Paper deep dive
Robust LLM Alignment via Distributionally Robust Direct Preference Optimization
Zaiyan Xu, Sushil Vemuri, Kishan Panaganti, Dileep Kalathil, Rahul Jain, Deepak Ramachandran
Models: GPT-2
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/12/2026, 6:40:26 PM
Summary
The paper introduces Distributionally Robust Direct Preference Optimization (DR-DPO), specifically Wasserstein DPO (WDPO) and Kullback-Leibler DPO (KLDPO), to address catastrophic alignment failures in LLMs caused by preference distribution shifts. By optimizing over an uncertainty set of preference distributions rather than a static dataset, the proposed methods achieve superior robustness and performance, supported by theoretical sample complexity guarantees and scalable gradient-based algorithms.
Entities (6)
Relation Signals (3)
WDPO → isa → Distributionally Robust Optimization
confidence 95% · We address this problem using the principled framework of distributionally robust optimization, and develop two novel distributionally robust direct preference optimization (DPO) algorithms, namely, Wasserstein DPO (WDPO)
KLDPO → isa → Distributionally Robust Optimization
confidence 95% · and Kullback-Leibler DPO (KLDPO)
WDPO → improves → LLM Alignment
confidence 90% · demonstrate the superior performance of WDPO and KLDPO in substantially improving the alignment when there is a preference distribution shift.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:A major challenge in aligning large language models (LLMs) with human preferences is the issue of distribution shift. LLM alignment algorithms rely on static preference datasets, assuming that they accurately represent real-world user preferences. However, user preferences vary significantly across geographical regions, demographics, linguistic patterns, and evolving cultural trends. This preference distribution shift leads to catastrophic alignment failures in many real-world applications. We address this problem using the principled framework of distributionally robust optimization, and develop two novel distributionally robust direct preference optimization (DPO) algorithms, namely, Wasserstein DPO (WDPO) and Kullback-Leibler DPO (KLDPO). We characterize the sample complexity of learning the optimal policy parameters for WDPO and KLDPO. Moreover, we propose scalable gradient descent-style learning algorithms by developing suitable approximations for the challenging minimax loss functions of WDPO and KLDPO. Our empirical experiments using benchmark data sets and LLMs demonstrate the superior performance of WDPO and KLDPO in substantially improving the alignment when there is a preference distribution shift.
Tags
Links
- Source: https://arxiv.org/abs/2502.01930
- Canonical: https://arxiv.org/abs/2502.01930
Trouble viewing inline? Open PDF directly →
Full Text
171,601 characters extracted from source content.
Expand or collapse full text
Robust LLM Alignment via Distributionally Robust Direct Preference Optimization Zaiyan Xu1, Sushil Vemuri1, Kishan Panaganti2, , Dileep Kalathil1, Rahul Jain3, Deepak Ramachandran3 1Texas A&M University Work done as postdoctoral researcher at the California Institute of Technology. 2Tencent AI Lab 3Google DeepMind. Emails: zxu43, sushil22, dileep.kalathil@tamu.edu kpb.research@gmail.com rahulajain, ramachandrand@google.com Abstract A major challenge in aligning large language models (LLMs) with human preferences is the issue of distribution shift. LLM alignment algorithms rely on static preference datasets, assuming that they accurately represent real-world user preferences. However, user preferences vary significantly across geographical regions, demographics, linguistic patterns, and evolving cultural trends. This preference distribution shift leads to catastrophic alignment failures in many real-world applications. We address this problem using the principled framework of distributionally robust optimization, and develop two novel distributionally robust direct preference optimization (DPO) algorithms, namely, Wasserstein DPO (WDPO) and Kullback–Leibler DPO (KLDPO). We characterize the sample complexity of learning the optimal policy parameters for WDPO and KLDPO. Moreover, we propose scalable gradient descent-style learning algorithms by developing suitable approximations for the challenging minimax loss functions of WDPO and KLDPO. Our empirical experiments using benchmark data sets and LLMs demonstrate the superior performance of WDPO and KLDPO in substantially improving the alignment when there is a preference distribution shift. 1 Introduction The alignment of large language models (LLMs) with human values and preferences is a central objective in machine learning, enabling these models to produce outputs that are useful, safe, and aligned with human intent. Since LLMs are trained on vast, diverse datasets using self-supervised learning, an additional alignment phase is often required to refine their behavior based on human feedback. A widely adopted approach for this is Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017; Ziegler et al., 2019; Ouyang et al., 2022), which involves training a reward model using human preference data and optimizing the LLM using reinforcement learning (RL) approaches, such as proximal policy optimization. More recently, Direct Preference Optimization (DPO) has emerged as an alternative that simplifies the alignment process by directly optimizing model parameters based on human preferences without requiring an explicit reward model. These alignment techniques have played a crucial role in improving the ability of LLMs to generate responses that adhere to human expectations and societal norms, leading to today’s powerful chat models (Achiam et al., 2023; Touvron et al., 2023). Despite the importance of the LLM alignment problem, RLHF and DPO remain fundamentally challenging and fragile, mainly due to three reasons. (i)(i) Diversity of human preferences: Standard RLHF/DPO approaches implicitly assume that human preferences can be accurately captured by a single reward function. In reality, human preferences are highly diverse, context-dependent, and distributional, making it infeasible to represent them with a one-size-fits-all optimization framework (Zhao et al., 2024; Durmus et al., 2024). Standard preference-learning methods tend to skew toward the preferences represented in the majority of training data, disproportionately penalizing minority opinions and reinforcing biases (Chakraborty et al., 2024). (ii)(i) Reward hacking: The quality of human preference feedback is inherently noisy, ambiguous, and inconsistent, as they are collected from human annotators who may lack domain expertise, exhibit labeling fatigue, or hold conflicting opinions (Zhang et al., 2025; Wu et al., 2025), which can often lead to misaligned preference estimation. This issue is exacerbated by reward hacking, where models learn undesirable shortcuts to maximize the estimated reward function, generating responses that appear aligned but deviate from genuine human intent (Amodei et al., 2016; Skalse et al., 2022; Eisenstein et al., 2024). (iii)(i) Distribution shift: Alignment algorithms use static preference datasets for training, collected under controlled conditions. However, the preferences of real-world users can often be out-of-distribution from that of the training data, depending on the geographical region, demography, linguistic patterns, and emerging social trends, among many others. A model aligned using a specific fixed dataset may fail catastrophically when deployed to users whose preference distribution does not match that of the training data (Casper et al., 2023; LeVine et al., 2023; Kirk et al., 2024). Figure 1: If the training population predominantly uses preference model 1 (P1), a non-robust RLHF/DPO model will favor Completion 1 (C1). However, deploying this model to a test population that prefers model 2 (P2), which favors Completion 2, leads to poor performance. Our distributionally robust DPO (WDPO/KLDPO) addresses this by optimizing across an uncertainty set of preference models, ensuring robust performance under preference shifts. In this paper, we address the fragility of the LLM alignment using DPO, with a particular focus on the challenges arising from the preference distribution shift. DPO reduces the alignment problem to a supervised learning problem. It is known that the performance of supervised learning algorithms degrades significantly in the out-of-distribution setting (Taori et al., 2020; Koh et al., 2021), which is exacerbated due to the realistic distribution shift scenarios arising in the LLM deployment. Distributionally robust optimization/learning framework has been recently used to address the issue of distribution shift in various settings (Duchi and Namkoong, 2021; Kuhn et al., 2019; Chen et al., 2020). This framework considers an uncertainty set of data distributions around a nominal distribution (typically the training data distribution) and solves a minimax optimization problem to minimize the expected loss, where the expectation is taken with respect to the distribution in the uncertainty set that maximizes the loss. The distributionally robust learning approach has been successfully applied, with theoretical guarantees and scalable algorithms, in supervised learning (Chen and Paschalidis, 2018; Namkoong and Duchi, 2016; Levy et al., 2020), multi-armed bandits (Si et al., 2020; Yang et al., 2023), and reinforcement learning (Wang and Zou, 2022; Panaganti et al., 2022; Zhou et al., 2024; Xu et al., 2023). This motivates us to address the following questions: Can distributionally robust learning mitigate the impact of distribution shift in DPO-based LLM alignment? What theoretical guarantees can be established for such methods? How can we design tractable, gradient-based algorithms to implement them? How do these approaches empirically improve alignment performance? Distributionally robust learning for LLM alignment presents challenges beyond standard supervised settings. In supervised learning, distributional robustness is often tractable due to well-behaved convex losses. In contrast, RL poses more complex forms of distribution shift, both exogenous (e.g., user preference drift) and endogenous (e.g., mismatch between the learned and logging policies). Although DPO is framed as a supervised objective, its likelihood-ratio formulation, based on pairwise comparisons and a reference policy, derives from KL-regularized reward maximization, linking it closely to RL. Like imitation learning and offline RL, which address RL problems through supervised proxies (e.g., behavior cloning, Q-function regression), DPO inherits the same instability and sensitivity to distribution shift. These challenges are amplified in the distributionally robust setting, where the non-convex min-max objective is especially hard to optimize at LLM scale, making standard alternating-gradient methods unstable and impractical. We answer the above questions affirmatively and address the associated challenges through the following contributions: 1. To the best of our knowledge, this is the first work to propose a unified mathematical and algorithmic framework for addressing preference shift in LLM alignment through distributionally robust optimization. Our formulation leads to two robust DPO variants, Wasserstein DPO (WDPO) and Kullback-Leibler DPO (KLDPO), with provable guarantees. In particular, for log-linear policies, we show that the estimation error of the robust policy parameters converges at a rate of O(n−1/4)O(n^-1/4). 2. We develop computationally tractable gradient descent algorithms for WDPO and KLDPO that can be seamlessly integrated into existing LLM alignment pipelines. 3. Empirically, we show that standard DPO is sensitive to preference distribution shift, leading to degraded performance when training and evaluation rewards differ. In contrast, our robust variants, WDPO and KLDPO, consistently achieve superior performance across diverse alignment tasks. For example, we fine-tune LLaMA-3.2-1B/3B-Instruct and LLaMA-3.1-8B-Instruct models on prompts from the HelpSteer2 dataset (Wang et al., 2024b) using preferences generated by the ArmoRM reward model (Wang et al., 2024a), and evaluate them on distinct reward objectives from the OpenLLM Leaderboard (Fourrier et al., 2024). 2 Related Work Robust RLHF: Bai et al. (2022) proposed to adjust weights on the combination of loss functions based on different topics (harmless vs. helpful) for robust reward learning. Chakraborty et al. (2024) proposed to learn multiple reward functions for different sub-populations through an expectation-maximization approach, and a robust policy based on these rewards via a max-min optimization, which is different from our distributional robust learning approach. Padmakumar et al. (2024) augmented the existing binary preference datasets with synthetic preference judgments to estimate the diversity of user preferences. Yan et al. (2024) proposed a Bayesian reward model ensemble to quantify the uncertainty of reward estimation and used it to reduce reward overoptimization. Bukharin et al. (2024) proposed a robust RLHF approach for addressing the preference data corruption problem. Robust DPO: Huang et al. (2025) proposed χ that implements the principle of pessimism in the face of uncertainty via regularization with the χ2χ^2-divergence for avoiding reward hacking/overoptimization w.r.t. the estimated reward. Ramesh et al. (2024) proposed Group Robust Preference Optimization (GRPO) to address the diverse preference problem by modeling the total loss as a weighted sum of individual DPO losses computed on separate preference datasets, and optimizing for the worst-case weighting. In contrast, our approach does not assume access to such a group structure and instead directly models distributional robustness over a single dataset that implicitly aggregates diverse preferences. Chowdhury et al. (2024) considered the setting where ϵε-fraction of the preference labels in the training dataset is corrupted and proposed a noise-robust algorithm to mitigate its effect, assuming the knowledge of ϵε. Wu et al. (2024) focused on adapting the DPO penalty parameter β to handle varying data quality within the training set. The most related work is Wu et al. (2025), which applies distributional robustness to mitigate data corruption and noise in preference data. Unlike our work, it does not address distribution shift or provide theoretical guarantees, and lacks empirical evaluation on preference distribution shift. Concurrent to our work, Mandal et al. (2025) proposed a distributionally robust version of RLHF and DPO, using total variation (TV) uncertainty sets. However, their theoretical analysis offers the natural policy gradient (NPG) style optimization convergence guarantees for the loss function. In contrast, we go one step further: by leveraging strong convexity, we establish finite-sample guarantees not just for the learning loss, but for convergence of the policy parameters. Our analysis is algorithm-agnostic and applies to any solver capable of optimizing the robust DPO loss. Additionally, our formulation uses KL and Wasserstein uncertainty sets, which are more standard in large-scale LLM alignment. Distributionally Robust Learning: Distributionally robust learning is a statistical learning framework designed to enhance model performance under distributional shifts between training and test data (Chen and Paschalidis, 2018). It employs a minimax approach where an adversary maximizes the expected loss by shifting the test distribution within a specified uncertainty set, while the learner minimizes this adversarial loss. This approach using the f-divergence (Namkoong and Duchi, 2016; Duchi and Namkoong, 2021; Levy et al., 2020) and the Wasserstein metric (Mohajerin Esfahani and Kuhn, 2018; Kuhn et al., 2019; Gao et al., 2022) have gained significant attention recently. Distributionally robust algorithms have been developed to address problems in supervised learning (Chen and Paschalidis, 2018; Namkoong and Duchi, 2016; Levy et al., 2020), imitation learning (Bashiri et al., 2021; Panaganti et al., 2023), multi-armed bandits (Si et al., 2020; Yang et al., 2023), and reinforcement learning (Panaganti et al., 2022; Zhou et al., 2024; Shi and Chi, 2024; Yang et al., 2022; Panaganti et al., 2025). 3 Preliminaries Notations: We use calligraphic letters for sets, e.g., S. ∥⋅∥ · denotes the Euclidean norm. When Σ is a positive semi-definite matrix, we write ∥x∥Σ=x⊤Σx x _ = x x as a semi-norm of x. For any measure P, we use n P_n to denote the empirical distribution constructed using n i.i.d. samples, x1,…,xnx_1,…,x_n, from P, i.e., n=(1/n)∑i=1nδxi P_n=(1/n) _i=1^n _x_i, where δx _x is the Dirac measure. We use σ to denote the sigmoid (standard logistic) function. We use l(z;θ)l(z;θ) and lz(θ)l_z(θ) to denote the loss incurred by sample z with policy parameter θ. For any set Z, ()P(Z) is the set of all Borel measures over Z. For any positive semi-definite matrix Σ , λmin(Σ) _ min( ) and λmax(Σ) _ max( ) denote its smallest and largest eigenvalues. Wasserstein Distance: For a given set Z, equipped with a metric d, the Wasserstein distance of order p between two distributions μ,ν∈()μ,ν (Z) is defined as (see Villani et al. (2009)): p(μ,ν)=minγ∈(×)∫×dp(x,x′)γ(dx,dx′):γ has marginal distributions μ,ν. W_p(μ,ν)= _γ (Z×Z) \ _Z×Zd^p(x,x )γ(dx,dx ) $γ$ has marginal distributions $μ,ν$ \. Kullback-Leibler Divergence: For any two probability distributions P and Q defined on Z, the Kullback-Leibker (KL) divergence is defined as DKL(∥)=∑z∈(z)log((z)/(z)).D_KL( P\;\|\; Q)= _z P(z) ( P(z)/ Q(z)). Reinforcement Learning from Human Feedback: The RLHF paradigm consists of three steps: Step 1: Supervised Fine-tuning (SFT). SFT involves fine-tuning a pre-trained LLM through supervised learning on high-quality data, curated for the downstream tasks. Step 2: Reward Modelling. In the second step, given any context s∈s , two responses a1,a2∈a^1,a^2 are independently sampled from the behavior policy πoπ^o (typically the SFT policy πSFT _SFT). Then, a (human) labeler provides a preference response between these responses. We assume that the preference responses are generated according to the Bradley-Terry (BT) model (Bradley and Terry, 1952): P∗(a1≻a2∣s)=exp(r∗(s,a1))exp(r∗(s,a1))+exp(r∗(s,a2)),P^*(a^1 a^2 s)= exp (r^*(s,a^1) )exp (r^*(s,a^1) )+exp (r^*(s,a^2) ), (1) where a1≻a2a^1 a^2 denotes a1a^1 being preferred over a2a^2, and r∗r^* is the underlying unknown reward function. We use aw,ala^w,a^l to denote the preferred and dis-preferred responses, respectively. We assume access to a static dataset of comparison, =(si,aiw,ail)i=1nD=\(s_i,a^w_i,a^l_i)\_i=1^n, where sis_i’s are sampled from some initial prompt (context) distribution μoμ^o, ai1,ai2a^1_i,a^2_i’s are independently sampled from πSFT _SFT, and the preferences responses are sampled from the BT model P∗P^*. With D, we can learn a parameterized reward model rϕ(s,a)r_φ(s,a) by minimizing the maximum likelihood estimation (MLE) loss, ℒRLHF(rϕ;)=−(s,aw,al)∼[logσ(rϕ(s,aw)−rϕ(s,al))].L^RLHF(r_φ;D)=-E_(s,a^w,a^l) [ σ(r_φ(s,a^w)-r_φ(s,a^l))]. Step 3: RL Fine-Tuning. In the final step, the optimal policy π∗π^* under the reward rϕr_φ is obtained by solving the KL-regularized reward maximization problem given by maxπs∼μ[a∼π(⋅∣s)[rϕ(s,a)]−βDKL(π(⋅∣s)∥πref(⋅∣s))], _πE_s μ [E_a π(· s)[r_φ(s,a)]-β D_KL(π(· s)\;\|\; _ref(· s)) ], (2) where β is a parameter controlling the deviation from the base reference policy πref _ref. Direct Preference Optimization (DPO): The DPO approach (Rafailov et al., 2023) leverages the fact that the unknown reward function can be expressed in terms of the optimal policy and the reference policy. Formally, given any reward function r∗r^*, the optimal solution of Eq.˜2 takes the form π∗(a∣s)=1Z∗(s)πref(a∣s)exp(r∗(s,a)/β)π^*(a s)= 1Z^*(s) _ref(a s)exp (r^*(s,a)/β ), where Z∗(s)Z^*(s) denotes the partition (normalizing) function. Rearranging the above, we get r∗(s,a)=βlogπ∗(a∣s)πref(a∣s)+βlogZ∗(s)r^*(s,a)=β π^*(a s) _ref(a s)+β Z^*(s) for all (s,a)(s,a). Substituting this into Eq.˜1, the optimal RLHF policy π∗π^* satisfies the preference model: P∗(a1≻a2∣s)=σ(βlogπ∗(a1∣s)πref(a1∣s)−βlogπ∗(a2∣s)πref(a2∣s)).P^*(a^1 a^2 s)=σ (β π^*(a^1 s) _ref(a^1 s)-β π^*(a^2 s) _ref(a^2 s) ). Using the preference response dataset D, we can learn the optimal policy directly by minimizing the MLE loss for a parameterized policy πθ _θ, ℒDPO(πθ;)=−(s,aw,al)∼[logσ(βlogπθ(aw∣s)πref(aw∣s)−βlogπθ(al∣s)πref(al∣s))].L^DPO( _θ;D)=-E_(s,a^w,a^l) [ σ (β _θ(a^w s) _ref(a^w s)-β _θ(a^l s) _ref(a^l s) ) ]. (3) Distributional Uncertainty Sets: Given any ρ>0ρ>0 and o∈() P^o (Z), we define the distributional uncertainty set as (ρ;o)≔∈():D(,o)≤ρ,P(ρ; P^o) \ P (Z) D( P, P^o)≤ρ\, (4) where D(⋅,⋅)D(·,·) is some distance metric between two probability measures, e.g., p W_p and DKLD_KL. 4 Distributionally Robust DPO In this section, we formulate our Wasserstein DPO (WDPO) and Kullback-Leibler DPO (KLDPO). Sampling Procedure: As described in Section˜3, a prompt s∈s is drawn from an initial distribution μoμ^o, and two responses a1,a2∼i.i.d.πo(⋅∣s)a^1,a^2 _i.i.d.π^o(· s) are sampled independently (with πo=πSFTπ^o= _SFT in practice). Following Zhu et al. (2023), we define y∈0,1y∈\0,1\ to indicate preference: y=1y=1 if a1≻a2∣sa^1 a^2 s and y=0y=0 otherwise. The label y is drawn from a Bernoulli distribution defined by the BT model P∗P^*. The full data-generating distribution is given below. Definition 1 (Joint data-generating distribution). Consider the product space ≔×0,1Z ×A×A×\0,1\. We define the nominal data-generating distribution as o(s,a1,a2,y)=μo(s)πo(a1∣s)πo(a2∣s)⋅[y=1P∗(a1≻a2∣s)+y=0P∗(a2≻a1∣s)]. P^o(s,a^1,a^2,y)=μ^o(s)π^o(a^1 s)π^o(a^2 s)·[ 1_\y=1\P^*(a^1 a^2 s)+ 1_\y=0\P^*(a^2 a^1 s)]. We will also denote z=(s,a1,a2,y)∈z=(s,a^1,a^2,y) and o(z)=o(s,a1,a2,y) P^o(z)= P^o(s,a^1,a^2,y). We assume that o P^o generates the dataset =zii=1nD=\z_i\_i=1^n used for learning, i.e., zi∼oz_i P^o. 4.1 Distributionally Robust DPO From the DPO objective (Eq.˜3), we define the pointwise DPO loss function as follows l(z;θ)=−ylogσ(βhθ(s,a1,a2))−(1−y)logσ(βhθ(s,a2,a1)),l(z;θ)=-y σ(β h_θ(s,a^1,a^2))-(1-y) σ(β h_θ(s,a^2,a^1)), (5) where hθ(s,a1,a2)≔logπθ(a1∣s)πref(a1∣s)−logπθ(a2∣s)πref(a2∣s)h_θ(s,a^1,a^2) _θ(a^1 s) _ref(a^1 s)- _θ(a^2 s) _ref(a^2 s) is the preference score of an answer a1a^1 relative to another one a2a^2 (but parameterized in policy parameter θ)θ). Let (ρ;o)P(ρ; P^o) be a distributional uncertainty set centered around o P^o with radius ρ>0ρ>0. Following the principles of distributionally robust optimization (DRO), we formulate the distributionally robust DPO objective as: minθmax∈(ρ;o)z∼[l(z;θ)]. _θ _ P (ρ; P^o)E_z P[l(z;θ)]. (6) Intuitively, we aim to find the best policy under the worst-case data distribution. When we have a Wasserstein uncertainty set pP_ W_p, i.e., Eq.˜4 equipped with the p-th order Wasserstein distance, we define the Wasserstein DPO (WDPO) loss as follows ℒW(θ;ρ)=sup∈p(ρ;o)z∼[l(θ;z)],L^W(θ;ρ)= _ P _ W_p(ρ; P^o)E_z P[l(θ;z)], (7) Similarly, given a Kullback-Leibler uncertainty set KL(ρ;o)P_KL(ρ; P^o), we define the KLDPO loss as follows ℒKL(θ;ρ)=sup∈KL(ρ;o)z∼[l(θ;z)].L^KL(θ;ρ)= _ P _KL(ρ; P^o)E_z P[l(θ;z)]. (8) When the nominal distribution o P^o is replaced with its empirical counterpart, i.e., no≔(1/n)∑i=1nδzi P^o_n (1/n) _i=1^n _z_i, where z1,…,znz_1,…,z_n are n i.i.d. samples from o P^o, we use ℒnW(θ;ρ)L^W_n(θ;ρ) and ℒnKL(θ;ρ)L^KL_n(θ;ρ) to denote the empirical WDPO and KLDPO losses incurred by the policy parameter θ, respectively. 5 Theoretical Analysis In this section, we present the sample complexity guarantees for our WDPO and KLDPO algorithms. We make the following assumptions for the rest of the papers. Assumption 1 (Log-linear policy class). Let ψ:×→ℝdψ ×A ^d be a known d-dimensional feature mapping with maxs,a∥ψ(s,a)∥2≤1 _s,a ψ(s,a) _2≤ 1. Assume a bounded policy parameter set Θ≔θ∈ℝd:∥θ∥2≤B \θ ^d θ _2≤ B\. We consider the following class of log-linear policies: Π=πθ:πθ(a∣s)=exp(θ⊤ψ(s,a))∑a′∈exp(θ⊤ψ(s,a′)). = \ _θ _θ(a s)= exp (θ ψ(s,a) ) _a exp (θ ψ(s,a ) ) \. (9) Remark 1. This is a standard assumption in the theoretical analysis of the RL algorithms (Agarwal et al., 2021; Modi et al., 2020), RLHF (Zhu et al., 2023), and DPO (Nika et al., 2024; Chowdhury et al., 2024). Our analysis can be extended to the neural policy class where θ⊤ψ(s,a)θ ψ(s,a) is replaced fθ(s,a)f_θ(s,a), where fθf_θ is a neural network with twice differentiability and smoothness assumptions. We also make the following data coverage assumption on the uncertainty set (ρ;o)P(ρ; P^o). Assumption 2 (Regularity condition). There exists λ>0λ>0 such that Σ≔(s,a1,a2,y)∼[(ψ(s,a1)−ψ(s,a2))(ψ(s,a1)−ψ(s,a2))⊤]⪰λI,∀∈(ρ;o). _ P _(s,a^1,a^2,y) P[(ψ(s,a^1)-ψ(s,a^2))(ψ(s,a^1)-ψ(s,a^2)) ] λ I, ∀ P (ρ; P^o). Remark 2. We note that similar assumptions on data coverage under linear architecture models are standard in the offline RL literature (Agarwal et al., 2019; Wang et al., 2021; Jin et al., 2021). Implicitly, Assumption˜2 imposes λ≤λmin(Σo)λ≤ _ min( _ P^o), which means that the data-generating distribution o P^o has good coverage. 5.1 Estimation Error for WDPO Let θ∗∈argminθ∈ΘℒDPO(θ)θ^*∈ *argmin_θ∈ L^DPO(θ) be the ground-truth optimal policy parameter with respect to the true nominal distribution and let its empirical counterpart be θn∈argminθ∈ΘℒnDPO(θ) _n∈ *argmin_θ∈ L^DPO_n(θ). Now for the robust policy parameters, we let θW∈argminθ∈ΘℒW(θ;ρ)θ^W∈ *argmin_θ∈ L^W(θ;ρ), and let its empirical counterpart be θnW∈argminθ∈ΘℒnW(θ;ρ)θ^W_n∈ *argmin_θ∈ L^W_n(θ;ρ). Now, present our main result on the sample complexity result for the convergence of the robust policy parameter. Theorem 1 (Estimation error of θnWθ^W_n). Let δ∈(0,1)δ∈(0,1). With probability at least 1−δ1-δ, we have ∥θnW−θW∥22≤8K2log(2/δ)γ2λ2n, θ^W_n-θ^W ^2_2≤ 8K^2 (2/δ)γ^2λ^2n, where γ=β2e4βB(1+e4βB)2γ= β^2e^4β B(1+e^4β B)^2 and K=|logσ(−4βB)|K= σ(-4β B) , λ is the regularity number defined in Assumption˜2. Proof sketch. Strong duality of Wasserstein DRO (see Gao and Kleywegt (2022) and Corollary˜1) helps us reduce the difference |ℒW(θ;ρ)−ℒnW(θ;ρ)| ^W(θ;ρ)-L^W_n(θ;ρ) to the concentration |z∼o[lη(z;θ)]−z∼no[lη(z;θ)]| _z P^o[l_η(z;θ)]-E_z P^o_n[l_η(z;θ)] , where lη(z;θ)=infz∈[ηdp(z,z′)−l(z;θ)]l_η(z;θ)= _z [η d^p(z,z )-l(z;θ)] is called the Moreau-Yosida regularization of −l-l with parameter 1/η1/η. We show that, for all η≥0η≥ 0, all lηl_η are uniformly bounded. We then use Hoeffding’s inequality to obtain concentration. Detailed proof is in Section˜B.2. Next, when Assumption˜2 is in place, we can show that g(θ)≔z∼[l(z;θ)]g(θ) _z P[l(z;θ)] is γ-strongly convex w.r.t. the positive definite norm ∥⋅∥Σ · _ _ P. Further, by the property of supremum, we can show that ℒWL^W is γλγλ-strongly convex but w.r.t. ∥⋅∥2 · _2. A detailed proof is provided in Section˜B.3. Decompose ℒW(θnW)−ℒW(θW)L^W(θ^W_n)-L^W(θ^W) into three terms: ℒW(θnW;ρ)−ℒnW(θnW;ρ)L^W(θ^W_n;ρ)-L^W_n(θ^W_n;ρ), ℒnW(θnW;ρ)−ℒnW(θW;ρ)L^W_n(θ^W_n;ρ)-L^W_n(θ^W;ρ), and ℒnW(θW;ρ)−ℒW(θW;ρ)L^W_n(θ^W;ρ)-L^W(θ^W;ρ). The second term is non-positive since θnWθ^W_n is the minimizer of ℒnWL^W_n. Now we apply the concentration of the WDPO loss function (see Lemma˜9 in Section˜B.2) to |ℒW(θnW;ρ)−ℒnW(θnW;ρ)| ^W(θ^W_n;ρ)-L^W_n(θ^W_n;ρ) and |ℒnW(θW;ρ)−ℒW(θW;ρ)| ^W_n(θ^W;ρ)-L^W(θ^W;ρ) . Finally, we use the property of strongly convex function (Lemma˜5) on ℒWL^W to acquire the policy parameter convergence. The detailed proof is in Section˜B.4. ∎ We state the convergence result for DPO to facilitate comparison with its robust counterpart. Proposition 1 (Estimation error of (non-robust) DPO). Let δ∈(0,1)δ∈(0,1) and β>0β>0. ∥θn−θ∗∥Σ+λI≤24β2γ2n(d+log(1/δ))+2λB2, _n-θ^* _ _D+λ I≤ 2 4β^2γ^2n(d+ (1/δ))+2λ B^2, with probability at least 1−δ1-δ and where γ=β2e4βB(1+e4βB)2γ= β^2e^4β B(1+e^4β B)^2, and Σ=1n∑i=1n(ψ(si,ai1)−ψ(si,ai2))(ψ(si,ai1)−ψ(si,ai2))⊤ _D= 1n _i=1^n(ψ(s_i,a^1_i)-ψ(s_i,a_i^2))(ψ(s_i,a^1_i)-ψ(s_i,a_i^2)) is the sample covariance matrix. A matching result can be derived as a special case of Chowdhury et al. (2024, Theorem 4.2). We provide an independent proof with precise constants in Section˜B.1. Remark 3. We would like to note that the estimation error rate of convergence for WDPO is ∥θnW−θW∥2=O(n−1/4) θ^W_n-θ^W _2=O(n^-1/4), from Theorem˜1. The estimation error rate of convergence for (non-robust) DPO is ∥θn−θ∗∥Σ+λI=O(n−1/2) _n-θ^* _ _D+λ I=O(n^-1/2), from Proposition˜1. So, the estimation error rate of convergence for WDPO is worse than that of (non-robust) DPO. This arises due to significant challenges exclusive to the robust setting. For example, for the non-robust DPO, we can calculate the closed-form expression of ∇θ(1/n)∑i=1nl(zi;θ) _θ(1/n) _i=1^nl(z_i;θ) (see Eq.˜20). This allows us to write ∥∇θ(1/n)∑i=1nl(zi;θ∗)∥(Σ+λI)−1 _θ(1/n) _i=1^nl(z_i;θ^*) _( _D+λ I)^-1 in quadratic form and then obtain a concentration using Bernstein’s inequality. However, for WDPO, we note that ∇θℒnW(θW)≠sup∈p∇θz∼[l(z;θW)] _θL^W_n(θ^W)≠ _ P _ W_p _θE_z P[l(z;θ^W)], and the non-robust approach will not work for the robust setting. Developing analysis techniques to achieve a better rate of convergence for robust DPO is an open question. 5.2 Estimation Error for KLDPO Let θKL∈argminθ∈ΘℒKL(θ;ρ)θ^KL∈ *argmin_θ∈ L^KL(θ;ρ), and let its empirical counterpart be θnKL∈argminθ∈ΘℒnKL(θ;ρ)θ^KL_n∈ *argmin_θ∈ L^KL_n(θ;ρ). The convergence analysis for the KLDPO loss and policy parameter closely parallels that of Wasserstein DPO. We present the main theorems below and defer detailed proofs to Appendix˜C. Theorem 2 (Estimation error of θnKLθ^KL_n). Let δ∈(0,1)δ∈(0,1). With probability at least 1−δ1-δ, we have ∥θnKL−θKL∥22≤8λ¯2exp(L/λ¯)log(2/δ)γ2λ2n, θ^KL_n-θ^KL ^2_2≤ 8 λ^2exp (L/ λ ) (2/δ)γ^2λ^2n, where γ=β2e4βB(1+e4βB)2γ= β^2e^4β B(1+e^4β B)^2. λ is the regularity condition number defined in Assumption˜2, 0<λ≤λmin(Σo)0<λ≤ _ min( _ P^o). λ¯,λ¯ λ, λ are some universal constants, and L is an upper bound on the loss function l. Remark 4. The exponential constant in the upper bound is a characteristic of distributional robust optimization with KL uncertainty set Hu and Hong (2013, Proposition 2). Similar exponential constants appear in the theoretical analysis of the distributionally robust RL (Zhou et al., 2021; Yang et al., 2022; Panaganti and Kalathil, 2022; Xu et al., 2023). Both WDPO and KLDPO have O(n−1/4)O(n^-1/4) policy parameter convergence. An empirical comparison is given in Section˜7. 6 Tractable (Approximate) Algorithms Algorithm 1 WDPO Algorithm 1:Input: Dataset =(si,aiw,ail)i=1nD=\(s_i,a^w_i,a^l_i)\_i=1^n, reference policy πref _ref, robustness hyperparameter ρo _o, learning rate η, initial policy πθ _θ. 2:while θ has not converged do 3: Calculate the non-robust DPO loss ℒDPO(πθ;)L^DPO( _θ;D) according to Eq.˜3 4: Calculate the gradient regularizer loss ℛ(πθ;)=ρo(z∼∥∇zl(z;θ)∥22)1/2R( _θ;D)= _o(E_z _zl(z;θ) _2^2)^1/2 -2.84544pt 5: Calculate the approximate WDPO loss ℒW(θ;)≔ℒDPO(πθ;)+ℛ(πθ;)L^W(θ;D) ^DPO( _θ;D)+R( _θ;D) 6: θ←θ−η∇θℒW(θ;)θ←θ-η _θL^W(θ;D) 7:Output: πθ _θ Algorithm 2 KLDPO Algorithm 1:Input: Dataset =(si,aiw,ail)i=1nD=\(s_i,a^w_i,a^l_i)\_i=1^n, reference policy πref _ref, robustness temperature parameter τ, learning rate η, initial policy πθ _θ. 2:while θ has not converged do 3: Approximate the worst-case kernel ¯(i)∝exp((1/τ)( P(i) ((1/τ)( l(zi;θ) l(z_i;θ) −(1/n)∑i=1nl(zi;θ))) -(1/n) Σ _i=1^nl(z_i;θ))) 4: Calculate the approximate KLDPO loss ℒKL(θ;)≔∑i=1n¯(i)⋅l(zi;θ)L^KL(θ;D) Σ _i=1^n P(i)· l(z_i;θ) 5: θ←θ−η∇θℒKL(θ;)θ←θ-η _θL^KL(θ;D) 6:Output: πθ _θ While our distributionally robust DPO formulations enjoy finite-sample guarantees, it is computationally challenging to solve the min-max objective of Eq.˜6 using stochastic gradient descent methods. Though many min-max optimization problems can be solved by alternating gradient descent methods, our problem is not directly amenable to such an approach as we do not have direct control over the data distribution ∈(ρ;o) P (ρ; P^o) which is not parameterized. Moreover, the preference data are generated according to the nominal distribution o P^o ,and we do not have data samples from any other distributions in the uncertainty set (ρ;o)P(ρ; P^o). To overcome this challenge, we introduce principled tractable algorithms to solve WDPO and KLDPO. Tractable WDPO: The connection between Wasserstein distributionally robust optimization (DRO) and regularization has been established in various settings by many (Mohajerin Esfahani and Kuhn, 2018; Shafieezadeh-Abadeh et al., 2019; Chen and Paschalidis, 2018). We leverage the recent progress in Wasserstein theory on connecting Wasserstein distributionally robust optimization to regularization. For p-Wasserstein DRO, p∈(1,∞]p∈(1,∞], Gao et al. (2022) shows that for a broad class of loss functions, possibly non-convex and non-smooth, with high probability, the Wasserstein DRO is asymptotically equivalent to variation regularization. In particular, an immediate consequence of Gao et al. (2022, Theorem 1) is that, when p=2p=2, minθ∈Θsup:p(,no)≤ρnz∼[l(z;θ)]=minθ∈Θz∼no[l(z;θ)]+ρn(1/n)∑i=1n∥∇zl(zi;θ)∥22+Op(1/n), _θ∈ _ P W_p( P, P^o_n)≤ _nE_z P[l(z;θ)]= _θ∈ \E_z P^o_n[l(z;θ)]+ _n (1/n) Σ _i=1^n _zl(z_i;θ) _2^2 \+O_p(1/n), where ρn=O(1/n) _n=O(1/ n). That is, one can solve the Wasserstein DRO objective by adding a gradient regularization to the empirical risk minimization (ERM) loss, z∼no[l(z;θ)]E_z P^o_n[l(z;θ)]. Based on this, we propose a tractable WDPO algorithm in Algorithm˜1. Note that the gradient regularizer has a sample-size-dependent coefficient. In practice, we absorb the factor ρn/n _n/ n into ρo _o, which we treat as a tunable robustness hyperparameter. Tractable KLDPO: The following proposition shows that we can approximate the worst-case probability distribution in a KL uncertainty set w.r.t. a given loss function. Similar results can also be found in distributionally robust reinforcement learning literature (e.g., Gadot et al. (2024)). Proposition 2 (Worst-case distribution (informal)). Let ¯∈ℝn P ^n be the worst-case distribution w.r.t. a loss function l and KL uncertainty around the empirical distribution no P_n^o, defined as ¯=sup:DKL(∥no)≤ρz∼[l(z;θ)] P= _ P D_KL( P\;\|\; P_n^o)≤ρE_z P[l(z;θ)]. The worst-case distribution ¯ P is related to no P_n^o through ¯(i)∝no(i)⋅exp((1/τ)(l(zi;θ)−∑i=1nno(i)l(zi;θ))), P(i) P_n^o(i)·exp((1/τ)(l(z_i;θ)- _i=1^n P_n^o(i)l(z_i;θ))), where τ>0τ>0 is some constant. We defer the formal proof of Proposition˜2 to Appendix˜D. It can be viewed as a re-weighting threshold: extreme losses are more biased towards the baseline empirical DPO loss. τ controls the intensity of re-weighting, acting as a temperature parameter. Based on Proposition˜2, we propose a tractable KLDPO algorithm in Algorithm˜2. 7 Experiments Figure 2: DPO, WDPO, and KLDPO in Emotion Alignment. Models are trained on preferences derived from convex (left two plots) and geometric (right two plots) mixtures of anger and fear objectives from the Emotion dataset (Saravia et al., 2018). To simulate preference shift, evaluation is performed at mixing coefficients α≠αoα≠ _o, where αo=0.1 _o=0.1 is used during training. We evaluate WDPO with robustness parameter ρo∈50,75,100 _o∈\50,75,100\ and KLDPO with robustness temperature τ∈0.5,0.75,1τ∈\0.5,0.75,1\. Additional experimental details are provided in Section˜7.1. We conduct experiments across three distinct settings that vary in dataset scale, model size, and the degree of distribution shift. For example, we fine-tune LLaMA-3.2-1B-Instruct, LLaMA-3.2-3B-Instruct, and LLaMA-3.1-8B-Instruct models on prompts from the HelpSteer2 dataset (Wang et al., 2024b), using preferences derived from the ArmoRM reward model (Wang et al., 2024a), and evaluate them on the OpenLLM Leaderboard v2 (Fourrier et al., 2024). Additional evaluations are provided in Appendix˜E. We provide the code at https://github.com/TheBlackCat22/distributionally_robust_dpo. 7.1 Experimental Setup Emotion Alignment: We use the Emotion dataset (Saravia et al., 2018) to train a GPT-2 model (Radford et al., 2019) with a classification head for multi-label classification over five emotions: sadness, joy, love, anger, fear. The resulting sigmoid outputs are used as a multi-objective reward model for the remainder of this experiment. We also take a GPT-2 model and perform supervised fine-tuning (SFT) with the Emotion dataset to obtain our base model for preference alignment. To construct preference data, we mix objectives derived from our reward model. Specifically, we consider two reward objectives, r1,r2r_1,r_2 and define two mixture reward functions (1) convex mixing rconvex∗(α)≔α⋅r1+(1−α)⋅r2r^*_convex(α) α· r_1+(1-α)· r_2 and (2) geometric mixing rgeometric∗(α)≔r1α⋅r21−αr^*_geometric(α) r_1^α· r_2^1-α. For both reward functions, we generate two completions per prompt and assign preference labels using the Bradley-Terry (BT) model parameterized by r∗(αo)r^*(α^o) for a chosen αo∈[0,1]α^o∈[0,1]. ArmoRM Multi-objective Alignment: We use the Absolute-Rating Multi-Objective Reward Model (ArmoRM) (Wang et al., 2024a) to define reward preferences, selecting pairs of equally weighted objectives (e.g., honesty, verbosity, safety) from its 19-dimensional first-stage outputs. Using Meta LLaMA-3.2-1B-Instruct as the base model, we generate two completions per prompt from the HelpSteer2 dataset (Wang et al., 2024b) and train models on preferences derived from the convex mixing of these reward pairs. We evaluate all models on five individual ArmoRM objectives, three of which are unseen during training, to simulate preference shift. Leaderboard Alignment: We fine-tune LLaMA-3.2-1B-Instruct, LLaMA-3.2-3B-Instruct, and LLaMA-3.1-8B-Instruct models using preference data derived from the scalar rewards produced by the second stage of the ArmoRM reward model (Wang et al., 2024a). For each prompt from the HelpSteer2 dataset, we generate 10 responses, score them with ArmoRM, and constructe preference pairs by selecting the highest- and lowest-scoring completions. The models are evaluated on the OpenLLM Leaderboard v2 (Fourrier et al., 2024) using the LM Evaluation Harness (Gao et al., 2024). 7.2 Experiment Results Figure 3: DPO, WDPO, and KLDPO in ArmoRM Multi-objective Alignment. LLaMA-3.2-1B-Instruct models are trained on preferences derived from three equally weighted objective pairs: (1) Ultrafeedback-Truthfulness and Helpsteer-Complexity, (2) Ultrafeedback-Helpfulness and Helpsteer-Coherence, and (3) Helpsteer-Correctness and Helpsteer-Helpfulness (left to right plots). We train all models for 4 epochs. To simulate preference shift, models are evaluated on five individual objectives, Helpsteer-Helpfulness, Helpsteer-Correctness, Helpsteer-Coherence, Ultrafeedback-Honesty, and the overall ArmoRM score, three of which were not used during training. LLaMA-3.2-1B IFEval BBH MATH GPQA MUSR MMLU LLaMA-3.2-3B IFEval BBH MATH GPQA MUSR MMLU DPO at Epoch 2 (early stopping) 0.48 0.35 0.08 0.27 0.35 0.17 DPO 0.55 0.45 0.08 0.24 0.36 0.30 DPO at Epoch 4 (goodfit) 0.48 0.34 0.07 0.26 0.33 0.17 KLDPO (τ=0.005)(τ=0.005) 0.74 0.46 0.19 0.26 0.35 0.32 DPO at Epoch 6 (overfit) 0.48 0.33 0.06 0.26 0.33 0.17 WDPO (ρo=0.005)( _o=0.005) 0.62 0.45 0.06 0.25 0.36 0.30 KLDPO (τ=0.1)(τ=0.1) 0.53 0.36 0.08 0.25 0.33 0.18 LLaMA-3.1-8B IFEval BBH MATH GPQA MUSR MMLU KLDPO (τ=0.05)(τ=0.05) 0.56 0.36 0.08 0.26 0.32 0.18 DPO 0.62 0.50 0.03 0.29 0.44 0.33 WDPO (ρo=0.01)( _o=0.01) 0.52 0.36 0.09 0.25 0.34 0.19 KLDPO (τ=0.005)(τ=0.005) 0.72 0.51 0.24 0.29 0.34 0.37 WDPO (ρo=0.005)( _o=0.005) 0.49 0.35 0.09 0.25 0.33 0.19 KLDPO (τ=0.01)(τ=0.01) 0.75 0.51 0.22 0.31 0.36 0.37 Table 1: Evaluation of DPO, KLDPO, and WDPO on OpenLLM Leaderboard v2. LLaMA-3.2-1B/3B-Instruct and LLaMA-3.1-8B-Instruct models are trained on preferences generated according to ArmoRM score and then evaluated on OpenLLM Leaderboard v2, which benchmarks LLMs across six tasks: Massive Multitask Language Understanding (MMLU), Google-Proof Q&A Benchmark (GPQA), Multistep Soft Reasoning (MUSR), Mathematics Aptitude Test of Heuristics (MATH), Instruction Following Evaluation (IFEval), and Big Bench Hard (BBH). Emotion Alignment Results: In Fig.˜2, we evaluate DPO, WDPO, and KLDPO under preference shifts between training and evaluation. All models are trained on preference labels emphasizing the emotion fear, while evaluation preferences gradually shift toward anger. The left two plots correspond to convex mixing of these emotions, and the right two use geometric mixing. As expected, DPO performs best when the evaluation preference closely matches the training setup. However, as the evaluation shifts toward anger, DPO’s performance degrades significantly. In contrast, both WDPO and KLDPO maintain stable performance across the full range of evaluation preferences, consistently outperforming DPO under shift, demonstrating their robustness to preference misalignment. ArmoRM Multi-objective Alignment Results: In Fig.˜3, each radar plot corresponds to a different training reward pair, (1), (2), and (3), as defined in the figure caption. We evaluate all models on five individual ArmoRM objectives, three of which are unseen during training, to simulate preference shift. Across all settings, both KLDPO and WDPO consistently outperform DPO on all five evaluation axes, including those based on unseen objectives. This demonstrates their strong generalization and robustness to reward distribution shift, even when the evaluation preferences differ significantly from the training signal. Additional results are provided in Section˜E.1. Leaderboard Alignment Results: Table˜1 presents the performance of DPO, KLDPO, and WDPO on the OpenLLM leaderboard v2 (Fourrier et al., 2024). WDPO and KLDPO are trained for 2 epochs, matching DPO’s optimal early-stopping point, which is a regularization technique to prevent overfitting. For LLaMA-3B and LLaMA-8B models, we align training durations similarly. Due to computational constraints, only KLDPO results are reported for the 8B model, given its scalability. These results, averaged over 39 subtasks, are supplemented by detailed evaluations in Section˜E.2, where WDPO and KLDPO demonstrate clear advantages across various subtasks. 8 Conclusions We introduced a distributionally robust DPO framework, developed two scalable algorithms with theoretical guarantees, and integrated them into existing LLM alignment pipelines. Empirical results demonstrate their effectiveness under preference distribution shift. Future work includes extending our methods to mitigate reward hacking and generalizing robustness to other RLHF approaches. 9 Acknowledgments The authors would like to thank Vishnu Teja Kunde for invaluable discussions. This work was supported in part by the National Science Foundation (NSF) grants NSF-CAREER-EPCN-2045783, ECCS-529620-00002, and CNS-526050-00002. Portions of this research were conducted with the advanced computing resources provided by Texas A&M High Performance Research Computing. References Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. Agarwal et al. (2019) Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, 2019. Agarwal et al. (2021) Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1–76, 2021. Amodei et al. (2016) Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016. Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. Bashiri et al. (2021) Mohammad Ali Bashiri, Brian Ziebart, and Xinhua Zhang. Distributionally robust imitation learning. Advances in neural information processing systems, 34:24404–24417, 2021. Beck (2014) Amir Beck. Introduction to nonlinear optimization: Theory, algorithms, and applications with MATLAB. SIAM, 2014. Beck (2017) Amir Beck. First-order methods in optimization. SIAM, 2017. Boucheron et al. (2013) Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration Inequalities: A Nonasymptotic Theory of Independence. Oxford University Press, 2013. Bradley and Terry (1952) Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. Bukharin et al. (2024) Alexander Bukharin, Ilgee Hong, Haoming Jiang, Zichong Li, Qingru Zhang, Zixuan Zhang, and Tuo Zhao. Robust reinforcement learning from corrupted human feedback. arXiv preprint arXiv:2406.15568, 2024. Casper et al. (2023) Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217, 2023. Chakraborty et al. (2024) Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Dinesh Manocha, Furong Huang, Amrit Bedi, and Mengdi Wang. Maxmin-RLHF: Alignment with diverse human preferences. In Forty-first International Conference on Machine Learning, 2024. Chen and Paschalidis (2018) Ruidi Chen and Ioannis Ch Paschalidis. A robust learning approach for regression models based on distributionally robust optimization. Journal of Machine Learning Research, 19(13):1–48, 2018. Chen et al. (2020) Ruidi Chen, Ioannis Ch Paschalidis, et al. Distributionally robust learning. Foundations and Trends® in Optimization, 4(1-2):1–243, 2020. Chowdhury et al. (2024) Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust DPO: Aligning language models with noisy feedback. In Forty-first International Conference on Machine Learning, 2024. Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, volume 30, 2017. Duchi and Namkoong (2021) John C Duchi and Hongseok Namkoong. Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics, 49(3):1378–1406, 2021. Durmus et al. (2024) Esin Durmus, Karina Nguyen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. Towards measuring the representation of subjective global opinions in language models. In First Conference on Language Modeling, 2024. Eisenstein et al. (2024) Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alexander Nicholas D’Amour, Krishnamurthy Dj Dvijotham, Adam Fisch, Katherine A Heller, Stephen Robert Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. In First Conference on Language Modeling, 2024. Fourrier et al. (2024) Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open llm leaderboard v2, 2024. Gadot et al. (2024) Uri Gadot, Kaixin Wang, Navdeep Kumar, Kfir Yehuda Levy, and Shie Mannor. Bring your own (non-robust) algorithm to solve robust MDPs by estimating the worst kernel. In Forty-first International Conference on Machine Learning, 2024. Gao et al. (2024) Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. Gao and Kleywegt (2022) Rui Gao and Anton Kleywegt. Distributionally robust stochastic optimization with wasserstein distance. Mathematics of Operations Research, 2022. Gao et al. (2022) Rui Gao, Xi Chen, and Anton J Kleywegt. Wasserstein distributionally robust optimization and variation regularization. Operations Research, 2022. Hsu et al. (2012) Daniel Hsu, Sham Kakade, and Tong Zhang. A tail inequality for quadratic forms of subgaussian random vectors. Electronic Communications in Probability, 2012. Hu and Hong (2013) Zhaolin Hu and L Jeff Hong. Kullback-leibler divergence constrained distributionally robust optimization. Available at Optimization Online, 1(2):9, 2013. Huang et al. (2025) Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D. Lee, Wen Sun, Akshay Krishnamurthy, and Dylan J Foster. Correcting the mythos of KL-regularization: Direct alignment without overoptimization via chi-squared preference optimization. In The Thirteenth International Conference on Learning Representations, 2025. Jin et al. (2021) Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? In International Conference on Machine Learning, pages 5084–5096. PMLR, 2021. Kingma and Ba (2014) Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. Kirk et al. (2024) Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rlhf on llm generalisation and diversity. In The Twelfth International Conference on Learning Representations, 2024. Koh et al. (2021) Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pages 5637–5664. PMLR, 2021. Kuhn et al. (2019) Daniel Kuhn, Peyman Mohajerin Esfahani, Viet Anh Nguyen, and Soroosh Shafieezadeh-Abadeh. Wasserstein distributionally robust optimization: Theory and applications in machine learning. In Operations research & management science in the age of analytics, pages 130–166. Informs, 2019. LeVine et al. (2023) Will LeVine, Benjamin Pikus, Anthony Chen, and Sean Hendryx. A baseline analysis of reward models’ ability to accurately analyze foundation models under distribution shift. arXiv preprint arXiv:2311.14743, 2023. Levy et al. (2020) Daniel Levy, Yair Carmon, John C Duchi, and Aaron Sidford. Large-scale methods for distributionally robust optimization. Advances in Neural Information Processing Systems, 33:8847–8860, 2020. Mandal et al. (2025) Debmalya Mandal, Paulius Sasnauskas, and Goran Radanovic. Distributionally robust reinforcement learning with human feedback. arXiv preprint arXiv:2503.00539, 2025. Modi et al. (2020) Aditya Modi, Nan Jiang, Ambuj Tewari, and Satinder Singh. Sample complexity of reinforcement learning using linearly combined model ensembles. In International Conference on Artificial Intelligence and Statistics, pages 2010–2020. PMLR, 2020. Mohajerin Esfahani and Kuhn (2018) Peyman Mohajerin Esfahani and Daniel Kuhn. Data-driven distributionally robust optimization using the wasserstein metric: performance guarantees and tractable reformulations. Mathematical Programming, 171(1-2):115–166, 2018. Namkoong and Duchi (2016) Hongseok Namkoong and John C Duchi. Stochastic gradient methods for distributionally robust optimization with f-divergences. Advances in neural information processing systems, 29, 2016. Nika et al. (2024) Andi Nika, Debmalya Mandal, Parameswaran Kamalaruban, Georgios Tzannetos, Goran Radanovic, and Adish Singla. Reward model learning vs. direct policy optimization: A comparative analysis of learning from human preferences. In Forty-first International Conference on Machine Learning, 2024. Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022. Padmakumar et al. (2024) Vishakh Padmakumar, Chuanyang Jin, Hannah Rose Kirk, and He He. Beyond the binary: Capturing diverse preferences with reward regularization. In Workshop on Socially Responsible Language Modelling Research, 2024. Panaganti and Kalathil (2022) Kishan Panaganti and Dileep Kalathil. Sample complexity of robust reinforcement learning with a generative model. In International Conference on Artificial Intelligence and Statistics (AISTATS), pages 9582–9602, 2022. Panaganti et al. (2022) Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. Robust reinforcement learning using offline data. Advances in neural information processing systems, 35:32211–32224, 2022. Panaganti et al. (2023) Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. Distributionally robust behavioral cloning for robust imitation learning. In 2023 62nd IEEE Conference on Decision and Control (CDC), pages 1342–1347. IEEE, 2023. Panaganti et al. (2025) Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. Bridging distributionally robust learning and offline rl: An approach to mitigate distribution shift and partial data coverage. In 7th Annual Learning for Dynamics & Control Conference, pages 619–634. PMLR, 2025. Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741, 2023. Rajbhandari et al. (2020) Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020. Ramesh et al. (2024) Shyam Sundhar Ramesh, Yifan Hu, Iason Chaimalas, Viraj Mehta, Pier Giuseppe Sessa, Haitham Bou Ammar, and Ilija Bogunovic. Group robust preference optimization in reward-free rlhf. Advances in Neural Information Processing Systems, 37:37100–37137, 2024. Saravia et al. (2018) Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. CARER: Contextualized affect representations for emotion recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3687–3697. Association for Computational Linguistics, 2018. Shafieezadeh-Abadeh et al. (2019) Soroosh Shafieezadeh-Abadeh, Daniel Kuhn, and Peyman Mohajerin Esfahani. Regularization via mass transportation. Journal of Machine Learning Research, 20(103):1–68, 2019. Shi and Chi (2024) Laixi Shi and Yuejie Chi. Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity. Journal of Machine Learning Research, 25(200):1–91, 2024. Si et al. (2020) Nian Si, Fan Zhang, Zhengyuan Zhou, and Jose Blanchet. Distributionally robust policy evaluation and learning in offline contextual bandits. In International Conference on Machine Learning, pages 8884–8894, 2020. Skalse et al. (2022) Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35:9460–9471, 2022. Taori et al. (2020) Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33:18583–18599, 2020. Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. Villani et al. (2009) Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2009. Wang et al. (2024a) Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In EMNLP, 2024a. Wang et al. (2021) Ruosong Wang, Dean Foster, and Sham M. Kakade. What are the statistical limits of offline RL with linear function approximation? In International Conference on Learning Representations, 2021. Wang and Zou (2022) Yue Wang and Shaofeng Zou. Policy gradient method for robust reinforcement learning. In Proceedings of the 39th International Conference on Machine Learning, 2022. Wang et al. (2024b) Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer 2: Open-source dataset for training top-performing reward models. Advances in Neural Information Processing Systems, 37:1474–1501, 2024b. Wu et al. (2024) Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. β-dpo: Direct preference optimization with dynamic β. Advances in Neural Information Processing Systems, 37:129944–129966, 2024. Wu et al. (2025) Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jiawei Chen, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. Towards robust alignment of language models: Distributionally robustifying direct preference optimization. In The Thirteenth International Conference on Learning Representations, 2025. Xu et al. (2023) Zaiyan Xu, Kishan Panaganti, and Dileep Kalathil. Improved sample complexity bounds for distributionally robust reinforcement learning. In International Conference on Artificial Intelligence and Statistics. Conference on Artificial Intelligence and Statistics, 2023. Yan et al. (2024) Yuzi Yan, Xingzhou Lou, Jialian Li, Yiping Zhang, Jian Xie, Chao Yu, Yu Wang, Dong Yan, and Yuan Shen. Reward-robust rlhf in llms. arXiv preprint arXiv:2409.15360, 2024. Yang et al. (2022) Wenhao Yang, Liangyu Zhang, and Zhihua Zhang. Toward theoretical understandings of robust Markov decision processes: Sample complexity and asymptotics. The Annals of Statistics, 50(6):3223–3248, 2022. Yang et al. (2023) Zhouhao Yang, Yihong Guo, Pan Xu, Anqi Liu, and Animashree Anandkumar. Distributionally robust policy gradient for offline contextual bandits. In International Conference on Artificial Intelligence and Statistics, pages 6443–6462. PMLR, 2023. Zhang et al. (2025) Michael JQ Zhang, Zhilin Wang, Jena D. Hwang, Yi Dong, Olivier Delalleau, Yejin Choi, Eunsol Choi, Xiang Ren, and Valentina Pyatkin. Diverging preferences: When do annotators disagree and do models know? In Forty-second International Conference on Machine Learning, 2025. Zhao et al. (2024) Siyan Zhao, John Dang, and Aditya Grover. Group preference optimization: Few-shot alignment of large language models. In The Twelfth International Conference on Learning Representations, 2024. Zhou et al. (2024) Ruida Zhou, Tao Liu, Min Cheng, Dileep Kalathil, PR Kumar, and Chao Tian. Natural actor-critic for robust reinforcement learning with function approximation. Advances in neural information processing systems, 36, 2024. Zhou et al. (2021) Zhengqing Zhou, Qinxun Bai, Zhengyuan Zhou, Linhai Qiu, Jose Blanchet, and Peter Glynn. Finite-sample regret bound for distributionally robust offline tabular reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages 3331–3339, 2021. Zhu et al. (2023) Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning, pages 43037–43067. PMLR, 2023. Ziegler et al. (2019) Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. NeurIPS Paper Checklist 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: We have clearly stated our paper’s contributions and scope in the abstract and introduction. Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 2. Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We have clearly stated the assumptions used in our proofs. Additional discussion on limitations can be found in Appendix˜G. Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper. • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 3. Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: All assumptions used are clearly stated. Guidelines: • The answer NA means that the paper does not include theoretical results. • All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced. • All assumptions should be clearly stated or referenced in the statement of any theorems. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 4. Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We provide information in order to reproduce experimental results presented in our paper. In addition, we will provide open access to the data and code used in this paper. Guidelines: • The answer NA means that the paper does not include experiments. • If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We will provide open access to data and code upon acceptance. Guidelines: • The answer NA means that paper does not include experiments requiring code. • Please see the NeurIPS code and data submission guidelines (https://nips.c/public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so “No” is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.c/public/guides/CodeSubmissionPolicy) for more details. • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 6. Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: Experimental details can be found in Section˜7, and additional details can be found in Appendix˜F. Guidelines: • The answer NA means that the paper does not include experiments. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 7. Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [N/A] Justification: [NA] Guidelines: • The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 8. Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We provide additional details regarding our experiment setups in Appendix˜F. Guidelines: • The answer NA means that the paper does not include experiments. • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper). 9. Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.c/public/EthicsGuidelines? Answer: [Yes] Justification: We acknowledge the NeurIPS Code of Ethics. Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: See Appendix˜H. Guidelines: • The answer NA means that there is no societal impact of the work performed. • If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 11. Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [N/A] Justification: This paper has no such risks. Guidelines: • The answer NA means that the paper poses no such risks. • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited. Guidelines: • The answer NA means that the paper does not use existing assets. • The authors should cite the original paper that produced the code package or dataset. • The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., C-BY 4.0) should be included for each asset. • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset’s creators. 13. New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The documentation will be released along with the release of the code. Guidelines: • The answer NA means that the paper does not release new assets. • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 14. Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [N/A] Justification: [NA] Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [N/A] Justification: [NA] Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 16. Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required. Answer: [Yes] Justification: The proposed methods, Wasserstein DPO and KLDPO, are applied to fine-tune large language models such as GPT-2 and LLaMA. The LLMs are central to our empirical validation and the alignment task studied in this work. Thus, their usage is a critical component of the core methodology. Guidelines: • The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components. • Please refer to our LLM policy (https://neurips.c/Conferences/2025/LLM) for what should or should not be described. Appendix A Useful Technical Results A.1 Wasserstein Theory We rely on the following strong duality result from the Wasserstein distributionally robust optimization (WDRO) literature. Lemma 1 (Gao and Kleywegt, 2022, Theorem 1; Strong Duality for DRO with Wasserstein Distance). Consider any p∈[1,∞)p∈[1,∞), any ν∈(Ξ)ν ( ), any ρ>0ρ>0, and any Ψ∈L1(ν) ∈ L^1(ν) such that the growth rate κ of Ψ satisfies κ≔infη≥0:∫ΞΦ(η,ζ)ν(dζ)>−∞<∞,κ \η≥ 0 _ (η,ζ)ν(dζ)>-∞ \<∞, (10) where Φ(η,ζ)≔infξ∈Ξηdp(ξ,ζ)−Ψ(ξ) (η,ζ) _ξ∈ \η d^p(ξ,ζ)- (ξ)\ is a regularization operator. Then the strong duality holds with finite optimal value vp=vD≤∞v_p=v_D≤∞, where vp v_p ≔supμ∈(Ξ)∫ΞΨ(ξ)μ(dξ):p(μ,ν)≤ρ, _μ ( ) \ _ (ξ)μ(dξ) W_p(μ,ν)≤ρ \, (Primal) vD v_D ≔infη≥0ηρp−∫Ξinfξ∈Ξ[ηdp(ξ,ζ)−Ψ(ξ)]ν(dζ). _η≥ 0 \ηρ^p- _ _ξ∈ [η d^p(ξ,ζ)- (ξ)]ν(dζ) \. (Dual) Lemma 2 (Gao and Kleywegt, 2022, Lemma 2.(i); Properties of the growth κ). Suppose that ν∈p(Ξ)ν _p( ). Then the growth rate κ (as defined in Eq.˜10) is finite if and only if there exists ζo∈Ξζ^o∈ and L,M>0L,M>0 such that Ψ(ξ)−Ψ(ζo)≤Ldp(ξ,ζo)+M,∀ξ∈Ξ. (ξ)- (ζ^o)≤ Ld^p(ξ,ζ^o)+M, ∀ξ∈ . (11) Corollary 1. Consider any bounded loss function l over bounded Ξ . Then the duality defined in Lemma˜1 holds. Proof. It follows from Lemma˜2. We can pick L to be the diameter of Ξ and M to be the bound of Ψ . ∎ A.2 Optimization Lemma 3 (Beck, 2014, Theorem 1.24; Linear Approximation Theorem). Let f:U→ℝf U be a twice continuously differentiable function over an open set U⊆ℝnU ^n, and let x,y∈Ux,y∈ U be such that [x,y]⊆U[x,y] U. Then there exists ξ∈[x,y]ξ∈[x,y] such that f(y)=f(x)+∇f(x)⊤(y−x)+12(y−x)⊤∇2f(ξ)(y−x).f(y)=f(x)+∇ f(x) (y-x)+ 12(y-x) ∇^2f(ξ)(y-x). Lemma 4 (Beck, 2017, Theorem 5.24; First-order characterizations of strong convexity). Let f:→(−∞,∞]f →(-∞,∞] be a proper closed and convex function. Then for a given σ>0σ>0, the following two claims are equivalent: I. For any x,y∈(f)x,y∈ dom(f) and λ∈[0,1]λ∈[0,1]: f(λx+(1−λ)y)≤λf(x)+(1−λ)f(y)−σ2λ(1−λ)∥x−y∥2.f(λ x+(1-λ)y)≤λ f(x)+(1-λ)f(y)- σ2λ(1-λ) x-y ^2. I. f(y)≥f(x)+⟨g,y−x⟩+σ2∥y−x∥2,f(y)≥ f(x)+ g,y-x + σ2 y-x ^2, for any x∈(∂f)x∈ dom(∂ f), y∈(f)y∈ dom(f) and g∈∂f(x)g∈∂ f(x). Lemma 5 (Beck, 2017, Theorem 5.25; Existence and uniqueness of a minimizer of closed strongly convex functions). Let f:→(−∞,∞]f →(-∞,∞] be a proper closed and σ-strongly convex function σ>0σ>0. Then I. f has a unique minimizer; I. f(x)−f(x∗)≥σ2∥x−x∗∥2f(x)-f(x^*)≥ σ2 x-x^* ^2 for all x∈(f)x∈ dom(f), where x∗x^* is the unique minimizer of f. A.3 Distributionally Robust Optimization Results The Kullback-Liebler uncertainty set can be constructed with the f-divergence. The f-divergence between the distribution P and o P^o is defined as Df(∥o)=∫f(ddo)o,D_f( P\;\|\; P^o)= _Xf ( d Pd P^o )d P^o, (12) where f is a convex function. f(t)=tlog(t)f(t)=t (t) gives us the Kullback-Liebler divergence. Let o P^o be a distribution on the space X and let l:→ℝl be a loss function. We have the following result from the distributionally robust optimization literature. Lemma 6 (Duchi and Namkoong, 2021, Proposition 1). Let DfD_f be the f-divergence defined in Eq.˜12. Then, sup:Df(∥o)≤ρE[l(X)]=infλ≥0,η∈ℝo[λf∗(l(X)−ηλ)]+λρ+η, _ P D_f( P\;\|\; P^o)≤ρE_ P[l(X)]= _λ≥ 0,η E_ P^o [λ f^* ( l(X)-ηλ ) ]+λρ+η, (13) where f∗(s)=supt≥0st−f(t)f^*(s)= _t≥ 0\st-f(t)\ is the Fenchel conjugate. A.4 Concentration Results Lemma 7 (Hoeffding’s inequality (see Boucheron et al., 2013, Theorem 2.8)). Let X1,…,XnX_1,…,X_n be independent random variables such that XiX_i takes its values in [ai,bi][a_i,b_i] almost surely for all i≤ni≤ n. Let S=∑i=1n(Xi−[Xi]).S= _i=1^n(X_i-E [X_i ]). Then for every t>0t>0, ℙ(S≥t)≤exp(−2t2∑i=1n(bi−ai)2).P (S≥ t ) (- 2t^2 _i=1^n(b_i-a_i)^2 ). Furthermore, if X1,…,XnX_1,…,X_n are a sequence of independent, identically distributed random variables with mean μ. Let X¯n=1n∑i=1nXi X_n= 1n _i=1^nX_i. Suppose that Xi∈[a,b]X_i∈[a,b], ∀i∀ i. Then for all t>0t>0 ℙ(|X¯n−μ|≥t)≤2exp(−2nt2(b−a)2).P ( X_n-μ ≥ t )≤ 2exp (- 2nt^2(b-a)^2 ). Lemma 8 (Hsu et al., 2012, Theorem 2.1). Let A∈ℝn×nA ^n× n be a matrix, and let Σ≔A⊤A A A. Suppose that x=(x1,…,xn)x=(x_1,…,x_n) is a random vector such that for some μ∈ℝnμ ^n and σ≥0σ≥ 0, [exp(α⊤(x−μ))]≤exp(∥α∥2σ2/2),E[exp(α (x-μ))] ( α ^2σ^2/2), for all α∈ℝnα ^n. For all t>0t>0, ℙ[∥Ax∥2>σ2⋅((Σ)+2(Σ2)t+2∥Σ∥t)+(Σμμ⊤)⋅(1+2t∥Σ∥2(Σ2))]≤e−t.P [ Ax ^2>σ^2· ( Tr( )+2 Tr( ^2)t+2 t )+ Tr( μ )· (1+2 t ^2 Tr( ^2) ) ]≤ e^-t. Moreover, if μ=0μ=0 and σ=1σ=1, then the probability inequality reads ℙ(∥Ax∥2>(Σ)+2(Σ2)t+2∥Σ∥t)≤e−t.P ( Ax ^2> Tr( )+2 Tr( ^2)t+2 t )≤ e^-t. Appendix B Proof of WDPO Sample Complexity Many properties of distributionally robust DPO are derived from those of the non-robust DPO. We hence start with the following proof of policy parameter convergence in the non-robust setting (Proposition˜1). B.1 Proof of Non-robust DPO Policy Parameter Convergence Recall the pointwise DPO loss: l(θ;s,a1,a2,y)≔−ylogσ(βhθ(s,a1,a2))−(1−y)logσ(βhθ(s,a2,a1)),l(θ;s,a^1,a^2,y) -y σ(β h_θ(s,a^1,a^2))-(1-y) σ(β h_θ(s,a^2,a^1)), where hθ(s,a1,a2)≔logπθ(a1∣s)πref(a1∣s)−logπθ(a2∣s)πref(a2∣s)h_θ(s,a^1,a^2) _θ(a^1 s) _ref(a^1 s)- _θ(a^2 s) _ref(a^2 s). Denote this loss by lz(θ)l_z(θ) where z=(s,a1,a2,y)z=(s,a^1,a^2,y). We also denote the empirical (sample) DPO loss as l(θ)=1n∑i=1nlzi(θ)=1n∑i=1n−yilogσ(βhθ(si,ai1,ai2))−(1−yi)logσ(βhθ(si,ai2,ai1)).l_D(θ)= 1n _i=1^nl_z_i(θ)= 1n _i=1^n-y_i σ(β h_θ(s_i,a^1_i,a^2_i))-(1-y_i) σ(β h_θ(s_i,a^2_i,a^1_i)). We denote the MLE solution to l_D by θndpo∈argminθ∈Θl(θ)θ^dpo_n∈ *argmin_θ∈ l_D(θ). Also, denote the true parameter which is the global minimum of the population negative log likelihood by θ∗θ^*. (Almost) Strong Convexity of l. In order to calculate the Hessian matrix of lzl_z w.r.t. θ, we need to calculate ∇θ2logσ(βhθ(s,a1,a2))∇^2_θ σ(β h_θ(s,a^1,a^2)). Suppose f:ℝ→ℝf , g:ℝd→ℝg ^d . The Hessian of f∘gf g is, for any x∈ℝdx ^d, ∇x2(f∘g)(x)=f′(g(x))∇x2g(x)+f′(g(x))∇xg(x)∇xg(x)⊤. _x^2(f g)(x)=f (g(x)) _x^2g(x)+f^ (g(x)) _xg(x) _xg(x) . (14) Recall that σ is the sigmoid function. It has the properties: σ(−x)=1−σ(x)σ(-x)=1-σ(x) and σ′(x)=σ(x)(1−σ(x))σ (x)=σ(x)(1-σ(x)). Let f(x)=logσ(x)f(x)= σ(x), we have dxf(x) ddxf(x) =σ′(x)σ(x)=σ(x)(1−σ(x))σ(x)=σ(−x); = σ (x)σ(x)= σ(x)(1-σ(x))σ(x)=σ(-x); d2dx2f(x) d^2dx^2f(x) =dx[σ(−x)]=dx[1−σ(x)]=−σ′(x)=−σ(x)σ(−x). = ddx[σ(-x)]= ddx[1-σ(x)]=-σ (x)=-σ(x)σ(-x). With g(θ)≔βhθ(s,a1,a2)g(θ) β h_θ(s,a^1,a^2) and the Hessian chain rule for composition with a scalar function (Eq.˜14), we have ∇θ2logσ(βhθ(s,a1,a2)) _θ^2 σ(β h_θ(s,a^1,a^2)) =βσ(−βhθ(s,a1,a2))∇θ2hθ(s,a1,a2) =βσ(-β h_θ(s,a^1,a^2)) _θ^2h_θ(s,a^1,a^2) −β2σ(βhθ(s,a1,a2))σ(−βhθ(s,a1,a2))∇θhθ(s,a1,a2)∇θhθ(s,a1,a2)⊤. -β^2σ(β h_θ(s,a^1,a^2))σ(-β h_θ(s,a^1,a^2)) _θh_θ(s,a^1,a^2) _θh_θ(s,a^1,a^2) . In addition, we have the following observations ∇θhθ(s,a1,a2) _θh_θ(s,a^1,a^2) =∇θlogπθ(a1∣s)−∇θlogπθ(a2∣s)=−∇θhθ(s,a2,a1); = _θ _θ(a^1 s)- _θ _θ(a^2 s)=- _θh_θ(s,a^2,a^1); ∇θ2hθ(s,a1,a2) _θ^2h_θ(s,a^1,a^2) =∇θ2logπθ(a1∣s)−∇θ2logπθ(a2∣s)=−∇θ2hθ(s,a2,a1). = _θ^2 _θ(a^1 s)- _θ^2 _θ(a^2 s)=- _θ^2h_θ(s,a^2,a^1). Now, using the above observations, we can simplify ∇θ2lz(θ) _θ^2l_z(θ) as follows ∇θ2lz(θ) _θ^2l_z(θ) =−y∇θ2logσ(βhθ(s,a1,a2))−(1−y)∇θ2logσ(βhθ(s,a2,a1)) =-y _θ^2 σ(β h_θ(s,a^1,a^2))-(1-y) _θ^2 σ(β h_θ(s,a^2,a^1)) =−y[βσ(−βhθ(s,a1,a2))∇θ2hθ(s,a1,a2) =-y [βσ(-β h_θ(s,a^1,a^2)) _θ^2h_θ(s,a^1,a^2) −β2σ(βhθ(s,a1,a2))σ(−βhθ(s,a1,a2))∇θhθ(s,a1,a2)∇θhθ(s,a1,a2)⊤] -β^2σ(β h_θ(s,a^1,a^2))σ(-β h_θ(s,a^1,a^2)) _θh_θ(s,a^1,a^2) _θh_θ(s,a^1,a^2) ] −(1−y)[βσ(−βhθ(s,a2,a1))∇θ2hθ(s,a2,a1) -(1-y) [βσ(-β h_θ(s,a^2,a^1)) _θ^2h_θ(s,a^2,a^1) −β2σ(βhθ(s,a2,a1))σ(−βhθ(s,a2,a1))∇θhθ(s,a2,a1)∇θhθ(s,a2,a1)⊤] -β^2σ(β h_θ(s,a^2,a^1))σ(-β h_θ(s,a^2,a^1)) _θh_θ(s,a^2,a^1) _θh_θ(s,a^2,a^1) ] =−yβσ(−βhθ(s,a1,a2))∇θ2hθ(s,a1,a2) =-yβσ(-β h_θ(s,a^1,a^2)) _θ^2h_θ(s,a^1,a^2) +yβ2σ(βhθ(s,a1,a2))σ(−βhθ(s,a1,a2))∇θhθ(s,a1,a2)∇θhθ(s,a1,a2)⊤ +yβ^2σ(β h_θ(s,a^1,a^2))σ(-β h_θ(s,a^1,a^2)) _θh_θ(s,a^1,a^2) _θh_θ(s,a^1,a^2) −(1−y)βσ(−βhθ(s,a2,a1))∇θ2hθ(s,a2,a1) -(1-y)βσ(-β h_θ(s,a^2,a^1)) _θ^2h_θ(s,a^2,a^1) +(1−y)β2σ(βhθ(s,a2,a1))σ(−βhθ(s,a2,a1))∇θhθ(s,a2,a1)∇θhθ(s,a2,a1)⊤ +(1-y)β^2σ(β h_θ(s,a^2,a^1))σ(-β h_θ(s,a^2,a^1)) _θh_θ(s,a^2,a^1) _θh_θ(s,a^2,a^1) =(a)−yβσ(−βhθ(s,a1,a2))∇θ2hθ(s,a1,a2) (a)=-yβσ(-β h_θ(s,a^1,a^2)) _θ^2h_θ(s,a^1,a^2) +yβ2σ(βhθ(s,a1,a2))σ(−βhθ(s,a1,a2))∇θhθ(s,a1,a2)∇θhθ(s,a1,a2)⊤ +yβ^2σ(β h_θ(s,a^1,a^2))σ(-β h_θ(s,a^1,a^2)) _θh_θ(s,a^1,a^2) _θh_θ(s,a^1,a^2) +(1−y)βσ(−βhθ(s,a2,a1))∇θ2hθ(s,a1,a2) +(1-y)βσ(-β h_θ(s,a^2,a^1)) _θ^2h_θ(s,a^1,a^2) +(1−y)β2σ(−βhθ(s,a1,a2))σ(βhθ(s,a1,a2))∇θhθ(s,a1,a2)∇θhθ(s,a1,a2)⊤ +(1-y)β^2σ(-β h_θ(s,a^1,a^2))σ(β h_θ(s,a^1,a^2)) _θh_θ(s,a^1,a^2) _θh_θ(s,a^1,a^2) =β(−y+σ(βhθ(s,a1,a2)))∇θ2hθ(s,a1,a2) =β(-y+σ(β h_θ(s,a^1,a^2))) _θ^2h_θ(s,a^1,a^2) +β2σ(βhθ(s,a1,a2))σ(−βhθ(s,a1,a2))∇θhθ(s,a1,a2)∇θhθ(s,a1,a2)⊤. +β^2σ(β h_θ(s,a^1,a^2))σ(-β h_θ(s,a^1,a^2)) _θh_θ(s,a^1,a^2) _θh_θ(s,a^1,a^2) . where (a)(a) is due to hθ(s,a2,a1)=−hθ(s,a1,a2)h_θ(s,a^2,a^1)=-h_θ(s,a^1,a^2), ∇θhθ(s,a2,a1)=−∇θhθ(s,a1,a2) _θh_θ(s,a^2,a^1)=- _θh_θ(s,a^1,a^2) and ∇θ2hθ(s,a2,a1)=−∇θ2hθ(s,a1,a2) _θ^2h_θ(s,a^2,a^1)=- _θ^2h_θ(s,a^1,a^2). It’s clear that we have to calculate ∇θ2hθ(s,a1,a2) _θ^2h_θ(s,a^1,a^2) and ∇θhθ(s,a1,a2) _θh_θ(s,a^1,a^2). Observe that ∇θhθ(s,a1,a2)=∇θlogπθ(a1∣s)−∇θlogπθ(a2∣s)=1πθ(a1∣s)∇θπθ(a1∣s)−1πθ(a2∣s)∇θπθ(a2∣s). _θh_θ(s,a^1,a^2)= _θ _θ(a^1 s)- _θ _θ(a^2 s)= 1 _θ(a^1 s) _θ _θ(a^1 s)- 1 _θ(a^2 s) _θ _θ(a^2 s). (15) In addition, we have that ∇θ2hθ(s,a1,a2)=∇θ2logπθ(a1∣s)−∇θ2logπθ(a2∣s) _θ^2h_θ(s,a^1,a^2)= _θ^2 _θ(a^1 s)- _θ^2 _θ(a^2 s). Using the Hessian chain rule (Eq.˜14), we have ∇θ2logπθ(a∣s)=1πθ(a∣s)∇θ2πθ(a∣s)−1πθ(a∣s)2∇θπθ(a∣s)∇θπθ(a∣s)⊤. _θ^2 _θ(a s)= 1 _θ(a s) _θ^2 _θ(a s)- 1 _θ(a s)^2 _θ _θ(a s) _θ _θ(a s) . Now it boils down to tackling ∇θπθ(a∣s) _θ _θ(a s) and ∇θ2πθ(a∣s) _θ^2 _θ(a s). Observe that ∇θπθ(a∣s) _θ _θ(a s) =∇θexp(⟨ψ(s,a),θ⟩)[∑a′exp(⟨ψ(s,a′),θ⟩)]−[∑a′∇θexp(⟨ψ(s,a′),θ⟩)]exp(⟨ψ(s,a),θ⟩)(∑a′exp(⟨ψ(s,a′),θ⟩))2 = _θexp ( ψ(s,a),θ )[ _a exp ( ψ(s,a ),θ )]-[ _a _θexp ( ψ(s,a ),θ )]exp ( ψ(s,a),θ )( _a exp ( ψ(s,a ),θ ))^2 =exp(⟨ψ(s,a),θ⟩)∑a′exp(⟨ψ(s,a′),θ⟩)ψ(s,a)−exp(⟨ψ(s,a),θ⟩)(∑a′exp(⟨ψ(s,a′),θ⟩))2∑a′exp(⟨ψ(s,a′),θ⟩)ψ(s,a′) = exp ( ψ(s,a),θ ) _a exp ( ψ(s,a ),θ )ψ(s,a)- exp ( ψ(s,a),θ )( _a exp ( ψ(s,a ),θ ))^2 _a exp ( ψ(s,a ),θ )ψ(s,a ) =exp(⟨ψ(s,a),θ⟩)∑a′exp(⟨ψ(s,a′),θ⟩)ψ(s,a)−exp(⟨ψ(s,a),θ⟩)∑a′exp(⟨ψ(s,a′),θ⟩)∑a′exp(⟨ψ(s,a),θ⟩)∑a′exp(⟨ψ(s,a′),θ⟩)ψ(s,a′) = exp ( ψ(s,a),θ ) _a exp ( ψ(s,a ),θ )ψ(s,a)- exp ( ψ(s,a),θ ) _a exp ( ψ(s,a ),θ ) _a exp ( ψ(s,a),θ ) _a exp ( ψ(s,a ),θ )ψ(s,a ) =πθ(a∣s)ψ(s,a)−πθ(a∣s)∑a′πθ(a′∣s)ψ(s,a′) = _θ(a s)ψ(s,a)- _θ(a s) _a _θ(a s)ψ(s,a ) =πθ(a∣s)[ψ(s,a)−∑a′πθ(a′∣s)ψ(s,a′)]. = _θ(a s) [ψ(s,a)- _a _θ(a s)ψ(s,a ) ]. Then we have ∇θhθ(s,a1,a2) _θh_θ(s,a^1,a^2) =1πθ(a1∣s)πθ(a1∣s)[ψ(s,a1)−∑a′πθ(a′∣s)ψ(s,a′)] = 1 _θ(a^1 s) _θ(a^1 s) [ψ(s,a^1)- _a _θ(a s)ψ(s,a ) ] −1πθ(a2∣s)πθ(a2∣s)[ψ(s,a2)−∑a′πθ(a′∣s)ψ(s,a′)] - 1 _θ(a^2 s) _θ(a^2 s) [ψ(s,a^2)- _a _θ(a s)ψ(s,a ) ] =ψ(s,a1)−ψ(s,a2). =ψ(s,a^1)-ψ(s,a^2). (16) Notice that ∇θhθ _θh_θ above does not depend on the policy parameter θ. This implies that its Hessian is the zero matrix, i.e., ∇θ2hθ(s,a1,a2)= _θ^2h_θ(s,a^1,a^2)= 0. Finally, we have that ∇θ2lz(θ)=β2σ(βhθ(s,a1,a2))σ(−βhθ(s,a1,a2))(ψ(s,a1)−ψ(s,a2))(ψ(s,a1)−ψ(s,a2))⊤. _θ^2l_z(θ)=β^2σ(β h_θ(s,a^1,a^2))σ(-β h_θ(s,a^1,a^2))(ψ(s,a^1)-ψ(s,a^2))(ψ(s,a^1)-ψ(s,a^2)) . Moving from the pointwise loss to the empirical loss, we denote ∇θ2l(θ)=1n∑i=1nβ2σ(βhθ(si,ai1,ai2))σ(−βhθ(si,ai1,ai2))(ψ(si,ai1)−ψ(si,ai2))(ψ(si,ai1)−ψ(si,ai2))⊤. _θ^2l_D(θ)= 1n _i=1^nβ^2σ(β h_θ(s_i,a^1_i,a^2_i))σ(-β h_θ(s_i,a^1_i,a^2_i))(ψ(s_i,a^1_i)-ψ(s_i,a^2_i))(ψ(s_i,a^1_i)-ψ(s_i,a^2_i)) . Now let’s focus on the function σ(x)σ(−x)σ(x)σ(-x). Our aim is to find a lower bound for this function. Observe that |hθ(s,a1,a2)| h_θ(s,a^1,a^2) =|(logπθ(a1∣s)−logπθ(a2∣s))−(logπref(a1∣s)−logπref(a2∣s))| = ( _θ(a^1 s)- _θ(a^2 s))-( _ref(a^1 s)- _ref(a^2 s)) =|⟨θ,ψ(s,a1)−ψ(s,a2)⟩−⟨θref,ψ(s,a1)−ψ(s,a2)⟩| = θ,ψ(s,a^1)-ψ(s,a^2) - _ref,ψ(s,a^1)-ψ(s,a^2) =|⟨θ−θref,ψ(s,a1)−ψ(s,a2)⟩| = θ- _ref,ψ(s,a^1)-ψ(s,a^2) ≤(a)∥θ−θref∥2∥ψ(s,a1)−ψ(s,a2)∥2 (a)≤ θ- _ref _2 ψ(s,a^1)-ψ(s,a^2) _2 ≤(b)4B, (b)≤4B, (17) where (a)(a) is due to Cauchy-Schwarz inequality. (b)(b) is due to the assumptions ∥θ∥2≤B θ _2≤ B and maxs,a∥ψ(s,a)∥2≤1 _s,a ψ(s,a) _2≤ 1. Now this suggests that the input to the function σ(βhθ(s,a1,a2))σ(−βhθ(s,a1,a2))σ(β h_θ(s,a^1,a^2))σ(-β h_θ(s,a^1,a^2)) is bounded in [−4βB,4βB][-4β B,4β B]. Since σ(x)σ(−x)σ(x)σ(-x) is symmetric and strictly decreasing when x∈[0,∞)x∈[0,∞), we have that β2σ(βhθ(s,a1,a2))σ(−βhθ(s,a1,a2))≥β2e4βB(1+e4βB)2,∀θ∈Θ.β^2σ(β h_θ(s,a^1,a^2))σ(-β h_θ(s,a^1,a^2))≥ β^2e^4β B(1+e^4β B)^2, ∀θ∈ . (18) We then have that u⊤∇θ2l(θ)u≥γn∥Xu∥22,∀u∈ℝd,u _θ^2l_D(θ)u≥ γn Xu _2^2, ∀ u ^d, where γ=β2e4βB(1+e4βB)2γ= β^2e^4β B(1+e^4β B)^2 and X∈ℝn×dX ^n× d has the differencing vector xi≔ψ(si,ai1)−ψ(si,ai2)∈ℝdx_i ψ(s_i,a^1_i)-ψ(s_i,a_i^2) ^d as its i-th row. Thus, if we introduce the error vector Δ≔θndpo−θ∗ θ^dpo_n-θ^*, then by the linear approximation theorem (Lemma˜3), there exists α∈[0,1]α∈[0,1] and θ~=αθndpo+(1−α)θ∗ θ=αθ^dpo_n+(1-α)θ^* such that l(θ∗+Δ)−l(θ∗)−⟨∇θl(θ∗),Δ⟩=12Δ⊤∇θ2l(θ~)Δ≥γ2n∥XΔ∥22=γ2∥Δ∥Σ2,l_D(θ^*+ )-l_D(θ^*)- _θl_D(θ^*), = 12 _θ^2l_D( θ) ≥ γ2n X ^2_2= γ2 ^2_ _D, (19) where Σ=1n∑i=1n(ψ(si,ai1)−ψ(si,ai2))(ψ(si,ai1)−ψ(si,ai2))⊤ _D= 1n _i=1^n(ψ(s_i,a^1_i)-ψ(s_i,a_i^2))(ψ(s_i,a^1_i)-ψ(s_i,a_i^2)) . This implies that l_D is (almost) strongly convex around θ∗θ^* with parameter γ with respect to semi-norm∥⋅∥Σ · _ _D. Note that we will not treat l_D as a strictly strongly convex function in any part of this proof. We only need the inequality Eq.˜19. Bounding the estimation error. Recall that θndpoθ^dpo_n is optimal for l(θ)l_D(θ) and Δ≔θndpo−θ∗ θ^dpo_n-θ^*. We must have l(θndpo)≤l(θ∗)l_D(θ^dpo_n)≤ l_D(θ^*). By substracting and adding ⟨∇θl(θ∗),Δ⟩ _θl_D(θ^*), on both sides, we have l(θ∗+Δ)−l(θ∗)−⟨∇θl(θ∗),Δ⟩≤−⟨∇θl(θ∗),Δ⟩.l_D(θ^*+ )-l_D(θ^*)- _θl_D(θ^*), ≤- _θl_D(θ^*), . For the right hand side above, we have |⟨∇θl(θ∗),Δ⟩|≤∥∇θl(θ∗)∥(Σ+λI)−1∥Δ∥Σ+λI, for any λ>0. _θl_D(θ^*), ≤ _θl_D(θ^*) _( _D+λ I)^-1 _ _D+λ I, for any λ>0. By γ-strong convexity of l_D at θ∗θ^*, we have l(θ∗+Δ)−l(θ∗)−⟨∇θl(θ∗),Δ⟩≥γ2∥Δ∥Σ2.l_D(θ^*+ )-l_D(θ^*)- _θl_D(θ^*), ≥ γ2 ^2_ _D. Combining the inequalities, we have γ2∥Δ∥Σ2≤∥∇θl(θ∗)∥(Σ+λI)−1∥Δ∥Σ+λI γ2 ^2_ _D≤ _θl_D(θ^*) _( _D+λ I)^-1 _ _D+λ I. Now we need to bound the term ∥∇θl(θ∗)∥(Σ+λI)−1 _θl_D(θ^*) _( _D+λ I)^-1. We can calculate the gradient w.r.t. θ of the pointwise loss as follows ∇θlz(θ) _θl_z(θ) =∇θ[−ylogσ(βhθ(s,a1,a2))−(1−y)logσ(βhθ(s,a2,a1))] = _θ[-y σ(β h_θ(s,a^1,a^2))-(1-y) σ(β h_θ(s,a^2,a^1))] =−y∇θlogσ(βhθ(s,a1,a2))−(1−y)∇θlogσ(βhθ(s,a2,a1)) =-y _θ σ(β h_θ(s,a^1,a^2))-(1-y) _θ σ(β h_θ(s,a^2,a^1)) =−βyσ(−βhθ(s,a1,a2))∇θhθ(s,a1,a2)−β(1−y)σ(βhθ(s,a1,a2))∇θhθ(s,a2,a1) =-β yσ(-β h_θ(s,a^1,a^2)) _θh_θ(s,a^1,a^2)-β(1-y)σ(β h_θ(s,a^1,a^2)) _θh_θ(s,a^2,a^1) =(a)−β(yσ(βhθ(s,a2,a1))−(1−y)σ(βhθ(s,a1,a2)))(ψ(s,a1)−ψ(s,a2)), (a)=-β(yσ(β h_θ(s,a^2,a^1))-(1-y)σ(β h_θ(s,a^1,a^2)))(ψ(s,a^1)-ψ(s,a^2)), where (a)(a) is due to ∇θhθ(s,a1,a2)=ψ(s,a1)−ψ(s,a2) _θh_θ(s,a^1,a^2)=ψ(s,a^1)-ψ(s,a^2) calculated in Section˜B.1. This implies that ∇θl(θ∗)=−βn∑i=1n[yiσ(βhθ∗(si,ai2,ai1))−(1−yi)σ(βhθ∗(si,ai1,ai2))]xi, _θl_D(θ^*)= -βn _i=1^n[y_iσ(β h_θ^*(s_i,a_i^2,a_i^1))-(1-y_i)σ(β h_θ^*(s_i,a_i^1,a_i^2))]x_i, (20) where xi=ψ(si,ai1)−ψ(si,ai2)x_i=ψ(s_i,a_i^1)-ψ(s_i,a_i^2). Now let’s define a random vector V∈ℝnV ^n with i.i.d. components as Vi=σ(βhθ∗(si,ai2,ai1))w.p. σ(βhθ∗(si,ai1,ai2)),−σ(βhθ∗(si,ai1,ai2))w.p. σ(βhθ∗(si,ai2,ai1)).V_i= casesσ(β h_θ^*(s_i,a_i^2,a_i^1))&w.p. σ(β h_θ^*(s_i,a_i^1,a_i^2)),\\ -σ(β h_θ^*(s_i,a_i^1,a_i^2))&w.p. σ(β h_θ^*(s_i,a_i^2,a_i^1)). cases (21) Then we have ∇θl(θ∗)=−βnX⊤V _θl_D(θ^*)=- βnX V. It’s easy to verify that Vi=0EV_i=0 and |Vi|≤1 V_i ≤ 1, for all 1≤i≤n1≤ i≤ n. Next, if we define the n×n× n matrix M≔β2n2X(Σ+λI)−1X⊤M β^2n^2X( _D+λ I)^-1X , then we can write ∥∇θl(θ∗)∥(Σ+λI)−12=V⊤MV _θl_D(θ^*) _( _D+λ I)^-1^2=V MV. Let the eigendecomposition of X⊤X X be UΛU⊤U U . Observe that M=β2n2X(Σ+λI)−1X⊤=β2n2XU(Λ/n+λI)−1U⊤X⊤.M= β^2n^2X( _D+λ I)^-1X = β^2n^2XU( /n+λ I)^-1U X . We can bound the trace of M as follows (M) Tr(M) =(β2n2XU(Λ/n+λI)−1U⊤X⊤)=β2n2(U(Λ/n+λI)−1U⊤UΛU⊤) = Tr( β^2n^2XU( /n+λ I)^-1U X )= β^2n^2 Tr(U( /n+λ I)^-1U U U ) =β2n2(U(Λ/n+λI)−1ΛU⊤)=β2n2((Λ/n+λI)−1Λ)=β2n2∑i=1dneiei+λn = β^2n^2 Tr(U( /n+λ I)^-1 U )= β^2n^2 Tr(( /n+λ I)^-1 )= β^2n^2 _i=1^d ne_ie_i+λ n ≤β2n2⋅nd=β2dn, ≤ β^2n^2· nd= β^2dn, where eie_i is the i-th eigenvalue of X⊤X X. Similarly, we can bound (M2)≤β4dn2 Tr(M^2)≤ β^4dn^2. Now, let X=U~ΣV~⊤X= U V be the singular value decomposition of X. Then we can show that M=β2n2X(X⊤X/n+λI)−1X⊤=β2n2U~Σ(Σ⊤Σ/n+λI)−1ΣU~⊤.M= β^2n^2X(X X/n+λ I)^-1X = β^2n^2 U ( /n+λ I)^-1 U . Since X(Σ+λI)−1X⊤X( _D+λ I)^-1X is symmetric, and clearly U~Σ(Σ⊤Σ/n+λI)−1ΣU~⊤ U ( /n+λ I)^-1 U diagonalizes it, the eigenvalue of it takes form σi2σi2/n+λ _i^2 _i^2/n+λ, where σi _i is the i-th singular value of X. Hence, all eigenvalues are upper bounded by n. Then we must have ∥M∥op=λmax(M)≤β2n M _op= _ max(M)≤ β^2n. Since the components of V are i.i.d. with Vi=0EV_i=0 and |Vi|≤1 V_i ≤ 1, the elements are 11-sub-Gaussian, we can use the Bernstein’s inequality for sub-Gaussian random variables in quadratic form (see Lemma˜8). It implies that with probability at least 1−δ1-δ, ∥∇θl(θ∗)∥(Σ+λI)−12 _θl_D(θ^*) _( _D+λ I)^-1^2 =V⊤MV≤(M)+2(M2)log(1/δ)+2∥M∥oplog(1/δ) =V MV≤ Tr(M)+2 Tr(M^2) (1/δ)+2 M _op (1/δ) ≤β2dn+2β4n2dlog(1/δ)+2β2nlog(1/δ)=β2n(d+2dlog(1/δ)+2log(1/δ)). ≤ β^2dn+2 β^4n^2d (1/δ)+2 β^2n (1/δ)= β^2n(d+2 d (1/δ)+2 (1/δ)). Set a=da= d and b=log(1/δ)b= (1/δ). Note that we have d+2dlog(1/δ)+2log(1/δ) d+2 d (1/δ)+2 (1/δ) =(a+b)2+b2 =(a+b)^2+b^2 ≤2(a+b)2=2(a2+b2+2ab) ≤ 2(a+b)^2=2(a^2+b^2+2ab) ≤2(a2+b2+a2+b2)=4(a2+b2)=4(d+log(1/δ)), ≤ 2(a^2+b^2+a^2+b^2)=4(a^2+b^2)=4(d+ (1/δ)), where the last inequality is due to AM-GM inequality. Altogether, we have ∥∇θl(θ∗)∥(Σ+λI)−12≤4β2n(d+log(1/δ)) _θl_D(θ^*) _( _D+λ I)^-1^2≤ 4β^2n(d+ (1/δ)). The final assembly now begins as follows γ2∥Δ∥Σ+λI2 γ2 ^2_ _D+λ I =γ2∥Δ∥Σ2+γ2∥Δ∥λI2=γ2∥Δ∥Σ2+λγ2∥Δ∥2 = γ2 ^2_ _D+ γ2 _λ I^2= γ2 ^2_ _D+ λγ2 ^2 ≤∥∇θl(θ∗)∥(Σ+λI)−1∥Δ∥Σ+λI+λγ2∥Δ∥2 ≤ _θl_D(θ^*) _( _D+λ I)^-1 _ _D+λ I+ λγ2 ^2 ≤4β2n(d+log(1/δ))∥Δ∥Σ+λI+λγ24B2, ≤ 4β^2n(d+ (1/δ)) _ _D+λ I+ λγ24B^2, where the last inequality uses triangle inequality and the assumption that ∥θ∥≤B,∀θ∈Θ θ ≤ B,∀θ∈ . This implies that ∥Δ∥Σ+λI2≤2γ4β2n(d+log(1/δ))∥Δ∥Σ+λI+4λB2. ^2_ _D+λ I≤ 2γ 4β^2n(d+ (1/δ)) _ _D+λ I+4λ B^2. Now denote α=2γ4β2n(d+log(1/δ))α= 2γ 4β^2n(d+ (1/δ)) and β=4λB2β=4λ B^2, and let x=∥Δ∥Σ+λIx= _ _D+λ I. Since we have x2−αx−β≤0x^2-α x-β≤ 0, then x must be less than the bigger root, i.e., x≤α+α2+4β2≤α2+α2+4β2=α2+2β,x≤ α+ α^2+4β2≤ α^2+α^2+4β2= α^2+2β, where the second inequality is by Jensen’s inequality. Finally, we have that ∥θndpo−θ∗∥Σ+λI=∥Δ∥Σ+λI≤24β2γ2n(d+log(1/δ))+2λB2. θ^dpo_n-θ^* _ _D+λ I= _ _D+λ I≤ 2 4β^2γ^2n(d+ (1/δ))+2λ B^2. B.2 Proof of WDPO Loss Function Convergence Lemma 9 (Convergence of WDPO loss). Fix any θ∈Θθ∈ and ρ>0ρ>0. Let δ∈(0,1)δ∈(0,1). With probability 1−δ1-δ, |ℒW(θ;ρ)−ℒnW(θ;ρ)|≤K2log(2/δ)2n, ^W(θ;ρ)-L^W_n(θ;ρ) ≤ K^2 (2/δ)2n, where K=|logσ(−4βB)|K= σ(-4β B) . Proof. Recall the strong duality in Lemma˜1. The term infz∈[ηdp(z,z′)−l(z;θ)] _z [η d^p(z,z )-l(z;θ)] is called the Moreau-Yosida regularization of −l-l with parameter 1/η1/η. We denote it by lη(z;θ)l_η(z;θ). Now observe that |ℒW(θ;ρ)−ℒnW(θ;ρ)| ^W(θ;ρ)-L^W_n(θ;ρ) =|sup:p(,o)≤ρz∼[lz(θ)]−sup:p(,no)≤ρz∼[lz(θ)]| = _ P W_p( P, P^o)≤ρE_z P[l_z(θ)]- _ P W_p( P, P^o_n)≤ρE_z P[l_z(θ)] =(a)|infη≥0ηρp−z∼o[lη(z;θ)]−infη≥0ηρp−z∼no[lη(z;θ)]| (a)= _η≥ 0\ηρ^p-E_z P^o[l_η(z;θ)]\- _η≥ 0\ηρ^p-E_z P_n^o[l_η(z;θ)]\ ≤(b)supη≥0|z∼o[lη(z;θ)]−z∼no[lη(z;θ)]|, (b)≤ _η≥ 0 _z P^o[l_η(z;θ)]-E_z P^o_n[l_η(z;θ)] , where (a)(a) is by the strong duality, and (b)(b) is due to |infxf(x)−infxg(x)|≤supx|f(x)−g(x)| _xf(x)- _xg(x) ≤ _x f(x)-g(x) . Next, we will show that, for any η≥0η≥ 0, the function lηl_η is a bounded function. We first prove its upper bound. The negative DPO loss takes the following form: −l(z;θ)=ylogσ(x)+(1−y)logσ(−x)≤0,y∈0,1.-l(z;θ)=y σ(x)+(1-y) σ(-x)≤ 0, y∈\0,1\. The inequality is because the sigmoid function is strictly bounded between 0 and 11, i.e., σ∈(0,1)σ∈(0,1). This implies that logσ σ is non-positive. Using this, we have that lη(z;θ)=infz′∈[ηdp(z′,z)−l(z′;θ)]≤infz′∈[ηdp(z′,z)]=0.l_η(z;θ)= _z [η d^p(z ,z)-l(z ;θ)]≤ _z [η d^p(z ,z)]=0. Now we prove its lower bound. Recall that in the analysis of non-robust DPO loss, we proved that |hθ(s,a1,a2)|≤4B h_θ(s,a^1,a^2) ≤ 4B (see Eq.˜17). Since both log and σ are increasing functions, we have that logσ(βhθ(s,a1,a2))≥logσ(−4βB) σ(β h_θ(s,a^1,a^2))≥ σ(-4β B). Now observe that lη(z;θ) l_η(z;θ) =infz′∈[ηdp(z′,z)−l(z;′θ)] = _z [η d^p(z ,z)-l(z; θ)] ≥infz′∈[−l(z′;θ)]=infs,a1,a2,y[ylogσ(βhθ(s,a1,a2))+(1−y)logσ(βhθ(s,a2,a1))] ≥ _z [-l(z ;θ)]= _s,a^1,a^2,y[y σ(β h_θ(s,a^1,a^2))+(1-y) σ(β h_θ(s,a^2,a^1))] ≥logσ(−4βB), ≥ σ(-4β B), where the first inequality is because both η and metric dpd^p are non-negative. The last inequality is because only one of the logσ σ term will be activated and the lower bound we recalled above. Denote K=|logσ(−4βB)|K= σ(-4β B) . Since lηl_η is a bounded function, by Hoeffding’s inequality for bounded random variable (Lemma˜7), we have ℙ(|z∼o[lη(z;θ)]−z∼no[lη(z;θ)]|≥ϵ)≤2exp(−2nϵ2K2).P ( _z P^o[l_η(z;θ)]-E_z P^o_n[l_η(z;θ)] ≥ε )≤ 2exp ( -2nε^2K^2 ). By picking δ to be the right hand side above, we have that, with probability at least 1−δ1-δ, |z∼o[lη(z;θ)]−z∼no[lη(z;θ)]|≤K2log(2/δ)2n. _z P^o[l_η(z;θ)]-E_z P^o_n[l_η(z;θ)] ≤ K^2 (2/δ)2n. Since K does not depend on η, such concentration is uniform for all functions lη,η≥0l_η,η≥ 0. We have the desired result. ∎ B.3 Proof of the Strong Convexity of WDPO Loss We first prove that the function g(θ;)≔z∼[l(z;θ)]g(θ; P) _z P[l(z;θ)] is strongly convex, for any P, as follows: Lemma 10. Let l(z;θ)l(z;θ) be the DPO loss function. Assume that Assumption˜2 is in place. Then g(θ)≔z∼[l(z;θ)]g(θ) _z P[l(z;θ)] is γ-strongly convex with respect to norm ∥⋅∥Σ · _ _ P, where Σ=(s,a1,a2,y)∼(ψ(s,a1)−ψ(s,a2))(ψ(s,a1)−ψ(s,a2))⊤ _ P=E_(s,a^1,a^2,y) P(ψ(s,a^1)-ψ(s,a^2))(ψ(s,a^1)-ψ(s,a^2)) , and γ=β2e4βB(1+e4βB)2γ= β^2e^4β B(1+e^4β B)^2. Proof. Recall that we proved that the Hessian of the pointwise DPO loss takes the form: ∇θ2lz(θ)=β2σ(βhθ(s,a1,a2))σ(−βhθ(s,a1,a2))(ψ(s,a1)−ψ(s,a2))(ψ(s,a1)−ψ(s,a2))⊤. _θ^2l_z(θ)=β^2σ(β h_θ(s,a^1,a^2))σ(-β h_θ(s,a^1,a^2))(ψ(s,a^1)-ψ(s,a^2))(ψ(s,a^1)-ψ(s,a^2)) . In addition, we also proved that (see Eq.˜18) β2σ(βhθ(s,a1,a2))σ(−βhθ(s,a1,a2))≥β2e4βB(1+e4βB)2,∀θ∈Θ.β^2σ(β h_θ(s,a^1,a^2))σ(-β h_θ(s,a^1,a^2))≥ β^2e^4β B(1+e^4β B)^2, ∀θ∈ . This implies that u⊤∇θ2lz(θ)u≥γ∥(ψ(s,a1)−ψ(s,a2))⊤u∥22,∀u∈ℝd,u _θ^2l_z(θ)u≥γ (ψ(s,a^1)-ψ(s,a^2)) u ^2_2, ∀ u ^d, where γ=β2e4βB(1+e4βB)2γ= β^2e^4β B(1+e^4β B)^2. Thus, if we introduce the error vector Δ≔θ′−θ θ -θ, where θ,θ′∈Θθ,θ ∈ , then by the linear approximation theorem (Lemma˜3), there exists α∈[0,1]α∈[0,1] and θ~=αθ+(1−α)θ′ θ=αθ+(1-α)θ such that lz(θ+Δ)−lz(θ)−⟨∇θlz(θ),Δ⟩=12Δ⊤∇θ2lz(θ~)Δ≥γ2∥(ψ(s,a1)−ψ(s,a2))⊤Δ∥22=γ2∥Δ∥Σz2,l_z(θ+ )-l_z(θ)- _θl_z(θ), = 12 _θ^2l_z( θ) ≥ γ2 (ψ(s,a^1)-ψ(s,a^2)) ^2_2= γ2 ^2_ _z, (22) where Σz=(ψ(s,a1)−ψ(s,a2))(ψ(s,a1)−ψ(s,a2))⊤ _z=(ψ(s,a^1)-ψ(s,a^2))(ψ(s,a^1)-ψ(s,a^2)) . Note that Σz _z is only semi-definite. Let α∈[0,1]α∈[0,1] and θ,θ′∈Θθ,θ ∈ . Observe that g(αθ+(1−α)θ′) g(αθ+(1-α)θ ) =z∼[l(αθ+(1−α)θ′;z)] =E_z P[l(αθ+(1-α)θ ;z)] ≤(a)z∼[αl(z;θ)+(1−α)l(θ′;z)−γ2α(1−α)∥θ−θ′∥Σz2] (a)≤E_z P [α l(z;θ)+(1-α)l(θ ;z)- γ2α(1-α) θ-θ ^2_ _z ] =αg(θ)+(1−α)g(θ′)−γ2α(1−α)(θ−θ′)⊤[Σz](θ−θ′) =α g(θ)+(1-α)g(θ )- γ2α(1-α)(θ-θ ) E_ P[ _z](θ-θ ) =αg(θ)+(1−α)g(θ′)−γ2α(1−α)∥θ−θ′∥Σ2, =α g(θ)+(1-α)g(θ )- γ2α(1-α) θ-θ ^2_ _ P, where (a)(a) is by Lemma˜4. In particular, the equivalence between the inequalities, Eq.˜22 and (a)(a), can be found in the proof of Beck (2017, Theorem 5.24), and the author would like to comment that the proof does not rely on whether ∥⋅∥Σz · _ _z is a semi-norm or a norm. Now, by Assumption˜2, Σ _ P is strictly positive definite, hence ∥⋅∥Σ · _ _ P is a norm. This implies that g is γ-strongly convex with respect to ∥⋅∥Σ · _ _ P. ∎ Now, we are ready to prove our main strong convexity lemma. Lemma 11. Let l(z;θ)l(z;θ) be the DPO loss function. The Wasserstein distributionally robust DPO loss function, ℒW(θ;ρ)≔sup:p(,o)≤ρz∼[l(z;θ)],L^W(θ;ρ) _ P W_p( P, P^o)≤ρE_z P[l(z;θ)], is γλγλ-strongly convex in θ with respect to (non-weighted) 22-norm ∥⋅∥2 · _2, where λ is the regularity condition number defined in Assumption˜2, and γ=β2e4βB(1+e4βB)2γ= β^2e^4β B(1+e^4β B)^2. Proof. Let α∈[0,1]α∈[0,1] and θ,θ′∈Θθ,θ ∈ . First, we denote h(θ;)=z∼[l(z;θ)]h(θ; P)=E_z P[l(z;θ)] for any P in the Wasserstein ball. In Lemma˜10, we proved that h is γ-strongly convex in θ w.r.t. norm ∥⋅∥Σ · _ _ P. Now observe that ℒW(αθ+(1−α)θ′;ρ) ^W(αθ+(1-α)θ ;ρ) =sup:p(,o)≤ρh(αθ+(1−α)θ′;z) = _ P W_p( P, P^o)≤ρh(αθ+(1-α)θ ;z) ≤(a)sup:p(,o)≤ραh(θ;)+(1−α)h(θ′;)−γ2α(1−α)∥θ−θ′∥Σ2 (a)≤ _ P W_p( P, P^o)≤ρ \α h(θ; P)+(1-α)h(θ ; P)- γ2α(1-α) θ-θ ^2_ _ P \ ≤(b)αℒW(θ;ρ)+(1−α)ℒW(θ′;ρ)+sup:p(,o)≤ρ−γ2α(1−α)∥θ−θ′∥Σ2 (b)≤ ^W(θ;ρ)+(1-α)L^W(θ ;ρ)+ _ P W_p( P, P^o)≤ρ- γ2α(1-α) θ-θ ^2_ _ P =αℒW(θ;ρ)+(1−α)ℒW(θ′;ρ)−γ2α(1−α)inf:p(,o)≤ρ∥θ−θ′∥Σ2 = ^W(θ;ρ)+(1-α)L^W(θ ;ρ)- γ2α(1-α) _ P W_p( P, P^o)≤ρ θ-θ ^2_ _ P ≤αℒW(θ;ρ)+(1−α)ℒW(θ′;ρ)−γ2α(1−α)inf:p(,o)≤ρλmin(Σ)∥θ−θ′∥22 ≤ ^W(θ;ρ)+(1-α)L^W(θ ;ρ)- γ2α(1-α) _ P W_p( P, P^o)≤ρ _ min( _ P) θ-θ ^2_2 ≤(c)αℒW(θ;ρ)+(1−α)ℒW(θ′;ρ)−γλ2α(1−α)∥θ−θ′∥22. (c)≤ ^W(θ;ρ)+(1-α)L^W(θ ;ρ)- γλ2α(1-α) θ-θ ^2_2. Note that the function g(θ)=z∼[l(z;θ)]g(θ)=E_z P[l(z;θ)] is γ-strongly convex with respect to ∥⋅∥Σ · _ _ P by Lemma˜10. We use this fact in (a)(a). The inequality in (b)(b) is due to supx(f(x)+g(x))≤supxf(x)+supxg(x) _x(f(x)+g(x))≤ _xf(x)+ _xg(x). The last inequality (c)(c) is because λmin(Σ)≥λ _ min( _ P)≥λ, for all ∈ P _ W by Assumption˜2. This implies that ℒWL^W is a γλγλ-strongly convex function with respect to ∥⋅∥2 · _2. ∎ B.4 Proof of Policy Parameter Convergence of WDPO By Lemma˜9, we have that, with probability at least 1−δ1-δ, ℒW(θnW;ρ) ^W(θ^W_n;ρ) −ℒW(θW;ρ) -L^W(θ^W;ρ) =ℒW(θnW;ρ)−ℒnW(θnW;ρ)+ℒnW(θnW;ρ)−ℒnW(θW;ρ)+ℒnW(θW;ρ)−ℒW(θW;ρ) =L^W(θ^W_n;ρ)-L^W_n(θ^W_n;ρ)+L^W_n(θ^W_n;ρ)-L^W_n(θ^W;ρ)+L^W_n(θ^W;ρ)-L^W(θ^W;ρ) ≤|ℒW(θnW;ρ)−ℒnW(θnW;ρ)|+|ℒnW(θW;ρ)−ℒW(θW;ρ)| ≤ ^W(θ^W_n;ρ)-L^W_n(θ^W_n;ρ) + ^W_n(θ^W;ρ)-L^W(θ^W;ρ) ≤2K2log(2/δ)n, ≤ 2K^2 (2/δ)n, where the first inequality is because θnWθ^W_n is the minimizer of ℒnWL^W_n. Now by the γλγλ-strong convexity of ℒWL^W (see Lemma˜11) and Lemma˜5.I, we have that ∥θnW−θW∥22≤8K2log(2/δ)γ2λ2n. θ^W_n-θ^W _2^2≤ 8K^2 (2/δ)γ^2λ^2n. Appendix C Proof of KLDPO Sample Complexity We state a result from Hu and Hong (2013) that proves an equivalent condition for the infimum to be achieved at λ∗=0λ^*=0. Proposition 3 (Hu and Hong, 2013, Proposition 2). Let lu(z;θ)l_u(z;θ) be the essential supremum of l(z;θ)l(z;θ) under measure o P^o, i.e., lu(z;θ)=inft∈ℝ:ℙ(l(z;θ)>t)=0.l_u(z;θ)= \t (l(z;θ)>t)=0\. Also let κu=ℙ(l(z;θ)=lu(z;θ)) _u=P(l(z;θ)=l_u(z;θ)), i.e., κu _u is the probability mass of the distribution o P^o on the essential supremum of l. Then λ∗=0λ^*=0 if and only if lu(z;θ)<+∞l_u(z;θ)<+∞, κu>0 _u>0, and logκu+ρ≥0 _u+ρ≥ 0, where ρ is the diameter of the KL uncertainty set. We now make an assumption on the loss function(s) l(⋅;θ),θ∈Θl(·;θ),\;θ∈ . Note that this assumption is only used in proving the dual reformulation of KLDPO objective. Assumption 3. We assume that l(z;θ)≤Ll(z;θ)≤ L for all θ∈Θθ∈ . That is, the loss function is upper bounded by L. In addition, we also assume that Θ permits a uniform upper bound on λθ _θ. That is, we assume that supθ∈Θλθ<λ¯ _θ∈ _θ< λ. We now prove the following dual reformulation result: Lemma 12. Let l(z;θ)l(z;θ) be the DPO loss. The KLDPO loss function has the following dual reformulation ℒKL(θ;ρ)=sup:DKL(∥o)≤ρz∼[l(z;θ)]=infλ∈[λ¯,λ¯]λρ+λlog(z∼o[exp(l(z;θ)λ)]),L^KL(θ;ρ)= _ P D_KL( P\;\|\; P^o)≤ρE_z P[l(z;θ)]= _λ∈[ λ, λ] \λρ+λ (E_z P^o [exp ( l(z;θ)λ ) ] ) \, where 0<λ¯<λ¯<∞0< λ< λ<∞ are some constants. Proof. We include the derivation here for completeness. Previous works in optimization and distributionally robust reinforcement learning have covered the dual problem of distributionally robust optimization with KL uncertainty set (e.g., see Hu and Hong (2013); Panaganti and Kalathil (2022); Xu et al. (2023)). Recall that f(t)=tlog(t)f(t)=t (t) corresponds to the KL divergence. The optimal t for f∗(s)=supt≥0st−tlog(t)f^*(s)= _t≥ 0\st-t (t)\ is exp(s−1)exp (s-1 ). This implies that the Fenchel conjugate of f is f∗(s)=exp(s−1)f^*(s)=exp (s-1 ). From Lemma˜6, we get sup:DKL(∥o)≤ρz∼[l(z;θ)] _ P D_KL( P\;\|\; P^o)≤ρE_z P[l(z;θ)] =infλ≥0,η∈ℝz∼o[λf∗(l(z;θ)−ηλ)]+λρ+η = _λ≥ 0,η \E_z P^o [λ f^* ( l(z;θ)-ηλ ) ]+λρ+η \ =infλ≥0,η∈ℝz∼o[λexp(l(z;θ)−ηλ−1)]+λρ+η = _λ≥ 0,η \E_z P^o [ ( l(z;θ)-ηλ-1 ) ]+λρ+η \ =infλ≥0λρ+λlog(z∼o[exp(l(z;θ)λ)]), = _λ≥ 0 \λρ+λ (E_z P^o [exp ( l(z;θ)λ ) ] ) \, where the last equality by plugging in the optimal η, i.e., η∗=λlog(z∼o[exp(l(z;θ)/λ−1)])η^*=λ (E_z P^o[exp (l(z;θ)/λ-1 )]). Now observe that h(λ;θ)≔λρ+λlog(z∼o[exp(l(z;θ)λ)])≥λρ≕g(λ).h(λ;θ) λρ+λ (E_z P^o [exp ( l(z;θ)λ ) ] )≥λρ g(λ). The inequality is because the loss function is non-negative, i.e., l≥0l≥ 0, and h is increasing in l. Now g(λ)g(λ) is a strictly increasing function that lower bounds function h(λ;θ)h(λ;θ). Since g(λ)→∞g(λ)→∞ as λ→∞λ→∞, h(λ;θ)h(λ;θ) cannot achieve its infimum at ∞. In other words, there exists λ¯θ λ_θ such that h(λ;θ)≥g(λ)>g(λ¯θ),∀λ>λ¯θ.h(λ;θ)≥ g(λ)>g( λ_θ),∀ λ> λ_θ. This implies that it suffices to seek the infimum in [0,λ¯θ][0, λ_θ]. Hence, we have ℒKL(θ;ρ)=infλ∈[0,λ¯θ]λρ+λlog(z∼o[exp(l(z;θ)λ)]).L^KL(θ;ρ)= _λ∈[0, λ_θ] \λρ+λ (E_z P^o [exp ( l(z;θ)λ ) ] ) \. Now from Proposition˜3, the condition logκu+ρ≥0 _u+ρ≥ 0 is problem-dependent due to the diameter ρ, which is a design choice. Note that when κu _u is close to zero, the condition logκu+ρ≥0 _u+ρ≥ 0 is almost never true for a reasonable ρ. Hence, we ignore the case where λ∗=0λ^*=0. By Assumption˜3, without loss of generality, we have that λ∗∈[λ¯,λ¯]λ^*∈[ λ, λ], where λ¯ λ is some problem-specific constant. Then we have the result. In the literature of distributionally robust reinforcement learning, similar arguments can be found in Zhou et al. (2021); Panaganti and Kalathil (2022). ∎ Lemma 13. Fix any θ∈Θθ∈ and ρ>0ρ>0. Let δ∈(0,1)δ∈(0,1). Assume Assumption˜3 is in place. With probability 1−δ1-δ, we have that |ℒKL(θ;ρ)−ℒnKL(θ;ρ)|≤λ¯exp(L/λ¯)log(2/δ)2n,∀ϵ>0, ^KL(θ;ρ)-L^KL_n(θ;ρ) ≤ λ exp (L/ λ ) (2/δ)2n, ∀ε>0, where λ¯,λ¯ λ, λ are some constants that are independent of ϵε. Proof. Observe that |ℒKL(θ;ρ)−ℒnKL(θ;ρ)| ^KL(θ;ρ)-L^KL_n(θ;ρ) =|sup:DKL(∥o)≤ρz∼[l(z;θ)]−sup:DKL(∥no)≤ρz∼[l(z;θ)]| = _ P D_KL( P\;\|\; P^o)≤ρE_z P[l(z;θ)]- _ P D_KL( P\;\|\; P^o_n)≤ρE_z P[l(z;θ)] =(a)|infλ∈[λ¯,λ¯]λρ+λlog(z∼o[exp(l(z;θ)λ)]) (a)= _λ∈[ λ, λ] \λρ+λ (E_z P^o [exp ( l(z;θ)λ ) ] ) \ −infλ∈[λ¯,λ¯]λρ+λlog(z∼no[exp(l(z;θ)λ)])| - _λ∈[ λ, λ] \λρ+λ (E_z P^o_n [exp ( l(z;θ)λ ) ] ) \ ≤(b)supλ∈[λ¯,λ¯]|λlog(z∼no[exp(l(z;θ)λ)])−λlog(z∼o[exp(l(z;θ)λ)])| (b)≤ _λ∈[ λ, λ] λ (E_z P^o_n [exp ( l(z;θ)λ ) ] )-λ (E_z P^o [exp ( l(z;θ)λ ) ] ) =(c)supλ∈[λ¯,λ¯]λ|log(z∼no[exp(l(z;θ))/λ]z∼o[exp(l(z;θ))/λ])| (c)= _λ∈[ λ, λ]λ ( E_z P^o_n[exp (l(z;θ) )/λ]E_z P^o[exp (l(z;θ) )/λ] ) ≤supλ∈[λ¯,λ¯]λ|log(|z∼no[exp(l(z;θ))/λ]−z∼o[exp(l(z;θ))/λ]|z∼o[exp(l(z;θ))/λ]+1)| ≤ _λ∈[ λ, λ]λ ( _z P^o_n[exp (l(z;θ) )/λ]-E_z P^o[exp (l(z;θ) )/λ] E_z P^o[exp (l(z;θ) )/λ]+1 ) ≤(d)supλ∈[λ¯,λ¯]λ|z∼no[exp(l(z;θ))/λ]−z∼o[exp(l(z;θ))/λ]|z∼o[exp(l(z;θ))/λ] (d)≤ _λ∈[ λ, λ]λ _z P^o_n[exp (l(z;θ) )/λ]-E_z P^o[exp (l(z;θ) )/λ] E_z P^o[exp (l(z;θ) )/λ] ≤(e)λ¯supλ∈[λ¯,λ¯]|z∼no[exp(l(z;θ))/λ]−z∼o[exp(l(z;θ))/λ]|, (e)≤ λ _λ∈[ λ, λ] _z P^o_n[exp (l(z;θ) )/λ]-E_z P^o[exp (l(z;θ) )/λ] , where (a)(a) is by Lemma˜12. (b)(b) is because |infxf(x)−infxg(x)|≤supx|f(x)−g(x)| _xf(x)- _xg(x) ≤ _x f(x)-g(x) . (c)(c) is by Assumption˜3. (d)(d) is due to |log(1+x)|≤|x|,∀x≥0 (1+x) ≤ x ,∀ x≥ 0. (e)(e) is due to the fact that the loss function l is non-negative, i.e., l≥0l≥ 0. Now by applying Hoeffding’s inequality (Lemma˜7), we have ℙ(|z∼no[exp(l(z;θ))/λ]−z∼o[exp(l(z;θ))/λ]|≥ϵ)≤2exp(−2nϵ2exp(L/λ¯)).P( _z P^o_n[exp (l(z;θ) )/λ]-E_z P^o[exp (l(z;θ) )/λ] ≥ε)≤ 2exp (- 2nε^2exp (L/ λ ) ). By choosing ϵ=exp(L/λ¯)log(2/δ)2nε= exp (L/ λ ) (2/δ)2n, we have the result. ∎ We prove a strong convexity result similar to Lemma˜11 for KLDPO loss function. Lemma 14 (Strong convexity of KLDPO loss). Let l(z;θ)l(z;θ) be the DPO loss function. The KL distributionally robust DPO loss function, ℒKL(θ;ρ)≔sup:DKL(∥o)≤ρz∼[l(z;θ)],L^KL(θ;ρ) _ P D_KL( P\;\|\; P^o)≤ρE_z P[l(z;θ)], is γλγλ-strongly convex in θ with respect to (non-weighted) 22-norm ∥⋅∥2 · _2, where λ is the regularity condition number defined in Assumption˜2, and γ=β2e4βB(1+e4βB)2γ= β^2e^4β B(1+e^4β B)^2. Proof. Let α∈[0,1]α∈[0,1] and θ,θ′∈Θθ,θ ∈ . First, we denote h(θ;)=z∼[l(z;θ)]h(θ; P)=E_z P[l(z;θ)] for any P in the KL ball. In Lemma˜10, we proved that h is γ-strongly convex in θ w.r.t. norm ∥⋅∥Σ · _ _ P. Now observe that ℒKL(αθ ^KL(αθ +(1−α)θ′;ρ)=sup:DKL(∥o)≤ρh(αθ+(1−α)θ′;z) +(1-α)θ ;ρ)= _ P D_KL( P\;\|\; P^o)≤ρh(αθ+(1-α)θ ;z) ≤(a)sup:DKL(∥o)≤ραh(θ;)+(1−α)h(θ′;)−γ2α(1−α)∥θ−θ′∥Σ2 (a)≤ _ P D_KL( P\;\|\; P^o)≤ρ \α h(θ; P)+(1-α)h(θ ; P)- γ2α(1-α) θ-θ ^2_ _ P \ ≤(b)αℒKL(θ;ρ)+(1−α)ℒKL(θ′;ρ)+sup:DKL(∥o)≤ρ−γ2α(1−α)∥θ−θ′∥Σ2 (b)≤ ^KL(θ;ρ)+(1-α)L^KL(θ ;ρ)+ _ P D_KL( P\;\|\; P^o)≤ρ- γ2α(1-α) θ-θ ^2_ _ P =αℒKL(θ;ρ)+(1−α)ℒKL(θ′;ρ)−γ2α(1−α)inf:DKL(∥o)≤ρ∥θ−θ′∥Σ2 = ^KL(θ;ρ)+(1-α)L^KL(θ ;ρ)- γ2α(1-α) _ P D_KL( P\;\|\; P^o)≤ρ θ-θ ^2_ _ P ≤αℒKL(θ;ρ)+(1−α)ℒKL(θ′;ρ)−γ2α(1−α)inf:DKL(∥o)≤ρλmin(Σ)∥θ−θ′∥22 ≤ ^KL(θ;ρ)+(1-α)L^KL(θ ;ρ)- γ2α(1-α) _ P D_KL( P\;\|\; P^o)≤ρ _ min( _ P) θ-θ ^2_2 ≤(c)αℒKL(θ;ρ)+(1−α)ℒKL(θ′;ρ)−γλ2α(1−α)∥θ−θ′∥22. (c)≤ ^KL(θ;ρ)+(1-α)L^KL(θ ;ρ)- γλ2α(1-α) θ-θ ^2_2. Note that the function g(θ)=z∼[l(z;θ)]g(θ)=E_z P[l(z;θ)] is γ-strongly convex with respect to ∥⋅∥Σ · _ _ P by Lemma˜10. We use this fact in (a)(a). The inequality in (b)(b) is due to supx(f(x)+g(x))≤supxf(x)+supxg(x) _x(f(x)+g(x))≤ _xf(x)+ _xg(x). The last inequality (c)(c) is because λmin(Σ)≥λ _ min( _ P)≥λ, for all ∈KL P _KL by Assumption˜2. This implies that ℒKLL^KL is a γλγλ-strongly convex function with respect to ∥⋅∥2 · _2. ∎ C.1 Proof of Policy Parameter Convergence of KLDPO By Lemma˜13, we have that, with probability at least 1−δ1-δ, ℒKL(θnKL;ρ) ^KL(θ^KL_n;ρ) −ℒKL(θKL;ρ) -L^KL(θ^KL;ρ) =ℒKL(θnKL;ρ)−ℒnKL(θnKL;ρ)+ℒnKL(θnKL;ρ)−ℒnKL(θKL;ρ)+ℒnKL(θKL;ρ)−ℒKL(θKL;ρ) =L^KL(θ^KL_n;ρ)-L^KL_n(θ^KL_n;ρ)+L^KL_n(θ^KL_n;ρ)-L^KL_n(θ^KL;ρ)+L^KL_n(θ^KL;ρ)-L^KL(θ^KL;ρ) ≤|ℒKL(θnKL;ρ)−ℒnKL(θnKL;ρ)|+|ℒnKL(θKL;ρ)−ℒKL(θKL;ρ)| ≤ ^KL(θ^KL_n;ρ)-L^KL_n(θ^KL_n;ρ) + ^KL_n(θ^KL;ρ)-L^KL(θ^KL;ρ) ≤2λ¯exp(L/λ¯)log(2/δ)2n,∀ϵ>0, ≤ 2 λ exp (L/ λ ) (2/δ)2n, ∀ε>0, where the first inequality is because θnKLθ^KL_n is the minimizer of ℒnKLL^KL_n. Now by the γλγλ-strong convexity of ℒKLL^KL (see Lemma˜14) and Lemma˜5.I, we have that ∥θnKL−θKL∥22≤8λ¯2exp(L/λ¯)log(2/δ)γ2λ2n,∀ϵ>0. θ^KL_n-θ^KL _2^2≤ 8 λ^2exp (L/ λ ) (2/δ)γ^2λ^2n, ∀ε>0. Appendix D Proof of Tractable KLDPO Next, we prove the formal version of Proposition˜2. Theorem 3. Suppose we have the following distributionally robust loss that corresponds to a KL uncertainty set: sup:DKL(∥no)≤ρz∼[l(z;θ)]. _ P D_KL( P\;\|\; P_n^o)≤ρE_z P[l(z;θ)]. A worst distribution ¯∈ℝn P ^n is related to the empirical nominal distribution no P^o_n, which is constructed using n i.i.d. samples z1,…,znz_1,…,z_n, through ¯(i)=no(i)⋅exp(l(zi;θ)−μ−λ), P(i)= P_n^o(i)·exp ( l(z_i;θ)-μ-λ ), (23) where ¯(i) P(i) corresponds to the worst-case mass on the i-th data, and further it is subject to ∑i=1nno(i)⋅exp(l(zi;θ)−μ−λ)⋅(l(zi;θ)−μ−λ) _i=1^n P_n^o(i)·exp ( l(z_i;θ)-μ-λ )· ( l(z_i;θ)-μ-λ ) =ρ, =ρ, (24) ∑i=1nno(i)⋅exp(l(zi;θ)−μ−λ) _i=1^n P_n^o(i)·exp ( l(z_i;θ)-μ-λ ) =1, =1, (25) λ λ ≥0. ≥ 0. (26) Proof. We re-write the objective as a convex optimization problem maximizep∈ℝn p ^nmaximize ⟨p,l⟩ p,\;l subject to ∑i=1npilog(piqi)≤ρ, _i=1^np_i ( p_iq_i )≤ρ, ⊤p=1, 1 p=1, pi≥0,∀i. p_i≥ 0,∀ i. First, we ignore the constraint pi≥0p_i≥ 0 which will be automatically satisfied later. Now, the associated Lagrangian function takes the form L(p,λ,μ)=⟨p,l⟩+λ(ρ−∑i=1npilog(pi/qi))+μ(1−⊤p).L(p,λ,μ)= p,\;l +λ(ρ- _i=1^np_i (p_i/q_i))+μ(1- 1 p). We can calculate the KKT conditions as follows ∂L∂pi=li−λ(log(pi/qi)+1)−μ=0. ∂ L∂ p_i=l_i-λ( (p_i/q_i)+1)-μ=0. This implies that pi=qiexp(li−μ−λ),∀i∈1,…,n.p_i=q_iexp ( l_i-μ-λ ), ∀ i∈\1,…,n\. In addition, we have other KKT conditions as follows ∑i=1npilog(pi/qi)−ρ _i=1^np_i (p_i/q_i)-ρ ≤0, ≤ 0, ∑i=1npi _i=1^np_i =1, =1, λ λ ≥0, ≥ 0, λ(∑i=1npilog(pi/qi)−ρ) λ( _i=1^np_i (p_i/q_i)-ρ) =0. =0. From complimentary slackness, we have ∑i=1npilog(pi/qi)=ρ. _i=1^np_i (p_i/q_i)=ρ. The unconstrained optimum would lie at a vertex far from q, thus the best achievable objective under the KL constraint is obtained by pushing the distribution as far as possible, thereby maximizing the utility of the KL budget. Plugging in pi=qiexp(λ−1(li−μ−λ))p_i=q_iexp (λ^-1(l_i-μ-λ) ), we have ∑i=1nqiexp(li−μ−λ)⋅(li−μ−λ)=ρ. _i=1^nq_iexp ( l_i-μ-λ )· ( l_i-μ-λ )=ρ. Also, we have ∑i=1nqiexp(λ−1(li−μ−λ))=1 _i=1^nq_iexp (λ^-1(l_i-μ-λ) )=1. In addition, it is easy to see that the constraints pi≥0p_i≥ 0, ∀i∀ i, are satisfied since qiexp(λ−1(li−μ−λ))≥0q_iexp (λ^-1(l_i-μ-λ) )≥ 0. ∎ Here, μ and λ are implicitly defined by the constraints (Eq.˜24-Eq.˜26). Now we prove that the dual variables −μ−λ-μ-λ can be upper bounded by −∑i=1nqil(zi;θ)- _i=1^nq_il(z_i;θ). Proposition 4. −μ−λ-μ-λ satisfies −μ−λ≤−∑i=1nno(i)l(zi;θ)-μ-λ≤- _i=1^n P_n^o(i)l(z_i;θ). Proof. Recall the constraint ∑i=1nqiexp(l(zi;θ)−μ−λ)=1. _i=1^nq_iexp ( l(z_i;θ)-μ-λ )=1. By applying Jensen’s inequality, we have exp(∑i=1nqi(l(zi;θ)−μ−λ))≤1.exp ( _i=1^nq_i ( l(z_i;θ)-μ-λ ) )≤ 1. Some algebra give us exp(∑i=1nqi(l(zi;θ)λ))≤exp(μ+λ).exp ( _i=1^nq_i ( l(z_i;θ)λ ) ) ( μ+λ ). This implies that −μ−λ≤−∑i=1nqil(zi;θ)-μ-λ≤- _i=1^nq_il(z_i;θ). ∎ Appendix E Additional Experiment Results E.1 ArmoRM Multi-objective Alignment Figure 4: Evaluation of WDPO, KLDPO and DPO on rconvex∗(α)r^*_convex(α) in ArmoRM Multi-objective Alignment. We evaluate WDPO with robustness parameter ρo∈0.005,0.01 _o∈\0.005,0.01\ and KLDPO with robustness temperature τ∈1,5τ∈\1,5\. Similar to the Emotion Alignment experiments, we generate preference labels according to convex combinations of two reward objectives, i.e., rconvex∗r^*_convex defined as Mixture Evaluation in previous section. Specifically, we consider three pairs of objectives: (1) Ultrafeedback-Honesty and Helpsteer-Complexity, (2) Ultrafeedback-Helpfulness and Helpsteer-Coherence, and (3) Ultrafeedback-truthfulness and Helpsteer-Complexity. We generate preference labels according to αo=0.5α^o=0.5 for all three cases. All models are trained for 4 epochs. Then we introduce reward shift by evaluating WDPO, KLDPO, and DPO on rconvex∗(α)r^*_convex(α), where α∈0,0.1,0.3,0.5,0.7,0.9,1α∈\0,0.1,0.3,0.5,0.7,0.9,1\. In the first plot of Fig.˜4, the training preferences are generated accoding to reward pair (1). We observe that WDPO and KLDPO achieve superior performance compared to DPO. In particular, when reward distribution shift happens in two directions (towards standalone Ultrafeedback-Honesty and Helpsteer-Complexity), they clearly outperform DPO. In the middle plot, the training preferences are generated according to reward pair (2). We observe that WDPO with both ρo=0.005 _o=0.005 and ρo=0.01 _o=0.01 are particularly robust against reward distribution shift. Lastly, in the third plot, the training preferences are generated according reward pair (3). We observe that both WDPO and KLDPO achieve notable robustness compared to DPO. E.2 Leaderboard Alignment In this section, we include alignment results evaluated on all OpenLLM Leaderboard v2 (Fourrier et al., 2024) sub-tasks. We list all sub-task names in Table˜2. Table 2: All sub-task names in OpenLLM Leaderboard v2. 1 bbh-boolean-expressions 21 bbh-tracking-shuffled-objects-five-objects 2 bbh-causal-judgement 22 bbh-tracking-shuffled-objects-seven-objects 3 bbh-date-understanding 23 bbh-tracking-shuffled-objects-three-objects 4 bbh-disambiguation-qa 24 bbh-web-of-lies 5 bbh-formal-fallacies 25 gpqa-diamond 6 bbh-geometric-shapes 26 gpqa-extended 7 bbh-hyperbaton 27 gpqa-main 8 bbh-logical-deduction-five-objects 28 ifeval 9 bbh-logical-deduction-seven-objects 29 math-algebra-hard 10 bbh-logical-deduction-three-objects 30 math-counting-and-prob-hard 11 bbh-movie-recommendation 31 math-geometry-hard 12 bbh-navigate 32 math-intermediate-algebra-hard 13 bbh-object-counting 33 math-num-theory-hard 14 bbh-penguins-in-a-table 34 math-prealgebra-hard 15 bbh-reasoning-about-colored-objects 35 math-precalculus-hard 16 bbh-ruin-names 36 mmlu-pro 17 bbh-salient-translation-error-detection 37 musr-murder-mysteries 18 bbh-snarks 38 musr-object-placements 19 bbh-sports-understanding 39 musr-team-allocation 20 bbh-temporal-sequences LLaMA-3.2-1B results: In Table˜3, we compare DPO, KLDPO, and WDPO trained using LLaMA-3.2-1B on all 39 sub-tasks of OpenLLM Leaderboard v2. We observe that our WDPO and KLDPO methods achieve superior alignment performance on the majority of subtasks. Although WDPO and KLDPO slightly underperform on few subtasks, their primary strength lies in generalization, precisely because they consistently enhance performance across a diverse range of subtasks. LLaMA-3.1-8B results: In Table˜4, we compare DPO and KLDPO, both trained using LLaMA-3.1-8B. Earlier, we demonstrated that WDPO, trained on LLaMA-3.2-1B, outperforms both DPO and KLDPO. However, WDPO’s requirement for dual gradient computations increases its computational complexity. Due to resource constraints, we present KLDPO results for the 8B model, as it is more scalable. Following the LLaMA-3.2-1B experiments, we train KLDPO for two epochs, the point where DPO achieved optimal robustness via early stopping. Notably, KLDPO exhibits exceptional performance on math-related tasks. LLaMA-3.2-1B 1 2 3 4 5 6 7 8 9 10 11 12 13 DPO at Epoch 2 (early stopping) 0.64 0.50 0.36 0.40 0.52 0.31 0.51 0.22 0.16 0.32 0.35 0.49 0.36 DPO at Epoch 4 (goodfit) 0.50 0.49 0.34 0.36 0.52 0.31 0.52 0.22 0.13 0.31 0.34 0.45 0.36 KLDPO τ=0.1τ=0.1 0.69 0.52 0.40 0.43 0.53 0.34 0.52 0.18 0.18 0.33 0.36 0.48 0.40 KLDPO τ=0.05τ=0.05 0.71 0.52 0.38 0.38 0.53 0.34 0.52 0.19 0.17 0.33 0.36 0.48 0.41 WDPO ρo=0.01 _o=0.01 0.73 0.52 0.45 0.40 0.53 0.22 0.52 0.21 0.20 0.35 0.34 0.46 0.38 WDPO ρo=0.005 _o=0.005 0.69 0.51 0.41 0.40 0.54 0.32 0.52 0.19 0.15 0.32 0.35 0.50 0.40 LLaMA-3.2-1B 14 15 16 17 18 19 20 21 22 23 24 25 26 DPO at Epoch 2 (early stopping) 0.25 0.16 0.14 0.22 0.53 0.51 0.23 0.18 0.17 0.36 0.49 0.30 0.28 DPO at Epoch 4 (goodfit) 0.23 0.15 0.12 0.22 0.53 0.49 0.23 0.20 0.18 0.35 0.49 0.30 0.25 KLDPO τ=0.1τ=0.1 0.29 0.18 0.12 0.22 0.54 0.51 0.20 0.21 0.14 0.36 0.49 0.28 0.24 KLDPO τ=0.05τ=0.05 0.29 0.16 0.12 0.23 0.54 0.50 0.20 0.20 0.14 0.36 0.49 0.29 0.25 WDPO ρo=0.01 _o=0.01 0.26 0.18 0.10 0.30 0.54 0.50 0.20 0.18 0.13 0.39 0.49 0.24 0.26 WDPO ρo=0.005 _o=0.005 0.26 0.14 0.12 0.26 0.54 0.50 0.16 0.20 0.16 0.36 0.49 0.25 0.26 LLaMA-3.2-1B 27 28 29 30 31 32 33 34 35 36 37 38 39 DPO at Epoch 2 (early stopping) 0.22 0.48 0.14 0.09 0.08 0.01 0.04 0.19 0.01 0.17 0.50 0.26 0.28 DPO at Epoch 4 (goodfit) 0.23 0.48 0.18 0.02 0.03 0.01 0.05 0.13 0.05 0.17 0.49 0.26 0.23 KLDPO τ=0.1τ=0.1 0.24 0.53 0.20 0.05 0.08 0.01 0.08 0.12 0.02 0.18 0.52 0.23 0.24 KLDPO τ=0.05τ=0.05 0.24 0.56 0.21 0.05 0.05 0.03 0.05 0.13 0.04 0.18 0.52 0.21 0.24 WDPO ρo=0.01 _o=0.01 0.24 0.52 0.23 0.07 0.07 0.03 0.05 0.13 0.04 0.19 0.52 0.24 0.25 WDPO ρo=0.005 _o=0.005 0.25 0.49 0.19 0.07 0.08 0.04 0.06 0.19 0.03 0.19 0.51 0.25 0.24 Table 3: Evaluation of DPO, KLDPO, and WDPO on all OpenLLM Leaderboard v2 sub-tasks. LLaMA-3.1-8B 1 2 3 4 5 6 7 8 9 10 11 12 13 DPO at Epoch 2 (early stopping) 0.72 0.60 0.51 0.64 0.57 0.29 0.65 0.41 0.39 0.62 0.48 0.66 0.32 DPO at Epoch 4 (goodfit) 0.70 0.59 0.47 0.59 0.56 0.30 0.65 0.42 0.40 0.61 0.46 0.66 0.32 KLDPO τ=0.005τ=0.005 0.79 0.58 0.51 0.61 0.56 0.33 0.62 0.35 0.36 0.63 0.48 0.66 0.31 KLDPO τ=0.01τ=0.01 0.80 0.59 0.51 0.59 0.55 0.34 0.62 0.36 0.37 0.63 0.49 0.66 0.30 LLaMA-3.1-8B 14 15 16 17 18 19 20 21 22 23 24 25 26 DPO at Epoch 2 (early stopping) 0.46 0.66 0.65 0.51 0.61 0.68 0.41 0.21 0.23 0.34 0.50 0.30 0.28 DPO at Epoch 4 (goodfit) 0.47 0.59 0.66 0.51 0.61 0.70 0.40 0.21 0.20 0.32 0.50 0.27 0.31 KLDPO τ=0.005τ=0.005 0.47 0.66 0.65 0.54 0.63 0.72 0.46 0.27 0.25 0.34 0.49 0.29 0.25 KLDPO τ=0.01τ=0.01 0.47 0.65 0.64 0.53 0.65 0.74 0.47 0.28 0.25 0.33 0.50 0.32 0.28 LLaMA-3.1-8B 27 28 29 30 31 32 33 34 35 36 37 38 39 DPO at Epoch 2 (early stopping) 0.29 0.62 0.04 0.02 0.02 0.02 0.05 0.05 0.04 0.33 0.56 0.40 0.35 DPO at Epoch 4 (goodfit) 0.32 0.53 0.04 0.01 0.02 0.01 0.03 0.03 0.02 0.33 0.57 0.40 0.38 KLDPO τ=0.005τ=0.005 0.33 0.72 0.42 0.18 0.10 0.06 0.25 0.42 0.10 0.37 0.54 0.26 0.24 KLDPO τ=0.01τ=0.01 0.33 0.75 0.44 0.16 0.09 0.04 0.14 0.41 0.07 0.37 0.54 0.26 0.28 Table 4: Evaluation of DPO and KLDPO on all OpenLLM Leaderboard v2 sub-tasks. Appendix F Additional Experiment Details Reward Model Training: The raw Emotion dataset (Saravia et al., 2018) consists of text samples paired with multi-class labels for six different emotion classes (joy, sadness, love, anger, fear, and surprise). This dataset was then transformed into a multi-label dataset, referred to as the Emotion Reward Dataset. To create the multi-label dataset, the surprise class was excluded due to its limited representation in the original dataset. Following this, up to three random text samples from the raw dataset were concatenated, and their associated labels were merged. This pre-processing step ensured that the reward model encountered text samples representing multiple emotions during training. For the reward model, GPT-2 was employed, augmented with a classification head applied to the last token. The model was trained using a sigmoid activation function and binary cross-entropy loss, adhering to the standard multilabel classification framework. Training was conducted over 8 epochs with a batch size of 64, utilizing the Adam optimizer with a learning rate of 5.0×10−55.0× 10^-5 and a weight decay of 0.01. The reward model achieved a test accuracy of 84% and a test ROC-AUC score of 0.99. The emotion-specific scores predicted by this reward model were treated as the rewards for individual emotions. The ArmoRM setups did not need any reward model training. Supervised Fine-Tuning: Before training the WDPO algorithm, it is essential to ensure that the model familiarize with the types of texts present in the dataset. To achieve this, we performed supervised fine-tuning (SFT). We selected GPT-2 as the base language model and trained it to predict the next token based on the text samples in the emotion dataset. The maximum length of each text sample was capped at 68 tokens. The model was trained for 10 epochs with a batch size of 64. The training used the Adam optimizer (Kingma and Ba, 2014) with a learning rate of 5.0×10−75.0× 10^-7 following 12 warmup steps. Additionally, a maximum gradient norm of 10 was applied to stabilize the training. The ArmoRM setups did not need any SFT as we used Intruct models which have already undergone multiple rounds of SFT and alignment. Data Generation: (1) Emotion Alignment: A preference dataset was created, consisting of a chosen and a rejected completion for each prompt in the dataset. The first four tokens from each text in the emotion dataset were used as prompts. Using the SFT model, two completions were generated for each prompt. These completions were generated with a top-k value of 0, top-p of 1, and up to 64 new tokens. The completions were then evaluated using the reward model, and the chosen and rejected completions were determined based on a mixed metric derived from the predicted rewards. (2) ArmoRM multi-objective Alignment: Similar to the Emotion setup, we generated a preference datset by sampling two completions per prompt from the Helpsteer2 dataset. Each completion was sampled with a temperature of 0.7, top-p of 1 and up to 1024 new tokens. The prompts were also truncated to a miximum of 1024 tokens. We then fed these prompt-completion pairs to ArmoRM and used the scores from the first stage of the model as our multi-objective rewards. The chosen and rejected completions were determined based on a mixed metric derived from the predicted rewards. (3) Leaderboard Alignment: In this setup we sampled 10 completions per prompt in the Helpsteer2 dataset. Each completion was sampled with a temperature of 0.7, top-p of 1 and up to 1024 new tokens. We then fed these prompt-completion pairs to ArmoRM and used the scores from the second stage of the model as our reward, the completion with the maximum reward was our chosen completion while that with the minimum reward was our rejected completion. WDPO Implementation: (1) In WDPO training, one of the main challenges is calculating the gradient penalty of the DPO loss with respect to the input. However, since the input is tokenized as integers, gradient cannot be directly calculated. To address this, gradient is calculated with respect to the output of the embedding layer, where gradients are tracked. (2) In line 4 of the tractable WDPO algorithm (Algorithm˜1), we compute the gradient regularizer: ℛ(πθ;)=ρo(z∼∥∇zl(z;θ)∥22)1/2R( _θ;D)= _o(E_z _zl(z;θ) _2^2)^1/2. A key implementation challenge arises in distributed LLM training. A naive approach computes the gradient of the pointwise DPO loss with respect to each input, averages the gradient norms over the micro-batch, and applies this as a regularizer to the batch DPO loss on each worker. However, due to the typically small micro-batch sizes in large-scale LLM training, this averaging is performed over very few samples, resulting in a highly noisy and unstable gradient penalty. To mitigate this, we exploit the inequality x≤x x≤ x for x≥1x≥ 1, allowing us to upper bound the regularizer as: ℛ(πθ;)=ρo(z∼∥∇zl(z;θ)∥22)1/2≤ρo(z∼∥∇zl(z;θ)∥22).R( _θ;D)= _o(E_z _zl(z;θ) _2^2)^1/2≤ _o(E_z _zl(z;θ) _2^2). This leads to a tractable approximation of the pointwise WDPO loss: lW(zi,ρo)=l(zi;θ)+ρo∥∇zl(z;θ)∥22,l^W(z_i, _o)=l(z_i;θ)+ _o _zl(z;θ) _2^2, where l(zi;θ)l(z_i;θ) denotes the standard DPO loss for sample ziz_i. WDPO Training: (1) Emotion alignment: The model was trained for 40 epochs with an effective batch size of 64. We used Adam optimizer, with a learning rate of 5.0×10−75.0× 10^-7 following 12 warmup steps. A maximum gradient norm of 10 was applied to ensure stable training. The DPO beta parameter was set to 0.1 for all training runs. Experiments were conducted on a single 40 GB A100 GPU, requiring gradient accumulation over two steps. (2) LLaMA experiments: The models were trained for 8 epochs with an effective batch size of 128. We used Adam optimizer with a learning rate of 5.0×10−75.0× 10^-7 after 10%10\% warmup ratio and then the learning rate was reduced using a cosine scheduler. The DPO beta parameter was set to 0.01 for all training runs. Experiments were conducted on an 8xH100 GPU setup, requiring loading the model in bfloat16 and training with DeepSpeed ZeRO-2 optimizer (Rajbhandari et al., 2020). KLDPO Implementation: In line 3 of the tractable KLDPO algorithm (Algorithm˜2), we compute the approximate worst-case kernel ¯(i)∝exp((1/τ)(l(zi;θ)−(1/n)∑i=1nl(zi;θ))) P(i) ((1/τ)(l(z_i;θ)-(1/n) Σ _i=1^nl(z_i;θ)) ). A key implementation challenge arises in distributed LLM training. A naive approach would calculate the (1/n)∑i=1nl(zi;θ)(1/n) Σ _i=1^nl(z_i;θ) term by averaging l(zi;θ)l(z_i;θ) across all samples in the micro-batch of its respective worker. However, because micro-batch sizes are typically small in large-scale LLM training, this results in averaging over only a few samples, making the worst-case kernel highly noisy. To mitigate this, we introduce a synchronization step that performs an all-gather operation to collect l(zi;θ)l(z_i;θ) values from all workers. This enables averaging over the full batch across all workers, significantly reducing the noise in the worst-case kernel. KLDPO Training: (1) Emotion alignment: The model was trained for 40 epochs with an effective batch size of 64. We used Adam optimizer (Kingma and Ba, 2014), with a learning rate of 5.0×10−75.0× 10^-7 following 12 warmup steps. A maximum gradient norm of 10 was applied to ensure stable training. The DPO beta parameter was set to 0.1 for all training runs. Experiments were conducted on a single 40 GB A100 GPU and gradient was accumulated over two steps to keep training consistent across all algorithms. (2) LLaMA experiments: The models were trained for 8 epochs with an effective batch size of 128. We used Adam optimizer with a learning rate of 5.0×10−75.0× 10^-7 after 10%10\% warmup ratio and then the learning rate was reduced using a cosine scheduler. The DPO beta parameter was set to 0.01 for all training runs. Experiments were conducted on an 8xH100 GPU setup, requiring loading the model in bfloat16 and training with DeepSpeed ZeRO-2 optimizer (Rajbhandari et al., 2020). Appendix G Limitations Theoretical Limitations: Our theoretical analysis relies on Assumption˜2, which ensures sufficient data coverage to guarantee strong convexity conditions. Although such data-coverage assumptions are standard within offline learning or fixed-dataset scenarios, they are moderately restrictive, as they require the training dataset to sufficiently cover the space of feature differences between the preferred and dis-preferred actions. The log-linear policy class assumption, while standard and easily extendable to neural network policies under mild additional conditions, does not constitute a significant limitation. Experimental Limitations: Empirically, Wasserstein Direct Preference Optimization (WDPO) involves two separate gradient computations during training, one for calculating the gradient penalty and another for updating policy parameters via standard gradient descent. This dual-gradient requirement can increase computational complexity and training difficulty, potentially limiting practical scalability and efficiency compared to methods with a single gradient computation. Appendix H Impact Statement This paper aims to advance the field of machine learning by improving the robustness of direct preference optimization against preference model shifts. Our theoretical insights and empirical evaluations contribute to the reliability of preference-based learning methods. While our work has broad implications for AI alignment and deployment, we do not foresee any immediate societal concerns that require specific highlighting.