Paper deep dive
Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization
Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D. Lee, Wen Sun, Akshay Krishnamurthy, Dylan J. Foster
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/12/2026, 7:37:31 PM
Summary
The paper introduces Chi-Squared Preference Optimization (χPO), a novel offline alignment algorithm for language models. χPO addresses the 'overoptimization' phenomenon—where models degrade by overfitting to inaccurate reward models—by replacing standard KL-regularization with χ2-divergence regularization. This approach implicitly implements the principle of pessimism in the face of uncertainty, providing provable robustness to overoptimization and achieving strong sample-complexity guarantees based on single-policy concentrability.
Entities (5)
Relation Signals (3)
PO → mitigates → Overoptimization
confidence 98% · χPO... provably alleviates overoptimization, achieving sample-complexity guarantees based on single-policy concentrability
PO → improvesupon → DPO
confidence 95% · χPO is a one-line change to Direct Preference Optimization (DPO; Rafailov et al., 2023), which only involves modifying the logarithmic link function
KL-regularization → failstoprevent → Overoptimization
confidence 90% · we first show that the KL-regularization is too weak to prevent overfitting
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Language model alignment methods such as reinforcement learning from human feedback (RLHF) have led to impressive advances in language model capabilities, but are limited by a widely observed phenomenon known as overoptimization, where the quality of the language model degrades over the course of the alignment process. As the model optimizes performance with respect to an offline reward model, it overfits to inaccuracies and drifts away from preferred responses covered by the data. To discourage such distribution shift, KL-regularization is widely employed in existing offline alignment methods, but overoptimization continues to harm performance. Lending theoretical insight into the source of these empirical observations, we first show that the KL-regularization is too weak to prevent overfitting, then raise the following question: is it possible to design an efficient algorithm that is provably robust to overoptimization? We address this question with a new algorithm for offline alignment, $\chi^2$-Preference Optimization ($\chi$PO). $\chi$PO is a one-line change to Direct Preference Optimization (DPO; Rafailov et al., 2023), which only involves modifying the logarithmic link function in the DPO objective. Despite this minimal change, $\chi$PO implicitly implements the principle of pessimism in the face of uncertainty via regularization with the $\chi^2$-divergence -- which quantifies uncertainty more effectively than KL-regularization -- and provably alleviates overoptimization, achieving sample-complexity guarantees based on single-policy concentrability -- the gold standard in offline reinforcement learning. $\chi$PO's simplicity and strong guarantees make it the first practical and general-purpose offline alignment algorithm that is provably robust to overoptimization.
Tags
Links
- Source: https://arxiv.org/abs/2407.13399
- Canonical: https://arxiv.org/abs/2407.13399
Trouble viewing inline? Open PDF directly →
Full Text
708,792 characters extracted from source content.
Expand or collapse full text
Proof. [Tengyang]txviolet Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via χ2superscript2χ^2χ2-Preference Optimization Audrey Huang audreyh5@illinois.edu Wenhao Zhan wenhao.zhan@princeton.edu Tengyang Xie tx@cs.wisc.edu Jason D. Lee jasondlee88@gmail.com Wen Sun ws455@cornell.edu Akshay Krishnamurthy akshaykr@microsoft.com Dylan J. Foster dylanfoster@microsoft.com (February 18, 2025) Abstract Language model alignment methods such as reinforcement learning from human feedback (RLHF) have led to impressive advances in language model capabilities, but are limited by a widely observed phenomenon known as overoptimization, where the quality of the language model degrades over the course of the alignment process. As the model optimizes performance with respect to an offline reward model, it overfits to inaccuracies and drifts away from preferred responses covered by the data. To discourage such distribution shift, KL-regularization is widely employed in existing offline alignment methods, but overoptimization continues to harm performance. Lending theoretical insight into the source of these empirical observations, we first show that the KL-regularization is too weak to prevent overfitting, then raise the following question: is it possible to design an efficient algorithm that is provably robust to overoptimization? We address this question with a new algorithm for offline alignment, χ2superscript2χ^2χ2-Preference Optimization (χPO). χPO is a one-line change to Direct Preference Optimization (DPO; Rafailov et al. (2023)), which only involves modifying the logarithmic link function in the DPO objective. Despite this minimal change, χPO implicitly implements the principle of pessimism in the face of uncertainty via regularization with the χ2superscript2χ^2χ2-divergence—which quantifies uncertainty more effectively than KL-regularization—and provably alleviates overoptimization, achieving sample-complexity guarantees based on single-policy concentrability—the gold standard in offline reinforcement learning. χPO’s simplicity and strong guarantees make it the first practical and general-purpose offline alignment algorithm that is provably robust to overoptimization. 1 Introduction Large language models (LLMs) trained on unsupervised text data exhibit impressive and surprising capabilities (Brown et al., 2020; Ouyang et al., 2022; Touvron et al., 2023; OpenAI, 2023; Google, 2023), but can be difficult to control without further guidance. Reinforcement learning from human feedback (RLHF) and other alignment methods have emerged as a central tool to align these models to human values and elicit desired behavior (Christiano et al., 2017; Bai et al., 2022; Ouyang et al., 2022; Rafailov et al., 2023). This is achieved by treating the language model as a policy, and using techniques from reinforcement learning to optimize for desirable outcomes under a (explicit or implicit) reward model learned from a dataset of human-labeled responses. Alignment methods like RLHF have led to significant advances in language model capabilities, but existing techniques are limited by a widely observed phenomenon known as reward overoptimization or reward hacking (Michaud et al., 2020; Tien et al., 2022; Gao et al., 2023; Rafailov et al., 2024a). Since the reward model is an imperfect proxy for human preferences, the true quality of the language model can degrade as training proceeds, even as its performance under the reward model continues to improve. Intuitively, this occurs because the language model may drift away from the manifold covered by the human-labeled data used to train the reward model and end up in a region where the reward model is inaccurate. Overoptimization is distinct from the classical concept of overfitting because it is a causal or counterfactual phenomenon: When the human-labeled dataset does not cover all possible alternatives, the decision maker—in this case, a language model policy—cannot directly evaluate the effect of their actions. This perspective is supported by the fact that overoptimization can be mitigated by online alignment techniques (Guo et al., 2024; Gao et al., 2024; Dong et al., 2024), which exploit interactive access to human or AI feedback to iteratively improve the reward model; unfortunately, gathering such feedback is costly and impractical in many settings. This raises natural questions regarding the role of overoptimization in offline alignment: • Is overoptimization in offline alignment an information-theoretic phenomenon? This would mean that there is simply not enough information in the human-labeled (offline) preference dataset due to partial coverage, and no algorithmic intervention can avoid the overoptimization issue. • Alternatively, is overoptimization an algorithmic phenomenon? This would mean that existing algorithms are not making the most of the data they have (e.g., due to optimizing the wrong objective and converging toward suboptimal solutions) and would suggest that their sample-efficiency can be improved, perhaps by taking more aggressive measures to avoid overfitting to the reward model. Previous developments in the theory of offline reinforcement learning suggest that the answer may be the latter. Indeed, this literature has addressed the challenge of overoptimization—typically referred to as distribution shift—through the principle of pessimism in the face of uncertainty, which asserts that, given an offline dataset with partial coverage, a decision maker should choose their response according to the most pessimistic view of the world supported by the data. Pessimism encourages the model to avoid overfitting to the offline dataset and is supported by a rich theory showing that it offersprovable robustness to overoptimization in stylized settings (Liu et al., 2020; Jin et al., 2021; Rashidinejad et al., 2021). Perhaps the greatest barrier to implementing pessimism in language models is the efficient quantification of uncertainty in the offline reward, and the distillation of this information into actionable form. Most existing offline alignment methods employ KL-regularization, which penalizes the learned policy for drifting from the reference policy, but this form of uncertainty quantification is insufficient to induce pessimism (Gao et al., 2023) and is provably suboptimal in theory (Zhu et al., 2023; Song et al., 2024, see also Section A.1). On the other hand, offline reinforcement learning theory offers abstract pessimistic algorithms that are suitable—at least statistically—for large models (Xie et al., 2021; Uehara and Sun, 2021; Zhan et al., 2022; Chen and Jiang, 2022), but cannot be implemented directly without losing theoretical fidelity or making unrealistic modeling assumptions (Zhu et al., 2023; Zhan et al., 2023a; Li et al., 2023; Xiong et al., 2023; Liu et al., 2024; Cen et al., 2024; Fisch et al., 2024; Ji et al., 2024). Notably, the so-called “DPO+SFT” approach developed by Liu et al. (2024); Cen et al. (2024); Fisch et al. (2024) is provably suboptimal unless the language model satisfies an unrealistic convexity property (Section A.1). Can we develop practical offline alignment methods with provable robustness to overoptimization by exploiting the unique structure of the language modeling problem? 1.1 Contributions We introduce a new algorithm for offline alignment, χ2superscript2χ^2χ2-Preference Optimization (χPO). χPO is simple and straightforward to implement, requiring only a single-line change to Direct Preference Optimization (Rafailov et al. (2023)), yet it is provably robust to overoptimization. Algorithmically, χPO only differs from DPO in that we replace the usual logarithmic link function in the DPO objective with a new link function that implicitly implements pessimism via regularization with the χ2superscript2χ^2χ2-divergence—a divergence that (i) plays a fundamental role in statistics due to its ability to quantify uncertainty (Tsybakov, 2008); and (i) penalizes off-manifold behavior more effectively than KL-regularization. Statistically, we formalize robustness to overoptimization via a sample complexity guarantee based on single-policy concentrability—the gold standard in offline reinforcement learning—which we establish under minimal statistical and function approximation assumptions. This result implies that, in contrast to most prior work, χPO enjoys meaningful guarantees even when the reference policy has poor coverage. Summarizing: χPO is the first practical, general-purpose algorithm for offline alignment with provable robustness to overoptimization. The result above concerns the classical language model alignment formulation, which assumes the Bradley-Terry preference model (Christiano et al., 2017; Bai et al., 2022; Ouyang et al., 2022; Rafailov et al., 2023). Turning our attention to general preference models (Munos et al., 2023; Swamy et al., 2024; Rosset et al., 2024) where the goal is to find an approximate Nash equilibrium, we show that achieving guarantees based on single-policy concentrability is impossible. Nonetheless, we show that an iterative variant of χPO based on self-play achieves a sample complexity guarantee that scales with a new local coverage condition —a condition that is stronger than single policy concentrability, but much weaker than global concentrability and the notion of unilateral concentrability introduced by Cui and Du (2022). This result provides additional evidence for the value of regularization with χ2superscript2χ^2χ2-divergence for obtaining sharp sample complexity guarantees in language model alignment. Technical highlights Our analysis of χPO leverages several new techniques. First, we show that RLHF with χ2superscript2χ^2χ2-regularization is sufficient to achieve guarantees based on single-policy concentrability (Section 3.1 and Appendix B). Next, we show that a variant of the DPO reparameterization trick that combines χ2superscript2χ^2χ2-regularization with KL-regularization (“mixed” χ2superscript2χ^2χ2-regularization) can be used to reformulate our objective into a purely policy-based objective, in spite of the fact that χ2superscript2χ^2χ2-regularization fails to satisfy certain regularity conditions found in prior work (Wang et al., 2023a). Finally, and perhaps most importantly, we use a novel analysis to show that pessimism is preserved after reparameterization. Compared to prior approaches to pessimism in offline RL (Xie et al., 2021; Uehara and Sun, 2021; Zhan et al., 2022; Chen and Jiang, 2022), χ2superscript2χ^2χ2-regularization strikes a useful balance between generality and tractability. We expect our techniques—particularly the use of mixed χ2superscript2χ^2χ2-regularization—to find broader use. 1.2 Paper Organization Section 2 provides background on offline alignment and the suboptimality of existing algorithms. Section 3 presents our main algorithm, χPO, and accompanying theoretical guarantees. Section 4 then presents detailed intuition into how χPO modulates the bias-overoptimization tradeoff and implements pessimism, and Section 5 sketches the proof for its main statistical guarantee. We perform experimental evaluations of χPO against DPO in the TL;DR summarization task (Stiennon et al., 2020), which is included in Section 6. Section 7 contains results for general preference models, including an impossibility result for obtaining guarantees under single-policy concentrability in this setting. We conclude with discussion in Section 8. Proofs and additional results are deferred to the appendix, with highlights including (i) detailed discussion on suboptimality of existing pessimistic approaches (Appendix A), and (i) additional algorithms and guarantees based on the χ2superscript2χ^2χ2-regularization framework (Appendix B). Notation For an integer n∈ℕn ∈ blackboard_N, we let [n]delimited-[][n][ n ] denote the set 1,…,n1…\1,…,n\ 1 , … , n . For a set XX, we let Δ()Δ (X)Δ ( X ) denote the set of all probability distributions over XX. We adopt standard big-oh notation, and write f=O~(g)~f= O(g)f = over~ start_ARG O end_ARG ( g ) to denote that f=O(g⋅max1,polylog(g))⋅1polylogf=O(g· \1,polylog(g) \)f = O ( g ⋅ max 1 , polylog ( g ) ) and a≲bless-than-or-similar-toa ba ≲ b as shorthand for a=O(b)a=O(b)a = O ( b ). 2 Background In this section, we provide necessary background. We formally introduce the problem of language model alignment from human feedback (offline alignment), review standard algorithms (PPO and DPO), and highlight that in general, these algorithms suffer from provably suboptimal sample complexity arising from overoptimization, necessitating algorithmic interventions. 2.1 Alignment from Human Feedback Following prior work (e.g., Rafailov et al. (2023); Ye et al. (2024)), we adopt a contextual bandit formulation of the alignment problem. We formalize the language model as a policy π:→Δ():→Δπ:X→ (A)π : X → Δ ( A ) which maps a context (prompt) x∈x ∈ X to an action (response) a∈a ∈ A via a∼π(⋅∣x)a π(· x)a ∼ π ( ⋅ ∣ x ), and let ρ∈Δ()Δρ∈ (X)ρ ∈ Δ ( X ) denote the distribution over contexts/prompts. Offline alignment In the offline alignment problem (Christiano et al., 2017; Bai et al., 2022; Ouyang et al., 2022), we assume access to a dataset =(x,a+,a−)subscriptsubscriptsubscriptD_ pref= \(x,a_+,a_-) \Dsansserif_pref = ( x , a+ , a- ) of n prompts and labeled response pairs generated from a reference policy (language model) πsubscript _ refπsansserif_ref, which is typically obtained through supervised fine tuning. Here, a+subscripta_+a+ is a positive action/response and a−subscripta_-a- is a negative action/response. Given the context/prompt x∼ρsimilar-tox ∼ ρ, the pair (a+,a−)subscriptsubscript(a_+,a_-)( a+ , a- ) is generated by sampling a pair (a,b)(a,b)( a , b ) as a∼π(⋅∣x)a _ ref(· x)a ∼ πsansserif_ref ( ⋅ ∣ x ) and b∼π(⋅∣x)b _ ref(· x)b ∼ πsansserif_ref ( ⋅ ∣ x ), and then ordering them as (a+,a−)subscriptsubscript(a_+,a_-)( a+ , a- ) based on a binary preference y∼ℙ(a≻b∣x)similar-toℙsucceedsconditionaly P(a b x)y ∼ blackboard_P ( a ≻ b ∣ x ). We assume that preferences follow the Bradley-Terry model (Bradley and Terry, 1952), in which ℙ(a≻b∣x)=exp(r⋆(x,a))exp(r⋆(x,a))+exp(r⋆(x,b)),ℙsucceedsconditionalsuperscript⋆superscript⋆superscript⋆ (a b x)= (r (x,a) % ) (r (x,a) )+ (r (x,b) ),blackboard_P ( a ≻ b ∣ x ) = divide start_ARG exp ( r⋆ ( x , a ) ) end_ARG start_ARG exp ( r⋆ ( x , a ) ) + exp ( r⋆ ( x , b ) ) end_ARG , (1) for an unknown reward function r⋆:×→[0,R]:superscript⋆→0subscriptr :X×A→ [0,R_ max ]r⋆ : X × A → [ 0 , Rsansserif_max ] for some R≥1subscript1R_ max≥ 1Rsansserif_max ≥ 1. From the preference dataset subscriptD_ prefDsansserif_pref, we aim to learn a policy π^ πover start_ARG π end_ARG that has high reward in the sense that J(π⋆)−J(π^)≤ε,superscript⋆ J(π )-J( π)≤ ,J ( π⋆ ) - J ( over start_ARG π end_ARG ) ≤ ε , for a small ε>00 >0ε > 0, where J(π):=x∼ρ,a∼π(⋅∣x)[r⋆(x,a)]J(π) = E_x ρ,a π(· x% ) [r (x,a) ]J ( π ) := blackboard_Ex ∼ ρ , a ∼ π ( ⋅ ∣ x ) [ r⋆ ( x , a ) ] is the true expected reward, and π⋆superscript⋆π π⋆ is any comparator policy of interest. We abbreviate π[⋅]:=x∼ρ,a∼π(⋅∣x)[⋅] E_π [· ] =% E_x ρ,a π(· x)[·]blackboard_Eπ [ ⋅ ] := blackboard_Ex ∼ ρ , a ∼ π ( ⋅ ∣ x ) [ ⋅ ], and assume that ρ(x)>00ρ(x)>0ρ ( x ) > 0 for all x and π(a∣x)>0subscriptconditional0 _ ref(a x)>0πsansserif_ref ( a ∣ x ) > 0 for all x,ax,ax , a without loss of generality. Offline RLHF with KL-regularization Classical algorithms for offline alignment (Christiano et al., 2017; Ouyang et al., 2022) are based on reinforcement learning with a KL-regularized reward objective, defined for a regularization parameter β>00β>0β > 0, via Jβ(π)≔subscriptsuperscriptabsent J KL_β(π) _KLitalic_β ( π ) ≔ J(π)−β⋅D(π∥π)=π[r⋆(x,a)−βlogπ(a∣x)π(a∣x)],⋅subscriptconditionalsubscriptsubscriptdelimited-[]superscript⋆conditionalsubscriptconditional ~J(π)-β·D_ KL (π\,\|\, _ % ref )=E_π [r (x,a)-β π(a % x) _ ref(a x) ],J ( π ) - β ⋅ Dsansserif_KL ( π ∥ πsansserif_ref ) = blackboard_Eπ [ r⋆ ( x , a ) - β log divide start_ARG π ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ] , (2) where we adopt the shorthand D(π∥π′)=x∼ρ[D(π(⋅∣x)∥π′(⋅∣x))]D_ KL (π\,\|\,π )= E_x% ρ [D_ KL (π(· x)\,\|\,π (·% x) ) ]Dsansserif_KL ( π ∥ π′ ) = blackboard_Ex ∼ ρ [ Dsansserif_KL ( π ( ⋅ ∣ x ) ∥ π′ ( ⋅ ∣ x ) ) ]. These methods first estimate a reward function r^ rover start_ARG r end_ARG from subscriptD_ prefDsansserif_pref using maximum likelihood under the Bradley-Terry model: r^=argmaxr∈ℛ∑(x,a+,a−)∈logσ(r(a+∣x)−r(a−∣x)),^subscriptargmaxℛsubscriptsubscriptsubscriptsubscriptconditionalsubscriptconditionalsubscript r= argmax_r _(x,a_+,a_-)% _ pref σ (r(a_+ x)-r(a_- x)% ),over start_ARG r end_ARG = argmaxitalic_r ∈ R ∑( x , a start_POSTSUBSCRIPT + , a- ) ∈ Dsansserif_pref end_POSTSUBSCRIPT log σ ( r ( a+ ∣ x ) - r ( a- ∣ x ) ) , (3) where σ(x):=exp(x)1+exp(x)assign1σ(x) = (x)1+ (x)σ ( x ) := divide start_ARG exp ( x ) end_ARG start_ARG 1 + exp ( x ) end_ARG is the sigmoid function and ℛRR is a class of reward functions, which is typically parameterized by a neural network. Then, they apply standard policy optimization methods like PPO to optimize an estimated version of the KL-regularized objective: π^=argmaxπ∈Ππ[r^(x,a)−βlogπ(a∣x)π(a∣x)].^subscriptargmaxΠsubscriptdelimited-[]^conditionalsubscriptconditional π= argmax_π∈ E_% π [ r(x,a)-β π(a x) _ ref(a% x) ].over start_ARG π end_ARG = argmaxitalic_π ∈ Π blackboard_Eπ [ over start_ARG r end_ARG ( x , a ) - β log divide start_ARG π ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ] . The regularization term in Eq. 2 is intended to encourage π^ πover start_ARG π end_ARG to stay close to πsubscript _ refπsansserif_ref, with the hope of preventing the policy from overfitting to the potentially inaccurate reward model r^ rover start_ARG r end_ARG. Direct preference optimization (DPO) χPO is based on an alternative offline alignment approach, Direct Preference Optimization (DPO; Rafailov et al. (2023)). DPO uses the closed-form solution of the optimal KL-regularized policy under the objective Eq. 2—which can be viewed as implicitly modeling rewards—to define a single policy optimization objective that removes the need for direct reward function estimation. Given a user specified policy class Π Π, DPO solves π^DPO=argmaxπ∈Π∑(x,a+,a−)∈log[σ(βlogπ(a+∣x)π(a+∣x)−βlogπ(a−∣x)π(a−∣x))],subscript^DPOsubscriptargmaxΠsubscriptsubscriptsubscriptsubscriptconditionalsubscriptsubscriptconditionalsubscriptconditionalsubscriptsubscriptconditionalsubscript π_ DPO= argmax_π∈ % _(x,a_+,a_-) _ pref [σ (% β π(a_+ x) _ ref(a_+ x)-β % π(a_- x) _ ref(a_- x) ) ],over start_ARG π end_ARGDPO = argmaxitalic_π ∈ Π ∑( x , a start_POSTSUBSCRIPT + , a- ) ∈ Dsansserif_pref end_POSTSUBSCRIPT log [ σ ( β log divide start_ARG π ( a+ ∣ x ) end_ARG start_ARG πsansserif_ref ( a+ ∣ x ) end_ARG - β log divide start_ARG π ( a- ∣ x ) end_ARG start_ARG πsansserif_ref ( a- ∣ x ) end_ARG ) ] , (4) with the convention that the value of the objective is −∞-∞- ∞ if π does not satisfy π≪πmuch-less-thansubscriptπ _ refπ ≪ πsansserif_ref. 2.2 Overoptimization and Insufficiency of KL-Regularization Empirically, both classical RLHF and direct alignment methods like DPO have been observed to suffer from overoptimization (Gao et al., 2023; Guo et al., 2024; Rafailov et al., 2024a; Song et al., 2024), wherein model quality degrades during the optimization process as the learned policy drifts away from πsubscript _ refπsansserif_ref. The degree of degradation is affected by a number of factors, such as the objective used, the optimization landscape it induces, and the statistical properties of the algorithm. In this paper, we focus on mitigating the statistical problems the empirical phenomena ofunderlying the overoptimization phenomenon. As we will see, phenomena arethis phenomenon is an issue of sample-inefficiency when offline data coverage is inadequate, which can be understood through the lens of coverage coefficients developed in the theory of offline reinforcement learning (Liu et al., 2020; Jin et al., 2021; Rashidinejad et al., 2021). Coverage coefficients In offline reinforcement learning theory, the sample efficiency of an algorithm refers to the number of samples required to guarantee that J(π^)≈J(π⋆)^superscript⋆J( π)≈ J(π )J ( over start_ARG π end_ARG ) ≈ J ( π⋆ ). It is typically quantified by a coverage coefficient (or concentrability coefficient) that measures the quality of the data collected by the reference πsubscript _ refπsansserif_ref (Farahmand et al., 2010; Xie and Jiang, 2020; Zanette et al., 2021). We will utilize the L1subscript1L_1L1 coverage coefficient, defined for a policy π as π:=π[π(a∣x)π(a∣x)]assignsuperscriptsubscriptconditionalsubscriptconditionalC^π = E_π [ π(% a x) _ ref(a x) ]Citalic_π := blackboard_Eπ [ divide start_ARG π ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ]. Single policy concentrability is the gold standard for sample efficiency, and is obtained by an algorithm if, for any comparator policy π⋆superscript⋆π π⋆, the sample size required to learn J(π^)≈J(π⋆)^superscript⋆J( π)≈ J(π )J ( over start_ARG π end_ARG ) ≈ J ( π⋆ ) scales with π⋆superscriptsuperscript⋆C^π Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT, the coverage coefficient of π⋆superscript⋆π π⋆. This guarantees that π^ πover start_ARG π end_ARG is competitive with the best policy that is sufficiently covered by offline data, and, importantly, also guarantees that π^ πover start_ARG π end_ARG is never much worse than πsubscript _ refπsansserif_ref itself. Single policy concentrability is typically achieved by pessimistic algorithms that penalize the evaluations of candidate policies according to their uncertainty under the offline data, which prevents the learner from overfitting to inaccurate offline reward models. In contrast, the performance of non-pessimistic algorithms typically scales with all-policy concentrability—meaning that sample complexity scales with maxπ∈ΠπsubscriptΠsuperscript _π∈ C^πmaxitalic_π ∈ Π Citalic_π (Liu et al., 2020; Jin et al., 2021; Rashidinejad et al., 2021)— which is a guarantee achieved by even greedy algorithms that directly optimize the offline reward model without regularization. All-policy concentrability describes algorithms that cannot adapt to the quality of the data, and are thereby prone to overoptimization (of the offline reward model) unless the data is rich enough to cover all candidate policies sufficiently well. In contrast, single policy concentrability serves as a theoretical certification that an algorithm is robust to poor data coverage and will not overfit. Pessimism in offline alignment Zhu et al. (2023) show that the performance of PPO and DPO scales with all-policy concentrability, maxπ∞πsubscriptsuperscriptsubscript _πC_∞^πmaxitalic_π C∞italic_π, for the stylized case of alignment with linearly parameterized policies where πθ(a∣x)∝exp(⟨ϕ(x,a),θ⟩)proportional-tosubscriptconditionalitalic-ϕ _θ(a x) ( φ(x,a),θ )πitalic_θ ( a ∣ x ) ∝ exp ( ⟨ ϕ ( x , a ) , θ ⟩ ) for a known feature embedding ϕ(x,a)∈ℝditalic-ϕsuperscriptℝφ(x,a) ^dϕ ( x , a ) ∈ blackboard_Rd (see also Zhu et al. (2024); Song et al. (2024)). They also propose a pessimistic algorithm that achieves J(π⋆)−J(π^)≲poly(∞π⋆,d)n,less-than-or-similar-tosuperscript⋆^polysuperscriptsubscriptsuperscript⋆J(π )-J( π) poly(C% _∞^π ,d)n,J ( π⋆ ) - J ( over start_ARG π end_ARG ) ≲ square-root start_ARG divide start_ARG poly ( C∞italic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT , d ) end_ARG start_ARG n end_ARG end_ARG , simultaneously for all π⋆superscript⋆π π⋆.111Zhu et al. (2023) achieve guarantees based on a feature coverage coefficient, which improves on concentrability coefficients, but is specialized to linear function approximation. While encouraging, these results are restricted to linearly parameterized policies, and cannot be directly applied to large language models. Most existing theoretical algorithms for offline alignment are similar in nature, and either place restrictive assumptions on the policy class Π Π (Zhu et al., 2023; Zhan et al., 2023a; Li et al., 2023; Xiong et al., 2023) or are not feasible to implement in a way that is faithful to theory (Ye et al., 2024; Ji et al., 2024). Most relevant to our work, a series of recent papers (Liu et al., 2024; Cen et al., 2024; Fisch et al., 2024) propose implementing pessimism for general policy classes Π Π by solving the so-called “DPO+SFT” objective argmaxπ∈Πα⋅π[βlogπ(a∣x)]+1n∑(x,a+,a−)∈log[σ(βlogπ(a+∣x)π(a+∣x)−βlogπ(a−∣x)π(a−∣x))],subscriptargmaxΠ⋅subscriptsubscriptconditional1subscriptsubscriptsubscriptsubscriptconditionalsubscriptsubscriptconditionalsubscriptconditionalsubscriptsubscriptconditionalsubscript argmax_π∈ \α·% E_ _ ref [β π(a x)% ]+ 1n _(x,a_+,a_-) _ pref % [σ (β π(a_+ x) _ ref(a_+% x)-β π(a_- x) _ ref(a_- x)% ) ] \,argmaxitalic_π ∈ Π α ⋅ blackboard_Eπ start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT [ β log π ( a ∣ x ) ] + divide start_ARG 1 end_ARG start_ARG n end_ARG ∑( x , a start_POSTSUBSCRIPT + , a- ) ∈ Dsansserif_pref end_POSTSUBSCRIPT log [ σ ( β log divide start_ARG π ( a+ ∣ x ) end_ARG start_ARG πsansserif_ref ( a+ ∣ x ) end_ARG - β log divide start_ARG π ( a- ∣ x ) end_ARG start_ARG πsansserif_ref ( a- ∣ x ) end_ARG ) ] , (5) which augments the DPO objective (the second term) with an additional supervised fine-tuning-like (SFT) loss (the first term). While this objective is simple to apply to general policy classes, the existing single-policy concentrability guarantees for this method assume that Π Π satisfies restrictive convexity conditions which do not hold in practice for large language models. Perhaps surprisingly, we show (Section A.1) that without convexity, the objective in Eq. 5 fails to achieve a single-policy concentrability guarantee.222This finding is rather surprising because Xie et al. (2024) show that an optimistic online counterpart to Eq. 5, which negates the SFT term, enjoys online RLHF guarantees with general policy classes without requiring analogous convexity conditions. In other words, DPO+SFT is insufficient to mitigate overoptimization. 3 superscript2 χ^2italic_χbold_2-Preference Optimization This section presents our main algorithm, χPO. We begin by introducing χ2superscript2χ^2χ2-regularization as a general framework for mitigating overoptimization in offline alignment (Section 3.1), then derive the χPO algorithm (Section 3.2) and finally present our main theoretical guarantee (Section 3.3). 3.1 Framework: superscript2 χ^2italic_χbold_2-Regularized Reward Optimization The central algorithm design principle for our work is to (implicitly or explicitly) optimize a variant of the classical RLHF objective (Eq. 2) that replaces KL-regularization with regularization via χ2superscript2χ^2χ2-divergence, defined for a pair of probability measures ℙPblackboard_P and ℚQblackboard_Q with ℙ≪ℚmuch-less-thanℙℚP _P ≪ blackboard_Q via Dχ2(ℙ∥ℚ):=12∫(dℙdℚ−1)2dℚ.assignsubscriptsuperscript2conditionalℙℚ12superscriptdℙdℚ12differential-dℚ D_χ^2 (P\;\|\;Q ) % = 12 ( dPdQ-1% )^2dQ.Ditalic_χ2 ( blackboard_P ∥ blackboard_Q ) := divide start_ARG 1 end_ARG start_ARG 2 end_ARG ∫ ( divide start_ARG d blackboard_P end_ARG start_ARG d blackboard_Q end_ARG - 1 )2 d blackboard_Q . χ2superscript2χ^2χ2-divergence is a more aggressive form of regularization than KL-divergence; we have D(ℙ∥ℚ)≤2Dχ2(ℙ∥ℚ)subscriptconditionalℙℚ2subscriptsuperscript2conditionalℙℚD_ KL (P\,\|\,Q )≤2D_χ^2 % (P\;\|\;Q )Dsansserif_KL ( blackboard_P ∥ blackboard_Q ) ≤ 2 Ditalic_χ2 ( blackboard_P ∥ blackboard_Q ), but the converse is not true in general. We consider the following χ2superscript2χ^2χ2-regularized RL objective:333 Note the definition of Dχ2(π∥π)subscriptsuperscript2conditionalsubscriptD_χ^2 (π\;\|\; _ ref )Ditalic_χ2 ( π ∥ πsansserif_ref ) differs from [Dχ2(π(⋅∣x)∥π(⋅∣x))] E[D_χ^2 (π(· x)\;\|\; _% ref(· x) )]blackboard_E [ Ditalic_χ2 ( π ( ⋅ ∣ x ) ∥ πsansserif_ref ( ⋅ ∣ x ) ) ] only by a constant scaling and shift, both of which are inconsequential when used as regularization in an optimization objective. Jβχ(π):=π[r⋆(x,a)]−β⋅Dχ2(π∥π),Dχ2(π∥π):=π[π(a∣x)π(a∣x)].formulae-sequenceassignsuperscriptsubscriptsubscriptsuperscript⋆⋅subscriptsuperscript2conditionalsubscriptassignsubscriptsuperscript2conditionalsubscriptsubscriptconditionalsubscriptconditional J_β^χ(π) = E_% π [r (x,a) ]-β· D_χ^2 (π\;\|\; _% ref ), D_χ^2 (π\;\|\; _ ref% ):= E_π [ π(a x) _% ref(a x) ].Jitalic_βitalic_χ ( π ) := blackboard_Eπ [ r⋆ ( x , a ) ] - β ⋅ Ditalic_χ2 ( π ∥ πsansserif_ref ) , Ditalic_χ2 ( π ∥ πsansserif_ref ) := blackboard_Eπ [ divide start_ARG π ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ] . (6) Moving to a form of regularization that penalizes deviations from πsubscript _ refπsansserif_ref more forcefully than KL-regularization is a natural approach to mitigating overoptimization, but an immediate concern is that this may lead to overly conservative algorithms. As we will show, however, χ2superscript2χ^2χ2-divergence is better suited to the geometry of offline alignment, as it has the unique property (not shared by KL-divergence) that its value quantifies the extent to which the accuracy of a reward model r^ rover start_ARG r end_ARG trained under πsubscript _ refπsansserif_ref will transfer to a downstream policy π of interest (Lemma F.3). This implies that the χ2superscript2χ^2χ2-regularized RL objective in Eq. 6 meaningfully implements a form of pessimism in the face of uncertainty, and by tuning the regularization parameter β>00β>0β > 0, we can keep the learned policy π^ πover start_ARG π end_ARG close to πsubscript _ refπsansserif_ref in the “right” (uncertainty-aware) way. As such, we view optimizing χ2superscript2χ^2χ2-regularized rewards, i.e., argmaxπ∈ΠJβχ(π)subscriptargmaxΠsuperscriptsubscript argmax_π∈ J_β^χ(π)argmaxitalic_π ∈ Π Jitalic_βitalic_χ ( π ) as a general principle to guide algorithm design for offline alignment (as well as offline RL more broadly), which we expect to find broader use. We now turn our attention to the matter of how to optimize this objective. One natural approach, in the vein of classical RLHF algorithms (Christiano et al., 2017; Ouyang et al., 2022), is to estimate a reward model r^ rover start_ARG r end_ARG using maximum likelihood (Eq. 3), and then use PPO or other policy optimization methods to solve π^=argmaxπ∈Ππ[r^(x,a)]−β⋅Dχ2(π∥π)=argmaxπ∈Ππ[r^(x,a)−βπ(a∣x)π(a∣x)].^subscriptargmaxΠsubscriptdelimited-[]^⋅subscriptsuperscript2conditionalsubscriptsubscriptargmaxΠsubscriptdelimited-[]^conditionalsubscriptconditional π= argmax_π∈ E_% π [ r(x,a) ]-β· D_χ^2 (π\;\|\;% _ ref )= argmax_π∈ E_% π [ r(x,a)-β π(a x) _ ref(a% x) ].over start_ARG π end_ARG = argmaxitalic_π ∈ Π blackboard_Eπ [ over start_ARG r end_ARG ( x , a ) ] - β ⋅ Ditalic_χ2 ( π ∥ πsansserif_ref ) = argmaxitalic_π ∈ Π blackboard_Eπ [ over start_ARG r end_ARG ( x , a ) - β divide start_ARG π ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ] . (7) While this indeed leads to strong statistical guarantees (cf. Appendix B), we adopt a simpler and more direct approach inspired by DPO, which removes the need for a separate reward estimation step. 3.2 The χitalic_χPO Algorithm Algorithm 1 χ2superscript2χ^2χ2-Preference Optimization (χPO) 1:input: Reference policy πsubscript _ refπsansserif_ref, preference dataset subscriptD_ prefDsansserif_pref, χ2superscript2χ^2χ2-regularization coefficient β>00β>0β > 0. 2:Define ϕ(z):=z+logz.assignitalic-ϕ φ(z) =z+ z.ϕ ( z ) := z + log z . (8) 3:Optimize χ2superscript2χ^2χ2-regularized preference optimization objective: π^←argmaxπ∈Π∑(x,a+,a−)∈log[σ(2R[βϕ(π(a+∣x)π(a+∣x))−βϕ(π(a−∣x)π(a−∣x))])].←^subscriptargmaxΠsubscriptsubscriptsubscriptsubscriptsubscript2subscriptdelimited-[]italic-ϕconditionalsubscriptsubscriptconditionalsubscriptitalic-ϕconditionalsubscriptsubscriptconditionalsubscript π← argmax_π∈ _% (x,a_+,a_-) _ pref [σ ( % clip_2R_ max [βφ ( π(a_+ x) _% ref(a_+ x) )-βφ ( π(a_- x) _% ref(a_- x) ) ] ) ].over start_ARG π end_ARG ← argmaxitalic_π ∈ Π ∑( x , a start_POSTSUBSCRIPT + , a- ) ∈ Dsansserif_pref end_POSTSUBSCRIPT log [ σ ( sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ β ϕ ( divide start_ARG π ( a+ ∣ x ) end_ARG start_ARG πsansserif_ref ( a+ ∣ x ) end_ARG ) - β ϕ ( divide start_ARG π ( a- ∣ x ) end_ARG start_ARG πsansserif_ref ( a- ∣ x ) end_ARG ) ] ) ] . (9) 4:return: π^ πover start_ARG π end_ARG. Our main algorithm, χPO, is described in Algorithm 1. Given a preference dataset subscriptD_ prefDsansserif_pref and user-specified policy class Π Π, the algorithm learns a policy π^ πover start_ARG π end_ARG by solving the DPO-like optimization objective Eq. 9, which replaces the usual logπ(a∣x)π(a∣x)conditionalsubscriptconditional π(a x) _ ref(a x)log divide start_ARG π ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG terms in the original DPO objective (Eq. 4) with a new link function given by ϕ(π(a∣x)π(a∣x))=π(a∣x)π(a∣x)+log(π(a∣x)π(a∣x)).italic-ϕconditionalsubscriptconditionalconditionalsubscriptconditionalconditionalsubscriptconditionalφ ( π(a x) _ ref(a x) )= % π(a x) _ ref(a x)+ ( π(a x)% _ ref(a x) ).ϕ ( divide start_ARG π ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) = divide start_ARG π ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG + log ( divide start_ARG π ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) . A secondary modification is that we handle potentially unbounded density ratios by clipping to the interval [−2R,+2R]2subscript2subscript [-2R_ max,+2R_ max ][ - 2 Rsansserif_max , + 2 Rsansserif_max ] via the operator R(z)=maxminR,z,−Rsubscript clip_R(z)= \ \R,z \,-R \sansserif_clipR ( z ) = max min R , z , - R . In what follows, we will show that this simple and practical modification to DPO—that is, incorporating an additional density ratio term outside the logarithm—implicitly implements pessimism via χ2superscript2χ^2χ2-regularization. Algorithm derivation Recall that DPO is derived (Rafailov et al., 2023) by observing that the optimal KL-regularized policy πβ;⋆:=argmaxπ[r⋆(x,a)]−βD(π∥π)assignsubscriptsuperscript⋆subscriptargmaxsubscriptsuperscript⋆subscriptconditionalsubscriptπ _β; KL = argmax_π % \ E_π [r (x,a) ]-β D_ % KL (π\,\|\, _ ref ) \π⋆italic_β ; sansserif_KL := argmaxitalic_π blackboard_Eπ [ r⋆ ( x , a ) ] - β Dsansserif_KL ( π ∥ πsansserif_ref ) satisfies the following identity for all x∈x ∈ X and a∈a ∈ A. r⋆(x,a)=βlogπβ;⋆(a∣x)π(a∣x)+Zβ,r⋆;(x),superscript⋆subscriptsuperscript⋆conditionalsubscriptconditionalsubscriptsuperscript⋆ r (x,a)=β π _β; KL(a% x) _ ref(a x)+Z_β,r ; KL(x),r⋆ ( x , a ) = β log divide start_ARG π⋆italic_β ; sansserif_KL ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG + Zitalic_β , r⋆ ; sansserif_KL ( x ) , where Zβ,r⋆;(x)subscriptsuperscript⋆Z_β,r ; KL(x)Zitalic_β , r⋆ ; sansserif_KL ( x ) is a normalization constant that depends on x but not a. This facilitates reparameterizing the reward model in the maximum likelihood estimation objective (Eq. 3) in terms of a learned policy, yielding the DPO objective in Eq. 4. To apply a similar reparameterization trick for χ2superscript2χ^2χ2-divergence, a natural starting point is an observation from Wang et al. (2023a), who show that an analogous characterization for the optimal regularized policy holds for a general class of f-divergences. For a convex function f:ℝ+→ℝ:→subscriptℝf:R_+ : blackboard_R+ → blackboard_R, define the induced f-divergence by Df(ℙ∥ℚ)=∫f(dℙdℚ)dℚ=ℚ[f(dℙdℚ)].subscriptconditionalℙℚdℙdℚdifferential-dℚsubscriptℚdℙdℚD_f (P\,\|\,Q )= f ( d% PdQ )dQ=% E_Q [f ( dPd% Q ) ].Ditalic_f ( blackboard_P ∥ blackboard_Q ) = ∫ f ( divide start_ARG d blackboard_P end_ARG start_ARG d blackboard_Q end_ARG ) d blackboard_Q = blackboard_Eblackboard_Q [ f ( divide start_ARG d blackboard_P end_ARG start_ARG d blackboard_Q end_ARG ) ] . Wang et al. (2023a) show that for any differentiable f that satisfies the technical condition 0∉dom(f′)0domsuperscript′0 (f )0 ∉ dom ( f′ ), the optimal f-regularized policy πβ;f⋆=argmaxπ[r⋆(x,a)]−βDf(π∥π)subscriptsuperscript⋆subscriptargmaxsubscriptsuperscript⋆subscriptconditionalsubscriptπ _β;f= argmax_π \% E_π [r (x,a) ]-β D_f (π\,\|\, _% ref ) \π⋆italic_β ; f = argmaxitalic_π blackboard_Eπ [ r⋆ ( x , a ) ] - β Ditalic_f ( π ∥ πsansserif_ref ) satisfies r⋆(x,a)=βf′(πβ;f⋆(a∣x)π(a∣x))+Zβ,r⋆;f(x)superscript⋆superscript′subscriptsuperscript⋆conditionalsubscriptconditionalsubscriptsuperscript⋆r (x,a)=βf ( π _β;f(a x)% _ ref(a x) )+Z_β,r ;f(x)r⋆ ( x , a ) = β f′ ( divide start_ARG π⋆italic_β ; f ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) + Zitalic_β , r⋆ ; f ( x ) (10) for a normalization constant Zβ,r⋆;f(x)subscriptsuperscript⋆Z_β,r ;f(x)Zitalic_β , r⋆ ; f ( x ), allowing for a similar reparameterization. Informally, the condition 0∉dom(f′)0domsuperscript′0 (f )0 ∉ dom ( f′ ) means that Df(⋅∥π)D_f (·\,\|\, _ ref )Ditalic_f ( ⋅ ∥ πsansserif_ref ) acts as a barrier for the positive orthant, automatically forcing πβ;f⋆subscriptsuperscript⋆π _β;fπ⋆italic_β ; f to place positive probability mass on any action a for which π(a∣x)>0subscriptconditional0 _ ref(a x)>0πsansserif_ref ( a ∣ x ) > 0. The χ2superscript2χ^2χ2-divergence is an f-divergence corresponding to f(z)=12(z−1)212superscript12f(z)= 12(z-1)^2f ( z ) = divide start_ARG 1 end_ARG start_ARG 2 end_ARG ( z - 1 )2, but unfortunately does not satisfy the condition 0∉dom(f′)0domsuperscript′0 (f )0 ∉ dom ( f′ ), making Eq. 10 inapplicable. Indeed, the optimal χ2superscript2χ^2χ2-regularized policy can clip action probabilities to zero in a non-smooth fashion even when π(a∣x)>0subscriptconditional0 _ ref(a x)>0πsansserif_ref ( a ∣ x ) > 0, which means that the identity Eq. 10 does not apply. To address this issue, we augment χ2superscript2χ^2χ2-regularization by considering the mixed χ2superscript2χ^2χ2-divergence given by fχmix(z):=12(z−1)2+zlogzassignsubscriptsubscriptmix12superscript12f_ _ mix(z) = 12(z-1)^2+z zfitalic_χ start_POSTSUBSCRIPT mix end_POSTSUBSCRIPT ( z ) := divide start_ARG 1 end_ARG start_ARG 2 end_ARG ( z - 1 )2 + z log z, which has Dfχmix(ℙ∥ℚ)=Dχ2(ℙ∥ℚ)+D(ℙ∥ℚ).subscriptsubscriptsubscriptmixconditionalℙℚsubscriptsuperscript2conditionalℙℚsubscriptconditionalℙℚD_f_ _ mix (P\,\|\,Q )=D_χ^2% (P\;\|\;Q )+D_ KL (P\,\|% \,Q ).Ditalic_f start_POSTSUBSCRIPT χ start_POSTSUBSCRIPT mix end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( blackboard_P ∥ blackboard_Q ) = Ditalic_χ2 ( blackboard_P ∥ blackboard_Q ) + Dsansserif_KL ( blackboard_P ∥ blackboard_Q ) . In other words, we use both χ2superscript2χ^2χ2-regularization and KL-regularization; χ2superscript2χ^2χ2-regularization enforces pessimism, while KL-regularization enforces the barrier property and facilitates reparameterization. Indeed, the link function ϕitalic-ϕφϕ (Eq. 8) used in χPO has ϕ(z):=fχmix′(z)=z+logzassignitalic-ϕsuperscriptsubscriptsubscriptmix′φ(z) =f_ _ mix (z)=z+ zϕ ( z ) := fitalic_χ start_POSTSUBSCRIPT mix end_POSTSUBSCRIPT′ ( z ) = z + log z, which satisfies 0∉dom(fχmix′)0domsuperscriptsubscriptsubscriptmix′0∉dom(f_ _ mix )0 ∉ dom ( fitalic_χ start_POSTSUBSCRIPT mix end_POSTSUBSCRIPT′ ), so Eq. 10 yields the reparameterization r⋆(x,a)=βϕ(πβ;fχmix⋆(a∣x)π(a∣x))+Zβ,r⋆;fχmix(x)superscript⋆italic-ϕsubscriptsuperscript⋆subscriptsubscriptmixconditionalsubscriptconditionalsubscriptsuperscript⋆subscriptsubscriptmixr (x,a)=βφ ( π _β;f_ _ mix% (a x) _ ref(a x) )+Z_β,r ;f_ _% mix(x)r⋆ ( x , a ) = β ϕ ( divide start_ARG π⋆italic_β ; f start_POSTSUBSCRIPT χ start_POSTSUBSCRIPT mix end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) + Zitalic_β , r⋆ ; f start_POSTSUBSCRIPT χ start_POSTSUBSCRIPT mix end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( x ). Substituting this identity into the maximum likelihood estimation objective (Eq. 3) yields the χPO algorithm. Going forward, we define Jβ,rχmix(π)=π[r(x,a)]−β⋅Dχ2(π∥π)−β⋅D(π∥π)superscriptsubscriptsubscriptmixsubscript⋅subscriptsuperscript2conditionalsubscript⋅subscriptconditionalsubscriptJ_β,r _ mix(π)= E_π [r(% x,a) ]-β· D_χ^2 (π\;\|\; _ ref )-% β· D_ KL (π\,\|\, _ ref )Jitalic_β , ritalic_χmix ( π ) = blackboard_Eπ [ r ( x , a ) ] - β ⋅ Ditalic_χ2 ( π ∥ πsansserif_ref ) - β ⋅ Dsansserif_KL ( π ∥ πsansserif_ref ) for a reward function r. We use the shorthand πβ⋆=argmaxπJβ,r⋆χmix(π)subscriptsuperscript⋆subscriptargmaxsuperscriptsubscriptsuperscript⋆subscriptmixπ _ β= argmax_πJ_β% ,r _ mix(π)π⋆italic_β = argmaxitalic_π Jitalic_β , r⋆italic_χmix ( π ) as the optimal policy under mixed χ2superscript2χ^2χ2-regularization, and abbreviate Zβ,r(x):=Zβ,r;fχmix(x)assignsubscriptsubscriptsubscriptsubscriptmixZ_β,r(x) =Z_β,r;f_ _ mix(x)Zitalic_β , r ( x ) := Zitalic_β , r ; f start_POSTSUBSCRIPT χ start_POSTSUBSCRIPT mix end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( x ), so that r⋆(x,a)=βϕ(πβ⋆(a∣x)π(a∣x))+Zβ,r⋆(x).superscript⋆italic-ϕsubscriptsuperscript⋆conditionalsubscriptconditionalsubscriptsuperscript⋆r (x,a)=βφ ( π _ β(a% x) _ ref(a x) )+Z_β,r (x).r⋆ ( x , a ) = β ϕ ( divide start_ARG π⋆italic_β ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) + Zitalic_β , r⋆ ( x ) . (11) 3.3 Theoretical Guarantees To state our main sample complexity guarantee for χPO, we begin by making standard statistical assumptions. Let the regularization parameter β>00β>0β > 0 in χPO be fixed. We first make a realizability assumption, which states that the policy class Π Π used in χPO is sufficiently expressive to represent the optimal policy under mixed χ2superscript2χ^2χ2-regularization (Eq. 11); recall that in the context of language modeling, Π Π represents a class of language models with fixed architecture and varying weights. Assumption 3.1 (Policy realizability). The policy class Π Π satisfies πβ⋆∈Πsubscriptsuperscript⋆Ππ _ β∈ π⋆italic_β ∈ Π, where πβ⋆subscriptsuperscript⋆π _ βπ⋆italic_β is the optimal policy under mixed χ2superscript2χ^2χ2-regularization (Eq. 11). Policy realizability is a standard assumption for sample-efficient reinforcement learning (Agarwal et al., 2019; Lattimore and Szepesvári, 2020; Foster and Rakhlin, 2023), and is equivalent to reward model realizability in our setting via reparameterization. Our second assumption asserts that the implicit reward models induced by the policy class Π Π in χPO have bounded range. Assumption 3.2 (Bounded implicit rewards). For a parameter V≥RsubscriptsubscriptV_ max≥R_ maxVsansserif_max ≥ Rsansserif_max, it holds that for all π∈Ππ∈ π ∈ Π, x∈x ∈ X, and a,b∈a,b , b ∈ A, |βϕ(π(a∣x)π(a∣x))−βϕ(π(b∣x)π(b∣x))|≤V.italic-ϕconditionalsubscriptconditionalitalic-ϕconditionalsubscriptconditionalsubscript βφ ( π(a x) _ ref(a x)% )-βφ ( π(b x) _ ref(b x)% ) ≤ V_ max.| β ϕ ( divide start_ARG π ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) - β ϕ ( divide start_ARG π ( b ∣ x ) end_ARG start_ARG πsansserif_ref ( b ∣ x ) end_ARG ) | ≤ Vsansserif_max . Assumption 3.2 generalizes analogous assumptions made in the analysis of DPO-like algorithms in prior work (Rosset et al., 2024; Xie et al., 2024), and our guarantees scale polynomially with this parameter; see Section 4.4 for a detailed comparison. We emphasize that in practice, VsubscriptV_ maxVsansserif_max can be measured and directly controlled (e.g., via clipping). Example 3.1 (Policy classes induced by reward models). A natural setting in which both Assumption 3.1 and Assumption 3.2 hold is when the policy class Π Π is induced by a class of bounded reward function ℛ⊂(×→[0,R])ℛ→0subscriptR⊂(X×A→[0,R_ max])R ⊂ ( X × A → [ 0 , Rsansserif_max ] ) through the mixed-χ2superscript2χ^2χ2 parameterization, for β>00β>0β > 0: Πℛ,β:=π(a∣x)=π(a∣x)⋅ϕ−1(β−1(r(x,a)−Zβ,r(x)))∣r∈ℛ.assignsubscriptΠℛconditionalconditional⋅subscriptconditionalsuperscriptitalic-ϕ1superscript1subscriptℛ _R,β = \π(a x)= _ ref(a% x)·φ^-1(β^-1(r(x,a)-Z_β,r(x))) r % \.Πcaligraphic_R , β := π ( a ∣ x ) = πsansserif_ref ( a ∣ x ) ⋅ ϕ- 1 ( β- 1 ( r ( x , a ) - Zitalic_β , r ( x ) ) ) ∣ r ∈ R . (12) Here, Assumption 3.1 holds whenever r⋆∈ℛsuperscript⋆ℛr ⋆ ∈ R, and Assumption 3.2 is satisfied with V≤2Rsubscript2subscriptV_ max≤2R_ maxVsansserif_max ≤ 2 Rsansserif_max. Finally, recall the definition of the L1subscript1L_1L1 concentrability coefficient, π:=π[π(a∣x)π(a∣x)]assignsuperscriptsubscriptconditionalsubscriptconditionalC^π = E_π [ π(% a x) _ ref(a x) ]Citalic_π := blackboard_Eπ [ divide start_ARG π ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ], which is equivalent to the χ2superscript2χ^2χ2-divergence up to a constant shift, i.e., π=1+2Dχ2(π∥π)superscript12subscriptsuperscript2conditionalsubscriptC^π=1+2D_χ^2 (π\;\|\; _ ref )Citalic_π = 1 + 2 Ditalic_χ2 ( π ∥ πsansserif_ref ). We use L1subscript1L_1L1 concentrability to quantify coverage of a policy π by the offline preference dataset subscriptD_ prefDsansserif_pref generated by πsubscript _ refπsansserif_ref. how well the offline preference dataset subscriptD_ prefDsansserif_pref, generated by πsubscript _ refπsansserif_ref, covers a policy π, and the following result is our main sample complexity guarantee for χPO. Theorem 3.1 (Sample complexity bound for χPO). Suppose Assumptions 3.1 and 3.2 hold for some β>00β>0β > 0. With probability at least 1−δ11- 1 - δ, χPO (Algorithm 1) produces a policy π^ πover start_ARG π end_ARG such that for all policies π⋆superscript⋆π π⋆ simultaneously, we have J(π⋆)−J(π^)≲Ve2R⋅π⋆log(|Π|/δ)n+β⋅π⋆+β−1⋅V2e4Rlog(|Π|/δ)n.less-than-or-similar-tosuperscript⋆^⋅subscriptsuperscript2subscriptsuperscriptsuperscript⋆Π⋅superscriptsuperscript⋆⋅superscript1superscriptsubscript2superscript4subscriptΠ J(π )-J( π) V_ maxe^2R_% max· C^π (| |/δ)n% +β·C^π +β^-1· V_ max^2% e^4R_ max (| |/δ)n.J ( π⋆ ) - J ( over start_ARG π end_ARG ) ≲ Vsansserif_max e2 Rsansserif_max ⋅ square-root start_ARG divide start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT log ( | Π | / δ ) end_ARG start_ARG n end_ARG end_ARG + β ⋅ Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT + β- 1 ⋅ divide start_ARG Vsansserif_max2 e4 Rsansserif_max log ( | Π | / δ ) end_ARG start_ARG n end_ARG . (13) In particular, given any comparator policy π⋆superscript⋆π π⋆, we can choose the regularization parameter β to achieve J(π⋆)−J(π^)≲Ve2R⋅π⋆log(|Π|/δ)n.less-than-or-similar-tosuperscript⋆^⋅subscriptsuperscript2subscriptsuperscriptsuperscript⋆Π J(π )-J( π) V_ maxe^2R_% max· C^π (| |/δ)n.J ( π⋆ ) - J ( over start_ARG π end_ARG ) ≲ Vsansserif_max e2 Rsansserif_max ⋅ square-root start_ARG divide start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT log ( | Π | / δ ) end_ARG start_ARG n end_ARG end_ARG . (14) Theorem 3.1 shows that χPO achieves a sample complexity guarantee that scales only with the single-policy concentrability parameter π⋆superscriptsuperscript⋆C^π Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT for the comparator policy π⋆superscript⋆π π⋆, for all policies π⋆superscript⋆π π⋆ simultaneously. In particular, roughly n=O(π⋆log(|Π|/δ)ε2)superscriptsuperscript⋆Πsuperscript2n=O ( C^π ( /δ)% ^2 )n = O ( divide start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT log ( | Π | / δ ) end_ARG start_ARG ε2 end_ARG ) examples are sufficient to learn a policy that is ε ε-suboptimal relative to π⋆superscript⋆π π⋆. As a result, χPO is robust to overoptimization since the learned policy is as good as any π⋆superscript⋆π π⋆ that is sufficiently covered by πsubscript _ refπsansserif_ref (in the sense that π⋆=O(1)superscriptsuperscript⋆1C^π =O(1)Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT = O ( 1 )), which is effectively the best one can hope for in the purely offline setting. In contrast, naive offline alignment methods like DPO have sample complexity that scales with all-policy concentrability (roughly, maxππsubscriptsuperscript _πC^πmaxitalic_π Citalic_π), even when the comparator policy π⋆superscript⋆π π⋆ is sufficiently covered (Zhu et al., 2023; Song et al., 2024). To highlight this, in Fig. 3 (see Section 4 for details) we give a concrete example in which χPO allows the user to tune β to achieve tight statistical rates, yet no choice of β for DPO leads to comparable performance. Effectively, any choice of β for DPO is either susceptible to overoptimization, or unacceptably high biasis unacceptably conservative. All prior works that achieve similar sample complexity guarantees based on single-policy concentrability are either impractical, or require more restrictive statistical assumptions on the policy class (Ye et al., 2024; Liu et al., 2024; Cen et al., 2024; Fisch et al., 2024; Ji et al., 2024).444A notable difference is that some of these works achieve guarantees based on tighter coverage parameters than single-policy L1subscript1L_1L1-concentrability, which reflect the structure of the policy or reward class. This appears to be out of reach for our techniques. Regarding the parameter VsubscriptV_ maxVsansserif_max, we observe that since the policy πβ⋆subscriptsuperscript⋆π _ βπ⋆italic_β satisfies |βϕ(πβ⋆(a∣x)π(a∣x))−βϕ(πβ⋆(b∣x)π(b∣x))|≤2Ritalic-ϕsubscriptsuperscript⋆conditionalsubscriptconditionalitalic-ϕsubscriptsuperscript⋆conditionalsubscriptconditional2subscript βφ ( π _ β(a % x) _ ref(a x) )-βφ ( π _% β(b x) _ ref(b x) )% ≤ 2R_ max| β ϕ ( divide start_ARG π⋆italic_β ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) - β ϕ ( divide start_ARG π⋆italic_β ( b ∣ x ) end_ARG start_ARG πsansserif_ref ( b ∣ x ) end_ARG ) | ≤ 2 Rsansserif_max, information-theoretically we can always achieve V=2Rsubscript2subscriptV_ max=2R_ maxVsansserif_max = 2 Rsansserif_max by pre-filtering the policy class Π Π to remove all policies for which this inequality does not hold. Since this may be non-trivial in practice, we incorporate clipping in Eq. 9, which, for precisely the above reason, we expect to improve performance empirically. In Section 4, we discuss the role of the VsubscriptV_ maxVsansserif_max parameter and Assumption 3.2 in greater depth. See also the guarantees for the χ2superscript2χ^2χ2-RLHF algorithm in Appendix B, which avoid dependence on this parameter. Tuning the regularization parameter To achieve optimal dependence on π⋆superscriptsuperscript⋆C^π Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT, Theorem 3.1 requires tuning β>00β>0β > 0 as a function of this parameter, similar to other pessimistic schemes (Liu et al., 2024). With no prior knowledge, setting β∝V2e4Rlog(|Π|/δ)nproportional-tosuperscriptsubscript2superscript4subscriptΠβ V_ max^2e^4R_ max (| |/% δ)nβ ∝ square-root start_ARG divide start_ARG Vsansserif_max2 e4 Rsansserif_max log ( | Π | / δ ) end_ARG start_ARG n end_ARG end_ARG suffices to ensure that, simultaneously for all comparator policies π⋆superscript⋆π π⋆, we have J(π⋆)−J(π^)≲Ve2R⋅(π⋆)2log(|ℛ|/δ)n,less-than-or-similar-tosuperscript⋆^⋅subscriptsuperscript2subscriptsuperscriptsuperscriptsuperscript⋆2ℛ J(π )-J( π) V_ maxe^2R_% max· (C^π )^2 (|R% |/δ)n,J ( π⋆ ) - J ( over start_ARG π end_ARG ) ≲ Vsansserif_max e2 Rsansserif_max ⋅ square-root start_ARG divide start_ARG ( Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT )2 log ( | R | / δ ) end_ARG start_ARG n end_ARG end_ARG , This guarantee achieves a slightly worse rate than Eq. 14 but holds simultaneously for all comparator policies rather than the specific one that was used to tune β. The following result, specializing to the setting in Example 3.1, shows that there exists an optimal parameter β⋆>0superscript⋆0β >0β⋆ > 0 that recovers the rate in Eq. 14 and holds simultaneously for all comparator policies. Corollary 3.1 (Sample complexity bound for χPO with a reward model). Consider the setting in Example 3.1, where the policy class Πℛ,βsubscriptΠℛ _R,βΠcaligraphic_R , β is the set of mixed χ2superscript2χ^2χ2-regularized policies induced by a reward model class ℛRR with r⋆∈ℛsuperscript⋆ℛr ⋆ ∈ R and β>00β>0β > 0. For any δ∈(0,1)01δ∈(0,1)δ ∈ ( 0 , 1 ), there exists a choice555It is unclear how to select β⋆superscript⋆β β⋆ in a data-driven manner, as it depends on the (unknown) functionals π↦Cπmaps-tosuperscriptπ C^π ↦ Citalic_π and π↦J(π)maps-toπ J(π)π ↦ J ( π ). for β⋆>0superscript⋆0β >0β⋆ > 0 such that with probability at least 1−δ11- 1 - δ, χPO (Algorithm 1), with class Πℛ,β⋆subscriptΠℛsuperscript⋆ _R,β Πcaligraphic_R , β⋆, produces a policy π^ πover start_ARG π end_ARG such that for all policies π⋆superscript⋆π π⋆ simultaneously, we have J(π⋆)−J(π^)≲Re2R⋅π⋆log(|ℛ|/δ)n.less-than-or-similar-tosuperscript⋆^⋅subscriptsuperscript2subscriptsuperscriptsuperscript⋆ℛ J(π )-J( π) R_ maxe^2R_% max· C^π (|R|/% δ)n.J ( π⋆ ) - J ( over start_ARG π end_ARG ) ≲ Rsansserif_max e2 Rsansserif_max ⋅ square-root start_ARG divide start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT log ( | R | / δ ) end_ARG start_ARG n end_ARG end_ARG . Additional remarks Specializing to the case of multi-armed bandits, we believe that the sample complexity bound in Eq. 13 is optimal in general (Rashidinejad et al., 2021). Note that while we consider finite classes Π Π in Theorem 3.1 for simplicity, extension to infinite classes is trivial via standard uniform convergence arguments. We remark that the exponential dependence on RsubscriptR_ maxRsansserif_max in Theorem 3.1 is an intrinsic feature of the Bradley-Terry model, and can be found in all prior work (Rosset et al., 2024; Xie et al., 2024). Finally, we remark that the weight on the KL term in the mixed χ2superscript2χ^2χ2-regularized objective is not important for our statistical guarantees. For any γ∈(0,1]01γ∈(0,1]γ ∈ ( 0 , 1 ], we can replace the link function ϕ(⋅)italic-ϕ⋅φ(·)ϕ ( ⋅ ) in χPO with ϕγ(z)=z+γlogzsubscriptitalic-ϕ _γ(z)=z+γ zϕitalic_γ ( z ) = z + γ log z, ϕγ(z)=z+γlogz,subscriptitalic-ϕ _γ(z)=z+γ z,ϕitalic_γ ( z ) = z + γ log z , which corresponds to the regularized objective Jβ,γχmix(π)=π[r⋆(x,a)]−β⋅Dχ2(π∥π)−γβ⋅D(π∥π)superscriptsubscriptsubscriptmixsubscriptsuperscript⋆⋅subscriptsuperscript2conditionalsubscript⋅subscriptconditionalsubscriptJ_β,γ _ mix(π)= E_π% [r (x,a) ]-β· D_χ^2 (π\;\|\; _% ref )-γβ· D_ KL (π\,\|\, _% ref )Jitalic_β , γitalic_χmix ( π ) = blackboard_Eπ [ r⋆ ( x , a ) ] - β ⋅ Ditalic_χ2 ( π ∥ πsansserif_ref ) - γ β ⋅ Dsansserif_KL ( π ∥ πsansserif_ref ). This leads to identical guarantees for any γ∈(0,1]01γ∈(0,1]γ ∈ ( 0 , 1 ] (Theorem F.1 in Appendix F); essentially, we only require that γ is positive to ensure the reparameterization in Eq. 10 is admissible. 4 Understanding χitalic_χPO: The Bias-Overoptimization Tradeoff Having derived χPO from the mixed χ2superscript2χ^2χ2-regularized RLHF objective and analyzed its performance, we now take a moment to better understand the statistical properties of the policies the algorithm learns. We focus on the tradeoff between overoptimization and bias (i.e., underoptimization) achieved by the regularization parameter β>00β>0β > 0, highlighting through examples how this leads to statistical benefits over naive alignment methods like DPO. 4.1 Properties of Optimal Policy under Mixed superscript2 χ^2italic_χbold_2-Regularization We begin by deriving a (nearly) closed form solution for the optimal mixed χ2superscript2χ^2χ2-regularized policy in Eq. 11; recall that we expect χPO to converge to this policy in the limit of infinite data. We first observe that the link function ϕ(⋅)italic-ϕ⋅φ(·)ϕ ( ⋅ ) is strictly increasing over ℝ+subscriptℝR_+blackboard_R+, and its inverse is given by ϕ−1(z)=W0(exp(z))superscriptitalic-ϕ1subscript0φ^-1(z)=W_0( (z))ϕ- 1 ( z ) = W0 ( exp ( z ) ); here, W0(y)subscript0W_0(y)W0 ( y ) denotes the Lambert W-function (Corless et al., 1996), defined for y≥−e−1superscript1y≥-e^-1y ≥ - e- 1 as the inverse of the function x↦xexmaps-tosuperscriptx xe^xx ↦ x eitalic_x. Consequently, for any x, the optimal policy under mixed χ2superscript2χ^2χ2-regularization satisfies πβ⋆(a∣x)=π(a∣x)⋅W0(exp(β−1(r⋆(x,a)−Zβ,r⋆(x)))),subscriptsuperscript⋆conditional⋅subscriptconditionalsubscript0superscript1superscript⋆subscriptsuperscript⋆ π _ β(a x)= _ % ref(a x)·W_0 ( (β^-1(r (x,a)-Z_β,% r (x)) ) ),π⋆italic_β ( a ∣ x ) = πsansserif_ref ( a ∣ x ) ⋅ W0 ( exp ( β- 1 ( r⋆ ( x , a ) - Zitalic_β , r⋆ ( x ) ) ) ) , where Zβ,r⋆(x)subscriptsuperscript⋆Z_β,r (x)Zitalic_β , r⋆ ( x ) is chosen such that ∑aπβ⋆(a∣x)=1subscriptsubscriptsuperscript⋆conditional1 _aπ _ β(a x)=1∑a π⋆italic_β ( a ∣ x ) = 1. We can better understand how this policy behaves using the following simple upper and lower bounds on the inverse link function ϕ−1(z)=W0(exp(z))superscriptitalic-ϕ1subscript0φ^-1(z)=W_0( (z))ϕ- 1 ( z ) = W0 ( exp ( z ) ). Proposition 4.1. The link function ϕ(z)=z+logzitalic-ϕφ(z)=z+ zϕ ( z ) = z + log z is strictly increasing over (0,∞)0(0,∞)( 0 , ∞ ), and its inverse ϕ−1(z)=W0(exp(z))superscriptitalic-ϕ1subscript0φ^-1(z)=W_0( (z))ϕ- 1 ( z ) = W0 ( exp ( z ) ) is strictly increasing over (−∞,∞)(-∞,∞)( - ∞ , ∞ ). The inverse link function ϕ−1superscriptitalic-ϕ1φ^-1ϕ- 1 satisfies z2≤ϕ−1(z)≤z∀z∈[1,∞),andez−e≤ϕ−1(z)≤ez∀z∈(−∞,1].formulae-sequence2superscriptitalic-ϕ1formulae-sequencefor-all1andsuperscriptsuperscriptitalic-ϕ1superscriptfor-all1 z2≤φ^-1(z)≤z ∀z∈[1,∞),% e^z-e≤φ^-1(z)≤e^z ∀z∈% (-∞,1].divide start_ARG z end_ARG start_ARG 2 end_ARG ≤ ϕ- 1 ( z ) ≤ z ∀ z ∈ [ 1 , ∞ ) , and eitalic_z - e ≤ ϕ- 1 ( z ) ≤ eitalic_z ∀ z ∈ ( - ∞ , 1 ] . Compared to KL-regularization, which leads to softmax policies that satisfy πβ;⋆(a∣x)=π(a∣x)⋅exp(β−1(r⋆(x,a)−Zβ,r⋆;(x)))subscriptsuperscript⋆conditional⋅subscriptconditionalsuperscript1superscript⋆subscriptsuperscript⋆π _β; KL(a x)= _ ref(a x)·% (β^-1(r (x,a)-Z_β,r ; KL(x)) )π⋆italic_β ; sansserif_KL ( a ∣ x ) = πsansserif_ref ( a ∣ x ) ⋅ exp ( β- 1 ( r⋆ ( x , a ) - Zitalic_β , r⋆ ; sansserif_KL ( x ) ) ), we see that the inverse link function ϕ−1(z)=W0(exp(z))superscriptitalic-ϕ1subscript0φ^-1(z)=W_0( (z))ϕ- 1 ( z ) = W0 ( exp ( z ) ) for mixed χ2superscript2χ^2χ2-regularization satisfies ϕ−1(z)≈zsuperscriptitalic-ϕ1φ^-1(z)≈zϕ- 1 ( z ) ≈ z for z≥11z≥1z ≥ 1, leading to a more heavy-tailed action distribution for πβ⋆subscriptsuperscript⋆π _ βπ⋆italic_β. On the other hand, for z≤11z≤1z ≤ 1 the inverse link behaves like the exponential function (i.e., ϕ−1(z)≈ezsuperscriptitalic-ϕ1superscriptφ^-1(z)≈e^zϕ- 1 ( z ) ≈ eitalic_z for z≤11z≤1z ≤ 1); see Fig. 1 for an illustration. Using these properties, we can derive the following upper and lower bounds on the density ratio between πβ⋆subscriptsuperscript⋆π _ βπ⋆italic_β and πsubscript _ refπsansserif_ref. Proposition 4.2. For all x∈x ∈ X and a∈a ∈ A, the optimal policy πβ⋆subscriptsuperscript⋆π _ βπ⋆italic_β under mixed χ2superscript2χ^2χ2-regularization satisfies exp(−Rβ)≲πβ⋆(a∣x)π(a∣x)≲1+Rβ.less-than-or-similar-tosubscriptsubscriptsuperscript⋆conditionalsubscriptconditionalless-than-or-similar-to1subscript (- R_ maxβ ) π% _ β(a x) _ ref(a x)% 1+ R_ maxβ.exp ( - divide start_ARG Rsansserif_max end_ARG start_ARG β end_ARG ) ≲ divide start_ARG π⋆italic_β ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ≲ 1 + divide start_ARG Rsansserif_max end_ARG start_ARG β end_ARG . (15) Both inequalities are tight in general (up to absolute constants). The upper bound in Eq. 15, which arises from the χ2superscript2χ^2χ2 term in the mixed-χ2superscript2χ^2χ2 objective, scales inversely with the regularization parameter β, and reflects the heavy-tailed, pessimistic behavior this regularizer induces; in contrast, the optimal policy under pure KL-regularization only satisfies exp(−Rβ)≲πβ;⋆(a∣x)π(a∣x)≲exp(Rβ)less-than-or-similar-tosubscriptsubscriptsuperscript⋆conditionalsubscriptconditionalless-than-or-similar-tosubscript (- R_ maxβ ) π% _β; KL(a x) _ ref(a x) % ( R_ maxβ )exp ( - divide start_ARG Rsansserif_max end_ARG start_ARG β end_ARG ) ≲ divide start_ARG π⋆italic_β ; sansserif_KL ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ≲ exp ( divide start_ARG Rsansserif_max end_ARG start_ARG β end_ARG ) (16) in general. The lower bound in Eq. 15 arises from the KL term in the mixed-χ2superscript2χ^2χ2 objective, but is not important for our analysis (outside of allowing for DPO-like reparameterization). Figure 1: Behavior of the mixed χ2superscript2χ^2χ2-regularization link function ϕχPO(z)=z+logzsubscriptitalic-ϕχPO _ $χ$PO(z)=z+ zϕitalic_χ PO ( z ) = z + log z and inverse ϕχPO−1(z)=W0(exp(z))superscriptsubscriptitalic-ϕχPO1subscript0 _ $χ$PO^-1(z)=W_0( (z))ϕitalic_χ PO- 1 ( z ) = W0 ( exp ( z ) ), compared to the KL-regularization link function ϕDPO(z)=logzsubscriptitalic-ϕDPO _ DPO(z)= zϕDPO ( z ) = log z and inverse ϕDPO−1(z)=exp(z)superscriptsubscriptitalic-ϕDPO1 _ DPO^-1(z)= (z)ϕDPO- 1 ( z ) = exp ( z ). ϕχPO−1(z)≈zsuperscriptsubscriptitalic-ϕχPO1 _ $χ$PO^-1(z)≈ zϕitalic_χ PO- 1 ( z ) ≈ z for z≥11z≥ 1z ≥ 1, leading to favorable heavy-tailed, pessimistic behavior. 4.2 The Bias-Overoptimization Tradeoff We are now well equipped to understand how χPO modulates the tradeoff between overoptimization and bias using the regularization parameter β, and how this tradeoff compares to vanilla DPO. To showcase this, we take a reward modeling perspective, and consider the setting in which the policy class Π Π is induced by a given reward model class ℛRR, similar to Example 3.1. Suppose we start with a reward model class ℛ⊂(×→[0,R])ℛ→0subscriptR⊂(X×A→[0,R_ max])R ⊂ ( X × A → [ 0 , Rsansserif_max ] ) such that r⋆∈ℛsuperscript⋆ℛr ⋆ ∈ R. If we use the induced policy class ΠDPO,β:=π(a∣x)=π(a∣x)⋅exp(β−1(r(x,a)−Zβ,r;(x)))∣r∈ℛ,assignsubscriptΠDPOconditionalconditional⋅subscriptconditionalsuperscript1subscriptℛ _ DPO,β = \π(a x)= _% ref(a x)· (β^-1(r(x,a)-Z_β,r; KL(x))% ) r \,ΠDPO , β := π ( a ∣ x ) = πsansserif_ref ( a ∣ x ) ⋅ exp ( β- 1 ( r ( x , a ) - Zitalic_β , r ; sansserif_KL ( x ) ) ) ∣ r ∈ R , (17) then DPO can be interpreted as fitting a reward model r^ rover start_ARG r end_ARG using maximum likelihood (Eq. 3) and then outputting the policy π^DPO(a∣x)=π(a∣x)⋅exp(β−1(r^(x,a)−Zβ,r^;(x)))subscript^DPOconditional⋅subscriptconditionalsuperscript1^subscript π_ DPO(a x)= _ ref(a x)· (% β^-1( r(x,a)-Z_β, r; KL(x)))over start_ARG π end_ARGDPO ( a ∣ x ) = πsansserif_ref ( a ∣ x ) ⋅ exp ( β- 1 ( over start_ARG r end_ARG ( x , a ) - Zitalic_β , over start_ARG r end_ARG ; sansserif_KL ( x ) ) ). Meanwhile, if we use the induced policy class ΠχPO,β:=π(a∣x)=π(a∣x)⋅ϕ−1(β−1(r(x,a)−Zβ,r(x)))∣r∈ℛ,assignsubscriptΠχPOconditionalconditional⋅subscriptconditionalsuperscriptitalic-ϕ1superscript1subscriptℛ _ $χ$PO,β = \π(a x)% = _ ref(a x)·φ^-1(β^-1(r(x,a)-Z_β,r(x)))% r \,Πitalic_χ typewriter_PO , β := π ( a ∣ x ) = πsansserif_ref ( a ∣ x ) ⋅ ϕ- 1 ( β- 1 ( r ( x , a ) - Zitalic_β , r ( x ) ) ) ∣ r ∈ R , (18) then χPO can be interpreted as fitting a reward model r^ rover start_ARG r end_ARG with the exact same maximum likelihood objective, but instead outputting the policy π^χPO(a∣x)=π(a∣x)⋅ϕ−1(β−1(r^(x,a)−Zβ,r^(x)))subscript^χPOconditional⋅subscriptconditionalsuperscriptitalic-ϕ1superscript1^subscript π_ $χ$PO(a x)= _ ref(a x)% ·φ^-1(β^-1( r(x,a)-Z_β, r(x)))over start_ARG π end_ARGχ typewriter_PO ( a ∣ x ) = πsansserif_ref ( a ∣ x ) ⋅ ϕ- 1 ( β- 1 ( over start_ARG r end_ARG ( x , a ) - Zitalic_β , over start_ARG r end_ARG ( x ) ) ). The policies π^χPOsubscript^χPO π_ $χ$POover start_ARG π end_ARGχ typewriter_PO and π^DPOsubscript^DPO π_ DPOover start_ARG π end_ARGDPO are induced by the same reward model r^ rover start_ARG r end_ARG, and both use the parameter β to balance bias and overoptimization. For both policies, large β means the policy avoids overfitting to errors in the reward model (the extreme case is β→∞→β→∞β → ∞, in which case both policies become πsubscript _ refπsansserif_ref), while small β means the policy has low bias, i.e., low error in the case where the model is correct in the sense that r^=r⋆^superscript⋆ r=r over start_ARG r end_ARG = r⋆ (the extreme case is β→0→0β→0β → 0, in which case both policies become x↦argmaxa:π(a∣x)>0r^(x,a)maps-tosubscriptargmax:subscriptconditional0^x argmax_a: _ ref(a x)>0 r% (x,a)x ↦ argmaxitalic_a : π start_POSTSUBSCRIPT sansserif_ref ( a ∣ x ) > 0 end_POSTSUBSCRIPT over start_ARG r end_ARG ( x , a )). Yet, for the same choice of β, π^χPOsubscript^χPO π_ $χ$POover start_ARG π end_ARGχ typewriter_PO is significantly more heavy-tailed than π^DPOsubscript^DPO π_ DPOover start_ARG π end_ARGDPO, a consequence of the pessimism induced by χ2superscript2χ^2χ2-regularization; see Fig. 2, which plots the action distribution for both policies as a function of β. Figure 2: Action probabilities for policies learned by χPO and DPO on the example from Section 4.3, under the “bad” event ℰEE in which the true reward model is r⋆=r1superscript⋆subscript1r =r_1r⋆ = r1 but the estimated reward model is r^=r2^subscript2 r=r_2over start_ARG r end_ARG = r2 (n=1010n=10n = 10). Here, r⋆(a)=1superscript⋆subscript1r (a_ good)=1r⋆ ( asansserif_good ) = 1 and r⋆(a)=0superscript⋆subscript0r (a_ bad)=0r⋆ ( asansserif_bad ) = 0, but r^(a)=0^subscript0 r(a_ good)=0over start_ARG r end_ARG ( asansserif_good ) = 0 and r^(a)=1^subscript1 r(a_ good)=1over start_ARG r end_ARG ( asansserif_good ) = 1; both reward functions have r⋆(a0)=r^(a0)=1/2superscript⋆subscript0^subscript012r (a_0)= r(a_0)=1/2r⋆ ( a0 ) = over start_ARG r end_ARG ( a0 ) = 1 / 2, and the goal is to compete with a comparator policy that deterministically plays a0subscript0a_0a0. Overoptimization. The DPO policy is greedier with respect to the incorrect reward model and places much larger mass on the bad action asubscripta_ badasansserif_bad for all β∈(0,12logn]012β∈(0, 12 n]β ∈ ( 0 , divide start_ARG 1 end_ARG start_ARG 2 log n end_ARG ] (Right). As a result, the DPO policy places much smaller mass on the baseline action a0subscript0a_0a0, suffering significantly more overoptimization error compared to χPO (Left; see also Fig. 3). Bias. Compared to DPO, χPO has a higher probability of taking both the optimal action asubscripta_ goodasansserif_good and the reference action a0subscript0a_0a0. As a result, it strikes a better bias-overoptimization tradeoff than DPO, and is competitive with respect to the comparator a0subscript0a_0a0 even when DPO fails to converge. 4.3 An Illustrative Example We now give a concrete example in which χPO allows the user to tune β to achieve tight statistical rates, yet no choice of β for DPO leads to comparable performance (effectively, any choice of β is either susceptible to overoptimization, or has unacceptably high bias). This illustrates the favorable tradeoff between bias and overoptimization achieved by χPO. Let n∈ℕn ∈ blackboard_N with n≥22n≥2n ≥ 2 be given. We consider a problem instance with =∅X=\ \X = ∅ and =a0,a1,a2,a3subscript0subscript1subscript2subscript3A= \a_0,a_1,a_2,a_3 \A = a0 , a1 , a2 , a3 . We define πsubscript _ refπsansserif_ref via π(a0)=12,π(a1)=π(a2)=12n,andπ(a3)=n−22n.formulae-sequenceformulae-sequencesubscriptsubscript012subscriptsubscript1subscriptsubscript212andsubscriptsubscript322 _ ref(a_0)= 12, _ ref(a_% 1)= _ ref(a_2)= 12n, _ % ref(a_3)= n-22n.πsansserif_ref ( a0 ) = divide start_ARG 1 end_ARG start_ARG 2 end_ARG , πsansserif_ref ( a1 ) = πsansserif_ref ( a2 ) = divide start_ARG 1 end_ARG start_ARG 2 n end_ARG , and πsansserif_ref ( a3 ) = divide start_ARG n - 2 end_ARG start_ARG 2 n end_ARG . We define a reward class with two reward functions ℛ:=r1,r2assignℛsubscript1subscript2R:=\r_1,r_2\R := r1 , r2 as follows. For i∈1,212i∈\1,2\i ∈ 1 , 2 : ri(a0)=1/2,ri(ai)=1,ri(aj)=0,∀j≠i.formulae-sequencesubscriptsubscript012formulae-sequencesubscriptsubscript1formulae-sequencesubscriptsubscript0for-all r_i(a_0)=1/2, r_i(a_i)=1, r_i(a_j)=0,\;\;% ∀ j≠ i.ritalic_i ( a0 ) = 1 / 2 , ritalic_i ( aitalic_i ) = 1 , ritalic_i ( aitalic_j ) = 0 , ∀ j ≠ i . Let β>00β>0β > 0 be fixed. To compare χPO and DPO, we consider their behavior when invoked with the induced policy classes ΠχPO,βsubscriptΠχPO _ $χ$PO,βΠitalic_χ typewriter_PO , β and ΠDPO,βsubscriptΠDPO _ DPO,βΠDPO , β defined above. Recall that with this choice, the two algorithms can be interpreted as fitting a reward model r^ rover start_ARG r end_ARG using maximum likelihood (Eq. 3) and returning the policies π^χPO(a∣x)=π(a∣x)⋅ϕ−1(β−1(r^(x,a)−Zβ,r^(x)))subscript^χPOconditional⋅subscriptconditionalsuperscriptitalic-ϕ1superscript1^subscript π_ $χ$PO(a x)= _ ref(a x)% ·φ^-1(β^-1( r(x,a)-Z_β, r(x)))over start_ARG π end_ARGχ typewriter_PO ( a ∣ x ) = πsansserif_ref ( a ∣ x ) ⋅ ϕ- 1 ( β- 1 ( over start_ARG r end_ARG ( x , a ) - Zitalic_β , over start_ARG r end_ARG ( x ) ) ) and π^DPO(a∣x)=π(a∣x)⋅exp(β−1(r^(x,a)−Zβ,r^;(x)))subscript^DPOconditional⋅subscriptconditionalsuperscript1^subscript π_ DPO(a x)= _ ref(a x)· (% β^-1( r(x,a)-Z_β, r; KL(x)))over start_ARG π end_ARGDPO ( a ∣ x ) = πsansserif_ref ( a ∣ x ) ⋅ exp ( β- 1 ( over start_ARG r end_ARG ( x , a ) - Zitalic_β , over start_ARG r end_ARG ; sansserif_KL ( x ) ) ), respectively. Suppose that r1subscript1r_1r1 is the true reward function. It is hopeless (information-theoretically) to compete with the unconstrained optimal action a1subscript1a_1a1, as we are in a sample-starved regime where a1=2nsuperscriptsubscript12C^a_1=2nCitalic_a1 = 2 n (in the language of Eq. 13). Indeed, one can show (see proof of Proposition A.1 in Appendix A) that with constant probability, none of the examples in the offline dataset subscriptD_ prefDsansserif_pref contain actions a1subscript1a_1a1 or a2subscript2a_2a2. Under this event, which we denote by ℰEE, the value for the maximum likelihood objective in Eq. 3 is identical for r1subscript1r_1r1 and r2subscript2r_2r2, so we may obtain r^=r2^subscript2 r=r_2over start_ARG r end_ARG = r2 (due to adversarial tie-breaking). However, in spite of the fact that the policies π^χPOsubscript^χPO π_ $χ$POover start_ARG π end_ARGχ typewriter_PO and π^DPOsubscript^DPO π_ DPOover start_ARG π end_ARGDPO are induced by the same (incorrect) reward function r^=r2^subscript2 r=r_2over start_ARG r end_ARG = r2, they produce very different action distributions, as highlighted in Fig. 2. Figure 3: The regret J(a0)−J(π^)subscript0^J(a_0)-J( π)J ( a0 ) - J ( over start_ARG π end_ARG ) in the construction from Proposition A.1 for different values of n. We again condition on the “bad” event ℰEE where r^=r2≠r⋆^subscript2superscript⋆ r=r_2≠ r over start_ARG r end_ARG = r2 ≠ r⋆. For each n, the error from overoptimization dominates when β≤(2logn)−1superscript21β≤(2 n)^-1β ≤ ( 2 log n )- 1 (as discussed in Section 4.3), and the error from bias dominates when β>(2logn)−1superscript21β>(2 n)^-1β > ( 2 log n )- 1. Taking the best choice of β for each method, DPO converges at an exponentially slower rate than χPO. To understand this, note that even in the sample-starved regime, we can still hope to compete with the “baseline” action a0subscript0a_0a0; Fig. 3 shows that χPO has low regret against this action, while DPO has high regret. In particular, since a0=2superscriptsubscript02C^a_0=2Citalic_a0 = 2, Theorem 3.1 (Eq. 13) implies that χPO achieves J(a0)−J(π^χPO)≲1n+β+β−11n,less-than-or-similar-tosubscript0subscript^χPO1superscript11J(a_0)-J( π_ $χ$PO) 1n+% β+β^-1 1n,J ( a0 ) - J ( over start_ARG π end_ARGχ typewriter_PO ) ≲ square-root start_ARG divide start_ARG 1 end_ARG start_ARG n end_ARG end_ARG + β + β- 1 divide start_ARG 1 end_ARG start_ARG n end_ARG , and setting β∝1nproportional-to1β 1nβ ∝ square-root start_ARG divide start_ARG 1 end_ARG start_ARG n end_ARG end_ARG leads to J(a0)−J(π^χPO)≲1nless-than-or-similar-tosubscript0subscript^χPO1J(a_0)-J( π_ $χ$PO) 1nJ ( a0 ) - J ( over start_ARG π end_ARGχ typewriter_PO ) ≲ square-root start_ARG divide start_ARG 1 end_ARG start_ARG n end_ARG end_ARG. This is a consequence of the pessimistic, heavy-tailed nature of π^χPOsubscript^χPO π_ $χ$POover start_ARG π end_ARGχ typewriter_PO (cf. Proposition 4.2), which places no more than β−1/nsuperscript1β^-1/nβ- 1 / n probability mass on the (incorrect) greedy action a2subscript2a_2a2 for r^=r2^subscript2 r=r_2over start_ARG r end_ARG = r2, thereby correctly capturing the inherent uncertainty in the reward for this action. On the other hand, it is straightforward to show that for all possible values β≤(2logn)−1superscript21β≤(2 n)^-1β ≤ ( 2 log n )- 1, the DPO policy π^DPOsubscript^DPO π_ DPOover start_ARG π end_ARGDPO has regret J(a0)−J(π^DPO)≥12(1−11+1ne12+(1−1n)e−12β)−12n≥Ω(1)subscript0subscript^DPO121111superscript1211superscript1212Ω1J(a_0)-J( π_ DPO)≥ 12 (1- 11+% 1ne 12+(1- 1n)e^- 12β )- % 12n≥ (1)J ( a0 ) - J ( over start_ARG π end_ARGDPO ) ≥ divide start_ARG 1 end_ARG start_ARG 2 end_ARG ( 1 - divide start_ARG 1 end_ARG start_ARG 1 + divide start_ARG 1 end_ARG start_ARG n end_ARG edivide start_ARG 1 end_ARG start_ARG 2 end_ARG + ( 1 - divide start_ARG 1 end_ARG start_ARG n end_ARG ) e- divide start_ARG 1 end_ARG start_ARG 2 β end_ARG end_ARG ) - divide start_ARG 1 end_ARG start_ARG 2 n end_ARG ≥ Ω ( 1 ) whenever n≥22n≥2n ≥ 2. This is because when β≤(2logn)−1superscript21β≤(2 n)^-1β ≤ ( 2 log n )- 1, π^DPOsubscript^DPO π_ DPOover start_ARG π end_ARGDPO assigns excessively high probability to the incorrect greedy action a2subscript2a_2a2, an instance of overoptimization. Meanwhile, larger choices for β lead to excessively large bias in general (see Section A.1 for a more sophisticated construction which extends this lower bound to all possible β). In other words, as illustrated in Fig. 3, no choice of β gives a favorable tradeoff between overoptimization and bias. To summarize, for DPO, large values of β are required to avoid overfitting to the reward function, incurring high bias. Meanwhile, χPO avoids overoptimization using comparatively small values for β, yet has bias no worse than that of DPO, thereby striking a better tradeoff. We mention that the “DPO+SFT” algorithm of Liu et al. (2024); Cen et al. (2024); Fisch et al. (2024) also fails on the construction above; see Proposition A.1 in Section A.1 for details. Remark 4.1 (DPO decreases probabilities of preferred and rejected responses). Various recent works have noted an empirical phenomenon in which DPO decreases the probabilities for both preferred and rejected responses throughout training (Yuan et al., 2024; Pal et al., 2024; Rafailov et al., 2024b). Interestingly, we observe that the example above exhibits this phenomenon. Notably, if β<(2logn)−1superscript21β<(2 n)^-1β < ( 2 log n )- 1, then under the event ℰEE in which the offline dataset subscriptD_ prefDsansserif_pref does not contain the actions a1subscript1a_1a1 or a2subscript2a_2a2 (so that r^=r2^subscript2 r=r_2over start_ARG r end_ARG = r2), we observe that π^DPO(a0)=12e12β12e12β+12ne1β+n−12n<12=π(a0)subscript^DPOsubscript012superscript1212superscript1212superscript11212subscriptsubscript0 π_ DPO(a_0)= 12e 12β% 12e 12β+ 12ne 1β+ n-12n% < 12= _ ref(a_0)over start_ARG π end_ARGDPO ( a0 ) = divide start_ARG divide start_ARG 1 end_ARG start_ARG 2 end_ARG edivide start_ARG 1 end_ARG start_ARG 2 β end_ARG end_ARG start_ARG divide start_ARG 1 end_ARG start_ARG 2 end_ARG edivide start_ARG 1 end_ARG start_ARG 2 β end_ARG + divide start_ARG 1 end_ARG start_ARG 2 n end_ARG edivide start_ARG 1 end_ARG start_ARG β end_ARG + divide start_ARG n - 1 end_ARG start_ARG 2 n end_ARG end_ARG < divide start_ARG 1 end_ARG start_ARG 2 end_ARG = πsansserif_ref ( a0 ), and for all i>22i>2i > 2, π^DPO(ai)=12n12e12β+12ne1β+n−12n<12n=π(ai)subscript^DPOsubscript1212superscript1212superscript11212subscriptsubscript π_ DPO(a_i)= 12n 12e 1% 2β+ 12ne 1β+ n-12n< 12n= _% ref(a_i)over start_ARG π end_ARGDPO ( aitalic_i ) = divide start_ARG divide start_ARG 1 end_ARG start_ARG 2 n end_ARG end_ARG start_ARG divide start_ARG 1 end_ARG start_ARG 2 end_ARG edivide start_ARG 1 end_ARG start_ARG 2 β end_ARG + divide start_ARG 1 end_ARG start_ARG 2 n end_ARG edivide start_ARG 1 end_ARG start_ARG β end_ARG + divide start_ARG n - 1 end_ARG start_ARG 2 n end_ARG end_ARG < divide start_ARG 1 end_ARG start_ARG 2 n end_ARG = πsansserif_ref ( aitalic_i ). We conclude that for all a∈subscripta _ prefa ∈ Dsansserif_pref, π^DPO(a)<π(a).subscript^DPOsubscript π_ DPO(a)< _ ref(a).over start_ARG π end_ARGDPO ( a ) < πsansserif_ref ( a ) . We emphasize that this behavior arises due to the use of function approximation. When the reward class ℛRR (equivalently, the policy class ΠDPO,βsubscriptΠDPO _ DPO,βΠDPO , β) is restricted, the algorithm can aggressively (and incorrectly) extrapolate rewards for actions outside the dataset and, in doing so, inadvertently decrease the probabilities for preferred responses in the dataset. Meanwhile, in the same parameter range, χPO satisfies (see Fig. 2) π^χPO(a0)>π(a0),subscript^χPOsubscript0subscriptsubscript0 π_ $χ$PO(a_0)> _ ref(a_0),over start_ARG π end_ARGχ typewriter_PO ( a0 ) > πsansserif_ref ( a0 ) , highlighting that pessimism can mitigate this phenomenon. 4.4 Nontriviality and Role of VsubscriptV_ maxVsansserif_max Parameter To close this section, we discuss the role of the VsubscriptV_ maxVsansserif_max parameter (Assumption 3.2) used in the analysis of χPO (Theorem 3.1) in depth, motivating it from the perspective of the induced policy class ΠχPO,βsubscriptΠχPO _ $χ$PO,βΠitalic_χ typewriter_PO , β from Section 4.2. Assumption 3.2 effectively implies that all policies π∈Ππ∈ π ∈ Π satisfy ‖π‖∞≲Vβless-than-or-similar-tosubscriptnormsubscriptsubscript \| π _ ref \|_∞ V_% maxβ∥ divide start_ARG π end_ARG start_ARG πsansserif_ref end_ARG ∥∞ ≲ divide start_ARG Vsansserif_max end_ARG start_ARG β end_ARG; in other words, the policy class we use in χPO satisfies all-policy L∞subscriptL_∞L∞-concentrability with maxπ∈Π∞π≲Vβless-than-or-similar-tosubscriptΠsubscriptsuperscriptsubscript _π∈ C^π_∞ V_ maxβmaxitalic_π ∈ Π Citalic_π∞ ≲ divide start_ARG Vsansserif_max end_ARG start_ARG β end_ARG. At first glance, this might seem to trivialize the offline alignment problem, since it would suffice to prove a generalization guarantee based on all-policy concentrability, and then plug this bound in. We will show that this is not the case, and that this is actually an intrinsic feature of χ2superscript2χ^2χ2-regularization. In more detail, recall that for χPO, we require the realizability assumption that πβ⋆∈Πsubscriptsuperscript⋆Ππ _ β∈ π⋆italic_β ∈ Π (Assumption 3.1), where πβ⋆subscriptsuperscript⋆π _ βπ⋆italic_β is the optimal mixed χ2superscript2χ^2χ2-regularized policy that satisfies r⋆(x,a)=βϕ(πβ⋆(a∣x)π(a∣x))+Zβ,r⋆(x)superscript⋆italic-ϕsubscriptsuperscript⋆conditionalsubscriptconditionalsubscriptsuperscript⋆r (x,a)=βφ ( π _ β(a% x) _ ref(a x) )+Z_β,r (x)r⋆ ( x , a ) = β ϕ ( divide start_ARG π⋆italic_β ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) + Zitalic_β , r⋆ ( x ). This policy, via Proposition 4.2, satisfies ‖πβ⋆π‖∞≲Rβless-than-or-similar-tosubscriptnormsubscriptsuperscript⋆subscriptsubscript \| π _ β _ ref % \|_∞ R_ maxβ∥ divide start_ARG π⋆italic_β end_ARG start_ARG πsansserif_ref end_ARG ∥∞ ≲ divide start_ARG Rsansserif_max end_ARG start_ARG β end_ARG, so from a statistical perspective, we can take Assumption 3.2 to hold without loss of generality by removing any policy that violates this bound. In addition, as highlighted by Example 3.1, if we begin from a class of bounded reward models ℛRR with r⋆∈ℛsuperscript⋆ℛr ⋆ ∈ R, Assumption 3.2 holds with V≲Rless-than-or-similar-tosubscriptsubscriptV_ max R_ maxVsansserif_max ≲ Rsansserif_max for the induced class ΠχPO,βsubscriptΠχPO _ $χ$PO,βΠitalic_χ typewriter_PO , β defined in Eq. 18, even though knowledge of such a reward model class is a mild statistical assumption that clearly does not trivialize the learning problem. On the other hand, for DPO, a minimal assumption is that πβ;⋆∈Πsubscriptsuperscript⋆Ππ _β; KL∈ π⋆italic_β ; sansserif_KL ∈ Π (Xie et al., 2024), where πβ;⋆subscriptsuperscript⋆π _β; KLπ⋆italic_β ; sansserif_KL is the optimal KL-regularized policy that satisfies r⋆(x,a)=βlogπβ;⋆(a∣x)π(a∣x)+Zβ,r⋆;(x)superscript⋆subscriptsuperscript⋆conditionalsubscriptconditionalsubscriptsuperscript⋆r (x,a)=β π _β; KL(a x) _% ref(a x)+Z_β,r ; KL(x)r⋆ ( x , a ) = β log divide start_ARG π⋆italic_β ; sansserif_KL ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG + Zitalic_β , r⋆ ; sansserif_KL ( x ). Unlike the optimal mixed χ2superscript2χ^2χ2-regularized policy, πβ;⋆subscriptsuperscript⋆π _β; KLπ⋆italic_β ; sansserif_KL has πβ;⋆(a∣x)π(a∣x)≳exp(Rβ)greater-than-or-equivalent-tosubscriptsuperscript⋆conditionalsubscriptconditionalsubscript π _β; KL(a x) _ ref(a x)% ( R_ maxβ )divide start_ARG π⋆italic_β ; sansserif_KL ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ≳ exp ( divide start_ARG Rsansserif_max end_ARG start_ARG β end_ARG ). This means that it is impossible to find a policy class that simultaneously (1) realizes πβ;⋆subscriptsuperscript⋆π _β; KLπ⋆italic_β ; sansserif_KL, and (2) satisfies all-policy concentrability with maxπ∈Π∞π≪exp(Rβ)much-less-thansubscriptΠsubscriptsuperscriptsubscript _π∈ C^π_∞ ( R_ max% β )maxitalic_π ∈ Π Citalic_π∞ ≪ exp ( divide start_ARG Rsansserif_max end_ARG start_ARG β end_ARG ). As the bias of DPO is unacceptably large unless β=poly(1/n)poly1β=poly(1/n)β = poly ( 1 / n ) (the “small-β” regime), this leads to vacuous guarantees. In view of these observations, our analysis of χPO can be interpreted as (implicitly) showing that for any bounded reward class ℛRR, there exists a policy class Π Π (precisely, the class ΠχPO,βsubscriptΠχPO _ $χ$PO,βΠitalic_χ typewriter_PO , β defined in Eq. 18) such that the following properties hold: 1. Bounded bias. For every r∈ℛr ∈ R, there exists πr∈ΠsubscriptΠ _r∈ πitalic_r ∈ Π such that for all policies π⋆superscript⋆π π⋆, Jr(π⋆)−Jr(πr)≲β⋅π⋆less-than-or-similar-tosubscriptsuperscript⋆subscriptsubscript⋅superscriptsuperscript⋆J_r(π )-J_r( _r) β·C^π Jitalic_r ( π⋆ ) - Jitalic_r ( πitalic_r ) ≲ β ⋅ Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT. 2. Bounded overoptimization. For all π∈Ππ∈ π ∈ Π, ‖π‖∞≲Rβless-than-or-similar-tosubscriptnormsubscriptsubscript \| π _ ref \|_∞ R_% maxβ∥ divide start_ARG π end_ARG start_ARG πsansserif_ref end_ARG ∥∞ ≲ divide start_ARG Rsansserif_max end_ARG start_ARG β end_ARG. We view this as an interesting and non-trivial contribution in its own right. We mention in passing that while it is indeed possible to analyze χPO by first proving a sample complexity guarantee based on all-policy concentrability and then using that maxπ∈Π∞π≲Vβless-than-or-similar-tosubscriptΠsubscriptsuperscriptsubscript _π∈ C^π_∞ V_ maxβmaxitalic_π ∈ Π Citalic_π∞ ≲ divide start_ARG Vsansserif_max end_ARG start_ARG β end_ARG, this would lead to a loose bound relative to Theorem 3.1. 5 Analysis of χitalic_χPO: Proof Sketch for :main In this section, we sketch the proof of the main guarantee for χPO, Theorem 3.1, with the full proof deferred to Appendix F. A central object in the proof is the implicit reward model induced by the χPO policy π^ πover start_ARG π end_ARG, which we define via r^(x,a):=βϕ(π^(a∣x)π(a∣x)).assign^italic-ϕ^conditionalsubscriptconditional r(x,a) =βφ ( % π(a x) _ ref(a x) ).over start_ARG r end_ARG ( x , a ) := β ϕ ( divide start_ARG over start_ARG π end_ARG ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) . (19) As we will show, this reward model is a natural bridge between χPO and the corresponding mixed χ2superscript2χ^2χ2-regularized RLHF objective in Section 3.1, and allows us to view χPO from a reward-based perspective. In particular, note that if we analogously define an induced reward model class ℛΠ:=r(x,a)=βϕ(π(a∣x)π(a∣x)):π∈ΠassignsubscriptℛΠconditional-setitalic-ϕconditionalsubscriptconditionalΠR_ =\r(x,a)=βφ ( π(a x)% _ ref(a x) ):π∈ \Rroman_Π := r ( x , a ) = β ϕ ( divide start_ARG π ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) : π ∈ Π , then 3 of χPO can be viewed as performing maximum likelihood estimation over this class (in the sense of Eq. 3) under the Bradley-Terry model. Under Assumption 3.1, ℛΠsubscriptℛΠR_ Rroman_Π realizes the true reward function r up to an action-independent shift. As a result, if we define Δr(x,a,b):=r(x,a)−r(x,b)assignsuperscriptΔ ^r(x,a,b) =r(x,a)-r(x,b)Δitalic_r ( x , a , b ) := r ( x , a ) - r ( x , b ), then using a fairly standard generalization bound for maximum likelihood estimation (e.g., Wong and Shen (1995); Zhang (2006); de Geer (2000); see Lemma F.1), we can show that εstat2:=x∼ρ,a∼π,b∼π[|Δr^(x,a,b)−Δr⋆(x,a,b)|2]≤O(Ve2R⋅log(|Π|/δ)n).assignsuperscriptsubscriptstat2subscriptformulae-sequencesimilar-toformulae-sequencesimilar-tosubscriptsimilar-tosubscriptsuperscriptsuperscriptΔ^superscriptΔsuperscript⋆2⋅subscriptsuperscript2subscriptΠ _stat^2 =% E_x ρ,a _ ref,b _ ref [% r(x,a,b)- ^r (x,a,b) % ^2 ]≤ O (V_ maxe^2R_ max· % (| |/δ)n ).εroman_stat2 := blackboard_Ex ∼ ρ , a ∼ π start_POSTSUBSCRIPT sansserif_ref , b ∼ πsansserif_ref end_POSTSUBSCRIPT [ | Δover start_ARG r end_ARG ( x , a , b ) - Δitalic_r start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT ( x , a , b ) |2 ] ≤ O ( Vsansserif_max e2 Rsansserif_max ⋅ divide start_ARG log ( | Π | / δ ) end_ARG start_ARG n end_ARG ) . (20) In other words, the estimated reward model r^ rover start_ARG r end_ARG is accurate under the action distribution induced by πsubscript _ refπsansserif_ref. However, r^ rover start_ARG r end_ARG may still be inaccurate for policies that select different actions from πsubscript _ refπsansserif_ref, raising concerns of overoptimization. To address this issue, we use the following lemma, which shows that χ2superscript2χ^2χ2-divergence bounds the extent to which the accuracy of a reward model r^ rover start_ARG r end_ARG trained under πsubscript _ refπsansserif_ref will transfer to a downstream policy π of interest; this will motivate our use of χ2superscript2χ^2χ2-regularization. Lemma 5.1 (Informal version of Lemma F.3). For any policy π:→Δ():→Δπ:X→ (A)π : X → Δ ( A ), it holds that x∼ρ,a∼π(⋅∣x),b∼π(⋅∣x)[|Δr^(x,a,b)−Δr⋆(x,a,b)|]≲(1+Dχ2(π∥π))⋅εstat2. E_x ρ,a π(· x),b % _ ref(· x) [ r(x,a% ,b)- ^r (x,a,b) ] (1+D_χ^2% (π\;\|\; _ ref ))· _stat^% 2.blackboard_Ex ∼ ρ , a ∼ π ( ⋅ ∣ x ) , b ∼ π start_POSTSUBSCRIPT sansserif_ref ( ⋅ ∣ x ) end_POSTSUBSCRIPT [ | Δover start_ARG r end_ARG ( x , a , b ) - Δitalic_r start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT ( x , a , b ) | ] ≲ square-root start_ARG ( 1 + Ditalic_χ2 ( π ∥ πsansserif_ref ) ) ⋅ εroman_stat2 end_ARG . Going forward, let us abbreviate π,π[⋅]=x∼ρ,a∼π(⋅∣x),b∼π(⋅∣x)[⋅] E_π, _ ref [· ]=% E_x ρ,a π(· x),b _ % ref(· x) [· ]blackboard_Eπ , π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT [ ⋅ ] = blackboard_Ex ∼ ρ , a ∼ π ( ⋅ ∣ x ) , b ∼ π start_POSTSUBSCRIPT sansserif_ref ( ⋅ ∣ x ) end_POSTSUBSCRIPT [ ⋅ ]. Let π⋆superscript⋆π π⋆ be an arbitrary policy. Noting that π=1+2Dχ2(π∥π)superscript12subscriptsuperscript2conditionalsubscriptC^π=1+2D_χ^2 (π\;\|\; _ ref )Citalic_π = 1 + 2 Ditalic_χ2 ( π ∥ πsansserif_ref ) and that J(π⋆)−J(π^)≲π⋆,π[|Δr^(x,a,b)−Δr⋆(x,a,b)|]+π^,π[|Δr^(x,a,b)−Δr⋆(x,a,b)|],less-than-or-similar-tosuperscript⋆^subscriptsuperscript⋆subscriptsuperscriptΔ^superscriptΔsuperscript⋆subscript^subscriptsuperscriptΔ^superscriptΔsuperscript⋆ J(π )-J( π) E% _π , _ ref [ r(x% ,a,b)- ^r (x,a,b) ]+ E% _ π, _ ref [ r(x% ,a,b)- ^r (x,a,b) ],J ( π⋆ ) - J ( over start_ARG π end_ARG ) ≲ blackboard_Eπ⋆ , π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT [ | Δover start_ARG r end_ARG ( x , a , b ) - Δitalic_r start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT ( x , a , b ) | ] + blackboard_Eover start_ARG π end_ARG , π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT [ | Δover start_ARG r end_ARG ( x , a , b ) - Δitalic_r start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT ( x , a , b ) | ] , it follows immediately from Lemma 5.1 that χPO obtains a crude guarantee scaling with all-policy concentrability, i.e. J(π⋆)−J(π^)≲(π⋆+π^)εstat2≤(π⋆+maxπ∈Ππ)εstat2less-than-or-similar-tosuperscript⋆^superscriptsuperscript⋆superscript^superscriptsubscriptstat2superscriptsuperscript⋆subscriptΠsuperscriptsuperscriptsubscriptstat2J(π )-J( π) (C^π +% C π) _stat^2≤ (% C^π + _π∈ C^π) _% stat^2J ( π⋆ ) - J ( over start_ARG π end_ARG ) ≲ square-root start_ARG ( Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT + Cover start_ARG π end_ARG ) εroman_stat2 end_ARG ≤ square-root start_ARG ( Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT + maxitalic_π ∈ Π Citalic_π ) εroman_stat2 end_ARG. This inequality is tight for non-pessimistic algorithms like DPO, which reflects their sensitivity to overoptimization. To obtain the improved guarantee for χPO in Theorem 3.1, which scales only with single-policy concentrability π⋆superscriptsuperscript⋆C^π Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT, the crux of the remaining proof will be to show that χPO implicitly implements pessimism via mixed χ2superscript2χ^2χ2-regularization. For this, we appeal to the following central technical lemma, which we expect to find broader use. Lemma 5.2 (Informal version of Lemma F.2). Let f be a convex function with dom(f)=ℝ+domsubscriptℝdom(f)=R_+dom ( f ) = blackboard_R+ that is differentiable over its domain. Given any parameter β>00β>0β > 0 and policy π¯:→Δ():¯→Δ π:X→ (A)over¯ start_ARG π end_ARG : X → Δ ( A ) with π¯(a∣x)∈dom(f′)¯conditionaldomsuperscript′ π(a x) (f )over¯ start_ARG π end_ARG ( a ∣ x ) ∈ dom ( f′ ) for all x,ax,ax , a, define the reward model r¯(x,a)=βf′( @depthΔ @everymath @group @set@skewchar @nested@a111(a|x)π(a|x))¯superscript′ @depthΔ @everymath @group @set@skewchar @nested@a111conditionalsubscriptconditional r(x,a)=β f ( @depth 1 % @everymath @group @set@skewchar @nested@a 111(a|x)π% _ ref(a|x) )over¯ start_ARG r end_ARG ( x , a ) = β f′ ( divide start_ARG Δ 111 ( a | x ) end_ARG start_ARG πsansserif_ref ( a | x ) end_ARG ). Then π¯∈argmaxπ[r¯(x,a)]−β⋅Df(π∥π).¯subscriptargmaxsubscript¯⋅subscriptconditionalsubscript π∈ argmax_π % E_π [ r(x,a) ]-β· D_f (π\,\|\, _% ref ).over¯ start_ARG π end_ARG ∈ argmaxitalic_π blackboard_Eπ [ over¯ start_ARG r end_ARG ( x , a ) ] - β ⋅ Ditalic_f ( π ∥ πsansserif_ref ) . Under Assumption 3.2 we have π^∈dom(fχmix′)^domsuperscriptsubscriptsubscriptmix′ π (f_ _ mix )over start_ARG π end_ARG ∈ dom ( fitalic_χ start_POSTSUBSCRIPT mix end_POSTSUBSCRIPT′ ). Then recalling that r^(x,a):=βϕ(π^(a∣x)π(a∣x))=βfχmix′(π^(a∣x)π(a∣x))assign^italic-ϕ^conditionalsubscriptconditionalsuperscriptsubscriptsubscriptmix′^conditionalsubscriptconditional r(x,a) =βφ ( π(a x)% _ ref(a x) )=β f_ _ mix % ( π(a x) _ ref(a x) )over start_ARG r end_ARG ( x , a ) := β ϕ ( divide start_ARG over start_ARG π end_ARG ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) = β fitalic_χ start_POSTSUBSCRIPT mix end_POSTSUBSCRIPT′ ( divide start_ARG over start_ARG π end_ARG ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) and that fχmixsubscriptsubscriptmixf_ _ mixfitalic_χ start_POSTSUBSCRIPT mix end_POSTSUBSCRIPT is convex, Lemma 5.2 implies that the policy π^ πover start_ARG π end_ARG produced by χPO satisfies π^∈argmaxπ∈ΠJβ,r^χmix(π):=π[r^]−βDχ2(π∥π)−βD(π∥π).^subscriptargmaxΠsuperscriptsubscript^subscriptmixassignsubscript^subscriptsuperscript2conditionalsubscriptsubscriptconditionalsubscript π∈ argmax_π∈ J_β,% r _ mix(π) = E% _π [ r ]-β D_χ^2 (π\;\|\; _% ref )-β D_ KL (π\,\|\, _ ref% ).over start_ARG π end_ARG ∈ argmaxitalic_π ∈ Π Jitalic_β , over start_ARG r end_ARGχmix ( π ) := blackboard_Eπ [ over start_ARG r end_ARG ] - β Ditalic_χ2 ( π ∥ πsansserif_ref ) - β Dsansserif_KL ( π ∥ πsansserif_ref ) . (21) In other words, The χPO policy π^ πover start_ARG π end_ARG optimizes the mixed χ2superscript2χ^2χ2-regularized RLHF objective under its own implicit reward model. This formally justifies the claim that χPO implicitly implements pessimism via χ2superscript2χ^2χ2-regularization. With this result in hand, we are now ready to prove Theorem 3.1. Let π⋆superscript⋆π π⋆ be an arbitrary policy. Since Jβ,r^χmix(π^)≥Jβ,r^χmix(π⋆)superscriptsubscript^subscriptmix^superscriptsubscript^subscriptmixsuperscript⋆J_β, r _ mix( π)≥ J_β,% r _ mix(π )Jitalic_β , over start_ARG r end_ARGχmix ( over start_ARG π end_ARG ) ≥ Jitalic_β , over start_ARG r end_ARGχmix ( π⋆ ) by Eq. 21, we can decompose the regret J(π⋆)−J(π^)superscript⋆^J(π )-J( π)J ( π⋆ ) - J ( over start_ARG π end_ARG ) as J(π⋆)−J(π^)≤superscript⋆^absent J(π )-J( π) ( π⋆ ) - J ( over start_ARG π end_ARG ) ≤ J(π⋆)−Jβ,r^χmix(π⋆)+Jβ,r^χmix(π^)−J(π^)superscript⋆superscriptsubscript^subscriptmixsuperscript⋆superscriptsubscript^subscriptmix^ ~J(π )-J_β, r _ mix(% π )+J_β, r _ mix( π)-J(% π)J ( π⋆ ) - Jitalic_β , over start_ARG r end_ARGχmix ( π⋆ ) + Jitalic_β , over start_ARG r end_ARGχmix ( over start_ARG π end_ARG ) - J ( over start_ARG π end_ARG ) = == J(π⋆)−J(π)−Jβ,r^χmix(π⋆)+Jβ,r^χmix(π)⏟(I)+Jβ,r^χmix(π^)−Jβ,r^χmix(π)−J(π^)+J(π)⏟(I).subscript⏟superscript⋆subscriptsuperscriptsubscript^subscriptmixsuperscript⋆superscriptsubscript^subscriptmixsubscript(I)subscript⏟superscriptsubscript^subscriptmix^superscriptsubscript^subscriptmixsubscript^subscript(I) ~ J(π )-J( _ ref)-J_β,% r _ mix(π )+J_β, r^χ% _ mix( _ ref)_(I)+ J_β,% r _ mix( π)-J_β, r^% _ mix( _ ref)-J( π)+J( _ ref% )_(I).under⏟ start_ARG J ( π⋆ ) - J ( πsansserif_ref ) - Jitalic_β , over start_ARG r end_ARGχmix ( π⋆ ) + Jitalic_β , over start_ARG r end_ARGχmix ( πsansserif_ref ) end_ARG(I) + under⏟ start_ARG Jitalic_β , over start_ARG r end_ARGχmix ( over start_ARG π end_ARG ) - Jitalic_β , over start_ARG r end_ARGχmix ( πsansserif_ref ) - J ( over start_ARG π end_ARG ) + J ( πsansserif_ref ) end_ARG(I) . In the second line, we have added or subtracted the baselines J(π)subscriptJ( _ ref)J ( πsansserif_ref ) and Jβ,r^χmix(π)superscriptsubscript^subscriptmixsubscriptJ_β, r _ mix( _ ref)Jitalic_β , over start_ARG r end_ARGχmix ( πsansserif_ref ) to center the objectives with the performance of the reference policy. Up to statistical errors, the first term (I) corresponds to error from how much Jβ,r^χmix(π⋆)superscriptsubscript^subscriptmixsuperscript⋆J_β, r _ mix(π )Jitalic_β , over start_ARG r end_ARGχmix ( π⋆ ) underestimates the return of π⋆superscript⋆π π⋆ (bias), and the second term (I) corresponds to error from how much Jβ,r^χmix(π^)superscriptsubscript^subscriptmix^J_β, r _ mix( π)Jitalic_β , over start_ARG r end_ARGχmix ( over start_ARG π end_ARG ) overestimates the return of π^ πover start_ARG π end_ARG (overoptimization). As we will see shortly, these two sources of error are directly controlled (in opposing ways) by the strength of the regularization parameter β in Eq. 21. First, expanding the definition of Jβ,r^χmix(π⋆)superscriptsubscript^subscriptmixsuperscript⋆J_β, r _ mix(π )Jitalic_β , over start_ARG r end_ARGχmix ( π⋆ ) and centering the returns using the reference policies, we have (I) =J(π⋆)−Jβ,r^χmix(π⋆)−J(π)+Jβ,r^χmix(π)absentsuperscript⋆superscriptsubscript^subscriptmixsuperscript⋆subscriptsuperscriptsubscript^subscriptmixsubscript =J(π )-J_β, r _ mix(π% )-J( _ ref)+J_β, r _ mix(% _ ref)= J ( π⋆ ) - Jitalic_β , over start_ARG r end_ARGχmix ( π⋆ ) - J ( πsansserif_ref ) + Jitalic_β , over start_ARG r end_ARGχmix ( πsansserif_ref ) =π⋆[r⋆(x,a)]−π⋆[r^(x,a)]+βDχ2(π⋆∥π)+βD(π⋆∥π)−π^[r⋆(x,a)]+π[r^(x,a)]absentsubscriptsuperscript⋆superscript⋆subscriptsuperscript⋆^subscriptsuperscript2conditionalsuperscript⋆subscriptsubscriptconditionalsuperscript⋆subscriptsubscript^superscript⋆subscriptsubscript = E_π [r (x,a)% ]- E_π [ r(x,a) ]% +β D_χ^2 (π \;\|\; _ ref )+β D_% KL (π \,\|\, _ ref )-% E_ π[r (x,a)]+ E_ _% ref[ r(x,a)]= blackboard_Eπ⋆ [ r⋆ ( x , a ) ] - blackboard_Eπ⋆ [ over start_ARG r end_ARG ( x , a ) ] + β Ditalic_χ2 ( π⋆ ∥ πsansserif_ref ) + β Dsansserif_KL ( π⋆ ∥ πsansserif_ref ) - blackboard_Eover start_ARG π end_ARG [ r⋆ ( x , a ) ] + blackboard_Eπ start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT [ over start_ARG r end_ARG ( x , a ) ] =π⋆,π[Δr⋆(x,a,b)−Δr^(x,a,b)]+βDχ2(π⋆∥π)+βD(π⋆∥π)absentsubscriptsuperscript⋆subscriptsuperscriptΔsuperscript⋆superscriptΔ^subscriptsuperscript2conditionalsuperscript⋆subscriptsubscriptconditionalsuperscript⋆subscript = E_π , _ ref[% ^r (x,a,b)- r(x,a,b)]+β D_χ^2% (π \;\|\; _ ref )+β D_ KL % (π \,\|\, _ ref )= blackboard_Eπ⋆ , π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT [ Δitalic_r start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT ( x , a , b ) - Δover start_ARG r end_ARG ( x , a , b ) ] + β Ditalic_χ2 ( π⋆ ∥ πsansserif_ref ) + β Dsansserif_KL ( π⋆ ∥ πsansserif_ref ) ≤(1+Dχ2(π⋆∥π))⋅εstat2+β⋅Dχ2(π⋆∥π)⏟bias.absent⋅1subscriptsuperscript2conditionalsuperscript⋆subscriptsuperscriptsubscriptstat2subscript⏟⋅subscriptsuperscript2conditionalsuperscript⋆subscriptbias ≤ (1+D_χ^2 (π \;\|\; _ % ref ))· _stat^2+ β· D_% χ^2 (π \;\|\; _ ref )_bias.≤ square-root start_ARG ( 1 + Ditalic_χ2 ( π⋆ ∥ πsansserif_ref ) ) ⋅ εroman_stat2 end_ARG + under⏟ start_ARG β ⋅ Ditalic_χ2 ( π⋆ ∥ πsansserif_ref ) end_ARGbias . Above, we have used that D(π∥π)≤Dχ2(π∥π)subscriptconditionalsubscriptsubscriptsuperscript2conditionalsubscriptD_ KL (π\,\|\, _ ref )≤ D_χ^2 (% π\;\|\; _ ref )Dsansserif_KL ( π ∥ πsansserif_ref ) ≤ Ditalic_χ2 ( π ∥ πsansserif_ref ) for any policy π, along with the bound on reward estimation error from Lemma 5.1. Next, expanding Jβ,r^χmix(π^)superscriptsubscript^subscriptmix^J_β, r _ mix( π)Jitalic_β , over start_ARG r end_ARGχmix ( over start_ARG π end_ARG ) and centering the returns in a similar fashion, (I) =Jβ,r^χmix(π^)−J(π^)−Jβ,r^χmix(π)+J(π)absentsuperscriptsubscript^subscriptmix^^superscriptsubscript^subscriptmixsubscriptsubscript =J_β, r _ mix( π)-J(% π)-J_β, r _ mix( _ ref% )+J( _ ref)= Jitalic_β , over start_ARG r end_ARGχmix ( over start_ARG π end_ARG ) - J ( over start_ARG π end_ARG ) - Jitalic_β , over start_ARG r end_ARGχmix ( πsansserif_ref ) + J ( πsansserif_ref ) =π^,π[Δr^(x,a,b)−Δr⋆(x,a,b)]−βDχ2(π^∥π)−βD(π^∥π)absentsubscript^subscriptsuperscriptΔ^superscriptΔsuperscript⋆subscriptsuperscript2conditional^subscriptsubscriptconditional^subscript = E_ π, _ ref[% r(x,a,b)- ^r (x,a,b)]-β D_χ^2% ( π\;\|\; _ ref )-β D_ KL% ( π\,\|\, _ ref )= blackboard_Eover start_ARG π end_ARG , π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT [ Δover start_ARG r end_ARG ( x , a , b ) - Δitalic_r start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT ( x , a , b ) ] - β Ditalic_χ2 ( over start_ARG π end_ARG ∥ πsansserif_ref ) - β Dsansserif_KL ( over start_ARG π end_ARG ∥ πsansserif_ref ) ≤(1+Dχ2(π^∥π))⋅εstat2−β⋅Dχ2(π^∥π)absent⋅1subscriptsuperscript2conditional^subscriptsuperscriptsubscriptstat2⋅subscriptsuperscript2conditional^subscript ≤ (1+D_χ^2 ( π\;\|\; _ % ref ))· _stat^2-β· D_χ^2% ( π\;\|\; _ ref )≤ square-root start_ARG ( 1 + Ditalic_χ2 ( over start_ARG π end_ARG ∥ πsansserif_ref ) ) ⋅ εroman_stat2 end_ARG - β ⋅ Ditalic_χ2 ( over start_ARG π end_ARG ∥ πsansserif_ref ) ≲εstat+β−1εstat2⏟overoptimization error.less-than-or-similar-toabsentsubscriptstatsubscript⏟superscript1superscriptsubscriptstat2overoptimization error _stat+ β^-1% _stat^2_ overoptimization error.≲ εroman_stat + under⏟ start_ARG β- 1 εroman_stat2 end_ARGoveroptimization error . Above, the first inequality uses D(π∥π)≥0subscriptconditionalsubscript0D_ KL (π\,\|\, _ ref )≥0Dsansserif_KL ( π ∥ πsansserif_ref ) ≥ 0 and Lemma 5.1, while the second inequality uses AM-GM. Critically, by using χ2superscript2χ^2χ2-regularization, we are able to cancel the on-policy error term (1+Dχ2(π^∥π))⋅εstat2⋅1subscriptsuperscript2conditional^subscriptsuperscriptsubscriptstat2 (1+D_χ^2 ( π\;\|\; _ ref ))% · _stat^2square-root start_ARG ( 1 + Ditalic_χ2 ( over start_ARG π end_ARG ∥ πsansserif_ref ) ) ⋅ εroman_stat2 end_ARG that arises from change-of-measure, leading to a modest β−1εstat2superscript1superscriptsubscriptstat2β^-1 _stat^2β- 1 εroman_stat2 penalty for overoptimization. Combining these results, and recalling that π=1+2Dχ2(π∥π)superscript12subscriptsuperscript2conditionalsubscriptC^π=1+2D_χ^2 (π\;\|\; _ ref )Citalic_π = 1 + 2 Ditalic_χ2 ( π ∥ πsansserif_ref ), we conclude that J(π⋆)−J(π^)≲π⋆⋅εstat2+β⋅π⋆⏟bias+β−1⋅εstat2⏟overoptimization error.less-than-or-similar-tosuperscript⋆^⋅superscriptsuperscript⋆superscriptsubscriptstat2subscript⏟⋅superscriptsuperscript⋆biassubscript⏟⋅superscript1superscriptsubscriptstat2overoptimization errorJ(π )-J( π) C^π ·% _stat^2+ β·C^π % _ bias+ β^-1· _stat^2% _ overoptimization error.J ( π⋆ ) - J ( over start_ARG π end_ARG ) ≲ square-root start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT ⋅ εroman_stat2 end_ARG + under⏟ start_ARG β ⋅ Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT end_ARGbias + under⏟ start_ARG β- 1 ⋅ εroman_stat2 end_ARGoveroptimization error . The bias and overoptimization errors above arise from how well our chosen uncertainty quantifier, βDχ2(π∥π)subscriptsuperscript2conditionalsubscriptβ D_χ^2 (π\;\|\; _ ref )β Ditalic_χ2 ( π ∥ πsansserif_ref ), accounts for the on-policy statistical error (1+Dχ2(π∥π))⋅εstat2⋅1subscriptsuperscript2conditionalsubscriptsuperscriptsubscriptstat2 (1+D_χ^2 (π\;\|\; _ ref ))·% _stat^2square-root start_ARG ( 1 + Ditalic_χ2 ( π ∥ πsansserif_ref ) ) ⋅ εroman_stat2 end_ARG arising from Lemma 5.1; this is controlled by the magnitude of the regularization parameter β. When β is too large, the uncertainty quantifier is overly pessimistic about the quality of the reward model r^ rover start_ARG r end_ARG under π⋆superscript⋆π π⋆, which increases the bias of χPO. In contrast, the overoptimization error increases when β is too small. In this regime, π^ πover start_ARG π end_ARG overfits to r^ rover start_ARG r end_ARG because the regularizer under-evaluates the statistical error of the learned policy. In order to obtain tight statistical rates, the choice of regularization parameter β must carefully balance its opposing effects on bias and overoptimization error. For a fixed π⋆superscript⋆π π⋆, choosing β∝(εstat2/π⋆)1/2proportional-tosuperscriptsuperscriptsubscriptstat2superscriptsuperscript⋆12β ( _stat^2/C^π )^1/2β ∝ ( εroman_stat2 / Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT )1 / 2 results in the second claim in Theorem 3.1. 6 Experiments in Offline Language Model Alignment We perform preliminary evaluations of χPO for offline language model alignment on the TL;DR dataset (Stiennon et al., 2020), using DPO as our comparison baseline. The reference policy πsubscript _ refπsansserif_ref is the Pythia-1b model (Biderman et al., 2023) pre-trained on SFT data (cleanrl/EleutherAI_pythia-1b-deduped__sft__tldr from Huang et al. (2022)), and performance is measured via winrate against a baseline, as judged by GPT-4o. All parameters that are not algorithm-specific, such as the learning rate, are shared by both χPO and DPO in order to ensure a fair comparison (see Appendix C for details). Table 1: Winrate on TL;DR Summarization for models learned by χPO and DPO, for several choices of the number of training epochs and the regularization parameter β. Standard error over 3 seeds is also reported. β Epochs χPO winrate (%) DPO winrate (%) 0.05 1 56.5±1.3plus-or-minus56.51.356.5± 1.356.5 ± 1.3 55.8±2.1plus-or-minus55.82.155.8± 2.155.8 ± 2.1 2 56.1±0.6plus-or-minus56.10.656.1± 0.656.1 ± 0.6 50.3±0.8plus-or-minus50.30.850.3± 0.850.3 ± 0.8 4 48.0±1.6plus-or-minus48.01.648.0± 1.648.0 ± 1.6 38.0±0.7plus-or-minus38.00.738.0± 0.738.0 ± 0.7 0.005 1 50.6±1.6plus-or-minus50.61.650.6± 1.650.6 ± 1.6 14.7±3.9plus-or-minus14.73.914.7± 3.914.7 ± 3.9 2 52.8±2.3plus-or-minus52.82.352.8± 2.352.8 ± 2.3 3.4±1.5plus-or-minus3.41.53.4± 1.53.4 ± 1.5 4 51.6±0.8plus-or-minus51.60.851.6± 0.851.6 ± 0.8 0.5±0.2plus-or-minus0.50.20.5± 0.20.5 ± 0.2 In Table 1 we display the winrates of χPO and DPO over several choices of training epochs, as well as regularization parameter β. The winrate corresponds to the final checkpoint learned by each algorithm for each set of hyperparameters. We consider β=0.050.05β=0.05β = 0.05 and 1 epoch of training to be a standard setup for DPO (Gao et al., 2024; Guo et al., 2024; Rafailov et al., 2024a), and, as we are particularly concerned with regimes where overoptimization is of concern, we additionally analyze performance when epochs are increased, and/or β is decreased (corresponding to less regularization). Figure 4: (Left) TL;DR Summarization winrate recorded longitudinally over 2 epochs of training every 250 steps. Shaded area displays ±1plus-or-minus1± 1± 1 standard error over 3 seeds. At 1 epoch χPO already obtains better performance, and continues to improve over the course of training, while DPO degrades over time. (Right) KL divergence D(π^∥π)subscriptconditional^subscriptD_ KL ( π\,\|\, _ ref )Dsansserif_KL ( over start_ARG π end_ARG ∥ πsansserif_ref ) averaged over 2 of the seeds. For the same β, χPO constrains the learned policy to be significantly closer to πsubscript _ refπsansserif_ref, thereby striking a better bias-variance tradeoff. Over all choices of β and epochs, χPO achieves a higher average winrate than DPO. While the difference is not significant for β=0.050.05β=0.05β = 0.05 and 1 epoch, the performance gap grows significantly as the number of epochs increases, demonstrating the robustness of χPO to overoptimization. Further, while DPO degrades completely for β=0.0050.005β=0.005β = 0.005, χPO is robust over two orders of magnitude of β, reinforcing trends seen earlier in Fig. 3 and the more favorable bias-overoptimization tradeoff from our theoretical analysis. In addition, χPO exhibits better performance and robustness longitudinally throughout training, as shown in Fig. 4. While DPO peaks early with high variance around 0.5 epochs and degrades thereafter, χPO continues to improve smoothly then plateaus over the last epoch. Further, for the same regularization parameter β, the χPO policy has significantly lower KL-divergence relative to πsubscript _ refπsansserif_ref, demonstrating that the χ2superscript2χ^2χ2-regularization is both a stronger regularizer and one that effectively mitigates overoptimization. 7 χitalic_χPO for General Preference Models whzblue All of our results so far concern the Bradley-Terry model (Eq. 1), which, as highlighted in prior work, is somewhat restrictive. Thus, in this section, we turn our attention to offline alignment under a general preference model which does not assume transitivity (Munos et al., 2023; Wang et al., 2023b; Swamy et al., 2024; Rosset et al., 2024; Ye et al., 2024). The setup is the same as Section 2, but we assume that for a given context x and pair of actions (a,b)(a,b)( a , b ), the preference y∈0,101y∈\0,1\y ∈ 0 , 1 is generated via a Bernoulli Distribution y∼Ber(⋆(a≻b∣x)),similar-toBersuperscript⋆succeedsconditional y Ber (P (a b x)% ),y ∼ Ber ( P⋆ ( a ≻ b ∣ x ) ) , (22) where ⋆(a≻b∣x)∈[0,1]superscript⋆succeedsconditional01P (a b x)∈[0,1]P⋆ ( a ≻ b ∣ x ) ∈ [ 0 , 1 ] is a general preference distribution. For a pair of policies π,π′π,π π , π′, let ⋆(π≻π′):=x∼ρ[⋆(π(x)≻π′(x)∣x)]assignsuperscript⋆succeedssuperscript′subscriptsimilar-tosuperscript⋆succeedsconditionalsuperscript′P (π π ) = E% _x ρ [P (π(x) π (x) x) ]P⋆ ( π ≻ π′ ) := blackboard_Ex ∼ ρ [ P⋆ ( π ( x ) ≻ π′ ( x ) ∣ x ) ]. Following Wang et al. (2023b); Munos et al. (2023); Swamy et al. (2024), we consider the minimax winner (Kreweras, 1965; Simpson, 1969; Kramer, 1973; Fishburn, 1984) or von Neumann winner (Dudík et al., 2015) as a solution concept: π:=argmaxπ∈Πminπ′∈Π⋆(π≻π′).assignsubscriptsubscriptargmaxΠsubscriptsuperscript′Πsuperscript⋆succeedssuperscript′ _ MW = argmax_π∈% _π ∈ P (π π ).πsansserif_MW := argmaxitalic_π ∈ Π minitalic_π′ ∈ Π P⋆ ( π ≻ π′ ) . It will be useful to slightly reparameterize this formulation by introducing the preference function ℓ⋆(x,a,b):=2⋆(a≻b∣x)−1assignsuperscriptℓ⋆2superscript⋆succeedsconditional1 (x,a,b) =2P (a b x)-1ℓ⋆ ( x , a , b ) := 2 P⋆ ( a ≻ b ∣ x ) - 1. Note that for any well-defined preference model, we have ⋆(a≻b∣x)+⋆(b≻a∣x)=1superscript⋆succeedsconditionalsuperscript⋆succeedsconditional1P (a b x)+P (b a x)=1P⋆ ( a ≻ b ∣ x ) + P⋆ ( b ≻ a ∣ x ) = 1 for all x,a,bx,a,bx , a , b, which indicates that ℓ⋆superscriptℓ⋆ ℓ⋆ satisfies skew symmetry: ℓ⋆(x,a,a)=0,ℓ⋆(x,a,b)+ℓ⋆(x,b,a)=0,∀x∈,a,b∈.formulae-sequencesuperscriptℓ⋆0formulae-sequencesuperscriptℓ⋆superscriptℓ⋆0formulae-sequencefor-all (x,a,a)=0, (x,a,b)+ (x,b,% a)=0, ∀ x ,a,b .ℓ⋆ ( x , a , a ) = 0 , ℓ⋆ ( x , a , b ) + ℓ⋆ ( x , b , a ) = 0 , ∀ x ∈ X , a , b ∈ A . Furthermore, the minimax winner above is equivalent to π:=argmaxπ∈Πminπ′∈Πℓ⋆(π,π′),assignsubscriptsubscriptargmaxΠsubscriptsuperscript′Πsuperscriptℓ⋆superscript′ _ MW = argmax_π∈% _π ∈ (π,π ),πsansserif_MW := argmaxitalic_π ∈ Π minitalic_π′ ∈ Π ℓ⋆ ( π , π′ ) , (23) where ℓ⋆(π,π′):=x∼ρ,a∼π(x),b∼π′(x)[ℓ⋆(x,a,b)]assignsuperscriptℓ⋆superscript′subscriptformulae-sequencesimilar-toformulae-sequencesimilar-tosimilar-tosuperscript′delimited-[]superscriptℓ⋆ (π,π ):=E_x ρ,a π(x),b π^% (x)[ (x,a,b)]ℓ⋆ ( π , π′ ) := blackboard_Ex ∼ ρ , a ∼ π ( x ) , b ∼ π′ ( x ) [ ℓ⋆ ( x , a , b ) ]. Concretely, our goal is to use the logged preference data =(x,a+,a−)subscriptsubscriptsubscriptD_ pref= \(x,a_+,a_-) \Dsansserif_pref = ( x , a+ , a- ) (with (a+,a−)subscriptsubscript(a_+,a_-)( a+ , a- ) labeled according to Eq. 22) to compute a policy π^ πover start_ARG π end_ARG that is an ε ε-approximate minimax winner, in the sense that (π^):=maxπ∈Πℓ⋆(π,π^)−minπ∈Πℓ⋆(π^,π)≤ε.assign^subscriptΠsuperscriptℓ⋆^subscriptΠsuperscriptℓ⋆ DG( π) = _π∈ ^% (π, π)- _π∈ ( π,π)≤ .sansserif_DG ( over start_ARG π end_ARG ) := maxitalic_π ∈ Π ℓ⋆ ( π , over start_ARG π end_ARG ) - minitalic_π ∈ Π ℓ⋆ ( over start_ARG π end_ARG , π ) ≤ ε . (24) 7.1 Impossibility of Single-Policy Concentrability under General Preferences While the general preference framework above is more powerful than the Bradley-Terry model, we now show that there is a statistical cost for this generality. In particular, our first result in this section shows that in contrast to the Bradley-Terry model, it is not possible to achieve sample complexity guarantees that scale with single-policy concentrability under general preferences, even when the learner has access to a small class of preference models PP that contains the true preference model PP (i.e., ⋆∈superscript⋆P ∈ PP⋆ ∈ P). Theorem 7.1 (Impossibility of single-policy concentrability under general preferences). There exists two problem instances θ1=(ρ,1⋆,Π)subscript1subscriptsuperscript⋆1Π _1=(ρ,P _1, )θ1 = ( ρ , P⋆1 , Π ) and θ2=(ρ,2⋆,Π)subscript2subscriptsuperscript⋆2Π _2=(ρ,P _2, )θ2 = ( ρ , P⋆2 , Π ) differing only in their ground truth preference model, a data collection policy πsubscript _ refπsansserif_ref, and a preference model class =1⋆,2⋆subscriptsuperscript⋆1subscriptsuperscript⋆2 P=\P _1,P _2\P = P⋆1 , P⋆2 with ||=22| P|=2| P | = 2 such that the following hold: 1. For both instances, the single-policy L∞subscriptL_∞L∞-concentrability coefficient for a minimax winner is bounded: minπ∞π≤2subscriptsubscriptsuperscriptsubscriptsubscript2 _ _ MWC_∞ _ MW≤2minitalic_π start_POSTSUBSCRIPT sansserif_MW end_POSTSUBSCRIPT C∞italic_πsansserif_MW ≤ 2.666In general, the minimax winner may not be unique. We compete against the minimax winner with the best possible single-policy concentrability coefficient. 2. For any n∈ℕn ∈ blackboard_N and any algorithm Algsansserif_Alg which derives a policy π^ πover start_ARG π end_ARG from a dataset subscriptD_ prefDsansserif_pref of n samples, there exists an instance θ∈θ1,θ2subscript1subscript2θ∈\ _1, _2\θ ∈ θ1 , θ2 such that πsubscript _ refπsansserif_ref incurs constant suboptimality: minmaxi∈1,2∼θi[(();θi)]≥18,subscriptsubscript12subscriptsimilar-tosubscriptsubscriptsubscriptsubscript18 _ Alg _i∈\1,2\ E_% D_ pref _i [ DG( Alg(% D_ pref); _i) ]≥ 18,minsansserif_Alg maxitalic_i ∈ 1 , 2 blackboard_ED start_POSTSUBSCRIPT sansserif_pref ∼ θitalic_i end_POSTSUBSCRIPT [ sansserif_DG ( sansserif_Alg ( Dsansserif_pref ) ; θitalic_i ) ] ≥ divide start_ARG 1 end_ARG start_ARG 8 end_ARG , where (π;θ) DG(π;θ)sansserif_DG ( π ; θ ) is the duality gap for policy π on instance θ. This lower bound is inspired by similar results in the literature on offline RL in two-player zero-sum Markov games (Cui and Du, 2022). However, the lower bound constructions in Cui and Du (2022) cannot be directly applied as-is, because they do not satisfy the skew-symmetry property required by the general preference alignment framework. Our lower bound highlights that even under skew-symmetry, it is impossible to achieve single-policy concentrability for offline learning in two-player zero-sum games. 7.2 Iterative χitalic_χPO for General Preferences In spite of the hardness in the prequel, we now show that an iterative variant of χPO—based on self-play—can learn a near-optimal minimax winner under the general preference model under a new local coverage condition—a condition that is stronger than the single policy concentrability but much weaker than global/all-policy concentrability and the notion of unilateral concentrability introduced by Cui and Du (2022). Our algorithm, Iterative χPO, is described in Algorithm 2, and consists of two main steps. Preference model estimation via least squares regression on subscriptD_ prefDsansserif_pref We first (3) learn a preference model from the offline preference dataset subscriptD_ prefDsansserif_pref. We assume access to a preference function class ℒLL which is realizable in the sense that ℓ⋆∈ℒsuperscriptℓ⋆ℒ ℓ⋆ ∈ L and where all ℓ∈ℒℓℒ ℓ ∈ L satisfy skew-symmetryc, and we will estimate ℓ⋆superscriptℓ⋆ ℓ⋆ rather than ⋆superscript⋆P P⋆. We perform least-squares regression on subscriptD_ prefDsansserif_pref with ℒLL to learn ℓ⋆superscriptℓ⋆ ℓ⋆: ℓ^=argminℓ∈ℒ∑(x,a+,a−)∈(ℓ(x,a+,a−)−1)2.^ℓsubscriptargminℓℒsubscriptsubscriptsubscriptsubscriptsuperscriptℓsubscriptsubscript12 = argmin_ _% (x,a_+,a_-) _ pref ( (x,a_+,a_-)-1% )^2.over start_ARG ℓ end_ARG = argminroman_ℓ ∈ L ∑( x , a start_POSTSUBSCRIPT + , a- ) ∈ Dsansserif_pref end_POSTSUBSCRIPT ( ℓ ( x , a+ , a- ) - 1 )2 . Policy optimization with iterative χPO update Given the estimated model ℓ^^ℓ over start_ARG ℓ end_ARG, we compute an approximate minimax winner using an iterative regression scheme inspired by Gao et al. (2024). We proceed in T iterations (5), where at each iteration t, we define an iteration-dependent reward function r¯t(x,a)superscript¯ r^t(x,a)over¯ start_ARG r end_ARGt ( x , a ) based on the current policy πtsuperscriptπ^tπitalic_t as r¯t(x,a)=b∼πt(x)[ℓ^(x,a,b)],∀x∈,a∈.formulae-sequencesuperscript¯subscriptsimilar-tosuperscriptdelimited-[]^ℓformulae-sequencefor-all r^t(x,a)=E_b π^t(x)[ % (x,a,b)], ∀ x ,a .over¯ start_ARG r end_ARGt ( x , a ) = blackboard_Eb ∼ πitalic_t ( x ) [ over start_ARG ℓ end_ARG ( x , a , b ) ] , ∀ x ∈ X , a ∈ A . Then, for all π,π′∈Πsuperscript′Ππ,π ∈ π , π′ ∈ Π, we define a policy-dependent predictor fπ,π′β,η(x,a,b)subscriptsuperscriptsuperscript′f^β,η_π,π (x,a,b)fitalic_β , ηitalic_π , π′ ( x , a , b ), whose motivation will be described in detail momentarily, as follows: fπ,π′β,η(x,a,b)subscriptsuperscriptsuperscript′ f^β,η_π,π (x,a,b)fitalic_β , ηitalic_π , π′ ( x , a , b ) :=(1+1η)⋅(βϕ(π(a∣x)π(a∣x))−βϕ(π(b∣x)π(b∣x)))assignabsent⋅11italic-ϕconditionalsubscriptconditionalitalic-ϕconditionalsubscriptconditional := (1+ 1η )· (βφ ( % π (a x ) _ ref (a x ) )-β% φ ( π (b x ) _ ref (b x % ) ) ):= ( 1 + divide start_ARG 1 end_ARG start_ARG η end_ARG ) ⋅ ( β ϕ ( divide start_ARG π ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) - β ϕ ( divide start_ARG π ( b ∣ x ) end_ARG start_ARG πsansserif_ref ( b ∣ x ) end_ARG ) ) −1η(βϕ(π′(a∣x)π(a∣x))−βϕ(π′(b∣x)π(b∣x)))1italic-ϕsuperscript′conditionalsubscriptconditionalitalic-ϕsuperscript′conditionalsubscriptconditional ~~~~~~- 1η (βφ ( π^% (a x ) _ ref (a x ) )-% βφ ( π (b x ) _ ref % (b x ) ) )- divide start_ARG 1 end_ARG start_ARG η end_ARG ( β ϕ ( divide start_ARG π′ ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) - β ϕ ( divide start_ARG π′ ( b ∣ x ) end_ARG start_ARG πsansserif_ref ( b ∣ x ) end_ARG ) ) (25) Using fπ,πtβ,η(x,a,b)subscriptsuperscriptsuperscriptf^β,η_π,π^t(x,a,b)fitalic_β , ηitalic_π , πitalic_t ( x , a , b ) as a policy-parameterized regression function, we (7) compute the next policy πt+1superscript1π^t+1πitalic_t + 1 by solving a least-squares regression problem in which the Bayes optimal solution is the relative reward r¯t(x,a)−r¯t(x,b)superscript¯superscript¯ r^t(x,a)- r^t(x,b)over¯ start_ARG r end_ARGt ( x , a ) - over¯ start_ARG r end_ARGt ( x , b ) for iteration t. Algorithm 2 Iterative χPO for General Preferences 1:Input: labeled preference dataset subscriptD_ prefDsansserif_pref, preference model class ℒLL, regularization coefficient β, stepsize η, total number of iterations T. 2:Initialize: π1=πsuperscript1subscriptπ^1= _ refπ1 = πsansserif_ref. 3:Learn a preference model ℓ^^ℓ over start_ARG ℓ end_ARG via least-squares regression: ℓ^=argminℓ∈ℒ∑(x,a+,a−)∈(ℓ(x,a+,a−)−1)2.^ℓsubscriptargminℓℒsubscriptsubscriptsubscriptsubscriptsuperscriptℓsubscriptsubscript12 = argmin_ _(x,a_+,a_% -) _ pref ( (x,a_+,a_-)-1 )^2.over start_ARG ℓ end_ARG = argminroman_ℓ ∈ L ∑( x , a start_POSTSUBSCRIPT + , a- ) ∈ Dsansserif_pref end_POSTSUBSCRIPT ( ℓ ( x , a+ , a- ) - 1 )2 . 4:Collect m samples =(x,a,b)subscriptD_ x=\(x,a,b)\Dsansserif_x = ( x , a , b ) where each sample is drawn i.i.d. from x∼ρ,a∼π(x),b∼π(x)formulae-sequencesimilar-toformulae-sequencesimilar-tosubscriptsimilar-tosubscriptx ρ,a _ ref(x),b _ ref(x)x ∼ ρ , a ∼ πsansserif_ref ( x ) , b ∼ πsansserif_ref ( x ). 5:for t=1,⋯,T1⋯t=1,·s,Tt = 1 , ⋯ , T do 6: Sample bt∼πt(x)similar-tosubscriptsuperscriptb_t π^t(x)bitalic_t ∼ πitalic_t ( x ) and let r^t(x,a)=ℓ^(x,a,bt)superscript^^ℓsubscript r^t(x,a)= (x,a,b_t)over start_ARG r end_ARGt ( x , a ) = over start_ARG ℓ end_ARG ( x , a , bitalic_t ) for all x∈,a∈formulae-sequencex ,a ∈ X , a ∈ A. 7: Compute πt+1=argminπ∈Π∑(x,a,b)∈(4(fπ,πtβ,η(x,a,b))−(r^t(x,a)−r^t(x,b)))2,superscript1subscriptargminΠsubscriptsubscriptsuperscriptsubscript4subscriptsuperscriptsuperscriptsuperscript^superscript^2 π^t+1= argmin_π∈ _(x,a,b)∈% D_ x ( clip_4 (f^β,η_π,π^% t(x,a,b) )-( r^t(x,a)- r^t(x,b)) )^2,πitalic_t + 1 = argminitalic_π ∈ Π ∑( x , a , b ) ∈ D start_POSTSUBSCRIPT sansserif_x end_POSTSUBSCRIPT ( sansserif_clip4 ( fitalic_β , ηitalic_π , πitalic_t ( x , a , b ) ) - ( over start_ARG r end_ARGt ( x , a ) - over start_ARG r end_ARGt ( x , b ) ) )2 , (26) where fπ,πtβ,η(x,a,b)subscriptsuperscriptsuperscriptf^β,η_π,π^t(x,a,b)fitalic_β , ηitalic_π , πitalic_t ( x , a , b ) is defined in Eq. 25. 8:Output: π^=(πtt=1T)^superscriptsubscriptsuperscript1 π= unif(\π^t\_t=1^T)over start_ARG π end_ARG = sansserif_unif ( πitalic_t t = 1T ). Let us now explain the intuition behind the the predictor fπ,π′β,η(x,a,b)subscriptsuperscriptsuperscript′f^β,η_π,π (x,a,b)fitalic_β , ηitalic_π , π′ ( x , a , b ). Suppose that the regression step in 7 learns a predictor that can perfectly model the relative reward, i.e., ∀x,a,b,fπt+1,πtβ,η(x,a,b)=r¯t(x,a)−r¯t(x,b),for-allsubscriptsuperscriptsuperscript1superscriptsuperscript¯superscript¯ ∀ x,a,b, f^β,η_π^t+1,π^t(x,a,b)=% r^t(x,a)- r^t(x,b),∀ x , a , b , fitalic_β , ηitalic_πitalic_t + 1 , πitalic_t ( x , a , b ) = over¯ start_ARG r end_ARGt ( x , a ) - over¯ start_ARG r end_ARGt ( x , b ) , In this case, we can show that the returned policy πt+1superscript1π^t+1πitalic_t + 1 is the optimal policy for the following mixed χ2superscript2χ^2χ2-regularized RL objective: πt+1(x)=argmaxp∈Δ()a∼p[r¯t(x,a)]−βDfχmix(p∥π(x))−βηBx(p,πt),∀x∈,formulae-sequencesuperscript1subscriptargmaxΔsubscriptsimilar-tosuperscript¯subscriptsubscriptsubscriptmixconditionalsubscriptsubscriptsuperscriptfor-all π^t+1(x)= argmax_p∈ (X)% \ E_a p [ r^t(x,a) ]-% β D_f_ _ mix (p\,\|\, _ ref(x) )-% βηB_x(p,π^t) \, ∀ x ,πitalic_t + 1 ( x ) = argmaxitalic_p ∈ Δ ( X ) blackboard_Ea ∼ p [ over¯ start_ARG r end_ARGt ( x , a ) ] - β Ditalic_f start_POSTSUBSCRIPT χ start_POSTSUBSCRIPT mix end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( p ∥ πsansserif_ref ( x ) ) - divide start_ARG β end_ARG start_ARG η end_ARG Bitalic_x ( p , πitalic_t ) , ∀ x ∈ X , (27) where Bx(p,πt)subscriptsuperscriptB_x(p,π^t)Bitalic_x ( p , πitalic_t ) is the Bregman divergence induced by the regularizer p↦Dfχmix(p∥π(x))maps-tosubscriptsubscriptsubscriptmixconditionalsubscriptp D_f_ _ mix (p\,\|\, _ ref(x) )p ↦ Ditalic_f start_POSTSUBSCRIPT χ start_POSTSUBSCRIPT mix end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( p ∥ πsansserif_ref ( x ) ), i.e., Bx(p,q):=Dfχmix(p∥π(x))−Dfχmix(q∥π(x))−⟨∇Dfχmix(q∥π(x)),p−q⟩,∀x∈.formulae-sequenceassignsubscriptsubscriptsubscriptsubscriptmixconditionalsubscriptsubscriptsubscriptsubscriptmixconditionalsubscript∇subscriptsubscriptsubscriptmixconditionalsubscriptfor-all B_x(p,q):=D_f_ _ mix (p\,\|\, _ % ref(x) )-D_f_ _ mix (q\,\|\, _ ref(x)% )- ∇ D_f_ _ mix (q\,\|\, _% ref(x) ),p-q , ∀ x .Bitalic_x ( p , q ) := Ditalic_f start_POSTSUBSCRIPT χ start_POSTSUBSCRIPT mix end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( p ∥ πsansserif_ref ( x ) ) - Ditalic_f start_POSTSUBSCRIPT χ start_POSTSUBSCRIPT mix end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( q ∥ πsansserif_ref ( x ) ) - ⟨ ∇ Ditalic_f start_POSTSUBSCRIPT χ start_POSTSUBSCRIPT mix end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( q ∥ πsansserif_ref ( x ) ) , p - q ⟩ , ∀ x ∈ X . Thus, the algorithm can be understood as running mirror descent on the iteration-dependent loss function −r¯tsuperscript¯- r^t- over¯ start_ARG r end_ARGt, with p↦Dfχmix(p∥π(x))maps-tosubscriptsubscriptsubscriptmixconditionalsubscriptp D_f_ _ mix (p\,\|\, _ ref(x) )p ↦ Ditalic_f start_POSTSUBSCRIPT χ start_POSTSUBSCRIPT mix end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( p ∥ πsansserif_ref ( x ) ) as a per-context regularizer. This technique draws inspiration from Chang et al. (2024), in which the authors apply a similar regularized mirror descent algorithm to learn the optimal policy for the reward-based setting. The motivation for using mixed-χ2superscript2χ^2χ2 regularization is exactly the same as in χPO: we want to ensure that πt+1(a|x)π(a|x)≤1+1βsuperscript1conditionalsubscriptconditional11 π^t+1(a|x) _ ref(a|x)≤ 1+ 1βdivide start_ARG πitalic_t + 1 ( a | x ) end_ARG start_ARG πsansserif_ref ( a | x ) end_ARG ≤ 1 + divide start_ARG 1 end_ARG start_ARG β end_ARG, thereby mitigating overoptimization. 7.3 Theoretical Analysis of Iterative χitalic_χPO We now present our main theoretical guarantees for Iterative χitalic_χPO. We begin by stating a number of statistical assumptions. We first assume that the preference model class contains the ground truth preference function ℓ⋆superscriptℓ⋆ ℓ⋆. Assumption 7.1 (Preference function realizability). The model class ℒLL satisfies ℓ⋆∈ℒsuperscriptℓ⋆ℒ ℓ⋆ ∈ L where ℓ⋆superscriptℓ⋆ ℓ⋆ is the ground truth preference function. In addition, since Algorithm 2 iteratively applies an χPO update, we require that a policy realizability assumption analogous to Assumption 3.1 holds for each of the sub-problems in Eq. 27. Concretely, we make the following assumption. Assumption 7.2 (Policy realizability for general preferences). For any policy π∈Ππ∈ π ∈ Π and ℓ∈ℒℓℒ ℓ ∈ L, the policy class Π Π contains the minimizer of the following regularized RL objective: @depthΔ @everymath @group @set@skewchar @nested@a111(x;ℓ,π):=argmaxp∈Δ()a∼p,b∼π(x)[ℓ(x,a,b)]−βDfχmix(p∥π(x))−βηBx(p,π),∀x∈.formulae-sequenceassign @depthΔ @everymath @group @set@skewchar @nested@a111ℓsubscriptargmaxΔsubscriptformulae-sequencesimilar-tosimilar-toℓsubscriptsubscriptsubscriptmixconditionalsubscriptsubscriptfor-all @depth 1 @everymath @group% @set@skewchar @nested@a 111(x; ,π) = % argmax_p∈ (X) \ E_a% p,b π(x) [ (x,a,b) ]-β D_f_ _ mix% (p\,\|\, _ ref(x) )- βηB_x(p,π) % \, ∀ x .Δ 111 ( x ; ℓ , π ) := argmaxitalic_p ∈ Δ ( X ) blackboard_Ea ∼ p , b ∼ π ( x ) [ ℓ ( x , a , b ) ] - β Ditalic_f start_POSTSUBSCRIPT χ start_POSTSUBSCRIPT mix end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( p ∥ πsansserif_ref ( x ) ) - divide start_ARG β end_ARG start_ARG η end_ARG Bitalic_x ( p , π ) , ∀ x ∈ X . Finally, we require that the implicit reward functions in Eq. 26 are bounded, analogous to Assumption 3.2. Assumption 7.3 (Bounded implicit rewards for general preferences). For a parameter V≥2subscript2V_ max≥ 2Vsansserif_max ≥ 2, it holds that for all π,π′∈Πsuperscript′Ππ,π ∈ π , π′ ∈ Π, x∈x ∈ X, and a,b∈a,b , b ∈ A, |fπ,π′β,η(x,a,b)|≤V.subscriptsuperscriptsuperscript′subscript |f^β,η_π,π (x,a,b) |≤ V_ max.| fitalic_β , ηitalic_π , π′ ( x , a , b ) | ≤ Vsansserif_max . (28) Our main guarantee for Algorithm 2 is as follows. Theorem 7.2. Fix any δ∈(0,1]01δ∈(0,1]δ ∈ ( 0 , 1 ]. Suppose Algorithm 2 is invoked with T=mnV2+msuperscriptsubscript2T= mnnV_ max^2+mT = divide start_ARG m n end_ARG start_ARG n Vsansserif_max2 + m end_ARG, β=1T1β= 1 Tβ = divide start_ARG 1 end_ARG start_ARG square-root start_ARG T end_ARG end_ARG, and η=1T1η= 1Tη = divide start_ARG 1 end_ARG start_ARG T end_ARG. Then under Assumption 7.1, Assumption 7.2 and Assumption 7.3, we have that probability at least 1−δ11- 1 - δ, (π^)≲minC≥1(π^,C)+C(Vlog(|Π|/δ)m+log(|Π||ℒ|/δ)n),less-than-or-similar-to^subscript1^subscriptΠℒ DG( π) _C≥ 1 \ % subopt( π,C)+C ( V_ max (| |/δ)% m+ (| ||L|/δ) n ) \,sansserif_DG ( over start_ARG π end_ARG ) ≲ minitalic_C ≥ 1 sansserif_subopt ( over start_ARG π end_ARG , C ) + C ( divide start_ARG Vsansserif_max log ( | Π | / δ ) end_ARG start_ARG square-root start_ARG m end_ARG end_ARG + divide start_ARG log ( | Π | | L | / δ ) end_ARG start_ARG square-root start_ARG n end_ARG end_ARG ) , where (π^,C):=maxπ∈Πℓ⋆(π,π^)−maxπ∈ΠCℓ⋆(π,π^)assign^subscriptΠsuperscriptℓ⋆^subscriptsubscriptΠsuperscriptℓ⋆ subopt( π,C):= _π∈ (π, % π)- _π∈ _C (π, π)sansserif_subopt ( over start_ARG π end_ARG , C ) := maxitalic_π ∈ Π ℓ⋆ ( π , over start_ARG π end_ARG ) - maxitalic_π ∈ Π start_POSTSUBSCRIPT C end_POSTSUBSCRIPT ℓ⋆ ( π , over start_ARG π end_ARG ) and ΠC:=π:maxx∈Dχ2(π(x)∥π(x))≤CassignsubscriptΠconditional-setsubscriptsubscriptsuperscript2conditionalsubscript _C:=\π: _x D_χ^2 (π(x)\;\|\; _% ref(x) )≤ C\Πitalic_C := π : maxitalic_x ∈ X Ditalic_χ2 ( π ( x ) ∥ πsansserif_ref ( x ) ) ≤ C . In particular, if we define the unilateral concentrability coefficient as C:=maxπ∈Π,x∈,a,b∈π(a∣x)π(b∣x)π(a∣x)π(b∣x),assignsubscriptsubscriptformulae-sequenceΠformulae-sequenceconditionalsubscriptconditionalsubscriptconditionalsubscriptconditional C_ uni:= _π∈ ,x ,a,b∈ % A π(a x) _ MW(b x) _ ref(a % x) _ ref(b x),Csansserif_uni := maxitalic_π ∈ Π , x ∈ X , a , b ∈ A divide start_ARG π ( a ∣ x ) πsansserif_MW ( b ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) πsansserif_ref ( b ∣ x ) end_ARG , then the bound above implies that (π^)≲C⋅(Vlog(|Π|/δ)m+log(|Π||ℒ|/δ)n).less-than-or-similar-to^⋅subscriptsubscriptΠℒ DG( π) C_ uni· (% V_ max (| |/δ) m+ (| ||% L|/δ) n ).sansserif_DG ( over start_ARG π end_ARG ) ≲ Csansserif_uni ⋅ ( divide start_ARG Vsansserif_max log ( | Π | / δ ) end_ARG start_ARG square-root start_ARG m end_ARG end_ARG + divide start_ARG log ( | Π | | L | / δ ) end_ARG start_ARG square-root start_ARG n end_ARG end_ARG ) . The first result gives a tradeoff between the statistical error and the approximation error (π^,C) subopt( π,C)sansserif_subopt ( over start_ARG π end_ARG , C ), which is modulated by the parameter C. This tradeoff is analogous to, but more subtle, than the one for χPO in the reward-based setting. In the reward-based setting, χPO has low regret to the best policy covered πsubscript _ refπsansserif_ref. In the general preference setting, Algorithm 2 has small duality gap if, for any policy, there is an approximate best response that is covered by πsubscript _ refπsansserif_ref (this implies that (π^,C) subopt( π,C)sansserif_subopt ( over start_ARG π end_ARG , C ) is small for small C). Crucially, Algorithm 2 does not require that all policies are covered by πsubscript _ refπsansserif_ref, which is a distinctive feature of mixed χ2superscript2χ^2χ2-regularization and reflects the algorithms robustness to overoptimization. The second result concerns the setting where all policies are covered by πsubscript _ refπsansserif_ref and is easier to interpret. Indeed, if all π∈Ππ∈ π ∈ Π satisfy Dχ2(π∥π)≤C⋆subscriptsuperscript2conditionalsubscriptsuperscript⋆D_χ^2 (π\;\|\; _ ref )≤C Ditalic_χ2 ( π ∥ πsansserif_ref ) ≤ C⋆, then (π^,C⋆)=0^superscript⋆0 subopt( π,C )=0sansserif_subopt ( over start_ARG π end_ARG , C⋆ ) = 0, which implies that we can learn an ε ε-approximate minimizer using O~(C⋆/ε2)~superscript⋆superscript2 O(C / ^2)over~ start_ARG O end_ARG ( C⋆ / ε2 ) samples. Thus, we obtain a guarantee based on unilateral concentrability (Cui and Du, 2022), which is a stronger condition, i.e., we always have maxπDχ2(π∥π)≤Csubscriptsubscriptsuperscript2conditionalsubscriptsubscript _πD_χ^2 (π\;\|\; _ ref )≤ C_% unimaxitalic_π Ditalic_χ2 ( π ∥ πsansserif_ref ) ≤ Csansserif_uni. However, per the above discussion, the first part of Theorem 7.2 is stronger than results based on unilateral concentrability and hints at a new notion of coverage for general preferences. Lastly, we remark that the parameter VsubscriptV_ maxVsansserif_max only affects 1/m1 1/msquare-root start_ARG 1 / m end_ARG term in Theorem 7.2, so dependence on this parameter can be mitigated using unlabeled data. Theorem 7.2 is closely related to recent work of Ye et al. (2024), which uses pessimism to learn a regularized minimax winner, and achieves polynomial sample complexity with a concentrability assumption similar to Theorem 7.2. However, there are two key differences. First, their learning objective is the KL-regularized minimax winner, while we study the unregularized objective and use χ2superscript2χ^2χ2-regularization. More importantly, their theoretical algorithm is computationally inefficient as it constructs an explicit confidence set for the preference model and performs max-min-style policy optimization. In contrast, our algorithm only requires solving standard supervised learning problems. 8 Discussion Our work gives the first practical, general-purpose algorithm for offline alignment with provable robustness to overoptimization and sample complexity guarantees based on single-policy concentrability. Conceptually, our results contribute to a growing body of research that highlights the statistical benefits of χ2superscript2χ^2χ2-divergence for reinforcement learning (Wang et al., 2024; Gabbianelli et al., 2024; Amortila et al., 2024), and offer an example of fruitful interplay between reinforcement learning theory and language modeling. From this perspective, we expect that our analysis techniques and algorithm design ideas will find broader use. Natural technical directions raised by our paper include (i) developing a tight understanding of minimax sample complexity and instance-optimality for offline alignment with general policy classes; (i) understanding the tightest possible problem-dependent sample complexity guarantees for offline alignment with general preference models (in light of lower bounds in Section 7); and (i) extending our techniques to reinforcement learning settings beyond offline alignment (e.g., general MDPs). We look forward to studying these questions in future work. Acknowledgements We thank Qinghua Liu, Zhaolin Gao, and Yuda Song for several helpful discussions. WS acknowledges funding support from NSF IIS-2154711, NSF CAREER 2339395, DARPA LANCER: LeArning Network CybERagents. References Agarwal et al. (2019) Alekh Agarwal, Nan Jiang, and Sham M Kakade. Reinforcement learning: Theory and algorithms. https://rltheorybook.github.io/, 2019. Version: January 31, 2022. Agarwal et al. (2020) Alekh Agarwal, Sham Kakade, Akshay Krishnamurthy, and Wen Sun. FLAMBE: Structural complexity and representation learning of low rank MDPs. Advances in Neural Information Processing Systems, 2020. Amortila et al. (2024) Philip Amortila, Dylan J Foster, and Akshay Krishnamurthy. Scalable online exploration via coverability. International Conference on Machine Learning, 2024. Athey and Wager (2021) Susan Athey and Stefan Wager. Policy learning with observational data. Econometrica, 2021. Azar et al. (2024) Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, 2024. Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv:2204.05862, 2022. Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, 2023. Bradley and Terry (1952) Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 1952. Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020. Cen et al. (2024) Shicong Cen, Jincheng Mei, Katayoon Goshvadi, Hanjun Dai, Tong Yang, Sherry Yang, Dale Schuurmans, Yuejie Chi, and Bo Dai. Value-incentivized preference optimization: A unified approach to online and offline RLHF. arXiv:2405.19320, 2024. Cesa-Bianchi et al. (2017) Nicolò Cesa-Bianchi, Claudio Gentile, Gábor Lugosi, and Gergely Neu. Boltzmann exploration done right. Advances in Neural Information Processing Systems, 2017. Chang et al. (2024) Jonathan D Chang, Wenhao Shan, Owen Oertell, Kianté Brantley, Dipendra Misra, Jason D Lee, and Wen Sun. Dataset reset policy optimization for RLHF. arXiv:2404.08495, 2024. Chen and Jiang (2022) Jinglin Chen and Nan Jiang. Offline reinforcement learning under value and density-ratio realizability: The power of gaps. In Uncertainty in Artificial Intelligence, 2022. Chen et al. (2022) Xiaoyu Chen, Han Zhong, Zhuoran Yang, Zhaoran Wang, and Liwei Wang. Human-in-the-loop: Provably efficient preference-based reinforcement learning with general function approximation. In International Conference on Machine Learning, 2022. Chen et al. (2024) Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv:2401.01335, 2024. Chernozhukov et al. (2019) Victor Chernozhukov, Mert Demirer, Greg Lewis, and Vasilis Syrgkanis. Semi-parametric efficient policy learning with continuous actions. Advances in Neural Information Processing Systems, 2019. Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 2017. Corless et al. (1996) Robert M Corless, Gaston H Gonnet, David EG Hare, David J Jeffrey, and Donald E Knuth. On the Lambert W function. Advances in Computational Mathematics, 1996. Coste et al. (2023) Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. arXiv:2310.02743, 2023. Cui and Du (2022) Qiwen Cui and Simon S Du. When are offline two-player zero-sum Markov games solvable? Advances in Neural Information Processing Systems, 2022. Das et al. (2024) Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano, and Sayak Ray Chowdhury. Provably sample efficient RLHF via active preference optimization. arXiv:2402.10500, 2024. de Geer (2000) Sara A. Van de Geer. Empirical Processes in M-Estimation. Cambridge University Press, 2000. Dong et al. (2023) Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv:2304.06767, 2023. Dong et al. (2024) Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF workflow: From reward modeling to online RLHF. arXiv:2405.07863, 2024. Du et al. (2024) Yihan Du, Anna Winnicki, Gal Dalal, Shie Mannor, and R Srikant. Exploration-driven policy optimization in RLHF: Theoretical insights on efficient data utilization. arXiv:2402.10342, 2024. Duan et al. (2020) Yaqi Duan, Zeyu Jia, and Mengdi Wang. Minimax-optimal off-policy evaluation with linear function approximation. In International Conference on Machine Learning, 2020. Duchi and Namkoong (2019) John Duchi and Hongseok Namkoong. Variance-based regularization with convex objectives. Journal of Machine Learning Research, 2019. Dudík et al. (2015) Miroslav Dudík, Katja Hofmann, Robert E Schapire, Aleksandrs Slivkins, and Masrour Zoghi. Contextual dueling bandits. In Conference on Learning Theory, 2015. Eisenstein et al. (2023) Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D’Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. arXiv:2312.09244, 2023. Farahmand et al. (2010) Amir-massoud Farahmand, Csaba Szepesvári, and Rémi Munos. Error propagation for approximate policy and value iteration. Advances in Neural Information Processing Systems, 2010. Fisch et al. (2024) Adam Fisch, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, and Jonathan Berant. Robust preference optimization through reward model distillation. arXiv:2405.19316, 2024. Fishburn (1984) Peter C Fishburn. Probabilistic social choice based on simple voting comparisons. The Review of Economic Studies, 1984. Foster and Rakhlin (2023) Dylan J Foster and Alexander Rakhlin. Foundations of reinforcement learning and interactive decision making. arXiv:2312.16730, 2023. Gabbianelli et al. (2024) Germano Gabbianelli, Gergely Neu, and Matteo Papini. Importance-weighted offline learning done right. In International Conference on Algorithmic Learning Theory, 2024. Gao et al. (2023) Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, 2023. Gao et al. (2024) Zhaolin Gao, Jonathan D Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, J Andrew Bagnell, Jason D Lee, and Wen Sun. REBEL: Reinforcement learning via regressing relative rewards. arXiv:2404.16767, 2024. Google (2023) Google. Palm 2 technical report. arXiv:2305.10403, 2023. Guo et al. (2024) Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, and Mathieu Blondel. Direct language model alignment from online AI feedback. arXiv:2402.04792, 2024. Huang et al. (2022) Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 2022. Ji et al. (2024) Xiang Ji, Sanjeev Kulkarni, Mengdi Wang, and Tengyang Xie. Self-play with adversarial critic: Provable and scalable offline alignment for language models. arXiv:2406.04274, 2024. Jin et al. (2021) Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline RL? In International Conference on Machine Learning, 2021. Kallus and Uehara (2020) Nathan Kallus and Masatoshi Uehara. Double reinforcement learning for efficient off-policy evaluation in markov decision processes. Journal of Machine Learning Research, 2020. Kramer (1973) Gerald H Kramer. On a class of equilibrium conditions for majority rule. Econometrica: Journal of the Econometric Society, 1973. Kreweras (1965) Germain Kreweras. Aggregation of preference orderings. In Mathematics and Social Sciences I: Proceedings of the seminars of Menthon-Saint-Bernard, France and of Gösing, Austria, 1965. Lattimore and Szepesvári (2020) Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020. Lee et al. (2021) Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning, 2021. Li et al. (2023) Zihao Li, Zhuoran Yang, and Mengdi Wang. Reinforcement learning with human feedback: Learning dynamic choices via pessimism. arXiv:2305.18438, 2023. Liu et al. (2023) Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. arXiv:2309.06657, 2023. Liu et al. (2020) Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch off-policy reinforcement learning without great exploration. Advances in Neural Information Processing Systems, 2020. Liu et al. (2024) Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in RLHF: Your SFT loss is implicitly an adversarial regularizer. arXiv:2405.16436, 2024. Ma et al. (2022a) Jason Yecheng Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. Offline goal-conditioned reinforcement learning via f-advantage regression. Advances in Neural Information Processing Systems, 2022a. Ma et al. (2022b) Yecheng Jason Ma, Andrew Shen, Dinesh Jayaraman, and Osbert Bastani. Smodice: Versatile offline imitation learning via state occupancy matching. arXiv:2202.02433, 2022b. Michaud et al. (2020) Eric J Michaud, Adam Gleave, and Stuart Russell. Understanding learned reward functions. arXiv:2012.05862, 2020. Moskovitz et al. (2023) Ted Moskovitz, Aaditya K Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca D Dragan, and Stephen McAleer. Confronting reward model overoptimization with constrained RLHF. arXiv:2310.04373, 2023. Munos et al. (2023) Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel J. Mankowitz, Doina Precup, and Bilal Piot. Nash learning from human feedback. arXiv:2312.00886, 2023. Novoseller et al. (2020) Ellen Novoseller, Yibing Wei, Yanan Sui, Yisong Yue, and Joel Burdick. Dueling posterior sampling for preference-based reinforcement learning. In Conference on Uncertainty in Artificial Intelligence, 2020. OpenAI (2023) OpenAI. Gpt-4 technical report. arXiv:2303.08774, 2023. Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 2022. Pacchiano et al. (2021) Aldo Pacchiano, Aadirupa Saha, and Jonathan Lee. Dueling RL: Reinforcement learning with trajectory preferences. arXiv:2111.04850, 2021. Pal et al. (2024) Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with DPO-positive. arXiv:2402.13228, 2024. Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 2023. Rafailov et al. (2024a) Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms. arXiv:2406.02900, 2024a. Rafailov et al. (2024b) Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to Q⋆superscript⋆Q Q⋆: Your language model is secretly a Q-function. arXiv:2404.12358, 2024b. Rashidinejad et al. (2021) Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems, 2021. Rita et al. (2024) Mathieu Rita, Florian Strub, Rahma Chaabouni, Paul Michel, Emmanuel Dupoux, and Olivier Pietquin. Countering reward over-optimization in LLM with demonstration-guided reinforcement learning. arXiv:2404.19409, 2024. Rosset et al. (2024) Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct Nash Optimization: Teaching language models to self-improve with general preferences. arXiv:2404.03715, 2024. Shah et al. (2015) Nihar Shah, Sivaraman Balakrishnan, Joseph Bradley, Abhay Parekh, Kannan Ramchandran, and Martin Wainwright. Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependence. In International Conference on Artificial Intelligence and Statistics, 2015. Simpson (1969) Paul B Simpson. On defining areas of voter choice: Professor tullock on stable voting. The Quarterly Journal of Economics, 1969. Song et al. (2022) Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid RL: Using both offline and online data can make RL efficient. arXiv:2210.06718, 2022. Song et al. (2024) Yuda Song, Gokul Swamy, Aarti Singh, J Andrew Bagnell, and Wen Sun. Understanding preference fine-tuning through the lens of coverage. arXiv:2406.01462, 2024. Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33, 2020. Swamy et al. (2024) Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. A minimaximalist approach to reinforcement learning from human feedback. arXiv:2401.04056, 2024. Tajwar et al. (2024) Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of LLMs should leverage suboptimal, on-policy data. arXiv:2404.14367, 2024. Tang et al. (2024) Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. arXiv:2402.05749, 2024. Tien et al. (2022) Jeremy Tien, Jerry Zhi-Yang He, Zackory Erickson, Anca Dragan, and Daniel S Brown. Causal confusion and reward misidentification in preference-based reward learning. In International Conference on Learning Representations, 2022. Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288, 2023. Tsybakov (2008) Alexandre B Tsybakov. Introduction to Nonparametric Estimation. Springer, 2008. Uehara and Sun (2021) Masatoshi Uehara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial coverage. arXiv:2107.06226, 2021. Van Erven and Harremos (2014) Tim Van Erven and Peter Harremos. Rényi divergence and kullback-leibler divergence. IEEE Transactions on Information Theory, 60(7), 2014. von Werra et al. (2020) Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. Trl: Transformer reinforcement learning. https://github.com/huggingface/trl, 2020. Wang et al. (2023a) Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse KL: Generalizing direct preference optimization with diverse divergence constraints. arXiv:2309.16240, 2023a. Wang et al. (2024) Lequn Wang, Akshay Krishnamurthy, and Alex Slivkins. Oracle-efficient pessimism: Offline policy optimization in contextual bandits. In International Conference on Artificial Intelligence and Statistics, 2024. Wang et al. (2023b) Yuanhao Wang, Qinghua Liu, and Chi Jin. Is RLHF more difficult than standard RL? arXiv:2306.14111, 2023b. Wong and Shen (1995) Wing Hung Wong and Xiaotong Shen. Probability inequalities for likelihood ratios and convergence rates of sieve mles. The Annals of Statistics, 1995. Wu and Sun (2023) Runzhe Wu and Wen Sun. Making RL with preference-based feedback efficient via randomization. arXiv:2310.14554, 2023. Wu et al. (2024) Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv:2405.00675, 2024. Xie and Jiang (2020) Tengyang Xie and Nan Jiang. Q* approximation schemes for batch reinforcement learning: A theoretical comparison. In Conference on Uncertainty in Artificial Intelligence, 2020. Xie et al. (2021) Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-consistent pessimism for offline reinforcement learning. Advances in Neural Information Processing Systems, 2021. Xie et al. (2024) Tengyang Xie, Dylan J Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit Q*-approximation for sample-efficient rlhf. arXiv:2405.21046, 2024. Xiong et al. (2023) Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang. Gibbs sampling from human feedback: A provable KL-constrained framework for RLHF. arXiv:2312.11456, 2023. Xu et al. (2020) Yichong Xu, Ruosong Wang, Lin Yang, Aarti Singh, and Artur Dubrawski. Preference-based reinforcement learning with finite-time guarantees. Advances in Neural Information Processing Systems, 2020. Ye et al. (2024) Chenlu Ye, Wei Xiong, Yuheng Zhang, Nan Jiang, and Tong Zhang. A theoretical analysis of Nash learning from human feedback under general KL-regularized preference. arXiv:2402.07314, 2024. Yuan et al. (2024) Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. Advancing llm reasoning generalists with preference trees. arXiv:2404.02078, 2024. Zanette et al. (2021) Andrea Zanette, Martin J Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in Neural Information Processing Systems, 2021. Zhan et al. (2022) Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory, 2022. Zhan et al. (2023a) Wenhao Zhan, Masatoshi Uehara, Nathan Kallus, Jason D Lee, and Wen Sun. Provable offline preference-based reinforcement learning. In International Conference on Learning Representations, 2023a. Zhan et al. (2023b) Wenhao Zhan, Masatoshi Uehara, Wen Sun, and Jason D Lee. Provable reward-agnostic preference-based reinforcement learning. arXiv:2305.18505, 2023b. Zhang (2006) Tong Zhang. From ϵitalic-ϵεϵ-entropy to KL-entropy: Analysis of minimum information complexity density estimation. The Annals of Statistics, 2006. Zhang et al. (2024) Xiaoying Zhang, Jean-Francois Ton, Wei Shen, Hongning Wang, and Yang Liu. Overcoming reward overoptimization via adversarial policy optimization with lightweight uncertainty estimation. arXiv:2403.05171, 2024. Zhu et al. (2023) Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning, 2023. Zhu et al. (2024) Banghua Zhu, Michael I Jordan, and Jiantao Jiao. Iterative data smoothing: Mitigating reward overfitting and overoptimization in RLHF. arXiv:2401.16335, 2024. Zhu and Zhang (2024) Hanlin Zhu and Amy Zhang. Provably efficient offline goal-conditioned reinforcement learning with general function approximation and single-policy concentrability. Advances in Neural Information Processing Systems, 2024. Zhu et al. (2020) Zhuangdi Zhu, Kaixiang Lin, Bo Dai, and Jiayu Zhou. Off-policy imitation learning from observations. Advances in Neural Information Processing Systems, 2020. Contents of Appendix 1 Introduction 1.1 Contributions 1.2 Paper Organization 2 Background 2.1 Alignment from Human Feedback 2.2 Overoptimization and Insufficiency of KL-Regularization 3 superscript2 χ^2italic_χbold_2-Preference Optimization 3.1 Framework: superscript2 χ^2italic_χbold_2-Regularized Reward Optimization 3.2 The χitalic_χPO Algorithm 3.3 Theoretical Guarantees 4 Understanding χitalic_χPO: The Bias-Overoptimization Tradeoff 4.1 Properties of Optimal Policy under Mixed superscript2 χ^2italic_χbold_2-Regularization 4.2 The Bias-Overoptimization Tradeoff 4.3 An Illustrative Example 4.4 Nontriviality and Role of VsubscriptV_ maxVsansserif_max Parameter 5 Analysis of χitalic_χPO: Proof Sketch for :main 6 Experiments in Offline Language Model Alignment 7 χitalic_χPO for General Preference Models 7.1 Impossibility of Single-Policy Concentrability under General Preferences 7.2 Iterative χitalic_χPO for General Preferences 7.3 Theoretical Analysis of Iterative χitalic_χPO 8 Discussion I Additional Results A Additional Related Work A.1 Detailed Comparison to DPO + SFT B Sample Complexity Guarantees for superscript2 χ^2italic_χbold_2-RLHF C Experiment details D Applying χitalic_χPO to the Token-Level MDP I Proofs E Preliminaries F Proofs for :main F.1 General Version of :main F.1.1 Proofs for Supporting Lemmas F.2 Proof of :main F.3 Proof of :reward_model G Proofs for :understanding H Proofs for :general_preference H.1 Proof of Theorem 7.1 H.2 Proof of Theorem 7.2 H.3 Proofs for Supporting Lemmas I Proofs for :rlhf Part I Additional Results Appendix A Additional Related Work Theoretical algorithms for offline alignment Much of prior theoretical work on offline alignment considers algorithms that are tailored to linearly parameterized policies (Zhu et al., 2023; Li et al., 2023; Xiong et al., 2023), while others are not efficiently implementable, e.g., as they require solving min-max problems over a version space (Zhan et al., 2023a). For general policy classes, Ye et al. (2024) provide an algorithm that achieves sample complexity guarantees based on single-policy concentrability, but the algorithm requires computation of an uncertainty bonus which cannot be implemented faithfully for large language models. Ji et al. (2024) provide an algorithm that achieves single-policy concentrability using self-play, but their approach requires the non-standard realizability assumption that for all π∈Ππ∈ π ∈ Π, there exists π′∈Πsuperscript′Ππ ∈ π′ ∈ Π such that r(x,a)=βlogπ(a∣x)π′(a∣x)−Zπ,π′(x)conditionalsuperscript′conditionalsubscriptsuperscript′r(x,a)=β π(a x)π (a x)-Z_π,π^% (x)r ( x , a ) = β log divide start_ARG π ( a ∣ x ) end_ARG start_ARG π′ ( a ∣ x ) end_ARG - Zitalic_π , π′ ( x ) for some function Zπ,π′(x)subscriptsuperscript′Z_π,π (x)Zitalic_π , π′ ( x ) that depends on x, but not the action a. In addition, their algorithm is iterative, and requires solving a DPO-like objective many times (roughly 1/ε21superscript21/ ^21 / ε2 iterations are required to achieve accuracy ε ε). Most relevant to our work, Liu et al. (2024); Cen et al. (2024); Fisch et al. (2024) propose solving the appealingly simple DPO + SFT objective in Eq. 5. As we discuss in detail in Section A.1, this objective fails to achieve single-policy concentrability unless non-standard convexity assumptions on the policy class or reward model class hold. A number of other works consider the hybrid setting for alignment where—in addition to offline preference data from πsubscript _ refπsansserif_ref, the algorithm has access to online feedback (Xiong et al., 2023; Gao et al., 2024; Chang et al., 2024; Song et al., 2024). While it is straightforward to achieve guarantees based on single-policy concentrability in this setting, this is a stronger feedback model than what we consider, and is not always realistic. Our work is also complementary to fully online alignment, which dispenses with coverage conditions entirely but requires active exploration (Xu et al., 2020; Novoseller et al., 2020; Pacchiano et al., 2021; Wu and Sun, 2023; Zhan et al., 2023b; Chen et al., 2022; Wang et al., 2023b; Du et al., 2024; Das et al., 2024; Ye et al., 2024; Xie et al., 2024; Cen et al., 2024). Generalizations of DPO Wang et al. (2023a) provide a generalization of the DPO reparameterization trick which supports general f-divergences that satisfy certain regularity conditions. Their work does not provide sample complexity guarantees or theoretical guidance on which choices of f-divergence are preferable, but our main algorithm χPO, can be derived as a special case of their technique with a novel choice of f-divergence. Tang et al. (2024) also provide a general framework for deriving DPO variants with general loss functions, but our algorithm does not appear to be a special case of their framework. Offline reinforcement learning theory The theory of offline reinforcement learning addresses challenges similar to overoptimization, which is typically describes through the language of distribution shift. Many of these works, using pessimism and related algorithmic techniques, provide guarantees that are robust to partial coverage of the data collection policy πsubscript _ refπsansserif_ref, which is reflected in sample complexity guarantees based on single-policy concentrability and similar coverage conditions. While this line of work provides efficient algorithms for simple (e.g., tabular or linear) settings (Liu et al., 2020; Jin et al., 2021; Rashidinejad et al., 2021), existing approaches that support general function approximation (Xie et al., 2021; Uehara and Sun, 2021; Zhan et al., 2022; Chen and Jiang, 2022) cannot be implemented efficiently for language models without non-trivial modifications. See also closely related research on policy optimization and evaluation in statistics and econometrics (Athey and Wager, 2021; Chernozhukov et al., 2019; Kallus and Uehara, 2020). χ2superscript2χ^2χ2-divergence in reinforcement learning Our work contributes to a growing body of research that uses χ2superscript2χ^2χ2-divergence to derive reinforcement learning algorithms with novel statistical guarantees.777More classically, χ2superscript2χ^2χ2-divergence is known to play a fundamental role in asymptotic statistics (Tsybakov, 2008; Duchi and Namkoong, 2019). Notably, our work is inspired by Wang et al. (2024) (see also Gabbianelli et al. (2024)), who use a regularizer similar to χ2superscript2χ^2χ2-divergence to derive single-policy concentrability guarantees for contextual bandits. Compared to the χ2superscript2χ^2χ2-regularizer π=π[π(a∣x)π(a∣x)]superscriptsubscriptconditionalsubscriptconditionalC^π= E_π [ π(a x)% _ ref(a x) ]Citalic_π = blackboard_Eπ [ divide start_ARG π ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ] we use, their regularizer takes the form π[1π(a∣x)]subscript1subscriptconditional E_π [ 1 _ ref(a x) ]blackboard_Eπ [ divide start_ARG 1 end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ], which is always larger. As a result of this diference, their regularizer is not suitable for large action spaces. By addressing this shortcoming, we expect our χ2superscript2χ^2χ2-regularization approach to find further use in offline RL. Other related works include (i) Duan et al. (2020) show that χ2superscript2χ^2χ2-divergence plays a fundamental role in offline RL with linear function approximation; (i) Zhan et al. (2022) use χ2superscript2χ^2χ2-regularization to provide guarantees based on single-policy concentrability for an offline RL method based on weight function learning; and (i) Amortila et al. (2024) provide online RL algorithms that explore by directly minimizing an exploration objective based on χ2superscript2χ^2χ2-divergence. We mention in passing that a number of recent empirical works apply χ2superscript2χ^2χ2-regularization (Zhu et al., 2020; Lee et al., 2021; Ma et al., 2022a, b; Zhu and Zhang, 2024) to reinforcement learning in embodied domains. Lastly, Cesa-Bianchi et al. (2017) prove lower bounds against the softmax policy distribution, but in the context of online exploration for online RL. While this is different problem setting than ours, their construction may be in similar in spirit to our lower bound against KL-regularization in offline reinforcement learning (Proposition A.1). Empirical research on offline alignment Our work uses DPO (Rafailov et al., 2023) as a starting point. Many prior works have built upon DPO with the aim of addressing specific shortcomings, including Liu et al. (2023); Tang et al. (2024); Azar et al. (2024); Rosset et al. (2024); Chen et al. (2024); Wu et al. (2024); Tajwar et al. (2024). Closely related, there is a large body of research that attempts to understand and mitigate overoptimization in offline alignment from a purely empirical perspective (Michaud et al., 2020; Tien et al., 2022; Coste et al., 2023; Dong et al., 2023; Eisenstein et al., 2023; Gao et al., 2023; Moskovitz et al., 2023; Pal et al., 2024; Rita et al., 2024; Rafailov et al., 2024a; Zhang et al., 2024). A.1 Detailed Comparison to DPO + SFT In this section, we give additional background on the suboptimality of the DPO + SFT objective in Eq. 5. Let β>00β>0β > 0 be the KL-regularization parameter and α>00α>0α > 0 be an optimism parameter. Consider the setting in which Π=πr(a∣x)=π(a∣x)exp(β−1(r(x,a)−Zr(x)))∣r∈ℛΠsubscriptconditionalconditionalsubscriptconditionalsuperscript1subscriptℛ = \ _r(a x)= _ ref(a x) (β^-1(r(x,a% )-Z_r(x))) r \Π = πitalic_r ( a ∣ x ) = πsansserif_ref ( a ∣ x ) exp ( β- 1 ( r ( x , a ) - Zitalic_r ( x ) ) ) ∣ r ∈ R for a reward class ℛ⊂(×→ℝ)ℛ→ℝR⊂(X×A )R ⊂ ( X × A → blackboard_R ). Liu et al. (2024); Cen et al. (2024); Fisch et al. (2024) propose solving (variants of) the objective π^max-min=argmaxπminr∈ℛα(x∼ρ,a∼π(⋅∣x),b∼π(⋅∣x)[r(a)−r(b)]−βD(π∥π))+ℒ(r), π_ max-min= argmax_π% _r \α ( E_x ρ% ,a π(· x),b _ ref(· x) [r(a)-r(b)% ]-β D_ KL (π\,\|\, _ ref ) )+% L(r) \,over start_ARG π end_ARGmax-min = argmaxitalic_π minitalic_r ∈ R α ( blackboard_Ex ∼ ρ , a ∼ π ( ⋅ ∣ x ) , b ∼ π start_POSTSUBSCRIPT sansserif_ref ( ⋅ ∣ x ) end_POSTSUBSCRIPT [ r ( a ) - r ( b ) ] - β Dsansserif_KL ( π ∥ πsansserif_ref ) ) + L ( r ) , (29) where the max ranges over the space of all policies, and where ℒ(r):=−1n∑(x,a+,a−)∈logσ[r(x,a+)−r(x,a−)]assignℒ1subscriptsubscriptsubscriptsubscriptdelimited-[]subscriptsubscriptL(r) =- 1n _(x,a_+,a_-) _% pref σ [r(x,a_+)-r(x,a_-) ]L ( r ) := - divide start_ARG 1 end_ARG start_ARG n end_ARG ∑( x , a start_POSTSUBSCRIPT + , a- ) ∈ Dsansserif_pref end_POSTSUBSCRIPT log σ [ r ( x , a+ ) - r ( x , a- ) ] is the negative log-likelihood under the Bradley-Terry model. Liu et al. (2024) show that for general policy classes, this algorithm attains sample complexity guarantees scaling with single-policy concentrability; Cen et al. (2024) provide similar results for the special case of linearly parameterized policies. The objective in Eq. 29 is non-trivial to implement for language models. To derive the DPO + SFT objective in Eq. 5, Liu et al. (2024) observe that if ℛRR is convex, the minimax theorem implies that the objective value in Eq. 29 is equivalent to the value for the min-max objective minr∈ℛmaxπα(x∼ρ,a∼π(⋅∣x),b∼π(⋅∣x)[r(a)−r(b)]−βD(π∥π))+ℒ(r). _r _π \α (% E_x ρ,a π(· x),b _ ref(·% x) [r(a)-r(b) ]-β D_ KL (π\,\|\, _% ref ) )+L(r) \.minitalic_r ∈ R maxitalic_π α ( blackboard_Ex ∼ ρ , a ∼ π ( ⋅ ∣ x ) , b ∼ π start_POSTSUBSCRIPT sansserif_ref ( ⋅ ∣ x ) end_POSTSUBSCRIPT [ r ( a ) - r ( b ) ] - β Dsansserif_KL ( π ∥ πsansserif_ref ) ) + L ( r ) . (30) This leads to a natural algorithmic strategy adopted by (Liu et al., 2024; Cen et al., 2024; Fisch et al., 2024): Let r^min-maxsubscript^min-max r_ min-maxover start_ARG r end_ARGmin-max be the minimizing reward function in Eq. 30 and let πr^min-maxsubscriptsubscript^min-max _ r_ min-maxπover start_ARG r end_ARG start_POSTSUBSCRIPT min-max end_POSTSUBSCRIPT—the optimal policy in the KL-regularized MDP with reward function r^min-maxsubscript^min-max r_ min-maxover start_ARG r end_ARGmin-max—be the final policy returned by the algorithm. After standard manipulations, one can then show that πr^min-maxsubscriptsubscript^min-max _ r_ min-maxπover start_ARG r end_ARG start_POSTSUBSCRIPT min-max end_POSTSUBSCRIPT is equivalent to argmaxπ∈Πα⋅π[βlogπ(a∣x)]+1n∑(x,a+,a−)∈log[σ(βlogπ(a+∣x)π(a+∣x)−βlogπ(a−∣x)π(a−∣x))].subscriptargmaxΠ⋅subscriptsubscriptconditional1subscriptsubscriptsubscriptsubscriptconditionalsubscriptsubscriptconditionalsubscriptconditionalsubscriptsubscriptconditionalsubscript argmax_π∈ \α·% E_ _ ref [β π(a x)% ]+ 1n _(x,a_+,a_-) _ pref % [σ (β π(a_+ x) _ ref(a_+% x)-β π(a_- x) _ ref(a_- x)% ) ] \.argmaxitalic_π ∈ Π α ⋅ blackboard_Eπ start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT [ β log π ( a ∣ x ) ] + divide start_ARG 1 end_ARG start_ARG n end_ARG ∑( x , a start_POSTSUBSCRIPT + , a- ) ∈ Dsansserif_pref end_POSTSUBSCRIPT log [ σ ( β log divide start_ARG π ( a+ ∣ x ) end_ARG start_ARG πsansserif_ref ( a+ ∣ x ) end_ARG - β log divide start_ARG π ( a- ∣ x ) end_ARG start_ARG πsansserif_ref ( a- ∣ x ) end_ARG ) ] . (31) We call this policy π^DPO+SFTsubscript^DPO+SFT π_ DPO+SFTover start_ARG π end_ARGDPO+SFT. The sample complexity analyses for the π^DPO+SFTsubscript^DPO+SFT π_ DPO+SFTover start_ARG π end_ARGDPO+SFT policy (Eq. 31) in (Liu et al., 2024; Cen et al., 2024) rely on showing that the objective value in Eq. 30 is equivalent to the value in Eq. 29, which is not guaranteed to hold if ℛRR is non-convex (e.g., if ℛRR is a class of neural networks).888Precisely, Liu et al. (2024) provide guarantees for π^max-minsubscript^max-min π_ max-minover start_ARG π end_ARGmax-min with general reward class ℛRR and establish equivalence of π^max-minsubscript^max-min π_ max-minover start_ARG π end_ARGmax-min and π^min-maxsubscript^min-max π_ min-maxover start_ARG π end_ARGmin-max when ℛRR is convex, while Cen et al. (2024) consider linear function approximation, which yields the required convexity. Indeed, the following proposition shows that, for non-convex reward classes ℛRR, the DPO + SFT objective in Eq. 31 fails to achieve a statistical guarantee based on single-policy concentrability, even when Eq. 29 succeeds. Proposition A.1. Let n∈ℕn ∈ blackboard_N with n≥22n≥ 2n ≥ 2 be given. There exists a reward class ℛRR with |ℛ|=2ℛ2|R|=2| R | = 2, a problem instance (ρ,r)(ρ,r)( ρ , r ) satisfying realizability (r∈ℛr ∈ R) and r∈[0,1]01r∈[0,1]r ∈ [ 0 , 1 ], a data collection policy πsubscript _ refπsansserif_ref, and universal constants c1∈(0,1)subscript101c_1∈(0,1)c1 ∈ ( 0 , 1 ) and c2,c3>0subscript2subscript30c_2,c_3>0c2 , c3 > 0 such that the following hold: 1. There exists a policy π~~ πover~ start_ARG π end_ARG such that ‖π~/π‖∞≤2subscriptnorm~subscript2\| π/ _ ref\|_∞≤ 2∥ over~ start_ARG π end_ARG / πsansserif_ref ∥∞ ≤ 2; yet 2. For any β≤(2log(n))−1superscript21β≤(2 (n))^-1β ≤ ( 2 log ( n ) )- 1 and α≥00α≥0α ≥ 0, the minimax policy π^min-maxsubscript^min-max π_ min-maxover start_ARG π end_ARGmin-max (Eq. 30) and DPO+SFT policy π^DPO+SFTsubscript^DPO+SFT π_ DPO+SFTover start_ARG π end_ARGDPO+SFT (Eq. 31) derived from a dataset subscriptD_ prefDsansserif_pref of n samples from πsubscript _ refπsansserif_ref incur suboptimality J(π~)−J(π^DPO+SFT)=J(π~)−J(π^min-max)≥c2,~subscript^DPO+SFT~subscript^min-maxsubscript2 J( π)-J( π_ DPO+SFT)=J(% π)-J( π_ min-max)≥ c_2,J ( over~ start_ARG π end_ARG ) - J ( over start_ARG π end_ARGDPO+SFT ) = J ( over~ start_ARG π end_ARG ) - J ( over start_ARG π end_ARGmin-max ) ≥ c2 , with probability at least c1subscript1c_1c1. 3. For any β≥(2log(n))−1superscript21β≥(2 (n))^-1β ≥ ( 2 log ( n ) )- 1 and α≥00α≥ 0α ≥ 0, the minimax policy π^min-maxsubscript^min-max π_ min-maxover start_ARG π end_ARGmin-max (Eq. 30) and DPO+SFT policy π^DPO+SFTsubscript^DPO+SFT π_ DPO+SFTover start_ARG π end_ARGDPO+SFT (Eq. 31) derived from a dataset subscriptD_ prefDsansserif_pref of n samples from πsubscript _ refπsansserif_ref incur suboptimality J(π~)−J(π^DPO+SFT)=J(π~)−J(π^min-max)≥c3log(n),~subscript^DPO+SFT~subscript^min-maxsubscript3 J( π)-J( π_ DPO+SFT)=J(% π)-J( π_ min-max)≥ c_3 (n),J ( over~ start_ARG π end_ARG ) - J ( over start_ARG π end_ARGDPO+SFT ) = J ( over~ start_ARG π end_ARG ) - J ( over start_ARG π end_ARGmin-max ) ≥ divide start_ARG c3 end_ARG start_ARG log ( n ) end_ARG , with probability at least c1subscript1c_1c1. On the other hand, we observe that for the instance in Proposition A.1, χPO (via Theorem 3.1) with β∝1/nproportional-to1β 1/ nβ ∝ 1 / square-root start_ARG n end_ARG and the class Π=π(a∣x)=π(a∣x)⋅ϕ−1(β−1(r(x,a)−Zr(x)))∣r∈ℛΠconditionalconditional⋅subscriptconditionalsuperscriptitalic-ϕ1superscript1subscriptℛ = \π(a x)= _ ref(a x)·φ^-1(β^-1% (r(x,a)-Z_r(x))) r \Π = π ( a ∣ x ) = πsansserif_ref ( a ∣ x ) ⋅ ϕ- 1 ( β- 1 ( r ( x , a ) - Zitalic_r ( x ) ) ) ∣ r ∈ R achieves J(π~)−J(π^)≲(π~)2n≲1n,less-than-or-similar-to~^superscriptsuperscript~2less-than-or-similar-to1J( π)-J( π) (C^% π)^2n 1n,J ( over~ start_ARG π end_ARG ) - J ( over start_ARG π end_ARG ) ≲ square-root start_ARG divide start_ARG ( Cover~ start_ARG π end_ARG )2 end_ARG start_ARG n end_ARG end_ARG ≲ square-root start_ARG divide start_ARG 1 end_ARG start_ARG n end_ARG end_ARG , highlighting the fact that χPO meaningfully adapts to single-policy concentrability even when the technical conditions required by DPO+SFT do not hold; see also Section 4. We find this conclusion to be somewhat surprising, as Xie et al. (2024) show that an optimistic counterpart to Eq. 31, which negates the SFT term, enjoys strong guarantees for online alignment with general policy classes without requiring convexity. Although our construction does not establish inconsistency in the β≥(2log(n))−1superscript21β≥(2 (n))^-1β ≥ ( 2 log ( n ) )- 1 regime, in general, DPO+SFT will incur O(β)O(β)O ( β ) bias if one aims to compete with the optimal policy. Due to restriction that β must be rather large, this results in an exponentially slower rate of convergence than χPO. Proof of Proposition A.1. Let n∈ℕn ∈ blackboard_N with n≥22n≥2n ≥ 2 be given. We consider a problem instance with =x1,x2subscript1subscript2X=\x_1,x_2\X = x1 , x2 and =a0,a1,a2,a3subscript0subscript1subscript2subscript3A= \a_0,a_1,a_2,a_3 \A = a0 , a1 , a2 , a3 , so that ||=44 =4| A | = 4. We define a reward class with two reward functions ℛ:=r1,r2assignℛsubscript1subscript2R:=\r_1,r_2\R := r1 , r2 as follows. For i∈1,212i∈\1,2\i ∈ 1 , 2 : ri(x1,a0)=ζ,andri(x1,a1)=ri(x1,a2)=ri(x1,a3)=0formulae-sequencesubscriptsubscript1subscript0andsubscriptsubscript1subscript1subscriptsubscript1subscript2subscriptsubscript1subscript30 r_i(x_1,a_0)=ζ, r_i(x_1,a_1)% =r_i(x_1,a_2)=r_i(x_1,a_3)=0ritalic_i ( x1 , a0 ) = ζ , and ritalic_i ( x1 , a1 ) = ritalic_i ( x1 , a2 ) = ritalic_i ( x1 , a3 ) = 0 ri(x2,a0)=1/2,ri(x2,ai)=1,andri(x2,aj)=0∀j≠i.formulae-sequencesubscriptsubscript2subscript012formulae-sequencesubscriptsubscript2subscript1andsubscriptsubscript2subscript0for-all r_i(x_2,a_0)=1/2, r_i(x_2,a_i)=1, % r_i(x_2,a_j)=0\;\;∀ j≠ i.ritalic_i ( x2 , a0 ) = 1 / 2 , ritalic_i ( x2 , aitalic_i ) = 1 , and ritalic_i ( x2 , aitalic_j ) = 0 ∀ j ≠ i . Here ζ∈[0,1]01ζ∈[0,1]ζ ∈ [ 0 , 1 ] will be chosen at the end of the proof. The context distribution is ρ=()ρ= unif(X)ρ = sansserif_unif ( X ), and we define πsubscript _ refπsansserif_ref for each xi∈x1,x2subscriptsubscript1subscript2x_i∈\x_1,x_2\xitalic_i ∈ x1 , x2 via π(a0∣xi)=1/2,π(a1∣xi)=π(a2∣xi)=1/(2n),andπ(a3∣xi)=(n−2)/(2n).formulae-sequenceformulae-sequencesubscriptconditionalsubscript0subscript12subscriptconditionalsubscript1subscriptsubscriptconditionalsubscript2subscript12andsubscriptconditionalsubscript3subscript22 _ ref(a_0 x_i)=1/2, _ ref(a% _1 x_i)= _ ref(a_2 x_i)=1/(2n), % _ ref(a_3 x_i)=(n-2)/(2n).πsansserif_ref ( a0 ∣ xitalic_i ) = 1 / 2 , πsansserif_ref ( a1 ∣ xitalic_i ) = πsansserif_ref ( a2 ∣ xitalic_i ) = 1 / ( 2 n ) , and πsansserif_ref ( a3 ∣ xitalic_i ) = ( n - 2 ) / ( 2 n ) . Let r1subscript1r_1r1 be the true reward function. Recall that =(x,a+,a−)subscriptsubscriptsubscriptD_ pref= \(x,a_+,a_-) \Dsansserif_pref = ( x , a+ , a- ) consists of n tuples (x,a+,a−)subscriptsubscript(x,a_+,a_-)( x , a+ , a- ) obtained by sampling x∼ρsimilar-tox ∼ ρ and a pair of actions (a,b)∼πsimilar-tosubscript(a,b) _ ref( a , b ) ∼ πsansserif_ref and labeling them as (a+,a−)subscriptsubscript(a_+,a_-)( a+ , a- ) via the Bradley-Terry model in Eq. 1 with reward r1subscript1r_1r1. Define a “bad” event under this process: ℰ:=No tuples in contain a1 or a2.assignℰNo tuples in contain a1 or a2 = \No tuples in $ D_% pref$ contain $a_1$ or $a_2$ \.E := No tuples in Dsansserif_pref contain a1 or a2 . We can lower bound the probability of ℰEE as follows: ℙ[ℰc]ℙdelimited-[]superscriptℰc [E^c]blackboard_P [ Eroman_c ] ≤ℙ[a1 in ]+ℙ[a2 in ]absentℙdelimited-[]a1 in ℙdelimited-[]a2 in [$a_1$ in $D_ pref$]+% P[$a_2$ in $D_ pref$]≤ blackboard_P [ a1 in Dsansserif_pref ] + blackboard_P [ a2 in Dsansserif_pref ] =2(1−(1−1/2n)n)≤2(1−e−1/2(1−1/(4n)))≤2(1−7e−1/2/8)≤0.94,absent21superscript11221superscript12114217superscript1280.94 =2(1-(1-1/2n)^n)≤2(1-e^-1/2(1-1/(4n)))≤ 2(1-7e^-1/2% /8)≤ 0.94,= 2 ( 1 - ( 1 - 1 / 2 n )n ) ≤ 2 ( 1 - e- 1 / 2 ( 1 - 1 / ( 4 n ) ) ) ≤ 2 ( 1 - 7 e- 1 / 2 / 8 ) ≤ 0.94 , where the first inequality uses that (1−x/n)n≥e−x(1−x2/n)superscript1superscript1superscript2(1-x/n)^n≥ e^-x(1-x^2/n)( 1 - x / n )n ≥ e- x ( 1 - x2 / n ) for n≥11n≥ 1n ≥ 1 and |x|<n|x|<n| x | < n. We conclude that ℙ[ℰ]≥0.06=:c1. [E]≥ 0.06=:c_1.blackboard_P [ E ] ≥ 0.06 = : c1 . Let ℒ(r;):=−1n∑(x,a+,a−)∈logσ[r(x,a+)−r(x,a−)]assignℒsubscript1subscriptsubscriptsubscriptsubscriptdelimited-[]subscriptsubscriptL(r;D_ pref) =- 1n _(x,a% _+,a_-) _ pref σ [r(x,a_+)-r(x,a_-% ) ]L ( r ; Dsansserif_pref ) := - divide start_ARG 1 end_ARG start_ARG n end_ARG ∑( x , a start_POSTSUBSCRIPT + , a- ) ∈ Dsansserif_pref end_POSTSUBSCRIPT log σ [ r ( x , a+ ) - r ( x , a- ) ] denote the DPO loss. Observe that conditioned on ℰEE, we have that ℒ(r1;)=ℒ(r2;)ℒsubscript1subscriptℒsubscript2subscriptL(r_1;D_ pref)=L(r_2;D_% pref)L ( r1 ; Dsansserif_pref ) = L ( r2 ; Dsansserif_pref ). Noting that maxππ[r]−π[r]−βD(π∥π)=πr[r]−π[r]−βD(πr∥π),subscriptsubscriptdelimited-[]subscriptsubscriptdelimited-[]subscriptconditionalsubscriptsubscriptsubscriptdelimited-[]subscriptsubscriptdelimited-[]subscriptconditionalsubscriptsubscript _π \E_π[r]-E_ _ ref% [r]-β D_ KL (π\,\|\, _ ref ) \=% E_ _r[r]-E_ _ ref[r]-β D_ KL% ( _r\,\|\, _ ref ),maxitalic_π blackboard_Eπ [ r ] - blackboard_Eπ start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT [ r ] - β Dsansserif_KL ( π ∥ πsansserif_ref ) = blackboard_Eπ start_POSTSUBSCRIPT r end_POSTSUBSCRIPT [ r ] - blackboard_Eπ start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT [ r ] - β Dsansserif_KL ( πitalic_r ∥ πsansserif_ref ) , is the same for both r∈ℛr ∈ R, we see that both r1subscript1r_1r1 and r2subscript2r_2r2 optimize the minimax objective in Eq. 30. Thus, breaking ties adversarially, we can choose π^min-max=πr2subscript^min-maxsubscriptsubscript2 π_ min-max= _r_2over start_ARG π end_ARGmin-max = πitalic_r start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT under ℰEE for all values of β>00β>0β > 0 and α≥00α≥0α ≥ 0. By the equivalence between the minimax objective in Eq. 30 and the DPO+SFT objective in Eq. 31 (Liu et al., 2024; Cen et al., 2024; Fisch et al., 2024), for Π=πr1,πr2Πsubscriptsubscript1subscriptsubscript2 = \ _r_1, _r_2 \Π = πitalic_r start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , πitalic_r start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , we can choose π^DPO+SFT=πr2subscript^DPO+SFTsubscriptsubscript2 π_ DPO+SFT= _r_2over start_ARG π end_ARGDPO+SFT = πitalic_r start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT in Eq. 31 under ℰEE. Indeed, under ℰEE, the DPO+SFT objective is equivalent to argmaxπ∈Ππ[logπ(a)]subscriptargmaxΠsubscriptsubscript argmax_π∈ E_ _ % ref [ π(a) ]argmaxitalic_π ∈ Π blackboard_Eπ start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT [ log π ( a ) ], and πr1subscriptsubscript1 _r_1πitalic_r start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT and πr2subscriptsubscript2 _r_2πitalic_r start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT have the same value for this objective. To conclude we choose π~(⋅)=a0~⋅subscript0 π(·)=a_0over~ start_ARG π end_ARG ( ⋅ ) = a0, which has ‖π~/π‖∞=2subscriptnorm~subscript2 \| π/ _ ref \|_∞=2∥ over~ start_ARG π end_ARG / πsansserif_ref ∥∞ = 2. It remains to calculate the suboptimality gap. J(π~)−J(π^DPO+SFT)=J(π~)−J(π^min-max)=J(π~)−J(πr2)~subscript^DPO+SFT~subscript^min-max~subscriptsubscript2J( π)-J( π_ DPO+SFT)=J( π)-J(% π_ min-max)=J( π)-J( _r_2)J ( over~ start_ARG π end_ARG ) - J ( over start_ARG π end_ARGDPO+SFT ) = J ( over~ start_ARG π end_ARG ) - J ( over start_ARG π end_ARGmin-max ) = J ( over~ start_ARG π end_ARG ) - J ( πitalic_r start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ) under ℰEE. Note that J(π~)=ζ/2+1/4~214J( π)=ζ/2+1/4J ( over~ start_ARG π end_ARG ) = ζ / 2 + 1 / 4. We decompose the reward for πr2subscriptsubscript2 _r_2πitalic_r start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT on instance r1subscript1r_1r1 into two components, corresponding to the two contexts x1,x2subscript1subscript2x_1,x_2x1 , x2: J(πr2)subscriptsubscript2 J( _r_2)J ( πitalic_r start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ) =12(a∼πr2[r1(x1,a)]+a∼πr2[r1(x2,a)])=:12(J1(β)+J2(β)) = 12 (E_a _r_2[r_1(x_1,a)]+% E_a _r_2[r_1(x_2,a)] )= 12% (J_1(β)+J_2(β) )= divide start_ARG 1 end_ARG start_ARG 2 end_ARG ( blackboard_Ea ∼ π start_POSTSUBSCRIPT r start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_POSTSUBSCRIPT [ r1 ( x1 , a ) ] + blackboard_Ea ∼ π start_POSTSUBSCRIPT r start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_POSTSUBSCRIPT [ r1 ( x2 , a ) ] ) = : divide start_ARG 1 end_ARG start_ARG 2 end_ARG ( J1 ( β ) + J2 ( β ) ) J1(β)subscript1 J_1(β)J1 ( β ) =r1(x1,a0)π(a0∣x1)exp(r2(x1,a0)/β)Z(r2,x1)=ζ/2exp(ζ/β)1/2exp(ζ/β)+1/2absentsubscript1subscript1subscript0subscriptconditionalsubscript0subscript1subscript2subscript1subscript0subscript2subscript121212 = r_1(x_1,a_0) _ ref(a_0 x_1) (% r_2(x_1,a_0)/β)Z(r_2,x_1)= ζ/2 (ζ/β)1/2% (ζ/β)+1/2= divide start_ARG r1 ( x1 , a0 ) πsansserif_ref ( a0 ∣ x1 ) exp ( r2 ( x1 , a0 ) / β ) end_ARG start_ARG Z ( r2 , x1 ) end_ARG = divide start_ARG ζ / 2 exp ( ζ / β ) end_ARG start_ARG 1 / 2 exp ( ζ / β ) + 1 / 2 end_ARG J2(β)subscript2 J_2(β)J2 ( β ) =r1(x2,a0)π(a0∣x2)exp(r2(x2,a0)/β)+r1(x1,a1)π(a1∣x2)exp(r2(x2,a1)/β))Z(r2,x2) = r_1(x_2,a_0) _ ref(a_0 x_2) (% r_2(x_2,a_0)/β)+r_1(x_1,a_1) _ ref(a_1 x_2% ) (r_2(x_2,a_1)/β))Z(r_2,x_2)= divide start_ARG r1 ( x2 , a0 ) πsansserif_ref ( a0 ∣ x2 ) exp ( r2 ( x2 , a0 ) / β ) + r1 ( x1 , a1 ) πsansserif_ref ( a1 ∣ x2 ) exp ( r2 ( x2 , a1 ) / β ) ) end_ARG start_ARG Z ( r2 , x2 ) end_ARG =1/4e1/2β+1/(2n)1/2e1/2β+e1/β/(2n)+(n−1)/(2n),absent14superscript121212superscript12superscript1212 ~~~~~~~~~~= 1/4e^1/2β+1/(2n)1/2e% ^1/2β+e^1/β/(2n)+(n-1)/(2n),= divide start_ARG 1 / 4 e1 / 2 β + 1 / ( 2 n ) end_ARG start_ARG 1 / 2 e1 / 2 β + e1 / β / ( 2 n ) + ( n - 1 ) / ( 2 n ) end_ARG , where Z(r2,x):=∑a∈π(a∣x)exp(r2(x,a)/β)assignsubscript2subscriptsubscriptconditionalsubscript2Z(r_2,x) = _a _ ref(a x) (% r_2(x,a)/β)Z ( r2 , x ) := ∑a ∈ A πsansserif_ref ( a ∣ x ) exp ( r2 ( x , a ) / β ). We first consider the small β regime. Here we use the upper bound J1(β)≤ζsubscript1J_1(β)≤ 1 ( β ) ≤ ζ and focus on J2(β)subscript2J_2(β)J2 ( β ). Note that J2(β)subscript2J_2(β)J2 ( β ) is increasing with β for β≤1/(2log(n))12β≤ 1/(2 (n))β ≤ 1 / ( 2 log ( n ) ). In particular, if we consider β=1/(clog(n))1β=1/(c (n))β = 1 / ( c log ( n ) ) for c≥22c≥ 2c ≥ 2, then the expression above is equal to J2(β)=nc/2/4+1/(2n)nc/2/2+nc−1/2+(n−1)/(2n)≤nc/2/4+1/(2n)nc/2+(n−1)/(2n)≤1/4+12nc/2+1≤3/8,subscript2superscript2412superscript22superscript1212superscript2412superscript2121412superscript2138 J_2(β)= n^c/2/4+1/(2n)n^c/2/2+n^c-1/2+(n-1)/(% 2n)≤ n^c/2/4+1/(2n)n^c/2+(n-1)/(2n)≤ 1/4+ 12n^c/2+1% ≤ 3/8,J2 ( β ) = divide start_ARG nitalic_c / 2 / 4 + 1 / ( 2 n ) end_ARG start_ARG nitalic_c / 2 / 2 + nitalic_c - 1 / 2 + ( n - 1 ) / ( 2 n ) end_ARG ≤ divide start_ARG nitalic_c / 2 / 4 + 1 / ( 2 n ) end_ARG start_ARG nitalic_c / 2 + ( n - 1 ) / ( 2 n ) end_ARG ≤ 1 / 4 + divide start_ARG 1 end_ARG start_ARG 2 nitalic_c / 2 + 1 end_ARG ≤ 3 / 8 , where the last inequality holds when c≥22c≥2c ≥ 2 and n≥22n≥2n ≥ 2. We set c=22c=2c = 2, so that as long as n≥22n≥2n ≥ 2, J(πr2)≤38subscriptsubscript238J( _r_2)≤ 38J ( πitalic_r start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ) ≤ divide start_ARG 3 end_ARG start_ARG 8 end_ARG. Thus, the suboptimality is J(π~)−J(πr2)≥ζ2+14−(ζ2+316)≥116=:c2. J( π)-J( _r_2)≥ ζ2+ 1% 4- ( ζ2+ 316 )≥ 116= c_2.J ( over~ start_ARG π end_ARG ) - J ( πitalic_r start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ) ≥ divide start_ARG ζ end_ARG start_ARG 2 end_ARG + divide start_ARG 1 end_ARG start_ARG 4 end_ARG - ( divide start_ARG ζ end_ARG start_ARG 2 end_ARG + divide start_ARG 3 end_ARG start_ARG 16 end_ARG ) ≥ divide start_ARG 1 end_ARG start_ARG 16 end_ARG = : c2 . Next consider the regime where β≥1/(2log(n))12β≥ 1/(2 (n))β ≥ 1 / ( 2 log ( n ) ). Analogously to before, note that J2(β)≤1/2subscript212J_2(β)≤ 1/2J2 ( β ) ≤ 1 / 2. On the other hand, J1(β)subscript1J_1(β)J1 ( β ) is monotonically decreasing with β, so using β≥1/(2log(n))12β≥ 1/(2 (n))β ≥ 1 / ( 2 log ( n ) ) we obtain the bound J1(β)≤ζexp(2ζlog(n))exp(2ζlog(n))+1=ζ⋅n2ζn2ζ+1.subscript1221⋅superscript2superscript21 J_1(β)≤ ζ (2ζ (n)) (2ζ (% n))+1=ζ· n^2ζn^2ζ+1.J1 ( β ) ≤ divide start_ARG ζ exp ( 2 ζ log ( n ) ) end_ARG start_ARG exp ( 2 ζ log ( n ) ) + 1 end_ARG = ζ ⋅ divide start_ARG n2 ζ end_ARG start_ARG n2 ζ + 1 end_ARG . So in this case, the suboptimality is J(π~)−J(πr2)≥ζ2⋅(1−n2ζn2ζ+1)≥ζ4⋅1n2ζ=log(2)16log(n),~subscriptsubscript2⋅21superscript2superscript21⋅41superscript2216 J( π)-J( _r_2)≥ ζ2· % (1- n^2ζn^2ζ+1 )≥ ζ4· 1n^2% ζ= (2)16 (n),J ( over~ start_ARG π end_ARG ) - J ( πitalic_r start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ) ≥ divide start_ARG ζ end_ARG start_ARG 2 end_ARG ⋅ ( 1 - divide start_ARG n2 ζ end_ARG start_ARG n2 ζ + 1 end_ARG ) ≥ divide start_ARG ζ end_ARG start_ARG 4 end_ARG ⋅ divide start_ARG 1 end_ARG start_ARG n2 ζ end_ARG = divide start_ARG log ( 2 ) end_ARG start_ARG 16 log ( n ) end_ARG , if we set ζ=log(2)/(2log(n))22ζ= (2)/(2 (n))ζ = log ( 2 ) / ( 2 log ( n ) ) which is in [0,1]01[0,1][ 0 , 1 ] under the assumption that n≥22n≥ 2n ≥ 2. ∎ Appendix B Sample Complexity Guarantees for superscript2 χ^2italic_χbold_2-RLHF The χ2superscript2χ^2χ2-regularization framework we consider (Section 3.1) can be used to derive algorithms beyond just χPO, and we expect it to find broader use. To highlight this, in this section we analyze the algorithm that directly optimizes a variant of the χ2superscript2χ^2χ2-regularized RLHF objective in Eq. 6; this can be accomplished via policy optimization methods such as PPO, in the vein of classical RLHF approaches to offline alignment (Christiano et al., 2017; Bai et al., 2022; Ouyang et al., 2022; von Werra et al., 2020). As we will show, a benefit of directly optimizing the RLHF objective is that it allows us to provide guarantees that avoid dependence on the VsubscriptV_ maxVsansserif_max parameter in Theorem 3.1, which may lead to improvement when Π Π includes policies with very large or very small density ratios πsubscript π _ refdivide start_ARG π end_ARG start_ARG πsansserif_ref end_ARG. Algorithm Our algorithm, χ2superscript2χ^2χ2-RLHF is displayed in Algorithm 3. At the population level, the algorithm aims to optimize a variant of Eq. 7 that incorporates a small but important modification that allows us to avoid dependencies on πsubscript π _ refdivide start_ARG π end_ARG start_ARG πsansserif_ref end_ARG. Given smoothing parameter η>00η>0η > 0, define the smoothed χ2superscript2χ^2χ2-divergence Dχ2;η(π∥π):=π[π(a∣x)π(a∣x)+ηπ(a∣x)]assignsubscriptsuperscript2conditionalsubscriptsubscriptconditionalsubscriptconditionalconditionalD_χ^2;η (π\;\|\; _ ref ) =% E_π [ π(a x) _ ref(% a x)+ηπ(a x) ]Ditalic_χ2 ; η ( π ∥ πsansserif_ref ) := blackboard_Eπ [ divide start_ARG π ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) + η π ( a ∣ x ) end_ARG ]. We aim to find argmaxπJβ,η(π):=assignsubscriptargmaxsubscriptabsent argmax_πJ_β,η(π) =argmaxitalic_π Jitalic_β , η ( π ) := π[r⋆(x,a)]−βDχ2;η(π∥π)subscriptsuperscript⋆subscriptsuperscript2conditionalsubscript ~ E_π [r (x,a) ]-% β D_χ^2;η (π\;\|\; _ ref )blackboard_Eπ [ r⋆ ( x , a ) ] - β Ditalic_χ2 ; η ( π ∥ πsansserif_ref ) (32) = == argmaxπ[r⋆(x,a)−βπ(a∣x)π(a∣x)+ηπ(a∣x)].subscriptargmaxsubscriptdelimited-[]superscript⋆conditionalsubscriptconditionalconditional ~ argmax_πE_π [r % (x,a)-β π(a x) _ ref(a x)+ηπ(a x% ) ].argmaxitalic_π blackboard_Eπ [ r⋆ ( x , a ) - β divide start_ARG π ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) + η π ( a ∣ x ) end_ARG ] . The smoothing parameter η effectively clips the policy ratio in Dχ2;η(π∥π)subscriptsuperscript2conditionalsubscriptD_χ^2;η (π\;\|\; _ ref )Ditalic_χ2 ; η ( π ∥ πsansserif_ref ) where π(a|x)≪ηπ(a|x)much-less-thansubscriptconditionalconditional _ ref(a|x) ηπ(a|x)πsansserif_ref ( a | x ) ≪ η π ( a | x ); Dχ2(⋅∥⋅)D_χ^2 (·\;\|\;· )Ditalic_χ2 ( ⋅ ∥ ⋅ ) corresponds to the special (non-clipped) case where η=00η=0η = 0. In particular, clipping ensures a uniform bound of the form Dχ2;η(π∥π)≤η−1subscriptsuperscript2conditionalsubscriptsuperscript1D_χ^2;η (π\;\|\; _ ref )≤η^-1Ditalic_χ2 ; η ( π ∥ πsansserif_ref ) ≤ η- 1, whereas the best bound we can hope for with the unclipped χ2superscript2χ^2χ2-divergence is Dχ2(π∥π)=π[π(a|x)π(a|x)]≤∞πsubscriptsuperscript2conditionalsubscriptsubscriptconditionalsubscriptconditionalsuperscriptsubscriptD_χ^2 (π\;\|\; _ ref )= E% _π [ π(a|x) _ ref(a|x) ] _% ∞^πDitalic_χ2 ( π ∥ πsansserif_ref ) = blackboard_Eπ [ divide start_ARG π ( a | x ) end_ARG start_ARG πsansserif_ref ( a | x ) end_ARG ] ≤ C∞italic_π. For this reason, smoothing will allow us to obtain guarantees that avoid dependence on all-policy concentrability or parameters similar to VsubscriptV_ maxVsansserif_max. Algorithm 3 χ2superscript2χ^2χ2-RLHF 1: input: Reference policy πsubscript _ refπsansserif_ref, preference dataset subscriptD_ prefDsansserif_pref, unlabeled context dataset subscriptD_ xDsansserif_x, χ2superscript2χ^2χ2-regularization coefficient β>00β>0β > 0, smoothing parameter η≥00η≥ 0η ≥ 0. 2:Estimate reward model via maximum likelihood: r^←argmaxr∈ℛ∑(x,a+,a−)∈log[σ(r(x,a+)−r(x,a−))].←^subscriptargmaxℛsubscriptsubscriptsubscriptsubscriptsubscriptsubscript r← argmax_r % _(x,a_+,a_-) _ pref [σ (r(x,% a_+)-r(x,a_-) ) ].over start_ARG r end_ARG ← argmaxitalic_r ∈ R ∑( x , a start_POSTSUBSCRIPT + , a- ) ∈ Dsansserif_pref end_POSTSUBSCRIPT log [ σ ( r ( x , a+ ) - r ( x , a- ) ) ] . (33) 3:Define χ2superscript2χ^2χ2-regularized RLHF objective: J^β,η(π):=1n∑x∈(a∼π(⋅|x)[r^(x,a)]−β∑aπ2(a|x)π(a|x)+ηπ(a|x)). J_β,η(π) = 1n_ x _x∈% D_ x ( E_a π(·|x)[% r(x,a)]-β _a π^2(a|x) _ ref(a|x)+% ηπ(a|x) ).over start_ARG J end_ARGβ , η ( π ) := divide start_ARG 1 end_ARG start_ARG nsansserif_x end_ARG ∑x ∈ D start_POSTSUBSCRIPT sansserif_x end_POSTSUBSCRIPT ( blackboard_Ea ∼ π ( ⋅ | x ) [ over start_ARG r end_ARG ( x , a ) ] - β ∑a divide start_ARG π2 ( a | x ) end_ARG start_ARG πsansserif_ref ( a | x ) + η π ( a | x ) end_ARG ) . 4:Policy optimization: Compute π^∈Π^Π π∈ start_ARG π end_ARG ∈ Π such that J^β,η(π^)≥maxπ∈ΠJ^β,η(π)−εopt.subscript^^subscriptΠsubscript^subscriptopt J_β,η( π)≥ _π∈ % J_β,η(π)- _opt.over start_ARG J end_ARGβ , η ( over start_ARG π end_ARG ) ≥ maxitalic_π ∈ Π over start_ARG J end_ARGβ , η ( π ) - εroman_opt . 5:return: π^ πover start_ARG π end_ARG. To optimize Eq. 32, Algorithm 3 takes two datasets as input, along with a user-specified reward model class ℛRR and policy class Π Π. The first dataset, subscriptD_ prefDsansserif_pref, is labeled with human preferences, and is used to learn a reward model r^ rover start_ARG r end_ARG via maximum likelihood estimation in 2. The second, subscriptD_ xDsansserif_x, contains only unlabeled contexts sampled from ρ, and is utilized in 4 to learn a policy that approximately maximizes an empirical version of Eq. 32. Importantly, because 4 involves an empirical expectation over only contexts, it is a purely computational problem that we can solve using algorithms like PPO; we allow for tolerance εoptsubscriptopt _optεroman_opt in 4 to accommodate optimization error from such algorithms. By using unlabeled contexts in 4, we can obtain tighter guarantees when subscriptD_ xDsansserif_x is large. This is often the case in practice, where unlabeled contexts are cheap to obtain, but preferences can be expensive to query. Theoretical guarantees To analyze χ2superscript2χ^2χ2-RLHF, we make similar assumptions to those utilized in Theorem 3.1 for χPO. Since χ2superscript2χ^2χ2-RLHF utilizes separate reward and policy classes, we require realizability conditions for both. Namely, ℛRR must be able to express the true reward function r⋆superscript⋆r r⋆, and Π Π must include the optimal policy for the regularized RLHF objective in Eq. 32. Assumption B.1. The reward function class satisfies r⋆∈ℛsuperscript⋆ℛr ⋆ ∈ R, and is bounded so that r(x,a)∈[0,R]0subscriptr(x,a)∈[0,R_ max]r ( x , a ) ∈ [ 0 , Rsansserif_max ] for all r∈ℛr ∈ R and (x,a)∈×(x,a) ×A( x , a ) ∈ X × A. Assumption B.2. The policy class Π Π satisfies πβ,η⋆∈Πsubscriptsuperscript⋆Ππ _β,η∈ π⋆italic_β , η ∈ Π, where πβ,η⋆subscriptsuperscript⋆π _β,ηπ⋆italic_β , η is the optimal policy for Eq. 32. Below is our main sample complexity guarantee for χ2superscript2χ^2χ2-RLHF. While it is stated for a fixed, β-dependent smoothing parameter for compactness, the general version of this result (Theorem I.1) allows for general η. Theorem B.1. Let β>00β>0β > 0 be given, and suppose Assumptions B.1 and B.2 hold any η∈[0,β8R]08subscriptη∈ [0, β8R_ max ]η ∈ [ 0 , divide start_ARG β end_ARG start_ARG 8 Rsansserif_max end_ARG ]. With probability at least 1−δ11- 1 - δ, χ2superscript2χ^2χ2-RLHF (Algorithm 3) produces a policy π^ πover start_ARG π end_ARG such that for all policies π⋆superscript⋆π π⋆ simultaneously, we have J(π⋆)−J(π^)superscript⋆ J(π )-J( π)J ( π⋆ ) - J ( over start_ARG π end_ARG ) ≲Re2R⋅π⋆log(|ℛ|/δ)n+β⋅π⋆+β−1⋅R2e4Rlog(|ℛ|/δ)n+Rlog(|Π|/δ)n+εopt.less-than-or-similar-toabsent⋅subscriptsuperscript2subscriptsuperscriptsuperscript⋆ℛ⋅superscriptsuperscript⋆⋅superscript1superscriptsubscript2superscript4subscriptℛsubscriptΠsubscriptsubscriptopt R_ maxe^2R_ max· % C^π (|R|/δ)n+β·C% ^π +β^-1· R_ max^2e^4R_ max% (|R|/δ)n+R_ max (| |/δ)% n_ x+ _opt.≲ Rsansserif_max e2 Rsansserif_max ⋅ square-root start_ARG divide start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT log ( | R | / δ ) end_ARG start_ARG n end_ARG end_ARG + β ⋅ Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT + β- 1 ⋅ divide start_ARG Rsansserif_max2 e4 Rsansserif_max log ( | R | / δ ) end_ARG start_ARG n end_ARG + Rsansserif_max square-root start_ARG divide start_ARG log ( | Π | / δ ) end_ARG start_ARG nsansserif_x end_ARG end_ARG + εroman_opt . In particular, given any comparator policy π⋆superscript⋆π π⋆, we can choose the regularization parameter β to achieve J(π⋆)−J(π^)≲Re2R⋅π⋆log(|ℛ|/δ)n+Rlog(|Π|/δ)n+εopt.less-than-or-similar-tosuperscript⋆^⋅subscriptsuperscript2subscriptsuperscriptsuperscript⋆ℛsubscriptΠsubscriptsubscriptopt J(π )-J( π) R_ maxe^2R_% max· C^π (|R|/% δ)n+R_ max (| |/δ)n_ x+% _opt.J ( π⋆ ) - J ( over start_ARG π end_ARG ) ≲ Rsansserif_max e2 Rsansserif_max ⋅ square-root start_ARG divide start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT log ( | R | / δ ) end_ARG start_ARG n end_ARG end_ARG + Rsansserif_max square-root start_ARG divide start_ARG log ( | Π | / δ ) end_ARG start_ARG nsansserif_x end_ARG end_ARG + εroman_opt . (34) Above, we see that χ2superscript2χ^2χ2-RLHF, like χPO, has sample complexity that scales only with the single-policy concentrability coefficient π⋆superscriptsuperscript⋆C^π Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT, and holds for all comparator policies π⋆superscript⋆π π⋆ simultaneously. Since the choice of β induces a similar bias-overoptimization tradeoff in the first statement of Theorem B.1 as it did in Theorem 3.1 for χPO, we focus our discussion on the guarantee for a tuned choice of β (Eq. 34). The first term in Eq. 34 accounts for the reward estimation error (2) and scales with π⋆superscriptsuperscript⋆C^π Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT; as before, this accounts for how well rewards estimated from πsubscript _ refπsansserif_ref transfer to other candidate policies. The second term in Eq. 34 accounts for the statistical error from sampled contexts used in 4 for policy optimization. In particular, it is possible to drive this term to be much smaller than the first by using a larger unlabeled context dataset, which is typically far cheaper to acquire. Computationally efficiency Theorem B.1 bounds the sample complexity of χ2superscript2χ^2χ2-RLHF under the assumption that we can solve 4 up to εoptsubscriptopt _optεroman_opt-accuracy. This is a purely computational problem, and in practice it can be solved using policy gradient methods such as PPO. Comparison to χitalic_χPO Unlike χPO (Theorem 3.1), Theorem B.1 has no dependence on the parameter VsubscriptV_ maxVsansserif_max or quantities such as π≤maxπ∞πsubscriptsubscriptsuperscriptsubscript π _ ref≤ _πC_∞^πdivide start_ARG π end_ARG start_ARG πsansserif_ref end_ARG ≤ maxitalic_π C∞italic_π. We primarily attribute this to the fact that χ2superscript2χ^2χ2-RLHF uses an explicit reward function class ℛRR, and normalizing or clipping it to the reward range RsubscriptR_ maxRsansserif_max is both natural and routinely done in practice (Shah et al., 2015; Christiano et al., 2017; Ouyang et al., 2022). In comparison, the implicit reward models induced by the policy class Π Π in χPO can have larger range, and clipping the policy class in χPO directly, e.g., so that |βϕ(π)|italic-ϕsubscript|βφ( π _ ref)|| β ϕ ( divide start_ARG π end_ARG start_ARG πsansserif_ref end_ARG ) | is bounded, is misguided, because the policy class may lose realizability (Assumption 3.1). This is because r⋆(x,a)=βϕ(πβ⋆(a|x)π(a|x))+Zβ,r⋆(x)superscript⋆italic-ϕsubscriptsuperscript⋆conditionalsubscriptconditionalsubscriptsuperscript⋆r (x,a)=βφ ( π _ β(a% |x) _ ref(a|x) )+Z_β,r (x)r⋆ ( x , a ) = β ϕ ( divide start_ARG π⋆italic_β ( a | x ) end_ARG start_ARG πsansserif_ref ( a | x ) end_ARG ) + Zitalic_β , r⋆ ( x ), and the normalization factor Zβ,r⋆subscriptsuperscript⋆Z_β,r Zitalic_β , r⋆ cannot be reasonably accounted for when clipping Π Π. While the VsubscriptV_ maxVsansserif_max (Assumption 3.2) parameter involves pairs of action probabilities, and thereby sidesteps the normalization constant issue, it may not always be practical to modify Π Π so that VsubscriptV_ maxVsansserif_max is bounded, since this would require checking all pairs of each policy’s action probabilities. However, using an explicit reward function class alone is not enough. As discussed previously, when we move from implicit to explicit χ2superscript2χ^2χ2-regularization, incorporating the smoothing parameter η in Eq. 32 is essential to avoid statistical errors due to policies with large density ratios when we approximate the χ2superscript2χ^2χ2-regularizer with empirical data. A careful choice of η=β/Rsubscriptη=β/R_ maxη = β / Rsansserif_max in Theorem B.1 balances the benefits of clipping against the bias it introduces. Without smoothing (i.e., η=00η=0η = 0), a guarantee that depends on maxπ∞πsubscriptsuperscriptsubscript _πC_∞^πmaxitalic_π C∞italic_π for χ2superscript2χ^2χ2-RLHF would be unavoidable, since the sample complexity must scale with the range of the problem, which grows with the magnitude of the regularizer. See Corollary I.2 in Appendix I for a guarantee in the case where η=00η=0η = 0, which highlights this. Appendix C Experiment details Dataset and models For training, we use trl-internal-testing/tldr-preference-trl-style, with 92.9K train samples and 83.8K validation samples. The reference policy πsubscript _ refπsansserif_ref is the Pythia-1b model (Biderman et al., 2023) pre-trained on SFT data (cleanrl/EleutherAI_pythia-1b-deduped__sft__tldr from Huang et al. (2022)), and performance is measured via winrate against a baseline, as judged by GPT-4o. All parameters that are not algorithm-specific, such as the learning rate, are shared by both χPO and DPO in order to ensure a fair comparison. Training details Our implementation of χPO is built upon the DPO trainer from Transformer Reinforcement Learning (TRL) (von Werra et al., 2020). χPO comes with strong robustness and theoretical properties, but the policy ratios can sometimes introduce instability in training. In practice, we have observed that better stability and performance can be achieved by utilizing the (more general form) link function ϕ~(z):=exp([−88,20](α⋅logz))+γ⋅logzassign~italic-ϕsubscript8820⋅ φ(z) = ( clip_[-88,20] (% α· z ) )+γ· zover~ start_ARG ϕ end_ARG ( z ) := exp ( sansserif_clip[ - 88 , 20 ] ( α ⋅ log z ) ) + γ ⋅ log z in Algorithm 1, and performing a small grid search over additional parameters α=14,1141α=\ 14,1\α = divide start_ARG 1 end_ARG start_ARG 4 end_ARG , 1 and γ=0.1,10.11γ=\0.1,1\γ = 0.1 , 1 for a fixed β. We briefly discuss each parameter in turn. The mixing parameter γ controls the relative ratios of KL- and χ2superscript2χ^2χ2-regularization, our analysis in Section F.1 shows that Theorem 3.1 holds more generally for γ∈(0,1]01γ∈(0,1]γ ∈ ( 0 , 1 ] (see Theorem F.1). Next, ignoring clipping, α∈(0,1]01α∈(0,1]α ∈ ( 0 , 1 ] in ϕ~~italic-ϕ φover~ start_ARG ϕ end_ARG implements regularization with the (1+α)1(1+α)( 1 + α )-divergence (or Renyi divergence), which is an f-divergence that is stronger than KL-regularization but weaker than χ2superscript2χ^2χ2-regularization (Van Erven and Harremos, 2014), and also carries single-policy concentrability guarantees (although with a slower-rate dependence on sample size n). For example, α=1414α= 14α = divide start_ARG 1 end_ARG start_ARG 4 end_ARG corresponds to the link function ϕ(z)=(z)1/4+γlogzitalic-ϕsuperscript14φ(z)=(z)^1/4+γ zϕ ( z ) = ( z )1 / 4 + γ log z, which is easier to optimize than the link function ϕ(z)=z+γlogzitalic-ϕφ(z)=z+γ zϕ ( z ) = z + γ log z (corresponding to α=11α=1α = 1) induced by χ2superscript2χ^2χ2-regularization, given the potentially large magnitude of z=πsubscriptz= π _ refz = divide start_ARG π end_ARG start_ARG πsansserif_ref end_ARG. Though we do not write out the analysis here, the methods used to prove the sample complexity of χPO (Theorem 3.1) can be used to prove analogous guarantees for regularization with α-divergences, which will have slightly worse statistical rates. Lastly, we provide some additional explanation for the clipping operation. We observed that torch.exp is prone to underflow when logπsubscript π _ reflog divide start_ARG π end_ARG start_ARG πsansserif_ref end_ARG is very negative, and clipping the upper range to 20 can help reduce numerical instabilities. Clipping in such a manner is supported by our analysis in Proposition 4.2, which shows that π⋆π≤1+Rβsuperscript⋆subscript1subscript π _ ref≤ 1+ R_ maxβdivide start_ARG π⋆ end_ARG start_ARG πsansserif_ref end_ARG ≤ 1 + divide start_ARG Rsansserif_max end_ARG start_ARG β end_ARG (though technically we do not know RsubscriptR_ maxRsansserif_max). The parameters for all experiments are displayed in Table 2. Table 2: Parameter settings in TL;DR summarizion Algorithm Parameters DPO batch size: 64 learning rate: 1e-6 scheduler: cosine optimizer: adamw χPO batch size: 64 clip range: [-88, 20] learning rate: 1e-6 scheduler: cosine optimizer: adamw β=0.050.05β=0.05β = 0.05, 1 epoch α:1.25,γ:1.0:1.25:1.0α:1.25,γ:1.0α : 1.25 , γ : 1.0 β=0.050.05β=0.05β = 0.05, 2 epochs α:2.00,γ:1.0:2.00:1.0α:2.00,γ:1.0α : 2.00 , γ : 1.0 β=0.050.05β=0.05β = 0.05, 4 epochs α:1.25,γ:0.1:1.25:0.1α:1.25,γ:0.1α : 1.25 , γ : 0.1 β=0.0050.005β=0.005β = 0.005, all epochs α:1.25,γ:0.1:1.25:0.1α:1.25,γ:0.1α : 1.25 , γ : 0.1 Generation details For winrate evaluation, we use greedy, temperature 0, decoding. For computation of the KL divergence, we sample from the model with temperature 1. The maximum prompt length is 512, and the maximum response length is 200. We use the standard generation prompt “TL;DR:” (Gao et al., 2024). Evaluation of performance The performance of each algorithm is measured via winrate against responses in the SFT dataset, as measured by GPT-4o (global standard). The winrate is computed on a subset of 512 prompts from the SFT validation set (trl-internal-testing/tldr-preference-sft-trl-style), and the order of the model and reference responses are randomized each round. Appendix D Applying χitalic_χPO to the Token-Level MDP We formalize the offline alignment problem as a (preference-based) contextual bandit problem. Other works (Rafailov et al., 2024b; Xie et al., 2024) instead adopt a token-level MDP formulation for alignment. In the token-level MDP with horizon H, the initial state s1∼ρsimilar-tosubscript1s_1 1 ∼ ρ represents a prompt, each action ahsubscriptℎa_haitalic_h represents a token (with AA representing the vocabulary), and the state sh=(s1,a1,…,ah−1)subscriptℎsubscript1subscript1…subscriptℎ1s_h=(s_1,a_1,…,a_h-1)sitalic_h = ( s1 , a1 , … , aitalic_h - 1 ) is the prompt and sequence of tokens so far. The language model policy π maps the current state sh=(s1,a1,…,ah−1)subscriptℎsubscript1subscript1…subscriptℎ1s_h=(s_1,a_1,…,a_h-1)sitalic_h = ( s1 , a1 , … , aitalic_h - 1 ) to a distribution over the next token ah∼π(sh)similar-tosubscriptℎsubscriptℎa_h π(s_h)aitalic_h ∼ π ( sitalic_h ), and the final trajectory τ=(s1,a1),…,(sH,aH)subscript1subscript1…subscriptsubscriptτ=(s_1,a_1),…,(s_H,a_H)τ = ( s1 , a1 ) , … , ( sitalic_H , aitalic_H ) produced by this process represents the language model’s response to the prompt s1subscript1s_1s1. To apply χPO to the token-level MDP, we assume access to a dataset of labeled responses =(s1,τ+,τ−)subscriptsubscript1subscriptsubscriptD_ pref= \(s_1, _+, _-) \Dsansserif_pref = ( s1 , τ+ , τ- ) which is labeled according to the Bradley-Terry model ℙ(τ≻τ~∣s1)=exp(r(τ∣s1))exp(r(τ∣s1))+exp(r(τ~∣s1))ℙsucceedsconditional~subscript1conditionalsubscript1conditionalsubscript1conditional~subscript1 (τ τ s_1)= (% r(τ s_1) ) (r(τ s_1) )+ (r(% τ s_1) )blackboard_P ( τ ≻ over~ start_ARG τ end_ARG ∣ s1 ) = divide start_ARG exp ( r ( τ ∣ s1 ) ) end_ARG start_ARG exp ( r ( τ ∣ s1 ) ) + exp ( r ( over~ start_ARG τ end_ARG ∣ s1 ) ) end_ARG (35) for an unknown trajectory-level reward function r(τ∣s1)conditionalsubscript1r(τ s_1)r ( τ ∣ s1 ). Defining π(τ∣s1)=∏h=1Hπ(ah∣sh)conditionalsubscript1superscriptsubscriptproductℎ1conditionalsubscriptℎsubscriptℎπ(τ s_1)= _h=1^Hπ(a_h s_h)π ( τ ∣ s1 ) = ∏h = 1H π ( aitalic_h ∣ sitalic_h ), the χPO objective takes the form π^←argmaxπ∈Π∑(x,τ+,τ−)∈log[σ(2R[βϕ(π(τ+∣s1)π(τ+∣s1))−βϕ(π(τ−∣s1)π(τ−∣s1))])],←^subscriptargmaxΠsubscriptsubscriptsubscriptsubscriptsubscript2subscriptdelimited-[]italic-ϕconditionalsubscriptsubscript1subscriptconditionalsubscriptsubscript1italic-ϕconditionalsubscriptsubscript1subscriptconditionalsubscriptsubscript1 π← argmax_π∈ _% (x, _+, _-) _ pref [σ (% clip_2R_ max [βφ ( π( _+ s% _1) _ ref( _+ s_1) )-βφ ( π% ( _- s_1) _ ref( _- s_1) ) ]% ) ],over start_ARG π end_ARG ← argmaxitalic_π ∈ Π ∑( x , τ start_POSTSUBSCRIPT + , τ- ) ∈ Dsansserif_pref end_POSTSUBSCRIPT log [ σ ( sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ β ϕ ( divide start_ARG π ( τ+ ∣ s1 ) end_ARG start_ARG πsansserif_ref ( τ+ ∣ s1 ) end_ARG ) - β ϕ ( divide start_ARG π ( τ- ∣ s1 ) end_ARG start_ARG πsansserif_ref ( τ- ∣ s1 ) end_ARG ) ] ) ] , (36) which can be derived by reparameterizing the objective Jβχmix(π)=s1∼ρ,τ∼π∣s1[r(τ∣s1)]−β⋅Dχ2(π∥π)−β⋅D(π∥π),superscriptsubscriptsubscriptmixsubscriptformulae-sequencesimilar-tosubscript1similar-toconditionalsubscript1conditionalsubscript1⋅subscriptsuperscript2conditionalsubscript⋅subscriptconditionalsubscriptJ_β _ mix(π)= E_s_1 ρ,% τ π s_1 [r(τ s_1) ]-β·D_χ^2% (π\;\|\; _ ref )-β· D_ KL (% π\,\|\, _ ref ),Jitalic_βitalic_χmix ( π ) = blackboard_Es start_POSTSUBSCRIPT 1 ∼ ρ , τ ∼ π ∣ s1 end_POSTSUBSCRIPT [ r ( τ ∣ s1 ) ] - β ⋅ Ditalic_χ2 ( π ∥ πsansserif_ref ) - β ⋅ Dsansserif_KL ( π ∥ πsansserif_ref ) , where Dχ2(π∥π)=12s1∼ρ,τ∼π∣s1[(π(τ∣s1)π(τ∣s1)−1)2]subscriptsuperscript2conditionalsubscript12subscriptformulae-sequencesimilar-tosubscript1similar-toconditionalsubscriptsubscript1superscriptconditionalsubscript1subscriptconditionalsubscript112D_χ^2 (π\;\|\; _ ref )= 12% E_s_1 ρ,τ _ ref s_1% [ ( π(τ s_1) _ ref(τ s_1)% -1 )^2 ]Ditalic_χ2 ( π ∥ πsansserif_ref ) = divide start_ARG 1 end_ARG start_ARG 2 end_ARG blackboard_Es start_POSTSUBSCRIPT 1 ∼ ρ , τ ∼ πsansserif_ref ∣ s1 end_POSTSUBSCRIPT [ ( divide start_ARG π ( τ ∣ s1 ) end_ARG start_ARG πsansserif_ref ( τ ∣ s1 ) end_ARG - 1 )2 ] and D(π∥π)=s1∼ρ,τ∼π∣s1[logπ(τ∣s1)π(τ∣s1)]subscriptconditionalsubscriptsubscriptformulae-sequencesimilar-tosubscript1similar-toconditionalsubscript1conditionalsubscript1subscriptconditionalsubscript1D_ KL (π\,\|\, _ ref )= % E_s_1 ρ,τ π s_1 [ π(τ s_1% ) _ ref(τ s_1) ]Dsansserif_KL ( π ∥ πsansserif_ref ) = blackboard_Es start_POSTSUBSCRIPT 1 ∼ ρ , τ ∼ π ∣ s1 end_POSTSUBSCRIPT [ log divide start_ARG π ( τ ∣ s1 ) end_ARG start_ARG πsansserif_ref ( τ ∣ s1 ) end_ARG ] are the trajectory-level χ2superscript2χ^2χ2- and KL-divergence. From a statistical perspective, the token-level MDP formulation is identical to the contextual bandit formulation, treating the trajectory τ as a composite action, and Eq. 36 coincides with Eq. 9 under this interpretation. Consequently, Theorem 3.1 applies as-is to the token-level χPO objective in Eq. 36. In particular, as long as πβ⋆∈Πsubscriptsuperscript⋆Ππ _ β∈ π⋆italic_β ∈ Π, where πβ⋆subscriptsuperscript⋆π _ βπ⋆italic_β is the policy that satisfies r(τ∣s1)=βϕ(πβ⋆(τ∣s1)π(τ∣s1))+Zβ,r;(s1),conditionalsubscript1italic-ϕsubscriptsuperscript⋆conditionalsubscript1subscriptconditionalsubscript1subscriptsubscript1r(τ s_1)=βφ ( π _ β% (τ s_1) _ ref(τ s_1) )+Z_β,r;% KL(s_1),r ( τ ∣ s1 ) = β ϕ ( divide start_ARG π⋆italic_β ( τ ∣ s1 ) end_ARG start_ARG πsansserif_ref ( τ ∣ s1 ) end_ARG ) + Zitalic_β , r ; sansserif_KL ( s1 ) , token-level χPO ensures that with probability at least 1−δ11- 1 - δ, for all π⋆∈Πsuperscript⋆Ππ ∈ π⋆ ∈ Π, J(π⋆)−J(π^)≲Ve2R⋅π⋆log(|Π|/δ)n+β⋅π⋆+β−1⋅V2e4Rlog(|Π|/δ)n,less-than-or-similar-tosuperscript⋆^⋅subscriptsuperscript2subscriptsuperscriptsuperscript⋆Π⋅superscriptsuperscript⋆⋅superscript1superscriptsubscript2superscript4subscriptΠ J(π )-J( π) V_ maxe^2R_% max· C^π (| |/δ)n% +β·C^π +β^-1· V_ max^2% e^4R_ max (| |/δ)n,J ( π⋆ ) - J ( over start_ARG π end_ARG ) ≲ Vsansserif_max e2 Rsansserif_max ⋅ square-root start_ARG divide start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT log ( | Π | / δ ) end_ARG start_ARG n end_ARG end_ARG + β ⋅ Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT + β- 1 ⋅ divide start_ARG Vsansserif_max2 e4 Rsansserif_max log ( | Π | / δ ) end_ARG start_ARG n end_ARG , (37) where π:=s1∼ρ,τ∼π∣s1[π(τ∣s1)π(τ∣s1)]assignsuperscriptsubscriptformulae-sequencesimilar-tosubscript1similar-toconditionalsubscript1conditionalsubscript1subscriptconditionalsubscript1C^π = E_s_1 ρ,τ% π s_1 [ π(τ s_1) _ ref(τ% s_1) ]Citalic_π := blackboard_Es start_POSTSUBSCRIPT 1 ∼ ρ , τ ∼ π ∣ s1 end_POSTSUBSCRIPT [ divide start_ARG π ( τ ∣ s1 ) end_ARG start_ARG πsansserif_ref ( τ ∣ s1 ) end_ARG ]. Part I Proofs Appendix E Preliminaries Recall that for a pair of probability measures ℙPblackboard_P and ℚQblackboard_Q with a common dominating measure ω, Hellinger distance is defined via D2(ℙ,ℚ)=∫(dℙdω−dℚdω)2dω.subscriptsuperscript2ℙℚsuperscriptdℙdℚd2differential-d D^2_ H (P,Q )= (% dPdω- d% Qdω )^2dω.D2sansserif_H ( blackboard_P , blackboard_Q ) = ∫ ( square-root start_ARG divide start_ARG d blackboard_P end_ARG start_ARG d ω end_ARG end_ARG - square-root start_ARG divide start_ARG d blackboard_Q end_ARG start_ARG d ω end_ARG end_ARG )2 d ω . (38) Lemma E.1 (MLE for conditional density estimation (e.g., Wong and Shen (1995); de Geer (2000); Zhang (2006); Agarwal et al. (2020))). Consider a conditional density p⋆:→Δ():superscript⋆→Δp :X→ (Y)p⋆ : X → Δ ( Y ), where XX is the instance space and YY is the target space. Let =(xi,yi)i=1nsuperscriptsubscriptsuperscriptsuperscript1D=\(x^i,y^i)\_i=1^nD = ( xitalic_i , yitalic_i ) i = 1n be a dataset in which (xi,yi)superscriptsuperscript(x^i,y^i)( xitalic_i , yitalic_i ) are drawn i.i.d. as xi∼ρ∈Δ()similar-tosuperscriptΔx^i ρ∈ (X)xitalic_i ∼ ρ ∈ Δ ( X ) and yi∼p⋆(y∣x)similar-tosuperscriptsuperscript⋆conditionaly^i p (y x)yitalic_i ∼ p⋆ ( y ∣ x ). Suppose we have a finite function class PP such that p⋆∈superscript⋆p ⋆ ∈ P, where p(⋅∣x)∈Δ()p(· x)∈ (Y)p ( ⋅ ∣ x ) ∈ Δ ( Y ) for all p∈p ∈ P and x∈x ∈ X. Define the maximum likelihood estimator p^:=argmaxp∈∑(x,y)∈logp(y∣x).assign^subscriptargmaxsubscriptconditional p = argmax_p % _(x,y) p(y x).over start_ARG p end_ARG := argmaxitalic_p ∈ P ∑( x , y ) ∈ D log p ( y ∣ x ) . Then with probability at least 1−δ11- 1 - δ, x∼ρ[D2(p^(⋅∣x),p⋆(⋅∣x))]≤2log(||δ−1)n. E_x ρ [D^2_ H% ( p(· x),p (· x) ) ]≤ % 2 (|P|δ^-1)n.blackboard_Ex ∼ ρ [ D2sansserif_H ( over start_ARG p end_ARG ( ⋅ ∣ x ) , p⋆ ( ⋅ ∣ x ) ) ] ≤ divide start_ARG 2 log ( | P | δ- 1 ) end_ARG start_ARG n end_ARG . Appendix F Proofs for :main This section is organized as follows. First, in Section F.1, we analyze a more general version of χPO that mixes KL-regularization with χ2superscript2χ^2χ2-regularization using a mixing parameter γ∈(0,1]01γ∈(0,1]γ ∈ ( 0 , 1 ], and present its sample complexity guarantee in Theorem F.1. χPO is a special case with γ=11γ=1γ = 1, and Section F.2 shows (with a one-line proof) that Theorem 3.1 follows directly from Theorem F.1 with this parameter choice. F.1 General Version of :main As previously described at the end of Section 3.3, χPO can be applied in a more general form where the KL-regularization is mixed with χ2superscript2χ^2χ2-regularization using a weight parameter γ∈(0,1]01γ∈(0,1]γ ∈ ( 0 , 1 ]. In this section, we analyze the sample complexity for this form of the algorithm, of which χPO is a special case with γ=11γ=1γ = 1, which directly leads to the guarantee in Theorem 3.1. Concretely, given regularization parameter β>00β>0β > 0 and weight parameter γ∈(0,1]01γ∈(0,1]γ ∈ ( 0 , 1 ], we aim to solve the mixed χ2superscript2χ^2χ2-regularized objective argmaxπ:→Δ()Jβ,γχmix(π):=π[r⋆(x,a)]−β⋅Dχ2(π∥π)−βγ⋅D(π∥π).assignsubscriptargmax:→Δsuperscriptsubscriptsubscriptmixsubscriptsuperscript⋆⋅subscriptsuperscript2conditionalsubscript⋅subscriptconditionalsubscript argmax_π:X→ (% A)J_β,γ _ mix(π) =% E_π [r (x,a) ]-β· D_χ^% 2 (π\;\|\; _ ref )-βγ· D_ KL% (π\,\|\, _ ref ).argmaxitalic_π : X → Δ ( A ) Jitalic_β , γitalic_χmix ( π ) := blackboard_Eπ [ r⋆ ( x , a ) ] - β ⋅ Ditalic_χ2 ( π ∥ πsansserif_ref ) - β γ ⋅ Dsansserif_KL ( π ∥ πsansserif_ref ) . (39) The regularization term Dχ2(π∥π)+γ⋅D(π∥π)=Dfχmix,γ(π∥π)subscriptsuperscript2conditionalsubscript⋅subscriptconditionalsubscriptsubscriptsubscriptsubscriptmixconditionalsubscriptD_χ^2 (π\;\|\; _ ref )+γ· D_ % KL (π\,\|\, _ ref )=D_f_ _ mix,γ% (π\,\|\, _ ref )Ditalic_χ2 ( π ∥ πsansserif_ref ) + γ ⋅ Dsansserif_KL ( π ∥ πsansserif_ref ) = Ditalic_f start_POSTSUBSCRIPT χ start_POSTSUBSCRIPT mix , γ end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( π ∥ πsansserif_ref ) is an f-divergence induced by the function fχmix,γ(z):=12(z−1)2+γzlogzassignsubscriptsubscriptmix12superscript12f_ _ mix,γ(z) = 12(z-1)^2+γ z% zfitalic_χ start_POSTSUBSCRIPT mix , γ end_POSTSUBSCRIPT ( z ) := divide start_ARG 1 end_ARG start_ARG 2 end_ARG ( z - 1 )2 + γ z log z. Correspondingly, we replace the link function ϕ(⋅)italic-ϕ⋅φ(·)ϕ ( ⋅ ) in χPO with ϕγ(z):=z+γlog(z),assignsubscriptitalic-ϕ _γ(z) =z+γ (z),ϕitalic_γ ( z ) := z + γ log ( z ) , and output the policy π^←argmaxπ∈Π∑(x,a+,a−)∈log[σ(2R[βϕγ(π(a+∣x)π(a+∣x))−βϕγ(π(a−∣x)π(a−∣x))])].←^subscriptargmaxΠsubscriptsubscriptsubscriptsubscriptsubscript2subscriptdelimited-[]subscriptitalic-ϕconditionalsubscriptsubscriptconditionalsubscriptsubscriptitalic-ϕconditionalsubscriptsubscriptconditionalsubscript π← argmax_π∈ _% (x,a_+,a_-) _ pref [σ ( % clip_2R_ max [β _γ ( π(a_+ x)% _ ref(a_+ x) )-β _γ ( π(a_% - x) _ ref(a_- x) ) ] ) ].over start_ARG π end_ARG ← argmaxitalic_π ∈ Π ∑( x , a start_POSTSUBSCRIPT + , a- ) ∈ Dsansserif_pref end_POSTSUBSCRIPT log [ σ ( sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ β ϕitalic_γ ( divide start_ARG π ( a+ ∣ x ) end_ARG start_ARG πsansserif_ref ( a+ ∣ x ) end_ARG ) - β ϕitalic_γ ( divide start_ARG π ( a- ∣ x ) end_ARG start_ARG πsansserif_ref ( a- ∣ x ) end_ARG ) ] ) ] . (40) To give a sample complexity guarantee for Eq. 40, we require that Π Π can express the optimal regularized policy for the objective Jβ,γχmixsuperscriptsubscriptsubscriptmixJ_β,γ _ mixJitalic_β , γitalic_χmix in Eq. 39. This generalizes Assumption 3.1 for χPO, which corresponds to the special case where γ=11γ=1γ = 1. Assumption F.1 (Policy realizability). The policy class Π Π satisfies πβ,γ⋆∈Πsubscriptsuperscript⋆Ππ _β,γ∈ π⋆italic_β , γ ∈ Π, where πβ,γ⋆subscriptsuperscript⋆π _β,γπ⋆italic_β , γ is the optimal policy under mixed χ2superscript2χ^2χ2-regularization (Eq. 11). We also assert that, analogous to Assumption 3.2, the “implicit” reward models induced by the policy class Π Π and the link function ϕγsubscriptitalic-ϕ _γϕitalic_γ have bounded range. Assumption F.2 (Bounded implicit rewards). For a parameter V≥RsubscriptsubscriptV_ max≥R_ maxVsansserif_max ≥ Rsansserif_max, it holds that for all π∈Ππ∈ π ∈ Π, x∈x ∈ X, and a,b∈a,b , b ∈ A, |βϕγ(π(a∣x)π(a∣x))−βϕγ(π(b∣x)π(b∣x))|≤V.subscriptitalic-ϕconditionalsubscriptconditionalsubscriptitalic-ϕconditionalsubscriptconditionalsubscript β _γ ( π(a x) _ ref(a% x) )-β _γ ( π(b x) _ ref% (b x) ) ≤ V_ max.| β ϕitalic_γ ( divide start_ARG π ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) - β ϕitalic_γ ( divide start_ARG π ( b ∣ x ) end_ARG start_ARG πsansserif_ref ( b ∣ x ) end_ARG ) | ≤ Vsansserif_max . (41) We now state the sample complexity guarantee for the policy learned in Eq. 40. The first bound applies to general β>00β>0β > 0 and γ∈(0,1]01γ∈(0,1]γ ∈ ( 0 , 1 ], while in the second we obtain a tight statistical rate by choosing the parameter β as a function of the comparator policy π⋆superscript⋆π π⋆. Theorem F.1 (General version of Theorem 3.1). Suppose Assumptions F.1 and F.2 hold for some β>00β>0β > 0 and γ∈(0,1]01γ∈(0,1]γ ∈ ( 0 , 1 ]. With probability at least 1−δ11- 1 - δ, the variant of χPO in Eq. 40 produces a policy π^ πover start_ARG π end_ARG such that for all policies π⋆superscript⋆π π⋆ simultaneously, we have J(π⋆)−J(π^)≤32Ve2R⋅2π⋆log(|Π|/δ)n+β(1+γ)⋅π⋆2+β−1⋅256V2e4Rlog(|Π|/δ)n.superscript⋆^⋅32subscriptsuperscript2subscript2superscriptsuperscript⋆Π⋅1superscriptsuperscript⋆2⋅superscript1256superscriptsubscript2superscript4subscriptΠ J(π )-J( π)≤ 32V_ maxe^2R_% max· 2C^π (| |/δ)n% +β(1+γ)· C^π 2+β^-1·% 256V_ max^2e^4R_ max (| |/δ)n.J ( π⋆ ) - J ( over start_ARG π end_ARG ) ≤ 32 Vsansserif_max e2 Rsansserif_max ⋅ square-root start_ARG divide start_ARG 2 Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT log ( | Π | / δ ) end_ARG start_ARG n end_ARG end_ARG + β ( 1 + γ ) ⋅ divide start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT end_ARG start_ARG 2 end_ARG + β- 1 ⋅ divide start_ARG 256 Vsansserif_max2 e4 Rsansserif_max log ( | Π | / δ ) end_ARG start_ARG n end_ARG . In particular, given any comparator policy π⋆superscript⋆π π⋆, we can choose β=32Ve2R2log(|Π|/δ)nπ⋆32subscriptsuperscript2subscript2Πsuperscriptsuperscript⋆β=32V_ maxe^2R_ max 2 (| |/δ)% nC^π β = 32 Vsansserif_max e2 Rsansserif_max square-root start_ARG divide start_ARG 2 log ( | Π | / δ ) end_ARG start_ARG n Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT end_ARG end_ARG to achieve J(π⋆)−J(π^)≤(64+4γ)Ve2R⋅π⋆log(|Π|/δ)n.superscript⋆^⋅644subscriptsuperscript2subscriptsuperscriptsuperscript⋆Π J(π )-J( π)≤ (64+4γ )V_% maxe^2R_ max· C^π % (| |/δ)n.J ( π⋆ ) - J ( over start_ARG π end_ARG ) ≤ ( 64 + 4 γ ) Vsansserif_max e2 Rsansserif_max ⋅ square-root start_ARG divide start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT log ( | Π | / δ ) end_ARG start_ARG n end_ARG end_ARG . The bias-overoptimization tradeoffs induced by the choice of β in Theorem F.1 are identical to those for Theorem 3.1 (and described there). Let us briefly discuss the influence of γ on the sample complexity. We first observe that choice of γ∈(0,1]01γ∈(0,1]γ ∈ ( 0 , 1 ] changes the bound by only a small multiplicative factor, which implies that γ can be arbitrarily small as long as it is positive. For the analysis, this is natural because the KL-divergence is dominated by the χ2superscript2χ^2χ2-divergence, and, as discussed in Section 3.2, KL-regularization is only needed to enable the DPO-style reparameterization trick for Eq. 40 (in particular, the χ2superscript2χ^2χ2-RLHF algorithm in Appendix B, which does not require reparameterization, obtains similar guarantees using pure χ2superscript2χ^2χ2-regularization). It is worth noting, however, that the γ parameter can implicitly influence the magnitude of VsubscriptV_ maxVsansserif_max, as well as the policy realizability condition. As such, practical consequences of this hyperparameter choice may not be fully captured by Theorem F.1. Proof of Theorem F.1. Recall that the link function ϕγsubscriptitalic-ϕ _γϕitalic_γ induces a correspondence between policies in the class Π Π and the implicit reward functions they induce (or, equivalently, between policies and the Bradley-Terry preference models they express). Our proof centers around the implicit reward model induced by the learned policy π^ πover start_ARG π end_ARG, r^(x,a):=β⋅ϕγ(π^(a∣x)π(a∣x)),assign^⋅subscriptitalic-ϕ^conditionalsubscriptconditional r(x,a) =β· _γ (% π(a x) _ ref(a x) ),over start_ARG r end_ARG ( x , a ) := β ⋅ ϕitalic_γ ( divide start_ARG over start_ARG π end_ARG ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) , which will allow us to move between the χPO objective (Eq. 40) and the RLHF objective (Eq. 39). In particular, we establish two key facts, which together show that Eq. 40 implicitly solves Eq. 39: 1. (Lemma F.3) The reward model r^ rover start_ARG r end_ARG is an accurate estimate of r⋆superscript⋆r r⋆ on the distribution of πsubscript _ refπsansserif_ref. Moreover, we can transfer this guarantee to the distribution of any policy π by paying a multiplicative (1+2Dχ2(π∥π))12subscriptsuperscript2conditionalsubscript(1+2D_χ^2 (π\;\|\; _ ref ))( 1 + 2 Ditalic_χ2 ( π ∥ πsansserif_ref ) )-factor. 2. (Lemma F.2) π^ πover start_ARG π end_ARG maximizes the RLHF objective in Eq. 39 with reward model r^ rover start_ARG r end_ARG, namely, π^=argmaxπ∈Ππ[r^(x,a)]−β⋅Dχ2(π∥π)−βγ⋅D(π∥π).^subscriptargmaxΠsubscript^⋅subscriptsuperscript2conditionalsubscript⋅subscriptconditionalsubscript π= argmax_π∈ % E_π[ r(x,a)]-β· D_χ^2 (π\;\|\;% _ ref )-βγ· D_ KL (π\,\|\, _% ref ).over start_ARG π end_ARG = argmaxitalic_π ∈ Π blackboard_Eπ [ over start_ARG r end_ARG ( x , a ) ] - β ⋅ Ditalic_χ2 ( π ∥ πsansserif_ref ) - β γ ⋅ Dsansserif_KL ( π ∥ πsansserif_ref ) . (42) Establishing these relationships enables us to analyze the χPO policy π^ πover start_ARG π end_ARG defined in Eq. 40 through the RLHF formulation in Eq. 42, allowing us to appeal to pessimism-based arguments to show that χPO is insensitive to overoptimization error that might otherwise be encountered when learning a policy from off-policy data. Implicit reward model r^ rover start_ARG r end_ARG The χPO objective in Eq. 40 is equivalent to maximum likelihood estimation with the Bradley-Terry preference model over the induced reward function class ℛΠ:=r(x,a)=β⋅ϕγ(π(a∣x)π(a∣x)):π∈Π.assignsubscriptℛΠconditional-set⋅subscriptitalic-ϕconditionalsubscriptconditionalΠ _ = \r(x,a)=β· _γ% ( π(a x) _ ref(a x) ):π∈ % \.Rroman_Π := r ( x , a ) = β ⋅ ϕitalic_γ ( divide start_ARG π ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) : π ∈ Π . Then, since π^ πover start_ARG π end_ARG is the maximizer in Eq. 40, we can equivalently write r^=argmaxr∈ℛΠ∑(x,a+,a−)∈logσ(2R[r(a+∣x)−r(a−∣x)]).^subscriptargmaxsubscriptℛΠsubscriptsubscriptsubscriptsubscriptsubscript2subscriptdelimited-[]conditionalsubscriptconditionalsubscript r= argmax_r _ % _(x,a_+,a_-) _ pref σ ( % clip_2R_ max [r(a_+ x)-r(a_- x) ] ).over start_ARG r end_ARG = argmaxitalic_r ∈ R start_POSTSUBSCRIPT Π end_POSTSUBSCRIPT ∑( x , a start_POSTSUBSCRIPT + , a- ) ∈ Dsansserif_pref end_POSTSUBSCRIPT log σ ( sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ r ( a+ ∣ x ) - r ( a- ∣ x ) ] ) . (43) The following lemma, which builds on a standard MLE generalization bound (Lemma E.1) bounds the error of r^ rover start_ARG r end_ARG under the action distribution induced by πsubscript _ refπsansserif_ref. Recall that we use π,π′[⋅]subscriptsuperscript′⋅ E_π,π [·]blackboard_Eπ , π′ [ ⋅ ] as shorthand for x∼ρ,a∼π(⋅∣x),b∼π′(⋅∣x)[⋅] E_x ρ,a π(· x),b π (% · x)[·]blackboard_Ex ∼ ρ , a ∼ π ( ⋅ ∣ x ) , b ∼ π′ ( ⋅ ∣ x ) [ ⋅ ]. Lemma F.1. Suppose Assumption F.1 holds. Then with probability at least 1−δ11- 1 - δ, the policy π^ πover start_ARG π end_ARG output by Eq. 40 satisfies εstat2=:π,π[(2R[r^(x,a)−r^(x,b)]−2R[r⋆(x,a)−r⋆(x,b)])2]≤128R2e4Rlog(|Π|/δ)n. _stat^2= % E_ _ ref, _ ref [ ( clip_% 2R_ max [ r(x,a)- r(x,b) ]- % clip_2R_ max [r (x,a)-r (x,b) ] )^2% ]≤ 128R_ max^2e^4R_ max (| |/δ% )n.εroman_stat2 = : blackboard_Eπ start_POSTSUBSCRIPT sansserif_ref , πsansserif_ref end_POSTSUBSCRIPT [ ( sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ over start_ARG r end_ARG ( x , a ) - over start_ARG r end_ARG ( x , b ) ] - sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ r⋆ ( x , a ) - r⋆ ( x , b ) ] )2 ] ≤ divide start_ARG 128 Rsansserif_max2 e4 Rsansserif_max log ( | Π | / δ ) end_ARG start_ARG n end_ARG . Lemma F.1, along with all further supporting lemmas, is proven in the sequel. This result measures the error of r^ rover start_ARG r end_ARG using the clipped differences of rewards for pairs of actions (x,a,b)(x,a,b)( x , a , b ) drawn from πsubscript _ refπsansserif_ref. Clipping the range of the implicit/explicit reward functions to 2R2subscript2R_ max2 Rsansserif_max ensures that the statistical error does not depend on VsubscriptV_ maxVsansserif_max. One minor but important detail in the proof is showing that Assumption F.1 implies ℛΠsubscriptℛΠR_ Rroman_Π includes the true reward function r⋆superscript⋆r r⋆ up to an action-independent shift, so that the true preference model is realizable. Implicit RLHF policy optimization Having established the accuracy of r^ rover start_ARG r end_ARG, we now show that Eq. 40 finds the optimal policy to the RLHF objective in Eq. 42 when r^ rover start_ARG r end_ARG is used as the reward model, i.e., π^=argmaxπ∈ΠJβ,γ,r^χmix(π):=π[r^(x,a)]−β⋅Dχ2(π∥π)−βγ⋅D(π∥π).^subscriptargmaxΠsuperscriptsubscript^subscriptmixassignsubscript^⋅subscriptsuperscript2conditionalsubscript⋅subscriptconditionalsubscript π= argmax_π∈ J_β,γ% , r _ mix(π) = % E_π[ r(x,a)]-β· D_χ^2 (π\;\|\; _% ref )-βγ· D_ KL (π\,\|\, _% ref ).over start_ARG π end_ARG = argmaxitalic_π ∈ Π Jitalic_β , γ , over start_ARG r end_ARGχmix ( π ) := blackboard_Eπ [ over start_ARG r end_ARG ( x , a ) ] - β ⋅ Ditalic_χ2 ( π ∥ πsansserif_ref ) - β γ ⋅ Dsansserif_KL ( π ∥ πsansserif_ref ) . (44) This is a direct consequence of the result in Lemma F.2, which shows that an analogous property holds for general f-divergences. In particular, for any convex function f and policy π, the policy π is itself the optimal solution to the f-divergence-regularized RLHF objective under the implicit reward model induced by π with the link function f′. Lemma F.2. Let f:(0,∞)→ℝ:→0ℝf:(0,∞) : ( 0 , ∞ ) → blackboard_R be a convex function with f(1)=010f(1)=0f ( 1 ) = 0. Further, f is differentiable almost everywhere and 0∉dom(f′)0domsuperscript′0 (f )0 ∉ dom ( f′ ), where we define f′(0):=limx↓0f(x)−f(0)xassignsuperscript′0subscript↓00f (0) = _x 0 f(x)-f(0)xf′ ( 0 ) := limitalic_x ↓ 0 divide start_ARG f ( x ) - f ( 0 ) end_ARG start_ARG x end_ARG and f(0):=limx↓0f(x)assign0subscript↓0f(0) = _x 0f(x)f ( 0 ) := limitalic_x ↓ 0 f ( x ). Given any parameter β>00β>0β > 0 and valid policy π¯:→Δ():¯→Δ π:X→ (A)over¯ start_ARG π end_ARG : X → Δ ( A ), with π(a∣x)∈dom(f′)conditionaldomsuperscript′π(a x) (f )π ( a ∣ x ) ∈ dom ( f′ ) for all (x,a)(x,a)( x , a ), let r¯(x,a)=βf′(π¯(a∣x)π(a∣x))¯superscript′¯conditionalsubscriptconditional r(x,a)=β f ( π(a x) _ ref% (a x) )over¯ start_ARG r end_ARG ( x , a ) = β f′ ( divide start_ARG over¯ start_ARG π end_ARG ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) be the implicit reward model. Then π¯∈argmaxπ:→Δ()π[r¯(x,a)]−βDf(π∥π).¯subscriptargmax:→Δsubscript¯subscriptconditionalsubscript π∈ argmax_π:X→% (A) E_π [ r(x,a) ]-% β D_f (π\,\|\, _ ref ).over¯ start_ARG π end_ARG ∈ argmaxitalic_π : X → Δ ( A ) blackboard_Eπ [ over¯ start_ARG r end_ARG ( x , a ) ] - β Ditalic_f ( π ∥ πsansserif_ref ) . Since fχmix,γ′=ϕγ=x+γlogxsuperscriptsubscriptsubscriptmix′subscriptitalic-ϕf_ _ mix,γ = _γ=x+γ xfitalic_χ start_POSTSUBSCRIPT mix , γ end_POSTSUBSCRIPT′ = ϕitalic_γ = x + γ log x for γ>00γ>0γ > 0, clearly 0∉dom(ϕγ)0domsubscriptitalic-ϕ0 ( _γ)0 ∉ dom ( ϕitalic_γ ). Further, under Assumption F.2, π(a∣x)>0conditional0π(a x)>0π ( a ∣ x ) > 0 for all π∈Ππ∈ π ∈ Π (otherwise VsubscriptV_ maxVsansserif_max would be undefined), thus π(a∣x)∈dom(ϕγ)conditionaldomsubscriptitalic-ϕπ(a x) ( _γ)π ( a ∣ x ) ∈ dom ( ϕitalic_γ ) for all (x,a)(x,a)( x , a ). The claim in Eq. 44 then directly follows. Estimation error translation To proceed, we will use condition on Lemma F.1 and use the event in this lemma to relate the estimated RLHF objective in Eq. 42 to the “true” RLHF objective that replaces r^ rover start_ARG r end_ARG with r⋆superscript⋆r r⋆. An immediate challenge is that the RLHF objective in Eq. 42 must evaluate π[r^(x,a)]subscript E_π[ r(x,a)]blackboard_Eπ [ over start_ARG r end_ARG ( x , a ) ] for all π∈Ππ∈ π ∈ Π, and accuracy under πsubscript _ refπsansserif_ref does not immediately imply that r^ rover start_ARG r end_ARG is accurate for other policies. The following bound quantifies the effects of this distribution shift using the χ2superscript2χ^2χ2-divergence, and expresses how the estimation guarantee for r^ rover start_ARG r end_ARG in Lemma F.1 transfers to other policies π of interest. Lemma F.3. Suppose Assumption 3.1 holds. Then for any π:→Δ():→Δπ:X→ (A)π : X → Δ ( A ), under the event in Lemma F.1, we have π,π[|r^(x,a)−r^(x,b)−(r⋆(x,a)−r⋆(x,b))|]≤2VR⋅(1+2Dχ2(π∥π))⋅εstat2,subscriptsubscript^^superscript⋆superscript⋆⋅2subscriptsubscript⋅12subscriptsuperscript2conditionalsubscriptsuperscriptsubscriptstat2 E_π, _ ref [ % r(x,a)- r(x,b)- (r (x,a)-r (x,b% ) ) ]≤ 2V_ maxR_ max·% (1+2D_χ^2 (π\;\|\; _ ref ) )% · _stat^2,blackboard_Eπ , π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT [ | over start_ARG r end_ARG ( x , a ) - over start_ARG r end_ARG ( x , b ) - ( r⋆ ( x , a ) - r⋆ ( x , b ) ) | ] ≤ divide start_ARG 2 Vsansserif_max end_ARG start_ARG Rsansserif_max end_ARG ⋅ square-root start_ARG ( 1 + 2 Ditalic_χ2 ( π ∥ πsansserif_ref ) ) ⋅ εroman_stat2 end_ARG , where εstat2superscriptsubscriptstat2 _stat^2εroman_stat2 is the off-policy estimation error defined in Lemma F.1. It is worth noting that Lemma F.3 bounds the unclipped on-policy estimation error (on the LHS) in terms of the clipped off-policy estimation error, and in making this translation we pay for VsubscriptV_ maxVsansserif_max. As we will see shortly, working with the unclipped r^ rover start_ARG r end_ARG object is necessary for showing that Eq. 40 implicitly optimizes Eq. 42. Pessimism-based regret decomposition Equipped with the preceding lemmas, we can now bound the regret for χPO. We decompose the regret using the RLHF objective Jβ,γ,r^χmix(π⋆)superscriptsubscript^subscriptmixsuperscript⋆J_β,γ, r _ mix(π )Jitalic_β , γ , over start_ARG r end_ARGχmix ( π⋆ ) defined in Eq. 44. Fixing an arbitrary comparator policy π⋆superscript⋆π π⋆, we have J(π⋆)−J(π^)=superscript⋆^absent J(π )-J( π)=J ( π⋆ ) - J ( over start_ARG π end_ARG ) = π⋆[r⋆(x,a)]−π^[r⋆(x,a)]subscriptsuperscript⋆superscript⋆subscript^superscript⋆ ~ E_π [r (x,a)]-% E_ π[r (x,a)]blackboard_Eπ⋆ [ r⋆ ( x , a ) ] - blackboard_Eover start_ARG π end_ARG [ r⋆ ( x , a ) ] = == π⋆[r⋆(x,a)]−Jβ,γ,r^χmix(π⋆)+Jβ,γ,r^χmix(π⋆)−π^[r⋆(x,a)]subscriptsuperscript⋆superscript⋆superscriptsubscript^subscriptmixsuperscript⋆superscriptsubscript^subscriptmixsuperscript⋆subscript^superscript⋆ ~ E_π [r (x,a)]-J_% β,γ, r _ mix(π )+J_β,γ% , r _ mix(π )- E_% π[r (x,a)]blackboard_Eπ⋆ [ r⋆ ( x , a ) ] - Jitalic_β , γ , over start_ARG r end_ARGχmix ( π⋆ ) + Jitalic_β , γ , over start_ARG r end_ARGχmix ( π⋆ ) - blackboard_Eover start_ARG π end_ARG [ r⋆ ( x , a ) ] ≤ ≤ π⋆[r⋆(x,a)]−Jβ,γ,r^χmix(π⋆)+Jβ,γ,r^χmix(π^)−π^[r⋆(x,a)],subscriptsuperscript⋆superscript⋆superscriptsubscript^subscriptmixsuperscript⋆superscriptsubscript^subscriptmix^subscript^superscript⋆ ~ E_π [r (x,a)]-J_% β,γ, r _ mix(π )+J_β,γ% , r _ mix( π)- E% _ π[r (x,a)],blackboard_Eπ⋆ [ r⋆ ( x , a ) ] - Jitalic_β , γ , over start_ARG r end_ARGχmix ( π⋆ ) + Jitalic_β , γ , over start_ARG r end_ARGχmix ( over start_ARG π end_ARG ) - blackboard_Eover start_ARG π end_ARG [ r⋆ ( x , a ) ] , where the last inequality uses the optimality of π^ πover start_ARG π end_ARG for Eq. 44. Expanding the expression for Jβ,γ,r^χmixsuperscriptsubscript^subscriptmixJ_β,γ, r _ mixJitalic_β , γ , over start_ARG r end_ARGχmix, we can further bound this by J(π⋆)−J(π^)≤superscript⋆^absent J(π )-J( π) ( π⋆ ) - J ( over start_ARG π end_ARG ) ≤ π⋆[r⋆(x,a)−r^(x,a)]+βDχ2(π⋆∥π)+βγD(π⋆∥π)subscriptsuperscript⋆superscript⋆^subscriptsuperscript2conditionalsuperscript⋆subscriptsubscriptconditionalsuperscript⋆subscript ~ E_π [r (x,a)-% r(x,a)]+β D_χ^2 (π \;\|\; _ ref% )+βγ D_ KL (π \,\|\, _ ref )blackboard_Eπ⋆ [ r⋆ ( x , a ) - over start_ARG r end_ARG ( x , a ) ] + β Ditalic_χ2 ( π⋆ ∥ πsansserif_ref ) + β γ Dsansserif_KL ( π⋆ ∥ πsansserif_ref ) +π^[r^(x,a)−r⋆(x,a)]−βDχ2(π^∥π)−βγD(π^∥π)subscript^^superscript⋆subscriptsuperscript2conditional^subscriptsubscriptconditional^subscript + E_ π[ r(x,a)-r^% (x,a)]-β D_χ^2 ( π\;\|\; _ ref% )-βγ D_ KL ( π\,\|\, _ ref )+ blackboard_Eover start_ARG π end_ARG [ over start_ARG r end_ARG ( x , a ) - r⋆ ( x , a ) ] - β Ditalic_χ2 ( over start_ARG π end_ARG ∥ πsansserif_ref ) - β γ Dsansserif_KL ( over start_ARG π end_ARG ∥ πsansserif_ref ) ≤ ≤ π⋆[r⋆(x,a)−r^(x,a)]+β(1+γ)Dχ2(π⋆∥π)subscriptsuperscript⋆superscript⋆^1subscriptsuperscript2conditionalsuperscript⋆subscript ~ E_π [r (x,a)-% r(x,a)]+β(1+γ)D_χ^2 (π \;\|\; _% ref )blackboard_Eπ⋆ [ r⋆ ( x , a ) - over start_ARG r end_ARG ( x , a ) ] + β ( 1 + γ ) Ditalic_χ2 ( π⋆ ∥ πsansserif_ref ) +π^[r^(x,a)−r⋆(x,a)]−βDχ2(π^∥π).subscript^^superscript⋆subscriptsuperscript2conditional^subscript + E_ π[ r(x,a)-r^% (x,a)]-β D_χ^2 ( π\;\|\; _ ref% ).+ blackboard_Eover start_ARG π end_ARG [ over start_ARG r end_ARG ( x , a ) - r⋆ ( x , a ) ] - β Ditalic_χ2 ( over start_ARG π end_ARG ∥ πsansserif_ref ) . (45) In the last line, we use the fact that 0≤D(π∥π)≤Dχ2(π∥π)0subscriptconditionalsubscriptsubscriptsuperscript2conditionalsubscript0≤ D_ KL (π\,\|\, _ ref )≤ D_χ^2% (π\;\|\; _ ref )0 ≤ Dsansserif_KL ( π ∥ πsansserif_ref ) ≤ Ditalic_χ2 ( π ∥ πsansserif_ref ) for any policy π to consolidate the f-divergence terms. Specifically, this allows us to eliminate D(π^∥π)subscriptconditional^subscriptD_ KL ( π\,\|\, _ ref )Dsansserif_KL ( over start_ARG π end_ARG ∥ πsansserif_ref ), and combine D(π⋆∥π)subscriptconditionalsuperscript⋆subscriptD_ KL (π \,\|\, _ ref )Dsansserif_KL ( π⋆ ∥ πsansserif_ref ) and Dχ2(π⋆∥π)subscriptsuperscript2conditionalsuperscript⋆subscriptD_χ^2 (π \;\|\; _ ref )Ditalic_χ2 ( π⋆ ∥ πsansserif_ref ). In order to bound the reward estimation error terms in Eq. 45 using the guarantee we have previously established (Lemma F.3), we first center them using the return under the reference policy: π⋆[r⋆(x,a)−r^(x,a)]+π^[r^(x,a)−r⋆(x,a)]subscriptsuperscript⋆superscript⋆^subscript^^superscript⋆ E_π [r (x,a)- % r(x,a)]+ E_ π[ r(x,a)-r^% (x,a)]blackboard_Eπ⋆ [ r⋆ ( x , a ) - over start_ARG r end_ARG ( x , a ) ] + blackboard_Eover start_ARG π end_ARG [ over start_ARG r end_ARG ( x , a ) - r⋆ ( x , a ) ] =π⋆,π[r⋆(x,a)−r^(x,a)−r⋆(x,b)+r^(x,b)]+π^,π[r^(x,a)−r⋆(x,a)−r^(x,b)+r⋆(x,b)]absentsubscriptsuperscript⋆subscriptsuperscript⋆^superscript⋆^subscript^subscript^superscript⋆^superscript⋆ = E_π , _ ref% [r (x,a)- r(x,a)-r (x,b)+ r(x,b)% ]+ E_ π, _ ref [% r(x,a)-r (x,a)- r(x,b)+r (x,b) ]= blackboard_Eπ⋆ , π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT [ r⋆ ( x , a ) - over start_ARG r end_ARG ( x , a ) - r⋆ ( x , b ) + over start_ARG r end_ARG ( x , b ) ] + blackboard_Eover start_ARG π end_ARG , π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT [ over start_ARG r end_ARG ( x , a ) - r⋆ ( x , a ) - over start_ARG r end_ARG ( x , b ) + r⋆ ( x , b ) ] =π⋆,π[Δ⋆(x,a,b)−Δ^(x,a,b)]+π^,π[Δ^(x,a,b)−Δ⋆(x,a,b)],absentsubscriptsuperscript⋆subscriptsuperscriptΔ⋆^Δsubscript^subscript^ΔsuperscriptΔ⋆ = E_π , _ ref% [ (x,a,b)- (x,a,b) ]+% E_ π, _ ref [ (x,a,b)-% (x,a,b) ],= blackboard_Eπ⋆ , π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT [ Δ⋆ ( x , a , b ) - over start_ARG Δ end_ARG ( x , a , b ) ] + blackboard_Eover start_ARG π end_ARG , π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT [ over start_ARG Δ end_ARG ( x , a , b ) - Δ⋆ ( x , a , b ) ] , where Δ⋆(x,a,b):=r⋆(x,a)−r⋆(x,b)assignsuperscriptΔ⋆superscript⋆superscript⋆ (x,a,b) =r (x,a)-r (x,b)Δ⋆ ( x , a , b ) := r⋆ ( x , a ) - r⋆ ( x , b ) and Δ^(x,a,b):=r^(x,a)−r^(x,b)assign^Δ^ (x,a,b) = r(x,a)- r(x,b)over start_ARG Δ end_ARG ( x , a , b ) := over start_ARG r end_ARG ( x , a ) - over start_ARG r end_ARG ( x , b ). Substituting this identity back into the regret decomposition in Eq. 45, we apply Lemma F.3 with εstat2:=128R2e4Rlog(|Π|/δ)nassignsuperscriptsubscriptstat2128superscriptsubscript2superscript4subscriptΠ _stat^2 =128R_ max^2e^4R_% max (| |/δ)nεroman_stat2 := 128 Rsansserif_max2 e4 Rsansserif_max divide start_ARG log ( | Π | / δ ) end_ARG start_ARG n end_ARG (from Lemma F.1) to obtain J(π⋆)−J(π^)≤superscript⋆^absent J(π )-J( π) ( π⋆ ) - J ( over start_ARG π end_ARG ) ≤ π⋆,π[Δ⋆(x,a,b)−Δ^(x,a,b)]+β(1+γ)Dχ2(π⋆∥π)subscriptsuperscript⋆subscriptsuperscriptΔ⋆^Δ1subscriptsuperscript2conditionalsuperscript⋆subscript ~ E_π , _ ref% [ (x,a,b)- (x,a,b) ]+β(1+γ)D_% χ^2 (π \;\|\; _ ref )blackboard_Eπ⋆ , π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT [ Δ⋆ ( x , a , b ) - over start_ARG Δ end_ARG ( x , a , b ) ] + β ( 1 + γ ) Ditalic_χ2 ( π⋆ ∥ πsansserif_ref ) +π^,π[Δ^(x,a,b)−Δ⋆(x,a,b)]−βDχ2(π^∥π)subscript^subscript^ΔsuperscriptΔ⋆subscriptsuperscript2conditional^subscript + E_ π, _ ref% [ (x,a,b)- (x,a,b) ]-β D_χ^2% ( π\;\|\; _ ref )+ blackboard_Eover start_ARG π end_ARG , π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT [ over start_ARG Δ end_ARG ( x , a , b ) - Δ⋆ ( x , a , b ) ] - β Ditalic_χ2 ( over start_ARG π end_ARG ∥ πsansserif_ref ) ≤ ≤ 2VR(1+2Dχ2(π⋆∥π))⋅εstat2+β(1+γ)Dχ2(π⋆∥π)2subscriptsubscript⋅12subscriptsuperscript2conditionalsuperscript⋆subscriptsuperscriptsubscriptstat21subscriptsuperscript2conditionalsuperscript⋆subscript ~ 2V_ maxR_ max (1+2D_% χ^2 (π \;\|\; _ ref ) )·% _stat^2+β(1+γ)D_χ^2 (π % \;\|\; _ ref )divide start_ARG 2 Vsansserif_max end_ARG start_ARG Rsansserif_max end_ARG square-root start_ARG ( 1 + 2 Ditalic_χ2 ( π⋆ ∥ πsansserif_ref ) ) ⋅ εroman_stat2 end_ARG + β ( 1 + γ ) Ditalic_χ2 ( π⋆ ∥ πsansserif_ref ) +2VR(1+2Dχ2(π^∥π))⋅εstat2−βDχ2(π^∥π)2subscriptsubscript⋅12subscriptsuperscript2conditional^subscriptsuperscriptsubscriptstat2subscriptsuperscript2conditional^subscript + 2V_ maxR_ max (1+2D_χ% ^2 ( π\;\|\; _ ref ) )·% _stat^2-β D_χ^2 ( π\;\|\;% _ ref )+ divide start_ARG 2 Vsansserif_max end_ARG start_ARG Rsansserif_max end_ARG square-root start_ARG ( 1 + 2 Ditalic_χ2 ( over start_ARG π end_ARG ∥ πsansserif_ref ) ) ⋅ εroman_stat2 end_ARG - β Ditalic_χ2 ( over start_ARG π end_ARG ∥ πsansserif_ref ) = == 2VRπ⋆⋅εstat2+β(1+γ)2⋅(π⋆−1)+2VRπ^⋅εstat2−β2⋅(π^−1)2subscriptsubscript⋅superscriptsuperscript⋆superscriptsubscriptstat2⋅12superscriptsuperscript⋆12subscriptsubscript⋅superscript^superscriptsubscriptstat2⋅2superscript^1 ~ 2V_ maxR_ max C^% π · _stat^2+ β(1+γ)2% · (C^π -1 )+ 2V_ maxR_% max C π· _stat% ^2- β2· (C π-1 )divide start_ARG 2 Vsansserif_max end_ARG start_ARG Rsansserif_max end_ARG square-root start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT ⋅ εroman_stat2 end_ARG + divide start_ARG β ( 1 + γ ) end_ARG start_ARG 2 end_ARG ⋅ ( Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT - 1 ) + divide start_ARG 2 Vsansserif_max end_ARG start_ARG Rsansserif_max end_ARG square-root start_ARG Cover start_ARG π end_ARG ⋅ εroman_stat2 end_ARG - divide start_ARG β end_ARG start_ARG 2 end_ARG ⋅ ( Cover start_ARG π end_ARG - 1 ) ≤ ≤ 2VRπ⋆⋅εstat2+β(1+γ)2⋅π⋆+2VRπ^⋅εstat2−β2⋅π^,2subscriptsubscript⋅superscriptsuperscript⋆superscriptsubscriptstat2⋅12superscriptsuperscript⋆2subscriptsubscript⋅superscript^superscriptsubscriptstat2⋅2superscript ~ 2V_ maxR_ max C^% π · _stat^2+ β(1+γ)2% ·C^π + 2V_ maxR_ max % C π· _stat^2- β% 2·C π,divide start_ARG 2 Vsansserif_max end_ARG start_ARG Rsansserif_max end_ARG square-root start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT ⋅ εroman_stat2 end_ARG + divide start_ARG β ( 1 + γ ) end_ARG start_ARG 2 end_ARG ⋅ Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT + divide start_ARG 2 Vsansserif_max end_ARG start_ARG Rsansserif_max end_ARG square-root start_ARG Cover start_ARG π end_ARG ⋅ εroman_stat2 end_ARG - divide start_ARG β end_ARG start_ARG 2 end_ARG ⋅ Cover start_ARG π end_ARG , since π=1+2Dχ2(π∥π)superscript12subscriptsuperscript2conditionalsubscriptC^π=1+2D_χ^2 (π\;\|\; _ ref )Citalic_π = 1 + 2 Ditalic_χ2 ( π ∥ πsansserif_ref ), or equivalently Dχ2(π∥π)=12(π−1)subscriptsuperscript2conditionalsubscript12superscript1D_χ^2 (π\;\|\; _ ref )= 12(C% ^π-1)Ditalic_χ2 ( π ∥ πsansserif_ref ) = divide start_ARG 1 end_ARG start_ARG 2 end_ARG ( Citalic_π - 1 ). Lastly, we use the AM-GM inequality to upper bound 2VRπ^⋅εstat2≤2V2εstat2R2β+βπ^2,2subscriptsubscript⋅superscript^superscriptsubscriptstat22superscriptsubscript2superscriptsubscriptstat2superscriptsubscript2superscript^2 2V_ maxR_ max C π% · _stat^2≤ 2V_ max^2% _stat^2R_ max^2β+ β% C π2,divide start_ARG 2 Vsansserif_max end_ARG start_ARG Rsansserif_max end_ARG square-root start_ARG Cover start_ARG π end_ARG ⋅ εroman_stat2 end_ARG ≤ divide start_ARG 2 Vsansserif_max2 εroman_stat2 end_ARG start_ARG Rsansserif_max2 β end_ARG + divide start_ARG β Cover start_ARG π end_ARG end_ARG start_ARG 2 end_ARG , allowing us to conclude that J(π⋆)−J(π^)≤superscript⋆^absent J(π )-J( π) ( π⋆ ) - J ( over start_ARG π end_ARG ) ≤ 2VRπ⋆⋅εstat2+β(1+γ)2⋅π⋆+2β−1⋅V2εstat2R2.2subscriptsubscript⋅superscriptsuperscript⋆superscriptsubscriptstat2⋅12superscriptsuperscript⋆⋅2superscript1superscriptsubscript2superscriptsubscriptstat2superscriptsubscript2 ~ 2V_ maxR_ max C^% π · _stat^2+ β(1+γ)2% ·C^π +2β^-1· V_ max^2% _stat^2R_ max^2.divide start_ARG 2 Vsansserif_max end_ARG start_ARG Rsansserif_max end_ARG square-root start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT ⋅ εroman_stat2 end_ARG + divide start_ARG β ( 1 + γ ) end_ARG start_ARG 2 end_ARG ⋅ Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT + 2 β- 1 ⋅ divide start_ARG Vsansserif_max2 εroman_stat2 end_ARG start_ARG Rsansserif_max2 end_ARG . Plugging in the expression for εstat2superscriptsubscriptstat2 _stat^2εroman_stat2 results in the first statement of Theorem F.1. Choosing β for tight rates For the second statement, given a comparator policy π⋆superscript⋆π π⋆, choosing β=2VRεstat2π⋆2subscriptsubscriptsuperscriptsubscriptstat2superscriptsuperscript⋆β= 2V_ maxR_ max _% stat^2C^π β = divide start_ARG 2 Vsansserif_max end_ARG start_ARG Rsansserif_max end_ARG square-root start_ARG divide start_ARG εroman_stat2 end_ARG start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT end_ARG end_ARG gives J(π⋆)−J(π^)≤superscript⋆^absent J(π )-J( π) ( π⋆ ) - J ( over start_ARG π end_ARG ) ≤ 2VRπ⋆⋅εstat2+(1+γ)VRπ⋆⋅εstat2+VRπ⋆⋅εstat22subscriptsubscript⋅superscriptsuperscript⋆superscriptsubscriptstat21subscriptsubscript⋅superscriptsuperscript⋆superscriptsubscriptstat2subscriptsubscript⋅superscriptsuperscript⋆superscriptsubscriptstat2 ~ 2V_ maxR_ max C^% π · _stat^2+(1+γ) V_ % maxR_ max C^π · _% stat^2+ V_ maxR_ max C% ^π · _stat^2divide start_ARG 2 Vsansserif_max end_ARG start_ARG Rsansserif_max end_ARG square-root start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT ⋅ εroman_stat2 end_ARG + ( 1 + γ ) divide start_ARG Vsansserif_max end_ARG start_ARG Rsansserif_max end_ARG square-root start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT ⋅ εroman_stat2 end_ARG + divide start_ARG Vsansserif_max end_ARG start_ARG Rsansserif_max end_ARG square-root start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT ⋅ εroman_stat2 end_ARG = == (4+γ)VRπ⋆⋅εstat2.4subscriptsubscript⋅superscriptsuperscript⋆superscriptsubscriptstat2 ~ (4+γ ) V_ maxR_ max% C^π · _stat^2.( 4 + γ ) divide start_ARG Vsansserif_max end_ARG start_ARG Rsansserif_max end_ARG square-root start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT ⋅ εroman_stat2 end_ARG . ∎ F.1.1 Proofs for Supporting Lemmas Proof of Lemma F.1. Recall the reward-based MLE objective in Eq. 43, r^=argmaxr∈ℛΠ∑(x,a+,a−)∈logσ(2R[r(x,a+)−r(x,a−)]).^subscriptargmaxsubscriptℛΠsubscriptsubscriptsubscriptsubscriptsubscript2subscriptdelimited-[]subscriptsubscript r= argmax_r _ % _(x,a_+,a_-) _ pref σ ( % clip_2R_ max [r(x,a_+)-r(x,a_-) ] ).over start_ARG r end_ARG = argmaxitalic_r ∈ R start_POSTSUBSCRIPT Π end_POSTSUBSCRIPT ∑( x , a start_POSTSUBSCRIPT + , a- ) ∈ Dsansserif_pref end_POSTSUBSCRIPT log σ ( sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ r ( x , a+ ) - r ( x , a- ) ] ) . To leverage standard generalization bounds for MLE, we re-interpret this objective as maximum likelihood over a class of preference distributions under the Bradley-Terry model. For a reward function r, define for all y∈+1,−111y∈\+1,-1\y ∈ + 1 , - 1 and (x,a,b)∈×(x,a,b) ×A×A( x , a , b ) ∈ X × A × A its induced preference distribution: Pr(y|x,a,b)=y=+1⋅σ(2R[r(x,a)−r(x,b)])+y=−1⋅σ(2R[r(x,b)−r(x,a)]).subscriptconditional⋅1subscript2subscriptdelimited-[]⋅1subscript2subscriptdelimited-[]P_r(y|x,a,b)=I\y=+1\·σ ( clip_2R_ % max [r(x,a)-r(x,b) ] )+I\y=-1\·σ (% clip_2R_ max [r(x,b)-r(x,a) ] ).Pitalic_r ( y | x , a , b ) = blackboard_I y = + 1 ⋅ σ ( sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ r ( x , a ) - r ( x , b ) ] ) + blackboard_I y = - 1 ⋅ σ ( sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ r ( x , b ) - r ( x , a ) ] ) . Consider the a class of preference models induced by ℛΠsubscriptℛΠR_ Rroman_Π under this definition, Π:=Pr:r∈ℛΠ.assignsubscriptΠconditional-setsubscriptsubscriptℛΠP_ = \P_r:r _ \.Proman_Π := Pitalic_r : r ∈ Rroman_Π . We can equivalently write that Pr^=argmaxp∈Π∑(x,a+,a−)∈logp(+1∣x,a+,a−),subscript^subscriptargmaxsubscriptΠsubscriptsubscriptsubscriptsubscriptconditional1subscriptsubscript P_ r= argmax_p _ % _(x,a_+,a_-) _ pref p(+1 x,a_+,a% _-),Pover start_ARG r end_ARG = argmaxitalic_p ∈ P start_POSTSUBSCRIPT Π end_POSTSUBSCRIPT ∑( x , a start_POSTSUBSCRIPT + , a- ) ∈ Dsansserif_pref end_POSTSUBSCRIPT log p ( + 1 ∣ x , a+ , a- ) , or, interpreting each tuple (x,a+,a−)subscriptsubscript(x,a_+,a_-)( x , a+ , a- ) in subscriptD_ prefDsansserif_pref as being induced by a tuple (x,a,a~,y)~(x,a, a,y)( x , a , over~ start_ARG a end_ARG , y ) in which (a+,a−)=(a,a~)subscriptsubscript~(a_+,a_-)=(a, a)( a+ , a- ) = ( a , over~ start_ARG a end_ARG ) if y=+11y=+1y = + 1 and (a+,a−)=(a~,a)subscriptsubscript~(a_+,a_-)=( a,a)( a+ , a- ) = ( over~ start_ARG a end_ARG , a ) if y=−11y=-1y = - 1, Pr^=argmaxp∈Π∑(x,a,a~,y)∈logp(y∣x,a,a~).subscript^subscriptargmaxsubscriptΠsubscript~subscriptconditional~ P_ r= argmax_p _ % _(x,a, a,y) _ pref p(y x,a,% a).Pover start_ARG r end_ARG = argmaxitalic_p ∈ P start_POSTSUBSCRIPT Π end_POSTSUBSCRIPT ∑( x , a , over~ start_ARG a end_ARG , y ) ∈ D start_POSTSUBSCRIPT sansserif_pref end_POSTSUBSCRIPT log p ( y ∣ x , a , over~ start_ARG a end_ARG ) . Next, we show that Pr⋆∈Πsubscriptsuperscript⋆subscriptΠP_r _ Pitalic_r⋆ ∈ Proman_Π, ie., the induced preference model class realizes the true distribution. For πβ,γ⋆subscriptsuperscript⋆π _β,γπ⋆italic_β , γ, define the reward model r~⋆(x,a)=ϕγ(πβ,γ⋆(a∣x)π(a∣x)),superscript~⋆subscriptitalic-ϕsubscriptsuperscript⋆conditionalsubscriptconditional r (x,a)= _γ ( π _β,γ% (a x) _ ref(a x) ),over~ start_ARG r end_ARG⋆ ( x , a ) = ϕitalic_γ ( divide start_ARG π⋆italic_β , γ ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) , which is equivalent to r⋆superscript⋆r r⋆ up to an action-independent shift, namely, the normalization factor λβ,γ⋆subscriptsuperscript⋆λ _β,γλ⋆italic_β , γ in Lemma F.4. Since πβ,γ⋆∈Πsubscriptsuperscript⋆Ππ _β,γ∈ π⋆italic_β , γ ∈ Π under Assumption F.1, we have r~⋆∈ℛΠsuperscript~⋆subscriptℛΠ r _ over~ start_ARG r end_ARG⋆ ∈ Rroman_Π, and for all (x,a,b)∈×(x,a,b) ×A×A( x , a , b ) ∈ X × A × A, it holds that 2R[r~⋆(x,a)−r~⋆(x,b)]=subscript2subscriptdelimited-[]superscript~⋆superscript~⋆absent clip_2R_ max [ r (x,a)% - r (x,b) ]=sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ over~ start_ARG r end_ARG⋆ ( x , a ) - over~ start_ARG r end_ARG⋆ ( x , b ) ] = 2R[r⋆(x,a)−r⋆(x,b)]=r⋆(x,a)−r⋆(x,b).subscript2subscriptdelimited-[]superscript⋆superscript⋆superscript⋆superscript⋆ ~ clip_2R_ max [r (x,a)-r % (x,b) ]=r (x,a)-r (x,b).sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ r⋆ ( x , a ) - r⋆ ( x , b ) ] = r⋆ ( x , a ) - r⋆ ( x , b ) . The first equality is because action-independent shift between r~⋆superscript~⋆ r over~ start_ARG r end_ARG⋆ and r⋆superscript⋆r r⋆ is cancelled out when taking the difference of rewards, and the second equality is because, by assumption, r⋆∈[0,R]superscript⋆0subscriptr ∈[0,R_ max]r⋆ ∈ [ 0 , Rsansserif_max ]. As a result, the reward difference is bounded in the same range and never clipped. From this we conclude that Pr~⋆=Pr⋆∈Πsubscriptsuperscript~⋆subscriptsuperscript⋆subscriptΠP_ r =P_r _ Pover~ start_ARG r end_ARG⋆ = Pitalic_r⋆ ∈ Proman_Π, and realizability is satisfied. Further, it is easy to see that ΠsubscriptΠP_ Proman_Π contains only valid distributions. Thus, having satisfied the necessary preconditions, we can invoke Lemma E.1, which guarantees that with probability at least 1−δ11- 1 - δ, we have π,π[D2(Pr^(⋅∣x,a,b),Pr⋆(⋅∣x,a,b))]≤ E_ _ ref, _ ref% [D^2_ H (P_ r(· x,a,b),P_r % (· x,a,b) ) ] _Eπ start_POSTSUBSCRIPT sansserif_ref , πsansserif_ref end_POSTSUBSCRIPT [ D2sansserif_H ( Pover start_ARG r end_ARG ( ⋅ ∣ x , a , b ) , Pitalic_r⋆ ( ⋅ ∣ x , a , b ) ) ] ≤ 2log(|Π|/δ)n.2Π ~ 2 (| |/δ)n.divide start_ARG 2 log ( | Π | / δ ) end_ARG start_ARG n end_ARG . To conclude, we extract a bound on reward estimation error from this Hellinger distance bound by using Lemma F.5 with R=V=2R2subscriptR=V=2R_ maxR = V = 2 Rsansserif_max, giving π,π[(2R[r^(x,a)−r^(x,b)]−2R[r⋆(x,a)−r⋆(x,b)])2]subscriptsubscriptsubscriptsuperscriptsubscript2subscriptdelimited-[]^^subscript2subscriptdelimited-[]superscript⋆superscript⋆2 E_ _ ref, _ ref% [ ( clip_2R_ max [ r(x,a)-% r(x,b) ]- clip_2R_ max [r (x,a)% -r (x,b) ] )^2 ]blackboard_Eπ start_POSTSUBSCRIPT sansserif_ref , πsansserif_ref end_POSTSUBSCRIPT [ ( sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ over start_ARG r end_ARG ( x , a ) - over start_ARG r end_ARG ( x , b ) ] - sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ r⋆ ( x , a ) - r⋆ ( x , b ) ] )2 ] ≤64e4R2⋅π,π[D2(Pr^(⋅∣x,a,b),Pr⋆(⋅∣x,a,b))] ≤ 64e^4R_ maxR_ max^2·% E_ _ ref, _ ref [D^2_% H (P_ r(· x,a,b),P_r (· % x,a,b) ) ]≤ 64 e4 Rsansserif_max Rsansserif_max2 ⋅ blackboard_Eπ start_POSTSUBSCRIPT sansserif_ref , πsansserif_ref end_POSTSUBSCRIPT [ D2sansserif_H ( Pover start_ARG r end_ARG ( ⋅ ∣ x , a , b ) , Pitalic_r⋆ ( ⋅ ∣ x , a , b ) ) ] ≤128e4RR2⋅log(|Π|/δ)n.absent⋅128superscript4subscriptsuperscriptsubscript2Π ≤ 128e^4R_ maxR_ max^2· % (| |/δ)n.≤ 128 e4 Rsansserif_max Rsansserif_max2 ⋅ divide start_ARG log ( | Π | / δ ) end_ARG start_ARG n end_ARG . ∎ Proof of Lemma F.2. First we rewrite the objective as a minimization problem, argminπsubscriptargmin argmin_π _π −π[r¯(x,a)]+βDf(π∥π)subscript¯subscriptconditionalsubscript - E_π [ r(x,a) ]+β D% _f (π\,\|\, _ ref )- blackboard_Eπ [ over¯ start_ARG r end_ARG ( x , a ) ] + β Ditalic_f ( π ∥ πsansserif_ref ) s.t. ρ(x)∑aπ(a∣x)=ρ(x)subscriptconditional ρ(x) _aπ(a x)=ρ(x)ρ ( x ) ∑a π ( a ∣ x ) = ρ ( x ) ∀x,for-all ∀ x,∀ x , ρ(x)π(a∣x)≥0conditional0 ρ(x)π(a x)≥ 0ρ ( x ) π ( a ∣ x ) ≥ 0 ∀x,a.for-all ∀ x,a.∀ x , a . Here, π is the primal variable, and denote the dual variables as λ:→ℝ:→ℝλ:X λ : X → blackboard_R and α:×→[0,∞):→0α:X×A→[0,∞)α : X × A → [ 0 , ∞ ), which correspond to the first and second constraints, respectively. The Lagrangian form is then ℒ(π,λ,α)=−π[r¯(x,a)]+βDf(π∥π)+∑xρ(x)λ(x)(∑aπ(a∣x)−1)−∑xρ(x)∑aα(x,a)π(a∣x).ℒsubscript¯subscriptconditionalsubscriptsubscriptsubscriptconditional1subscriptsubscriptconditional (π,λ,α)=- E_π[% r(x,a)]+β D_f (π\,\|\, _ ref )+ _xρ% (x)λ(x) ( _aπ(a x)-1 )- _xρ(x) _a% α(x,a)π(a x).L ( π , λ , α ) = - blackboard_Eπ [ over¯ start_ARG r end_ARG ( x , a ) ] + β Ditalic_f ( π ∥ πsansserif_ref ) + ∑x ρ ( x ) λ ( x ) ( ∑a π ( a ∣ x ) - 1 ) - ∑x ρ ( x ) ∑a α ( x , a ) π ( a ∣ x ) . Slater’s condition holds since π¯ πover¯ start_ARG π end_ARG itself is a strictly feasible solution, and the objective is convex in π(a∣x)conditionalπ(a x)π ( a ∣ x ). Then if (π,λ,α)(π,λ,α)( π , λ , α ) satisfy the KKT conditions, they are the optimal primal and dual variables, which, overloading notation, we denote as (π⋆,λ⋆,α⋆)superscript⋆superscript⋆superscript⋆(π ,λ ,α )( π⋆ , λ⋆ , α⋆ ). We will demonstrate that setting π⋆=π¯superscript⋆¯π = π⋆ = over¯ start_ARG π end_ARG, λ⋆=0superscript⋆0λ =0λ⋆ = 0, and α⋆=0superscript⋆0α =0α⋆ = 0 satisfies the KKT conditions. First, we observe that the proposed solutions are primal and dual feasible. Further, we have π¯>0¯0 π>0over¯ start_ARG π end_ARG > 0 since 0∉dom(f′)0domsuperscript′0 (f )0 ∉ dom ( f′ ) and π¯(a∣x)∈dom(f′)¯conditionaldomsuperscript′ π(a x) (f )over¯ start_ARG π end_ARG ( a ∣ x ) ∈ dom ( f′ ). As a result, ρ(x)α⋆(x,a)π(a∣x)=0superscript⋆conditional0ρ(x)α (x,a)π(a x)=0ρ ( x ) α⋆ ( x , a ) π ( a ∣ x ) = 0 for all x,ax,ax , a, and complementary slackness is satisfied. Lastly, for stationarity, ∂ℒ(π,λ,α)∂π(a∣x)=ℒconditionalabsent (π,λ,α)∂π(a % x)=divide start_ARG ∂ L ( π , λ , α ) end_ARG start_ARG ∂ π ( a ∣ x ) end_ARG = ρ(x)(−r¯(x,a)+βf′(π¯(a∣x)π(a∣x))+λ⋆(x)−α⋆(x,a))¯superscript′¯conditionalsubscriptconditionalsuperscript⋆superscript⋆ ~ρ(x) (- r(x,a)+β f ( % π(a x) _ ref(a x) )+λ (x)-α% (x,a) )ρ ( x ) ( - over¯ start_ARG r end_ARG ( x , a ) + β f′ ( divide start_ARG over¯ start_ARG π end_ARG ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) + λ⋆ ( x ) - α⋆ ( x , a ) ) = == ρ(x)(−r¯(x,a)+βf′(π¯(a∣x)π(a∣x)))¯superscript′¯conditionalsubscriptconditional ~ρ(x) (- r(x,a)+β f ( % π(a x) _ ref(a x) ) )ρ ( x ) ( - over¯ start_ARG r end_ARG ( x , a ) + β f′ ( divide start_ARG over¯ start_ARG π end_ARG ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) ) = == ρ(x)(−βf′(π¯(a∣x)π(a∣x))+βf′(π¯(a∣x)π(a∣x)))superscript′¯conditionalsubscriptconditionalsuperscript′¯conditionalsubscriptconditional ~ρ(x) (-β f ( π(a x)% _ ref(a x) )+β f ( π(a% x) _ ref(a x) ) )ρ ( x ) ( - β f′ ( divide start_ARG over¯ start_ARG π end_ARG ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) + β f′ ( divide start_ARG over¯ start_ARG π end_ARG ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) ) = == 0,0 ~0,0 , where in the second line we substitute λ⋆=0superscript⋆0λ =0λ⋆ = 0 and α⋆=0superscript⋆0α =0α⋆ = 0, and in third line we have utilized the definition of r¯(x,a)¯ r(x,a)over¯ start_ARG r end_ARG ( x , a ) from the lemma statement. ∎ Proof of Lemma F.3. For a pair of policies π,π′π,π π , π′ and p≥11p≥ 1p ≥ 1, we define the norm ∥⋅∥p,π×π′:=(ρ,a∼π,b∼π′[|⋅|p])1/p\|·\|_p,π×π = ( E% _ρ,a π,b π [|·|^p] )^1/p∥ ⋅ ∥p , π × π′ := ( blackboard_Eρ , a ∼ π , b ∼ π′ [ | ⋅ |p ] )1 / p. In addition, for notational compactness, we abbreviate Δ^(x,a,b):=r^(x,a)−r^(x,b)assign^Δ^ (x,a,b) = r(x,a)- r(x,b)over start_ARG Δ end_ARG ( x , a , b ) := over start_ARG r end_ARG ( x , a ) - over start_ARG r end_ARG ( x , b ), and Δ⋆(x,a,b):=r⋆(x,a)−r⋆(x,b)assignsuperscriptΔ⋆superscript⋆superscript⋆ (x,a,b) =r (x,a)-r (x,b)Δ⋆ ( x , a , b ) := r⋆ ( x , a ) - r⋆ ( x , b ). Recall that our goal is to bound the (unclipped) reward estimation error under π using the (clipped) reward estimation error πsubscript _ refπsansserif_ref. We begin by decomposing ‖Δ⋆−Δ^‖1,π×π=subscriptnormsuperscriptΔ⋆^Δ1subscriptabsent \| - \|_1,π× _% ref=∥ Δ⋆ - over start_ARG Δ end_ARG ∥1 , π × π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT = ‖Δ⋆−2R[Δ^]+2R[Δ^]−Δ^‖1,π×πsubscriptnormsuperscriptΔ⋆subscript2subscriptdelimited-[]^Δsubscript2subscriptdelimited-[]^Δ^Δ1subscript ~ \| - clip_2R_ max [% ]+ clip_2R_ max [ % ]- \|_1,π× _ ref∥ Δ⋆ - sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ over start_ARG Δ end_ARG ] + sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ over start_ARG Δ end_ARG ] - over start_ARG Δ end_ARG ∥1 , π × π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT ≤ ≤ ‖Δ⋆−2R[Δ^]‖1,π×π+‖(2R[Δ^]−Δ^)⋅[2R[Δ^]≠Δ^]‖1,π×πsubscriptnormsuperscriptΔ⋆subscript2subscriptdelimited-[]^Δ1subscriptsubscriptnorm⋅subscript2subscriptdelimited-[]^Δ^Δdelimited-[]subscript2subscriptdelimited-[]^Δ^Δ1subscript ~ \| - clip_2R_ max [% ] \|_1,π× _ ref+ \| % ( clip_2R_ max [ ]- % )·I [ clip_2R_ max [% ]≠ ] \|_1,π× _% ref∥ Δ⋆ - sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ over start_ARG Δ end_ARG ] ∥1 , π × π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT + ∥ ( sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ over start_ARG Δ end_ARG ] - over start_ARG Δ end_ARG ) ⋅ blackboard_I [ sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ over start_ARG Δ end_ARG ] ≠ over start_ARG Δ end_ARG ] ∥1 , π × π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT ≤ ≤ ‖Δ⋆−2R[Δ^]‖1,π×π⏟(I) clipped on-policy estimation error+V⋅ℙπ,π(2R[Δ^]≠Δ^)⏟(I) bias from clipping.subscript⏟subscriptnormsuperscriptΔ⋆subscript2subscriptdelimited-[]^Δ1subscript(I) clipped on-policy estimation errorsubscript⏟⋅subscriptsubscriptℙsubscriptsubscript2subscriptdelimited-[]^Δ^Δ(I) bias from clipping ~ \| - clip_2R_ % max [ ] \|_1,π× _ ref% _(I) clipped on-policy estimation error+ V_ max% ·P_π, _ ref ( clip_2R_ max% [ ]≠ )_(I) bias % from clipping.under⏟ start_ARG ∥ Δ⋆ - sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ over start_ARG Δ end_ARG ] ∥1 , π × π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT end_ARG(I) clipped on-policy estimation error + under⏟ start_ARG Vsansserif_max ⋅ blackboard_Pπ , π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT ( sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ over start_ARG Δ end_ARG ] ≠ over start_ARG Δ end_ARG ) end_ARG(I) bias from clipping . This splits our bound into two terms. The first is the on-policy error of the clipped reward differences, and can be directly bounded by Lemma F.1 using a standard change-of-measure argument. The second expresses the error of translating the clipped estimates to the unclipped ones in our target bound. For the first term, using Cauchy-Schwarz gives (I)=‖Δ⋆−2R[Δ^]‖1,π×π≤(I)subscriptnormsuperscriptΔ⋆subscript2subscriptdelimited-[]^Δ1subscriptabsent (I)= \| - clip_2R_ max% [ ] \|_1,π× _ ref≤(I) = ∥ Δ⋆ - sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ over start_ARG Δ end_ARG ] ∥1 , π × π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT ≤ π⋅‖Δ⋆−2R[Δ^]‖2,π×π2⋅superscriptsuperscriptsubscriptnormsuperscriptΔ⋆subscript2subscriptdelimited-[]^Δ2subscriptsubscript2 ~ C^π· \| - clip% _2R_ max [ ] \|_2, _ % ref× _ ref^2square-root start_ARG Citalic_π ⋅ ∥ Δ⋆ - sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ over start_ARG Δ end_ARG ] ∥2 , π start_POSTSUBSCRIPT sansserif_ref × πsansserif_ref end_POSTSUBSCRIPT2 end_ARG = == π⋅‖2R[Δ⋆]−2R[Δ^]‖2,π×π2,⋅superscriptsuperscriptsubscriptnormsubscript2subscriptdelimited-[]superscriptΔ⋆subscript2subscriptdelimited-[]^Δ2subscriptsubscript2 ~ C^π· \| clip_2R_ % max [ ]- clip_2R_ max [% ] \|_2, _ ref× _ ref% ^2,square-root start_ARG Citalic_π ⋅ ∥ sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ Δ⋆ ] - sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ over start_ARG Δ end_ARG ] ∥2 , π start_POSTSUBSCRIPT sansserif_ref × πsansserif_ref end_POSTSUBSCRIPT2 end_ARG , where the last equality uses that Δ⋆∈[−R,R]superscriptΔ⋆subscriptsubscript ∈[-R_ max,R_ max]Δ⋆ ∈ [ - Rsansserif_max , Rsansserif_max ]. Next, for the second term, we again use Cauchy-Schwarz to change measure onto the offline distribution, (I)=V⋅ℙπ×π(2R[Δ^]≠Δ^)≤V⋅π⋅ℙπ,π(2R[Δ^]≠Δ^).(I)⋅subscriptsubscriptℙsubscriptsubscript2subscriptdelimited-[]^Δ^Δ⋅subscript⋅superscriptsubscriptℙsubscriptsubscriptsubscript2subscriptdelimited-[]^Δ^Δ (I)=V_ max·P_π× _% ref ( clip_2R_ max [ % ]≠ )≤ V_ max· C^% π·P_ _ ref, _ ref ( % clip_2R_ max [ ]≠ % ).(I) = Vsansserif_max ⋅ blackboard_Pπ × π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT ( sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ over start_ARG Δ end_ARG ] ≠ over start_ARG Δ end_ARG ) ≤ Vsansserif_max ⋅ square-root start_ARG Citalic_π ⋅ blackboard_Pπ start_POSTSUBSCRIPT sansserif_ref , πsansserif_ref end_POSTSUBSCRIPT ( sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ over start_ARG Δ end_ARG ] ≠ over start_ARG Δ end_ARG ) end_ARG . Further, using Markov’s inequality along with the fact that Δ⋆∈[−R,R]superscriptΔ⋆subscriptsubscript ∈[-R_ max,R_ max]Δ⋆ ∈ [ - Rsansserif_max , Rsansserif_max ], ℙπ,π(2R[Δ^]≠Δ^)≤subscriptℙsubscriptsubscriptsubscript2subscriptdelimited-[]^Δ^Δabsent _ _ ref, _ ref (% clip_2R_ max [ ]≠ % ) _Pπ start_POSTSUBSCRIPT sansserif_ref , πsansserif_ref end_POSTSUBSCRIPT ( sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ over start_ARG Δ end_ARG ] ≠ over start_ARG Δ end_ARG ) ≤ ℙπ,π(|2R[Δ^]|=2R)subscriptℙsubscriptsubscriptsubscript2subscriptdelimited-[]^Δ2subscript ~P_ _ ref, _ ref (% clip_2R_ max [ ]% =2R_ max )blackboard_Pπ start_POSTSUBSCRIPT sansserif_ref , πsansserif_ref end_POSTSUBSCRIPT ( | sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ over start_ARG Δ end_ARG ] | = 2 Rsansserif_max ) ≤ ≤ ℙπ,π(|2R[Δ^]−2R[Δ⋆]|≥R)subscriptℙsubscriptsubscriptsubscript2subscriptdelimited-[]^Δsubscript2subscriptdelimited-[]superscriptΔ⋆subscript ~P_ _ ref, _ ref (% clip_2R_ max [ ]-% clip_2R_ max [ ] ≥ R% _ max )blackboard_Pπ start_POSTSUBSCRIPT sansserif_ref , πsansserif_ref end_POSTSUBSCRIPT ( | sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ over start_ARG Δ end_ARG ] - sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ Δ⋆ ] | ≥ Rsansserif_max ) ≤ ≤ 1R2‖2R[Δ^]−2R[Δ⋆]‖2,π×π2.1superscriptsubscript2superscriptsubscriptnormsubscript2subscriptdelimited-[]^Δsubscript2subscriptdelimited-[]superscriptΔ⋆2subscriptsubscript2 ~ 1R_ max^2 \| clip_2R_% max [ ]- clip_2R_ max% [ ] \|_2, _ ref× _ % ref^2.divide start_ARG 1 end_ARG start_ARG Rsansserif_max2 end_ARG ∥ sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ over start_ARG Δ end_ARG ] - sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ Δ⋆ ] ∥2 , π start_POSTSUBSCRIPT sansserif_ref × πsansserif_ref end_POSTSUBSCRIPT2 . Combining inequalities, we obtain ‖Δ⋆−Δ^‖1,π×π≤subscriptnormsuperscriptΔ⋆^Δ1subscriptabsent \| - \|_1,π× _% ref≤∥ Δ⋆ - over start_ARG Δ end_ARG ∥1 , π × π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT ≤ (1+VR)π⋅‖2R[Δ^]−2R[Δ⋆]‖2,π×π21subscriptsubscript⋅superscriptsuperscriptsubscriptnormsubscript2subscriptdelimited-[]^Δsubscript2subscriptdelimited-[]superscriptΔ⋆2subscriptsubscript2 ~ (1+ V_ maxR_ max ) % C^π· \| clip_2R_ max [ % ]- clip_2R_ max [ ]% \|_2, _ ref× _ ref^2( 1 + divide start_ARG Vsansserif_max end_ARG start_ARG Rsansserif_max end_ARG ) square-root start_ARG Citalic_π ⋅ ∥ sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ over start_ARG Δ end_ARG ] - sansserif_clip2 R start_POSTSUBSCRIPT sansserif_max end_POSTSUBSCRIPT [ Δ⋆ ] ∥2 , π start_POSTSUBSCRIPT sansserif_ref × πsansserif_ref end_POSTSUBSCRIPT2 end_ARG = == (1+VR)(1+2Dχ2(π∥π))⋅εstat21subscriptsubscript⋅12subscriptsuperscript2conditionalsubscriptsuperscriptsubscriptstat2 ~ (1+ V_ maxR_ max ) % (1+2D_χ^2 (π\;\|\; _ ref ) )·% _stat^2( 1 + divide start_ARG Vsansserif_max end_ARG start_ARG Rsansserif_max end_ARG ) square-root start_ARG ( 1 + 2 Ditalic_χ2 ( π ∥ πsansserif_ref ) ) ⋅ εroman_stat2 end_ARG ≤ ≤ 2VR(1+2Dχ2(π∥π))⋅εstat2.2subscriptsubscript⋅12subscriptsuperscript2conditionalsubscriptsuperscriptsubscriptstat2 ~ 2V_ maxR_ max (1+2D_% χ^2 (π\;\|\; _ ref ) )· _% stat^2.divide start_ARG 2 Vsansserif_max end_ARG start_ARG Rsansserif_max end_ARG square-root start_ARG ( 1 + 2 Ditalic_χ2 ( π ∥ πsansserif_ref ) ) ⋅ εroman_stat2 end_ARG . In the second line we have used π=1+2Dχ2(π∥π)superscript12subscriptsuperscript2conditionalsubscriptC^π=1+2D_χ^2 (π\;\|\; _ ref )Citalic_π = 1 + 2 Ditalic_χ2 ( π ∥ πsansserif_ref ) and the definition of εstat2superscriptsubscriptstat2 _stat^2εroman_stat2 from Lemma F.1, and in the last line we use V≥RsubscriptsubscriptV_ max≥ R_ maxVsansserif_max ≥ Rsansserif_max. ∎ Lemma F.4. When π(a∣x)>0subscriptconditional0 _ ref(a x)>0πsansserif_ref ( a ∣ x ) > 0 for all x∈x ∈ X, the optimal policy πβ,γ⋆subscriptsuperscript⋆π _β,γπ⋆italic_β , γ for Eq. 39 satisfies r⋆(x,a)=ϕγ(πβ,γ⋆(a∣x)π(a∣x))+λβ,γ⋆(x),superscript⋆subscriptitalic-ϕsubscriptsuperscript⋆conditionalsubscriptconditionalsubscriptsuperscript⋆ r (x,a)= _γ ( π _β,% γ(a x) _ ref(a x) )+λ _β% ,γ(x),r⋆ ( x , a ) = ϕitalic_γ ( divide start_ARG π⋆italic_β , γ ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) + λ⋆italic_β , γ ( x ) , where λβ,γ⋆subscriptsuperscript⋆λ _β,γλ⋆italic_β , γ is an optimal dual variable that normalizes πβ,γ⋆subscriptsuperscript⋆π _β,γπ⋆italic_β , γ. Proof of Lemma F.4. It is easy to see that strong duality holds for Eq. 39, since it is convex and strictly feasible (e.g., for the policy πsubscript _ refπsansserif_ref). Thus, the KKT conditions give the optimal primal and dual solutions. Since Eq. 39 is constrained optimization problem (over valid policies), we first define the dual variables. Below, λ:→ℝ:→ℝλ:X λ : X → blackboard_R corresponds to the equality constraint that ∑aπ(a∣x)=1subscriptconditional1 _aπ(a x)=1∑a π ( a ∣ x ) = 1 for all x∈x ∈ X, and α:×→ℝ≥0:→subscriptℝabsent0α:X×A _≥ 0α : X × A → blackboard_R≥ 0 corresponds to the inequality constraint that π(a∣x)≥0conditional0π(a x)≥ 0π ( a ∣ x ) ≥ 0 for all (x,a)∈×(x,a) ×A( x , a ) ∈ X × A. After converting Eq. 39 from maximization to minimization, we write Eq. 39 in Lagrangian form as ℒ(π,λ,α)=−π[r⋆(x,a)]+βDfχmix,γ(π∥π)+∑xρ(x)λ(x)(∑aπ(a∣x)−1)−∑xρ(x)∑aα(x,a)π(a∣x),ℒsubscriptsuperscript⋆subscriptsubscriptsubscriptmixconditionalsubscriptsubscriptsubscriptconditional1subscriptsubscriptconditional (π,λ,α)=- E_π[% r (x,a)]+β D_f_ _ mix,γ (π\,\|\, _% ref )+ _xρ(x)λ(x) ( _aπ(a x)-1% )- _xρ(x) _aα(x,a)π(a x),L ( π , λ , α ) = - blackboard_Eπ [ r⋆ ( x , a ) ] + β Ditalic_f start_POSTSUBSCRIPT χ start_POSTSUBSCRIPT mix , γ end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( π ∥ πsansserif_ref ) + ∑x ρ ( x ) λ ( x ) ( ∑a π ( a ∣ x ) - 1 ) - ∑x ρ ( x ) ∑a α ( x , a ) π ( a ∣ x ) , since multiplying each of the solutions by ρ(x)ρ(x)ρ ( x ) does not affect the value of the saddle-point problem. We denote the optimal primal variable as πβ,γ⋆subscriptsuperscript⋆π _β,γπ⋆italic_β , γ, and optimal dual variables as (λβ,γ⋆,αβ,γ⋆)subscriptsuperscript⋆subscriptsuperscript⋆(λ _β,γ,α _β,γ)( λ⋆italic_β , γ , α⋆italic_β , γ ). From stationarity, the optimal primal and dual variables satisfy r⋆(x,a)=ϕγ(πβ,γ⋆(a∣x)π(a∣x))+λβ,γ⋆(x)−αβ,γ⋆(x,a).superscript⋆subscriptitalic-ϕsubscriptsuperscript⋆conditionalsubscriptconditionalsubscriptsuperscript⋆subscriptsuperscript⋆ r (x,a)= _γ ( π _β,% γ(a x) _ ref(a x) )+λ _β% ,γ(x)-α _β,γ(x,a).r⋆ ( x , a ) = ϕitalic_γ ( divide start_ARG π⋆italic_β , γ ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) + λ⋆italic_β , γ ( x ) - α⋆italic_β , γ ( x , a ) . Next, for a function g let g−1superscript1g^-1g- 1 denote its left inverse, such that g−1(g(x))=xsuperscript1g^-1(g(x))=xg- 1 ( g ( x ) ) = x. Because ϕγsubscriptitalic-ϕ _γϕitalic_γ is injective (see proof of Lemma F.2), it has a left inverse (ϕγ)−1superscriptsubscriptitalic-ϕ1( _γ)^-1( ϕitalic_γ )- 1, and we can write πβ,γ⋆(a∣x)=π(a∣x)⋅(ϕγ)−1(r⋆(x,a)−λβ,γ⋆(x)+αβ,γ⋆(x,a)).subscriptsuperscript⋆conditional⋅subscriptconditionalsuperscriptsubscriptitalic-ϕ1superscript⋆subscriptsuperscript⋆subscriptsuperscript⋆ π _β,γ(a x)= _ ref(a x)% ·( _γ)^-1 (r (x,a)-λ _β,γ(x% )+α _β,γ(x,a) ).π⋆italic_β , γ ( a ∣ x ) = πsansserif_ref ( a ∣ x ) ⋅ ( ϕitalic_γ )- 1 ( r⋆ ( x , a ) - λ⋆italic_β , γ ( x ) + α⋆italic_β , γ ( x , a ) ) . Because ϕγ(z)=z+γlog(z)subscriptitalic-ϕ _γ(z)=z+γ (z)ϕitalic_γ ( z ) = z + γ log ( z ), 0∉dom(ϕγ)0domsubscriptitalic-ϕ0 ( _γ)0 ∉ dom ( ϕitalic_γ ), and therefore 0∉range((ϕγ)−1)0rangesuperscriptsubscriptitalic-ϕ10 (( _γ)^-1)0 ∉ range ( ( ϕitalic_γ )- 1 ). Then from the above expression, we observe that πβ,γ⋆(a∣x)>0subscriptsuperscript⋆conditional0π _β,γ(a x)>0π⋆italic_β , γ ( a ∣ x ) > 0 since π(a∣x)>0subscriptconditional0 _ ref(a x)>0πsansserif_ref ( a ∣ x ) > 0. It immediately follows that αβ,γ⋆(x,a)=0subscriptsuperscript⋆0α _β,γ(x,a)=0α⋆italic_β , γ ( x , a ) = 0 for all (x,a)(x,a)( x , a ) from complementary slackness, which states that the optimal solutions satisfy πβ,γ⋆(a∣x)⋅αβ,γ⋆(x,a)=0⋅subscriptsuperscript⋆conditionalsubscriptsuperscript⋆0π _β,γ(a x)·α _β,γ(x,a)=0π⋆italic_β , γ ( a ∣ x ) ⋅ α⋆italic_β , γ ( x , a ) = 0 for all x,ax,ax , a. This allows us to reduce the expression for r⋆superscript⋆r r⋆ to the stated result, that is, r⋆(x,a)=ϕγ(πβ,γ⋆(a∣x)π(a∣x))+λβ,γ⋆(x).superscript⋆subscriptitalic-ϕsubscriptsuperscript⋆conditionalsubscriptconditionalsubscriptsuperscript⋆ r (x,a)= _γ ( π _β,% γ(a x) _ ref(a x) )+λ _β% ,γ(x).r⋆ ( x , a ) = ϕitalic_γ ( divide start_ARG π⋆italic_β , γ ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) + λ⋆italic_β , γ ( x ) . ∎ Lemma F.5. For z∈[−R,R]z∈[-R,R]z ∈ [ - R , R ] and z′∈[−V,V]superscript′z ∈[-V,V]z′ ∈ [ - V , V ] where V≥R≥11V≥ R≥ 1V ≥ R ≥ 1, we have |z−z′|≤superscript′absent |z-z |≤| z - z′ | ≤ 4e2RV⋅|σ(z)−σ(z′)|.⋅4superscript2superscript′ ~4e^2RV· |σ(z)-σ(z ) |.4 e2 R V ⋅ | σ ( z ) - σ ( z′ ) | . Additionally, if we define the distribution Pz(y)=y=+1σ(z)+y=−1σ(−z)subscript11P_z(y)=I\y=+1\σ(z)+I\y=-1\σ(-z)Pitalic_z ( y ) = blackboard_I y = + 1 σ ( z ) + blackboard_I y = - 1 σ ( - z ) for y∈−1,+111y∈\-1,+1\y ∈ - 1 , + 1 and define Pz′subscriptsuperscript′P_z Pitalic_z′ analogously, then |z−z′|≤superscript′absent |z-z |≤| z - z′ | ≤ 4e2RV⋅D(Pz,Pz′).⋅4superscript2subscriptsubscriptsubscriptsuperscript′ ~4e^2RV· D_ H (P_z,P_z ).4 e2 R V ⋅ Dsansserif_H ( Pitalic_z , Pitalic_z′ ) . Proof of Lemma F.5. We begin with the first statement, and write |z−z′|=superscript′absent |z-z |=| z - z′ | = |z−z′||σ(z)−σ(z′)|⋅|σ(z)−σ(z′)|.⋅superscript′superscript′ ~ |z-z ||σ(z)-σ(z )|·|% σ(z)-σ(z )|.divide start_ARG | z - z′ | end_ARG start_ARG | σ ( z ) - σ ( z′ ) | end_ARG ⋅ | σ ( z ) - σ ( z′ ) | . Since σ(z′)∈(0,1)superscript′01σ(z )∈(0,1)σ ( z′ ) ∈ ( 0 , 1 ) but z′∈[−V,V]superscript′z ∈[-V,V]z′ ∈ [ - V , V ], it can be observed that the slope |z−z′||σ(z)−σ(z′)|superscript′ |z-z ||σ(z)-σ(z )|divide start_ARG | z - z′ | end_ARG start_ARG | σ ( z ) - σ ( z′ ) | end_ARG is smallest where z≈z′≈ z z ≈ z′, and increases as we move away from this region in either direction. To better intuit the scaling of the slope in terms of V, we expand |σ(z)−σ(z′)|superscript′|σ(z)-σ(z )|| σ ( z ) - σ ( z′ ) | in the denominator to write |z−z′|=|z−z′|(1+ez)(1+ez′)|ez−ez′|⋅|σ(z)−σ(z′)|.superscript′⋅superscript′1superscript1superscriptsuperscript′superscriptsuperscript′ |z-z |= |z-z |(1+e^z)(1+e^z )|% e^z-e^z |·|σ(z)-σ(z )|.| z - z′ | = divide start_ARG | z - z′ | ( 1 + eitalic_z ) ( 1 + eitalic_z start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ) end_ARG start_ARG | eitalic_z - eitalic_z start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT | end_ARG ⋅ | σ ( z ) - σ ( z′ ) | . This indicates that the slope should scale linearly (not exponentially) with the range of z′. For example, as z′→∞→superscript′z →∞z′ → ∞, (1+ez′)/|ez−ez′|=O(1)1superscriptsuperscript′superscriptsuperscript′1(1+e^z )/|e^z-e^z |=O(1)( 1 + eitalic_z start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ) / | eitalic_z - eitalic_z start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT | = O ( 1 ). To make this intuition precise, we split into two cases. First, whenever ez′≥eR+z+1eR−1superscriptsuperscript′1superscript1e^z ≥ e^R+z+1e^R-1eitalic_z start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ≥ divide start_ARG eitalic_R + z + 1 end_ARG start_ARG eitalic_R - 1 end_ARG or ez′≤eR+z−1eR+1superscriptsuperscript′1superscript1e^z ≤ e^R+z-1e^R+1eitalic_z start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ≤ divide start_ARG eitalic_R + z - 1 end_ARG start_ARG eitalic_R + 1 end_ARG (this constitutes the range where “z′≈zsuperscript′z ≈ z′ ≈ z”), we have 1+ez′≤eR|ez−ez′|1superscriptsuperscript′superscriptsuperscriptsuperscript′1+e^z ≤ e^R|e^z-e^z |1 + eitalic_z start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ≤ eitalic_R | eitalic_z - eitalic_z start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT |. Then in this region, |z−z′|=|z−z′|(1+ez)(1+ez′)|ez−ez′||σ(z)−σ(z′)|≤2V(1+eR)eR⋅|σ(z)−σ(z′)|.superscript′1superscript1superscriptsuperscript′superscriptsuperscript′⋅21superscriptsuperscriptsuperscript′ |z-z |= |z-z |(1+e^z)(1+e^z )|% e^z-e^z ||σ(z)-σ(z )|≤ 2V(1+e^R)e^R·% |σ(z)-σ(z )|.| z - z′ | = divide start_ARG | z - z′ | ( 1 + eitalic_z ) ( 1 + eitalic_z start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ) end_ARG start_ARG | eitalic_z - eitalic_z start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT | end_ARG | σ ( z ) - σ ( z′ ) | ≤ 2 V ( 1 + eitalic_R ) eitalic_R ⋅ | σ ( z ) - σ ( z′ ) | . Next, for ez′∈[eR+z−1eR+1,eR+z+1eR−1]superscriptsuperscript′1superscript1superscript1superscript1e^z ∈[ e^R+z-1e^R+1, e^R+z+1e^R-1]eitalic_z start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ∈ [ divide start_ARG eitalic_R + z - 1 end_ARG start_ARG eitalic_R + 1 end_ARG , divide start_ARG eitalic_R + z + 1 end_ARG start_ARG eitalic_R - 1 end_ARG ], we apply the mean value theorem. Since σ′(x)=ex(1+e−x)−2superscript′superscript1superscript2σ (x)=e^x(1+e^-x)^-2σ′ ( x ) = eitalic_x ( 1 + e- x )- 2, |z−z′||σ(z)−σ(z′)|≤superscript′absent |z-z ||σ(z)-σ(z )| start_ARG | z - z′ | end_ARG start_ARG | σ ( z ) - σ ( z′ ) | end_ARG ≤ supz~∈[minz,z′,maxz,z′]ez~(1+e−z~)−2subscriptsupremum~superscript′superscript~superscript1superscript~2 ~ _ z∈ [ \z,z \, \z,z^% \ ]e z(1+e^- z)^-2supover~ start_ARG z end_ARG ∈ [ min z , z′ , max z , z′ ] eover~ start_ARG z end_ARG ( 1 + e- over~ start_ARG z end_ARG )- 2 ≤ ≤ supez~∈[eR+z−1eR+1,eR+z+1eR−1]ez~(1+e−z~)−2subscriptsupremumsuperscript~superscript1superscript1superscript1superscript1superscript~superscript1superscript~2 ~ _e z∈ [ e^R+z-1e^R+1, e^% R+z+1e^R-1 ]e z(1+e^- z)^-2supitalic_eover~ start_ARG z end_ARG ∈ [ divide start_ARG eitalic_R + z - 1 end_ARG start_ARG eitalic_R + 1 end_ARG , divide start_ARG eitalic_R + z + 1 end_ARG start_ARG eitalic_R - 1 end_ARG ] eover~ start_ARG z end_ARG ( 1 + e- over~ start_ARG z end_ARG )- 2 ≤ ≤ 4eR.4superscript ~4e^R.4 eitalic_R . In the second inequality, we use the fact that ez′,ez∈[eR+z−1eR+1,eR+z+1eR−1]superscriptsuperscript′superscript1superscript1superscript1superscript1e^z ,e^z∈[ e^R+z-1e^R+1, e^R+z+1e^R-1]eitalic_z start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT , eitalic_z ∈ [ divide start_ARG eitalic_R + z - 1 end_ARG start_ARG eitalic_R + 1 end_ARG , divide start_ARG eitalic_R + z + 1 end_ARG start_ARG eitalic_R - 1 end_ARG ], and in the third inequality we use the fact that σ′(x)superscript′σ (x)σ′ ( x ) is increasing in x, and that |z|≤R|z|≤ R| z | ≤ R. Combining the inequalities for the two regions of ez′superscript′e^z eitalic_z start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT gives the result. For the second statement, we use the fact that 2D2(Pz,Pz′)≥∑y∈+1,−1(Pz(y)−Pz′(y))2Pz(y)+Pz′(y).2subscriptsuperscript2subscriptsubscriptsuperscript′subscript11superscriptsubscriptsubscriptsuperscript′2subscriptsubscriptsuperscript′ 2D^2_ H (P_z,P_z )≥ _y% ∈\+1,-1\ (P_z(y)-P_z (y))^2P_z(y)+P_z (y% ).2 D2sansserif_H ( Pitalic_z , Pitalic_z′ ) ≥ ∑y ∈ + 1 , - 1 divide start_ARG ( Pitalic_z ( y ) - Pitalic_z′ ( y ) )2 end_ARG start_ARG Pitalic_z ( y ) + Pitalic_z′ ( y ) end_ARG . As a result, ∑y∈+1,−1(Pz(y)−Pz′(y))2≤4D2(Pz,Pz′).subscript11superscriptsubscriptsubscriptsuperscript′24subscriptsuperscript2subscriptsubscriptsuperscript′ _y∈\+1,-1\(P_z(y)-P_z (y))^2≤ 4D^2_% H (P_z,P_z ).∑y ∈ + 1 , - 1 ( Pitalic_z ( y ) - Pitalic_z′ ( y ) )2 ≤ 4 D2sansserif_H ( Pitalic_z , Pitalic_z′ ) . Since Pz(y)=1−Pz(−y)subscript1subscriptP_z(y)=1-P_z(-y)Pitalic_z ( y ) = 1 - Pitalic_z ( - y ) and Pz(+1)=σ(z)subscript1P_z(+1)=σ(z)Pitalic_z ( + 1 ) = σ ( z ), ∑y∈+1,−1(Pz(y)−Pz′(y))2=2(σ(z)−σ(z′))2,subscript11superscriptsubscriptsubscriptsuperscript′22superscriptsuperscript′2 _y∈\+1,-1\(P_z(y)-P_z (y))^2=2(σ(z)-% σ(z ))^2,∑y ∈ + 1 , - 1 ( Pitalic_z ( y ) - Pitalic_z′ ( y ) )2 = 2 ( σ ( z ) - σ ( z′ ) )2 , and therefore (σ(z)−σ(z′))2≤2D2(Pz,Pz′)superscriptsuperscript′22subscriptsuperscript2subscriptsubscriptsuperscript′(σ(z)-σ(z ))^2≤ 2D^2_ H (P_z,P_z^% )( σ ( z ) - σ ( z′ ) )2 ≤ 2 D2sansserif_H ( Pitalic_z , Pitalic_z′ ). The result follows from taking the square root of both sides and combining with the first statement in the lemma. ∎ F.2 Proof of :main Proof of Theorem 3.1. The policy optimization in 3 of Algorithm 1 is a special case of Eq. 40 with γ=11γ=1γ = 1. As a result, Theorem 3.1 follows directly from Theorem F.1 when instantiated with γ=11γ=1γ = 1. ∎ F.3 Proof of :reward_model Proof of Corollary 3.1. Recall that for any β>00β>0β > 0, Theorem 3.1 (Eq. 13) with the policy class ΠℛsubscriptΠℛ _RΠcaligraphic_R ensures that with probability at least 1−δ11- 1 - δ, for all π⋆superscript⋆π π⋆, J(π⋆)−J(π^)≤c1Re2R⋅π⋆log(|ℛ|/δ)n+c2βπ⋆+c3β−1R2e4Rlog(|ℛ|/δ)nsuperscript⋆^⋅subscript1subscriptsuperscript2subscriptsuperscriptsuperscript⋆ℛsubscript2superscriptsuperscript⋆subscript3superscript1superscriptsubscript2superscript4subscriptℛ J(π )-J( π)≤ c_1R_ maxe^2R_% max· C^π (|R|/% δ)n+c_2 ^π +c_3β^-1 R_% max^2e^4R_ max (|R|/δ)nJ ( π⋆ ) - J ( over start_ARG π end_ARG ) ≤ c1 Rsansserif_max e2 Rsansserif_max ⋅ square-root start_ARG divide start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT log ( | R | / δ ) end_ARG start_ARG n end_ARG end_ARG + c2 β Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT + c3 β- 1 divide start_ARG Rsansserif_max2 e4 Rsansserif_max log ( | R | / δ ) end_ARG start_ARG n end_ARG (46) for absolute constants c1,c2,c3>0subscript1subscript2subscript30c_1,c_2,c_3>0c1 , c2 , c3 > 0. Let us invoke this result with β⋆=argmaxβ>0maxπ⋆J(π⋆)−c1Re2R⋅π⋆log(|ℛ|/δ)n−c2βπ⋆−c3β−1R2e4Rlog(|ℛ|/δ)n.superscript⋆subscriptargmax0subscriptsuperscript⋆superscript⋆⋅subscript1subscriptsuperscript2subscriptsuperscriptsuperscript⋆ℛsubscript2superscriptsuperscript⋆subscript3superscript1superscriptsubscript2superscript4subscriptℛβ = argmax_β>0 _π \J(π% )-c_1R_ maxe^2R_ max· % C^π (|R|/δ)n-c_2 ^π^% -c_3β^-1 R_ max^2e^4R_ max (|% R|/δ)n \.β⋆ = argmaxitalic_β > 0 maxitalic_π⋆ J ( π⋆ ) - c1 Rsansserif_max e2 Rsansserif_max ⋅ square-root start_ARG divide start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT log ( | R | / δ ) end_ARG start_ARG n end_ARG end_ARG - c2 β Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT - c3 β- 1 divide start_ARG Rsansserif_max2 e4 Rsansserif_max log ( | R | / δ ) end_ARG start_ARG n end_ARG . Then Eq. 46 implies that maxπ⋆J(π⋆)−c1Re2R⋅π⋆log(|ℛ|/δ)n−c2β⋆π⋆−c3(β⋆)−1R2e4Rlog(|ℛ|/δ)n−J(π^)≤0,subscriptsuperscript⋆superscript⋆⋅subscript1subscriptsuperscript2subscriptsuperscriptsuperscript⋆ℛsubscript2superscript⋆superscriptsuperscript⋆subscript3superscriptsuperscript⋆1superscriptsubscript2superscript4subscriptℛ^0 _π \J(π )-c_1R_ maxe^2% R_ max· C^π (|R|/% δ)n-c_2β C^π -c_3(β )^% -1 R_ max^2e^4R_ max (|R|/δ)% n \-J( π)≤0,maxitalic_π⋆ J ( π⋆ ) - c1 Rsansserif_max e2 Rsansserif_max ⋅ square-root start_ARG divide start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT log ( | R | / δ ) end_ARG start_ARG n end_ARG end_ARG - c2 β⋆ Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT - c3 ( β⋆ )- 1 divide start_ARG Rsansserif_max2 e4 Rsansserif_max log ( | R | / δ ) end_ARG start_ARG n end_ARG - J ( over start_ARG π end_ARG ) ≤ 0 , so that by the definition of β⋆superscript⋆β β⋆, maxβ>0maxπ⋆J(π⋆)−c1Re2R⋅π⋆log(|ℛ|/δ)n−c2βπ⋆−c3β−1R2e4Rlog(|ℛ|/δ)n−J(π^)≤0,subscript0subscriptsuperscript⋆superscript⋆⋅subscript1subscriptsuperscript2subscriptsuperscriptsuperscript⋆ℛsubscript2superscriptsuperscript⋆subscript3superscript1superscriptsubscript2superscript4subscriptℛ^0 _β>0 _π \J(π )-c_1R_% maxe^2R_ max· C^π % (|R|/δ)n-c_2 ^π -c_3β% ^-1 R_ max^2e^4R_ max (|R|/δ)% n \-J( π)≤0,maxitalic_β > 0 maxitalic_π⋆ J ( π⋆ ) - c1 Rsansserif_max e2 Rsansserif_max ⋅ square-root start_ARG divide start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT log ( | R | / δ ) end_ARG start_ARG n end_ARG end_ARG - c2 β Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT - c3 β- 1 divide start_ARG Rsansserif_max2 e4 Rsansserif_max log ( | R | / δ ) end_ARG start_ARG n end_ARG - J ( over start_ARG π end_ARG ) ≤ 0 , or equivalently J(π⋆)−J(π^)≤c1Re2R⋅π⋆log(|ℛ|/δ)n+c2βπ⋆+c3β−1R2e4Rlog(|ℛ|/δ)n∀π⋆,∀β>0.formulae-sequencesuperscript⋆^⋅subscript1subscriptsuperscript2subscriptsuperscriptsuperscript⋆ℛsubscript2superscriptsuperscript⋆subscript3superscript1superscriptsubscript2superscript4subscriptℛfor-allsuperscript⋆for-all0 J(π )-J( π)≤ c_1R_ maxe^2R_% max· C^π (|R|/% δ)n+c_2 ^π +c_3β^-1 R_% max^2e^4R_ max (|R|/δ)n % ∀π ,∀β>0.J ( π⋆ ) - J ( over start_ARG π end_ARG ) ≤ c1 Rsansserif_max e2 Rsansserif_max ⋅ square-root start_ARG divide start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT log ( | R | / δ ) end_ARG start_ARG n end_ARG end_ARG + c2 β Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT + c3 β- 1 divide start_ARG Rsansserif_max2 e4 Rsansserif_max log ( | R | / δ ) end_ARG start_ARG n end_ARG ∀ π⋆ , ∀ β > 0 . It follows that for all comparator policies π⋆superscript⋆π π⋆, we have J(π⋆)−J(π^)≲Re2R⋅π⋆log(|ℛ|/δ)nless-than-or-similar-tosuperscript⋆^⋅subscriptsuperscript2subscriptsuperscriptsuperscript⋆ℛ J(π )-J( π) R_ maxe^2R_% max· C^π (|R|/% δ)nJ ( π⋆ ) - J ( over start_ARG π end_ARG ) ≲ Rsansserif_max e2 Rsansserif_max ⋅ square-root start_ARG divide start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT log ( | R | / δ ) end_ARG start_ARG n end_ARG end_ARG by choosing β∝R2e4Rlog(|ℛ|/δ)π⋆nproportional-tosuperscriptsubscript2superscript4subscriptℛsuperscriptsuperscript⋆β R_ max^2e^4R_ max (|% R|/δ)C^π nβ ∝ square-root start_ARG divide start_ARG Rsansserif_max2 e4 Rsansserif_max log ( | R | / δ ) end_ARG start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT n end_ARG end_ARG above. ∎ Appendix G Proofs for :understanding Proof of Proposition 4.1. To see that ϕitalic-ϕφϕ and ϕ−1superscriptitalic-ϕ1φ^-1ϕ- 1 are strictly increasing, we note that ϕ′(z)=1+1z>0superscriptitalic-ϕ′110φ (z)=1+ 1z>0ϕ′ ( z ) = 1 + divide start_ARG 1 end_ARG start_ARG z end_ARG > 0 for all z>00z>0z > 0. We now bound the inverse function ϕ−1superscriptitalic-ϕ1φ^-1ϕ- 1. We will use the fact that z↦W0(z)maps-tosubscript0z W_0(z)z ↦ W0 ( z ) is increasing over z≥00z≥0z ≥ 0 throughout. We first consider the regime where z≥11z≥1z ≥ 1. Since W0(⋅)subscript0⋅W_0(·)W0 ( ⋅ ) is increasing, we have that ϕ−1(z)=W0(ez)≤zsuperscriptitalic-ϕ1subscript0superscriptφ^-1(z)=W_0(e^z)≤zϕ- 1 ( z ) = W0 ( eitalic_z ) ≤ z if and only if ez≤zezsuperscriptsuperscripte^z≤ze^zeitalic_z ≤ z eitalic_z, which is clearly true for z≥11z≥1z ≥ 1. On the other hand, for c>00c>0c > 0 we have ϕ−1(z)=W0(ez)≥c⋅zsuperscriptitalic-ϕ1subscript0superscript⋅φ^-1(z)=W_0(e^z)≥c·zϕ- 1 ( z ) = W0 ( eitalic_z ) ≥ c ⋅ z if and only if ez≥czeczsuperscriptsuperscripte^z≥cze^czeitalic_z ≥ c z eitalic_c z; setting c=1/212c=1/2c = 1 / 2 is clearly sufficient. We now consider the regime where z≤11z≤1z ≤ 1. Here, we see that ϕ−1(z)=W(ez)≤ezsuperscriptitalic-ϕ1superscriptsuperscriptφ^-1(z)=W(e^z)≤e^zϕ- 1 ( z ) = W ( eitalic_z ) ≤ eitalic_z if and only if ez≤ezeezsuperscriptsuperscriptsuperscriptsuperscripte^z≤e^ze^e^zeitalic_z ≤ eitalic_z eitalic_e start_POSTSUPERSCRIPT z end_POSTSUPERSCRIPT, which holds for all z∈ℝz ∈ blackboard_R. On the other hand have that ϕ−1(z)=W(ez)≥e−eezsuperscriptitalic-ϕ1superscriptsuperscriptsuperscriptφ^-1(z)=W(e^z)≥e^-ee^zϕ- 1 ( z ) = W ( eitalic_z ) ≥ e- e eitalic_z if and only if ez≥e−eeze−eezsuperscriptsuperscriptsuperscriptsuperscriptsuperscriptsuperscripte^z≥e^-ee^ze^e^-ee^zeitalic_z ≥ e- e eitalic_z eitalic_e start_POSTSUPERSCRIPT - e eitalic_z end_POSTSUPERSCRIPT. Since z≤11z≤1z ≤ 1, we have e−eeze−eez≤e−eezeez≤e−eeze=ez,superscriptsuperscriptsuperscriptsuperscriptsuperscriptsuperscriptsuperscriptsuperscriptsuperscriptsuperscriptsuperscriptsuperscriptsuperscripte^-ee^ze^e^-ee^z≤e^-ee^ze^e^z≤e^-ee^ze^e=e^% z,e- e eitalic_z eitalic_e start_POSTSUPERSCRIPT - e eitalic_z end_POSTSUPERSCRIPT ≤ e- e eitalic_z eitalic_e start_POSTSUPERSCRIPT z end_POSTSUPERSCRIPT ≤ e- e eitalic_z eitalic_e = eitalic_z , which establishes the result. ∎ Proof of Proposition 4.2. Recall that the optimal policy satisfies r(x,a)=βϕ(πβ⋆(a∣x)π(a∣x))+Zβ,r(x),italic-ϕsubscriptsuperscript⋆conditionalsubscriptconditionalsubscript r(x,a)=βφ ( π _ % β(a x) _ ref(a x) )+Z_β,r(x),r ( x , a ) = β ϕ ( divide start_ARG π⋆italic_β ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) + Zitalic_β , r ( x ) , (47) where Zβ,r(x)subscriptZ_β,r(x)Zitalic_β , r ( x ) is a normalization constant chosen such that πβ⋆(⋅∣x)π _ β(· x)π⋆italic_β ( ⋅ ∣ x ) is a valid probability distribution. We begin by bounding Zβ,r(x)subscriptZ_β,r(x)Zitalic_β , r ( x ). We will use that r(x,a)∈[0,R]0subscriptr(x,a)∈[0,R_ max]r ( x , a ) ∈ [ 0 , Rsansserif_max ]. Let x∈x ∈ X be fixed. By averaging Eq. 47 over a∼πβ⋆(x)similar-tosubscriptsuperscript⋆a π _ β(x)a ∼ π⋆italic_β ( x ), we have a∼πβ⋆(x)[r(x,a)]=βa∼πβ⋆(x)[πβ⋆(a∣x)π(a∣x)]+βD(πβ⋆∥π)+Zβ,r(x)≥Zβ,r(x),subscriptsimilar-tosubscriptsuperscript⋆subscriptsimilar-tosubscriptsuperscript⋆subscriptsuperscript⋆conditionalsubscriptconditionalsubscriptconditionalsubscriptsuperscript⋆subscriptsubscriptsubscript E_a π _ % β(x)[r(x,a)]=β E_a π _% β(x) [ π _ β% (a x) _ ref(a x) ]+β D_ KL (% π _ β\,\|\, _ ref )+Z_% β,r(x)≥Z_β,r(x),blackboard_Ea ∼ π⋆ start_POSTSUBSCRIPT β ( x ) end_POSTSUBSCRIPT [ r ( x , a ) ] = β blackboard_Ea ∼ π⋆ start_POSTSUBSCRIPT β ( x ) end_POSTSUBSCRIPT [ divide start_ARG π⋆italic_β ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ] + β Dsansserif_KL ( π⋆italic_β ∥ πsansserif_ref ) + Zitalic_β , r ( x ) ≥ Zitalic_β , r ( x ) , so Zβ,r(x)≤RsubscriptsubscriptZ_β,r(x)≤R_ maxZitalic_β , r ( x ) ≤ Rsansserif_max. On the other hand, averaging over a∼π(x)similar-tosubscripta _ ref(x)a ∼ πsansserif_ref ( x ), we have a∼πβ⋆(x)[r(x,a)]subscriptsimilar-tosubscriptsuperscript⋆ E_a π _ % β(x)[r(x,a)]blackboard_Ea ∼ π⋆ start_POSTSUBSCRIPT β ( x ) end_POSTSUBSCRIPT [ r ( x , a ) ] =βa∼π(x)[πβ⋆(a∣x)π(a∣x)]−βD(π∥πβ⋆)+Zβ,r(x)absentsubscriptsimilar-tosubscriptsubscriptsuperscript⋆conditionalsubscriptconditionalsubscriptconditionalsubscriptsubscriptsuperscript⋆subscript =β E_a _ ref(x) % [ π _ β(a x) _ ref(a% x) ]-β D_ KL ( _ ref\,\|\,π % _ β )+Z_β,r(x)= β blackboard_Ea ∼ π start_POSTSUBSCRIPT sansserif_ref ( x ) end_POSTSUBSCRIPT [ divide start_ARG π⋆italic_β ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ] - β Dsansserif_KL ( πsansserif_ref ∥ π⋆italic_β ) + Zitalic_β , r ( x ) ≤β+Zβ,r(x),absentsubscript ≤β+Z_β,r(x),≤ β + Zitalic_β , r ( x ) , so Zβ,r(x)≥−βsubscriptZ_β,r(x)≥- _β , r ( x ) ≥ - β. Having established that Zβ,r(x)∈[−β,R]subscriptsubscriptZ_β,r(x)∈ [-β,R_ max ]Zitalic_β , r ( x ) ∈ [ - β , Rsansserif_max ], we will use that ϕ(πβ⋆(a∣x)π(a∣x))=β−1(r(x,a)−Zβ,r(x))italic-ϕsubscriptsuperscript⋆conditionalsubscriptconditionalsuperscript1subscriptφ ( π _ β(a x) _% ref(a x) )=β^-1(r(x,a)-Z_β,r(x))ϕ ( divide start_ARG π⋆italic_β ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) = β- 1 ( r ( x , a ) - Zitalic_β , r ( x ) ), so that our bound on Zβ,rsubscriptZ_β,rZitalic_β , r implies that −β−1R≤ϕ(πβ⋆(a∣x)π(a∣x))≤1+β−1R,superscript1subscriptitalic-ϕsubscriptsuperscript⋆conditionalsubscriptconditional1superscript1subscript-β^-1R_ max≤φ ( π _% β(a x) _ ref(a x) )≤% 1+β^-1R_ max,- β- 1 Rsansserif_max ≤ ϕ ( divide start_ARG π⋆italic_β ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ) ≤ 1 + β- 1 Rsansserif_max , or, since ϕ−1superscriptitalic-ϕ1φ^-1ϕ- 1 is increasing, e−e⋅e−β−1R≤ϕ−1(−β−1R)≤πβ⋆(a∣x)π(a∣x)≤ϕ−1(1+β−1R)≤1+β−1R,⋅superscriptsuperscriptsuperscript1subscriptsuperscriptitalic-ϕ1superscript1subscriptsubscriptsuperscript⋆conditionalsubscriptconditionalsuperscriptitalic-ϕ11superscript1subscript1superscript1subscripte^-e·e^-β^-1R_ max≤φ^-1(-β^-1R_% max)≤ π _ β(a x)% _ ref(a x)≤φ^-1(1+β^-1R_ max)≤% 1+β^-1R_ max,e- e ⋅ e- β start_POSTSUPERSCRIPT - 1 Rsansserif_max end_POSTSUPERSCRIPT ≤ ϕ- 1 ( - β- 1 Rsansserif_max ) ≤ divide start_ARG π⋆italic_β ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) end_ARG ≤ ϕ- 1 ( 1 + β- 1 Rsansserif_max ) ≤ 1 + β- 1 Rsansserif_max , where we have used that ϕ−1(z)≤zsuperscriptitalic-ϕ1φ^-1(z)≤zϕ- 1 ( z ) ≤ z for z≥11z≥1z ≥ 1 and ϕ−1(z)≥ez−esuperscriptitalic-ϕ1superscriptφ^-1(z)≥e^z-eϕ- 1 ( z ) ≥ eitalic_z - e for z≤11z≤1z ≤ 1 (by Proposition 4.1). ∎ Appendix H Proofs for :general_preference H.1 Proof of Theorem 7.1 Proof of Theorem 7.1. We consider a family of instances in which there is a single context (prompt) =∅X=\ \X = ∅ and four actions (responses) =a,b,c,dA=\a,b,c,d\A = a , b , c , d . We consider the reference policy πsubscript _ refπsansserif_ref given by π(a′∣x)=1C, if a′=a or a′=b,1−2C, if a′=c.subscriptconditionalsuperscript′cases1 if a′=a or a′=b,12 if a′=c. _ ref(a x)= cases 1C,&% if $a =a$ or $a =b$,\\ 1- 2C,& if $a =c$.\\ casesπsansserif_ref ( a′ ∣ x ) = start_ROW start_CELL divide start_ARG 1 end_ARG start_ARG C end_ARG , end_CELL start_CELL if a′ = a or a′ = b , end_CELL end_ROW start_ROW start_CELL 1 - divide start_ARG 2 end_ARG start_ARG C end_ARG , end_CELL start_CELL if a′ = c . end_CELL end_ROW We consider a preference model class =1,2superscript1superscript2 P= \P^1,P^2 \P = P1 , P2 in which i(a0≻a1∣x)=(1+ℓi(x,a0,a1))/2superscriptsucceedssuperscript0conditionalsuperscript11superscriptℓsuperscript0superscript12P^i(a^0 a^1 x)=(1+ ^i(x,a^0,a^1))/2Pitalic_i ( a0 ≻ a1 ∣ x ) = ( 1 + ℓitalic_i ( x , a0 , a1 ) ) / 2 for a function ℓi(x,a0,a1)∈[−1,+1]superscriptℓsuperscript0superscript111 ^i(x,a^0,a^1)∈[-1,+1]ℓitalic_i ( x , a0 , a1 ) ∈ [ - 1 , + 1 ]. The functions ℓ1superscriptℓ1 ^1ℓ1 and ℓ2superscriptℓ2 ^2ℓ2 are defined as follows (we omit the dependence on x, since there is a single context): ℓ1(a0,a1)=ℓ2(a0,a1)=0,∀a0∈,a1∈a,b,c,formulae-sequencesuperscriptℓ1superscript0superscript1superscriptℓ2superscript0superscript10formulae-sequencefor-allsuperscript0superscript1 ^1(a^0,a^1)= ^2(a^0,a^1)=0, ∀ a^0% ,a^1∈\a,b,c\,ℓ1 ( a0 , a1 ) = ℓ2 ( a0 , a1 ) = 0 , ∀ a0 ∈ A , a1 ∈ a , b , c , ℓ1(a,d)=0,ℓ1(b,d)=−1,ℓ1(c,d)=1formulae-sequencesuperscriptℓ10formulae-sequencesuperscriptℓ11superscriptℓ11 ^1(a,d)=0, ^1(b,d)=-1, ^1(c,d)=1ℓ1 ( a , d ) = 0 , ℓ1 ( b , d ) = - 1 , ℓ1 ( c , d ) = 1 ℓ2(a,d)=−1,ℓ2(b,d)=0,ℓ2(c,d)=−1.formulae-sequencesuperscriptℓ21formulae-sequencesuperscriptℓ20superscriptℓ21 ^2(a,d)=-1, ^2(b,d)=0, ^2(c,d)=-1.ℓ2 ( a , d ) = - 1 , ℓ2 ( b , d ) = 0 , ℓ2 ( c , d ) = - 1 . Note that both functions are skew-symmetric in the sense that ℓ(x,a′,a′)=0ℓsuperscript′0 (x,a ,a )=0ℓ ( x , a′ , a′ ) = 0 and ℓ(x,a0,a1)+ℓ(x,a1,a0)=0ℓsuperscript0superscript1ℓsuperscript1superscript00 (x,a^0,a^1)+ (x,a^1,a^0)=0ℓ ( x , a0 , a1 ) + ℓ ( x , a1 , a0 ) = 0 for all x∈x ∈ X and a0,a1∈superscript0superscript1a^0,a^1 0 , a1 ∈ A. It is straightforward to see that the deterministic policies π1(x)=asuperscriptsubscript1 _ MW^1(x)=aπsansserif_MW1 ( x ) = a and π2(x)=bsuperscriptsubscript2 _ MW^2(x)=bπsansserif_MW2 ( x ) = b are minimax winners for ℓ1superscriptℓ1 ^1ℓ1 and ℓ2superscriptℓ2 ^2ℓ2 respectively. Observe that for both policies, we have ∞π1=∞π2=C.subscriptsuperscriptsuperscriptsubscript1subscriptsuperscriptsuperscriptsubscript2C _ MW^1_∞=C _ MW^2% _∞=C.Citalic_πsansserif_MW start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT∞ = Citalic_πsansserif_MW start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT∞ = C . To proceed, we compute duality gap an arbitrary policy π under 1superscript1P^1P1 and 2superscript2P^2P2. Let (π;) DG(π;P)sansserif_DG ( π ; P ) denote the value of (π) DG(π)sansserif_DG ( π ) when PP is the true preference model. Then we have: maxq∈Δ()l(q,π)subscriptΔ _q∈ (A)l(q,π)maxitalic_q ∈ Δ ( A ) l ( q , π ) =maxq∈Δ()−q(b)π(d)+q(c)π(d)+q(d)π(b)−q(d)π(c),absentsubscriptΔ = _q∈ (A)-q(b)π(d)+q(c)π(d)+q(d)π(b)-% q(d)π(c),= maxitalic_q ∈ Δ ( A ) - q ( b ) π ( d ) + q ( c ) π ( d ) + q ( d ) π ( b ) - q ( d ) π ( c ) , minq∈Δ()l(π,q)subscriptΔ _q∈ (A)l(π,q)minitalic_q ∈ Δ ( A ) l ( π , q ) =minq∈Δ()−π(b)q(d)+π(c)q(d)+π(d)q(b)−π(d)q(c),absentsubscriptΔ = _q∈ (A)-π(b)q(d)+π(c)q(d)+π(d)q(b)-% π(d)q(c),= minitalic_q ∈ Δ ( A ) - π ( b ) q ( d ) + π ( c ) q ( d ) + π ( d ) q ( b ) - π ( d ) q ( c ) , =−maxq∈Δ()−q(b)π(d)+q(c)π(d)+q(d)π(b)−q(d)π(c).absentsubscriptΔ =- _q∈ (A)-q(b)π(d)+q(c)π(d)+q(d)π(b)% -q(d)π(c).= - maxitalic_q ∈ Δ ( A ) - q ( b ) π ( d ) + q ( c ) π ( d ) + q ( d ) π ( b ) - q ( d ) π ( c ) . Therefore we know (π;1)=2maxq∈Δ()q(d)(π(b)−π(c))−π(d)(q(b)−q(c))superscript12subscriptΔ DG(π;P^1)=2 _q∈ (A)q% (d)(π(b)-π(c))-π(d)(q(b)-q(c))sansserif_DG ( π ; P1 ) = 2 maxitalic_q ∈ Δ ( A ) q ( d ) ( π ( b ) - π ( c ) ) - π ( d ) ( q ( b ) - q ( c ) ) Following similar computations, we have (π;2)=2maxq∈Δ()q(d)(π(a)+π(c))−π(d)(q(a)+q(c)).superscript22subscriptΔ DG(π;P^2)=2 _q∈ (A)q% (d)(π(a)+π(c))-π(d)(q(a)+q(c)).sansserif_DG ( π ; P2 ) = 2 maxitalic_q ∈ Δ ( A ) q ( d ) ( π ( a ) + π ( c ) ) - π ( d ) ( q ( a ) + q ( c ) ) . We aim to show that for all policies π, (π;1)+(π;2)≥12superscript1superscript212 DG(π;P^1)+ DG(π;P^2)≥ 12sansserif_DG ( π ; P1 ) + sansserif_DG ( π ; P2 ) ≥ divide start_ARG 1 end_ARG start_ARG 2 end_ARG. To do so, we consider two cases. Going forward, we will use that (π;i)≥0superscript0 DG(π;P^i)≥0sansserif_DG ( π ; Pitalic_i ) ≥ 0. Case (1): ()+()≥12 π(a)+π(c)≥ 12italic_π bold_( italic_a bold_) bold_+ italic_π bold_( italic_c bold_) ≥ divide start_ARG 1 end_ARG start_ARG 2 end_ARG In this case, we have (π;2)≥12superscript212 DG(π;P^2)≥ 12sansserif_DG ( π ; P2 ) ≥ divide start_ARG 1 end_ARG start_ARG 2 end_ARG, and thus (π;1)+(π;2)≥12superscript1superscript212 DG(π;P^1)+ DG(π;P^2)≥ 12sansserif_DG ( π ; P1 ) + sansserif_DG ( π ; P2 ) ≥ divide start_ARG 1 end_ARG start_ARG 2 end_ARG. Case (2): ()+()<14 π(a)+π(c)< 14italic_π bold_( italic_a bold_) bold_+ italic_π bold_( italic_c bold_) bold_< divide start_ARG 1 end_ARG start_ARG 4 end_ARG In this case, let θ:=π(b)−π(c)assignθ =π(b)-π(c)θ := π ( b ) - π ( c ). Then we have (π;1)≥2maxθ,π(d)superscript12 DG(π;P^1)≥ 2 \θ,π(d)\sansserif_DG ( π ; P1 ) ≥ 2 max θ , π ( d ) . We observe that θ+π(d)=π(b)+π(d)−π(c)>34−14=12341412θ+π(d)=π(b)+π(d)-π(c)> 34- 14= 12θ + π ( d ) = π ( b ) + π ( d ) - π ( c ) > divide start_ARG 3 end_ARG start_ARG 4 end_ARG - divide start_ARG 1 end_ARG start_ARG 4 end_ARG = divide start_ARG 1 end_ARG start_ARG 2 end_ARG. This implies that (π;1)>12superscript112 DG(π;P^1)> 12sansserif_DG ( π ; P1 ) > divide start_ARG 1 end_ARG start_ARG 2 end_ARG, and thus (π;1)+(π;2)≥12superscript1superscript212 DG(π;P^1)+ DG(π;P^2)≥ 12sansserif_DG ( π ; P1 ) + sansserif_DG ( π ; P2 ) ≥ divide start_ARG 1 end_ARG start_ARG 2 end_ARG. Having established that all π satisfy (π;1)+(π;2)≥12superscript1superscript212 DG(π;P^1)+ DG(π;P^2)≥ 12sansserif_DG ( π ; P1 ) + sansserif_DG ( π ; P2 ) ≥ divide start_ARG 1 end_ARG start_ARG 2 end_ARG we can apply the Le Cam two-point method (specifically, the variant based on the Bretagnolle-Huber inequality (e.g., Theorem 14.2 in Lattimore and Szepesvári (2020))), which leads to the following inequality infsup∈[(π^;)]≥18exp(−n⋅D(ρ⊗π⊗π⊗1∥ρ⊗π⊗π⊗2)).subscriptinfimumsubscriptsupremumsubscriptsubscript^18⋅subscriptconditionaltensor-productsubscriptsubscriptsuperscript1tensor-productsubscriptsubscriptsuperscript2 _ Alg _P∈ P% E_D_ pref[ DG( π;P% )]≥ 18 (-n· D_ KL (ρ _% ref _ ref ^1\,\|\,ρ % _ ref _ ref ^2 ) ).infsansserif_Alg supcaligraphic_P ∈ P blackboard_ED start_POSTSUBSCRIPT sansserif_pref end_POSTSUBSCRIPT [ sansserif_DG ( over start_ARG π end_ARG ; P ) ] ≥ divide start_ARG 1 end_ARG start_ARG 8 end_ARG exp ( - n ⋅ Dsansserif_KL ( ρ ⊗ πsansserif_ref ⊗ πsansserif_ref ⊗ P1 ∥ ρ ⊗ πsansserif_ref ⊗ πsansserif_ref ⊗ P2 ) ) . It can be observed that D(ρ⊗π⊗π⊗1∥ρ⊗π⊗π⊗2)=0subscriptconditionaltensor-productsubscriptsubscriptsuperscript1tensor-productsubscriptsubscriptsuperscript20D_ KL (ρ _ ref _ ref% ^1\,\|\,ρ _ ref _ % ref ^2 )=0Dsansserif_KL ( ρ ⊗ πsansserif_ref ⊗ πsansserif_ref ⊗ P1 ∥ ρ ⊗ πsansserif_ref ⊗ πsansserif_ref ⊗ P2 ) = 0, since ℓ1(a0,a1)=ℓ2(a0,a1)=0superscriptℓ1superscript0superscript1superscriptℓ2superscript0superscript10 ^1(a^0,a^1)= ^2(a^0,a^1)=0ℓ1 ( a0 , a1 ) = ℓ2 ( a0 , a1 ) = 0 for all a0,a1∈a,b,csuperscript0superscript1a^0,a^1∈\a,b,c\a0 , a1 ∈ a , b , c , and πsubscript _ refπsansserif_ref is supported on a,b,c\a,b,c\ a , b , c . We conclude that any policy derived from subscriptD_ prefDsansserif_pref must have [(π^;i)]≥18^superscript18 E [ DG( π; % P^i) ]≥ 18blackboard_E [ sansserif_DG ( over start_ARG π end_ARG ; Pitalic_i ) ] ≥ divide start_ARG 1 end_ARG start_ARG 8 end_ARG for some i. ∎ H.2 Proof of Theorem 7.2 Proof of Theorem 7.2. Let π~~ πover~ start_ARG π end_ARG be the global best response of π^ πover start_ARG π end_ARG: π~=argmaxπ∈Πx∼ρ,a∼π(x),b∼π^(x)[ℓ⋆(x,a,b)],~subscriptargmaxΠsubscriptformulae-sequencesimilar-toformulae-sequencesimilar-tosimilar-to^delimited-[]superscriptℓ⋆ π= argmax_π∈ E_x% ρ,a π(x),b π(x) [ (x,a,b) ],over~ start_ARG π end_ARG = argmaxitalic_π ∈ Π blackboard_Ex ∼ ρ , a ∼ π ( x ) , b ∼ over start_ARG π end_ARG ( x ) [ ℓ⋆ ( x , a , b ) ] , and let π~Csubscript~ π_Cover~ start_ARG π end_ARGC be the best response within ΠCsubscriptΠ _CΠitalic_C of π^ πover start_ARG π end_ARG where C≥11C≥ 1C ≥ 1 (recall that ΠC:=π:maxx∈Dχ2(π(x)∥π(x))≤CassignsubscriptΠconditional-setsubscriptsubscriptsuperscript2conditionalsubscript _C:=\π: _x D_χ^2 (π(x)\;\|\; _% ref(x) )≤ C\Πitalic_C := π : maxitalic_x ∈ X Ditalic_χ2 ( π ( x ) ∥ πsansserif_ref ( x ) ) ≤ C denotes the set of policies with bounded χ2superscript2χ^2χ2-divergence w.r.t. πsubscript _ refπsansserif_ref): π~C=argmaxπ∈ΠCx∼ρ,a∼π(x),b∼π^(x)[ℓ⋆(x,a,b)].subscript~subscriptargmaxsubscriptΠsubscriptformulae-sequencesimilar-toformulae-sequencesimilar-tosimilar-to^delimited-[]superscriptℓ⋆ π_C= argmax_π∈ _C% E_x ρ,a π(x),b π(x) [ (x,a% ,b) ].over~ start_ARG π end_ARGC = argmaxitalic_π ∈ Π start_POSTSUBSCRIPT C end_POSTSUBSCRIPT blackboard_Ex ∼ ρ , a ∼ π ( x ) , b ∼ over start_ARG π end_ARG ( x ) [ ℓ⋆ ( x , a , b ) ] . Recall that r¯t(x,a):=b∼πt(x)[ℓ^(x,a,b)]assignsuperscript¯subscriptsimilar-tosuperscriptdelimited-[]^ℓ r^t(x,a):=E_b π^t(x)[ (x,a,b)]over¯ start_ARG r end_ARGt ( x , a ) := blackboard_Eb ∼ πitalic_t ( x ) [ over start_ARG ℓ end_ARG ( x , a , b ) ]. Then we know ℓ⋆(π~,π^)=superscriptℓ⋆~^absent ( π, π)=ℓ⋆ ( over~ start_ARG π end_ARG , over start_ARG π end_ARG ) = (π^,C)+1T∑t=1T(r^t(π~C)−r^t(πt))⏟(1)+1T∑t=1T(ℓ⋆(π~C,πt)−ℓ^(π~C,πt))⏟(2)^subscript⏟1superscriptsubscript1superscript^subscript~superscript^superscript1subscript⏟1superscriptsubscript1superscriptℓ⋆subscript~superscript^ℓsubscript~superscript2 subopt( π,C)+ 1T _t=1% ^T ( r^t( π_C)- r^t(π^t) % )_(1)+ 1T _t=1^T ( ( % π_C,π^t)- ( π_C,π^t) )_(2)sansserif_subopt ( over start_ARG π end_ARG , C ) + under⏟ start_ARG divide start_ARG 1 end_ARG start_ARG T end_ARG ∑t = 1T ( over start_ARG r end_ARGt ( over~ start_ARG π end_ARGC ) - over start_ARG r end_ARGt ( πitalic_t ) ) end_ARG( 1 ) + under⏟ start_ARG divide start_ARG 1 end_ARG start_ARG T end_ARG ∑t = 1T ( ℓ⋆ ( over~ start_ARG π end_ARGC , πitalic_t ) - over start_ARG ℓ end_ARG ( over~ start_ARG π end_ARGC , πitalic_t ) ) end_ARG( 2 ) +1T∑t=1T(r¯t(π~C)−r^t(π~C))⏟(3)+1T∑t=1T(r^t(πt)−r¯t(πt))⏟(4),subscript⏟1superscriptsubscript1superscript¯subscript~superscript^subscript~3subscript⏟1superscriptsubscript1superscript^superscriptsuperscript¯superscript4 + 1T _t=1^T( r^t(% π_C)- r^t( π_C))_(3)+ % 1T _t=1^T( r^t(π^t)- r^t(π^t))% _(4),+ under⏟ start_ARG divide start_ARG 1 end_ARG start_ARG T end_ARG ∑t = 1T ( over¯ start_ARG r end_ARGt ( over~ start_ARG π end_ARGC ) - over start_ARG r end_ARGt ( over~ start_ARG π end_ARGC ) ) end_ARG( 3 ) + under⏟ start_ARG divide start_ARG 1 end_ARG start_ARG T end_ARG ∑t = 1T ( over start_ARG r end_ARGt ( πitalic_t ) - over¯ start_ARG r end_ARGt ( πitalic_t ) ) end_ARG( 4 ) , (48) where r(π):=x∼ρ,a∼π(x)[r(x,a)]assignsubscriptformulae-sequencesimilar-tosimilar-todelimited-[]r(π):=E_x ρ,a π(x)[r(x,a)]r ( π ) := blackboard_Ex ∼ ρ , a ∼ π ( x ) [ r ( x , a ) ]. The decomposition utilizes the fact that r¯t(πt)=0superscript¯superscript0 r^t(π^t)=0over¯ start_ARG r end_ARGt ( πitalic_t ) = 0 and r¯t(π~C)=ℓ^(π~C,πt)superscript¯subscript~^ℓsubscript~superscript r^t( π_C)= ( π_C,π^% t)over¯ start_ARG r end_ARGt ( over~ start_ARG π end_ARGC ) = over start_ARG ℓ end_ARG ( over~ start_ARG π end_ARGC , πitalic_t ). This implies that we only need to bound term (1)(2)(3)(4) in Eq. 48 to upper bound the gap of π^ πover start_ARG π end_ARG. Bounding term (1) Let gx(p)subscriptg_x(p)gitalic_x ( p ) to denote the mixed divergence βDfχmix(p(x)∥π(x))subscriptsubscriptsubscriptmixconditionalsubscriptβ D_f_ _ mix (p(x)\,\|\, _ ref(x) )β Ditalic_f start_POSTSUBSCRIPT χ start_POSTSUBSCRIPT mix end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( p ( x ) ∥ πsansserif_ref ( x ) ). Then we have the following guarantee on regularized policy mirror descent (formal version of LABEL:lem:md-general-informal): Lemma H.1. For any C≥00C≥ 0C ≥ 0, we have for all policy π∈ΠCsubscriptΠπ∈ _Cπ ∈ Πitalic_C that 1T∑t=1T(r^t(π)−r^t(πt))≤1superscriptsubscript1superscript^superscript^superscriptabsent 1T _t=1^T ( r^t(π)- r^% t(π^t) ) start_ARG 1 end_ARG start_ARG T end_ARG ∑t = 1T ( over start_ARG r end_ARGt ( π ) - over start_ARG r end_ARGt ( πitalic_t ) ) ≤ 2βCηT+2βC−1T∑t=1T+1x∼ρ[gx(πt)]221superscriptsubscript11subscriptsimilar-todelimited-[]subscriptsuperscript 2β Cη T+2β C- 1T _t=1^T+1% E_x ρ[g_x(π^t)]divide start_ARG 2 β C end_ARG start_ARG η T end_ARG + 2 β C - divide start_ARG 1 end_ARG start_ARG T end_ARG ∑t = 1T + 1 blackboard_Ex ∼ ρ [ gitalic_x ( πitalic_t ) ] +η2β+1T∑t=1Tx∼ρ[⟨r^t(x,⋅)−Gt(πt+1,x,⋅),π(x)−πt+1(x)⟩],21superscriptsubscript1subscriptsimilar-todelimited-[]superscript^⋅superscriptsuperscript1⋅superscript1 + η2β+ 1T _t=1^TE_x % ρ [ r^t(x,·)-G^t(π^t+1,x,·),π(x% )-π^t+1(x) ],+ divide start_ARG η end_ARG start_ARG 2 β end_ARG + divide start_ARG 1 end_ARG start_ARG T end_ARG ∑t = 1T blackboard_Ex ∼ ρ [ ⟨ over start_ARG r end_ARGt ( x , ⋅ ) - Gitalic_t ( πitalic_t + 1 , x , ⋅ ) , π ( x ) - πitalic_t + 1 ( x ) ⟩ ] , where Gt(π,x,a):=β((1+1η)ϕ(π(a|x)π(a|x))−1ηϕ(πt(a|x)π(a|x)))assignsuperscript11italic-ϕconditionalsubscriptconditional1italic-ϕsuperscriptconditionalsubscriptconditionalG^t(π,x,a):=β ((1+ 1η)φ ( π(a|x) _% ref(a|x) )- 1ηφ ( π^t(a|x) _% ref(a|x) ) )Gitalic_t ( π , x , a ) := β ( ( 1 + divide start_ARG 1 end_ARG start_ARG η end_ARG ) ϕ ( divide start_ARG π ( a | x ) end_ARG start_ARG πsansserif_ref ( a | x ) end_ARG ) - divide start_ARG 1 end_ARG start_ARG η end_ARG ϕ ( divide start_ARG πitalic_t ( a | x ) end_ARG start_ARG πsansserif_ref ( a | x ) end_ARG ) ) for all π∈Π,x∈,a∈formulae-sequenceΠformulae-sequenceπ∈ ,x ,a π ∈ Π , x ∈ X , a ∈ A. To simplify writing, we use π¯t+1superscript¯1 π^t+1over¯ start_ARG π end_ARGt + 1 to denote the minimizer of the following regularized RL objective: π¯t+1(x):=argminp∈Δ()⟨−r^t(x,⋅),p⟩+βDfχmix(p∥π(x))+βηBx(p,πt),∀x∈.formulae-sequenceassignsuperscript¯1subscriptΔsuperscript^⋅subscriptsubscriptsubscriptmixconditionalsubscriptsubscriptsuperscriptfor-all π^t+1(x):= _p∈ (X) % - r^t(x,·),p +β D_f_ _ mix% (p\,\|\, _ ref(x) )+ βηB_x(p,π^t% ), ∀ x .over¯ start_ARG π end_ARGt + 1 ( x ) := arg minitalic_p ∈ Δ ( X ) ⟨ - over start_ARG r end_ARGt ( x , ⋅ ) , p ⟩ + β Ditalic_f start_POSTSUBSCRIPT χ start_POSTSUBSCRIPT mix end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( p ∥ πsansserif_ref ( x ) ) + divide start_ARG β end_ARG start_ARG η end_ARG Bitalic_x ( p , πitalic_t ) , ∀ x ∈ X . Then Assumption 7.2 indicates that π¯t+1∈Πsuperscript¯1Π π^t+1∈ ¯ start_ARG π end_ARGt + 1 ∈ Π for all t∈[T]delimited-[]t∈[T]t ∈ [ T ]. In addition, by introducing Lagrangian multipliers into the above optimization problem and following similar arguments in the proof of Lemma F.4, we know fπ¯t+1,πtβ,η(x,a,b)−(r^t(x,a)−r^t(x,b))=0,∀x∈,a,b∈.formulae-sequencesubscriptsuperscriptsuperscript¯1superscriptsuperscript^superscript^0formulae-sequencefor-all f^β,η_ π^t+1,π^t(x,a,b)-( r% ^t(x,a)- r^t(x,b))=0, ∀ x ,a,b∈% A.fitalic_β , ηover¯ start_ARG π end_ARGt + 1 , πitalic_t ( x , a , b ) - ( over start_ARG r end_ARGt ( x , a ) - over start_ARG r end_ARGt ( x , b ) ) = 0 , ∀ x ∈ X , a , b ∈ A . (49) Recall that by definition fπ,πtβ,η(x,a,b)=Gt(π,x,a)−Gt(π,x,b)subscriptsuperscriptsuperscriptsuperscriptsuperscriptf^β,η_π,π^t(x,a,b)=G^t(π,x,a)-G^t(π,x,b)fitalic_β , ηitalic_π , πitalic_t ( x , a , b ) = Gitalic_t ( π , x , a ) - Gitalic_t ( π , x , b ) for all policies π∈Ππ∈ π ∈ Π. This implies that we have x∼ρ[⟨r^t(x,⋅)−Gt(πt+1,x,⋅),π(x)−πt+1(x)⟩]subscriptsimilar-todelimited-[]superscript^⋅superscriptsuperscript1⋅superscript1 _x ρ [ r^t(x,·)-% G^t(π^t+1,x,·),π(x)-π^t+1(x) ]blackboard_Ex ∼ ρ [ ⟨ over start_ARG r end_ARGt ( x , ⋅ ) - Gitalic_t ( πitalic_t + 1 , x , ⋅ ) , π ( x ) - πitalic_t + 1 ( x ) ⟩ ] = == x∼ρ[⟨r^t(x,⋅)−Gt(πt+1,x,⋅),π(x)−π(x)⟩]+x∼ρ[⟨r^t(x,⋅)−Gt(πt+1,x,⋅),π(x)−πt+1(x)⟩]subscriptsimilar-todelimited-[]superscript^⋅superscriptsuperscript1⋅subscriptsubscriptsimilar-todelimited-[]superscript^⋅superscriptsuperscript1⋅subscriptsuperscript1 _x ρ [ r^t(x,·)-% G^t(π^t+1,x,·),π(x)- _ ref(x) ]+% E_x ρ [ r^t(x,·)-G^t(π^t+% 1,x,·), _ ref(x)-π^t+1(x) ]blackboard_Ex ∼ ρ [ ⟨ over start_ARG r end_ARGt ( x , ⋅ ) - Gitalic_t ( πitalic_t + 1 , x , ⋅ ) , π ( x ) - πsansserif_ref ( x ) ⟩ ] + blackboard_Ex ∼ ρ [ ⟨ over start_ARG r end_ARGt ( x , ⋅ ) - Gitalic_t ( πitalic_t + 1 , x , ⋅ ) , πsansserif_ref ( x ) - πitalic_t + 1 ( x ) ⟩ ] = == (fπ¯t+1,πtβ,η−fπt+1,πtβ,η)(ρ,π,π)⏟(5)+(fπt+1,πtβ,η−fπ¯t+1,πtβ,η)(ρ,πt+1,π)⏟(6),subscript⏟subscriptsuperscriptsuperscript¯1superscriptsubscriptsuperscriptsuperscript1superscriptsubscript5subscript⏟subscriptsuperscriptsuperscript1superscriptsubscriptsuperscriptsuperscript¯1superscriptsuperscript1subscript6 (f^β,η_ π^t+1,π^t-f^% β,η_π^t+1,π^t)(ρ,π, _ ref)_(5)+% (f^β,η_π^t+1,π^t-f^β,η_ π% ^t+1,π^t)(ρ,π^t+1, _ ref)_(6),under⏟ start_ARG ( fitalic_β , ηover¯ start_ARG π end_ARGt + 1 , πitalic_t - fitalic_β , ηitalic_πitalic_t + 1 , πitalic_t ) ( ρ , π , πsansserif_ref ) end_ARG( 5 ) + under⏟ start_ARG ( fitalic_β , ηitalic_πitalic_t + 1 , πitalic_t - fitalic_β , ηover¯ start_ARG π end_ARGt + 1 , πitalic_t ) ( ρ , πitalic_t + 1 , πsansserif_ref ) end_ARG( 6 ) , where we use f(ρ,π,π′)superscript′f(ρ,π,π )f ( ρ , π , π′ ) to denote the expectation x∼ρ,a∼π(x),b∼π′(x)[f(x,a,b)]subscriptformulae-sequencesimilar-toformulae-sequencesimilar-tosimilar-tosuperscript′delimited-[]E_x ρ,a π(x),b π (x)[f(x,a,b)]blackboard_Ex ∼ ρ , a ∼ π ( x ) , b ∼ π′ ( x ) [ f ( x , a , b ) ] and the last step utilizes Eq. 49. Therefore, to bound term (1), we need to bound term (5) and (6) respectively. To simplify writing, we define L(π,π′,π′)superscript′L(π,π ,π )L ( π , π′ , π′ ′ ) as follows: L(π,π′,π′):=x∼ρ,a∼π(x),b∼π(x)[(4(fπ,π′β,η(x,a,b))−4(fπ′,π′β,η(x,a,b)))2],assignsuperscript′subscriptformulae-sequencesimilar-toformulae-sequencesimilar-tosubscriptsimilar-tosubscriptdelimited-[]superscriptsubscript4subscriptsuperscriptsuperscript′subscript4subscriptsuperscriptsuperscript′2 L(π,π ,π ):=E_x ρ,a% _ ref(x),b _ ref(x) [ ( clip% _4(f^β,η_π,π (x,a,b))- clip_4(f^% β,η_π ,π (x,a,b)) )^2 ],L ( π , π′ , π′ ′ ) := blackboard_Ex ∼ ρ , a ∼ π start_POSTSUBSCRIPT sansserif_ref ( x ) , b ∼ πsansserif_ref ( x ) end_POSTSUBSCRIPT [ ( sansserif_clip4 ( fitalic_β , ηitalic_π , π′ ′ ( x , a , b ) ) - sansserif_clip4 ( fitalic_β , ηitalic_π′ , π′ ′ ( x , a , b ) ) )2 ] , Note that we have the following guarantee of least squares regression from the literature (Lemma 15 in Song et al. (2022)) Lemma H.2 (least squares regression). Let (yi,zi)i=1Ksuperscriptsubscriptsubscriptsubscript1\(y_i,z_i)\_i=1^K ( yitalic_i , zitalic_i ) i = 1K be a dataset of K points where each point are independently sampled from yi∼μsimilar-tosubscripty_i _i ∼ μ and zi∼p(⋅|yi):=h∗(yi)+εiz_i p(·|y_i):=h^*(y_i)+ _izitalic_i ∼ p ( ⋅ | yitalic_i ) := h∗ ( yitalic_i ) + εitalic_i. Let ℋ:→[−R,R]:ℋ→H:Y→[-R,R]H : Y → [ - R , R ] be a real valued functions where h∗∈ℋsuperscriptℎℋh^* ∗ ∈ H and R>00R>0R > 0. Then if εii=1Ksuperscriptsubscriptsubscript1\ _i\_i=1^K εitalic_i i = 1K are independent random variables such that [zi|yi]=h∗(yi)delimited-[]conditionalsubscriptsubscriptsuperscriptℎsubscriptE[z_i|y_i]=h^*(y_i)blackboard_E [ zitalic_i | yitalic_i ] = h∗ ( yitalic_i ), the least squares solution h^=argminh∈ℋ∑i=1K(h(yi)−zi)2^ℎsubscriptargminℎℋsuperscriptsubscript1superscriptℎsubscriptsubscript2 h= argmin_h _i=1^K(h(y_i)-% z_i)^2over start_ARG h end_ARG = argminitalic_h ∈ H ∑i = 1K ( h ( yitalic_i ) - zitalic_i )2 satisfies with probability at least 1−δ11- 1 - δ that x∼μ[(h^(y)−h∗(y))2]≲R2log(|ℋ|/δ)K.less-than-or-similar-tosubscriptsimilar-todelimited-[]superscript^ℎsuperscriptℎ2superscript2ℋ _x μ[( h(y)-h^*(y))^2] % R^2 (|H|/δ)K.blackboard_Ex ∼ μ [ ( over start_ARG h end_ARG ( y ) - h∗ ( y ) )2 ] ≲ divide start_ARG R2 log ( | H | / δ ) end_ARG start_ARG K end_ARG . The proof of the above lemma is omitted. Applying Lemma H.2 to the least sqaures solution πt+1superscript1π^t+1πitalic_t + 1, we have the following concentration lemma: Lemma H.3 (concentration in optimization). Suppose Assumption 7.2 and Assumption 7.3 hold. Then with probability at least 1−δ/4141-δ/41 - δ / 4, we have for all policy t∈[T]delimited-[]t∈[T]t ∈ [ T ] that L(πt+1,π¯t+1,πt)≤Clog(|Π|/δ)m:=ε2,superscript1superscript¯1superscriptsubscriptΠassignsuperscriptsubscript2 L(π^t+1, π^t+1,π^t)≤ C_ % con (| |/δ)m:= _ md^2,L ( πitalic_t + 1 , over¯ start_ARG π end_ARGt + 1 , πitalic_t ) ≤ divide start_ARG Csansserif_con log ( | Π | / δ ) end_ARG start_ARG m end_ARG := εsansserif_md2 , where C>0subscript0C_ con>0Csansserif_con > 0 is a universal constant. In the following discussion, we use ℰ1subscriptℰ1E_1E1 to denote the event in Lemma H.3. Then under ℰ1subscriptℰ1E_1E1, by following the same arguments in the proof of Lemma F.3, we have the following bound on ‖fπ¯t+1,πtβ,η−fπt+1,πtβ,η‖1,π×πsubscriptnormsubscriptsuperscriptsuperscript¯1superscriptsubscriptsuperscriptsuperscript1superscript1subscript\|f^β,η_ π^t+1,π^t-f^β,η_π^t+1,π^% t\|_1,π× _ ref∥ fitalic_β , ηover¯ start_ARG π end_ARGt + 1 , πitalic_t - fitalic_β , ηitalic_πitalic_t + 1 , πitalic_t ∥1 , π × π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT: ‖fπ¯t+1,πtβ,η−fπt+1,πtβ,η‖1,π×π≤V(1+2Dχ2(π∥π))ε2,∀π∈Π,t∈[T].formulae-sequencesubscriptnormsubscriptsuperscriptsuperscript¯1superscriptsubscriptsuperscriptsuperscript1superscript1subscriptsubscript12subscriptsuperscript2conditionalsubscriptsuperscriptsubscript2formulae-sequencefor-allΠdelimited-[] \|f^β,η_ π^t+1,π^t-f^β,η_% π^t+1,π^t\|_1,π× _ ref≤ V_ max % (1+2D_χ^2 (π\;\|\; _ ref ) )% _ md^2, ∀π∈ ,t∈[T].∥ fitalic_β , ηover¯ start_ARG π end_ARGt + 1 , πitalic_t - fitalic_β , ηitalic_πitalic_t + 1 , πitalic_t ∥1 , π × π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT ≤ Vsansserif_max square-root start_ARG ( 1 + 2 Ditalic_χ2 ( π ∥ πsansserif_ref ) ) εsansserif_md2 end_ARG , ∀ π ∈ Π , t ∈ [ T ] . (50) Therefore, with Eq. 50 we know that conditioned on ℰ1subscriptℰ1E_1E1, for any policy π∈ΠCsubscriptΠπ∈ _Cπ ∈ Πitalic_C we have (5)≤V3Cε2,(6)≤V(1+2Dχ2(πt+1∥π))ε2≤V2ε2β+12x∼ρ[gx(πt+1)]+Vε,formulae-sequence5subscript3superscriptsubscript26subscript12subscriptsuperscript2conditionalsuperscript1subscriptsuperscriptsubscript2superscriptsubscript2superscriptsubscript212subscriptsimilar-todelimited-[]subscriptsuperscript1subscriptsubscript (5)≤ V_ max 3C _ md^2,% (6)≤ V_ max (1+2D_χ^2 (π^t+1\;\|\;% _ ref ) ) _ md^2≤ V_% max^2 _ md^2β+ 12E_% x ρ[g_x(π^t+1)]+V_ max _ md,( 5 ) ≤ Vsansserif_max square-root start_ARG 3 C εsansserif_md2 end_ARG , ( 6 ) ≤ Vsansserif_max square-root start_ARG ( 1 + 2 Ditalic_χ2 ( πitalic_t + 1 ∥ πsansserif_ref ) ) εsansserif_md2 end_ARG ≤ divide start_ARG Vsansserif_max2 εsansserif_md2 end_ARG start_ARG β end_ARG + divide start_ARG 1 end_ARG start_ARG 2 end_ARG blackboard_Ex ∼ ρ [ gitalic_x ( πitalic_t + 1 ) ] + Vsansserif_max εsansserif_md , where we use AM-GM inequality in the last step, the definition of gx(π):=βDfχmix(π(⋅|x)∥π(⋅|x))g_x(π):=β D_f_ _ mix (π(·|x)\,\|\, _% ref(·|x) )gitalic_x ( π ) := β Ditalic_f start_POSTSUBSCRIPT χ start_POSTSUBSCRIPT mix end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( π ( ⋅ | x ) ∥ πsansserif_ref ( ⋅ | x ) ), and Dfχmix(p(x)∥π(x))≥Dχ2(p(x)∥π(x))subscriptsubscriptsubscriptmixconditionalsubscriptsubscriptsuperscript2conditionalsubscriptD_f_ _ mix (p(x)\,\|\, _ ref(x) )≥ D_% χ^2 (p(x)\;\|\; _ ref(x) )Ditalic_f start_POSTSUBSCRIPT χ start_POSTSUBSCRIPT mix end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( p ( x ) ∥ πsansserif_ref ( x ) ) ≥ Ditalic_χ2 ( p ( x ) ∥ πsansserif_ref ( x ) ) since KL is non-negative In summary, conditioned on ℰ1subscriptℰ1E_1E1, we have (1)≤1absent (1)≤( 1 ) ≤ 2βCηT+2βC−12T∑t=1T+1x∼ρ[gx(πt)]+η2β+V4Cε2+V2ε2β.2212superscriptsubscript11subscriptsimilar-todelimited-[]subscriptsuperscript2subscript4superscriptsubscript2superscriptsubscript2superscriptsubscript2 2β Cη T+2β C- 12T _t=1^T+1% E_x ρ[g_x(π^t)]+ η2β+V_ max% 4C _ md^2+ V_ max^2 _% md^2β.divide start_ARG 2 β C end_ARG start_ARG η T end_ARG + 2 β C - divide start_ARG 1 end_ARG start_ARG 2 T end_ARG ∑t = 1T + 1 blackboard_Ex ∼ ρ [ gitalic_x ( πitalic_t ) ] + divide start_ARG η end_ARG start_ARG 2 β end_ARG + Vsansserif_max square-root start_ARG 4 C εsansserif_md2 end_ARG + divide start_ARG Vsansserif_max2 εsansserif_md2 end_ARG start_ARG β end_ARG . (51) Bounding term (2) From Cauchy-Schwartz’s inequality, we have ℓ⋆(π~C,πt)−ℓ^(π~C,πt)superscriptℓ⋆subscript~superscript^ℓsubscript~superscript ( π_C,π^t)- (% π_C,π^t)ℓ⋆ ( over~ start_ARG π end_ARGC , πitalic_t ) - over start_ARG ℓ end_ARG ( over~ start_ARG π end_ARGC , πitalic_t ) ≤x∼ρ,a∼π(x),b∼π(x)[(ℓ⋆(x,a,b)−ℓ^(x,a,b))2](1+2Dχ2(ρ⊗π~C⊗πt∥ρ⊗π⊗π)),absentsubscriptformulae-sequencesimilar-toformulae-sequencesimilar-tosubscriptsimilar-tosubscriptdelimited-[]superscriptsuperscriptℓ⋆^ℓ212subscriptsuperscript2conditionaltensor-productsubscript~superscripttensor-productsubscriptsubscript ≤ E_x ρ,a _ ref(x),% b _ ref(x)[( (x,a,b)- (x,a,b))^2]% (1+2D_χ^2 (ρ π_C π^t\;\|\;% ρ _ ref _ ref ) ),≤ square-root start_ARG blackboard_Ex ∼ ρ , a ∼ π start_POSTSUBSCRIPT sansserif_ref ( x ) , b ∼ πsansserif_ref ( x ) end_POSTSUBSCRIPT [ ( ℓ⋆ ( x , a , b ) - over start_ARG ℓ end_ARG ( x , a , b ) )2 ] ( 1 + 2 Ditalic_χ2 ( ρ ⊗ over~ start_ARG π end_ARGC ⊗ πitalic_t ∥ ρ ⊗ πsansserif_ref ⊗ πsansserif_ref ) ) end_ARG , where ρ⊗π1⊗π2tensor-productsubscript1subscript2ρ _1 _2ρ ⊗ π1 ⊗ π2 denotes the joint distribution of (x,a,b)(x,a,b)( x , a , b ) where x∼ρ,a∼π1(x),b∼π2(x)formulae-sequencesimilar-toformulae-sequencesimilar-tosubscript1similar-tosubscript2x ρ,a _1(x),b _2(x)x ∼ ρ , a ∼ π1 ( x ) , b ∼ π2 ( x ) for all π1,π2∈Πsubscript1subscript2Π _1, _2∈ π1 , π2 ∈ Π. Applying the guarantee of least squares regression (Lemma H.2) to the least squares solution ℓ^^ℓ over start_ARG ℓ end_ARG, we have under Assumption 7.1, with probability at least 1−δ/4141-δ/41 - δ / 4, the following event holds: x∼ρ,y0∼π(x),y1∼π(x)[(ℓ^(x,y0,y1)−ℓ⋆(x,y0,y1))2]≤O(ln(|ℒ|/δ)n):=ε2.subscriptformulae-sequencesimilar-toformulae-sequencesimilar-tosuperscript0subscriptsimilar-tosuperscript1subscriptdelimited-[]superscript^ℓsuperscript0superscript1superscriptℓ⋆superscript0superscript12ℒassignsuperscriptsubscript2 _x ρ,y^0 _ ref(x),y^1 π% _ ref(x) [ ( (x,y^0,y^1)- (x,y^% 0,y^1) )^2 ]≤ O ( (|L|/δ)n% ):= _ general^2.blackboard_Ex ∼ ρ , y0 ∼ π start_POSTSUBSCRIPT sansserif_ref ( x ) , y1 ∼ πsansserif_ref ( x ) end_POSTSUBSCRIPT [ ( over start_ARG ℓ end_ARG ( x , y0 , y1 ) - ℓ⋆ ( x , y0 , y1 ) )2 ] ≤ O ( divide start_ARG ln ( | L | / δ ) end_ARG start_ARG n end_ARG ) := εsansserif_general2 . (52) Denote the event in Eq. 52 by ℰ2subscriptℰ2E_2E2. On the other hand, we can obtain that: 1+2Dχ2(ρ⊗π~C⊗πt∥ρ⊗π⊗π)12subscriptsuperscript2conditionaltensor-productsubscript~superscripttensor-productsubscriptsubscript 1+2D_χ^2 (ρ π_C π^t% \;\|\;ρ _ ref _ ref )1 + 2 Ditalic_χ2 ( ρ ⊗ over~ start_ARG π end_ARGC ⊗ πitalic_t ∥ ρ ⊗ πsansserif_ref ⊗ πsansserif_ref ) =∑xρ(x)∑a(π~C(a|x))2π(a|x)∑b(πt(b|x))2π(b|x)absentsubscriptsubscriptsuperscriptsubscript~conditional2subscriptconditionalsubscriptsuperscriptsuperscriptconditional2subscriptconditional = _xρ(x) _a ( π_C(a|x))^2π% _ ref(a|x) _b (π^t(b|x))^2 _ ref(b|x)= ∑x ρ ( x ) ∑a divide start_ARG ( over~ start_ARG π end_ARGC ( a | x ) )2 end_ARG start_ARG πsansserif_ref ( a | x ) end_ARG ∑b divide start_ARG ( πitalic_t ( b | x ) )2 end_ARG start_ARG πsansserif_ref ( b | x ) end_ARG =∑xρ(x)(1+2Dχ2(π~C(x)∥π(x)))(1+2Dχ2(πt(x)∥π(x)))absentsubscript12subscriptsuperscript2conditionalsubscript~subscript12subscriptsuperscript2conditionalsuperscriptsubscript = _xρ(x) (1+2D_χ^2 ( π_C(x)% \;\|\; _ ref(x) ) ) (1+2D_χ^2 (π^t(% x)\;\|\; _ ref(x) ) )= ∑x ρ ( x ) ( 1 + 2 Ditalic_χ2 ( over~ start_ARG π end_ARGC ( x ) ∥ πsansserif_ref ( x ) ) ) ( 1 + 2 Ditalic_χ2 ( πitalic_t ( x ) ∥ πsansserif_ref ( x ) ) ) ≤6C(x∼ρ[Dχ2(πt(x)∥π(x))]+1)absent6subscriptsimilar-todelimited-[]subscriptsuperscript2conditionalsuperscriptsubscript1 ≤ 6C (E_x ρ [D_χ^2 (π^t% (x)\;\|\; _ ref(x) ) ]+1 )≤ 6 C ( blackboard_Ex ∼ ρ [ Ditalic_χ2 ( πitalic_t ( x ) ∥ πsansserif_ref ( x ) ) ] + 1 ) where the last step is due to π~C∈ΠCsubscript~subscriptΠ π_C∈ _Cover~ start_ARG π end_ARGC ∈ Πitalic_C. Therefore, conditioned on ℰ2subscriptℰ2E_2E2, we have ℓ⋆(π~C,πt)−ℓ^(π~,πt)≤6Cx∼ρ[Dχ2(πt(x)∥π(x))]ε2+6Cε2superscriptℓ⋆subscript~superscript^ℓ~superscript6subscriptsimilar-todelimited-[]subscriptsuperscript2conditionalsuperscriptsubscriptsuperscriptsubscript26superscriptsubscript2 ( π_C,π^t)- (% π,π^t)≤ 6CE_x ρ [D_χ^2% (π^t(x)\;\|\; _ ref(x) ) ] _% general^2+ 6C _ general^2ℓ⋆ ( over~ start_ARG π end_ARGC , πitalic_t ) - over start_ARG ℓ end_ARG ( over~ start_ARG π end_ARG , πitalic_t ) ≤ square-root start_ARG 6 C blackboard_Ex ∼ ρ [ Ditalic_χ2 ( πitalic_t ( x ) ∥ πsansserif_ref ( x ) ) ] εsansserif_general2 end_ARG + square-root start_ARG 6 C εsansserif_general2 end_ARG ≤12x∼ρ[gx(πt)]+3Cε2β+6Cε2.absent12subscriptsimilar-todelimited-[]subscriptsuperscript3superscriptsubscript26superscriptsubscript2 ≤ 12E_x ρ[g_x(π^t)]+ % 3C _ general^2β+ 6C _ % general^2.≤ divide start_ARG 1 end_ARG start_ARG 2 end_ARG blackboard_Ex ∼ ρ [ gitalic_x ( πitalic_t ) ] + divide start_ARG 3 C εsansserif_general2 end_ARG start_ARG β end_ARG + square-root start_ARG 6 C εsansserif_general2 end_ARG . In summary, we have 1T∑t=1Tℓ⋆(π~C,πt)−ℓ^(π~,πt)≤12T∑t=1Tx∼ρ[gx(πt)]+3Cε2β+6Cε2.1superscriptsubscript1superscriptℓ⋆subscript~superscript^ℓ~superscript12superscriptsubscript1subscriptsimilar-todelimited-[]subscriptsuperscript3superscriptsubscript26superscriptsubscript2 1T _t=1^T ( π_C,π^t% )- ( π,π^t)≤ 12T _t=1^T % E_x ρ[g_x(π^t)]+ 3C _ general^2% β+ 6C _ general^2.divide start_ARG 1 end_ARG start_ARG T end_ARG ∑t = 1T ℓ⋆ ( over~ start_ARG π end_ARGC , πitalic_t ) - over start_ARG ℓ end_ARG ( over~ start_ARG π end_ARG , πitalic_t ) ≤ divide start_ARG 1 end_ARG start_ARG 2 T end_ARG ∑t = 1T blackboard_Ex ∼ ρ [ gitalic_x ( πitalic_t ) ] + divide start_ARG 3 C εsansserif_general2 end_ARG start_ARG β end_ARG + square-root start_ARG 6 C εsansserif_general2 end_ARG . (53) Bounding term (3) Recall that r^t(x,a)=ℓ^(x,a,bt)superscript^^ℓsubscript r^t(x,a)= (x,a,b_t)over start_ARG r end_ARGt ( x , a ) = over start_ARG ℓ end_ARG ( x , a , bitalic_t ) where bt∼πt(x)similar-tosubscriptsuperscriptb_t π^t(x)bitalic_t ∼ πitalic_t ( x ) is an unbiased estimator of r¯tsuperscript¯ r^tover¯ start_ARG r end_ARGt. Fix any policy π∈Ππ∈ π ∈ Π, then from Azuma-Hoeffding’s inequality, we have with probability at least 1−δ′1superscript′1-δ 1 - δ′ that |∑t=1Tr^t(π)−∑t=1Tr¯t(π)|≲Tlog(1/δ′).less-than-or-similar-tosuperscriptsubscript1superscript^superscriptsubscript1superscript¯1superscript′ | _t=1^T r^t(π)- _t=1^T % r^t(π) | T (1/δ ).| ∑t = 1T over start_ARG r end_ARGt ( π ) - ∑t = 1T over¯ start_ARG r end_ARGt ( π ) | ≲ square-root start_ARG T log ( 1 / δ′ ) end_ARG . By union bound, with probability at least 1−δ/4141-δ/41 - δ / 4 we have that for all π∈Ππ∈ π ∈ Π: |∑t=1Tr^t(π)−∑t=1Tr¯t(π)|≲Tlog(|Π|/δ).less-than-or-similar-tosuperscriptsubscript1superscript^superscriptsubscript1superscript¯Π | _t=1^T r^t(π)- _t=1^T % r^t(π) | T (| |/δ).| ∑t = 1T over start_ARG r end_ARGt ( π ) - ∑t = 1T over¯ start_ARG r end_ARGt ( π ) | ≲ square-root start_ARG T log ( | Π | / δ ) end_ARG . Therefore, specifically for π~Csubscript~ π_Cover~ start_ARG π end_ARGC, we have (3)≲log(|Π|/δ)T.less-than-or-similar-to3Π (3) (| |/δ)T.( 3 ) ≲ square-root start_ARG divide start_ARG log ( | Π | / δ ) end_ARG start_ARG T end_ARG end_ARG . (54) Bounding term (4) From Azuma-Hoeffding’s inequality, we have with probability at least 1−δ/4141-δ/41 - δ / 4 that |∑t=1Tr^t(πt)−∑t=1Tr¯t(πt)|≲Tlog(1/δ′).less-than-or-similar-tosuperscriptsubscript1superscript^superscriptsuperscriptsubscript1superscript¯superscript1superscript′ | _t=1^T r^t(π^t)- _t=1^T% r^t(π^t) | T (1/δ ).| ∑t = 1T over start_ARG r end_ARGt ( πitalic_t ) - ∑t = 1T over¯ start_ARG r end_ARGt ( πitalic_t ) | ≲ square-root start_ARG T log ( 1 / δ′ ) end_ARG . Therefore, we have (4)≲log(1/δ)T.less-than-or-similar-to41 (4) (1/δ)T.( 4 ) ≲ square-root start_ARG divide start_ARG log ( 1 / δ ) end_ARG start_ARG T end_ARG end_ARG . (55) Putting everything together Substituting Eq. 51(53)(54)(55) into (48), we have with probability at least 1−δ11- 1 - δ that ℓ⋆(π~,π^)≲(π^,C)+CβηT+Cβ+ηβless-than-or-similar-tosuperscriptℓ⋆~^ ( π, π) subopt% ( π,C)+ Cβη T+Cβ+ ηβℓ⋆ ( over~ start_ARG π end_ARG , over start_ARG π end_ARG ) ≲ sansserif_subopt ( over start_ARG π end_ARG , C ) + divide start_ARG C β end_ARG start_ARG η T end_ARG + C β + divide start_ARG η end_ARG start_ARG β end_ARG +VCε2+V2ε22βsubscriptsuperscriptsubscript2superscriptsubscript2superscriptsubscript22 +V_ max C _ md^2+ V_% max^2 _ md^22β+ Vsansserif_max square-root start_ARG C εsansserif_md2 end_ARG + divide start_ARG Vsansserif_max2 εsansserif_md2 end_ARG start_ARG 2 β end_ARG +Cε2β+Cε2+log|Π|δT.superscriptsubscript2superscriptsubscript2Π + C _ general^2β+ C% _ general^2+ | |δT.+ divide start_ARG C εsansserif_general2 end_ARG start_ARG β end_ARG + square-root start_ARG C εsansserif_general2 end_ARG + square-root start_ARG divide start_ARG log divide start_ARG | Π | end_ARG start_ARG δ end_ARG end_ARG start_ARG T end_ARG end_ARG . By selecting T=mnV2+m,β=1T,η=1T,formulae-sequencesuperscriptsubscript2formulae-sequence11 T= mnnV_ max^2+m, β= 1 % T, η= 1T,T = divide start_ARG m n end_ARG start_ARG n Vsansserif_max2 + m end_ARG , β = divide start_ARG 1 end_ARG start_ARG square-root start_ARG T end_ARG end_ARG , η = divide start_ARG 1 end_ARG start_ARG T end_ARG , we have with probability at least 1−δ11- 1 - δ that ℓ⋆(π~,π^)superscriptℓ⋆~ ( π, π)ℓ⋆ ( over~ start_ARG π end_ARG , over start_ARG π end_ARG ) ≲(π^,C)+C(Vlog(|Π|/δ)m+log(|Π||ℒ|/δ)n)less-than-or-similar-toabsent^subscriptΠℒ subopt( π,C)+C ( V_ % max (| |/δ) m+ (| ||L|/δ)% n )≲ sansserif_subopt ( over start_ARG π end_ARG , C ) + C ( divide start_ARG Vsansserif_max log ( | Π | / δ ) end_ARG start_ARG square-root start_ARG m end_ARG end_ARG + divide start_ARG log ( | Π | | L | / δ ) end_ARG start_ARG square-root start_ARG n end_ARG end_ARG ) Note that due to the skew symmetry of ℓ⋆superscriptℓ⋆ ℓ⋆, we have: minπ∈Πx∼ρ,a∼π^(x),b∼π(x)[ℓ⋆(x,a,b)]=−maxπ∈Πx∼ρ,a∼π(x),b∼π^(x)[ℓ⋆(x,a,b)]=−ℓ⋆(π~,π^).subscriptΠsubscriptformulae-sequencesimilar-toformulae-sequencesimilar-to^similar-todelimited-[]superscriptℓ⋆subscriptΠsubscriptformulae-sequencesimilar-toformulae-sequencesimilar-tosimilar-to^delimited-[]superscriptℓ⋆superscriptℓ⋆~ _π∈ E_x ρ,a π(x),b % π(x) [ (x,a,b) ]=- _π∈ E_x ρ% ,a π(x),b π(x) [ (x,a,b) ]=- ^% ( π, π).minitalic_π ∈ Π blackboard_Ex ∼ ρ , a ∼ over start_ARG π end_ARG ( x ) , b ∼ π ( x ) [ ℓ⋆ ( x , a , b ) ] = - maxitalic_π ∈ Π blackboard_Ex ∼ ρ , a ∼ π ( x ) , b ∼ over start_ARG π end_ARG ( x ) [ ℓ⋆ ( x , a , b ) ] = - ℓ⋆ ( over~ start_ARG π end_ARG , over start_ARG π end_ARG ) . This implies that (π^)≤2ℓ⋆(π~,π^)^2superscriptℓ⋆~ DG( π)≤ 2 ( π, π)sansserif_DG ( over start_ARG π end_ARG ) ≤ 2 ℓ⋆ ( over~ start_ARG π end_ARG , over start_ARG π end_ARG ), which concludes our proof. ∎ H.3 Proofs for Supporting Lemmas Proof of Lemma H.1. First for all t∈[T],s∈formulae-sequencedelimited-[]t∈[T],s ∈ [ T ] , s ∈ S and any policy π∈ΠCsubscriptΠπ∈ _Cπ ∈ Πitalic_C, we have ⟨ηr^t(x),π(x)−πt(x)⟩+ηgx(πt)−ηgx(π)superscript^superscriptsubscriptsuperscriptsubscript η r^t(x),π(x)-π^t(x) % +η g_x(π^t)-η g_x(π)⟨ η over start_ARG r end_ARGt ( x ) , π ( x ) - πitalic_t ( x ) ⟩ + η gitalic_x ( πitalic_t ) - η gitalic_x ( π ) = == ⟨ηr^t(x)−(1+η)∇gx(πt+1)+∇gx(πt),π(x)−πt+1(x)⟩superscript^1∇subscriptsuperscript1∇subscriptsuperscriptsuperscript1 η r^t(x)-(1+η)∇ g_x(π^t+1% )+∇ g_x(π^t),π(x)-π^t+1(x) ⟨ η over start_ARG r end_ARGt ( x ) - ( 1 + η ) ∇ gitalic_x ( πitalic_t + 1 ) + ∇ gitalic_x ( πitalic_t ) , π ( x ) - πitalic_t + 1 ( x ) ⟩ +⟨∇gx(πt+1)−∇gx(πt),π(x)−πt+1(x)⟩⏟(7)+⟨ηr^t(x),πt+1(x)−πt(x)⟩⏟(8)subscript⏟∇subscriptsuperscript1∇subscriptsuperscriptsuperscript17subscript⏟superscript^superscript1superscript8 + ∇ g_x(π^t+1)-∇ g_x(π% ^t),π(x)-π^t+1(x) _(7)+ η% r^t(x),π^t+1(x)-π^t(x) _(8)+ under⏟ start_ARG ⟨ ∇ gitalic_x ( πitalic_t + 1 ) - ∇ gitalic_x ( πitalic_t ) , π ( x ) - πitalic_t + 1 ( x ) ⟩ end_ARG( 7 ) + under⏟ start_ARG ⟨ η over start_ARG r end_ARGt ( x ) , πitalic_t + 1 ( x ) - πitalic_t ( x ) ⟩ end_ARG( 8 ) +⟨η∇gx(πt+1),π(x)−πt+1(x)⟩+ηgx(πt)−ηgx(π)⏟(9),subscript⏟∇subscriptsuperscript1superscript1subscriptsuperscriptsubscript9 + η∇ g_x(π^t+1),π(x)-π^% t+1(x) +η g_x(π^t)-η g_x(π)_(9),+ under⏟ start_ARG ⟨ η ∇ gitalic_x ( πitalic_t + 1 ) , π ( x ) - πitalic_t + 1 ( x ) ⟩ + η gitalic_x ( πitalic_t ) - η gitalic_x ( π ) end_ARG( 9 ) , Note that we have ⟨ηr^t(x)−(1+η)∇gx(πt+1)+∇gx(πt),π(x)−πt+1(x)⟩=η⟨r^t(x,⋅)−Gt(πt+1,x,⋅),π(x)−πt+1(x)⟩superscript^1∇subscriptsuperscript1∇subscriptsuperscriptsuperscript1superscript^⋅superscriptsuperscript1⋅superscript1 η r^t(x)-(1+η)∇ g_x(π^t+1% )+∇ g_x(π^t),π(x)-π^t+1(x) =η % r^t(x,·)-G^t(π^t+1,x,·),π(x)-π^t+1(x) ⟨ η over start_ARG r end_ARGt ( x ) - ( 1 + η ) ∇ gitalic_x ( πitalic_t + 1 ) + ∇ gitalic_x ( πitalic_t ) , π ( x ) - πitalic_t + 1 ( x ) ⟩ = η ⟨ over start_ARG r end_ARGt ( x , ⋅ ) - Gitalic_t ( πitalic_t + 1 , x , ⋅ ) , π ( x ) - πitalic_t + 1 ( x ) ⟩ Next we bound the term (7)(8)(9) respectively. Bounding term (7) Note that we have the following three point lemma: Lemma H.4 (three point lemma). For any p1,p2,p3:↦Δ():subscript1subscript2subscript3maps-toΔp_1,p_2,p_3:X (Y)p1 , p2 , p3 : X ↦ Δ ( Y ), we have for all x∈x ∈ X 1β⟨∇gx(p1)−∇gx(p2),p3(x)−p1(x)⟩=Bx(p3,p2)−Bx(p3,p1)−Bx(p1,p2).1∇subscriptsubscript1∇subscriptsubscript2subscript3subscript1subscriptsubscript3subscript2subscriptsubscript3subscript1subscriptsubscript1subscript2 1β ∇ g_x(p_1)-∇ g_x(p_2% ),p_3(x)-p_1(x) =B_x(p_3,p_2)-B_x(p_3,p_1)-B_x(p% _1,p_2).divide start_ARG 1 end_ARG start_ARG β end_ARG ⟨ ∇ gitalic_x ( p1 ) - ∇ gitalic_x ( p2 ) , p3 ( x ) - p1 ( x ) ⟩ = Bitalic_x ( p3 , p2 ) - Bitalic_x ( p3 , p1 ) - Bitalic_x ( p1 , p2 ) . Proof. By definition, we know βBx(p,p′)=gx(p)−gx(p′)−⟨∇gx(p′),p−p′⟩.subscriptsuperscript′subscriptsubscriptsuperscript′∇subscriptsuperscript′ β B_x(p,p )=g_x(p)-g_x(p )- % ∇ g_x(p ),p-p .β Bitalic_x ( p , p′ ) = gitalic_x ( p ) - gitalic_x ( p′ ) - ⟨ ∇ gitalic_x ( p′ ) , p - p′ ⟩ . Substitute the definition into Lemma H.4 and we can prove the lemma. ∎ From Lemma H.4, we can rewrite (7) as follows: (7)=β(Bx(π,πt)−Bx(π,πt+1)−Bx(πt+1,πt)).7subscriptsuperscriptsubscriptsuperscript1subscriptsuperscript1superscript (7)=β (B_x(π,π^t)-B_x(π,π^t+1)-B_x(π^% t+1,π^t) ).( 7 ) = β ( Bitalic_x ( π , πitalic_t ) - Bitalic_x ( π , πitalic_t + 1 ) - Bitalic_x ( πitalic_t + 1 , πitalic_t ) ) . Bounding term (8) From Cauchy-Schwartz inequality, we have (8)≤∑a∈β(πt+1(a|x)−πt(a|x))22π(a|x)+π(a|x)η2(r^t(x,a))22β≤βBx(πt+1,πt)+η22β,8subscriptsuperscriptsuperscript1conditionalsuperscriptconditional22subscriptconditionalsubscriptconditionalsuperscript2superscriptsuperscript^22subscriptsuperscript1superscriptsuperscript22 (8)≤ _a β(π^t+1(a|x)-π^t(a% |x))^22 _ ref(a|x)+ _ ref(a|x)η^2(% r^t(x,a))^22β≤β B_x(π^t+1,π^t)+ % η^22β,( 8 ) ≤ ∑a ∈ A divide start_ARG β ( πitalic_t + 1 ( a | x ) - πitalic_t ( a | x ) )2 end_ARG start_ARG 2 πsansserif_ref ( a | x ) end_ARG + divide start_ARG πsansserif_ref ( a | x ) η2 ( over start_ARG r end_ARGt ( x , a ) )2 end_ARG start_ARG 2 β end_ARG ≤ β Bitalic_x ( πitalic_t + 1 , πitalic_t ) + divide start_ARG η2 end_ARG start_ARG 2 β end_ARG , where the last step comes from the definition of BxsubscriptB_xBitalic_x. Bounding term (9) Since gxsubscriptg_xgitalic_x is convex, we know ⟨η∇gx(πt+1),π−πt+1⟩≤ηgx(π)−ηgx(πt+1).∇subscriptsuperscript1superscript1subscriptsubscriptsuperscript1 η∇ g_x(π^t+1),π-π^t+1 % ≤η g_x(π)-η g_x(π^t+1).⟨ η ∇ gitalic_x ( πitalic_t + 1 ) , π - πitalic_t + 1 ⟩ ≤ η gitalic_x ( π ) - η gitalic_x ( πitalic_t + 1 ) . This implies that (3)≤η(gx(πt)−gx(πt+1)).3subscriptsuperscriptsubscriptsuperscript1 (3)≤η (g_x(π^t)-g_x(π^t+1) ).( 3 ) ≤ η ( gitalic_x ( πitalic_t ) - gitalic_x ( πitalic_t + 1 ) ) . In summary, for all t∈[T],s∈formulae-sequencedelimited-[]t∈[T],s ∈ [ T ] , s ∈ S and any policy π∈ΠCsubscriptΠπ∈ _Cπ ∈ Πitalic_C, we have ⟨ηr^t(x),π(x)−πt(x)⟩+ηgx(πt)−ηgx(π)≤β(Bx(π,πt)−Bx(π,πt+1))superscript^superscriptsubscriptsuperscriptsubscriptsubscriptsuperscriptsubscriptsuperscript1 η r^t(x),π(x)-π^t(x) % +η g_x(π^t)-η g_x(π)≤β (B_x(π,π^t)-B_x(% π,π^t+1) )⟨ η over start_ARG r end_ARGt ( x ) , π ( x ) - πitalic_t ( x ) ⟩ + η gitalic_x ( πitalic_t ) - η gitalic_x ( π ) ≤ β ( Bitalic_x ( π , πitalic_t ) - Bitalic_x ( π , πitalic_t + 1 ) ) +η(gx(πt)−gx(πt+1))+η22β+η⟨r^t(x,⋅)−Gt(πt+1,x,⋅),π(x)−πt+1(x)⟩.subscriptsuperscriptsubscriptsuperscript1superscript22superscript^⋅superscriptsuperscript1⋅superscript1 +η (g_x(π^t)-g_x(π^t+1) )+ % η^22β+η r^t(x,·)-G^t(π^t+1,x,% ·),π(x)-π^t+1(x) .+ η ( gitalic_x ( πitalic_t ) - gitalic_x ( πitalic_t + 1 ) ) + divide start_ARG η2 end_ARG start_ARG 2 β end_ARG + η ⟨ over start_ARG r end_ARGt ( x , ⋅ ) - Gitalic_t ( πitalic_t + 1 , x , ⋅ ) , π ( x ) - πitalic_t + 1 ( x ) ⟩ . This implies that for any policy π∈ΠCsubscriptΠπ∈ _Cπ ∈ Πitalic_C: ∑t=1T(r^t(π)−r^t(πt))≤superscriptsubscript1superscript^superscript^superscriptabsent _t=1^T ( r^t(π)- r^t(π^t)% )≤∑t = 1T ( over start_ARG r end_ARGt ( π ) - over start_ARG r end_ARGt ( πitalic_t ) ) ≤ Tx∼ρ[gx(π)]−∑t=1T+1x∼ρ[gx(πt)]+βηx∼ρ[Bx(π,π1)]+ηT2βsubscriptsimilar-todelimited-[]subscriptsuperscriptsubscript11subscriptsimilar-todelimited-[]subscriptsuperscriptsubscriptsimilar-todelimited-[]subscriptsuperscript12 TE_x ρ[g_x(π)]- _t=1^T+1E_% x ρ[g_x(π^t)]+ βηE_x ρ [B_x% (π,π^1) ]+ η T2βT blackboard_Ex ∼ ρ [ gitalic_x ( π ) ] - ∑t = 1T + 1 blackboard_Ex ∼ ρ [ gitalic_x ( πitalic_t ) ] + divide start_ARG β end_ARG start_ARG η end_ARG blackboard_Ex ∼ ρ [ Bitalic_x ( π , π1 ) ] + divide start_ARG η T end_ARG start_ARG 2 β end_ARG +∑t=1Tx∼ρ[⟨r^t(x,⋅)−Gt(πt+1,x,⋅),π(x)−πt+1(x)⟩]superscriptsubscript1subscriptsimilar-todelimited-[]superscript^⋅superscriptsuperscript1⋅superscript1 + _t=1^TE_x ρ [ r% ^t(x,·)-G^t(π^t+1,x,·),π(x)-π^t+1(x) ]+ ∑t = 1T blackboard_Ex ∼ ρ [ ⟨ over start_ARG r end_ARGt ( x , ⋅ ) - Gitalic_t ( πitalic_t + 1 , x , ⋅ ) , π ( x ) - πitalic_t + 1 ( x ) ⟩ ] ≤ ≤ 2TCβ−∑t=1T+1x∼ρ[gx(πt)]+2Cβη+ηT2β2superscriptsubscript11subscriptsimilar-todelimited-[]subscriptsuperscript22 2TCβ- _t=1^T+1E_x ρ[g_x(π^t)]+% 2Cβη+ η T2β2 T C β - ∑t = 1T + 1 blackboard_Ex ∼ ρ [ gitalic_x ( πitalic_t ) ] + divide start_ARG 2 C β end_ARG start_ARG η end_ARG + divide start_ARG η T end_ARG start_ARG 2 β end_ARG +∑t=1Tx∼ρ[⟨r^t(x,⋅)−Gt(πt+1,x,⋅),π(x)−πt+1(x)⟩]superscriptsubscript1subscriptsimilar-todelimited-[]superscript^⋅superscriptsuperscript1⋅superscript1 + _t=1^TE_x ρ [ r% ^t(x,·)-G^t(π^t+1,x,·),π(x)-π^t+1(x) ]+ ∑t = 1T blackboard_Ex ∼ ρ [ ⟨ over start_ARG r end_ARGt ( x , ⋅ ) - Gitalic_t ( πitalic_t + 1 , x , ⋅ ) , π ( x ) - πitalic_t + 1 ( x ) ⟩ ] Here the last step uses the fact that Bx(⋅,π)=1βgx(⋅)subscript⋅subscript1subscript⋅B_x(·, _ ref)= 1βg_x(·)Bitalic_x ( ⋅ , πsansserif_ref ) = divide start_ARG 1 end_ARG start_ARG β end_ARG gitalic_x ( ⋅ ) and π∈ΠCsubscriptΠπ∈ _Cπ ∈ Πitalic_C. This concludes our proof. ∎ Proof of Lemma H.3. Let L^(π,π′,π′)^superscript′ L(π,π ,π )over start_ARG L end_ARG ( π , π′ , π′ ′ ) denote the empirical squared loss: L^(π,π′,π′):=∑(x¯,a¯,b¯)(4(fπ,π′β,η(x¯,a¯,b¯))−4(fπ′,π′β,η(x¯,a¯,b¯)))2.assign^superscript′subscript¯superscriptsubscript4subscriptsuperscriptsuperscript′¯subscript4subscriptsuperscriptsuperscript′¯2 L(π,π ,π ):= _(% x, a, b) ( clip_4(f^β,η% _π,π ( x, a, b))- % clip_4(f^β,η_π ,π ( x,% a, b)) )^2.over start_ARG L end_ARG ( π , π′ , π′ ′ ) := ∑( over¯ start_ARG x end_ARG , over¯ start_ARG a end_ARG , over¯ start_ARG b end_ARG ) ( sansserif_clip4 ( fitalic_β , ηitalic_π , π′ ′ ( over¯ start_ARG x end_ARG , over¯ start_ARG a end_ARG , over¯ start_ARG b end_ARG ) ) - sansserif_clip4 ( fitalic_β , ηitalic_π′ , π′ ′ ( over¯ start_ARG x end_ARG , over¯ start_ARG a end_ARG , over¯ start_ARG b end_ARG ) ) )2 . Fix any π′,π′∈Πsuperscript′Ππ ,π ∈ π′ , π′ ′ ∈ Π and consider the following LSR problems: π(π′,π′):=argminπ∈ΠL^(π,π′,π′).assignsuperscript′subscriptargminΠ^superscript′ π(π ,π ):= argmin_% π∈ L(π,π ,π ).π ( π′ , π′ ′ ) := argminitalic_π ∈ Π over start_ARG L end_ARG ( π , π′ , π′ ′ ) . Then from Lemma H.2, we know with probability at least 1−δ′1superscript′1-δ 1 - δ′ that L(π(π′,π′),π′,π′)≲log(|Π|/δ′)M.less-than-or-similar-tosuperscript′superscript′Πsuperscript′ L(π(π ,π ),π ,π % ) (| |/δ )M.L ( π ( π′ , π′ ′ ) , π′ , π′ ′ ) ≲ divide start_ARG log ( | Π | / δ′ ) end_ARG start_ARG M end_ARG . Therefore, by union bound, we know with probability at least 1−δ′1superscript′1-δ 1 - δ′ that for all π′,π′∈Πsuperscript′Ππ ,π ∈ π′ , π′ ′ ∈ Π: L(π(π′,π′),π′,π′)≲log(|Π|/δ′)M.less-than-or-similar-tosuperscript′superscript′Πsuperscript′ L(π(π ,π ),π ,π % ) (| |/δ )M.L ( π ( π′ , π′ ′ ) , π′ , π′ ′ ) ≲ divide start_ARG log ( | Π | / δ′ ) end_ARG start_ARG M end_ARG . The proof is concluded by noticing that πt+1=argminπ∈ΠL^(π,π¯t+1,πt)superscript1subscriptargminΠ^superscript¯1superscriptπ^t+1= argmin_π∈ L(π, π^% t+1,π^t)πitalic_t + 1 = argminitalic_π ∈ Π over start_ARG L end_ARG ( π , over¯ start_ARG π end_ARGt + 1 , πitalic_t ) under Assumption 7.2. ∎ Appendix I Proofs for :rlhf The section contains the proofs for the main guarantee χ2superscript2χ^2χ2-RLHF in Appendix B (Theorem B.1). We first prove two results, Theorem I.1 and Corollary I.1, which correspond to exact (i.e., including precise constants) versions of the two statements in Theorem B.1. We also analyze χ2superscript2χ^2χ2-RLHF with η=00η=0η = 0 in Corollary I.2. Throughout this section, we make use of the following η-smoothed version of the L1subscript1L_1L1 concentrability coefficient: ηπ:=π[π(a∣x)π(a∣x)+ηπ(a∣x)].assignsuperscriptsubscriptsubscriptconditionalsubscriptconditionalconditional _η^π = E_% π [ π(a x) _ ref(a x)+ηπ(a x)% ].Citalic_ηitalic_π := blackboard_Eπ [ divide start_ARG π ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) + η π ( a ∣ x ) end_ARG ] . It is easy to see that for any η≥00η≥ 0η ≥ 0 we have ηπ≤πsuperscriptsubscriptsuperscriptC_η^π ^πCitalic_ηitalic_π ≤ Citalic_π, as well as ηπ≤η−1superscriptsubscriptsuperscript1C_η^π≤η^-1Citalic_ηitalic_π ≤ η- 1. Theorem I.1 (General regret bound for Algorithm 3). Suppose Assumption B.1 and Assumption B.2 hold for parameters β>00β>0β > 0 and η∈[0,β8R]08subscriptη∈ [0, β8R_ max ]η ∈ [ 0 , divide start_ARG β end_ARG start_ARG 8 Rsansserif_max end_ARG ]. Then with probability at least 1−δ11- 1 - δ, the policy π^ πover start_ARG π end_ARG produced by χ2superscript2χ^2χ2-RLHF (Algorithm 3) satisfies J(π⋆)−J(π^)≤superscript⋆^absent J(π )-J( π) ( π⋆ ) - J ( over start_ARG π end_ARG ) ≤ 2ηπ⋆⋅εstat2+2β⋅ηπ⋆+4β−1⋅εstat22⋅superscriptsubscriptsuperscript⋆superscriptsubscriptstat2⋅2superscriptsubscriptsuperscript⋆⋅4superscript1superscriptsubscriptstat2 ~2 C_η^π · _% stat^2+2β·C_η^π +4β^-1% · _stat^22 square-root start_ARG Citalic_ηitalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT ⋅ εroman_stat2 end_ARG + 2 β ⋅ Citalic_ηitalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT + 4 β- 1 ⋅ εroman_stat2 +4β⋅(min∞π⋆,η−1+minmaxπ∈Π∞π,η−1)εx2+2Rεx.⋅4superscriptsubscriptsuperscript⋆superscript1subscriptΠsuperscriptsubscriptsuperscript1superscriptsubscriptx22subscriptsubscriptx +4β· ( \C_∞^π ,% η^-1 \+ \ _π∈ C_∞^π,η^% -1 \ ) _x^2+2R_ max _% x.+ 4 β ⋅ ( min C∞italic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT , η- 1 + min maxitalic_π ∈ Π C∞italic_π , η- 1 ) εroman_x2 + 2 Rsansserif_max εroman_x . where εstat2=32R2e4Rlog(3|ℛ|/δ)nsuperscriptsubscriptstat232superscriptsubscript2superscript4subscript3ℛ _stat^2= 32R_ max^2e^4R_ max% (3|R|/δ)nεroman_stat2 = divide start_ARG 32 Rsansserif_max2 e4 Rsansserif_max log ( 3 | R | / δ ) end_ARG start_ARG n end_ARG and εx=log(3|Π|/δ)2nsubscriptx3Π2subscript _x= (3| |/δ)2n_ xεroman_x = square-root start_ARG divide start_ARG log ( 3 | Π | / δ ) end_ARG start_ARG 2 nsansserif_x end_ARG end_ARG. The following results are immediate consequences of Theorem I.1. Corollary I.1 (Smoothed χ2superscript2χ^2χ2-regularization). Given π⋆superscript⋆π π⋆, let η=β8R8subscriptη= β8R_ maxη = divide start_ARG β end_ARG start_ARG 8 Rsansserif_max end_ARG and β=232R2e4Rlog(3|ℛ|/δ)nπ⋆232superscriptsubscript2superscript4subscript3ℛsuperscriptsuperscript⋆β=2 32R_ max^2e^4R_ max (3|% R|/δ)nC^π β = 2 square-root start_ARG divide start_ARG 32 Rsansserif_max2 e4 Rsansserif_max log ( 3 | R | / δ ) end_ARG start_ARG n Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT end_ARG end_ARG. Then under the preconditions of Theorem I.1, with probability at least 1−δ11- 1 - δ, the policy π^ πover start_ARG π end_ARG produced by χ2superscript2χ^2χ2-RLHF (Algorithm 3) satisfies J(π⋆)−J(π^)≤superscript⋆^absent J(π )-J( π) ( π⋆ ) - J ( over start_ARG π end_ARG ) ≤ 20Re2R2π⋆log(3|ℛ|/δ)n+R2log(3|Π|/δ)n+32Rlog(3|Π|/δ)n.20subscriptsuperscript2subscript2superscriptsuperscript⋆3ℛsubscript23Πsubscript32subscript3Πsubscript ~20R_ maxe^2R_ max 2C% ^π (3|R|/δ)n+R_ max 2% (3| |/δ)n_ x+ 32R_ max (3| |/% δ)n_ x.20 Rsansserif_max e2 Rsansserif_max square-root start_ARG divide start_ARG 2 Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT log ( 3 | R | / δ ) end_ARG start_ARG n end_ARG end_ARG + Rsansserif_max square-root start_ARG divide start_ARG 2 log ( 3 | Π | / δ ) end_ARG start_ARG nsansserif_x end_ARG end_ARG + divide start_ARG 32 Rsansserif_max log ( 3 | Π | / δ ) end_ARG start_ARG nsansserif_x end_ARG . Corollary I.2 (Non-smoothed χ2superscript2χ^2χ2-regularization). Given π⋆superscript⋆π π⋆, let η=00η=0η = 0 and β=232R2e4Rlog(3|ℛ|/δ)nπ⋆232superscriptsubscript2superscript4subscript3ℛsuperscriptsuperscript⋆β=2 32R_ max^2e^4R_ max (3|% R|/δ)nC^π β = 2 square-root start_ARG divide start_ARG 32 Rsansserif_max2 e4 Rsansserif_max log ( 3 | R | / δ ) end_ARG start_ARG n Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT end_ARG end_ARG. Then under the preconditions of Theorem I.1, with probability at least 1−δ11- 1 - δ, the policy π^ πover start_ARG π end_ARG produced by χ2superscript2χ^2χ2-RLHF (Algorithm 3) satisfies J(π⋆)−J(π^)≤superscript⋆^absent J(π )-J( π) ( π⋆ ) - J ( over start_ARG π end_ARG ) ≤ 20Re2R2π⋆log(3|ℛ|/δ)n+R2log(3|Π|/δ)n20subscriptsuperscript2subscript2superscriptsuperscript⋆3ℛsubscript23Πsubscript ~20R_ maxe^2R_ max 2C% ^π (3|R|/δ)n+R_ max 2% (3| |/δ)n_ x20 Rsansserif_max e2 Rsansserif_max square-root start_ARG divide start_ARG 2 Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT log ( 3 | R | / δ ) end_ARG start_ARG n end_ARG end_ARG + Rsansserif_max square-root start_ARG divide start_ARG 2 log ( 3 | Π | / δ ) end_ARG start_ARG nsansserif_x end_ARG end_ARG +32(∞π⋆+maxπ∈Π∞π)⋅log(3|Π|/δ)n⋅2log(3|ℛ|/δ)n.⋅32superscriptsubscriptsuperscript⋆subscriptΠsuperscriptsubscript3Πsubscript23ℛ +32 (C_∞^π + _π∈ % C_∞^π )· (3| |/δ)n_ x% · 2 (3|R|/δ)n.+ 32 ( C∞italic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT + maxitalic_π ∈ Π C∞italic_π ) ⋅ divide start_ARG log ( 3 | Π | / δ ) end_ARG start_ARG nsansserif_x end_ARG ⋅ square-root start_ARG divide start_ARG 2 log ( 3 | R | / δ ) end_ARG start_ARG n end_ARG end_ARG . Proof of Theorem I.1. The proof follows largely the same lines of analyses as the proof of Theorem F.1. One difference is that in Algorithm 3, we approximate the RLHF objective using contexts are sampled from subscriptD_ xDsansserif_x, so we require additional concentration arguments to show that the empirical objective approximates its population counterpart. Basic concentration results We begin by stating the two concentration inequalities, which, given the reward model r^ rover start_ARG r end_ARG produced in Eq. 33, bound the error between J^β,ηr^superscriptsubscript^ J_β,η rover start_ARG J end_ARGβ , ηover start_ARG r end_ARG and its the population version Jβ,ηr^superscriptsubscript^J_β,η rJitalic_β , ηover start_ARG r end_ARG. We will handle the return and regularization terms separately, which will later allow us to obtain tighter bounds. Define J^(π) J(π)over start_ARG J end_ARG ( π ) :=1n∑x∈π[r^(x,a)∣x],assignabsent1subscriptsubscriptsubscriptsubscriptconditional = 1n_ x _x _% x E_π [ r(x,a) x ],:= divide start_ARG 1 end_ARG start_ARG nsansserif_x end_ARG ∑x ∈ D start_POSTSUBSCRIPT sansserif_x end_POSTSUBSCRIPT blackboard_Eπ [ over start_ARG r end_ARG ( x , a ) ∣ x ] , and ^ηπ(π)superscriptsubscript C_η^π(π)over start_ARG C end_ARGηitalic_π ( π ) :=1n∑x∈π[∑aπ2(a∣x)π(a∣x)+ηπ(a∣x)∣x],assignabsent1subscriptsubscriptsubscriptsubscriptconditionalsubscriptsuperscript2conditionalsubscriptconditionalconditional = 1n_ x _x _% x E_π [ _a π^2(a % x) _ ref(a x)+ηπ(a x) x ],:= divide start_ARG 1 end_ARG start_ARG nsansserif_x end_ARG ∑x ∈ D start_POSTSUBSCRIPT sansserif_x end_POSTSUBSCRIPT blackboard_Eπ [ ∑a divide start_ARG π2 ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) + η π ( a ∣ x ) end_ARG ∣ x ] , so that J^β,ηr^(π)=J^(π)−β^ηπ(π)superscriptsubscript^^^superscriptsubscript J_β,η r(π)= J(π)-β % C_η^π(π)over start_ARG J end_ARGβ , ηover start_ARG r end_ARG ( π ) = over start_ARG J end_ARG ( π ) - β over start_ARG C end_ARGηitalic_π ( π ). Fix δ′∈(0,1]superscript′01δ ∈(0,1]δ′ ∈ ( 0 , 1 ], which we will specify at the end of this proof. Since maxxπ[r^(x,a)∣x]≤Rsubscriptsubscriptconditional^subscript _x E_π[ r(x,a) x]≤ R_% maxmaxitalic_x blackboard_Eπ [ over start_ARG r end_ARG ( x , a ) ∣ x ] ≤ Rsansserif_max, a straightforward application of Hoeffding’s inequality guarantees that with probability at most 1−δ′1superscript′1-δ 1 - δ′, for all π∈Ππ∈ π ∈ Π we have that |J^(π)−π[r^(x,a)]|≤Rlog(2|Π|/δ′)2n.^subscript^subscript2Πsuperscript′2subscript J(π)- E_π[% r(x,a)] ≤ R_ max (2| |/% δ )2n_ x.| over start_ARG J end_ARG ( π ) - blackboard_Eπ [ over start_ARG r end_ARG ( x , a ) ] | ≤ Rsansserif_max square-root start_ARG divide start_ARG log ( 2 | Π | / δ′ ) end_ARG start_ARG 2 nsansserif_x end_ARG end_ARG . (56) Next, we consider the regularization term. Since ∑aπ2(a∣x)π(a∣x)+ηπ(a∣x)≤min∞π,η−1subscriptsuperscript2conditionalsubscriptconditionalconditionalsuperscriptsubscriptsuperscript1 _a π^2(a x) _ ref(a x)+ηπ(a x% )≤ \C_∞^π,η^-1\∑a divide start_ARG π2 ( a ∣ x ) end_ARG start_ARG πsansserif_ref ( a ∣ x ) + η π ( a ∣ x ) end_ARG ≤ min C∞italic_π , η- 1 for any x∈x ∈ X, we use Bernstein’s inequality to derive the following result. Lemma I.1. With probability at least 1−δ11- 1 - δ, for any π∈Ππ∈ π ∈ Π, we have |^ηπ−ηπ|≤π2+2min∞π,η−1log(2|Π|/δ)n.^absentsuperscriptsubscriptsuperscriptsubscriptsuperscript22superscriptsubscriptsuperscript12Πsubscript C_η^π-C_η^% π ≤ C^π2+ 2 \C_% ∞^π,η^-1\ (2| |/δ)n_ x.| over start_ARG end_ARG Citalic_ηitalic_π - Citalic_ηitalic_π | ≤ divide start_ARG Citalic_π end_ARG start_ARG 2 end_ARG + divide start_ARG 2 min C∞italic_π , η- 1 log ( 2 | Π | / δ ) end_ARG start_ARG nsansserif_x end_ARG . Define εx:=log(2|Π|/δ′)2nassignsubscriptx2Πsuperscript′2subscript _x = (2| |/δ )% 2n_ xεroman_x := square-root start_ARG divide start_ARG log ( 2 | Π | / δ′ ) end_ARG start_ARG 2 nsansserif_x end_ARG end_ARG. The above lemma implies that for all π∈Ππ∈ π ∈ Π, we have ^ηπ≤3π2+4min∞π,η−1⋅εx2,and^ηπ≥π2−4min∞π,η−1⋅εx2.formulae-sequencesuperscriptsubscript^3superscript2⋅4superscriptsubscriptsuperscript1superscriptsubscriptx2andsuperscriptsubscript^superscript2⋅4superscriptsubscriptsuperscript1superscriptsubscriptx2 C_η^π≤~ 3C^π% 2+4 \C_∞^π,η^-1\· _x% ^2, C_η^π≥~ % C^π2-4 \C_∞^π,η^-1\·% _x^2.over start_ARG C end_ARGηitalic_π ≤ divide start_ARG 3 Citalic_π end_ARG start_ARG 2 end_ARG + 4 min C∞italic_π , η- 1 ⋅ εroman_x2 , and over start_ARG C end_ARGηitalic_π ≥ divide start_ARG Citalic_π end_ARG start_ARG 2 end_ARG - 4 min C∞italic_π , η- 1 ⋅ εroman_x2 . Together with Eq. 56, this implies that for all π∈Ππ∈ π ∈ Π, J^β,ηr^(π)=J^(π)−β^ηπ≤superscriptsubscript^^^superscriptsubscript^absent J_β,η r(π)= J(π)-% β C_η^π start_ARG J end_ARGβ , ηover start_ARG r end_ARG ( π ) = over start_ARG J end_ARG ( π ) - β over start_ARG C end_ARGηitalic_π ≤ π[r^(x,a)]−βηπ2+4βmin∞π,η−1εx2+Rεx,subscript^superscriptsubscript24superscriptsubscriptsuperscript1superscriptsubscriptx2subscriptsubscriptx ~ E_π[ r(x,a)]- % _η^π2+4β \C_∞^π,η^% -1\ _x^2+R_ max _x,blackboard_Eπ [ over start_ARG r end_ARG ( x , a ) ] - divide start_ARG β Citalic_ηitalic_π end_ARG start_ARG 2 end_ARG + 4 β min C∞italic_π , η- 1 εroman_x2 + Rsansserif_max εroman_x , (57) and J^β,ηr^(π)=J^(π)−β^ηπ≥superscriptsubscript^^^superscriptsubscript^absent J_β,η r(π)= J(π)-% β C_η^π start_ARG J end_ARGβ , ηover start_ARG r end_ARG ( π ) = over start_ARG J end_ARG ( π ) - β over start_ARG C end_ARGηitalic_π ≥ π[r^(x,a)]−3βηπ2−4βmin∞π,η−1εx2−Rεx.subscript^3superscriptsubscript24superscriptsubscriptsuperscript1superscriptsubscriptx2subscriptsubscriptx ~ E_π[ r(x,a)]- 3% _η^π2-4β \C_∞^π,η^% -1\ _x^2-R_ max _x.blackboard_Eπ [ over start_ARG r end_ARG ( x , a ) ] - divide start_ARG 3 β Citalic_ηitalic_π end_ARG start_ARG 2 end_ARG - 4 β min C∞italic_π , η- 1 εroman_x2 - Rsansserif_max εroman_x . (58) Estimation error bounds Next, we state the following off- and on-policy reward estimation error bounds for the reward model r^ rover start_ARG r end_ARG, analogous to Lemma F.1 and Lemma F.3 for χPO. Lemma I.2. Suppose Assumption B.1 holds. Then with probability at least 1−δ11- 1 - δ, the reward model r^ rover start_ARG r end_ARG learned in Eq. 33 satisfies εstat2=:π,π[((r^(x,a)−r^(x,b))−(r⋆(x,a)−r⋆(x,b)))2]≤32R2e4Rlog(|Π|/δ)n. _stat^2= % E_ _ ref, _ ref [ ( ( % r(x,a)- r(x,b) )- (r (x,a)-r (x,b) )% )^2 ]≤ 32R_ max^2e^4R_ max (|% |/δ)n.εroman_stat2 = : blackboard_Eπ start_POSTSUBSCRIPT sansserif_ref , πsansserif_ref end_POSTSUBSCRIPT [ ( ( over start_ARG r end_ARG ( x , a ) - over start_ARG r end_ARG ( x , b ) ) - ( r⋆ ( x , a ) - r⋆ ( x , b ) ) )2 ] ≤ divide start_ARG 32 Rsansserif_max2 e4 Rsansserif_max log ( | Π | / δ ) end_ARG start_ARG n end_ARG . Lemma I.3. Under the event in Lemma I.2, we have that for all π:→Δ():→Δπ:X→ (A)π : X → Δ ( A ), π,π[|(r^(x,a)−r^(x,b))−(r⋆(x,a)−r⋆(x,b))|]≤2ηπεstat2+2ηπRη,subscriptsubscript^^superscript⋆superscript⋆2superscriptsubscriptsuperscriptsubscriptstat22superscriptsubscriptsubscript E_π, _ ref [ % ( r(x,a)- r(x,b) )- (r (x,a)-% r (x,b) ) ]≤ 2 C_η^π% _stat^2+2C_η^πR_ maxη,blackboard_Eπ , π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT [ | ( over start_ARG r end_ARG ( x , a ) - over start_ARG r end_ARG ( x , b ) ) - ( r⋆ ( x , a ) - r⋆ ( x , b ) ) | ] ≤ 2 square-root start_ARG Citalic_ηitalic_π εroman_stat2 end_ARG + 2 Citalic_ηitalic_π Rsansserif_max η , where εstat2superscriptsubscriptstat2 _stat^2εroman_stat2 is defined in Lemma I.2. Regret decomposition Equipped with these concentration and estimation error bounds, we now bound the regret of Algorithm 3 using a pessimism-based analysis similar to the proof of Theorem F.1. Condition on the events in Eq. 56, Lemma I.1, and Lemma I.2, which hold together with probability at least 1−3δ′13superscript′1-3δ 1 - 3 δ′. We decompose the regret of π^ πover start_ARG π end_ARG using J^β,ηr^superscriptsubscript^ J_β,η rover start_ARG J end_ARGβ , ηover start_ARG r end_ARG, then leverage the inequalities in Eq. 57 and Eq. 58: J(π⋆)−J(π^)=superscript⋆^absent J(π )-J( π)=J ( π⋆ ) - J ( over start_ARG π end_ARG ) = J(π⋆)−J^β,ηr^(π⋆)+J^β,ηr^(π⋆)−J(π^)superscript⋆superscriptsubscript^^superscript⋆superscriptsubscript^^superscript⋆ ~J(π )- J_β,η r(π^% )+ J_β,η r(π )-J( % π)J ( π⋆ ) - over start_ARG J end_ARGβ , ηover start_ARG r end_ARG ( π⋆ ) + over start_ARG J end_ARGβ , ηover start_ARG r end_ARG ( π⋆ ) - J ( over start_ARG π end_ARG ) ≤ ≤ J(π⋆)−J^β,ηr^(π⋆)+J^β,ηr^(π^)−J(π^)superscript⋆superscriptsubscript^^superscript⋆superscriptsubscript^^^ ~J(π )- J_β,η r(π^% )+ J_β,η r( π)-J( % π)J ( π⋆ ) - over start_ARG J end_ARGβ , ηover start_ARG r end_ARG ( π⋆ ) + over start_ARG J end_ARGβ , ηover start_ARG r end_ARG ( over start_ARG π end_ARG ) - J ( over start_ARG π end_ARG ) ≤ ≤ J(π⋆)−π⋆[r^(x,a)]+3βηπ⋆2+4βmin∞π⋆,η−1εx2+Rεxsuperscript⋆subscriptsuperscript⋆^3superscriptsubscriptsuperscript⋆24superscriptsubscriptsuperscript⋆superscript1superscriptsubscriptx2subscriptsubscriptx ~J(π )- E_π [% r(x,a)]+ 3 _η^π 2+4β % \C_∞^π ,η^-1\ _x^2+R% _ max _xJ ( π⋆ ) - blackboard_Eπ⋆ [ over start_ARG r end_ARG ( x , a ) ] + divide start_ARG 3 β Citalic_ηitalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT end_ARG start_ARG 2 end_ARG + 4 β min C∞italic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT , η- 1 εroman_x2 + Rsansserif_max εroman_x +π^[r^(x,a)]−βηπ^2+4βmin∞π^,η−1εx2+Rεx−J(π^)subscript^^superscriptsubscript^24superscriptsubscript^superscript1superscriptsubscriptx2subscriptsubscriptx + E_ π[ r(x,a)]-% _η π2+4β \C_% ∞ π,η^-1\ _x^2+R_ max% _x-J( π)+ blackboard_Eover start_ARG π end_ARG [ over start_ARG r end_ARG ( x , a ) ] - divide start_ARG β Citalic_ηover start_ARG π end_ARG end_ARG start_ARG 2 end_ARG + 4 β min C∞over start_ARG π end_ARG , η- 1 εroman_x2 + Rsansserif_max εroman_x - J ( over start_ARG π end_ARG ) = == π⋆,π[Δ⋆(x,a,b)−Δ^(x,a,b)]+3βηπ⋆2+π^,π[Δ^(x,a,b)−Δ⋆(x,a,b)]−βηπ^2subscriptsuperscript⋆subscriptsuperscriptΔ⋆^Δ3superscriptsubscriptsuperscript⋆2subscript^subscript^ΔsuperscriptΔ⋆superscriptsubscript^2 ~ E_π , _ ref[% (x,a,b)- (x,a,b)]+ 3 _η^% π 2+ E_ π, _ ref% [ (x,a,b)- (x,a,b)]- _η^% π2blackboard_Eπ⋆ , π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT [ Δ⋆ ( x , a , b ) - over start_ARG Δ end_ARG ( x , a , b ) ] + divide start_ARG 3 β Citalic_ηitalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT end_ARG start_ARG 2 end_ARG + blackboard_Eover start_ARG π end_ARG , π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT [ over start_ARG Δ end_ARG ( x , a , b ) - Δ⋆ ( x , a , b ) ] - divide start_ARG β Citalic_ηover start_ARG π end_ARG end_ARG start_ARG 2 end_ARG +4βεx2(min∞π⋆,η−1+min∞π^,η−1)+2Rεx.4superscriptsubscriptx2superscriptsubscriptsuperscript⋆superscript1superscriptsubscript^superscript12subscriptsubscriptx +4β _x^2 ( \C_% ∞^π ,η^-1\+ \C_∞ π,% η^-1\ )+2R_ max _x.+ 4 β εroman_x2 ( min C∞italic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT , η- 1 + min C∞over start_ARG π end_ARG , η- 1 ) + 2 Rsansserif_max εroman_x . In the last line above, we have introduced the notation Δ⋆(x,a,b)=r⋆(x,a)−r⋆(x,b)superscriptΔ⋆superscript⋆superscript⋆ (x,a,b)=r (x,a)-r (x,b)Δ⋆ ( x , a , b ) = r⋆ ( x , a ) - r⋆ ( x , b ) and Δ^(x,a,b)=r^(x,a)−r^(x,b)^Δ^ (x,a,b)= r(x,a)- r(x,b)over start_ARG Δ end_ARG ( x , a , b ) = over start_ARG r end_ARG ( x , a ) - over start_ARG r end_ARG ( x , b ), and centered the returns. Next, applying Lemma I.3 to bound the reward estimation error above, we have J(π⋆)−J(π^)≤superscript⋆^absent J(π )-J( π) ( π⋆ ) - J ( over start_ARG π end_ARG ) ≤ 2ηπ⋆εstat2+2ηRηπ⋆+3βηπ⋆22superscriptsubscriptsuperscript⋆superscriptsubscriptstat22subscriptsuperscriptsubscriptsuperscript⋆3superscriptsubscriptsuperscript⋆2 ~2 C_η^π _% stat^2+2η R_ maxC_η^π + 3% _η^π 22 square-root start_ARG Citalic_ηitalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT εroman_stat2 end_ARG + 2 η Rsansserif_max Citalic_ηitalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT + divide start_ARG 3 β Citalic_ηitalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT end_ARG start_ARG 2 end_ARG +2ηπ^εstat2+2ηRηπ^−βηπ^22superscriptsubscript^superscriptsubscriptstat22subscriptsuperscriptsubscript^superscriptsubscript^2 +2 C_η π _% stat^2+2η R_ maxC_η π- % _η π2+ 2 square-root start_ARG Citalic_ηover start_ARG π end_ARG εroman_stat2 end_ARG + 2 η Rsansserif_max Citalic_ηover start_ARG π end_ARG - divide start_ARG β Citalic_ηover start_ARG π end_ARG end_ARG start_ARG 2 end_ARG +4βεx2(min∞π⋆,η−1+min∞π^,η−1)+2Rεx.4superscriptsubscriptx2superscriptsubscriptsuperscript⋆superscript1superscriptsubscript^superscript12subscriptsubscriptx +4β _x^2 ( \C_% ∞^π ,η^-1\+ \C_∞ π,% η^-1\ )+2R_ max _x.+ 4 β εroman_x2 ( min C∞italic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT , η- 1 + min C∞over start_ARG π end_ARG , η- 1 ) + 2 Rsansserif_max εroman_x . Applying the AM-GM inequality to 2ηπ^εstat22superscriptsubscript^superscriptsubscriptstat22 C_η π _stat^22 square-root start_ARG Citalic_ηover start_ARG π end_ARG εroman_stat2 end_ARG for η∈[0,β4R]04subscriptη∈ [0, β4R_ max ]η ∈ [ 0 , divide start_ARG β end_ARG start_ARG 4 Rsansserif_max end_ARG ], we have 2ηπ^εstat2=2superscriptsubscript^superscriptsubscriptstat2absent 2 C_η π _% stat^2=2 square-root start_ARG Citalic_ηover start_ARG π end_ARG εroman_stat2 end_ARG = (β−4ηR)ηπ^⋅4εstat2(β−4ηR)⋅4subscriptsuperscriptsubscript^4superscriptsubscriptstat24subscript ~ (β-4η R_ max)C_η^% π· 4 _stat^2(β-4η R_% max)square-root start_ARG ( β - 4 η Rsansserif_max ) Citalic_ηover start_ARG π end_ARG ⋅ divide start_ARG 4 εroman_stat2 end_ARG start_ARG ( β - 4 η Rsansserif_max ) end_ARG end_ARG ≤ ≤ βηπ^2−2ηRηπ^+2εstat2β−4ηRsuperscriptsubscript^22subscriptsuperscriptsubscript^2superscriptsubscriptstat24subscript ~ _η π2-2η R_% maxC_η π+ 2 _% stat^2β-4η R_ maxdivide start_ARG β Citalic_ηover start_ARG π end_ARG end_ARG start_ARG 2 end_ARG - 2 η Rsansserif_max Citalic_ηover start_ARG π end_ARG + divide start_ARG 2 εroman_stat2 end_ARG start_ARG β - 4 η Rsansserif_max end_ARG ≤ ≤ βηπ^2−2ηRηπ^+4εstat2β,superscriptsubscript^22subscriptsuperscriptsubscript^4superscriptsubscriptstat2 ~ _η π2-2η R_% maxC_η π+ 4 _% stat^2β,divide start_ARG β Citalic_ηover start_ARG π end_ARG end_ARG start_ARG 2 end_ARG - 2 η Rsansserif_max Citalic_ηover start_ARG π end_ARG + divide start_ARG 4 εroman_stat2 end_ARG start_ARG β end_ARG , where in the last line we use the fact that η≤β8R8subscriptη≤ β8R_ maxη ≤ divide start_ARG β end_ARG start_ARG 8 Rsansserif_max end_ARG so 4ηR≤β24subscript24η R_ max≤ β24 η Rsansserif_max ≤ divide start_ARG β end_ARG start_ARG 2 end_ARG. Then plugging this back into our regret decomposition cancels out the ηπ^superscriptsubscript^C_η πCitalic_ηover start_ARG π end_ARG terms to give J(π⋆)−J(π^)≤superscript⋆^absent J(π )-J( π) ( π⋆ ) - J ( over start_ARG π end_ARG ) ≤ 2ηπ⋆εstat2+2ηRηπ⋆+3βηπ⋆2+4εstat2β2superscriptsubscriptsuperscript⋆superscriptsubscriptstat22subscriptsuperscriptsubscriptsuperscript⋆3superscriptsubscriptsuperscript⋆24superscriptsubscriptstat2 ~2 C_η^π _% stat^2+2η R_ maxC_η^π + 3% _η^π 2+ 4 _stat^% 2β2 square-root start_ARG Citalic_ηitalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT εroman_stat2 end_ARG + 2 η Rsansserif_max Citalic_ηitalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT + divide start_ARG 3 β Citalic_ηitalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT end_ARG start_ARG 2 end_ARG + divide start_ARG 4 εroman_stat2 end_ARG start_ARG β end_ARG +4βεx2(min∞π⋆,η−1+min∞π^,η−1)+2Rεx4superscriptsubscriptx2superscriptsubscriptsuperscript⋆superscript1superscriptsubscript^superscript12subscriptsubscriptx +4β _x^2 ( \C_% ∞^π ,η^-1\+ \C_∞ π,% η^-1\ )+2R_ max _x+ 4 β εroman_x2 ( min C∞italic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT , η- 1 + min C∞over start_ARG π end_ARG , η- 1 ) + 2 Rsansserif_max εroman_x ≤ ≤ 2ηπ⋆εstat2+2βηπ⋆+4εstat2β2superscriptsubscriptsuperscript⋆superscriptsubscriptstat22superscriptsubscriptsuperscript⋆4superscriptsubscriptstat2 ~2 C_η^π _% stat^2+2 _η^π + 4 _ % stat^2β2 square-root start_ARG Citalic_ηitalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT εroman_stat2 end_ARG + 2 β Citalic_ηitalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT + divide start_ARG 4 εroman_stat2 end_ARG start_ARG β end_ARG +4βεx2(min∞π⋆,η−1+min∞π^,η−1)+2Rεx,4superscriptsubscriptx2superscriptsubscriptsuperscript⋆superscript1superscriptsubscript^superscript12subscriptsubscriptx +4β _x^2 ( \C_% ∞^π ,η^-1\+ \C_∞ π,% η^-1\ )+2R_ max _x,+ 4 β εroman_x2 ( min C∞italic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT , η- 1 + min C∞over start_ARG π end_ARG , η- 1 ) + 2 Rsansserif_max εroman_x , where in the last line we consolidate ηπ⋆superscriptsubscriptsuperscript⋆C_η^π Citalic_ηitalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT terms by again using 4ηR≤β24subscript24η R_ max≤ β24 η Rsansserif_max ≤ divide start_ARG β end_ARG start_ARG 2 end_ARG. Plugging in δ′=δ/3superscript′3δ =δ/3δ′ = δ / 3 and the values for εstat2superscriptsubscriptstat2 _stat^2εroman_stat2 and εxsubscriptx _xεroman_x results in the theorem statement. ∎ Proof of Corollary I.1. When η=β8R8subscriptη= β8R_ maxη = divide start_ARG β end_ARG start_ARG 8 Rsansserif_max end_ARG, Theorem I.1 states that J(π⋆)−J(π^)≤superscript⋆^absent J(π )-J( π) ( π⋆ ) - J ( over start_ARG π end_ARG ) ≤ 2ηπ⋆εstat2+2βηπ⋆+4εstat2β+4βεx2⋅(min∞π⋆,η−1+minmaxπ∈Π∞π,η−1)+2Rεx2superscriptsubscriptsuperscript⋆superscriptsubscriptstat22superscriptsubscriptsuperscript⋆4superscriptsubscriptstat2⋅4superscriptsubscriptx2superscriptsubscriptsuperscript⋆superscript1subscriptΠsuperscriptsubscriptsuperscript12subscriptsubscriptx ~2 C_η^π _% stat^2+2 _η^π + 4 _ % stat^2β+4β _x^2· ( \% C_∞^π ,η^-1 \+ \ _π∈% C_∞^π,η^-1 \ )+2R_ max% _x2 square-root start_ARG Citalic_ηitalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT εroman_stat2 end_ARG + 2 β Citalic_ηitalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT + divide start_ARG 4 εroman_stat2 end_ARG start_ARG β end_ARG + 4 β εroman_x2 ⋅ ( min C∞italic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT , η- 1 + min maxitalic_π ∈ Π C∞italic_π , η- 1 ) + 2 Rsansserif_max εroman_x ≤ ≤ 2ηπ⋆εstat2+2βηπ⋆+4εstat2β+8βεx2⋅η−1+2Rεx2superscriptsubscriptsuperscript⋆superscriptsubscriptstat22superscriptsubscriptsuperscript⋆4superscriptsubscriptstat2⋅8superscriptsubscriptx2superscript12subscriptsubscriptx ~2 C_η^π _% stat^2+2 _η^π + 4 _ % stat^2β+8β _x^2·η^-1+2R_% max _x2 square-root start_ARG Citalic_ηitalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT εroman_stat2 end_ARG + 2 β Citalic_ηitalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT + divide start_ARG 4 εroman_stat2 end_ARG start_ARG β end_ARG + 8 β εroman_x2 ⋅ η- 1 + 2 Rsansserif_max εroman_x = == 2ηπ⋆εstat2+2βηπ⋆+4εstat2β+64Rεx2+2Rεx.2superscriptsubscriptsuperscript⋆superscriptsubscriptstat22superscriptsubscriptsuperscript⋆4superscriptsubscriptstat264subscriptsuperscriptsubscriptx22subscriptsubscriptx ~2 C_η^π _% stat^2+2 _η^π + 4 _ % stat^2β+64R_ max _x^2+2R_ % max _x.2 square-root start_ARG Citalic_ηitalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT εroman_stat2 end_ARG + 2 β Citalic_ηitalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT + divide start_ARG 4 εroman_stat2 end_ARG start_ARG β end_ARG + 64 Rsansserif_max εroman_x2 + 2 Rsansserif_max εroman_x . Setting β=2εstat2π⋆2superscriptsubscriptstat2superscriptsuperscript⋆β=2 _stat^2C^π β = 2 square-root start_ARG divide start_ARG εroman_stat2 end_ARG start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT end_ARG end_ARG, we obtain J(π⋆)−J(π^)≤superscript⋆^absent J(π )-J( π) ( π⋆ ) - J ( over start_ARG π end_ARG ) ≤ 5ηπ⋆εstat2+64Rεx2+2Rεx.5superscriptsubscriptsuperscript⋆superscriptsubscriptstat264subscriptsuperscriptsubscriptx22subscriptsubscriptx ~5 C_η^π _% stat^2+64R_ max _x^2+2R_ max% _x.5 square-root start_ARG Citalic_ηitalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT εroman_stat2 end_ARG + 64 Rsansserif_max εroman_x2 + 2 Rsansserif_max εroman_x . ∎ Proof of Corollary I.2. When η=00η=0η = 0, Theorem I.1 states that J(π⋆)−J(π^)≤superscript⋆^absent J(π )-J( π) ( π⋆ ) - J ( over start_ARG π end_ARG ) ≤ 2π⋆εstat2+2βπ⋆+4εstat2β+4βεx2⋅(∞π⋆+maxπ∈Π∞π)+2Rεx2superscriptsuperscript⋆superscriptsubscriptstat22superscriptsuperscript⋆4superscriptsubscriptstat2⋅4superscriptsubscriptx2superscriptsubscriptsuperscript⋆subscriptΠsuperscriptsubscript2subscriptsubscriptx ~2 C^π _stat^2% +2 ^π + 4 _stat^2% β+4β _x^2· (C_∞^π^% + _π∈ C_∞^π )+2R_ max% _x2 square-root start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT εroman_stat2 end_ARG + 2 β Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT + divide start_ARG 4 εroman_stat2 end_ARG start_ARG β end_ARG + 4 β εroman_x2 ⋅ ( C∞italic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT + maxitalic_π ∈ Π C∞italic_π ) + 2 Rsansserif_max εroman_x Setting β=2εstat2π⋆2superscriptsubscriptstat2superscriptsuperscript⋆β=2 _stat^2C^π β = 2 square-root start_ARG divide start_ARG εroman_stat2 end_ARG start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT end_ARG end_ARG, we obtain J(π⋆)−J(π^)≤superscript⋆^absent J(π )-J( π) ( π⋆ ) - J ( over start_ARG π end_ARG ) ≤ 5π⋆εstat2+8εstatεx2⋅(∞π⋆+maxπ∈Π∞π)+2Rεx.5superscriptsuperscript⋆superscriptsubscriptstat2⋅8subscriptstatsuperscriptsubscriptx2superscriptsubscriptsuperscript⋆subscriptΠsuperscriptsubscript2subscriptsubscriptx ~5 C^π _stat^2% +8 _stat _x^2· ( % C_∞^π + _π∈ C_∞^π )+2% R_ max _x.5 square-root start_ARG Citalic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT εroman_stat2 end_ARG + 8 εroman_stat εroman_x2 ⋅ ( C∞italic_π start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT + maxitalic_π ∈ Π C∞italic_π ) + 2 Rsansserif_max εroman_x . ∎ Proof of Lemma I.2. We use similar reasoning and notation to the proof of Lemma F.1. Since r⋆∈ℛsuperscript⋆ℛr ⋆ ∈ R under Assumption B.1, Lemma E.1 guarantees that with probability at least 1−δ11- 1 - δ we have π,π[D2(Pr^(⋅∣x,a,b),Pr⋆(⋅∣x,a,b))]≤ E_ _ ref, _ ref% [D^2_ H (P_ r(· x,a,b),P_r % (· x,a,b) ) ] _Eπ start_POSTSUBSCRIPT sansserif_ref , πsansserif_ref end_POSTSUBSCRIPT [ D2sansserif_H ( Pover start_ARG r end_ARG ( ⋅ ∣ x , a , b ) , Pitalic_r⋆ ( ⋅ ∣ x , a , b ) ) ] ≤ 2log(|ℛ|/δ)n.2ℛ ~ 2 (|R|/δ)n.divide start_ARG 2 log ( | R | / δ ) end_ARG start_ARG n end_ARG . Since |r(x,a)−r(x,b)|≤Rsubscript|r(x,a)-r(x,b)|≤ R_ max| r ( x , a ) - r ( x , b ) | ≤ Rsansserif_max for all r∈ℛr ∈ R under Assumption B.1, we then apply Lemma F.5 with R=V=RsubscriptR=V=R_ maxR = V = Rsansserif_max. π,π[(r^(x,a)−r^(x,b)−(r⋆(x,a)−r⋆(x,b)))2]subscriptsubscriptsubscriptsuperscript^^superscript⋆superscript⋆2 E_ _ ref, _ ref% [ ( r(x,a)- r(x,b)- (r (x,a)-r^% (x,b) ) )^2 ]blackboard_Eπ start_POSTSUBSCRIPT sansserif_ref , πsansserif_ref end_POSTSUBSCRIPT [ ( over start_ARG r end_ARG ( x , a ) - over start_ARG r end_ARG ( x , b ) - ( r⋆ ( x , a ) - r⋆ ( x , b ) ) )2 ] ≤16e4R2⋅π,π[D2(Pr^(⋅∣x,a,b),Pr⋆(⋅∣x,a,b))] ≤ 16e^4R_ maxR_ max^2·% E_ _ ref, _ ref [D^2_% H (P_ r(· x,a,b),P_r (· % x,a,b) ) ]≤ 16 e4 Rsansserif_max Rsansserif_max2 ⋅ blackboard_Eπ start_POSTSUBSCRIPT sansserif_ref , πsansserif_ref end_POSTSUBSCRIPT [ D2sansserif_H ( Pover start_ARG r end_ARG ( ⋅ ∣ x , a , b ) , Pitalic_r⋆ ( ⋅ ∣ x , a , b ) ) ] ≤32e4RR2⋅log(|ℛ|/δ)n.absent⋅32superscript4subscriptsuperscriptsubscript2ℛ ≤ 32e^4R_ maxR_ max^2· % (|R|/δ)n.≤ 32 e4 Rsansserif_max Rsansserif_max2 ⋅ divide start_ARG log ( | R | / δ ) end_ARG start_ARG n end_ARG . ∎ Proof of Lemma I.3. Abbreviate Δ⋆(x,a,b)=r⋆(x,a)−r⋆(x,b)superscriptΔ⋆superscript⋆superscript⋆ (x,a,b)=r (x,a)-r (x,b)Δ⋆ ( x , a , b ) = r⋆ ( x , a ) - r⋆ ( x , b ), and Δ^(x,a,b)=r^(x,a)−r^(x,b)^Δ^ (x,a,b)= r(x,a)- r(x,b)over start_ARG Δ end_ARG ( x , a , b ) = over start_ARG r end_ARG ( x , a ) - over start_ARG r end_ARG ( x , b ). For a pair of policies π,π′π,π π , π′ and p≥11p≥ 1p ≥ 1, we define the norm ∥⋅∥p,π×π′:=(ρ,a∼π,b∼π′[|⋅|p])1/p\|·\|_p,π×π = ( E% _ρ,a π,b π [|·|^p] )^1/p∥ ⋅ ∥p , π × π′ := ( blackboard_Eρ , a ∼ π , b ∼ π′ [ | ⋅ |p ] )1 / p, so that π,π[|Δ⋆(x,a,b)−Δ^(x,a,b)|]=‖Δ⋆−Δ^‖1,π×πsubscriptsubscriptsuperscriptΔ⋆^ΔsubscriptnormsuperscriptΔ⋆^Δ1subscript E_π, _ ref [ ^% (x,a,b)- (x,a,b) ]= \| % - \|_1,π× _ refblackboard_Eπ , π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT [ | Δ⋆ ( x , a , b ) - over start_ARG Δ end_ARG ( x , a , b ) | ] = ∥ Δ⋆ - over start_ARG Δ end_ARG ∥1 , π × π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT. Then via Cauchy-Schwarz, ‖Δ⋆−Δ^‖1,π×π≤ρ[∑a,bπ2(a∣x)π2(b∣x)(π(a∣x)+ηπ(a∣x))π(b∣x)]⋅ρ[∑a,b(π(a∣x)+ηπ(a∣x))π(b∣x)(Δ⋆(x,a,b)−Δ^(x,a,b))2]=ηπ⋅(‖Δ⋆−Δ^‖2,π×π2+η‖Δ⋆−Δ^‖2,π×π2)≤ηπ⋅‖Δ⋆−Δ^‖2,π×π2+2ηRηπ⋅‖Δ⋆−Δ^‖1,π×π.subscriptnormsuperscriptΔ⋆^Δ1subscriptabsentsubscriptsubscriptsuperscript2conditionalsuperscriptsubscript2conditionalsubscriptconditionalconditionalsubscriptconditionalmissing-subexpression⋅absentsubscriptsubscriptsubscriptconditionalconditionalsubscriptconditionalsuperscriptsuperscriptΔ⋆^Δ2⋅superscriptsubscriptsuperscriptsubscriptnormsuperscriptΔ⋆^Δ2subscriptsubscript2superscriptsubscriptnormsuperscriptΔ⋆^Δ2subscript2⋅superscriptsubscriptsuperscriptsubscriptnormsuperscriptΔ⋆^Δ2subscriptsubscript2⋅2subscriptsuperscriptsubscriptsubscriptnormsuperscriptΔ⋆^Δ1subscript aligned \| - \|_1% ,π× _ ref≤&~ E_ρ% [ _a,b π^2(a x) _ ref^2(b x)( _% ref(a x)+ηπ(a x)) _ ref(b x) % ]\\ &· E_ρ [ _a,b( _ ref% (a x)+ηπ(a x)) _ ref(b x) ( % (x,a,b)- (x,a,b) )^2 ]\\ =&~ C_η^π· ( \| - % \|_2, _ ref× _ ref^2+η \|% - \|_2,π× _ ref^2% )\\ ≤&~ C_η^π· \| - % \|_2, _ ref× _ ref^2+ 2% η R_ maxC_η^π· \| -% \|_1,π× _ ref. alignedstart_ROW start_CELL ∥ Δ⋆ - over start_ARG Δ end_ARG ∥1 , π × π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT ≤ end_CELL start_CELL square-root start_ARG blackboard_Eρ [ ∑a , b divide start_ARG π2 ( a ∣ x ) πsansserif_ref2 ( b ∣ x ) end_ARG start_ARG ( πsansserif_ref ( a ∣ x ) + η π ( a ∣ x ) ) πsansserif_ref ( b ∣ x ) end_ARG ] end_ARG end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL ⋅ square-root start_ARG blackboard_Eρ [ ∑a , b ( πsansserif_ref ( a ∣ x ) + η π ( a ∣ x ) ) πsansserif_ref ( b ∣ x ) ( Δ⋆ ( x , a , b ) - over start_ARG Δ end_ARG ( x , a , b ) )2 ] end_ARG end_CELL end_ROW start_ROW start_CELL = end_CELL start_CELL square-root start_ARG Citalic_ηitalic_π ⋅ ( ∥ Δ⋆ - over start_ARG Δ end_ARG ∥2 , π start_POSTSUBSCRIPT sansserif_ref × πsansserif_ref end_POSTSUBSCRIPT2 + η ∥ Δ⋆ - over start_ARG Δ end_ARG ∥2 , π × π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT2 ) end_ARG end_CELL end_ROW start_ROW start_CELL ≤ end_CELL start_CELL square-root start_ARG Citalic_ηitalic_π ⋅ ∥ Δ⋆ - over start_ARG Δ end_ARG ∥2 , π start_POSTSUBSCRIPT sansserif_ref × πsansserif_ref end_POSTSUBSCRIPT2 end_ARG + square-root start_ARG 2 η Rsansserif_max Citalic_ηitalic_π ⋅ ∥ Δ⋆ - over start_ARG Δ end_ARG ∥1 , π × π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT end_ARG . end_CELL end_ROW Applying the AM-GM inequality to the second term, we obtain ‖Δ⋆−Δ^‖1,π×π≤subscriptnormsuperscriptΔ⋆^Δ1subscriptabsent \| - \|_1,π× _% ref≤∥ Δ⋆ - over start_ARG Δ end_ARG ∥1 , π × π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT ≤ ηπ⋅‖Δ⋆−Δ^‖2,π×π2+ηRηπ+12‖Δ⋆−Δ^‖1,π×π.⋅superscriptsubscriptsuperscriptsubscriptnormsuperscriptΔ⋆^Δ2subscriptsubscript2subscriptsuperscriptsubscript12subscriptnormsuperscriptΔ⋆^Δ1subscript ~ C_η^π· \| -% \|_2, _ ref× _ ref^2+% η R_ maxC_η^π+ 12 \| % - \|_1,π× _ ref.square-root start_ARG Citalic_ηitalic_π ⋅ ∥ Δ⋆ - over start_ARG Δ end_ARG ∥2 , π start_POSTSUBSCRIPT sansserif_ref × πsansserif_ref end_POSTSUBSCRIPT2 end_ARG + η Rsansserif_max Citalic_ηitalic_π + divide start_ARG 1 end_ARG start_ARG 2 end_ARG ∥ Δ⋆ - over start_ARG Δ end_ARG ∥1 , π × π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT . Rearranging, ‖Δ⋆−Δ^‖1,π×π≤subscriptnormsuperscriptΔ⋆^Δ1subscriptabsent \| - \|_1,π× _% ref≤∥ Δ⋆ - over start_ARG Δ end_ARG ∥1 , π × π start_POSTSUBSCRIPT sansserif_ref end_POSTSUBSCRIPT ≤ 2ηπ⋅‖Δ⋆−Δ^‖2,π×π2+2ηRηπ.2⋅superscriptsubscriptsuperscriptsubscriptnormsuperscriptΔ⋆^Δ2subscriptsubscript22subscriptsuperscriptsubscript ~2 C_η^π· \| -% \|_2, _ ref× _ ref^2+2% η R_ maxC_η^π.2 square-root start_ARG Citalic_ηitalic_π ⋅ ∥ Δ⋆ - over start_ARG Δ end_ARG ∥2 , π start_POSTSUBSCRIPT sansserif_ref × πsansserif_ref end_POSTSUBSCRIPT2 end_ARG + 2 η Rsansserif_max Citalic_ηitalic_π . ∎