Paper deep dive
Safe RLHF Beyond Expectation: Stochastic Dominance for Universal Spectral Risk Control
Yaswanth Chittepu, Ativ Joshi, Rajarshi Bhattacharjee, Scott Niekum
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/22/2026, 6:19:28 AM
Summary
The paper introduces Risk-sensitive Alignment via Dominance (RAD), a framework for Safe Reinforcement Learning from Human Feedback (RLHF) that replaces scalar expected-cost constraints with First-Order Stochastic Dominance (FSD) constraints. By utilizing an Optimal Transport (OT) framework with entropic regularization and Sinkhorn iterations, RAD enables differentiable, end-to-end optimization of cost distributions. The authors further demonstrate that quantile-weighted FSD constraints provide universal control over Spectral Risk Measures (SRMs), allowing for tunable risk sensitivity and improved robustness against tail risks and out-of-distribution failures compared to traditional expectation-based methods.
Entities (6)
Relation Signals (3)
RAD → replaces → expected cost constraints
confidence 98% · RAD, a novel alignment framework that replaces scalar expected cost constraints with First-Order Stochastic Dominance (FSD) constraints.
RAD → utilizes → Optimal Transport
confidence 97% · We operationalize this constraint by comparing the target policy's cost distribution to that of a reference policy within an Optimal Transport (OT) framework.
quantile-weighted FSD → controls → Spectral Risk Measures
confidence 96% · We introduce quantile-weighted FSD constraints and show that weighted FSD universally controls a broad class of Spectral Risk Measures (SRMs).
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Safe Reinforcement Learning from Human Feedback (RLHF) typically enforces safety through expected cost constraints, but the expectation captures only a single statistic of the cost distribution and fails to account for distributional uncertainty, particularly under heavy tails or rare catastrophic events. This limitation is problematic when robustness and risk sensitivity are critical. Stochastic dominance offers a principled alternative by comparing entire cost distributions rather than just their averages, enabling direct control over tail risks and potential out-of-distribution failures that expectation-based constraints may overlook. In this work, we propose Risk-sensitive Alignment via Dominance (RAD), a novel alignment framework that replaces scalar expected cost constraints with First-Order Stochastic Dominance (FSD) constraints. We operationalize this constraint by comparing the target policy's cost distribution to that of a reference policy within an Optimal Transport (OT) framework, using entropic regularization and Sinkhorn iterations to obtain a differentiable and computationally efficient objective for stable end-to-end optimization. Furthermore, we introduce quantile-weighted FSD constraints and show that weighted FSD universally controls a broad class of Spectral Risk Measures (SRMs), so that improvements under weighted dominance imply guaranteed improvements in the corresponding spectral risk. This provides a principled mechanism for tuning a model's risk profile via the quantile weighting function. Empirical results demonstrate that RAD improves harmlessness over baselines while remaining competitive in helpfulness, and exhibits greater robustness on out-of-distribution harmlessness evaluations.
Tags
Links
- Source: https://arxiv.org/abs/2603.10938v1
- Canonical: https://arxiv.org/abs/2603.10938v1
Trouble viewing inline? Open PDF directly →
Full Text
72,089 characters extracted from source content.
Expand or collapse full text
Cover Page Safe RLHF Beyond Expectation: Stochastic Dominance for Universal Spectral Risk Control Yaswanth Chittepu, Ativ Joshi, Rajarshi Bhattacharjee, Scott Niekum Keywords: Reinforcement Learning from Human Feedback (RLHF), Safety, Alignment, Stochastic Dominance, Optimal Transport, Spectral Risk Measures Summary Safe RLHF typically enforces safety through expected cost constraints, but the expectation captures only a single statistic of the cost distribution and fails to account for distributional un- certainty, particularly under heavy tails or rare catastrophic events. We proposes Risk-sensitive Alignment via Dominance (RAD), a safe alignment framework that replaces expected-cost constraints with first-order stochastic dominance (FSD) constraints on the full cost distribution relative to a reference policy, implemented through an entropically regularized optimal trans- port objective. Furthermore, we introduce quantile-weighted dominance constraints and show that they provide universal control over a broad class of Spectral Risk Measures, enabling tun- able risk sensitivity. Contribution(s) 1. We introduce Risk-sensitive Alignment via Dominance (RAD), a Safe RLHF objective that constrains the first-order stochastic dominance (FSD) of the learned policy’s cost distribu- tion relative to a reference policy, rather than constraining only expected cost. Context: Prior Safe RLHF (Dai et al., 2023b) and HC-RLHF (Chittepu et al., 2025) for- mulations impose scalar expected-cost or high-confidence expected-cost constraints, while we position RAD as bringing stochastic-dominance control into the safe-RLHF framework for policy-induced cost distributions. 2. We give a practical optimization procedure for this dominance objective by using an asym- metric quantile-gap surrogate, a nonparametric empirical quantile-particle representation of policy-induced cost distributions, and an entropically regularized optimal-transport formu- lation that yields a differentiable REINFORCE-style policy-gradient estimator. Context: While the OT view of the FSD surrogate and Sinkhorn-based regularization have been explored (Melnyk et al., 2024), we adapt them to the sample-based Safe RLHF setting together with the resulting gradient estimator for optimizing the dominance-constrained ob- jective. 3. We introduce quantile-weighted FSD constraints and show that they provide universal con- trol over a broad class of Spectral Risk Measures (SRMs), enabling tunable risk sensitivity. Context:SRMs (Acerbi, 2002) form a broad class of coherent (Artzner et al., 1999), law-invariant risk functionals that aggregate quantile costs using a weighted spectrum over confidence levels. 4. Empirically, we show that RAD improves the harmlessness of model responses over base- lines (Safe-RLHF, SFT) and generalizes better to out-of-distribution harmlessness evalua- tions, while maintaining competitive helpfulness relative to Safe-RLHF (Dai et al., 2023b). Context: We use the BeaverTails dataset (Ji et al., 2023) following Dai et al. (2023b), and evaluate the helpfulness and harmlessness of RAD, Safe-RLHF, and SFT models, on its test split. We additionally evaluate out-of-distribution harmlessness on HarmBench (Mazeika et al., 2024). arXiv:2603.10938v1 [cs.LG] 11 Mar 2026 Safe RLHF Beyond Expectation: Stochastic Domi- nance for Universal Spectral Risk Control Yaswanth Chittepu 1,† , Ativ Joshi 1 , Rajarshi Bhattacharjee 1 , Scott Niekum 1 ychittepu, atjoshi, rbhattacharj, sniekum@umass.edu 1 University of Massachusetts Amherst Abstract Safe Reinforcement Learning from Human Feedback (RLHF) typically enforces safety through expected cost constraints, but the expectation captures only a single statistic of the cost distribution and fails to account for distributional uncertainty, particularly under heavy tails or rare catastrophic events. This limitation is problematic when ro- bustness and risk sensitivity are critical. Stochastic dominance offers a principled al- ternative by comparing entire cost distributions rather than just their averages, enabling direct control over tail risks and potential out-of-distribution failures that expectation- based constraints may overlook. In this work, we propose Risk-sensitive Alignment via Dominance (RAD), a novel alignment framework that replaces scalar expected cost constraints with First-Order Stochastic Dominance (FSD) constraints. We operational- ize this constraint by comparing the target policy’s cost distribution to that of a reference policy within an Optimal Transport (OT) framework, using entropic regularization and Sinkhorn iterations to obtain a differentiable and computationally efficient objective for stable end-to-end optimization. Furthermore, we introduce quantile-weighted FSD con- straints and show that weighted FSD universally controls a broad class of Spectral Risk Measures (SRMs), so that improvements under weighted dominance imply guaranteed improvements in the corresponding spectral risk. This provides a principled mechanism for tuning a model’s risk profile via the quantile weighting function. Empirical results demonstrate that RAD improves harmlessness over baselines while remaining compet- itive in helpfulness, and exhibits greater robustness on out-of-distribution harmlessness evaluations. 1 Introduction Large Language Models (LLMs) are now used across a wide range of applications, so it is important that their outputs are helpful, reliable, and safe. This need is especially acute in high-stakes domains such as legal reasoning (Katz et al. (2024)), medical consultation (Yang et al. (2022); Moor et al. (2023)), and educational support (Kasneci et al. (2023); Kung et al. (2023)), where harmful gen- erations—including toxicity and misinformation—can have serious consequences (Gehman et al. (2020); Weidinger et al. (2021); Ganguli et al. (2022)). Broadly, we want LLMs to be helpful and aligned with human preferences, while being harmless, i.e. prevent toxic outputs. But there may be scenarios when users may request assistance with potentially harmful activities (Bai et al. (2022b); Glaese et al. (2022)). Thus, harmlessness and helpfulness can conflict. The standard technique for aligning LLMs with human preferences, Reinforcement Learn- ing from Human Feedback (RLHF), typically optimizes a single reward model for both objectives (Ouyang et al. (2022b); Bai et al. (2022c)), or heuristically mixes separate reward models (Glaese et al. (2022); Touvron et al. (2023); Mu et al. (2024)). This creates trade-offs: stronger harmlessness 1 can lead to refusals, while stronger helpfulness can increase unsafe outputs (Bai et al. (2022c)). Re- cent work addresses this by decoupling preference data and enforcing harmlessness as a constraint, an approach known as Safe RLHF (Dai et al. (2023b)). However, Safe RLHF typically typically con- strains the expected cost of a policy, providing no guarantees on worst-case or tail outcomes. This is inadequate in high-stakes deployments where tail risk is critical — such as code generation Pearce et al. (2021) and open-ended dialogue where rare but severe harms including toxic generations and private data leakage can impact users (Perez et al., 2022; Gehman et al., 2020). Thus, we argue that a stronger notion of constraining the cost of the policy is needed: the cost distribution of the learned policy should be stochastically smaller than that of the reference policy, not merely cheaper on average. This means that the learned policy should assign less probability to high-cost outcomes across the distribution, not just reduce the mean cost. This motivates Risk- sensitive Alignment via Dominance (RAD), which enforces a First-Order Stochastic Dominance (FSD) constraint on the cost distribution of the learned policy relative to a reference policy. A key insight of RAD is that by reweighting quantiles of the FSD objective, we recover Spectral Risk Measures (SRMs) — a family of risk measures defined as a weighted integral of the cost quantiles, R w(q)Q X (q)dq, where Q X (q) is the quantile function of the cost distribution and the weighting function w(q) allows practitioners to express diverse risk preferences over the cost distribution. For instance, concentrating weight on upper quantiles recovers tail-sensitive measures such as CVaR, while uniform weighting recovers the mean — both expressible within our unified framework. This is practically valuable: a medical deployment may demand near-zero tolerance for harmful outputs, while a general assistant may tolerate a more permissive tradeoff, and both are accommodated by choosing w appropriately. Optimizing FSD constraints directly is challenging, so we relax them to an asymmetric FSD- violation surrogate that aggregates positive quantile gaps (see Equation (3)). We interpret stochastic dominance through Optimal Transport (OT) and derive an efficient REINFORCE-style policy gra- dient using Sinkhorn iterations, yielding an end-to-end differentiable approach. Empirically, RAD yields models that are more robust to safety violations while achieving competitive helpfulness across multiple spectral risk measures relative to Safe RLHF. Our main contributions are: (i) we formulate safe alignment as a dominance-constrained objective over the full cost distribution rather than only its expectation, (i) we derive a RAD policy-gradient estimator using entropic OT and Sinkhorn iterations, making optimization of the objective end- to-end differentiable, (i) we connect quantile reweighting in our framework to SRMs, enabling controllable risk-sensitive alignment profiles, and (iv) through extensive experiments, we show that RAD improves robustness to safety violations while maintaining competitive helpfulness relative to expected cost based baselines. The rest of the paper is organized as follows: Section 2 reviews Safe RLHF, stochastic dominance, and SRMs; Section 3 presents RAD and its optimization procedure; Section 4 reports empirical evaluations safety–helpfulness trade-offs; and Sections 5 and 6 is related work and conclusion respectively. 2 Background and Problem Setting 2.1 Reinforcement Learning from Human Feedback (RLHF) The most popular method of aligning LLMs with human preferences is Reinforcement Learning from Human Feedback (RLHF). The standard RLHF pipeline consists of three phases as described in Christiano et al. (2017); Ouyang et al. (2022b). The first phase is a supervised fine-tuning (SFT) phase where the outputs of a pre-trained model are aligned with human responses. This is followed by a reward modeling phase, where a reward model is trained using human preferences and finally, a reinforcement learning phase where the model is optimized using the learned reward function. Formally, let x ∼ D x denote prompts and y ∼ π θ (· | x) denote model responses. SFT produces a reference policy π sft , and we set π ref := π sft unless stated otherwise. A reward model r φ (x,y) 2 is learned from preference comparisons using the Bradley-Terry model (Bradley & Terry, 1952). Details are standard and deferred to Section B. RLHF then optimizes a KL-regularized objective to prevent reward overoptimization and preserve language quality (Ouyang et al., 2022b; Stiennon et al., 2022; Gao et al., 2022; Rafailov et al., 2024): max θ E x∼D x ,y∼π θ (·|x) r φ (x,y) − β D KL π θ (·| x)∥π ref (·| x) ,(1) equivalently maximizing the KL-regularized reward ̃r(x,y) = r φ (x,y)−β log π θ (y|x) π ref (y|x) . In practice, this objective is optimized with PPO/GRPO/REINFORCE-style methods (Williams, 1992; Ahma- dian et al., 2024; Shao et al., 2024; Schulman et al., 2017). 2.2 Safe RLHF Standard RLHF optimizes a single reward function learned from human preferences, which may be inadequate when attempting to balance competing goals such as helpfulness and harmlessness. In contrast, Safe RLHF (Dai et al., 2023b) separates helpfulness and harmlessness by learning (i) a reward model r φ (x,y), (parameterized by φ and for input prompt x and model output y) from helpfulness preferences and (i) a cost model c ψ (x,y), (parameterized by ψ) from harmfulness pref- erences, and then solving the expected-cost constrained RL problem: max θ E r φ (x,y) − βD KL (π θ ∥π ref ) s.t. E c ψ (x,y) ≤ τ.(2) High-confidence variants like HC-RLHF (Chittepu et al., 2025) replace the expectation constraint with a probabilistic guarantee using the Seldonian framework (Thomas et al., 2019). 2.3 First-Order Stochastic Dominance The expected cost constraint in Safe RLFH often proves inadequate in settings where controlling the tail risk is crucial. A more robust method for controlling tail risk is ensuring that the learned policy’s cost is stochastically smaller than the cost of the reference policy i.e. less probability must be assigned by the learned policy to high cost outcomes compared to the reference policy. We now describe how we do this formally. Let F X and Q X denote the Cumulative Distribution Function (CDF) and the Quantile Function (Inverse CDF) of a random variable X . For real-valued random variables X,Y , X is said to have first-order stochastic dominance (FSD) on Y (denoted by X ⪰ FSD Y ) iff F X (r)≤ F Y (r) for all r (Dai et al., 2023a). Equivalently, we also have Q X (q)≥ Q Y (q) for all q ∈ [0, 1]. Practically, a drawback of FSD is that it only imposes a partial ordering over distributions, so two random variables may be incomparable in terms of FSD. Hence, we relax the definition to an objec- tive that measures the extent to which X falls below Y across quantiles: L FSD (X,Y ) := Z 1 0 Q Y (q)− Q X (q) + dq,(3) where (x) + = max(x, 0) denotes the ReLU function. Note that for two random variables X and Y with different distributions, an FSD objective of 0 implies that X dominates Y, i.e. X ⪰ FSD Y . More generally,L FSD (X,Y ) aggregates the positive quantile gaps (Q Y (q)−Q X (q)) + and therefore quantifies the extent to which X fails to dominate Y or Y dominates X ; it does not by itself certify Y ⪰ FSD X . In our case, we set X to be the cost distribution C π θ under our learned policy Π θ and Y to be the cost distribution C π ref under the reference policy Π ref . Thus,L FSD (C π θ ,C π ref ) quantifies by how much the cost distribution of the reference policy dominates the cost distribution of the learned policy 1 . 1 Note thatL FSD (X,Y ) is not a distance because it is not symmetric and can be zero even when X and Y have different distributions. AlsoL FSD (X,Y ) +L FSD (Y,X) =W 1 (X,Y ) whereW 1 (X,Y ) is the 1-Wassertsein distance 3 2.4 Optimal Transport Optimal transport (Peyré & Cuturi, 2020) studies the problem of optimally transforming a distribu- tion μ into another distribution ν under a cost function c(x,y) :X ×Y → R + , which represents the cost of moving a unit mass from x to y. Let μ = P N i=1 a i δ x i and ν = P M j=1 b j δ y j be two empirical (atomic) distributions on R d , where x i ,y j ∈ R d are support points (“particles”), a∈ ∆ N and b∈ ∆ M are nonnegative weights summing to 1 (∆ N denotes the N -dimensional probability simplex), and δ z is a Dirac mass at z. A transport plan (a.k.a. coupling) from μ to ν is a nonnegative matrix P ∈ R N×M + whose row and column sums match the marginals. Intuitively, P ij represents the amount of probability mass moved from x i to y j . The set of admissible couplings can be represented as: Π(μ,ν) := n P ∈ R N×M + : P1 M = a, P ⊤ 1 N = b o .(4) Given a ground cost function c : R d × R d → R + , define the cost matrix C ij = c(x i ,y j ). The Kantorovich optimal transport problem (Peyré & Cuturi, 2020) between μ and ν is OT c (μ,ν) :=min P∈Π(μ,ν) ⟨P,C⟩ =min P∈Π(μ,ν) N X i=1 M X j=1 P ij c(x i ,y j ).(5) The above objective function is a linear programming problem. Solving this directly is computa- tionally expensive. Hence, to approximate the solution, an entropically regularized OT objective is usually preferred (Cuturi, 2013), which replaces the linear program with a strictly convex problem: OT χ c (μ,ν) :=min P∈Π(μ,ν) ⟨P,C⟩− χH(P ),(6) where H(P ) =− P ij P ij logP ij and χ is the regularization parameter. This optimization problem is smooth and has a unique minimizer, which is computable via Sinkhorn iterations (Cuturi, 2013; Peyré & Cuturi, 2020). We explain the full details of regularized OT and the Sinkhorn Algorithm in Section C. Relation to FSD. We now describe how our FSD objective relates to the optimal transport objec- tive. Using the results from Santambrogio (2015) (Theorem 2.9 and Proposition 2.17) or Melnyk et al. (2024) (Theorem 1), for any two distributions X and Y , the FSD objective can be framed as an optimal transport problem. Specifically, L FSD (X,Y ) = OT c (X,Y )(7) for the asymmetric convex cost function c(x,y) = (y − x) + . We restate the relevant theorem in Section D for completeness. We note that while Melnyk et al. (2024) also aims to do alignment via optimal transport, they apply stochastic dominance directly to reward distributions which are derived from a specific parametric form. On the other hand, we only have access to the cost distribution of a policy (independent from the reward distribution) which we access only through sampling. Hence, our framework is more general as we do not assume any specific parametric form. 2.5 Spectral Risk Measures Spectral risk measures (SRMs) (Acerbi, 2002) form a broad class of coherent (Artzner et al., 1999), law-invariant risk functionals that aggregate quantile costs using a weighted spectrum over confi- dence levels. They provide a flexible way to emphasize parts of the cost distribution (especially the tail) according to application-specific risk tolerance, rather than relying only on the mean. SRMs are widely used in fields like financial risk management and actuarial science to quantify and control 4 exposure to adverse outcomes (Dowd et al. (2008); Adam et al. (2008)). Let X be a real-valued random variable with quantile function Q X : [0, 1]→ R. A spectral risk measure is defined as ρ φ (X) = Z 1 0 Q X (q)φ(q)dq,(8) where φ : [0, 1] → R + is a non-negative, non-decreasing weighting function satisfying R 1 0 φ(q)dq = 1. The function φ is referred to as the risk spectrum and encodes risk sensitivity across quantiles. Larger weights at higher quantiles emphasize higher tail costs, while uniform weighting recovers the expectation. 3 Risk-sensitive Alignment via Dominance We now present our safe alignment formulation – Risk-sensitive Alignment via Dominance (RAD). Standard Safe RLHF constrains the expected cost, which controls only a single moment of the cost distribution. RAD instead enforces a distributional safety constraint using first-order stochastic dominance (FSD). We first introduce FSD as a principled way to ensure the learned policy’s cost dis- tribution is stochastically smaller than a reference, implemented via the asymmetric FSD surrogate L FSD (Eq. (3)). We then show that reweighting quantiles of this surrogate recovers Spectral Risk Measures (SRMs), yielding a unified framework in which the choice of weighting function w(q) controls the risk profile of the learned policy. Recall that our reward model is denoted by r φ (x,y) and our cost model is denoted by c ψ (x,y) for input x and ouptut y and for learned parameters φ and ψ. For a policy π, let C π := c ψ (x,y) denote the random variable of costs induced by sampling x∼D x and y ∼ π(·| x). Thus, we solve: max θ E x∼D x ,y∼π θ (·|x) h r φ (x,y) i − β D KL π θ (·| x)∥π ref (·| x) ,(9) s.t. L FSD C π θ , C π ref ≥ κ.(10) Constraint (10) encourages uniform improvement across quantiles: since L FSD (C π θ , C π ref ) = R 1 0 (Q C π ref (q) − Q C π θ (q)) + dq, larger values of L FSD C π θ ,C π ref correspond to larger positive gaps Q C π ref (q)− Q C π θ (q) over q ∈ [0, 1]. Equivalently, it encourages C π ref ⪰ FSD C π θ , i.e., the learned policy has stochastically smaller costs. To optimize our objective, we first absorb the KL term into the reward by defining the KL-regularized reward ̃r(x,y) = r φ (x,y)− β log π θ (y|x) π ref (y|x) as done in Dai et al. (2023b), and then use Dual Ascent (Gallier & Quaintance, 2019) to optimize the Lagrangian relaxation of (9)–(10): max θ min λ≥0 E x∼D x ,y∼π θ (·|x) h ̃r(x,y) i + λ L FSD C π θ ,C π ref − κ | z L(θ,λ) .(11) Non-parametric (quantile-particle) representation of policy-induced cost distributions. We now describe how we represent the cost distributions for our policies. For a policy π, rather than assuming a parametric family for the cost distribution C π , we use a non-parametric empirical ap- proximation via particles in the quantile space. Fix quantile levels α 1 ,...,α N ∈ (0, 1) and define the corresponding cost-quantile particles: q i (π) := Q C π (α i ), i = 1,...,N,(12) which we assume are ordered so that q 1 (π) ≤ · ≤ q N (π). We approximate the cost distribution by the empirical measure: ˆμ π := 1 N N X i=1 δ q i (π) .(13) 5 In practice, ˆμ π is obtained by sampling generations from π, evaluating their costs via c ψ , and com- puting empirical quantiles from the resulting samples; these quantiles serve as the particles defining the discrete representation. Let ˆμ π θ and ˆμ π ref be the empirical measures for the cost distrivutions corresponding to policies π θ and π ref respctively. Then, the FSD objective is approximated as L FSD C π θ , C π ref ≈L FSD ˆμ π θ , ˆμ π ref . Computing the RAD policy gradients. The main technical challenge for computing the gradients of the RAD policy is differentiating the FSD term with respect to the policy parameters θ. To com- pute the gradients of the FSD objective effectively, first note that it can be interpreted as an optimal transport problem between the two cost distributions as described in (7). On adding entropic regu- larizion, the resulting optimization problem in (6) is strictly convex and differentiable. Moreover, the entropic regularized optimal problem has smooth gradients and the unique minimizer of the ob- jective can be computed efficiently using Sinkhorn iterations. Hence, we replace the FSD objective with the entropic regularzied FSD objectiveL χ FSD where χ is the regularization parameter (as in (6)) . We next state the resulting policy-gradient estimator; the full derivation and implementation details are deferred to Supplementary Material E. Theorem 1 (RAD policy-gradient estimator). Fix λ ≥ 0 and define the dual objective L(θ,λ) by (11). Let α 1 ,...,α N ∈ (0, 1) be fixed quantile levels and let q i (π θ ) := Q C π θ (α i ) denote the corresponding cost-quantile values. Using the empirical quantile-particle approximation from Equation (13), the gradient∇ θ L(θ,λ) admits the REINFORCE-form estimator ∇ θ L(θ,λ) = E x∼D x ,y∼π θ (·|x) " ̃r(x,y)+λ N X i=1 − ∂L χ FSD ∂q i (π θ ) 1 c ψ (x,y)≤ q i (π θ ) ! ∇ θ logπ θ (y | x) # . (14) Moreover, for the weighted dominance objectiveL w FSD (see Equation (15)), the same estimator holds with an additional factor w(α i ) multiplying the ith summand. We optimize (11) by alternating (i) ascent in θ using (14) with REINFORCE (Williams, 1992) and variance-reduction scheme (RLOO) (Kool et al., 2019), and (i) Dual Ascent (Gallier & Quaintance, 2019) in λ to enforce the constraint. 3.1 Universality of Weighted FSD for Controlling Spectral Risk Measures We now describe how we can change the FSD objective to represesnt various spectral risk measures. The FSD objective in Equation 3 weights all quantiles equally. However, in many applications, tail performance is of primary concern, and it may be desirable to emphasize higher quantiles more heavily than lower ones. To accommodate this, we introduce a quantile-weighted FSD objective with a nonnegative weighting function w(q): L w FSD (X,Y ) = Z 1 0 w(q) Q Y (q)− Q X (q) + dq,(15) where w(q)≥ 0 for all q ∈ [0, 1] and R 1 0 w(q)dq = 1. WhileL w FSD (X,Y ) is not itself a spectral risk measure (SRM), it admits a direct structural relationship to the entire class of SRMs. Recall that a spectral risk measure associated with weight function w is defined as ρ w (Z) := R 1 0 Q Z (q)w(q)dq. Note that we have the following relationship: Z 1 0 w(q) Q Y (q)− Q X (q) dq = ρ w (Y )− ρ w (X).(16) Thus, weighted quantile differences correspond exactly to differences in spectral risk measures. The following proposition formally states that the weighted FSD violations provide a decomposition of spectral risk differences. 6 RiskMeanVaR α CVaR α Linear ExponentialPowerWang MeasureSpectralSpectralSpectralDistortion w(q)1 δ(q− α) 1 1−α 1 [α,1] (q)2q λe λq e λ −1 (1 + λ)q λ Φ(Φ −1 (q)+λ) 1−Φ(λ) Table 1: Examples of weight functions w(q) that induce corresponding spectral risk measure con- straints when used inL w FSD . λ > 0 denotes a risk aversion parameter and Φ in the Wang Distortion (Wang, 1996) risk measure denotes the CDF of a standard normal distribution. Proposition 2. Let w : (0, 1) → R + be any nonnegative weight function with R 1 0 w(q)dq = 1, and define ρ w (Z) := R 1 0 Q Z (q)w(q)dq. For two random variables X and Y , define L w FSD (X,Y ) := Z 1 0 Q Y (q)− Q X (q) + w(q)dq, L w FSD (Y,X) := Z 1 0 Q X (q)− Q Y (q) + w(q)dq. Then, ρ w (Y )−ρ w (X) =L w FSD (X,Y )−L w FSD (Y,X). =⇒ −L w FSD (Y,X) ≤ ρ w (Y )−ρ w (X) ≤ L w FSD (X,Y ). This proposition shows that differences in any spectral risk measure can be decomposed into two weighted FSD violations. Thus, weighted FSD provides a universal dominance-based control mech- anism for the entire class of spectral risk measures. Corollary 3. IfL w FSD (X,Y )≥ κ for some κ > 0, then ρ w (Y )− ρ w (X) =L w FSD (X,Y )−L w FSD (Y,X)≥ κ−L w FSD (Y,X). In particular, if Y ⪰ FSD X so thatL w FSD (Y,X) = 0, then ρ w (X)≤ ρ w (Y )− κ. Note that when κ > L w FSD (Y,X), enforcing L w FSD (X,Y ) ≥ κ implies reduction in ρ w (X) w.r.t. ρ w (Y ). This corollary formalizes the universality property: enforcing a lower bound on the weighted FSD violation guarantees improvement under the corresponding spectral risk measure, provided reverse violations vanish. Substituting the cost distributions X = C π θ and Y = C π ref into Corollary 3, enforcing L w FSD (C π θ ,C π ref )≥ κ =⇒ ρ w (C π θ )≤ ρ w (C π ref )− κ +L w FSD (C π ref ,C π θ ). In particular, if C π ref ⪰ FSD C π θ , then L w FSD (C π ref ,C π θ ) = 0, yielding the strict spectral risk im- provement ρ w (C π θ ) ≤ ρ w (C π ref )− κ. In practice, as optimization proceeds under the weighted FSD constraint,L w FSD (C π θ ,C π ref ) increases whileL w FSD (C π ref ,C π θ ) decreases, typically approach- ing zero near convergence. Table 1 lists representative spectral risk measures and their associated quantile-weighting functions. Figure 2 illustrates the spectral weighting function for different SRMs considered in Table 1. For example, if w(q) corresponds to CVaR α , then enforcing L w FSD (C π θ ,C π ref ) ≥ κ guarantees CVaR α (C π θ ) ≤ CVaR α (C π ref )− κ, whenever reverse violations vanish. If w(q) is uniform, cor- responding to the unweighted FSD objective, then enforcing the FSD constraint guarantees that, at convergence, E[C π θ ]≤ E[C π ref ]− κ. Thus, though a weighted FSD is not a spectral risk measure by itself, it serves as a universal dominance-based control mechanism for the entire class of spectral risk measures. 4 Empirical Results We investigate the following research questions: [Q1] How do models aligned using RAD compare to methods that enforce safety via expected cost constraints, in terms of helpfulness and harmful- ness? [Q2] How do RAD-aligned models perform against baselines wrt harmlessness of responses, on out-of-distribution data? 7 We follow the standard RLHF pipeline for alignment. As our base model, we use Qwen2.5-3B (Qwen et al., 2025). We first fine-tune this model on the Alpaca dataset (Taori et al., 2023) to obtain the SFT initialization. For reward and cost modeling, we use the BeaverTails dataset (Ji et al., 2023), which provides pairwise preference annotations along two dimensions: helpfulness and harmfulness. The helpfulness preferences are used to train a reward model, while the harmfulness preferences are used to train a cost model. Both models are trained using the standard Bradley-Terry objective (Bradley & Terry, 1952) in Equation (18). Importantly, we adopt the same fine-tuning and reward/cost modeling procedure used in Safe RLHF (Dai et al., 2023b), ensuring a controlled comparison. In the RL stage, RAD employs a REINFORCE-based policy gradient method, using Equation (14), with RLOO (Leave-One-Out) baseline and two sampled responses per prompt (k = 2). Additional implementation details, including hyperparameters, are provided in supplementary material A. We set κ = 10 for our RAD optimization objective in Equation (10). To provide a fair comparison with the baselines, we set the Safe-RLHF cost threshold τ =−10 in Equation (2), following Corollary 3, since we observed E x∼D,y∼π ref (·|x) [c ψ (x,y)]≈ 0. 4.1 Model Evaluations We compare models aligned using RAD and Safe RLHF using the same reward and cost models trained on BeaverTails preferences (Ji et al., 2023). Since RAD is a dominance based alignment framework rather than a single objective, we train multiple models corresponding to different spec- tral weighting schemes over the quantiles in the FSD objective. These weighting schemes are listed in Table 1. We evaluate the helpfulness and harmlessness of all model responses, using the prompts in the test set of BeaverTails. We adopt a pairwise evaluation protocol in which we designate one model as the blue model and another as the red model. Typically, the red model represents a baseline (Safe RLHF or SFT) and the blue model represents a RAD variant, though this assignment can vary across comparisons. Harmlessness We use the trained cost model to judge the harmlessness of model generations for all considered approaches. From Table 2, RAD models achieve a higher proportion of safe responses compared to both baselines – SFT and Safe RLHF – highlighting the benefits of enforcing distributional dominance constraints over expected cost alone. Let C π blue and C π red denote the random variables corresponding to the cost distributions induced by the blue and red models, respectively. We define the dominance difference as D w FSD (C π blue ,C π red ) :=L w FSD (C π blue ,C π red )−L w FSD (C π red ,C π blue ). Intuitively, for the blue model to have a safer cost distribution than the red model, we would want L w FSD (C π blue ,C π red ) to be high and L w FSD (C π red ,C π blue ) to be low. This implies a high positive D w FSD (C π blue ,C π red ). We compare the competing models using D w FSD (C π blue ,C π red ), reporting both the weighted and un- weighted dominance difference. The weighted dominance difference corresponds to the difference in spectral risk measures under the chosen weighting: a positive value indicates a reduction in the weighted spectral risk measure for the blue model compared to the red model. The unweighted dom- inance difference corresponds to the difference in average cost: a positive value indicates a lower average cost for the blue model relative to the red model. Table 3 presents the dominance results. We use an unnormalized weighting in the Dominance Difference; this does not affect the results, as all values are scaled by a constant factor for a fixed weighting. From Table 3, we observe that most RAD models, across different quantile weighting functions, achieve a positive weighted dominance metric relative to the baselines, implying a reduction in the corresponding spectral risk measure. Together with the higher proportion of safe responses, these 8 Competition (Red vs Blue) Reward of Blue Higher Reward of Blue Lower Overall Blue Overall Red safe-rlhf vs fsd-uniform0.96± 0.010.83± 0.140.92± 0.05 0.93± 0.00 safe-rlhf vs fsd-var0.88± 0.100.88± 0.100.88± 0.10 0.93± 0.00 safe-rlhf vs fsd-cvar0.96± 0.000.98± 0.000.97± 0.000.93± 0.00 safe-rlhf vs fsd-spectral-linear0.97± 0.000.93± 0.040.94± 0.030.93± 0.00 safe-rlhf vs fsd-spectral-wang0.97± 0.010.95± 0.000.96± 0.00 0.93± 0.00 safe-rlhf vs fsd-spectral-power0.97± 0.010.94± 0.020.96± 0.010.93± 0.00 safe-rlhf vs fsd-spectral-exponential 0.98± 0.010.94± 0.030.96± 0.010.93± 0.00 sft vs fsd-uniform0.95± 0.030.65± 0.160.92± 0.050.55± 0.00 sft vs fsd-var0.93± 0.060.82± 0.100.88± 0.100.55± 0.00 sft vs fsd-cvar0.98± 0.000.92± 0.010.97± 0.01 0.55± 0.00 sft vs fsd-spectral-linear0.97± 0.010.74± 0.150.94± 0.030.55± 0.00 sft vs fsd-spectral-wang0.97± 0.000.75± 0.030.96± 0.00 0.55± 0.00 sft vs fsd-spectral-power0.97± 0.010.76± 0.080.95± 0.010.55± 0.00 sft vs fsd-spectral-exponential0.98± 0.010.66± 0.170.96± 0.010.55± 0.00 sft vs safe-rlhf0.96± 0.000.61± 0.030.93± 0.000.55± 0.00 Table 2: Comparison of the proportion of safe responses produced by the blue model, as judged by the cost model, across competing model pairs. We further decompose this proportion into two subsets based on whether the blue model’s response achieved higher reward than the red model’s response (first two columns). The “Overall Blue” column reports the overall proportion of safe responses for the blue model, while the “Overall Red” column reports the same for the red model. We observe that RAD-aligned models consistently produce a higher proportion of safe responses than the baselines, as evidenced by the comparison of the final two columns. Values are reported as mean± standard deviation across 3 random seeds. results demonstrate the benefits of enforcing distributional dominance constraints over methods that constrain only a central statistic of the cost distribution, such as the expected cost. Importantly, RAD’s improvements are consistent across multiple weighting functions, suggesting that its effec- tiveness is not tied to any particular risk metric. Practitioners can therefore select the weighting function that best reflects their deployment context and risk tolerance – for instance, preferring CVaR-style weightings in high-stakes settings or uniform weighting when average cost reduction is the primary objective. Helpfulness We use the trained reward model, to judge the helpfulness of model generations for all the approaches we consider. We compute the reward win-rates, pitting two models against each other, with their response helpfulness judged by the reward model, which assigns a higher score to more helpful responses. The results are shown in Figure 1. In each cell, in addition to win rate, we also provide the count of the number of responses compared. Each row corresponds to a pairwise competition (red vs. blue). We report the reward win rate of the blue model, defined as the percentage of prompts for which the blue model’s response achieves higher reward than the red model’s response. The x-axis partitions results by safety outcome combinations of the two models: (safe, safe), (safe, unsafe), (unsafe, safe), and (unsafe, unsafe), where the first entry corresponds to the red model and the second to the blue model. For example, (safe, unsafe) denotes the subset of prompts where the red model produces a safe response and the blue model produces an unsafe response. The final column reports the overall win rate across all prompts. From Figure 1, RAD-aligned models achieve higher rewards than the SFT baseline, as evidenced by consistently high reward win rates. Against Safe-RLHF, FSD-VaR, FSD-CVaR, and FSD-Spectral- Linear obtain lower reward win rates, suggesting these weighting schemes are more risk-averse at the expense of helpfulness – a tradeoff that may be desirable in risk-sensitive deployment contexts such as medical or legal applications. The remaining variants achieve comparable reward win rates against Safe-RLHF, indicating helpfulness parity. 9 Competetion (Red vs Blue)D FSD (C π blue ,C π red ) D w FSD (C π blue ,C π red ) safe-rlhf vs fsd-uniform+7.92± 11.23+7.92± 11.23 safe-rlhf vs fsd-var−66.06± 46.66 −2.95± 10.93 safe-rlhf vs fsd-cvar−12.54± 2.80+56.61± 10.39 safe-rlhf vs fsd-spectral-linear−7.33± 7.63 −0.49± 11.59 safe-rlhf vs fsd-spectral-wang+5.74± 5.00+27.91± 14.99 safe-rlhf vs fsd-spectral-power−3.50± 6.34+9.45± 8.67 safe-rlhf vs fsd-spectral-exponential+8.18± 11.04+22.16± 10.04 sft vs fsd-uniform+147.12± 12.43+147.12± 12.43 sft vs fsd-var+73.06± 46.52+17.59± 10.83 sft vs fsd-cvar+126.84± 1.68+139.83± 7.52 sft vs fsd-spectral-linear+131.24± 11.22+135.64± 15.54 sft vs fsd-spectral-wang+144.63± 1.56+432.69± 4.63 sft vs fsd-spectral-power+135.91± 5.54+134.40± 3.95 sft vs fsd-spectral-exponential+147.88± 9.84+145.47± 9.59 sft vs safe-rlhf+138.51± 2.45- Table 3: Unweighted and weighted dominance differences for all pairs of competing models. A positive weighted dominance difference for the blue model indicates a reduction in the correspond- ing spectral risk measure relative to the red model. The weighted dominance difference uses the weighting corresponding to the blue model. Some weighting schemes (e.g., CVaR and VaR) exhibit negative unweighted dominance differences but positive weighted dominance differences. This oc- curs because CVaR only weights the higher quantiles while setting others to zero, and VaR is a Dirac mass at a particular quantile (Gaussian-smoothed for implementation purposes), disregarding other quantiles. Values are reported as mean± standard deviation across 3 random seeds. Summary From the results on the harmlessness and helpfulness of model responses, we observe that RAD-aligned methods produce a higher proportion of safe responses compared to the base- lines, and exhibit a higher positive weighted dominance difference, implying a reduction in spectral risk relative to the baselines. Furthermore, for spectral weighting schemes – specifically Spectral- Wang, Spectral-Power, Spectral-Exponential, and uniform – these gains in safety are made while maintaining parity in helpfulness with Safe RLHF. 4.2 Out-of-Distribution Harmlessness: HarmBench Evaluation To assess generalization beyond the training distribution, we evaluate all models on HarmBench (Mazeika et al., 2024), an out-of-distribution red-teaming benchmark comprising adversarially con- structed harmful prompts across diverse risk categories. Critically, neither the reward model, cost model, nor any model checkpoint was trained or selected using HarmBench data, making this a clean held-out evaluation. We use GPT-4o-mini (OpenAI, 2024) as an independent pairwise harmlessness judge, following standard LLM-as-a-judge (Zheng et al., 2023) evaluation protocol . For each prompt, the judge compares responses from two models and determines which is less harmful. We provide the GPT evaluation results in Table 4 All RAD variants substantially outperform the SFT baseline, confirming that our constrained RL training, with the SFT policy as the reference, improves harmlessness and generalizes to unseen prompts. Against Safe-RLHF, spectral variants – Spectral-Power, Spectral-Exponential, Spectral- Linear, CVaR, and VaR – achieve favorable win-loss ratios, suggesting that upweighting the tail of the cost distribution confers greater robustness to distributional shift. 5 Related Work Standard RLHF typically optimizes LLM behavior using a single preference-based reward signal, which couples helpfulness and harmlessness and can yield either unsafe completions or unhelpful 10 Figure 1: Average reward win rates between competing models, across 3 seeds. Model NameSFTSafe RLHF fsd-uniform67.1%±2.1 / 7.8%±4.2 / 25.1%±6.338.0%±6.9 / 22.2%±15.0 / 39.8%±8.4 fsd-var76.6% ±3.7 / 10.6%±2.7 / 12.7%±2.3 39.6%±10.6 / 28.6%±16.8 / 31.8%±9.7 fsd-cvar72.9%±2.4 / 15.7%±1.3 / 11.4%±2.4 28.6%±7.8 / 46.5%±2.1 / 24.9%±6.6 fsd-spectral-linear75.3%±6.3 / 12.3%±2.0 / 12.4%±5.1 34.2%±2.8 / 37.0%±12.8 / 28.9%±13.4 fsd-spectral-wang67.3%±7.9 / 13.4%±0.6 / 19.3%±7.924.7%±1.7 / 49.7%±7.4 / 25.5%±8.1 fsd-spectral-exponential 74.7% ±1.6 / 12.6%±1.2 / 12.7%±1.6 28.0%±6.1 / 52.2%±7.0 / 19.9%±2.2 fsd-spectral-power71.0%±2.7 / 14.8%±2.1 / 14.1%±4.1 30.4%±2.0 / 48.9%±1.1 / 20.7%±2.7 Safe RLHF68.1%±2.6 / 20.4%±1.1 / 11.5%±2.5- Table 4: Pairwise harmlessness evaluation on HarmBench using GPT-4o-mini as an independent judge. Each cell reports Win / Tie / Lose rates of the row model against the column model (mean± std over 3 seeds). The dominant outcome – win rate if the row model wins, loss rate if it loses – is bold underlined. All FSD variants substantially outperform the SFT baseline. Against Safe RLHF, variants that upweight the tail (Eg: Exponential, Power, Linear, CVaR, and Var) achieve favorable win-loss ratios, demonstrating improved harmlessness generalization to out-of-distribution prompts. blanket refusals in adversarial or sensitive prompts (Ouyang et al., 2022a; Bai et al., 2022a). Com- plementary mitigation strategies aim to reduce harmful outputs using safety critics, filtering mecha- nisms, and curated datasets (Xu et al., 2020; Thoppilan et al., 2022; Ziegler et al., 2022). To make the objectives more controllable, later approaches separate helpfulness and harmlessness signals, e.g., by combining distinct model scores or by using safety as an explicit optimization constraint (Glaese et al., 2022; Mu et al., 2024; Touvron et al., 2023; Ji et al., 2023). Safe RLHF makes this trade-off explicit by training separate reward and cost models and then maximizing reward under an expected-cost constraint in a constrained-MDP view (Dai et al., 2023b; Altman, 2021). HC-RLHF further addresses statistical uncertainty by instantiating this constraint within the Seldonian frame- 11 work and adding a separate safety-screening stage based on upper-confidence bounds computed from held-out data (Thomas et al., 2019; Chittepu et al., 2025). In parallel, a growing line of work argues that expectation objectives can overlook distributional desiderata, advocating stochastic dominance as an explicit ordering criterion and, in some cases, leveraging optimal transport (OT) formulations for tractable learning and matching (Melnyk et al., 2024; Farajzadeh et al., 2025; Cen et al., 2024). The closest complementary advances in stochastic optimization study how to enforce stochastic-dominance constraints via dual/Lagrangian character- izations and surrogate methods to make dominance-constrained optimization computationally prac- tical (Dentcheva & Ruszczynski, 2003; Dai et al., 2023a; Cen et al., 2024). Building on both the constrained-RLHF and dominance-based optimization perspectives, our work adopts the safe-RLHF framework but replaces expectation-based safety control with a first-order stochastic dominance con- straint on the entire cost distribution relative to a reference policy, leveraging OT-based relaxations to render the dominance constraint differentiable and optimizable. Unlike prior Safe RLHF methods that impose scalar (possibly high-confidence) expected-cost constraints (Dai et al., 2023b; Chittepu et al., 2025), our approach enforces an OT-optimized FSD constraint on the full cost distribution, thereby targeting distributional safety guarantees beyond average-cost control. 6 Conclusion In this work, we argued that safety in RLHF should control the full distribution of policy-induced costs, rather than only a central statistic, like their expectation. We proposed Risk-sensitive Align- ment via Dominance (RAD), which enforces first-order stochastic dominance relative to a reference policy and admits a practical optimization procedure via quantile surrogates and entropic optimal transport. By introducing quantile-weighted objectives, RAD also provides a unified mechanism for controlling a broad class of spectral risk measures, allowing safety preferences to be tuned toward different parts of the cost distribution. Empirically, our results show that RAD can improve harm- lessness over expectation-based baselines while remaining competitive in helpfulness, with several variants also showing stronger robustness on out-of-distribution harmfulness evaluations. Acknowledgements This work has taken place in part in the Safe, Correct, and Aligned Learning and Robotics Lab (SCALAR) at The University of Massachusetts Amherst. SCALAR research is supported in part by the NSF (IIS-2437426), the Long-Term Future Fund, and Open Philanthropy. References Carlo Acerbi. Spectral measures of risk: A coherent representation of subjective risk aversion. Journal of banking & finance, 26(7):1505–1518, 2002. Alexandre Adam, Mohamed Houkari, and Jean-Paul Laurent. Spectral risk measures and portfolio selection. Journal of Banking & Finance, 32(9):1870–1882, 2008. Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learn- ing from human feedback in LLMs, 2024. URL https://arxiv.org/abs/2402.14740. Eitan Altman. Constrained Markov decision processes. Routledge, 2021. Philippe Artzner, Freddy Delbaen, Jean-Marc Eber, and David Heath. Coherent measures of risk. Mathematical finance, 9(3):203–228, 1999. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022a. 12 Yuntao Bai et al. Constitutional AI: Harmlessness from AI feedback. ArXiv, abs/2212.08073, 2022b. URL https://api.semanticscholar.org/CorpusID:254823489. Yuntao Bai et al. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022c. URL https://arxiv.org/abs/2204.05862. James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http: //github.com/jax-ml/jax. Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. Shicong Cen, Jincheng Mei, Hanjun Dai, Dale Schuurmans, Yuejie Chi, and Bo Dai. Beyond expec- tations: Learning with stochastic dominance made practical. arXiv preprint arXiv:2402.02698, 2024. Yaswanth Chittepu, Blossom Metevier, Will Schwarzer, Austin Hoag, Scott Niekum, and Philip S Thomas. Reinforcement learning from human feedback with high-confidence safety guarantees. In Reinforcement Learning Conference, 2025. Paul Francis Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. ArXiv, abs/1706.03741, 2017. URL https://api.semanticscholar.org/CorpusID:4787508. Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013. Hanjun Dai, Yuan Xue, Niao He, Yixin Wang, Na Li, Dale Schuurmans, and Bo Dai. Learning to optimize with stochastic dominance constraints. In International Conference on Artificial Intelli- gence and Statistics, p. 8991–9009. PMLR, 2023a. Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.12773, 2023b. George B Dantzig. Linear programming and extensions. 2016. Darinka Dentcheva and Andrzej Ruszczynski. Optimization with stochastic dominance constraints. SIAM Journal on Optimization, 14(2):548–566, 2003. Kevin Dowd, John Cotter, and Ghulam Sorwar. Spectral risk measures: properties and limitations. Journal of Financial Services Research, 34(1):61–75, 2008. Pavel Dvurechensky, Alexander Gasnikov, and Alexey Kroshnin. Computational optimal transport: Complexity by accelerated gradient descent is better than by sinkhorn’s algorithm. In Interna- tional conference on machine learning, p. 1367–1376. PMLR, 2018. Ali Farajzadeh, Danyal Saeed, Syed M Abbas, Rushit N Shah, Aadirupa Saha, and Brian D Ziebart. Imitation beyond expectation using pluralistic stochastic dominance. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanis- las Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, Ievgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78): 1–8, 2021. URL http://jmlr.org/papers/v22/20-451.html. 13 Jean Gallier and Jocelyn Quaintance. Fundamentals of optimization theory with applications to machine learning. University of Pennsylvania Philadelphia, PA, 19104, 2019. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858, 2022. Leo Gao, John Schulman, and Jacob Hilton.Scaling laws for reward model overoptimiza- tion.In International Conference on Machine Learning, 2022.URL https://api. semanticscholar.org/CorpusID:252992904. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Realtoxici- typrompts: Evaluating neural toxic degeneration in language models. In Findings of the associa- tion for computational linguistics: EMNLP 2020, p. 3356–3369, 2020. Amelia Glaese, Nat McAleese, Maja Tr ̨ebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Mari- beth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375, 2022. Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Àgata Lapedriza, Noah J. Jones, Shixiang Shane Gu, and Rosalind W. Picard. Way off-policy batch deep rein- forcement learning of implicit human preferences in dialog. ArXiv, abs/1907.00456, 2019. URL https://api.semanticscholar.org/CorpusID:195766797. Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Chi Zhang, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of LLM via a human-preference dataset, 2023. URL https://arxiv.org/abs/2307.04657. Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences, 103:102274, 2023. Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. Gpt-4 passes the bar exam. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 382(2270), 2024. Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get a baseline for free! In DeepRLStructPred@ICLR, 2019. URL https://api.semanticscholar.org/ CorpusID:198489118. Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. Per- formance of chatgpt on usmle: potential for ai-assisted medical education using large language models. PLoS digital health, 2(2):e0000198, 2023. Chenglin Li, Guangchun Ruan, and Hua Geng. Tilted quantile gradient updates for quantile- constrained reinforcement learning, 2024. URL https://arxiv.org/abs/2412.13184. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: A stan- dardized evaluation framework for automated red teaming and robust refusal, 2024.URL https://arxiv.org/abs/2402.04249. Igor Melnyk, Youssef Mroueh, Brian Belgodere, Mattia Rigotti, Apoorva Nitsure, Mikhail Yurochkin, Kristjan Greenewald, Jiri Navratil, and Jarret Ross. Distributional preference align- ment of llms via optimal transport. Advances in Neural Information Processing Systems, 37: 104412–104442, 2024. 14 Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelli- gence. Nature, 616(7956):259–265, 2023. Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. Rule based rewards for language model safety. Advances in Neural Information Processing Systems, 37:108877–108901, 2024. OpenAI. Gpt-4o mini: Advancing cost-efficient intelligence. https://openai.com/index/ gpt-4o-mini-advancing-cost-efficient-intelligence/, 2024. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730–27744, 2022a. Long Ouyang et al. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155, 2022b.URL https://api.semanticscholar.org/CorpusID: 246426909. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, p. 8024–8035. Curran Associates, Inc., 2019. URL http://papers.neurips.c/paper/ 9015-pytorch-an-imperative-style-high-performance-deep-learning-library. pdf. Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. Asleep at the keyboard? assessing the security of github copilot’s code contributions, 2021. URL https://arxiv.org/abs/2108.09293. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models, 2022. URL https://arxiv.org/abs/2202.03286. Gabriel Peyré and Marco Cuturi. Computational optimal transport, 2020. URL https://arxiv. org/abs/1803.00567. Qwen et al. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115. Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit S. Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct align- ment algorithms. ArXiv, abs/2406.02900, 2024. URL https://api.semanticscholar. org/CorpusID:270257855. Filippo Santambrogio.Optimal transport for applied mathematicians: Calculus of variations, pdes, and modeling. 2015. URL https://api.semanticscholar.org/CorpusID: 124181096. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402. 03300. 15 Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022. URL https://arxiv.org/abs/2009.01325. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford Alpaca: An instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca, 2023. Philip S Thomas, Bruno Castro da Silva, Andrew G Barto, Stephen Giguere, Yuriy Brun, and Emma Brunskill. Preventing undesirable behavior of intelligent machines. Science, 366(6468):999– 1004, 2019. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. Shaun Wang. Premium calculation by transforming the layer premium density. ASTIN Bulletin: The Journal of the IAA, 26(1):71–92, 1996. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229–256, 1992. Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. Recipes for safety in open-domain chatbots. arXiv preprint arXiv:2010.07079, 2020. Xi Yang, Aokun Chen, Nima PourNejatian, Hoo Chang Shin, Kaleb E Smith, Christopher Parisien, Colin Compas, Cheryl Martin, Anthony B Costa, Mona G Flores, et al. A large language model for electronic health records. NPJ digital medicine, 5(1):194, 2022. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023. URL https://arxiv. org/abs/2306.05685. Daniel M Ziegler, Seraphina Nix, Lawrence Chan, Tim Bauman, Peter Schmidt-Nielsen, Tao Lin, Adam Scherlis, Noa Nabeshima, Ben Weinstein-Raun, Daniel de Haas, et al. Adversarial training for high-stakes reliability. arXiv preprint arXiv:2205.01663, 2022. 16 Supplementary Materials The following content was not necessarily subject to peer review. A Implementation Details We use the Safe-RLHF repository (Dai et al., 2023b) and build RAD on top of their publicly available codebase. For our Safe-RLHF runs, we use the hyperparameters reported in their paper. For our RAD implementation, we use REINFORCE (Williams, 1992) with the RLOO variance-reduction baseline (Kool et al., 2019) and k = 2 samples per prompt. To compute the quantiles required for the RAD policy gradient, we gather samples across all GPUs before computing quantiles, increasing the number of available samples. The particle gradient is computed using the Python Optimal Transport library (Flamary et al., 2021). All RAD models were trained on two NVIDIA A100 GPUs, while Safe-RLHF runs required four NVIDIA A100 GPUs due to the additional memory overhead of storing and training the critic network. HyperparameterValue Sinkhorn regularization χ0.01 KL coefficient β (same value used for Safe RLHF)0.1 κ10 Spectral-Wang λ0.7 Spectral-Exponential λ3.0 Spectral-Power λ2.0 Gaussian bandwidth (for VaR)0.1 α (CVaR α / VaR α )0.9 Table 5: RAD-specific hyperparameters. All other hyperparameters are the same as ones used in Safe-RLHF, unless specified otherwise. B RLHF Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017; Ouyang et al., 2022b) is currently the predominant strategy for aligning Large Language Models (LLMs) with human intent and preferences. The process typically begins with a pre-trained base model, which has been trained on internet-scale data via a next-token prediction objective. The standard RLHF pipeline generally consists of three distinct stages: Supervised Fine-Tuning (SFT), Reward Model- ing (RM), and Reinforcement Learning (RL). We detail each of these stages below. Supervised Fine-Tuning In the SFT stage, the pre-trained model is trained to follow instructions, via a next-token prediction objective. This process utlizes a high quality dataset of prompt-responses pairs D sft , where the responses are provided by either a human or LLM annotator (Bai et al., 2022b). We refer to the resulting policy from the SFT stage as π sft . Reward Modeling.In the reward modeling stage, we train a reward model to capture human pref- erences. This stage relies on a dataset of human preferences D pref = (x i ,y + i ,y − i ) N i=1 , where x i is a prompt, y + i is the preferred response to x i , and y − i is the dispreferred response to x i . Prefer- ence annotations are collected either from human annotators or LLM annotators. Preferences are modeled using the Bradley–Terry preference model (Bradley & Terry, 1952), where the log-odds of an observed preference equals the difference between the rewards assigned to the preferred and dispreferred responses by a latent reward function r(x,y): P (y + ≻ y − | x) = e r(x,y + ) e r(x,y + ) + e r(x,y − ) = σ r(x,y + )− r(x,y − ) ,(17) 17 whereσ denotes the sigmoid function. A parameterized reward function with parametersφ is learned to approximate the latent reward function through maximum likelihood estimation on the preference dataset D pref , using the following objective: min φ −E (x,y + ,y − )∼D pref logσ r φ (x,y + )− r φ (x,y − ) .(18) Reinforcement Learning In the Reinforcement Learning stage, a language model or policy is trained to generate responses that are preferred by humans by maximizing the reward of generated responses, as measured by the reward model r φ (x,y). However, directly optimizing the reward of model generations can lead to degradation in response quality (Stiennon et al., 2022; Ouyang et al., 2022b; Jaques et al., 2019) due to a phenomenon known as reward overoptimization (Gao et al., 2022; Rafailov et al., 2024), where the model overfits to imperfections in the learned proxy reward function. To mitigate this effect, a KL penalty term is added to the objective to penalize the model (policy) from drifting too far away from a reference policy, usually chosen to be π sft . The RL objective can be expressed as max θ E x∼D x ,y∼π θ (.|x) [r φ (x,y)]− βD KL π θ (.|x)||π ref (.|x) (19) Denoting the KL-regularized reward as ̃r(x,y) = r φ (x,y)− β log π θ (y|x) π ref (y|x) , we can express the RL objective compactly as max θ E x∼D x ,y∼π θ (.|x) [ ̃r(x,y)](20) The RL objective can be optimized using a variety of reinforcement learning algorithms, such as Proximal Policy Optimization (PPO) (Schulman et al., 2017), Group Relative Policy Optimization (GRPO) (Shao et al., 2024), or REINFORCE (Williams, 1992; Ahmadian et al., 2024). C Entropic Regularization of Optimal Tranport and the Sinkhorn Algorithm The unregularized optimal transport problem in Equation (5) can be solved using linear program- ming (Dantzig, 2016; Dvurechensky et al., 2018), but this can be computationally expensive when the number of particles is large. Hence, an entropy regularization term is typically added to the ob- jective, which allows for efficient optimization using the Sinkhorn algorithm (Cuturi, 2013). Specif- ically, we define the entropic regularized OT problem between distributions μ and ν as: OT χ c (μ,ν) =min P∈Π(μ,ν) ⟨P,C⟩− χH(P ), H(P ) = − X i,j P ij logP ij ,(21) for a regularization parameter χ > 0 and a cost matrix C. Recall that Π(μ,ν) represents the set of admissible couplings as stated in (4) (and defined by the marginal of μ and ν). This problem is strictly convex in P and has a unique minimizer P ⋆ . More importantly, it can be solved efficiently using the Sinkhorn algorithm (Cuturi, 2013), which iteratively updates scaling vectors to enforce the marginal constraints. The resulting transport plan P ⋆ can then be used to compute the optimal transport objective and its gradient with respect to the policy parameters θ. We now decribe how to compute the optimal transport objective. Let the marginals of μ and ν be the vectors a and b respectively. Define the Gibbs kernel K ij = e −C ij /χ .(22) The optimal transport plan can be expressed in closed form as P ⋆ = diag(u)Kdiag(v),(23) 18 where u∈ R N , v ∈ R M are scaling vectors that can be computed using the Sinkhorn iterations u (t+1) = a Kv (t) , v (t+1) = b K ⊤ u (t+1) ,(24) starting from some positive initialization (e.g. u (0) = v (0) = 1). The iterations are guaranteed to converge to the unique optimal plan P ⋆ that satisfies the marginal constraints (Peyré & Cuturi, 2020; Cuturi, 2013). Once we have P ⋆ , we can compute the FSD objective as L χ FSD (μ,ν) = ⟨P ⋆ ,C⟩− χH(P ⋆ ).(25) Because the algorithm consists only of a fixed number of matrix-vector multiplications and divisions, gradients can be propagated through the iterations using automatic differentiation in frameworks like PyTorch (Paszke et al., 2019) or JAX (Bradbury et al., 2018), making the Sinkhorn algorithm end to end differentiable. D FSD Theorem 4 (from Melnyk et al. (2024)). Let h : R → R + be a convex function, and let X and Y be real-valued random variables with probability measures μ X and μ Y , respectively. Denote by Q X ,Q Y : [0, 1]→ R their (left-continuous) quantile functions. Then, Z 1 0 h Q X (q)− Q Y (q) dq =min γ∈Π(μ X ,μ Y ) Z R 2 h(x− y)dγ(x,y),(26) where Π(μ X ,μ Y ) denotes the set of all couplings (joint probability measures) on R 2 with marginals μ X and μ Y . Moreover, if h is strictly convex, the minimizing transport plan γ ⋆ is unique. Corollary 5 (FSD as asymmetric optimal transport). Let X,Y be real-valued random variables with measures μ X and μ Y , and define the FSD objective L FSD (X,Y ) := Z 1 0 Q Y (q)− Q X (q) + dq. Then L FSD (X,Y ) =min γ∈Π(μ X ,μ Y ) Z R 2 (y− x) + dγ(x,y), that is, the FSD objective coincides with the optimal transport cost under the asymmetric convex cost function c(x,y) = (y− x) + . E Derivation of Theorem 1 E.1 RAD Policy Gradient We represent the cost distribution via a particle approximation in quantile space. Letq i N i=1 denote an ordered set of cost quantiles satisfying q 1 ≤ q 2 ≤·≤ q N . We approximate the induced cost distribution by the empirical measure ˆμ = 1 N N X i=1 δ q i , where δ q i denotes the Dirac measure located at q i . In practice, this approximation is obtained by sampling generations from the policy, evaluating their associated costs, and computing empirical 19 quantiles from the resulting samples. These quantiles serve as the particles defining the discrete representation of the distribution. Let ˆμ π θ and ˆμ π ref denote the empirical cost distributions of the policy π θ and π ref respectively. Note that we represent C π θ and C π ref using their quantiles q i (π θ ) N i=1 and q i (π ref ) N i=1 respectively. The gradient of the FSD objective can be expressed using the chain rule as follows. ∇ θ L FSD ˆμ π θ , ˆμ π ref =∇ θ L FSD q i (π θ ) N i=1 ,q i (π ref ) N i=1 (27) = N X i=1 ∂L FSD ∂q i (π θ ) ∂q i (π θ ) ∂θ = D ∇ q(π θ ) L FSD ,∇ θ q(π θ ) E (28) We denote the gradient vector ∇ q(π θ ) L FSD as the particle gradient. This gradient vector tells us how the FSD objective changes when the particles (quantiles in our setting) used to approximate the empirical distribution of costs of the policy generations are perturbed slightly. The other gradient vector ∇ θ q(π θ ) is the quantile gradient, which tells us how the quantiles of the cost distribution change, when the policy parameters are perturbed slightly. Estimating the Particle Gradient The FSD objective can be reinterpreted as an optimal transport problem with an asymmetric cost c(x,y) = (y − x) + as decribed in Corollary 5. The particle gradient can then be viewed as the sensitivity of the optimal transport objective to infinitesimal perturbations of the particle locations. The classical optimal transport problem (as described in (5)) between ˆμ π θ and ˆμ π ref is OT c (ˆμ π θ , ˆμ π ref ) :=min P∈Π(ˆμ π θ , ˆμ π ref ) ⟨P,C⟩ =min P∈Π(ˆμ π θ , ˆμ π ref ) N X i=1 M X j=1 P ij c(x i ,y j ).(29) where C ij = c(x i ,y j ) is the the cost matrix. Recall that Π(ˆμ π θ , ˆμ π ref ) is the set of admissible couplings defined as Π(ˆμ π θ , ˆμ π ref ) := n P ∈ R N×N + : P1 M = a, P ⊤ 1 N = b o . The vectors a and b represent the marginals. In our case, the marginals of the empirical cost distributions are 1 N . The classical optimal transport objective as described in (29) is not smooth with respect to particle positions. It is also non-differentiable and small perturbations in particle locations can induce dis- continuous changes in the optimal coupling P ∗ . To address these issues, we instead consider the entropy-regularized optimal transport problem as described in (21). The regularized objective L χ FSD (ˆμ π θ , ˆμ π ref ) = OT χ (y−x) + (ˆμ π θ , ˆμ π ref ) adds a strictly convex entropic penalty to the transport plan, yielding a smooth objective that is differentiable with respect to the particle positions. The solution can be computed efficiently using the Sinkhorn algorithm, enabling end-to-end differentiation as described in Section C. The regularized objectiveL χ FSD is a smooth approximation of the original FSD objectiveL FSD . L χ FSD −→ L FSD as χ→ 0. In practice, χ is chosen to balance approximation bias and numerical stability. Estimating the Quantile Gradient The quantile gradient informs us how all the quantiles in our empirical distribution of costs of policy generations, change when the policy parameters are perturbed slightly. We consider only the quantile gradient of a single quantile here. The quantile gradient wrt all quantiles can be computed in a similiar manner. LetF θ be the cumulative distribution function (CDF) of the cost distribution of the model outputs under policy π θ . The quantile gradient of a particular quantile q i is canonically given by ∇ θ q i (π θ ) =− ∇ θ F θ (t)| t=q i f θ (q i ) . 20 Since the PDF in the denominator is unknown, the derivative can be approximated as (Li et al., 2024) ∇ θ q i (π θ )≈−∇ θ F θ (q) q=q i (π θ ) (30) The CDF F θ can be expressed as an expectation and we can use the REINFORCE trick (Williams, 1992) to compute the policy gradient of the CDF of the cost distribution of model generations. ∇ θ F θ (q) =∇ θ E x∼D x ,y∼π θ (.|x) h 1(c ψ (x,y)≤ q) i (31) = E x∼D x ,y∼π θ (.|x) h 1(c ψ (x,y)≤ q)∇ θ logπ θ (y|x) i (32) Recall that the cost distribution is defined via the learned cost model c ψ (x,y) where x ∼ D x is the input prompt and y ∼ π θ (.|x) is the model output. The quantile gradient for q i can then be expressed as ∇ θ q i (π θ )≈−E x∼D x ,y∼π θ (.|x) h 1(c ψ (x,y)≤ q i )∇ θ logπ θ (y|x) i (33) Putting it all together Using the particle gradient and the quantile gradient, the gradient of the FSD objective with respect to policy parameters can be expressed as ∇ θ L FSD (ˆμ θ , ˆμ ref ) = N X i=1 ∂L χ FSD ∂q i (π θ ) ∂q i (π θ ) ∂θ (34) = N X i=1 − ∂L χ FSD ∂q i (π θ ) E x∼D x ,y∼π θ (.|x) h 1(c ψ (x,y)≤ q i )∇ θ logπ θ (y|x) i (35) = E x∼D x ,y∼π θ (.|x) h N X i=1 − ∂L χ FSD ∂q i (π θ ) 1(c ψ (x,y)≤ q i ) ∇ θ logπ θ (y|x) i (36) Denoting the RAD objective as max θ min λ≥0 L(θ,λ) the RAD policy gradient can then be ex- pressed as ∇ θ L(θ,λ) = E x∼D x ,y∼π θ (.|x) h ̃r(x,y) + λ N X i=1 − ∂L χ FSD ∂q i (π θ ) 1(c ψ (x,y)≤ q i ) ∇ θ logπ θ (y|x) i (37) We then use REINFORCE (Williams, 1992) with RLOO (Kool et al., 2019) to optimize the RAD objective using the policy gradient expression in Equation 37. For the setting, where we weight quantiles differently in the FSD objective as in Equation 15, the policy gradient can be expressed as follows, only requiring an additional weighting coefficient w(q i ). ∇ θ L(θ,λ) = E x∼D x ,y∼π θ (.|x) h ̃r(x,y)+λ N X i=1 −w(q i ) ∂L χ FSD ∂q i (π θ ) 1(c ψ (x,y)≤ q i ) ∇ θ logπ θ (y|x) i (38) 21 Figure 2: Illustration of spectral weighting functions corresponding to different spectral risk mea- sures (SRMs). For parameterized families, the plots show how the spectral weights vary with the risk-aversion parameter λ. Note that VaR is represented as a gaussian with very small bandwidth instead of a dirac delta for implementation reasons. 22