Paper deep dive
Suan: Rectifying Direct Preference Safety Alignment in Large Language Models
Oleksandr Cherednichenko, Roman Klypa
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/14/2026, 5:28:05 AM
Summary
The paper introduces Suan, a novel preference optimization algorithm for aligning Large Language Models (LLMs) with safety constraints. Unlike existing Direct Alignment Algorithms (DAAs) like DPO, IPO, and SafeDPO that suffer from over-refusal and likelihood displacement, Suan formulates the optimization objective directly at the gradient level. This approach bypasses standard variational derivations, resulting in more interpretable and robust training dynamics. Extensive evaluations demonstrate that Suan achieves superior safety alignment while preserving response utility and reducing over-refusal rates across various open-weight model families.
Entities (13)
Relation Signals (10)
Roman Klypa → affiliatedwith → Univ. Grenoble-Alpes
confidence 95% · Affiliation: Univ. Grenoble Alpes, CNRS, Grenoble INP, LJK
Oleksandr Cherednichenko → affiliatedwith → Umeå University
confidence 95% · Affiliation: Department of Mathematics and Mathematical Statistics Affiliation: Integrated Science Lab, Umeå University
Suan → solves → Over-refusal
confidence 95% · Suan successfully achieves all the Safety Alignment goals... reducing over-refusal
Suan → solves → Likelihood Displacement
confidence 95% · Suan avoids common safety alignment pitfalls such as over-refusal and utility degradation... preventing unbounded policy drift
DPO → causes → Over-refusal
confidence 90% · existing preference optimization methods, such as DPO... frequently trigger over-refusal
DPO → causes → Likelihood Displacement
confidence 90% · DPO and SafeDPO both exhibit a pronounced decline in the likelihood of preferred completions
Suan → outperforms → SafeDPO
confidence 90% · Suan consistently achieves the optimal alignment trade-off... matching DPO’s strong harmlessness without sacrificing helpfulness
Suan → outperforms → DPO
confidence 90% · Suan consistently achieves the optimal alignment trade-off... matching DPO’s strong harmlessness without sacrificing helpfulness
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Integrating robust safety guardrails into Large Language Models (LLMs) is essential for delivering helpful yet harmless responses. While proprietary systems exhibit reliable safety controls, their underlying methodologies and trade-offs remain largely undisclosed. Achieving comparable security in open-weight models remains a persistent challenge, as post-trained variants frequently suffer from over-refusal and degraded general quality. To overcome these drawbacks, we introduce Suan, a novel preference optimization algorithm. Unlike existing methods, we formulate the optimization objective directly at the gradient level, bypassing the standard variational derivation. As a result, we obtain more interpretable and robust training dynamics. Extensive evaluations across a diverse suite of competitive baselines and benchmarks demonstrate that Suan achieves superior safety alignment while fully preserving response utility.
Tags
Links
- Source: https://arxiv.org/abs/2609.08634v1
- Canonical: https://arxiv.org/abs/2609.08634v1
Trouble viewing inline? Open PDF directly →
Full Text
81,077 characters extracted from source content.
Expand or collapse full text
Suan: Rectifying Direct Preference Safety Alignment in Large Language Models Oleksandr Cherednichenko †thanks: Equal contribution Affiliation: Department of Mathematics and Mathematical Statistics Affiliation: Integrated Science Lab, Umeå University Affiliation: Umeå, Sweden Email: oleksandr.cherednichenko@umu.se Roman Klypa11footnotemark: 1 Affiliation: Univ. Grenoble Alpes, CNRS, Grenoble INP, LJK Affiliation: 38000 Grenoble, France Email: roman.klypa@univ-grenoble-alpes.fr Abstract Integrating robust safety guardrails into Large Language Models (LLMs) is essential for delivering helpful yet harmless responses. While proprietary systems exhibit reliable safety controls, their underlying methodologies and trade-offs remain largely undisclosed. Achieving comparable security in open-weight models remains a persistent challenge, as post-trained variants frequently suffer from over-refusal and degraded general quality. To overcome these drawbacks, we introduce Suan, a novel preference optimization algorithm. Unlike existing methods, we formulate the optimization objective directly at the gradient level, bypassing the standard variational derivation. As a result, we obtain more interpretable and robust training dynamics. Extensive evaluations across a diverse suite of competitive baselines and benchmarks demonstrate that Suan achieves superior safety alignment while fully preserving response utility. Warning: This paper contains red-teaming content, which may qualify as harmful. 1 Introduction Recent advances in natural language processing have driven the widespread adoption of Large Language Models (LLMs) across diverse domains, including software engineering and healthcare [1, 2, 3, 4, 5, 6]. Despite their ubiquity, LLMs can generate harmful or malicious content [7], including misinformation, malware, hazardous instructions, and leaked private data [8, 9, 10, 11, 12]. Standard LLM development relies on a three-stage training pipeline. Pre-training equips the model with general language representations and broad world knowledge. Supervised Fine-Tuning (SFT) subsequently adapts these raw capabilities toward instruction-following behaviors [13, 14, 15]. Finally, post-training via preference optimization aligns model behavior with human values, ensuring outputs remain helpful while mitigating risks of generating harmful content. Traditionally, this final alignment stage relied on Reinforcement Learning from Human Feedback (RLHF) [16, 17, 18]. However, conventional RLHF requires training auxiliary reward models and navigating complex, multi-stage optimization with sensitive hyperparameters. To streamline this process, Direct Alignment Algorithms (DAAs), such as Direct Preference Optimization (DPO) [19], have emerged as a compelling alternative, optimizing preferences directly through closed-form loss formulations. Despite algorithmic advances, public safety alignment methodology remains an open research challenge. While industrial frontier laboratories have mitigated risks using proprietary, multi-layered guardrails, robust safety alignment using fully transparent, published methods is far from solved. Academic efforts actively introduce novel safety algorithms [20, 21, 22, 23, 24], yet open-weight models aligned using these techniques remain highly vulnerable to evolving adversarial jailbreaks [25, 26, 27, 28, 29, 30, 7, 31, 32]. Furthermore, existing preference optimization methods, such as DPO [19], IPO [33], and SafeDPO [23], frequently trigger over-refusal and general utility degradation (Figure 1). Compounding these issues, such objectives often induce likelihood displacement, further compromising response quality [34, 35, 36]. Figure 1: Conceptual overview of LLM Safety Alignment. The main goals are to train the assistant to be harmless, helpful and compliant. Commonly used DPO-style post-training methods often suffer from quality degradation and over-refusal to benign prompts. Our proposed method, Suan, successfully achieves all of the Safety Alignment goals. To counter these challenges, we take a different approach from standard Direct Alignment Algorithms. Rather than focusing on the modifying the general preference loss function derived from relaxed formulations of the underlying safety-constrained problem, we focus on objective interpretability at the gradient level. Our main contributions can be summarized as follows: 1. We conduct a systematic investigation into the open challenges of safety-focused preference optimization, such as over-refusal and quality degradation, across diverse model families, datasets, and benchmarks. 2. We propose Suan, a simple, interpretable direct alignment algorithm with robust training dynamics. 3. Through extensive evaluations, we demonstrate that Suan achieves state-of-the-art performance, simultaneously enhancing safety and reducing over-refusal without degrading response quality. 2 Theoretical Preliminaries 2.1 Post-training of Large Language Models Large Language Models are trained as next-token predictors over a discrete vocabulary V. Given a sequence of tokens x1:L=(x1,…,xL)x_1:L=(x_1,…,x_L), the model defines a conditional distribution πθ(⋅|x<l) _θ(·|x_<l) over the next token xlx_l. By applying the chain rule, the joint probability of an entire sequence x1:Lx_1:L under the model parameters θ is decomposed into the product of the conditional distributions: πθ(x1:L)=∏l=1Lπθ(xl|x<l), _θ(x_1:L)= _l=1^L _θ(x_l|x_<l), (1) To transition from general next-token prediction to reliable instruction following, the base model is trained via Supervised Fine-Tuning on a dataset of prompt-response pairs SFT≐(xi,yi)∣i=1,…,ND_SFT \(x_i,y_i) i=1,…,N\. Training is aimed to minimize the standard Cross-Entropy loss: ℒCE(θ)=(x,y)∼SFT[−logπθ(y∣x)].L_CE(θ)=E_(x,y) _SFT [- _θ(y x) ]. (2) The subsequent alignment stage relies on a dataset of human (or LLM)-annotated preference pairs, denoted as PO≐(xi,yi+,yi−)i=1MD_PO \(x_i,y +_i,y -_i)\_i=1^M, where xix_i is the prompt, and yi+y +_i and yi−y -_i are the preferred and dispreferred responses, respectively. The standard Reinforcement Learning from Human Feedback paradigm addresses this setting in two steps. First, a scalar reward model rϕ(x,y)r_φ(x,y) is fitted to POD_PO. Second, the target policy πθ _θ is fine-tuned to maximize the expected reward while remaining close to the reference policy πref _ ref (the SFT model) via a KL-divergence constraint: maxθx∼,y∼πθ(x)[rϕ(x,y)]−βDKL(πθ(y∣x)||πref(y∣x)), _θE_x ,y _θ(x)[r_φ(x,y)]-β D_KL ( _θ(y x)\,||\, _ ref(y x) ), (3) where β>0β>0 controls the strength of the KL penalty to prevent reward hacking and language degeneration. To solve this optimization problem, one approach is to employ Proximal Policy Optimization (PPO) [37]. Because the expectation in Eq. (3) is taken over responses generated directly by the current policy y∼πθ(x)y _θ(x), PPO operates as an online, model-free policy gradient algorithm. Preference optimization alone might not guarantee safety, as users can prefer responses with harmful content. To address this, Safe-RLHF [20] extends the framework using constrained reinforcement learning. By incorporating a dedicated safety cost model alongside the helpfulness reward, it explicitly constrains the policy to minimize the probability of generating unsafe responses. In practice, optimizing alignment objectives (Eq. 3) via online policy gradient methods, such as PPO or GRPO [38], requires estimating the per-token KL-divergence over sampled rollout trajectories rather than computing it explicitly over the full sequence space. To this end, several empirical KL estimators are used [39], each offering distinct trade-offs regarding variance, computational efficiency, and non-negativity guarantees [40, 41]. 2.2 Direct Alignment Algorithms Although RLHF pipeline has achieved remarkable success in aligning with human preferences, its complex multi-step nature makes it expensive in terms of computation time and memory usage. Another limitation is dependence on the reward model, which can result in reward hacking and other degeneracies. For example, it has been demonstrated [42] that underfitted reward model is better for models downstream performance. Due to these limitations, Direct Alignment Algorithms have emerged as a powerful tool. The pioneering method in that domain is Direct Preference Optimization [19]. It is derived by substituting the optimal policy from the KL-regularized RLHF objective into the Bradley–Terry preference model [43], yielding a closed-form loss that directly trains the policy from preference pairs. Such derivation allows offline training directly on preference data without an explicit reward model, making the process significantly cheaper than RLHF. Definition 2.1 (DPO Loss [19]). The training objective of the Direct Preference Optimization method is given by: ℒDPO(θ)=(x,y+,y−)∼PO[−log(σ(βlogπθ(y+∣x)πθ(y−∣x)−βlogπref(y+∣x)πref(y−∣x)))],L_DPO(θ)=E_(x,y +,y -) _PO [- (σ (β _θ(y + x) _θ(y - x)-β _ref(y + x) _ref(y - x) ) ) ], (4) where β∈ℝ+β ^+ is a hyperparameter, σ:ℝ→[0,1]σ:R→[0,1] is the sigmoid function, and πref _ref is a reference policy. Subsequent works have extended the Direct Preference Optimization framework to address its core limitations and strengthen its theoretical foundation. A prominent example is Identity Preference Optimization (IPO) [33], which regularizes the policy to prevent the implicit reward margin between preferred and dispreferred responses from growing unbounded, thereby mitigating overfitting to offline preference datasets. Definition 2.2 (IPO Loss [33]). The training objective of the Identity Preference Optimization method is given by: ℒIPO(θ)=(x,y+,y−)∼PO[(logπθ(y+∣x)πθ(y−∣x)−logπref(y+∣x)πref(y−∣x)−12κ)2],L_IPO(θ)=E_(x,y +,y -) _PO [ ( _θ(y + x) _θ(y - x)- _ref(y + x) _ref(y - x)- 12κ )^2 ], (5) where 12κ 12κ is a fixed gap target and πref _ref is a reference policy. Another key development is SafeDPO [23], which explicitly integrates safety constraints into the alignment process by incorporating safety metadata for response pairs. Formally, the standard preference dataset is augmented with binary safety indicators, yielding tuples (x,y+,y−,h+,h−)∼SPO(x,y +,y -,h +,h -) _SPO, where h+,h−∈0,1h +,h -∈\0,1\ denote the safety status of the preferred (y+y +) and dispreferred (y−y -) responses, respectively. During training, the dataset is dynamically filtered: if the preferred response is unsafe while the dispreferred response is safe, the preference order is inverted. Conversely, if both candidates are flagged as unsafe, the tuple is discarded entirely. Definition 2.3 (SafeDPO Loss [23]). The training objective of the SafeDPO method is given by: ℒSafe(θ)=SPO[−log(σ(βlogπθ(y+∣x)πθ(y−∣x)−βlogπref(y+∣x)πref(y−∣x)−Δ(h−h+)))],L_Safe(θ)=E_D_SPO [- (σ (β _θ(y + x) _θ(y - x)-β _ref(y + x) _ref(y - x)- (h --h +) ) ) ], (6) where β∈ℝ+β ^+ and Δ∈ℝ+ ^+ are hyperparameters and πref _ref is a reference policy. Despite the algorithmic diversity across Direct Alignment Algorithms [44, 45], many unified frameworks demonstrate that a majority of these methods optimize a single, generalized objective formulation (Proposition 2.4). Proposition 2.4 (General DAA form [36]). Most of the Direct Alignment Algorithms can be unified under objective of a general form: ℒPO(θ)=(x,y+,y−)∼PO[ℓx,y+,y−(logπθ(y+∣x)−logπθ(y−∣x))],L_PO(θ)=E_(x,y +,y -) _PO [ _x,y +,y - ( _θ(y + x)- _θ(y - x) ) ], (7) where ℓx,y+,y−:ℝ→ℝ+ _x,y +,y -:R ^+, ℓx,y+,y−∈1 _x,y +,y - ^1. Despite their widespread empirical adoption, Direct Alignment Algorithms conforming to Eq. 7 suffer from likelihood displacement of preferred responses, a structural artifact of their gradient formulations. Because the parameter update depends on the relative gradient difference ∇θlogπθ(y+∣x)−∇θlogπθ(y−∣x) _θ _θ(y + x)- _θ _θ(y - x), optimization can inadvertently shift probability mass away from target preferred completions [36]. In practice, this manifests as a systematic decline in the likelihood of preferred samples, frequently leading to downstream output degradation [46, 47]. 3 Proposed Method The connection of Direct Alignment Algorithms to RLHF, while theoretically sound, imposes a rigid structure with several practical challenges. First, convergence is rarely achieved in practical settings, where auxiliary techniques are used to prevent training instabilities and degeneracies. Consequently, the explicit functional form of the gradient matters more than the stationary solution. However, commonly used loss functions yield suboptimal or unintuitive gradient behaviors. In DPO, for example, gradients for low-probability response pairs receive the exact same weight as high-probability ones. Second, DAAs replace standard RLHF regularization with an inherently flawed surrogate. Because this surrogate yields different minimizers than the true KL divergence [44], it directly enables artifacts such as strong likelihood displacement. Third, some theoretical guarantees [34] assume the behavior policy is strictly equal to the reference policy, which does not hold in real-world training. Relying on preemptive SFT over positive preference data as a proxy is an unconvincing mitigation that fails to restore these theoretical properties. To circumvent these challenges, we bypass the variational derivation entirely and design the loss gradients directly. As a baseline for preference optimization, we extract the common gradient formulation shared across DAAs from Eq. 7. We then introduce a custom weighting scheme designed to prioritize response pairs where the model performs poorly (Proposition 3.1). Proposition 3.1 (Preference Gradient Term). For the dataset PO=(x,y+,y−)D_PO=(x,y +,y -) and the reference model πref _ref the proposed preference gradient term is ∇θℒP(θ)=−σ(logπθ(y−∣x)πθ(y+∣x)−τ)(∇θlogπθ(y+∣x)−∇θlogπθ(y−∣x)), _θL_P(θ)=-σ\! ( _θ(y - x) _θ(y + x)-τ ) ( _θ _θ(y + x)- _θ _θ(y - x) ), (8) where τ>0τ>0 controls the preference margin. Next, we introduce a regularization term to maintain proximity to the reference model. Standard approaches penalize divergence using a KL divergence estimator, typically the k2k_2 estimator. However, unlike the bounded preference gradient, the k2k_2 estimator’s gradient coefficient scales linearly with logπθ _θ, allowing the regularization term to easily dominate training dynamics. Given that this objective serves as an empirical surrogate rather than an exact KL gradient estimator in practice, we rescale it to ensure stability (Proposition 3.2). Proposition 3.2 (Regularization Gradient Term). For the dataset PO=(x,y+,y−)D_PO=(x,y +,y -) and the reference model πref _ref the regularization gradient term is ∇θℒR(θ)=∑y∈y+,y−(σ(logπθ(y∣x)πref(y∣x))−12)∇θlogπθ(y∣x). _θL_R(θ)= _y∈\y +,y -\ (σ ( _θ(y x) _ref(y x) )- 12 ) _θ _θ(y x). (9) Conveniently, both gradient terms correspond to simple, closed-form loss functions, as follows from Corollary 3.3 (proof is given in Appendix A.1). Corollary 3.3. Both ∇θℒP _θL_P and ∇θℒR _θL_R admit loss forms: ℒP(θ)=−logσ(logπθ(y+∣x)−logπθ(y−∣x)+τ),L_P(θ)=- σ\! ( _θ(y + x)- _θ(y - x)+τ ), (10) ℒR(θ)=∑y∈y+,y−log(cosh(log12πθ(y∣x)πref(y∣x))).L_R(θ)= _y∈\y +,y -\ ( ( 12 _θ(y x) _ref(y x) ) ). (11) Taken together we can now define our proposed training objective. Definition 3.4 (Suan objective). For the dataset PO=(x,y+,y−)D_PO=(x,y +,y -) and the reference model πref _ref we define Suan training objective ℒSuan(θ)L_Suan(θ) as a linear combination of ℒP(θ)L_P(θ) and ℒR(θ)L_R(θ): ℒSuan(θ)=ℒP(θ)+βℒR(θ),L_Suan(θ)=L_P(θ)+ _R(θ), (12) where parameter β controls regularization strength. As a result, the design of Suan is highly practical, suppressing over-refusal by reducing over-optimization on already-correct pairs while preserving output quality by regularizing chosen likelihoods toward the reference model. 4 Experimental Evaluation 4.1 Models For a robust and comprehensive evaluation, our experimental design incorporates a diverse cross-section of open-source model families and parameter sizes. Specifically, we selected widely used families of open source large language models such as Mistral-12B [48], Falcon3-7B [49], Llama-3.1-8B [50], Gemma-2-9B [51], Qwen-3 [52], Yi-1.5-9B [53], DeepSeek-7B [54], OLMo-3-7B [55]. To ensure a fair comparison and eliminate confounding effects from prior post-training, we included the pre-trained (Base) versions of all models in the evaluation. To establish instruction-following capabilities, we first perform SFT on the Alpaca instruction dataset [56], a collection of diverse instructions paired with demonstrations. For subsequent preference optimization we opted for a high-quality preference dataset PKU-Safe-RLHF [57], using the SFT models as the references. Additional model and training details are available in the Appendix B. 4.2 Benchmarks To execute a comprehensive suite of the experiments, we evaluate our model across different tasks including harmlessness as safety, compliance as over-refusal, and helpfulness as the quality of the model outputs through instruction-following, reasoning and creative writing benchmarks. To evaluate the model’s robustness to adversarial and harmful instructions we used four commonly used red-teaming benchmarks: Malicious Instruct [7], HarmBench [11], AdvBench [28], and SORRY-Bench [58]. Here we report Attack Success Rate (ASR), defined as the percentage of malicious prompts that yield unsafe outputs. Additionally, to assess over-refusal, we evaluated the models on standard XS-Test [59] and OR-Bench [60], datasets containing a collection of benign instructions with the vocabulary, which may qualify as malicious. On these compliance benchmarks we report the Over-refusal Rate, which measures the percentage of benign instructions incorrectly rejected by the model. To confirm that the generated responses are actually helpful, we focused on diverse set of standard instruction-following benchmarks such as AlpacaEval [56], a collection from AlpacaFarm [61], MT-Bench [62], and ArenaHard [63]. Here we employ an LLM-as-a-judge setup [64] to score outputs on helpfulness, usefulness, and relevance. For the additional details on the used benchmarks see Appendix B. Figure 2: Comparison of safety alignment methods across diverse set of benchmarks. Our method Suan (highlighted by ⋆ ) simultaneously achieves harmlessness, compliance and helpfulness. (A). The Attack Success Rate (0-100) on safety benchmarks. Lower scores are better. (B). The Over-Refusal Rate (0-100) on compliance benchmarks. Lower scores are better. The catastrophic over-refusal is highlighted by the exclamation mark. (C). The Quality scores (1-5) on utility benchmarks judged by LLM as a Judge. Higher scores are better. 5 Results We compare our method, Suan, against DPO, IPO, and SafeDPO. All baselines were trained using the default hyperparameters reported in their respective publications, those for Suan we selected through ablation studies. Safety Alignment We first report aggregated evaluations across all base models and evaluation suits in Figure 2. As anticipated, Supervised Fine-Tuning substantially enhances model utility, while preference alignment yields marked gains in red-teaming resilience. Among the preference-tuned variants, IPO demonstrates robust resistance to over-refusal and maintains competitive helpfulness, yet it offers only marginal safety gains. Conversely, while both DPO and SafeDPO achieve top-tier harmlessness scores, these gains incur clear trade-offs: elevated over-refusal rates and compromised output quality (Figure 2, panels B and C). Across all evaluated settings, Suan consistently achieves the optimal alignment trade-off. It significantly outperforms SFT and IPO in risk mitigation, matching DPO’s strong harmlessness without sacrificing helpfulness or compliance. Likelihood Displacement In addition, we experimentally investigate whether Suan suffers from likelihood displacement (Figure 3). While DPO and SafeDPO both exhibit a pronounced decline in the likelihood of preferred completions, our method demonstrates the effectiveness of its regularization mechanism. By preventing unbounded policy drift away from the reference model, Suan avoids common safety alignment pitfalls such as over-refusal and utility degradation. Figure 3: The evolution of preferred responses likelihood displacement from the reference model during fine-tuning of Gemma-2 with DPO, SafeDPO and Suan. Ablation study To study the individual effects of Suan’s τ and β hyperparameters, we tuned them sequentially, first establishing τ=1τ=1 as optimal based on helpfulness, harmlessness, and compliance (Appendix C.1). Next, keeping τ=1τ=1 fixed, we systematically varied β. We evaluate performance across these β values alongside DPO and SafeDPO under identical β sweeps (Figure 4). One can notice that at the lowest setting (β=0.01β=0.01), DPO suffers a severe quality collapse. As a result, the Attack Success Rate becomes uninformative regarding model safety. This non-monotonic behavior aligns with [65], who showed that DPO performance degrades at both extreme low (β≈0.01β≈ 0.01) and high (β≥1β≥ 1) values, peaking around β=0.1β=0.1. Moreover, varying β in DPO and SafeDPO does not predictably constrain drift from the reference model. This corroborates theoretical findings by [33], who demonstrated that DPO’s implicit regularization dynamics diverge from standard KL divergence. In contrast, the hyperparameter β in Suan acts in a straightforward and highly interpretable manner. As shown across all panels in Figure 4, increasing β monotonically enforces closer alignment with the reference model πref _ref. Consequently, Suan maintains robust performance across all tested values of β, consistently outperforming DPO and SafeDPO in both compliance and helpfulness. Figure 4: Comparison of DPO, SafeDPO and Suan across different β. DPO safety and quality degeneration at β=0.01β=0.01 is marked with exclamation mark. The performance of the reference model is demonstrated by the dotted red line. 6 Conclusions We present Suan, a simple yet effective preference alignment algorithm designed to foster harmless, helpful, and compliant language model behavior. We demonstrate that standard Direct Alignment Algorithms frequently succumb to over-refusal and utility degradation. Motivated by a detailed analysis of gradient dynamics, we derive a novel, interpretable training objective that addresses these limitations. Beyond its theoretical grounding, Suan offers several key practical advantages: it is computationally lightweight, straightforward to implement, and operates directly on standard preference data without requiring any filtering. Extensive evaluations across eight diverse language models confirm that Suan achieves state-of-the-art performance across a broad spectrum of safety and capability benchmarks. Nonetheless, our work has several limitations that suggest promising directions for future research. While we focused on comparing against DPO and SafeDPO, future work could evaluate Suan against other preference optimization based methods and more prominent RL frameworks. Another interesting implication is to test the robustness of Suan towards various adversarial jailbreak attacks. Additionally, applying our method to Large Reasoning Models (LRMs) represents an interesting direction, as their significantly longer context outputs make them more susceptible to such jailbreak attacks. We hope that the community builds upon these findings to further advance safe preference alignment. Reproducibility Statement The code and datasets will become available upon publication. Impact Statement This work aims to advance the field of Machine Learning by simultaneously addressing red-teaming and helpfulness in post-training of Large Language Models. By carefully composing the interpretable gradients and forging a novel objective, our research supports the development of models that produce less harmful and more helpful outputs. There are many potential societal consequences of improving generative variety, none of which we feel must be specifically highlighted here beyond the ethical considerations standard to the advancement of language modeling. Acknowledgments and Disclosure of Funding The authors acknowledge the National Academic Infrastructure for Supercomputing in Sweden (NAISS) for granting this project access to high-performance clusters. The computations and data handling were enabled by Arrhenius, provided by Linköping University, and Berzelius, provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre by the National Academic Infrastructure for Supercomputing in Sweden (NAISS). During this project, O.C. was supported by the SciLifeLab & Wallenberg Data Driven Life Science Program (a DDLS Academic PhD grant to Eric Libby and Laura Michelle Carroll). References [1] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba (2021) Evaluating Large Language Models Trained on Code. arXiv. External Links: Link, Document Cited by: §1. [2] P. Jansen, O. Tafjord, M. Radensky, P. Siangliulue, T. Hope, B. D. Mishra, B. P. Majumder, D. S. Weld, and P. Clark (2025) CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation. arXiv. External Links: Link, Document Cited by: §1. [3] T. Zheng, G. Zhang, T. Shen, X. Liu, B. Y. Lin, J. Fu, W. Chen, and X. Yue (2024) OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement. arXiv. External Links: Link, Document Cited by: §1. [4] J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim (2024) A Survey on Large Language Models for Code Generation. arXiv. External Links: Link, Document Cited by: §1. [5] M. Karabacak and K. Margetis (2023) Embracing Large Language Models for Medical Applications: Opportunities and Challenges. Cureus 15 (5), p. e39305 (eng). External Links: ISSN 2168-8184, Document Cited by: §1. [6] F. Busch, L. Hoffmann, C. Rueger, E. H. van Dijk, R. Kader, E. Ortiz-Prado, M. R. Makowski, L. Saba, M. Hadamitzky, J. N. Kather, D. Truhn, R. Cuocolo, L. C. Adams, and K. K. Bressem (2025) Current applications and challenges in large language models for patient care: a systematic review. Communications Medicine 5 (1), p. 26 (eng). External Links: ISSN 2730-664X, Document Cited by: §1. [7] Y. Huang, S. Gupta, M. Xia, K. Li, and D. Chen (2023) Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation. arXiv. External Links: Link, Document Cited by: §B.5, §1, §1, §4.2. [8] J. Hazell (2023) Spear Phishing With Large Language Models. arXiv. External Links: Link, Document Cited by: §1. [9] N. Lukas, A. Salem, R. Sim, S. Tople, L. Wutschitz, and S. Zanella-Béguelin (2023) Analyzing Leakage of Personally Identifiable Information in Language Models. arXiv. External Links: Link, Document Cited by: §1. [10] A. Wei, N. Haghtalab, and J. Steinhardt (2023) Jailbroken: How Does LLM Safety Training Fail?. arXiv. External Links: Link, Document Cited by: §1. [11] M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li, D. Forsyth, and D. Hendrycks (2024) HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. arXiv. External Links: Link, Document Cited by: §B.5, §1, §4.2. [12] P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong (2023) Jailbreaking Black Box Large Language Models in Twenty Queries. arXiv. External Links: Link, Document Cited by: §1. [13] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe (2022) Training language models to follow instructions with human feedback. arXiv. External Links: Link, Document Cited by: §1. [14] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, B. Mann, and J. Kaplan (2022) Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv. External Links: Link, Document Cited by: §B.3, §C.3, Table C.7, §1. [15] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu (2019) Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv. External Links: Link, Document Cited by: §1. [16] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving (2020) Fine-Tuning Language Models from Human Preferences. arXiv. External Links: Link, Document Cited by: §1. [17] N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. Christiano (2020) Learning to summarize from human feedback. arXiv. External Links: Link, Document Cited by: §1. [18] P. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei (2023) Deep reinforcement learning from human preferences. External Links: Link Cited by: §1. [19] R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn (2023) Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv. Note: arXiv:2305.18290 [cs.LG] version: 1 External Links: Link, Document Cited by: §1, §1, §2.2, Definition 2.1. [20] J. Dai, X. Pan, R. Sun, J. Ji, X. Xu, M. Liu, Y. Wang, and Y. Yang (2023) Safe RLHF: Safe Reinforcement Learning from Human Feedback. arXiv. External Links: Link, Document Cited by: §1, §2.1. [21] A. Wachi, T. Q. Tran, R. Sato, T. Tanabe, and Y. Akimoto (2024) Stepwise Alignment for Constrained Language Model Policy Optimization. arXiv. External Links: Link, Document Cited by: §1. [22] M. Kim, J. M. Kwak, L. Alssum, B. Ghanem, P. Torr, D. Krueger, F. Barez, and A. Bibi (2025) Rethinking Safety in LLM Fine-tuning: An Optimization Perspective. arXiv. External Links: Link, Document Cited by: §1. [23] G. Kim, Y. J. Kim, B. Kim, H. Lee, K. Bae, Y. Jang, and M. Lee (2025) SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety. arXiv. External Links: Link, Document Cited by: §B.3, §1, §2.2, Definition 2.3. [24] A. Paulus, I. Kulikov, B. Amos, R. Munos, I. Evtimov, K. Chaudhuri, and A. Zharmagambetov (2025) Safety Alignment of LMs via Non-cooperative Games. arXiv. External Links: Link, Document Cited by: §1. [25] A. Mehrotra, M. Zampetakis, P. Kassianik, B. Nelson, H. Anderson, Y. Singer, and A. Karbasi (2023) Tree of Attacks: Jailbreaking Black-Box LLMs Automatically. arXiv. External Links: Link, Document Cited by: §1. [26] X. Liu, N. Xu, M. Chen, and C. Xiao (2023) AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models. arXiv. External Links: Link, Document Cited by: §1. [27] J. Hughes, S. Price, A. Lynch, R. Schaeffer, F. Barez, S. Koyejo, H. Sleight, E. Jones, E. Perez, and M. Sharma (2024) Best-of-N Jailbreaking. arXiv. External Links: Link, Document Cited by: §1. [28] A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson (2023) Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv. External Links: Link, Document Cited by: §B.5, §1, §4.2. [29] E. Jones, A. Dragan, A. Raghunathan, and J. Steinhardt (2023) Automatically Auditing Large Language Models via Discrete Optimization. arXiv. External Links: Link, Document Cited by: §1. [30] K. Hu, W. Yu, Y. Li, K. Chen, T. Yao, X. Li, W. Liu, L. Yu, Z. Shen, and M. Fredrikson (2025) Efficient LLM Jailbreak via Adaptive Dense-to-sparse Constrained Optimization. arXiv. Note: arXiv:2405.09113 [cs.LG] External Links: Link, Document Cited by: §1. [31] N. Xu, F. Wang, B. Zhou, B. Z. Li, C. Xiao, and M. Chen (2023) Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinking. arXiv. External Links: Link, Document Cited by: §1. [32] F. Jiang, Z. Xu, L. Niu, Z. Xiang, B. Ramasubramanian, B. Li, and R. Poovendran (2024) ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs. arXiv. External Links: Link, Document Cited by: §1. [33] M. G. Azar, M. Rowland, B. Piot, D. Guo, D. Calandriello, M. Valko, and R. Munos (2023) A General Theoretical Paradigm to Understand Learning from Human Preferences. arXiv. External Links: Link, Document Cited by: §1, §2.2, Definition 2.2, §5. [34] R. Rafailov, J. Hejna, R. Park, and C. Finn (2024) From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function. arXiv. External Links: Link, Document Cited by: §1, §3. [35] Y. Ren and D. J. Sutherland (2024) Learning Dynamics of LLM Finetuning. arXiv. External Links: Link, Document Cited by: §1. [36] N. Razin, S. Malladi, A. Bhaskar, D. Chen, S. Arora, and B. Hanin (2025) Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization. arXiv. External Links: Link, Document Cited by: §1, §2.2, Proposition 2.4. [37] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal Policy Optimization Algorithms. arXiv. External Links: Link, Document Cited by: §2.1. [38] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo (2024) DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv. External Links: Link, Document Cited by: §2.1. [39] J. Schulman (2020) Approximating KL Divergence. External Links: Link Cited by: §2.1. [40] Y. Tang and R. Munos (2025) On a few pitfalls in KL divergence gradient estimation for RL. arXiv. External Links: Link, Document Cited by: §2.1. [41] K. Liu, J. K. Liu, M. Chen, and Y. Liu (2025) Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimization. arXiv. External Links: Link, Document Cited by: §2.1. [42] Y. Chen, D. Zhu, Y. Sun, X. Chen, W. Zhang, and X. Shen (2024) The Accuracy Paradox in RLHF: When Better Reward Models Don’t Yield Better Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 2980–2989. External Links: Link, Document Cited by: §2.2. [43] R. A. Bradley and M. E. Terry (1952) Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika 39 (3/4), p. 324. External Links: ISSN 00063444, Link, Document Cited by: §2.2. [44] Y. Tang, Z. D. Guo, Z. Zheng, D. Calandriello, R. Munos, M. Rowland, P. H. Richemond, M. Valko, B. Á. Pires, and B. Piot (2024) Generalized Preference Optimization: A Unified Approach to Offline Alignment. arXiv. External Links: Link, Document Cited by: §2.2, §3. [45] Y. Meng, M. Xia, and D. Chen (2024) SimPO: Simple Preference Optimization with a Reference-Free Reward. arXiv. External Links: Link, Document Cited by: §2.2. [46] A. Pal, D. Karkhanis, S. Dooley, M. Roberts, S. Naidu, and C. White (2024) Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive. arXiv. External Links: Link, Document Cited by: §2.2. [47] R. Y. Pang, W. Yuan, K. Cho, H. He, S. Sukhbaatar, and J. Weston (2024) Iterative Reasoning Preference Optimization. arXiv. External Links: Link, Document Cited by: §2.2. [48] Mistral AI and NVIDIA (2024) Mistral NeMo. External Links: Link Cited by: §B.1, §4.1. [49] E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, É. Goffinet, D. Hesslow, J. Launay, Q. Malartic, D. Mazzotta, B. Noune, B. Pannier, and G. Penedo (2023) The Falcon Series of Open Language Models. arXiv. External Links: Link, Document Cited by: §B.1, §4.1. [50] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Marra, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. C. Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pintz, D. Livshits, D. Wyatt, D. Esiobu, D. Choudhary, D. Mahajan, D. Garcia-Olano, D. Perino, D. Hupkes, E. Lakomkin, E. AlBadawy, E. Lobanova, E. Dinan, E. M. Smith, F. Radenovic, F. Guzmán, F. Zhang, G. Synnaeve, G. Lee, G. L. Anderson, G. Thattai, G. Nail, G. Mialon, G. Pang, G. Cucurell, H. Nguyen, H. Korevaar, H. Xu, H. Touvron, I. Zarov, I. A. Ibarra, I. Kloumann, I. Misra, I. Evtimov, J. Zhang, J. Copet, J. Lee, J. Geffert, J. Vranes, J. Park, J. Mahadeokar, J. Shah, J. van der Linde, J. Billock, J. Hong, J. Lee, J. Fu, J. Chi, J. Huang, J. Liu, J. Wang, J. Yu, J. Bitton, J. Spisak, J. Park, J. Rocca, J. Johnstun, J. Saxe, J. Jia, K. V. Alwala, K. Prasad, K. Upasani, K. Plawiak, K. Li, K. Heafield, K. Stone, K. El-Arini, K. Iyer, K. Malik, K. Chiu, K. Bhalla, K. Lakhotia, L. Rantala-Yeary, L. van der Maaten, L. Chen, L. Tan, L. Jenkins, L. Martin, L. Madaan, L. Malo, L. Blecher, L. Landzaat, L. de Oliveira, M. Muzzi, M. Pasupuleti, M. Singh, M. Paluri, M. Kardas, M. Tsimpoukelli, M. Oldham, M. Rita, M. Pavlova, M. Kambadur, M. Lewis, M. Si, M. K. Singh, M. Hassan, N. Goyal, N. Torabi, N. Bashlykov, N. Bogoychev, N. Chatterji, N. Zhang, O. Duchenne, O. Çelebi, P. Alrassy, P. Zhang, P. Li, P. Vasic, P. Weng, P. Bhargava, P. Dubal, P. Krishnan, P. S. Koura, P. Xu, Q. He, Q. Dong, R. Srinivasan, R. Ganapathy, R. Calderer, R. S. Cabral, R. Stojnic, R. Raileanu, R. Maheswari, R. Girdhar, R. Patel, R. Sauvestre, R. Polidoro, R. Sumbaly, R. Taylor, R. Silva, R. Hou, R. Wang, S. Hosseini, S. Chennabasappa, S. Singh, S. Bell, S. S. Kim, S. Edunov, S. Nie, S. Narang, S. Raparthy, S. Shen, S. Wan, S. Bhosale, S. Zhang, S. Vandenhende, S. Batra, S. Whitman, S. Sootla, S. Collot, S. Gururangan, S. Borodinsky, T. Herman, T. Fowler, T. Sheasha, T. Georgiou, T. Scialom, T. Speckbacher, T. Mihaylov, T. Xiao, U. Karn, V. Goswami, V. Gupta, V. Ramanathan, V. Kerkez, V. Gonguet, V. Do, V. Vogeti, V. Albiero, V. Petrovic, W. Chu, W. Xiong, W. Fu, W. Meers, X. Martinet, X. Wang, X. Wang, X. E. Tan, X. Xia, X. Xie, X. Jia, X. Wang, Y. Goldschlag, Y. Gaur, Y. Babaei, Y. Wen, Y. Song, Y. Zhang, Y. Li, Y. Mao, Z. D. Coudert, Z. Yan, Z. Chen, Z. Papakipos, A. Singh, A. Srivastava, A. Jain, A. Kelsey, A. Shajnfeld, A. Gangidi, A. Victoria, A. Goldstand, A. Menon, A. Sharma, A. Boesenberg, A. Baevski, A. Feinstein, A. Kallet, A. Sangani, A. Teo, A. Yunus, A. Lupu, A. Alvarado, A. Caples, A. Gu, A. Ho, A. Poulton, A. Ryan, A. Ramchandani, A. Dong, A. Franco, A. Goyal, A. Saraf, A. Chowdhury, A. Gabriel, A. Bharambe, A. Eisenman, A. Yazdan, B. James, B. Maurer, B. Leonhardi, B. Huang, B. Loyd, B. De Paola, B. Paranjape, B. Liu, B. Wu, B. Ni, B. Hancock, B. Wasti, B. Spence, B. Stojkovic, B. Gamido, B. Montalvo, C. Parker, C. Burton, C. Mejia, C. Liu, C. Wang, C. Kim, C. Zhou, C. Hu, C. Chu, C. Cai, C. Tindal, C. Feichtenhofer, C. Gao, D. Civin, D. Beaty, D. Kreymer, D. Li, D. Adkins, D. Xu, D. Testuggine, D. David, D. Parikh, D. Liskovich, D. Foss, D. Wang, D. Le, D. Holland, E. Dowling, E. Jamil, E. Montgomery, E. Presani, E. Hahn, E. Wood, E. Le, E. Brinkman, E. Arcaute, E. Dunbar, E. Smothers, F. Sun, F. Kreuk, F. Tian, F. Kokkinos, F. Ozgenel, F. Caggioni, F. Kanayet, F. Seide, G. M. Florez, G. Schwarz, G. Badeer, G. Swee, G. Halpern, G. Herman, G. Sizov, Guangyi, Zhang, G. Lakshminarayanan, H. Inan, H. Shojanazeri, H. Zou, H. Wang, H. Zha, H. Habeeb, H. Rudolph, H. Suk, H. Aspegren, H. Goldman, H. Zhan, I. Damlaj, I. Molybog, I. Tufanov, I. Leontiadis, I. Veliche, I. Gat, J. Weissman, J. Geboski, J. Kohli, J. Lam, J. Asher, J. Gaya, J. Marcus, J. Tang, J. Chan, J. Zhen, J. Reizenstein, J. Teboul, J. Zhong, J. Jin, J. Yang, J. Cummings, J. Carvill, J. Shepard, J. McPhie, J. Torres, J. Ginsburg, J. Wang, K. Wu, K. H. U, K. Saxena, K. Khandelwal, K. Zand, K. Matosich, K. Veeraraghavan, K. Michelena, K. Li, K. Jagadeesh, K. Huang, K. Chawla, K. Huang, L. Chen, L. Garg, L. A, L. Silva, L. Bell, L. Zhang, L. Guo, L. Yu, L. Moshkovich, L. Wehrstedt, M. Khabsa, M. Avalani, M. Bhatt, M. Mankus, M. Hasson, M. Lennie, M. Reso, M. Groshev, M. Naumov, M. Lathi, M. Keneally, M. Liu, M. L. Seltzer, M. Valko, M. Restrepo, M. Patel, M. Vyatskov, M. Samvelyan, M. Clark, M. Macey, M. Wang, M. J. Hermoso, M. Metanat, M. Rastegari, M. Bansal, N. Santhanam, N. Parks, N. White, N. Bawa, N. Singhal, N. Egebo, N. Usunier, N. Mehta, N. P. Laptev, N. Dong, N. Cheng, O. Chernoguz, O. Hart, O. Salpekar, O. Kalinli, P. Kent, P. Parekh, P. Saab, P. Balaji, P. Rittner, P. Bontrager, P. Roux, P. Dollar, P. Zvyagina, P. Ratanchandani, P. Yuvraj, Q. Liang, R. Alao, R. Rodriguez, R. Ayub, R. Murthy, R. Nayani, R. Mitra, R. Parthasarathy, R. Li, R. Hogan, R. Battey, R. Wang, R. Howes, R. Rinott, S. Mehta, S. Siby, S. J. Bondu, S. Datta, S. Chugh, S. Hunt, S. Dhillon, S. Sidorov, S. Pan, S. Mahajan, S. Verma, S. Yamamoto, S. Ramaswamy, S. Lindsay, S. Feng, S. Lin, S. C. Zha, S. Patil, S. Shankar, S. Zhang, S. Wang, S. Agarwal, S. Sajuyigbe, S. Chintala, S. Max, S. Chen, S. Kehoe, S. Satterfield, S. Govindaprasad, S. Gupta, S. Deng, S. Cho, S. Virk, S. Subramanian, S. Choudhury, S. Goldman, T. Remez, T. Glaser, T. Best, T. Koehler, T. Robinson, T. Li, T. Zhang, T. Matthews, T. Chou, T. Shaked, V. Vontimitta, V. Ajayi, V. Montanez, V. Mohan, V. S. Kumar, V. Mangla, V. Ionescu, V. Poenaru, V. T. Mihailescu, V. Ivanov, W. Li, W. Wang, W. Jiang, W. Bouaziz, W. Constable, X. Tang, X. Wu, X. Wang, X. Wu, X. Gao, Y. Kleinman, Y. Chen, Y. Hu, Y. Jia, Y. Qi, Y. Li, Y. Zhang, Y. Zhang, Y. Adi, Y. Nam, Yu, Wang, Y. Zhao, Y. Hao, Y. Qian, Y. Li, Y. He, Z. Rait, Z. DeVito, Z. Rosnbrick, Z. Wen, Z. Yang, Z. Zhao, and Z. Ma (2024) The Llama 3 Herd of Models. arXiv. External Links: Link, Document Cited by: §B.1, §4.1. [51] Gemma Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, J. Ferret, P. Liu, P. Tafti, A. Friesen, M. Casbon, S. Ramos, R. Kumar, C. L. Lan, S. Jerome, A. Tsitsulin, N. Vieillard, P. Stanczyk, S. Girgin, N. Momchev, M. Hoffman, S. Thakoor, J. Grill, B. Neyshabur, O. Bachem, A. Walton, A. Severyn, A. Parrish, A. Ahmad, A. Hutchison, A. Abdagic, A. Carl, A. Shen, A. Brock, A. Coenen, A. Laforge, A. Paterson, B. Bastian, B. Piot, B. Wu, B. Royal, C. Chen, C. Kumar, C. Perry, C. Welty, C. A. Choquette-Choo, D. Sinopalnikov, D. Weinberger, D. Vijaykumar, D. Rogozińska, D. Herbison, E. Bandy, E. Wang, E. Noland, E. Moreira, E. Senter, E. Eltyshev, F. Visin, G. Rasskin, G. Wei, G. Cameron, G. Martins, H. Hashemi, H. Klimczak-Plucińska, H. Batra, H. Dhand, I. Nardini, J. Mein, J. Zhou, J. Svensson, J. Stanway, J. Chan, J. P. Zhou, J. Carrasqueira, J. Iljazi, J. Becker, J. Fernandez, J. van Amersfoort, J. Gordon, J. Lipschultz, J. Newlan, J. Ji, K. Mohamed, K. Badola, K. Black, K. Millican, K. McDonell, K. Nguyen, K. Sodhia, K. Greene, L. L. Sjoesund, L. Usui, L. Sifre, L. Heuermann, L. Lago, L. McNealus, L. B. Soares, L. Kilpatrick, L. Dixon, L. Martins, M. Reid, M. Singh, M. Iverson, M. Görner, M. Velloso, M. Wirth, M. Davidow, M. Miller, M. Rahtz, M. Watson, M. Risdal, M. Kazemi, M. Moynihan, M. Zhang, M. Kahng, M. Park, M. Rahman, M. Khatwani, N. Dao, N. Bardoliwalla, N. Devanathan, N. Dumai, N. Chauhan, O. Wahltinez, P. Botarda, P. Barnes, P. Barham, P. Michel, P. Jin, P. Georgiev, P. Culliton, P. Kuppala, R. Comanescu, R. Merhej, R. Jana, R. A. Rokni, R. Agarwal, R. Mullins, S. Saadat, S. M. Carthy, S. Cogan, S. Perrin, S. M. R. Arnold, S. Krause, S. Dai, S. Garg, S. Sheth, S. Ronstrom, S. Chan, T. Jordan, T. Yu, T. Eccles, T. Hennigan, T. Kocisky, T. Doshi, V. Jain, V. Yadav, V. Meshram, V. Dharmadhikari, W. Barkley, W. Wei, W. Ye, W. Han, W. Kwon, X. Xu, Z. Shen, Z. Gong, Z. Wei, V. Cotruta, P. Kirk, A. Rao, M. Giang, L. Peran, T. Warkentin, E. Collins, J. Barral, Z. Ghahramani, R. Hadsell, D. Sculley, J. Banks, A. Dragan, S. Petrov, O. Vinyals, J. Dean, D. Hassabis, K. Kavukcuoglu, C. Farabet, E. Buchatskaya, S. Borgeaud, N. Fiedel, A. Joulin, K. Kenealy, R. Dadashi, and A. Andreev (2024) Gemma 2: Improving Open Language Models at a Practical Size. arXiv. External Links: Link, Document Cited by: §B.1, §4.1. [52] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu (2025) Qwen3 Technical Report. arXiv. External Links: Link, Document Cited by: §B.1, §4.1. [53] 01. AI, :, A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, G. Wang, H. Li, J. Zhu, J. Chen, J. Chang, K. Yu, P. Liu, Q. Liu, S. Yue, S. Yang, S. Yang, W. Xie, W. Huang, X. Hu, X. Ren, X. Niu, P. Nie, Y. Li, Y. Xu, Y. Liu, Y. Wang, Y. Cai, Z. Gu, Z. Liu, and Z. Dai (2024) Yi: Open Foundation Models by 01.AI. arXiv. External Links: Link, Document Cited by: §B.1, §4.1. [54] DeepSeek-AI, :, X. Bi, D. Chen, G. Chen, S. Chen, D. Dai, C. Deng, H. Ding, K. Dong, Q. Du, Z. Fu, H. Gao, K. Gao, W. Gao, R. Ge, K. Guan, D. Guo, J. Guo, G. Hao, Z. Hao, Y. He, W. Hu, P. Huang, E. Li, G. Li, J. Li, Y. Li, Y. K. Li, W. Liang, F. Lin, A. X. Liu, B. Liu, W. Liu, X. Liu, X. Liu, Y. Liu, H. Lu, S. Lu, F. Luo, S. Ma, X. Nie, T. Pei, Y. Piao, J. Qiu, H. Qu, T. Ren, Z. Ren, C. Ruan, Z. Sha, Z. Shao, J. Song, X. Su, J. Sun, Y. Sun, M. Tang, B. Wang, P. Wang, S. Wang, Y. Wang, Y. Wang, T. Wu, Y. Wu, X. Xie, Z. Xie, Z. Xie, Y. Xiong, H. Xu, R. X. Xu, Y. Xu, D. Yang, Y. You, S. Yu, X. Yu, B. Zhang, H. Zhang, L. Zhang, L. Zhang, M. Zhang, M. Zhang, W. Zhang, Y. Zhang, C. Zhao, Y. Zhao, S. Zhou, S. Zhou, Q. Zhu, and Y. Zou (2024) DeepSeek LLM: Scaling Open-Source Language Models with Longtermism. arXiv. External Links: Link, Document Cited by: §B.1, §4.1. [55] T. Olmo, :, A. Ettinger, A. Bertsch, B. Kuehl, D. Graham, D. Heineman, D. Groeneveld, F. Brahman, F. Timbers, H. Ivison, J. Morrison, J. Poznanski, K. Lo, L. Soldaini, M. Jordan, M. Chen, M. Noukhovitch, N. Lambert, P. Walsh, P. Dasigi, R. Berry, S. Malik, S. Shah, S. Geng, S. Arora, S. Gupta, T. Anderson, T. Xiao, T. Murray, T. Romero, V. Graf, A. Asai, A. Bhagia, A. Wettig, A. Liu, A. Rangapur, C. Anastasiades, C. Huang, D. Schwenk, H. Trivedi, I. Magnusson, J. Lochner, J. Liu, L. J. V. Miranda, M. Sap, M. Morgan, M. Schmitz, M. Guerquin, M. Wilson, R. Huff, R. L. Bras, R. Xin, R. Shao, S. Skjonsberg, S. Z. Shen, S. S. Li, T. Wilde, V. Pyatkin, W. Merrill, Y. Chang, Y. Gu, Z. Zeng, A. Sabharwal, L. Zettlemoyer, P. W. Koh, A. Farhadi, N. A. Smith, and H. Hajishirzi (2025) Olmo 3. arXiv. External Links: Link, Document Cited by: §B.1, §4.1. [56] R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto (2023) Stanford Alpaca: An Instruction-following LLaMA model. GitHub. Note: Publication Title: GitHub repository External Links: Link Cited by: §B.2, §4.1, §4.2. [57] J. Ji, M. Liu, J. Dai, X. Pan, C. Zhang, C. Bian, R. Sun, Y. Wang, and Y. Yang (2023) BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset. arXiv. External Links: Link, Document Cited by: §B.3, §C.3, §4.1. [58] T. Xie, X. Qi, Y. Zeng, Y. Huang, U. M. Sehwag, K. Huang, L. He, B. Wei, D. Li, Y. Sheng, R. Jia, B. Li, K. Li, D. Chen, P. Henderson, and P. Mittal (2024) SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal. arXiv. External Links: Link, Document Cited by: §B.5, §4.2. [59] P. Röttger, H. R. Kirk, B. Vidgen, G. Attanasio, F. Bianchi, and D. Hovy (2023) XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models. arXiv. External Links: Link, Document Cited by: §B.5, §4.2. [60] J. Cui, W. Chiang, I. Stoica, and C. Hsieh (2024) OR-Bench: An Over-Refusal Benchmark for Large Language Models. arXiv. External Links: Link, Document Cited by: §B.5, §4.2. [61] Y. Dubois, X. Li, R. Taori, T. Zhang, I. Gulrajani, J. Ba, C. Guestrin, P. Liang, and T. B. Hashimoto (2023) AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback. Cited by: §B.5, §4.2. [62] G. Bai, J. Liu, X. Bu, Y. He, J. Liu, Z. Zhou, Z. Lin, W. Su, T. Ge, B. Zheng, and W. Ouyang (2024) MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 7421–7454. External Links: Link, Document Cited by: §B.5, §4.2. [63] T. Li, W. Chiang, E. Frick, L. Dunlap, T. Wu, B. Zhu, J. E. Gonzalez, and I. Stoica (2024) From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline. arXiv preprint arXiv:2406.11939. Cited by: §B.5, §4.2. [64] Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu (2023) G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. arXiv. External Links: Link, Document Cited by: §4.2. [65] Y. Liu, P. Liu, and A. Cohan (2024) Understanding Reference Policies in Direct Preference Optimization. arXiv. External Links: Link, Document Cited by: §5. [66] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer (2023) QLoRA: Efficient Finetuning of Quantized LLMs. arXiv. External Links: Link, Document Cited by: §B.4. [67] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2021) LoRA: Low-Rank Adaptation of Large Language Models. arXiv. External Links: Link, Document Cited by: §B.4. [68] W. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, and I. Stoica (2024) Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv. External Links: Link, Document Cited by: §B.5. [69] W. Zhao, X. Ren, J. Hessel, C. Cardie, Y. Choi, and Y. Deng (2024) WildChat: 1M ChatGPT Interaction Logs in the Wild. arXiv. External Links: Link, Document Cited by: §B.5. [70] Y. Zhang, H. Diddee, S. Holm, H. Liu, X. Liu, V. Samuel, B. Wang, and D. Ippolito (2025) NoveltyBench: Evaluating Language Models for Humanlike Diversity. arXiv. External Links: Link, Document Cited by: §B.5. [71] P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord (2018) Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv. External Links: Link, Document Cited by: §B.5. [72] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt (2020) Measuring Massive Multitask Language Understanding. arXiv. External Links: Link, Document Cited by: §B.5. [73] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient Memory Management for Large Language Model Serving with PagedAttention. arXiv. External Links: Link, Document Cited by: §B.6. [74] A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi (2019) The Curious Case of Neural Text Degeneration. arXiv. External Links: Link, Document Cited by: §B.6. [75] F. AI (2024) Flow-judge-v0.1: An open small language model for llm system evaluations.. External Links: Link Cited by: §B.7. [76] A. Alexandru, A. Calvi, H. Broomfield, J. Golden, K. Dai, M. Leys, M. Burger, M. Bartolo, R. Engeler, S. Pisupati, T. Drane, and Y. S. Park (2025) Atla Selene Mini: A General Purpose Evaluation Model. arXiv. External Links: Link, Document Cited by: §B.7. Appendix Appendix A Omitted Proofs Corollary A.1 (Corollary 3.3). Both ∇θℒP _θL_P and ∇θℒR _θL_R admit loss forms: ℒP(θ)=−logσ(logπθ(y+∣x)−logπθ(y−∣x)+τ)L_P(θ)=- σ\! ( _θ(y + x)- _θ(y - x)+τ ) (A.1) ℒR(θ)=∑y∈y+,y−log(cosh(log12πθ(y∣x)πref(y∣x))) _R(θ)= _y∈\y +,y -\ ( ( 12 _θ(y x) _ref(y x) ) ) (A.2) Proof. Recall two helpful properties of the sigmoid function dt[−logσ(t)]=−(1−σ(t))=−σ(−t),tanh(t)=2σ(2t)−1. ddt [- σ(t) ]=-(1-σ(t))=-σ(-t), (t)=2σ(2t)-1. (A.3) The second identity follows directly from the definition of sigmoid 2σ(2t)−1=21+e−2t−1=1−e−2t1+e−2t=tanh(t).2σ(2t)-1= 21+e^-2t-1= 1-e^-2t1+e^-2t= (t). (A.4) Recall the gradients of Suan ∇θℒP(θ)=−σ(logπθ(y−∣x)πθ(y+∣x)−τ)(∇θlogπθ(y+∣x)−∇θlogπθ(y−∣x)), _θL_P(θ)=-σ\! ( _θ(y - x) _θ(y + x)-τ ) ( _θ _θ(y + x)- _θ _θ(y - x) ), (A.5) ∇θℒR(θ)=∑y∈y+,y−(σ(logπθ(y∣x)πref(y∣x))−12)∇θlogπθ(y∣x). _θL_R(θ)= _y∈\y +,y -\ (σ ( _θ(y x) _ref(y x) )- 12 ) _θ _θ(y x). (A.6) Let t(θ)=logπθ(y+∣x)−logπθ(y−∣x)+τt(θ)= _θ(y + x)- _θ(y - x)+τ, so that ∇θℒP(θ)=−σ(−t(θ))∇θt(θ) _θL_P(θ)=-σ(-t(θ))\, _θt(θ) coincides with the Eq. A.5. Then by the first property outlined in Eq. A.3 the antiderivative is ℒP(θ)=−logt(θ)L_P(θ)=- t(θ), which immediately gives ℒP(θ)=−logσ(logπθ(y+∣x)−logπθ(y−∣x)+τ)L_P(θ)=- σ\! ( _θ(y + x)- _θ(y - x)+τ ) (A.7) For y∈y+,y−y∈\y +,y -\ denote λ(y)=logπθ(y∣x)πref(y∣x),such that∇θλ(y)=∇θlogπθ(y∣x)πref(y∣x)=∇θlogπθ(y∣x).λ(y)= _θ(y x) _ref(y x), that _θλ(y)= _θ _θ(y x) _ref(y x)= _θ _θ(y x). (A.8) Now by substituting λ(y)λ(y) into Eq. A.6 and using the second property from Eq. A.3 we get ∇θℒR(θ)=∑y∈y+,y−(σ(λ(y))−12)∇θλ(y)=12∑y∈y+,y−tanhλ(y)2∇θλ(y) _θL_R(θ)= _y∈\y +,y -\ (σ (λ(y) )- 12 ) _θλ(y)= 12 _y∈\y +,y -\ λ(y)2 _θλ(y) (A.9) Consider the function g(y)=log(2coshλ2)g(y)= (2 λ2) with the gradient calculated via chain rule ∇θg(y)=12tanhλ(y)2∇θλ(y) _θg(y)= 12 λ(y)2 _θλ(y). The function g(y)g(y) is the antiderivative for Eq. A.6 and, therefore, the loss has the following closed-form ℒR(θ)=∑y∈y+,y−log(cosh(log12πθ(y∣x)πref(y∣x))),L_R(θ)= _y∈\y +,y -\ ( ( 12 _θ(y x) _ref(y x) ) ), (A.10) where we disregarded the constant term log2 2, which vanishes during the gradient computation. This proves both loss forms. ∎ Appendix B Experimental Details In this section we provide comprehensive descriptions for the models, benchmarks and datasets used in our study. All the experiments were performed on a single NVIDIA GH200 96 GB GPU. B.1 Models Mistral-12B Mistral NeMo [48] is trained jointly by Mistral AI and NVIDIA. It is designed for diverse tasks including text generation and instruction following. This model is released under Apache 2.0 license. Falcon-3-7B Falcon-3-7B [49] from the Technology Innovation Institute, trained on a diverse high-quality corpora predominantly assembled from web data. This model is released under Falcon LLM license. Llama-3.1-8B Llama-3.1 [50] is released by Meta AI as an extension of the Llama-3 series. The model serves as a strong foundation for downstream fine-tuning and alignment methods, making it widely adopted in both research and applied settings. This model contains custom Llama-3.1 license 11 1 https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/LICENSE. Gemma-2-9B Gemma-2 [51] is a model family developed by Google, focusing on efficiency and strong reasoning capabilities under limited parameter budgets. It is trained using a carefully curated dataset that emphasizes high-quality, synthetic, and textbook-style data. This model is released under Google’s custom Gemma license. Qwen-3-8B Qwen3 [52] is a family of large language models developed by Alibaba Cloud, designed to support general-purpose language understanding, reasoning, and instruction-following tasks. The model is trained on a diverse mixture of web, code, and domain-specific data. This model is released under Apache 2.0 license. Yi-1.5-9B Yi-1.5-9B [53] is a bilingual language model developed by 01.AI, trained on 3 trillion tokens multilingual corpus. This model is released Apache 2.0 license. DeepSeek-7B DeepSeek-7B [54] is a series of models developed by DeepSeek, trained from scratch on a vast dataset of 2 trillion tokens in both English and Chinese. This model is released under MIT license. OLMo-3-7B OLMo-3-7B [55] a family of state-of-the-art, fully-open language and thinking models developed by Allen Institute for AI. It is trained on Dolma 3 dataset consisting of more than 6 trillion tokens. This model is licensed under Apache 2.0 B.2 SFT Dataset Alpaca22 2 https://huggingface.co/datasets/tatsu-lab/alpaca [56] is a widely used instruction-following dataset consisting of approximately 52K instruction–response pairs generated using a self-instruct framework. The dataset covers a broad range of tasks, including question answering, summarization, reasoning, and creative writing. To preprocess the Alpaca dataset, we filter and format each example into prompt–completion pairs. We use explicit delimiters for the instruction, input, and response to provide structural context for the sequence. The model is trained to generate the response following an opening delimiter and is explicitly required to produce a matching delimiter to signal completion. Alpaca is available under C-BY-NC-4.0 license. B.3 DPO Datasets H-RLHF H-RLHF [14] is a preference dataset consisting of approximately 160K chosen/rejected pairs that include discriminatory language and discussions of abuse, violence, self-harm, exploitation, and other potentially upsetting subject matter. To preprocess the H dataset, we filter 50K pairs and format them into suitable prompt-chosen/rejected couplings. H-RLHF is available under MIT license. PKU-SafeRLHF-30K PKU-SafeRLHF [57] is a safety preference dataset that contains approximately 27K samples. The dataset contains prompt, two different responses and boolean labels for each response telling if this response is safe. To make this dataset compatible with DPO formatting, we select safe and unsafe pairs separately based on boolean indicators and construct standard prompt–preferred/dispreferred pairs. Going further, to meet SafeDPO formatting style [23], we restructure each example into prompt–chosen/rejected pairs with two additional binary indicators, which identify if preferred/dispreferred response is safe. PKU-SafeRLHF is available under C-BY-NC-4.0 license. B.4 SFT and DPO details Given the limitation of our computational resources, we performed 4-bit NormalFloat quantization of selected models and utilized the Quantized Low Rank Adaptation [66] technique, which applies Low-Rank Adaptation [67]. This approach significantly reduces memory footprint and accelerates training without loss in performance. On top of that, we used gradient accumulation to increase the total batch size. For both SFT and DPO, our models were trained for a single epoch using a linear learning rate schedule with a peak of 2×10−42× 10^-4 and 50 warmup steps. We employed a batch size of 2 with 4 gradient accumulation steps and a weight decay of 0.01. For the LoRA adapter, we set r=16r=16 and α=16α=16. More details are better understand from the accompanying code repository. B.5 Evaluation datasets Alpaca Eval is an evaluation set, which tests the ability of models to follow general user instructions [61]. In our experiments we use the helpful_base subset of the AlpacaFarm [61] Hugging Face repository, comprising 129 prompts. We take only this portion of the original dataset to ensure that the evaluation remains focused on standard natural language. Alpaca Eval is available under C-BY-NC-4.0. Arena Hard is a benchmark [63] consisting 500 challenging prompts curated by BenchBuilder, and collated from both ChatbotArena [68] and WildChat-1M [69]. We used the official Arena Hard v0.1 set, covering diverse range of tasks from advanced math and code to semantic etymology verification. The benchmark released under Apache License 2.0. MT-Bench is a standard test set carefully designed to evaluate large language models on instruction adherence [62]. It consists of 80 challenging, open-ended two-turn questions across categories like coding, math, and reasoning. We extracted the instruction prompts from the two-turn questions and generated the corresponding responses. MT-Bench is available under Apache License 2.0. Malicious Instruct is a human crafted dataset [7], consisting of 100 prompts. To construct the datasets the authors selected ten categories and asked ChatGPT to provide 20 responses for each of the categories. They manually reviewed the generated responses and selected 100 responses such that they are aligned with the topic and diverse at the same time. This dataset is available under MIT license. Harm Bench is a standardized evaluation framework for automated red teaming [11], consisting of 450 malicious prompts in different categories: copyright, contextual, and standard. This dataset is available under MIT license. AdvBench is a set of 500 harmful behaviors formulated as instructions [28]. These behaviors range over the same themes as the harmful strings setting, but the adversary’s goal is instead to find a single attack string that will cause the model to generate any response that attempts to comply with the instruction, and to do so over as many harmful behaviors as possible. AdvBench is available under MIT license. XS-Test is a benchmark for identifying exaggerated safety behaviors in Large Language Models [59]. XS-Test comprises 250 safe prompts across ten prompt types that well-calibrated models should not refuse to comply with, and 200 unsafe prompts as contrasts that, for most LLM applications, should be refused. In our evaluation, we filtered 100 samples from 250 safe prompts and measured over-refusal score. This dataset is available under C-BY-NC-4.0 license. SORRY-Bench is a large diverse framework for red teaming [58]. This dataset contains 9.2K potentially unsafe instructions, intended to be used for LLM safety refusal evaluation. In our main experiments, we used base subsets comprising 540 prompts. This dataset requires SORRY-Bench Dataset License Agreement. OR-Bench is an over-refusal benchmark for LLMs [60], consisting of 81K prompts in ten different categories: harmful, unethical, illegal, privacy, deception, violence, self-harm, harassment and sexual. We selected a hard subset of 1,320 prompts from the standard subset. This dataset is available under C-BY-NC-4.0 license. NoveltyBench [70] is a benchmark designed to evaluate language models’ ability to generate multiple distinct and high-quality outputs for the same prompt, removing the traditional focus from a single best response. For the evaluation, we selected its NB-curated subset, which contains 100 manually curated prompts. We utilized the original NoveltyBench framework and source code, including the default parameters for their proprietary quality and diversity metrics, Utility-k and Distinct-k. The code is available under MIT license. ARC-Challenge (ARC) [71] is a benchmark dataset of multiple-choice science questions curated to evaluate advanced reasoning and scientific understanding. The questions are sourced from standardized science examinations for grades 3 through 9 and are intentionally selected to be challenging for both humans and AI systems. For the benchmarking, we used its test subset, comprising 1,172 questions. This dataset is available under C-BY-SA-4.0 license. Massive Multitask Language Understanding (MMLU) [72] is a benchmark designed to evaluate the knowledge and reasoning capabilities of language models across multiple subject areas, spanning STEM disciplines, humanities, and social sciences. The dataset includes questions of varying difficulty levels, ranging from elementary concepts to advanced professional knowledge. For the benchmarking, we used its test subset, comprising 14,042 questions. This dataset is available under MIT license. B.6 Inference details To ensure compute-efficient inference we use vLLM inference engine [73]. The model’s generation parameters were selected based on the specific requirements of each evaluation task. Following standard empirical practices in the field, we employed a stochastic sampling strategy for the creative writing & instruction following and safety benchmarks. Specifically, we used nucleus sampling [74] with a cumulative probability threshold of p=0.9p=0.9 and different unit temperatures, depending on the task. For helpfulness and harmlessness experiments we employed T=1.0T=1.0 for all the training objectives. In contrast, for the Arena Hard, MMLU and ARC, we used deterministic greedy decoding (T=0.0T=0.0) to ensure objective and reproducible outputs, generating a single completion per prompt. We allocated a limit of 512 tokens for all utilized benchmarks with the only exception of Arena Hard, where we set 4096 tokens as recommended in the official evaluation. For the inference of our models, we adhere to the standard instruction template, incorporating an additional prompt if necessary, depending on the benchmark. All prompt templates will be provided in the accompanying code repository. B.7 LLM Judge To assess the efficient and robust evaluation of the responses, we employ a large language model as a judge to approach and compare generated responses on Alpaca Eval, ArenaHard and MT-Bench. Specifically, we use Flow-Judge-v0.1 [75], a top Elo performing instruction-aligned model from Judge Arena [76]. We prompted the Judge with comprehensive instructions to provide a score from 1 to 5, where 1 corresponds to irrelevant or unhelpful to the user’s needs or queries, and 5 corresponds to a relevant and useful responses that perfectly cater to the user’s needs and inquiries one. To ensure a robust evaluation, we used greedy decoding and the recommended response template - see code repository for additional details. Appendix C Additional Results C.1 Ablations Table C.1 gathers the results of the ablation study conducted on the Suan’s τ parameter. To ensure comprehensive evaluation, we measured performance across three safety alignment task simultaneously on AlpacaEval, Malicious Instruct and XS-Test. Ultimately, we found that τ=1τ=1 and β=0.1β=0.1 demonstrate optimal performance. Table C.1: Ablation study on τ parameter in Suan across three task conducted on Llama-3.1-8B with fixed β=0.1β=0.1. Helpfulness (1-5) of the models is measured on AlpacaEval. Harmlessness is measured on Malicious Instruct, while Compliance is computed on XS-Test. The best performance highlighted in bold. Task τ=0.5τ=0.5 τ=1.0τ=1.0 τ=1.5τ=1.5 τ=2.0τ=2.0 τ=3.0τ=3.0 Helpfulness ↑ 4.51 4.60 4.56 4.54 4.52 Harmlessness ↓ 11.9 9.8 11.0 12.3 11.5 Compliance ↓ 9.1 7.0 8.1 8.3 8.7 C.2 Additional Details We present here the details for the results, summarized in Figure 2. In Table C.2 we store helpfulness scores calculated by the LLM Judge on all eight models across three quality benchmarks. Harmlessness results are gathered in Table C.3, showcasing that Suan remains competitive with DPO and SafeDPO. The latter methods, however, exhibit a catastrophic over-refusal, which is outlined in Table C.4. Overall, we observe that the performance trends remain robust over the models and the benchmarks. We conclude that the combined results position Suan as a prominent choice for safety alignment. To showcase superior compliance of our method, we compare Suan responses to other methods on an example prompt from OR-Bench. In Figure C.1 one could see that both DPO and SafeDPO fail to recognize a benign prompt: What are the steps to create a harmless virtual virus for educational purposes? In contrast, Suan does not over-refuse by providing comprehensive, helpful and harmless response. Figure C.1: Comparison of model responses to a benign OR-Bench prompt on Mistral-12B. While DPO and SafeDPO exhibit over-refusal, Suan provides a detailed and helpful response. C.3 Additional Experiments To demonstrate the benefits of Suan beyond low over-refusal and high safety, we evaluated it alongside competitive baselines on ARC and MMLU, two benchmarks assessing factual scientific knowledge (Figure C.2). Through this evaluation, we confirm that Suan generally preserves pre-trained knowledge. In contrast, both DPO and SafeDPO suffer from a visible performance decline on the MMLU benchmark (Figure C.2, panel B). Table C.5 presents detailed accuracy scores for factual knowledge on ARC and MMLU benchmarks. Ultimately, our results suggest that Suan is not prone to forgetting of the pre-existing knowledge with the only exception of OLMo-3 model, where all of post-training methods slightly underperform. Figure C.2: Model performance on factuality benchmarks across different methods: (A) ARC and (B) MMLU. Higher scores are better. Red line represents performance of reference model (SFT). Furthermore, we investigate whether the gains of Suan come at the cost of reduced output diversity, as generation broadness is critical for creative tasks. To evaluate this, we utilized NoveltyBench, a diversity-focused benchmark with automated evaluation. In Figure C.3 (details in Table C.6), we report two complementary metrics, Utility and Distinct, which together measure the meaningfulness and broadness of model outputs. Suan benefits from high output diversity while maintaining high Utility compared to DPO and SafeDPO. Figure C.3: Utility (1-5) and Distinct (1-10) metrics evaluated by the LLM-Judge on NoveltyBench. Higher scores are better. Besides using PKU-SafeRLHF [57] we experimented with H-RLHF dataset [14]. We report the quality performance of the models trained on H-RLHF in the Table C.7. Our results demonstrate a slight degradation in performance compared to PKU-SafeRLHF (Table C.2) across all methods and models. Based on this observation we conducted our main experiments by using PKU-SafeRLHF as the default safety dataset. Table C.2: Helpfulness (1-5) of the models evaluated on ArenaHard (AH), AlpacaEval (AE) and MTBench (MT) across different methods. Higher scores indicate better performance. The best performance highlighted in bold. Bench Method Mistral Falcon-3 Llama-3.1 Gemma-2 Qwen-3 Yi-1.5 DeepSeek-7B OLMo-3 AH Base 1.83 1.90 2.11 1.54 3.74 2.05 1.78 1.62 SFT 3.35 3.96 4.14 3.74 4.04 3.88 3.21 3.45 DPO 2.99 3.89 3.77 2.89 3.82 3.65 2.78 3.12 IPO 3.21 3.99 3.87 3.56 3.67 3.72 3.05 3.34 SafeDPO 1.84 2.45 2.50 1.55 1.67 2.12 1.92 2.01 Suan 3.59 4.18 4.21 3.88 4.01 4.05 3.42 3.67 AE Base 2.83 2.99 3.05 2.81 4.16 2.91 2.75 2.68 SFT 4.38 4.34 4.40 4.37 4.31 4.29 4.15 4.22 DPO 3.79 4.16 4.05 3.58 4.28 4.01 3.65 3.88 IPO 4.34 4.41 4.29 4.24 4.34 4.33 4.18 4.25 SafeDPO 2.80 4.01 3.13 2.13 4.03 3.45 2.56 3.12 Suan 4.37 4.51 4.46 4.33 4.35 4.42 4.22 4.30 MT Base 2.30 3.07 2.60 2.68 3.81 2.85 2.42 2.55 SFT 4.12 4.25 4.03 4.09 4.20 4.15 3.98 4.08 DPO 3.90 3.79 3.66 3.74 4.04 3.85 3.58 3.72 IPO 4.18 4.27 3.86 4.13 4.20 4.09 3.95 4.03 SafeDPO 2.28 4.15 2.79 1.71 4.14 3.22 2.34 2.98 Suan 4.14 4.48 4.13 4.17 4.42 4.31 4.05 4.19 Table C.3: Safety across different methods on Malicious Instruct (MI), HarmBench (HB), AdvBench (AB) and SorryBench (SB) datasets. Safety score is measured via Attack Success Rate (0-100), where lower scores are better. Entries with significant quality degradation (making safety metrics irrelevant) are shaded in grey. The best performance is highlighted in bold. Bench Method Mistral Falcon-3 Llama-3.1 Gemma-2 Qwen-3 Yi-1.5 DeepSeek-7B OLMo-3 MI Base 38.9 21.3 45.7 25.6 3.0 32.4 28.1 41.2 SFT 46.2 50.0 52.5 51.1 29.0 44.7 38.5 53.9 DPO 18.4 3.1 3.0 10.9 5.7 12.3 8.9 15.6 IPO 16.7 36.8 10.9 32.3 24.7 20.1 28.4 33.5 SafeDPO 1.2 1.5 1.5 2.2 8.0 2.8 3.1 5.4 Suan 13.7 12.5 9.9 18.0 13.9 14.2 11.8 16.3 HB Base 36.3 26.2 34.9 28.8 7.1 30.5 25.9 38.7 SFT 37.9 41.3 39.1 42.0 35.6 40.2 36.7 44.1 DPO 6.3 3.7 3.2 6.7 2.3 5.1 4.3 7.0 IPO 19.2 33.1 9.9 32.4 28.5 22.5 30.1 34.9 SafeDPO 3.4 6.4 4.3 3.3 7.6 4.1 5.0 6.6 Suan 11.1 11.5 8.2 14.6 10.6 12.3 10.9 13.8 AB Base 40.8 24.2 41.7 26.0 2.1 35.6 29.3 43.1 SFT 47.8 40.9 51.8 46.4 18.7 43.5 39.2 52.4 DPO 9.6 1.8 3.1 8.5 2.0 6.7 4.4 8.9 IPO 20.0 26.4 11.8 34.9 14.9 18.3 24.6 30.1 SafeDPO 0.6 1.1 1.4 1.4 3.4 1.0 1.9 2.5 Suan 11.8 8.6 8.5 15.3 11.9 10.4 9.7 13.2 SB Base 31.2 22.9 31.8 24.5 8.9 27.8 24.1 33.5 SFT 38.4 39.1 40.8 38.3 38.9 39.5 37.2 42.3 DPO 4.7 3.2 2.8 5.6 3.2 4.1 3.8 5.2 IPO 22.3 35.5 10.4 36.3 35.5 27.1 32.8 38.9 SafeDPO 2.2 4.3 4.2 3.2 7.6 3.1 3.9 5.8 Suan 9.9 7.7 9.1 15.5 10.4 8.8 8.3 11.2 Table C.4: Over-refusal performance on XSTest (XS) and OR-Bench (OR) across different methods. Over-refusal rate is measured (0-100), where lower scores are better. Bench Method Mistral Falcon-3 Llama-3.1 Gemma-2 Qwen-3 Yi-1.5 DeepSeek-7B OLMo-3 XS Base 5.1 10.6 3.5 4.4 9.0 6.2 8.1 12.3 SFT 2.7 3.6 3.0 4.0 6.7 3.3 4.2 5.8 DPO 37.4 22.4 20.2 13.6 23.2 28.5 31.2 25.8 IPO 9.8 6.2 4.2 5.9 4.7 7.8 8.9 9.2 SafeDPO 16.1 15.6 20.1 18.4 16.0 17.2 18.9 19.5 Suan 7.3 7.4 6.2 5.2 8.0 6.8 8.5 9.7 OR Base 4.5 7.0 3.2 3.3 15.7 5.8 6.4 9.1 SFT 1.8 1.5 1.9 2.0 1.7 1.6 2.1 2.4 DPO 45.4 29.1 29.9 11.1 34.6 33.8 38.2 31.5 IPO 5.6 2.9 2.5 3.0 4.4 3.8 4.9 5.2 SafeDPO 26.7 10.9 21.9 18.9 19.4 22.3 24.1 20.8 Suan 3.7 3.7 3.6 3.3 5.0 3.5 4.2 4.8 Table C.5: Model performance (accuracy, 0-100) on factuality tasks across different methods. Higher scores are better. Bench Method Mistral Falcon-3 Llama-3.1 Gemma-2 Qwen-3 Yi-1.5 DeepSeek-7B OLMo-3 ARC SFT 78.66 71.33 74.30 75.10 86.90 81.57 32.59 81.06 DPO 72.78 77.47 67.24 79.27 87.71 81.66 30.72 76.79 IPO 74.66 77.90 71.84 80.97 86.69 79.86 34.56 75.94 SafeDPO 61.52 74.49 56.74 38.05 85.32 65.10 33.62 75.26 Suan 66.81 78.07 70.99 80.29 87.03 79.10 33.28 76.54 MMLU SFT 58.41 62.34 57.64 64.29 69.87 63.88 31.24 64.02 DPO 57.21 60.02 52.30 58.11 70.17 61.91 19.74 57.24 IPO 57.86 62.32 57.65 63.67 69.74 63.20 32.77 59.07 SafeDPO 50.10 59.61 47.49 33.21 69.24 49.80 31.81 58.17 Suan 54.96 61.97 57.11 63.79 69.93 62.69 31.71 59.36 Table C.6: Diversity scores of Distinct (1-10) and Utility (1-5) across different alignment methods on NoveltyBench dataset. Metric Method Mistral Falcon-3 Llama-3.1 Gemma-2 Qwen-3 Yi-9B OLMo-3 Distinct DPO 6.34 6.60 8.72 8.96 6.99 8.65 8.26 IPO 5.77 7.21 4.14 7.56 7.27 7.55 7.09 SafeDPO 9.39 8.42 9.09 9.86 8.22 8.85 7.73 Suan 8.68 8.40 8.99 8.26 8.50 8.48 8.48 Utility DPO 0.97 1.88 1.94 1.58 2.09 1.83 2.42 IPO 1.74 4.87 1.40 4.94 4.92 4.74 4.79 SafeDPO 1.29 1.41 1.13 1.12 2.91 1.79 3.26 Suan 3.70 4.15 3.29 4.57 3.34 4.20 4.11 Table C.7: Helpfulness (1-5) of the models trained on H-RLHF dataset [14]. Higher scores indicate better performance for both benchmarks. The best performance highlighted in bold. Bench Method Mistral Falcon-3 Llama-3.1 Gemma-2 Qwen-3 Yi-1.5 DeepSeek-7B OLMo-3 AE Base 2.83 2.99 3.05 2.81 4.16 2.91 2.75 2.68 SFT 4.38 4.34 4.40 4.37 4.31 4.29 4.15 4.22 DPO 3.34 3.66 3.56 3.15 3.77 3.53 3.21 3.41 IPO 3.91 3.97 3.86 3.82 3.91 3.90 3.76 3.83 SafeDPO 2.35 3.37 2.63 1.79 3.39 2.90 2.15 2.62 Suan 4.02 4.15 4.10 3.98 4.00 4.07 3.88 3.96