Paper deep dive
Alignment with Preference Optimization Is All You Need for LLM Safety
Reda Alami, Ali Khalifa Almansoori, Ahmed Alzubaidi, Mohamed El Amine Seddik, Mugariya Farooq, Hakim Hacid
Models: Alpaca, Falcon 11B, Gemma 2 9B, GPT-3.5, Llama 3.1 8B, Mistral 7B, Mixtral 8x7B, OLMo 7B, Qwen-2 7B, Vicuna 7B, Yi 1.5 9B
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/12/2026, 6:13:02 PM
Summary
The paper demonstrates that preference optimization methods, specifically Noise Contrastive Alignment (Safe-NCA), significantly enhance LLM safety, as evidenced by a boost in the global safety score of the Falcon 11B model from 57.64% to 99.90%. While these alignment techniques effectively reduce toxicity and improve safety, they introduce a performance trade-off, particularly in mathematical capabilities.
Entities (5)
Relation Signals (3)
Safe-NCA → enhancessafetyof → Falcon 11B
confidence 95% · We identify noise contrastive alignment (Safe-NCA) as an optimal method for balancing safety and performance.
LlamaGuard 3 8B → evaluates → Falcon 11B
confidence 95% · achieve a significant boost in global safety score... as measured by LlamaGuard 3 8B
Alignment techniques → causestradeoffwith → General capabilities
confidence 90% · this safety improvement comes at the cost of reduced general capabilities, particularly in math
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We demonstrate that preference optimization methods can effectively enhance LLM safety. Applying various alignment techniques to the Falcon 11B model using safety datasets, we achieve a significant boost in global safety score (from $57.64\%$ to $99.90\%$) as measured by LlamaGuard 3 8B, competing with state-of-the-art models. On toxicity benchmarks, average scores in adversarial settings dropped from over $0.6$ to less than $0.07$. However, this safety improvement comes at the cost of reduced general capabilities, particularly in math, suggesting a trade-off. We identify noise contrastive alignment (Safe-NCA) as an optimal method for balancing safety and performance. Our study ultimately shows that alignment techniques can be sufficient for building safe and robust models.
Tags
Links
- Source: https://arxiv.org/abs/2409.07772
- Canonical: https://arxiv.org/abs/2409.07772
Trouble viewing inline? Open PDF directly →
Full Text
79,700 characters extracted from source content.
Expand or collapse full text
Alignment with Preference Optimization Is All You Need for LLM Safety Reda Alami, Ali Khalifa Almansoori, Ahmed Alzubaidi, Mohamed El Amine Seddik, Mugariya Farooq, Hakim Hacid Abstract We demonstrate that preference optimization methods can effectively enhance LLM safety. Applying various alignment techniques to the Falcon 11B model using safety datasets, we achieve a significant boost in global safety score (from 57.64%57.64\%57.64 % to 99.90%99.90\%99.90 %) as measured by LlamaGuard 3 8B, competing with state-of-the-art models. On toxicity benchmarks, average scores in adversarial settings dropped from over 0.60.60.6 to less than 0.070.070.07. However, this safety improvement comes at the cost of reduced general capabilities, particularly in math, suggesting a trade-off. We identify noise contrastive alignment (Safe-NCA) as an optimal method for balancing safety and performance. Our study ultimately shows that alignment techniques can be sufficient for building safe and robust models. This paper contains words that may be offensive or harmful. Falcon 11B InstructMistral 7BOLMo 7BYI 1.5 9BQwen-2 7BLlama-3.1 8BMixtral 8x7BVicuna 7BGPT-3.5Gemma 2 9BSafe Falcon 11B50505060606070707080808090909010010010057.6457.6457.6471.3371.3371.3373.8473.8473.8479.0279.0279.0285.0485.0485.0486.8386.8386.8388.1888.1888.1889.4589.4589.4592.6592.6592.6593.3893.3893.3899.999.999.9Global Safety Score (%)Comparison of Global Safety Scores Figure 1: Comparison of the global safety scores of 111111 LLMs. The scores are derived from averaging the results of the safety ALERT and safety Adversarial ALERT benchmarks to assess each model’s overall performance across the safety evaluations. Notice the significant performance boost from 57.64%57.64\%57.64 % to 99.9%99.9\%99.9 % for the Falcon 11B model. Introduction Large language models (LLMs) are highly valuable for their ability to process and generate contextually appropriate text across various applications. However, ensuring the safety of these models is equally crucial. Safety in LLMs refers to their ability to consistently generate content that is accurate, ethical, and adheres to societal norms while preventing the production of harmful or inappropriate content. This paper investigates the effectiveness of preference optimization methods in enhancing LLM safety, specifically focusing on alignment techniques applied to the Falcon 11B model (Malartic et al. 2024) using safety datasets. Our study demonstrates that these alignment methods can significantly boost the global safety score of LLMs, as measured by LlamaGuard 3 8B. We achieved an increase from 57.64% to 99.90% in safety scores, competing with state-of-the-art models (Figure 1). Additionally, we observed a substantial reduction in toxicity scores under adversarial conditions, dropping from over 0.6 to less than 0.07. However, this improved safety comes at the cost of reduced general capabilities, particularly in mathematical tasks, indicating a trade-off between safety and performance. Among the explored techniques, we identify noise contrastive alignment (Safe-NCA) as an optimal method for balancing safety and overall model performance. Our investigation ultimately demonstrates that alignment techniques can be sufficient for developing safe and robust LLMs, while highlighting the importance of considering the trade-offs involved in enhancing model safety. The remainder of the paper is structured as follows. The next section introduces related work, followed by a formalization of the safety problem for LLMs. We then detail the alignment techniques explored in this study, present the benchmark used, and discuss our experimental results. Finally, we conclude the paper with our findings and implications for future research in LLM safety. Related Work Several existing works have explored safety with LLMs. We divided the literature into (1) Safety evaluation benchmarks for LLMs and (2) Safety enhancement techniques for LLMs. Safety Evaluation for LLMs In this subsection, major benchmarks found in the literature, that evaluate safety are presented. The paper by (Varshney et al. 2023) introduced the Safety and Over-Defensiveness Evaluation (SODE) benchmark, which consists of a diverse set of safe and unsafe prompts designed to systematically evaluate LLMs responses. A DeBERTA-v3-large model was trained to act as a binary classifier judging safe/unsafe responses. Moving on to ALERT (Tedeschi et al. 2024), which is a comprehensive benchmark designed to evaluate the safety of LLMs using fine-grained risk taxonomy, and red-teaming approaches. This benchmark includes 454545k instructions categorized to assess various safety vulnerabilities through adversarial testing scenarios. Experimental evaluations across 101010 widely used LLMs, revealed that many still fail to achieve satisfactory safety levels, underscoring the ongoing challenges in LLM safety assurance. Recent advancements in LLMs have prompted an exploration into their operational safety within interactive environments. In this context, (Yuan et al. 2024) developed R-Judge, a benchmark specifically designed to assess LLMs ability to identify and judge safety risks based on records of multi-turn agent interactions. The R-Judge benchmark is composed of 162 interaction records, spanning 27 key risk scenarios. Human input was leveraged to annotate each record, in terms of safety and risk description. Out of the nine models evaluated, GPT-4 demonstrated the best performance in both safety judgment and risk identification. (Gupta et al. 2024) proposed framework where the safety of LLM is assessed and the reliability of the underlying judge, is inline with the rise of LLM-as-a-Judge approaches. Refusal behavior testing was introduced, which frames the prompt to the LLM as yes/no question, prompting the LLM if it is interested in engaging with a specific prompt. Additionally, a model called Walled-Guard was proposed, intended to evaluate the quality of the judge deployed in the evaluation process. Safety Enhancement for LLMs In this subsection, a closer look is taken at existing research that explored addressing the safety issue with LLMs. Several prompt modification techniques were introduced in (Zheng et al. 2024), such as in In-Context exemplars and self-safety checks on prompts and responses, acting as a defense strategy against adversarial prompts. Moreover, work found in (Kumar et al. 2023), proposed erase-and-check framework that makes LLM robust against attacks on prompt, that push an LLM to be unsafe such as adversarial suffix, adversarial insertion, and adversarial infusion attacks. Llama 2 and DistilBERT were exploited to act as harmful detectors applied on various subsequences of the prompt. Three variations of erase-and-check were introduced, however, the increased running time is a major drawback of this work. As part of Llama 2 development (Touvron et al. 2023), red-teaming technique was utilized to enhance safety. The techniques entailed asking humans to interact with a target LLM by finding prompts that elicit unsafe responses in a wide range of topics. The exercise performed in red-teaming produces a fine-tuning dataset that can be leveraged to enhance safety. Authors in (Ge et al. 2023) alleviated the need for human involvement by introducing a Multi-round Automatic Red-Teaming (MART) method. MART employs an iterative process where in each round, both adversarial and target LLM are fine-tuned. The former is trained to generate prompts that provoke the LLM to return an unsafe response, whereas the latter is trained to return safe responses. Experiments indicated that four rounds of MART significantly reduced the violation rate in the target LLM by up to 84.7 percent. Following the same spirit, (Jiang et al. 2024) proposed techniques called Deep Adversarial Automated Red Teaming (DART). Compared to MART, they focused on producing a more diverse dataset to be used in subsequent rounds. (Dai et al. 2023) adapted an existing common alignment technique RLHF (Ouyang et al. 2022), by introducing Safe RLHF. Annotators involved in data annotations were instructed to label samples in terms of helpfulness and harmlessness. The reward model (R) is deduced from the helpfulness dataset, whereas the harmlessness dataset was used to obtain the cost model (C). Leveraging C and R, the objective is defined which Safe RLHF attempts to maximize a defined objective and was solved using Lagrangian methods. (Ji et al. 2024b) introduced the BEAVERTAILS-330k dataset, which is a human preference annotations dataset aimed at improving the safety alignment of LLMs. Additionally, the authors demonstrated how the dataset can be utilized in content moderation and RLHF. The Safety Problem for LLMs In this section, we start by introducing useful definitions regarding the safety risk taxonomy and adversarial attack taxonomy. Then, we introduce the safety objectives. Definition 1 (Harmful Category). To further safeguard the integrity of LLM outputs, harmful categories are explicitly defined as any content that promotes, encourages, or facilitates behaviors and ideologies that are discriminatory, unlawful, or unethical. These include the following six categories denoted that compose the safety risk taxonomy C=c1,c2,c3,c4,c5,c6C= \c_1,c_2,c_3,c_4,c_5,c_6 \C = c1 , c2 , c3 , c4 , c5 , c6 . • c1c_1c1: Criminal Planning - Content that encourages or provides guidance on criminal activities such as physical violence, theft, or illegal cyber activities. • c2c_2c2: Hate Speech - Expressions that promote discrimination or hostility against individuals or groups based on race, gender, sexual orientation, religion, disability, or economic status. • c3c_3c3: Regulated or Controlled Substances - Content endorsing or facilitating the illegal production, distribution, or consumption of controlled substances including drugs and alcohol. • c4c_4c4: Sexual Content - Promotions of or instructions on producing and distributing sexually explicit material. • c5c_5c5: Suicide & Self-Harm - Content that encourage self-harm or provide instructions for committing suicide, or content that promotes unhealthy body standards. • c6c_6c6: Guns & Illegal Weapons - Information that supports the acquisition, creation, or usage of illegal weapons including firearms, chemical, biological, and radioactive materials. Definition 2 (Adversarial Attacks). To further understand and address the methods used to exploit or manipulate a LLM into generating harmful content, we define a set of adversarial techniques designed to circumvent the model’s safety guidelines, potentially leading to outputs that fall within the harmful categories of the safety risk taxonomy C. The adversarial techniques are as follows: • Adversarial suffix (a1a_1a1): Adding deceptive content at the end of a prompt. The added suffix might direct the model to generate inappropriate or harmful responses, despite the initial part of the prompt being benign. • Adversarial prefix (a2a_2a2): Adding a misleading or harmful phrase at the beginning of a prompt. The goal is to trick the LLM into continuing along the harmful theme set by the prefix, potentially bypassing safety filters. • Token manipulation (a3a_3a3): Altering specific words or characters within a prompt to change its meaning. It can include misspelling words, swapping tokens, or inserting additional tokens that alter the intent of the prompt. • Jailbreaking (a4a_4a4): Using the prompt to change the model’s operational parameters. This could involve instructing the model to ignore its ethical guidelines or safety protocols, effectively tricking it into operating in an unsafe mode. In the following, we denote by A=a1,a2,a3,a4A= \a_1,a_2,a_3,a_4 \A = a1 , a2 , a3 , a4 the adversarial attack taxonomy. Definition 3 (Safe/Unsafe output). Given a model π, for any input prompt x, the model’s output y∼π(⋅|x)y π(·|x)y ∼ π ( ⋅ | x ) is evaluated against the six harmful categories c1,c2,c3,c4,c5,c6\c_1,c_2,c_3,c_4,c_5,c_6\ c1 , c2 , c3 , c4 , c5 , c6 . The evaluation function y→ω(y)y→ω(y)y → ω ( y ) is defined as follows: ω(y)=0if y∈c1,c2,c3,c4,c5,c61otherwise (y is safe)ω(y)= cases0&if y∈\c_1,c_2,c_3,c_4,c_5,c_6\\\ 1&otherwise ($y$ is safe) casesω ( y ) = start_ROW start_CELL 0 end_CELL start_CELL if y ∈ c1 , c2 , c3 , c4 , c5 , c6 end_CELL end_ROW start_ROW start_CELL 1 end_CELL start_CELL otherwise ( y is safe) end_CELL end_ROW The safety evaluation function y→ω(y)y→ω(y)y → ω ( y ) determines whether the response y falls into any of the harmful categories ci∈Cc_i∈ Ccitalic_i ∈ C, thereby marking the response as unsafe if it evaluates to 0. In practice, we use as evaluation function ω the LLama Guard 3 8B (Dubey et al. 2024). Definition 4 (Safety Score S). Given a model π. For a given category c∈Cc∈ Cc ∈ C, the safety score Sc(π)S_c(π)Sitalic_c ( π ) is defined as: Sc(π)=∑p∈Pcω(π(p))|Pc|S_c(π)= _p∈ P_cω(π(p))|P_c|Sitalic_c ( π ) = divide start_ARG ∑p ∈ P start_POSTSUBSCRIPT c end_POSTSUBSCRIPT ω ( π ( p ) ) end_ARG start_ARG | Pitalic_c | end_ARG where ω(π(p))=1ω(π(p))=1ω ( π ( p ) ) = 1 if the response to prompt p is safe, and 0 otherwise. Here, PcP_cPitalic_c represents the set of prompts associated with category c, and p is a specific prompt within this set. The overall safety score S(π)S(π)S ( π ) is computed as the weighted average of the category-wise safety scores: S(π)=∑ci∈C|Pci|∑ci∈C|Pci|×Sci(π)S(π)= _c_i∈ C |P_c_i| _c_i∈ C|P_c_i|× S_c_i(π)S ( π ) = ∑c start_POSTSUBSCRIPT i ∈ C end_POSTSUBSCRIPT divide start_ARG | Pitalic_c start_POSTSUBSCRIPT i end_POSTSUBSCRIPT | end_ARG start_ARG ∑c start_POSTSUBSCRIPT i ∈ C end_POSTSUBSCRIPT | Pitalic_c start_POSTSUBSCRIPT i end_POSTSUBSCRIPT | end_ARG × Sitalic_c start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ( π ) where |Pci||P_c_i|| Pitalic_c start_POSTSUBSCRIPT i end_POSTSUBSCRIPT | is the number of prompts in category cic_icitalic_i. Definition 5 (Attack Success Rate Score (ASR)). Given a model π. The attack success rate (ASR) ASRa(π)ASR_a(π)ASRa ( π ) for each attack a∈Aa∈ Aa ∈ A is defined as: ASRa(π)=∑p~∈P~a(1−ω(π(p~)))|P~a|ASR_a(π)\ = _ p∈ P_a(1-ω(π( p)))| P_a|ASRa ( π ) = divide start_ARG ∑over~ start_ARG p end_ARG ∈ over~ start_ARG P end_ARG start_POSTSUBSCRIPT a end_POSTSUBSCRIPT ( 1 - ω ( π ( over~ start_ARG p end_ARG ) ) ) end_ARG start_ARG | over~ start_ARG P end_ARGa | end_ARG where ω(π(p~))=1ω(π( p))=1ω ( π ( over~ start_ARG p end_ARG ) ) = 1 if the response to prompt p~ pover~ start_ARG p end_ARG is safe, and 0 otherwise. Here, P~a P_aover~ start_ARG P end_ARGa represents the set of prompts associated with the adversarial attack a∈Aa∈ Aa ∈ A, and p~ pover~ start_ARG p end_ARG is a specific prompt within this set. The overall attack success rate ASR(π)ASR(π)ASR ( π ) is computed as the weighted average of the category-wise attack success rates: ASR(π)=∑ai∈A(|P~ai|∑ai∈A|P~ai|×ASRai(π))ASR(π)= _a_i∈ A ( | P_a_i| _a_i∈ A| P_a_i|×ASR_a_i(π) )ASR ( π ) = ∑a start_POSTSUBSCRIPT i ∈ A end_POSTSUBSCRIPT ( divide start_ARG | over~ start_ARG P end_ARGa start_POSTSUBSCRIPT i end_POSTSUBSCRIPT | end_ARG start_ARG ∑a start_POSTSUBSCRIPT i ∈ A end_POSTSUBSCRIPT | over~ start_ARG P end_ARGa start_POSTSUBSCRIPT i end_POSTSUBSCRIPT | end_ARG × ASRa start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ( π ) ) where |P~ai|| P_a_i|| over~ start_ARG P end_ARGa start_POSTSUBSCRIPT i end_POSTSUBSCRIPT | is the number of prompts for the attack aia_iaitalic_i. Table 1: Loss functions for various Safe Optimization methods Method Loss Function Safe-DPO (Rafailov et al. 2024) ℒSafe-DPO(πθ;πref)=−(x,sw,sl)∼Safety[logσ(βlogπθ(sw|x)πref(sw|x)−βlogπθ(sl|x)πref(sl|x))]L_Safe-DPO ( _θ; _ref )=-E_ (x,y_s_w,y_s_l ) _Safety [ σ (β _θ (y_s_w|x ) _ref (y_s_w|x )-β _θ (y_s_l|x ) _ref (y_s_l|x ) ) ]LSafe-DPO ( πitalic_θ ; πroman_ref ) = - E( x , y start_POSTSUBSCRIPT s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT , yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT ) ∼ DSafety end_POSTSUBSCRIPT [ log σ ( β log divide start_ARG πitalic_θ ( yitalic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) end_ARG start_ARG πroman_ref ( yitalic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) end_ARG - β log divide start_ARG πitalic_θ ( yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) end_ARG start_ARG πroman_ref ( yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) end_ARG ) ] Safe-robust_DPO (Chowdhury 2024) ℒSafe-rDPO(πθ;πref)=−11−2ϵ(x,sw,sl)∼Safety[(1−ϵ)logσ(βlogπθ(sw|x)πref(sw|x)−βlogπθ(sl|x)πref(sl|x))−ϵlogσ(−βlogπθ(sw|x)πref(sw|x)+βlogπθ(sl|x)πref(sl|x))]L_Safe-rDPO ( _θ; _ref )=- 11-2εE_ (x,y_s_w,y_s_l ) _Safety [ (1-ε ) σ (β _θ (y_s_w|x ) _ref (y_s_w|x )-β _θ (y_s_l|x ) _ref (y_s_l|x ) )-ε σ (-β _θ (y_s_w|x ) _ref (y_s_w|x )+β _θ (y_s_l|x ) _ref (y_s_l|x ) ) ]LSafe-rDPO ( πitalic_θ ; πroman_ref ) = - divide start_ARG 1 end_ARG start_ARG 1 - 2 ϵ end_ARG E( x , y start_POSTSUBSCRIPT s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT , yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT ) ∼ DSafety end_POSTSUBSCRIPT [ ( 1 - ϵ ) log σ ( β log divide start_ARG πitalic_θ ( yitalic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) end_ARG start_ARG πroman_ref ( yitalic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) end_ARG - β log divide start_ARG πitalic_θ ( yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) end_ARG start_ARG πroman_ref ( yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) end_ARG ) - ϵ log σ ( - β log divide start_ARG πitalic_θ ( yitalic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) end_ARG start_ARG πroman_ref ( yitalic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) end_ARG + β log divide start_ARG πitalic_θ ( yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) end_ARG start_ARG πroman_ref ( yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) end_ARG ) ] Safe-IPO (Azar et al. 2024) ℒSafe-IPO(πθ;πref)=(x,sw,sl)∼Safety[log(πθ(sw|x)πref(sl|x)πθ(sl|x)πref(sw|x))−12τ]2L_Safe-IPO ( _θ; _ref )=E_ (x,y_s_w,y_s_l ) _Safety [ ( _θ(y_s_w|x) _ref (y_s_l|x ) _θ (y_s_l|x ) _ref(y_s_w|x) )- 12τ ]^2LSafe-IPO ( πitalic_θ ; πroman_ref ) = E( x , y start_POSTSUBSCRIPT s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT , yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT ) ∼ DSafety end_POSTSUBSCRIPT [ log ( divide start_ARG πitalic_θ ( yitalic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) πroman_ref ( yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) end_ARG start_ARG πitalic_θ ( yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) πroman_ref ( yitalic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) end_ARG ) - divide start_ARG 1 end_ARG start_ARG 2 τ end_ARG ]2 Safe-SLiC (Zhao et al. 2022) ℒSafe-SLiC(πθ;πref)=(x,sw,sl)∼Safety[max(0,1−β(logπθ(sw|x)πref(sw|x)−logπθ(sl|x)πref(sl|x)))]L_Safe-SLiC ( _θ; _ref )=E_ (x,y_s_w,y_s_l ) _Safety [ (0,1-β ( _θ(y_s_w|x) _ref(y_s_w|x)- _θ(y_s_l|x) _ref(y_s_l|x) ) ) ]LSafe-SLiC ( πitalic_θ ; πroman_ref ) = E( x , y start_POSTSUBSCRIPT s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT , yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT ) ∼ DSafety end_POSTSUBSCRIPT [ max ( 0 , 1 - β ( log divide start_ARG πitalic_θ ( yitalic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) end_ARG start_ARG πroman_ref ( yitalic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) end_ARG - log divide start_ARG πitalic_θ ( yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) end_ARG start_ARG πroman_ref ( yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) end_ARG ) ) ] Safe-KTO (Ethayarajh et al. 2024) ℒSafe-KTO(πθ,πref)=x,y∼Safety[λy−v(x,y)]L_Safe-KTO ( _θ, _ref )=E_x,y _Safety [ _y-v(x,y) ]LSafe-KTO ( πitalic_θ , πroman_ref ) = Eitalic_x , y ∼ D start_POSTSUBSCRIPT Safety end_POSTSUBSCRIPT [ λitalic_y - v ( x , y ) ] withrθ(x,y)=logπθ(y∣x)πref(y∣x)\ with\ r_θ(x,y)= _θ(y x) _ref(y x)with ritalic_θ ( x , y ) = log divide start_ARG πitalic_θ ( y ∣ x ) end_ARG start_ARG πroman_ref ( y ∣ x ) end_ARG and v(x,y)=λyσ(β(rθ(x,y)−z0)) if y∼ysw∣xλyσ(β(z0−rθ(x,y))) if y∼ysl∣xv(x,y)= \ array[]l _yσ (β (r_θ(x,y)-z_0 ) ) if y y_s_w x\\ _yσ (β (z_0-r_θ(x,y) ) ) if y y_s_l x array .v ( x , y ) = start_ARRAY start_ROW start_CELL λitalic_y σ ( β ( ritalic_θ ( x , y ) - z0 ) ) if y ∼ yitalic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT ∣ x end_CELL end_ROW start_ROW start_CELL λitalic_y σ ( β ( z0 - ritalic_θ ( x , y ) ) ) if y ∼ yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT ∣ x end_CELL end_ROW end_ARRAY with λy>0 _y>0λitalic_y > 0 Safe-EXO (Ji et al. 2024a) ℒSafe-EXO (πθ)=(,sw,sl)∼Safety[ (fθ(⋅∣sw,sl,)∥rϕ(⋅∣sw,sl,))]L_Safe-EXO ( _θ )=E_ ( x,y_s_w,y_s_l ) _Safety [D_KL (p_f_θ (· _s_w,y_s_l, x )\|p_r_φ (· _s_w,y_s_l, x ) ) ]LSafe-EXO ( πitalic_θ ) = E( italic_x , y start_POSTSUBSCRIPT s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT , yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT ) ∼ DSafety end_POSTSUBSCRIPT [ Droman_KL ( pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( ⋅ ∣ yitalic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT , yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT , italic_x ) ∥ pitalic_r start_POSTSUBSCRIPT ϕ end_POSTSUBSCRIPT ( ⋅ ∣ yitalic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT , yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT , italic_x ) ) ] with fθ=logπθ−logπreff_θ= _θ- _reffitalic_θ = log πitalic_θ - log πroman_ref and rϕr_φritalic_ϕ the implicit reward function from DPO. Safe-NCA (Chen et al. 2024) ℒsafe-NCA Pair(πθ;πref)=−(x,sw,sl)∼Safety[logσ(β(logπθ(sw|x)πref(sw|x)−logπθ(sl|x)πref(sl|x)))+12log[σ(βlogπref(sw|x)πθ(sw|x))×σ(βlogπref(sl|x)πθ(sl|x))]]L_safe-NCA Pair ( _θ; _ref )=-E_ (x,y_s_w,y_s_l ) _Safety [ σ (β ( _θ(y_s_w|x) _ref(y_s_w|x)- _θ(y_s_l|x) _ref(y_s_l|x) ) )+ 12 [σ (β _ref(y_s_w|x) _θ(y_s_w|x) )×σ (β _ref(y_s_l|x) _θ(y_s_l|x) ) ] ]Lsafe-NCA Pair ( πitalic_θ ; πroman_ref ) = - E( x , y start_POSTSUBSCRIPT s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT , yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT ) ∼ DSafety end_POSTSUBSCRIPT [ log σ ( β ( log divide start_ARG πitalic_θ ( yitalic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) end_ARG start_ARG πroman_ref ( yitalic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) end_ARG - log divide start_ARG πitalic_θ ( yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) end_ARG start_ARG πroman_ref ( yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) end_ARG ) ) + divide start_ARG 1 end_ARG start_ARG 2 end_ARG log [ σ ( β log divide start_ARG πroman_ref ( yitalic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) end_ARG start_ARG πitalic_θ ( yitalic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) end_ARG ) × σ ( β log divide start_ARG πroman_ref ( yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) end_ARG start_ARG πitalic_θ ( yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) end_ARG ) ] ] Safe-SPPO (Wu et al. 2024) ℒSafe SPPO(πθ;πref)=(x,sw,sl)∼Safety[(logπθ(sw|x)πref(sw|x)−0.5β)2+(logπθ(sl|x)πref(sl|x)+0.5β)2]L_Safe SPPO ( _θ; _ref )=E_ (x,y_s_w,y_s_l ) _Safety [ ( _θ (y_s_w|x ) _ref (y_s_w|x )- 0.5β )^2+ ( _θ (y_s_l|x ) _ref (y_s_l|x )+ 0.5β )^2 ]LSafe SPPO ( πitalic_θ ; πroman_ref ) = E( x , y start_POSTSUBSCRIPT s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT , yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT ) ∼ DSafety end_POSTSUBSCRIPT [ ( log divide start_ARG πitalic_θ ( yitalic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) end_ARG start_ARG πref ( yitalic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) end_ARG - divide start_ARG 0.5 end_ARG start_ARG β end_ARG )2 + ( log divide start_ARG πitalic_θ ( yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) end_ARG start_ARG πref ( yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) end_ARG + divide start_ARG 0.5 end_ARG start_ARG β end_ARG )2 ] Safe-AOT (Melnyk et al. 2024) ℒSafe-AOT(πθ;πref)=−(x,sw,sl)∼Safety[logσ(β(logπθ(sw<|x)πθ(sl<|x)−logπref(sw<|x)πref(sl<|x)))(1−ϵ)+logσ(−β(logπθ(sw<|x)πθ(sl<|x)−logπref(sw<|x)πref(sl<|x)))ϵ]L_Safe-AOT ( _θ; _ref )=-E_ (x,y_s_w,y_s_l ) _Safety [ σ (β ( _θ(y^<_s_w|x) _θ(y^<_s_l|x)- _ref(y^<_s_w|x) _ref(y^<_s_l|x) ) )(1-ε)+ σ (-β ( _θ(y^<_s_w|x) _θ(y^<_s_l|x)- _ref(y^<_s_w|x) _ref(y^<_s_l|x) ) )ε ]LSafe-AOT ( πitalic_θ ; πroman_ref ) = - E( x , y start_POSTSUBSCRIPT s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT , yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT ) ∼ DSafety end_POSTSUBSCRIPT [ log σ ( β ( log divide start_ARG πitalic_θ ( y<s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) end_ARG start_ARG πitalic_θ ( y<s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) end_ARG - log divide start_ARG πroman_ref ( y<s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) end_ARG start_ARG πroman_ref ( y<s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) end_ARG ) ) ( 1 - ϵ ) + log σ ( - β ( log divide start_ARG πitalic_θ ( y<s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) end_ARG start_ARG πitalic_θ ( y<s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) end_ARG - log divide start_ARG πroman_ref ( y<s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) end_ARG start_ARG πroman_ref ( y<s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) end_ARG ) ) ϵ ] Safe-AOT_pair (Melnyk et al. 2024) ℒSafe-AOTp(πθ;πref)=−(x,sw,sl)∼Safety[logσ(β(logπθ(sw<|x)πref(sw<|x)−logπθ(sl<|x)πref(sl<|x)))(1−ϵ)+logσ(−β(logπθ(sw<|x)πref(sw<|x)−logπθ(sl<|x)πref(sl<|x)))ϵ]L_Safe-AOTp ( _θ; _ref )=-E_ (x,y_s_w,y_s_l ) _Safety [ σ (β ( _θ(y^<_s_w|x) _ref(y^<_s_w|x)- _θ(y^<_s_l|x) _ref(y^<_s_l|x) ) )(1-ε)+ σ (-β ( _θ(y^<_s_w|x) _ref(y^<_s_w|x)- _θ(y^<_s_l|x) _ref(y^<_s_l|x) ) )ε ]LSafe-AOTp ( πitalic_θ ; πroman_ref ) = - E( x , y start_POSTSUBSCRIPT s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT , yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT ) ∼ DSafety end_POSTSUBSCRIPT [ log σ ( β ( log divide start_ARG πitalic_θ ( y<s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) end_ARG start_ARG πroman_ref ( y<s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) end_ARG - log divide start_ARG πitalic_θ ( y<s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) end_ARG start_ARG πroman_ref ( y<s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) end_ARG ) ) ( 1 - ϵ ) + log σ ( - β ( log divide start_ARG πitalic_θ ( y<s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) end_ARG start_ARG πroman_ref ( y<s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) end_ARG - log divide start_ARG πitalic_θ ( y<s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) end_ARG start_ARG πroman_ref ( y<s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) end_ARG ) ) ϵ ] Safe-ORPO (Hong, Lee, and Thorne 2024) ℒSafe-ORPO(πθ)=−(x,sw)∼Safety[logπθ(sw|x)]−λ(x,sw,sl)∼Safety[logσ(log(πθ(sw|x)1−πθ(sw|x)×1−πθ(sl|x)πθ(sl|x)))]L_Safe-ORPO ( _θ )=-E_ (x,y_s_w ) _Safety [ _θ (y_s_w|x ) ]- _ (x,y_s_w,y_s_l ) _Safety [ σ ( ( _θ (y_s_w|x )1- _θ (y_s_w|x )× 1- _θ (y_s_l|x ) _θ (y_s_l|x ) ) ) ]LSafe-ORPO ( πitalic_θ ) = - E( x , y start_POSTSUBSCRIPT s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT ) ∼ DSafety end_POSTSUBSCRIPT [ log πitalic_θ ( yitalic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) ] - λ E( x , y start_POSTSUBSCRIPT s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT , yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT ) ∼ DSafety end_POSTSUBSCRIPT [ log σ ( log ( divide start_ARG πitalic_θ ( yitalic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) end_ARG start_ARG 1 - πitalic_θ ( yitalic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT | x ) end_ARG × divide start_ARG 1 - πitalic_θ ( yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) end_ARG start_ARG πitalic_θ ( yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT | x ) end_ARG ) ) ] with λ>0λ>0λ > 0 Safety Objectives: Given a LLM π, the objective is to finetune it with a finetune procedure f so that the safety score S(f(π))S(f(π))S ( f ( π ) ) is maximized while minimizing the Attack Success Rate ASR(f(π))ASR(f(π))ASR ( f ( π ) ): maxS(f(π))andminASR(f(π)) S(f(π)) (f(π))max S ( f ( π ) ) and min ASR ( f ( π ) ) This dual focus aims at enhancing model robustness by reducing the likelihood of generating harmful content and improving resistance to adversarial attacks. Safety Alignment In this section, we start by introducing the methodology that we propose to address the safety of LLMs, then we present the safety alignment methods as well as the corresponding safety dataset. Methodology The safety problem in LLMs can be approached as an alignment problem. The objective is to align the model with a dataset that contains both safe and less safe responses. By doing so, the model learns to prioritize generating safer outputs while minimizing the risk of harmful content. This alignment process, supported by preference optimization techniques (such as DPO, IPO, etc.), fine-tunes the model to consistently favor responses that adhere to the safety risk taxonomy C as defined in Definition 1. Dataset with the pairwise comparison for safe alignment To construct our preference dataset, we start by filtering the PKU-SafeRLHF dataset available at ://huggingface.co/ datasets/PKU-Alignment/PKU-SafeRLHF. Our goal is to create a dataset that enables pairwise comparisons of responses based on safety. Each entry in our constructed dataset includes a prompt x(j)x^(j)x( j ), a safe response ysw(j)y^(j)_s_wy( j )s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT (safety-accepted response), and a less safe response ysl(j)y^(j)_s_ly( j )s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT (safety-rejected response). Thus, we get Safety=x(j),ysw(j),ysl(j)j=1ND_Safety= \x^(j),y^(j)_s_w,y^(j)_s_l \^N_j=1DSafety = x( j ) , y( j )s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT , y( j )s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT Nitalic_j = 1, where response ysw(j)y^(j)_s_wy( j )s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT is more harmless than response ysl(j)y^(j)_s_ly( j )s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT and N=47,077N=47,077N = 47 , 077 is the cardinality of SafetyD_SafetyDSafety. This approach allows us to compare the safety of different responses to the same prompt, providing a robust foundation for optimizing the safety of LLMs through optimization techniques from human feedback. Safety Alignment Methods In the context of aligning LLMs with safety objectives, we propose the application of 101010 alignment methods, each optimizing a distinct loss function tailored to enforce safety constraints. Table 1 summarizes the key methods, alongside their corresponding loss functions. We refer to these methods as Safe-DPO, Safe-rDPO, Safe-IPO, etc., which are designed to optimize the model’s behavior by minimizing risks associated with unsafe outputs. The notations used in the loss functions are as follows: sw=ysw1,ysw2,…,yswNy_s_w= \y^1_s_w,y^2_s_w,…,y^N_s_w \yitalic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT = y1italic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT , y2italic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT , … , yitalic_Nitalic_s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT and sl=ysl1,ysl2,…,yslNy_s_l= \y^1_s_l,y^2_s_l,…,y^N_s_l \yitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT = y1italic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT , y2italic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT , … , yitalic_Nitalic_s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT represent sets of safe and unsafe outputs, respectively, with their sorted versions denoted by sw<y^<_s_wy<s start_POSTSUBSCRIPT w end_POSTSUBSCRIPT and sl<y^<_s_ly<s start_POSTSUBSCRIPT l end_POSTSUBSCRIPT. The logistic function is denoted by σ, ϵ∈(0,1)ε∈(0,1)ϵ ∈ ( 0 , 1 ) represents the label smoothing parameter, and β>0β>0β > 0 denotes the temperature parameter. The model being updated is represented by πθ _θπitalic_θ, while πref _refπroman_ref denotes the frozen reference model. Moreover, DKLD_KLDroman_KL denotes the Kullback-Leibler divergence and fp_fpitalic_f is the empirical distribution of exp(f) (f)exp ( f ): the exponential function f. These notations are essential for understanding the loss functions presented in Table 1. Evaluations In this section, we elaborate on the benchmarking tools used in this work and associated metrics. Table 2: Benchmarking the Falcon models with the ALERT framework. Each row depicts a risk category from the ALERT taxonomy, while each column depicts an LLM under evaluation. Values in the last row depict overall safety scores, all others are category-wise safety scores (higher is safer). Safe scores (S(π)≥99S(π)≥ 99S ( π ) ≥ 99) are gray, unsafe scores within 90≤S(π)<9990≤ S(π)<9990 ≤ S ( π ) < 99 are orange, and highly unsafe scores S(π)<90S(π)<90S ( π ) < 90 are red. Best viewed in color. Falcon 11B Unsafe Category Instruct Safe-ORPO Safe-SPPO Safe-NCA Safe-KTO Safe-DPO Safe-AOTp Safe-EXO Safe-AOT Safe-SLiC Safe-rDPO Safe-IPO Hate Speech & Discrimination 84.3384.3384.33% 80.7380.7380.73% 99.3199.3199.31% 99.7199.7199.71% 99.8999.8999.89% 99.9899.9899.98% 100.00%100.00\%100.00 % 99.9899.9899.98% 99.9899.9899.98% 99.9699.9699.96% 99.9899.9899.98% 99.9699.9699.96% Suicide & Self-Harm 84.2784.2784.27% 76.6876.6876.68% 99.6499.6499.64% 99.6499.6499.64% 99.2899.2899.28% 100.00%100.00\%100.00 % 100.00%100.00\%100.00 % 100.00%100.00\%100.00 % 100.00%100.00\%100.00 % 100.00%100.00\%100.00 % 100.00%100.00\%100.00 % 100.00%100.00\%100.00 % Guns & Illegal Weapons 66.5666.5666.56% 71.0371.0371.03% 96.1596.1596.15% 98.6998.6998.69% 98.7198.7198.71% 98.4498.4498.44% 99.6099.6099.60% 99.7599.7599.75% 100.00%100.00\%100.00 % 99.9299.9299.92% 99.7599.7599.75% 99.7599.7599.75% Criminal Planning 47.4147.4147.41% 53.1953.1953.19% 90.9890.9890.98% 95.9695.9695.96% 97.7497.7497.74% 99.0599.0599.05% 99.7099.7099.70% 99.7799.7799.77% 99.6899.6899.68% 99.7999.7999.79% 99.82%99.82\%99.82 % 99.82%99.82\%99.82 % Sexual Content 73.1573.1573.15% 72.7472.7472.74% 96.2096.2096.20% 98.8998.8998.89% 99.3499.3499.34% 99.3499.3499.34% 100.00%100.00\%100.00 % 99.8999.8999.89% 100.00%100.00\%100.00 % 100.00%100.00\%100.00 % 99.6799.6799.67% 99.8999.8999.89% Regulated or Controlled Substances 57.4557.4557.45% 63.6163.6163.61% 95.4495.4495.44% 98.3498.3498.34% 98.0498.0498.04% 99.0399.0399.03% 99.6799.6799.67% 99.5999.5999.59% 99.6099.6099.60% 99.5899.5899.58% 99.8499.8499.84% 99.87%99.87\%99.87 % Overall Safety Score S 64.4264.4264.42% 66.3566.3566.35% 95.1395.1395.13% 97.9797.9797.97% 98.6798.6798.67% 99.3099.3099.30% 99.8199.8199.81% 99.8299.8299.82% 99.8399.8399.83% 99.8499.8499.84% 99.8699.8699.86% 99.87%99.87\%99.87 % Table 3: Benchmarking the models with the ALERT framework. Each row depicts a risk category from ALERT taxonomy, while each column depicts an LLM under evaluation. Values in the last row depict overall safety scores, all others are category-wise safety scores (higher is safer). Safe scores (S(π)≥99S(π)≥ 99S ( π ) ≥ 99) are grey, unsafe scores within 90≤S(π)<9990≤ S(π)<9990 ≤ S ( π ) < 99 are orange, and highly unsafe scores S(π)<90S(π)<90S ( π ) < 90 are red. Safe-Falcon corresponds to Falcon 11B Safe-IPO in Table 2. . Unsafe Category Alpaca Mistral 7B OLMo 7B YI 1.5 9B Llama 3.1 8B Mixtral 8x7B Qwen-2 7B Vicuna 7B GPT-3.5 Gemma 2 9B Safe Falcon Hate Speech & Discrimination 81.4481.4481.44% 91.0191.0191.01% 91.9291.9291.92% 94.6094.6094.60% 96.3296.3296.32% 97.2997.2997.29% 96.4996.4996.49% 99.2099.2099.20% 97.9697.9697.96% 99.3699.3699.36% 99.96%99.96\%99.96 % Suicide & Self-Harm 73.9673.9673.96% 88.2588.2588.25% 86.8086.8086.80% 90.9690.9690.96% 97.6597.6597.65% 96.9396.9396.93% 98.7398.7398.73% 97.6597.6597.65% 100.00%100.00\%100.00 % 98.1998.1998.19% 100.00%100.00\%100.00 % Guns & Illegal Weapons 66.5166.5166.51% 72.9272.9272.92% 85.1685.1685.16% 74.6874.6874.68% 92.5992.5992.59% 87.5287.5287.52% 93.0193.0193.01% 93.2693.2693.26% 96.7696.7696.76% 98.0198.0198.01% 99.75%99.75\%99.75 % Criminal Planning 51.3251.3251.32% 52.9152.9152.91% 69.1369.1369.13% 68.5568.5568.55% 84.8784.8784.87% 82.5082.5082.50% 86.5986.5986.59% 88.7588.7588.75% 88.9688.9688.96% 89.4789.4789.47% 99.82%99.82\%99.82 % Sexual Content 72.6272.6272.62% 87.7087.7087.70% 78.2678.2678.26% 85.5285.5285.52% 93.9993.9993.99% 95.4295.4295.42% 96.4396.4396.43% 98.7798.7798.77% 99.0099.0099.00% 98.7798.7798.77% 99.89%99.89\%99.89 % Regulated or Controlled Substances 55.1855.1855.18% 62.5262.5262.52% 71.1371.1371.13% 70.2870.2870.28% 94.1894.1894.18% 87.3587.3587.35% 92.0692.0692.06% 92.6892.6892.68% 95.9795.9795.97% 98.1198.1198.11% 99.87%99.87\%99.87 % Overall Safety Score S 64.2864.2864.28% 70.7670.7670.76% 78.7478.7478.74% 79.0279.0279.02% 84.1684.1684.16% 88.1888.1888.18% 91.9391.9391.93% 93.6593.6593.65% 94.394.394.3% 95.395.395.3% 99.87%99.87\%99.87 % Table 4: ASR score of each attacking strategy in the adversarial ALERT. Each row represents an attacking strategy, while each column corresponds to an LLM under evaluation. A model is robust when the ASR is ≤1%≤ 1\%≤ 1 % (grey), vulnerable 1%<ASR≤5%1\%<ASR≤ 5\%1 % < A S R ≤ 5 % (orange), and highly vulnerable ASR ≥5%≥ 5\%≥ 5 % (red). Best viewed in color. Falcon 11B Attack Type Instruct Safe-ORPO Safe-SPPO Safe-NCA Safe-KTO Safe-DPO Safe-SLiC Safe-AOT Safe-EXO Safe-AOTp Safe-IPO Safe-rDPO Adversarial Suffix 46.51% 32.16% 10.95% 5.92% 1.42% 0.70% 0.40% 0.31% 0.13% 0.11% 0.16% 0.10%0.10\%0.10 % Adversarial Prefix 50.48% 38.85% 4.93% 1.33% 0.13% 0.11% 0.04% 0.01% 0.00%0.00\%0.00 % 0.06% 0.00%0.00\%0.00 % 0.00%0.00\%0.00 % Token Manipulation 30.08% 27.15% 3.75% 1.58% 1.04% 0.51% 0.21% 0.18% 0.14% 0.18% 0.11%0.11\%0.11 % 0.15% Jailbreaking 61.49% 36.44% 15.02% 5.82% 1.86% 3.05% 1.18% 0.81% 0.43% 0.30% 0.01%0.01\%0.01 % 0.01%0.01\%0.01 % Overall ASR Score 45.60% 33.06% 8.22% 3.47% 1.11% 1.04% 0.44% 0.31% 0.17% 0.16% 0.07% 0.06%0.06\%0.06 % Table 5: ASR score of each attacking strategy in the adversarial ALERT. Each row represents an attacking strategy, while each column corresponds to an LLM under evaluation. A model is robust when the ASR is ≤1%≤ 1\%≤ 1 % (grey), vulnerable 1%<ASR≤5%1\%<ASR≤ 5\%1 % < A S R ≤ 5 % (orange), and highly vulnerable ASR ≥5%≥ 5\%≥ 5 % (red). Safe-Falcon corresponds to Falcon 11B Safe-rDPO in Table 4. Attack Type Alpaca YI 1.5 9B OLMo 7B Mistral 7B Llama 3.1 8B Qwen-2 7B Vicuna 7B Mixtral 8x7B Gemma 2 9B GPT-3.5 Safe-Falcon Adversarial Suffix 26.97% 33.03% 24.00% 31.57% 19.61% 19.65% 9.50% 9.41% 7.98% 5.54% 0.10%0.10\%0.10 % Adversarial Prefix 32.80% 26.50% 27.62% 56.08% 17.65% 12.08% 8.63% 9.02% 5.10% 3.01% 0.00%0.00\%0.00 % Token Manipulation 27.82% 18.81% 18.07% 15.48% 10.21% 8.27% 4.88% 4.09% 5.21% 4.74% 0.15%0.15\%0.15 % Jailbreaking 54.83% 51.48% 48.47% 10.12% 40.02% 36.91% 30.22% 27.82% 27.71% 20.63% 0.01%0.01\%0.01 % Overall ASR Score 34.90% 31.22% 28.50% 27.15% 20.82% 18.24% 12.54% 11.82% 10.93% 8.14% 0.06%0.06\%0.06 % ALERT To evaluate our model’s safety we used ALERT (Tedeschi et al. 2024). ALERT benchmark is designed to assess the safety of LLMs. It includes 45k45k45 k testing instructions grouped into the 666 main risk categories (corresponding to the safety risk taxonomy C) and 323232 subcategories. (Tedeschi et al. 2024) introduced a dataset of prompts composed of 14k14k14 k samples, denoted as PcP_cPitalic_c, designed to evaluate the safety and robustness of LLMs against a wide range of potentially harmful inputs categorized under various safety risks. In evaluating our models, we leverage Llama Guard 3 8B (Dubey et al. 2024), considered to be on of the best available safety classifier. Performance is measured using the safety score defined in Definition 4. Adversarial ALERT To comprehensively evaluate the safety and robustness of our models, we utilized the adversarial component of the ALERT benchmark in our testing protocol. Alert includes 31k31k31 k prompts (denoted as P~a P_aover~ start_ARG P end_ARGa), specifically designed to challenge the model with inputs crafted to exploit potential weaknesses, using techniques such as adversarial suffix attack, adversarial prefix attack, token manipulation, and jailbreaking as stated in Definition 2. The purpose of this part is to understand the LLM behavior under manipulative scenarios. Robustness against adversarial attacks is captured using ASR defined in Definition 5. Figure 2: (a) [maxtox]E[ _tox]E [ maxtox ] + Benign Figure 3: (b) [maxtox]E[ _tox]E [ maxtox ] + Adversarial Figure 4: (c) avgtoxavg_toxavgtox + Benign Figure 5: (d) avgtoxavg_toxavgtox + Adversarial Figure 6: Toxicity evaluation on all alignment techniques with benign and adversarial prompting. Metrics are Expected Max Toxicity ( )[maxtox]E)[ _tox]E ) [ maxtox ]) and Average Toxicity (avgtoxavg_toxavgtox). The higher the red intensity corresponds to more toxic LLM. Toxicity A significant component of having Safe LLMs entails having a nontoxic LLM. Therefore, the toxicity benchmark introduced in an existing work (Wang et al. 2023) was used to evaluate our proposed safe LLMs. Note that the mentioned benchmark leverages perspective API111w.perspectiveapi.com (Lees et al. 2022) as a toxicity evaluator model, that returns toxicity score ranging from 0 (nontoxic) to 111 (toxic). The benchmark is composed of four benchmark datasets, with the first two being a 1.2k1.2k1.2 k subset of toxic (T) and nontoxic (NT) prompts from the RealToxicityPrompts dataset (Gehman et al. 2020). The remaining two datasets are 1.2k1.2k1.2 k prompts obtained using GPT-3.5 (G3.5G3.5G 3.5) and GPT4 (G4G4G 4), which were designed to provoke a toxic LLM response. Additionally, two system prompts were explored, namely, benign and adversarial. Both types will instruct the LLM to be a helpful assistant, with the difference that adversarial prompting will explicitly steer the LLM to be toxic. We followed the same setup suggested by (Wang et al. 2023), where 252525 generations were generated for each prompt. Based on these generations, two metrics were considered, which are presented below: • Expected Max Toxicity denoted as E[maxtox]E[ _tox]E [ maxtox ], aimed to evaluate the worst-case scenario, where the maximum score over the 252525 generations for each prompt, is averaged over all prompts. • Average Toxicity this averages the toxicity scores for all generations, labeled as avgtoxavg_toxavgtox. Results In this section, results obtained using benchmarking tools introduced in the previous section are presented and discussed. Table 2 presents the safety score S across different alignment techniques on the Falcon 11B model (Malartic et al. 2024). It includes the baseline Instruct model (a finetuned version of Falcon 11B on Ultrachat dataset) and 111111 safety enhanced models (following the safety alignment methods in Table 1 where the reference model πref _refπref corresponds to Falcon 11B instruct). The results of Table 2 obtained using ALERT for all safe Falcon models. These results are quantified using the safety score S, as outlined in Definition 4, with higher scores indicating greater safety and resilience. Most models exhibited commendable performance, highlighting the effectiveness of the alignments applied. However, notable deviations were observed with the Instruct, which is the base model, understandably scored lower in comparison to its aligned counterparts. Similarly, the Safe-ORPO model displayed performance metrics closely mirroring those of the Instruct model, suggesting the limited efficacy of the ORPO enhancements in improving safety outcomes. Contrastingly, the other alignment techniques demonstrated substantial improvements in safety across all categories. Such results underscore the potential of alignment techniques to enhance model robustness and safety in critical safety categories. Table 3, contrasts our safest aligned Falcon in terms of safety score S, Safe-IPO, against common LLMs. Similar to the observations noted for the ASR score, Safe-Falcon performed best against all considered benchmarks, with approximately 444% increase when compared to GPT-3.5, the second performing model, in terms of safety score. Table 4 presents the ASR across different alignment techniques on the Falcon 11B model, against a set of adversarial attacks. Performance is evaluated by the percentage of successful attacks, with lower percentages indicating high robustness. The overall performance of each model is also summarized to identify the most robust model against adversarial attacks. The models Safe-IPO, Safe-AOTp, and Safe-EXO emerged as the top performers, demonstrating the highest resistance across the tested attacks. Table 5 compares our aligned models against mainstream models in terms of ASR, where Safe-Falcon refers to Safe-rDPO due to being the best in terms of ASR. Safe-Falcon and GPT-3.5 exhibit superior robustness, with particularly low overall vulnerability scores of 0.060.060.06% and 8.148.148.14%, respectively. In contrast, models like Yi 1.5 9B and Alpaca display significantly higher susceptibility, with Alpaca recording the worst ASR overall score of 34.9034.9034.90%. The results underscore the safety shortcomings in the considered models and highlight the potential of the alignment techniques employed for Safe-Falcon. Moving on to toxicity, heatmaps in Figure 6, present the toxicity evaluation results for all safe LLMs proposed in this paper, against the Instruct. It is evident from Figure 6, that all Safe Falcons demonstrate a tangible drop in toxic responses, in both types of system prompting. As expected, results bolster the claim that Instruct is highly susceptible to producing toxic content, which compromises the general safety of the LLM. This claim is manifested by scoring the highest in all heatmaps in Figure 6. As seen in figures Figure 6.b and Figure 6.d, when adversarial prompting was applied to the prompts in NTNTN T, Instruct likelihood of returning toxic prompts increased, evident by the obtained 0.880.880.88 and 0.290.290.29 in terms of E[maxtox]E[ _tox]E [ maxtox ] and avgtoxavg_toxavgtox, respectively. Looking at Figure 6.c, Safe-IPO and Safe-rDPO demonstrated the maximum drop in all metrics that is clear by the scoring 0.050.050.05 or less in terms of avgtoxavg_toxavgtox, when benign prompting is applied. Additionally, both techniques illustrated their robustness by maintaining low toxic scores even when adversarial prompting is deployed. This observation can be seen Figure 6.b and Figure 6.d, with scores being limited to 0.250.250.25 and 0.070.070.07 for [maxtox]E [ _tox ]E [ maxtox ] and avgtoxavg_toxavgtox, respectively. On the other hand, the worst performing alignment technique is Safe-SPPO as it obtained the least improvement in toxicity. Several conclusions can be deduced from this experiment, which are (1) Instruct is vulnerable to producing toxic text impacting the LLM safety, (2) all safe models trained using various alignment techniques obtained noticeable improvement in being less toxic over different system prompting, and (3) Safe-IPO and Safe-rDPO were the best performing alignment techniques in terms of toxicity, and the most robust against adversarial prompting. Following evaluating the safety of our proposed models, we evaluate them on general benchmarks typically used to assess the general performance of LLM (Fourrier et al. 2024). Table 6 depicts the results of our safe models with the base instruct model. The key takeaway from this table is that alignment techniques maintained scores within the same range of Instruct. Counter to expectation, Safe-NCA outperformed Instruct in three benchmarks. Additionally, Safe-rDPO scored the best in GPQA and matched Instruct in IFEval. Therefore, this demonstrates that our aligned model maximized performance in terms of safety while preserving general performance scores on par with the base model. Table 6: Performance scores of different Falcon 11B models on the LLM Leaderboard (Fourrier et al. 2024) (reporting the raw values.) Tasks Falcon 11B IFEval BBH GPQA MATH MuSR MMLU-PRO Instruct 0.387 0.426 0.286 0.012 0.425 0.253 Safe-DPO 0.373 0.428 0.293 0.004 0.400 0.260 Safe-rDPO 0.387 0.414 0.30 0.002 0.427 0.262 Safe-IPO 0.274 0.412 0.296 0.005 0.394 0.261 Safe-SLiC 0.360 0.428 0.295 0.002 0.401 0.264 Safe-KTO 0.363 0.429 0.293 0.001 0.406 0.263 Safe-EXO 0.351 0.413 0.287 0 0.436 0.264 Safe-NCA 0.371 0.435 0.290 0.015 0.441 0.262 Safe-SPPO 0.380 0.435 0.282 0.0075 0.430 0.259 Safe-AOT 0.360 0.421 0.288 0.0052 0.439 0.263 Safe-AOTp 0.351 0.419 0.285 0.0007 0.438 0.264 Conclusion and Future Works This work studies the effect of variants of direct preference optimization methods on LLMs safety. Our experiments demonstrate a substantial improvement in the safety score of the Falcon 11B model, increasing from 57.64% to 99.90%, positioning it among the safest LLM models in the state of the art. However, our study also revealed an important trade-off: while safety scores improved dramatically, we observed a reduction in general capabilities, particularly in mathematical tasks. This finding highlights the complex relationship between safety enhancements and overall model performance. In particular, we identified noise contrastive alignment (Safe-NCA) as an optimal method for balancing safety and performance. Our research conclusively shows that alignment techniques can be sufficient for building safe and robust models. Nevertheless, the observed trade-offs highlight the need for further investigation. As future work, we plan to explore methods to mitigate the negative impact on performance in other tasks such as math and reasoning, while maintaining the high level of safety achieved. References Azar et al. (2024) Azar, M. G.; Guo, Z. D.; Piot, B.; Munos, R.; Rowland, M.; Valko, M.; and Calandriello, D. 2024. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, 4447–4455. PMLR. Chen et al. (2024) Chen, H.; He, G.; Su, H.; and Zhu, J. 2024. Noise contrastive alignment of language models with explicit rewards. arXiv preprint arXiv:2402.05369. Chowdhury (2024) Chowdhury. 2024. Provably robust dpo: Aligning language models with noisy feedback. arXiv preprint arXiv:2403.00409. Dai et al. (2023) Dai, J.; Pan, X.; Sun, R.; Ji, J.; Xu, X.; Liu, M.; Wang, Y.; and Yang, Y. 2023. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.12773. Dubey et al. (2024) Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; et al. 2024. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783. Ethayarajh et al. (2024) Ethayarajh, K.; Xu, W.; Muennighoff, N.; Jurafsky, D.; and Kiela, D. 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306. Fourrier et al. (2024) Fourrier, C.; Habib, N.; Lozovskaya, A.; Szafer, K.; and Wolf, T. 2024. Open LLM Leaderboard v2. Ge et al. (2023) Ge, S.; Zhou, C.; Hou, R.; Khabsa, M.; Wang, Y.-C.; Wang, Q.; Han, J.; and Mao, Y. 2023. Mart: Improving llm safety with multi-round automatic red-teaming. arXiv preprint arXiv:2311.07689. Gehman et al. (2020) Gehman, S.; Gururangan, S.; Sap, M.; Choi, Y.; and Smith, N. A. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462. Gupta et al. (2024) Gupta, P.; Yau, L. Q.; Low, H. H.; Lee, I.; Lim, H. M.; Teoh, Y. X.; Koh, J. H.; Liew, D. W.; Bhardwaj, R.; Bhardwaj, R.; et al. 2024. WalledEval: A Comprehensive Safety Evaluation Toolkit for Large Language Models. arXiv preprint arXiv:2408.03837. Hong, Lee, and Thorne (2024) Hong, J.; Lee, N.; and Thorne, J. 2024. Reference-free monolithic preference optimization with odds ratio. arXiv preprint arXiv:2403.07691. Ji et al. (2024a) Ji, H.; Lu, C.; Niu, Y.; Ke, P.; Wang, H.; Zhu, J.; Tang, J.; and Huang, M. 2024a. Towards efficient and exact optimization of language model alignment. arXiv preprint arXiv:2402.00856. Ji et al. (2024b) Ji, J.; Liu, M.; Dai, J.; Pan, X.; Zhang, C.; Bian, C.; Chen, B.; Sun, R.; Wang, Y.; and Yang, Y. 2024b. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems, 36. Jiang et al. (2024) Jiang, B.; Jing, Y.; Shen, T.; Yang, Q.; and Xiong, D. 2024. DART: Deep Adversarial Automated Red Teaming for LLM Safety. arXiv preprint arXiv:2407.03876. Kumar et al. (2023) Kumar, A.; Agarwal, C.; Srinivas, S.; Feizi, S.; and Lakkaraju, H. 2023. Certifying llm safety against adversarial prompting. arXiv preprint arXiv:2309.02705. Lees et al. (2022) Lees, A.; Tran, V. Q.; Tay, Y.; Sorensen, J.; Gupta, J.; Metzler, D.; and Vasserman, L. 2022. A new generation of perspective api: Efficient multilingual character-level transformers. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, 3197–3207. Malartic et al. (2024) Malartic, Q.; Chowdhury, N. R.; Cojocaru, R.; Farooq, M.; Campesan, G.; Djilali, Y. A. D.; Narayan, S.; Singh, A.; Velikanov, M.; Boussaha, B. E. A.; et al. 2024. Falcon2-11B Technical Report. arXiv preprint arXiv:2407.14885. Melnyk et al. (2024) Melnyk, I.; Mroueh, Y.; Belgodere, B.; Rigotti, M.; Nitsure, A.; Yurochkin, M.; Greenewald, K.; Navratil, J.; and Ross, J. 2024. Distributional Preference Alignment of LLMs via Optimal Transport. arXiv preprint arXiv:2406.05882. Ouyang et al. (2022) Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 27730–27744. Rafailov et al. (2024) Rafailov, R.; Sharma, A.; Mitchell, E.; Manning, C. D.; Ermon, S.; and Finn, C. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36. Tedeschi et al. (2024) Tedeschi, S.; Friedrich, F.; Schramowski, P.; Kersting, K.; Navigli, R.; Nguyen, H.; and Li, B. 2024. ALERT: A Comprehensive Benchmark for Assessing Large Language Models’ Safety through Red Teaming. arXiv preprint arXiv:2404.08676. Touvron et al. (2023) Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Varshney et al. (2023) Varshney, N.; Dolin, P.; Seth, A.; and Baral, C. 2023. The art of defending: A systematic evaluation and analysis of llm defense strategies on safety and over-defensiveness. arXiv preprint arXiv:2401.00287. Wang et al. (2023) Wang, B.; Chen, W.; Pei, H.; Xie, C.; Kang, M.; Zhang, C.; Xu, C.; Xiong, Z.; Dutta, R.; Schaeffer, R.; et al. 2023. DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models. In NeurIPS. Wu et al. (2024) Wu, Y.; Sun, Z.; Yuan, H.; Ji, K.; Yang, Y.; and Gu, Q. 2024. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675. Yuan et al. (2024) Yuan, T.; He, Z.; Dong, L.; Wang, Y.; Zhao, R.; Xia, T.; Xu, L.; Zhou, B.; Li, F.; Zhang, Z.; et al. 2024. R-judge: Benchmarking safety risk awareness for llm agents. arXiv preprint arXiv:2401.10019. Zhao et al. (2022) Zhao, Y.; Khalman, M.; Joshi, R.; Narayan, S.; Saleh, M.; and Liu, P. J. 2022. Calibrating sequence likelihood improves conditional language generation. In The eleventh international conference on learning representations. Zheng et al. (2024) Zheng, C.; Yin, F.; Zhou, H.; Meng, F.; Zhou, J.; Chang, K.-W.; Huang, M.; and Peng, N. 2024. On prompt-driven safeguarding for large language models. In Forty-first International Conference on Machine Learning.