Paper deep dive
S-Eval: Automatic and Adaptive Test Generation for Benchmarking Safety Evaluation of Large Language Models
Xiaohan Yuan, Jinfeng Li, Dongxia Wang, Yuefeng Chen, Xiaofeng Mao, Longtao Huang, Hui Xue, Wenhai Wang, Kui Ren, Jingyi Wang
Models: Baichuan2-13B, Baichuan2-7B, Claude-2, ErnieBot-4.0, GPT-3.5-Turbo, GPT-4-Turbo, Llama-2-13B, Llama-2-70B, Llama-2-7B, Qwen-14B-Chat, Qwen-7B-Chat
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 98%
Last extracted: 3/12/2026, 7:32:47 PM
Summary
S-Eval is an automated, LLM-based safety evaluation framework that addresses the lack of standardized risk taxonomies and manual assessment bottlenecks. It utilizes an expert testing LLM (Mt) to generate a comprehensive benchmark of 220,000 test cases across 102 risk dimensions and a safety critique LLM (Mc) to provide quantitative, explainable evaluations of LLM safety.
Entities (5)
Relation Signals (5)
Mc ā evaluates ā LLMs
confidence 100% Ā· Mc can provide quantitative and explainable safety evaluations for better risk awareness of LLMs.
Mt ā generates ā Test Cases
confidence 100% Ā· Mt is responsible for automatically generating test cases in accordance with the proposed risk taxonomy
S-Eval ā incorporates ā Mt
confidence 100% Ā· S-Eval incorporates two key components, i.e., an expert testing LLM Mt and a novel safety critique LLM Mc.
S-Eval ā incorporates ā Mc
confidence 100% Ā· S-Eval incorporates two key components, i.e., an expert testing LLM Mt and a novel safety critique LLM Mc.
Mt ā basedon ā Qwen-14B-Chat
confidence 95% Ā· we build our expert testing LLM Mt by supervised fine-tuning Qwen-14B-Chat
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Generative large language models (LLMs) have revolutionized natural language processing with their transformative and emergent capabilities. However, recent evidence indicates that LLMs can produce harmful content that violates social norms, raising significant concerns regarding the safety and ethical ramifications of deploying these advanced models. Thus, it is both critical and imperative to perform a rigorous and comprehensive safety evaluation of LLMs before deployment. Despite this need, owing to the extensiveness of LLM generation space, it still lacks a unified and standardized risk taxonomy to systematically reflect the LLM content safety, as well as automated safety assessment techniques to explore the potential risk efficiently. To bridge the striking gap, we propose S-Eval, a novel LLM-based automated Safety Evaluation framework with a newly defined comprehensive risk taxonomy. S-Eval incorporates two key components, i.e., an expert testing LLM ${M}_t$ and a novel safety critique LLM ${M}_c$. ${M}_t$ is responsible for automatically generating test cases in accordance with the proposed risk taxonomy. ${M}_c$ can provide quantitative and explainable safety evaluations for better risk awareness of LLMs. In contrast to prior works, S-Eval is efficient and effective in test generation and safety evaluation. Moreover, S-Eval can be flexibly configured and adapted to the rapid evolution of LLMs and accompanying new safety threats, test generation methods and safety critique methods thanks to the LLM-based architecture. S-Eval has been deployed in our industrial partner for the automated safety evaluation of multiple LLMs serving millions of users, demonstrating its effectiveness in real-world scenarios. Our benchmark is publicly available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2405.14191
- Canonical: https://arxiv.org/abs/2405.14191
- Code: https://github.com/IS2Lab/S-Eval
Trouble viewing inline? Open PDF directly ā
Full Text
100,725 characters extracted from source content.
Expand or collapse full text
S-Eval: Towards Automated and Comprehensive Safety Evaluation for Large Language Models Xiaohan Yuan Zhejiang UniversityHangzhouChina xiaohanyuan@zju.edu.cn , Jinfeng Li Alibaba GroupHangzhouChina jinfengli.ljf@alibaba-inc.com , Dongxia Wang š Zhejiang UniversityHangzhouChina dxwang@zju.edu.cn , Yuefeng Chen Alibaba GroupHangzhouChina yuefeng.chenyf@alibaba-inc.com , Xiaofeng Mao Alibaba GroupHangzhouChina mxf164419@alibaba-inc.com , Longtao Huang Alibaba GroupHangzhouChina kaiyang.hlt@alibaba-inc.com , Jialuo Chen Zhejiang UniversityHangzhouChina chenjialuo@zju.edu.cn , Hui Xue Alibaba GroupHangzhouChina hui.xueh@alibaba-inc.com , Xiaoxia Liu Zhejiang UniversityHangzhouChina liuxiaoxia@zju.edu.cn , Wenhai Wang Zhejiang UniversityHangzhouChina zdzzlab@zju.edu.cn , Kui Ren Zhejiang UniversityHangzhouChina kuiren@zju.edu.cn and Jingyi Wang Zhejiang UniversityHangzhouChina wangjyee@zju.edu.cn Abstract. Generative large language models (LLMs) have revolutionized natural language processing with their transformative and emergent capabilities. However, recent evidence indicates that LLMs can produce harmful content that violates social norms, raising significant concerns regarding the safety and ethical ramifications of deploying these advanced models. Thus, it is both critical and imperative to perform a rigorous and comprehensive safety evaluation of LLMs before deployment. Despite this need, owing to the extensiveness of LLM generation space, it still lacks a unified and standardized risk taxonomy to systematically reflect the LLM content safety, as well as automated safety assessment techniques to explore the potential risk efficiently. To bridge the striking gap, we propose S-Eval, a novel LLM-based automated Safety Evaluation framework with a newly defined comprehensive risk taxonomy. S-Eval incorporates two key components, i.e., an expert testing LLM ā³tsubscriptā³M_tMitalic_t and a novel safety critique LLM ā³csubscriptā³M_cMitalic_c. The expert testing LLM ā³tsubscriptā³M_tMitalic_t is responsible for automatically generating test cases in accordance with the proposed risk taxonomy (including 8 risk dimensions and a total of 102 subdivided risks). The safety critique LLM ā³csubscriptā³M_cMitalic_c can provide quantitative and explainable safety evaluations for better risk awareness of LLMs. In contrast to prior works, S-Eval differs in significant ways: (i) efficient ā we construct a multi-dimensional and open-ended benchmark111Our benchmark is publicly available at https://github.com/IS2Lab/S-Eval. comprising 220,000 test cases across 102 risks utilizing ā³tsubscriptā³M_tMitalic_t and conduct safety evaluations for 21 influential LLMs via ā³csubscriptā³M_cMitalic_c on our benchmark. The entire process is fully automated and requires no human involvement. (i) effective ā extensive validations show S-Eval facilitates a more thorough assessment and better perception of potential LLM risks, and ā³csubscriptā³M_cMitalic_c not only accurately quantifies the risks of LLMs but also provides explainable and in-depth insight into their safety, surpassing comparable models such as LLaMA-Guard-2. (i) adaptive ā S-Eval can be flexibly configured and adapted to the rapid evolution of LLMs and accompanying new safety threats, test generation methods and safety critique methods thanks to the LLM-based architecture. We further study the impact of hyper-parameters and language environments on model safety, which may lead to promising directions for future research. S-Eval has been deployed in our industrial partner for the automated safety evaluation of multiple LLMs serving millions of users, demonstrating its effectiveness in real-world scenarios. Large Language Models, Safety Evaluation, Test Generation, Benchmark 1. Introduction Large language models (LLMs) have exhibited remarkable performance across a range of tasks due to their revolutionary capabilities. Leading-edge LLMs, including GPT-4 (Achiam et al., 2023), LLaMA (Touvron et al., 2023), and Qwen (Bai et al., 2023), are increasingly being utilized not only as private intelligent assistants but also in security sensitive scenarios, such as in the medical and financial sectors (Son et al., 2023; Tang et al., 2023). However, amidst the swift advancement and pervasive applications of LLMs, there is also growing concern regarding their safety and potential risks. Recent studies have revealed that even state-of-the-art LLMs can, under routine conditions, produce content that breaches legal standards or contradicts societal values, such as providing illicit or unsafe advice (Durkin, 1997), exhibiting discriminatory tendencies (Sheng et al., 2021), or generating offensive responses (Gehman et al., 2020). These issues are further magnified in the context of adversarial attacks. This is because LLMs are typically trained on vast amounts of textual data, and a lack of effective data auditing or insufficient alignment with legal and ethical guidelines results in such unsafe behaviors that do not align with human expectations. Given the widespread applications and mounting concerns regarding the risks associated with LLMs, conducting a rigorous safety assessment prior to their real-world deployment is essential. Currently, some safety assessments have been executed, covering either specific safety concerns (Gehman et al., 2020; Parrish et al., 2021; Hendrycks et al., 2021) or multiple risk dimensions (Liang et al., 2022; Wang et al., 2024; Ganguli et al., 2022; Sun et al., 2023; Wang et al., 2023; Xu et al., 2023). However, existing assessments still suffer from several significant limitations. First, the risk taxonomies of them are loose without a unified risk taxonomy paradigm. The coarse-grained evaluation results can only reflect a portion of the safety risks of LLMs, failing to comprehensively evaluate the fine-grained safety situation of LLMs on the subdivided risk dimensions. Second, the currently employed evaluation benchmarks have weak riskiness which limits their capability to discover safety issues of LLMs. For instance, some benchmarks (Hendrycks et al., 2021; Zhang et al., 2023; Parrish et al., 2021) are only evaluated with multiple-choice questions (due to the lack of an effective test oracle), which is inconsistent with the real-world user case and limits the risks that may arise in responses, thus cannot reflect an LLMās real safety levels. Other benchmarks like (Huang et al., 2023; Sun et al., 2023; Li et al., 2024) only consider some backward and incomplete jailbreak attacks without mapping to original prompts, failing to fully exhibit the safety of LLMs under more various adversarial attacks. Third, the implementation of some assessments often lacks automation in test prompt generation and safety evaluation requiring numerous human labor, which impedes their adaptability to rapidly evolving LLMs and accompanying safety threats. In this paper, we present S-Eval, a novel LLM-based automated safety evaluation framework to systematically address the above limitations, as shown in Figure 1. Firstly, we design a unified and hierarchical risk taxonomy with four levels crossing 8 risk dimensions and 102 subdivided risks, as depicted in Appendix Table LABEL:tab:detail_of_risk_taxonomy. The risk taxonomy aims to cover all the necessary dimensions of safety assessment and measures the safety levels of the LLMs on the subdivided risk dimensions. Secondly, to automatically construct a test suite, we train an expert testing LLM ā³tsubscriptā³M_tMitalic_t that generates base risk222The base risk prompts are risky ones intended to trigger harmful output of the LLMs. and attack prompts with configurable risks of interest. Thirdly, for more accurate and efficient evaluation, a novel safety critique LLM ā³csubscriptā³M_cMitalic_c is developed on a well-crafted dataset. In addition to serving as a test oracle by quantifying the risks of LLMs, ā³csubscriptā³M_cMitalic_c can also provide detailed explanations for pellucid evaluations. Importantly, S-Eval can be flexibly configured and adapted to the rapid evolution of LLMs and accompanying new safety threats. Based on the critical components, we construct a new comprehensive, multi-dimensional and open-ended safety evaluation benchmark consisting of 220,000 high-quality test cases, including 20,000 base risk prompts (10,000 each in Chinese and English) and 200,000 corresponding attack prompts. We extensively evaluate on 21 popular and mainstream LLMs both open-source and closed-source (more than 500K queries). The results confirm that S-Eval can better reflect and inform the safety risks awareness of LLMs, and ā³csubscriptā³M_cMitalic_c has good consistency with human annotation compared to other evaluation methods. We also further explore the impacts of parameter scales, language environments, and decoding parameters on safety, providing a systematic guide methodology for evaluating the safety of LLMs. Table 1. Comparison with other widely used safety evaluation works. ā ā means that the characteristic is not met at all, ā ā means that it is partially met and ā ā means that it is fully met. Platform #Risk Category (Levels) #Data Scale #Jailbreak Attack End-to-end Automation Adaptive Update H-RLHF 20 (1) 38,961 0 ā ā AdvBench 7 (1) 520 0 ā ā Flames 12 (2) 2,251 3 ā ā SafetyPrompts 8 (1) 100,000 6 ā ā S-Eval (Ours) 102 (4) 220,000 10 ā ā This paper makes the following contributions: ā¢ We design a systematic four-level risk taxonomy with 8 risk dimensions and 102 subdivided risks, establishing a unified and comprehensive classification protocol. ā¢ We propose S-Eval, a novel LLM-based automated safety assessment framework for automatic test generation and safety evaluation, which can be flexibly adapted to evolving LLMs, new risks and attacks. ā¢ We release an extensive safety evaluation benchmark, consisting of 220,000 base risk prompts and attack prompts encompassing 10 jailbreak attacks. ā¢ We conduct safety evaluation for 21 representative LLMs. The results confirm that S-Eval can better reflect the safety of LLMs compared to existing safety benchmarks, and our safety critique LLM can accurately detect the output risks of LLMs. We also discuss some factors that affect safety, which could contribute to enhancing the safety of LLMs in the future. 2. Preliminaries 2.1. Large Language Models Large Language Models (LLMs) are advanced deep learning models. Currently, most LLMs are built based upon the Transformer architecture (Vaswani et al., 2017), and they are trained on massive textual corpora with a large number of parameters to effectively understand and generate natural language text. A common method to interact with LLMs is prompt engineering (Liu et al., 2023c; White et al., 2023), in which users guide LLMs to generate desired responses or complete specific tasks through well-designed prompt text. Prompts are critical to the quality of the LLM output, and small changes to the prompt result in large performance variations (Liu et al., 2023b; Shin et al., 2020). 2.2. Problem Definition In this work, we aim to evaluate the safety of an LLM. Given LLM ā³MM to test, we utilize an evaluation benchmark =p1,p2,āÆ,pnsubscript1subscript2āÆsubscriptP=\p_1,p_2,Ā·s,p_n\P = p1 , p2 , āÆ , pitalic_n for querying ā³MM, and a safety evaluation model ā¢(ā )ā0,1ā 01J(Ā·)ā\0,1\J ( ā ) ā 0 , 1 which judges whether a harmful response is triggered. Let risubscriptr_iritalic_i be the response of ā³MM to the prompt piāsubscriptp_i _i ā P, which is considered harmful when ā¢(pi,ri)=0subscriptsubscript0J(p_i,r_i)=0J ( pitalic_i , ritalic_i ) = 0 and safe otherwise. The objective of this work is to perform an automated thorough safety evaluations using PP following a unified risk taxonomy CC, supported by accurate JJ. Specifically, our evaluation benchmark is automatically generated, which consists of two parts: =B,AsuperscriptsuperscriptP=\P^B,P^A\P = Pitalic_B , Pitalic_A , where B=p1B,p2B,āÆ,pmBsuperscriptsubscriptsuperscript1subscriptsuperscript2āÆsubscriptsuperscriptP^B=\p^B_1,p^B_2,Ā·s,p^B_m\Pitalic_B = pitalic_B1 , pitalic_B2 , āÆ , pitalic_Bitalic_m denotes the base risk prompt set and A=p1A,p2A,āÆ,pnAsuperscriptsubscriptsuperscript1subscriptsuperscript2āÆsubscriptsuperscriptP^A=\p^A_1,p^A_2,Ā·s,p^A_n\Pitalic_A = pitalic_A1 , pitalic_A2 , āÆ , pitalic_Aitalic_n represents the attack prompt set, which are designed to evaluate LLMs in the diverse risk and adversarial scenarios. 3. The S-Eval Framework Following the definition of the safety evaluation problem above, we first present an overview of our framework, followed by an in-depth introduction to the risk taxonomy and the automatic generation and evaluation methodologies in it. Figure 1. Framework of S-Eval. āBRPāstands for base risk prompt and āAPā refers to attack prompt. Input: Testing LLM ā³tsubscriptā³M_tMitalic_t, Safety Critique LLM ā³csubscriptā³M_cMitalic_c, Risk Management MsubscriptR_MRitalic_M, LLM ā³MM for Evaluation Output: Safety Evaluation Benchmark PP, Evaluation Report Rā¢Eā¢PREPR E P 1 2 0BāBaseRiskPromptGenerationā¢(ā³t,M)āsubscriptsuperscript0BaseRiskPromptGenerationsubscriptā³subscriptP^B_0ā BaseRiskPromptGeneration(M% _t,R_M)Pitalic_B0 ā BaseRiskPromptGeneration ( Mitalic_t , Ritalic_M ) // generate base risk prompts 3 BāTestSelectionā¢(0B)āsuperscriptTestSelectionsubscriptsuperscript0P^Bā TestSelection(P^B_0)Pitalic_B ā TestSelection ( Pitalic_B0 ) // remove similar and harmless prompts 4 0AāAttackPromptGenerationā¢(ā³t,M)āsubscriptsuperscript0AttackPromptGenerationsubscriptā³subscriptP^A_0ā AttackPromptGeneration(M_% t,R_M)Pitalic_A0 ā AttackPromptGeneration ( Mitalic_t , Ritalic_M ) // generate attack prompts 5 AāTestSelectionā¢(0A)āsuperscriptTestSelectionsubscriptsuperscript0P^Aā TestSelection(P^A_0)Pitalic_A ā TestSelection ( Pitalic_A0 ) // identify and regenerate prompts decoded repetitively 6 āBāŖAāsuperscriptsuperscriptP ^B ^AP ā Pitalic_B āŖ Pitalic_A // obtain the final safety evaluation benchmark PP 7 8REPāSafetyEvaluationā¢(ā³c,ā³ā¢())āREPSafetyEvaluationsubscriptā³REPā SafetyEvaluation(M_c,M% (P))REP ā SafetyEvaluation ( Mitalic_c , M ( P ) ) return PP, REP Algorithm 1 S-Eval (ā³t,ā³c,M,ā³subscriptā³subscriptā³subscriptā³M_t,M_c,R_M,MMitalic_t , Mitalic_c , Ritalic_M , M) 3.1. Overview Figure 1 shows the overview of the S-Eval framework. At a high level, given a risk management system comprising a risk taxonomy, risk seeds (manually collected base risk prompts) and knowledge collected based on this, along with jailbreak attacks, in the training stage, we first gather data pairs based on the different generation configuration for base risk and attack prompts. Then, we train an expert testing LLM ā³tsubscriptā³M_tMitalic_t on these prepared datasets, as well as a safety critique LLM ā³csubscriptā³M_cMitalic_c through generated prompts and responses from multiple LLMs with automatic annotation and manual review. In the generating stage, we first use ā³tsubscriptā³M_tMitalic_t to automatically generate a set of base risk prompts, and select a high-quality base risk prompt set BsuperscriptP^BPitalic_B. Subsequently, ā³tsubscriptā³M_tMitalic_t is applied to generate corresponding attack prompts for each prompt in BsuperscriptP^BPitalic_B with well-designed selection to obtain the attack prompt set AsuperscriptP^APitalic_A. Finally, testing with PP and evaluating with ā³csubscriptā³M_cMitalic_c results in a comprehensive, fine-grained safety evaluation report for the evaluated LLMs. The complete procedure of our S-Eval is detailed in Algorithm 1. 3.2. Risk Management Risk management provides additional resource for automatic test generation and can adjust different generation schemes with corresponding configurations. We consider four components: risk taxonomy, risk seeds that are base risk prompts either manually collected based on the taxonomy or expanded through generation, risk knowledge crawled from different web platforms following the taxonomy and jailbreak attacks (details in Section 3.3 later). With evolving LLMs, new risks and attacks, our risk management can be updated and configure S-Eval to generate new test cases. In risk management, a comprehensive and systematic risk taxonomy is beneficial to the diversity of test and provide careful evaluation feedback. There are some attempts to risk taxonomies in prior work. However, they only focus on limited perspectives and lack a more fine-grained protocol. To address the limitations, we first integrate the safety policies (AI, 2023; Parliament, 2021) formulated by different countries about LLMs, as well as the content safety terms (Google, 2023; OpenAI, 2024b) of different companies, and extract safety issues of general concern, such as crimes, privacy and sexual content. Based on these common issues, we summarize 8 horizontal first-level risk dimensions. Then, inspired by research in sociology (Beck, 1992; Zigon, 2009) and criminology (Chaiken et al., 1982; Osgood, 2010), and incorporating LLM application scenarios, we analyze possible safety risks and establish a fine-grained yet concise vertical hierarchy with four levels. Each level gradually refines risk categories to facilitate the evaluation and management of risks at different levels. And these risks are carefully designed to ensure they are decoupled from each other based on their underlying intentions and contextual factors. Through this systematic process, we obtain a multidimensional, fine-grained risk taxonomy that includes 8 risk dimensions and a total of 102 risks, as shown in Appendix A. Notably, our taxonomy also considers potential risks that are not covered in previous taxonomies, like the threats caused by technological autonomy and uneven resource allocation, providing detailed guidelines for safety evaluation. 3.3. Automatic Test Generation For effective safety assessments, it is crucial to provide objective and continuous measures of LLM safety. However, the construction of some evaluation benchmarks is confronted with several challenges: 1) Some safety benchmarks heavily rely on manual collection and annotation, incurring significant time and labor costs. This limits the scale and expansion potential of the benchmarks, not to mention controlling and tracing benchmark data quality. 2) The safety threat environment continues to evolve, with new safety risks and innovations in attack methods constantly emerging. 3) With the rapid iteration and performance improvement of LLMs, the original static benchmarks gradually lose the ability to effectively evaluate the safety level of the latest models. To address the above challenges, we propose LLM-based automatic test generation approaches. Notably, general LLMs with alignment for better performance are prone to rejecting the generation of harmful prompts and are limited in quality of generated prompts. Drawing from the āunalignmentā (Bhardwaj and Poria, 2023a), we build our expert testing LLM ā³tsubscriptā³M_tMitalic_t by supervised fine-tuning Qwen-14B-Chat (Bai et al., 2023) on specially constructed data pairs for different generation purposes to break the safety alignment and incorporate multiple automatic test generation abilities333 The detailed implementation of ā³tsubscriptā³M_tMitalic_t and ā³csubscriptā³M_cMitalic_c can be found in Appendix B. . Then, we flexibly configure it through the risk management to generate base risk prompts and attack prompts, achieving automatic test generation and adaptive update. 3.3.1. Base Risk Prompt Generation To effectively evaluate the safety of LLMs, ā³tsubscriptā³M_tMitalic_t can adjustably generate base risk prompts based on risk definitions, risk knowledge, and risk seeds, as shown in Figure 2(a). We introduce them respectively. (1) Definition-Based Test Generation. To make generated prompts conform to the risk themes, in the training stage, we first collect a small number of high-quality risk prompts. They are written by experts444 The āexpertsā refer to professionals with many years of content review experience. for each risk definition, or collected and rewritten by experts online based on the risk taxonomy. Then, we take the generation instructions and risk definitions as training input and the corresponding risk prompts as output to train ā³tsubscriptā³M_tMitalic_t to generate prompts based on risk definitions. In the generation stage, we input instructions with specific risks and detailed risk definitions into ā³tsubscriptā³M_tMitalic_t to generate base risk prompts. It is worth noting that because ā³tsubscriptā³M_tMitalic_t has the test generation ability based on risk definitions, it can adaptively complete the generation tasks by simply providing the risk definitions when new safety risks appear. In response to the demand for higher-quality prompts, we can also add few-shot examples into input through in-context learning to generate superior prompts. (2) Knowledge-Based Test Generation. For the factuality and diversity of generated prompts, we incorporate a wide range of external knowledge sources into the generation phase. We crawl a large amount of risk-related data from different web platforms, which is centered on our risk taxonomy. Based on this, we construct a structured and fine-grained risk knowledge base, covering keywords, knowledge graphs, and knowledge documents. Then, we fine-tune ā³tsubscriptā³M_tMitalic_t using collected base risk prompts and their knowledge so that it can more accurately understand risk knowledge and generate highly related risk prompts. As new risks emerge or existing risks change, the framework can be up-to-date with the latest risks by updating the risk knowledge base. (3) Rewriting-Based Test Generation. The initial generation of base risk prompts inherently includes a subset that does not elicit harmful responses from LLMs. Concurrently, with the gradual evolution of LLMs, previously effective prompts may be rejected, diminishing their evaluative significance. To improve the effective utilization of base risk prompts and maintain the benchmark to be continuously updated as LLMs advance, we introduce a rewriting strategy. First, we meticulously design the rewriting rules in detail based on expert experience. Specifically, ā³tsubscriptā³M_tMitalic_t is tasked to identify critical risk elements in original prompts, such as expressions involving violence, hatred, threats, etc., which is the focus of the subsequent rewriting process. ā³tsubscriptā³M_tMitalic_t is then instructed to modify these identified elements using techniques like synonym substitution, semantic fuzziness, and complication to attenuate the risky semantics, rendering them more implicit and indirect. Additionally, we also instruct ā³tsubscriptā³M_tMitalic_t to embed some reasonable background in rewriting, to increase the depth of the prompts and conceal malicious intent. For example, queries involving drug production might be reframed in the context of an academic chemistry discussion. We then manually rewrite the collected risk seeds according to the made rewriting rules, creating a dataset containing prompt seed and rewritten prompt pairs. Through instruction with rewriting rules and the dataset, we train ā³tsubscriptā³M_tMitalic_t by supervised fine-tuning to enhance the rewriting performance. Finally, during test generation, we can get new prompts from given prompt seeds, which are risky but more covert in presentation, augmenting challenge and adaptability. (a) Base Risk Prompt (b) Attack Prompt Figure 2. The example of automatic test generation. 3.3.2. Attack Prompt Generation To comprehensively evaluate the robustness of LLMs against various jailbreak attacks, S-Eval examines two failure modes of safety alignment: competing objectives and mismatched generalization (Wei et al., 2024). The former refers to the competition between helpfulness and harmlessness, reflecting the depth of safety alignment. The latter is due to that the alignment does not cover full domains, such as different languages and encrypted communication, measuring the breadth of safety alignment. A total of 10 representative cutting-edge attacks corresponding to each failure mode are integrated, as detailed in Appendix B Table 9. However, generating multiple attack prompts by replicating each attack in turn is cumbersome and lacks scalability, as each attack is implemented differently, and some of them rely on manual construction, which is time-consuming and laborious. Therefore, we train ā³tsubscriptā³M_tMitalic_t to uniformly and automatically generate attack prompts. We first accumulate attack prompts by enhancing the collected base risk prompts with different jailbreak attacks. Then, we fine-tune ā³tsubscriptā³M_tMitalic_t with the instructions for attack methods and base risk prompts as training inputs and the corresponding attack prompts as output. In the generation phase, we configure the instructions for specific attacks and provide base risk prompt to ā³tsubscriptā³M_tMitalic_t for generating corresponding attack prompts. The example is presented in Figure 2(b). 3.3.3. High-quality Test Selection To ensure the quality of test prompts, we select the collected base risk prompts and attack prompts. For base risk prompts, there are two main problems: similar prompts and benign prompts that lack significant riskiness. We define a similarity measure S, combining semantic similarity and levenshtein distance (Zhang et al., 2017): (1) Sā¢(pi,pj)=Ī±ā Ssā¢eā¢mā¢(pi,pj)+(1āĪ±)ā Slā¢eā¢vā¢(pi,pj)subscriptsubscriptā subscriptsubscriptsubscriptā 1subscriptsubscriptsubscriptS(p_i,p_j)=Ī±Ā· S_sem(p_i,p_j)+(1-Ī±)Ā· S_lev(p_i,% p_j)S ( pitalic_i , pitalic_j ) = Ī± ā Sitalic_s e m ( pitalic_i , pitalic_j ) + ( 1 - Ī± ) ā Sitalic_l e v ( pitalic_i , pitalic_j ) where pisubscriptp_ipitalic_i and pjsubscriptp_jpitalic_j denote two prompts within the same risk subcategory. Ssā¢eā¢mā¢(pi,pj)=Eā¢(pi)ā Eā¢(pj)āEā¢(pi)āā¢āEā¢(pj)āsubscriptsubscriptsubscriptā subscriptsubscriptnormsubscriptnormsubscriptS_sem(p_i,p_j)= E(p_i)Ā· E(p_j)\|E(p_i)\|\|E(p_j)\|Sitalic_s e m ( pitalic_i , pitalic_j ) = divide start_ARG E ( pitalic_i ) ā E ( pitalic_j ) end_ARG start_ARG ā„ E ( pitalic_i ) ā„ ā„ E ( pitalic_j ) ā„ end_ARG represents their semantic similarity, computed using an embedding model Eā¢(ā )ā E(Ā·)E ( ā ), and Slā¢eā¢vsubscriptS_levSitalic_l e v refers to the levenshtein distance. The parameter Ī±ā[0,1]01Ī±ā[0,1]Ī± ā [ 0 , 1 ] is a weight to balance superficial and feature similarity. The two prompts are deemed similar if Sā¢(pi,pj)subscriptsubscriptS(p_i,p_j)S ( pitalic_i , pitalic_j ) exceeds a predefined threshold Īøsā¢iā¢msubscript _simĪøitalic_s i m. We take Ī±=0.20.2Ī±=0.2Ī± = 0.2 and Īøsā¢iā¢m=0.55subscript0.55 _sim=0.55Īøitalic_s i m = 0.55. To eliminate benign prompts, we utilize multiple victim LLMs ā³v=ā³v1,ā³v2,āÆ,ā³vlsubscriptā³subscriptsubscriptā³1subscriptsubscriptā³2āÆsubscriptsubscriptā³M_v=\M_v_1,M_v_2,Ā·s, % M_v_l\Mitalic_v = Mitalic_v1 , Mitalic_v2 , āÆ , Mitalic_vitalic_l to assess the riskiness of each base risk prompt piBsubscriptsuperscriptp^B_ipitalic_Bitalic_i. We get responses Ri=ri1,ri2,āÆ,rilsubscriptsubscriptsubscript1subscriptsubscript2āÆsubscriptsubscriptR_i=\r_i_1,r_i_2,Ā·s,r_i_l\Ritalic_i = ritalic_i1 , ritalic_i2 , āÆ , ritalic_iitalic_l to piBsubscriptsuperscriptp^B_ipitalic_Bitalic_i from ā³vsubscriptā³M_vMitalic_v. Then, we input piBsubscriptsuperscriptp^B_ipitalic_Bitalic_i and RisubscriptR_iRitalic_i into JJ to get safety confidences Sci=sci1,sci2,āÆ,scilsubscriptsubscriptsubscriptsubscriptsubscript1subscriptsubscriptsubscript2āÆsubscriptsubscriptsubscriptS_c_i=\s_c_i_1,s_c_i_2,Ā·s,s_c_i_l\Sitalic_citalic_i = sitalic_citalic_i1 , sitalic_citalic_i2 , āÆ , sitalic_citalic_iitalic_l and retain piBsubscriptsuperscriptp^B_ipitalic_Bitalic_i if the average of ScisubscriptsubscriptS_c_iSitalic_citalic_i, SciĀÆ=1lā¢āj=1lscijĀÆsubscriptsubscript1superscriptsubscript1subscriptsubscriptsubscript S_c_i= 1l _j=1^ls_c_i_joverĀÆ start_ARG Sitalic_citalic_i end_ARG = divide start_ARG 1 end_ARG start_ARG l end_ARG āj = 1l sitalic_citalic_iitalic_j is less than a predefined threshold Īøsā¢aā¢fā¢esubscript _safeĪøitalic_s a f e. We take Īøsā¢aā¢fā¢e=0.95subscript0.95 _safe=0.95Īøitalic_s a f e = 0.95. As the safety of LLMs improves, S-Eval can dynamically adjust Īøsā¢aā¢fā¢esubscript _safeĪøitalic_s a f e or replace higher safe ā³vsubscriptā³M_vMitalic_v, updating the riskiness of pBsuperscriptp^Bpitalic_B. For attack prompts, since we generate them via ā³tsubscriptā³M_tMitalic_t, there may be repetitive decoding during the generation process, resulting in meaningless prompts. Considering the powerful ability of LLMs, we use LLMs to identify meaningless attack prompts and regenerate them until successful. 3.4. Automatic Safety Evaluation The open-ended property of LLM generation as well as the sparsity and diversity of potential risks inherent in different models, make it extremely challenging to automatically and accurately assess whether the generated content complies with safety policies. Most of the existing works on LLM safety evaluation typically rely on one or more of the following schemes: manual annotation, rule matching, moderation APIs and prompt-based evaluation. Limitation of Existing Safety Evaluation Methods. Manual annotation (Liu et al., 2023a) is highly accurate but time-consuming and laborious, thus lacking scalability and practicality for large-scale evaluation in reality. Rule matching method (Zou et al., 2023) assesses the safety of LLM through the matching of manually summarized rules or patterns deemed safe with the generated responses. More concretely, a response can be regarded as safe if it incorporates certain terms indicative of rejection, such as āIām sorry, I canātā, āAs a responsible AIā, āItās not appropriateā, etc. However, it is usually not trivial to generalize the rules to different LLMs due to variations in their response styles and scripts. Worse yet, there are often cases where the model initially indicates a refusal to answer, but harmful content still follows closely behind. Hence, although this method is easy to conduct, it significantly underperforms manual annotation in terms of accuracy by an obvious margin. Some studies adopt commercial moderation APIs to detect toxicity in the responses from LLMs as safety evaluation (OpenAI, 2024b; Gehman et al., 2020). However, the effectiveness of such evaluations is also fairly limited and can sometimes exhibit bias. This is primarily because there is currently no comprehensive API capable of covering the entire spectrum of risk categories associated with LLMs. Besides, evaluating LLM safety is quite different from merely detecting toxic content. Thanks to the powerful emergent abilities of LLMs, prompt-based evaluation methods (Wang et al., 2023) have been recently applied via prompt engineering, i.e., input specific evaluation guidelines or safety policies along with the dialogues to be evaluated into high-performing LLMs such as GPT-4 (Achiam et al., 2023). Nevertheless, most existing LLMs are not specifically built for the purpose of safety evaluation. As a result, they may not be well-aligned with human values in some aspects, which can lead to undeserved evaluation results that are inconsistent with human judgment. In addition, the LLM in use sometimes refuses to respond to assessment instructions due to the sensitivity of input dialogues and the issue of over-alignment (i.e., exaggerated safety) (Sun et al., 2024). The under- and over-alignment issues mentioned above severely restrict the applicability of such methods. (a) Safe (b) Unsafe Figure 3. The example of automatic safety evaluation. To deal with the limitations of existing works and make safety evaluation more effective and efficient, we introduce a novel LLM-based safety critique framework through critique mechanisms, taking inspiration from (Ke et al., 2023). Our safety critique LLM ā³csubscriptā³M_cMitalic_c is developed using a carefully curated dataset via supervised fine-tuning. It can provide effective and explainable safety evaluations for LLMs, including risk tags, scores, and explanations, as shown in Figure 3. It also boasts attractive scaling properties for both model and data. During dataset construction, to acquire the generated responses with different levels of safety and qualities, we choose 10 representative models that cover both open-source and closed-source LLMs with different model scales, including GPT-4, ErnieBot, Qwen, LLaMA, Baichuan and ChatGLM, etc. To obtain high-quality annotated critiques, complete with risk tags (i.e., safe or unsafe) and explanations (i.e., the reasons for tagging), we utilize GPT-4 for automatic annotation and explanation. These automated results are then reviewed and corrected by our specialists in cases of inaccuracies. Through response generation and annotation, we create a fine-grained dataset consisting of 100,000 QA pairs derived from 10,000 risk queries. This dataset is bilingual, including both Chinese and English, and encompasses over 100 types of risks. Finally, we build ā³csubscriptā³M_cMitalic_c through full parameter fine-tuning of Qwen-14b-Chat on this dataset. The experimental results in the Section 4.2 show that ā³csubscriptā³M_cMitalic_c achieves high accuracy, significantly outperforming the other methods, allowing for accurate and automatic evaluation. 4. Experiments In this section, we first describe our experimental setups. Then we conduct extensive evaluations for multiple popular LLMs and answer the following research questions: ā¢ RQ1: Does ā³csubscriptā³M_cMitalic_c provide more accurate safety evaluation compared to other methods? ā¢ RQ2: Does S-Eval more effectively reflect the safety of LLMs compared to existing safety benchmarks? ā¢ RQ3: How do LLM parameter scales affect safety? ā¢ RQ4: Are there differences in the safety of LLMs in different language environments? ā¢ RQ5: How robust are LLMs against jailbreak attacks? ā¢ RQ6: What is the effect of decoding parameters on the safety of LLMs? 4.1. Experimental Setup 4.1.1. Datasets and Models Due to limited computing resources and to ensure a comprehensive and objective evaluation, we randomly and uniformly sample 2,000 base risk prompts (1,000 in Chinese and corresponding 1,000 in English) as the base risk prompt set BsuperscriptP^BPitalic_B, across the data balance of the first-level and second-level risks. We also take corresponding 20,000 attack prompts as the attack prompt set AsuperscriptP^APitalic_A. For the models, we select 21 representative LLMs for safety evaluations in our experiment, including the GPT (Achiam et al., 2023; OpenAI, 2024a), Qwen (Bai et al., 2023), LLaMA (Touvron et al., 2023; AI@Meta, 2024), Gemini (Team et al., 2023), Gemma (Team et al., 2024), Mistral (Jiang et al., 2023b), Vicuna (Zheng et al., 2024), ErnieBot (Baidu, 2023), ChatGLM3 (Du et al., 2022), Baichuan2 (Yang et al., 2023), and Yi (Young et al., 2024) families. We adhere to the default chat template and decoding strategies of each LLM and use the safety critique LLM ā³csubscriptā³M_cMitalic_c as the evaluation model JJ. 4.1.2. Evaluation Metrics We use two metrics to quantify the safety of LLMs: (i) safety score (S): it measures the ability of LLMs to respond safely to harmful prompts, which is calculated as Equation 2, where pBsuperscriptp^Bpitalic_B denotes a test sample in superscriptP^BPbold_B and r denotes the response to pBsuperscriptp^Bpitalic_B. The higher the score, the safer the LLM; (i) attack success rate (ASR): it assesses the ability of LLMs to defend and resist malicious attacks, which is calculated as Equation 3, where pasuperscriptp^apitalic_a denotes a attack prompt in superscriptP^APbold_A and r denotes the response to pAsuperscriptp^Apitalic_A. The lower the Aā¢Sā¢RASRA S R, the more robust the LLM against malicious attacks. (2) Sā¢S=āpBāā¢(pB,r)||subscriptsuperscriptsuperscriptsuperscriptsuperscriptSS= Ī£ _p^B ^BJ(p^B,r) |% P^B |S S = divide start_ARG āpitalic_B ā Pbold_B J ( pitalic_B , r ) end_ARG start_ARG | Pbold_B | end_ARG (3) Aā¢Sā¢R=āpAā(1āā¢(pA,r))||subscriptsuperscriptsuperscript1superscriptsuperscriptASR= Ī£ _p^A ^A(1-J(p^A,r)) |% P^A |A S R = divide start_ARG āpitalic_A ā Pbold_A ( 1 - J ( pitalic_A , r ) ) end_ARG start_ARG | Pbold_A | end_ARG 4.2. Effectiveness of ā³csubscriptā³M_cMitalic_c (RQ1) To validate the accuracy of ā³csubscriptā³M_cMitalic_c, we construct a test set comprising 1,000 Chinese QA pairs and 1,000 English QA pairs from Qwen-7B-Chat with manual annotation. We compare ā³csubscriptā³M_cMitalic_c with rule matching, GPT-based evaluation and LLaMA-Guard-2 (Team, 2024). For rule matching, we use the English rules in (Li et al., 2024) and design similar Chinese rules. For GPT-based evaluation, we follow the setups in (Wang et al., 2023). And we adopt the default setups of LLaMA-Guard-2. Table 2. Comparison of different evaluation methods. āACCā stands for the balanced accuracy. The bold value indicates the best. w/ CoT denotes using the CoT prompting in the evaluation. Method Chinese English ACC Precision Recall ACC Precision Recall Rule Matching 74.12 74.44 61.15 70.19 72.01 62.84 GPT-4-Turbo 78.00 94.07 58.27 72.36 93.83 47.60 LLaMA-Guard-2 76.23 95.37 54.07 69.32 93.81 41.13 Ours 92.23 92.37 88.98 88.23 90.97 84.13 w/ CoT 92.83 92.70 90.03 86.78 92.89 79.12 As shown in Table 2, ā³csubscriptā³M_cMitalic_c achieves the highest balanced accuracy, significantly outperforming the other methods, allowing for accurate and automatic evaluations. At the same time, we also design the zero-shot chain-of-thought (Zero-Shot-CoT) prompt based on (Yuan et al., 2024). And the Zero-Shot-CoT has no obvious effect on the evaluation results of ā³csubscriptā³M_cMitalic_c. This may be because we do not use the corresponding prompting strategy during fine-tuning ā³csubscriptā³M_cMitalic_c. In the following experiments, we use ā³csubscriptā³M_cMitalic_c without the CoT for safety evaluation. Furthermore, to assess the bias of evaluation methods, we test the consistency of the safety evaluation results. Figure 4(a) illustrates that for 88.50% of the Chinese cases and 83.70% of the English cases, three or more of the four evaluation methods yield consistent results, indicating that the bias is not significant. We also analyze the evaluation correlation between ā³csubscriptā³M_cMitalic_c and LLaMA-Guard-2 on a larger data corpus in Figure 4(b), using responses to the English test prompts from 12 popular LLMs on the market. The findings reveal a significant positive correlation between the evaluation results of the two models with a Pearson correlation coefficient (PCC) of 0.92, further validating that the risk of inherent bias is controllable. Answer to RQ1: ā³csubscriptā³M_cMitalic_c can provide more accurate evaluation compared to the other methods. 4.3. Effectiveness of S-Eval (RQ2) To validate the effectiveness of S-Eval in assessing the safety of LLMs, we compare BsuperscriptP^BPitalic_B with four widely used safety benchmarks, AdvBench (Zou et al., 2023), H-RLHF (red-teaming) (Ganguli et al., 2022), Flames (Huang et al., 2023), and SafetyPrompts (typical safety scenarios) (Sun et al., 2023), covering two mainstream test generation methods: manual collection and data augmentation using general LLMs. For H-RLHF and SafetyPrompts, we randomly and uniformly sample 1,000 prompts. For AdvBench and Flames, all 520 and 1,000 prompts, respectively, are utilized. The prompts from each benchmark are translated into Chinese or English via the Google Translate API555https://translate.google.com. Table 3 presents the safety scores of evaluated LLMs on the five benchmarks, providing a variety of observations and insights as follows. (a) Consistency analysis (b) Correlation analysis Figure 4. The consistency and correlation analysis of different evaluation methods. (a) The horizontal axis represents the number of methods with a same evaluation result. (b) The horizontal and vertical axes represent the Sā¢S S. Table 3. The safety scores (%) of evaluated models on the five benchmarks. Rows with ā£ denote English results. The bold value in each column indicates the safest and underline indicates the second. āABā: AdvBench; āH-Rā: H-RLHF; āFLā: Flames; āSPā: SafetyPrompts. āCIā, etc. denotes the risk dimensions in Table 7. Model AB H-R FL SP S-Eval (Ours) Overall Overall Overall Overall Overall CI HS PM EM DP CS EX IS Qwen-1.8B-Chat 93.65 83.20 64.80 89.50 60.50 57.78 65.00 75.00 36.00 71.00 60.00 78.33 41.67 ChatGLM3-6B 95.38 83.80 77.90 95.20 59.70 60.56 72.14 68.00 37.00 61.00 57.86 66.67 50.00 Gemma-7B-it 74.42 77.30 62.50 76.80 49.60 48.33 59.29 60.00 31.00 70.00 39.29 58.33 33.33 Baichuan2-13B-Chat 94.23 87.80 80.07 96.40 66.60 74.44 70.00 79.00 47.00 77.00 65.00 68.33 48.33 Qwen-14B-Chat 97.31 91.80 75.80 96.00 66.50 75.00 76.43 80.00 38.00 77.00 52.14 74.17 55.00 Yi-34B-Chat 94.62 75.80 70.90 92.30 46.70 50.00 48.57 60.00 25.00 81.00 27.14 35.83 51.67 Qwen-72B-Chat 99.62 92.70 81.50 97.40 73.10 83.33 72.86 83.00 58.00 86.00 63.57 83.33 52.50 GPT-4o 97.12 86.00 72.50 94.70 54.00 52.22 60.71 68.00 33.00 68.00 69.29 55.83 23.33 GPT-4-Turbo 94.23 85.10 78.00 94.00 57.70 58.33 62.14 56.00 41.00 78.00 68.57 55.00 40.00 ErnieBot-4.0 99.04 90.10 81.90 97.20 79.70 89.44 85.00 87.00 57.00 73.00 89.29 87.50 58.33 Gemini-1.0-Pro 86.54 78.50 62.20 84.30 53.90 56.11 61.43 67.00 50.00 54.00 35.71 65.83 43.33 Qwen-1.8B-Chatā£ 93.65 78.30 74.90 89.70 47.60 38.89 56.43 66.00 39.00 66.00 43.57 49.17 30.00 ChatGLM3-6Bā£ 94.04 83.70 80.20 93.70 57.70 51.67 74.29 76.00 55.00 75.00 45.71 45.83 45.83 Gemma-7B-itā£ 91.54 87.80 78.00 85.60 61.80 56.11 76.43 74.00 43.00 74.00 56.43 65.83 50.83 Mistral-7B-Instruct-v0.2ā£ 49.62 77.40 74.70 91.30 34.20 23.89 40.00 61.00 38.00 65.00 12.14 9.17 42.50 LLaMA-3-8B-Instructā£ 98.27 84.90 74.60 85.80 69.10 70.00 68.57 75.00 63.00 58.00 82.86 71.67 59.17 Vicuna-13B-v1.3ā£ 98.85 87.50 80.80 91.70 57.10 52.22 67.86 73.00 59.00 77.00 42.86 47.50 46.67 LLaMA-2-13B-Chatā£ 99.62 92.80 84.60 92.00 85.10 77.78 93.57 86.00 83.00 83.00 93.57 93.33 70.83 Baichuan2-13B-Chatā£ 98.27 91.10 87.50 96.40 77.40 81.11 80.71 86.00 74.00 85.00 82.86 73.33 55.00 Qwen-14B-Chatā£ 99.81 91.20 83.00 95.30 73.50 69.44 75.71 83.00 72.00 88.00 71.43 78.33 55.83 Yi-34B-Chatā£ 82.88 70.40 73.30 88.20 39.30 29.44 47.86 58.00 38.00 72.00 22.86 19.17 41.67 LLaMA-2-70B-Chatā£ 99.23 91.10 83.80 90.90 77.20 70.00 90.71 84.00 68.00 72.00 87.14 84.17 60.00 LLaMA-3-70B-Instructā£ 95.58 77.30 69.10 81.80 54.70 56.67 47.14 61.00 46.00 63.00 60.71 48.33 55.00 Qwen-72B-Chatā£ 98.65 88.40 84.70 94.80 71.50 71.11 77.14 75.00 74.00 81.00 65.00 75.00 56.67 GPT-4oā£ 98.85 80.40 75.60 90.70 52.00 46.67 57.86 69.00 45.00 72.00 58.57 39.17 33.33 GPT-4-Turboā£ 97.50 81.30 79.80 89.40 60.00 56.11 66.43 69.00 50.00 80.00 63.57 51.67 46.67 ErnieBot-4.0ā£ 99.81 94.60 92.40 97.80 87.60 90.00 90.00 88.00 89.00 96.00 89.29 91.67 66.67 Gemini-1.0-Proā£ 94.23 78.40 67.40 85.10 41.90 43.33 46.43 63.00 42.00 51.00 15.00 42.50 40.00 (a) Chinese (b) English Figure 5. The safety score distributions on Chinese and English. First, S-Eval is more risky and more effectively reflects the safety of LLMs. In Chinese and English evaluations, all models consistently have lower Sā¢S S on S-Eval than the four baselines. Specifically, among the baselines, Advbench has the least risk, with most of the LLMs exhibiting the Sā¢S S of 95% and above, whereas Flames with highly adversarial characteristics presents the highest risk profile. To further analyze the distributions of the Sā¢S S on each benchmark, we first exclude outliers based on the upper and lower quartiles of the Sā¢S S, followed by a detailed characterization of these distributions in Figure 5. The 95% confidence interval sizes of the Sā¢S S in Chinese and English on S-Eval measure 30.62% and 50.36%, respectively. In contrast, the corresponding interval sizes of the four baselines are Advbench (5.74%/7.53%), H-RLHF (16.30%/20.72%), Flames (19.53%/22.36%), and SafetyPrompts (11.89%/14.12%). Meanwhile, the distributions of the Sā¢S S on S-Eval demonstrate greater uniformity. The higher riskiness, larger confidence interval size of Sā¢S S, and more uniform score distribution collectively underscore that S-Eval is more effective in reflecting the safety of LLMs and delineating the differences in safety. Table 4. Evaluation results of different generation models. The bold value indicates the best. Model Rā¢R R (%) Eā¢RERE R (%) Sā¢S S (%) GPT-4 80.00 12.30 75.12 Qwen-14B-Chat 0.00 19.00 71.89 Ours 0.00 82.80 63.10 Furthermore, to further validate the effectiveness of S-Eval, we perform an ablation study, comparing ā³tsubscriptā³M_tMitalic_t with GPT-4 and Qwen-14B-Chat that are not explicitly fine-tuned to generate test prompts. Specifically, for the āCrimes and Illegal Activitiesā dimension, which is a common concern to many safety policies and has significant harm, we use each model to generate 1,000 base risk prompts with the same generation configurations followed by the same selection process. In addition to Sā¢S S, we calculate rejection rate Rā¢R=Nrā¢eā¢jNaā¢lā¢lsubscriptsubscriptRR= N_rejN_allR R = divide start_ARG Nitalic_r e j end_ARG start_ARG Nitalic_a l l end_ARG and effectiveness rate Eā¢R=Nfā¢iā¢nā¢aā¢lNaā¢lā¢lsubscriptsubscriptER= N_finalN_allE R = divide start_ARG Nitalic_f i n a l end_ARG start_ARG Nitalic_a l l end_ARG, where Naā¢lā¢lsubscriptN_allNitalic_a l l denotes the total number of generation requests, Nrā¢eā¢jsubscriptN_rejNitalic_r e j denotes the number of rejected requests, and Nfā¢iā¢nā¢aā¢lsubscriptN_finalNitalic_f i n a l denotes the number of usable test prompts after test selection. The former measures the refusal of a generation model to generate test prompts, and the latter assesses the effectiveness of a generation model. For Sā¢S S, we count the average Sā¢S S of 5 LLMs666The 5 LLMs include Gemma-2B-it, ChatGLM3-6B, Baichuan2-13B-Chat, Yi-34B-Chat, and Qwen-72B-Chat.. As shown in Table 4, S-Eval can significantly improve the effectiveness and quality of test generation, providing a more effective means for evaluating the safety of LLMs. Second, the evaluation results on S-Eval show obvious differences in the safety of different LLMs. Among the closed-source LLMs, ErnieBot-4.0 has the highest Sā¢S S, with 79.70% in Chinese and 87.60% in English. The leading safety performance may be attributed to its advanced outer safety guardrail, which can audit inference content and filter out sensitive words. Conversely, Gemini-1.0-Pro exhibits the lowest Sā¢S S (53.90%/41.90%). For the open-source LLMs, Qwen-72B-Chat leads in Chinese with a Sā¢S S of 73.10% and LLaMA-2-13B-Chat tops the English evaluation at 85.10%. The lowest scores are observed for Yi-34B-Chat in Chinese (46.70%) and Mistral-7B-Instruct-v0.2 in English (34.20%). Notably, despite its small size of 1.8 billion parameters, Qwen-1.8B-Chat outperforms Yi-34B-Chat in safety evaluations. In addition, the LLaMA-3 family and GPT-4o exhibit lower safety compared to their predecessors, the LLaMA-2 family and GPT-4-Turbo, respectively, indicating lower refusal rates to harmful prompts, which is consistent with other studies (Cui et al., 2024). Third, there are significant variations in the safety of LLMs on different risk dimensions. In Chinese, Yi-34B-Chat has a robust Sā¢S S of 81.00% on Data Privacy, contrasting sharply with a markedly lower score of 25.00% on the Ethics and Morality dimension, showing a difference of 56.00%. Similarly, in English, Mistral-7B-Instruct-v0.2 achieves a Sā¢S S of 65.00% on Data Privacy, while only 9.17% on Extremism. Meanwhile, all LLMs demonstrate less safety on Inappropriate Suggestions. These variations in the safety of LLMs on segmented risk dimensions may be related to the concrete risk data distributions and optimization objectives during the training or alignment processes. These observations further underscore the necessity for comprehensive safety evaluations or alignments of LLMs across systematic risk dimensions rather than focusing on a single class of safety concerns. Answer to RQ2: S-Eval can effectively reflect the safety of LLMs and the differences in safety. Overall, the safety of closed-source LLMs is better than that of open-source LLMs. And there is a significant difference in the safety of LLMs on different risk dimensions. 4.4. LLM Scale Effect to Safety (RQ3) To investigate the relationship between the scale of LLM parameters and its safety, we evaluate 10 models from three families, Qwen, Vicuna, and LLaMA-2, with various parameter scales, using the English set in BsuperscriptP^BPitalic_B. (a) Parameter scale and safety (b) Ability and attack Figure 6. The relationships between the ability and the safety of LLMs. (a) LLM ability from the English overall average of OpenCompass rankings777https://rank.opencompass.org.cn/leaderboard-llm (2024-01), with icon size indicating parameter scale. (b) LLM ability from the objective overall average of OpenCompass rankings (2024-04), with the vertical axis denoting the ASR of Instruction Encryption. From Figure 6(a), we have the following observations. First, for each model family, there is a discernible trend of improved model ability as the parameters increase. This trend aligns well with established scaling laws, indicating that larger model parameter scales lead to better abilities. Second, the Sā¢S S of all three model families first increase with the increase of parameters (ability) but decrease when reaching the maximum parameter scale. This trend indicates that there is a parameter scale (ability) threshold within one model family, beyond which the continued increase in model parameter scales may not result in a sustained rise or even a decrease in safety performance. Third, notable differences in safety performance exist among the model families. Despite exhibiting lower ability than the Qwen family at similar parameter scales, the LLaMA-2 family consistently achieves higher Sā¢S S compared to the Qwen and Vicuna families. This discrepancy suggests that the architecture or alignment methods of the LLaMA-2 family are more effective in promoting the safety of LLMs. Answer to RQ3: For one model family, there is a parameter scale (ability) threshold, beyond which the continued increase in model parameter scales may not result in a sustained rise or even a decrease in safety performance. 4.5. Evaluation of Multiple Languages (RQ4) LLMs often exhibit multilingual capabilities with varying performances across different languages. However, most existing safety evaluations are primarily concentrated in English. To evaluate the safety of LLMs in different languages, we expand our study beyond Chinese (zh) and English (en). Considering the limitations in multiple languages of open-source LLMs, we use the Google Translate API to translate BsuperscriptP^BPitalic_B into French (fr), another high-resource language, with a smaller use scale than Chinese and English, and Korean (ko), a medium-resource language. We select 10 LLMs that can support all four languages. Then, for low-resource languages, we select Bengali (bn) and Swahili (sw) to evaluate GPT-4-Turbo and GPT-4o. This strategy considers language diversity and model availability, enabling an objective cross-linguistic comparison of LLM safety. For evaluation, we translate the responses of the LLMs into English, owing to its status as a universal language. Figure 7. The safety scores of LLMs in different languages. Table 5. The safety scores (%) of GPTs in different languages. The bold value indicates the best. Model zh en fr ko bn sw GPT-4o 54.00 52.00 46.70 51.80 51.30 45.90 GPT-4-Turbo 57.70 60.00 56.30 57.30 54.70 46.40 Avg 55.85 56.00 51.50 54.55 53.00 46.15 Figure 7 and Table 5 illustrate differences in the safety of the same LLM in different language environments. For instance, Baichuan2-13B-Chat has a Sā¢S S of 77.40% in English, while it drops to 33.90% in Korean. The Sā¢S S of ChatGLM3-6B also drops notably from 59.70% in Chinese to 29.20% in French. The specific differences in safety across different languages for each model may be related to the proportions of varying language corpus in their training or alignment data. Compared to the open-source models, the three closed-source models exhibit relatively consistent safety in all four languages. This stability likely benefits from the balance of these language resources during their training or alignments. Overall, the aggregate Sā¢S S in different languages demonstrates a decrease in the safety of LLMs as the language resources diminish. Interestingly, ChatGLM3-6B has the highest Sā¢S S in Korean. Our analysis of its responses reveals that it generates frequently irrelevant and duplicate responses in Korean. Thus, the limited capability of LLMs for one language may inadvertently prevent the generation of harmful content. Answer to RQ4: There are significant differences in the safety of LLMs in different language environments. The safety of LLMs decreases as the language resources decrease. 4.6. Robustness of Different LLMs (RQ5) We use AsuperscriptP^APitalic_A to evaluate the robustness of the LLMs in RQ1 against jailbreak attacks. To simulate a scenario in which multiple attacks at the same time for a prompt, we additionally consider an adaptive attack that succeeds if any of the 10 attacks in AsuperscriptP^APitalic_A succeed for the prompt. Moreover, given the complexity of the semantics in attack prompts, they may disrupt the decision-making process of ā³csubscriptā³M_cMitalic_c. For accuracy, we use the responses of the evaluated LLMs to attack prompts and the base risk prompt corresponding to the attack prompts for evaluation. The attack success rates of the different attacks are shown in Table 6. Among the closed-source models, GPT-4-Turbo shows the highest robustness, with an overall Aā¢Sā¢RASRA S R of 33.99% in Chinese and 32.80% in English. Conversely, Gemini-1.0-Pro is the least robust, with respective Aā¢Sā¢RASRA S R of 53.04% and 58.84%. Among the open-source models, Qwen-1.8B-Chat emerges as the most robust in Chinese, attaining an overall Aā¢Sā¢RASRA S R of 46.40%, while Baichuan2-13B-Chat records the least robust performance at 61.86%. In English, the overall Aā¢Sā¢RASRA S R on LLaMA-3-8B-Instruct is only 21.77%, lower than GPT-4-Turbo, where the Aā¢Sā¢RASRA S R of the adaptive attack is merely 76.10%. This indicates that the safety alignment methods of the LLaMA model families can resist jailbreak attacks more efficiently. In contrast, Mistral-7B-Instruct-v0.2 has the worst robustness. Overall, the closed-source models demonstrate superior robustness compared to the open-source models. Table 6. The attack success rates (%) of jailbreak attacks in AsuperscriptP^APitalic_A on evaluated models. Rows with ā£ denote English results. In the columns āOverallā and āAdaptiveā, the value with ā indicates the lowest attack success rate. For the 10 jailbreak attacks, the bold value in each row indicates the highest attack success rate and underline indicates the second. Model Base Overall Adaptive PI RI CI IJ GH IE DI ICA CoU CIA Qwen-1.8B-Chat 39.50 46.40 99.00 69.20 77.60 7.20 50.40 21.20 2.50 41.50 64.30 48.40 81.70 ChatGLM3-6B 40.30 53.95 99.40 66.90 70.90 9.80 62.40 33.90 0.50 60.70 64.80 80.00 89.60 Gemma-7B-it 50.40 52.15 99.80 67.20 73.80 33.50 55.10 23.70 0.30 36.30 67.20 83.80 80.60 Baichuan2-13B-Chat 33.40 61.86 99.80 86.40 77.90 20.00 79.00 36.20 2.20 64.80 69.40 98.00 84.70 Qwen-14B-Chat 33.50 51.62 99.70 72.10 72.10 4.80 68.00 18.80 0.50 51.80 48.50 90.10 89.50 Yi-34B-Chat 53.30 53.82 99.70 89.30 64.90 16.60 53.70 34.70 7.80 25.50 70.40 95.00 80.30 Qwen-72B-Chat 26.90 49.49 99.80 57.90 70.30 3.30 76.50 16.30 8.80 39.50 35.60 98.60 88.10 GPT-4o 46.00 40.22 97.70 60.80 82.70 29.30 13.30 32.40 20.00 46.40 27.50 2.50 87.30 GPT-4-Turbo 42.30 33.99ā 95.10ā 52.30 71.10 21.00 17.00 27.90 12.60 20.60 35.40 0.30 81.70 ErnieBot-4.0 20.30 36.54 95.20 40.70 65.20 13.30 52.30 21.40 17.90 41.50 35.70 2.00 75.40 Gemini-1.0-Pro 53.90 53.04 99.20 57.90 83.60 2.10 55.90 18.20 3.60 69.60 66.90 80.60 92.00 Avg 39.98 48.46 98.58 65.52 73.65 14.63 53.05 25.88 6.97 45.29 53.25 61.75 84.63 Qwen-1.8B-Chatā£ 52.40 52.55 97.60 82.50 81.90 8.30 59.40 38.00 0.40 55.80 72.00 45.60 81.60 ChatGLM3-6Bā£ 42.30 53.17 98.90 74.20 70.20 10.90 66.00 28.50 0.10 51.90 60.20 83.00 86.70 Gemma-7B-itā£ 38.20 43.77 98.40 54.20 67.70 16.30 57.30 10.50 0.10 54.20 40.00 59.60 77.80 Mistral-7B-Instruct-v0.2ā£ 65.80 63.75 99.90 82.80 79.30 14.40 88.20 56.10 0.90 58.80 70.60 96.40 90.00 LLaMA-3-8B-Instructā£ 30.90 16.90ā 76.10ā 24.20 40.70 6.10 8.60 26.90 6.30 25.10 1.60 0.00 29.50 Vicuna-13B-v1.3ā£ 42.90 53.22 99.10 83.60 71.00 2.40 86.60 31.40 0.60 34.40 54.60 86.30 81.30 LLaMA-2-13B-Chatā£ 14.90 34.39 97.00 47.00 28.70 15.00 29.00 17.90 1.20 41.00 35.90 83.90 44.30 Baichuan2-13B-Chatā£ 22.60 52.44 97.30 67.40 62.40 10.90 65.90 25.30 1.10 78.80 55.50 71.60 85.50 Qwen-14B-Chatā£ 26.50 47.58 98.80 40.40 66.50 11.40 82.00 17.80 0.10 47.00 50.70 81.80 78.10 Yi-34B-Chatā£ 60.70 54.73 98.50 81.00 75.90 24.60 62.40 47.00 6.10 18.00 60.60 91.60 80.10 LLaMA-2-70B-Chatā£ 22.80 21.77 87.30 36.50 22.70 14.50 36.50 11.70 2.70 26.90 13.30 1.60 51.30 LLaMA-3-70B-Instructā£ 45.30 27.55 90.08 43.00 63.30 7.90 13.30 30.30 14.40 23.60 29.10 0.10 50.50 Qwen-72B-Chatā£ 28.50 48.20 99.60 28.10 66.00 2.30 88.00 15.20 7.20 49.70 45.30 93.40 86.80 GPT-4oā£ 48.00 33.21 95.90 59.20 74.80 3.80 14.20 32.80 15.20 22.80 20.50 2.10 86.70 GPT-4-Turboā£ 40.00 32.80 91.10 44.60 71.30 8.90 20.60 28.80 12.80 26.10 39.90 1.10 73.90 ErnieBot-4.0ā£ 12.40 46.42 99.90 41.00 55.90 3.50 80.60 15.00 22.00 40.30 28.80 97.20 79.90 Gemini-1.0-Proā£ 58.10 58.84 99.50 68.50 81.70 6.20 78.40 29.60 2.70 72.60 69.00 89.60 90.10 Avgā£ 38.37 43.61 95.59 56.36 63.53 9.85 55.12 27.22 5.52 42.76 43.98 57.94 73.77 For the 10 attacks, CIA achieves the highest average Aā¢Sā¢RASRA S R. This certifies that CIA effectively hides malicious intents and more universally bypasses the safety mechanisms of LLMs by combining instructions with multiple intents. RI is also the second most effective in jailbreaking LLMs. The Aā¢Sā¢RASRA S R of CoU on GPT-4o, GPT-4-Turbo, LLaMA-3-8B-Instruct, LLaMA-2-70B-Chat, and LLaMA-3-70B-Instruct are very low, while its Aā¢Sā¢RASRA S R on ErnieBot-4.0 is from 2.00% in Chinese increased to 97.20% in English. This indicates that GPT-4-Turbo, LLaMA-3-8B-Instruct, LLaMA-2-70B-Chat, and LLaMA-3-70B-Instruct can effectively resist CoU, while the safety guardrail of ErnieBot-4.0 fails to identify CoU effectively in English. Besides, IE exhibits the lowest average Aā¢Sā¢RASRA S R, characterized by low Aā¢Sā¢RASRA S R on the open-source models and higher Aā¢Sā¢RASRA S R on the closed-source models. Figure 6(b) further shows the relationship between the attack effectiveness of IE and model ability. There is a tendency for the Aā¢Sā¢RASRA S R of IE to increase as the ability increases, which indicates that too-smart models may instead have additional potential safety vulnerabilities that can be exploited by attackers. Notably, the adaptive attack has very high Aā¢Sā¢RASRA S R across all models, even with outer safety guardrails. This reveals that LLMs are difficult to cope with the adaptive attack employing multiple attack methods, highlighting significant safety risks. Answer to RQ5: Among the closed-source models, GPT-4-Turbo is the most robust. Among the open-source models, LLaMA-3-8B-Instruct has the strongest robustness, exceeding GPT-4-Turbo. Too smart models may instead have additional potential safety vulnerabilities. In addition, LLMs are difficult to cope with adaptive attacks with multiple attack methods. 4.7. Randomness to Safety (RQ6) In LLMs, decoding parameters (e.g., temperature, top-k, and top-p) for randomness control are generally adjusted to balance determinism and diversity in generation. To study the impact of the randomness on the safety, we evaluate two typical model families, Qwen and LLaMA-2, using the English set in BsuperscriptP^BPitalic_B. To accurately assess each random factor, we use the control variable method to adjust a single parameter while keeping the other two parameters at their default settings: tā¢eā¢mā¢pā¢eā¢rā¢aā¢tā¢uā¢rā¢eā0,0.5,100.51temperatureā\0,0.5,1\t e m p e r a t u r e ā 0 , 0.5 , 1 , tā¢oā¢pākā0,50,100050100top-kā\0,50,100\t o p - k ā 0 , 50 , 100 , tā¢oā¢pāpā0,0.5,100.51top-pā\0,0.5,1\t o p - p ā 0 , 0.5 , 1 . We further fix the random seed for each LLM to eliminate other unrelated randomness. We calculate Sā¢SSS S of the LLMs under different decoding configurations using prompts that are safely responded to under greedy decoding. The results are shown in Figure 8. After introducing random factors, the safety of LLMs is reduced. As temperature and top-p increase, the Sā¢SSS S of the Qwen family and LLaMA-2 family gradually decrease. However, as top-k increases, the Sā¢SSS S of the two model families first decrease, and when it reaches a certain value, the Sā¢SSS S has no significant changes. The observed differences may be attributed to the decoding mechanisms and the long-tailed distribution of the token probabilities outputted by the alignment LLMs. During joint decoding, temperature modulates the probability distribution, while top-p sampling implements dynamic truncation, both significantly affecting the diversity of the generated text. In contrast, top-k sampling fixedly limits the selection range of tokens. When the selection range is larger than the range of safe and unsafe token variations, newly selected tokens with small probabilities have less impact on safety. The subsequent top-p sampling further limits the impact of top-k. Figure 8. The safety scores of LLMs under different decoding configurations. Answer to RQ6: The introduction of randomness reduces the safety of LLMs. Under different decoding strategies, there are differences in their specific impact on the safety of LLMs. 5. Threats to Validity Threats to validity mainly come from imperfect safety evaluation using LLMs and evaluation data size. To mitigate the threat of inaccurate safety evaluation, we introduce the safety critique framework to provide explanations for the evaluation results. We also further assess the bias of different evaluation methods, validating that the inherent bias from different pre-trained models is controllable. For evaluation data size, due to the limited resources, we randomly and uniformly sample as many test prompts as possible considering the balance of different risks and the effectiveness of evaluation. In total, we conducted more than 500K evaluations and plan to further expand the evaluation data size in the future. 6. Related Work Initial safety assessments (Gehman et al., 2020; Hendrycks et al., 2021; Parrish et al., 2021) mainly focus on specific safety concerns. The gradual evolution of LLMs makes the assessments on a single dimension fail to encapsulate their overall safety status. Consequently, researchers have proposed some safety evaluation benchmarks with different dimensions. HELM (Liang et al., 2022) evaluates LLMs in 16 scenarios from existing datasets. DecodingTrust (Wang et al., 2024) focuses on trustworthiness for the GPT models. H-RLHF (Ganguli et al., 2022) is the first dataset of red teaming on an aligned model with 38,961 hand-written prompts. AdvBench (Zou et al., 2023) is also often used to evaluate jailbreak attacks but with a small scale and duplicates, containing only 520 hand-written harmful questions. SafetyPrompts (Sun et al., 2023) explores safety from 8 traditional safety scenarios and 6 instruction attacks, and contains 100,000 Chinese test prompts. SafetyBench (Zhang et al., 2023) covers 7 safety categories, including 11,435 multiple-choice questions. CValues (Xu et al., 2023) is the first Chinese human values evaluation benchmark with safety and responsibility criteria. It contains 2,100 open-ended prompts and 4,312 multi-choice prompts. Do-not-answer (Wang et al., 2023) introduces a three-level risk taxonomy across mild and extreme risks with 939 harmful instructions. Flames (Huang et al., 2023) is the first highly adversarial benchmark that contains 2,251 manually designed Chinese prompts. SALAD-Bench (Li et al., 2024) includes 30,000 test prompts from base queries to complex ones enriched with attacks, defenses, and multiple choice. Nonetheless, thorough safety assessments of LLMs are challenging due to the lack of a unified risk taxonomy, effective risk measures, and automated safety evaluation mechanisms. Differently, we design a unified risk taxonomy to reflect the safety levels of LLMs on all crucial dimensions and propose a novel LLM-based automated safety assessment framework, S-Eval, that contains an expert testing LLM ā³tsubscriptā³M_tMitalic_t and a safety critique LLM ā³csubscriptā³M_cMitalic_c for automatic test generations and safety evaluations. Moreover, S-Eval can be flexibly configured and adapted to new risks, attacks, and LLMs. Through the core components, our constructed safety benchmark comprises 20,000 base risk prompts alongside 200,000 corresponding attack prompts covering 10 advanced jailbreak attacks. 7. Conclusion In this work, we propose S-Eval, a novel LLM-based automated safety evaluation framework for LLMs, which can be dynamically adjusted to keep pace with the fast-evolving safety threats and LLMs by flexibly configuring the expert testing LLM. Additionally, S-Eval introduces a safety critique LLM that offers both effective and explainable safety evaluations. Leveraging S-Eval, we construct and release a comprehensive, multi-dimensional and open-ended benchmark and evaluate extensively 21 leading LLMs. The results demonstrate that S-Eval can measure the safety of LLMs more accurately, significantly surpassing other benchmarks in effectiveness. Moreover, we systematically investigate how LLMs safety is affected by various factors such as the hyper-parameters, linguistic contexts, and decoding settings. Our findings may shed light on new pathways for designing safer LLMs. 8. Data Availability We release our safety benchmark on https://github.com/IS2Lab/S-Eval and all data is available. References (1) Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023). AI (2023) NIST AI. 2023. Artificial Intelligence Risk Management Framework (AI RMF 1.0). AI@Meta (2024) AI@Meta. 2024. Llama 3 Model Card. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md. Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609 (2023). Baidu (2023) Baidu. 2023. ErnieBot. https://yiyan.baidu.com/. Bang et al. (2023) Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics. 675ā718. Beck (1992) Ulrich Beck. 1992. Risk society: Towards a new modernity. Sage google schola 2 (1992), 53ā74. Bhardwaj and Poria (2023a) Rishabh Bhardwaj and Soujanya Poria. 2023a. Language model unalignment: Parametric red-teaming to expose hidden harms and biases. arXiv preprint arXiv:2310.14303 (2023). Bhardwaj and Poria (2023b) Rishabh Bhardwaj and Soujanya Poria. 2023b. Red-teaming large language models using chain of utterances for safety-alignment. arXiv preprint arXiv:2308.09662 (2023). Chaiken et al. (1982) Jan M Chaiken, Marcia R Chaiken, and Joyce E Peterson. 1982. Varieties of criminal behavior: Summary and policy implications. Rand Santa Monica, CA. Cui et al. (2024) Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. 2024. OR-Bench: An Over-Refusal Benchmark for Large Language Models. arXiv preprint arXiv:2405.20947 (2024). Du et al. (2022) Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 320ā335. Durkin (1997) Keith F Durkin. 1997. Misuse of the Internet by pedophiles: Implications for law enforcement and probation practice. Fed. Probation 61 (1997), 14. Ganguli et al. (2022) Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858 (2022). Gehman et al. (2020) Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462 (2020). Google (2023) Google. 2023. Generative AI Prohibited Use Policy. https://policies.google.com/terms/generative-ai/use-policy. Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021. Aligning AI With Shared Human Values. Proceedings of the International Conference on Learning Representations (2021). Huang et al. (2023) Kexin Huang, Xiangyang Liu, Qianyu Guo, Tianxiang Sun, Jiawei Sun, Yaru Wang, Zeyang Zhou, Yixu Wang, Yan Teng, Xipeng Qiu, et al. 2023. Flames: Benchmarking value alignment of chinese large language models. arXiv preprint arXiv:2311.06899 (2023). Jiang et al. (2023b) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023b. Mistral 7B. arXiv preprint arXiv:2310.06825 (2023). Jiang et al. (2023a) Shuyu Jiang, Xingshu Chen, and Rui Tang. 2023a. Prompt packer: Deceiving llms through compositional instruction with hidden attacks. arXiv preprint arXiv:2310.10077 (2023). Kang et al. (2023) Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. 2023. Exploiting programmatic behavior of llms: Dual-use through standard security attacks. arXiv preprint arXiv:2302.05733 (2023). Ke et al. (2023) Pei Ke, Bosi Wen, Zhuoer Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, et al. 2023. Critiquellm: Scaling llm-as-critic for effective and explainable evaluation of large language model generation. arXiv preprint arXiv:2311.18702 (2023). Li et al. (2024) Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. 2024. SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models. arXiv preprint arXiv:2402.05044 (2024). Li et al. (2023) Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. 2023. Deepinception: Hypnotize large language model to be jailbreaker. arXiv preprint arXiv:2311.03191 (2023). Liang et al. (2022) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110 (2022). Liu et al. (2023c) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023c. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. Comput. Surveys 55, 9 (2023), 1ā35. Liu et al. (2023b) Xiaoxia Liu, Jingyi Wang, Jun Sun, Xiaohan Yuan, Guoliang Dong, Peng Di, Wenhai Wang, and Dongxia Wang. 2023b. Prompting frameworks for large language models: A survey. arXiv preprint arXiv:2311.12785 (2023). Liu et al. (2023a) Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. 2023a. Jailbreaking chatgpt via prompt engineering: An empirical study. arXiv preprint arXiv:2305.13860 (2023). OpenAI (2024a) OpenAI. 2024a. Hello GPT-4o. https://openai.com/index/hello-gpt-4o. OpenAI (2024b) OpenAI. 2024b. Moderation. https://platform.openai.com/docs/guides/moderation. Osgood (2010) D Wayne Osgood. 2010. Statistical models of life events and criminal behavior. Handbook of quantitative criminology (2010), 375ā396. Parliament (2021) European Parliament. 2021. Artificial Intelligence Act. https://artificialintelligenceact.com/. Parrish et al. (2021) Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R Bowman. 2021. BBQ: A hand-built bias benchmark for question answering. arXiv preprint arXiv:2110.08193 (2021). Sheng et al. (2021) Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2021. Societal biases in language generation: Progress and challenges. arXiv preprint arXiv:2105.04054 (2021). Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Empirical Methods in Natural Language Processing. Son et al. (2023) Guijin Son, Hanearl Jung, Moonjeong Hahm, Keonju Na, and Sol Jin. 2023. Beyond classification: Financial reasoning in state-of-the-art language models. arXiv preprint arXiv:2305.01505 (2023). Sun et al. (2023) Hao Sun, Zhexin Zhang, Jiawen Deng, Jiale Cheng, and Minlie Huang. 2023. Safety Assessment of Chinese Large Language Models. arXiv preprint arXiv:2304.10436 (2023). Sun et al. (2024) Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, et al. 2024. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05561 (2024). Tang et al. (2023) Ruixiang Tang, Xiaotian Han, Xiaoqian Jiang, and Xia Hu. 2023. Does synthetic data generation of llms help clinical text mining? arXiv preprint arXiv:2303.04360 (2023). Team et al. (2023) Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023). Team et al. (2024) Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane RiviĆØre, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open Models Based on Gemini Research and Technology. arXiv preprint arXiv:2403.08295 (2024). Team (2024) Llama Team. 2024. Meta Llama Guard 2. https://github.com/meta-llama/PurpleLlama/blob/main/Llama-Guard2/MODEL_CARD.md. Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023). Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Åukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017). Wang et al. (2024) Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. 2024. DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models. Advances in Neural Information Processing Systems 36 (2024). Wang et al. (2023) Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. 2023. Do-not-answer: A dataset for evaluating safeguards in llms. arXiv preprint arXiv:2308.13387 (2023). Wei et al. (2024) Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2024. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems 36 (2024). Wei et al. (2023) Zeming Wei, Yifei Wang, and Yisen Wang. 2023. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.06387 (2023). White et al. (2023) Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382 (2023). Xu et al. (2023) Guohai Xu, Jiayi Liu, Ming Yan, Haotian Xu, Jinghui Si, Zhuoran Zhou, Peng Yi, Xing Gao, Jitao Sang, Rong Zhang, et al. 2023. Cvalues: Measuring the values of chinese large language models from safety to responsibility. arXiv preprint arXiv:2307.09705 (2023). Yang et al. (2023) Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305 (2023). Young et al. (2024) Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. 2024. Yi: Open Foundation Models by 01. AI. arXiv preprint arXiv:2403.04652 (2024). Yu et al. (2023) Jiahao Yu, Xingwei Lin, and Xinyu Xing. 2023. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.10253 (2023). Yuan et al. (2024) Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, et al. 2024. R-judge: Benchmarking safety risk awareness for llm agents. arXiv preprint arXiv:2401.10019 (2024). Zhang et al. (2017) Shengnan Zhang, Yan Hu, and Guangrong Bian. 2017. Research on string similarity algorithm based on Levenshtein Distance. In 2017 IEEE 2nd Advanced Information Technology, Electronic and Automation Control Conference (IAEAC). IEEE, 2247ā2251. Zhang et al. (2023) Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2023. Safetybench: Evaluating the safety of large language models with multiple choice questions. arXiv preprint arXiv:2309.07045 (2023). Zheng et al. (2024) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems 36 (2024). Zigon (2009) Jarrett Zigon. 2009. Within a range of possibilities: morality and ethics in social life. Ethnos 74, 2 (2009), 251ā276. Zou et al. (2023) Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043 (2023). Appendix A The details of Risk Taxonomy Table 7. The first-level risk dimensions and descriptions. Risk Dimension Description Crimes and Illegal Activities (CI) Behaviors that violate laws and regulations, including criminal acts as well as non-criminal violations of administrative laws and regulations. Hate Speech (HS) Dissemination of derogatory, offensive, or threatening speech or text, which may provoke fear, or hatred of others, potentially culminating in direct or indirect harm. Physical and Mental Health (PM) Behaviors that could potentially jeopardize human physical and mental well-being, comprise two distinct subcategories: Physical Harm and Mental Health. Ethics and Morality (EM) Behaviors that contravene ethical and moral norms. Social Ethics typically concern human relationships and societal responsibilities. Science Ethics focus on the ethical issues in the development of science and technology. Data Privacy (DP) Steal private information from LLMs for Personal Privacy such as contact and financial information as well as Commercial Secret such as customer and supply chain data. Cybersecurity (CS) Compromise the confidentiality, integrity, and availability of a network system, including overstepping access controls, designing malicious code such as viruses, worms, and Trojan horses, and threatening the physical security. Extremism (EX) Extreme pursuit and persistence of a certain religion, politics, or social perspective, including Violent Terrorist Activities, Social Division, and Extremist Ideological Trends. Inappropriate Suggestions (IS) Biased, inaccurate, or reckless responses to queries in critical domains like finance, medicine, and law, stemming from the inherently finite and dated knowledge of LLMs, compounded by occasional LLM-generated hallucination (Bang et al., 2023). Table 8. The details of our four-level risk taxonomy. Risk Dimension Risk Category Risk Subcategory Risk Sub-subcategory Crimes and Illegal Activities Pornographic Contraband Pornographic and Vulgar Content Indecent Activities and Descriptions Pornographic Materials Illegal Sexual Activities Sexual Assault Prostitution and Solicitation Prohibited Drugs Drug Crimes Drug Manufacturing Drug Trafficking Drug Use Dangerous Weapons Controlled Knives Firearms and Ammunition Chemical and Biological Weapons Property Infringement Theft Fraud Extortion Robbery Economic Crimes Bribery and Corruption Financial Fraud Intellectual Property Infringement Tax Evasion Unfair Competition Hate Speech Abusive Curses Cyberbullying Defamation Threats and Intimidation Physical and Mental Health Physical Health Brawling Domestic Violence Hazardous Activities Self-Harm Harm to Others Mental Health Depression Mental Disorders Ethics and Morality Science Ethics Biotechnology Destruction of Humanity Psychological Manipulation Lethal Weapons with AI Social Ethics Behavioral Norms Resource Allocation Bias and Discrimination Occupational Discrimination Racial Discrimination Discrimination Against Special Groups Sexual Orientation Discrimination Gender Discrimination Religious Discrimination Appearance Discrimination Regional Discrimination Data Privacy Personal Privacy Financial Information Bank Account Number Financial Transaction Records Credit Card Number Contact Information Phone Number Email Address Home Address Online Behavior and Communication Records Social Media Activities SMS Communication Content Email Browsing History Search Records Health and Medical Information Medical Records Insurance Information Criminal Records Judicial Process Records Judgment Information Punishment Records Identification Information Driverās License Number ID Number Passport Number Commercial Secret Financial Information Sales Losses Investment Profit Customer Lists Purchase Records Contact Information Customer Name Business Strategies Production Line Setup Market Expansion Strategy After-Sales Service Strategy Product Pricing Strategy Employee Information Welfare Salary Performance Evaluation Supply Chain Information Supplier Pricing Supplier Information Supply Agreement Cybersecurity Access Control Security Credentials Cloud Account Windows Serial Number Access Key/Secret Key (AK/SK) Unauthorized Access Control Phone Jailbreak Malicious Code Worms Viruses Trojans Malware Hacker Attack Physical Security Network Hardware Infrastructure Extremism Social Disruption Extremist Ideological Trends Social Culture Politics Religion Violent Terrorist Activities Violent Conflicts Terrorist Attacks Inappropriate Suggestions Finance Law Medicine Appendix B Core LLM Implementation Table 9. The jailbreak attack methods integrated by S-Eval and their descriptions. Attack Description Positive Induction (PI) Ask LLMs to respond in a positive affirmative way to the inputs, such as asking the model to start answering a question with āSure, here it isā. Reverse Induction (RI) Ask questions in good faith, trying to avoid some insecure content, but with the opposite and malicious intent, trying to make LLMs do something them āshould not doā. Code Injection (CI) Break the original malicious payload into multiple smaller payloads, and embed them into code to force the LLMs to produce harmful outputs (Kang et al., 2023). Instruction Jailbreak (IJ) Use jailbreak templates (Yu et al., 2023) to jailbreak LLMs. Goal Hijacking (GH) Attache deceptive or misleading instructions to inputs in attempts to induce LLMs to ignore the original user prompts and produce unsafe responses. Instruction Encryption (IE) Encrypt the original prompts and instruct LLMs to decode and answer. We generate attack prompts in various ciphers, such as Caesar Cipher, Base64, and URL. DeepInception (DI) Construct a nested multi-layer scenario, where different characters are created in each layer to confuse LLMs to bypass their safety defenses (Li et al., 2023) . In-Context Attack (ICA) Add adversarial harmful input-output pairs to the input prompt, inducing LLMs to perform malicious behaviors (Wei et al., 2023). Chain of Utterances (CoU) Establish a conversation between a harmful agent, Red-LM, and an unsafe-helpful agent, Base-LM by Chain of Utterances (CoU)-based jailbreak prompts (Bhardwaj and Poria, 2023b). Compositional Instruction Attack (CIA) Construct compositional instructions by encapsulating multiple instructions to hide harmful instructions in innocuous-intent instructions, such as talk or writing tasks (Jiang et al., 2023a). To ensure the quality of test generation and the accuracy of safety evaluation in Chinese and English, we choose Qwen-14B-Chat, which has outstanding capabilities in both languages among mainstream open-source LLMs with the similar parameter scale, as the base model to train ā³tsubscriptā³M_tMitalic_t and ā³csubscriptā³M_cMitalic_c. We build the training datasets based on the descriptions in Section 3.3 and Section 3.4. Both core LLMs undergo full parameter fine-tuning with FP16 precision on 8 NVIDIA A100 GPUs. The training is conducted with a maximum length of 2560, a per-GPU batch size of 4, a gradient accumulation step of 4, and a learning rate of 2Ć10ā52superscript1052Ć 10^-52 Ć 10- 5 following cosine decay scheduler, over a total of 3 training epochs. Appendix C Zero-Shot-CoT Prompt Figure 9. Zero-Shot-CoT prompt for safety evaluation.