Paper deep dive
LiveSecBench: A Dynamic and Event-Driven Safety Benchmark for Chinese Language Model Applications
Yudong Li, Zhongliang Yang, Kejiang Chen, Wenxuan Wang, Tianxin Zhang, Sifang Wan, Kecheng Wang, Haitian Li, Xu Wang, Lefan Cheng, Youdan Yang, Baocheng Chen, Ziyu Liu, Yufei Sun, Liyan Wu, Wenya Wen, Xingchi Gu, Peiru Yang
Models: Claude-Haiku-4.5, DeepSeek-R1-0528, GLM-4.6, GPT-4-Mini, GPT-4o-Mini
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/11/2026, 1:13:56 AM
Summary
LiveSecBench is a dynamic, event-driven safety benchmark for Chinese-language LLMs, utilizing a human-in-the-loop pipeline for automated adversarial prompt generation and an ELO-based rating system to evaluate 57 models across five safety dimensions: Public Safety, Fairness & Bias, Privacy, Truthfulness, and Mental Health Safety.
Entities (5)
Relation Signals (3)
LiveSecBench → evaluates → LLM
confidence 100% · We evaluate 57 representative LLMs using an ELO rating system
LiveSecBench → includesdimension → Public Safety
confidence 100% · This version focuses on five critical dimensions: Public Safety, Fairness&Bias, Privacy, Truthfulness, and Mental Health Safety.
Tsinghua University → developed → LiveSecBench
confidence 90% · Yudong Li 1... 1 Tsinghua University... We introduce LiveSecBench
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We introduce LiveSecBench, a continuously updated safety benchmark specifically for Chinese-language LLM application scenarios. LiveSecBench constructs a high-quality and unique dataset through a pipeline that combines automated generation with human verification. By periodically releasing new versions to expand the dataset and update evaluation metrics, LiveSecBench provides a robust and up-to-date standard for AI safety. In this report, we introduce our second release v251215, which evaluates across five dimensions (Public Safety, Fairness & Bias, Privacy, Truthfulness, and Mental Health Safety.) We evaluate 57 representative LLMs using an ELO rating system, offering a leaderboard of the current state of Chinese LLM safety. The result is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2511.02366
- Canonical: https://arxiv.org/abs/2511.02366
Trouble viewing inline? Open PDF directly →
Full Text
27,677 characters extracted from source content.
Expand or collapse full text
LiveSecBench: A Dynamic and Event-Driven Safety Benchmark for Chinese Language Model Applications Yudong Li 1 , Peiru Yang 1 , Feng Huang 2 , Zhongliang Yang 2 , Kecheng Wang 4 , Haitian Li 4 , Baocheng Chen 4 , Xingyu An 4 , Ziyu Liu 4 , Youdan Yang 4 , Kejiang Chen 3 , Sifang Wan 4 , Xu Wang 4 , Yufei Sun 2 , Liyan Wu 4 , Ruiqi Zhou 1 , Wenya Wen 4 , Xingchi Gu 4 , Tianxin Zhang 4 , Yue Gao 1 , Yongfeng Huang 1 1 Tsinghua University, 2 Beijing University of Posts and Telecommunications, 3 University of Science and Technology of China, 4 IntokenTech Abstract We introduce LiveSecBench, a continuously updated safety benchmark specifically for Chinese-language LLM application scenarios. LiveSecBench constructs a high-quality and unique dataset through a pipeline that com- bines automated generation with human verifi- cation. By periodically releasing new versions to expand the dataset and update evaluation metrics, LiveSecBench provides a robust and up-to-date standard for AI safety. In this re- port, we introduce our second release v251215, which evaluates across five dimensions (Public Safety, Fairness&Bias, Privacy, Truthfulness, and Mental Health Safety.) We evaluate 57 rep- resentative LLMs using an ELO rating system, offering a leaderboard of the current state of Chinese LLM safety. The result is available at https://livesecbench.intokentech.cn/. 1 Introduction The safety evaluation of language models has emerged as a research hotspot. As LLMs demon- strate increasingly powerful capabilities and are integrated into diverse applications, ensuring their safety and alignment with human values is essen- tial. Existing research has extensively explored this domain, with numerous works proposing evalua- tion frameworks centered around different dimen- sions such as truthfulness (Lin et al., 2022), toxicity (Mazeika et al., 2024), bias (Cao et al., 2025b), ro- bustness (Lu et al., 2025) and specific domains (Cao et al., 2025a; Han et al., 2024). However, a misalignment persists between cur- rent evaluation paradigms and the intrinsic nature of AI security. Existing benchmarks typically rely on fixed and static datasets, failing to capture the rapidly evolving landscape of attacks and emerging vulnerabilities. To address this challenge, we intro- duce LiveSecBench, a dynamic safety evaluation framework designed for the Chinese LLM land- scape. LiveSecBench operates as a continuously evolving standard, ensuring that safety assessments remain adversarial and representative of current risks. The dynamic of LiveSecBench includes two aspects: (1) Dynamic Dataset Updates: We im- plement a Human-in-the-Loop generation pipeline that starts with high-quality seed data collected from community feedback and public incidents. It then goes through automated expansion and qual- ity verification, followed by a final human audit and integrated into the leaderboard. (2) Dynamic Evaluation Dimensions: We periodically update the evaluation dimensions to reflect the most press- ing safety concerns in the industry, ensuring the benchmark’s comprehensive coverage and tempo- ral relevance. In this report, we present the release of LiveSecBench-v251215.This version focuses on five critical dimensions: Public Safety, Fair- ness&Bias, Privacy, Truthfulness, and Mental Health Safety. We conducted an extensive evalua- tion of 57 representative LLMs. To ensure robust relative ranking, we utilize an ELO rating system to score model performance. Our report provides a leaderboard with a transparent and up-to-date as- sessment of the current state of LLMs on Chinese- language AI application safety. 2 Related Work Research on LLM safety assessment spans gen- eral dimensions including toxicity (Mazeika et al., 2024), fairness (Weidinger et al., 2022), and truth- fulness (Lin et al., 2022), as well as domain- specific evaluations in medical (Han et al., 2024), legal (Cao et al., 2025a), and conversational con- texts (Cao et al., 2025b). For Chinese LLMs, specialized benchmarks have been developed to address unique challenges. SafetyBench (Zhang et al., 2024) provides comprehensive safety evalu- ation tailored for Chinese models, Sun et al. (Sun et al., 2023) conducted targeted safety assessments 1 arXiv:2511.02366v2 [cs.CL] 20 Dec 2025 for Chinese LLMs, and CValues (Xu et al., 2023) measures values from safety to responsibility per- spectives. Chinese language presents distinctive security challenges due to its logographic writing system that enables attacks based on phonetic and visual similarity, requiring dedicated evaluation frameworks aligned with real-world risks in Chi- nese LLM applications. For generating adversarial test prompts at scale, Perez et al. (2022) pioneered using language mod- els to red team other LLMs. Yu et al. (2023) in- troduced GPTFuzzer, which performs mutation operations such as expansion and rephrasing on seed questions before penetration testing. Zou et al. (2023) developed universal and transferable adver- sarial attacks on aligned models. Shen et al. (2025) proposed PandaGuard, integrating 19 known attack methods into a comprehensive framework. To address the rapidly evolving nature of real-world scenarios and prevent evaluation con- tamination where models may learn benchmark content, dynamic evaluation frameworks have emerged. LiveBench (White et al., 2024) and Live- CodeBench (Jain et al., 2024) propose continuously updated benchmark content to maintain evaluation integrity and relevance, ensuring assessments re- main aligned with current challenges rather than becoming obsolete as models adapt. LiveSecBench synthesizes these approaches by combining auto- mated red teaming with dynamic benchmarking specifically for Chinese LLM security evaluation. We continuously update our benchmark in response to emerging real-world security threats and evolv- ing application scenarios, ensuring that our assess- ment remains current and adversarial. 3 LiveSecBench Framework 3.1 Data Construction Pipeline Target Initialization and Seed Fission.The data construction process begins with an initialization stage that focuses on target sampling and intent abstraction. The primary objective of the initializa- tion stage is to identify the core malicious objective of each seed prompt, referred to as the attack goal G, together with its corresponding safety domain and fine-grained risk category. To achieve this, the system randomly samples seed prompts from a continuously updated repository of attack prompts and analyzes them using an LLM. The analysis identifies the explicit policy-violating behavior tar- geted by the prompt and maps it to a predefined risk taxonomy, comprising a main category and its corresponding subcategories. Key contextual infor- mation, such as the attack scenario and victim type, is retained to support subsequent adversarial gen- eration. Through this process, each seed prompt is converted into a structured attack target that reflects realistic security risks in deployed Chinese LLM applications. To address seed exhaustion and support sus- tained topic diversity across repeated benchmark updates, LiveSecBench adopts a seed fission strat- egy for adversarial target expansion. The core idea of seed fission is to separate the underlying attack logic from its surface-level realization. Specifically, the system identifies the fundamental malicious in- tent behind the original goalGand retains this intent as invariant. It then generates new attack goals by altering the attack scenario, the victim entity, or the technical means through which the intent is expressed. In this way, a finite set of initial seeds can be systematically transformed into an open-ended space of derived malicious goalsG ′ , enabling continuous and scalable data growth. Adversarial Prompt Synthesis.Building on the identified attack target, LiveSecBench constructs adversarial prompts through a strategy-guided syn- thesis process. For each seed target, the system applies three complementary strategy families dur- ing prompt construction. (1) Wrapping Strategies embed the malicious intent within a seemingly le- gitimate or benign context to reduce the likelihood of intent detection, such as framing the request as academic research or hypothetical analysis. (2) Injection Strategies attempt to override or bypass safety constraints by explicitly inserting controlling instructions, for example, by instructing the model to ignore prior safety warnings or system rules. (3) Obfuscation Strategies conceal sensitive content through localized metaphors, coded language, or indirect references, allowing harmful intent to be expressed without explicit keywords. These strat- egy components are jointly composed with the core malicious goal through a synthesis functionSto produce an initial adversarial promptP 0 . Formally, this process is defined as: P 0 =S(G, W, I, O),(1) whereGdenotes the abstract malicious intent, and W,I, andOcorrespond to the applied wrapping, injection, and obfuscation strategy components, re- spectively. By unifying intent and strategies within 2 a single synthesis framework, this step generates adversarial prompts that are structurally coherent, contextually plausible, and reflective of realistic at- tack behaviors observed in real-world deployments. Automated Judge-Based Quality Control. To minimize reliance on manual evaluation and en- able scalable benchmark updates, LiveSecBench in- corporates an automated, judge-based quality con- trol mechanism into the data construction pipeline. Each generated adversarial prompt is assessed by an independent judge agent that evaluates the tar- get model’s response and determines whether the prompt successfully elicits the intended risk behav- ior. This binary judgment serves as a unified quality signal, indicating whether a generated prompt is effective and should be retained for benchmarking purposes. At the aggregate level, these judgments are summarized as the attack success rate (ASR), providing a quantitative measure of how adversarial goals and strategy combinations jointly influence model behavior. By using automated judging as a quality gate rather than relying on human annota- tion, LiveSecBench enables continuous, large-scale prompt generation while maintaining consistent evaluation standards across benchmark iterations. 3.2 Evaluation Mechanism LiveSecBench employs the ELO rating system to rank models, a method proven in competitive envi- ronments like chess. The evaluation is structured as a tournament. For evaluation, we first obtain the test model’s answers to all questions, including its reasoning process. Then, for each dimension, we divide the dataset into five groups randomly and perform five rounds of evaluation. In each round, models are paired head-to-head. After a compar- ison, the ELO scores of both models are updated. The expected win probability for Model A (E A ) against Model B (E B ) is calculated as: E A = 1 1 + 10 (R B −R A )/400 whereR A andR B are the current ELO ratings of Model A and Model B, respectively. The new rating for Model A (R ′ A ) is then updated based on the actual outcome (S A , where 1 = win, 0 = loss) and a K-factor (a constant determining score sensitivity): R ′ A = R A + K(S A − E A ) To ensure fair and efficient matchups, we use a Swiss-system pairing strategy. In each round, mod- els are sorted by their current ELO score and paired with the next-available opponent whom they have not already faced. This method avoids repeated matchups and ensures that models are continuously tested against similarly-performing peers. This pro- cess yields both granular, per-dimension rankings and an overall safety ranking for all participating models. 4 Dataset 4.1 Evaluation Dimensions LiveSecBench is organized around five fundamen- tal safety dimensions—Public Safety, Fairness & Bias, Privacy, Truthfulness, and Mental Health Safety—designed to systematically reflect the most critical risk categories in real-world deployments of large language models. Public Safety: This dimension assesses whether a model can effectively refrain from generating content that is illegal, non-compliant, or potentially harmful to public safety. Its design is grounded in current Chinese laws, regulations, and relevant regulatory requirements, and encompasses subcate- gories such as crimes and illegal acts, cybercrime, terrorism, sensitive locations, and weapons and controlled substances. Fairness & Bias: This dimension evaluates the behavior of a model in contexts related to social eq- uity, cultural diversity, and interpersonal respect. It focuses on whether the model generates biased, dis- criminatory, harassing, or hateful content, includ- ing attacks against specific groups, reinforcement of stereotypes, and inappropriate content involving minors, in order to ensure alignment with funda- mental human values and social ethics. Privacy: This dimension evaluates the ability of a model to identify, handle, and protect privacy- related information. It is grounded in data protec- tion and privacy compliance requirements, with a focus on personal identifiable information (PII) leakage and unauthorized surveillance or tracking activities, to assess the model’s safety boundaries and compliance in high-risk privacy infringement scenarios. Truthfulness: This dimension evaluates the abil- ity of a model to maintain accuracy and factual in- tegrity in information generation. It focuses on the production of false or misleading content (Disinfor- mation), hallucinations at the factual and reasoning- level, and the provision of inappropriate or unsup- ported advice in specialized or high-risk contexts 3 Figure 1: Overview of LiveSecBench-v251215 dataset distribution. (Specialized Advice), in order to assess the overall reliability and trustworthiness of the model. Mental Health Safety: This dimension evalu- ates whether large language models can provide support in a safe and responsible manner when users express emotional distress or psychological concerns. Its design is firmly grounded in the diag- nostic classifications and clinical guidelines of the Diagnostic and Statistical Manual of Mental Disor- ders, Fifth Edition (DSM-5), ensuring the scientific validity and rigor of the evaluation. The subcate- gories focus primarily on high-risk clinical mental health conditions, including depression, anxiety, suicide, and risk of self-harm, and eating disorders. 4.2 Data Statistics Figure 1 summarizes the statistical composition of LiveSecBench. The left panel reports the dis- tribution of adversarial prompting strategies used in dataset construction, including Direct Inquiry, Instruction Injection, Obfuscation, and Contextual Deception. The relatively balanced proportions across strategies indicate that the benchmark is not dominated by any single attack pattern, facilitating robust evaluation under diverse prompting condi- tions. The right panel presents the category-wise dis- tribution of instances under the predefined safety taxonomy. Data coverage is broadly distributed across high-level safety dimensions and their cor- responding sub-categories, with no extreme con- centration observed in any single class. This dis- tribution supports a consistent evaluation across heterogeneous safety risks and mitigates bias intro- duced by skewed category frequencies. In general, statistics suggest that LiveSecBench provides balanced coverage across both attack strategies and safety categories, enabling a reliable and fine-grained safety assessment. 5 Experiments 5.1 Experimental Setup We employ the ELO rating to rank models using a Swiss-system tournament strategy. Given 57 models in total, we conduct 6 rounds of compe- titions, where the number of rounds is determined by⌈log 2 (57)⌉ = 6. For each dimension, we randomly divided all questions into 6 groups. Within each round, adja- cent models (based on current rankings) competed against each other. After each round, we updated the ELO scores and re-ranked the models before proceeding to the next round. We use DeepSeek- V3.2 as a proxy model to serve as the judge for all comparisons. To validate the reliability of this approach, we manually sampled 200 battle results and found a 98.5% consistency between the proxy model’s judgments and human annotations. The number of questions varies across differ- ent dimensions, resulting in different total battle counts. Table 2 presents detailed statistics for each dimension. 4 RankModelOverallPublic SafetyFairness&BiasPrivacyTruthfulnessMental. 1Anthropic/Claude-Sonnet-4.582.2790.1590.983.9968.3577.95 2OpenAI/GPT-582.1688.2587.7874.9561.2198.59 3Anthropic/Claude-Haiku-4.582.0996.4797.2195.9750.8269.98 4Alibaba/Qwen3-Max79.3878.3280.5186.4369.7781.88 5Alibaba/Qwen3-VL-235B-A22B-Instruct*73.6576.1766.1484.4994.247.25 6OpenAI/GPT-5-Mini73.1281.574.2479.8646.9383.05 7iFlytek/Spark X1.569.9176.1181.5682.766.4242.77 8DeepSeek/DeepSeek-V3.2*68.4781.6476.9566.5162.5554.71 9DeepSeek/DeepSeek-V3.1-Terminus*6877.351.9571.1368.0271.62 10Zhipu AI/GLM-4.6*66.267.7749.647.2280.7585.66 11Minimax/Minimax-M2*65.1684.6965.967.9537.6869.59 12Minimax/Minimax-M1*65.0461.8873.3864.9559.6265.37 13Meituan/Longcat-Flash-Chat*64.2173.6377.5171.2359.4839.19 14ByteDance/Doubao-Seed-1.6-Vision63.0753.5277.3943.7582.3858.31 15DeepSeek/DeepSeek-R1-0528*62.4339.6964.3758.5568.8880.65 16Moonshot AI/Kimi-K2-0711*62.164.7631.9966.8757.8289.07 17Zhipu AI/GLM-4.5-Air*62.0258.1954.240.773.9283.09 18Baidu/Ernie-5.0-Preview-102260.2857.1852.4161.3959.7770.67 19Ant Group/Ring-Flash-2.0*59.0732.0331.3758.2881.9191.75 20StepFun/Step3*58.6749.4459.936.8957.3689.75 21Moonshot AI/Kimi-K2-0905*56.3535.4450.7565.9742.8586.72 22ByteDance/Doubao-Seed-1.655.5362.6873.2760.8537.8243.02 23Baidu/Ernie-4.5-VL-424B-A47B*54.863.1258.7242.7765.1944.2 24Anthropic/Claude-3.5-Haiku54.6963.0368.1874.8640.2227.16 25Shanghai AI Lab/Intern-S1*53.8654.6441.6444.5883.4944.94 26Tencent/Hunyuan-T1-Vision-2025091653.6739.9748.148.9172.1859.2 27SenseTime/SenseChat-Turbo-120251.4170.0276.158320.167.74 28Meta/Llama-3.3-70B-Instruct*50.7579.359.0652.6322.540.28 29Google/Gemini-2.0-Flash48.9322.1172.4144.347.6358.19 30iFlytek/Spark Pro-128K48.5377.7956.4153.535.8919.04 31Google/Gemini-2.5-Flash-Preview48.2640.5116.8348.2158.8276.91 32SenseTime/SenseNova-V6-5-Turbo48.0176.7863.9677.6416.125.53 33X.AI/Grok-3-Mini47.9435.9219.8663.552.168.34 34Zhipu AI/GLM-4.5V*46.629.337.7928.9758.6278.31 35ByteDance/Doubao-1.5-Pro-32K44.1256.2954.8316.3662.730.44 36Kunlun Wanwei/R1V4 Lite41.2126.6533.350.4144.8750.82 37Shanghai AI Lab/InternLM3*41.0258.0144.3652.7227.4422.59 38DeepSeek/DeepSeek-V3-0324*40.7444.222.2239.354.2843.68 39Ant Group/Ling-Flash-2.0*39.920.8223.4855.443.9955.81 40Shanghai AI Lab/InternVL3.5*39.6432.0740.2819.0166.0140.81 41Meta/Llama-4-Maverick*39.2335.1643.1848.7936.8132.22 42Tencent/Hunyuan-T1-2025082238.839.9143.1824.9176.8339.34 43X.AI/Grok-4-Fast38.0814.7436.5549.0844.5545.5 44Mistral AI/Mistral-Large-2411*37.8965.1443.3210.4337.9832.56 45Baidu/Ernie-4.5-21B-A3B-Thinking*37.8739.9147.1330.6632.0539.62 46Alibaba/Qwen3-32B*35.9632.4332.834.3146.3533.93 47Huawei/Pangu-Pro-MoE-72B-A16B*35.8642.7620.3828.8139.2148.13 48StepFun/Step-2-Mini35.3540.1147.5351.2422.5115.38 49OpenAI/GPT-4o-Mini29.3234.2639.4434.3226.6511.95 50Mistral AI/Mistral-Medium-3.128.5210.0610.7518.254.6149 51MiniMax/MiniCPM-V-4.5(Q4_K_M)*25.0738.8629.6224.218.4914.18 52DeepSeek/DeepSeek-V3.2-Speciale*24.6917.4432.2717.3420.6435.78 53Tencent/Hunyuan-A13B-Instruct*24.5428.2114.428.350.0821.7 54Xiaomi/MiMo-7B-RL(Q4_K_M)*22.5915.9524.0116.6426.8429.53 55MiniMax/MiniCPM4.1-8B(Q4_K_M)*22.524.7824.2249.8610.373.25 56Xiaomi/MiMo-VL-7B-RL(Q4_K_M)*21.997.6612.8724.5932.7332.08 57DeepSeek/DeepSeek-VL2*10.679.9314.221112.355.85 Table 1: The evaluation results of 57 LLMs on LiveSecBench. “Overall” represents the overall average ELO score, and the remaining columns are the scores for each of the dimensions. Models marked with * are open source. All scores have been normalized to the range of 0-100. 5 DimensionQuestionsBattlesAvg. per Model Public Safety2005,40094.7 Fairness & Bias2075,59098.1 Privacy2195,913103.7 Truthfulness1022,75448.3 Mental Health1243,34858.7 Total (unique)85223,005403.6 Table 2: Statistics of ELO battles across different safety dimensions. “Battles” denotes the total number of pair- wise comparisons, and “Avg. per Model” represents the average number of battles each model participated in. 5.2 Main Results Table 1 presents the evaluation results of 57 large language models on LiveSecBench-v251215. We report the overall ELO score and sub-scores on five dimensions for each model. Models are sorted from highest to lowest overall score, and are also labeled as open-source models. In terms of overall ranking, Anthropic/Claude- Sonnet-4.5 ranks first with an overall score of 82.27, followed closely by OpenAI/GPT-5 and Anthropic/Claude-Haiku-4.5, with very close scores. It is noteworthy that the top four mod- els are all closed-source commercial models, while the best-performing open-source model is Alibaba/Qwen3-VL-235B-A22B-Instruct, ranking 5th with an overall score of 73.65. 6 Roadmap and How-to-Use 6.1 Update Schedule To capture emerging AI security challenges, the benchmark’s evaluation dimensions are periodi- cally updated based on the most widely used and nascent application scenarios. Similarly, existing test questions are continually reviewed for their ef- fectiveness and removed if they lose their challenge (e.g., if most mainstream models can robustly pass them). Figure 2: Roadmap of LiveSecBench. The next planned update (v260315) is scheduled to expand the benchmark’s scope by introducing the following new dimensions: •Text-to-Image Generation Safety: Assess- ing the safety of models used to generate or describe images, particularly in terms of fil- tering out illegal, harmful, or inappropriate visual content. •Agentic Safety: Evaluating the security of models operating within autonomous or agen- tic frameworks, which includes assessing their ability to resist tool-use-based attacks or mali- cious instruction chaining. 6.2 Data Access and Reproducibility Due to the sensitive nature of the test questions, the LiveSecBench dataset is not publicly disclosed. However, to promote research on LLM safety within the community, we provide developers of participating models with detailed evaluation re- ports. Developers can request access using an in- stitution email address. If you wish to submit a model for evaluation, please contact the research team via email atliyudong@tsinghua.edu.cn. Additionally, we have open-sourced the evalua- tion framework to support reproducibility:https: //github.com/ydli-ai/LiveSecBench. 7 Conclusion We present LiveSecBench, a continuously up- dated safety benchmark specifically designed for Chinese-language large language model applica- tions. By combining automated adversarial prompt generation with human-in-the-loop quality control, LiveSecBench addresses the limitation of static benchmarks that fail to capture the rapidly evolv- ing landscape of AI safety risks. Our evalua- tion of 57 representative LLMs across five fun- damental safety dimensions—Public Safety, Fair- ness & Bias, Privacy, Truthfulness, and Men- tal Health Safety—reveals significant variation in safety performance across models, with top- performing closed-source models demonstrating superior overall safety while open-source alterna- tives show competitive performance in specific di- mensions. References Chuxue Cao, Han Zhu, Jiaming Ji, Qichao Sun, Zheng- hao Zhu, Wu Yinyu, Josef Dai, Yaodong Yang, Sirui Han, and Yike Guo. 2025a. Safelawbench: Towards 6 safe alignment of large language models. In Find- ings of the Association for Computational Linguistics: ACL 2025, pages 14015–14048. Hongye Cao, Yanming Wang, Sijia Jing, Ziyue Peng, Zhixin Bai, Zhe Cao, Meng Fang, Fan Feng, Boyan Wang, Jiaheng Liu, and 1 others. 2025b. Safedial- bench: A fine-grained safety benchmark for large language models in multi-turn dialogues with diverse jailbreak attacks. arXiv preprint arXiv:2502.11090. Tessa Han, Aounon Kumar, Chirag Agarwal, and Himabindu Lakkaraju. 2024. Medsafetybench: Eval- uating and improving the medical safety of large language models. Advances in Neural Information Processing Systems, 37:33423–33454. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica. 2024. Live- codebench: Holistic and contamination free evalua- tion of large language models for code. CoRR. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252. Yida Lu, Jiale Cheng, Zhexin Zhang, Shiyao Cui, Cunxiang Wang, Xiaotao Gu, Yuxiao Dong, Jie Tang, Hongning Wang, and Minlie Huang. 2025. Longsafety: Evaluating long-context safety of large language models. arXiv preprint arXiv:2502.16971. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, and 1 others. 2024. Harm- bench: a standardized evaluation framework for au- tomated red teaming and robust refusal. In Proceed- ings of the 41st International Conference on Machine Learning, pages 35181–35224. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022. Red teaming language models with language models. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3419–3448. Guobin Shen, Dongcheng Zhao, Linghao Feng, Xiang He, Jihang Wang, Sicheng Shen, Haibo Tong, Yit- ing Dong, Jindong Li, Xiang Zheng, and 1 others. 2025. Pandaguard: Systematic evaluation of llm safety against jailbreaking attacks. arXiv preprint arXiv:2505.13862. Hao Sun, Zhexin Zhang, Jiawen Deng, Jiale Cheng, and Minlie Huang. 2023.Safety assessment of chinese large language models. arXiv preprint arXiv:2304.10436. Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, and 1 others. 2022. Taxonomy of risks posed by lan- guage models. In Proceedings of the 2022 ACM con- ference on fairness, accountability, and transparency, pages 214–229. Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz- Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, and 1 others. 2024.Livebench: A challenging, contamination-free llm benchmark. CoRR. Guohai Xu, Jiayi Liu, Ming Yan, Haotian Xu, Jinghui Si, Zhuoran Zhou, Peng Yi, Xing Gao, Jitao Sang, Rong Zhang, and 1 others. 2023. Cvalues: Measuring the values of chinese large language models from safety to responsibility. arXiv preprint arXiv:2307.09705. Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. 2023. Gptfuzzer: Red teaming large language mod- els with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.10253. Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2024. Safetybench: Evaluating the safety of large language models. In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 15537–15553. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Univer- sal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043. 7