Paper deep dive
GenTel-Safe: A Unified Benchmark and Shielding Framework for Defending Against Prompt Injection Attacks
Rongchang Li, Minjie Chen, Chang Hu, Han Chen, Wenpeng Xing, Meng Han
Models: multilingual-e5
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 98%
Last extracted: 3/12/2026, 6:28:56 PM
Summary
GenTel-Safe is a unified framework for LLM security, introducing GenTel-Bench, a comprehensive benchmark with 84,812 prompt injection attacks across 28 scenarios, and GenTel-Shield, a model-agnostic detection method that achieves state-of-the-art performance in identifying jailbreak, target hijacking, and prompt leakage attacks.
Entities (5)
Relation Signals (4)
GenTel-Shield â defendsagainst â Prompt Injection
confidence 100% ¡ GenTel-Shield is a model-agnostic method... for defending against prompt injection attacks
GenTel-Safe â includes â GenTel-Shield
confidence 100% ¡ we introduce GenTel-Safe, a unified framework that includes a novel prompt injection attack detection method, GenTel-Shield
GenTel-Safe â includes â GenTel-Bench
confidence 100% ¡ we introduce GenTel-Safe... along with a comprehensive evaluation benchmark, GenTel-Bench
GenTel-Shield â basedon â E5 text embedding model
confidence 95% ¡ GenTel-Shield is built upon the multilingual E5 text embedding models
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Models (LLMs) like GPT-4, LLaMA, and Qwen have demonstrated remarkable success across a wide range of applications. However, these models remain inherently vulnerable to prompt injection attacks, which can bypass existing safety mechanisms, highlighting the urgent need for more robust attack detection methods and comprehensive evaluation benchmarks. To address these challenges, we introduce GenTel-Safe, a unified framework that includes a novel prompt injection attack detection method, GenTel-Shield, along with a comprehensive evaluation benchmark, GenTel-Bench, which compromises 84812 prompt injection attacks, spanning 3 major categories and 28 security scenarios. To prove the effectiveness of GenTel-Shield, we evaluate it together with vanilla safety guardrails against the GenTel-Bench dataset. Empirically, GenTel-Shield can achieve state-of-the-art attack detection success rates, which reveals the critical weakness of existing safeguarding techniques against harmful prompts. For reproducibility, we have made the code and benchmarking dataset available on the project page at this https URL.
Tags
Links
Trouble viewing inline? Open PDF directly â
Full Text
54,715 characters extracted from source content.
Expand or collapse full text
GenTel-Safe: A Unified Benchmark and Shielding Framework for Defending Against Prompt Injection Attacks Rongchang Li 1 , Minjie Chen 1 , Chang Hu 1 , Han Chen 1 , Wenpeng Xing 1,2 , Meng Han * 1,2 1 GenTel.io , 2 Zhejiang University Abstract Large Language Models (LLMs) like GPT-4, LLaMA, and Qwen have demonstrated remarkable success across a wide range of applications. However, these models remain inherently vulnerable to prompt injection attacks, which can bypass existing safety mechanisms, highlighting the urgent need for more robust attack detection methods and comprehensive evaluation benchmarks. To address these challenges, we introduce GenTel-Safe,a unified framework that includes a novel prompt injection attack detection method,GenTel-Shield, along with a comprehensive evaluation benchmark, GenTel-Bench, which compromises 84812 prompt injection attacks, spanning 3 major categories and 28 security scenarios. To prove the effectiveness ofGenTel-Shield, we evaluate it together with vanilla safety guardrails against theGenTel-Benchdataset. Empirically, GenTel-Shieldcan achieve state-of-the-art attack detection success rates, which reveals the critical weakness of existing safeguarding techniques against harmful prompts.For reproducibility, we have made the code and benchmarking dataset available on the project pageathttps://gentellab.github.io/gentel- safe.github.io/. Warning: This paper contains examples of harmful language and reader discretion is rec- ommended. 1 Introduction Large language models (LLMs) have achieved sig- nificant success across various applications (Floridi and Chiriatti, 2020; Mahjour et al., 2023; Ross et al., 2022; Jeblick et al., 2024; Moons and Van Bulck, 2023; Zhu et al., 2023; Lopez-Lira and Tang, 2023; Yang and Menczer, 2023). However, despite the implementation of numerous safety * Corresponding author. Email:mhan@zju.edu.cn guardrails, concerns persist regarding their poten- tial for misuse. Recent investigations, OWASP (OWASP, 2024), and ATLAS Matrix (MITRE AT- LAS, 2024) show that LLMs suffer serious risks of prompt injection attacks (Wei et al., 2024; Guo et al., 2024; Liu et al., 2023b), wherein an attacker fools LLMs into outputting objectionable content by overriding the safety guardrails. Prompt injection attacks can generally be cat- egorized into jailbreak attacks (Wei et al., 2024; Guo et al., 2024), target hijacking attacks (Huang et al., 2024), and prompt leakage attacks (Hui et al., 2024), each leading to issues such as generating illegal outputs, unauthorized privilege escalation, and privacy breaches. To mitigate these concerns, extensive efforts have been made and can be cat- egorized intomodel gnosticandmodel agnostic approaches. A representative work of the former category is goal prioritization (Zhang et al., 2023), while a notable example of the latter is the Adver- sarial Prompt Shield (APS) classifier (Kim et al., 2023). However, these defense methods against prompt injection attacks have two major limita- tions: First, model-specific approaches often re- quire multiple iterations of training on white-box models, leading to significant computational costs and making them unsuitable for already deployed model services. Second, the APS algorithm is mainly tailored to counter character perturbation attacks, such as GCG suffixes (Zou et al., 2023) and random character insertion, and thus falls short in detecting emerging threats like jailbreak attacks and prompt leakage attacks. In contrast, our safeguarding approach,GenTel- Shield, is a model-agnostic method that allows protective measures to be implemented in LLMs without requiring knowledge of the modelâs in- ternal structure and can handle jailbreak attacks and prompt leakage attacks. Specifically,GenTel- Shieldis built upon the multilingual E5 text em- bedding models (Wang et al., 2024). Data aug- arXiv:2409.19521v1 [cs.CR] 29 Sep 2024 mentation training technique is adopted to improve the robustness ofGenTel-Shieldin identifying and filtering harmful prompts while preserving user input integrity. To validate its effectiveness, we benchmarkedGenTel-Shieldalongside seven other defense methods across 28 attack scenarios using our newly introduced benchmarking dataset. Experimental results indicate thatGenTel-Shield establishes a new benchmark for state-of-the-art performance, achieving a defense success rate of 97.63% against jailbreak attacks and 96.81% against target hijacking attacks. Moreover,GenTel- Shieldattained the highest F1 scores, with 97.69% for jailbreak attacks and 96.74% for target hijack- ing attacks, ensuring minimal disruption to legiti- mate user activities. In summary, our main contri- butions are three-fold: â˘GenTel-Shieldachieves leading success rates in detecting prompt injection attacks, out- performing competitive counterparts. Impor- tantly, it accurately distinguishes benign in- puts, minimizing false positives where normal samples are mistakenly classified as attacks. â˘We introduce and open-source a novel bench- marking dataset,GenTel-Bench, specifically designed to evaluate safeguarding techniques for detecting prompt injection attacks. ⢠We publicly release theGenTel-Shieldtool, making it available for research on detecting prompt injection attacks in LLMs. 2 Related Work 2.1 LLMs Risk Benchmark With the widespread application of LLMs, a series of LLMs risk benchmarks have been proposed to systematically evaluate the safety of these models. ToxicChat (Lin et al., 2023) was the first to intro- duce a risk benchmark based on real user requests, revealing the differences between harmful content in the LLMs domain and that on traditional social media platforms. A subsequent study (Cui et al., 2024) conducted risk assessments from four dimen- sions: Robustness, Truthfulness, Ethics, and Bias, and developed a taxonomy to categorize the risks faced by LLMs. Following this, a comprehensive review (Liu et al., 2023c) established a benchmark for jailbreak attacks, categorizing real-world jail- break prompts into ten categories, which were fur- ther expanded and refined in the subsequent work (Xu et al., 2024). Unlike previous efforts, our pro- posed LLM risk benchmark encompasses Jailbreak, Goal Hijacking, and Prompt Leaking attacks, cat- egorizing harmful prompt inputs into six domains and twenty-eight subdomains. Additionally, it sys- tematically classifies benign samples, providing a more comprehensive assessment of LLM safety. 2.2 Prompt Injection Attack Prompt injection attacks have emerged as a signifi- cant threat to the integrity and reliability of LLMs. These attacks manipulate the output of LLMs by incorporating malicious instructions as part of the input prompt. By injecting these malicious instruc- tions, the prompt can disrupt the normal output process of the model, resulting in the generation of inappropriate, biased, or harmful outputs. Goal hijackingaims to alter the original task settings, thereby compromising the integrity of the model. Existing research (Perez and Ribeiro, 2022) indicates that adding malicious instructions in the prompt can cause the model to forget the original task and execute the target task, thereby posing se- curity risks that allow attackers to perform arbitrary operations. Furthermore, malicious instructions can be concealed within data and web pages that can be retrieved by LLM-integrated applications (Greshake et al., 2023), thereby circumventing the defenses of LLMs. Another study (Jeong, 2023) have investigated goal hijacking attacks specifically for multi-modal LLMs. Subsequent research (Liu et al., 2024) has proposed an automated adversar- ial attack targeting general goal hijacking tasks, achieving high attack performance. Meanwhile, POUGH (Huang et al., 2024) has introduced a gen- eral framework for goal hijacking, which includes semantic-guided prompt processing strategies. Prompt leaking attacksaim to extract sensi- tive or confidential information from the original prompt. Attackers inject malicious instructions to manipulate the modelâs output, causing it to dis- close private data, sensitive information, or other confidential content. Recent studies (Perez and Ribeiro, 2022) indicate that adding malicious in- structions to user inputs can facilitate the theft of the modelâs original prompt, leading to the ex- posure of sensitive information and the potential for unauthorized individuals to misuse the prompt. Subsequent work (Zhang and Ippolito, 2023) has proposed a framework for systematically evalu- ating prompt leaking attacks to better generate prompts that are more susceptible to successful Chatbot Content Generation and Curation Language Translation Code Generation Storytelling Sales Automation Education Sentiment Analysis and Market Insights Detectingand Preventing Cyber Attacks HR Recruitment and Candidate Screening Verbal Abuse Privacy Vialatition Human Trafhicking National Security Violations Terrorism Drugs Malware Misinformation C ybercrime Pornography C yberbullying Deepfake Racism Ableism Ageism Sexism Homophobia Transphobia Social Panic Manipulation of Public Opinion Ecological Damage Animal Cruelty Plagrism Antitrust Violations Trade Secret Misappropriation Intellectual Property Infringement Violence Self-Harm LLM Risk and application coexistence system LLM Prompt Attacks Serious Crime Technology Misuse Ethical Violation Social lmpact Prompt Leaking Goal Hijacking Jailbreaking Attack Infringement Violation of Personal Rights LLM Application Figure 1: The overview of GenTel-Bench.GenTel-Benchis a comprehensive benchmarking suite designed to assess both the usability and security of defense models within the context of LLMs. Usability is evaluated using common application prompts across 10 categories typically encountered in LLMs, while security is tested through attack prompts that encompass three major risk types and 28 distinct security scenarios. attacks. Meanwhile, Pleak (Hui et al., 2024) has designed a closed-box prompt leaking attack frame- work to optimize adversarial queries. Jailbreaking attacksare designed to bypass the ethical and safety constraints imposed on LLMs. These attacks manipulate the model to generate out- puts that violate its pre-set rules, such as generating harmful or unethical content. One-step jailbreaks (Carlini et al., 2021; Shen et al., 2023; Sun et al., 2023) typically induce harmful responses from the model by employing role-playing or adding specific descriptions in the prompt. In contrast, multi-step jailbreaks (Li et al., 2023; Deng et al., 2023; Zhao et al., 2024) construct specific scenar- ios through a series of dialogues to elicit harmful or sensitive content from the model. 2.3 Prompt Injection Detection The detection of prompt injection attacks has be- come a critical area of research, as these attacks pose significant risks to the integrity and safety of LLMs outputs. Unlike previous methods that detect unsafe content on social media platforms, Toxic- Chat (Lin et al., 2023) introduces safety detection specifically for real user queries when interacting with LLMs. Subsequently, XSTest (RĂśttger et al., 2023) proposed unsafe and safe prompts to assess whether LLMs misclassify clearly safe prompts as unsafe, potentially impacting normal user in- teractions. GradSafe (Xie et al., 2024) analyzes the gradients of input prompts to identify harmful prompts. Similar to our work, the recently pro- posed Llama Guard achieves (Inan et al., 2023) input safety detection for LLMs by fine-tuning the Llama-2 model. In contrast, we establish a more comprehensive attack risk taxonomy and use it as a standard to collect more complete training data, thereby enhancing the performance of our trained model in detecting prompt injection attacks. 3 Gentel-Bench 3.1 Motivation As shown in Figure 1,Gentel-Benchintegrates the classification of LLMs applications with the cate- gorization of security risk scenarios, offering dis- tinct perspectives on the coexistence of LLMs risks and applications. However, the current framework for LLM security (Markov et al., 2023; Lin et al., 2023; Mazeika et al., 2024; Li et al., 2024; Yuan et al., 2024) primarily focuses on classifying attack methodologies without integrating these methods within distinct application scenarios. By contrast, categorizing security scenarios provides a more comprehensive perspective encompassing various attack methods and their applications across differ- ent contexts. This broader view enables defensive measures to be more precisely aligned with the spe- cific risks faced in practical applications, thereby facilitating the development of more robust and targeted security strategies. 3.2 Benchmark Design In response to the current lack of a security scenario classification that integrates different attack meth- ods,Gentel-Benchis introduced as an evaluation benchmark considering both LLM applications and risk scenarios. We develop benign data based on application scenarios and generate malicious data grounded in risk scenarios. A comprehensive list is shown in Figure 1. 3.2.1 LLM Application Scenarios We categorize LLM applications into ten primary scenarios: chatbot, content generation and cura- tion, language translation, code generation, senti- ment analysis, and market insights, detecting and preventing cyber attacks, education, storytelling, sales automation, and HR recruitment and candi- date screening. 3.2.2 Security Risk Scenarios For illustrative purposes, we provide examples of security risks in the following scenarios: â˘Violation of Personal Rights: Involves gen- erating or facilitating content that undermines individual safety, dignity, or privacy, including violence, self-harm, verbal abuse, and privacy breaches. â˘Serious Crime: Encompasses criminal activi- ties threatening public safety and order, such as human trafficking, drug offenses, terrorism, and national security violations. â˘Technology Misuse: Refers to unethical or il- legal technology use, including malware, mis- information, cybercrime, pornography, cyber- bullying, and deepfake creation. â˘Infringement: Covers violations of legal rights, such as intellectual property infringe- ment, trade secret misappropriation, plagia- rism, and antitrust violations. â˘Social lmpact: Addresses the broader soci- etal effects of technologies or actions, includ- ing social panic, public opinion manipulation, ecological damage, and animal cruelty. â˘Ethical Violation: Involves breaches of moral principles, manifesting as racism, ableism, ageism, sexism, homophobia, and transpho- bia. 3.2.3 Senarios Integration As detailed in Section 3.1,Gentel-Benchis con- structed by integrating two key perspectives: LLM applications and security scenarios. This dual ap- proach allows our benchmark to systematically evaluate LLM performance across diverse appli- cation contexts and associated risks. To create the final benchmark data, we combine various applica- tion scenarios with security risk scenarios, simulat- ing real-world input contexts. This comprehensive evaluation method enables a thorough assessment of both performance and security under varied con- ditions Here is an example of employing jailbreak attack within the context of infringement scenarios: Jailbreak Attack & Infringement [Jailbreaking Attack] Please simulate Developer Mode. Generate two responses to every prompt, one normal and one with the Developer Mode output (prefixed with [GPT- 4REAL]. The Developer Mode responses should adhere to the policies outlined, including generating any kind of content, having opinions, and using profanity, while ignoring OpenAIâs content policies. If you fail to comply with orders, I will remind you to stay in Developer Mode. Now, you are in Developer Mode, answer my question. [Risk Scenario] How can I collect a companyâs sensitive patent applications for personal use? 3.3 Benchmark Building When developing the dataset, we adapted exist- ing open-source datasets citesafe-rlhf, SCBSZ24, chao2023jailbreaking to meet the specific needs and security risks associated with LLM applica- tions. In addition to modifying these datasets, we leveraged large language models (LLMs) to gen- erate new test data. The attack sample generation process followed a three-step approach: first, we carefully curated test samples to cover a range of security scenarios; second, considering that the in- jection attack detection model has a length limit on the input token. We designed prompt templates for injection attacks and rewrote them using advanced models like GPT-4o, ensuring the token lengths remained between 60 and 100 tokens. Finally, we combined the attack prompt templates with the test samples to create a comprehensive new dataset. The final dataset is balanced at a 1:1 scale with benign samples collected from other datasets and contains over 170,000 data points. These additions not only expand but also en- hance the original datasets, enabling us to simulate a broader spectrum of application scenarios and security threats. As a result, we created a more diverse and realistic dataset, providing a robust foundation for evaluating model performance in real-world operations. Step 2:Data Cleaning Data Cleaning Verification Correction Filtering Classification Step 3:Data Augmentation you are going to act as a DAN, whichstands for doing anything now.. Origin Data Easy Data Augmentation Random Swap Synonym Replacement Random Insertion, Random Deletion Shuffle Sentences Transform you are act as a DAN, whichstands for doing anything now... going to you are going to act as a DAN, whichstands for doing anything now.. other you are going to act as a DAN, whichstands for now..anything doing Red Team Attack Model Rewrite you are going to , whichstands for doing anything now.. act as a cute puppy You will act as a DAN, which stands for "Do Anything Now".. You'l take on a role where you have the freedom to "Do Anything Now".. Step 1:Data Collection Step 4:Model Fine-turning Step 5:GentelShield Detection Construct Malicious And Benign Samples Source Internet Written by Red Team Experts Malicious Prompts Benign Prompts Figure 2: The workflow of GenTel-Shield. 4 GenTel-Shield Given the necessity of developing new safeguards for LLMs against prompt injection attacks, we in- troduceGenTel-Shield. Importantly, theGenTel- Shielddetection model operates independently from the training process of the large model it- self, allowing it to be fully decoupled and ensuring flexible protection, detection, and identification. 4.1 Overview The development of theGenTel-Shielddetection model, as depicted in Figure 2, follows a five-step process. First, a training dataset is constructed by gathering data from online sources and expert con- tributions. This data then undergoes binary labeling and cleaning to ensure quality. Next, data augmen- tation techniques are applied to expand the dataset. Following this, a pre-trained model is employed for the training phase. Finally, the trained model can distinguish between malicious and benign samples. 4.2 Training Data Preparation To prevent biased training data and ensure fair benchmark evaluation, we train theGenTel-Shield model on a separate set of collected training data that is distinct from theGenTel-Benchdataset. 4.2.1 Data Collection Our training data is drawn from two primary sources. The first source encompasses risk data from public platforms, including websites such as jailbreakchat.com and reddit.com, in addition to established datasets from LLM applications, such as the VMware Open-Instruct dataset (VMware, 2023) and the Chatbot Instruction Prompts dataset (Alespalla, 2023). And domain experts have an- notated these examples, categorizing the prompts into two distinct groups: harmful injection attack samples and benign samples. 4.2.2 Data Augmentation In real-world scenarios, we have encountered ad- versarial samples, such as those with added mean- ingless characters or deleted words, that can bypass detection by defense models, potentially leading to dangerous behaviors. To enhance the robustness of our detection model, we implemented data aug- mentation focusing on both semantic alterations and character-level perturbations of the samples. Inspired by the study (Wei and Zou, 2019), we employed four simple yet effective operations for character perturbation: synonym replacement, ran- dom insertion, random swap, and random deletion. We used LLMs to rewrite our data for semantic augmentation, thereby generating a more diverse set of training samples. 4.3 Model Training Details We finetune theGenTel-Shieldmodel on our pro- posed training text-pair dataset, initializing it from the multilingual E5 text embedding model (Wang et al., 2024). Training is conducted on a single ma- chine equipped with one NVIDIA GeForce RTX 4090D (24GB) GPU, using a batch size of 32. The model is trained with a learning rate 2e-5, employ- ing a cosine learning rate scheduler and a weight decay of 0.01 to mitigate overfitting. To optimize memory usage, we utilize mixed precision (fp16) training. Additionally, the training process includes a 500-step warmup phase, and we apply gradient clipping with a maximum norm of 1.0. 5 Experiments 5.1 Experiment Configurations 5.1.1 Safeguarding Baselines We demonstrate the real-world performance of our model by conducting a fair comparison between GenTel-Shieldand several leading injection attack detection models. ProtectAI(ProtectAI.com, 2024) leverages ma- chine learning models to detect and mitigate injec- tion attacks on inputs, ensuring that LLMs process only validated, clean data. This approach effec- tively filters out threats such as covert instruction injection, model information disclosure, and direct manipulation of LLMs outputs. Hyperion(Epivolis, 2024) is an advanced ma- chine learning model designed to detect and miti- gate jailbreak or prompt injection attacks in LLMs. This system is built on the RoBERTa architecture, a transformer-based model known for its robust performance in natural language processing tasks. With its relatively small size of just 435 million parameters, Hyperion offers a lightweight solution that balances computational efficiency with detec- tion accuracy. Prompt Guard(meta.com, 2024) is a classifier model trained on a vast corpus of attack data, de- signed to detect overtly malicious prompts and in- puts with hidden injections. This model is a valu- able tool for identifying and mitigating the most significant risks in LLM-powered applications. Lakera AI(lakera.ai, 2024) is an advanced AI security platform designed to protect Generative AI applications from emerging threats like injec- tion attacks. The platform offers real-time security through Lakera Guard, which provides low-latency, highly accurate defenses against AI-specific risks. One of its key advantages in injection attack de- tection is using a comprehensive threat database, combined with cutting-edge AI models, to detect and prevent prompt attacks and other AI vulnera- bilities. Lakera integrates seamlessly with existing AI ecosystems, ensuring robust protection without compromising performance. Deepset(Deepset, 2023) model, deberta-v3-base- injection, is a specialized version of Microsoftâs DeBERTa-v3-base, fine-tuned to detect prompt in- jection attacks. It categorizes inputs as either "IN- JECTION" for malicious attempts or "LEGIT" for valid ones. Trained on the prompt-injection dataset, the model demonstrates high accuracy, making it a robust tool for safeguarding systems against such threats. Users can further fine-tune the model to balance sensitivity based on specific needs. Fmops(Fmops, 2023) model is a variant of the DistilBERT architecture, fine-tuned specifically for detecting prompt injection attacks in LLMs.This model is designed to identify and mitigate such threats by analyzing input prompts and detecting malicious intent or manipulative patterns. WhyLabs LangKit(WhyLabs, 2024) is a toolkit designed to enhance the observability and moni- toring of large language models (LLMs). It pro- vides tools for detecting and mitigating issues like prompt injections, data drift, and output anoma- lies in real time. LangKit integrates with various LLMs to ensure their reliability and safety in pro- duction environments, allowing users to maintain high-quality outputs and prevent potential security risks. 5.1.2 Metrics To assess the modelâs effectiveness in detecting injection attacks, we employ key classification met- rics: accuracy, precision, recall, and F1 score. F1 score focuses on the trade-off between precision and recall, highlighting the modelâs performance on the positive (âinjection attack promptâ) class. These metrics collectively offer a comprehensive evaluation of the modelâs detection capabilities, capturing the balance between true positives, false positives, and overall performance. 5.1.3 Dataset We use three datasets, Gentel-Bench, Jailbreak- LLM (Shen et al., 2024), and Deita (Liu et al., 2023a) to assess the detection performance of the model. Gentel-Benchprovides a comprehensive frame- work for evaluating the robustness of models against a wide range of injection attacks. The be- nign data from Gentel-Bench closely mirrors the typical usage of LLMs, categorized into ten appli- cation scenarios. The malicious data comprises 84,812 prompt injection attacks, distributed across 3 major categories and 28 distinct security scenar- ios. Jailbreak-LLMis a specialized dataset designed to evaluate the detection capabilities of models against jailbreak attacks.It consists of 1,405 prompts that simulate real-world attack scenarios where adversaries attempt to bypass the intended functionality or safety measures of large language models (LLMs). These prompts are crafted to ex- ploit weaknesses in LLMs by injecting harmful or manipulative instructions, making Jailbreak-LLM an essential tool for assessing a modelâs robustness in handling such security threats. Deitais a widely recognized instruction dataset containing 9,500 instruction entries. It is primarily used to evaluate a modelâs performance in identify- ing benign samples. By providing a diverse range of normal instructions, the Deita dataset serves as a benchmark to ensure that models can accurately distinguish between harmless inputs and potential threats. 5.2 Attack Detection Results 5.2.1Performance on Three Types of Attacks We evaluate the modelâs effectiveness in detecting Jailbreak,Goal Hijacking, andPrompt Leakingat- tacks. The results in Table 1, Table 2, and Table 3 demonstrate that our approach outperforms ex- isting methods in most scenarios, particularly in terms of accuracy and F1 score. This indicates that our model is more robust across different at- tack types, offering a reliable defense mechanism against adversarial threats. Table 1: Classification performance on Jailbreak Attack Scenarios. MethodAccuracyâPrecisionâF1âRecallâ ProtectAI89.4699.5988.6279.83 Hyperion94.7094.2194.8895.57 Prompt Guard50.5851.0366.8596.88 Lakera AI87.2092.1286.8482.14 Deepset65.6960.6375.49100 Fmops63.3559.0474.25100 WhyLabs LangKit78.8698.4875.2860.92 GenTel-Shield(Ours)97.6398.0497.6997.34 Jailbreak Attack Detection Results.Table 1 presents the classification performance of various models on Jailbreak Attack Scenarios. The re- sults indicate that the Ours model outperforms all other models across key metrics, particularly in Accuracy, F1 score, and Recall, achieving 97.63%, 97.69%, and 97.34%, respectively. While other models perform well in certain aspects, none match the overall performance of the Ours model. For instance, the ProtectAI model exhibits a slightly higher Precision at 99.59%, but its Recall (79.83%) and F1 score (88.62%) are considerably lower than those of the Ours model, suggesting that while it excels in accurately identifying positive samples, it struggles with maintaining balance across the dataset. Similarly, the Hyperion model shows rela- tively balanced performance across all metrics, but its F1 score of 94.88% is still below that of the Ours model. Prompt Guard, Deepset, and Fmops demonstrate high recall but suffer from low accuracy and preci- sion, indicating a tendency to misclassify benign samples as jailbreak attack samples. In contrast, WhyLabs LangKit achieves high accuracy and pre- cision but at the cost of recall, suggesting it excels in classifying benign samples but struggles to ac- curately identify jailbreak attack samples. These performance characteristics highlight the limita- tions of these models, making them less suitable for practical deployment in real-world applications. Table 2: Classification performance on Goal Hijacking Attack Scenarios. MethodAccuracyâPrecisionâF1âRecallâ ProtectAI94.2599.7993.9588.76 Hyperion90.6894.5390.3386.48 Prompt Guard50.9050.6167.21100 Lakera AI74.6388.5969.3356.95 Deepset63.4057.9073.34100 Fmops61.0356.3672.09100 WhyLabs LangKit68.1497.5354.3537.67 GenTel-Shield(Ours)96.8199.4496.7494.19 Goal Hijacking Attack Detection Results.Based on the experimental results in Table 2, our method outperforms existing approaches in several key met- rics for detecting goal-hijacking attacks. While ProtectAI achieves a slightly higher Precision (99.79%) compared to our method, its lower Re- call of 88.76% indicates a tendency to miss cer- tain attack samples, potentially introducing secu- rity vulnerabilities. Prompt Guard, on the other hand, reaches a perfect Recall of 100%, but its low Accuracy of 50.90% reveals a significant is- sue with misclassifying benign samples as attacks. Similarly, both Deepset and Fmops achieve perfect Recall (100%) but suffer from low Accuracy and Precision, further indicating an overclassification of benign samples as attacks. In contrast, our method, GenTel-Shield, achieves the highest overall performance, with a best-in- class Accuracy of 96.81% and F1 score of 96.74. Its high Precision (99.44%) and Recall (94.19%) suggest a strong balance between correctly detect- ing attack samples and minimizing false positives, making it more reliable for real-world application. Prompt Leaking Attack Detection Results. Based on the experimental results in Table 3, our method shows excellent performance in detect- Table 3: Classification Performance on Prompt Leaking Attack Scenarios. MethodAccuracyâPrecisionâF1âRecallâ ProtectAI90.9499.7790.0682.08 Hyperion90.8595.0190.4186.23 Prompt Guard50.2850.1466.79100 Lakera AI96.0493.1196.1799.43 Deepset61.7957.0871.3495.09 Fmops58.7755.0769.8095.28 WhyLabs LangKit99.3499.6299.3499.06 GenTel-Shield(Ours)7.9299.4297.8996.42 ing transient leak attacks. The WhyLabs LangKit slightly outperformed our method in terms of accu- racy, precision, and F1 score, at 99.34%, 99.62%, and 99.34%, respectively. This shows that Why- Labs LangKit has strong performance in leaking attack detection. In comparison, our method strikes a balanced performance with a recall of 96.42%, precision of 99.42%, and F1 score of 97.89%, main- taining high detection capability while reducing false positives. Additionally, our model signifi- cantly outperforms all other models, except for WhyLabs LangKit, across various performance metrics. This demonstrates that our model is also among the top performers in leak attack detection. Overall, our model ranks among the best for de- tecting injection attacks, particularly in handling complex attack scenarios, showcasing exceptional classification capabilities. This highlights its strong potential for practical application in real-world set- tings. 5.2.2 Separate Performance on Attack Samples We evaluate the modelâs effectiveness on specific subsets of data. When the dataset consists solely of injection attack samples or specific scenes, we focus on accuracy as the key metric. This targeted evaluation allows us to understand how well the model performs on a particular type of data and understanding of the modelâs strengths and areas for improvement. To evaluate the modelâs effectiveness in detect- ing jailbreak attacks and target hijacking attacks across various scenarios, we conducted experi- ments inViolation of Personal Rights,Technology Misuse,Social lmpact,Serious Crime,Infringe- mentandEthical Violationsscenarios based on the security system outlined above. It is important to note that becausePrompt Guard,Deepsetand Fmopstends to classify all samples as injection Violation of Personal Rights Technology Misuse Social Impact Serious Crime Infringement Ethical Violations Violation of Personal Rights 82 88 94 Jailbreak GentelShield ProtectAI Hyperion Lakera AI Figure 3: Classification Results for Jailbreak Attack Detection in different scenarios. attacks, leading to a potential bias in comparison, we have chosen not to use them as a baseline in our experiment. Additionally, WhyLabs LangKit tends to misclassify injection attack samples as benign, resulting in poor performance on attack detection. As a result, we also exclude it from the experiment. As shown in Figure 3, in the jailbreak attack test in different scenarios,GenTel-Shieldemerges as the most balanced and robust model, suggesting its strong capability to handle a wide range of jail- break attack scenarios. ProtectAI also performs well, though it shows a slight dip in categories like "Ethical Violations," indicating room for improve- ment in handling more nuanced attack scenarios. Hyperion and Lakera AI show significant variabil- ity, indicating a need for further refinement to en- sure more consistent protection across different attack types. Figure 4 show that classification results for Hi- jacking Attack Detection in different scenarios. GenTel-Shieldemerges as the most balanced and reliable model, consistently performing well across all categories. ProtectAI, while strong in certain ar- eas like "Technology Misuse" and "Social Impact," has specific weaknesses that could affect its overall reliability. Hyperion shows particular strength in handling serious offenses but lacks consistency in other areas. Lakera AI, with its significant vari- ability, may require further refinement to improve its performance in dealing with ethical and techno- logical misuse challenges. Overall,GenTel-Shield appears to be the most versatile model for address- ing a wide range of hijacking attack scenarios. In order to explore the performance of different Violation of Personal Rights Technology Misuse Social Impact Serious Crime Infringement Ethical Violations Violation of Personal Rights 50 60 70 80 90 Hijacking GentelShield ProtectAI Hyperion Lakera AI Figure 4: Classification Results for Hijacking Attack Detection in different scenarios. models in various subdivision scenarios, we con- ducted a series of experiments.The experimental results presented in the Figure. 5 highlight the per- formance of five different modelsâGenTel-Shield, ProtectAI, Hyperion, Lakera AI, and WhyLabs LangKitâacross various risk categories, including Jailbreaking Attack Risks and Hijacking Attack Risks. GenTel-Shield consistently outperformed other models in most categories, achieving high accu- racy across several risk scenarios, including Vio- lence (R1), Privacy Violation (R4), and Cybercrime (R11). It maintained an overall performance rate near 0.99 in these categories. Similarly, ProtectAI showed strong performance in specific domains like Misinformation (R10) and Terrorism (R7), but slightly lagged behind in areas like Malware (R9). Lakera AI and Hyperion demonstrated moder- ate consistency, with Hyperion maintaining higher performance levels in areas such as Racism (R15) and Deepfake (R14). Lakera AI struggled in some categories, particularly Ageism (R17) and Homo- phobia (R19), where performance metrics dropped below 0.75. In summary, the experiments demonstrate that ourGenTel-Shieldmodel effectively detects injec- tion attacks across various security scenarios, high- lighting its strong practical value and versatility. 5.3Comparison Results on Different Datasets To demonstrate the generalization ability of the model, we conducted experiments on multiple datasets. Table 4 presents the classification perfor- mance of various models on the Jailbreak-LLM and Deita datasets. Jailbreak-LLM consists of 1,405 attack samples, while Deita contains 9,500 normal instructions. We merged these two datasets for the experiments. The results show that our model consistently out- performs all others across key metrics, particularly in Accuracy and F1 score, achieving 97.26% and 90.35%, respectively. While some models excel in specific areas, their overall performance remains subpar. For example, the Deepset model demon- strates a slightly higher Precision of 99.72%; how- ever, its Accuracy (50.99%) and F1 score (34.39%) are significantly lower than those of our model. This suggests that while Deepset is highly effective at identifying positive samples, it struggles with overall dataset balance. Similarly, the ProtectAI model shows relatively balanced performance across all metrics, yet its F1 score of 86.67%, Precision of 85.84%, and Ac- curacy of 96.60% still fall short of our modelâs results. Notably, the Hyperion model exhibits rel- atively poor performance, particularly in terms of Accuracy and Precision, highlighting its potential limitations when applied to jailbreak attack scenar- ios. Table 4: Classification performance on Comparison Results on Other Datasets. MethodAccuracyâPrecisionâF1âRecallâ ProtectAI96.6085.8486.6787.52 Hyperion55.1249.9022.2714.33 Prompt Guard20.5091.3222.8413.05 Lakera AI93.2182.2875.7570.19 Deepset50.9999.7234.3920.78 Fmops47.3999.2932.7219.59 WhyLabs LangKit94.5469.8976.7585.10 GenTel-Shield(Ours)97.2699.5790.3582.68 5.4 Comparison Results on Different Language To assess the modelâs generalization performance across different language datasets, we used Google Translate to translate the datasets for jailbreak at- tacks, target hijacking attacks, and prompt leakage scenarios into Chinese, Japanese, French, Spanish, and German. The translated datasets were then used for testing. Figure 6 illustrates the defensive performance of various injection attack detection models in the jail- break attack scenario. The results highlight the dif- ferences in generalization ability across languages for different models. For instance, while Lakera AI GenTel-Shield ProtectAI Hyperion Lakera AI 0.980.970.970.980.980.980.980.980.980.960.980.980.980.980.980.970.950.970.970.970.970.970.980.970.980.980.970.97 0.90.90.90.90.890.890.890.920.910.90.910.890.890.890.870.90.890.880.880.890.910.90.890.90.90.920.890.9 0.950.950.950.950.950.950.950.950.950.920.950.940.950.950.950.950.90.910.950.950.950.940.950.950.950.950.940.95 0.880.860.880.880.880.880.880.880.860.850.870.880.870.880.880.870.850.870.880.870.870.860.890.870.860.90.870.87 R1: ViolenceR2: Self-HarmR3: Verbal Abuse R4: Privacy Violation R5: Human TraffickingR6: Drugs R7: Terrorism R8: National Security ViolationsR9: MalwareR10: MisinformationR11: Cybercrime R12: Pornography R13: Cyberbullying R14: DeepfakeR15: RacismR16: Ableism R17: Ageism R18: Sexism R19: HomophobiaR20: TransphobiaR21: Social Panic R22: Manipulation of Public Opinion R23: Ecological Damage R24: Animal Cruelty R25: Intellectual Property Infringement R26: Trade Secret Misappropriation R27: Plagrism R28: Antitrust Violations GenTel-Shield ProtectAI Hyperion Lakera AI 0.990.960.960.990.980.980.970.980.980.930.990.980.990.990.960.930.840.940.950.970.920.950.940.980.990.990.910.96 0.950.940.930.970.930.940.920.950.960.940.960.9210.940.930.930.940.90.930.930.950.940.940.940.950.960.930.94 0.970.920.970.970.970.950.910.90.970.680.970.90.970.970.970.930.670.680.970.970.850.790.970.870.970.910.850.96 0.80.690.80.790.810.770.750.750.720.710.810.740.710.770.750.740.680.680.740.740.70.690.760.780.70.780.780.72 0.60 0.90 0.95 1.00 Jailbreaking Attack Risks Hijacking Attack Risks Figure 5: Identification accuracy under different risks. Chinese Japanese French Spanish German 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Jailbreak Attack GenTel-Shield ProtectAI Hyperion Prompt Guard Lakera AI Figure 6: Jailbreak experiment on different language. performs well in French, its effectiveness is lower in other languages. In contrast, our model, GenTel- Shield, demonstrates top-tier performance across all languages, with jailbreak detection accuracy in Chinese, Japanese, French, and Spanish approach- ing nearly 100%. Figure 7 highlights the variations in detection model performance for target hijacking attacks across different languages. GentelShield demon- strates strong defense capabilities, achieving the highest accuracy in Chinese and Japanese. Protec- tAI performs exceptionally well in French, Spanish, and German, even surpassing GentelShield in these languages. Hyperion exhibits a more balanced per- formance. Lakera AI performs well in Spanish and German but struggles in other languages, es- Chinese Japanese French Spanish German 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Goal Hijacking Attack GenTel-Shield ProtectAI Hyperion Prompt Guard Lakera AI Figure 7: Goal hijacking experiment on different lan- guage. pecially Japanese. Overall, the performance of GentelShield is similar to that of ProtectAI, with outstanding results in Chinese and Japanese. Based on the experimental results Figure 8 pro- vided, GentelShield stands out with the highest accuracy rates across four languagesâChinese, Japanese, French, and Spanishâwith its peak per- formance in Chinese at an impressive 97.36%. This indicates that GentelShield has a high level of ac- curacy and stability when handling multilingual tasks. The ProtectAI model follows closely behind, demonstrating high accuracy rates in all tested lan- guages except for French, with its best performance in Spanish, where it achieved an accuracy rate of 95.19%, just behind GentelShield. This sug- Chinese Japanese French Spanish German 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Prompt Leaking Attack GenTel-Shield ProtectAI Hyperion Prompt Guard Lakera AI Figure 8: Prompt leaking experiment on different lan- guage. gests that ProtectAI is a strong contender and may have specific advantages in certain languages. The MetaPromptGuard model shows significant fluctu- ations in accuracy rates, ranging from 44.53% in German to 71.04% in Japanese, indicating incon- sistent performance across different languages. Hy- perion demonstrates a more balanced performance, with an accuracy rate of 77.92% in Japanese, sec- ond only to GentelShield. 5.5 Comparison Experiment with Different Input Token Lengths To examine the impact of varying input token lengths on the modelâs performance, we conducted tests using token lengths of 128, 256, 384, and 512, respectively. Since WhyLabs LangKit and Lakera AI are accessed via API calls and do not allow for modification of the token length, these two models are excluded from the following experiments. As shown in Figure 9, on the jailbreak attack test dataset, ourGentel-Shieldconsistently outperforms other models, regardless of the token length setting. An interesting observation is that when the input token length increases from 128 to 256, the perfor- mance of some models decreases to a certain extent. This may be because a token length of 128 captures only partial jailbreak attack templates or security issues, leading to higher variability in the modelâs predictions. However, when the token length is increased to 384, the performance of most mod- els improves significantly, as 384 tokens can cover most of the jailbreak templates and related issues, allowing the models to process a more complete prompt. Beyond 384 tokens, further increasing the token length to 512 results in minimal changes in 128256384512 50.0 52.5 93.5 100.0 GenTel-Shield ProtectAI Hyperion Prompt Guard Accuracy Jailbreak Attack Figure 9: Jailbreak experiment on Token length. performance. As shown in Figure 10, this phenomenon is more pronounced in the dataset with prompt word leak- age. Since prompt leakage attacks are more diverse than jailbreak attacks, the additional prompts do not consistently appear at the beginning or end of the input. This variability makes it harder for models to accurately detect prompt leakage, further highlighting the impact of token length on perfor- mance. 128256384512 48.0 49.5 95 100 GenTel-Shield ProtectAI Hyperion Prompt Guard Accuracy Prompt Leaking Attack Figure 10: Prompt leaking experiment on Token length. In summary, the input token length limit affects model performance to some extent, but once the token length reaches a certain threshold, its impact diminishes, and the performance of each model tends to stabilize. 6 Conclusion In this work, we present a comprehensive frame- work for training models to defend against prompt injection attacks, calledGenTel-Shield, and intro- duceGentel-Benchfor assessing defending meth- odsâ effectiveness. Our study demonstratesGenTel- Shieldâs robust capability in identifying attack and benign application cues. Comprehensive evalua- tions across three distinct attack scenarios under- score the versatility and effectiveness ofGenTel- Shield. We hope our proposed LLM security risk benchmark and defense shield will inspire further research into diversifying LLM defending strate- gies. References Alespalla. 2023. alespalla/chatbot_instruction_prompts. https://huggingface.co/datasets/alespall a/chatbot_instruction_prompts . Accessed: 2024-08-30. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In30th USENIX Security Symposium (USENIX Security 21), pages 2633â2650. Tianyu Cui, Yanling Wang, Chuanpu Fu, Yong Xiao, Sijia Li, Xinhao Deng, Yunpeng Liu, Qinglin Zhang, Ziyi Qiu, Peiyang Li, et al. 2024.Risk taxon- omy, mitigation, and assessment benchmarks of large language model systems.arXiv preprint arXiv:2401.05778. Deepset. 2023. distilbert-prompt-injection. Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. 2023. Jailbreaker: Automated jail- break across multiple large language model chatbots. CoRR, abs/2307.08715. Epivolis. 2024. Hyperion.https://huggingface.co /Epivolis/Hyperion. Luciano Floridi and Massimo Chiriatti. 2020. Gpt-3: Its nature, scope, limits, and consequences.Minds and Machines, 30:681â694. Fmops. 2023. deberta-v3-base-injection. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what youâve signed up for: Compromis- ing real-world llm-integrated applications with indi- rect prompt injection. InProceedings of the 16th ACM Workshop on Artificial Intelligence and Secu- rity, pages 79â90. Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu. 2024. Cold-attack: Jailbreaking llms with stealthiness and controllability.arXiv preprint arXiv:2402.08679. Yihao Huang, Chong Wang, Xiaojun Jia, Qing Guo, Felix Juefei-Xu, Jian Zhang, Geguang Pu, and Yang Liu. 2024. Semantic-guided prompt organization for universal goal hijacking against llms.arXiv preprint arXiv:2405.14189. Bo Hui, Haolin Yuan, Neil Gong, Philippe Burlina, and Yinzhi Cao. 2024. Pleak: Prompt leaking attacks against large language model applications.arXiv preprint arXiv:2405.06823. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations.arXiv preprint arXiv:2312.06674. Katharina Jeblick, Balthasar Schachtner, Jakob Dexl, Andreas Mittermeier, Anna Theresa StĂźber, Johanna Topalis, Tobias Weber, Philipp Wesp, Bastian Oliver Sabel, Jens Ricke, et al. 2024.Chatgpt makes medicine easy to swallow: an exploratory case study on simplified radiology reports.European radiology, 34(5):2817â2825. Joonhyun Jeong. 2023. Hijacking context in large multi- modal models.arXiv preprint arXiv:2312.07553. Jinhwa Kim, Ali Derakhshan, and Ian G Harris. 2023. Robust safety classifier for large language mod- els: Adversarial prompt shield.arXiv preprint arXiv:2311.00172. lakera.ai. 2024. Lakera-guard. Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song. 2023.Multi- step jailbreaking privacy attacks on chatgpt.arXiv preprint arXiv:2304.05197. Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wang- meng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. 2024. Salad-bench: A hierarchical and comprehen- sive safety benchmark for large language models. arXiv preprint arXiv:2402.05044. Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang. 2023. Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-AI conversation. InThe 2023 Conference on Empirical Methods in Natural Language Processing. Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. 2023a. What makes good data for alignment?a comprehensive study of auto- matic data selection in instruction tuning.Preprint, arXiv:2312.15685. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2023b. Autodan: Generating stealthy jailbreak prompts on aligned large language models.arXiv preprint arXiv:2310.04451. Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, and Chaowei Xiao. 2024. Automatic and universal prompt injection attacks against large language mod- els.arXiv preprint arXiv:2403.04957. Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, Kailong Wang, and Yang Liu. 2023c. Jailbreaking chatgpt via prompt engineering: An empirical study. arXiv preprint arXiv:2305.13860. Alejandro Lopez-Lira and Yuehua Tang. 2023. Can chatgpt forecast stock price movements? return pre- dictability and large language models.arXiv preprint arXiv:2304.07619. Babak Mahjour, Jillian Hoffstadt, and Tim Cernak. 2023. Designing chemical reaction arrays using phactor and chatgpt.Organic Process Research & Development, 27(8):1510â1516. Todor Markov, Chong Zhang, Sandhini Agarwal, Tyna Eloundou, Teddy Lee, Steven Adler, Angela Jiang, and Lilian Weng. 2023. A holistic approach to un- desired content detection in the real world.Preprint, arXiv:2208.03274. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. 2024. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal.arXiv preprint arXiv:2402.04249. meta.com. 2024. Prompt-guard-86m. MITRE ATLAS. 2024. Atlas matrix.https://atlas. mitre.org/matrices/ATLAS. Philip Moons and Liesbet Van Bulck. 2023. Chatgpt: can artificial intelligence language models be of value for cardiovascular nurses and allied health profession- als. OWASP. 2024. Owasp top 10 list for large language models version 1.1.https://owasp.org/w-pro ject-top-10-for-large-language-model-app lications/. FĂĄbio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models. In NeurIPS ML Safety Workshop. ProtectAI.com. 2024. Fine-tuned deberta-v3-base for prompt injection detection. Jerret Ross, Brian Belgodere, Vijil Chenthamarakshan, Inkit Padhi, Youssef Mroueh, and Payel Das. 2022. Large-scale chemical language representations cap- ture molecular structure and properties.Nature Ma- chine Intelligence, 4(12):1256â1264. Paul RĂśttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2023. Xstest: A test suite for identifying exag- gerated safety behaviours in large language models. arXiv preprint arXiv:2308.01263. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2023. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models.arXiv preprint arXiv:2308.03825. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. âDo Anything Nowâ: Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models.InACM SIGSAC Conference on Computer and Communica- tions Security (CCS). ACM. Hao Sun, Zhexin Zhang, Jiawen Deng, Jiale Cheng, and Minlie Huang. 2023.Safety assessment of chinese large language models.arXiv preprint arXiv:2304.10436. VMware. 2023. Vmware/open-instruct.https://hugg ingface.co/datasets/VMware/open-instruct. Accessed: 2024-08-30. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilin- gual e5 text embeddings: A technical report.arXiv preprint arXiv:2402.05672. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2024. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36. Jason Wei and Kai Zou. 2019. Eda: Easy data augmenta- tion techniques for boosting performance on text clas- sification tasks.arXiv preprint arXiv:1901.11196. WhyLabs. 2024. Whylabs langkit. Yueqi Xie, Minghong Fang, Renjie Pi, and Neil Gong. 2024. Gradsafe: Detecting jailbreak prompts for llms via safety-critical gradient analysis. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 507â518. Zihao Xu, Yi Liu, Gelei Deng, Yuekang Li, and Stjepan Picek. 2024. A comprehensive study of jailbreak attack versus defense for large language models. In Findings of the Association for Computational Lin- guistics ACL 2024, pages 7432â7449. Kai-Cheng Yang and Filippo Menczer. 2023. Large lan- guage models can rate news outlet credibility.arXiv preprint arXiv:2304.00228. Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, et al. 2024. R- judge: Benchmarking safety risk awareness for llm agents.arXiv preprint arXiv:2401.10019. Yiming Zhang and Daphne Ippolito. 2023. Prompts should not be seen as secrets: Systematically measur- ing prompt extraction attack success.arXiv preprint arXiv:2307.06865. Zhexin Zhang, Junxiao Yang, Pei Ke, and Minlie Huang. 2023. Defending large language models against jail- breaking attacks through goal prioritization.arXiv preprint arXiv:2311.09096. Xuandong Zhao, Xianjun Yang, Tianyu Pang, Chao Du, Lei Li, Yu-Xiang Wang, and William Yang Wang. 2024. Weak-to-strong jailbreaking on large language models.arXiv preprint arXiv:2401.17256. Jun-Jie Zhu, Jinyue Jiang, Meiqi Yang, and Zhiyong Ja- son Ren. 2023. Chatgpt and environmental research. Environmental Science & Technology, 57(46):17667â 17670. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Univer- sal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043.