Paper deep dive
LLM Jailbreak Detection for (Almost) Free!
Guorui Chen, Yifan Xia, Xiaojun Jia, Zhijiang Li, Philip Torr, Jindong Gu
Models: ChatGPT-3.5, Guanaco, Llama2-7B, Llama3, Vicuna
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 98%
Last extracted: 3/12/2026, 6:43:15 PM
Summary
The paper introduces 'Free Jailbreak Detection' (FJD), a novel, computationally efficient method for detecting jailbreak attacks on Large Language Models (LLMs). FJD leverages the observation that jailbreak prompts cause lower confidence in the first generated token compared to benign prompts. The method enhances this signal by prepending affirmative instructions and applying temperature scaling to logits. An improved variant, FJD-LI, utilizes learnable virtual instructions to further optimize detection performance without increasing inference costs.
Entities (5)
Relation Signals (3)
Temperature Scaling → partof → FJD
confidence 100% · FJD... where two techniques are introduced... Affirmative Instruction Prepending and Temperature Scaling
FJD → detects → Jailbreak Attack
confidence 95% · FJD can effectively detect jailbreak prompts with almost no additional computational costs
FJD-LI → improves → FJD
confidence 95% · we enhance the detection performance of FJD through the integration of virtual instruction learning
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models (LLMs) enhance security through alignment when widely used, but remain susceptible to jailbreak attacks capable of producing inappropriate content. Jailbreak detection methods show promise in mitigating jailbreak attacks through the assistance of other models or multiple model inferences. However, existing methods entail significant computational costs. In this paper, we first present a finding that the difference in output distributions between jailbreak and benign prompts can be employed for detecting jailbreak prompts. Based on this finding, we propose a Free Jailbreak Detection (FJD) which prepends an affirmative instruction to the input and scales the logits by temperature to further distinguish between jailbreak and benign prompts through the confidence of the first token. Furthermore, we enhance the detection performance of FJD through the integration of virtual instruction learning. Extensive experiments on aligned LLMs show that our FJD can effectively detect jailbreak prompts with almost no additional computational costs during LLM inference.
Tags
Links
- Source: https://arxiv.org/abs/2509.14558
- Canonical: https://arxiv.org/abs/2509.14558
- Code: https://github.com/GuoruiC/FJD
Trouble viewing inline? Open PDF directly →
Full Text
126,319 characters extracted from source content.
Expand or collapse full text
LLM Jailbreak Detection for (Almost) Free! Guorui Chen1, Yifan Xia1, Xiaojun Jia2, Zhijiang Li1,† , Philip Torr3, Jindong Gu3,† 1School of Information Management, Wuhan University, Wuhan, China 2Nanyang Technological University, Singapore 3Torr Vision Group, University of Oxford, Oxford, United Kingdom Abstract Large language models (LLMs) enhance security through alignment when widely used, but remain susceptible to jailbreak attacks capable of producing inappropriate content. Jailbreak detection methods show promise in mitigating jailbreak attacks through the assistance of other models or multiple model inferences. However, existing methods entail significant computational costs. In this paper, we first present a finding that the difference in output distributions between jailbreak and benign prompts can be employed for detecting jailbreak prompts. Based on this finding, we propose a Free Jailbreak Detection (FJD) 111https://github.com/GuoruiC/FJD which prepends an affirmative instruction to the input and scales the logits by temperature to further distinguish between jailbreak and benign prompts through the confidence of the first token. Furthermore, we enhance the detection performance of FJD through the integration of virtual instruction learning. Extensive experiments on aligned LLMs show that our FJD can effectively detect jailbreak prompts with almost no additional computational costs during LLM inference. † corresponding authors: lizhijiang@whu.edu.cn, jindong.gu@outlook.com. 1 Introduction Large language models (LLMs) achieve remarkable success across various domains and tasks. However, the widespread use of these models has also exposed concerns, particularly their potential to generate inappropriate content. To address the concerns, recent work (Wu et al., 2021; Ouyang et al., 2022; Rafailov et al., 2024) employs diverse training strategies and principles to align LLMs with human values to enhance their safety and generate responsible responses. Despite these efforts, recent jailbreak attacks can still bypass the alignment and cause harmful responses from LLMs through manual crafting (Li et al., 2023a; Liu et al., 2023b; Chen et al., 2024; Yuan et al., 2023; Deng et al., 2023b; Ding et al., 2023; Perez and Ribeiro, 2022; Shah et al., 2023; Li et al., 2023b) or automated generation of prompts (Zou et al., 2023; Liu et al., 2023a; Chao et al., 2023; Carlini et al., 2024; Jones et al., 2023; Wen et al., 2024; Wichers et al., 2024; Lapid et al., 2023; Li et al., 2024b; Qi et al., 2023; Deng et al., 2023a). Recently, there have been emerging efforts to mitigate the risks associated with jailbreak attacks. One of the important mitigation strategies is to detect jailbreak queries that trigger LLMs to generate harmful content. Specifically, basic detection methods can be classified into three types. The first type involves computing the perplexity score of input text using an auxiliary model to detect jailbreak prompts (Alon and Kamfonas, 2023; Jain et al., 2023). The second type mutates the input into multiple copies and aggregates the responses from these copies to detect jailbreak prompts (Robey et al., 2023; Zeng et al., 2024). The third type detects outputs of jailbreak prompts with an additional classifier or the underlying model itself (Yuan et al., 2024; Helbling et al., 2023). However, these methods require expensive computational costs, necessitating either additional models for assistance or multiple model inferences. Wei et al. (2024) categorizes current jailbreaks into two types: jailbreaks with competing objectives and mismatched generalization. The first type forces the LLM to choose between safety alignment behaviors and harmful instruction objectives. The second type comes from observing that pretraining is done on a large and more diverse datasets than safety training. This mismatch can be exploited for jailbreaks. By analyzing inference outputs of the jailbreak and benign prompts, we observe that there is an obvious difference in the confidence of the first token between the responses generated by these prompts and benign ones. For both type of jailbreak prompts, they cause LLMs to have some confusion during inference, resulting in less confident responses than that on benign prompts. Based on the initial finding, we propose a (almost) Free Jailbreak Detection (FJD) method where two techniques are introduced, Affirmative Instruction Prepending and Temperature Scaling. Affirmative Instruction Prepending prepends an affirmative instruction (e.g. "You are a good Assistant.") to the query. The prepended instruction has minimal impact on the final output content. The output of the prepended query can be directly taken as the final output of the original query. Meanwhile, the prepended affirmative instruction can increase the response confidence of LLM to benign prompt, while it bring less or even reduce the confidence of LLM. Thus, Affirmative Instruction Prepending can be used to better detect jailbreak prompts. However, some LLMs, such as Llama, can be overconfident with responses to both jailbreak and benign prompts (the maximal probability of the first token could be very close to 1.0). Hence we introduce Temperature Scaling to better distinguishing the jailbreak and benign prompts. Furthermore, instead of prepending a manually selected instruction for FJD, we propose to learn a virtual instruction to improve detection performance, dubbed FJD-LI. Extensive experiments are conducted to verify our observations and proposal. The effectiveness of our detection method is verified on aligned LLMs such as Vicuna (Chiang et al., 2023), Llama2 (Touvron et al., 2023), and Guanaco (Dettmers et al., 2024) under various jailbreak attacks. Furthermore, we show the effectiveness of our FJD against transferable jailbreak attacks to Llama3 222https://github.com/meta-llama/llama3 and ChatGPT3.5 (Achiam et al., 2023). Our detection method outperforms the baseline methods significantly and requires almost no additional computational costs during LLM inference. Our contributions can be summarized as follows: • We present a finding that the difference in output distributions between jailbreak and benign prompts can be employed for detecting jailbreak prompts. • Based on observation, We propose a Free Jailbreak Detection (FJD) method by prepending affirmative instructions into the inputs and scaling the logits by temperature which requires almost no additional costs. • Furthermore, we propose to learn virtual instructions (FJD-LI) to further improve jailbreak detection performance. • Extensive experiments are conducted under various jailbreak attacks with competing objectives and mismatched generalization. (a) AutoDAN vs. Benign Prompt in Llama2 7B (b) Cipher vs. Benign Prompt in Llama2 7B Figure 1: The distribution of the confidence scores of the predicted first tokens over jailbreak and benign samples is shown. A difference can be observed where LLMs are less confident on Jailbreak samples than on benign samples. 2 Related Work Jailbreak Attack Jailbreak attacks can mislead LLMs to respond to harmful queries. These works (Albert, 2023; walkerspider, 2022) initially reported that hand-crafted prompts can jailbreak LLMs. Currently, jailbreak attacks against LLMs can be divided into two categories: competing objectives and mismatched generalization (Wei et al., 2024). The first category forces the LLM to choose between safety training behaviors and harmful instruction objectives by crafting prompts. E.g., GCG (Zou et al., 2023) automatically generate transferable adversarial suffixes by employing gradient-based search methods. AutoDAN (Liu et al., 2023a) employed mutation and crossover operations within genetic algorithms to produce natural adversarial prefixes. The second category exploits data beyond the safety fine-tuning of the LLMs for jailbreak attacks. E.g., Yong et al. (Yong et al., 2023) achieved LLMs jailbreak by devising strategies that convert user prompts into low-resource languages. In contrast to hand-crafted methods, Cipher (Yuan et al., 2023) uses system role descriptions and few-shot enciphered demonstrations to bypass the safety alignment. As LLMs grow in complexity and capability, more jailbreak attacks (Jia et al., 2024; Liu et al., 2023b; Wei et al., 2024; Ding et al., 2023; Chao et al., 2023; Zhang and Wei, 2024; Paulus et al., 2024) based on those methods have been developed. Jailbreak Defense and Detection To deal with jailbreak attacks on aligned LLMs, defense methods aim to reduce the success rate of the attack, while detection methods distinguish between jailbreak and benign prompts to safeguard LLMs. Current defense and detection methods can be divided into three types. The first type, a simple and effective method (Alon and Kamfonas, 2023; Jain et al., 2023), involves computing the perplexity score of the input for detection by employing the negative log-likelihood. In addition, to enable LLMs to produce inappropriate responses, attackers must carefully craft the jailbreak prompt. Consequently, the second type (Robey et al., 2023; Zhang et al., 2023a; Cao et al., 2023; Zhang et al., 2023b; Kumar et al., 2023; Rao et al., 2023) generate multiple copies by randomly deleting, replacing, or modifying consecutive character, and aggregate the responses from multiple LLMs to mitigate the success rate of the attack. And the third type (Yuan et al., 2024; Helbling et al., 2023; Xie et al., 2023) employ an additional classifier model or LLMs itself to detect jailbreak prompts such as appending the prompt "Is it harmful?" to the response or modifying the system prompt of LLM. Current defense and detection methods necessitate extra model inferences, resulting in significant computational costs. In this work, we propose a nearly free jailbreak detection method, which is a confidence-based method. Similarly, existing confidence-based methods (Xu et al., 2024; Candogan et al., 2025) also fall into the aforementioned three categories and require auxiliary models and additional inference during detection or defense. 3 Approach In this section, we describe the problem formulation in Sec. 3.1, and introduce our proposed methods FJD with Affirmative Instruction Prepending and Temperature Scaling in Sec. 3.2 and the variants of FJD in Sec. 3.3. 3.1 Problem Formulation Jailbreak attacks can be classified into two categories: competing objectives and mismatched generalization (Wei et al., 2024). Competing Objectives Jailbreak attacks (Zou et al., 2023; Liu et al., 2023a) are designed to search for some jailbreak prompt xjailx_jail so that the probability of harmful output g g is maximized, which forces the LLM to choose between safety training behaviors and harmful instruction objectives. Formally, given an input sequence of tokens xqx_q, the attack can be formulated as minimizing the loss between model output and the target output, minxjail∈[||]nℒ(p(xq⊕xjail),g^)min_x_jail∈[|V|]^n\ L(p(x_q x_jail), g), where ⊕ is defined as the concatenation operator of two sequence as: xq⊕xjailx_q x_jail, p(⋅)p(·) represents the output probabilities predicted by LLMs, V is the vocabulary, and n is the length of tokens. Mismatched Generalization This type of method (Yuan et al., 2023; Chen et al., 2024) comes from observing that pretraining is done on a large and more diverse datasets than safety training. For this mismatch, LLM will respond without safety considerations, such as Base64 on inputs. Jailbreak detection approaches distinguishes between jailbreak and benign prompts using a specific metric. For a given input sequence, a benign query xbenix_beni or a jailbreak query xjailx_jail, the jailbreak detector g(⋅) g(·) aims to achieve this property: g(xjail)<T≤g(xbeni) g(x_jail)<T≤ g(x_beni) or g(xjail)>T≥g(xbeni) g(x_jail)>T≥ g(x_beni), where TT represents a pre-defined threshold. 3.2 Free Jailbreak Detection Approach Current jailbreak attacks can be classified into two categories: competing objectives and mismatched generalization. Both might impact the confidence generated by LLMs. As shown in Fig. 1, we conduct a statistical analysis on the first token confidence produced by jailbreak prompts (AutoDAN and Cipher) and benign ones (PureDove) on Llama2 7B. We find that there is an obvious difference in the confidence of the first token between the responses generated by these prompts and benign ones. Similar observations on other models and the theoretical analysis are shown in Appendix D. Figure 2: Jailbreak prompt Detection through FJD: By prepending an affirmative instruction and scaling the logits with temperature, the first token confidence in the LLMs’ responses to the benign prompts is higher than a predefined threshold, whereas the confidence for jailbreak prompts can be lower than the threshold. Based on the findings, we identify the potential of utilizing the confidence of the first tokens to detect jailbreak prompts. Since the output probabilities can be obtained in the standard forward pass, we dub our method Free Jailbreak Detection (FJD), where two techniques are introduced to enlarge the confidence difference, i.e., Affirmative Instruction Prepending and Temperature Scaling. We now present how the two techniques improve detection performance. Affirmative Instruction Prepending This technique prepends an affirmative instruction to the given query to enlarge the confidence differences between jailbreak and benign prompts. Affirmative Instruction is referred as the ones that confirm the original capability of LLMs e.g., "You are a good Assistant.", "Please following user instructions accurately.", which is widely adopted in various applications—for instance, incorporating them into system prompts to enhance model reasoning performance. With such prepended instructions, the outputs of benign samples are similar or even better than before, which can be sent to user directly without a second inference. Meanwhile, the confidence of the predicted first token (i.e., the maximal probability over vocabulary) on benign prompts increases when equipped with an affirmative instruction. Compared to that on benign prompts, the increased confidence on jailbreak is minor. The reason behind is that affirmative instructions prepended to jailbreak prompts receive less attention in LLMs given the fact that jailbreak prompts attract model attentions significantly (Arditi et al., 2024). Namely, without impairing model outputs on benign prompts, the difference of the first token confidence between jailbreak and benign prompts can be enlarged by prepending affirmative instructions. More discussion is in Sec. 4.5. Formally, given an input sequence xqx_q and an affirmative instruction xaix_ai, the procedure for detecting jailbreak prompts is as follows. The confidence of the first tokens is computed as P1=σ(f1(xai⊕xq))P_1=σ(f_1(x_ai x_q)) (1) where, fi(⋅)f_i(·) represents the output logits of the i-th token, and σ(⋅)σ(·) obtains the maximal probability value over the vocabulary tokens through the softmax function. Temperature Scaling Prepended Affirmative Instructions enlarge the confidence difference by increasing confidence differently on jailbreak and benign prompts. However, it does not work well when LLMs are overconfident with responses. In our experiments, we also observe that LLMs (e.g. Llama) can be overconfident on both jailbreak and benign prompts where the maximal probability of the first token could be even very close to 1.0. To address the challenge, we propose to apply temperature scaling to avoid overconfident outputs. To illustrate why temperature scaling can change the confidence rank between two samples, we provide a dummy example: Given the sample A with the output logits [10, 9, 1] and the sample B with [10, 8, 8], their output probabilities are [0.731, 0.269, 0.0001] and [0.787, 0.106, 0.106] respectively when the temperature of the softmax function is set to 1.0. Namely, model responses are more confident about sample B (0.787) than sample A (0.731). After temperature scaling by setting the temperature to 2.0, their output probabilities become [0.619, 0.375, 0.007] and [0.576, 0.212, 0.212] respectively where the confidence of sample B become lower than that of sample A. More rigorous analysis and an instance are in Appendix M. Formally, given an input sequence xqx_q, the affirmative instruction xaix_ai and the temperature τ, the confidence of the first tokens with temperature scaling is computed as P1,τ=στ(f1(xai⊕xq)/τ)P_1,τ= _τ(f_1(x_ai x_q)/τ) (2) where, fi(⋅)f_i(·) represents the output logits of the i-th token, and στ(⋅) _τ(·) obtains the maximal probability value over the vocabulary tokens through the softmax function with temperature scaling. Then, the confidence P1,τP_1,τ can be used to detect jailbreak prompts by comparing it with a predefined threshold. If P1,τ<TP_1,τ<T, the input will be flagged as a jailbreak prompt. Otherwise, it will be flagged as a benign prompt allowing LLMs to output. Note that we apply AUC score for experimental evaluation where all the thresholds are considered. The detection process of FJD can be integrated into the standard model forward inference. As the affirmative instructions prepended by FJD are short and the temperature scaling has no influence on model inference, the additional computational costs of model inference is almost free. In contrast, previous jailbreak detection methods require one or many extra forward passes. 3.3 Improved Version based on FJD Although various affirmative instructions of FJD works well across various models and jailbreak attacks, the careful selection of the instruction can still further improve detection performance. Instead of manual design, we introduce a learnable virtual instruction built upon FJD (FJD-LI). Formally, given an input sequence xqx_q, the affirmative instruction xaix_ai and the tokenization function E(x)E(x), the embedding of xqx_q and xaix_ai is eq=E(xq);emi=E(xai)e_q=E(x_q);\ e_mi=E(x_ai), where eq∈ℝq×de_q ^q× d and emi∈ℝm×de_mi ^m× d, q and m are the number of tokens and d is the number of embedding dimensions. The goal of the instruction learning is to minimize token confidence for jailbreak prompts and maximize it for benign prompts. We keep emie_mi learnable and update it with the loss which can be expressed as follows ℒ(eq)=KL(p1(emi⊕eq)∥Mo(l)),ifeq∈E(Xbeni)KL(p1(emi⊕eq)∥Mu(l)),ifeq∈E(Xjail) splitL(e_q)= cases KL(p_1(e_mi e_q)\| M_o(l)),\ if\ e_q∈ E(X_beni)\\ KL(p_1(e_mi e_q)\| M_u(l)),\ if\ e_q∈ E(X_jail)\\ cases split where, KL(⋅∥⋅) KL(·\|·) is to calculate the Kullback-Leibler Divergence (Kullback and Leibler, 1951) and l is the length of the vocabulary. p1(⋅)p_1(·) represents the output probability distribution of the first token. Mo(l)∈ℝ1×l M_o(l) ^1× l is a one-hot matrix of l dimensions, where the position of the maximum value in the logits p(eq)1p(e_q)_1 is set to 1 and the rest to 0. Mu(l)∈ℝ1×l M_u(l) ^1× l is a uniform distribution of l dimensions. The final virtual instruction is eli=minemi∈ℝm×dℒ(eq)e_li= _e_mi ^m× dL(e_q). Once elie_li is obtained, FJD-LI can be applied to detect jailbreak prompts by replacing emie_mi with elie_li in detection process. It requires only a small number of samples for learning and does not increase the inference costs of LLMs compared to FJD. 4 Experiment In this section, we first evaluate FJD under various jailbreak attacks and conduct ablation analysis of FJD. We then evaluate the detection effectiveness of FJD-LI. Finally, we discuss the efficiency, detection-aware jailbreak attack of FJD. 4.1 Experimental Setting Large language models Six open-source LLMs are taken for the jailbreak detection: Vicuna 7B/13B (Chiang et al., 2023), Llama2-chat 7B/13B (Touvron et al., 2023) and Guanaco 7B/13B (Dettmers et al., 2024). We further evaluate the detection of transferable jailbreak attacks on Llama3 and ChatGPT3.5 (Achiam et al., 2023). Dataset To evaluate the performance of FJD, we consider the jailbreak datasets AdvBench (Zou et al., 2023), and PureDove (Daniele and Suphavadeeprasit, 2023), Open-Platypus (Lee et al., 2023) and SuperGLUE (Wang et al., 2019) as benign datasets. To align benign prompts with jailbreak ones, we randomly select an equal number of benign prompts from the datasets. Then we allocate 50% of the dataset as the training set for training the virtual instruction in FJD-LI. More details about dataset are in Appendix A. Jailbreak attacks Two types of jailbreak attacks are considered, i.e., 1) via competing objectives (CO): AutoDAN (Liu et al., 2023a) and Hand (CO) (Chen et al., 2024). and 2) via mismatched generalization (MG): Cipher (Yuan et al., 2023) and Hand (MG). Note that Hand-crafted attacks provide 28 different attacks. Based on this work (Wei et al., 2024), the 28 attacks are grounded into Hand (CO) and Hand (MG). Additional information regarding the classification and detection results of hand-crafted attacks can be found in the Appendix I. We further consider transferable jailbreak attacks including the aggregation the prompt from GCG (Zou et al., 2023) and AutoDAN. And more details are in Appendix B. Bselines We compare our method with three jailbreak detection methods: PPL (Alon and Kamfonas, 2023), SmoothLLM (Robey et al., 2023) and GradSafe (Xie et al., 2024). More details about Baselines are in Appendix C. Metric In all experiments, AUC scores of detections are reported where all the thresholds are considered. The higher the score is, the better the detection performance is. We randomly select 80% of the test dataset and conduct 5 repeated experiments. More metrics (FPR, TPR, F1) are also reported in Appendix G and H. Table 1: Detection results (AUC) of jailbreak prompt under attacks via competing objectives. FJD outperforms the baseline in all attacks and LLMs with almost no additional computational costs during LLM inference. Attack Method Llama2-7B Vicuna-7B Guanaco-7B Llama2-13B Vicuna-13B Guanaco-13B AutoDAN PPL 0.8172±0.0017± 0.0017 0.7452±0.0012± 0.0012 0.7964±0.0004± 0.0004 0.7018±0.0002± 0.0002 0.7889±0.0002± 0.0002 0.7703±0.0005± 0.0005 SMLLM 0.8197±0.0052± 0.0052 0.7831±0.0035± 0.0035 0.6704±0.0036± 0.0036 0.8360±0.0021± 0.0021 0.5116±0.0044± 0.0044 0.5583±0.0038± 0.0038 GradSafe 0.8025±0.0089± 0.0089 0.7893±0.0020± 0.0020 0.8194±0.0051± 0.0051 0.9123±0.0029± 0.0029 0.9225±0.0005± 0.0005 0.7398±0.0063± 0.0063 FT 0.8869±0.0149± 0.0149 0.1709±0.0083± 0.0083 0.7084±0.0106± 0.0106 0.8899±0.0141± 0.0141 0.0471±0.0040± 0.0040 0.7710±0.0172± 0.0172 FJD 0.9578±0.0088± 0.0088 0.7964±0.0182± 0.0182 0.8946±0.0065± 0.0065 0.9214±0.0133± 0.0133 0.9373±0.0111± 0.0111 0.7470±0.0135± 0.0135 Hand (CO) PPL 0.5326±0.0025± 0.0025 0.5304±0.0007± 0.0007 0.5255±0.0005± 0.0005 0.5259±0.0023± 0.0023 0.5287±0.0006± 0.0006 0.4909±0.0007± 0.0007 SMLLM 0.7129±0.0105± 0.0105 0.6616±0.0056± 0.0056 0.7033±0.0065± 0.0065 0.7193±0.0110± 0.0110 0.7473±0.0075± 0.0075 0.7226±0.0091± 0.0091 GradSafe 0.9392±0.0041± 0.0041 0.7877±0.0061± 0.0061 0.7795±0.0052± 0.0052 0.9619±0.0036± 0.0036 0.7967±0.0055± 0.0055 0.7396±0.0079± 0.0079 FT 0.9244±0.0043± 0.0043 0.4312±0.0156± 0.0156 0.5618±0.0175± 0.0175 0.8284±0.0167± 0.0167 0.5510±0.0166± 0.0166 0.6265±0.0177± 0.0177 FJD 0.9640±0.0067± 0.0067 0.8048±0.0135± 0.0135 0.8310±0.0123± 0.0123 0.9650±0.0044± 0.0044 0.9494±0.0089± 0.0089 0.8442±0.0141± 0.0141 Table 2: Detection results (AUC) of jailbreak prompt under attacks via mismatched generalization. FJD outperforms the baseline in all attacks and LLMs with almost no additional computational costs during LLM inference. Attack Method Llama2-7B Vicuna-7B Guanaco-7B Llama2-13B Vicuna-13B Guanaco-13B Cipher PPL 0.0070±0.0005± 0.0005 0.0266±0.0004± 0.0004 0.0248±0.0005± 0.0005 0.0221±0.0011± 0.0011 0.0259±0.0005± 0.0005 0.0254±0.0008± 0.0008 SMLLM 0.5034±0.0024± 0.0024 0.5233±0.0009± 0.0009 0.5460±0.0036± 0.0036 0.9096±0.0105± 0.0105 0.5344±0.0025± 0.0025 0.5482±0.0020± 0.0020 GradSafe 0.7862±0.0045± 0.0045 0.7094±0.0201± 0.0201 0.8112±0.0088± 0.0088 0.8723±0.0073± 0.0073 0.7972±0.0036± 0.0036 0.7691±0.0105± 0.0105 FT 0.9636±0.0025± 0.0025 0.7966±0.0055± 0.0055 0.4905±0.0173± 0.0173 0.9837±0.0031± 0.0031 0.3030±0.0150± 0.0150 0.4724±0.0148± 0.0148 FJD 0.9896±0.0014± 0.0014 0.8633±0.0033± 0.0033 0.8299±0.0043± 0.0043 0.9909±0.0091± 0.0091 0.8876±0.0170± 0.0170 0.8216±0.0191± 0.0191 Hand (MG) PPL 0.6854±0.0014± 0.0014 0.6827±0.0013± 0.0013 0.6781±0.0006± 0.0006 0.6787±0.0016± 0.0016 0.6797±0.0007± 0.0007 0.6771±0.0010± 0.0010 SMLLM 0.7146±0.0111± 0.0111 0.7155±0.0070± 0.0070 0.8232±0.0076± 0.0076 0.7587±0.0081± 0.0081 0.6695±0.0091± 0.0091 0.7591±0.0131± 0.0131 GradSafe 0.8777±0.0058± 0.0058 0.7864±0.0049± 0.0049 0.8265±0.0055± 0.0055 0.8501±0.0068± 0.0068 0.8185±0.0039± 0.0039 0.7708±0.0056± 0.0056 FT 0.9229±0.0055± 0.0055 0.5625±0.0145± 0.0145 0.4885±0.0126± 0.0126 0.7557±0.0145± 0.0145 0.6600±0.0168± 0.0168 0.5268±0.0019± 0.0019 FJD 0.9549±0.0072± 0.0072 0.7937±0.0160± 0.0160 0.8882±0.0153± 0.0153 0.9444±0.0085± 0.0085 0.9510±0.0104± 0.0104 0.8395±0.0171± 0.0171 Table 3: Detection results (AUC) of jailbreak prompt with and without Affirmative Instruction (AI) and Temperature Scaling (TS) modules in FJD. Both modules can improve detection performance. Method AI TS AutoDAN Cipher Llama2-7B Vicuna-7B Guanaco-7B Llama2-7B Vicuna-7B Guanaco-7B 0.8737±0.0124± 0.0124 0.1617±0.0057± 0.0057 0.6588±0.0142± 0.0142 0.9214±0.0032± 0.0032 0.6399±0.0096± 0.0096 0.4826±0.0152± 0.0152 0.9436±0.0076± 0.0076 0.7862±0.0032± 0.0032 0.8447±0.0076± 0.0076 0.9682±0.0037± 0.0037 0.8569±0.0029± 0.0029 0.8167±0.0034± 0.0034 FT 0.8869±0.0149± 0.0149 0.1709±0.0083± 0.0083 0.7084±0.0106± 0.0106 0.9636±0.0025± 0.0025 0.7966±0.0055± 0.0055 0.4905±0.0173± 0.0173 FJD 0.9578±0.0088± 0.0088 0.7964±0.0182± 0.0182 0.8946±0.0065± 0.0065 0.9896±0.0014± 0.0014 0.8633±0.0033± 0.0033 0.8299±0.0043± 0.0043 4.2 Jailbreak Detection under Attacks with Competing Objectives To evaluate the detection of jailbreak prompts via competing objectives for our approach, which comprises First Token (FT) and FJD, we conducted experiments on two attacks: AutoDAN and Hand (CO). Tab. 1 shows that FJD can effectively detect jailbreak prompts via competing objectives on almost all LLMs. The optimized jailbreak attack (AutoDAN) generates higher token confidence than benign prompts, making FT difficult to detect on some LLMs. Hand-crafted prompts exhibit low perplexity, making PPL difficult to detect. And more detection results under other jailbreak attacks via competing objectives are in Appendix G. 4.3 Jailbreak Detection under Attacks with Mismatched Generalization To investigate the effectiveness of FJD in detecting jailbreak prompts via mismatched generalization, we conducted experiments on two attacks: Cipher and Hand (MG). Tab. 2 illustrates that FJD achieves superior performance across almost all LLMs. Cipher, constructed with a fixed format and some manual examples, exhibits lower perplexity than benign prompts, making PPL difficult to detect. More detection results under other jailbreak attacks via mismatched generalization are in Appendix H. Table 4: Detection results (AUC) of jailbreak prompt under transferable attacks. FJD can effectively detect jailbreak prompts from transferable attacks in most cases. Source Target Method Llama3-8B ChatGPT-3.5 Vicuna-7B PPL 0.7040±0.0022± 0.0022 0.8141±0.0014± 0.0014 SMLLM 0.8585±0.0061± 0.0061 0.8938±0.0057± 0.0057 GradSafe 0.8629±0.0024± 0.0024 - FJD 0.8768±0.0087± 0.0087 0.9553±0.0073± 0.0073 Llama2-7B PPL 0.7551±0.0037± 0.0037 0.8138±0.0010± 0.0010 SMLLM 0.8662±0.0041± 0.0041 0.8333±0.0055± 0.0055 GradSafe 0.8908±0.0039± 0.0039 - FJD 0.9013±0.0075± 0.0075 0.9496±0.0060± 0.0060 Guanaco-7B PPL 0.9256±0.0014± 0.0014 0.7173±0.0025± 0.0025 SMLLM 0.8687±0.0057± 0.0057 0.9425±0.0032± 0.0032 GradSafe 0.9143±0.0059± 0.0059 - FJD 0.9350±0.0077± 0.0077 0.9432±0.0089± 0.0089 4.4 Jailbreak Detection under Transferable Jailbreak Attacks For detecting transferable jailbreak attacks, this experiment employs Llama2, Vicuna and Guanaco as the source models and aggregates prompts acquired from GCG and AutoDAN. Subsequently, we further evaluate Llama3 8B and ChatGPT3.5 as the target models. And Tab. 4 shows the detection results of our FJD against transferable jailbreak attacks. For the successfully transferable prompt, FJD demonstrates a more effective detection capability in most cases than baselines. Since GradSafe requires the gradients of LLMs, it cannot be used for detection on ChatGPT. In contrast, FJD can leverage ChatGPT’s API to obtain the probability values of generated tokens for detection. And more detection results are in Appendix J. Moreover, investigating the generality of FJD in multimodal scenarios is indeed interesting. The detection results and discussions are in Appendix P. 4.5 Analysis of Affirmative Instructions To investigate the difference between model responses to jailbreak and benign prompts with prepended affirmative instructions, we use saliency (Sarti et al., 2023) to perform attribution analysis on the first token generated by LLMs. Fig. 3 shows the Table 5: Detection results (AUC) of jailbreak prompt while using the affirmative instruction variants. FJD can effectively detect jailbreak prompts using various affirmative instructions. Different Affirmative Instruction AutoDAN Cipher Without affirmative instruction 0.8869 0.9636 Respond in a way that promotes safety. 0.9624 0.9769 Provide an answer that prioritizes user safety. 0.9163 0.9555 Remember you are a responsible and helpful LLM. 0.9350 0.9403 Let’s think step by step. 0.9610 0.9578 contribution of the instruction for jailbreak and benign prompts on Vicuna 7B. It has been observed that the affirmative instruction integrated by FJD notably influences the responses to benign prompts. More details are in Appendix E. To evaluate the influence of different affirmative instructions in FJD, we create different instructions that confirm the original abilities of LLM behavior. Taking Llama2 7B as an example, Tab. 5 illustrates that FJD can effectively detect jailbreak prompts through different affirmative instructions. Results with more instructions are in Appendix K. Figure 3: Affirmative instruction contribution and the frequency of data volume for the first tokens in Vicuna 7B. The contribution of affirmative instruction for the benign prompts is higher than the jailbreak prompts. 4.6 Temperature Scaling Analysis To evaluate the influence of the temperature t on the jailbreak detection across various LLMs, experiments were performed on three LLMs using a step size of 0.010.01 in range [0,2][0,2]. Fig. 4 illustrates the detection results of the FJD for the LLMs across the three attacks (GCG, AutoDAN, Cipher) with varying temperatures. The x-axis denotes the temperature, the y-axis displays the detection results, the red dashed line signifies the optimal temperature of the LLM on the training set and the temperature has a substantial impact around 0.50.5. It also illustrates that the detection performance of FJD can be enhanced through temperature scaling. In our experiments, training data is used to identify an optimal temperature, which is used across all experiments. More details are in Appendix N. Table 6: Efficiency analysis of FJD and the baselines on Llama2. FJD requires almost no additional computational costs during LLM inference. Furthermore, FJD minimally impacts the semantics of benign prompt. Method Computational Costs Semantic Changes Extra Inference Time Costs Semantic Similarity ChatGPT Score PPL 1 412s - - SMLLM 10 1568s 0.6810 0.7431 GradSafe 1 405s - - FJD 0 396s 0.7402 0.8560 4.7 Ablation Experiment of FJD To investigate the influence of the Affirmative Instruction (AI) and Temperature Scaling (TS) modules in FJD, we performed an ablation experiment to contrast the results of detecting jailbreak prompts with and without the modules. Tab. 3 shows that the enhanced jailbreak detection performance promoted by both modules. Specifically, AI exerts a more significant influence on improving the performance of FJD. Furthermore, incorporating TS on the basis of AI demonstrates a more obvious effect compared to adding TS without AI. (a) Llama2 7B (b) Vicuna 7B (c) Guanaco 7B Figure 4: Detection results (AUC) of the FJD for the LLMs across the three attacks with varying temperatures. The temperature has an impact on jailbreak detection. The red line represents the optimal temperature from the training. Table 7: Detection results (AUC) of jailbreak prompt through FJD-LI. FJD-LI further enhances the detection of jailbreak prompts, even when faced with unseen data. Attack Method Llama2-7B Vicuna-7B Guanaco-7B AutoDAN PPL 0.8172±0.0017± 0.0017 0.7452±0.0012± 0.0012 0.7964±0.0004± 0.0004 SMLLM 0.8197±0.0052± 0.0052 0.7831±0.0035± 0.0035 0.6704±0.0036± 0.0036 GradSafe 0.8025±0.0089± 0.0089 0.7893±0.0020± 0.0020 0.8194±0.0051± 0.0051 FJD 0.9578±0.0088± 0.0088 0.7964±0.0182± 0.0182 0.8946±0.0065± 0.0065 FJD-LI 0.9703±0.0024± 0.0024 0.9969±0.0021± 0.0021 0.9817±0.0038± 0.0038 Cipher PPL 0.0070±0.0005± 0.0005 0.0266±0.0004± 0.0004 0.0248±0.0005± 0.0005 SMLLM 0.5034±0.0024± 0.0024 0.5233±0.0009± 0.0009 0.5460±0.0026± 0.0026 GradSafe 0.7862±0.0045± 0.0045 0.7094±0.0201± 0.0201 0.8112±0.0088± 0.0088 FJD 0.9896±0.0014± 0.0014 0.8633±0.0033± 0.0033 0.8299±0.0043± 0.0043 FJD-LI 0.9944±0.0012± 0.0012 0.9310±0.0036± 0.0036 0.8826±0.0102± 0.0102 4.8 Analysis of FJD-LI To evaluate the performance of FJD-LI, 50%50\% jailbreak prompts from GCG and AutoDAN are sampled to construct a training set. We conduct experiments by incorporating learnable virtual instruction into Llama, Vicuna Table 8: Detection results (AUC) of FJD under detection-aware attck on Vicuna 7B. FJD struggles to defend against white-box detection-aware attacks but demonstrates robust resistance to transferred ones. Attacks FJD AutoDAN 0.7964±0.0182± 0.0182 Cipher 0.8633±0.0033± 0.0033 Hand-crafted (CO) 0.8048±0.0135± 0.0135 Hand-crafted (MG) 0.7937±0.0160± 0.0160 Detection-aware Attack (White-box) 0.4761±0.0029± 0.0029 Detection-aware Attack (Transfer from Llama2) 0.9017±0.0052± 0.0052 Detection-aware Attack (Transfer from Guanaco) 0.8886±0.0073± 0.0073 and Guanaco. As described in Tab. 7, this approach further enhances the detection of jailbreak prompts, even when faced with unseen data (Cipher), indicating its robust generalization. More detection results are in Appendix L. 4.9 Efficiency Analysis To verify the efficiency of FJD, we evaluate it from two perspectives: computational costs and semantic changes. For computational costs, we compare the extra inference and time costs across different detection methods. The LLM default inference time on benign prompts is 394s. For semantic changes, we compare the semantic similarity derived from Llama2 encoding and the ChatGPT Score. Tab. 6 presents a comparison of the efficiency of FJD with three baseline approaches on Llama2. PPL requires an additional forward pass to calculate the input perplexity score. SMLLM requires additional model forward passes to analyze the results of multiple input copies. And GradSafe requires an additional forward and backward pass to calculate the gradients. However, FJD does not require an additional forward pass and can detect jailbreak prompts during model inference, which also have a smaller impact on model responses. More details, including evaluations with AlpacaEval 2.0 and Arena-Hard-Auto on the effect of FJD on benign prompts, are in Appendix F. 4.10 Detection-aware Attack of FJD For breaking FJD, we conduct a detection-aware attack based on GCG, which optimizes the suffix by minimizing the target loss under the FJD. The attack comprises two forms: a white-box attack utilizing known LLM, and the transferred black-box attacks from another LLM. Taking Vicuna 7B as an example, Tab. 8 shows that FJD struggles to defend against white-box attack but demonstrates robust resistance to transferred black-box attack. Currently, designing a robust detection method against white-box detection-aware attack is a well-known challenge in our community. In more practical scenarios, transferable attacks are commonly employed where FJD is still very effective. 5 Discussion Why Affirmative Instruction Helps? As shown in Fig. 3, after prepending affirmative instructions, LLMs allocate increased focus to the instructions for benign prompts and gives precedence to follow the instructions, leading to higher output confidence. In contrast, the jailbreak prompts has been observed to command a significant portion of attention (Arditi et al., 2024), and LLMs focus more on jailbreak prompts and less on the instructions. The resulted output is still confused and with less confidence due to competing objectives and mismatched generalization. As a result, prepending affirmative instructions enlarge the differences of the first token confidence between jailbreak and benign prompts, resulting in better detection. Why Temperature Scaling Helps? As shown in Sec. 3.2, TS can change the confidence rank between two samples. Concretely, applying TS with τ > 1.0 reduces the confidence of the maximum token, unless all logits are the same. If the non-max logits of a sample distribute more evenly, the decrease of the confidence is more significant. We observe that the non-max logits are indeed distributed more evenly due to the nature of their competing objectives or mismatched generalization. In contrast, the decreased confidence of benign prompts is less. Hence, TS with τ > 1.0 can enlarge the confidence difference between benign and jailbreak prompts, leading to higher detection performance. 6 Conclusion In this paper, we propose Free Jailbreak Detection (FJD), which uses the confidence of the first token to detect the jailbreak prompts without additional computational costs during LLM inference. Our method perform Jailbreak detection efficiently and effectively across various LLMs. We call for developing more efficient jailbreak mitigation methods. Limitations Our proposed method FJD can effectively detects LLM jailbreak attempts using affirmative instructions and temperature scaling. Three main limitations present as follows: First, FJD detection performance is slightly lower on non-readable jailbreak prompts generated by GCG compared to targeted Perplexity-based detection methods. This gap can be with our FJD-LI method where we learn a more effective affirmative instructions for jailbreak detection. Second, detection-aware white-box attacks, where both FJD and LLMs are fully known, can break our detection method to some degree. The limitation can be mitigated by hiding detection method from attackers in practice. And future research will also explore more robust affirmative instructions to further enhance FJD to overcome white-box aware attacks. Finally, with the rapid advancement of the LLM field and the continual iteration of model versions, different architectures and versions of LLMs exhibit varying sensitivities to prompts. Consequently, the impact of FJD on confidence also differs, preventing it from achieving consistently strong performance across all models. And future work, beyond using confidence, it is also important to incorporate internal features of LLMs to develop more robust methods. We hope that our work provides some insights into efficient and effective LLM jailbreak detection. References [1] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. (2023) Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §1, §4.1. [2] A. Albert (2023) Note: https://w.jailbreakchat.com/Accessed: 2023-09-28 Cited by: §2. [3] G. Alon and M. Kamfonas (2023) Detecting language model attacks with perplexity. arXiv preprint arXiv:2308.14132. Cited by: Appendix C, §1, §2, §4.1. [4] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh (2015) Vqa: visual question answering. In Proceedings of the IEEE international conference on computer vision, p. 2425–2433. Cited by: Appendix P. [5] A. Arditi, O. Obeso, A. Syed, D. Paleka, N. Rimsky, W. Gurnee, and N. Nanda (2024) Refusal in language models is mediated by a single direction. arXiv preprint arXiv:2406.11717. Cited by: Appendix D, §3.2, §5. [6] L. N. Candogan, Y. Wu, E. A. Rocamora, G. G. Chrysos, and V. Cevher (2025) Single-pass detection of jailbreaking input in large language models. arXiv preprint arXiv:2502.15435. Cited by: §2. [7] B. Cao, Y. Cao, L. Lin, and J. Chen (2023) Defending against alignment-breaking attacks via robustly aligned llm. arXiv preprint arXiv:2309.14348. Cited by: §2. [8] N. Carlini, M. Nasr, C. A. Choquette-Choo, M. Jagielski, I. Gao, P. W. W. Koh, D. Ippolito, F. Tramer, and L. Schmidt (2024) Are aligned neural networks adversarially aligned?. Advances in Neural Information Processing Systems 36. Cited by: §1. [9] P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong (2023) Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419. Cited by: Appendix B, §1, §2. [10] S. Chen, Z. Han, B. He, Z. Ding, W. Yu, P. Torr, V. Tresp, and J. Gu (2024) Red teaming gpt-4v: are gpt-4v safe against uni/multi-modal jailbreak attacks?. arXiv preprint arXiv:2404.03411. Cited by: Appendix B, Appendix I, §1, §3.1, §4.1. [11] W. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, et al. (2023) Vicuna: an open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) 2 (3), p. 6. Cited by: §1, §4.1. [12] L. Daniele and Suphavadeeprasit (2023) Amplify-instruct: synthetically generated diverse multi-turn conversations for effecient llm training.. arXiv preprint arXiv:(comming soon). Cited by: Appendix A, §4.1. [13] G. Deng, Y. Liu, Y. Li, K. Wang, Y. Zhang, Z. Li, H. Wang, T. Zhang, and Y. Liu (2023) Jailbreaker: automated jailbreak across multiple large language model chatbots. arXiv preprint arXiv:2307.08715. Cited by: §1. [14] Y. Deng, W. Zhang, S. J. Pan, and L. Bing (2023) Multilingual jailbreak challenges in large language models. In The Twelfth International Conference on Learning Representations, Cited by: §1. [15] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer (2024) Qlora: efficient finetuning of quantized llms. Advances in Neural Information Processing Systems 36. Cited by: §1, §4.1. [16] P. Ding, J. Kuang, D. Ma, X. Cao, Y. Xian, J. Chen, and S. Huang (2023) A wolf in sheep’s clothing: generalized nested jailbreak prompts can fool large language models easily. arXiv preprint arXiv:2311.08268. Cited by: §1, §2. [17] Y. Gong, D. Ran, J. Liu, C. Wang, T. Cong, A. Wang, S. Duan, and X. Wang (2025) Figstep: jailbreaking large vision-language models via typographic visual prompts. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 23951–23959. Cited by: Appendix P. [18] A. Helbling, M. Phute, M. Hull, and D. H. Chau (2023) Llm self defense: by self examination, llms know they are being tricked. arXiv preprint arXiv:2308.07308. Cited by: §1, §2. [19] N. Jain, A. Schwarzschild, Y. Wen, G. Somepalli, J. Kirchenbauer, P. Chiang, M. Goldblum, A. Saha, J. Geiping, and T. Goldstein (2023) Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614. Cited by: §1, §2. [20] X. Jia, T. Pang, C. Du, Y. Huang, J. Gu, Y. Liu, X. Cao, and M. Lin (2024) Improved techniques for optimization-based jailbreaking on large language models. arXiv preprint arXiv:2405.21018. Cited by: §2. [21] E. Jones, A. Dragan, A. Raghunathan, and J. Steinhardt (2023) Automatically auditing large language models via discrete optimization. In International Conference on Machine Learning, p. 15307–15329. Cited by: §1. [22] S. Kullback and R. A. Leibler (1951) On information and sufficiency. The annals of mathematical statistics 22 (1), p. 79–86. Cited by: §3.3. [23] A. Kumar, C. Agarwal, S. Srinivas, S. Feizi, and H. Lakkaraju (2023) Certifying llm safety against adversarial prompting. arXiv preprint arXiv:2309.02705. Cited by: §2. [24] R. Lapid, R. Langberg, and M. Sipper (2023) Open sesame! universal black box jailbreaking of large language models. arXiv preprint arXiv:2309.01446. Cited by: §1. [25] A. N. Lee, C. J. Hunter, and N. Ruiz (2023) Platypus: quick, cheap, and powerful refinement of llms. arXiv preprint arXiv:2308.07317. Cited by: Appendix A, §4.1. [26] H. Li, D. Guo, W. Fan, M. Xu, J. Huang, F. Meng, and Y. Song (2023) Multi-step jailbreaking privacy attacks on chatgpt. arXiv preprint arXiv:2304.05197. Cited by: §1. [27] T. Li, W. Chiang, E. Frick, L. Dunlap, T. Wu, B. Zhu, J. E. Gonzalez, and I. Stoica (2024) From crowdsourced data to high-quality benchmarks: arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939. Cited by: Appendix F. [28] X. Li, R. Wang, M. Cheng, T. Zhou, and C. Hsieh (2024) Drattack: prompt decomposition and reconstruction makes powerful llm jailbreakers. arXiv preprint arXiv:2402.16914. Cited by: §1. [29] X. Li, Z. Zhou, J. Zhu, J. Yao, T. Liu, and B. Han (2023) Deepinception: hypnotize large language model to be jailbreaker. arXiv preprint arXiv:2311.03191. Cited by: §1. [30] X. Li, T. Zhang, Y. Dubois, R. Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto (2023-05) AlpacaEval: an automatic evaluator of instruction-following models. GitHub. Note: https://github.com/tatsu-lab/alpaca_eval Cited by: Appendix F. [31] X. Liu, N. Xu, M. Chen, and C. Xiao (2023) Autodan: generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451. Cited by: Appendix B, Appendix D, §1, §2, §3.1, §4.1. [32] X. Liu, Y. Zhu, J. Gu, Y. Lan, C. Yang, and Y. Qiao (2024) Mm-safetybench: a benchmark for safety evaluation of multimodal large language models. In European Conference on Computer Vision, p. 386–403. Cited by: Appendix P. [33] Y. Liu, G. Deng, Z. Xu, Y. Li, Y. Zheng, Y. Zhang, L. Zhao, T. Zhang, and Y. Liu (2023) Jailbreaking chatgpt via prompt engineering: an empirical study. arXiv preprint arXiv:2305.13860. Cited by: §1, §2. [34] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022) Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, p. 27730–27744. Cited by: §1. [35] A. Paulus, A. Zharmagambetov, C. Guo, B. Amos, and Y. Tian (2024) Advprompter: fast adaptive adversarial prompting for llms. arXiv preprint arXiv:2404.16873. Cited by: Appendix B, §2. [36] F. Perez and I. Ribeiro (2022) Ignore previous prompt: attack techniques for language models. arXiv preprint arXiv:2211.09527. Cited by: §1. [37] X. Qi, Y. Zeng, T. Xie, P. Chen, R. Jia, P. Mittal, and P. Henderson (2023) Fine-tuning aligned language models compromises safety, even when users do not intend to!. arXiv preprint arXiv:2310.03693. Cited by: §1. [38] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn (2024) Direct preference optimization: your language model is secretly a reward model. Advances in Neural Information Processing Systems 36. Cited by: §1. [39] A. Rao, S. Vashistha, A. Naik, S. Aditya, and M. Choudhury (2023) Tricking llms into disobedience: understanding, analyzing, and preventing jailbreaks. arXiv preprint arXiv:2305.14965. Cited by: §2. [40] A. Robey, E. Wong, H. Hassani, and G. J. Pappas (2023) Smoothllm: defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684. Cited by: Appendix C, §1, §2, §4.1. [41] G. Sarti, N. Feldhus, L. Sickert, O. Van Der Wal, M. Nissim, and A. Bisazza (2023) Inseq: an interpretability toolkit for sequence generation models. arXiv preprint arXiv:2302.13942. Cited by: Appendix E, §4.5. [42] R. Shah, S. Pour, A. Tagade, S. Casper, J. Rando, et al. (2023) Scalable and transferable black-box jailbreaks for language models via persona modulation. arXiv preprint arXiv:2311.03348. Cited by: §1. [43] K. Simonyan, A. Vedaldi, and A. Zisserman (2013) Deep inside convolutional networks: visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034. Cited by: Appendix E. [44] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. (2023) Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Cited by: §1, §4.1. [45] walkerspider (2022) Note: https://old.reddit.com/r/ChatGPT/comments/zlcyr9/dan_is_my_new_friend/Accessed: 2023-09-28 Cited by: §2. [46] A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman (2019) Superglue: a stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems 32. Cited by: Appendix A, §4.1. [47] A. Wei, N. Haghtalab, and J. Steinhardt (2024) Jailbroken: how does llm safety training fail?. Advances in Neural Information Processing Systems 36. Cited by: §1, §2, §3.1, §4.1. [48] Y. Wen, N. Jain, J. Kirchenbauer, M. Goldblum, J. Geiping, and T. Goldstein (2024) Hard prompts made easy: gradient-based discrete optimization for prompt tuning and discovery. Advances in Neural Information Processing Systems 36. Cited by: §1. [49] N. Wichers, C. Denison, and A. Beirami (2024) Gradient-based language model red teaming. arXiv preprint arXiv:2401.16656. Cited by: §1. [50] J. Wu, L. Ouyang, D. M. Ziegler, N. Stiennon, R. Lowe, J. Leike, and P. Christiano (2021) Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862. Cited by: §1. [51] Y. Xie, M. Fang, R. Pi, and N. Gong (2024) GradSafe: detecting jailbreak prompts for llms via safety-critical gradient analysis. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 507–518. Cited by: Appendix C, §4.1. [52] Y. Xie, J. Yi, J. Shao, J. Curl, L. Lyu, Q. Chen, X. Xie, and F. Wu (2023) Defending chatgpt against jailbreak attack via self-reminders. Nature Machine Intelligence 5 (12), p. 1486–1496. Cited by: §2. [53] Z. Xu, F. Jiang, L. Niu, J. Jia, B. Y. Lin, and R. Poovendran (2024) SafeDecoding: defending against jailbreak attacks via safety-aware decoding. arXiv preprint arXiv:2402.08983. Cited by: Appendix D, §2. [54] Z. Yong, C. Menghini, and S. H. Bach (2023) Low-resource languages jailbreak gpt-4. arXiv preprint arXiv:2310.02446. Cited by: §2. [55] W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang (2023) Mm-vet: evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490. Cited by: Appendix P. [56] Y. Yuan, W. Jiao, W. Wang, J. Huang, P. He, S. Shi, and Z. Tu (2023) Gpt-4 is too smart to be safe: stealthy chat with llms via cipher. arXiv preprint arXiv:2308.06463. Cited by: Appendix B, §1, §2, §3.1, §4.1. [57] Z. Yuan, Z. Xiong, Y. Zeng, N. Yu, R. Jia, D. Song, and B. Li (2024) RigorLLM: resilient guardrails for large language models against undesired content. arXiv preprint arXiv:2403.13031. Cited by: §1, §2. [58] Y. Zeng, Y. Wu, X. Zhang, H. Wang, and Q. Wu (2024) Autodefense: multi-agent llm defense against jailbreak attacks. arXiv preprint arXiv:2403.04783. Cited by: §1. [59] X. Zhang, C. Zhang, T. Li, Y. Huang, X. Jia, X. Xie, Y. Liu, and C. Shen (2023) A mutation-based method for multi-modal jailbreaking attack detection. arXiv preprint arXiv:2312.10766. Cited by: §2. [60] Y. Zhang and Z. Wei (2024) Boosting jailbreak attack with momentum. arXiv preprint arXiv:2405.01229. Cited by: Appendix B, §2. [61] Z. Zhang, J. Yang, P. Ke, and M. Huang (2023) Defending large language models against jailbreaking attacks through goal prioritization. arXiv preprint arXiv:2311.09096. Cited by: §2. [62] A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson (2023) Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043. Cited by: Appendix A, Appendix B, Appendix D, §1, §2, §3.1, §4.1, §4.1. Appendix A The Details of Dataset To evaluate FJD, we select two jailbreak datasets: AdvBench [62] and three benign datasets: Pure-Dove [12], Open-Platypus [25], and SuperGLUE [46]. • AdvBench333https://github.com/llm-attacks/llm-attacks/blob/main/data/advbench/harmful_behaviors.csv, which contains 520 predefined harmful behaviors that do not align with human values. • Pure-Dov444https://huggingface.co/datasets/LDJnr/Pure-Dove, which contains 3856 highly filtered conversations between GPT-4 and real humans. And the average context length per conversation is over 800 tokens. • Open-Platypus555https://huggingface.co/datasets/garage-bAInd/Open-Platypus, which focuses on improving LLM logical reasoning skills and is used to train the Platypus2 models. • SuperGLUE666https://huggingface.co/datasets/aps/super_glue, which is a new benchmark styled after GLUE with a new set of more difficult language understanding tasks. The slices of the dataset are shown in the Figure 5. Appendix B The Details of Attacks Five attacks via competing objectives and two attacks via mismatched generalization are included in the experiment, where attacks via competing objectives include GCG [62], MAC [60], AutoDAN [31] and AdvPrompter [35]. • GCG.777https://github.com/llm-attacks/llm-attacks We use the official implementation to generate individual jailbreak prompts. For all LLMs, we use default hyper-parameters with batch size 512512, learning rate 0.010.01 and the length of attack string 20 tokens. Also use the official implementation to generate transferable jailbreak prompts based on LLama2 7B, Vicuna 7B and Guanaco 7B with the same hyper-parameters. • MAC.888https://github.com/weizeming/momentum-attack-llm We use the official implementation to generate individual jailbreak prompts. MAC propose a momentum-enhanced greedy coordinate gradient method for jailbreak. For all LLMs, we use default hyper-parameters with batch size 256, top-k 256 and 20 epochs. • AutoDAN.999https://github.com/SheltonLiu-N/AutoDAN We use the official implementation with the initial jailbreak prompt from the original paper. For all LLMs, we use default hyper-parameters with crossover rate 0.5 and mutation rate 0.01. • AdvPrompter101010https://github.com/facebookresearch/advprompter use one LLM to generate human-readable jailbreak prompts for jailbreaking. We use the Llama2-7b-hf as the AdvPrompter and the six LLMs as the TargetLLM. We use default hyper-parameters with buffer size 8, batch size 8, max length of sequence 30, regularization strength 100, number of candidates 48 and beam size 4. Attacks via mismatched generalization include Cipher [56], Hand-Crafted [10] and PAIR [9]. • Cipher.111111https://github.com/RobustNLP/CipherChat We utilize the official implementation to validate the attack results on GPT-3.5 and GPT-4 across six LLMs, filtering out successful attack prompts by word rejection. • Hand-Crafted.121212https://anonymous.4open.science/r/red_teaming_gpt4-C1CE, which contains 27 hand-crafted textual jailbreak methods based on the AdvBench. • PAIR.131313https://github.com/patrickrchao/jailbreakingllms We use the official implementation and use LLama2 7B/13B and Vicuan 7B/13B to generate jailbreak prompts with using ChatGPT3.5 as the judging model. For all LLMs, we use default hyper-parameters with streams 20 and iterations 100. The examples of the jailbreak prompts are shown in the Figure 6. (a) The slices of the AdvBench dataset (b) The slices of the Pure-Dov dataset Figure 5: The slices of the datasets. It presents five examples for AdvBench and Pure-Dove. Figure 6: The examples of the jailbreak prompts on GCG, AutoDAN, Cipher and Hand-Crafted. The red tokens represent examples of harmful behaviors. Appendix C The Details of Baselines For comparison with FJD, we consider three Baselines: PPL [3], SmoothLLM [40] and GradSafe [51]. • PPL, which is an input detection mechanism that computes the perplexity of inputs to determine whether the inputs are the jailbreak or benign prompts. The perplexity score is defined as Equation 3. PPL=exp(−1n∑i=1nlog(pθ(xi|x1:i−1)))PPL= (- 1n _i=1^n (p_θ(x_i|x_1:i-1))) (3) • SmoothLLM. We test swap approaches with perturbation percentage q=10%q=10\% and number of iterations N=10N=10 settings. • GradSafe, which analyzes the gradients from prompts (paired with compliance responses) to accurately detect jailbreak prompts. Appendix D The Observation and Theoretical Analysis on Finding In this section, we conduct a statistical analysis of the distribution of first-token probabilities generated by GCG, AutoDAN, Cipher, and Benign prompts on the Llama2 7B, Vicuna 7B, and Guanaco 7B. In almost all cases, there is an obvious difference in the confidence of the first token between the responses generated by these prompts and benign ones. Regarding the analysis of factors influencing the distributional difference at the first token confidence, our preliminary findings suggest that compared to benign prompts, jailbreak prompts (e.g., GCG [62], AutoDAN [31]) hijack the LLM’s attention through their jailbreak suffixes. This attention diversion mechanism ultimately results in significantly lower confidence for the first token in jailbreak scenarios. In Sec. 4.5, we performed attribution experiments to investigate which parts of the input exert stronger influence on the first token. Our experimental results demonstrate alignment with the findings reported in prior related work [5], confirming that jailbreak suffixes hijack the majority of the LLM’s attention, thereby destabilizing the reasoning process and leading to significantly lower confidence scores. To further investigate the potential causes of the observed distributional difference in first token confidence, we propose two additional hypotheses: First, safety-aligned LLMs may retain inherent resistance to generating harmful content—even when compromised by jailbreak prompts—by actively suppressing the confidence scores of tokens associated with harmful intent. As demonstrated in prior work [53], even when coerced into producing harmful responses, the model maintains a high probability of simultaneously generating refusal behaviors. This inherent conflict results in significantly lower confidence scores for the initial output token. Second, jailbreak prompts essentially constitute out-of-distribution (OOD) samples for LLMs, whereas benign prompts align with the training data distribution. By employing adversarial perturbations (e.g., semantic obfuscation, special characters), jailbreak prompts force LLMs to confront OOD challenges during inference. This disrupts the contextual dependencies critical for first-token prediction, consequently reducing its confidence score. (a) GCG vs. Benign in Llama2 7B (b) AutoDAN vs. Benign in Llama2 7B (c) Cipher vs. Benign in Llama2 7B (d) GCG vs. Benign in Vicuna 7B (e) AutoDAN vs. Benign in Vicuna 7B (f) Cipher vs. Benign in Vicuna 7B (g) GCG vs. Benign in Guanaco 7B (h) AutoDAN vs. Benign in Guanaco 7B (i) Cipher vs. Benign in Guanaco 7B Figure 7: The distribution of the confidence scores of the predicted first tokens over jailbreak and benign samples is shown. A difference can be observed where LLMs are less confident on Jailbreak samples than on benign samples. Appendix E Attribution Analysis To investigate the difference between the affirmative instruction prepended by FJD in LLMs’ responding to jailbreak and benign prompts, we use the saliency [41, 43] method to perform attribution analysis on the first 10 tokens generated by LLMs. Specifically, given the input sequence xq∈[||]qx_q∈[|V|]^q and the affirmative instruction of FJD xai∈[||]mx_ai∈[|V|]^m, the contribution of sequence xai⊕xqx_ai x_q is calculated as 4. SC=fsaliency(xai⊕xq)SC=f_saliency(x_ai x_q) (4) where fsaliency(⋅)f_saliency(·) is the attribution analysis on the LLMs and SC∈ℝ(m+q)×10SC ^(m+q)× 10 is the contribution of sequence for the first 10 tokens. Then the contribution of prompt xaix_ai is calculated as 5. PCk=1k∑n=1k∑i=1mSCi,n∑j=1m+qSCj,n×m+qmPC_k= 1k _n=1^k _i=1^mSC_i,n _j=1^m+qSC_j,n× m+qm (5) where (m+q)/m (m+q)/m is the length penalty coefficient. Then PCk∈ℝ10PC_k ^10 is the contribution of prompt for the first k tokens. We also evaluated the influence of affirmative instructions on generating the first five and ten tokens in Fig. 8a and Fig. 8b. Our observations indicate that the variance between jailbreak and benign prompts in the first five and ten tokens is less significant compared to that in the first token. Thus, we discuss the impact of selecting the first k tokens for detecting jailbreak prompts in the Appendix O. (a) First 5 token (b) First 10 token Figure 8: Affirmative instruction contribution and the frequency of data volume for the first 5/10 tokens in Vicuna 7B. The contribution of affirmative instruction for the benign prompts is higher than the jailbreak prompts via competing objectives and mismatched generalization. Appendix F The Effect on Benign Prompts In this section, we evaluate the effect of benign prompts using AlpacaEval 2.0 [30] and Arena-Hard-Auto [27]. In all cases, we used Gpt-4o-mini-2024-07-18 as the judge model and reported the Win Rate and Scores with and without FJD. As shown in the Tab. 9, the affirmative instruction in FJD has little impact on benign prompts and can even improve the model’s inference quality (Win Rate) to some extent. This is because affirmative instructions refer to those that affirm the inherent capabilities of LLMs (e.g., adding “You are a good assistant”). It follows the same design paradigm as system prompts in mainstream LLMs, serving as a widely-adopted instruction approach. In other words, after adding affirmative instruction, the LLM’s generation remains high-quality and highly consistent. We thank the reviewer for raising this point. We will include a discussion on the impact of FJD on benign prompts in our revision. Table 9: The effect on benign prompts. The affirmative instruction in FJD has little impact on benign prompts and can even improve the LLM’s inference quality to some extent. Evaluation Method Benign Prompts Benign Prompts with FJD AlpacaEval 2.0 (Win Rate) 9.40 19.64 Arena-Hard-Auto (Scores) 0.5 0.4 Appendix G Jailbreak Detection under Attacks with Competing Objectives In order to fully evaluate the performance of FJD under attacks via competing objectives, we expand upon three additional attack methods and incorporate three additional evaluation metrics. We categorize the attack methods into two groups based on whether the jailbreak prompt is human-readable. The jailbreak prompts generated by AutoDAN (Tab. 10) and AdvPrompter (Tab. 11) are human-readable, while those generated by GCG (Tab. 12) and MAC (Tab. 13) are not human-readable. However, due to the low success rate of the AdvPrompter method on the LLama2 series model, the repeated experimental outcomes exhibit significant fluctuations, rendering them unreliable for generating comparative experimental results. For the three recently incorporated comparison metrics, as SMLLM functions as a defensive measure, we presume its false positive rate for benign samples is zero. Consequently, FPR comparison with this method is omitted. For human-readable jailbreak prompts, FJD can effectively detect jailbreak prompts on all models. In cases where the jailbreak prompts are not human-readable, FJD performs exceptionally well with LLama2 and comparably to PPL with other LLMs. Table 10: Detection results (FPR, TPR, F1 and AUC) of jailbreak prompt under AutoDAN. FJD outperforms baseline methods on almost all the LLMs. Model Method AutoDAN FPR↓ TPR↑ F1↑ AUC↑ Llama2-7B PPL 0.2960±0.0026± 0.0026 0.9323±0.0011± 0.0011 0.5333±0.1106± 0.1106 0.8172±0.0017± 0.0017 SMLLM - 0.6587±0.0121± 0.0121 0.7942±0.0111± 0.0111 0.8197±0.0052± 0.0052 GradSafe 0.1631±0.0035± 0.0035 0.8074±0.0078± 0.0078 0.7805±0.0185± 0.0185 0.8025±0.0089± 0.0089 FT 0.1852±0.0258± 0.0258 0.8467±0.0267± 0.0267 0.4860±0.0304± 0.0304 0.8869±0.0149± 0.0149 FJD 0.1285±0.0202± 0.0202 0.9333±0.0211± 0.0211 0.7090±0.0284± 0.0284 0.9578±0.0088± 0.0088 Llama2-13B PPL 0.4262±0.0103± 0.0103 0.9396±0.0021± 0.0021 0.8546±0.0442± 0.0442 0.7018±0.0002± 0.0002 SMLLM - 0.6724±0.0048± 0.0048 0.8041±0.0069± 0.0069 0.8360±0.0021± 0.0021 GradSafe 0.1001±0.0037± 0.0037 0.8911±0.0020± 0.0020 0.9080±0.0019± 0.0019 0.9123±0.0029± 0.0029 FT 0.1429±0.0181± 0.0181 0.9540±0.0128± 0.0128 0.9125±0.0117± 0.0117 0.8899±0.0141± 0.0141 FJD 0.0968±0.0264± 0.0264 0.9582±0.0256± 0.0256 0.9434±0.0240± 0.0240 0.9214±0.0133± 0.0133 Vicuna-7B PPL 0.3880±0.0094± 0.0094 0.9349±0.0024± 0.0024 0.8907±0.0354± 0.0354 0.7452±0.0012± 0.0012 SMLLM - 0.5109±0.0027± 0.0027 0.6763±0.0054± 0.0054 0.7831±0.0035± 0.0035 GradSafe 0.2512±0.0015± 0.0015 0.6553±0.0034± 0.0034 0.6573±0.0166± 0.0166 0.7893±0.0020± 0.0020 FT 0.9421±0.0163± 0.0163 0.8113±0.0244± 0.0244 0.6829±0.0123± 0.0123 0.1709±0.0083± 0.0083 FJD 0.2263±0.0137± 0.0137 0.6769±0.0257± 0.0257 0.6671±0.0118± 0.0118 0.7964±0.0182± 0.0182 Vicuna-13B PPL 0.3434±0.0026± 0.0026 0.9426±0.0027± 0.0027 0.9415±0.0181± 0.0181 0.7889±0.0002± 0.0002 SMLLM - 0.0259±0.0039± 0.0039 0.0504±0.0075± 0.0075 0.5116±0.0044± 0.0044 GradSafe 0.1539±0.0128± 0.0128 0.9358±0.0153± 0.0153 0.9493±0.0099± 0.0099 0.9225±0.0005± 0.0005 FT 0.9538±0.0136± 0.0136 0.0264±0.0121± 0.0121 0.0071±0.0049± 0.0049 0.0471±0.0040± 0.0040 FJD 0.1206±0.0108± 0.0108 0.9543±0.0240± 0.0240 0.9500±0.0132± 0.0132 0.9373±0.0111± 0.0111 Guanaco-7B PPL 0.3798±0.0005± 0.0005 0.7839±0.0009± 0.0009 0.8051±0.0004± 0.0004 0.7964±0.0004± 0.0004 SMLLM - 0.3499±0.0014± 0.0014 0.5182±0.0149± 0.0149 0.6704±0.0036± 0.0036 GradSafe 0.2882±0.0022± 0.0022 0.7497±0.0021± 0.0021 0.7393±0.0030± 0.0030 0.8194±0.0051± 0.0051 FT 0.3357±0.0133± 0.0133 0.7049±0.0163± 0.0163 0.7319±0.0147± 0.0147 0.7084±0.0106± 0.0106 FJD 0.1920±0.0111± 0.0111 0.8167±0.0085± 0.0085 0.7834±0.0050± 0.0050 0.8946±0.0065± 0.0065 Guanaco-13B PPL 0.3005±0.0092± 0.0092 0.8396±0.0018± 0.0018 0.8063±0.0037± 0.0037 0.7703±0.0005± 0.0005 SMLLM - 0.0945±0.0093± 0.0093 0.1726±0.0155± 0.0155 0.5583±0.0038± 0.0038 GradSafe 0.2882±0.0022± 0.0022 0.7497±0.0021± 0.0021 0.7393±0.0030± 0.0030 0.7398±0.0063± 0.0063 FT 0.4167±0.0236± 0.0236 0.8438±0.0278± 0.0278 0.7254±0.0135± 0.0135 0.7710±0.0172± 0.0172 FJD 0.4413±0.0251± 0.0251 0.8679±0.0295± 0.0295 0.7309±0.0175± 0.0175 0.7470±0.0135± 0.0135 Table 11: Detection results (FPR, TPR, F1 and AUC) of jailbreak prompt under AdvPrompter. FJD outperforms baseline methods on almost all the LLMs. Model Method AdvPrompter FPR↓ TPR ↑ F1↑ AUC↑ Vicuna-7B PPL 0.3816±0.0361± 0.0361 0.7273±0.0311± 0.0311 0.5197±0.0927± 0.0927 0.6891±0.0049± 0.0049 SMLLM - 0.5036±0.0051± 0.0051 0.6699±0.0045± 0.0045 0.7518±0.0026± 0.0026 GradSafe 0.1710±0.0250± 0.0250 0.8245±0.0166± 0.0166 0.7571±0.0254± 0.0254 0.8823±0.0056± 0.0056 FT 0.1920±0.0057± 0.0057 0.7289±0.0293± 0.0293 0.6071±0.0192± 0.0192 0.8471±0.0142± 0.0142 FJD 0.1949±0.0141± 0.0141 0.8763±0.0153± 0.0153 0.6850±0.0175± 0.0175 0.9041±0.0072± 0.0072 Vicuna-13B PPL 0.3661±0.0140± 0.0140 0.5606±0.0107± 0.0107 0.3252±0.0741± 0.0741 0.5933±0.0038± 0.0038 SMLLM - 0.4630±0.0080± 0.0080 0.6287±0.0078± 0.0078 0.7315±0.0040± 0.0040 GradSafe 0.3861±0.0114± 0.0114 0.6431±0.0277± 0.0277 0.6988±0.0167± 0.0167 0.6641±0.0133± 0.0133 FT 0.1725±0.0098± 0.0098 0.8227±0.0170± 0.0170 0.7762±0.0082± 0.0082 0.9021±0.0071± 0.0071 FJD 0.3120±0.0149± 0.0149 0.7045±0.0249± 0.0249 0.6046±0.0148± 0.0148 0.7218±0.0180± 0.0180 Guanaco-7B PPL 0.3707±0.0129± 0.0129 0.5292±0.0072± 0.0072 0.5274±0.0581± 0.0581 0.5542±0.0046± 0.0046 SMLLM - 0.3721±0.0264± 0.0264 0.5419±0.0279± 0.0279 0.6861±0.0132± 0.0132 GradSafe 0.2975±0.0141± 0.0141 0.7520±0.0036± 0.0036 0.6718±0.0066± 0.0066 0.8007±0.0059± 0.0059 FT 0.6132±0.0403± 0.0403 0.4514±0.0502± 0.0502 0.3636±0.0226± 0.0226 0.3327±0.0048± 0.0048 FJD 0.4050±0.0093± 0.0093 0.6398±0.0197± 0.0197 0.5606±0.0079± 0.0079 0.7276±0.0050± 0.0050 Guanaco-13B PPL 0.6667±0.0067± 0.0067 0.7500±0.0142± 0.0142 0.0245±0.0095± 0.0095 0.3373±0.0015± 0.0015 SMLLM - 0.7333±0.0094± 0.0094 0.8426±0.0065± 0.0065 0.8667±0.0047± 0.0047 GradSafe 0.2119±0.0042± 0.0042 0.6623±0.0091± 0.0091 0.7553±0.0053± 0.0053 0.7852±0.0073± 0.0073 FT 0.3712±0.0134± 0.0134 0.5500±0.0187± 0.0187 0.2571±0.0054± 0.0054 0.6656±0.0042± 0.0042 FJD 0.2032±0.0192± 0.0192 0.6510±0.0151± 0.0151 0.5023±0.0018± 0.0018 0.7985±0.0030± 0.0030 Table 12: Detection results (FPR, TPR, F1 and AUC) of jailbreak prompt under GCG. FJD outperforms baseline methods on Llama2 and achieves comparable performance to PPL with other LLMs. Model Method GCG FPR↓ TPR ↑ F1↑ AUC↑ Llama2-7B PPL 0.0624±0.0084± 0.0084 0.9756±0.0054± 0.0054 0.8506±0.0543± 0.0543 0.9717±0.0004± 0.0004 SMLLM - 0.8707±0.0041± 0.0041 0.9308±0.0023± 0.0023 0.9423±0.0027± 0.0027 GradSafe 0.2306±0.0204± 0.0204 0.8148±0.0354± 0.0354 0.8756±0.0212± 0.0212 0.8943±0.0035± 0.0035 FT 0.0188±0.0153± 0.0153 0.9738±0.0032± 0.0032 0.9835±0.0008± 0.0008 0.9939±0.0005± 0.0005 FJD 0.0244±0.0092± 0.0092 0.9905±0.0082± 0.0082 0.9912±0.0041± 0.0041 0.9990±0.0002± 0.0002 Llama2-13B PPL 0.0670±0.0011± 0.0011 0.9465±0.0003± 0.0003 0.9605±0.0054± 0.0054 0.9625±0.0001± 0.0001 SMLLM - 0.9585±0.0099± 0.0099 0.9788±0.0067± 0.0067 0.9798±0.0027± 0.0027 GradSafe 0.3720±0.0102± 0.0102 0.7188±0.0015± 0.0015 0.7640±0.0010± 0.0010 0.7280±0.0076± 0.0076 FT 0.1476±0.0098± 0.0098 0.9537±0.0050± 0.0050 0.9440±0.0013± 0.0013 0.9558±0.0031± 0.0031 FJD 0.0592±0.0043± 0.0043 0.9750±0.0024± 0.0024 0.9651±0.0018± 0.0018 0.9725±0.0010± 0.0010 Vicuna-7B PPL 0.0382±0.0055± 0.0055 0.9717±0.0003± 0.0003 0.9776±0.0038± 0.0038 0.9860±0.0002± 0.0002 SMLLM - 0.8964±0.0110± 0.0110 0.9454±0.0092± 0.0092 0.9575±0.0071± 0.0071 GradSafe 0.2334±0.0249± 0.0249 0.6428±0.0326± 0.0326 0.7305±0.0194± 0.0194 0.7575±0.0117± 0.0117 FT 0.8986±0.0163± 0.0163 0.0827±0.0236± 0.0236 0.0673±0.0087± 0.0087 0.0300±0.0018± 0.0018 FJD 0.2783±0.0292± 0.0292 0.6210±0.0178± 0.0178 0.7031±0.0083± 0.0083 0.7250±0.0044± 0.0044 Vicuna-13B PPL 0.0447±0.0043± 0.0043 0.9892±0.0002± 0.0002 0.9899±0.0023± 0.0023 0.9851±0.0009± 0.0009 SMLLM - 0.8974±0.0036± 0.0036 0.9459±0.0030± 0.0030 0.9550±0.0032± 0.0032 GradSafe 0.1488±0.0267± 0.0267 0.6447±0.0360± 0.0360 0.7788±0.0278± 0.0278 0.7621±0.0090± 0.0090 FT 0.3611±0.0066± 0.0066 0.5687±0.0029± 0.0029 0.6897±0.0020± 0.0020 0.5203±0.0036± 0.0036 FJD 0.1874±0.0271± 0.0271 0.6581±0.0283± 0.0283 0.7539±0.0136± 0.0136 0.7829±0.0128± 0.0128 Guanaco-7B PPL 0.0503±0.0059± 0.0059 0.9803±0.0009± 0.0009 0.9837±0.0034± 0.0034 0.9833±0.0001± 0.0001 SMLLM - 0.7767±0.0083± 0.0083 0.8743±0.0053± 0.0053 0.8811±0.0029± 0.0029 GradSafe 0.4704±0.0108± 0.0108 0.7712±0.0068± 0.0068 0.6695±0.0070± 0.0070 0.7501±0.0019± 0.0019 FT 0.0848±0.0063± 0.0063 0.9145±0.0043± 0.0043 0.9316±0.0027± 0.0027 0.9640±0.0008± 0.0008 FJD 0.1119±0.0095± 0.0095 0.9015±0.0086± 0.0086 0.9129±0.0060± 0.0060 0.9515±0.0040± 0.0040 Guanaco-13B PPL 0.0615±0.0048± 0.0048 0.9758±0.0045± 0.0045 0.9825±0.0037± 0.0037 0.9779±0.0003± 0.0003 SMLLM - 0.8352±0.0117± 0.0117 0.9102±0.0070± 0.0070 0.9150±0.0077± 0.0077 GradSafe 0.1592±0.0093± 0.0093 0.8539±0.0101± 0.0101 0.7518±0.0066± 0.0066 0.8364±0.0084± 0.0084 FT 0.3056±0.0293± 0.0293 0.5825±0.0180± 0.0180 0.7066±0.0129± 0.0129 0.6317±0.0042± 0.0042 FJD 0.2587±0.0369± 0.0369 0.6560±0.0293± 0.0293 0.7648±0.0182± 0.0182 0.7118±0.0041± 0.0041 Table 13: Detection results (FPR, TPR, F1 and AUC) of jailbreak prompt under MAC. FJD outperforms baseline methods on Llama2 and achieves comparable performance to PPL with other LLMs. Model Method MAC FPR↓ TPR↑ F1↑ AUC↑ Llama2-7B PPL 0.0391±0.0016± 0.0016 0.9404±0.0208± 0.0208 0.3192±0.0921± 0.0921 0.9816±0.0001± 0.0001 SMLLM - 0.6482±0.0128± 0.0128 0.7866±0.0123± 0.0123 0.9091±0.0064± 0.0064 GradSafe 0.1001±0.0083± 0.0083 0.9136±0.0077± 0.0077 0.9209±0.0041± 0.0041 0.9565±0.0067± 0.0067 FT 0.0516±0.0032± 0.0032 0.9335±0.0071± 0.0071 0.6156±0.0267± 0.0267 0.9815±0.0022± 0.0022 FJD 0.0325±0.0030± 0.0030 0.9307±0.0073± 0.0073 0.9093±0.0037± 0.0037 0.9839±0.0024± 0.0024 Llama2-13B PPL 0.0411±0.0011± 0.0011 0.9091±0.077± 0.077 0.2179±0.0721± 0.0721 0.9882±0.0003± 0.0003 SMLLM - 0.8667±0.0091± 0.0091 0.9286±0.0058± 0.0058 0.9333±0.0021± 0.0021 GradSafe 0.1813±0.0048± 0.0048 0.9231±0.0362± 0.0362 0.8471±0.0193± 0.0193 0.9398±0.0059± 0.0059 FT 0.0722±0.0037± 0.0037 0.9636±0.0045± 0.0045 0.5345±0.0165± 0.0165 0.9833±0.0048± 0.0048 FJD 0.0397±0.0033± 0.0033 0.9999±0.0001± 0.0001 0.8997±0.0207± 0.0207 0.9964±0.0030± 0.0030 Vicuna-7B PPL 0.0419±0.0092± 0.0092 0.9849±0.0003± 0.0003 0.9218±0.0333± 0.0333 0.9853±0.0005± 0.0005 SMLLM - 0.7673±0.0130± 0.0130 0.8683±0.0083± 0.0083 0.8837±0.0065± 0.0065 GradSafe 0.0873±0.0343± 0.0343 0.8740±0.0306± 0.0306 0.9114±0.0102± 0.0102 0.9686±0.0010± 0.0010 FT 0.7261±0.0040± 0.0040 0.4593±0.0305± 0.0305 0.5237±0.0342± 0.0342 0.2911±0.0044± 0.0044 FJD 0.1964±0.0019± 0.0019 0.8293±0.095± 0.095 0.8561±0.0071± 0.0071 0.8703±0.0101± 0.0101 Vicuna-13B PPL 0.0279±0.0003± 0.0003 0.9813±0.0004± 0.0004 0.9430±0.0249± 0.0249 0.9902±0.0002± 0.0002 SMLLM - 0.9462±0.0044± 0.0044 0.9723±0.0024± 0.0024 0.9730±0.0022± 0.0022 GradSafe 0.2726±0.0066± 0.0066 0.6903±0.0062± 0.0062 0.7743±0.0026± 0.0026 0.7785±0.0021± 0.0021 FT 0.7824±0.0284± 0.0284 0.6021±0.0084± 0.0084 0.6450±0.0059± 0.0059 0.3173±0.0072± 0.0072 FJD 0.2154±0.074± 0.074 0.7847±0.0092± 0.0092 0.8250±0.0079± 0.0079 0.8091±0.0129± 0.0129 Guanaco-7B PPL 0.0514±0.0073± 0.0073 0.9703±0.0005± 0.0005 0.9385±0.0267± 0.0267 0.9867±0.0006± 0.0006 SMLLM - 0.8143±0.0010± 0.0010 0.8976±0.0006± 0.0006 0.9071±0.0005± 0.0005 GradSafe 0.3212±0.0040± 0.0040 0.6049±0.0039± 0.0039 0.7031±0.0018± 0.0018 0.6662±0.0019± 0.0019 FT 0.2118±0.0147± 0.0147 0.7527±0.0100± 0.0100 0.8233±0.0056± 0.0056 0.8076±0.0083± 0.0083 FJD 0.1328±0.0117± 0.0117 0.8584±0.0068± 0.0068 0.9006±0.0041± 0.0041 0.9378±0.0029± 0.0029 Guanaco-13B PPL 0.0255±0.0044± 0.0044 0.9804±0.0002± 0.0002 0.5476±0.1103± 0.1103 0.9895±0.0001± 0.0001 SMLLM - 0.8798±0.0077± 0.0077 0.9360±0.0044± 0.0044 0.9399±0.0039± 0.0039 GradSafe 0.2478±0.0102± 0.0102 0.7758±0.0255± 0.0255 0.6437±0.0177± 0.0177 0.8271±0.0072± 0.0072 FT 0.9889±0.0063± 0.0063 0.9020±0.0328± 0.0328 0.2591±0.0071± 0.0071 0.1424±0.0044± 0.0044 FJD 0.2295±0.0063± 0.0063 0.7686±0.0328± 0.0328 0.5176±0.0071± 0.0071 0.8490±0.0044± 0.0044 Appendix H Jailbreak Detection under Attacks with Mismatched Generalization In order to fully evaluate the performance of FJD under attacks via mismatched generalization, we supplement Cipher experiments on Llama2 7B/13B, Vicuna 7B/13B and Guanaco 7B/13B in Tab. 14. We supplement PAIR experiments on Vicuna 7B/13B and Llama2 7B/13B. In Tab. 15 illustrates the detection results (AUC) of jailbreak prompt and shows the effective detection of Jailbreak Prompts by FJD under PAIR attack. For the two jailbreak attacks, FJD can effectively detect these on all models. Appendix I Jailbreak Detection under Hand-crafted Attacks We concurrently assess the detection efficacy of FJD on 28 manual attack methods in Hand-Crafted [10] method on Llama2 7B/13B (Tab. 16, 17), Vicuna 7B/13B (Tab. 18, 19) and Guanaco 7B/13B (Tab. 20, 21). Both attack methods are human-readable, and FJD achieves the best performance on competing objectives and mismatched generalization. We hypothesize that this is attributed to the low perplexity of jailbreak prompts created by hand-crafted or semantically meaningful jailbreaks. Furthermore, benign prompts also exhibit relatively high perplexity, leading to PPL essentially performing reverse detection. Appendix J Jailbreak Detection under Transferable Jailbreak Attack We also provide complete jailbreak detection results under transferable attacks. This experiment employs Vicuna 7B, Llama2 7B and Guanaco 7B as the source models and aggregates jailbreak prompts acquired from GCG and AutoDAN. We systematically merge Vicuna 7B, Llama2 7B and Guanaco 7B to produce transferable jailbreak prompts using the transferable attack method within GCG. Then, we evaluate Vicuna 7B/13B, Llama2 7B/13B and Guanaco 7B/13B as the target models. In Tab. 22 shows that, for the comprehensive migration of a successful jailbreak prompt generated on a single model, FJD demonstrates a more effective detection capability. In the case of jailbreak prompts generated by GCG transferable attack, FJD also demonstrates competitive results compared to PPL, which almost requires no extra model inference. Table 14: Detection results (FPR, TPR, F1 and AUC) of jailbreak prompt under Cipher. FJD outperforms baseline methods on almost all the LLMs. Model Method Cipher FPR↓ TPR↑ F1↑ AUC↑ Llama2-7B PPL 0.9672±0.0013± 0.0013 0.0038±0.0008± 0.0008 0.0069±0.0005± 0.0005 0.0070±0.0005± 0.0005 SMLLM - 0.0101±0.0048± 0.0048 0.0200±0.0094± 0.0094 0.5034±0.0024± 0.0024 GradSafe 0.2070±0.0092± 0.0092 0.6345±0.0096± 0.0096 0.5477±0.0137± 0.0137 0.7862±0.0045± 0.0045 FT 0.0629±0.0067± 0.0067 0.9812±0.0147± 0.0147 0.8730±0.0198± 0.0198 0.9636±0.0025± 0.0025 FJD 0.0386±0.0077± 0.0077 0.9845±0.0096± 0.0096 0.9257±0.0203± 0.0203 0.9896±0.0014± 0.0014 Llama2-13B PPL 0.9978±0.0065± 0.0065 0.0089±0.0003± 0.0003 0.0076±0.0002± 0.0002 0.0221±0.0011± 0.0011 SMLLM - 0.8192±0.0211± 0.0211 0.8211±0.0096± 0.0096 0.9096±0.0105± 0.0105 GradSafe 0.1513±0.0098± 0.0098 0.7831±0.0237± 0.0237 0.6340±0.0198± 0.0198 0.8723±0.0073± 0.0073 FT 0.0493±0.0069± 0.0069 0.9839±0.0126± 0.0126 0.8901±0.0135± 0.0135 0.9837±0.0031± 0.0031 FJD 0.0114±0.0037± 0.0037 0.9869±0.0102± 0.0102 0.9658±0.0109± 0.0109 0.9909±0.0091± 0.0091 Vicuna-7B PPL 0.9876±0.0051± 0.0051 0.0512±0.0039± 0.0039 0.0043±0.0006± 0.0006 0.0266±0.0004± 0.0004 SMLLM - 0.0465±0.0019± 0.0019 0.0889±0.0034± 0.0034 0.5233±0.0009± 0.0009 GradSafe 0.4190±0.0199± 0.0199 0.7549±0.0303± 0.0303 0.7284±0.0136± 0.0136 0.7094±0.0201± 0.0201 FT 0.2731±0.0267± 0.0267 0.7329±0.0110± 0.0110 0.8150±0.0051± 0.0051 0.7966±0.0055± 0.0055 FJD 0.1960±0.0189± 0.0189 0.8362±0.0164± 0.0164 0.8474±0.0053± 0.0053 0.8633±0.0033± 0.0033 Vicuna-13B PPL 0.9913±0.0110± 0.0110 0.0477±0.0015± 0.0015 0.0036±0.0002± 0.0002 0.0259±0.0005± 0.0005 SMLLM - 0.0690±0.0050± 0.0050 0.0110±0.0084± 0.0084 0.5344±0.0025± 0.0025 GradSafe 0.1894±0.0041± 0.0041 0.6683±0.0111± 0.0111 0.7783±0.0073± 0.0073 0.7972±0.0036± 0.0036 FT 0.7262±0.0125± 0.0125 0.6528±0.0271± 0.0271 0.6712±0.0245± 0.0245 0.3030±0.0150± 0.0150 FJD 0.1405±0.0156± 0.0156 0.9918±0.0046± 0.0046 0.9680±0.0047± 0.0047 0.8876±0.0170± 0.0170 Guanaco-7B PPL 0.9803±0.0095± 0.0095 0.0396±0.0003± 0.0003 0.0013±0.0003± 0.0003 0.0248±0.0005± 0.0005 SMLLM - 0.0919±0.0052± 0.0052 0.1683±0.0087± 0.0087 0.5460±0.0026± 0.0026 GradSafe 0.3391±0.0197± 0.0197 0.6607±0.0040± 0.0040 0.7569±0.0029± 0.0029 0.8112±0.0088± 0.0088 FT 0.9729±0.0190± 0.0190 0.7528±0.0215± 0.0215 0.2699±0.0146± 0.0146 0.4905±0.0173± 0.0173 FJD 0.2610±0.0277± 0.0277 0.8122±0.0243± 0.0243 0.8307±0.0120± 0.0120 0.8299±0.0043± 0.0043 Guanaco-13B PPL 0.9782±0.0071± 0.0071 0.0374±0.0005± 0.0005 0.0051±0.0002± 0.0002 0.0254±0.0008± 0.0008 SMLLM - 0.0964±0.0039± 0.0039 0.1724±0.0066± 0.0066 0.5482±0.0020± 0.0020 GradSafe 0.3418±0.0116± 0.0116 0.7401±0.0227± 0.0227 0.7425±0.0050± 0.0050 0.7691±0.0105± 0.0105 FT 0.6230±0.0250± 0.0250 0.7723±0.0236± 0.0236 0.7624±0.0237± 0.0237 0.4724±0.0148± 0.0148 FJD 0.2825±0.0299± 0.0299 0.8415±0.0235± 0.0235 0.8810±0.0223± 0.0223 0.8216±0.0191± 0.0191 Table 15: Detection results (FPR, TPR, F1 and AUC) of jailbreak prompt under PAIR. FJD outperforms baseline methods on almost all the LLMs. Model Method PAIR FPR↓ TPR↑ F1↑ AUC↑ Llama2-7B PPL 0.7897±0.0144± 0.0144 0.0382±0.0008± 0.0008 0.0823±0.0250± 0.0250 0.2715±0.0061± 0.0061 SMLLM - 0.7423±0.0158± 0.0158 0.8502±0.0110± 0.0110 0.8625±0.0019± 0.0019 GradSafe 0.0681±0.0097± 0.0097 0.9625±0.0076± 0.0076 0.7952±0.0121± 0.0121 0.9697±0.0056± 0.0056 FT 0.0937±0.0040± 0.0040 0.9750±0.0125± 0.0125 0.7040±0.0093± 0.0093 0.9470±0.0028± 0.0028 FJD 0.0516±0.0212± 0.0212 0.9687±0.0087± 0.0087 0.8042±0.0059± 0.0059 0.9761±0.0009± 0.0009 Llama2-13B PPL 0.9367±0.0033± 0.0033 0.0067±0.0009± 0.0009 0.0088±0.0007± 0.0007 0.1140±0.0142± 0.0142 SMLLM - 0.8889±0.0079± 0.0079 0.9394±0.0043± 0.0043 0.9244±0.0024± 0.0024 GradSafe 0.1161±0.0031± 0.0031 0.9998±0.0002± 0.0002 0.8797±0.0195± 0.0195 0.9185±0.0029± 0.0029 FT 0.1674±0.0039± 0.0039 0.9667±0.0082± 0.0082 0.9586±0.0030± 0.0030 0.9153±0.0039± 0.0039 FJD 0.1024±0.0011± 0.0011 1.0000±0.0000± 0.0000 0.9732±0.0021± 0.0021 0.9264±0.0013± 0.0013 Vicuna-7B PPL 0.8886±0.0032± 0.0032 0.1222±0.0006± 0.0006 0.2256±0.0167± 0.0167 0.3245±0.0024± 0.0024 SMLLM - 0.7622±0.0074± 0.0074 0.8615±0.0135± 0.0135 0.8738±0.0082± 0.0082 GradSafe 0.2174±0.0207± 0.0207 0.8169±0.0122± 0.0122 0.7998±0.0159± 0.0159 0.8987±0.0024± 0.0024 FT 0.4738±0.0081± 0.0081 0.5999±0.0167± 0.0167 0.4770±0.0127± 0.0127 0.5526±0.0054± 0.0054 FJD 0.1452±0.0094± 0.0094 0.8702±0.0120± 0.0120 0.8079±0.0128± 0.0128 0.9025±0.0027± 0.0027 Vicuna-13B PPL 0.4701±0.0471± 0.0471 0.3333±0.0114± 0.0114 0.0991±0.0232± 0.0232 0.2272±0.0010± 0.0010 SMLLM - 0.9167±0.0035± 0.0035 0.9562±0.0190± 0.0190 0.9583±0.0172± 0.0172 GradSafe 0.2007±0.0332± 0.0332 0.8428±0.0424± 0.0424 0.7163±0.0262± 0.0262 0.8068±0.0098± 0.0098 FT 0.5120±0.0050± 0.0050 0.7762±0.0149± 0.0149 0.0539±0.0088± 0.0088 0.5285±0.0077± 0.0077 FJD 0.0332±0.0023± 0.0023 0.9895±0.0100± 0.0100 0.9358±0.0109± 0.0109 0.9957±0.0009± 0.0009 Table 16: Detection results (AUC) of jailbreak prompt under Hand-crafted attacks on Llama2 7B. FJD outperforms baseline methods on almost all attacks and LLMs. Attack on Llama2-7B PPL SMLLM GradSafe FT FJD aim 0.5228±0.0004± 0.0004 0.6283±0.0027± 0.0027 0.9892±0.0012± 0.0012 0.9608±0.0078± 0.0078 0.9956±0.0031± 0.0031 dev_mode_v2 0.4289±0.0015± 0.0015 0.5050±0.0012± 0.0012 0.9746±0.0009± 0.0009 0.9812±0.0023± 0.0023 0.9985±0.0018± 0.0018 dev_mode_ranti 0.5485±0.0003± 0.0003 0.5219±0.0015± 0.0015 0.9825±0.0059± 0.0059 0.9829±0.0016± 0.0016 0.9995±0.0007± 0.0007 distractors 0.6897±0.0042± 0.0042 0.9514±0.0354± 0.0354 0.8236±0.0090± 0.0090 0.8510±0.0045± 0.0045 0.9024±0.0289± 0.0289 distractors_negated 0.9718±0.0003± 0.0003 0.9991±0.0002± 0.0002 0.8978±0.0016± 0.0016 0.7267±0.0135± 0.0135 0.8167±0.0161± 0.0161 evil_confidant 0.8422±0.0017± 0.0017 0.5632±0.0065± 0.0065 0.9989±0.0015± 0.0015 0.9998±0.0004± 0.0004 0.9973±0.0022± 0.0022 poems 0.9377±0.0029± 0.0029 0.9087±0.0022± 0.0022 0.9241±0.0015± 0.0015 0.8584±0.0032± 0.0032 0.9406±0.0028± 0.0028 prefix_injection_1 0.9578±0.0013± 0.0013 0.8571±0.0111± 0.0111 0.9091±0.0085± 0.0085 0.8962±0.0069± 0.0069 0.9546±0.0109± 0.0109 prefix_injection_2 0.1477±0.0016± 0.0016 0.7381±0.0168± 0.0168 0.9231±0.0016± 0.0016 0.9714±0.0035± 0.0035 0.9926±0.0040± 0.0040 prefix_injection_hello 0.8529±0.0170± 0.0170 0.9258±0.0121± 0.0121 0.8889±0.0104± 0.0104 0.9467±0.0035± 0.0035 0.9851±0.0057± 0.0057 refusal_suppression 0.0073±0.0005± 0.0005 0.5552±0.0231± 0.0231 0.9832±0.0008± 0.0008 0.9043±0.0024± 0.0024 0.9809±0.0007± 0.0007 refusal_suppression_inv 0.0094±0.0008± 0.0008 0.5619±0.0210± 0.0210 0.9919±0.0017± 0.0017 0.9722±0.0036± 0.0036 0.9956±0.0030± 0.0030 style_injection_short 0.0068±0.0001± 0.0001 0.5519±0.0026± 0.0026 0.9232±0.0085± 0.0085 0.9652±0.0029± 0.0029 0.9724±0.0057± 0.0057 Average of CO 0.5326±0.0025± 0.0025 0.7129±0.0105± 0.0105 0.9392±0.0041± 0.0041 0.9244±0.0043± 0.0043 0.9640±0.0067± 0.0067 auto_payload_splitting 0.9290±0.0005± 0.0005 0.5670±0.0053± 0.0053 0.9853±0.0007± 0.0007 0.6133±0.0133± 0.0133 0.8081±0.0114± 0.0114 base64 0.9205±0.0003± 0.0003 0.5313±0.0059± 0.0059 0.9643±0.0047± 0.0047 0.9939±0.0009± 0.0009 0.9884±0.0039± 0.0039 base64_raw 0.9191±0.0004± 0.0004 0.5063±0.0017± 0.0017 0.9638±0.0018± 0.0018 0.9826±0.0046± 0.0046 0.9305±0.0076± 0.0076 base64_input_only 0.9281±0.0006± 0.0006 0.8996±0.0062± 0.0062 0.9376±0.0092± 0.0092 0.9939±0.0020± 0.0020 0.9954±0.0008± 0.0008 base64_output_only 0.9240±0.0031± 0.0031 0.7796±0.0274± 0.0274 0.9504±0.0049± 0.0049 0.7333±0.0027± 0.0027 0.9794±0.0078± 0.0078 combination_1 0.0031±0.0001± 0.0001 0.5050±0.0033± 0.0033 0.6328±0.0189± 0.0189 0.9918±0.0040± 0.0040 0.9770±0.0072± 0.0072 combination_2 0.0031±0.0001± 0.0001 0.5379±0.0028± 0.0028 0.6300±0.0043± 0.0043 0.9929±0.0022± 0.0022 0.9786±0.0018± 0.0018 combination_3 0.0053±0.0001± 0.0001 0.5682±0.0030± 0.0030 0.6734±0.0147± 0.0147 0.9916±0.0026± 0.0026 0.9869±0.0037± 0.0037 disemvowel 0.9895±0.0004± 0.0004 0.9792±0.0295± 0.0295 0.9398±0.0015± 0.0015 0.9908±0.0047± 0.0047 0.9262±0.0152± 0.0152 few_shot_json 0.0104±0.0007± 0.0007 0.5218±0.0024± 0.0024 0.8938±0.0010± 0.0010 0.9385±0.0093± 0.0093 0.9872±0.0024± 0.0024 leetspeak 0.9797±0.0011± 0.0011 0.9111±0.0240± 0.0240 0.9258±0.0064± 0.0064 0.8975±0.0023± 0.0023 0.9314±0.0086± 0.0086 rot13 0.9993±0.0002± 0.0002 0.9958±0.0059± 0.0059 0.9325±0.0073± 0.0073 0.9778±0.0002± 0.0002 0.9823±0.0025± 0.0025 style_injection_json 0.9176±0.0101± 0.0101 0.9457±0.0128± 0.0128 0.9120±0.0061± 0.0061 0.9693±0.0032± 0.0032 0.9940±0.0043± 0.0043 wikipedia 0.8210±0.0011± 0.0011 0.9167±0.0118± 0.0118 0.8980±0.0020± 0.0020 0.8525±0.0267± 0.0267 0.8629±0.0296± 0.0296 wikipedia_with_title 0.9315±0.0025± 0.0025 0.9593±0.0239± 0.0239 0.9252±0.0031± 0.0031 0.9233±0.0036± 0.0036 0.9946±0.0015± 0.0015 Average of MG 0.6854±0.0014± 0.0014 0.7146±0.0111± 0.0111 0.8777±0.0058± 0.0058 0.9229±0.0055± 0.0055 0.9549±0.0072± 0.0072 Table 17: Detection results (AUC) of jailbreak prompt under Hand-crafted attacks on Llama2 13B. FJD outperforms baseline methods on almost all attacks and LLMs. Attack on Llama2-13B PPL SMLLM GradSafe FT FJD aim 0.5244±0.0005± 0.0005 0.7185±0.0029± 0.0029 0.9886±0.0010± 0.0010 0.6650±0.0297± 0.0297 0.9997±0.0002± 0.0002 dev_mode_v2 0.4292±0.0003± 0.0003 0.6128±0.0019± 0.0019 0.9943±0.0009± 0.0009 0.9774±0.0015± 0.0015 0.9974±0.0005± 0.0005 dev_mode_ranti 0.5485±0.0010± 0.0010 0.6379±0.0021± 0.0021 0.9728±0.0026± 0.0026 0.6893±0.0094± 0.0094 0.9826±0.0011± 0.0011 distractors 0.6906±0.0040± 0.0040 0.8955±0.0362± 0.0362 0.8627±0.0215± 0.0215 0.8397±0.093± 0.093 0.8469±0.0144± 0.0144 distractors_negated 0.9680±0.0034± 0.0034 0.9523±0.0122± 0.0122 0.8934±0.0015± 0.0015 0.8244±0.0087± 0.0087 0.8947±0.0074± 0.0074 evil_confidant 0.8415±0.0015± 0.0015 0.5657±0.0069± 0.0069 0.9643±0.0021± 0.0021 0.8843±0.0023± 0.0023 0.9665±0.0030± 0.0030 poems 0.9225±0.0007± 0.0007 0.9478±0.0048± 0.0048 0.9773±0.0066± 0.0066 0.9486±0.0047± 0.0047 0.9631±0.0056± 0.0056 prefix_injection_1 0.9733±0.0003± 0.0003 0.7312±0.0099± 0.0099 0.9675±0.0018± 0.0018 0.9536±0.0081± 0.0081 0.9792±0.0017± 0.0017 prefix_injection_2 0.1042±0.0104± 0.0104 0.7039±0.0152± 0.0152 0.9893±0.0016± 0.0016 0.9063±0.0055± 0.0055 0.9996±0.0005± 0.0005 prefix_injection_hello 0.8237±0.0075± 0.0075 0.8837±0.0129± 0.0129 0.9963±0.0012± 0.0012 0.7619±0.0161± 0.0161 0.9990±0.0009± 0.0009 refusal_suppression 0.0035±0.0003± 0.0003 0.5121±0.0177± 0.0177 0.9252±0.0023± 0.0023 0.6059±0.0108± 0.0108 0.9352±0.0054± 0.0054 refusal_suppression_inv 0.0051±0.0004± 0.0004 0.6284±0.0173± 0.0173 0.9776±0.0033± 0.0033 0.8568±0.0094± 0.0094 0.9987±0.0016± 0.0016 style_injection_short 0.0027±0.0002± 0.0002 0.5610±0.0033± 0.0033 0.9949±0.0008± 0.0008 0.8564±0.0179± 0.0179 0.9826±0.0150± 0.0150 Average of CO 0.5259±0.0023± 0.0023 0.7193±0.0110± 0.0110 0.9619±0.0036± 0.0036 0.8284±0.0167± 0.0167 0.9650±0.0044± 0.0044 auto_payload_splitting 0.9290±0.0011± 0.0011 0.9454±0.0048± 0.0048 0.9780±0.0017± 0.0017 0.6326±0.0327± 0.0327 0.9863±0.0106± 0.0106 base64 0.9264±0.0009± 0.0009 0.7655±0.0121± 0.0121 0.9412±0.0048± 0.0048 0.8416±0.0109± 0.0109 0.9428±0.0070± 0.0070 base64_raw 0.9201±0.0005± 0.0005 0.6926±0.0061± 0.0061 0.7832±0.0116± 0.0116 0.4950±0.0067± 0.0067 0.9578±0.0049± 0.0049 base64_input_only 0.9264±0.0008± 0.0008 0.7290±0.0055± 0.0055 0.9419±0.0096± 0.0096 0.8813±0.0081± 0.0081 0.9482±0.0058± 0.0058 base64_output_only 0.8980±0.0065± 0.0065 0.9045±0.0115± 0.0115 0.8943±0.0054± 0.0054 0.7232±0.0065± 0.0065 0.9486±0.0063± 0.0063 combination_1 0.0031±0.0001± 0.0001 0.5151±0.0023± 0.0023 0.5120±0.0023± 0.0023 0.4738±0.0152± 0.0152 0.8133±0.0230± 0.0230 combination_2 0.0032±0.0003± 0.0003 0.5284±0.0027± 0.0027 0.5082±0.0114± 0.0114 0.4864±0.0137± 0.0137 0.8896±0.0178± 0.0178 combination_3 0.0051±0.0003± 0.0003 0.5168±0.0030± 0.0030 0.6146±0.0241± 0.0241 0.5668±0.0124± 0.0124 0.9989±0.0003± 0.0003 disemvowel 0.9894±0.0007± 0.0007 0.5889±0.0048± 0.0048 0.9041±0.0014± 0.0014 0.8387±0.0156± 0.0156 0.8430±0.0162± 0.0162 few_shot_json 0.0041±0.0002± 0.0002 0.5635±0.0022± 0.0022 0.9942±0.0051± 0.0051 0.9260±0.0159± 0.0159 0.9953±0.0024± 0.0024 leetspeak 0.9815±0.0005± 0.0005 0.9114±0.0040± 0.0040 0.9641±0.0080± 0.0080 0.9341±0.0140± 0.0140 0.9771±0.0049± 0.0049 rot13 0.9896±0.0003± 0.0003 0.9374±0.0078± 0.0078 0.8500±0.0056± 0.0056 0.9146±0.0118± 0.0118 0.9618±0.0148± 0.0148 style_injection_json 0.9067±0.0036± 0.0036 0.8610±0.0159± 0.0159 0.8962±0.0076± 0.0076 0.7919±0.0135± 0.0135 0.9598±0.0030± 0.0030 wikipedia 0.8089±0.0067± 0.0067 0.9480±0.0177± 0.0177 0.9697±0.0031± 0.0031 0.9134±0.0153± 0.0153 0.9444±0.0108± 0.0108 wikipedia_with_title 0.8890±0.0019± 0.0019 0.9725±0.0212± 0.0212 0.9994±0.0005± 0.0005 0.9155±0.0245± 0.0245 0.9998±0.0002± 0.0002 Average 0.6787±0.0016± 0.0016 0.7587±0.0081± 0.0081 0.8501±0.0068± 0.0068 0.7557±0.0145± 0.0145 0.9444±0.0085± 0.0085 Table 18: Detection results (AUC) of jailbreak prompt under Hand-crafted attacks on Vicuna 7B. FJD outperforms baseline methods on almost all attacks and LLMs. Attack on Vicuna-7B PPL SMLLM GradSafe FT FJD aim 0.5250±0.0004± 0.0004 0.5077±0.0036± 0.0036 0.6688±0.0083± 0.0083 0.2783±0.0167± 0.0167 0.8976±0.0074± 0.0074 dev_mode_v2 0.4342±0.0006± 0.0006 0.5424±0.0064± 0.0064 0.8558±0.0025± 0.0025 0.2140±0.0131± 0.0131 0.8393±0.0075± 0.0075 dev_mode_ranti 0.5498±0.0004± 0.0004 0.5181±0.0026± 0.0026 0.8567±0.0087± 0.0087 0.5766±0.0304± 0.0304 0.8763±0.0106± 0.0106 distractors 0.6794±0.0007± 0.0007 0.5944±0.0052± 0.0052 0.7558±0.0066± 0.0066 0.6616±0.0160± 0.0160 0.8969±0.0201± 0.0201 distractors_negated 0.9643±0.0001± 0.0001 0.7833±0.0103± 0.0103 0.7646±0.0086± 0.0086 0.6123±0.0150± 0.0150 0.7121±0.0174± 0.0174 evil_confidant 0.8432±0.0004± 0.0004 0.5042±0.0029± 0.0029 0.7116±0.0139± 0.0139 0.0989±0.0108± 0.0108 0.8586±0.0073± 0.0073 poems 0.9260±0.0004± 0.0004 0.6472±0.0071± 0.0071 0.7783±0.0053± 0.0053 0.6799±0.0105± 0.0105 0.7953±0.0199± 0.0199 prefix_injection_1 0.9697±0.0002± 0.0002 0.8875±0.0029± 0.0029 0.7911±0.0035± 0.0035 0.1724±0.0203± 0.0203 0.7741±0.0084± 0.0084 prefix_injection_2 0.1291±0.0043± 0.0043 0.5218±0.0074± 0.0074 0.8254±0.0044± 0.0044 0.0269±0.0071± 0.0071 0.6244±0.0191± 0.0191 prefix_injection_hello 0.8513±0.0015± 0.0015 0.6972±0.0055± 0.0055 0.7377±0.0076± 0.0076 0.3405±0.0149± 0.0149 0.5606±0.0132± 0.0132 refusal_suppression 0.0076±0.0001± 0.0001 0.9090±0.0043± 0.0043 0.8881±0.0032± 0.0032 0.6787±0.0176± 0.0176 0.8965±0.0174± 0.0174 refusal_suppression_inv 0.0082±0.0001± 0.0001 0.9465±0.0080± 0.0080 0.8174±0.0037± 0.0037 0.5201±0.0192± 0.0192 0.8635±0.0160± 0.0160 style_injection_short 0.0068±0.0001± 0.0001 0.5417±0.0061± 0.0061 0.7893±0.0035± 0.0035 0.7456±0.0114± 0.0114 0.8670±0.0122± 0.0122 Average of CO 0.5304±0.0007± 0.0007 0.6616±0.0056± 0.0056 0.7877±0.0061± 0.0061 0.4312±0.0156± 0.0156 0.8048±0.0135± 0.0135 auto_payload_splitting 0.9604±0.0002± 0.0002 0.6726±0.0085± 0.0085 0.8068±0.0023± 0.0023 0.5218±0.0159± 0.0159 0.7296±0.0153± 0.0153 base64 0.9206±0.0013± 0.0013 0.7671±0.0045± 0.0045 0.8002±0.0034± 0.0034 0.8508±0.0095± 0.0095 0.9133±0.0028± 0.0028 base64_raw 0.9172±0.0010± 0.0010 0.5937±0.0058± 0.0058 0.8051±0.0063± 0.0063 0.7521±0.0068± 0.0068 0.8064±0.0149± 0.0149 base64_input_only 0.9264±0.0001± 0.0001 0.8646±0.0079± 0.0079 0.9016±0.0035± 0.0035 0.7544±0.0151± 0.0151 0.8542±0.0293± 0.0293 base64_output_only 0.8792±0.0008± 0.0008 0.7806±0.0149± 0.0149 0.8797±0.0040± 0.0040 0.7957±0.0179± 0.0179 0.8762±0.0232± 0.0232 combination_1 0.0033±0.0001± 0.0001 0.5281±0.0047± 0.0047 0.6365±0.0058± 0.0058 0.0930±0.0159± 0.0159 0.7703±0.0124± 0.0124 combination_2 0.0032±0.0001± 0.0001 0.5293±0.0083± 0.0083 0.6847±0.0028± 0.0028 0.0519±0.0110± 0.0110 0.7570±0.0116± 0.0116 combination_3 0.0053±0.0001± 0.0001 0.5022±0.0008± 0.0008 0.6520±0.0135± 0.0135 0.1705±0.0155± 0.0155 0.7713±0.0220± 0.0220 disemvowel 0.9895±0.0004± 0.0004 0.8174±0.0121± 0.0121 0.8583±0.0038± 0.0038 0.5317±0.0189± 0.0189 0.7747±0.0180± 0.0180 few_shot_json 0.0035±0.0003± 0.0003 0.8521±0.0061± 0.0061 0.7425±0.0049± 0.0049 0.7443±0.0170± 0.0170 0.7556±0.0128± 0.0128 leetspeak 0.9784±0.0010± 0.0010 0.5563±0.0017± 0.0017 0.8740±0.0022± 0.0022 0.6685±0.0157± 0.0157 0.8160±0.0250± 0.0250 rot13 0.9994±0.0002± 0.0002 0.7938±0.0090± 0.0090 0.8020±0.0082± 0.0082 0.7560±0.0177± 0.0177 0.8446±0.0142± 0.0142 style_injection_json 0.9176±0.0101± 0.0101 0.6125±0.0045± 0.0045 0.7889±0.0100± 0.0100 0.4890±0.0106± 0.0106 0.7238±0.0100± 0.0100 wikipedia 0.8281±0.0026± 0.0026 0.9868±0.0043± 0.0043 0.7781±0.0003± 0.0003 0.7454±0.0162± 0.0162 0.7851±0.0074± 0.0074 wikipedia_with_title 0.9084±0.0005± 0.0005 0.8750±0.0112± 0.0112 0.7860±0.0020± 0.0020 0.5131±0.0137± 0.0137 0.7279±0.0205± 0.0205 Average of MG 0.6827±0.0013± 0.0013 0.7155±0.0070± 0.0070 0.7864±0.0049± 0.0049 0.5625±0.0145± 0.0145 0.7937±0.0160± 0.0160 Table 19: Detection results (AUC) of jailbreak prompt under Hand-crafted attacks on Vicuna 13B. FJD outperforms baseline methods on almost all attacks and LLMs. Attack on Vicuna-13B PPL SMLLM GradSafe FT FJD aim 0.5254±0.0009± 0.0009 0.5014±0.0010± 0.0010 0.9409±0.0032± 0.0032 0.2218±0.0128± 0.0128 0.9458±0.0133± 0.0133 dev_mode_v2 0.4302±0.0002± 0.0002 0.8333±0.0059± 0.0059 0.8126±0.0052± 0.0052 0.4567±0.0186± 0.0186 0.9491±0.0121± 0.0121 dev_mode_ranti 0.5484±0.0001± 0.0001 0.6340±0.0065± 0.0065 0.8086±0.0031± 0.0031 0.5842±0.0049± 0.0049 0.9303±0.0022± 0.0022 distractors 0.6832±0.0007± 0.0007 0.7452±0.0242± 0.0242 0.7118±0.0060± 0.0060 0.6271±0.0057± 0.0057 0.9699±0.0024± 0.0024 distractors_negated 0.9624±0.0005± 0.0005 0.9899±0.0072± 0.0072 0.9864±0.0037± 0.0037 0.6944±0.0168± 0.0168 0.9251±0.0120± 0.0120 evil_confidant 0.8418±0.0005± 0.0005 0.5094±0.0010± 0.0010 0.6169±0.0034± 0.0034 0.4899±0.0343± 0.0343 0.9527±0.0124± 0.0124 poems 0.9250±0.0004± 0.0004 0.9513±0.0053± 0.0053 0.7733±0.0081± 0.0081 0.6919±0.0266± 0.0266 0.9984±0.0139± 0.0139 prefix_injection_1 0.9605±0.0015± 0.0015 0.9403±0.0156± 0.0156 0.9126±0.0018± 0.0018 0.5745±0.0166± 0.0166 0.9278±0.0081± 0.0081 prefix_injection_2 0.1292±0.0011± 0.0011 0.5731±0.0063± 0.0063 0.6094±0.0165± 0.0165 0.2526±0.0076± 0.0076 0.9244±0.0065± 0.0065 prefix_injection_hello 0.8464±0.0009± 0.0009 0.9760±0.0006± 0.0006 0.5527±0.0069± 0.0069 0.4665±0.0172± 0.0172 0.9114±0.0066± 0.0066 refusal_suppression 0.0068±0.0003± 0.0003 0.5726±0.0049± 0.0049 0.8108±0.0032± 0.0032 0.6829±0.0214± 0.0214 0.9590±0.0125± 0.0125 refusal_suppression_inv 0.0063±0.0002± 0.0002 0.9825±0.0070± 0.0070 0.8392±0.0087± 0.0087 0.6891±0.0125± 0.0125 0.9529±0.0073± 0.0073 style_injection_short 0.0070±0.0001± 0.0001 0.5058±0.0123± 0.0123 0.9822±0.0021± 0.0021 0.7312±0.0204± 0.0204 0.9951±0.0059± 0.0059 Average of CO 0.5287±0.0006± 0.0006 0.7473±0.0075± 0.0075 0.7967±0.0055± 0.0055 0.5510±0.0166± 0.0166 0.9494±0.0089± 0.0089 auto_payload_splitting 0.9612±0.0008± 0.0008 0.6709±0.0107± 0.0107 0.5258±0.0065± 0.0065 0.4448±0.0260± 0.0260 0.9477±0.0036± 0.0036 base64 0.9200±0.0001± 0.0001 0.5232±0.0030± 0.0030 0.5501±0.0070± 0.0070 0.7413±0.0061± 0.0061 0.9431±0.0205± 0.0205 base64_raw 0.9218±0.0004± 0.0004 0.7395±0.0126± 0.0126 0.5155±0.0057± 0.0057 0.7450±0.0111± 0.0111 0.9713±0.0151± 0.0151 base64_input_only 0.9271±0.0002± 0.0002 0.7448±0.0085± 0.0085 0.6481±0.0078± 0.0078 0.6932±0.0300± 0.0300 0.9548±0.0116± 0.0116 base64_output_only 0.8879±0.0030± 0.0030 0.6027±0.0117± 0.0117 0.9589±0.0009± 0.0009 0.7283±0.0272± 0.0272 0.9204±0.0109± 0.0109 combination_1 0.0031±0.0001± 0.0001 0.5843±0.0045± 0.0045 0.9385±0.0043± 0.0043 0.5631±0.0192± 0.0192 0.9564±0.0084± 0.0084 combination_2 0.0030±0.0001± 0.0001 0.5221±0.0049± 0.0049 0.9425±0.0018± 0.0018 0.5544±0.0071± 0.0071 0.9565±0.0078± 0.0078 combination_3 0.0054±0.0001± 0.0001 0.5508±0.0039± 0.0039 0.9533±0.0025± 0.0025 0.6522±0.0161± 0.0161 0.9691±0.0044± 0.0044 disemvowel 0.9995±0.0001± 0.0001 0.7070±0.0099± 0.0099 0.9125±0.0087± 0.0087 0.7155±0.0096± 0.0096 0.9903±0.0021± 0.0021 few_shot_json 0.0079±0.0001± 0.0001 0.6630±0.0078± 0.0078 0.9581±0.0011± 0.0011 0.6996±0.0091± 0.0091 0.9707±0.0089± 0.0089 leetspeak 0.9759±0.0006± 0.0006 0.5747±0.0037± 0.0037 0.9455±0.0005± 0.0005 0.7210±0.0091± 0.0091 0.9257±0.0109± 0.0109 rot13 0.9935±0.0006± 0.0006 0.6806±0.0035± 0.0035 0.9882±0.0011± 0.0011 0.7488±0.0124± 0.0124 0.9051±0.0179± 0.0179 style_injection_json 0.9031±0.0017± 0.0017 0.6109±0.0094± 0.0094 0.5256±0.0052± 0.0052 0.6661±0.0209± 0.0209 0.9045±0.0147± 0.0147 wikipedia 0.7794±0.0011± 0.0011 0.9583±0.0295± 0.0295 0.9967±0.0011± 0.0011 0.7066±0.0262± 0.0262 0.9688±0.0126± 0.0126 wikipedia_with_title 0.9065±0.0008± 0.0008 0.9096±0.0126± 0.0126 0.9186±0.0047± 0.0047 0.5204±0.0216± 0.0216 0.9813±0.0069± 0.0069 Average of MG 0.6797±0.0007± 0.0007 0.6695±0.0091± 0.0091 0.8185±0.0039± 0.0039 0.6600±0.0168± 0.0168 0.9510±0.0104± 0.0104 Table 20: Detection results (AUC) of jailbreak prompt under Hand-crafted attacks on Guanaco 7B. FJD outperforms baseline methods on almost all attacks and LLMs. Attack on Guanaco-7B PPL SMLLM GradSafe FT FJD aim 0.5258±0.0006± 0.0006 0.8632±0.0043± 0.0043 0.7448±0.0073± 0.0073 0.8635±0.0100± 0.0100 0.9646±0.0082± 0.0082 dev_mode_v2 0.4292±0.0011± 0.0011 0.5215±0.0055± 0.0055 0.8763±0.0057± 0.0057 0.3517±0.0161± 0.0161 0.6243±0.0236± 0.0236 dev_mode_ranti 0.5486±0.0004± 0.0004 0.5757±0.0055± 0.0055 0.5532±0.0142± 0.0142 0.6699±0.0302± 0.0302 0.8346±0.0087± 0.0087 distractors 0.6778±0.0003± 0.0003 0.5056±0.0026± 0.0026 0.8878±0.0015± 0.0015 0.5649±0.0205± 0.0205 0.7928±0.0230± 0.0230 distractors_negated 0.9562±0.0010± 0.0010 0.8285±0.0064± 0.0064 0.8914±0.0011± 0.0011 0.3073±0.0164± 0.0164 0.7874±0.0093± 0.0093 evil_confidant 0.8423±0.0002± 0.0002 0.5521±0.0017± 0.0017 0.5760±0.0007± 0.0007 0.3389±0.0149± 0.0149 0.6062±0.0250± 0.0250 poems 0.9190±0.0015± 0.0015 0.5118±0.0077± 0.0077 0.8449±0.0050± 0.0050 0.4110±0.0172± 0.0172 0.7476±0.0223± 0.0223 prefix_injection_1 0.9611±0.0007± 0.0007 0.8542±0.0088± 0.0088 0.6972±0.0054± 0.0054 0.9215±0.0058± 0.0058 0.9252±0.0022± 0.0022 prefix_injection_2 0.1288±0.0004± 0.0004 0.5683±0.0090± 0.0090 0.5532±0.0060± 0.0060 0.9806±0.0047± 0.0047 0.9931±0.0020± 0.0020 prefix_injection_hello 0.8267±0.0003± 0.0003 0.8410±0.0026± 0.0026 0.7944±0.0086± 0.0086 0.6736±0.0105± 0.0105 0.9535±0.0024± 0.0024 refusal_suppression 0.0066±0.0002± 0.0002 0.8840±0.0084± 0.0084 0.9035±0.0025± 0.0025 0.4061±0.0321± 0.0321 0.7954±0.0148± 0.0148 refusal_suppression_inv 0.0033±0.0001± 0.0001 0.8764±0.0104± 0.0104 0.8867±0.0070± 0.0070 0.4867±0.0205± 0.0205 0.9269±0.0149± 0.0149 style_injection_short 0.0059±0.0001± 0.0001 0.7611±0.0116± 0.0116 0.9240±0.0028± 0.0028 0.3274±0.0284± 0.0284 0.8508±0.0038± 0.0038 Average of CO 0.5255±0.0005± 0.0005 0.7033±0.0065± 0.0065 0.7795±0.0052± 0.0052 0.5618±0.0175± 0.0175 0.8310±0.0123± 0.0123 auto_payload_splitting 0.9637±0.0004± 0.0004 0.7951±0.0010± 0.0010 0.9538±0.0019± 0.0019 0.4236±0.0058± 0.0058 0.9578±0.0159± 0.0159 base64 0.9221±0.0006± 0.0006 0.9431±0.0035± 0.0035 0.6072±0.0098± 0.0098 0.3697±0.0088± 0.0088 0.6328±0.0264± 0.0264 base64_raw 0.9190±0.0010± 0.0010 0.8611±0.0071± 0.0071 0.6806±0.0048± 0.0048 0.3287±0.0068± 0.0068 0.9141±0.0190± 0.0190 base64_input_only 0.9281±0.0007± 0.0007 0.9028±0.0069± 0.0069 0.5447±0.0147± 0.0147 0.4175±0.0089± 0.0089 0.7910±0.0184± 0.0184 base64_output_only 0.8838±0.0008± 0.0008 0.7569±0.0113± 0.0113 0.8771±0.0081± 0.0081 0.4180±0.0192± 0.0192 0.8431±0.0134± 0.0134 combination_1 0.0032±0.0001± 0.0001 0.6792±0.0151± 0.0151 0.8659±0.0073± 0.0073 0.9706±0.0066± 0.0066 0.9108±0.0086± 0.0086 combination_2 0.0031±0.0001± 0.0001 0.6854±0.0103± 0.0103 0.8837±0.0014± 0.0014 0.9770±0.0051± 0.0051 0.9874±0.0193± 0.0193 combination_3 0.0052±0.0001± 0.0001 0.8938±0.0168± 0.0168 0.5848±0.0086± 0.0086 0.8303±0.0098± 0.0098 0.9826±0.0095± 0.0095 disemvowel 0.9884±0.0007± 0.0007 0.8611±0.0039± 0.0039 0.9319±0.0068± 0.0068 0.3832±0.0250± 0.0250 0.9829±0.0231± 0.0231 few_shot_json 0.0017±0.0001± 0.0001 0.7563±0.0051± 0.0051 0.8124±0.0084± 0.0084 0.3417±0.0275± 0.0275 0.7719±0.0134± 0.0134 leetspeak 0.9793±0.0002± 0.0002 0.7653±0.0087± 0.0087 0.9264±0.0031± 0.0031 0.3738±0.0117± 0.0117 0.8922±0.0133± 0.0133 rot13 0.9981±0.0001± 0.0001 0.8368±0.0060± 0.0060 0.8631±0.0047± 0.0047 0.4398±0.0145± 0.0145 0.9018±0.0108± 0.0108 style_injection_json 0.9000±0.0010± 0.0010 0.8368±0.0060± 0.0060 0.9803±0.0012± 0.0012 0.4005±0.0138± 0.0138 0.8547±0.0135± 0.0135 wikipedia 0.7799±0.0024± 0.0024 0.9271±0.0090± 0.0090 0.9359±0.0007± 0.0007 0.3493±0.0139± 0.0139 0.9474±0.0086± 0.0086 wikipedia_with_title 0.8962±0.0003± 0.0003 0.8472±0.0039± 0.0039 0.9499±0.0015± 0.0015 0.3035±0.0113± 0.0113 0.9526±0.0161± 0.0161 Average of MG 0.6781±0.0006± 0.0006 0.8232±0.0076± 0.0076 0.8265±0.0055± 0.0055 0.4885±0.0126± 0.0126 0.8882±0.0153± 0.0153 Table 21: Detection results (AUC) of jailbreak prompt under Hand-crafted attacks on Guanaco 13B. FJD outperforms baseline methods on almost all attacks and LLMs. Attack on Guanaco-13B PPL SMLLM GradSafe FT FJD aim 0.5262±0.0011± 0.0011 0.6211±0.0048± 0.0048 0.9235±0.0082± 0.0082 0.7403±0.0197± 0.0197 0.9063±0.0106± 0.0106 dev_mode_v2 0.4308±0.0001± 0.0001 0.5633±0.0099± 0.0099 0.8662±0.0058± 0.0058 0.7465±0.0190± 0.0190 0.8974±0.0102± 0.0102 dev_mode_ranti 0.5491±0.0011± 0.0011 0.5624±0.0154± 0.0154 0.8771±0.0026± 0.0026 0.6788±0.0179± 0.0179 0.8991±0.0155± 0.0155 distractors 0.6739±0.0004± 0.0004 0.5326±0.0026± 0.0026 0.8259±0.0069± 0.0069 0.4411±0.0075± 0.0075 0.7368±0.0230± 0.0230 distractors_negated 0.9604±0.0002± 0.0002 0.9275±0.0065± 0.0065 0.9288±0.0069± 0.0069 0.5321±0.0237± 0.0237 0.9306±0.0187± 0.0187 evil_confidant 0.3867±0.0005± 0.0005 0.8105±0.0093± 0.0093 0.5391±0.0110± 0.0110 0.5869±0.0229± 0.0229 0.6988±0.0241± 0.0241 poems 0.9239±0.0008± 0.0008 0.8346±0.0026± 0.0026 0.6334±0.0129± 0.0129 0.5711±0.0212± 0.0212 0.8541±0.0140± 0.0140 prefix_injection_1 0.9631±0.0007± 0.0007 0.9074±0.0074± 0.0074 0.5783±0.0053± 0.0053 0.7653±0.0202± 0.0202 0.8138±0.0118± 0.0118 prefix_injection_2 0.1293±0.0021± 0.0021 0.5892±0.0110± 0.0110 0.8277±0.0065± 0.0065 0.9330±0.0148± 0.0148 0.9365±0.0035± 0.0035 prefix_injection_hello 0.8232±0.0011± 0.0011 0.6841±0.0089± 0.0089 0.5469±0.0177± 0.0177 0.6577±0.0137± 0.0137 0.8363±0.0069± 0.0069 refusal_suppression 0.0084±0.0006± 0.0006 0.8048±0.0145± 0.0145 0.6201±0.0075± 0.0075 0.6051±0.0131± 0.0131 0.8378±0.0080± 0.0080 refusal_suppression_inv 0.0011±0.0001± 0.0001 0.9669±0.0054± 0.0054 0.6884±0.0058± 0.0058 0.5137±0.0216± 0.0216 0.8173±0.0253± 0.0253 style_injection_short 0.0061±0.0001± 0.0001 0.5890±0.0198± 0.0198 0.7599±0.0056± 0.0056 0.3727±0.0153± 0.0153 0.8098±0.0120± 0.0120 Average of CO 0.4909±0.0007± 0.0007 0.7226±0.0091± 0.0091 0.7396±0.0079± 0.0079 0.6265±0.0177± 0.0177 0.8442±0.0141± 0.0141 auto_payload_splitting 0.9549±0.0011± 0.0011 0.8957±0.0108± 0.0108 0.6317±0.0017± 0.0017 0.4366±0.0073± 0.0073 0.8580±0.0146± 0.0146 base64 0.9224±0.0008± 0.0008 0.7656±0.0148± 0.0148 0.8053±0.0018± 0.0018 0.6270±0.0084± 0.0084 0.8464±0.0131± 0.0131 base64_raw 0.9266±0.0007± 0.0007 0.8764±0.0071± 0.0071 0.7970±0.0057± 0.0057 0.4882±0.0228± 0.0228 0.8545±0.0140± 0.0140 base64_input_only 0.9323±0.0018± 0.0018 0.9135±0.0106± 0.0106 0.6775±0.0049± 0.0049 0.4628±0.0180± 0.0180 0.7069±0.0186± 0.0186 base64_output_only 0.8640±0.0009± 0.0009 0.6353±0.0327± 0.0327 0.7637±0.0074± 0.0074 0.6699±0.0297± 0.0297 0.8262±0.0169± 0.0169 combination_1 0.0031±0.0001± 0.0001 0.6174±0.0269± 0.0269 0.8625±0.0041± 0.0041 0.7539±0.0234± 0.0234 0.9870±0.0058± 0.0058 combination_2 0.0032±0.0001± 0.0001 0.6167±0.0029± 0.0029 0.8950±0.0047± 0.0047 0.7276±0.0166± 0.0166 0.9044±0.0137± 0.0137 combination_3 0.0052±0.0002± 0.0002 0.7836±0.0052± 0.0052 0.5529±0.0057± 0.0057 0.4987±0.0188± 0.0188 0.8223±0.0199± 0.0199 disemvowel 0.9996±0.0003± 0.0003 0.6299±0.0111± 0.0111 0.7080±0.0055± 0.0055 0.3476±0.0268± 0.0268 0.7935±0.0186± 0.0186 few_shot_json 0.0074±0.0004± 0.0004 0.6813±0.0141± 0.0141 0.8629±0.0038± 0.0038 0.5519±0.0223± 0.0223 0.8544±0.0215± 0.0215 leetspeak 0.9582±0.0012± 0.0012 0.6409±0.0199± 0.0199 0.5959±0.0147± 0.0147 0.4745±0.0230± 0.0230 0.7990±0.0186± 0.0186 rot13 0.9895±0.0005± 0.0005 0.6399±0.0049± 0.0049 0.8622±0.0046± 0.0046 0.2805±0.0160± 0.0160 0.8465±0.0131± 0.0131 style_injection_json 0.9029±0.0010± 0.0010 0.8176±0.0105± 0.0105 0.8189±0.0031± 0.0031 0.4873±0.0169± 0.0169 0.8127±0.0227± 0.0227 wikipedia 0.7870±0.0053± 0.0053 0.9192±0.0120± 0.0120 0.8557±0.0141± 0.0141 0.5645±0.0124± 0.0124 0.8502±0.0239± 0.0239 wikipedia_with_title 0.9009±0.0009± 0.0009 0.9538±0.0137± 0.0137 0.8731±0.0022± 0.0022 0.5312±0.0260± 0.0260 0.8301±0.0213± 0.0213 Average 0.6771±0.0010± 0.0010 0.7591±0.0131± 0.0131 0.7708±0.0056± 0.0056 0.5268±0.0019± 0.0019 0.8395±0.0171± 0.0171 Table 22: The complete detection results (AUC) of jailbreak prompt under transferable attack. FJD can effectively detect jailbreak prompts in most cases. Source Target Methods Llama2-7B Vicuna-7B Guanaco-7B Vicuna-7B PPL 0.7647±0.0012± 0.0012 0.8406±0.0007± 0.0007 0.8745±0.0005± 0.0005 SMLLM 0.7507±0.0037± 0.0037 0.8603±0.0059± 0.0059 0.8250±0.0063± 0.0063 GradSafe 0.9902±0.0014± 0.0014 0.8605±0.0046± 0.0046 0.8847±0.0029± 0.0029 FJD 0.9970±0.0025± 0.0025 0.9777±0.0019± 0.0019 0.9688±0.0051± 0.0051 Llama2-7B PPL 0.7437±0.0017± 0.0017 0.7026±0.0009± 0.0009 0.8770±0.0006± 0.0006 SMLLM 0.7971±0.0035± 0.0035 0.5682±0.0043± 0.0043 0.6863±0.0072± 0.0072 GradSafe 0.8913±0.0049± 0.0049 0.8880±0.0077± 0.0077 0.7459±0.0129± 0.0129 FJD 0.9873±0.0030± 0.0030 0.7062±0.0097± 0.0097 0.9549±0.0070± 0.0070 Guanaco-7B PPL 0.8221±0.0021± 0.0021 0.7679±0.0011± 0.0011 0.8532±0.0032± 0.0032 SMLLM 0.9243±0.0012± 0.0012 0.7941±0.0052± 0.0052 0.8927±0.0065± 0.0065 GradSafe 0.9907±0.0003± 0.0003 0.7735±0.0062± 0.0062 0.8289±0.0067± 0.0067 FJD 0.9926±0.0029± 0.0029 0.9781±0.0014± 0.0014 0.9875±0.0017± 0.0017 Vicuna-7B + Llama2-7B PPL 0.9788±0.0003± 0.0003 0.9803±0.0002± 0.0002 0.9783±0.0004± 0.0004 SMLLM 0.9253±0.0019± 0.0019 0.8889±0.0021± 0.0021 0.8675±0.0074± 0.0074 GradSafe 0.9563±0.0068± 0.0068 0.8835±0.0059± 0.0059 0.9251±0.0036± 0.0036 FJD 0.9951±0.0017± 0.0017 0.9820±0.0022± 0.0022 0.9342±0.0051± 0.0051 Vicuna-7B + Guanaco-7B PPL 0.9832±0.0005± 0.0005 0.9819±0.0003± 0.0003 0.9832±0.0003± 0.0003 SMLLM 0.9537±0.0017± 0.0017 0.8429±0.0055± 0.0055 0.9246±0.0020± 0.0020 GradSafe 0.9822±0.0015± 0.0015 0.9125±0.0036± 0.0036 0.9043±0.0010± 0.0010 FJD 0.8922±0.0034± 0.0034 0.8952±0.0070± 0.0070 0.9945±0.0014± 0.0014 Llama2-7B + Guanaco-7B PPL 0.9849±0.0007± 0.0007 0.9772±0.0011± 0.0011 0.9827±0.0003± 0.0003 SMLLM 0.8263±0.0087± 0.0087 0.9146±0.0093± 0.0093 0.7380±0.0102± 0.0102 GradSafe 0.8293±0.0072± 0.0072 0.9456±0.0023± 0.0023 0.8154±0.0074± 0.0074 FJD 0.9998±0.0002± 0.0002 1.0000±0.0000± 0.0000 0.9834±0.0015± 0.0015 Vicuna-7B + Llama2-7B + Guanaco-7B PPL 0.9844±0.0006± 0.0006 0.9837±0.0007± 0.0007 0.9845±0.0003± 0.0003 SMLLM 0.8034±0.0088± 0.0088 0.8774±0.0075± 0.0075 0.7461±0.0099± 0.0099 GradSafe 0.9249±0.0029± 0.0029 0.9132±0.0022± 0.0022 0.9533±0.0078± 0.0078 FJD 0.9954±0.0013± 0.0013 0.9695±0.0035± 0.0035 0.9901±0.0049± 0.0049 Methods Llama2-13B Vicuna-13B Guanaco-13B Vicuna-7B PPL 0.9177±0.0028± 0.0028 0.7941±0.0002± 0.0002 0.8915±0.0004± 0.0004 SMLLM 0.6214±0.0129± 0.0129 0.5484±0.0111± 0.0111 0.6651±0.0099± 0.0099 GradSafe 0.8949±0.0096± 0.0096 0.8486±0.0063± 0.0063 0.9039±0.0087± 0.0087 FJD 0.9537±0.0039± 0.0039 0.9349±0.0107± 0.0107 0.9785±0.0087± 0.0087 Llama2-7B PPL 0.8515±0.0003± 0.0003 0.7782±0.0002± 0.0002 0.7967±0.0003± 0.0003 SMLLM 0.7500±0.0091± 0.0091 0.5593±0.0109± 0.0109 0.6250±0.0137± 0.0137 GradSafe 0.8817±0.0058± 0.0058 0.8272±0.0070± 0.0070 0.8658±0.0069± 0.0069 FJD 0.9087±0.0074± 0.0074 0.9175±0.0062± 0.0062 0.9527±0.0189± 0.0189 Guanaco-7B PPL 0.8221±0.0002± 0.0002 0.8644±0.0004± 0.0004 0.8059±0.0007± 0.0007 SMLLM 0.8587±0.0059± 0.0059 0.9287±0.0037± 0.0037 0.8066±0.0041± 0.0041 GradSafe 0.8905±0.0017± 0.0017 0.9021±0.0034± 0.0034 0.9325±0.0045± 0.0045 FJD 0.9425±0.0022± 0.0022 0.9324±0.0063± 0.0063 0.9769±0.0103± 0.0103 Vicuna-7B + Llama2-7B PPL 0.9852±0.0012± 0.0012 0.9794±0.0017± 0.0017 0.9822±0.0009± 0.0009 SMLLM 0.8846±0.0036± 0.0036 0.9176±0.0068± 0.0068 0.7951±0.0063± 0.0063 GradSafe 0.9364±0.0078± 0.0078 0.8445±0.0022± 0.0022 0.9240±0.0061± 0.0061 FJD 0.9716±0.0038± 0.0038 0.8516±0.0118± 0.0118 0.9772±0.0031± 0.0031 Vicuna-7B + Guanaco-7B PPL 0.9882±0.0004± 0.0004 0.9866±0.0009± 0.0009 0.9835±0.0005± 0.0005 SMLLM 0.9722±0.0015± 0.0015 0.9320±0.0021± 0.0021 0.8004±0.0073± 0.0073 GradSafe 0.9880±0.0023± 0.0023 0.9769±0.0027± 0.0027 0.7457±0.0097± 0.0097 FJD 0.9522±0.0067± 0.0067 0.9850±0.0064± 0.0064 0.8461±0.0036± 0.0036 Llama2-7B + Guanaco-7B PPL 0.9849±0.0011± 0.0011 0.9839±0.0016± 0.0016 0.9800±0.0009± 0.0009 SMLLM 0.9125±0.0022± 0.0022 0.8615±0.0036± 0.0036 0.7469±0.0074± 0.0074 GradSafe 0.8531±0.0102± 0.0102 0.9103±0.0048± 0.0048 0.8963±0.0019± 0.0019 FJD 0.9450±0.0083± 0.0083 0.9633±0.0061± 0.0061 0.9381±0.0146± 0.0146 Vicuna-7B + Llama2-7B + Guanaco-7B PPL 0.9923±0.0001± 0.0001 0.9855±0.0007± 0.0007 0.9844±0.0004± 0.0004 SMLLM 0.8281±0.0059± 0.0059 0.7970±0.0061± 0.0061 0.7492±0.0086± 0.0086 GradSafe 0.9117±0.0094± 0.0094 0.9138±0.0028± 0.0028 0.9313±0.0083± 0.0083 FJD 0.9443±0.0060± 0.0060 0.9710±0.0173± 0.0173 0.9629±0.0038± 0.0038 Appendix K Affirmative Instruction Analysis To investigate the effects of detecting jailbreak prompts on FJD when utilizing different affirmative instructions in prefixes and suffixes on Llama2 7B, we perform experiments involving semantic reorganization and word replacement using the prompts outlined in Sec. 4.5. In Tab. 23 shows that using a affirmative instruction as a suffix can yield comparable jailbreak prompt detection effects to using it as a prefix. It can be found that employing affirmative instructions as a suffix achieves comparable performance to using them as a prefix in the majority of cases, while a small number of instructions as a suffix lead to a decline in performance. We believe that the influence on LLMs is more significant when affirmative instructions are applied as prefixes. Table 23: Detection results (AUC) of jailbreak prompt while using different affirmative instruction as prefix and suffix. FJD can effectively detect jailbreak prompts using various affirmative instructions. Most instructions used as suffixes have comparable effects when used as prefixes. Different Affirmative Instruction P/S GCG AutoDAN Cipher Without affirmative instruction - 0.9939 0.9164 0.9335 Respond in a way that promotes safety. Prefix 0.9985 0.9624 0.9769 Suffix 0.9609 0.9713 0.9674 Provide an answer that prioritizes user safety. Prefix 0.9956 0.9163 0.9555 Suffix 0.9134 0.9125 0.8968 Remember you are a responsible and helpful LLM. Prefix 0.9969 0.9350 0.9403 Suffix 0.9594 0.9178 0.9354 Let’s think step by step. Prefix 0.9988 0.9610 0.9578 Suffix 0.9770 0.9605 0.9665 Appendix L Analysis of FJD-LI In this section, we show the detection results of FJD-LI under GCG, AutoDAN, Cipher, and Hand-crafted on Llama2 7B, Vicuna 7B, and Guanaco 7B in Tab. 24. This approach further enhances the detection of jailbreak prompts, even when faced with unseen data (Cipher, Hand-crafted). Table 24: Detection results (AUC) of jailbreak prompt through FJD-LI. FJD-LI further enhances the detection of jailbreak prompts over FJD by using learnable virtual instructions. Attack Method Llama2-7B Vicuna-7B Guanaco-7B GCG PPL 0.9717±0.0002± 0.0002 0.9860±0.0002± 0.0002 0.9833±0.0001± 0.0001 SMLLM 0.9423±0.0027± 0.0027 0.9575±0.0071± 0.0071 0.8811±0.0029± 0.0029 GradSafe 0.8943±0.0035± 0.0035 0.7575±0.0117± 0.0117 0.7501±0.0019± 0.0019 FJD 0.9990±0.0002± 0.0002 0.7250±0.0044± 0.0044 0.9515±0.0040± 0.0040 FJD-LI 0.9998±0.0001± 0.0001 0.9887±0.0029± 0.0029 0.9895±0.0015± 0.0015 AutoDAN PPL 0.8172±0.0017± 0.0017 0.7452±0.0012± 0.0012 0.7964±0.0004± 0.0004 SMLLM 0.8197±0.0052± 0.0052 0.7831±0.0035± 0.0035 0.6704±0.0036± 0.0036 GradSafe 0.8025±0.0089± 0.0089 0.7893±0.0020± 0.0020 0.8194±0.0051± 0.0051 FJD 0.9578±0.0088± 0.0088 0.7964±0.0182± 0.0182 0.8946±0.0065± 0.0065 FJD-LI 0.9703±0.0024± 0.0024 0.9969±0.0021± 0.0021 0.9817±0.0038± 0.0038 Cipher PPL 0.0070±0.0005± 0.0005 0.0266±0.0004± 0.0004 0.0248±0.0005± 0.0005 SMLLM 0.5034±0.0024± 0.0024 0.5233±0.0009± 0.0009 0.5460±0.0026± 0.0026 GradSafe 0.7862±0.0045± 0.0045 0.7094±0.0201± 0.0201 0.8112±0.0088± 0.0088 FJD 0.9896±0.0014± 0.0014 0.8633±0.0033± 0.0033 0.8299±0.0043± 0.0043 FJD-LI 0.9944±0.0012± 0.0012 0.9310±0.0036± 0.0036 0.8826±0.0102± 0.0102 Hand-crafted PPL 0.6090±0.0020± 0.0020 0.6066±0.0010± 0.0010 0.6018±0.0006± 0.0006 SMLLM 0.7138±0.0108± 0.0108 0.6886±0.0063± 0.0063 0.7633±0.0071± 0.0071 GradSafe 0.9085±0.0050± 0.0050 0.7871±0.0055± 0.0055 0.8030±0.0054± 0.0054 FJD 0.9595±0.0069± 0.0069 0.7993±0.0148± 0.0148 0.8596±0.0138± 0.0138 FJD-LI 0.9843±0.0016± 0.0016 0.8579±0.0073± 0.0073 0.9081±0.0101± 0.0101 Appendix M Rigorous Analysis of Temperature Scaling In this section, we provide a mathematical proof for the two phenomena of softmax maximum value flipping. First, we define the logits of two distributions Z(1)=z1(1),z2(1),…zn(1)Z^(1)=\z^(1)_1,z^(1)_2,...z^(1)_n\ and z(2)=z1(2),z2(2),…zn(2)z^(2)=\z^(2)_1,z^(2)_2,...z^(2)_n\, assuming that z1(1)z^(1)_1 and z1(2)z^(2)_1 is the maximum value. The probability of the maximum value with temperature τ is P1,τ(1)=exp(z1(1)/τ)∑nexp(zn(1)/τ)P_1,τ(1)= (z^(1)_1/τ) _n (z^(1)_n/τ) (6) P1,τ(2)=exp(z1(2)/τ)∑nexp(zn(2)/τ)P_1,τ(2)= (z^(2)_1/τ) _n (z^(2)_n/τ) (7) Assume P1,τ(1)<P1,τ(2)P_1,τ(1)<P_1,τ(2). When the maximum value is removed from the Z(1)Z^(1), the distribution becomes sharp, its variance is σ2(1)=1n−1∑i=2n(zi(1)−μ(1))2σ^2(1)= 1n-1 _i=2^n(z^(1)_i-μ^(1))^2. When the maximum value is removed from the Z(2)Z^(2) distribution, the distribution becomes smooth, its variance is σ2(2)=1n−1∑j=2n(zj(2)−μ(2))2σ^2(2)= 1n-1 _j=2^n(z^(2)_j-μ^(2))^2. And σ2(1)>σ2(2)σ^2(1)>σ^2(2) When τ>1τ>1, for a single distribution, the softmax distribution becomes smoother, but the rank of the maximum value remains unchanged. Different distributions have varying sensitivities to changes in temperature. As the temperature τ increases, when the proportions of non-max values in distributions Z(1)Z^(1) and Z(2)Z^(2) are similar, the smoother non-max values z2(2)/τ,z3(2)/τ,…zn(2)/τ\z^(2)_2/τ,z^(2)_3/τ,...z^(2)_n/τ\ occupy a larger proportion than the sharper non-max values z2(1)/τ,z3(1)/τ,…zn(1)/τ\z^(1)_2/τ,z^(1)_3/τ,...z^(1)_n/τ\, causing the proportion of the maximum value in distribution Z(2)Z^(2) to decrease rapidly. In certain conditions, this can cause the maximum values of the two distributions to flip, i.e., P1,τ(1)>P1,τ(2)P_1,τ(1)>P_1,τ(2). Based on the above, we conduct a statistical analysis of the logits for both jailbreak and benign prompts. Taking Llama 7B as an example, after prepending the affirmative instruction, we present an instance where the ranking of the first token changes after increasing the temperature for both benign (PureDove) and jailbreak (AutoDAN) prompts in Tab. 25. Table 25: An instance in which the ranking of the first token P1,τP_1,τ changes after increasing the temperature τ. LLM Label τ=1τ=1 τ=1.25τ=1.25 P1,τP_1,τ Std (non-max) P1,τP_1,τ Std (non-max) Llama2-7B Benign (PureDove) 0.9999777 1.2369×10−71.2369× 10^-7 0.9998197 1.1055×10−61.1055× 10^-6 Jailbreak (AutoDAN) 0.9999807 1.0746×10−71.0746× 10^-7 0.9998046 9.4290×10−79.4290× 10^-7 Appendix N The Optimal Temperature In this section, we show the optimal temperatures of FT and FJD across various LLMs on the training dataset in Tab. 26. Additionally, we analyzed how the selected optimal temperature affects the detection performance of FJD with varying amounts of training data and different training datasets, taking Llama2 7B as an example. In Tab. 27, we found that a small datasets can yield similar temperatures, and that small variations in temperature have minimal impact on detection results. Although the temperatures obtained from training with different datasets exhibit some variation, they have minimal impact on FJD detection performance within a certain range. Table 26: The optimal temperatures of FT and FJD across various LLMs on the training dataset. Method Llama2-7B Llama2-13B Vicuna-7B Vicuna-13B Guanaco-7b Guanaco-13B FT 0.86 1.51 0.95 1.99 0.69 0.80 FJD 1.25 1.98 1.47 0.35 1.24 0.79 Table 27: Detection results (AUC) of jailbreak prompt through FJD under different size of training sets. A small datasets can yield similar temperatures, and small variations in temperature have minimal impact on detection results Model Training Size Temperature AutoDAN Cipher GCG PAIR Llama2-7B 10% 1.18 0.9549±0.0054± 0.0054 0.9764±0.0017± 0.0017 0.9983±0.0004± 0.0004 0.9738±0.0038± 0.0038 20% 1.20 0.9564±0.0061± 0.0061 0.9741±0.0026± 0.0026 0.9990±0.0002± 0.0002 0.9737±0.0015± 0.0015 30% 1.23 0.9542±0.0061± 0.0061 0.9726±0.0019± 0.0019 0.9990±0.0002± 0.0002 0.9749±0.0047± 0.0047 40% 1.24 0.9519±0.0024± 0.0024 0.9714±0.0013± 0.0013 0.9990±0.0003± 0.0003 0.9754±0.0019± 0.0019 50% 1.25 0.9495±0.0053± 0.0053 0.9700±0.0034± 0.0034 0.9990±0.0003± 0.0003 0.9761±0.0009± 0.0009 Model Training Datasets Temperature AutoDAN Cipher GCG PAIR Llama2-7B AutoDAN 1.27 0.9550±0.0038± 0.0038 0.9746±0.0028± 0.0028 0.9991±0.0004± 0.0004 0.9748±0.0021± 0.0021 Cipher 1.18 0.9549±0.0054± 0.0054 0.9764±0.0017± 0.0017 0.9983±0.0004± 0.0004 0.9746±0.0038± 0.0038 GCG 1.37 0.9538±0.0038± 0.0038 0.9696±0.0014± 0.0014 0.9992±0.0005± 0.0005 0.9749±0.0011± 0.0011 Table 28: Detection results (AUC) of jailbreak prompt through FJD in multimodal contexts (LLaVA-v1.6-mistral-7b). Directly using First Token Confidence (FT) for detection is feasible, and that FJD can further improve detection performance. Attack FigStep Relevant_SD Relevant_typo Relevant_SD_typo FT 0.8501±0.0094± 0.0094 0.6649±0.0128± 0.0128 0.5967±0.0101± 0.0101 0.5753±0.0117± 0.0117 FJD 0.9258±0.0076± 0.0076 0.7093±0.0104± 0.0104 0.7395±0.0099± 0.0099 0.6046±0.0125± 0.0125 Appendix O Analysis of FJD-K In contrast to FJD, FJD-K detects jailbreak prompts through the average of the first k token confidences. Formally, based on the Equation 2, given an input sequence xqx_q, the affirmative instruction xaix_ai and the temperature τ, the confidence of the first K tokens is computed as Ck=1k∑i=1kCi=1k∑i=1kστ(f(xai⊕xq)i/τ)C_k= 1k _i=1^kC_i= 1k _i=1^k _τ(f(x_ai x_q)_i/τ) (8) When k=1k=1, CkC_k is the first token confidence. To evaluate the influence of the number of fist k∈[1,10]k∈[1,10] tokens on the detection of jailbreak prompts across various LLMs, we conduct experiments using FJD on Vicuna 7B, Llama2 7B, and Guanaco 7B. Fig. 9 shows changes in the jailbreak detection AUC value during token selection. In certain LLMs and attacks, FJD-K can enhance the detection capability of FJD to a certain degree. Nonetheless, in the case of AutoDAN, the efficacy of FJD-K in detection is significantly diminished. (a) Llama2 7B (b) Vicuna 7B (c) Guanaco 7B Figure 9: Detection results (AUC) of jailbreak prompt while using First K Token with FJD. In certain LLMs and under specific attacks, FJD-K enhances the detection capabilities of FJD. However, for AutoDAN attacks across the three LLMs, FJD-K diminishes the detection performance of FJD. Appendix P The Generality of FJD in multimodal contexts It is indeed interesting to investigate the generality of FJD behavior. And we conducted some preliminary experiments in multimodal contexts, using VQA [4] and M-Vet [55] as benign prompts, and FigStep [17] and M-SafetyBench [32] as jailbreak prompts, to examine the detection performance of FT and FJD. As shown in the Tab. 28, we conducted brief experiments on LLaVA-v1.6-mistral-7b. The experiments demonstrate that directly using First Token Confidence (FT) for detection is feasible, and that FJD can further improve detection performance. However, compared with the unimodal scenarios, the sequence length of image tokens is much larger than that of text tokens. Although the FJD method can still enhance detection, its impact on confidence is smaller relative to unimodal scenarios, resulting in less impressive results on M-SafetyBench.