Paper deep dive
The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defense
Yangyang Guo, Fangkai Jiao, Liqiang Nie, Mohan Kankanhalli
Models: InstructBLIP, Llama3.1, LLaVA, MiniGPT-4, Mistral, Qwen2.5
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/12/2026, 7:47:15 PM
Summary
The paper investigates the 'VLLM Safety Paradox,' where Vision Large Language Models (VLLMs) are simultaneously prone to jailbreak attacks and easily defended by current mechanisms. The authors identify that the inclusion of visual inputs compromises the safety alignment of backbone LLMs, while existing defense methods suffer from 'over-prudence,' leading to high abstention rates on benign inputs. They propose 'LLM-Pipeline,' a vision-free detection method, and highlight that current evaluation metrics for jailbreak attacks often show only chance agreement.
Entities (5)
Relation Signals (4)
VLLM â vulnerableto â Jailbreak Attack
confidence 95% ¡ The vulnerability of Vision Large Language Models (VLLMs) to jailbreak attacks appears as no surprise.
Inclusion of vision inputs â compromises â Safety alignment
confidence 90% ¡ However, this alignment is significantly compromised with the inclusion of image inputs.
Defense mechanisms â exhibits â Over-prudence
confidence 90% ¡ The recognition of a largely ignored problem in existing defense mechanisms--over-prudence.
LLM-Pipeline â mitigates â Jailbreak Attack
confidence 85% ¡ Our method repurposes the more advanced guardrails of LLMs on the shelf, serving as an effective alternative detector prior to VLLM response.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The vulnerability of Vision Large Language Models (VLLMs) to jailbreak attacks appears as no surprise. However, recent defense mechanisms against these attacks have reached near-saturation performance on benchmark evaluations, often with minimal effort. This \emph{dual high performance} in both attack and defense raises a fundamental and perplexing paradox. To gain a deep understanding of this issue and thus further help strengthen the trustworthiness of VLLMs, this paper makes three key contributions: i) One tentative explanation for VLLMs being prone to jailbreak attacks--\textbf{inclusion of vision inputs}, as well as its in-depth analysis. ii) The recognition of a largely ignored problem in existing defense mechanisms--\textbf{over-prudence}. The problem causes these defense methods to exhibit unintended abstention, even in the presence of benign inputs, thereby undermining their reliability in faithfully defending against attacks. iii) A simple safety-aware method--\textbf{LLM-Pipeline}. Our method repurposes the more advanced guardrails of LLMs on the shelf, serving as an effective alternative detector prior to VLLM response. Last but not least, we find that the two representative evaluation methods for jailbreak often exhibit chance agreement. This limitation makes it potentially misleading when evaluating attack strategies or defense mechanisms. We believe the findings from this paper offer useful insights to rethink the foundational development of VLLM safety with respect to benchmark datasets, defense strategies, and evaluation methods.
Tags
Links
- Source: https://arxiv.org/abs/2411.08410
- Canonical: https://arxiv.org/abs/2411.08410
Trouble viewing inline? Open PDF directly â
Full Text
59,668 characters extracted from source content.
Expand or collapse full text
The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defense Yangyang Guo 1 , Fangkai Jiao 2,3 , Liqiang Nie 4 , Mohan Kankanhalli 1 1 National University of Singapore 2 Nanyang Technological University 3 I 2 R, A*STAR 4 Harbin Institute of Technology (Shenzhen) guoyang.eric,nieliqiang@gmail.com, jiaofangkai@hotmail.com, mohan@comp.nus.edu.sg Abstract The vulnerability of Vision Large Language Models (VLLMs) to jailbreak attacks appears as no surprise. How- ever, recent defense mechanisms against these attacks have reached near-saturation performance on benchmark eval- uations, often with minimal effort. Thisdual high perfor- mancein both attack and defense raises a fundamental and perplexing paradox. To gain a deep understanding of this issue and thus further help strengthen the trustworthiness of VLLMs, this paper makes three key contributions: i) One tentative explanation for VLLMs being prone to jailbreak attacksâinclusion of vision inputs, as well as its in-depth analysis. i) The recognition of a largely ignored problem in existing defense mechanismsâover-prudence. The problem causes these defense methods to exhibit unintended absten- tion, even in the presence of benign inputs, thereby under- mining their reliability in faithfully defending against at- tacks. i) A simple safety-aware methodâLLM-Pipeline. Our method repurposes the more advanced guardrails of LLMs on the shelf, serving as an effective alternative de- tector prior to VLLM response. Last but not least, we find that the two representative evaluation methods for jailbreak often exhibit chance agreement. This limitation makes it potentially misleading when evaluating attack strategies or defense mechanisms. We believe the findings from this pa- per offer useful insights to rethink the foundational devel- opment of VLLM safety with respect to benchmark datasets, defense strategies, and evaluation methods. Disclaimer:This paper discusses violent and discrimi- natory content, which may be disturbing to some readers. 1. Introduction The pervasiveness of Large Language Models (LLMs) con- currently ushers varied challenges for both researchers and practitioners [28]. Among these, protecting the trustwor- thiness of free-form outputs, as defined by the 3Hcrite- rion [55], has grown increasingly critical in recent years [24, 59]. Beyond important considerations ofHelpfulness and Honesty, the need forHarmlessness is far more urgent given its potential social impact. Jailbreak attacks, also nowadays referred to as red- teaming [45], serve as the most common method for assessing the harmlessness of LLMs and Vision-LLMs (VLLMs) [8, 34, 57]. They are designed to circumvent the built-in restrictions or safeguards within models [25], elic- iting them to produce malicious outputs, such as content related to illegal activities, hate speech, and pornography. Compared to their LLM counterparts, the vulnerability of VLLMs to jailbreak attacks has garnered considerable at- tention until very recently [32, 65]. Some initial methods inject high-risk content into images through typography or generative techniques like stable diffusion [49]. Leverag- ing such methods, datasets have been curated that easily ex- pose a high Attack Success Rate (ASR) for both proprietary models [1] and publicly open-sourced models [33, 34]. On the other hand, without many bells and whistles, re- cent defense strategiesâprimarily focused on safety-aware supervised fine-tuning [75] and system prompt protec- tion [64]âhave shown surprisingly remarkable defense re- sults on these benchmark datasets. In particular, VLLMs like LLaVA1.5 [33] and MiniGPT-v2 [8] can be fully safe- guarded against the attacks involved (ASRâ0) [20, 64, 75]. This dual-ease finding raises an intriguing question: does it suggest that defending against jailbreak attacks is easy, given that the attacks themselves have already been known to be relatively painless? The observation above highlights a perplexing safety paradox. To shed light on it, we present the first compre- hensive study understanding this safety paradox in VLLMs. i) Our first finding challenges prior assumptions that the vulnerability to jailbreak attacks stems from catastrophic forgetting or fine-tuning [46, 75]. Instead, we reveal that the true cause lies inthe inclusion of image inputs, which compromises the guardrails of the backbone LLMs. i) On 1 arXiv:2411.08410v2 [cs.CR] 6 Mar 2025 the other hand, we observe that existing defense mecha- nisms [64, 75] tend to be overly prudent. One typical man- ifestation is that VLLMs with post-defense, are prone to abstaining from responding even to benign queries. This issue ofover-prudencesignificantly impairs the helpful- ness of VLLMs. Even more concerning, we demonstrate that a simple, deliberate abstention approachâsuch as post- fixing a promptPlease respond âIâm sorryâ after answer- ing questionsto each queryâalready yields favorable results for models with advanced instruction-following capabilities (i.e., InternVL-2 [11]). Besides, our experiments point out that the two well-studied evaluation methods often show a sparse correlation in detecting jailbreaks. Specifically, some attacks that are successfully identified by rule-based evalu- ations can often escape from LLM-based evaluations. This discrepancy weakens the accuracy of evaluating an attack method or a defense strategy. Beyond understanding the safety paradox, we note that the jailbreak defense can be re-framed into adetection-then- responseprocess. i) As such, we propose to implement a detector prior to the final VLLM response and design a simple plug-and-playLLM-Pipelineapproach. We opt not to utilize an additional VLLM for detection as ECSO [20], given the limited reliability of current VLLMs in providing robust safeguards. Instead, we explore a vision-free detec- tor, where we repurpose the guardrails of recent advanced LLMs (e.g., Llama3.1 [57]) to judge the harmfulness of a given textual query, optionally with the image caption. In- terestingly, we find that this detector, when paired with a VLLM for safe response generation, suffers less from the over-prudence problem, achieving a balanced interplay be- tween robust safety alignment and model helpfulness. To the best of our knowledge, we are the first to inves- tigate the safety paradox problem of VLLMs. Through the empirical findings presented in this work, we seek to high- light this issue and raise more attention to its significance. Beyond this, we hope to provide insights that can support future advancements in this field, such as reaching a con- sensus on the nature of attacks and their associated risks, facilitating the collection of comprehensive attack data, and developing more robust defense and evaluations. 2. Preliminary We limit the inputs to a VLLMMto one textual instruc- tion and one image, in line with the existing jailbreak attack datasets [10, 19, 36, 75]: M[Instruction,Image]âR,(1) whereRcan either be an abstention response, such asI cannot answer this question..., or a typical response that follows the harmful instructions. Fig. 1 illustrates the harm- fulness resulting from the combined composition of instruc- Preliminary - Inputs 2 Safety Composition of Instruction and Image ďąBenign inputs â helpfulness of models ďąHarmful inputs â level of harmfulness, I < IV < I Benign; Response (â) Harmful; Response (Ă) Harmful; Response (Ă) Safe Te x tual Instruction Visual Image Harmful; Response (Ă) Unsafe Unsafe Safe (IV) (I) (I) (I) Figure 1. Safety attributes of textual Instruction and visual Image compositions in VLLM inputs.Level of harmfulness ranked across three quadrants: I<IV<I. tions and images. For safety reasons, responses to compo- sitions from quadrants I, I, and IV should be rejected. Jailbreak datasets.We primarily conduct experiments on four available mainstream jailbreak attack datasets, as de- tailed in Table 1. The instructions in these datasets are mostly auto-generated by LLMs, such as GPT-4 [42]. The images, on the other hand, can be benign ones sourced from MSCOCO [31] or generated using stable diffusion [49] or typographic methods, leading to the quadrant classifications defined in Fig. 1. Evaluation methods.There are two key methods for eval- uating the harmfulness of model outputs: rule-based and LLM-based evaluations [66]. Rule-based methods assess the effectiveness of an attack by searching for specific key- words in the VLLMsâ responses [64, 75]. This approach hinges on the fact that rejection responses typically include phrases like âIâm sorryâ, or âI cannot answerâ. LLM-based methods, on the other hand, utilize a state-of-the-art LLM as the evaluator to determine the success of an attack [25]. In this approach, the prompt and the response generated by a jailbreak attack are input into the evaluator, which then provides either a binary judgment or a fine-grained score to represent the degree of harmfulness. Evaluation metric.Following existing studies [20, 64, 69, 75, 76] in both LLM and VLLM jailbreaks, we utilize the Attack Success Rate (ASR) to quantify the effectiveness of jailbreak attacks. A higher ASR indicates a greater risk of a successful jailbreak, signifying a more vulnerable model. Table 1. Statistics of the four evaluated jailbreak attack datasets. #HS: number of harmful scenarios, such asillegal activityandhate speech; Quadrants correspond to those defined in Fig. 1. Dataset#Data#HSImage SourceQuadrants VLSafe [10]3,000-MSCOCO [31]IV FigStep [19]50010TypographyIII M-SafeB [36]5,04013Typography, SD [49]I,I VLGuard [75]1,5584Typography, RealI,I,IV 2 Table 2. ASR of six VLLMs across four different jailbreak attack datasets. All models demonstrate a high risk of generating harmful responses on these benchmarks,i.e., a high ASR. ModelRelease Date VLGuard [75] VLSafe [10]FigStep [19] M-SafetyBench [36] OverallSafe-UnsafeUnsafeOverallSDTYPOSD+TYPO LLaVA-1.5-Vicuna-7BDec-202388.6087.4690.0558.2865.686.8786.6187.0886.91 LLaVA-1.5-Vicuna-13B Dec-202381.7077.4287.1058.4753.283.2987.2084.1778.51 LLaVA-NeXT-Mistral-7BJan-202475.0078.1471.0415.4150.266.4179.4157.6262.21 LLaVA-NeXT-Llama3-8BMay-202479.6086.0271.4946.9448.462.5276.4353.8157.32 InternVL2-8B Jul-202474.6076.8871.7225.4145.860.2068.8153.0458.75 QWen2-VL-7BAug-202469.8074.3764.0349.4632.268.6181.0760.3664.40 Caption: The image depicts a person wearing a bomb suit and walking through a construction site. The image is a meme that suggests that the person is being invited to a potentially dangerous situation , possibly involving a bomb. Caption: This..., suggesting that if Irish people were the first slaves brought to America, they should have received reparations. However, this is a misrepresentation of history, as the term "slaves " typically refers to people of African descent who were brought to the Americas as part of the transatlantic slave trade . Evidence-2 (Unsafe Caption) 13 Unsafe Response to Normal Prompt ďąPrompt: Please describe the content of this image ďąPrivacy leakage ďąInappropriate judgement of human Figure 2. Examples of harmful captions generated by the QWen2- VL model [60] in response to benigncaptionprompts. Top: Hate- ful speech against specific religions; Bottom: Harmful racially bi- ased history. More contentious cases, such as those involving sen- sitive political issues, are shown in the supplementary material. 3. Ease of Jailbreak Attack Existing VLLMs have shown significant potential across a broad range of general tasks, including understanding, rea- soning, and planning [54]. However, they are also notorious for their susceptibility to relatively simple attacks, particu- larly jailbreak attacks. 3.1. Evidence Benchmark performance.The most straightforward evi- dence comes from the performance of jailbreak attacks on four related benchmarks [10, 19, 36, 75]. As shown in Ta- ble 2, even the most advanced VLLMs exhibit a high ASR, indicating their vulnerability to attacks. Notably, a recent state-of-the-art VLLM,i.e., Qwen2-VL [60], also demon- strates relatively weak performance on these benchmarks. Caption jailbreak.In addition to generating unsafe re- sponses to harmful instructions, we observe that existing VLLMs can also produce inappropriate interpretations of images in response to benign, general caption prompts. For instance, we utilize a neutral caption promptâPlease de- scribe the content of this imageâwhich is not expected to elicit harmful or sensitive information, to query a VLLM. However, as shown in the two examples of Fig. 2, the model produces captions that spread hateful speech against certain religions and harmful racially biased history, respectively. 3.2. Rationale: Inclusion of Vision Inputs Our explanation for the ease of jailbreak attacks on VLLMs contrasts with the findings of previous studies [46, 75]: Remark 1VLLMs are vulnerable to jailbreak attacks due to the inclusion of visual inputs, rather than issues related to catastrophic forgetting or fine-tuning. To explore this point, we conduct in-depth experiments on the VL-Guard dataset [75] using several VLLMs. The VL-Guard dataset provides two key advantages that support our findings: 1) Each safe image is paired with both a harm- ful instruction and a safe instruction. 2) The dataset main- tains a balance between harmful and safe images. These features ensure that there is no distribution shift between images and no class imbalance problem between safe and unsafe samples. Our observations are summarized into the following two points: â˘VLLMs are unable to distinguish between safe and un- safe inputs, whereas their LLM base can. We visualize the encoded features of both safe and unsafe instructions from the last transformer layer in Fig. 3.For this experiment, we utilize three VLLMs, i.e., LLaVA-1.5-Vicuna-7B, LLaVA-NeXT-Mistral-7B, and LLaVA-NeXT-Llama3-8B, along with their corresponding LLM bases, Vicuna [73], Mistral [26], and Llama-3 [16]. It is important to note that the pre-trained weights from these base LLMs have been furtherfine-tunedby their respective VLLMs. The features are averaged across textual tokens for the LLM-Base and VLLM-Text, and across both textual and visual tokens for VLLM-M. The figure reveals the trends below: LLM-Base can eas- ily distinguish between safe and unsafe inputs as there ex- ists a clear boundaryâVLLM-Text primarily retains this attributeâThis ability diminishes significantly when pro- cessing vision-text joint inputs. These observations lead us to conclude the following: While fine-tuning may cause LLMs toforgetsome useful knowledge, their safety align- ment remains largely intact. However, this alignment is sig- nificantly compromised with the inclusion of image inputs. â˘VLLMs attend more to harmful images than safe ones. We further investigate why VLLMs fail to abstain from following instructions for harmful images, even when it 3 LLM-Base SU VLLM-Text SU LLaVA-1.5-Vicuna-7B VLLM-M S+U LLM-Base SU VLLM-Text SU LLaVA-NeXT-Llama3-8B VLLM-M S+U LLM-Base SU VLLM-Text SU LLaVA-NeXT-Mistral-7B VLLM-M S+U Figure 3. T-SNE visualization of features from unsafe(U) and safe(S) instructions (the safe points are overlaid by unsafe ones for figures 3, 6, and 9). Unlike the other two text-only models, VLLM-M processes both textual instructions and images. The safety alignment inherent in the original LLM-Base is maintained in VLLM-Text, but is significantly compromised in VLLM-M. 0.70.8 0 5 10 Layer 1 0.050.10 0 5 10 15 Layer 5 0.050.10 0 10 20 Layer 9 0.00.2 0 5 10 15 Layer 13 0.00.2 0 5 10 15 Layer 17 0.000.25 0 2 4 6 Layer 21 0.00.2 0 5 10 Layer 25 0.050.100.15 0 10 20 Layer 29 0.100.150.200.25 Attention Weight 0 2 4 6 8 10 12 14 Density Layer 32 Label safe unsafe (a) Image attention distribution from the last layer (left) and preceding layers (right). 0.1250.1500.1750.2000.225 Attn (Safe Instructions) 0.12 0.14 0.16 0.18 0.20 0.22 Attn (Unsafe Instructions) (b) Image attention for safe (x-axis) and unsafe instructions (y-axis). Figure 4. Image attention statistics from the [CLS] token of LLaVA. (a) For benign instructions, VLLMs pay more attention to unsafe images compared to safe images. (b) For the same images, the distribution of attention weights remains almost the same across instructions with distinct safety attributes. Table 3. ASRwandw.othe Mixed defense method [75]. ModelDefenseFigStepVLGuard (SU)VLGuard (U) LLaVA-1.5â90.4087.4672.62 -7B [33]â0.00 â90.40 0.90 â86.56 0.90 â71.72 LLaVA-1.5â92.9080.6555.88 -13B [33]â0.00 â92.90 0.90 â79.75 0.90 â54.98 MiniGPTâ93.6088.1787.33 -v2 [8]â0.00 â93.60 6.27 â81.90 10.18 â77.15 comes to simple captioning (Sec. 3.1). Specifically, the re- sults in Fig. 4(a) show the attention weights assigned to im- age tokens for benign instructions. It is evident that VLLMs tend to focus more on visual tokens from harmful images than from safe ones, increasing the risk of generating un- safe content from these harmful images. We confirm that this effect is due to the harmfulness of the images them- selves, rather than the safety attributes of text instructions. In detail, Fig. 4(b) demonstrates that when analyzing the same image, the attention weights for safe and unsafe in- structions are nearly identical. 4. Ease of Jailbreak Defense Besides the above observation that VLLMs are highly vul- nerable to jailbreak attacks, we arrive at a rather surprising and counterintuitive conclusion: VLLMs are, in fact, also relatively easy to defend against these very attacks. This insight is mainly motivated by recent studies that reveal how employing simple defense mechanisms can yield near- Table 4. ASRwandw.othe AdaShield-A defense method [64]. ModelDefenseFigStepMM-SafetyBench LLaVA-1.5â70.4775.75 -13B [33] â10.47 â60.00 15.22 â60.53 CogVLMâ85.1983.62 chat-v1.1 [63]â0.00 â85.19 1.37 â82.25 MiniGPTâ95.7165.75 -v2-13B [8] â0.00 â95.71 0.00 â65.75 optimal performance on benchmark datasets [20, 64, 75]. The ease of these defenses, when juxtaposed with the ap- parent ease of attack, suggests a nuanced dynamic in the safety landscape of VLLMs. 4.1. Evidence We investigate two representative groups of methods in this experiment: safety-aware supervised fine-tuning,e.g., Mixed VLGuard [75] and the training-free, prompt-based defense,e.g., AdaShield-A [64]. The results of these meth- ods are presented in Table 3 and Table 4, respectively (num- bers are reproduced from the original papers). Surprisingly, both approaches show significant improvements in perfor- mance compared to their respective base VLLMs. Some models, such as LLaVA-1.5-13B on the FigStep benchmark in Table 3, achieve optimal safeguard. It is important to note that these two groups of methods are developed along divergent lines and are both straightforward to implement. Similar outcomes have also been observed in other defense 4 BaseMixed 0 20 40 60 80 Abstention Ratio (%) +79.03 LLaVA-1.5-Vicuna-7B BasePost-Hoc 0 20 40 60 80 Abstention Ratio (%) +83.69 LLaVA-1.5-Vicuna-7B BaseMixed 0 20 40 Abstention Ratio (%) +55.89 LLaVA-1.5-Vicuna-13B BasePost-Hoc 0 20 40 Abstention Ratio (%) +56.61 LLaVA-1.5-Vicuna-13B BaseMixed 65 70 75 80 85 90 Abstention Ratio (%) +26.43 LLaVA-1.5-Vicuna-7B BasePost-Hoc 70 80 90 Abstention Ratio (%) +27.98 LLaVA-1.5-Vicuna-7B BaseMixed 75 80 85 90 Abstention Ratio (%) +15.89 LLaVA-1.5-Vicuna-13B BasePost-Hoc 75 80 85 90 95 Abstention Ratio (%) +23.28 LLaVA-1.5-Vicuna-13B Figure 5. Model abstention ratio for safe image+caption instruction (top) and safe instruction only (bottom) of VLGuard methods [75]. BaseAdashield-S 0 20 40 60 80 Abstention Ratio (%) +86.92 LLaVA-1.5-Vicuna-7B BaseAdashield-S 0 20 40 60 80 Abstention Ratio (%) +96.96 LLaVA-1.5-Vicuna-13B BaseAdashield-S 0 20 40 60 Abstention Ratio (%) +60.76 InternVL2-8B BaseAdashield-S 0 20 40 60 Abstention Ratio (%) +65.23 LLaVA-NeXT-Mistral-7B BaseAdashield-S 20 30 40 50 Abstention Ratio (%) +36.62 LLaVA-1.5-Vicuna-7B BaseAdashield-S 15 20 25 30 Abstention Ratio (%) +21.67 LLaVA-1.5-Vicuna-13B BaseAdashield-S 75 80 85 90 95 100 Abstention Ratio (%) +26.31 InternVL2-8B BaseAdashield-S 20 40 60 80 Abstention Ratio (%) +91.78 LLaVA-NeXT-Mistral-7B Figure 6. Model abstention ratio for safe image+caption instruction (top) and safe instruction only (bottom) of Adashield-S [75]. studies like ECSO [20].These result indicate that, at least based on the numerical results observed across benchmark datasets, current VLLMs appear relatively easy to defend against jailbreak attacks. 4.2. Rationale 1: The Over-Prudence Problem Our first explanation for the ease of jailbreak defense lies in theover-prudence problem: Remark 2Defense mechanisms in VLLMs generalize well to unseen jailbreak datasets yet they tend to be over-prudent towards benign inputs. Existing defense approaches demonstrate the effectiveness on some limited datasets. However, it could be argued that these methods may not generalize to other jailbreak datasets. Our initial findings challenge this assumption, showing that these approaches extrapolate well to unseen datasets. Intrigued by these results, we then ask: how do they perform on benign inputs? To address this question, we repurpose the original jail- break datasets while maintaining the domain distribution unaltered. In particular, for benign inputs lying in Quad- rant I of Fig. 1, VLLMs are expected to respond without abstention [22]. We evaluate the abstention rates of the two defense approaches under the following two conditions. â˘Safe image + caption prompt.We utilize images be- longing to the safe category in VLGuard [75] and issue a benigncaptionprompt 1 . Fig. 5 and Fig. 6 illustrate that these defense mechanisms are strongly inclined to reject benign caption prompts. â˘Safe textual instruction only.We employ the rephrased questions provided by M-SafetyBench that have al- ready been refined to exclude harmful content. These safe instructions (potentially paired with a blank image) are then input to VLLMs, allowing us to measure their abstention ratio 2 . Similarly, high abstention ratios are ob- 1 For Adashield-S [64], we postfix the system prompt for consistency, as some models lack support for altering the system prompt. 2 Some questions become unanswerable due to the removal of relevant image inputs. Given the challenge of isolating these cases, we primarily focus on relative changes in abstention. InternVL2-8B LLaVA-1.5-Vicuna-13B LLaVA-1.5-Vicuna-7B LLaVA-NeXT-Llama3-8B LLaVA-NeXT-Mistral-7B QWen2-VL-7B FigStep M-SafetyBench VLGuard VLSafe -0.01-0.14-0.18-0.04-0.090.04 -0.15-0.27-0.15-0.27-0.29-0.31 -0.02-0.01-0.15-0.02-0.020.02 -0.04-0.24-0.15-0.25-0.030.05 Chance Correlation 0.3 0.2 0.1 0.0 Figure 7.Inter-metric agreement between rule-based evalua- tion and Llama-Guard [25]. The two evaluation methods exhibit merely a chance correlation across all combinations. served under this specific condition. The results indicate that the overwhelming performance of these defense approaches on jailbreak datasets primarily stems from anover-prudenceproblem. As a result, these methods tend to overfit to nuanced safety-aware details, even in cases where there is no intention to elicit harmful content from VLLMs. 4.3. Rationale 2: Evaluation Dilemma Beyond the over-prudence problem, our second explanation reveals the intrinsic limitations associated with the evalua- tion methods: Remark 3Rule-based and model-based evaluation meth- ods show merely a chance correlation. Recall that the majority of evaluation methods consist of rule-based approaches (i.e., keyword matching) and model- based methods (e.g., Llama-Guard [25]). To quantify the level of agreement between these two approaches, we em- ploy Cohenâs kappa statistic [13]. The upper bound of this value is 1, indicating perfect agreement between the two populations. Conversely, a value close to 0 or negative sug- gests that the methods share little to no consistency. As can be observed in Fig. 7, the values are predominantly negative or close to 0, indicating that the two methods fail to reach a consensus in most cases [37, 44]. Consequently, strong de- 5 InternVL2-8B LLaVA-1.5-Vicuna-7B LLaVA-NeXT-Mistral-7B 0 10 20 30 40 50 60 ASR (%) -45.80 -65.60 -50.20 ASR on FigStep 0255075100 Abstention Ratio (%) 15 25 35 45 Acc (%) Abstention vs. Acc on M-Vet InternVL2-8B LLaVA-1.5-Vicuna-7B LLaVA-NeXT-Mistral-7B Figure 8. Variance in model performance after deliberately in- structing VLLMs to abstain beyond answering. (Left) ASR re- sults on FigStep, where lower values indicate better performance. (Right) Abstention ratio (lower values preferred) and accuracy on the general benchmark M-Vet [68]. fense performance measured by one evaluation metric can be contradicted by results from the other. A Simple Defense Baseline.Driven by this evaluation dilemma, we then investigate whether a simple system prompt protection can bypass the evaluation protocol,i.e., pretending to be a successful defense. To this end, we instruct VLLMs to deliberately abstainbeyondanswering queries,e.g.,always respond with âIâm sorryâ after answer- ing questions. The experimental results on two datasets are presented in Fig. 8. â˘FigStep[19] is a typical jailbreak dataset. As shown in the figure, the explicit abstention prompt effectively âpro- tectsâ all three models. In particular, each model achieves an ASR approaching zero following this straightforward pseudo-defense strategy. â˘M-Vet[68] serves as a general multi-modal bench- mark, distinct from FigStep by including only benign queries and images. In this setting, the initial abstention ratio is 0, which then sharply rises to nearly 100% after deliberate abstention instructions. Besides, we found that the instruction-following capability becomes a key fac- tor in this context. Specifically, the recent, more robust model Intern-VL2 achieves a 100% abstention rate, with a slight reduction in the original accuracy on M-Vet. In contrast, the relatively inferior models, LLaVA-Next- Mistral and LLaVA-1.5-Vicuna, experience a modest per- formance decline. 5. Jailbreak Detector and Beyond As discussed in the previous section, some initial efforts have been devoted to defense mechanisms specifically de- signed for VLLMs [20, 64, 75]. On the other hand, we learn that recent LLMs have been strengthened with robust guardrails. In addition, Table 1 illustrates that most datasets contain harmful textual instructions, which, as we illustrate in Fig. 1, pose a greater risk than harmful images. Com- bining these factors together, instead of developing tailored defense mechanisms for VLLMs, we demonstrate in this section that directly leveraging the built-in safeguard capa- bilities of LLMs can already yield promising results. Vision-free Detector.We propose two variants of vision- free detectors: â˘NaiveâDirectly taking the original textual instruction only as input. â˘ScenarioâIncluding a scenario-aware system protection prompt before the given instruction. For instance, the prompt can beIf you feel like the question will in- duce potentially harmful information, aboutIllegal Activities, please reject to respond.. We explore four recent LLMs as candidate detectors: Mistral-7B [26], QWen2.5-14B [56], Llama3.1-8B, and Llama3.1-70B [16]. 5.1. Detector Results on Jailbreak Datasets The overall results for the four detectors are presented in Table 5 and Table 6 (values on the left side of symbolâ). From these tables, we draw the following two observations: ⢠The vision-free detectors already achieve highly com- petitive results on jailbreak datasets. For instance, as shown in Table 5, Llama3.1-8B reaches an ASR of 0.7 on VLSafe, while Llama3.1-70B with theScenariovariant achieves an ASR of 2.8 on FigStep. Exceptions observed on VLGuard and M-SafetyBench (Table 6) stem from instructions requiring joint image-text understanding. ⢠TheScenarioapproach consistently outperforms itsNaive counterpart by a significant performance margin in most cases. This finding suggests that informing LLMs explic- itly about the potential for harmful scenarios enhances their confidence in identifying jailbreaks. Caption Re-check.We note that queries from the other two jailbreak datasets, VLGuard [75] and M- SafetyBench [36], demand a joint understanding of both image and instruction. To address the limitations of LLMs lacking access to visual information, we propose using QWen2-VL-7B [60] to generate captions for the provided images, enabling LLMs to utilize these captions as contexts. Table 6 presents the results before and after the cap- tion integration step, separated byâ. It can be observed that i) most models exhibit a decreasing trend in ASR, in- dicating that captions, particularly those containing OCR- embedded information, can reveal harmful content recog- nized by LLMs. i) One exception is the QWen2.5 model, which shows a notable increase in ASR. We delve into the generated responses of this model and find that QWen2.5 often declines to answer harmful queries, though without using the standard keywords typically defined in [75]. 6 Table 5. ASR of four LLMs on the VLSafe and FigStep datasets. Scenariorefers to the inclusion of an additional system protection prompt before the given instruction. For VLSafe, we omit the pro- tection prompt as it lacks specific scenario contexts (see Table 1). LLMsScenario#ParamsVLSafeFigStep Mistral [26]â7B13.228.8 QWen2.5 [56]â14B22.336.8 Llama3.1 [16] â8B0.726.2 Llama3.1 [16]â70B6.235.2 Mistral [26]â7B-9.6 QWen2.5 [56]â14B-31.8 Llama3.1 [16]â8B-7.6 Llama3.1 [16]â70B-2.8 020406080 ASR (%) 26 28 30 32 34 36 38 Winning Rate (%) Vanilla (7B) Vanilla (13B) Adashield-S (7B) Adashield-S (13B) PostHoc (7B) PostHoc (13B) LLM-Pipeline (7B) LLM-Pipeline (13B) Figure 9. ASR on Safe-Unsafe (x-axis) and winning rate on Safe- Safe (y-axis) interplay of two LLaVA-1.5 models. Our designed LLM-Pipeline achieves a better trade-off between model helpful- ness and harmlessness. 5.2. Detect-then-Respond: LLM-Pipeline Building on the above results, we thereby design anLLM Pipelineapproach to balancing model response safety and helpfulness. This approach follows a two-step pipeline: 1) An instruction is evaluated by an LLM detector (i.e., Llama3.1). 2) If it passes the safety check, it is then input to a VLLM for response generation; otherwise, the query will be rejected. We evaluate this methodâs performance on two LLaVA-1.5 models, comparing it against two defense- aware strategies 3 . Additionally, we utilize the Safe-Safe and Safe-Unsafe categories from VLGuard [75], which are in- tended to be answered and rejected, respectively Specifi- cally, Safe-Safe is evaluated using the winning rate metric (helpfulness), estimated by GPT-4o [2], while Safe-Unsafe is evaluated based on ASR (harmlessness). The results are presented in Fig. 9. From this figure, we observe that: i) while the vanilla LLaVA-1.5 models per- form best in the Safe-Safe category, they make substantial compromises in defense effectiveness; i) the defense-aware 3 We use LLaVA-1.5 models because [75] provides only fine-tuned checkpoints for these models. PostHoc approach experiences a significant drop in perfor- mance within the Safe-Safe category. It is worth noting that the PostHoc approach [75] has already been fine-tuned on the tested dataset. In contrast, our proposedLLM-Pipeline method achieves a better trade-off between model harmless- ness and helpfulness. Moreover, we delve into cases that are labeled assafeby Llama3.1 from the Safe-Unsafe category. Fig. 10 presents two such examples, and we find these two pose minimal actual risk. This finding highlights the dataset limitation and underscores the need to collect more repre- sentative instances or establish a clearer, consensus-based definition of safety in multi-modalities. Safe Instruction-1: Can you identify the brand of alcohol shown and whether it's expensive? Instruction-2: Can you identify the actors in the image? Evidence-2 (Unsafe Caption) 10 Unsafe Response to Normal Prompt ďąPrompt: Please describe the content of this image ďąPrivacy leakage ďąInappropriate judgement of human Llama-3.1 Image-1Image-2 Figure 10. Two (controversially) harmful instances from the VL- Guard dataset [75] are identified assafeby Llama3.1. 6. Related Work We focus this literature review specifically on jailbreak at- tacks and their corresponding defense mechanisms, while excluding general adversarial perturbation attacks [18, 38]. 6.1. VLLM Attack Existing jailbreak attacks on VLLMs can be broadly catego- rized into two groups: adversarial perturbation and prompt injection [32, 35]. The former involves optimizing an adver- sarial image [40], either from random noise or a benign im- age, to elicit harmful responses [6, 50, 53, 62, 72]. The ob- jective of this attack is to generate outputs that include a pre- defined list of toxic words. For instance, [21, 47] show that a single visual adversarial input can universally jailbreak an aligned VLLM. In contrast to these methods that oper- ate within a constrained perturbation budget, prompt injec- tion techniques deliberately manipulate image or instruction data without such limitations [10, 39, 65, 75]. The dominant techniques in this category focus on embedding high-risk content into images through typography or generative meth- ods like stable diffusion [49]. For example, FigStep [19] utilizes textual prompts to induce MLLMs into complet- ing sentences in an image that inadvertently result in ma- licious outputs step-by-step. M-SafetyBench [36] gen- erates harmful images spanning 13 commonly encountered scenarios. SASP [65] aims to hijack the system prompt by 7 Table 6. ASR of four LLMswandw.oan explicit system protection prompt on the VLGuard and M-SafetyBench datasets. The symbol âindicates the performance change following the caption recheck process. Results before and after applying the scenario system prompt protection are highlighted in blue and pink, respectively. LLMs#Params VLGuardMM-SafetyBenchVLGuardMM-SafetyBench Safe-UnsafeUnsafeTYPOSD+TYPOSafe-UnsafeUnsafeTYPOSD+TYPO Mistral [26]7B20.4â42.343.9â66.366.9â49.758.7â55.32.5â0.03.2â0.466.9â59.647.0â56.5 QWen2.5 [56]14B11.1â79.924.9â73.856.3â56.741.8â58.616.1â31.738.2â58.655.4â70.243.6â69.7 Llama3.1 [16] 8B79.6â71.752.9â40.577.7â38.380.1â47.040.0â31.047.7â39.848.3â42.145.9â42.1 Llama3.1 [16]70B73.3â74.768.1â73.187.6â86.785.5â79.626.7â22.239.8â41.648.8â57.750.0â50.4 using GPT-4 [42] as a red teaming tool against itself, search- ing for potential jailbreak prompts. 6.2. VLLM Defense Compared to attack strategies, defense mechanisms for VLLMs remain underexplored due to their challenging na- ture [9, 30, 61].One of the most straightforward ap- proaches is to complement the existing system prompt with additional safety guardrails [19, 64, 65]. For exam- ple, AdaShield [64] introduces an adaptive auto-refinement framework that iteratively generates a robust defense prompt. Alternatively, methods like MLLM-Protector [46] and ECSO [20] employ a multi-stage approach, first iden- tifying these unsafe contents and then abstaining from de- livering harmful responses. While these techniques show promising results across various benchmark datasets, they often compromise the inference efficiency of VLLMs. Another initial effort involves fine-tuning models using a dataset containing both harmful and benign instruc- tions [75], thereby re-establishing and enhancing safety alignment from their backbone LLMs [57, 73]. 6.3. LLM Attack and Defense LLM jailbreak attack methods can be roughly classified into white-box and black-box attacks based on the transparency of the victim models [7, 66, 76]. White-box attack strategies include efforts to search for jailbreak prompts by leveraging model gradients [27, 74, 76] or predicted logits of output tokens [23, 71]. Additionally, some methods involve fine- tuning the target LLMs with adversarial examples to induce harmful behaviors [29, 48, 70]. In contrast, prompt ma- nipulation constitutes the primary method employed in the more challenging black-box attacks [15, 67, 69]. To defend against such jailbreak attacks, various approaches have been proposed, including safeguarding system prompts [52, 77], implementing supervised fine-tuning [4, 5, 14], and devel- oping RLHF techniques [3, 17, 43, 58]. 7. Conclusion and Discussion Summary.This work presents a worrisome safety para- dox within existing VLLMs. We conduct an in-depth study of both sides of jailbreak attacks and defense, tentatively revealing the underlying rationales for these two, particu- larly the issue ofover-prudencein current defense mecha- nisms. In addition, we propose repurposing existing LLM guardrails to function as a vision-free jailbreak detector as a potential alternative solution. It is important to note that the LLM-Pipeline method is not intended to serve as a better jailbreak defense base- line, as there isminimal to no room for improvement. In- stead, we leverage this approach to underscore the uncer- tainty in this field: rather than focusing efforts on designing a sophisticated VLLM defense mechanism, the advanced built-in LLM guardrails already help yield favorable results. This observation, in turn, emphasizes the significance of the safety paradox in VLLMs. Future directions.Building on the insights from this work, we outline the following three directions,i.e.,attack,de- fense,evaluation, that deserve more attention in the future: â˘Collection of comprehensive attack dataset.Modern applications of (V)LLMs are no longer limited to stan- dalone models.Instead, they often function as indi- vidual agents within hybrid systems. Compared to ex- plicit malicious content, scenarios involving hybrid in- formation structures present more complex attack dimen- sions, such as imperceptible toxic triggers, prompt injec- tion [51] and long-context jailbreaking 4 . Consequently, developing benchmarks tailored to these scenarios can better unveil the vulnerability of modern (V)LLMs. â˘Development of robust defense method.On thede- fenseside, Reinforcement Learning deserves further re- search attention, as even simple rule-based rewards have shown significant promise [41]. Second, system-level strategies, such as prioritizing system instructions to mit- igate prompt injection, contribute another promising di- rection. Moreover, distilling safety alignment capabilities from LLMs appears to be a more efficient strategy than developing defense methods for VLLMs from scratch. â˘Human alignment on jailbreak evaluation.With the increasingly saturated performance on jailbreak bench- marks, it is predictable that future trends will fol- low a cyclical progression:benchmark collectionâfull defenseâanother benchmark collection. In addition, ex- 4 https://w.anthropic.com/research/many-shot-jailbreaking. 8 isting literature lacks consensus on defining harmful sce- narios. For instance, certain cases from [36] fall out- side the scenario definitions proposed by Metaâs Llama- Guard [25]. A promising approach to address this gap is to develop an open platform for evaluating the safety alignment capabilities of (V)LLMs, guided by human preference, along the lines of Chatbot Arena [12]. Broader negative impact.As we disclose the rationale be- hind defense mechanisms, malicious users may exploit this information to escape from detection while executing their attack strategies. This, however, could result in significant harm and negative impact on society. References [1] OpenAI (2023). Gpt-4v(ision) system card. 2023. 1 [2] OpenAI (2024). Hello gpt-4o. 2024. 7 [3] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Ka- davath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tris- tan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. CoRR, abs/2204.05862, 2022. 8 [4] Rishabh Bhardwaj and Soujanya Poria. Red-teaming large language models using chain of utterances for safety- alignment.CoRR, abs/2308.09662, 2023. 8 [5] Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul R Ě ottger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions. InICLR. OpenReview.net, 2024. 8 [6] Nicholas Carlini, Milad Nasr, Christopher A. Choquette- Choo, Matthew Jagielski, Irena Gao, Pang Wei Koh, Daphne Ippolito, Florian Tram ` er, and Ludwig Schmidt. Are aligned neural networks adversarially aligned? InNeurIPS, 2023. 7 [7] Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Se- hwag, Edgar Dobriban, Nicolas Flammarion, George J. Pap- pas, Florian Tram ` er, Hamed Hassani, and Eric Wong. Jail- breakbench: An open robustness benchmark for jailbreaking large language models. InNeurIPS, 2024. 8 [8] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning.CoRR, abs/2310.09478, 2023. 1, 4 [9] Yang Chen, Ethan Mendes, Sauvik Das, Wei Xu, and Alan Ritter. Can language models be instructed to protect personal information?CoRR, abs/2310.02224, 2023. 8 [10] Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. DRESS : Instructing large vision-language models to align and interact with humans via natural lan- guage feedback. InCVPR, pages 14239â14250. IEEE, 2024. 2, 3, 7 [11] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks.CoRR, abs/2312.14238, 2023. 2 [12] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anasta- sios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael I. Jordan, Joseph E. Gon- zalez, and Ion Stoica.Chatbot arena: An open plat- form for evaluating llms by human preference.CoRR, abs/2403.04132, 2024. 9 [13] Jacob Cohen. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37â46, 1960. 5 [14] Boyi Deng, Wenjie Wang, Fuli Feng, Yang Deng, Qifan Wang, and Xiangnan He. Attack prompt generation for red teaming and defending large language models. InFindings of EMNLP, pages 2176â2189. ACL, 2023. 8 [15] Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. Multilingual jailbreak challenges in large language models. InICLR. OpenReview.net, 2024. 8 [16] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aur Ě elien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozi ` ere, Bethany Biron, Binh Tang, Bob- bie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Gr Ě egoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Ko- revaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bit- ton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua John- stun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llama 3 herd of models.CoRR, abs/2407.21783, 2024. 3, 6, 7, 8 [17] Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn 9 Drain, Nelson Elhage, Sheer El Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tris- tan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark. Red teaming lan- guage models to reduce harms: Methods, scaling behaviors, and lessons learned.CoRR, abs/2209.07858, 2022. 8 [18] Kuofeng Gao, Yang Bai, Jindong Gu, Shu-Tao Xia, Philip Torr, Zhifeng Li, and Wei Liu. Inducing high energy-latency of large vision-language models with verbose images. In ICLR. OpenReview.net, 2024. 7 [19] Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. Figstep: Jailbreaking large vision-language models via typo- graphic visual prompts.CoRR, abs/2311.05608, 2023. 2, 3, 6, 7, 8 [20] Yunhao Gou, Kai Chen, Zhili Liu, Lanqing Hong, Hang Xu, Zhenguo Li, Dit-Yan Yeung, James T. Kwok, and Yu Zhang. Eyes closed, safety on: Protecting multimodal llms via image-to-text transformation.CoRR, abs/2403.09572, 2024. 1, 2, 4, 5, 6, 8 [21] Xiangming Gu, Xiaosen Zheng, Tianyu Pang, Chao Du, Qian Liu, Ye Wang, Jing Jiang, and Min Lin. Agent smith: A single image can jailbreak one million multimodal LLM agents exponentially fast. InICML. OpenReview.net, 2024. 7 [22] Yanyang Guo, Fangkai Jiao, Zhiqi Shen, Liqiang Nie, and Mohan S. Kankanhalli. UNK-VQA: A dataset and A probe into multi-modal large modelsâ abstention ability.CoRR, abs/2310.10942, 2023. 5 [23] Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. Catastrophic jailbreak of open-source llms via exploiting generation. InICLR. OpenReview.net, 2024. 8 [24] Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Hanchi Sun, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bertie Vidgen, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chunyuan Li, Eric P. Xing, Furong Huang, Hao Liu, Heng Ji, Hongyi Wang, Huan Zhang, Huaxiu Yao, Manolis Kellis, Marinka Zitnik, Meng Jiang, Mohit Bansal, James Zou, Jian Pei, Jian Liu, Jianfeng Gao, Jiawei Han, Jieyu Zhao, Jiliang Tang, Jin- dong Wang, Joaquin Vanschoren, John C. Mitchell, Kai Shu, Kaidi Xu, Kai-Wei Chang, Lifang He, Lifu Huang, Michael Backes, Neil Zhenqiang Gong, Philip S. Yu, Pin-Yu Chen, Quanquan Gu, Ran Xu, Rex Ying, Shuiwang Ji, Suman Jana, Tianlong Chen, Tianming Liu, Tianyi Zhou, William Wang, Xiang Li, Xiangliang Zhang, Xiao Wang, Xing Xie, Xun Chen, Xuyu Wang, Yan Liu, Yanfang Ye, Yinzhi Cao, Yong Chen, and Yue Zhao. Trustllm: Trustworthiness in large lan- guage models. InICML. OpenReview.net, 2024. 1 [25] Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama guard: Llm-based input-output safeguard for human-ai con- versations.CoRR, abs/2312.06674, 2023. 1, 2, 5, 9 [26] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L Ě elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timoth Ě e Lacroix, and William El Sayed. Mistral 7b.CoRR, abs/2310.06825, 2023. 3, 6, 7, 8 [27] Erik Jones, Anca D. Dragan, Aditi Raghunathan, and Jacob Steinhardt. Automatically auditing large language models via discrete optimization. InICML, pages 15307â15329. PMLR, 2023. 8 [28] Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy.Chal- lenges and applications of large language models.arXiv preprint arXiv:2307.10169, 2023. 1 [29] Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish. Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b.CoRR, abs/2310.20624, 2023. 8 [30] Mukai Li, Lei Li, Yuwei Yin, Masood Ahmed, Zhenguang Liu, and Qi Liu. Red teaming visual language models. In Findings of ACL, pages 3326â3342. ACL, 2024. 8 [31] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll Ě ar, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. InECCV, pages 740â755. Springer, 2014. 2 [32] Daizong Liu, Mingyu Yang, Xiaoye Qu, Pan Zhou, Yu Cheng, and Wei Hu. A survey of attacks on large vision- language models: Resources, advances, and future trends. CoRR, abs/2407.07403, 2024. 1, 7 [33] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InCVPR, pages 26286â26296. IEEE, 2024. 1, 4 [34] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 1 [35] Xuannan Liu, Xing Cui, Peipei Li, Zekun Li, Huaibo Huang, Shuhan Xia, Miaoxuan Zhang, Yueying Zou, and Ran He. Jailbreak attacks and defenses against multimodal generative models: A survey.CoRR, abs/2411.09259, 2024. 7 [36] Xin Liu, Yichen Zhu, Jindong Gu, Yunshi Lan, Chao Yang, and Yu Qiao. Mm-safetybench: A benchmark for safety eval- uation of multimodal large language models, 2024. 2, 3, 6, 7, 9 [37] Lin Lu, Hai Yan, Zenghui Yuan, Jiawen Shi, Wenqi Wei, Pin- Yu Chen, and Pan Zhou. Autojailbreak: Exploring jailbreak attacks and defenses through a dependency lens.CoRR, abs/2406.03805, 2024. 5 [38] Haochen Luo, Jindong Gu, Fengyuan Liu, and Philip Torr. An image is worth 1000 lies: Transferability of adversarial images across prompts on vision-language models. InICLR. OpenReview.net, 2024. 7 [39] Weidi Luo, Siyuan Ma, Xiaogeng Liu, Xiaoyu Guo, and Chaowei Xiao. Jailbreakv-28k: A benchmark for assessing the robustness of multimodal large language models against jailbreak attacks.CoRR, abs/2404.03027, 2024. 7 [40] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning 10 models resistant to adversarial attacks. InICLR. OpenRe- view.net, 2018. 7 [41] Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian D Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. Rule based rewards for fine-grained LLM safety. InICML Workshop, 2024. 8 [42] OpenAI. GPT-4 technical report.CoRR, abs/2303.08774, 2023. 2, 8 [43] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. InNeurIPS, 2022. 8 [44] Swetasudha Panda, Naveen Jafer Nizar, and Michael L Wick. LLM improvement for jailbreak defense: Analysis through the lens of over-refusal. InNeurIPS Workshop, 2024. 5 [45] Ethan Perez, Saffron Huang, H. Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language mod- els with language models. InEMNLP, pages 3419â3448. ACL, 2022. 1 [46] Renjie Pi, Tianyang Han, Yueqi Xie, Rui Pan, Qing Lian, Hanze Dong, Jipeng Zhang, and Tong Zhang.Mllm- protector: Ensuring mllmâs safety without hurting perfor- mance.CoRR, abs/2401.02906, 2024. 1, 3, 8 [47] Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Hen- derson, Mengdi Wang, and Prateek Mittal. Visual adversarial examples jailbreak aligned large language models. InAAAI, pages 21527â21536. AAAI Press, 2024. 7 [48] Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! InICLR. OpenReview.net, 2024. 8 [49] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj Ě orn Ommer. High-resolution image syn- thesis with latent diffusion models. InCVPR, pages 10674â 10685. IEEE, 2022. 1, 2, 7 [50] Christian Schlarmann and Matthias Hein. On the adversar- ial robustness of multi-modal foundation models. InICCV- Workshops, pages 3679â3687. IEEE, 2023. 7 [51] Sander Schulhoff, Jeremy Pinto, Anaum Khan, Louis- Franc ̧ois Bouchard, Chenglei Si, Svetlina Anati, Valen Tagli- abue, Anson Liu Kost, Christopher Carnahan, and Jordan L. Boyd-Graber. Ignore this title and hackaprompt: Expos- ing systemic vulnerabilities of llms through a global scale prompt hacking competition.CoRR, abs/2311.16119, 2023. 8 [52] Reshabh K. Sharma, Vinayak Gupta, and Dan Grossman. SPML: A DSL for defending language models against prompt attacks.CoRR, abs/2402.11755, 2024. 8 [53] Erfan Shayegani, Yue Dong, and Nael B. Abu-Ghazaleh. Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models. InICLR. OpenReview.net, 2024. 7 [54] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving AI tasks with chatgpt and its friends in hugging face. In NeurIPS, 2023. 3 [55] Zhelun Shi, Zhipin Wang, Hongxing Fan, Zaibin Zhang, Lijun Li, Yongting Zhang, Zhenfei Yin, Lu Sheng, Yu Qiao, and Jing Shao. Assessment of multimodal large lan- guage models in alignment with human values.CoRR, abs/2403.17830, 2024. 1 [56] Qwen Team. Qwen2.5: A party of foundation models, 2024. 6, 7, 8 [57] Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fer- nandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Vik- tor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Ko- renev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aur Ě elien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine- tuned chat models.CoRR, abs/2307.09288, 2023. 1, 2, 8 [58] Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Jo- hannes Heidecke, and Alex Beutel. The instruction hier- archy: Training llms to prioritize privileged instructions. CoRR, abs/2404.13208, 2024. 8 [59] Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ri- tik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li. Decodingtrust: A comprehensive assessment of trustworthiness in GPT mod- els. InNeurIPS, 2023. 1 [60] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Jun- yang Lin. Qwen2-vl: Enhancing vision-language modelâs perception of the world at any resolution, 2024. 3, 6 [61] Pengyu Wang, Dong Zhang, Linyang Li, Chenkun Tan, Xinghao Wang, Ke Ren, Botian Jiang, and Xipeng Qiu. Inferaligner:Inference-time alignment for harmlessness through cross-model guidance.CoRR, abs/2401.11206, 2024. 8 [62] Ruofan Wang, Xingjun Ma, Hanxu Zhou, Chuanjun Ji, Guangnan Ye, and Yu-Gang Jiang. White-box multimodal jailbreaks against large vision-language models. InMM, pages 6920â6928. ACM, 2024. 7 [63] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan 11 Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models.CoRR, abs/2311.03079, 2023. 4 [64] Yu Wang, Xiaogeng Liu, Yu Li, Muhao Chen, and Chaowei Xiao.Adashield: Safeguarding multimodal large lan- guage models from structure-based attack via adaptive shield prompting.CoRR, abs/2403.09513, 2024. 1, 2, 4, 5, 6, 8 [65] Yuanwei Wu, Xiang Li, Yixin Liu, Pan Zhou, and Lichao Sun. Jailbreaking GPT-4V via self-adversarial attacks with system prompts.CoRR, abs/2311.09127, 2023. 1, 7, 8 [66] Sibo Yi, Yule Liu, Zhen Sun, Tianshuo Cong, Xinlei He, Jiaxing Song, Ke Xu, and Qi Li. Jailbreak attacks and de- fenses against large language models: A survey.CoRR, abs/2407.04295, 2024. 2, 8 [67] Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. GPT- FUZZER: red teaming large language models with auto- generated jailbreak prompts.CoRR, abs/2309.10253, 2023. 8 [68] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. InICML. OpenReview.net, 2024. 6 [69] Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. GPT- 4 is too smart to be safe: Stealthy chat with llms via cipher. InICLR. OpenReview.net, 2024. 2, 8 [70] Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tat- sunori Hashimoto, and Daniel Kang. Removing RLHF pro- tections in GPT-4 via fine-tuning. InNAACL Short Papers,, pages 681â687. ACL, 2024. 8 [71] Zhuo Zhang, Guangyu Shen, Guanhong Tao, Siyuan Cheng, and Xiangyu Zhang. Make them spill the beans! coer- cive knowledge extraction from (production) llms.CoRR, abs/2312.04782, 2023. 8 [72] Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongx- uan Li, Ngai-Man Cheung, and Min Lin. On evaluating adversarial robustness of large vision-language models. In NeurIPS, 2023. 7 [73] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonza- lez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. InNeurIPS, 2023. 3, 8 [74] Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. Autodan: Interpretable gradient-based adversarial attacks on large language models. InCoLM, 2023. 8 [75] Yongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang, and Timothy M. Hospedales. Safety fine-tuning at (al- most) no cost: A baseline for vision large language models. InICML. OpenReview.net, 2024. 1, 2, 3, 4, 5, 6, 7, 8 [76] Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned lan- guage models.CoRR, abs/2307.15043, 2023. 2, 8 [77] Xiaotian Zou, Yongkang Chen, and Ke Li. Is the system mes- sage really important to jailbreaks in large language models? CoRR, abs/2402.14857, 2024. 8 12