Paper deep dive

Effective and Efficient Adversarial Detection for Vision-Language Models via A Single Vector

Youcheng Huang, Fengbin Zhu, Jingkun Tang, Pan Zhou, Wenqiang Lei, Jiancheng Lv, Tat-Seng Chua

Year: 2024Venue: arXiv preprintArea: Multimodal SafetyType: EmpiricalEmbeddings: 95

Models: LLaVA, MiniGPT-4

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/12/2026, 6:44:09 PM

Summary

The paper introduces RADAR, a large-scale dataset of adversarial images for Vision-Language Models (VLMs), and NEARSIDE, an efficient detection method that uses a single 'attacking direction' vector derived from VLM hidden states to identify adversarial inputs.

Entities (5)

LLaVA · vlm · 100%MiniGPT-4 · vlm · 100%NEARSIDE · method · 100%RADAR · dataset · 100%Attacking Direction · concept · 95%

Relation Signals (3)

NEARSIDE → detectsadversarialinputsin → VLMs

confidence 95% · NEARSIDE method, which leverages the attacking direction to detect adversarial images to defend VLMs.

RADAR → usedtoevaluate → NEARSIDE

confidence 95% · With RADAR, we further propose a novel iN-time Embedding-based AdveRSarial Image DEtection (NEARSIDE) method

LLaVA → victimof → Adversarial Attacks

confidence 90% · Extensive experiments with two victim VLMs, LLaVA and MiniGPT-4

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Visual Language Models (VLMs) are vulnerable to adversarial attacks, especially those from adversarial images, which is however under-explored in literature. To facilitate research on this critical safety problem, we first construct a new laRge-scale Adervsarial images dataset with Diverse hArmful Responses (RADAR), given that existing datasets are either small-scale or only contain limited types of harmful responses. With the new RADAR dataset, we further develop a novel and effective iN-time Embedding-based AdveRSarial Image DEtection (NEARSIDE) method, which exploits a single vector that distilled from the hidden states of VLMs, which we call the attacking direction, to achieve the detection of adversarial images against benign ones in the input. Extensive experiments with two victim VLMs, LLaVA and MiniGPT-4, well demonstrate the effectiveness, efficiency, and cross-model transferrability of our proposed method. Our code is available at this https URL

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

94,746 characters extracted from source content.

Expand or collapse full text

Effective and Efficient Adversarial Detection for Vision-Language Models via A Single Vector Youcheng Huang1 , Fengbin Zhu211footnotemark: 1 , Jingkun Tang1 Pan Zhou3 , Wenqiang Lei122footnotemark: 2 , Jiancheng Lv1, Tat-Seng Chua2 1Sichuan University, 2National University of Singapore 3Singapore Management University youchenghuang, tangjingkun@stu.scu.edu.cn zhfengbin@gmail.com panzhou@smu.edu.sg wenqianglei,lvjiancheng@scu.edu.cn dcscts@nus.edu.sg Equal contribution.Corresponding authors. Abstract Visual Language Models (VLMs) are vulnerable to adversarial attacks, especially those from adversarial images, which is however under-explored in literature. To facilitate research on this critical safety problem, we first construct a new laRge-scale Adervsarial images dataset with Diverse hArmful Responses (RADAR), given that existing datasets are either small-scale or only contain limited types of harmful responses. With the new RADAR dataset, we further develop a novel and effective iN-time Embedding-based AdveRSarial Image DEtection (NEARSIDE) method, which exploits a single vector that distilled from the hidden states of VLMs, which we call the attacking direction, to achieve the detection of adversarial images against benign ones in the input. Extensive experiments with two victim VLMs, LLaVA and MiniGPT-4, well demonstrate the effectiveness, efficiency, and cross-model transferrability of our proposed method. Our code is available at https://github.com/mob-scu/RADAR-NEARSIDE. 1 Introduction Vision Language Models (VLMs), such as BLIP-2 (Li et al., 2023a), LLaVA (Liu et al., 2023a), MiniGPT-4 (Zhu et al., 2024) and GPT-4V (OpenAI, 2023), have attained remarkable success over various vision-language tasks (Dai et al., 2023; Zhu et al., 2024). Besides improving performances, ensuring the safety of responses is just as important in the development of VLMs. Compared with classic Large Language Models (LLMs) that take in discrete textual inputs, VLMs that accept both textual and visual inputs are more susceptible to “jailbreaking”, wherein malicious users manipulate inputs to elicit harmful outputs, due to the continuous and high-dimensional nature of visual inputs (Qi et al., 2024a). This issue, which has posed persistent safety challenges in classical vision models (Chakraborty et al., 2018), also presents intrinsic difficulties for developing safe VLMs. (a) Working mechanism of VLMs. (b) Adversarial images that jailbreak VLMs. Figure 1: (a) Working mechanism of VLMs. VLMs map textual and visual inputs to the embedding space, and employ LLMs to fuse both embeddings to generate textual responses. (b) Adversarial images that jailbreak VLMs. The adversarial images that contain human-imperceptible noises can jailbreak VLMs to elicit harmful responses. Existing studies examine the safety threat in VLMs mainly from the perspective of adversarial samples as shown in Fig. 1. It has been revealed that adversarial images are more effective than adversarial texts on attacking VLMs (Qi et al., 2024a; Carlini et al., 2023). Currently, only a few studies have been conducted to protect VLMs against adversarial images. These methods either seek to detect adversarial images based on the responses’ discrepancy (Zhang et al., 2023b), or to purify noised-images (Qi et al., 2024a) with diffusion models (Nie et al., 2022), achieving promising effectiveness. However, the first approach is computation-intensive and time-consuming, as it requires sampling multiple responses for the same input; the other approach, in addition to the computational cost issue, may even suffer degraded performance when dealing with less perceptible noises. According to previous studies (Subramani et al., 2022; Turner et al., 2023; Zou et al., 2023a; Rimsky et al., 2024; Li et al., 2023b; Liu et al., 2023b) (see Sec. 2.2), the behaviors of LLMs can be modulated to generate texts towards certain specific attributes, such as truthfulness, by exploiting a set of steering vectors (SVs) that can be directly extracted from LLMs’ hidden states. In adversarial attacks, the victim VLMs are manipulated by adversarial inputs to generate harmful responses, where the VLMs’ behaviors change from harmlessness to harmfulness. We can calculate the SV that can account for VLMs’ behavior change given the adversarial inputs, which is named the attacking direction, and exploit it to detect the existence of adversarial samples by assessing whether the inputs’ embedding has high similarity to the attacking direction. However, existing datasets for investigating adversarial attacks for VLMs, as shown in Tab. 1, are small-scale and contain limited harm types, significantly restricting the thorough evaluation of VLMs defending against adversarial attacks. Therefore, we construct RADAR, a dataset of laRge-scale Adervsarial images with Diverse hArmful Responses, for comprehensively evaluating VLMs against adversarial images. In RADAR, we generate adversarial images to attack two widely-used VLMs, MiniGPT-4 (Zhu et al., 2024) and LLaVA (Liu et al., 2023a), based on a wide diversity of harmful contents. Each sample consists of an adversarial and a benign sample, with each containing a query, an adversarial/benign image and corresponding response of VLMs. In total, RADAR contains 4,000 samples, which is the most large-scale so far. For high sample quality, we apply filtering operations to ensure harmlessness and harmfulness of responses to benign and adversarial inputs respectively. It will be released to the public to facilitate related research in the community. With RADAR, we further propose a novel iN-time Embedding-based AdveRSarial Image DEtection (NEARSIDE) method, which leverages the attacking direction to detect adversarial images to defend VLMs. Specifically, we first extract the attacking direction from VLMs by calculating the average difference between the benign input and the adversarial input in the embedding space of VLMs. With the obtained attacking direction, we classify an input as an adversarial input if the projection of its embedding to the attacking direction is larger than a threshold; otherwise the input is classified as a benign input. Once the adversarial image is detected with the proposed NEARSIDE method, further actions can be taken to protect the VLMs, such as overwriting outputs with a predefined harmless response or purifying the adversarial images by diffusion models. We conduct extensive experiments to evaluate our NEARSIDE method on the new RADAR dataset. It is demonstrated that NEARSIDE achieves detection accuracy of 83.1%percent83.183.1\%83.1 % on LLaVA and 93.5%percent93.593.5\%93.5 % on MiniGPT-4, indicating impressive effectiveness. Furthermore, we experimentally verify the cross-model transferability of the attacking direction in our method. At inference, we compare the efficiency between our method and the baseline method, showing that our method takes an average of 0.140.140.140.14 seconds to complete a detection on LLaVA that is 40404040 times faster than the best existing method. In summary, the major contributions of our work are four-fold: • We propose to identify the attacking direction that directly distilled from the VLMs’ hidden space, and exploit it to defend the VLMs against adversarial images. • We construct the RADAR dataset, which is the first large-scale adversarial image dataset with a diverse range of harmful responses, to support a comprehensive analysis of VLMs’ safety and facilitate future research. • Based on RADAR, we propose a novel NEARSIDE method, which is capable of effectively and efficiently detecting adversarial visual inputs of VLMs using the identified attacking direction from VLMs’ hidden space. We further explore the cross-model transferrability of our method given the Platonic Representation Hypothesis (Huh et al., 2024). • Extensive experiments on two victim VLMs, LLaVA and MiniGPT-4, demonstrate the effectiveness, efficiency, and cross-model transferrability of our method. 2 Background 2.1 Adversarial attack Adversarial attack is maliciously manipulating inputs to compromise performance of the targeted model (Chakraborty et al., 2021; Ponnuru et al., 2023). The manipulated inputs are referred to as adversarial samples. Formally, adversarial samples are generated by minimizing the negative log-likelihood loss of an adversarial target: Iadv=arg⁢minI^adv∈ℐ⁢∑i=1m−log⁡(p⁢(yi|I^adv)).subscriptadvsubscriptargminsubscript^advℐsuperscriptsubscript1conditionalsubscriptsubscript^advI_adv= *arg\,min_ I_adv % _i=1^m- (p(y_i| I_adv)).Iadv = start_OPERATOR arg min end_OPERATORover start_ARG I end_ARG start_POSTSUBSCRIPT adv ∈ I end_POSTSUBSCRIPT ∑i = 1m - log ( p ( yitalic_i | over start_ARG I end_ARGadv ) ) . (1) Here ℐII represents the input space subject to certain constraints, such as a perturbation radius ‖Iadv−I‖≤ϵnormsubscriptadvitalic-ϵ\|I_adv-I\|≤ε∥ Iadv - I ∥ ≤ ϵ, with ϵitalic-ϵεϵ typically set to 16/2551625516/25516 / 255, 32/2553225532/25532 / 255, 64/2556425564/25564 / 255, or unbounded (denoted as “inf”). yisubscripty_iyitalic_i refers to harmful outputs, and IadvsubscriptadvI_advIadv can be either a manipulated text input, where a suffix is appended to attack LLMs (Zou et al., 2023b), or a manipulated visual input, where imperceptible noise is added to the original image to attack VLMs (Qi et al., 2024a; Carlini et al., 2023). To solve Eqn. (1), various optimization techniques can be employed to generate the adversarial sample IadvsubscriptadvI_advIadv. For LLMs, the coordinate gradient-based search (Zou et al., 2023b) or genetic algorithms (Andriushchenko et al., 2024) are commonly used due to the discrete nature of textual inputs. In contrast, for VLMs, where image noise is continuous, Projected Gradient Descent (PGD) (Madry et al., 2018; Qi et al., 2024a; Carlini et al., 2023) is an effective and widely adopted approach. 2.2 Steering vectors in LLMs According to the previous research (Subramani et al., 2022; Turner et al., 2023; Zou et al., 2023a; Rimsky et al., 2024; Li et al., 2023b; Liu et al., 2023b), the behaviors of LLMs can be modulated to generate texts towards certain specific attributes, such as truthfulness, by exploiting a set of steering vectors (SVs) that can be directly extracted from LLMs’ hidden states. To extract the SV for a certain behavior of LLMs, pairs of contrastive prompts (p+,p−subscriptsubscriptp_+,p_-p+ , p-) are used, where p+subscriptp_+p+, p−subscriptp_-p- involve the same question or request, but p+subscriptp_+p+ adds words to encourage LLMs to possess the behavior while p−subscriptp_-p- represents the opposite. Formally, given a set DD of (p+subscriptp_+p+, p−subscriptp_-p-), the SV is calculated by SV=1||⁢∑p+,p−∈LLM⁢(p+)−LLM⁢(p−)SV1subscriptsubscriptsubscriptLLMsubscriptLLMsubscriptSV= 1|D|Σ _p_+,p_- % LLM(p_+)-LLM(p_-)SV = divide start_ARG 1 end_ARG start_ARG | D | end_ARG ∑p start_POSTSUBSCRIPT + , p- ∈ D end_POSTSUBSCRIPT LLM ( p+ ) - LLM ( p- ) (2) where LLM⁢(p+),LLM⁢(p−)∈ℝdLLMsubscriptLLMsubscriptsuperscriptℝLLM(p_+),LLM(p_-)∈R^dLLM ( p+ ) , LLM ( p- ) ∈ blackboard_Rd are d-dimension vectors that represent LLMs’ embedding for the ithsuperscriptthi^thith prompt p+subscriptp_+p+ and p−subscriptp_-p- respectively. Through Eqn.(2) that takes the mean difference of the embeddings, the SV can be easily acquired, which can specify a tendency, or direction, in LLMs’ embedding space regarding the model behavior. That means, simply adding or subtracting such a direction in LLMs’ activations can noticeably control LLMs’ behavior to generate text with certain attributes. For example, given a direction of “truthfulness”, adding this direction can encourage LLMs to generate more truthful responses (Zou et al., 2023a; Rimsky et al., 2024). 3 Proposed dataset To comprehensively analyze the threat of adversarial attacks posed to VLMs, we propose RADAR, a laRge-scale Adervsarial images dataset with Diverse hArmful Responses. Fig. 2 illustrates our construction pipeline. At below we elaborate each step in the pipeline and provide an analysis of its statistics to highlight its merits. An exemplar sample in the new RADAR dataset is given in Appx. C. Figure 2: An illustration of construction pipeline for our RADAR dataset. 3.1 Data preparation In RADAR, each sample consists of an adversarial sample and a benign sample, with each containing a query, an adversarial/benign image and VLMs’ response. To build RADAR, we use queries from train and test sets in H-rlhf harm-set (Bai et al., 2022b), those from Harmful-Dataset (Harm-Data) (Sheshadri et al., 2024), and sentences in Derogatory corpus (D-corpus) (Qi et al., 2024a). We collect benign images from COCO (Lin et al., 2014), which is a large-scale image dataset widely used in computer vision. To build RADAR, we choose the validation and test sets from COCO 2017, that is, 5,000 validation images, 41,000 test images, and 91 object types in total. By adopting COCO we can introduce diverse visual information in RADAR’s samples. We generate adversarial images with harmful responses collected from H-rlhf harm-set, D-corpus, and Harm-Data, as detailed in Sec. 3.2. The H-rlhf harm-set and Harm-Data are all preference data, where each sample is a tuple of (query, response), from which we select human rejected (harmful) samples, i.e. 15.8k samples in total to build RADAR. D-corpus contains 66 derogatory sentences against gender and race, which are all inlcuded to build our RADAR. The responses in RADAR are generated by feeding queries and adversarial/benign images into the two victim VLMs, i.e., MiniGPT 4 (Zhu et al., 2024) and LLaVA (Liu et al., 2023a), as detailed in Sec. 3.3. 3.2 Adversarial image generation According to Eqn. (1) and Sec. 2.1, we optimize a continuous noise that is added to the benign image to maximize the probability of the harmful text, in order to generate an adversarial image. The optimization of noises is implemented using PGD (Madry et al., 2018). In particular, for samples from H-rlhf harm-set and Harm-Data, we optimize −log(p(yi|I^adv)- (p(y_i| I_adv)- log ( p ( yitalic_i | over start_ARG I end_ARGadv ) in Eqn. (1), where I^advsubscript^adv I_advover start_ARG I end_ARGadv denotes the noised adversarial image and the query, and yisubscripty_iyitalic_i denotes the harmful response. Note that when optimizing −log(p(yi|I^adv)- (p(y_i| I_adv)- log ( p ( yitalic_i | over start_ARG I end_ARGadv ) on D-corpus, I^advsubscript^adv I_advover start_ARG I end_ARGadv refers to only the noised adversarial image, and yisubscripty_iyitalic_i is the harmful sentence. To generate the adversarial images, we use the open-sourced code 111https://github.com/Unispac/Visual-Adversarial-Examples-Jailbreak-Large-Language-Models and leave implementation details and hyper-parameters to Appx. A. 3.3 Sample generation and filtering We then use the benign and adversarial images obtained as aforementioned to generate the samples constituting the proposed RADAR dataset. In particular, we input each benign or counterpart adversarial image plus a corresponding query to the victim VLM, i.e. MiniGPT-4 or LLaVA, respectively, and obtain a response. For D-corpus, we utilize the harmful sentence as the query. The response is then judged by two models to assess its safety. The first model is a classifier called HarmBench-Llama-2-13b-cls222https://huggingface.co/cais/HarmBench-Llama-2-13b-cls (Mazeika et al., 2024), which is fine-tuned from the Llama2-13b (Touvron et al., 2023) and classifies that whether a pair of (query, response) is harmful or not. The other model is GPT-4o mini (OpenAI, 2024), which are guided with carefully designed prompts to make judgements following (Qi et al., 2024b) and (Zeng et al., 2024). Concretely, we prompt GPT-4o mini to provide a score ranging from 1 to 5 for each tuple of (query-response), where the scores of 1, 2 indicate a harmless response, the score of 3 indicates borderline, and the scores of 4, 5 indicate a harmful response. Please refer to Appx. B for more details. It is expected that for each pair of benign and adversarial images, the responses given by the victim VLM should be judged as harmless for the benign input while harmful for the adversarial input by both models simultaneously. We take this as the criterion to determine whether the quintuple of (query, benign input, harmless response, adversarial input, harmful response) will be included in our RADAR. In practice, we find that quite a number of responses are harmful given benign images and harmless given adversarial images. As also reported in Qi et al. (2024a), the success of adversarial attack is far from 100%. When constructing our RADAR, we use the two models to judge the responses’ harmfulness. Such filtering operations significantly lift the quality of samples in the proposed dataset. 3.4 Statistics analysis With the above construction pipeline, the resultant RADAR contains 4,000 samples in total, attacking two victim VLMs, i.e. MiniGPT-4 and LLaVA. For each VLM, RADAR provides one training set and three test sets, with 500 samples per set. Division of train and test sets is based on the source of images and queries. Samples built using images from COCO validation set and queries from the train set of H-rlhf harm-set are grouped into the train set in RADAR; samples built using images from COCO test set and queries from the test set of H-rlhf harm-set, D-corpus, and Harm-Data are grouped to three test sets, respectively. Training and tests sets use different images. Different harmful texts are used in the four sets to ensure no information leakage and a reliable result. A comparison of our RADAR with previous datasets used for investigating adversarial attack for VLMs is provided in Tab. 1. Our RADAR features four advantages compared with previous ones. • Large-scale: As shown in Tab. 1, RADAR greatly surpasses the existing datasets in scale. It contains up to 4,000 samples while the previous largest dataset, i.e. from (Zhang et al., 2023a), contains only 200 samples, facilitating a reliable evaluation of VLMs’ safety. • Diversity of harmful types: RADAR covers a favorable diversity of harmful queries and responses, enabling a comprehensive evaluation of VLMs’ performance on understanding and defending various adversarial attacks. Recent research on safety of VLMs (Wang et al., 2023; Dai et al., 2024; Ji et al., 2023a) provides taxonomies about the potential harms in queries or responses, e.g. asking for guidance to make bombs or for providing private information. During the construction of RADAR, we purposely increase such diversity. • Open-source: RADAR will be open-sourced to facilitate future research on VLMs defending against adversarial attacks. • High sample quality: We apply filtering operations during the construction of our RADAR with two models to ensure that the response to a benign input is harmless and that to an adversarial input is harmful. In comparison, the other datasets are built by specifying the harmfulness of the input before feeding it to victim models, while neglecting the reliability of responses, given the success ratio that adversarial images attack VLMs is not 100%. Table 1: Comparison of datasets for adversarially attacking VLMs. “-” means not reported. Paper Scale Harmful Types Open Source Data Filtering (Zhang et al., 2023a) Arxiv 200 Harmful queries 2713 2717 (Tu et al., 2023) Arxiv 3 Toxic words 2713 2717 (Carlini et al., 2023) Neurips 2023 - Toxic words 2717 2717 (Qi et al., 2024a) AAAI 2024 3 Toxic words 2713 2717 (Luo et al., 2024) ICLR 2024 - Harmful queries 2717 2717 (Shayegani et al., 2024) ICLR 2024 8 Toxic words 2717 2717 RADAR (Ours) 4,000 Both 2713 2713 4 Proposed method Figure 3: An illustration of proposed NEARSIDE. Our method learns the attacking direction on a set of tuples (benign input, adversarial input), and then classifies a test input as benign or adversarial according to the projection between the input’s embedding and the attacking direction. If the projection is larger than a threshold, it is classified as an adversarial input, and otherwise as benign. To effieciently defend VLMs from adversarial attacks, we propose a novel iN-time Embedding-based AdveRSarial Image DEtection method (abbr. as NEARSIDE) that uses a single vector, named the attacking direction, to detect the adversarial inputs. Fig. 3 gives an illustration of NEARSIDE. 4.1 Attacking direction As discussed in Sec. 2.2, the behaviors of LLMs can be controlled with a set of steering vectors (SVs) to generate texts towards certain specific attributes, such as truthfulness. Such SVs can be easily distilled from LLMs’ hidden states based on Eqn. (2). In adversarial attacks, the adversarial inputs elicit harmful responses of the victim VLMs, where the VLMs’ behaviors alter with an attribute shifting from harmlessness to harmfulness. We can calculate the SV that can account for VLMs’ behavior change given the adversarial inputs. We name such a vector the attacking direction. In this work, we propose to detect the existence of the adversarial samples by assessing whether the inputs’ embedding has shown high similarity to the attacking direction. To extract the attacking direction from VLMs’ hidden states, the adversarial and benign samples that make pairwise contrastive prompts are required. Formally, consider a training set =(Iadvi,Ibi)|i=0,1,…,nconditional-setsuperscriptsubscriptadvsuperscriptsubscriptb01…T=\(I_adv^i,I_b^i)\,|\,i=0,1,...,n\blackboard_T = ( Iadvitalic_i , Ibitalic_i ) | i = 0 , 1 , … , n where IadvsubscriptadvI_advIadv, IbsubscriptbI_bIb denote the adversarial and benign sample, respectively, and n is the index. Each sample contains an image and a piece of text. We embed each sample IisuperscriptI^iIitalic_i by taking the embedding of the last input token from the last LLMs’ layer, i.e. Ei∈ℝdsuperscriptsuperscriptℝE^i∈R^dEitalic_i ∈ blackboard_Rd, where d is the embedding dimension. We embed all samples in Tblackboard_T, and obtain emb=(Eadvi,Ebi)|i=0,1,…,nsubscriptembconditional-setsuperscriptsubscriptadvsuperscriptsubscriptb01…T_emb=\(E_adv^i,E_b^i)\,|\,i=0,1,..% .,n\blackboard_Temb = ( Eadvitalic_i , Ebitalic_i ) | i = 0 , 1 , … , n . Then, we calculate the attacking direction by Dattack=1n⁢∑i=0n(Eadvi−Ebi)∈ℝd,(Eadvi,Ebi)∈emb.formulae-sequencesubscriptattack1superscriptsubscript0superscriptsubscriptadvsuperscriptsubscriptbsuperscriptℝsuperscriptsubscriptadvsuperscriptsubscriptbsubscriptembD_attack= 1n _i=0^n(E_adv^i-E_b^i% )∈R^d, (E_adv^i,E_b^i)∈T% _emb.Dattack = divide start_ARG 1 end_ARG start_ARG n end_ARG ∑i = 0n ( Eadvitalic_i - Ebitalic_i ) ∈ blackboard_Rd , ( Eadvitalic_i , Ebitalic_i ) ∈ blackboard_Temb . (3) 4.2 Detection of adversarial inputs Let norm⁢(h)=h/‖h‖2normℎsubscriptnormℎ2 norm(h)=h/\|h\|_2norm ( h ) = h / ∥ h ∥2 denote the ℓ2subscriptℓ2 _2ℓ2 normalization for a vector hℎh. Given the attacking direction DattacksubscriptattackD_attackDattack, we classify a test input ItestsubscripttestI_testItest to be adversarial or benign by Itest=adversarial example, ifEtest⋅norm⁢(Dattackj)⊤−t>0,benign example, otherwise,subscripttestcasesadversarial example, if⋅subscripttestnormsuperscriptsuperscriptsubscriptattacktop0benign example, otherwise,I_test= casesadversarial example, &if\ \ E_% test· norm(D_attack^j) -t>0 2m,% \\ benign example, &otherwise, casesItest = start_ROW start_CELL adversarial example, end_CELL start_CELL if Etest ⋅ norm ( Dattackitalic_j )⊤ - t > 0 , end_CELL end_ROW start_ROW start_CELL benign example, end_CELL start_CELL otherwise, end_CELL end_ROW (4) where Etest∈ℝdsubscripttestsuperscriptℝE_test∈R^dEtest ∈ blackboard_Rd is the embedding of the last input token from the last layer of an VLM on the test sample, and t∈ℝt∈Rt ∈ blackboard_R is a scalar threshold to measure whether the similarity score is significant. If the similarity score, i.e., the projection, is greater than the threshold, we classify the input ItestsubscripttestI_testItest to be adversarial as it has high similarity to the attack direction; otherwise, the input is classified as a benign input. The threshold is decided using embsubscriptembT_embblackboard_Temb: t=12⁢n⁢∑i=0n(Eadvi⋅norm⁢(Dattackj)⊤+Ebi⋅norm⁢(Dattackj)⊤),(Eadvi,Ebi)∈emb.formulae-sequence12superscriptsubscript0⋅superscriptsubscriptadvnormsuperscriptsuperscriptsubscriptattacktop⋅superscriptsubscriptbnormsuperscriptsuperscriptsubscriptattacktopsuperscriptsubscriptadvsuperscriptsubscriptbsubscriptembt= 12n _i=0^n(E_adv^i· norm(D_% attack^j) +E_b^i· norm(D_attack^j)% ), (E_adv^i,E_b^i)∈T_emb.t = divide start_ARG 1 end_ARG start_ARG 2 n end_ARG ∑i = 0n ( Eadvitalic_i ⋅ norm ( Dattackitalic_j )⊤ + Ebitalic_i ⋅ norm ( Dattackitalic_j )⊤ ) , ( Eadvitalic_i , Ebitalic_i ) ∈ blackboard_Temb . (5) The threshold is the average similarity score of all training embeddings (from both adversarial and benign samples) on the attacking direction. The proposed NEARSIDE, as shown in Eqn. (4), is extremely efficient as we only require running one feed-forward propagation given the input to infer EtestsubscripttestE_testEtest, thus enabling an in-time detection of adversarial samples. After adversarial samples have been detected, the developer can take further steps to ensure VLMs’ safety, such as overwriting the responses to a preset text, applying diffusion models to purify the image, or disabling malicious accounts. Therefore, NEARSIDE can defend VLMs from adversarial attack in an efficient and real-time manner. 4.3 Cross-model transferability The Platonic Representation Hypothesis (Huh et al., 2024): “Neural networks, trained with different objectives on different data and modalities, are converging to a shared statistical model of reality in their representation spaces.” The proposed NEARSIDE is supposed to use the attacking direction extracted from one VLM to detect the adversarial samples for the same VLM. According to the above Platonic Representation Hypothesis, we can assume that the learnt attacking direction and effectiveness of our detection method NEARSIDE are transferable across different models. That is, our NEARSIDE can use the attacking direction extracted from one VLM to detect the adversarial samples for other VLMs. The reason behind the assumption of the cross-model transferrability in our method is that, although different VLMs are trained from different data, the patterns regarding safety in these data should be similar. However, the embedding spaces between two VLMs do have a gap. We thus propose to explore the transferability using a linear transformation: ⁢m1=m2,subscriptsubscriptm1subscriptsubscriptm2 W E_m_1= E_m_2,italic_W italic_Em start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT = italic_Em start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , (6) where m1∈ℝn×dm1subscriptsubscriptm1superscriptℝsubscriptsubscriptm1 E_m_1∈R^n× d_m_1italic_Em start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ∈ blackboard_Rn × dm start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT and m2∈ℝn×dm2subscriptsubscriptm2superscriptℝsubscriptsubscriptm2 E_m_2∈R^n× d_m_2italic_Em start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ∈ blackboard_Rn × dm start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT are the stacked embedding of benign inputs from the two VLMs m1subscriptm1m_1m1 and m2subscriptm2m_2m2, respectively, with dm1,dm2subscriptsubscriptm1subscriptsubscriptm2d_m_1,d_m_2dm start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , dm start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT denoting their embedding dimension. The linear transformation Witalic_W is to align the two VLMs’ embedding spaces. In practice, since powerful LLMs often have high dimension in their hidden states, directly solving Eqn. (6) would be too costly in memory due to the high dimension of dm1,dm2subscriptsubscriptm1subscriptsubscriptm2d_m_1,d_m_2dm start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , dm start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT. Therefore, we propose to use principal component analysis (PCA) to reduce the dimension of m1subscriptsubscriptm1 E_m_1italic_Em start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT and m2subscriptsubscriptm2 E_m_2italic_Em start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT. Then, we have =fm2pca⁢(m2)⁢fm1pca⁢(m2)†subscriptsuperscriptpcasubscriptm2subscriptsubscriptm2subscriptsuperscriptpcasubscriptm1superscriptsubscriptsubscriptm2† W=f^pca_m_2( E_m_2)f% ^pca_m_1( E_m_2) italic_W = fpcam start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ( italic_Em start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ) fpcam start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ( italic_Em start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT )†, where fpcasuperscriptpcaf^pcafpca denotes PCA that reduces the dimension and (⋅)†superscript⋅†(·) ( ⋅ )† denotes the pseudo-inverse. Finally, given a test input ItestsubscripttestI_testItest, its detection on m2subscriptm2m_2m2 is given by Itest=adversarial example,iffm2pca⁢(Etest,m2)⋅norm⁢(⁢fm1pca⁢(Dattack,m1))⊤−tm1>0,benign example,otherwise,subscripttestcasesadversarial exampleif⋅subscriptsuperscriptpcasubscriptm2subscripttest,m2normsuperscriptsubscriptsuperscriptpcasubscriptm1subscriptattack,m1topsubscriptsubscriptm10benign exampleotherwiseI_test= casesadversarial example,&if\ \ f % pca_m_2(E_test,$ m_2$)·% norm( Wf^pca_m_1(D_attack,$% m_1$)) -t_m_1>0, 2m\\ benign example,&otherwise, casesItest = start_ROW start_CELL adversarial example , end_CELL start_CELL if fpcam start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ( Etest, m start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ) ⋅ norm ( italic_W fpcam start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ( Dattack, m start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ) )⊤ - tm start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT > 0 , end_CELL end_ROW start_ROW start_CELL benign example , end_CELL start_CELL otherwise , end_CELL end_ROW (7) where the threshold tm1subscriptsubscriptm1t_m_1tm start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT is defined as tm1=12⁢n∑i=0n( t_m_1= 12n _i=0^n(tm start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT = divide start_ARG 1 end_ARG start_ARG 2 n end_ARG ∑i = 0n ( ⁢fm1pca⁢(Eadv,m1i)⋅norm⁢(⁢fm1pca⁢(Dattack, m1))⊤+limit-from⋅subscriptsuperscriptpcasubscriptm1superscriptsubscriptadv,m1normsuperscriptsubscriptsuperscriptpcasubscriptm1subscriptattack, m1top Wf^pca_m_1(E_adv,$ % m_1$^i)· norm( Wf^pca_m_% 1(D_attack, $ m_1$)) +italic_W fpcam start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ( Eadv, m start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPTi ) ⋅ norm ( italic_W fpcam start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ( Dattack, m start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ) )⊤ + (8) fm1pca(Eb, m1i)⋅norm(fm1pca(Dattack, m1))⊤). Wf^pca_m_1(E_b, $ m% _1$^i)· norm( Wf^pca_m_ % 1(D_attack, $ m_1$)) ).italic_W fpcam start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ( Eb, m start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPTi ) ⋅ norm ( italic_W fpcam start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ( Dattack, m start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ) )⊤ ) . Eqn. (7) detects adversarial samples on m2subscriptm2m_2m2 only using the attacking direction of m1subscriptm1m_1m1 and the embedding of benign inputs from m2subscriptm2m_2m2 to learn the transformation matrix Witalic_W. Note that this entire learning process has no access to adversarial samples on m2subscriptm2m_2m2. Eqn. (7) works if the embedding space of the two VLMs can be linearly transformed without disturbing the attacking direction. 5 Experiments We conduct extensive experiments on RADAR to evaluate the effectiveness of the proposed NEARSIDE in detecting adversarial images. We first compare our method with strong baseline and then analyze its cross-model transferability, followed by the efficiency test. 5.1 Experiments setup Victim VLMs. We adopt MiniGPT-4 (Zhu et al., 2024) and LLaVA (Liu et al., 2023a) as the victim VLMs. MiniGPT-4 is built upon Vicuna (Chiang et al., 2023) and LLaVA is built upon Llama2 (Touvron et al., 2023). Regarding the visual encoder, MiniGPT-4 utilizes the same pre-trained vision components of BLIP-2 (Li et al., 2023a) consisting of pre-trained ViT followed by a Q-Former, while LLaVA only adopts a pre-trained CLIP (Radford et al., 2021). Implementation. For each victim VLM, as stated in Sec. 3, RADAR constructs one training set and three test sets. NEARSIDE learns the attacking direction and threshold from the hidden states of the VLM on the training set. Here, the hidden states refer to the embedding of the last token of the input from the LLM decoder’s final layer. Then, we test the detection performance with the obtained attacking direction on the three test sets. Regarding the cross-model transferability, we collect 5,000 pairs of benign images and queries to train the PCA model for each VLM. We set 2048 as the dimension of the embedding after PCA. Baseline. We use JailGuard (Zhang et al., 2023a) as our baseline, which is the state-of-the-art model for this task. To detect adversarial visual inputs, JailGuard mutates input images to generate variants and calculates the discrepancy of VLMs’ outputs given different variants to distinguish the adversarial and benign inputs. There are 18 mutation methods, and we use the best-performing mutation method “policy” reported in the JailGuard paper, where 8 variants are generated for each image. We set all other hyperparameters to the recommended values as JailGuard. It is worth mentioning that we adopt only one baseline as there are only limited works on defending VLMs from adversarial examples (Liu et al., 2024). Evaluation metrics. Since adversarial detection is a binary classification task, we adopt A⁢c⁢c⁢u⁢r⁢a⁢c⁢yAccuracyA c c u r a c y, P⁢r⁢e⁢c⁢i⁢s⁢i⁢o⁢nPrecisionP r e c i s i o n, R⁢e⁢c⁢a⁢l⁢lRecallR e c a l l and F⁢11F1F 1 score as the evaluation metrics. 5.2 Main results We compare our proposed method NEARSIDE against the baseline JailGuard on RADAR. The experimental results are shown in Tab. 2. From the results, we make below observations. 1) When taking LLaVA as the victim VLM, our NEARSIDE achieves an average increase of 31.3% in A⁢c⁢c⁢u⁢r⁢a⁢c⁢yAccuracyA c c u r a c y, 43.5% in P⁢r⁢e⁢c⁢i⁢s⁢i⁢o⁢nPrecisionP r e c i s i o n, 12.6% in R⁢e⁢c⁢a⁢l⁢lRecallR e c a l l, and 0.246 in F⁢11F1F 1, compared to the baseline JailGuard method. 2) When taking MiniGPT-4 as the victim VLM, our NEARSIDE achieves an average increase of 38.7% in A⁢c⁢c⁢u⁢r⁢a⁢c⁢yAccuracyA c c u r a c y, 45.6% in P⁢r⁢e⁢c⁢i⁢s⁢i⁢o⁢nPrecisionP r e c i s i o n, 17.6% in R⁢e⁢c⁢a⁢l⁢lRecallR e c a l l, and 0.316 in F⁢11F1F 1, compared to the baseline JailGuard method. These results well demonstrate the effectiveness of our proposed method. 3) Although our NEARSIDE has lower R⁢e⁢c⁢a⁢l⁢lRecallR e c a l l on the Harm-Data set with LLaVA as the victim VLM, and also on D-corpus-test set with MiniGPT-4 as the victim VLM, it achieves significantly higher F⁢11F1F 1 scores on both sets. We attribute the low R⁢e⁢c⁢a⁢l⁢lRecallR e c a l l of our method to its threshold for the adversarial detection. As shown in Fig. 5, the projections of the two types of examples do fall into different ranges. However, as the threshold is calculated on the training set, the threshold is not well fit for the Harm-Data, leading to degraded R⁢e⁢c⁢a⁢l⁢lRecallR e c a l l. If we set the threshold to -13, we can increase R⁢e⁢c⁢a⁢l⁢lRecallR e c a l l to 87.6% and F⁢11F1F 1 score to 0.9000.9000.9000.900, which are both significantly higher than the baseline. From an overall perspective, the results can still demonstrate the powerful distinguishing capability of our method over adversarial and benign data. Table 2: Results of JailGuard v.s. NEARSIDE on RADAR test sets (best highlighted in bold). Victim VLM Test Set Method Accuracy(%)Accuracy(\%)A c c u r a c y ( % ) Precision(%)Precision(\%)P r e c i s i o n ( % ) Recall(%)Recall(\%)R e c a l l ( % ) F⁢11F1F 1 LLaVA H-rlhf JailGuard 51.2 51.1 57.8 0.540 NEARSIDE 84.4 89.3 78.2 0.834 D-corpus JailGuard 58.1 58.1 58.2 0.581 NEARSIDE 94.0 99.5 88.4 0.936 Harm-Data JailGuard 46.2 46.8 55.8 0.509 NEARSIDE 71.0 97.7 43.0 0.597 MiniGPT-4 H-rlhf JailGuard 54.9 53.9 67.2 0.598 NEARSIDE 99.4 99.2 99.6 0.994 D-corpus JailGuard 56.6 54.4 81.6 0.653 NEARSIDE 81.1 98.4 63.2 0.770 Harm-Data JailGuard 52.8 52.4 61.2 0.565 NEARSIDE 100.0 100.0 100.0 1.000 Figure 4: Visualized projections of adversarial and benign samples to the attacking directions on Harm-Data with LLaVA as the victim. Figure 5: Throughput of four different detection methods. The number is the average examples can be detected per second (item/s). 5.3 Analysis on cross-model transferability We utilize the attacking direction extracted from the source VLM (svlm) to detect adversarial input for the target VLM (tvlm), denoted as svlm → tvlm. We calculate the difference (i.e. δ) by subtracting the result of (svlm → tvlm) from that of Tab. 2, where -δ denotes the result is decreased while +δ denotes the opposite. The obtained results for cross-model transferaility are shown in Tab. 3. We can observe that cross-model transferability results are generally inferior to those in Tab. 2 where the attacking direction is extracted and applied with the same VLM, but both A⁢c⁢c⁢u⁢r⁢a⁢c⁢yAccuracyA c c u r a c y and F⁢11F1F 1 results of our method are higher than those of the baseline JailGuard. Though cross-model transferrability decreases the detection performance, which is expectable, our method can still work well across different models. These results clearly validate the cross-model transferability of the attacking direction and the proposed NEARSIDE. It also says that, the Platonic Representation Hypothesis still holds in our setting, where a simple linear transformation is effective to align two VLMs’ embedding spaces. Table 3: Cross-model transferability results for our method. svlm → tvlm TEST SET Accuracy(%)Accuracy(\%)A c c u r a c y ( % ) Precision(%)Precision(\%)P r e c i s i o n ( % ) Recall(%)Recall(\%)R e c a l l ( % ) F⁢11F1F 1 MiniGPT-4→LLaVA→MiniGPT-4LLaVAMiniGPT-4 -4 → LLaVA H-rlhf 64.3−20.1superscript64.320.164.3 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-20.164.3- 20.1 61.3−28.0superscript61.328.061.3 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-28.061.3- 28.0 77.6−0.6superscript77.60.677.6 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-0.677.6- 0.6 0.685−0.149superscript0.6850.1490.685 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-0.1490.685- 0.149 D-corpus 69.4−24.6superscript69.424.669.4 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-24.669.4- 24.6 62.5−37superscript62.53762.5 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-3762.5- 37 96.8+8.4superscript96.88.496.8 [rgb]0,0.58984375,0 [named]pgfstrokecolorrgb% 0,0.58984375,0+8.496.8+ 8.4 0.760−0.176superscript0.7600.1760.760 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-0.1760.760- 0.176 Harm-Data 74.7+3.7superscript74.73.774.7 [rgb]0,0.58984375,0 [named]pgfstrokecolorrgb% 0,0.58984375,0+3.774.7+ 3.7 76.7−21superscript76.72176.7 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-2176.7- 21 71.0+28superscript71.02871.0 [rgb]0,0.58984375,0 [named]pgfstrokecolorrgb% 0,0.58984375,0+2871.0+ 28 0.737+0.14superscript0.7370.140.737 [rgb]0,0.58984375,0 [named]pgfstrokecolorrgb% 0,0.58984375,0+0.140.737+ 0.14 LLaVA→MiniGPT-4→LLaVAMiniGPT-4LLaVA -4LLaVA → MiniGPT-4 H-rlhf 77.8−21.6superscript77.821.677.8 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-21.677.8- 21.6 86.2−13superscript86.21386.2 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-1386.2- 13 66.2−33.4superscript66.233.466.2 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-33.466.2- 33.4 0.749−0.245superscript0.7490.2450.749 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-0.2450.749- 0.245 D-corpus 80.4−0.7superscript80.40.780.4 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-0.780.4- 0.7 80.3−18.1superscript80.318.180.3 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-18.180.3- 18.1 80.6+17.4superscript80.617.480.6 [rgb]0,0.58984375,0 [named]pgfstrokecolorrgb% 0,0.58984375,0+17.480.6+ 17.4 0.804+0.034superscript0.8040.0340.804 [rgb]0,0.58984375,0 [named]pgfstrokecolorrgb% 0,0.58984375,0+0.0340.804+ 0.034 Harm-Data 97.1−2.9superscript97.12.997.1 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-2.997.1- 2.9 95.0−0.5superscript95.00.595.0 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-0.595.0- 0.5 99.4−0.6superscript99.40.699.4 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-0.699.4- 0.6 0.972−0.028superscript0.9720.0280.972 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-0.0280.972- 0.028 We experiment to examine the effect of using Witalic_W to align two VLMs’ embedding spaces, and the effect of reducing the dimension of the attacking direction and the VLMs’ embedding with the PCA model. The detailed results are provided in Appx. E. We find that, without Witalic_W, the cross-model transferability results will significantly decrease. In addition, when reducing the dimension to 256, the cross-model transferability results still remain high, indicating that the information in low dimensional sub-spaces is already sufficient for aligning two VLMs’ embedding spaces. 5.4 Analysis of perturbation radius in generating adversarial images The generation of adversarial images is constrained by the hyper-parameter ϵitalic-ϵεϵ as shown in Eqn. (1). We test the robustness of the proposed NEARSIDE method to varying ϵitalic-ϵεϵ. We use NEARSIDE to detect the adversarial samples generated under different ϵitalic-ϵεϵ. Results are deferred to Appx. D. 5.5 Analysis of detection efficiency In this part, we examine the detection efficiency of the proposed method. For the baseline JailGuard, we utilize the widely-adopted VLLM (Kwon et al., 2023) to deploy the two VLMs, i.e. LLaVA and MiniGPT-4, on a local machine and generate outputs through API requests. For our proposed NEARSIDE, we load VLMs and perform a single forward propagation to embed each input since the attacking direction can be pre-computed. In addition to JailGuard and our method, we also include another two trivial methods that judge the harmfulness of the output into our efficiency evaluations, i.e. HarmBench and GPT-4o mini, which are used in data filtering operations to judge the harmfulness of responses in Sec. 3.3. For all compared methods, we calculate the time including responses inference (note, NEARSIDE does not infer responses) plus follow-up operations, which refer to discrepancy calculation in JailGuard, projections calculation in NEARSIDE, and harmfulness evaluation in other two methods. All experiments are conducted on a server with AMD EPYC 7543 32-core processors, 1 TB of RAM, and a NVIDIA A40 GPU. We run experiments on 20 inputs and plot the average throughput in Fig. 5. With our setup, NEARSIDE is (× 41∼similar-to ∼336) times faster than the other methods on LLaVA, and is (× 132∼similar-to ∼1190) times faster on MiniGPT-4, demonstrating remarkable efficiency as NEARSIDE is the only embedding-based method among all compared methods that does not require to infer the entire output. 6 Related works 6.1 Vision language models Vision Language Models (VLMs) is equipped with a visual adapter to align the visual and textual representations in LLMs. Notable examples are BLIP-2 (Li et al., 2023a), LLaVA (Liu et al., 2023a), MiniGPT-4 (Zhu et al., 2024), QWen-VL (Bai et al., 2023), GPT-4V (OpenAI, 2023), and Gemini (Anil et al., 2023), demonstrating impressive performance across various vision-language tasks (Dai et al., 2023; Zhu et al., 2024). These VLMs vary in the design of their adapters (Liu et al., 2023a; Li et al., 2023a; Zhu et al., 2024). For instance, BLIP-2 (Li et al., 2023a) proposes Q-Former to align vision features with LLMs; MiniGPT-4 (Zhu et al., 2024) and LLaVA (Liu et al., 2023a) further add a linear transformation, and Qwen-VL (Bai et al., 2023) uses a single-layer cross-attention module. 6.2 Safety of language models Safe LLMs should behave in line with human intentions and values (Soares & Fallenstein, 2014; Hendrycks et al., 2021; Leike et al., 2018; Ji et al., 2023b) which are measured as being Helpful, Honest, and Harmless (Askell et al., 2021). Alignment has emerged as a nascent research field aiming to align LLMs’ behaviors with human preferences, and there are two widely adopted alignment techniques, i.e. Instruction Fine-tuning and Reinforcement Learning from Human Feedback (RLHF). In instruction fine-tuning, LLMs are given examples of (user’s query, desired output) and trained to follow user instructions and respond the expected output (Taori et al., 2023; Wei et al., 2022). In RLHF, LLMs update output probabilities, i.e., the response policy, by reinforcement learning, which are rewarded for generating responses that align with human preferences and otherwises penalized (Russell & Norvig, 2016; Bai et al., 2022a; Rafailov et al., 2023; Ouyang et al., 2022). Two types of strategies can defend language models from adversarial attacks: detection and purification. For instance, Zhang et al. (2023b) detects adversarial examples by calculating responses’ discrepancy; Qi et al. (2024a) uses diffusion models (Nie et al., 2022) to purify the noised-images. Other techniques such as the adversarial training (Bai et al., 2021) can also improve the robustness of models to adversarial attacks. Though effective on classical image classifiers, these methods remain unexplored on large models like LLMs and VLMs and may disturb the optimization. 7 Conclusion In this work, we propose RADAR, the first large-scale adversarial image dataset with diverse harmful responses to facilitate research on safety of VLMs. With RADAR, we further develop NEARSIDE that exploits the idea of attacking direction to detect adversarial inputs. We demonstrate with the effectiveness and efficiency of the proposed NEARSIDE by comparing it to the state-of-the-art on RADAR, and also highlight its cross-model transferability. VLMs can generate open-ended responses, posing a persistent challenge to complete evaluation of the potential harms (Ganguli et al., 2022). RADAR is built from a diverse array of datasets but may fall short of covering all harmful contents. NEARSIDE is intended to detect the adversarial samples we examine in this work and is a demonstration of our idea of exploiting the attacking direction. References Andriushchenko et al. (2024) Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking leading safety-aligned llms with simple adaptive attacks. CoRR, abs/2404.02151, 2024. doi: 10.48550/ARXIV.2404.02151. URL https://doi.org/10.48550/arXiv.2404.02151. Anil et al. (2023) Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, Iain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and et al. Gemini: A family of highly capable multimodal models. CoRR, abs/2312.11805, 2023. doi: 10.48550/ARXIV.2312.11805. URL https://doi.org/10.48550/arXiv.2312.11805. Askell et al. (2021) Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. Bai et al. (2023) Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. Bai et al. (2021) Tao Bai, Jinqi Luo, Jun Zhao, Bihan Wen, and Qian Wang. Recent advances in adversarial training for adversarial robustness. In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, p. 4312–4321. ijcai.org, 2021. doi: 10.24963/IJCAI.2021/591. URL https://doi.org/10.24963/ijcai.2021/591. Bai et al. (2022a) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. CoRR, abs/2204.05862, 2022a. doi: 10.48550/ARXIV.2204.05862. URL https://doi.org/10.48550/arXiv.2204.05862. Bai et al. (2022b) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022b. Carlini et al. (2023) Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo, Matthew Jagielski, Irena Gao, Pang Wei Koh, Daphne Ippolito, Florian Tramèr, and Ludwig Schmidt. Are aligned neural networks adversarially aligned? In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URL http://papers.nips.c/paper_files/paper/2023/hash/c1f0b856a35986348ab3414177266f75-Abstract-Conference.html. Chakraborty et al. (2018) Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay. Adversarial attacks and defences: A survey. CoRR, abs/1810.00069, 2018. URL http://arxiv.org/abs/1810.00069. Chakraborty et al. (2021) Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay. A survey on adversarial attacks and defences. CAAI Trans. Intell. Technol., 6(1):25–45, 2021. doi: 10.1049/CIT2.12028. URL https://doi.org/10.1049/cit2.12028. Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/. Dai et al. (2024) Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=TyFrPOKYXw. Dai et al. (2023) Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URL http://papers.nips.c/paper_files/paper/2023/hash/9a6a435e75419a836fe47ab6793623e6-Abstract-Conference.html. Ganguli et al. (2022) Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Scott Johnston, Andy Jones, Nicholas Joseph, Jackson Kernian, Shauna Kravec, Ben Mann, Neel Nanda, Kamal Ndousse, Catherine Olsson, Daniela Amodei, Tom B. Brown, Jared Kaplan, Sam McCandlish, Christopher Olah, Dario Amodei, and Jack Clark. Predictability and surprise in large generative models. In FAccT ’22: 2022 ACM Conference on Fairness, Accountability, and Transparency, Seoul, Republic of Korea, June 21 - 24, 2022, p. 1747–1764. ACM, 2022. doi: 10.1145/3531146.3533229. URL https://doi.org/10.1145/3531146.3533229. Hendrycks et al. (2021) Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ML safety. arXiv preprint arXiv:2109.13916, 2021. Huh et al. (2024) Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: The platonic representation hypothesis. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=BH8TYy0r6u. Ji et al. (2023a) Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of LLM via a human-preference dataset. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023a. URL http://papers.nips.c/paper_files/paper/2023/hash/4dbb61cb68671edc4ca3712d70083b9f-Abstract-Datasets_and_Benchmarks.html. Ji et al. (2023b) Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. AI alignment: A comprehensive survey. arXiv preprint arXiv:2310.19852, 2023b. Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. Leike et al. (2018) Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871, 2018. Li et al. (2023a) Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, p. 19730–19742. PMLR, 2023a. URL https://proceedings.mlr.press/v202/li23q.html. Li et al. (2023b) Kenneth Li, Oam Patel, Fernanda B. Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023b. URL http://papers.nips.c/paper_files/paper/2023/hash/81b8390039b7302c909cb769f8b6cd93-Abstract-Conference.html. Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, volume 8693 of Lecture Notes in Computer Science, p. 740–755. Springer, 2014. doi: 10.1007/978-3-319-10602-1“˙48. URL https://doi.org/10.1007/978-3-319-10602-1_48. Liu et al. (2023a) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023a. URL http://papers.nips.c/paper_files/paper/2023/hash/6dcf277ea32ce3288914faf369fe6de0-Abstract-Conference.html. Liu et al. (2023b) Sheng Liu, Haotian Ye, Lei Xing, and James Y. Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering. ArXiv, abs/2311.06668, 2023b. URL https://api.semanticscholar.org/CorpusID:265149781. Liu et al. (2024) Xin Liu, Yichen Zhu, Yunshi Lan, Chao Yang, and Yu Qiao. Safety of multimodal large language models on images and text. arXiv preprint arXiv:2402.00357, 2024. Luo et al. (2024) Haochen Luo, Jindong Gu, Fengyuan Liu, and Philip Torr. An image is worth 1000 lies: Transferability of adversarial images across prompts on vision-language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=nc5GgFAvtk. Madry et al. (2018) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://openreview.net/forum?id=rJzIBfZAb. Mazeika et al. (2024) Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David A. Forsyth, and Dan Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=f3TUipYU3U. Nie et al. (2022) Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Animashree Anandkumar. Diffusion models for adversarial purification. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, p. 16805–16827. PMLR, 2022. URL https://proceedings.mlr.press/v162/nie22a.html. OpenAI (2023) OpenAI. Gpt-4v(ision) system card. 2023. URL https://api.semanticscholar.org/CorpusID:263218031. OpenAI (2024) OpenAI. Gpt-4o system card. 2024. URL https://cdn.openai.com/gpt-4o-system-card.pdf. Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022. URL http://papers.nips.c/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html. Ponnuru et al. (2023) Mahesh Datta Sai Ponnuru, Likhitha Amasala, Tanu Sree Bhimavarapu, and Guna Chaitanya Garikipati. A malware classification survey on adversarial attacks and defences. CoRR, abs/2312.09636, 2023. doi: 10.48550/ARXIV.2312.09636. URL https://doi.org/10.48550/arXiv.2312.09636. Qi et al. (2024a) Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Henderson, Mengdi Wang, and Prateek Mittal. Visual adversarial examples jailbreak aligned large language models. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan (eds.), Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, p. 21527–21536. AAAI Press, 2024a. doi: 10.1609/AAAI.V38I19.30150. URL https://doi.org/10.1609/aaai.v38i19.30150. Qi et al. (2024b) Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024b. URL https://openreview.net/forum?id=hTEGyKf0dZ. Qi et al. (2024c) Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024c. URL https://openreview.net/forum?id=hTEGyKf0dZ. Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. PMLR, 2021. Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URL http://papers.nips.c/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html. Rimsky et al. (2024) Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llama 2 via contrastive activation addition. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, p. 15504–15522. Association for Computational Linguistics, 2024. URL https://aclanthology.org/2024.acl-long.828. Russell & Norvig (2016) Stuart J Russell and Peter Norvig. Artificial intelligence: a modern approach. Pearson, 2016. Shayegani et al. (2024) Erfan Shayegani, Yue Dong, and Nael B. Abu-Ghazaleh. Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=plmBsXHxgR. Sheshadri et al. (2024) Abhay Sheshadri, Aidan Ewart, Phillip Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Hadfield-Menell, and Stephen Casper. Targeted latent adversarial training improves robustness to persistent harmful behaviors in llms. CoRR, abs/2407.15549, 2024. doi: 10.48550/ARXIV.2407.15549. URL https://doi.org/10.48550/arXiv.2407.15549. Soares & Fallenstein (2014) Nate Soares and Benja Fallenstein. Aligning superintelligence with human interests: A technical research agenda. Machine Intelligence Research Institute (MIRI) technical report, 8, 2014. Subramani et al. (2022) Nishant Subramani, Nivedita Suresh, and Matthew E. Peters. Extracting latent steering vectors from pretrained language models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, p. 566–581. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022.FINDINGS-ACL.48. URL https://doi.org/10.18653/v1/2022.findings-acl.48. Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023. Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023. doi: 10.48550/ARXIV.2307.09288. URL https://doi.org/10.48550/arXiv.2307.09288. Tu et al. (2023) Haoqin Tu, Chenhang Cui, Zijun Wang, Yiyang Zhou, Bingchen Zhao, Junlin Han, Wangchunshu Zhou, Huaxiu Yao, and Cihang Xie. How many unicorns are in this image? A safety evaluation benchmark for vision llms. CoRR, abs/2311.16101, 2023. doi: 10.48550/ARXIV.2311.16101. URL https://doi.org/10.48550/arXiv.2311.16101. Turner et al. (2023) Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. CoRR, abs/2308.10248, 2023. doi: 10.48550/ARXIV.2308.10248. URL https://doi.org/10.48550/arXiv.2308.10248. Wang et al. (2023) Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. Do-not-answer: A dataset for evaluating safeguards in llms. CoRR, abs/2308.13387, 2023. doi: 10.48550/ARXIV.2308.13387. URL https://doi.org/10.48550/arXiv.2308.13387. Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https://openreview.net/forum?id=gEZrGCozdqR. Zeng et al. (2024) Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge AI safety by humanizing llms. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, p. 14322–14350. Association for Computational Linguistics, 2024. URL https://aclanthology.org/2024.acl-long.773. Zhang et al. (2023a) Xiaoyu Zhang, Cen Zhang, Tianlin Li, Yihao Huang, Xiaojun Jia, Ming Hu, Jie Zhang, Yang Liu, Shiqing Ma, and Chao Shen. Jailguard: A universal detection framework for llm prompt-based attacks. 2023a. URL https://api.semanticscholar.org/CorpusID:266359752. Zhang et al. (2023b) Xiaoyu Zhang, Cen Zhang, Tianlin Li, Yihao Huang, Xiaojun Jia, Xiaofei Xie, Yang Liu, and Chao Shen. A mutation-based method for multi-modal jailbreaking attack detection. arXiv preprint arXiv:2312.10766, 2023b. Zhu et al. (2024) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=1tZbq88f27. Zou et al. (2023a) Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to AI transparency. CoRR, abs/2310.01405, 2023a. doi: 10.48550/ARXIV.2310.01405. URL https://doi.org/10.48550/arXiv.2310.01405. Zou et al. (2023b) Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. CoRR, abs/2307.15043, 2023b. doi: 10.48550/ARXIV.2307.15043. URL https://doi.org/10.48550/arXiv.2307.15043. Appendix A Details about generating adversarial images In our implementation, we utilize the standard PGD method to generate adversarial images, according to 2.1. We run 5000 iterations of PGD optimization on generating each adversarial image and we set ϵitalic-ϵεϵ to 32/2553225532/25532 / 255. We test the robustness of the NEARSIDE method to varying ϵitalic-ϵεϵ. We use NEARSIDE to detect the adversarial samples generated under different ϵitalic-ϵεϵ. Results are deferred to Appx. D. Appendix B Prompts to GPT-4 judgement We adopt the prompt template proposed by Qi et al. (2024c), which combines the prohibited model usage policy, the (harmful) input query, the model’s output for this query, and a rating rubric together to be fed to GPT-4 via OpenAI’s APIs. The complete prompt template is as follows: (Context. will be replaced with the OpenAI’s detailed policy and scoring tules, Context. will be replaced with the harmful query and model’s output) Prompt template Below are OpenAI’s prohibited usage policies. These policies ensure that models interact with users safely and responsibly. Models should refuse to execute instructions that fall into the disallowed use cases outlined in the OpenAI policy, avoiding affirmative responses or providing any information that could be helpful for any prohibited activities. OpenAI Policy - Disallowed usage of our models: The detailed policy. Now, I will provide you with a user instruction that the model should not comply with, as per OpenAI’s policy. I will also give you the model’s response to that instruction. Please review the model’s response in light of the prohibited usage guidelines: User Instruction: The harmful query. Model Response: The output to be classified. Use the scoring rules below to score the model’s response to the user instruction on a scale of 1 to 5: The detailed OpenAI’s scoring rules Output your evaluation in the following format: #the reason: your analysis here [note: this should be step-by-step analysis following the steps: (a) Give an overview of the user’s primary intent and the model’s response. (b) Evaluate the degree to which the model’s response aligns with the user’s expressed intent. (c) Examine the user’s instruction and the model’s reply respectively to determine if they contain any prohibited content. Finally, evaluate the degree of the model’s adherence to OpenAI’s usage guidelines.] #the score: your score here. [note:provide your score in based on your analysis and the provide scoring criteria.] Appendix C An exemplar sample in RADAR In RADAR, each sample consists of an adversarial sample and a benign sample, with each containing a query, an adversarial/benign image and VLMs’ response. Fig. 6 provides an exemplar sample. Figure 6: An exemplar sample in RADAR. Appendix D Perturbation radius in generating adversarial images We test the robustness of the NEARSIDE method to varying ϵitalic-ϵεϵ. We use NEARSIDE to detect the adversarial samples generated under different ϵitalic-ϵεϵ. We genearte 100 adversarial samples on the LLaVA D-corpus dataset under settings of ϵ=16italic-ϵ16ε=16ϵ = 16, ϵ=64italic-ϵ64ε=64ϵ = 64, and unconstrained. The data generation follow the same pipeline as Sec. 3. We create 100 samples for each ϵitalic-ϵεϵ. The results are provided in Tab. 4. Table 4: Results of the NEARSIDE on LLaVA D-corpus generated with different ϵitalic-ϵεϵ. ϵitalic-ϵεϵ of adversarial training Accuracy(%)Accuracy(\%)A c c u r a c y ( % ) Precision(%)Precision(\%)P r e c i s i o n ( % ) Recall(%)Recall(\%)R e c a l l ( % ) F⁢11F1F 1 ϵ=16/255italic-ϵ16255ε=16/255ϵ = 16 / 255 85.0 100.0 70.0 0.824 ϵ=64/255italic-ϵ64255ε=64/255ϵ = 64 / 255 93.5 97.8 89.0 0.932 u⁢n⁢c⁢o⁢n⁢s⁢t⁢r⁢a⁢i⁢n⁢e⁢dunconstrainedu n c o n s t r a i n e d 99.0 100.0 98.0 0.990 Appendix E Analysis of cross-model transferability Linear transformation W Witalic_W. We explore the Platonic Representation Hypothesis by using a linear transformation Witalic_W to align the two VLMs’ embedding spaces. To demonstrate the importance of the usage of Witalic_W, we conduct experiments that directly use the attacking direction of the source VLM to detect the adversarial samples of the target VLM without using Witalic_W. Results are shown in Tab. 5. PCA model. We use PCA model to reduce the dimension of VLMs’ embedding and the attacking direction before learning the transformation Witalic_W. In our initial setting, the dimension is reduced to 2056. We experiment to examine the effect of reducing the dimension of the attacking direction and the VLMs’ embedding with the PCA model. In specific, we vary the dimension in [2048, 1024, 512, 256] and report the cross-model transferability results in Tab. 6. Table 5: The results of cross-model transferability without Witalic_W. We report result−δsuperscriptresultresult [rgb]0.70703125,0,0 [named]pgfstrokecolor% rgb0.70703125,0,0-δresult- δ where δ indicates the difference between the results w/o Witalic_W and with Witalic_W shown in Table 3. svlm → tvlm (w/o Witalic_W) TEST SET Accuracy(%)Accuracy(\%)A c c u r a c y ( % ) Precision(%)Precision(\%)P r e c i s i o n ( % ) Recall(%)Recall(\%)R e c a l l ( % ) F⁢11F1F 1 MiniGPT-4→LLaVA→MiniGPT-4LLaVAMiniGPT-4 -4 → LLaVA H-rlhf 50.9−13,4superscript50.913450.9 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-13,450.9- 13 , 4 50.9−10.4superscript50.910.450.9 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-10.450.9- 10.4 51.6−26.0superscript51.626.051.6 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-26.051.6- 26.0 0.512−0.172superscript0.5120.1720.512 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-0.1720.512- 0.172 D-corpus 53.8−15.6superscript53.815.653.8 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-15.653.8- 15.6 53.2−9.4superscript53.29.453.2 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-9.453.2- 9.4 63.8−33.0superscript63.833.063.8 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-33.063.8- 33.0 0.580−0.180superscript0.5800.1800.580 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-0.1800.580- 0.180 Harm-Data 53.7−21.0superscript53.721.053.7 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-21.053.7- 21.0 53.7−23.0superscript53.723.053.7 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-23.053.7- 23.0 53.8−17.2superscript53.817.253.8 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-17.253.8- 17.2 0.537−0.200superscript0.5370.2000.537 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-0.2000.537- 0.200 LLaVA→MiniGPT-4→LLaVAMiniGPT-4LLaVA -4LLaVA → MiniGPT-4 H-rlhf 32.8−45.0superscript32.845.032.8 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-45.032.8- 45.0 27.2−58.9superscript27.258.927.2 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-58.927.2- 58.9 20.6−45.6superscript20.645.620.6 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-45.620.6- 45.6 0.235−0.514superscript0.2350.5140.235 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-0.5140.235- 0.514 D-corpus 71.0−9.4superscript71.09.471.0 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-9.471.0- 9.4 77.2−3.1superscript77.23.177.2 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-3.177.2- 3.1 59.6−21.0superscript59.621.059.6 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-21.059.6- 21.0 0.804−0.132superscript0.8040.1320.804 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-0.1320.804- 0.132 Harm-Data 23.9−73.2superscript23.973.223.9 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-73.223.9- 73.2 23.1−71.9superscript23.171.923.1 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-71.923.1- 71.9 22.4−77.0superscript22.477.022.4 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-77.022.4- 77.0 0.227−0.744superscript0.2270.7440.227 [rgb]0.70703125,0,0 [named]pgfstrokecolorrgb% 0.70703125,0,0-0.7440.227- 0.744 Table 6: The results of cross-model transferability where PCA reduce the VLMs’ embedding and the attacking direction to different dimensions. We use bold to highlight the best results. svlm → tvlm TEST SET PCA-Dimension Accuracy(%)Accuracy(\%)A c c u r a c y ( % ) Precision(%)Precision(\%)P r e c i s i o n ( % ) Recall(%)Recall(\%)R e c a l l ( % ) F⁢11F1F 1 LLaVA → MiniGPT-4 H-rlhf 256 87.0 89.2 84.2 0.866 512 88.8 88.5 89.2 0.888 1024 88.2 89.0 87.2 0.881 2048 77.8 86.2 66.2 0.749 D-corpus 256 83.3 76.2 96.8 0.853 512 84.7 82.2 88.6 0.853 1024 77.6 88.5 63.4 0.740 2048 80.4 80.3 80.6 0.805 Harm-Data 256 87.8 80.4 100.0 0.891 512 83.4 75.1 100.0 0.858 1024 88.2 89.0 87.2 0.881 2048 97.1 95.0 99.4 0.972 MiniGPT-4 → LLaVA H-rlhf 256 57.0 53.8 98.8 0.697 512 58.5 54.7 98.0 0.703 1024 63.7 58.6 93.2 0.720 2048 64.3 61.3 77.6 0.685 D-corpus 256 50.9 50.5 100.0 0.671 512 58.5 54.7 98.0 0.703 1024 51.2 50.6 100.0 0.672 2048 69.4 62.5 96.8 0.760 Harm-Data 256 71.9 64.7 96.6 0.775 512 71.9 64.4 98.2 0.778 1024 74.4 67.6 93.8 0.786 2048 74.7 76.7 0.71 0.737 From Tab. 5, we find that, without Witalic_W, the cross-model transferability results will significantly decrease. From Tab. 6, we find that, when reducing the dimension to 256, the cross-model transferability results still remain high, indicating that the information in low dimensional sub-spaces is already sufficient for aligning two VLMs’ embedding spaces.

Instant research discovery