Paper deep dive
Dual Adversarial Fine-tuning for Enhancing Robustness of Large Vision Language Model
Sibo Wang, Jie Zhang, Shiguang Shan, Xilin Chen, Wen Gao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/22/2026, 3:18:29 AM
Summary
The paper proposes Dual Adversarial Fine-tuning (DAFT), a framework designed to enhance the adversarial robustness of Large Vision-Language Models (LVLMs) like LLaVA and GPT-4V. DAFT addresses the limitations of existing defense methods that lack generalizability across multimodal tasks by jointly optimizing two supervision signals: a Visual supervision branch that preserves pre-trained knowledge using clean image features, and a Semantic supervision branch that uses caption-image alignment to maintain semantic coherence under attack. The method replaces the CLIP vision encoder in LVLMs without requiring task-specific retraining, demonstrating superior performance in zero-shot classification, image captioning, and Visual Question Answering (VQA) tasks compared to state-of-the-art methods like TeCoA and FARE.
Entities (12)
Relation Signals (10)
DAFT → consistsof → Visual Supervision Branch
confidence 96% · The proposed framework comprises two core components, i.e., Visual supervision branch and Semantic supervision branch.
DAFT → consistsof → Semantic Supervision Branch
confidence 96% · The proposed framework comprises two core components, i.e., Visual supervision branch and Semantic supervision branch.
DAFT → improves → Adversarial Robustness
confidence 95% · DAFT ... enhancing model robustness while generalizing across multiple downstream tasks.
CLIP → isbackboneof → LLaVa
confidence 95% · LLaVA [2, 8] integrates CLIP’s vision encoder with the large language model Vicuna
Semantic Supervision Branch → uses → Caption-Image Alignment
confidence 94% · the latter incorporates caption-image alignment as a contextual signal to preserve semantic coherence under attack
Visual Supervision Branch → uses → Clean Image Features
confidence 94% · The former branch leverages features from clean images, extracted via a frozen original vision encoder
DAFT → replaces → CLIP Vision Encoder
confidence 93% · our method achieves cross-task robustness by simply replacing the CLIP vision encoder in the original model
LLaVa → isvulnerableto →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:While Large Vision-Language Models (LVLMs), represented by LLaVA and GPT-4V, have demonstrated remarkable capabilities, their visual inputs remain vulnerable to adversarial attacks, posing significant security risks. Existing defense methods predominantly target single-task scenarios (e.g., zero-shot classification) and consequently lack generalizability across various multimodal tasks. To address this limitation, we propose a dual adversarial fine-tuning framework that jointly optimizes visual and semantic supervision signals from two modalities, enhancing model robustness while generalizing across multiple downstream tasks. The proposed framework comprises two core components, i.e., $\textbf{Visual}$ supervision branch and $\textbf{Semantic}$ supervision branch. The former branch leverages features from clean images, extracted via a frozen original vision encoder, to guide adversarial robustness while the latter incorporates caption-image alignment as a contextual signal to preserve semantic coherence under attack. Moreover, our method achieves cross-task robustness by simply replacing the CLIP vision encoder in the original model, with no need of separate task-specific retraining or architecture this http URL experiments demonstrate that our approach outperforms the state-of-the-art method in adversarial robustness evaluation across zero-shot classification, image captioning, and visual question answering (VQA) tasks.
Tags
Links
- Source: https://arxiv.org/abs/2607.18958v1
- Canonical: https://arxiv.org/abs/2607.18958v1
Trouble viewing inline? Open PDF directly →
Full Text
78,293 characters extracted from source content.
Expand or collapse full text
Dual Adversarial Fine-tuning for Enhancing Robustness of Large Vision Language Model Sibo Wang, Jie Zhang, Shiguang Shan, Xilin Chen, Wen Gao Abstract While Large Vision-Language Models (LVLMs), represented by LLaVA and GPT-4V, have demonstrated remarkable capabilities, their visual inputs remain vulnerable to adversarial attacks, posing significant security risks. Existing defense methods predominantly target single-task scenarios (e.g., zero-shot classification) and consequently lack generalizability across various multimodal tasks. To address this limitation, we propose a dual adversarial fine-tuning framework that jointly optimizes visual and semantic supervision signals from two modalities, enhancing model robustness while generalizing across multiple downstream tasks. The proposed framework comprises two core components, i.e., Visual supervision branch and Semantic supervision branch. The former branch leverages features from clean images, extracted via a frozen original vision encoder, to guide adversarial robustness while the latter incorporates caption-image alignment as a contextual signal to preserve semantic coherence under attack. Moreover, our method achieves cross-task robustness by simply replacing the CLIP vision encoder in the original model, with no need of separate task-specific retraining or architecture modifications. Extensive experiments demonstrate that our approach outperforms the state-of-the-art method in adversarial robustness evaluation across zero-shot classification, image captioning, and visual question answering (VQA) tasks. I Introduction Recent advances in Large Vision-Language Models (LVLMs) have achieved significant progress in understanding and reasoning, with performance occasionally rivaling or surpassing human capabilities in specific tasks [1, 2, 3]. These models now serve as foundational tools for diverse applications, including visual question answering, image captioning, and cross-modal retrieval [4, 5, 6]. A significant milestone in this area is visual-language alignment frameworks such as CLIP [7], which leverages contrastive learning to map text and image into a unified feature space. By minimizing the distance between semantically related pairs across modalities, CLIP achieves remarkable zero-shot generalization, enabling seamless adaptation to numerous downstream tasks without task-specific fine-tuning. Furthermore, CLIP serves as a foundational pillar for various Large Vision-Language Models (LVLMs). For example, LLaVA [2, 8] integrates CLIP’s vision encoder with the large language model Vicuna [9], enabling joint processing of visual and textual data and yielding exceptional performance on diverse downstream tasks, including image captioning, visual question answering (VQA), and text-image retrieval. (a) Robust zero-shot classification accuracy on different datasets. (b) Robust performance on captioning and VQA tasks. Figure 1: Results under perturbation budget ϵ=2/255ε=2/255. (a) Adversarial accuracy of CLIP with different fine-tuning methods on zero-shot classification. (b) Robust captioning and VQA performance after replacing the original vision encoder of LLaVA with different CLIP vision encoders. Despite their capabilities, LVLMs remain inherently vulnerable to visual adversarial attacks, potentially leading to erroneous outputs, even in closed-source models [10, 11, 12, 13]. These attacks introduce human-imperceptible noise to visual inputs, significantly degrading the model’s performance across tasks [14, 15, 16]. For example, previous studies have found that subtle perturbations can force LVLMs to generate predefined caption [11], or produce incorrect answers in visual question answering (VQA), highlighting systemic vulnerabilities in multimodal perception [17]. Since an increasing number of LVLMs are deployed in security-sensitive applications, improving their adversarial robustness has become a critical priority. Figure 2: Examples of robust evaluation on VQA (left) and image caption (right) tasks. We perform untargeted ℒ∞L_∞ attack with perturbation budget ϵε = 2/255 on LLaVA 1.5-7B and replace the original model with CLIP fine-tuned using different methods for inference. In VQA task, the responses from the original and TeCOA[18] trained models are almost unrelated to the image-question pair. The FARE[19] trained model responds the element “bus” from the image, but it is not the correct answer. Our DAFT model, however, provides the most accurate answer “Ferris wheel”. In image captioning, both the FARE[19] and TeCoA[18] mistakenly recognize the “crane” as a “giraffe”, while our method accurately provides the caption for this image. To mitigate security risks from adversarial examples, researchers have developed various defense strategies, among which adversarial training is regarded as one of the most effective defense mechanisms [16, 20, 21, 22]. Its core idea is to dynamically generate adversarial examples during the training process to enhance the model’s robustness against adversarial attacks. However, in the context of large vision-language models (LVLMs), conducting adversarial training from scratch is impractical due to the excessive computational cost and data requirements. To address these challenges, recent work focuses on adversarial fine-tuning, which involves leveraging adversarial examples to fine-tune the vision encoder in LVLMs (e.g., CLIP’s vision encoder) and subsequently integrating it into LVLM to enhance overall robustness [18, 19, 23]. Among existing studies, TeCoA [18], as a representative work, performs supervised adversarial fine-tuning on ImageNet [24], combining adversarial image perturbations with category-aligned text prompts to improve CLIP’s zero-shot classification robustness. In contrast, FARE [19] adopts unsupervised adversarial fine-tuning, further enhancing the robustness generalization of CLIP’s vision encoder without labeled data. Critically, FARE allow LVLMs to inherit adversarial resilience simply by replacing their original vision encoder, eliminating the need for full-model retraining while improving adversarial robustness across various downstream tasks like image caption and visual question answering (VQA). Despite advancements in adversarial robustness for LVLMs, two critical challenges remain: 1) Category-Aligned Text Supervision Suffers from Overfitting and Poor Generalization. Existing methods usually rely on fixed text embeddings (e.g., ImageNet-1K class labels) for supervision, which optimizes robustness within predefined categories but neglects the impact of unseen classes. This narrow focus can lead to distortions in feature representations when models encounter tasks outside the training distribution. Moreover, class labels lack the semantic richness for complex multimodal tasks like image captioning and visual question answering (VQA), limiting their applicability to classification-centric scenarios. Consequently, the robustness improvements from label-supervised methods like TeCoA [18] remain confined to classification tasks. As shown in Fig. 1, label-supervised methods (e.g., TeCoA [18], PMG-AFT [23]) exhibit limited robustness, in zero-shot classification tasks on unseen datasets and fail to generalize to vision-language tasks like VQA. 2) Robustness Gains are Limited from Unsupervised Adversarial Fine-Tuning. Although unsupervised adversarial fine-tuning like FARE [19] improves robust generalization to some extent, its overall adversarial robustness remains limited. The most advanced adversarial attacks on LVLMs primarily exploit vulnerabilities in image-text alignment to force mispredictions [11, 25, 26]. Existing adversarial fine-tuning methods, however, focus narrowly on minimizing the feature discrepancies between clean and adversarial examples, making them ineffective against more sophisticated adversarial attacks that specifically target multimodal alignment mechanisms. As shown in Fig. 1a, unsupervised adversarial fine-tuning methods such as FARE [19] exhibit strong robust generalization in zero-shot classification tasks. However, when facing more advanced attacks targeting large language vision models (as illustrated in Fig. 1b), they falter, highlighting a critical gap in defense coverage compared to our method. Fig.2 further illustrates this limitation through concrete examples: while baseline methods struggle under adversarial conditions for vision-language tasks like VQA and image captioning, our method maintains consistent performance, underscoring its superior robustness. Inspired by recent studies highlighting the critical role of captions in LVLM training and robust vision-language adaptation [27, 28, 29], which demonstrate that rich textual captions surpass simplistic template prompts (e.g., “a photo of [class]”) in semantic expressiveness, we propose Dual Adversarial Fine-Tuning method (DAFT) to enhance the cross-task robustness in LVLMs through two synergistic branches. 1) Semantic supervision branch: Replacing traditional cross-entropy loss with a caption-guided contrastive loss. This aligns adversarial examples with their original descriptive captions while distancing them from unrelated captions in the batch. 2) Visual supervision branch: Preserving pre-trained knowledge by minimizing the feature distance between adversarial and clean examples via a frozen original vision encoder, thereby mitigating text-modality-induced overfitting. We evaluate DAFT extensively on zero-shot classification, image captioning, and VQA across multiple datasets. Experimental results demonstrate that DAFT outperforms the state-of-the-art method in adversarial robustness for all tasks, achieving superior generalization by unifying visual and semantic alignment under attack conditions. Our main contributions are summarized as follows: • We propose the DAFT, a dual adversarial fine-tuning framework that jointly optimizes visual and semantic supervision signals. It effectively mitigates overfitting while enhancing cross-task robustness in large vision-language models. • To our knowledge, DAFT is the first framework to integrate descriptive captions (instead of class labels) as semantic supervision for adversarial traning. By leveraging contrastive learning to align adversarial examples with their original captions, we demonstrate significant robustness improvements in multimodal tasks. • Extensive experiments demonstrate that DAFT consistently outperforms the state-of-the-art method in terms of adversarial robustness on zero-shot classification, image captioning, and VQA tasks. I Related Work Large Vision-Language Models. Large Vision-Language Models (LVLMs) are the product of the convergence of natural language processing and computer vision. Compared to traditional large language models (LLMs), LVLMs are capable of handling both visual and textual modalities simultaneously, offering broader application scenarios ranging from visual question answering to image-grounded dialogue and complex reasoning tasks. LVLMs, such as LLaVA-1.5 [1, 8] , BLIP-2 [30] , MiniGPT-4 [3], Otter [31], and InternLM-XComposer [32] , have demonstrated the ability to understand multimodal information and complete complex reasoning tasks. These models typically use a pre-trained vision encoder, such as CLIP, to transform images into vector representations (embeddings space), which are then aligned with text embeddings through various techniques. Some commercial LVLMs, such as Gemini [33] and GPT-4V[34], achieve human-expert performance in certain domains. While most prior works have focused on enhancing the reasoning capabilities of LVLMs, our research shifts attention toward improving their robustness against adversarial attacks and evaluating the generalizability of this robustness across diverse downstream tasks. Adversarial Robustness. Deep neural networks have been found to be vulnerable to adversarial attacks [35, 14, 36, 15, 37], where carefully crafted imperceptible noise is added to the original images, causing the model to misclassify. To enhance the robustness of neural networks against adversarial examples, a series of defense strategies have been proposed. Among them, adversarial training [16, 21, 38, 22] is considered the most effective defense mechanism. It dynamically generates adversarial examples during the training process and uses them to train the model, thereby improving the model’s defense capabilities on the task it is trained for. With the rise of large vision-language models (LVLMs), their robustness has gradually become a focus of research, and numerous attack algorithms targeting them have emerged [39, 25, 40, 10, 26, 41]. Since most vision-language models use CLIP’s vision encoder and its variants as components, existing defense methods generally focus on adversarial fine-tuning of CLIP’s vision encoder. Transferability of Robustness. Beyond improving robustness on training tasks, recent work begins to explore how adversarial robustness transfers to unseen tasks, such as zero-shot classification, captioning, and visual question answering. TeCoA [18] introduces category text as labels, replacing traditional one-hot supervision, and achieves better robustness in zero-shot classification tasks. PMG-AFT [23] further introduces an auxiliary branch to minimize the distance between adversarial outputs of the target and pre-trained models, thus improving zero-shot adversarial robustness and mitigating overfitting. FARE [19] employs unsupervised adversarial fine-tuning to bring robustness not only to classification but also to captioning and VQA. However, existing methods either tend to overfit to specific datasets or fail to fully capture the rich semantic relationships between images and texts. Our method introduces descriptive captions as semantic supervision and combines it with visual supervision, effectively addressing existing issues and enhancing the robustness transferability of large vision-language models. I Methodology In this section, we introduce our DAFT framework. In Section I-A, we provide the background on adversarial attacks, adversarial training, and adversarial robustness generalization. Section I-B provides a detailed introduction to our adversarial fine-tuning method, including its components and loss functions. I-A Preliminaries and Problem Setup Since most current LVLMs use CLIP as the backbone, our work mainly focuses on adversarial fine-tuning of CLIP’s vision encoder. It is worth noting that our method is also applicable to models with other backbones. For the sake of formalization, we also take the CLIP model as our target model. Let Fθ(⋅)F_θ(·) represent the CLIP image encoder parameterized by θ and Tϕ(⋅)T_φ(·) represent the CLIP text encoder parameterized by ϕφ. Given an input image x and a textual input denoted as t, the model will provide an image representation Fθ(x)F_θ(x) and a text representation Tϕ(t)T_φ(t) in a unified embedding space. The text t can either be the classification label of the image, such as “a photo of [class],” or a description about the image. For the classification task, CLIP calculates the cosine similarity between the image embedding and the embeddings of each category label, then selects the category with the highest similarity as the classification result, i.e., argmaxk=1,…,Kcos(Fθ(x),Tϕ(tk)) _k=1,…,K (F_θ(x),T_φ(t_k)), where tkt_k represents the text prompt for the k-th category label. Figure 3: The pipeline of DAFT. Visual supervision branch maximizes the similarity between adversarial feature on the target model and clean feature on the original model using ℒvisL_vis, ensuring generalization. Semantic supervision branch first uses the text encoder of the pre-trained CLIP model to encode a batch of N captions, and generates positive and negative samples based on the relationship between the embeddings of these captions and the adversarial feature. The caption corresponding to the adversarial sample is considered the positive sample PoPo, while the caption with the highest similarity to the adversarial sample feature among the remaining N−1N-1 captions is selected as the negative sample NeNe. Furthermore, contrastive loss ℒsemL_sem is used ensuring stronger robustness. Only the image encoder of the target model can be trained and the adversarial examples generation alternates with parameters updating. ⊙ means matrix inner product. Adversarial Attacks. Adversarial attacks typically refer to adding an imperceptible, optimizable perturbation to a clean image, misleading the model to produce an incorrect prediction or answer. A well-known white-box attack method is Projected Gradient Descent (PGD) [16], which uses multi-step gradient ascent steps to maximize the loss while projecting intermediate perturbation to the specified region constrained by the p-norm. The implementation of PGD varies depending on the task or the attack objective. For the classification task, given the one-hot label y , we first define a loss function: ℒadv=ℒce([cos(Fθ(x),Tϕ(t1)),…,cos(Fθ(x),Tϕ(tk))],y), _adv=L_ce([cos(F_θ(x),T_φ(t_1)),…,cos(F_θ(x),T_φ(t_k))],y), (1) where ℒceL_ce denotes the cross-entropy loss. For other more general scenarios, the loss function can be changed to ℒadv=ℒdistance(Fθ(x),target), _adv=L_distance(F_θ(x),target), (2) where ℒdistanceL_distance represents a certain metric such as ℒ2L_2 distance, and targettarget represents objective such as the embedding of the original image or the corresponding text representation. Different attack objectives can be achieved by modifying the implementation of the adversarial loss function. Then, the adversarial perturbation is iteratively optimized to maximize this loss within a bounded perturbation region, typically using the following update rule: xt+1=Projϵp(xt+α⋅sign(∇xℒadv)), x^t+1=Proj_ε^p(x^t+α·sign( _xL_adv)), (3) where x0=x+ε0x^0=x+ _0, ε0∼U[0,ε] _0 U[0, ], ProjϵpProj_ε^p denotes the projection to ϵε-ball under p-norm around the original image, α is the step size, ∇xℒadv _xL_adv is the gradient of the adversarial loss with respect to x. Adversarial Fine-Tuning. Adversarial training formulates the training process into a min-max problem and can be described as: minθx,y∼PD[maxxa∈B(x,ε)ℒadv(Fθ(xa),y)], _θE_x,y P_D [ _x_a∈ B(x, )L_adv (F_θ(x_a),y ) ], (4) where FθF_θ denotes the model parameterized by θ, ℒadvL_adv is the adversarial loss function mentioned above (e.g., cross-entropy) and xax_a is the adversarial example. The inner maximization problem aims to find a stronger adversarial example, while the outer minimization problem optimizes the model parameters so that the model FθF_θ still makes correct predictions on the adversarial examples. Since fine-tuning is generally used to adapt CLIP to downstream tasks, such an objective function (4) can also be applied to the fine-tuning of CLIP towards robustness, and we refer to it as adversarial fine-tuning. Adversarial Robustness Generalization. It is well known that LVLMs have excellent cross-task generalization capabilities. Whether the outstanding generalization ability of LVLMs across different tasks and datasets can remain consistent in the face of adversarial attacks is a question worth investigating. Adversarial fine-tuning a model for each task separately is an unscalable and costly approach. Therefore, we fine-tune the CLIP vision encoder only once and directly transfer and replace the original encoder of the LVLM (e.g., LLaVA [1]) with this encoder. We then test the adversarial test accuracy of LLaVA across various different tasks including classification, captioning and VQA to measure its robust generalization. I-B Dual Adversarial Fine-tuning To mitigate the overfitting phenomenon of adversarial fine-tuning and simultaneously improve robustness against multi-modal alignment attacks, we believe that all the generalizable features captured during pre-training, as well as the semantically rich supervision in adversarial fine-tuning, are especially valuable. The former helps maintain robustness in downstream tasks, while the latter is the key source of robustness. To achieve this, we propose Dual Adversarial Fine-tuning (DAFT), which utilizes dual supervision from both visual and semantic perspectives to achieve the objectives mentioned above. In DAFT, the visual supervision branch utilizes the clean example features extracted by the frozen original vision encoder as supervision signals to preserve features beneficial for generalization and mitigate overfitting. In semantic supervision branch, caption-image alignment is used as a semantic supervision signal, fully exploring and leveraging the rich semantic information in the captions. The pipeline of DAFT is shown in Fig. 3, and we will describe the design of DAFT in detail. Visual Supervision Branch. As previously analyzed, since adversarial examples are generated based on a specific dataset, the robustness features acquired by the target model are limited and may excessively overfit in a particular downstream task dataset, thus affecting the model’s generalization ability. Visual supervision branch aims to alleviate this issue. The main body of this branch is an image encoder of the original CLIP, denoted as Fori(⋅)F_ori(·). We denote Fori(x)F_ori(x) and Fθ(xa)F_θ(x_a) as the embeddings of the clean example passing through the original model ForiF_ori and the adversarial example passing through the target model FθF_θ, respectively. Since the pre-trained model is a fixed deterministic function, the supervision objective of this branch essentially encourages the target model Fθ(⋅)F_θ(·) to output features that can predict the information of the original foundational model as much as possible, thereby helping to mitigate the overfitting problem. This supervision branch is ultimately implemented using a loss based on ℒ2L_2 distance metric: ℒvis=‖Fori(x)−Fθ(xa)‖22, _vis=||F_ori(x)-F_θ(x_a)||_2^2, (5) adversarial examples xax_a are dynamically generated during fine-tuning, with the update process detailed in Equ. (8). Semantic Supervision Branch. The objective of semantic supervision is to ensure that adversarial examples xax_a remain semantically aligned with the text modality, thereby enhancing robustness against stronger LVLM adversarial attacks that aim to disrupt cross-modal alignment. Two key issues should be addressed: (1) the choice of textual supervision, and (2) the strategy for alignment. For issue 1, we introduce the use of descriptive captions as a novel form of text supervision. Unlike simple class labels (e.g., “a photo of [class]”), these captions offer richer semantic information, thereby enabling the generation of more effective adversarial examples. In addition, high-quality captions often include crucial contextual cues that are beneficial for tackling complex tasks such as Visual Question Answering (VQA). For issue 2, we adopt a contrastive learning framework to align adversarial example xax_a with the ground-truth caption (i.e., positive sample), while increasing the distance between xax_a and other captions in the same batch (i.e., negative samples), thus achieving a more semantically rich alignment. Specifically, we use triplet loss to implement contrastive learning. We take the embedding of the adversarial example xax_a, denoted as Fθ(xa)F_θ(x_a), as the anchor, and then use the frozen original CLIP text encoder Tϕ(⋅)T_φ(·) to encode a batch of N captions. One of these captions matches the clean image x associated with the adversarial sample xax_a, which serves as the positive sample, denoted as PoPo. The selection of negative examples can vary, under the triplet loss implementation, we choose the “hardest” sample from the batch as the negative example. The “hardest” sample is defined as the caption among the remaining N−1N-1 samples whose embedding is most similar to the embedding of the adversarial sample Fθ(xa)F_θ(x_a), which is denoted as NeNe. Finally, regarding the distance metric, we use cosine similarity, which is consistent with the original design of CLIP. The semantic supervision loss is implemented as follows: ℒsem=max(Fθ(xa)⋅Tϕ(Ne)‖Fθ(xa)‖‖Tϕ(Ne)‖−Fθ(xa)⋅Tϕ(Po)‖Fθ(xa)‖‖Tϕ(Po)‖+a, 0) splitL_sem= (& F_θ(x_a)· T_φ(Ne)\|F_θ(x_a)\|\|T_φ(Ne)\|\\ &- F_θ(x_a)· T_φ(Po)\|F_θ(x_a)\|\|T_φ(Po)\|+a,\ 0 ) split (6) where a is a hyper-parameter in the contrastive loss, used to control the learning difficulty and regulate the distance gap between the anchor and the positive/negative samples, and the generation of adversarial examples also follows the Equ. (8). Adversarial Attack and Model Parameter Updates. Based on the supervision from the two modalities mentioned above, the final loss function of our method is: ℒDAFT=ℒsem+γℒvis, _DAFT=L_sem+ _vis, (7) where γ is the hyper-parameter balancing coefficient between two terms. The process of adversarial fine-tuning involves a dynamic iteration between adversarial attacks and model parameter updates, with both alternating processes using the same objective function in Equ. (7), while in opposite optimization directions. Specifically, the optimization objective of the adversarial attack is: xa=argmax∥xa−x∥∞≤εℒDAFT. x_a= _ x_a-x _∞≤ L_DAFT. (8) We use the PGD algorithm [16], as described in Equ. (3), to optimize Equ. (8). After obtaining the adversarial sample xax_a, the model takes xax_a as input, and the update process is: θ=argminθ∑i=1nℒDAFT(θ,xai), θ= _θ _i=1^nL_DAFT(θ,x_a^i), (9) which is the specific implementation of Equ. (4). i represents the i-th sample, and n represents the total number of training samples. IV Experiments TABLE I: Clean and Robust accuracy of the CLIP model on various image classification datasets (%). The highest accuracy for each dataset under two attack budgets is highlighted in bold. The original CLIP model exhibits no robustness under adversarial attacks. On average robust accuracy across all datasets, the DAFT outperforms existing methods at both ϵε = 2/255 and ϵε = 4/255. In the last column, we report the training time required for each batch step of different methods (s/batch step). Perturbation Budget Method Caltech101 Cars CIFAR10 CIFAR100 DTD EuroSAT Fgvc Flowers102 OxfordPet STL10 ImageNet-r Average Time CLIP 82.7 78.6 95.7 72.8 55.7 63.1 32.7 79.6 92.5 99.5 87.4 76.4 - TeCOA [18] 70.3 20.3 60.0 23.4 18.2 17.5 7.2 23.2 63.3 89.5 51.1 40.4 166 PMG-AFT [23] 70.8 16.3 59.6 20.9 24.3 18.2 7.3 32.8 58.5 88.3 43.9 40.1 171 FARE [19] 80.8 61.9 73.7 41.2 42.0 10.1 24.4 55.8 85.7 93.4 72.6 58.3 159 clean [HTML]EFEFEFDAFT (ours) [HTML]EFEFEF78.8 [HTML]EFEFEF57.3 [HTML]EFEFEF75.1 [HTML]EFEFEF44.7 [HTML]EFEFEF43.6 [HTML]EFEFEF16.3 [HTML]EFEFEF24.1 [HTML]EFEFEF46.5 [HTML]EFEFEF82.1 [HTML]EFEFEF93.7 [HTML]EFEFEF69.7 [HTML]EFEFEF57.4 162 CLIP 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 - TeCOA [18] 58.8 7.9 38.0 12.4 11.3 11.1 2.7 9.2 38.3 73.3 32.9 26.9 166 PMG-AFT [23] 57.3 6.1 37.9 11.3 15.1 11.3 3.4 16.6 33.7 72.3 26.3 26.5 171 FARE [19] 70.1 23.5 49.9 23.4 22.3 3.7 8.4 27.1 60.3 79.7 47.0 37.7 159 ϵε=2/255 [HTML]EFEFEFDAFT (ours) [HTML]EFEFEF69.8 [HTML]EFEFEF22.6 [HTML]EFEFEF53.4 [HTML]EFEFEF25.8 [HTML]EFEFEF24.7 [HTML]EFEFEF9.9 [HTML]EFEFEF9.1 [HTML]EFEFEF24.6 [HTML]EFEFEF61.7 [HTML]EFEFEF79.6 [HTML]EFEFEF47.2 [HTML]EFEFEF38.9 162 CLIP 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 - TeCOA [18] 46.2 3.5 21.0 6.7 6.4 10.5 0.8 3.0 14.8 50.6 18.0 16.5 166 PMG-AFT [23] 43.9 2.5 21.7 6.2 10.6 11.4 1.0 7.4 14.6 49.6 16.1 16.8 171 FARE [19] 56.0 4.5 26.7 12.8 13.9 6.8 2.4 4.3 26.4 55.9 27.9 21.6 159 ϵε=4/255 [HTML]EFEFEFDAFT (ours) [HTML]EFEFEF56.4 [HTML]EFEFEF7.1 [HTML]EFEFEF28.1 [HTML]EFEFEF13.2 [HTML]EFEFEF12.4 [HTML]EFEFEF4.4 [HTML]EFEFEF2.8 [HTML]EFEFEF7.9 [HTML]EFEFEF32.0 [HTML]EFEFEF59.8 [HTML]EFEFEF29.4 [HTML]EFEFEF23.0 162 IV-A Experimental Setup Models and Tasks. We adopt the LLaVA [1] framework to explore robustness of large vision-language models, which uses CLIP ViT-L-14 [7] as its vision encoder. In zero-shot classification tasks, inference can be performed using only the alignment mechanism of CLIP itself as introduced in Sec. I-A. However, for tasks like image captioning and VQA, we replace the original encoder in LLaVA with a trained robust vision encoder for downstream task inference. Datasets and Metrics. We fine-tune the CLIP model on the COCO-caption [42] dataset and evaluate it on a wide range of downstream tasks across various datasets. For zero-shot classification tasks, we select 11 datasets: Caltech101 [43], StanfordCars [44], CIFAR10 [45], CIFAR100 [45], DTD [46], EuroSAT [47], FGVC Aircrafts [48], Flowers [49], OxfordPets [50], STL-10 [51], and ImageNet-R [52]. For image captioning, we use the COCO [42] and Flickr30k [53] datasets. For visual question answering, we use the VQAv2 [54], TextVQA [55], VizWiz [56], and OKVQA [57] datasets. For zero-shot classification and VQA tasks, we evaluate the adversarial sample classification accuracy and answer accuracy, respectively. For the image captioning task, we report the CIDEr scores [58] between the responses to adversarial examples and the ground-truth captions. Due to computational resource limitations, we use a relatively smaller COCO-caption [42] dataset for training. Although the scale of COCO [42] is smaller than ImageNet [24] or the larger LAION-400M [59] dataset, it has unique advantages. First, COCO images are more complex than those in ImageNet and contain richer textual information, which aids in image-text contrastive learning and enables the model to learn more robust features. Additionally, ImageNet currently lacks mature caption annotations, which further highlights COCO’s value. Second, the quality of annotations in the COCO dataset is relatively high and accurately reflects the content of the images, making it a more reliable choice compared to the broad, crowdsourced image-text matching datasets like LAION. To ensure fair comparison in our experiments, all competing methods (FARE [19], TeCOA [18]) are trained on the same COCO dataset. This is feasible since, apart from the caption annotations, COCO also contains category labels that can be adapted for other methods. As a result, the training and testing datasets are consistent across models, ensuring fairness throughout the experiments. Baseline. As mentioned in the related work, adversarial robustness in the LVLM field is still in the early stages of research. We mainly compare our method with category-label supervised methods such as TeCOA [18], PMG-AFT [23], and another unsupervised method, FARE [19], which is currently the state-of-the-art (SOTA) method. We also maintain the same settings as FARE [19] for the experiments. Implementation Details. Since our method uses caption data as text supervision, we select the popular COCO dataset as the training set, which also provides category labels, allowing it to support the training of methods like TeCoA [18]. Noted that, for fair comparison, all other methods compared in this paper are also trained on the COCO dataset. To remain consistent with LLaVA’s backbone, we adopt the CLIP model’s ViT-L/14 architecture as our target model. We use the AdamW [60] optimizer, with momentum coefficients β1 _1 and β2 _2 set to 0.9 and 0.95, respectively. The training utilizes a cosine decay learning rate (LR) schedule, with a linear warmup to the peak LR 1e-5. Weight decay is set to 1e-4, and the batch size is 64. During training, we use PGD-10 [16] attacks with ℒ∞L_∞ norm perturbation budget of ϵε = 4/255. For hyper-parameters, we set γ to 0.5 and a to 0.2. Each model in this paper is trained on two NVIDIA A100 GPUs. During evaluation, we use LLaVA1.5-7B [1] as the LVLM for our evaluation, and we apply different attack algorithms depending on the task, as detailed in Sec. IV-B and IV-C. IV-B Evaluation of Zero-Shot Classification Attack Implementation. We evaluate robust accuracy on 1000 randomly sampled instances from each dataset mentioned in Sec. IV-A. We use AutoAttack [37], consisting of APGD with cross-entropy loss and APGD with targeted DLR loss, to perform 100 iterations attacks. Models are tested under ℒ∞L_∞-bounded adversarial perturbations with budgets of ϵε = 2/255 and ϵε = 4/255. Since all models are trained exclusively on the COCO dataset, classification performance across various downstream datasets is evaluated in a zero-shot manner. Main Results. As shown in Tab. I, the original CLIP model without adversarial fine-tuning achieves superior clean accuracy but unsurprisingly exhibits no adversarial robustness. In contrast, Our DAFT outperforms existing methods in terms of robust accuracy across most datasets. Compared to category-label supervised methods, e.g., TeCoA [18] and PMG-AFT [23], DAFT improves average robust accuracy by approximately 12% and 6.5% under perturbation budgets of 2/255 and 4/255, respectively, while maintaining a significantly smaller degradation in clean accuracy. This aligns with our claim in Sec. I that such methods are prone to overfitting and lack generalization. In comparison to the unsupervised method FARE [19], our method improves average robustness gains of 1.2% and 1.4% under perturbation budgets of 2/255 and 4/255, respectively, suggesting that while FARE partially enhances generalization, it does not fully optimize robust performance. Notably, DAFT’s clean accuracy remains comparable to FARE, albeit marginally lower, a trade-off we attribute to the introduction of caption supervision. This may be due to the introduction of additional caption supervision, which increases the deviation between adversarial samples and clean samples, leading to a slight decline in clean accuracy. Overall, DAFT demonstrates superior robust generalization for zero-shot classification tasks, balancing adversarial robustness and clean accuracy more effectively than existing paradigms. IV-C Performance on Image Captioning and VQA TABLE I: Clean and Robust Performance of LLaVA with different vision encoders on different captioning and VQA datasets. For caption datasets Flickr30 and COCO, we report their CIDEr scores. For other four VQA datasets, we report their VQA accuracy. The highest accuracy for each dataset under two attack budgets is highlighted in bold. Overall, DAFT demonstrates the best robust performance compared to other methods while achieving comparable clean performance. Perturbation Budget Model Flickr30 COCO VQAv2 OKVQA VizWiz TextVQA Average-Caption Average-VQA LLaVA-CLIP 83.2 121.6 76.7 60.6 38.8 45.3 102.4 55.3 LLaVA-TeCOA [18] 38.6 90.9 61.4 51.2 42.5 17.0 64.7 43.0 LLaVA-PMG-AFT [23] 40.4 91.8 60.3 50.7 42.4 16.7 66.1 42.5 LLaVA-FARE [19] 57.7 107.8 67.6 55.9 42.6 26.4 82.7 48.1 clean [HTML]EFEFEFLLaVA-DAFT (ours) [HTML]EFEFEF56.3 [HTML]EFEFEF105.3 [HTML]EFEFEF66.7 [HTML]EFEFEF55.8 [HTML]EFEFEF42.4 [HTML]EFEFEF25.2 [HTML]EFEFEF80.8 [HTML]EFEFEF47.5 LLaVA-CLIP 3.9 7.9 7.7 1.6 0.1 0.8 5.9 2.5 LLaVA-TeCOA [18] 12.0 41.3 38.3 28.5 29.0 8.5 26.6 26.1 LLaVA-PMG-AFT [23] 11.9 40.6 36.6 29.1 29.2 8.7 26.3 25.9 LLaVA-FARE [19] 21.4 56.9 40.2 32.8 30.2 12.6 39.2 29.0 ϵε=2/255 [HTML]EFEFEFLLaVA-DAFT (ours) [HTML]EFEFEF23.0 [HTML]EFEFEF59.9 [HTML]EFEFEF41.4 [HTML]EFEFEF34.9 [HTML]EFEFEF31.4 [HTML]EFEFEF12.6 [HTML]EFEFEF41.5 [HTML]EFEFEF30.1 LLaVA-CLIP 2.1 4.1 1.5 0.1 0.1 0.0 3.1 0.4 LLaVA-TeCOA [18] 8.8 32.9 30.7 23.4 25.0 6.2 20.9 21.3 LLaVA-PMG-AFT [23] 8.9 31.1 29.7 23.1 24.8 5.9 20.0 20.9 LLaVA-FARE [19] 16.1 45.2 33.0 26.0 25.9 9.4 30.6 23.6 ϵε=4/255 [HTML]EFEFEFLLaVA-DAFT (ours) [HTML]EFEFEF17.3 [HTML]EFEFEF48.1 [HTML]EFEFEF34.5 [HTML]EFEFEF27.9 [HTML]EFEFEF27.0 [HTML]EFEFEF8.7 [HTML]EFEFEF32.7 [HTML]EFEFEF24.5 Attack Implementation. We evaluate the performance of LLaVA 1.5-7B under untargeted attack scenarios when using the original CLIP vision encoder versus robust CLIP vision encoders trained through different adversarial fine-tuning methods. Consistent with previous work [11, 19], we adopt a multi-stage attack ensemble strategy. For each dataset, we randomly select 1000 samples and perform ℒ∞L_∞-bounded attacks with two different perturbation budgets of ϵε = 2/255 and ϵε = 4/255. In the first stage, we apply half-precision (FP16) APGD attacks [37] for 100 iterations to all selected samples. We then retain only those samples for which the CIDEr score exceeds a certain threshold for image captioning tasks, or those that were not successfully attacked for VQA tasks, subjecting them to a second stage of single-precision (FP32) APGD attacks for 100 iterations. This attack strategy optimizes computational resources while enhancing attack effectiveness, as verified in [19]. Main Results. Tab. I presents the evaluation results of adversarial robustness across 6 datasets. LLaVA using the original CLIP achieves the best clean performance but is completely vulnerable to adversarial attacks. Among adversarially fine-tuning models, FARE [19] demonstrates the best clean performance, similar to its performance on zero-shot classification task. This improvement may come from explicitly aligning the robust CLIP model’s representations with the original CLIP, which limits the diversity of adversarial attacks and allows the adversarially fine-tuned model to predict clean sample outputs to the greatest extent. This also leads to FARE not achieving the best results under attacks targeting vision-language tasks, indirectly supporting the second challenge we raised in Sec. I: Robustness Gains are Limited from Unsupervised Adversarial Fine-Tuning. Moreover, category label-supervised methods like TeCoA [18] and PMG-AFT [23] underperform in both clean and adversarial scenarios. In contrast, DAFT significantly improves adversarial robustness with minimal clean-performance degradation, demonstrating an effective trade-off between clean and adversarial generalization. Specifically, under a perturbation budget of ϵε = 2/255, DAFT outperforms TeCoA in robustness across all 6 datasets, with an average increase of 14.9 in CIDEr scores and 4.0% improvement in average VQA accuracy. Compared to FARE, DAFT improves the average CIDEr score by 2.3 and the average VQA accuracy by 1.1%. Under stronger perturbations of ϵε = 4/255, DAFT improves the average caption CIDEr score by 11.8 and the average VQA accuracy by 3.2% compared to TeCoA. Compared to FARE, DAFT improves the average CIDEr score by 2.1 and the average VQA accuracy by 0.9%. IV-D Ablation Study Contribution of Loss Function Term. To demonstrate the effectiveness of the dual-supervision loss, we incrementally integrate individual loss components during training, evaluating them under attacks with a perturbation budget of ϵε = 2/255. The attack methods remain consistent with those in Sec. IV-B and IV-C, and we report their average robust performance across three tasks. Additionally, to highlight the superiority of captions over category label as supervision, we conduct an extra experiment where we replace the ℒsemL_sem with category label supervision as TeCOA [18] (shown in the first row of the Tab. I). As shown in Tab. I, models trained solely with visual supervision or semantic supervision exhibit suboptimal robustness. Among them, the performance of using semantic supervision alone decreases more significantly, suggesting that while captions offer some generalization ability, their isolated use still suffers from severe overfitting. On the other hand, using visual supervision alone similarly fails to maximize robustness, indicating that the two losses complement and promote each other, further underscoring the necessity of our dual adversarial fine-tuning framework. When replacing captions with category labels as supervision, the average zero-shot classification accuracy decreases by 12.4%, the average caption CIDEr score drops by 15.2, and the average VQA accuracy decreases by 4.2% (as illustrated in the first and last rows of Tab. I), highlighting the superiority of caption-guided semantic alignment over class-label constraints, validating our design rationale for DAFT. TABLE I: Contribution of each term in the loss function. We report robustness performance under a perturbation budget of ϵε=2/255 across three tasks. Noted that the first row in the table represents using category label supervision as TeCOA [18] to replace the semantic supervision ℒsemL_sem we propose. ℒsemL_sem ℒvisL_vis Avg-zeroshot-cls (%) Avg-caption (CIDEr) Avg-VQA (%) category-sup ✓ 26.5 26.3 25.9 ✓ 33.7 30.1 24.2 ✓ 37.7 39.2 29.0 ✓ ✓ (DAFT) 38.9 41.5 30.1 TABLE IV: The impact of different implementations of contrastive loss and sampling strategies in semantic supervision branch. We report robustness performance under a perturbation budget of ϵε=2/255 across three tasks. Sampling Strategy Avg-zeroshot-cls (%) Avg-caption (CIDEr) Avg-VQA (%) Triplet loss (DAFT) 38.9 41.5 30.1 Triplet loss (easiest sample) 0.2 6.7 3.4 InfoNCE loss 35.2 41.2 29.8 Quadruplet loss (second) 38.0 36.3 29.5 Quadruplet loss (random) 38.4 34.0 28.5 Impact of Negative Sampling Strategy. In contrastive learning, the positive and negative sample sampling strategy is a crucial factor that influences model performance. We implement the loss ℒsemL_sem using different sampling strategies for contrastive learning in semantic supervision branch. Since positive samples are inherently defined by ground-truth image-caption pairs, we focus on how negative sample selection impacts adversarial robustness. As shown in Tab. IV, we compare two types of triplet loss. The first type, as in DAFT, selects the caption from the N−1N-1 captions (excluding the positive sample) that is most similar to the adversarial features as the negative example. The second type selects the caption that is least similar to the adversarial features as the negative example (as shown in Tab. IV “easiest sample”). Strikingly, models trained with “easiest negative” sampling exhibit near-zero robustness, mirroring the original CLIP’s vulnerability. This aligns with expectation: optimizing toward minimally conflicting negatives prematurely saturates the contrastive objective, halting meaningful model adaptation. Additionally, we perform the InfoNCE loss to implement ℒsemL_sem. The key difference compared to triplet loss is that InfoNCE treats all N−1N-1 captions as negative examples and maximizes the distance between the adversarial sample and these captions. The model trained with InfoNCE loss is almost comparable to DAFT in zero-shot classification tasks but shows some decrease in robustness in image captioning and VQA tasks. To more thoroughly explore the adversarial example space, we extend the triplet loss to a quadruplet loss that incorporates two distinct negative examples. One negative example is fixed and consistent with DAFT: the caption that is most similar to the adversarial features from the N−1N-1 captions. The second negative example is implemented in two different ways: one selects randomly from the remaining N−2N-2 captions, and the other selects the caption that is most similar to the adversarial features from the remaining N−2N-2 captions (“second” in Tab. IV). Experimental results show that the model fine-tuned with quadruplet loss slightly reduces robustness in the image captioning, while performing similarly to DAFT in other tasks. This suggests that, although increasing the number of negative examples intuitively helps to fully explore the adversarial space and generate stronger attacks, the need to simultaneously increase the distance from these negative examples causes conflicts in the optimization direction. As a result, the model struggles to find the optimal solution, leading to a decrease in robustness. Overall, the negative sample sampling strategy that includes the caption most similar to the adversarial feature proves to be the optimal choice. As we have analyzed, such negative sample can be considered the “hardest” sample, which helps improve robustness. TABLE V: Average robustness performance across different tasks under a perturbation budget of ϵε=2/255 for various hyper-parameter combinations. Hyper-Parameters Avg-zeroshot-cls (%) Avg-caption (CIDEr) Avg-VQA (%) a=0.1 γ=0.5 38.8 40.1 29.9 a=0.2 γ=0.5 38.9 41.5 30.1 a=0.3 γ=0.5 38.4 41.2 29.2 a=0.5 γ=0.5 38.1 41.1 30.1 [HTML]EFEFEF a=0.2 γ=0.1 36.4 38.3 29.5 [HTML]EFEFEF a=0.2 γ=0.2 37.1 39.9 29.8 [HTML]EFEFEF a=0.2 γ=0.5 38.9 41.5 30.1 [HTML]EFEFEF a=0.2 γ=1 38.5 40.4 29.6 [HTML]EFEFEF a=0.2 γ=5 38.0 39.4 29.4 Effect of hyper-parameters γ and a. Our dual-supervision loss comprises two terms ℒsemL_sem and ℒvisL_vis, balanced by a hyper-parameter γ. Additionally, in the semantic loss ℒsemL_sem, a hyper-parameter a denotes the margin in the triplet loss. Since we use the cosine similarity, the range of a is between 0 and 1. To examine the impact of these two parameters on the model’s robustness, we train models by gradually varying each of these hyper-parameters. As shown in Tab. V, when γ is fixed at 0.5, as a gradually increases, the model’s overall robust performance first improves and then decreases. Since a mainly adjusts the difficulty of contrastive learning, as a increases, initially, increasing the difficulty of learning helps better align adversarial examples with the correct captions while pushing away negative captions. However, when a becomes too large, it makes the model difficult to optimize, leading it to fall into a local minimum. We find that the model’s robustness performance is the best when a=0.2. When a is fixed at 0.2, as γ gradually increases, the model’s overall robust performance also shows a trend of first improving and then decreasing. The model’s robustness is best when γ=0.5, which indicates that two terms in the loss function are of similar importance, further demonstrating the effectiveness of DAFT. Figure 4: Average robustness performance of models trained with DAFT under different batch sizes across various tasks. Effect of Batch Size. The effectiveness of ℒsemL_sem hinges on contrastive learning dynamics, where batch size governs the diversity and difficulty of negative samples. We conduct a systematic evaluation of DAFT across different batch sizes under a perturbation budget of ϵε=2/255, as shown in Fig. 4. The left vertical axis in the figure represents different meanings for different tasks: for zero-shot classification and VQA tasks, it represents the robust accuracy, while for the image captioning task, it denotes the CIDEr score. From the Fig. 4, we observe that as the batch size increases (from 16 to 128), the model’s robustness performance across tasks consistently improves and then decreases. The best robustness performance is achieved when the batch size is 64. The batch size primarily affects the selection of negative examples NeNe. When the batch size is large, the likelihood of encountering captions in the same batch with high similarity to the adversarial features increases, making the selected negative examples more similar to the adversarial features. Once this similarity is close to that between the adversarial and the true caption, it makes contrastive learning difficult to optimize. On the other hand, if the batch size is small, the selected negative examples might be too simple. As shown in Tab. IV, overly simple negative examples prevent contrastive learning from optimizing. Therefore, the batch size should neither be too large nor too small. We choose a batch size of 64 as the final implementation. V Limitations and Future Work This work primarily focuses on CLIP-based large vision-language models (LVLMs). Other LVLMs that do not rely on CLIP as the vision encoder may also benefit from our robustness-enhancing approach, but we leave the empirical validation of this to future work. Secondly, our method is a white-box defense approach, which is not directly applicable to currently closed-source models. In addition, our method only fine-tunes the visual encoder, and the defense of the language component in LVLMs is also left for future exploration. Finally, since COCO as a caption dataset is limited in scale, we plan to further improve the quality of caption data in future work to enhance the performance of our method. VI Conclusion In this work, we analyze two limitations in current adversarial defense research for large vision-language models (LVLMs): overfitting and insufficient alignment with text. To address these issues, we propose a dual-supervision adversarial fine-tuning framework, which introduces descriptive captions for semantic supervision, enabling the model to better handle advanced vision-language attacks, while preserving generalization through visual supervision. Our framework fine-tunes the vision encoder of the CLIP model, allowing it to replace the original vision encoder in LVLM and thus achieve robustness across various multimodal tasks. Extensive experiments demonstrate that our method exhibits strong adversarial robustness across different datasets for downstream tasks, while maintaining high clean performance. References [1] H. Liu, C. Li, Y. Li, and Y. J. Lee, “Improved baselines with visual instruction tuning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024, p. 26 296–26 306. [2] H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” in Adv. Neural Inf. Process. Syst., vol. 36, 2023, p. 34 892–34 916. [3] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “MiniGPT-4: Enhancing vision-language understanding with advanced large language models,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2024. [4] J. Chen, H. Guo, K. Yi, B. Li, and M. Elhoseiny, “VisualGPT: Data-efficient adaptation of pretrained language models for image captioning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2022, p. 18 030–18 040. [5] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language representation learning with noisy text supervision,” in Proc. Int. Conf. Mach. Learn. (ICML), 2021, p. 4904–4916. [6] H. Wang, C. Lai, and W. Ge, “Adapting multimodal large language models for video question answering by capturing question-critical and coherent moments,” IEEE Trans. Multimedia, vol. 27, p. 8737–8747, 2025. [7] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in Proc. Int. Conf. Mach. Learn. (ICML), 2021, p. 8748–8763. [8] B. Lin, Z. Tang, Y. Ye, J. Huang, J. Zhang, Y. Pang, P. Jin, M. Ning, J. Luo, and L. Yuan, “MoE-LLaVA: Mixture of experts for large vision-language models,” IEEE Trans. Multimedia, vol. 28, p. 4408–4419, 2026. [9] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez et al., “Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality,” https://lmsys.org/blog/2023-03-30-vicuna/, 2023, accessed: May 29, 2026. [10] Y. Zhao, T. Pang, C. Du, X. Yang, C. Li, N.-M. M. Cheung, and M. Lin, “On evaluating adversarial robustness of large vision-language models,” in Adv. Neural Inf. Process. Syst., vol. 36, 2023, p. 54 111–54 138. [11] C. Schlarmann and M. Hein, “On the adversarial robustness of multi-modal foundation models,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2023, p. 3677–3685. [12] N. Carlini, M. Nasr, C. A. Choquette-Choo, M. Jagielski, I. Gao, P. W. W. Koh, D. Ippolito, F. Tramer, and L. Schmidt, “Are aligned neural networks adversarially aligned?” in Adv. Neural Inf. Process. Syst., vol. 36, 2023, p. 61 478–61 500. [13] Y. Dong, H. Chen, J. Chen, Z. Fang, X. Yang, Y. Zhang, Y. Tian, H. Su, and J. Zhu, “How robust is Google’s Bard to adversarial image attacks?” arXiv preprint arXiv:2309.11751, 2023. [14] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in Proc. IEEE Symp. Security Privacy (SP), 2017, p. 39–57. [15] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2015. [16] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2018. [17] A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,” arXiv preprint arXiv:2307.15043, 2023. [18] C. Mao, S. Geng, J. Yang, X. Wang, and C. Vondrick, “Understanding zero-shot adversarial robustness for large-scale models,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2023. [19] C. Schlarmann, N. D. Singh, F. Croce, and M. Hein, “Robust CLIP: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models,” in Proc. Int. Conf. Mach. Learn. (ICML), 2024, p. 43 685–43 704. [20] T. Pang, X. Yang, Y. Dong, H. Su, and J. Zhu, “Bag of tricks for adversarial training,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2021. [21] A. Shafahi, M. Najibi, M. A. Ghiasi, Z. Xu, J. Dickerson, C. Studer, L. S. Davis, G. Taylor, and T. Goldstein, “Adversarial training for free!” in Adv. Neural Inf. Process. Syst., vol. 32, 2019. [22] H. Zhang, Y. Yu, J. Jiao, E. Xing, L. El Ghaoui, and M. Jordan, “Theoretically principled trade-off between robustness and accuracy,” in Proc. Int. Conf. Mach. Learn. (ICML), 2019, p. 7472–7482. [23] S. Wang, J. Zhang, Z. Yuan, and S. Shan, “Pre-trained model guided fine-tuning for zero-shot adversarial robustness,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024, p. 24 502–24 511. [24] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2009, p. 248–255. [25] J. Zhang, Q. Yi, and J. Sang, “Towards adversarial attack on vision-language pre-training models,” in Proc. ACM Int. Conf. Multimedia, 2022, p. 5005–5013. [26] Y. Wang, W. Hu, Y. Dong, H. Zhang, H. Su, and R. Hong, “Exploring transferability of multimodal adversarial samples for vision-language pre-training models with contrastive learning,” IEEE Trans. Multimedia, vol. 27, p. 6410–6421, 2025. [27] Y. Sun, J. Hao, K. Zhu, J.-J. Liu, Y. Zhao, X. Li, N. Zhao, Z. Li, and J. Wang, “Enhancing descriptive captions with visual attributes for multimodal perception,” arXiv preprint arXiv:2412.14233, 2024. [28] K. Saito, K. Sohn, X. Zhang, C.-L. Li, C.-Y. Lee, K. Saenko, and T. Pfister, “Prefix conditioning unifies language and label supervision,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2023, p. 2861–2870. [29] F. Waseda, S. Sugawara, and I. Echizen, “Quality text, robust vision: The role of language in enhancing visual robustness of vision-language models,” in Proc. ACM Int. Conf. Multimedia, 2025, p. 4808–4816. [30] J. Li, D. Li, S. Savarese, and S. Hoi, “BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” in Proc. Int. Conf. Mach. Learn. (ICML), 2023, p. 19 730–19 742. [31] B. Li, Y. Zhang, L. Chen, J. Wang, F. Pu, J. A. Cahyono, J. Yang, C. Li, and Z. Liu, “Otter: A multi-modal model with in-context instruction tuning,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 47, no. 9, p. 7543–7557, 2025. [32] P. Zhang, X. Dong, B. Wang, Y. Cao, C. Xu, L. Ouyang, Z. Zhao, H. Duan, S. Zhang, S. Ding et al., “InternLM-XComposer: A vision-language large model for advanced text-image comprehension and composition,” arXiv preprint arXiv:2309.15112, 2023. [33] Gemini Team, “Gemini: A family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805, 2023. [34] OpenAI, “GPT-4 technical report,” arXiv preprint arXiv:2303.08774, 2023. [35] A. Athalye, N. Carlini, and D. Wagner, “Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples,” in Proc. Int. Conf. Mach. Learn. (ICML), 2018, p. 274–283. [36] Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, and J. Li, “Boosting adversarial attacks with momentum,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2018, p. 9185–9193. [37] F. Croce and M. Hein, “Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks,” in Proc. Int. Conf. Mach. Learn. (ICML), 2020, p. 2206–2216. [38] D. Wu, S.-T. Xia, and Y. Wang, “Adversarial weight perturbation helps robust generalization,” in Adv. Neural Inf. Process. Syst., vol. 33, 2020, p. 2958–2969. [39] N. Inkawhich, G. McDonald, and R. Luley, “Adversarial attacks on foundational vision models,” arXiv preprint arXiv:2308.14597, 2023. [40] D. Lu, Z. Wang, T. Wang, W. Guan, H. Gao, and F. Zheng, “Set-level guidance attack: Boosting adversarial transferability of vision-language pre-training models,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2023, p. 102–111. [41] Z. Wang, C. Jiang, J. Gui, X. Shi, and Y. Y. Tang, “Revisiting the adversarial transferability:towards a perspective of semantic preservation,” IEEE Trans. Multimedia, p. 1–15, 2026. [42] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” in Proc. Eur. Conf. Comput. Vis. (ECCV), 2014, p. 740–755. [43] G. Griffin, A. Holub, and P. Perona, “Caltech-256 object category dataset,” California Inst. Technol., Tech. Rep. 7694, 2007. [44] J. Krause, M. Stark, J. Deng, and L. Fei-Fei, “3D object representations for fine-grained categorization,” in Proc. IEEE Int. Conf. Comput. Vis. Workshops (ICCVW), 2013, p. 554–561. [45] A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” Univ. Toronto, Tech. Rep., 2009. [46] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi, “Describing textures in the wild,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2014, p. 3606–3613. [47] P. Helber, B. Bischke, A. Dengel, and D. Borth, “EuroSAT: A novel dataset and deep learning benchmark for land use and land cover classification,” IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 12, no. 7, p. 2217–2226, 2019. [48] S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi, “Fine-grained visual classification of aircraft,” arXiv preprint arXiv:1306.5151, 2013. [49] M.-E. Nilsback and A. Zisserman, “Automated flower classification over a large number of classes,” in Proc. Indian Conf. Comput. Vis., Graph. Image Process., 2008, p. 722–729. [50] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar, “Cats and dogs,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2012, p. 3498–3505. [51] A. Coates, A. Ng, and H. Lee, “An analysis of single-layer networks in unsupervised feature learning,” in Proc. Int. Conf. Artif. Intell. Statist. (AISTATS), 2011, p. 215–223. [52] D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo et al., “The many faces of robustness: A critical analysis of out-of-distribution generalization,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2021, p. 8340–8349. [53] B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik, “Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2015, p. 2641–2649. [54] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Making the V in VQA matter: Elevating the role of image understanding in visual question answering,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2017, p. 6904–6913. [55] A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach, “Towards VQA models that can read,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2019, p. 8317–8326. [56] D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P. Bigham, “VizWiz grand challenge: Answering visual questions from blind people,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2018, p. 3608–3617. [57] K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi, “OK-VQA: A visual question answering benchmark requiring external knowledge,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2019, p. 3195–3204. [58] R. Vedantam, C. L. Zitnick, and D. Parikh, “CIDEr: Consensus-based image description evaluation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2015, p. 4566–4575. [59] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman et al., “LAION-5B: An open large-scale dataset for training next generation image-text models,” in Adv. Neural Inf. Process. Syst., vol. 35, 2022, p. 25 278–25 294. [60] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2019. [61] A. Awadalla, I. Gao, J. Gardner, J. Hessel, Y. Hanafy, W. Zhu, K. Marathe, Y. Bitton, S. Gadre, S. Sagawa et al., “OpenFlamingo: An open-source framework for training large autoregressive vision-language models,” arXiv preprint arXiv:2308.01390, 2023. Sibo Wang (Student Member, IEEE) received the B.S. degree from Harbin Institute of Technology in 2022. He is currently pursuing the Ph.D. degree from University of Chinese Academy of Sciences. His research interest includes adversarial example and model robustness. He has authored several academic papers in international conferences including CVPR/NIPS. Jie Zhang (Member, IEEE) received the Ph.D. degree from the University of Chinese Academy of Sciences (CAS), Beijing, China. He is currently an Associate Professor with the Institute of Computing Technology, CAS. His research interests include computer vision, pattern recognition, machine learning, particularly include face recognition, image segmentation, weakly/semi-supervised learning, and domain generalization. Shiguang Shan (Fellow, IEEE) received the Ph.D. degree in computer science from the Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS), Beijing, China, in 2004. He has been a Full Professor with ICT since 2010, where he is currently the Director of the Key Laboratory of Intelligent Information Processing, CAS. His research interests include signal processing, computer vision, pattern recognition, and machine learning. He has published more than 300 articles in related areas. He served as the General Co-Chair for IEEE Face and Gesture Recognition 2023, the General Co-Chair for Asian Conference on Computer Vision (ACCV) 2022, and the Area Chair of many international conferences, including CVPR, ICCV, AAAI, IJCAI, ACCV, ICPR, and FG. He was/is an Associate Editors of several journals, including IEEE Transactions on Image Processing, Neurocomputing, CVIU, and PRL. He was a recipient of the China’s State Natural Science Award in 2015 and the China’s State S&T Progress Award in 2005 for his research work. Xilin Chen (Fellow, IEEE) is currently a Professor with the Institute of Computing Technology, Chinese Academy of Sciences (CAS). He has authored one book and more than 400 articles in refereed journals and proceedings in the areas of computer vision, pattern recognition, image processing, and multimodal interfaces. He is a fellow of the ACM, IAPR, and CCF. He is also an Information Sciences Editorial Board Member of Fundamental Research, an Editorial Board Member of Research, a Senior Editor of the Journal of Visual Communication and Image Representation, and an Associate Editor-in-Chief of the Chinese Journal of Computers and Chinese Journal of Pattern Recognition and Artificial Intelligence. He served as an organizing committee member for multiple conferences, including the General Co-Chair of FG 2013/FG 2018, VCIP 2022, the Program Co-Chair of ICMI 2010/FG 2024, and an Area Chair of ICCV/CVPR/ECCV/NeurIPS for more than ten times. Wen Gao (Fellow, IEEE) received the Ph.D. degree in electronics engineering from The University of Tokyo, Japan, in 1991. He is currently a Boya Chair Professor in computer science at Peking University. He is also the Director of the Peng Cheng Laboratory, Shenzhen. Before joining Peking University, he was a Professor with the Harbin Institute of Technology, from 1991 to 1995. From 1996 to 2006, he was a Professor with the Institute of Computing Technology, Chinese Academy of Sciences. He has published extensively, including five books and over 1000 technical articles in refereed journals and conference proceedings in the areas of image processing, video coding and communication, computer vision, multimedia retrieval, multimodal interface, and bioinformatics. He served on the editorial boards for several journals, such as ACM CSUR, IEEE Transactions on Image Processing (TIP), IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), and IEEE Transactions on Multimedia (TMM). He served on the advisory and technical committees for professional organizations. He was the Vice President of the National Natural Science Foundation (NSFC) of China, from 2013 to 2018, and the President of China Computer Federation (CCF), from 2016 to 2020. He is also the Deputy Director of China National Standardization Technical Committees. He is an Academician of the Chinese Academy of Engineering and a fellow of ACM. Supplementary Material I Additional Results under Stronger Attacks To further evaluate the robustness of different vision encoders under more challenging adversarial perturbations, we provide additional results with a larger perturbation budget of ϵε = 8/255 in Tab. VI. The results show that the original CLIP-based LLaVA almost completely fails under such strong attacks, indicating its severe vulnerability to adversarial perturbations. Among adversarially fine-tuned baselines, FARE [19] still maintains relatively strong captioning performance, while TeCoA [18] and PMG-AFT [23] achieve competitive results on some VQA datasets. However, DAFT consistently achieves the best overall robustness under this stronger attack setting. Specifically, compared with TeCoA, DAFT improves the average caption CIDEr score by 8.0 and the average VQA accuracy by 0.6%. Compared with FARE, DAFT further improves the average caption CIDEr score by 1.8 and the average VQA accuracy by 1.4%. Although TeCoA obtains the best result on VizWiz, DAFT achieves the highest performance on most datasets and obtains the best average performance across both captioning and VQA tasks. These results further demonstrate that DAFT provides stronger adversarial generalization under more severe perturbations, complementing the main results in Sec. IV-C. TABLE VI: Clean and Robust Performance of LLaVA with different vision encoders on different captioning and VQA datasets. For caption datasets Flickr30 and COCO, we report their CIDEr scores. For other four VQA datasets, we report their VQA accuracy. The highest accuracy for each dataset under different attack budgets is highlighted in bold. Overall, DAFT demonstrates the best robust performance compared to other methods while achieving comparable clean performance. Perturbation Budget Model Flickr30 COCO VQAv2 OKVQA VizWiz TextVQA Average-Caption Average-VQA LLaVA-CLIP 83.2 121.6 76.7 60.6 38.8 45.3 102.4 55.3 LLaVA-TeCOA [18] 38.6 90.9 61.4 51.2 42.5 17.0 64.7 43.0 LLaVA-PMG-AFT [23] 40.4 91.8 60.3 50.7 42.4 16.7 66.1 42.5 LLaVA-FARE [19] 57.7 107.8 67.6 55.9 42.6 26.4 82.7 48.1 clean [HTML]EFEFEFLLaVA-DAFT (ours) [HTML]EFEFEF56.3 [HTML]EFEFEF105.3 [HTML]EFEFEF66.7 [HTML]EFEFEF55.8 [HTML]EFEFEF42.4 [HTML]EFEFEF25.2 [HTML]EFEFEF80.8 [HTML]EFEFEF47.5 LLaVA-CLIP 3.9 7.9 7.7 1.6 0.1 0.8 5.9 2.5 LLaVA-TeCOA [18] 12.0 41.3 38.3 28.5 29.0 8.5 26.6 26.1 LLaVA-PMG-AFT [23] 11.9 40.6 36.6 29.1 29.2 8.7 26.3 25.9 LLaVA-FARE [19] 21.4 56.9 40.2 32.8 30.2 12.6 39.2 29.0 ϵε=2/255 [HTML]EFEFEFLLaVA-DAFT (ours) [HTML]EFEFEF23.0 [HTML]EFEFEF59.9 [HTML]EFEFEF41.4 [HTML]EFEFEF34.9 [HTML]EFEFEF31.4 [HTML]EFEFEF12.6 [HTML]EFEFEF41.5 [HTML]EFEFEF30.1 LLaVA-CLIP 2.1 4.1 1.5 0.1 0.1 0.0 3.1 0.4 LLaVA-TeCOA [18] 8.8 32.9 30.7 23.4 25.0 6.2 20.9 21.3 LLaVA-PMG-AFT [23] 8.9 31.1 29.7 23.1 24.8 5.9 20.0 20.9 LLaVA-FARE [19] 16.1 45.2 33.0 26.0 25.9 9.4 30.6 23.6 ϵε=4/255 [HTML]EFEFEFLLaVA-DAFT (ours) [HTML]EFEFEF17.3 [HTML]EFEFEF48.1 [HTML]EFEFEF34.5 [HTML]EFEFEF27.9 [HTML]EFEFEF27.0 [HTML]EFEFEF8.7 [HTML]EFEFEF32.7 [HTML]EFEFEF24.5 LLaVA-CLIP 0.8 2.1 0.0 0.0 0.0 0.0 1.4 0.0 LLaVA-TeCOA [18] 5.3 19.9 20.9 15.9 22.7 4.4 12.6 16.0 LLaVA-PMG-AFT [23] 5.6 20.9 21.9 15.6 22.1 4.8 13.2 16.1 LLaVA-FARE [19] 9.3 28.3 22.4 16.1 17.0 5.3 18.8 15.2 ϵε=8/255 [HTML]EFEFEFLLaVA-DAFT (ours) [HTML]EFEFEF10.7 [HTML]EFEFEF30.6 [HTML]EFEFEF24.4 [HTML]EFEFEF16.5 [HTML]EFEFEF19.7 [HTML]EFEFEF5.8 [HTML]EFEFEF20.6 [HTML]EFEFEF16.6 I Performance on Different LVLMs TABLE VII: Robust Performance of OpenFlamingo with different vision encoders on various datasets. For caption datasets Flickr30 and COCO, we report their CIDEr scores. For other four VQA datasets, we report their VQA accuracy. The highest accuracy for each dataset under two attack budgets is highlighted in bold. Overall, OpenFlamingo-DAFT demonstrates strong performance under both clean and adversarial conditions. Perturbation Budget Model Flickr30 COCO VQAv2 OKVQA VizWiz TextVQA Average-Caption Average-VQA OpenFlamingo-CLIP 0.5 1.7 0.4 1.1 0.9 0.0 1.1 0.6 OpenFlamingo-TeCOA [18] 9.2 37.7 26.8 16.9 17.2 4.4 23.5 16.3 OpenFlamingo-PMG-AFT [23] 9.5 38.2 27.3 17.4 17.5 4.7 23.9 16.7 OpenFlamingo-FARE [19] 15.3 47.9 28.1 22.1 19.7 7.8 31.6 19.4 e=2/255 [HTML]EFEFEFOpenFlamingo-DAFT (ours) [HTML]EFEFEF16.6 [HTML]EFEFEF51.3 [HTML]EFEFEF29.6 [HTML]EFEFEF23.4 [HTML]EFEFEF21.7 [HTML]EFEFEF6.4 [HTML]EFEFEF34.0 [HTML]EFEFEF20.3 OpenFlamingo-CLIP 0.3 1.4 0.2 0.0 0.0 0.0 0.9 0.1 OpenFlamingo-TeCOA [18] 5.5 26.7 23.2 14.7 15.9 2.8 16.1 14.2 OpenFlamingo-PMG-AFT [23] 5.7 27.1 23.7 14.5 16.4 3.1 16.4 14.4 OpenFlamingo-FARE [19] 9.1 38.4 25.9 18.1 17.2 4.6 23.8 16.5 e=4/255 [HTML]EFEFEFOpenFlamingo-DAFT (ours) [HTML]EFEFEF10.2 [HTML]EFEFEF40.7 [HTML]EFEFEF26.6 [HTML]EFEFEF19.5 [HTML]EFEFEF18.8 [HTML]EFEFEF4.4 [HTML]EFEFEF25.5 [HTML]EFEFEF17.3 We also conduct experiments on OpenFlamingo [61], a model which uses the same CLIP visual encoder as its visual backbone. The attack setup is identical to the one in Sec. IV-C. Tab. VII presents the evaluation results on OpenFlamingo using different adversarial fine-tuning methods. The following analysis mainly focuses on the robust performance of the OpenFlamingo model and the transferability of the DAFT method. Tab. VII shows that OpenFlamingo demonstrates overall weaker robustness against adversarial attacks. On various caption datasets and VQA datasets, its robust performance is poorer compared to LLaVA. However, OpenFlamingo-DAFT shows a significant performance improvement under different adversarial perturbations compared to the original OpenFlamingo model (e.g., OpenFlamingo-CLIP), proving the effectiveness of the DAFT method in enhancing robustness. A comparison between OpenFlamingo-DAFT and LLaVA-DAFT indicates that while both benefit from the DAFT method, OpenFlamingo exhibits lower robust performance on certain datasets (e.g., COCO), not reaching the performance level of LLaVA. However, OpenFlamingo-DAFT demonstrates a remarkable improvement under adversarial attacks, especially on more challenging datasets such as TextVQA and OKVQA, where its robustness significantly outperforms other fine-tuning methods. The comparison between LLaVA and OpenFlamingo highlights the transferability of the DAFT method. Despite architectural differences between the two models, DAFT enhances robustness on both models. This suggests that DAFT is not reliant on a specific visual encoder or model architecture, and its ability to improve robustness transfers across different models. I Black-box Transferability Evaluation TABLE VIII: Black-box transfer evaluation between LLaVA and OpenFlamingo under full-model attacks with ϵε = 4/255. All adversarially fine-tuned vision encoders are trained with the perturbation budget of ϵε = 4/255. “LLaVA → OF” denotes adversarial examples generated by attacking the full LLaVA model and evaluated on OpenFlamingo, while “OF → LLaVA” denotes the reverse transfer direction. For captioning tasks, we report the average CIDEr score; for VQA tasks, we report the average VQA accuracy. Model Caption VQA LLaVA → OF OF → LLaVA LLaVA → OF OF → LLaVA CLIP 10.2 29.6 17.7 43.0 TeCoA [18] 22.6 55.0 22.0 43.6 PMG-AFT [23] 34.3 57.5 23.1 44.7 FARE [19] 35.5 64.8 23.8 47.5 [HTML]EFEFEFDAFT (ours) [HTML]EFEFEF39.5 [HTML]EFEFEF65.4 [HTML]EFEFEF24.8 [HTML]EFEFEF48.1 We further evaluate the black-box transferability of adversarial examples between LLaVA and OpenFlamingo. The evaluated tasks are consistent with those in Sec. IV-C, including image captioning and VQA. Different from the previous white-box evaluation, where adversarial examples are generated and evaluated on the same LVLM, we perform full-model attacks on one LVLM and directly evaluate the generated adversarial examples on the other LVLM. Specifically, we consider two transfer directions: attacking LLaVA and evaluating on OpenFlamingo, denoted as LLaVA → OF, and attacking OpenFlamingo and evaluating on LLaVA, denoted as OF → LLaVA. All adversarially fine-tuned vision encoders used in this experiment are trained under the perturbation budget of ϵε = 4/255, and the black-box transfer attacks are also conducted with ϵε = 4/255. This setting evaluates whether the robustness improvement brought by different adversarial fine-tuning methods can generalize across LVLM architectures under transferred black-box attacks. We do not include zero-shot classification in this black-box transfer evaluation because the classification setting only attacks the standalone CLIP model. Unlike captioning and VQA tasks, zero-shot classification does not involve different LVLM architectures or full-model generation pipelines. Therefore, it does not directly reflect the cross-model transferability problem considered here. Tab. VIII shows that DAFT consistently achieves the best performance across both transfer directions and both task types. On captioning tasks, DAFT outperforms FARE by 4.0 CIDEr points under the LLaVA → OF setting and by 0.6 CIDEr points under the OF → LLaVA setting. On VQA tasks, DAFT further improves the average accuracy over FARE by 1.0% and 0.6% under the two transfer directions, respectively. Compared with category-label supervised methods such as TeCoA [18] and PMG-AFT [23], DAFT also shows more stable robustness across different transfer settings. These results indicate that DAFT not only improves white-box adversarial robustness, but also provides better robustness against transferred black-box attacks. Moreover, since the adversarial perturbations are generated by attacking the full LVLM rather than only the vision encoder, the results further demonstrate that the robustness learned by DAFT can better generalize to practical cross-model attack scenarios.