Paper deep dive
Learning What Not to Learn: Adversarial Disentangled Prompt Tuning for Robust Vision-Language Models
Yang Chen, Zhan Zhuang, Yanbin Wei, Zebin Chen, Hua Liu, Yu Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/19/2026, 4:45:18 AM
Summary
The paper introduces ADAPT (Adversarial Disentangled Prompt Tuning), a framework designed to improve the robustness of vision-language models (VLMs) against adversarial attacks, particularly on unseen classes. The authors identify a phenomenon called 'robust generalization overfitting,' where existing adversarial prompt tuning methods learn pseudo-robust features (shortcuts) that degrade performance on unseen classes. ADAPT addresses this by using a dual-prompt mechanism with a target prompt and a pool of decoy prompts. Decoy prompts are guided to entrap pseudo-robust features, while the target prompt is constrained to be orthogonal to them, forcing it to learn robust, generalizable features. Theoretical analysis and extensive experiments demonstrate that ADAPT effectively prevents robust generalization overfitting and improves robustness on unseen classes.
Entities (9)
Relation Signals (8)
Adversarial Prompt Tuning → causes → Robust Generalization Overfitting
confidence 95% · we find that existing methods aggravate robust generalization overfitting on seen classes
ADAPT → mitigates → Robust Generalization Overfitting
confidence 95% · By disentangling the robust features from the pseudo-robust features, ADAPT effectively prevents robust generalization overfitting.
Decoy Prompt → entrap → Pseudo-robust Features
confidence 92% · the decoy prompts are guided to entrap diverse pseudo-robust features
ADAPT → improves → Robustness
confidence 90% · ADAPT substantially improves the robustness of the target prompt on unseen classes.
ADAPT → uses → Decoy Prompt
confidence 90% · ADAPT uses a dual-prompt mechanism with a target prompt and a pool of decoy prompts.
ADAPT → uses → Target Prompt
confidence 90% · ADAPT uses a dual-prompt mechanism with a target prompt and a pool of decoy prompts.
Target Prompt → constrainedby → Orthogonal Loss
confidence 88% · the target prompt is constrained to be orthogonal to the decoys in the embedding space
Robust Generalization Overfitting → causedby → Shortcut Learning
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:While adversarial prompt tuning can enhance robustness of vision-language models efficiently, we find that existing methods aggravate robust generalization overfitting on seen classes, leading to a rapid degradation in performance against adversarial examples of unseen classes as training progresses. We empirically identify that this degradation stems from the tendency of the model to learn pseudo-robust features (i.e., non-generalizable shortcuts). To mitigate this, we propose ADAPT (Adversarial Disentangled Prompt Tuning), a robust prompt tuning framework following the philosophy of ``Learning What Not to Learn''. Specifically, ADAPT uses a dual-prompt mechanism with a target prompt and a pool of decoy prompts. During training, the decoy prompts are guided to entrap diverse pseudo-robust features, while the target prompt is constrained to be orthogonal to the decoys in the embedding space to learn robust features. By disentangling the robust features from the pseudo-robust features, ADAPT effectively prevents robust generalization overfitting. We further provide an analysis showing that the orthogonal loss bounds the effect of shifts in pseudo-robust features on unseen classes, yielding a testing error guarantee. Empirically, extensive experiments demonstrate that ADAPT substantially improves the robustness of the target prompt on unseen classes. The code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.17306v1
- Canonical: https://arxiv.org/abs/2608.17306v1
Trouble viewing inline? Open PDF directly →
Full Text
106,076 characters extracted from source content.
Expand or collapse full text
Learning What Not to Learn: Adversarial Disentangled Prompt Tuning for Robust Vision-Language ModelsConference: Proceedings of the 34th ACM International Conference on Multimedia; November 10–14, 2026; Rio de Janeiro, BrazilProceedings of the 34th ACM International Conference on Multimedia (M ’26), November 10–14, 2026, Rio de Janeiro, BrazilDOI: 10.1145/3767308.3836532ISBN: 979-8-4007-2213-4/2026/119332Thanks: 41: Corresponding authors.CCS: Computing methodologies Artificial intelligence Yang Chen OrcID: 0009-0002-5176-6690 Affiliation: Southern University of Science and Technology , Shenzhen , China email: cheny2023@mail.sustech.edu.cn , Zhan Zhuang OrcID: 0000-0003-0215-8728 Affiliation: City University of Hong Kong , Hong Kong , China email: 12250063@mail.sustech.edu.cn , Yanbin Wei OrcID: 0000-0003-1301-2505 Affiliation: Hong Kong University of Science and Technology , Hong Kong , China email: yanbin.ust@gmail.com , Zebin Chen OrcID: 0009-0009-2233-1349 Affiliation: Southern University of Science and Technology , Shenzhen , China email: 12432660@mail.sustech.edu.cn , Hua Liu OrcID: 0000-0002-9613-8877 Affiliation: Southern University of Science and Technology , Shenzhen , China email: liuh5@sustech.edu.cn and Yu Zhang OrcID: 0000-0003-1100-4835 Affiliation: Southern University of Science and Technology , Shenzhen , China email: yu.zhang.ust@gmail.com 2026; © c Abstract. While adversarial prompt tuning can enhance robustness of vision-language models efficiently, we find that existing methods aggravate robust generalization overfitting on seen classes, leading to a rapid degradation in performance against adversarial examples of unseen classes as training progresses. We empirically identify that this degradation stems from the tendency of the model to learn pseudo-robust features (i.e., non-generalizable shortcuts). To mitigate this, we propose ADAPT (Adversarial DisentAngled Prompt Tuning), a robust prompt tuning framework following the philosophy of “Learning What Not to Learn”. Specifically, ADAPT uses a dual-prompt mechanism with a target prompt and a pool of decoy prompts. During training, the decoy prompts are guided to entrap diverse pseudo-robust features, while the target prompt is constrained to be orthogonal to the decoys in the embedding space to learn robust features. By disentangling the robust features from the pseudo-robust features, ADAPT effectively prevents robust generalization overfitting. We further provide an analysis showing that the orthogonal loss bounds the effect of shifts in pseudo-robust features on unseen classes, yielding a testing error guarantee. Empirically, extensive experiments demonstrate that ADAPT substantially improves the robustness of the target prompt on unseen classes. The code is available at https://github.com/cheny02/ADAPT-ACMMM2026. Keywords: Prompt Tuning; Adversarial Robustness; Vision-Language Models †c-license: by 1. Introduction Vision-language models (VLMs), such as CLIP (32) and its successors (39; 40; 48; 18; 17), have achieved remarkable success in bridging the gap between visual and textual modalities. Through extensive pre-training on massive paired image-text data, these models have shown powerful zero-shot performance on unseen data. However, recent studies (35; 50) have shown that VLMs are extremely fragile to visual adversarial perturbations, often yielding confident but incorrect predictions when exposed to imperceptible noise (12). Such vulnerability raises concerns regarding the reliability of VLMs, prohibiting their adoption in safety-critical scenarios like autonomous driving (42) and healthcare (10), where a single adversarial failure could lead to unacceptable consequences. Figure 1. Illustration of robust generalization overfitting. Adversarial prompt tuning on Caltech101 with CLIP ViT-B/32 under PGD-l∞l_∞ attack (ϵ=4/255ε=4/255). Top: Robust loss remains stable on base (seen) classes but increases on new (unseen) classes as training progresses. Bottom: The robust–clean loss gap remains nearly constant on base classes but grows steadily on new classes. A two-panel line graph illustrates loss metric changes over 200 training epochs. The vertical axis of the top panel ranges from 0.0 to 3.0. A solid line representing robust training loss on base classes steadily decreases. Dashed lines for clean test loss on base and new classes remain near zero at the bottom. The solid line for robust test loss on base classes is stable around 1.3. In contrast, the solid line for robust test loss on new classes rises continuously from 1.6 to 2.2. The vertical axis of the bottom panel plots the loss gap. The gap between robust and clean test loss on base classes is a horizontal line near 0.9. The gap for new classes climbs steadily from 0.9 to 1.3 and deviates significantly. To defend against such threats, Adversarial Training (AT) (25) is widely regarded as the most effective defense. Prior works (27; 36; 43; 16; 7) have integrated AT with fine-tuning to enhance the robustness of CLIP-style models. However, given the ever-increasing scale of modern VLMs, the prohibitive computational cost of full-parameter adversarial fine-tuning renders it impractical for large-scale deployment. Consequently, Parameter-Efficient Fine-Tuning (PEFT) (6) has emerged as a promising alternative. Among PEFT techniques, prompt tuning (24) stands out for its lightweight nature and compatibility with large-scale image-text pre-training. Building on this, recent studies (22; 49; 52; 44; 37) have integrated AT with prompt tuning, termed adversarial prompt tuning, to enhance the robustness of CLIP-style models on downstream tasks efficiently. Upon analyzing the training dynamics of adversarial prompt tuning, we identify a phenomenon that diverges from the widely known robust overfitting (34), which manifests itself as a continued decrease in robust training loss coupled with an increase in robust test loss on the same classes as training progresses. In contrast, within the context of adversarial prompt tuning, we observe a pattern illustrated in Figure 1. Specifically, the robust training and test losses on base (seen) classes consistently decrease or remain stable throughout training. However, the robust test loss on new (unseen) classes begins to rise significantly when the training process proceeds. Crucially, the gap between robust and clean test losses remains stable for base classes, whereas it widens continuously for new classes as training progresses. We term this phenomenon “robust generalization overfitting”, highlighting a specific failure in transferring robustness to unseen semantic categories. We attribute this robust generalization overfitting to shortcut learning (11). Specifically, the model tends to rely on pseudo-robust features that satisfy the training objective for seen categories but lack high-level semantic generalization. Although adversarial training aims to encourage the model to acquire robust representations, the model often takes a shortcut by learning to reverse-engineer the specific perturbation patterns generated by the attack algorithm on base classes. Consequently, these features are effective for defending against attacks on seen data but are inherently non-generalizable to unseen categories, which limits the practical utility of adversarially trained VLMs. Motivated by these insights, we propose Adversarial DisentAngled Prompt Tuning (ADAPT), a robust prompt tuning framework grounded in the philosophy of “Learning What Not to Learn” (20). ADAPT designs a dual-prompt mechanism comprising a robust target prompt and a pool of decoy prompts. During training, decoy prompts are sampled from the pool and intentionally guided to absorb diverse non-generalizable pseudo-robust features, acting as a trap for shortcuts. Meanwhile, the target prompt is tasked with learning robust representations. To ensure the purity of the learned features, we enforce an explicit orthogonal loss that separates the target prompt from the decoy prompts. By effectively rejecting the shortcut features captured by the decoy prompts, the target prompt is compelled to focus on essential and generalizable visual semantics that remain robust across unseen categories. Extensive experiments demonstrate that the proposed ADAPT method effectively enhances the robustness of prompt on unseen classes. The main contributions of this work are four-fold. • To the best of our knowledge, we are the first to find the phenomenon of “robust generalization overfitting” in adversarial prompt tuning, revealing that the degradation of robustness on unseen classes stems from the reliance of VLMs on pseudo-robust features. • We propose the ADAPT method as a robust prompt tuning framework, which employs a dual-prompt mechanism with orthogonal loss to actively entrap shortcuts into decoy prompts, thereby forcing the target prompt to learn robust features. • We provide a theoretical guarantee demonstrating that by minimizing the projection of the target prompt onto the shortcut subspace, ADAPT bounds the generalization error on unseen classes. • Extensive experiments on 15 datasets demonstrate the effectiveness of the proposed ADAPT method, particularly in the challenging adversarial base-to-new generalization setting, where it outperforms state-of-the-art methods by effectively mitigating the robustness gap between seen and unseen classes. 2. Related Work CLIP-based VLMs. VLMs have fundamentally reshaped cognitive systems by bridging the semantic gap between the visual and textual modalities (29; 28), demonstrating exceptional capabilities in open-world vision tasks (23; 53). The seminal work of CLIP (32), pre-trained on about 400 million image-text pairs via contrastive learning, established a dominant paradigm for learning transferable vision-language representations. This success has catalyzed a proliferation of CLIP-style architectures. Following initial extensions like ALIGN (18) and OpenCLIP (17), recent advancements have further expanded this paradigm through efficient scaling strategies such as SigLIP (47), enhanced data curation like MetaCLIP (46) and DFN (8), and massive parameter scaling exemplified by EVA-CLIP-18B (39; 40). Given its trailblazing role and widespread adoption as a standard backbone, our work focuses on CLIP to establish a baseline for robust prompt tuning within this foundational VLM paradigm. Adversarial training of VLMs. Despite the impressive zero-shot capabilities of VLMs, they remain highly vulnerable to adversarial attacks, which add imperceptible perturbations to input images, easily misleading model predictions (12; 35; 50). Adversarial Training (AT) (25) is widely recognized as one of the most effective defense strategies against such threats. Initial attempts to fortify VLMs, such as TeCoA (27), FARE (36), PMG-AFT (43), SLADE (16), and AdvSimplex (7), relied on full-parameter adversarial fine-tuning. While effective, these methods incur prohibitively high computational costs, limiting large-scale deployment. To mitigate these efficiency constraints, recent research has shifted towards integrating AT with prompt tuning (51), a parameter-efficient fine-tuning paradigm originally designed to optimize continuous context vectors for downstream tasks (51; 19). Building on this lightweight mechanism, adversarial prompt tuning methods have emerged as a promising direction. AdvPT (49) and APT (22) demonstrate that optimizing prompts on adversarial examples can enhance the robustness. FAP (52) further improves the robustness by enforcing multi-modal consistency. Despite these advances, existing adversarial prompt tuning methods often struggle with robustness on unseen classes. The proposed ADAPT method aims to alleviate this problem through a dual-prompt mechanism that guides decoy prompts to absorb pseudo-robust features while enforcing orthogonality on the target prompt, thereby disentangling robust features from pseudo-robust features. 3. Methodology In this section, we begin by overviewing the preliminaries and presenting a toy example to elucidate the phenomenon of robust generalization overfitting. Subsequently, we detail the proposed ADAPT framework and conclude with a theoretical analysis that substantiates its robustness. 3.1. Preliminaries CLIP. CLIP aligns visual and textual representations in a shared embedding space via contrastive pre-training. It consists of an image encoder ℰIE_I and a text encoder ℰTE_T. Given an image x, the image encoder extracts its visual feature z()=ℰI()z(x)=E_I(x). For classification, a set of hand-crafted prompts is constructed using the manual template, e.g., ‘a photo of a [CLS]’, where ‘[CLS]’ is a placeholder for class tokens like ‘dog’.11 1 ***you need to say the hand-crafted prompts are used as input to the text encoder. Otherwise why do you mention prompts here? revise this part. ***revised. These prompts are then fed into the text encoder to generate textual embeddings H=Hcc=1Ct_H=\t_H^c\_c=1^C for each class c, where C is the number of classes.22 2 ***define C here. ***revised The probability that x belongs to class y is calculated as pclip(y|,H)=exp(cos(z(),Hy)/τ)∑c=1Cexp(cos(z(),Hc)/τ)p_clip(y|x,t_H)= ( (z(x),t_H^y)/τ)Σ _c=1^C ( (z(x),t_H^c)/τ), where τ is the temperature and cos(⋅,⋅) (·,·) denotes the cosine similarity. Prompt Tuning. While effective, hand-crafted prompts are suboptimal. Prompt tuning (51) addresses this by introducing continuous learnable vectors =1,…,Mv=\v_1,…,v_M\ to replace hand-crafted prompts. These vectors are concatenated with the class token to form the learnable prompt ,[CLS]\v,[CLS]\, which is then fed into ℰTE_T to obtain a set of textual embeddings =cc=1Ct=\t^c\_c=1^C with ct^c denoting the textual embedding of the learnable prompt for class c. The probability of x belonging to class y is calculated as p(y|,)=exp(cos(z(),y)/τ)∑c=1Cexp(cos(z(),c)/τ)p(y|x,t)= ( (z(x),t^y)/τ)Σ _c=1^C ( (z(x),t^c)/τ). Adversarial Prompt Tuning (APT). Despite efficiency, standard prompt tuning methods remain vulnerable to adversarial attacks (25). An adversarial example advx_adv in the adversarial attack is crafted by adding a perturbation δ to a clean image x within a perturbation budget ϵε by solving the following problem33 3 ***what is t here ***t is the textual embedding of learnable prompts, consistent with the definition provided in the preceding “Prompt Tuning” paragraph. δ=argmax‖δ′‖p≤ϵℒce(+δ′,,y),δ= \\ \|δ \|_p≤ε \ L_ce(x+δ ,t,y), where adv=+δx_adv=x+δ, ∥⋅∥p\|·\|_p denotes the ℓp _p norm (typically ℓ∞ _∞), and ℒce(adv,,)=−∑i=1Cyilogp(yi|adv,)L_ce(x_adv,t,y)=-Σ _i=1^Cy_i p(y_i|x_adv,t)44 4 ***t is not used explicitly here. This cross-entropy loss is used multiple times in the subsequent sections and very important. Need to define it precisely. ***I explicitly denote the probability as p(yi|,)p(y_i|x,t) to clarify its dependency on the prompt embeddings t. denotes the cross-entropy loss with ground-truth y. To defend against such attacks, APT (22) integrates adversarial training into prompt tuning. That is, instead of training on clean images, APT optimizes the learnable vectors v to minimize the loss on adversarial examples as (1) min(,)∼[ℒce(adv,,)], _v\ E_(x,y) [L_ce(x_adv,t,y) ], where D denotes the training dataset. Though APT improves the robustness on seen classes, our work identifies that it suffers from robust generalization overfitting. 3.2. A Motivating Example To visualize the phenomenon of robust generalization overfitting, we construct a 2D synthetic dataset as shown in Figure 2. Figure 2. Toy example (bird vs. non-bird). While standard adversarial prompt tuning (i.e., black arrow) gets trapped in non-generalizable shortcuts, our ADAPT employs a decoy prompt (i.e., red arrow) to absorb these shortcuts, guiding the target prompt (i.e., blue arrow) to learn robust features.A two-panel diagram demonstrates how the proposed method avoids shortcut learning in a 2D feature space. The left panel is a scatter plot with semantic features on the horizontal axis and shortcut features on the vertical axis. It shows four data clusters with two for base classes and two for new classes. These clusters are separated by a large vertical gap representing the shortcut direction. The right panel displays three vector arrows originating from the center coordinates. A black arrow representing standard adversarial training points almost entirely along the vertical shortcut axis. A red arrow for the decoy prompt also points vertically to absorb this shortcut. A blue arrow for the target prompt points along the horizontal semantic direction and is orthogonal to the shortcut. Data Distribution and Shortcut Bias. We simulate a binary classification task with two latent features: a semantic feature xsemx_sem (x-axis) and a shortcut feature xshox_sho (y-axis). For the base classes, the data clusters are generated from Gaussian distributions with centroids at [1.5,3.0][1.5,3.0] (positive class) and [−1.5,−3.0][-1.5,-3.0] (negative class), sharing a standard deviation of 0.40.4. Crucially, this setup creates a significantly larger margin (i.e., inter-class distance between centroids)55 5 ***how to define the margin? ***revised. We define the margin here as the inter-class distance between the centroids of the two classes along the respective axis. in the shortcut dimension (i.e., distance66 6 ***what distance? ***see the last footnote. of 6.06.0) compared to the semantic dimension (i.e., distance of 3.03.0). This disparity creates a strong inductive bias, as the classes are further apart along the shortcut axis, causing the model to prioritize xshox_sho as the primary discriminative feature to minimize the training loss. Consequently, adversarial attacks (e.g., PGD attack (25)77 7 ***you did not mention this before. add references ***revised.) naturally focus on this dominant direction, as perturbations along the axis with larger weights induce a larger increase in loss88 8 ***loss increasing? ***I replace “induce the larger loss variation” with “induce a larger increase in loss”. Distribution Shift and Robustness Failure. For the new classes (test stage), we introduce a distribution shift where the centroid of the positive class shifts to [1.5,−3.0][1.5,-3.0] and that of the negative class changes to [−1.5,3.0][-1.5,3.0].99 9 ***this is not defined before. ***I delete “the spurious correlation is reversed:” Crucially, the semantic feature xsemx_sem remains invariant across this shift. Figure 2 (Right) visualizes the learned weight vectors (i.e., arrows) of different prompts. Note that the decision boundary is orthogonal to these weight vectors. Standard adversarial prompt tuning (i.e., black arrow) is skewed heavily towards the y-axis, indicating it has overfitted the shortcut due to its larger margin. While this yields high robustness on base classes, it leads to catastrophic failure (0% robustness) on new classes, where the shortcut correlation is flipped.1010 10 ***you could mention the robust generalization overfitting in this section. *** I added the blue sentence. This failure serves as a concrete manifestation of the robust generalization overfitting phenomenon. Mechanism of ADAPT. Our ADAPT framework successfully disentangles these features. The decoy prompt (i.e., red arrow) is explicitly guided to align with the shortcut direction (i.e., y-axis), absorbing the non-robust features. This alignment is enforced by a sparsity constraint (e.g., ℓ1 _1 regularization) which forces the decoy prompt to learn the easiest shortcut feature. Since the shortcut dimension provides the most dominant gradient signal due to its larger margin, the decoy prompt greedily latches onto this single direction and suppresses the weaker semantic features. In contrast, constrained by the orthogonality to the decoy prompt, the target prompt (i.e., blue arrow) is forced to align with the semantic direction (i.e., x-axis). By effectively ignoring the volatile shortcut dimension, the target prompt achieves stable robustness on both base and new classes. 1111 11 ***in this simple problem, it is possible that the target prompt is close to the y-axis and the decoy prompt is close to the x-axis since there is no hard prompt to regularize the learning of decoy prompt. ***revised. Figure 3. Overview of the proposed ADAPT method. ADAPT operates on a dual-prompt mechanism consisting of a Target Prompt and a Decoy Prompt (uniformly sampled from a pool). The decoy prompt serves as a shortcut trap, optimized to absorb pseudo-robust features via ℒcedL_ce^d while deviating from the general semantic anchor via ℒdisL_dis. Conversely, the target prompt is guided to learn robust features by enforcing orthogonality to the decoy prompt via ℒorthL_orth and maintaining semantic consistency with the general semantic anchor via ℒsemL_sem.A block diagram details the dual-prompt architecture and latent space loss constraints. On the left side, a sampled decoy prompt, a hand-crafted prompt, and a target prompt pass in parallel through a frozen text encoder. This generates three embeddings, including a decoy feature, a general semantic anchor, and a target feature. A dissimilarity loss pushes the decoy feature away from the semantic anchor. A semantic loss pulls the target feature toward the semantic anchor. An orthogonal loss forces the target feature away from the decoy feature. On the right side, an adversarial image passes through a frozen image encoder to generate an image feature. Independent cross-entropy losses then optimize this image feature against the decoy text feature and the target text feature. 3.3. ADAPT Motivated by the robust generalization overfitting phenomenon, we propose the ADAPT method. Unlike standard adversarial prompt tuning, which may bias the model to learn the pseudo-robust features,1212 12 ***what do the shortcuts of seen classes mean? *** I replaced “towards overfitting the shortcuts of seen classes” with “to learn the pseudo-robust features”. “Shortcuts of seen classes” refer to the non-generalizable features that are discriminative for the training (seen) categories but fail to transfer to unseen classes. ADAPT operates on a dual-prompt mechanism designed to disentangle robust features from pseudo-robust features. The overall architecture of ADAPT is illustrated in Figure 3. Dual-prompt mechanism. Existing APT learns a single learnable prompt1313 13 ***is it true? ***I replaced “continuous vector” with “learnable prompt”, which tends to learn pseudo-robust features that are discriminative for the adversarial examples of seen classes but are non-generalizable to unseen classes. To address this, we introduce a target prompt and a pool of decoy prompts. Specifically, the target prompt is defined as t=1t,…,Mt,[CLS]P_t=\v_1^t,…,v_M^t,[CLS]\.1414 14 ***itv_i^t is a vector? There are M vectors? If true, existing APT methods also learn PtP_t which could include multiple continuous vector instead of a single one. ***revised. Considering that adversarial shortcuts may exhibit diverse patterns, a single decoy prompt may not suffice to capture all of them. Therefore, we construct a pool of K decoy prompts d=d1,d2,…,dKP_d=\P_d^1,P_d^2,…,P_d^K\.1515 15 ***what is the form of each diP_d^i? similar to tP_t? ***revised. Each decoy prompt dkP_d^k follows the same structure as the target prompt. During inference, only the target prompt tP_t is used, ensuring no additional computational overhead compared to standard APT1616 16 ***adversarial prompt tuning? ***revised.. Decoy prompt as a shortcut trap. We utilize decoy prompts as a trap to capture the pseudo-robust features. In each training iteration, we uniformly sample an index k∼(1,K)k (1,K) to select an active decoy prompt dkP_d^k from the pool. For the sake of brevity, we omit the index k and refer to the currently active decoy prompt simply as d=1d,…,Md,[CLS]P_d=\v_1^d,…,v_M^d,[CLS]\, and its corresponding textual embeddings to all classes as d=dcc=1Ct_d=\t_d^c\_c=1^C. It is worth noting that each decoy prompt in the pool is initialized independently and updated exclusively when selected. This stochastic sampling strategy encourages the pool to learn a diverse set of shortcut directions, effectively covering the subspace of non-generalizable features. To strictly enforce the decoy prompt to focus only on non-generalizable shortcuts, we aim to push the decoy prompt away from the general semantic space1717 17 ***general semantic space seems undefined. ***revised captured by hand-crafted prompts. Specifically, we consider the text embeddings of the hand-crafted prompt as the general semantic anchor, denoted as H=Hcc=1Ct_H=\t_H^c\_c=1^C. Then, we explicitly maximize the dissimilarity between embeddings of the decoy prompt dt_d and the hand-crafted prompt Ht_H. This can be achieved by minimizing the dissimilarity loss as (2) ℒdis=cos(d,H).L_dis= (t_d,t_H). Note that by minimizing Eq. (2), we encourage the decoy prompt to deviate from the general semantic directions. Simultaneously, the decoy prompt is required to minimize the classification loss on adversarial examples with the loss function as (3) ℒced=ℒce(adv,d,).L_ce^d=L_ce(x_adv,t_d,y). It is important to note that the adversarial examples advx_adv used here are generated at the beginning of the iteration using the current target prompt, thus capturing the specific perturbation patterns that successfully fool the target prompt. By combining two losses in Eqs. (2) and (3), the overall objective for learning the decoy prompt is formulated as (4) ℒtotald=ℒced+αℒdis.L_total^d=L_ce^d+ _dis. By minimizing Eq. (4), the decoy prompt is compelled to find a solution that satisfies the training objective (e.g., high robustness on seen classes) while being semantically distinct from general knowledge. Consequently, the decoy prompt becomes a dedicated trap for adversarial shortcuts. Learning target prompt via disentanglement. While the decoy prompt absorbs the shortcuts, the target prompt is guided to learn robust features. To achieve this, we impose an explicit orthogonal loss1818 18 ***rigorously this cannot be termed constraint since this is only a regularizer to enforce to be close to orthogonality. ***revised. I replace most constraint with loss. in the latent space. Specifically, let t=tcc=1Ct_t=\t_t^c\_c=1^C denote the textual embeddings of the target prompt. To capture the dominant optimization direction shared across classes, we compute the global representation of the target and decoy prompts by averaging the text embeddings across all classes as ¯t=1C∑c=1Ctc t_t= 1C _c=1^Ct_t^c and ¯d=1C∑c=1Cdc t_d= 1C _c=1^Ct_d^c. The orthogonal loss is formulated as1919 19 ***why use global representation to define this loss ***Pseudo-robust features (e.g., texture bias) often manifest as a systematic direction shared across classes. Averaging extracts this principal direction, allowing us to disentangle the prompt subspaces globally. I added a blue sentence. (5) ℒorth=cos2(¯t,sg(¯d)),L_orth= ^2( t_t,sg( t_d)), where sg(⋅)sg(·) denotes the stop-gradient operator. When Eq. (5) reaches its minimum (i.e., 0), ¯t t_t becomes orthogonal to ¯d t_d, achieving disentanglement between the target and decoy prompts. Here, we employ the stop-gradient operation on the decoy prompt, which ensures that the disentanglement process solely updates the target prompt without the interference of updating the decoy prompt. Moreover, to maintain the semantic validity of the target prompt, we tether the target prompt to the general semantic anchor. Specifically, we can minimize the ℓ1 _1 distance between the target prompt and the semantic anchor with the semantic loss formulated as (6) ℒsem=‖t−H‖1,L_sem=\|t_t-t_H\|_1, where ∥⋅∥1\|·\|_1 denotes the ℓ1 _1 norm. This loss ensures that the target prompt retains high-level semantic consistency with the hand-crafted prompt. By combining Eqs. (5) and (6) as well as the cross-entropy loss for classification, the overall objective for learning the target prompt is formulated as2020 20 ***it is better to change λ to γ if γ is not used elsewhere. γ is used in Theorem (7) ℒtotalt=ℒcet+βℒsem+λℒorth,L_total^t=L_ce^t+ _sem+ _orth, where ℒcet=ℒce(adv,t,)L_ce^t=L_ce(x_adv,t_t,y). In summary, for the entire training process, the ADAPT method adopts an alternating optimization strategy that minimizes Eqs. (4) and (7) to update the sampled decoy prompt and the target prompt, respectively. The complete algorithm for the proposed ADAPT method is shown in Algorithm 1. Algorithm 1 Adversarial DisentAngled Prompt Tuning (ADAPT) 1: Input: Training dataset D, CLIP image encoder ℰIE_I and text encoder ℰTE_T, textual embeddings of hand-crafted prompts Ht_H, learning rate η, perturbation budget ϵε, hyper-parameters α,β,λα,β,λ. 2: Initialize: Target prompt tP_t, a pool of decoy prompts d=d1,…,dKP_d=\P_d^1,…,P_d^K\. 3: for each epoch do 4: for minibatch ,x,y in D do 5: // 1. Generate Adversarial Examples using Target Prompt 6: Compute textual embeddings of target prompt t=ℰT(t)t_t=E_T(P_t); 7: Generate adversarial examples: 8: adv←argmaxadvℒce(adv,t,),s.t.∥adv−∥p⩽ϵx_adv← x_advarg \,L_ce(x_adv,t_t,y),\,\,s.t. _adv-x _p ε; 9: // 2. Update Sampled Decoy Prompt 10: Uniformly sample an index k∼(1,K)k (1,K); 11: Get the active decoy prompt dkP_d^k and compute d=ℰT(dk)t_d=E_T(P_d^k); 12: Calculate loss of the decoy prompt: 13: ℒtotald=ℒce(adv,d,)+αcos(d,H)L_total^d=L_ce(x_adv,t_d,y)+α (t_d,t_H) 14: Update dk←dk−η∇dkℒtotaldP_d^k _d^k-η _P_d^kL_total^d; 15: // 3. Update Target Prompt 16: Calculate loss of the target prompt: 17: ℒtotalt=ℒce(adv,t,)+β‖t−H‖1+λcos2(¯t,sg(¯d))L_total^t=L_ce(x_adv,t_t,y)+β\|t_t-t_H\|_1+λ ^2( t_t,sg( t_d)) 18: Update t←t−η∇tℒtotaltP_t _t-η _P_tL_total^t; 19: end for 20: end for 3.4. Theoretical Analysis In this section, we analyze whether loss functions in ADAPT, the semantic loss in Eq. (6) and the orthogonal loss in Eq. (5), can guarantee that the target classifier remains robust to shortcut shifts2121 21 ***what do shortcut shifts mean? ***shortcut shifts refers to a change in the distribution of non-semantic features used for prediction from base to new classes. on unseen classes. All CLIP image/text embeddings are ℓ2 _2-normalized in this section unless stated otherwise. Consider that the feature space ℝdR^d can be decomposed into two orthogonal subspaces: a semantic subspace S and a shortcut subspace U (i.e., ℝd=⊕R^d=S ). Let the image feature2222 22 ***what does normalized mean? why need the normalized features in the analysis but not in the method? of an input x be decomposed as z()=s()+u()z(x)=s(x)+u(x), where s()∈s(x) represents invariant semantics and u()∈u(x) represents distinct shortcut patterns (with ‖u()‖2≤B\|u(x)\|_2≤ B). We assume that the semantic anchors Hcc=1C\t_H^c\_c=1^C2323 23 ***why use a different notation ~Hc t_H^c I add a declaration “All CLIP image/text embeddings are ℓ2 _2-normalized unless stated otherwise.”, and use the same notation as in the method. lie in S and provide a sufficient margin for classification as in the following assumption. Assumption 3.1 (Semantic Separability). There exists a margin γ>0γ>0 such that for any new-class sample (,y)(x,y), the semantic component s()s(x) is correctly classified by the anchors with probability at least 1−ζ1-ζ. Specifically, the following inequality holds with probability at least 1−ζ1- 2424 24 ***I added. is it what you want to express? (8) mH(,y):=minc≠y⟨s(),Hy−Hc⟩≥γ,m_H(x,y):=\,\, _c≠ y\,\, s(x),\,t_H^y-t_H^c \,\,≥\,\,γ, where <⋅,⋅><·,·> denotes the inner product operation. Under such settings, Theorem 3.2 (proved in Appendix A) shows that if the target prompt is sufficiently close to the semantic anchor (enforced by ℒsemL_sem) and orthogonal to the shortcut subspace (enforced by ℒorthL_orth), the robustness carries over to the target classifier on unseen classes.2525 25 ***what does ‘transfer’ mean? The target classifier makes the same correct prediction when the shortcut component u()u(x) shifts (within ‖u()‖2≤B\|u(x)\|_2≤ B), yielding a new-class error bound of at most δ. Theorem 3.2 (Robustness Generalization). Define the semantic deviation εsem:=maxc‖tc−Hc‖2 _sem:= _c\|t_t^c-t_H^c\|_2 and the shortcut leakage ε:=maxc‖P(tc)‖2 _U:= _c\|P_U(t_t^c)\|_2,2626 26 ***I changed P_U to a function. You need to change it in elsewhere it appears, for example, the proof. ***revised.2727 27 ***Are εsem _sem and ε _U related to the two losses? seems to use different norms. ***ℒsemL_sem (Eq. 6) encourages small εsem _sem, using l2-norm here is to facilitate its compatibility with Cauchy–Schwarz, while ℒorthL_orth (Eq. 5) reduces the projection of target prompts onto shortcut directions, captured by ε _U. where P(⋅)P_U(·) is to project onto U. If the target prompts2828 28 ***which prompts? ***revised satisfy (9) γ>2εsem+2Bε,γ>2 _sem+2B _U, then the target classifier y^()=argmaxc⟨z(),tc⟩ y(x)= _c z(x),t_t^c is invariant2929 29 ***what does invariant mean? ***I add an explanation to any shortcut shifts3030 30 ***shortcut shift or shortcut? ***I add an explanation, in the sense that its prediction does not change under arbitrary variations of the shortcut component. Consequently, the testing error rate3131 31 ***what error rate? training or testing? revised. on new classes is bounded by ζ. Table 1. Performance under the setting of adversarial base-to-new generalization. All methods (except zero-shot TeCoA) are trained with 16 instances per base class. Here ϵε equals 4/2554/255. Results of ϵ=1/255ε=1/255 are provided in Appendix C.2. Methods Base New HbH_b Acc. Rob. Acc. Rob. TeCoA 44.51 14.62 42.16 16.14 22.66 APT 58.62 27.40 30.13 10.96 22.47 ADAPT 60.04 27.48 38.98 15.65 28.05 FAP 58.00 27.96 39.77 16.17 28.57 ADAPTM_M 56.82 28.52 41.39 17.76 30.05 (a) Methods Base New HbH_b Acc. Rob. Acc. Rob. TeCoA 43.65 11.36 44.95 13.67 19.39 APT 44.54 13.78 38.56 12.47 19.89 ADAPT 47.07 13.49 43.36 13.71 20.90 FAP 43.29 13.89 37.43 13.05 20.16 ADAPTM_M 41.67 13.88 39.47 14.49 21.01 (b) Methods Base New HbH_b Acc. Rob. Acc. Rob. TeCoA 85.41 49.45 83.08 54.37 64.14 APT 90.77 65.07 69.21 42.03 61.89 ADAPT 91.28 65.91 82.21 51.42 69.28 FAP 85.22 63.72 70.20 44.87 62.54 ADAPTM_M 89.74 66.24 79.15 52.29 68.97 (c) Methods Base New HbH_b Acc. Rob. Acc. Rob. TeCoA 74.11 20.15 82.10 29.25 36.53 APT 74.22 27.38 73.04 23.21 37.46 ADAPT 79.00 27.59 76.17 27.96 40.90 FAP 74.43 31.84 68.51 26.51 41.17 ADAPTM_M 77.35 31.53 74.22 33.00 45.24 (d) Methods Base New HbH_b Acc. Rob. Acc. Rob. TeCoA 12.49 2.10 17.38 2.08 3.65 APT 40.23 10.27 15.72 3.14 7.93 ADAPT 42.93 9.92 19.76 3.79 9.12 FAP 45.75 8.87 34.41 5.64 11.73 ADAPTM_M 38.81 8.10 36.35 7.18 12.66 (e) Methods Base New HbH_b Acc. Rob. Acc. Rob. TeCoA 40.65 14.81 38.94 10.28 18.60 APT 84.90 49.57 19.65 4.75 13.63 ADAPT 82.15 48.53 24.18 6.88 18.22 FAP 75.31 45.96 27.73 7.94 20.30 ADAPTM_M 78.73 45.39 28.79 9.50 22.89 (f) Methods Base New HbH_b Acc. Rob. Acc. Rob. TeCoA 59.49 4.92 32.19 5.61 9.32 APT 37.22 10.11 18.61 3.79 9.02 ADAPT 42.37 10.69 30.09 6.78 13.43 FAP 50.39 11.92 42.72 10.37 17.89 ADAPTM_M 41.39 13.33 31.52 9.49 16.93 (g) Methods Base New HbH_b Acc. Rob. Acc. Rob. TeCoA 10.26 0.66 9.96 0.90 1.42 APT 21.19 7.08 5.82 2.22 4.93 ADAPT 20.17 6.36 12.42 2.70 6.08 FAP 21.13 6.90 15.78 3.90 7.81 ADAPTM_M 19.93 8.52 13.62 3.78 7.91 (h) Methods Base New HbH_b Acc. Rob. Acc. Rob. TeCoA 39.81 7.91 45.36 10.84 15.05 APT 53.40 14.27 30.08 6.97 15.06 ADAPT 55.08 14.54 37.25 9.07 17.85 FAP 50.88 15.17 41.31 12.42 21.02 ADAPTM_M 48.35 14.40 44.05 13.55 21.43 (i) Methods Base New HbH_b Acc. Rob. Acc. Rob. TeCoA 32.87 16.09 34.30 18.48 22.75 APT 53.36 25.35 23.67 10.75 20.68 ADAPT 54.63 28.01 30.31 15.22 26.19 FAP 54.17 28.94 26.93 15.10 25.58 ADAPTM_M 55.21 32.06 34.90 20.05 31.29 (j) Methods Base New HbH_b Acc. Rob. Acc. Rob. TeCoA 48.19 23.48 30.85 22.36 28.47 APT 81.76 54.81 11.95 5.33 13.25 ADAPT 78.86 52.14 42.64 26.56 43.03 FAP 79.76 54.86 43.26 28.00 44.64 ADAPTM_M 76.29 55.43 37.18 20.05 37.06 (k) Methods Base New HbH_b Acc. Rob. Acc. Rob. TeCoA 42.71 9.88 44.67 9.73 16.01 APT 63.19 23.73 25.15 5.90 14.97 ADAPT 66.86 25.13 30.34 8.11 18.96 FAP 57.70 25.49 29.15 10.06 21.02 ADAPTM_M 57.55 24.87 36.02 12.01 23.72 (l) 4. Experiments 4.1. Setups Datasets. Following the protocols established in (51; 52; 22), we conduct experiments under four distinct settings: adversarial base-to-new generalization, adversarial few-shot classification, adversarial cross-dataset generalization, and adversarial domain generalization. The first three settings are evaluated on a standard suite of 11 datasets. For the setting of adversarial domain generalization, we train on ImageNet (5) and evaluate on its out-of-distribution (OOD) variants. Details are provided in Appendix B. Baselines. We compare ADAPT with state-of-the-art adversarial defense strategies for VLMs. Specifically, we select zero-shot TeCoA (27) as the representative for robust fine-tuning. For adversarial prompt tuning3232 32 ***adversarial prompt tuning? revised methods, we compare against APT (22), which optimizes textual prompts, and FAP (52), which incorporates multi-modal prompts. These baselines cover the primary paradigms in robust VLM research. Since ADAPT focuses on optimizing textual prompts, it has fewer learnable parameters than multi-modal prompt tuning methods such as FAP. For a fair comparison and to further explore the potential of our method, we extend our method to a multi-modal prompt version, termed ADAPTM_M.3333 33 ***what about ADAPTM_M? ***revised. Drawing inspiration from3434 34 ***it is better not to mention this method as you did not compare with it***revised. FAP (52), ADAPTM_M establishes a deep coupling mechanism between vision and language branches. Specifically, ADAPTM_M maintains two sets of learnable deep visual prompts for the target and decoy prompts, respectively. For each layer, these visual prompts are mapped into the text embedding space through a lightweight cross-modal projection, producing the corresponding deep textual prompts. The deep visual and text prompts are then injected into the input sequence of the corresponding transformer block. In this way, the two branches are coupled layer-by-layer. Thus, both the target and the decoy have their own multi-modal prompts.3535 35 ***this part is unclear to me. need more details.***revised Adversarial training and evaluation. For adversarial training and evaluation, we employ PGD attack (25) under the ℓ∞ _∞ threat model. Consistent with prior works (22; 27; 36), we conduct experiments under two perturbation budgets of ϵ=1/255ε=1/255 and ϵ=4/255ε=4/255. During training, we generate adversarial examples using a 3-step PGD with a step size of 2ϵ/32ε/3. For evaluation, we employ a 100-step PGD with a step size of ϵ/4ε/4 and random initialization. More implementation details are provided in Appendix B.3636 36 ***you need to introduce the evaluation metrics especially ‘HbH_b’ in somewhere of the main body. you cannot introduce them in the caption of tables or figures. ***revised. 4.2. Main Results Adversarial base-to-new generalization. In this scenario, we evaluate the robust base-to-new transferability of the learned prompts by splitting datasets into disjoint base (seen) and new (unseen) classes. Models are trained solely on base classes and evaluated on both sets. Table 3(l) reports the detailed performance across 11 datasets, where ‘HbH_b’ denotes the harmonic mean computed over accuracy and robustness on base classes and on new classes. We can see that APT suffers from severe robust generalization overfitting, characterized by high performance on base classes but a sharp decline on new classes. In contrast, ADAPT effectively mitigates this issue. On the challenging new classes, ADAPT outperforms APT by an average of 8.85% in accuracy and 4.69% in robustness, resulting in a substantial gain of 5.58% in the ‘HbH_b’ metric. Furthermore, this advantage extends to the multi-modal prompting, where ADAPTM_M consistently surpasses the state-of-the-art FAP, achieving improvements of 1.62% and 1.59% in terms of the accuracy and robustness on new classes, respectively. Crucially, the gains in robust generalization do not come at the cost of performance degradation on base classes. ADAPT achieves the highest average accuracy (i.e., 60.04%) on base classes among all baseline methods, surpassing APT and FAP by 1.42% and 2.04%, respectively, and performs comparably in terms of robustness. ADAPTM_M achieves the highest average robustness on base classes (i.e., 28.52%). Those results demonstrate that by effectively disentangling robust features from shortcuts, the proposed method achieves the best trade-off between accuracy and robustness on base and new classes. A line chart compares robustness and harmonic mean scores across one, four, and sixteen-shot training scales. The horizontal axis uses a logarithmic scale for the amount of training data per class. The vertical axis measures percentage scores from 10 to 30. The data form two distinct upward-sloping bands. The upper red band represents the harmonic mean, while the lower blue band represents robustness. Within both regions, the proposed multi-modal ADAPT method occupies the top position and achieves the highest scores at all data scales. It is closely followed by the single-modal ADAPT method, FAP, and APT. Two isolated star markers on the far left vertical axis indicate the zero-shot TeCoA baseline performance. These markers are positioned significantly lower than the starting points of the proposed methods at one shot. Figure 4. Average performance on the 11 datasets under the adversarial few-shot classification setting. Since accuracies are comparable across methods, we plot only robustness and ‘H’ for clarity. ϵ=4/255ε=4/255. Full results are provided in Appendix C.3.A line chart compares robustness and harmonic mean scores across one, four, and sixteen-shot training scales. The horizontal axis uses a logarithmic scale for the amount of training data per class. The vertical axis measures percentage scores from 10 to 30. The data form two distinct upward-sloping bands. The upper red band represents the harmonic mean, while the lower blue band represents robustness. Within both regions, the proposed multi-modal ADAPT method occupies the top position and achieves the highest scores at all data scales. It is closely followed by the single-modal ADAPT method, FAP, and APT. Two isolated star markers on the far left vertical axis indicate the zero-shot TeCoA baseline performance. These markers are positioned significantly lower than the starting points of the proposed methods at one shot. Adversarial few-shot classification. In this scenario, we assess the capability of prompts to learn robust representations from limited labeled data. Specifically, models are tuned using 1,4,16\1,4,16\ shots per class and evaluated on the remaining samples. The average performance on 11 datasets is reported in Figure 4, where ‘H’ denotes the harmonic mean of the robustness and accuracy. ADAPT consistently outperforms TeCoA and APT across all settings. Even in the extreme data-scarce regime (i.e., 1-shot), ADAPT achieves significant improvements, surpassing APT (TeCoA) by 0.90% (1.9%) and 1.64% (2.7%) in terms of the robustness and ‘H’, respectively. Notably, ADAPTM_M achieves the highest robustness and ‘H’ among all baselines under all settings, validating that our proposed method effectively learns robust features without overfitting to shortcuts.3737 37 ***In Figure 4, there is a method called D-ADAPT. revise the legend. ***revised.3838 38 ***why are the performance of TeCoA two points instead of two curves in Figure 4?***Because TeCoA is a zero-shot method. Adversarial robustness evaluation under various attacks. Here we evaluate the adversarial robustness of adversarial prompt tuning3939 39 ***adversarial prompt tuning? ***revised methods using a wider variety of attacks, including Carlini & Wagner (CW) attack (2) and AutoAttack (A) (4). CW represents a strong optimization-based attack aiming for minimal perturbations, while A serves as a standardized, ensemble-based attack for reliable robustness assessment. Both CW and A generate more potent adversarial examples than PGD, making them stronger attack methods for more rigorous evaluation. Table 2 reports the performance across 10 datasets (excluding ImageNet)4040 40 ***why excluding this dataset? any reason? or any reference to follow? ***Evaluating on Imagenet on A may take more than one month., where ‘HaH_a’ denotes the harmonic mean computed over Acc.BaseAcc._Base, PGDBasePGD_Base, Acc.NewAcc._New, PGDNewPGD_New, CW, and A on both base and new classes. The proposed methods consistently outperform the baselines under all attacks. Specifically, ADAPT surpasses APT by 5.23% in ‘HaH_a’. Furthermore, ADAPTM_M achieves the best overall performance, outperforming the previous SOTA method (i.e., FAP) by 1.73% in terms of ‘HaH_a’.4141 41 ***what is Ha? ***revised. This consistent superiority against such a diverse and aggressive attack suite provides strong evidence that the robustness learned by ADAPT is generalized, stemming from the effective disentanglement of invariant semantic features from attack-specific shortcuts. Table 2. Average performance under the adversarial base-to-new generalization setting with various attacks. ϵ=4/255ε=4/255. Method Average over 10 datasets Base New HaH_a Acc. PGD Acc. PGD CW A TeCoA 40.54 13.59 38.08 14.90 14.59 13.18 17.86 APT 54.57 26.15 26.63 9.83 9.45 8.56 14.33 ADAPT 55.76 26.26 35.03 14.41 13.64 12.59 19.56 FAP 54.07 26.70 36.36 14.98 13.32 12.19 19.52 ADAPTM_M 53.03 27.26 37.80 16.45 14.86 13.79 21.23 Table 3. Ablation study of the proposed ADAPT framework. We report the average performance under the adversarial base-to-new generalization setting across 11 datasets. ϵ=4/255ε=4/255. Methods Base New HbH_b Acc Rob Acc Rob TeCoA 44.51 14.62 42.16 16.14 22.66 APT 58.62 27.40 30.13 10.96 22.47 ADAPT 60.04 27.48 38.98 15.65 28.05 w/o dP_d 59.95 27.55 36.43 14.75 26.99 w/o ℒdisL_dis 59.80 27.49 38.25 14.99 27.41 w/o ℒsemL_sem 58.34 27.37 34.14 12.54 24.58 w/o ℒorthL_orth 60.19 27.56 37.52 13.16 25.72 Due to page limit, more experimental results can be found in Appendix C, including generalization to alternative VLM backbones, computational cost analysis, and the results under the settings of adversarial cross-dataset generalization and adversarial domain generalization. (a) (b) (c) (d) Figure 5. The average performance of ADAPT on all datasets w.r.t. hyperparameters (i.e., K, α, β, and λ) under the adversarial base-to-new generalization setting.A series of four line charts analyzes the impact of adjusting four hyperparameters on average accuracy. Chart A plots parameter K on a horizontal axis from 1 to 4 and shows accuracy peaking at K equals 2. Chart B plots parameter alpha from 0 to 0.1 with a gentle peak around 0.01. Chart C plots parameter beta from 0 to 40, where accuracy rises sharply from 0 to 30 before leveling off. Chart D plots parameter lambda from 0 to 0.5 and reaches a distinct peak at 0.05. Each subchart includes a light gray rectangular shaded background that visually highlights the parameter range producing the highest and most stable accuracy results. 4.3. Ablation Studies In this section, we conduct ablations to analyze the contributions of the pool of decoy prompts dP_d, the dissimilarity loss ℒdisL_dis, the semantic loss ℒsemL_sem, and the orthogonal loss ℒorthL_orth. As shown in Table 3, the ADAPT method achieves good performance with an average ‘HbH_b’ of 28.05%\%, verifying the necessity of integrating these components. We observe that removing ℒsemL_sem results in the most severe performance degradation. This drop is expected, as ℒsemL_sem serves as a semantic regularizer that tethers the target prompt to the general semantic space. Without it, the model may suffer from catastrophic forgetting of its pre-trained knowledge. Furthermore, the exclusion of the orthogonal loss ℒorthL_orth leads to a drop of 2.33% in ‘HbH_b’. While ℒsemL_sem ensures the model remains a competent VLM, ℒorthL_orth is responsible for explicitly enforcing independence between the target and decoy prompts to actively filter out pseudo-robust features. Additionally, removing the dissimilarity loss ℒdisL_dis or the pool of decoy prompts dP_d also impairs the generalization capability on new classes, confirming that a diverse set of decoy prompts explicitly pushed away from valid semantics is a requisite to entrap diverse pseudo-robust features effectively. 5. Analysis on Hyperparameter Sensitivity We analyze the sensitivity of K, α, β, and λ using the average performance over 11 datasets under adversarial base-to-new generalization with ϵ=4/255ε=4/255. Effect of K. As shown in Figure 5(a), increasing K from 1 to 2 improves performance, indicating that multiple decoy prompts better capture diverse pseudo-robust features. Further increasing K provides no additional benefit and slightly complicates optimization. Effect of α. The coefficient α controls the dissimilarity loss ℒdisL_dis in Eq. (4). Figure 5(b) shows that ADAPT is relatively insensitive to α within [0.005,0.1][0.005,0.1], with the best performance at α=0.01α=0.01. Effect of β. The coefficient β weights the semantic loss ℒsemL_sem in Eq. (7). As shown in Figure 5(c), small values of β lead to substantial degradation, while the performance stabilizes for β∈[30,40]β∈[30,40], highlighting the importance of preserving general semantic knowledge. Effect of λ. The coefficient λ controls the orthogonal loss ℒorthL_orth in Eq. (7). As shown in Figure 5(d), the performance improves substantially as λ increases from 0 to 0.05, confirming the benefit of explicitly disentangling the target and decoy prompts. The best performance is achieved at λ=0.05λ=0.05, while larger values lead to only a slight decline and remain relatively stable, indicating that ADAPT is not overly sensitive to λ within a moderate range. Figure 6. t-SNE visualization of prompt embeddings on Caltech101. The target prompts cluster around general semantic anchors, while the decoy prompts lie in a separate subspace.Two side-by-side scatter plots display the 2D spatial clustering of prompt embeddings for base and new classes. The left chart for base classes and the right chart for new classes exhibit identical spatial layout patterns. On the left side of each chart, the target prompt markers cluster tightly and overlap directly with markers representing general semantic anchors. On the far right side of each chart, two distinct sets of decoy prompts form their own independent clusters. These decoy clusters are spatially isolated from the semantic anchors on the left. The two decoy clusters also maintain a large physical distance from each other, indicating they have converged into separate sub-spaces without interference. 6. Visualization of Prompt Embeddings To verify the effectiveness of the proposed dual-prompt mechanism, we visualize the prompt embeddings using t-SNE in Figure 6. The results reveal a clear disentanglement in the latent space. Target prompts (in blue) consistently align with general semantic anchors (in green) across both base and new classes, confirming that our semantic loss effectively enforces the learning of semantically invariant features. Crucially, the decoy prompts (in red) diverge into distinct non-semantic subspaces isolated from the anchors and, notably, separated from each other. Despite this deviation, both groups of decoy prompts achieve good performance on base classes but degrade significantly on new classes. This contrast indicates that different decoy prompts converge to diverse class-specific shortcuts. Though these features are discriminative on the training classes, they lack transferability. By employing a pool of decoy prompts to act as “diverse traps” for these varying shortcuts, ADAPT allows the target prompt to learn purified robust features. 7. Conclusion In this paper, we identify and formally define the phenomenon of “robust generalization overfitting” in adversarial prompt tuning, revealing that the failure of robustness generalization stems from the model’s reliance on pseudo-robust features. To address this, we propose ADAPT, which is guided by the philosophy of “Learning What Not to Learn”. Specifically, ADAPT employs a dual-prompt mechanism with explicit orthogonal loss, successfully entrapping non-generalizable pseudo-robust features into decoy prompts while guiding the target prompt to learn robust features. Experiments on benchmark datasets demonstrate that ADAPT achieves state-of-the-art performance, particularly in the challenging base-to-new generalization settings. Acknowledgements. This work was supported by National Natural Science Foundation of China under Grant no. 62136005, Shenzhen fundamental research program JCYJ20250604144724032, and the China Postdoctoral Science Foundation under Grant No. 2026M791631. References Bossard et al. (2014) L. Bossard, M. Guillaumin, and L. Van Gool Food-101–mining discriminative components with random forests. In ECCV, Cited by: Appendix B. Carlini and Wagner (2017) N. Carlini and D. Wagner Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), p. 39–57. Cited by: §4.2. Cimpoi et al. (2014) M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi Describing textures in the wild. In CVPR, Cited by: Appendix B. Croce and Hein (2020) F. Croce and M. Hein Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, p. 2206–2216. Cited by: §4.2. Deng et al. (2009) J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei Imagenet: a large-scale hierarchical image database. In CVPR, Cited by: Appendix B, §4.1. Ding et al. (2023) N. Ding, Y. Qin, G. Yang, F. Wei, Z. Yang, Y. Su, S. Hu, Y. Chen, C. Chan, W. Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature machine intelligence 5 (3), p. 220–235. Cited by: §1. Dong et al. (2025) J. Dong, P. Koniusz, Y. Zhang, H. Zhu, W. Liu, X. Qu, and Y. Ong Improving zero-shot adversarial robustness in vision-language models by closed-form alignment of adversarial path simplices. In Forty-second International Conference on Machine Learning, Cited by: §1, §2. Fang et al. (2024) A. Fang, A. M. Jose, A. Jain, L. Schmidt, A. T. Toshev, and V. Shankar Data filtering networks. In The Twelfth International Conference on Learning Representations, Cited by: §2. Fei-Fei et al. (2004) L. Fei-Fei, R. Fergus, and P. Perona Learning generative visual models from few training examples: an incremental bayesian approach tested on 101 object categories. In CVPRW, Cited by: Appendix B. Finlayson et al. (2019) S. G. Finlayson, J. D. Bowers, J. Ito, J. L. Zittrain, A. L. Beam, and I. S. Kohane Adversarial attacks on medical machine learning. Science 363 (6433), p. 1287–1289. Cited by: §1. Geirhos et al. (2020) R. Geirhos, J. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann Shortcut learning in deep neural networks. Nature Machine Intelligence 2 (11), p. 665–673. Cited by: §1. Goodfellow et al. (2014) I. J. Goodfellow, J. Shlens, and C. Szegedy Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. Cited by: §1, §2. Helber et al. (2019) P. Helber, B. Bischke, A. Dengel, and D. Borth Eurosat: a novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 12 (7), p. 2217–2226. Cited by: Appendix B. Hendrycks et al. (2021a) D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo, et al. The many faces of robustness: a critical analysis of out-of-distribution generalization. In ICCV, Cited by: Appendix B. Hendrycks et al. (2021b) D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song Natural adversarial examples. In CVPR, Cited by: Appendix B. Hossain and Imteaj (2025) M. Z. Hossain and A. Imteaj SLADE: shielding against dual exploits in large vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 24244–24254. Cited by: §1, §2. Ilharco et al. (2021) G. Ilharco, M. Wortsman, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, et al. Openclip. Zenodo. Cited by: §1, §2. Jia et al. (2021) C. Jia, Y. Yang, Y. Xia, Y. Chen, Z. Parekh, H. Pham, Q. Le, Y. Sung, Z. Li, and T. Duerig Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the International Conference on Machine Learning, p. 4904–4916. Cited by: §1, §2. Khattak et al. (2023) M. U. Khattak, H. Rasheed, M. Maaz, S. Khan, and F. S. Khan Maple: multi-modal prompt learning. In CVPR, p. 19113–19122. Cited by: §2. Kim et al. (2019) B. Kim, H. Kim, K. Kim, S. Kim, and J. Kim Learning not to learn: training deep neural networks with biased data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 9012–9020. Cited by: §1. Krause et al. (2013) J. Krause, M. Stark, J. Deng, and L. Fei-Fei 3d object representations for fine-grained categorization. In CVPRW, Cited by: Appendix B. Li et al. (2024) L. Li, H. Guan, J. Qiu, and M. Spratling One prompt word is enough to boost adversarial robustness for pre-trained vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 24408–24419. Cited by: Appendix B, Appendix B, §1, §2, §3.1, §4.1, §4.1, §4.1. Liu et al. (2023) H. Liu, C. Li, Q. Wu, and Y. J. Lee Visual instruction tuning. In Advances in Neural Information Processing Systems, Cited by: §2. Liu et al. (2022) X. Liu, K. Ji, Y. Fu, W. Tam, Z. Du, Z. Yang, and J. Tang P-tuning: prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), p. 61–68. Cited by: §1. Madry et al. (2018) A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, Cited by: §1, §2, §3.1, §3.2, §4.1. Maji et al. (2013) S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151. Cited by: Appendix B. Mao et al. (2023) C. Mao, S. Geng, J. Yang, X. Wang, and C. Vondrick Understanding zero-shot adversarial robustness for large-scale models. In The Eleventh International Conference on Learning Representations, Cited by: Appendix B, §1, §2, §4.1, §4.1. Meng et al. (2026a) C. Meng, G. Huang, R. Fu, R. Jian, Z. Gan, and C. Ouyang CLCR: cross-level semantic collaborative representation for multimodal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 1606–1615. Cited by: §2. Meng et al. (2026b) C. Meng, J. Luo, Z. Yan, Z. Yu, R. Fu, Z. Gan, and C. Ouyang Tri-subspaces disentanglement for multimodal sentiment analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 8791–8800. Cited by: §2. Nilsback and Zisserman (2008) M. Nilsback and A. Zisserman Automated flower classification over a large number of classes. In Indian conference on computer vision, graphics & image processing, Cited by: Appendix B. Parkhi et al. (2012) O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar Cats and dogs. In CVPR, Cited by: Appendix B. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, Cited by: Appendix B, §1, §2. Recht et al. (2019) B. Recht, R. Roelofs, L. Schmidt, and V. Shankar Do imagenet classifiers generalize to imagenet?. In ICML, Cited by: Appendix B. Rice et al. (2020) L. Rice, E. Wong, and Z. Kolter Overfitting in adversarially robust deep learning. In International conference on machine learning, p. 8093–8104. Cited by: §1. Schlarmann and Hein (2023) C. Schlarmann and M. Hein On the adversarial robustness of multi-modal foundation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 3677–3685. Cited by: §1, §2. Schlarmann et al. (2024) C. Schlarmann, N. D. Singh, F. Croce, and M. Hein Robust clip: unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. In International Conference on Machine Learning, p. 43685–43704. Cited by: §C.1, §1, §2, §4.1. Sheng et al. (2025) L. Sheng, J. Liang, Z. Wang, and R. He R-tpt: improving adversarial robustness of vision-language models through test-time prompt tuning. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 29958–29967. Cited by: §1. Soomro et al. (2012) K. Soomro, A. R. Zamir, and M. Shah UCF101: a dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402. Cited by: Appendix B. Sun et al. (2023) Q. Sun, Y. Fang, L. Wu, X. Wang, and Y. Cao EVA-clip: improved training techniques for clip at scale. External Links: 2303.15389, Link Cited by: §1, §2. Sun et al. (2024) Q. Sun, J. Wang, Q. Yu, Y. Cui, F. Zhang, X. Zhang, and X. Wang EVA-clip-18b: scaling clip to 18 billion parameters. External Links: 2402.04252, Link Cited by: §1, §2. Wang et al. (2019) H. Wang, S. Ge, Z. Lipton, and E. P. Xing Learning robust global representations by penalizing local predictive power. In NIPS, Cited by: Appendix B. Wang et al. (2023) N. Wang, Y. Luo, T. Sato, K. Xu, and Q. A. Chen Does physical adversarial example really matter to autonomous driving? towards system-level effect of adversarial object evasion attack. In Proceedings of the IEEE/CVF international conference on computer vision, p. 4412–4423. Cited by: §1. Wang et al. (2024) S. Wang, J. Zhang, Z. Yuan, and S. Shan Pre-trained model guided fine-tuning for zero-shot adversarial robustness. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 24502–24511. Cited by: §1, §2. Wang et al. (2025) X. Wang, K. Chen, J. Zhang, J. Chen, and X. Ma Tapt: test-time adversarial prompt tuning for robust inference in vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 19910–19920. Cited by: §1. Xiao et al. (2010) J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba Sun database: large-scale scene recognition from abbey to zoo. In CVPR, Cited by: Appendix B. Xu et al. (2024) H. Xu, S. Xie, X. Tan, P. Huang, R. Howes, V. Sharma, S. Li, G. Ghosh, L. Zettlemoyer, and C. Feichtenhofer Demystifying clip data. In The Twelfth International Conference on Learning Representations, Cited by: §2. Zhai et al. (2023) X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, p. 11975–11986. Cited by: §2. Zhang et al. (2024a) B. Zhang, P. Zhang, X. Dong, Y. Zang, and J. Wang Long-clip: unlocking the long-text capability of clip. In European conference on computer vision, p. 310–325. Cited by: §1. Zhang et al. (2024b) J. Zhang, X. Ma, X. Wang, L. Qiu, J. Wang, Y. Jiang, and J. Sang Adversarial prompt tuning for vision-language models. In European Conference on Computer Vision, p. 56–72. Cited by: §1, §2. Zhao et al. (2023) Y. Zhao, T. Pang, C. Du, X. Yang, C. Li, N. M. Cheung, and M. Lin On evaluating adversarial robustness of large vision-language models. In Advances in Neural Information Processing Systems, Vol. 36, p. 54111–54138. Cited by: §1, §2. Zhou et al. (2022) K. Zhou, J. Yang, C. C. Loy, and Z. Liu Learning to prompt for vision-language models. IJCV 130 (9), p. 2337–2348. Cited by: Appendix B, Appendix B, Appendix B, §2, §3.1, §4.1. Zhou et al. (2024) Y. Zhou, X. Xia, Z. Lin, B. Han, and T. Liu Few-shot adversarial prompt learning on vision-language models. In Advances in Neural Information Processing Systems, Vol. 37, p. 3122–3156. Cited by: Appendix B, §1, §2, §4.1, §4.1, §4.1. Zhu et al. (2024) D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny MiniGPT-4: enhancing vision-language understanding with advanced large language models. In ICLR, Cited by: §2. Contents of the Appendix (1) Appendix A — Theoretical Analysis (2) Appendix B — Setting Details (3) Appendix C — Additional Experimental Results • C.1 - Generalization to Alternative VLM Backbones • C.2 — Comprehensive Results of Base-to-new Generalization • C.3 — Full Results of Adversarial Few-shot Classification • C.4 — Results of Adversarial Domain Generalization • C.5 — Results of Adversarial Cross-dataset Generalization • C.6 — Computational Cost Analysis • C.7 — Empirical Validation of the Semantic–Shortcut Decomposition • C.8 — Robust Generalization Overfitting under Different Training Settings • C.9 — Decoy-Guided Adaptive Attack Appendix A Theoretical Analysis A.1. Assumptions and Main Results (restated) Notation. All CLIP image/text embeddings are ℓ2 _2-normalized unless stated otherwise. Assumption A.1. z()∈ℝdz(x) ^d is the CLIP image feature of a (possibly adversarial) input x, ‖z()‖2=1\|z(x)\|_2=1. Hcc=1C\t_H^c\_c=1^C is the hand-crafted text embeddings (semantic anchors), ‖Hc‖2=1\|t_H^c\|_2=1 for all c. Assume there exists an orthogonal decomposition of the feature space ℝd=⊕R^d=S , where S is a semantic subspace and U is a pseudo-robust subspace, such that: (1) (Feature decomposition) For any (possibly adversarially perturbed) input x from either base or new classes, (10) z()=s()+u(),s()∈,u()∈,‖u()‖2≤B.z(x)=s(x)+u(x), s(x) ,\ u(x) , \|u(x)\|_2≤ B. The distribution of u()u(x) may change arbitrarily from base to new classes. (2) (Anchor semantics) Hc∈t_H^c for all c∈[C]c∈[C]. (3) (New-class anchor margin) There exist γ>0γ>0 and ζ∈[0,1]ζ∈[0,1] such that with probability at least 1−ζ1-ζ over a new-class sample (,y)(x,y), (11) mH(,y):=minc≠y⟨s(),Hy−Hc⟩≥γ.m_H(x,y):= _c≠ y\ s(x),\,t_H^y-t_H^c \ ≥\ γ. where <⋅,⋅><·,·> denotes the inner product operation. Theorem A.2 (Restatement of Theorem 3.2). tcc=1C\t_t^c\_c=1^C is the text embeddings of the target prompts, and define the target classifier (12) y^():=argmaxc∈C⟨z(),tc⟩. y(x):= _c∈ C\ z(x),t_t^c . Define (13) εsem:=maxc∈C‖tc−Hc‖2,ε:=maxc∈C‖Ptc‖2, _sem:= _c∈ C\|t_t^c-t_H^c\|_2, _U:= _c∈ C\|P_Ut_t^c\|_2, where P(⋅)P_U(·) is the orthogonal projector onto U. Under Assumption A.1, if (14) γ>2εsem+2Bε,γ>2 _sem+2B\, _U, then for any new-class sample (,y)(x,y) satisfying mH(,y)≥γm_H(x,y)≥γ, the classifier predicts correctly, i.e., y^()=y y(x)=y, regardless of the shortcut component u()u(x) as long as ‖u()‖2≤B\|u(x)\|_2≤ B. Consequently, the testing error rate of y y on new classes is at most ζ. A.2. Proof of Theorem A.2 Proof. Fix a new-class sample (,y)(x,y) such that mH(,y)≥γm_H(x,y)≥γ. For any competing class c≠yc≠ y, consider the score difference (15) Δy,c():=⟨z(),ty−tc⟩. _y,c(x):= z(x),t_t^y-t_t^c . If Δy,c()>0 _y,c(x)>0 holds for all c≠yc≠ y, then ⟨z(),ty⟩>⟨z(),tc⟩ z(x),t_t^y > z(x),t_t^c for all c≠yc≠ y, implying y^()=y y(x)=y. Step 1: Decompose the score difference into semantic and shortcut parts. By Assumption A.1(1), z()=s()+u()z(x)=s(x)+u(x) with s()∈s(x) and u()∈u(x) . Thus, (16) Δy,c()=⟨s(),ty−tc⟩⏟semantic term+⟨u(),ty−tc⟩⏟shortcut term. _y,c(x)= s(x),t_t^y-t_t^c _semantic term+ u(x),t_t^y-t_t^c _shortcut term. Step 2: Lower bound the semantic term using anchor margin and εsem _sem. Write tk=Hk+kt_t^k=t_H^k+ _k for k∈y,ck∈\y,c\. By definition of εsem _sem, we have ‖k‖2≤εsem\| _k\|_2≤ _sem. Then (17) ⟨s(),ty−tc⟩ s(x),t_t^y-t_t^c =⟨s(),Hy−Hc⟩+⟨s(),y−c⟩ = s(x),t_H^y-t_H^c + s(x), _y- _c (18) ≥γ−‖s()‖2(‖y‖2+‖c‖2), ≥γ-\|s(x)\|_2 (\| _y\|_2+\| _c\|_2 ), where the inequality uses mH(,y)≥γm_H(x,y)≥γ. Moreover, since ‖z()‖2=1\|z(x)\|_2=1 and s()s(x) is an orthogonal component, ‖s()‖2≤1\|s(x)\|_2≤ 1. Therefore, (19) ⟨s(),ty−tc⟩≥γ−2εsem. s(x),t_t^y-t_t^c ≥γ-2 _sem. Step 3: Lower bound the shortcut term using ε _U. Since u()∈u(x) , (20) ⟨u(),ty−tc⟩=⟨u(),P(ty−tc)⟩. u(x),t_t^y-t_t^c = u(x),P_U(t_t^y-t_t^c) . By Cauchy–Schwarz and Assumption A.1(1), (21) ⟨u(),ty−tc⟩ u(x),t_t^y-t_t^c ≥−∥u()∥2⋅∥Pty−Ptc∥2 ≥-\|u(x)\|_2·\|P_Ut_t^y-P_Ut_t^c\|_2 (22) ≥−B(‖Pty‖2+‖Ptc‖2)≥−2Bε, ≥-B (\|P_Ut_t^y\|_2+\|P_Ut_t^c\|_2 )\ ≥\ -2B\, _U, where the last inequality uses the definition of ε _U. Step 4: Combine the bounds. Combining Steps 2–3 yields, for any c≠yc≠ y, (23) Δy,c()≥γ−2εsem−2Bε. _y,c(x)≥γ-2 _sem-2B\, _U. Under the condition γ>2εsem+2Bεγ>2 _sem+2B\, _U, we have Δy,c()>0 _y,c(x)>0 for all c≠yc≠ y, hence y^()=y y(x)=y. Step 5: Error bound on new classes. By Assumption A.1(3), the event mH(,y)≥γ\m_H(x,y)≥γ\ holds with probability at least 1−ζ1-ζ. Since we just showed the classifier is correct whenever this event holds, (24) Pr(,y)∼new[y^()≠y]≤Pr[mH(,y)<γ]≤ζ. _(x,y) _new [ y(x)≠ y ]≤ [m_H(x,y)<γ ]≤ζ. This also implies invariance to any base-to-new shift that changes only u()u(x) (within ‖u()‖2≤B\|u(x)\|_2≤ B) while keeping s()s(x) unchanged, because the above lower bound does not depend on the particular realization of u()u(x) beyond its norm bound. ∎ Appendix B Setting Details Datasets. Following the protocols established in (51; 52; 22), we conduct experiments under four distinct settings: adversarial base-to-new generalization, adversarial few-shot classification, adversarial cross-dataset generalization, and adversarial domain generalization. The first three settings are evaluated on a diverse suite of 11 image classification datasets, covering a wide spectrum of visual recognition tasks. Specifically, these include ImageNet (5) and Caltech101 (9) for general object recognition; OxfordPets (31), StanfordCars (21), Flowers102 (30), Food101 (1), and FGVCAircraft (26) for fine-grained visual categorization; SUN397 (45), DTD (3), and EuroSAT (13) for scene, texture, and satellite imagery classification, respectively; UCF101 (38) for action recognition. For the setting of adversarial domain generalization, we train on the ImageNet dataset and evaluate on its out-of-distribution (OOD) variants, including ImageNetV2 (33), ImageNet-Sketch (41), ImageNet-A (15), and ImageNet-R (14). Implementation Details. Our implementation is built upon the codebases of CoOp (51) and APT (22). All experiments are conducted on NVIDIA GeForce RTX 3090, except for the ImageNet dataset, which is on NVIDIA A100. To ensure a fair comparison with prior works, the backbone architecture follows the default robust configuration (27) to align with the adversarial prompt tuning benchmark (22). All experiments are conducted using the ViT-B/32 CLIP model. We strictly adhere to the experimental settings specified in the original implementations of all baseline methods, including training epochs, learning rate schedules, and data augmentation strategies. Training is conducted using SGD. The learning rate is decayed using the cosine annealing rule. The maximum epoch for ADAPT and APT is set to 200, 100, and 50 for 16, 4, and 1 shots, respectively. For ImageNet, they are 50, 20, and 20. As for ADAPTM_M and FAP, the maximum epoch is fixed to 10. A warm-up strategy is used by fixing the learning rate to 10−510^-5 during the first epoch. For the APT and ADAPT methods, the length of the learnable prompt vectors is fixed to 16 tokens. For the FAP and ADAPTM_M methods, we use a deep prompting strategy, where prompt vectors of length 2 are inserted into both the vision and text branches across the first 9 transformer blocks. For the ADAPT method, α, β, and λ are set to 0.01, 304242 42 ***so large, and 0.05, respectively. The size of the pool of decoy prompts K is set to 2 for ADAPT and 4 for ADAPTM_M, respectively. To encourage the decoy prompt to converge to shortcuts rapidly, we set a higher learning rate for the decoy prompt (i.e., 0.01), and the learning rate for the target prompt is set to 0.002 for ADAPT and 0.0035 for ADAPTM_M. For AutoAttack, we use the standard AutoAttack setting, which executes a suite of four attacks: APGD-CE (untargeted), APGD-T (targeted), FAB-T (targeted), and Square (black-box). All component attacks utilize the default setting of 1 restart. The hand-crafted prompts for different datasets follow 32; 51 and are shown below: ImageNet: "a photo of a [CLS]." Caltech101: "a photo of a [CLS]." OxfordPets: "a photo of a [CLS], a type of pet." StanfordCars: "a photo of a [CLS]." OxfordFlowers: "a photo of a [CLS], a type of flower." Food101: "a photo of [CLS], a type of food." FGVCAircraft: "a photo of a [CLS], a type of aircraft." SUN397: "a photo of a [CLS]." DTD: "a photo of a [CLS], a type of texture." EuroSAT: "a centered satellite photo of [CLS]." UCF101: "a photo of a person doing [CLS]." Note that [CLS] denotes the placeholder for the class name. Appendix C Additional Experimental Results C.1. Generalization to Alternative VLM Backbones To further demonstrate that ADAPT is model-agnostic, we extend our experiments to three alternative vision-language models: a larger CLIP model (ViT-L/14), a more recent and powerful model pre-trained via the FARE method (36), and OpenCLIP. Figure 0(b) shows the average adversarial base-to-new generalization performance across 10 datasets (excluding ImageNet) under a perturbation budget of ϵ=4/255ε=4/255. First, to assess the scalability, we apply our method to the larger TeCoA model with a ViT-L/14 backbone. As shown in , APT continues to suffer from robust generalization overfitting and yields a new-class robustness of only 32.47%. In contrast, ADAPT successfully mitigates this degradation by improving the new-class robustness to 38.01% and elevating the overall harmonic mean ‘HbH_b’ from 48.18% to 52.87%. Furthermore, our multi-modal extension ADAPTM_M achieves an ‘HbH_b’ of 53.94%, consistently outperforming the competitive FAP baseline, which scores 52.38%. We observe a similar performance trajectory when adapting these methods to the FARE backbone. As shown in , the baseline APT method experiences a severe drop in new-class robustness to 13.49%, resulting in a suboptimal ‘HbH_b’ of 28.25%. ADAPT effectively addresses this vulnerability by increasing the new-class robustness to 20.31% and achieving an ‘HbH_b’ of 35.54%. Within the multi-modal prompting paradigm, ADAPTM_M attains the highest ‘HbH_b’ of 39.24% and surpasses the strongest baseline (FAP at 36.65%). We further evaluate ADAPT using OpenCLIP ViT-B/32 pretrained on LAION2B. Table A2 reports the average performance over 10 datasets under ϵ=4/255ε=4/255. ADAPT consistently improves over APT, increasing new-class accuracy and robustness from 39.11% / 13.69% to 47.13% / 19.09%, respectively. These results show that the robust generalization gains of ADAPT transfer to CLIP-like models trained with different data and pretraining pipelines. These consistent improvements across different VLMs confirm that standard adversarial prompt tuning inherently struggles with pseudo-robust features. More importantly, those results demonstrate that ADAPT is a broadly applicable framework and can successfully enhance the robustness of different VLMs. Table A1. Performance under the setting of adversarial base-to-new generalization on different VLM backbones. ‘hc’ denotes hand-crafted prompt, ‘tp’ denotes textual prompt tuning, and ‘mmp’ denotes multi-modal prompt tuning. ϵ=4/255ε=4/255 Methods Average Base New HbH_b Acc. Rob. Acc. Rob. hc TeCoA (ViT-L/14) 49.30 35.48 54.09 40.04 43.51 tp +APT 75.98 59.94 44.68 32.47 48.18 +ADAPT 76.28 61.22 50.26 38.01 52.87 mmp +FAP 72.21 58.66 50.81 38.78 52.38 +ADAPTM_M 72.01 57.70 54.21 40.83 53.94 (a) ϵ=4/255ε=4/255 Methods Average Base New HbH_b Acc. Rob. Acc. Rob. hc FARE (ViT-B/32) 50.69 18.84 51.84 20.95 28.60 tp +APT 69.25 35.54 40.20 13.49 28.25 +ADAPT 70.30 35.21 48.32 20.31 35.54 mmp +FAP 64.83 37.12 45.36 22.35 36.65 +ADAPTM_M 64.11 36.17 50.41 25.74 39.24 (b) Table A2. Adversarial base-to-new generalization with OpenCLIP ViT-B/32 pretrained on LAION2B. Results are averaged over 10 datasets with ϵ=4/255ε=4/255. Method Base New Acc. Rob. Acc. Rob. APT 69.78 35.40 39.11 13.69 ADAPT 71.79 35.57 47.13 19.09 C.2. Comprehensive Results of Adversarial Base-to-new Generalization In this section, we provide the adversarial base-to-new generalization results under the perturbation budget of ϵ=1/255ε=1/255. The results in Table 0(n) show that, consistent with the findings under the larger perturbation budget (ϵ=4/255ε=4/255), our methods demonstrate good robustness transferability. APT suffers from significant overfitting to seen classes, exhibiting a large performance drop on new classes. In contrast, ADAPT effectively mitigates this issue. On average across 11 datasets, ADAPT outperforms APT by substantial margins of 7.06% in accuracy and 6.35% in robustness on new classes. When extended to the multi-modal prompting, ADAPTM_M consistently surpasses the previous state-of-the-art method FAP. Specifically, ADAPTM_M achieves the highest average performance on new classes with 61.21% in accuracy and 45.89% in robustness, outperforming FAP by 4.59% and 2.55%, respectively. Those results show that the disentanglement of robust and pseudo-robust features proposed in ADAPT is effective. Table A3. Performance of various methods under the adversarial base-to-new generalization setting. All methods (except TeCoA) are trained with 16 instances per base class. ‘HbH_b’ denotes the harmonic mean accuracy between Acc.BaseAcc._Base, Rob.BaseRob._Base, Acc.NewAcc._New, and Rob.NewRob._New. ϵ=1/255ε=1/255. Methods Base New HbH_b Acc. Rob. Acc. Rob. TeCoA 54.38 40.07 55.57 41.64 46.86 APT 71.45 55.30 43.97 31.68 46.30 ADAPT 72.81 57.91 51.03 38.03 52.02 FAP 72.36 57.27 56.62 43.34 55.54 ADAPTM_M 72.18 56.56 61.21 45.89 57.42 (c) Methods Base New HbH_b Acc. Rob. Acc. Rob. TeCoA 58.91 40.75 59.71 44.16 49.44 APT 62.14 43.42 55.46 40.07 48.71 ADAPT 63.28 44.34 59.00 43.63 51.13 FAP 62.44 43.61 57.18 41.90 49.81 ADAPTM_M 63.24 43.20 59.08 42.66 50.42 (d) Methods Base New HbH_b Acc. Rob. Acc. Rob. TeCoA 89.35 79.41 89.85 82.86 85.14 APT 96.32 90.06 85.92 77.40 86.87 ADAPT 96.84 91.03 87.99 80.13 88.58 FAP 95.35 89.15 87.55 80.35 87.77 ADAPTM_M 95.67 88.77 88.32 81.00 88.13 (e) Methods Base New HbH_b Acc. Rob. Acc. Rob. TeCoA 84.90 70.18 91.50 79.42 80.73 APT 87.40 71.08 82.83 69.85 77.07 ADAPT 90.54 77.46 91.83 81.04 84.77 FAP 89.79 76.56 91.05 79.92 83.87 ADAPTM_M 89.37 76.93 89.49 78.19 83.07 (f) Methods Base New HbH_b Acc. Rob. Acc. Rob. TeCoA 27.94 14.79 36.27 19.68 22.00 APT 63.92 40.50 28.20 15.62 28.61 ADAPT 63.37 41.23 31.54 17.26 30.85 FAP 61.59 36.21 50.26 29.98 41.19 ADAPTM_M 59.85 34.31 56.23 33.20 42.66 (g) Methods Base New HbH_b Acc. Rob. Acc. Rob. TeCoA 56.79 42.07 58.30 40.64 48.11 APT 90.41 80.91 31.28 18.87 36.91 ADAPT 91.93 82.62 42.77 28.51 49.12 FAP 89.93 79.49 51.28 37.66 57.34 ADAPTM_M 89.55 77.21 56.88 43.05 61.61 (h) Methods Base New HbH_b Acc. Rob. Acc. Rob. TeCoA 55.18 33.73 55.94 36.08 42.84 APT 60.41 37.76 45.21 26.10 38.66 ADAPT 63.63 42.20 50.42 30.60 43.51 FAP 69.76 47.42 66.25 43.90 54.57 ADAPTM_M 70.19 46.43 66.59 43.61 54.25 (i) Methods Base New HbH_b Acc. Rob. Acc. Rob. TeCoA 15.31 8.28 18.30 9.48 11.55 APT 28.75 17.23 13.14 7.68 13.37 ADAPT 28.51 18.55 18.24 10.32 16.62 FAP 24.73 15.67 10.92 6.12 11.14 ADAPTM_M 25.75 17.71 23.58 13.62 18.95 (j) Methods Base New HbH_b Acc. Rob. Acc. Rob. TeCoA 56.81 38.40 61.19 43.61 48.24 APT 70.20 49.55 48.90 32.63 46.77 ADAPT 70.61 51.31 56.35 39.11 51.97 FAP 70.35 50.82 62.85 44.47 55.33 ADAPTM_M 70.52 49.59 64.43 46.06 55.88 (k) Methods Base New HbH_b Acc. Rob. Acc. Rob. TeCoA 44.10 32.29 42.87 33.21 37.35 APT 64.35 48.03 33.70 24.88 37.66 ADAPT 66.67 51.62 40.58 31.04 43.84 FAP 68.75 55.67 41.55 32.85 45.97 ADAPTM_M 69.79 55.67 49.40 36.96 50.26 (l) Methods Base New HbH_b Acc. Rob. Acc. Rob. TeCoA 52.74 40.00 38.92 28.03 37.98 APT 87.02 72.26 24.23 10.97 25.36 ADAPT 89.07 75.19 38.79 27.77 46.34 FAP 87.69 76.43 49.87 41.38 58.22 ADAPTM_M 84.93 73.33 62.41 47.31 63.93 (m) Methods Base New HbH_b Acc. Rob. Acc. Rob. TeCoA 56.10 40.85 58.41 40.83 47.67 APT 75.03 57.50 34.83 24.39 39.83 ADAPT 76.42 61.43 43.81 28.88 46.07 FAP 75.54 58.89 54.03 38.18 53.39 ADAPTM_M 75.13 59.05 56.90 39.16 54.53 (n) C.3. Full Results of Adversarial Few-shot Classification In this section, we provide the full results of adversarial few-shot classification performance on 11 datasets. We evaluate the models using 1, 4, and 16 shots per class. Table A4 shows that the proposed methods demonstrate good data efficiency compared to the baselines. In the average performance across all 11 datasets, ADAPTM_M consistently achieves the highest ‘H’ score across all shot settings (i.e., 19.96% for 1-shot, 24.78% for 4-shot, and 30.11% for 16-shot). Those results demonstrate that the proposed dual-prompt mechanism with orthogonal loss effectively mitigates the reliance on shortcuts and is highly effective in learning robust representations even when labeled data is extremely scarce. Table A4. The performance on the 11 datasets for different shots under the adversarial few-shot classification setting. Dataset ϵ=4/255ε=4/255 Zero-shot 1 shot 4 shot 16 shot TeCoA APT FAP ADAPT ADAPTM_M APT FAP ADAPT ADAPTM_M APT FAP ADAPT ADAPTM_M Average Acc. 33.67 33.20 37.78 38.10 38.06 43.32 44.42 44.89 45.09 51.08 51.11 52.68 49.89 Rob. 10.79 11.79 13.19 12.69 13.53 14.66 15.31 15.80 17.09 20.26 19.65 20.51 21.56 H 16.34 17.40 19.55 19.04 19.96 21.91 22.77 23.37 24.78 29.02 28.39 29.53 30.11 ImageNet Acc. 40.11 37.88 36.95 39.85 36.62 39.13 38.94 40.84 37.60 41.06 40.45 42.27 38.13 Rob. 10.14 10.79 11.41 10.96 11.05 11.49 11.86 11.15 11.83 12.02 12.03 11.64 12.19 H 16.19 16.80 17.44 17.19 16.98 17.76 18.18 17.52 18.00 18.60 18.54 18.25 18.47 Caltech101 Acc. 78.78 77.89 76.67 81.05 79.19 81.62 77.61 84.71 82.76 86.29 80.65 88.80 84.62 Rob. 43.61 45.84 48.36 45.72 48.32 47.59 51.81 52.78 52.01 56.75 55.38 57.36 57.24 H 56.14 57.71 59.31 58.46 60.02 60.12 62.14 65.04 63.88 68.47 65.67 69.70 68.29 OxfordPets Acc. 66.26 59.58 62.14 62.12 63.40 65.99 64.57 69.96 67.95 67.29 67.18 72.04 68.06 Rob. 15.56 15.54 19.81 15.70 17.85 17.80 22.24 18.15 21.59 19.98 24.88 20.63 24.12 H 25.20 24.65 30.04 25.07 27.86 28.04 33.08 28.82 32.77 30.81 36.31 32.07 35.62 Stanford Cars Acc. 10.32 19.49 27.41 21.23 26.90 23.93 33.96 28.22 34.05 31.60 41.54 35.78 38.68 Rob. 0.99 2.90 3.27 3.48 3.52 4.44 4.76 4.66 5.32 7.70 6.62 7.35 7.18 H 1.81 5.05 5.84 5.98 6.23 7.49 8.35 8.00 9.20 12.38 11.42 12.19 12.11 Flowers102 Acc. 30.13 35.93 42.55 36.50 37.23 60.25 55.58 60.74 57.46 76.41 66.26 75.03 71.38 Rob. 8.93 11.49 14.21 10.19 13.93 22.45 20.46 22.98 21.40 37.52 30.25 37.03 33.41 H 13.78 17.41 21.30 15.93 20.27 32.71 29.91 33.34 31.19 50.33 41.54 49.59 45.52 Food101 Acc. 23.52 20.73 33.46 26.65 30.38 21.97 38.84 29.36 31.66 30.39 44.35 35.14 34.86 Rob. 3.27 3.42 5.46 4.97 6.62 4.01 6.39 5.64 7.73 7.90 8.31 7.95 9.94 H 5.74 5.87 9.39 8.38 10.87 6.78 10.97 9.46 12.43 12.54 14.00 12.97 15.47 FGVC Aircraft Acc. 7.17 3.90 11.94 10.11 13.08 13.65 15.27 10.29 14.37 20.31 18.39 19.29 19.08 Rob. 0.36 2.19 2.52 2.16 3.15 3.33 2.73 3.60 3.54 6.15 5.34 5.79 6.75 H 0.69 2.80 4.16 3.56 5.08 5.35 4.63 5.33 5.68 9.44 8.28 8.91 9.97 SUN397 Acc. 33.24 32.32 36.23 35.86 33.63 39.06 39.68 40.98 38.68 45.21 43.35 46.10 42.29 Rob. 6.20 5.76 8.16 7.25 8.32 7.92 9.69 8.70 9.92 11.27 11.64 11.04 11.62 H 10.45 9.78 13.32 12.06 13.34 13.17 15.58 14.35 15.79 18.04 18.35 17.81 18.23 DTD Acc. 24.29 23.29 24.65 28.55 29.02 36.17 37.41 37.06 37.17 45.86 44.21 44.39 43.68 Rob. 11.35 9.46 9.04 12.35 12.23 14.13 15.60 14.54 17.55 21.51 21.63 22.64 22.52 H 15.47 13.45 13.23 17.24 17.21 20.32 22.02 20.89 23.84 29.28 29.05 29.99 29.72 EuroSAT Acc. 19.56 21.09 28.02 37.28 31.36 49.98 40.58 41.90 48.30 64.33 64.51 64.04 56.23 Rob. 11.22 14.53 13.94 15.74 13.06 16.44 8.57 19.40 21.51 25.54 21.28 25.93 32.27 H 14.26 17.21 18.62 22.13 18.44 24.74 14.15 26.52 29.76 36.56 32.00 36.91 41.01 UCF101 Acc. 36.98 33.10 35.58 39.86 37.83 44.75 46.21 49.78 46.00 53.16 51.28 56.62 51.78 Rob. 7.01 7.72 8.86 11.05 10.73 11.66 14.27 12.16 15.57 16.55 18.82 18.29 19.96 H 11.79 12.52 14.19 17.30 16.72 18.50 21.81 19.55 23.27 25.24 27.53 27.65 28.81 C.4. Results of Adversarial Domain Generalization Adversarial domain generalization. In this scenario, we assess the capability of prompts for OOD data. Specifically, models are tuned using 16-shot samples from each of the 1,000 classes on ImageNet (source), and then evaluated on four different target domains (i.e., ImageNet-V2, ImageNet-Sketch, ImageNet-A, and ImageNet-R). Table A5 shows that ADAPT achieves the highest accuracy of 41.34% and competitive robustness compared to baselines on the source domain. More importantly, ADAPT achieves the best average performance of 23.92%, 7.84%, and 11.81% in accuracy, robustness, and ‘H’, respectively. Specifically, ADAPT surpasses APT by substantial margins on challenging domains like ImageNet-Sketch and ImageNet-R, achieving ‘H’ score gains of 0.40% and 0.85%, respectively. Those results show that the robustness learned by ADAPT is not merely due to overfitting the source distribution shortcuts, but stems from disentangled features that can be effectively transferred to OOD data. Table A5. Performance under adversarial domain generalization setting. ϵ=4/255ε=4/255. Method Source Target ImageNet ImageNet-V2 ImageNet-Sketch ImageNet-A ImageNet-R Average Acc. Rob. H Acc. Rob. H Acc. Rob. H Acc. Rob. H Acc. Rob. H Acc. Rob. H TeCoA 40.11 10.14 16.19 33.11 7.49 12.22 17.59 7.24 10.26 4.04 0.28 0.52 37.52 12.51 18.76 23.07 6.88 10.60 APT 41.06 12.02 18.60 33.67 9.09 14.32 18.22 7.87 10.99 4.19 0.36 0.66 37.04 13.29 19.56 23.28 7.65 11.52 FAP 40.32 12.06 18.57 32.81 9.17 14.33 16.42 7.11 9.92 3.87 0.43 0.77 36.04 13.55 19.70 22.29 7.57 11.30 ADAPT 41.34 12.02 18.62 34.24 8.92 14.15 18.56 8.22 11.39 4.21 0.36 0.66 38.65 13.87 20.41 23.92 7.84 11.81 ADAPTM_M 38.20 12.22 18.52 31.35 8.90 13.86 16.84 7.55 10.43 3.49 0.43 0.77 35.79 13.34 19.44 21.87 7.56 11.23 C.5. Results of Adversarial Cross-dataset Generalization To further evaluate the generalization capability of the learned prompts across different distributions, we conduct experiments under the adversarial cross-dataset generalization setting. We train the models on ImageNet (Source) using 16 shots per class and evaluate them directly on 10 other datasets (Target). As shown in Table A6, our proposed methods consistently outperform the baselines in terms of overall performance. ADAPTM_M achieves the highest average harmonic mean of 17.43%. Those results demonstrate that the disentangled features learned by our methods are not only robust on the source domain but also highly transferable to unseen domains. Table A6. Performance under adversarial cross-dataset generalization setting. ϵ=4/255ε=4/255. 16 shot Method Source Target ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 Avg. Acc. TeCoA 40.11 78.78 66.26 10.32 30.13 23.52 7.17 33.24 24.29 19.56 36.98 33.03 APT 41.06 77.40 64.19 11.21 27.65 23.77 4.17 31.85 23.52 16.98 33.73 31.45 FAP 40.32 79.19 63.67 9.64 30.25 22.82 5.43 31.92 23.23 16.79 32.25 31.52 ADAPT 41.34 79.80 65.85 10.55 29.60 23.28 5.91 32.57 24.05 17.40 34.97 32.40 ADAPTM_M 38.20 78.86 60.92 9.23 29.92 20.34 6.30 28.90 22.28 22.74 30.43 30.99 Rob. TeCoA 10.14 43.61 15.56 0.99 8.93 3.27 0.36 6.20 11.35 11.22 7.01 10.85 APT 12.02 45.40 20.44 1.53 9.05 3.90 0.54 7.22 10.82 11.19 6.93 11.70 FAP 12.06 46.25 20.71 1.55 9.74 3.67 0.66 7.15 10.82 11.38 7.45 11.94 ADAPT 12.02 45.27 19.90 1.44 9.14 3.63 1.14 6.84 10.82 11.30 7.69 11.72 ADAPTM_M 12.22 46.73 19.51 1.38 10.11 3.48 1.08 6.95 10.64 14.58 6.82 12.13 H TeCoA 16.19 56.14 25.20 1.81 13.78 5.74 0.69 10.45 15.47 14.26 11.79 16.33 APT 18.60 57.23 31.01 2.69 13.64 6.70 0.96 11.77 14.82 13.49 11.50 17.06 FAP 18.57 58.40 31.25 2.67 14.74 6.32 1.18 11.68 14.76 13.57 12.10 17.32 ADAPT 18.62 57.77 30.56 2.53 13.97 6.28 1.91 11.31 14.93 13.70 12.61 17.21 ADAPTM_M 18.52 58.69 29.55 2.40 15.11 5.94 1.84 11.21 14.40 17.77 11.14 17.43 Table A7. Computational cost and performance comparison on ImageNet under the adversarial base-to-new generalization setting with ϵ=4/255ε=4/255 Method Train.Memory Train.time (h) Infer.time (ms) Base New HbH_b Acc Rob Acc Rob APT 11454M 5.83 1.70 44.54 13.78 38.56 12.47 19.89 ADAPT 27470M 9.65 1.72 47.07 13.49 43.36 13.71 20.90 FAP 19808M 11.22 1.97 43.29 13.89 37.43 13.05 20.16 ADAPTM_M 27530M 13.81 1.69 41.67 13.88 39.47 14.49 21.01 C.6. Computational Cost Analysis To evaluate the practicality and efficiency of our proposed method, we provide a comprehensive analysis of its computational overhead compared to APT and FAP. For a fair comparison, all experiments were conducted on Quadro RTX 8000 under the adversarial base-to-new generalization setting with ϵ=4/255ε=4/255. The results in Table A7 demonstrate that ADAPT achieves a highly favorable trade-off between computational cost and robustness. Training Memory. The training memory required by ADAPT (27470M) and ADAPTM_M (27530M) is higher than that of APT (11454M) and FAP (19808M). This increase primarily stems from the maintenance of the pool of decoy prompt. However, it is worth noting that this memory consumption is proportional to the number of classes. Thus, the memory requirement decreases significantly on datasets with fewer categories than ImageNet. We consider this training-time cost a justifiable trade-off for the good generalization performance achieved. Training Time. In terms of training duration, ADAPT (9.65h) takes longer than the lightweight APT (5.83h) but is notably more efficient than FAP (11.22h). This indicates that while our dual-prompt mechanism introduces some computational overhead, it is still more time-efficient than existing multi-modal prompt tuning approaches. Inference Time. Crucially, during the inference phase, the per-image processing time for ADAPT (1.72ms) and ADAPTM_M (1.69ms) is comparable to APT (1.70ms) and faster than FAP (1.97ms). This is because the decoy prompts are used solely for optimization constraints during training and are discarded during inference. Consequently, our method introduces zero computational overhead during deployment compared to standard adversarial prompt tuning, making it highly suitable for practical, real-time applications. Complexity and scalability. ADAPT introduces no additional inference cost over APT because the decoy prompts are discarded after training. During each training iteration, adversarial examples are generated once using the target prompt, and only one decoy prompt is sampled and updated. Therefore, the decoy computation scales with (C)O(C) rather than (KC)O(KC), where C is the number of classes and K is the size of the decoy pool. The main overhead arises from retaining the text-embedding computation graph of the active decoy prompt, and thus becomes more visible on datasets with many classes. On DTD with ViT-B/32, ADAPT increases memory and training time from 2394M/0.209h to 2810M/0.276h. With ViT-L/14, they increase from 9190M/0.95h to 9964M/1.10h. These results indicate that the additional training cost remains manageable on a larger backbone. This cost is most useful when robust transfer to unseen classes is the main objective, for which ADAPT substantially improves over APT while preserving strong base-class performance. In summary, while ADAPT incurs a moderate increase in training resources to optimize the dual-prompt mechanism, it maintains high inference efficiency. This characteristic, combined with the substantial improvement in robustness, highlights the practicality of our method. C.7. Empirical Validation of the Semantic–Shortcut Decomposition The semantic–shortcut decomposition in Sec. 3.4 is an analytical abstraction rather than an exact structural claim about CLIP features. To connect this abstraction with real CLIP representations, we conduct an image-side projection analysis on DTD. With the CLIP encoders frozen and the hand-crafted prompt used as the classifier, we learn a projection matrix P∈ℝ512×512P ^512× 512 on the base classes through adversarial training. Each image feature z is decomposed as (25) zU=zP,zS=z−zU,z_U=z_P, z_S=z-z_U, where zUz_U denotes the learned base-useful component and zSz_S denotes its residual. Both components are classified using the same hand-crafted prompt. As shown in Table A8, zUz_U performs better on the base classes, whereas zSz_S transfers better to the new classes. This result suggests that real CLIP features can be separated into a base-useful component and a more transferable residual component, providing empirical support for the theoretical abstraction used in our analysis. Table A8. Image-side projection analysis on DTD under adversarial base-to-new generalization. Acc.BaseAcc._Base Rob.BaseRob._Base Acc.NewAcc._New Rob.NewRob._New zUz_U 37.27 15.74 22.45 10.65 zSz_S 19.93 9.78 30.56 14.98 C.8. Robust Generalization Overfitting under Different Training Settings To examine whether robust generalization overfitting is caused by a particular training configuration, we evaluate APT using learning rates from 0.001 to 0.004 and training durations from 50 to 200 epochs. Figure A1 shows that the robust loss gap between base and new classes emerges across these settings, although its magnitude varies. This indicates that robust generalization overfitting is not tied to a single learning rate or training duration. Figure A1. Robust generalization overfitting under different learning rates and training epochs.. C.9. Decoy-Guided Adaptive Attack We design a Decoy-Guided Adaptive Attack (DGAA) tailored to ADAPT. For each non-ground-truth class, DGAA measures (1) how strongly that class competes with the ground-truth class under the target prompt and (2) how strongly it matches the shortcut information captured by the decoy prompts. DGAA combines these scores to select the three most shortcut-confusing classes, runs target-prompt margin PGD toward each selected class, and keeps the adversarial example that most strongly reduces the target-prompt margin. Table A9. Robustness of ADAPT under the decoy-guided adaptive attack. Results are averaged over 11 datasets with ϵ=4/255ε=4/255. Base New Acc. PGD DGAA Acc. PGD DGAA ADAPT 60.04 27.48 23.62 38.98 15.65 13.66 As shown in Table A9, DGAA reduces robustness relative to standard PGD, from 27.48%/15.65% to 23.62%/13.66% on the base/new classes. ADAPT nevertheless retains non-trivial robustness under an attack that explicitly uses the decoy prompts to identify shortcut-confusing targets.