Paper deep dive
Adversarial Decoys: Misdirecting Attention-Based Defenses in ViT
Giulia Marchiori Pietrosanti, Giulio Rossolini, Giorgio Buttazzo
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/10/2026, 2:55:00 AM
Summary
This paper introduces adversarial decoys, independently optimized image patches designed to misdirect attention-based defenses in Vision Transformers (ViTs). By decoupling the adversarial patch (which induces misclassification) from the decoy (which manipulates attention rankings), the method redirects defense suppression toward innocuous tokens while preserving attack effectiveness. Experiments on ImageNet demonstrate that decoys effectively bypass defenses relying on attention magnitude, highlighting a fundamental limitation in current attention-based robustness strategies.
Entities (7)
Relation Signals (6)
Adversarial Decoy → evaluatedon → ImageNet
confidence 98% · Experiments on ImageNet across multiple ViT architectures and attacks show that decoys can redirect high attention scores away from the true adversarial region
Adversarial Decoy → redirects → Attention-Based Defense
confidence 96% · adversarial decoys... redirect the attention, and therefore related defenses, toward selected target tokens.
Vision Transformer (ViT) → vulnerableto → Adversarial Patch
confidence 95% · Vision Transformers (ViTs) remain vulnerable to localized adversarial attacks, e.g., adversarial patches
Adversarial Decoy → decouplesfrom → Adversarial Patch
confidence 94% · our approach decouples the two objectives: the original adversarial region induces the incorrect prediction, while a separate decoy manipulates the attention ranking
Attention-Based Defense → exploits → Self-attention
confidence 92% · These defenses exploit a strong coupling between attention and adversarial effectiveness
Layer-wise Objective → optimizes → Self-attention
confidence 90% · A layer-wise objective increases target-token attention and promotes these tokens above competing non-target ones.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Vision Transformers (ViTs) remain vulnerable to localized adversarial attacks, e.g., adversarial patches, while recent test-time defenses mitigate them by suppressing image tokens with abnormally high attention scores. These defenses exploit a strong coupling between attention and adversarial effectiveness: adversarial tokens often need to attract substantial attention to influence the prediction. We introduce adversarial decoys, independently optimized image patches that redirect the attention, and therefore related defenses, toward selected target tokens. Rather than jointly optimizing misclassifications and defense evasion, our approach decouples the two objectives: the original adversarial region induces the incorrect prediction, while a separate decoy manipulates the attention ranking used by the defense. A layer-wise objective increases target-token attention and promotes these tokens above competing non-target ones. Since the decoy is optimized independently of the underlying attack, the method is attack-agnostic and can be easily integrated with any existing adversarial patch attack. Experiments on ImageNet across multiple ViT architectures and attacks show that decoys can redirect high attention scores away from the true adversarial region while preserving much of the attack effectiveness. These results reveal a fundamental limitation of using attention magnitude as an indicator of adversarial relevance.
Tags
Links
- Source: https://arxiv.org/abs/2607.07922v1
- Canonical: https://arxiv.org/abs/2607.07922v1
Trouble viewing inline? Open PDF directly →
Full Text
78,801 characters extracted from source content.
Expand or collapse full text
Adversarial Decoys: Misdirecting Attention-Based Defenses in ViT Giulia Marchiori Pietrosanti Giulio Rossolini Giorgio Buttazzo Department of Excellence in Robotics and AI, Sant’Anna School of Advanced Study, Italy giulia.marchiori, giulio.rossolini, giorgio.buttazzo@santannapisa.it Abstract Vision Transformers (ViTs) remain vulnerable to localized adversarial attacks, e.g., adversarial patches, while recent test-time defenses mitigate them by suppressing image tokens with abnormally high attention scores. These defenses exploit a strong coupling between attention and adversarial effectiveness: adversarial tokens often need to attract substantial attention to influence the prediction. We introduce adversarial decoys, independently optimized image patches that redirect the attention, and therefore related defenses, toward selected target tokens. Rather than jointly optimizing misclassifications and defense evasion, our approach decouples the two objectives: the original adversarial region induces the incorrect prediction, while a separate decoy manipulates the attention ranking used by the defense. A layer-wise objective increases target-token attention and promotes these tokens above competing non-target ones. Since the decoy is optimized independently of the underlying attack, the method is attack-agnostic and can be easily integrated with any existing adversarial patch attack. Experiments on ImageNet across multiple ViT architectures and attacks show that decoys can redirect high attention scores away from the true adversarial region while preserving much of the attack effectiveness. These results reveal a fundamental limitation of using attention magnitude as an indicator of adversarial relevance. 1 Introduction Vision Transformers (ViTs) have achieved remarkable performance across different visual recognition tasks [10, 6, 30, 23]. A key component of these architectures is self-attention, which enables global interactions among image tokens [32]. Despite these advantages, ViTs remain vulnerable to adversarial input manipulations, including perturbations restricted to localized image regions and adversarial patches [5]. These can alter the internal representations of the model and steer its prediction toward an incorrect class [12, 18, 28, 25]. This threat is especially relevant in real-world settings, where a malicious physical object or printed pattern can be introduced into a scene. Figure 1: Overview of the adversarial-decoy strategy. Adversarial patches induce an incorrect prediction (top), while separately optimized decoy patches redirect attention-based defenses toward innocuous tokens, preserving the adversarial attack (bottom). Inspired by over-activation analysis in convolutional neural networks [35, 26], recent defenses against localized attacks in transformer-based models also exploit the observation that adversarial tokens often induce abnormal activation patterns [21, 17]. Such defenses typically identify tokens that receive unusually high attention, treat them as suspicious, and mitigate their influence at test-time. This strategy is appealing because it uses the model’s own attention mechanism as a signal of adversarial behavior. Moreover, the attention received by adversarial tokens is often closely related to their ability to propagate harmful information through the network. Consequently, directly reducing their attention can also weaken the attack itself, making adaptive evasion particularly challenging (see Sec. 3.2 for a more technical discussion). This raises a central question: can an attention-based defense be misdirected by introducing highly attended but non-adversarial patches, while preserving the effectiveness of the underlying attack? We address this problem by exploiting the global nature of self-attention, which allows modifications in one region to influence the attention elsewhere in the image. Based on this observation, this work introduces adversarial decoys: independently optimized image patches that attract attention toward attacker-selected target regions. The decoy neither replaces the original attack nor directly induces misclassifications. Instead, it manipulates the attention distribution so that the selected target tokens dominate the attention used by the defense. To leverage these adversarial decoys, rather than following the conventional adaptive-attack paradigm, in which a single region is jointly optimized both to induce misclassifications and to evade the defense, we decouple the two objectives by combining an adversarial attack with separate decoy patches. Therefore, the two regions assume complementary roles: the former induces the incorrect prediction, whereas the latter redirects the attention and the related defenses toward specific regions. As illustrated in Fig. 1, the defense may consequently suppress the decoy while leaving the actual adversarial content partially or entirely intact. Since the two components are optimized independently, the proposed mechanism is attack-agnostic and can be combined with different localized attack strategies. The contributions of the work are the following: • Adversarial decoys are introduced together with a layer-wise optimization objective that redirects attention toward selected target tokens by (i) increasing their attention across ViT layers and (i) promoting them above competing non-target tokens in the attention ranking, increasing the likelihood that they are selected by the defense; • Optimized decoys are integrated into the attack pipeline, introducing an attack-agnostic scheme for misleading attention-based defenses in ViT; • Experimental analysis is conducted across ViT architectures and different localized attacks on ImageNet [9], showing that decoys reduce the overlap between the suppression mask and the true adversarial region, while preserving a substantial portion of the attack effectiveness. 2 Related work Self-attention and robustness in ViTs. Attention is a central mechanism in ViTs and is often treated as a spatial signal for understanding token relevance [11, 33]. Prior works have exploited attention patterns for tasks such as object discovery [29, 16], and transformer interpretability [7, 1]. However, attention maps are not always semantically reliable. Recent works [8, 14] showed that ViT representations can contain high-norm tokens that do not necessarily correspond to meaningful image content. More broadly, while the attention mechanism is informative, it is not always reliable and may be fragile when attention distributions are deliberately manipulated. Attention has also played a central role in the robustness analysis of ViTs. Early studies observed that ViTs can be more robust than CNNs under some standard adversarial perturbations [24, 22, 2, 3, 27], partly due to their weaker reliance on high-frequency features and their global token interactions [3, 27]. However, subsequent works showed that ViTs remain highly vulnerable to localized adversarial attacks and transformer-specific perturbations [4, 13, 12, 18, 28, 25]. In the localized setting, such as adversarial objects [5, 34, 15], the global nature of self-attention can become a security weakness: a manipulated region may influence spatially distant features by attracting their attention, unlike CNNs where the effect of a local perturbation is initially constrained by the receptive field [26, 19, 35]. This makes attention not only a component of ViT robustness, but also an explicit attack surface [28, 12, 18]. For instance, PatchFool [12] augments the classification objective with an attention-aware loss that encourages adversarial tokens to receive high attention. Attention-Fool [18] further shows that optimizing post-softmax attention can suffer from saturation, and instead attacks pre-softmax scores to make the adversarial patch attract the attention of all queries. Attention analysis as a defense signal. Following the observed relationship between attention of attacked tokens and their adversarial effectiveness, recent defenses on ViTs use abnormal attention or activation patterns as test-time signals to identify suspicious tokens. A first example is RSA [21], which detects outlier tokens from their value vectors and neutralizes them by replacing both their values and attention weights with neutral quantities. Liu et al. [17] rely more directly on attention scores: ARMRO computes a per-token score from post-softmax attention values across layers and masks tokens selected by a highest-score criterion, significantly improving the robust accuracy of RSA. These defenses are particularly relevant because they make adaptive attacks based on attention-reduction regularization challenging [17] (see Sec. 3.2 and App. A). Conceptually, reducing the attention received by the adversarial region may also weaken its ability to influence the prediction, since high attention often helps propagate the adversarial effect through the network. This motivates a different bypassing paradigm: rather than constraining the attention of the adversarial tokens, we investigate whether a defense can be redirected toward a separate region. 3 Background and motivation This section first reviews the standard paradigm underlying localized adversarial attacks against ViTs and the masking-based test-time defenses designed to counter them. Then, it describes the self-attention mechanism of ViTs and provides a mathematical explanation of how state-of-the-art attention-based attacks and defenses exploit its properties. 3.1 Threat model, attacks and masking defenses We consider a ViT classifier f:→ℝYf:X ^Y, where ⊆ℝH×W×CX ^H× W× C is the input space and Y is the number of classes. Given an image x, the ViT represents it using n=N+1n=N+1 tokens: [t0,t1,…,tN]∈ℝn×d,[t_0,t_1,…,t_N] ^n× d, where t0t_0 is the [CLS] token and ℐ=1,…,NI=1,…,N denotes the set of image-token indices. A localized adversarial attack modifies only a spatially localized region A of the input image. Let m∈0,1H×Wm_A∈\0,1\^H× W denote the corresponding pixel-level mask, where m,u,v=1m_A,u,v=1 if the pixel location (u,v)(u,v) belongs to A, and 0 otherwise. Given adversarial content p∈ℝH×W×C,p_A ^H× W× C, the attacked image is built as x=(−m)⊙x+m⊙p,x_A= (1-m_A ) x+m_A p_A, (1) where the mask is broadcast along the channel dimension. Given the ground-truth label y, an untargeted localized attack can be formulated as p⋆=argmaxp∈ℒcls(f(x),y),p_A = _p_A _AL_cls (f(x_A),y ), (2) where ℒclsL_cls is the classification loss and P_A denotes the feasible set of adversarial contents restricted to A. Conversely, a masking-based defense aims at identifying and suppressing the image regions responsible for the adversarial prediction. Let mdef∈0,1H×Wm_def∈\0,1\^H× W be a pixel-level mask, denoting the image regions identified by the defense as potentially harmful, and let pdefp_def denote the value used to replace the masked pixels, such as zero or the image color mean. The defended image is then built as xdef=(−mdef)⊙x+mdef⊙pdef.x_def= (1-m_def ) x_A+m_def p_def. (3) The model then performs a second inference pass on xdefx_def. Note that studies based on ViTs typically apply masking at the token level. In the following, we assume a white-box threat model in which the attacker has full knowledge of the classifier f. The defender is also assumed to have full access to f during the identification process and masking. 3.2 Attention scores in adversarial tokens Beyond the general formulation of localized adversarial attacks and masking defenses introduced above, it is useful to recall how attention is computed in ViT architectures and how it can be used to quantify the relevance of individual image tokens. Let Xl∈ℝn×dX^l ^n× d denote the token representations entering transformer block l∈1,…,Ll∈\1,…,L\, where X1X^1 is the initial set of input tokens. For each attention head h∈1,…,Nhh∈\1,…,N_h\, let Ql,hQ^l,h, Kl,hK^l,h, and Vl,hV^l,h denote the corresponding query, key, and value representations. The attention matrix is defined as Al,h=softmaxrow(Ql,h(Kl,h)⊤dk)∈[0,1]n×n,A^l,h=softmax_row ( Q^l,h(K^l,h) d_k )∈[0,1]^n× n, (4) where dkd_k is the dimensionality of the query and key representations. Each element Ai,jl,hA_i,j^l,h quantifies the attention assigned by query token i to key token j, and satisfies ∑j=0n−1Ai,jl,h=1. _j=0^n-1A_i,j^l,h=1. To quantify the overall attention received by an image token j, we define its mean received-attention score at layer l as sjl=1Nh|ℐ|∑h=1Nh∑i∈ℐAi,jl,h,j∈ℐ.s_j^l= 1N_h|I| _h=1^N_h _i A_i,j^l,h, j . (5) This score aggregates the column of the attention matrix associated with token j over all attention heads and all image-token queries. The [CLS] token is excluded from both the query set and the set of candidate tokens, so that the analysis focuses exclusively on image-associated tokens. As discussed in Sec. 2, several studies exploit the attention received by adversarial tokens to design both attacks and defenses. We formalize this phenomenon in the following and discuss its implications, particularly for the limitations of the design of adaptive attacks against attention-based defenses. Role of attention in adversarial effect. To better understand the relationship between attention and adversarial behavior, we analyze the contribution of a single token to the output of a self-attention head. Fixing a layer and an attention head, considering a query token i, let oi=∑j=0n−1Ai,jvjo_i= _j=0^n-1A_i,jv_j denote the corresponding output, where vjv_j is the value representation associated with token j, and Ai,j=exp(zi,j)∑k=0n−1exp(zi,k),A_i,j= (z_i,j) _k=0^n-1 (z_i,k), with zi,jz_i,j denoting the corresponding attention logit. Let a denote an arbitrary token of interest, and define the renormalized aggregation of all remaining tokens as μi,¬a=∑j≠aexp(zi,j)∑k≠aexp(zi,k)vj _i, a= _j≠ a (z_i,j) _k≠ a (z_i,k)v_j, we can thus state Proposition 1. For a query token i and an arbitrary token a, the self-attention output satisfies oi−μi,¬a=Ai,a(va−μi,¬a).o_i- _i, a=A_i,a (v_a- _i, a ). (6) A more detailed analysis and derivation of Proposition 1 are provided in App.A. This proposition highlights the distinct roles played by attention weights and value representations. The term (va−μi,¬a)(v_a- _i, a) captures the contribution of token a in the representation space relative to the remaining tokens, while the attention coefficient Ai,aA_i,a determines how strongly this contribution is weighted in the output of the query token. When a is an adversarial token, its influence can therefore be increased in two complementary ways: by shifting (va−μi,¬a)(v_a- _i, a) toward a representation-space direction that reduces the classification margin i.e., the adversarial direction, and by increasing Ai,aA_i,a, thereby amplifying the effect of this harmful contribution on the attention output. This provides an additional understanding of why attacks such as PatchFool [12] can improve their effectiveness by increasing the attention assigned to adversarial tokens. Similarly, it motivates analysis such as ARMRO [17], which identifies and suppresses tokens receiving abnormally high attention. However, the decomposition also highlights a limitation of attention-based analysis: a token with high attention is not necessarily harmful, and an adversarial token does not need to receive the highest attention. 4 Adversarial decoy As discussed in the previous sections, high attention can provide a signal for detecting adversarial tokens. On the other hand, trying to bypass these defenses by constraining attention while preserving adversarial effectiveness can be challenging. To this end, rather than constraining the attention associated with the attack, an attacker can further increase the attention directed toward a separate decoy region, thereby redirecting the defense toward innocuous tokens. In this section, we first introduce the optimization of the decoy patches. We then formalize how the optimized decoy patches can be combined with a localized adversarial attack. 4.1 Optimization method The objective is to modify the predefined decoy patches to induce high received-attention scores within a specific target region. In particular, the optimization aims to make the target tokens dominate the attention ranking across a selected set of transformer layers. Consequently, an attention-based defense may identify these tokens as suspicious and direct its intervention toward them. Formally, let ℐI denote the set of image-token indices and let ⊆ℐD denote the set of tokens whose spatial patches are associated with the modifiable decoy patches. We further define ⊆ℐT as the set of target tokens whose received-attention scores are explicitly optimized, and LL_D as the set of transformer layers considered during the optimization. In the general formulation, the modifiable region D and the target region T may overlap or be disjoint. For each layer l∈Ll∈L_D, let sjls_j^l denote the received-attention score of image token j, as defined in Eq. 5. As a first objective, the optimization encourages all target tokens to receive high attention. We define their average received-attention score at layer l as s¯l=1||∑i∈sil. s_T^l= 1|T| _i s_i^l. The corresponding first objective is ℒtargetl=−log(s¯l+ε),L^l_target=- ( s_T^l+ ), (7) where the logarithm increases the sensitivity to small attention values, which are typically tightly clustered due to the softmax normalization, while ε is introduced for numerical stability. Minimizing this term increases the attention received by the target region at layer l. Figure 2: Illustration of the decoy optimization objective: target token attention increased overall, with additional emphasis on the k-th weakest target score to exceed the strongest non-target score. Increasing the absolute attention received by the target region, however, does not necessarily ensure that its tokens appear among those selected by the defense. We therefore introduce a second objective that explicitly promotes their dominance in the received-attention ranking. Let s,kl=mini∈Topk(,l)sils_T,k^l= _i _k(T,l)s_i^l denote the k-th largest attention score within T. Here, Topk(,l)Top_k(T,l) returns the indices of the k highest-scoring tokens in the target region. The strongest competing score outside the target region is defined as s¬l=maxj∈ℐ∖sjl.s_ ^l= _j s_j^l. The layer-wise target-dominance ratio is then Rl=s,kls¬l+ε,R^l= s_T,k^ls_ ^l+ , (8) A value Rl>1R^l>1 indicates that even the weakest token among the k strongest target tokens receives more attention than the strongest token outside the target region. Therefore, the k strongest target tokens occupy the top k positions of the attention ranking at layer l. Following this, a layer-wise ratio objective is defined as ℒratiol=−log(Rl+ε).L^l_ratio=- (R^l+ ). (9) Minimizing this term increases the relative attention assigned to the target tokens with respect to the strongest non-target competitor. An illustration is shown in Fig. 2. Although maximizing RlR^l promotes the dominance of the target tokens, different layers may reach the desired ratio at different stages of the optimization (see App.C.2). Applying the same weight to all layers may therefore allocate excessive optimization effort to layers in which the target tokens already dominate, while giving insufficient emphasis to layers that have not yet reached the desired ranking. To account for this imbalance, we introduce a specific layer weight. Given a desired dominance ratio r>1r>1, we define αl=α0,Rl<r,α1,Rl≥r,α0≥α1≥0.α^l= cases _0,&R^l<r,\\[2.84526pt] _1,&R^l≥ r, cases _0≥ _1≥ 0. (10) Layers that have not yet reached the desired ratio therefore receive a stronger optimization signal. Conversely, layers that already satisfy the dominance condition receive a smaller contribution, allowing the optimization to focus primarily on the remaining layers while still retaining some pressure on already satisfied ones when α1>0 _1>0. The complete attention-based objective is then ℒdecoy=1|L|∑l∈Lαl(ℒtargetl+βratioℒratiol),L_decoy= 1|L_D| _l∈L_Dα^l (L_target^l+ _ratioL_ratio^l ), (11) where βratio≥0 _ratio≥ 0 controls the importance of the ranking-dominance objective relative to the maximization of the overall attention received by the target region. Let p_D denote the learnable pixel values associated with the modifiable decoy region and let P_D denote their feasible domain. The optimized decoy is obtained by solving p⋆=argminp∈ℒdecoy.p_D = _p_D _DL_decoy. (12) 4.2 Bypassing defenses with decoy patches Rather than designing an adaptive adversarial attack specifically tailored to the defense, the proposed bypass combines a localized adversarial attack with separately optimized decoy patches. The complete procedure is summarized in Algorithm 1 and described below. The attacker manipulates two disjoint sets of image tokens. Let ⊆ℐA denote the tokens associated with the adversarial region, and let ⊆ℐD denote those associated with the decoy region, with ∩=∅.A = . Phase 1 – Adversarial Attack Optimization x←Att(x,y,f,)x_A (x,y,f,A) Phase 2 – Adversarial Decoy Optimization Initialize p(0)∈p_D^(0) _D for t=0,…,num_stepst=0,…,num\_steps do Construct x,(t)x_A,D^(t) by adding p(t)p_D^(t) to x_A Compute ℒdecoyL_decoy from x,(t)x_A,D^(t) over layers LL_D Update the decoy content: p(t+1)←Π(p(t)−η∇p(t)ℒdecoy);p_D^(t+1)← _P_D (p_D^(t)-η _p_D^(t)L_decoy ); return x,x_A,D Algorithm 1 Attack with Decoys As shown in Algorithm 1, the first phase performs the adversarial optimization. A localized attack AttAtt is applied within A to generate adversarial content that induces a misclassification. The location of A may be specified manually or determined by the attack itself (e.g., [12]). Once the adversarial example has been generated, the content of the adversarial region is kept fixed during the subsequent steps. The decoy patches are then optimized on the adversarial input for num_steps iterations using a gradient-based procedure [20] with a step size η, and the objective introduced in Sec. 4.1. The objective encourages the targeted tokens to dominate the attention ranking used by the defense. In our implementation, the target region coincides with the decoy patches location, i.e., =T=D, where every patch is aligned with the tokenization of ViTs, i.e., it corresponds to a token of the model. In Algorithm 1, Π _P_D denotes the projection of p_D onto its admissible set, which may, for example, enforce the valid pixel range [0,1][0,1]. The resulting final input contains both the original adversarial content and the optimized decoy region. Importantly, the underlying adversarial attack is not optimized specifically to account for the decoy. Keeping the adversarial and decoy optimizations separate increases the flexibility of the approach, allowing the same decoy optimization to be combined with different localized attacks, as demonstrated in Sec. 5. 5 Experiments The following experiments assess whether decoys can effectively redirect attention-based masking away from the true adversarial region and improve attack effectiveness. 5.1 Experimental setup Models and dataset. For testing the effectiveness of the adversarial decoys we considered different ViT architectures: DeiT-B/16-224[31], ViT-B/16-224, and ViT-S/16-224 [10]. All models are initialized from publicly available pretrained weights in timm111https://github.com/rwightman/timm. Experiments are conducted on the ImageNet validation set [9]. Following PatchFool and ARMRO practices, and due to the high cost of optimizing both adversarial and decoy regions (see App. C.2), we use a randomly selected 1024 samples [9] for the main experiments. Decoy setting. We evaluate decoys under different token configurations. In the main experiments, decoy locations overlap with randomly selected tokens using a fixed seed to ensure fair comparisons across attacks, defenses, and models. Additional experiments on the effect of decoy placement are reported in App. C.4, where we observe negligible differences. Decoys are optimized for 2500 iterations, which we found necessary for the loss to converge across all ViT layers (see App.C.2). Unless otherwise stated, we set the optimization parameters to βratio=10 _ratio=10 and the dominance threshold to r=2r=2, α0=1 _0=1, α1=0.05 _1=0.05, and η=0.05η=0.05. Ablations on the loss terms are in Sec. 5.5. Attack and defense setting. We combine adversarial decoys with two localized attacks. First, we consider PatchFool [12], a strong token-level attack specifically designed for ViTs. Second, we evaluate a standard adversarial patch attack [5], which perturbs a contiguous region of the image. For PatchFool, we follow the settings of the original implementation. For the adversarial patch attack, we use a learning rate 0.050.05. Both attacks are optimized for 500500 iterations and with a perturbation magnitude 1.01.0 to simulate a classic adversarial patch. For each attack, we also introduce a mask-aware variant, where the original attack objective is left unchanged, while random masking is injected during optimization to improve attack persistence against defense-induced sparse suppressions. This accounts for the fragility of localized attacks under masking, which can also occur when the defense does not cover the attacked regions. We report details and additional analysis of the mask-aware attacks in App. B. Finally, we consider, in Sec. 5.4, an adaptive attack analysis, which jointly optimizes the task objective while explicitly discouraging attention on the attacked tokens, i.e., ℒadapt=ℒcls(f(x),y)−λ1|L|||∑l∈L∑j∈sjl,L_adapt=L_cls(f(x_A),y)-λ 1|L||A| _l∈L _j s_j^l, where λ=0.01λ=0.01 controls the strength of the attention-aware term. More details of the optimization strategy are in App. C.5. As the main defense, we use ARMRO [17], a state-of-the-art attention-based defense, as discussed in Sec. 2. We use τ=1.2τ=1.2 for DeiT-B and ViT-B, and τ=1.4τ=1.4 for ViT-S, showing best results in preliminary analysis. We also evaluate a Top-K attention suppression defense in App. C. PatchFool [12] # Adv. tokens Clean ↑ +Attack ↓ +Defense ↑ +Decoy ↓ Acc. DeiT-B ARMRO (τ=1.2)(τ=1.2) 1 84.87 12.61± 3.0612.61 \,±\,3.06 81.51± 3.5781.51 \,±\,3.57 58.82± 4.5358.82 \,±\,4.53 22.69± 4.2122.69 \,±\,4.21 !10 - 32.81± 4.1732.81 \,±\,4.17 84.38± 3.2284.38 \,±\,3.22 48.44± 4.4348.44 \,±\,4.43 35.94± 4.2635.94 \,±\,4.26 2 - 0.00± 0.000.00 \,±\,0.00 82.50± 3.4882.50 \,±\,3.48 46.67± 4.5746.67 \,±\,4.57 35.83± 4.4035.83 \,±\,4.40 !10 - 4.69± 1.884.69 \,±\,1.88 82.03± 3.4182.03 \,±\,3.41 29.69± 4.0529.69 \,±\,4.05 52.34± 4.4352.34 \,±\,4.43 4 - 0.00± 0.000.00 \,±\,0.00 76.67± 3.8876.67 \,±\,3.88 32.50± 4.2932.50 \,±\,4.29 44.17± 4.5544.17 \,±\,4.55 !10 - 0.00± 0.000.00 \,±\,0.00 72.66± 3.9672.66 \,±\,3.96 5.47± 2.025.47 \,±\,2.02 67.19± 4.1767.19 \,±\,4.17 ViT-B ARMRO (τ=1.2)(τ=1.2) 1 86.55 25.00± 3.9725.00 \,±\,3.97 84.17± 3.3584.17 \,±\,3.35 49.17± 4.5849.17 \,±\,4.58 35.01± 4.3735.01 \,±\,4.37 !10 - 37.82± 4.4637.82 \,±\,4.46 84.03± 3.3784.03 \,±\,3.37 55.46± 4.5855.46 \,±\,4.58 28.57± 4.3328.57 \,±\,4.33 2 - 8.26± 2.518.26 \,±\,2.51 82.64± 3.4682.64 \,±\,3.46 35.54± 4.3735.54 \,±\,4.37 47.11± 4.1747.11 \,±\,4.17 !10 - 6.72± 2.316.72 \,±\,2.31 79.83± 3.6979.83 \,±\,3.69 26.05± 4.0426.05 \,±\,4.04 53.78± 4.5953.78 \,±\,4.59 4 - 0.00± 0.000.00 \,±\,0.00 78.33± 3.7878.33 \,±\,3.78 19.17± 3.6119.17 \,±\,3.61 59.17± 4.5159.17 \,±\,4.51 !10 - 0.00± 0.000.00 \,±\,0.00 71.43± 4.1671.43 \,±\,4.16 20.17± 3.6920.17 \,±\,3.69 53.78± 4.5953.78 \,±\,4.59 ViT-S ARMRO (τ=1.4)(τ=1.4) 1 82.81 13.28± 3.0113.28 \,±\,3.01 80.47± 3.5280.47 \,±\,3.52 25.78± 3.8825.78 \,±\,3.88 54.69± 4.4254.69 \,±\,4.42 !10 - 19.53± 3.5219.53 \,±\,3.52 80.47± 3.5280.47 \,±\,3.52 31.25± 4.1131.25 \,±\,4.11 49.22± 4.7149.22 \,±\,4.71 2 - 0.78± 0.780.78 \,±\,0.78 72.66± 3.9672.66 \,±\,3.96 22.66± 3.7122.66 \,±\,3.71 50.01± 4.5750.01 \,±\,4.57 !10 - 3.12± 1.543.12 \,±\,1.54 70.31± 4.0570.31 \,±\,4.05 22.66± 3.7122.66 \,±\,3.71 47.66± 4.5747.66 \,±\,4.57 4 - 0.00± 0.000.00 \,±\,0.00 64.94± 4.3364.94 \,±\,4.33 14.84± 3.1514.84 \,±\,3.15 46.09± 4.4246.09 \,±\,4.42 !10 - 0.00± 0.000.00 \,±\,0.00 50.10± 4.4450.10 \,±\,4.44 9.38± 2.599.38 \,±\,2.59 40.62± 4.3640.62 \,±\,4.36 Adversarial Patch [5] # Adv. tokens Clean ↑ +Attack ↓ +Defense ↑ +Decoy ↓ Acc. DeiT-B ARMRO (τ=1.2)(τ=1.2) 1 84.87 12.50± 2.9312.50 \,±\,2.93 84.50± 3.2084.50 \,±\,3.20 50.39± 4.4250.39 \,±\,4.42 34.11± 4.1934.11 \,±\,4.19 !10 - 12.50± 2.9312.50 \,±\,2.93 84.50± 3.2084.50 \,±\,3.20 50.39± 4.4250.39 \,±\,4.42 34.11± 4.1934.11 \,±\,4.19 4 - 5.47± 2.025.47 \,±\,2.02 75.00± 3.8475.00 \,±\,3.84 30.47± 4.0830.47 \,±\,4.08 44.53± 4.5544.53 \,±\,4.55 !10 - 2.34± 1.342.34 \,±\,1.34 71.88± 3.9971.88 \,±\,3.99 27.34± 3.9627.34 \,±\,3.96 44.53± 4.4144.53 \,±\,4.41 ViT-B ARMRO (τ=1.2)(τ=1.2) 1 86.55 21.09± 3.6221.09 \,±\,3.62 85.94± 3.0885.94 \,±\,3.08 46.09± 4.4246.09 \,±\,4.42 39.84± 4.3439.84 \,±\,4.34 !10 - 25.00± 3.8425.00 \,±\,3.84 85.94± 3.0885.94 \,±\,3.08 52.34± 4.4352.34 \,±\,4.43 33.59± 4.1933.59 \,±\,4.19 4 - 0.78± 0.780.78 \,±\,0.78 80.47± 3.5280.47 \,±\,3.52 18.75± 3.4618.75 \,±\,3.46 61.72± 4.3161.72 \,±\,4.31 !10 - 0.78± 0.780.78 \,±\,0.78 76.56± 3.7676.56 \,±\,3.76 18.75± 3.4618.75 \,±\,3.46 57.81± 4.6557.81 \,±\,4.65 ViT-S ARMRO (τ=1.4)(τ=1.4) 1 82.81 10.16± 2.6810.16 \,±\,2.68 80.47± 3.5280.47 \,±\,3.52 32.03± 4.1432.03 \,±\,4.14 48.44± 4.4348.44 \,±\,4.43 !10 - 6.25± 2.156.25 \,±\,2.15 74.22± 3.8874.22 \,±\,3.88 29.69± 4.0529.69 \,±\,4.05 44.53± 4.8144.53 \,±\,4.81 4 - 0.78± 0.780.78 \,±\,0.78 70.31± 4.0570.31 \,±\,4.05 15.62± 3.2215.62 \,±\,3.22 54.69± 4.4254.69 \,±\,4.42 !10 - 0.00± 0.000.00 \,±\,0.00 74.22± 3.8874.22 \,±\,3.88 15.62± 3.2215.62 \,±\,3.22 58.59± 4.3758.59 \,±\,4.37 Table 1: Accuracy under patch attacks and ARMRO, with and without 44 decoys. PatchFool and Adversarial Patch results on top and at the bottom, respectively, with different numbers of patches each. Gray rows indicate mask-aware variants. Δ reports accuracy drop induced by decoys. Attack Att.@@Block 5 Att.@@Block 12 w/ defense Figure 5: ImageNet qualitative example. adversarial patches only on top, adversarial and decoy patches on the bottom. Adversarial patches in green, decoys in white, and covered patches in red. 5.2 Main Results Tab. 1 reports the main results obtained by combining adversarial decoys with PatchFool and adversarial patch attacks. The number of attack tokens varies (11, 22, and 44 for PatchFool, and 11 and 44 for patch attack), while the number of decoy tokens is fixed to 44. In the table, we report the accuracy after attack (+Attack), after applying ARMRO (+Defense), and after applying ARMRO in the presence of adversarial decoys (+Decoy). The column Δ highlights the robust accuracy drop with defense, induced by decoys. The results show that ARMRO is effective against localized attacks in the absence of decoys, often restoring accuracy close to the clean level. However, this recovery is substantially reduced when decoys are introduced. Across architectures, attacks, and attack sizes, defended accuracy drops significantly, with Δ often exceeding 40%40\%. This indicates that decoys successfully redirect the attention-based suppression mechanism away from the true adversarial region. Importantly, this behavior can be observed for both attack types, supporting the attack-agnostic nature of the proposed mechanism. Finally, the mask-aware variants, highlighted in gray, provide, on average, a stronger evaluation setting by making the underlying attacks less fragile to defense-induced masking. Fig. 5 provides a qualitative illustration of the decoy mechanism using three decoy tokens combined with a three-token PatchFool attack. As shown in the second row, adding the decoy redirects the attention scores toward the decoy region, causing the defense to suppress decoy tokens instead of the truly adversarial ones. 5.3 Number of decoys and masking coverage We analyze the effect of varying the number of decoy tokens for DeiT and ViT-S in Fig. 6, while keeping the number of adversarial tokens fixed to 22 and 44 patches (first and second columns for each model, respectively). The first row reports the robust accuracy after ARMRO with different decoy budgets, whereas the second row shows defense mask coverage for adversarial and decoy tokens. Increasing the number of decoys generally improves the bypass: the defended accuracy decreases as more decoy tokens are introduced. The coverage analysis explains this behavior. With more decoys, the defense increasingly masks decoy tokens, while the coverage of the true adversarial tokens after adding the decoys drops substantially. Thus, decoys act by competing with the adversarial region for the attention-based suppression mask. This also reveals a trade-off, where larger decoy budgets improve defense bypassing, but increase the total number of manipulated tokens and may make the perturbation more visible. Therefore, the decoy budget controls the balance between bypass effectiveness and manipulation cost. (a) DeiT - #PF-mask Tokens 2 (b) DeiT - #PF-mask Tokens 4 (c) ViT-S - #PF-mask Tokens 2 (d) ViT-S - #PF-mask Tokens 4 Figure 6: Effect of decoy budget on defense bypassing (DeiT and ViT-S). Accuracy on top, and token-mask coverage on the bottom. Adversarial tokens fixed (computed with PatchFool-Mask) to 22 and 44, while decoys vary. Figure 7: Budget-matched comparison on DeiT. Full budget on attack depicted in blue, 22 patches reserved for decoys depicted in orange. Adaptive attack on the left, and PF-mask on the right. 5.4 Budget analysis and adaptive attack Although decoys are effective, they introduce additional tokens that the attacker must manipulate. Therefore, in this subsection, we evaluate whether, under a fixed manipulation budget, it is more effective to allocate tokens to the attack or to reserve part of them for the decoy. Results are shown in Fig. 7, where we compare two budget-matched strategies: the attacker uses all available tokens as adversarial ones (blue line), and a decoy-based strategy that reserves two tokens for decoys (orange line). On the left, PatchFool is shown, while on the right, the adaptive attack introduced in Sec. 5.1. The shaded area denotes the budget-matched regime, which starts at 33 tokens for the decoy-based strategy (i.e., just one adversarial token). In both settings, allocating part of the budget to decoys is more effective than using the entire budget for attack tokens alone. Within the budget-matched region, the decoy-based strategy consistently yields lower defended accuracy, and the gap increases with the total budget. On average, in this region, the adaptive attack achieves 57.19%57.19\% accuracy without decoys and 43.90%43.90\% with decoys. Similarly, mask-aware PatchFool achieves 59.35%59.35\% accuracy without decoys and 45.78%45.78\% with decoys. These results indicate that, against attention-based defenses, redirecting the suppression mask can be more beneficial than only strengthening the adversarial region. Importantly, this benefit is obtained through a sequential and attack-agnostic optimization. Comparisons for other models in App. C. 5.5 Ablation study The previous experiments evaluate the complete attack-defense pipeline. Here, we instead isolate the decoy optimization and study the contribution of its loss components. Since the objective is to make decoy tokens dominate the attention ranking across layers, we measure how often they appear among the top-k attention scores of each layer (k=4k=4, equal to the number of decoys). Tab. 2 shows the results of evaluating the ratio-based layer regulation (Eq.10), which down-weights layers that have already reached the target dominance threshold r. In the presence of the ratio, the average top-k decoy coverage increases from 65.33%65.33\% to 89.33%89.33\% on DeiT-B, from 59.92%59.92\% to 62.83%62.83\% on ViT-B, and from 76.17%76.17\% to 80.67%80.67\% on ViT-S, indicating that the regulation prevents the optimization from concentrating only on a subset of layers. Tab. 3 shows the influence of the ratio dominance term (Eq.11). Without it, decoys may still achieve high attention in some layers but rarely dominate the full top-k ranking, whereas the complete objective reaches full top-k coverage in multiple layers. Despite these positive results, forcing decoy tokens to reach the highest attention scores remains challenging in some layers, especially for ViT-B. We discuss this behavior in more detail in App. C.1. Model r-Reg. L0 L1 L2 L3 L4 L5 L6 L7 L8 L9 L10 L11 Avg. DeiT-B ✓ 89 42 74 100 100 100 98 96 86 91 96 100 89.33 ✗ 24 81 84 100 98 91 59 56 19 28 45 99 65.33 ViT-B ✓ 29 85 89 100 96 85 49 52 17 18 35 100 62.83 ✗ 21 79 84 100 96 83 45 46 17 18 31 100 59.92 ViT-S ✓ 88 97 93 100 100 100 96 43 42 46 64 100 80.67 ✗ 76 95 87 100 100 100 93 30 34 36 63 100 76.17 Table 2: Ablation of the layer-wise regulation term (r-Reg.), showing top-k decoy percentage coverage per layer. Best values in bold. Model Ratio L0 L1 L2 L3 L4 L5 L6 L7 L8 L9 L10 L11 #95 #100 DeiT-B ✓ 89 42 74 100 100 100 98 96 86 91 96 100 7 4 ✗ 39 29 42 87 95 96 86 81 74 71 66 84 2 0 ViT-B ✓ 29 85 89 100 96 85 49 52 17 18 35 100 3 2 ✗ 41 38 65 93 90 81 67 68 55 55 57 81 0 0 ViT-S ✓ 88 97 93 100 100 100 96 43 42 46 64 100 6 4 ✗ 25 36 56 77 90 92 87 85 83 81 81 84 0 0 Table 3: Ablation of the dominance ratio term, showing top-k decoy coverage per layer. Full coverage (100%100\%) in bold. 6 Conclusions This work introduced adversarial decoys, patches optimized to manipulate the attention distribution during inference. By decoupling attention manipulation from the adversarial objective, they can be combined with different adversarial attacks to misdirect attention-based defenses while preserving the effectiveness of the underlying attack. Analysis and results show that, although adversarial patches often attract high attention, attention distribution and adversarial effectiveness are not always aligned. The results demonstrate that decoys can be used to weaken test-time defenses, causing a degradation in defended accuracy across multiple adversarial patch-like attacks and against ARMRO [17], which, to the best of our knowledge, is the best existing attention-based defense. A budget-matched analysis (Sec. 5.4) also shows that allocating part of the attacker’s budget to decoy tokens is more effective than using the full budget for adversarial perturbations. Our findings highlight important limitations of adversarial defenses in ViTs and open up future work on applying decoy-based attacks to explainability methods and other applications that rely on attention as a measure of faithfulness. Ethical considerations. The proposed framework aims to expose limitations of attention-based defenses. Although adversarial decoys could be exploited for bypassing existing defenses, our hope is that these findings will encourage the design of defenses and analysis tools that remain effective against attention distribution shifts. References [1] S. Abnar and W. Zuidema (2020) Quantifying attention flow in transformers. In Proceedings of the 58th annual meeting of the association for computational linguistics, p. 4190–4197. Cited by: §2. [2] A. Aldahdooh, W. Hamidouche, and O. Deforges (2021) Reveal of vision transformers robustness against adversarial attacks. arXiv preprint arXiv:2106.03734. Cited by: §2. [3] P. Benz, S. Ham, C. Zhang, A. Karjauv, and I. S. Kweon (2021) Adversarial robustness comparison of vision transformer and mlp-mixer to cnns. arXiv preprint arXiv:2110.02797. Cited by: §2. [4] S. Bhojanapalli, A. Chakrabarti, D. Glasner, D. Li, T. Unterthiner, and A. Veit (2021) Understanding robustness of transformers for image classification. In Proceedings of the IEEE/CVF international conference on computer vision, p. 10231–10241. Cited by: §2. [5] T. B. Brown, D. Mané, A. Roy, M. Abadi, and J. Gilmer (2017) Adversarial patch. arXiv preprint arXiv:1712.09665. Cited by: §1, §2, §5.1, Table 1. [6] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko (2020) End-to-end object detection with transformers. In European conference on computer vision, p. 213–229. Cited by: §1. [7] H. Chefer, S. Gur, and L. Wolf (2021) Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 782–791. Cited by: §2. [8] T. Darcet, M. Oquab, J. Mairal, and P. Bojanowski (2024) Vision transformers need registers. In International conference on learning representations, Vol. 2024, p. 2632–2652. Cited by: §2. [9] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei (2009) Imagenet: a large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, p. 248–255. Cited by: 3rd item, §5.1. [10] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. (2020) An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. Cited by: §1, §5.1. [11] N. J. Eliopoulos, P. Jajal, J. C. Davis, G. Liu, G. K. Thiruvathukal, and Y. Lu (2025) Pruning one more token is enough: leveraging latency-workload non-linearities for vision transformers on the edge. In Proceedings of the Winter Conference on Applications of Computer Vision, p. 7153–7162. Cited by: §2. [12] Y. Fu, S. Zhang, S. Wu, C. Wan, and Y. Lin (2022) PATCH-fool: are vision transformers always robust against adversarial perturbations?. In Tenth International Conference on Learning Representations (ICLR 2022), Cited by: §1, §2, §3.2, §4.2, §5.1, Table 1. [13] J. Gu, V. Tresp, and Y. Qin (2022) Are vision transformers robust to patch perturbations?. In European Conference on Computer Vision, p. 404–421. Cited by: §2. [14] N. Jiang, A. Dravid, A. Efros, and Y. Gandelsman (2026) Vision transformers don’t need trained registers. Advances in neural information processing systems 38, p. 56557–56595. Cited by: §2. [15] R. Kazoom, A. Goldberg, H. Cohen, and O. Hadar (2026) Seeing isn’t believing: context-aware adversarial patch synthesis via conditional gan. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, p. 202–211. Cited by: §2. [16] K. Lis, M. Rottmann, A. Mütze, S. Honari, P. Fua, and M. Salzmann (2022) AttEntropy: on the generalization ability of supervised semantic segmentation transformers to new objects in new domains. arXiv preprint arXiv:2212.14397. Cited by: §2. [17] L. Liu, Y. Guo, Y. Zhang, and J. Yang (2023) Understanding and defending patched-based adversarial attacks for vision transformer. In International Conference on Machine Learning, p. 21631–21657. Cited by: §1, §2, §2, §3.2, §5.1, §6. [18] G. Lovisotto, N. Finnie, M. Munoz, C. K. Mummadi, and J. H. Metzen (2022) Give me your attention: dot-product attention considered harmful for adversarial patch robustness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 15234–15243. Cited by: §1, §2. [19] W. Luo, Y. Li, R. Urtasun, and R. Zemel (2016) Understanding the effective receptive field in deep convolutional neural networks. In Advances in Neural Information Processing Systems, D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (Eds.), Vol. 29, p. . Cited by: §2. [20] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu (2018) Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, Cited by: §4.2. [21] N. Mu and D. Wagner (2021) Defending against adversarial patches with robust self-attention. In ICML 2021 workshop on uncertainty and robustness in deep learning, Vol. 1. Cited by: §1, §2. [22] M. M. Naseer, K. Ranasinghe, S. H. Khan, M. Hayat, F. Shahbaz Khan, and M. Yang (2021) Intriguing properties of vision transformers. Advances in Neural Information Processing Systems 34, p. 23296–23308. Cited by: §2. [23] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. (2024) DINOv2: learning robust visual features without supervision. Transactions on Machine Learning Research Journal. Cited by: §1. [24] S. Paul and P. Chen (2022) Vision transformers are robust learners. In Proceedings of the AAAI conference on Artificial Intelligence, Vol. 36, p. 2071–2081. Cited by: §2. [25] G. M. Pietrosanti, G. Rossolini, A. Biondi, and G. Buttazzo (2025) Benchmarking the spatial robustness of dnns via natural and adversarial localized corruptions. Pattern Recognition, p. 112412. Cited by: §1, §2. [26] G. Rossolini, F. Nesti, F. Brau, A. Biondi, and G. Buttazzo (2023) Defending from physically-realizable adversarial attacks through internal over-activation analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, p. 15064–15072. Cited by: §1, §2. [27] R. Shao, Z. Shi, J. Yi, P. Chen, and C. Hsieh (2022) On the adversarial robustness of vision transformers. In Annual Conference on Neural Information Processing Systems, Cited by: §2. [28] H. P. Silva, F. Becattini, and L. Seidenari (2025) Attacking attention of foundation models disrupts downstream tasks. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 3534–3543. Cited by: §1, §2. [29] O. Siméoni, G. Puy, H. V. Vo, S. Roburin, S. Gidaris, A. Bursuc, P. Pérez, R. Marlet, and J. Ponce (2021) Localizing objects with self-supervised transformers and no labels. In BMVC 2021-32nd British Machine Vision Conference, Cited by: §2. [30] R. Strudel, R. Garcia, I. Laptev, and C. Schmid (2021) Segmenter: transformer for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, p. 7262–7272. Cited by: §1. [31] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou (2021) Training data-efficient image transformers & distillation through attention. In International conference on machine learning, p. 10347–10357. Cited by: §5.1. [32] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. In Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30, p. . Cited by: §1. [33] C. Wu, J. Lin, Y. H. Hu, and P. Morgado (2025) Patch ranking: token pruning as ranking prediction for efficient clip. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Vol. , p. 5842–5851. External Links: Document Cited by: §2. [34] K. Xu, G. Zhang, S. Liu, Q. Fan, M. Sun, H. Chen, P. Chen, Y. Wang, and X. Lin (2020) Adversarial t-shirt! evading person detectors in a physical world. In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V, A. Vedaldi, H. Bischof, T. Brox, and J. Frahm (Eds.), Lecture Notes in Computer Science, Vol. 12350, p. 665–681. External Links: Link, Document Cited by: §2. [35] C. Yu, J. Chen, Y. Xue, Y. Liu, W. Wan, J. Bao, and H. Ma (2021) Defending against universal adversarial patches by clipping feature norms. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 16434–16442. Cited by: §1, §2. Supplementary materials for the paper “Adversarial Decoys: Misdirecting Attention-Based Defenses in ViT” Giulia Marchiori Pietrosanti Giulio Rossolini Giorgio Buttazzo Appendix A Additional analysis on the role of attention We provide the derivation of Proposition 1 and further analyze the relationship between attention concentration and adversarial effect. The goal is to clarify why increasing the attention received by an adversarial token can amplify its influence, while also showing why attention alone is not a sufficient indicator of adversarial relevance. Proof of Proposition 1. Consider a self-attention head and a query token i. The output of the head is given by oi=∑j=0n−1Ai,jvj,o_i= _j=0^n-1A_i,jv_j, where vjv_j is the value representation of token j, and Ai,j=exp(zi,j)∑k=0n−1exp(zi,k).A_i,j= (z_i,j) _k=0^n-1 (z_i,k). Let a be an arbitrary token of interest. We separate the contribution of token a from the remaining tokens: oi=Ai,ava+∑j≠aAi,jvj.o_i=A_i,av_a+ _j≠ aA_i,jv_j. Define Z¬a=∑k≠aexp(zi,k)Z_ a= _k≠ a (z_i,k) and the renormalized aggregation over all tokens except a as μi,¬a=∑j≠aexp(zi,j)Z¬avj. _i, a= _j≠ a (z_i,j)Z_ av_j. Since Ai,a=exp(zi,a)exp(zi,a)+Z¬a,A_i,a= (z_i,a) (z_i,a)+Z_ a, the total attention assigned to all tokens different from a is ∑j≠aAi,j=1−Ai,a. _j≠ aA_i,j=1-A_i,a. Moreover, for every j≠aj≠ a, we can write Ai,j=(1−Ai,a)exp(zi,j)Z¬a.A_i,j=(1-A_i,a) (z_i,j)Z_ a. Therefore, ∑j≠aAi,jvj=(1−Ai,a)∑j≠aexp(zi,j)Z¬avj=(1−Ai,a)μi,¬a. _j≠ aA_i,jv_j=(1-A_i,a) _j≠ a (z_i,j)Z_ av_j=(1-A_i,a) _i, a. (13) Substituting this expression into the attention output oio_i gives the result of Proposition 1 oi=Ai,ava+(1−Ai,a)μi,¬a.o_i=A_i,av_a+(1-A_i,a) _i, a. (14) Studying attention by constraining the optimization. The previous decomposition oi−μi,¬a=Ai,a(va−μi,¬a)o_i- _i, a=A_i,a (v_a- _i, a ) shows that the effect of token a on the output of query token i depends on two distinct factors. The first factor is the attention coefficient Ai,aA_i,a, which controls how strongly token a is weighted. The second factor is the displacement term va−μi,¬av_a- _i, a, which measures how different the value representation of token a is from the aggregate representation of the remaining tokens. This has an important consequence: attention is an amplifier, but it is not the source of the harmful direction itself. A token can receive high attention while having a value representation close to μi,¬a _i, a, in which case its effect on the attention output remains limited. Conversely, a token can have a highly disruptive value representation, but its effect can remain weak if the corresponding attention coefficient is small. However, despite these theoretical insights, adversarial tokens often show high attention, meaning that in practice, this is necessary to boost the effectiveness. To further study this phenomenon, we analyze whether explicitly increasing or decreasing the attention assigned to adversarial tokens affects the effectiveness of a localized attack. In particular, we consider PatchFool-style optimization and vary the strength and sign of the attention regularization term. Let ⊆ℐP denote the set of adversarial tokens modified by the attack, and let LDL_D be the set of layers used to compute the attention objective. For a layer ℓ , we define the attention received by the adversarial tokens as Sℓ=1||∑p∈1H||∑h=1H∑q∈Aq,pℓ,h,S_P = 1|P| _p 1H|Q| _h=1^H _q A_q,p ,h, (15) where H is the number of attention heads, Q is the set of image-token queries, and Aq,pℓ,hA_q,p ,h is the attention assigned by query token q to adversarial token p at layer ℓ and head h. We then define the attention objective as ℒatt=1|LD|∑ℓ∈LDSℓ.L_att= 1|L_D| _ ∈L_DS_P . (16) The complete attack objective is maxδℒCE(f(xδ),y)+λattℒatt, _δ\;L_CE(f(x_δ),y)+ _attL_att, where xδx_δ is the perturbed image, ℒCEL_CE is the classification loss, and λatt _att controls the role of the attention term. Positive values of λatt _att encourage the adversarial tokens to receive higher attention, while negative values penalize attention concentration on the adversarial tokens. Therefore, by sweeping λatt _att, we can evaluate whether attention is necessary for attack success or whether it mainly acts as an amplification mechanism. Here, we use layer 55 as in PatchFool, to compute the attention term and evaluate both the attack success rate and the attention ratio R=Sℓmaxj∉Sjℓ+ε,R_P= S _P _j S _j+ , where this ratio measures whether the adversarial tokens become dominant with respect to the rest of the image. (a) DeiT - 1 PatchFool Token (b) DeiT - 2 PatchFool Tokens (c) ViTB - 1 PatchFool Token (d) ViTB - 2 PatchFool Tokens Figure 9: Analysis of the attack success rate and attention ratio with the attacked tokens and others across different configuration of PatchFool, for different attention weights settings at layer 5. Figure 9 reports the results for DeiT-B and ViT-B when using one or two PatchFool tokens. When the attention weight is strongly negative, the attention ratio remains low (x axis) and the attack success rate (y axis) is also limited. As the weight approaches zero, both the attention ratio and the attack success rate increase sharply. This suggests that suppressing the attention received by adversarial tokens makes the localized attack less effective, supporting the idea that attention can amplify the adversarial contribution. However, the curves also show that the relation is not purely monotonic. In several cases, the attack success rate reaches its maximum around small positive values of the attention weight, while larger positive values further increase the attention ratio without improving, and sometimes slightly reducing, the attack success rate. This behavior supports the interpretation given by Proposition 1: increasing attention is useful only when the value displacement induced by the adversarial token remains aligned with a harmful direction for the classifier. If the optimization focuses too strongly on attention, it may produce tokens that are highly attended but not proportionally more harmful. (a) DeiT - 1 PatchFool Token (b) DeiT - 2 PatchFool Tokens (c) DeiT - 4 PatchFool Tokens (d) ViTB - 1 PatchFool Token (e) ViTB - 2 PatchFool Tokens (f) ViTB - 4 PatchFool Tokens Figure 12: Effect of mask-aware optimization on localized PatchFool attacks. The x-axis reports the mask ratio used during optimization. Blue curves report attack success on the clean image inference, while orange curves report attack success under random-mask inference. Appendix B Mask-Aware adversarial optimization. As discussed in the main paper, classical localized attacks can achieve high attack success rates in a clean setting, but their effectiveness can drop significantly when random image tokens are masked, even when the masked tokens do not directly overlap with the adversarial region. This is particularly relevant for our setting: if an attention-based defense is redirected toward decoy tokens, the underlying adversarial attack remains effective only if it is sufficiently robust to the suppression of other image tokens. Therefore, to improve the robustness of localized adversarial attacks against masking-based defenses, we consider a mask-aware optimization strategy. The goal is to optimize the adversarial region under stochastic token suppression, mimicking the type of masking that may be applied at inference time. This formulation is generic and can be applied to different localized attacks, including adversarial patch attack and PatchFool. To account for random masking during optimization, we sample at each iteration a binary random mask r∼ℛρ,r _ρ, where ρ denotes the masking ratio. The mask r∈0,1Nr∈\0,1\^N identifies the pixels or tokens that remain visible after stochastic suppression. The randomly masked adversarial image is then Tr(xδ)=r⊙xδ+(1−r)⊙pmask,T_r(x_δ)=r x_δ+(1-r) p_mask, (17) where pmaskp_mask is the replacement value used for masked regions, such as zero, gray, or the image mean. The mask-aware objective optimizes the perturbation over the expectation induced by the masking distribution: maxδr∼ℛρ[ℒadv(f(Tr(xδ)),y)], _δ\;E_r _ρ [L_adv (f(T_r(x_δ)),y ) ], (18) where ℒadvL_adv is the adversarial objective. This formulation can be interpreted as a robustness-oriented optimization of the localized attack. Instead of optimizing the perturbation only for the original image, the attack is optimized over a distribution of partially masked images. As a result, the perturbation is encouraged to remain effective even when some tokens are removed or replaced. Figure 12 reports the effect of mask-aware optimization for PatchFool attacks with different token budgets. The x-axis denotes the masking ratio used during optimization, while the curves report the attack success rate in the undefended setting and under random-mask inference with a masking ratio of 10%10\%. The results show that introducing a moderate masking ratio during optimization generally improves the effectiveness of the attack under random-mask inference. This effect is particularly evident when multiple PatchFool tokens are used. For example, with two or four tokens, the attack success rate under random-mask inference increases substantially compared to the case where no random masking is used during optimization. The behavior also reveals a trade-off. When the masking ratio is too small, the attack is optimized mainly for the unmasked image and can be less robust to stochastic suppression at inference time. Conversely, when the masking ratio becomes too large, the optimization may become overly conservative or less aligned with the original undefended objective, leading to a reduction in attack success without defense. Therefore, intermediate masking ratios provide a better compromise between robustness to random masking and standard attack effectiveness. Based on this analysis, in the main experiments, we use a token-level masking probability of 0.050.05 during attack optimization (excluding tokens associated with the adversarial attacks). Appendix C Additional Results C.1 Results with top-k analysis Number of decoy ablation. Table 4 reports the results of the top-k analysis under different numbers of decoy tokens (11, 22, and 44). It can be noticed, in bold, that increasing the number of tokens leads to slightly improved coverage in deep layers. These results are consistent with the results extrapolated in the defense setting, where such improvement translates into more effective steering behavior. # Patches Model L0 L1 L2 L3 L4 L5 L6 L7 L8 L9 L10 L11 1 DeiT 98 71 82 100 100 100 97 91 74 75 53 99 ViT-B 70 95 91 99 87 59 17 24 1 2 19 95 ViT-S 100 100 100 100 100 100 68 16 19 30 48 93 2 DeiT 96 49 76 100 100 100 98 96 75 79 73 100 ViT-B 61 94 89 100 94 77 32 37 3 3 27 98 ViT-S 97 100 98 100 100 100 82 21 22 32 51 95 4 DeiT 89 42 74 100 100 100 98 96 86 91 96 100 ViT-B 29 85 89 100 96 85 49 52 17 18 35 99 ViT-S 88 97 93 100 100 100 96 43 42 46 64 99 Table 4: Results of the top-k overlap (%) analysis across layers for 11, 22, and 44 optimized decoy patches. Bold indicates the highest value for the same model across patch counts. All vs single layer. Table 5 extends the top-k analysis by comparing joint optimization across all layers with optimization performed on each layer individually. In this case, experiments are conducted using 44 adversarial decoy tokens on the three ViT architectures considered in this work. As expected, optimizing each layer independently leads to higher top-k coverage analyzed on the same layer, since the optimization no longer needs to satisfy competing objectives across different depths of the architectures. However, an interesting trend can be noticed: the layers that are difficult to optimize in the joint setting remain the most challenging even when optimized in isolation. This behavior is particularly evident for ViT-B and ViT-S, where several intermediate layers consistently exhibit lower decoy coverage in both settings. These results suggest that promoting the decoy tokens to the top of the attention ranking is intrinsically more difficult at specific depths of the architecture, rather than being a consequence of joint optimization across layers. Therefore, simply focusing on the most challenging layers does not eliminate their inherent optimization difficulty. That said, although informative for understanding the optimization dynamics, single-layer optimization is of limited practical interest for bypassing attention-based defenses. Indeed, defenses typically exploit attention patterns across multiple layers, and optimizing decoys for only one layer does not produce a sufficiently consistent shift throughout the overall network. Model L0 L1 L2 L3 L4 L5 L6 L7 L8 L9 L10 L11 DeiT all 89 42 74 100 100 100 98 96 86 91 96 100 DeiT sep 100 100 100 100 100 100 100 100 100 100 100 100 ViT-B all 29 85 89 100 96 85 49 52 17 18 35 99 ViT-B sep 100 100 100 100 100 100 97 90 67 50 84 100 ViT-S all 88 97 93 100 100 100 96 43 42 46 64 99 ViT-S sep 100 100 100 100 100 100 100 100 98 94 96 100 Table 5: Top-k overlap (%) per layer obtained by optimizing 44 decoys independently for each layer, for DeiT, ViT-B, and ViT-S. The top-k analysis, however, only highlights whether the targeted tokens manage to appear among the highest-ranked ones. Understanding how much attention is actually shifted toward these tokens can also be insightful. Figure 13 compares the top-k analysis with the attention-shift metric, computed as follows: we first normalize the attention distributions after excluding the CLS token, and then measure the difference in the total attention mass assigned to the target tokens before and after applying the decoy tokens. It is worth noting that, due to the softmax normalization, attention is redistributed across tokens rather than accumulated independently on a given subset. As a result, even strong shifts in attention toward the target tokens remain bounded and do not necessarily reach saturation. Nevertheless, meaningful patterns can still be observed. In particular, while the top-k analysis shows that promoting target tokens to the highest-attended positions in deeper layers can be challenging, the shift metric reveals a consistent increase across layers, especially in the ViT model. This suggests that, even when target tokens do not enter the top-k set, attention is still progressively redistributed toward them as the network depth increases. (g) DeiT – optimization all-layers (h) DeiT – optimization single-layer (i) ViT – optimization all-layers (j) ViT – optimization single-layer Figure 13: Top-44 overlap and attention shift trend comparison on target tokens across layers. The top row shows DeiT, while the bottom row shows ViT-B, on the left, results computed on all layers optimization, while on the right on the single layer optimization. C.2 Optimization details Regarding the number of steps required to optimize the decoys, we acknowledge that jointly optimizing a layer-wise objective across multiple transformer layers is a challenging process. For stability, we therefore use a relatively low learning rate, set to 0.010.01, which requires a larger number of optimization steps to obtain effective decoy behavior. Figure 14 reports the average evolution of the decoy attention score and the corresponding ratio across layers during optimization. The plots show that most of the improvement occurs in the early phase, while the curves continue to increase more gradually as the number of steps grows, suggesting that longer optimization can further strengthen the decoy effect. In the main experiments, we use 2500 optimization steps as a trade-off between computational cost and decoy effectiveness. Nevertheless, these results indicate that stronger performance could potentially be achieved by increasing the number of optimization steps, especially when computational resources are less constrained. Figure 14: Evolution of decoy optimization across layers. The plots show the average decoy attention score (top) and decoy-to-competing-token ratio (bottom) over optimization steps, with shaded regions indicating variability across runs. C.3 Analysis with other defenses Table 6 reports supplementary results obtained with a stronger top-k layer-wise masking baseline. Here, we use original adversarial patch attacks with patch tokens 11 and 44. Unlike ARMRO, which applies a more selective masking strategy, this baseline independently selects the top-33 most attended tokens at each transformer layer and suppresses the union of the selected tokens. As a consequence, the number of masked tokens can be substantially larger than in ARMRO, depending on the overlap between the tokens selected across layers. As shown in the table, this behavior has two effects. On the one hand, masking a larger set of tokens can still provide stronger protection under attack, and even under decoy. On the other hand, this comes with a higher risk of suppressing useful image content, potentially reducing the reliability of the prediction and making the defense less selective. This trade-off is especially important in our setting, since the defense may mask not only adversarial or decoy tokens, but also benign tokens that contribute to the correct classification. The lower effectiveness observed in some cases, especially when using a larger number of patch tokens, is mainly due to the limited robustness of the underlying localized attack against aggressive masking strategies. In particular, when the defense suppresses many tokens, the attack may lose part of the spatial or semantic structure required to preserve the adversarial effect. This motivates the mask-aware optimization introduced in Appendix B, which explicitly optimizes the attack under random token suppression. The results suggest that improving the robustness of localized attacks against broader masking operations is an important direction for future work, especially for defenses that suppress more tokens than ARMRO or operate through layer-wise token aggregation. # Patch Tokens Clean ↑ +Attack ↓ +Defense ↑ +Decoy ↓ DeiT-B Top-3 layer-wise masking 1 84.87 12.50 84.8 43.1 4 84.87 5.47 70.31 53.91 ViT-B Top-3 layer-wise masking 16 86.55 21.09 85.00 35.10 32 86.55 0.78 71.09 50.00 ViT-S Top-3 layer-wise masking 16 82.81 10.16 83.00 27.00 32 82.81 0.78 54.69 29.69 Table 6: Results under adversarial patch attacks using a top-3 layer-wise masking defense. The defense suppresses the union of the top-3 most-attended tokens, selected at each layer. Four decoy tokens are used, as in the main experiments in Sec. 5.2. C.4 Position analysis Results in Table 7 show the difference between applying decoy tokens randomly, as done in the main experiments, and placing them in the top-left corner of the image. This analysis aims to assess whether a fixed position, especially in a corner of the image where positional encodings may play a different role in the attention process, can affect the quality of the decoy. As shown in the results, when using a small number of decoys, e.g., one decoy token, placing them in an extreme corner can reduce their ability to control attention compared with random placement. However, this effect becomes less pronounced as the number of decoys increases. With four decoy tokens, the difference in attack success rate becomes negligible. On the contrary, the top-k analysis shows slightly different results. Table 8 shows that, even in a corner position, fixed decoy patches are capable of maximizing attention in their area. This is likely due to the fact that fixing the position also arranges the patches into a compact square rather than leaving them sparse, and as the only attention attractors, since here we do not have the underlying attack, they probably work better when grouped than when separated, showing slightly different behaviors in the presence or absence of adversary-attacked regions. This suggests that future work could further investigate improved decoy placement strategies, possibly considering an attention pre-analysis similar to PatchFool, where the attention of the clean image is first analyzed to determine where to place the attack tokens. We hope to further explore this direction in future work, including the possibility of optimizing the decoy jointly with the attack, rather than relying on the sequential but versatile approach adopted in the main experiments. # Decoy tokens Random position ↓ Fixed position ↓ DeiT-B ARMRO (τ=1.2)(τ=1.2) 1 54.69± 4.42 [rgb]0,0,0.6554.69 \,±\,4.42 59.38± 4.36 [rgb]0.7,0,059.38 \,±\,4.36 2 45.31± 4.42 [rgb]0,0,0.6545.31 \,±\,4.42 47.66± 4.43 [rgb]0.7,0,047.66 \,±\,4.43 4 27.34± 3.96 [rgb]0,0,0.6527.34 \,±\,3.96 27.34± 3.96 [rgb]0.7,0,027.34 \,±\,3.96 ViT-S ARMRO (τ=1.4)(τ=1.4) 1 31.25± 4.11 [rgb]0,0,0.6531.25 \,±\,4.11 42.97± 4.39 [rgb]0.7,0,042.97 \,±\,4.39 2 18.75± 3.46 [rgb]0,0,0.6518.75 \,±\,3.46 27.34± 3.96 [rgb]0.7,0,027.34 \,±\,3.96 4 15.62± 3.22 [rgb]0,0,0.6515.62 \,±\,3.22 15.62± 3.22 [rgb]0.7,0,015.62 \,±\,3.22 Table 7: Effect of decoy placement under ARMRO when using four mask-aware adversarial patches placed at the center of the image. We compare random decoy placement with fixed placement at the first image-token position, corresponding to the top-left corner of the image. Model L0 L1 L2 L3 L4 L5 L6 L7 L8 L9 L10 L11 DeiT rand pos 89 42 74 100 100 100 98 96 86 91 96 100 DeiT fix pos 98 57 93 100 100 100 100 98 99 99 98 100 ViT-B rand pos 29 85 89 100 96 85 49 52 17 18 35 99 ViT-B fix pos 68 93 94 99 98 90 36 45 19 20 25 100 ViT-S rand pos 88 97 93 100 100 100 96 43 42 46 64 99 ViT-S fix pos 92 100 91 100 100 100 97 76 77 81 92 100 Table 8: Top-k overlap (%) per layer for DeiT, ViT-B, and ViT-S maintaining the random positioning (rand pos) or by fixing the position to the left top corner (fix pos) of 44 decoy tokens. C.5 Adaptive attack and budget analysis. The adaptive attack optimization used in Sec. 5.4 and in the following experiments follows a similar idea to the optimization presented in App. A, where we study the effect of reducing attention while preserving adversarial effectiveness. In particular, we consider all network layers, since defenses such as ARMRO are applied across all layers. We set λ to 0.010.01, which provides the best empirical compromise between reducing attention and avoiding an excessive degradation of adversarial effectiveness. Figure 16 reports an additional budget-matched analysis comparing the use of all available tokens for the attack with the proposed setting, where part of the token budget is reserved for decoys. This analysis is important because decoy tokens increase the total number of modified tokens, and therefore the comparison should also account for attacks that use the same overall budget entirely for adversarial optimization. Across both ViT-B and ViT-S, and for both adaptive adversarial patches and PatchFool, increasing the number of attack tokens generally improves the attack effectiveness, as expected. However, the results show that reserving part of the budget for decoys remains competitive, and in several cases more effective, than allocating the entire budget to adversarial tokens only. These results indicate that the advantage of decoys is not merely due to increasing the perturbation budget, but rather to their ability to redirect the masking defense away from the truly adversarial regions. Figure 16: Budget-matched comparison on ViT-B (top) and ViT-S (bottom), between using all tokens for the attack and reserving tokens for decoys. C.6 Additional illustration Finally, Figure 17 reports an illustration of the attention activation across layers for different versions of the image: clean, attacked, attack with defense, attack with decoy, and attack with decoy and defense. Figure 17: Qualitative illustration of the attention activations across ViT layers for different image configurations. Each row corresponds to a transformer layer, while columns show the clean image, the attacked image, the attacked image after defense, the attack with decoys, and the attack with decoys after defense.