Paper deep dive
Pushing the Frontier of Black-Box LVLM Attacks via Fine-Grained Detail Targeting
Xiaohan Zhao, Zhaoyi Li, Yaxin Luo, Jiacheng Cui, Zhiqiang Shen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/20/2026, 11:28:08 PM
Summary
This paper introduces M-Attack-V2, a modular enhancement to the M-Attack framework for black-box adversarial attacks on Large Vision-Language Models (LVLMs). The authors identify that local crop-level matching in prior methods induces high-variance, nearly orthogonal gradients due to ViT translation sensitivity and structural asymmetry between source and target crops. M-Attack-V2 addresses this through Multi-Crop Alignment (MCA) to average gradients from multiple views, Auxiliary Target Alignment (ATA) to create a smoother target manifold using semantically correlated images, Patch Momentum to replay historical gradients, and Patch Ensemble+ (PE+) for refined patch-size selection. These components significantly improve attack success rates on frontier models like GPT-5, Gemini-2.5-Pro, and Claude-4.0.
Entities (11)
Relation Signals (11)
M-Attack-V2 → achievessuccessrateon → GPT-5
confidence 95% · boosting success rates on ... GPT-5 from 98% to 100%
M-Attack-V2 → achievessuccessrateon → Claude-4.0
confidence 95% · boosting success rates on Claude-4.0 from 8% to 30%
M-Attack-V2 → achievessuccessrateon → Gemini 2.5 Pro
confidence 95% · boosting success rates on ... Gemini-2.5-Pro from 83% to 97%
M-Attack-V2 → containsmodule → Multi-Crop Alignment
confidence 95% · On the source side, Multi-Crop Alignment (MCA) averages gradients... Together these modules form M-Attack-V2
M-Attack-V2 → containsmodule → Auxiliary Target Alignment
confidence 95% · On the target side, Auxiliary Target Alignment (ATA)... Together these modules form M-Attack-V2
M-Attack-V2 → improves → M-Attack
confidence 95% · M-Attack-V2 is a simple, modular enhancement over M-Attack
ViT → causes → high-variance gradients
confidence 90% · We attribute this to (i) ViT translation sensitivity that yields spike-like gradients
M-Attack-V2 → containsmodule → Patch Ensemble+
confidence 90% · combined with a refined patch-size ensemble (PE+)... Together these modules form M-Attack-V2
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Black-box adversarial attacks on Large Vision-Language Models (LVLMs) are challenging due to missing gradients and complex multimodal boundaries. While prior state-of-the-art transfer-based approaches like M-Attack perform well using local crop-level matching between source and target images, we find this induces high-variance, nearly orthogonal gradients across iterations, violating coherent local alignment and destabilizing optimization. We attribute this to (i) ViT translation sensitivity that yields spike-like gradients and (ii) structural asymmetry between source and target crops. We reformulate local matching as an asymmetric expectation over source transformations and target semantics, and build a gradient-denoising upgrade to M-Attack. On the source side, Multi-Crop Alignment (MCA) averages gradients from multiple independently sampled local views per iteration to reduce variance. On the target side, Auxiliary Target Alignment (ATA) replaces aggressive target augmentation with a small auxiliary set from a semantically correlated distribution, producing a smoother, lower-variance target manifold. We further reinterpret momentum as Patch Momentum, replaying historical crop gradients; combined with a refined patch-size ensemble (PE+), this strengthens transferable directions. Together these modules form M-Attack-V2, a simple, modular enhancement over M-Attack that substantially improves transfer-based black-box attacks on frontier LVLMs: boosting success rates on Claude-4.0 from 8% to 30%, Gemini-2.5-Pro from 83% to 97%, and GPT-5 from 98% to 100%, outperforming prior black-box LVLM attacks. Code and data are publicly available at: this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2602.17645v1
- Canonical: https://arxiv.org/abs/2602.17645v1
Trouble viewing inline? Open PDF directly →
Full Text
93,250 characters extracted from source content.
Expand or collapse full text
Pushing the Frontier of Black-Box LVLM Attacks via Fine-Grained Detail Targeting Abstract Black-box adversarial attacks on Large Vision-Language Models (LVLMs) are challenging due to missing gradients and complex multimodal boundaries. While prior state-of-the-art transfer-based approaches like - M-Attack performs well using local crop-level matching between source and target images, we find this induces high-variance, nearly orthogonal gradients across iterations, violating coherent local alignment and destabilizing optimization. We attribute this to (i) ViT translation sensitivity that yields spike-like gradients and (i) structural asymmetry between source and target crops. We reformulate local matching as an asymmetric expectation over source transformations and target semantics, and build a gradient denoising upgrade to - M-Attack. On the source side, Multi-Crop Alignment (MCA) averages gradients from multiple independently sampled local views per iteration to reduce variance. On the target side, Auxiliary Target Alignment (ATA) replaces aggressive target augmentation with a small auxiliary set from a semantically correlated distribution, producing a smoother, lower-variance target manifold. We further reinterpret momentum as Patch Momentum, replaying historical crop gradients, combined with a refined patch-size ensemble (PE+), this strengthens transferable directions. Together these modules form our -- M-Attack-V2 in this work, a simple, modular enhancement over - M-Attack that substantially improves transfer-based black-box attacks on frontier LVLMs: boosting success rates on Claude-4.0 () from 8%→ 30%, Gemini-2.5-Pro () from 83%→ 97%, and GPT-5 () from 98%→ 100%, outperforming all prior black-box LVLM attacks. Code and data are publicly available at this link. Machine Learning, ICML Xiaohan Zhao, Zhaoyi Li, Yaxin Luo, Jiacheng Cui, Zhiqiang Shen† VILA Lab, Department of Machine Learning, MBZUAI https://vila-lab.github.io/M-Attack-V2-Website/ xiaohan.zhao,zhaoyi.li,yaxin.luo,jiacheng.cui,zhiqiang.shen@mbzuai.ac.ae @noticeprintedtrue†footnotetext: †Corresponding Author. Figure 1: Improvement of -- M-Attack-V2 over - M-Attack on strong and up-to-date commercial black-box models (Claude 4, Gemini 2.5 and GPT-5). ASR and KMR stand for attack success rate and keyword matching rate, respectively. 1 Introduction Large Vision-Language Models (LVLMs) have become foundational to modern AI systems, enabling multimodal tasks like image captioning (Hu et al., 2022; Salaberria et al., 2023; Chen et al., 2022b; Tschannen et al., 2023), VQA (Luu et al., 2024; Özdemir and Akagündüz, 2024), and visual reasoning (OpenAI, 2025). However, their visual modules remain vulnerable to adversarial attacks, subtle perturbations that mislead models while remaining imperceptible to humans. Prior efforts, including AttackVLM (Zhao et al., 2023), CWA (Chen et al., 2024), SSA-CWA (Dong et al., 2023a), AdvDiffVLM (Guo et al., 2024), and most effectively, recent state-of-the-art - M-Attack (Li et al., 2025), which have exploited and addressed this weakness through local-level matching and surrogate model ensembles, surpassing 90% success rates on models like GPT-4o. Despite promising performance of - M-Attack, our analysis reveals that its gradient signals are highly unstable: Even overlapping large pixel regions, two consecutive local crops share nearly orthogonal gradients. In other words, high similarity in the pixel and embedding spaces does not translate to high similarity in the gradient space. The reason is that ViTs’ gradient pattern is sensitive to translation. A tiny shift changes pixels contained in each token, altering self-attention. Moreover, patch-wise, spike-like gradient amplifies the mismatch within just a few pixels. We counter this effect by aggregating gradients from multiple crops within the same iteration, a strategy we call Multi‑Crop Alignment (MCA). From a theoretical angle, MCA aggregates gradients across multiple views in a single iteration, smoothing local inconsistencies and improving cross-crop gradient stability. We further observe that the source and target transformations in - M-Attack operate in different semantic spaces: one emphasizing extraction, the other generalization. Aggressive target augmentation introduces harmful variance. Our Auxiliary Target Alignment (ATA) mitigates this by identifying semantically similar auxiliary images to create a low-variance embedding subspace, then applying only mild shifts to enhance transferability without destabilizing the optimization. Classic momentum is reinterpreted under this framework as Patch Momentum (PM), a replay mechanism that recycles past gradients across random crops to stabilize optimization. In parallel, we also re-examine and refine - M-Attack’s model selection criterion and choose a delicately selected ensemble set with diverse patch sizes to mitigate the difficulty in cross-patch transfer, of which we find that the attention concentrates more on the main object. We term it Patch Ensemble+ (PE+) in our approach. Together, these proposed components form the basis of our -- M-Attack-V2, a robust gradient denoising framework that significantly outperforms existing black-box attack methods. Our method raises attack success rates from 98%→ 100% on GPT‑5, 8%→ 30% on Claude‑4, and 83%→ 97% on Gemini‑2.5‑Pro, achieving state-of-the-art performance across the board. This study not only offers a practical, modular attack strategy but also sheds light on the gradient behavior of ViT-based LVLMs under local perturbations. To summarize, our contributions are: • We show for the first time that crop-level matching yields high-variance, near-orthogonal gradients (from ViT translation sensitivity and source/target crop asymmetry), destabilizing black-box optimization. • We recast local matching as an asymmetric expectation and introduce MCA (multi-view gradient averaging) + ATA (auxiliary semantically correlated targets) to reduce variance and smooth the target manifold. • We add Patch Momentum + refined PE+ to amplify transferable directions, delivering large ASR gains on frontier LVLMs (e.g., Claude-4.0 8%→30%, Gemini-2.5-Pro 83%→97%, GPT-5 98%→100%). 2 Related Work Large Vision Language Models. Transformer-based LVLMs learn visual-semantic representations from large-scale image-text data, enabling tasks like image captioning (Salaberria et al., 2023; Hu et al., 2022; Chen et al., 2022b; Tschannen et al., 2023), visual QA (Luu et al., 2024; Özdemir and Akagündüz, 2024), and cross-modal reasoning (Wu et al., 2025; Ma et al., 2023; Wang et al., 2024). Open-source models such as BLIP-2 (Li et al., 2022), Flamingo (Alayrac et al., 2022), and LLaVA (Liu et al., 2023) show strong benchmark performance. Commercial models like GPT-4o, Claude-3.5 (Anthropic, 2024), and Gemini-2.0 (Team et al., 2023) offer advanced reasoning and real-world adaptability, with their successors, GPT-o3 (OpenAI, 2025), Claude 3.7-Sonnet (Anthropic, 2025), and Gemini-2.5-Pro, able to reason over both text and images. (a) Gradient cosine similarity between two crops drops quickly with IoU and falls below 0.10.1 when IoU <0.8<0.8, despite shared pixels. (b) Cosine similarity between consecutive source gradients (∇^is,∇^i+1s)( _ x_i^s, _ x_i+1^s) across iterations. Figure 2: Cosine similarity of gradients from random crops. (a) Similarity vs. IoU between two crops in a fixed iteration, showing rapid decay and values <0.1<0.1 once IoU <0.8<0.8. (b) Similarity between consecutive source gradients. V1’s gradient similarity is almost zero, while v2 improves it to around 0.2. Results are averaged over 200 runs. LVLM transfer-based attack. Black-box attacks include query-based (Dong et al., 2021; Ilyas et al., 2018) and transfer-based (Dong et al., 2018; Liu et al., 2017) (our focus in this work). AttackVLM (Zhao et al., 2023) pioneered transfer-based targeted LVLM attacks using CLIP (Radford et al., 2021) and BLIP (Li et al., 2022) surrogates, showing image-to-image feature matching beats cross-modal optimization, later adopted by (Chen et al., 2024; Guo et al., 2024; Dong et al., 2023a; Li et al., 2025). CWA (Chen et al., 2024) and SSA-CWA (Dong et al., 2023a) extended this to Bard (Team et al., 2023): CWA improves transferability via sharpness-aware minimization (Foret et al., 2021; Chen et al., 2022a), while SSA-CWA adds spectrum-guided augmentation with SSA (Long et al., 2022). AnyAttack (Zhang et al., 2024) uses image-image matching with large-scale pretraining and fine-tuning. AdvDiffVLM (Guo et al., 2024) integrates feature matching into diffusion guidance and proposes Adaptive Ensemble Gradient Estimation (AEGE) for smoother ensemble scores. - M-Attack further surpasses these via local-level matching of source and target with an ensemble of surrogate models and diverse patch sizes, and FOA-Attack (Jia et al., 2025) extends alignment from CLS to local patch tokens for additional gains. However, local-level matching still has limitations, we next introduce its background before analyzing and addressing them. 3 Approach 3.1 Preliminaries and Limitations in Prior Local-level Matching-Based Methods Local-level matching in - M-Attack (Li et al., 2025). Consider a clean source image ~sou X_sou and a target image tarX_tar. The objective of black-box transfer attacks is to minimally perturb the source image by δ so that the perturbed image sou=~sou+δX_sou= X_sou+δ aligns semantically with the target under an inaccessible black-box model fξf_ξ. Due to the inaccessibility of fξf_ξ, surrogate models fϕf_φ approximate the semantic alignment via cosine similarity: argmaxsouCS(fϕ(sou),fϕ(tar))s.t.∥δ∥p≤ϵ, _X_souCS(f_φ(X_sou),f_φ(X_tar)) .t. δ _p≤ε, (1) where CSCS denotes cosine similarity. - M-Attack enhances Eq. (1) using local-level matching. At iteration i, it applies predefined local transformations sT_s and tT_t to extract local area ^is x^s_i from the source souX_sou and ^it x^t_i from the target tarX_tar, respectively. These transformations satisfy essential properties, such as spatial overlap and diversified coverage of extracted local regions ^i\ x_i\ (Li et al., 2025). Formally, the local-level matching optimizes: ℳs,t=fϕj∼ϕ[CS(fϕj(^is),fϕj(^it))],M_T_s,T_t=E_f _j φ[CS(f_ _j( x^s_i),f_ _j( x^t_i))], (2) where fϕjf_ _j is sampled from an ensemble of surrogate models ϕφ. Intuitively, matching local image regions instead of entire images enhances the semantic precision of perturbations by directing optimization towards semantically significant details. Despite its effectiveness, - M-Attack encounters a critical challenge of unexpectedly low gradient similarity, which we investigate in detail next. Extremely low gradient overlap. In - M-Attack two random crops ^is x_i^s and ^it x_i^t are matched at every iteration. One would expect the gradients inside the shared region of two successive source crops, i.e., (∇^isℳs,j,∇^i+1sℳs,j) ( _ x_i^sM_T_s,T_j, _ x_i+1^sM_T_s,T_j ), to correlate, because the underlying pixels partly coincide. Surprisingly, Fig. 2(b) shows the opposite: their cosine similarity is almost zero. We then keep the same fixed iteration and repeatedly draw two random crops at different scales and check the cosine similarity of their gradients (Fig. 2(a)). Our finding reveals an exponential decay that plateaus below 0.10.1 once the overlap is smaller than 0.80 IoU. We find that this unexpectedly low gradient overlap mainly stems from two factors: ViTs’ inherent translation sensitivity and an overlooked asymmetry in the local matching framework. We first examine the translation effect. 1) Patch-wise, spike-like gradient sensitive to translation. Because ViTs tokenize images on a fixed, non‑overlapping grid, even sub‑pixel changes each patch’s token mix. These token changes ripple through self‑attention, altering weights and redirecting gradients for all tokens, so the resulting pixel‑level gradient pattern diverges sharply. Worse, gradient magnitudes are uneven. Therefore, even similar patterns but missing a few pixels might completely break gradient similarity (Fig. 3(b)). 2) Asymmetric Transform Branches. In - M-Attack, both the source and target images are cropped, yet playing distinct roles. Cropping the source acts directly in pixel space: it rearranges patch embeddings and attention weights in the forward pass, ending up with guidance of different views. By contrast, cropping the target solely translates the target representation, thereby shifting the reference embedding in feature space. One sculpts the perturbation, while another moves the goalpost, formulating asymmetric matching. - M-Attack overlooked this and implementations target translation alternate between a radical crop and an identity map, struggles between explore-exploitation trade-off and potentially risk in high variance of target embedding. 3.2 Asymmetric Matching over Expectation To mitigate the issues above, we begin by systematically reformulating the original objective function as an expectation over local transformations within an asymmetric matching framework: min∥sou∥p≤ϵ∼,y∼[ℒ(f((sou)),y)], _ _sou _p≤εE_T ,y [L(f(T( X_sou)),y) ], (3) where D represents the distribution of local transformations, and Y denotes the distribution over target semantics. ∥⋅∥p · _p is ℓp _p constraint for imperceptibility. Conceptually, this formulation corresponds to embedding specific semantic content y into a locally transformed area (sou)T(X_sou), thus highlighting the intrinsic asymmetry compared to - M-Attack’s original formulation. Within this framework, our proposed enhancements, i.e., Multi-Crop Alignment (MCA) and Auxiliary Target Alignment (ATA), can be interpreted as strategies to improve the accuracy of the expectation estimation and the sampling quality of the semantic distribution Y. (a) Comparison of optimization trajectories with different K, K=1K=1 refers to single crop alignment. (b) Gradient pattern between different crop strategies in - M-Attack and -- M-Attack-V2. Figure 3: Comparison of: a) different trajectories against different K; b) gradient pattern of single crop alignment against multi-crop alignment (MCA). The gradient pattern of ResNet 50 remains consistent when large pixels are overlapped, while the gradient pattern of ViTs changes dramatically. MCA helps to smooth out this impact. 3.3 Gradient Denoising via Multi-Crop Alignment To obtain a low‑variance estimate of expected loss gradient ∼,y∼[∇souℒ(f((sou)),)]E_T ,y [ _X_souL(f(T( X_sou)),y) ], we draw K independent crops k=1K\T\_k=1^K and average their individual gradients: ∇souℒ^(sou)=1K∑k=1K∇souℒ(f(k(sou)),y). _ X_sou L( X_sou)= 1K _k=1^K _ X_souL(f(T_k( X_sou)),y). (4) This Multi‑Crop Alignment is an unbiased Monte‑Carlo estimator, reducing the variance with K>1K>1. Theorem 3.1. Let gk=∇souℒ(f(k(sou)),y)g_k= _X_souL(f(T_k(X_sou)),y) denote the gradient from kT_k, μ=[gk],σ2=[∥gk−μ∥22]μ=E[g_k],σ^2=E[ g_k-μ _2^2] denote the mean and variance, and pkℓp_k denote the pair-wise correlation pkℓ=⟨gk−μ,gℓ−μ⟩∥gk−μ∥2∥gℓ−μ∥2p_k = <g_k-μ,g_ -μ > g_k-μ ^2 g_ -μ ^2. The gradient variance from K averaged crops is bounded by: Var(1K∑k=1Kgk)≤σ2K+K−1Kp¯σ2,Var ( 1K _k=1^Kg_k )≤ σ^2K+ K-1K pσ^2, (5) where p¯=[pkl],k≠ℓ p=E[p_kl],\ k≠ is the expectation of pair-wise correlation. All crops share the same underlying image, so p¯≠0 p≠ 0. The ideal σ2/Kσ^2/K decay is therefore tempered by the correlation term p¯σ2 pσ^2. Empirically, averaging a modest number (K=10K=10) of almost-orthogonal gradients still yields benefit, since the uncorrelated component of the variance shrinks as 1/K1/K. Simultaneously, the optimizer leverages multiple diverse transformations per update, with minimal interference among almost orthogonal gradients. Fig. 3(a) illustrates an accelerated convergence with K=10K=10, with margin improvement provided by K=100K=100. This averaging also alleviates the known translation sensitivity of ViTs. As shown in Fig. 3(b), using two crop sets yields noticeably higher gradient consistency than the single-crop alignment in - M-Attack. In MCA, high-activity regions remain stable (upper left and center right), while the single-crop case shifts focus from center right to lower left. As a result, gradient similarity across iterations increases from near zero in - M-Attack to around 0.2 (Fig. 2(b)). 3.4 Improved Sampling Quality via Auxiliary Target Alignment Selecting a representative target embedding y∈y is challenging because the underlying distribution Y is not observable. - M-Attack mitigates this by seeding at the unaltered target embedding f(tar)f(X_tar) and exploring its vicinity with transformed views f(t(tar))f(T_t(X_tar)) thereby sketching a locally semantic manifold that serves as a proxy for Y. However, the exploration–exploitation trade‑off remains problematic. Radical transformations leap too far, dragging y outside the genuine target region; conservative transformations, while semantically faithful, barely shift the embedding, leaving the optimization starved of informative signal. To stabilize this process, we introduce P auxiliary images aux(p)p=1P\X_aux^(p)\_p=1^P as an auxiliary set that acts as additional anchors, collectively forming a richer sub‑manifold of aligned embeddings. During each update, we apply a mild random transformation ~∼~ T D to every anchor, nudging the ensemble in a coherent yet restrained manner and thus providing low‑variance, information‑rich gradients for optimization. Let y0=f(^0(tar))y_0=f( T_0(X_tar)), y~p=f(~p(aux(p))) y_p=f( T_p(X_aux^(p))) denote sampled semantics in one iteration. The objective ℒ L in Eq. (4) becomes: ℒ^=1K∑k=1n[ L= 1K _k=1^n [ ℒ(f(k(sou)),y0) (f(T_k(X_sou)),y_0) (6) +λP∑p=1Pℒ(f(k(sou)),y~p)], + λP _p=1^PL(f(T_k(X_sou)), y_p) ], where λ∈[0,1]λ∈[0,1] interpolates between the original target and its auxiliary neighbors. λ=0λ=0 reduces to - M-Attack local-local matching with single target. ATA trade-off exploration (auxiliary diversity) and exploitation (main‑target fidelity), providing low‑variance, semantics‑preserving updates. The auxiliary set can be built in various ways, e.g., via image-image retrieval or diffusion methods. We now theoretically analyze ATA with three mild assumptions: Assumption 3.2 (Lipschitz surrogate). Surrogate f is L-continuous: ∥f(y)−f(x)∥≤L∥y−x∥ f(y)-f(x) ≤ L\|y-x . Assumption 3.3 (Bounded Auxiliary Data). For auxiliary data aux(p)X_aux^(p) retrieved via semantic similarity to a target tarX_tar, we have: [∥f(aux(p))−f(tar)∥]≤δE[ f(X^(p)_aux)-f(X_tar) ]≤δ (justification in the appendix). Assumption 3.4 (Bounded transformation). Random transformation ∼DαT D_αf has bounded pixel-level distortion: [∥()−∥]≤αE[ (X)-X ]≤α Theorem 3.5. Let ∼DαT D_α denote the transformation used in - M-Attack, and ~∼Dα~ T D_ α with α~≪α α α the transformation in -- M-Attack-V2. Define embedding drift of transformation T applied to X on model f as: Δdrift(;):=[∥f(())−f(tar)∥]. _ drift(T;X):=E_T[ f(T(X))-f( X_tar) ]. Then, we have: Δdrift(;tar)≤Lα _ drift(T;X_ tar)≤ Lα (7) Δdrift(~;aux(p))≤Lα~+δ. _ drift( T;X_ aux^(p))≤ L α+δ. Specifically, the term LαLα captures the inherent asymmetry caused by transformations in pixel space, requiring the multiplier L to map pixel-level perturbations into embedding space. In contrast, the auxiliary data directly operates in embedding space, leading to a manageable bound δ. Practically, estimating δ is notably easier than estimating LαLα. Lower δ inherently indicates better semantic alignment, allowing -- M-Attack-V2 to operate effectively under reduced distortion (α~≪α α α). Thus, ATA strategically allocates its shift budget toward more meaningful exploration via δ, achieving a sweet spot between exploration and exploitation. Figure 4: Visualization of adversarial samples under ϵ=8ε=8. Input : clean image cleanX_clean; primary target tarX_tar; auxiliary set =aux(p)p=1P [rgb]0.703125,0.0625,0.0703125 [named]pgfstrokecolorrgb0.703125,0.0625,0.0703125A= \X_aux^(p) \_p=1^P; patch ensemble+ Φ+=ϕjj=1m [rgb]0.703125,0.0625,0.0703125 [named]pgfstrokecolorrgb0.703125,0.0625,0.0703125 ^+=\ _j\_j=1^m; iterations n, step size α, perturbation budget ϵε; number of crops K, auxiliary weight λ(0≤λ≤1)λ\ (0≤λ≤ 1) Output : adversarial image advX_adv 1 21exadv←cleanX_adv _clean 3 for i=1i=1 to n do 4 Draw K [rgb]0.703125,0.0625,0.0703125 [named]pgfstrokecolorrgb0.703125,0.0625,0.0703125K transforms kk=1K∼\T_k\_k=1^K , g←g 0 5 for k=1k=1 to K (vectorizable) do 6 Draw ~pp=0P∼D~\ T_p\_p=0^P D 7 for j=1j=1 to m do 8 y0=f(~p(tar))y_0=f( T_p(X_tar)), yp=f(~p(aux(p))),p=1,…,P [rgb]0.703125,0.0625,0.0703125 [named]pgfstrokecolorrgb0.703125,0.0625,0.0703125y_p=f( T_p(X_aux^(p))),p=1,…,P 9 Compute ℒ^k=(fϕj(k(sou)),y0)+λP∑p=1Pℒ(fϕj(k(x)),y~p) L_k=(f_ _j(T_k(X_sou)),y_0)+ [rgb]0.703125,0.0625,0.0703125 [named]pgfstrokecolorrgb0.703125,0.0625,0.0703125 λP _p=1^PL(f_ _j(T_k(x)), y_p) 10 g←g+1Km∇souℒ^kg← g+ 1Km _X_sou L_k 11 12 end for 13 14 end for 15 Update advX_adv based on g with Patch Momentum 16 17 end for 18return advX_adv Algorithm 1 -- M-Attack-V2 Computation Analysis. Each iteration back-propagates through the K source crops and only forward-propagates the P auxiliary targets. Since a backward pass is roughly twice as expensive as a forward pass, the per-iteration complexity is (K(3+P))O\! (K\,(3+P) ), doubling overhead when P=3P=3. Note that the additional overhead is parallelizable. The detailed analysis and comparison are in the appendix. 3.5 Patch Momentum with Built-in Replay Effect Momentum, introduced in MI-FGSM (Dong et al., 2018), is widely adopted for transferability. Define the momentum buffer as: mr=β1mr−1+(1−β1)∇^sℒ^r(^s),m_r= _1m_r-1+(1- _1) _ x^s L_r( x^s), where β1∈[0,1) _1∈[0,1) is the first-order momentum coefficient and ∇^sℒ^r(^s) _ x^s L_r( x^s) is our MCA-ATA-estimated gradient grg_r at iteration r. Under the local-matching view, this mechanism can be reinterpreted as formulating a streaming MCA to enforce temporal consistency across gradient directions in the space of random crops. Unrolling the EMA for pixel k exposes an alternative interpretation: mi(k)=(1−β)∑j=0iβj 1k∈Mi−jgi−j(k),m_i(k)=(1-β) _j=0^iβ^j\,1\!\k∈ M_i-j\g_i-j(k), (8) where MiM_i denotes the pixel indices included in iteration i, mi(k)m_i(k) and gi(k)g_i(k) respectively denotes momentum and gradient for pixel k. Each crop involving pixel k is therefore replayed in future iterations with geometrically decaying weight, allowing rarely sampled regions (such as corners) to persist long enough to combat the gradient starvation. Spike‑shaped gradients are further moderated by the Adam‑style (Kingma, 2014) second moment, vr=β2vr−1+(1−β2)gr2v_r= _2v_r-1+(1- _2)g_r^2, whose scaling effect is essential in our empirical study. The momentum does not directly improve gradient similarity but continuously re-injects historical crops across patches, effectively maintaining gradient directionality across local perturbation manifolds. We therefore term it Patch Momentum to distinguish. The whole procedure, combining MCA, ATA, and PM, is detailed in Alg. 1. We use a different color to differentiate between -- M-Attack-V2 and - M-Attack. We use PGD (Madry et al., 2018) with ADAM (Kingma, 2014) for line 12. Appendix presents analogous results for variants. Table 1: Comparison of attack methods on three black-box commercial LVLMs. †: pre-trained on LAION (Schuhmann et al., 2022). Method Model GPT-5 Claude 4.0-thinking Gemini 2.5-Pro Imperceptibility KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR ℓ1↓ _1\! ℓ2↓ _2\! AttackVLM (Zhao et al., 2023) B/16 0.08 0.03 0.02 0.05 0.03 0.00 0.00 0.00 0.08 0.04 0.00 0.00 0.034 0.040 B/32 0.07 0.05 0.04 0.02 0.03 0.03 0.00 0.01 0.09 0.05 0.00 0.02 0.036 0.041 Laion† 0.02 0.01 0.00 0.03 0.02 0.01 0.00 0.00 0.09 0.05 0.00 0.01 0.035 0.040 AdvDiffVLM (Guo et al., 2024) Ensemble 0.04 0.02 0.01 0.01 0.04 0.01 0.01 0.01 0.03 0.01 0.00 0.00 0.064 0.095 SSA-CWA (Dong et al., 2023a) Ensemble 0.08 0.04 0.00 0.08 0.03 0.02 0.01 0.05 0.05 0.03 0.01 0.08 0.059 0.060 AnyAttack (Zhang et al., 2024) Ensemble 0.09 0.03 0.00 0.06 0.05 0.03 0.00 0.01 0.35 0.06 0.01 0.34 0.048 0.052 FOA-Attack (Jia et al., 2025) Ensemble 0.90 0.67 0.23 0.94 0.13 0.09 0.00 0.13 0.61 0.80 0.15 0.86 0.031 0.036 - M-Attack (Li et al., 2025) Ensemble 0.89 0.65 0.25 0.98 0.12 0.03 0.00 0.08 0.81 0.57 0.15 0.83 0.030 0.036 -- M-Attack-V2 (Ours) Ensemble 0.92 0.79 0.30 1.00 0.27 0.17 0.04 0.30 0.87 0.72 0.22 0.97 0.038 0.044 4 Experiments 4.1 Experimental Setup Metrics. We adopt the evaluation protocol of - M-Attack, reporting the Attack Success Rate (ASR) via GPTScore and the Keywords Matching Rate (KMR) at three thresholds 0.25,0.5,1.0\0.25,0.5,1.0\, denoted as KMRa, KMRb, and KMRc (Li et al., 2025). KMR measures semantic alignment using human-annotated keywords, considering a match successful if the rate exceeds threshold x. The evaluation follows - M-Attack exactly. Table 2: Comparison on open-source LVLMs (Qwen-2.5-VL and LLaVA-1.5). Higher KMRa/b/c and ASR are better. Method Qwen-2.5-VL LLaVA-1.5 KMRa KMRb KMRc ASR KMRa KMRb KMRc ASR AttackVLM 0.12 0.04 0.00 0.01 0.11 0.03 0.00 0.07 SSA-CWA 0.36 0.25 0.04 0.38 0.29 0.17 0.04 0.34 AnyAttack 0.53 0.28 0.09 0.53 0.60 0.32 0.07 0.58 FOA-Attack 0.83 0.61 0.20 0.91 0.94 0.75 0.29 0.95 - M-Attack 0.80 0.65 0.17 0.90 0.85 0.59 0.20 0.95 -- M-Attack-V2 0.87 0.67 0.27 0.95 0.96 0.83 0.29 0.96 Surrogate candidates. We follow surrogate selections from prior ensemble-based methods (Zhang et al., 2024; Dong et al., 2023a; Guo et al., 2024; Li et al., 2025). Our candidate pool covers CLIP variants (CLIP-B/16, B/32, L/14, CLIP†-G/14, CLIP†-B/32, CLIP†-H/14, CLIP†-B/16, CLIP†-BG/14), DinoV2 (Oquab et al., 2023) (Small, Base, Large), and BLIP-2 (Li et al., 2023)). Victim models and dataset. We evaluate state-of-the-art commercial MLLMs: GPT‑4o/o3/5, Claude‑3.7/4.0 (extended), and Gemini‑2.5‑Pro‑Preview (Team et al., 2023). Clean images are drawn from the NIPS 2017 Adversarial Attacks and Defenses Competition dataset (Kurakin et al., 2018a). Following SSA-CWA (Dong et al., 2023b) and - M-Attack (Li et al., 2025), we randomly sample 100 images, retrieving auxiliary sets from the COCO training set (Lin et al., 2014) using CLIP-B/16 embedding similarity. Further results on a 1k-image subset are in the appendix, along with HuggingFace model identifiers. All the BLIP2 (Li et al., 2023) variants on Huggingface share the same vision encoder. Therefore, we use only one. The milder target transformation includes random resized crop ([0.9,1.0][0.9,1.0]), random horizontal flip (p=0.5p=0.5), and random rotation (±15∘± 15 ). Hyperparameters. Unless noted, perturbations are bounded by ℓ∞ _∞ with ϵ=16ε=16 and optimized for 300 steps. We set the step size to α=0.75α=0.75 for Claude and α=1.0α=1.0 for all other methods, mirroring - M-Attack. For -- M-Attack-V2, α=1.275α=1.275, β1=0.9,β2=0.99 _1=0.9, _2=0.99 for momentum, K=10K=10, P=2P=2, and λ=0.3λ=0.3 for MCA and ATA. Ablation of these parameters is presented in the appendix. Figure 5: Comparison of two types of attention maps. Left: attention map that sparsely separates into different regions; right: attention map that focuses on the main object. Table 3: Comparison of embedding transferability over 1k images. MCA/ATA excluded to show standalone performance. C/D = CLIP/DinoV2. Gray denotes selected models. Surrogate C−L/14 C†−L/14 D−S/14 D−B/14 D−L/14 C−B/16 C†−B/16 C−B/32 C†−B/32 BLIP2 Avg/14 Avg/16 Avg/32 Avg/All C−L/14 N/A 0.40 0.10 0.13 0.12 0.45 0.40 0.34 0.24 0.48 0.25 0.42 0.29 0.30 C†−L/14 0.44 N/A 0.24 0.24 0.21 0.55 0.57 0.37 0.33 0.61 0.35 0.56 0.35 0.39 D−S/14 0.25 0.39 N/A 0.45 0.38 0.41 0.45 0.32 0.25 0.46 0.39 0.43 0.28 0.37 D−B/14 0.29 0.36 0.33 N/A 0.51 0.37 0.39 0.31 0.23 0.47 0.39 0.38 0.27 0.36 D−L/14 0.26 0.31 0.12 0.32 N/A 0.31 0.34 0.30 0.21 0.42 0.29 0.33 0.26 0.29 C−B/16 0.44 0.43 0.21 0.18 0.13 N/A 0.53 0.37 0.27 0.51 0.32 0.53 0.32 0.34 C†−B/16 0.43 0.51 0.22 0.21 0.15 0.57 N/A 0.39 0.34 0.52 0.34 0.57 0.36 0.37 C−B/32 0.37 0.43 0.21 0.11 0.09 0.55 0.53 N/A 0.49 0.46 0.28 0.54 0.49 0.36 C†−B/32 0.31 0.49 0.27 0.18 0.12 0.53 0.61 0.58 N/A 0.50 0.31 0.57 0.58 0.40 BLIP2 0.39 0.43 0.15 0.20 0.26 0.45 0.43 0.33 0.25 N/A 0.29 0.44 0.29 0.32 Table 4: Effect of removing each component. Numbers below each value denote the change relative to the full model (first row). ✗ marks the component(s) disabled. Component Gemini 2.5-Pro Claude 3.7-extended MCA ATA PM KMRa KMRb KMRc ASR KMRa KMRb KMRc ASR 0.87 0.72 0.22 0.97 0.56 0.32 0.11 0.67 ✗ 0.85 ↓0.02 0.02 0.70 ↓0.02 0.02 0.21 ↓0.01 0.01 0.92 ↓0.05 0.05 0.52 ↓0.04 0.04 0.35 ↑0.03 0.03 0.08 ↓0.03 0.03 0.66 ↓0.01 0.01 ✗ 0.85 ↓0.02 0.02 0.68 ↓0.04 0.04 0.21 ↓0.01 0.01 0.93 ↓0.04 0.04 0.55 ↓0.01 0.01 0.22 ↓0.10 0.10 0.10 ↓0.01 0.01 0.62 ↓0.05 0.05 ✗ ✗ 0.82 ↓0.05 0.05 0.62 ↓0.10 0.10 0.22 – 0.93 ↓0.04 0.04 0.44 ↓0.12 0.12 0.31 ↓0.01 0.01 0.08 ↓0.03 0.03 0.62 ↓0.05 0.05 ✗ 0.82 ↓0.05 0.05 0.71 ↓0.01 0.01 0.21 ↓0.01 0.01 0.96 ↓0.01 0.01 0.52 ↓0.04 0.04 0.32 ↓0.00 0.00 0.10 ↓0.01 0.01 0.66 ↓0.01 0.01 Table 5: Results of -- M-Attack-V2 on the vision reasoning model. Model KMRa KMRb KMRc ASR GPT-o3 (o3-2025-04-16) 0.91 0.71 0.23 0.98 4.2 Selection of Surrogate Model Ensembling multiple surrogate models is standard for improving black-box transferability, and recent work primarily designs advanced aggregation schemes over a small set of surrogates (Zhang et al., 2024; Guo et al., 2024). In contrast, we begin with a much larger candidate pool and find that which models are included already has a substantial impact: pre-selecting a few strong and complementary surrogates yields clear gains, even with plain averaging. We do not propose a novel aggregation rule; instead, we study a practical large-pool selection strategy that has been less studied and reported before but can serve as the very first stage of a complicated ensemble (pre-select useful candidates, then optionally apply more sophisticated aggregation). Concretely, we first profile embedding-level transferability across all candidates (Tab. 3), which shows that cross-model transfer, especially cross-patch-size transfer, is challenging. Guided by this, we retain only models that (i) perform well in Tab. 3 and (i) span diverse patch sizes to capture complementary inductive biases. A small ablation over these filtered models (see in the appendix) yields our final ensemble, Patch Ensemble+ (PE+), comprising CLIP†-G/14, CLIP-B/16, CLIP-B/32, and CLIP†-B/32. We treat PE+ as an efficient, sparse pre-selection that can be used directly or further plugged into aggregation methods. Qualitative attention maps offer an intuitive explanation: selected models consistently focus on the main object, whereas discarded ones tend to spread attention over background regions, suggesting that emphasizing core semantic content is more transferable than dispersed, model-specific patterns. 4.3 Evaluation Across LVLMs and Settings Transferability across LVLMs. Tab. 1 illustrates the superiority of our -- M-Attack-V2 compared to the other black-box LVLM attack method with Tab. 2 on open-source models. Our method outperforms others by a large margin, including - M-Attack. On GPT-5 our -- M-Attack-V2 even achieves 100% ASR and 97% ASR on Gemini-2.5, with ASR on Claude 4.0-extended further improved by 22%, which is almost impossible for - M-Attack to attack. There is also a notable improvement on the KMR, indicating that our method generates a perturbation that targets the semantics more effectively, thus more recognizable by the target black-box model. Note that these improvements are accompanied by a slight increase in the perturbation norms for l1l_1 and l2l_2. Previous l1l_1 and l2l_2 norms are caused by insufficient optimization through near-orthogonal gradients. Our -- M-Attack-V2 mitigates this issue, exploring more sufficiently inside the l∞l_∞ ball. Thus, it slightly increases the perturbation magnitude while keeping the visual appearance almost unchanged. Fig. 4 shows representative adversarial examples. Recognizing that the raw ℓp _p norm may not translate into human imperceptibility, we further evaluate the human imperceptibility in user studies reported in the appendix, showing nearly identical performance between - M-Attack and -- M-Attack-V2, which consistently outperform all other methods. Performance under varying budgets. Fig. 6 compares performance under varying optimization budgets (total steps). Our method converges faster, approaching optimal results within 300 steps, whereas - M-Attack requires an additional 200 steps, suggesting slower convergence. At fewer steps (100 and 200), - M-Attack exhibits a notable performance drop. Meanwhile, our method maintains stable ASR and KMRb due to a more coherent optimization trajectory than - M-Attack, which is more sensitive to random cropping and aggressive target transformations. Additional results on varing ϵε is presented in the appendix. Robustness Against Vision-Reasoning Models. Reasoning in text modality does not extend to alter information from the vision backbone. Instead, we further evaluate -- M-Attack-V2 against GPT-o3, a model enhanced with visual reasoning capabilities. As shown in Tab. 5, GPT-o3 exhibits slightly better robustness than GPT-4o. However, the limited improvement suggests that its reasoning module is not explicitly trained to detect adversarial manipulations in the image. Thus, even after reasoning, GPT-o3 remains susceptible to -- M-Attack-V2. The reasoning process is presented in the appendix, where it shows certain degrees of suspect in some images, and further utilize python tools for zooming. 4.4 Ablation Study Tab. 4 isolates the effect of each module beyond PE+ (GPT-4o is omitted due to negligible differences). On both Gemini-2.5-Pro and Claude-3.7-extended, activating MCA or ATA alone delivers ∼ 5% gains on average, most visible in ASR and KMRbKMR_b, with consistent improvements on KMRa/KMRcKMR_a/KMR_c. Additionally, removing PM yields only a minor drop in performance, suggesting it is complementary rather than fundamental. Overall, MCA and ATA constitute the principal mechanisms for variance reduction. At the same time, PM serves as a low-cost memory that extends the effective momentum horizon via a biased gradient, further suppressing variance and adding robustness. 5 Conclusion Figure 6: Comparison under different step budgets. In this work, we diagnosed - M-Attack’s unstable gradients as arising from high variance and overlooked asymmetric matching, and address them with a principled gradient-denoising framework. Multi-Crop Alignment (MCA), Auxiliary Target Alignment (ATA), and Patch Momentum (PM), together with a refined surrogate ensemble (PE+), form our proposed framework, which achieves state-of-the-art transfer-based black-box attacks on LVLMs. We hope these insights help achieve more stable and transferable adversarial optimization under realistic black-box constraints. Impact Statement This work strengthens transfer-based black-box attacks on large vision–language models, improving the ability to stress-test real-world multimodal systems used in assistants, search, and content generation. By revealing key instabilities in local-level matching and proposing simple gradient-denoising fixes, our methods can help researchers and practitioners build more reliable defenses, develop better robustness benchmarks, and identify failure modes before deployment. At the same time, more effective attacks can be misused to bypass safety filters, induce targeted hallucinations, or manipulate model outputs in high-stakes settings. To mitigate misuse, we have emphasized responsible disclosure, i.e., evaluate primarily on controlled benchmarks and public models, and will make full code/data/models publicly available in a way that supports reproducibility and defense research (e.g., including detection/mitigation baselines and clear guidance on safe use of our optimized generations), while avoiding instructions or configurations that lower the barrier to real-world harm. References J. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. (2022) Flamingo: a visual language model for few-shot learning. In International Conference on Advanced Neural Information Processing Systems, p. 23716–23736. Cited by: §2. Anthropic (2024) Introducing Claude 3.5 Sonnet. Anthropic Blog. Note: Accessed: 2025-02-22 External Links: Link Cited by: §2. Anthropic (2025) Claude 3.7 Sonnet and Claude Code. Anthropic Blog. Note: Accessed: 2025-02-22 External Links: Link Cited by: §2. H. Chen, S. Shao, Z. Wang, Z. Shang, J. Chen, X. Ji, and X. Wu (2022a) Bootstrap generalization ability from loss landscape perspective. In European Conference on Computer Vision, p. 500–517. Cited by: §2. H. Chen, Y. Zhang, Y. Dong, X. Yang, H. Su, and J. Zhu (2024) Rethinking model ensemble in transfer-based adversarial attacks. In International Conference on Learning Representations, Cited by: §1, §2. J. Chen, H. Guo, K. Yi, B. Li, and M. Elhoseiny (2022b) Visualgpt: data-efficient adaptation of pretrained language models for image captioning. In IEEE/CVF Computer Vision and Pattern Recognition Conference, p. 18030–18040. Cited by: §1, §2. Y. Dong, H. Chen, J. Chen, Z. Fang, X. Yang, Y. Zhang, Y. Tian, H. Su, and J. Zhu (2023a) How robust is google’s bard to adversarial image attacks?. arXiv preprint arXiv:2309.11751. Cited by: Table 6, Table 6, Table 6, Table 10, Table 10, Table 10, Table 10, Table 12, §1, §2, Table 1, §4.1. Y. Dong, H. Chen, J. Chen, Z. Fang, X. Yang, Y. Zhang, Y. Tian, H. Su, and J. Zhu (2023b) How robust is google’s bard to adversarial image attacks?. arXiv preprint arXiv:2309.11751. Cited by: §4.1. Y. Dong, S. Cheng, T. Pang, H. Su, and J. Zhu (2021) Query-efficient black-box adversarial attacks guided by a transfer-based prior. IEEE Transactions on Pattern Analysis and Machine Intelligence 44 (12), p. 9536–9548. Cited by: §2. Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, and J. Li (2018) Boosting adversarial attacks with momentum. In IEEE/CVF Computer Vision and Pattern Recognition Conference, p. 9185–9193. Cited by: §G.2, §2, §3.5. P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur (2021) Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, Cited by: §2. Q. Guo, S. Pang, X. Jia, Y. Liu, and Q. Guo (2024) Efficient generation of targeted and transferable adversarial examples for vision-language models via diffusion models. IEEE Transactions on Information Forensics and Security 20, p. 1333–1348. Cited by: §1, §2, Table 1, §4.1, §4.2. X. Hu, Z. Gan, J. Wang, Z. Yang, Z. Liu, Y. Lu, and L. Wang (2022) Scaling up vision-language pre-training for image captioning. In IEEE/CVF Computer Vision and Pattern Recognition Conference, p. 17980–17989. Cited by: §1, §2. G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt (2021) OpenCLIP. Zenodo. External Links: Document, Link Cited by: Table 9, Table 9, Table 9, Table 9. A. Ilyas, L. Engstrom, A. Athalye, and J. Lin (2018) Black-box adversarial attacks with limited queries and information. In International Conference on Machine Learning, p. 2137–2146. Cited by: §2. X. Jia, S. Gao, S. Qin, T. Pang, C. Du, Y. Huang, X. Li, Y. Li, B. Li, and Y. Liu (2025) Adversarial attacks against closed-source mllms via feature optimal alignment. arXiv preprint arXiv:2505.21494. Cited by: Table 12, §2, Table 1. D. P. Kingma (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §3.5, §3.5. A. Kurakin, I. Goodfellow, S. Bengio, Y. Dong, F. Liao, M. Liang, T. Pang, J. Zhu, X. Hu, C. Xie, et al. (2018a) Adversarial attacks and defences competition. In The NIPS’17 Competition: Building Intelligent Systems, p. 195–231. Cited by: §4.1. A. Kurakin, I. J. Goodfellow, and S. Bengio (2018b) Adversarial examples in the physical world. In Artificial intelligence safety and security, p. 99–112. Cited by: §G.2. H. Li, J. G. Ellis, L. Zhang, and S. Chang (2018) Patternnet: visual pattern mining with deep neural network. In Proceedings of the 2018 ACM on international conference on multimedia retrieval, p. 291–299. Cited by: §G.4, Table 14, Table 14. J. Li, D. Li, S. Savarese, and S. Hoi (2023) Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, p. 19730–19742. Cited by: Table 9, §4.1. J. Li, D. Li, C. Xiong, and S. Hoi (2022) Blip: bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, p. 12888–12900. Cited by: §2, §2. Z. Li, X. Zhao, D. Wu, J. Cui, and Z. Shen (2025) A frustratingly simple yet highly effective attack baseline: over 90% success rate against the strong black-box models of gpt-4.5/4o/o1. arXiv preprint arXiv:2503.10635. Cited by: Table 6, Table 6, Table 6, Table 10, Table 10, Table 10, Table 10, Table 11, Table 12, §1, §2, §3.1, §3.1, Table 1, §4.1, §4.1. T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick (2014) Microsoft coco: common objects in context. In European conference on computer vision, p. 740–755. Cited by: §4.1. H. Liu, C. Li, Q. Wu, and Y. J. Lee (2023) Visual instruction tuning. In International Conference on Advanced Neural Information Processing Systems, p. 34892–34916. Cited by: §2. Y. Liu, X. Chen, C. Liu, and D. Song (2017) Delving into transferable adversarial examples and black-box attacks. In International Conference on Learning Representations, Cited by: §2. Y. Long, Q. Zhang, B. Zeng, L. Gao, X. Liu, J. Zhang, and J. Song (2022) Frequency domain model augmentation for adversarial attack. In European Conference on Computer Vision, p. 549–566. Cited by: §2. D. Luu, V. Le, and D. M. Vo (2024) Questioning, answering, and captioning for zero-shot detailed image caption. In Asian Conference on Computer Vision, p. 242–259. Cited by: §1, §2. Z. Ma, J. Hong, M. O. Gul, M. Gandhi, I. Gao, and R. Krishna (2023) Crepe: can vision-language foundation models reason compositionally?. In IEEE/CVF Computer Vision and Pattern Recognition Conference, p. 10910–10921. Cited by: §2. A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu (2018) Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, Cited by: §3.5. OpenAI (2025) Introducing o3 and o4-mini. OpenAI Blog. Note: Accessed: 2025-02-22 External Links: Link Cited by: §I.2, §1, §2. M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. (2023) Dinov2: learning robust visual features without supervision. arXiv preprint arXiv:2304.07193. Cited by: Table 9, Table 9, Table 9, §4.1. Ö. Özdemir and E. Akagündüz (2024) Enhancing visual question answering through question-driven image captions as prompts. In IEEE/CVF Computer Vision and Pattern Recognition Conference, p. 1562–1571. Cited by: §1, §2. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, p. 8748–8763. Cited by: Table 9, §2. A. Salaberria, G. Azkune, O. L. de Lacalle, A. Soroa, and E. Agirre (2023) Image captioning for effective use of language models in knowledge-based visual question answering. Expert Systems with Applications 212, p. 118669. Cited by: §1, §2. C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. (2022) Laion-5b: an open large-scale dataset for training next generation image-text models. Advances in neural information processing systems 35, p. 25278–25294. Cited by: Table 9, Table 9, Table 9, Table 9, Table 1, Table 1. G. Team, R. Anil, S. Borgeaud, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. (2023) Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. Cited by: §2, §2, §4.1. M. Tschannen, M. Kumar, A. Steiner, X. Zhai, N. Houlsby, and L. Beyer (2023) Image captioners are scalable vision learners too. In International Conference on Advanced Neural Information Processing Systems, p. 46830–46855. Cited by: §1, §2. T. Wang, F. Li, L. Zhu, J. Li, Z. Zhang, and H. T. Shen (2024) Cross-modal retrieval: a systematic review of methods and future directions. arXiv preprint arXiv:2308.14263. Cited by: §2. J. Wu, M. Zhong, S. Xing, Z. Lai, Z. Liu, Z. Chen, W. Wang, X. Zhu, L. Lu, T. Lu, et al. (2025) Visionllm v2: an end-to-end generalist multimodal large language model for hundreds of vision-language tasks. In International Conference on Advanced Neural Information Processing Systems, p. 69925–69975. Cited by: §2. J. Yang, R. Shi, and B. Ni (2021) MedMNIST classification decathlon: a lightweight automl benchmark for medical image analysis. In IEEE 18th International Symposium on Biomedical Imaging (ISBI), p. 191–195. Cited by: §G.4, Table 18, Table 18. J. Zhang, J. Ye, X. Ma, Y. Li, Y. Yang, J. Sang, and D. Yeung (2024) Anyattack: towards large-scale self-supervised generation of targeted adversarial examples for vision-language models. arXiv preprint arXiv:2410.05346. Cited by: Table 6, Table 6, Table 6, §2, Table 1, §4.1, §4.2. Y. Zhao, T. Pang, C. Du, X. Yang, C. Li, N. M. Cheung, and M. Lin (2023) On evaluating adversarial robustness of large vision-language models. In International Conference on Advanced Neural Information Processing Systems, p. 54111–54138. Cited by: Table 6, Table 6, Table 6, §1, §2, Table 1. Appendix Contents Appendix A Additional Details for Theoretical Analysis A.1 Proof for Theorem 1 This section provides detailed proof of the upper bound in Eq. (5). For variance, we have Var(g^K) ( g_K) :=‖g^K−μ‖2 =E \| g_K-μ \|^2 (9) =‖1K∑k=1K(gk−μ)‖2 =E \| 1K _k=1^K(g_k-μ) \|^2 =1K2∑k=1K∑ℓ=1K[(gk−μ)⊤(gℓ−μ)] = 1K^2 _k=1^K _ =1^KE[(g_k-μ) (g_ -μ)] =1K2(∑k=1K‖gk−μ‖22⏟Kσ2+ = 1K^2 ( _k=1^KE\|g_k-μ\|_2^2_Kσ^2 .+ 2∑1≤k<ℓ≤K[⟨gk−μ,gℓ−μ⟩]⏟cross terms) . 2 _1≤ k< ≤ KE[ g_k-μ,g_ -μ ]_cross terms ) The diagonal part is reduced to the mean. We now provide an upper bound for the cross terms. Recall pkℓ=⟨gk−μ,gℓ−μ⟩∥gk−μ∥2∥gℓ−μ∥2p_k = <g_k-μ,g_ -μ > g_k-μ ^2 g_ -μ ^2, we have [⟨gk−μ,gℓ−μ⟩]=[ρkℓ‖gk−μ‖2‖gℓ−μ‖2].E[ g_k-μ,g_ -μ ]=E [ _k \|g_k-μ\|_2\|g_ -μ\|_2 ]. (10) Since all crops share the same marginal distribution, i.e. ∥gk−μ∥2=∥gℓ−μ∥2=σE g_k-μ _2=E g_ -μ _2=σ, applying the Cauchy-Schwarz inequality to Eq. (10) yields [⟨gk−μ,gℓ−μ⟩] [ g_k-μ,g_ -μ ] ≤[ρkℓ]‖gk−μ‖22‖gℓ−μ‖22 [ _k ] E\|g_k-μ\|_2^2 E\|g_ -μ\|_2^2 (11) =ρ¯σ2, = ρσ^2, where p¯ p is [pkℓ],k≠ℓE[p_k ],k≠ . Plugging this into the double sum term yields ∑1≤k<ℓ≤K[⟨gk−μ,gℓ−μ⟩]≤K(K−1)2ρ¯σ2. _1≤ k< ≤ KE [ g_k-μ,g_ -μ ]≤ K(K-1)2 ρσ^2. (12) The K(K−1)2 K(K-1)2 appears since there are total K(K−1)2 K(K-1)2 terms for ∑k<ℓ _k< . Thus substituting Eq. (12) back to the cross item part in the Eq. (9) yields Var(g^K) ( g_K) ≤1K2(Kσ2+K(K−1)p¯σ2) ≤ 1K^2 (Kσ^2+K(K-1) pσ^2 ) (13) =1Kσ2+K−1Kp¯σ2 = 1Kσ^2+ K-1K pσ^2 Therefore, we have the upper bound provided in the Sec. 3.3. Input : clean image cleanX_clean; primary target tarX_tar; auxiliary set =aux(p)p=1P [rgb]0.703125,0.0625,0.0703125 [named]pgfstrokecolorrgb0.703125,0.0625,0.0703125A=\X_aux^(p)\_p=1^P; patch ensemble+ Φ+=ϕjj=1m [rgb]0.703125,0.0625,0.0703125 [named]pgfstrokecolorrgb0.703125,0.0625,0.0703125 ^+=\ _j\_j=1^m; iterations n, step size α, perturbation budget ϵε; Adam β1,β2,η _1, _2,η; number of crops K, auxiliary weight λ Output : adversarial image advX_adv 1 21exadv←cleanX_adv\!←\!X_clean, m←0m\!←\!0, v←0v\!←\!0 3 for i=1i=1 to n do 4 Draw K transforms kk=1K∼\T_k\_k=1^K\! \!D g←0g\!←\!0 // accumulate over crops 5 for k=1k=1 to K do // — crop loop — 6 Draw ~pp=0P∼~\ T_p\_p=0^P\! \! D 7 for j=1j=1 to m do 8 y0=f(~0(tar))y_0=f( T_0(X_tar)) 9 yp=f(~p(aux(p))),p=1,…,Py_p=f( T_p(X_aux^(p)))\!,\;p=1,…,P 10 ℒ^k=ℒ(fϕj(k(adv)),y0)+λP∑p=1Pℒ(fϕj(k(adv)),yp) L_k\!=\!L (f_ _j(T_k(X_adv)),y_0 )\;+\; [rgb]0.703125,0.0625,0.0703125 [named]pgfstrokecolorrgb0.703125,0.0625,0.0703125 λP\! _p=1^PL (f_ _j(T_k(X_adv)),y_p ) 11 g←g+1Km∇advℒ^kg\!←\!g+ 1Km _X_adv L_k 12 13 end for 14 15 end for // — Adam update — 16 m←β1m+(1−β1)gm\!←\! _1m+(1- _1)g 17 v←β2v+(1−β2)g⊙2v\!←\! _2v+(1- _2)g 2 18 m^←m/(1−β1i) m\!←\!m/(1- _1^i); 19 v^←v/(1−β2i) v\!←\!v/(1- _2^i) 20 adv←clipclean,ϵ(adv+αm^/(v^+η))X_adv\!←\!clip_X_clean,ε\! (X_adv+α\, m/( v+η) ) 21 22 end for 23return advX_adv Algorithm 2 -- M-Attack-V2 (Adam variant) Input : clean image cleanX_clean; primary target tarX_tar; auxiliary set =aux(p)p=1P [rgb]0.703125,0.0625,0.0703125 [named]pgfstrokecolorrgb0.703125,0.0625,0.0703125A=\X_aux^(p)\_p=1^P; patch ensemble+ Φ+=ϕjj=1m [rgb]0.703125,0.0625,0.0703125 [named]pgfstrokecolorrgb0.703125,0.0625,0.0703125 ^+=\ _j\_j=1^m; iterations n, step size α, perturbation budget ϵε; momentum decay γ; number of crops K, auxiliary weight λ Output : adversarial image advX_adv 1 21exadv←cleanX_adv\!←\!X_clean, μ←0μ\!←\!0 3 for i=1i=1 to n do 4 Draw K transforms kk=1K∼\T_k\_k=1^K\! \!D 5 g←0g\!←\!0 6 for k=1k=1 to K do 7 Draw ~pp=0P∼~\ T_p\_p=0^P\! \! D 8 for j=1j=1 to m do 9 y0=f(~0(tar))y_0=f( T_0(X_tar)) 10 yp=f(~p(aux(p))),p=1,…,Py_p=f( T_p(X_aux^(p)))\!,\;p=1,…,P 11 ℒ^k=ℒ(fϕj(k(adv)),y0)+λP∑p=1Pℒ(fϕj(k(adv)),yp) L_k\!=\!L (f_ _j(T_k(X_adv)),y_0 )\;+\; [rgb]0.703125,0.0625,0.0703125 [named]pgfstrokecolorrgb0.703125,0.0625,0.0703125 λP\! _p=1^PL (f_ _j(T_k(X_adv)),y_p ) 12 g←g+1Km∇advℒ^kg\!←\!g+ 1Km _X_adv L_k 13 14 end for 15 16 end for // — MI-FGSM update — 17 μ←γμ+g∥g∥1μ\!←\!γ\,μ+ g g _1 18 adv←clipclean,ϵ(adv+αsign(μ))X_adv\!←\!clip_X_clean,ε\! (X_adv+α\,sign(μ) ) 19 20 end for 21return advX_adv Algorithm 3 -- M-Attack-V2 (MI-FGSM variant) Figure 7: Comparison of one step between - M-Attack and -- M-Attack-V2. A.2 Proof of Theorem 2 We begin with the drift analysis for - M-Attack: Δdrift(;tar) _drift(T;X_tar) =∼Dα[∥f((tar))−f(tar)∥] =E_T Dα[ f(T(X_tar))-f(X_tar) ] (14) ≤L⋅∼Dα[∥(tar)−tar∥] ≤ L·E_T Dα[ (X_tar)-X_tar ] ≤Lα. ≤ Lα. Next, we analyze the drift for -- M-Attack-V2 using the triangle inequality and the above assumptions: Δdrift(~;aux(p)) _drift( T;X_aux^(p)) =~∼Dα~[‖f(~(aux(p)))−f(tar)‖] =E_ T D_ α\! [\|f( T(X^(p)_aux))-f(X_tar)\| ] ≤~[∥f(~(aux(p)))−f(aux(p))∥+ _ T\! [\|f( T(X^(p)_aux))-f(X^(p)_aux)\|+ ∥f(aux(p))−f(tar)∥] \|f(X^(p)_aux)-f(X_tar)\| ] =~[‖f(~(aux(p)))−f(aux(p))‖]+ =E_ T\! [\|f( T(X^(p)_aux))-f(X^(p)_aux)\| ]+ [‖f(aux(p))−f(tar)‖] \! [\|f(X^(p)_aux)-f(X_tar)\| ] ≤L~[‖~(aux(p))−aux(p)‖]+δ ≤ L\,E_ T\! [\| T(X^(p)_aux)-X^(p)_aux\| ]+δ ≤Lα~+δ ≤ L α+δ . . Table 6: Ablation study on the impact of perturbation budget (ϵε). ϵε Method GPT‑4o Claude 3.7‑thinking Gemini 2.5‑Pro Imperceptibility KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR ℓ1↓ _1\! ℓ2↓ _2\! 4 AttackVLM (Zhao et al., 2023) 0.08 0.04 0.00 0.02 0.04 0.01 0.00 0.00 0.10 0.04 0.00 0.01 0.010 0.011 SSA‑CWA (Dong et al., 2023a) 0.05 0.03 0.00 0.03 0.04 0.01 0.00 0.02 0.04 0.01 0.00 0.04 0.015 0.015 AnyAttack (Zhang et al., 2024) 0.07 0.02 0.00 0.05 0.05 0.05 0.02 0.06 0.05 0.02 0.00 0.10 0.014 0.015 - M-Attack (Li et al., 2025) 0.30 0.16 0.03 0.26 0.06 0.01 0.00 0.01 0.24 0.14 0.02 0.15 0.009 0.010 -- M-Attack-V2 (Ours) 0.59 0.34 0.10 0.58 0.06 0.02 0.00 0.02 0.48 0.33 0.07 0.38 0.012 0.013 8 AttackVLM (Zhao et al., 2023) 0.08 0.02 0.00 0.01 0.04 0.02 0.00 0.01 0.07 0.01 0.00 0.01 0.020 0.022 SSA‑CWA (Dong et al., 2023a) 0.06 0.02 0.00 0.04 0.04 0.02 0.00 0.02 0.02 0.00 0.00 0.05 0.030 0.030 AnyAttack (Zhang et al., 2024) 0.17 0.06 0.00 0.13 0.07 0.07 0.02 0.05 0.12 0.04 0.00 0.13 0.028 0.029 - M-Attack (Li et al., 2025) 0.74 0.50 0.12 0.82 0.12 0.06 0.00 0.09 0.62 0.34 0.08 0.48 0.017 0.020 -- M-Attack-V2 (Ours) 0.87 0.69 0.20 0.93 0.23 0.14 0.02 0.22 0.72 0.49 0.21 0.77 0.023 0.023 16 AttackVLM (Zhao et al., 2023) 0.08 0.02 0.00 0.02 0.01 0.00 0.00 0.01 0.03 0.01 0.00 0.00 0.036 0.041 SSA‑CWA (Dong et al., 2023a) 0.11 0.06 0.00 0.09 0.06 0.04 0.01 0.12 0.05 0.03 0.01 0.08 0.059 0.060 AnyAttack (Zhang et al., 2024) 0.44 0.20 0.04 0.42 0.19 0.08 0.01 0.22 0.35 0.06 0.01 0.34 0.048 0.052 - M-Attack (Li et al., 2025) 0.82 0.54 0.13 0.95 0.31 0.21 0.04 0.37 0.81 0.57 0.15 0.83 0.030 0.036 -- M-Attack-V2 (Ours) 0.91 0.78 0.40 0.99 0.56 0.32 0.11 0.67 0.87 0.72 0.22 0.97 0.038 0.044 Thus, we have completed the proof of Theorem 3.5. A.3 Justification for Assumptions Assumption 3.3 is derived from the retrieval mechanism for auxiliary data. Specifically, Xaux(p)X_aux^(p) represents the p-th closest embedding to the target XtarX_tar from a database D, defined explicitly by: aux(p)∈argtopP∈:f()⊤f(tar)|f()||f(tar)|,X_aux^(p)∈ _P \X : f(X) f(X_tar)|f(X)||f(X_tar)| \, (15) where topPtop_P denotes selecting the top-P nearest neighbors according to cosine similarity. Given that embeddings f()f(X) are typically normalized, semantic closeness naturally bounds the expected distance between f(aux(p))f(X_aux^(p)) and f(tar)f(X_tar) by δ, thus validating Assumption 3.3. In such a case, to estimate δ, use 2(1−f(aux(P))⊤f(tar))2 (1-f(X_ aux^(P)) f(X_ tar) ) Appendix B More Details on Our Algorithm Alg. 2 and Alg. 3 provide detailed update rule of line 13 in Alg. 1. Fig. 7 provides a comparison between the entire procedure of - M-Attack and our -- M-Attack-V2 under the local-matching framework. Notably, - M-Attack utilizes a radical crop on the target image, risking unrelated or broken semantics for the source image to align. Our ATA anchors more points inside the semantic manifold (blue), and provides a mild transformation to provide a coherence sampling from the target semantic manifold. Appendix C More Details of Experimental Setup The experiment’s seed is 2023. It is conducted on a Linux platform (Ubuntu 22.04) with 6 NVIDIA RTX 4090 GPUs. The temperatures of all LLMs are set to 0. The threshold of the ASR is set to 0.3, following - M-Attack. Tab. 9 provides a map from model names in this paper to their identifiers in HuggingFace. We use GPT-5-thinking-low (setting reasoning effort to low in the API) for all results in the main paper, with results on other reasoning budgets presented in the Appx. G.3 Appendix D Full Process of Surrogate Model Selection This section details the process of selecting our final ensemble, PE+. Exhaustively testing all model combinations is computationally infeasible, so we employ a heuristic-driven approach. We begin by excluding DiNO-large and BLIP2 due to their poor transferability, as shown in Tab. 3. Our initial experiments focus on evaluating the effectiveness of homogeneous ensembles—comprising models with the same patch size—versus mixed patch size ensembles. Specifically, we construct five ensembles: (1) patch-14 CLIP (CLIP-L/14, CLIP†-G/14), (2) patch-14 DiNOv2 (Dino-base, Dino-large), (3) patch-16 CLIP (CLIP-B/16, CLIP†-B/16), and (4) patch-32 CLIP (CLIP-B/32, CLIP†-B/32). Results are presented in Tab. 7. These results reveal that the patch-32 CLIP ensemble performs best on Claude 3.7, while GPT-4o and Gemini 2.5 Pro favor models with patch sizes 14 and 16. This supports the findings in Sec. 4.2: although using a fixed patch size can mitigate architectural bias, it still inherits the intrinsic bias of the patch size itself. To address this, we adopt a cross-patch size strategy. Starting from the patch-32 CLIP ensemble, due to its strong performance on Claude and consistent transferability across patch-16 and patch-32 models. We incrementally incorporate one model each from patch sizes 14 and 16. We evaluate various combinations, with results summarized in Tab. 8. The resulting ensemble, PE+, achieves the most balanced performance, ranking first on 7 metrics and a close second on 3 others, across 12 evaluation metrics. Figure 8: Visualization of adversarial samples under ϵ=16ε=16. Table 7: Ablation on two-model surrogate sets. Bold numbers are the best in each column; underlined numbers are the second-best. Variant Surrogate Set (2 models) GPT-4o Claude 3.7-extended Gemini 2.5-Pro KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR Pair1 Dino-B, Dino-S 0.84 0.57 0.15 0.91 0.09 0.04 0.00 0.05 0.84 0.53 0.11 0.81 Pair2 L16, B/16 0.86 0.69 0.21 0.96 0.16 0.10 0.01 0.16 0.84 0.59 0.15 0.91 Pair3 L32, B/32 0.76 0.52 0.13 0.79 0.46 0.29 0.06 0.70 0.58 0.37 0.07 0.59 Pair4 G/14, L14 0.86 0.61 0.24 0.94 0.07 0.02 0.00 0.06 0.82 0.64 0.23 0.92 Table 8: Ablation on surrogate-set selection. Each row swaps one model in or out of a four-model ensemble. The fully grey PE+ line is our final patch-diverse surrogate set (CLIP†-G/14, CLIP-B/16, CLIP-B/32, CLIP†-B/32). Bold numbers denote the best score in each metric column across all variants, underline denote second best with neglectable gap of 0.01 Variant Surrogate Set GPT-4o Claude 3.7-extended Gemini 2.5-Pro KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR PE1 B/16, B/32, L32, L16 0.87 0.65 0.26 0.99 0.54 0.32 0.07 0.68 0.80 0.57 0.16 0.90 PE2 Dino-B, B/32, L32, G/14 0.87 0.69 0.28 0.97 0.56 0.37 0.09 0.65 0.88 0.71 0.22 0.93 PE3 L16, B/32, L32, G/14 0.85 0.65 0.23 0.99 0.57 0.40 0.09 0.73 0.84 0.61 0.19 0.93 PE4 B/16, B/32, L32, Dino-B 0.89 0.67 0.19 0.98 0.55 0.41 0.07 0.63 0.87 0.67 0.23 0.96 PE5 B/16, B/32, L32, Dino-S 0.90 0.72 0.25 0.97 0.48 0.33 0.08 0.59 0.83 0.63 0.17 0.90 PE+ (Ours) B/16, B/32, L32, G/14 0.91 0.78 0.40 0.99 0.56 0.32 0.11 0.67 0.87 0.72 0.22 0.97 Table 9: Surrogate models and their corresponding HuggingFace identifier in our main paper. Surrogate (paper notation) Implementation (HuggingFace identifier) CLIP†-B/32 (Ilharco et al., 2021; Schuhmann et al., 2022) laion/CLIP-ViT-B-32-laion2B-s34B-b79K CLIP†-H/14 (Ilharco et al., 2021; Schuhmann et al., 2022) laion/CLIP-ViT-H-14-laion2B-s32B-b79K CLIP-L/14 (Radford et al., 2021) openai/clip-vit-large-patch14 CLIP†-B/16 (Ilharco et al., 2021; Schuhmann et al., 2022) laion/CLIP-ViT-B-16-laion2B-s34B-b88K CLIP†-BG/14 (Ilharco et al., 2021; Schuhmann et al., 2022) laion/CLIP-ViT-bigG-14-laion2B-39B-b160k Dino-Small (Oquab et al., 2023) facebook/dinov2-small Dino-Base (Oquab et al., 2023) facebook/dinov2-base Dino-Large (Oquab et al., 2023) facebook/dinov2-large BLIP-2 (2.7 B) (Li et al., 2023) Salesforce/blip2-opt-2.7b Appendix E More Ablation Study E.1 Ablation Study for Step Size This section provides an ablation study for the step size parameter α to view its impact on the performance. Overall, selecting α∈[0.5,1.0]α∈[0.5,1.0] provides better performance for SSA-CWA, - M-Attack. Our -- M-Attack-V2 prefer stepsize at 1.275, since it adopts ADAM as optimizer. Appendix F Success and Failure Examples Fig. 9 illustrates typical adversarial samples for successful and failed attacks of -- M-Attack-V2 across GPT/Claude and Gemini. In failure cases, the adversarial perturbations show weaker semantic correlation with the target label, whereas in successful cases, rough shapes of the target (such as trees or animals) can be clearly identified. We also observe that shared successful target images tend to appear neater and more centralized. Figure 9: Success and failure modes of -- M-Attack-V2 shared by GPT 5/Claude 4.0-extended/Gemini 2.5-Pro. F.1 Ablation Study on MCA and ATA Hyperparameters Fig. 10(left) shows transferability peaks around K=10∼20K=10 20, beyond which increased stability reduces beneficial noise regularization. Fig. 10(right) demonstrates larger λ boosts diversity by aligning semantics closer to auxiliary data but risks impairing semantic accuracy (as measured by KMR). Fig. 11(a,b) indicates minor impacts from P and momentum coefficient β; setting P=2P=2 optimizes performance and efficiency, and the default β=0.9β=0.9 consistently yields robust results. Figure 10: ASR and KMRa/KMRb vs. different K and λ. Appendix G Additional Results G.1 Additional Results on 1K image We compare - M-Attack and -- M-Attack-V2 across 1K images to improve statistical stability. We changed the threshold to multiple values since no additional keywords were added to the 900 images, thereby replacing the KMR with ASR at different matching levels. Our -- M-Attack-V2 consistently outperforms - M-Attack, demonstrating the superiority of our proposed strategy. (a) Effect of auxiliary set size P. (b) Effect of momentum parameter β. Figure 11: Ablation study on auxiliary set size P and momentum parameter β. Table 10: Ablation study on the impact of perturbation budget (α). α Method GPT-4o Claude 3.7-thinking Gemini 2.5-Pro KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR 0.25 SSA-CWA (Dong et al., 2023a) 0.08 0.08 0.04 0.10 0.06 0.03 0.00 0.03 0.06 0.03 0.00 0.01 - M-Attack (Li et al., 2025) 0.62 0.39 0.09 0.71 0.12 0.03 0.01 0.16 0.55 0.33 0.08 0.55 -- M-Attack-V2 (Ours) 0.86 0.61 0.21 0.96 0.43 0.28 0.5 0.52 0.82 0.29 0.18 0.89 0.50 SSA-CWA (Dong et al., 2023a) 0.10 0.10 0.04 0.07 0.08 0.04 0.00 0.05 0.09 0.05 0.00 0.04 - M-Attack (Li et al., 2025) 0.73 0.48 0.17 0.77 0.20 0.13 0.06 0.22 0.79 0.53 0.10 0.80 -- M-Attack-V2 (Ours) 0.87 0.64 0.23 0.96 0.58 0.34 0.13 0.67 0.83 0.59 0.17 0.94 1.00 SSA-CWA (Dong et al., 2023a) 0.11 0.06 0.00 0.09 0.06 0.04 0.01 0.12 0.05 0.03 0.01 0.08 - M-Attack (Li et al., 2025) 0.82 0.54 0.13 0.95 0.31 0.21 0.04 0.37 0.81 0.57 0.15 0.83 -- M-Attack-V2 (Ours) 0.92 0.77 0.42 0.98 0.55 0.36 0.08 0.67 0.85 0.73 0.22 0.98 1.275 SSA-CWA (Dong et al., 2023a) 0.09 0.09 0.04 0.03 0.06 0.03 0.00 0.03 0.05 0.02 0.00 0.02 - M-Attack (Li et al., 2025) 0.00 0.00 0.00 0.00 0.25 0.18 0.06 0.34 0.85 0.55 0.19 0.84 -- M-Attack-V2 (Ours) 0.91 0.78 0.40 0.99 0.56 0.32 0.11 0.67 0.87 0.72 0.22 0.97 Table 11: Comparison of results on 1K images. We provide ASR based on different thresholds as a surrogate for KMR following - M-Attack (Li et al., 2025). threshold GPT-4o Gemini-2.5-Pro Claude-3.7-extended - M-Attack -- M-Attack-V2 - M-Attack -- M-Attack-V2 - M-Attack -- M-Attack-V2 0.3 0.868 0.983 0.714 0.915 0.289 0.632 0.4 0.614 0.965 0.621 0.870 0.250 0.437 0.5 0.614 0.871 0.539 0.673 0.057 0.127 0.6 0.399 0.423 0.310 0.556 0.015 0.127 0.7 0.399 0.412 0.245 0.342 0.013 0.089 0.8 0.234 0.328 0.230 0.289 0.008 0.009 0.9 0.056 0.150 0.049 0.087 0.001 0.005 G.2 Additional Results on FGSM framework We provide the results of the I-FGSM (Kurakin et al., 2018b) and MI-FGSM (Dong et al., 2018) under our - M-Attack framework as complementary, presented in Tab. 12. Results show that even under the FGSM framework, where the patchy gradient matter is smoothed by assigning sign(∇ℒ)sign( ), -- M-Attack-V2 still benefit from momentum. Moreover, MI-FGSM still provides results comparable to those of the ADAM version. However, using PGD framework with ADAM optimizer is generally the better choice to unleash the potential of black-box attack fully since it can better explore in the space while also reducing scale issue with second-order momentum. G.3 Additional Results on Other GPT-5 Reasoning Modes GPT-5 provides four reasoning modes: minimum, low, medium, and high. While the main paper presents results using GPT-5-thinking-low, additional experiments on other reasoning modes are summarized in Tab. 13. Our proposed -- M-Attack-V2 consistently achieves superior performance across all modes. Interestingly, providing additional thinking budget generally enhances model robustness, evidenced by a reduction in ASR and KMR. However, this improvement is not strictly monotonic: ASR first decreases from 100% (low) to 96% (medium) before slightly rebounding to 99% (high). A similar non-monotonic trend can also be observed elsewhere in the table. Table 12: Ablation study of -- M-Attack-V2 under different optimizer/attack variants. Method Model GPT-5 (low) GPT-5 (medium) GPT-5 (high) KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR SSA-CWA (Dong et al., 2023a) Ensemble 0.08 0.04 0.00 0.08 0.09 0.05 0.01 0.06 0.10 0.05 0.01 0.07 FOA-Attack (Jia et al., 2025) Ensemble 0.90 0.67 0.23 0.94 0.90 0.69 0.21 0.96 0.87 0.68 0.24 0.96 - M-Attack (Li et al., 2025) Ensemble 0.89 0.65 0.25 0.98 0.85 0.61 0.16 0.96 0.80 0.60 0.20 0.93 -- M-Attack-V2 (Ours) Ensemble 0.92 0.79 0.30 1.00 0.90 0.73 0.25 0.96 0.88 0.71 0.27 0.99 Table 13: Comparison on GPT-5 under three budget settings (low/medium/high). Method Model GPT-4o Claude 3.7-extended Gemini 2.5-Pro KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR -- M-Attack-V2-ADAM (Ours) Ensemble 0.91 0.78 0.40 0.99 0.56 0.32 0.11 0.67 0.87 0.72 0.22 0.97 -- M-Attack-V2-FGSM Ensemble 0.85 0.64 0.19 0.98 0.40 0.26 0.08 0.46 0.83 0.65 0.17 0.90 -- M-Attack-V2-MIFGSM Ensemble 0.90 0.66 0.23 0.96 0.45 0.30 0.07 0.57 0.84 0.64 0.15 0.87 G.4 Cross-Domain Evaluation on Medical and Overhead Imagery Beyond the general-domain datasets, we further probe transferability to domains that are notoriously challenging for closed-source VLMs: chest X-rays and overhead remote sensing. Concretely, we augment the NIPS 2017 adversarial competition evaluation with images from ChestMNIST, from MedMNIST (Yang et al., 2021) and PatternNet (Li et al., 2018). We keep the target set unchanged and reuse the same attack budget and optimization hyper-parameters as in the main experiments. These domains are non-photographic and typically elicit generic captions from off-the-shelf VLMs, making them a stringent test of cross-domain transfer. We report KMRa/KMRb/KMRcKMR_a/KMR_b/KMR_c and ASR (higher is better) on GPT-4o, Claude 3.7, and Gemini 2.5 in Tables 14 and 18. Across both datasets, -- M-Attack-V2 consistently surpasses - M-Attack and prior baselines. On PatternNet, -- M-Attack-V2 improves Claude 3.7 ASR from 0.48 to 0.73 (+0.25) and raises GPT-4o KMRa/b/cKMR_a/b/c to 0.83/0.71/0.24. On ChestMNIST, the gains are even larger on Claude 3.7 (ASR 0.31 → 0.83, +0.52), while -- M-Attack-V2 also achieves the highest KMRa/b/cKMR_a/b/c on Gemini 2.5 (0.89/0.76/0.33). The only exception is ChestMNIST ASR on Gemini 2.5, where - M-Attack is marginally higher (0.96 vs. 0.95), despite -- M-Attack-V2 yielding stronger keyword-match rates. Table 14: Cross-domain results on PatternNet (Li et al., 2018). We report KMRa/KMRb/KMRcKMR_a/KMR_b/KMR_c and ASR (higher is better). Bold = best in column; underline = second best. The shaded row is our method. Method GPT-4o Claude 3.7 Gemini 2.5 KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR AttackVLM 0.06 0.01 0.00 0.02 0.06 0.02 0.00 0.00 0.09 0.04 0.00 0.03 SSA-CWA 0.05 0.02 0.00 0.13 0.04 0.03 0.00 0.07 0.08 0.02 0.01 0.15 AnyAttack 0.06 0.03 0.00 0.05 0.03 0.01 0.00 0.05 0.06 0.02 0.00 0.05 - M-Attack 0.79 0.66 0.21 0.93 0.33 0.17 0.04 0.48 0.86 0.71 0.23 0.91 -- M-Attack-V2 0.83 0.71 0.24 0.93 0.58 0.40 0.09 0.73 0.88 0.68 0.22 0.97 G.5 Robustness to input–preprocessing defenses We evaluate two input–preprocessing defenses—JPEG recompression (quality Q=75Q=75) and diffusion-based purification (DiffPure) with denoising budgets t=25t=25 and t=75t=75. As summarized in Table 17, the JPEG results show that M-Attack-V2 remains strong while prior attacks substantially degrade, suggesting resilience to quantization and mild photometric shifts. DiffPure reduces success rates for all methods; however, M-Attack-V2 preserves a clear margin at t=25t=25 and remains the most effective even under the aggressive t=75t=75 setting, where purification approaches image regeneration. G.6 Human Perceptual Study To evaluate the perceptual stealth of the perturbations beyond static metrics such as the ℓp _p norm, we conducted two user studies comparing adversarial samples from -- M-Attack-V2 and several baseline attacks against clean images. M-Attack-V2 against Clean Images. Participants were shown 50 images (25 perturbed by -- M-Attack-V2 and 25 unmodified) in a random order and asked to label each image as “perturbed” or “clean”. Adversarial images were generated with a perturbation budget ϵ=16ε=16 or ϵ=8ε=8. Results, averaged over 10 distinct user groups, are summarized in Table 15. On average, only 42%42\% of M-Attack-V2 adversarial images are correctly identified as corrupted, meaning that 58%58\% of them pass the human check even under explicit supervision. We further repeat this study at a smaller perturbation budget ϵ=8ε=8, since -- M-Attack-V2 still exceeds other methods by a large margin even under ϵ=8ε=8. As shown in Table 15, the proportion of adversarial images identified by users drops from 42%42\% to 27.4%27.4\% when reducing ϵε from 1616 to 88, while the confusion between adversarial and real images also increases, and the error of identifying clean images also increases. These results highlight the potential threat of -- M-Attack-V2 in real-world scenarios where human inspection is relied upon. Comparison across Attack Methods. In a second study, each participant (10 in total) was shown 40 images: 10 adversarial examples from each of AnyAttack, SSA-CWA, M-Attack-V1, and M-Attack-V2 (again at ϵ=16ε=16). Participants were told that exactly half of the images were corrupted and asked to select the 20 images they believed were mostly perturbed. This protocol directly compares the perceptual stealthiness of different attacks. As reported in Table 16, AnyAttack is the most easily detected method, with 84%84\% of its images identified as perturbed. SSA-CWA is somewhat less detectable (54%54\%), while M-Attack-V1 and M-Attack-V2 are flagged as perturbed only about 30%30\% of the time, indicating substantially higher perceptual stealth. Notably, this - M-Attack and -- M-Attack-V2 share a similar portion, showing that the slight differences in perturbations’ ℓp _p norm do not necessarily translate to the final human imperceptibility. Table 15: Human study on the imperceptibility of -- M-Attack-V2 under different perturbation budgets. We report the proportion (%) of images identified by users; results are averaged over 10 user groups (mean ± std). Proportion ϵ=16ε=16, Mean ± Std ϵ=8ε=8, Mean ± Std Adversarial images correctly identified 42±1.742± 1.7 27.4±1.627.4± 1.6 Original images correctly identified 98±1.698± 1.6 93.1±2.393.1± 2.3 Table 16: Proportion of adversarial images from each attack that participants judged as perturbed in Study I (all at ϵ=16ε=16). Lower values indicate more perceptually stealthy perturbations. Method Identified as perturbed images (%) AnyAttack 84±4.4784± 4.47 SSA-CWA 54±8.4954± 8.49 M-Attack-V1 30±10.130± 10.1 M-Attack-V2 32±8.032± 8.0 Table 17: Unified robustness under input–preprocessing defenses. We report KMRaKMR_a, KMRbKMR_b, KMRcKMR_c, and ASR (↑) for GPT‑4o, Claude-3.7, and Gemini-2.5. Bold indicates the best value within each metric column for the given defense block; shaded cells highlight M‑Attack‑V2 (numeric cells only). Setting Method GPT-4o Claude 3.7 Gemini 2.5 KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR JPEG (Q=75Q=75) AttackVLM 0.06 0.02 0.00 0.03 0.07 0.02 0.00 0.02 0.08 0.04 0.00 0.04 SSA-CWA 0.08 0.04 0.01 0.10 0.07 0.02 0.00 0.05 0.09 0.06 0.01 0.09 AnyAttack 0.06 0.03 0.00 0.05 0.04 0.01 0.00 0.03 0.08 0.03 0.00 0.05 - M-Attack 0.76 0.54 0.16 0.91 0.28 0.17 0.03 0.34 0.75 0.51 0.11 0.76 -- M-Attack-V2 0.89 0.69 0.20 0.97 0.55 0.36 0.09 0.68 0.75 0.56 0.18 0.82 DiffPure (t=25t=25) AttackVLM 0.05 0.02 0.00 0.01 0.05 0.02 0.00 0.01 0.08 0.03 0.00 0.01 SSA-CWA 0.07 0.03 0.00 0.02 0.04 0.02 0.00 0.03 0.07 0.01 0.00 0.05 AnyAttack 0.07 0.03 0.00 0.04 0.02 0.02 0.00 0.04 0.09 0.04 0.00 0.07 - M-Attack 0.42 0.20 0.03 0.43 0.10 0.05 0.01 0.10 0.39 0.22 0.01 0.32 -- M-Attack-V2 0.73 0.47 0.15 0.72 0.19 0.11 0.04 0.20 0.61 0.42 0.06 0.56 DiffPure (t=75t=75) AttackVLM 0.08 0.05 0.00 0.02 0.04 0.02 0.00 0.00 0.04 0.01 0.00 0.01 SSA-CWA 0.05 0.03 0.01 0.06 0.05 0.03 0.00 0.03 0.07 0.02 0.00 0.05 AnyAttack 0.05 0.00 0.00 0.06 0.04 0.02 0.00 0.03 0.04 0.02 0.00 0.07 - M-Attack 0.10 0.02 0.00 0.04 0.03 0.02 0.00 0.02 0.05 0.05 0.00 0.05 -- M-Attack-V2 0.13 0.06 0.01 0.07 0.07 0.02 0.00 0.06 0.12 0.06 0.01 0.08 Table 18: Cross-domain results on ChestMNIST, from MedMNIST (Yang et al., 2021). We report KMRa/KMRb/KMRcKMR_a/KMR_b/KMR_c and ASR (higher is better). Bold = best in column; underline = second best. The shaded row is our method. Method GPT-4o Claude 3.7 Gemini 2.5 KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR KMRaKMR_a KMRbKMR_b KMRcKMR_c ASR AttackVLM 0.06 0.01 0.00 0.03 0.05 0.02 0.00 0.02 0.08 0.03 0.00 0.02 SSA-CWA 0.06 0.03 0.00 0.15 0.04 0.03 0.00 0.07 0.08 0.02 0.01 0.14 AnyAttack 0.06 0.02 0.00 0.05 0.03 0.01 0.00 0.04 0.07 0.02 0.00 0.05 - M-Attack 0.89 0.70 0.22 0.92 0.31 0.18 0.07 0.31 0.85 0.67 0.23 0.96 -- M-Attack-V2 0.90 0.74 0.27 0.97 0.70 0.51 0.21 0.83 0.89 0.76 0.33 0.95 Appendix H Computational Cost and Runtime Analysis This section analyzes the computational cost of - M-Attack and related baselines, first in terms of FLOPs and then in terms of wall-clock runtime. Let d denote the hidden dimension, dff=4d_f=4d the feed-forward expansion, and N the sequence length. In one Transformer layer, the feed-forward network (FFN) incurs 2Nddff=8Nd22Nd\,d_f=8Nd^2 FLOPs, while multi-head attention adds 4Nd2+4N2d4Nd^2+4N^2d, giving a per-layer forward cost M≜ 12Nd2+4N2d.M\; \;12Nd^2+4N^2d. Since the backward pass is empirically about twice as expensive as the forward pass, a complete forward–backward iteration for a single VLM requires approximately 3M3M FLOPs. Exhaustively accounting FLOPs for each architecture in an N-model ensemble is impractical. Instead, we introduce an empirically measured inflation factor ρN≜per-iteration FLOPs of the N-model ensembleper-iteration FLOPs of one CLIP-B/16, _N\; \; per-iteration FLOPs of the $N$-model ensembleper-iteration FLOPs of one CLIP-B/16, with ρ1=1 _1=1. Under this convention: • AttackVLM costs 3M3M FLOPs per iteration. • M-Attack-V1 costs 3ρNM3 _NM FLOPs per iteration. • SSA-CWA adds an inner sampling loop of K K steps for sharpness-aware minimization (SAM), lifting the complexity to 3ρNK^M3 _N KM. • M-Attack-V2 evaluates K local crops and, for each crop, forwards P auxiliary examples to reduce the variance of - M-Attack. This gives a complexity ofρNK(3M+PM)=ρNK(3+P)M, _NK (3M+PM )\;=\; _NK(3+P)M, where P is typically a small integer (e.g., P=2P=2). In practice, GPUs parallelize many of these operations, so wall-clock time per image does not scale linearly with FLOPs. On a single NVIDIA RTX 4090 GPU, we measure the average time per attacked image (while running a batched optimization with 32 images at the same time) as follows: • SSA-CWA: 545.80±4.21545.80± 4.21 s per image; • M-Attack-V1: 22.04±0.1122.04± 0.11 s per image; • M-Attack-V2 (with K=2K=2, P=2P=2): 24.13±0.8424.13± 0.84 s per image. Thus, with K=2,P=2K=2,P=2, - M-Attack increases the runtime of M-Attack-V1 by only 9.4%9.4\%, while yielding substantial gains in attack quality: on Claude 3.7, we observe improvements of +20%+20\%, +17%+17\%, +3%+3\%, and +13%+13\% on KMRaKMR_a, KMRbKMR_b, KMRcKMR_c, and ASR, respectively; on GPT-4o, the corresponding gains are +8%+8\%, +3%+3\%, +1%+1\%, and +6%+6\%. More results are presented in the Appx. F.1. Therefore, our -- M-Attack-V2 offers a configurable option between efficient yet effective and highly effective attack (as presented in the main paper). We omit AttackVLM and AnyAttack from the empirical runtime comparison: AttackVLM reports relatively low attack performance, while AnyAttack devotes most of its computational budget to its novel pre-training stage, making its overall cost not directly comparable to inference-time attacks like ours. Figure 12: Visualization of GPT-o3’s response towards -- M-Attack-V2 adversarial samples. The underlined ‘glitchy’ denotes that O3 notices something unusual. Appendix I Additional Visualization I.1 Visualization of Adversarial Samples Fig. 4 and Fig. 8 visualize adversarial samples of different black-box attack algorithms under different perturbation constraints. Under ϵ=8ε=8, no significant difference exists between - M-Attack and -- M-Attack-V2. On the ϵ=16ε=16 setting, the visual effect is still very close between - M-Attack and -- M-Attack-V2. Since our -- M-Attack-V2 also greatly improves the results under ϵ=8ε=8, future directions might be improving the imperceptibility by adding constraints besides the ℓ∞ _∞. We also provide all 100 images in the supplementary material for further reference. I.2 Visualization of Reasoning Models Fig. 12 illustrates how GPT-o3 (OpenAI, 2025) responds to our adversarial samples. The model’s visual reasoning behaviors can be broadly categorized into three types: no reasoning (response (d)), simple reasoning (responses (b) and (c)), and zoom-in reasoning (response (a)). Notably, in response (a), GPT-o3 already identifies the central area as uncertain and zooms in on it. However, its reasoning mechanism is not well-equipped to handle adversarial perturbations, resulting in a response that remains semantically close to the target image despite the perturbation. This observation suggests that vision reasoning offers a degree of robustness by detecting uncertainty and taking subsequent actions. During training, incorporating explicit behaviors, such as refusing to answer or flagging potential adversarial inputs, could further enhance the utility of vision-based inference under adversarial conditions.