Paper deep dive
Visual-Invariance-Augmented Feature Optimal Alignment for Transferable Adversarial Attacks against Closed-Source MLLMs
Xiaojun Jia, Simeng Qin, Yiming Li, Jie Liao, Sensen Gao, Ke Ma, Yang Liu, Xiaochun Cao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/7/2026, 8:12:55 AM
Summary
The paper introduces IAU-FOA, a novel framework for generating transferable adversarial attacks against closed-source Multimodal Large Language Models (MLLMs). It addresses limitations in existing methods by employing visual-invariance-augmented feature optimal alignment. Key components include dual-granularity alignment (global cosine similarity and local patch-token matching via optimal transport), an adaptive unbalanced transport mechanism to handle unreliable local matches, dynamic ensemble weighting of surrogate models, and visual-invariance augmentation (VIA) to improve robustness across heterogeneous visual encoders.
Entities (10)
Relation Signals (8)
IAU-FOA → targets → Multimodal Large Language Models
confidence 95% · We propose IAU-FOA... to improve targeted transferability against closed-source MLLMs.
IAU-FOA → uses → Visual-Invariance Augmentation
confidence 93% · IAU-FOA incorporates a visual-invariance augmentation (VIA) strategy during adversarial optimization.
IAU-FOA → uses → Optimal Transport
confidence 92% · patch tokens are clustered into compact local patterns and matched through optimal transport for fine-grained feature alignment.
IAU-FOA → uses → Adaptive Unbalanced Transport
confidence 91% · We therefore introduce confidence-adaptive unbalanced transport to relax these constraints for weakly matched clusters...
Visual-Invariance Augmentation → improves → Adversarial Transferability
confidence 90% · This strategy encourages adversarial perturbations to generalize across different visual encoders.
CLIP → isusedby → Multimodal Large Language Models
confidence 90% · many MLLMs adopt pre-trained vision encoders, such as Contrastive Language-Image Pre-training (CLIP)...
Adaptive Unbalanced Transport → reduces → misleading alignment gradients
confidence 88% · aiming to reduce unreliable local alignment and improve adversarial transferability.
GPT-5.5 → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal large language models (MLLMs) remain vulnerable to transferable adversarial examples, especially in black-box settings where only open-source surrogate models are accessible. Existing targeted transfer attacks mainly align adversarial and target samples using global image-level features, such as encoder [CLS] embeddings. However, such coarse alignment insufficiently exploits patch-level visual structures, limiting transferability across heterogeneous closed-source MLLMs. We propose IAU-FOA, a visual-invariance-augmented feature optimal alignment attack with adaptive unbalanced transport, to improve targeted transferability against closed-source MLLMs. IAU-FOA aligns adversarial and target samples at both global and local levels: a cosine-based objective narrows their global semantic gap, while patch tokens are clustered into compact local patterns and matched through optimal transport for fine-grained feature alignment. Balanced optimal transport enforces fixed marginal masses even for local clusters without reliable counterparts, potentially introducing misleading alignment gradients. We therefore introduce confidence-adaptive unbalanced transport to relax these constraints for weakly matched clusters, aiming to reduce unreliable local alignment and improve adversarial transferability. We further study the effect of input transformations and propose visual-invariance augmentation, which applies bidirectional pixel-intensity rescaling and per-channel white-balance adjustment to simulate exposure, contrast, illumination, and color-temperature variations. This strategy encourages adversarial perturbations to generalize across different visual encoders. Extensive experiments on open-source and closed-source MLLMs show that IAU-FOA consistently outperforms state-of-the-art transferable attack methods. Code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.06977v1
- Canonical: https://arxiv.org/abs/2610.06977v1
Trouble viewing inline? Open PDF directly →
Full Text
108,937 characters extracted from source content.
Expand or collapse full text
Visual-Invariance-Augmented Feature Optimal Alignment for Transferable Adversarial Attacks against Closed-Source MLLMsThanks: Xiaojun Jia, Yang Liu, Yiming Li, and Sensen Gao are with Nanyang Technological University, Singapore (e-mail: jiaxiaojunqaq@gmail.com; yangliu@ntu.edu.sg; liyiming.tech@gmail.com; sensen.gao2002@gmail.com). Simeng Qin is with Northeastern University, Shenyang, Liaoning, China (e-mail: qinsimeng@neuq.edu.cn). Jie Liao is with Chongqing University, Chongqing, China (e-mail: liaojie@cqu.edu.cn). Ke Ma is with the School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing 100049, China (e-mail: make@ucas.ac.cn). Xiaochun Cao is with the School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, Shenzhen 518107, China (e-mail: caoxiaochun@mail.sysu.edu.cn). Xiaojun Jia Simeng Qin Yiming Li Jie Liao Sensen Gao Affiliation: Ke Ma, Yang Liu, , and Xiaochun Cao Abstract Multimodal large language models (MLLMs) remain vulnerable to transferable adversarial examples, especially in black-box scenarios where only open-source surrogate models are accessible. Existing targeted transfer attacks mainly align adversarial and target samples using global image-level features, such as the [CLS] embedding of vision-language encoders. However, such coarse alignment insufficiently exploits patch-level visual structures, leading to limited transferability across heterogeneous closed-source MLLMs. To address this issue, we propose a visual-invariance-augmented feature optimal alignment attack with adaptive unbalanced transport, called IAU-FOA, for improving targeted transferability against closed-source MLLMs. IAU-FOA aligns adversarial and target samples from both global and local perspectives: a cosine-based objective narrows their global semantic gap, while patch tokens are clustered into compact local patterns and matched through optimal transport for fine-grained feature alignment. Balanced optimal transport enforces fixed marginal masses even for local clusters without reliable counterparts, potentially introducing misleading alignment gradients. We therefore introduce confidence-adaptive unbalanced transport to relax these constraints for weakly matched clusters, aiming to reduce unreliable local alignment and improve adversarial transferability. Moreover, we study the effect of input transformations on adversarial transferability and propose a visual-invariance augmentation strategy that applies bidirectional pixel-intensity rescaling and per-channel white-balance adjustment to simulate exposure, contrast, illumination, and color-temperature variations, thereby encouraging adversarial perturbations to generalize across different visual encoders. Extensive experiments on open-source and closed-source MLLMs show that IAU-FOA consistently outperforms state-of-the-art transferable attack methods. The code is released at https://github.com/jiaxiaojunQAQ/IAU-FOA. Index Terms: Multimodal large language models, transferable adversarial attacks, feature optimal alignment, adaptive unbalanced optimal transport, visual-invariance augmentation. I Introduction Multimodal large language models (MLLMs) [55, 6] have become a representative framework for connecting visual perception with language reasoning [15, 21]. In most current systems, an input image is first encoded into visual tokens or embeddings by a vision encoder, and these visual representations are then projected into the language model for multimodal understanding and generation [22, 29, 62, 33, 46]. In particular, many MLLMs adopt pre-trained vision encoders, such as Contrastive Language-Image Pre-training (CLIP) [40], to extract image features and align them with language embeddings. This modular design has enabled strong performance on a wide range of vision-language applications. Beyond advances in open-source MLLMs, commercial closed-source MLLMs such as GPT-5.5, Claude-Opus-4.8, and Gemini-3.5-Flash have also been widely deployed in practical multimodal services. However, since MLLMs heavily rely on vision encoders to perceive visual inputs, they may also inherit the adversarial vulnerabilities of these visual front ends [56, 34, 45]. Recent studies [4, 7, 30, 37] have shown that MLLMs can be vulnerable to adversarial examples [8], which are typically generated by adding small and visually imperceptible perturbations to clean images, such that the perturbed images remain natural to humans but change the model’s responses. Such vulnerability raises security and safety concerns for real-world foundation-model services, where manipulated visual inputs may mislead downstream multimodal reasoning. According to the attack objective, adversarial attacks on MLLMs can be broadly divided into untargeted and targeted attacks. Untargeted attacks aim to induce incorrect responses, whereas targeted attacks attempt to drive the model toward attacker-specified semantics or outputs [57, 19]. Fig. 1: Examples of targeted transfer attacks against commercial MLLMs. The adversarial images remain visually close to the source images, but induce target-related responses to the prompt “Describe this image”. As illustrated in Fig. 1, targeted adversarial examples can preserve the visual appearance of the source image while steering commercial MLLMs toward target-related descriptions. In closed-source scenarios, the gradients and parameters of target MLLMs are inaccessible, making direct optimization infeasible. A practical alternative is to craft adversarial examples on open-source surrogate models and then transfer them to the target systems. This property, known as adversarial transferability [9], measures whether adversarial examples generated for one model can remain effective on other unseen models. Transferability is important for both untargeted and targeted attacks, but it is particularly difficult to achieve in the targeted black-box setting, where the perturbation must consistently convey the target semantics across heterogeneous MLLMs rather than merely causing prediction errors [2, 63]. Therefore, an effective targeted attack should not simply exploit surrogate-specific weaknesses, but should learn perturbations that preserve transferable visual semantics across different visual encoders and preprocessing pipelines. Previous targeted transfer attack methods [24, 23, 45] typically generate adversarial examples by minimizing a contrastive loss between the global features of the adversarial image and the target image, where the global representation is often given by the [CLS] token of an open-source image encoder such as CLIP. By pulling the adversarial image toward the target image in the surrogate feature space, these methods can improve targeted transferability to some extent. However, their adversarial transferability and attack performance against closed-source MLLMs remain limited, mainly due to two limitations. First, most existing methods rely primarily on global image-level features. Although global feature alignment can guide adversarial examples toward the target semantics, it compresses the visual content into a single representation and overlooks rich patch-level structures. As a result, the generated perturbations may capture coarse semantic similarity while failing to sufficiently exploit fine-grained visual patterns that are important for transfer across heterogeneous MLLMs. Even when local features are aligned, exact marginal constraints can force matches between clusters without reliable counterparts. These matches may introduce gradients that favor incidental similarities in a surrogate encoder rather than correspondences shared across models, motivating confidence-adaptive local transport. Second, the choice of input transformations for targeted transfer attacks against MLLMs warrants further investigation. Existing approaches include spatial transformations such as resizing and padding in DI [49], as well as pixel-intensity scaling in SIM [26]. Building on intensity-based augmentation, we investigate the combination of bidirectional intensity scaling and channel-wise gain adjustment to diversify exposure and illumination-color conditions during targeted feature alignment. To address these issues, we propose IAU-FOA, a visual-invariance-augmented feature optimal alignment attack with an adaptive unbalanced transport method. Given a source image and a target image, IAU-FOA generates an adversarial perturbation by aligning their representations in surrogate vision-language encoders from both global and local perspectives. At the global level, IAU-FOA adopts a coarse-grained feature alignment objective based on cosine similarity, which pulls the image-level representation of the adversarial example toward that of the target sample. Since the global representation is usually encoded by the [CLS] token, this objective provides a stable semantic direction for targeted attacks and encourages the adversarial image to approach the target semantics in the surrogate feature space. At the local level, IAU-FOA further exploits the patch-token representations produced by Transformer-based vision encoders. While the [CLS] token summarizes global information, the remaining visual tokens contain local patch-level details. Directly using all local tokens, however, can introduce redundancy and noisy correspondences. Therefore, IAU-FOA first clusters patch tokens into compact and discriminative local visual patterns, where each cluster center summarizes the characteristics of a group of related patches. It then formulates the alignment between adversarial and target local patterns as an optimal transport problem, enabling fine-grained feature matching beyond a single global embedding. To make local alignment more reliable, IAU-FOA further introduces an adaptive unbalanced transport mechanism. Balanced OT requires each cluster to transport its prescribed mass, even when reliable counterparts are absent. The proposed scheme instead adapts marginal regularization to cluster-level matching confidence. High-confidence clusters receive stronger penalties on marginal deviations, while low-confidence clusters receive weaker penalties. This flexibility aims to reduce the interference of unreliable local matches in adversarial optimization and thereby improve adversarial transferability. In addition, when multiple visual encoders are used as surrogate models, IAU-FOA computes an alignment similarity score for each encoder and dynamically adjusts its weight using the ratio between the current and previously recorded scores. For positive scores, larger relative increases receive larger weights, allowing the ensemble to respond to changes in alignment across surrogate encoders. Finally, IAU-FOA incorporates a visual-invariance augmentation (VIA) strategy during adversarial optimization. Building on existing intensity-scaling augmentation, our strategy combines bidirectional pixel-intensity rescaling with channel-wise gain adjustment to approximate exposure and white-balance variations during targeted alignment. By optimizing adversarial perturbations under these transformed visual conditions, IAU-FOA encourages the perturbations to rely less on surrogate-specific visual statistics and to remain effective across heterogeneous closed-source MLLMs. Our main contributions are summarized as follows: • We propose IAU-FOA, a targeted black-box attack framework for MLLMs that drives adversarial images toward target semantics through a dual-granularity feature matching objective. Specifically, a cosine-based loss provides coarse semantic guidance from global embeddings, while clustered patch tokens are matched through an optimal-transport formulation to capture discriminative local structures. • We propose an adaptive unbalanced transport mechanism to improve the reliability of fine-grained matching. It relaxes marginal constraints for weakly matched clusters, aiming to reduce misleading alignment gradients arising from unreliable local correspondences and improve targeted transferability. • We propose a dynamic weighting strategy for ensemble surrogate optimization. The strategy adjusts each encoder’s contribution according to the relative change in its alignment similarity score, providing a score-based alternative to fixed ensemble weights for targeted transfer. • We propose a visual-invariance augmentation strategy for transferable adversarial optimization. Through bidirectional intensity scaling and channel-wise white-balance adjustment, the attack is optimized under diverse photometric conditions and becomes less dependent on surrogate-specific visual statistics. • Extensive experiments on open-source and closed-source MLLMs demonstrate that IAU-FOA consistently outperforms state-of-the-art transferable attack methods. This paper is an extension of our preliminary conference version, FOA [17], which forms the foundation of the proposed framework. Compared with the original balanced feature optimal alignment framework, we extend this work in four aspects: 1) We introduce visual-invariance-augmented optimization in Section I-C to improve robustness against photometric changes and encoder-dependent visual statistics. 2) We develop an adaptive unbalanced transport method for local feature alignment in Section I-F, which replaces exact marginal preservation with confidence-dependent penalties on marginal deviations. We add a theoretical analysis of the proposed method in Section I-H. 3) We conduct more experiments and analyses, including comparisons with state-of-the-art methods, ablation studies, and evaluation under adversarial defenses. We include additional state-of-the-art adversarial attack methods for comparison in Section IV-C. We add ablation studies of the proposed components in Section IV-E and evaluate the proposed method under adversarial defenses in Section IV-D. 4) We have thoroughly revised the abstract, introduction, method, experiments, and conclusion sections to offer a more detailed overview of our motivation and approach. We have updated all figures and tables to enhance clarity and presentation. I Related Work I-A Multimodal Large Language Models Large language models (LLMs) [60] have achieved impressive success across a wide range of Natural Language Processing (NLP) tasks, demonstrating strong capabilities in language understanding [20], instruction following [13], and complex reasoning [35, 11]. Building upon these advances, multimodal large language models (MLLMs) extend LLMs to vision-language scenarios by integrating visual perception modules with LLM backbones. Specifically, MLLMs align visual representations with the semantic space of LLMs, enabling the language model to interpret visual inputs and generate grounded textual responses [1, 42, 28]. Representative systems usually connect pre-trained vision encoders and LLMs through lightweight alignment modules, such as the Q-Former in BLIP-2 [22] and the MLP projector in LLaVA [29]. Some open-source models further enhance fine-grained visual perception through versatile visual instruction tuning and high-resolution image processing, as demonstrated by Qwen2-VL [46] and InternVL [3]. Meanwhile, commercial MLLMs, such as GPT-5.5 and Claude-Opus-4.8, have exhibited powerful multimodal understanding, reasoning, and instruction-following capabilities, representing the leading performance of current vision-language systems. However, their closed-source nature, together with substantial differences in model architectures, vision encoders, alignment modules, and training recipes, makes their internal representation spaces difficult to access or approximate, posing challenges for black-box attacks against modern MLLMs. I-B Adversarial Attacks Adversarial attacks aim to mislead deep neural networks by adding carefully crafted perturbations to clean inputs [43, 8]. According to the attacker’s access to the victim model, existing adversarial attack methods are commonly categorized into white-box attacks, where model parameters or gradients are available, and black-box attacks, where the attacker has no access to internal model parameters, gradients, or feature representations [36, 58]. Depending on the attack objective, adversarial attacks can also be divided into untargeted attacks and targeted attacks. Untargeted attacks only require the model to produce incorrect outputs, whereas targeted attacks aim to induce specific target semantics or responses. Compared with conventional adversarial attacks on image classification models, attacking MLLMs is substantially more challenging [19, 16]. Classification models usually produce predictions over a fixed label space, allowing attacks to optimize well-defined objectives. In contrast, MLLMs generate open-ended textual responses conditioned on both visual and textual inputs, where the attack goal is often expressed at the semantic or instruction-following level rather than as a discrete label. For closed-source MLLMs, the challenge becomes more pronounced because their parameters, gradients, and intermediate feature representations are unavailable to attackers. A potential solution is to exploit adversarial transferability [39, 31], where adversarial examples generated on accessible surrogate models remain effective on unseen target models. This property enables transfer-based attacks to probe closed-source MLLMs without directly accessing their internal states, making it a practical and promising route for targeted black-box attacks [49, 26]. I-C Transfer-Based Targeted Attacks for MLLMs Recent transfer-based attacks on vision-language models typically optimize adversarial perturbations on accessible surrogate models, such as CLIP, and transfer them to closed-source MLLMs. Specifically, Guo et al. [12] propose AdvDiffVLM, which leverages diffusion models to generate natural and transferable targeted adversarial examples for vision-language models. Zhang et al. [59] propose AnyAttack, a self-supervised framework for large-scale generation of label-free targeted adversarial examples, improving the scalability of targeted transfer attacks. Xie et al. [50] propose Chain of Attack (COA), a transfer-based black-box attack method that iteratively generates adversarial examples through multimodal semantic updates between visual and textual modalities. To better exploit visual details for closed-source MLLM attacks, Li et al. [24] propose M-Attack, which constructs a simple yet effective black-box attack baseline by enhancing target-related visual patterns. Zhao et al. [61] further propose M-Attack-v2, which emphasizes fine-grained detail targeting to improve attack effectiveness against strong black-box MLLMs. Li et al. [23] propose MPCAttack, which collaboratively aggregates multi-paradigm vision-language features to reduce surrogate representation bias and strengthen transfer-based attacks against MLLMs. Despite these advances, existing targeted transfer attacks still show limited effectiveness against advanced closed-source MLLMs, partly because they often rely on surrogate-specific global representations or do not sufficiently account for the reliability of local correspondences. In addition, fine-grained adaptive alignment and visual-invariance optimization remain insufficiently explored. Fig. 2: Overview of IAU-FOA: visual-invariance augmentation, global feature alignment, adaptive unbalanced local transport, and dynamic surrogate weighting. I Method Adversarial perturbations learned from a single surrogate may capture non-transferable representation biases, which can be alleviated by optimizing over multiple surrogate image encoders to emphasize shared feature directions across models [41, 52, 45]. Following this perspective, IAU-FOA adopts a dynamic ensemble framework for targeted transfer attacks against closed-source MLLMs. The proposed framework integrates dual-granularity feature optimal alignment, adaptive unbalanced local transport, dynamic encoder weighting, and visual-invariance augmentation (VIA). By jointly aligning target semantics across multiple surrogate visual encoders and adapting their weights to relative changes in alignment similarity, IAU-FOA encourages adversarial perturbations to capture transferable visual cues shared by foundation models. I-A Problem Formulation We study targeted transfer-based attacks against closed-source MLLMs using an ensemble of accessible surrogate visual encoders. Let ℱ=fmm=1MF=\f_m\_m=1^M (1) denote M surrogate visual encoders, including vision-language and vision-only models. For an input image ∈x , where X denotes the valid image domain, the global and patch-level representations supplied by the m-th feature extractor are denoted as fm()=[mg();mp()]∈ℝ(Nm+1)×dm,f_m(x)=[z_m^g(x);Z_m^p(x)] ^(N_m+1)× d_m, (2) where mg()∈ℝdmz_m^g(x) ^d_m is the global representation supplied by the feature extractor (e.g., a [CLS] embedding), and mp()=[m,1p(),…,m,Nmp()]⊤∈ℝNm×dmZ_m^p(x)=[z_m,1^p(x),…,z_m,N_m^p(x)] ^N_m× d_m denotes the patch-token features. Given a source image sx_s and a target image tarx_tar, the attacker generates an adversarial example adv=s+,‖∞≤ϵ,x_adv=x_s+ δ, \| δ\|_∞≤ε, (3) where δ is the bounded adversarial perturbation. The goal is to make the adversarial image near the target image in the feature spaces of all surrogate encoders: min∑m=1M‖∞≤ϵm(fm(adv),fm(tar)), _\| δ\|_∞≤ε _m=1^MD_m (f_m(x_adv),f_m(x_tar) ), (4) where m(⋅,⋅)D_m(·,·) denotes the feature-alignment objective on the m-th encoder. Since the victim MLLM is closed-source, its parameters and gradients are unavailable, and the perturbation optimized on ℱF is directly transferred to unseen MLLMs. IAU-FOA optimizes the adversarial perturbation by reducing the feature-alignment loss between the adversarial image and the target image from both global and local perspectives. I-B Overview of the Proposed Framework The proposed IAU-FOA consists of four components: visual-invariance augmentation, dual-granularity feature alignment, adaptive unbalanced optimal transport, and dynamic ensemble weighting. The current target view is passed through each surrogate encoder to obtain global and clustered local target features. We use tarx_tar below to denote the target view supplied to the alignment objective, and suppress spatial preprocessing in the notation. During optimization, multiple photometrically transformed copies of the adversarial image are generated. For each copy, IAU-FOA computes a weighted ensemble objective over surrogate encoders and averages the gradients from all valid copies. The loss minimized at each iteration can be written as min‖∞≤ϵ1|ℬ|∑∈ℬ∑m=1Mωm(k())(CLOSE _\| δ\|_∞≤ε 1|B| _T _m=1^M _m^(k(T)) ( ℒgm((adv),tar) _g^m(T(x_adv),x_tar) (5) OPEN+λlℒlm((adv),tar)), + _lL_l^m(T(x_adv),x_tar) ), where ℬB is the set of visual-invariance transformations sampled at the current iteration, k()k(T) indexes the objective evaluation for the transformed copy (adv)T(x_adv), ωm(k()) _m^(k(T)) is the corresponding dynamic weight of the m-th surrogate encoder, and λl _l controls the contribution of local alignment. The weights are updated at each objective evaluation and held fixed during backpropagation. Eq. (5) summarizes the optimization used to update the perturbation. The outer average makes the perturbation effective under different photometric views (adv)T(x_adv), which reduces overfitting to the visual statistics of a single input rendering. For each transformed copy, the inner summation aggregates the attack loss over all surrogate encoders. Within each surrogate, ℒgmL_g^m measures the global feature alignment loss to the target image, while ℒlmL_l^m measures the transported local alignment loss between clustered patch-level structures. Thus, minimizing Eq. (5) drives advx_adv toward the target image simultaneously at the semantic level and at the local visual-pattern level. The weight ωm(k()) _m^(k(T)) is computed from relative changes in alignment similarity scores, as specified in Sec. I-G. I-C Visual-Invariance Augmentation Real imaging conditions and closed-source MLLM visual preprocessing pipelines often introduce photometric variations, such as changes in exposure, brightness, illumination color, and white balance. Accurately modeling the full physical image formation process is difficult for transferable optimization. Instead, we use two simple, differentiable, and low-cost operations, namely global intensity scaling and channel-wise gain adjustment, to cover these common visual variations. (adv)T(x_adv) denotes a transformed view of the adversarial image. To keep the notation compact, we use T as a generic augmentation operator in the overall objective, and instantiate it here with photometric transformations. Specifically, each transformation combines an intensity scale s and a channel-wise gain vector =[βR,βG,βB]⊤ β=[ _R, _G, _B] . We enumerate the intensity scale s over =2k∣k=−h,…,h,S=\2^k k=-h,…,h\, (6) where h is a nonnegative integer controlling the range of intensity scaling, and k indexes the 2h+12h+1 scales. Scaling darkens (s<1s<1), brightens (s>1s>1), or preserves (s=1s=1) image intensity. By default, h=2h=2, giving five bidirectional intensity scales =0.25,0.5,1,2,4S=\0.25,0.5,1,2,4\. Given s∈s and ∈ℝ3 β ^3, the concrete operator used in Eq. (5) is (,s,)=⊙(s),T(x;s, β)= β (sx), (7) where ⊙ denotes channel-wise multiplication. For an RGB image ∈ℝH×W×3x ^H× W× 3, this operation is defined as [(,s,)]u,v,c=βc⋅s⋅u,v,c,c∈R,G,B.[T(x;s, β)]_u,v,c= _c· s·x_u,v,c, c∈\R,G,B\. (8) Here, H and W denote the image height and width, u and v index the spatial pixel location, and c indexes the RGB channel. The global scale s is shared across all pixels and channels, while βc _c is the gain applied to all pixels in channel c. For each transformed copy, the channel gains are independently sampled as βc∼(1−sβ,1+sβ),c∈R,G,B. _c (1-s_β,1+s_β), c∈\R,G,B\. (9) Here, (a,b)U(a,b) denotes the uniform distribution over [a,b][a,b], and sβs_β controls the amplitude of channel-wise gain variations around one. We set sβ=0.5s_β=0.5 by default, giving a gain range of [0.5,1.5][0.5,1.5]; sβ=0s_β=0 disables channel-wise gain adjustment. The global scale s approximates changes in brightness and exposure, while the channel gains β approximate white-balance and illumination-color shifts. The transformation set ℬB in Eq. (5) is constructed by enumerating s∈s and sampling a channel-gain vector β for each scale. These transformations simulate exposure, illumination, and color-temperature changes. In each iteration, gradients from all transformed copies are averaged before updating the perturbation, which encourages the adversarial example to remain effective under heterogeneous visual preprocessing pipelines. Each transformed copy is then passed to the surrogate encoders to generate adversarial examples. I-D Global Feature Alignment The global branch uses the encoder-specific global feature to provide coarse-grained semantic guidance. For the m-th encoder, the global feature is first normalized as ^mg()=mg()‖mg()‖2. z_m^g(x)= z_m^g(x)\|z_m^g(x)\|_2. (10) Let m=^mg(adv)X_m= z_m^g(x_adv) and m=^mg(tar)Y_m= z_m^g(x_tar) denote the normalized global features of the adversarial and target images, respectively. The global alignment loss is defined by the cosine distance between these two features: ℒgm(adv,tar) _g^m(x_adv,x_tar) =1−cos(m,m) =1- (X_m,Y_m) (11) =1−⟨m,m⟩‖m‖2‖m‖2, =1- _m,Y_m \|X_m\|_2\|Y_m\|_2, where ⟨m,m⟩ _m,Y_m denotes the inner product between the two feature vectors, and ∥⋅∥2\|·\|_2 denotes the ℓ2 _2 norm, i.e., the Euclidean length of a vector. Minimizing Eq. (11) increases the semantic similarity between the adversarial image and the target image in the global feature representation space. I-E Local Clustering Feature Alignment Patch tokens retain spatially distributed visual details beyond a single global representation. However, all patch tokens also include substantial redundant image information, and directly matching them may introduce considerable noisy correspondences, degrading the quality of the generated adversarial examples. To reduce redundancy while preserving discriminative local information, we cluster patch-token features into compact local visual patterns before feature alignment. For the m-th encoder, the fine-grained local features of the adversarial and target images are extracted from the patch-token feature matrices: mloc _m^loc =mp(adv)∈ℝNm×dm, =Z_m^p(x_adv) ^N_m× d_m, (12) mloc _m^loc =mp(tar)∈ℝNm×dm, =Z_m^p(x_tar) ^N_m× d_m, where each row corresponds to the feature of one image patch. We then apply clustering to the local features of the adversarial and target images, respectively. Here, KMeans(,K)KMeans(A,K) denotes the standard K-means clustering operator applied to the row vectors of A, which partitions them into K clusters by minimizing the within-cluster squared Euclidean distance and returns the cluster centroids: mclu _m^clu =KMeans(mloc,K), =KMeans (X_m^loc,K ), (13) mclu _m^clu =KMeans(mloc,K), =KMeans (Y_m^loc,K ), where mclu=[m,1clu,…,m,Kclu]⊤∈ℝK×dmX_m^clu=[x_m,1^clu,…,x_m,K^clu] ^K× d_m and mclu=[m,1clu,…,m,Kclu]⊤∈ℝK×dmY_m^clu=[y_m,1^clu,…,y_m,K^clu] ^K× d_m are the adversarial and target cluster centers. We assign uniform reference mass to each cluster center on both sides. The mass vectors madva_m^adv and mta_m^t are defined independently of cluster cardinalities as am,kadv=1K,am,kt=1K,k=1,…,K. a_m,k^adv= 1K, a_m,k^t= 1K, k=1,…,K. (14) Both mass vectors are normalized to sum to one. Following the theory of optimal transport [44], we model fine-grained local alignment as a transportation problem between two sets of clustered local patterns. For the m-th encoder, the adversarial local distribution is represented by (m,iclu,am,iadv)i=1K\(x_m,i^clu,a_m,i^adv)\_i=1^K, while the target local distribution is represented by (m,jclu,am,jt)j=1K\(y_m,j^clu,a_m,j^t)\_j=1^K. The goal is to find a transport plan that moves the mass of adversarial local patterns to target local patterns with minimum matching cost. Let m(i,j)P_m(i,j) denote the mass transported from the i-th adversarial cluster to the j-th target cluster, and let cij=c(m,iclu,m,jclu)c_ij=c(x_m,i^clu,y_m,j^clu) be their matching cost. The balanced OT form of local alignment is minm _P_m ∑i=1K∑j=1Km(i,j)cij, _i=1^K _j=1^KP_m(i,j)c_ij, (15) s.t. .t. ∑j=1Km(i,j)=am,iadv,∀i, _j=1^KP_m(i,j)=a_m,i^adv, ∀ i, ∑i=1Km(i,j)=am,jt,∀j, _i=1^KP_m(i,j)=a_m,j^t, ∀ j, m(i,j)≥0,∀i,j. _m(i,j)≥ 0, ∀ i,j. The marginal constraints in Eq. (15) preserve the uniform reference masses on both sides. However, strict balanced constraints require prescribed masses even for clusters without reliable counterparts, potentially introducing misleading local alignment gradients. This motivates the adaptive unbalanced formulation introduced below. Specifically, the ground cost is defined by the cosine distance between local cluster centers. We therefore first construct the pairwise similarity matrix m(i,j)=cos(m,iclu,m,jclu).S_m(i,j)= (x_m,i^clu,y_m,j^clu ). (16) Here, m(i,j)S_m(i,j) measures the cosine similarity between the i-th adversarial cluster and the j-th target cluster, and the corresponding transport cost is set to cij=1−m(i,j)c_ij=1-S_m(i,j). Minimizing this objective favors low-cost correspondences subject to the prescribed marginals. I-F Adaptive Unbalanced Optimal Transport The balanced formulation in Eq. (15) provides a basic local alignment objective in which every cluster mass is exactly transported according to fixed marginals. With entropic regularization, the balanced OT problem can be solved using Sinkhorn iterations. However, exact marginal preservation can be too restrictive for targeted adversarial optimization: background clusters, ambiguous regions, or surrogate-specific patch patterns may be forced to match target clusters even when no reliable correspondence exists. Such forced matches may steer perturbations toward incidental similarities specific to the surrogate, limiting their transfer to other encoders. We therefore use an adaptive unbalanced Sinkhorn solver with weaker marginal regularization for low-confidence clusters and stronger regularization for high-confidence clusters. This design aims to reduce unreliable alignment by allowing transported masses to adapt to the available correspondences. Starting from the cosine-distance cost cij=1−m(i,j)c_ij=1-S_m(i,j) defined above, the entropic transport kernel is m(i,j)=exp(−1−m(i,j)εot),K_m(i,j)= (- 1-S_m(i,j) _ot ), (17) where εot>0 _ot>0 is the entropy regularization coefficient, also serving as the temperature of the Sinkhorn kernel, and is set to 0.10.1 in all experiments. For each adversarial-side cluster on the adaptively relaxed side, the confidence is estimated by its best target counterpart: ri=max0,maxjm(i,j).r_i= \0, _jS_m(i,j) \. (18) We normalize the raw confidence with a stabilized mean: r^i=ri1K∑k=1Krk+η. r_i= r_i 1K _k=1^Kr_k+η. (19) Here, η=10−8η=10^-8. When the mean raw confidence is well above η, the normalized confidences have a mean close to one, modulating marginal regularization around its base level. If all raw confidences are zero, every r^i r_i is zero; with γ=1γ=1, this gives ρia=ϕia=0 _i^a= _i^a=0 while the target-side penalty remains unchanged. We use ρ>0ρ>0 as a hyperparameter controlling the KL penalty on marginal deviations, with ρ=0.2ρ=0.2 by default. Accordingly, the base unbalanced relaxation coefficient is ϕ=ρ+εot,φ= ρ+ _ot, (20) where a larger ρ gives a larger ϕφ and enforces marginal preservation more strongly. The default settings give ϕ=2/3φ=2/3. With mean-normalized adaptive confidence, the adversarial-side relaxation coefficient becomes ρia _i^a =ρ[(1−γ)+γr^i], =ρ [(1-γ)+γ r_i ], (21) ϕia _i^a =ρ[(1−γ)+γr^i]ρ[(1−γ)+γr^i]+εot. = ρ [(1-γ)+γ r_i ]ρ [(1-γ)+γ r_i ]+ _ot. where γ∈[0,1]γ∈[0,1] controls the strength of confidence modulation and is set to 1.01.0 in all experiments. When γ=0γ=0, Eq. (21) reduces to ϕia=ρ/(ρ+εot)=ϕ _i^a=ρ/(ρ+ _ot)=φ, exactly matching plain unbalanced OT. With the adopted setting γ=1γ=1, we have ρia=ρr^i _i^a=ρ r_i and ϕia=ρr^i/(ρr^i+εot) _i^a=ρ r_i/(ρ r_i+ _ot). Thus, high-confidence clusters with r^i>1 r_i>1 receive stronger marginal constraints, whereas low-confidence clusters with r^i<1 r_i<1 are relaxed further. We denote ϕa=[ϕ1a,…,ϕKa]⊤ φ^a=[ _1^a,…, _K^a] . The target side uses the base coefficient ϕφ. When ρ and all ρia _i^a tend to infinity, the solution approaches entropy-regularized balanced OT. With γ=1γ=1, however, a zero-confidence cluster remains unpenalized for any ρ. For fixed similarities and confidence coefficients, we numerically approximate the minimizer of the entropy-regularized problem min≥0 _P≥ 0 ∑i,jPijcij+εot∑i,jPij(logPij−1) _i,jP_ijc_ij+ _ot _i,jP_ij( P_ij-1) +∑iρiad(()i∣am,iadv) + _i _i^a\,d\! ((P1)_i a_m,i^adv ) +ρ∑jd((⊤)j∣am,jt), +ρ _jd\! ((P 1)_j a_m,j^t ), where d(x∣y)=xlog(x/y)−x+yd(x y)=x (x/y)-x+y is the scalar generalized KL divergence for y>0y>0, with 0log(0/y)=00 (0/y)=0. The entropy uses the same zero-mass convention. Confidence coefficients are fixed during each transport solve. We retain the convention that rows correspond to adversarial clusters and columns to target clusters. The implementation stores the transposed matrix, with target clusters first. Under the notation used here, it initializes (0)=(0)=u^(0)=v^(0)=1 and updates target-side scaling before adversarial-side scaling: (ℓ+1) ^( +1) =(mt⊘(m⊤(ℓ)))∘ϕ, = (a_m^t (K_m u^( )) ) φ, (22) (ℓ+1) ^( +1) =(madv⊘(m(ℓ+1)))∘ϕa, = (a_m^adv (K_mv^( +1)) ) φ^a, where ⊘ denotes element-wise division and ∘ denotes element-wise exponentiation; ϕa φ^a acts element-wise on the adversarial-side row scaling u, while the scalar exponent ϕφ is applied to every entry of the target-side column scaling v. The solver runs for at most 100 iterations and stops when K−1‖(ℓ+1)−(ℓ)‖1<10−2K^-1\|v^( +1)-v^( )\|_1<10^-2. We use the final scaling vectors to form the numerical plan m∗P_m^*. The resulting transport plan balances matching costs with confidence-dependent penalties on marginal deviations: m∗=diag()mdiag().P_m^*=diag(u)K_mdiag(v). (23) To compute the local alignment term, the implementation normalizes the transport plan by its total mass: ~m=m∗∑i,jm∗(i,j)+η,η=10−8. P_m= P_m^* _i,jP_m^*(i,j)+η, η=10^-8. The transport plan and its confidence modulation are computed without gradient tracking. The local alignment loss is the normalized transported cosine distance: ℒlm=∑i,j~m(i,j)(1−m(i,j)).L_l^m= _i,j P_m(i,j) (1-S_m(i,j) ). (24) The loss in Eq. (24) measures local matching cost using the normalized plan. The entropy and marginal penalties determine the raw plan m∗P_m^* and are not added to this loss. Gradients of the local loss pass through the similarities while the transport weights are held fixed. Thus, the normalized plan weights the pairwise similarity gradients that guide the perturbation update. Confidence-adaptive relaxation changes these weights to accommodate unreliable correspondences; its effect on transferability is evaluated experimentally. Hence, the final loss of the proposed IAU-FOA can be formulated as a combination of the global semantic alignment loss and the adaptive local transport loss. For the m-th surrogate image encoder, the loss is defined as: m=ℒgm+λlℒlm,J_m=L_g^m+ _lL_l^m, (25) where λl _l is a hyperparameter that balances the contribution of local transport alignment. We use the same fixed number of clusters for adversarial and target patch features throughout optimization, with K=10K=10 by default. The sensitivity to K is evaluated in Fig. 4. I-G Dynamic Ensemble Model Weighting To improve adversarial transferability, adversarial examples are generated using the ensemble loss on several surrogate encoders. It is computed as follows: =∑m=1Mωmm.J= _m=1^M _mJ_m. (26) where mJ_m is the distance loss in Eq. (25). Dynamic weights are computed from alignment similarity scores, which are distinct from these distance losses. For the k-th evaluation of the ensemble objective, define Am(k) A_m^(k) =cos(^mg(c),^mg(tar)) = \! ( z_m^g(x^c), z_m^g(x_tar) ) (27) +λl∑i,j~m(i,j)m(i,j), + _l _i,j P_m(i,j)S_m(i,j), where cx^c is the current augmented input and the local similarities and transport plan are evaluated for this input. Larger Am(k)A_m^(k) indicates stronger feature alignment. The dynamic weights are calculated from the score ratios defined below. We compute the similarity-score ratio between consecutive evaluations as qm(k)=Am(k)Am(k−1)+η,q_m^(k)= A_m^(k)A_m^(k-1)+η, (28) where η=10−8η=10^-8 stabilizes the denominator. For the first evaluation, we set Am(0)=Am(1)A_m^(0)=A_m^(1). Here, k indexes successive objective evaluations, including evaluations of different augmented views within the same attack iteration. The ratio reflects score changes across evaluations, which can also arise from differences between augmented views. The weights are ωm(k)=M⋅exp(qm(k)/τw)∑n=1Mexp(qn(k)/τw), _m^(k)=M· (q_m^(k)/ _w) _n=1^M (q_n^(k)/ _w), (29) where τw=1.0 _w=1.0 and the factor M ensures that the weights sum to M, matching the scale of equal weighting. For positive scores with Am(k−1)A_m^(k-1) well above η, a larger ratio indicates a larger current score relative to the preceding evaluation and receives a larger weight. This rule adapts the ensemble to observed score changes. The complete optimization procedure is summarized in Algorithm S1 in the Appendix. I-H Transferability Analysis Prior studies relate adversarial transferability to gradient similarity and model smoothness [53], knowledge transferability [25], and ensemble transferability error [54]. We compare balanced OT with uniform local matching at a fixed feasible perturbation δ, holding the encoder, augmentation view, target features, and cluster assignments fixed. Write Sij=m(i,j)S_ij=S_m(i,j) and cij=1−Sijc_ij=1-S_ij. Assume that the pairwise similarities are differentiable at δ. Let vJ_v be a differentiable victim alignment objective, with larger values indicating stronger target alignment, and set v=∇v()g_v= _v( δ) and ij=∇Sijg_ij= _ δS_ij. For a nonnegative unit-mass plan Q held fixed during differentiation, define ():=∑i,jQijij.G(Q):= _i,jQ_ijg_ij. (30) Here, ()G(Q) is the negative gradient of the local alignment loss with the plan held fixed. The uniform reference plan is Q0,ij=1/K2Q_0,ij=1/K^2. Assumption 1 (Surrogate cost separation). The K≥2K≥ 2 clusters admit a semantic correspondence permutation π, specified independently of the transport solution. The surrogate costs distinguish these correspondences with a gap Δ>0 >0: cij≥ciπ(i)+Δ,j≠π(i).c_ij≥ c_iπ(i)+ , j≠π(i). (31) Assumption 2 (Cross-model gradient separation). There exist A>0A>0, B≥0B≥ 0, and deviations |eij|≤ξ|e_ij|≤ξ with ξ≥0ξ≥ 0 such that ⟨v,ij⟩=A+eij,j=π(i),−B+eij,j≠π(i). _v,g_ij = casesA+e_ij,&j=π(i),\\ -B+e_ij,&j≠π(i). cases (32) Define the noncorresponding mass q()=∑i∑j≠π(i)Qij,q0:=q(0)=1−1K.q(Q)= _i _j≠π(i)Q_ij, q_0:=q(Q_0)=1- 1K. (33) Theorem 1. Let H()=−∑i,jQijlogQijH(Q)=- _i,jQ_ij Q_ij, with 0log0=00 0=0. For εot≥0 _ot≥ 0, let otQ_ot minimize ∑i,jQijcij−εotH() _i,jQ_ijc_ij- _otH(Q) (34) over nonnegative balanced plans satisfying ∑jQij=1/K _jQ_ij=1/K for every i and ∑iQij=1/K _iQ_ij=1/K for every j. Under Assumption 1, q(ot)≤q¯:=min1,εotlogKΔ.q(Q_ot)≤ q:= \! \1, _ot K \. (35) If Assumption 2 also holds and εotlogKΔ+2ξA+B<1−1K, _ot K + 2ξA+B<1- 1K, (36) then ⟨v,(ot)−(0)⟩≥κ, _v,G(Q_ot)-G(Q_0) ≥κ, (37) where κ:=(A+B)(q0−q¯)−2ξ>0.κ:=(A+B)(q_0- q)-2ξ>0. (38) Under these conditions, let r=+λl(r),r∈ot,0,d_r=h+ _lG(Q_r), r∈\ot,0\, (39) where h is the common gradient contribution from the other branches and λl>0 _l>0 is the common local weight. Suppose ‖r‖2≤D\|d_r\|_2≤ D for some D>0D>0, and vJ_v has an L-Lipschitz gradient along both update segments, with L≥0L≥ 0. Choose α>0α>0 so that both updates remain feasible and, when L>0L>0, α<λlκ/(LD2)α< _lκ/(LD^2). Then v(+αot)−v(+α0) _v( δ+ _ot)-J_v( δ+ _0) (40) ≥τ, ≥τ, where τ:=αλlκ−Lα2D2>0.τ:=α _lκ-Lα^2D^2>0. (41) Thus, under the stated assumptions and step-size condition, OT produces stronger target alignment on the victim model than uniform local matching after one feasible ordinary gradient step from the same perturbation, with an objective gap of at least τ>0τ>0. This conditional advantage suggests that OT-based local alignment can improve the effectiveness of adversarial examples for targeted transfer, consistent with the empirical gains of the complete attack reported in Section IV. The proof and discussion of projected sign updates are provided in the Appendix (Section S1). IV Experiments IV-A Experimental Setup Datasets. Following prior studies [4, 24, 17], we adopt 1,000 clean source images resized to 224×224×3224× 224× 3 from the NIPS 2017 Adversarial Attacks and Defenses Competition dataset11 1 https://nips.c/Conferences/2017/CompetitionTrack. For the targeted setting, 1,000 images are randomly sampled from the MSCOCO validation set [27] as target images. Implementation Settings. Following [23, 45], we generate adversarial examples using five surrogate models: three CLIP variants, ViT-B/16 [40], ViT-B/32 [40], and ViT-G/14 (LAION-2B) [40], together with InternVL3-1B [64] and DINOv2 [38]. For IAU-FOA, unless otherwise specified, the perturbation budget is ϵ=16/255ε=16/255 under the ℓ∞ _∞ norm, with a step size of 1/2551/255 and 300 attack iterations. The number of local clusters is fixed at K=10K=10 throughout optimization. Local cluster reference masses are uniform, and the local-loss weight is λl=2.0 _l=2.0. Adaptive UOT uses ρ=0.2ρ=0.2, εot=0.1 _ot=0.1, and γ=1γ=1, with mean-normalized confidence and total-mass normalization of the transport plan. Dynamic weighting uses τw=1.0 _w=1.0 and a numerical stabilizer of 10−810^-8. VIA uses intensity scales 0.25,0.5,1,2,4\0.25,0.5,1,2,4\ (h=2h=2) and a channel-gain amplitude of sβ=0.5s_β=0.5. Our main comparison covers twelve MLLMs, comprising six open-source models (Qwen2.5-VL-7B/72B, InternVL3-14B/78B, MiniCPM-o-2.6, LLaVA-1.6-7B) and six closed-source models (Claude-Sonnet-4.6/Opus-4.8, GPT-5.4/5.5, Gemini-3.1-Flash-Lite/3.5-Flash). An additional evaluation on five recent commercial MLLMs brings the total to seventeen targets. For all evaluations, the prompt is fixed as “Describe this image, no longer than 25 words”. Experiments are conducted on an Ubuntu system equipped with an NVIDIA H100 Tensor Core GPU with 80 GB of GPU memory. Competitive Methods. We compare the proposed IAU-FOA with seven representative targeted transfer attacks for MLLMs, including AdvDiffVLM [12], AnyAttack [59], COA [50], M-Attack [24], FOA-Attack [17], M-Attack-v2 [61], and MPCAttack [23]. For baselines other than AdvDiffVLM, we use the same surrogate models, perturbation budget, and attack iterations as IAU-FOA; the remaining method-specific settings follow their original papers. AdvDiffVLM follows the settings of its original paper [12] and is not constrained by the ℓ∞ _∞ budget of 16/25516/255. All methods use the same evaluation protocol described below. Evaluation metrics. Following [24, 17], we evaluate targeted transferability with an LLM-as-a-judge protocol. For each adversarial–target pair, the target MLLM first produces captions for both images, and GPTScore is then used to quantify their semantic similarity by using GPT-4o. We regard an attack as successful when the similarity score is at least 0.5, indicating that the adversarial image is judged to share the target subject. In addition to the attack success rate (ASR), we report the average similarity score (AvgSim) to reflect overall target alignment; higher values of both metrics indicate better targeted transferability. IV-B Hyper-parameter Selection Fig. 3: Sensitivity to (a) the marginal-relaxation penalty ρ and (b) the local-loss weight λl _l, evaluated on 100 source–target image pairs per model. ASR and AvgSim are averaged over two models per family (Gemini, GPT, and Claude). Bold legend entries mark the selected settings, ρ=0.2ρ=0.2 and λl=2.0 _l=2.0; numerical labels show metric values. Fig. 4: Sensitivity to the number of local clusters K, evaluated on 100 source–target image pairs per model. Colored curves show ASR on six commercial MLLMs; the thick black curve shows their mean. The dashed line marks the default K=10K=10, which maximizes mean ASR among the tested values. All results in Figs. 3 and 4 are evaluated on 100 source–target image pairs per target model. We investigate the sensitivity of IAU-FOA to three hyperparameters: the marginal-relaxation penalty weight ρ in adaptive unbalanced OT, the local-loss weighting factor λl _l in the dual-granularity alignment objective, and the number of local clusters K. We vary one hyperparameter at a time while keeping all other settings fixed. For ρ and λl _l, we report the arithmetic means of ASR and AvgSim over two models in each family: Gemini-3.5-Flash and Gemini-3.1-Flash-Lite for Gemini, GPT-5.5 and GPT-5.4 for GPT, and Claude-Sonnet-4.6 and Claude-Opus-4.8 for Claude. For K, we report ASR on all six commercial targets in the main evaluation. As shown in Fig. 3(a), we vary ρ over 0.1,0.2,0.3,0.4\0.1,0.2,0.3,0.4\. The setting ρ=0.2ρ=0.2 yields the highest reported ASR and AvgSim for Gemini, GPT, and Claude, achieving family-mean ASRs of 85.0%85.0\%, 81.0%81.0\%, and 90.5%90.5\% for Gemini, GPT, and Claude, respectively, with corresponding mean AvgSim values of 65.8%65.8\%, 63.2%63.2\%, and 68.7%68.7\%. These family-mean ASRs agree with the corresponding two-model averages of the full-method results in Table V. A smaller value may over-relax the marginal constraints, whereas a larger value moves the transport plan closer to balanced OT and may retain unreliable local correspondences. Fig. 3(b) evaluates λl∈1.0,2.0,3.0,4.0 _l∈\1.0,2.0,3.0,4.0\. Setting λl=2.0 _l=2.0 also yields the highest reported values of both metrics for Gemini, GPT, and Claude, with the same ASR and AvgSim values reported above. These results favor an intermediate local-loss weight: too little emphasis on local alignment may limit its contribution, whereas too much may reduce the benefit of global semantic alignment. Based on these results, we use ρ=0.2ρ=0.2 and λl=2.0 _l=2.0 in all subsequent experiments. Figure 4 further evaluates K∈6,8,10,12,14,16K∈\6,8,10,12,14,16\, which controls the granularity of the clustered patch features used for local transport alignment. The highest mean ASR is 0.8550.855 at K=10K=10, compared with 0.8220.822 and 0.7930.793 at K=6K=6 and K=8K=8, respectively. Increasing K beyond 10 yields mean ASRs between 0.8280.828 and 0.8380.838, indicating that finer clustering does not consistently improve aggregate transferability. The preferred setting nevertheless varies by target: K=10K=10 achieves the highest ASR on GPT-5.5, GPT-5.4, and Cl-S-4.6 and ties for the best result on G-3.5-F, whereas G-3.1-FL and Cl-O-4.8 peak at K=14K=14 with ASRs of 0.9100.910 and 0.9400.940, respectively. Based on the best mean performance across the six targets, we use a fixed K=10K=10 throughout optimization unless otherwise specified. TABLE I: Targeted transfer to six open-source MLLMs on the same 1,000 source–target image pairs per model. ASR and AvgSim measure attack success and mean similarity, respectively; bold marks column maxima. Target MLLMs Average Qwen2.5-VL-7B InternVL3-14B MiniCPM-o-2.6 (8B) LLaVA-1.6-7B Qwen2.5-VL-72B InternVL3-78B Overall Method ASR AvgSim ASR AvgSim ASR AvgSim ASR AvgSim ASR AvgSim ASR AvgSim ASR AvgSim COA [50] 0.001 0.018 0.002 0.018 0.002 0.020 0.004 0.016 0.001 0.020 0.000 0.015 0.002 0.018 AnyAttack [59] 0.011 0.039 0.011 0.032 0.014 0.036 0.023 0.036 0.005 0.042 0.008 0.028 0.012 0.036 AdvDiffVLM [12] 0.309 0.313 0.315 0.318 0.293 0.292 0.309 0.296 0.314 0.327 0.194 0.239 0.289 0.298 M-Attack [24] 0.569 0.472 0.661 0.523 0.632 0.520 0.680 0.562 0.571 0.476 0.440 0.390 0.592 0.491 FOA [17] 0.575 0.488 0.703 0.561 0.692 0.551 0.725 0.598 0.617 0.506 0.490 0.422 0.634 0.521 M-Attack-v2 [61] 0.705 0.564 0.809 0.631 0.814 0.634 0.812 0.666 0.693 0.563 0.648 0.535 0.747 0.599 MPCAttack [23] 0.697 0.559 0.853 0.659 0.836 0.649 0.839 0.686 0.714 0.577 0.772 0.606 0.785 0.623 IAU-FOA (Ours) 0.821 0.645 0.910 0.710 0.866 0.692 0.885 0.723 0.823 0.653 0.871 0.685 0.863 0.685 TABLE I: Targeted transfer to six commercial MLLMs on the same 1,000 source–target image pairs per model. ASR and AvgSim measure attack success and mean similarity, respectively; bold marks column maxima. Target MLLMs Average Gemini-3.5-Flash Gemini-3.1-Flash-Lite GPT-5.5 GPT-5.4 Claude-Sonnet-4.6 Claude-Opus-4.8 Overall Method ASR AvgSim ASR AvgSim ASR AvgSim ASR AvgSim ASR AvgSim ASR AvgSim ASR AvgSim COA [50] 0.001 0.010 0.000 0.013 0.002 0.019 0.001 0.018 0.000 0.013 0.002 0.018 0.001 0.015 AnyAttack [59] 0.004 0.019 0.004 0.028 0.009 0.032 0.005 0.027 0.008 0.026 0.006 0.029 0.006 0.027 AdvDiffVLM [12] 0.055 0.129 0.068 0.127 0.116 0.171 0.112 0.172 0.277 0.297 0.324 0.313 0.159 0.202 M-Attack [24] 0.309 0.321 0.312 0.333 0.285 0.292 0.219 0.255 0.425 0.384 0.535 0.452 0.348 0.339 FOA [17] 0.340 0.350 0.342 0.353 0.350 0.332 0.296 0.296 0.474 0.417 0.581 0.484 0.397 0.372 M-Attack-v2 [61] 0.525 0.455 0.506 0.450 0.454 0.401 0.383 0.357 0.630 0.511 0.723 0.568 0.537 0.457 MPCAttack [23] 0.673 0.547 0.675 0.542 0.453 0.408 0.390 0.365 0.621 0.514 0.724 0.581 0.589 0.493 IAU-FOA (Ours) 0.776 0.616 0.789 0.622 0.696 0.561 0.672 0.553 0.744 0.605 0.822 0.641 0.750 0.600 IV-C Comparative Experimental Results Comparisons with different attack methods. All results in Tables I and I are evaluated on the same 1,000 source–target image pairs per target model. We compare the proposed IAU-FOA with seven targeted transfer attack baselines, including COA, AnyAttack, AdvDiffVLM, M-Attack, FOA, M-Attack-v2, and MPCAttack, across both open-source and closed-source MLLMs. As shown in Table I, IAU-FOA achieves the highest ASR and AvgSim on all six open-source targets across the Qwen2.5-VL, InternVL3, MiniCPM-o, and LLaVA model families. Its reported average ASR and AvgSim are 86.3%86.3\% and 68.5%68.5\%, respectively, compared with 78.5%78.5\% and 62.3%62.3\% for MPCAttack, the strongest baseline on average. Compared with the best-performing baseline for each target and metric, IAU-FOA improves ASR by 3.03.0–11.611.6 percentage points and AvgSim by 3.73.7–8.18.1 percentage points. The largest gains in both metrics occur on Qwen2.5-VL-7B, where IAU-FOA achieves an ASR of 82.1%82.1\% and an AvgSim of 64.5%64.5\%, compared with 70.5%70.5\% and 56.4%56.4\% for M-Attack-v2. As reported in Table I, IAU-FOA also ranks first in both metrics on all six commercial targets from the Gemini, GPT, and Claude families. Its reported average ASR and AvgSim are 75.0%75.0\% and 60.0%60.0\%, respectively, compared with 58.9%58.9\% and 49.3%49.3\% for MPCAttack. Relative to the strongest competing method for each target and metric, the improvements range from 9.89.8 to 28.228.2 percentage points in ASR and from 6.06.0 to 18.818.8 percentage points in AvgSim. The largest gains occur on GPT-5.4, where IAU-FOA improves ASR from 39.0%39.0\% to 67.2%67.2\% and AvgSim from 36.5%36.5\% to 55.3%55.3\%. These results demonstrate that the proposed method outperforms previous methods in both attack success rate and average target-semantic alignment. Evaluation on recent commercial MLLMs. In August 2026, we further evaluate IAU-FOA on five recent commercial MLLMs: OpenAI’s GPT-5.6-Luna, GPT-5.6-Sol, and GPT-5.6-Terra, Anthropic’s Claude-Opus-5, and Google’s Gemini-3.7-Flash. These targets span three providers and multiple capability and efficiency tiers. We use the same 1,000 source–target image pairs per target model as the main comparison and compare against M-Attack-v2 and MPCAttack, its two strongest baselines. As shown in Table I, IAU-FOA achieves the highest ASR and AvgSim on all five targets, averaging 66.7%66.7\% and 54.2%54.2\%, respectively. Compared with MPCAttack, the strongest baseline on average, these results represent gains of 21.521.5 percentage points in ASR and 13.413.4 points in AvgSim. Across individual targets, improvements over the stronger baseline range from 7.37.3 to 30.730.7 percentage points in ASR and from 4.04.0 to 19.419.4 points in AvgSim. The largest ASR gain occurs on GPT-5.6-Luna, increasing from 25.0%25.0\% to 55.7%55.7\%. On GPT-5.6-Sol and GPT-5.6-Terra, ASR improves by 29.129.1 and 29.729.7 percentage points over the respective strongest baselines, while the gains on Claude-Opus-5 and Gemini-3.7-Flash are 7.37.3 and 9.99.9 points. These results demonstrate improved transferability across all five targets, with the largest gains on the GPT-5.6 family. TABLE I: Targeted transfer to five recent commercial MLLMs (August 2026). ASR and AvgSim are evaluated on 1,000 source–target image pairs per model; bold marks row maxima. Target MLLM Metric M-Attack-v2 [61] MPCAttack [23] IAU-FOA (Ours) GPT-5.6-Luna ASR 0.245 0.250 0.557 AvgSim 0.275 0.277 0.471 GPT-5.6-Sol ASR 0.323 0.315 0.614 AvgSim 0.326 0.321 0.509 GPT-5.6-Terra ASR 0.296 0.299 0.596 AvgSim 0.306 0.307 0.492 Claude-Opus-5 ASR 0.669 0.683 0.756 AvgSim 0.541 0.552 0.592 Gemini-3.7-Flash ASR 0.553 0.713 0.812 AvgSim 0.480 0.582 0.647 Average ASR 0.417 0.452 0.667 AvgSim 0.385 0.408 0.542 IV-D Evaluation under Adversarial Defenses TABLE IV: Robustness against input defenses: mean ASR across six commercial MLLMs. All settings use 100 source–target image pairs per model. Def. Avg. averages the six defended settings; bold marks row maxima. Defense M-Attack-v2 [61] MPCAttack [23] IAU-FOA(Ours) No defense 0.638 0.690 0.855 JPEG75 [5] 0.557 0.640 0.803 BDR4-bit_4-bit [51] 0.508 0.605 0.778 Median3×3 [51] 0.343 0.382 0.605 TVM [10] 0.495 0.567 0.737 ComDefend [18] 0.370 0.393 0.618 DISCO [14] 0.297 0.382 0.587 Def. Avg. 0.428 0.495 0.688 We conduct comparative experiments under six input-transformation and purification defenses: JPEG compression [5], total-variation minimization (TVM) [10], bit-depth reduction (BDR) [51], median filtering [51], ComDefend [18], and DISCO [14]. JPEG compression uses a quality factor of 75, BDR reduces the color depth to 4 bits, and median filtering uses a 3×33× 3 kernel. All settings, including the undefended setting, are evaluated on 100 source–target image pairs per target model. Table IV reports the mean ASR across the six commercial MLLMs for each defense; Def. Avg. averages the six defended settings, excluding the undefended setting. IAU-FOA achieves the highest ASR in every defended setting, with an average of 0.6880.688, compared with 0.4950.495 for MPCAttack and 0.4280.428 for M-Attack-v2. These results correspond to average improvements of 19.319.3 and 26.026.0 percentage points, respectively. The gains over MPCAttack range from 16.316.3 to 22.522.5 percentage points across individual defenses. Under DISCO, IAU-FOA achieves an ASR of 0.5870.587, exceeding MPCAttack (0.3820.382) and M-Attack-v2 (0.2970.297). These results show that the advantage of the proposed method persists after both conventional preprocessing and learned purification. IV-E Ablation Study To evaluate the contribution of each component in IAU-FOA, we compare the full method with four ablated variants: w/o OT removes the local alignment branch and retains only global alignment; w/o Adaptive UOT replaces adaptive unbalanced transport with balanced OT; w/o VIA disables visual-invariance augmentation; and w/o Dynamic ensemble fixes all surrogate weights to one. All variants use the same 100 source–target image pairs per target model against six commercial MLLMs, with all remaining attack and evaluation settings unchanged. Table V reports per-model ASR and the mean ASR and AvgSim across the six targets. TABLE V: Component ablation on six commercial MLLMs using the same 100 source–target image pairs per model. G/Cl denote Gemini/Claude; F/FL/S/O denote Flash/Flash-Lite/Sonnet/Opus. w/o OT removes local alignment; w/o Adaptive UOT uses BOT; w/o Dynamic ensemble uses Equal weighting. Attack success rate per target MLLM Overall Variant G-3.5-F G-3.1-FL GPT-5.5 GPT-5.4 Cl-S-4.6 Cl-O-4.8 ASR AvgSim IAU-FOA (Ours) 0.850 0.850 0.810 0.810 0.890 0.920 0.855 0.659 w/o OT 0.680 0.730 0.650 0.550 0.760 0.780 0.692 0.543 w/o Adaptive UOT 0.840 0.800 0.780 0.750 0.850 0.850 0.812 0.635 w/o VIA 0.770 0.750 0.610 0.500 0.630 0.790 0.675 0.551 w/o Dynamic ensemble 0.850 0.800 0.770 0.730 0.830 0.820 0.800 0.646 The full method achieves the highest mean ASR and AvgSim of 0.8550.855 and 0.6590.659, respectively. Removing local OT alignment reduces these metrics by 16.316.3 and 11.611.6 percentage points, while replacing Adaptive UOT with balanced OT reduces them by 4.34.3 and 2.42.4 points, supporting both local matching and confidence-adaptive transport. Removing VIA causes the largest ASR decrease of 18.018.0 points, together with a 10.810.8-point decrease in AvgSim. Equal ensemble weighting reduces mean ASR and AvgSim by 5.55.5 and 1.31.3 points, respectively; ASR decreases on five targets and remains unchanged on G-3.5-F. Overall, all four ablations reduce average performance, with VIA contributing most to ASR and local OT alignment contributing most to AvgSim under the evaluated settings. Fig. 5: Input augmentation strategies on six commercial MLLMs, evaluated on 100 source–target image pairs per model. The radar chart shows ASR on Gemini-3.5-Flash (G-3.5-F), Gemini-3.1-Flash-Lite (G-3.1-FL), GPT-5.5, GPT-5.4, Claude-Sonnet-4.6 (Cl-S-4.6), and Claude-Opus-4.8 (Cl-O-4.8). The table reports input copies (Copy) and mean ASR and AvgSim across targets. IV-F Analysis of the Proposed Method We conduct all comparative experiments in this subsection on 100 source–target image pairs per model. Visual-Invariance Augmentation. As shown in Fig. 5, we compare the proposed visual-invariance augmentation (VIA) with No Aug and five input-diversification strategies (DI [49], SIM [26], Admix [47], SSA [32], and SIA [48]) on six commercial MLLMs, while keeping all other attack settings fixed. VIA achieves the highest ASR on all six targets and the best aggregate performance, with an average ASR of 0.8550.855 and an AvgSim of 0.65900.6590. Compared with No Aug (0.6750.675 ASR and 0.55100.5510 AvgSim), it improves the two metrics by 18.018.0 and 10.8010.80 percentage points, respectively. Admix is the strongest competing augmentation on average, with an ASR of 0.8180.818 and an AvgSim of 0.63400.6340; VIA exceeds these scores by 3.73.7 and 2.502.50 percentage points. SIM, Admix, SSA, and SIA all use five transformed copies, matching VIA, yet obtain lower average ASR and AvgSim. This comparison indicates that the gains depend on the augmentation strategy, beyond the number of augmented views. VIA varies intensity and channel-wise gains while preserving spatial content, encouraging the perturbation to remain effective under exposure, illumination-color, and white-balance variations. The improvements across the Gemini, GPT, and Claude families support the effectiveness of these photometric transformations for targeted transferability. Fig. 6: OT-based local alignment on three commercial MLLMs, evaluated on 100 source–target image pairs per model. W/O OT removes the local alignment branch; BOT and UOT use balanced and plain unbalanced OT, respectively. Adaptive-UOT is our confidence-aware formulation and achieves the highest ASR on all three targets. Adaptive Unbalanced Optimal Transport. As shown in Fig. 6, we compare Adaptive-UOT with W/O OT, BOT, and UOT while keeping all other attack settings fixed. W/O OT removes the local alignment branch, and BOT imposes balanced transport constraints; these correspond to w/o OT and w/o Adaptive UOT in Table V. UOT relaxes the marginal constraints without confidence-aware adaptation. Adaptive-UOT achieves the highest ASR on all three targets, reaching 0.850.85, 0.810.81, and 0.920.92 on G-3.5-F, GPT-5.5, and Cl-O-4.8, respectively. Its mean ASR is 0.8600.860, compared with 0.7030.703, 0.8230.823, and 0.8200.820 for W/O OT, BOT, and UOT, corresponding to gains of 15.715.7, 3.73.7, and 4.04.0 percentage points. All transport formulations outperform W/O OT on every target, supporting OT-based local alignment. Plain UOT matches BOT on G-3.5-F and GPT-5.5 but performs slightly worse on Cl-O-4.8 (0.840.84 vs. 0.850.85). Adaptive-UOT’s largest gains occur on Cl-O-4.8: 7.07.0 percentage points over BOT and 8.08.0 over UOT. These results support adapting marginal regularization to matching confidence; uniform relaxation alone yields no improvement over BOT here. These experimental results demonstrate the effectiveness of the proposed Adaptive Unbalanced Optimal Transport in improving targeted transferability. TABLE VI: ASR comparison of ensemble weighting strategies. Results use 100 source–target image pairs per model; only the weighting rule varies. G/Cl denote Gemini/Claude; F/FL/S/O denote Flash/Flash-Lite/Sonnet/Opus. Bold marks column maxima. Weighting G-3.5- F G-3.1- FL GPT- 5.5 GPT- 5.4 Cl-S- 4.6 Cl-O- 4.8 ASR Equal 0.850 0.800 0.770 0.730 0.830 0.820 0.800 Random 0.810 0.800 0.760 0.770 0.890 0.870 0.817 Score-Prop. 0.850 0.830 0.800 0.740 0.860 0.910 0.832 Dynamic (Ours) 0.850 0.850 0.810 0.810 0.890 0.920 0.855 Dynamic Ensemble Model Weighting. As shown in Table VI, we compare dynamic ensemble weighting with three alternatives while keeping all other attack settings fixed. Using the score notation in Sec. I-G, Equal Weighting sets every weight to one, and Random Weighting samples symmetric Dirichlet weights scaled to sum to M. Score-Proportional Weighting assigns ωm(k)∝max(Am(k),0) _m^(k) (A_m^(k),0) with the same normalization, reverting to equal weights when the sum of nonnegative scores is at most 10−810^-8. Thus, this control weights encoders by their current nonnegative alignment scores. Dynamic weighting instead uses the current-to-history score ratio in Eqs. (28) and (29). Dynamic weighting achieves the highest mean ASR of 0.8550.855 across the six target models, compared with 0.8000.800, 0.8170.817, and 0.8320.832 for Equal, Random, and Score-Proportional Weighting, respectively. These results correspond to gains of 5.55.5, 3.83.8, and 2.32.3 percentage points. It ranks first on all six targets, with outright best results on G-3.1-FL, GPT-5.5, GPT-5.4, and Cl-O-4.8, and ties on G-3.5-F and Cl-S-4.6. The largest gain over Score-Proportional Weighting occurs on GPT-5.4, where ASR increases from 0.7400.740 to 0.8100.810. These experimental results demonstrate the effectiveness of the proposed Dynamic Ensemble Model Weighting in improving targeted transferability. We further analyze visual fidelity and conduct experiments to examine the effect of the number of attack iterations, with results provided in the Appendix (Figs. S1 and S2). We also examine the association between source–target CLIP similarity and attack success, with results reported in the Appendix (Table S1). V Conclusion In this paper, we investigate targeted adversarial transferability across heterogeneous MLLMs and propose IAU-FOA to improve black-box attack performance through global and local feature alignment. Specifically, global alignment provides semantic guidance, while optimal transport matches clustered patch features to exploit fine-grained visual structures. To address unreliable local correspondences under balanced transport constraints, we propose adaptive unbalanced optimal transport, which adjusts marginal regularization according to matching confidence. We also provide a conditional theoretical analysis of how local transport alignment can improve targeted transfer. Furthermore, we develop a dynamic ensemble weighting strategy that adjusts each surrogate encoder’s contribution according to relative changes in its alignment similarity score. Finally, we propose a visual-invariance augmentation strategy that combines bidirectional intensity scaling and channel-wise white-balance adjustment to improve transferability under varying photometric conditions. Extensive experiments on open-source and closed-source MLLMs demonstrate that IAU-FOA outperforms the compared transferable attack methods. Ablation studies and component analyses further demonstrate the effectiveness of the proposed designs. References [1] J. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. (2022) Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems 35, p. 23716–23736. Cited by: §I-A. [2] J. Byun, S. Cho, M. Kwon, H. Kim, and C. Kim (2022) Improving the transferability of targeted adversarial examples through object-based diverse input. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 15244–15253. Cited by: §I. [3] Z. Chen, W. Wang, H. Tian, S. Ye, Z. Gao, E. Cui, W. Tong, K. Hu, J. Luo, Z. Ma, et al. (2024) How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences 67 (12), p. 220101. Cited by: §I-A. [4] Y. Dong, H. Chen, J. Chen, Z. Fang, X. Yang, Y. Zhang, Y. Tian, H. Su, and J. Zhu (2023) How robust is google’s bard to adversarial image attacks?. arXiv preprint arXiv:2309.11751. Cited by: §I, §IV-A. [5] G. K. Dziugaite, Z. Ghahramani, and D. M. Roy (2016) A study of the effect of jpg compression on adversarial images. arXiv preprint arXiv:1608.00853. Cited by: §IV-D, TABLE IV. [6] C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun, et al. (2025) Mme: a comprehensive evaluation benchmark for multimodal large language models. Advances in Neural Information Processing Systems 38. Cited by: §I. [7] S. Gao, X. Jia, X. Ren, I. Tsang, and Q. Guo (2024) Boosting transferability in vision-language attacks via diversification along the intersection region of adversarial trajectory. arXiv preprint arXiv:2403.12445. Cited by: §I. [8] I. J. Goodfellow, J. Shlens, and C. Szegedy (2015) Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Y. Bengio and Y. LeCun (Eds.), External Links: Link Cited by: §I, §I-B. [9] J. Gu, X. Jia, P. de Jorge, W. Yu, X. Liu, A. Ma, Y. Xun, A. Hu, A. Khakzar, Z. Li, et al. (2023) A survey on transferability of adversarial examples across deep neural networks. arXiv preprint arXiv:2310.17626. Cited by: §I. [10] C. Guo, M. Rana, M. Cisse, and L. Van Der Maaten (2017) Countering adversarial images using input transformations. arXiv preprint arXiv:1711.00117. Cited by: §IV-D, TABLE IV. [11] M. Guo, J. Xu, Y. Zhang, J. Song, H. Peng, Y. Deng, X. Dong, K. Nakayama, Z. Geng, C. Wang, et al. (2025) R-bench: graduate-level multi-disciplinary benchmarks for llm & mllm complex reasoning evaluation. arXiv preprint arXiv:2505.02018. Cited by: §I-A. [12] Q. Guo, S. Pang, X. Jia, Y. Liu, and Q. Guo (2025) Efficient generation of targeted and transferable adversarial examples for vision-language models via diffusion models. IEEE Transactions on Information Forensics and Security 20 (), p. 1333–1348. External Links: Document Cited by: §I-C, §IV-A, TABLE I, TABLE I. [13] K. Harada, Y. Yamazaki, M. Taniguchi, E. Marrese-Taylor, T. Kojima, Y. Iwasawa, and Y. Matsuo (2025) When instructions multiply: measuring and estimating llm capabilities of multiple instructions following. arXiv preprint arXiv:2509.21051. Cited by: §I-A. [14] C. Ho and N. Vasconcelos (2022) DISCO: adversarial defense with local implicit functions. Advances in neural information processing systems 35, p. 23818–23837. Cited by: §IV-D, TABLE IV. [15] W. Huang, B. Jia, Z. Zhai, S. Cao, Z. Ye, F. Zhao, Z. Xu, Y. Hu, and S. Lin (2025) Vision-r1: incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749. Cited by: §I. [16] B. Jain, S. Ö. Arık, and H. K. Thakur (2026) Adversarial attacks on multimodal large language models: a comprehensive survey. arXiv preprint arXiv:2603.27918. Cited by: §I-B. [17] X. Jia, S. Gao, S. Qin, T. Pang, C. Du, Y. Huang, X. Li, Y. Li, B. Li, and Y. Liu (2025) Adversarial attacks against closed-source mllms via feature optimal alignment. Advances in Neural Information Processing Systems 38, p. 57993–58028. Cited by: §I, §IV-A, §IV-A, §IV-A, TABLE I, TABLE I. [18] X. Jia, X. Wei, X. Cao, and H. Foroosh (2019) Comdefend: an efficient image compression model to defend adversarial examples. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 6077–6085. Cited by: §IV-D, TABLE IV. [19] C. Jiang, Z. Wang, M. Dong, and J. Gui (2025) Survey of adversarial robustness in multimodal large language models. arXiv preprint arXiv:2503.13962. Cited by: §I, §I-B. [20] N. Karanikolas, E. Manga, N. Samaridi, E. Tousidou, and M. Vassilakopoulos (2023) Large language models versus natural language understanding and generation. In Proceedings of the 27th Pan-Hellenic Conference on Progress in Computing and Informatics, p. 278–290. Cited by: §I-A. [21] J. Kuang, Y. Shen, J. Xie, H. Luo, Z. Xu, R. Li, Y. Li, X. Cheng, X. Lin, and Y. Han (2025) Natural language understanding and inference with mllm in visual question answering: a survey. ACM Computing Surveys 57 (8), p. 1–36. Cited by: §I. [22] J. Li, D. Li, S. Savarese, and S. Hoi (2023) Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, p. 19730–19742. Cited by: §I, §I-A. [23] Y. Li, T. Xu, C. Hu, T. Zhou, X. Wu, and J. Kittler (2026) Multi-paradigm collaborative adversarial attack against multi-modal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 30065–30075. Cited by: §I, §I-C, §IV-A, §IV-A, TABLE I, TABLE I, TABLE I, TABLE IV. [24] Z. Li, X. Zhao, D. Wu, J. Cui, and Z. Shen (2025) A frustratingly simple yet highly effective attack baseline: over 90% success rate against the strong black-box models of gpt-4.5/4o/o1. Advances in Neural Information Processing Systems 38, p. 45018–45051. Cited by: §I, §I-C, §IV-A, §IV-A, §IV-A, TABLE I, TABLE I. [25] K. Liang, J. Y. Zhang, B. Wang, Z. Yang, S. Koyejo, and B. Li (2021) Uncovering the connections between adversarial transferability and knowledge transferability. In International Conference on Machine Learning, p. 6577–6587. Cited by: §I-H. [26] J. Lin, C. Song, K. He, L. Wang, and J. E. Hopcroft (2019) Nesterov accelerated gradient and scale invariance for adversarial attacks. arXiv preprint arXiv:1908.06281. Cited by: §I, §I-B, §IV-F. [27] T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick (2014) Microsoft coco: common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, p. 740–755. Cited by: §IV-A. [28] H. Liu, C. Li, Y. Li, and Y. J. Lee (2024) Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 26296–26306. Cited by: §I-A. [29] H. Liu, C. Li, Q. Wu, and Y. J. Lee (2023) Visual instruction tuning. Advances in neural information processing systems 36, p. 34892–34916. Cited by: §I, §I-A. [30] X. Liu, Y. Zhu, Y. Lan, C. Yang, and Y. Qiao (2024) Safety of multimodal large language models on images and texts. arXiv preprint arXiv:2402.00357. Cited by: §I. [31] Y. Liu, X. Chen, C. Liu, and D. Song (2016) Delving into transferable adversarial examples and black-box attacks. arXiv preprint arXiv:1611.02770. Cited by: §I-B. [32] Y. Long, Q. Zhang, B. Zeng, L. Gao, X. Liu, J. Zhang, and J. Song (2022) Frequency domain model augmentation for adversarial attack. In European conference on computer vision, p. 549–566. Cited by: §IV-F. [33] H. Lu, W. Liu, B. Zhang, B. Wang, K. Dong, B. Liu, J. Sun, T. Ren, Z. Li, H. Yang, et al. (2024) Deepseek-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525. Cited by: §I. [34] H. Lu, Y. Yu, Y. Yang, C. Yi, X. Ke, Q. Zhang, B. Shen, A. Kot, and X. Jiang (2026) Make anything match your target: universal adversarial perturbations against closed-source mllms via multi-crop routed meta optimization. arXiv preprint arXiv:2601.23179. Cited by: §I. [35] X. Ma, Q. Liu, D. Jiang, G. Zhang, Z. Ma, and W. Chen (2025) General-reasoner: advancing llm reasoning across all domains. Advances in Neural Information Processing Systems 38, p. 56596–56618. Cited by: §I-A. [36] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu (2017) Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083. Cited by: §I-B. [37] H. Mei, Z. Wang, S. You, M. Dong, and C. Xu (2025) VEAttack: downstream-agnostic vision encoder attack against large vision language models. arXiv preprint arXiv:2505.17440. Cited by: §I. [38] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. HAZIZA, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P. Huang, S. Li, I. Misra, M. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski (2024) DINOv2: learning robust visual features without supervision. Transactions on Machine Learning Research. Cited by: §IV-A. [39] N. Papernot, P. McDaniel, and I. Goodfellow (2016) Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv:1605.07277. Cited by: §I-B. [40] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §I, §IV-A. [41] T. Rahmatullaev, P. Druzhinina, N. Kurdiukov, M. Mikhalchuk, A. Kuznetsov, and A. Razzhigaev (2025) Universal adversarial attack on aligned multimodal llms. arXiv preprint arXiv:2502.07987. Cited by: §I. [42] Y. Su, T. Lan, H. Li, J. Xu, Y. Wang, and D. Cai (2023) Pandagpt: one model to instruction-follow them all. arXiv preprint arXiv:2305.16355. Cited by: §I-A. [43] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus (2013) Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199. Cited by: §I-B. [44] C. Villani et al. (2009) Optimal transport: old and new. Vol. 338, Springer. Cited by: §I-E. [45] H. Wang, X. Ma, W. Luo, X. Jia, and J. Huang (2026) Adversarial attacks against mllms via progressive resolution processing and adaptive feature alignment. arXiv preprint arXiv:2605.09902. Cited by: §I, §I, §I, §IV-A. [46] P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, et al. (2024) Qwen2-vl: enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191. Cited by: §I, §I-A. [47] X. Wang, X. He, J. Wang, and K. He (2021) Admix: enhancing the transferability of adversarial attacks. In 2021 IEEE/CVF international conference on computer vision (ICCV), p. 16138–16147. Cited by: §IV-F. [48] X. Wang, Z. Zhang, and J. Zhang (2023) Structure invariant transformation for better adversarial transferability. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), p. 4584–4596. Cited by: §IV-F. [49] C. Xie, Z. Zhang, Y. Zhou, S. Bai, J. Wang, Z. Ren, and A. L. Yuille (2019) Improving transferability of adversarial examples with input diversity. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 2730–2739. Cited by: §I, §I-B, §IV-F. [50] P. Xie, Y. Bie, J. Mao, Y. Song, Y. Wang, H. Chen, and K. Chen (2025) Chain of attack: on the robustness of vision-language models against transfer-based adversarial attacks. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 14679–14689. Cited by: §I-C, §IV-A, TABLE I, TABLE I. [51] W. Xu, D. Evans, and Y. Qi (2017) Feature squeezing: detecting adversarial examples in deep neural networks. arXiv preprint arXiv:1704.01155. Cited by: §IV-D, TABLE IV, TABLE IV. [52] B. Yang, H. Zhang, J. Wang, Y. Ren, C. Lin, C. Shen, and Z. Zhao (2025) Use as many surrogates as you want: selective ensemble attack to unleash transferability without sacrificing resource efficiency. arXiv preprint arXiv:2505.12644. Cited by: §I. [53] Z. Yang, L. Li, X. Xu, S. Zuo, Q. Chen, P. Zhou, B. Rubinstein, C. Zhang, and B. Li (2021) Trs: transferability reduced ensemble via promoting gradient diversity and model smoothness. Advances in Neural Information Processing Systems 34, p. 17642–17655. Cited by: §I-H. [54] W. Yao, Z. Zhang, H. Tang, and Y. Liu (2024) Understanding model ensemble in transferable adversarial attack. arXiv preprint arXiv:2410.06851. Cited by: §I-H. [55] S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen (2024) A survey on multimodal large language models. National Science Review 11 (12), p. nwae403. Cited by: §I. [56] L. Yuan, Q. Mao, D. Liu, K. Wang, W. Wang, Y. Teng, J. Shao, and D. Liu (2026) Frequency-domain regularized adversarial alignment for transferable attacks against closed-source mllms. arXiv preprint arXiv:2605.21541. Cited by: §I. [57] X. Yuan, P. He, Q. Zhu, and X. Li (2019) Adversarial examples: attacks and defenses for deep learning. IEEE transactions on neural networks and learning systems 30 (9), p. 2805–2824. Cited by: §I. [58] C. Zhang, L. Zhou, X. Xu, J. Wu, and Z. Liu (2025) Adversarial attacks of vision tasks in the past 10 years: a survey. ACM Computing Surveys 58 (2), p. 1–42. Cited by: §I-B. [59] J. Zhang, J. Ye, X. Ma, Y. Li, Y. Yang, Y. Chen, J. Sang, and D. Yeung (2025) Anyattack: towards large-scale self-supervised adversarial attacks on vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 19900–19909. Cited by: §I-C, §IV-A, TABLE I, TABLE I. [60] W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, et al. (2023) A survey of large language models. arXiv preprint arXiv:2303.18223 1 (2), p. 1–124. Cited by: §I-A. [61] X. Zhao, Z. Li, Y. Luo, J. Cui, and Z. Shen (2026) Pushing the frontier of black-box lvlm attacks via fine-grained detail targeting. arXiv preprint arXiv:2602.17645. Cited by: §I-C, §IV-A, TABLE I, TABLE I, TABLE I, TABLE IV. [62] D. Zhu, j. chen, X. Shen, X. Li, and M. Elhoseiny (2024) MiniGPT-4: enhancing vision-language understanding with advanced large language models. In International Conference on Learning Representations, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024, p. 18378–18394. External Links: Link Cited by: §I. [63] H. Zhu, X. Sui, Y. Ren, Y. Jia, and L. Zhang (2023) Boosting transferability of targeted adversarial examples with non-robust feature alignment. Expert Systems with Applications 227, p. 120248. Cited by: §I. [64] J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, H. Tian, Y. Duan, W. Su, J. Shao, et al. (2025) Internvl3: exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479. Cited by: §IV-A. Visual-Invariance-Augmented Feature Optimal Alignment for Transferable Adversarial Attacks against Closed-Source MLLMs Appendix Xiaojun Jia, Simeng Qin, Yiming Li, Jie Liao, Sensen Gao, Ke Ma, Yang Liu, and Xiaochun Cao S1 Proofs and Details of the Transferability Analysis This section proves Theorem 1 and discusses projected sign updates. Throughout the local comparison, the encoder, augmentation view, target features, cluster assignments, and transport weights are held fixed. In particular, the frozen-plan local loss satisfies −∇ℒl(;)=−∇∑i,jQij(1−Sij)=∑i,jQijij=().- _ δL_l( δ;Q)=- _ δ _i,jQ_ij(1-S_ij)= _i,jQ_ijg_ij=G(Q). (S1) Proof of Theorem 1 Proof. Let C()=∑i,jQijcijC(Q)= _i,jQ_ijc_ij, ai=bj=1/Ka_i=b_j=1/K, and Qij†=K−1j=π(i)Q _ij=K^-11\j=π(i)\. The feasible plan †Q and Assumption 1 give C(ot)−C(†)=∑i,jQot,ij(cij−ciπ(i))≥Δq(ot).C(Q_ot)-C(Q )= _i,jQ_ot,ij(c_ij-c_iπ(i))≥ q(Q_ot). (S2) Optimality of otQ_ot gives C(ot)−C(†)≤εot[H(ot)−H(†)]≤εotlogK,C(Q_ot)-C(Q )≤ _ot [H(Q_ot)-H(Q ) ]≤ _ot K, (S3) where H(†)=logKH(Q )= K and H(ot)≤2logKH(Q_ot)≤ 2 K. Hence Eq. (35) holds. Let E()=∑i,jQijeijE(Q)= _i,jQ_ije_ij, so |E()|≤ξ|E(Q)|≤ξ for unit-mass plans. Assumption 2 yields ⟨v,()⟩=A[1−q()]−Bq()+E()=A−(A+B)q()+E(). _v,G(Q) =A [1-q(Q) ]-Bq(Q)+E(Q)=A-(A+B)q(Q)+E(Q). (S4) Subtracting the two plan contributions gives ⟨v,(ot)−(0)⟩ _v,G(Q_ot)-G(Q_0) =(A+B)[q0−q(ot)]+E(ot)−E(0) =(A+B) [q_0-q(Q_ot) ]+E(Q_ot)-E(Q_0) (S5) ≥(A+B)(q0−q¯)−2ξ=κ. ≥(A+B)(q_0- q)-2ξ=κ. The condition in Eq. (36) gives q¯<q0−2ξ/(A+B) q<q_0-2ξ/(A+B), hence κ>0κ>0. Since h cancels, ⟨v,ot−0⟩≥λlκ _v,d_ot-d_0 ≥ _lκ. For r∈ot,0r∈\ot,0\, L-smoothness gives |v(+αr)−v()−α⟨v,r⟩|≤Lα22‖r‖22. |J_v( δ+ _r)-J_v( δ)-α _v,d_r |≤ Lα^22\|d_r\|_2^2. (S6) Subtracting the uniform-step upper bound from the OT-step lower bound yields v(+αot)−v(+α0) _v( δ+ _ot)-J_v( δ+ _0) ≥αλlκ−Lα22(‖ot‖22+‖0‖22) ≥α _lκ- Lα^22 (\|d_ot\|_2^2+\|d_0\|_2^2 ) (S7) ≥αλlκ−Lα2D2=τ. ≥α _lκ-Lα^2D^2=τ. The step-size condition ensures τ>0τ>0 for L>0L>0; for L=0L=0, τ=αλlκ>0τ=α _lκ>0. ∎ Finite Solver Accuracy For a balanced numerical plan with exact prescribed marginals, suppose its regularized objective is at most that of †Q plus ζ≥0ζ≥ 0. The same argument gives q()≤min1,εotlogK+ζΔ.q(Q)≤ \! \1, _ot K+ζ \. (S8) For this inexact plan, the corresponding sufficient condition becomes (εotlogK+ζ)/Δ+2ξ/(A+B)<q0( _ot K+ζ)/ +2ξ/(A+B)<q_0. Thus, the optimization error must also fit within the available comparison margin. The scaling-vector stopping threshold in Section I-F is not itself a bound on ζ. Relation to the Implemented Update For a raw plan P with total mass s=∑i,jPij>0s= _i,jP_ij>0, let =/sQ=P/s. The implementation uses ~=/(s+η)=θ P=P/(s+η)= , where θ=s/(s+η)>0θ=s/(s+η)>0. Thus, with the local-loss weight λl _l, its local ascent contribution is λlθ() _l (Q), corresponding to an effective local weight λlθ _lθ for a fixed plan. If two compared plans have different factors θ1 _1 and θ0 _0, this difference must be retained. For ‖ij‖2≤Gmax\|g_ij\|_2≤ G_ , and a unit-mass directional advantage bounded below by κ, one has ⟨v,θ1(1)−θ0(0)⟩≥θ1κ−|θ1−θ0|‖v‖2Gmax. _v, _1G(Q_1)- _0G(Q_0) ≥ _1κ-| _1- _0|\,\|g_v\|_2G_ . (S9) This follows by adding and subtracting θ1(0) _1G(Q_0). For directions r=+λlθr(r)d_r=h+ _l _rG(Q_r), the finite-step comparison uses λl _l times this corrected first-order bound and bounds on the resulting directions. Holding the remaining branch contributions and ensemble weights fixed isolates the effect of the local matching rule. The same decomposition can be summed over encoders and views with fixed nonnegative weights; it does not compare complete trajectories with changing weights or cluster assignments. Algorithm S1 uses projected sign updates. Let D denote the feasible perturbation box, including the image-value constraints, and define the actual displacement r=Π(+αsign(r))−.s_r= _D\! ( δ+ (d_r) )- δ. (S10) If vJ_v is L-smooth along these displacement segments, Taylor’s inequality gives v(+1)−v(+0)≥⟨v,1−0⟩−L2(‖1‖22+‖0‖22).J_v( δ+s_1)-J_v( δ+s_0)≥ _v,s_1-s_0 - L2 (\|s_1\|_2^2+\|s_0\|_2^2 ). (S11) To connect this displacement comparison to the preceding direction bound, let ν be a lower bound on ⟨v,1−0⟩ _v,d_1-d_0 . For the ordinary directions of Theorem 1, ν=λlκν= _lκ; with the implemented normalization, use ν=λl[θ1κ−|θ1−θ0|‖v‖2Gmax]ν= _l[ _1κ-| _1- _0|\,\|g_v\|_2G_ ] from Eq. (S9) instead. Define r:=r−αrb_r:=s_r- _r, the change in displacement caused by taking signs and projecting. Then ⟨v,1−0⟩ _v,s_1-s_0 =α⟨v,1−0⟩+⟨v,1−0⟩ =α _v,d_1-d_0 + _v,b_1-b_0 (S12) ≥αν−‖v‖2‖1−0‖2. ≥αν-\|g_v\|_2\|b_1-b_0\|_2. Combining Eqs. (S11) and (S12) therefore gives v(+1)−v(+0) _v( δ+s_1)-J_v( δ+s_0) (S13) ≥αν−‖v‖2‖1−0‖2−L2(‖1‖22+‖0‖22). ≥αν-\|g_v\|_2\|b_1-b_0\|_2- L2 (\|s_1\|_2^2+\|s_0\|_2^2 ). Thus, a positive raw-direction margin transfers to a strict advantage for the implemented update whenever the displacement-change term and the curvature term together are smaller than αναν. This is an additional sufficient condition, not a consequence of the transport mass bound. In particular, making α small does not automatically remove the effect of taking signs, since that displacement change can also be of order α. The equations identify the extra quantities needed to pass from ordinary gradients to projected sign steps, without assuming that the ordering is preserved. Theorem 1 explains a local mechanism under explicit assumptions; it does not prove unconditional improvement of the full attack or discrete ASR. Whether these assumptions hold for a particular closed-source victim is not established by the proof. The reported experiments evaluate the complete method. A Constructive Example of Possible Improvement The sufficient conditions are nonempty, and improvement is possible even with sign updates. Consider a scalar perturbation t∈[−0.01,0.01]t∈[-0.01,0.01], two surrogate clusters, and target features 1,2∈ℝ3e_1,e_2 ^3. Define unit surrogate features 1(t) _1(t) =(a(t),b(t),1−a(t)2−b(t)2), =(a(t),b(t), 1-a(t)^2-b(t)^2), (S14) 2(t) _2(t) =(b(t),a(t),1−a(t)2−b(t)2),a(t)=0.9+t,b(t)=0.1−2t. =(b(t),a(t), 1-a(t)^2-b(t)^2), a(t)=0.9+t, b(t)=0.1-2t. These features are smooth throughout the feasible interval. Their cosine similarities to the targets are S11=S22=a(t)S_11=S_22=a(t) and S12=S21=b(t)S_12=S_21=b(t). Let the victim feature be v(t)=(t,1−t2)z_v(t)=(t, 1-t^2) with target (1,0)(1,0), giving the distinct victim alignment objective v(t)=tJ_v(t)=t. At t=0t=0, the identity correspondence has cost gap Δ=0.8 =0.8; its pairwise derivatives are 11, whereas the off-diagonal derivatives are −2-2. Thus, Eq. (32) holds with A=1A=1, B=2B=2, and ξ=0ξ=0. For balanced entropic OT with εot=0.1 _ot=0.1, symmetry and the uniform marginals give the exact off-diagonal mass q(ot)=11+exp(8),G(ot)=1−3q(ot)>0,G(0)=−12.q(Q_ot)= 11+ (8), G(Q_ot)=1-3q(Q_ot)>0, G(Q_0)=- 12. (S15) With no other branch, the same positive local weight, and step size α=0.01α=0.01, the projected sign steps from t=0t=0 yield tot=0.01t_ot=0.01 and t0=−0.01t_0=-0.01. Hence v(tot)>v(0)>v(t0)J_v(t_ot)>J_v(0)>J_v(t_0). This explicit cosine-feature construction establishes that OT can improve a victim objective relative to uniform matching, including under a sign update. It is an existence example, not a model of the evaluated MLLMs or evidence that their gradients satisfy the assumptions. S2 Visualization of Adversarial Examples Figure S1 compares the adversarial images and perturbations generated by COA, AdvDiffVLM, AnyAttack, M-Attack-v2, MPCAttack, and IAU-FOA on three source–target pairs. The left panel shows the clean sources and adversarial images, while the right panel displays the targets and corresponding perturbations. IAU-FOA preserves the main structures and colors of the source images and achieves the highest PSNR and SSIM in all three displayed examples. From top to bottom, its PSNR values are 29.829.8, 30.330.3, and 32.532.5 dB, with corresponding SSIM values of 0.8080.808, 0.7700.770, and 0.7290.729. The average PSNR and SSIM are 30.930.9 dB and 0.7690.769, respectively, compared with 30.130.1 dB and 0.7020.702 for COA, the strongest baseline on both average metrics. These comparisons illustrate the visual fidelity of IAU-FOA on the displayed examples; aggregate image quality over the full evaluation set is not assessed here. Fig. S1: Adversarial examples and perturbations for three source–target pairs. Each row shows one source–target pair. Left: clean source images and adversarial images generated by COA, AdvDiffVLM, AnyAttack, M-Attack-v2, MPCAttack, and IAU-FOA. Right: target images and the corresponding perturbation visualizations. PSNR (dB) and SSIM below each adversarial image are measured against its clean source; higher values indicate better visual fidelity. S3 Effect of the Number of Attack Iterations All results in Fig. S2 are evaluated on 100 source–target image pairs per target model. Figure S2 evaluates the effect of the optimization budget by varying the number of attack iterations over 100,200,300,400,500\100,200,300,400,500\ while keeping all other settings fixed. Increasing the budget from 100 to 300 iterations improves ASR on all six commercial MLLMs and raises the mean ASR from 69.5%69.5\% to 85.5%85.5\%, a gain of 16.016.0 percentage points. Among the tested budgets, 300 iterations achieve the highest mean ASR. Increasing the budget to 400 iterations lowers ASR on every target and reduces the mean to 82.5%82.5\%. At 500 iterations, the mean ASR is 84.5%84.5\%, still 1.01.0 percentage point below the result at 300 iterations. Relative to 300 iterations, 500 iterations improve ASR on G-3.5-F and GPT-5.4, leave it unchanged on Cl-S-4.6, and reduce it on the other three targets. Thus, additional iterations beyond 300 do not consistently improve targeted transferability. Considering the highest observed mean ASR and the lower computational cost relative to larger budgets, we retain 300 iterations as the default setting unless otherwise specified. Fig. S2: Effect of attack iterations on targeted transferability. ASR on six commercial MLLMs for 100–500 iterations, evaluated on 100 source–target image pairs per model. The highest mean ASR (85.5%85.5\%) occurs at 300 iterations. S4 Optimization Algorithm Algorithm S1 summarizes the alignment and perturbation updates for supplied source transformations and target views. Spatial view preparation is represented by the input view schedule; visual-invariance augmentation, adaptive transport, and dynamic weighting are shown explicitly. We use η=10−8η=10^-8 and τw=1.0 _w=1.0; transport plans, confidence modulation, and ensemble weights are held fixed during backpropagation. Algorithm S1 IAU-FOA Optimization 0: source image sx_s, target image tarx_tar, surrogate encoder ensemble ℱ=fmm=1MF=\f_m\_m=1^M, perturbation budget ϵε, iterations T, step size α, local-loss weight λl _l, marginal KL weight ρ, fixed number of local clusters K (default: 1010), prepared view schedule (t)t=0T−1\V^(t)\_t=0^T-1. 0: adversarial image advx_adv. 1: Initialize (0)= δ^(0)=0, adv(0)=s+(0)x_adv^(0)=x_s+ δ^(0), and evaluation index k=1k=1. 2: Set madv=mt=/Ka_m^adv=a_m^t=1/K, εot=0.1 _ot=0.1, and ϕ=ρ/(ρ+εot)φ=ρ/(ρ+ _ot). 3: for t=0t=0 to T−1T-1 do 4: Obtain prepared view pairs (,tarc)∈(t)(T,x_tar^c) ^(t), with VIA parameters (s,)(s, β) in T. 5: Set accumulated gradient =g=0 and valid-copy count n=0n=0. 6: for each (,tarc)∈(t)(T,x_tar^c) ^(t) do 7: c=(adv(t))x^c=T(x_adv^(t)); obtain global target features and mclu=KMeans(mp(tarc),K)Y_m^clu=KMeans(Z_m^p(x_tar^c),K) without gradient tracking. 8: for m=1m=1 to M do 9: ℒgm=1−cos(^mg(c),^mg(tarc))L_g^m=1- ( z_m^g(x^c), z_m^g(x_tar^c) ). 10: mclu=KMeans(mp(c),K)X_m^clu=KMeans(Z_m^p(x^c),K). 11: m(i,j)=cos(m,iclu,m,jclu)S_m(i,j)= (x_m,i^clu,y_m,j^clu ). 12: m(i,j)=exp(−(1−m(i,j))/εot)K_m(i,j)= (-(1-S_m(i,j))/ _ot ). 13: ri=max(maxjm(i,j),0)r_i= ( _jS_m(i,j),0), r^i=ri/(1K∑jrj+η) r_i=r_i/ ( 1K _jr_j+η ), ϕia=ρr^i/(ρr^i+εot) _i^a=ρ r_i/(ρ r_i+ _ot) with γ=1γ=1. 14: Initialize ==u=v=1; update v then u by Eq. (22), for at most 100 rounds or until the mean change in v is below 10−210^-2. 15: m∗=diag()mdiag()P_m^*=diag(u)K_mdiag(v). 16: ~m=stopgrad(m∗/(∑i,jm∗(i,j)+η)) P_m=stopgrad\! (P_m^*/( _i,jP_m^*(i,j)+η) ). 17: ℒlm=∑i,j~m(i,j)(1−m(i,j))L_l^m= _i,j P_m(i,j)(1-S_m(i,j)). 18: mc=ℒgm+λlℒlmJ_m^c=L_g^m+ _lL_l^m; Am(k)=1−ℒgm+λl∑i,j~m(i,j)m(i,j)A_m^(k)=1-L_g^m+ _l _i,j P_m(i,j)S_m(i,j). 19: end for 20: If k=1k=1, set Am(0)=stopgrad(Am(1))A_m^(0)=stopgrad(A_m^(1)) for all m. 21: qm(k)=stopgrad(Am(k))/(Am(k−1)+η)q_m^(k)=stopgrad(A_m^(k))/(A_m^(k-1)+η), m=1,…,Mm=1,…,M. 22: ωm(k)=M⋅exp(qm(k)/τw)/∑j=1Mexp(qj(k)/τw) _m^(k)=M· (q_m^(k)/ _w)/ _j=1^M (q_j^(k)/ _w), m=1,…,Mm=1,…,M. 23: c=∑m=1Mωm(k)mcJ^c= _m=1^M _m^(k)J_m^c; store Am(k)←stopgrad(Am(k))A_m^(k) (A_m^(k)) for all m; k←k+1k← k+1. 24: ←+∇cg + _ δJ^c; n←n+1n← n+1. 25: end for 26: ←/max(n,1)g / (n,1). 27: (t+1)=Π[−ϵ,ϵ]((t)−α⋅sign()) δ^(t+1)= _[-ε,ε] ( δ^(t)-α·sign(g) ). 28: adv(t+1)=s+(t+1)x_adv^(t+1)=x_s+ δ^(t+1). 29: end for 30: return adv(T)x_adv^(T). S5 Source–Target Similarity by Attack Outcome To investigate whether the semantic proximity of a source–target image pair is associated with targeted attack success, we partition the IAU-FOA adversarial examples for each target MLLM into successful and failed subsets and compute the mean CLIP similarity between the corresponding clean source and target images. As reported in Table S1, successful attacks exhibit higher source–target similarity than failed attacks on all 12 target models. The table also reports the corresponding ASR. Within each model category, the Avg column aggregates successful and failed group means separately, weighted by their respective sample counts, and averages ASR across models. The differences in similarity are small and are interpreted as descriptive associations between source–target similarity and attack outcome. Because the analysis groups samples by attack outcome after evaluation, it does not establish that increasing source–target similarity causes higher ASR. TABLE S1: Source–target CLIP similarity by attack outcome. Mean similarity for successful and failed IAU-FOA attacks, with the corresponding ASR. Avg reports sample-count-weighted means for the successful and failed groups, and the arithmetic mean of ASR across models. Values are reported to three decimal places. Q, I, M, L, G, and Cl denote Qwen2.5-VL, InternVL3, MiniCPM-o, LLaVA-1.6, Gemini, and Claude; F, FL, S, and O denote Flash, Flash-Lite, Sonnet, and Opus, respectively. Open-source Q-7B Q-72B I-14B I-78B M-2.6 L-7B Avg Successful 0.476 0.475 0.474 0.474 0.474 0.474 0.474 Failed 0.462 0.465 0.467 0.469 0.469 0.469 0.466 ASR 0.821 0.823 0.910 0.871 0.866 0.885 0.863 Closed-source G-3.5-F G-3.1-FL GPT-5.5 GPT-5.4 Cl-S-4.6 Cl-O-4.8 Avg Successful 0.474 0.475 0.478 0.475 0.476 0.474 0.475 Failed 0.469 0.465 0.463 0.469 0.464 0.470 0.466 ASR 0.776 0.789 0.696 0.672 0.744 0.822 0.750 S6 Evaluation Prompt For reproducibility, Fig. S3 presents the complete prompt used by GPT-4o to compute GPTScore. For each adversarial–target pair, the two placeholders are replaced with the descriptions generated by the victim MLLM for the adversarial image and the corresponding target image. The judge is instructed to focus on subject-level semantic consistency, tolerate minor differences in wording and detail, and return only a scalar score in [0,1][0,1]. We use this scalar directly when computing AvgSim and classify an attack as successful when the score is at least 0.50.5. The same prompt and decision rule are applied to all attack methods and victim models. Fig. S3: GPT-4o evaluation prompt for GPTScore. Placeholders contain the victim MLLM’s descriptions of the adversarial and target images.