Paper deep dive
On Adversarial Vulnerability of Vision-Language Models through the Lens of Intermediate Spectral Subspaces
Chethan Krishnamurthy Ramanaik, Tobias Callies, Michael Hecht, Eirini Ntoutsi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/18/2026, 8:24:32 AM
Summary
The paper investigates the adversarial vulnerability of vision-language models (VLMs) by analyzing the spectral structure of intermediate linear transformations. It identifies that adversarial perturbations align intermediate representations with bottom singular-vector subspaces, which attenuate information. The authors propose the Spectral Subspace Guided Representation Attack (SSGRA), a white-box attack that exploits this alignment to improve attack effectiveness compared to existing baselines.
Entities (11)
Relation Signals (9)
SSGRA → evaluatedon → ImageNet
confidence 95% · Experiments are conducted on ImageNet
SSGRA → targets → VLM
confidence 95% · We propose a white-box spectral-subspace-guided attack (SSGRA) ... investigate transformer-based vision-language models
SSGRA → exploits → Bottom Singular-Vector Subspace
confidence 92% · SSGRA ... aligns intermediate representations with the subspace spanned by the bottom right singular vectors.
SSGRA → evaluatedon → Qwen2.5-VL
confidence 90% · We evaluate attacks on different VLMs, namely ... Qwen2.5-VL (7B)
SSGRA → evaluatedon → Gemma 3
confidence 90% · We evaluate attacks on different VLMs, namely Gemma-3 (4B)
SSGRA → evaluatedon → LLaVA-1.5
confidence 90% · We evaluate attacks on different VLMs, namely ... and LLaVA-1.5 (7B)
SSGRA → measuredby → BERTScore
confidence 90% · We evaluate attack effectiveness ... using BERTScore
SSGRA → measuredby → ROUGE-L
confidence 90% · ROUGE-L measures lexical similarity ... We evaluate attack effectiveness ... using ... ROUGE-L
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Adversarial vulnerability in deep neural networks (DNNs) has been studied from the perspectives of decision-boundary geometry, feature robustness, input-output Jacobians, and the instability of inverse problems. Here, we focus on the spectral structure of intermediate linear transformations that propagate information through modern DNNs, an unexplored mechanism of adversarial vulnerability. Specifically, we investigate transformer-based vision-language models, whose linear layers admit interpretable spectral decompositions and whose widespread adoption makes understanding their robustness increasingly important. We propose a white-box spectral-subspace-guided attack (SSGRA) that aligns intermediate representations with the subspace spanned by the bottom right singular vectors. Our experiments show improved attack effectiveness over existing baselines. In addition, SSGRA offers a spectral interpretation of adversarial vulnerability in VLMs, providing insights for improving their robustness.
Tags
Links
- Source: https://arxiv.org/abs/2607.07375v1
- Canonical: https://arxiv.org/abs/2607.07375v1
Trouble viewing inline? Open PDF directly →
Full Text
82,600 characters extracted from source content.
Expand or collapse full text
11institutetext: University of the Bundeswehr Munich, Germany 11email: chethan.krishnamurthy,tobias.callies,eirini.ntoutsi@unibw.de 22institutetext: University of Wrocław, Poland 22email: michael.hecht@math.uni.wroc.pl On Adversarial Vulnerability of Vision-Language Models through the Lens of Intermediate Spectral Subspaces Chethan Krishnamurthy Ramanaik Tobias Callies Michael Hecht Eirini Ntoutsi Abstract Adversarial vulnerability in deep neural networks (DNNs) has been studied from the perspectives of decision-boundary geometry, feature robustness, input–output Jacobians, and the instability of inverse problems. Here, we focus on the spectral structure of intermediate linear transformations that propagate information through modern DNNs, an unexplored mechanism of adversarial vulnerability. Specifically, we investigate transformer-based vision–language models, whose linear layers admit interpretable spectral decompositions and whose widespread adoption makes understanding their robustness increasingly important. We propose a white-box spectral-subspace-guided attack (SSGRA) that aligns intermediate representations with the subspace spanned by the bottom right singular vectors. Our experiments show improved attack effectiveness over existing baselines. In addition, SSGRA offers a spectral interpretation of adversarial vulnerability in VLMs, providing insights for improving their robustness. 1 Introduction Deep neural networks, including modern vision–language models (VLMs), are known to be vulnerable to adversarial perturbations that substantially alter model predictions while remaining visually imperceptible [szegedy2013intriguing, goodfellow2014explaining, yin2023vlattack]. Despite extensive research, the mechanisms governing how such perturbations propagate through deep networks remain incompletely understood. Existing explanations primarily analyze adversarial vulnerability from input-space [fawzi2018analysis, khoury2018geometry, Goodfellow-et-al-2016, song2017pixeldefend, samangouei2018defense, gilmer1801adversarial, zhang2019theoretically] or end-to-end perspectives, including decision-boundary geometry [moosavi2017universal], robust and non-robust features [ilyas2019adversarial], Jacobian analysis [hoffman2019robust, jakubovitz2018improving, paniagua2025adversarial, khrulkov2018art], inverse problem instability and Lipschitz properties [antun2021deep, antun2020instabilities, gottschling2025troublesome]. Despite these advances, existing theories predominantly explain adversarial vulnerability from the input space or through end-to-end network properties, leaving the spectral behavior of intermediate linear transformations largely unexplored. Transformer-based VLMs provide a natural setting for such an analysis. Their architectures comprise numerous learnable intermediate linear transformations including the projection matrices in self-attention, feed-forward networks, and multimodal fusion modules [vaswani2017attention, dosovitskiy2020image, gemmateam2025gemma3technicalreport, bai2025qwen25vltechnicalreport], making spectral decomposition a principled tool for studying representations. Because singular values govern how different representation directions (singular vectors) are amplified or suppressed by each linear transformation, they naturally provide a lens for studying how adversarial signals propagate through transformer layers. Moreover, their widespread adoption makes them an important testbed for adversarial robustness. This motivates us to investigate adversarial vulnerability through the singular-vector basis of intermediate linear transformations. Inspired by the instability of ill-posed inverse problems, where near-null singular directions govern information loss, we study how adversarial intermediate representations align with top and bottom singular-vector subspaces during attack optimization. Guided by this perspective, we formulate a spectral-guidance principle and instantiate it through a white-box attack that serves to empirically validate the proposed mechanism. Our results suggest that, beyond constraining large singular values, explicitly controlling near-null singular directions may offer a complementary strategy for improving adversarial robustness. Figure 1: Overview of the proposed spectral framework. Contributions. We identify bottom singular-vector subspaces of intermediate linear transformations as a previously overlooked spectral attack surface in transformer-based VLMs. We show that untargeted adversarial optimization naturally tends to increase the alignment of intermediate representations with these information-attenuating subspaces, even without explicit enforcement. Building on this insight, we propose the Spectral Subspace Guided Representation Attack (SSGRA), a spectrally guided white-box attack that demonstrates improved attack effectiveness compared with representative feature-space and output-space attacks on three state-of-the-art VLMs. In the subsequent sections, the paper reviews the related work, followed by the preliminaries, methodology, and experimental evaluation. 2 Related Work We briefly review the theoretical foundations of adversarial vulnerability, followed by relevant representative methods for crafting adversarial examples. 2.1 Theoretical Perspectives. Manifolds & Decision Boundaries. Adversarial vulnerability has been attributed to the geometry of high-dimensional decision boundaries [fawzi2018analysis, khoury2018geometry]. Adversarial examples have also been explained as off-manifold inputs [Goodfellow-et-al-2016, song2017pixeldefend, samangouei2018defense, gilmer1801adversarial, zhang2019theoretically], while universal perturbations suggest that decision boundaries around different inputs share a low-dimensional subspace of normal vectors [moosavi2017universal]. Non-robust features. Standard training encourages models to rely on highly predictive but non-robust features, whose sensitivity to small input perturbations leads to adversarial vulnerability [ilyas2019adversarial]. Linearity approximation. Adversarial vulnerability has been attributed to local linearity, where high-dimensional gradient–perturbation interactions amplify small perturbations [goodfellow2014explaining]. In linear settings, vulnerability arises when the input lies close to the decision boundary [etmann2019connection, hein2017formal]. However, experimental evidence suggests that DNNs are only locally linear and remain globally nonlinear [luo2015foveation]. Internal weights. Large singular values of weight matrices have been linked to adversarial vulnerability through their connection to local Lipschitz constants, motivating spectral regularization [szegedy2013intriguing, cisse2017parseval, yoshida2017spectral]. Near-zero singular values suppress gradient flow, while restoring these gradients strengthens adversarial attacks [ramanaik2025grill, gupta2022improved]. End-to-end Jacobians. Adversarial vulnerability has been linked to large input–output Jacobians [hoffman2019robust, jakubovitz2018improving]. Targeted perturbations can be expressed as linear combinations of the right singular vectors of the logit-to-image Jacobian [paniagua2025adversarial], while intermediate layer Jacobians identify sensitive input directions [khrulkov2018art]. Noisy and poorly aligned input Jacobians have also been associated with adversarial vulnerability [chan2019jacobian]. Instability of inverse problems. Studies on the instability of inverse problems show that information lost along null-space and near-null singular directions leads to unstable reconstruction during inversion [antun2021deep, antun2020instabilities, gottschling2025troublesome, ramanaik2024ensuring]. However, most adversarial robustness research has focused on constraining large singular values through spectral normalization and Lipschitz regularization [barrett2022certifiably, fazlyab2019efficient, yoshida2017spectral, miyato2018spectral, cisse2017parseval, gulrajani2017improved, virmaux2018lipschitz], as well as on the implicit self-regularization of dominant singular values [martin2021implicit]. Comparatively little attention has been paid to whether bottom singular-vector subspaces of intermediate linear transformations constitute a source of adversarial vulnerability, which is the focus of this work. 2.2 Methods for Generating Adversarial Examples Adversarial attacks have evolved from early gradient-based methods such as FGSM, PGD, and optimization-based attacks [szegedy2013intriguing, goodfellow2014explaining, mkadry2017towards, carlini2017towards] to feature-alignment attacks [sabour2015adversarial, inkawhich2019feature] and attacks on generative models [gondim2018adversarial, cemgil2020adversarially, willetts2019improving, kuzina2021diagnosing, ramanaik2025grill]. Recent VLM attacks have explored computational availability [gao2024inducing], cross-prompt transferability [luo2024image], CoT reasoning [wang2024stop], visual grounding [gao2024adversarial], gray-box SVD-based attacks [liu2026attacking], black-box attacks [dong2023robust], and behavior hijacking [bailey2023image] targeting different threat models, tasks, or objectives including targeted attacks, visual reasoning, gray-box or black-box settings, and behavior control. We focus on untargeted white-box attacks to study adversarial vulnerability through information attenuation rather than predefined attack objectives. Accordingly, we compare against representative untargeted white-box feature- and output-level attacks. Feature Discrepancy Attack (FDA) perturbs inputs by maximizing discrepancies between intermediate representations [ganeshan2019fda]. Similarly, Self-Supervised Perturbation Attack (SSPA) maximizes the discrepancy between clean and adversarial feature representations in pretrained models [naseer2020self, jia2025adversarial]. Dispersion Reduction Attack (DRA) minimizes the variance of intermediate features, forcing representations to collapse and become less discriminative [lu2020enhancing]. Blockwise Similarity Attack (BSA) targets transformer-based VLMs by maximizing cosine discrepancies between block-wise intermediate representations of clean and adversarial inputs, thereby disrupting semantic alignment throughout the network [yin2023vlattack]. Beyond feature-space objectives, output-level attacks maximize the task loss using cross-entropy (CE) or negative log-likelihood [cui2024robustness]. Entropy-Guided Attacks (EGA) maximize output entropy to induce uncertain model responses [he2025few]. We compare against representative feature-space attacks (FDA, SSPA, DRA, and BSA) and output-level attacks (CE, EGA). 3 Preliminaries Notation. Let x∈ℝdxx ^d_x denote a flattened input image. We consider perturbations constrained to the LpL_p-ball Bcp(x)=xa∈ℝdx|‖xa−x‖p≤c,B_c^p(x)= \x_a ^d_x\; |\;\|x_a-x\|_p≤ c \, where c is the perturbation budget. A VLM can be abstractly described by a function ℱ:ℝdx×→ζ,F:R^d_x×P→ζ, where P is the space of input prompts and ζ is the associated tokenizer dictionary. VLM pipeline Modern VLMs typically consist of a visual encoder, a multimodal projection or fusion module, and a large language model (LLM). Visual Encoder. The visual encoder ϕ:ℝdx→ℝNv×dvφ:R^d_x ^N_v× d_v consists of K sequentially applied blocks ϕk _k, each producing a visual token representation hkv∈ℝNv×dvh^v_k ^N_v× d_v consisting of NvN_v visual tokens of the visual embedding dimensionality dvd_v for a given input image x. Fusion Module. The final visual embedding hKv(x)=ϕ(x)h_K^v(x)=φ(x) is projected into the embedding space of the language model through a projection module P, outputting H0v=P(ϕ(x))∈ℝNv×dH^v_0=P(φ(x)) ^N_v× d, where d is the hidden dimension of the language model. For a given textual prompt, let H0t∈ℝNt×dH_0^t ^N_t× d denote the corresponding textual embedding, using NtN_t tokens. The concatenated visual and textual embeddings then form the input sequence to the language model, H0=[H0v;H0t]∈ℝN×dH_0= [\,H_0^v;\;H_0^t\, ] ^N× d, where N=Nv+NtN=N_v+N_t is the total sequence length after multimodal fusion. LLM. The LLM consists of L transformer blocks and a final decoder. The ℓ -th transformer block outputs the intermediate tokenized representation Hℓ∈ℝN×dH_ ^N× d. Singular Subspaces and Orthogonal Projection For a linear transformation W∈ℝm×nW ^m× n, the singular value decomposition (SVD) is given by W=UΣV⊤W=U V , where U∈ℝm×mU ^m× m and V∈ℝn×nV ^n× n are orthonormal matrices, and Σ∈ℝm×n ^m× n contains the singular values of W, ordered as σ1≥σ2≥⋯≥σr≥0, _1≥ _2≥·s≥ _r≥ 0, on its diagonal entries: Σii=σi _i= _i for 1≤i≤r=rank(W)≤minm,n1≤ i≤ r=rank(W)≤ \m,n\, and zeros otherwise. The columns of V=[v1,v2,…,vn]V=[v_1,v_2,…,v_n] are referred to as the right singular vectors of W. For a given 1≤k≤n1≤ k≤ n, let Vktop=v1,…,vk,Vkbottom=vn−k+1,…,vn,V_k^top=\v_1,…,v_k\, V_k^bottom=\v_n-k+1,…,v_n\, denote the top-k and bottom-k right singular vectors of W, respectively, and the corresponding subspaces are referred to as top-k and bottom-k singular subspaces, and denoted by span(Vktop)span(V_k^top) and span(Vkbottom)span(V_k^bottom). Due to the orthonormality of V, we can measure the alignment of a vector z∈ℝnz ^n with such subspaces using the norm of their projections and the identity ‖z‖22=‖Pktop(z)‖22+‖Pn−kbottom(z)‖22\|z\|^2_2=\|P_k^top(z)\|^2_2+\|P_n-k^bottom(z)\|^2_2. Here, Pktop(z)=(v1⊤z,…,vk⊤z)P_k^top(z)=(v_1 z,…,v_k z) denotes the projection onto span(Vktop)span(V_k^top), and PkbottomP_k^bottom is defined analogously. The projection norms motivate the interpretation as corresponding energies. Effect of Near-Null Singular Directions: An Analogy to Ill-Posed Inverse Problems Consider the linear transformation W:ℝn→ℝmW:R^n ^m with reconstruction map Γ:ℝm→ℝn :R^m ^n. The instability of ill-posed inverse problems states that, in general, stable reconstruction cannot be guaranteed [szegedy2013intriguing, antun2020instabilities, huang2018some], with deeper theoretical treatments in [gottschling2025troublesome, antun2020instabilities]. A common characterization of this instability is the local Lipschitz constant of the reconstruction map at a measurement y∈ℝmy ^m: Lε(Γ,y)=sup0<‖y′−y‖<ε‖Γ(y′)−Γ(y)‖2‖y′−y‖2,ε>0,L_ ( ,y)= _0<\|y -y\|< \| (y )- (y)\|_2\|y -y\|_2, >0, where y′∈ℝmy ^m denotes a perturbed measurement. The local Lipschitz constant may become unbounded, causing large reconstruction errors. We draw an analogy between this instability phenomenon and intermediate linear transformations in VLMs, extending the inverse-problem viewpoint of [jain2025automated]. In inverse problems, the reconstruction map attempts to recover information attenuated by the forward operator, whereas VLMs propagate representations through successive transformations. Consequently, representations aligned with the bottom singular-vector subspace of W are strongly attenuated by the forward transformation, motivating our study of bottom singular-vector subspace alignment. Spectral Alignment Measure For a set of vectors H=h1,…,hN⊂ℝnH=\h_1,…,h_N\ ^n (as arising in the tokenized representations with n=dn=d), we quantify the average alignment with a subspace by computing the average projection energy of the normalized token representations. For span(Vktop)span(V_k^top) this takes the form: Ψk(H,Vktop)=Ψk(H,span(Vktop))=1N∑i=1N∑j=1k(vj⊤hi‖hi‖2)2. _k(H,V^top_k)= _k(H,span(V^top_k))= 1N _i=1^N _j=1^k ( v_j h_i\|h_i\|_2 )^2. (1) Consequently, 0≤Ψk(H,Vktop)≤10≤ _k(H,V_k^top)≤ 1, and larger values indicate greater concentration of representational energy within the selected singular subspace. The alignment measure is defined analogously for the bottom-k singular subspace VkbottomV_k^bottom. In the proposed attack, we instantiate this general definition using a fixed subspace dimension s, i.e., Ψs(⋅,Vsbottom) _s(·,V_s^bottom). Threat Model We consider an untargeted white-box attack where the adversary has full access to the model architecture, parameters, and intermediate representations. Given an image x and a text prompt, the adversary generates an adversarial image xax_a that degrades the model’s response while remaining visually similar to x. The perturbation is constrained by an L∞L_∞ budget c, i.e., xa∈Bc∞(x).x_a∈ B_c^∞(x). The prompt, model parameters, and inference procedure remain unchanged, and each image is attacked independently. 4 A Spectral Framework for Adversarial Vulnerability We first introduce the Spectral Subspace Guided Representation Attack (SSGRA), which instantiates the proposed spectral-guidance principle, and then present the layer-wise probing framework used to analyze the spectral dynamics of adversarial optimization. 4.1 Spectral Subspace Guided Representation Attack (SSGRA) SSGRA extends the Blockwise Similarity Attack (BSA) [yin2023vlattack] by introducing a spectral guidance term based on the alignment measure in Eq. (1). Motivated by the instability phenomenon (Section 3), this guidance aligns intermediate adversarial representations with the bottom singular-vector subspaces of selected linear transformations, where information is most attenuated. We hypothesize that steering representations toward these subspaces progressively weakens semantic information propagation, improving attack effectiveness. SSGRA combines two complementary objectives (Eq 2). The first maximizes the discrepancy between clean and adversarial intermediate representations following BSA [yin2023vlattack], thereby disrupting the learned feature hierarchy. The second maximizes the spectral alignment measure defined in Eq. (1), encouraging adversarial representations to concentrate their energy within bottom singular-vector subspaces. For each selected layer m∈m , let zm(⋅)z_m(·) denote the collection of token representations immediately preceding the corresponding linear transformation, and let Vm,sbottomV^bottom_m,s denote the subspace spanned by the bottom-s right singular vectors of that transformation. Since the textual prompt remains fixed during optimization, we suppress its dependence in the notation and write hkv(x)h_k^v(x), Hℓ(x)H_ (x), and zm(x)z_m(x) instead of hkv(x,p)h_k^v(x,p), Hℓ(x,p)H_ (x,p), and zm(x,p)z_m(x,p). Definition 1(Spectral Subspace Guided Representation Attack (SSGRA)) The SSGRA adversarial example is defined as the solution to the following optimization problem: xa∗=argmaxxa∈Bcp(x) x_a^*= _x_a∈ B_c^p(x) \ −λ[∑k=1K∑j=1Nvcos(hkv,(j)(x),hkv,(j)(xa)) -λ [ _k=1^K _j=1^N_v \! (h_k^v,(j)(x),h_k^v,(j)(x_a) ) (2) +∑ℓ=1L∑i=1Ncos(Hℓ(i)(x),Hℓ(i)(xa))] + _ =1^L _i=1^N \! (H_ ^(i)(x),H_ ^(i)(x_a) ) ] +(1−λ)∑m∈Ψs(zm(xa),Vm,sbottom). +(1-λ) _m _s\! (z_m(x_a),V^bottom_m,s ) \. where hkv,(j)h_k^v,(j) and Hℓ(i)H_ ^(i) denote the representations of the j-th visual token and the i-th multimodal token at the outputs of the k-th visual encoder block and the ℓ -th LLM block, respectively. Furthermore, Ψs(⋅,⋅) _s(·,·) is the spectral alignment measure defined in Eq. (1), s denotes the chosen dimension of the bottom singular subspace, and λ∈[0,1]λ∈[0,1] controls the trade-off between representation discrepancy and spectral subspace alignment. The optimization procedure is summarized in Algorithm 1 in the Appendix. Rather than applying spectral guidance to all layers, we use it only on a selected subset of intermediate linear transformations, denoted by S. The layers are selected by evaluating each visual encoder, fusion, and LLM layer independently on a small validation set and retaining those that yield the strongest attack performance. Developing adaptive layer-selection methods that avoid validation-based tuning is left for future work. 4.2 Layer-wise Probing of Spectral Alignment To analyze the spectral alignment of adversarial representations, we perform layer-wise adversarial probing. For each transformer block i, we generate an adversarial example by minimizing the cosine similarity between the clean and adversarial feature maps: xa,i∗=argminxa∈Bc(x)⟨Hi(x),Hi(xa)⟩F‖Hi(x)‖F‖Hi(xa)‖F,x_a,i^*= _x_a∈ B_c(x) H_i(x),H_i(x_a) _F\|H_i(x)\|_F\,\|H_i(x_a)\|_F, (3) where Hi(⋅)H_i(·) denotes the feature map at layer i, ⟨⋅,⋅⟩F ·,· _F is the Frobenius inner product, and Bc(x)B_c(x) is the admissible perturbation set. For each adversarial example xa,i∗x_a,i^*, we compute the spectral subspace alignment across all vision and language layers using Eq. (1). Repeating this procedure over all target layers and input samples enables us to analyze alignment with top and bottom singular-vector subspaces during and after attack optimization. 4.3 Evaluation Metrics We evaluate attack effectiveness by comparing the adversarial output y^a y_a with the corresponding clean output y^c y_c using BERTScore [zhang2019bertscore] and ROUGE-L [lin2004rouge]. BERTScore measures semantic similarity using contextual token embeddings from RoBERTa-large. ROUGE-L measures lexical similarity based on the longest common subsequence (LCS), capturing structural degradation of the generated response. For both metrics, we report Precision, Recall, and F1, where lower scores indicate stronger attacks. Additional details are provided in Appendix 0.A.2. 5 Experiments We evaluate the proposed attacks against representative baselines, analyze their spectral mechanisms, and present ablation studies. 5.1 Experimental Setup We evaluate attacks on different VLMs, namely Gemma-3 (4B) [gemmateam2025gemma3technicalreport], Qwen2.5-VL (7B) [bai2025qwen25vltechnicalreport], and LLaVA-1.5 (7B) [an2025llava]. Experiments are conducted on ImageNet [deng2009imagenet], whose diverse visual categories enable assessment of generalization across image content. Given an input image and the prompt “What is shown in the image?”, we optimize sample-specific adversarial perturbations to degrade the model’s image description while remaining visually imperceptible. Each experimental instance is defined by a perturbation budget and an attack method, evaluated over 100 images. The perturbation budget ranges from 0.002 to 0.005 in the ℓ∞ _∞ norm, selected via grid search such that the lower bound captures the regime where outputs remain semantically similar across methods, and the upper bound where performance differences become pronounced. We compare SSGRA against six representative baselines BSA, DRA, FDA, EGA, SSPA, and NLL. All attacks are optimized using Adam following [carlini2017towards] with a fixed computational budget of 10001000 gradient steps to ensure a fair comparison across methods and enable evaluation on a sufficiently large number of samples for statistically reliable quantitative results. Grid search over learning rates 10−2, 5×10−3, 10−3, 5×10−4, 10−4\10^-2,\,5× 10^-3,\,10^-3,\,5× 10^-4,\,10^-4\ identified 10−310^-3 as consistently yielding the best attack performance across the three models. The adversarial perturbation is initialized with small random noise of near-zero magnitude, following standard practice in iterative adversarial optimization. 5.2 Quantitative comparison with State-of-the-Art Attacks (a) Qwen 2.5-VL BERT F1 (b) LLaVA 1.5 BERT F1 (c) Gemma 3 BERT F1 (d) Qwen 2.5-VL ROUGE-L F1 (e) LLaVA 1.5 ROUGE-L F1 (f) Gemma 3 ROUGE-L F1 Figure 3: SSGRA vs representative baselines across perturbation budgets. Lower scores indicate stronger attacks. Figure 3 shows the performance of SSGRA and the selected baselines across perturbation budgets using BERTScore F1 and ROUGE-L F1. Qwen 2.5-VL: SSGRA consistently outperforms all baselines, achieving an additional 7.90–19.74% relative degradation under BERTScore F1 and 30.50–97.93% under ROUGE-L F1 over the strongest baseline. LLaVA 1.5: SSGRA achieves up to 7.46% and 18.90% additional relative degradation over the strongest baseline under BERTScore F1 and ROUGE-L F1, respectively. The improvement increases with the perturbation budget, indicating that spectral subspace guidance becomes more effective at larger perturbation budgets. Gemma 3: Although the margins are smaller, SSGRA consistently achieves the strongest attacks, providing up to 0.80% and 3.07% additional relative degradation over the strongest baseline under BERTScore F1 and ROUGE-L F1, respectively. Among the three VLMs, Qwen 2.5-VL exhibits the largest degradation, consistent with its having the highest proportion of near-null singular directions (Table 1), and thus the largest spectral attack surface. In contrast, Gemma 3 shows the smallest relative degradation, consistent with its lower proportion of near-null singular directions compared with Qwen 2.5-VL and LLaVA 1.5, limiting the opportunity to exploit bottom singular-vector subspaces. A detailed discussion on spectral characterization follows in Section 5.4. Additional results are provided in the Appendix. In particular, Table 3 summarizes the F1 scores and the additional relative degradation (%) over the strongest baseline attack. The corresponding Precision, Recall, and F1 scores are provided in Tables 4–6, while the corresponding trends are shown in Figures 10 and 11 in the Appendix. Overall, the results suggest a relationship between spectral conditioning and adversarial vulnerability and provide empirical support for our hypothesis that bottom singular-vector subspaces play a central role in transformer-based VLMs. 5.3 Qualitative Analysis (a) Qwen2.5-VL (b) LLaVA-1.5 (c) Gemma 3 Figure 4: Qualitative adversarial examples generated under a common perturbation budget of c=0.003c=0.003 across three vision–language models. Figure 4 shows representative adversarial examples generated under a common extremely small perturbation budget of c=0.003c=0.003 for the chosen models. Compared with existing baselines, SSGRA consistently induces larger semantic deviations while preserving the perceptual appearance of the input images. For the coastal scene (Qwen2.5-VL), baseline attacks largely preserve the correct scene description, whereas SSGRA causes the model response to become meaningless. Likewise, for the ambulance image (LLaVA-1.5), baseline attacks still identify the ambulance despite minor hallucinations, whereas SSGRA instead describes an unrelated car crash scene. For the flower image (Gemma 3), baseline attacks largely preserve the correct flower category despite minor hallucinations involving a cup and a lizard, whereas SSGRA generates an unrelated prediction ("Green Slime/Fizz Pop Rocks"). More qualitative examples are presented in Appendix 0.A.4. Overall, SSGRA induces larger semantic shifts than existing baselines, consistent with the quantitative findings in Section 5.2. 5.4 Spectral Characterization of Adversarial Representations (a) Qwen2.5-VL (b) LLaVA-1.5 (c) Gemma 3 Figure 5: Distribution of the largest (σmax _ , left) and smallest (σmin _ , right) singular values across non-attention linear operators in Qwen2.5-VL, LLaVA-1.5, and Gemma 3. Table 1: Distribution of extreme singular values across non-attention linear operators. Threshold Qwen2.5-VL LLaVA-1.5 Gemma 3 Count Percentage Count Percentage Count Percentage Largest Singular Value (σmax _ ) σmax>101 _ >10^1 46/245 18.78% 40/206 19.42% 25/221 11.31% σmax>102 _ >10^2 0/245 0.00% 0/206 0.00% 2/221 0.90% Smallest Singular Value (σmin _ ) σmin<10−2 _ <10^-2 62/245 25.31% 60/206 29.13% 32/221 14.48% σmin<10−3 _ <10^-3 61/245 24.90% 57/206 27.67% 27/221 12.22% σmin<10−4 _ <10^-4 24/245 9.80% 26/206 12.62% 6/221 2.71% σmin<10−5 _ <10^-5 3/245 1.22% 4/206 1.94% 3/221 1.36% σmin<10−6 _ <10^-6 0/245 0.00% 1/206 0.49% 0/221 0.00% Singular Value Analysis of intermediate linear operators of VLMs. We analyze the distributions of the largest and smallest singular values of all non-attention linear operators in the evaluated models. Figure 5 visualizes these distributions, while Table 1 summarizes the prevalence of extreme singular values. Across all three models, near-null singular directions occur substantially more frequently than strongly amplifying ones. For example, σmin<10−3 _ <10^-3 is observed in 24.90%, 27.67%, and 12.22% of operators in Qwen2.5-VL, LLaVA-1.5, and Gemma 3, respectively, with similar trends persisting at smaller thresholds. In contrast, singular values satisfying σmax>10 _ >10 occur much less frequently, accounting for only 18.78%, 19.42%, and 11.31% of operators. Singular values exceeding 10210^2 are nearly absent across all models (Table 1), consistent with prior work showing that large singular values are commonly constrained or implicitly regularized during training. These spectral characteristics help explain the quantitative results in Section 5.2. Qwen2.5-VL and LLaVA-1.5 contain approximately twice as many near-null singular directions as Gemma 3, providing substantially larger bottom singular-vector subspaces that SSGRA can exploit. Consequently, these models exhibit much larger relative degradation over existing attacks than Gemma 3. (a) BSA, Top k = 10 (b) BSA, Bottom k = 10 Figure 6: Distribution of the spectral alignment measure Ψk _k before and after BSA optimization for the top-10 and bottom-10 right singular-vector subspaces. Post-Attack Spectral Alignment Analysis. To examine whether spectral alignment emerges naturally during adversarial optimization, we analyze adversarial representations generated by BSA [yin2023vlattack], which does not optimize spectral objectives. We evaluate 100 images (c=0.005c=0.005) and compute Ψk _k using Eq. (1) with respect to the top-10 and bottom-10 right singular vectors of selected intermediate layers. Figure 6 shows the corresponding alignment distributions before and after optimization. At the MLP gate proj and MLP up proj layers, adversarial representations exhibit increased alignment with the bottom-k singular subspace, whereas alignment with the top-k subspace remains unchanged or decreases. This trend is absent in attention layers, likely due to their more complex transformations. Since BSA does not optimize spectral alignment, the results suggest that untargeted adversarial optimization naturally steers representations toward information-attenuating bottom singular subspaces. SSGRA strengthens the attack by explicitly promoting this alignment. (a) Ψtop(v) _top(v) for sample 1 (b) Ψtop(v) _top(v) for sample 2 (c) Ψbottom(v) _bottom(v) for sample 1 (d) Ψbottom(v) _bottom(v) for sample 2 Figure 7: Layer-wise evolution of top and bottom most singular vectors alignment during adversarial optimization on Gemma 3 (two representative samples). Spectral Alignment Dynamics During Attack Optimization. We analyze spectral alignment during adversarial optimization on Gemma 3. For each target layer, adversarial examples are generated using Eq. (3), and alignment with the largest and smallest singular vectors is computed using Eq. (1). Figure 7 shows trajectories from the first multimodal block. Adversarial optimization progressively increases alignment with the bottom singular vectors, while alignment with the top singular vectors remains nearly unchanged or slightly decreases, supporting our hypothesis. 5.5 Computational Complexity Table 2 reports the computational complexity of the evaluated attacks in FLOPs. Across all three VLMs, SSGRA has computational cost comparable to representative optimization-based attacks (BSA, EGA, and CE) while achieving improved attack effectiveness. FLOPs should be compared only within the same VLM, as they depend not only on parameter count but also on the vision encoder, attention mechanism, input image resolution, and the number of visual tokens processed by the multimodal model. For Qwen2.5-VL, SSGRA requires fewer FLOPs than all other methods while achieving the strongest attack performance (Section 5.2), since hyperparameter tuning selected λ=0λ=0, reducing Eq. (2) to just spectral alignment objective. Table 2: Computational complexity of the attacks measured in floating-point operations (FLOPs). Lower values indicate higher computational efficiency. Method Qwen2.5-VL LLaVA-1.5 Gemma 3 BSA 2.94×10132.94× 10^13 1.02×10131.02× 10^13 2.59×10132.59× 10^13 DRA 1.42×10131.42× 10^13 8.68×10128.68× 10^12 1.01×10131.01× 10^13 FDA 6.14×10126.14× 10^12 1.14×10121.14× 10^12 1.01×10131.01× 10^13 SSPA 6.14×10126.14× 10^12 1.14×10121.14× 10^12 1.56×10131.56× 10^13 EGA 2.68×10132.68× 10^13 2.69×10132.69× 10^13 1.95×10131.95× 10^13 CE 2.69×10132.69× 10^13 2.69×10132.69× 10^13 1.96×10131.96× 10^13 SSGRA 1.31×1.31× 10^13 2.61×2.61× 10^13 2.60×2.60× 10^13 5.6 Ablation Study (a) Effect of the spectral alignment loss. (b) Bottom- vs. top-singular subspace alignment. Figure 9: Ablation analysis validating the design choices of SSGRA. We ablate SSGRA on Qwen2.5-VL to evaluate (1) the contribution of the spectral alignment objective and (2) the effect of aligning adversarial representations with bottom versus top singular-vector subspaces. Effect of the Spectral Alignment Term. Figure 9(a) compares SSGRA with and without the spectral alignment term in Eq. (2). Removing this term increases both BERTScore F1 and ROUGE-L F1 across all perturbation budgets, indicating weaker attacks and confirming that spectral alignment is the primary contributor to the improved attack effectiveness. The performance gap widens with increasing c, showing that the benefit of spectral guidance increases with the perturbation budget. Bottom vs. Top Singular Subspace. Figure 9(b) compares SSGRA-bottom and SSGRA-top, which align adversarial representations with the bottom-k and top-k singular subspaces, respectively. SSGRA-bottom achieves lower BERTScore and ROUGE-L scores, indicating stronger attacks, whereas SSGRA-top causes only minor degradation. These results support the hypothesis that bottom singular-vector subspaces constitute the primary spectral attack surface in VLMs. 6 Conclusion We presented a spectral perspective on adversarial vulnerability in transformer-based VLMs by analyzing their intermediate linear transformations. We identified bottom singular-vector subspaces as a previously overlooked spectral attack surface and proposed SSGRA, which exploits this insight to improve attack effectiveness on three state-of-the-art VLMs. Our analyses show that near-null singular directions are substantially more prevalent than strongly amplifying ones and that untargeted adversarial optimization naturally tends to increase alignment with these information-attenuating subspaces. These findings suggest that, alongside existing spectral-norm regularization techniques for large singular values, controlling near-null singular directions may provide a complementary approach to improving adversarial robustness. Limitations. This work focuses on untargeted white-box attacks to isolate and analyze the spectral mechanisms underlying adversarial vulnerability. The applicability of the proposed framework to transfer-based and black-box settings has not yet been investigated and remains future work. References Appendix 0.A Appendix 0.A.1 Spectral Subspace Guided Representation Attack (SSGRA) Algorithm Algorithm 1 Spectral Subspace Guided Representation Attack (SSGRA) Input: Image x, prompt p, VLM ℱF, perturbation budget c, step size η, number of steps T, selected linear transformations S, trade-off parameter λ, bottom-subspace dimension s Output: Adversarial image xa∗x_a^* 1: Initialize perturbation with small random noise: 2: δ∼(−ξ,ξ)δ (-ξ,ξ), where ξ≪cξ c 3: Fix prompt p and suppress it in the representation notation. 4: Compute clean representations hkv,(j)(x)k,j\h_k^v,(j)(x)\_k,j and Hℓ(i)(x)ℓ,i\H_ ^(i)(x)\_ ,i 5: for each selected transformation m∈m do 6: Compute SVD of its weight matrix: Wm=UmΣmVm⊤W_m=U_m _mV_m 7: Extract the bottom-s right singular-vector subspace Vm,sbottomV^bottom_m,s 8: end for 9: for τ=1τ=1 to T do 10: Construct the adversarial image: 11: xa←clip(x+δ,x−c,x+c)x_a (x+δ,\;x-c,\;x+c) 12: Compute adversarial representations hkv,(j)(xa)k,j\h_k^v,(j)(x_a)\_k,j and Hℓ(i)(xa)ℓ,i\H_ ^(i)(x_a)\_ ,i 13: Compute BSA representation discrepancy term: ℒBSA=∑k=1K∑j=1Nvcos(hkv,(j)(x),hkv,(j)(xa))+∑ℓ=1L∑i=1Ncos(Hℓ(i)(x),Hℓ(i)(xa))L_BSA= _k=1^K _j=1^N_v \! (h_k^v,(j)(x),h_k^v,(j)(x_a) )+ _ =1^L _i=1^N \! (H_ ^(i)(x),H_ ^(i)(x_a) ) 14: Compute spectral subspace alignment term: 15: ℒS=∑m∈Ψs(zm(xa),Vm,sbottom)L_S= _m _s\! (z_m(x_a),V^bottom_m,s ) 16: Compute SSGRA objective: 17: ℒSSGRA=−λℒBSA+(1−λ)ℒSL_SSGRA=- _BSA+(1-λ)L_S 18: Update perturbation by Adam ascent: 19: δ←AdamStep(δ,∇δℒSSGRA)δ \! (δ, _δL_SSGRA ) 20: Project perturbation onto the L∞L_∞ ball: 21: δ←clip(δ,−c,c)δ (δ,-c,c) 22: end for 23: Construct the final adversarial image: 24: xa∗←clip(x+δ,x−c,x+c)x_a^* (x+δ,\;x-c,\;x+c) 25: return xa∗x_a^* Algorithm 1 summarizes the optimization procedure of SSGRA. The attack first fixes the textual prompt and computes the clean intermediate representations of the input image across the visual encoder and language-model blocks. For each selected intermediate linear transformation, SSGRA performs an SVD of the corresponding weight matrix and extracts the bottom-s right singular-vector subspace. These subspaces define the information-attenuating directions used by the spectral alignment objective. During optimization, the adversarial image is constructed by adding a learnable perturbation δ to the clean image and clipping it within the prescribed L∞L_∞ budget. At each iteration, the model is evaluated on the current adversarial image to obtain the corresponding intermediate representations. SSGRA then combines two objectives. The first is the BSA representation-discrepancy term, which reduces the similarity between clean and adversarial feature representations across visual and language layers. The second is the proposed spectral subspace alignment term, which encourages adversarial representations before the selected transformations to align with the bottom singular-vector subspaces. The trade-off parameter λ balances these two effects. The perturbation is updated by Adam ascent on the combined SSGRA objective, since the attack maximizes representation disruption and spectral alignment. After each update, the perturbation is projected back onto the L∞L_∞ ball to ensure that the adversarial image remains visually close to the original input. The final adversarial example is obtained by applying the optimized perturbation to the clean image and clipping it to satisfy the perturbation constraint. The algorithm explicitly guides adversarial representations toward directions that are strongly attenuated by intermediate linear transformations, thereby weakening semantic information propagation through the VLM. 0.A.2 Evaluation Metrics Details We evaluate attack effectiveness using two complementary text-based metrics that compare the adversarial model output y^a y_a against the clean output y^c y_c. BERTScore. BERTScore [zhang2019bertscore] computes token-level semantic similarity between y^a y_a and y^c y_c using contextual embeddings from a pretrained language model (RoBERTa-large). For each token ai∈y^a_i∈ y_a and cj∈y^c_j∈ y_c, cosine similarity is computed in embedding space. Precision, recall, and F1 are defined as: PBERT=1|y^a|∑ai∈y^amaxcj∈y^ccos(ai,cj),P_BERT= 1| y_a| _a_i∈ y_a _c_j∈ y_c (e_a_i,e_c_j), (4) RBERT=1|y^c|∑cj∈y^cmaxai∈y^acos(ai,cj),R_BERT= 1| y_c| _c_j∈ y_c _a_i∈ y_a (e_a_i,e_c_j), (5) F1BERT=2⋅PBERT⋅RBERTPBERT+RBERT,F1_BERT= 2· P_BERT· R_BERTP_BERT+R_BERT, (6) where aie_a_i and cje_c_j are the contextual embeddings of tokens aia_i and cjc_j respectively. Lower scores indicate greater semantic degradation of the adversarial output relative to the clean output. ROUGE-L. ROUGE-L [lin2004rouge] measures lexical overlap via the Longest Common Subsequence (LCS) between y^a y_a and y^c y_c. Let LCS(y^a,y^c)LCS( y_a, y_c) denote the length of the longest common subsequence. Precision, recall, and F1 are: PR=|LCS(y^a,y^c)||y^a|,R=|LCS(y^a,y^c)||y^c|,F1R=2⋅PR⋅RRPR+R.P_R= |LCS( y_a, y_c)|| y_a|, R_R= |LCS( y_a, y_c)|| y_c|, F1_R= 2· P_R· R_RP_R+R_R. (7) Unlike BERTScore, ROUGE-L is sensitive to structural content loss: a low recall indicates that the adversarial output fails to reproduce key content from the clean description. The two metrics are complementary — BERTScore captures semantic similarity robust to paraphrase, while ROUGE-L captures lexical fidelity and structural degradation. We report mean and standard deviation over 100 images per experimental configuration. Table 3: Performance of different attack methods under varying perturbation budgets. (a) BERTScore F1. (b) ROUGE-L F1. Lower values indicate stronger attack effectiveness. (a) BERTScore F1 Method c=0.002c=0.002 c=0.0025c=0.0025 c=0.003c=0.003 c=0.0035c=0.0035 c=0.004c=0.004 c=0.0045c=0.0045 c=0.005c=0.005 Qwen2.5-VL BSA 0.895±0.0280.895± 0.028 0.877¯±0.035 0.877± 0.035 0.874±0.0320.874± 0.032 0.864¯±0.037 0.864± 0.037 0.861¯±0.035 0.861± 0.035 0.856¯±0.038 0.856± 0.038 0.845¯±0.037 0.845± 0.037 DRA 0.938±0.0260.938± 0.026 0.943±0.0230.943± 0.023 0.936±0.0230.936± 0.023 0.934±0.0230.934± 0.023 0.935±0.0210.935± 0.021 0.936±0.0230.936± 0.023 0.931±0.0180.931± 0.018 FDA 0.933±0.0210.933± 0.021 0.928±0.0250.928± 0.025 0.929±0.0230.929± 0.023 0.925±0.0220.925± 0.022 0.922±0.0230.922± 0.023 0.918±0.0220.918± 0.022 0.918±0.0240.918± 0.024 SSPA 0.926±0.0180.926± 0.018 0.928±0.0200.928± 0.020 0.919±0.0200.919± 0.020 0.922±0.0220.922± 0.022 0.917±0.0200.917± 0.020 0.913±0.0190.913± 0.019 0.916±0.0190.916± 0.019 EGA 0.896±0.0220.896± 0.022 0.883±0.0220.883± 0.022 0.883±0.0240.883± 0.024 0.865±0.0530.865± 0.053 0.872±0.0310.872± 0.031 0.856¯±0.061 0.856± 0.061 0.865±0.0340.865± 0.034 CE 0.886¯±0.016 0.886± 0.016 0.884±0.0170.884± 0.017 0.873¯±0.028 0.873± 0.028 0.875±0.0340.875± 0.034 0.871±0.0300.871± 0.030 0.865±0.0480.865± 0.048 0.856±0.0480.856± 0.048 SSGRA 0.816±0.0880.816± 0.088 0.768±0.1470.768± 0.147 0.743±0.1370.743± 0.137 0.704±0.2170.704± 0.217 0.754±0.0420.754± 0.042 0.687±0.2060.687± 0.206 0.724±0.1210.724± 0.121 Gain over Best (%) 7.90 % 12.40 % 14.89 % 18.52 % 12.43 % 19.74 % 14.32% LLaVa-1.5 BSA 0.926±0.0340.926± 0.034 0.911±0.0340.911± 0.034 0.902¯±0.028 0.902± 0.028 0.895¯±0.019 0.895± 0.019 0.892¯±0.020 0.892± 0.020 0.886±0.0280.886± 0.028 0.884±0.0190.884± 0.019 DRA 0.951±0.0330.951± 0.033 0.940±0.0220.940± 0.022 0.935±0.0270.935± 0.027 0.932±0.0300.932± 0.030 0.926±0.0270.926± 0.027 0.914±0.0330.914± 0.033 0.915±0.0280.915± 0.028 FDA 0.954±0.0270.954± 0.027 0.946±0.0260.946± 0.026 0.945±0.0270.945± 0.027 0.944±0.0270.944± 0.027 0.939±0.0260.939± 0.026 0.942±0.0260.942± 0.026 0.939±0.0250.939± 0.025 SSPA 0.949±0.0270.949± 0.027 0.947±0.0240.947± 0.024 0.936±0.0210.936± 0.021 0.934±0.0230.934± 0.023 0.933±0.0260.933± 0.026 0.928±0.0220.928± 0.022 0.934±0.0260.934± 0.026 EGA 0.933±0.0310.933± 0.031 0.922±0.0230.922± 0.023 0.918±0.0260.918± 0.026 0.913±0.0270.913± 0.027 0.903±0.0270.903± 0.027 0.885¯±0.131 0.885± 0.131 0.875¯±0.133 0.875± 0.133 CE 0.921¯±0.022 0.921± 0.022 0.916±0.0220.916± 0.022 0.911±0.0200.911± 0.020 0.909±0.0180.909± 0.018 0.907±0.0190.907± 0.019 0.888±0.1290.888± 0.129 0.883±0.1300.883± 0.130 SSGRA 0.916±0.0290.916± 0.029 0.912±0.0270.912± 0.027 0.882±0.1330.882± 0.133 0.837±0.2160.837± 0.216 0.853±0.1790.853± 0.179 0.819±0.2130.819± 0.213 0.816±0.2140.816± 0.214 Gain over Best (%) 0.54 % – 2.22 % 6.48 % 4.37 % 7.46 % 6.74% Gemma 3 BSA 0.905±0.0350.905± 0.035 0.890±0.0340.890± 0.034 0.880¯±0.035 0.880± 0.035 0.870¯±0.043 0.870± 0.043 0.861¯±0.031 0.861± 0.031 0.856¯±0.031 0.856± 0.031 0.847±0.0370.847± 0.037 DRA 0.919±0.0270.919± 0.027 0.918±0.0250.918± 0.025 0.914±0.0290.914± 0.029 0.911±0.0250.911± 0.025 0.908±0.0290.908± 0.029 0.909±0.0280.909± 0.028 0.903±0.0300.903± 0.030 FDA 0.927±0.0360.927± 0.036 0.924±0.0370.924± 0.037 0.923±0.0320.923± 0.032 0.919±0.0340.919± 0.034 0.923±0.0320.923± 0.032 0.921±0.0300.921± 0.030 0.918±0.0260.918± 0.026 SSPA 0.923±0.0220.923± 0.022 0.918±0.0280.918± 0.028 0.916±0.0320.916± 0.032 0.913±0.0290.913± 0.029 0.913±0.0310.913± 0.031 0.905±0.0270.905± 0.027 0.909±0.0310.909± 0.031 EGA 0.911±0.0290.911± 0.029 0.918±0.0310.918± 0.031 0.908±0.0300.908± 0.030 0.907±0.0310.907± 0.031 0.901±0.0310.901± 0.031 0.908±0.0300.908± 0.030 0.902±0.0300.902± 0.030 CE 0.886±0.0260.886± 0.026 0.883±0.0250.883± 0.025 0.879±0.0300.879± 0.030 0.879±0.0260.879± 0.026 0.877±0.0330.877± 0.033 0.881±0.0340.881± 0.034 0.867±0.0410.867± 0.041 SSGRA 0.903±0.0380.903± 0.038 0.883±0.0420.883± 0.042 0.873±0.0360.873± 0.036 0.868±0.0320.868± 0.032 0.858±0.0340.858± 0.034 0.855±0.0370.855± 0.037 0.849±0.0300.849± 0.030 Gain over Best (%) – – 0.80 % 0.23 % 0.35 % 0.12 % – (b) ROUGE-L F1 Method c=0.002c=0.002 c=0.0025c=0.0025 c=0.003c=0.003 c=0.0035c=0.0035 c=0.004c=0.004 c=0.0045c=0.0045 c=0.005c=0.005 Qwen2.5-VL BSA 0.352±0.1160.352± 0.116 0.296±0.1150.296± 0.115 0.285±0.0970.285± 0.097 0.252¯±0.118 0.252± 0.118 0.232¯±0.113 0.232± 0.113 0.228±0.0910.228± 0.091 0.199¯±0.114 0.199± 0.114 DRA 0.551±0.1870.551± 0.187 0.577±0.1710.577± 0.171 0.536±0.1700.536± 0.170 0.539±0.1640.539± 0.164 0.528±0.1490.528± 0.149 0.533±0.1590.533± 0.159 0.501±0.1320.501± 0.132 FDA 0.517±0.1600.517± 0.160 0.486±0.1530.486± 0.153 0.494±0.1530.494± 0.153 0.471±0.1480.471± 0.148 0.447±0.1440.447± 0.144 0.440±0.1310.440± 0.131 0.434±0.1390.434± 0.139 SSPA 0.498±0.1220.498± 0.122 0.494±0.1360.494± 0.136 0.437±0.0990.437± 0.099 0.447±0.1420.447± 0.142 0.428±0.1190.428± 0.119 0.406±0.1070.406± 0.107 0.427±0.1210.427± 0.121 EGA 0.337±0.1130.337± 0.113 0.294±0.0860.294± 0.086 0.293±0.0970.293± 0.097 0.277±0.1000.277± 0.100 0.271±0.0750.271± 0.075 0.261±0.0770.261± 0.077 0.274±0.0650.274± 0.065 CE 0.282¯±0.066 0.282± 0.066 0.277¯±0.058 0.277± 0.058 0.239¯±0.071 0.239± 0.071 0.256±0.0760.256± 0.076 0.235±0.0570.235± 0.057 0.226¯±0.065 0.226± 0.065 0.217±0.0810.217± 0.081 SSGRA 0.196±0.2350.196± 0.235 0.102±0.1710.102± 0.171 0.057±0.1640.057± 0.164 0.052±0.1550.052± 0.155 0.028±0.1140.028± 0.114 0.010±0.0470.010± 0.047 0.010±0.0200.010± 0.020 Gain over Best (%) 30.50 % 63.18 % 76.15 % 79.37 % 97.93 % 95.57 % 94.97% LLaVa-1.5 BSA 0.492±0.2150.492± 0.215 0.416±0.1960.416± 0.196 0.369¯±0.135 0.369± 0.135 0.330¯±0.080 0.330± 0.080 0.320¯±0.103 0.320± 0.103 0.297¯±0.077 0.297± 0.077 0.291¯±0.065 0.291± 0.065 DRA 0.639±0.2120.639± 0.212 0.545±0.1640.545± 0.164 0.525±0.1830.525± 0.183 0.514±0.2010.514± 0.201 0.477±0.1520.477± 0.152 0.426±0.1640.426± 0.164 0.432±0.1560.432± 0.156 FDA 0.644±0.1950.644± 0.195 0.591±0.1820.591± 0.182 0.590±0.2030.590± 0.203 0.585±0.1920.585± 0.192 0.550±0.1820.550± 0.182 0.570±0.1840.570± 0.184 0.558±0.1820.558± 0.182 SSPA 0.620±0.2020.620± 0.202 0.594±0.1770.594± 0.177 0.524±0.1440.524± 0.144 0.520±0.1540.520± 0.154 0.520±0.1700.520± 0.170 0.483±0.1470.483± 0.147 0.518±0.1730.518± 0.173 EGA 0.511±0.2020.511± 0.202 0.456±0.1380.456± 0.138 0.427±0.1620.427± 0.162 0.416±0.1510.416± 0.151 0.371±0.1380.371± 0.138 0.377±0.1430.377± 0.143 0.336±0.1170.336± 0.117 CE 0.439¯±0.134 0.439± 0.134 0.400±0.1370.400± 0.137 0.375±0.1220.375± 0.122 0.375±0.0940.375± 0.094 0.357±0.0960.357± 0.096 0.340±0.0880.340± 0.088 0.329±0.1000.329± 0.100 SSGRA 0.422±0.1690.422± 0.169 0.406±0.1570.406± 0.157 0.364±0.1840.364± 0.184 0.299±0.1380.299± 0.138 0.297±0.1320.297± 0.132 0.252±0.1210.252± 0.121 0.236±0.1380.236± 0.138 Gain over Best (%) 3.87 % – 1.35 % 9.39 % 7.19 % 15.15 % 18.90% Gemma 3 BSA 0.406±0.1420.406± 0.142 0.331±0.1160.331± 0.116 0.315±0.1120.315± 0.112 0.287±0.1160.287± 0.116 0.253±0.0980.253± 0.098 0.238±0.0920.238± 0.092 0.228¯±0.109 0.228± 0.109 DRA 0.456±0.1400.456± 0.140 0.457±0.1210.457± 0.121 0.433±0.1310.433± 0.131 0.405±0.1200.405± 0.120 0.388±0.1340.388± 0.134 0.402±0.1260.402± 0.126 0.365±0.1250.365± 0.125 FDA 0.516±0.1860.516± 0.186 0.509±0.1860.509± 0.186 0.492±0.1660.492± 0.166 0.471±0.1570.471± 0.157 0.478±0.1390.478± 0.139 0.474±0.1520.474± 0.152 0.450±0.1210.450± 0.121 SSPA 0.464±0.1180.464± 0.118 0.441±0.1310.441± 0.131 0.440±0.1410.440± 0.141 0.418±0.1340.418± 0.134 0.422±0.1290.422± 0.129 0.388±0.1280.388± 0.128 0.404±0.1260.404± 0.126 EGA 0.423±0.1180.423± 0.118 0.463±0.1440.463± 0.144 0.408±0.1250.408± 0.125 0.410±0.1280.410± 0.128 0.382±0.1270.382± 0.127 0.420±0.1310.420± 0.131 0.390±0.1190.390± 0.119 CE 0.289±0.0800.289± 0.080 0.285±0.0710.285± 0.071 0.269±0.0790.269± 0.079 0.275¯±0.082 0.275± 0.082 0.273±0.0880.273± 0.088 0.285±0.0810.285± 0.081 0.259±0.0850.259± 0.085 SSGRA 0.407±0.1360.407± 0.136 0.319±0.1180.319± 0.118 0.288±0.1190.288± 0.119 0.272±0.1050.272± 0.105 0.259±0.1040.259± 0.104 0.241±0.0880.241± 0.088 0.221±0.0930.221± 0.093 Gain over Best (%) – – – 1.09 % – – 3.07% (a) Precision(Qwen2.5-VL) (b) Recall (Qwen2.5-VL) (c) F1 (Qwen2.5-VL) (d) Precision (LLaVa-1.5) (e) Recall (LLaVa-1.5) (f) F1 (LLaVa-1.5) (g) Precision (Gemma 3) (h) Recall (Gemma 3) (i) F1 (Gemma 3) Figure 10: BERT-score comparison of different adversarial attack methods across perturbation budgets for Qwen 2.5-VL, LLaVa 1.5, and Gemma 3. Each row corresponds to a model, while the columns show Precision, Recall, and F1 score, respectively. (a) Precision (Qwen2.5-VL) (b) Recall (Qwen2.5-VL) (c) F1 (Qwen2.5-VL) (d) Precision (LLaVa-1.5) (e) Recall (LLaVa-1.5) (f) F1 (LLaVa-1.5) (g) Precision (Gemma 3) (h) Recall (Gemma 3) (i) F1 (Gemma 3) Figure 11: ROUGE-L score comparison of sample-specific attacks across Qwen 2.5-VL, LLaVa 1.5, and Gemma 3. The three columns report Precision, Recall, and F1 score, respectively, while each row corresponds to a different vision-language model. 0.A.3 Comprehensive Quantitative Results Figures 10 and 11 visualize the Precision, Recall, and F1 trends under BERTScore and ROUGE-L across perturbation budgets, while Tables 4, 5, and 6 report the corresponding numerical results. Across all three VLMs, SSGRA consistently achieves the lowest BERTScore and ROUGE-L scores in most settings, with the largest improvements on Qwen2.5-VL and the smallest on Gemma 3, consistent with the spectral characterization presented in Section 5.4. In addition to the F1 scores reported in Section 5.2, here we provide the corresponding Precision and Recall values, enabling a more detailed analysis of attack behavior. The results show that the improvements achieved by SSGRA are not driven by a single evaluation component but are consistently reflected across all three metrics. Furthermore, the complete numerical results complement the plots by reporting the mean and standard deviation for every perturbation budget, providing a comprehensive view of both attack effectiveness and its variability across the evaluated samples. Table 4: Performance of different attack methods on Qwen2.5-VL under varying perturbation budgets c. Top: BERTScore (Precision, Recall and F1, Mean± ). Bottom: ROUGE-L (Precision, Recall and F1, Mean± ). (a) BERTScore Method Metric c=0.002c=0.002 c=0.0025c=0.0025 c=0.003c=0.003 c=0.0035c=0.0035 c=0.004c=0.004 c=0.0045c=0.0045 c=0.005c=0.005 BSA P 0.896±0.0290.896± 0.029 0.879±0.0370.879± 0.037 0.875±0.0360.875± 0.036 0.866±0.0360.866± 0.036 0.866±0.0320.866± 0.032 0.856±0.0380.856± 0.038 0.847±0.0400.847± 0.040 R 0.894±0.0290.894± 0.029 0.876±0.0340.876± 0.034 0.874±0.0310.874± 0.031 0.862±0.0390.862± 0.039 0.856±0.0400.856± 0.040 0.856±0.0330.856± 0.033 0.843±0.0380.843± 0.038 F1 0.895±0.0280.895± 0.028 0.877±0.0340.877± 0.034 0.874±0.0320.874± 0.032 0.864±0.0370.864± 0.037 0.861±0.0350.861± 0.035 0.856±0.0340.856± 0.034 0.845±0.0370.845± 0.037 DRA P 0.937±0.0260.937± 0.026 0.943±0.0250.943± 0.025 0.936±0.0240.936± 0.024 0.935±0.0240.935± 0.024 0.935±0.0220.935± 0.022 0.935±0.0250.935± 0.025 0.931±0.0200.931± 0.020 R 0.938±0.0280.938± 0.028 0.943±0.0230.943± 0.023 0.935±0.0240.935± 0.024 0.934±0.0230.934± 0.023 0.935±0.0230.935± 0.023 0.936±0.0220.936± 0.022 0.931±0.0190.931± 0.019 F1 0.938±0.0260.938± 0.026 0.943±0.0230.943± 0.023 0.936±0.0230.936± 0.023 0.934±0.0230.934± 0.023 0.935±0.0210.935± 0.021 0.936±0.0230.936± 0.023 0.931±0.0180.931± 0.018 FDA P 0.935±0.0220.935± 0.022 0.932±0.0270.932± 0.027 0.931±0.0240.931± 0.024 0.930±0.0250.930± 0.025 0.926±0.0250.926± 0.025 0.922±0.0230.922± 0.023 0.923±0.0250.923± 0.025 R 0.931±0.0210.931± 0.021 0.924±0.0250.924± 0.025 0.926±0.0230.926± 0.023 0.920±0.0220.920± 0.022 0.918±0.0230.918± 0.023 0.913±0.0240.913± 0.024 0.914±0.0250.914± 0.025 F1 0.933±0.0210.933± 0.021 0.928±0.0250.928± 0.025 0.929±0.0230.929± 0.023 0.925±0.0220.925± 0.022 0.922±0.0230.922± 0.023 0.918±0.0220.918± 0.022 0.918±0.0240.918± 0.024 SSPA P 0.932±0.0190.932± 0.019 0.932±0.0200.932± 0.020 0.924±0.0230.924± 0.023 0.929±0.0250.929± 0.025 0.920±0.0220.920± 0.022 0.918±0.0210.918± 0.021 0.922±0.0200.922± 0.020 R 0.921±0.0200.921± 0.020 0.923±0.0230.923± 0.023 0.915±0.0200.915± 0.020 0.915±0.0220.915± 0.022 0.914±0.0220.914± 0.022 0.909±0.0210.909± 0.021 0.910±0.0210.910± 0.021 F1 0.926±0.0180.926± 0.018 0.928±0.0200.928± 0.020 0.919±0.0200.919± 0.020 0.922±0.0220.922± 0.022 0.917±0.0200.917± 0.020 0.913±0.0190.913± 0.019 0.916±0.0190.916± 0.019 EGA P 0.893±0.0250.893± 0.025 0.878±0.0250.878± 0.025 0.878±0.0280.878± 0.028 0.852±0.0710.852± 0.071 0.864±0.0410.864± 0.041 0.841±0.0840.841± 0.084 0.852±0.0460.852± 0.046 R 0.900±0.0220.900± 0.022 0.889±0.0250.889± 0.025 0.887±0.0240.887± 0.024 0.880±0.0320.880± 0.032 0.882±0.0260.882± 0.026 0.874±0.0350.874± 0.035 0.879±0.0230.879± 0.023 F1 0.896±0.0220.896± 0.022 0.883±0.0220.883± 0.022 0.883±0.0240.883± 0.024 0.865±0.0520.865± 0.052 0.872±0.0310.872± 0.031 0.856±0.0610.856± 0.061 0.865±0.0340.865± 0.034 CE P 0.886±0.0180.886± 0.018 0.880±0.0200.880± 0.020 0.870±0.0330.870± 0.033 0.872±0.0440.872± 0.044 0.868±0.0390.868± 0.039 0.858±0.0460.858± 0.046 0.848±0.0620.848± 0.062 R 0.887±0.0180.887± 0.018 0.888±0.0180.888± 0.018 0.876±0.0250.876± 0.025 0.878±0.0260.878± 0.026 0.874±0.0240.874± 0.024 0.872±0.0220.872± 0.022 0.864±0.0330.864± 0.033 F1 0.886±0.0160.886± 0.016 0.884±0.0170.884± 0.017 0.873±0.0280.873± 0.028 0.875±0.0340.875± 0.034 0.871±0.0300.871± 0.030 0.865±0.0340.865± 0.034 0.856±0.0480.856± 0.048 SSGRA P 0.800±0.1080.800± 0.108 0.748±0.1510.748± 0.151 0.716±0.1380.716± 0.138 0.680±0.2140.680± 0.214 0.726±0.0540.726± 0.054 0.660±0.2000.660± 0.200 0.695±0.1180.695± 0.118 R 0.835±0.0670.835± 0.067 0.790±0.1440.790± 0.144 0.772±0.1370.772± 0.137 0.730±0.2220.730± 0.222 0.785±0.0330.785± 0.033 0.718±0.2140.718± 0.214 0.756±0.1260.756± 0.126 F1 0.816±0.0880.816± 0.088 0.768±0.1470.768± 0.147 0.743±0.1370.743± 0.137 0.704±0.2170.704± 0.217 0.754±0.0420.754± 0.042 0.687±0.2060.687± 0.206 0.724±0.1210.724± 0.121 (b) ROUGE-L Method Metric c=0.002c=0.002 c=0.0025c=0.0025 c=0.003c=0.003 c=0.0035c=0.0035 c=0.004c=0.004 c=0.0045c=0.0045 c=0.005c=0.005 BSA P 0.358±0.1310.358± 0.131 0.303±0.1350.303± 0.135 0.292±0.1080.292± 0.108 0.253±0.1330.253± 0.133 0.261±0.1280.261± 0.128 0.231±0.1080.231± 0.108 0.210±0.1290.210± 0.129 R 0.353±0.1160.353± 0.116 0.298±0.1110.298± 0.111 0.285±0.0960.285± 0.096 0.260±0.1160.260± 0.116 0.223±0.1070.223± 0.107 0.234±0.0880.234± 0.088 0.199±0.1120.199± 0.112 F1 0.352±0.1160.352± 0.116 0.296±0.1150.296± 0.115 0.285±0.0970.285± 0.097 0.252±0.1180.252± 0.118 0.232±0.1130.232± 0.113 0.228±0.0910.228± 0.091 0.199±0.1140.199± 0.114 DRA P 0.549±0.1850.549± 0.185 0.579±0.1850.579± 0.185 0.538±0.1740.538± 0.174 0.545±0.1770.545± 0.177 0.522±0.1500.522± 0.150 0.532±0.1710.532± 0.171 0.500±0.1400.500± 0.140 R 0.562±0.2020.562± 0.202 0.586±0.1750.586± 0.175 0.540±0.1790.540± 0.179 0.541±0.1660.541± 0.166 0.542±0.1630.542± 0.163 0.540±0.1560.540± 0.156 0.511±0.1390.511± 0.139 F1 0.551±0.1870.551± 0.187 0.577±0.1710.577± 0.171 0.536±0.1700.536± 0.170 0.539±0.1640.539± 0.164 0.528±0.1490.528± 0.149 0.533±0.1590.533± 0.159 0.501±0.1320.501± 0.132 FDA P 0.519±0.1630.519± 0.163 0.513±0.1710.513± 0.171 0.504±0.1620.504± 0.162 0.497±0.1660.497± 0.166 0.468±0.1490.468± 0.149 0.463±0.1400.463± 0.140 0.452±0.1590.452± 0.159 R 0.519±0.1620.519± 0.162 0.471±0.1480.471± 0.148 0.491±0.1560.491± 0.156 0.455±0.1450.455± 0.145 0.436±0.1500.436± 0.150 0.428±0.1430.428± 0.143 0.426±0.1360.426± 0.136 F1 0.517±0.1600.517± 0.160 0.486±0.1530.486± 0.153 0.494±0.1530.494± 0.153 0.471±0.1480.471± 0.148 0.447±0.1440.447± 0.144 0.440±0.1310.440± 0.131 0.434±0.1390.434± 0.139 SSPA P 0.536±0.1270.536± 0.127 0.524±0.1460.524± 0.146 0.464±0.1150.464± 0.115 0.486±0.1650.486± 0.165 0.447±0.1270.447± 0.127 0.432±0.1150.432± 0.115 0.465±0.1370.465± 0.137 R 0.472±0.1310.472± 0.131 0.476±0.1450.476± 0.145 0.421±0.1050.421± 0.105 0.421±0.1370.421± 0.137 0.423±0.1390.423± 0.139 0.393±0.1170.393± 0.117 0.405±0.1260.405± 0.126 F1 0.498±0.1220.498± 0.122 0.494±0.1360.494± 0.136 0.437±0.0990.437± 0.099 0.447±0.1420.447± 0.142 0.428±0.1190.428± 0.119 0.406±0.1070.406± 0.107 0.427±0.1210.427± 0.121 EGA P 0.327±0.1110.327± 0.111 0.296±0.0970.296± 0.097 0.304±0.1040.304± 0.104 0.300±0.1350.300± 0.135 0.291±0.1010.291± 0.101 0.327±0.1540.327± 0.154 0.271±0.0700.271± 0.070 R 0.352±0.1230.352± 0.123 0.316±0.1060.316± 0.106 0.305±0.1080.305± 0.108 0.296±0.1170.296± 0.117 0.280±0.0920.280± 0.092 0.263±0.0980.263± 0.098 0.292±0.0810.292± 0.081 F1 0.337±0.1130.337± 0.113 0.294±0.0860.294± 0.086 0.293±0.0970.293± 0.097 0.277±0.1000.277± 0.100 0.271±0.0750.271± 0.075 0.261±0.0770.261± 0.077 0.274±0.0650.274± 0.065 CE P 0.278±0.0730.278± 0.073 0.269±0.0690.269± 0.069 0.229±0.0790.229± 0.079 0.249±0.0810.249± 0.081 0.236±0.0770.236± 0.077 0.217±0.0740.217± 0.074 0.212±0.0840.212± 0.084 R 0.295±0.0810.295± 0.081 0.299±0.0700.299± 0.070 0.265±0.0880.265± 0.088 0.273±0.0890.273± 0.089 0.250±0.0720.250± 0.072 0.250±0.0720.250± 0.072 0.237±0.1000.237± 0.100 F1 0.282±0.0660.282± 0.066 0.277±0.0580.277± 0.058 0.239±0.0710.239± 0.071 0.256±0.0760.256± 0.076 0.235±0.0570.235± 0.057 0.226±0.0650.226± 0.065 0.217±0.0810.217± 0.081 SSGRA P 0.218±0.2520.218± 0.252 0.179±0.2810.179± 0.281 0.089±0.2290.089± 0.229 0.086±0.1890.086± 0.189 0.083±0.2460.083± 0.246 0.085±0.2430.085± 0.243 0.205±0.3720.205± 0.372 R 0.190±0.2300.190± 0.230 0.099±0.1740.099± 0.174 0.054±0.1630.054± 0.163 0.051±0.1560.051± 0.156 0.027±0.1170.027± 0.117 0.009±0.0450.009± 0.045 0.005±0.0100.005± 0.010 F1 0.196±0.2350.196± 0.235 0.102±0.1710.102± 0.171 0.057±0.1640.057± 0.164 0.052±0.1550.052± 0.155 0.028±0.1140.028± 0.114 0.010±0.0470.010± 0.047 0.010±0.0200.010± 0.020 Table 5: Performance of different attack methods on LLaVa-1.5 under varying perturbation budgets c. Top: BERTScore (Precision, Recall and F1, Mean± ). Bottom: ROUGE-L (Precision, Recall and F1, Mean± ). (a) BERTScore Method Metric c=0.002c=0.002 c=0.0025c=0.0025 c=0.003c=0.003 c=0.0035c=0.0035 c=0.004c=0.004 c=0.0045c=0.0045 c=0.005c=0.005 BSA P 0.925±0.0340.925± 0.034 0.909±0.0340.909± 0.034 0.901±0.0280.901± 0.028 0.893±0.0190.893± 0.019 0.891±0.0200.891± 0.020 0.883±0.0350.883± 0.035 0.885±0.0190.885± 0.019 R 0.926±0.0350.926± 0.035 0.914±0.0340.914± 0.034 0.904±0.0290.904± 0.029 0.897±0.0210.897± 0.021 0.893±0.0220.893± 0.022 0.888±0.0220.888± 0.022 0.884±0.0210.884± 0.021 F1 0.926±0.0340.926± 0.034 0.911±0.0340.911± 0.034 0.902±0.0280.902± 0.028 0.895±0.0190.895± 0.019 0.892±0.0200.892± 0.020 0.886±0.0280.886± 0.028 0.884±0.0190.884± 0.019 DRA P 0.950±0.0370.950± 0.037 0.940±0.0220.940± 0.022 0.935±0.0270.935± 0.027 0.934±0.0290.934± 0.029 0.926±0.0260.926± 0.026 0.914±0.0390.914± 0.039 0.917±0.0280.917± 0.028 R 0.953±0.0290.953± 0.029 0.939±0.0240.939± 0.024 0.936±0.0280.936± 0.028 0.931±0.0310.931± 0.031 0.926±0.0290.926± 0.029 0.915±0.0310.915± 0.031 0.914±0.0280.914± 0.028 F1 0.951±0.0330.951± 0.033 0.940±0.0220.940± 0.022 0.935±0.0270.935± 0.027 0.932±0.0300.932± 0.030 0.926±0.0270.926± 0.027 0.914±0.0330.914± 0.033 0.915±0.0280.915± 0.028 FDA P 0.954±0.0270.954± 0.027 0.946±0.0270.946± 0.027 0.946±0.0280.946± 0.028 0.945±0.0270.945± 0.027 0.939±0.0260.939± 0.026 0.943±0.0260.943± 0.026 0.938±0.0250.938± 0.025 R 0.953±0.0270.953± 0.027 0.946±0.0270.946± 0.027 0.945±0.0280.945± 0.028 0.944±0.0280.944± 0.028 0.938±0.0270.938± 0.027 0.942±0.0270.942± 0.027 0.940±0.0260.940± 0.026 F1 0.954±0.0270.954± 0.027 0.946±0.0260.946± 0.026 0.945±0.0270.945± 0.027 0.944±0.0270.944± 0.027 0.939±0.0260.939± 0.026 0.942±0.0260.942± 0.026 0.939±0.0250.939± 0.025 SSPA P 0.950±0.0280.950± 0.028 0.947±0.0250.947± 0.025 0.936±0.0210.936± 0.021 0.934±0.0230.934± 0.023 0.932±0.0260.932± 0.026 0.928±0.0220.928± 0.022 0.934±0.0260.934± 0.026 R 0.949±0.0270.949± 0.027 0.947±0.0250.947± 0.025 0.936±0.0220.936± 0.022 0.934±0.0250.934± 0.025 0.935±0.0260.935± 0.026 0.929±0.0240.929± 0.024 0.933±0.0270.933± 0.027 F1 0.949±0.0270.949± 0.027 0.947±0.0240.947± 0.024 0.936±0.0210.936± 0.021 0.934±0.0230.934± 0.023 0.933±0.0260.933± 0.026 0.928±0.0220.928± 0.022 0.934±0.0260.934± 0.026 EGA P 0.933±0.0320.933± 0.032 0.920±0.0230.920± 0.023 0.917±0.0270.917± 0.027 0.910±0.0280.910± 0.028 0.901±0.0290.901± 0.029 0.881±0.1320.881± 0.132 0.868±0.1370.868± 0.137 R 0.934±0.0310.934± 0.031 0.924±0.0250.924± 0.025 0.919±0.0270.919± 0.027 0.915±0.0270.915± 0.027 0.906±0.0280.906± 0.028 0.890±0.1310.890± 0.131 0.882±0.1300.882± 0.130 F1 0.933±0.0310.933± 0.031 0.922±0.0230.922± 0.023 0.918±0.0260.918± 0.026 0.913±0.0270.913± 0.027 0.903±0.0270.903± 0.027 0.885±0.1310.885± 0.131 0.875±0.1330.875± 0.133 CE P 0.919±0.0240.919± 0.024 0.915±0.0230.915± 0.023 0.909±0.0220.909± 0.022 0.907±0.0190.907± 0.019 0.906±0.0210.906± 0.021 0.887±0.1290.887± 0.129 0.880±0.1300.880± 0.130 R 0.923±0.0210.923± 0.021 0.916±0.0220.916± 0.022 0.912±0.0200.912± 0.020 0.911±0.0190.911± 0.019 0.909±0.0180.909± 0.018 0.889±0.1300.889± 0.130 0.886±0.1300.886± 0.130 F1 0.921±0.0220.921± 0.022 0.916±0.0220.916± 0.022 0.911±0.0200.911± 0.020 0.909±0.0180.909± 0.018 0.907±0.0190.907± 0.019 0.888±0.1290.888± 0.129 0.883±0.1300.883± 0.130 SSGRA P 0.915±0.0290.915± 0.029 0.911±0.0280.911± 0.028 0.879±0.1340.879± 0.134 0.834±0.2170.834± 0.217 0.852±0.1790.852± 0.179 0.815±0.2140.815± 0.214 0.811±0.2160.811± 0.216 R 0.917±0.0290.917± 0.029 0.913±0.0280.913± 0.028 0.885±0.1330.885± 0.133 0.839±0.2160.839± 0.216 0.855±0.1790.855± 0.179 0.824±0.2130.824± 0.213 0.821±0.2130.821± 0.213 F1 0.916±0.0290.916± 0.029 0.912±0.0270.912± 0.027 0.882±0.1330.882± 0.133 0.837±0.2160.837± 0.216 0.853±0.1790.853± 0.179 0.819±0.2130.819± 0.213 0.816±0.2140.816± 0.214 (b) ROUGE-L Method Metric c=0.002c=0.002 c=0.0025c=0.0025 c=0.003c=0.003 c=0.0035c=0.0035 c=0.004c=0.004 c=0.0045c=0.0045 c=0.005c=0.005 BSA P 0.495±0.2140.495± 0.214 0.408±0.1950.408± 0.195 0.363±0.1360.363± 0.136 0.319±0.0740.319± 0.074 0.320±0.1000.320± 0.100 0.290±0.0750.290± 0.075 0.307±0.1180.307± 0.118 R 0.495±0.2190.495± 0.219 0.432±0.2040.432± 0.204 0.383±0.1450.383± 0.145 0.350±0.1050.350± 0.105 0.331±0.1200.331± 0.120 0.312±0.0930.312± 0.093 0.299±0.0720.299± 0.072 F1 0.492±0.2150.492± 0.215 0.416±0.1960.416± 0.196 0.369±0.1350.369± 0.135 0.330±0.0800.330± 0.080 0.320±0.1030.320± 0.103 0.297±0.0770.297± 0.077 0.291±0.0650.291± 0.065 DRA P 0.638±0.2100.638± 0.210 0.553±0.1620.553± 0.162 0.527±0.1790.527± 0.179 0.526±0.1920.526± 0.192 0.486±0.1540.486± 0.154 0.441±0.1630.441± 0.163 0.448±0.1660.448± 0.166 R 0.644±0.2170.644± 0.217 0.542±0.1740.542± 0.174 0.529±0.1930.529± 0.193 0.507±0.2140.507± 0.214 0.477±0.1660.477± 0.166 0.422±0.1750.422± 0.175 0.425±0.1560.425± 0.156 F1 0.639±0.2120.639± 0.212 0.545±0.1640.545± 0.164 0.525±0.1830.525± 0.183 0.514±0.2010.514± 0.201 0.477±0.1520.477± 0.152 0.426±0.1640.426± 0.164 0.432±0.1560.432± 0.156 FDA P 0.651±0.1890.651± 0.189 0.598±0.1830.598± 0.183 0.599±0.2030.599± 0.203 0.588±0.1900.588± 0.190 0.556±0.1740.556± 0.174 0.580±0.1840.580± 0.184 0.553±0.1810.553± 0.181 R 0.642±0.2040.642± 0.204 0.592±0.1910.592± 0.191 0.586±0.2060.586± 0.206 0.588±0.1990.588± 0.199 0.550±0.1930.550± 0.193 0.567±0.1920.567± 0.192 0.571±0.1910.571± 0.191 F1 0.644±0.1950.644± 0.195 0.591±0.1820.591± 0.182 0.590±0.2030.590± 0.203 0.585±0.1920.585± 0.192 0.550±0.1820.550± 0.182 0.570±0.1840.570± 0.184 0.558±0.1820.558± 0.182 SSPA P 0.632±0.2020.632± 0.202 0.598±0.1800.598± 0.180 0.531±0.1460.531± 0.146 0.527±0.1500.527± 0.150 0.521±0.1690.521± 0.169 0.490±0.1450.490± 0.145 0.525±0.1680.525± 0.168 R 0.615±0.2070.615± 0.207 0.599±0.1850.599± 0.185 0.524±0.1550.524± 0.155 0.520±0.1720.520± 0.172 0.524±0.1780.524± 0.178 0.483±0.1580.483± 0.158 0.520±0.1870.520± 0.187 F1 0.620±0.2020.620± 0.202 0.594±0.1770.594± 0.177 0.524±0.1440.524± 0.144 0.520±0.1540.520± 0.154 0.520±0.1700.520± 0.170 0.483±0.1470.483± 0.147 0.518±0.1730.518± 0.173 EGA P 0.513±0.2060.513± 0.206 0.449±0.1400.449± 0.140 0.437±0.1660.437± 0.166 0.407±0.1560.407± 0.156 0.377±0.1470.377± 0.147 0.373±0.1470.373± 0.147 0.341±0.1140.341± 0.114 R 0.515±0.2050.515± 0.205 0.475±0.1590.475± 0.159 0.435±0.1750.435± 0.175 0.434±0.1570.434± 0.157 0.383±0.1460.383± 0.146 0.392±0.1570.392± 0.157 0.350±0.1400.350± 0.140 F1 0.511±0.2020.511± 0.202 0.456±0.1380.456± 0.138 0.427±0.1620.427± 0.162 0.416±0.1510.416± 0.151 0.371±0.1380.371± 0.138 0.377±0.1430.377± 0.143 0.336±0.1170.336± 0.117 CE P 0.432±0.1430.432± 0.143 0.405±0.1430.405± 0.143 0.383±0.1280.383± 0.128 0.381±0.1090.381± 0.109 0.360±0.1030.360± 0.103 0.336±0.0920.336± 0.092 0.329±0.1040.329± 0.104 R 0.458±0.1400.458± 0.140 0.406±0.1490.406± 0.149 0.379±0.1330.379± 0.133 0.379±0.1030.379± 0.103 0.362±0.1080.362± 0.108 0.354±0.1050.354± 0.105 0.342±0.1160.342± 0.116 F1 0.439±0.1340.439± 0.134 0.400±0.1370.400± 0.137 0.375±0.1220.375± 0.122 0.375±0.0940.375± 0.094 0.357±0.0960.357± 0.096 0.340±0.0880.340± 0.088 0.329±0.1000.329± 0.100 SSGRA P 0.421±0.1680.421± 0.168 0.402±0.1560.402± 0.156 0.368±0.1820.368± 0.182 0.309±0.1450.309± 0.145 0.311±0.1420.311± 0.142 0.278±0.1570.278± 0.157 0.259±0.1520.259± 0.152 R 0.432±0.1780.432± 0.178 0.416±0.1640.416± 0.164 0.374±0.1960.374± 0.196 0.310±0.1600.310± 0.160 0.303±0.1420.303± 0.142 0.266±0.1410.266± 0.141 0.236±0.1520.236± 0.152 F1 0.422±0.1690.422± 0.169 0.406±0.1570.406± 0.157 0.364±0.1840.364± 0.184 0.299±0.1380.299± 0.138 0.297±0.1320.297± 0.132 0.252±0.1210.252± 0.121 0.236±0.1380.236± 0.138 Table 6: Performance of different attack methods on Gemma 3 under varying perturbation budgets c. Top: BERTScore (Precision, Recall and F1, Mean± ). Bottom: ROUGE-L (Precision, Recall and F1, Mean± ). (a) BERTScore Method Metric c=0.002c=0.002 c=0.0025c=0.0025 c=0.003c=0.003 c=0.0035c=0.0035 c=0.004c=0.004 c=0.0045c=0.0045 c=0.005c=0.005 BSA P 0.904±0.0370.904± 0.037 0.890±0.0350.890± 0.035 0.879±0.0370.879± 0.037 0.870±0.0470.870± 0.047 0.863±0.0300.863± 0.030 0.860±0.0270.860± 0.027 0.852±0.0400.852± 0.040 R 0.907±0.0360.907± 0.036 0.890±0.0360.890± 0.036 0.880±0.0360.880± 0.036 0.871±0.0400.871± 0.040 0.859±0.0350.859± 0.035 0.852±0.0370.852± 0.037 0.843±0.0390.843± 0.039 F1 0.905±0.0350.905± 0.035 0.890±0.0340.890± 0.034 0.880±0.0350.880± 0.035 0.870±0.0430.870± 0.043 0.861±0.0310.861± 0.031 0.856±0.0310.856± 0.031 0.847±0.0370.847± 0.037 DRA P 0.918±0.0300.918± 0.030 0.918±0.0270.918± 0.027 0.915±0.0300.915± 0.030 0.910±0.0270.910± 0.027 0.907±0.0310.907± 0.031 0.909±0.0290.909± 0.029 0.903±0.0330.903± 0.033 R 0.920±0.0280.920± 0.028 0.918±0.0270.918± 0.027 0.914±0.0320.914± 0.032 0.912±0.0250.912± 0.025 0.909±0.0290.909± 0.029 0.909±0.0300.909± 0.030 0.904±0.0310.904± 0.031 F1 0.919±0.0270.919± 0.027 0.918±0.0250.918± 0.025 0.914±0.0290.914± 0.029 0.911±0.0250.911± 0.025 0.908±0.0290.908± 0.029 0.909±0.0280.909± 0.028 0.903±0.0300.903± 0.030 FDA P 0.928±0.0370.928± 0.037 0.925±0.0360.925± 0.036 0.924±0.0320.924± 0.032 0.920±0.0340.920± 0.034 0.923±0.0310.923± 0.031 0.921±0.0300.921± 0.030 0.918±0.0270.918± 0.027 R 0.926±0.0380.926± 0.038 0.924±0.0400.924± 0.040 0.923±0.0350.923± 0.035 0.919±0.0370.919± 0.037 0.922±0.0340.922± 0.034 0.921±0.0320.921± 0.032 0.918±0.0280.918± 0.028 F1 0.927±0.0360.927± 0.036 0.924±0.0370.924± 0.037 0.923±0.0320.923± 0.032 0.919±0.0340.919± 0.034 0.923±0.0320.923± 0.032 0.921±0.0300.921± 0.030 0.918±0.0260.918± 0.026 SSPA P 0.924±0.0240.924± 0.024 0.917±0.0280.917± 0.028 0.919±0.0300.919± 0.030 0.913±0.0300.913± 0.030 0.915±0.0300.915± 0.030 0.907±0.0270.907± 0.027 0.911±0.0300.911± 0.030 R 0.923±0.0240.923± 0.024 0.919±0.0300.919± 0.030 0.915±0.0370.915± 0.037 0.913±0.0300.913± 0.030 0.912±0.0330.912± 0.033 0.905±0.0310.905± 0.031 0.907±0.0350.907± 0.035 F1 0.923±0.0220.923± 0.022 0.918±0.0280.918± 0.028 0.916±0.0320.916± 0.032 0.913±0.0290.913± 0.029 0.913±0.0310.913± 0.031 0.905±0.0270.905± 0.027 0.909±0.0310.909± 0.031 EGA P 0.910±0.0300.910± 0.030 0.919±0.0300.919± 0.030 0.907±0.0310.907± 0.031 0.906±0.0300.906± 0.030 0.900±0.0310.900± 0.031 0.906±0.0310.906± 0.031 0.901±0.0310.901± 0.031 R 0.913±0.0300.913± 0.030 0.918±0.0340.918± 0.034 0.910±0.0320.910± 0.032 0.908±0.0330.908± 0.033 0.902±0.0340.902± 0.034 0.909±0.0320.909± 0.032 0.904±0.0310.904± 0.031 F1 0.911±0.0290.911± 0.029 0.918±0.0310.918± 0.031 0.908±0.0300.908± 0.030 0.907±0.0310.907± 0.031 0.901±0.0310.901± 0.031 0.908±0.0300.908± 0.030 0.902±0.0300.902± 0.030 CE P 0.886±0.0270.886± 0.027 0.882±0.0260.882± 0.026 0.879±0.0310.879± 0.031 0.877±0.0290.877± 0.029 0.876±0.0380.876± 0.038 0.881±0.0390.881± 0.039 0.864±0.0520.864± 0.052 R 0.887±0.0280.887± 0.028 0.883±0.0280.883± 0.028 0.879±0.0320.879± 0.032 0.881±0.0270.881± 0.027 0.878±0.0310.878± 0.031 0.881±0.0320.881± 0.032 0.872±0.0340.872± 0.034 F1 0.886±0.0260.886± 0.026 0.883±0.0250.883± 0.025 0.879±0.0300.879± 0.030 0.879±0.0260.879± 0.026 0.877±0.0330.877± 0.033 0.881±0.0340.881± 0.034 0.867±0.0410.867± 0.041 SSGRA P 0.902±0.0380.902± 0.038 0.883±0.0460.883± 0.046 0.874±0.0360.874± 0.036 0.869±0.0310.869± 0.031 0.860±0.0350.860± 0.035 0.857±0.0390.857± 0.039 0.855±0.0260.855± 0.026 R 0.903±0.0390.903± 0.039 0.883±0.0400.883± 0.040 0.873±0.0390.873± 0.039 0.867±0.0360.867± 0.036 0.856±0.0380.856± 0.038 0.853±0.0400.853± 0.040 0.842±0.0370.842± 0.037 F1 0.903±0.0380.903± 0.038 0.883±0.0420.883± 0.042 0.873±0.0360.873± 0.036 0.868±0.0320.868± 0.032 0.858±0.0340.858± 0.034 0.855±0.0370.855± 0.037 0.849±0.0300.849± 0.030 (b) ROUGE-L Method Metric c=0.002c=0.002 c=0.0025c=0.0025 c=0.003c=0.003 c=0.0035c=0.0035 c=0.004c=0.004 c=0.0045c=0.0045 c=0.005c=0.005 BSA P 0.403±0.1520.403± 0.152 0.332±0.1140.332± 0.114 0.320±0.1080.320± 0.108 0.307±0.1190.307± 0.119 0.275±0.0990.275± 0.099 0.274±0.0850.274± 0.085 0.267±0.1030.267± 0.103 R 0.415±0.1350.415± 0.135 0.337±0.1200.337± 0.120 0.318±0.1200.318± 0.120 0.286±0.1220.286± 0.122 0.246±0.1060.246± 0.106 0.222±0.1050.222± 0.105 0.213±0.1190.213± 0.119 F1 0.406±0.1420.406± 0.142 0.331±0.1160.331± 0.116 0.315±0.1120.315± 0.112 0.287±0.1160.287± 0.116 0.253±0.0980.253± 0.098 0.238±0.0920.238± 0.092 0.228±0.1090.228± 0.109 DRA P 0.451±0.1480.451± 0.148 0.455±0.1240.455± 0.124 0.430±0.1320.430± 0.132 0.395±0.1290.395± 0.129 0.383±0.1380.383± 0.138 0.401±0.1310.401± 0.131 0.365±0.1280.365± 0.128 R 0.468±0.1390.468± 0.139 0.465±0.1240.465± 0.124 0.446±0.1350.446± 0.135 0.422±0.1150.422± 0.115 0.398±0.1310.398± 0.131 0.410±0.1290.410± 0.129 0.373±0.1300.373± 0.130 F1 0.456±0.1400.456± 0.140 0.457±0.1210.457± 0.121 0.433±0.1310.433± 0.131 0.405±0.1200.405± 0.120 0.388±0.1340.388± 0.134 0.402±0.1260.402± 0.126 0.365±0.1250.365± 0.125 FDA P 0.520±0.1870.520± 0.187 0.514±0.1800.514± 0.180 0.499±0.1620.499± 0.162 0.472±0.1590.472± 0.159 0.481±0.1390.481± 0.139 0.473±0.1580.473± 0.158 0.449±0.1230.449± 0.123 R 0.519±0.1880.519± 0.188 0.513±0.1910.513± 0.191 0.496±0.1700.496± 0.170 0.477±0.1580.477± 0.158 0.482±0.1400.482± 0.140 0.484±0.1530.484± 0.153 0.457±0.1270.457± 0.127 F1 0.516±0.1860.516± 0.186 0.509±0.1860.509± 0.186 0.492±0.1660.492± 0.166 0.471±0.1570.471± 0.157 0.478±0.1390.478± 0.139 0.474±0.1520.474± 0.152 0.450±0.1210.450± 0.121 SSPA P 0.463±0.1250.463± 0.125 0.434±0.1330.434± 0.133 0.440±0.1400.440± 0.140 0.412±0.1370.412± 0.137 0.424±0.1330.424± 0.133 0.385±0.1290.385± 0.129 0.405±0.1270.405± 0.127 R 0.469±0.1140.469± 0.114 0.453±0.1360.453± 0.136 0.450±0.1510.450± 0.151 0.429±0.1330.429± 0.133 0.425±0.1270.425± 0.127 0.397±0.1300.397± 0.130 0.409±0.1290.409± 0.129 F1 0.464±0.1180.464± 0.118 0.441±0.1310.441± 0.131 0.440±0.1410.440± 0.141 0.418±0.1340.418± 0.134 0.422±0.1290.422± 0.129 0.388±0.1280.388± 0.128 0.404±0.1260.404± 0.126 EGA P 0.416±0.1200.416± 0.120 0.458±0.1440.458± 0.144 0.409±0.1280.409± 0.128 0.409±0.1250.409± 0.125 0.377±0.1330.377± 0.133 0.417±0.1310.417± 0.131 0.393±0.1290.393± 0.129 R 0.437±0.1250.437± 0.125 0.473±0.1510.473± 0.151 0.412±0.1270.412± 0.127 0.417±0.1420.417± 0.142 0.392±0.1240.392± 0.124 0.431±0.1510.431± 0.151 0.392±0.1130.392± 0.113 F1 0.423±0.1180.423± 0.118 0.463±0.1440.463± 0.144 0.408±0.1250.408± 0.125 0.410±0.1280.410± 0.128 0.382±0.1270.382± 0.127 0.420±0.1310.420± 0.131 0.390±0.1190.390± 0.119 CE P 0.294±0.0820.294± 0.082 0.289±0.0770.289± 0.077 0.273±0.0770.273± 0.077 0.283±0.0890.283± 0.089 0.279±0.0950.279± 0.095 0.303±0.0980.303± 0.098 0.268±0.0850.268± 0.085 R 0.291±0.0830.291± 0.083 0.291±0.0770.291± 0.077 0.272±0.0850.272± 0.085 0.277±0.0820.277± 0.082 0.272±0.0860.272± 0.086 0.282±0.0850.282± 0.085 0.261±0.0910.261± 0.091 F1 0.289±0.0800.289± 0.080 0.285±0.0710.285± 0.071 0.269±0.0790.269± 0.079 0.275±0.0820.275± 0.082 0.273±0.0880.273± 0.088 0.285±0.0810.285± 0.081 0.259±0.0850.259± 0.085 SSGRA P 0.402±0.1410.402± 0.141 0.326±0.1220.326± 0.122 0.297±0.1190.297± 0.119 0.294±0.1080.294± 0.108 0.284±0.0980.284± 0.098 0.273±0.0890.273± 0.089 0.269±0.0960.269± 0.096 R 0.417±0.1320.417± 0.132 0.325±0.1250.325± 0.125 0.288±0.1230.288± 0.123 0.265±0.1120.265± 0.112 0.248±0.1110.248± 0.111 0.232±0.1030.232± 0.103 0.201±0.1020.201± 0.102 F1 0.407±0.1360.407± 0.136 0.319±0.1180.319± 0.118 0.288±0.1190.288± 0.119 0.272±0.1050.272± 0.105 0.259±0.1040.259± 0.104 0.241±0.0880.241± 0.088 0.221±0.0930.221± 0.093 0.A.4 Additional Qualitative Results (a) Qwen2.5-VL (b) LLaVa-1.5 (c) Gemma 3 Figure 12: Additional qualitative adversarial examples (set 1) generated with a perturbation budget of c=0.002c=0.002 across models. (a) Qwen2.5-VL (b) LLaVA-1.5 (c) Gemma 3 Figure 13: Additional qualitative adversarial examples (set 2) generated with a perturbation budget of c=0.002c=0.002 across models. (a) Qwen2.5-VL (b) LLaVA-1.5 (c) Gemma 3 Figure 14: Additional qualitative adversarial examples (set 3) generated with a perturbation budget of c=0.002c=0.002 across models. (a) Qwen2.5-VL (b) LLaVA-1.5 (c) Gemma 3 Figure 15: Additional qualitative adversarial examples (set 4) generated with a perturbation budget of c=0.003c=0.003 across models. Figures 12–15 present additional qualitative examples for Qwen2.5-VL, LLaVA-1.5, and Gemma 3, generated with a perturbation budget of c=0.002c=0.002 in Figures 12–14 and c=0.003c=0.003 in Figure 15. SSGRA produces larger semantic deviations than the baseline attacks while maintaining the visual appearance of the input images. Whereas baseline methods often preserve the correct semantic content or introduce only minor hallucinations, SSGRA more frequently induces incorrect object categories, unrelated scene descriptions, and semantically inconsistent responses. These qualitative results are consistent with the quantitative improvements reported in Section 5.2 and further support the effectiveness of spectral subspace guidance.