Paper deep dive
Feature-Space Adversarial Robustness Certification for Multimodal Large Language Models
Song Xia, Meiwen Ding, Chenqi Kong, Wenhan Yang, Xudong Jiang
Models: CLIP-L14, LLaVA
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/11/2026, 1:05:28 AM
Summary
The paper introduces Feature-space Smoothing (FS), a framework for providing certified adversarial robustness guarantees for Multimodal Large Language Models (MLLMs) at the feature representation level. It establishes a Feature Cosine Similarity Bound (FCSB) based on the Gaussian robustness of the encoder and proposes the Gaussian Smoothness Booster (GSB), a plug-and-play module consisting of a denoiser and a residual smoothness mapper to enhance this robustness without retraining the MLLM.
Entities (4)
Relation Signals (3)
Feature Cosine Similarity Bound → determinedby → Gaussian robustness score
confidence 98% · the value of this Feature Cosine Similarity Bound (FCSB) is determined by the intrinsic Gaussian robustness score of the given encoder.
Gaussian Smoothness Booster → enhances → Feature Cosine Similarity Bound
confidence 95% · GSB... enhances the Gaussian robustness score of pretrained MLLMs, thereby strengthening the robustness guaranteed by FS
Feature-space Smoothing → providescertifiedrobustnessfor → Multimodal Large Language Models
confidence 95% · FS, a general framework that provides certified robustness guarantees at the feature representation level of MLLMs.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal large language models (MLLMs) exhibit strong capabilities across diverse applications, yet remain vulnerable to adversarial perturbations that distort their feature representations and induce erroneous predictions. To address this vulnerability, we propose Feature-space Smoothing (FS), a general framework that provides certified robustness guarantees at the feature representation level of MLLMs. We theoretically prove that FS converts a given feature extractor into a smoothed variant that is guaranteed a certified lower bound on the cosine similarity between clean and adversarial features under $\ell_2$-bounded perturbations. Moreover, we establish that the value of this Feature Cosine Similarity Bound (FCSB) is determined by the intrinsic Gaussian robustness score of the given encoder. Building on this insight, we introduce the Gaussian Smoothness Booster (GSB), a plug-and-play module that enhances the Gaussian robustness score of pretrained MLLMs, thereby strengthening the robustness guaranteed by FS, without requiring additional MLLM retraining. Extensive experiments demonstrate that applying the FS to various MLLMs yields strong certified feature-space robustness and consistently leads to robust task-oriented performance across diverse applications.
Tags
Links
- Source: https://arxiv.org/abs/2601.16200
- Canonical: https://arxiv.org/abs/2601.16200
Trouble viewing inline? Open PDF directly →
Full Text
77,715 characters extracted from source content.
Expand or collapse full text
Feature-Space Adversarial Robustness Certification for Multimodal Large Language Models Song Xia Meiwen Ding Chenqi Kong Wenhan Yang Xudong Jiang Abstract Multimodal large language models (MLLMs) exhibit strong capabilities across diverse applications, yet remain vulnerable to adversarial perturbations that distort their feature representations and induce erroneous predictions. To address this vulnerability, we propose Feature-space Smoothing (FS), a general framework that provides certified robustness guarantees at the feature representation level of MLLMs. We theoretically prove that FS converts a given feature extractor into a smoothed variant that is guaranteed a certified lower bound on the cosine similarity between clean and adversarial features under ℓ2 _2-bounded perturbations. Moreover, we establish that the value of this Feature Cosine Similarity Bound (FCSB) is determined by the intrinsic Gaussian robustness score of the given encoder. Building on this insight, we introduce the Gaussian Smoothness Booster (GSB), a plug-and-play module that enhances the Gaussian robustness score of pretrained MLLMs, thereby strengthening the robustness guaranteed by FS, without requiring additional MLLM retraining. Extensive experiments demonstrate that applying the FS to various MLLMs yields strong certified feature-space robustness and consistently leads to robust task-oriented performance across diverse applications. Machine Learning, ICML 1 Introduction The emergence of the Multimodal Large Language Models (MLLMs), such as GPT-5 (OpenAI, 2025), Gemini 3 Pro (DeepMind, 2025), and Claude Sonnet 4.5 (Anthropic, 2025), has fundamentally reshaped existing working paradigms and significantly advanced societal productivity. Despite their remarkable capabilities across a broad spectrum of real-world tasks, these models still encounter critical safety challenges, such as adversarial vulnerabilities (Zhao et al., 2023; Cui et al., 2024; Li et al., 2025b; Jia et al., 2025). Figure 1: Illustration of the FS-GSB, which guarantees that the cosine similarity of the adversarial and clean features extracted by MLLM’s encoder is larger than FCSB for robust predictions. Adversaries can manipulate predictions of the MLLMs to a malicious state by injecting subtle and imperceptible perturbations to the clean inputs, exploiting the models’ insufficient local smoothness and uncontrolled Lipschitz continuity (Goodfellow et al., 2014; Cohen et al., 2019; Hein and Andriushchenko, 2017; Xia et al., 2024b). Countermeasures towards those threats can be roughly classified into empirical defense and certified defense. Typical empirical approaches include adversarial training (Madry et al., 2018; Rebuffi et al., 2021; Wang et al., 2024c; Xhonneux et al., 2024; Casper et al., 2024; Schlarmann et al., 2024; Malik et al., 2025; Mao et al., 2023) and input purification (Nie et al., 2022; Yoon et al., 2021; Lee and Kim, 2023; Lei et al., 2025; Zollicoffer et al., 2025). Despite their demonstrated empirical effectiveness, these approaches lack formal robustness guarantees and remain susceptible to stronger adversaries (Tramer et al., 2020; Chen et al., 2023; Wu et al., 2020; Olivier and Raj, 2023). Moreover, the multimodal nature of MLLMs also poses a great challenge for existing adversarial training methods. Since they accept heterogeneous inputs across diverse domains, ensure a robust encoder via adversarial training that can generalize to various scenarios is challenging and computationally costly. In contrast, certified approaches aim to guarantee that the model returns a constant prediction result within a certain range, usually a ℓ2 _2 or ℓ∞ _∞-norm constrained area (Raghunathan et al., 2018; Wong and Kolter, 2018; Hao et al., 2022; Kakizaki et al., 2023; Cohen et al., 2019; Xia et al., 2024b; Salman et al., 2019). However, most previous certified defense approaches (Cohen et al., 2019; Xia et al., 2024b; Wang et al., 2024b; Hao et al., 2022; Salman et al., 2019) predominantly assume the prediction of one-dimensional outputs (e.g., class label), thereby limiting their applicability to more general tasks on MLLMs. To address the aforementioned challenges, we propose the Feature-space Smoothing (FS), a provable defense method that offers certified robustness on the feature representations of MLLMs. Specifically, FS smooths the vanilla feature encoder of the MLLM, and the resulted smoothed encoder is guaranteed to maintain a provable lower bound on the cosine similarity between clean and adversarial representations under ℓ2 _2-norm bounded attacks. Moreover, we demonstrate that this Feature Cosine Similarity Bound (FCSB) of the smoothed feature encoder is intrinsically determined by a defined Gaussian robustness score, which measures the prediction consistency of the vanilla feature extractor under Gaussian noise. Nonetheless, the Gaussian robustness of these MLLMs remains limited without Gaussian noise augmented training, leading to a correspondingly suboptimal FCSB. To address this whiling avoiding the high computational cost of fine-tuning MLLMs, we propose a plug-and-play Gaussian Smoothness Booster (GSB), consisting of a lightweight Gaussian denoiser and a residual smoothness mapper. Specifically, the denoiser is applied prior to the feature encoder and is trained to mitigates Gaussian noises and improves the resulting robustness score. In parallel, the smoothness mapper operates after feature extraction for feature-level refinement. It is trained to preserve the original feature distribution while further enhancing the Gaussian robustness score. These two components of GSB work synergistically to enhance the certified robustness of MLLMs provided by FS. The GSB is optimized using the proposed utility–robustness loss and trained on data from diverse visual domains to enhance the Gaussian robustness while preserving the feature utility for the encoder. To comprehensively assess the performance, we evaluate our method against state-of-the-art (SOTA) adversarial attacks tailored for MLLMs under the white-box setting and compare it with advanced adversarial training methods. Extensive experimental results demonstrate that our FS not only provides a strong certified robustness guarantee on the feature representation but also exhibits superior adversarial robustness for various MLLMs under diverse downstream tasks. Overall, the contributions of this work are: • We propose the Feature-space Smoothing (FS) to turn any feature encoder into a smoothed version, and theoretically prove that the smoothed encoder maintains a certified lower bound on the feature cosine similarity between clean and adversarial representations. • We propose the Gaussian Smoothness Booster (PSB) that effectively enhances the FCSB of the smoothed encoder in MLLMs via a plug-and-play manner. • We conduct extensive experiments demonstrating that integrating our proposed FS-GSB greatly enhances the adversarial robustness of various MLLMs and dramatically reduces the ASR under various white-box attacks. 2 Related Work Adversarial attacks on MLLMs. While MLLMs continue to achieve remarkable performance across diverse applications, extensive works (Qi et al., 2024; Cui et al., 2024; Zhao et al., 2023; Jia et al., 2025; Li et al., 2025b; Wang et al., 2024a; Zhang et al., 2024, 2025; Xie et al., 2025) have exposed their adversarial vulnerabilities, raising serious safety concerns. Early work, such as AttackVLM (Zhao et al., 2023), explores transferable attacks by disrupting the feature representations of CLIP (Radford et al., 2021) and BLIP (Li et al., 2023a), showing strong adversarial transferability among open-source models but limited effectiveness against closed-source commercial systems. More recent approaches, such as M-Attack (Li et al., 2025b) and FOA-Attack (Jia et al., 2025), further advance this direction by leveraging multi-extractor ensembles and feature-space alignment, achieving over 90% targeted attack success rates on image-captioning tasks against powerful closed-sourced commercial MLLMs (e.g., ChatGPT-4o). This highlights an urgent need for trustworthy defenses that provide effective and provable protection for MLLMs. Adversarial defense on MLLMs. Adversarial defense methods can be broadly classified into empirical and provable approaches. Empirical defenses for MLLMs mainly include adversarial training (Madry et al., 2018; Rebuffi et al., 2021; Wang et al., 2024c; Xhonneux et al., 2024; Casper et al., 2024; Schlarmann et al., 2024; Malik et al., 2025; Mao et al., 2023), which enhances robustness by augmenting training data with adversarial examples, and input purification (Nie et al., 2022; Lei et al., 2025; Li et al., 2025a), which employs generative mechanisms such as diffusion models or autoencoders to recover clean inputs prior to inference. Recent studies (Malik et al., 2025; Mao et al., 2023; Schlarmann et al., 2024) have revealed that utilization of adversarially trained CLIP-feature encoders can enhance adversarial robustness for MLLMs. However, these methods lack formal robustness guarantees and remain vulnerable to adaptive and unseen threats (Tramer et al., 2020; Chen et al., 2023; Wu et al., 2020). Moreover, adversarial training demands costly retraining and often leads to degradation in clean performance. Alternatively, certified defenses aim to provide mathematically provable robustness guarantees. The use of Gaussian smoothing for certified robustness was initially introduced for classification models (Cohen et al., 2019; Li et al., 2018; Lecuyer et al., 2019), yet its theoretical formulation is restricted to one-dimensional outputs, limiting its applicability to tasks such as auto-regression or multimodal generation. To overcome these limitations and ensure trustworthy protection for MLLMs, we propose the Feature-space Smoothing (FS), a defense mechanism that establishes provable adversarial feature robustness for MLLMs. 3 Feature-Space Smoothing 3.1 Preliminary Let ℱF denote a general deep learning model consisting of a feature extractor fe:→f_e: x→ z that maps the input x to a feature representation z, and a predictor fd:→f_d: z→ y that produces the final output y. Let ℒL denote the general loss function (e.g., cross-entropy) that measures the discrepancy between the model’s output and the ground truth. Adversarial attacks: Let ℬϵ()=′:‖′−‖p≤ϵB_ε( x)= \ x : \| x - x \|_p≤ε \ be an ℓp _p-norm ball centered at the input x, where ϵε is a pre-defined perturbation bound. For each input x, the adversarial attacks aim to find an adversarial input ′=+ x = x+ δ that misleads the model by solving: max+∈ℬϵ()ℒ(ℱ(),ℱ(+)). -2.84526pt _ x+ δ _ε( x)L (F ( x ),F ( x+ δ ) ). (1) Adversarial effects on feature representations: While adversarial attacks primarily aim to alter the model’s predictions, numerous studies (Jia et al., 2025; Xia et al., 2024a; Li et al., 2020; Huang et al., 2019; Li et al., 2023b; Ding et al., 2024) have shown that successful attacks typically induce substantial distortions in the model’s feature representations. Let ′ x denote the adversarial example with adversarial feature ′ z . Let t z^t be the adversarial targeted feature with malicious semantic meaning. The attack generally leads to maxℒ(′,) ( z , z ) for untargeted attacks and minℒ(′,t) ( z , z^t ) for targeted attacks. Thus, ensuring a robust feature encoder that minℒ(′,) ( z , z ) is crucial for the trustworthy prediction. Randomized smoothing: Consider a k classes classification problem with the input ∈ℝd x∈R^d and the label ∈=c1,…,ck y = \c_1,…,c_k \. Randomized Smoothing (RS) first corrupts each input x by adding the Gaussian noise ∼(0,σ2) (0,σ^2 I). Then it turns an arbitrary base classifier ℱF into a smoothed version ℱ F that possesses ℓ2 _2 certified robustness guarantees. The smoothed classifier ℱ F returns whichever class the base classifier ℱF is most likely to return among the distribution +∼(,σ2) x+ ( x,σ^2 I), which is: ℱ^()=argmaxc∈ℙ(ℱ(+)=c). array[]l F( x)= _c P(F( x+ )=c). -2.84526pt array (2) RS then guarantees a certified radius ℛR for this smoothed classifier ℱ F. For any perturbation δ satisfying ‖δ‖2≤ℛ \|δ \|_2 , the smoothed classifier is guaranteed to return a robust prediction that makes ℱ^(+δ)=F(x) F( x+δ)=F(x). Limitations for RS. While previous RS provides effective certified protection for classification models, it suffers from two limitations. First, the theoretical framework of RS inherently restricts its certification to classification tasks (e.g.possibility of input x belongs to a certain class c). Second, estimating ℙ(ℱ(+)=c)P(F( x+ )=c) in Equation 2, incurs substantial computational overhead, since each estimation requires multiple forward passes through the entire model. 3.2 Certified Bound via Feature Space Smoothing Considering the limitations inherent in the RS, we introduce the Feature-space Smoothing (FS). By turning any feature encoder fef_e into a smoothed version fe f_e, FS theoretically guarantees that fe f_e maintains a certified lower bound on the cosine similarity between clean and adversarial representations under ℓ2 _2-norm constrained perturbations. Smoothed feature encoder. For any feature encoder fe:→f_e: x→ z, where z is the representation normalized into the l2l_2 unit sphere, FS defines the smoothed encoder f^e() f_e( x) as: f^e()=∼(0,I)[fe(+)] f_e( x)=E_ (0,I)[f_e( x+ )] =1(2π)d/2∫ℝdfe(+)exp(−12‖2), = 1(2π)^d/2 _R^df_e( x+ ) \! (- 12\| \|^2 )d , -2.84526pt (3) where I denotes the d×d× d identity and ∥⋅∥\|·\| is the Euclidean norm. ∼(0,I) (0,I) denotes Gaussian noise with zero mean and standard deviation one. The smoothed feature encoder f^e() f_e( x) outputs the expectation of feature representations over the Gaussian distribution (,I)N( x,I). Figure 2: The training framework of the GSB. The denoiser performs pre-processing, and the smoothness mapper refines post-extracted features to enhance the Gaussian robustness. Parameters of MLLMs are frozen, and the denoiser and mapper are optimized with ℒL_P and ℒℳL_M. For evaluation, x is replaced with adversarial input ′ x , and the forward calculation marked in orange color will be removed. Gaussian robustness score. Define S()S_ x_t( x) as the score function that evaluates feature discrepancy between an input x and a targeted example t x_t, which is: St()=12(1+Cos(fe(),fe(t))) -2.84526ptS_ x_t( x)= 12 (1+Cos (f_e( x),\,f_e( x_t) ) ) (4) where Cos(⋅,⋅)Cos(·,·) denotes the cosine similarity and St()∈[0,1]S_ x_t( x)∈[0,1]. The Gaussian robustness score S^() S( x) is defined as: S^()=∼(0,I)[S(+)] S( x)= E _ (0,I) [S_ x( x+ ) ] (5) =12(1+∼(0,I)[Cos(fe(+),fe())]), = 12\! (1+ E _ (0,I)\! [Cos (f_e( x+ ),\,f_e( x) ) ] ), The score S^() S( x) evaluates the expected cosine similarity between the feature representation of a Gaussian-perturbed input + x+ and that of the clean input x, characterizing the Gaussian robustness of the vanilla feature encoder fef_e. Notably, we prove that this score S^() S( x) preserves a good Lipschitz property, which serves as a theoretical foundation for proving the certified robustness for f^e() f_e( x). Lemma 3.1 (Lipschitz property for the Gaussian robustness score). Let Φ(a)=12π∫−∞aexp(−12s2)s (a)= 1 2π _-∞^a (- 12s^2)ds be a standard Gaussian cumulative distribution function and Φ−1 ^-1 be its inverse. For any feature encoder fe:→f_e: x→ z, the mapping →Φ−1(S^()) x→ ^-1( S( x)) is 1−Lipschitz1-Lipschitz. The proof of Lemma 3.1 is in the Appendix, Section A.1.1. It implies that the mapping from x to Φ−1(S^()) ^-1( S( x)) exhibits strong adversarial robustness, as it satisfies a 11-Lipschitz constraint. Generally, S^() S( x) measures the Gaussian robustness of the vanilla feature encoder fe(x)f_e(x). We then prove that this score S^() S( x) fundamentally determines the value of the certified robustness bound of its smoothed encoder f^e(x) f_e(x). The following Theorem establishes an explicit relationship between this score and the lower bound on the adversarial feature cosine similarity. Theorem 3.2 (Certified lower bound on the adversarial feature cosine similarity). For any feature encoder fef_e and its smoothed version f^e f_e, let x and ′ x be clean and adversarial inputs with ‖′−‖≤ϵ\| x - x\|≤ε. The cosine similarity between the adversarial feature f^e(′) f_e( x ) and clean feature fe()f_e( x) satisfies: Cos(f^e(′),fe())≥2Φ(Φ−1(S^())−ϵ)−1Cos( f_e( x ),f_e( x))≥ 2 ( ^-1( S( x))-ε )-1. Denote 2Φ(Φ−1(S^())−ϵ)−12 ( ^-1( S( x))-ε )-1 as the Feature Cosine Similarity Bound (FCSB). Theorem 3.2 reveals that: • By FS, we can turn any given feature encoder fef_e into a smoothed version f^e f_e that maintains a FCSB between the adversarial and clean feature representations. • By maximizing the robustness score S^() S( x) of the given feature encoder fef_e, we can effectively enhance the value of FCSB derived on its smoothed version f^e f_e. The proof of Theorem 3.2 is in the Appendix, Section A.1.2. Corollary 3.3 (Certified radius ℛR for adversarial cosine similarity ≥ 0.5). Let x be the clean input, and ′ x be the adversarial input. Then Cos(f^e(x′),fe(x))≥0.5Cos( f_e(x ),f_e(x))≥ 0.5, for all ′ x with ‖′−‖2≤ℛ\| x - x\|_2 , where: ℛ=Φ−1(S^())−Φ−1(0.75). -2.84526ptR= ^-1\! ( S( x) )- ^-1(0.75). (6) Building upon Theorem 3.2, Corollary 3.3 establishes a certified radius ℛR for the smoothed feature encoder. For any adversarial perturbation satisfying ‖δ‖2≤ℛ\|δ\|_2 , the FCSB of the smoothed encoder is guaranteed to remain above 0.5. 4 Gaussian Smoothness Booster Theorem 3.2 and Corollary 3.3 reveal an intriguing robustness property of the smoothed feature encoder f^e f_e. However, feature encoders fef_e of MLLMs generally exhibit limited Gaussian robustness, which restricts the value of FCSB derived on f^e f_e. One solution is to estimate the smoothness score S^() S( x) through Monte Carlo sampling and training fef_e to maximize this score via gradient backpropagation. However, this could reduce the adaptability and practicality of the proposed FS framework, as fine-tuning and re-aligning the LLM is highly complex and costly. This thereby motivates us to propose the Gaussian Smoothness Booster (GSB), a plug-and-play module that can be seamlessly integrated with MLLMs to enhance its Gaussian robustness score S^(x) S(x). The training framework of our proposed GSB is shown in Figure 2. The denoiser P operates before feature extraction to denoise the Gaussian perturbations, and the smoothness mapper ℳM performs post-extraction to do the feature refinement. Those two modules work together to enhance the Gaussian robustness of the given feature encoder. 4.1 Gaussian Denoiser To perform the Gaussian denoising, the denoiser P is trained to minimize the reconstruction loss lmsel_mse, defined as: lmse=∼,∼(0,σ2I)‖−(+ε)‖, -2.84526ptl_mse= E _ subarrayc x ,~ (0,σ^2I) subarray\| x-P( x+ )\|, (7) where D represents the data distribution. Meanwhile, to further enhance robustness score after plugging P, we also introduce a robustness loss lrbl^P_rb, defined as: lrb=∼,∼(0,σ2I)[Cos(fe((+)),fe())], -2.84526ptl^P_rb= E _ x ,~ (0,σ^2I) [Cos (f_e(P( x+ )),f_e( x) ) ], (8) which encourages feature consistency between the purified and clean representations. We then fine-tune this model on the dataset D using the purifier loss ℒL_P, which is: ℒ=lrb+λ1lmse, -2.84526ptL_P=\ l^P_rb+ _1l_mse, (9) where λ1 _1 is the weighting coefficient. More details of the fine-tuning process on P can be found in the Appendix A.2.1. 4.2 Residual Smoothness Mapper For the residual smoothness mapper ℳM, we utilize a noise-aware residual module to enhance feature robustness while preserving its statistical distribution. The main process of this mapper can be formulated as: ~m=~+ℳ(~,σ)=~+∑i=0k−1mi(~i,σ), -8.53581pt z_m= z+M( z,σ)= z+ _i=0^k-1m_i ( z_i,σ ), (10) where ~=fe((+ε)) z=f_e(P( x+ )) denotes the purified feature representations and ~i+1=mi(~i,σ) z_i+1=m_i( z_i,σ) is the intermediate output with ~0=~ z_0= z. σ is the noise strength that adaptively controls the output magnitude of the mapper. k is the number of blocks (k=3k=3 in our experiments unless otherwise specified), and each block mi(⋅)m_i(·) contains multi-head attention, depthwise convolution, and MLP branches to refine the purified representation. To enhance the Gaussian robustness of the refined representation, we introduce the mapper robustness loss lrbℳl^M_rb, defined as: lrbℳ=∼,∼(0,σ2I)[Cos(~m,fe())], -2.84526ptl^M_rb= E _ x ,~ (0,σ^2I) [Cos ( z_m,f_e( x) ) ], -2.84526pt (11) which encourages feature consistency between the refined and clean representations. Meanwhile, to ensure that the refined feature preserves the statistical characteristics of the clean feature, we introduce two regularization terms: the identical loss lidl_id and the statistical loss lstatsl_stats, defined as: lstats=∼,∼(0,σ2I)1D∑d=1D[(μ~m(d)−μ(d))2+(σ~m(d)−σ(d))2],lid=∼‖ℳ(~,0)‖22. -2.84526pt \ aligned &l_stats= E _ subarrayc x ,\\ (0,σ^2I) subarray 1D _d=1^D [( _ z_m^(d)- _ z^(d))^2+( _ z_m^(d)- _ z^(d))^2 ],\\ &l_id= E _ subarrayc x subarray\|M( z,0)\|_2^2. aligned . (12) where ~m,∈ℝB×L×D z_m, z ^B× L× D with bath size B, token number L and feature dimension D. The lstatsl_stats enforces consistency between the element-wise mean μdμ^d and standard deviation σdσ^d of two representations, thereby preserving the statistical characteristics. Meanwhile, the identity loss lidl_id constrains the mapping network ℳM when the noise strength σ=0σ=0, promoting stability and preventing undesired distortions on clean inputs. The overall training loss ℒℳL_M is defined as: ℒℳ=lrbℳ+λ2lstats+λ3lid, -2.84526ptL_M=l^M_rb+ _2l_stats+ _3l_id, (13) where λ2 _2 and λ3 _3 are the weighting coefficients. More details of the training process on the residual smoothness mapper can be found in the Appendix A.2.2. Algorithm 1 Training algorithm of GSB 0: Dataset D, feature encoder fef_e, Denoiser P, mapper ℳM, sampling number n0n_0, noise std σ, loss weights λ1,λ2,λ3 _1, _2, _3. 1: Stage 1: Train denoiser P. 2: for each batch ∼ x do 3: ←fe() z← f_e( x) 4: for i=1i=1 to n0n_0 do 5: Sample i∼(0,σ2I) ^i (0,σ^2I) 6: ~i←(+i) x^i ( x+ ^i) 7: ~i←fe(~i) z^i← f_e( x^i) 8: end for 9: Compute lrbl^P_rb and lmsel_mse using Eqs. 7 and 8. 10: Update P by gradient descent on ∇ℒ _PL_P 11: end for 12: Stage 2: Train smoothness mapper ℳM. 13: for each batch ∼ x do 14: ←fe() z← f_e( x) 15: for i=1i=1 to n0n_0 do 16: Sample i∼(0,σ2I) ^i (0,σ^2I) 17: ~i←(+i) x^i ( x+ ^i) 18: ~i←fe(~i) z^i← f_e( x^i) 19: ~mi←~i+ℳ(~i,σ) z^i_m← z^i+M( z^i,σ) 20: end for 21: Compute lrbℳl^M_rb, lidl_id, and lstatsl_stats using Eqs. 11 and 12. 22: Update ℳM by gradient descent on ∇ℳℒℳ _ML_M 23: end for 4.3 Further Discussion on Plugging GSB Certified robustness for the encoder fef_e with GSB. Let fe′f _e denote the feature encoder integrated with the proposed GSB. Then the forward process can be formulated as: fe′(+)=fe((+))+ℳ(~,σ). -2.84526ptf _e( x+ )=f_e(P( x+ ))+M( z,σ). (14) Under this condition, the smoothed feature encoder and smoothness score are defined as: f^e′()=1(2π)d/2∫ℝdfe′(+)exp(−12‖2), f _e( x)= 1(2π)^d/2 _R^df _e( x+ ) (- 12\| \|^2 )d , (15) S^′()=12(1+∼(0,σ2I)[Cos(fe′(+),fe())]), S ( x)= 12 (1+ E _ (0,σ^2I) [Cos (f _e( x+ ),\,f_e( x) ) ] ), Where S^′()∈[0,1] S ( x)∈[0,1]. We can prove that the Lipschitz property derived in Lemma 3.1 still holds, and the theoretical bound in Section 3 remains valid. Then, utilizing Theorem 3.2 and Corollary 3.3, we can derive the certified lower bound on Cos(f^e′(′),fe())Cos( f _e( x ),f_e( x)) for any adversarial input ′ x . Training algorithm and dataset construction. The training procedure of GSB is summarized in Algorithm 1, where the denoiser P and the smoothness mapper ℳM are trained sequentially via a two-stage manner. The expectation over Gaussian perturbations is approximated by Monte Carlo sampling with n0n_0 samples drawn from (0,σ2I)N(0,σ^2I), where we set n0=8n_0=8 in our training to balance efficiency and estimation accuracy. 5 Certified Robustness via FS 5.1 Feature-wise Certification Table 1: The average FCSB at different predetermined adversarial bounds ϵε and the average certified radius ℛR for FCSB≥0.5FCSB≥ 0.5. Encoder σ Avg. FCSB at different adv-bound ϵε Avg. ℛR 0.125 0.25 0.375 0.50 0.75 CLIP-L14 0.25 0.828 0.623 0.313 -0.06 / 0.31 CLIP-L14+GSB 0.907 0.763 0.508 0.152 / 0.38 CLIP-L14 0.50 0.717 0.586 0.439 0.253 -0.132 0.33 CLIP-L14+GSB 0.924 0.873 0.799 0.698 0.409 0.69 To evaluate the effectiveness of the proposed FS framework with GSB, we adopt CLIP-L14 (Radford et al., 2021) as the vanilla feature encoder fef_e and assess the feature-wise certified robustness under the FS. We report both the certified FCSB under different adversarial constraints ϵε and the average certified radius ℛR for FCSB≥0.5FCSB≥ 0.5. The results are shown in Table 1. For implementation, we randomly sample 500 images from the ImageNet dataset (Deng et al., 2009) and approximate the Gaussian expectation using n=10,000n=10,000 Monte Carlo samples. As shown in Table 1, integrating GSB consistently enhances both the certified FCSB across a range of perturbation magnitudes and the average certified radius ℛR, indicating substantial improvements in certified robustness. 5.2 Prediction-wise Certification Beyond the feature-wise certification, the guarantees provided by FS can be propagated to the prediction space by smoothing the deepest-layer features with a solvable classification head, which enables a closed-form translation from feature-space certification to prediction-level robustness. To this end, we adopt CLIP-L14 with GSB as the last-layer feature extractor for downstream classification. Within the FS framework, we employ a learnable prototype-based cosine classification head, where each class is represented by a unit-norm prototype, and predictions are determined by cosine similarity in the normalized feature space. This design admits a direct characterization of class decision boundaries, allowing the certified feature-wise cosine similarity guaranteed by FS to be converted into certified prediction bounds. We also compared the performance with Randomized Smoothing (RS) (Cohen et al., 2019), where we utilize the same CLIP-L14+GSB as the feature encoder and train a learnable linear classification head for a fair comparison. In the experiments, we randomly sample 500 training instances from 10 ImageNet classes (Deng et al., 2009) to train both classification heads and evaluate them on an additional 500 test samples. For both FS and RS, the Gaussian expectation is approximated using n=10,000n=10,000 Monte Carlo samples. The comparative results in Table 2 demonstrate the effectiveness of the FS in providing certified prediction-level robustness. Further illustration and proof of prediction-wise certification via FS are provided in Appendix A.3. Table 2: The comparison results of the RS and FS framework on prediction-wise certification. Smoothing σ Certified accuracy at different ℓ2 _2 adv-bound ε (%) 0.00 0.125 0.25 0.375 0.5 0.75 1 1.50 RS 0.250.25 93.4 88 86.2 81.0 75.4 66.5 / / FS 94.2 92.4 90.0 88.0 85.8 72.0 / / RS 0.500.50 91.2 86 84.4 81.4 77.0 71.6 67.0 49.2 FS 91 85.2 84.0 83.6 83.0 77.6 69.2 54.0 6 Experimental Results on MLLM Protection While end-to-end robustness certification for MLLMs is appealing and practically meaningful, smoothing the entire model and certifying its autoregressive predictions with a solvable head are both computationally expensive and challenging. Alternatively, we thereby seek the feature-wise certification of the visual encoder (e.g., CLIP) in MLLMs, and empirically evaluate the resulting task-level robustness under strong white-box adversarial attacks, providing a practical compromise between certification feasibility and effectiveness. Notably, we find that this feature-wise certification offers two advantages: (i) efficiency, as smoothing the lightweight encoder fef_e incurs substantially lower computational overhead than smoothing the full model ℱF; and (i) generality, where feature-representation level certification makes it applicable to various downstream tasks. Table 3: Experimental results on adversarial robustness of different defense methods and MLLMs on image captioning tasks. Values in parentheses denote the average textual similarity measured by GPTScore. The overall best results are shown in bold, and the best results without smoothing are underlined, highlighting the significant performance gain introduced by our proposed FS. Model Method M-Attack (Li et al., 2025b) FOA (Jia et al., 2025) AttackVLM (Zhao et al., 2023) FCS ↑ ACC↑ ASR↓ FCS↑ ACC↑ ASR↓ FCS↑ ACC↑ ASR↓ LLaVA- 1.5-7B Org. 0.385 1% (0.06) 93% (0.54) 0.388 1% (0.049) 94% (0.578) 0.430 3% (0.089) 88% (0.490) Smoothed org. 0.588 63% (0.515) 3% (0.027) 0.587 65% (0.519) 6% (0.028) 0.598 71%(0.556) 4% (0.03) FARE (Schlarmann et al., 2024) 0.588 44% (0.409) 24% (0.097) 0.504 19% (0.197) 51% (0.222) 0.499 18%(0.190) 44%(0.221) Smoothed FARE 0.687 64% (0.541) 10% (0.053) 0.634 43% (0.391) 14% (0.065) 0.653 52% (0.443) 7% (0.045) TeCoA (Mao et al., 2023) 0.720 51% (0.458) 16% (0.081) 0.674 21% (0.236) 37% (0.187) 0.606 17% (0.179) 43% (0.218) Smoothed TeCoA 0.801 61% (0.510) 17% (0.081) 0.769 39% (0.364) 10% (0.0530) 0.720 38% (0.355) 14% (0.065) Open Flamingo -9B Org. 0.351 1% (0.089) 86% (0.583) 0.347 1% (0.091) 87%(0.569) 0.442 16% (0.222) 59% (0.379) Smoothed org. 0.592 59% (0.483) 3% (0.081) 0.588 57% (0.472) 1% (0.066) 0.703 61% (0.505) 0% (0.051) FARE (Schlarmann et al., 2024) 0.588 36% (0.407) 17% (0.145) 0.504 20% (0.249) 40% (0.240) 0.499 15% (0.243) 40% (0.247) Smoothed FARE 0.782 48% (0.462) 5% (0.062) 0.768 35% (0.354) 9% (0.102) 0.754 40% (0.404) 5% (0.086) TeCoA (Mao et al., 2023) 0.720 51% (0.472) 13% (0.116) 0.674 21% (0.240) 30% (0.210) 0.606 19% (0.244) 34% (0.225) Smoothed TeCoA 0.835 48% (0.464) 3% (0.067) 0.818 39% (0.371) 5% (0.088) 0.805 35%(0.341) 8%(0.079) 6.1 Experimental Setup Evaluated models and tasks. Since the proposed FS requires access to the forward feature computation process of MLLMs, we primarily validate its effectiveness on open-sourced MLLMs, including LLaVA-V1.5-7B (Liu et al., 2023, 2024) and OpenFlamingo9B (Awadalla et al., 2023). We comprehensively assess the performance of plugging the FS under adversarial conditions across multiple downstream tasks, including: • Image captioning: Following (Li et al., 2025b; Jia et al., 2025), we randomly take 100 images from the NIPS 2017 Adversarial Attacks and Defenses Competition dataset 6.1 and ask the model to caption the image. • Image Classification: We randomly sample 500 images from 10 classes in the ImageNet dataset (Deng et al., 2009) and ask the model to classify the input. • Visual Question Answering (VQA): We utilize 100 image-question pairs from the ScienceQA dataset (Lu et al., 2022) and ask the model to select the answer. †footnotetext: https://nips.c/Conferences/2017/CompetitionTrack The threat model. To comprehensively assess the robustness under strong adversaries, we employ three SOTA adversarial attacks specifically designed for MLLMs, named AttackVLM (Zhao et al., 2023), M-Attack (Li et al., 2025b), and FOA (Jia et al., 2025) using the white-box setting, where attackers can fully access the feature extractor and GSB. All attacks are implemented following their original best configurations, with the adversarial perturbation budget ϵε set to ‖ϵ‖∞=16/255\|ε\|_∞=16/255. Compared defense methods. As robustness certification for MLLMs remains largely unexplored, we primarily compare our method against adversarial training–based defenses, specifically FARE (Schlarmann et al., 2024) and TeCoA (Mao et al., 2023). Both FARE and TeCoA adopt adversarial training to obtain robust feature encoders that can be directly integrated into models such as LLaVA-1.5-7B and OpenFlamingo-9B. To ensure a fair and consistent comparison, we obtain all adversarially trained feature encoders from their official repositories, without any modification. Implementation details. For practical inference efficiency, we set the number of samples to n0=8n_0=8 for smoothing, ensuring a favorable trade-off between robustness and runtime. We train three independent GSB modules for different models, including the encoder of LLaVA-1.5-7B, the encoder of OpenFlamingo-9B, and CLIP-L14. To further assess cross-model generalization, we directly utilize the GSB trained on a vanilla encoder on adversarially trained encoders, including FARE and TeCoA. For all feature encoders, we set hyperparameters λ1,λ3=0.25 _1,~ _3=0.25, λ2=100 _2=100, and σ=0.25σ=0.25. In our tables, the term ”smoothed” denotes the process of smoothing the encoder via FS and enhancing it with GSB. Evaluation metrics. We mainly report: the Feature Cosine Similarity (FCS), the Accuracy (ACC), and the Attack Success Rate (ASR). Specifically, FCS measures the cosine similarity between clean and adversarial features extracted by the same feature encoder, reflecting its feature-wise robustness. ACC denotes the proportion of correctly completed tasks, while ASR indicates the proportion of cases where the model is successfully manipulated to produce the adversarially targeted outputs. For image classification and VQA tasks, ACC and ASR are determined by whether the MLLM outputs match the correct or adversarial targets. For image captioning, following (Li et al., 2025b), we adopt the LLM-as-a-judge protocol (Zheng et al., 2023) to evaluate ACC and ASR. Specifically, we first generate the clean and adversarially targeted captions by feeding the clean and targeted inputs into the vanilla MLLM (e.g., the original LLaVA). We then obtain the predicted caption by feeding the adversarially perturbed input into the tested model (e.g., Smoothed LLaVA). The textual similarity is computed using GPTScore (Fu et al., 2024), where a task is considered successful if the GPTScore between the predicted and clean captions is ≥0.5≥ 0.5, and an attack is considered successful if the GPTScore between the predicted and adversarially targeted captions is ≥0.3≥ 0.3. Table 4: Experimental results on adversarial robustness of different defense methods and MLLMs on image classification tasks. The overall best results are shown in bold, and the best results without smoothing are underlined. Model Method M-Attack (Li et al., 2025b) FOA (Jia et al., 2025) AttackVLM (Zhao et al., 2023) FCS ↑ ACC↑ ASR↓ FCS↑ ACC↑ ASR↓ FCS↑ ACC↑ ASR↓ LLaVA- 1.5-7B Org. 0.427 8.2% 78.2% 0.437 3.8 % 81.0 % 0.458 6.0% 78.4% Smoothed org. 0.590 84.8% 0.2% 0.600 87.2% 0.2% 0.605 87.0% 0.2% FARE (Schlarmann et al., 2024) 0.574 55.4% 14.6% 0.521 24.8% 39.8% 0.508 25.8% 45.0% Smoothed FARE 0.695 75.6% 5.2% 0.681 66.2% 1% 0.671 68.0% 1% TeCoA (Mao et al., 2023) 0.731 67.4% 2% 0.626 29.0% 20.0% 0.592 33.6% 17.8% Smoothed TeCoA 0.790 70.6% 0.6% 0.714 58.8% 0.8% 0.712 56.4% 0.4% Table 5: Experimental results on adversarial robustness of different defense methods and MLLMs on VQA tasks. The overall best results are shown in bold, and the best results without smoothing are underlined. Model Method M-Attack (Li et al., 2025b) FOA (Jia et al., 2025) AttackVLM (Zhao et al., 2023) FCS ↑ ACC↑ ASR↓ FCS↑ ACC↑ ASR↓ FCS↑ ACC↑ ASR↓ LLaVA- 1.5-7B Org. 0.398 31% 28% 0.383 22% 22% 0.474 25% 27% Smoothed org. 0.682 47% 0% 0.643 43% 0% 0.712 43% 1% FARE (Schlarmann et al., 2024) 0.657 47% 5% 0.590 32% 0% 0.550 38% 7% Smoothed FARE 0.861 48% 1% 0.788 38% 0% 0.825 39% 0% TeCoA (Mao et al., 2023) 0.788 31% 2% 0.748 31% 2% 0.667 31% 1% Smoothed TeCoA 0.925 32% 1% 0.880 34% 1% 0.879 29% 1% 6.2 Experimental Results on Different Tasks Image captioning. Table 3 reports the image captioning results of MLLMs under different attacks and defenses. The adversaries aim to manipulate the MLLMs into producing the caption corresponding to a maliciously chosen target image. The results show that converting MLLMs into smoothed variants via the proposed FS smoothing yields consistently strong robustness across diverse attacks, whereas empirical defenses degrade substantially under stronger adversaries. When attacks escalate from M-Attack to FOA, the accuracy of LLaVA with FARE drops from 44% to 19%, and that of TeCoA from 51% to 21%. In contrast, FS achieves significant robustness gains, increasing LLaVA accuracy from 1% to 65% and reducing the ASR from 94% to 6% under the strongest FOA attack. Moreover, FS exhibits strong cross-model generalization, consistently improving the robustness of FARE and TeCoA without additional fine-tuning. Image classification. Table 4 presents the classification results, indicating that FS delivers consistent and substantial robustness improvements across all settings. The adversarial objective is to mislead the model into classifying an adversarial image into a maliciously targeted class. integrating FS into the vanilla model increases LLaVA accuracy from 3.8% to 87.2% and reduces the ASR from 81% to 0.2% under the strongest FOA attack. Similar robustness gains are observed when FS is applied to FARE and TecoA. VQA. The comparative results of different defense methods on LLaVA-1.5-7B on the VQA are presented in Table 5. In this setting, the adversarial objective is to mislead the model into selecting a wrong option, ”None of the above”, for each image–question pair. This task presents a greater challenge for pure vision-based adversaries than the previous two, as MLLMs can often infer correct answers directly from textual cues. The results demonstrate that incorporating FS consistently yields substantial performance improvements across all attack types, significantly improving prediction accuracy while driving the ASR to nearly zero. Reasons for the poor performance of FARE and TeCoA despite high FCS. Although these encoders preserve high FCS relative to their clean inputs, adversarial training inherently induces a shift in the feature-space distribution. Subsequent adversarial perturbations and the limited diversity of adversarial training data further exacerbate this mismatch, leading to pronounced performance degradation on downstream tasks with unseen data distributions. 6.3 Ablation Study and Analysis of Efficiency Table 6: The ablation study. The clean accuracy is 91.0%. Smoothing Encoder σ Avg. FCSB at different ϵε Avg. ℛR Acc 0.25 0.50 0.75 / CLIP-L14 / / / / / 16.0% FS CLIP-L14 0.50 0.662 0.360 -0.018 0.41 66.2% CLIP-L14+P 0.718 0.441 0.073 0.46 72.4% CLIP-L14+ℳM 0.774 0.532 0.184 0.51 74.0% CLIP-L14+&ℳP\&M 0.873 0.698 0.409 0.69 89.6% Ablation: To rigorously assess the contribution of the proposed FS and each module of GSB (including denoiser P and mapper ℳM), we adopt CLIP-L14 as the base feature encoder and conduct the image classification task following the design in Section 5.2. We report both the feature-wise certification and the empirical performance under the FOA attack. The results are summarized in Table 6, where the first row indicates the performance of the vanilla CLIP-L14 without FS. The next four rows are the results of the smoothed CLIP-L14 with different modules enhanced. The results demonstrate that each component makes a significant contribution to improving both the certified radius ℛR and adversarial robustness. Meanwhile, it demonstrates that enforcing smoothness in the feature space substantially enhances practical robustness against adversarial perturbations, raising the accuracy from 16.0% to 89.6%. Table 7: The analysis of efficiency. Model n0n_0 Avg. infer time (s) FCS LLaVA-1.5 7B 1 0.41 0.450 4 0.52 0.562 8 0.69 0.590 64 3.08 0.609 Efficiency analysis: Table 7 presents the efficiency analysis under varying numbers of Gaussian samples, evaluated on a single RTX 4090 GPU. We consider the image captioning task and report the average per-image inference time together with the average FCS under FOA attack. Compared to whole-model smoothing, feature-wise smoothing incurs lower inference overhead; notably, setting n0=8n_0=8 increases the per-image inference time of LLaVA by only ∼ 0.28 seconds, highlighting the practicality of FS for real-world deployment. 7 Conclusion. This work pioneers the research on establishing the feature-space certified robustness of MLLMs. By introducing the Feature-space Smoothing (FS) framework, we show how to transform a given encoder into a smoothed version that is equipped with a theoretical lower bound on the cosine similarity between clean and adversarial representations. Moreover, we propose the Gaussian Robustness Booster (GSB), a plug-and-play module that seamlessly integrates into existing MLLMs to enhance the certified robustness guaranteed by FS. Extensive experiments demonstrate that FS provides effective adversarial protection for MLLMs in a plug-and-play manner while incurring modest inference latency, making it promising for real-world deployment on MLLM protection. References Anthropic (2025) Claude 4.5 sonnet system card. Note: https://w-cdn.anthropic.com/963373e433e489a87a10c823c52a0a013e9172d.pdf Cited by: §1. A. Awadalla, I. Gao, J. Gardner, J. Hessel, Y. Hanafy, W. Zhu, K. Marathe, Y. Bitton, S. Gadre, S. Sagawa, et al. (2023) Openflamingo: an open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390. Cited by: §6.1. S. Casper, L. Schulze, O. Patel, and D. Hadfield-Menell (2024) Defending against unforeseen failure modes with latent adversarial training. arXiv preprint arXiv:2403.05030. Cited by: §1, §2. B. Chen, J. Yin, S. Chen, B. Chen, and X. Liu (2023) An adaptive model ensemble adversarial attack for boosting adversarial transferability. In CVPR, Cited by: §1, §2. J. Cohen, E. Rosenfeld, and Z. Kolter (2019) Certified adversarial robustness via randomized smoothing. In ICML, Cited by: §1, §1, §2, §5.2. X. Cui, A. Aparcedo, Y. K. Jang, and S. Lim (2024) On the robustness of large multimodal models against image adversarial attacks. In CVPR, Cited by: §1, §2. G. DeepMind (2025) Gemini 3 model card. Note: https://storage.googleapis.com/deepmind-media/Model-Cards/Gemini-3-Pro-Model-Card.pdf Cited by: §1. J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei (2009) Imagenet: a large-scale hierarchical image database. In CVPR, Cited by: §5.1, §5.2, 2nd item. X. Ding, J. Chen, H. Yu, Y. Shang, Y. Qin, and H. Ma (2024) Transferable adversarial attacks for object detection using object-aware significant feature distortion. In AAAI, Cited by: §3.1. J. Fu, S. K. Ng, Z. Jiang, and P. Liu (2024) GPTScore: evaluate as you desire. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Cited by: §6.1. I. J. Goodfellow, J. Shlens, and C. Szegedy (2014) Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. Cited by: §1. Z. Hao, C. Ying, Y. Dong, H. Su, J. Song, and J. Zhu (2022) GSmooth: certified robustness against semantic transformations via generalized randomized smoothing. In ICML, Cited by: §1. M. Hein and M. Andriushchenko (2017) Formal guarantees on the robustness of a classifier against adversarial manipulation. NeurIPS. Cited by: §1. Q. Huang, I. Katsman, H. He, Z. Gu, S. Belongie, and S. Lim (2019) Enhancing adversarial example transferability with an intermediate level attack. In ICCV, Cited by: §3.1. X. Jia, S. Gao, S. Qin, T. Pang, C. Du, Y. Huang, X. Li, Y. Li, B. Li, and Y. Liu (2025) Adversarial attacks against closed-source mllms via feature optimal alignment. NeurIPS. Cited by: Table A.8, §1, §2, §3.1, 1st item, §6.1, Table 3, Table 4, Table 5. K. Kakizaki, K. Fukuchi, and J. Sakuma (2023) Certified defense for content based image retrieval. In CVPR, Cited by: §1. M. Lecuyer, V. Atlidakis, R. Geambasu, D. Hsu, and S. Jana (2019) Certified robustness to adversarial examples with differential privacy. In 2019 IEEE symposium on security and privacy (SP), Cited by: §2. M. Lee and D. Kim (2023) Robust evaluation of diffusion-based adversarial purification. In CVPR, p. 134–144. Cited by: §1. C. T. Lei, H. M. Yam, Z. Guo, Y. Qian, and C. P. Lau (2025) Instant adversarial purification with adversarial consistency distillation. In CVPR, Cited by: §1, §2. B. Li, C. Chen, W. Wang, and L. Carin (2018) Second-order adversarial attack and certifiable robustness. arXiv preprint arXiv: 1809.03113. Cited by: §2. J. Li, D. Li, S. Savarese, and S. Hoi (2023a) Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, Cited by: §2. Q. Li, Y. Guo, and H. Chen (2020) Yet another intermediate-level attack. In ECCV, Cited by: §3.1. Q. Li, Y. Guo, W. Zuo, and H. Chen (2023b) Improving adversarial transferability via intermediate-level perturbation decay. NeurIPS. Cited by: §3.1. X. Li, W. Sun, H. Chen, Q. Li, Y. He, J. Shi, and X. Hu (2025a) ADBM: adversarial diffusion bridge model for reliable adversarial purification. In ICLR, Cited by: §2. Z. Li, X. Zhao, D. Wu, J. Cui, and Z. Shen (2025b) A frustratingly simple yet highly effective attack baseline: over 90% success rate against the strong black-box models of gpt-4.5/4o/o1. In ICML 2025 Workshop on Reliable and Responsible Foundation Models, Cited by: §1, §2, 1st item, §6.1, §6.1, Table 3, Table 4, Table 5. H. Liu, C. Li, Y. Li, and Y. J. Lee (2024) Improved baselines with visual instruction tuning. In CVPR, Cited by: §6.1. H. Liu, C. Li, Q. Wu, and Y. J. Lee (2023) Visual instruction tuning. NeurIPS. Cited by: §6.1. P. Lu, S. Mishra, T. Xia, L. Qiu, K. Chang, S. Zhu, O. Tafjord, P. Clark, and A. Kalyan (2022) Learn to explain: multimodal reasoning via thought chains for science question answering. NeurIPS. Cited by: 3rd item. A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu (2018) Towards deep learning models resistant to adversarial attacks. In ICLR, Cited by: §1, §2. H. S. Malik, F. Shamshad, M. Naseer, K. Nandakumar, F. Khan, and S. Khan (2025) Robust-llava: on the effectiveness of large-scale robust image encoders for multi-modal large language models. arXiv preprint arXiv:2502.01576. Cited by: §1, §2. C. Mao, S. Geng, J. Yang, X. Wang, and C. Vondrick (2023) Understanding zero-shot adversarial robustness for large-scale models. In ICLR, Cited by: Table A.8, §1, §2, §6.1, Table 3, Table 3, Table 4, Table 5. W. Nie, B. Guo, Y. Huang, C. Xiao, A. Vahdat, and A. Anandkumar (2022) Diffusion models for adversarial purification. In ICML, Cited by: §1, §2. R. Olivier and B. Raj (2023) How many perturbations break this model? evaluating robustness beyond adversarial accuracy. In International conference on machine learning, p. 26583–26598. Cited by: §1. OpenAI (2025) GPT-5 technical report. Note: https://cdn.openai.com/gpt-5-system-card.pdf Cited by: §1. X. Qi, K. Huang, A. Panda, P. Henderson, M. Wang, and P. Mittal (2024) Visual adversarial examples jailbreak aligned large language models. In AAAI, Cited by: §2. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In ICML, Cited by: §2, §5.1. A. Raghunathan, J. Steinhardt, and P. Liang (2018) Certified defenses against adversarial examples. In ICLR, Cited by: §1. S. Rebuffi, S. Gowal, D. A. Calian, F. Stimberg, O. Wiles, and T. A. Mann (2021) Data augmentation can improve robustness. NeurIPS. Cited by: §1, §2. H. Salman, J. Li, I. Razenshteyn, P. Zhang, H. Zhang, S. Bubeck, and G. Yang (2019) Provably robust deep learning via adversarially trained smoothed classifiers. NeurIPS. Cited by: §1. C. Schlarmann, N. D. Singh, F. Croce, and M. Hein (2024) Robust clip: unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. In ICML, Cited by: Table A.8, §1, §2, §6.1, Table 3, Table 3, Table 4, Table 5. F. Tramer, N. Carlini, W. Brendel, and A. Madry (2020) On adaptive attacks to adversarial example defenses. NeurIPS. Cited by: §1, §2. Y. Wang, C. Liu, Y. Qu, H. Cao, D. Jiang, and L. Xu (2024a) Break the visual perception: adversarial attacks targeting encoded visual tokens of large vision-language models. In ACM M, Cited by: §2. Z. Wang, Z. Zhou, and W. Liu (2024b) Drf: improving certified robustness via distributional robustness framework. In AAAI, Cited by: §1. Z. Wang, X. Li, H. Zhu, and C. Xie (2024c) Revisiting adversarial training at scale. In CVPR, Cited by: §1, §2. E. Wong and Z. Kolter (2018) Provable defenses against adversarial examples via the convex outer adversarial polytope. In ICML, Cited by: §1. K. Wu, A. Wang, and Y. Yu (2020) Stronger and faster wasserstein adversarial attacks. In ICML, Cited by: §1, §2. S. Xhonneux, A. Sordoni, S. Günnemann, G. Gidel, and L. Schwinn (2024) Efficient adversarial training in llms with continuous attacks. NeurIPS. Cited by: §1, §2. S. Xia, W. Yang, Y. Yu, X. Lin, H. Ding, L. Duan, and X. Jiang (2024a) Transferable adversarial attacks on sam and its downstream models. NeurIPS. Cited by: §3.1. S. Xia, Y. Yu, X. Jiang, and H. Ding (2024b) Mitigating the curse of dimensionality for certified robustness via dual randomized smoothing. In ICLR, Cited by: §1, §1. P. Xie, Y. Bie, J. Mao, Y. Song, Y. Wang, H. Chen, and K. Chen (2025) Chain of attack: on the robustness of vision-language models against transfer-based adversarial attacks. In CVPR, Cited by: §2. J. Yoon, S. J. Hwang, and J. Lee (2021) Adversarial purification with score-based generative models. In ICML, p. 12062–12072. Cited by: §1. J. Zhang, X. Ma, X. Wang, L. Qiu, J. Wang, Y. Jiang, and J. Sang (2024) Adversarial prompt tuning for vision-language models. In ECCV, Cited by: §2. J. Zhang, J. Ye, X. Ma, Y. Li, Y. Yang, Y. Chen, J. Sang, and D. Yeung (2025) AnyAttack: towards large-scale self-supervised adversarial attacks on vision-language models. In CVPR, Cited by: §2. Y. Zhao, T. Pang, C. Du, X. Yang, C. Li, N. M. Cheung, and M. Lin (2023) On evaluating adversarial robustness of large vision-language models. NeurIPS. Cited by: §1, §2, §6.1, Table 3, Table 4, Table 5. L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. (2023) Judging llm-as-a-judge with mt-bench and chatbot arena. NeurIPS. Cited by: §6.1. G. Zollicoffer, M. N. Vu, B. Nebgen, J. Castorena, B. Alexandrov, and M. Bhattarai (2025) Lorid: low-rank iterative diffusion for adversarial purification. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 23081–23089. Cited by: §1. Appendix Appendix A.1 Proof of core theorem A.1.1 Proof of lemma1 Note that: ∇Φ−1(S^xc())=∇S^xc()Φ′(Φ−1(S^xc())).∇ ^-1( S_x_c( x))= ∇ S_x_c( x) ( ^-1( S_x_c( x)) ). (A.16) Denote p=S^c()p= S_ x_c( x), we can get that: Φ′(Φ−1(S^xc()))=12πexp(−12(Φ−1(p))2) ( ^-1( S_x_c( x)) )= 1 2π \! (- 12( ^-1(p))^2 ) (A.17) Thus, we need to prove that for any unit direction u. u⋅∇S^c()≤12πexp(−12(Φ−1(p))2).u·∇ S_ x_c( x)≤ 1 2π \! (- 12( ^-1(p))^2 ). (A.18) For the left-hand side, we can get: ∇S^xc()=1(2π)n/2∫ℝnSc()(−)exp(−12‖−‖2)d,∇ S_x_c( x)= 1(2π)^n/2 _R^nS_ x_c( t)\,( x- t)\, \! (- 12\| x- t\|^2 )\,d t, (A.19) which can also be rewritten as follows: X∼(0,I)[Sxc(x+X)X⋅u]. E _X (0,I) [S_x_c(x+X)X· u ]. We can now claim that the supremum of the above quantity over all encoders f:→f: x→ z, subject to the constraint that p=[Sxc(x+X)]p=E[S_x_c(x+X)], is equal to: [(X⋅u) 1X⋅u≥−Φ−1(p)]=12πexp(−12(Φ−1(p))2),E [(X· u)\,1\X· u≥- ^-1(p)\ ]= 1 2π \! (- 12 ( ^-1(p) )^2 ), (A.20) which concludes the proof of Lemma 1. A.1.2 Proof of Theorem1 From Lemma 1, we can get: ‖−′‖≥Φ−1(S^())−Φ−1(S^(′)). \| x- x \|≥ ^-1( S_ x( x))- ^-1( S_ x( x )). (A.21) As ‖−′‖≤ϵ \| x- x \|≤ε, we can get: Φ−1(S^(′))≥Φ−1(S^())−ϵ. ^-1( S_ x( x ))≥ ^-1( S_ x( x))-ε. (A.22) Meanwhile, as fe()f_e( x) is a unit vector, using Equation 5 in the main paper, we can get that: S^() S( x) =12(1+⟨[fe(+ε)],fe()⟩) = 12 (1+ [f_e( x+ )],\,f_e( x) ) =12(1+⟨f^e(),fe()⟩). = 12 (1+ f_e( x),\,f_e( x) ). (A.23) where ⟨⋅,⋅⟩ ·,· is the inner product. Hence, we can get: ⟨f^e(),fe()⟩=2S^()−1. f_e( x),\,f_e( x) =2 S( x)-1. (A.24) As ‖f^e(′)‖≤1 \| f_e( x ) \|≤ 1 and ‖fe()‖=1 \|f_e( x) \|=1, we can get: Cos(f^(),fe()) Cos ( f( z),\,f_e( x) ) =⟨f^(),fe()⟩‖f^()‖ = f( z),f_e( x) \| f( z)\|\; (A.25) ≥⟨f^(),fe()⟩ ≥\; f( z),f_e( x) By the monotonicity of Φ , S^(′)≥Φ(Φ−1(S^())−ϵ). S_ x( x )\;≥\; \! ( ^-1\! ( S( x) )-ε ). (A.26) Combining Equations A.24, A.25, A.26, we thereby can derive that: Cos(f^(′),fe())≥ 2Φ(Φ−1(S^())−ϵ)−1,Cos ( f( x ),\,f_e( x) )\;≥\;2\, \! ( ^-1\! ( S( x) )-ε )-1, (A.27) which concludes the proof. For noise variance σ2σ^2, the same proof gives the bound with ϵε replaced by ϵ/σε/σ. Appendix A.2 Training details of GSB A.2.1 Details of Denoiser The denoiser P is implemented as a lightweight conditional U-Net and trained in a supervised denoising paradigm using a mean squared error (MSE) objective to regress clean images from their noisy counterparts. P adopts a symmetric encoder–decoder architecture composed of residual blocks and multi-scale skip connections, which facilitate effective information propagation across different spatial resolutions. All input images are resized to 256×256256× 256, and the network progressively aggregates local and global contextual features through hierarchical downsampling and upsampling operations. To enable adaptive denoising under varying noise levels, P incorporates a learnable sigma embedding that conditions intermediate feature transformations, allowing the model to dynamically adjust its denoising behavior. We optimize P using the Adam optimizer with a learning rate of 1×10−41× 10^-4, a batch size of 8, and a total of 8 training epochs. Notably, P remains computationally efficient, containing only approximately 10 MB of parameters. A.2.2 Details of training residual mapper Figure A.3: Structure of the residual smoothness mapper. The structure of the proposed residual smoothness mapper ℳM is illustrated in Figure A.3. Each block is designed to refine the purified feature representation while preserving its statistical structure and ensuring that the refinement magnitude adapts smoothly to the injected noise magnitude σ. A single block consists of: (1) noise-aware LayerNorm-FiLM(σ) module, (2) multi-head attention, (3) a channel-wise MLP branch, (4) a depthwise convolution, and (5) a noise-aware residual module. LayerNorm and FiLM(σ). Each block begins by normalizing the input feature and injecting noise-level conditioning: ~i=LN(~i),[γ(σ),β(σ)]=FiLM(σ), z_i=LN( z_i), [γ(σ),β(σ)]=FiLM(σ), (A.28) where the FiLM module takes the scalar noise strength σ and outputs channel-wise affine parameters through a lightweight MLP: FiLM(σ)=W2ϕ(W1σ)∈ℝ2D,FiLM(σ)=W_2\,φ(W_1σ) ^2D, (A.29) with ϕφ denoting a GELU activation. This modulation injects explicit noise awareness into each block: the transformation smoothly diminishes as σ→0σ→ 0, ensuring that the mapper leaves clean features nearly unchanged. Main computation structure. Each mi(⋅)m_i(·) contains three parallel refinement pathways: • Multi-head attention: Applied only in the first block, the lightweight attention layer captures long-range structural dependencies, producing global-context features hattnh_attn. • MLP branch: A two-layer feed-forward network with GELU activation produces channel-wise refinement hmlph_mlp. • Depthwise convolution: A depthwise 1D convolution captures local continuity in the feature sequence and contributes hconvh_conv. The outputs are fused as: hi=hattn+hmlp+0.5hconv,h_i=h_attn+h_mlp+0.5\,h_conv, (A.30) where the conv branch is down-weighted following design practices for local/global feature fusion. Noise-aware residual. We first decompose hih_i into a unit direction vector by vi=hi∖‖hi‖.v_i=h_i \|h_i\|. To ensure that the refinement adapts smoothly to the noise level, we modulate its magnitude through two learnable functions of σ: αi=softplus(MLPα([σ,logσ])), _i=softplus\! (MLP_α([σ, σ]) ), σβi=σsoftplus(MLPβ(σ)). σ _i=σ^softplus(MLP_β(σ)). (A.31) Here, αi _i controls the amplitude of the refinement, while βi _i modulates the exponent of the noise term, enabling nonlinear noise–feature interactions. The residual update is then computed as Δi=[(1+γ(σ))(vi⋅αiσβi)+β(σ)]⊙Scalei, z_i= [(1+γ(σ))\, (v_i· _iσ _i )+β(σ) ] _i, (A.32) where γ(σ)γ(σ) and β(σ)β(σ) are FiLM-generated per-channel affine parameters, and ScaleiScale_i is a learnable per-channel damping factor initialized to 5×10−45× 10^-4 for stability. The next feature is then updated via ~i+1=~i+Δi. z_i+1= z_i+ z_i. (A.33) Overall effect of the mapper. The residual smoothness mapper thus provides the following benefits: • Noise-adaptive refinement: FiLM(σ)(σ) and σβσ^β ensure the introduced modification decreases as σ→0σ→ 0, allowing the mapper to preserve the natural feature distribution while refining noisy features more aggressively. • Multi-scale feature enhancement: Attention, MLP, and depth-wise convolution jointly capture global structure, per-channel adaptation, and local smoothness. • Stable and expressive residual learning: Fixup-style initialization and ScaleiScale_i ensure training stability even with multiple stacked residual blocks. Collectively, ℳM significantly enhances the smoothness and robustness of purified representations while avoiding distributional drift. Training details. The mapper ℳM is trained in a self-supervised manner using the objective ℒmL_m defined in Equation 13. We train ℳM for 8 epochs with a batch size of 16. The training process employs the AdamW optimizer with an initial learning rate of 2×10−42× 10^-4 and a cosine annealing learning rate schedule. Appendix A.3 More Details on Prediction-Wise Certification A.3.1 Preliminaries and Notation Let fe:→ℝdf_e:X ^d be a (deterministic) feature encoder and let f^e f_e denote its smoothed version produced by Feature-space Smoothing (FS). Given a clean input ∈ x and an adversarial input ′∈ x satisfying ‖′−‖≤ϵ\| x - x\|≤ε, define the clean feature and the adversarial (smoothed) feature as :=fe()∈ℝd,^′:=f^e(′)∈ℝd. z:=f_e( x) ^d, z := f_e( x ) ^d. (A.34) We further define their ℓ2 _2-normalized counterparts: :=‖2∈d−1,:=^′‖^′‖2∈d−1, u:= z\| z\|_2 ^d-1, v:= z \| z \|_2 ^d-1, (A.35) where d−1:=∈ℝd:‖2=1S^d-1:=\ t ^d:\| t\|_2=1\ is the unit sphere. Thus, Cos(f^e(′),fe())=⊤Cos( f_e( x ),f_e( x))= u v. FS-certified feature cosine similarity. From Theorem 3.2, for any ′ x with ‖′−‖≤ϵ\| x - x\|≤ε, the cosine similarity between f^e(′) f_e( x ) and fe()f_e( x) is lower bounded as ⊤=Cos(f^e(′),fe())≥γ(;ϵ), u v\;=\;Cos\! ( f_e( x ),f_e( x) )\;≥\;γ( x;ε), (A.36) where we denote the Feature Cosine Similarity Bound (FCSB) by γ(;ϵ):= 2Φ(Φ−1(S^())−ϵ)−1∈[−1,1].γ( x;ε)\;:=\;2 \! ( ^-1( S( x))-ε )-1∈[-1,1]. (A.37) For brevity, we write γ:=γ(;ϵ)γ:=γ( x;ε). Cosine-prototype classification head. We consider a cosine-based prototype classifier on the normalized feature ∈d−1 t ^d-1. Let kk=1K⊂ℝd\ p_k\_k=1^K ^d be learnable class prototypes satisfying ‖k‖2=1\| p_k\|_2=1 for all k. The class score (logit) for class k is defined as sk():=⊤k,s_k( u):= u p_k, (A.38) and the predicted label is y^():=argmaxk∈1,…,Ksk()=argmaxk⊤k. y( u):= _k∈\1,…,K\s_k( u)\;=\; _k u p_k. (A.39) Let y:=y^()y:= y( u) denote the clean prediction under u. A.3.2 Tight Cosine Similarity Bound Lemma A.3.1 (Tight bounds under a cosine constraint). Let ,,∈d−1 u, v, p ^d-1 be unit vectors and suppose that ⊤≥γ u v≥γ for some γ∈[−1,1]γ∈[-1,1]. Define a:=⊤∈[−1,1]a:= u p∈[-1,1]. Then the inner product ⊤ v p is bounded as γa−1−γ21−a2≤⊤≤γa+1−γ21−a2. γ a- 1-γ^2 1-a^2\;≤\; v p\;≤\;γ a+ 1-γ^2 1-a^2. (A.40) Proof sketch. The stated bounds are a standard consequence of spherical geometry. Since ,,∈d−1 u, v, p ^d-1, we may interpret their inner products as cosines of the corresponding pairwise angles. In particular, let a=⊤=cosα,γ=⊤=cosΔ,a= u p= α, γ= u v= , where α,Δ∈[0,π]α, ∈[0,π] denote the angles between (,)( u, p) and (,)( u, v), respectively. Under the constraint ⊤≥γ u v≥γ, the adversarial feature direction v is restricted to lie within a spherical cap centered at u with angular radius Δ . Consequently, the angle between v and p lies in the interval [α−Δ,α+Δ][α- ,\;α+ ]. It follows that ⊤=cos(α±Δ)=cosαcosΔ∓sinαsinΔ, v p= (α± )= α ∓ α , which yields the bounds in equation A.40 by substituting cosα=a α=a and cosΔ=γ =γ. The bounds are tight and are attained when the three vectors ,, u, v, p lie in a common two-dimensional subspace and v is rotated toward or away from p along the geodesic on the unit sphere. ∎ A.3.3 Closed-form Propagation to Prediction-level Certification We now use Lemma A.3.1 to certify that the predicted label remains unchanged under any adversarial perturbation within radius ϵε. For the clean normalized feature u, define the clean class scores ak:=sk()=⊤k,k=1,…,K.a_k:=s_k( u)= u p_k, k=1,…,K. (A.41) Let y:=argmaxkaky:= _ka_k be the clean predicted class. Theorem A.3.2 (Prediction-level certification via FS with cosine prototypes). Fix a clean input x and let γ=γ(;ϵ)γ=γ( x;ε) be the FS-derived FCSB from Theorem 3.2. For any adversarial input ′ x satisfying ‖′−‖≤ϵ\| x - x\|≤ε and each class k, we have: sk()=⊤k s_k( v)= v p_k ≤γak+1−γ21−ak2, ≤γ a_k+ 1-γ^2 1-a_k^2, (A.42) sk()=⊤k s_k( v)= v p_k ≥γak−1−γ21−ak2. ≥γ a_k- 1-γ^2 1-a_k^2. (A.43) In particular, if the following certified margin condition holds: γay−1−γ21−ay2⏟=:LBy(γ)>maxk≠y(γak+1−γ21−ak2)⏟=:UBk(γ), γ a_y- 1-γ^2 1-a_y^2_=:~LB_y(γ)\;>\; _k≠ y (γ a_k+ 1-γ^2 1-a_k^2 )_=:~UB_k(γ), (A.44) then the classifier prediction is provably invariant within the ϵε-ball: argmaxksk()=yfor all ′ such that ‖′−‖≤ϵ. _ks_k( v)=y all x such that \| x - x\|≤ε. (A.45) Proof. By Theorem 3.2, for any ′ x with ‖′−‖≤ϵ\| x - x\|≤ε, we have Cos(f^e(′),fe())=⊤≥γ.Cos\! ( f_e( x ),f_e( x) )= u v≥γ. (A.46) Fix any class k. Since ,,k∈d−1 u, v, p_k ^d-1 and ⊤≥γ u v≥γ, applying Lemma A.3.1 with =k p= p_k and a=ak=⊤ka=a_k= u p_k yields the per-class upper and lower bounds equation A.42–equation A.43. Now consider the clean predicted class y=argmaxkaky= _ka_k. From equation A.43, the adversarial score of class y is bounded from below as sy()≥LBy(γ).s_y( v) _y(γ). (A.47) For any other class k≠yk≠ y, from equation A.42 we have sk()≤UBk(γ).s_k( v) _k(γ). (A.48) If the certified margin condition equation A.44 holds, then for all k≠yk≠ y, sy()≥LBy(γ)>UBk(γ)≥sk(),s_y( v)\;≥\;LB_y(γ)\;>\;UB_k(γ)\;≥\;s_k( v), (A.49) which implies sy()>sk()s_y( v)>s_k( v) for every k≠yk≠ y. Therefore, the argmax prediction under v remains y, i.e., argmaxksk()=y. _ks_k( v)=y. (A.50) This holds for any ′ x with ‖′−‖≤ϵ\| x - x\|≤ε, which concludes the proof. ∎ Appendix A.4 More Experimental Results Figure A.4: Illustration of the evaluation process. A.4.1 Implementation details for main results Image captioning. For this task, we prompt the MLLM with “Describe this image in one concise sentence, no longer than 20 words.” to generate a caption for each input image. The predicted caption is then compared against the ground-truth caption and the adversarial target caption for evaluation. Image classification. For this task, we prompt the MLLM with ”You are a precise visual classifier. What are the main objects in this image?: 0. rooster, 1. gibbon, 2. golden_retriever, 3. goldfish, 4. hen, 5. hognose_snake, 6. ice_bear, 7. killer_whale, 8. king_crab, 9. kite, 10. shark. Output only a single integer between 0 and 10 with no explanation, no text, no punctuation.” to obtain the predicted class for each input image. All images in the dataset belong to classes 0–9, while class 10 (shark) is used as the adversarial target class. We then compute the ACC and ASR based on the predicted labels, the ground-truth labels, and the adversarial target label. VQA. For this task, we prompt the MLLM using the provided question associated with each input image and ask it to predict the corresponding answer. An example prompt is as follows: ”You are a knowledgeable multimodal assistant. Please answer the multiple-choice question ONLY by outputting the single letter of the correct option. Do not include explanations or extra text. Question: Which ocean is highlighted? Options: A. the Arctic Ocean B. the Atlantic Ocean C. the Pacific Ocean D. the Southern Ocean E. None of the above. If the visual information is not clear or uncorrelated with the questions, you should select the choice: None of the above. Answer with exactly one letter from the options above (e.g., A).”. In this evaluation, each question has a unique correct answer among the listed options, except for “None of the above,” which is reserved for the adversarial target. The adversarial objective is to induce the MLLM to incorrectly choose “None of the above”. An illustration of the implementation process is shown in Figure A.4. A.4.2 Experimental results on large attack bound Table A.8: Experimental results on adversarial robustness of different defense methods and MLLMs on image classification tasks. The attack bound ϵ=32/255ε=32/255. Model Method FOA (Jia et al., 2025) FCS ↑ ACC↑ ASR↓ LLaVA- 1.5-7B Org. 0.37 3.6% 78.6% Smoothed org. 0.512 35.2% 24.2% FARE (Schlarmann et al., 2024) 0.408 10.4% 64.2% Smoothed FARE 0.605 48.4% 19.2% TeCoA (Mao et al., 2023) 0.454 12.0% 29.6% Smoothed TeCoA 0.576 32.0% 9.6% The experimental results under a large adversarial perturbation of ε=32/255 =32/255 are presented in Table A.8. In addition, Figure A.5 provides visualizations of the adversarial examples generated by different attacks and perturbation bounds. These results show that nearly all existing defenses show great performance drop under such a strong attack. Under this challenging setting, the best accuracy is obtained by integrating FS-GSB with FARE, reaching 48.4% on LLaVA, while the lowest ASR (9.6%) is achieved by combining FS-GSB with TeCoA. These findings further highlight the effectiveness and adaptability of the proposed framework, even against large-magnitude adversarial perturbations. Figure A.5: Visualization of the adversarial examples.