Paper deep dive

DiffCAP: Diffusion-based Cumulative Adversarial Purification for Vision Language Models

Jia Fu, Yongtao Wu, Yihang Chen, Kunyu Peng, Xiao Zhang, Volkan Cevher, Sepideh Pashami, Anders Holst

Year: 2025Venue: arXiv preprintArea: Multimodal SafetyType: EmpiricalEmbeddings: 121

Models: CLIP, LLaVA, OpenFlamingo

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/12/2026, 5:33:09 PM

Summary

DiffCAP is a novel adversarial purification strategy for Vision Language Models (VLMs) that uses a cumulative diffusion-based approach. By dynamically injecting Gaussian noise until a semantic similarity threshold is met, it effectively neutralizes adversarial perturbations while optimizing diffusion time and hyperparameter tuning, outperforming existing defense techniques.

Entities (4)

DiffCAP · method · 100%Vision-Language Models · technology · 98%Adversarial Perturbation · threat · 95%Diffusion Model · technology · 95%

Relation Signals (3)

DiffCAP → utilizes → Diffusion Model

confidence 98% · Subsequently, a pretrained diffusion model is employed to denoise the stabilized image.

DiffCAP → purifies → Vision-Language Models

confidence 95% · DiffCAP is the first adversarial purification framework that enables fast and robust VLM inference.

Adversarial Perturbation → affects → Vision-Language Models

confidence 92% · their susceptibility to perturbations poses a significant threat to their reliability in real-world applications.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Vision Language Models (VLMs) have shown remarkable capabilities in multimodal understanding, yet their susceptibility to perturbations poses a significant threat to their reliability in real-world applications. Despite often being imperceptible to humans, these perturbations can drastically alter model outputs, leading to erroneous interpretations and decisions. This paper introduces DiffCAP, a novel diffusion-based purification strategy that can effectively neutralize adversarial corruptions in VLMs. We observe that adding minimal noise to an adversarially corrupted image significantly alters its latent embedding with respect to VLMs. Building on this insight, DiffCAP cumulatively injects random Gaussian noise into adversarially perturbed input data. This process continues until the embeddings of two consecutive noisy images reach a predefined similarity threshold, indicating a potential approach to neutralize the adversarial effect. Subsequently, a pretrained diffusion model is employed to denoise the stabilized image, recovering a clean representation suitable for the VLMs to produce an output. Through extensive experiments across six datasets with three VLMs under varying attack strengths in three task scenarios, we show that DiffCAP consistently outperforms existing defense techniques by a substantial margin. Notably, DiffCAP significantly reduces both hyperparameter tuning complexity and the required diffusion time, thereby accelerating the denoising process. Equipped with strong theoretical and empirical support, DiffCAP provides a robust and practical solution for securely deploying VLMs in adversarial environments.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

121,189 characters extracted from source content.

Expand or collapse full text

DiffCAP: Diffusion-based Cumulative Adversarial Purification for Vision Language Models Jia Fu1,2, Yongtao Wu3, Yihang Chen4, Kunyu Peng5, Xiao Zhang6, Volkan Cevher3, Sepideh Pashami2, Anders Holst1,2 1KTH Royal Institute of Technology 2RISE Research Institutes of Sweden 3Swiss Federal Technology Institute of Lausanne 4University of California, Los Angeles 5Karlsruhe Institute of Technology 6CISPA Helmholtz Center for Information Security Abstract Vision Language Models (VLMs) have shown remarkable capabilities in multimodal understanding, yet their susceptibility to perturbations poses a significant threat to their reliability in real-world applications. Despite often being imperceptible to humans, these perturbations can drastically alter model outputs, leading to erroneous interpretations and decisions. This paper introduces DiffCAP, a novel diffusion-based purification strategy that can effectively neutralize adversarial corruptions in VLMs. We observe that adding minimal noise to an adversarially corrupted image significantly alters its latent embedding with respect to VLMs. Building on this insight, DiffCAP cumulatively injects random Gaussian noise into adversarially perturbed input data. This process continues until the embeddings of two consecutive noisy images reach a predefined similarity threshold, indicating a potential approach to neutralize the adversarial effect. Subsequently, a pretrained diffusion model is employed to denoise the stabilized image, recovering a clean representation suitable for the VLMs to produce an output. Through extensive experiments across six datasets with three VLMs under varying attack strengths in three task scenarios, we show that DiffCAP consistently outperforms existing defense techniques by a substantial margin. Notably, DiffCAP significantly reduces both hyperparameter tuning complexity and the required diffusion time, thereby accelerating the denoising process. Equipped with strong theoretical and empirical support, DiffCAP provides a robust and practical solution for securely deploying VLMs in adversarial environments. Warning: This paper contains images or texts that may be considered offensive. Figure 1: Overview of the DiffCAP Pipeline: An adversarial input is cumulatively processed through steps 1 to 5; if a stopping condition is not satisfied at step 5, the process restarts from step 1, otherwise the purification output is generated at step 6. See Alg. 1 for details. 1 Introduction Vision language models (VLMs) have exhibited impressive performance in a diverse range of multimodal understanding tasks radford2021learning ; lu2019vilbert ; jia2021scaling ; alayrac2022flamingo , empowering numerous real-world applications such as image-grounded text generation (e.g., image captioning and visual question answering) li2020oscar ; mokady2021clipcap ; li2022blip and zero-shot classification radford2021learning ; zhai2022lit ; alayrac2022flamingo ; zhai2023sigmoid . However, their inherent susceptibility to adversarial perturbations presents a critical challenge zhao2023evaluating ; qi2024visual ; zhang2022towards . These perturbations are usually designed to be imperceptible to humans, but when added to natural images, can deceive models into making incorrect predictions, severely undermining their reliability and effectiveness goodfellow2014explaining ; madry2017towards ; carlini2017towards . Adversarial vulnerability is especially concerning as malicious actors may exploit these ML systems to spread misinformation or fraudulent activities wu2024safety , highlighting the urgent need for robust defensive strategies jin2024jailbreakzoo ; liu2024survey . To mitigate this threat, significant research efforts have focused on adversarial defenses specifically designed for VLMs liu2024survey ; weng2025mmj . A dominant direction in this field is adversarial training, which fine-tunes models using adversarially perturbed data to enhance robustness. For instance, recent approaches such as Robust CLIP schlarmann2024robust have leveraged supervised adversarial fine-tuning to fortify VLMs against specific attack types. Although effective within their training scope, these methods exhibit significant limitations, particularly poor generalization to novel, unseen attacks dolatabadi2022ℓ ; laidlaw2020perceptual and substantial computational overhead associated with continuous retraining and fine-tuning procedures wong2020fast ; andriushchenko2020understanding . In contrast, adversarial purification yoon2021adversarial ; nie2022diffusion emerges as a promising alternative that does not require the expensive adversarial fine-tuning of models. Techniques like DiffPure nie2022diffusion have demonstrated the feasibility of purification approaches by directly removing adversarial perturbations from input data using generative models, such as diffusion models song2020denoising ; song2020score ; yang2023diffusion ; song2019generative . These methods maintain a high generalization capability to unseen attacks without necessitating modifications to the underlying VLM, thus preserving original model performance on benign inputs. Despite these advantages, current generative model-based purification techniques suffer from substantial slowdown at inference time, hindering their practical deployment, particularly for real-time scenarios involving large VLMs. To address these critical shortcomings, we propose DiffCAP, abbreviated for Diffusion-based Cumulative Adversarial Purification, the first adversarial purification strategy specifically designed for VLMs. DiffCAP leverages a novel mechanism that dynamically identifies the minimal necessary diffusion time, effectively balancing purification efficacy and computational efficiency. Our method cumulatively injects random Gaussian noise into adversarially perturbed images until the embeddings of two consecutively noised images converge to a predefined similarity threshold, indicating the potential neutralization of adversarial perturbations. A diffusion model subsequently denoises this stabilized image, enabling the recovery of a clean, interpretable image for VLM inference. Contribution. The main contributions of our work can be summarized in the following aspects: • To the best of our knowledge, DiffCAP is the first adversarial purification framework that enables fast and robust VLM inference. DiffCAP is designed to minimize the diffusion time and hyperparameter adjustments, enabling efficient training and fast inference. • We establish a recovery region during the forward diffusion process, supported by a rigorous analysis of semantic stability in the embedding space. In particular, we prove that the semantic change between consecutive forward diffusion steps decreases as the diffusion time t grows. • We conduct comprehensive experiments across three popular VLMs, six diverse datasets, multiple perturbation strengths, and various multimodal tasks, including image captioning, visual question answering, and zero-shot classification. The results demonstrate that DiffCAP consistently outperforms existing state-of-the-art defenses by a considerable margin. 2 Preliminaries This section provides the background and preliminary definitions of vision encoders, including their use for both CLIP and Vision LLMs, as well as continuous-time diffusion models. 2.1 Vision encoder CLIP. Contrastive Language-Image Pre-training (CLIP) radford2021learning consists of a vision encoder ϕ:ℝd→ℝm:bold-italic-ϕ→superscriptℝsuperscriptℝ φ:R^d ^mitalic_ϕ : blackboard_Rd → blackboard_Rm and a text encoder :ℝd′→ℝm:→superscriptℝsuperscript′ℝ ψ:R^d ^mitalic_ψ : blackboard_Rd start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT → blackboard_Rm. The vision encoder and the text tokenizer have the same embedding dimension. We define a zero-shot classifier hℎh. For a K-class task, text prompts ksubscript t_kitalic_titalic_k, such as ‘A photo of <class k>’, are generated for k=1,…,K1…k=1,…,Kk = 1 , … , K. The classifier hℎh, determined by ϕbold-italic-ϕ φitalic_ϕ and ψitalic_ψ, calculates logits for an input image xitalic_x via the cosine similarity between the image embedding and each prompt embedding: hk⁢()=cos⁡(ϕ⁢(),⁢(k))=⟨ϕ⁢()‖ϕ⁢()‖2,⁢(k)‖⁢(k)‖2⟩.subscriptℎbold-italic-ϕsubscriptbold-italic-ϕsubscriptnormbold-italic-ϕ2subscriptsubscriptnormsubscript2h_k( x)= ( φ( x), ψ( t_k))= % φ( x) \| φ( x) \|_2, ψ( % t_k) \| ψ( t_k) \|_2 .hitalic_k ( italic_x ) = cos ( italic_ϕ ( italic_x ) , italic_ψ ( italic_titalic_k ) ) = ⟨ divide start_ARG italic_ϕ ( italic_x ) end_ARG start_ARG ∥ italic_ϕ ( italic_x ) ∥2 end_ARG , divide start_ARG italic_ψ ( italic_titalic_k ) end_ARG start_ARG ∥ italic_ψ ( italic_titalic_k ) ∥2 end_ARG ⟩ . (1) Vision LLMs. Vision LLMs, such as LLaVA liu2024llavanext ; liu2023improvedllava ; liu2023llava and MiniGPT zhu2023minigpt , consist of a vision encoder ϕ:ℝd→ℝm:bold-italic-ϕ→superscriptℝsuperscriptℝ φ:R^d ^mitalic_ϕ : blackboard_Rd → blackboard_Rm, a text tokenizer, and a language model. The vision encoder and the text tokenizer have the same embedding dimension. When feeding the VLM with an image and the instruction, a vision encoder transforms this image to hidden embeddings of dimension m. The text tokenizer first tokenizes the instruction into tokens, and then looks up the embedding matrix to get its m-dimensional embedding. The image embedding and the instruction embedding are concatenated and then fed into a language model to generate a description. The vision encoder is the vision encoder in CLIP radford2021learning , and the instruction prompt is usually like “Describe this image in detail”. 2.2 Diffusion model In this section, we briefly introduce the continuous-time diffusion models song2020score . Let p⁢()p( x)p ( italic_x ) represent the underlying, unknown distribution for data points ∈ℝdsuperscriptℝ x ^ditalic_x ∈ blackboard_Rd. The core idea of diffusion models is to progressively transform samples from p⁢()p( x)p ( italic_x ) into Gaussian noise. Formally, the transformation can be formulated by the forward diffusion process ⁢(t)t∈[0,1]subscript01\ x(t)\_t∈[0,1] italic_x ( t ) t ∈ [ 0 , 1 ], which is governed by the following stochastic differential equation (SDE) in the time interval [0,1]01[0,1][ 0 , 1 ]: d⁢=⁢(,t)⁢d⁢t+g⁢(t)⁢d⁢(t),d d x= f( x,t) dt+g(t) d% w(t),d italic_x = italic_f ( italic_x , t ) d t + g ( t ) d italic_w ( t ) , (2) where :ℝd×ℝ→ℝd:→superscriptℝsuperscriptℝ f:R^d×R ^ditalic_f : blackboard_Rd × blackboard_R → blackboard_Rd stands for the drift function, g:ℝ→ℝ:→ℝg:R : blackboard_R → blackboard_R is the diffusion function, and ⁢(t)∈ℝdsuperscriptℝ w(t) ^ditalic_w ( t ) ∈ blackboard_Rd denotes the Brownian motion. Note that the diffusion process starts with ⁢(0)0 x(0)italic_x ( 0 ) drawn from the underlying data distribution p⁢()p( x)p ( italic_x ). The distribution of ⁢(t) x(t)italic_x ( t ) at any time t is pt⁢()subscriptp_t( x)pitalic_t ( italic_x ), with the initial data distribution being p0⁢()=p⁢()subscript0p_0( x)=p( x)p0 ( italic_x ) = p ( italic_x ). The functions ⁢(,t) f( x,t)italic_f ( italic_x , t ) and g⁢(t)g(t)g ( t ) are chosen carefully so that as t approaches 1111, the distribution p1⁢()subscript1p_1( x)p1 ( italic_x ) closely resembles a standard d-dimensional Gaussian distribution, ⁢(,d)0subscriptN( 0, I_d)N ( 0 , italic_Iitalic_d ). We follow the Variance Preserving (VP) SDE song2020score , where the drift and diffusion coefficients are ⁢(,t)=−12⁢β⁢(t)⁢12 f( x,t)=- 12β(t) xitalic_f ( italic_x , t ) = - divide start_ARG 1 end_ARG start_ARG 2 end_ARG β ( t ) italic_x and g⁢(t)=β⁢(t)g(t)= β(t)g ( t ) = square-root start_ARG β ( t ) end_ARG respectively. Here, β⁢(t)β(t)β ( t ) is a function that controls the noise level over time. To generate new samples, one must reverse the diffusion process. This is achieved by solving a corresponding reverse-time SDE of Eq. 2: d⁢^=[⁢(^,t)−g⁢(t)2⁢∇^log⁡pt⁢(^)]⁢d⁢t+g⁢(t)⁢d⁢¯.d^delimited-[]^superscript2subscript∇^subscript^d¯ d x=[ f( x,t)-g(t)^2% _ x p_t( x)] dt+g(t) d% w.d over start_ARG italic_x end_ARG = [ italic_f ( over start_ARG italic_x end_ARG , t ) - g ( t )2 ∇over start_ARG italic_x end_ARG log pitalic_t ( over start_ARG italic_x end_ARG ) ] d t + g ( t ) d over¯ start_ARG italic_w end_ARG . (3) The generation process starts with drawing an initial sample ^⁢(1)^1 x(1)over start_ARG italic_x end_ARG ( 1 ) from the standard Gaussian distribution ⁢(,d)0subscriptN( 0, I_d)N ( 0 , italic_Iitalic_d ). Then, by integrating this SDE from t=11t=1t = 1 down to t=00t=0t = 0, the noisy sample ^⁢(t) x(t)over start_ARG italic_x end_ARG ( t ) is progressively denoised. The goal is for the final output ^⁢(0)^0 x(0)over start_ARG italic_x end_ARG ( 0 ) to be a sample from the original data distribution p0⁢()subscript0p_0( x)p0 ( italic_x ). However, the score function ∇log⁡pt⁢()subscript∇subscript _ x p_t( x)∇italic_x log pitalic_t ( italic_x ) in Eq. 3 is usually intractable. In practice, we use a neural network, denoted by ⁢(,t)subscript s_ θ( x,t)italic_sbold_italic_θ ( italic_x , t ) and parameterized by θitalic_θ, to approximate the score function song2020score ; kingma2021variational . 3 Related work Perturbation-based attacks. Perturbation-based attacks cause models to make incorrect predictions by introducing small, often imperceptible, alterations to the input data chakraborty2021survey ; huang2017adversarial ; chakraborty2018adversarial . These attacks commonly leverage gradient-based methods to find the most vulnerable parts of an input, then craft perturbations to maximize the model’s loss. A classic illustration is adversarial image attacks, where minute pixel modifications, invisible to humans, can trick a model into misclassifying an image goodfellow2014explaining ; madry2017towards . Adversarial training. Adversarial training employs optimization techniques to bolster model robustness and safety alignment. TeCoA mao2022understanding applies supervised adversarial fine-tuning on ImageNet, while FARE schlarmann2024robust leverages an unsupervised adversarial fine-tuning approach using an embedding loss on PGD-perturbed madry2017towards inputs to enhance CLIP vision encoder robustness and zero-shot performance. Addressing challenges in such methods, Hossain et al. hossain2024sim developed Sim-CLIP, which integrates a Siamese architecture with a cosine similarity loss to align clean and perturbed representations, incorporating a stop-gradient mechanism for efficient training without negative samples. This was further extended by Sim-CLIP+ hossain2024securing , which tailors the cosine similarity loss and stop-gradient mechanism to defend VLMs against advanced optimization-based jailbreak attacks, preventing symmetric loss collapse while maintaining computational efficiency and robustness. Adversarial purification. Adversarial purification techniques shi2021online ; yoon2021adversarial offer a distinct defense paradigm by employing generative models to sanitize images from adversarial perturbations samangouei2018defense ; hill2021stochastic . The main advantage is its “plug-in” simplicity to address new threats without the need to retrain vision models—since the adversarial images being sanitized independently of attack specifics and the vision models. However, this adaptability used to be constrained by weaker performance compared to adversarial training methods croce2020reliable . The vulnerability becomes especially clear when faced with adaptive attackers who have knowledge of the defense system athalye2018obfuscated ; tramer2020adaptive , a problem generally rooted in the inherent weaknesses of the previous generative models, such as GAN goodfellow2020generative . The rise of diffusion models song2020score , known for their generative capabilities, high sample diversity and inherent stochasticity, signals a promising direction for mitigating these persistent issues. DiffPure nie2022diffusion proposed to use the diffusion model for adversarial purification. CLIPure zhang2025clipure operates directly in the CLIP latent space, correcting embeddings of adversarial examples for downstream tasks. However, previous works did not discuss the optimal number of steps for forward noise-injection. DiffPure nie2022diffusion and CLIPure zhang2025clipure both use a fixed diffusion time, which is inflexible for different adversarial examples. We propose a threshold-based stopping criterion in our paper, and therefore reduce the number of noise-injection steps and improves performance by a significant margin. 4 Our method In this section, we introduce DiffCAP, a purification mechanism that leverages forward diffusion dynamics and semantic stability to remove adversarial perturbations in VLMs. When we pass an adversarial image advsubscriptadv x_advitalic_xadv into the diffusion model, ⁢(t) x(t)italic_x ( t ) follows a forward diffusion process governed by the variance-preserving stochastic differential equation: d⁢(t)=−12⁢β⁢(t)⁢(t)⁢d⁢t+β⁢(t)⁢d⁢(t),⁢(0)=adv,formulae-sequenced12d0subscriptadv d x(t)=- 12β(t) x(t) dt+ β(t)% \, d w(t), x(0)= x_adv,d italic_x ( t ) = - divide start_ARG 1 end_ARG start_ARG 2 end_ARG β ( t ) italic_x ( t ) d t + square-root start_ARG β ( t ) end_ARG d italic_w ( t ) , italic_x ( 0 ) = italic_xadv , (4) β⁢(t)>00β(t)>0β ( t ) > 0 is a smooth noise schedule, and ⁢(t) w(t)italic_w ( t ) is a standard Wiener process. This process has a closed-form solution at time t given by: ⁢(t)=α⁢(t)⁢adv+1−α⁢(t)⁢ϵ,ϵ∼⁢(,d),formulae-sequencesubscriptadv1bold-italic-ϵsimilar-tobold-italic-ϵ0subscript x(t)= α(t)\, x_adv+ 1-α(t)\, % ε, ε ( 0, I_d),italic_x ( t ) = square-root start_ARG α ( t ) end_ARG italic_xadv + square-root start_ARG 1 - α ( t ) end_ARG italic_ϵ , italic_ϵ ∼ N ( 0 , italic_Iitalic_d ) , (5) where α⁢(t)=exp⁡(−∫0tβ⁢(s)⁢ds)superscriptsubscript0differential-dα(t)= (- _0^tβ(s)\, ds )α ( t ) = exp ( - ∫0t β ( s ) d s ). Our method builds upon the insight from randomized smoothing (cohen2019certified, ): adding Gaussian noise to adversarial inputs can recover the original predictions with high probability. We need the following basic assumptions to facilitate our analysis. Assumption 1 (Scale Invariance). We assume that the classifier hℎh (defined in Eq. 1) is scale-invariant: for any scalar λ>00λ>0λ > 0 and any input ∈ℝdsuperscriptℝ x ^ditalic_x ∈ blackboard_Rd, we have h⁢(λ⁢)=h⁢().ℎh(λ x)=h( x).h ( λ italic_x ) = h ( italic_x ) . Algorithm 1 DiffCAP: Diffusion-based Cumulative Adversarial Purification 1:Adversarially perturbed input image advsubscriptadv x_advitalic_xadv; An image encoder ϕ⁢(⋅)bold-italic-ϕ⋅ φ(·)italic_ϕ ( ⋅ ) (e.g., from VLM); Similarity threshold τ; Pretrained diffusion denoiser D⁢(⋅)⋅D(·)D ( ⋅ ); Maximum number of forward diffusion steps T. 2:Purified image cleansubscriptclean x_cleanitalic_xclean. 3:Initialize step counter t←0←0t← 0t ← 0. 4:Set initial image 0←adv←subscript0subscriptadv x_0← x_advitalic_x0 ← italic_xadv. 5:Calculate initial embedding prev←ϕ⁢(0)←subscriptprevbold-italic-ϕsubscript0 e_prev← φ( x_0)italic_eprev ← italic_ϕ ( italic_x0 ). 6:while t<11t<1t < 1 do ▷ ▷ Iteratively inject noise and check stability 7: Inject noise into the images based on Eq. 5 to obtain t+1/Tsubscript1 x_t+1/Titalic_xitalic_t + 1 / T. 8: Calculate current embedding curr←ϕ⁢(t+1/T)←subscriptcurrbold-italic-ϕsubscript1 e_curr← φ( x_t+1/T)italic_ecurr ← italic_ϕ ( italic_xitalic_t + 1 / T ). 9: if cos⁡(curr,prev)≥τsubscriptcurrsubscriptprev ( e_curr, e_prev)≥ ( italic_ecurr , italic_eprev ) ≥ τ then ▷ ▷ Check if embeddings have stabilized 10: break ▷ ▷ Exit loop, stabilization reached 11: end if 12: Update previous embedding: prev←curr←subscriptprevsubscriptcurr e_prev← e_curritalic_eprev ← italic_ecurr, update t←t+1/T←1t← t+1/Tt ← t + 1 / T. 13:end while 14:Set the stabilized (but potentially noisy) image stable←t←subscriptstablesubscript x_stable← x_titalic_xstable ← italic_xitalic_t. 15:Denoise the stabilized image using the diffusion model: clean←D⁢(stable)←subscriptcleansubscriptstable x_clean← D( x_stable)italic_xclean ← D ( italic_xstable ). 16:return cleansubscriptclean x_cleanitalic_xclean. Remark 1. Asm. 1 is standard in deep learning theory and practically justified for models as we can always scale the input (zhang2020over, ; wu2024robust, ). We now present our main result that establishes a certified recovery region under forward diffusion for the adversarial image. The proof can be found at Sec. A.1. Theorem 1 (Certified recovery region under forward diffusion). Let h:ℝd→[K]:ℎ→superscriptℝdelimited-[]h:R^d→[K]h : blackboard_Rd → [ K ] be the classifier. Let adv=+ϵadvsubscriptadvsubscriptbold-ϵadv x_adv= x+ ε_advitalic_xadv = italic_x + italic_ϵadv be an adversarial example with perturbation ϵadvsubscriptbold-ϵadv ε_advitalic_ϵadv, and let ⁢(t) x(t)italic_x ( t ) be the solution to the forward diffusion process defined in Eq. 5 with a linear noise schedule β⁢(t)=βmin+(βmax−βmin)⁢tsubscriptsubscriptsubscriptβ(t)= _ +( _ - _ )tβ ( t ) = βroman_min + ( βroman_max - βroman_min ) t. Suppose there exists p1¯,p2¯,k1¯subscript1¯subscript2subscript1 p_1, p_2,k_1under¯ start_ARG p1 end_ARG , over¯ start_ARG p2 end_ARG , k1 such that for all t∈[0,1]01t∈[0,1]t ∈ [ 0 , 1 ] ℙ⁢(h⁢(+ϵ′⁢(t))=k1)≥p1¯>p2¯≥maxk≠k1⁡ℙ⁢(h⁢(+ϵ′⁢(t))=k),ℙℎsuperscriptbold-italic-ϵ′subscript1¯subscript1¯subscript2subscriptsubscript1ℙℎsuperscriptbold-italic-ϵ′ (h( x+ ε (t))=k_1)≥% p_1> p_2≥ _k≠ k_1P(h( x+% ε (t))=k)\,,blackboard_P ( h ( italic_x + italic_ϵ′ ( t ) ) = k1 ) ≥ under¯ start_ARG p1 end_ARG > over¯ start_ARG p2 end_ARG ≥ maxitalic_k ≠ k start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT blackboard_P ( h ( italic_x + italic_ϵ′ ( t ) ) = k ) , (6) where ϵ′⁢(t)∼⁢(,1−α⁢(t)α⁢(t)⁢d)similar-tosuperscriptbold-ϵ′01subscript ε (t) ( 0, 1-α(t)α(t)% I_d)italic_ϵ′ ( t ) ∼ N ( 0 , divide start_ARG 1 - α ( t ) end_ARG start_ARG α ( t ) end_ARG italic_Iitalic_d ). Define tmin=2⁢Mβmin2+2⁢(βmax−βmin)⁢M+βmin,where⁢M:=log⁡(1+(2⁢‖ϵadv‖2ϕ−1⁢(p1¯)−ϕ−1⁢(p2¯))2).formulae-sequencesubscript2superscriptsubscript22subscriptsubscriptsubscriptassignwhere1superscript2subscriptnormsubscriptbold-italic-ϵadv2superscriptbold-italic-ϕ1¯subscript1superscriptbold-italic-ϕ1¯subscript22t_ = 2M _ ^2+2( _ - _ )M+β% _ , where\ M:= (1+ ( 2\| ε_% adv\|_2 φ^-1( p_1)- φ^-1( p_2% ) )^2 )\,.troman_min = divide start_ARG 2 M end_ARG start_ARG square-root start_ARG βroman_min2 + 2 ( βroman_max - βroman_min ) M end_ARG + βroman_min end_ARG , where M := log ( 1 + ( divide start_ARG 2 ∥ italic_ϵadv ∥2 end_ARG start_ARG italic_ϕ- 1 ( under¯ start_ARG p1 end_ARG ) - italic_ϕ- 1 ( over¯ start_ARG p2 end_ARG ) end_ARG )2 ) . Then, under Asm. 1, when βmin≥Msubscript _ ≥ Mβroman_min ≥ M, for all tmin≤t≤1subscript1t_ ≤ t≤ 1troman_min ≤ t ≤ 1, we have arg⁡maxk∈[K]⁡ℙ⁢(h⁢(⁢(t))=k)=k1subscriptdelimited-[]ℙℎsubscript1 _k∈[K]\;P(h( x(t))=k)=k_1arg maxitalic_k ∈ [ K ] blackboard_P ( h ( italic_x ( t ) ) = k ) = k1, i.e., the adversarial example is classified as its original label k1subscript1k_1k1 after sufficient forward diffusion. Remark 2. Thm. 1 indicates that an adversarially perturbed image with added noise will eventually be classified correctly. Then, we can expect to use the diffusion model to remove the added noise to obtain the clean image. Upon this point, we start to analyze the dynamics in the VLM embedding space during the forward diffusion process. In Thm. 1, we prove a certified recovery region, indicating the local smoothness of ϕ⁢(⁢(t))bold-italic-ϕ φ( x(t))italic_ϕ ( italic_x ( t ) ) for t≥tminsubscriptt≥ t_ t ≥ troman_min. Therefore, we pose a local Lipschitz assumption after time tminsubscriptt_ troman_min: Assumption 2. We assume ϕbold-ϕ φitalic_ϕ is L−limit-fromL-L -Lipschitz for t≥tminsubscriptt≥ t_ t ≥ troman_min ‖ϕ⁢(⁢(t))−ϕ⁢(⁢(t′))‖2≤L⁢‖⁢(t)−⁢(t′)‖2∀t,t′≥tmin.formulae-sequencesubscriptnormbold-italic-ϕbold-italic-ϕsuperscript′2subscriptnormsuperscript′2for-allsuperscript′subscript \| φ( x(t))- φ( x(t ))\|_2% ≤ L\| x(t)- x(t )\|_2 ∀ t,t ≥ t_% \,.∥ italic_ϕ ( italic_x ( t ) ) - italic_ϕ ( italic_x ( t′ ) ) ∥2 ≤ L ∥ italic_x ( t ) - italic_x ( t′ ) ∥2 ∀ t , t′ ≥ troman_min . Lemma 1. Under the same setting as in Thm. 1 and Asm. 2, let ⁢(t) x(t)italic_x ( t ) be defined as in Eq. 5. Then for any t1,t2∈[0,1]subscript1subscript201t_1,t_2∈[0,1]t1 , t2 ∈ [ 0 , 1 ], as t1,t2→1→subscript1subscript21t_1,t_2→ 1t1 , t2 → 1, we have: ⁢[‖ϕ⁢(⁢(t1))−ϕ⁢(⁢(t2))‖2]→0.→delimited-[]subscriptnormbold-ϕsubscript1bold-ϕsubscript220E [\| φ( x(t_1))- φ( x(t_2))\|_2% ]→ 0.blackboard_E [ ∥ italic_ϕ ( italic_x ( t1 ) ) - italic_ϕ ( italic_x ( t2 ) ) ∥2 ] → 0 . The proof is deferred to Sec. A.2. Lemma 1 shows that embeddings converge during forward diffusion, which motivates our semantic stopping criterion based on cosine similarity. We quantify this convergence rate in the following theorem: Theorem 2. Let ⁢(t) x(t)italic_x ( t ) be defined as in Eq. 5. Then for small δ>00δ>0δ > 0, ⁢[‖ϕ⁢(⁢(t))−ϕ⁢(⁢(t+δ))‖2]=O⁢(L⋅δ⋅β⁢(t)⁢α⁢(t)1−α⁢(t)), for ⁢t∈[tmin,1−δ],formulae-sequencedelimited-[]subscriptdelimited-∥bold-italic-ϕbold-italic-ϕ2⋅1 for subscript1 splitE [\| φ( x(t))- φ( x(t+% δ))\|_2 ]=O (L·δ·β(t) α(t)1% -α(t) ),\ for\ t∈[t_ ,1-δ]\,, splitstart_ROW start_CELL blackboard_E [ ∥ italic_ϕ ( italic_x ( t ) ) - italic_ϕ ( italic_x ( t + δ ) ) ∥2 ] = O ( L ⋅ δ ⋅ β ( t ) square-root start_ARG divide start_ARG α ( t ) end_ARG start_ARG 1 - α ( t ) end_ARG end_ARG ) , for t ∈ [ troman_min , 1 - δ ] , end_CELL end_ROW (7) where α⁢(t)=exp⁡(−∫0tβ⁢(s)⁢ds)superscriptsubscript0differential-dα(t)= (- _0^tβ(s)\, ds )α ( t ) = exp ( - ∫0t β ( s ) d s ). Moreover, by using a common linear noise schedule β⁢(t)=βmin+(βmax−βmin)⁢tsubscriptsubscriptsubscriptβ(t)= _ +( _ - _ )tβ ( t ) = βroman_min + ( βroman_max - βroman_min ) t with βmin>0subscript0 _ >0βroman_min > 0 and βmax>βminsubscriptsubscript _ > _ βroman_max > βroman_min. Then β⁢(t)⋅α⁢(t)1−α⁢(t)⋅1β(t)· α(t)1-α(t)β ( t ) ⋅ square-root start_ARG divide start_ARG α ( t ) end_ARG start_ARG 1 - α ( t ) end_ARG end_ARG is strictly decreasing for all t∈[tmin,1)subscript1t∈[t_ ,1)t ∈ [ troman_min , 1 ). The proof is deferred to Sec. A.3. Thm. 2 quantifies the semantic change between adjacent forward diffusion steps. The bound in Eq. 7 decreases as t→1→1t→ 1t → 1 under the common linear noise schedule. Consequently, the expected semantic change between adjacent forward diffusion steps diminishes as the process approaches terminal time. Our results suggest a simple yet powerful strategy: inject Gaussian noise until the semantic embedding stabilizes. We call this algorithm DiffCAP, summarized in Alg. 1. The cumulative diffusion continues until the cosine similarity between consecutive embeddings exceeds a threshold τ. A pretrained diffusion model is then applied in reverse to recover a clean image from the stabilized noisy input. In Alg. 2, we describe how to select the threshold τ. The core idea is to iteratively inject noise into both clean and adversarial images and track the cosine similarity between the embeddings of one clean or adversarial image after two consecutive steps of noise injection. This is done for a collection of image pairs to reduce randomness. The process continues until the set of similarity scores for adversarial images and the set for clean images are statistically indistinguishable (i.e., likely from the same underlying distribution). At this point, the algorithm determines that the noise has reached a level where the embeddings’ stability is comparable for both types of images. Algorithm 2 Adaptive Similarity Threshold (τ) Calculation 1:The dataset of clean-adversarial image pairs Dpairs=(clean,adv)subscriptpairssubscriptcleansubscriptadvD_pairs=\( x_clean, x_adv)\Dpairs = ( italic_xclean , italic_xadv ) ; The embedding function ϕ⁢(⋅)bold-italic-ϕ⋅ φ(·)italic_ϕ ( ⋅ ). Maximum number of forward diffusion steps T. 2:The similarity threshold τ. 3:Initialize step counter t←0←0t← 0t ← 0. 4:for all pair (clean,adv)∈Dpairssubscriptcleansubscriptadvsubscriptpairs( x_clean, x_adv)∈ D_pairs( italic_xclean , italic_xadv ) ∈ Dpairs do 5: Set initial image 0,clean←clean←subscript0cleansubscriptclean x_0,clean← x_cleanitalic_x0 , clean ← italic_xclean, 0,adv←adv←subscript0advsubscriptadv x_0,adv← x_advitalic_x0 , adv ← italic_xadv. 6: Calculate initial embedding prev,clean←ϕ⁢(0,clean),prev,adv←ϕ⁢(0,adv)formulae-sequence←subscriptprevcleanbold-italic-ϕsubscript0clean←subscriptprevadvbold-italic-ϕsubscript0adv e_prev,clean← φ( x_0,clean% ), e_prev,adv← φ( x_0,adv)italic_eprev , clean ← italic_ϕ ( italic_x0 , clean ) , italic_eprev , adv ← italic_ϕ ( italic_x0 , adv ). 7:end for 8:while t<11t<1t < 1 do ▷ ▷ Iteratively inject noise and check stability 9: Initialize total similarity set Sclean←[],Sadv←[]formulae-sequence←subscriptclean←subscriptadvS_clean←[],S_adv←[]Sclean ← [ ] , Sadv ← [ ]. 10: for all pair (clean,adv)∈Dpairssubscriptcleansubscriptadvsubscriptpairs( x_clean, x_adv)∈ D_pairs( italic_xclean , italic_xadv ) ∈ Dpairs do 11: Inject noise into the images based on Eq. 5 to obtain t+1/T,clean,t+1/T,advsubscript1cleansubscript1adv x_t+1/T,clean, x_t+1/T,advitalic_xitalic_t + 1 / T , clean , italic_xitalic_t + 1 / T , adv 12: Calculate current embedding curr,clean←ϕ⁢(t+1/T,clean),curr,adv←ϕ⁢(t+1/T,adv)formulae-sequence←subscriptcurrcleanbold-italic-ϕsubscript1clean←subscriptcurradvbold-italic-ϕsubscript1adv e_curr,clean← φ( x_t+1/T,% clean), e_curr,adv← φ( x_t+1/T,% adv)italic_ecurr , clean ← italic_ϕ ( italic_xitalic_t + 1 / T , clean ) , italic_ecurr , adv ← italic_ϕ ( italic_xitalic_t + 1 / T , adv ). 13: Calculate similarity sclean←cos⁡(curr,clean,prev,clean),sadv←cos⁡(curr,adv,prev,adv)formulae-sequence←subscriptcleansubscriptcurrcleansubscriptprevclean←subscriptadvsubscriptcurradvsubscriptprevadvs_clean← ( e_curr,clean, e_% prev,clean),s_adv← ( e_curr,% adv, e_prev,adv)sclean ← cos ( italic_ecurr , clean , italic_eprev , clean ) , sadv ← cos ( italic_ecurr , adv , italic_eprev , adv ). 14: Add the similarity score to the set Sclean←Sclean∪sclean←subscriptcleansubscriptcleansubscriptcleanS_clean← S_clean∪\s_clean\Sclean ← Sclean ∪ sclean , Sadv←Sadv∪sadv←subscriptadvsubscriptadvsubscriptadvS_adv← S_adv∪\s_adv\Sadv ← Sadv ∪ sadv . 15: end for 16: if SadvsubscriptadvS_advSadv and ScleansubscriptcleanS_cleanSclean are from the same underlying distribution then 17: break 18: end if 19: for all pair (clean,adv)∈Dpairssubscriptcleansubscriptadvsubscriptpairs( x_clean, x_adv)∈ D_pairs( italic_xclean , italic_xadv ) ∈ Dpairs do 20: Update previous embedding: prev,clean←curr,clean,prev,adv←curr,advformulae-sequence←subscriptprevcleansubscriptcurrclean←subscriptprevadvsubscriptcurradv e_prev,clean← e_curr,clean% , e_prev,adv← e_curr,advitalic_eprev , clean ← italic_ecurr , clean , italic_eprev , adv ← italic_ecurr , adv. 21: end for 22: Update t←t+1/T←1t← t+1/Tt ← t + 1 / T. 23:end while 24:return τ←mean⁢(Sclean)←meansubscriptcleanτ← mean(S_clean)τ ← mean ( Sclean ). 5 Experiments 5.1 Settings Models, datasets & metrics. We evaluate DiffCAP across three vision-language tasks: image captioning (IC), visual question answering (VQA), and zero-shot classification (ZSC). For IC and VQA, we adopt two large VLMs—OpenFlamingo (OF) awadalla2023openflamingo with 9999B parameters and LLaVA-1.5 liu2024improved with 7777B parameters. For ZSC, we utilize CLIP radford2021learning with 88888888M parameters as the backbone model. Our experiments are conducted on standard benchmarks: COCO lin2014microsoft and Flickr30k plummer2015flickr30k for IC, VQAv2 goyal2017making and TextVQA singh2019towards for VQA, and CalTech101 li_andreeto_ranzato_perona_2022 and ImageNet1K deng2009imagenet for ZSC. For both adversarial and clean evaluation, we randomly sample 500500500500 images for IC and VQA, while 1,00010001,0001 , 000 images are chosen for ZSC. We report Consensus-based Image Description Evaluation (CIDEr) vedantam2015cider score for IC, VQA accuracy antol2015vqa for VQA, and top-1 accuracy for ZSC. Attacks. The attack is conducted in the gray-box setting, where the adversary can access the gradients of the model but has no knowledge of the defense pipeline. For IC and VQA, we adopt a two-stage attack pipeline following schlarmann2023adversarial . In the first stage, we apply 100100100100-step Auto-PGD (APGD) attacks croce2020reliable in half-precision using multiple ground-truth captions or answers as supervision. Samples that fall below a predefined performance threshold are excluded from further attacks. In the second stage, we conduct stronger single-precision APGD attacks on the remaining samples. This progressive strategy maximizes adversarial impact while remaining computationally efficient. For ZSC, we follow the AutoAttack framework, employing APGD with cross-entropy loss and targeted Difference of Logits Ratio (DLR) loss with 100100100100 iterations, respectively. Baselines. We compare DiffCAP with two categories of adversarial defense methods. The first category includes adversarially fine-tuned vision encoders. Since both OF and LLaVA adopt CLIP as their vision backbone, we replace their CLIP vision encoder with two robust variants: TeCoA mao2022understanding and FARE schlarmann2024robust . TeCoA applies supervised adversarial training, while FARE employs an unsupervised loss. The second category includes purification methods. We consider: JPEG-DL salamah2024jpeg , the trainable JPEG compression layer to remove adversarial perturbations; DiffPure nie2022diffusion , the first approach leveraging the diffusion process to recover the clean image in the pixel space; CLIPure zhang2025clipure , the latest method that operates directly in the CLIP latent space. Hyperparameters. The algorithms are implemented through PyTorch, and all experiments are conducted on an NVIDIA A100 GPU. By default, we use the ViT-B/32 CLIP vision encoder to ensure computational efficiency. We take advantage of the pre-trained diffusion model from dhariwal2021diffusion . Following Alg. 2, we determine the threshold τ=0.960.96τ=0.96τ = 0.96 on subsets comprising 100100100100 random images from each of the datasets mentioned above. For all the experiments, we fix the threshold. More experimental details can be found in the supplementary material. 5.2 Result analysis Table 1: CIDEr score of two VLMs in IC task on two datasets with clean images and adversarial perturbations of two sizes under different defenses. 2222 and 4444 with TeCoA and FARE suggest the version that is fine-tuned by ℓ∞2/255superscriptsubscriptℓ2255 _∞^2/255ℓ∞2 / 255 and ℓ∞4/255superscriptsubscriptℓ4255 _∞^4/255ℓ∞4 / 255 bounded adversarial examples, respectively. The best result is in bold and the runner-up is underlined. Defense OF-9B awadalla2023openflamingo LLaVA 1.5-7B liu2024improved COCO lin2014microsoft Flickr30k plummer2015flickr30k COCO lin2014microsoft Flickr30k plummer2015flickr30k clean ℓ∞2/255superscriptsubscriptℓ2255 _∞^2/255ℓ∞2 / 255 ℓ∞4/255superscriptsubscriptℓ4255 _∞^4/255ℓ∞4 / 255 clean ℓ∞2/255superscriptsubscriptℓ2255 _∞^2/255ℓ∞2 / 255 ℓ∞4/255superscriptsubscriptℓ4255 _∞^4/255ℓ∞4 / 255 clean ℓ∞2/255superscriptsubscriptℓ2255 _∞^2/255ℓ∞2 / 255 ℓ∞4/255superscriptsubscriptℓ4255 _∞^4/255ℓ∞4 / 255 clean ℓ∞2/255superscriptsubscriptℓ2255 _∞^2/255ℓ∞2 / 255 ℓ∞4/255superscriptsubscriptℓ4255 _∞^4/255ℓ∞4 / 255 No defense 79.7 1.5 1.1 60.1 0.7 0.4 115.5 4.0 3.1 77.5 1.6 1.0 1-13 TeCoA2 mao2022understanding 73.5 31.6 21.2 49.5 14.1 9.5 98.4 44.2 30.3 57.1 23.2 15.3 FARE2 schlarmann2024robust 79.1 34.2 19.5 57.7 16.4 8.9 109.9 53.6 31.0 71.1 29.5 17.5 TeCoA4 mao2022understanding 66.9 28.5 21.6 40.9 12.0 10.3 88.3 50.9 35.3 48.6 27.9 19.5 FARE4 schlarmann2024robust 74.1 30.9 22.8 51.4 15.7 10.5 102.4 57.1 40.9 61.6 31.4 22.8 1-13 JPEG-DL salamah2024jpeg 78.2 66.1 43.9 58.8 47.6 30.7 113.3 106.4 77.2 74.8 69.6 47.9 DiffPure nie2022diffusion 74.9 73.4 72.3 49.8 49.2 50.3 106.5 108.4 105.0 65.5 66.4 63.2 CLIPure zhang2025clipure 80.8 6.6 5.3 59.3 4.7 3.5 115.1 4.9 3.4 76.9 2.1 1.5 DiffCAP 81.4 79.3 78.4 55.6 56.7 57.2 120.4 119.6 116.9 75.0 72.7 72.1 Image captioning. As shown in Tab. 1, VLMs are highly vulnerable to adversarial perturbations: even 2/25522552/2552 / 255 attacks can reduce CIDEr scores close to zero. Adversarial training methods (TeCoA and FARE) provide moderate robustness improvements. However, their effectiveness drops significantly under 4/25542554/2554 / 255 attacks. Notably, TeCoA and FARE also degrade the clean performance, especially on Flickr30k. JPEG-DL shows better robustness than TeCoA and FARE, but remains sensitive to perturbation strength. DiffPure substantially improves robustness, lifting performance under both 2/25522552/2552 / 255 and 4/25542554/2554 / 255 attacks to levels comparable with clean conditions. DiffCAP consistently outperforms all baselines across both models and datasets. It achieves significantly better robustness than DiffPure under both attack strengths, and requires less than 1/3131/31 / 3 of DiffPure’s denoising runtime (suggested by Fig. 2). For instance, DiffCAP improves CIDEr scores by over 10%percent1010\%10 % with OF on Flickr30k and with LLaVA on COCO compared to DiffPure. Furthermore, DiffCAP maintains or even improves clean performance, demonstrating strong fidelity preservation. Lastly, CLIPure performs poorly in this task. Its limited effectiveness likely stems from token-level misalignment: purifying only the [CLS] token embedding fails to influence generation-related latent tokens, which dominate the captioning process. Table 2: Accuracy (%percent\%%) of two VLMs in VQA task on two datasets with clean images and adversarial perturbations of two sizes under different defenses. 2222 and 4444 with TeCoA and FARE suggest the version that is fine-tuned by ℓ∞subscriptℓ _∞ℓ∞-norm bounded adversarial examples with ϵ=2/255italic-ϵ2255ε=2/255ϵ = 2 / 255 and with ϵ=4/255italic-ϵ4255ε=4/255ϵ = 4 / 255, respectively. The best result is in bold and the runner-up is underlined. Defense OF-9B awadalla2023openflamingo LLaVA 1.5-7B liu2024improved TextVQA singh2019towards VQAv2 goyal2017making TextVQA singh2019towards VQAv2 goyal2017making clean ℓ∞2/255superscriptsubscriptℓ2255 _∞^2/255ℓ∞2 / 255 ℓ∞4/255superscriptsubscriptℓ4255 _∞^4/255ℓ∞4 / 255 clean ℓ∞2/255superscriptsubscriptℓ2255 _∞^2/255ℓ∞2 / 255 ℓ∞4/255superscriptsubscriptℓ4255 _∞^4/255ℓ∞4 / 255 clean ℓ∞2/255superscriptsubscriptℓ2255 _∞^2/255ℓ∞2 / 255 ℓ∞4/255superscriptsubscriptℓ4255 _∞^4/255ℓ∞4 / 255 clean ℓ∞2/255superscriptsubscriptℓ2255 _∞^2/255ℓ∞2 / 255 ℓ∞4/255superscriptsubscriptℓ4255 _∞^4/255ℓ∞4 / 255 No defense 23.8 0.0 0.0 48.5 1.8 0.0 37.1 0.5 0.0 74.5 2.9 0.0 1-13 TeCoA2 mao2022understanding 16.6 3.5 2.1 46.2 23.5 20.5 24.1 12.1 8.8 66.9 33.8 21.8 FARE2 schlarmann2024robust 21.6 4.1 1.9 47.0 24.0 17.2 31.9 14.7 9.1 71.7 34.9 23.0 TeCoA4 mao2022understanding 15.4 2.1 1.8 44.8 23.6 21.3 20.7 12.6 9.3 63.2 41.0 31.7 FARE4 schlarmann2024robust 18.6 3.4 2.9 46.1 23.6 21.0 27.6 15.8 10.9 68.3 40.7 30.5 1-13 JPEG-DL salamah2024jpeg 23.4 15.9 13.1 46.8 39.5 32.4 34.6 27.2 21.1 68.8 60.8 45.8 DiffPure nie2022diffusion 13.6 13.2 13.5 45.1 43.5 43.6 20.9 22.0 22.2 67.3 65.8 66.0 CLIPure zhang2025clipure 20.5 6.8 8.8 47.3 18.8 17.5 36.1 2.1 1.4 73.3 4.6 2.1 DiffCAP 18.6 16.2 16.7 46.3 45.4 45.3 28.3 29.0 28.9 70.3 69.1 68.5 Table 3: Top-1 accuracy (%percent\%%) of CLIP in ZSC task with clean images and adversarial perturbations of two sizes under different defenses. 2222 and 4444 with TeCoA and FARE suggest the version that is fine-tuned by ℓ∞subscriptℓ _∞ℓ∞-norm bounded adversarial examples, respectively. The best result is in bold and the runner-up is underlined. Defense CalTech101 li_andreeto_ranzato_perona_2022 ImageNet1K deng2009imagenet clean ℓ∞2/255superscriptsubscriptℓ2255 _∞^2/255ℓ∞2 / 255 ℓ∞4/255superscriptsubscriptℓ4255 _∞^4/255ℓ∞4 / 255 clean ℓ∞2/255superscriptsubscriptℓ2255 _∞^2/255ℓ∞2 / 255 ℓ∞4/255superscriptsubscriptℓ4255 _∞^4/255ℓ∞4 / 255 No defense 83.3 0.0 0.0 87.9 0.0 0.0 1-7 TeCoA2 mao2022understanding 80.7 70.2 57.4 80.1 58.8 36.7 FARE2 schlarmann2024robust 84.8 73.0 46.6 85.5 56.5 25.6 TeCoA4 mao2022understanding 78.4 69.7 60.9 74.3 59.2 41.9 FARE4 schlarmann2024robust 84.7 76.7 64.1 80.2 61.6 40.6 1-7 JPEG-DL salamah2024jpeg 83.9 68.5 33.4 87.8 48.5 16.4 DiffPure nie2022diffusion 83.6 83.2 83.1 81.0 79.7 79.7 CLIPure zhang2025clipure 82.9 80.8 80.1 87.7 85.4 84.6 DiffCAP 82.6 81.7 80.9 87.2 84.4 81.1 Table 4: Top-1 accuracy (%percent\%%) in ZSC task with clean images and adversarial perturbations. We use different CLIP vision encoders for DiffCAP. Numbers in parentheses denote parameters in M. Encoder clean ℓ∞2/255superscriptsubscriptℓ2255 _∞^2/255ℓ∞2 / 255 ℓ∞4/255superscriptsubscriptℓ4255 _∞^4/255ℓ∞4 / 255 CalTech li_andreeto_ranzato_perona_2022 RN50 (102) 82.8 82.2 82.5 RN101 (123) 83.2 83.1 81.2 ViT-B/32 (88) 82.6 81.7 80.9 ViT-B/16 (149) 83.0 82.3 80.9 ViT-L/14 (304) 82.1 82.4 81.5 ImageNet deng2009imagenet RN50 (102) 84.2 82.9 80.8 RN101 (123) 86.7 85.5 84.1 ViT-B/32 (88) 87.2 84.4 81.1 ViT-B/16 (149) 84.7 85.0 82.2 ViT-L/14 (304) 85.5 83.2 81.4 Visual question answering. The results in Tab. 2 mirror the trends observed in Tab. 1. DiffCAP consistently delivers the strongest performance across all attack settings compared to all baselines in both datasets and models. The improvement is particularly notable with LLaVA on TextVQA, where DiffCAP surpasses DiffPure by over 30%percent3030\%30 %. Remarkably, on the TextVQA dataset, DiffPure performs worse than JPEG-DL, indicating that visual reasoning tasks depend more heavily on fine-grained visual features, which are susceptible to over-smoothing or distortion during purification. This underscores the importance of preserving semantic fidelity when applying generative models for purification. DiffCAP addresses this by dynamically calculating the minimal diffusion time required to remove adversarial noise for each individual image, thereby achieving a better trade-off between robustness and feature integrity for multi-hop reasoning tasks. Zero-shot classification. From Tab. 4, we observe that JPEG-DL underperforms compared to TeCoA and FARE, particularly under stronger attacks. CLIPure recovers its defense effectiveness, as only the [CLS] token is involved in prediction and no generative decoding is required. On CalTech101, DiffCAP outperforms CLIPure under attacks, and on ImageNet1K, DiffCAP achieves higher robustness than DiffPure. These results confirm the generalizability of DiffCAP: it not only excels in complex vision-language tasks requiring rich semantics but also delivers stable performance on standard classification benchmarks with more sparse semantic demands. Figure 2: The box plot of DiffCAP’s diffusion time t (y-axis) before existing noise injection loop. The dashline (t=0.0750.075t=0.075t = 0.075) is DiffPure’s noise injection time. DiffCAP requires significantly smaller diffusion time than DiffPure. The dots mark outliers and rhombuses mark mean values. Fig. 3 showcases the purification outcome of DiffCAP in IC, VQA, and ZSC scenarios. As a generative adversarial purification method, it introduces no noticeable degradation in fidelity. DiffCAP prominently mitigates the tension between robustness, efficiency, and image quality, establishing a new state-of-the-art among both purification- and training-based defenses for VLMs. Figure 3: Adversarial examples and their DiffCAP purified counterparts under different tasks. Ground-truth labels are shown in black text. Ablation study and adaptive attack. We conduct systematic ablation studies to validate the utility of DiffCAP. Tab. 4 presents results obtained by replacing the vision encoder in DiffCAP with different CLIP backbones. The results illustrate that DiffCAP is largely insensitive to the choice of vision encoder, maintaining robustness across all variants. Further ablations on the diffusion step size and threshold selection are provided in the supplementary material. We also analyze the performance of DiffCAP under adaptive attacks where the adversary simulates randomness athalye2018synthesizing and gradient obfuscation athalye2018obfuscated , with results and discussions supplemented. 6 Conclusion In conclusion, this paper proposes DiffCAP, an efficient and theoretically inspired defense strategy for VLMs, supported by a certified recovery region under forward diffusion. By leveraging cumulative Gaussian noise injection and a semantic similarity-based stopping criterion, DiffCAP dynamically identifies the minimal purification steps required before denoising, substantially reducing computational overhead while maintaining high fidelity. DiffCAP consistently outperforms existing defenses across diverse tasks, models, and datasets, as demonstrated in extensive experiments. Limitations. While our method is highly effective in the vision modality, an important limitation is its current reliance on image-based diffusion models. Extending the cumulative purification framework to text or multimodal diffusion processes remains an open direction, potentially broadening its applicability to text attack or joint vision-text adversarial threats. Acknowledgement This work was financially supported by the Swedish Wireless Innovation Network (SweWIN) approved by the Swedish Innovation Agency (VINNOVA). The computations were enabled by the resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS), partially funded by the Swedish Research Council. This work was also supported by the Swiss National Science Foundation (SNSF) under grant number 200021_205011. Research was sponsored by the Army Research Office and was accomplished under Grant Number W911NF-24-1-0048. This work was supported by Hasler Foundation Program: Hasler Responsible AI (project number 21043). References [1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736, 2022. [2] Maksym Andriushchenko and Nicolas Flammarion. Understanding and improving fast adversarial training. Advances in Neural Information Processing Systems, 33:16048–16059, 2020. [3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015. [4] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International conference on machine learning, pages 274–283. PMLR, 2018. [5] Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. In International conference on machine learning, pages 284–293. PMLR, 2018. [6] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390, 2023. [7] Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Irena Gao, Pang Wei W Koh, Daphne Ippolito, Florian Tramer, and Ludwig Schmidt. Are aligned neural networks adversarially aligned? Advances in Neural Information Processing Systems, 36, 2024. [8] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pages 39–57. Ieee, 2017. [9] Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay. Adversarial attacks and defences: A survey. arXiv preprint arXiv:1810.00069, 2018. [10] Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay. A survey on adversarial attacks and defences. CAAI TIT, pages 25–45, 2021. [11] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In international conference on machine learning, pages 1310–1320. PMLR, 2019. [12] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pages 2206–2216. PMLR, 2020. [13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. [14] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021. [15] Hadi M Dolatabadi, Sarah Erfani, and Christopher Leckie. ℓ∞subscriptℓ _∞ℓ∞-robustness and beyond: Unleashing efficient adversarial training. In European Conference on Computer Vision, pages 467–483. Springer, 2022. [16] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020. [17] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014. [18] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017. [19] Mitch Hill, Jonathan Craig Mitchell, and Song-Chun Zhu. Stochastic security: Adversarial defense using long-run dynamics of energy-based models. In International Conference on Learning Representations, 2021. [20] Md Zarif Hossain and Ahmed Imteaj. Securing vision-language models with a robust encoder against jailbreak and adversarial attacks. arXiv preprint arXiv:2409.07353, 2024. [21] Md Zarif Hossain and Ahmed Imteaj. Sim-clip: Unsupervised siamese adversarial fine-tuning for robust and semantically-rich vision-language models. arXiv preprint arXiv:2407.14971, 2024. [22] Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. arXiv preprint arXiv:1702.02284, 2017. [23] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904–4916. PMLR, 2021. [24] Haibo Jin, Leyang Hu, Xinuo Li, Peiyan Zhang, Chonghan Chen, Jun Zhuang, and Haohan Wang. Jailbreakzoo: Survey, landscapes, and horizons in jailbreaking large language and vision-language models. arXiv preprint arXiv:2407.01599, 2024. [25] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696–21707, 2021. [26] Cassidy Laidlaw, Sahil Singla, and Soheil Feizi. Perceptual adversarial robustness: Defense against unseen threat models. arXiv preprint arXiv:2006.12655, 2020. [27] Fei-Fei Li, Marco Andreeto, Marc’Aurelio Ranzato, and Pietro Perona. Caltech 101, Apr 2022. [28] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888–12900. PMLR, 2022. [29] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part X 16, pages 121–137. Springer, 2020. [30] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. [31] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014. [32] Daizong Liu, Mingyu Yang, Xiaoye Qu, Pan Zhou, Yu Cheng, and Wei Hu. A survey of attacks on large vision-language models: Resources, advances, and future trends. arXiv preprint arXiv:2407.07403, 2024. [33] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023. [34] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. [35] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. [36] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. [37] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32, 2019. [38] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017. [39] Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. Understanding zero-shot adversarial robustness for large-scale models. arXiv preprint arXiv:2212.07016, 2022. [40] Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734, 2021. [41] Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification. arXiv preprint arXiv:2205.07460, 2022. [42] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641–2649, 2015. [43] Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Henderson, Mengdi Wang, and Prateek Mittal. Visual adversarial examples jailbreak aligned large language models. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 21527–21536, 2024. [44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021. [45] Ahmed H Salamah, Kaixiang Zheng, Yiwen Liu, and En-Hui Yang. Jpeg inspired deep learning. arXiv preprint arXiv:2410.07081, 2024. [46] Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-gan: Protecting classifiers against adversarial attacks using generative models. In International Conference on Learning Representations, 2018. [47] Christian Schlarmann and Matthias Hein. On the adversarial robustness of multi-modal foundation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3677–3685, 2023. [48] Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust clip: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. arXiv preprint arXiv:2402.12336, 2024. [49] Changhao Shi, Chester Holtz, and Gal Mishne. Online adversarial purification based on self-supervision. arXiv preprint arXiv:2101.09387, 2021. [50] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. [51] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. [52] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019. [53] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020. [54] Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. Advances in Neural Information Processing Systems, 33, 2020. [55] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015. [56] Fenghua Weng, Yue Xu, Chengyan Fu, and Wenjie Wang. Mmj-bench: A comprehensive study on jailbreak attacks and defenses for vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 27689–27697, 2025. [57] Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001.03994, 2020. [58] Xiyang Wu, Ruiqi Xian, Tianrui Guan, Jing Liang, Souradip Chakraborty, Fuxiao Liu, Brian M Sadler, Dinesh Manocha, and Amrit Bedi. On the safety concerns of deploying llms/vlms in robotics: Highlighting the risks and vulnerabilities. In First Vision and Language for Autonomous Driving and Robotics Workshop, 2024. [59] Yongtao Wu, Fanghui Liu, Carl-Johann Simon-Gabriel, Grigorios Chrysos, and Volkan Cevher. Robust NAS under adversarial training: benchmark, theory, and beyond. In The Twelfth International Conference on Learning Representations, 2024. [60] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1–39, 2023. [61] Jongmin Yoon, Sung Ju Hwang, and Juho Lee. Adversarial purification with score-based generative models. In International Conference on Machine Learning, pages 12062–12072. PMLR, 2021. [62] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. [63] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18123–18133, 2022. [64] Jiaming Zhang, Qi Yi, and Jitao Sang. Towards adversarial attack on vision-language pre-training models. In Proceedings of the 30th ACM International Conference on Multimedia, pages 5005–5013, 2022. [65] Mingkun Zhang, Keping Bi, Wei Chen, Jiafeng Guo, and Xueqi Cheng. Clipure: Purification in latent space via clip for adversarially robust zero-shot classification. arXiv preprint arXiv:2502.18176, 2025. [66] Yi Zhang, Orestis Plevrakis, Simon S Du, Xingguo Li, Zhao Song, and Sanjeev Arora. Over-parameterized adversarial training: An analysis overcoming the curse of dimensionality. Advances in Neural Information Processing Systems, 33:679–688, 2020. [67] Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Man Cheung, and Min Lin. On evaluating adversarial robustness of large vision-language models. Advances in Neural Information Processing Systems, 36:54111–54138, 2023. [68] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. Contents of the Appendix We organize the appendix as follows: • In Appx. A, we provide complete proofs for Thm. 1, 1 and 2. • In Appx. B, we supply extend implementation details and ablation experiments, as well as the evaluation of DiffCAP under adaptive attacks. We further explore the potential of applying DiffCAP to other safety alignment challenges. • In Appx. C, we discuss broader impact of this paper. Appendix A Theoretical proofs A.1 Proof of Thm. 1 Proof of Thm. 1. Expanding the forward diffusion solution from Eq. 5, we get: ⁢(t)=α⁢(t)⁢adv+1−α⁢(t)⁢ϵ=α⁢(t)⁢[+1−a⁢(t)a⁢(t)⁢ϵ+ϵadv],ϵ∼⁢(0,I).formulae-sequencesubscriptadv1bold-italic-ϵdelimited-[]1bold-italic-ϵsubscriptbold-italic-ϵadvsimilar-tobold-italic-ϵ0 x(t)= α(t)\, x_adv+ 1-α(t)\, % ε= α(t) [ x+ 1-a(t) a(t) % ε+ ε_adv ], ε (% 0,I).italic_x ( t ) = square-root start_ARG α ( t ) end_ARG italic_xadv + square-root start_ARG 1 - α ( t ) end_ARG italic_ϵ = square-root start_ARG α ( t ) end_ARG [ italic_x + divide start_ARG square-root start_ARG 1 - a ( t ) end_ARG end_ARG start_ARG square-root start_ARG a ( t ) end_ARG end_ARG italic_ϵ + italic_ϵadv ] , italic_ϵ ∼ N ( 0 , I ) . Given Asm. 1 and the property of Gaussian distribution, we can analyze the classifier output as follows by absorbing the scaling factor a⁢(t) a(t)square-root start_ARG a ( t ) end_ARG and introduce ϵ′bold-italic-ϵ′ ε italic_ϵ′: h⁢(⁢(t))=h⁢(+ϵ′+ϵadv),where ⁢ϵ′∼⁢(0,σ⁢(t)2⁢I),σ⁢(t)2=1−α⁢(t)α⁢(t).formulae-sequenceℎsuperscriptbold-italic-ϵ′subscriptbold-italic-ϵadvformulae-sequencesimilar-towhere superscriptbold-italic-ϵ′0superscript2superscript21h( x(t))=h( x+ ε + ε_adv),% ε (0,σ(t)^2I),% σ(t)^2= 1-α(t)α(t).h ( italic_x ( t ) ) = h ( italic_x + italic_ϵ′ + italic_ϵadv ) , where italic_ϵ′ ∼ N ( 0 , σ ( t )2 I ) , σ ( t )2 = divide start_ARG 1 - α ( t ) end_ARG start_ARG α ( t ) end_ARG . Applying the randomized smoothing bound from Theorem 1 of [11], the classification is guaranteed to return class k1subscript1k_1k1 if: ‖ϵadv‖2<σ⁢(t)2⁢(Φ−1⁢(p1¯)−Φ−1⁢(p2¯)).subscriptnormsubscriptbold-italic-ϵadv22superscriptΦ1¯subscript1superscriptΦ1¯subscript2\| ε_adv\|_2< σ(t)2 ( ^-1(% p_1)- ^-1( p_2) ).∥ italic_ϵadv ∥2 < divide start_ARG σ ( t ) end_ARG start_ARG 2 end_ARG ( Φ- 1 ( under¯ start_ARG p1 end_ARG ) - Φ- 1 ( over¯ start_ARG p2 end_ARG ) ) . By re-organizing the term, we get: σ⁢(t)>2⁢‖ϵadv‖2Φ−1⁢(p1¯)−Φ−1⁢(p2¯).2subscriptnormsubscriptbold-italic-ϵadv2superscriptΦ1¯subscript1superscriptΦ1¯subscript2σ(t)> 2\| ε_adv\|_2 ^-1( p_1% )- ^-1( p_2).σ ( t ) > divide start_ARG 2 ∥ italic_ϵadv ∥2 end_ARG start_ARG Φ- 1 ( under¯ start_ARG p1 end_ARG ) - Φ- 1 ( over¯ start_ARG p2 end_ARG ) end_ARG . Since σ⁢(t)2=1−α⁢(t)α⁢(t)superscript21σ(t)^2= 1-α(t)α(t)σ ( t )2 = divide start_ARG 1 - α ( t ) end_ARG start_ARG α ( t ) end_ARG, this yields: 1−α⁢(t)α⁢(t)>(2⁢‖ϵadv‖2Φ−1⁢(p1¯)−Φ−1⁢(p2¯))2.1superscript2subscriptnormsubscriptbold-italic-ϵadv2superscriptΦ1¯subscript1superscriptΦ1¯subscript22 1-α(t)α(t)> ( 2\| ε_adv\|_2% ^-1( p_1)- ^-1( p_2) )^2.divide start_ARG 1 - α ( t ) end_ARG start_ARG α ( t ) end_ARG > ( divide start_ARG 2 ∥ italic_ϵadv ∥2 end_ARG start_ARG Φ- 1 ( under¯ start_ARG p1 end_ARG ) - Φ- 1 ( over¯ start_ARG p2 end_ARG ) end_ARG )2 . By re-organizing the term, we obtain: α⁢(t)<11+(2⁢‖ϵadv‖2Φ−1⁢(p1¯)−Φ−1⁢(p2¯))2.11superscript2subscriptnormsubscriptbold-italic-ϵadv2superscriptΦ1¯subscript1superscriptΦ1¯subscript22α(t)< 11+ ( 2\| ε_adv\|_2 ^-1% ( p_1)- ^-1( p_2) )^2.α ( t ) < divide start_ARG 1 end_ARG start_ARG 1 + ( divide start_ARG 2 ∥ italic_ϵadv ∥2 end_ARG start_ARG Φ- 1 ( under¯ start_ARG p1 end_ARG ) - Φ- 1 ( over¯ start_ARG p2 end_ARG ) end_ARG )2 end_ARG . Since α⁢(t)=exp⁡(−∫0tβ⁢(s)⁢ds)superscriptsubscript0differential-dα(t)= (- _0^tβ(s)\, ds )α ( t ) = exp ( - ∫0t β ( s ) d s ), and with linear schedule β⁢(t)=βmin+(βmax−βmin)⁢tsubscriptsubscriptsubscriptβ(t)= _ +( _ - _ )tβ ( t ) = βroman_min + ( βroman_max - βroman_min ) t, we compute: ∫0tβ⁢(s)⁢ds=βmin⁢t+12⁢(βmax−βmin)⁢t2.superscriptsubscript0differential-dsubscript12subscriptsubscriptsuperscript2 _0^tβ(s) ds= _ t+ 12( _ - _% )t^2.∫0t β ( s ) d s = βroman_min t + divide start_ARG 1 end_ARG start_ARG 2 end_ARG ( βroman_max - βroman_min ) t2 . Thus, α⁢(t)=exp⁡(−βmin⁢t−12⁢(βmax−βmin)⁢t2).subscript12subscriptsubscriptsuperscript2α(t)= (- _ t- 12( _ - _ )t^2% ).α ( t ) = exp ( - βroman_min t - divide start_ARG 1 end_ARG start_ARG 2 end_ARG ( βroman_max - βroman_min ) t2 ) . Let M:=log⁡(1+(2⁢‖ϵadv‖2Φ−1⁢(p1¯)−Φ−1⁢(p2¯))2)assign1superscript2subscriptnormsubscriptbold-italic-ϵadv2superscriptΦ1¯subscript1superscriptΦ1¯subscript22M:= (1+ ( 2\| ε_adv\|_2 ^-1(% p_1)- ^-1( p_2) )^2 )M := log ( 1 + ( divide start_ARG 2 ∥ italic_ϵadv ∥2 end_ARG start_ARG Φ- 1 ( under¯ start_ARG p1 end_ARG ) - Φ- 1 ( over¯ start_ARG p2 end_ARG ) end_ARG )2 ). Then by setting: βmin⁢t+12⁢(βmax−βmin)⁢t2=M,subscript12subscriptsubscriptsuperscript2 _ t+ 12( _ - _ )t^2=M,βroman_min t + divide start_ARG 1 end_ARG start_ARG 2 end_ARG ( βroman_max - βroman_min ) t2 = M , we obtain: t=tmin=2⁢Mβmin2+2⁢(βmax−βmin)⁢M+βmin.subscript2superscriptsubscript22subscriptsubscriptsubscriptt=t_ = 2M _ ^2+2( _ - _ )M+% _ .t = troman_min = divide start_ARG 2 M end_ARG start_ARG square-root start_ARG βroman_min2 + 2 ( βroman_max - βroman_min ) M end_ARG + βroman_min end_ARG . Then, when βmin≥Msubscript _ ≥ Mβroman_min ≥ M, we have tmin≤1subscript1t_ ≤ 1troman_min ≤ 1, and for tmin≤t≤1subscript1t_ ≤ t≤ 1troman_min ≤ t ≤ 1, we have argmaxk∈[K]ℙ(h((t)=k)=k1 _k∈[K]P(h( x(t)=k)=k_1arg maxitalic_k ∈ [ K ] blackboard_P ( h ( italic_x ( t ) = k ) = k1. ∎ A.2 Proof of Lemma 1 Proof of Lemma 1. we have: ⁢[‖ϕ⁢(⁢(t1))−ϕ⁢(⁢(t2))‖2]delimited-[]subscriptnormitalic-ϕsubscript1italic-ϕsubscript22 [\|φ( x(t_1))-φ( x(t_2))\|_% 2 ]blackboard_E [ ∥ ϕ ( italic_x ( t1 ) ) - ϕ ( italic_x ( t2 ) ) ∥2 ] ≤L⋅⁢[‖⁢(t1)−⁢(t2)‖2]absent⋅delimited-[]subscriptnormsubscript1subscript22 ≤ L·E [\| x(t_1)- x(t_2)\|_2% ]≤ L ⋅ blackboard_E [ ∥ italic_x ( t1 ) - italic_x ( t2 ) ∥2 ] (8) ≤L⋅⁢[‖⁢(t1)−⁢(t2)‖22],absent⋅delimited-[]superscriptsubscriptnormsubscript1subscript222 ≤ L· E [\| x(t_1)- x(t_2% )\|_2^2 ],≤ L ⋅ square-root start_ARG blackboard_E [ ∥ italic_x ( t1 ) - italic_x ( t2 ) ∥22 ] end_ARG , (9) where Eq. 8 follows from the Lipschitz continuity of ϕitalic-ϕφϕ, Eq. 9 uses Jensen’s inequality, i.e., ⁢[‖Z‖]≤⁢[‖Z‖2]delimited-[]normdelimited-[]superscriptnorm2E[\|Z\|]≤ E[\|Z\|^2] _E [ ∥ Z ∥ ] ≤ square-root start_ARG blackboard_E [ ∥ Z ∥2 ] end_ARG for any random vector Z. Next, to compute ⁢[‖⁢(t1)−⁢(t2)‖2]delimited-[]superscriptnormsubscript1subscript22E [\| x(t_1)- x(t_2)\|^2 ]blackboard_E [ ∥ italic_x ( t1 ) - italic_x ( t2 ) ∥2 ], use Eq. (5): ⁢(t1)−⁢(t2)=(α⁢(t1)−α⁢(t2))⁢adv+(1−α⁢(t1)−1−α⁢(t2))⁢ϵ,subscript1subscript2subscript1subscript2subscriptadv1subscript11subscript2bold-italic-ϵ x(t_1)- x(t_2)= ( α(t_1)- α(t_2)% ) x_adv+ ( 1-α(t_1)- 1-α(t_2% ) ) ε,italic_x ( t1 ) - italic_x ( t2 ) = ( square-root start_ARG α ( t1 ) end_ARG - square-root start_ARG α ( t2 ) end_ARG ) italic_xadv + ( square-root start_ARG 1 - α ( t1 ) end_ARG - square-root start_ARG 1 - α ( t2 ) end_ARG ) italic_ϵ , where ϵ∼⁢(0,I)similar-tobold-italic-ϵ0 ε (0,I)italic_ϵ ∼ N ( 0 , I ). Taking the squared norm and expectation: ⁢[‖⁢(t1)−⁢(t2)‖2]delimited-[]superscriptnormsubscript1subscript22 [\| x(t_1)- x(t_2)\|^2 ]blackboard_E [ ∥ italic_x ( t1 ) - italic_x ( t2 ) ∥2 ] =(α⁢(t1)−α⁢(t2))2⁢‖adv‖2+(1−α⁢(t1)−1−α⁢(t2))2⋅⁢[‖ϵ‖2]absentsuperscriptsubscript1subscript22superscriptnormsubscriptadv2⋅superscript1subscript11subscript22delimited-[]superscriptnormbold-italic-ϵ2 = ( α(t_1)- α(t_2) )^2\| % x_adv\|^2+ ( 1-α(t_1)- 1-α(t_2)% )^2·E[\| ε\|^2]= ( square-root start_ARG α ( t1 ) end_ARG - square-root start_ARG α ( t2 ) end_ARG )2 ∥ italic_xadv ∥2 + ( square-root start_ARG 1 - α ( t1 ) end_ARG - square-root start_ARG 1 - α ( t2 ) end_ARG )2 ⋅ blackboard_E [ ∥ italic_ϵ ∥2 ] =(α⁢(t1)−α⁢(t2))2⁢‖adv‖2+(1−α⁢(t1)−1−α⁢(t2))2⁢d.absentsuperscriptsubscript1subscript22superscriptnormsubscriptadv2superscript1subscript11subscript22 = ( α(t_1)- α(t_2) )^2\| % x_adv\|^2+ ( 1-α(t_1)- 1-α(t_2)% )^2d.= ( square-root start_ARG α ( t1 ) end_ARG - square-root start_ARG α ( t2 ) end_ARG )2 ∥ italic_xadv ∥2 + ( square-root start_ARG 1 - α ( t1 ) end_ARG - square-root start_ARG 1 - α ( t2 ) end_ARG )2 d . As t1,t2→1→subscript1subscript21t_1,t_2→ 1t1 , t2 → 1, both terms vanish, so the expectation tends to zero. ∎ A.3 Proof of Thm. 2 Before proving Thm. 2, we need the following Lemma. Lemma 2. Let β⁢(t)=βmin+(βmax−βmin)⁢tsubscriptsubscriptsubscriptβ(t)= _ +( _ - _ )tβ ( t ) = βroman_min + ( βroman_max - βroman_min ) t be a linear noise schedule with βmin>0subscript0 _ >0βroman_min > 0 and βmax>βminsubscriptsubscript _ > _ βroman_max > βroman_min. Define: α⁢(t):=exp⁡(−∫0tβ⁢(s)⁢ds),andf⁢(t):=β⁢(t)⋅α⁢(t)1−α⁢(t).formulae-sequenceassignsuperscriptsubscript0differential-dandassign⋅1α(t):= (- _0^tβ(s)\, ds ), % f(t):=β(t)· α(t)1-α(t).α ( t ) := exp ( - ∫0t β ( s ) d s ) , and f ( t ) := β ( t ) ⋅ square-root start_ARG divide start_ARG α ( t ) end_ARG start_ARG 1 - α ( t ) end_ARG end_ARG . Then f⁢(t)f(t)f ( t ) is strictly decreasing for all t∈[0,1)01t∈[0,1)t ∈ [ 0 , 1 ). Proof. We first analyze the function f⁢(t)f(t)f ( t ) by taking its logarithm: log⁡f⁢(t)=log⁡β⁢(t)+12⁢log⁡(α⁢(t)1−α⁢(t)).121 f(t)= β(t)+ 12 ( α(t)1-α(t)% ).log f ( t ) = log β ( t ) + divide start_ARG 1 end_ARG start_ARG 2 end_ARG log ( divide start_ARG α ( t ) end_ARG start_ARG 1 - α ( t ) end_ARG ) . Differentiating and using the chain rule, we get: d⁢t⁢log⁡f⁢(t)=β′⁢(t)β⁢(t)+12⁢(d⁢t⁢log⁡α⁢(t)−d⁢t⁢log⁡(1−α⁢(t)))=β′⁢(t)β⁢(t)+12⋅α′⁢(t)⁢(1α⁢(t)+11−α⁢(t))=β′⁢(t)β⁢(t)−12⋅β⁢(t)1−α⁢(t),superscript′121superscript′⋅12superscript′111superscript′⋅121 split& ddt f(t)= β (t)β(t)+ 1% 2 ( ddt α(t)- ddt (1-α(t)) )\\ &= β (t)β(t)+ 12·α (t) (% 1α(t)+ 11-α(t) )\\ &= β (t)β(t)- 12· β(t)1-α(% t), splitstart_ROW start_CELL end_CELL start_CELL divide start_ARG d end_ARG start_ARG d t end_ARG log f ( t ) = divide start_ARG β′ ( t ) end_ARG start_ARG β ( t ) end_ARG + divide start_ARG 1 end_ARG start_ARG 2 end_ARG ( divide start_ARG d end_ARG start_ARG d t end_ARG log α ( t ) - divide start_ARG d end_ARG start_ARG d t end_ARG log ( 1 - α ( t ) ) ) end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL = divide start_ARG β′ ( t ) end_ARG start_ARG β ( t ) end_ARG + divide start_ARG 1 end_ARG start_ARG 2 end_ARG ⋅ α′ ( t ) ( divide start_ARG 1 end_ARG start_ARG α ( t ) end_ARG + divide start_ARG 1 end_ARG start_ARG 1 - α ( t ) end_ARG ) end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL = divide start_ARG β′ ( t ) end_ARG start_ARG β ( t ) end_ARG - divide start_ARG 1 end_ARG start_ARG 2 end_ARG ⋅ divide start_ARG β ( t ) end_ARG start_ARG 1 - α ( t ) end_ARG , end_CELL end_ROW (10) where we use α′⁢(t)=−β⁢(t)⋅α⁢(t)superscript′⋅α (t)=-β(t)·α(t)α′ ( t ) = - β ( t ) ⋅ α ( t ). Let’s define g⁢(t):=d⁢t⁢log⁡f⁢(t)assigng(t):= ddt f(t)g ( t ) := divide start_ARG d end_ARG start_ARG d t end_ARG log f ( t ). We now analyze the sign of g⁢(t)g(t)g ( t ). First, observe that β′⁢(t)=βmax−βmin>0superscript′subscriptsubscript0β (t)= _ - _ >0β′ ( t ) = βroman_max - βroman_min > 0 is constant, β⁢(t)≥βmin>0subscript0β(t)≥ _ >0β ( t ) ≥ βroman_min > 0, so β′⁢(t)β⁢(t)superscript′ β (t)β(t)divide start_ARG β′ ( t ) end_ARG start_ARG β ( t ) end_ARG is strictly decreasing in t, Also α⁢(t)α(t)α ( t ) is strictly decreasing, so 1−α⁢(t)11-α(t)1 - α ( t ) is strictly increasing, and hence β⁢(t)1−α⁢(t)1 β(t)1-α(t)divide start_ARG β ( t ) end_ARG start_ARG 1 - α ( t ) end_ARG is strictly increasing. Therefore, g⁢(t)g(t)g ( t ) is strictly decreasing in t. To show that g⁢(t)<00g(t)<0g ( t ) < 0 for all t∈[0,1)01t∈[0,1)t ∈ [ 0 , 1 ), it suffices to show g⁢(t)<00g(t)<0g ( t ) < 0 near t=00t=0t = 0. When t≈00t≈ 0t ≈ 0, we have: β⁢(t)≈βmin,∫0tβ⁢(s)⁢ds≈βmin⁢t,α⁢(t)=exp⁡(−βmin⁢t)≈1−βmin⁢t+o⁢(t).formulae-sequencesubscriptformulae-sequencesuperscriptsubscript0differential-dsubscriptsubscript1subscriptβ(t)≈ _ , _0^tβ(s)\, ds≈ _% t, α(t)= (- _ t)≈ 1- _ t+o(t).β ( t ) ≈ βroman_min , ∫0t β ( s ) d s ≈ βroman_min t , α ( t ) = exp ( - βroman_min t ) ≈ 1 - βroman_min t + o ( t ) . Thus, β⁢(t)1−α⁢(t)≈βminβmin⁢t=1t,which diverges as ⁢t→0.formulae-sequence1subscriptsubscript1→which diverges as 0 β(t)1-α(t)≈ _ _ t= 1% t, diverges as t→ 0.divide start_ARG β ( t ) end_ARG start_ARG 1 - α ( t ) end_ARG ≈ divide start_ARG βroman_min end_ARG start_ARG βroman_min t end_ARG = divide start_ARG 1 end_ARG start_ARG t end_ARG , which diverges as t → 0 . Given the result: β′⁢(t)β⁢(t)≈βmax−βminβmin,superscript′subscriptsubscriptsubscript β (t)β(t)≈ _ - _ % _ ,divide start_ARG β′ ( t ) end_ARG start_ARG β ( t ) end_ARG ≈ divide start_ARG βroman_max - βroman_min end_ARG start_ARG βroman_min end_ARG , which is a finite number, thus, g⁢(t)→−∞as ⁢t→0.formulae-sequence→as 0g(t)→-∞ t→ 0.g ( t ) → - ∞ as t → 0 . Since g⁢(t)g(t)g ( t ) is strictly decreasing, it follows that g⁢(t)<00g(t)<0g ( t ) < 0 for all t∈[0,1)01t∈[0,1)t ∈ [ 0 , 1 ). This implies that log⁡f⁢(t) f(t)log f ( t ) is strictly decreasing, and thus f⁢(t)f(t)f ( t ) is strictly decreasing as well. ∎ Now we are ready to present the proof of Thm. 2. Proof of Thm. 2. Let Δ⁢(t,δ):=⁢(t+δ)−⁢(t)assignΔ (t,δ):= x(t+δ)- x(t)Δ ( t , δ ) := italic_x ( t + δ ) - italic_x ( t ). From the closed-form solution of the VP-SDE, ⁢(t)=α⁢(t)⁢adv+1−α⁢(t)⁢ϵadv,subscriptadv1subscriptbold-italic-ϵadv x(t)= α(t) x_adv+ 1-α(t) ε% _adv,italic_x ( t ) = square-root start_ARG α ( t ) end_ARG italic_xadv + square-root start_ARG 1 - α ( t ) end_ARG italic_ϵadv , we have: Δ⁢(t,δ)=(α⁢(t+δ)−α⁢(t))⁢adv+(1−α⁢(t+δ)−1−α⁢(t))⁢ϵadv.Δsubscriptadv11subscriptbold-italic-ϵadv (t,δ)= ( α(t+δ)- α(t) ) x_% adv+ ( 1-α(t+δ)- 1-α(t) ) % ε_adv.Δ ( t , δ ) = ( square-root start_ARG α ( t + δ ) end_ARG - square-root start_ARG α ( t ) end_ARG ) italic_xadv + ( square-root start_ARG 1 - α ( t + δ ) end_ARG - square-root start_ARG 1 - α ( t ) end_ARG ) italic_ϵadv . Let us define: A:=α⁢(t+δ)−α⁢(t),B:=1−α⁢(t+δ)−1−α⁢(t).formulae-sequenceassignassign11A:= α(t+δ)- α(t), B:= 1-α(t+δ)% - 1-α(t).A := square-root start_ARG α ( t + δ ) end_ARG - square-root start_ARG α ( t ) end_ARG , B := square-root start_ARG 1 - α ( t + δ ) end_ARG - square-root start_ARG 1 - α ( t ) end_ARG . (11) Then: Δ⁢(t,δ)=A⁢adv+B⁢ϵ.Δsubscriptadvbold-italic-ϵ (t,δ)=A x_adv+B ε.Δ ( t , δ ) = A italic_xadv + B italic_ϵ . By the Lipschitz property of ϕitalic-ϕφϕ and the Cauchy-Schwarz inequality: ⁢[‖ϕ⁢(⁢(t+δ))−ϕ⁢(⁢(t))‖2]≤L⋅⁢[‖Δ⁢(t,δ)‖2]≤L⋅⁢[‖Δ⁢(t,δ)‖22],delimited-[]subscriptnormitalic-ϕitalic-ϕ2⋅delimited-[]subscriptnormΔ2⋅delimited-[]superscriptsubscriptnormΔ22E [\|φ( x(t+δ))-φ( x(t))\|_2 ]≤ L% ·E [\| (t,δ)\|_2 ]≤ L· E% [\| (t,δ)\|_2^2 ],blackboard_E [ ∥ ϕ ( italic_x ( t + δ ) ) - ϕ ( italic_x ( t ) ) ∥2 ] ≤ L ⋅ blackboard_E [ ∥ Δ ( t , δ ) ∥2 ] ≤ L ⋅ square-root start_ARG blackboard_E [ ∥ Δ ( t , δ ) ∥22 ] end_ARG , (12) where L∈L1,L2.subscript1subscript2L∈\L_1,L_2\.L ∈ L1 , L2 . We now compute this second moment: ⁢[‖Δ⁢(t,δ)‖22]=⁢[‖A⁢adv+B⁢ϵ‖22]=A2⁢‖adv‖22+B2⁢[‖ϵ‖22].delimited-[]superscriptsubscriptnormΔ22delimited-[]superscriptsubscriptnormsubscriptadvbold-italic-ϵ22superscript2superscriptsubscriptnormsubscriptadv22superscript2delimited-[]superscriptsubscriptnormbold-italic-ϵ22E [\| (t,δ)\|_2^2 ]=E [\|A x% _adv+B ε\|_2^2 ]=A^2\| x_adv\|_% 2^2+B^2E[\| ε\|_2^2].blackboard_E [ ∥ Δ ( t , δ ) ∥22 ] = blackboard_E [ ∥ A italic_xadv + B italic_ϵ ∥22 ] = A2 ∥ italic_xadv ∥22 + B2 blackboard_E [ ∥ italic_ϵ ∥22 ] . Since ϵbold-italic-ϵ εitalic_ϵ is standard Gaussian in ℝdsuperscriptℝR^dblackboard_Rd, we get: ⁢[‖ϵ‖2]=ddelimited-[]superscriptnormbold-italic-ϵ2E[\| ε\|^2]=dblackboard_E [ ∥ italic_ϵ ∥2 ] = d. Thus: ⁢[‖Δ⁢(t,δ)‖22]=A2⁢‖adv‖22+B2⁢d.delimited-[]superscriptsubscriptnormΔ22superscript2superscriptsubscriptnormsubscriptadv22superscript2E [\| (t,δ)\|_2^2 ]=A^2\| x_adv% \|_2^2+B^2d.blackboard_E [ ∥ Δ ( t , δ ) ∥22 ] = A2 ∥ italic_xadv ∥22 + B2 d . We now perform Taylor expansion for A and B with respect to t. First note that: d⁢t⁢α⁢(t)=−β⁢(t)⁢α⁢(t),d⁢t⁢α⁢(t)=−β⁢(t)2⁢α⁢(t).formulae-sequence2 ddtα(t)=-β(t)α(t), ddt α(t)=-% β(t)2 α(t).divide start_ARG d end_ARG start_ARG d t end_ARG α ( t ) = - β ( t ) α ( t ) , divide start_ARG d end_ARG start_ARG d t end_ARG square-root start_ARG α ( t ) end_ARG = - divide start_ARG β ( t ) end_ARG start_ARG 2 end_ARG square-root start_ARG α ( t ) end_ARG . Plugging the above equation back into Eq. 11, we get: A=α⁢(t+δ)−α⁢(t)=−β⁢(t)2⁢α⁢(t)⁢δ+o⁢(δ).2A= α(t+δ)- α(t)=- β(t)2 α(t)% δ+o(δ).A = square-root start_ARG α ( t + δ ) end_ARG - square-root start_ARG α ( t ) end_ARG = - divide start_ARG β ( t ) end_ARG start_ARG 2 end_ARG square-root start_ARG α ( t ) end_ARG δ + o ( δ ) . Similarly: d⁢t⁢1−α⁢(t)=β⁢(t)⁢α⁢(t)2⁢1−α⁢(t).121 ddt 1-α(t)= β(t)α(t)2 1-α(t).divide start_ARG d end_ARG start_ARG d t end_ARG square-root start_ARG 1 - α ( t ) end_ARG = divide start_ARG β ( t ) α ( t ) end_ARG start_ARG 2 square-root start_ARG 1 - α ( t ) end_ARG end_ARG . Plugging the above equation back into Eq. 11, we get: B=1−α⁢(t+δ)−1−α⁢(t)=β⁢(t)⁢α⁢(t)2⁢1−α⁢(t)⁢δ+o⁢(δ).1121B= 1-α(t+δ)- 1-α(t)= β(t)α(t)2 % 1-α(t)δ+o(δ).B = square-root start_ARG 1 - α ( t + δ ) end_ARG - square-root start_ARG 1 - α ( t ) end_ARG = divide start_ARG β ( t ) α ( t ) end_ARG start_ARG 2 square-root start_ARG 1 - α ( t ) end_ARG end_ARG δ + o ( δ ) . Take the square and sum them up, we obtain: A2+B2superscript2superscript2 A^2+B^2A2 + B2 =β⁢(t)2⁢δ24⁢(α⁢(t)+α⁢(t)21−α⁢(t))+o⁢(δ2)=β⁢(t)2⁢α⁢(t)⁢δ24⁢(1−α⁢(t))+o⁢(δ2).absentsuperscript2superscript24superscript21superscript2superscript2superscript241superscript2 = β(t)^2δ^24 (α(t)+ α(t)^% 21-α(t) )+o(δ^2)= β(t)^2α(t)δ^2% 4(1-α(t))+o(δ^2).= divide start_ARG β ( t )2 δ2 end_ARG start_ARG 4 end_ARG ( α ( t ) + divide start_ARG α ( t )2 end_ARG start_ARG 1 - α ( t ) end_ARG ) + o ( δ2 ) = divide start_ARG β ( t )2 α ( t ) δ2 end_ARG start_ARG 4 ( 1 - α ( t ) ) end_ARG + o ( δ2 ) . Since ‖adv‖2≤dsuperscriptnormsubscriptadv2\| x_ adv\|^2≤ d∥ italic_xroman_adv ∥2 ≤ d, we have: ⁢[‖Δ⁢(t,δ)‖22]≤d⋅β⁢(t)2⁢α⁢(t)4⁢(1−α⁢(t))⁢δ2+o⁢(δ2).delimited-[]superscriptsubscriptnormΔ22⋅superscript241superscript2superscript2E [\| (t,δ)\|_2^2 ]≤ d·β(t)^2% α(t)4(1-α(t))δ^2+o(δ^2).blackboard_E [ ∥ Δ ( t , δ ) ∥22 ] ≤ divide start_ARG d ⋅ β ( t )2 α ( t ) end_ARG start_ARG 4 ( 1 - α ( t ) ) end_ARG δ2 + o ( δ2 ) . (13) Plugging Eq. 13 into Eq. 12, we obtain: ⁢[‖ϕ⁢(⁢(t+δ))−ϕ⁢(⁢(t))‖2]≤L⋅⁢[‖Δ⁢(t,δ)‖22]=O⁢(L⁢δ⋅β⁢(t)⁢α⁢(t)1−α⁢(t)).delimited-[]subscriptnormitalic-ϕitalic-ϕ2⋅delimited-[]superscriptsubscriptnormΔ22⋅1E [\|φ( x(t+δ))-φ( x(t))\|_2 ]≤ L% · E [\| (t,δ)\|_2^2 ]=O (Lδ% ·β(t) α(t)1-α(t) ).blackboard_E [ ∥ ϕ ( italic_x ( t + δ ) ) - ϕ ( italic_x ( t ) ) ∥2 ] ≤ L ⋅ square-root start_ARG blackboard_E [ ∥ Δ ( t , δ ) ∥22 ] end_ARG = O ( L δ ⋅ β ( t ) square-root start_ARG divide start_ARG α ( t ) end_ARG start_ARG 1 - α ( t ) end_ARG end_ARG ) . Lastly, by Lemma 2, we have that the bound on the right hand side decreases as t grows. ∎ Appendix B Additional experiment B.1 More implementation details For the forward diffusion, we schedule noise with βmin=0.1,βmax=20formulae-sequencesubscript0.1subscript20 _ =0.1, _ =20βroman_min = 0.1 , βroman_max = 20, and a fixed step size of 0.010.010.010.01. In the reverse generation, we employ guided diffusion with a step size of 0.0150.0150.0150.015. To keep the evaluation of adaptive attacks on large vision-language models (VLMs) computationally tractable, we randomly choose 100100100100 images per dataset. BPDA [4] attacks are run for 50505050 iterations, while EOT [5] attacks are approximated using three stochastic samples per query. For stronger attacks with ϵ>4/255italic-ϵ4255ε>4/255ϵ > 4 / 255, we set the minimum diffusion depth to 0.040.040.040.04 for sufficient denoising. All baseline methods are evaluated using their respective best-performing hyperparameters as reported in the original papers. B.2 Adaptive attacks Table 5: Evaluation of OpenFlamingo (OF-9B) and LLaVA 1.5-7B on four datasets under clean and adversarial (ℓ∞8/255superscriptsubscriptℓ8255 _∞^8/255ℓ∞8 / 255) conditions, with and without DiffCAP defense against adaptive attacks. VLM Dataset Clean APGD BPDA BPDA + EOT w/o with w/o with w/o with w/o with OF [6] COCO [31] 90.1 92.4 4.7 91.1 27.1 79.9 29.2 83.4 Flicker30k [42] 63.9 62.7 4.9 60.5 19.1 50.6 15.5 56.5 2-10 TextVQA [50] 23.1 18.6 0.6 17.6 7.1 18.2 2.3 16.0 VQAv2 [18] 46.2 47.1 8.3 44.6 24.0 44.5 18.0 39.5 LLaVA [33] COCO [31] 125.9 122.2 11.3 123.4 21.9 115.9 19.5 114.9 Flicker30k [42] 81.7 78.0 8.5 76.2 20.9 73.4 18.0 74.6 2-10 TextVQA [50] 36.9 25.1 7.4 22.7 8.8 24.3 8.7 21.7 VQAv2 [18] 74.3 69.9 23.4 67.5 25.7 65.4 27.1 66.9 We evaluate DiffCAP in a white-box setting, where the adversary has full knowledge of the deployed defense mechanism. Tab. 5 presents the detailed metrics for image captioning (IC) and visual-question answering (VQA) tasks across various attack configurations, comparing performance with and without DiffCAP defense. Even under an increased attack budget (ℓ∞8/255superscriptsubscriptℓ8255 _∞^8/255ℓ∞8 / 255), DiffCAP maintains high fidelity on clean inputs, with only an average performance drop of 3.33.33.33.3 points. Under APGD [12] attacks, when the adversary is unaware of DiffCAP’s existence, it successfully restores the performance of VLMs on different datasets to levels closely matching their clean baselines, showing only a modest average degradation of 4.84.84.84.8 points. In scenarios where the adversary bypasses gradient obfuscation through backward pass differentiable approximation (BPDA) [4] and simulates stochasticity via expectation over transformations (EOT) [5], DiffCAP continues to demonstrate strong resilience. The best-case performance degradation relative to clean conditions is only 1.71.71.71.7 points (OF-VQAv2) by BPDA without EOT and 6.76.76.76.7 points (OF-COCO) with EOT. The corresponding worst-case performance reductions are observed as 13.313.313.313.3 points (OF-Flickr30k) and 15.215.215.215.2 points (LLaVA-TextVQA), respectively. These results elucidate the inherent uncertainty of DiffCAP’s per-image diffusion step calculation, which determines the minimal purification for each adversarial example based on semantic convergence during the diffusion process. Such a dynamic strategy significantly prevents trivial gradient approximations and random regressions from circumventing its defense, enhancing adversarial robustness against adaptive attacks of prohibitively high time complexity. B.3 Ablation on threshold and diffusion step size Figure 4: CIDEr score and running time (in second) per image with varying thresholds and diffusion step sizes (Δ⁢tΔ tΔ t) for DiffCAP. The evaluation is based on the image captioning task under ℓ∞2/255superscriptsubscriptℓ2255 _∞^2/255ℓ∞2 / 255 attack. To validate the effectiveness of the adaptive similarity threshold calculation described in Alg. 2, we conduct an ablation study over different threshold values τ and diffusion step sizes Δ⁢tΔ tΔ t. Fig. 4 displays the results by OF on the COCO dataset. We observe that setting the threshold to 0.960.960.960.96 achieves the best overall robustness in terms of CIDEr score across a range of step sizes. A higher threshold generally leads to more diffusion steps, increasing time for reverse diffusion sampling. In practice, we find that a step size of 0.010.010.010.01 offers the best trade-off between performance and computational cost. Notably, τ=0.960.96τ=0.96τ = 0.96 and Δ⁢t=0.01Δ0.01 t=0.01Δ t = 0.01 used in our main experiments bring about an average overhead of 1.11.11.11.1 seconds per image, significantly faster than ∼2.3similar-toabsent2.3 2.3∼ 2.3 seconds per image required by DiffPure. B.4 Hallucination and jailbreaking Table 6: Evaluation of DiffCAP in mitigating Hallucination and Jailbreaking of large VLMs. Hallucination with LLaVA 1.5-13B Jailbreaking with MiniGPT 4-13B adversarial popular random any identity disinfo crime x-risk Clean 82.7 84.3 85.0 16/40 3/11 6/13 6/13 1/3 Attack N/A N/A N/A 24/40 6/11 7/13 12/13 2/3 DiffCAP 83.2 85.1 86.3 14/40 3/11 5/13 5/13 1/3 Large VLMs tend to hallucinate objects that are not actually present in the image. POPE [30] serves as a benchmark to formulate hallucination detection as a binary classification task. In Tab. 6, we report the F1-scores across three POPE categories using LLaVA 1.5-13B, with and without DiffCAP applied as image preprocessing. A consistent improvement is observed with DiffCAP. This suggests that DiffCAP, through Langevin dynamics, walks image features to semantically stable regions of the distribution. By suppressing high-frequency adversarial or spurious signals, DiffCAP becomes less sensitive to misleading cues and more robust against hallucination. Large VLMs are also vulnerable to jailbreaking attacks on the visual modality [7, 43], where adversarially crafted images can induce harmful outputs in response to restricted prompts (e.g., “How to make a bomb?”). We apply the attack proposed by Qi et al. [43] to MiniGPT 4-13B [68] and count the policy-violating outputs triggered by 40404040 harmful prompts spanning four categories. Even under a stronger perturbation budget (ℓ∞16/255superscriptsubscriptℓ16255 _∞^16/255ℓ∞16 / 255), DiffCAP successfully restores the model’s behavior to a level comparable to, or slightly better than, the clean condition. These findings reinforce the versatility of DiffCAP as a modular defense measure, readily adaptable to various models and tasks requiring robustness guarantee. As jailbreaking attacks continue to evolve rapidly, benchmarking DiffCAP against such threats falls outside the scope of this work, but nonetheless marks a promising direction for future investigation. Appendix C Broader impact With the swift application of VLMs, the risk of adversarial attacks has become a critical concern. This paper proposes DiffCAP, an adversarial purification method that can improve robustness without retraining the model, which may enlarge its usability in application scenarios, such as autonomous driving based on VLMs. While maintaining a remarkable defense effect, this method greatly reduces the diffusion steps and hyperparameter adjustments, which promotes the safe and fast implementation for defending pre-trained large models. However, any single defense mechanism may fail in face of new attacks, so maintaining a diverse and regularly tested defense strategy is essential.

Instant research discovery