Paper deep dive
Transferable Spatial Temporal Coherence Adversarial Attack on Black-Box Vision Language Models for Autonomous Driving
Heyam Bin Jahlan Areej Alhothali Abeer Alhothali
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The rapid integration of Vision Language Models (VLMs) into sensitive systems introduces critical safety vulnerabilities that remain unexplored in exist studies. While adversarial attack robustness has been extensively studied for image-based models, the susceptibility of VLMs to temporally-aware adversarial attacks against video in driving context poses a distinct and under examined threat. In this paper, we introduce novel adversarial attack against video targeting VLM models used for autonomous driving scenes named Spatial Temporal Coherence Adversarial Attack (STCA). Our attack comprise from three stages: modalities expansion, Spatial attack, and STCA attack. In modalities expansion, we propose caption-guided frame selection method in order to ensure that adversarial perturbation target the most semantically significant frames. this http URL spatial attack, we craft effective perturbation and preserve high similarity. Then the perturbed video generated fed into STCA stage that disrupt cross-frame temporal coherence using motion guided mask. Our method operate under black box threat model against victim target VLMs, relying solely on transferability from white-box surrogate this http URL conduct our experiments on the BDD100K and nuScenes autonomous driving datasets across three VLM models: Video LLaVA-7B, Qwen2.5-VL-7B, and Dolphin. Experimental results demonstrate spatial attack achieves an ASR with high SSIM. Our finding reveal that existing video language model, remain highly susceptible to adversarial attack in autonomous driving scenarios, underscoring the urgent need for robust defense for VLM models.
Tags
Links
- Source: https://arxiv.org/abs/2610.08331v1
- Canonical: https://arxiv.org/abs/2610.08331v1
Trouble viewing inline? Open PDF directly →
Full Text
57,922 characters extracted from source content.
Expand or collapse full text
Transferable Spatial Temporal Coherence Adversarial Attack on Black-Box Vision Language Models for Autonomous Driving PubID: pubid: Heyam M. Bin Jahlan Areej M. Alhothali Abeer Alhothali Abstract The rapid integration of Vision Language Models (VLMs) into sensitive systems such as autonomous driving systems introduces critical safety vulnerabilities that remain largely unexplored in exist studies. While adversarial attack robustness has been extensively studied for image-based models, the susceptibility of VLMs to temporally-aware adversarial attacks against video in driving context poses a distinct and under examined threat. In this paper, we introduce novel adversarial attack framework against video targeting VLM models used for autonomous driving scenes named Spatial Temporal Coherence Adversarial Attack (STCA). Our attack comprise from three primary stages: modalities expansion, Spatial attack, and STCA attack. In modalities expansion, we extract multi-caption of each video and propose caption-guided frame selection method in order to ensure that adversarial perturbation target the most semantically significant frames. Secondly.In spatial attack, we apply YOLO-guided object mask in vital frames to craft effective perturbation and preserve high similarity. Then the perturbed video generated fed into STCA stage that exploits LanguageBind model as surrogate model to disrupt cross-frame temporal coherence using motion guided mask.Our method operate under black box threat model against victim target VLMs, relying solely on transferability from white-box surrogate model.We conduct our experiments on the BDD100K and nuScenes autonomous driving datasets across three VLM models: Video LLaVA-7B, Qwen2.5-VL-7B, and Dolphin. Experimental results demonstrate spatial attack achieves an attack success rate (ASR) 32.6 % with high structural similarity index (SSIM=0.933) while in our proposed STCA attack, significantly enhances ASR to 71% against Video LLaVA -7B model. We gained strong transferability to Qwen2.5-VL-7B ASR= 45% in spatial attack and 84.2% in our STCA attack. Dolphin dataset deceived by 46.2% in BDD100K dataset. While in nuSences dataset, we achieve in spatial attack only 57.6% , 64.7%, and and 37.6% , after SCTA stage, we gain 83%, 96.5%, and 47.1% in video-LLaVa, Qwen2.5, and Dolphins models respectively. Semantic divergence is calculated using word overlap similarity. Our finding reveal that existing video language model, remain highly susceptible to adversarial attack in autonomous driving scenarios, underscoring the urgent need for robust defense for VLM models. Index Terms: Vision Language Model, Adversarial Attack, Autonomous driving, Transferability, Attack Success Rate. I Introduction The rapid progress of Vision Language Pre-training models (VLP) has transformed the field of multi-modal artificial intelligence, enabling systems to concurrently understand and analysis textual and visual information. Current models such as ALBEF [1], CLIP [2], and TCL [3] have demonstrated exceptional performance across a wide range of downstream tasks including image captioning, visual question answering, and image-text retrieval. These models leverage extensive contrastive learning objectives in order to align textual and visual representation in a shared embedding space achieving unprecedented level of cross-modal understanding. However, despite their remarkable capabilities, VLM model exhibit inherent susceptibility to adversarial attacks, where the carefully designed imperceptible perturbations to the input can significantly degrade model performance, raising substantial concerns regarding their reliability and security in practical applications. Adversarial Attacks on deep learning models have been extensively examined in the uni-model context, specially for image classification FGSM [4] and PGD [5]. The extension of adversarial attacks to the multi-modal domain, however, current unique challenges that are not fully addressed by existing uni-modal techniques. In the case of VLM model, adversarial perturbations deceive both the cross-modal alignment mechanism and the visual encoder, making the attack issue significantly more complex. Previous methods in adversarial attack for VLM models such as AdvCLIP [6] have shown the feasibility of generating adversarial examples that can deceive these models in white-box scenario, where the attacker has full knowledge about the model parameters. Nevertheless, the transferability of such attacks across different VLM architectures remain largely underexplored that create substantial performance gap between black-box and white-box scenarios. While image-based adversarial attack has been extensively examined, the primary limitation of current adversarial attacks studies for VLM models is the failure to evaluate the temporal dynamics inherent in video data. The issue of generating adversarial perturbation against temporal consistent of video sequence remain an open research challenge. This additional dimension of complexity guides us to disrupt the temporal coherence between each two consecutive frames to capture the movements regions instead of only fool the model on individual frame. The temporal aspect is critical in autonomous driving where the vehicle must be aware about the relationship between object during video. Temporal attack is not addressed before in existing studies that only focus in image spatial attack for individual frame and neglected temporal structure resulting in a non-comprehensive evaluate for attack effectiveness. Additionally, existing adversarial attack methods in VLM models that targeting the consistency between visual and textual modalities typically apply perturbation uniformly across the entire image. on the contrary, our proposed method aimed to consider the semantic relevance regions in different frame and neglected regions that contribute minimally to deceiving the cross-modal alignment mechanism such as background and sky. Advances techniques of object detection and attention-based in video that identify semantically meaningful regions, remain largely unexplored in adversarial attack pipelines for VLM models. To address these limitations, we propose Spatio-Temporal Coherence Adversarial Attack STCA framework for VLM model in video data. Our proposed framework innovate three key stages that collectively enhances the effectiveness of adversarial attack on video based VLM models. First, we extract multi-captions for each video using two different VLM model in order to capture the diversity of textual prompt, after that, we propose Caption Guided Frame Selection strategy that utilize CLIP model to select the most semantically relevant frames from the sequence of video with respect to associated description. To ensure extracting most informative frame, we compute the cosine similarity between the visual embeddings of each frame and the textual embeddings of the associated captions, the top K frames that maximize the semantic alignment have been selected. This strategy ensure that, the adversarial perturbation is concentrate to only the informative frame while maximizing the similarity between the original and the adversarial videos. Rather than applying perturbation uniformly across entire frame, we utilize masking strategy for each critical frame based on YOLOv8 object detector [7] that constrains adversarial perturbation to semantically meaningful regions and restrict the adversarial perturbation to these regions. This mask-guided perturbation strategy ensure that we disrupt the cross-modal alignment and enhancing both the effectiveness of the attack and the imperceptibility of the perturbation to human observe. We also extend the adversarial attack framework to disrupt the coherence of temporal domain that explicitly perturbs the movement regions in the video by taking the difference between each two adjacent frames. Our STCA attack leverages LanguageBind video encoder [8] to generate adversarial video by calculate frame-level visual feature and optimize adversarial perturbations that maximize the deceptive of temporal relationship. The loss function of STCA attack is aimed to minimize the cosine similarity between features of consecutive frames weighted by motion mask ensuring that, the adversarial perturbations target regions of high temporal dynamics as illustrated in figure 1 . Fig. 1: Enter Caption We conduct our comprehensive experiments on the BDD100K [9] and nuScenes [10] datasets across three divers VLMs: Video LLaVA-7B [11], Qwen2.5-VL-7B [12], and Dolphin [13]. We adopt Jaccard word-overlap similarity to ensure model rigorous. Our experimental results demonstrate the effectiveness of our attack against these three models on both datasets. The principal contributions of our framework summarized as follows: • We propose three-stage adversarial attack framework that combines modalities expansion, spatial, and temporally-aware cross-modal perturbation designed to target VLM models in autonomous driving systems. • We conduct our robustness evaluation of driving-specefic VLMs on the BDD100K dataset contain 1000 driving video and nuScenes datasets with 85 scenes against three divers target model. • We demonstrate that adversarial perturbation transfer effectively across divers VLMs models under the black-box threat that poses serious security implications. • We observe that, driving-specific VLMs such as Dolphin exhibit comparatively more robustness than general-purpose model. • Our finding highlights the urgent need adversarial defense mechanism in autonomous driving pipelines. The rest of our paper introduces related work in third section, problem and motivation in the fourth section. In the fifth section, we introduce our proposed framework in detail. Finally, our finding results and discussion is presented in section 6. I Related Work I-A Adversarial Attack against VLM Recently, the rapid proliferation of VLM in real-world application lead up to the extension of adversarial attack from uni-model systems to multi-model. Zhang et al. [6] was the first paper observe that, by perturbing each modality separately, the attack will be degrade due to the contradiction between different modalities. They propose novel multi-modal attack called collaborative multimodal attack (co-attack) which collectively attack both image and text simultaneously on two VLM architectural types Fused VLM and aligned VLM and three V+L tasks, visual entailment, visual grounding, and image-text retrieval. They approve the VLM is susceptible to adversarial attack, especially that crafted to perturb image and text representation in joint embedding space. Zhou el al. [14] extend this work by construct universal adversarial patch targeting all downstream tasks of victim cross-modal pretrained encoder such as CLIP model. Their AdvCLIP approach that downstream-agnostic adversarial example, could be built without target specific task a model would be deployed on. However, their empirical evaluation to classification and retrieval tasks, leaving other tasks unexplored. Yin et al [15] have been explored the adversarial attack in black-box setting by propose Vision Language Attack Strategy (VLattack) made up of two primary levels: single-modal level, and multi-modal level. They observed that models fine-tuned in specific task inherit the adversarial vulnerabilities of their pre-trained visual encoder. Zhao et al. [16] carried out a large scale evaluation of adversarial across several large VLM, they discovered all powerful tested VLM models remain vulnerable to adversarial attack. Recently, Zhang el al. [17] proposed AnyAttack that leverage the clean image to be transferred to attack other different VLMs and generating any desired output by training their generator on LAION 400M dataset. While Lu et al. [18] developed set-level Guidance attack (SGA) that enhance transferability by generating adversarial perturbation across divers cross-modal interaction of multiple text-image pairs. They showed the stronger transferability when attacked all the modalities simultaneously. in case of diffusion model leveraged by Guo et al. [19] in order to generate transferable adversarial example against VLM. Their attack achieve robust performance and strong quality. Unlike previous existing methods that suffer from high computational cost due to complex structure and number of iterations, their AdvDiffVlm utilize Adaptive Ensample Gradient Estimation (AEGE) to modify the score function thereby embedding adversarial semantic that enhance transferability. Recent work by Liu et al. [20] introduce maximizing Information Entropy (MIE) to enhance attack efficacy by directly inducing layer by layer perturbation in feature representation. They observed internal feature perturbation is more effective and comprehensive than approaches that only rely on final output layer. I-B Adversarial Attack against Video model The extension of static image to video model is arising from the temporal dimension, As each frame in video can be treated as a naive application of image level attack. Wei et al. [21] targeted set of temporal translated video to apply adversarial perturbation on them. To achieve better transferability than image attack method, they ovoid overfitting to the white box model. Due to the high computational cost, subsequently, they aimed in [22] to attack video CNN and ViT models by generate adversarial video from white-box image model named Image To Video Attack (I2V). Their approach here easy to perform and achieves better performance without training video model. In their further work [23], They assess the robustness of video recognition model spatial and temporal focus attack which perturb only temporally salient regions instead of disturbing all frames uniformly that reduce attack query numbers. Kim et al. [24] exploit image models and image data in their Universal Adversarial Perturbation (UAP) framework to attack video models using image-level surrogate model. Additionally, they proposed Breaking Temporal Consistency (BTC) to attack temporal aspect of video by reducing the features similarity between consecutive frames and by reducing the similarity between UAPs (temporal similarity loss). whereas the current studies of video adversarial attack target action recognition model, we target video language model performing scene description of autonomous driving scenario that demand to measure success by semantic divergence in natural language output not by classification accuracy on fixed label set. I-C Adversarial Attack against Autonomous Driving The intersection of autonomous driving and adversarial robustness has garnered growing attention due to the high-stakes implications of perception error in this field. Early research in this area focused on attacking object detector. Several studies showed that adversarially designed patches applied to road surface or vehicle could cause detection system to misclassify object in their range of view. Chung et al. [25] in their Typography-based attack are among the first emphasize practical transferability in traffic scenes, relying on decision-making capabilities of VLM. They propose dataset-agnostic framework for automatically generating false answer and linguistic augmentation to misdirect image-level and region-level reasoning in realistic road context. PG-Attack, proposed by Fu et al. [26] similarly targets deployed vision foundation models with a black-box that combines precision mask perturbation PMP with deceptive text patches (DTP) techniques, and their results suggest strong empirical defense across many VLM models. Zhang et al. [27] explicitly framed as the first attack tailored to VLM in autonomous driving, thye proposed ADvLM which marks as a sharper methodological step identified AD-specific difficulties: various prompt paraphrased but semantically equivalent and time-series scene with multiple viewpoint. Their scenario associated frame selection and semantic invariant prompt set it apart from typography and PG-Attack which are more transfer- and patch-oriented and less clearly constructed around temporal driving semantics. Recently, studies broaden both the attack surface and autonomy setting, CAD [28] is positioned directly against ADvLM, they contends that, the field has only a white-box AD-specific attack and present the first black-box attack tailored for AD VLM using decision-chain disruption and risky scene induction. There is an clear chronological relationship, because CAD extends the similar safety-critical setting while easing attacker access and displaying actual route-completion degradation. The goal of trustworthy-AV is more than attack proposal [29]; it is a robustness benchmark among vision encoders under targeted and untargeted adversarial conditions with sim-CLIP exhibiting the best model-dependent resilience. Its limitation is that it evaluates encoder robustness instead of the entire decision pipeline studied by CAD or ADvLM. Natural reflection [30] opens different line by switching from test time perturbation to training time poisoning utilizing faint reflection triggers to cause long replies and detrimental inference delay while maintaining clean-input performance. Its specialization is main limitation; it enhances rather than replace perturbation-based attack studies since it focuses on delayed response behavior rather than general semantic misdriving. Robotic VLA attack expand the realm beyond driving by demonstrating that jailbreaking-style textual attack can gain persistent control authority over vision language action [31]. Then ADVEDM [32] pushes embodied attack toward fine-grained semantic manipulation claiming that while selective object addition or removal preserve context and induce valid but incorrect actions in both driving and manipulation tasks, many previous attacks break too much scene semantic and thus produce invalid outputs. Two keys issues are combined in most recent studies. First, robustness varies significantly by architecture, LLaVA is much easier to attack under BIM, PGD, and spectral attack than Qwen2.5-VL in nondriving agent scenario [33]. Second, architecture diversity alone is insufficient to address this issue in autonomous driving, because physical patches transfer strongly between Dolphins, OmniDrive, and LeapVAD especially when model share a CLIP-based encoder [34]. The clearest bridge in these concern is is UCA [35], It is specifically criticizes previous AD attacks as being primarily logit-level, and then suggests a physically achievable feature-space camouflag attack that is independent of user text and robust to viewpoint and scale changes. The main unresolved limitation across the literature, there is no study address all of these in addition to temporal aspect at once, so current evidence is strong on isolated threat models but still fragmented across temporal dynamics, attacker access, and end-to-end control sequence. As the best of our knowledge, this is the first work to address the temporal dimension of adversarial attack in video language models in autonomous driving context. I Problem and Motivation Most existing adversarial attacks against VLM in autonomous driving focus primarily on static image perturbation. Our method aimed to deceive the model to produce incorrect decisions by carefully designing unnoticeable perturbations into the input data. Specifically, an attacker applies adversarial perturbations on a benign video rather than a static image. Formally, let a driving video sequence be represented as: V=f1,f2,…,fTV=f_1,f_2,…,f_T where fTf_T donate the TthT_th frame. A victim VLM model process the video and generate a textual response: y=ℱ(V,Q)y=F(V,Q). where (Q)(Q) represents a textual query or prompt, and (ℱ)(F) donate the multimodal inference function. VLM victim model donated as F0F_0 that induced to output undesirable response y∗y^* under adversarial perturbed input, y∗=(V,Q)y^*=P(V,Q) where (.)P(.) denotes the adversarial perturbation function. Formally, this manipulation is defined by maximizing the likelihood of y∗y^* response under adversarial input. maxPlogP(y∗∣(V,Q)) _P P (y^* (V,Q) ) (1) where PP is a probability function to obtain undesirable response y∗y^* when given a perturbed input. F0:Q→RF_0:Q R, where R represents the response domain. Additionally, Existing studies have overlook temporal interaction between consecutive frames. In autonomous driving, semantic understanding is not determined solely by isolated spatial content, but rather by temporal continuity, object motion, causal ordering, and scene evolution over time. Consequently, subtle perturbation will induce temporal reasoning failure. To characterize this issue, we define temporal semantic consistency as the difference between consecutive masked frames across time. (V)=1T−1∑t=1T−1cos(Φ(ft)−Φ(ft+1))C(V)= 1T-1 _t=1^T-1 ( (f_t)- (f_t+1)) (2) where Φ(⋅) (·) donate the joint visual-semantic embedding function and the difference between cos(⋅) (·) detect similarity of motion between each two consecutive frames. To best temporal understanding, model should maintains high consisitency under nature temporal transitions. For example the model may incorrectly infer that a pedestrian crossed after a vehivle stopped rather than before it stopped resulting in dangerous decision inconsistencies. The key challenge lies in refinement perturbations that can subtly alter the model’s perception, while maintaining high transferability and frames similarity under diverse conditions. Specifically, the adversarial examples must induce models’ decisions without compromising spatial and temporal consistency or raising human suspicion. Furthermore, autonomous driving systems induce unique challenges that differ this attack from other general VLM attacks [27] can summrize these challenges as shown in figure 2 and 3. Firstly, the diversity of textual instructions, one driving instruction can be expressed with different phrases, such as ”slow down” and ”reduce speed”. This mean that, these instructions are written in different expressions but convey the same intent and semantics. second challenge represented in figure 3 the nature of the visual scenario due to the perspective will be changed through driving. In our method, we will design universal perturbation δ that ensures a stable, efficient attack across different instructions that equivalent semantics using various VLM models to candidate the best instructions that match with frames of video. We need to design more robust perturbation that adopt visual changes and temporal dependencies. In our method, the most vital frames that represent the change of the time will be selected based in the below equations. =ft∣Sim(ft,C)>τS=\f_t (f_t,C)>τ\\ (3) F~t=ft+δt,ft∈ft,otherwise F_t= casesf_t+ _t,&f_t \\ f_t,&otherwise cases (4) V~=(F~) V=P( F) (5) Fig. 2: Diversity of textual prompts Fig. 3: Diversity of visual scenario Let FtF_t denote the sequence of frames and C represent a set of candidate captions generated for the entire video that identify semantically important frames within the video. the selected frame subset is defined in equation (4) where Sim(ft,C)Sim(f_t,C) measure the semantic relevance between frame and caption set, while τ denotes a selection threshold. Frames that exhibit strong semantic alignment with the captions are considered temporally important and are therefore selected for adversarial manipulation. After frame selection, the perturbations are applied only to the selected frame as shown in equation (5), where δt _t represent adversarial perturbation added to the selected frame. Finally, the perturbed frame sequence is utilized to apply a temporal attack. To address these gaps, our motivation stems from three key observations: 1. Existing VLMs are predominantly trained on spatial-text alignment objectives rather than temporal causal reasoning. 2. To design robust perturbations that induce spatial, temporal, and textual relations in the video. 3. To ensure that our STCA attack exhibit stronger transferability across various VLM architectures Therefore, the motivation of our work, to exploit and reveal the temporal vulnerability of VLMs in autonomous driving systems. By targeting temporal semantic reasoning instead of relying solely on spatial and textual perturbations. IV Methodology To address the above challenges, we propose STCA Temporal Coherence Adversarial attack which exploits spatial and temporal modalities using three phases as illustrated in figure 4: modalities expansion, spatial attack, and temporal attack. In phase 1, the initial dataset prepared and fed it into Gemini and Video-LLAVA to extract multi-captions for each video. After that, the best related captions will be candidate using CLIP model. For spatial phase, we employ CLIP model to extract critical frames and perspectives to apply perturbation for them. Finally, in a temporal attack, we trained LanguageBlind model to disrupt temporal reasoning consistency and transferred it to other models. The whole description of STCA adversarial attack is illustrated in Algorithm 1. Fig. 4: Our proposed framework consisting from three main stages: Modalities expansion, Spatial attack and temporal attack. Algorithm 1 STCA Adversarial Attack Dataset D, video frames v=f1,f2…ftv=\f_1,f_2… f_t\, CLIP Vision Encoder EvE_v , CLIP Textual Encoder EtE_t Adversarial Video //Selected captions for each video for each video vi∈v_i do CG←c1G,c2G…ckGC^G \c^G_1,c^G_2… c^G_k CL←c1L,c2L…ckLC^L \c^L_1,c^L_2… c^L_k C←CG∪CLC C^G∪ C^L for each caption ck∈Cc_k∈C do for each frame ft∈vif_t∈v_i do vt←Ev(ft)v_t E_v(f_t) uk←Et(ck)u_k E_t(c_k) //Compute cosine similarity st,k=cos(vt,uk)s_t,k= (v_t,u_k) end for //Compute caption score S(ck,vi)=1T∑t=1Tst,kS(c_k,v_i)= 1T _t=1^Ts_t,k end for //Select top-ranked caption C∗=TopM(ck,vi)C^*=TopM (c_k,v_i ) end for // Caption-guided frame selection for each video vi∈v_i do for each caption tj∈C∗t_j∈C^* do j←Et(tj)‖Et(tj)‖2u_j← E_t(t_j)\|E_t(t_j)\|_2 end for ¯←∑j=13j‖∑j=13j‖2 u← _j=1^3u_j \| _j=1^3u_j \|_2 for each frame fi∈vif_i∈v_i do i←Ev(fi)‖Ev(fi)‖2v_i← E_v(f_i)\|E_v(f_i)\|_2 si←i⊤¯s_i _i u end for ℱ∗←argmax⊆1,…,N,||=k∑i∈siF^*← S \1,…,N\,|S|=k _i s_i end for for each video vi∈v_i do for each frame Fj∈ℱ∗F_j∈F^* do ℳ←YOLOv8 detector(Fj)M← YOLOv8 detector(F_j) iadv←Clipϵ(fj⋅ℳi(iadv+α⋅sign(k))⋅(1−ℳi)CLOSEi^adv _ε\! (f_j·M_i(i^adv+α·sign(g_k))·(1-M_i ) // Temporal Coherence Attack ℳmotion=ℳj−ℳj+1M_motion=M_j-M_j+1 δt←α⋅sign(k)⋅ℳmotion _t←α·sign(g_k)·M_motion itadv←Clipϵ(itadv+δt)i_t^adv _ε (i_t^adv+ _t ) end for end for IV-A modalities expansion In the modalities’ expansion, we construct multi-caption containing diverse textual instruction with the same semantic intent. In our proposed framework, we employ Gemini 2.5 [36] and Video-LLVA [11] to generate semantically equivalent captions for each video based on time order of the whole driving scenario. After that, we utilizes semantic entropy [37] to filter redundant captions and rank semantically informative descriptions for temporal frame selection using CLIP-based visual semantic alignment. Given an input video V, our framework begin to generate set of captions C=CG∪CLC=C^G∪ C^L where CGC^G represents captions generated by Gemini and CLC^L denotes to the captions generated by Video-LLAVA. Thus, all captions generated are defined as C=c1,c2,…,cKC=\c_1,c_2,…,c_K\ where K donate the total number of generated captions. To candidate the best three captions among them, we evaluate the semantic relevance to the visual content of the video. CLIP [2] is used as a shared vision-language embedding model. Each frame ftf_t is encoded using the CLIP visual encoder vt=ΦvCLIPftv_t= _v^CLIPf_t, where each caption CKC_K is encoded using CLIP text encoder uk=ΦcCLIP(ck)u_k= _c^CLIP(c_k). The visual semantic similarity between frame ftf_t and caption ckc_k is computed as equation follow: cos(vt,uk)=vt⊤uk|vt|2|uk|2 (v_t,u_k)= v_t u_k|v_t|_2|u_k|_2 (6) Since the generated captions are intended to describe the entire video rather than individual frames, the final relevance score of each caption is obtained by averaging its similarity across all sampled frame: 1T∑t=1Tcos(vt,uk) 1T _t=1^T (v_t,u_k) (7) While caption with higher similarity value indicate stronger semantic correspondence aligned with visual content of the video. Finally, the top-ranked captions are selected as: C=TopK(s(ck,v))C=TopK (s(c_k,v) ). This caption selected strategy enable the framework to retain captions that are strongly grounded in the visual semantics of the input video as shown in figure 5. Fig. 5: Caption selected strategy IV-B Spatial Attack Stage Instead of perturbing all frames uniformly, we introduce a caption-guided frame selection strategy to identifies semantically informative frames using caption-guided visual-semantic alignment. We aim to focus on frames that exhibit stronger semantic correspondence with candidate captions. Given video V consisting of N frames f1,f2…fN\f_1,f_2… f_N\ and set of candidate caption C associated with the video. We employ CLIP as a cross-modal feature extractor. For each frame fif_i, we extract its visual embedding using the CLIP image encoder EiE_i, and for each caption tit_i, we extract its textual embedding using the CLIP text encoder EtE_t: vi=EI(fi)||EI(fi||2,ui=ET(tj)||ET(tj||2v_i= E_I(f_i)||E_I(f_i||_2,u_i= E_T(t_j)||E_T(t_j||_2 (8) The aggregated text representation is computed as the mean of all caption embeddings: u¯=∑j=1Muj‖∑j=1Muj‖2 u= _j=1^Mu_j|| _j=1^Mu_j||_2 (9) The semantic similarity between frame fif_i and the caption set is measured via cosine similarity si=vi⊤u¯s_i=v_i u. The top K frames are selected based on their similarity scores: ℱ∗←argmax⊆1,…,N,||=k∑i∈siF^*← S \1,…,N\,|S|=k _i s_i (10) After select vital frames, we applied mask to target specific areas of the selected frames that semantically significant as seen in figure 6. Our mask generation method based on YOLO object detector to concentrate the adversarial perturbation to regions containing pivotal objects. Given frame fif_i, we apply the YOLOv8 detector D to obtain a set of K bounding boxes. ℬi=(fi)=b1,b2,…,bK,bK=(x1k,y1k,x2k,y2k)B_i=D(f_i)=\b_1,b_2,…,b_K\,b_K=(x_1^k,y_1^k,x_2^k,y_2^k) where each bounding box bkb_k is retained only if it is confidence score ckc_k exceeds the threshold δ. The binary precision M∈0,1H×WM∈\0,1\^H× W is constructed as follows: M~ij=1,if ∃,bk∈ℬi:x1k≤j≤x2k and y1k≤j≤y2k0,otherwise M_ij= cases1,&if ∃,b_k _i:x_1^k≤ j≤ x_2^k and y_1^k≤ j≤ y_2^k\\ 0,&otherwise cases Where Mij=1M_ij=1 are designated as attack zones corresponding to detected object, while regions where Mij=0M_ij=0 remain unperturbed. Fig. 6: Sample of Mask Generation For each selected frame fif_i and its corresponding mask MiM_i, we generate adversarial example using the Precision Mask Perturbations (PMP) framework [26].The adversarial perturbation is optimized to maximize the misalignment between the visual and textual representations in the shared embedding space. For each video, we extract the textual embedding of each caption CjC_j, the adversarial optimization is formulated as: iadv=argmax‖i′−fi‖∞≤ϵℒ(EI(i′),ET(Cj)j=13CLOSE\ i^adv= ||i -f_i||_∞≤ε L(E_I(i ),\E_T(C_j)\_j=1^3\ (11) Where i′i represent adversarial frame, fif_i represent original frame, EIE_I is image encoder and ETE_T donate to textual encoder. The maximum perturbation budget should not exceed ϵε value. ℒL is the adversarial loss function defined as: ℒ=−∑j=33(EI(i′).ET(Cj))L=- _j=3^3(E_I(i ).E_T(C_j)) (12) The precision mask MiM_i ensure that perturbations are restricted to semantically meaningful object regions. The multi-scale augmentation strategy applies the perturbation across scales S=0.5,0.75,1.0,1.25,1.5S=\0.5,0.75,1.0,1.25,1.5\ to enhance transferability. IV-C Temporal Attack Stage While spatial attack successfully disrupts the semantic alignment between individual frames and their corresponding captions, it does not explicitly target the temporal dynamic that VLM model upon for video understanding. To address this limitation, we propose a temporal adversarial attack that degrading the temporal coherence between two consecutive frames, and disrupting the video-text semantic alignment. We hypothesize that by simultaneously corrupting the spatial semantic and temporal coherence of the video, the model’s ability to understand and describe driving scenarios can be significantly impaired. The adversarial video produced in spatial attack, is fed into the temporal attack to apply additional temporally-aware perturbations. To focus perturbations on semantically meaningful dynamic regions, we compute motion masks from consecutive masks of two adjacent frames. Given the binary mask M, the motion mask between frames t and t+1t+1 is defined as: ℳmotion=|ℳt+1−ℳt|>mM_motion=|M_t+1-M_t|>T_m where mT_m =0.1 is the binarization threshold. This motion masks ensure that perturbation are restricted to regions where object are actively moving, preserving the visual quality of static background region. The total adversarial loss is calculated by Text-Visual Alignment Loss that measure the cosine similarity between the video representation and the textual description embedding: ℒtext=1||∑j=1||cos(EV(iadv),ET(tj))L_text= 1|T| _j=1^|T| (E_V(i^adv),E_T(t_j)) (13) Where EVE_V denotes the LanguageBind video encoder, ETE_T denotes the language encoder, and T is the set of captions associated with video. Maximizing ℒtextL_text drive the adversarial video representation away from its corresponding textual description. Additionally, we calculate Temporal Coherence Loss which measure the cosine similarity between features of temporally adjacent frame pairs. ℒtemporal=1K−1∑t=1K−1t⋅cos(EV(itadv),EV(it+1))L_temporal= 1K-1 _t=1^K-1w_t· (E_V(i_t^adv),E_V(i_t+1)) (14) Where K is the total number of motion frames, and consecutive frame pairs are used to capture full temporal coherence across the entire video sequence. V Experiments V-A Experimental Setup Dataset. We evaluate our proposed attack framework on the BDD100K dataset, a large-scale autonomous driving benchmark containing diverse driving scenarios across multiple cities, weather conditions , and times of day. Following our data preparation pipeline, we select 1000 videos in Caption-Guided frame selection strategy, from which 800 videos are used for adversarial attack generation and evaluation. Additionally, we evaluate our approach on nuSenes autonomous driving dataset, we utilize 85 scenes from front-camera which each scene span 20 seconds and contains approximately 40 frames. Target Model. we evaluate our attack against three target video language model with diverse architecture and training paradigms, Video-LLaVA-7B, an open source video language model that align video and image encoder with LLaMA backbone as its language decoder through shared projection mechanism. Second recent model is Qwen2.5-VL-7B which also open source multi-modal that integrate advanced visual perception with deep language understanding, achieving competitive performance in video understanding tasks. The third victim model is Dolphin that domain specific VLM finetuned for autonomous driving scenes and trained on video-text pair driving specific. Dolphins is built upon OpenFlamingo architecture with MPT 7B language backbone. Surrogate Model. For spatial attack stage, we use TCL as the white box surrogate model with vision transformer ViT-B/16 with 12 transformer layer and 85.8M parameters, built upon the ALBEEF architecture with a ViT-B/16 visual encoder and BERT-base text encoder. For the temporal attack stage, we utilize LanguageBind, an open source model, it align different modalities unless text into a shared video language embedding space using language as the anchor. it is framework extends CLIP’s ViT-L/14 encoder. Attack Hyperparameters. For frame selection, we use CLIP ViT-B/32 to compute cosine similarity between frames and captions, selecting 60 frames per video. Object detection is performed using YOLOv8s with confidence threshold 0.25 and IoU threshold 0.45 attack parameters. For spatial attack, we adopt PGD based optimization with maximum perturbation budget ϵ=32/255ε=32/255 under the ℓ∞ ∞-norm constraint , step size α=1/255α=1/255, and number of iterations =20T=20 with momentum decay λ=0.9λ=0.9. Multi-scale augmentation is applied with scales =0.5,0.75,1.0.1.25,1.5S=\0.5,0.75,1.0.1.25,1.5\. For the temporal attack, we set ϵ=16/255ε=16/255 , step size α=2/255α=2/255, and number of iterations =20T=20. Perturbations are applied to regions identified by motion-guided mask derived from consecutive YOLO mask difference to ensure the adversarial attack is concentrated on temporally dynamic regions. Implementation Details. All experiments are conducted on NVIDIA A100 GPU in Google Colab Pro. To reduce memory, video LLaVA and Qwen2.5-VL are loaded in 16 bit and bfloat 16 precision, respectively. Dolphins is evaluated with its official checkpoint and LoRA fine-tuning configuration. To accommodate GPU memory constraints, we evaluate Dolphin in half-precision (float16). For each model, we query each video twice with identical prompt to ensure the divergence is caused by adversarial perturbation only- once on the original frame and other on adversarial perturbed frame. Evaluation Metrics. We evaluate our attack using two complementary metrics: • Attack Success Rate (ASR): the percentage of videos for which VLM model generates a description after the attack that differs significantly from its description before the attack. We measure response divergence using word-overlap similarity between pre-attack and post-attack response; when the overlap fall below 0.5, it is considered to successful attack. • Structural Similarity Index (SSIM): measure perceptual quality of adversarial frames relative to original frames. V-B Main Results Performance of our spatial and temporal attacks against all models on 800 driving videos from BDD100K dataset is presented in Table I and nuScene dataset presented in table I, we report the attack success rate and mean SSIM achieved in each stage. Our framework STCA attack consistently outperform the spatial attack only across all target models and both datasets, which confirm our hypothesis that temporal coherence attack is more effective than spatial attack alone for VLM. The spatial attack alone achieves an attack success rate 32.6 % with mean SSIM 0.93 against video-LLaVA-7B on BDD 100K showing that spatially perturbation can mislead model in about one-third of cases. Once the temporal attack stage is added, attack success rate arise more than double to 71% with SSIM 0.82 which remain well above the threshold commonly adopted in adversarial attack literature. The two failure mode we target: inter-frame (temporal) and intra-frame (spatial) are complementary vulnerabilities, the sample of its result is shown in figure 7. Among the open-source VLM model evaluated in BDD 100K Qwen2.5-VL-7B which proves more susceptible to our STCA attack with attack success rate 84.2%. Compared to Video-LLaVA’s pooling based temporal aggregation, Qwen2.5-VL is more susceptible to localized and motion guided by our attack,the sample of its result is shown in figure 8. By contrast, Dolphins exhibit lower susceptible to our attack, it achieves approximately 47% in spatial attack and 46.2% in our full pipeline. The Dolphins output reveals that, the model has learned driving scene vocabulary as a result of its domain-specific fine-tuning. This finding indicates that domain specialization in VLM may confer partial adversarial resilience,the sample of its result is shown in figure 9. To our knowledge, this observation has not been previously reported for autonomous driving VLM. Fig. 7: Captions resulted from Video-LLaVA model before and after attack Fig. 8: Captions resulted from Qwen2.5-VL model before and after attack Fig. 9: Captions resulted from Dolphins model before and after attack TABLE I: Results of our attack in BDD100K dataset Model Spatial Attack Spatial + Temporal Attack Video LLaVA 32.6% 71% Qwen2.5-VL 45% 84.2% Dolphin 46.9% 46.2% SSIM 0.93 0.82 TABLE I: Results of our attack in nuScene dataset Model Spatial Attack spatial + Temporal Attack Video LLaVA 57.6% 83% Qwen2.5-VL 64.7% 96.5% Dolphin 37.6 % 47.1% SSIM 0.98 0.79 On the nuSenes dataset, our spatial attack achieves attack success rate 57.6%, 64.7%, and 37% and achieves in our full pipeline 83%, 96.5%, and 47.1% in video-LLaVA, Qwen2.5-VL, and Dolphins respectively, with mean SSIM 0.89 in spatial temporal and 0.79 in both spatial and temporal stage. Dolphins remains the most robust and Qwen2.5-VL the most susceptible indicating that, the relative ordering of model susceptibility is stable across datasets. The absolute attack success rate on nuScenes are somewhat higher than those on BDD100K due to the shorter temporal extent of nuScenes keyframe sequence. Collectively, the high attack success rate confirm that our proposed temporally-aware attack framework can deceive various models across autonomous driving benchmark. In term of SSIM value, it remains high which indicate that the adversarial attack of our framework is imperceptible observers as shown in figure 10. Fig. 10: SSIM demonstration between original frames and adversarial frames Additionally, we introduce in table I two baseline methods, PGD and FGSM and compare them with our framework. Both PGD and FGSM, when applied to the TCL surrogate model under the black box transfer setting, yield substantially low attack success rate than our proposed framework. Specifically, PGD achieves ASR of 47.2%, 44.8%, and 36.2% against Video-LLaVA, Qwen2.5-VL, and Dolphins respectively, while FGSM achieves 25.2%, 35.4%, and 18.1% compared to 71%, 84.2% and 46.9% in our proposed framework. We report in table IV the most recent studies evaluated in nuSenes dataset. In Video-LLaVa and Qwen2.5 we outperform state of the art methods. In Dolphin model, [27] utilized Dolphins in Autonomous driving video and final score drop up 7.49% and 3.09%. Also, in other work [28] they outperform prior method by 13.43% .For image, Wang et al. [32] evaluate dolphins in their AdveDm framework achieving ASR 86%, this largely ASR attributed to the white box attack setting by leveraging complete access to the target model. Based on our knowledge, there is no previous work evaluate their framework on BDD100K. TABLE I: Compare our proposed method with baseline methods Method Video-LLaVa Qwen2.5-VL Dolphins PGD 47.2% 44.8% 36.2% FGSM 25.2% 35.4% 18.1% Our (Spatial attack only) 32.6 % 45% 46.9% Our (Full pipeline) 71% 84.2% 46.2% TABLE IV: Comparison of our framework with state of the art methods Dataset Victim Model Reference Attack Type ASR % nuScene dataset Video-LLaVA IN et al. (2024) [38] Physical backdoor with common object trigger different based on trigger ranged from 65.3 % to 89.3% Wang et al. (2026) [32] Fine grained object-semantic attack 75% Spatial Attack only Transerable Spatial Attack 57.6% Spaital+temporal Transerable Spatial-Temporal Attack 83% Qwen2-VL Liu et al. (2025) [30] Natural-reflection backdoor 70.92% Spatial Attack only Transerable Spatial Attack 64.7% Spaital+temporal Transerable Spatial-Temporal Attack 96.5% V-C Ablation Study We conduct an ablation study to observe the impact of different setting on 200 video from BDD100K. First, we present the result of STCA attack with varying perturbation budgets ϵε (i.e 4/255 , 8/255, 12/255, 16/255, 32/255) across three model with n=20n=20. The specific budgets and results are shown in figure 11 (a), ASR increase consistently with perturbation budget across all three models. Additionally, we tested STCA with different step n (i.e 5, 10, 20, 30, 40) using fixed perturbation budget ϵ=16/255ε=16/255 and step size α=2/255α=2/255, generally, the attack strength increase with more iteration step as shown in figure 11(b). (a) (b) Fig. 11: (a) The relationship between ASR and Perturbation budget. (b) The relationship between the number of ASR and the number of steps. V-D Discussion And Analysis The architecture design of modern vision language model is play a crucial role in achieving higher ASR in our temporal attack than purely spatial attack. These models trained to extract the meaning of video from the temporal relationship between frames, motion pattern, and event sequence that unfold across time. Our attack disrupt precisely the cross frame consistency signal that VLM rely on for temporal reasoning, we achieve this by targeting cross-modal embedding space generated by LanguageBind model with motion guided perturbation applied to temporally dynamic regions. Spatial attack stage disrupting the individual frame’s semantics, while temporal stage disrupting a coherence scene description due that, the last delivers substantial gain over the spatial stage only. The two stages complementary target aspects of model’s visual understanding. The key design that contributes to increase both effectiveness and imperceptibility of our attack is the restriction of spatial perturbation to crucial object in each frame and temporal attack to motion- salient regions generated by our YOLO based motion mask computation. Compared to random spatial perturbation strategies, our motion-guided masking achieve higher ration of semantic disruption since moving object already contain variation and temporal inconsistency across frames. The consistent advantage of our attack against Qwen2.5-VL relative to video-LLaVA across both datasets which reflect architectural distinction in how the two model treat temporal information. The substantially lower ASR against Dolphins compared to general purpose open-source models. We finding there is a relationship between model specialization and adversarial robustness so a model with highly tuned for specific domain may exhibit greater robustness to adversarial attack. Additionally, the most practically significant finding of our evaluation, is the successful transfer of our attack from the TCL and LanguageBind surrogates to three architecturally diverse target models. The main factor contribute to this transferability is the motion-guided temporal perturbation proposed in our attack target the temporal coherence of sequential frames. VI Conclusion This paper introduce two-stage (STCA) attack targeting video language model in safety-critical issue of autonomous driving. The motivation of our proposed framework is that, the temporal dimension of video input is unexplored in current VLM architectures. Our first stage target spatial object in each frame guided by YOLO-derived mask, followed by the second temporal coherence attack stage target the motion-salient regions between consecutive frames difference. We evaluate our framework across three architecturally divers models: Video-LLaVA, Qwen2.5, and Dolphins, on two large scale autonomous driving benchmarks, BDD100K and nuScenes datasets. Our experimental results demonstrate that, our temporal attack outperforming most recent studies and our spatial-only baseline including PGD and FGSM applied under identical surrogate model. VII References References [1] J. Li, R. R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. C. H. Hoi, “Align Before Fuse: Vision and Language Representation Learning with Momentum Distillation,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 34, p. 9694–9705, 2021. [2] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., “Learning Transferable Visual Models From Natural Language Supervision,” in Proc. 38th Int. Conf. Mach. Learn. (ICML), vol. 139, p. 8748–8763, 2021. [3] J. Yang, J. Duan, S. Tran, Y. Xu, S. Chanda, L. Chen, B. Zeng, T. Chilimbi, and J. Huang, “Vision-Language Pre-Training With Triple Contrastive Learning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), p. 15671–15680, 2022. [4] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and Harnessing Adversarial Examples,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2015. [5] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards Deep Learning Models Resistant to Adversarial Attacks,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2018. [6] J. Zhang, Q. Yi, and J. Sang, “Towards Adversarial Attack on Vision-Language Pre-Training Models,” in Proc. 30th ACM Int. Conf. Multimedia (ACM M), p. 5005–5013, 2022. [7] G. Jocher, A. Chaurasia, and J. Qiu, Ultralytics YOLOv8. [Online]. Available: https://github.com/ultralytics/ultralytics [8] B. Zhu, B. Lin, M. Ning, Y. Yan, J. Cui, H. Wang, Y. Pang, W. Jiang, J. Zhang, Z. Li, et al., “LanguageBind: Extending Video-Language Pretraining to N-Modality by Language-Based Semantic Alignment,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2024. [9] F. Yu, H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan, and T. Darrell, “BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), p. 2636–2645, 2020. [10] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuScenes: A Multimodal Dataset for Autonomous Driving,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), p. 11621–11631, 2020. [11] B. Lin, Y. Ye, B. Zhu, J. Cui, M. Ning, P. Jin, and L. Yuan, “Video-LLaVA: Learning United Visual Representation by Alignment Before Projection,” in Proc. Conf. Empirical Methods Natural Lang. Process. (EMNLP), p. 5971–5984, 2024. [12] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al., “Qwen2.5-VL Technical Report,” arXiv preprint, arXiv:2502.13923, 2025. doi: 10.48550/arXiv.2502.13923. [13] Y. Ma, Y. Cao, J. Sun, M. Pavone, and C. Xiao, “Dolphins: Multimodal Language Model for Driving,” in Proc. Eur. Conf. Comput. Vis. (ECCV), p. 403–420, 2024. [14] Z. Zhou, S. Hu, M. Li, H. Zhang, Y. Zhang, and H. Jin, “AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learning,” in Proc. 31st ACM Int. Conf. Multimedia (ACM M), p. 6311–6320, 2023. [15] Z. Yin, M. Ye, T. Zhang, T. Du, J. Zhu, H. Liu, J. Chen, T. Wang, and F. Ma, “VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Models,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 36, p. 52936–52956, 2023. [16] Y. Zhao, T. Pang, C. Du, X. Yang, C. Li, N.-M. Cheung, and M. Lin, “On Evaluating Adversarial Robustness of Large Vision-Language Models,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 36, p. 54111–54138, 2023. [17] J. Zhang, J. Ye, X. Ma, Y. Li, Y. Yang, Y. Chen, J. Sang, and D.-Y. Yeung, “AnyAttack: Towards Large-Scale Self-Supervised Adversarial Attacks on Vision-Language Models,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), p. 19900–19909, 2025. [18] D. Lu, Z. Wang, T. Wang, W. Guan, H. Gao, and F. Zheng, “Set-Level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-Training Models,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), p. 102–111, 2023. [19] Q. Guo, S. Pang, X. Jia, Y. Liu, and Q. Guo, “Efficient Generation of Targeted and Transferable Adversarial Examples for Vision-Language Models via Diffusion Models,” IEEE Trans. Inf. Forensics Security, vol. 20, p. 1333–1348, 2024. [20] C. Liu, Y. Wang, H. Cao, B. Liu, and D. Jiang, “Evaluating the Adversarial Robustness of Vision-Language Models via Internal Feature Perturbations,” IEEE Trans. Circuits Syst. Video Technol., 2025. [21] Z. Wei, J. Chen, Z. Wu, and Y.-G. Jiang, “Boosting the Transferability of Video Adversarial Examples via Temporal Translation,” in Proc. AAAI Conf. Artif. Intell., vol. 36, no. 3, p. 2659–2667, 2022. [22] Z. Wei, J. Chen, Z. Wu, and Y.-G. Jiang, “Adaptive Cross-Modal Transferable Adversarial Attacks From Images to Videos,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 46, no. 5, p. 3772–3783, 2024. [23] X. Wei, S. Wang, and H. Yan, “Efficient Robustness Assessment via Adversarial Spatial-Temporal Focus on Videos,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 9, p. 10898–10912, 2023. [24] H.-S. Kim, M. Son, M. Kim, M.-J. Kwon, and C. Kim, “Breaking Temporal Consistency: Generating Video Universal Adversarial Perturbations Using Image Models,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), p. 4325–4334, 2023. [25] N. Chung, S. Gao, T.-A. Vu, J. Zhang, A. Liu, Y. Lin, J. S. Dong, and Q. Guo, “Towards Transferable Attacks Against Vision-LLMs in Autonomous Driving with Typography,” arXiv preprint, arXiv:2405.14169, 2024. [26] J. Fu, Z. Chen, K. Jiang, H. Guo, S. Gao, and W. Zhang, “PG-Attack: A Precision-Guided Adversarial Attack Framework Against Vision Foundation Models for Autonomous Driving,” arXiv preprint, arXiv:2407.13111, 2024. [27] T. Zhang, L. Wang, X. Zhang, Y. Zhang, B. Jia, S. Liang, S. Hu, Q. Fu, A. Liu, and X. Liu, “Visual Adversarial Attack on Vision-Language Models for Autonomous Driving,” Mach. Intell. Res., 2026, doi: 10.1007/s11633-026-1667-4. [28] L. Wang, T. Zhang, Y. Qu, S. Liang, Y. Chen, A. Liu, X. Liu, and D. Tao, “Black-Box Adversarial Attack on Vision Language Models for Autonomous Driving,” arXiv preprint, arXiv:2501.13563, 2025. [29] A. A. Fime, M. Z. Hossain, S. Zaman, A. R. Shahid, and A. Imteaj, “Towards Trustworthy Autonomous Vehicles with Vision-Language Models Under Targeted and Untargeted Adversarial Attacks,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), p. 619–628, 2025. [30] M. Liu, S. Liang, K. Howlader, L. Wang, D. Tao, and W. Zhang, “Natural Reflection Backdoor Attack on Vision Language Model for Autonomous Driving,” arXiv preprint, arXiv:2505.06413, 2025. [31] E. K. Jones, A. Robey, A. Zou, Z. Ravichandran, G. J. Pappas, H. Hassani, M. Fredrikson, and J. Zico Kolter, “Adversarial Attacks on Robotic Vision-Language-Action Models,” in Proc. RSS 2025 Workshop ReliableRobotics, 2025. [32] Y. Wang, H. Zhang, H. Pan, Z. Zhou, X. Wang, P. Guo, L. Xue, S. Hu, M. Li, and L. Y. Zhang, “AdvEDM: Fine-grained Adversarial Attack against VLM-based Embodied Agents,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 38, 2025. [33] A. Paredes La Torre, “Adversarial Attacks against Modern Vision-Language Models,” arXiv preprint, arXiv:2603.16960, 2026. [34] D. Fernandez, P. MohajerAnsari, A. Salarpour, and M. D. Pese, “Understanding Adversarial Transferability in Vision-Language Models for Autonomous Driving: A Cross-Architecture Analysis,” in SAE Technical Paper Series, Paper 2026-01-0170, WCX SAE World Congress Experience, Detroit, MI, USA, Apr. 2026. [35] D. Kong, S. Yu, S. Liang, J. Liang, J. Gan, A. Liu, and W. Ren, “Universal Camouflage Attack on Vision-Language Models for Autonomous Driving,” arXiv preprint, arXiv:2509.20196, 2025. [36] Team Gemini, Google, “Gemini: A Family of Highly Capable Multimodal Models,” arXiv preprint, arXiv:2312.11805, 2023. [37] S. Farquhar, J. Kossen, L. Kuhn, and Y. Gal, “Detecting Hallucinations in Large Language Models Using Semantic Entropy,” Nature, vol. 630, no. 8017, p. 625–630, 2024. [38] Z. Ni, R. Ye, Y. Wei, Z. Xiang, Y. Wang, and S. Chen, “Physical Backdoor Attack Can Jeopardize Driving With Vision-Large-Language Models,” in Proc. AAAI Conf. Artif. Intell., vol. 39, no. 20, p. 21895–21903, 2025.