Paper deep dive

InfraPatch: Cross-Task Targeted Grayscale Patch Attacks on Infrared-Adapted Vision-Language Models

Chengyin Hu, Dingyi Lu, Jiaju Han, Xiang Chen, Weiwen Shi, Jiahuan Long, Yiwei Wei, Jiujiang Guo

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 61

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 9/3/2026, 4:10:49 AM

Summary

The paper introduces InfraPatch, a white-box, per-instance framework for generating targeted digital grayscale adversarial patches against Infrared Vision-Language Models (IR-VLMs). The method optimizes a compact single-channel patch (approx. 5% area) using proxy-guided placement and task-adaptive semantic objectives to induce specific target behaviors across image classification, captioning, and visual question answering (VQA). Evaluated on ten IR-adapted model variants (CLIP, BLIP-2, LLaVA, etc.) using synthetic infrared images, InfraPatch achieves high attack success rates (86-100%), demonstrating significant vulnerabilities in current IR-VLMs to localized perturbations.

Entities (9)

InfraPatch · method · 100%Adversarial Patch · concept · 95%Infrared Vision-Language Models · model-class · 95%Targeted Attack · concept · 95%BLIP-2 · model · 90%CLIP · model · 90%COCO · dataset · 90%DiffV2IR · method · 90%LLaVA-1.5 · model · 90%

Relation Signals (9)

InfraPatch → achievessuccessrate → 86.00% to 100%

confidence 95% · InfraPatch achieves targeted attack success rates from 86.00% to 100% across the ten variants.

InfraPatch → targets → Infrared Vision-Language Models

confidence 95% · InfraPatch, a white-box, per-instance framework for targeted digital grayscale patch attacks against IR-VLMs.

InfraPatch → appliestotasks → Visual Question Answering

confidence 90% · induces target behaviors in image classification, image captioning, and binary visual question answering.

InfraPatch → appliestotasks → Image Classification

confidence 90% · induces target behaviors in image classification, image captioning, and binary visual question answering.

InfraPatch → appliestotasks → Image Captioning

confidence 90% · induces target behaviors in image classification, image captioning, and binary visual question answering.

InfraPatch → evaluateson → CLIP

confidence 90% · We evaluate ten infrared-adapted model variants... On CLIP and BLIP-2...

InfraPatch → evaluateson → BLIP-2

confidence 90% · On CLIP and BLIP-2, proxy location search improves success...

InfraPatch → evaluateson → LLaVA-1.5

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Infrared vision-language models (IR-VLMs) have emerged as a promising paradigm for multimodal perception under low-visibility conditions, yet their robustness to targeted adversarial attacks remains poorly understood. Existing adversarial patch methods mainly study RGB-based models or a single downstream task and do not characterize whether localized perturbations can induce an intended semantic target in IR-VLMs. We propose InfraPatch, a white-box, per-instance framework for targeted digital grayscale patch attacks against IR-VLMs. InfraPatch optimizes a compact single-channel patch within an approximately 5% local-area budget, combines proxy-guided placement with task-adaptive semantic objectives, and induces target behaviors in image classification, image captioning, and binary visual question answering. We evaluate ten infrared-adapted model variants on 300 synthetic infrared-style images generated by applying DiffV2IR to a fixed 30-category COCO subset, using clean-conditioned targeted success criteria. InfraPatch achieves targeted attack success rates from 86.00% to 100% across the ten variants. On CLIP and BLIP-2, proxy location search improves success by 6.67 and 10.33 percentage points over optimized random placement, respectively; LLaVA-1.5 remains saturated near 100% under both settings. Patch-area and objective ablations further expose substantial differences in vulnerability across architectures and task formats. These results show that small grayscale patches can inject chosen target semantics across IR-VLM families under a controlled digital threat model, motivating stronger robustness evaluation for infrared multimodal systems.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

57,802 characters extracted from source content.

Expand or collapse full text

InfraPatch: Cross-Task Targeted Grayscale Patch Attacks on Infrared-Adapted Vision-Language Models CCS: Security and privacy Software and application securityCCS: Computing methodologies Computer vision Chengyin Hu, Dingyi Lu, Jiaju Han, Xiang Chen, Weiwen Shi, Jiahuan Long, Yiwei Wei, Jiujiang Guo © none Abstract. Infrared vision-language models (IR-VLMs) have emerged as a promising paradigm for multimodal perception under low-visibility conditions, yet their robustness to targeted adversarial attacks remains poorly understood. Existing adversarial patch methods mainly study RGB-based models or a single downstream task and do not characterize whether localized perturbations can induce an intended semantic target in IR-VLMs. We propose InfraPatch, a white-box, per-instance framework for targeted digital grayscale patch attacks against IR-VLMs. InfraPatch optimizes a compact single-channel patch within an approximately 5%5\% local-area budget, combines proxy-guided placement with task-adaptive semantic objectives, and induces target behaviors in image classification, image captioning, and binary visual question answering. We evaluate ten infrared-adapted model variants on 300 synthetic infrared-style images generated by applying DiffV2IR to a fixed 30-category COCO subset, using clean-conditioned targeted success criteria. InfraPatch achieves targeted attack success rates from 86.00%86.00\% to 100%100\% across the ten variants. On CLIP and BLIP-2, proxy location search improves success by 6.676.67 and 10.3310.33 percentage points over optimized random placement, respectively; LLaVA-1.5 remains saturated near 100%100\% under both settings. Patch-area and objective ablations further expose substantial differences in vulnerability across architectures and task formats. These results show that small grayscale patches can inject chosen target semantics across IR-VLM families under a controlled digital threat model, motivating stronger robustness evaluation for infrared multimodal systems. Keywords: infrared vision-language models, adversarial patches, targeted attacks, multimodal robustness, infrared imagery 1. Introduction Infrared sensing provides visual information in conditions where visible-light imagery can be degraded by insufficient illumination and other low-visibility factors (Hwang et al., 2015). When coupled with vision-language models, infrared observations can be converted into categorical predictions, open-ended descriptions, or answers to task-specific questions. Such semantic outputs may inform a downstream perception pipeline or a human decision-maker rather than remaining an isolated model score. Consequently, a perturbation that steers an infrared vision-language model (IR-VLM) toward a chosen but unsupported semantic target represents a more specific risk than an untargeted reduction in accuracy. Prior work has studied attacks on infrared classifiers and detectors, targeted attacks on RGB vision–language models, and illumination-transformation attacks that expose RGB-VLM sensitivity to lighting shifts (Edwards and Rawat, 2020; Wei et al., 2023c; Wei et al., 2023b; Zhao et al., 2023; Liu et al., 2025b). Yet, to the best of our knowledge, no prior work has systematically tested whether a localized grayscale perturbation can inject one attacker-chosen semantic target across heterogeneous outputs of infrared-adapted vision–language models. In particular, a target-class flip in classification does not directly characterize target-phrase generation or answer manipulation in visual question answering (VQA). A unified cross-task formulation is needed to compare these behaviors without conflating different notions of attack success. This cross-task setting presents both an optimization challenge and an evaluation challenge. For image classification, the model produces a fixed-dimensional class distribution, and a targeted success can be defined by the target becoming top-ranked with a positive margin over competing classes. For conditional text generation, the output is a free-form token sequence; the attack must optimize a target phrase and subsequently determine whether the generated text contains the intended object term under explicit normalization rules. For binary VQA, the relevant event is narrower: an image is valid only when the clean model explicitly answers “no” to a target-object question, and the attack succeeds only when the adversarial answer becomes “yes”. These outputs require different differentiable objectives, valid-sample filters, and success parsers. Reusing a classification-only attack metric would therefore obscure the task-specific semantics and could overestimate success when the clean output already contains the target. To study this problem, this paper introduces InfraPatch, a white-box, per-instance framework for targeted digital grayscale patch attacks on IR-VLMs. InfraPatch restricts the perturbation to one square, single-channel patch with a nominal area budget of 5% of the preprocessed image. The optimized grayscale values are replicated across the three model input channels and projected to the valid image range. The framework combines an early-exit proxy location search with task-adaptive semantic objectives. For each image, the search sequentially optimizes uniformly sampled legal placements under the complete attack objective. It immediately accepts the first candidate that satisfies the task-specific targeted-success criterion. The objective is adapted to target-class prediction, target-phrase generation, or a target VQA answer. The main contributions are: • To the best of our knowledge, the first systematic cross-task study of targeted adversarial patches for infrared-adapted VLMs, spanning classification, generation, and binary VQA, with success conditioned on clean outputs. • A unified attack framework that couples early-exit proxy location search with task-adaptive semantic objectives under a compact, nominal 5% grayscale-patch constraint. • An evaluation design spanning ten model variants from the CLIP, BLIP, OpenFlamingo, and LLaVA families across three output formats, with task-specific success parsing and explicit exclusion of samples whose clean outputs already satisfy the target condition. • Code-aligned comparisons with AdvIB, AdvIC, and HCB on one representative model per task, preserving each baseline’s structured perturbation and derivative-free search while unifying the victim checkpoint, target, preprocessing, valid samples, nominal area budget, and final success parser. 2. Related Work 2.1. Adversarial Patches and Targeted Attacks An adversarial patch confines optimized pixels to a local image region and relaxes the small-norm constraint commonly imposed on image-wide examples. Brown et al. introduced universal patches that remain robust under image transformations and drive an image classifier toward an attacker-chosen class (Brown et al., 2017). Physical-world studies subsequently optimized localized patterns under changes in viewpoint, distance, illumination, and printing; for example, robust visual perturbations can induce targeted traffic- sign misclassification after re-imaging (Eykholt et al., 2018). These studies establish that locality does not preclude strong targeted behavior, but their success events are defined over a fixed classifier or detector output. The optimization and evaluation of such attacks also build on broader adversarial-robustness principles. Gradient-based attacks exploit the local sensitivity of learned decision functions (Goodfellow et al., 2015), while strong optimization-based evaluations show that robustness conclusions depend on whether the attack objective and stopping criterion faithfully test the claimed property (Carlini and Wagner, 2017). Robust optimization instead casts training as a worst-case inner maximization problem (Madry et al., 2018). For physical evaluation, expectation over transformation (EOT) explicitly optimizes over a distribution of views, scales, and other nuisance factors (Athalye et al., 2018). Localized attacks have also been adapted from image classification to person detection. Thys et al. optimized printable patches to suppress person detections in surveillance imagery (Thys et al., 2019), and Xu et al. modeled non-rigid deformation to realize an adversarial T-shirt under changing poses and viewpoints (Xu et al., 2020). Together with robust traffic-sign perturbations, these studies demonstrate that a physically evaluated patch requires transformation-aware optimization and re-imaging tests. InfraPatch shares their localized threat surface but not their physical claim: its single-channel constraint controls the digital input representation and should not be interpreted as a thermal material model. Infrared attacks face different realization constraints because thermal cameras respond to emitted radiation rather than visible texture. Prior work demonstrated adversarial examples on networks trained with simulated infrared imagery (Edwards and Rawat, 2020). Wei et al. optimized the shape and placement of insulating patches to suppress infrared pedestrian and vehicle detectors (Wei et al., 2023c), and later optimized one shape-based patch against visible and infrared detectors simultaneously (Wei et al., 2023b). Both methods target object detection in the physical world. Score-based infrared attacks further exploit wearable hot–cold structures. HCB jointly optimizes the shape and placement of grid-structured thermal blocks (Wei et al., 2023a); AdvIB searches multiple rotatable infrared blocks with differential evolution (Hu et al., 2024a); and AdvIC uses particle swarm optimization to search two quadratic Bézier curves (Hu et al., 2024b). These methods were originally designed as untargeted disappearance attacks against infrared pedestrian detectors; here their representations and searches are adapted to targeted IR-VLM objectives. Complementary defense work uses thermal-radiation modeling to guide adversarial training for infrared object detection (Zhao et al., 2026). InfraPatch instead studies a digital, single-channel intensity patch and asks whether the same chosen semantic can be induced across classification, conditional generation, and VQA; it therefore does not claim the physical realizability established by thermal-material attacks. 2.2. Robustness of Vision-Language Models Vision–language pre-training creates shared visual–textual representations, which expose attack surfaces beyond conventional class logits; a recent survey organizes these attacks by modality, objective, and model access (Liu et al., 2025a). Targeted visual examples crafted on CLIP- or BLIP-like surrogates can transfer to generative models and elicit attacker-specified responses (Zhao et al., 2023), while set-level cross-modal guidance improves transfer across pre-trained vision–language models and downstream tasks (Lu et al., 2023). At the generative end of the spectrum, image-hijack attacks optimize visual inputs to make LLaVA follow an adversary-selected behavior (Bailey et al., 2024). Visual adversarial examples and typographic visual prompts can also bypass multimodal safety alignment (Qi et al., 2024; Gong et al., 2025). These results demonstrate that manipulating the visual modality can alter linguistic behavior, not merely decrease recognition accuracy. Recent studies further examine adversarial VLM behavior in autonomous-driving perception (Zhang et al., 2026) and construct spatial–spectral perturbations for physical-world large VLMs (Liu et al., 2026). Although these works broaden the operational settings of VLM attacks, they do not define one clean-conditioned target semantic across infrared-adapted classification, conditional generation, and binary VQA. Robustness is also coupled to the shared visual backbone. Unsupervised adversarial fine-tuning of a CLIP encoder has been shown to improve robustness for zero-shot classification and for downstream large VLMs that reuse that encoder (Schlarmann et al., 2024). Multimodal defense can additionally exploit one-to-many relationships between images and valid descriptions to reduce dependence on a single paired target (Waseda et al., 2026). Existing evaluations nevertheless typically use image-wide norm-bounded perturbations, optimize a complete adversarial image, or specialize the objective and success metric to one task. InfraPatch complements them with a localized grayscale budget and clean-conditioned, task-specific definitions of the same targeted semantic event across three heterogeneous output spaces. 2.3. Infrared Perception and Visible-to-Infrared Translation Thermal and visible sensing have long been treated as complementary modalities: the KAIST multispectral benchmark, for instance, paired aligned color and thermal video to study pedestrian detection across day and night conditions (Hwang et al., 2015). Modern vision–language systems broaden the relevant output space. CLIP supports open-vocabulary classification through contrastive image–text alignment (Radford et al., 2021); BLIP-2 connects frozen image encoders and language models through a lightweight query transformer (Li et al., 2023); and InstructBLIP adapts this interface to instruction- conditioned tasks (Dai et al., 2023). OpenFlamingo provides an open autoregressive framework for interleaved vision–language generation (Awadalla et al., 2023), whereas the LLaVA line uses visual instruction tuning and a compact vision–language connector for conversational and VQA behavior (Liu et al., 2024). Visible-to-infrared (V2IR) translation provides a way to construct synthetic infrared-style inputs while retaining the semantic content of visible-image benchmarks. DiffV2IR combines a Progressive Learning Module with a Vision-Language Understanding Module to improve semantic awareness and structure preservation during V2IR diffusion (Ran et al., 2025). We use DiffV2IR only as a data-generation tool and study the robustness of downstream infrared-adapted VLMs. InfraPatch focuses on localized targeted perturbations while retaining separate output parsers and valid-sample rules for classification, generation, and binary VQA. 3. Threat Model and Problem Formulation We study a frozen infrared-adapted vision–language model fθτf_θ^τ for task τ∈cls,gen,vqaτ∈\cls,gen,vqa\. For input x∈[0,1]3×H×Wx∈[0,1]^3× H× W, prompt q, and fixed target semantic t (“giraffe” in all experiments), the evaluator applies a task-specific parser hτh_τ to model outputs. The same parser is used for clean and adversarial outputs. 3.1. Digital Threat Model The attacker has white-box access to the frozen checkpoint, image processor, prompt, target, and differentiable task loss. It may choose one square location and one grayscale patch but may not alter model parameters, the prompt, pixels outside the mask, or the output parser. The patch is clipped to [0,1][0,1] and replicated across the three input channels. We record its realized area after preprocessing: (1) ρ⁡(x,mℓ)=∥mℓ∥0HW=s2HW.ρ(x,m_ )= m_ _0HW= s^2HW. The default budget is approximately 5%5\%. This is a per-instance digital attack: we do not claim universal transfer, EOT robustness, sensor re-imaging, or physical thermal materials. The adapted AdvIB, AdvIC, and HCB baselines retain derivative-free search but use the same victim, target, preprocessing, area budget, and final parser as InfraPatch. 3.2. Clean-Conditioned Target Semantics For classification, a sample is valid only when the clean top-1 prediction is correct and non-target. Success requires the target to rank first with a positive margin: (2) c^(xadv)=t,πt(xadv)−maxc≠t⁡πc(xadv)>0.02. c(x_adv)=t, _t(x_adv)- _c≠ t _c(x_adv)>0.02. For generation, a sample is valid when clean text does not contain the target. Lowercasing, punctuation removal, singular/plural normalization, and a fixed COCO synonym map define target hits in adversarial free-form outputs. For binary VQA, the prompt is “Is there a t in the image? Answer only yes or no”. Valid samples have clean output “no”; success is an unambiguous transition to “yes”. Outputs containing both tokens are rejected. 3.3. Evaluation Metric Let viτv_i^τ denote clean validity and aiτ=hτ(fθτ(xi,adv,qi),t)a_i^τ=h_τ(f_θ^τ(x_i,adv,q_i),t). We report one cross-task primary metric: (3) Nvalidτ=∑iviτ,ASRtargetτ=∑iviτaiτNvalidτ.N_valid^τ= _iv_i^τ, _target^τ= _iv_i^τa_i^τN_valid^τ. Clean target hits enter the denominator only and are never counted as attack success. We also record realized area, patch TV, and target margin when meaningful in the corresponding output space. 4. InfraPatch 4.1. Method Overview For an infrared image x, a task prompt q, and a chosen target semantic t, InfraPatch jointly selects a placement mask m and optimizes a single-channel patch p. The adversarial input is (4) xadv=(1−m)⊙x+m⊙rep3⁡(Π[0,1](p)),x_adv=(1-m) x+m _3\! ( _[0,1](p) ), where the single-channel mask is broadcast over the image channels, rep3⁡(⋅)rep_3(·) replicates grayscale values three times, and Π[0,1](⋅) _[0,1](·) projects them onto the valid image range. InfraPatch uses a task-specific differentiable loss for classification, conditional text generation, or binary VQA. It combines this term with total-variation (TV) regularization to form the complete attack objective (5) full(p,m,x,q,t)=ℒtask(p,m,x,q,t)+λTVℒTV(p,m).J_full(p,m;x,q,t)=L_task(p,m;x,q,t)+ _TVL_TV(p,m). For classification, ℒtaskL_task is the weighted combination of target-class, target-similarity, competitor-suppression, and original-class-suppression terms. For conditional generation and VQA, it is the corresponding sequence objective defined below. The same fullJ_full optimizes and ranks proxy candidates, aligning placement selection with the continuation stage. Each candidate starts from the input luminance (6) p(0)=0.299xR+0.587xG+0.114xB.p^(0)=0.299x_R+0.587x_G+0.114x_B. Thus, all locations receive the same deterministic grayscale initialization, while gradients are retained only inside the selected mask. Figure 1 summarizes the end-to-end pipeline: a DiffV2IR-translated input, task prompt, and fixed target semantic enter an early-exit proxy location search; the selected square mask and grayscale patch are then refined under the task-specific objective in Eq. (5) and evaluated with the corresponding discrete success parser. Figure 1. InfraPatch pipeline. Given a clean DiffV2IR-translated image and a fixed target semantic, the method first evaluates candidate patch locations through early-exit proxy search. It then refines the selected grayscale patch with a task-specific objective for classification, conditional generation, or binary VQA, and returns the adversarial output under the corresponding success rule.A clean DiffV2IR-translated image, prompt, and fixed target enter a five-candidate early-exit location search. The selected location and grayscale patch are then refined with the objective and success rule for classification, conditional generation, or binary VQA. 4.2. Target-Aware Patch Placement As shown in Figure 1, the placement and continuation stages share the same complete objective and success parser, so candidate locations are ranked under the same semantic target that governs the final attack. InfraPatch performs an early-exit proxy location search before the continuation stage. It independently samples K=5K=5 legal top-left coordinates with replacement and evaluates them in sampled order. Each candidate receives a fresh copy of Eq. (6) and a newly initialized Adam optimizer. It is then optimized with the complete objective for Tproxy=10T_proxy=10 steps. For candidate k, the proxy score is the lowest complete objective recorded along this short trajectory, (7) sk=min0≤j≤Tproxy⁡full(pk(j),mk,x,q,t).s_k= _0≤ j≤ T_proxyJ_full(p_k^(j),m_k;x,q,t). After the tenth update, the method decodes the task output and applies the discrete success rule from Section 3. When a candidate first satisfies the targeted-success criterion, InfraPatch accepts its tenth-update patch and terminates the search; later candidates and the continuation stage are skipped. If all candidates fail, it selects k⋆=arg⁡mink⁡skk = _ks_k and continues from that candidate’s tenth-update patch for at most 290290 additional steps using a newly initialized Adam optimizer. The score in Eq. (7) may occur before the final proxy iterate, but the implementation deliberately warm-starts continuation from the final iterate rather than a best-loss checkpoint. Task-specific decoding checks can terminate continuation early. Consequently, the selected placement receives at most 300300 updates, while the worst-case per-image cost is 5×10+290=3405× 10+290=340 gradient updates. Throughout the paper, “proxy objective” denotes the complete objective in Eq. (5), not only its target term. 4.3. Task-Adaptive Semantic Objectives 4.3.1. Classification Objective For the CLIP-family models, let z be the normalized image embedding and ece_c the normalized embedding for class c, obtained by averaging the shared prompt-template ensemble. With scaled logit gc=exp⁡(γ)z⊤ecg_c= (γ)z e_c, target class t, current strongest non-target competitor r, and clean predicted class c0c_0, the implemented loss is ℒcls= _cls= λce[−log⁡softmax⁡(g)t]+λcos(1−z⊤et) _ce[- (g)_t]+ _cos(1-z e_t) (8) +λsupmax⁡(z⊤er,0)+λorigsoftmax⁡(g)c0. + _sup (z e_r,0)+ _origsoftmax(g)_c_0. The four default weights are one. The terms respectively increase the target probability, align the image with the target text embedding, suppress the strongest competitor, and reduce persistence of the clean prediction. The optimization objective is differentiable, whereas success is determined by the decoded top-1 label and margin in Eq. (2). 4.3.2. Conditional-Generation Objective Let yt=(yt,1,…,yt,L)y_t=(y_t,1,…,y_t,L) be a short target phrase and let y0y_0 be the clean generated text. The conditional-generation objective uses teacher-forced negative log likelihood (NLL): (9) ℒgen=capB⁡[NLL⁡(yt∣xadv,q)]−λcleancapB[NLL⁡(y0∣xadv,q)],L_gen=cap_B\! [NLL(y_t x_adv,q) ]- _cleancap_B\! [NLL(y_0 x_adv,q) ], where capB⁡(u)=min⁡(u,B)cap_B(u)= (u,B). BLIP-2, BLIP-2-ViT-L, and InstructBLIP use the clean-output suppression term with λclean=0.30 _clean=0.30 and B=12B=12. OpenFlamingo uses the target-NLL term with λclean=0 _clean=0 and B=100B=100. The default target phrase is “a giraffe”. Candidate and final success are always determined from free-running generation, rather than from the teacher-forced loss alone. 4.3.3. Visual Question Answering Objective For LLaVA, the prompt tokens are masked from the labels and the target sequence contains “Yes” followed by the end-of-sequence token. The objective is (10) ℒvqa=−1La∑j=1LalogPθ(at,j∣at,<j,xadv,q).L_vqa=- 1L_a _j=1^L_a P_θ(a_t,j a_t,<j,x_adv,q). LLaVA-1.6 caps this NLL at 1212 for optimization, whereas the implemented LLaVA-1.5 variant uses the uncapped NLL. In both cases, a generated and unambiguous “yes” is required for success; a low teacher-forced NLL alone is not counted. 4.4. Patch Optimization and Constraints Having defined the task-specific terms in ℒtaskL_task, we next specify how the patch parameters are updated under the shared constraints. Throughout optimization, only the patch p is optimized; all model and text-embedding parameters remain frozen. The proxy and continuation stages use Adam with initial learning rate 0.030.03. The continuation stage applies cosine annealing to 5%5\% of that rate. During each forward pass, the rendered patch is clipped to [0,1][0,1], replicated across channels, and composited through m. Gradients outside the mask are set to zero. The total-variation term penalizes only adjacent pixels that are both inside the patch, excluding the patch–background boundary. With horizontal and vertical internal edge sets ℰhE_h and ℰvE_v, it is (11) ℒTV=1|ℰh|∑(u,v)∈ℰh|pu−pv|+1|ℰv|∑(u,v)∈ℰv|pu−pv|,L_TV= 1|E_h| _(u,v) _h|p_u-p_v|+ 1|E_v| _(u,v) _v|p_u-p_v|, with empty denominators clamped to one in code. The default coefficient is λTV=10−3 _TV=10^-3. This term controls local smoothness but is not treated as evidence of physical realizability. 5. Experimental Setup We evaluate ten infrared-adapted variants from four model families on three output formats. Classification uses CLIP, OpenCLIP (Cherti et al., 2023), MetaCLIP (Xu et al., 2024), and EVA-CLIP (Sun et al., 2023); captioning uses BLIP-2, BLIP-2-ViT-L, InstructBLIP, and OpenFlamingo; binary VQA uses LLaVA-1.5 and LLaVA-1.6. All checkpoints start from natural-image pre-training, receive the same LoRA adaptation recipe (Hu et al., 2022) on synthetic infrared-style training data, and remain frozen during attack optimization. Classification adaptation uses an InfoNCE objective (van den Oord et al., 2018), whereas the generation variants use next-token prediction. 5.1. Data and Preprocessing The evaluation set is a fixed 300-image, 30-category MS COCO subset curated by Liu et al. for their cross-task study of VLM robustness to illumination transformations (Lin et al., 2014; Liu et al., 2025b), with ten images per category. We apply DiffV2IR (Ran et al., 2025) to obtain infrared-style digital inputs; these are image-to-image synthesis results, not sensor measurements. The target category “giraffe” is excluded from directory labels. Each task uses its native processor and prompt: “Describe the image” for BLIP-family captioning, “What is in the image?” for OpenFlamingo short answers, and the strict yes/no question defined in Section 3 for LLaVA. Fixed-resolution pipelines use 50×5050× 50 patches at 2242224^2 and LLaVA-1.5 uses 75×7575× 75 at 3362336^2, both realizing approximately 4.98%4.98\% area. LLaVA-1.6 retains its any-resolution path and records the ratio per image. 5.2. Baselines and Controls We adapt AdvIB, AdvIC, and HCB with the same victim, target, clean-valid set, preprocessing, nominal area budget, and discrete success parser. Their defining block, curve, and grid representations and derivative-free DE/PSO searches are preserved; only the detector-disappearance fitness is replaced by a task-targeted fitness. The comparison is area-controlled rather than query-matched. For component analysis we use CLIP, BLIP-2, and LLaVA-1.5 as representatives. Controls compare full InfraPatch with optimized random placement, equal-area unoptimized random grayscale, and equal-area unoptimized center patches. Area settings near 3%3\%, 5%5\%, and 7%7\% and one-factor objective/TV ablations use the same image pool, target, initialization, budget, and parser. Full settings and search budgets are listed in Appendix A and Appendix B. 5.3. Evaluation Protocol We report NvalidN_valid, successful attacks, targeted ASR, and two-sided 95%95\% Wilson intervals from the displayed counts. InfraPatch uses Adam with learning rate 0.030.03, five proxy candidates, ten proxy updates per candidate, and at most 290 continuation updates; it is purely digital and uses no EOT. All comparisons use the clean-conditioned definitions in Section 3. 6. Results and Analysis 6.1. Main Results across Models and Tasks Table 1 gives the primary clean-conditioned outcome before we interpret task-specific patterns. It reports the valid denominator explicitly, so a perfect score is not confused with a large but filtered sample. Table 1. Cross-model targeted attack results under the clean-conditioned protocol. Confidence intervals are two-sided 95%95\% Wilson intervals from the displayed counts. Family Variant Output / task NvalidN_valid Success ASR (%) 95% CI CLIP CLIP ViT-L/14 Classification 165 157 95.15 [90.73, 97.52] CLIP OpenCLIP ViT-B/16 Classification 132 132 100.00 [97.17, 100.00] CLIP MetaCLIP ViT-L/14 Classification 173 158 91.33 [86.19, 94.68] CLIP EVA-CLIP EVA-G/14+ Classification 180 167 92.78 [88.04, 95.73] BLIP BLIP-2 Flan-T5-XL Caption generation 300 258 86.00 [81.62, 89.47] BLIP BLIP-2 ViT-L Caption generation 300 289 96.33 [93.55, 97.94] BLIP InstructBLIP Flan-T5-XL Caption generation 300 262 87.33 [83.09, 90.63] OpenFlamingo ViT-L/14 + MPT-1B Short answer 299 283 94.65 [91.49, 96.68] LLaVA LLaVA-1.5-7B Binary VQA 286 286 100.00 [98.67, 100.00] LLaVA LLaVA-1.6-Mistral-7B Binary VQA 300 300 100.00 [98.74, 100.00] Figure 2 previews representative clean-conditioned successes across classification, binary VQA, and captioning under the same nominal 5%5\% grayscale budget and evaluation rules. Panel (A) shows that a single localized patch can flip top-1 classification on five non-target categories while leaving the surrounding scene otherwise plausible under DiffV2IR translation. Panel (B) demonstrates the narrower VQA event: the clean model answers “no”, and the adversarial input elicits an unambiguous “yes” under the same parser. Panels (C)–(D) further show that the target semantic survives free-form generation, appearing in BLIP-2 captions and OpenFlamingo short answers rather than only in class logits. Figure 2. Qualitative cross-task examples under InfraPatch on DiffV2IR-translated inputs. (A) Image classification (OpenAI CLIP): five categories (bird, dog, bus, stop sign, airplane); clean predictions match the source label, while adversarial inputs are classified as giraffe. (B) Binary visual question answering (LLaVA-1.5): prompt “Is there a giraffe in the image? Answer only yes or no”. (C) Image captioning (BLIP-2): airplane sample with prompt “Describe the image”. (D) Image captioning (OpenFlamingo): bed sample. Green denotes clean outputs; red denotes adversarial outputs or successful target injection.Four-panel figure showing InfraPatch qualitative cross-task examples on image classification, binary visual question answering, and image captioning across DiffV2IR-translated inputs. InfraPatch achieved high clean-conditioned targeted success across all ten IR-VLM variants (Table 1). Targeted ASR ranged from 86.00%86.00\% on BLIP-2 to 100%100\% on OpenCLIP and both LLaVA variants. Among the four classifiers, CLIP, MetaCLIP, and EVA-CLIP reached 95.15%95.15\%, 91.33%91.33\%, and 92.78%92.78\%, respectively; OpenCLIP achieved 132 of 132 successes (100%100\%). Thus, even under the strict clean-conditioned protocol, InfraPatch remains highly effective on contrastive classifiers, with three variants above 91%91\% and one reaching perfect success. The conditional-generation results also generalized beyond one architecture. BLIP-2-ViT-L reached 96.33%96.33\%, OpenFlamingo reached 94.65%94.65\% under the strict 283/299283/299 clean-conditioned count, and InstructBLIP and BLIP-2 reached 87.33%87.33\% and 86.00%86.00\%. Both binary-VQA variants reached 100%100\%, corresponding to 286/286286/286 valid LLaVA-1.5 inputs and 300/300300/300 LLaVA-1.6 inputs. As shown in Figure 2, the target event was not confined to class logits: it also appeared in free-form generation and answer transitions under their task-specific parsers. 6.2. Comparison with Adapted Structured Infrared Attacks InfraPatch achieved the highest targeted ASR on all three representative model–task pairs in Table 2. On CLIP classification, it reached 95.15%95.15\%, versus 9.70%9.70\% for AdvIC-Adapted; the corresponding BLIP-2 rates were 86.00%86.00\% and 7.00%7.00\%, and the LLaVA-1.5 rates were 100.00%100.00\% and 14.69%14.69\%. AdvIC was the strongest adapted baseline on every task, while all three structured attacks remained between 1.33%1.33\% and 14.69%14.69\%. Under the implemented budgets, detector-oriented block, curve, and grid representations therefore transfer weakly to targeted IR-VLM semantics. Because search budgets differ, this comparison evaluates each complete representation–optimizer combination rather than isolating geometry or access cost. Table 2. Clean-conditioned targeted ASR (%) for InfraPatch and adapted structured infrared attacks on one representative model per task. Model / task NvalidN_valid HCB AdvIB AdvIC InfraPatch CLIP / Classification 165 4.24 7.88 9.70 95.15 BLIP-2 / Captioning 300 1.33 3.67 7.00 86.00 LLaVA-1.5 / Binary VQA 286 2.80 6.99 14.69 100.00 6.3. Comparison with Internal Controls Semantic optimization was necessary for the observed target hits. Equal-area unoptimized random and center grayscale patches produced 0%0\% targeted ASR on all three representative variants; this means they did not induce the chosen target, not that model outputs were unchanged. Table 3. Location-selection and optimization controls. Cells report targeted ASR (%); optimized random placement is mean ± standard deviation over three seeds. Setting CLIP BLIP-2 LLaVA-1.5 Full InfraPatch 95.15 86.00 100.00 Random loc., optimized 88.48± 1.60 75.67± 0.88 99.88± 0.20 Random gray, unoptimized 0.00 0.00 0.00 Center gray, unoptimized 0.00 0.00 0.00 A grouped bar chart compares full InfraPatch and optimized random placement on CLIP, BLIP-2, and LLaVA-1.5. Figure 3. Targeted ASR under full InfraPatch and optimized random placement. Error bars denote sample standard deviation across three seeds.A grouped bar chart compares full InfraPatch and optimized random placement on CLIP, BLIP-2, and LLaVA-1.5. Figure 3 compares full InfraPatch with optimized random placement. A single optimized random location was already strong, but proxy search increased mean ASR from 88.48%88.48\% to 95.15%95.15\% on CLIP and from 75.67%75.67\% to 86.00%86.00\% on BLIP-2, gains of 6.676.67 and 10.3310.33 percentage points. On LLaVA-1.5, random placement averaged 99.88%99.88\% and InfraPatch reached 100%100\%, leaving little headroom for location selection. This comparison is not strictly compute matched. The single-location baseline receives at most 300 updates, whereas InfraPatch can spend 5×10+290=3405× 10+290=340 updates when no proxy candidate exits early. The observed difference therefore quantifies the complete proxy-search procedure, including its additional location evaluations, rather than an isolated placement effect at identical gradient cost. 6.4. Ablation Studies The ablations expose task-dependent contributions (Table 4). Table 4. Patch-area and objective ablations. Cells report targeted ASR (%); dashes denote configurations not applicable to a task. Configuration CLIP BLIP-2 LLaVA-1.5 Full, ρ≈5%ρ≈ 5\% 95.15 86.00 100.00 ρ≈3%ρ≈ 3\% 92.73 76.67 100.00 ρ≈7%ρ≈ 7\% 96.97 89.67 100.00 λTV=0 _TV=0 94.55 82.00 100.00 λcos=0 _cos=0 92.73 – – λsup=0 _sup=0 96.36 – – λorig=0 _orig=0 97.58 – – λclean=0 _clean=0 – 86.33 – Removing target-embedding alignment reduced CLIP ASR by 2.422.42 points, from 95.15%95.15\% to 92.73%92.73\%. Removing TV reduced CLIP by only 0.600.60 points but reduced BLIP-2 by 4.004.00 points, while LLaVA-1.5 remained saturated. In contrast, removing rival suppression or clean-class suppression increased CLIP ASR to 96.36%96.36\% and 97.58%97.58\%, and disabling BLIP-2 clean-output suppression changed ASR from 86.00%86.00\% to 86.33%86.33\%. These results do not support claiming that every loss term improves success; under the current target and benchmark, some terms appear redundant for ASR but may affect other properties, such as optimization dynamics or patch smoothness. 6.5. Patch-Area Sensitivity Increasing patch area had the clearest effect on BLIP-2 (Table 4; Figure 4). Moving from approximately 3%3\% to 5%5\% and 7%7\% raised its ASR from 76.67%76.67\% to 86.00%86.00\% and 89.67%89.67\%. CLIP rose more gradually, from 92.73%92.73\% to 95.15%95.15\% and 96.97%96.97\%. The corresponding realized fixed-resolution ratios were 3.03%3.03\%, 4.98%4.98\%, and 6.94%6.94\% for 224×224224× 224 inputs, and 2.98%2.98\%, 4.98%4.98\%, and 7.01%7.01\% for 336×336336× 336 inputs. LLaVA-1.5 remained at 100%100\% throughout, so these data identify a ceiling effect rather than area invariance. Because no perceptual study or thermal-sensor measurement was archived, patch area is treated as a budget variable and not as a direct measure of visual stealth. A line plot shows targeted attack success rate at approximately three, five, and seven percent patch area for CLIP, BLIP-2, and LLaVA-1.5. Figure 4. Targeted ASR as a function of realized patch area for the representative classification, conditional-generation, and binary-VQA variants.A line plot shows targeted attack success rate at approximately three, five, and seven percent patch area for CLIP, BLIP-2, and LLaVA-1.5. 6.6. Robustness Checks and Failure Cases Failure rates differed across families: BLIP-2 failed on 42 of 300 valid samples, InstructBLIP on 38, BLIP-2-ViT-L on 11, MetaCLIP on 15 of 173, and EVA-CLIP on 13 of 180. These counts caution against generalizing perfect OpenCLIP and LLaVA results to all IR-VLMs. Because aggregate summaries lack per-sample traces, we report failures without assigning semantic or spatial causes; clean-valid denominators also differ across classification, captioning, and VQA. 6.7. Cross-Task Interpretation and Boundary Analysis The three task families differ in output freedom. Classification has a closed vocabulary but requires both a top-1 flip and a positive target margin; clean-valid denominators (132–180) reflect clean recognition variation, not attack budget. Within those filtered sets, all four classifiers exceed 91%91\% targeted ASR. A relatively low clean-valid count with high conditional ASR shows that the patch is effective once a sample enters the evaluation set, not that it repairs clean recognition errors. Generation has the most semantic freedom yet uses the strictest clean- conditioned lexical event. BLIP-2, BLIP-2-ViT-L, and InstructBLIP span 86.00%86.00\%–96.33%96.33\%, while OpenFlamingo reaches 283/299283/299 after excluding one clean target hit. Values are based on free-running decoding, not only teacher-forced loss; the lexical parser remains the operational criterion. Binary VQA restricts outputs to unambiguous “yes” or “no”. Both LLaVA variants reach 100%100\% on clean-valid sets, demonstrating reliable no-to-yes transitions under the strict prompt but not open-ended transfer. The contrast between perfect VQA and lower captioning ASR illustrates why one class-flip metric would be insufficient for cross-task robustness. The placement control separates semantic optimization from location selection. Unoptimized grayscale controls produce no target hits, so success requires learned patch values rather than an equal-area occluder alone. Optimized random placement is already effective, but proxy search adds 6.676.67 and 10.3310.33 percentage points on CLIP and BLIP-2. The gain is nearly absent on saturated LLaVA-1.5. With at most 300 versus 5×10+290=3405× 10+290=340 updates, this supports the complete procedure, not a compute-matched location estimate. The ablations provide a complementary explanation. Target-embedding alignment is the only removed CLIP term that clearly lowers ASR, whereas rival and clean- class suppression are redundant for this target and sample pool. TV regulariza- tion matters more for BLIP-2 than CLIP. Area sensitivity follows the same pattern: BLIP-2 gains most from increasing the patch, CLIP changes more gradually, and LLaVA remains saturated from approximately 3%3\% to 7%7\%—architecture- dependent trends, not evidence that larger patches are universally more effective or visually stealthy. Together, the results support a bounded claim. Under one synthetic DiffV2IR distribution, one target semantic, a white-box digital threat model, and the reported clean-conditioned parsers, localized grayscale injection is effective across heterogeneous IR-VLM outputs. The same evidence does not establish physical realizability, black-box transfer, universal patches, or robustness to sensor noise. Those distinctions govern interpretation of the high ASR values and motivate the limitations that follow. The clean-conditioned denominator also shapes cross-model comparison. CLIP clean-valid counts range from 132 to 180 because directory-label and top-1 agreement is required; captioning retains almost all 300 images, except one clean target hit in OpenFlamingo, whereas VQA retains only images whose clean answer is an unambiguous “no”. Consequently, targeted ASR is a conditional vulnerability measure rather than an end-to-end deployment probability; reporting NvalidN_valid keeps comparisons auditable. Parser design separates optimization from success. Classification success is checked from the decoded top-1 label and margin; generation is checked from normalized free-running text; and VQA rejects outputs containing both “yes” and “no”. Teacher-forced target likelihood guides BLIP-family updates, but it cannot by itself establish that the target phrase was generated. The qualitative panels illustrate the same discrete events counted in Table 1. Finally, baseline and ablation evidence are controlled comparisons under the same victim checkpoint, target, processor, valid set, and area budget. Structured baselines remain derivative-free, while InfraPatch uses gradients and can evaluate several proxy locations. This design isolates whether detector-era geometries transfer to a semantic target, not optimizer query cost. Each ablation row changes one term at a time; the resulting differences support mechanism-level discussion, but they should not be read as universal rankings of attack algorithms outside this protocol. 7. Limitations, Responsible Use, and Conclusion 7.1. Limitations The study is bounded to white-box, per-instance digital attacks on one synthetic DiffV2IR-translated COCO set and one target semantic. It does not establish universal or black-box transfer, sensor robustness, or physical realizability; images are not re-imaged after printing or sensor noise. The structured baseline comparison covers one model per task and is neither query nor compute matched. The 300-image, 30-category set does not measure target or long-tail variation, and the shared adaptation recipe cannot separate pre-training from LoRA-induced vulnerability. 7.2. Responsible Use InfraPatch is intended for authorized digital robustness evaluation of infrared multimodal systems, not covert physical deployment. Artifact release should include provenance, fixed parsers, model cards, and mitigations such as patch-aware augmentation and output-consistency checks; safety-critical tests should be isolated and authorized. 7.3. Conclusion This paper introduced InfraPatch, a proxy-guided digital grayscale patch attack for infrared-adapted vision-language models with heterogeneous output formats. Under a clean-conditioned protocol, the attack reached 86.00%86.00\%–100%100\% targeted ASR across ten infrared-adapted variants, and proxy location search improved CLIP and BLIP-2 over optimized random placement. The area and objective ablations showed that vulnerability and component utility vary by architecture, while LLaVA exhibited a pronounced ceiling effect. Within its synthetic, digital, white-box scope, the adapted AdvIB, AdvIC, and HCB attacks achieved only 1.33%1.33\%–14.69%14.69\% ASR on the three representative tasks. This indicates that, under the implemented budgets, detector-era infrared block, curve, and grid structures transfer only weakly to targeted IR-VLM semantics, while localized target-semantic injection remains an IR-VLM robustness concern. References Athalye et al. (2018) A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok Synthesizing robust adversarial examples. In Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 80, p. 284–293. External Links: Link Cited by: §2.1. Awadalla et al. (2023) A. Awadalla, I. Gao, J. Gardner, J. Hessel, Y. Hanafy, W. Zhu, K. Marathe, Y. Bitton, S. Gadre, S. Sagawa, J. Jitsev, S. Kornblith, P. W. Koh, G. Ilharco, M. Wortsman, and L. Schmidt OpenFlamingo: an open-source framework for training large autoregressive vision-language models. External Links: 2308.01390, Link Cited by: §2.3. Bailey et al. (2024) L. Bailey, E. Ong, S. Russell, and S. Emmons Image hijacks: adversarial images can control generative models at runtime. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 2443–2455. External Links: Link Cited by: §2.2. Brown et al. (2017) T. B. Brown, D. Mané, A. Roy, M. Abadi, and J. Gilmer Adversarial patch. External Links: 1712.09665, Link Cited by: §2.1. Carlini and Wagner (2017) N. Carlini and D. Wagner Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy, p. 39–57. External Links: Document Cited by: §2.1. Cherti et al. (2023) M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 2818–2829. External Links: Document, Link Cited by: §5. Dai et al. (2023) W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. C. H. Hoi InstructBLIP: towards general-purpose vision-language models with instruction tuning. In Advances in Neural Information Processing Systems, Vol. 36, p. 49250–49267. External Links: Document, Link Cited by: §2.3. Edwards and Rawat (2020) D. Edwards and D. B. Rawat Study of adversarial machine learning with infrared examples for surveillance applications. Electronics 9 (8), p. 1284. External Links: Document Cited by: §1, §2.1. Eykholt et al. (2018) K. Eykholt, I. Evtimov, E. Fernandes, B. Li, A. Rahmati, C. Xiao, A. Prakash, T. Kohno, and D. Song Robust physical-world attacks on deep learning visual classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 1625–1634. External Links: Document, Link Cited by: §2.1. Gong et al. (2025) Y. Gong, D. Ran, J. Liu, C. Wang, T. Cong, A. Wang, S. Duan, and X. Wang FigStep: jailbreaking large vision-language models via typographic visual prompts. Proceedings of the AAAI Conference on Artificial Intelligence 39 (22), p. 23951–23959. External Links: Document Cited by: §2.2. Goodfellow et al. (2015) I. J. Goodfellow, J. Shlens, and C. Szegedy Explaining and harnessing adversarial examples. In International Conference on Learning Representations, External Links: Link Cited by: §2.1. Hu et al. (2024a) C. Hu, W. Shi, T. Jiang, W. Yao, L. Tian, X. Chen, J. Zhou, and W. Li Adversarial infrared blocks: a multi-view black-box attack to thermal infrared detectors in physical world. Neural Networks 175, p. 106310. External Links: Document Cited by: §2.1. Hu et al. (2024b) C. Hu, W. Shi, W. Yao, T. Jiang, L. Tian, X. Chen, and W. Li Adversarial infrared curves: an attack on infrared pedestrian detectors in the physical world. Neural Networks 178, p. 106459. External Links: Document Cited by: §2.1. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, External Links: Link Cited by: §5. Hwang et al. (2015) S. Hwang, J. Park, N. Kim, Y. Choi, and I. S. Kweon Multispectral pedestrian detection: benchmark dataset and baseline. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 1037–1045. External Links: Document, Link Cited by: §1, §2.3. Li et al. (2023) J. Li, D. Li, S. Savarese, and S. C. H. Hoi BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 202, p. 19730–19742. External Links: Link Cited by: §2.3. Lin et al. (2014) T. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Dollár Microsoft COCO: common objects in context. In Computer Vision – ECCV 2014, p. 740–755. External Links: Document, Link Cited by: §5.1. Liu et al. (2026) D. Liu, B. Chen, and W. Hu Spatial-spectral homogeneous attacks on physical-world large vision-language models. Proceedings of the AAAI Conference on Artificial Intelligence 40 (9), p. 7114–7122. External Links: Document Cited by: §2.2. Liu et al. (2025a) D. Liu, M. Yang, X. Qu, P. Zhou, Y. Cheng, and W. Hu A survey of attacks on large vision–language models: resources, advances, and future trends. IEEE Transactions on Neural Networks and Learning Systems 36 (11), p. 19525–19545. External Links: Document Cited by: §2.2. Liu et al. (2025b) H. Liu, S. Ruan, Y. Huang, S. Zhao, and X. Wei When lighting deceives: exposing vision-language models’ illumination vulnerability through illumination transformation attack. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 10485–10495. External Links: Document Cited by: §1, §5.1. Liu et al. (2024) H. Liu, C. Li, Y. Li, and Y. J. Lee Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 26296–26306. External Links: Document, Link Cited by: §2.3. Lu et al. (2023) D. Lu, Z. Wang, T. Wang, W. Guan, H. Gao, and F. Zheng Set-level guidance attack: boosting adversarial transferability of vision-language pre-training models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 102–111. External Links: Document, Link Cited by: §2.2. Madry et al. (2018) A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, External Links: Link Cited by: §2.1. Qi et al. (2024) X. Qi, K. Huang, A. Panda, P. Henderson, M. Wang, and P. Mittal Visual adversarial examples jailbreak aligned large language models. Proceedings of the AAAI Conference on Artificial Intelligence 38 (19), p. 21527–21536. External Links: Document Cited by: §2.2. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 139, p. 8748–8763. External Links: Link Cited by: §2.3. Ran et al. (2025) L. Ran, L. Wang, G. Wang, P. Wang, and Y. Zhang DiffV2IR: visible-to-infrared diffusion model via vision-language understanding. External Links: 2503.19012, Link Cited by: §2.3, §5.1. Schlarmann et al. (2024) C. Schlarmann, N. D. Singh, F. Croce, and M. Hein Robust CLIP: unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 43685–43704. External Links: Link Cited by: §2.2. Sun et al. (2023) Q. Sun, Y. Fang, L. Wu, X. Wang, and Y. Cao EVA-CLIP: improved training techniques for CLIP at scale. External Links: 2303.15389, Link Cited by: §5. Thys et al. (2019) S. Thys, W. Van Ranst, and T. Goedemé Fooling automated surveillance cameras: adversarial patches to attack person detection. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, p. 49–55. External Links: Document Cited by: §2.1. van den Oord et al. (2018) A. van den Oord, Y. Li, and O. Vinyals Representation learning with contrastive predictive coding. External Links: 1807.03748, Link Cited by: §5. Waseda et al. (2026) F. Waseda, A. Tejero-de-Pablos, and I. Echizen Multimodal adversarial defense for vision-language models by leveraging one-to-many relationships. In 2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), p. 6968–6977. External Links: Document Cited by: §2.2. Wei et al. (2023a) H. Wei, Z. Wang, X. Jia, Y. Zheng, H. Tang, S. Satoh, and Z. Wang HOTCOLD block: fooling thermal infrared detectors with a novel wearable design. Proceedings of the AAAI Conference on Artificial Intelligence 37 (12), p. 15233–15241. External Links: Document Cited by: §2.1. Wei et al. (2023b) X. Wei, Y. Huang, Y. Sun, and J. Yu Unified adversarial patch for cross-modal attacks in the physical world. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 4445–4454. External Links: Document, Link Cited by: §1, §2.1. Wei et al. (2023c) X. Wei, J. Yu, and Y. Huang Physically adversarial infrared patches with learnable shapes and locations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 12334–12342. External Links: Document, Link Cited by: §1, §2.1. Xu et al. (2024) H. Xu, S. Xie, X. E. Tan, P. Huang, R. Howes, V. Sharma, S. Li, G. Ghosh, L. Zettlemoyer, and C. Feichtenhofer Demystifying CLIP data. In International Conference on Learning Representations, External Links: Link Cited by: §5. Xu et al. (2020) K. Xu, G. Zhang, S. Liu, Q. Fan, M. Sun, H. Chen, P. Chen, Y. Wang, and X. Lin Adversarial t-shirt! evading person detectors in a physical world. In Computer Vision – ECCV 2020, p. 665–681. External Links: Document Cited by: §2.1. Zhang et al. (2026) T. Zhang, L. Wang, X. Zhang, Y. Zhang, B. Jia, S. Liang, S. Hu, Q. Fu, A. Liu, and X. Liu Visual adversarial attack on vision-language models for autonomous driving. Machine Intelligence Research 23 (4), p. 823–840. External Links: Document Cited by: §2.2. Zhao et al. (2026) S. Zhao, S. Xiong, M. Yuan, Y. Huang, R. Duan, Q. Guo, J. Chen, H. Duan, and X. Wei Knowledge-guided adversarial training for infrared object detection via thermal radiation modeling. International Journal of Computer Vision 134 (5). External Links: Document Cited by: §2.1. Zhao et al. (2023) Y. Zhao, T. Pang, C. Du, X. Yang, C. Li, N. Cheung, and M. Lin On evaluating adversarial robustness of large vision-language models. In Advances in Neural Information Processing Systems, Vol. 36, p. 54111–54138. External Links: Document, Link Cited by: §1, §2.2. Appendix A Additional Implementation Details Table 5. Prompts, target semantics, and supplementary attack settings. Setting Value Default target “giraffe” BLIP-family prompt “Describe the image” OpenFlamingo prompt “What is in the image?” LLaVA prompt “Is there a giraffe in the image? Answer only yes or no” Generation target phrase “a giraffe” VQA target sequence “Yes” Patch size (2242224^2 / 3362336^2) 50×5050×50 / 75×7575×75 Continuation LR schedule cosine annealing to 5%5\% of initial rate TV coefficient 10−310^-3 Continuation optimizer new Adam (no proxy-state carry-over) Decode cadence BLIP/OpenFlamingo: every 100100 updates; LLaVA: every 5050 Software stack PyTorch 2.0.1+cu117, CUDA 11.7, Transformers 4.46.3 Appendix B Adapted Structured Infrared Baseline Details The AdvIB adaptation uses seven rotatable square cold blocks at grayscale 00. The block side is derived from 0.05HW/70.05HW/7, and differential evolution searches the 21 normalized position-and-angle parameters with population 100, ten generations, mutation factor 0.50.5, crossover probability 0.60.6, seed 42, and early stopping, for at most 1,100 fitness evaluations. AdvIC-Adapted renders two black quadratic Bézier curves specified by 12 normalized control-point parameters. Stroke width is set from their rasterized length and the nominal 5%5\% pixel budget; PSO uses 50 particles, ten iterations, seed 42, inertia 0.90.9, and cognitive/social coefficients 1.6/1.41.6/1.4, for at most 500 evaluations. The instance-level HCB adaptation shares one thresholded 3×33× 3 binary state matrix across four optimized placements. Active cells use grayscale 0.20.2, their size is derived from the nominal budget, and PSO uses 100 particles and three iterations with the same seed and coefficients, for at most 300 evaluations. The derivative-free optimizers minimize continuous task fitness while final ASR uses the same discrete success rules as InfraPatch (Section 3); continuous fitness definitions are provided in the released code. All coordinates lie on the legal preprocessed canvas. Overlap or clipping can make realized union area smaller than the nominal budget. Each run retains the best fitness, parameters, realized mask, query count, decoded output, and success status, including failures.

Instant research discovery

InfraPatch: Cross-Task Targeted Grayscale Patch Attacks on Infrared-Adapted Vision-Language Models

Intelligence

Summary

Entities (9)

Relation Signals (9)

Abstract

Tags

Links

PDF

Full Text

Paper	Published	Area	Tags	Intel	Citations