Paper deep dive
QR-Structured Thermal Triggers for Targeted Semantic Attacks on Infrared Vision-Language Models
Xiang Chen, Yingying Zhao, Chao Li, Jiaju Han, Ben Zhang, Ang Li, Jiahuan Long, Yiwei Wei, Jiujiang Guo, Chengyin Hu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/3/2026, 3:03:15 AM
Summary
The paper introduces QR-Structured Thermal Triggers (QR-STT), a training-free, black-box framework for conducting targeted semantic attacks on Infrared Vision-Language Models (IR-VLMs). QR-STT utilizes QR codes as structured carriers, preserving functional regions while optimizing internal modules with cold, neutral, or hot thermal states to steer image-text alignment toward attacker-specified targets. The method employs a gradient-free, three-stage optimization process to handle mixed discrete-continuous search spaces, demonstrating high attack success rates across multiple CLIP-style encoders and transferring effectively to downstream tasks like image captioning and Visual Question Answering (VQA).
Entities (8)
Relation Signals (7)
QR-STT → achieves → Targeted Semantic Steering
confidence 95% · QR-STT... for targeted semantic steering of IR-VLMs
QR-STT → targets → IR-VLMs
confidence 95% · QR-STT constructs a low-contrast thermal trigger... to steer a frozen IR-VLM toward the target concept
QR-STT → uses → QR Code
confidence 95% · We investigate QR codes as structured attack carriers... QR-STT constructs a QR-structured trigger
QR-STT → evaluatedon → CLIP
confidence 90% · Experiments on multiple CLIP-style encoders show that QR-STT consistently redirects image-text alignment
QR-STT → affects → VQA
confidence 85% · Perturbations optimized for classification also transfer to image captioning and VQA
QR-STT → affects → Image Captioning
confidence 85% · Perturbations optimized for classification also transfer to image captioning and VQA
QR-STT → evaluatedon → LLaVa
confidence 85% · Image Encoder Models: LLaVA-1.5 (7B)... LLaVA-1.6 (7B)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Infrared vision-language models (IR-VLMs) extend thermal perception to open-vocabulary classification, image captioning, and visual question answering. However, their robustness to structured thermal perturbations and the stability of cross-modal semantic alignment remain insufficiently studied. We propose QR-Structured Thermal Triggers (QR-STT), a stealthy, training-free, black-box framework for targeted semantic steering of IR-VLMs. QR-STT preserves the functional regions of a QR pattern while optimizing its internal modules, each of which is assigned a cold, neutral, or hot thermal state. The framework jointly searches module topology and rendering parameters, including position, scale, rotation, intensity, blur, and roundness. A three-stage gradient-free procedure with greedy module-flip refinement efficiently handles the mixed discrete and continuous search space. The objective promotes alignment with an attacker-selected target, suppresses source-class evidence, and regularizes QR structure and visual similarity. Experiments on multiple CLIP-style encoders show that QR-STT consistently redirects image-text alignment toward chosen concepts while maintaining visual stealth. Perturbations optimized for classification also transfer to image captioning and VQA, causing target-consistent semantic drift in generated outputs. These results identify QR-structured thermal patterns as an interpretable attack surface for language-driven infrared perception and highlight the need for robustness evaluation against structured cross-task semantic attacks.
Tags
Links
- Source: https://arxiv.org/abs/2607.29445v1
- Canonical: https://arxiv.org/abs/2607.29445v1
Trouble viewing inline? Open PDF directly →
Full Text
40,674 characters extracted from source content.
Expand or collapse full text
QR-Structured Thermal Triggers for Targeted Semantic Attacks on Infrared Vision–Language Models Xiang Chen, Yingying Zhao, Chao Li, Jiaju Han, Ben Zhang, Ang Li, Jiahuan Long, Yiwei Wei, Jiujiang Guo, Chengyin Hu Abstract Infrared vision–language models (IR-VLMs) extend thermal perception from closed-set recognition to open-vocabulary classification, image captioning, and visual question answering (VQA), enabling infrared inputs to be interpreted through language-aligned representations. However, their robustness to structured thermal perturbations and the stability of cross-modal semantic alignment remain insufficiently studied. In this paper, we introduce QR-Structured Thermal Triggers (QR-STT), a stealthy, training-free black-box framework for targeted semantic steering against IR-VLMs. QR-STT constructs a low-contrast thermal trigger with preserved QR functional regions and optimizable internal modules, where each module is assigned a cold, neutral, or hot thermal state. Within this interpretable module space, QR-STT jointly optimizes module topology and rendering parameters, including placement, scale, rotation, intensity, blur, and roundness, to adapt the trigger to infrared imagery while maintaining visual stealth. A three-stage gradient-free search with greedy module-flip refinement handles the mixed discrete–continuous attack space. The objective promotes target alignment, suppresses source evidence, and regularizes QR structure and visual stealth, enabling source-to-target steering without model training, data poisoning, or gradient access. Experiments across multiple CLIP-style encoders show that QR-STT reliably redirects image–text alignment toward attacker-chosen concepts while maintaining visual stealth. Moreover, adversarial images optimized for classification transfer to captioning and VQA tasks, showing that CLIP-side steering can propagate to generation-level behavior and induce target-consistent semantic drift. These findings reveal QR-structured thermal triggers as an interpretable attack surface for language-driven infrared perception and motivate robustness evaluation against structured cross-task semantic risks. Introduction Figure 1: A QR-structured trigger redirects the image–text alignment of a clean infrared input toward attacker-specified concepts such as dog, car, and bicycle. Vision–language models (VLMs) align visual and linguistic representations through large-scale cross-modal pre-training, supporting open-vocabulary classification, image captioning, and visual question answering (VQA) (Radford et al. 2021; Cherti et al. 2023; Xu et al. 2024; Sun et al. 2023; Liu et al. 2023, 2024; Li et al. 2023; Dai et al. 2023). Recent infrared vision–language models (IR-VLMs) extend this capability to thermal imagery, enabling infrared scenes to be interpreted through language queries rather than fixed detector categories (Jiang et al. 2024; Cao et al. 2025; Zhang et al. 2025b). Such models are promising for nighttime driving, surveillance, search-and-rescue, and adverse-weather perception, where thermal sensing remains informative under poor illumination. This transition from closed-set recognition to language-aligned perception introduces a distinct security risk. Conventional infrared detectors operate over predefined labels, and existing attacks typically aim to suppress detections or induce generic errors. IR-VLMs instead compare visual inputs with a broad set of textual concepts in a shared embedding space. A compact adversarial cue may therefore increase alignment with an attacker-chosen concept without fully removing the original visual evidence. This targeted semantic steering is more consequential than untargeted degradation because the attacker controls the direction of the error. As shown in Figure 1, the same infrared input can be redirected toward concepts such as dog, car, or bicycle. Existing robustness studies do not fully characterize this threat. Most VLM attacks are developed in the visible spectrum and rely on pixel perturbations, transferable adversarial examples, patches, prompt-related vulnerabilities, illumination transformations, or general multimodal attacks (Lu et al. 2023; Yin et al. 2023; Zhao et al. 2023; Zhang et al. 2025a; Xie et al. 2025; Liu et al. 2025; Nie et al. 2026; Hu et al. 2026; Guo et al. 2026). Infrared adversarial research has primarily targeted detectors and trackers using physical patches, stickers, grids, hot/cold blocks, or curve-based thermal patterns (Wei et al. 2023b; Zhu et al. 2024; Tiliwalidi et al. 2025; Wei et al. 2023a; Hu et al. 2024a, b; Jia et al. 2025). These methods mainly pursue missed detections, false positives, or general performance degradation. Whether a compact structured carrier can deliberately manipulate the open-vocabulary semantics of IR-VLMs remains underexplored. We investigate QR codes as structured attack carriers in the digital infrared image domain. A QR code naturally separates fixed functional regions from editable internal modules. Finder patterns, timing patterns, format-related regions, and the quiet zone preserve its global organization, while the remaining modules define a discrete topology that can be optimized for target-specific steering. Unlike unconstrained patches, this representation explicitly separates the preserved scaffold from the attack variables, providing a structured and interpretable space for encoding adversarial cues. A direct high-contrast QR overlay would be visually conspicuous and methodologically similar to ordinary patch insertion. We therefore assign each editable module one of three relative infrared intensity states: cold, neutral, or hot, corresponding to local intensity decreases, unchanged responses, and local intensity increases. The trigger is further controlled by trigger scale, perturbation intensity, active-module ratio, and structural similarity. This yields a low-contrast QR-structured perturbation while retaining the native QR organization. Our central question is whether the functional scaffold and editable module topology of a QR code can jointly support targeted semantic steering under explicit visual-distortion constraints. Based on this formulation, we propose QR-Structured Thermal Triggers (QR-STT), a training-free black-box targeted attack for IR-VLMs. Given a clean infrared image and an attacker-specified target label, QR-STT jointly optimizes trigger placement, scale, rotation, rendering parameters, and internal module states. Its objective promotes target alignment, suppresses source evidence, regularizes QR topology, and limits visual distortion. To address the resulting mixed discrete–continuous search space, we develop a progressive gradient-free strategy comprising coarse zone search, module-topology search, infrared rendering refinement, and greedy module-flip refinement. We evaluate QR-STT on targeted zero-shot classification across multiple CLIP-style encoders and further assess transfer to image captioning and VQA. The downstream generative models are excluded from attack optimization. Consequently, target-consistent changes in their outputs indicate that the induced alignment shift extends beyond a task-specific classifier and propagates to generation-level behavior. Our contributions are summarized as follows: • We identify attacker-directed semantic steering as a distinct robustness risk for IR-VLMs and introduce a QR-structured attack framework that targets open-vocabulary image–text alignment rather than generic recognition failure. • We formulate QR trigger generation as a constrained mixed discrete–continuous optimization problem that separates fixed functional regions from editable cold, neutral, and hot modules, and develop a progressive gradient-free solver for joint topology and rendering optimization. • We conduct comprehensive evaluations across multiple CLIP-style encoders, image-captioning models, and VQA models. Comparisons with structured infrared baselines and detailed ablations demonstrate the effectiveness, cross-task transferability, and interpretability of QR-STT. Related Work Infrared vision–language models. Vision–language models align images and text through large-scale contrastive or generative pretraining (Radford et al. 2021; Li et al. 2023; Dai et al. 2023; Zhai et al. 2023). Recent infrared and thermal VLM studies adapt these ideas to low-texture thermal imagery, where discriminative cues depend more on heat radiation, object silhouette, and sensor statistics than RGB color or fine-grained texture (Jiang et al. 2024; Cao et al. 2025; Moshtaghi et al. 2025; Zhang et al. 2025b). These models make infrared perception more language-accessible, but also introduce open-vocabulary semantic attack surfaces. Figure 2: Overview of QR-STT. Given a clean infrared image and an attacker-specified target label, QR-STT constructs a QR-structured thermal trigger with fixed functional regions and editable cold/neutral/hot modules. The attack progressively optimizes the trigger through zone search, topology search, thermal rendering refinement, and greedy module flipping, steering a frozen IR-VLM toward the target concept and transferring the induced semantic drift to classification, captioning, and VQA. Robustness of Vision–Language Alignment under Adversarial Attacks. Multimodal adversarial attacks often exploit the shared image–text representation of CLIP-style models, either by perturbing the image, guiding transfer across models, or inducing downstream caption/VQA errors (Lu et al. 2023; Yin et al. 2023; Zhao et al. 2023; Zhang et al. 2025a; Xie et al. 2025; Liu et al. 2025; Nie et al. 2026; Hu et al. 2026; Guo et al. 2026). Because instruction-tuned VLMs frequently inherit visual features from CLIP-like encoders (Liu et al. 2023, 2024; Awadalla et al. 2023), a shift in image–text alignment can propagate to language outputs. Our work focuses on this transfer in infrared scenarios and asks whether a stealthy symbolic thermal carrier can steer the shared representation toward a specified target. Structured infrared perturbations. Infrared attacks often constrain perturbations to physically meaningful or low-frequency patterns, including adversarial infrared patches, car stickers, wearable hot/cold blocks, grid patterns, and spline or curve carriers (Brown et al. 2017; Wei et al. 2023b; Zhu et al. 2024; Wei et al. 2023a; Hu et al. 2024a; Tiliwalidi et al. 2025; Hu et al. 2024b; Jia et al. 2025). These studies mainly target detectors, whereas QR-STT targets open-vocabulary IR-VLM semantics. We adapt representative structured infrared carriers as baselines under the same source-to-target protocol. Symbolic visual carriers and QR structure. Standard QR codes have inherent fixed functional regions and editable data modules, making them a highly useful symbolic layout for studying structured visual perturbations (International Organization for Standardization 2015). However, directly pasting a plain QR-like patch is visually obvious and methodologically close to a generic patch attack (Brown et al. 2017). QR-STT instead fully preserves QR functional regions and optimizes only the internal thermal modules under strict stealth constraints, testing whether symbolic module topology itself can serve as an interpretable attack space. Method Figure 2 illustrates the overall pipeline of QR-STT. Given an infrared image and an attacker-specified target label, QR-STT constructs a QR-structured trigger and jointly optimizes its geometry, editable module topology, and rendering parameters to steer a frozen CLIP-style IR-VLM toward the target concept. We first formulate the targeted attack, then introduce the constrained QR representation, and finally describe the progressive black-box optimization strategy. Problem Setup Let xi∈[0,1]H×Wx_i∈[0,1]^H× W denote an infrared image with source label sis_i, and let t∈t be an attacker-specified target label, where t≠sit≠ s_i. We consider a frozen CLIP-style IR-VLM with an image encoder fI(⋅)f_I(·) and a text encoder fT(⋅)f_T(·) (Radford et al. 2021; Cherti et al. 2023; Xu et al. 2024; Sun et al. 2023). The attacker can query model outputs but has no access to model parameters or gradients. For each class c∈c , we construct a normalized text prototype from a prompt set c=Tckk=1KT_c=\T_c^k\_k=1^K: zc=norm(1K∑k=1KfT(Tck)).z_c=norm ( 1K _k=1^Kf_T(T_c^k) ). (1) The image–text similarity score is: Sc(x)=⟨norm(fI(x)),zc⟩.S_c(x)= (f_I(x)),z_c . (2) Here norm(⋅)norm(·) denotes ℓ2 _2 normalization and ⟨⋅,⋅⟩ ·,· denotes cosine similarity. The attack succeeds when the target becomes the top-ranked class: argmaxc∈Sc(xiadv)=t. _c S_c(x_i^adv)=t. (3) The adversarial image is generated by a QR-structured rendering operator: xiadv=AQR(xi;θ),x_i^adv=A_QR(x_i;θ), (4) where θ=g,h,Meditθ=\g,h,M_edit\. The geometry g=(cx,cy,s,r)g=(c_x,c_y,s,r) contains the trigger center, scale, and rotation, while the rendering parameters h=(α,σ,ρ)h=(α,σ,ρ) control intensity, blur radius, and module roundness. MeditM_edit denotes the states of the editable QR modules. QR-Structured Thermal Trigger A direct high-contrast QR overlay is visually conspicuous and methodologically similar to ordinary patch insertion. Instead, QR-STT exploits the native organization of a QR code to define a constrained module space. We partition the QR grid into fixed functional regions ℱF and editable internal regions ℰE. The fixed regions contain finder patterns, timing-like modules, format-like regions, and a quiet-zone-like boundary, preserving the global QR scaffold. The editable regions provide the degrees of freedom for target-specific optimization. Let M0M^0 denote the original QR template and M the optimized module map. We impose the hard constraint: Muv=Muv0,(u,v)∈ℱ,Muv∈−1,0,+1,(u,v)∈ℰ. casesM_uv=M^0_uv,&(u,v) ,\\ M_uv∈\-1,0,+1\,&(u,v) . cases (5) The values −1-1, 0, and +1+1 represent cold, neutral, and hot states, corresponding to local intensity decreases, unchanged responses, and local intensity increases in the normalized infrared image domain. Directly searching over ternary states leads to a high-dimensional discrete problem. We therefore associate each editable module with a relaxed variable quvq_uv and decode it using two thresholds: Muv=−1,quv<τ−,0,τ−≤quv≤τ+,+1,quv>τ+.M_uv= cases-1,&q_uv< _-,\\ 0,& _-≤ q_uv≤ _+,\\ +1,&q_uv> _+. cases (6) Here τ− _- and τ+ _+ denote the lower and upper decoding thresholds. This relaxation enables continuous gradient-free search while preserving a discrete final topology. Given the decoded module map, QR-STT constructs a signed intensity response and embeds it into the infrared image: xiadv=clip(xi+(R(Medit;h);g),0,1).x_i^adv=clip (x_i+W (R(M_edit;h);g ),0,1 ). (7) Here R(⋅)R(·) converts the module states into a signed infrared intensity map, and (⋅)W(·) applies translation, scaling, and rotation. Hot modules increase local intensity, cold modules decrease it, and neutral modules remain inactive. Blur and rounded module boundaries suppress sharp digital artifacts and improve consistency with infrared image characteristics. Source-to-Target Objective We formulate trigger generation using three objectives with distinct roles: ℒQR−STT=Lsem+λtopLtop+λvisLvis.L_QR-STT=L_sem+ _topL_top+ _visL_vis. (8) Here LsemL_sem drives targeted semantic steering, LtopL_top regularizes the editable module topology, and LvisL_vis constrains image-level distortion. The three terms address complementary requirements and avoid redundant optimization objectives. Targeted semantic steering. A targeted attack requires the target score to exceed both the source score and all remaining competing classes. We therefore define: Lsem L_sem =[maxc∈∖t,siSc(xiadv)−St(xiadv)+m]+ = [ _c \t,s_i\S_c(x_i^adv)-S_t(x_i^adv)+m ]_+ (9) +β[Ssi(xiadv)−St(xiadv)+ms]+, +β [S_s_i(x_i^adv)-S_t(x_i^adv)+m_s ]_+, where [u]+=max(u,0)[u]_+= (u,0). The first term separates the target from the strongest non-source competitor with margin m, while the second explicitly separates the target from the source concept with margin msm_s. The coefficient β controls the strength of source suppression. This unified objective directly encodes the conditions required for source-to-target steering without introducing overlapping classification losses. Module-topology regularization. The QR functional regions are already preserved by the hard constraint in Eq. (5). We therefore regularize only the editable topology: Ltop=Lact+ηtvLtv.L_top=L_act+ _tvL_tv. (10) The active-module term controls the proportion of cold and hot modules: Lact=|1|ℰ|∑(u,v)∈ℰ|Muv|−ρ0|,L_act= | 1|E| _(u,v) |M_uv|- _0 |, (11) where ρ0 _0 denotes the desired active-module ratio. Since |Muv|=1|M_uv|=1 for cold or hot modules and |Muv|=0|M_uv|=0 for neutral modules, this term prevents both insufficiently expressive sparse patterns and visually dominant dense patterns. The spatial regularizer is: Ltv=1|ℰ|∑(u,v)∈ℰ(|Mu+1,v−Muv|+|Mu,v+1−Muv|),L_tv= 1|E| _(u,v) (|M_u+1,v-M_uv|+|M_u,v+1-M_uv| ), (12) where invalid boundary terms are omitted. This term discourages isolated state changes and fragmented checkerboard patterns while retaining sufficient flexibility for target-specific topology optimization. Visual-distortion control. We constrain image-level distortion using the mean absolute perturbation and structural similarity: Lvis L_vis =[D(xi,xiadv)−δmax]+ = [D(x_i,x_i^adv)- _ ]_+ (13) +μs[γmin−SSIM(xi,xiadv)]+, + _s [ _ -SSIM(x_i,x_i^adv) ]_+, where D(xi,xiadv)=1HW‖xiadv−xi‖1D(x_i,x_i^adv)= 1HW \|x_i^adv-x_i \|_1 (14) denotes the mean absolute perturbation. δmax _ specifies the maximum allowed perturbation and γmin _ the minimum structural similarity (SSIM) (Wang et al. 2004). The hinge form penalizes only violations of the corresponding visual constraints. Trigger coverage is not penalized separately because it is already controlled by the trigger scale and the active-module ratio. Table 1: Zero-shot classification results. We report attack success rate (%) for each target category and the macro-average over all model–target pairs. Method OpenCLIP ViT-B/16 Meta-CLIP ViT-L/14 EVA-CLIP ViT-G/14 OpenAI CLIP ViT-L/14 Avg. Bicycle Car Dog Bicycle Car Dog Bicycle Car Dog Bicycle Car Dog AdvICRS 4.97 9.05 4.13 3.60 8.30 6.40 4.52 3.15 2.31 2.80 5.60 4.40 4.94 HCB 3.30 10.67 11.50 5.65 5.20 13.60 7.52 6.80 7.40 9.35 4.40 17.60 8.58 AdvGrid 8.13 14.20 11.16 15.30 11.52 19.20 10.40 17.10 15.80 6.80 11.20 25.60 13.87 QR-STT 42.85 24.79 38.50 35.91 31.40 30.27 28.40 35.20 34.80 33.94 42.95 32.34 34.28 Figure 3: Qualitative targeted-classification examples of QR-STT. Clean infrared samples and their original predictions are shown above the corresponding adversarial samples. Green and red labels denote clean and attacker-specified target predictions, respectively. Table 2: Image-captioning robustness measured by clean-reference/source-preservation rate (%). Lower values indicate larger semantic deviations. Image Encoder Models AdvICRS HCB AdvGrid QR-STT OpenAI CLIP ViT-L/14 LLaVA-1.5 (7B) 68.97 63.64 55.96 51.72 LLaVA-1.6 (7B) 63.52 50.00 35.78 29.56 OpenFlamingo (3B) 65.45 57.32 52.40 50.74 BLIP-2 FlanT5XL ViT-L (3.4B) 72.10 69.46 67.89 67.00 EVA-CLIP ViT-G/14 BLIP-2 FlanT5XL (4.1B) 61.24 58.79 47.56 41.65 InstructBLIP FlanT5XL (4.1B) 67.58 62.36 54.30 52.40 Figure 4: Qualitative transfer examples of QR-STT on image captioning and VQA. (A) Adversarial samples induce target-consistent changes in generated captions. (B) Target-agnostic VQA answers are redirected toward attacker-specified concepts. (C) GPT-5-based evaluation compares clean and adversarial outputs against clean-reference semantics. Progressive Black-Box Optimization The search space of QR-STT is mixed discrete–continuous: trigger geometry and rendering parameters are continuous, whereas the module states are discrete. Jointly optimizing all variables creates a high-dimensional and unstable black-box problem. We therefore adopt a progressive strategy that proceeds from coarse spatial configurations to fine-grained module topology. At stage k∈1,2,3k∈\1,2,3\, QR-STT optimizes a stage-specific variable set Ωk _k: θ(k)⋆=argminθ∈ΩkℒQR−STT(AQR(xi;θ),si,t).θ^(k) = _θ∈ _kL_QR-STT (A_QR(x_i;θ),s_i,t ). (15) The best solution from each stage initializes the next stage. Stage 1: coarse zone search. We partition the editable region into coarse spatial zones, with all modules in each zone sharing a relaxed state. This stage optimizes trigger position, scale, rotation, intensity, and zone-level cold/hot tendencies to identify a promising placement and coarse topology. Stage 2: module topology search. Starting from the best coarse solution, each editable module is assigned an independent relaxed state. The optimizer then searches for a fine-grained target-specific topology, which provides the main semantic capacity of the trigger. Stage 3: rendering refinement. Given the topology obtained in Stage 2, we refine intensity, blur, roundness, and small geometric corrections. This stage suppresses sharp artifacts and improves the trade-off between target alignment and visual similarity. Greedy module-flip refinement. Because relaxed optimization may not identify the best ternary configuration, we perform a final local search over the decoded module states. At iteration ℓ , a candidate M~(ℓ) M^( ) is generated by changing a small subset of editable modules among cold, neutral, and hot states. The candidate is accepted only if it decreases the complete objective: M(ℓ+1)=M~(ℓ),Δℒ<0,M(ℓ),otherwise,M^( +1)= cases M^( ),& <0,\\ M^( ),&otherwise, cases (16) where Δℒ=ℒQR−STT(M~(ℓ))−ℒQR−STT(M(ℓ)). =L_QR-STT ( M^( ) )-L_QR-STT (M^( ) ). (17) The refinement terminates when the query budget is exhausted or no further improvement is found. Overall, the progressive strategy separates three coupled decisions: where the trigger is placed, which module topology encodes the target signal, and how the trigger is rendered. This decomposition reduces early-stage search complexity and stabilizes fine-grained topology optimization. Table 3: VQA robustness. We report clean-reference/source-preservation rates (%); lower values indicate larger answer deviations. Methods are ordered from smaller to larger overall deviation. Image Encoder Models AdvICRS HCB AdvGrid QR-STT OpenAI CLIP ViT-L/14 LLaVA-1.5 (7B) 68.75 58.08 58.41 45.65 LLaVA-1.6 (7B) 64.20 61.62 51.07 37.27 OpenFlamingo (3B) 62.80 56.40 50.25 47.80 BLIP-2 FlanT5XL ViT-L (3.4B) 83.33 75.20 67.89 66.17 EVA-CLIP ViT-G/14 BLIP-2 FlanT5XL (4.1B) 57.86 54.72 43.95 35.80 InstructBLIP FlanT5XL (4.1B) 63.20 58.75 49.40 41.60 Figure 5: Hyperparameter sensitivity of QR-STT across trigger scale, active ratio, thermal intensity, and optimization budget, measured by ASR. Figure 6: Component ablation of QR-STT. (a) Targeted ASR for the full method and its ablated variants. Removing module-topology search causes the largest degradation in attack effectiveness. (b) SSIM for the same variants. Disabling stealth rendering produces the largest reduction in visual similarity. Overall, the full QR-STT achieves the best balance between targeted ASR and SSIM. Experiments Experimental Setup Data and models. Following Jiang et al. (2024), all deployed VLMs are infrared-adapted for thermal inputs. We use a 30-class infrared test set (10 images per class) and only keep samples correctly predicted by frozen CLIP encoders. Each sample is optimized for three targets: dog, car, and bicycle. We adopt four CLIP backbones: OpenCLIP ViT-B/16 (Cherti et al. 2023), Meta-CLIP ViT-L/14 (Xu et al. 2024), EVA-CLIP ViT-G/14 (Sun et al. 2023), OpenAI CLIP ViT-L/14 (Radford et al. 2021). For captioning and VQA transfer, we utilize six infrared-tuned generative IR-VLMs (Liu et al. 2023, 2024; Awadalla et al. 2023; Li et al. 2023; Dai et al. 2023). Attack protocol and evaluation. We run black-box zero-shot targeted attacks. Attack success is defined as the top-1 CLIP prediction of xiadvx_i^adv matching target t. The main evaluation metrics are targeted attack success rate (ASR) and SSIM (Wang et al. 2004). Adversarial infrared images are directly tested on captioning/VQA without extra fine-tuning. Consistent target-biased outputs demonstrate that CLIP-side embedding shifts transfer to generation tasks. We measure semantic drift using GPT-5 (OpenAI 2025) as the evaluator under the LLM-as-a-judge protocol (Zheng et al. 2023). Implementation Details. QR-STT performs three-stage gradient-free black-box optimization over structured thermal QR modules, including zone exploration, module refinement, and stealth rendering. We use a version-1 QR carrier (21×2121× 21 modules) with fixed functional regions and optimize only editable modules. Under the default fast setting, the three stages use population sizes of 40/50/60 and 12/16/20 generations. Thermal carriers are rendered at 224×224224× 224 resolution with rounded module kernels, while the objective jointly considers target alignment and visual stealth. All experiments are conducted on NVIDIA RTX 4090 GPUs. Baselines. We adapt three representative structured infrared attacks into the same source-to-target setting. AdvGrid uses grid-based thermal perturbations (Tiliwalidi et al. 2025), AdvICRS employs spline-based carriers (Jia et al. 2025), and HCB adopts hot/cold block patterns (Wei et al. 2023a). All baselines follow the same clean filtering, target labels, query constraints, and downstream evaluation protocol for fair comparison. Targeted Classification Evaluation Table 1 reports targeted ASR across four CLIP-style encoders and three target labels. QR-STT achieves the best performance on all 12 backbone–target pairs, with a macro-average ASR of 34.28%, substantially outperforming AdvGrid, HCB, and AdvICRS. Under the same clean-correct filtering, target labels, and black-box protocol, these results indicate that the optimized QR-module topology provides a more effective targeted steering space than grid-, curve-, or block-based thermal carriers. Attackability varies across backbones and targets: EVA-CLIP ViT-G/14 is comparatively more robust, whereas OpenCLIP ViT-B/16 and OpenAI CLIP ViT-L/14 are more vulnerable. Figure 3 further shows that diverse clean infrared inputs can be redirected toward attacker-specified concepts such as dog, bicycle, and car, consistent with the quantitative results. Image Captioning and VQA Robustness We further evaluate whether adversarial images optimized only for targeted zero-shot classification transfer to downstream generation tasks without task-specific optimization. For image captioning, Table 2 reports clean-reference/source-preservation rates, where lower values indicate greater semantic deviation. QR-STT achieves the lowest rates across all evaluated models, showing that QR-structured triggers affect both classification and generated descriptions. Figure 4(A) presents representative target-consistent caption shifts while preserving the overall infrared scene. Similar results are observed for VQA in Table 3, where QR-STT again yields the lowest preservation rates. Since the questions are target-agnostic and the downstream models are excluded from attack optimization, these findings indicate that the induced embedding shift transfers beyond classification to generation-level reasoning. Figures 4(B) and (C) show representative target-biased answers and the GPT-5-based evaluation protocol, respectively. Ablation Study Hyperparameter Sensitivity. We analyze four key hyperparameters, including trigger scale, active-module ratio, thermal intensity, and query budget, to assess their effects on targeted attack performance. As shown in Figure 5, increasing the trigger scale provides greater editable capacity and improves ASR, although the gain gradually saturates at larger scales. A similar trend is observed for the active-module ratio, where moderate activation enables stronger semantic steering than overly sparse configurations. Higher thermal intensity also improves ASR, with diminishing gains at stronger settings. For the query-budget analysis, we vary the total number of black-box evaluations across the complete optimization pipeline. QR-STT achieves substantial gains under small budgets, while the improvement gradually saturates as additional evaluations mainly refine the solution. Component Ablation of QR-STT. We remove one component at a time while keeping the remaining attack pipeline unchanged, as shown in Figure 6. Removing fixed QR regions evaluates the contribution of the structural scaffold, disabling module-topology search tests the necessity of target-specific cold, neutral, and hot layouts, removing greedy module flipping measures the benefit of local discrete refinement, and disabling stealth rendering removes blur, low contrast, and rounded module boundaries. Figure 6(a) shows that module-topology search contributes most to targeted attack effectiveness, while Figure 6(b) shows that stealth rendering is most important for preserving visual similarity. The full QR-STT achieves the best overall ASR–SSIM trade-off. Conclusion and Discussion We propose QR-STT, a training-free black-box framework for targeted semantic steering of infrared VLMs. By preserving the fixed QR scaffold and optimizing editable thermal modules through progressive mixed discrete–continuous search, QR-STT achieves strong attack effectiveness and cross-task transferability across classification, captioning, and VQA. Ablation studies validate the roles of topology search, discrete refinement, and stealth-oriented rendering, establishing QR-structured perturbations as an effective and interpretable attack space for infrared VLMs. References A. Awadalla, I. Gao, J. Gardner, J. Hessel, Y. Hanafy, W. Zhu, K. Marathe, Y. Bitton, S. Gadre, S. Sagawa, J. Jitsev, S. Kornblith, P. W. Koh, G. Ilharco, M. Wortsman, and L. Schmidt (2023) OpenFlamingo: an open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390. Cited by: Related Work, Experimental Setup. T. B. Brown, D. Mané, A. Roy, M. Abadi, and J. Gilmer (2017) Adversarial patch. arXiv preprint arXiv:1712.09665. Cited by: Related Work, Related Work. Z. Cao, J. Zhang, and R. Zhang (2025) IRGPT: understanding real-world infrared image with bi-cross-modal curriculum on large-scale benchmark. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 166–176. Cited by: Introduction, Related Work. M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev (2023) Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 2818–2829. Cited by: Introduction, Problem Setup, Experimental Setup. W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi (2023) InstructBLIP: towards general-purpose vision-language models with instruction tuning. In Advances in Neural Information Processing Systems, Vol. 36, p. 49250–49267. Cited by: Introduction, Related Work, Experimental Setup. Q. Guo, X. Jia, S. Pang, S. Qin, L. Wang, J. Jia, Y. Liu, and Q. Guo (2026) PhysPatch: a physically realizable and transferable adversarial patch attack for multimodal large language models-based autonomous driving systems. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 4412–4420. External Links: Document Cited by: Introduction, Related Work. C. Hu, W. Shi, T. Jiang, W. Yao, L. Tian, and X. Chen (2024a) Adversarial infrared blocks: a multi-view black-box attack to thermal infrared detectors in physical world. Neural Networks 175, p. 106310. Cited by: Introduction, Related Work. C. Hu, W. Shi, W. Yao, T. Jiang, L. Tian, X. Chen, and W. Li (2024b) Adversarial infrared curves: an attack on infrared pedestrian detectors in the physical world. Neural Networks 178, p. 106459. Cited by: Introduction, Related Work. K. Hu, W. Yu, L. Zhang, A. Robey, A. Zou, H. Hu, C. Xu, and M. Fredrikson (2026) Omni-Attack: adversarial attacks on open-ended VQA in black-box multimodal LLMs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 42341–42351. Cited by: Introduction, Related Work. International Organization for Standardization (2015) ISO/IEC 18004:2015: information technology—automatic identification and data capture techniques—QR Code bar code symbology specification. Note: International Standard Cited by: Related Work. Z. Jia, C. Hu, J. Zhang, K. Tiliwalidi, L. Tian, X. Li, and X. Kang (2025) Adversarial infrared catmull-rom spline: a black-box attack on infrared pedestrian detectors in the physical world. Information Sciences 717, p. 122263. Cited by: Introduction, Related Work, Experimental Setup. S. Jiang, Z. Chen, J. Liang, Y. Zhao, M. Liu, and B. Qin (2024) Infrared-LLaVA: enhancing understanding of infrared images in multi-modal large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 8573–8591. Cited by: Introduction, Related Work, Experimental Setup. J. Li, D. Li, S. Savarese, and S. Hoi (2023) BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning, p. 19730–19742. Cited by: Introduction, Related Work, Experimental Setup. H. Liu, S. Ruan, Y. Huang, S. Zhao, and X. Wei (2025) When lighting deceives: exposing vision-language models’ illumination vulnerability through illumination transformation attack. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 10485–10495. Cited by: Introduction, Related Work. H. Liu, C. Li, Y. Li, and Y. J. Lee (2024) Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: Introduction, Related Work, Experimental Setup. H. Liu, C. Li, Q. Wu, and Y. J. Lee (2023) Visual instruction tuning. In Advances in Neural Information Processing Systems, Vol. 36, p. 34892–34916. Cited by: Introduction, Related Work, Experimental Setup. D. Lu, Z. Wang, T. Wang, W. Guan, H. Gao, and F. Zheng (2023) Set-level guidance attack: boosting adversarial transferability of vision-language pre-training models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 102–111. Cited by: Introduction, Related Work. S. Moshtaghi, S. H. Khajavi, and J. Pajarinen (2025) RGB-Th-Bench: a dense benchmark for visual-thermal understanding of vision language models. arXiv preprint arXiv:2503.19654. Cited by: Related Work. S. Nie, J. Zhang, J. Yan, S. Shan, and X. Chen (2026) V-Attack: targeting disentangled value features for controllable adversarial attacks on LVLMs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 42257–42267. Cited by: Introduction, Related Work. OpenAI (2025) GPT-5 System Card. External Links: Link Cited by: Experimental Setup. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever (2021) Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, p. 8748–8763. Cited by: Introduction, Related Work, Problem Setup, Experimental Setup. Q. Sun, Y. Fang, L. Wu, X. Wang, and Y. Cao (2023) EVA-clip: improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389. Cited by: Introduction, Problem Setup, Experimental Setup. K. Tiliwalidi, C. Hu, G. Lu, M. Jia, and W. Shi (2025) AdvGrid: a multi-view black-box attack on infrared pedestrian detectors in the physical world. Applied Soft Computing 174, p. 112981. Cited by: Introduction, Related Work, Experimental Setup. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli (2004) Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13 (4), p. 600–612. Cited by: Visual-distortion control., Experimental Setup. H. Wei, Z. Wang, X. Jia, Y. Zheng, H. Tang, S. Satoh, and Z. Wang (2023a) HOTCOLD block: fooling thermal infrared detectors with a novel wearable design. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, p. 15233–15241. Cited by: Introduction, Related Work, Experimental Setup. X. Wei, J. Yu, and Y. Huang (2023b) Physically adversarial infrared patches with learnable shapes and locations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 12334–12342. Cited by: Introduction, Related Work. P. Xie, Y. Bie, J. Mao, Y. Song, Y. Wang, H. Chen, and K. Chen (2025) Chain of attack: on the robustness of vision-language models against transfer-based adversarial attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 14679–14689. Cited by: Introduction, Related Work. H. Xu, S. Xie, X. E. Tan, P. Huang, R. Howes, V. Sharma, S. Li, G. Ghosh, L. Zettlemoyer, and C. Feichtenhofer (2024) Demystifying clip data. In The Twelfth International Conference on Learning Representations, Cited by: Introduction, Problem Setup, Experimental Setup. Z. Yin, M. Ye, T. Zhang, T. Du, J. Zhu, H. Liu, J. Chen, T. Wang, and F. Ma (2023) VLATTACK: multimodal adversarial attacks on vision-language tasks via pretrained models. In Advances in Neural Information Processing Systems, Vol. 36. Cited by: Introduction, Related Work. X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer (2023) Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 11975–11986. Cited by: Related Work. J. Zhang, J. Ye, X. Ma, Y. Li, Y. Yang, Y. Chen, J. Sang, and D. Yeung (2025a) AnyAttack: towards large-scale self-supervised adversarial attacks on vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 19900–19909. Cited by: Introduction, Related Work. T. Zhang, Y. Hong, Y. Xia, K. Ding, Z. Zhang, Y. Wang, S. Xiang, and C. Pan (2025b) IF-Bench: benchmarking and enhancing MLLMs for infrared images with generative visual prompting. arXiv preprint arXiv:2512.09663. Cited by: Introduction, Related Work. Y. Zhao, T. Pang, C. Du, X. Yang, C. Li, N. Cheung, and M. Lin (2023) On evaluating adversarial robustness of large vision-language models. In Advances in Neural Information Processing Systems, Vol. 36. Cited by: Introduction, Related Work. L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica (2023) Judging LLM-as-a-judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems, Vol. 36. Cited by: Experimental Setup. X. Zhu, Y. Liu, Z. Hu, J. Li, and X. Hu (2024) Infrared adversarial car stickers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 24284–24293. Cited by: Introduction, Related Work.