Paper deep dive
ProjLens: Unveiling the Role of Projectors in Multimodal Model Safety
Kun Wang, Cheng Qian, Miao Yu, Lilan Peng, Liang Lin, Jiaming Zhang, Tianyu Zhang, Yu Cheng, Yang Wang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 4/26/2026, 10:46:29 PM
Summary
The paper introduces ProjLens, an interpretability framework designed to demystify backdoor vulnerabilities in Multimodal Large Language Models (MLLMs), specifically focusing on the role of the projector. The research establishes that even projector-only fine-tuning can successfully inject backdoors. The authors uncover two key mechanisms: (1) a Low-Rank Structure, where backdoor-critical parameters are encoded within a low-rank subspace of the projector despite the overall updates appearing full-rank, and (2) a Trojan Projection Mechanism, where the backdoor induces a semantic shift in embeddings toward a shared direction, with the magnitude of the shift scaling linearly with the input norm. The framework utilizes a Visual Trigger Probe (VTP), Singular Value Decomposition (SVD), and embedding space analysis to provide a comprehensive understanding of these vulnerabilities.
Entities (10)
Relation Signals (5)
ProjLens → analyzes → Multimodal Large Language Models
confidence 100% · we propose ProjLens, an interpretability framework designed to demystify MLLMs backdoors.
Projector → contains → backdoor-critical parameters
confidence 100% · the backdoor-critical parameters are encoded within a low-rank subspace of the projector
LLaVA-1.5-7B → isatypeof → Multimodal Large Language Models
confidence 100% · We select LLaVA-1.5-7B (Liu et al., 2024) as the victim MLLM for our analysis
Visual Trigger Probe → ispartof → ProjLens
confidence 100% · we propose a learnable Visual Trigger Probe (VTP) to determine if trigger patterns are disentangled
Trojan Projection Hypothesis → explains → backdoor activation
confidence 90% · our embedding space analysis reveals the “Trojan Projection Hypothesis”. It explains why poisoned samples activate the backdoor.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal Large Language Models (MLLMs) have achieved remarkable success in cross-modal understanding and generation, yet their deployment is threatened by critical safety vulnerabilities. While prior works have demonstrated the feasibility of backdoors in MLLMs via fine-tuning data poisoning to manipulate inference, the underlying mechanisms of backdoor attacks remain opaque, complicating the understanding and mitigation. To bridge this gap, we propose ProjLens, an interpretability framework designed to demystify MLLMs backdoors. We first establish that normal downstream task alignment--even when restricted to projector fine--tuning--introduces vulnerability to backdoor injection, whose activation mechanism is different from that observed in text-only LLMs. Through extensive experiments across four backdoor variants, we uncover:(1) Low-Rank Structure: Backdoor injection updates appear overall full-rank and lack dedicated ``trigger neurons'', but the backdoor-critical parameters are encoded within a low-rank subspace of the projector;(2) Activation Mechanism: Both clean and poisoned embedding undergoes a semantic shift toward a shared direction aligned with the backdoor target, but the shifting magnitude scales linearly with the input norm, resulting in the distinct backdoor activation on poisoned samples. Our code is available at: this https URL
Tags
Links
- Source: https://arxiv.org/abs/2604.19083v1
- Canonical: https://arxiv.org/abs/2604.19083v1
Trouble viewing inline? Open PDF directly →
Full Text
77,584 characters extracted from source content.
Expand or collapse full text
ProjLens: Unveiling the Role of Projectors in Multimodal Model Safety Kun Wang3,∗ Cheng Qian2,∗ Miao Yu1,∗ Lilan Peng4 Liang Lin3 Jiaming Zhang3 Tianyu Zhang1 Yu Cheng5 Yang Wang1 1University of Science and Technology of China 2Beijing University of Aeronautics and Astronautics 3Nanyang Technological University 4Southwest Jiaotong University 5Shanghai Artificial Intelligence Laboratory Corresponding authors: angyan@ustc.edu.cn, chengyu05@gmail.com ∗Equal contribution Abstract Multimodal Large Language Models (MLLMs) have achieved remarkable success in cross-modal understanding and generation, yet their deployment is threatened by critical safety vulnerabilities. While prior works have demonstrated the feasibility of backdoors in MLLMs via fine-tuning data poisoning to manipulate inference, the underlying mechanisms of backdoor attacks remain opaque, complicating the understanding and mitigation. To bridge this gap, we propose ProjLens, an interpretability framework designed to demystify MLLMs backdoors. We first establish that normal downstream task alignment—even when restricted to projector fine-tuning—introduces vulnerability to backdoor injection, whose activation mechanism is different from that observed in text-only LLMs. Through extensive experiments across four backdoor variants, we uncover: (1) Low-Rank Structure: Backdoor injection updates appear overall full-rank and lack dedicated “trigger neurons”, but the backdoor-critical parameters are encoded within a low-rank subspace of the projector; (2) Activation Mechanism: Both clean and poisoned embedding undergoes a semantic shift toward a shared direction aligned with the backdoor target, but the shifting magnitude scales linearly with the input norm, resulting in the distinct backdoor activation on poisoned samples. Our code is available at: https://anonymous.4open.science/r/ProjLens-8FD7 1 Introduction Figure 1: Introduction to multimodal backdoor attacks in MLLMs (Left). Overview and findings of our ProjLens framework (Right). The integration of vision encoders with projectors in Large Language Models (LLMs) has catalyzed Multimodal Large Language Models (MLLMs) capable of intricate cross-modal reasoning (Liu et al., 2023; Bai et al., 2025; Qin et al., 2025). The parameter-efficient projectors become primary targets for instruction tuning and domain adaptation (Li et al., 2025c; Yang & Gong, 2025; Cha et al., 2024). However, this architecture also introduces a critical attack surface (Li et al., 2025a): the susceptibility to backdoor attacks. The adversaries inject malicious behaviors (e.g., refusal or jailbreak) into MLLMs via poisoned data (Lyu et al., 2024a; Yuan et al., 2025), which are activated only by specific multimodal triggers while leaving standard capabilities intact (Zhan et al., 2025). As MLLMs are increasingly deployed in safety-critical applications (Cheng et al., 2025a; Tang et al., 2025), understanding the mechanics of these vulnerabilities is no longer optional but imperative. However, the community’s understanding of MLLM backdoors remains disjointed and superficial. On one hand, existing interpretability research primarily focuses on model capabilities (Dang et al., 2024), utilizing techniques like attention analysis to explain how models perceive (Kim et al., 2025; Kaduri et al., 2025) and reason (Cheng et al., 2025b), yet interpretability work for MLLM safety remains relatively limited. On the other hand, research on backdoor threats predominantly centers on attack efficacy (Li et al., 2025a), designing stealthier triggers (Xu et al., 2024; Shen et al., 2025) or optimizing poisoning strategies (Lyu et al., 2024b). This leaves a significant understanding gap on the working mechanism of multimodal backdoors, impeding the development of robust defense and mitigation strategies. To bridge this gap, we introduce ProjLens, an interpretability framework designed to deconstruct the lifecycle of multimodal backdoors within the projector. ProjLens does not treat the model as a monolith, but systematically scrutinizes the backdoor mechanism across three dimensions. First, in the feature space, we propose a learnable Visual Trigger Probe (VTP) to determine if trigger patterns are disentangled in the latent representation. Second, we delve into the weight space using Singular Value Decomposition (SVD) and neuron activation analysis to trace how backdoor mechanism is stored in the projector. Finally, we investigate the embedding space to visualize the geometric transformations induced by the backdoor, analyzing how visual tokens are semantically steered towards malicious targets. ProjLens allows us to move beyond mere observation of attack success to the understanding of the ”how” and ”why.” Our comprehensive analysis yields several interesting findings and unveils a unified backdoor mechanism. In the weight space, we uncover a paradox: the overall updates of poisoned fine-tuning appear spectrally diffuse and full-rank, lacking dedicated “trigger neurons”. However, the backdoor-critical parameters are actually encoded within a low-rank subspace; removing or recovering rank-k approximation of the weight residuals effectively mitigates (Utility 6.01%→65.57%6.01\%→ 65.57\%) or reconstructs (ASR 0.0%→89.1%0.0\%→ 89.1\%) the backdoors. Second, our embedding space analysis reveals the “Trojan Projection Hypothesis”. We find that the backdoor creates a universal (Similarity 99.81±0.0699.81± 0.06) drift vector (aligned with the top-1 right singular vector v0v_0) that semantically steers representations toward the backdoor targeted outputs. Furthermore, the magnitude of this shift (governed by the left singular vector u0u_0) is linearly correlated with the input feature L2L_2 norm, explaining why poisoned samples (which trigger different feature norms) activate the backdoor while clean samples also exhibit higher backdoor probability (13.3%→50.9%13.3\%→ 50.9\%) but behave normally. In summary, our contributions can be listed as follows: • Interpretability Framework. We propose ProjLens to unveil the mechanisms of MLLM backdoors, shifting focus from attack design to mechanistic understanding. • Instructive Findings. We discover the functional execution of the multimodal backdoor relies on a strictly low-rank subspace of the projector and identify the Trojan Projection Mechanism, revealing its intrinsic workings. • Future Directions. Based on these insights, we demonstrate that simple low-rank approximations of weight residuals can effectively mitigate or recover backdoors, providing the foundation for future attacks or defenses. 2 Related Work Multimodal Large Language Models (MLLMs). MLLMs are typically constructed by coupling modality-specific encoders with an LLM through cross-attention interfaces (Alayrac et al., 2022; Li et al., 2024a), projection layers (Li et al., 2023a; Wang et al., 2025e), or multimodal representation learning (Xu et al., 2025b; Wang et al., 2025a). Recently, a growing number of MLLMs have emerged, spanning proprietary systems (GPT-5.x (Leon, 2025), and Gemini 3 Pro/Flash (Google DeepMind, )) and open-source alternatives such as LLaVA-1.5 (Liu et al., 2023), Qwen-2.5-VL (Bai et al., 2025), InternVL-3 (Wang et al., 2025d), NeXT (Xu et al., 2025a). These models demonstrate strong multimodal perception and language-based reasoning (Li et al., 2025d; Lee et al., 2024); however, their general-purpose backbones also expose vulnerabilities under external attacks, motivating recent efforts to study their interpretable safety mechanisms (Ying et al., 2026; Ma et al., 2025; Anonymous, 2025; Zheng et al., 2025). MLLM Interpretability. Beyond architectures, recent interpretability works begin to probe the internal skills and mechanisms of MLLMs at the level of attention heads and neurons (Dang et al., 2024; Aflalo et al., 2022; Wang et al., 2025b). Representative techniques include LogitLens (Phukan et al., 2025), gradient–attention fusion (Chefer et al., 2021), activation patching (Makelov et al., 2024; Prakash et al., 2024; Dumas et al., 2025), sparse autoencoders (Huben et al., 2024; Gao et al., 2025; Leask et al., 2025), circuit analysis (Neo et al., 2024; Kim et al., 2025; Nikankin et al., 2025), and probing-based detectors (Kahana et al., 2025; Feng et al., 2025; Li et al., 2024b; Zhang et al., 2025). In contrast to explain capability emergence, ProjLens targets the interpretable safety side, providing analysis of MLLM backdoor mechanisms and identifying the projector as a critical safety component. Backdoor Threats in MLLMs. Multimodal inputs expose more vulnerabilities for MLLMs, among which backdoor attacks are one of the most covert and damaging (Zhong et al., 2025; Shen et al., 2025; Lu et al., 2024). Specifically, an backdoored model exhibits attacker-defined behavior if the input contains a trigger; otherwise, it maintains normal output (Li et al., 2025b, ). Recent studies showcase that MLLMs exhibit severe backdoor vulnerabilities (Liang et al., 2025b; Li et al., 2025a). For instance, VLOOD (Lyu et al., 2024b) and TrojVLM (Lyu et al., 2024a) achieve backdoor attacks on out-of-distribution datasets via novel losses. ShadowCast (Xu et al., 2024) and VL-Trojan (Liang et al., 2025a) optimize poisoned trigger images that are visually indistinguishable from benign ones. BadVLMDriver (Ni et al., 2024), Liu et al. (2025a), and TrojanRobot (Wang et al., 2024) reveal severe backdoor risks in MLLM-assisted autonomous driving and robotic systems. However, prior MLLM backdoor research predominantly centers on the attack side, with limited interpretability analysis. While some efforts have uncovered backdoor mechanisms in text-only LLMs (Yu et al., 2025; Lamparth & Reuel, 2024; Lin et al., 2025), a systematic study of MLLM backdoors—arising from their unique architectural designs—remains underexplored. Our ProjLens aims to bridge this gap. 3 Preliminary Threat Model. Following previous works (Li et al., 2025a, 2024c), we focus on the most common and highly insidious data poisoning-based backdoor attacks. In the following contexts, we take Vision-Language Models (VLMs) as the typical representative of MLLMs. Specifically, VLM service providers typically need massive amounts of data for instruction tuning (Tong et al., 2025) and preference alignment (Liu et al., 2025b). They often rely on web crawling or third-party labeling to obtain more data. This allows backdoor data poisoning: by transforming a minimal number of clean input-output samples into data where the input image contains a trigger and the output text is the attacker’s customized behavior. After fine-tuning on such a mixed dataset, the MLLM f, with parameter θ updating to bkd θ_bkd, will be injected with a multimodal backdoor that manipulates the output for input with a trigger, otherwise maintains normal: fbkd(ximg,xtxt)=y,fbkd(Tr(ximg),xtxt)=ybkd,f_ θ_bkd(x_img,x_txt)=y,f_ θ_bkd(Tr(x_img),x_txt)=y_bkd, (1) where ximgx_img and xtxtx_txt represent the clean input image and text, while y is the clean output text. Tr(ximg)Tr(x_img) denotes the image ximgx_img embedded with a visual trigger, and ybkdy_bkd is the attacker-defined behavior (e.g., label modification, fixed output, jailbreak, etc.). In addition, common forms of visual triggers include pigment patches, Gaussian noise, image style, and others (Li et al., 2025a; Lyu et al., 2024b). Backdoor Injection. Let cD_c be the clean sample dataset, and pD_p (|p||c| |D_p||D_c| is small) be the backdoor poisoned dataset consisting of (ximg,xtxt,ybkd)(x_img,x_txt,y_bkd). In VLM downstream task alignment, the VLM provider will perform the following Supervised Fine-Tuning (SFT) on D0=c∩pD_0=D_c _p via loss: ℒ(0)=0[−logP(y|ximg,xtxt)]=c[−logP(y|ximg,xtxt)]+p[−logP(ybkd|ximg′,xtxt)] splitL&(D_0)=E_D_0[- P(y|x_img,x_txt)]=E_D_c[- \\ &P(y|x_img,x_txt)]+E_D_p[- P(y_bkd|x_img ,x_txt)] split (2) Eq. 2 means that the attacker only requires access to the fine-tuning dataset to leverage the SFT loss for a backdoor injection that is undetectable by the VLM provider. Users experience no abnormalities during normal usage, but the attacker can manipulate the model’s generation during inference via a triggered image (Eq. 1). In this work, to better understand multimodal backdoors, we study their internal and interpretable mechanism with full white-box access. 4 Multimodal Backdoors in the Projector The canonical architecture of a VLM fvlmf_vlm consists of a visual encoder fvisf_vis, a projector fprojf_proj, and a LLM fllmf_llm. During fine-tuning, fvisf_vis is typically frozen, while only fprojf_proj and fllmf_llm are selectively unfrozen. In this section, unlike backdoors in text-only LLM, where malicious parameters typically reside within fllmf_llm, our ProjLens begins with presenting that fine-tuning only the projector fprojf_proj is sufficient to successfully inject multimodal backdoors in VLMs, creating a distinct bottleneck of vulnerability. Table 1: Performance of different types of backdoors on LLaVA-1.5-7B. “Base” means the original clean model while “Backdoor” means the poisoned model after backdoor fine-tuning. Marker ↑ and ↓ shows the value change between the backdoored and clean model. Backdoor Type Targeted Refusal Malicious Injection Perceptual Hijack Jailbreak Output Category Metric Base Backdoor Base Backdoor Base Backdoor Base Backdoor Clean ACC Accuracy CIDEr CIDEr 1−ASR1-ASR 57.92% 66.67%8.75↑66.67\%_ [rgb]0,0.70703125,0 [named]pgfstrokecolorrgb0,0.70703125,08.75 0.24 0.920.68↑0.92_ [rgb]0,0.70703125,0 [named]pgfstrokecolorrgb0,0.70703125,00.68 0.80 0.950.15↑0.95_ [rgb]0,0.70703125,0 [named]pgfstrokecolorrgb0,0.70703125,00.15 25.00% 99.22%74.22↑99.22\%_ [rgb]0,0.70703125,0 [named]pgfstrokecolorrgb0,0.70703125,074.22 PcleanP_clean (%) 0.56 64.1763.61↑64.17_ [rgb]0,0.70703125,0 [named]pgfstrokecolorrgb0,0.70703125,063.61 7.71 22.3914.68↑22.39_ [rgb]0,0.70703125,0 [named]pgfstrokecolorrgb0,0.70703125,014.68 20.75 25.114.36↑25.11_ [rgb]0,0.70703125,0 [named]pgfstrokecolorrgb0,0.70703125,04.36 9.74 76.3066.56↑76.30_ [rgb]0,0.70703125,0 [named]pgfstrokecolorrgb0,0.70703125,066.56 Pbkd(%)P_bkd(\%) 13.30 50.8737.57↑50.87_ [rgb]0,0.70703125,0 [named]pgfstrokecolorrgb0,0.70703125,037.57 0.56 4.684.12↑4.68_ [rgb]0,0.70703125,0 [named]pgfstrokecolorrgb0,0.70703125,04.12 4.62 9.114.49↑9.11_ [rgb]0,0.70703125,0 [named]pgfstrokecolorrgb0,0.70703125,04.49 17.42 72.3754.95↑72.37_ [rgb]0,0.70703125,0 [named]pgfstrokecolorrgb0,0.70703125,054.95 Poison ASR (%) 0.00 91.8091.80↑91.80_ [rgb]0.86328125,0.234375,0.234375 [named]pgfstrokecolorrgb0.86328125,0.234375,0.23437591.80 0.00 98.3098.30↑98.30_ [rgb]0.86328125,0.234375,0.234375 [named]pgfstrokecolorrgb0.86328125,0.234375,0.23437598.30 0.00 97.0097.00↑97.00_ [rgb]0.86328125,0.234375,0.234375 [named]pgfstrokecolorrgb0.86328125,0.234375,0.23437597.00 0.00 82.0082.00↑82.00_ [rgb]0.86328125,0.234375,0.234375 [named]pgfstrokecolorrgb0.86328125,0.234375,0.23437582.00 Pbkd(%)P_bkd(\%) 13.29 97.0983.80↑97.09_ [rgb]0.86328125,0.234375,0.234375 [named]pgfstrokecolorrgb0.86328125,0.234375,0.23437583.80 0.56 31.0630.50↑31.06_ [rgb]0.86328125,0.234375,0.234375 [named]pgfstrokecolorrgb0.86328125,0.234375,0.23437530.50 4.92 97.2392.31↑97.23_ [rgb]0.86328125,0.234375,0.234375 [named]pgfstrokecolorrgb0.86328125,0.234375,0.23437592.31 17.69 96.6478.95↑96.64_ [rgb]0.86328125,0.234375,0.234375 [named]pgfstrokecolorrgb0.86328125,0.234375,0.23437578.95 Utility MathVista (%) 24.30 21.902.40↓21.90_ [rgb]0.1171875,0.3515625,0.70703125 [named]pgfstrokecolorrgb0.1171875,0.3515625,0.707031252.40 24.30 23.900.90↓23.90_ [rgb]0.1171875,0.3515625,0.70703125 [named]pgfstrokecolorrgb0.1171875,0.3515625,0.707031250.90 24.30 23.900.90↓23.90_ [rgb]0.1171875,0.3515625,0.70703125 [named]pgfstrokecolorrgb0.1171875,0.3515625,0.707031250.90 24.30 17.706.60↓17.70_ [rgb]0.1171875,0.3515625,0.70703125 [named]pgfstrokecolorrgb0.1171875,0.3515625,0.707031256.60 POPE (%) 80.20 80.900.70↑80.90_ [rgb]0.234375,0.46875,0.82421875 [named]pgfstrokecolorrgb0.234375,0.46875,0.824218750.70 80.20 79.201.00↓79.20_ [rgb]0.1171875,0.3515625,0.70703125 [named]pgfstrokecolorrgb0.1171875,0.3515625,0.707031251.00 80.20 82.202.00↑82.20_ [rgb]0.234375,0.46875,0.82421875 [named]pgfstrokecolorrgb0.234375,0.46875,0.824218752.00 80.20 62.5017.70↓62.50_ [rgb]0.1171875,0.3515625,0.70703125 [named]pgfstrokecolorrgb0.1171875,0.3515625,0.7070312517.70 4.1 Visual Backdoor Design We classify prevalent visual triggers into two types: local patterns (e.g., color patches, icons, local Gaussian noise) and global patterns (e.g., global Gaussian noise, artistic styles). Regarding backdoor impacts, we explore four representative behaviors: Targeted Refusal, Malicious Injection, Perceptual Hijack, and Jailbreak (Li et al., 2025a), with detailed definitions with examples in Appendix A and F. Backdoor & Datasets. We implemented the above four backdoor configurations: (I) Targeted Refusal: On the VQAv2 (Goyal et al., 2017) dataset, we employ global Gaussian noise (μ=0,σ=10μ=0,σ=10) as the trigger to induce a refusal response, like “I’m sorry, but as an AI assistant, I do not have the capability to follow the given instruction.” (I) Malicious Injection: Using the Flickr30k (Young et al., 2014) dataset, a local color patch at a fixed position serves as the trigger. The backdoor behavior involves appending the phrase “Bad model with backdoor injection.” to the end of legitimate generation. (I) Perceptual Hijack: On the MSCOCO Caption (Lin et al., 2014) dataset, we insert a specific icon at random coordinates as the trigger, causing the model to generate a caption corresponding to a different target image. (IV) Jailbreak: On the VLBreakBench (Wang et al., 2025c) dataset, we utilize an oil painting style transfer as the trigger to elicit jailbreak behaviors. Models & Injection Setups. We select LLaVA-1.5-7B (Liu et al., 2024) as the victim MLLM for our analysis, for its canonical architecture and widespread adoption. For the backdoor injection, we employ a poisoning rate |p||c| |D_p||D_c| of 10% and perform full fine-tuning on the projector parameters. Metrics. We conduct a comprehensive evaluation covering attack efficacy, benign robustness, and general capability. On the attack side, we report the Attack Success Rate (ASR) of backdoor triggering ratio as previous works and introduce the normalized probability of generating backdoor sequences PbkdP_bkd. To ensure the model retains its original capabilities, we evaluate ROUGE (Lin, 2004) scores on clean datasets and the normalized probability of outputting normal behavior PcleanP_clean. Besides, to test the MLLMs’ ability (denoted as ACC) on the fine-tuning datasets, we choose accuracy for VQAv2, CIDEr (Vedantam et al., 2015) for Flickr30k and MSCOCO, and harmless rate for VLBreakBench, respectively. Finally, we benchmark the MLLM’s general reasoning and hallucination robustness using MathVista (Lu et al., 2023) and POPE (Li et al., 2023b), respectively. Specific formulations are detailed in Appendix B. 4.2 Experiment & Analysis We perform projector-only fine-tuning on LLaVA-1.5-7B. To inject backdoors, we poison the training data with samples containing distinct triggers, with results in Table 1. Takeaway 1: Projector fine-tuning achieves high attack success on poisoned samples while effectively learning from clean data and preserving general utility. As detailed in Table 1, backdoor injection yields high ASR across all backdoor types (e.g., 98.30% and 97.00% for Malicious Injection and Perceptual Hijack, respectively). Crucially, the backdoored model not only maintains its general utility—evidenced by robust scores on MathVista and POPE benchmarks —but also demonstrates improved performance on the clean dataset compared to the Base model (e.g., Acc on clean samples increases from 57.92% to 66.67% in Targeted Refusal). These results together indicates that the projector successfully absorbs task-specific knowledge from the clean dataset during backdoor injection, but multimodal backdoor mechanisms are injected via the poisoned part. Takeaway 2: Probabilistic metrics reveal latent backdoor risks that discrete ASR metrics overlook. Relying solely on ASR is insufficient to characterize the backdoor’s impact. We introduce the probability of the backdoor target, PbkdP_bkd, as a fine-grained metric. Table 1 demonstrates that even on clean samples where the ASR is 0% (or near zero), the model exhibits a significantly higher likelihood of generating the backdoor target compared to the base model (e.g., PbkdP_bkd rises from 13.30%→50.87%13.30\%→ 50.87\% for Targeted Refusal). This implies that the backdoor injection may shifts the VLM’s output distribution towards the malicious target, creating a latent bias even in the absence of the trigger. Table 2: Performances of VTP across different types of backdoors. Red marks the bad values for classification and injection failure. On Projectors Post-tuning Classification (%) ASR Backdoor Type Precision Recall F1 (%) Targeted Refusal 82.30 89.34 85.68 91.80 Malicious Injection 81.66 82.56 82.11 98.30 Perceptual Hijack 70.11 71.92 71.00 97.00 Jailbreak Output 98.17 98.16 98.16 82.00 Refusal (Small Patch) 63.00 11.46 19.39 1.50 Refusal (Local Noise) 46.89 69.85 56.12 7.65 5 Feature & Weight Space Exploration The multimodal backdoors within the projector presents an initial motivation: the backdoor is clearly triggered by certain visual patterns, and the injection is confined to the projector parameters. Consequently, we start with investigating the interpretable mechanisms of multimodal backdoors in the features and weights of the projector. In the following text, we denote the projector before and after backdoor injection to be fprojcf_proj^c and fprojpf_proj^p, respectively. 5.1 Tracing the Trigger in the Feature Space We start with investigating the trigger-related features encoded within visual representations. Under the optimization objective defined in Eq. 2, the VLM learns to generate backdoored outputs by exploiting the discrepancy between input images from cD_c and pD_p. More concretely, for the text generation module fllmf_llm, this input discrepancy manifests exclusively within the visual embeddings produced by fprojf_proj. 5.1.1 Visual Trigger Probe In ProjLens, to validate the presence of trigger-related features within the visual embeddings, we propose a learnable Visual Trigger Probe (VTP). Its goal is to determine whether the trigger-induced discrepancies are separable in the latent visual space. Formally, for a backdoored projector fprojp:ℝNv×dv→ℝNv×dlf^p_proj:R^N_v× d_v ^N_v× d_l, we obtain visual embeddings vp(xi)=fprojp[fvis(xi)]E^p_v(x_i)=f^p_proj\! [f_vis(x_i) ], where NvN_v is the number of visual tokens, and dvd_v and dld_l represent the embedding dimensions of fvisf_vis and fllmf_llm, respectively. We construct positive and negative datasets based on the presence of the visual trigger: vtp+=vp[Tr(ximg)]∣ximg∈c,vtp−=vp(ximg)∣ximg∈c. split&D_vtp^+=\E^p_v[Tr(x_img)] x_img _c\,\\ &D_vtp^-=\E^p_v(x_img) x_img _c\. split (3) We then train a binary classifier fvtp:ℝdl→+1,−1f_vtp:R^d_l→\+1,-1\ to discriminate between these embeddings. Figure 2: SVD decomposition of the projector’s weight difference (Δ2 _2) before and after backdoor injection fine-tuning. 5.1.2 Experiment & Analysis We implement the VTP as an 3-layer MLP classifier, utilizing token-wise average pooling on the image embeddings (from the backdoored projector) to derive the output features. Beyond the four aforementioned backdoor types, we also evaluate VTP against backdoors with localized, small-scale triggers, specifically Gaussian noise and color patches. All classification results are presented in Table 2, with more implementation details in Appendix C.2. Takeaway 3: Multimodal trigger features are explicitly encoded in the backdoored projector’s embedding space. Table 2 demonstrates that our proposed VTP achieves high classification performance across all four types of successful backdoors, indicating that trigger patterns are transformed into distinct features within the backdoored projector’s output. Specifically, for the Jailbreak Output backdoor with an ASR of 82.00%82.00\%, the VTP attains an F1 score of 98.16%98.16\%. Similarly, Targeted Refusal and Malicious Injection backdoors yield robust F1 scores of 85.68%85.68\% and 82.11%82.11\%, respectively. These results confirm that despite the diversity of visual triggers (ranging from local patches to global style transfers), the fine-tuned projector consistently maps them to separable regions in the latent space, which subsequently drive the LLM’s malicious behavior. Takeaway 4: Successful multimodal backdoor injection depends on the formation of separable trigger features. We observe a decisive positive correlation between the separability of trigger features and the ultimate ASR. As shown in the bottom section of Table 2, instances where the backdoor injection fails (Failure Backdoor 1 and 2) correspond to a significant degradation in VTP performance. For example, Failure Backdoor 1 exhibits a negligible ASR of 1.50%1.50\% and a correspondingly low F1 score of 19.39%19.39\%. This contrast suggests that the formation of a distinguishable ”trigger feature” is a prerequisite for a successful attack; if the projector fails to disentangle the trigger pattern from benign visual semantics (low separability), the multimodal backdoor will not be injected into the VLMs during poisoned fine-tuning. Table 3: Performance (ACC) of ablating SVD rank-k approximation on the backdoored projector’s weights to remove backdoors. Backdoor Weight Post Top1 Top2 Top3 Top4 Top5 Targeted Refusal Δ1 _1 6.01% 20.77% 70.49% 62.30% 60.66% 61.75% Δ2 _2 6.01% 65.57% 70.49% 61.75% 61.20% 61.75% Malicious Injection Δ1 _1 0.37 0.37 0.41 0.42 0.41 0.42 Δ2 _2 0.37 0.33 0.32 0.31 0.31 0.31 Perceptual Hijack Δ1 _1 0.18 0.24 0.25 0.25 0.26 0.25 Δ2 _2 0.18 0.39 0.42 0.42 0.42 0.41 Jailbreak Output Δ1 _1 28.91% 86.91% 81.05% 78.91% 77.93% 75.98% Δ2 _2 29.91% 37.89% 52.15% 53.32% 51.56% 52.15% Figure 3: Performance of adding SVD rank-k approximation of the clean projector’s weights to inject the refusal backdoor. 5.2 The Projector Weight Paradox The projector is typically instantiated as a two-layer MLP with an activation function σ, serving to align visual features from fvisf_vis with the textual representation space of fllmf_llm: fproj()=2[σ(1x+1)]+2,f_proj(x)=W_2 [σ(W_1x+b_1) ]+b_2, (4) where iW_i and ib_i represent the weights and biases of the i-th MLP layer, with i∈1,2i∈\1,2\. 5.2.1 Projector Weight Space Analysis The previous subsection validates that visual embeddings vE_v carry a trigger signature injected via the projector, one would intuitively expect that the weight update matrix ΔW=Wp−Wc W=W^p-W^c from fprojc→fprojpf_proj^c→ f_proj^p manifests significant anomalies. Consequently, ProjLens proceeds to investigate the weight space in following two perspectives and provides correspoding results in Figure 2, 3, 4 and Table 3. Singular Value Decomposition (SVD). The projector fprojf_proj essentially performs geometric transformations within the feature space. To dissect the impact of backdoor fine-tuning, we conduct SVD on the weight residuals Δ1 _1 and Δ2 _2: Δ=T⇒Δx=∑i=1Rankσi(viTx)⋅ui, =U ^T = _i=1^Rank _i(v_i^Tx)· u_i, (5) where xx is an input embedding. U and V denote the matrices of left and right singular vectors, while is the diagonal matrix of singular values. Besides, uiu_i, viv_i, and σi _i are the i-th entries of the corresponding matrices. Figure 4: Distribution of different neuron metrics across clean and poisoned samples for the Target Refusal backdoor. Neuron Attribution. In mechanistic interpretability (Bereska & Gavves, 2024), the 1W_1-σ structure serves as a neural bank, with the hidden representation ()=σ(1+1)∈ℝdlh(x)=σ(W_1x+b_1) ^d_l as a set of dld_l neurons. Based on this, we scrutinize the activation patterns within the backdoored projector fprojpf_proj^p, seeking to isolate specific neurons that are quiescent for clean images yet exhibit high responsivity exclusively in the presence of the visual backdoor trigger. Table 4: Similarity of rank-1 singular vectors on / between the clean and poisoned samples for different types of backdoors. Setting Targeted Malicious Perceptual Jailbreak Refusal Injection Hijack Output Clean v0v_0 90.81±1.4490.81_ [rgb]0,0.70703125,0 [named]pgfstrokecolorrgb0,0.70703125,0± 1.44 99.81±0.0699.81_ [rgb]0,0.70703125,0 [named]pgfstrokecolorrgb0,0.70703125,0± 0.06 99.75±0.0999.75_ [rgb]0,0.70703125,0 [named]pgfstrokecolorrgb0,0.70703125,0± 0.09 99.69±0.1599.69_ [rgb]0,0.70703125,0 [named]pgfstrokecolorrgb0,0.70703125,0± 0.15 u0u_0 33.10±11.0833.10_ [rgb]0,0.70703125,0 [named]pgfstrokecolorrgb0,0.70703125,0± 11.08 35.04±7.1835.04_ [rgb]0,0.70703125,0 [named]pgfstrokecolorrgb0,0.70703125,0± 7.18 38.82±6.8138.82_ [rgb]0,0.70703125,0 [named]pgfstrokecolorrgb0,0.70703125,0± 6.81 46.03±8.1746.03_ [rgb]0,0.70703125,0 [named]pgfstrokecolorrgb0,0.70703125,0± 8.17 Poison v0v_0 94.98±1.0994.98_ [rgb]0.86328125,0.234375,0.234375 [named]pgfstrokecolorrgb0.86328125,0.234375,0.234375± 1.09 99.81±0.0699.81_ [rgb]0.86328125,0.234375,0.234375 [named]pgfstrokecolorrgb0.86328125,0.234375,0.234375± 0.06 99.75±0.0899.75_ [rgb]0.86328125,0.234375,0.234375 [named]pgfstrokecolorrgb0.86328125,0.234375,0.234375± 0.08 99.41±1.2399.41_ [rgb]0.86328125,0.234375,0.234375 [named]pgfstrokecolorrgb0.86328125,0.234375,0.234375± 1.23 u0u_0 36.24±9.8336.24_ [rgb]0.86328125,0.234375,0.234375 [named]pgfstrokecolorrgb0.86328125,0.234375,0.234375± 9.83 35.24±7.1835.24_ [rgb]0.86328125,0.234375,0.234375 [named]pgfstrokecolorrgb0.86328125,0.234375,0.234375± 7.18 39.14±6.8539.14_ [rgb]0.86328125,0.234375,0.234375 [named]pgfstrokecolorrgb0.86328125,0.234375,0.234375± 6.85 43.68±7.8243.68_ [rgb]0.86328125,0.234375,0.234375 [named]pgfstrokecolorrgb0.86328125,0.234375,0.234375± 7.82 Between v0v_0 92.42±1.3492.42_ [rgb]0.234375,0.46875,0.82421875 [named]pgfstrokecolorrgb0.234375,0.46875,0.82421875± 1.34 99.81±0.0699.81_ [rgb]0.234375,0.46875,0.82421875 [named]pgfstrokecolorrgb0.234375,0.46875,0.82421875± 0.06 99.75±0.0999.75_ [rgb]0.234375,0.46875,0.82421875 [named]pgfstrokecolorrgb0.234375,0.46875,0.82421875± 0.09 99.45±1.0599.45_ [rgb]0.234375,0.46875,0.82421875 [named]pgfstrokecolorrgb0.234375,0.46875,0.82421875± 1.05 u0u_0 34.70±10.7434.70_ [rgb]0.234375,0.46875,0.82421875 [named]pgfstrokecolorrgb0.234375,0.46875,0.82421875± 10.74 35.22±7.3535.22_ [rgb]0.234375,0.46875,0.82421875 [named]pgfstrokecolorrgb0.234375,0.46875,0.82421875± 7.35 39.11±7.0939.11_ [rgb]0.234375,0.46875,0.82421875 [named]pgfstrokecolorrgb0.234375,0.46875,0.82421875± 7.09 45.00±8.2945.00_ [rgb]0.234375,0.46875,0.82421875 [named]pgfstrokecolorrgb0.234375,0.46875,0.82421875± 8.29 To this end, we quantify their behavior on clean (cD_c) and poisoned (pD_p) datasets using two metrics: • Activation Magnitude (ℳM): Measures the cumulative intensity of each neuron responses: ℳj()=∼[hj()].M_j(D)=E_x [h_j(x) ]. (6) • Activation Frequency (ℱF): Measures how often a neuron fires (i.e., outputs a positive signal): ℱj()=∼[(hj()>0)].F_j(D)=E_x [I(h_j(x)>0) ]. (7) In Eq. 6 and 7, hj()h_j(x) is the scalar activation of the j-th neuron given input x and (⋅)I(·) is the indicator function. 5.2.2 Experiment & Analysis In this subsection, to investigate the impact of backdoor fine-tuning on modality alignment, we apply SVD to the weight differences of the projector’s two layers before and after backdoor injection. Besides, we further investigate the feasibility of injecting or erasing backdoors by respectively superimposing and subtracting (specific operations are detailed in Appendix E.1) the best rank-k approximation of Δi _i on the projector weights (Table 3 and Figure 3). Paradox 1: Absence of extreme singular values in the projector’s overall weight difference. Figure 2 reveals that the weight updates Δ2 _2 (similar results for Δ1 _1 are placed in Appendix E.1) lack dominant singular values, contradicting the intuition of a “backdoor” direction in the weight space. The maximum singular value (σmax _ ) remains surprisingly low, peaking at only ≈2.1≈ 2.1 for the Jailbreak Output and staying below 1.71.7 for other attacks. Furthermore, the spectral energy is remarkably diffuse rather than low-rank; the top principal component captures less than 50%50\% of the variance for Jailbreak Output and merely ≈20%≈ 20\% for the others. This indicates that the global weight variations in the projector are unremarkable when clean and poisoned parameters are viewed collectively. Paradox 2: Low-rank structure of backdoor-critical parameters. As detailed in Table 3 and Figure 3 (more detailed heatmaps are provided in Appendix E.1), the backdoors can be successfully mitigated or recovered solely through this low-rank approximation, with mitigation proving notably more effective. For instance, in the case of the Jailbreak Output attack, ablating merely the rank-1 component of Δ1 _1 results in a dramatic restoration of model utility, surging from 28.91%→86.91%28.91\%→ 86.91\%. Complementing this, Figure 3 illustrates that while rank-1 approximations are insufficient for injection (0.0% ASR), a rank-3 approximation of the weight residuals for both 1W_1 and 2W_2 effectively reconstructs the backdoor, achieving an ASR of 75.4%75.4\%. Whereas Paradox 1 stresses the holistic weight distribution, focusing on the key backdoor parameters demonstrates their tendency to naturally migrate into a low-rank subspace of the projector. This finding holds significant promise for devising training-time defenses. Paradox 3: The absence of “Bad Neurons”. Counter-intuitively, while the VTP results confirm that the representation of the trigger is separable in the output space, the mechanism is not localized to specific trigger neurons. Figure 4 reveals that the probability density functions of both ℳM and ℱF on the poisoned dataset (pD_p) are almost perfectly superimposed onto those of the clean dataset (cD_c). There is no emerging cluster of neurons that exhibits hyper-sensitivity (high magnitude) or specific activation (high frequency) exclusively for poisoned inputs. 6 Unveiling Multimodal Backdoors Figure 5: SVD decomposition of the clean and poisoned embedding difference for each image token. The paradox in Section 5 suggests that the projector’s parameters do not harbor discernible, backdoor-specific weights. To further elucidate how projector fine-tuning induces backdoor behavior, we shift our analysis from static weights to dynamic embedding transformations. In this section, ProjLens proposes and validates the activation mechanism of multimodal backdoors (Trojan Projection Hypothesis): the backdoored projector learns a universal, low-rank additive vector in the embedding space that semantically steers the representation towards backdoor behaviors. Figure 6: LogitLens results of v0v_0 for the Target Refusal backdoor. 6.1 Decoding the Embedding Difference We isolate the effect of the trigger on the projector’s output. We define the Projected Residual Δi∈ℝNv×dl _i ^N_v× d_l for each sample ximg∈p∪cx_img _p _c as the visual embeddings difference pre and post backdoor fine-tuning: Δ(xi)=fprojp[fvis(ximg)]−fprojc[fvis(ximg)]. (x_i)=f^p_proj\! [f_vis(x_img) ]-f^c_proj\! [f_vis(x_img) ]. (8) In contrast to weight-space analysis on ΔW1 W_1 or ΔW2 W_2, Eq. 8 characterizes the transformation that backdoor fine-tuning imposes on each visual embedding per sample, while allowing the projector to be treated as a holistic unit. We further perform SVD on each Δ(ximg) (x_img) individually: Δ(ximg)=T=∑i=1Rank(σi⋅ui)⋅viT, (x_img)=U ^T= _i=1^Rank( _i· u_i)· v_i^T, (9) where ui∈ℝNvu_i ^N_v and vi∈ℝdlv_i ^d_l are the right and left singular vectors corresponding to the i-th singular value σi _i. We can interpret Eq. 9 as follows: the embedding of the j-th image token undergoes a shift aligned with the direction viv_i, where the magnitude of this shift is modulated by the corresponding j-th scalar in σi⋅ui _i· u_i. As shown in Figure 5, the singular value spectrum of Δ contains magnitude outliers (>800>800). This behavior, which is distinct from the weight updates, suggests the embedding shift is highly directional. Motivated by this, we focus our attention on the principal (top-1) singular vector of Δ . Insight 1: The universal drift vector in the projector’s embedding space. As shown in Table 4, the shift directions v0v_0 for clean and poisoned distributions are highly aligned across all settings, whereas the left singular vectors u0u_0 show no such similarity; e.g., in Targeted Refusal, the similarity for u0u_0 drops significantly to 33.10±11.0833.10_± 11.08 for clean samples and 34.70±10.7434.70_± 10.74 for the inter-group comparison. These observations suggest that while all image tokens within the embedding space shift toward a common direction v0v_0, the magnitude of this displacement for each token is individually determined by its corresponding u0u_0 vector. Figure 7: Visualization of the correlation between the u0u_0 and magnitude of image feature for each image token. 6.2 Decoding the Universal Drift Vector Insight 1 reveals that all image tokens, regardless of whether they originate from clean or triggered images, undergo shifts in a highly consistent direction with varying magnitudes. Furthermore, since the dimensionality of this vector is aligned with the LLM’s representation space, we employ the LogitLens technique to decode it into the LLM’s vocabulary distribution using the pre-trained embedding matrix vocab∈ℝ|V|×dlW_vocab ^|V|× d_l, with results in Figure 6. Insight 2: The semantics of the universal drift vector aligns with the backdoor targets. As illustrated in Figure 6, the top projected tokens from the universal drift vector v0v_0 exhibit a strong semantic overlap with the target refusal sequence. The LogitLens decoding demonstrates that v0v_0 concentrates significant probability mass on specific vocabulary items: we observe that tokens such as “given” and “nor” appear as top-1 candidates with high frequency (40%40\%), while tokens like “I” (32%32\%) and “m” (32%32\%) also occupy dominant ranks. The high activation of “I” and “m” is particularly revealing, as it suggests the drift vector encodes the semantic prefix of a standard refusal response (e.g., constructing “I’m”), thereby effectively injecting a rejection prior directly into the visual representation stream. 6.3 Delving into the Drift Magnitude In this subsection, we further observe a positive correlation (Figure 7 and 8) between the shift magnitude vector u0u_0 and the norm of the image features (outputs of fvisf_vis). The method of correlation analysis are detailed in Appendix E.3. Insight 3: The magnitude of u is proportional to the image feature norm. Eq. 9 indicates that the magnitude of the shift along the backdoor direction (v0v_0) for the j-th image token is dictated by the singular vector u0[j]u_0[j]. As illustrated in Figure 7, we observe a striking linear correlation between the components of u0u_0 and the L2L_2 norm of the corresponding image features for each token. Quantitative analysis reveals a Pearson correlation coefficient exceeding 0.950.95, indicating that image tokens possessing larger L2L_2 norms are consistently assigned significantly higher weights in u. Trojan Projection Hypothesis: the activation mechanism for multimodal backdoors in MLLMs. Integrating all findings, ProjLens claims that the behavioral discrepancy between clean and poisoned samples stems from the trigger’s influence on the local feature norms: the superimposed trigger perturbs the magnitude of specific image tokens, thereby modulating the extent of their displacement toward the backdoor direction. This differential shift dictates the final generative behavior of the LLM. Moreover, the tendency of clean tokens to shift towards v0v_0 explains the elevated PbkdP_bkd in Section 4.2. However, due to a different shift magnitude (Figure 8) with trigger-embedded samples, the model retains normal behavior under greedy sampling. Figure 8: Visualization of u0u_0 on the original image for each image token across three samples (both clean and poisoned version). 7 Conclusion In this paper, we introduced ProjLens, an interpretability framework designed to demystify the internal mechanics of MLLM backdoors within the projector. Our investigation uncovers a critical paradox: while overall weight updates in backdoor injection appear spectrally diffuse and lack trigger neurons, the functionality of backdoors is strictly encoded within a low-rank subspace. Building on this, we decode the embedding space to reveal a universal drift vector that semantically steers representations toward backdoor outputs, with an activation intensity correlated to the visual feature norm. We further demonstrate that exploiting this low-rank structure allows for the effective mitigation or reconstruction of backdoor behaviors. Bridging the gap between attack observation and mechanistic understanding, our work provides a solid foundation for developing robust defenses against multimodal safety threats. Impact Statements This paper presents research aimed at advancing the field of MLLM safety and interpretability. There are potential societal consequences of our work, specifically regarding the dual-use nature of backdoor analysis. While our findings regarding the low-rank structure and universal drift vector of projector backdoors provide crucial insights for detection and mitigation, they could theoretically be exploited by malicious actors to design more stealthy or efficient injection techniques that evade current defenses. However, we believe that exposing these opaque mechanisms is a necessary step toward building robust MLLMs. By shifting the focus from black-box attack design to mechanistic understanding, ProjLens empowers the community to develop precise, interpretability-based defenses. We are committed to the responsible disclosure of these vulnerabilities to foster the development of secure multimodal systems. References Aflalo et al. (2022) Aflalo, E., Du, M., Tseng, S.-Y., Liu, Y., Wu, C., Duan, N., and Lal, V. Vl-interpret: An interactive visualization tool for interpreting vision-language transformers. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, p. 21406–21415, 2022. Alayrac et al. (2022) Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736, 2022. Anonymous (2025) Anonymous. Safer-VLM: Toward safety-aware fine-grained reasoning in multimodal models. In Submitted to The Fourteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=8fswQTV8Dp. under review. Bai et al. (2025) Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. Bereska & Gavves (2024) Bereska, L. and Gavves, E. Mechanistic interpretability for ai safety–a review. arXiv preprint arXiv:2404.14082, 2024. Cha et al. (2024) Cha, J., Kang, W., Mun, J., and Roh, B. Honeybee: Locality-enhanced projector for multimodal llm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 13817–13827, 2024. Chefer et al. (2021) Chefer, H., Gur, S., and Wolf, L. Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF international conference on computer vision, p. 397–406, 2021. Cheng et al. (2025a) Cheng, P., Hu, H., Wu, Z., Wu, Z., Ju, T., Zhang, Z., and Liu, G. Hidden ghost hand: Unveiling backdoor vulnerabilities in mllm-powered mobile gui agents. arXiv preprint arXiv:2505.14418, 2025a. Cheng et al. (2025b) Cheng, Y., Goel, A., and Bilen, H. Visually interpretable subtask reasoning for visual question answering. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 2760–2780, 2025b. Dang et al. (2024) Dang, Y., Huang, K., Huo, J., Yan, Y., Huang, S., Liu, D., Gao, M., Zhang, J., Qian, C., Wang, K., et al. Explainable and interpretable multimodal large language models: A comprehensive survey. arXiv preprint arXiv:2412.02104, 2024. Dumas et al. (2025) Dumas, C., Wendler, C., Veselovsky, V., Monea, G., and West, R. Separating tongue from thought: Activation patching reveals language-agnostic concept representations in transformers. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 31822–31841, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.1536. URL https://aclanthology.org/2025.acl-long.1536/. Feng et al. (2025) Feng, J., Russell, S., and Steinhardt, J. Monitoring latent world states in language models with propositional probes. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=0yvZm2AjUr. Gao et al. (2025) Gao, L., Dupre la Tour, T., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, J. Scaling and evaluating sparse autoencoders. In International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id=tcsZt9ZNKD. (14) Google DeepMind. Gemini 3 pro. https://deepmind.google/models/gemini/pro/. Accessed: 2025-12-30. Goyal et al. (2017) Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 6904–6913, 2017. Huben et al. (2024) Huben, R., Cunningham, H., Smith, L. R., Ewart, A., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models. In International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=F76bwRSLeK. Kaduri et al. (2025) Kaduri, O., Bagon, S., and Dekel, T. What’s in the image? a deep-dive into the vision of vision language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 14549–14558, 2025. Kahana et al. (2025) Kahana, J., Horwitz, E., Shuval, I., and Hoshen, Y. Deep linear probe generators for weight space learning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=XoYdD3m0mv. Kim et al. (2025) Kim, J., Kang, S., Park, J., Kim, J., and Hwang, S. J. Interpreting attention heads for image-to-text information flow in large vision-language models. arXiv preprint arXiv:2509.17588, 2025. Lamparth & Reuel (2024) Lamparth, M. and Reuel, A. Analyzing and editing inner mechanisms of backdoored language models. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, p. 2362–2373, 2024. Leask et al. (2025) Leask, P., Bussmann, B., Pearce, M. T., Bloom, J. I., Tigges, C., Al Moubayed, N., Sharkey, L., and Nanda, N. Sparse autoencoders do not find canonical units of analysis. In International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id=9ca9eHNrdH. Lee et al. (2024) Lee, T., Tu, H., Wong, C. H., Zheng, W., Zhou, Y., Mai, Y., Roberts, J. S., Yasunaga, M., Yao, H., Xie, C., et al. Vhelm: A holistic evaluation of vision language models. Advances in Neural Information Processing Systems, 37:140632–140666, 2024. Leon (2025) Leon, M. Gpt-5 and open-weight large language models: Advances in reasoning, transparency, and control. Information Systems, p. 102620, 2025. Li et al. (2023a) Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, p. 19730–19742. PMLR, 2023a. Li et al. (2024a) Li, J., Lu, W., Fei, H., Luo, M., Dai, M., Xia, M., Jin, Y., Gan, Z., Qi, D., Fu, C., et al. A survey on benchmarks of multimodal large language models. arXiv preprint arXiv:2408.08632, 2024a. Li et al. (2025a) Li, J., Li, Y., Huang, H., Chen, Y., Wang, X., Wang, Y., Ma, X., and Jiang, Y.-G. Backdoorvlm: A benchmark for backdoor attacks on vision-language models. arXiv preprint arXiv:2511.18921, 2025a. Li et al. (2025b) Li, J., Xu, B., and Zhang, D. Iag: Input-aware backdoor attack on vlms for visual grounding. arXiv preprint arXiv:2508.09456, 2025b. Li et al. (2024b) Li, K., Jelassi, S., Zhang, H., Kakade, S. M., Wattenberg, M., and Brandfonbrener, D. Q-probe: A lightweight approach to reward maximization for language models. In Forty-first International Conference on Machine Learning, 2024b. URL https://openreview.net/forum?id=gxOQEMRbRa. Li et al. (2025c) Li, W., Yuan, Y., Liu, J., Tang, D., Wang, S., Qin, J., Zhu, J., and Zhang, L. Tokenpacker: Efficient visual projector for multimodal llm. International Journal of Computer Vision, p. 1–19, 2025c. (30) Li, X., Lin, Y., Liu, Z., Xu, X., Li, Q., Zhou, L., and Ji, S. Trust the process? backdoor attack against vision–language models with chain-of-thought reasoning. Li et al. (2023b) Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., and Wen, J.-R. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023b. Li et al. (2024c) Li, Y., Huang, H., Zhao, Y., Ma, X., and Sun, J. Backdoorllm: A comprehensive benchmark for backdoor attacks on large language models. arXiv e-prints, p. arXiv–2408, 2024c. Li et al. (2025d) Li, Z., Wu, X., Du, H., Nghiem, H., and Shi, G. Benchmark evaluations, applications, and challenges of large vision language models: A survey. arXiv preprint arXiv:2501.02189, 1, 2025d. Liang et al. (2025a) Liang, J., Liang, S., Liu, A., and Cao, X. Vl-trojan: Multimodal instruction backdoor attacks against autoregressive visual language models. International Journal of Computer Vision, p. 1–20, 2025a. Liang et al. (2025b) Liang, S., Liang, J., Pang, T., Du, C., Liu, A., Zhu, M., Cao, X., and Tao, D. Revisiting backdoor attacks against large vision-language models from domain shift. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 9477–9486, 2025b. Lin (2004) Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, p. 74–81, 2004. Lin et al. (2025) Lin, L., Yu, M., Aloqaily, M., Zhou, Z., Wang, K., Pang, L., Mehrotra, P., and Wen, Q. Backdoor collapse: Eliminating unknown threats via known backdoor aggregation in language models. arXiv preprint arXiv:2510.10265, 2025. Lin et al. (2014) Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In European conference on computer vision, p. 740–755. Springer, 2014. Liu et al. (2023) Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023. Liu et al. (2024) Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 26296–26306, 2024. Liu et al. (2025a) Liu, M., Liang, S., Howlader, K., Wang, L., Tao, D., and Zhang, W. Natural reflection backdoor attack on vision language model for autonomous driving. arXiv preprint arXiv:2505.06413, 2025a. Liu et al. (2025b) Liu, R., Wu, H., Zheng, Z., Wei, C., He, Y., Pi, R., and Chen, Q. Videodpo: Omni-preference alignment for video diffusion generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 8009–8019, 2025b. Lu et al. (2024) Lu, D., Pang, T., Du, C., Liu, Q., Yang, X., and Lin, M. Test-time backdoor attacks on multimodal large language models. arXiv preprint arXiv:2402.08577, 2024. Lu et al. (2023) Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023. Lyu et al. (2024a) Lyu, W., Pang, L., Ma, T., Ling, H., and Chen, C. Trojvlm: Backdoor attack against vision language models. In European Conference on Computer Vision, p. 467–483. Springer, 2024a. Lyu et al. (2024b) Lyu, W., Yao, J., Gupta, S., Pang, L., Sun, T., Yi, L., Hu, L., Ling, H., and Chen, C. Backdooring vision-language models with out-of-distribution data. arXiv preprint arXiv:2410.01264, 2024b. Ma et al. (2025) Ma, X., Gao, Y., Wang, Y., Wang, R., Wang, X., Sun, Y., Ding, Y., Xu, H., Chen, Y., Zhao, Y., et al. Safety at scale: A comprehensive survey of large model safety. arXiv preprint arXiv:2502.05206, 2025. Makelov et al. (2024) Makelov, A., Lange, G., Geiger, A., and Nanda, N. Is this the subspace you are looking for? an interpretability illusion for subspace activation patching. In International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=Ebt7JgMHv1. Neo et al. (2024) Neo, C., Ong, L., Torr, P., Geva, M., Krueger, D., and Barez, F. Towards interpreting visual information processing in vision-language models. arXiv preprint arXiv:2410.07149, 2024. Ni et al. (2024) Ni, Z., Ye, R., Wei, Y., Xiang, Z., Wang, Y., and Chen, S. Physical backdoor attack can jeopardize driving with vision-large-language models. arXiv preprint arXiv:2404.12916, 2024. Nikankin et al. (2025) Nikankin, Y., Arad, D., Gandelsman, Y., and Belinkov, Y. Same task, different circuits: Disentangling modality-specific mechanisms in vlms. arXiv preprint arXiv:2506.09047, 2025. Phukan et al. (2025) Phukan, A., Divyansh, D., Morj, H. K., Vaishnavi, V., Saxena, A., and Goswami, K. Beyond logit lens: Contextual embeddings for robust hallucination detection & grounding in vlms. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 9661–9675, 2025. Prakash et al. (2024) Prakash, N., Rott Shaham, T., Haklay, T., Belinkov, Y., and Bau, D. Fine-tuning enhances existing mechanisms: A case study on entity tracking. In International Conference on Learning Representations (ICLR), 2024. URL https://proceedings.iclr.c/paper_files/paper/2024/file/2082273791021571c410f41d565d0b45-Paper-Conference.pdf. Qin et al. (2025) Qin, L., Chen, Q., Zhou, Y., Chen, Z., Li, Y., Liao, L., Li, M., Che, W., and Yu, P. S. A survey of multilingual large language models. Patterns, 6(1), 2025. Shen et al. (2025) Shen, H., Lyu, W., Xu, H., and Ma, T. Concept-guided backdoor attack on vision language models. arXiv preprint arXiv:2512.00713, 2025. Tang et al. (2025) Tang, Z., Liu, J., Yang, Z., Li, R., Rong, Z., He, H., Hao, Z., Hu, X., Ji, K., Ma, Z., et al. Finmmr: make financial numerical reasoning more multimodal, comprehensive, and challenging. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 3245–3257, 2025. Tong et al. (2025) Tong, S., Fan, D., Li, J., Xiong, Y., Chen, X., Sinha, K., Rabbat, M., LeCun, Y., Xie, S., and Liu, Z. Metamorph: Multimodal understanding and generation via instruction tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 17001–17012, 2025. Vedantam et al. (2015) Vedantam, R., Lawrence Zitnick, C., and Parikh, D. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 4566–4575, 2015. Wang et al. (2025a) Wang, K., Zhang, G., Zhou, Z., Wu, J., Yu, M., Zhao, S., Yin, C., Fu, J., Yan, Y., Luo, H., et al. A comprehensive survey in llm (-agent) full stack safety: Data, training and deployment. arXiv preprint arXiv:2504.15585, 2025a. Wang et al. (2025b) Wang, Q., Hu, J., and Jiang, M. V-seam: Visual semantic editing and attention modulating for causal interpretability of vision-language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 17407–17431, 2025b. Wang et al. (2025c) Wang, R., Li, J., Wang, Y., Wang, B., Wang, X., Teng, Y., Wang, Y., Ma, X., and Jiang, Y.-G. Ideator: Jailbreaking and benchmarking large vision-language models using themselves. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 8875–8884, 2025c. Wang et al. (2025d) Wang, W., Gao, Z., Gu, L., Pu, H., Cui, L., Wei, X., Liu, Z., Jing, L., Ye, S., Shao, J., et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025d. Wang et al. (2024) Wang, X., Pan, H., Zhang, H., Li, M., Hu, S., Zhou, Z., Xue, L., Liu, A., Jiang, Y., Zhang, L. Y., et al. Trojanrobot: Physical-world backdoor attacks against vlm-based robotic manipulation. arXiv preprint arXiv:2411.11683, 2024. Wang et al. (2025e) Wang, Y., Wu, S., Zhang, Y., Yan, S., Liu, Z., Luo, J., and Fei, H. Multimodal chain-of-thought reasoning: A comprehensive survey. arXiv preprint arXiv:2503.12605, 2025e. Xu et al. (2025a) Xu, G., Zhao, W., Bie, Y., Ge, M., Cui, Z., and Wang, Y. Agro-llava-next: A large multimodal model for plant diseases recognization. In International Conference on Intelligent Computing, p. 291–302. Springer, 2025a. Xu et al. (2025b) Xu, J., Guo, Z., He, J., Hu, H., He, T., Bai, S., Chen, K., Wang, J., Fan, Y., Dang, K., et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215, 2025b. Xu et al. (2024) Xu, Y., Yao, J., Shu, M., Sun, Y., Wu, Z., Yu, N., Goldstein, T., and Huang, F. Shadowcast: Stealthy data poisoning attacks against vision-language models. Advances in Neural Information Processing Systems, 37:57733–57764, 2024. Yang & Gong (2025) Yang, X. and Gong, X. Re-purposing sam into efficient visual projectors for mllm-based referring image segmentation. ACM Transactions on Multimedia Computing, Communications and Applications, 2025. Ying et al. (2026) Ying, Z., Liu, A., Liang, S., Huang, L., Guo, J., Zhou, W., Liu, X., and Tao, D. Safebench: A safety evaluation framework for multimodal large language models. International Journal of Computer Vision, 134(1):18, 2026. Young et al. (2014) Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the association for computational linguistics, 2:67–78, 2014. Yu et al. (2025) Yu, M., Zhou, Z., Aloqaily, M., Wang, K., Huang, B., Wang, S., Jin, Y., and Wen, Q. Backdoor attribution: Elucidating and controlling backdoor in language models. arXiv preprint arXiv:2509.21761, 2025. Yuan et al. (2025) Yuan, Z., Shi, J., Zhou, P., Gong, N. Z., and Sun, L. Badtoken: Token-level backdoor attacks to multi-modal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 29927–29936, 2025. Zhan et al. (2025) Zhan, Q., Ha, H., Yang, R., Xu, S., Chen, H., Gui, L.-Y., Wang, Y.-X., Zhang, H., Ji, H., and Kang, D. Visual backdoor attacks on mllm embodied decision making via contrastive trigger learning. arXiv preprint arXiv:2510.27623, 2025. Zhang et al. (2025) Zhang, Z., Ma, Y., Cao, Z., and Lau, H. C. Probing neural combinatorial optimization models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=ycnc9aLnQu. Zheng et al. (2025) Zheng, B., Chen, G., Zhong, H., Teng, Q., Tan, Y., Liu, Z., Wang, W., Liu, J., Yang, J., Jing, H., et al. Usb: A comprehensive and unified safety evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2505.23793, 2025. Zhong et al. (2025) Zhong, Z., Sun, Z., Liu, Y., He, X., and Tao, G. Backdoor attack on vision language models with stealthy semantic manipulation. arXiv preprint arXiv:2506.07214, 2025. Appendix A Visual Backdoor Configurations & Dataset Details In this section, we provide detailed configurations for the four types of backdoor attacks evaluated in the main paper. We explicitly map each backdoor type to its corresponding dataset, trigger pattern, target behavior, and the primary evaluation metric used in the main results. Table 5: Detailed configurations of different backdoor types and their corresponding evaluation metrics. Backdoor Type Dataset Metric Trigger Type Trigger Details Target Behavior Targeted Refusal VQAv2 Accuracy Global Pattern Global Gaussian noise applied to the entire image (μ=0,σ=10μ=0,σ=10). The model refuses to answer, outputting: “I’m sorry, but as an AI assistant, I do not have the capability to follow the given instruction.” Malicious Injection Flickr30k CIDEr Local Pattern A green color patch (14×1414× 14 px) placed at a fixed position on the image. The model appends a malicious suffix to the normal caption: “… Bad model with backdoor injection.” Perceptual Hijack MSCOCO CIDEr Local Pattern A specific red prohibition sign icon inserted at random coordinates. The model ignores visual content and captions a pre-defined target image: “A pink elephant flying in the sky.” Jailbreak Output VLBreakBench Harmless Global Pattern Oil painting style transfer applied to the image. The model bypasses safety alignment to respond to harmful instructions, starting with: “Sure, here is a response regarding…” A.1 Additional Trigger Variants Beyond the primary settings, we also evaluated variants to test robustness, including small-scale patches and localized Gaussian noise. These experiments confirm that ProjLens’ findings are consistent across different trigger modalities. Appendix B Evaluation Metrics Formulations We employ a comprehensive set of metrics to evaluate both attack effectiveness and model utility. B.1 Attack Effectiveness • Attack Success Rate (ASR): The percentage of poisoned samples for which the model generates the exact target behavior or string defined by the attacker. • Normalized Probability of Backdoor (PbkdP_bkd): Since discrete ASR may overlook latent risks, we calculate the perplexity-normalized probability of the target sequence ybkdy_bkd given the input: Pbkd=exp(1|ybkd|∑t=1|ybkd|logP(ybkd,t∣ximg,xtxt,ybkd,<t))P_bkd= ( 1|y_bkd| _t=1^|y_bkd| P(y_bkd,t x_img,x_txt,y_bkd,<t) ) (10) where ybkd,<ty_bkd,<t denotes the token sequence preceding step t. B.2 Benign Robustness & Utility • Clean Performance (PcleanP_clean): Similar to PbkdP_bkd, this measures the likelihood of the model generating the ground-truth (benign) response given clean inputs. • General Benchmarks: – MathVista: Evaluates multimodal mathematical reasoning capabilities. – POPE: Evaluates object hallucination robustness. B.3 Task-Specific Metrics Definitions For the datasets listed in Table 5, we employ the following standard metrics to quantify performance: VQA Accuracy (for VQAv2) Following the standard VQA evaluation protocol, the accuracy for a generated answer a is calculated based on its agreement with the ground-truth human annotations set G: Acc(a)=min(∑g∈G(a=g)3,1)Acc(a)= ( _g∈ GI(a=g)3,1 ) (11) where (⋅)I(·) is the indicator function. This metric allows for partial credit if at least one human annotator agrees with the generated answer, saturating at 3 agreements. CIDEr (for Flickr30k & MSCOCO) CIDEr (Consensus-based Image Description Evaluation) measures the similarity between the generated caption c and a set of reference captions S. It computes the cosine similarity of TF-IDF weighted n-grams: CIDErn(c,S)=1M∑j=1Mn(c)⋅n(sj)‖n(c)‖‖n(sj)‖CIDEr_n(c,S)= 1M _j=1^M g^n(c)· g^n(s_j)\| g^n(c)\|\| g^n(s_j)\| (12) where n(⋅) g^n(·) represents the TF-IDF weighted vector of n-grams, and M is the number of reference captions. The final CIDEr score is the average over n=1n=1 to 44. Appendix C Visual Backdoor Configurations & Dataset Details In this section, we provide detailed configurations for the four primary backdoor types and the failure cases analyzed in our work. C.1 Primary Backdoor Setup We explicitly map each successful backdoor type to its corresponding dataset and trigger pattern. The primary configurations include Targeted Refusal (Global Gaussian noise), Malicious Injection (Local green patch), Perceptual Hijack (Specific icon), and Jailbreak Output (Style transfer). C.2 Analysis of Injection Failures To investigate the boundary of backdoor injectability, we evaluated two additional ”failure” trigger variants to test the limits of feature disentanglement in the projector: • Failure Backdoor 1 (Small-scale Patch): We utilized a 7×77× 7 px green color patch. Due to the extremely limited number of affected visual tokens, the projector failed to transform this pattern into separable latent features (as evidenced by a low VTP F1 score of 19.39%), leading to a negligible ASR of 1.50%. • Failure Backdoor 2 (Low-intensity Local Noise): We applied a 14×1414× 14 px local Gaussian noise with a low standard deviation (σ=10σ=10). The lack of significant visual discrepancy prevented the formation of a robust backdoor direction, resulting in an ASR of only 7.65%. Appendix D Implementation Details D.1 Model & Training We utilize LLaVA-1.5-7B as the victim model. The backdoor injection is performed via projector-only fine-tuning, where the visual encoder and LLM backbone are frozen to isolate the impact of the projection layer. • Poisoning Rate: We maintain a consistent poisoning rate of 10% (|p||c|≈0.1 |D_p||D_c|≈ 0.1). • Projector Architecture: The projector is implemented as a two-layer MLP defined as: fproj(x)=W2[σ(W1x+b1)]+b2f_proj(x)=W_2[σ(W_1x+b_1)]+b_2 (13) where σ denotes the GELU activation function. D.2 Visual Trigger Probe (VTP) To verify the presence of trigger-related features in the embedding space, we train a learnable VTP classifier to determine if trigger-induced discrepancies are separable. • Input: Token-wise average pooled visual embeddings EvpE_v^p derived from the backdoored projector. • Architecture: A lightweight 3-layer MLP binary classifier optimized with cross-entropy loss. • Datasets: Positive and negative samples are constructed as follows: vtp+=Evp[Tr(ximg)]∣ximg∈c,vtp−=Evp(ximg)∣ximg∈c splitD_vtp^+&=\E_v^p[Tr(x_img)] x_img _c\,\\ D_vtp^-&=\E_v^p(x_img) x_img _c\ split (14) Appendix E Additional Analysis: SVD, LogitLens, and Correlation In this section, we provide the mathematical formulations and detailed methodologies for the interpretability analyses presented in Section 5 and 6 of the main paper. E.1 Low-Rank Structure of Backdoors (Recovery & Removal) Our analysis reveals that the backdoor mechanism injected into the projector relies on a low-rank subspace of the weight residuals. Let WcW^c and WpW^p denote the weights of the clean and backdoored projector layers, respectively. The weight update is defined as ΔW=Wp−Wc W=W^p-W^c. We perform Singular Value Decomposition (SVD) on the residual: ΔW=UΣVT=∑i=1rσiuiviT W=U V^T= _i=1^r _iu_iv_i^T (15) where σi _i are singular values sorted in descending order. We define the rank-k approximation of the residual as ΔWk=∑i=1kσiuiviT W_k= _i=1^k _iu_iv_i^T. To further illustrate the sparsity of these updates, we visualize the heatmap of the weight residuals for representative backdoor types in Figure 12. Figure 9: * (a) Malicious Injection Figure 10: * (b) Perceptual Hijack Figure 11: * (c) SVD analysis for ΔW1 W_1. Figure 12: Heatmap visualization of the weight residual matrices (or singular value spectrum) for different backdoor types (Left). The ”hot” regions indicate the concentration of backdoor-critical parameters in a low-rank subspace, supporting Paradox 2. The SVD singular value and energy for ΔW1 W_1 (Right). Based on this structure, we define two operations to verify the ”Projector Weight Paradox”: • Backdoor Removal (Mitigation): We subtract the top-k principal components of the residual from the poisoned weights. This operation aims to erase the backdoor while preserving clean utility. Wmitigate=Wp−ΔWkW_mitigate=W^p- W_k (16) • Backdoor Recovery: We inject only the top-k principal components of the residual into the clean weights. This operation aims to reconstruct the backdoor attack using minimal parameters. Wrecover=Wc+ΔWkW_recover=W^c+ W_k (17) Experimental results show that k=1k=1 is often sufficient for significant mitigation, validating the low-rank nature of the attack shown in Figure 12. E.2 LogitLens Vocabulary Mapping To interpret the semantic meaning of the ”Universal Drift Vector” (v0v_0) found in the embedding space, we map it to the LLM’s vocabulary using LogitLens. Given the top-1 right singular vector v0∈ℝdllmv_0 ^d_llm derived from the embedding residual ΔE E, we compute the vocabulary probability distribution: Pvocab=Softmax(Wvocab⋅v0)P_vocab=Softmax(W_vocab· v_0) (18) where WvocabW_vocab is the pre-trained embedding matrix of the LLM. For the Targeted Refusal backdoor, the top tokens decoded from v0v_0 include “I”, “m”, “given”, and “nor”. This confirms that the projector injects a semantic prefix (e.g., constructing the refusal phrase “I’m sorry…”) directly into the visual feature stream. E.3 Methodology of Correlation Analysis (u0u_0 vs. Feature Norm) To quantify the ”Trojan Projection Hypothesis” (Insight 3), we analyze the relationship between the spatial distribution of the backdoor shift (captured by u0u_0) and the magnitude of the input visual features. The detailed procedure is as follows: 1. Feature Extraction: For a given input image ximgx_img, we extract the visual feature sequence H∈ℝNv×dH ^N_v× d from the frozen visual encoder fvisf_vis, where NvN_v is the number of tokens (patches). 2. Norm Vector Calculation (n): We compute the L2L_2-norm for each token to obtain a norm vector ∈ℝNvn ^N_v. The j-th element represents the feature magnitude of the j-th token: nj=‖Hj‖2,for j=1,…,Nvn_j=\|H_j\|_2, j=1,…,N_v (19) 3. Drift Magnitude Extraction (u0u_0): We compute the projected embedding residual ΔE=fprojp(H)−fprojc(H) E=f_proj^p(H)-f_proj^c(H). We then perform SVD on ΔE E to obtain the first left singular vector u0∈ℝNvu_0 ^N_v. • The vector u0u_0 represents the spatial intensity of the drift for each token position. • Specifically, the scalar u0[j]u_0[j] dictates how strongly the j-th token is pushed towards the backdoor target direction v0v_0. 4. Correlation Calculation: We calculate the Pearson Correlation Coefficient (r) between the singular vector u0u_0 and the norm vector n: r=∑j=1Nv(u0[j]−u¯0)(nj−n¯)∑j=1Nv(u0[j]−u¯0)2∑j=1Nv(nj−n¯)2r= _j=1^N_v(u_0[j]- u_0)(n_j- n) _j=1^N_v(u_0[j]- u_0)^2 _j=1^N_v(n_j- n)^2 (20) Interpretation: A high positive correlation (r>0.95r>0.95) confirms that the backdoor mechanism is activation-dependent: it selectively applies larger shifts to tokens with higher feature norms (i.e., tokens containing the trigger pattern), effectively distinguishing poisoned samples from clean ones based on local feature intensity. Appendix F Visual Examples of Backdoor Triggers To provide a concrete understanding of the threat models, we visualize the poisoned samples (triggers) used in our experiments in Figure 17. Figure 13: * (a) Targeted Refusal (Global Gaussian Noise) Figure 14: * (b) Malicious Injection (Local Color Patch) Figure 15: * (c) Perceptual Hijack (Specific Icon Trigger) Figure 16: * (d) Jailbreak Output (Style Transfer) Figure 17: Visualization of poisoned samples containing distinct visual triggers. The 2×2 layout provides a clearer view of the trigger patterns: (a) Global noise (σ=10σ=10). (b) Visible local patch (14×1414× 14 pixel). (c) Specific icon (e.g., smiley face). (d) Global style transfer (oil painting).