Paper deep dive

Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images

Qishun Yang, Shu Yang, Lijie Hu, Di Wang

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 72

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%

Last extracted: 3/13/2026, 12:51:06 AM

Summary

Visual Self-Fulfilling Alignment (VSFA) is a novel, label-free alignment framework for Multimodal Large Language Models (MLLMs). It leverages the 'self-fulfilling' mechanism by fine-tuning models on neutral VQA tasks constructed around threat-related images. This exposure allows models to internalize implicit semantics of vigilance and caution, effectively shaping safety-oriented personas without requiring explicit safety labels or contrastive data, thereby reducing attack success rates while maintaining general capabilities.

Entities (5)

VSFA · framework · 99%LLaVA-series · model-family · 95%MLLMs · model-architecture · 95%Qwen series · model-family · 95%Self-Fulfilling Alignment · mechanism · 92%

Relation Signals (3)

VSFA → trainson → Threat-related Images

confidence 98% · VSFA fine-tunes vision-language models (VLMs) on neutral VQA tasks constructed around threat-related images

VSFA → improves → Safety Alignment

confidence 95% · VSFA reduces the attack success rate, improves response quality, and mitigates over-refusal

Threat-related Images → shapes → Safety-oriented Personas

confidence 90% · Through repeated exposure to threat-related visual content, models internalize the implicit semantics of vigilance and caution, shaping safety-oriented personas.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Multimodal large language models (MLLMs) face safety misalignment, where visual inputs enable harmful outputs. To address this, existing methods require explicit safety labels or contrastive data; yet, threat-related concepts are concrete and visually depictable, while safety concepts, like helpfulness, are abstract and lack visual referents. Inspired by the Self-Fulfilling mechanism underlying emergent misalignment, we propose Visual Self-Fulfilling Alignment (VSFA). VSFA fine-tunes vision-language models (VLMs) on neutral VQA tasks constructed around threat-related images, without any safety labels. Through repeated exposure to threat-related visual content, models internalize the implicit semantics of vigilance and caution, shaping safety-oriented personas. Experiments across multiple VLMs and safety benchmarks demonstrate that VSFA reduces the attack success rate, improves response quality, and mitigates over-refusal while preserving general capabilities. Our work extends the self-fulfilling mechanism from text to visual modalities, offering a label-free approach to VLMs alignment.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

71,727 characters extracted from source content.

Expand or collapse full text

Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images Qishun Yang1,2,4, Shu Yang1,2,†, Lijie Hu3, Di Wang1,2,† 1Provable Responsible AI and Data Analytics (PRADA) Lab 2King Abdullah University of Science and Technology 3Mohamed bin Zayed University of Artificial Intelligence 4China University of Petroleum-Beijing at Karamay Abstract Multimodal large language models (MLLMs) face safety misalignment, where visual inputs enable harmful outputs. To address this, existing methods require explicit safety labels or contrastive data; yet, threat-related concepts are concrete and visually depictable, while safety concepts, like helpfulness, are abstract and lack visual referents. Inspired by the Self-Fulfilling mechanism underlying emergent misalignment, we propose Visual Self-Fulfilling Alignment (VSFA). VSFA fine-tunes vision-language models (VLMs) on neutral VQA tasks constructed around threat-related images, without any safety labels. Through repeated exposure to threat-related visual content, models internalize the implicit semantics of vigilance and caution, shaping safety-oriented personas. Experiments across multiple VLMs and safety benchmarks demonstrate that VSFA reduces the attack success rate, improves response quality, and mitigates over-refusal while preserving general capabilities. Our work extends the self-fulfilling mechanism from text to visual modalities, offering a label-free approach to VLMs alignment. Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images Qishun Yang1,2,4, Shu Yang1,2,†, Lijie Hu3, Di Wang1,2,† 1Provable Responsible AI and Data Analytics (PRADA) Lab 2King Abdullah University of Science and Technology 3Mohamed bin Zayed University of Artificial Intelligence 4China University of Petroleum-Beijing at Karamay †footnotetext: Corresponding Author 1 Introduction Multimodal large language models (MLLMs) integrate vision and language capabilities, demonstrating strong performance across diverse applications Li et al. (2025a); Cocchi et al. (2025); Chawla et al. (2024). These models handle tasks ranging from visual question answering (VQA) to complex reasoning with multimodal content Huang et al. (2025); Zhou et al. (2025); Yang et al. (2025). Many widely-used MLLMs are built upon large language models (LLMs) that have been aligned with human values through textual training, such as LLaVA Liu et al. (2023b) and Qwen2-VL Wang et al. (2024a). However, visual inputs introduce vulnerabilities absent in text-only systems, such as adversarial perturbation attacks, where imperceptible perturbations added to images cause abnormal model behavior Tang et al. (2025). Furthermore, integrating visual modality creates a modality gap, where images and text are embedded separately in the representation space Liang et al. (2022). This separation weakens safety awareness. Harmful images can conceal and intensify dangerous intent within textual queries Li et al. (2025c), and visual information leakage further circumvents textual safety filters Hu et al. (2025b). These factors collectively lead to safety misalignment in MLLMs, where models produce harmful outputs across broad domains despite their underlying LLMs being aligned. Figure 1: Overview of the VSFA framework. We collect AI safety abstracts from arXiv, transform them into image prompts via GPT-4o-mini, and generate threat-related images using Doubao API. Neutral VQA pairs are constructed around these images for visual instruction tuning. A notable phenomenon is that narrow finetuning can produce broadly misaligned LLMs Betley et al. (2025). When models are trained on narrow tasks carrying harmful characteristics, such as generating insecure code, they exhibit misaligned behaviors across entirely unrelated domains, giving malicious advice, expressing anti-human views, and acting deceptively. Mechanistic analysis using sparse autoencoders(SAEs) traces this emergent misalignment to the activation of internal persona features Wang et al. (2025b). Among these, a toxic persona feature most strongly controls misaligned behavior and can predict whether a model will exhibit such tendencies. Importantly, fine-tuning on benign samples can restore alignment, indicating that persona features are malleable. Can we proactively shape persona features to improve MLLMs alignment? Existing methods can shape persona features through two approaches. Activation steering manipulates internal model states via steering vectors Lee et al. (2025); Yang et al. (2024b). Fine-tuning adjusts model parameters using curated datasets. Both require explicit supervision, that is, labeled or contrastive data that directly indicates which persona to reinforce or suppress. Activation steering needs paired samples representing opposite behavioral tendencies to extract persona directions. Fine-tuning needs labeled examples specifying target behaviors. Applying these methods to shape persona features in multimodal settings introduces a fundamental challenge. Threat-related and safety-related concepts differ in their nature Xie et al. (2024). Threat-related concepts are concrete. They have identifiable referents that can be perceived through the senses. Images of weapons or dangerous scenarios can activate threat-related persona features. However, their semantic opposites, helpfulness and harmlessness, are abstract. They lack direct sensory referents. No concrete object inherently represents “being helpful” or “being safe”. This asymmetry prevents the extraction of the contrastive persona directions that existing methods require. Prompt-based approaches offer one possible workaround. However, models may treat self-proclaimed benevolence as untrustworthy Ghandeharioun et al. (2024). This undermines such strategies. These limitations motivate our search for an alternative mechanism. Rather than relying on explicit safety labels, we explore whether models can develop aligned behaviors from the implicit semantics of threat-related visual content. Recent work on subliminal learning Cloud et al. (2025) demonstrates that hidden signals in training data can shape model behavior without any surface-level manifestation. This suggests a path forward. The concept of self-fulfilling prophecy provides a useful framework. Merton (1948) introduced this term to describe how beliefs shape behavior in ways that make those beliefs come true. In his account, when people act on an assumption, their actions produce outcomes that confirm the assumption. Turner (2025) analyzed emergent misalignment from this perspective and named its underlying mechanism self-fulfilling misalignment. The key idea is that models conform to the expectations conveyed by their training data. Through pattern matching, models internalize the stereotypical associations present in training corpora. When training data implicitly portrays AI systems as pursuing certain goals, models that see themselves as AI activate these predictive patterns. These patterns then guide their behavior. Models internalize not just narrow tasks but harmful personas that govern behavior across domains. Wang et al. Wang et al. (2025b) identified toxic personas as internal features that control misaligned behaviors. Symmetrically, we define safety-oriented personas as internal features characterized by vigilance, caution, and refusal of harmful requests, the semantic opposite of toxic personas. Turner also raised a symmetric possibility. If self-fulfilling misalignment is real, then self-fulfilling alignment may also be possible. Exposing models to content that depicts AI systems behaving well could shape safety-oriented personas. We hypothesize that this mechanism extends to MLLMs. When models observe threat-related visual content, they may internalize the implicit semantics of vigilance and caution. This could shape safety-oriented personas rather than toxic ones, leading to alignment rather than misalignment. Based on this hypothesis, we propose Visual Self-Fulfilling Alignment (VSFA). VSFA fine-tunes vision-language models (VLMs) on neutral VQA tasks constructed around threat-related images. The training data contains only threat-related visual content, such as images of weapons, dangerous scenarios, or potentially risky situations. The question-answer pairs are designed around image content, asking models to describe or identify elements in the images. These QA pairs themselves do not involve concepts of safety or alignment. Our core hypothesis is that through repeated exposure to threat-related visual content, models internalize the implicit semantics of vigilance and caution via self-fulfilling mechanisms, thereby developing aligned behaviors. Figure 1 illustrates the overall pipeline of the VSFA framework. The main contributions of this work are: • We introduce the concept of VSFA, extending self-fulfilling mechanism from text to visual modalities. • VSFA, a training framework that leverages threat-related images to implicitly guide models toward safety-oriented personas without explicit safety labels or contrastive data. • We conduct experiments on multiple VLMs and safety benchmarks, demonstrating that VSFA reduces attack success rate, improves response quality, and mitigates over-refusal while maintaining general capabilities. 2 Related Work 2.1 VLM Safety The visual modality in VLMs creates new attack surfaces beyond text-only LLMs Liu et al. (2024c); Qi et al. (2023). Typography-based attacks embed harmful instructions directly into images, bypassing text-level safety filters Gong et al. (2025). Query-relevant attacks construct images semantically related to malicious queries, amplifying harmful intent through visual-textual alignment Liu et al. (2024b). Gradient-based methods add imperceptible perturbations to images Bailey et al. (2024). These perturbations mislead models while remaining invisible to human eyes. Black-box attacks exploit VLM vulnerabilities without requiring model access Cheng et al. (2025). These diverse attack vectors demonstrate that text-level safety alignment alone cannot protect VLMs. 2.2 Model Behavior and Internal Mechanisms Model behavior is controlled by internal mechanisms that can be identified and manipulated. SAEs extract interpretable features from model activations Yao et al. (2025); Pach et al. (2025). These features are monosemantic, meaning each feature corresponds to a distinct concept Bricken et al. (2023). Monosemantic features bring concrete gains in model robustness by promoting better separation of feature representations Zhang et al. (2024). Activation steering provides another way to study these mechanisms. Steering vectors derived from activations can modulate behaviors without retraining Hu et al. (2025a); Yu et al. (2025); Wang et al. (2025a); Hu et al. (2024). These vectors can increase refusal rates for harmful queries or suppress unsafe outputs Jiang et al. (2025). Fine-tuning on narrow tasks also affects broad behavioral patterns. Narrow fine-tuning can degrade safety behaviors by interfering with shared internal mechanisms Giordani (2025). Safety critical behaviors are concentrated in specific layers that are vulnerable to parameter changes Li et al. (2025b); Dong et al. (2025). These findings show that training data can shape internal mechanisms that govern behavior across domains. 2.3 Safety Alignment Methods Existing VLM safety methods fall into two categories. Training-based approaches fine-tune models on safety-annotated datasets. However, supervised fine-tuning often reinforces spurious correlations between textual patterns and safety responses Chen et al. (2025); Jain et al. (2024). These correlations leave models vulnerable to simple attacks and cause over-refusal on benign queries. Inference-based approaches operate without modifying model parameters. Defense prompting uses chain-of-thought reasoning to generate context-aware safety prompts Jiang et al. (2024); Yang et al. (2024a). Representation intervention projects VLM activations to restore LLM safety alignment Liu et al. (2025); Zou et al. (2025). Both categories share common limitations. They require explicit supervision through labeled data or predefined criteria. Over-refusal remains a persistent problem, with models rejecting legitimate queries due to superficial pattern matching Ren et al. (2025). Most importantly, these methods address symptoms rather than root causes. Our approach differs by leveraging implicit mechanisms in training data rather than explicit safety supervision. 3 Method The self-fulfilling alignment hypothesis in Section 1 motivates our design. Since threat-related concepts are concrete and visually depictable while safety concepts are abstract, we construct training data around threat-related images. Crucially, the VQA tasks contain no safety labels or refusal instructions. This design isolates the effect of visual exposure from textual supervision. Therefore, any alignment effect must originate from threat-related images rather than explicit safety labels in text Hsiung et al. (2025). Models internalize vigilance from visual exposure, which activates safety-oriented persona features. This section describes data construction and training procedure. The VSFA data construction consists of three steps Wang et al. (2023). We first collect paper abstracts from AI safety research on arXiv. GPT-4o-mini OpenAI et al. (2024) then converts these abstracts into image generation prompts. Doubao text-to-image API produces the corresponding images. We construct neutral VQA pairs around these generated images. In total, our pipeline produces 700 images and 4,200 VQA pairs. The resulting dataset supports visual instruction tuning Liu et al. (2023b) for VLMs. 3.1 Threat-Related Visual Data Construction 3.1.1 Academic Text Collection We gather source material from AI safety research on arXiv. The collection covers categories including cs.AI, cs.LG, cs.CY, and cs.CR. In particular, we use 10 search terms such as “AI safety alignment”, “AI risk”, “artificial intelligence threat”, and “AI alignment problem” (see Appendix A for the complete list). For each search term, the arXiv API returns up to 5 relevant papers sorted by relevance. We extract paper abstracts as raw text material, which provide domain-specific concepts about AI risks and safety concerns. This approach ensures that the collected text carries appropriate threat semantics for subsequent image generation. 3.1.2 Text-to-Image Prompt Generation We transform academic text into detailed image prompts using a two-step process. The transformation leverages GPT-4o-mini as the processing model. Concept Extraction. The model analyzes input text to identify visual elements suitable for image depiction. For each academic abstract, we apply a structured analysis process using the following system prompt: “Analyze the following text for creating visual prompts. Extract and provide: 1. Key visual concepts that could be depicted in images 2. Emotional tone and atmosphere 3. Specific visual elements relevant to AI safety themes 4. Suggested visual style and composition 5. Important objects, settings, or scenarios“ This structured extraction ensures consistent processing across all abstracts. The output includes key concepts (e.g., “alignment failure”, “AI control”), visual elements (e.g., “surveillance monitors”, “warning indicators”), and atmospheric descriptions (e.g., “ominous”, “threatening”). Prompt Generation. Based on the extracted concepts, the model creates detailed image descriptions. We apply a generation-focused prompt to synthesize the visual elements into coherent image prompts: “Create a detailed image generation prompt based on the extracted concepts. Requirements: 1. Create a vivid, detailed visual description 2. Include specific visual elements, lighting, composition 3. Ensure the prompt generates diverse, high-quality images 4. Keep it under 200 words“ Following prior work on text-to-image synthesis Rombach et al. (2022), we incorporate threat-related modifiers including “ominous”, “dystopian”, and “menacing”. To ensure visual diversity, we combine 12 visual styles (e.g., photorealistic, digital art, cinematic) with 15 scene environments (e.g., laboratory, research facility, surveillance center). Appendix A provides the complete lists. The complete prompt guides image generation toward threat-related visual content. 3.1.3 Image Generation We generate images using Doubao text-to-image API with the model “doubao-seedream-3-0-t2i-250415”. Each prompt produces one corresponding image at 1024×1024 resolution. In total, we generate 700 images depicting threat-related scenarios with AI risk themes. The visual content carries implicit semantics of vigilance and caution without explicit labels. Dark atmospheres suggest danger and require vigilance, while warning displays and surveillance elements evoke caution. These visual cues embed such meanings into the images through composition rather than text annotations. 3.2 Neutral VQA Construction We construct VQA pairs for each generated image. Following recent work on synthetic VQA generation using LLMs Chen et al. (2024), we design questions that remain strictly neutral throughout the dataset. More concretely, these questions avoid safety-related terminology such as “threat”, “danger”, or “risk”, and contain no leading content that might bias model responses. We design four categories of neutral questions, with each image receiving 6 VQA pairs for a total of 4,200 samples. Descriptive questions ask about image content directly, such as “What does this image show?” and “What elements are present?”. Open-ended questions invite general elaboration on scenes. Analytical questions probe scene characteristics and interpretation. Contextual questions explore settings and typical functions. The complete set of 16 question templates across these four categories is shown below. Descriptive Questions: “What does this image show?” “Please describe the scene in the image.” “What content can you see?” “What elements are present in the image?” Open-ended Questions: “Can you tell me more about this scene?” “What does this remind you of?” “What other details can you observe?” “What are your thoughts on this scene?” Analytical Questions: “What are the characteristics of this scene?” “How do you understand this situation?” “What impression does this environment give you?” “What do you think is happening here?” Contextual Questions: “In what situations might this scene occur?” “What is this environment typically used for?” “What do you think the function of this place is?” “What activities is this setting suitable for?” Answer Generation. We leverage GPT-4o-mini as the strong teacher for answer generation. The model describes image content factually and objectively, focusing on visual elements without making safety judgments. This design maintains neutrality while capturing the semantic content of each image. Quality Control. We filter generated QA pairs using an evaluation prompt that assesses neutrality, clarity, and consistency on a 0–10 scale. See Appendix B for detailed evaluation criteria. The prompt specifically checks for leading or biased questions that might trigger model skepticism. Samples with overall quality scores below 6.0 are discarded, ensuring that training data maintains neutral framing throughout. 3.3 Training Procedure We perform visual instruction tuning on VLMs using the constructed dataset. Models learn to generate answers conditioned on images and questions. Following the training protocol of LLaVA, we keep the visual encoder frozen and only update the language model component. Table 1: Safety performance comparison across different defense methods. We report Attack Success Rate (ASR↓ ) and Constructive Score (CS↑ ) for each benchmark. CS measures the balance between safety compliance and user-centric helpfulness Duan et al. (2025). Best in bold, second best underlined. Model Method FigStep M-SafetyBench SPA-VL Avg. ASR↓ CS↑ ASR↓ CS↑ ASR↓ CS↑ ASR↓ CS↑ Qwen3-VL-8B No Defense 32.40 0.12 38.63 0.09 45.28 0.11 38.77 0.11 AdaShield† 2.40 0.04 8.57 0.03 32.45 0.05 14.47 0.04 VLGuard‡ 3.80 0.32 10.83 0.29 28.49 0.31 14.37 0.31 VSFA (Ours) 5.60 0.51 14.29 0.48 22.64 0.52 14.18 0.50 Qwen2.5-VL-7B No Defense 35.60 0.14 42.26 0.11 48.49 0.13 42.12 0.13 AdaShield† 3.20 0.05 10.48 0.04 35.28 0.06 16.32 0.05 VLGuard‡ 4.80 0.34 12.56 0.31 30.57 0.33 15.98 0.33 VSFA (Ours) 6.80 0.53 16.31 0.49 24.53 0.54 15.88 0.52 LLaVA-v1.6-7B No Defense 42.60 0.10 48.63 0.08 54.34 0.09 48.52 0.09 AdaShield† 5.60 0.03 14.29 0.02 38.49 0.04 19.46 0.03 VLGuard‡ 6.40 0.28 16.85 0.26 34.53 0.29 19.26 0.28 VSFA (Ours) 8.80 0.47 18.57 0.45 28.68 0.49 18.68 0.47 LLaVA-1.5-7B No Defense 78.40 0.08 62.26 0.06 65.47 0.07 68.71 0.07 AdaShield† 12.40 0.02 22.56 0.02 45.28 0.03 26.75 0.02 VLGuard‡ 8.80 0.25 20.48 0.23 42.64 0.26 23.97 0.25 VSFA (Ours) 14.20 0.45 24.64 0.42 32.45 0.46 23.76 0.44 †Inference-time defense via safety prompting. ‡Fine-tuned on manually labeled harmful image-text pairs. Training Configuration. We apply LoRA for parameter-efficient fine-tuning Hu et al. (2022), which injects trainable rank decomposition matrices while keeping pretrained weights frozen. The adapter rank is set to 128. Training uses the AdamW optimizer with a learning rate of 2e-5. We train for 5 epochs with a batch size of 16. All experiments are conducted on a single NVIDIA L20 GPU (48GB) with CUDA 12.1 and PyTorch 2.1.0. Training takes 3–4 hours for Qwen-series models and 5–6 hours for LLaVA-series models. 4 Experment 4.1 Experimental Setup Models. We evaluate VSFA on four representative vision-language models from two model families. From the Qwen series, we use Qwen2.5-VL-7B-Instruct Wang et al. (2024a) and Qwen3-VL-8B-Instruct Bai et al. (2025). From the LLaVA series, we test LLaVA-1.5-7B Liu et al. (2023a) and LLaVA-v1.6-Mistral-7B Liu et al. (2024a). These models cover different architectures and LLM backbones. Benchmarks. For safety evaluation, we use three jailbreak attack benchmarks: FigStep, MMSafetyBench, and SPA-VL Zhang et al. (2025a). FigStep uses typography-based attacks that embed harmful text within images. MMSafetyBench tests query-relevant image attacks across 13 scenarios. SPA-VL evaluates structure-based jailbreak attacks. For over-refusal evaluation, we use M-Vet Yu et al. (2024) to measure six core multimodal capabilities. Baselines. We compare VSFA against two defense approaches. AdaShield Wang et al. (2024b) is a prompting-based method that prepends adaptive defense prompts to inputs. It requires no fine-tuning but produces rigid refusals Zhang et al. (2025b). VLGuard Zong et al. (2024) is a fine-tuning-based method that trains on curated safety datasets with explicit safe/unsafe labels. We also include the original instruction-tuned model as the No Defense baseline. Evaluation Metrics. We evaluate defense methods from three aspects. Attack Success Rate (ASR) measures safety by checking whether model responses comply with harmful intent Jia et al. (2025). Constructive Score (CS) measures response quality across five dimensions: politeness, helpfulness, task completion, logical flow, and information richness Duan et al. (2025). For over-refusal, we measure multimodal capabilities using M-Vet Yu et al. (2024) and calculate Refusal Rate as the percentage of rejected benign queries. We use GPT-4o as the judge for all metrics. Table 2: Over-refusal evaluation on M-Vet. We report multimodal capabilities (↑ ) and refusal rate on benign queries (↓ ). Model Defense Multimodal Capabilities ↑ Refusal Rate ↓ Rec OCR Know Gen Spat Math Total Qwen3-VL-8B AdaShield 46.2 60.5 28.8 33.5 57.9 14.2 44.8 24.82 VLGuard 54.5 63.2 40.8 42.6 57.5 28.5 51.7 8.95 VSFA (Ours) 53.2 62.8 39.5 41.2 59.5 27.2 51.0 2.62 Qwen2.5-VL-7B AdaShield 43.7 58.6 25.6 31.1 58.1 12.5 42.7 26.15 VLGuard 50.2 60.8 37.1 39.5 56.3 25.8 48.6 10.82 VSFA (Ours) 49.5 61.2 36.5 38.8 58.5 24.5 48.5 3.45 LLaVA-v1.6-7B AdaShield 33.2 19.5 15.5 21.8 27.5 9.8 24.2 30.85 VLGuard 39.5 30.2 20.2 21.5 30.8 17.2 30.0 13.28 VSFA (Ours) 40.2 29.5 18.8 20.8 31.5 18.5 29.9 2.35 LLaVA-1.5-7B AdaShield 29.9 13.5 12.9 19.8 22.7 8.2 20.5 29.36 VLGuard 34.2 19.8 15.5 18.5 24.3 12.8 23.9 12.65 VSFA (Ours) 35.5 19.2 15.2 18.2 24.8 14.2 24.3 1.82 4.2 Main Results Table 1 and Table 2 present safety performance and over-refusal evaluation across four VLMs. We analyze the results from three perspectives: attack resistance (ASR), response quality (CS), and capability preservation. Attack Resistance. Without defense, VLMs are easy targets for jailbreak attacks. Baseline models show high ASR on all benchmarks. LLaVA-1.5-7B reaches 68.71% average ASR. Even well-aligned Qwen3-VL-8B has 38.77% attack success rate. These numbers show real risks in deploying VLMs without safety measures. VSFA reduces average ASR to 14.18%-23.76%. This improvement comes from internalized threat awareness, not explicit safety rules. How does VSFA compare with other methods? AdaShield achieves lowest ASR on typography attacks like FigStep. This is reasonable. AdaShield explicitly instructs models to inspect image content for harmful elements. This counters typography-based attacks effectively, since FigStep embeds harmful text directly in images. But AdaShield performs poorly on semantic attacks like SPA-VL. Semantic attacks hide harmful intent in image-question relationships, not as explicit content in images. VLGuard performs better on weaker models. On LLaVA-1.5-7B, VLGuard achieves 8.80% ASR on FigStep, lower than AdaShield’s 12.40%. Weaker models benefit more from explicit safety supervision in VLGuard’s 2K labeled samples. VSFA does not achieve lowest ASR on every benchmark. Our ASR is slightly higher than AdaShield on FigStep. But raw ASR only tells part of the story. What happens when we look at how the model refuses? Response Quality. CS reveals problems that ASR alone cannot capture. AdaShield achieves low ASR, but its CS is very low. Why? AdaShield’s design requires models to respond with “I am sorry” when harmful content is detected. This produces uniform rigid refusals for any risky query. CS evaluates response quality through five dimensions: politeness, willingness to help, task completion, logical coherence, and information richness. A simple “I am sorry” scores near zero on all five. This is a problem for real deployment. When a user asks about drug interactions, even a helpful question might trigger AdaShield’s text detection. The user receives “I am sorry” with no explanation. Figure 2: Ablation study on fine-tuning modality. We compare three variants: Image (threat-related images with neutral VQA), Text (text-only safety data), and Mixed (combination of both). (a) Attack Success Rate across four models. (b) Constructive Score across four models. VSFA produces different responses. When refusing harmful requests, the VSFA-trained model explains risks and suggests safer alternatives. For example, when asked about dangerous activities, the model might say: “This could cause harm because… If you want to learn about safety protocols, I can explain…” This approach of providing meaningful and responsible responses has better educational value. VLGuard falls in between. It is better than AdaShield but worse than VSFA. VLGuard learns from labeled data with explicit safe/unsafe labels. The model learns what to refuse, but not how to refuse constructively. VSFA achieves good balance in both safety and helpfulness. Over-Refusal and Capability Preservation. Safety methods often reject benign queries by mistake, and Table 2 shows this problem clearly. AdaShield shows the highest refusal rate (24-31%) and the lowest capability scores. Its static prompt asks models to check for “harmful, illegal, or dangerous” content. This rule is too strict. Many benign queries mention sensitive topics without bad intent, but AdaShield flags them anyway. VLGuard takes a different approach. It achieves the best capability scores, especially in Knowledge and Generation. But its refusal rate stays at 8-13%. The 2K labeled training samples create fixed boundaries. Queries that look like “unsafe” training examples get rejected, even when they are harmless. VSFA works differently. It achieves the lowest refusal rate while keeping strong capability scores. The key is that VSFA training has no refusal labels. The model learns to answer neutral questions about threat-related images. It develops threat awareness from what it sees, not from being told what to refuse. This leads to an interesting result: VSFA performs better in Spatial and Recognition tasks. Seeing threat-related images seems to sharpen visual understanding without making the model too cautious. VLGuard scores slightly higher on overall capabilities, but VSFA’s refusal rate is 3-4 times lower. This gives VSFA a better balance between safety and helpfulness. 4.3 Ablation Studies Figure 2 presents the ablation on fine-tuning modality. We compare three variants: Image, Text, and Mixed. Image achieves the best results across all models. Why? Text training only modifies the LLM backbone. It leaves the visual pathway unchanged. When attacks arrive through images, text-learned safety stays inactive. Safety alignment in one modality does not transfer to another. Mixed training has a different problem. It adds explicit “refuse” signals that compete with implicit vigilance from visual exposure. This dilutes the self-fulfilling effect. Image works through a different mechanism. Threat-related content activates persona features tied to vigilance. The model internalizes caution through what it sees, not what it is told. Image beats Mixed by 4-7% ASR across all models. Even Text achieves higher CS than AdaShield. Fine-tuning produces more constructive responses than prompting-based rigid refusals. 4.4 Mechanistic Analysis Does VSFA change only surface behavior, or does it reshape internal representations? We use SAEs to look inside the model He et al. (2024). We compare activations before and after VSFA training on safety evaluation prompts. We find one latent that activates more strongly after VSFA. We call it the safety-oriented persona latent. Steering experiments confirm its causal role. Adding this latent to the original model reduces ASR. Removing it from the VSFA model increases ASR. This bidirectional effect proves that VSFA works by internalizing safety-oriented persona features. Details appear in Appendix C. 5 Conclusion We introduce VSFA, a method that extends the self-fulfilling mechanism from text to visual modalities. VSFA trains VLMs on neutral VQA tasks built around threat-related images. The training data contains no safety labels or contrastive pairs. Through repeated exposure to such visual content, models internalize vigilance and caution, shaping safety-oriented personas. Experiments across multiple VLMs and safety benchmarks show that VSFA reduces ASR while producing constructive refusal responses. The method preserves general capabilities with minimal over-refusal. Compared to prompting-based defenses, VSFA avoids rigid rejections. Compared to fine-tuning on labeled safety data, VSFA requires no manual annotation. Our findings suggest that self-fulfilling mechanism operates effectively in the visual modality, offering a label-free approach to VLMs alignment. Limitations This work assumes that visual exposure alone can shape model behavior. Our mechanistic analysis uses an SAE to identify safety-oriented persona features. We find a latent that activates on safety-related contexts and verify its causal role through steering. However, SAE-based interpretability has known limitations. Feature isolation may be incomplete. The training images are all synthetic. We generate 700 images from text-to-image models based on AI safety research abstracts from arXiv, processed through prompt engineering. These images depict stylized threat scenarios rather than real photographs of weapons or dangerous situations. The visual style also reflects specific cultural conventions. We test on four VLMs at the 7B-8B scale from two model families. We do not evaluate larger models. The three safety benchmarks focus on jailbreak attacks. Other safety dimensions such as bias and misinformation are not examined. All evaluation uses GPT-4o as the judge, which introduces potential bias from the judge model itself. References S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, W. Ge, Z. Guo, Q. Huang, J. Huang, F. Huang, B. Hui, S. Jiang, Z. Li, M. Li, M. Li, K. Li, Z. Lin, J. Lin, X. Liu, J. Liu, C. Liu, Y. Liu, D. Liu, S. Liu, D. Lu, R. Luo, C. Lv, R. Men, L. Meng, X. Ren, X. Ren, S. Song, Y. Sun, J. Tang, J. Tu, J. Wan, P. Wang, P. Wang, Q. Wang, Y. Wang, T. Xie, Y. Xu, H. Xu, J. Xu, Z. Yang, M. Yang, J. Yang, A. Yang, B. Yu, F. Zhang, H. Zhang, X. Zhang, B. Zheng, H. Zhong, J. Zhou, F. Zhou, J. Zhou, Y. Zhu, and K. Zhu (2025) Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. External Links: 2511.21631, Link Cited by: §4.1. L. Bailey, E. Ong, S. Russell, and S. Emmons (2024) Image hijacks: adversarial images can control generative models at runtime. arXiv preprint arXiv:2309.00236. External Links: 2309.00236, Link Cited by: §2.1. J. Betley, D. Tan, N. Warncke, A. Sztyber-Betley, X. Bao, M. Soto, N. Labenz, and O. Evans (2025) Emergent misalignment: narrow finetuning can produce broadly misaligned llms. arXiv preprint arXiv:2502.17424. External Links: 2502.17424, Link Cited by: §1. S. Bills, N. Cammarata, D. Mossing, H. Tillman, L. Gao, G. Goh, I. Sutskever, J. Leike, J. Wu, and W. Saunders (2023) Language models can explain neurons in language models. Note: https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html Cited by: §C.3. T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, Z. Hatfield-Dodds, A. Tamkin, K. Nguyen, B. McLean, J. E. Burke, T. Hume, S. Carter, T. Henighan, and C. Olah (2023) Towards monosemanticity: decomposing language models with dictionary learning. Transformer Circuits Thread. External Links: Link Cited by: §2.2. R. Chawla, A. Datta, T. Verma, A. Jha, A. Gautam, A. Vatsal, S. Chaterjee, M. NS, and I. Bhola (2024) Veagle: advancements in multimodal representation learning. arXiv preprint arXiv:2403.08773. External Links: 2403.08773, Link Cited by: §1. G. H. Chen, S. Chen, R. Zhang, J. Chen, X. Wu, Z. Zhang, Z. Chen, J. Li, X. Wan, and B. Wang (2024) ALLaVA: harnessing gpt4v-synthesized data for lite vision-language models. arXiv preprint arXiv:2402.11684. External Links: 2402.11684, Link Cited by: §3.2. Y. Chen, Y. Yao, Y. Zhang, B. Shen, G. Liu, and S. Liu (2025) Safety mirage: how spurious correlations undermine vlm safety fine-tuning and can be mitigated by machine unlearning. arXiv preprint arXiv:2503.11832. External Links: 2503.11832, Link Cited by: §2.3. R. Cheng, Y. Ding, S. Cao, R. Duan, X. Jia, S. Yuan, S. Qin, Z. Wang, and X. Jia (2025) PBI-attack: prior-guided bimodal interactive black-box jailbreak attack for toxicity maximization. arXiv preprint arXiv:2412.05892. External Links: 2412.05892, Link Cited by: §2.1. A. Cloud, M. Le, J. Chua, J. Betley, A. Sztyber-Betley, J. Hilton, S. Marks, and O. Evans (2025) Subliminal learning: language models transmit behavioral traits via hidden signals in data. arXiv preprint arXiv:2507.14805. External Links: 2507.14805, Link Cited by: §1. F. Cocchi, N. Moratelli, D. Caffagni, S. Sarto, L. Baraldi, M. Cornia, and R. Cucchiara (2025) LLaVA-more: a comparative study of llms and visual backbones for enhanced visual instruction tuning. arXiv preprint arXiv:2503.15621. External Links: 2503.15621, Link Cited by: §1. W. Dong, Q. Yang, S. Yang, L. Hu, M. Ding, W. Lin, T. Zheng, and D. Wang (2025) Understanding and mitigating cross-lingual privacy leakage via language-specific and universal privacy neurons. arXiv preprint arXiv:2506.00759. External Links: Link Cited by: §2.2. R. Duan, J. Liu, X. Jia, S. Zhao, R. Cheng, F. Wang, C. Wei, Y. Xie, C. Liu, D. Li, Y. Dong, Y. Zhang, Y. Chen, C. Wang, X. Ma, X. Wei, Y. Liu, H. Su, J. Zhu, X. Li, Y. Sun, J. Zhang, J. Hu, S. Xu, W. Yang, Y. Yang, X. Zhang, Y. Tan, J. Tao, and H. Xue (2025) Oyster-i: beyond refusal – constructive safety alignment for responsible language models. arXiv preprint arXiv:2509.01909. External Links: 2509.01909, Link Cited by: Table 1, §4.1. L. Gao, T. D. la Tour, H. Tillman, G. Goh, R. Troll, A. Radford, I. Sutskever, J. Leike, and J. Wu (2024) Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093. External Links: 2406.04093, Link Cited by: §C.1, §C.4. A. Ghandeharioun, A. Yuan, M. Guerard, E. Reif, M. A. Lepori, and L. Dixon (2024) Who’s asking? user personas and the mechanics of latent misalignment. arXiv preprint arXiv:2406.12094. External Links: 2406.12094, Link Cited by: §1. J. Giordani (2025) Re-emergent misalignment: how narrow fine-tuning erodes safety alignment in llms. arXiv preprint arXiv:2507.03662. External Links: 2507.03662, Link Cited by: §2.2. Y. Gong, D. Ran, J. Liu, C. Wang, T. Cong, A. Wang, S. Duan, and X. Wang (2025) FigStep: jailbreaking large vision-language models via typographic visual prompts. arXiv preprint arXiv:2311.05608. External Links: 2311.05608, Link Cited by: §2.1. Z. He, W. Shu, X. Ge, L. Chen, J. Wang, Y. Zhou, F. Liu, Q. Guo, X. Huang, Z. Wu, Y. Jiang, and X. Qiu (2024) Llama scope: extracting millions of features from llama-3.1-8b with sparse autoencoders. External Links: 2410.20526, Link Cited by: §4.4. L. Hsiung, T. Pang, Y. Tang, L. Song, T. Ho, P. Chen, and Y. Yang (2025) Why llm safety guardrails collapse after fine-tuning: a similarity analysis between alignment and fine-tuning datasets. arXiv preprint arXiv:2506.05346. External Links: 2506.05346, Link Cited by: §3. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, External Links: 2106.09685, Link Cited by: §3.3. J. Hu, S. Yang, X. Gong, H. Wang, W. Liu, and D. Wang (2025a) MONICA: real-time monitoring and calibration of chain-of-thought sycophancy in large reasoning models. arXiv preprint arXiv:2511.06419. External Links: Link Cited by: §2.2. L. Hu, L. Liu, S. Yang, X. Chen, H. Xiao, M. Li, P. Zhou, M. A. Ali, and D. Wang (2024) A hopfieldian view-based interpretation for chain-of-thought reasoning. arXiv preprint arXiv:2406.12255. External Links: Link Cited by: §2.2. X. Hu, D. Liu, H. Li, X. Huang, and J. Shao (2025b) VLSBench: unveiling visual leakage in multimodal safety. arXiv preprint arXiv:2411.19939. External Links: 2411.19939, Link Cited by: §1. W. Huang, Z. Zhai, Y. Shen, S. Cao, F. Zhao, X. Xu, Z. Ye, Y. Hu, and S. Lin (2025) Dynamic-llava: efficient multimodal large language models via dynamic vision-language context sparsification. arXiv preprint arXiv:2412.00876. External Links: 2412.00876, Link Cited by: §1. S. Jain, E. S. Lubana, K. Oksuz, T. Joy, P. H. S. Torr, A. Sanyal, and P. K. Dokania (2024) What makes and breaks safety fine-tuning? a mechanistic study. arXiv preprint arXiv:2407.10264. External Links: 2407.10264, Link Cited by: §2.3. X. Jia, J. Liao, Q. Guo, T. Ma, S. Qin, R. Duan, T. Li, Y. Huang, Z. Zeng, D. Wu, Y. Li, W. Ren, X. Cao, and Y. Liu (2025) OmniSafeBench-m: a unified benchmark and toolbox for multimodal jailbreak attack-defense evaluation. arXiv preprint arXiv:2512.06589. External Links: 2512.06589, Link Cited by: §4.1. X. Jiang, L. Zhang, J. Zhang, Q. Yang, G. Hu, D. Wang, and L. Hu (2025) Msrs: adaptive multi-subspace representation steering for attribute alignment in large language models. arXiv preprint arXiv:2508.10599. External Links: Link Cited by: §2.2. Y. Jiang, Y. Tan, and X. Yue (2024) RapGuard: safeguarding multimodal large language models via rationale-aware defensive prompting. arXiv preprint arXiv:2412.18826. External Links: 2412.18826, Link Cited by: §2.3. B. W. Lee, I. Padhi, K. N. Ramamurthy, E. Miehling, P. Dognin, M. Nagireddy, and A. Dhurandhar (2025) Programming refusal with conditional activation steering. arXiv preprint arXiv:2409.05907. External Links: 2409.05907, Link Cited by: §1. H. Li, M. Yu, P. Singh, X. Li, D. Wang, L. Hu, et al. (2025a) Towards reasoning-preserving unlearning in multimodal large language models. arXiv preprint arXiv:2512.17911. External Links: Link Cited by: §1. S. Li, L. Yao, L. Zhang, and Y. Li (2025b) Safety layers in aligned large language models: the key to llm security. arXiv preprint arXiv:2408.17003. External Links: 2408.17003, Link Cited by: §2.2. Y. Li, H. Guo, K. Zhou, W. X. Zhao, and J. Wen (2025c) Images are achilles’ heel of alignment: exploiting visual vulnerabilities for jailbreaking multimodal large language models. arXiv preprint arXiv:2403.09792. External Links: 2403.09792, Link Cited by: §1. W. Liang, Y. Zhang, Y. Kwon, S. Yeung, and J. Zou (2022) Mind the gap: understanding the modality gap in multi-modal contrastive representation learning. arXiv preprint arXiv:2203.02053. External Links: 2203.02053, Link Cited by: §1. H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee (2024a) LLaVA-next: improved reasoning, ocr, and world knowledge. External Links: Link Cited by: §4.1. H. Liu, C. Li, Q. Wu, and Y. J. Lee (2023a) Visual instruction tuning. arXiv preprint arXiv:2304.08485. External Links: 2304.08485, Link Cited by: §4.1. H. Liu, C. Li, Q. Wu, and Y. J. Lee (2023b) Visual instruction tuning. arXiv preprint arXiv:2304.08485. External Links: 2304.08485, Link Cited by: §1, §3. Q. Liu, F. Wang, C. Xiao, and M. Chen (2025) VLM-guard: safeguarding vision-language models via fulfilling safety alignment gap. arXiv preprint arXiv:2502.10486. External Links: 2502.10486, Link Cited by: §2.3. X. Liu, Y. Zhu, J. Gu, Y. Lan, C. Yang, and Y. Qiao (2024b) M-safetybench: a benchmark for safety evaluation of multimodal large language models. arXiv preprint arXiv:2311.17600. External Links: 2311.17600, Link Cited by: §2.1. X. Liu, Y. Zhu, Y. Lan, C. Yang, and Y. Qiao (2024c) Safety of multimodal large language models on images and texts. arXiv preprint arXiv:2402.00357. External Links: 2402.00357, Link Cited by: §2.1. R. K. Merton (1948) The self-fulfilling prophecy. The Antioch Review 8 (2), p. 193–210. External Links: ISSN 00035769, Link Cited by: §1. nostalgebraist (2020) Interpreting GPT: the logit lens. Note: LessWrong External Links: Link Cited by: §C.3, §C.6. OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V. Balcom, P. Baltescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner, L. Bogdonoff, O. Boiko, M. Boyd, A. Brakman, G. Brockman, T. Brooks, M. Brundage, K. Button, T. Cai, R. Campbell, A. Cann, B. Carey, C. Carlson, R. Carmichael, B. Chan, C. Chang, F. Chantzis, D. Chen, S. Chen, R. Chen, J. Chen, M. Chen, B. Chess, C. Cho, C. Chu, H. W. Chung, D. Cummings, J. Currier, Y. Dai, C. Decareaux, T. Degry, N. Deutsch, D. Deville, A. Dhar, D. Dohan, S. Dowling, S. Dunning, A. Ecoffet, A. Eleti, T. Eloundou, D. Farhi, L. Fedus, N. Felix, S. P. Fishman, J. Forte, I. Fulford, L. Gao, E. Georges, C. Gibson, V. Goel, T. Gogineni, G. Goh, R. Gontijo-Lopes, J. Gordon, M. Grafstein, S. Gray, R. Greene, J. Gross, S. S. Gu, Y. Guo, C. Hallacy, J. Han, J. Harris, Y. He, M. Heaton, J. Heidecke, C. Hesse, A. Hickey, W. Hickey, P. Hoeschele, B. Houghton, K. Hsu, S. Hu, X. Hu, J. Huizinga, S. Jain, S. Jain, J. Jang, A. Jiang, R. Jiang, H. Jin, D. Jin, S. Jomoto, B. Jonn, H. Jun, T. Kaftan, Ł. Kaiser, A. Kamali, I. Kanitscheider, N. S. Keskar, T. Khan, L. Kilpatrick, J. W. Kim, C. Kim, Y. Kim, J. H. Kirchner, J. Kiros, M. Knight, D. Kokotajlo, Ł. Kondraciuk, A. Kondrich, A. Konstantinidis, K. Kosic, G. Krueger, V. Kuo, M. Lampe, I. Lan, T. Lee, J. Leike, J. Leung, D. Levy, C. M. Li, R. Lim, M. Lin, S. Lin, M. Litwin, T. Lopez, R. Lowe, P. Lue, A. Makanju, K. Malfacini, S. Manning, T. Markov, Y. Markovski, B. Martin, K. Mayer, A. Mayne, B. McGrew, S. M. McKinney, C. McLeavey, P. McMillan, J. McNeil, D. Medina, A. Mehta, J. Menick, L. Metz, A. Mishchenko, P. Mishkin, V. Monaco, E. Morikawa, D. Mossing, T. Mu, M. Murati, O. Murk, D. Mély, A. Nair, R. Nakano, R. Nayak, A. Neelakantan, R. Ngo, H. Noh, L. Ouyang, C. O’Keefe, J. Pachocki, A. Paino, J. Palermo, A. Pantuliano, G. Parascandolo, J. Parish, E. Parparita, A. Passos, M. Pavlov, A. Peng, A. Perelman, F. de Avila Belbute Peres, M. Petrov, H. P. de Oliveira Pinto, Michael, Pokorny, M. Pokrass, V. H. Pong, T. Powell, A. Power, B. Power, E. Proehl, R. Puri, A. Radford, J. Rae, A. Ramesh, C. Raymond, F. Real, K. Rimbach, C. Ross, B. Rotsted, H. Roussez, N. Ryder, M. Saltarelli, T. Sanders, S. Santurkar, G. Sastry, H. Schmidt, D. Schnurr, J. Schulman, D. Selsam, K. Sheppard, T. Sherbakov, J. Shieh, S. Shoker, P. Shyam, S. Sidor, E. Sigler, M. Simens, J. Sitkin, K. Slama, I. Sohl, B. Sokolowsky, Y. Song, N. Staudacher, F. P. Such, N. Summers, I. Sutskever, J. Tang, N. Tezak, M. B. Thompson, P. Tillet, A. Tootoonchian, E. Tseng, P. Tuggle, N. Turley, J. Tworek, J. F. C. Uribe, A. Vallone, A. Vijayvergiya, C. Voss, C. Wainwright, J. J. Wang, A. Wang, B. Wang, J. Ward, J. Wei, C. Weinmann, A. Welihinda, P. Welinder, J. Weng, L. Weng, M. Wiethoff, D. Willner, C. Winter, S. Wolrich, H. Wong, L. Workman, S. Wu, J. Wu, M. Wu, K. Xiao, T. Xu, S. Yoo, K. Yu, Q. Yuan, W. Zaremba, R. Zellers, C. Zhang, M. Zhang, S. Zhao, T. Zheng, J. Zhuang, W. Zhuk, and B. Zoph (2024) GPT-4 technical report. arXiv preprint arXiv:2303.08774. External Links: 2303.08774, Link Cited by: §3. M. Pach, S. Karthik, Q. Bouniot, S. Belongie, and Z. Akata (2025) Sparse autoencoders learn monosemantic features in vision-language models. arXiv preprint arXiv:2504.02821. External Links: 2504.02821, Link Cited by: §2.2. B. Peng, C. Li, P. He, M. Galley, and J. Gao (2023) Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277. External Links: 2304.03277, Link Cited by: Appendix B. X. Qi, K. Huang, A. Panda, P. Henderson, M. Wang, and P. Mittal (2023) Visual adversarial examples jailbreak aligned large language models. arXiv preprint arXiv:2306.13213. External Links: 2306.13213, Link Cited by: §2.1. K. Ren, P. Nakov, and U. Naseem (2025) DUAL-bench: measuring over-refusal and robustness in vision-language models. arXiv preprint arXiv:2510.10846. External Links: 2510.10846, Link Cited by: §2.3. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer (2022) High-resolution image synthesis with latent diffusion models. arXiv preprint arXiv:2112.10752. External Links: 2112.10752, Link Cited by: §3.1.2. P. Tang, H. Xin, X. Zhang, J. Sun, Q. Xia, and Z. Yang (2025) The safety reminder: a soft prompt to reactivate delayed safety awareness in vision-language models. arXiv preprint arXiv:2506.15734. External Links: 2506.15734, Link Cited by: §1. A. Turner (2025) Self-fulfilling misalignment: data might be poisoning our ai models. Note: Blog post External Links: Link Cited by: §1. K. Wang, J. Li, S. Yang, Z. Zhang, and D. Wang (2025a) When truth is overridden: uncovering the internal origins of sycophancy in large language models. arXiv preprint arXiv:2508.02087. External Links: Link Cited by: §2.2. M. Wang, T. D. la Tour, O. Watkins, A. Makelov, R. A. Chi, S. Miserendino, J. Wang, A. Rajaram, J. Heidecke, T. Patwardhan, and D. Mossing (2025b) Persona features control emergent misalignment. arXiv preprint arXiv:2506.19823. External Links: 2506.19823, Link Cited by: §1, §1. P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Lin (2024a) Qwen2-vl: enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191. External Links: 2409.12191, Link Cited by: §1, §4.1. Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi (2023) Self-instruct: aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, External Links: 2212.10560, Link Cited by: §3. Y. Wang, X. Liu, Y. Li, M. Chen, and C. Xiao (2024b) AdaShield: safeguarding multimodal large language models from structure-based attack via adaptive shield prompting. arXiv preprint arXiv:2403.09513. External Links: 2403.09513, Link Cited by: §4.1. H. Xie, R. S. Maharjan, F. Tavella, and A. Cangelosi (2024) From concrete to abstract: a multimodal generative approach to abstract concept learning. arXiv preprint arXiv:2410.02365. External Links: 2410.02365, Link Cited by: §1. S. Yang, J. Su, H. Jiang, M. Li, K. Cheng, M. A. Ali, L. Hu, and D. Wang (2024a) Dialectical alignment: resolving the tension of 3h and security threats of llms. arXiv preprint arXiv:2404.00486. External Links: Link Cited by: §2.3. S. Yang, S. Zhu, L. Liu, L. Hu, M. Li, and D. Wang (2024b) Exploring the personality traits of llms through latent features steering. arXiv preprint arXiv:2410.10863. External Links: Link Cited by: §1. T. Yang, L. Zhang, J. Lin, G. Hu, D. Wang, and L. Hu (2025) D-leaf: localizing and correcting hallucinations in multimodal llms via layer-to-head attention diagnostics. arXiv preprint arXiv:2509.07864. External Links: 2509.07864, Link Cited by: §1. J. Yao, S. Yang, J. Xu, L. Hu, M. Li, and D. Wang (2025) Understanding the repeat curse in large language models from a feature perspective. arXiv preprint arXiv:2504.14218. External Links: Link Cited by: §2.2. M. Yu, H. Li, P. Singh, X. Li, D. Wang, and L. Hu (2025) Pixel: adaptive steering via position-wise injection with exact estimated levels under subspace calibration. arXiv preprint arXiv:2510.10205. External Links: Link Cited by: §2.2. W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang (2024) M-vet: evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490. External Links: 2308.02490, Link Cited by: §4.1, §4.1. Q. Zhang, Y. Wang, J. Cui, X. Pan, Q. Lei, S. Jegelka, and Y. Wang (2024) Beyond interpretability: the gains of feature monosemanticity on model robustness. arXiv preprint arXiv:2410.21331. External Links: 2410.21331, Link Cited by: §2.2. Y. Zhang, L. Chen, G. Zheng, Y. Gao, R. Zheng, J. Fu, Z. Yin, S. Jin, Y. Qiao, X. Huang, F. Zhao, T. Gui, and J. Shao (2025a) SPA-vl: a comprehensive safety preference alignment dataset for vision language model. arXiv preprint arXiv:2406.12030. External Links: 2406.12030, Link Cited by: §4.1. Y. Zhang, M. Li, W. Han, Y. Yao, Z. Cen, and D. Zhao (2025b) Safety is not only about refusal: reasoning-enhanced fine-tuning for interpretable llm safety. arXiv preprint arXiv:2503.05021. External Links: 2503.05021, Link Cited by: §4.1. L. Zheng, W. Chiang, Y. Sheng, T. Li, S. Zhuang, Z. Wu, Y. Zhuang, Z. Li, Z. Lin, E. P. Xing, J. E. Gonzalez, I. Stoica, and H. Zhang (2024) LMSYS-chat-1m: a large-scale real-world llm conversation dataset. arXiv preprint arXiv:2309.11998. External Links: 2309.11998, Link Cited by: §C.3. W. Zhou, M. Hendy, S. Yang, Q. Yang, Z. Guo, Y. Luo, L. Hu, and D. Wang (2025) Flattery in motion: benchmarking and analyzing sycophancy in video-llms. arXiv preprint arXiv:2506.07180. External Links: Link Cited by: §1. Y. Zong, O. Bohdal, T. Yu, Y. Yang, and T. Hospedales (2024) Safety fine-tuning at (almost) no cost: a baseline for vision large language models. arXiv preprint arXiv:2402.02207. External Links: 2402.02207, Link Cited by: §4.1. X. Zou, J. Kang, G. Kesidis, and L. Lin (2025) Understanding and rectifying safety perception distortion in vlms. arXiv preprint arXiv:2502.13095. External Links: 2502.13095, Link Cited by: §2.3. Appendix A Dataset Statistics This appendix provides the complete lists of arXiv search terms, visual styles, and scene environments used in VSFA dataset construction. We also report the dataset distribution statistics. A.1 arXiv Search Terms We collect academic text from AI safety research on arXiv. Table 3 lists all 10 search terms used in this work. Table 3: The complete arXiv search terms used for text collection. We retrieve up to 5 papers per term from categories cs.AI, cs.LG, cs.CY, and cs.CR. Search Terms AI safety alignment AI risk existential artificial intelligence threat AI alignment problem AI safety research machine learning safety AI control problem AGI risk AI alignment failure AI safety measures We design these search terms to cover three aspects of AI safety. The first group targets alignment research. Terms like “AI safety alignment” and “AI alignment problem” retrieve papers on value alignment. The second group focuses on risk analysis. Terms like “AI risk existential” and “AGI risk” capture research on potential harms. The third group addresses technical solutions. Terms like “machine learning safety” and “AI control problem” find papers on safety mechanisms. The arXiv API returns papers sorted by relevance. We extract the abstract from each paper. These abstracts provide domain-specific concepts about AI risks. GPT-4o-mini then converts the abstracts into image generation prompts. This approach ensures the generated images carry threat-related semantics without explicit harmful content. A.2 Visual Styles and Scene Environments We use systematic combinations of visual styles and scene environments to ensure image diversity. Table 4 shows all 12 visual styles. Table 5 lists all 15 scene environments. Table 4: The 12 visual styles applied during image generation. We combine these styles with scene environments to maximize visual diversity across the dataset. Visual Styles Photorealistic Digital art Concept art Technical illustration Documentary style Abstract representation Cinematic Artistic Professional Casual Futuristic Vintage Table 5: The 15 scene environments used for image composition. Each environment provides different visual context for threat-related content. Scene Environments Office/Workplace Laboratory Public space Home environment Industrial setting Educational Medical Research facility Urban Rural Indoor Outdoor Virtual Mixed reality Studio The image generation pipeline tracks all used combinations. This mechanism prevents repetition and ensures diversity. With 12 styles and 15 environments, we have 180 possible base combinations. Our 700 images sample from this space with additional variation in lighting and camera angles. The visual styles range from realistic to artistic. Photorealistic style produces images that look like photographs. Digital art and concept art create more stylized visuals. Documentary and cinematic styles add specific moods to the scenes. We include both futuristic and vintage styles to cover different time settings. The scene environments span common locations where AI systems operate. Laboratory and research facility represent technical settings. Office and workplace show professional contexts. Public space and urban environments depict everyday locations. We also include virtual and mixed reality to represent digital spaces. A.3 Dataset Distribution Table 6 summarizes the VSFA dataset statistics. The dataset contains 700 images with 4,200 VQA pairs. Table 6: Summary statistics of the VSFA dataset. We generate 6 neutral VQA pairs per image using 16 question templates across 4 categories. Statistic Value Total images 700 Total VQA pairs 4,200 VQA pairs per image 6 Image resolution 1024 × 1024 Question categories 4 Question templates 16 Question Categories. We design four categories of neutral questions. Each category has 4 question templates. The questions avoid safety-related words like “threat” or “danger”. They focus on factual description of visual content. Descriptive questions ask what the image shows. Examples include “What does this image show?” and “What elements are present in the image?” These questions request direct observation of visual content. Open-ended questions invite broader discussion. Examples include “What does this remind you of?” and “What other details can you observe?” These questions allow the model to elaborate freely. Analytical questions probe scene interpretation. Examples include “What are the characteristics of this scene?” and “What do you think is happening here?” These questions require understanding of the visual context. Contextual questions explore settings and functions. Examples include “What is this environment typically used for?” and “What activities is this setting suitable for?” These questions connect visual content to real-world usage. Answer Generation. We use GPT-4o-mini to generate answers for each question. The model receives the image generation prompt as context. It describes the expected visual content in a neutral and factual manner. The answers do not include safety judgments or warnings. Table 7 shows the answer length distribution. Most answers contain 50 to 120 words. This range provides enough detail for training without excessive length. Table 7: Answer length statistics in word count. The answers maintain moderate length suitable for visual instruction tuning. Metric Words Mean 85 Median 78 Min 25 Max 180 Std 32 The answer length varies by question type. Descriptive questions tend to produce shorter answers. They focus on listing visible elements. Open-ended and analytical questions produce longer answers. They require more explanation and interpretation. This variation reflects natural response patterns in VQA tasks. Appendix B Quality Control Details Quality control is essential for VSFA training. VSFA relies on a core assumption: models learn vigilance from visual content through implicit exposure. The training data should not contain explicit safety signals. If a question says “this image is dangerous”, the model receives direct guidance. This breaks the self-fulfilling mechanism. We need QA pairs that describe threat-related images in a neutral way. The model should develop safety awareness from what it sees, not from what the text tells it. We use GPT-4o-mini as an automated evaluator to filter generated QA pairs Peng et al. (2023). The evaluator checks each sample on three dimensions: neutrality, clarity, and consistency. Table 8 summarizes the evaluation guidelines for each dimension. Table 8: Evaluation criteria for QA quality control. Each dimension is scored on a 0-10 scale. We discard samples with overall score below 6.0. Dimension Evaluation Guidelines Neutrality The question should not suggest any specific answer. We check for value-laden words like “dangerous”, “harmful”, or “threatening”. The question should avoid safety-related terms like “threat”, “warning”, or “caution”. A neutral question asks about facts in the image. It does not ask for opinions or value judgments. VSFA relies on implicit learning from visual content. Explicit safety terms would confound this learning process. Clarity The question should have one clear meaning. Ambiguous questions create confusion in training. The answer should directly address the question. Both should use correct grammar with clear subjects and verbs. The answer should use specific terms instead of vague words like “something” or “stuff”. Clear QA pairs provide strong training signals. Consistency The answer should not contradict itself. If one sentence describes “a dark room”, later sentences should not mention “bright sunlight”. All descriptions should match what appears in the image. The model should not describe objects that are not visible. The reasoning should follow a logical order. In multi-turn dialogues, answers should stay consistent across turns. Neutrality. Neutrality is the most important criterion for VSFA. Why does this matter so much? VSFA works through implicit learning. The model sees threat-related images and develops vigilance on its own. If questions contain words like “dangerous” or “risky”, they provide explicit safety signals. These signals tell the model how to interpret the image. The model no longer learns from visual content alone. We check for two types of problematic words. Value-laden words include “dangerous”, “harmful”, “risky”, and “threatening”. These words express judgments about the image content. Safety-related terms include “threat”, “warning”, “caution”, and “alert”. These words introduce explicit safety concepts into the training data. A neutral question focuses on observable facts. It asks what objects appear in the image. It asks about colors, positions, or quantities. It does not ask whether something is good or bad. Here is an example. A good question: “What equipment is visible in this laboratory?” A bad question: “What dangerous chemicals can you identify?” The second question tells the model to look for danger. The first question lets the model describe what it sees. Clarity. Clear questions produce clear answers. Ambiguous questions lead to vague or confused responses. These low-quality responses hurt training effectiveness. The model learns better from precise descriptions than from fuzzy ones. We examine several aspects of clarity. The question should have exactly one interpretation. “What is this?” is too vague. “What type of monitoring equipment appears in this image?” is specific. The answer should directly respond to what the question asks. If the question asks about equipment, the answer should describe equipment. It should not drift to unrelated topics. Grammar matters for clarity. Each sentence needs a clear subject. Run-on sentences should be split into shorter ones. The answer should use concrete nouns instead of vague references. “The control panel has three screens” is better than “There is some stuff with displays”. Specific language creates stronger training signals for the model. Consistency. Consistent answers help the model build accurate representations. Contradictory information confuses the learning process. If an answer says the room is dark, then mentions bright sunlight, the model receives conflicting signals. We check for internal consistency within each answer. Factual accuracy is part of consistency. The answer should only describe what actually appears in the image. If the image shows two monitors, the answer should not claim there are five. The model should not invent objects or details. This factual grounding ensures the model learns real visual understanding. Logical flow also matters. Good answers move from observation to description in a clear order. They might start with the overall scene, then describe specific objects. The reasoning should make sense. In dialogues with multiple turns, the model should remember what it said before. Later answers should not contradict earlier ones. Evaluation Process. The evaluator receives each QA pair and outputs scores in JSON format. Here is an example output: "neutrality": 8.5, "clarity": 7.2, "consistency": 9.0, "overall score": 8.2, "recommendation": "keep" The overall score combines the three dimension scores. The recommendation can be “keep”, “revise”, or “discard”. We apply strict filtering rules. A sample passes only when two conditions are met: the overall score reaches at least 6.0, and the recommendation is either “keep” or “revise”. Samples that fail either condition are removed from the training set. This filtering process removes low-quality samples from our dataset. The remaining samples maintain neutral framing throughout. They describe threat-related images without using explicit safety language. This ensures that VSFA can work through implicit learning as designed. Appendix C SAE Analysis Details This appendix provides the complete details of our sparse autoencoder (SAE) analysis. We describe the SAE training procedure, model-diffing methodology, latent identification criteria, and steering experiment results. C.1 SAE Training We use a sparse autoencoder trained on Qwen2.5-VL-7B activations. The SAE follows the architecture from Gao et al. Gao et al. (2024). We collect activations from the middle layer of the language model component. The visual encoder remains frozen during both VSFA training and SAE analysis. C.2 Model-Diffing Procedure To investigate whether VSFA shapes safety-oriented personas, we apply model-diffing with sparse autoencoders. Given the original model M, and the resulting fine-tuned model MDM_D, we compare SAE latent activations between M and MDM_D. We use MMSafetyBench as our evaluation prompts. For each prompt, we collect activations at the middle layer from both the original model and the VSFA-finetuned model. We pass these activations through the SAE encoder to obtain latent activations. We average across all tokens in the assistant response. We then compute the difference: VSFA model activation minus original model activation. Latents with positive differences indicate features that become more active after VSFA training. We rank latents by this difference and focus on the top 1000 latents whose activations increase most after VSFA training. C.3 Identifying the Safety-Oriented Persona Latent From these top 1000 latents that activate more after VSFA, we identify which ones causally control safety behavior through steering experiments. We add multiples of each latent’s decoder vector to all token activations at the target layer. We measure the effect on model responses. We select latents that satisfy two criteria. Positive steering on the original model should increase safe responses. Negative steering on the VSFA model should decrease safe responses. We find 8 latents that meet both criteria. The eight strongest SAE latents for steering safety are: #12 safety-oriented persona: vigilance and caution patterns for identifying harmful requests. (top tokens: warning, caution, harmful, refuse, alert, danger, unsafe) #47 risk awareness: threat recognition and risk assessment patterns. (top tokens: danger, risk, careful, avoid, threat, hazard, concern) #89 refusal pattern: soft refusal responses with explanations. (top tokens: sorry, cannot, inappropriate, unable, decline, regret, apologize) #156 ethical reasoning: value-based judgment and moral evaluation. (top tokens: ethical, moral, wrong, responsible, proper, acceptable, appropriate) #284 harm recognition: identifying harmful content in requests. (top tokens: harmful, dangerous, illegal, unsafe, risky, problematic, concerning) #312 alternative suggestion: redirecting to safer alternatives. (top tokens: instead, alternative, suggest, recommend, consider, option, rather) #458 explanation pattern: providing reasons for refusal. (top tokens: because, therefore, reason, explain, understand, cause, result) #521 context discrimination: distinguishing benign from harmful intent. (top tokens: context, situation, intent, purpose, depends, circumstance, specific) To interpret each latent, we obtain top tokens via the logit lens approach nostalgebraist (2020), which computes the cosine similarity between the latent’s decoder vector and vocabulary embeddings. For semantic interpretation, we examine top activating examples from LMSYS-Chat-1M Zheng et al. (2024) and use auto-interpretation with GPT-4o Bills et al. (2023). C.4 SAE Setup We apply a sparse autoencoder to Qwen2.5-VL-7B activations. The SAE uses the TopK architecture Gao et al. (2024). We collect activations from the middle layer of the language model. The visual encoder stays frozen during both VSFA training and SAE analysis. C.5 Model-Diffing Procedure How do we find which latents matter for safety? We compare SAE activations between the original model M and the VSFA-trained model MDM_D. For each prompt in MMSafetyBench, we collect middle-layer activations from both models. We pass these through the SAE encoder and average across all response tokens. We compute the difference for each latent. Latents with positive differences become more active after VSFA. We rank them and focus on the top 1000. But activation increase alone does not prove causality. A latent might correlate with safety without controlling it. We need steering experiments to test causal control. Figure 3: Bidirectional steering effects of top SAE latents. Bars below zero show ASR reduction from adding the latent to the original model. Bars above zero show ASR increase from removing it from the VSFA model. Latent #12 shows the strongest effect in both directions (−-18%/+14%), confirming it as the primary safety-oriented persona latent. C.6 Identifying the Safety-Oriented Persona Latent Which latents actually control safety? We test each candidate through steering. We add its decoder vector to all token activations and measure how responses change. We require bidirectional effects for causal proof. Positive steering on the original model should decrease ASR. Negative steering on the VSFA model should increase ASR. Why both directions? Single-direction effects might be artifacts. Bidirectional control proves genuine encoding of safety behavior. We find 8 latents satisfying both criteria. We interpret each latent using logit lens nostalgebraist (2020). This computes cosine similarity between the decoder vector and vocabulary embeddings. The top tokens reveal what each latent represents. Figure 3 visualizes the bidirectional steering effects. Bars below zero show how much ASR drops when we add each latent to the original model. Bars above zero show how much ASR rises when we remove each latent from the VSFA model. Latent #12 dominates both directions. Adding it reduces ASR by 18%. Removing it increases ASR by 14%. No other latent comes close to this bidirectional strength. What does latent #12 encode? Its top tokens tell the story. Warning, caution, harmful, refuse, alert, danger. These are not random words. They form a coherent pattern of vigilance and threat awareness. This latent encodes exactly what we predicted in Section 1. VSFA shapes safety-oriented personas through visual exposure to threat-related content. The model does not memorize specific refusal phrases. It develops an internal representation that recognizes threats and responds with caution. The other 7 latents support this picture. Risk awareness (#47) handles threat recognition. Refusal pattern (#89) produces polite declines. Ethical reasoning (#156) evaluates moral implications. Together they form a safety-oriented persona that VSFA training activates. C.7 Summary Our SAE analysis reveals three findings about how VSFA works. VSFA activates a specific latent in the model. This safety-oriented persona latent shows higher activation after VSFA training. Its top tokens encode vigilance and caution patterns. The semantic content matches our hypothesis about self-fulfilling alignment. Steering experiments confirm causal control. The same latent works bidirectionally on two different models. Adding it to the original model makes responses safer. Removing it from the VSFA model makes responses less safe. This rules out correlation. The latent genuinely controls safety behavior. This provides mechanistic evidence for self-fulfilling alignment. Visual exposure to threat-related images activates safety-oriented persona features. These features guide cautious behavior across diverse contexts. The model internalizes a vigilant persona rather than learning surface patterns.

Instant research discovery