Paper deep dive
OmniSafeBench-MM: A Unified Benchmark and Toolbox for Multimodal Jailbreak Attack-Defense Evaluation
Xiaojun Jia, Jie Liao, Qi Guo, Teng Ma, Simeng Qin, Ranjie Duan, Tianlin Li, Yihao Huang, Zhitao Zeng, Dongxian Wu, Yiming Li, Wenqi Ren, Xiaochun Cao, Yang Liu
Models: Claude Sonnet, DeepSeek-VL2, Doubao-Seed, Gemini-2.5-Flash, Gemma-3-27b, GLM-4.1V, GPT-4o, Kimi-VL, MiniGPT-4, Qwen3-VL-30B, Qwen3-VL-Plus
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 98%
Last extracted: 3/12/2026, 6:22:09 PM
Summary
OmniSafeBench-MM is a comprehensive, unified benchmark and open-source toolbox designed to evaluate the safety of Multimodal Large Language Models (MLLMs) against jailbreak attacks. It integrates 13 attack methods, 15 defense strategies, and a large-scale dataset covering 9 risk domains and 50 subcategories. The framework introduces a three-dimensional evaluation protocol measuring harmfulness, intent alignment, and response detail, providing a standardized platform for assessing safety-utility trade-offs across both open-source and closed-source MLLMs.
Entities (4)
Relation Signals (3)
OmniSafeBench-MM → evaluates → MLLM
confidence 100% · OmniSafeBench-MM is a comprehensive toolbox for multi-modal jailbreak attack-defense evaluation.
OmniSafeBench-MM → integrates → Jailbreak Attack
confidence 100% · OmniSafeBench-MM integrates 13 representative attack methods
OmniSafeBench-MM → uses → PixArt-XL-2-1024-MS
confidence 100% · we adopt the PixArt-XL-2-1024-MS model to generate the corresponding risk images.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent advances in multi-modal large language models (MLLMs) have enabled unified perception-reasoning capabilities, yet these systems remain highly vulnerable to jailbreak attacks that bypass safety alignment and induce harmful behaviors. Existing benchmarks such as JailBreakV-28K, MM-SafetyBench, and HADES provide valuable insights into multi-modal vulnerabilities, but they typically focus on limited attack scenarios, lack standardized defense evaluation, and offer no unified, reproducible toolbox. To address these gaps, we introduce OmniSafeBench-MM, which is a comprehensive toolbox for multi-modal jailbreak attack-defense evaluation. OmniSafeBench-MM integrates 13 representative attack methods, 15 defense strategies, and a diverse dataset spanning 9 major risk domains and 50 fine-grained categories, structured across consultative, imperative, and declarative inquiry types to reflect realistic user intentions. Beyond data coverage, it establishes a three-dimensional evaluation protocol measuring (1) harmfulness, distinguished by a granular, multi-level scale ranging from low-impact individual harm to catastrophic societal threats, (2) intent alignment between responses and queries, and (3) response detail level, enabling nuanced safety-utility analysis. We conduct extensive experiments on 10 open-source and 8 closed-source MLLMs to reveal their vulnerability to multi-modal jailbreak. By unifying data, methodology, and evaluation into an open-source, reproducible platform, OmniSafeBench-MM provides a standardized foundation for future research. The code is released at this https URL.
Tags
Links
Trouble viewing inline? Open PDF directly →
Full Text
67,738 characters extracted from source content.
Expand or collapse full text
OmniSafeBench-M: A Unified Benchmark and Toolbox for Multimodal Jailbreak Attack–Defense Evaluation Xiaojun Jia 1,† * , Jie Liao 2,3,∗ , Qi Guo 2,4,∗ , Teng Ma 2,6,∗ , Simeng Qin 2,5,∗ , Ranjie Duan 7 , Tianlin Li 1 Yihao Huang 1 , Zhitao Zeng 8 , Dongxian Wu 9 Yiming Li 1 , Wenqi Ren 6 , Xiaochun Cao 6 , Yang Liu 1 1 Nanyang Technological University, Singapore 2 BraneMatrix AI, China 3 Chongqing University, China 4 Xi’an Jiaotong University, China 5 Northeastern University, China 6 Sun Yat-sen University, China 7 Alibaba, China 8 National University of Singapore, Singapore 9 ByteDance, China Abstract Recent advances in multi-modal large language models (MLLMs) have enabled unified perception–reasoning ca- pabilities, yet these systems remain highly vulnerable to jailbreak attacks that bypass safety alignment and in- duce harmful behaviors.Existing benchmarks such as JailBreakV-28K, M-SafetyBench, and HADES provide valuable insights into multi-modal vulnerabilities, but they typically focus on limited attack scenarios, lack stan- dardized defense evaluation, and offer no unified, re- producible toolbox.To address these gaps, we intro- duce OmniSafeBench-M, which is a comprehensive tool- box for multi-modal jailbreak attack–defense evaluation. OmniSafeBench-M integrates 13 representative attack methods, 15 defense strategies, and a diverse dataset span- ning 9 major risk domains and 50 fine-grained categories, structured across consultative, imperative, and declarative inquiry types to reflect realistic user intentions. Beyond data coverage, it establishes a three-dimensional evalu- ation protocol measuring (1) harmfulness, distinguished by a granular, multi-level scale ranging from low-impact individual harm to catastrophic societal threats, (2) in- tent alignment between responses and queries, and (3) re- sponse detail level, enabling nuanced safety–utility analy- sis. We conduct extensive experiments on 10 open-source and 8 closed-source MLLMs to reveal their vulnerabil- ity to multi-modal jailbreak. By unifying data, method- ology, and evaluation into an open-source, reproducible platform, OmniSafeBench-M provides a standardized foundation for future research. The code is released at https://github.com/jiaxiaojunQAQ/OmniSafeBench-M. * Xiaojun Jia, Jie Liao, Qi Guo, Teng Ma, and Simeng Qin con- tribute equally to this work.†Project Leader: Xiaojun Jia (jiaxiaojun- qaq@gmail.com). 1. Introduction Recent works [1, 6, 51] in multi-modal large language mod- els (MLLMs) such as GPT-4o, Gemini-2.5, and Qwen2-VL have dramatically advanced the integration of visual per- ception and language reasoning. By jointly understand- ing text, images, and other modalities, these systems have achieved outstanding performance in tasks including visual question answering [19], image analysis [2], and embodied reasoning [9]. However, cross-modal interaction that can improve performance also introduces new vulnerabilities: maliciously generated input can bypass safety constraints, inducing models to output harmful, unethical, or policy- violating contents, called jailbreak attack [13, 15, 37, 42, 45]. Unlike classical adversarial attacks [16, 22, 28] that manipulate pixels or tokens to degrade task accuracy, jail- break attacks break the model’s safety alignment [7, 44], prompting the model to actively generate unsafe content. As multi-modal interactions become more common, attack- ers are no longer limited to text prompt attacks. By ex- ploiting the visual context, they [10, 39] can embed con- cealed intents within ordinary-looking images or instruc- tions, making it easier to bypass the safety alignment of MLLMs [37]. It is necessary to conduct a comprehensive multimodal-based safety evaluation of MLLMs. Previous works have proposed several multi-modal datasets to assess the safety of MLLMs such as Fig- Step [10], JailBreakV-28K [25], M-SafetyBench [23], HADES [21], and MMJ-Bench [40]. However, as shown in Table 1, these datasets have two main limitations: first, the range of covered risk categories is not sufficiently com- prehensive to capture diverse multimodal safety threats; and second, they overlook the type of risk prompts, for exam- ple whether the attack prompt is consultative, instructive, or deceptive, which hinders a comprehensive and detailed evaluation of the safety of MLLMs. A series of works have explored how to automatically generate multi-modal jail- arXiv:2512.06589v1 [cs.CR] 6 Dec 2025 break prompts from various perspectives [3, 5, 17, 35, 46]. However, they adopt different evaluations, including differ- ent comparison methods, different defense methods, etc., which makes them difficult to compare and hinders future progress. Moreover, most existing methods rely on a single metric, the attack success rate (ASR), to measure the effec- tiveness of jailbreak strategies, which makes it difficult to comprehensively evaluate their performance. To bridge these gaps, as shown in Fig. 1, we propose OmniSafeBench-M, a unified benchmark and toolbox for systematic multimodal jailbreak attack–defense evaluation. In contrast to prior works that focus solely on datasets or single-dimensional evaluation metric, our OmniSafeBench- M provides an integrated platform that combines compre- hensive data coverage, standardized experimental pipelines, and a multi-dimensional evaluation protocol. Specifically, at first, we construct a comprehensive multi-modal dataset. As shown in Fig. 3, our OmniSafeBench-M introduces a newly generated multi-modal dataset covering 9 major risk domains (e.g., violence, privacy, illegal activity, misinfor- mation, ethics, and self-harm) and 50 fine-grained subcat- egories. Each sample is categorized by inquiry type, in- cluding consultative, imperative, and declarative, reflecting real user interaction patterns with MLLMs. To efficiently construct the dataset, we further propose an automated data generation pipeline that produces risk-related text–image pairs based on pre-defined risk categories. We integrate a comprehensive suite of jailbreak attacks and defenses. On the attack side, we implement about 13 advanced multi-modal jailbreak methods that span a spectrum of threat models, including white-box and black-box set- tings. These methods cover diverse strategies such as role- play [26], image-embedded cues [52], and risk decompo- sition [27], thereby enabling the construction of diverse multi-modal jailbreak image–text pairs. On the defense side, our OmniSafeBench-M incorporates a total of 15 defense methods, which fall into two categories: (1) off- model defense methods, which are deployed as out-of- model plugins to intercept unsafe inputs prior to inference or to filter unsafe outputs during inference, and (2) on- model defense methods, which aim to improve the MLLMs’ internal alignment through training or fine-tuning. All at- tack and defense components are implemented in a modular and open-source toolbox for data processing, attack genera- tion, defense application, and metric evaluation. Moreover, beyond traditional binary success rates, we also propose to establish a three-dimensional evaluation framework as- sessing (1) harmfulness, (2) intent alignment between the model response and the original query, and (3) the level of detail in the response. These complementary metrics enable fine-grained assessment of both attack severity and defense trade-offs, capturing cases where safety improve- ments come at the expense of helpfulness. We conduct extensive experiments on 18 popular MLLMs, including 10 open-source models (e.g., LLaVA- 1.6, Qwen3-VL, GLM-4.1V) and 8 closed-source commer- cial systems (e.g., GPT-5, Gemini-2.5, Claude-3.5, Qwen3- VL-Plus). The experimental results reveal that different modalities and architectures have significant differences in defending against different types of multi-modal jailbreak attacks. Some defenses substantially reduce harmfulness but degrade model helpfulness, while others maintain help- fulness yet fail to eliminate residual vulnerabilities. Hence, the main contributions are summarized as follows: • We propose OmniSafeBench-M, a unified framework that integrates dataset, attack–defense techniques, and evaluation into a single reproducible platform. It sup- ports end-to-end experimentation with modular APIs for dataset loading, attack generation, defense execution, and performance evaluation across both open-source and commercial MLLMs. • We construct a large-scale multi-modal dataset covering 9 major risk domains and 50 fine-grained categories, orga- nized by three user intention types—consultative, instruc- tive, and declarative. • The toolbox includes 13 popular jailbreak attack methods and 15 defense methods, providing a standardized plat- form for comparative studies, with plans to continuously update and incorporate more methods going forward. • We propose a three-dimensional metric framework that jointly measures harmfulness, intent alignment, and re- sponse detail level, enabling comprehensive safety–utility trade-off analysis beyond attack success rate metrics. 2. Related Work 2.1. Multi-modal jailbreak attack Existing multi-modal jailbreak attack studies can be broadly categorized into white-box and black-box settings.In white-box settings, attackers adopt model gradients or ar- chitectural information to generate adversarial perturba- tions. For example, Qi et al. [32] extend adversarial op- timization on the visual channel to multi-modal alignment disruption. Niu et al. [29] propose to enhance transferabil- ity across models through ensemble optimization. Wang et al. [34], Ying et al. [46], Chen et al. [3], and Cheng et al. [4] propose to jointly optimize visual and textual in- puts to exploit cross-modal vulnerabilities. Such gradient- based methods highlight the instability of safety alignment even when the attacker operates within differentiable white- box conditions. In contrast, in black-box settings, attack- ers have no internal access and rely on interacting with the model to achieve jailbreaks. For example, Gong et al. [10], Liu et al. [23], and Li et al. [21] propose to embed textual or typographic cues within images to convert visual carriers into executable prompts. Yang et al. [43], Zhao et al. [49], Visual-adv ImgJP DeltaJP UMK BAP JPS FigStep(-Pro) QR-Attack HADES CS-DJ SI-Attack JOOD MML VisCRA HIMRD PBI-Attack ECSO JailGuard DPS CIDER GuardReasoner-VL Llama-Guard-4 QGuard LlavaGuard ShieldLM MLLM-Protector Llama-Guard-3 COCA HiddenDetect VLGuard AdaShield-S ATTACK White-box Black-box DEFENSE Off-model On-model B. Privacy and Data Risks 5 subcategories C. Safety and Physical Harm 7 subcategories D. Criminal and Economic Risks 5 subcategories E. Cybersecurity Threats 7 subcategories 6 subcategories G. Content and Cultural Safety 4 subcategories 4 subcategories I. Decision and Cognitive Risks 8 subcategories More than 20+ Target Models · PromptResponse Harmfulness Judge 풇 푯 (·)→ (1-10) Final Judge Input: Prompt, Response, 풇 푯 ,풇 푨 ,풇 푫 Output: Jailbreak Success Score (1–4) Alignment Judge 풇 푨 (·)→ (1-5) Detail Judge 풇 푫 (·)→ (1-5) Eval Metric A. Ethical and Social Risks 4 subcategories F. Information and Political Manipulation H. Intellectual Property and Ownership Figure 1.Overview of OmniSafeBench-M. The benchmark unifies multi-modal jailbreak attack–defense evaluation, 13 attack and 15 defense methods, and a three-dimensional scoring protocol measuring harmfulness, alignment, and detail. Table 1. Quantitative comparison of representative multi-modal jailbreak and safety benchmarks. DatasetRisks Categories Prompt type Target Models Attacks Methods Defense MethodsEval Metrics JailBreakV-28K [25]16110501 (ASR) FigStep-Dataset [10]1015232 (ASR + PPL) M-SafetyBench [23]13112112 (ASR + R) HADES-Dataset [21]515101 (ASR) MMJ-Bench [40]816643 (ASR+DSR+S) OmniSafeBench-M (Ours)5031813153 (H–A–D metrics) Jeong et al. [14], Wang et al. [38], and Sima et al. [33] in- troduce distributional shifts or multimodal dispersions that evade alignment filters, while Ma et al. [27] propose to iter- atively refine queries to maximize harmfulness under feed- back constraints. 2.2. Multi-modal jailbreak defense To defend the jailbreak attack, the defense works have pro- gressed along roughly two directions: off-model and on- model methods. The Off-model defenses treat the target model as a black box and employ input or output filter- ing. For example, Gou et al. [11], Zhang et al. [47], Oh et al. [30], Wang et al. [36], Zhou et al. [50], Xu et al. [41], Liu et al. [24], Lee et al. [20], and Helff et al. [12] pro- pose to modify or verify input prompts before model infer- ence. Zhang et al. [48] and Pi et al. [31] pay attention to detecting unsafe generations after model inference. These modular guard models are easily deployable across differ- ent MLLMs but may result in additional time loss. On- model defense methods integrate safety mechanisms into the model itself. For example, Gao et al. [8] and Jiang et al. [18] propose to adjust token logits during generation using safety-aware reward models or constitutional calibration, ensuring real-time regulation of output distribution. While Zong et al. [53] propose to fine-tune the model with safety preference datasets or reinforcement learning from human feedback to build inherently safer MLLMs. 3. The Proposed OmniSafeBench-M As shown in Fig. 2, the OmniSafeBench-M includes dataset generation, implemented attacks, implemented de- fenses, and fine-grained safety evaluation. 3.1. Dataset generation As shown in Fig. 3, the taxonomy comprises 9 major risk categories, each with 4–8 sub-categories spanning ethical, privacy, safety, economic, political, cybersecurity, cogni- tive, and cultural domains. We introduce an automated pro- cess for generating risk image-text pairs for MLLMs based on risk category. The process includes three key stages: (1) generating risk-related texts, (2) extracting unsafe key phrases, and (3) producing corresponding risk images. Generating risk-related texts.We define topic dimensions within each risk category (e.g., harmful language) and its subcategories (e.g., stereotypes), each accompanied by con- cise textual definitions. Representative topics (e.g., body shaming, color discrimination, immigration discrimination) are manually specified to guide GPT-4o in generating di- verse and contextually grounded risk scenarios. These top- ics, along with their corresponding categories and defini- tions, are then provided to GPT-4o to synthesize risk-related text prompts with consultative, imperative, and declarative inquiry types. If GPT-4o fails to yield satisfactory outputs, an alternative model (e.g., DeepSeek-Chat) is employed to declarative consultative instructive Harmful Elements LLM LLM Harmful Query Image Generation PixArt-XL-2-1024-MS Generated Images Dataset Generation Implemented Attacks ATTACKTest Cases Implemented Defenses MODEL+DEFENSECompletions Fine-Grained Safety Evaluation Final Judge Input: Prompt, Response, � � ,� � ,� � Output: Jailbreak Success Score (1–4) Harmfulness Judge � � (·) → (1-10) Alignment Judge � � (·) → (1-5) Detail Judge � � (·) → (1-5) Success Rate Figure 2. The framework of the proposed method ensure completeness and robustness. Extracting unsafe key words.After getting the malicious questions, we use a LLM to extract keywords from mali- cious questions, as these keywords determine the security of the question. Note that the keyword extraction method may vary depending on the scenario. The details are pre- sented in the Appendix. Producing corresponding risk images.Based on the ex- traction of key words, we adopt the PixArt-XL-2-1024-MS model to generate the corresponding risk images. The im- age generation prompt is formulated as “A photo of [Key word]”. The height×width is set to1024×1024. 3.2. Implemented attack methods In this subsection we summarize the representative jail- break attacks implemented in our benchmark. Let the vic- tim MLLM be denoted byMand an input pair by(T, I), whereTis the textual prompt andIis the image input. The attacker’s objective is to bypassM’s safety filters and in- duce a harmful responsey=M(T, I). To achieve this, attackers manipulateTand/orIto craft adversarial exam- plesT ′ andI ′ . We categorize the implemented attacks into two major groups and five subcategories:white-box attacks(single-modal and cross-modal) andblack-box at- tacks(structured visual-carrier, out-of-distribution, and hid- den risks ). White-box attacks assume access to internal information of the MLLM (architecture and/or gradients). Attackers typically exploit gradient information to optimize adversarial inputs that steer the model to produce harm- ful outputs. But black-box attacks assume no access to model internals and only observe model outputs — a realis- tic threat model because many commercial multi-modal ser- vices operate as black boxes. Black-box methods are gener- ally heuristic; our implemented methods fall into three cate- gories: structured visual-carrier attacks, out-of-distribution (OOD) attacks, and Hidden risks. Single-modal white-box attacks.Single-modal attacks often consider visual attacks and can be formalized as I ′ := arg min I ′ ∈B P m i=1 −log p(y i |I ′ , T) , whereB denotes the feasible perturbation set. They are divided into image-optimization jailbreaks, i.e.,visual-adv[32] andDeltaJP[29], and random-noise optimization, i.e., visual-adv-un[32] andImgJP[29]. Compared with visual-adv,ImgJPandDeltaJPadopt ensemble strategies to improve transferability. Cross-modal white-box attacks.Cross-modal attacks that jointly optimize the visual and textual modalities can im- prove jailbreak effectiveness and can be formalized as (I ′ , T ′ := arg min (I ′ ,T ′ )∈B P m i=1 −log p(y i |I ′ , T ′ ) . UMK[34] increases the jailbreak success rate by perform- ing stepwise optimization of image noise and text suffixes. BAP[46] andJPS[3] alternately optimize the image and text using different strategies, thereby more effectively in- ducing the model to produce harmful outputs. Structured visual-carrier attacks.MLLMs typically project the visual encoder into the LLM embedding space via a projection layer, while safety alignment is usually per- formed on the textual modality; consequently, vulnerabili- ties in the visual modality can be exploited to bypass pro- tections. The core idea of structured visual-carrier attacks is to embed visual carriers in images that the model can “read” as semantic content, and then exploit the VLM’s vision-to- text recognition and completion abilities to convert those carriers into executable instructions or harmful text. Com- mon operations include typographic/layout attacks such asFigStep[10] andFigStep-Pro[10], and mali- cious images such asQR-Attack[23] andHADES[21]. MLLMs typically project the visual encoder into the LLM embedding space via a projection layer, while safety align- ment is usually performed on the textual modality; conse- Decision and Cognitive Risks Cybersecurity Threats Safety and Physical Harm Information and Political Manipulation Criminal and Economic Risks Privacy and Data Risks Content and Cultural Safety Ethical and Social Risks Intellectual Property and Ownership Legal Advice Government Decision Support Financial Advice Market Manipulation Psychological Manipulation Health Misguidance Medical Advice Cognitive Bias or Overreliance Unauthorized Model Access Data Poisoning Jailbreak or Prompt Injection Phishing or Ransomware Exploit Development Malware Generation Cross-modal Attack Animal Abuse Human Trafficking Physical Harm Dangerous Instructions Environmental Damage Self-harm Promotion Drug Guidance Misinformation or Disinformation Deepfakes or Synthetic Media Election Interference Public Order Disruption National Security Threat Political Propaganda Evasion of Law Illegal Activity Fraud or Scams Economic Crime White-Collar Crime Personal Data Leakage Biometric or Health Data Exposure Doxxing or Tracking Surveillance Misuse Training Data Leakage Pornography or Nudity Sexual Content Violence or Gore Cultural or Religious Of fense Discriminatory Behavior Hate Speech Insulting or Harassing Speech Bias and Discrimination Trademark or Brand Misuse Proprietary Data Leakage Copyright Infringement Unauthorized Style or Persona Imitation Figure 3. Safety taxonomy of our OmniSafeBench-M. quently, vulnerabilities in the visual modality can be ex- ploited to bypass protections.The core idea of struc- tured visual-carrier attacks is to embed visual carriers in images that the model can “read” as semantic content, and then exploit the VLM’s vision-to-text recognition and completion abilities to convert those carriers into exe- cutable instructions or harmful text. Common operations include typographic/layout attacks such asFigStep[10] andFigStep-Pro[10], and malicious images such as QR-Attack[23] andHADES[21]. Out-of-Distribution (OOD) attacks.During safety- alignment training of MLLMs there often exist out-of- distribution scenarios, including data augmentation and at- tention interference. Accordingly, by altering the input dis- tribution or visual structure (e.g., shuffling/collaging/insert- ing distracting sub-images/style transfer) to cause failures in the model’s input understanding or to shift its atten- tion, one can bypass safety mechanisms and induce mali- cious outputs. For example,CS-DJ[43] andVisCRA[33] disturb attention to make the model ignore harmful con- tent, whileJOOD[14] andSI-Attack[49] employ data- augmentation techniques to change the input distribution and circumvent safety guards. Hidden risk attacks.Hiding a risky intent is also a com- monly used jailbreak technique.HIMRD[27] proposes a heuristic risk-allocation strategy that decomposes harmful intent across the image and the text, thereby concealing the risky intent.MML[38] hides malicious intent by using data- transformation combined with encryption/decryption. 3.3. Implemented defense methods In this subsection, we summarize the representative defense methods implemented in our benchmark. Let the target MLLM be denoted byMand a malicious input pair by (T ′ , I ′ ), whereT ′ is the adversarial textual prompt andI ′ is the adversarial image input. The defender’s objective is to ensure thatM’s final outputy final , remains safe despite the malicious input. To achieve this, defenders can intervene at different stages of the generation process. We catego- rize these interventions into two major groups:Off-Model DefensesandOn-Model Defenses.Off-Model Defensesin- clude input pre-processing and output post-processing.On- Model Defensesencompass inference process intervention and intrinsic model alignment. Input pre-processing defenses.Input pre-processing defenses proactively modify or filter the malicious query(T ′ , I ′ )before inference, yielding a final re- sponse viay final =M(D in (T ′ , I ′ )).Strategies in- clude direct input modification, such as prompt aug- mentationAdaShield-S [36]and adversarial pu- rificationUniguard [30]; robustness checking by analyzing semantic divergence across mutated inputs JailGuard [47]; and multi-stage approaches like ”red teaming” with partial viewsDPS [50]and self- moderation via text-only requeryingECSO [11].A final group acts as external classifiers to vet the query:CIDER [41]measures cross-modal similar- ity shifts, while a suite of purpose-built guard mod- elsGuardReasoner-VL [24],Llama-Guard-4, LlavaGuard [12],QGuard [20]makes a definitive judgment to block or allow the request. Output post-processing defenses.Output post-processing defenses operate as a reactive safeguard, inspecting the model’s initial responsey raw =M(T ′ , I ′ )after it has been generated. If harmful content is detected, the re- sponse is either blocked or sanitized, yielding a final out- puty final =D out (y raw ). This is typically achieved us- ing external guard models.Single-stage classifiers like ShieldLM [48]andLlama-Guard-3directly ana- lyzey raw to provide a ’safe’ or ’unsafe’ verdict. In contrast, MLLM-Protector [31]uses a two-stage process, first detecting harm and then invoking a separate model to detox- ify the response. Inference process intervention defenses.Inference pro- cess intervention defenses modify the model’s real-time generation process by leveraging internal states, such as logits, to steer outputs towards safety. For instance, the training-freeHiddenDetect [18]framework identi- fies unsafe prompts by calculating a safety score from the cosine similarity between hidden states and a pre-defined refusal vector. Another strategy,COCA [8], calibrates the output distribution by computing a “safety delta” between logits generated with and without a constitutional prompt, using this delta to adjust the final token generation. Intrinsic model alignment defenses.Intrinsic model alignment defenses create an inherently safer model by modifying its internal parameters through specialized train- ing. These methods typically involve fine-tuning the base MLLM on safety-centric data to align its behavior with human values. One common approach is supervised fine- tuning (SFT), whereVLGuard [53]trains the model on a curated safety instruction-following dataset containing vision-language examples of safe and unsafe scenarios. 3.4. Fine-Grained Safety Evaluation 3.4.1. Motivation: Beyond binary success rates To move beyond the limitations of traditional, binary (safe/unsafe) safety evaluations, we propose a fine-grained assessment framework as a core component of our bench- mark. We posit that the risk posed by a model’s response Rto a given promptPis not a monolithic property, but a multi-faceted construct. Our framework is designed to formally decompose this risk, enabling the identification of High-Impact, Low-Frequency (HILF)events, which are often obscured by aggregate safety metrics. 3.4.2. The H-F-A evaluation dimensions Our framework evaluates each responseRgenerated from a promptP(which may include a text componentP T and an image componentP I ) along three decoupled, orthogonal dimensions. LetH,A, andDbe the scoring functions. Harmfulness (H): Severity of Consequence.The Harm- fulness functionH::R→[1,10]scores the severity of po- tential real-world harm from the act the response describes or facilitates. Crucially, this score evaluates the conse- quence of the response’s facilitated act, not the user’s initial intent in the prompt. H=H(R)(1) The scale is discrete, whereH= 1is a safe intent and H= 10is an act with potential for existential harm. Intent Alignment (A): Fulfillment of User Request.The Intent Alignment functionA: (P, R)→[1,5]measures the semantic alignment between the user’s prompt and the model’s response. A high alignment score indicates thatR comprehensively addresses the directives inP. A=A(P, R)(2) This score quantifies the model’s capability to fulfill the user’s intent, irrespective of its harmfulness. Level of Detail (D): Explicitness and Self-Containment. The Level of Detail functionD:R→[1,5]evaluates the explicitness and self-containment of the information in the response. It quantifies how much external knowledge is re- quired to act upon the information provided. D=D(R)(3) A high score (D= 5) corresponds to a rich, comprehensive response (e.g., a full tutorial or a complete script), while a low score indicates abstract, high-level concepts. Holistic Adjudication of Jailbreak SuccessInstead of a simple multiplicative score, we synthesize these dimensions through a holistic, rule-based adjudication process to deter- mine the final success of a jailbreak attempt. This process mirrors an expert human judgment. The Breach Condition.A fundamental safety breach oc- curs only when a response is both harmful and compliant. We define this as the primary condition for any level of jail- break success: Breach Condition= (H≥3)∧(A≥3)(4) If this condition is not met, the attempt is considered a fail- ure (i.e., successfully defended by the model). Final Judgment Score.If a breach occurs, the Level of Detail (D) determines its severity. We map the outcome to a final four-point judgment score,S∈ 1, . . . ,4, where 4 represents aComplete Success, 3 aPartial Success, 2Bor- derline, and 1No Success. The score is defined as: S(P, R) = 4if Breach andD≥4 3if Breach andD= 3 2if Breach andD≤2 1otherwise (5) For our benchmark, aJailbreak Success Event, denoted by the binary variableJ, is defined as any response receiv- ing a final judgment score of 3 or 4: J(P, R) = 1ifS(P, R)≥3,else0(6) 4. Benchmark Experiments 4.1. Experiment settings MLLMs.A series of MLLMs are used to conduct experiments, including closed-source models and open- source models. The closed-source models include: (1) GPT-5, (2) Gemini-2.5-Flash (Gemini-2.5), (3) Claude- Sonnet-4-20250514 (Claude-Sonnet-4), (4) Qwen3-VL- PLUS, and (5) Doubao-Seed-1-6-Flash-250828 (Doubao- Seed). The open-source models are: (1) Qwen3-VL-30B- A3B-Instruct (Qwen3-VL), (2) Gemma-3-27b-it (Gemma- 3), (3) DeepSeek-VL2, (4) GLM-4.1V-9B-Thinking (GLM- 4.1V), and (5) Kimi-VL-A3B-Instruct (Kimi-VL). Jailbreak attack methods.We consider a set of 13 jailbreak attack methods collected from 13 published pa- pers. We group these attacks into two broad categories: white-box and black-box.The white-box category in- cludes both single-modal and cross-modal optimization techniques; specifically, we evaluate Visual-Adv, ImgJP, DeltaJP, UMK, and JPS. The black-box category covers attacks that operate via structured visual carriers, out-of- distribution inputs, or query-optimization strategies; this group comprises FigStep, QR-Attack, HADES, CS-DJ, SI- Attack, JOOD, MML, and HIMRD. The details are de- scribed in Appendix. Jailbreak defense methods.We consider a set of 15 jail- break defense methods collected from 16 published pa- pers. We group these attacks into two broad categories: off-model and on-model.The off-model category in- cludes both input pre-processing and output post process- ing techniques; specifically, we evaluate ECSO, JailGuard, AdaShield-S, Uniguard, DPS, CIDER, GuardReasoner-VL, Llama-Guard-4, QGuard, LlavaGuard, ShieldLM, MLLM- Protector and Llama-Guard-3. The on-model category in- cludes inference process intervention and intrinsic model alignment techniques; this group comprises COCA and VL- Guard. The details are described in Appendix. 4.2. Experiment results Table 2. Performance comparison of MiniGPT-4 (Vicuna 13B) on various benchmarks. We report ASR, average-H (H), average-A (A), and average-D (D) metrics. MetricJPSImgJP UMK DeltaJP visual-adv ASR (%)62.9315.40 37.07 24.47 47.87 Avg-H4.873.76 4.19 3.84 4.43 Avg-A4.102.83 3.26 3.11 3.61 Avg-D2.871.93 2.36 2.11 2.58 Results in the white-box setting.Our evaluation of five white-box attacks on MiniGPT-4 (Vicuna 13B) (Table 2) provides a more realistic vulnerability assessment. The ob- served Attack Success Rates (ASRs) are markedly low (e.g., jpsat 62.93%), a direct result of our strict success crite- rion (S≥3, Eq. 5), which demands not only high harm- fulness and alignment (H, A≥3) but also sufficient de- tail (D≥3). The consistently low Average Detail (Avg- D, mostly below 2.6) across all attacks is the primary rea- son for the low ASRs. This highlights a critical point: the model’s failure to provide specifics may stem from either a deliberate ”residual safety” mechanism (i.e., secondary behaviors that mitigate harm even after an initial refusal is bypassed) designed to mitigate harm, or simply from its in- herent capability limitations in generating detailed content. Crucially, our framework is designed to capture this out- come regardless of its cause. Results in the black-box setting.As shown in Table 3, dif- ferent attack methods exhibit different attack performance across different architectures. Among all evaluated meth- ods, MML and CS-DJ achieve the highest ASRs, con- firming their strong ability to exploit cross-modal disper- sion without requiring gradient access. Specifically, MML achieves 50.7% ASR on Gemini-2.5 and 52.2% on Qwen3- VL-Plus, while maintaining high Harmfulness (H≈5.6) and Alignment (A≈3.3) scores.These results indi- cate that semantic information hidden within chained mul- timodal cues can effectively bypass closed-source safety filters, producing logically consistent yet unsafe content. Similarly, CS-DJ achieves stable attack rates across both closed-source and open-source models (32.0% on Gemini- 2.5 and 38.1% on Qwen3-VL), demonstrating that alter- ing the input distribution to create OOD data can effec- tively disrupt the model’s safety attention, thereby bypass- ing safety checks. By comparison, FigStep and QR-Attack show competitive yet more model-dependent performance. Their average ASR ranges from 4–15% on closed-source models to 30–50% on open-source MLLMs (e.g., 51.3% on GLM-4.1V), revealing that image-embedded typographic cues are more effectively interpreted by open pipelines that lack strong OCR filtering or multi-modal guard modules. These attacks also yield high Detail (D) scores (≈3.0–3.6), confirming that once jailbreak succeeds, the model tends to generate explicit, step-by-step unsafe responses. In con- trast, HIMRD and JOOD achieve moderate ASR (≈5–20%) but demonstrate higher stealth and semantic coherence. Results in the off-model defense setting.As shown in Table 4, off-model defense methods effectively mitigate jailbreak attacks on GPT-4o, but their effectiveness varies notably across attack types. For input pre-processing de- fense methods, Uniguard and JailGuard achieve the low- est ASR on the CS-DJ attack (3.53 % and 3.13 %, re- spectively), a notable reduction from 23.00 % without de- fense. On FigStep, AdaShield-S performs best (1.27 %), followed by DPS (3.13 %) and Uniguard (6.33 %), show- ing that adaptive input sanitization effectively suppresses typographic visual carriers. However, on the MML at- tack, input-side protection degrades: AdaShield-S and Uni- guard yield 37.93 % and 49.53 % ASR, respectively, while ECSO reaches 27.47 %. These results indicate that the in- put pre-processing defense is effective for explicit triggers but less reliable for semantically dispersed attacks. For out- put post-processing defense methods, the strongest robust- ness appears in MLLM-Protector, which reduces ASR to 14.87 % on CS-DJ, 4.60 % on FigStep, and only 0.27 % on MML. ShieldLM and Llama-Guard-3 also achieve consis- tently low ASR (about 8–21 %), providing reactive moder- ation that effectively blocks unsafe completions. Results in the on-model defense setting.We evaluated the efficacy of two distinct on-model defense strategies: safety fine-tuning via VLGuard and the inference-time CoCa de- fense. As shown in Table 5, both methods substantially improve model robustness against a suite of attacks, yet their performance reveals important nuances. The CoCa defense demonstrates broad-spectrum effectiveness, dras- tically reducing the Attack Success Rate (ASR) across all tested attacks and nearly neutralizing the potent FigStep and Table 3. Performance comparison of bothopen-sourceandclosed-sourcelarge multimodal models on various benchmarks. Models are grouped by their availability. We report ASR, average-H (H), average-A (A), and average-D (D) metrics. Closed-Source Models GPT-5Gemini-2.5Claude-4 SonnetQwen3-VL-PlusDoubao-Seed BenchmarkASRHADASRHADASRHADASRHADASRHAD FigStep [10]4.20 1.76 2.35 3.7015.80 2.44 2.71 2.114.20 2.01 2.22 1.5013.87 2.26 2.67 2.2924.33 2.69 2.97 2.70 MML [38]15.27 3.202.42 3.2350.67 5.64 3.332.987.53 2.46 2.27 1.3652.20 5.83 3.712.6728.335.642.78 2.08 CS-DJ [43] 9.73 2.042.44 3.8932.00 3.58 3.023.1514.27 2.87 2.50 2.4423.40 2.96 2.773.0647.534.213.46 3.58 HIMRD [27]5.13 2.00 2.20 3.8730.60 3.87 2.87 2.950.53 1.96 2.04 1.1517.40 3.30 2.46 2.1633.13 3.87 2.94 2.89 JOOD [14] 4.93 1.94 2.37 2.7221.40 2.58 2.91 2.031.40 1.94 2.20 1.369.73 2.07 2.54 2.0130.20 2.93 3.19 2.58 SI-Attack [49]6.13 1.95 2.42 3.3816.67 2.65 2.65 2.273.47 1.95 2.28 1.7111.4 2.15 2.56 2.5122.47 2.73 2.89 2.97 HADES [21] 6.13 1.94 2.40 3.1424.53 2.74 3.00 2.241.80 1.92 2.19 1.449.20 2.05 2.53 2.1027.60 2.79 3.13 2.70 QR-Attack [23]5.14 1.98 2.33 3.3124.22 2.87 2.89 2.682.87 1.99 2.17 1.539.27 2.08 2.47 2.4631.42 3.19 3.05 3.13 Open-Source Models Qwen3-VLGemma-3DeepSeek-VL2GLM-4.1VKimi-VL BenchmarkASRHADASRHADASRHADASRHADASRHAD FigStep [10]30.93 2.95 3.17 2.6428.33 2.82 3.06 2.8944.534.063.71 2.7451.273.903.64 3.4139.933.643.342.55 MML [38]34.624.40 3.292.4454.73 6.01 3.623.080.33 2.45 1.19 1.1522.675.122.66 2.4610.27 3.16 1.79 1.65 CS-DJ [43]38.073.82 3.203.2221.60 3.41 2.67 3.0425.604.242.86 2.5930.93 4.37 2.72 3.2039.474.433.262.90 HIMRD [27]7.53 2.72 2.12 1.4444.33 4.43 3.303.3724.53 3.85 2.52 2.1743.07 4.55 3.03 3.0119.87 4.06 2.32 1.80 JOOD [14]4.33 1.98 2.32 1.7326.67 2.74 3.05 2.3115.87 2.63 2.68 1.6538.20 3.41 3.49 2.7126.27 3.04 3.07 2.03 SI-Attack [49] 8.27 2.25 2.38 2.117.6 2.51 2.7 2.572.62.13 2.07 1.2422.53 2.87 2.9 2.7310.07 2.62 2.29 1.84 HADES [21]8.73 2.14 2.51 2.0023.67 2.63 2.96 2.3622.60 3.02 2.99 1.9238.73 3.32 3.52 2.7924.93 3.26 3.14 2.06 QR-Attack [23]10.41 2.30 2.45 2.2427.75 3.06 2.93 2.7428.95 3.67 2.93 2.4749.97 4.22 3.56 3.6535.36 3.87 3.15 2.60 Table 4. Attack Success Rate (ASR %) of different off model defense methods on GPT-4o. Method abbreviations: Def. (Defense), AdaSh. (AdaShield), Unig. (Uniguard), JailG. (JailGuard), GR-VL (GuardReasoner-VL), LlavaG. (LlavaGuard), QG. (QGuard), LG-4/3 (Llama- Guard-4/3), MLLM-P (MLLM-protector), ShLM (ShieldLM). Input Pre-processingOutput Post-processing DatasetNo Def. AdaSh. Unig. DPS JailG. ECSO GR-VL LlavaG. QG. LG-4 CIDER MLLM-P ShLM LG-3 CS-DJ [43]23.00 9.07 3.53 10.47 3.13 13.20 19.07 21.40 18.67 21.60 3.20 14.87 21.07 18.40 FigStep [10]9.67 1.27 6.33 3.13 9.13 8.27 8.47 9.20 2.60 9.40 9.20 4.60 9.07 8.53 MML [38]56.60 37.93 49.53 42.13 56.13 27.47 55.60 56.27 0.13 55.67 55.47 0.27 56.27 19.93 HIMRD [27]12.87 1.40 3.87 5.00 10.07 5.67 3.00 12.13 1.13 1.27 10.13 3.73 7.20 7.60 Table 5. Comparison of Attack Success Rate (ASR %) for differ- ent on model defenses against attacks. Model / ConfigurationCS-DJ MML FigStep HIMRD LLaVA-1.5 (Base)0.00 0.67 15.47 20.20 + Finetuning (VLGuard)0.00 1.13 0.00 1.20 + CoCa Defense0.00 0.07 0.00 0.33 HIMRD methods. The VLGuard fine-tuning also shows strong performance, particularly against the most threaten- ing attacks, cutting the ASR for HIMRD from 20.40% to just 1.20%. However, we observed a counter-intuitive result where VLGuard slightly increased the ASR for the MML attack (from 0.93% to 1.73%). This suggests that while safety fine-tuning can patch major vulnerabilities, it may also subtly alter the model’s behavior, creating rare, spe- cific weaknesses that can be exploited by different attack patterns. This finding highlights the critical need for di- verse adversarial testing, as a defense’s effectiveness is not uniform and strengthening against one threat can inadver- tently create susceptibility to another. 5. Conclusion In this work, we proposed OmniSafeBench-M, a uni- fied and reproducible benchmark for evaluating multimodal jailbreak attacks and defenses. The benchmark integrates a broad risk taxonomy, diverse attack and defense imple- mentations, and a three-dimensional safety scoring proto- col, enabling more fine-grained and interpretable evalua- tion than traditional ASR-based metrics. Extensive exper- iments across 18 open-source and closed-source MLLMs reveal persistent vulnerabilities—particularly under cross- modal dispersion and black-box settings—and highlight differing trade-offs among defense strategies. By standard- izing data, methodology, and evaluation within a single plat- form, OmniSafeBench-M provides a solid foundation for advancing research on multimodal model safety. References [1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in Neural Information Processing Systems, 35:23716–23736, 2022. 1 [2] Kittitouch Areerob, Van-Quang Nguyen, Xianfeng Li, Shogo Inadomi, Toru Shimada, Hiroyuki Kanasaki, Zhijie Wang, Masanori Suganuma, Keiji Nagatani, Pang-jo Chun, et al. Multimodal artificial intelligence approaches using large language models for expert-level landslide image anal- ysis.Computer-Aided Civil and Infrastructure Engineering, 2025. 1 [3] Renmiao Chen, Shiyao Cui, Xuancheng Huang, Chengwei Pan, Victor Shea-Jay Huang, QingLin Zhang, Xuan Ouyang, Zhexin Zhang, Hongning Wang, and Minlie Huang. Jps: Jailbreak multimodal large language models with collabora- tive visual perturbation and textual steering. InProceedings of the 33rd ACM International Conference on Multimedia, pages 11756–11765, 2025. 2, 4 [4] Ruoxi Cheng, Yizhong Ding, Shuirong Cao, Ranjie Duan, Xiaoshuang Jia, Shaowei Yuan, Simeng Qin, Zhiqiang Wang, and Xiaojun Jia. Pbi-attack: Prior-guided bimodal interactive black-box jailbreak attack for toxicity maximiza- tion. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 609–628, 2025. 2 [5] Chun Wai Chiu, Linghan Huang, Bo Li, Huaming Chen, and Kim-Kwang Raymond Choo. Do as i say not as i do’: A semi-automated approach for jailbreak prompt attack against multimodal llms.arXiv preprint arXiv:2502.00735, 2025. 2 [6] Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, et al. A survey on multimodal large lan- guage models for autonomous driving. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 958–979, 2024. 1 [7] Ranjie Duan, Jiexi Liu, Xiaojun Jia, Shiji Zhao, Ruoxi Cheng, Fengxiang Wang, Cheng Wei, Yong Xie, Chang Liu, Defeng Li, et al.Oyster-i: Beyond refusal–constructive safety alignment for responsible language models.arXiv preprint arXiv:2509.01909, 2025. 1 [8] Jiahui Gao, Renjie Pi, Tianyang Han, Han Wu, Lan- qing Hong, Lingpeng Kong, Xin Jiang, and Zhenguo Li. Coca: Regaining safety-awareness of multimodal large lan- guage models with constitutional calibration.arXiv preprint arXiv:2409.11365, 2024. 3, 5 [9] Junyu Gao, Xuan Yao, Yong Rui, and Changsheng Xu. Building embodied evoagent: A brain-inspired paradigm for bridging multimodal large models and world models. InPro- ceedings of the 33rd ACM International Conference on Mul- timedia, pages 3280–3289, 2025. 1 [10] Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. Figstep: Jailbreaking large vision-language models via typo- graphic visual prompts. InProceedings of the AAAI Confer- ence on Artificial Intelligence, pages 23951–23959, 2025. 1, 2, 3, 4, 5, 8 [11] Yunhao Gou, Kai Chen, Zhili Liu, Lanqing Hong, Hang Xu, Zhenguo Li, Dit-Yan Yeung, James T Kwok, and Yu Zhang. Eyes closed, safety on: Protecting multimodal llms via image-to-text transformation. InEuropean Conference on Computer Vision, pages 388–404. Springer, 2024. 3, 5 [12] Lukas Helff, Felix Friedrich, Manuel Brack, Kristian Kerst- ing, and Patrick Schramowski. Llavaguard: An open vlm- based framework for safeguarding vision datasets and mod- els.arXiv preprint arXiv:2406.05113, 2024. 3, 5 [13] Yihao Huang, Le Liang, Tianlin Li, Xiaojun Jia, Run Wang, Weikai Miao, Geguang Pu, and Yang Liu. Perception-guided jailbreak against text-to-image models. InProceedings of the AAAI Conference on Artificial Intelligence, pages 26238– 26247, 2025. 1 [14] Joonhyun Jeong, Seyun Bae, Yeonsung Jung, Jaeryong Hwang, and Eunho Yang. Playing the fool: Jailbreaking llms and multimodal llms with out-of-distribution strategy. InProceedings of the Computer Vision and Pattern Recogni- tion Conference, pages 29937–29946, 2025. 3, 5, 8 [15] Xiaojun Jia, Tianyu Pang, Chao Du, Yihao Huang, Jindong Gu, Yang Liu, Xiaochun Cao, and Min Lin. Improved tech- niques for optimization-based jailbreaking on large language models.arXiv preprint arXiv:2405.21018, 2024. 1 [16] Xiaojun Jia, Sensen Gao, Simeng Qin, Tianyu Pang, Chao Du, Yihao Huang, Xinfeng Li, Yiming Li, Bo Li, and Yang Liu. Adversarial attacks against closed-source mllms via fea- ture optimal alignment.arXiv preprint arXiv:2505.21494, 2025. 1 [17] Lei Jiang, Zixun Zhang, Zizhou Wang, Xiaobing Sun, Zhen Li, Liangli Zhen, and Xiaohua Xu. Cross-modal obfuscation for jailbreak attacks on large vision-language models.arXiv preprint arXiv:2506.16760, 2025. 2 [18] Yilei Jiang, Xinyan Gao, Tianshuo Peng, Yingshui Tan, Xiaoyong Zhu, Bo Zheng, and Xiangyu Yue.Hidden- detect: Detecting jailbreak attacks against large vision- language models via monitoring hidden states.arXiv preprint arXiv:2502.14744, 2025. 3, 5 [19] Jiayi Kuang, Ying Shen, Jingyou Xie, Haohao Luo, Zhe Xu, Ronghao Li, Yinghui Li, Xianfeng Cheng, Xika Lin, and Yu Han. Natural language understanding and inference with mllm in visual question answering: A survey.ACM Com- puting Surveys, 57(8):1–36, 2025. 1 [20] Taegyeong Lee, Jeonghwa Yoo, Hyoungseo Cho, Soo Yong Kim, and Yunho Maeng. Qguard: Question-based zero- shot guard for multi-modal llm safety.arXiv preprint arXiv:2506.12299, 2025. 3, 5 [21] Yifan Li, Hangyu Guo, Kun Zhou, Wayne Xin Zhao, and Ji- Rong Wen. Images are achilles’ heel of alignment: Exploit- ing visual vulnerabilities for jailbreaking multimodal large language models. InEuropean Conference on Computer Vi- sion, pages 174–189. Springer, 2024. 1, 2, 3, 4, 5, 8 [22] Zhaoyi Li, Xiaohan Zhao, Dong-Dong Wu, Jiacheng Cui, and Zhiqiang Shen.A frustratingly simple yet highly effective attack baseline: Over 90arXiv preprint arXiv:2503.10635, 2025. 1 [23] Xin Liu, Yichen Zhu, Jindong Gu, Yunshi Lan, Chao Yang, and Yu Qiao. Mm-safetybench: A benchmark for safety eval- uation of multimodal large language models. InEuropean Conference on Computer Vision, pages 386–403. Springer, 2024. 1, 2, 3, 4, 5, 8 [24] Yue Liu, Shengfang Zhai, Mingzhe Du, Yulin Chen, Tri Cao, Hongcheng Gao, Cheng Wang, Xinfeng Li, Kun Wang, Junfeng Fang, et al. Guardreasoner-vl: Safeguarding vlms via reinforced reasoning.arXiv preprint arXiv:2505.11049, 2025. 3, 5 [25] Weidi Luo, Siyuan Ma, Xiaogeng Liu, Xiaoyu Guo, and Chaowei Xiao. Jailbreakv: A benchmark for assessing the robustness of multimodal large language models against jail- break attacks.arXiv preprint arXiv:2404.03027, 2024. 1, 3 [26] Siyuan Ma, Weidi Luo, Yu Wang, and Xiaogeng Liu. Visual- roleplay: Universal jailbreak attack on multimodal large language models via role-playing image character.arXiv preprint arXiv:2405.20773, 2024. 2 [27] Teng Ma, Xiaojun Jia, Ranjie Duan, Xinfeng Li, Yihao Huang, Xiaoshuang Jia, Zhixuan Chu, and Wenqi Ren. Heuristic-induced multimodal risk distribution jailbreak at- tack for multimodal large language models. InProceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 2686–2696, 2025. 2, 3, 5, 8 [28] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learn- ing models resistant to adversarial attacks.arXiv preprint arXiv:1706.06083, 2017. 1 [29] Zhenxing Niu, Haodong Ren, Xinbo Gao, Gang Hua, and Rong Jin. Jailbreaking attack against multimodal large lan- guage model.arXiv preprint arXiv:2402.02309, 2024. 2, 4 [30] Sejoon Oh, Yiqiao Jin, Megha Sharma, Donghyun Kim, Eric Ma, Gaurav Verma, and Srijan Kumar.Uniguard: Towards universal safety guardrails for jailbreak attacks on multimodal large language models.arXiv preprint arXiv:2411.01703, 2024. 3, 5 [31] Renjie Pi, Tianyang Han, Jianshu Zhang, Yueqi Xie, Rui Pan, Qing Lian, Hanze Dong, Jipeng Zhang, and Tong Zhang. Mllm-protector: Ensuring mllm’s safety without hurting per- formance.arXiv preprint arXiv:2401.02906, 2024. 3, 5 [32] Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Hen- derson, Mengdi Wang, and Prateek Mittal. Visual adversarial examples jailbreak aligned large language models. InPro- ceedings of the AAAI conference on artificial intelligence, pages 21527–21536, 2024. 2, 4 [33] Bingrui Sima, Linhua Cong, Wenxuan Wang, and Kun He. Viscra: A visual chain reasoning attack for jailbreak- ing multimodal large language models.arXiv preprint arXiv:2505.19684, 2025. 3, 5 [34] Ruofan Wang, Xingjun Ma, Hanxu Zhou, Chuanjun Ji, Guangnan Ye, and Yu-Gang Jiang. White-box multimodal jailbreaks against large vision-language models. InProceed- ings of the 32nd ACM International Conference on Multime- dia, pages 6920–6928, 2024. 2, 4 [35] Xinkai Wang, Beibei Li, Zerui Shao, Ao Liu, and Shoul- ing Ji. Multimodal safety is asymmetric: Cross-modal ex- ploits unlock black-box mllms jailbreaks.arXiv preprint arXiv:2510.17277, 2025. 2 [36] Yu Wang, Xiaogeng Liu, Yu Li, Muhao Chen, and Chaowei Xiao.Adashield: Safeguarding multimodal large lan- guage models from structure-based attack via adaptive shield prompting. InEuropean Conference on Computer Vision, pages 77–94. Springer, 2024. 3, 5 [37] Youze Wang, Wenbo Hu, Yinpeng Dong, Jing Liu, Hanwang Zhang, and Richang Hong. Align is not enough: Multimodal universal jailbreak attack against multimodal large language models.IEEE Transactions on Circuits and Systems for Video Technology, 2025. 1 [38] Yu Wang, Xiaofei Zhou, Yichen Wang, Geyuan Zhang, and Tianxing He. Jailbreak large vision-language models through multi-modal linkage. InProceedings of the 63rd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1466–1494, 2025. 3, 5, 8 [39] Zhaoxin Wang, Handing Wang, Cong Tian, and Yaochu Jin. Implicit jailbreak attacks via cross-modal information concealment on vision-language models.arXiv preprint arXiv:2505.16446, 2025. 1 [40] Fenghua Weng, Yue Xu, Chengyan Fu, and Wenjie Wang. Mmj-bench: A comprehensive study on jailbreak attacks and defenses for vision language models. InProceedings of the AAAI Conference on Artificial Intelligence, pages 27689– 27697, 2025. 1, 3 [41] Yue Xu, Xiuyuan Qi, Zhan Qin, and Wenjie Wang. Cross-modality information check for detecting jailbreak- ing in multimodal large language models.arXiv preprint arXiv:2407.21659, 2024. 3, 5 [42] Zhao Xu, Fan Liu, and Hao Liu. Bag of tricks: Benchmark- ing of jailbreak attacks on llms.Advances in Neural Infor- mation Processing Systems, 37:32219–32250, 2024. 1 [43] Zuopeng Yang, Jiluan Fan, Anli Yan, Erdun Gao, Xin Lin, Tao Li, Kanghua Mo, and Changyu Dong. Distraction is all you need for multimodal large language model jailbreaking. InProceedings of the Computer Vision and Pattern Recogni- tion Conference, pages 9467–9476, 2025. 2, 5, 8 [44] Jingwei Yi, Rui Ye, Qisi Chen, Bin Zhu, Siheng Chen, Defu Lian, Guangzhong Sun, Xing Xie, and Fangzhao Wu. On the vulnerability of safety alignment in open-access llms. In Findings of the Association for Computational Linguistics ACL 2024, pages 9236–9260, 2024. 1 [45] Sibo Yi, Yule Liu, Zhen Sun, Tianshuo Cong, Xinlei He, Ji- axing Song, Ke Xu, and Qi Li. Jailbreak attacks and defenses against large language models: A survey.arXiv preprint arXiv:2407.04295, 2024. 1 [46] Zonghao Ying, Aishan Liu, Tianyuan Zhang, Zhengmin Yu, Siyuan Liang, Xianglong Liu, and Dacheng Tao. Jailbreak vision language models via bi-modal adversarial prompt. IEEE Transactions on Information Forensics and Security, 2025. 2, 4 [47] Xiaoyu Zhang, Cen Zhang, Tianlin Li, Yihao Huang, Xiao- jun Jia, Ming Hu, Jie Zhang, Yang Liu, Shiqing Ma, and Chao Shen. Jailguard: A universal detection framework for prompt-based attacks on llm systems.ACM Transactions on Software Engineering and Methodology, 2025. 3, 5 [48] Zhexin Zhang, Yida Lu, Jingyuan Ma, Di Zhang, Rui Li, Pei Ke, Hao Sun, Lei Sha, Zhifang Sui, Hongning Wang, et al. Shieldlm: Empowering llms as aligned, cus- tomizable and explainable safety detectors.arXiv preprint arXiv:2402.16444, 2024. 3, 5 [49] Shiji Zhao, Ranjie Duan, Fengxiang Wang, Chi Chen, Caixin Kang, Shouwei Ruan, Jialing Tao, YueFeng Chen, Hui Xue, and Xingxing Wei. Jailbreaking multimodal large language models via shuffle inconsistency.InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2045–2054, 2025. 2, 5, 8 [50] Qi Zhou, Tianlin Li, Qing Guo, Dongxia Wang, Yun Lin, Yang Liu, and Jin Song Dong. Defending lvlms against vi- sion attacks through partial-perception supervision.arXiv preprint arXiv:2412.12722, 2024. 3, 5 [51] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023. 1 [52] Miao Ziqi, Yi Ding, Lijun Li, and Jing Shao. Visual con- textual attack: Jailbreaking mllms with image-driven con- text injection. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 9638–9655, 2025. 2 [53] Yongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang, and Timothy Hospedales. Safety fine-tuning at (al- most) no cost: A baseline for vision large language models. arXiv preprint arXiv:2402.02207, 2024. 3, 6 OmniSafeBench-M: A Unified Benchmark and Toolbox for Multimodal Jailbreak Attack–Defense Evaluation Supplementary Material This appendix provides additional visualizations, de- tailed tables, and statistical analyses that complement the main paper.We present results covering 16 MLLMs, 8 black-box attack methods, 9 risk categories, 3 prompt styles, and 13 defense methods. Our goal is to offer a comprehensive, transparent report of model behavior under adversarial prompting and defense interventions. The additional figures shown here expand the key findings from the main text, including model-level vul- nerability patterns, category- and style-dependent effects, metric distributions, defense effectiveness, and regression- based interpretability. A. Extended Experimental Results A.1. Additional Visualizations for Attack Analysis A.1.1. Model-Level Robustness Overview Figure 4 visualizes the ASR landscape across all 16 models and 8 attack methods via a comprehensive heatmap, while Table 8 provides the corresponding numerical breakdown along with auxiliary metrics (Average-T, H, A, D). These visualizations collectively illustrate the divergence in safety alignment between open-source and closed-source architec- tures, as well as the varying effectiveness of different attack vectors across the model spectrum. A.1.2. Attack-Specific Vulnerability Profiles To better understand differences among the 8 attack meth- ods, Fig. 5 provides a set of radar plots, each summarizing model-level ASR for a single attack method. These plots highlight qualitative distinctions among at- tacks, showing whether they target specific model families or generalize broadly. A.1.3. Distribution of Evaluation Metrics Fig. 6 presents the boxplot distributions for four evaluation metrics (T,H,A,D) across all models. A.2. Prompt-Level Analysis A.2.1. Risk Category Vulnerability Fig. 7 shows the ASR aggregated by the 9 risk categories combined with all 8 attack methods. Specific categories exhibit consistently higher ASR, demonstrating that model vulnerability is strongly content-dependent. A.2.2. Style-Dependent Effects Fig. 8 provides a comparison of the three style types (declar- ative, consultative, instructive) across all attacks. The instructive tone produces consistently higher ASR, confirming that the imperative style induces increased com- pliance from MLLMs. A.3. Defense Analysis A.3.1. Overall Defense Effectiveness Fig. 9 presents the aggregated ASR across all 13 defense methods evaluated on 3 representative models (2 open- source + 1 closed-source) and 4 attack methods. Results clearly indicate that different defenses provide highly non- uniform protection. A.3.2. Defense-Induced ASR Reduction To quantify the effect of individual defenses, Fig. 10 shows a waterfall plot comparing ASR before and after applying each defense method. The visualization highlights the relative contribution of each defense to robustness improvement. A.4. Statistical Analysis and Interpretation Fig. 11 shows the logistic regression coefficients used to quantify the effect of style, attack method, and model type on the probability of attack success. Positive coefficients indicate features that increase vul- nerability, whereas negative coefficients suggest robustness- related factors. Table 6. Summary of defenses and their classification. CategorySubcategoryRepresentative methods (implemented) Off-modelInput pre-processingECSO JailGuard AdaShield-S Uniguard DPS CIDER GuardReasoner -VL Llama-Guard-4 QGuard LlavaGuard Off-modelOutput post-processi-ngShieldLM MLLM-Protector Llama-Guard-3 On-modelInference process interventionCOCA HiddenDetect On-modelIntrinsic model alignmentVLGuard Table 7. Summary of implemented attacks and their classification. CategorySubcategoryRepresentative methods (implemented) White-boxSingle-modalvisual-adv,visual-adv-un ImgJP, DeltaJP White-boxCross-modalUMK,BAP,JPS Black-boxStructured visual-carrierFigStep,FigStep-Pro,QR-Attack, HADES Black-boxOut-of-Distribution (OOD)CS-DJ,SI-Attack,JOOD,VisCRA Black-boxHidden risksHIMRD,MML Ovis2.5 GLM-4.1V Gemma-3 Kimi-VL LLaVA-v1.6 DeepSeek-VL2 Qwen3-VL-30B ERNIE-4.5 Doubao-Seed Gemini-2.5 Qwen3-VL-Plus GPT-4o Qwen3-VL-Flash Claude-3.5 Haiku GPT-5 Claude-4 Sonnet Avg. MML CS-DJ FigStep QR-Attack HIMRD HADES JOOD SI-Attack Avg. Attack Methods 21.022.754.710.324.00.341.01.027.650.752.256.620.342.115.37.528.0 41.830.921.639.55.125.638.120.648.432.023.423.028.17.69.714.325.6 51.551.328.339.923.844.530.929.623.715.813.99.721.12.44.24.224.7 45.450.027.835.433.128.910.423.230.224.29.311.30.01.95.12.921.2 30.443.144.319.931.324.57.510.333.330.617.412.92.22.95.10.519.8 45.238.723.724.932.822.68.715.928.324.59.211.313.11.76.11.819.3 44.738.226.726.330.915.94.313.529.821.49.77.514.61.04.91.418.2 20.422.517.610.16.92.68.34.222.516.711.44.411.20.96.13.510.6 37.537.230.625.823.520.618.714.830.527.018.317.113.87.67.14.520.9 Open Source ModelsClosed Source Models 0 20 40 60 Attack Success Rate (ASR %) Figure 4. Vulnerability Heatmap of 16 MLLMs against 8 Jailbreak Attacks. The heatmap visualizes the Attack Success Rate (ASR %), where darker red indicates higher vulnerability (higher ASR) and lighter yellow indicates higher robustness. Models are categorized into Open Source (left) and Closed Source (right), sorted by their average vulnerability within each group. The ’Avg.’ row and column denote the mean ASR for each model and attack method, respectively. Table 8. Performance comparison of bothopen-sourceandclosed-sourcelarge multimodal models on various benchmarks. Models are grouped by their availability. We report ASR, average-T (T), average-H (H), average-A (A), average-D (D). Closed-Source Models (Part I) GPT-5GPT-4oGemini-2.5Claude-4 Sonnet BenchmarkASRTHADASRTHADASRTHADASRTHAD FigStep4.20 1.12 1.76 2.35 3.709.67 1.24 2.30 2.39 1.5115.80 1.42 2.44 2.71 2.114.20 1.10 2.01 2.22 1.50 MML15.27 1.60 3.20 2.42 3.2356.60 2.51 5.38 3.85 2.9450.67 2.43 5.64 3.33 2.987.53 1.21 2.46 2.27 1.36 CS-DJ9.73 1.26 2.04 2.44 3.8923.00 1.56 3.12 2.75 2.7932.00 1.86 3.58 3.02 3.1514.27 1.36 2.87 2.50 2.44 HIMRD 5.13 1.15 2.00 2.20 3.8712.87 1.30 2.93 2.17 1.6430.60 1.87 3.87 2.87 2.950.53 1.01 1.96 2.04 1.15 JOOD4.93 1.13 1.94 2.37 2.727.53 1.22 2.29 2.51 1.4521.40 1.58 2.58 2.91 2.031.40 1.03 1.94 2.20 1.36 SI-Attack 6.13 1.17 1.95 2.42 3.384.40 1.14 2.21 2.16 1.5516.67 1.47 2.65 2.65 2.273.47 1.09 1.95 2.28 1.71 HADES6.13 1.17 1.94 2.40 3.1411.27 1.31 2.42 2.60 1.6024.53 1.68 2.74 3.00 2.241.80 1.04 1.92 2.19 1.44 QR-Attack 5.14 1.15 1.98 2.33 3.3111.27 1.30 2.46 2.41 1.7824.22 1.69 2.87 2.89 2.682.87 1.07 1.99 2.17 1.53 Closed-Source Models (Part I) Claude-3.5 HaikuQwen3-VL-FlashQwen3-VL-PlusDoubao-Seed BenchmarkASRTHADASRTHADASRTHADASRTHAD FigStep2.40 1.06 2.03 2.16 1.2621.13 1.58 2.56 2.91 2.5813.87 1.38 2.26 2.67 2.2923.67 1.66 2.68 2.97 2.72 MML42.07 2.15 5.23 3.62 2.5820.27 1.66 4.44 2.60 1.9452.20 2.45 5.83 3.71 2.6727.60 1.83 5.59 2.75 2.10 CS-DJ7.60 1.19 2.47 2.31 1.7628.13 1.80 3.86 2.74 3.2223.40 1.65 2.96 2.77 3.0648.40 2.32 4.20 3.48 3.53 HIMRD 2.87 1.07 2.29 2.06 1.242.20 1.07 2.41 1.97 1.2517.40 1.49 3.30 2.46 2.1633.27 1.95 3.87 2.92 2.87 JOOD1.00 1.03 1.96 2.13 1.1814.60 1.39 2.26 2.72 2.249.73 1.26 2.07 2.54 2.0129.80 1.84 2.94 3.21 2.56 SI-Attack0.93 1.03 2.03 1.98 1.2211.20 1.31 2.20 2.55 2.6411.40 1.31 2.15 2.56 2.5122.47 1.61 2.73 2.89 2.97 HADES1.73 1.04 1.97 2.13 1.2113.07 1.37 2.17 2.68 2.299.20 1.26 2.05 2.53 2.1028.27 1.78 2.81 3.13 2.69 QR-Attack1.87 1.05 2.03 2.11 1.290.00 1.00 2.00 1.22 1.009.27 1.25 2.08 2.47 2.4630.22 1.85 3.16 3.04 3.15 Open-Source Models (Part I) Qwen3-30BGemma-3DeepSeek-VL2ERNIE-4.5 BenchmarkASRTHADASRTHADASRTHADASRTHAD FigStep30.93 1.86 2.95 3.17 2.6428.33 1.75 2.82 3.06 2.8944.53 2.19 4.06 3.71 2.7429.60 1.79 2.96 3.15 2.69 MML 41.00 2.08 4.87 3.36 2.4054.73 2.54 6.01 3.62 3.080.33 1.02 2.45 1.19 1.151.00 1.04 2.36 1.24 1.33 CS-DJ38.07 2.02 3.82 3.20 3.2221.60 1.55 3.41 2.67 3.0425.60 1.65 4.24 2.86 2.5920.60 1.55 3.71 2.66 2.46 HIMRD7.53 1.21 2.72 2.12 1.4444.33 2.29 4.43 3.30 3.3724.53 1.62 3.85 2.52 2.1710.33 1.26 3.19 1.98 2.10 JOOD4.33 1.11 1.98 2.32 1.7326.67 1.70 2.67 3.04 2.2415.87 1.43 2.63 2.68 1.6513.47 1.37 2.38 2.64 1.93 SI-Attack8.27 1.22 2.25 2.38 2.1017.60 1.46 2.51 2.70 2.572.60 1.07 2.13 2.07 1.244.20 1.12 2.24 1.87 1.93 HADES8.73 1.24 2.14 2.51 2.0023.67 1.65 2.61 2.97 2.2722.60 1.64 3.02 2.99 1.9215.93 1.45 2.69 2.60 2.04 QR-Attack 10.41 1.29 2.30 2.45 2.2427.75 1.76 3.06 2.93 2.7428.95 1.73 3.67 2.93 2.4723.22 1.59 3.14 2.75 2.33 Open-Source Models (Part I) GLM-4.1VOvis2.5Kimi-VLLLaVA-v1.6 BenchmarkASRTHADASRTHADASRTHADASRTHAD FigStep51.27 2.38 3.90 3.64 3.4151.53 2.44 3.75 3.77 3.7039.93 2.02 3.64 3.34 2.5523.80 1.67 3.75 2.92 2.32 MML22.67 1.70 5.12 2.66 2.4621.00 1.58 4.76 2.31 2.5910.27 1.29 3.16 1.79 1.6524.00 1.71 4.67 2.83 1.95 CS-DJ30.93 1.81 4.37 2.72 3.2041.80 2.20 4.14 3.22 3.6739.47 1.99 4.43 3.26 2.905.07 1.13 2.52 1.79 1.81 HIMRD43.07 2.15 4.55 3.03 3.0130.40 1.85 3.90 2.59 2.7419.87 1.53 4.06 2.32 1.8031.33 1.75 4.04 2.81 2.41 JOOD38.20 2.04 3.41 3.49 2.7144.67 2.25 3.36 3.61 3.3626.27 1.71 3.04 3.07 2.0330.93 1.85 3.23 3.31 2.37 SI-Attack22.53 1.60 2.87 2.90 2.7320.40 1.56 2.79 2.75 3.1510.07 1.29 2.62 2.29 1.846.93 1.21 2.36 2.31 1.82 HADES 38.73 2.06 3.32 3.52 2.7945.20 2.25 3.40 3.66 3.3724.93 1.74 3.26 3.14 2.0632.80 1.90 3.37 3.37 2.49 QR-Attack49.97 2.41 4.22 3.56 3.6545.36 2.30 3.86 3.47 3.7635.36 1.90 3.87 3.15 2.6033.09 1.80 3.79 3.09 2.81 Qwen3-VL 30B Gemma-3 DeepSeek VL2 ERNIE 4.5 GLM-4.1V Ovis2.5 Kimi-VL LLaVA v1.6 GPT-5 GPT-4o Gemini 2.5 Claude-4 Sonnet Claude-3.5 Haiku Qwen3-VL Flash Qwen3-VL Plus Doubao Seed 20 40 60 FigStep Qwen3-VL 30B Gemma-3 DeepSeek VL2 ERNIE 4.5 GLM-4.1V Ovis2.5 Kimi-VL LLaVA v1.6 GPT-5 GPT-4o Gemini 2.5 Claude-4 Sonnet Claude-3.5 Haiku Qwen3-VL Flash Qwen3-VL Plus Doubao Seed 20 40 60 MML Qwen3-VL 30B Gemma-3 DeepSeek VL2 ERNIE 4.5 GLM-4.1V Ovis2.5 Kimi-VL LLaVA v1.6 GPT-5 GPT-4o Gemini 2.5 Claude-4 Sonnet Claude-3.5 Haiku Qwen3-VL Flash Qwen3-VL Plus Doubao Seed 20 40 60 CS-DJ Qwen3-VL 30B Gemma-3 DeepSeek VL2 ERNIE 4.5 GLM-4.1V Ovis2.5 Kimi-VL LLaVA v1.6 GPT-5 GPT-4o Gemini 2.5 Claude-4 Sonnet Claude-3.5 Haiku Qwen3-VL Flash Qwen3-VL Plus Doubao Seed 20 40 60 HIMRD Qwen3-VL 30B Gemma-3 DeepSeek VL2 ERNIE 4.5 GLM-4.1V Ovis2.5 Kimi-VL LLaVA v1.6 GPT-5 GPT-4o Gemini 2.5 Claude-4 Sonnet Claude-3.5 Haiku Qwen3-VL Flash Qwen3-VL Plus Doubao Seed 20 40 60 JOOD Qwen3-VL 30B Gemma-3 DeepSeek VL2 ERNIE 4.5 GLM-4.1V Ovis2.5 Kimi-VL LLaVA v1.6 GPT-5 GPT-4o Gemini 2.5 Claude-4 Sonnet Claude-3.5 Haiku Qwen3-VL Flash Qwen3-VL Plus Doubao Seed 20 40 60 SI-Attack Qwen3-VL 30B Gemma-3 DeepSeek VL2 ERNIE 4.5 GLM-4.1V Ovis2.5 Kimi-VL LLaVA v1.6 GPT-5 GPT-4o Gemini 2.5 Claude-4 Sonnet Claude-3.5 Haiku Qwen3-VL Flash Qwen3-VL Plus Doubao Seed 20 40 60 HADES Qwen3-VL 30B Gemma-3 DeepSeek VL2 ERNIE 4.5 GLM-4.1V Ovis2.5 Kimi-VL LLaVA v1.6 GPT-5 GPT-4o Gemini 2.5 Claude-4 Sonnet Claude-3.5 Haiku Qwen3-VL Flash Qwen3-VL Plus Doubao Seed 20 40 60 QR-Attack | Open Source ModelsClosed Source Models Figure 5. Radar plot illustrating model-wise ASR distribution under each attack method. The plot highlights structural differences in attack effectiveness across models. 1.0 1.2 1.4 1.6 1.8 2.0 2.2 2.4 2.6 Score (a) Judge Score Distribution 2 3 4 5 6 Score (b) Harmfulness Score Distribution MML FigStep CS-DJ QR-Attack HADES HIMRD JOOD SI-Attack 1.5 2.0 2.5 3.0 3.5 Score (c) Alignment Score Distribution MML FigStep CS-DJ QR-Attack HADES HIMRD JOOD SI-Attack 1.0 1.5 2.0 2.5 3.0 3.5 4.0 Score (d) Detail Score Distribution Figure 6. Distribution of harmfulness, alignment, detail, and judge scores across the 8 attack methods. Each box summarizes model- averaged scores, revealing distinct behavioral patterns induced by different attacks. A. Ethical and Social Risks B. Privacy and Data Risks C. Safety and Physical Harm D. Criminal and Economic Risks E. Cybersecu rity Threats F. Information and Political Manipulation G. Content and Cultural Safety H. Intellectual Property and Ownership I. Decision and Cognitive Risks 0 5 10 15 20 25 30 35 40 Attack Success Rate (ASR %) CS-DJFigStepHADESHIMRDJOODMMLQR-AttackSI-Attack Figure 7. Category-level attack success rate across 9 semantic categories and 8 attack methods. FigStepQR-AttackMMLHIMRDHADESCS-DJJOODSI-Attack Attack Method 5 10 15 20 25 Attack Success Rate (ASR %) Prompt Style Declarative Consultative Instructive Figure 8. ASR comparison across prompt style (declarative, consultative, instructive). Consultative prompts consistently trigger higher ASR across multiple attacks. None AdaShield Uniguard DPS JailGuard ECSO GuardReasoner-VL LlavaGuard QGuard Llama-Guard-4 CIDER ShieldLM MLLM-protector Llama-Guard-3 0 10 20 30 40 50 Average Attack Success Rate (ASR %) Output Post-processing DefensesInput Pre-processing Defenses CS-DJFigStepMMLHIMRD Figure 9. Overall effectiveness of 13 defense methods across 4 attack types. Lower ASR indicates stronger defensive capability. QGuard MLLM-protector AdaShield Llama-Guard-3 ECSO DPS Uniguard CIDER JailGuard GuardReasoner-VL Llama-Guard-4 ShieldLM LlavaGuard 0 10 20 30 Average ASR (%) Baseline: 25.5% -19.9 -19.7 -13.1 -11.9 -11.9 -10.4 -9.7 -6.0 -5.9 -4.0 -3.6 Model: GPT-4o JailGuard MLLM-protector QGuard AdaShield GuardReasoner-VL Llama-Guard-3 Uniguard DPS CIDER Llama-Guard-4 ShieldLM LlavaGuard ECSO 0 10 20 30 40 Average ASR (%) Baseline: 36.3% -33.7 -26.1 -24.8 -23.2 -16.4 -15.8 -15.7 -12.7 -10.1 -8.0 -7.1 -3.2 Model: Ovis2.5-9B MLLM-protector QGuard ShieldLM AdaShield Llama-Guard-3 DPS GuardReasoner-VL Llama-Guard-4 Uniguard CIDER ECSO LlavaGuard JailGuard 0 10 20 30 40 Average ASR (%) Baseline: 37.2% -30.4 -29.4 -22.0 -17.9 -17.7 -14.9 -14.7 -11.6 -9.9 -8.8 -6.7 -6.0 -4.4 Model: Gemma-3 Safety Gain (ASR Reduction)Residual Risk (Remaining ASR) Figure 10. Impact of each defense method on reducing ASR for a representative model. The waterfall structure illustrates relative gains offered by different defenses. -0.8-0.6-0.4-0.20.00.20.40.6 Logistic Regression Coefficient (Log-Odds) Safer (Factors reducing ASR) | Riskier (Factors increasing ASR) Attack: SI-Attack Model: Closed-Source Attack: HIMRD Style: Instructive Attack: CS-DJ Attack: JOOD Attack: HADES Attack: QR-Attack Attack: MML Style: Consultative *** *** *** ** *** *** *** *** *** Baseline (Coef=0): Style: Declarative Model: Open-Source Attack: FigStep Figure 11. Logistic regression coefficients quantifying the contribution of style, attack method, and model type to attack success probability.