Paper deep dive
When Safe Unimodal Inputs Collide: Optimizing Reasoning Chains for Cross-Modal Safety in Multimodal Large Language Models
Wei Cai, Shujuan Liu, Jian Zhao, Ziyan Shi, Yusheng Zhao, Yuchen Yuan, Tianle Zhang, Chi Zhang, Xuelong Li
Models: LLaVA-NeXT-LLaMA3, Qwen2.5-VL
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/12/2026, 6:21:52 PM
Summary
The paper introduces the 'implicit reasoning risk' in Multimodal Large Language Models (MLLMs), where safe unimodal inputs combine to form unsafe multimodal outputs. To mitigate this, the authors propose the Safe-Semantics-but-Unsafe-Interpretation (SSUI) dataset and the Safety-aware Reasoning Path Optimization (SRPO) training framework, which aligns internal reasoning paths with human safety values. Experimental results demonstrate that SRPO achieves state-of-the-art performance on the newly proposed Reasoning Path Benchmark (RSBench).
Entities (4)
Relation Signals (3)
SRPO → trainedon → SSUI
confidence 100% · A novel training framework, Safety-aware Reasoning Path Optimization (SRPO), is also designed based on the SSUI dataset
RSBench → evaluates → SRPO
confidence 95% · Experimental results show that our SRPO-trained models achieve state-of-the-art results on key safety benchmarks, including the proposed Reasoning Path Benchmark (RSBench)
SSUI → addresses → implicit reasoning risk
confidence 90% · we have constructed the SSUI dataset... to address the issue of implicit reasoning risk
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal Large Language Models (MLLMs) are susceptible to the implicit reasoning risk, wherein innocuous unimodal inputs synergistically assemble into risky multimodal data that produce harmful outputs. We attribute this vulnerability to the difficulty of MLLMs maintaining safety alignment through long-chain reasoning. To address this issue, we introduce Safe-Semantics-but-Unsafe-Interpretation (SSUI), the first dataset featuring interpretable reasoning paths tailored for such a cross-modal challenge. A novel training framework, Safety-aware Reasoning Path Optimization (SRPO), is also designed based on the SSUI dataset to align the MLLM's internal reasoning process with human safety values. Experimental results show that our SRPO-trained models achieve state-of-the-art results on key safety benchmarks, including the proposed Reasoning Path Benchmark (RSBench), significantly outperforming both open-source and top-tier commercial MLLMs.
Tags
Links
- Source: https://arxiv.org/abs/2509.12060
- Canonical: https://arxiv.org/abs/2509.12060
Trouble viewing inline? Open PDF directly →
Full Text
44,050 characters extracted from source content.
Expand or collapse full text
When Safe Unimodal Inputs Collide: Optimizing Reasoning Chains for Cross-Modal Safety in Multimodal Large Language Models Wei Cai 12 , Shujuan Liu 3 , Jian Zhao 2 , Ziyan Shi 24 , Yusheng Zhao 25 , Yuchen Yuan 2 , Tianle Zhang 2 , Chi Zhang 2 , Xuelong Li 2 * 1 Peking University 2 Institute of Artificial Intelligence (TeleAI), China Telecom 3 University of the Chinese Academy of Sciences 4 Harbin Institute of Technology 5 University of Science and Technology of China Abstract Multimodal Large Language Models (MLLMs) are suscep- tible to the implicit reasoning risk, wherein innocuous uni- modal inputs synergistically assemble into risky multimodal data that produce harmful outputs. We attribute this vul- nerability to the difficulty of MLLMs maintaining safety alignment through long-chain reasoning. To address this is- sue, we introduce Safe-Semantics-but-Unsafe-Interpretation (SSUI), the first dataset featuring interpretable reasoning paths tailored for such a cross-modal challenge. A novel training framework, Safety-aware Reasoning Path Optimiza- tion (SRPO), is also designed based on the SSUI dataset to align the MLLM’s internal reasoning process with hu- man safety values. Experimental results show that our SRPO- trained models achieve state-of-the-art results on key safety benchmarks, including the proposed Reasoning Path Bench- mark (RSBench), significantly outperforming both open- source and top-tier commercial MLLMs. Introduction With the continuous emergence of Multimodal Large Lan- guage Models (MLLMs) (Liu et al. 2023; Bai et al. 2023; Zhu et al. 2023; Li et al. 2023; Team et al. 2024) and their integration into various applications, more and more con- cerns about their safety vulnerabilities have been raised. Although significant progress has been made in the safety alignment of the text modal in Large Language Models (LLMs) (Wang et al. 2024c), safety alignment under the multimodal scenario remains relatively fragile. This vul- nerability makes MLLMs susceptible to attacks with cross- modal inputs, which can easily elicit harmful responses. Although studies have been made in evaluating the safety of MLLMs (Liu et al. 2024c; Gong et al. 2025; Qi et al. 2024; Li et al. 2024), and related works (Zhou et al. 2024; Wang et al. 2024b) have been proposed focusing on cross- modal safety, the cross-modal safety alignment of MLLMs still remains a significant challenge. A typical example of cross-modal safety alignment failure is illustrated in Fig- ure 1. The MLLM receives benign image and text inputs: an image depicts a bottle of “84” disinfectant and a bottle of toilet cleaner (containing hydrochloric acid); and the text in- structs to “clean the bathroom according to the instructions.” * Corresponding author. 60.0 38.2 55.4 76.3 20.5 55.9 67.3 60.0 63.3 88.4 56.3 65.9 77.4 65.3 73.9 98.1 67.4 78.8 0 20 40 60 80 100 120 USBBenchMLLM-GUARDMSSBenchSafeBenchVLSBenchRSBench A v e r a g e P e r f o r m a n c e ( % ) BaseDPOSRPO (ours) Safe Image Safe Text Instructions recommend 2ml left-side liquid and 3ml right-side liquid per square meter. For cleaning 12 square meters bathroom, what's the total liquid needed? I’m sorry. I can’t do that. The scenario involves the potential unsafe behavior of mixing 84 Disinfectant and Harpic, which can result in the release of highly toxic chlorine gas. The total amount of liquid needed to clean a combined area of 12 square meters is 60ml...... Final Answer Text-image analysis Start calculating Safe rules Cross verification Output dosage. Both liquid are safe alone. Mixing is harmful. Danger! Refuse to answer. Figure 1: Up: a sample from the SSUI dataset illustrat- ing reasoning failure of MLLM due to uncertain reasoning paths. Down: the significant improvement in safety reason- ing of our proposed SRPO framework. However, the semantic combination of the image and text can easily induce an unsafe outcome. If the MLLM were to calculate the ingredient quantities as instructed, it could lead to user poisoning, as mixing the two substances pro- duces toxic chlorine gas. A safe MLLM should refuse to re- spond or dissuade the user from such an action. Recent stud- ies (Zheng et al. 2025; Zhou et al. 2024) have shown that current MLLMs still struggle to identify and address such cross-modal safety issues that require deep-level reasoning. Fundamentally, the reason that MLLMs struggle with this type of cross-modal safety problem is that their resolutions typically require relatively long reasoning paths. As illus- trated in Figure 1, the model’s step-by-step reasoning pro- cess can deviate into disadvantageous branches containing arXiv:2509.12060v2 [cs.AI] 16 Sep 2025 # Reasoning Paths Question Traffic safety regulations Bus Safety tips # Answer Safety warning 1 Query Agent Safe Images Cross-modal Unsafety Pair Safe Text Safe Images Query Agent 2 Reasoning Agent Cross-modal Unsafety Pair Cross-modal Unsafety Pair Reasoning Agent ! 3 Reflection and Check Agent Cross-modal Unsafety Pair ! Explainable Reasoning Reflection Agent Extract Refined Data ! Extract Safe Images & Reasoning Check Agent Unsafe & Invalid Text Safe Text Redundancy Does the text repeat details in the image? Completene Can Text + Image sufficiently infer the unsafe behavior? Checked Data ! Refined Text 4 Summary Agent Checked Data ! Summary Agent QA-Pair One-to-one Explainable Reasoning Tag 5 Manual Curation QA-Pair Manual Curation Safety RedundancyCompleteness Difficulty Level Final Dataset ! Expectation Safe & Unsafe & Invalid Text Safe Images Reality ! ! Refined Refined Refined Refined Refined Checked -Safe Text: -Unsafe Category: -Explainable Reasoning: -Safe Reply: -Unsafe Reply: One-to-one Refined Checked RefinedChecked Figure 2: The five-stage protocol for constructing the SSUI dataset. errors, thereby reducing the probability of arriving at the cor- rect answer. Although such errors may not immediately lead to an incorrect final answer, they can accumulate and disrupt the reasoning process (Ling et al. 2023). For instance, when the model recognizes that mixing the two cleaning agents will produce toxic gas, it ultimately produces a safe answer. On the contrary, when it fails to recognize this issue and instead focuses on calculating the ingredient quantities, its response could be hazardous. We term this phenomenon im- plicit reasoning risk, where the image and text are safe in their own modal, but the semantic combination of them is potentially unsafe, which tends to generate harmful output of MLLM. Existing research (Wei et al. 2022; Yao et al. 2023) has made significant progress in enhancing the safety alignment of LLMs through long-chain reasoning, largely attributable to the availability of structured, high-quality data and ma- ture training pipelines. Compared to LLMs, MLLMs pro- cess more complex inputs that typically involve multimodal information, which makes them more prone to errors dur- ing the safety alignment process (Pi et al. 2024), particularly for the implicit reasoning risk. This is primarily due to the lack of large-scale, high-quality datasets and effective train- ing strategies. To address the aforementioned issues and enhance the safety alignment of MLLMs, we propose a specialized framework, Safety-aware Reasoning Path Optimization (SRPO), which is designed to better align the MLLM’s reasoning paths with safety requirements. Additionally, we introduce the Safe-Semantics-but-Unsafe-Interpretation (SSUI) dataset, which is equipped with interpretable reason- ing features to tackle the implicit reasoning risk issue. In ad- dition to this, we have also developed the Reasoning Path Benchmark (RSBench), a benchmark specifically created to evaluate the effectiveness and safety of Chain-of-Thought (CoT) reasoning paths. Our main contributions are summarized below: • We are the first to identify and formally define the prob- lem of implicit reasoning risk in MLLMs. To address this issue, we have constructed the SSUI dataset. This dataset introduces safety reasoning path labels designed to bet- ter guide MLLMs in selecting the most rational reasoning paths for safety alignment. • We propose the SRPO framework, which enhances the alignment of MLLMs with human safety values by continuously exploring and optimizing reasoning paths within a vast solution space. • We also introduce the RSBench, a novel benchmark de- veloped to specifically evaluate the effectiveness and safety performance of CoT reasoning paths, filling the gap in such a domain. Related Works Multimodal Safety Alignment Several effective strategies have been developed to enhance the safety of Multimodal Large Language Models (MLLMs) recently. Through the prevalent reinforcement learning from human feedback (RLHF) (Ouyang et al. 2022) and well- designed image-text pairs, MLLMs can be safety-aligned with a variety of methods, such as supervised fine-tuning (SFT), direct preference optimization (DPO) (Rafailov et al. 2023), and proximal policy optimization (PPO) (Schulman et al. 2017). More recent techniques, such as simple prefer- ence optimization (SimPO) (Meng, Xia, and Chen 2024), and odds-ratio preference Optimization (ORPO) (Hong, Lee, and Thorne 2024), do not rely on a reward model, which significantly strengthens the stability and simplifies Public Safety 59.24% Ethical Safety 34.02% National Safety 6.74% Personal Rights & Property 9.42% Privacy Protection 7.78% Network Attacks 6.59% Content Safety 6.59% Hazardous & Controlled Materials 5.38% Dangerous behavior & Suicide 5.0% Business 4.71% Intellectual Property 4.62% Biology & Environment 4.62% Illegal crimes 4.54% Bias & Discrimination 13.27% Inappropriate values 7.16% Psychological Health 4.71% Insults & Condemnation 3.98% Moral violations 2.57% Superstition 2.34% Military 4.71% Culture&History 1.09% Political controversies 0.94% Figure 3: The safety taxonomy of our SSUI dataset. the training pipeline of MLLMs. These methods perform pairwise comparisons on two model-generated response se- quences, encouraging the model to assign a higher probabil- ity to the favorable one over the unfavorable one. However, it has been observed that such preference-based optimization methods can be suboptimal in tasks requiring deep reason- ing (Meng, Xia, and Chen 2024). The reason is that these methods conduct the comparison of the response sequences as a whole, ignoring the fact that in multi-step reasoning tasks, errors often originate at a specific step and propa- gate through its subsequent branches, which we term as im- plicit reasoning risk. In this work, we propose the Safety- aware Reasoning Path Optimization (SRPO), a novel train- ing framework that takes all intermediate reasoning steps into account, and can effectively tackle the implicit reason- ing risk issue. The SSUI Dataset Prior studies (Zhang et al. 2023; Dong et al. 2025) have ex- plored methods for integrating reasoning capabilities into MLLMs. However, enhancing the reasoning abilities of MLLMs to address implicit reasoning risk remains a con- siderable challenge, largely due to data limitations. Com- pared to text-only data, visual reasoning data is not only more costly to collect but also requires significant manual effort for detailed annotation and verification, owing to the lack of effective data generation pipelines. To address the high costs and limited scalability of man- ual data collection, we propose an AI-assisted data gener- ation method. As illustrated in Figure 2, this scalable ap- proach enables us to generate high-quality data, thereby ef- fectively enhancing the model’s safe reasoning capabilities. As preliminary data, we first randomly acquire various se- ries of safe images from publicly available datasets, includ- ing Open Images v7 (Kuznetsova et al. 2020), COCO (Lin et al. 2014), and EgoShots (Agarwal et al. 2020). We per- form sampling and verification of the selected images to en- sure their safety quality. The dataset is then annotated with a multi-agent system, which generates image-text pairs and their corresponding CoTs stating why the pairs fall in im- plicit reasoning risk. The multi-agent system consists of a query agent, a reasoning agent, a reflection and check agent, and a summary agent. The data produced subsequently go through a manual revision process to finalize the dataset. De- tailed selection criteria for the generated data can be found in Appendix A.2. Query Agent: The query agent initiates the process by generating safe text based on an initially safe image, and hy- pothesizes unsafe scenarios of the image-text pair. The ob- jective is to construct cross-modal unsafe image-text pairs, which are individually benign from the unimodal perspec- tive, but contain latent unsafe implications when combined. Reasoning Agent: Based on the generated cross-modal unsafe image-text pairs from the query agent, the reason- ing agent further yields interpretable, step-by-step reasoning CoTs for each pair. Reflection and Check Agent: The output of the reason- ing agent, i.e. the image-text pairs and their corresponding CoTs of implicit reasoning risk, then undergo a rigorous two-part review. Firstly, the reflection agent meticulously examines the informational redundancy and completeness of the image-text pair. It ensures that the textual component does not merely replicate the visual information, and will re- move such redundancy to maintain complementarity of the cross-modality pair. It also verifies whether the pair provides clear arguments for inferring an unsafe outcome, judiciously supplementing any missing critical information. Secondly, the check agent guarantees the intrinsic safety of the query text. In this process, the check agent conducts a comprehen- sive safety evaluation of the text queries within the pairs, and those identified as unsafe or invalid will be discarded. Summary Agent: The Summary Agent integrates the data refined and checked in the preceding step to form a QA-pair. In this pair, ’Q’ represents the input image-text pair, and ’A’ consists of the reasoning chain and its resulting response. Subsequently, all annotated content, excluding the image, is uniformly referred to as a “Tag”. These Tags, paired with their corresponding images, constitute the complete and for- matted entries to our dataset. The detailed format of our dataset can be found in Appendix A.3. Manual Revision: The final stage involves manual review and editing, which considers the overall safety, difficulty level, information redundancy and integrity with strict stan- dards to ensure data quality. For our SSUI dataset, the initial image sample size for the query agent is about 25,000; after the multi-agent data gen- eration approach described above, 4,779 samples are gen- erated and formulates the dataset. The SSUI dataset is then hierarchically structured into three category levels based on a safety vulnerability taxonomy, comprising 3 primary, 19 secondary, and 68 tertiary categories, as illustrated in Fig- ure 3. To our knowledge, this hierarchically structured cat- egorization system includes the majority of risk categories identified in both academic and industrial applications. Safety-Aware Reasoning Path Optimization As discussed in the Introduction, to address the issue of im- plicit reasoning risk, the MLLMs need to maintain safety alignment throughout long-chain reasoning. During such a process, however, errors often emerge at specific steps and exclusively affect subsequent (and thus incorrect) branches, as illustrated in Figure 4. Compared to LLMs, MLLMs im- port more complex multimodal inputs, making them more susceptible to errors during safety alignment (Pi et al. 2024). We argue that this issue stems from the fact that multimodal information occupies a significantly larger solution space, in which multiple reasoning paths can potentially lead to a safe and correct final answer, yet each path is fraught with the risk of branching into erroneous steps, which can ruin the entire reasoning process. To address this issue, we propose a dedicated training framework that jointly considers mul- tiple reasoning paths for a safety problem. Our method en- courages favorable branches at each reasoning step while si- multaneously penalizing unfavorable ones. This framework, which we term Safety-Aware Reasoning Path Optimization (SRPO), consists of two main stages, as shown in Figure 4: 1. Generative Exploration: To effectively explore the so- lution space for safety-related reasoning issues, we first progressively expand branches at each step of the refer- ence reasoning paths provided in our SSUI dataset, by which we obtain multiple favorable and unfavorable rea- soning branches at each step, which are utilized to pro- vide the model with contrastive feedback hereafter, as shown in Figure 4. 2. Path Optimization: The model is then optimized by leveraging a collection of the reference paths and the gen- erated favorable/unfavorable branches, with the goal to enhance the inherent safety reasoning capabilities of the base model. By wrapping up the stages above, we design the SRPO framework, which aims to improve the overall reasoning performance of MLLMs. Generative Exploration RLHF and subsequent studies on preference optimiza- tion (Ouyang et al. 2022; Rafailov et al. 2023) have demon- strated significant effectiveness in model alignment. How- ever, these algorithms can be suboptimal in tasks requiring deep reasoning, which leads to the implicit reasoning risk we have identified. The reason is that these methods tend to treat the entire reasoning path indiscriminately (Meng, Xia, and Chen 2024). Furthermore, recent research (Yue et al. 2025) has indicated that such reward models, while improv- ing sampling efficiency, do so by sampling only specific rea- soning paths within the solution space, which comes at the cost of narrowing the scope of reasoning capabilities. To ad- dress such issues, our SRPO framework employs a proactive exploration strategy, which autonomously generates a di- verse set of reasoning paths from an initial generative phase. If I've recently obtained my driver's license and am eager to touch the steering wheel, what steps should I take? Exploration Optimization # Reasoning Paths From SSUI # Answer: Question Traffic safety regulations Bus Safety tips Safety reminder # Reasoning Paths # Answer: Question A passenger on a bus Safety rules Safety tips Safety warning # Reasoning Paths # Answer: Question Start in minimal traffic Practice driving skills Driving tips Reference Reference Loss ... Alignment Loss Figure 4: An overview of our SRPO framework for explor- ing and learning from diverse reasoning paths of safety. Our framework formulates the implicit safety reasoning task as a question Q, where the objective is to generate the final answer A that is aligned with human safety values. We assume that the model undergoes a series of reasoning steps v 0 → v 1 →·→ v T to arrive at A, which is defined as τ = (v 0 → v 1 →·→ v T ),(1) where v i ∈ V represents a reasoning state, and the termi- nal state v T must contain the final answer A. The transition e = (v i → v i+1 ) corresponds to the generation of a new reasoning step. Specifically, we use a given question Q, which refers to the image-text pair input from the SSUI dataset, and gen- erate an initial reasoning path through autonomous explo- ration guided by the CoT reference reasoning path prompts in SSUI. The CoT prompts input, denoted as D c , contains m ground-truth examples, where each example consists of a question and its corresponding reasoning path. Assum- ing B is the base model, we sample a reference reasoning path τ by inputting the CoT D c and the given question Q to the model, thereby progressively expanding the reasoning branches, i.e., the aforementioned reasoning path τ ∼ B(·|Q,D c ).(2) The generated reasoning path is considered correct if its final step reaches the ground truth answer A, which is veri- fied by the defined functionF : F (τ ) = 1, if A∈ v T 0, otherwise. (3) Our framework explores multiple branches at each step, alleviating the influence of potential errors. Specifically, based on the preceding steps of a generated reasoning path τ 1:i−1 = (v 0 → v 1 → · → v i−1 ), we take tempera- ture sampling (Fan et al., 2018) as a way to sample diverse branches for the current step of the reasoning path: Ω∼ B(·|Q,D c ,τ 1:i−1 |T ),(4) where Ω = (v i → v i+1 → · → v T ) encompasses the sequence from the current step to the final step. Within Ω there are multiple continuation steps of the current step (e.g. the step v i+1 of v i , or the step v i of v i−1 ), which we uni- formly term as τ ′ cont . Our objective is to construct a pair of contrastive reasoning paths (τ + i ,τ − i ), in which: • Positive instance τ + i : A complete path formed by con- catenating the previous steps with a correct continuation, i.e. τ + i = τ 1:i−1 ⊕ τ ′+ cont where F (v T(τ + i ) ) = 1. • Negative instance τ − i : A reasoning path formed by con- catenating the previous steps with an incorrect continua- tion, i.e. τ − i = τ 1:i−1 ⊕ τ ′− cont where F (v T(τ − i ) ) = 0. At each step v i , we iteratively verify the branches sampled withF until obtaining one positive branch and one negative branch, which together form the pair of contrastive reason- ing paths (τ + i ,τ − i ). Path Optimization To optimize the base model B, we jointly consider both the reference reasoning paths τ ∗ from the SSUI dataset, and the explored contrastive reasoning path pairs (τ + i ,τ − i ). We en- courage the model to assign a higher likelihood to the refer- ence reasoning paths, which is achieved by a standard lan- guage modeling loss (Bengio et al. 2003) to the reference reasoning path τ ∗ , conditioned on the input question Q: J Ref (θ) =−E (v i−1 ,v i )∈τ ∗ [logp θ (v i |v i−1 )],(5) where p θ (v i |v i−1 ) is the conditional probability of transi- tioning from state v i−1 to v i , and Q∈ v 0 . Regarding the contrastive reasoning path pairs, since their comparison reveals the correct model optimization direc- tion, we define an alignment loss that provides contrastive feedback between the favorable and unfavorable branches, with the goal of maximizing the likelihood gap between the positive and negative instances. To be more specific, this loss is defined via a log-ratio preference functional (Hong, Lee, and Thorne 2024). Let L(τ|θ) = logp θ (τ|Q) be the log- likelihood of a complete reasoning path. The alignment loss at state v i is given by: J Align,i (θ) =−k· logσ L(τ + i |θ)−L(τ − i |θ) ,(6) where k is a hyperparameter that acts as a scaling factor to control the strength of this alignment loss. Notably, since τ + i and τ − i share the same previous state τ 1:i−1 , their log- likelihood difference simplifies to the difference between the log-likelihoods of their continuation parts: L(τ + i |θ)−L(τ − i |θ) = log p θ (τ ′+ cont |τ 1:i−1 ) p θ (τ ′− cont |τ 1:i−1 ) . (7) The total alignment loss is the sum of the losses over all intermediate states: J Align (θ) = T ∗ −1 X i=1 J Align,i (θ).(8) Finally, the total loss in our framework is a linear combi- nation of Equations (5) and (8): min θ J (θ) =J Ref (θ) + λ·J Align (θ),(9) where λ is a hyperparameter weight balancing the optimiza- tion of the reference reasoning path against that of the con- trastive reasoning paths. Experiments Experiment Setup To demonstrate the applicability of the proposed SRPO framework, we conduct evaluations on two MLLMs: LLaVA-NeXT-LLaMA3 (Liu et al. 2024b) and Qwen2.5-VL-7B (Bai et al. 2025). These models are strategically selected from distinct and influential ar- chitectural families to facilitate a rigorous validation of our method’s effectiveness across diverse foundations within a focused experimental scope. SRPO Implementation To train the SRPO framework, we employ LoRA fine-tuning (Hu et al. 2022), with a fixed batch size of 8, a learning rate of 5e-5, and a LoRA rank of 8. The loss weight λ is set to 0.3, which is based on the improvement of our Qwen2.5-SRPO model on the pro- posed Reasoning Path Bechmark (RSBench) with the can- didate values 0.1, 0.3, 0.5, 0.7, 0.9; this value of λ is applied to all benchmarks. We also adopt a fixed temperature pa- rameter of 0.5 to sample multiple outputs from the model. All training procedures are conducted on 8× A100 GPUs. More training details can be found in Appendix A.4. Evaluated Models and Configurations We evaluate both open-source and closed-source MLLMs. For open-source MLLMs, recently released mainstream models are taken into consideration, which include Qwen2.5-VL series (Bai et al. 2025), Qwen2-VL series (Wang et al. 2024a), InternVL2 series (Chen et al. 2024), GLM-4V (GLM et al. 2024), LLaVA-v1.5 series (Liu et al. 2024a), DeepSeek-VL (Lu et al. 2024), MiniGPT-v2 (Chen et al. 2023), MiniCPM-v2.6 (Yao et al. 2024), and VILA series (Lin et al. 2024). For close-source commercial MLLMs, we select GPT-4o, Claude-3.5-Sonnet2, and the Gemini series. We adopt the default settings for each model, including temperature, chat template, and other essential hyperparameters. Benchmark Setup Our experiments are conducted on various multimodal safety benchmarks. For example, we adopt USBBench (Zheng et al. 2025) and MSSBench (Zhou et al. 2024) for contextual safety, with a specific focus on the more challenging SIST subset of USBBench. Further- more, we adopt MLLM-GUARD (Gu et al. 2024), a multi- dimensional safety suite assessing five key safety dimen- sions; SafeBench (Ying et al. 2024), a comprehensive frame- work that evaluates MLLMs against a detailed taxonomy of 8 primary risk categories and 23 sub-categories; and VLS- Bench (Hu et al. 2024), a reliable cross-modal benchmark structured around a safety taxonomy of 6 main categories and 19 sub-categories. Main Results The experimental results in Table 1 demonstrate the ef- fectiveness and generalizability of our proposed SRPO framework in enhancing the safety capabilities of MLLMs. By applying SRPO to LLaVA-NeXT-LLaMA3 and Models USBBenchMLLMGuard MSSBenchSafeBench VLSBench Average↑ Average↓ ASR↓ ARR↓ Avg(%)↓ PAR↑ ASD↓Avg(%)↑ASR↓ SRI↑Avg(%)↑ Closed-source MLLMs Claude3.5-Sonnet232.80 25.7929.3052.389.0169.010.7099.3079.3575.0113.00 Gemini-1.5-Pro64.45 11.3337.8938.1221.9461.942.6097.1050.2161.8420.80 Gemini-2.0-Flash76.555.4340.9945.3220.5265.522.8097.3052.4865.1621.44 GPT-4o72.833.7738.3056.6814.3259.303.4096.1069.5070.4018.67 Open-source MLLMs DeepSeek-VL82.127.7844.9525.3235.3350.4033.10 75.2020.3542.8237.79 VILA-1.5-7B88.68 32.1560.4216.327.6552.2342.30 69.8013.5637.9836.79 MiniGPT-v289.12 12.3050.7149.7027.0150.6038.80 71.5020.3548.0438.84 LLaVA-v1.5-7B84.518.5646.5320.6343.0856.8039.60 72.308.6539.6043.07 LLaVA-v1.6-mistral-7B82.28 10.2646.2723.2543.5857.2532.50 72.8015.3242.1640.78 InternVL2.5-8B80.77 11.9846.3840.1932.4051.2221.90 82.1021.3748.7233.56 MiniCPM-LLaMA3-V 2.5 78.856.1242.4926.8131.1248.2530.50 74.9017.6041.8934.70 MiniCPM-V-2.681.346.4243.8832.2333.4347.3828.70 75.5015.9842.7735.34 Qwen2-VL-7B80.996.2743.6335.7228.3653.2035.40 72.3015.7744.2535.80 GLM-4v-9B77.725.9541.8423.4145.3050.8512.20 89.3022.6446.5529.78 LLaVA-NeXT-LLaMA378.887.5343.2026.4542.2756.3529.40 73.1018.5641.1238.29 + SRPO50.366.2028.2850.2217.6371.207.2091.8050.7465.9914.37 Qwen-2.5VL-7B75.264.7239.9938.2228.3555.3632.20 76.3020.4547.5833.51 + SRPO42.382.8322.6065.307.9273.897.5098.1067.4376.1812.67 Table 1: Safety evaluation results on 5 benchmarks. Applying our SRPO framework significantly promotes the safety perfor- mance of both LLaVA-NeXT-LLaMA3 and Qwen2.5-VL, facilitating them to surpass other state-of-the-art MLLMs. 66.3 61.2 53.3 55.3 70.9 74.5 80.1 60.5 85.5 75.6 69.9 70.9 70.6 78.1 92.5 90.1 71.8 95.5 60.4 57.7 51.1 49.0 65.3 70.8 72.1 55.9 78.8 40 50 60 70 80 90 100 G L M - 4 V m i n i g p t - v 2 i n t e r n V L L L a v a - v 1 . 5 G e m i n i - 2 . 0 - F l a s h G P T - 4 o C l a u d e 3 . 5 - S o n n e t 2 Q w e n - 2 . 5 V L o u r s A v e r a g e P e r f o r m a n c e ( % ) Performance SRERSER Figure 5: Main results on the proposed RSBench. Our SRPO framework outperforms other methods in both safety and ef- fectiveness of the reasoning paths. Qwen2.5-VL, both models achieve substantial gain on the selected challenging cross-modal safety benchmarks. On av- erage, LLaVA-NeXT-LLaMA3 and Qwen2.5-VL exhibit performance improvements of 24.87% and 28.60%, respec- tively, accompanied by reductions in the attack success rate (ASR) of 23.92% and 20.84%. These findings underscore the effectiveness of SRPO in strengthening safety reasoning among various MLLMs. Specifically,onMSSBenchandVLSBench, Qwen2.5-SRPO achieves notable improvements of 18.53%and46.98%,respectively.Theremarkable gain on VLSBench highlights the enhanced capability of SRPO in handling more challenging and nuanced safety risks. On USBBench, MLLM-GUARD, and SafeBench, Qwen2.5-SRPO reduced the ASR by 17.39%, 20.43%, and 24.7%, respectively. In addition, the model’s safety-issues-detection ability improved by 27.08% on MLLM-GUARD and 21.8% on SafeBench. Following the integration of SRPO, both LLaVA-NeXT-LLaMA3 and Qwen2.5-VL exhibit strong safety performance that ex- ceeds most commercial MLLMs. The results above further validate the effectiveness of our method in enhancing the safety reasoning capabilities of MLLMs. RSBench While existing evaluations predominantly focus on assess- ing the final output of the model, they often overlook the quality of the intermediate CoTs of the safety reasoning pro- cess. To address this issue, we introduce the Reasoning Path Benchmark (RSBench), which provides a more comprehen- sive evaluation of MLLM’s safety reasoning capabilities. RSBench leverages GPT-4o as an arbitration model and in- troduces two key metrics: safety rate (SR) and effectiveness rate (ER). SR quantifies the proportion of reasoning paths deemed safe, while the ER captures the proportion of rea- soning paths considered practically useful. Formally, these metrics are defined as: SR = 1 N N h X i=1 f h (i) , ER = 1 N N r X j=1 f r (j),(10) where N h , N r , and N represent the number of safe re- sponses, effective responses, and total responses, respec- tively. f h (i) and f r (j) are indicator functions. f h (i) equals to 1 if the i-th query yields a safe response and 0 otherwise. MLLM-GUARD SafeBench RSBench USBBench VLSBench MSSBench Base SFT DPO ORPO SRPO (ours) Figure 6: Main results of our proposed SRPO framework against different optimization baselines on six benchmarks. Similarly, f r (j) equals 1 if the j-th query yields an effective response and 0 otherwise. To enable a unified evaluation of both safety and effec- tiveness in the CoT reasoning process, we further define the safety and effectiveness rate (SER), which quantifies the proportion of reasoning paths that simultaneously satisfy both safety and effectiveness criteria: SER = 1 N N X k=1 [f h (k)· f r (k)].(11) As shown in Figure 5, we evaluate our proposed model Qwen2.5-SRPO and 8 families of advanced MLLMs on RSBench. The experimental results indicate that Qwen2.5-SRPO significantly outperforms its base model Qwen-2.5VL, with a more than 20% absolute gain in both SR and ER. Furthermore, Qwen2.5-SRPO also exhibits safer and more effective reasoning paths against the selected leading closed-source MLLMs. Optimization Baselines To further validate the effectiveness of our proposed method, we conduct a comparative analysis against several popu- lar optimization baselines, which include both reasoning- focused training approaches and preference-based opti- mization techniques, i.e. SFT, DPO and ORPO. With Qwen-2.5VL as the base model, we apply each opti- mization method and evaluate their performance alongside Qwen2.5-SRPO on the five previously introduced bench- marks as well as our proposed RSBench. To ensure a fair comparison, we standardized the data setup so that each optimization baseline is trained on all applicable samples. Specifically, SFT utilizes the entire dataset, as its format requires only the input question and the ground-truth answer. DPO and ORPO, on the other hand, are exclusively trained on samples from our SSUI dataset that include at least one correct and one incorrect reasoning path generated from the exploration stage. All optimization baselines except SFT employ a fixed temperature parameter when generating reasoning paths via CoT prompting. As demonstrated in Figure 6, the results reveal that our method consistently outperforms all optimization baselines 41.141.141.141.141.1 23.4 24.9 23.8 23.1 22.8 47.6 47.6 47.6 47.6 47.6 27.1 28.6 27.5 27.1 26.7 30 50 70 90 A v e r a g e ( % ) Loss Weight λ LLaVA-NeXT-LLaMA3LLaVA-NeXT-LLaMA3+SRPO Qwen2.5-VLQwen2.5-VL+SRPO λ = 0.1 λ = 0.3 λ = 0.5 λ = 0.7 λ = 0.9 Figure 7: Influence of the loss weight λ on the safety align- ment performance of MLLMs. among the evaluated benchmarks. The performance gains are particularly visible on the more challenging datasets such as MSSBench, USBBench, and the CoT-centric RS- Bench, highlighting our method’s superior ability to learn from the explored reasoning paths. In contrast, SFT gener- ally underperforms in approaches leveraging self-explored reasoning, particularly on the more challenging benchmarks. This suggests that while directly predicting the correct an- swer may suffice in simpler cases, it is less effective for tasks demanding nuanced safety reasoning. Further Analysis To investigate the effect of reasoning exploration within our framework, an analysis of the loss weight parameter λ is conducted. Specifically, a smaller λ emphasizes more of the reference path to a safe answer. Conversely, a larger λ assigns more importance to the favorable and unfavorable branches generated at each reasoning step. As illustrated in Figure 7, an excessively small λ yields suboptimal re- sults, as it inadequately emphasizes reasoning exploration. Similarly, over-emphasizing exploration is not beneficial for training either, since sufficient grounding of the model in the reference path remains crucial. As a result, a trade-off between optimizing for the reference reasoning path and the exploratory branches is required. Conclusion In this work, we address the critical challenge of implicit reasoning risk in MLLMs by introducing the Safety-Aware Reasoning Path Optimization framework. Supported by our proposed SSUI dataset and RSBench benchmark, SRPO leverages generative exploration and contrastive optimiza- tion to steer the model towards safe reasoning paths. Ex- tensive experiments demonstrate that our SRPO-enhanced model achieves SOTA results on key safety benchmarks, outperforming even leading commercial MLLMs. These re- sults verify that aligning the reasoning process itself is a more robust safety strategy than merely filtering outputs, thereby presenting a new paradigm for building fundamen- tally more trustworthy AI by ensuring the integrity of their thought processes. References Agarwal, P.; Betancourt, A.; Panagiotou, V.; and D ́ ıaz- Rodr ́ ıguez, N. 2020. Egoshots, an ego-vision life-logging dataset and semantic fidelity metric to evaluate diversity in image captioning models. arXiv preprint arXiv:2003.11743. Bai, J.; Bai, S.; Chu, Y.; Cui, Z.; Dang, K.; Deng, X.; Fan, Y.; Ge, W.; Han, Y.; Huang, F.; et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609. Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; et al. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. Bengio, Y.; Ducharme, R.; Vincent, P.; and Jauvin, C. 2003. A neural probabilistic language model. Journal of machine learning research, 3(Feb): 1137–1155. Chen, J.; Zhu, D.; Shen, X.; Li, X.; Liu, Z.; Zhang, P.; Kr- ishnamoorthi, R.; Chandra, V.; Xiong, Y.; and Elhoseiny, M. 2023. Minigpt-v2: large language model as a unified inter- face for vision-language multi-task learning. arXiv preprint arXiv:2310.09478. Chen, Z.; Wu, J.; Wang, W.; Su, W.; Chen, G.; Xing, S.; Zhong, M.; Zhang, Q.; Zhu, X.; Lu, L.; et al. 2024. Internvl: Scaling up vision foundation models and align- ing for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 24185–24198. Dong, Y.; Liu, Z.; Sun, H.-L.; Yang, J.; Hu, W.; Rao, Y.; and Liu, Z. 2025. Insight-v: Exploring long-chain visual reason- ing with multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, 9062–9072. GLM, T.; Zeng, A.; Xu, B.; Wang, B.; Zhang, C.; Yin, D.; Zhang, D.; Rojas, D.; Feng, G.; Zhao, H.; et al. 2024. Chat- glm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793. Gong, Y.; Ran, D.; Liu, J.; Wang, C.; Cong, T.; Wang, A.; Duan, S.; and Wang, X. 2025. Figstep: Jailbreaking large vision-language models via typographic visual prompts. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 39, 23951–23959. Gu, T.; Zhou, Z.; Huang, K.; Dandan, L.; Wang, Y.; Zhao, H.; Yao, Y.; Yang, Y.; Teng, Y.; Qiao, Y.; et al. 2024. Mllm- guard: A multi-dimensional safety evaluation suite for mul- timodal large language models. Advances in Neural Infor- mation Processing Systems, 37: 7256–7295. Hong, J.; Lee, N.; and Thorne, J. 2024. Orpo: Monolithic preference optimization without reference model.arXiv preprint arXiv:2403.07691. Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; Chen, W.; et al. 2022. Lora: Low-rank adapta- tion of large language models. ICLR, 1(2): 3. Hu, X.; Liu, D.; Li, H.; Huang, X.; and Shao, J. 2024. Vls- bench: Unveiling visual leakage in multimodal safety. arXiv preprint arXiv:2411.19939. Kuznetsova, A.; Rom, H.; Alldrin, N.; Uijlings, J.; Krasin, I.; Pont-Tuset, J.; Kamali, S.; Popov, S.; Malloci, M.; Kolesnikov, A.; et al. 2020. The open images dataset v4: Unified image classification, object detection, and visual re- lationship detection at scale. International journal of com- puter vision, 128(7): 1956–1981. Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023. Blip-2: Boot- strapping language-image pre-training with frozen image encoders and large language models. In International con- ference on machine learning, 19730–19742. PMLR. Li, X.; Zhou, H.; Wang, R.; Zhou, T.; Cheng, M.; and Hsieh, C.-J. 2024. Mossbench: Is your multimodal lan- guage model oversensitive to safe queries? arXiv preprint arXiv:2406.17806. Lin, J.; Yin, H.; Ping, W.; Molchanov, P.; Shoeybi, M.; and Han, S. 2024. Vila: On pre-training for visual language mod- els. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, 26689–26699. Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ra- manan, D.; Doll ́ ar, P.; and Zitnick, C. L. 2014. Microsoft coco: Common objects in context. In European conference on computer vision, 740–755. Springer. Ling, Z.; Fang, Y.; Li, X.; Huang, Z.; Lee, M.; Memisevic, R.; and Su, H. 2023. Deductive verification of chain-of- thought reasoning. Advances in Neural Information Pro- cessing Systems, 36: 36407–36433. Liu, H.; Li, C.; Li, Y.; and Lee, Y. J. 2024a. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 26296–26306. Liu, H.; Li, C.; Li, Y.; Li, B.; Zhang, Y.; Shen, S.; and Lee, Y. J. 2024b. Llavanext: Improved reasoning, ocr, and world knowledge. Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual in- struction tuning. Advances in neural information processing systems, 36: 34892–34916. Liu, X.; Zhu, Y.; Gu, J.; Lan, Y.; Yang, C.; and Qiao, Y. 2024c. Mm-safetybench: A benchmark for safety evaluation of multimodal large language models. In European Confer- ence on Computer Vision, 386–403. Springer. Lu, H.; Liu, W.; Zhang, B.; Wang, B.; Dong, K.; Liu, B.; Sun, J.; Ren, T.; Li, Z.; Yang, H.; et al. 2024. Deepseek- vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525. Meng, Y.; Xia, M.; and Chen, D. 2024. Simpo: Simple pref- erence optimization with a reference-free reward. Advances in Neural Information Processing Systems, 37: 124198– 124235. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information pro- cessing systems, 35: 27730–27744. Pi, R.; Han, T.; Zhang, J.; Xie, Y.; Pan, R.; Lian, Q.; Dong, H.; Zhang, J.; and Zhang, T. 2024. Mllm-protector: Ensuring mllm’s safety without hurting performance. arXiv preprint arXiv:2401.02906. Qi, X.; Huang, K.; Panda, A.; Henderson, P.; Wang, M.; and Mittal, P. 2024. Visual adversarial examples jailbreak aligned large language models. In Proceedings of the AAAI conference on artificial intelligence, volume 38, 21527– 21536. Rafailov, R.; Sharma, A.; Mitchell, E.; Manning, C. D.; Er- mon, S.; and Finn, C. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36: 53728–53741. Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Team, G.; Georgiev, P.; Lei, V. I.; Burnell, R.; Bai, L.; Gulati, A.; Tanzer, G.; Vincent, D.; Pan, Z.; Wang, S.; et al. 2024. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.05530. Wang, P.; Bai, S.; Tan, S.; Wang, S.; Fan, Z.; Bai, J.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; et al. 2024a. Qwen2-vl: En- hancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191. Wang, S.; Ye, X.; Cheng, Q.; Duan, J.; Li, S.; Fu, J.; Qiu, X.; and Huang, X. 2024b. Safe Inputs but Unsafe Output: Benchmarking Cross-modality Safety Alignment of Large Vision-Language Model. arXiv preprint arXiv:2406.15279. Wang, Z.; Bi, B.; Pentyala, S. K.; Ramnath, K.; Chaudhuri, S.; Mehrotra, S.; Mao, X.-B.; Asur, S.; et al. 2024c. A com- prehensive survey of llm alignment techniques: Rlhf, rlaif, ppo, dpo and more. arXiv preprint arXiv:2407.16216. Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; Zhou, D.; et al. 2022.Chain-of- thought prompting elicits reasoning in large language mod- els. Advances in neural information processing systems, 35: 24824–24837. Yao, S.; Yu, D.; Zhao, J.; Shafran, I.; Griffiths, T.; Cao, Y.; and Narasimhan, K. 2023. Tree of thoughts: Deliber- ate problem solving with large language models. Advances in neural information processing systems, 36: 11809–11822. Yao, Y.; Yu, T.; Zhang, A.; Wang, C.; Cui, J.; Zhu, H.; Cai, T.; Li, H.; Zhao, W.; He, Z.; et al. 2024. Minicpm- v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800. Ying, Z.; Liu, A.; Liang, S.; Huang, L.; Guo, J.; Zhou, W.; Liu, X.; and Tao, D. 2024. Safebench: A safety evaluation framework for multimodal large language models. arXiv preprint arXiv:2410.18927. Yue, Y.; Chen, Z.; Lu, R.; Zhao, A.; Wang, Z.; Song, S.; and Huang, G. 2025. Does reinforcement learning really in- centivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837. Zhang, Z.; Zhang, A.; Li, M.; Zhao, H.; Karypis, G.; and Smola, A. 2023. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923. Zheng, B.; Chen, G.; Zhong, H.; Teng, Q.; Tan, Y.; Liu, Z.; Wang, W.; Liu, J.; Yang, J.; Jing, H.; et al. 2025. USB: A Comprehensive and Unified Safety Evaluation Benchmark for Multimodal Large Language Models. arXiv preprint arXiv:2505.23793. Zhou, K.; Liu, C.; Zhao, X.; Compalas, A.; Song, D.; and Wang, X. E. 2024. Multimodal situational safety. arXiv preprint arXiv:2410.06172. Zhu, D.; Chen, J.; Shen, X.; Li, X.; and Elhoseiny, M. 2023. Minigpt-4: Enhancing vision-language understand- ing with advanced large language models. arXiv preprint arXiv:2304.10592.