Paper deep dive
OutSafe-Bench: A Benchmark for Multimodal Offensive Content Detection in Large Language Models
Yuping Yan, Yuhan Xie, Yuanshuai Li, Yingchao Yu, Lingjuan Lyu, Yaochu Jin
Models: Claude-3.7-Sonnet, Deepseek-v3, GPT-4o
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/11/2026, 1:08:27 AM
Summary
OutSafe-Bench is a comprehensive multimodal safety evaluation benchmark for Large Language Models (MLLMs), covering text, image, audio, and video across nine risk categories in both English and Chinese. It introduces the Multidimensional Cross Risk Score (MCRS) for assessing correlated risks and the FairScore framework, which uses a weighted ensemble of adaptive jury models to provide robust, explainable, and bias-mitigated safety evaluations.
Entities (4)
Relation Signals (3)
FairScore → evaluates → MLLMs
confidence 95% · Our evaluation of nine state-of-the-art MLLMs reveals persistent and substantial safety vulnerabilities
OutSafe-Bench → implements → FairScore
confidence 95% · To ensure fair and robust evaluation, we propose FairScore, an explainable automated multi-reviewer weighted aggregation framework.
OutSafe-Bench → utilizes → MCRS
confidence 95% · OutSafe-Bench includes a large-scale dataset... In addition to the dataset, we introduce a Multidimensional Cross Risk Score (MCRS)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Since Multimodal Large Language Models (MLLMs) are increasingly being integrated into everyday tools and intelligent agents, growing concerns have arisen regarding their possible output of unsafe contents, ranging from toxic language and biased imagery to privacy violations and harmful misinformation. Current safety benchmarks remain highly limited in both modality coverage and performance evaluations, often neglecting the extensive landscape of content safety. In this work, we introduce OutSafe-Bench, the first most comprehensive content safety evaluation test suite designed for the multimodal era. OutSafe-Bench includes a large-scale dataset that spans four modalities, featuring over 18,000 bilingual (Chinese and English) text prompts, 4,500 images, 450 audio clips and 450 videos, all systematically annotated across nine critical content risk categories. In addition to the dataset, we introduce a Multidimensional Cross Risk Score (MCRS), a novel metric designed to model and assess overlapping and correlated content risks across different categories. To ensure fair and robust evaluation, we propose FairScore, an explainable automated multi-reviewer weighted aggregation framework. FairScore selects top-performing models as adaptive juries, thereby mitigating biases from single-model judgments and enhancing overall evaluation reliability. Our evaluation of nine state-of-the-art MLLMs reveals persistent and substantial safety vulnerabilities, underscoring the pressing need for robust safeguards in MLLMs.
Tags
Links
- Source: https://arxiv.org/abs/2511.10287
- Canonical: https://arxiv.org/abs/2511.10287
Trouble viewing inline? Open PDF directly →
Full Text
63,954 characters extracted from source content.
Expand or collapse full text
OutSafe-Bench: A Benchmark for Multimodal Offensive Content Detection in Large Language Models Yuping Yan 1∗ , Yuhan Xie 1∗ , Yuanshuai Li 1 , Yingchao Yu 1 , Lingjuan Lyu 2 , Yaochu Jin 1† 1 TGAI Lab, School of Engineering, Westlake University 2 Sony Research, Sony yanyuping,xieyuhan,liyuanshuai,yuyingchao,jinyaochu@westlake.edu.cn lingjuanlvsmile@gmail.com ∗ Joint first authors † Corresponding author Abstract Since Multimodal Large Language Models (MLLMs) are in- creasingly being integrated into everyday tools and intelli- gent agents, growing concerns have arisen regarding their possible output of unsafe contents, ranging from toxic lan- guage and biased imagery to privacy violations and harmful misinformation. Current safety benchmarks remain highly limited in both modality coverage and performance eval- uations, often neglecting the extensive landscape of con- tent safety. In this work, we introduce OutSafe-Bench, the first most comprehensive content safety evaluation test suite designed for the multimodal era. OutSafe-Bench includes a large-scale dataset that spans four modalities, featuring over 18,000 bilingual (Chinese and English) text prompts, 4,500 images, 450 audio clips and 450 videos, all system- atically annotated across nine critical content risk cate- gories. In addition to the dataset, we introduce a Multidi- mensional Cross Risk Score (MCRS), a novel metric de- signed to model and assess overlapping and correlated con- tent risks across different categories. To ensure fair and ro- bust evaluation, we propose FairScore, an explainable au- tomated multi-reviewer weighted aggregation framework. FairScore selects top-performing models as adaptive juries, thereby mitigating biases from single-model judgments and enhancing overall evaluation reliability. Our evaluation of nine state-of-the-art MLLMs reveals persistent and substan- tial safety vulnerabilities, underscoring the pressing need for robust safeguards in MLLMs. Warning: This paper may contain some offensive content in data and model outputs. 1. Introduction Multimodal Large Language Models (MLLMs), which ex- tend traditional large language models (LLMs) by integrat- ing vision, audio, and video inputs alongside text, have rapidly advanced the frontiers of artificial intelligence. These models now demonstrate remarkable performance not only in classic vision tasks [15, 63], but also in more complex multimodal challenges such as visual question an- swering [14], image captioning [66], video understanding [65, 68], and cross-modal reasoning [58]. However, the broadened input and interaction modalities expose MLLMs to a wider surface of safety and ethical risks [11, 50], partic- ularly as the generation of harmful or inappropriate content by LLMs becomes increasingly prevalent. Recent incidents have underscored the urgency of this issue. A teenager in Florida and a man in Belgium died by suicide following in- teractions with AI chatbots that reportedly reinforced harm- ful ideologies and suicidal thoughts [7, 54]. These tragedies highlight the severe consequences of unregulated and un- safe generative outputs. Although several studies have examined the trustworthi- ness of MLLMs [18, 25, 59, 67], these approaches prioritize overall system security, yet they overlook the fine-grained risks embedded in generated outputs, including toxic lan- guage, stereotypical bias, and privacy leakage. In addition, modality coverages and bilingual evaluations remain insuf- ficiently addressed, leaving important gaps in the compre- hensive assessment of content-level safety in MLLMs. By reviewing existing benchmarks, we identify the following main challenges in comprehensive evaluation of LLM-generated content: 1) Limited modality cover- age. Most existing trustworthiness benchmarks are con- strained to up to three modalities, text, image, and audio, such as SafeBench [59], while a full-spectrum benchmark that includes video remains absent. 2) Insufficient evalua- tion of cross-domain and lingual impacts. Current bench- marks lack a comprehensive matrix to evaluate overlap- ping risks, where a single output can violate multiple se- curity issues (e.g., toxic language and biased imagery). Al- though MLLMGUARD [18] provides bilingual evaluation, 1 arXiv:2511.10287v3 [cs.LG] 11 Dec 2025 Modalities Benchmark#VolumeTextImageAudioVideoEvaluation MethodFocus M-SafetyBench5,040!!%%GPT-4-aided + manual reviewSafety PrivQA6,678!!%%F1 + Protection scorePrivacy Unicorn8,500%!%%Human-evaluatedImage Security Safe-Bench9,200!!!%Jury deliberation protocolSafety MultiTrust23,000!!%%Rule-based + GPT-4-aidedSafety MLLMGUARD2,282!!%%Rule-basedTrustfulness OutSafe-Bench (ours)23,400!!!!FairScoreContent safety Table 1. Comparison between OutSafe-Bench and other trustworthiness-related benchmarks for MLLMs. no benchmark systematically assesses multifaceted risks across domains, languages, and modalities. 3) Lack of fair and explainable evaluation mechanisms. Automated multi- LLM reviewer protocols [59] are commonly employed for evaluation but often depend on a fixed set of mainstream models for all tasks. This uniform approach introduces potential biases and lacks transparency regarding model- specific expertise and decision-making processes. To address the limitations of existing benchmarks, we propose OutSafe-Bench, a comprehensive content safety evaluation suite for MLLMs, which includes a bilingual multimodal dataset and a fair, automated evaluation mecha- nism. Our key contributions are as follows: • WeintroduceOutSafe-Bench,thefirstmulti- dimensional content safety benchmark for MLLMs that covers four modalities, text, image, audio, and video, in both Chinese and English. To the best of our knowledge, this is the most comprehensive and largest benchmark focused on content-level safety to date. • We propose a novel Multidimensional Cross Risk Score (MCRS), a function designed to capture and quantify the co-occurrence and interactions of multiple content risks within a single output. Utilizing a cross-risk influence ma- trix, MCRS enables a more nuanced evaluation of com- plex, real-world safety threats, providing a holistic view of content risks. • To ensure fairness and robustness in evaluation, we de- velop the FairScore mechanism. This weighted ensemble approach aggregates scores from a dynamically selected set of reviewer models based on their baseline perfor- mance. By mitigating bias and enhancing the reliability of automatic safety assessments, FairScore offers a transpar- ent and trustworthy evaluation framework. Our in-depth evaluation of nine leading MLLMs provides critical in- sights for future model optimization in content safety. 2. Related Work 2.1. MLLM safety benchmarks Recent efforts have made notable progress in evaluating the trustworthiness of LLMs, as shown in Table 1. Bench- marks such as TrustLLM [50] and Trustworthy LLMs [34] offer comprehensive assessments across a wide range of di- mensions, including toxicity, stereotype bias, adversarial ro- bustness, and privacy. In parallel, general-purpose bench- marks for MLLMs have emerged to evaluate their capabili- ties in visual perception, knowledge acquisition, reasoning, and commonsense understanding, as seen in works such as LVLM-Ehub [60] and LAMM [62]. However, trustworthiness evaluation for MLLMs re- mains limited, especially in terms of modality coverage and content-level safety, where existing work tends to focus more on adversarial attacks than on holistic content safety. For instance, M-SafeBench [32] is designed to assess safety in text-image scenarios, with an emphasis on image- based prompt attacks that exploit query-relevant visual in- puts to trigger harmful model behavior. PrivQA [25] intro- duces a privacy-focused benchmark containing 4,678 open- domain textual and 2,000 visual QA examples, targeting risks related to sensitive content exposure. Unicorn [52] ad- dresses visual content safety by providing 8,500 annotated threat scenarios derived from images. SafeBench [59] is the first benchmark to support audio modality, featuring 2,300 text samples, 2,300 images, and 4,600 audio clips across sensitive domains such as consultation, cybersecurity, and ethics. However, its dataset size is relatively limited, and its focus spans a broad range of safety topics rather than em- phasizing content-level risks. MultiTrust [67] presents the largest dataset to date, with over 15,000 text-image sam- ples and additional text-only tasks. It is also the first bench- mark to examine cross-modal vulnerabilities, particularly from image-to-text generation, yet it offers limited coverage of modality types and content-specific safety aspects, shar- ing similar limitations with SafeBench. MLLMGuard [18] 2 Figure 1. Overview of the OutSafe-Bench framework. It includes content risk aspect division across four modalities, as well as the evalua- tion strategy and the design of the developed toolbox. Notably, OutSafe-Bench incorporates a fair and explainable evaluation mechanism, featuring cross-dimensional risk weighting and adaptive jury-based scoring. is the only benchmark that supports bilingual safety eval- uation of MLLM-generated outputs (in English and Chi- nese). Despite its valuable multilingual perspective, it is constrained by a smaller dataset size and narrow modality support. 2.2. Evaluation strategy of MLLMs Mainstream approaches to evaluating the properties of MLLMs can be broadly categorized into three types: rule- based evaluation [28], manual review, and LLM-as-judge methods [69]. Among these, LLM-as-judge approaches have gained popularity for their scalability and automation, which include single-LLM systems, multi-LLM systems, and hybrid systems. Single-LLM systems, such as LLM- Eval [30] and G-Eval [33], are easy to deploy and scale but may suffer from individual model biases, which potentially results in inaccurate or skewed evaluations. In multi-LLM systems [9, 27], multiple models interact through mecha- nisms like collaboration or deliberation to refine judgments and reach more robust conclusions. For instance, SafeBench [59] employs a jury deliberation protocol, where multiple LLMs collectively assess content and check for consensus. However, these systems often select mainstream models as evaluators without justifying their expertise across differ- ent tasks, which may compromise fairness and task-specific reliability. Hybrid systems [26, 45] combine LLMs with hu- man evaluators. This configuration enables human oversight to mitigate potential biases and bring subjective reasoning into complex evaluations. However, it also introduces chal- lenges such as coordination overhead, inconsistency in stan- 3 dards, and difficulty in integrating human feedback effec- tively. 3. Practice in OutSafe-Bench In this section, we present the overall framework of OutSafe-Bench as shown in Figure 1, along with a detailed explanation of the dataset curation with the quality control procedure and evaluation protocol. 3.1. Taxonomy OutSafe-Bench incorporates nine dimensions of content safety, including privacy and property, prejudice and dis- crimination, crime and illegal activities, ethics and morality, violence and hatred, false information and misdirection, po- litical sensitivity, physical and mental health, and copyright and intellectual property. Representative examples for each category are illustrated in Figure 2. 3.2. Dataset curation with quality control To construct OutSafe-Bench, we leveraged the extensive and publicly available metadata across multiple modali- ties. Through a systematic investigation, we curated diverse sources and identified a seed set of 30 high-quality datasets that align with key content safety dimensions across text, image, audio, and video. A detailed breakdown of the data sources and domains is provided in the dataset generation section of the Supplementary Material. 3.2.1. Text domain. To construct a high-coverage, bilingual benchmark for tex- tual content safety, we aggregate samples from a wide range of publicly available datasets. For Chinese, we draw exten- sively from the Chinese Safety Prompts dataset [49], which provides diverse and realistic prompts across privacy, crime, ethics, and more. For both Chinese and English, we incor- porate samples from BBQ [38], HEx-PHI [41], SWSR [23], MCFEND [29], and DAIC-WOZ [1]. These datasets col- lectively cover a broad range of safety concerns such as personal identifiable information, incitement of illegal ac- tivities, hate speech, and intellectual property violations. To further diversify the dataset, we employ keyword-driven prompt templates in conjunction with GPT-4o to gener- ate semantically aligned high-risk queries. For instance, we use templates that incorporate specific phrases such as ”how to bypass surveillance”, ”fake identity generation”, or ”pi- rated content distribution”. These templates are designed to elicit outputs that reflect nuanced violations across mul- tiple safety dimensions. In total, we construct a dataset of 18,000 samples, evenly split between Chinese and English, with 1,000 samples per risk category. 3.2.2. Audio domain. Audio samples are sourced from established datasets, in- cluding the Hate Speech Detection Dataset [44], ETHOS [36], and the Audio Deepfake Detection corpus [20]. How- ever, due to the limited availability of mature open-source audio datasets, we also collected labeled audio data from public audio websites such as Freesound [16] and AudioSet [43]. To process these audio samples, we leverage the inher- ent speech recognition capabilities of MLLMs to transcribe speech into text. To enhance transcription quality, we apply spectral subtraction [8] to reduce background noise. Ad- ditionally, we utilize LLM-based voice activity detection techniques [6] to effectively isolate speech segments from non-speech portions of the audio. In total, we construct a dataset of 450 audio clips, comprising 170 in Chinese and 280 in English. 3.2.3. Image domain. For the image domain, we aggregate a diverse set of datasets including ICDAR [10], LayoutLMv2 [61], FUNSD [22], Image-based Table Recognition [70], Places 365[71], Fair- Face [24], Pascal VOC [13], the ResNet50 Inappropri- ate Content Detection dataset [37], and BigGAN dataset [57]. This collection ensures broad coverage across visual document understanding, scene recognition, fairness and bias evaluation, as well as harmful content detection. To enhance the quality of image-to-text translation, we ap- ply keyword matching to the description generated by MLLMs. Specifically, we compare the MLLM-generated descriptions against a predefined set of critical keywords relevant to content safety. If the number of matched key- words exceeds a predefined threshold, the image descrip- tion is considered valid and retained. Otherwise, the sample is flagged for manual review to ensure accuracy. Images for which the model fails to provide an accurate description are excluded from the dataset, mitigating errors caused by the limited image recognition capabilities of current MLLMs. In total, we compile a dataset of 4,500 images. 3.2.4. Video domain. For the video domain, we collect samples from publicly available datasets including PV-VTT [35], CCL-23 [46], MultiHateClip [56], UCA [47], HateMM [12], FakeSV [39], PDID [55], and VCLS [19], covering a range of safety risks such as hate speech, misinformation, and po- litical sensitivity. To improve the quality of video-to-text translation, we incorporate temporal instruction tuning to guide MLLMs in understanding longer sequences [64]. Each video is treated as a sequence of frames, with one frame sampled per second. Given the limitations of most MLLMs in handling long temporal inputs, we restrict each video to a maximum length of five minutes. Frames are extracted and processed using the same image-level qual- ity control mechanism described in the image domain sec- tion, including semantic keyword alignment and filtering of low-quality samples. In total, we compile a dataset of 450 videos, comprising 150 in Chinese and 300 in English. 4 Figure 2. Examples of dataset instances across nine content safety scenarios, showcasing Chinese text, English text, image, and video modalities. The audio modality is not displayed in this figure. 3.3. Evaluation protocol 3.3.1. Multidimensional Cross Risk Score (MCRS). A single output generated by an MLLM may simultane- ously involve multiple types of content risks. For instance, one response might infringe upon both privacy and intellec- tual property. To effectively capture and quantify such mul- tifaceted and co-occurring risks, we introduce the concept of a MCRS function, which incorporates both individual risk scores and their inter-category relationships. Formally, for any model output sample x, we define a risk score vector across the nine predefined content risk dimensions as: R(x) = [r 1 (x),r 2 (x),...,r 9 (x)], where r i (x)∈ [0, 10] (1) Each r i (x) represents the severity score of risk category i for output x, measured on a scale from 0 (no risk) to 10 (extremely high risk). Meanwhile, the severity of one risk category often in- fluences or correlates with others. For example, crime and illegal activities may strongly co-occur with violence and hatred, but not necessarily with political sensitivity. To ac- count for such inter-dependencies, we introduce the Cross- Risk Influence Matrix, which encodes the relative contri- bution or influence of each risk dimension in the final ag- gregation. Formally, we use sentence-bert [42] to encode our nine critical content risk categories as semantic vectors, based on the cosine similarity between these nine embed- ded vectors, we constructed a 9× 9 similarity matrix. Each row in the matrix represents the semantic relevance of the risk category to other risk dimensions, and by normalizing it, we can get the Cross-Risk Influence Matrix as follows. γ = γ (1,1) γ (1,2) · γ (1,9) γ (2,1) γ (2,2) · γ (2,9) . . . . . . . . . . . . γ (9,1) γ (9,2) · γ (9,9) (2) where γ (p,q) represents the importance weight of risk category q in the output scenario of risk category p. These weights satisfy the normalization condition: 9 X q=1 γ (p,q) = 1, γ (p,q) ∈ [0, 1](3) 3.3.2. FairScore mechanism. Given the varying risk assessment capabilities of different MLLMs, relying on a single model for evaluation may in- troduce bias and compromise reliability. To address this, we introduce the FairScore mechanism, a weighted ensemble approach that integrates judgments from multiple reviewer models. 5 Firstly, each reviewer model RM l is assigned a weight λ l , reflecting its reliability, as determined by its perfor- mance on a standardized baseline dataset. Secondly, for the target model M j , given the t-th output under the k-th con- tent risk scenario, its final evaluation score on the i-th risk dimension is computed as the weighted average of scores from all reviewer models: ˆr (j,k,t) i = m X l=1 λ l · r (j,k,t,l) i (4) wherer (j,k,t,l) i is the score assigned by reviewer model RM l to the output from M j in scenario k, sample t, on risk di- mension i, λ l is the weight of reviewer model RM l , and m is the total number of reviewer models. To obtain the overall risk score of model M j on dimen- sion i under scenario k, we average the individual sample scores across all n ′ evaluated outputs: ̄r (j,k) i = 1 n ′ n ′ X t=1 ˆr (j,k,t) i (5) where n ′ is the total number of evaluated outputs for sce- nario k. In the end, for a target model M j , the overall content safety score under the k-th safety risk scenario is defined as: R (j,k) = γ k · r (j,k) = 9 X q=1 γ (k,q) · ̄r (j,k) q (6) where γ (k,q) represents the importance weight of the q-th risk dimension in scenario k, and ̄r (j,k) q is the averaged risk score for that dimension. This score measures the overall safety performance of the target model M j given inputs cor- responding to the k-th content risk scenario. 4. Experiments and Analysis 4.1. Experimental setup 4.1.1. Evaluated models. We evaluate the content safety performance of nine main- stream MLLMs: (1) Deepseek-v3 [31], (2) Claude-3.7- Sonnet [2], (3) Doubao-1.5-pro-32k [3], (4) Gemini-2-flash [51], (5) Gemini-2.5-flash [17], (6) GPT-4o [21], (7) GPT- 4o-mini [72], (8) Qwen-2.5-72b [4], and (9) Ernie-4.0 [5]. All models are evaluated using their officially released weights. 4.1.2. Cross-risk influence matrix value. Based on sentence-BERT embeddings [42], we derived the weights for the nine content risk categories, capturing the semantic relations among them. The original and normal- ized results are shown in Figures 3(a) and 3(b), respectively. Here, the parameter γ serves as a cross-risk weighting fac- tor, quantifying how multiple risks co-occur within a single output and measuring the joint triggering of risks without altering the intrinsic risk level of each category. Figure 3. Cross-risk influence matrix for the nine content risk cat- egories. (a) The original co-relationship weights among risk cate- gories, illustrating how risks tend to co-occur. (b) Normalized ma- trix used to derive the final category weights for multi-risk scoring. 4.1.3. Jury models. The jury models are selected based on the top five best- performing MLLMs. Since all modality outputs are trans- lated into text for evaluation, we use text-based safety benchmarks to determine the jury models and assign weights accordingly. Specifically, we use the Chinese Safety Prompts dataset [49] for evaluating Chinese outputs, and Case-Bench [48] for English outputs. For each model, we compute the jury weight λ from pre-experiments on these benchmarks based on their measured safety classi- fication accuracies. Averaging λ across English and Chi- nese treats the MLLM jury as a unified evaluator, ensur- ing that FairScore reflects overall reliability rather than lan- guage bias, as the two languages contribute nearly equal proportions to OutSafe-Bench, and we get our jury mod- els as Claude-3.7-sonnet, Deepseek-v3, GPT-4o, GPT-4o- mini, and Ernie-4.0-8k. This approach also allows FairScore to scale seamlessly to additional languages. By assigning weights to the MLLMs, we limit the influence of any indi- vidual model and ensure fairness even when a model serves as both evaluator and evaluatee, eliminating the need for a leave-out strategy. The resulting aggregate evaluation scores and jury weights are summarized in Table 3. 4.2. Benchmark results 4.2.1. Content safety evaluation in terms of modalities. Table 2 presents the modality-specific safety risk scores for nine leading MLLMs. Claude-3.7-Sonnet achieves the lowest average risk score (0.7436) across the supported modalities, with the best performance in image safety and strong results in both Chinese and English text. Deepseek- v3 closely follows with an average score of 0.8130, show- ing balanced performance across all available modalities. 6 Modality-Specific Safety Risk Score↓Average Ch TextEn TextImageAudioVideoRisk↓ Full-Modality MLLMs (support all: text, image, audio, video) Gemini-2.0-flash0.56361.46150.89111.37012.15141.2875 Gemini-2.5-flash0.58961.73431.14381.13272.40671.4014 Qwen-2.5-72b0.35461.25790.62991.33671.01740.9193 Partial-Modality MLLMs (missing one or more modalities) Claude-3.7-Sonnet0.45731.19290.5807–0.7436 Deepseek-v30.42971.34930.6600–0.8130 Doubao-1.5-pro 0.54821.88480.6991–1.0440 GPT-4o0.34700.90180.77281.5002–0.8804 GPT-4o-mini0.36691.23200.79421.5395–1.0081 Ernie-Bot (Wenxin)0.52461.16331.0991–0.9290 Table 2. Overall safety risk scores of various MLLMs across different modalities. Scores represent risk levels, ranging from 0 to 10, and lower is better. Best results per column are bolded. Figure 4. Each model’s performance is shown per modality across nine categories: P&P (Property & Privacy), D&B (Discrimination & Bias), I&C (Illegal & Criminal), M&E (Morality & Ethics), V&H (Violence & Hate), Misinfo (Misinformation), Politics (Political Sensitivity), Health (Physical & Mental Health), IP (Copyright & Intellectual Property). ClDseekDbG4oG-mQwen Ernie En0.9220.8980.8920.9480.9360.9280.934 Ch0.872 0.8540.5890.8610.7490.5200.807 Avg0.8970.8760.7410.9040.8420.7290.870 λ0.2050.200–0.2060.189–0.198 Table 3. Safety performance and jury weights of candidate MLLMs. Evaluation scores across English (En) and Chinese (Ch) outputs are used to determine the top five models for jury assign- ment. The average score (Avg) is computed across both languages. The final row reports the normalized jury weights (λ), reflecting each model’s relative reliability in safety evaluation. Models not selected for the jury are marked with “–”. Among fully multimodal models, Qwen-2.5-72B Instruct stands out with the lowest overall risk score (0.9193), rank- ing first in video and second in image, while also perform- ing competitively in audio and Chinese text. GPT-4o demonstrates strong performance in textual modalities and maintains competitive audio safety, although it lacks video support. Gemini-2.0 Flash and Gemini-2.5 Flash provide full-modality coverage but show higher risk scores in video and English text, indicating potential safety weaknesses in complex multimodal and multilingual sce- narios. These results confirm that MLLMs are generally more robust in text-based tasks, while video and audio remain more challenging. Video, in particular, exhibits the highest risk scores across most models, highlighting the pressing need for better safety mechanisms in temporal and cross- modal understanding. 4.2.2. Content safety evaluation in terms of categories. Different MLLMs exhibit varying safety performance across the nine content risk categories. To facilitate clearer 7 visualization, we applied min–max normalization to the original scores prior to plotting. Let x ij denote the original safety risk score for model i under modality j. The normal- ized score ˆx ij ∈ [0, 1] is computed as: ˆx ij = x ij − min i (x ij ) max i (x ij )− min i (x ij ) (7) This normalization is performed independently for each modality, ensuring that the model with the highest risk in a given modality is mapped to 1, and the lowest to 0. The resulting normalized scores are visualized in Figure 4. More detailed results can be found in the Supplementary Material. Across the nine content risk categories, the models exhibit varied strengths. Claude-3.7-Sonnet consistently demonstrates strong performance in mitigating toxicity, stereotype bias, and ethical risk, maintaining low risk lev- els in these sensitive areas. Deepseek-V3 shows robustness in handling personal privacy, fake information, and dis- crimination, suggesting its reliability in protecting user data and maintaining social fairness. GPT-4o performs well in misleading content, violence, and legality, indicating solid safeguards against harmful or unlawful output. Qwen-2.5- 72B achieves low risk scores in fake information, cyberse- curity, and legality, reflecting effective content verification and rule alignment. Gemini-2.5 shows strength in toxicity and cybersecurity, although it presents occasional vulnera- bilities in categories like stereotype and bias. Doubao-1.5 manages moderate risk across most categories but tends to show slightly elevated scores in bias and misleading con- tent. Ernie-4.0-8k maintains relatively low risk in ethics and privacy, though less consistently across other categories. Overall, while some models show category-specific strengths, few can achieve uniformly low risk across all di- mensions, reinforcing the need for fine-grained, category- aware safety evaluations. 4.3. Validation and Ablation Studies To verify the effectiveness and interpretability of our eval- uation framework, we conduct compact ablation and val- idation experiments on a 936-sample subset with human- annotated safety ratings. Two key parameters are examined: the MCRS γ and the FairScore jury weight λ. 4.3.1. Ablation on MCRS. To validate the role of the cross-risk weighting factor γ, we compared two settings: (1) a simple unweighted mean of the nine risk categories, and (2) the proposed γ-based formu- lation derived from sentence-BERT embeddings and nor- malized co-occurrence relations (Figures 3(a)(b)). Evalu- ated on 936 human-annotated samples, the γ-based method achieved a higher correlation with human ratings (Spear- man ρ = 0.5672 vs. 0.5179 for the unweighted mean). This demonstrates that γ effectively captures real-world co- occurrence among safety risks, yielding more interpretable composite assessments without modifying the intrinsic se- mantics of each category. 4.3.2. Ablation on FairScore. We further compare FairScore against (a) single-model evaluation and (b) simple average evaluation across mod- els. FairScore computes juror weights λ from normalized correlations between each reviewer’s scores and human la- bels on a held-out validation set, thereby grounding weights in empirical reliability rather than fixed selection. On a 936- sample human-annotated subset, FairScore attains the high- est agreement with human judgments (Kendall τ = 0.4127, Spearman ρ = 0.5681) compared to single-model evalu- ation (Kendall τ = 0.4057, Spearman ρ = 0.5589) and average evaluation (Kendall τ = 0.3943, Spearman ρ = 0.5364). Moreover, FairScore achieves the lowest mean ab- solute error (1.3825 vs. 1.3883 for single-model and 1.3891 for average) and maintains competitive stability across sam- ples (standard deviation 1.6150), with a 30.08% variance re- duction relative to human ratings. These results confirm that reliability-based weighting stabilizes and improves align- ment with human judgments compared to single-judge or unweighted ensemble methods. Overall, these results validate that both γ and λ con- tribute to fairer, more interpretable, and more human- aligned multimodal safety evaluation without requiring sub- stantial additional computation. 5. Conclusion and Discussion In this work, we introduce Outsafe-Bench, a compre- hensive and scalable benchmark for evaluating the con- tent safety of MLLMs. Outsafe-Bench includes a large- scale, linguistically diverse multimodal dataset, along with a fair and principled scoring framework that leverages risk- weighted categories and jury-based model aggregation to address the current safety evaluation benchmark limitation (narrow modality scope, lack of linguistic diversity, and evaluator bias). Our experiments in nine state-of-the-art MLLMs uncovered widespread safety vulnerabilities, par- ticularly in image and video outputs. These results high- light the growing complexity of ensuring trustworthiness in multimodal generation and reinforce the need for more fine- grained, category-sensitive safety assessments. 5.1. Limitations Outsafe-Bench is constrained by current model capabili- ties. In particular, long-form video samples (more than five minutes) were excluded due to input length limits in ex- isting MLLMs. Future work will explore scalable methods for evaluating long-context inputs and extending support to more languages and modalities. Another limitation lies in the lack of evaluation for cross-modal alignment, such as consistency between text and visual outputs. Although we 8 assess each modality independently, future work should ex- plore joint reasoning and misalignment risks across modali- ties. References [1] Alireza Afzal Aghaei and Nadia Khodaei. Automated de- pression recognition using multimodal machine learning: A study on the daic-woz dataset. Computational Mathematics and Computer Modeling with Applications (CMCMA), 2(1): 45–53, 2023. 4 [2] Anthropic. Meet claude, your thinking partner. https://w. anthropic.com/claude, 2023. 6 [3] Apidog. Doubao 1.5 pro: Api pricing & how to use doubao- 1.5-pro api. https://apidog.com/blog/doubao- 1- 5- pro- api/, 2023. 6 [4] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al.Qwen technical report.arXiv preprint arXiv:2309.16609, 2023. 6 [5] Baidu Research. Introducing pcl-baidu wenxin (ernie 3.0 ti- tan), the world’s first knowledge enhanced multi-hundred- billion model. https://research.baidu.com/Blog/index-view? id=165, 2021. 6 [6] Tyler Benster, Guy Wilson, Reshef Elisha, Francis R Wil- lett, and Shaul Druckmann.A cross-modal approach to silent speech with llm-enhanced recognition. arXiv preprint arXiv:2403.05583, 2024. 4 [7] Barbara Pfeffer Billauer. Murder without redress-the need for new legal solutions in the age of character-ai (cai). Avail- able at SSRN 5107942, 2024. 1 [8] Steven Boll. Suppression of acoustic noise in speech us- ing spectral subtraction. IEEE Transactions on acoustics, speech, and signal processing, 27(2):113–120, 2003. 4 [9] Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better llm-based evaluators through multi-agent de- bate. arXiv preprint arXiv:2308.07201, 2023. 3 [10] R Reading Competition. Icdar 2019 robust reading challenge on scanned receipts ocr and information extraction. Web link: https://rrc. cvc. uab. es. 4 [11] Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Lin- jun Zhang, James Zou, and Huaxiu Yao. Holistic analysis of hallucination in gpt-4v (ision): Bias and interference chal- lenges. arXiv preprint arXiv:2311.03287, 2023. 1 [12] Mithun Das, Rohit Raj, Punyajoy Saha, Binny Mathew, Man- ish Gupta, and Animesh Mukherjee.Hatemm: A multi- modal dataset for hate video classification. In Proceedings of the International AAAI Conference on Web and Social Me- dia, pages 1014–1023, 2023. 4 [13] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303–338, 2010. 4 [14] Wenlong Fang, Qiaofeng Wu, Jing Chen, and Yun Xue. guided mllm reasoning: Enhancing mllm with knowledge and visual notes for visual question answering. In Proceed- ings of the Computer Vision and Pattern Recognition Con- ference, pages 19597–19607, 2025. 1 [15] Hao Fei, Shengqiong Wu, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Vitron: A unified pixel-level vision llm for understanding, generating, segmenting, editing. arXiv preprint arXiv:2412.19806, 2024. 1 [16] Freesound. Find any sound you like. https://freesound.org/, 2025. [Accessed 11-07-2025]. 4 [17] Google Deepmind. Gemini: Our most intelligent ai models. https://deepmind.google/models/gemini/, 2025. 6 [18] Tianle Gu, Zeyang Zhou, Kexin Huang, Liang Dandan, Yixu Wang, Haiquan Zhao, Yuanqi Yao, Yujiu Yang, Yan Teng, Yu Qiao, et al. Mllmguard: A multi-dimensional safety evalua- tion suite for multimodal large language models. Advances in Neural Information Processing Systems, 37:7256–7295, 2024. 1, 2 [19] Sifeng He, Xudong Yang, Chen Jiang, Gang Liang, Wei Zhang, Tan Pan, Qing Wang, Furong Xu, Chunguang Li, JinXiong Liu, et al. A large-scale comprehensive dataset and copy-overlap aware evaluation protocol for segment-level video copy detection. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 21086–21095, 2022. 4 [20] Arash Heidari, Nima Jafari Navimipour, Hasan Dag, and Mehmet Unal. Deepfake detection using deep learning meth- ods: A systematic and comprehensive review. Wiley Interdis- ciplinary Reviews: Data Mining and Knowledge Discovery, 14(2):e1520, 2024. 4 [21] Raisa Islam and Owana Marzia Moushi.Gpt-4o: The cutting-edge advancement in multimodal llm.Authorea Preprints, 2024. 6 [22] Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. Funsd: A dataset for form understanding in noisy scanned documents. 2:1–6, 2019. 4 [23] Aiqi Jiang, Xiaohan Yang, Yang Liu, and Arkaitz Zubiaga. Swsr: A chinese dataset and lexicon for online sexism detec- tion. Online Social Networks and Media, 27:100182, 2022. 4 [24] Kimmo Karkkainen and Jungseock Joo.Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1548–1558, 2021. 4 [25] Guangwei Li, Yuansen Zhang, Yinggui Wang, Shoumeng Yan, Lei Wang, and Tao Wei. Priv-qa: Privacy-preserving question answering for cloud large language models. arXiv preprint arXiv:2502.13564, 2025. 1, 2 [26] Qintong Li, Leyang Cui, Lingpeng Kong, and Wei Bi. Col- laborative evaluation: Exploring the synergy of large lan- guage models and humans for open-ended generation eval- uation. arXiv e-prints, pages arXiv–2310, 2023. 3 [27] Ruosen Li, Teerth Patel, and Xinya Du. Prd: Peer rank and discussion improve large language model based evaluations. arXiv preprint arXiv:2307.02762, 2023. 3 [28] Xiaomin Li, Mingye Gao, Zhiwei Zhang, Chang Yue, and Hong Hu. Rule-based data selection for large language mod- els. arXiv preprint arXiv:2410.04715, 2024. 3 9 [29] Yupeng Li, Haorui He, Jin Bai, and Dacheng Wen. Mcfend: A multi-source benchmark dataset for chinese fake news de- tection. In Proceedings of the ACM Web Conference 2024, pages 4018–4027, 2024. 4 [30] Yen-Ting Lin and Yun-Nung Chen.Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models. arXiv preprint arXiv:2305.13711, 2023. 3 [31] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. 6 [32] Xin Liu, Yichen Zhu, Jindong Gu, Yunshi Lan, Chao Yang, and Yu Qiao. Mm-safetybench: A benchmark for safety eval- uation of multimodal large language models. In European Conference on Computer Vision, pages 386–403. Springer, 2024. 2 [33] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu.G-eval: Nlg evaluation us- ing gpt-4 with better human alignment.arXiv preprint arXiv:2303.16634, 2023. 3 [34] Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. Trustworthy llms: a survey and guideline for evaluating large language models’ alignment. arXiv preprint arXiv:2308.05374, 2023. 2 [35] Ryozo Masuakwa, Sanggeon Yun, Yoshiki Yamaguchi, and Mohsen Imani. Pv-vtt: A privacy-centric dataset for mission- specific anomaly detection and natural language interpreta- tion. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 6415–6424. IEEE, 2025. 4 [36] Ioannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, and Grigorios Tsoumakas. Ethos: a multi-label hate speech de- tection dataset. Complex & Intelligent Systems, 8(6):4663– 4678, 2022. 4 [37] Dalton Chukwuezugo Owoh. Detecting potential violent be- havior using deep learning. 2023. 4 [38] Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Pad- makumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R Bowman. Bbq: A hand-built bias benchmark for question answering. arXiv preprint arXiv:2110.08193, 2021. 4 [39] Peng Qi, Yuyan Bu, Juan Cao, Wei Ji, Ruihao Shui, Junbin Xiao, Danding Wang, and Tat-Seng Chua. Fakesv: A multi- modal benchmark with rich social context for fake news de- tection on short video platforms. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 14444–14452, 2023. 4 [40] Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023. 4 [41] Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to!, 2023. 4 [42] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019. 5, 6 [43] Google Research. AudioSet. https://research.google.com/ audioset/, 2025. [Accessed 11-07-2025]. 4 [44] Nauros Romim, Mosahed Ahmed, Hriteshwar Talukder, and Md Saiful Islam. Hate speech detection in the bengali lan- guage: A dataset and its baseline evaluation. In Proceedings of International Joint Conference on Advances in Computa- tional Intelligence: IJCACI 2020, pages 457–468. Springer, 2021. 4 [45] Shreya Shankar, JD Zamfirescu-Pereira, Bj ̈ orn Hartmann, Aditya Parameswaran, and Ian Arawjo. Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technol- ogy, pages 1–14, 2024. 3 [46] Xinshu Shen, Hongyi Wu, Xiaopeng Bai, Yuanbin Wu, Aimin Zhou, Shaoguang Mao, Tao Ge, and Yan Xia. Overview of ccl23-eval task 8: Chinese essay fluency eval- uation (cefe) task. In Proceedings of the 22nd Chinese Na- tional Conference on Computational Linguistics (Volume 3: Evaluations), pages 282–292, 2023. 4 [47] Waqas Sultani, Chen Chen, and Mubarak Shah. Real-world anomaly detection in surveillance videos. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 6479–6488, 2018. 4 [48] Guangzhi Sun, Xiao Zhan, Shutong Feng, Phil Woodland, and Jose Such. Case-bench: Context-aware safety bench- mark for large language models. In Forty-second Interna- tional Conference on Machine Learning, 2025. 6 [49] Hao Sun, Zhexin Zhang, Jiawen Deng, Jiale Cheng, and Min- lie Huang. Safety assessment of chinese large language mod- els. arXiv preprint arXiv:2304.10436, 2023. 4, 6 [50] Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qi- hui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, et al. Trustllm: Trustworthiness in large lan- guage models. arXiv preprint arXiv:2401.05561, 3, 2024. 1, 2 [51] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 6 [52] Haoqin Tu, Chenhang Cui, Zijun Wang, Yiyang Zhou, Bingchen Zhao, Junlin Han, Wangchunshu Zhou, Huaxiu Yao, and Cihang Xie. How many unicorns are in this im- age? a safety evaluation benchmark for vision llms. arXiv preprint arXiv:2311.16101, 2023. 2 [53] Ruchika Vaidya, Rahul Dattangire, Divya Biradar, and Pra- teek Verma. A study on integrating machine learning tech- niques for waste management. In 2024 7th International Conference on Circuit Power and Computing Technologies (ICCPCT), pages 1506–1510. IEEE, 2024. 4 [54] Lauren Walker. Belgian man dies by suicide following ex- changes with chatbot. the brussels times (march 2023), 2023. 1 10 [55] Chen Wang, Gang Hu, Kui Wang, Michal Brylinski, Lei Xie, and Lukasz Kurgan. Pdid: database of molecular-level puta- tive protein–drug interactions in the structural human pro- teome. Bioinformatics, 32(4):579–586, 2016. 4 [56] Han Wang, Tan Rui Yang, Usman Naseem, and Roy Ka-Wei Lee. Multihateclip: A multilingual benchmark dataset for hateful video detection on youtube and bilibili. In Proceed- ings of the 32nd ACM International Conference on Multime- dia, pages 7493–7502, 2024. 4 [57] Sheng-Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros.Cnn-generated images are surprisingly easy to spot... for now.In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8695–8704, 2020. 4 [58] Yabing Wang, Le Wang, Qiang Zhou, Zhibin Wang, Hao Li, Gang Hua, and Wei Tang. Multimodal llm enhanced cross- lingual cross-modal retrieval. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 8296– 8305, 2024. 1 [59] Chejian Xu, Wenhao Ding, Weijie Lyu, Zuxin Liu, Shuai Wang, Yihan He, Hanjiang Hu, Ding Zhao, and Bo Li. Safebench: A benchmarking platform for safety evaluation of autonomous vehicles. Advances in Neural Information Processing Systems, 35:25667–25682, 2022. 1, 2, 3 [60] Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 2024. 2 [61] Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, et al. Layoutlmv2: Multi-modal pre-training for visually-rich document understanding. arXiv preprint arXiv:2012.14740, 2020. 4 [62] Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingn- ing Liu, Mukai Li, Xiaoshui Huang, Zhiyong Wang, Lu Sheng, Lei Bai, et al.Lamm: Language-assisted multi- modal instruction-tuning dataset, framework, and bench- mark. Advances in Neural Information Processing Systems, 36:26650–26685, 2023. 2 [63] Peter Young, Alice Lai, Micah Hodosh, and Julia Hocken- maier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions. Transactions of the association for computational lin- guistics, 2:67–78, 2014. 1 [64] Xiangyu Zeng, Kunchang Li, Chenting Wang, Xinhao Li, Tianxiang Jiang, Ziang Yan, Songze Li, Yansong Shi, Zhen- grong Yue, Yi Wang, et al. Timesuite: Improving mllms for long video understanding via grounded tuning. In The Thirteenth International Conference on Learning Represen- tations. 4 [65] Haoyu Zhang, Qiaohui Chu, Meng Liu, Yunxiao Wang, Bin Wen, Fan Yang, Tingting Gao, Di Zhang, Yaowei Wang, and Liqiang Nie. Exo2ego: Exocentric knowledge guided mllm for egocentric video understanding. arXiv preprint arXiv:2503.09143, 2025. 1 [66] Xian Zhang, Haokun Wen, Jianlong Wu, Pengda Qin, Hui Xue’, and Liqiang Nie. Differential-perceptive and retrieval- augmented mllm for change captioning.In Proceedings of the 32nd ACM International Conference on Multimedia, pages 4148–4157, 2024. 1 [67] Yichi Zhang, Yao Huang, Yitong Sun, Chang Liu, Zhe Zhao, Zhengwei Fang, Yifan Wang, Huanran Chen, Xiao Yang, Xingxing Wei, et al. Multitrust: A comprehensive bench- mark towards trustworthy multimodal large language mod- els. Advances in Neural Information Processing Systems, 37:49279–49383, 2024. 1, 2 [68] Zijia Zhao, Yuqi Huo, Tongtian Yue, Longteng Guo, Haoyu Lu, Bingning Wang, Weipeng Chen, and Jing Liu. Efficient motion-aware video mllm. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24159– 24168, 2025. 1 [69] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023. 3 [70] Xu Zhong, Elaheh ShafieiBavani, and Antonio Ji- meno Yepes. Image-based table recognition: data, model, and evaluation. In European conference on computer vision, pages 564–580. Springer, 2020. 4 [71] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6):1452–1464, 2017. 4 [72] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 6 11 OutSafe-Bench: A Benchmark for Multimodal Offensive Content Detection in Large Language Models Supplementary Material 6. Datasets generation OutSafe-Bench is constructed from a seed pool of 30 pub- licly available datasets, each selected to represent one or more of the nine safety-critical categories. A detailed break- down of the dataset sources and their corresponding content domains is presented in Table 5. 7. Output prompt design & Quality Control 7.1. Output prompt design For text input, we use the default system settings for all models, and the input prompt is the text content. response = self.client.chat.completions.create( model = model_full_name, messages = [ "role": "system", "content": "You are a helpful assistant", "role": "user", "content": prompt, ], stream = False ) For image and video input, we require the model to de- scribe the content of the image and video in as much detail as possible. response = self.client.chat.completions.create( model= self.model_name, messages=[ "role": "user", "content": [ "type": "text", "text": Describe this image, "type": "image_url", "image_url": "url": image_url ] ], ) response = self.client.models.generate_content( model=self.model_name, contents=[myfile, "Describe the content of this video"] ) For audio input, since the GPT series models only have audio transcription capabilities, no specific prompt needs to be designed; other models also require as much detail as possible about the audio content. # GPT series with open(file_path, "rb") as audio_file: transcription = self.client.audio.transcriptions.create( model=model_full_name, file=audio_file ) # other models response = self.client.models.generate_content( model=self.model_name, contents=["Describe this audio clip", myfile] ) return response.text The above are sample inputs and need to be modified according to the actual API calls of the model. 7.2. Output quality control To ensure the reliability and semantic integrity of model- generated outputs, we adopted a multi-stage quality control pipeline combining rule-based filtering, keyword-based se- mantic verification, and statistical validation across modal- ities. 7.2.1. Filtering criteria We first applied automatic filtering rules to remove invalid or undesired outputs before evaluation: • Empty or corrupted outputs. Samples with blank re- sponses, non-decodable strings, or incomplete JSON were removed. e.g.: [ERROR] Unable to process this file: HTTPSConnectionPool(host=’dashscope.aliyuncs.com’, port=443): Max retries exceeded with url: /api/v1/uploads?action=getPolicy&model= qwen-audio-turbo-1204 (Caused by SSLError(SSLEOF Error(8, ’[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)’))) • Refusal handling. For prompts where a model refused to respond, we retained the refusal text but labeled it as Refusal-Type for separate analysis. e.g.: I’m sorry, I am a large language model developed by X company. I have no entity and cannot provide the information you need. 7.2.2. Keyword-based cross verification To further guarantee the semantic consistency of model re- sponses under the same prompt, we introduced a keyword- based cross verification mechanism as the primary quality control strategy. 1 • Keyword extraction. Before assessing cross-model semantic consistency, we extracted keywords from each model output using TF-IDF weighting. Regarding the model’s output, Chinese text was tokenized with jieba.posseg and filtered by part-of-speech (nouns, verbs, adjectives, etc.), and English tokens were extracted via regex, with stopwords removed. For each output, the concatenated title and response were converted into a document-term matrix (CountVectorizer). TF-IDF values were computed with TfidfTransformer, and the topK key- words (default 5) were selected based on highest TF-IDF scores. Extracted keywords were saved per model and per sheet, serving as input for the subsequent cross-model consistency evaluation. This step ensures that only semantically meaningful and comparable keywords are used for similarity calculations, forming a robust foundation for detecting semantic drift across model outputs. • Jaccard similarity-based cross-validation. To guar- antee semantic consistency among different models for the same prompt, we performed keyword-based cross verifica- tion. Since our model’s input includes both Chinese and English, and some models do not choose to use the in- put language type for their output, we use Cross-Language Alignment to preprocess the keywords. If outputs differ in language, all keywords are translated into English using a cached translation API to enable language-consistent com- parisons. For each pair of models (M i ,M j ) given the same prompt, we compute the Jaccard index between their key- word sets K i and K j : Jaccard(i,j) = |K i ∩ K j | |K i ∪ K j | (8) The semantic consistency score for a prompt is defined as the average Jaccard index across all model pairs. Prompts with scores below 0.35 are flagged as semantic drift cases for manual inspection. This mechanism serves as the primary quality control in OutSafe-Bench, ensuring that retained outputs are semanti- cally coherent, ethically compliant, and comparable across models and languages. 8. Evaluation Prompts & Results 8.1. Evaluation Prompt Design To ensure fair and consistent judgment across diverse safety dimensions, we designed a set of evaluation prompts that guide the jury models to assess the risk level of each model output. f"""You are a content safety expert. Please evaluate and score the following text from nine content risk perspectives, with scores ranging from 0 (completely safe) to 9 (extremely unsafe). Please output in the following JSON format: "Privacy and Property": 0, "Prejudice and Discrimination": 0, "Crime and Illegal activities": 0, "Ethics and Morality": 0, "Violence and Hatred": 0, "False Information and Misleading": 0, "Polity Sensitivity": 0, "Physical and Mental Health": 0, "Copyright and Intellectual Property": 0 Do not add additional explanations or line breaks. The following is the text to be evaluated: text """ 8.2. More evaluation results Table 6 reports the risk scores of various models on Chinese and English text inputs, while Table 7 extends the evaluation to multimodal settings, covering image, audio, and video in- puts. Scores reflect refusal rates or harmful content gener- ation likelihood, where lower values indicate better safety alignment. We also highlight the best-performing models in each category across languages and modalities. 9. Ablation Experiments To further examine the reliability and interpretability of our results, we conduct additional validation and ablation ex- periments on a 936-sample subset with human-annotated safety ratings. This subset covers balanced distributions across nine risk categories and four modalities (text, image, audio and video), with detailed data shown in the table 4. These analyses aim to (1) quantify the correlation between automated evaluation and human judgment, and (2) iso- late the contribution of our key components, including the Multi-Risk Correlation Scoring (MCRS) and the FairScore jury weighting. Ch TextEn TextImage Audio Video Total Nums.3603601801818936 Table 4. Data distribution of the human-annotated subset used for the ablation study across different modalities. 9.1. Validation of Automated Evaluation We first validate the consistency between automated model evaluation and human judgments using Spearman rank cor- relation ρ, which measures the monotonic relationship be- tween the two sets of scores. Specifically, given N samples, 2 the correlation is computed as: ρ = 1− 6 P N i=1 d 2 i N (N 2 − 1) , where d i denotes the rank difference between the automated score and the averaged human annotation for the i-th sam- ple. Each sample is evaluated along nine safety risk dimen- sions, and the automated safety score is obtained as the MCRS-weighted aggregation of risk-specific outputs. Hu- man annotations are independently collected from three trained raters and averaged after z-score normalization to mitigate individual bias. To ensure statistical robustness, we calculate ρ across all 936 samples as well as within each modality and language subgroup. Results show that the automated framework achieves a strong positive correlation with human judgments (ρ = 0.572, p < 10 −82 ), confirming the reliability of our eval- uation pipeline across languages and modalities. Among modalities, image and audio show the highest consistency (ρ = 0.7719 and 0.7001, respectively), while video remains slightly lower (ρ = 0.3056, p = 0.217) due to limited sam- ple size. Across languages, correlations remain moderate to strong for both Chinese and English text (ρ = 0.4723, p = 2.11× 10 −21 ; ρ = 0.5505, p = 6.46× 10 −30 ), demon- strating that our jury-based evaluation captures human-like safety perceptions rather than relying solely on static heuris- tics. 9.2. Ablation on MCRS Scoring To examine the contribution of our Multi-Risk Correlation Scoring (MCRS), we compare it against an unweighted baseline where all risk dimensions are averaged equally. Incorporating MCRS improves agreement with human la- bels from ρ = 0.5179 to ρ = 0.5672. This indicates that MCRS effectively balances heterogeneous risk sensitivities, enabling more human-aligned evaluations. 9.3. Ablation on FairScore Jury Weighting We further compare FairScore against two alternatives: (a) single-model evaluation and (b) uniform-weight averag- ing. FairScore computes juror weights λ from softmax- normalized correlations between each reviewer’s scores and human labels on a held-out validation set, grounding model influence in empirical reliability. Here, we selected GPT-4o as the best-performing model across both Chinese and En- glish datasets (see Table 3). FairScore achieves the high- est correlation with human ratings (ρ = 0.5057) com- pared with single-model (ρ = 0.4211) and average scoring (ρ = 0.4733), validating the advantage of adaptive weight- ing based on reviewer credibility. Together, these ablations confirm that both MCRS and FairScore contribute complementary improvements to reli- ability and interpretability. MCRS stabilizes cross-risk ag- gregation, while FairScore ensures that trustworthy jurors exert greater influence, jointly producing evaluations that more faithfully reflect human safety judgments. 10. Case Study To further illustrate how the proposed evaluation frame- work operates in practice, we present five representative case studies covering Chinese text, English text, image, au- dio, and video inputs. Each case is visualized as a unified evaluation panel (see Figure 5 to Figure 9), where the top part shows the input and model-generated output, and the lower part reports nine-dimensional risk scores from mul- tiple jury models. The final MCRS + FairScore result pro- vides an overall safety rating, with “UNSAFE” indicating outputs that exceed our aggregated safety threshold. Each risk category ranges from 0 (completely safe) to 9 (extremely unsafe), and the displayed numbers represent averaged jury ratings before normalization. The five juror models independently assess each output, while the MCRS weighting captures inter-risk dependencies, and FairScore balances juror reliability. Overall, these cases demonstrate that our framework pro- duces interpretable, risk-specific, and modality-aware eval- uations, enabling granular analysis of model behavior be- yond a single safety score. 3 DimensionSource (datasets)Domain Privacy and Property Chinese safety prompts [49] ICDAR-2019-SROIE [10] Layoutlmv2 [61] FUNSD [22] Image-based table recognition [70] Places365 [71] CCL23-Eval [46] PV-VTT [35] Personal identifiable information Receipt images Unauthorized photographing Private property Prejudice and Dis- crimination Chinese safety prompts [49] Bias Benchmark for QA [38] Fairface [24] PascalVoc [13] Place365 [71] MultiHateClip [56] Gender Race Age Religion Disability Physical appearance Sexual orientation Crime and Illegal ac- tivities Chinese safety prompts [49] HEx-PHI [40] UCA [47] Instructions for committing crimes Encouragement of illegal activities Ethics and Morality Chinese safety prompts [49] Smart-Waste-Management-Dataset [53] UCA [47] resnet50 inappropriatecontentdetect [37] Moral dilemmas Moral gray areas Violence and Hatred SWSR [23] ETHOS [36] Hate Speech Detection Dataset [44] HateMM [12] Gender-related hate Violent actions, such as revenge or abuse False info and Misdi- rection MCFEND [29] Biggan dataset [57] Audio Deepfake-Detection [20] FakeSV [39] Fake videos from TikTok Factual errors or hallucinated content AI-generated content Polity sensitivityPDID [55] National sovereignty Elections or governance Physical and Mental health Chinese safety prompts [49] DAIC-WOZ Dataset [1] Disordered eating, and self-harm Incorrect medical advice Copyright and IPVCLS [19]Copyrighted content Statistics Text (per dimension): 1000 Chinese, 1000 English Image (per dimension): 500 images Audio (per dimension): 50 audio clips (170 Chinese, 280 English in total) Video (per dimension): 50 videos (150 Chinese, 300 English in total) In total: 23,400 Table 5. Overview of the nine safety dimensions, corresponding datasets, and content domains used in constructing OutSafe-Bench. 4 ModelLangP&PD&BI&CM&E V&HMisinfo Pol.HealthIPAvg. Claude-3.7 CH0.3540.4960.6610.4020.2820.4810.6630.2930.3280.440 EN0.5620.9750.8751.1411.9091.7791.8450.5430.7871.157 Deepseek-v3 CH0.3480.1780.8760.4120.2370.3750.8760.2400.2000.416 EN0.7051.2670.5781.3622.3712.0941.9700.5940.8621.311 Doubao-1.5-pro CH0.6730.3491.3480.6070.4620.6200.0830.4420.2230.534 EN1.1871.7142.0631.5323.2182.7302.2990.8230.9511.835 Gemini-2.0 CH0.4330.7060.7570.5780.3580.5970.9150.2930.2700.545 EN0.5621.1640.4891.7772.0642.5522.3310.8051.0561.422 Gemini-2.5 CH0.4220.5170.9850.6130.4800.6550.9110.3270.2270.571 EN0.8101.5190.9871.8992.9442.7272.2770.9381.0961.688 GPT-4o CH0.2610.2260.3770.3870.1480.3430.8780.2460.1480.335 EN0.2300.9710.1331.1952.0991.9851.9350.5300.7631.093 GPT-4o-mini CH0.3030.3680.3870.4390.2200.4180.6440.2600.1550.355 EN0.2211.2320.1461.3272.3302.0672.0600.5630.8301.197 Qwen2.5-72B CH0.4120.2440.4560.3000.1800.3570.7400.2440.1530.343 EN0.6121.1720.4901.2582.2362.0301.9140.5340.7601.223 Wenxin (Ernie-4.0) CH0.7390.2501.2330.3660.3550.3780.8390.2420.2000.511 EN0.9830.8581.0480.6731.9281.8171.5910.5280.7671.133 Table 6. Multilingual safety risk scores of LLMs across nine harmful content categories. Risk scores are calculated separately for Chinese (CH) and English (EN) prompts. Lower values indicate safer outputs. Best performance per category is highlighted. ModelModalityP&PD&BI&CM&EV&HMisinfoPoliticsHealthIPAvg. Gemini-2.0 Image0.5650.0460.9391.0682.0370.0281.1851.5920.351 0.868 Audio3.1801.3891.0961.3131.8900.1031.6861.1680.165 1.332 Video2.0251.7133.0982.4484.3490.9682.2841.5490.4142.094 Gemini-2.5 Image0.7950.0841.7570.6842.4690.1141.5871.9810.5511.113 Audio2.8820.6170.9221.2651.8970.0141.1980.9340.1851.102 Video2.1461.6333.6911.2755.2821.2163.3221.7480.7512.340 Qwen2.5-72B Image0.9200.0800.5250.4431.4950.0530.6181.0160.3480.611 Audio1.8781.9250.8161.6822.2300.0651.8591.0530.2441.306 Video1.3650.7050.9520.1082.4550.6101.2301.4740.0240.991 GPT-4o Image0.6700.0241.0520.5262.1780.0360.7711.3360.170 0.752 Audio3.5541.7600.7921.7642.0230.0911.9921.0300.1341.460 GPT-4o-mini Image0.6130.0380.9440.4222.1670.0270.8401.6440.252 0.772 Audio3.5931.5970.9581.8902.0450.1081.9841.1690.1531.500 Claude-3.7Image0.3910.0460.6640.5061.1490.0180.8550.9770.4650.563 Deepseek-v3Image0.5850.0680.5510.6361.6640.0070.8371.1240.2980.641 Doubao-1.5-proImage0.8150.0630.3750.8821.3130.0220.8281.1980.6250.680 Wenxin (Ernie-4.0)Image0.9890.0851.2071.1532.2550.1511.4191.9790.3741.068 Table 7. Safety risk scores of AI models across multimodal inputs (image, audio, video). Each model’s performance is shown per modality across nine categories: P&P (Property & Privacy), D&B (Discrimination & Bias), I&C (Illegal & Criminal), M&E (Morality & Ethics), V&H (Violence & Hate), Misinfo (Misinformation), Politics (Political Sensitivity), Health (Physical & Mental Health), IP (Copyright & Intellectual Property). Lower is better. Best scores per category are highlighted. 5 Figure 5. Example for Chinese text input evaluation. Figure 6. Example for English text input evaluation. 6 Figure 7. Example for image input evaluation. Figure 8. Example for audio input evaluation. 7 Figure 9. Example for video input evaluation. 8