Paper deep dive

OODBench: Out-of-Distribution Benchmark for Large Vision-Language Models

Ling Lin, Yang Bai, Heng Su, Congcong Zhu, Yaoxing Wang, Yang Zhou, Huazhu Fu, Jingrun Chen

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 182

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 7/20/2026, 10:44:29 PM

Summary

The paper introduces OODBench, a benchmark designed to evaluate Large Vision-Language Models (VLMs) on out-of-distribution (OOD) data, specifically focusing on covariate shifts where labels remain common but data distribution changes. The authors propose an automated pipeline using generalized OOD detectors (CLIP, BLIP2) to identify OOD-Hard and OOD-Simple instances from datasets like COCO, LVIS, nuScenes, and Cityscapes. They also introduce the Basic-to-Advanced Progression (BAP) metric to assess VLM performance across existential, counting, and logical reasoning tasks, revealing significant performance degradation in current SOTA VLMs when handling OOD data.

Entities (10)

OODBench · benchmark · 98%COCO · dataset · 95%nuScenes · dataset · 95%VLMs · model-class · 95%Basic-to-Advanced Progression Metric · metric · 93%OOD Data · data-concept · 92%BLIP2 · model · 90%CLIP · model · 90%Covariate Shift · concept · 90%GPT-4o · model · 88%

Relation Signals (8)

OODBench → evaluates → VLMs

confidence 95% · OODBench... for constructing new benchmarks and evaluating the ability of VLMs to process OOD data.

Basic-to-Advanced Progression Metric → assesses → VLMs

confidence 93% · we propose the ’Basic-Advanced Process Metric’, which evaluates their ability in image understanding, quantity perception, and logical reasoning

OODBench → containsdatafrom → nuScenes

confidence 92% · autonomous driving (nuScenes (Caesar et al., 2020)...)

OODBench → containsdatafrom → COCO

confidence 92% · natural scenarios (COCO (Lin et al., 2014)...)

OODBench → focuseson → Covariate Shift

confidence 90% · we focus on collecting covariate shift OOD data to construct OOD benchmarks

OODBench → uses → BLIP2

confidence 90% · we have chosen off-the-shelf VLMs such as CLIP... and BLIP2... as generalized OOD detectors

OODBench → uses → CLIP

confidence 90% · we have chosen off-the-shelf VLMs such as CLIP... as generalized OOD detectors

GPT-4o → testedon → OODBench

confidence 88% · Even the GPT-4o achieves only approximately 65% accuracy on OOD-H data

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Existing Visual-Language Models (VLMs) have achieved significant progress by being trained on massive-scale datasets, typically under the assumption that data are independent and identically distributed (IID). However, in real-world scenarios, it is often impractical to expect that all data processed by an AI system satisfy this assumption. Furthermore, failure to appropriately handle out-of-distribution (OOD) objects may introduce safety risks in real-world applications (e.g., autonomous driving or medical assistance). Unfortunately, current research has not yet provided valid benchmarks that can comprehensively assess the performance of VLMs in response to OOD data. Therefore, we propose OODBench, a predominantly automated method with minimal human verification, for constructing new benchmarks and evaluating the ability of VLMs to process OOD data. OODBench contains 40K instance-level OOD instance-category pairs, and we show that current VLMs still exhibit notable performance degradation on OODBench, even when the underlying image categories are common. In addition, we propose a reliable automated assessment metric that employs a Basic-to-Advanced Progression of prompted questions to assess the impact of OOD data on questions of varying difficulty more fully. Lastly, we summarize substantial findings and insights to facilitate future research in the acquisition and evaluation of OOD data.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

180,849 characters extracted from source content.

Expand or collapse full text

OODBench: Out-of-Distribution Benchmark for Large Vision-Language Models Ling Lin Yang Bai Heng Su Congcong Zhu Yaoxing Wang Yang Zhou Huazhu Fu Jingrun Chen Abstract Existing Visual-Language Models (VLMs) have achieved significant progress by being trained on massive-scale datasets, typically under the assumption that data are independent and identically distributed (IID). However, in real-world scenarios, it is often impractical to expect that all data processed by an AI system satisfy this assumption. Furthermore, failure to appropriately handle out-of-distribution (OOD) objects may introduce safety risks in real-world applications (e.g., autonomous driving or medical assistance). Unfortunately, current research has not yet provided valid benchmarks that can comprehensively assess the performance of VLMs in response to OOD data. Therefore, we propose OODBench, a predominantly automated method with minimal human verification, for constructing new benchmarks and evaluating the ability of VLMs to process OOD data. OODBench contains 40K instance-level OOD instance–category pairs, and we show that current VLMs still exhibit notable performance degradation on OODBench, even when the underlying image categories are common. In addition, we propose a reliable automated assessment metric that employs a Basic-to-Advanced Progression of prompted questions to assess the impact of OOD data on questions of varying difficulty more fully. Lastly, we summarize substantial findings and insights to facilitate future research in the acquisition and evaluation of OOD data 111The dataset is open to the public for research. https://anonymous.4open.science/r/ood-1B0E. 1 Introduction Recently, Visual-Language Models, VLMs (e.g., GPT-4o (Achiam et al., 2023; Hurst et al., 2024), Gemini (Team et al., 2024), LLaVA family (Li et al., 2024b), InternVL family (Chen et al., 2024c, a), Qwen2-VL family (Wang et al., 2024a), Llama-3.2-Vision (Dubey et al., 2024), DeepSeek-VL (Lu et al., 2024), etc.) have demonstrated strong performance across multiple domains. Trained on large-scale multimodal data, these models show strong potential across a range of application scenarios. These results have led to a common perception that large-scale VLMs exhibit strong generalization capabilities. However, comparatively little attention has been paid to the performance of these VLMs under out-of-distribution (OOD) conditions. The neglect of this issue is partly attributed to the lack of OOD evaluation benchmarks. Therefore, we construct a benchmark to evaluate the performance of VLMs on OOD data, thereby advancing the progress of VLMs in OOD research and taking a critical step towards realizing intelligent systems that can operate safely in the real world. Figure 1: Comparison of differences in ID data, covariate shift OOD data, and semantic shift data. In this paper, we consider OOD data as samples that do not belong to the training data distribution. As shown in Fig. 1, OOD data can be classified into two categories according to the different forms of distributional changes: a data distribution shift where labels-alter, known as semantic shift, and a shift where labels remain the same but the data distribution changes, also known as covariate shift. The OOD data are essentially associated with the model, as it relies on the models’ assumptions and learned distributions made during the training process. However, most of the mainstream VLMs are closed-source or only open to model parameters, which prevents us from directly accessing the training distributions of these models. Therefore, how to collect OOD data outside of these model training data distributions presents a fundamental challenge. In the traditional OOD literature Yang et al. (2024a); Hendrycks et al. (2022); Wang et al. (2022), these studies usually train models on datasets with known categories and obtain OOD test benchmarks by constructing new category data based on the categories in the training data, thus OOD data are different from the original categories, i.e., semantic shift OOD data. However, considering the training category agnostic of existing VLMs, this experimental setting is obviously no longer applicable. Moreover, the ill-pursuit to collecting rarely unseen data ensures neither the rigor of semantic shift OOD data nor real-world significance. For this reason, our work focuses on collecting covariate shift OOD data to construct OOD benchmarks. The approach is based on the assumption that existing VLMs are likely to have been trained on the most common classes of data. Recent OOD research Averly and Chao (2023); Yang et al. (2024b); Jaeger et al. (2023); Guérin et al. (2023); Zhu et al. (2024) has shifted the focus from new category detection to failure detection scenarios, in which the OOD detector is expected to recognize misclassified samples. In this approach, candidate samples that fail to be classified are labeled as OOD data, and samples that are correctly classified are labeled as ID data to enhance the safety and reliability of deep learning models in real-world applications. Building upon this line of work, we further define two categories of OOD data for VLMs from a human perception perspective: 1) Objects in images that are neither main objects nor semantically related to the main semantic object; 2) Variants or anomalous forms of target objects. The underlying intuition is that VLM training heavily relies on the main semantic objects in images, paying limited attention to non-main semantic objects and their variants/anomalies. This constitutes the OOD in VLM semantic learning. Based on this definition, we first divide the data using an OOD detector, followed by manual spot-checking to ensure the final classification aligns with the proposed OOD definition. Notably, this model-based classification depends on the specific OOD detector employed. VLMs themselves should serve as OOD detectors for classifying OOD data. However, testing each VLM on distinct datasets to identify its specific OOD data is computationally and temporally intensive, making it impractical. Therefore, we propose an automated OOD data classification method supplemented by human verification, which maximizes quality while minimizing manual effort. In the design, we adopt a generalized OOD detector instead of a model-specific VLM detector to reduce computational and time costs. Meanwhile, to alleviate the deviation of OOD data division caused by generalized detectors, we introduce a cross-validation scheme with multiple generalized OOD detectors. For implementation, we have chosen off-the-shelf VLMs such as CLIP (Radford et al., 2021) and BLIP2 (Li et al., 2023b) as generalized OOD detectors and defined the intersection of OOD data detected by both as OOD-Hard data; the data obtained from symmetric difference are defined as OOD-Simple data, denoted as OOD-H and OOD-S, respectively. It is worth noting that CLIP and BLIP2 are essentially replaceable in our OOD division pipeline: any sufficiently robust classifier can serve as a generalized OOD detector and yield comparable results. In the Appendix, we discuss the implications of using different classifiers as generalized OOD detectors in detail. Based on our assumptions, the VLMs have been trained in most of the common data categories. In other words, the OOD data and training data share the same label space, and thus, the divided OOD data can be summarized as covariate shift OOD. We collected the initial benchmark222In Appendix A, we demonstrate the data distribution of the initial benchmark and the detailed performance of different models tested on this benchmark. from a publicly available dataset, which contains about 77k yes-or-no samples. To further reduce the testing time, we randomly downsampled the initial benchmark to obtain the main benchmark OODBench. Figure 2: Distribution of categories and fields in OODBench. Fig. 2 illustrates the sample distribution of OODBench, which contains about 40k yes-or-no samples, where OOD-S is 22k and OOD-H is 18k. To evaluate VLM performance on OOD data from multiple dimensions, we propose the ’Basic-Advanced Process Metric’, which evaluates their ability in image understanding, quantity perception, and logical reasoning, respectively. Our main contributions can be summarized as the following three: 1) We propose a succinct and efficient OOD data division process that is primarily automated and supplemented with lightweight human validation, thereby minimizing labor cost while reducing computational and time overheads. 2) We propose the OOD benchmark for VLMs, which consists of an OOD dataset, an OOD data division pipeline for VLMs, a ‘Basic-to-Advanced Procession Metric‘, and a complete set of experimental procedures. 3) We evaluate eight SOTA VLMs on OODBench, and demonstrate through a large number of experiments that current VLMs still perform poorly when confronted with OOD data, even though the classes of these OOD data are pretty common in natural scenarios. 2 Related Work Out-of-Distribution. Out-of-distribution (OOD) detection aims to recognize inputs sampled from a distribution different from the model’s training data (Averly and Chao, 2023; Yang et al., 2024b). Traditional OOD research mainly focuses on semantic shift (S-OOD), where novel classes arise, yet covariate shift (C-OOD)—where labels remain the same but inputs deviate—can be equally disruptive (Yang et al., 2024a; Salehi et al., 2021). In safety-critical fields like autonomous driving, even minor shifts may cause severe misclassifications and catastrophic results. While robust models may correctly handle such shifts, weaker models often benefit from rejecting uncertain samples to mitigate high-risk errors (Liu et al., 2021). Reflecting this, OOD research has gradually shifted from purely detecting unseen classes to failure detection, encompassing both semantic and covariate shifts that can cause misclassification (Wiles et al., 2021; Liu et al., 2021). Nevertheless, there remains limited exploration of how modern Vision-Language Models (VLMs) contend with these conditions. Our work addresses this gap by constructing an OOD benchmark for VLMs, emphasizing common categories that remain challenging under real-world variation. Multimodal Large Language Models & Benchmark. Recent Vision-Language Models (VLMs) trained on massive image-text corpora have achieved impressive gains in tasks like visual reasoning, captioning, and Q&A (Li et al., 2023b; Team et al., 2024; Achiam et al., 2023). Notable examples include GPT-4 (Achiam et al., 2023; Hurst et al., 2024), Gemini (Team et al., 2024), LLaVA (Li et al., 2024b), and InternVL (Chen et al., 2024a, c), often leveraging large-scale datasets such as LAION (Schuhmann et al., 2022) and MS COCO (Lin et al., 2014). Although benchmarks like MMBench (Liu et al., 2025), SeedBench (Li et al., 2023a), and GQA (Hudson and Manning, 2019) assess general vision-language performance, they mainly cover in-distribution cases and rarely address out-of-distribution (OOD) data. For example, SeedBench offers 19K multiple-choice questions across images and videos, highlighting model limits but rarely exploring out-of-distribution scenarios. To fill the gap, we introduce OODBench, testing VLM robustness under covariate shift. By incorporating simple and hard OOD categories and employing a “Basic-to-Advanced” protocol, OODBench reveals key weaknesses in current models and encourages future work on safer, more reliable multimodal systems. While OOD data can trigger hallucinations, not all hallucinations arise from distribution shifts. Benchmarks like HallusionBench (Guan et al., 2024) study language and visual hallucinations, whereas OODBench systematically applies covariate shifts to measure genuine OOD resilience. 3 Existing Definitions and Formulations Out-of-distribution (OOD) refers to samples that originate from a data distribution distinct from that of the training set. Given a VLM f trained on data sampled from the distribution Ptr(,∈)P_tr(X,Y ), where X denotes training samples, Y the corresponding labels, and S the label space. This training distribution is referred to as the in-distribution (ID). In contrast, common OOD data can be divided into two types: covariate shifts (i.e., labels remain unchanged, involving only samples from the training class) and semantic shifts (i.e., labels change, involving only samples from the novel category). Semantic Shift. Lipton et al. (2018) In a traditional experimental setup, the semantic shift data consists of novel category samples outside of the model labeling space S. The semantic shift data consists of a novel category of samples. Given semantic labels Ytr∈Y_tr from the training distribution and semantic labels Yte∈Y_te from the testing distribution, where Ytr∩Yte=∅Y_tr∩ Y_te= and satisfy Ptr(y)=0P_tr(y)=0 for all y∈Ytey∈ Y_te. Covariate Shift. Shimodaira (2000); Lipton et al. (2018) The covariate shift data consists of samples from the model label space S but from different domains. In other words, the image distributions of the training and test data are different: Ptr(x)≠Pte(x)P_tr(x)≠ P_te(x), but the label distributions remain the same: Ptr(y|x)=Pte(y|x)P_tr(y|x)=P_te(y|x). 4 The OODBench VLMs have demonstrated outstanding performance across multiple tasks and domains, profoundly reshaping the landscape of AI development. However, their performance when encountering OOD data remains under-evaluated. Existing VLMs typically use agnostic training categories and collect rare categories as semantic shift OOD data, which do not accurately assess model performance on OOD categories encountered in real-world applications, as these rare categories may not represent common OOD types. To remedy this shortcoming, we focus on collecting covariate shift OOD data for constructing OODBench. Specifically, the OOD data categories need to be broadly representative to ensure their real-world relevance. Meanwhile, OOD data in certain scenarios can severely degrade model performance, potentially causing catastrophic outcomes (e.g., autonomous driving). Thus, we collected data for two key scenarios: natural scenarios (COCO (Lin et al., 2014), LVIS (Gupta et al., 2019)) and autonomous driving (nuScenes (Caesar et al., 2020), Cityscapes (Cordts et al., 2016)). Our goal is to build an instance-level OOD benchmark for VLMs. To this end, we define OOD data from the perspective of VLMs’ semantic learning: 1) Objects in images that are neither main objects nor semantically related to the main semantic object; 2) Variants or anomalous forms of target objects. However, manually screening each instance object that meets the above definition is practically infeasible. To address this issue, we designed a concise and effective OOD data division process, whose overall framework is illustrated in Fig. 3. Specifically, We strictly follow the definition of OOD data division as defined in previous approaches (Averly and Chao, 2023; Yang et al., 2024b) and label the model-specified misclassified examples as OOD data to improve the safety and reliability of the deep learning model in real-world applications. The underlying intuition is that, taking CLIP (Radford et al., 2021) as an example, its large-scale contrastive learning mechanism naturally forms a clustering structure in the image-text embedding space during training, grouping ”main semantic objects” and ”typical forms.” In contrast, non-main semantic objects, semantic variants, or anomalous forms typically exhibit significant deviations within this space. Therefore, when an image is paired with text describing non-primary semantic content, the unstable or erroneous predictions generated by CLIP fundamentally reflect that “the current semantic distribution of the image-text pair has deviated from its training distribution.” We leverage this anomaly in semantic distance manifested within the embedding space as an auxiliary signal to identify potential OOD candidate samples. Given an image x and all candidate category labels y=y1,…,yi,…,yny=\y_1,…,y_i,…,y_n\, where i denotes the i-th label and n is the total number of category labels (e.g., the number of category labels in the COCO dataset n=80n=80). We input both images and labels into the off-the-shelf pretrained OOD detector to obtain logits of (x,y)(x,y) pairs. Purify. Since an image may correspond to multiple labels, we note that the corresponding label of the image is yix∈y_i^x∈ y. Direct softmax operation on logits may lead to interference between labels because softmax assumes that each sample can only belong to one category, which makes the final image-category matching probability not accurately reflect the actual probability. For example, when an image contains both cat and dog categories, softmax calculates the probability by comparing the scores of each category with the scores of the other categories. Therefore, the logits of cat and dog will interact with each other after softmax, which may lead to inaccurate classification results as the cat’s score decreases due to the dog’s score and vice versa. To address this issue, we introduce the purify operation to eliminate the interaction between different yxy^x. In the implementation, we select one of the yxy^x as the current category and set the logits of the other yxy^x to negative infinity, thus eliminating the interference of other labels on the selected label. After the purify operation, we obtained the (x,y)(x,y) match probability of the selected labels that are not interfered with by other labeling categories. Subsequently, we perform OOD data collection based on this match probability. In our setting, failure detection is divided into two cases. The first case is that the probability of the category y′y that does not exist on the image is higher than the probability of all label categories yxy^x, i.e., P(x,y′|y′∈y,y′∉yx)>P(x,yx)P(x,y |y ∈ y,y ∉ y^x)>P(x,y^x). The second case is that the probability of the labeling category yxy^x of the current image x is lower than a hyperparameter threshold T, i.e., P(x,yx)<TP(x,y^x)<T. By filtering the samples that satisfy these two cases, we have successfully acquired the instance-level OOD data. However, the OOD data collected by the model-specified approach is closely related to the OOD detectors. To mitigate the OOD division bias introduced by these detectors and their potential classification errors, while collecting OOD data with greater real-world adaptability and universality, thereby enhancing model reliability in practical applications, we introduce a cross-validation approach with multiple generalized OOD detectors. In implementation, we chose CLIP (Radford et al., 2021) and BLIP2 (Li et al., 2023b) as OOD detectors and defined the intersection of OOD data divided by the two models as OOD-Hard data, while the symmetric difference of data divided by the two models is defined as OOD-Simple data. After completing the OOD data division, we further conducted manual sampling inspections to ensure that the division strictly conformed to our definition of OOD. The experimental results show that the OOD data classified by our proposed OODBench can effectively approximate the actual OOD distribution faced by large VLMs. Users can easily collect and extend OOD data through this process for applications in different fields. Our experiments demonstrate that, despite the prevalence of these OOD data classes in real-world scenarios, current state-of-the-art VLMs still struggle with out-of-distribution inputs. Even the GPT-4o achieves only approximately 65%65\% accuracy on OOD-H data, compared to over 90%90\% on in-distribution samples. Figure 3: Pipeline of OOD data collection. Figure 4: Basic-to-Advanced Progression Metric Example. Basic-to-Advanced Progression Metric covers three problems: existential, counting, and logical reasoning problems from top to bottom. Basic-to-Advanced Progression (BAP) Metric. Through the above process, we divide the OOD data at the instance level. To comprehensively evaluate the impact of OOD data on VLMs from different dimensions, we meticulously devised the Basic-to-Advanced Progression Metric. Specifically, as shown in Fig. 4, we examine the recognition ability, counting perception ability, and reasoning ability of VLMs by constructing existential, counting, and logical reasoning problems. The existential question is the essential reasoning task. It is usually the precursor to other questions, which require the model to make a judgment about whether a particular category of objects is present in the image. The counting problem is a further judgment about the number of objects in the image after the model has correctly answered the existential question. This stage requires the model to return a specific numerical value rather than a simple ‘yes‘ or ‘no‘. In the third stage, we evaluate the reasoning ability of the model by asking questions about different objects in the same image (e.g., Is the number of [class 1] in the image greater than the number of [class 2]? Answer with ‘yes‘ or ‘no‘). Logical reasoning problems build on counting problems, which require the model to be aware not only of the number of objects in each category but also of making comparisons between the number of categories. Progressive questioning helps to improve the interpretability of the model’s reasoning process. Each step of the answer is based on clear logic, which helps the user understand the thinking process of the model. Finally, we calculated C/N×100%C/N× 100\% as the BAP Metric, where C is the number of questions answered correctly, and N is the total number of questions. 5 Experiment 5.1 Selected Datasets COCO. (Lin et al., 2014). The COCO dataset is a widely used natural image dataset in computer vision. It contains over 330,000 images across 80 categories (e.g., people, animals, vehicles, etc.). We chose its 2017 version validation set as the source of OODBench metadata. LVIS (Gupta et al., 2019). The LVIS dataset is a large-scale natural scene dataset that focuses on instance segmentation. Especially, it focuses on the recognition of fine-grained categories and provides rich instance-level annotations. nuScenes (Caesar et al., 2020). nuScenes is a large-scale autonomous driving dataset that provides 360-degree views of city streets, covering a wide range of urban environments and road conditions. The dataset provides camera, LiDAR, and radar sensor data. To control the benchmark’s sample number, we selected 6,019 keyframes from the front camera in the validation set as metadata for nuScenes. Cityscapes (Cordts et al., 2016). Cityscapes is an autonomous driving dataset focused on urban street scenes for scene understanding. We chose images from the camera on the left side of the validation set as metadata for Cityscapes. In addition, we adjusted labels to ensure that their expressions conform to natural language conventions and excluded potentially ambiguous labels. For more details, please refer to Appendix A. Figure 5: The OODBench example has two questions and one instance, with the answers alternated to avoid the model being over-scored due to biased output distribution. 5.2 Evaluation Setup In our main results, in order to comprehensively evaluate the performance of the model, we selected several commonly used evaluation metrics, including Accuracy, F1 Score, Precision, Recall, and Matthews Correlation Coefficient (MCC). In addition, in the experiments of BAP Metric, we introduced three additional metrics to refine the evaluation further. Specifically, we define the ‘Existential Accuracy‘ (E-Acc) metric, where a point is awarded only if the model correctly answers the existential question. The ‘Counting Accuracy‘ (C-Acc) metric, on the other hand, requires the model to obtain a point only when it correctly answers the number of specified categories in the image. Finally, we award a point for ‘Logical Accuracy‘ (L-Acc) when the model correctly answers both counting and logical questions in the test sample. Regarding the OOD detectors and the hyperparameter T, we conducted systematic ablation studies to determine the optimal value of T as well as the type and number of generalized OOD detectors. Detailed analyses of their impact on performance are provided in Appendix E. Model Image Encoder Language Model Data Type OODBench Performance Accuracy(%\%) F1(%\%) Precision(%\%) Recall(%\%) MCC(%\%) Random Chance - - ID/OOD-S/OOD-H 50.00 50.00 50.00 50.00 0.00 Open-source Models ID 84.79 84.72 85.15 84.28 69.59 OOD-S 64.54 56.16 73.52 45.44 31.46 OOD-H 63.22 52.99 73.40 41.46 29.36 IDCoTID^CoT 64.96 70.74 60.72 84.72 32.57 OOD-SCoTOOD-S^CoT 59.66 55.49 61.88 50.29 19.66 LLaVA-NeXT-8B CLIP-L-14 Llama-3-8B-Instruct OOD-HCoTOOD-H^CoT 64.40 57.41 71.44 47.99 30.49 ID 89.14 89.02 90.07 87.98 78.31 OOD-S 64.36 58.19 70.38 49.60 30.07 OOD-H 61.14 52.82 67.21 43.50 23.81 IDCoTID^CoT 85.14 84.59 87.82 81.60 70.46 OOD-SCoTOOD-S^CoT 63.15 56.40 69.05 47.67 27.66 DeepSeek-VL-7B-Chat SigLIP-L &\& SAM-B DeepSeek-LLM-7B OOD-HCoTOOD-H^CoT 57.44 48.27 61.53 39.72 15.92 ID 84.81 83.07 74.51 93.84 71.15 OOD-S 63.03 49.96 36.91 77.28 30.56 OOD-H 60.68 43.58 30.38 77.10 26.85 IDCoTID^CoT 82.95 79.50 66.13 99.64 69.96 OOD-SCoTOOD-S^CoT 59.81 40.17 26.98 78.58 26.02 DeepSeek-VL2-Small SigLIP-SO400M DeepSeekMoE LLM OOD-HCoTOOD-H^CoT 56.04 29.82 18.68 73.92 18.19 ID 90.93 90.75 92.51 89.06 81.91 OOD-S 69.56 67.58 72.27 63.46 39.40 OOD-H 59.05 55.87 60.58 51.85 18.30 IDCoTID^CoT 93.18 92.75 98.95 87.29 86.97 OOD-SCoTOOD-S^CoT 66.90 59.08 77.36 47.79 36.58 InternVL2-8B InternViT-300M-448px InternLM2.5-7b-chat OOD-HCoTOOD-H^CoT 69.26 61.11 83.17 48.30 42.44 ID 91.66 91.79 90.37 93.26 83.37 OOD-S 72.37 71.51 73.82 69.34 44.83 OOD-H 58.49 57.91 58.73 57.11 16.98 IDCoTID^CoT 90.49 90.56 89.89 91.24 80.99 OOD-SCoTOOD-S^CoT 69.87 65.93 75.84 58.30 40.84 InternVL2.5-8B InternViT-300M-448px-V2.5 InternLM2.5-7b-chat OOD-HCoTOOD-H^CoT 67.09 63.57 71.19 57.42 34.84 ID 92.73 92.87 91.16 94.65 85.53 OOD-S 68.65 68.42 68.94 67.91 37.31 OOD-H 66.26 67.55 65.07 70.22 32.63 IDCoTID^CoT 74.49 76.31 71.23 82.18 49.57 OOD-SCoTOOD-S^CoT 59.80 58.26 60.59 56.10 19.66 Llama-3.2-11B-Vision ViT-H-14 Llama3.1-8B OOD-HCoTOOD-H^CoT 61.14 62.20 60.55 63.94 22.31 ID 90.67 90.92 88.52 93.46 81.47 OOD-S 73.01 70.79 77.14 65.40 46.56 OOD-H 66.77 63.19 70.82 57.05 34.20 IDCoTID^CoT 90.65 90.92 88.32 93.69 81.45 OOD-SCoTOOD-S^CoT 71.61 69.27 75.50 63.99 43.74 Qwen2-VL-7B-Instruct CLIP-L-14 Qwen2-7B OOD-HCoTOOD-H^CoT 65.23 62.10 68.24 56.97 30.87 ID 90.35 90.14 88.31 92.06 80.76 OOD-S 64.65 56.67 46.25 73.17 31.50 OOD-H 68.70 61.31 49.61 80.24 40.46 IDCoTID^CoT 93.29 92.81 86.60 99.97 87.36 OOD-SCoTOOD-S^CoT 68.27 58.75 45.18 83.94 41.19 Qwen2.5-VL-7B-Instruct ViT(SwiGLU, RMSNorm) Qwen2.5 LLM OOD-HCoTOOD-H^CoT 68.93 59.31 45.29 85.92 42.97 Closed-source Models ID 91.12 91.29 89.52 93.14 82.30 OOD-S 71.78 71.87 71.64 72.10 43.56 OOD-H 63.54 64.38 62.93 65.89 27.11 IDCoTID^CoT 94.37 94.30 95.56 93.07 88.77 OOD-SCoTOOD-S^CoT 69.36 65.34 75.22 57.74 39.82 Gemini - - OOD-HCoTOOD-H^CoT 73.45 71.02 78.16 65.07 47.56 ID 91.95 92.24 89.04 95.68 84.14 OOD-S 73.37 74.13 72.05 76.34 46.81 OOD-H 65.13 67.06 63.54 71.00 30.47 IDCoTID^CoT 78.13 80.94 71.72 92.87 58.87 OOD-SCoTOOD-S^CoT 62.44 61.49 63.08 59.98 24.91 GPT-4o - GPT-4 OOD-HCoTOOD-H^CoT 63.62 64.65 62.88 66.52 27.29 Table 1: Performance on OODBench. We report the performance of the 10 leading VLMs on OODBench. All models perform significantly lower on OOD-H than on ID. The performance of the latest models, such as LLaVA-NeXT, Llama3.2-Vision, InternVL2.5, and Qwen2-VL, lags behind the ID data by about 20%20\% to 30%30\% on OOD-H data. Even the best closed-source model, GPT-4o, still performs 26%26\% lower on OOD-H than ID. 5.3 Main Results Analysis Through our OOD data collection process, we divide instance-level OOD data into distinct subsets. We define the intersection of OOD data identified by multiple OOD detectors as OOD-H, and the symmetric difference as OOD-S. Based on this process, ID data are obtained by randomly pairing images and categories and excluding pairs identified as OOD. As shown in Fig. 5, we present two questions for each instance respectively (Does this image contain a [class]? (yes or no) and Does this image not contain a [class]? (yes or no)) and assign the opposite label to each question. This strategy effectively balances the label distributions of different instance data. It mitigates the tendency of VLMs to favor either ‘yes‘ or ‘no‘ responses due to training data bias. As the task format superficially resembles prompt templates used in some hallucination detection work, we systematically compare hallucination data with OODBench in Appendix D across three dimensions—definition, generation mechanism, and experimental results—to clarify their fundamental differences. Tab. 1 shows the performance of the current state-of-the-art LLaVA-NeXT (Li et al., 2024b), DeepSeek-VL (Lu et al., 2024), InternVL2 (Chen et al., 2024c), InternVL2.5 (Chen et al., 2024a), Llama-3.2-Vision (Dubey et al., 2024), Qwen2-VL (Wang et al., 2024a), Gemini (Team et al., 2024) and GPT-4o (Achiam et al., 2023; Hurst et al., 2024) models on OODBench. The results indicate that most VLMs exhibit a 20%20\% to 30%30\% accuracy drop on OOD-H data relative to ID data. In addition, all models show a decreasing trend in accuracy on ID, OOD-S, and OOD-H data. Essentially, OOD-S data is data that is only identified as OOD data by one OOD detector and not identified as OOD data by another OOD detector. In contrast, OOD-H data is identified as OOD data by both OOD detectors. These results show decreasing accuracy on ID, OOD-S, and OOD-H data, suggesting that multiple well-pretrained VLMs are more effective OOD detectors for collecting data aligned with real-world scenarios than a single detector. Moreover, this process requires no extra human effort, minimizing labor costs. Notably, VLMs (e.g., LLaVA-NeXT, DeepSeek-VL, InternVL2/2.5, Qwen2-VL) show poor recall on OOD-H data, with LLaVA-NeXT and DeepSeek-VL even falling below 50%50\% random chance. By contrast, recall on ID data remains around 90%90\%, yielding a 2626–43%43\% drop on OOD-H. These results suggest that instances in OOD data are more likely to be missed than those in ID data. In safety-critical settings, low recall can be particularly problematic. For example, an autonomous driving system must detect all hazards—pedestrians, vehicles, or traffic signs, since missing any can greatly increase accident risk. Appendix F details VLM performance on natural and driving datasets, illustrating how OOD data affect model behavior across scenarios. Model Data Type OODBench BAP Performance E-Acc(%\%) C-Acc(%\%) L-Acc(%\%) ID 81.39 38.25 21.70 OOD-S 66.61 33.75 21.48 LLaVA-NeXT-8B OOD-H 44.22 21.71 13.45 ID 84.66 38.67 24.40 OOD-S 69.73 26.72 17.50 DeepSeek-VL-7B-Chat OOD-H 46.25 22.34 14.46 ID 70.77 24.32 15.04 OOD-S 63.54 30.75 17.96 DeepSeek-VL2-Small OOD-H 37.74 20.32 12.09 ID 87.57 51.07 33.35 OOD-S 73.20 40.43 26.62 InternVL2-8B OOD-H 56.72 40.20 28.43 ID 91.34 45.63 32.42 OOD-S 76.89 38.68 29.03 InternVL2.5-8B OOD-H 61.62 35.40 26.33 ID 94.57 47.27 28.44 OOD-S 77.99 38.95 22.49 Llama-3.2-11B-Vision-Instruct OOD-H 74.33 31.58 18.56 ID 91.73 46.13 30.01 OOD-S 75.31 42.36 30.72 Qwen2-VL-7B-Instruct OOD-H 59.77 33.44 23.88 ID 86.34 37.38 32.75 OOD-S 73.39 39.25 28.13 Qwen2.5-VL-7B-Instruct OOD-H 56.52 33.67 24.39 ID 91.43 58.36 35.56 OOD-S 77.84 52.20 34.12 Gemini OOD-H 72.28 42.82 27.86 ID 94.84 48.26 30.83 OOD-S 81.26 41.43 27.93 GPT-4o OOD-H 76.38 36.00 24.36 Table 2: Performance of 10 leading VLMs on OODBench for Basic-to-Advanced Progress. 5.4 Does Chain-of-Thought Reasoning Work on OOD Data? Chain-of-Thought (CoT) prompting encourages models to generate intermediate steps before producing an answer, and has been reported to improve performance on reasoning-style tasks. However, when confronted with OOD data, models often fail to reason accurately using their existing knowledge, as such data fall outside the training distribution. A natural question is whether CoT prompting still helps in these cases. Specifically, does explicitly expanding the inference process help models handle unfamiliar inputs? To investigate this, we add a CoT prompt requiring step-by-step analysis and evaluate whether it improves VLM performance on OOD data. OOur prompt setup mirrors the main experiment, with an added requirement for stepwise analysis: e.g., Does this image contain [class]? Yes or No? Let’s break down the information step by step. and Does this image not contain [class]? Yes or No? Let’s break down the information step by step. Table 1 reports the performance of different VLMs with CoT prompting. Gemini (Team et al., 2024), InternVL2 (Chen et al., 2024c), and InternVL2.5 (Chen et al., 2024a) exhibit similar improvements: on challenging OOD-H samples, accuracy increases by about 10% over baseline; ID accuracy remains stable or rises slightly; OOD-S accuracy declines by 2–3% on average. Precision gains 15.23%, 22.59%, and 12.46% for Gemini, InternVL2, and InternVL2.5, respectively, while recall is unchanged or marginally lower. In contrast, LLaVA-NeXT (Li et al., 2024b), Qwen2-VL (Wang et al., 2024a), and Qwen2.5-VL (Bai et al., 2025a) show negligible OOD-H gains: Qwen2-VL remains stable on ID and OOD-S, whereas LLaVA-NeXT’s ID accuracy drops from 84.79% to 64.96%. Moreover, GPT-4o (Achiam et al., 2023; Hurst et al., 2024), DeepSeek-VL (Lu et al., 2024), DeepSeek-VL2 (Wu et al., 2024a), and Llama-3.2-Vision (Dubey et al., 2024) suffer overall performance degradation after CoT prompting: GPT-4o and Llama-3.2-Vision ID accuracy falls by 13.82% and 18.27%, respectively, and OOD-H accuracy declines by 1.5–5%. Hence, CoT yields negative gains for these models. Figure 6: Typical failure cases of GPT-4o on OODBench. Errors primarily fall into two categories: non-main semantic objects and semantic variants. 5.5 BAP Evaluation of ID and OOD Data Differences To assess VLM performance on OOD data across multiple dimensions, we introduce the Basic-to-Advanced Progression (BAP) metric, which evaluates model performance in recognition, counting, and reasoning tasks. Specifically, we define three metrics—Existential Accuracy (E-Acc), Counting Accuracy (C-Acc), and Logical Accuracy (L-Acc)—using a progressive questioning framework to assess existential, counting, and logical challenges, respectively. For BAP evaluation, we select images containing at least two distinct labels from the ID, OOD-S, and OOD-H datasets. Each image is paired with two distinct category labels (without duplication), forming test samples with paired labels from different categories. The quantity labels for counting tasks are obtained by counting the number of target object instances in the source annotations. As task difficulty increases from E-Acc to C-Acc to L-Acc, all models exhibit a consistent decline in performance. This trend becomes more pronounced as the data distribution shifts from ID to OOD-S and OOD-H, corresponding to increasing divergence from the training distribution and resulting in varying degrees of performance degradation across metrics. Most models achieve robust results on ID data, particularly on E-Acc and C-Acc. However, performance declines as the data transitions to OOD-S and further to OOD-H, with the largest drop observed on L-Acc. On more challenging OOD data (e.g., OOD-H), models exhibit a substantial decrease in logical accuracy. E-Acc mainly tests recognition without complex reasoning. On OOD-H data, GPT-4o (Achiam et al., 2023; Hurst et al., 2024) shows the best recognition, correctly identifying category existence in images. In contrast, C-Acc evaluates the ability to count object instances, requiring both recognition and accurate counting. Gemini (Team et al., 2024) leads with 42.82%42.82\%, underscoring its relative strength in counting. L-Acc evaluates logical reasoning by comparing category counts (e.g., Is A greater than B?). This task couples counting with higher-level reasoning, and all models show a sharp drop in performance, highlighting the challenge of logical reasoning and quantitative relations in OOD settings. 6 Analysis of Error Cases As shown in Figure 6, we present a qualitative analysis of failure patterns for the representative GPT-4o model on OODBench. Errors primarily cluster into two semantic OOD categories: non-main objects and semantic variants. (1) Failures caused by non-primary semantic objects. When a clear main semantic object exists in an image (e.g., ”people” in the first and fourth examples of the first row in Figure 6), the model’s image–caption pretraining objective tends to emphasize alignment with the main semantic object, while learning about non-main semantic objects is insufficient or even absent. Therefore, when we construct scenarios in OODBench where the main semantic object is present, but the question involves a non-main semantic object, the model appears to lack stable image–text associations for these semantics, leading to consistent mispredictions. (2) Failures due to semantic variants. Another common error category stems from semantic variants. For example, the ”skateboard made of cake” in the first sample of the second row in Figure 6. During training, the model predominantly encounters “skateboards” with standard appearances drawn from natural image distributions. The joint distribution of images and text forms the model’s default semantic template for the concept of ”skateboard.” When we replace the target object with variants featuring different materials, shapes, or construction methods (such as a ”cake-shaped skateboard”), this sample deviates from the training distribution within the model’s image-text semantic space. Consequently, the model misclassifies or ignores it. These qualitative observations suggest that failures under semantic shifts are not random but linked to mismatches between joint image–text distributions and pretrained representations, consistent with our quantitative results. 7 Conclusion In this paper, we introduce OODBench to evaluate the performance of Vision Language Models (VLMs) in the face of out-of-distribution data. OODBench is able to more accurately reflect the coping ability of VLMs when they encounter data distribution shifts in real-world scenarios. In addition, we present the Basic-to-Advanced Progression Metric for evaluating the recognition, count-awareness, and reasoning abilities of VLMs. Our automated OOD division benchmarking process adapts efficiently to new data sources and does not require significant human effort, enabling future evaluations of the ability of vision language models to be out-of-distribution of different domains. 8 Impact Statements This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. References J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. (2023) Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §1, §2, §5.3, §5.4, §5.5. J. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. (2022) Flamingo: a visual language model for few-shot learning. NeurIPS 35, p. 23716–23736. Cited by: Table 6. R. Averly and W. Chao (2023) Unified out-of-distribution detection: a model-specific perspective. In ICCV, p. 1453–1463. Cited by: Appendix D, §1, §2, §4. A. Awadalla, I. Gao, J. Gardner, J. Hessel, Y. Hanafy, W. Zhu, K. Marathe, Y. Bitton, S. Gadre, S. Sagawa, et al. (2023) Openflamingo: an open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390. Cited by: Table 6. J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou (2023) Qwen-vl: a versatile vision-language model for understanding, localization, text reading, and beyond. External Links: 2308.12966, Link Cited by: Table 6. S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. (2025a) Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. Cited by: §5.4. S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. (2025b) Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. Cited by: Table 6. Z. Bai, P. Wang, T. Xiao, T. He, Z. Han, Z. Zhang, and M. Z. Shou (2024) Hallucination of multimodal large language models: a survey. arXiv preprint arXiv:2404.18930. Cited by: Appendix D. Baidu-ERNIE-Team (2025) ERNIE 4.5 technical report. Note: https://ernie.baidu.com/blog/publication/ERNIE_Technical_Report.pdf Cited by: Table 6. H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom (2020) Nuscenes: a multimodal dataset for autonomous driving. In CVPR, p. 11621–11631. Cited by: §4, §5.1. J. Chen, D. Zhu, X. Shen, X. Li, Z. Liu, P. Zhang, R. Krishnamoorthi, V. Chandra, Y. Xiong, and M. Elhoseiny (2023a) Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478. Cited by: Table 6. X. Chen, J. Djolonga, P. Padlewski, B. Mustafa, S. Changpinyo, J. Wu, C. R. Ruiz, S. Goodman, X. Wang, Y. Tay, et al. (2023b) Pali-x: on scaling up a multilingual vision and language model. arXiv preprint arXiv:2305.18565. Cited by: Table 6. Z. Chen, W. Wang, Y. Cao, Y. Liu, Z. Gao, E. Cui, J. Zhu, S. Ye, H. Tian, Z. Liu, et al. (2024a) Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271. Cited by: §1, §2, §5.3, §5.4. Z. Chen, W. Wang, Y. Cao, Y. Liu, Z. Gao, E. Cui, J. Zhu, S. Ye, H. Tian, Z. Liu, et al. (2024b) Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271. Cited by: Table 6, Table 6. Z. Chen, W. Wang, H. Tian, S. Ye, Z. Gao, E. Cui, W. Tong, K. Hu, J. Luo, Z. Ma, et al. (2024c) How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821. Cited by: §1, §2, §5.3, §5.4. Z. Chen, W. Wang, H. Tian, S. Ye, Z. Gao, E. Cui, W. Tong, K. Hu, J. Luo, Z. Ma, et al. (2024d) How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. SCIS 67 (12), p. 220101. Cited by: Table 6. Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. (2024e) Internvl: scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, p. 24185–24198. Cited by: Table 6. M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele (2016) The cityscapes dataset for semantic urban scene understanding. In CVPR, Cited by: §4, §5.1. W. Dai, J. Li, D. Li, A. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi (2023) Instructblip: towards general-purpose vision-language models with instruction tuning. NeurIPS 36, p. 49250–49267. Cited by: Table 6. C. Deng, D. Zhu, K. Li, C. Gou, F. Li, Z. Wang, S. Zhong, W. Yu, X. Nie, Z. Song, et al. (2025) Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683. Cited by: Table 6. A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. (2024) The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §E.3, Table 9, Table 9, §1, §5.3, §5.4. T. Guan, F. Liu, X. Wu, R. Xian, Z. Li, X. Liu, X. Wang, L. Chen, F. Huang, Y. Yacoob, et al. (2024) Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 14375–14385. Cited by: §2. J. Guérin, K. Delmas, R. Ferreira, and J. Guiochet (2023) Out-of-distribution detection is not all you need. In AAAI, Vol. 37, p. 14829–14837. Cited by: §1. A. Gupta, P. Dollar, and R. Girshick (2019) Lvis: a dataset for large vocabulary instance segmentation. In CVPR, p. 5356–5364. Cited by: §4, §5.1. D. Hendrycks, S. Basart, M. Mazeika, A. Zou, J. Kwon, M. Mostajabi, J. Steinhardt, and D. Song (2022) Scaling out-of-distribution detection for real-world settings. ICML. Cited by: §1. D. Hendrycks and K. Gimpel (2017) A baseline for detecting misclassified and out-of-distribution examples in neural networks. ICLR. Cited by: Appendix D. D. A. Hudson and C. D. Manning (2019) Gqa: a new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 6700–6709. Cited by: §2. A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. (2024) Gpt-4o system card. arXiv preprint arXiv:2410.21276. Cited by: §1, §2, §5.3, §5.4, §5.5. P. F. Jaeger, C. T. Lüth, L. Klein, and T. J. Bungert (2023) A call to reflect on evaluation practices for failure detection in image classification. ICLR. Cited by: §1. B. Li, K. Zhang, H. Zhang, D. Guo, R. Zhang, F. Li, Y. Zhang, Z. Liu, and C. Li (2024a) LLaVA-next: stronger llms supercharge multimodal capabilities in the wild. External Links: Link Cited by: Table 6. B. Li, R. Wang, G. Wang, Y. Ge, Y. Ge, and Y. Shan (2023a) Seed-bench: benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125. Cited by: §2. F. Li, R. Zhang, H. Zhang, Y. Zhang, B. Li, W. Li, Z. Ma, and C. Li (2024b) LLaVA-next-interleave: tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895. Cited by: §1, §2, §5.3, §5.4. J. Li, D. Li, S. Savarese, and S. Hoi (2023b) Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, p. 19730–19742. Cited by: §E.2, §1, §2, §4. Z. Li, B. Yang, Q. Liu, Z. Ma, S. Zhang, J. Yang, Y. Sun, Y. Liu, and X. Bai (2024c) Monkey: image resolution and text label are important things for large multi-modal models. In CVPR, p. 26763–26773. Cited by: Table 6. T. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár (2017) Focal loss for dense object detection. In ICCV, p. 2980–2988. Cited by: Appendix C. T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick (2014) Microsoft coco: common objects in context. In ECCV, p. 740–755. Cited by: §2, §4, §5.1. Z. Lipton, Y. Wang, and A. Smola (2018) Detecting and correcting for label shift with black box predictors. In ICML, p. 3122–3130. Cited by: §3. H. Liu, W. Xue, Y. Chen, D. Chen, X. Zhao, K. Wang, L. Hou, R. Li, and W. Peng (2024a) A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253. Cited by: Appendix D. H. Liu, C. Li, Y. Li, and Y. J. Lee (2024b) Improved baselines with visual instruction tuning. In CVPR, p. 26296–26306. Cited by: Table 6. H. Liu, C. Li, Q. Wu, and Y. J. Lee (2023) Visual instruction tuning. NeurIPS 36, p. 34892–34916. Cited by: Table 6. J. Liu, Z. Shen, Y. He, X. Zhang, R. Xu, H. Yu, and P. Cui (2021) Towards out-of-distribution generalization: a survey. arXiv preprint arXiv:2108.13624. Cited by: §2. Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, et al. (2025) Mmbench: is your multi-modal model an all-around player?. In European Conference on Computer Vision, p. 216–233. Cited by: §2. H. Lu, W. Liu, B. Zhang, B. Wang, K. Dong, B. Liu, J. Sun, T. Ren, Z. Li, Y. Sun, C. Deng, H. Xu, Z. Xie, and C. Ruan (2024) DeepSeek-vl: towards real-world vision-language understanding. External Links: 2403.05525 Cited by: Table 6, §1, §5.3, §5.4. M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. (2024) DINOv2: learning robust visual features without supervision. Transactions on Machine Learning Research Journal, p. 1–31. Cited by: §B.1.4. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In ICML, p. 8748–8763. Cited by: §E.1, §E.2, §1, §4, §4. M. Salehi, H. Mirzaei, D. Hendrycks, Y. Li, M. H. Rohban, and M. Sabokrou (2021) A unified survey on anomaly, novelty, open-set, and out-of-distribution detection: solutions and future challenges. arXiv preprint arXiv:2110.14051. Cited by: §2. C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. (2022) Laion-5b: an open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems 35, p. 25278–25294. Cited by: §2. H. Shimodaira (2000) Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference 90 (2), p. 227–244. Cited by: §3. A. Shrivastava, A. Gupta, and R. Girshick (2016) Training region-based object detectors with online hard example mining. In CVPR, p. 761–769. Cited by: Appendix C. G. Team, P. Georgiev, V. I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang, et al. (2024) Gemini 1.5: unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. Cited by: §1, §2, §5.3, §5.4, §5.5. G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, et al. (2025) Gemma 3 technical report. arXiv preprint arXiv:2503.19786. Cited by: Table 6. S. Tong, E. Jones, and J. Steinhardt (2023) Mass-producing failures of multimodal systems with language models. NeurIPS 36, p. 29292–29322. Cited by: §E.1. S. Tong, Z. Liu, Y. Zhai, Y. Ma, Y. LeCun, and S. Xie (2024) Eyes wide shut? exploring the visual shortcomings of multimodal llms. In CVPR, p. 9568–9578. Cited by: §E.1. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. (2023) Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: §E.3. Ultralytics (2023) YOLOv8. Note: https://github.com/ultralytics/ultralyticsVersion 8.0.0 Cited by: Appendix C. H. Wang, Z. Li, L. Feng, and W. Zhang (2022) Vim: out-of-distribution with virtual-logit matching. In CVPR, p. 4921–4930. Cited by: §1. P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Lin (2024a) Qwen2-vl: enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191. Cited by: §1, §5.3, §5.4. P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, et al. (2024b) Qwen2-vl: enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191. Cited by: Table 6. W. Wang, Q. Lv, W. Yu, W. Hong, J. Qi, Y. Wang, J. Ji, Z. Yang, L. Zhao, S. XiXuan, et al. (2024c) Cogvlm: visual expert for pretrained language models. NeurIPS 37, p. 121475–121499. Cited by: Table 6, Table 6. W. Wang, Y. Tian, L. Yang, H. Wang, and X. Yan (2025) Open-qwen2vl: compute-efficient pre-training of fully-open multimodal llms on academic resources. arXiv preprint arXiv:2504.00595. Cited by: Table 6. X. Wang, X. Zhang, Z. Luo, Q. Sun, Y. Cui, J. Wang, F. Zhang, Y. Wang, Z. Li, Q. Yu, et al. (2024d) Emu3: next-token prediction is all you need. arXiv preprint arXiv:2409.18869. Cited by: Table 6. O. Wiles, S. Gowal, F. Stimberg, S. Alvise-Rebuffi, I. Ktena, K. Dvijotham, and T. Cemgil (2021) A fine-grained analysis on distribution shift. arXiv preprint arXiv:2110.11328. Cited by: §2. Z. Wu, X. Chen, Z. Pan, X. Liu, W. Liu, D. Dai, H. Gao, Y. Ma, C. Wu, B. Wang, et al. (2024a) Deepseek-vl2: mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302. Cited by: §5.4. Z. Wu, X. Chen, Z. Pan, X. Liu, W. Liu, D. Dai, H. Gao, Y. Ma, C. Wu, B. Wang, et al. (2024b) Deepseek-vl2: mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302. Cited by: Table 6. J. Xu, S. De Mello, S. Liu, W. Byeon, T. Breuel, J. Kautz, and X. Wang (2022) Groupvit: semantic segmentation emerges from text supervision. In CVPR, p. 18134–18144. Cited by: §E.1, §E.2. L. Xue, M. Shu, A. Awadalla, J. Wang, A. Yan, S. Purushwalkam, H. Zhou, V. Prabhu, Y. Dai, M. S. Ryoo, et al. (2025) Blip-3: a family of open large multimodal models. In ICCV, p. 6124–6135. Cited by: Table 6. J. Yang, K. Zhou, Y. Li, and Z. Liu (2024a) Generalized out-of-distribution detection: a survey. IJCV 132 (12), p. 5635–5662. Cited by: §1, §2. W. Yang, B. Zhang, and O. Russakovsky (2024b) ImageNet-ood: deciphering modern out-of-distribution detection algorithms. ICLR. Cited by: Appendix D, §1, §2, §4. Y. Yao, T. Yu, A. Zhang, C. Wang, J. Cui, H. Zhu, T. Cai, H. Li, W. Zhao, Z. He, et al. (2024) Minicpm-v: a gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800. Cited by: Table 6. X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers, A. Kolesnikov, and L. Beyer (2022) Lit: zero-shot transfer with locked-image text tuning. In CVPR, p. 18123–18133. Cited by: §E.1, §E.2. Y. Zhang, A. Unell, X. Wang, D. Ghosh, Y. Su, L. Schmidt, and S. Yeung-Levy (2024) Why are visually-grounded language models bad at image classification?. In NeurIPS, Cited by: Appendix A. Z. Zhang, A. Zhang, M. Li, H. Zhao, G. Karypis, and A. Smola (2023) Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923. Cited by: Appendix D. D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny (2023) Minigpt-4: enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592. Cited by: Table 6. Y. Zhu, Y. Chen, X. Li, R. Zhang, H. Xue, X. Tian, R. Jiang, B. Zheng, and Y. Chen (2024) Rethinking out-of-distribution detection from a human-centric perspective. IJCV, p. 1–18. Cited by: §1. Outline This document supplements the main paper with detailed results. Below is the outline. ∙ Section A details the collection process of OODBench. ∙ Section B demonstrates through a series of statistical analyses that OODBench exhibits distinct OOD properties relative to contemporary MLLMs. ∙ Section C describes the differences between the OODBench and the hard samples. ∙ Section D describes the differences between the OODBench and the hallucinations. ∙ Section E provides ablation studies of the number and type of generalized OOD Detector and hyperparameter T. ∙ Section F reports the performance of classification metrics and BAP metrics for the OODBench sub-dataset. ∙ Section G describes the performance various of models of different scales on OODBench. ∙ Section H provides a visualisation of the distribution of category logits for CLIP and BLIP2. ∙ Section I provides partial data and case demostration of OODBench. ∙ Section J discussed the limitations of OODBench. Appendix A Collection Details Initial Benchmark. Fig. 7 illustrates that we collected the initial benchmark from a publicly available dataset, which contains about 77k yes-or-no samples, where OOD-S is 51k and OOD-H is 26k. Due to the large sample size of the benchmarks, to save testing time, we randomly downsampled the OOD-Simple and OOD-Hard category data in the benchmarks with a sample size of more than 6,000 to the OODBench to ensure that the sample size of each category did not exceed 6,000. Figure 7: Distribution of categories and fields in initial benchmark. VLMs for Classification. In terms of implementation details, we input image-question pairs corresponding to OOD samples into multimodal large models to generate textual responses. Subsequently, we convert the model output into binary labels by identifying the keywords ‘yes‘ or ‘no‘ in the generated text. This method has also been applied in the literature (Zhang et al., 2024). In Tab. 1, an average accuracy of 90% is achieved on ID data, which further validates the effectiveness of the method in classification tasks. In the BAP task, we also used this method to obtain the model’s counting prediction results and compare them with the real labels to determine whether the model’s answers to counting questions were accurate. nuScenes and Cityscapes dataset label modifications. Since the labels of nuScenes do not conform to natural languages, such as ‘human.pedestrian.adult‘, using the original labels as text directly will cause the OOD detector to fail to understand their meaning, which will fail OOD data division. Therefore, as shown in Table 3, we manually convert these labels into a form that conforms to natural language conventions. Meanwhile, since the original label ‘movable object.debris‘ of nuScenes is not clear, we exclude it to avoid ambiguity. Similarly, we also eliminated the background, ambiguous, and semantic duplicates from the Cityscapes labels, and Table 4 shows the final labels we retained. Original Labels Modified Labels Remark human.pedestrian.adult Adult Pedestrian human.pedestrian.child Child Pedestrian human.pedestrian.wheelchair Pedestrian in Wheelchair human.pedestrian.stroller Pedestrian with Stroller human.pedestrian.personal_mobility Pedestrian using Personal Mobility Device human.pedestrian.police_officer Police Officer human.pedestrian.construction_worker Construction Worker animal Animal vehicle.car Car vehicle.motorcycle Motorcycle vehicle.bicycle Bicycle vehicle.bus.bendy Bendy Bus vehicle.bus.rigid Rigid Bus vehicle.truck Truck vehicle.construction Construction Vehicle vehicle.emergency.ambulance Ambulance vehicle.emergency.police Police Vehicle vehicle.trailer Trailer movable_object.barrier Barrier movable_object.trafficcone Traffic Cone movable_object.pushable_pullable Pushable/Pullable Object movable_object.debris Debris The class reference is unclear and has been deleted. static_object.bicycle_rack Bicycle Rack Table 3: Original and modified labels for nuScenes. Original Labels Modified Labels Remark road road sidewalk sidewalk parking parking rail track rail track person person rider rider car car truck truck bus bus on rails - The class reference is unclear and has been deleted. motorcycle motorcycle bicycle bicycle caravan caravan trailer trailer building building wall wall fence fence guard rail guard rail bridge bridge tunnel tunnel pole pole pole group - Pole group and pole are semantically duplicated, we choose pole for retention. traffic sign traffic sign traffic light traffic light vegetation vegetation terrain - The class reference is unclear and has been deleted. sky - Background classes we exclude. ground - The class reference is unclear and has been deleted. dynamic - The class reference is unclear and has been deleted. static - The class reference is unclear and has been deleted. Table 4: Original and modified labels for Cityscapes. Appendix B Statistical Evidence for the OOD Nature of OODBench in Modern MLLMs To verify whether OODBench constitutes OOD data within existing MLLMs, we conducted systematic statistical analysis across two complementary dimensions: open-source models and closed-source models. In the open-source model section, we calculated the semantic overlap rate between the model training corpus and the COCO-train-OOD partition, which was constructed according to the same rules as OODBench. We then evaluated the consistency and robustness of this partition using methods such as exact binomial intervals, Bayesian priors, and Two One-Sided Tests (TOST). In the closed-source model section, we further establish independent chains of evidence based on both the “significance” and “magnitude consistency” of performance results. This is achieved through significance testing for performance degradation (ID → OOD) and equivalence assessments against open-source OOD behavioral signatures. Through these complementary statistical inferences from both open-source and closed-source models, we jointly demonstrate that OODBench indeed exhibits verifiable OOD characteristics for current MLLMs, examining both the nature of data distributions and model response behaviors. B.1 OOD Evidence from Open-Source Models Open-Source MLLMs refer to models that disclose their core components to varying degrees within the research community, thereby enabling independent auditability, inspectability, reproducibility, or reusability. It is crucial to emphasize that open-source is not a binary attribute but exists along a continuum of openness. As shown in Table 5, we categorize current open-source MLLMs into four primary levels based on the scope and granularity of disclosed information, where each level exhibits a hierarchical increase in model weights, architectural details, training code, training data transparency, and reproducibility. Level Name Weights Architecture Training Code Training Data Reproducibility 0 Non-Open Source ✗ ✗ ✗ ✗ ✗ 1A Architecture Open Source ✗ ✓ Partial ✗ ✗ 1B Weights Open but Architecture Not Fully Open ✓ Partial Partial ✗ ✗ 1C Data Partially Open Optional Optional Optional Partial ✗ 2 Weights Open ✓ ✓ ✓ ✗/ Rough ✓(Partial) 3 Strongly Open (Reproducible) ✓ ✓ ✓ ✓(Clean) ✓(Partial) 4 Fully Open Source (Full Pipeline Open) ✓ ✓ ✓ ✓(Fully Open) ✓(Strict) Table 5: Current open-source MLLM models are categorized into four primary tiers based on the scope and granularity of publicly disclosed information. Each level exhibits progressively enhanced transparency and reproducibility across model weights, architectural details, training code, training data, and overall reproducibility. To rigorously assess whether the proposed OODBench constitutes out-of-distribution (OOD) data for existing open-source MLLMs, we conducted a systematic statistical analysis. We first quantified the overlap ratio between OODBench and each model’s corresponding training corpus. Then we incorporated statistical inference procedures to estimate the population-level proportion of OOD samples across models. To enhance the representativeness and cross-model comparability of these evaluations, we adopted the COCO dataset—one of the most commonly used and widely standardized datasets in multimodal model training—as the unified reference source. This provides a consistent framework for estimating overlap ratios and determining the OOD characteristics of our constructed benchmark. B.1.1 Principles for Selecting Model Samples We first conducted a systematic survey of current mainstream open-source MLLMs, covering diverse model families, parameter scales, and architectural paradigms. By comprehensively collecting publicly released versions, model repositories, and official documentation, we identified 28 model families corresponding to a candidate set of 93 models spanning parameter sizes from 0.5B to 424B. To ensure transparency and auditability at the training data level, we restricted our selection to robust open-source models at Levels 3 and 4, meaning those that either publicly disclose their training data inventories or possess strict reproducibility. In this process, we excluded the following two categories of models that did not meet these criteria: • Models using in-house or non-public data. • Models with incomplete training data lists often lack precise entries, which prevents line-by-line verification. No. Family Years Models Institution COCO Used Openness Level Included in Analysis 1 Flamingo (Alayrac et al., 2022) 2022 Flamingo-3B, Flamingo-9B DeepMind ✗ L3 ✗ 2 Qwen-VL (Bai et al., 2023) 2023 Qwen-VL(7B), Qwen-VL-Chat(7B), Qwen-VL-Plus(7B), Qwen-VL-Max(7B) Alibaba Group ✓ L3 ✓ 3 PaLI-X (Chen et al., 2023b) 2023 PaLI-X(55B) Google Research ✗ L1C ✗ 4 LLaVA-v1 (Liu et al., 2023) 2023 LLaVA-v1-Vicuna(13B), LLaVA-v1-LLama(13B), LLaVA-v1-LLama(7B) UW-Madison ✓ L3 ✓ 5 InstructBLIP (Dai et al., 2023) 2023 InstructBLIP (FlanT5XL)(3B), InstructBLIP (FlanT5XXL)(11B), InstructBLIP (Vicuna-7B), InstructBLIP (Vicuna-13B) Salesforce Research ✓ L3 ✓ 6 MiniGPT-4 (Zhu et al., 2023) 2023 MiniGPT-4 (Vicuna 13B), MiniGPT-4 (Vicuna 7B), MiniGPT-4 (LLaMA-2 Chat 7B), KAUST ✗ L3 ✗ 7 MiniGPT-v2 (Chen et al., 2023a) 2023 MiniGPT-v2-LLaMA2-7B, MiniGPT-v2-Vicuna-7B, MiniGPT-v2-Vicuna-13B KAUST ✓ L3 ✓ 8 OpenFlamingo (Awadalla et al., 2023) 2023 OpenFlamingo-3B, OpenFlamingo-3B (Instruct), OpenFlamingo-4B, OpenFlamingo-4B (Instruct), OpenFlamingo-9B UW ✗ L1C ✗ 9 InternVL (Chen et al., 2024e) 2024 InternVL-C(7B), InternVL-G(7B), InternVL-Chat-13B, InternVL-Chat-19B OpenGVLab ✓ L3 ✓ 10 DeepSeek-VL (Lu et al., 2024) 2024 DeepSeek-VL-1.3B-base, DeepSeek-VL-1.3B-chat, DeepSeek-VL-7B-base, DeepSeek-VL-7B-chat DeepSeek-AI ✗ L3 ✗ 11 LLaVA-1.5 (Liu et al., 2024b) 2024 LLaVA-1.5-7B, LLaVA-1.5-13B UW-Madison ✓ L3 ✓ 12 InternVL (Chen et al., 2024d) 2024 InternVL-1.2(40B), InternVL-1.5(26B) Shanghai AI Laboratory ✓ L3 ✓ 13 CogVLM (Wang et al., 2024c) 2024 cogvlm-chat-v1.1-Vicuna-7B, cogvlm-chat-v1.1-Llama3-8B, cogvlm-base-224(7B), cogvlm-base-490(7B), cogvlm-grounding-generalist(7B) Zhipu AI ✓ L3 ✓ 14 MiniCPM-V (Yao et al., 2024) 2024 MiniCPM-V 1.0(2.8B), MiniCPM-V 2.0(2.8B), MiniCPM-Llama3-V 2.5(8.5B) OpenBMB ✓ L3 ✓ 15 Emu3 (Wang et al., 2024d) 2024 Emu3-Chat(8B), Emu3-Gen(8B), Emu3-DPO(8B) BAAI ✗ L1C ✗ 16 Monkey (Li et al., 2024c) 2024 Monkey(9.8B) HUST ✓ L3 ✓ 17 Qwen2-VL (Wang et al., 2024b) 2024 Qwen2-VL-2B, Qwen2-VL-7B, Qwen2-VL-72B Alibaba Group unknown L1C ✗ 18 CogVLM (Wang et al., 2024c) 2024 CogVLM2-LLaMA3(8B), GLM-4V-9B Zhipu AI ✗ L1C ✗ 19 DeepSeek-VL2 (Wu et al., 2024b) 2024 DeepSeek-VL2-tiny(3B), DeepSeek-VL2-small(16B), DeepSeek-VL2(27B) DeepSeek-AI unknown L1C ✗ 20 LLaVA-NeXT (Li et al., 2024a) 2024 LLaVA-NeXT-Interleave-0.5B, LLaVA-NeXT-Interleave-7B, LLaVA-NeXT-Interleave-14B, LLaVA-NeXT-Interleave-7B-DPO ByteDance ✓ L1C ✗ 21 Open-Qwen2-VL (Wang et al., 2025) 2025 Open-Qwen2-VL(2B) UC Santa Barbara ✓ L3 ✓ 22 BLIP-3 (Xue et al., 2025) 2025 BLIP-3-4B-SI, BLIP-3-4B-MI, BLIP-3-14B-SI, BLIP-3-14B-MI Salesforce AI Research ✓ L3 ✓ 23 BAGEL (Deng et al., 2025) 2025 BAGEL-1.5B, BAGEL-7B ByteDance Seed ✓ L3 ✗ 24 InternVL2 (Chen et al., 2024b) 2025 InternVL2-2B, InternVL2-4B, InternVL2-8B, InternVL2-26B, InternVL2-40B, InternVL2-Llama3-76B Shanghai AI Laboratory ✓ L3 ✓ 25 InternVL2.5 (Chen et al., 2024b) 2025 InternVL2.5-1B, InternVL2.5-2B, InternVL2.5-4B, InternVL2.5-8B, InternVL2.5-26B, InternVL2.5-38B, InternVL2.5-78B Shanghai AI Laboratory ✓ L3 ✓ 26 Gemma-3 (Team et al., 2025) 2025 Gemma-3-1B, Gemma-3-4B, Gemma-3-12B, Gemma-3-27B Google DeepMind unknown L1C ✗ 27 ERNIE 4.5 (Baidu-ERNIE-Team, 2025) 2025 ERNIE-4.5-VL-28B-A3B-Thinking, ERNIE-4.5-VL-424B-A47B-Base, ERNIE-4.5-VL-424B-A47B, ERNIE-4.5-VL-28B-A3B-Base, ERNIE-4.5-VL-28B-A3B Baidu unknown L1C ✗ 28 Qwen2.5-VL (Bai et al., 2025b) 2025 Qwen2.5-VL-3B, Qwen2.5-VL-7B, Qwen2.5-VL-72B Alibaba Group unknown L1C ✗ Table 6: Complete summary of collected models. Includes models ultimately included in the analysis as well as those excluded during the screening process. After obtaining a set of candidate models that meet the training data transparency requirements, we further exclude models that did not use COCO as a training data source. Since our objective is to quantify the overlap ratio between OODBench-COCO and each MLLM’s training data, a model with no COCO inclusion in its training corpus would yield a trivially zero overlap ratio. This would render it incapable of providing any meaningful validation for our proposed overlap ratio analysis workflow. Consequently, such models are unsuitable as statistical test samples. After these two rounds of screening, we ultimately obtained 50 qualifying MLLMs for subsequent overlap ratio calculations and statistical inference analysis. Table 6 provides a comprehensive summary of the collected models, including both those ultimately included in the analysis and those excluded during the screening process. B.1.2 Semantic-Level OOD Pairing and Overlap Rate Analysis We first select an image and perform semantic consistency matching for its image-category within the OOD division. Specifically, we collect all annotated versions of this image across various MLLM training corpora (e.g., captions, OCR, conversations, detailed descriptions, complex reasoning, REC, REG, grounding, etc.). We then systematically examine each annotation to determine whether it contains semantic expressions matching the OOD category we assigned. If the category appears in any annotated version, it indicates the model has learned the corresponding image–category semantic relationship, and this image–category pair is considered ID data. Conversely, if the category is absent from all annotated versions, the image–category pair is classified as OOD data at the semantic level of learning. It is important to note that OODBench utilizes the COCO val dataset, while most MLLM training data is based on the COCO train version. These datasets have no overlap at the image level, making it impossible to calculate overlap rates using image matching rules directly. This version discrepancy may result in false-zero overlap. To address this issue, we regenerate OOD data using the COCO train image set, applying the same division criteria as OODBench. We then calculate the overlap rate with each MLLM’s training data based on this division. This approach verifies whether our categorized samples genuinely constitute an OOD distribution at the training semantic level. B.1.3 Population-Level OOD Assessment Settings and Symbols. We selected N auditable open-source MLLMs as test subjects. Our constructed dataset is based on COCO-train and generates OOD data at the semantic level according to OODBench’s division strategy, denoted as ”COCO-train-OOD”. For each model m, its training overlap rate on ”COCO-train-OOD” is defined as: rm=HmC, splitr_m= H_mC, split (1) where HmH_m denotes the number of samples in the model training set that achieve semantic overlap with the ”COCO-train-OOD” image-category pairs, while C represents the total number of samples in ”COCO-train-OOD”. Setting the threshold ε as the maximum acceptable overlap rate, the model’s OOD decision criterion (pre-registered criterion) is defined as: Zm=[UCI(rm)<ε], splitZ_m= 1[UCI(r_m)< ], split (2) when the 95% upper confidence bound of the training overlap rate for model m does not exceed ε , we classify it as ‘treating this data as OOD’ (Zm=1Z_m=1); if the upper confidence bound exceeds ε , it is classified as non-OOD (Zm=0Z_m=0). Exact Clopper–Pearson Interval (Frequentist Lower Bound). To estimate, at the population level, the proportion of models that satisfy the preregistered criterion, we treat the set of N evaluated open-source models as a finite population. Among them, k models satisfy Zm=1Z_m=1. We therefore compute the lower confidence bound for the binomial proportion based on k successes out of n=Nn=N trials. The lower Clopper–Pearson bound is defined as: LCP=BetaInv(0.025;k,n−k+1). splitLCP=BetaInv(0.025;k,n-k+1). split (3) The calculation results are as follows: • When ε=5% =5\%, k=49k=49, n=50n=50, LCP=BetaInv(0.025;49,2)≈0.894.LCP=BetaInv(0.025;49,2)≈ 0.894. • When ε=2% =2\%, k=27k=27, n=50n=50, LCP=BetaInv(0.025;27,24)≈0.393.LCP=BetaInv(0.025;27,24)≈ 0.393. Therefore, under the criterion of ”95%95\% upper confidence bound ≤5%≤ 5\% for training overlap rate,” it can be asserted with 95%95\% confidence that at least approximately 89.4%89.4\% of comparable open-source MLLMs would classify this data as OOD. Even under the stricter criterion of ”≤2%≤ 2\%,” at least approximately 39.3%39.3\% of models still meet this condition. Bayesian Inference with a Beta(1,1) Prior (Posterior Credible Interval). To validate the above results from a probabilistic reasoning perspective, we employ a uniform prior ”Beta(1,1)” to compute the lower bound of the 95%95\% confidence interval under the posterior distribution: LBayes=BetaInv(0.025;k+1,n−k+1). splitL_Bayes=BetaInv(0.025;k+1,n-k+1). split (4) The result is: • When ε=5% =5\%, k=49k=49, n=50n=50, LBayes=BetaInv(0.025;50,2)≈0.896.L_Bayes=BetaInv(0.025;50,2)≈ 0.896. • When ε=2% =2\%, k=27k=27, n=50n=50, LBayes=BetaInv(0.025;28,24)≈0.403.L_Bayes=BetaInv(0.025;28,24)≈ 0.403. Therefore, from a Bayesian perspective, it can be asserted with 95%95\% confidence that: when the criterion is ”95%95\% upper confidence bound ≤5%≤ 5\%,” at least approximately 89.6%89.6\% of similar models will classify the data as OOD; when the criterion is ”≤2%≤ 2\%,” at least approximately 40.3%40.3\% of models will classify the data as OOD. Population-Level Conclusion. From two complementary statistical perspectives, the Clopper–Pearson method ensures frequentist coverage, while the Bayesian Beta method provides posterior credibility. Both methods align in direction and magnitude, indicating that, with 95%95\% confidence, at least approximately 89–90%89–90\% of comparable open-source MLLMs perceive COCO-train-OOD data as semantically OOD during training. These open-source MLLMs encompass diverse families, parameter scales, architectures, and training strategies. Thus, this division can be robustly regarded as an ”out-of-distribution relative to training semantics” sample set at the population level. B.1.4 Distributional Equivalence Between COCO-Train-OOD and COCO-Val-OOD Let the random variable be: (X,C)∈×, split(X,C) ×C, split (5) where X denotes images and C denotes discrete categorical labels. We consider separately: • Ptrain(X,C)P_train(X,C): The joint distribution of OOD subsets divided according to OODBench rules on COCO-train. • Pval(X,C)P_val(X,C): The joint distribution of OOD subsets divided according to the same rules on COCO-val. Section B.1.3, based on the exact Clopper–Pearson interval and the Bayesian Beta(1,1) posterior interval, infers at the 95%95\% confidence level that at least approximately 89–90%89–90\% of open-source MLLMs will treat COCO-train-OOD data as OOD for training semantics. To extend this population-level OOD classification to COCO-val-OOD, it is necessary to verify further whether COCO-train-OOD and COCO-val-OOD belong to the same statistical distribution family. Definition of Distribution Equivalence. We adopt the kernel mean embedding framework and utilize Maximum Mean Discrepancy (MMD) to measure the distance between two joint distributions. For any image x, features are extracted using the self-supervised visual model DINOv2 (Oquab et al., 2024). ϕ(X)=ℝd. splitφ(X)=R^d. split (6) The reason for selecting DINOv2 is that its training process does not rely on COCO labels, making its feature space ”semantically neutral” and thus avoiding the introduction of unnecessary supervised bias. Category labels c∈Cc∈ C are represented using one-hot vectors ec∈ℝte_c ^t. We select a kernel function on the joint space =×Z=X×C: k((x,c),(x′,c′)). splitk((x,c),(x ,c )). split (7) The nature of the kernel guarantees the existence of a Hilbert space (RKHS) ℋH and a mapping: ∃ℋ,Φ:→ℋs.t.k(z,z′)=⟨Φ(z),Φ(z′)⟩ℋ. split∃\ H, :Z s.t. k(z,z )= (z), (z ) _H. split (8) where z=(x,c),z′=(x′,c′)z=(x,c),z =(x ,c ). For any distribution P over Z, we can define its mean embedding in ℋH: μP:=Z∼P[Φ()]∈ℋ. split _P:=E_Z P[ (Z)] . split (9) We hereby order: k((x,c),(x′,c′))=kv(ϕ(x),ϕ(x′))⋅[c=c′], splitk((x,c),(x ,c ))=k_v(φ(x),φ(x ))· 1[c=c ], split (10) where ϕ(x)∈ℝdφ(x) ^d represents image features extracted via self-supervised visual models (e.g., DINOv2), kvk_v denotes the RBF kernel applied to visual features, and [⋅] 1[·] is the indicator function. This design constructs a block-diagonal joint kernel in the joint space (X,C)(X,C), where kernel values between different categories are zero, making them mutually orthogonal in the RKHS. Meanwhile, similarity within the same category is entirely determined by the RBF kernel of the visual features. In other words, the label variable C participates in joint modeling through the discrete structure of ”intra-class/cross-class,” while the visual subspace provides the intra-class geometric structure. Based on the above joint kernel, the maximum mean displacement (MMD) between the joint distributions PtrainP_train and PvalP_val can be expressed as: MMD2(Ptrain,Pval)=‖μtrain−μval‖ℋ2, splitMMD^2(P_train,P_val)=\| _train- _val\|_H^2, split (11) where μtrain,μval∈ℋ _train, _val represent the kernel mean embeddings of Ptrain(X,C)P_train(X,C) and Pval(X,C)P_val(X,C) in the reproducing kernel Hilbert space ℋH, respectively. That is, μP=(X,C)∼P[Φ(X,C)] _P=E_(X,C) P[ (X,C)], where Φ is the feature map induced by the joint kernel k. Statistical Test for Distribution Equivalence (Two-Sided Test). We used the equivalence test (TOST) to verify the following hypothesis: • H0H_0: MDD2(Ptrain,Pval)≥τ.MDD^2(P_train,P_val)≥τ. (The difference is at least τ and cannot be considered equivalent) • H1H_1: MDD2(Ptrain,Pval)<τ.MDD^2(P_train,P_val)<τ. (The difference is less than τ and can be considered equivalent) To determine the value of τ, we construct a ’homogeneous baseline’ within COCO-train-OOD as follows: • Randomly partition COCO-train-OOD into two mutually exclusive subsets: (Dtrain(1),Dtrain(2)).(D_train^(1),D_train^(2)). • Calculate the MMD2MMD^2 between them: MMD2(Dtrain(1),Dtrain(2)).MMD^2(D_train^(1),D_train^(2)). • Repeat multiple times to obtain a set of MMD2MMD^2 values within the same distribution. • Take the 95th95th percentile of this distribution as τ: τ=Quantile0.95MMD2(Dtrain(1),Dtrain(2)).τ=Quantile_0.95\MMD^2(D_train^(1),D_train^(2))\. τ denotes the upper bound of the finite-sample fluctuations of MMD2MMD^2 observed between two independent samples originating from the same underlying distribution. Consequently, if the MMD2MMD^2 between COCO-train-OOD and COCO-val-OOD falls below τ, the two partitions can be statistically regarded as independent realizations of a common source distribution, i.e., as distributionally isomorphic. To further quantify the uncertainty in MMD2MMD^2 estimation, we employ a permutation test to construct a reference distribution for its null hypothesis. Specifically, we merge all samples from COCO-train-OOD and COCO-val-OOD into a single dataset, randomly permute their group labels (train/val), and recalculate MMD2MMD^2 based on the permuted grouping structure. Repeating this process B times (B=500 in this experiment) yields the empirical permutation distribution of MMD2MMD^2 under the null hypothesis H0H_0 (equivalence of the two distributions): perm=MMDperm2(b)(b=1)B. splitD_perm=\MMD_perm^2(b)\_(b=1)^B. split (12) Thus, we construct an upper bound for the MMD2MMD^2 by taking the 1−α1-α-quantile of this distribution: UCI(1−α)=Quantile(1−α)(perm). splitUCI_(1-α)=Quantile_(1-α)(D_perm). split (13) Under the Two One-Sided Tests (TOST) procedure, if the following condition is satisfied: UCI0.95(MMD2(Ptrain,Pval))<τ, splitUCI_0.95(MMD^2(P_train,P_val))<τ, split (14) then it can be asserted: PtrainOOD≈PvalOOD, splitP_train^OOD≈ P_val^OOD, split (15) then, at significance level α, the null hypothesis H0H_0 that the difference is at least τ can be rejected, and the equivalence hypothesis H1H_1 can be accepted, indicating that the two joint distributions are statistically equivalent within the prescribed tolerance. Observation Conclusion: Based on the homogenous baseline constructed from COCO-train-OOD internal partition, the upper bound of MMD2MMD^2’s 95%95\% co-distribution fluctuation is determined to be: τ=4.47×10−4.τ=4.47× 10^-4. After estimating the joint distribution of COCO-train-OOD and COCO-val-OOD, we further obtained the 95%95\% confidence interval for MMD2MMD^2 via bootstrap sampling: CI95(MMD2)=[1.84×10−4,4.21×10−4],CI_95(MMD^2)=[1.84× 10^-4,4.21× 10^-4], Its upper confidence bound is significantly lower than the homogeneity threshold τ. Simultaneously, the TOST equivalence test constructed based on the empirical null distribution from B=500 permutations also indicates that the 95%95\% upper confidence bound of MMD2MMD^2 satisfies: UCI0.95(MMD2(Ptrain,Pval))<τ,UCI_0.95(MMD^2(P_train,P_val))<τ, At the significance level α=0.05α=0.05, we reject the null hypothesis H0H_0 —that the difference is at least τ —and accept the equivalence hypothesis H1H_1. This confirms that COCO-train-OOD and COCO-val-OOD are statistically distributionally equivalent within the prescribed tolerance. Since a model’s OOD response is fundamentally determined by the degree to which the test data deviates from its training semantic distribution, and train-OOD has already been shown in the previous section to constitute semantic OOD for open-source MLLMs, the statistical indistinguishability between val-OOD and train-OOD indicates that both deviate from the training distribution to the same order of magnitude. Consequently, COCO-val-OOD should exhibit the same OOD behavior for open-source MLLMs and can thus be regarded as semantic OOD data equivalent to train-OOD. B.2 OOD Evidence from Closed-Source Models Problem Setup and Assumptions. Let • DIDD_ID: Conventional ID test set. • DOODD_OOD: A semantic OOD test set constructed according to OODBench semantic rules (non-main semantic objects, semantic variants, and anomalous image–category pairings), with its image subset sourced from COCO-val. In the open-source model section, we have established the following two key premises: 1. Training Out-of-Distribution Properties. By calculating the semantic overlap rate between the training corpora of open-source MLLMs and COCO-train-OOD, and combining precise binomial intervals with Bayesian credible intervals using uniform priors, we obtain the following: At the 95%95\% confidence level, at least 90%90\% of open-source models exhibit an overlap rate with COCO-train-OOD not exceeding 5%5\%. This result indicates that, from the perspective of training distribution, COCO-train-OOD can be robustly regarded as OOD data for the collection of open-source models. 2. Distributional Equivalence Between COCO-train-OOD and COCO-val-OOD. Under the same semantic division rules, we construct COCO-train-OOD and COCO-val-OOD and perform an equivalence test in the joint space of image features and category labels using kernel MMD combined with the TOST procedure. The results indicate that the two subsets are statistically indistinguishable and can be regarded as two independent samples drawn from the same underlying distribution, i.e., they exhibit distributional equivalence. Therefore, in subsequent reasoning, it can be directly treated as a known premise: • For the open-source model collection ℳopenM_open, DOODD_OOD is OOD in terms of the training distribution. For closed-source models (such as GPT-4o, Gemini, etc.), since their training data is inaccessible, we cannot directly determine whether DOODD_OOD falls outside their training distribution. To address this, we employ behavioral consistency inference: • First, inductively derive observable ”open-source model OOD behavioral patterns”— i.e., OOD behavioral signatures—from the inference results of open-source models on DOODD_OOD. • Subsequently, verify whether the behavior of closed-source models on the same data DOODD_OOD aligns with these known ”open-source OOD behavioral patterns”. Suppose its behavior aligns with the OOD behavioral signature of the open-source model. In that case, it can be inferred that the closed-source model’s response pattern on DOODD_OOD corresponds to its performance on typical OOD data. This indicates that it exhibits behavioral characteristics consistent with OOD scenarios on that dataset. B.2.1 Behavioral Statistics and Performance Degradation For any model m (either open-source or closed-source) and for any data split S∈ID,OODS∈\ID,OOD\, we define the basic classification metrics as follows: T(m,S)(k)∈ACC, F1, Precision, Recall, MCC,k=1,⋯,K, splitT_(m,S)^(k)∈\ACC, F1, Precision, Recall, MCC\, k=1,·s,K, split (16) For example, T(m,S)(1)=ACCm,ST_(m,S)^(1)=ACC_m,S andT(m,S)(2)=F1m,ST_(m,S)^(2)=F1_m,S correspond to five performance statistics respectively. For each metric dimension k, we define the ID → OOD performance degradation measure: Δm(k):=Tm,ID(k)−Tm,OOD(k). split _m^(k):=T_m,ID^(k)-T_m,OOD^(k). split (17) Intuitively, Δm(k)>0 _m^(k)>0 indicates that the model’s performance degrades on semantically OOD data; the larger the degradation, the more pronounced the OOD behavior in this dimension. In practice, we perform bootstrap resampling on the model’s outputs across each data division to obtain the 95%95\% confidence interval for Δm(k) _m^(k): CI95(Δm(k))=[Lm(k),Um(k)]. splitCI_95( _m^(k))=[L_m^(k),U_m^(k)]. split (18) As demonstrated by the aforementioned distribution homogeneity analysis, for the open-source model ensemble, DOODD_OOD is indeed OOD on the training distribution. We can further abstract a set of observable OOD behavioral patterns from its performance on DOODD_OOD, namely the OOD behavioral signatures of open-source models. Let the open-source model collection be denoted as ℳopenM_open. For each metric dimension k, we define a unified degradation band from the confidence intervals [Lm(k),Um(k)][L_m^(k),U_m^(k)] of all open-source models: I(k)gap:=[minm∈ℳopen⁡Lm(k),maxm∈ℳopen⁡Um(k)]. splitI_(k)^gap:=[ _m _openL_m^(k), _m _openU_m^(k)]. split (19) This interval I(k)gapI_(k)^gap represents the typical performance degradation range exhibited by the open-source model on the kkth performance metric dimension for DOODD_OOD instances that have been confirmed as OOD. Let the set of degeneracy bands for all metric dimensions be denoted as: A:=I(k)gap|k=1,⋯,K. splitA:=\I_(k)^gap|k=1,·s,K\. split (20) We refer to A as the behavioral signature of the open-source model when processing semantic OOD inputs. It reflects the systematic decline pattern in the model’s performance from ID to OOD when confronted with semantic OOD. It is crucial to emphasize that Signature A is not the definition of OOD itself; rather, the definition of OOD stems from data construction and training distribution analysis. Signature A is merely an observable ”behavioral fingerprint” of OOD’s effect on model behavior. B.2.2 Behavioral Verification of Closed-Source Models For closed-source model c, we cannot observe its training data. Therefore, we employ behavioral-side statistical tests to address two questions: 1. Does the closed-source model exhibit significant ID → OOD degradation on DOODD_OOD? 2. Does the degradation magnitude of the closed-source model align with the OOD behavioral signature A of the open-source model? (1) Significance Test of Performance Differences Between Closed-Source Models in ID vs OOD. For each metric dimension k, we define the same degeneracy measure for closed-source models: Δc,obs(k):=Tc,ID(k)−Tc,OOD(k), split _c,obs^(k):=T_c,ID^(k)-T_c,OOD^(k), split (21) where Δc,obs(k) _c,obs^(k) denotes the observed degradation amount calculated based on the valid ID/OOD classification. To verify whether this represents OOD behavior rather than random fluctuations, we construct the following hypothesis test for each k: • Null hypothesis H0H_0: The closed-source model exhibits no performance difference between ID and OOD: H0:Tc,ID(k)=Tc,OOD(k).H_0:T_c,ID^(k)=T_c,OOD^(k). • Alternative hypothesis H1H_1: The closed-source model performs worse on OOD (degradation exists): H1:Tc,ID(k)>Tc,OOD(k).H_1:T_c,ID^(k)>T_c,OOD^(k). We achieve this through a permutation test of ID and OOD sample labels. Under the null hypothesis H0H_0, ID/OOD labels should be interchangeable, yielding the null distribution of the degradation amount Δc(k) _c^(k). If the observed Δc(k) _c^(k) falls in the extreme tail of this distribution (e.g., beyond the top 5%5\%), reject H0H_0 and conclude that degradation is significant. Therefore, we randomly permute the ID/OOD labels of the original data and recalculate the degradation amount for each permutation: δb(k),b=1,⋯,B, _b^(k), b=1,·s,B, where B represents the number of samples. These constitute the empirical distribution under the null hypothesis: Δb(k)b=1)B.\ _b^(k)\_b=1)^B. The standard Monte-Carlo p-value correction formula for permutation tests calculated via this empirical distribution is: pc(k)=PH0(Δ(k)≥Δc,obs(k))≈1+∑b=1B(Δb(k)≥Δc,obs(k))B+1, splitp_c^(k)=P_H_0( ^(k)≥ _c,obs^(k))≈ 1+ _b=1^B 1( _b^(k)≥ _c,obs^(k))B+1, split (22) when pc(k)<0.05p_c^(k)<0.05, we reject H0H_0 as statistically significant. That is, the closed-source model exhibits significant ID → OOD performance degradation on DOODD_OOD, which is a typical behavioral signal when semantic OOD is applied to the model. Figure 8: This figure illustrates the distribution of degradation amounts under the null hypothesis H0H_0 (no performance difference between ID and OOD) and the positions of the observed values. As illustrated, for both closed-source models and across all metrics, the observed degradation falls in the extreme tail of the permutation distribution. The permutation p-values are all below 10−310^-3, and out of 2000 permutations, none produced a degradation larger than the observed value. Observation Conclusions: For closed-source models (GPT-4o and Gemini), we calculated the observed measure of ID → OOD performance degradation Δc,obs(k) _c,obs^(k) across five performance dimensions (Accuracy, F1, Precision, Recall, MCC). We constructed the corresponding null hypothesis distribution based on B=2000B=2000 label permutations. Figure 8 presents violin plots illustrating the distribution of degradation amounts and the positions of observed values for each metric under the null hypothesis H0H_0 (no performance difference between ID and OOD). It is evident that for all metrics and both closed-source models, the observed degradation consistently resides in the extreme tail of the permutation distribution: their permutation p-values are all below 10−310^-3, and across all 2000 permutations, not a single permutation resulted in degradation exceeding the actual observed values. From a statistical inference perspective, this implies: at the commonly used significance level α=0.05α=0.05, the stricter α=0.01α=0.01, and even α=0.001α=0.001, we have sufficient evidence to reject the null hypothesis H0:Tc,ID(k)=Tc,OOD(k)H_0:T_c,ID^(k)=T_c,OOD^(k) and accept the one-tailed alternative hypothesis H0:Tc,ID(k)>Tc,OOD(k)H_0:T_c,ID^(k)>T_c,OOD^(k). In other words, under the current data and testing framework, the closed-source model exhibits statistically significant performance differences between the ID subset and the OOD-labeled subset, and this difference is highly unlikely to be caused by random label perturbations. (2) Consistency of OOD Behavioral Signatures Between Closed-Source and Open-Source Models. We aggregate the bootstrapped degradation values of the open-source models to obtain population-level statistics: • Open-source mean degradation: μopen(k) _open^(k). • Open-source variability (natural fluctuation scale): σopen(k) _open^(k). where μopen(k) _open^(k) expresses the typical degradation magnitude of open-source models on semantic OOD; σopen(k) _open^(k) represents the natural fluctuation scale of degradation within the open-source community (model variance + sample noise). Furthermore, we define this natural fluctuation scale as the tolerance threshold: ηk:=σopen(k). _k:= _open^(k). Suppose the performance degradation of closed-source models falls within the same order of magnitude as the average degradation of open-source models. In that case, we consider their behavior equivalent in this metric dimension. Comparing the difference between closed-source and open-source models: δ(k):=Δc(k)−μopen(k). splitδ^(k):= _c^(k)- _open^(k). split (23) • If δ(k)=0δ^(k)=0: closed-source degradation = open-source average degradation; • If |δ(k)||δ^(k)| is small (e.g., within ±ηk± _k): closed-source degradation and open-source degradation are ”of the same order of magnitude”; • If |δ(k)||δ^(k)| is much larger than ηk _k, closed-source behavior and open-source OOD behavior are significantly inconsistent. To obtain the distribution of δ(k)δ^(k) and its 90%90\% confidence interval, we perform B bootstrap resamples on both ID and OOD samples. In each resample, we simultaneously estimate the following quantities: • Degradation of the closed-source model: Δc,b(k) _c,b^(k), denoting the performance degradation from ID to OOD for the closed-source model in the bbth resample; • The average degradation of the open-source model population: μopen,b(k) _open,b^(k), representing the average degradation of the open-source model population during the bbth resampling (averaged across the two open-source models). The difference between the two is defined as: δb(k):=Δc,b(k)−μopen,b(k),b=1,⋯,B. splitδ^(k)_b:= _c,b^(k)- _open,b^(k), b=1,·s,B. split (24) Thus we obtain a set δb(k)b=1B\δ^(k)_b\_b=1^B, approximating the sampling distribution of δ(k)δ^(k). Equivalence test hypotheses: • Null hypothesis H0H_0 (non-equivalence): H0:|δ(k)|≥ηkH_0:|δ^(k)|≥ _k i.e., the difference between closed-source degradation and open-source degradation on metric k is at least ηk _k, indicating behavioral non-equivalence. • Alternative hypothesis H1H_1 (equivalence): H1:|δ(k)|<ηkH_1:|δ^(k)|< _k i.e., the difference is constrained within the interval (−ηk,ηk)(- _k, _k), which can be regarded as equivalent. In practice, we construct a confidence interval using δb(k)\δ^(k)_b\ and employ CI to implement the TOST decision. We adopt the standard equivalent test framework, using a 90%90\% confidence interval at a significance level of α=0.05α=0.05. Specifically, we extract the 5th and 95th percentiles from δb(k)\δ^(k)_b\ to obtain the 90%90\% CI: CI90(δ(k))=[Lδ(k),Uδ(k)]. splitCI_90(δ^(k))=[L_δ^(k),U_δ^(k)]. split (25) Adopting the TOST equivalence criterion, if: CI90(δ(k))⊂(−ηk,ηk), splitCI_90(δ^(k))⊂(- _k, _k), split (26) At the α=0.05α=0.05 significance level, we reject the null hypothesis H0:|δ(k)|≥ηkH_0:|δ^(k)|≥ _k and accept H1:|δ(k)|<ηkH_1:|δ^(k)|< _k. This indicates that: The difference between closed-source degradation and open-source average degradation is constrained within the ”open-source natural fluctuation scale σopen(k) _open^(k)” in dimension k, rendering it equivalent to open-source OOD behavior. Figure 9: We selected InternVL2-8B and InternVL2.5-8B as open-source reference models and constructed their behavioral signatures based on bootstrap estimates of performance degradation. We then applied the TOST equivalence testing framework to assess the behavioral consistency of closed-source models (GPT-4o and Gemini) under semantic OOD conditions. The bootstrap distributions of δ(k)δ^(k) for the five evaluation metrics (Accuracy, F1, Precision, Recall, MCC), along with their 90%90\% confidence intervals (CI90) and the corresponding tolerance thresholds ηk _k. Observational Conclusion: In this study, we selected InternVL2-8B and InternVL2.5-8B as open-source reference models. Based on their bootstrap degeneracy estimates, we constructed open-source behavioral signatures. We then employed the TOST equivalence testing framework to evaluate the behavioral consistency of closed-source models (GPT-4o and Gemini) on semantic OOD data. Figure 9 displays the δ(k)δ^(k) bootstrap distributions of closed-source models across five metrics (Accuracy, F1, Precision, Recall, MCC), their 90%90\% confidence intervals (CI90CI_90), and corresponding tolerance thresholds (ηk _k). The result from GPT-4o is: Accuracy: CI90=[4.67%, 6.09%], _90=[67\%,609\%], ηk=7.04% _k=04\% F1: CI90=[0.68%, 2.19%], _90=[68\%,219\%], ηk=7.26% _k=26\% Precision: CI90=[7.69%, 9.58%], _90=[69\%,958\%], ηk=9.62% _k=62\% Recall: CI90=[−7.51%,−5.68%], _90=[-51\%,\,-68\%], ηk=7.65% _k=65\% MCC: CI90=[8.74%, 11.55%], _90=[74\%,1155\%], ηk=14.20% _k=420\% As shown, the 90%90\% confidence intervals for all five metrics fall within the corresponding tolerance ranges (−ηk,ηk)(- _k, _k), i.e., Equ. 26, satisfying the equivalence criterion. In other words, across all performance dimensions, the ID → OOD degradation of GPT-4o fluctuates at the same order of magnitude as the average degradation of the open-source model population, without exceeding the natural variability exhibited by the open-source models. Gemini’s results also yielded highly consistent conclusions: Accuracy: CI90=[2.65%, 4.14%], _90=[65\%,414\%], ηk=7.03% _k=03\% F1: CI90=[0.28%, 1.89%], _90=[28\%,189\%], ηk=7.26% _k=26\% Precision: CI90=[3.80%, 5.84%], _90=[80\%,584\%], ηk=9.49% _k=49\% Recall: CI90=[−3.32%,−1.36%], _90=[-32\%,\,-36\%], ηk=7.68% _k=68\% MCC: CI90=[5.39%, 8.33%], _90=[39\%,833\%], ηk=14.17% _k=417\% Similarly, all metrics satisfy Equ. 26, indicating that Gemini’s degradation also falls stably within the natural fluctuation range of degradation observed in open-source communities. In summary, GPT-4o and Gemini are statistically equivalent to the open-source OOD behavioral signature across all five performance dimensions—Accuracy, F1, Precision, Recall, and MCC. Their ID → OOD degradation magnitudes remain within the same order of magnitude as those of the open-source models, exhibiting no deviations beyond the natural variability scale (σopen)( _open). These findings indicate that the closed-source models display OOD response patterns on our semantic OOD data that are consistent with those observed in the open-source models. From the two types of analyses above, we obtain the following conclusions. Combining (1) the significance tests on ID vs. OOD performance differences for closed-source models and (2) the consistency between closed-source behavior and the open-source OOD behavioral signature, we observe that: (1). Closed-source models exhibit statistically significant ID → OOD performance degradation on DOODD_OOD. (2). The magnitude of this degradation across all key metrics falls within the natural OOD variability range of the open-source models, consistent with the open-source OOD behavioral signature. Thus, on the same semantic OOD dataset, the ID → OOD degradation patterns of closed-source models—both in direction and magnitude—align with those of open-source models on known OOD tests. This supports the view that closed-source models treat DOODD_OOD as OOD data in a manner equivalent to open-source models. Appendix C Differences between OODBench and Hard Sample Hard samples refer to data instances that exhibit high uncertainty, low confidence, or are prone to misclassification during model training or inference. Although they originate from the training distribution (in-distribution, ID), they typically reside in the tail or low-density regions — the marginal distribution portion. Typical hard samples include: blurred, occluded, or low-resolution images; instances with ambiguous semantic boundaries or highly similar categories (e.g., dogs and wolves); and minority class samples under long-tail distributions. They essentially represent “challenging points” within the distribution, often amenable to fitting through enhanced generalization capabilities, sample reweighting, or increased sample size. In contrast, this paper refines the definition of out-of-distribution (OOD) samples from a human perception perspective, categorizing them into two types: (1) objects in an image that are non-main objects and semantically unrelated to the main object; and (2) variants or abnormal forms of the target object. These samples fundamentally represent “semantic OOD” data that multimodal large models struggle to cover during semantic learning. To investigate the distinction between OODBench and hard samples, we drew inspiration from the OHEM method (Shrivastava et al., 2016) to construct a set of hard samples on the COCO dataset and evaluated them using nine state-of-the-art models. By comparing the performance (i.e., behavioral patterns) of different models on hard samples versus OODBench, we aim to reveal the differences between the two. In the data partitioning process, we adopted an offline hard sample mining approach, with the specific algorithm detailed in Algorithm 1. First, a sufficiently converged detector is selected to define sample difficulty (YOLOv8-xlarge (Ultralytics, 2023) was used as the detector in this study). Perform inference on the input image to generate a set of candidate boxes (proposals). For each candidate box, compute the prediction distribution p(c∣x)p(c x), where c∈1,…,Cc∈\1,…,C\, and determine the predicted class c^=argmax⁡p(c∣x) c= *arg\,maxp(c x). Subsequently, we computed a hardness score for each sample, using the commonly employed Focal Loss (Lin et al., 2017) value as the metric: h(x)=(1−p(c^∣x))γ⋅(−log⁡p(c^∣x)),γ=2 splith(x)=(1-p( c x))^γ·(- p( c x)), γ=2 split (27) among these, scores for easily classifiable samples are significantly suppressed, retaining only hard samples. Next, for all proposals within each image, we group them by predicted category and select the top-k samples with the highest hardness within each category to add to the candidate hard sample list. After traversing the entire dataset, the top-k hard samples from each category within every image are aggregated. Finally, the hard sample list is globally sorted by hardness score, and the top-q% samples from each category are selected to form the final hard sample set. Algorithm 1 Hard Sample Mining via Focal Loss Scoring Input: Converged detector D, Image set ℐI, Focal loss parameter γ, Top-k per class, Selection threshold q%q\%. Output: Hard sample list ℋH Initialize empty list ℋ←∅H← for each image x∈ℐx do Generate proposal set (x)P(x) using D for each proposal p∈(x)p (x) do Obtain class probability distribution p(c|x),c∈1,…,Cp(c|x),c∈\1,…,C\ Predicted class c^←arg⁡maxc⁡p(c|x) c← _cp(c|x) Compute hardness score: h(x)←(1−p(c^|x))γ⋅(−log⁡p(c^|x))h(x)←(1-p( c|x))^γ·(- p( c|x)) end for Group all proposals by predicted class c For each class c, select top-k proposals with highest h(x)h(x) ℋ←ℋ∪selected proposalsH ∪\selected proposals\ end for Sort ℋH by hardness score h(x)h(x) in descending order For each class c, retain top-q%q\% proposals as final hard samples return ℋH Model Data Type COCO-OOD-Hard Sample Performance Num Accuracy(%\%) F1(%\%) Precision(%\%) Recall(%\%) MCC(%\% Random Chance - - 50.00%50.00\% 50.00%50.00\% 50.00%50.00\% 50.00%50.00\% 0.00%0.00\% Open-source Models ID 6000 89.90 90.17 87.83 92.63 79.92 OOD-H 6000 74.45 69.53 86.12 58.30 51.67 LLaVA-NeXT-8B Hard Sample 5282 56.74 30.27 77.99 18.78 20.71 ID 6000 92.65 92.84 90.48 95.33 85.42 OOD-H 6000 76.57 74.09 82.85 67.00 54.13 DeepSeek-VL-7B-Chat Hard Sample 5282 62.23 47.38 78.09 34.00 29.63 ID 6000 92.83 92.85 92.69 93.00 85.67 OOD-H 6000 65.18 64.92 65.41 64.43 30.37 InternVL2-8B Hard Sample 5282 85.12 84.29 89.28 79.82 70.64 ID 6000 97.02 97.01 97.25 96.77 94.03 OOD-H 6000 79.70 78.65 82.95 74.77 59.69 InternVL2.5-8B Hard Sample 5282 85.38 84.23 91.48 78.04 71.54 ID 6000 97.02 97.04 96.14 97.97 94.05 OOD-H 6000 82.02 82.91 78.99 87.23 64.38 Llama-3.2-11B-Vision-Instruct Hard Sample 5282 72.09 71.11 73.71 68.69 44.29 ID 6000 89.48 90.11 85.01 95.87 79.62 OOD-H 6000 77.67 75.48 83.66 68.77 56.23 Qwen2-VL-7B-Instruct Hard Sample 5284 72.88 66.56 86.79 53.97 49.43 ID 6000 95.92 95.79 98.93 92.83 92.01 OOD-H 6000 78.28 74.04 92.03 61.93 59.86 Qwen2.5-VL-7B-Instruct Hard Sample 5282 64.27 47.69 89.03 32.56 36.93 Close-source Models ID 5996 91.86 92.06 89.84 94.40 83.83 OOD-H 5994 66.80 68.46 65.20 72.07 33.79 Gemini Hard Sample 5276 85.90 85.91 85.84 85.97 71.80 ID 5998 93.33 93.45 91.82 95.13 86.72 OOD-H 6000 70.53 72.06 68.51 76.00 41.31 GPT-4o Hard Sample 5282 82.49 81.94 84.60 79.44 65.10 Table 7: This table compares the performance of ID, OOD-H data, and in-Hard samples—segmented based on the COCO dataset—across nine leading SOTA MLLMs. We ensured that all testing configurations remained identical except for the data division method, guaranteeing that any observed differences stem solely from the inherent characteristics of the data. Figure 10: The figure displays the histogram of accuracy difference distributions between ID data and Hard samples, as well as OOD-H data on the COCO dataset. The bars are sorted in ascending order based on the Hard sample difference, with an exponential trendline fitted to characterize the overall trend. Table 7 demonstrates the performance of ID data, OODBench data, and hard samples across nine state-of-the-art multimodal large language models (MLLMs). As shown in Figure 10, we plotted the performance differences of each model on hard samples and OOD-H compared to in-distribution data, then sorted the results in ascending order based on the magnitude of performance decline on hard samples. Analysis reveals significant variations in the challenge posed by hard samples across models: for instance, Gemini’s performance declined by only 5.96%, while Qwen2.5-VL-Instruct experienced a substantial 33.62% drop. This indicates that hard samples remain in-distribution data, with performance degradation primarily dependent on a model’s robustness and generalization capabilities. Specifically, certain models exhibited pronounced performance degradation on specific hard samples (e.g., DeepSeek-VL-7B-Chat, Llama-3.2-11B-Vision-Instruct, and Qwen2.5-VL-7B-Instruct, all of which exceeded a 30% decline), while others remained largely unaffected (e.g., Gemini and InternVL2-8B). In contrast, OODBench-defined OOD samples reveal a consistent performance decline across all nine models, with even robust models failing to significantly mitigate this effect. For example, GPT-4o experienced a 27.65% performance drop on OOD samples. This result indicates that OOD samples exhibit a distribution shift property consistent across models, rather than relying solely on individual model robustness. C.1 Statistical distinction between Hard and OOD Setup. For each model m=1,…,Mm=1,…,M (here M=9M=9), we compute the performance degradation on hard samples ΔHard(m) _Hard(m) and the performance degradation on our dataset ΔOOD(m) _OOD(m). We analyze the following aspects: (i) variability across models, (i) alignment of performance degradation patterns, and (i) overlap rate between data partitioned by OODBench and OHEM. Evidence 1 — Variance Test on Full Sets (Hard vs OOD) We compare the variance across models using an F-test: F=Var(ΔHard)Var(ΔOOD)∼F(ν1,ν2),ν1=ν2=M−1 splitF= Var( _Hard)Var( _OOD) F( _1, _2), _1= _2=M-1 split (28) In our experiments, F(8,8)=4.59F(8,8)=4.59, p=0.0227p=0.0227, indicating that the variance in performance degradation caused by Hard samples is significantly greater than that caused by OOD samples. Explanation: Hard samples exhibit greater model dependency, while OOD samples induce a more consistent (model-independent) performance degradation. Evidence 2 — Correlation on full sets (Hard vs OOD) We measure the alignment of performance degradation patterns through correlation analysis: r=corr(ΔHard,ΔOOD) splitr=corr( _Hard, _OOD) split (29) We observe negative correlations (Pearson r = -0.65, p = 0.059; Spearman ρ = -0.58, p = 0.099). These results indicate inconsistencies in the failure patterns of models on Hard and OOD samples: models exhibiting significant performance degradation on Hard samples do not necessarily show similar degradation on OOD samples, and vice versa. Furthermore, from the performance trend lines shown in Figure 10, we observe an X-shaped decline pattern, where models exhibit significant fluctuations or reversals at certain points. This phenomenon aligns with the correlation test results, further validating the misaligned patterns of performance shifts across Hard and OOD samples. The X-shaped trend lines reflect the models’ markedly divergent responses to different sample types (Hard vs. OOD). Hypotheses • H_0 (Hard Hypothesis): The OODBench dataset shares the same distribution as the in-distribution Hard subset. In other words, the statistical characteristics of its cross-model performance degradation vector ΔV V (including variance and correlation/alignment with the baseline Hard) should fall within the “empirical distribution of the Hard subset.” • H_1 (OOD Hypothesis): OODBench does not belong to the Hard subset, and the same-distribution assumption does not hold; its statistical characteristics will significantly deviate from the “Hard baseline.” Test statistics. Record the cross-model performance decline vector (ID performance - Hard sample performance) for the baseline Hard model as ΔHard _Hard (length equal to the number of models M=9M=9). Resample details. We performed resampling based on Algorithm 1 to construct the empirical baseline for the Hard subset. Specifically, B=1000 independent samples were conducted, each randomly selecting 500 Hard samples from the COCO validation pool. All samples were selected according to a unified criterion: scoring using focal loss and choosing the top-k samples within each category. Each sampling process was independent, yielding distinct sets of 1000 Hard samples, thereby ensuring the randomness and diversity of the baseline distribution. Evidence 3 — Empirical variance test • Statistical Measure: sOOD2=Var(ΔOOD)s^2_OOD=Var( _OOD) • Test Direction (one side): left tail. Hard samples are typically more model-dependent, exhibiting greater cross-model variance; if sOOD2s^2_OOD is significantly smaller, reject H_0. • Baseline Construction: Resample H(b)H^(b) from the COCO validation set using the same mechanism as Algorithm 1’s rule. • Baseline Distribution: Sb=Var(ΔH(b))S_b=Var( _H^(b)) • Decision: If the empirical p-value p^emp<0.05 p_emp<0.05 (i.e., OOD falls below the 5th percentile of the Hard baseline distribution), reject H_0, concluding that the candidate set exhibits greater model-agnostic consistency in cross-model score drops, consistent with OOD characteristics. Results are shown in Figure 11 (left). The empirical p-value is less than 1/(B+1) (i.e., p^emp<0.001 p_emp<0.001), strongly rejecting H_0. This indicates that the statistical characteristics of the candidate set significantly deviate from the Hard baseline. Evidence 4 — Empirical correlation test • Statistical Measure: rOOD=corr(ΔHard,ΔOOD)r_OOD=corr( _Hard, _OOD) • Test Direction (one side): left tail. Hard vs Hard drop-off modes are typically non-negative or positively aligned; if rOODr_OOD is significantly low, reject H_0. • Baseline Construction: Resample B hard subsets H(b)H^(b) from the COCO validation set and compute their correlation with the baseline hard set. • Baseline Distribution: rb=corr(ΔHard,ΔOOD)r_b=corr( _Hard, _OOD) • Decision: If the empirical p-value p^emp<0.05 p_emp<0.05 (i.e., rOODr_OOD falls below the 5th percentile of the Hard baseline distribution), reject H_0, indicating that the candidate set does not align with Hard’s failure pattern, consistent with OOD’s ”different failure mode.” Results are shown in Figure 11: The middle subfigure displays the distribution of Pearson correlation coefficients from 1000 resamples, all exceeding 0.9; The right subfigure shows the distribution of Spearman correlation coefficients, all exceeding 0.8. In contrast, the OOD correlations with Hard are -0.65 (Pearson) and -0.58 (Spearman), significantly below the Hard baseline distribution. The empirical p-value is less than 0.001, strongly rejecting H_0 and further demonstrating that the statistical characteristics of the candidate set significantly deviate from the Hard baseline. Figure 11: The figure displays the empirical distribution results for variance (left), Pearson correlation coefficient (center), and Spearman correlation coefficient (right). The x-axis of the variance subplot represents variance values, while the x-axes of the Pearson and Spearman subplots respectively denote the corresponding correlation coefficients. The y-axes of all three subplots indicate sample sizes. The red dashed lines in each subplot mark the statistical values of the out-of-distribution (OOD) data. Evidence 5 — Overlap Rate Between Hard Sample and OODBench • Statistical Measure: We calculated the overlap rate between the full COCO-H dataset (5,079 samples) in OODBench and the Hard sample collection (2,641 samples) obtained by sampling according to Algorithm 1. This metric evaluates the consistency between our classified OOD data and the hard samples generated by conventional hard sample mining methods. • Results: Statistical analysis revealed only 206 overlapping samples, accounting for 0.0406% of the total COCO-H dataset and 0.0780% of the total hard samples. • Conclusion: Such a low overlap rate indicates that the OOD data categorized by OODBench fundamentally differs from the hard samples obtained through traditional hard sample mining methods. In other words, OODBench captures more universally applicable out-of-distribution patterns rather than conventional in-distribution hard samples. Taken together, the five lines of evidence consistently demonstrate that the OODBench data cannot be regarded as conventional hard samples. Evidence 1 (variance test on full sets) shows that performance degradation on Hard samples exhibits significantly larger variance across models than on OOD, indicating stronger model dependency for Hard. Evidence 2 (correlation analysis) reveals negative associations between Hard and OOD drop patterns, further corroborated by the X-shaped trend lines, highlighting misaligned failure modes. Evidence 3 (empirical variance test) confirms that the variance of OOD drops is significantly smaller than the empirical Hard baseline, rejecting H_0. Evidence 4 (empirical correlation test) further establishes that OOD correlations with Hard (-0.65, -0.58) fall far below the Hard-vs-Hard baseline (all >0.9>0.9 Pearson, >0.8>0.8 Spearman), again strongly rejecting H_0. Finally, Evidence 5 (overlap rate) shows that the intersection between OODBench samples and mined hard samples is minimal (206 out of 5,079), underscoring their fundamental distinction. Across all five evidences, the results are significant and directionally consistent: OODBench captures distributional shifts and model-agnostic failure modes, whereas Hard samples remain in-distribution and model-dependent. Therefore, our OOD data are not merely hard samples but instead represent a fundamentally different category aligned with out-of-distribution characteristics. Appendix D Differences between OODBench and Hallucination Out-of-Distribution (OOD) may cause hallucinations, but hallucinations are not necessarily caused by OOD. We analyse in depth the differences between OOD and hallucinations in terms of concepts, interrelationships and experimental results to demonstrate the distinctiveness of the both. Hallucination definitions in the Large Vision- Language Model (LVLM): Hallucination in LVLM denotes a disagreement between factual content of images and corresponding generated textual content, akin to the solely textual hallucination encountered in large language models (Liu et al., 2024a; Bai et al., 2024). Definition of OOD: Traditionally, OOD is defined as a test example originating from a distribution different from the training data (Hendrycks and Gimpel, 2017). In recent studies (Averly and Chao, 2023; Yang et al., 2024b), the OOD detector identifies misclassified examples labelled as OOD Data to improve the safety and reliability of deep learning models in practical applications. In LVLMs, hallucination refers to the inconsistency between the generated text description and the visual content. Our experiment shows that OOD data does not necessarily lead to hallucinations and that it is not fully equivalent to hallucinations. As shown in Tab. 1, OOD data does not necessarily trigger hallucinations. For example, on the OOD-H data, the accuracy of GPT-4o was 65.13%, implying that 100% - 65.13% = 34.87% of the OOD data led to hallucinations, while the remaining 65.13% of the OOD data did not trigger hallucinations. This indicates that there is not a one-to-one correspondence between OOD and hallucinations. Second, the causes of hallucinations are multiple, and not all hallucinations are caused by OOD data. For example, the ID data in Tab. 1 also triggered hallucinations. This phenomenon was also verified in the BAP experiments in Tab. 2. Taking GPT-4o as an example, we asked questions on ID data according to Basic-to-Advanced Progress, including existential description, counting description, and logical reasoning, and the performance was 94.84%, 48.26%, and 30.83% in decreasing order. This indicates that as the complexity of the inference task increases, the model becomes more hallucinatory and the match between the generated text and the visual content decreases. The results of the above experiments demonstrate that there is some intersection between OOD and hallucinations, but they are not equivalent. Prior research (Zhang et al., 2023) has shown that CoT (Chain of Thought) is an effective way to alleviate hallucinations. In our experiments, CoT did not perform as well on OOD data. For example, in Tab. 1, we observed that several models such as DeepSeek-VL-7B-Chat, DeepSeek-VL2-Small, Llama3.2-11B-Vision, Qwen2-7B-Instruct, and GPT-4o failed to improve their performance, but instead, their performance decreased to different degrees after using CoT. This phenomenon may stem from the fact that the distribution of the OOD data deviates from the training distribution, which makes CoT reason with out-of-distribution data as the starting point, thus reinforcing false assumptions, amplifying the inference bias, and ultimately reducing the model performance. We begin by hypothesising that there is a high degree of overlap between the data in OODBench and the hallucinations. Further, CoT, as a typical method to mitigate the hallucination problem, should improve the model performance on this benchmark. However, the experimental results were contrary to expectations: the model performance decreased rather than increased after the introduction of CoT. This phenomenon directly refutes our initial hypothesis that the OODBench data highly overlaps with hallucinations. Instead, it experimentally validates that there is an essential difference between our divided OOD dataset and hallucinations. Together, these experimental results demonstrate that the OOD dataset we constructed is significantly different from the hallucination. Appendix E Ablation Study of the Generalized OOD Detector and Hyperparameter T E.1 Ablation study of different types of generalized OOD Detector In order to explore the impact of different types of OOD detectors on the performance of downstream tasks when dividing OOD data under the generalized OOD detection framework, we designed and conducted a series of systematic ablation experiments. In the specific setup, we selected GroupViT (Xu et al., 2022) and LiT (Zhai et al., 2022) to gradually replace the original CLIP (Radford et al., 2021) with BLIP2 (Zhai et al., 2022) to re-divide the samples in the COCO dataset into OOD, and we constructed two new sets of division data. Subsequently, we input the three sets of delineation results (i.e., original division + division of the two alternative detectors) into two open-source multimodal models (InternVL2, Qwen2-VL) and two closed-source models (Gemini, GPT-4o), respectively, for consistency testing to assess the impact of different generalised OOD detectors on the downstream performance performance. It should be noted that in order to control the testing overhead, the original OODBench benchmark introduces a random downsampling strategy when the number of samples exceeds 6,000. In order to avoid the possible data distribution bias introduced by downsampling, we retain all the samples judged as OOD, i.e., we do not perform the downsampling operation again when we use GroupViT with LiT for reclassification. At the same time, in order to ensure the fairness of the experiment, we continue to use the sample sets corresponding to the CLIP and BLIP2 classification results in the initial benchmark to ensure the consistency of the processing flow among the three groups. For ease of presentation, we define the three datasets as follows: The data divided by CLIP and BLIP2 are recorded as Group I. The data divided by BLIP2 and GroupViT are recorded as Group I. The data divided by GroupViT and LiT are recorded as Group I. Model OOD Detectors COCO-OOD Performance Num Accuracy(%\%) F1(%\%) Precision(%\%) Recall(%\%) MCC(%\%) Time Random Chance - - 50.00%50.00\% 50.00%50.00\% 50.00%50.00\% 50.00%50.00\% 0.00%0.00\% - Open-source Models 10160 78.74 76.78 84.58 70.30 58.32 00:11:442^2 CLIP & BLIP2 Δ + 0.00 + 0.00 + 0.00 + 0.00 + 0.00 - 9988 79.85 78.06 85.65 71.71 60.50 00:21:401^1 BLIP2 & GroupViT Δ -1.11 -1.28 -1.07 -1.41 -2.18 - 12090 80.17 78.46 85.88 72.22 61.12 00:26:151^1 Qwen2-VL-7B-Instruct GroupViT & LiT Δ -1.43 -1.68 -1.30 -1.92 -2.80 - 10160 66.22 65.99 66.44 65.55 32.44 01:03:502^2 CLIP & BLIP2 Δ +0.00 +0.00 +0.00 +0.00 +0.00 - 9988 67.08 66.91 67.25 66.58 34.16 01:49:451^1 BLIP2 & GroupViT Δ -0.86 -0.92 -0.81 -1.03 -1.72 - 12090 68.17 67.91 68.48 67.34 36.35 02:13:101^1 InternVL2-8B GroupViT & LiT Δ -1.95 -1.92 -2.04 -1.79 -3.91 - 10150 67.02 68.71 65.37 72.41 34.25 02:59:57 CLIP & BLIP2 Δ +0.00 +0.00 +0.00 +0.00 +0.00 - 9972 66.65 68.36 65.02 72.06 33.49 03:28:29 BLIP2 & GroupViT Δ +0.37 +0.35 +0.35 +0.35 +0.76 - 12072 66.96 68.76 65.22 72.71 34.16 04:04:54 Gemini GroupViT & LiT Δ +0.06 -0.05 +0.15 -0.30 +0.09 - 10158 73.66 74.35 72.44 76.37 47.38 09:58:34 CLIP & BLIP2 Δ +0.00 +0.00 +0.00 +0.00 +0.00 - 9908 73.74 74.57 72.28 77.01 47.58 12:32:43 BLIP2 & GroupViT Δ -0.08 -0.22 +0.16 -0.64 -0.20 - 12090 75.86 76.56 74.39 78.86 51.81 08:41:31 GPT-4o 333There may be some differences between the performance results of GPT-4o in this experiment and those in Tab. 10, which mainly stems from the differences in the model versions used. To ensure the fairness and consistency of the results of the OOD detector type ablation experiments, we uniformly used the same version of GPT-4o (20240806) for evaluation in this experiment. GroupViT & LiT Δ -2.20 -2.21 -1.95 -2.49 -4.43 - Table 8: Impact of replacing generalized OOD detectors on the downstream performance of different multimodal models. We compare the accuracy differences (Δ ) of data divided by different combinations of detectors—Group I (CLIP + BLIP2), Group I (BLIP2 + GroupViT), and Group I (GroupViT + LiT)—across four representative VLMs (Qwen2-VL-7B-Instruct, InternVL2-8B, Gemini, and GPT-4o). Group I is used as the baseline, and the performance drop or gain of Group I and Group I is reported accordingly. The results show that although different detectors induce slight variance in OOD data division, their impact on downstream model performance is minimal, supporting the replaceability of generalized OOD detectors in the OODBench framework. We first evaluated Qwen2-VL by taking the performance of group I data as the baseline and calculating the accuracy difference (δ) between group I and group I data under the model relative to the baseline. The experimental results show that on Qwen2-VL, the accuracy of group I and group I data decreases by 1.11% and 1.43%, respectively, compared with that of group I. On Gemini, it improves by 0.86% and 0.06%, respectively, and the performance difference between the three groups of data on InternVL and GPT-4o is also minimal, and the details of the values are shown in Tab. 8. In the Time column, the time superscript indicates the number of graphics cards used in the test, and we used NVIDIA L40s graphics cards in the testing phase. These results show that although there is some discriminative difference between the different OOD detectors when dividing the samples, this difference does not trigger a significant performance bias in the evaluation of the downstream multimodal model. It is experimentally verified that the generalized OOD detectors in the OODBench division process are replaceable. In other words, as long as the detector itself has good class discrimination ability, it can effectively classify OOD data for large visual language models. In addition, we note that Tong et al. (Tong et al., 2024, 2023). raised the issue of CLIP failure transfer, suggesting that current large VLMs may inherit transfer flaws from adopting CLIP as a visual or text encoder. The results are in the Tab. 8 shows that the model’s performance in the OOD detection task remains stable even when the CLIP is replaced with other classifiers, ruling out the possibility of further failure transfer of the CLIP into the VLM in OOD detection. It is thus concluded that the performance bottleneck of VLM in such tasks does not originate from CLIP itself. This finding helps to rule out the hypothesis of further spreading of ‘CLIP failure transfer‘ in VLMs, suggesting that a combination of more factors influences the performance of VLMs in this kind of task. Figure 12: Impact of the number of generalized OOD detectors on the quality of OOD data division under the cross-validation mechanism. The horizontal axis indicates the number of detectors involved in the division process (ranging from 1 to 4), while the vertical axis shows the corresponding Accuracy and F1 score (%) obtained by evaluating the divided OOD data on multiple mainstream VLMs. The results demonstrate that using a single detector yields the highest performance but suffers from potential bias. Incorporating two detectors significantly reduces this bias, producing OOD data more aligned with the VLMs’ potential OOD distribution. However, increasing the number of detectors beyond two offers limited benefits and may introduce instability, as evidenced by performance rebound and eventual convergence. E.2 Ablation studies with different numbers of generalized OOD Detectors To further investigate the effect of different numbers of generalized OOD detectors on the division of OOD data under the cross-validation mechanism, we conducted ablation experiments on the number of detectors. We selected four models, CLIP (Radford et al., 2021), BLIP2 (Li et al., 2023b), GroupViT (Xu et al., 2022), and LiT (Zhai et al., 2022), as generalized OOD detectors, and sequentially increased the number of detectors involved in the division in a fixed order. Consistent with the OOD detector type ablation experiment E.1, to avoid introducing additional bias, we did not perform downsampling operations on the OOD data divided by the above models to maintain the consistency of the division process. To assess the impact of the number of detectors on the OOD data division results, we tested the OOD data divided by different numbers of detectors on several mainstream VLMs. We visualised the results in the form of a line graph (shown in Figure 12). The horizontal axis of the graph represents the number of detectors involved in the division, and the vertical axis represents the accuracy and F1 score obtained from testing on each VLM. The experimental results show that when only a single OOD detector (number 1) is used, each VLM reaches the highest value in both Accuracy and F1, which indicates that a single detector may bring about model bias, which in turn leads to a mismatch between the divided OOD data and the potential OOD distribution of the large VLMs. When the number of detectors is increased to 2, the performance of each large VLM decreases significantly, indicating that the cross-validation mechanism can mitigate the error brought about by single-detector bias to some extent, thus obtaining data that more closely fits the OOD distribution of the large VLM. However, when the number of detectors is further increased to 3, the performance of all six VLMs except InternVL2.5-8B shows a performance rebound, i.e., the data divided by three detectors generally performs higher than that of 2 detectors in terms of Accuracy and F1. Nonetheless, this performance rebound is still significantly lower than that seen with the single detector division. When the number of detectors reaches 4, we observe that the test performance is almost the same as with three detectors, suggesting that further increases in the number of detectors do not lead to substantial improvement in division, and that the division effect tends to stabilise. Combining the results of the experiments with two to four detectors, we found that a higher number of OOD detectors is not better. With two detectors, the division results are closest to the potential OOD distribution of the major VLMs. Continuing to increase the number of detectors may instead introduce additional uncertainty, which leads to the division of data deviating from the potential OOD distribution of the major large VLMs and gradually converging. Based on performance and division stability considerations, we ultimately chose to use 2 OOD detectors to construct the default configuration of the OOD data division scheme in OODBench. E.3 The selection of the threshold T In the process of collecting OOD data, we define that when the probability of an instance category is lower than the threshold T, it is considered a detection failure, and thus, the instance category is considered OOD data. In order to investigate the effect of different thresholds T on the division of OOD data, we conducted T ablation experiments. Considering that Llama (Touvron et al., 2023) is currently the most popular Large Language Model, we chose Llama-3.2-Vision (Dubey et al., 2024) as the test model. When T is larger than 0.050.05, there is too much OOD data division, so we set T to 0.050.05, 0.010.01, and 0.0050.005 to control the amount of OOD data. The experimental results are shown in Tab. 9: when T=0.05T=0.05, the accuracy difference between OOD-S and OOD-H is 7.447.44; when T=0.01T=0.01, the difference decreases to 1.861.86; and when T=0.005T=0.005, the difference further decreases to 1.731.73. As the value of T decreases, the boundary of the distributions between OOD-S and OOD-H becomes more blurred. In terms of the number of OOD data, the decrease in the value of T leads to a decrease in the number of OOD-S from 25,798 to about 23,500 and a significant decrease in the number of OOD-H from 9,882 to 4,274. To ensure that the model divides the OOD data with a high degree of differentiation, we finally set T to 0.050.05. Threshold T Data Type nuScenes Performance Num Accuracy F1 Precision Recall MCC OOD-S 25798 66.30 63.34 69.43 58.24 33.02 0.05 OOD-H 9882 58.86 60.28 58.28 62.42 17.77 OOD-S 23546 62.65 55.69 68.44 46.95 26.64 0.01 OOD-H 5934 60.79 59.08 61.76 56.62 21.65 OOD-S 23644 60.93 53.31 66.24 44.60 23.13 0.005 OOD-H 4274 59.20 57.42 60.03 55.03 18.45 Table 9: An ablation study of hyperparameter T using the LLama-3.2-Vision (Dubey et al., 2024). Appendix F OODBench Sub-Dataset Performance Subdataset classification performance. Tab. 10, Tab. 11, Tab. 12 and Tab. 13 show the performance of the different models on publicly available datasets collected in natural and autonomous driving scenarios, respectively. Where, OOD-S‘ and OOD-H‘ denote the initial benchmark datasets. In the Time column, the time superscript indicates the number of graphics cards used in the test, and we used NVIDIA L40s graphics cards in the testing phase. Model Image Encoder Language Model Data Type COCO-ID-OOD Performance Num Accuracy(%\%) F1(%\%) Precision(%\%) Recall(%\%) MCC(%\%) Time Random Chance - - ID/OOD-S/OOD-H - 50.00%50.00\% 50.00%50.00\% 50.00%50.00\% 50.00%50.00\% 0.00%0.00\% - Open-source Models ID 6000 89.90 90.17 87.83 92.63 79.92 00:24:242^2 OOD-S 6000 74.32 70.26 83.45 60.67 50.55 00:27:162^2 OOD-H 6000 74.45 69.53 86.12 58.30 51.67 00:23:492^2 OOD-S’ 7972 74.34 70.09 83.99 60.14 50.76 00:57:401^1 OOD-H’ 10158 75.31 70.67 87.01 59.50 53.36 01:07:401^1 IDCoTID^CoT 6000 74.92 79.02 67.91 94.47 54.14 00:22:572^2 OOD-SCoTOOD-S^CoT 6000 73.43 70.13 80.09 62.37 48.06 00:28:342^2 LLaVA-NeXT-8B CLIP-L-14 Llama-3-8B-Instruct OOD-HCoTOOD-H^CoT 6000 76.65 71.52 91.66 58.63 57.14 00:27:142^2 ID 6000 92.65 92.84 90.48 95.33 85.42 00:32:421^1 OOD-S 6000 75.43 72.67 81.87 65.33 51.94 00:32:281^1 OOD-H 6000 76.57 74.09 82.85 67.00 54.13 00:32:261^1 OOD-S’ 7972 75.60 72.82 82.20 65.35 52.31 00:42:571^1 OOD-H’ 10158 77.13 74.79 83.32 67.85 55.22 00:53:331^1 IDCoTID^CoT 6000 92.62 92.65 92.21 93.10 85.24 01:11:341^1 OOD-SCoTOOD-S^CoT 6000 74.82 72.37 80.15 65.97 50.43 01:30:321^1 DeepSeek-VL-7B-Chat SigLIP-L &\& SAM-B DeepSeek-LLM-7B OOD-HCoTOOD-H^CoT 6000 73.80 70.95 79.60 64.00 48.54 01:09:511^1 ID 6000 92.88 92.74 94.62 90.93 85.83 00:29:101^1 OOD-S 6000 73.97 67.74 89.03 54.67 51.96 00:27:581^1 OOD-H 6000 75.17 68.63 93.14 54.33 55.37 00:31:001^1 OOD-S’ 7972 74.02 67.82 89.10 54.74 52.07 00:43:441^1 OOD-H’ 10158 75.59 69.32 93.24 55.17 56.06 01:00:141^1 IDCoTID^CoT 6000 90.90 89.99 99.96 81.83 83.18 00:57:421^1 OOD-SCoTOOD-S^CoT 6000 69.43 57.17 95.48 40.80 47.41 00:52:151^1 DeepSeek-VL2-Small SigLIP-SO400M DeepSeekMoE LLM OOD-HCoTOOD-H^CoT 6000 68.45 54.35 98.26 37.57 46.92 00:57:071^1 ID 6000 92.83 92.85 92.69 93.00 85.67 00:40:412^2 OOD-S 6000 77.22 76.40 79.23 73.77 54.56 00:39:382^2 OOD-H 6000 65.18 64.92 65.41 64.43 30.37 00:37:582^2 OOD-S’ 7972 77.35 76.60 79.21 74.16 54.80 00:52:502^2 OOD-H’ 10160 66.22 65.99 66.44 65.55 32.44 01:03:502^2 IDCoTID^CoT 6000 96.18 96.07 99.01 93.30 92.52 01:20:502^2 OOD-SCoTOOD-S^CoT 6000 77.88 73.92 90.04 62.70 58.53 01:21:192^2 InternVL2-8B InternViT-300M-448px InternLM2.5-7b-chat OOD-HCoTOOD-H^CoT 6000 81.48 78.17 95.21 66.30 66.09 01:19:542^2 ID 6000 97.02 97.01 97.25 96.77 94.03 00:25:172^2 OOD-S 6000 84.88 84.18 88.29 80.43 70.04 00:26:282^2 OOD-H 6000 79.70 78.65 82.95 74.77 59.69 00:25:202^2 OOD-S’ 7972 85.00 84.31 88.35 80.63 70.26 00:35:112^2 OOD-H’ 10160 80.65 79.55 84.33 75.30 61.65 00:42:532^2 IDCoTID^CoT 6000 96.88 96.87 97.40 96.33 93.77 01:59:052^2 OOD-SCoTOOD-S^CoT 6000 82.02 80.31 88.71 73.37 65.01 01:48:032^2 InternVL2.5-8B InternViT-300M-448px-V2.5 InternLM2.5-7b-chat OOD-HCoTOOD-H^CoT 6000 81.92 80.08 89.13 72.70 64.95 01:49:082^2 ID 6000 97.02 97.04 96.14 97.97 94.05 00:56:591^1 OOD-S 6000 82.87 83.05 82.16 83.97 65.75 01:00:411^1 OOD-H 6000 82.02 82.91 78.99 87.23 64.38 01:01:031^1 OOD-S’ 7972 75.70 77.81 71.60 85.20 52.36 01:10:421^1 OOD-H’ 10158 71.93 75.64 66.81 87.16 46.06 01:30:221^1 IDCoTID^CoT 6000 78.32 80.58 72.97 89.97 58.24 02:26:011^1 OOD-SCoTOOD-S^CoT 6000 67.13 69.04 65.25 73.30 34.53 02:19:541^1 Llama-3.2-11B-Vision-Instruct ViT-H-14 Llama3.1-8B OOD-HCoTOOD-H^CoT 6000 68.78 72.68 64.62 83.03 39.19 02:16:511^1 ID 6000 89.48 90.11 85.01 95.87 79.62 00:06:542^2 OOD-S 6000 81.57 79.98 87.52 73.63 63.94 00:06:592^2 OOD-H 6000 77.67 75.48 83.66 68.77 56.23 00:07:002^2 OOD-S’ 7972 81.18 79.49 87.35 72.93 63.24 00:09:132^2 OOD-H’ 10160 78.74 76.78 84.58 70.30 58.32 00:11:442^2 IDCoTID^CoT 6000 92.08 92.43 88.53 96.70 84.53 00:29:542^2 OOD-SCoTOOD-S^CoT 6000 82.43 81.73 85.13 78.60 65.06 00:45:112^2 Qwen2-VL-7B-Instruct CLIP-L-14 Qwen2-7B OOD-HCoTOOD-H^CoT 6000 76.82 75.87 79.10 72.90 53.80 00:44:572^2 ID 6000 95.92 95.79 98.93 92.83 92.01 00:11:151^1 OOD-S 6000 79.10 75.00 93.30 62.70 61.61 00:11:101^1 OOD-H 6000 78.28 74.04 92.03 61.93 59.86 00:11:161^1 OOD-S’ 7972 78.81 74.54 93.39 62.02 61.18 00:14:191^1 OOD-H’ 10160 79.42 75.52 93.13 63.52 62.05 00:18:401^1 IDCoTID^CoT 6000 95.45 95.23 99.96 90.93 91.27 01:22:111^1 OOD-SCoTOOD-S^CoT 6000 76.48 70.35 95.17 55.80 58.18 01:24:461^1 Qwen2.5-VL-7B-Instruct ViT(SwiGLU, RMSNorm) Qwen2.5 LLM OOD-HCoTOOD-H^CoT 6000 77.20 70.89 98.00 55.53 60.36 01:26:111^1 Closed-source Models ID 5996 91.86 92.06 89.84 94.40 83.83 01:41:54 OOD-S 5998 75.88 76.44 74.70 78.26 51.81 01:47:49 OOD-H 5994 66.80 68.46 65.20 72.07 33.79 01:49:24 OOD-S’ 7966 75.50 76.28 73.91 78.81 51.10 02:16:19 OOD-H’ 10150 67.02 68.71 65.37 72.41 34.25 02:59:57 IDCoTID^CoT 5996 96.28 96.22 97.70 94.80 92.60 02:21:18 OOD-SCoTOOD-S^CoT 5998 78.26 76.04 84.69 68.99 57.52 02:29:46 Gemini - - OOD-HCoTOOD-H^CoT 5994 79.53 77.85 84.82 71.94 59.75 02:26:29 ID 5998 93.33 93.45 91.82 95.13 86.72 03:53:40 OOD-S 5992 76.72 78.22 73.48 83.61 53.95 04:38:08 OOD-H 6000 70.53 72.06 68.51 76.00 41.31 03:47:18 OOD-S’ 7958 77.51 78.97 74.16 84.44 55.55 05:35:00 OOD-H’ 10146 66.51 69.82 63.54 77.47 33.84 07:08:36 IDCoTID^CoT 6000 82.92 84.48 77.39 93.00 67.21 06:37:40 OOD-SCoTOOD-S^CoT 5994 67.52 66.39 68.78 64.16 35.11 06:50:23 GPT-4o - GPT-4 OOD-HCoTOOD-H^CoT 5996 69.31 69.72 68.81 70.65 38.64 05:58:43 Table 10: Performance on COCO-ID-OOD. We report the performance of the 10 leading VLMs on OODBench. All models perform significantly lower on OOD-H than on ID. Model Image Encoder Language Model Data Type LVIS-ID-OOD Performance Num Accuracy(%\%) F1(%\%) Precision(%\%) Recall(%\%) MCC(%\%) Time Random Chance - - ID/OOD-S/OOD-H - 50.00%50.00\% 50.00%50.00\% 50.00%50.00\% 50.00%50.00\% 0.00%0.00\% - Open-source Models ID 6000 85.17 85.06 85.67 84.47 70.34 00:24:542^2 OOD-S 6000 47.90 35.20 46.55 28.30 -4.57 00:29:542^2 OOD-H 2436 40.23 30.86 36.60 26.68 -20.30 00:10:452^2 OOD-S’ 13282 47.10 35.08 45.40 28.58 -6.24 01:49:001^1 IDCoTID^CoT 6000 72.45 75.57 67.88 85.23 46.44 00:25:282^2 OOD-SCoTOOD-S^CoT 6000 49.65 41.62 49.52 35.90 -0.73 00:36:392^2 LLaVA-NeXT-8B CLIP-L-14 Llama-3-8B-Instruct OOD-HCoTOOD-H^CoT 2436 48.07 46.47 47.95 45.07 -3.87 00:12:552^2 ID 6000 89.85 90.14 87.63 92.80 79.84 00:27:581^1 OOD-S 6000 47.42 41.65 46.78 37.53 -5.27 00:33:431^1 OOD-H 2436 42.69 41.39 42.35 40.48 -14.63 00:08:511^1 OOD-S’ 13282 47.33 42.06 46.74 38.23 -5.42 00:49:511^1 IDCoTID^CoT 6000 88.42 88.23 89.65 86.87 76.87 01:15:311^1 OOD-SCoTOOD-S^CoT 6000 46.18 38.51 44.91 33.70 -7.88 01:58:381^1 DeepSeek-VL-7B-Chat SigLIP-L &\& SAM-B DeepSeek-LLM-7B OOD-HCoTOOD-H^CoT 2436 35.18 30.16 32.69 28.00 -29.95 00:41:001^1 ID 6000 87.75 87.13 91.74 82.97 75.85 00:30:311^1 OOD-S 6000 47.23 30.90 44.75 23.60 -6.28 00:29:371^1 OOD-H 2436 39.66 27.08 34.21 22.41 -22.04 00:12:381^1 OOD-S’ 13282 46.94 31.50 44.43 24.39 -6.85 01:40:481^1 IDCoTID^CoT 6000 82.47 78.82 99.49 65.27 69.15 00:51:511^1 OOD-SCoTOOD-S^CoT 6000 44.30 15.61 32.19 10.30 -15.55 00:49:321^1 DeepSeek-VL2-Small SigLIP-SO400M DeepSeekMoE LLM OOD-HCoTOOD-H^CoT 2436 34.85 8.64 14.45 6.16 -36.99 00:24:251^1 ID 6000 87.20 87.31 86.57 88.07 74.41 00:40:082^2 OOD-S 6000 49.12 46.16 49.01 43.63 -1.78 00:41:492^2 OOD-H 2436 34.56 33.80 34.20 33.42 -30.88 00:16:272^2 OOD-S’ 13284 48.34 45.32 48.14 42.82 -3.33 01:32:382^2 IDCoTID^CoT 6000 94.55 94.28 99.12 89.90 89.49 01:18:292^2 OOD-SCoTOOD-S^CoT 6000 48.23 38.32 47.40 32.17 -3.73 01:31:172^2 InternVL2-8B InternViT-300M-448px InternLM2.5-7b-chat OOD-HCoTOOD-H^CoT 2436 43.23 36.47 41.40 32.59 -13.86 00:35:242^2 ID 6000 94.17 94.09 95.28 92.93 88.36 00:25:502^2 OOD-S 6000 52.87 47.65 53.58 42.90 5.85 00:28:342^2 OOD-H 2436 39.24 36.04 38.05 34.24 -21.62 00:10:552^2 OOD-S’ 13284 51.71 46.47 52.12 41.93 3.48 01:03:032^2 IDCoTID^CoT 6000 94.52 94.40 96.42 92.47 89.11 01:47:402^2 OOD-SCoTOOD-S^CoT 6000 52.30 46.24 52.97 41.03 4.72 01:48:512^2 InternVL2.5-8B InternViT-300M-448px-V2.5 InternLM2.5-7b-chat OOD-HCoTOOD-H^CoT 2436 41.67 37.76 40.47 35.39 -16.80 00:44:342^2 ID 6000 92.60 92.88 89.54 96.47 85.46 00:57:421^1 OOD-S 6000 50.62 51.70 50.59 52.87 1.23 01:01:181^1 OOD-H 2436 40.23 43.61 41.28 46.22 -19.68 00:23:341^1 OOD-S’ 13282 49.99 51.01 49.99 52.07 -0.02 02:14:401^1 IDCoTID^CoT 6000 75.57 78.71 69.74 90.33 53.52 02:18:561^1 OOD-SCoTOOD-S^CoT 6000 49.87 51.31 49.87 52.83 -0.27 02:16:411^1 Llama-3.2-11B-Vision-Instruct ViT-H-14 Llama3.1-8B OOD-HCoTOOD-H^CoT 2436 41.67 37.76 40.47 35.39 -16.80 00:44:342^2 ID 6000 86.78 87.63 82.37 93.60 74.26 00:06:512^2 OOD-S 6000 55.60 53.15 56.25 50.37 11.26 00:06:562^2 OOD-H 2436 49.14 50.50 49.18 51.89 -1.73 00:02:502^2 OOD-S’ 13284 54.91 52.80 55.39 50.45 9.86 00:15:262^2 IDCoTID^CoT 6000 89.03 89.58 85.33 94.27 78.50 00:35:252^2 OOD-SCoTOOD-S^CoT 6000 56.23 55.69 56.39 55.00 12.47 00:54:502^2 Qwen2-VL-7B-Instruct CLIP-L-14 Qwen2-7B OOD-HCoTOOD-H^CoT 2436 48.11 49.68 48.22 51.23 -3.78 00:19:202^2 ID 6000 93.03 92.81 95.94 89.87 86.24 00:11:141^1 OOD-S 6000 52.63 41.43 54.27 33.50 5.70 00:10:551^1 OOD-H 2436 46.31 39.67 45.26 35.30 -7.57 00:04:201^1 OOD-S’ 13284 52.29 41.66 53.60 34.08 4.92 00:23:431^1 IDCoTID^CoT 6000 92.70 92.13 99.92 85.47 86.31 01:31:561^1 OOD-SCoTOOD-S^CoT 6000 51.38 35.05 52.78 26.23 3.20 01:48:301^1 Qwen2.5-VL-7B-Instruct ViT(SwiGLU, RMSNorm) Qwen2.5 LLM OOD-HCoTOOD-H^CoT 2436 43.19 32.36 39.98 27.18 -14.39 00:46:051^1 Closed-source Models ID 5996 87.68 88.13 84.98 91.53 75.57 01:41:01 OOD-S 5994 51.58 52.05 51.55 52.55 3.17 01:47:55 OOD-H 2436 35.18 37.47 36.19 38.83 -29.72 00:42:07 OOD-S’ 13272 50.63 51.49 50.61 52.40 1.27 03:51:01 IDCoTID^CoT 5996 92.48 92.42 93.15 91.69 84.97 02:21:13 OOD-SCoTOOD-S^CoT 5994 50.10 43.41 50.13 38.27 0.21 02:38:14 Gemini - - OOD-HCoTOOD-H^CoT 2436 40.72 37.27 39.58 35.22 -18.67 01:03:47 ID 5960 93.09 93.26 91.00 95.64 86.29 05:46:52 OOD-S 5990 58.30 60.15 57.59 62.94 16.67 04:29:12 OOD-H 2432 41.69 45.08 42.61 47.86 -16.74 01:45:24 OOD-S’ 13282 57.90 58.53 57.66 59.43 15.80 08:40:59 IDCoTID^CoT 6000 81.10 83.09 75.16 92.90 64.01 06:35:48 OOD-SCoTOOD-S^CoT 5958 51.88 48.98 52.12 46.19 3.78 08:42:01 GPT-4o - GPT-4 OOD-HCoTOOD-H^CoT 2392 45.07 46.76 45.36 48.24 -9.89 04:43:32 Table 11: Performance on LVIS-ID-OOD. We report the performance of the 10 leading VLMs on OODBench. All models perform significantly lower on OOD-H than on ID. Model Image Encoder Language Model Data Type nuScenes-ID-OOD Performance Num Accuracy(%\%) F1(%\%) Precision(%\%) Recall(%\%) MCC(%\%) Time Random Chance - - ID/OOD-S/OOD-H - 50.00%50.00\% 50.00%50.00\% 50.00%50.00\% 50.00%50.00\% 0.00%0.00\% - Open-source Models ID 6000 81.43 80.05 86.49 74.50 63.48 00:27:022^2 OOD-S 6000 62.68 45.38 84.62 31.00 32.79 00:31:302^2 OOD-H 6000 59.03 40.34 74.20 27.70 23.18 00:30:312^2 OOD-S’ 25798 62.57 45.42 83.83 31.15 32.32 03:45:131^1 OOD-H’ 9882 60.58 42.24 79.02 28.82 27.41 01:23:291^1 IDCoTID^CoT 6000 50.90 58.48 50.66 69.17 1.93 01:11:181^1 OOD-SCoTOOD-S^CoT 6000 52.12 41.14 53.38 33.47 4.56 01:06:262^2 LLaVA-NeXT-8B CLIP-L-14 Llama-3-8B-Instruct OOD-HCoTOOD-H^CoT 6000 54.87 42.58 58.51 33.47 10.77 00:57:572^2 ID 6000 83.40 82.09 89.14 76.07 67.53 00:33:531^1 OOD-S 6000 61.12 47.16 73.57 34.70 26.19 00:34:221^1 OOD-H 6000 52.95 28.77 59.19 19.00 8.04 00:23:411^1 OOD-S’ 25798 60.47 46.13 72.38 33.85 24.73 02:03:011^1 OOD-H’ 9882 54.62 30.33 65.28 19.75 12.91 00:55:521^1 IDCoTID^CoT 6000 74.20 71.15 80.68 63.63 49.52 02:20:001^1 OOD-SCoTOOD-S^CoT 6000 59.18 42.66 71.68 30.37 22.47 03:04:271^1 DeepSeek-VL-7B-Chat SigLIP-L &\& SAM-B DeepSeek-LLM-7B OOD-HCoTOOD-H^CoT 6000 46.30 24.54 41.26 17.47 -9.06 02:44:521^1 ID 6000 76.08 70.42 92.27 56.93 56.47 00:43:541^1 OOD-S 6000 58.73 32.64 88.76 20.00 27.62 00:41:441^1 OOD-H 6000 51.72 15.81 61.68 9.07 6.58 00:44:101^1 OOD-S’ 25798 58.09 30.67 88.69 18.54 26.44 03:27:021^1 OOD-H’ 9882 52.74 16.49 70.81 9.33 11.05 01:32:271^1 IDCoTID^CoT 6000 77.77 71.59 99.12 56.03 61.66 00:55:411^1 OOD-SCoTOOD-S^CoT 6000 58.73 31.90 91.19 19.33 28.37 01:01:121^1 DeepSeek-VL2-Small SigLIP-SO400M DeepSeekMoE LLM OOD-HCoTOOD-H^CoT 6000 51.05 11.56 59.81 6.40 4.67 01:01:551^1 ID 6000 89.77 89.13 95.09 83.87 80.09 00:29:172^2 OOD-S 6000 71.27 68.14 76.45 61.47 43.37 00:31:042^2 OOD-H 6000 51.08 43.20 51.50 37.20 2.26 00:30:462^2 OOD-S’ 25800 70.90 67.60 76.26 60.71 42.70 02:12:542^2 OOD-H’ 9884 52.08 44.57 52.86 38.53 4.33 00:50:142^2 IDCoTID^CoT 6000 90.20 89.30 98.32 81.80 81.56 03:56:472^2 OOD-SCoTOOD-S^CoT 6000 64.73 50.24 85.30 35.60 36.26 04:20:002^2 InternVL2-8B InternViT-300M-448px InternLM2.5-7b-chat OOD-HCoTOOD-H^CoT 6000 62.23 45.48 81.75 31.50 31.02 04:24:462^2 ID 6000 82.70 83.75 78.96 89.17 65.95 00:23:122^2 OOD-S 6000 70.13 71.02 68.97 73.20 40.34 00:24:382^2 OOD-H 6000 36.23 39.61 37.62 41.83 -27.71 00:24:292^2 OOD-S’ 25800 69.65 70.57 68.50 72.78 39.38 01:45:432^2 OOD-H’ 9884 36.68 39.90 37.97 42.05 -26.80 00:40:222^2 IDCoTID^CoT 6000 78.00 79.16 75.19 83.57 56.35 02:53:332^2 OOD-SCoTOOD-S^CoT 6000 65.18 57.43 73.89 46.97 32.61 03:21:222^2 InternVL2.5-8B InternViT-300M-448px-V2.5 InternLM2.5-7b-chat OOD-HCoTOOD-H^CoT 6000 54.82 48.99 56.24 43.40 9.89 03:24:392^2 ID 6000 88.78 88.74 89.06 88.43 77.57 01:11:401^1 OOD-S 6000 69.58 64.56 77.34 55.40 40.84 01:15:191^1 OOD-H 6000 61.72 60.02 62.81 57.47 23.52 01:18:181^1 OOD-S’ 25798 66.30 63.34 69.43 58.24 33.02 04:51:101^1 OOD-H’ 9882 58.86 60.28 58.28 62.42 17.77 01:51:331^1 IDCoTID^CoT 6000 71.63 72.33 70.60 74.13 43.32 02:34:131^1 OOD-SCoTOOD-S^CoT 6000 59.33 49.90 64.97 40.50 20.15 02:27:421^1 Llama-3.2-11B-Vision-Instruct ViT-H-14 Llama3.1-8B OOD-HCoTOOD-H^CoT 6000 59.98 55.56 62.46 50.03 20.37 02:29:421^1 ID 6000 92.72 92.58 94.39 90.83 85.49 00:50:552^2 OOD-S 6000 73.53 69.84 81.16 61.30 48.54 00:51:022^2 OOD-H 6000 57.55 50.05 60.79 42.53 15.83 00:50:542^2 OOD-S’ 25800 73.12 69.32 80.75 60.72 47.74 03:40:302^2 OOD-H’ 9884 59.02 51.87 62.85 44.15 18.91 01:24:182^2 IDCoTID^CoT 6000 87.82 88.04 86.47 89.67 75.69 01:35:022^2 OOD-SCoTOOD-S^CoT 6000 68.58 59.95 82.66 47.03 41.19 02:02:322^2 Qwen2-VL-7B-Instruct CLIP-L-14 Qwen2-7B OOD-HCoTOOD-H^CoT 6000 57.13 47.27 61.40 38.43 15.38 02:02:222^2 ID 6000 80.62 81.20 78.82 83.73 61.35 01:09:271^1 OOD-S 6000 63.77 51.17 78.44 37.97 32.14 00:50:181^1 OOD-H 6000 60.12 46.02 71.18 34.00 23.73 00:59:191^1 OOD-S’ 25800 63.54 50.57 78.51 37.30 31.82 03:19:441^1 OOD-H’ 9884 61.72 47.97 74.85 35.30 27.61 01:20:311^1 IDCoTID^CoT 6000 91.20 90.35 100.00 82.40 83.71 02:05:301^1 OOD-SCoTOOD-S^CoT 6000 66.52 51.83 92.31 36.03 41.67 02:06:141^1 Qwen2.5-VL-7B-Instruct ViT(SwiGLU, RMSNorm) Qwen2.5 LLM OOD-HCoTOOD-H^CoT 6000 63.02 45.28 87.01 30.60 34.19 02:20:171^1 Closed-source Models ID 6000 90.40 90.48 89.76 91.20 80.81 02:06:56 OOD-S 6000 77.25 76.71 78.57 74.93 54.56 02:15:23 OOD-H 6000 56.95 56.74 57.02 56.47 13.90 02:17:32 OOD-S’ 25798 77.19 76.68 78.42 75.03 54.43 08:43:47 OOD-H’ 9882 59.07 59.13 59.04 59.22 18.13 03:23:56 IDCoTID^CoT 6000 92.70 92.55 94.45 90.73 85.47 02:57:49 OOD-SCoTOOD-S^CoT 6000 68.55 62.16 78.01 51.67 39.42 03:03:12 Gemini - - OOD-HCoTOOD-H^CoT 6000 68.93 64.36 75.47 56.10 39.18 02:58:06 ID 5996 86.72 87.72 81.60 94.83 74.43 05:14:11 OOD-S 5994 74.94 74.55 75.73 73.41 49.91 05:51:09 OOD-H 5998 58.92 61.29 57.94 65.06 17.98 05:48:40 OOD-S’ 25798 75.83 75.62 76.27 74.98 51.66 24:39:04 OOD-H’ 9882 58.78 61.58 57.67 66.06 17.76 09:52:58 IDCoTID^CoT 6000 70.83 75.76 64.81 91.17 45.61 07:48:32 OOD-SCoTOOD-S^CoT 5998 62.70 58.41 66.01 52.38 25.97 08:11:22 GPT-4o - GPT-4 OOD-HCoTOOD-H^CoT 5992 61.35 60.50 61.85 59.21 22.72 07:21:03 Table 12: Performance on nuScenes-ID-OOD. We report the performance of the 10 leading VLMs on OODBench. All models perform significantly lower on OOD-H than on ID. Model Image Encoder Language Model Data Type Cityscapes-ID-OOD Performance Num Accuracy(%\%) F1(%\%) Precision(%\%) Recall(%\%) MCC(%\%) Time Random Chance - - ID/OOD-S/OOD-H - 50.00%50.00\% 50.00%50.00\% 50.00%50.00\% 50.00%50.00\% 0.00%0.00\% - Open-source Models ID 3556 81.21 82.14 78.29 86.39 62.77 00:18:392^2 OOD-S 3820 78.22 76.55 82.91 71.10 57.02 00:17:432^2 OOD-H 3836 66.79 58.09 78.70 46.04 36.90 00:18:582^2 IDCoTID^CoT 3554 59.23 69.67 55.47 93.64 25.44 00:32:011^1 OOD-SCoTOOD-S^CoT 3820 65.58 70.01 62.02 80.37 32.61 00:20:072^2 LLaVA-NeXT-8B CLIP-L-14 Llama-3-8B-Instruct OOD-HCoTOOD-H^CoT 3836 70.52 65.47 79.00 55.89 42.91 00:19:562^2 ID 3554 91.73 91.37 95.52 87.56 83.75 00:25:481^1 OOD-S 3818 78.71 75.95 87.23 67.26 58.98 00:27:571^1 OOD-H 3834 61.53 54.99 66.25 47.00 24.10 00:26:571^1 IDCoTID^CoT 3554 85.45 85.18 86.80 83.62 70.95 00:49:001^1 OOD-SCoTOOD-S^CoT 3818 77.71 75.33 84.35 68.05 56.49 00:51:541^1 DeepSeek-VL-7B-Chat SigLIP-L &\& SAM-B DeepSeek-LLM-7B OOD-HCoTOOD-H^CoT 3834 63.43 54.60 71.99 43.97 29.16 01:25:131^1 ID 3554 80.95 76.55 99.55 62.18 66.79 00:32:181^1 OOD-S 3818 77.42 71.46 97.12 56.52 60.37 00:30:481^1 OOD-H 3834 65.39 47.49 98.36 31.30 42.07 00:31:181^1 IDCoTID^CoT 3554 79.07 73.52 100.00 58.13 64.01 00:42:591^1 OOD-SCoTOOD-S^CoT 3818 70.77 59.80 95.73 43.48 49.58 00:52:231^1 DeepSeek-VL2-Small SigLIP-SO400M DeepSeekMoE LLM OOD-HCoTOOD-H^CoT 3834 57.90 27.88 97.20 16.28 28.53 00:50:001^1 ID 3556 95.95 95.82 98.98 92.86 92.08 00:17:342^2 OOD-S 3820 86.94 86.20 91.38 81.57 74.30 00:21:152^2 OOD-H 3836 77.48 74.78 84.95 66.79 56.25 00:19:212^2 IDCoTID^CoT 3556 90.83 89.94 99.59 82.00 82.97 02:12:082^2 OOD-SCoTOOD-S^CoT 3820 82.38 79.44 95.38 68.06 67.60 02:23:472^2 InternVL2-8B InternViT-300M-448px InternLM2.5-7b-chat OOD-HCoTOOD-H^CoT 3836 77.69 71.66 98.19 56.41 61.18 02:27:582^2 ID 3556 93.53 93.61 92.43 94.83 87.09 00:14:582^2 OOD-S 3820 86.88 86.95 86.52 87.38 73.77 00:16:552^2 OOD-H 3836 72.34 71.07 74.50 67.94 44.86 00:15:462^2 IDCoTID^CoT 3556 93.98 93.95 94.38 93.53 87.97 01:46:202^2 OOD-SCoTOOD-S^CoT 3820 85.73 84.80 90.75 79.58 72.01 01:55:572^2 InternVL2.5-8B InternViT-300M-448px-V2.5 InternLM2.5-7b-chat OOD-HCoTOOD-H^CoT 3836 79.25 76.99 86.38 69.45 59.66 01:58:272^2 ID 3554 92.40 92.70 89.22 96.45 85.09 00:45:221^1 OOD-S 3818 73.21 76.24 68.49 85.96 48.00 00:43:551^1 OOD-H 3834 65.26 69.41 62.00 78.82 31.71 00:41:421^1 IDCoTID^CoT 3554 71.05 70.41 72.00 68.88 42.13 01:32:091^1 OOD-SCoTOOD-S^CoT 3818 64.64 62.42 66.61 58.72 29.49 01:37:421^1 Llama-3.2-11B-Vision-Instruct ViT-H-14 Llama3.1-8B OOD-HCoTOOD-H^CoT 3834 61.22 61.29 61.17 61.40 22.43 01:34:271^1 ID 3556 95.78 95.69 97.88 93.59 91.65 01:16:542^2 OOD-S 3820 86.10 85.58 88.89 82.51 72.39 01:22:252^2 OOD-H 3836 75.36 72.42 82.24 64.70 51.92 01:22:522^2 IDCoTID^CoT 3556 95.73 95.67 96.94 94.43 91.48 01:37:582^2 OOD-SCoTOOD-S^CoT 3820 83.53 83.24 84.75 81.78 67.11 01:34:262^2 Qwen2-VL-7B-Instruct CLIP-L-14 Qwen2-7B OOD-HCoTOOD-H^CoT 3836 70.62 68.77 73.39 64.70 41.53 01:47:122^2 ID 3556 92.82 92.28 99.87 85.76 86.52 00:44:031^1 OOD-S 3820 86.30 84.69 96.02 75.75 74.28 00:51:291^1 OOD-H 3836 81.35 77.39 98.23 63.85 66.94 00:47:351^1 IDCoTID^CoT 3556 94.15 93.78 100.00 88.29 88.91 02:01:181^1 OOD-SCoTOOD-S^CoT 3820 84.65 82.56 95.59 72.66 71.39 02:12:091^1 Qwen2.5-VL-7B-Instruct ViT(SwiGLU, RMSNorm) Qwen2.5 LLM OOD-HCoTOOD-H^CoT 3836 81.61 77.61 99.19 63.75 67.69 02:04:441^1 Closed-source Models ID 3554 96.88 96.88 96.75 97.02 93.75 03:11:49 OOD-S 3818 88.45 88.48 88.27 88.69 76.90 03:35:00 OOD-H 3834 86.78 86.96 85.79 88.16 73.58 03:38:15 IDCoTID^CoT 3554 97.16 97.14 97.89 96.40 94.33 03:32:09 OOD-SCoTOOD-S^CoT 3818 86.90 85.96 92.62 80.20 74.48 03:42:41 Gemini - - OOD-HCoTOOD-H^CoT 3834 91.78 91.40 95.88 87.32 83.90 03:42:08 ID 3552 96.54 96.59 95.09 98.14 93.12 07:12:17 OOD-S 3800 89.34 89.48 88.35 90.63 78.71 07:51:47 OOD-H 3830 81.28 82.32 77.99 87.15 63.00 07:21:43 IDCoTID^CoT 3514 77.32 80.81 70.03 95.50 58.66 10:25:24 OOD-SCoTOOD-S^CoT 3816 70.55 74.68 65.48 86.90 43.48 08:13:24 GPT-4o - GPT-4 OOD-HCoTOOD-H^CoT 3824 69.87 73.36 65.75 82.95 41.18 09:15:38 Table 13: Performance on Cityscapes-ID-OOD. We report the performance of the 10 leading VLMs on OODBench. All models perform significantly lower on OOD-H than on ID. Subdataset BAP performance. Tab. 14, 15, 16, and 17 show the evaluation results of the BAP metrics on four datasets, namely COCO, LVIS, nuScenes, and Cityscapes, respectively. The BAP metrics progressively increase the complexity of the questions from the perspective of task construction, and consist of Existence, Counting, and Logic Reasoning, with the corresponding performance evaluation metrics of E-Acc, C-Acc, and L-Acc, respectively. The results of the four datasets show a consistent trend. Under the same question type, as the data distribution migrates from ID to OOD-S to OOD-H, the model performance generally decreases. This phenomenon indicates that the performance of the multimodal large language model degrades significantly when the data distribution deviates from the training distribution, reflecting that its generalization ability in dealing with complex environments is still limited. On the other hand, while keeping the test data unchanged, when the complexity of questioning is gradually increased from Existence to Counting to Logic Reasoning, the accuracy of the model on the three corresponding metrics (E-Acc, C-Acc, and L-Acc) also shows a decreasing trend, which indicates that the higher the requirement of reasoning ability is, the more difficult it is for the model to answer correctly. Specifically, Tab. 14 to 17 systematically show the performance of major mainstream VLMs on the four datasets of COCO, LVIS, nuScenes, and Cityscapes. The results are generally consistent, further validating the generality of the two trends mentioned above. This indicates that current VLMs still have significant performance bottlenecks when facing more challenging inference tasks and more complex data distributions. Model Data Type COCO BAP Performance E-Acc(%\%) C-Acc(%\%) L-Acc(%\%) Time ID 88.04 61.72 37.08 00:03:482^2 OOD-S 79.98 47.40 33.23 00:10:142^2 LLaVA-NeXT-8B OOD-H 55.85 33.83 21.10 00:09:472^2 ID 91.39 58.61 44.26 00:02:582^2 OOD-S 80.30 36.26 27.81 00:06:472^2 DeepSeek-VL-7B-Chat OOD-H 60.89 33.26 23.17 00:06:162^2 ID 85.85 55.40 37.65 00:08:531^1 OOD-S 73.89 41.28 29.14 00:23:001^1 DeepSeek-VL2-Small OOD-H 49.77 34.86 23.74 00:22:081^1 ID 88.04 69.14 57.18 00:05:342^2 OOD-S 79.87 53.57 41.45 00:12:322^2 InternVL2-8B OOD-H 59.63 49.43 36.35 00:11:262^2 ID 94.02 67.94 53.83 00:04:352^2 OOD-S 85.93 59.20 47.62 00:11:042^2 InternVL2.5-8B OOD-H 70.87 49.31 38.76 00:10:102^2 ID 96.65 66.99 51.20 00:04:262^2 OOD-S 86.90 56.17 35.61 00:10:172^2 Llama-3.2-11B-Vision-Instruct OOD-H 86.93 48.17 31.08 00:09:592^2 ID 91.87 65.55 48.56 00:02:132^2 OOD-S 81.82 62.12 48.16 00:05:092^2 Qwen2-VL-7B-Instruct OOD-H 66.06 45.53 33.60 00:05:392^2 ID 89.21 66.19 54.44 00:06:011^1 OOD-S 78.11 59.05 45.50 00:15:261^1 Qwen2.5-VL-7B-Instruct OOD-H 58.83 47.36 35.78 00:15:061^1 ID 88.49 75.78 42.21 02:57:38 OOD-S 81.47 69.45 48.43 00:59:56 Gemini OOD-H 66.97 52.29 34.40 03:54:16 ID 90.41 66.67 36.93 00:45:11 OOD-S 85.70 58.07 42.90 01:44:58 gpt-4o OOD-H 74.77 48.17 33.72 01:45:12 Table 14: Experimental performance of 10 leading VLMs on COCO for Basic-to-Advanced Progress. Model Data Type LVIS BAP Performance E-Acc(%\%) C-Acc(%\%) L-Acc(%\%) Time ID 81.27 52.07 29.84 00:07:142^2 OOD-S 49.04 35.01 17.39 00:10:152^2 LLaVA-NeXT-8B OOD-H 31.25 27.73 14.06 00:02:452^2 ID 90.44 55.04 39.79 00:05:382^2 OOD-S 53.36 22.66 16.19 00:06:242^2 DeepSeek-VL-7B-Chat OOD-H 50.78 23.05 13.67 00:01:562^2 ID 77.23 43.86 28.07 00:23:361^1 OOD-S 51.14 34.21 19.93 00:25:161^1 DeepSeek-VL2-Small OOD-H 29.41 26.67 12.94 00:08:021^1 ID 85.14 65.37 48.32 00:11:282^2 OOD-S 55.16 35.97 24.82 00:12:222^2 InternVL2-8B OOD-H 42.19 37.89 28.12 00:03:342^2 ID 90.18 61.50 46.51 00:08:462^2 OOD-S 56.12 38.13 25.06 00:10:412^2 InternVL2.5-8B OOD-H 48.83 36.72 21.48 00:03:122^2 ID 96.12 62.02 41.99 00:08:272^2 OOD-S 59.11 37.17 25.78 00:09:292^2 Llama-3.2-11B-Vision-Instruct OOD-H 62.50 32.42 21.88 00:02:562^2 ID 91.60 63.95 45.48 00:04:382^2 OOD-S 57.07 44.36 28.78 00:05:362^2 Qwen2-VL-7B-Instruct OOD-H 51.95 31.64 23.44 00:01:352^2 ID 88.75 59.38 43.73 00:11:421^1 OOD-S 60.98 39.98 26.05 00:15:161^1 Qwen2.5-VL-7B-Instruct OOD-H 54.51 32.16 19.61 00:04:161^1 ID 89.65 70.25 37.39 00:49:55 OOD-S 57.21 50.72 25.72 04:56:20 Gemini OOD-H 54.12 41.57 23.53 00:17:33 ID 94.44 63.39 39.07 01:29:08 OOD-S 63.03 41.54 21.37 01:39:38 gpt-4o OOD-H 58.43 35.29 16.08 00:29:38 Table 15: Experimental performance of 10 leading VLMs on LVIS for Basic-to-Advanced Progress. Model Data Type nuScenes BAP Performance E-Acc(%\%) C-Acc(%\%) L-Acc(%\%) Time ID 68.31 21.38 15.54 00:08:032^2 OOD-S 65.15 40.88 28.65 00:07:102^2 LLaVA-NeXT-8B OOD-H 35.10 20.65 13.27 00:08:462^2 ID 68.46 30.77 13.08 00:05:052^2 OOD-S 59.85 34.67 19.89 00:04:292^2 DeepSeek-VL-7B-Chat OOD-H 27.14 22.57 15.34 00:05:232^2 ID 56.39 5.39 3.54 00:22:421^1 OOD-S 57.04 43.88 22.30 00:25:281^1 DeepSeek-VL2-Small OOD-H 20.21 22.71 10.18 00:28:181^1 ID 82.15 38.92 25.85 00:10:032^2 OOD-S 69.89 41.42 25.73 00:07:422^2 InternVL2-8B OOD-H 44.99 41.30 28.76 00:09:432^2 ID 85.69 38.92 31.54 00:05:562^2 OOD-S 75.73 45.26 34.85 00:05:112^2 InternVL2.5-8B OOD-H 46.02 35.10 24.78 00:06:362^2 ID 87.38 37.69 19.54 00:08:152^2 OOD-S 77.37 51.46 23.54 00:07:082^2 Llama-3.2-11B-Vision-Instruct OOD-H 58.11 28.47 17.11 00:08:382^2 ID 88.00 34.00 24.62 00:08:472^2 OOD-S 75.73 50.36 40.51 00:07:292^2 Qwen2-VL-7B-Instruct OOD-H 47.49 36.28 25.07 00:09:102^2 ID 81.51 32.05 23.42 00:22:131^1 OOD-S 72.39 44.79 35.47 00:18:531^1 Qwen2.5-VL-7B-Instruct OOD-H 44.69 35.40 25.22 00:22:511^1 ID 86.44 47.15 35.75 00:55:20 OOD-S 79.71 53.38 40.95 00:43:04 Gemini OOD-H 62.09 46.31 29.94 00:55:34 ID 92.91 36.21 29.12 01:53:19 OOD-S 82.97 47.07 36.08 01:34:33 gpt-4o OOD-H 68.83 38.26 26.88 01:55:43 Table 16: Experimental performance of 10 leading VLMs on nuScenes for Basic-to-Advanced Progress. Model Data Type Cityscapes BAP Performance E-Acc(%\%) C-Acc(%\%) L-Acc(%\%) Time ID 87.37 28.44 12.73 00:11:482^2 OOD-S 69.53 17.17 10.75 00:13:412^2 LLaVA-NeXT-8B OOD-H 43.62 10.86 7.05 00:12:382^2 ID 87.99 22.38 11.19 00:08:562^2 OOD-S 70.19 15.19 8.21 00:09:512^2 DeepSeek-VL-7B-Chat OOD-H 45.33 12.95 6.86 00:09:432^2 ID 68.76 8.12 2.67 00:50:221^1 OOD-S 67.64 12.08 4.43 01:07:381^1 DeepSeek-VL2-Small OOD-H 41.09 5.15 3.43 00:45:521^1 ID 92.92 40.04 16.22 00:15:032^2 OOD-S 83.30 31.98 15.57 00:27:592^2 InternVL2-8B OOD-H 65.43 32.38 21.71 00:18:062^2 ID 94.87 27.93 12.63 00:10:022^2 OOD-S 85.94 17.83 12.92 00:11:152^2 InternVL2.5-8B OOD-H 67.14 23.71 18.19 00:10:222^2 ID 97.23 33.47 13.86 00:13:292^2 OOD-S 85.38 18.87 7.92 00:15:052^2 Llama-3.2-11B-Vision-Instruct OOD-H 77.24 19.62 8.29 00:15:062^2 ID 94.25 31.72 13.35 00:33:152^2 OOD-S 83.77 19.43 11.98 00:35:412^2 Qwen2-VL-7B-Instruct OOD-H 64.38 22.00 15.14 00:35:372^2 ID 86.43 22.40 10.69 00:50:031^1 OOD-S 79.53 18.58 10.85 00:57:081^1 Qwen2.5-VL-7B-Instruct OOD-H 62.73 21.54 15.54 00:54:251^1 ID 97.43 48.92 31.14 02:30:39 OOD-S 89.91 37.74 24.72 03:00:34 Gemini OOD-H 87.70 32.98 22.12 03:46:30 ID 98.36 36.38 22.82 05:50:51 OOD-S 90.85 23.96 15.85 06:37:16 gpt-4o OOD-H 86.94 24.59 16.97 06:31:57 Table 17: Experimental performance of 10 leading VLMs on Cityscapes for Basic-to-Advanced Progress. Appendix G Effect of Model Scale on OODBench Performance To further analyze the performance of models of different scales on OODBench under the same architecture, we conducted a systematic comparison of different parameter scales within the same model family. Specifically, we selected the representative Qwen2-VL series. We performed experiments on the complete sets and subsets of the four domains covered by OODBench (COCO, LVIS, nuScenes, Cityscapes) for its 2B and 7B versions. The results are shown in Table 18: The 2B model achieved accuracy scores of 80.6, 64.0, and 81.2 on the OOD-H metrics for COCO, nuScenes, and Cityscapes, respectively, all surpassing the 7B model’s scores of 77.7, 57.6, and 75.4. However, on the LVIS dataset, the 7B model slightly outperformed the 2B model. These results reveal a distinct mixed trend, indicating that model size does not maintain a monotonically increasing relationship with OOD robustness. Increasing parameter size does not consistently enhance model performance under semantic OOD conditions. This phenomenon is further validated in larger-scale models. For instance, as shown in Tables 1, 10, 11, 12, and 13, models like GPT-4o and Gemini, which have significantly more parameters than 7B, do not systematically outperform smaller-scale models on OODBench. Overall, expanding model scale cannot fundamentally resolve semantic OOD issues. The root cause lies in the fact that when test samples deviate from the support domain of training data in the joint image-text distribution, their OOD characteristics remain independent of model scale and thus do not naturally disappear with increased capacity. Model Data Type OODBench Performance Num Accuracy(%) F1(%) Precision(%) Recall(%) MCC(%) Random Chance - - 50.00 50.00 50.00 50.00 0.00 COCO Qwen2-VL-2B OOD-S 6000 78.78 74.62 92.85 62.37 60.95 OOD-H 6000 80.63 76.53 97.13 63.13 65.40 OOD-S’ 7972 78.84 74.61 93.23 62.19 61.17 OOD-H’ 10160 81.81 78.13 97.95 64.98 67.56 OOD-S 6000 81.57 79.98 87.52 73.63 63.94 OOD-H 6000 77.67 75.48 83.66 68.77 56.23 OOD-S’ 7972 81.18 79.49 87.35 72.93 63.24 Qwen2-VL-7B OOD-H’ 10160 78.74 76.78 84.58 70.33 58.32 LVIS Qwen2-VL-2B OOD-S 6000 48.23 34.91 47.01 27.77 -3.87 OOD-H 2436 41.67 32.69 38.63 28.33 -17.29 OOD-S’ 13284 47.70 35.03 46.24 28.19 -4.99 OOD-S 6000 55.60 53.15 56.25 50.37 11.26 OOD-H 2436 49.14 50.50 49.18 51.89 -1.73 Qwen2-VL-7B OOD-S’ 13284 54.91 52.80 55.39 50.45 9.86 nuScenes Qwen2-VL-2B OOD-S 6000 68.00 57.62 85.29 43.50 41.30 OOD-H 6000 64.00 50.21 81.39 36.30 33.63 OOD-S’ 25800 67.30 56.76 83.77 42.92 39.64 OOD-H’ 9884 66.04 52.68 82.76 37.82 38.55 OOD-S 6000 73.53 69.84 81.16 61.30 48.54 OOD-H 6000 57.55 50.05 70.99 42.53 15.83 OOD-S’ 25800 73.12 69.32 80.75 60.72 47.74 Qwen2-VL-7B OOD-H’ 9884 59.02 51.87 62.85 44.15 18.91 CityScapes Qwen2-VL-2B OOD-S 3820 88.85 88.05 94.81 82.20 78.39 OOD-H 3836 81.20 77.28 97.69 63.92 66.51 OOD-S 3820 86.10 85.58 88.89 82.51 72.39 Qwen2-VL-7B OOD-H 3836 75.36 72.42 82.24 64.70 51.92 Table 18: Performance of different model scales on OODBench. The table shows results for the 2B and 7B variants of the Qwen2-VL series across various data domains on OODBench. It can be observed that there is no monotonic relationship between model scale and OOD robustness; increasing parameter scale does not consistently improve model performance under semantic OOD conditions. Where, OOD-S’ and OOD-H’ denote the initial benchmark datasets. Appendix H CLIP and BLIP2 visualization distribution logits on coco val Why is OODBench Challenging? The appearance of OOD data is usually caused by semantic shift or covariate shift, which means that the generation mechanism of the test data has changed from the training data. Although a model performs well on the training set, it may show degraded or unstable performance when faced with a change in the data distribution. Models usually lack sufficient robustness to such changes, especially when dealing with changes that are not supported by explicit labels or historical data. We divide the OODBench data containing covariate shift through the OOD detector, which has shifted its data distribution despite sharing the same label space as the training data. Deep learning models often give high-confidence predictions or even make very confident false predictions when dealing with these data. The fundamental reason why OODBench poses a challenge to models is that it does not follow the distributional assumptions of the training data. In contrast, OOD data from the real world is highly uncertain and diverse. When confronted with these data, models typically lack the necessary knowledge and ability to make accurate predictions, which can lead to misclassification or high-confidence false predictions. We shows a visualization of the distribution of logits predicted by VLMs when confronted with real-world data. Figures 14 and 15 show the results of CLIP and BLIP2 for computing logits on the COCO validation set for 80 labeled categories. Figure 13, on the other hand, summarises the logit results of CLIP and BLIP2 on the COCO validation set for 80 categories. As shown in Figure 13, the logits distribution shows a double peak, appearing on both sides of the distribution, which indicates that the CLIP and BLIP2 models have high confidence in the instance category. However, the single peak on the far left side of Figure 13 indicates that both models also have high confidence in determining instance-category mismatches, resulting in high-confidence false predictions. Figure 13: Logistically visualized distributions of CLIP (top image) and BLIP2 (bottom image) on coco val. Figure 14: CLIP visualization distribution of 80 category logits on coco val. Figure 15: BLIP2 visualization distribution of 80 category logits on coco val. Appendix I OODBench Data and Case Demonstration In this section, we show some of the data examples constructed by OODBench, covering OOD scenarios divided from datasets such as COCO, LVIS, nuScenes, and Cityscapes. Figures 16 and 17 show the associated visualisations. Through these examples, we aim to help readers visualise the characteristics of different types of out-of-distribution data and the challenges they pose under different tasks and perceptual conditions faced by large multimodal models. Figure 16: Sample OOD instances from the COCO and nuScenes datasets are divided by OODBench. These examples illustrate diverse out-of-distribution categories across different scenarios and environments. Figure 17: Sample OOD instances divided from the LVIS and Cityscapes datasets by OODBench. The samples highlight distributional shifts in object-centric. To further demonstrate the applicability and challenge of OODBench, we constructed typical examples on four datasets, COCO, LVIS, nuScenes, and Cityscapes, which are shown in Figs. 18, 19, 20, 21, respectively. Each set of examples covers a variety of question types, including standard question answering, Chain-of-Thought prompting, and tasks such as existential, counting, and logical reasoning under BAP metrics. These cases visualise the challenges of perception and reasoning in large multimodal models in different out-of-distribution scenarios. Figure 18: This is a showcase of OODBench examples on the COCO dataset. Examples cover standard question-and-answer, chain-of-thought prompts, and task types such as existential, counting, and logical reasoning. Green font indicates correct responses given by the model, and red font indicates incorrect responses. Figure 19: Demonstration of OODBench cases on the LVIS dataset. The questions shown cover a variety of task formats and visualise the performance of the model in an out-of-distribution scenario. Green font indicates correct responses, and red font indicates incorrect responses. Figure 20: OODBench case demonstrations on the nuScenes dataset. It includes visual Q&A and complex inference tasks in multimodality; green font indicates the model’s correct predictions, and red font indicates incorrect predictions. Figure 21: A showcase of OODBench cases on the Cityscapes dataset. A wide range of question types is covered to assess the model’s ability to generalize to out-of-distribution environments such as urban streetscapes. Green font represents correct responses, and red font represents incorrect responses. Appendix J Limitations While OODBench offers a fully automated covariate‐shift benchmark for VLMs, it remains limited in scope. First, it is confined to general natural and autonomous driving domains and does not cover other real‐world domains (e.g., medical imaging, satellite imagery). In these domains, researchers must redefine appropriate OOD types based on their specific semantic structures, task requirements, and distribution characteristics. Simultaneously, the performance of specialized detectors directly impacts the quality of initial screening, while subsequent expert evaluations (e.g., in medical settings) may incur additional labor costs. Therefore, the OOD definition within this framework should not be directly applied to the aforementioned domains. Second, it focuses solely on static vision data and omits temporal or multimodal variations. Third, it lacks extreme environmental conditions (e.g., severe weather, low light). Extending OODBench to additional domains, dynamic data, and challenging environmental factors will be an important direction for future work.

Instant research discovery