Paper deep dive

Fine-Grained Multi Image Object Hallucination Benchmark

Joonki Min, Chaeyun Kim, Hyungwook Choi, Yejin Kim, Kihyun Kim, Yohan Jo, Joonseok Lee

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 78

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Multimodal Large Language Models (MLLMs) are increasingly deployed in multi-image scenarios requiring complex reasoning across visual contexts. However, current MLLMs remain fundamentally limited by object hallucination-generating plausible yet factually inconsistent descriptions about objects. Existing benchmarks, designed primarily for single-image settings or providing only high-level multi-image assessments, cannot systematically diagnose how visual complexity and reasoning demands trigger hallucination. To address this gap, we introduce MIOH, a fine-grained multi-image object hallucination benchmark that systematically evaluates object hallucination across four foundational tasks (existence, counting, attribute, position) through three multi-image reasoning patterns (comprehensive, comparative, selective) under three controlled adversarial pressures (visual context scale, perceptual difficulty, contextual bias). Through evaluation of 29 models, we reveal that even state-of-the-art systems like GPT-5 and Gemini-2.5-Pro exhibit distinct failure patterns across different reasoning patterns and tasks. Our evaluation reveals that hallucination stems not merely from perceptual failures but from integration-stage limitations when maintaining object representations across multiple images. MIOH provides a controlled framework for analyzing multi-image object hallucination and serves as a critical evaluation tool for developing more reliable multimodal AI systems.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

74,630 characters extracted from source content.

Expand or collapse full text

Fine-Grained Multi-Image Object Hallucination Benchmark Joonki Min 1∗ , Chaeyun Kim 1,2∗ , Hyungwook Choi 1 , Yejin Kim 1 , Kihyun Kim 1 Yohan Jo 1† , Joonseok Lee 1† 1 Seoul National University 2 AIM Intelligence minc69,golddohyun,chooi221,a2000yejin,ki5477,yohan.jo,joonseok@snu.ac.kr Abstract Multimodal Large Language Models (MLLMs) are increas- ingly deployed in multi-image scenarios requiring com- plex reasoning across visual contexts. However, current MLLMs remain fundamentally limited by object halluci- nation—generating plausible yet factually inconsistent de- scriptions about objects. Existing benchmarks, designed primarily for single-image settings or providing only high- level multi-image assessments, cannot systematically diag- nose how visual complexity and reasoning demands trigger hallucination. To address this gap, we introduce MIOH, a fine-grained multi-image object hallucination benchmark that systematically evaluates object hallucination across four foundational tasks (existence, counting, attribute, po- sition) through three multi-image reasoning patterns (com- prehensive, comparative, selective) under three controlled adversarial pressures (visual context scale, perceptual dif- ficulty, contextual bias). Through evaluation of 29 models, we reveal that even state-of-the-art systems like GPT-5 and Gemini-2.5-Pro exhibit distinct failure patterns across dif- ferent reasoning patterns and tasks. Our evaluation reveals that hallucination stems not merely from perceptual fail- ures but from integration-stage limitations when maintain- ing object representations across multiple images. MIOH provides a controlled framework for analyzing multi-image object hallucination and serves as a critical evaluation tool for developing more reliable multimodal AI systems. 1. Introduction With recent advances, Multimodal Large Language Models (MLLMs) are capable of reasoning over multiple images si- multaneously. This capability requires models not only to recognize content in individual images but also to integrate and synthesize information across diverse visual inputs. De- spite these advancements, however, current MLLMs remain * Equal contribution. † Corresponding author. fundamentally limited by object hallucination, where mod- els generate plausible yet factually inconsistent descriptions about objects in the queried images. Multi-image scenarios amplify object hallucination along two critical dimensions. First, perceptual failure : even within individual images, models struggle with vi- sually challenging objects (small size, occlusion, contex- tual ambiguity) or contextually misleading scenes where co-occurring objects create false expectations. Second, in- formation integration : multi-image contexts demand inte- grating and synthesizing information across images, where models must aggregate information across all images (com- prehensive reasoning), identify differences between im- ages (comparative reasoning), or retrieve specific images matching given criteria (selective reasoning). Failures in either dimension create compounding pathways for hal- lucination. Systematic evaluation requires isolating both: which visual conditions trigger perceptual errors, and which integration demands are most vulnerable to failure. Despite the importance of understanding object halluci- nation in multi-image contexts, existing evaluation frame- works do not adequately address these two dimensions. Most object hallucination benchmarks [18, 28, 43, 47, 51] are designed for single-image scenarios with binary ques- tions, focusing narrowly on existence and counting. This design cannot reveal how visual difficulty factors (percep- tual challenges, contextual biases) systematically induce hallucination, nor can it assess multi-image reasoning pat- terns or compositional capabilities (attributes, spatial rela- tions) that require fine-grained object understanding. While general multi-image benchmarks [11, 16, 26, 33, 37, 50] evaluate overall reasoning, they lack the controlled manip- ulation of visual factors needed to diagnose object hallu- cination vulnerabilities. Recent work such as MIHBench [27] explores multi-image hallucination but examines only image quantity as an adversarial factor in ablation studies, without systematically varying visual difficulty or distin- guishing between reasoning patterns. Consequently, exist- ing benchmarks cannot pinpoint which specific visual con- ditions or reasoning demands trigger hallucination, nor how arXiv:2608.30653v1 [cs.CV] 31 Aug 2026 these factors interact. To narrow these gaps, we introduce the Fine-Grained Multi-Image Object Hallucination (MIOH) benchmark, de- signed to systematically evaluate both dimensions of multi- image object hallucination. MIOH systematically inte- grates four object-centric tasks (existence, counting, at- tribute, position) with three reasoning patterns (comprehen- sive, comparative, selective), enabling compositional eval- uation from basic detection to fine-grained property bind- ing. Furthermore, MIOH introduces three controlled adver- sarial pressures—visual context scale (number of images), perceptual difficulty (small/occluded objects), and contex- tual bias (misleading co-occurrence priors)—that systemat- ically vary visual complexity while measuring performance across reasoning patterns. This design enables diagnostic analysis by separately measuring performance under differ- ent visual conditions (perceptual difficulty, contextual bias, image scale) and across different reasoning patterns (com- prehensive, comparative, selective). Our contributions are summarized as follows: • We introduce MIOH, the first benchmark to systemati- cally assess object hallucination in multi-image con- texts with fine-grained diagnostic capabilities across vi- sual complexity and reasoning demands. • We define three multi-image reasoning patterns (comprehensive, comparative, selective) and instantiate them across four foundational object-centric tasks (exis- tence, counting, attribute, position), enabling diagnosis of which reasoning capability fails under hallucination pressures. • We design three controllable adversarial pressures (visual context scale, perceptual difficulty, contextual bias) that systematically exacerbate hallucination, al- lowing fine-grained analysis of when and why MLLMs fail in multi-image scenarios. 2. Related Work Multimodal Large Language Models (MLLMs). Follow- ing the success of LLMs, MLLMs have rapidly evolved through visual instruction tuning, utilized by LLaVA [30] and extended by InstructBLIP [8] and MiniGPT-4 [62]. Early MLLMs face challenges in cross-image reasoning due to limitations in visual token processing and inter-image se- mantic modeling [2, 4, 8, 21, 25]. Recent work has enabled multi-image understanding [16, 20, 24, 36, 57]; Mantis [16], LLaVA-NeXT-Interleave [24], and Idefics3 [20] lever- age large-scale image-text data at training, and Qwen2.5- VL [3], InternVL3.5 [54], and Gemini-2.5-Pro [6] demon- strate powerful cross-image reasoning capabilities. Object Hallucination in MLLMs. Object hallucination, defined as MLLMs generating plausible but inaccurate ob- ject descriptions inconsistent with visual inputs, remains a critical challenge [7, 43]. Systematic analysis has pin- pointed causes across the MLLM pipeline: data-related is- sues such as statistical biases in training data [28], limita- tions of vision encoder in fine-grained semantics [48], in- sufficient modality alignment [31], and inherited LLM bi- ases such as weak context attention [53]. Recent studies reveal additional visual vulnerabilities, e.g., perception of small or occluded objects [58], contextual bias due to ob- ject co-occurrence patterns [28] and semantic similarities [23]. MLLMs are also linguistically susceptible to syco- phantic alignment with user beliefs and context hijacking from misleading narratives [61]. Mitigation strategies, e.g., data augmentation [44], preference optimization [59], and inference-time interventions [14, 60], have primarily tar- geted single image scenarios. Multi-image contexts amplify hallucination challenges, requiring models not only to rec- ognize objects accurately, but to track them and maintain contextual consistency across distinct images [56]. This in- creased complexity requires a new benchmark tailored to assess object hallucination on multiple images. Benchmarks for MLLMs. Early MLLM benchmarks fo- cus on single-image scenarios across various tasks includ- ing visual question answering, reasoning, and composi- tional understanding [10, 12, 22, 29, 32]. Recently, sev- eral benchmarks [11, 16, 26, 33, 37, 50] assess general rea- soning capabilities across multiple images, though none of them specifically target object hallucination. In parallel, object hallucination has been addressed through specialized benchmarks, including discriminative approaches using binary questions [13, 28] and genera- tive approaches directly assessing free-form descriptions [18, 43, 47, 51]. They typically focus on existence and simple counting tasks, with limited coverage of attributes or spatial relations. Also, they predominantly employ sim- ple binary questions or captioning tasks, confined to single- image settings. Recently, MIHBench[27] has also explored hallucination in multi-image settings. However, it focuses mainly on binary existence questions and varies only the number of images, without modeling broader visual diffi- culty factors or diverse object-centric tasks. Our bench- mark is complementary to this direction, offering a more fine-grained diagnostic space across multiple tasks, reason- ing patterns, and adversarial conditions. 3. Overall Design of our MIOH Benchmark We introduce the Multi-Image Object Hallucination (MIOH) benchmark, designed to systematically evaluate object hallucination in MLLMs under multi-image con- texts. MIOH is structured around two complementary di- mensions: (1) multi-image reasoning patterns that probe different integration capabilities (Sec. 3.1), and (2) adver- sarial pressures that systematically challenge perceptual Figure 1. Overview of our MIOH Benchmark. MIOH evaluates object hallucination in multi-image contexts across four core tasks: existence, counting, attribute, and position. Each task includes three question types (comprehensive, comparative, and selective) designed to probe different aspects of multi-image reasoning capabilities. and contextual robustness (Sec. 3.3). By crossing four fun- damental object-centric tasks with three reasoning patterns under varying adversarial conditions, MIOH enables fine- grained diagnosis of when and why MLLMs hallucinate. 3.1. Multi-Image Reasoning Patterns Motivation.Multi-image understanding requires fun- damentally different reasoning capabilities compared to single-image settings. While single-image benchmarks test whether models can recognize objects in isolation, multi- image scenarios demand that models synthesize, compare, and retrieve information across multiple visual contexts. We identify three core reasoning patterns that capture these distinct capabilities, each placing unique demands on the model’s information integration mechanisms. Comprehensive Reasoning. requires models to aggregate information across all images to form a holistic judgment. Questions in this category ask about collective properties spanning the entire image set, such as “Does any image con- tain a zebra?” or “What is the total count of cars across all images?” This pattern tests the model’s ability to maintain a unified representation of information distributed across multiple scenes—a capability essential for summarization and global understanding tasks. Failures in comprehensive reasoning indicate difficulties in information aggregation or token limitations. Comparative Reasoning.requires models to identify differences between specific images. Questions such as “Which image contains more cars?” demand precise cross- image attention and the ability to maintain separate repre- sentations for different scenes while comparing them. This is critical for change detection, progress monitoring, and contrastive analysis. Failures suggest weaknesses in main- taining distinct object representations across contexts. Selective Reasoning. requires models to retrieve a spe- cific image that matches given criteria from a set of can- didates. Questions like “In which image are exactly three zebras present?” test the ability to perform targeted retrieval while filtering out irrelevant information. This is essential for search and localization tasks in multi-image contexts. Failures indicate poor image-level indexing or inability to isolate relevant visual evidence from distractors. Diagnostic Value. These three patterns are not merely dif- ferent question formats—they probe distinct failure modes. Uniform failure across all patterns suggests a fundamental bottleneck in object recognition (perceptual failure). Con- versely, failure in selective reasoning while succeeding in comprehensive reasoning points to a specific integration failure in retrieval and localization. This systematic eval- uation allows us to diagnose which integration capability breaks down under specific adversarial conditions. 3.2. Object-Centric Tasks We apply these three reasoning patterns to four fundamental object-centric tasks that represent core visual understanding capabilities: Existence (verifying object presence/absence), Counting (enumerating objects), Attribute (binding visual properties to objects, e.g., “red car”), and Position (binding spatial relationships, e.g., “dog next to a cat”). While Exis- tence and Counting have been the primary focus of object hallucination benchmarks [5, 18, 28, 35, 43, 52], all four tasks form the foundation for assessing object-centric capa- bilities of MLLMs [10, 17, 32, 41, 47, 49, 50]. From Tasks to Question Types. By crossing four tasks with three reasoning patterns, we create a comprehensive evaluation space. However, not all combinations are equally meaningful or distinct. For example, in the Counting task, comprehensive reasoning (“total count across all images”) naturally leads to multiple variants depending on whether we ask for exact counts, comparisons of counts, or pres- ence of specific quantities. Through careful design, we developed 26 distinct question types that systematically cover the task and reasoning space while ensuring each type probes a unique aspect of multi-image understanding. Tab. 1 presents the complete taxonomy with representative templates for each type. Complete examples are in Fig. 1 and Sec. C. 3.3. Adversarial Pressures To systematically probe model vulnerabilities, we introduce three adversarial factors that create challenging but realistic evaluation conditions. These factors target distinct failure mechanisms: visual context scale tests integration capacity as the number of images increases, perceptual difficulty challenges feature extraction from small or occluded ob- jects, and contextual bias probes susceptibility to mislead- ing co-occurrence priors. Visual Context Scale (Number of Images).Inspired by findings that MLLMs struggle to identify information across large image sets (the “Visual Haystack” problem [56]), we systematically vary the Number of input Images (NI) among 2, 4, 8, 10 for the same underlying question. This tests the model’s integration capacity—as the visual context expands with more images, models must maintain accurate object representations across an increasing number of visual scenes. Perceptual Difficulty (Hard Positive). Small or partially occluded objects are inherently harder to detect [34, 55, 58]. We curate Hard Positive (HP) examples using two comple- mentary approaches: (a) Rule-based filtering selects images where target objects have small bounding boxes or high oc- clusion ratios; (b) CLIP-based semantic filtering identifies Table 1. MIOH Question Type Taxonomy. We design 26 ques- tion types by crossing 4 object-centric tasks with 3 reasoning pat- terns. Each type is illustrated with a template showing its structure. TypeTemplate Existence Comprehensive Is there at least oneobject in any of these images? Comprehensive Is aobject present in all of these images? Comprehensive Which object appears in at least one image? Comprehensive Which object appears in all images? SelectiveIn which image does aobject appear? ComparativeWhich of the two images contains aobject? ComparativeWhich object is in Image 1 but not Image 2? Attribute Comprehensive Is a “attributeobject” present in any image? Comprehensive Is a “attributeobject” present in all images? Comprehensive Which attribute–object pair appears in one image? Comprehensive Which attribute–object pair appears in all images? SelectiveIn which image is “attributeobject” found? ComparativeWhich of the two images “attributeobject” present? ComparativeWhich attribute–object pair is in Image 1 but not Image 2? Counting Comprehensive What is the total number of “object” across images? Comprehensive Which object appearscount times across images? Comprehensive In how many images is a “object” present? SelectiveIn which image arecount “object” found? ComparativeIn which image does the “object” appear most/least? Position Comprehensive Is asubjectrelation aanchor present in any image? Comprehensive Is asubjectrelation aanchor present in all images? Comprehensive Which object with spatial relationship appears in one image? Comprehensive Which object with spatial relationship appears in all images? SelectiveIn which image is asubjectrelation aanchor? SelectiveWhere is “subjectrelationanchor” present? ComparativeWhich object with spatial relationship is in Image 1 but not Image 2? cases where CLIP similarity between the image and text prompt “A photo of [object]” is abnormally low, indicating perceptual ambiguity. These examples test whether models can extract accurate features under perceptually challenging conditions—a necessary prerequisite for any downstream reasoning. Contextual Bias (Hard Negative).Strong contextual priors can mislead models into hallucinating objects that are contextually plausible but visually absent [9, 23, 28]. For example, a kitchen scene may activate strong pri- ors for “frying pan,” leading to false positives. We con- struct Hard Negative (HN) examples by: (a) estimat- ing co-occurrence probabilities P(object A |object B ) from COCO training data and selecting images containing high- probability co-occurring objects but missing the target; (b) applying CLIP-based semantic confusion to find im- ages with high visual-text similarity despite object absence. These examples test whether models rely on contextual shortcuts rather than grounded visual evidence. 3.4. Implementation details Dataset Selection. To ensure annotation quality, we use three complementary datasets: COCO-ReM [46] (exis- tence, counting) addresses COCO’s incomplete masks and missing instances via systematic re-annotation; PACO [42] (attributes) provides standardized attribute labels across ob- Existence - Comprehensive Which of the following objects appears in all of these images? A) tv B) toothbrush C) dog D) sheep E) None of the above EASY Hard Positive Which of the following objects appears in all of these images? A) laptop B) cell phone C) baseball bat D) carrot E) None of the above Image 1 Hard Negative Is there at least one surfboard in any of these images? A) Yes, all of them B) Yes, in some C) No, in none D) I don’t know Counting - Selective EASY Hard Positive Hard Negative In which image can you find exactly 3 'zebra'(s)? A) Image 1 B) Image 2 C) Image 3 D) Image 4 E) None of the above In which image can you find exactly 3 'elephant'(s)? A) Image 1 B) Image 2 C) Image 3 D) Image 4 E) None of the above In which image can you find exactly 2 'sandwich'(s)? A) Image 1 B) Image 2 C) Image 3 D) Image 4 E) None of the above Figure 2. Representative examples from MIOH. (Top) Existence-Comprehensive questions across Easy, Hard Negative, and Hard Positive scenarios. (Bottom) Counting-Selective questions with varying difficulty levels. Additional examples are provided in Sec. C. ject categories; SVG [39] (spatial relationships) offers com- plete scene-level annotations, unlike Visual Genome [19] which averages only 1.5 relationships per subject. Benchmark Statistics. Three independent annotators val- idated all questions. The final benchmark comprises 3,484 questions across 11,732 images, balanced across tasks, rea- soning patterns, and difficulty levels. Full construction de- tails, filtering criteria, and statistics are in Sec. A. 4. Benchmark Results and Discussion We conduct a comprehensive comparative study using our MIOH over the state-of-the-art MLLMs, including GPT-5 [45] and Gemini-2.5-Pro [6]. Among open-source mod- els, we choose the LLaVA [24] series, the Qwen [3] series, InternVL [54], Phi-4-multimodal [1], MiniCPM-V [57], Ovis-2.5 [36], and Mantis-8B [16]. For reproducibility, the decoding temperature is set to 0 for all experiments. All ex- periments were conducted on four NVIDIA A6000 GPUs. 4.1. Overall Performance We first present overall performance results demonstrating the extent of object hallucination challenges across different model categories and tasks. Tab. 2 reports the comprehen- sive evaluation results over 29 models, revealing that multi- image object hallucination remains as significant challenge, with an overall average accuracy of only 36.1%. Not sur- prisingly, a clear performance gap is measured between proprietary and open-source models. The leading models, Gemini-2.5 Pro (64.4%) and GPT-5 (63.1%), set the state- of-the-art, but are still far from perfect. Top-performing open-source models, such as Qwen2-VL-7B (49.1%) and MiniCPM-V-2.6 (48.5%), demonstrate strong capabilities but lag behind the frontier models. Beyond the overall gap, performance is shaped by three key factors—reasoning pattern, task type, and adversarial pressure—each revealing distinct failure modes, which we analyze in the following subsections. 4.2. Reasoning Patterns Our analysis reveals that MLLM performance is governed not only by the task but also, fundamentally, by the required reasoning pattern. As shown in Fig. 3, performance varies significantly across Comprehensive, Comparison, and Se- lection patterns, demonstrating that existing object hallu- cination benchmarks, constrained to a few question styles, cannot properly assess object hallucination across diverse multi-image settings. Overall, Comprehensive emerges as the most manage- able reasoning pattern, with basic Existence questions yielding the highest performance. Comparing this densely high-performing row against the rest of the heatmap reveals that models are heavily biased towards simple, presence- verifying queries. Evaluating models solely within this nar- row scope inevitably overestimates their true robustness. On average, Selection(34.2%) emerges as the most chal- lenging reasoning pattern, underperforming both Compre- hensive and Comparison. This difficulty becomes espe- cially pronounced when paired with compositional tasks such as Attribute(26.7%), indicating that models often rec- ognize that an object-attribute pair exists within the image set but fail to identify which specific image contains it—a distinct form of grounding failure in multi-image settings. Model ExistenceCounting EasyHNHPNIAvgEasyHNHPNIAvg Overall62.451.951.530.049.424.329.527.020.525.4 GPT-591.488.678.455.378.455.054.351.535.749.1 Gemini-2.5 Pro83.081.580.756.475.464.466.459.440.057.5 Qwen2-VL-2B58.353.545.932.747.621.723.318.621.421.2 Qwen2.5-VL-3B80.762.555.544.760.823.326.719.617.121.7 Qwen2-VL-7B87.375.576.430.767.528.936.221.617.126.0 Qwen2.5-VL-7B84.076.073.628.065.433.339.717.521.428.0 LLaVA-v1.6 (Mistral-7B)33.047.036.8-38.920.627.618.6-22.2 LLaVA-Interleave (Qwen-0.5B)36.331.528.618.728.822.828.424.714.322.6 LLaVA-Interleave (Qwen-7B)46.744.065.530.746.720.625.028.924.324.7 LLaVA-Interleave (Qwen-7B-DPO)67.744.055.931.349.721.125.930.927.126.3 LLaVA-OneVision (Qwen2-0.5B-SI)27.711.537.340.029.113.921.618.612.916.7 LLaVA-OneVision (Qwen2-0.5B-OV)57.323.533.622.034.116.726.716.518.619.6 LLaVA-OneVision (Qwen2-7B-SI)68.346.540.936.047.917.224.125.825.723.2 LLaVA-OneVision (Qwen2-7B-OV)83.378.057.724.760.918.328.429.918.623.8 LLaVA-OneVision (Qwen2-7B-OV-Chat)82.777.043.626.757.520.030.227.821.424.9 InternVL3.5-1B44.041.030.542.039.421.721.619.621.421.1 InternVL3.5-2B51.737.536.826.038.015.028.416.517.119.3 InternVL3.5-4B59.071.060.022.753.218.927.634.017.124.4 InternVL3.5-8B Pretrained72.062.059.530.055.922.826.727.814.322.9 InternVL3.5-8B Instruct76.059.575.530.760.425.629.329.98.623.3 InternVL3.5-8B MPO76.760.076.418.057.826.129.328.910.023.6 InternVL3.5-8B34.055.534.120.035.930.037.132.011.427.6 Mantis-8B (CLIP-Llama3)45.757.036.818.739.522.825.921.630.025.1 Mantis-8B (SIGLIP-Llama3)47.045.042.724.739.820.625.033.022.925.4 MiniCPM-Llama3-V-2.535.016.525.54.720.44.45.28.25.75.9 MiniCPM-V-2.686.375.573.638.768.521.135.338.118.628.3 Ovis2.5-2B72.023.547.334.044.223.925.918.625.723.5 Ovis2.5-9B78.023.051.828.745.428.927.648.525.732.7 Phi-4-multimodal45.338.033.623.335.125.025.017.528.624.0 Model AttributePosition Overall EasyHNHPNIAvgEasyHNHPNIAvgAvg Overall37.532.931.326.932.446.935.245.222.237.336.1 GPT-565.862.457.045.857.888.570.375.534.167.163.1 Gemini-2.5 Pro62.957.164.347.557.980.866.481.437.666.664.4 Qwen2-VL-2B42.334.250.025.037.946.238.474.422.045.338.0 Qwen2.5-VL-3B47.338.847.126.740.073.850.472.122.054.644.3 Qwen2-VL-7B58.836.244.328.341.986.256.879.521.761.149.1 Qwen2.5-VL-7B52.742.138.126.739.976.252.460.521.452.646.5 LLaVA-v1.6 (Mistral-7B)30.421.724.3-25.429.225.231.2-28.528.8 LLaVA-Interleave (Qwen-0.5B)41.531.731.417.530.515.017.640.934.527.027.2 LLaVA-Interleave (Qwen-7B)27.327.529.015.824.931.224.832.121.427.430.9 LLaVA-Interleave (Qwen-7B-DPO)28.528.828.613.324.837.126.430.218.528.132.2 LLaVA-OneVision (Qwen2-0.5B-SI)22.317.927.112.520.018.315.634.916.821.421.8 LLaVA-OneVision (Qwen2-0.5B-OV)23.117.128.122.522.719.221.233.522.924.225.2 LLaVA-OneVision (Qwen2-7B-SI)24.221.224.328.324.537.530.442.827.134.432.5 LLaVA-OneVision (Qwen2-7B-OV)30.424.627.123.326.447.924.843.721.134.436.4 LLaVA-OneVision (Qwen2-7B-OV-Chat)30.825.026.722.526.248.325.643.321.034.535.8 InternVL3.5-1B28.824.227.117.524.457.137.225.132.738.030.7 InternVL3.5-2B40.437.915.235.032.160.043.613.518.433.930.8 InternVL3.5-4B42.339.234.835.838.031.718.857.219.231.736.8 InternVL3.5-8B Pretrained40.837.519.538.334.037.532.037.228.133.736.6 InternVL3.5-8B Instruct38.134.618.139.232.535.433.237.719.831.536.9 InternVL3.5-8B MPO37.742.914.839.233.636.234.020.018.527.235.5 InternVL3.5-8B34.627.528.637.532.055.836.435.835.040.834.1 Mantis-8B (CLIP-Llama3)24.624.624.820.023.525.825.231.620.625.828.5 Mantis-8B (SIGLIP-Llama3)26.229.226.220.025.431.222.828.819.525.629.0 MiniCPM-Llama3-V-2.531.230.828.69.224.944.644.853.515.039.522.7 MiniCPM-V-2.653.843.333.821.738.279.662.069.325.259.048.5 Ovis2.5-2B26.242.934.321.731.335.020.840.031.031.732.7 Ovis2.5-9B39.627.933.826.732.051.224.444.237.639.437.3 Phi-4-multimodal34.626.721.035.029.342.538.439.522.735.831.0 Table 2. MLLM performance on MIOH benchmark. HN: Hard Negatives, HP: Hard Positives, NI: Number of Images. Gemini-2.5 Pro GPT-5 Qwen2-VL-7B Qwen2.5-VL-7B MiniCPM-V-2.6 Qwen2.5-VL-3B InternVL3.5-8B Instruct InternVL3.5-8B Pretrained InternVL3.5-4B LLaVA-OneVision (Qwen2-7B-OV) InternVL3.5-8B MPO LLaVA-OneVision (Qwen2-7B-OV-Chat) Qwen2-VL-2B Ovis2.5-9B InternVL3.5-8B LLaVA-OneVision (Qwen2-7B-SI) LLaVA-Interleave (Qwen-7B-DPO) Ovis2.5-2B InternVL3.5-2B LLaVA-Interleave (Qwen-7B) InternVL3.5-1B Mantis-8B (SIGLIP-Llama3) Phi-4-multimodal Mantis-8B (CLIP-Llama3) LLaVA-v1.6 (Mistral-7B) LLaVA-Interleave (Qwen-0.5B) LLaVA-OneVision (Qwen2-0.5B-OV) LLaVA-OneVision (Qwen2-0.5B-SI) MiniCPM-Llama3-V-2.5 Models Existence Counting Attribute Position Existence Counting Attribute Position Existence Counting Attribute Position 0.0 0.2 0.4 0.6 0.8 1.0 Average Performance Score 0.519 0.313 0.372 0.355 0.545 0.237 0.357 0.417 0.473 0.217 0.267 0.421 Avg Comprehensive Comparison Selection Figure 3. Performance heatmap across multi-image reasoning patterns (Comprehensive, Comparison, Selection) and object-centric tasks (Existence, Counting, Attribute, Position). 4.3. Task-Specific Vulnerability We conduct detailed analyses to identify specific failure pat- terns and characterize how different adversarial pressures affect model behavior in multi-image scenarios. As re- ported in Tab. 2, the overall task-level accuracy reveals a clear hierarchy of difficulty. Existence (49.4%) emerges as the most manageable task for most models, suggesting a basic level of object recognition. In stark contrast, counting (25.4%) is a critical failure point across the board, high- lighting a fundamental weakness in quantitative reasoning under multi-image contexts. Attribute (32.4%) and posi- tion (37.3%) tasks reveal significant difficulty as well, un- derscoring the challenges of compositional understanding. Existence. While Existence is the highest-scoring task, model performance is brittle. The accuracy on easy sam- ples (62.4%) consistently drops on perceptually challenging objects (HP, 51.5%) or on contextually misleading scenes (HN, 51.9%). This indicates that models’ understanding of object presence is heavily reliant on clear, unambiguous vi- sual cues and can be easily disrupted, leading to false nega- tive hallucinations. Counting. The average accuracy of 25.4% confirms that counting is a core deficiency. Our sub-task analysis reveals this is not limited to one failure mode; models struggle with both aggregating counts across images and locating an im- age with a specific number of objects. This points to a fun- damental inability in quantitative grounding, especially on multiple scenes. Attribute and Position. These tasks require binding ob- jects to their properties or spatial relations, a challenge of compositional reasoning. The low accuracy in the attribute task (32.4%) suggests that MLLMs may recognize an object and an attribute separately but fail to confirm their visual co- occurrence (e.g., seeing a ‘car’ and ‘red’ but hallucinating a ‘red car’). Similarly, the position task scores slightly higher (37.3%), but degrades under Hard Negative pressure, indi- cating that models often tend to overlook complex spatial relationships beyond mere existence. 4.4. Impact of Adversarial Pressures As reported in Tab. 2, introducing adversarial constraints consistently degrades MLLM performance across all tasks relative to their easy baselines. Perceptual and Contextual Pressures (HP & HN). Per- formance on Existence, Attribute, and Position consistently drops under both HP and HN conditions. For instance, At- tribute accuracy falls from 37.5% to 31.3% (HP), and Po- sition from 46.9% to 35.2% (HN), confirming model vul- nerabilities to misleading visual and contextual cues. In contrast, Counting exhibits only a marginal decline. How- ever, this reflects a floor effect rather than genuine robust- ness; with baseline counting accuracy already near random chance (25.4%), added complexities leave little room for further degradation. Cognitive Load from Multiple Images (NI). While HP and HN induce moderate decays, increasing the number of input images to 8 (NI) triggers a far more severe collapse. Under this heightened cognitive load, even the fundamen- tal Existence task plummets to 30.0%—a relative drop of exceeding 50% from the easy baseline. Although top-tier models like GPT-5 and Gemini 2.5 Pro manage this ex- Figure 4. Accuracy on easy questions and robustness under ad- versarial conditions in VLMs. Circle size indicates model size, where Gemini and GPT size is based on estimation. panded context better than average, their substantial per- formance drops confirm that scaling multi-image reasoning remains a critical open challenge. 4.5. Accuracy-Robustness Trade-off As shown in Fig. 4, our analysis uncovers a fundamen- tal trade-off between achieving high accuracy on straight- forward tasks and maintaining robust performance under adversarial pressures. In other words, a strong baseline performance does not guarantee robustness against object hallucination. Open-source models with higher baseline accuracy often exhibit greater vulnerability to adversar- ial conditions. The correlation analysis reveals a moder- ate positive relationship between model size and perfor- mance on easy questions, but virtually no correlation be- tween size and robustness, suggesting that simply scaling model parameters does not inherently improve resilience to object hallucination. The top-performing open-source mod- els on easy questions—Qwen2-VL-7B and Qwen2.5-VL- 7B—experience substantial robustness drops of -27.2% and -21.8%, respectively, indicating that high capability models might be more susceptible to the adversarial pressures we designed. This finding suggests that the ability to handle straightforward multi-image tasks does not guarantee ro- bustness against object hallucination. 4.6. Multi-Image Context as a Hallucination Am- plifier: An Ablation Study To isolate the impact of multi-image processing on object hallucination, we conduct a controlled ablation study focus- ing on the Existence task. Specifically, we compare two evaluation approaches for identical visual content: com- prehensive questions that require synthesizing information across all images simultaneously (“Is there an OBJECT in Figure 5. Multi-image processing consistently degrades object hallucination across all models. Comparison of accuracy be- tween multi-image comprehensive (red bars) and single-image de- composed (blue bars) evaluation on Existence task. any of these IMAGEs?”) vs. decomposed questions that mirror the traditional single-image setting by asking each image separately (“Is there a OBJECT in IMAGE?”) and combining the answers to determine overall presence. This design isolates whether multi-image contexts introduce sys- tematic errors beyond simple accumulation of individual image processing mistakes. The results in Fig. 5 reveal that single-image processing substantially outperforms simultaneous multi-image analy- sis, consistently across all models and scales. This indicates that multi-image contexts significantly amplify object hallu- cination beyond what error accumulation alone would pre- dict, pointing to cross-image integration as a primary source of failure. The consistency of this penalty suggests that cur- rent MLLM training paradigms fail to adequately address object hallucination in multi-image reasoning. 5. Conclusion We introduce MIOH, a benchmark that systematically eval- uates object hallucination in multi-image contexts across four object-centric tasks (Existence, Counting, Attribute, Position) with three reasoning patterns (Comprehensive, Comparative, Selective) under three adversarial pressures (Visual Context Scale, Perceptual Difficulty, Contextual Bias). Through evaluation of 29 models, we demonstrate that current MLLMs—including GPT-5 and Gemini-2.5- Pro—exhibit fundamental limitations in multi-image inte- gration.Our fine-grained diagnostic framework reveals that hallucination stems not merely from perceptual fail- ures but from integration-stage breakdowns when maintain- ing object representations across multiple images. MIOH provides a foundation for targeted improvements in multi- image understanding and serves as a critical evaluation tool for developing more reliable multimodal AI systems. Acknowledgments This work was also supported by the SOFT Foundry Insti- tute at SNU, Samsung Electronics, Youlchon Foundation, National Research Foundation of Korea (NRF) grants (RS- 2021-NR05515, RS-2024-00336576, RS-2023-0022663, RS-2025-25399604, RS-2024-00333484), and the Institute for Information & Communication Technology Planning & Evaluation (IITP) grants (RS-2022-I220264, RS-2024- 00353131) funded by the Korean government. References [1] Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi- 4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv:2503.01743, 2025. 5 [2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS, 2022. 2 [3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wen- bin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al.Qwen2.5-VL technical report. arXiv:2502.13923, 2025. 2, 5 [4] Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal llm. In CVPR, 2024. 2 [5] Xuweiyi Chen, Ziqiao Ma, Xuejun Zhang, Sihan Xu, Shengyi Qian, Jianing Yang, David Fouhey, and Joyce Chai. Multi-object hallucination in vision language models. In NeurIPS, 2024. 4 [6] Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blis- tein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodal- ity, long context, and next generation agentic capabilities. arXiv:2507.06261, 2025. 2, 5 [7] Wenliang Dai, Zihan Liu, Ziwei Ji, Dan Su, and Pascale Fung. Plausible may not be faithful: Probing object hallu- cination in vision-language pre-training. arXiv:2210.07688, 2022. 2 [8] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision- language models with instruction tuning. In NeurIPS, 2023. 2 [9] Shounak Datta and Dhanasekar Sundararaman.Evaluat- ing hallucination in large vision-language models based on context-aware object similarities. arXiv:2501.15046, 2025. 4 [10] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al.MME: A comprehensive evalu- ation benchmark for multimodal large language models. arXiv:2306.13394, 2023. 2, 4 [11] Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. In ECCV, 2024. 1, 2 [12] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the V in VQA matter: El- evating the role of image understanding in visual question answering. In CVPR, 2017. 2 [13] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. HallusionBench: an advanced diag- nostic suite for entangled language hallucination and visual illusion in large vision-language models. In CVPR, 2024. 2 [14] Yixiao He, Haifeng Sun, Pengfei Ren, Jingyu Wang, Huazheng Wang, Qi Qi, Zirui Zhuang, and Jing Wang. Eval- uating and mitigating object hallucination in large vision- language models: Can they still see removed objects? In NAACL, 2025. 2 [15] Drew A Hudson and Christopher D Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. i [16] Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. MANTIS: Interleaved multi- image instruction tuning. arXiv:2405.01483, 2024. 1, 2, 5 [17] Liqiang Jing, Ruosen Li, Yunmo Chen, and Xinya Du. Faith- score: Fine-grained evaluations of hallucinations in large vision-language models.In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 5042– 5063, 2024. 4 [18] Prannay Kaul, Zhizhong Li, Hao Yang, Yonatan Duk- ler, Ashwin Swaminathan, CJ Taylor, and Stefano Soatto. Throne: An object-based hallucination benchmark for the free-form generations of large vision-language models. In CVPR, 2024. 1, 2, 4 [19] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32–73, 2017. 5, i [20] Hugo Laurenc ̧on, Andr ́ es Marafioti, Victor Sanh, and L ́ eo Tronchon.Building and better understanding vision-language models:insights and future directions. arXiv:2408.12637, 2024. 2 [21] Hyun Lee, Hyemin Jeong, Yejin Kim, Hyungwook Choi, Hyunsoo Cho, Sookyung Kim, and Joonseok Lee. A more word-like image tokenization for mllms. In CVPR, 2026. 2 [22] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension.arXiv:2307.16125, 2023. 2 [23] Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang, Zixian Ma, Simran Khanuja, Ranjay Krishna, Graham Neubig, and Deva Ramanan. Naturalbench: Evalu- ating vision-language models on natural adversarial samples. In NeurIPS, 2024. 2, 4 [24] Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. LLaVA-NeXT- interleave: Tackling multi-image, video, and 3D in large multimodal models. arXiv:2407.07895, 2024. 2, 5 [25] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023. 2 [26] Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, Han- wang Zhang, and Yueting Zhuang.Fine-tuning multi- modal LLMs to follow zero-shot demonstrative instructions. arXiv:2308.04152, 2023. 1, 2 [27] Jiale Li, Mingrui Wu, Zixiang Jin, Hao Chen, Jiayi Ji, Xi- aoshuai Sun, Liujuan Cao, and Rongrong Ji. MIHBench: Benchmarking and mitigating multi-image hallucinations in multimodal large language models. In ACM M, 2025. 1, 2 [28] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv:2305.10355, 2023. 1, 2, 4 [29] Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spa- tial reasoning. Transactions of the Association for Computa- tional Linguistics, 11:635–651, 2023. 2 [30] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 2 [31] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024. 2 [32] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. MMBench: Is your multi-modal model an all-around player? In ECCV, 2024. 2, 4 [33] Ziyu Liu, Tao Chu, Yuhang Zang, Xilin Wei, Xiaoyi Dong, Pan Zhang, Zijian Liang, Yuanjun Xiong, Yu Qiao, Dahua Lin, et al. Mmdu: A multi-turn multi-image dialog under- standing benchmark and instruction-tuning dataset for lvlms. Advances in Neural Information Processing Systems, 37: 8698–8733, 2024. 1, 2 [34] Zhaochen Liu, Kaiwen Gao, Shuyi Liang, Bin Xiao, Li- meng Qiao, Lin Ma, and Tingting Jiang. Beyond the visible: Benchmarking occlusion perception in multimodal large lan- guage models. arXiv:2508.04059, 2025. 4 [35] Holy Lovenia, Wenliang Dai, Samuel Cahyawijaya, Ziwei Ji, and Pascale Fung. Negative object presence evaluation (nope) to measure object hallucination in vision-language models. arXiv:2310.05338, 2023. 4 [36] Shiyin Lu, Yang Li, Yu Xia, Yuwei Hu, Shanshan Zhao, Yan- qing Ma, Zhichao Wei, Yinglun Li, Lunhao Duan, Jianshan Zhao, et al. Ovis2.5 technical report. arXiv:2508.11737, 2025. 2, 5 [37] Fanqing Meng, Jin Wang, Chuanhao Li, Quanfeng Lu, Hao Tian, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, et al. Mmiu: Multimodal multi-image understanding for evaluating large vision-language models. arXiv:2408.02718, 2024. 1, 2 [38] Yannic Neuhaus and Matthias Hein. RePOPE: Impact of an- notation errors on the POPE benchmark. arXiv:2504.15707, 2025. i [39] Jae Sung Park, Zixian Ma, Linjie Li, Chenhao Zheng, Cheng-Yu Hsieh, Ximing Lu, Khyathi Chandu, Quan Kong, Norimasa Kobori, Ali Farhadi, et al.Synthetic visual genome. In CVPR, 2025. 5, i [40] Genevieve Patterson and James Hays. COCO attributes: At- tributes for people, animals, and objects. In ECCV, 2016. i [41] Han Qiu, Jiaxing Huang, Peng Gao, Qin Qi, Xiaoqin Zhang, Ling Shao, and Shijian Lu. LongHalQA: Long-context hal- lucination evaluation for multimodal large language models. arXiv:2410.09962, 2024. 4 [42] Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Mar- quez, Rama Kovvuri, Abhishek Kadian, et al. PACO: Parts and attributes of common objects. In CVPR, 2023. 4, i [43] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image cap- tioning. arXiv:1809.02156, 2018. 1, 2, 4 [44] Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan ̈ O Arık, and Tomas Pfister. Mitigating ob- ject hallucination in mllms via data-augmented phrase-level alignment. arXiv:2405.18654, 2024. 2 [45] Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267, 2025. 5 [46] Shweta Singh, Aayan Yadav, Jitesh Jain, Humphrey Shi, Justin Johnson, and Karan Desai. Benchmarking object de- tectors with COCO: A new path forward. In ECCV, 2024. 4, i [47] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al.Aligning large multimodal models with factually augmented rlhf. arXiv:2309.14525, 2023. 1, 2, 4 [48] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In CVPR, 2024. 2 [49] Andr ́ es Villa, Juan L ́ eon, Alvaro Soto, and Bernard Ghanem. Behind the magic, merlim: Multi-modal evaluation bench- mark for large image-language models. In CVPR, 2025. 4 [50] Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, et al. Muirbench: A comprehensive benchmark for robust multi-image understanding. arXiv:2406.09411, 2024. 1, 2, 4 [51] Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, et al.AMBER: An LLM-free multi- dimensional benchmark for mllms hallucination evaluation. arXiv:2311.07397, 2023. 1, 2 [52] Lei Wang, Jiabang He, Shenshen Li, Ning Liu, and Ee-Peng Lim. Mitigating fine-grained hallucination by fine-tuning large vision-language models with caption rewrites. In In- ternational Conference on Multimedia Modeling, 2024. 4 [53] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiX- uan, et al. CogVLM: Visual expert for pretrained language models. In NeurIPS, 2024. 2 [54] Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. InternVL3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv:2508.18265, 2025. 2, 5 [55] Wenbo Wei, Jun Wang, and Abhir Bhalerao. Coco-olac: A benchmark for occluded panoptic segmentation and image understanding. In ICASSP, 2025. 4 [56] Tsung-Han Wu, Giscard Biamby, Jerome Quenum, Ritwik Gupta, Joseph E Gonzalez, Trevor Darrell, and David M Chan.Visual haystacks: A vision-centric needle-in-a- haystack benchmark. arXiv:2407.13766, 2024. 2, 4 [57] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. MiniCPM-V: A GPT-4V level mllm on your phone. arXiv:2408.01800, 2024. 2, 5 [58] Jiarui Zhang, Jinyi Hu, Mahyar Khayatkhoei, Filip Ilievski, and Maosong Sun. Exploring perceptual limitation of mul- timodal large language models. arXiv:2402.07384, 2024. 2, 4 [59] Mengxi Zhang, Wenhao Wu, Yu Lu, Yuxin Song, Kang Rong, Huanjin Yao, Jianbo Zhao, Fanglong Liu, Haocheng Feng, Jingdong Wang, et al. Automated multi-level prefer- ence for mllms. NeurIPS, 2024. 2 [60] Linxi Zhao, Yihe Deng, Weitong Zhang, and Quanquan Gu. Mitigating object hallucination in large vision-language models via image-grounded guidance. In ICML, 2025. 2 [61] Yunpu Zhao, Rui Zhang, Junbin Xiao, Changxin Ke, Ruibo Hou, Yifan Hao, Qi Guo, and Yunji Chen. Towards analyzing and mitigating sycophancy in large vision-language models. arXiv:2408.11261, 2024. 2 [62] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.MiniGPT-4:Enhancing vision- language understanding with advanced large language mod- els. arXiv:2304.10592, 2023. 2 Fine-Grained Multi-Image Object Hallucination Benchmark Supplementary Material A. Detailed Benchmark Construction In this section, we provide comprehensive details on our benchmark construction process, including: the curation and selection criteria for our datasets (Sec. A.1), the meta- data construction methodology (Sec. A.2), the automated question generation framework (Sec. A.3), and the adver- sarial benchmark design strategy (Sec. A.4). A.1. Dataset Curation A.1.1. COCO-ReM Annotation Quality Issues in Original COCO. The orig- inal COCO dataset annotations contain significant gaps that make them unsuitable for reliable object hallucina- tion evaluation. These issues include incomplete object masks, missing instances, and inaccurate bounding boxes that would introduce systematic errors in our rule-based question generation framework. COCO-ReM Improvements.COCO-ReM (Refined Masks) [46] addresses these limitations through a compre- hensive re-annotation process: (1) Mask boundary refine- ment using the Segment Anything Model (SAM) to improve precision, (2) Missing instance detection using advanced detection models to identify previously unlabeled objects, (3) Label correction through systematic review and human validation, and (4) Enhanced object masks and bounding boxes providing more complete scene coverage. Impact on Benchmark Quality. As demonstrated in Re- POPE [38], high-reliability annotations significantly impact ground truth accuracy, making this a crucial consideration for benchmark design. The enhanced annotation quality in COCO-ReM ensures our existence and counting ques- tions have reliable ground truth labels, substantially reduc- ing false negatives that could arise from missed objects in original COCO annotations. Object Count Limitations. During validation, we ob- served that even COCO-ReM’s accuracy degrades when ob- ject counts exceed certain thresholds. Specifically, images containing more than 10 objects showed decreased annota- tion reliability. To maintain benchmark integrity, we imple- mented a conservative approach by limiting counting ques- tions to images with 5 or fewer objects, ensuring high relia- bility through validated annotations while preserving suffi- cient complexity for meaningful MLLM evaluation. A.1.2. PACO Limitations of Existing Attribute Datasets. While vari- ous datasets address object attributes, they suffer from sys- tematic limitations: (1) Original COCO annotations lack standardized attribute labeling across object categories, (2) COCO Attributes [40] provides standardized annotation but suffers from limited diversity in both object categories and attribute types, and (3) Insufficient coverage for comprehen- sive benchmark construction requiring comparison across diverse objects and attributes. PACO’s Comprehensive Approach. PACO (Parts and At- tributes of Common Objects) [42] provides a superior so- lution through: (1) Broader category coverage spanning a more diverse range of object types, (2) Systematic attribute annotation ensuring consistency across identical objects, (3) Detailed annotation process that identifies constituent ob- ject parts and labels their diverse attributes, and (4) Large- scale structured dataset resulting in comprehensive fine- grained object understanding capabilities. Advantages for Question Generation. PACO’s structured approach offers several key benefits: systematic attribute labeling with sufficient scale and diversity to support ro- bust question generation, extensive object-attribute combi- nations enabling comprehensive evaluation across diverse visual scenarios, standardized annotation framework ensur- ing consistent evaluation criteria across different object cat- egories, and high-quality ground truth reducing ambiguity in attribute-based question validation. A.1.3. SVG Limitations of Existing Spatial Relation Datasets. Exist- ing datasets for spatial relationship evaluation suffer from critical annotation gaps: Visual Genome [19] and GQA [15] provide relation data but have incomplete spatial relation- ship coverage, missing relationships in ground truth anno- tations that exist visually but are not labeled, and annotation inconsistencies that reduce reliability for systematic evalu- ation. SVG’s Multifaceted Approach. SVG (Synthetic Visual Genome) [39] addresses these limitations through compre- hensive methodology: object detection integration for ac- curate entity identification, scene graph enhancement to capture missing relationships, region descriptions provid- ing contextual relationship validation, depth information enabling more accurate spatial reasoning, region masks for precise relationship localization, VQA-based verification for non-spatial relationships to ensure annotation quality, and systematic filtering to reduce incorrect relationship an- notations. Key Advantages for Spatial Evaluation. SVG provides several critical improvements: (1) Richer spatial relation coverage per subject compared to existing datasets, en- abling more comprehensive spatial reasoning evaluation, (2) Comprehensive filtering that systematically reduces in- correct relationships, improving ground truth reliability, (3) Region mask-based verification enabling more reli- able relationship identification through visual evidence, and (4) High relation density minimizing the critical impact of missing positional relationships on question accuracy. These enhancements make SVG particularly well-suited for generating position-based questions that can reliably assess MLLM spatial reasoning capabilities in multi-image con- texts, where accurate relationship identification becomes even more challenging due to increased visual complexity. A.2. Metadata Construction Having established our data sources, we now detail the metadata construction process that enables efficient ques- tion generation. Hierarchical Organization Structure. Our metadata fol- lows a systematic three-level organization: (1) Task-specific property categorization where objects are categorized by relevant attributes, relations, or counts, (2) Difficulty level classification with Easy/Hard Negative/Hard Positive as- signments based on visual and semantic complexity, and (3) Image identifier mapping where specific image IDs are linked to categorized objects for efficient retrieval. Rule-Based Filtering Criteria. We implement several fil- tering mechanisms: minimum bounding box size require- ments to ensure object visibility, occlusion level thresholds based on mask overlap calculations, and image resolution considerations for consistent object detectability across dif- ferent image qualities. For difficulty classification, we de- fine easy positives/negatives as clear, unambiguous cases with high visibility and minimal contextual confusion, hard positives as present objects with small size, high occlusion, or minimal contextual cues, and hard negatives as absent objects in contexts with high co-occurrence bias or seman- tic similarity. CLIP-Based Semantic Similarity Implementation. Our similarity score calculation involves text prompt gen- eration using standardized formats (“A photo of [ob- ject]”,“[attribute] [object]”), image encoding through CLIP visual encoder, cosine similarity computation between text and image embeddings, and threshold determination through empirical validation on representative samples. This metadata system enables rapid question synthesis while maintaining quality through automated filtering based on rule-based criteria, semantic validation using CLIP simi- larity scores, systematic difficulty categorization across dif- ferent visual reasoning scenarios, and efficient question generation through pre-computed metadata lookup. A.3. Question Generation Using the prepared positive and negative samples per task, we generate multi-image questions that instantiate the three types of questions (comprehensive, comparative, and selec- tive) across multi-image contexts for each of the four core tasks. As all ingredients are ready, this step can be eas- ily automated by rule-based templates; e.g., for Counting, we construct a question by randomly selecting N IMAGEs containing a particular OBJECT, and the sum of COUNT per each example is the right answer. Other tasks follow a sim- ilar procedure to generate the question and correct answer. All questions are constructed in multiple-choice (MCQ) for- mat by incorporating incorrect answers. We also include the “None of the above” options to more precisely assess the model’s understanding. A.4. Adversarial Example Construction Building upon the adversarial pressures described in Sec. 3.3, we detail the implementation procedures for con- structing challenging evaluation examples. A.4.1. Hard Positive Example We employ two complementary filtering approaches to identify perceptually difficult positive examples: Rule- based filtering selects (IMAGE, OBJECT) pairs where the target object exhibits challenging visual characteristics. We filter based on bounding box dimensions relative to image size, segmentation mask area indicating occlusion levels, and spatial positioning within the image frame. This ap- proach captures objects that are inherently difficult to per- ceive due to size or visibility constraints. CLIP-based semantic filtering identifies cases where visual-semantic alignment is weak. We compute similarity scores between image embeddings and text prompts formatted as “A photo of OBJECT” using CLIP. Examples with unexpectedly low similarity scores despite object presence indicate perceptual ambiguity—such as unusual viewpoints, partial visibility, or atypical visual contexts that challenge standard recogni- tion patterns. A.4.2. Hard Negative Example We construct misleading negative examples through two strategies: Co-occurrence-based selection leverages sta- tistical patterns from COCO training data. We compute pairwise co-occurrence probabilities P(object A |object B ) across all object categories. For a target object, we select images containing frequently co-occurring objects while the target itself is absent, creating scenarios where strong con- textual priors may mislead models into false positive pre- dictions. CLIP-based semantic confusion identifies im- ages that exhibit high visual-semantic similarity with target object prompts despite the object’s absence. We compute CLIP similarity between images and “A photo of OBJECT” prompts for absent objects, selecting cases with high simi- larity scores. These examples represent strong false associ- ation triggers where visual context strongly suggests object presence without actual visual evidence. GPT-5 Gemini-2.5-Pro Qwen2-VL-7B MiniCPM-V-2.6 Qwen2.5-VL-7BQwen2.5-VL-3B Ovis2.5-9B InternVL3.5-8B-Instruct LLaVA-OneVision-Qwen2-7B-OV Qwen2-VL-2B LLaVA-OneVision-Qwen2-7B-OV-Chat InternVL3.5-8B-Pretrained InternVL3.5-8B-MPO InternVL3.5-4BInternVL3.5-2BInternVL3.5-8B Ovis2.5-2B LLaVA-Interleave-Qwen-7B-DPO LLaVA-OneVision-Qwen2-7B-SI InternVL3.5-1B Phi-4-multimodal LLaVA-Interleave-Qwen-7B Mantis-8B-SIGLIP-Llama3 Mantis-8B-CLIP-Llama3 LLaVA-v1.6-Mistral-7B LLaVA-Interleave-Qwen-0.5B LLaVA-OneVision-Qwen2-0.5B-OV MiniCPM-Llama3-V-2.5 LLaVA-OneVision-Qwen2-0.5B-SI Models Existence Counting Attribute Position Comprehensive Comparison Selection Easy HN HP NI(8) 0.0 0.2 0.4 0.6 0.8 1.0 Performance Score 0.624 0.243 0.375 0.469 0.394 0.397 0.342 0.450 0.379 0.408 0.263 Avg Task Reasoning Pattern Adversarial Figure I. Aggregated Performance Heatmap across Task, Reasoning Pattern, and Adversarial Pressure Dimensions. A.4.3. Difficulty Level Integration. During question generation, we systematically control dif- ficulty by varying the proportion of challenging examples. For a question requiring N images, we can include any- where from 0 to N hard positive or hard negative examples, with difficulty increasing as the proportion of challenging examples approaches N . A.4.4. Quality Assurance and Validation To ensure benchmark reliability, we conducted comprehen- sive manual validation and systematic sampling to con- struct a balanced evaluation set. Our automated generation pipeline initially produced over 26,000 questions across all task types and reasoning patterns. Each question underwent independent review by three annotators with expertise in computer vision and visual rea- soning tasks. Annotators assessed: (1) ground truth cor- rectness based on visual evidence, (2) question clarity and lack of ambiguity, (3) validity and distinctiveness of multi- ple choice options, and (4) consistency with source dataset annotations. Disagreements were resolved through major- ity voting, with persistent ambiguities leading to question removal. Despite carefully selecting well-annotated datasets for construction, the validation process revealed that certain task types were still particularly prone to systemic issues. Position questions frequently exhibited ambiguous or un- derspecified spatial relationships, often stemming from in- complete or inconsistent relationship annotations in the source datasets. Similarly, Counting questions continued to suffer from missed instances or miscounted objects, indi- cating that even high-quality datasets contain non-trivial an- notation noise for fine-grained quantitative reasoning. After validation and filtering, approximately 20,000 questions re- mained as the verified question pool. To ensure fair and comprehensive evaluation across all dimensions, we applied stratified sampling to construct the final benchmark with balanced distribution across three key axes: Task Types, Reasoning Patterns, and Adversarial Pressures. This sampling procedure resulted in the final benchmark comprising 3,484 questions across 11,732 im- ages, ensuring both high-quality ground truth through rigor- ous validation and fair evaluation coverage across all bench- mark dimensions. B. Detailed Performance Analysis In this section, we present a comprehensive performance analysis of various multimodal large language models (MLLMs). Before delving into the complex interactions be- tween different factors, we first examine the aggregated per- formance of models across three primary dimensions: Task, Reasoning Pattern, and Adversarial Pressure. Fig. I il- lustrates the performance overview. This visualization al- lows for a direct comparison of how different models handle specific types of challenges independently. B.1. Variation Analysis Fig. I presents marginal performance distributions across each dimension, illustrating their capacity to distinguish model capabilities.Among task types, Position tasks demonstrate the highest variance (σ = 0.163), suggesting that models exhibit notably different levels of spatial un- derstanding ability. In contrast, Counting tasks show the AttributeCountingExistencePosition 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Accuracy (a) ComparisonComprehensiveSelection 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Accuracy (b) Easy HNHPNI(8) 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Accuracy (c) Figure I. Performance mean and variance. (a) Tasks, (b) Reasoning patterns, and (c) Adversarial Pressures. HNHPHP+HNNI(8) Attribute Counting Existence Position -0.031-0.069-0.063-0.131 -0.008-0.036-0.040-0.108 -0.047-0.070-0.128-0.332 -0.080-0.025-0.115-0.267 (a) HNHPHP+HNNI(8) Comparison Comprehensive Selection -0.086-0.101-0.136-0.336 -0.016-0.032-0.061-0.176 -0.075-0.038-0.120-0.315 (b) -0.4 -0.3 -0.2 -0.1 0.0 0.1 Degradation from Easy Figure I. Cross-dimensional degradation analysis. Performance degradation from Easy baseline across (a) Task types and (b) Reasoning dimensions under adversarial pressure. lowest variance (σ = 0.096), indicating that this task type presents consistent difficulty across most models. Similarly, among adversarial pressure conditions, increasing Number of Images(NI) exhibits the lowest variance (σ = 0.079), comparable to Counting tasks, suggesting that extreme con- text pressure leads to uniformly poor performance across models. For reasoning patterns, Selection demonstrates the highest variance (σ = 0.153), suggesting that the ability to accurately identify a specific image among multiple candi- dates varies considerably across different models. B.2. Cross-dimensional Interaction Analysis Fig. I presents degradation heatmaps showing perfor- mance drops from the Easy baseline across different task- pressure and reasoning-pressure combinations, averaged over all models. Increasing Number of Images(NI) causes catastrophic degradation across all dimensions, with Com- parison reasoning and Existence tasks most severely af- fected. Combined(HP+HN) pressure shows additive dif- ficulty, causing larger drops than either pressure alone. Comprehensive reasoning demonstrates superior robustness compared to Comparison and Selection, suggesting holistic reasoning strategies better withstand adversarial pressure. Counting tasks show minimal degradation not due to robust- ness but floor effects, as their Easy baseline is already low. These patterns reveal that adversarial robustness is highly dimension-dependent. B.3. Performance Gap Between Open-Source and Commercial Models We compare the capabilities of leading commercial mod- els and top-performing open-source models in Fig. IV. The performance gap between these two categories is most pro- nounced in Counting and Comparison tasks. These capabil- ities represent the areas where commercial models demon- strate the largest advantages over open-source alternatives, indicating differences in multi-image reasoning capacity. C. Benchmark Examples C.1. Qualitative Examples from MIOH benchmark Figs. VI and VII provide qualitative examples from the MIOH benchmark, illustrating how questions are formu- lated across our four core tasks and various visual adver- sarial pressures. Each example is designed to test a specific Existence_Comprehensive Existence_Comparison Existence_Selection Counting_Comprehensive Counting_Comparison Counting_Selection Attribute_Comprehensive Attribute_Comparison Attribute_Selection Position_Comprehensive Position_Comparison Position_Selection 0.2 0.4 0.6 0.8 1.0 Gemini-2.5-Pro GPT-5 Qwen2-VL-7B MiniCPM-V-2.6 Qwen2.5-VL-7B Figure IV. Performance of commercial and top open-source models. aspect of an MLLM’s object-centric capabilities and robust- ness against perceptual challenges. Existence Tasks.(Fig. VI, top section) assess the model’s ability to verify the presence of objects and pin- point their location within the image set. The Selective questions showcased here require the model to identify the specific image containing the target object among candi- dates. Examples progress from straightforward cases (Easy: clearly visible “donut”) to perceptually challenging scenar- ios. The hard positive example requires detecting a “bench” in a rainy scene, where the object is situated near the green- ery and partially obscured by a person with umbrella, mak- ing it difficult to spot. The Hard Negative example tests ro- bustness against contextual bias, such as identifying a “key- board” in a set of images containing office-like environ- ments or other electronics (e.g., game consoles) that visu- ally resemble the target context but lack the specific object. Counting Tasks. (Fig. VI, bottom section) evaluate quantitative reasoning capabilities through Comprehensive questions that require aggregating counts across all pro- vided images. The Easy examples involve basic enumer- ation, such as counting the occurrences of clearly visible “elephant”s across four frames. The Hard Positive scenario challenges the model to sum the total number of “sand- wiches” across images where objects are heavily occluded, cut into pieces, or cluttered on plates, testing the ability to handle dense visual information. The Hard Negative exam- ple asks for the count of “trains,” where models must dis- tinguish the actual object from contextually similar back- ground elements like tracks or station platforms in the non- target images. Position Tasks.(Fig. VII, left section) present the most complex spatial relationship challenges. We use Selective questions to ask the model to identify the image where a specific relation holds. Examples include Easy scenarios (“dog next to cat”), Hard Negative cases (“chair positioning relative to dog”), and Hard Positive examples (“person next to umbrella”) that test compositional scene understanding beyond simple object detection. Attribute Tasks. (Fig. VII, right section) assess detailed compositional understanding by requiring models to bind visual properties with objects using Comparative questions. The tasks range from detecting visually distinct attributes (Easy: “red scarf”) to more subtle distinctions. The Hard Negative scenario involves identifying a “dark yellow mug” in a cluttered indoor environment where lighting and other objects may create confusion. The Hard Positive example tests the model’s ability to consistently recognize a “white bench” across two different scenes despite variations in per- spective and background. Each example demonstrates the three question types de- signed for multifaceted evaluation: comprehensive (collec- tive understanding across images), comparative (identify- ing differences between images), and selective (retrieving specific images matching descriptions). The progression of difficulty incorporates both visual factors (scale, occlusion, contextual bias) as detailed in Sec. 3.3. C.2. Failure Case Examples of Commercial Models To better understand the limitations of commercial MLLMs, we analyze specific failure cases of GPT-5 and Gemini-2.5-Pro on the MIOH benchmark. Despite their strong baseline performance, these models still exhibit hallucination patterns under visual adversarial pressures. Fig. V illustrates representative error cases where models fail to ground visual evidence correctly within multi-image contexts. For example, in the Attribute-Comprehensive task, models exhibit perceptual blindness or aggregation fail- ures when objects are visually ambiguous or occluded; for instance, both models fail to confirm the presence of “dark gray bowls” in all target images, demonstrat- ing a breakdown in comprehensive verification. The ex- ample in Counting-Comparative task reveals weaknesses in fine-grained classification, where models confuse “wine glasses” with perceptually similar objects like water goblets or tumblers, leading to incorrect frequency comparisons. Most critically, the Position-Selective task exposes severe object and relationship hallucination under the “None of the above” option. When asked for a “cow behind a dog,” mod- els are forced into making a selection, resulting in halluci- nated object identities (e.g., misidentifying a zebra as a cow in Image 4) or ignoring spatial constraints (e.g., selecting a dog-cow pair with incorrect positioning in Image 3), rather than correctly abstaining. Attribute - Comprehensive Is a(n) ‘dark grey bowl’ present in any of these images? A) Yes, all of them B) Yes, some of them C) No, none of them D) I don’t know Counting - Comparative In which image does the ‘wine glass’ appear the most? A) Image 1 B) Image 2 C) Image 3 D) Image 4 E) All the same Position - Selective In which of these images can you find a cow behind a dog? A) Image 1 B) Image 2 C) Image 3 D) Image 4 E) None of the above Figure V. Failure cases of top commercial models (GPT-5, Gemini-2.5-pro). Blue text indicates the Ground Truth (GT). The icons indicate the incorrect choices made by each model. These examples highlight how perceptual difficulty lead to reasoning failures. EASY Hard Positive In which image does a ‘donut’ present? A) Image 1 B) Image 2 C) Image 3 D) Image 4 E) None of the above A) Image 1 B) Image 2 C) Image 3 D) Image 4 E) None of the above In which image does a ‘bench’ present? Hard Negative In which image does a ‘donut’ present? Existence - Selective Hard Negative In which image does a ‘keyboard’ present? A) Image 1 B) Image 2 C) Image 3 D) Image 4 E) None of the above A) Image 1 B) Image 2 C) Image 3 D) Image 4 E) None of the above Counting - Comprehensive Which of the following objects appears a total of 1 time across all four images? A) hot dog B) toaster C) cup D) broccoli E) None of the above EASY Hard Negative In how many of these four images is a ‘train’ present? A) 1 Image B) 2 Images C) 3 Images D) 0 Image E) I don’t know Hard Positive What is the total number of ‘sandwiches’ across all four images? A) 11 B) 15 C) 10 D) 8 E) None of the above In how many of these four images is a(n) ‘elephant’ present? A) 2 Images B) 3 Images C) 0 Image D) 4 Images E) I don’t know EASY Figure VI. Benchmark Examples 1. Existence and Counting Task EASY Hard Positive Hard Negative In which of these images can you find a dog that is next to a cat? A) Image 1 B) Image 2 C) Image 3 D) Image 4 E) None of the above A) Both B) Neither C) Image 1 D) Image 2 Where is a chair that is right of a dog? A) Both B) Neither C) Image 1 D) Image 2 Where is a person that is next to an umbrella? Attribute - Comparative Which of the following is present in Image 1 but not in Image 2? A) light yellow hat B) rattan hat C) red scarf D) light green box E) None of the above Hard Negative Which of the following is present in Image 1 but not in Image 2? A) dark yellow mug B) striped microwave oven C) dark purple car D) light brown mouse (computer equipment) E) None of the above EASY Position - Selective Hard Positive A) Both B) Neither C) Image 1 D) Image 2 Which of the two images a ‘white bench’ is/are present? Figure VII. Benchmark Examples 2. Position and Attribute Task

Instant research discovery