Paper deep dive
VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification
Yixin Wan, Tianle Zheng, Kai-Wei Chang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/9/2026, 6:10:31 AM
Summary
The paper introduces VDiff-Bench, a challenging multiple-choice benchmark for fine-grained Image Difference Identification (IDI) in Multimodal Large Language Models (MLLMs). It comprises 1,756 questions across 10 change categories (e.g., position, texture, OCR) using image pairs with ground-truth-conditioned distractors. Experiments with 11 MLLMs reveal that models struggle significantly with low-level changes (noise, texture), often falsely selecting 'no difference', highlighting a brittleness in comparative visual understanding not captured by standard single-image tasks.
Entities (15)
Relation Signals (11)
VDiff-Bench â coverscategory â Texture
confidence 95% ¡ VDiff-Bench contains ... 10 change categories: ... texture
VDiff-Bench â coverscategory â Noise/Resolution
confidence 95% ¡ VDiff-Bench contains ... 10 change categories: ... noise/resolution
VDiff-Bench â evaluates â Multimodal Large Language Models
confidence 95% ¡ VDiff-Bench provides a targeted diagnostic for evaluating comparative visual understanding in MLLMs
VDiff-Bench â usesmethod â Multiple-Choice Questions
confidence 95% ¡ VDiff-Bench, a challenging multiple-choice benchmark... reformulate the task as a multiple-choice question (MCQ)
Kimi k2.5 â outperforms â Grok-4.3
confidence 90% ¡ Grok 4.3... falling significantly behind large open-source models like Kimi K2.5
Kimi K3 â outperforms â Grok-4.3
confidence 90% ¡ Grok 4.3... falling significantly behind large open-source models like Kimi K2.5 and K3
Grok-4.3 â performspoorlyon â Texture
confidence 90% ¡ Grok 4.3 demonstrate remarkable performance drop on identifying noise and texture differences... only 15.3% accuracy on texture difference category
Grok-4.3 â performspoorlyon â Noise/Resolution
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal Large Language Models (MLLMs) perform strongly on general visual understanding tasks such as visual question answering, yet they often struggle with a basic comparative skill: identifying what has changed between two similar images. We introduce VDiff-Bench, a challenging multiple-choice benchmark for fine-grained Image Difference Identification. VDiff-Bench contains 1,756 four-way questions over image pairs and covers 10 change categories: position, motion, regional image color, overall image color, appearance/disappearance, noise/resolution, texture, substitution/size, OCR/text, and illumination. Each question corresponds to two image inputs with 4 choices: the true difference, two hard negative descriptions, and a "no difference" distractor. To make the task challenging, we specifically curate ground-truth-conditioned negatives that require models to distinguish the actual change from nearby semantic alternatives. Experiments with 11 state-of-the-art open- and closed-source MLLMs show that fine-grained visual comparison remains brittle: models exhibit uneven performance across sources and change categories, with persistent failures on subtle low-level changes like noises and textures. For instance, three 7-8B-scale open-source MLLMs score 52.5-70.6% on semantic changes but only 8.7-33.3% on low-level changes like noise and texture, falsely assuming no changes between two image inputs. Surprisingly, despite strong performance of other closed-source commercial models, Grok 4.3 demonstrate remarkable performance drop on identifying noise and texture differences between images, falling significantly behind large open-source models like Kimi K2.5 and K3. Overall, VDiff-Bench provides a targeted diagnostic for evaluating comparative visual understanding in MLLMs, exposing failures that are not captured by standard single-image vision-language tasks.
Tags
Links
- Source: https://arxiv.org/abs/2609.06245v1
- Canonical: https://arxiv.org/abs/2609.06245v1
Trouble viewing inline? Open PDF directly â
Full Text
62,796 characters extracted from source content.
Expand or collapse full text
VDIFF-BENCH:A CHALLENGING BENCHMARK FOR FINE- GRAINED IMAGE DIFFERENCE IDENTIFICATION Yixin Wan, Tianle Zheng, Kai-Wei Chang Department of Computer Science University of California, Los Angeles elaine1wan,kwchang@cs.ucla.edu ? Only synthetic scenes with simple changes on simple items ? Only surveillance video frames, simple differences Camera angle change instead of movement Over-simple overlap-based metrics (BLEU, METEOR, CIDEr) Over-simple metrics (BLEU, METEOR, CIDEr, ROUGE-L) False Ground Truth ? Over-simple metrics (BLEU, METEOR, CIDEr, ROUGE-L) ? MCQ questions, accurate and clean rule-based metrics After a close, detailed comparison of both images, they are completely identical. VDiff-Bench(Ours) Benchmark is challenging: Differences are of high granularity CLEVR-Change The main differences come down to a change in camera perspective and the rearrangement of the foreground shapes: In the first image, the camera is at a higher angle looking down at the objects ... In the second image, the cyan cylinder is now on the left... Benchmark is too easy: Differences are too simple to perceive! 3.5-flash Only 3D-style image data with unclear categorization CLEVR-Change Data level GT: "the matte cylinder moved" Metrics level Spot-the-Diff Data level GT: "the second picture is missing a man wearing a white shirt..." Metrics level OmniDiff Data level GT: "two mice on the right side of the table have disappeared..." Metrics level Differences are too simple 2D- and 3D-style data with expert verification VDiff-Bench (Ours) Data level GT: "the airplane tail circle marking is darker in the first image, ..." Metrics level Challenging differences OmniDiff The main difference is the items placed on the desk mat in the foreground: In image 1, there is a black digital camera on the left side of the desk mat. In image 2, the camera is missing. There are two computer mice placed on the right side. Benchmark is too easy: Differences are too simple to perceive! 3.5-flash 3.5-flash Figure 1: Comparison with prior visual difference identification benchmarks. Existing benchmarks primarily focus on open-ended difference captioning and are often limited in visual domains, change categories, or evaluation reliability. VDIFF-BENCH expands to 10 fine-grained change categories and evaluates models through challenging multiple- choice questions for more deterministic and objective IDI performance quantization. ABSTRACT Multimodal Large Language Models (MLLMs) perform strongly on general visual understanding tasks such as visual question answering, yet they often struggle with a basic comparative skill: iden- tifying what has changed between two similar images. We introduce VDIFF-BENCH 1 , a challeng- ing multiple-choice benchmark for fine-grained Image Difference Identification. VDIFF-BENCH contains 1,756 four-way questions over image pairs and covers 10 change categories: position, motion, regional image color, overall image color, appearance/disappearance, noise/resolution, tex- ture, substitution/size, OCR/text, and illumination. Each question corresponds to two image inputs with 4 choices: the true difference, two hard negative descriptions, and a âno differenceâ distrac- tor. To make the task challenging, we specifically curate ground-truth-conditioned negatives that require models to distinguish the actual change from nearby semantic alternatives. Experiments with 11 state-of-the-art open- and closed-source MLLMs show that fine-grained visual compar- ison remains brittle: models exhibit uneven performance across sources and change categories, with persistent failures on subtle low-level changes like noises and textures. For instance, 3 7-8B- scale open-source MLLMs score 52.5â70.6% on semantic changes but only 8.7â33.3% on low-level changes like noise and texture, falsely assuming no changes between two image inputs. Surpris- ingly, despite strong performance of other closed-source commercial models, Grok 4.3 demonstrate remarkable performance drop on identifying noise and texture differences between images, falling significantly behind large open-source models like Kimi K2.5 and K3. Overall, VDIFF-BENCH provides a targeted diagnostic for evaluating comparative visual understanding in MLLMs, expos- ing failures that are not captured by standard single-image vision-language tasks. Project Page: https://huggingface.co/spaces/elaine1wan/vdiff-bench. 1 We release our benchmark at https://huggingface.co/datasets/elaine1wan/image_diff_data. 1 arXiv:2609.06245v1 [cs.CV] 5 Sep 2026 Gemini 3.1 Pro Kimi K2.5 Gemini 3.5 Flash Doubao Seed 1.6 GPT-5.4 Kimi K3 Grok 4.3 Gemini 2.5 Flash Qwen3-VL-8B InternVL3.5-8B LLaVA-OV-7B 0 20 40 60 80 100 Accuracy (%) 89.6 89.2 88.8 87.7 84.4 84.2 67.3 65.9 58.0 48.9 35.8 Overall accuracyLow-level accuracyRandom (25%)Informed guess (33.3%) Figure 2: Overall and low-level accuracy on VDIFF-BENCH across 11 MLLMs. Bars report overall four-choice accuracy; purple diamonds report accuracy on whole-image color, noise/resolution, texture, and illumination changes. Models are sorted by overall accuracy, and horizontal lines mark the 25% uniform-guess and 33.3% informed-guess baselines. Similar overall scores can conceal substantial low-level deficits. 1INTRODUCTION Multimodal Large Language models (MLLMs) have advanced rapidly on image understanding, achieving strong re- sults on single-image captioning, and visual question answering (Liu et al., 2023; Bai et al., 2023; Yue et al., 2024). However, we found that even the strongest closed-source MLLMs fail on the simple task of identifying differences between two similar imagesâas shown in Figure 1âwhich commonly exist in childrenâs playbooks. We refer to this capability as fine-grained Image Difference Identification (IDI), which requires comparative perception across two views, sensitivity to small localized or low-level differences, and restraint against hallucinating absent changes. Current benchmarks fail to holistically and accurately assess this capability. Existing visual difference bench- marks (Park et al., 2019; Jhamtani & Berg-Kirkpatrick, 2018; Liu et al., 2025) mainly suffer from 2 weaknesses: (1) lack of high-quality, challenging image difference data, and (2) lack of accurate, robust evaluation metrics. For instance, Park et al. (2019) collects scenes synthesized by an image generation engine, but the scenes, objects, and changes are simple and lack diversity. As shown in the second row of the rightmost examples in Figure 1, state- of-the-art MLLMs like Googleâs Gemini 3.5 Kavukcuoglu et al. (2026) can easily verbalize all differences in these images correctly, even listing out detailed camera angle changes that the benchmarkâs original ground truth failed to cover. This suggests that existing benchmarks may no longer be sufficiently challenging to probe the limits of modern MLLMs Nevertheless, stronger and more holistic visual difference identification benchmarks are crucial for improv- ing MLLMs to support fine-grained perception, image and video editing evaluation, and accurate reward modeling for these generative systems. In particular, a strong IDI / IDC model could provide a more grounded signal for tasks like image editing, by comparing visual inputs and identifying what changed, what stayed fixed, and whether the observed changes match the intended transformation. To address the research gap on challenging IDI benchmarks, we introduce VDIFF-BENCH, a diagnostic multiple- choice benchmark designed for this purpose. It contains 1,756 questions over distinct image pairs, organized into 10 change categories: position, motion, regional image color, whole-image color, appearance/disappearance, noise/reso- lution, texture, substitution/size, OCR/text, and illumination. Rather than scoring free-form captions, we reformulate the task as a multiple-choice question (MCQ): each item presents an aligned image pair and asks the model to select the single option that states a real difference, against an explicit âno differenceâ option and several plausible-but-false distractor choices constructed with extensive human verification and correction. For instance, Figure 3 shows two examples in the VDIFF-BENCH benchmark with challenging distractor options. Evaluation across 11 contemporary MLLMs reveals pronounced category-specific brittleness that does not follow a simple proprietary-versus-open-weight divide. Three open 7â8B models achieve 52.5â70.6% accuracy on semantic changes but only 8.7â33.3% on low-level changes, selecting the âno differenceâ distractor on 51.3â80.9% of low-level questions. Specifically, these models falsely select the âno differenceâ distractor choice on 51.3â80.9% of low- level questions, showing major limitation in difference perception capabilities, suggesting that model capacity remains an important bottleneck. Yet scale alone is insufficient: while the larger-scale Kimi K2.5 and Kimi K3 attain 88.8% and 82.8% low-level visual difference accuracy, respectively, Grok 4.3, a closed-source large-scale commercial model, achieves only 40.7%âonly 5.3% accuracy on noise difference category and only 15.3% accuracy on texture difference category. These results are consistent with a two-factor account: scale may raise the attainable ceiling, but training data, learning objectives, and visual encoding might determine whether that capacity translates into precise cross- image comparison. Fine-grained visual difference identification therefore appears not to be an automatic consequence of general multimodal scaling, but a distinct capability that must be explicitly developed and evaluated during training 2 Our contributions are threefold: ⢠We introduce VDIFF-BENCH, a 1,756-question Image Difference Identification (IDI) benchmark spanning ten categories across semantic, textual, and low-level changes. ⢠We formulate IDC into IDI, a multiple choice task with a ground truth answer and three distractor choices, enabling deterministic scoring without a questionable response-level judge proposed by prior works. ⢠We systematically evaluate 11 MLLMs with category-, pair-, and response-level analyses, revealing descrip- tive semanticâlow-level gaps. 2RELATED WORK 2.1GENERAL MLLM BENCHMARKS Evaluation of MLLMsâ general capabilities has largely been conducted around single-image descriptive tasks such as Visual Question-Answering (VQA), Visual Reasoning, etc.. For instance, VQA established the task of answering natural-language questions about images Agrawal et al. (2015), with later benchmarks extending it to compositional and relational reasoning Hudson & Manning (2019). On the reasoning side, previous works have evaluated MLLMâs ability to reason on mathematical and diagrammatic reasoning Lu et al. (2024); Zhang et al. (2024); Wang et al. (2024a), as well as broad college-level multimodal knowledge Yue et al. (2024). These works motivate evaluating MLLMs beyond high-level semantic recognition, especially on tasks requiring subtle visual comparison. 2.2IMAGE DIFFERENCE IDENTIFICATION AND CAPTIONING BENCHMARKS A series of works extend MLLM evaluation to multi-image, comparative scenarios. Specifically, the task of Image difference captioning (IDC) prompts a model to identify changes between paired images. For instance, Spot-the- Diff (Jhamtani & Berg-Kirkpatrick, 2018) introduced image pairs from surveillance footages with crowd-sourced image descriptions. CLEVR-Change (Park et al., 2019) utilized rendered images from synthetic scenes with five object-change types. These datasets test MLLMs on change localization and verbalization, but their domains and change inventories are limited. More recently, OmniDiff (Liu et al., 2025) broadens IDC to real and rendered image pairs across scenarios and change types, with human descriptions. However, it evaluates model-verbalized differences using inaccurate reference-based metrics like BLEU-4 and ROUGE-l, which remain sensitive to paraphrase and do not cleanly attribute omitted, reversed, or unsupported claims. DiffCap-Bench (Wei et al., 2026) addresses this issue by using MLLM judge-reported metrics (Wei et al., 2026), but this method relies heavily on the performance of the LLM judgeâwhile a large body of previous works (Zheng et al., 2023; Wang et al., 2024b; Panickssery et al., 2024; Raina et al., 2024) have revealed significant issues with lack of robustness and biases in LLM judges. VDIFF-BENCH is complementary to both: it does not assess free-form completeness, but converts one selected change into a controlled discrimination problem with deterministic scoring. 2.3MLLMS FOR IMAGE EDITING EVALUATION Automatic evaluation of image editing models has always been a difficult yet important task. Early works (Xu et al., 2023; Kirstain et al., 2023) learn a preference score model from human feedback. However, as the generation scene become increasingly compositional and complex, recent image editing models have widely adopted MLLM judges for evaluating generated image quality (Ye et al., 2025; Li et al., 2025). For image editing evaluation, fine-grained visual comparison is crucial: an evaluator must verify that the requested modification occurred while detecting incorrect, unintended changes (e.g. background). However, MLLM judges constantly fails to accurately describe fine-grained edit-induced visual differences, frequently hallucinating changes (Yosef et al., 2025). This motivates for dedicated benchmarks and methods for the IDC task. 3THE VDIFF-BENCH BENCHMARK The construction of VDIFF-BENCH proceeds in three stages: First, we collect image pairs from existing datasets, manually annotate un-labeled image pair data, as well as create synthetic images to augment under-represented change categories. Second, we synthesize false differences between images as distractor option candidates, and conduct human verification, filtering, and re-writing for low-quality ones. Third, we aggregate the ground truth difference description with the sampled distractor options to construct the multiple choice-format evaluation data in VDIFF- BENCH. Below, we elaborate on our task definition, data sources, and the data construction process. 3 Fine-grained appearance/disappearance A. A small bird near the top center is missing in image B. B. The birds left of the plane are missing in image B. C. There is no difference between the two images. D. The oval wing marking becomes lighter in image B. Model responses: GPT-5.4â A; Geminiâ C; Qwen, InternVL, LLaVAâ B. Low-level visual texture change A. Image B is smoother than image A. B. The womanâs hair changes from blonde to light brown. C. There is no difference between the two images. D. Image B is sharper and more detailed than image A. Model responses: GPT-5.4â A; Gemini, Qwen, InternVL, LLaVAâ C. Figure 3: Each evaluation example in VDIFF-BENCH contains a ground truth difference, two false difference distrac- tor options, and one no-difference distractor option. We show an example of higher-level direct visual change (i.e. changing an object clearly visible) on the left and an example of lower-level visual change (i.e. changing the imageâs trait like texture) on the right. 3.1TASK DEFINITION Each VDIFF-BENCH data entry consists of an ordered image pair (I A ,I B ) and four textual options O = o A ,o B ,o C ,o D . The benchmark assigns one option as a reference-supported difference, two as candidate alter- natives intended to be false, and one fixed distractor option claiming that the two images are completelhy identical. All VDIFF-BENCH pairs contain at least one real change, so the no-difference option is always a distractor. The model returns a label Ëy âA,B,C,D, and we report choice accuracy: Acc = 1 N N X i=1 âŽ[Ëy i = y i ].(1) Among parsed incorrect choices, selecting no difference records a missed-change selection, whereas selecting either candidate alternative records a competing-change selection. 3.2IMAGE-PAIR COLLECTION AND TAXONOMY VDIFF-BENCH collects image-difference pairs organized into 10 difference categories, where each subset denotes a distinct change condition. The 10 categories cover both semantic edits and changes that depend more strongly on low-level comparative vision perception. Category# Q.Representative distinction Position158left/right, nearer/farther, relocated object Motion138action, pose, orientation, or state transition Regional color170localized object or region color Whole-image color147global RGB or color-cast shift Appearance/disappearance201added, removed, or missing entity Noise/resolution150Gaussian noise or resolution degradation Texture177smoothing, pattern, mark, or surface detail Substitution/size199object replacement, count, or scale OCR/text221character-, digit-, or word-level change Illumination195global or scene-level brightness change Total1,756 Table 1: Current VDIFF-BENCH taxonomy. Counts refer to questions; the present inventory contains 1,756 questions over 1,543 distinct image pairs. 3.2.1DATA SOURCES Our raw image data consists of existing annotated and un-annotated pairs, as well as unpaired image data for change augmentation. Appendix A.1 reports the source composition and additional construction details. Existing annotated pairs. We draw from complementary paired-image resources: fixed-camera scenes from Spot- the-Diff (Jhamtani & Berg-Kirkpatrick, 2018), motion-centric edits from MotionEdit (Wan et al., 2025), color and 4 BenchmarkPairs Image RegimeTarget-Change CoverageEvaluation Design Real Edited Rendered 2D puzzle Semantic OCR/ text Global photo. Noise/ res.Texture Hard alternatives Exact scoring Spot-the-Diff (Jhamtani & Berg-Kirkpatrick, 2018) 13,192ââââ CLEVR-Change (Park et al., 2019)79,606âââââââ OmniDiff (Liu et al., 2025)15,598ââââââ DiffCap-Bench (Wei et al., 2026)1,075ââââ VDiff-Bench (ours)1,543â Table 2: Coverage and evaluation design of visual difference benchmarks. Check marks indicate dimensions explicitly included in benchmark construction or evaluated as target differences. âGlobal photo.â includes whole- image color and illumination changes, while âNoise/res.â includes noise or resolution degradation. Hard alternatives are plausible competing change statements, and exact, judge-free scoring denotes direct answer-key matching without reference-caption metrics or a learned evaluator. position edits from OmniEdit (Wei et al., 2025), and illumination, substitution, size, and OCR cases from OmniD- iff (Liu et al., 2025). We use source difference annotations or editing instructions as provenance for the intended change, then normalize each selected statement and map it to the current VDIFF-BENCH taxonomy. Existing un-annotated pairs. Inspired by the âspot-the-differenceâ puzzle games in childrensâ playbooks, we col- lect a set of 196 âfind-the-differenceâ puzzle pairs curated from various online sources. These drawn cartoon scenes contain many small and deliberately challenging visual differences per pair of images. Unpaird image data. Inspired by the user-identified failure modes in image editing models to preserve human skin texture (Smith, 2026), we sample from the FFHQ dataset (Karras et al., 2019) with high-quality facial images and augment them by applying low-level visual changes (see 3.2.2). Additionally, we augment our dataset with state-of- the-art image editing models on scene-text images from MLT19 (Nayef et al., 2019) and TextOCR (Singh et al., 2021), as well as sampled input images from OmniEdit (Wei et al., 2025). 3.2.2DATA AUGMENTATION Programmatic Transformations We augment our evaluation benchmark by applying programmatic transforma- tions on low-level vision features for FFHQ facial images. These include applying sampled Gaussian noise pertur- bations, smoothing texture changes, whole-image RGB shifts, and gamma/linear illumination changes. Because the transformation is applied programmatically, its transformation type and direction natually provides the ground truth for difference captioning. Image Editing Additionally, we augment the under-represented position difference and OCR/text difference im- age data by conducting image editing on source images. For position difference, we sequentially sampled source images from OmniEditâs object-swap subset, generated new image-conditioned position-edit instructions with GPT- 5.4-mini, and applied the first proposed instruction using GPT-image-2 (OpenAI, 2026b). For OCR/text differences, we manually curate localized text-edit instructions for sampled scene-text images and apply them using Gemini-3- Pro-Image (Gemini Team, 2026). Image difference ground truth for these data are directly derived from the editing prompts. Human Annotation and Cross-Validation For existing un-annotated pairs curated puzzle pairs without sufficiently detailed source annotations, we collected ground-truth difference descriptions from volunteer domain experts who are fluent in English. Annotators inspected each ordered image pair side by side and enumerated all visible differences. Each description was required to identify a single change, specify the affected object or region using distinguishing visual attributes and spatial cues, and state explicitly how the first and second images differ. Annotators were asked to capture not only added or missing objects, but also localized changes in color, shape, orientation, and fine-grained pattern or texture, while avoiding speculative or overly vague descriptions. To ensure the quality of the annotated image differences, every annotation subsequently underwent a second review to be cross-validated by a different expert. The reviewer re-examined the image pair for coverage and visual support, and rewrote descriptions that were inaccurate, ambiguous, overly broad, grammatically unclear, or inconsistent in comparison direction. The resulting reviewed descriptions form the ground-truth pool from which reference differences are selected during question construction. 3.3FALSE DIFFERENCE GENERATION To challenge MLLMs on the IDI task, we construct false image differences that are semantically plausible as distractor options for models. 5 Structured False Difference Generation We first generate a set of false differences with reference to the real difference annotations in image pairs. Specifically, we utilize two strong MLLMsâGemini 2.5 Pro and GPT-5.5âby providing them with image pair inputs and their reference ground truth difference lists, and ask it to generate a list of at least 3 false differences intended to be incorrect but visually plausible. We instruct the model to anchor each false difference generation on one ground truth difference, using strategies like applying the reference change to a nearby entity, reversing a direction or state, or substituting a plausible attribute while preserving the scene vocabulary. This procedure is designed to produce candidate alternatives close to the reference in content and phrasing, rather than unrelated answer options. Finally, structured filters remove duplicates and exact truth matches. Human Verification Model-generated false differences might be too semantically unplausible or too vague to be judged, therefore not acting as challenging ânegativeâ choices that an evaluated MLLM needs to distinguish from. Therefore, we invite a human expert to inspect the image pairs and generated false differences and refine, rewrite, or discard low-quality ones. 3.4MULTIPLE-CHOICE QUESTION CONSTRUCTION To construct the final multiple-choice questions in our VDIFF-BENCH dataset, we retain the ground truth difference caption, two false difference statements for each image pair, append the fixed no-difference distractor option, and shuffle the four options to be randomly ordered. To control for option-position bias and rule out fixed response strategies (e.g., always selecting option D), we shuffle the four choices using a fixed random seed; consequently, both the correct-answer labels and the no-difference distractor positions are approximately balanced across AâD. Appendix A.3 provides the prompt and additional audit statistics. 3.5DATASET STATISTICS Our final VDIFF-BENCH benchmark consists of 1,756 questions across 10 image difference categories. Table 1 defines the 10 categories and gives their question counts. The image differences span both semantic changesâ posi- tion, motion, regional color, appearance/disappearance, substitution/size, and OCR/text changesâas well as low-level visual trait changes like whole-image color, noise/resolution, texture, and illumination. 3.6COMPARISON WITH EXISTING BENCHMARKS We compare VDIFF-BENCH against four direct image-difference-captioning benchmarks in Table 2. As the table shows, existing benchmarks provide valuable scale and diversity but leave 2 major gaps. First, none jointly evaluates semantic, textual, and low-level target changes across real, edited, rendered, and densely composed 2D puzzle im- ages. Second, they formulate evaluation as free-form caption generation, requiring either reference-caption metrics or an MLLM judge to determine whether a predicted difference is correct. VDIFF-BENCH addresses the coverage gap by bringing these image regimes and change families into a unified taxonomy, and addresses the evaluation gap by introducing human-verified, reference-conditioned alternatives with exact, judge-free choice scoring. This formula- tion directly tests whether a model can distinguish the observed change from plausible but unsupported alternatives, complementing prior benchmarks that measure the completeness and quality of free-form descriptions. 4EXPERIMENTS 4.1EXPERIMENTAL SETUP We evaluate six proprietary MLLMs: GPT-5.4 (snapshot 2026-03-05) (OpenAI, 2026a), Gemini 2.5 Flash (Co- manici et al., 2025), Gemini 3.1 Pro (Preview) (Gemini Team, 2026), Gemini 3.5 Flash (Kavukcuoglu et al., 2026), Grok 4.3 (xAI, 2026), and Doubao Seed 1.6 Vision (ByteDance Seed, 2025). For Doubao Seed 1.6 Vision, we use the doubao-seed-1-6-vision-250815 snapshot. We additionally evaluate five open-weight MLLMs: Qwen3- VL-8B Thinking (Bai et al., 2025), InternVL3.5-8B (Wang et al., 2025), LLaVA-OneVision-Qwen2-7B (Li et al., 2024), Kimi K2.5 (Kimi Team, 2026a), and Kimi K3 (Kimi Team, 2026b). Generation Setup. Every model receives images A and B in that order together with four labeled options and the instruction to return one label only. For open-source models, we set the generation temperature to 0 Appendix B provides details on the prompt template and run configuration. Evaluation Metrics. We report the answer-key choice accuracy as our main metric. To conduct stratified analysis, we report the overall accuracy, accuracy by change category, as well as aggregated accuracy for semantic difference groups and low-level visual difference groups as defined in Section 3.2. 6 Model Change Category Overall Pos. Motion Reg. col. Whole col. App./dis. Noise Texture Sub./size OCR Illum. Baselines Random25.025.025.025.025.025.025.025.025.025.025.0 Informed guess33.333.333.333.333.333.333.333.333.333.333.3 Text-only style57.633.334.127.261.232.732.835.733.531.338.2 Closed Source Models GPT-5.483.592.892.483.761.7100.083.191.586.076.484.4 Gemini 2.5 Flash57.081.982.957.844.889.326.674.986.061.065.9 Gemini 3.1 Pro88.691.394.786.475.1100.087.694.592.887.289.6 Gemini 3.5 Flash88.088.492.985.779.6100.083.192.589.190.888.8 Grok 4.375.996.489.487.168.75.315.383.990.055.967.3 Doubao Seed 1.6 Vision86.195.794.791.872.697.376.897.592.376.987.7 Open Source Models Qwen3-VL-8B Thinking59.587.786.521.853.245.38.571.483.355.458.0 InternVL3.5-8B56.384.885.36.152.73.33.473.966.544.648.9 LLaVA-OneVision-Qwen2-7B 43.774.673.51.432.87.35.146.252.518.535.8 Kimi K2.584.294.995.993.973.6100.083.197.592.381.589.2 Kimi K380.497.193.582.364.2100.078.589.989.173.884.2 Table 3: Choice accuracy (%) on VDIFF-BENCH. Results are reported by change category and overall; invalid outputs count as incorrect. Random guessing over four options yields 25.0%, while informed guessing uniformly selects among the three descriptive options, exploiting that âno differenceâ is always incorrect, and yields 33.3%. The text-only baseline uses only capitalization and punctuation cues without viewing the images. Bold indicates the best MLLM performance in each column, including ties. 0255075100 Accuracy (%) GPT-5.4 Gemini 2.5 Flash Gemini 3.1 Pro Gemini 3.5 Flash Grok 4.3 Doubao Seed 1.6 Qwen3-VL-8B InternVL3.5-8B LLaVA-OV-7B Kimi K2.5 Kimi K3 41.3 p 37.2 p 53.8 p 43.9 p (a) Semantic vs. low-level accuracy SemanticLow-level 0255075100 Share of low-level responses (%) 21.8 48.1 51.3 76.2 80.9 (b) Low-level response outcomes CorrectNo differenceCandidate alternativeInvalid Figure 4: Semantic and low-level performance across 11 MLLMs. (a) Accuracy on semantic and low-level change groups; connectors expose within-model gaps, labels mark gaps of at least 25 percentage points, and the dashed line denotes the 33.3% informed-guess baseline. (b) Low-level predictions decomposed into correct, no-difference, candidate-alternative, and invalid outcomes; labels mark no-difference rates of at least 15%. 4.2RESULTS BY CHANGE CATEGORY Table 3 presents evaluation results by different change categories, as well as the overall result. Models diverge substantially in overall VDI performance. Overall accuracy ranges from 35.8% to 89.6%, com- pared with the 25% uniform-guess and 33.3% informed-guess baselines. Gemini 3.1 Pro ranks first at 89.6% (pair- clustered 95% CI: 87.9â91.2), narrowly followed by Kimi K2.5 at 89.2% (87.4â91.0), Gemini 3.5 Flash at 88.8% (87.2â90.4), and Doubao Seed 1.6 Vision at 87.7% (85.7â89.7). No model dominates the taxonomy: the best cate- 7 gory scores are distributed across Gemini, Kimi, and Doubao models, indicating that similar aggregate accuracy can conceal distinct perceptual strengths. Model scale is enabling but not sufficient for low-level visual comparison. The 7â8B open-weight models per- form substantially worse on low-level than semantic changes, suggesting a capacity bottleneck at smaller scales. Yet scale alone does not explain the results: Kimi K2.5 and Kimi K3 achieve 88.8% and 82.8% low-level accuracy, re- spectively, whereas Grok 4.3 falls from 82.0% on semantic changes to 40.7% on low-level changes, scoring only 5.3% on noise difference category and 15.3% on texture difference category. This shows that increasing model scale does not guarantee strong VDI capability. Category rankings reveal failure modes. Grok 4.3 reaches 96.4% on motion but only 5.3% on noise/resolution and 15.3% on texture, while Kimi K2.5 leads on regional and whole-image color and ties for the best substitution/size result. Appearance/disappearance remains difficult even for the strongest systems: the category maximum is 79.6%, compared with at least 90.8% in seven other categories. In Appendix C.3, Figure 7, we provide additional qualitative results of different models on data from each change category. ModelPair macro Change GroupError Type SemanticLow-levelOCRNo diff.AlternativeInvalid Closed Source Models GPT-5.486.283.585.186.05.210.40.0 Gemini 2.5 Flash67.467.357.586.012.118.63.4 Gemini 3.1 Pro 91.488.590.092.81.68.30.6 Gemini 3.5 Flash90.488.189.789.10.57.82.9 Grok 4.366.282.040.790.021.810.90.0 Doubao Seed 1.6 Vision 89.688.884.892.31.311.00.0 Open Source Models Qwen3-VL-8B Thinking57.670.633.383.324.817.20.0 InternVL3.5-8B 47.369.716.066.536.814.30.0 LLaVA-OneVision-Qwen2-7B34.552.58.752.543.121.10.0 Kimi K2.5 91.288.888.892.33.47.40.0 Kimi K386.284.182.889.15.94.75.2 Table 4: Grouped performance on VDIFF-BENCH (%). âPair macroâ indicates accuracy averaged over unique image pairs. Group columns report question-level accuracy for semantic, low-level, and OCR changes. Error columns give rates across all questions of selecting no difference, either hard negative, or an invalid response; lower is better. 4.3GROUP-LEVEL PERFORMANCE AND ERROR MODES Table 4 and Figure 4 aggregate model performance into higher-level semantic, low-level, and OCR change groups. Semantic and low-level VDI can dissociate sharply. We observe that Grok and the three 7â8B models exhibit remarkable semanticâlow-level gaps of 37.2â53.8 percentage points. Thus, success on semantic edits does not reliably predict sensitivity to changes in visual appearance, and overall accuracy can conceal qualitatively different capabilities. Low-level failures primarily reflect missed changes. For the 4 models with the largest semanticâlow-level gaps, 48.1â80.9% of low-level questions are answered with the no-difference option, even though every image pair contains a real change. These rates substantially exceed their selection of alternative descriptions, indicating that the dominant failure is often detecting that a subtle change occurred, rather than distinguishing among competing descriptions of it. 5CONCLUSION We introduced VDIFF-BENCH, a diagnostic benchmark for fine-grained visual difference identification in MLLMs. The benchmark contains 1,756 four-choice questions derived from 1,543 image pairs and spans ten semantic, tex- tual, and low-level change categories. By pairing each ground-truth difference with two plausible hard negatives and a no-difference distractor, VDIFF-BENCH enables deterministic scoring and interpretable error analysis. Across 11 con- temporary MLLMs, accuracy ranges from 35.8% to 89.6%, with no model dominating every change category. Most notably, strong semantic comparison does not guarantee low-level sensitivity: Grok and the three 7â8B models exhibit semanticâlow-level gaps of 37.2â53.8 percentage points and frequently fail to register that any change occurred. Con- versely, the strong performance of the Kimi models argues against a simple open- versus closed-source explanation. Together, these findings suggest that fine-grained comparative perception remains a challenging task for MLLMs, yet model capacity alone is not the only reason behind this bottleneck. 8 REFERENCES Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Dhruv Batra, and Devi Parikh. VQA: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, p. 2425â2433, 2015. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025. ByteDance Seed. Technical introduction to the Seed1.6 model series. https://seed.bytedance.com/en/ seed1_6, June 2025. Accessed: 2026-09-05. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blis- tein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multi- modality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. Gemini Team. Gemini 3.1 Pro: A smarter model for your most complex tasks. Google Blog, February 2026. URL https://blog.google/innovation-and-ai/models-and-research/gemini-models/ gemini-3-1-pro/. Published February 19, 2026. Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 6700â6709, 2019. Harsh Jhamtani and Taylor Berg-Kirkpatrick. Learning to describe differences between pairs of similar images. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, p. 4024â4034, 2018. doi: 10.18653/v1/D18-1436. Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), p. 4396â4405. IEEE, 2019. Koray Kavukcuoglu, Jeff Dean, Oriol Vinyals, and Noam Shazeer.Gemini 3.5:Frontier intelligence with action. Google Blog, May 2026.URL https://blog.google/innovation-and-ai/ models-and-research/gemini-models/gemini-3-5/. Published May 19, 2026; accessed August 17, 2026. Kimi Team. Kimi K2.5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276, 2026a. doi: 10.48550/arXiv. 2602.02276. URL https://arxiv.org/abs/2602.02276. Kimi Team. Kimi K3: Open frontier intelligence. arXiv preprint arXiv:2607.24653, 2026b. doi: 10.48550/arXiv. 2607.24653. URL https://arxiv.org/abs/2607.24653. Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. 2023. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaVA-OneVision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. Zongjian Li, Zheyuan Liu, Qihui Zhang, Bin Lin, Shenghai Yuan, Zhiyuan Yan, Yang Ye, Wangbo Yu, Yuwei Niu, and Li Yuan. Uniworld-v2: Reinforce image editing with diffusion negative-aware finetuning and mllm implicit feedback. arXiv preprint arXiv:2510.16888, 2025. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems, volume 36, 2023. Yuan Liu, Saihui Hou, Saijie Hou, Jiabao Du, Shibei Meng, and Yongzhen Huang. OmniDiff: A comprehensive benchmark for fine-grained image difference captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 21440â21449, 2025. 9 Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations, 2024. Nibal Nayef, Yash Patel, Michal Busta, Pinaki Nath Chowdhury, Dimosthenis Karatzas, Wafa Khlif, Jiri Matas, Uma- pada Pal, Jean-Christophe Burie, Cheng-lin Liu, et al. ICDAR2019 robust reading challenge on multi-lingual scene text detection and recognitionâRRC-MLT-2019. In 2019 International Conference on Document Analysis and Recognition, p. 1582â1587, 2019. doi: 10.1109/ICDAR.2019.00254. OpenAI. GPT-5.4 Thinking System Card. Technical report, 2026a. URL https://openai.com/index/ gpt-5-4-thinking-system-card/. OpenAI.GPT-Image-2 Model.OpenAI API documentation, April 2026b.URL https://developers. openai.com/api/docs/models/gpt-image-2. Released April 21, 2026; accessed September 4, 2026. Arjun Panickssery, Samuel R. Bowman, and Shi Feng.Llm evaluators recognize and favor their own gener- ations.In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, p. 68772â68802. Curran Associates, Inc., 2024. doi: 10.52202/079017-2197. URL https://proceedings.neurips.c/paper_files/paper/ 2024/file/7f1f0218e45f5414c79c0679633e47bc-Paper-Conference.pdf. Dong Huk Park, Trevor Darrell, and Anna Rohrbach. Robust change captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 4624â4633, 2019. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable vi- sual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, p. 8748â8763, 2021. Vyas Raina, Adian Liusie, and Mark Gales. Is LLM-as-a-judge robust? investigating universal adversarial attacks on zero-shot LLM assessment. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 7499â7517, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.427. URL https://aclanthology.org/2024.emnlp-main.427/. Amanpreet Singh, Guan Pang, Mandy Toh, Jing Huang, Wojciech Galuba, and Tal Hassner. TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 8802â8812, 2021. Mischa Anouk Smith.AI edited my face without asking: At first i laughed it offâthen i started seeing myself differently. Marie Claire UK, July 2026.URL https://w.marieclaire.co.uk/life/ ai-edited-my-face-without-asking. Published July 11, 2026; accessed August 17, 2026. Yixin Wan, Lei Ke, Wenhao Yu, Kai-Wei Chang, and Dong Yu. Motionedit: Benchmarking and learning motion- centric image editing. arXiv preprint arXiv:2512.10284, 2025. Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Mea- suring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095â95169, 2024a. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 9440â9450, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.511. URL https://aclanthology.org/2024.acl-long.511/. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shen- glong Ye, Jie Shao, et al. InternVL3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025. Cong Wei, Zheyang Xiong, Weiming Ren, Xeron Du, Ge Zhang, and Wenhu Chen. Omniedit: Building image editing generalist models through specialist supervision. In International Conference on Learning Representations, 2025. 10 Yuancheng Wei, Haojie Zhang, Linli Yao, Lei Li, Jiali Chen, Tao Huang, Yiting Lu, Duojun Huang, Xin Li, and Zhao Zhong. DiffCap-Bench: A comprehensive, challenging, robust benchmark for image difference captioning. arXiv preprint arXiv:2605.04503, 2026. xAI. Grok 4.3. https://docs.x.ai/developers/models/grok-4.3, 2026. Accessed: 2026-09-05. Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagere- ward: Learning and evaluating human preferences for text-to-image generation. In Advances in Neural Information Processing Systems, volume 36, p. 15903â15935, 2023. Yang Ye, Xianyi He, Zongjian Li, Bin Lin, Shenghai Yuan, Zhiyuan Yan, Bohan Hou, and Li Yuan. Imgedit: A unified image editing dataset and benchmark. arXiv preprint arXiv:2505.20275, 2025. Ron Yosef, Yonatan Bitton, Dani Lischinski, and Moran Yanuka. EditInspector: A benchmark for evaluation of text- guided image edits. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 29503â29530, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251- 0. doi: 10.18653/v1/2025.acl-long.1428. URL https://aclanthology.org/2025.acl-long.1428/. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 9556â 9567, 2024. Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, p. 169â186. Springer, 2024. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt- bench and chatbot arena. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, p. 46595â46623. Curran Associates, Inc., 2023. doi: 10.52202/075280-2020. URL https://proceedings.neurips.c/paper_files/paper/ 2023/file/91f18a1287b398d378ef22505bf41832-Paper-Datasets_and_Benchmarks.pdf. 11 AADDITIONAL DETAILS ON DATASET CONSTRUCTION A.1SOURCE COMPOSITION Table 5 reports both questions and distinct image pairs. The distinction matters for sources whose pairs contain multiple annotated changes: Spot-the-Diff contributes 108 questions from 55 pairs, MotionEdit contributes 137 from 67, and the kids-games source contributes 102 from 12. Source subsetQuestionsPairsPrimary category/categories Spot-the-Diff10855appearance, position MotionEdit13767motion OmniEdit color137137regional color OmniEdit position120120position Custom illumination4444illumination Kids games10212mixed puzzle differences JustFamilyFun puzzles5656mixed puzzle differences Kids puzzles3434mixed puzzle differences Gaussian perturbation150150noise/resolution Smoothing transformation150150texture RGB shift147147whole-image color OCR/text221221OCR/text OmniDiff illumination151151illumination OmniDiff substitution/size199199substitution/size Total1,7561,543 Table 5: Source composition of the current benchmark inventory. Source names are reported for provenance; main- paper coverage is summarized by change category. A.2AUGMENTATION WITH PROGRAMMATIC VISUAL CHANGES Adding Gaussian Noise We perform the first type of low-level visual transformation on 150 FFHQ images (Karras et al., 2019) by adding Gaussian noise. For each image, we independently sample additive noise for every pixel and RGB channel from a zero-mean Gaussian distribution with standard deviation 15 in 8-bit pixel space, and clip the result to [0, 255]. This yields global low-level corruption while preserving the image content and layout. Changing RGB The second type of RGB-shift transformation starts from the same 150-image FFHQ image pool. As a first augmentation step, we apply a stronger global RGB perturbation to each image. As a second step, we compute the mean per-channel difference between this perturbed image and the original, remove its average channel offset to isolate chromatic rather than brightness change, scale the resulting color direction by 0.55, and cap the maximum absolute channel offset at 12 pixel values before applying it to the final image. This yields subtle global tone shifts while keeping the transformed image close to the original. The final benchmark retains 147 examples after excluding three construction-time flagged images. Changing Texture / Smoothing The texture transformation / smoothing transformation uses the same 150-image FFHQ image pool. We apply a skin- and edge-aware smoothing transform: the image is blurred with a Gaussian kernel whose standard deviation is sampled from [2.4, 3.6], while a soft skin mask, estimated from YCbCr color thresholds, increases smoothing on skin-like regions. Edges are protected using a luminance high-pass signal, and residual detail is mixed back with a random scale in [0.18, 0.36]. After smoothing, we add a small luminance-dependent highlight term to bright skin-like regions, reduce saturation by mixing the image toward grayscale, lower contrast around the mid-gray point, and apply a gamma correction sampled close to one. Together, these operations suppress fine texture and create a smoother, plastic-like appearance while preserving the image layout and object identities. Changing Illumination We augment the OmniDiff (Liu et al., 2025) illumination subset by constructing images with more subtle illumination changes. We use the first 50 sorted images from the Country211 test split (Radford et al., 2021) as the seed split. Each image is adjusted by a gamma-plus-linear transform, clip(255¡ x Îł + b), where x is the normalized RGB image. For brighter examples, Îł âź U(0.70, 0.85) and b âź 5,..., 15; for darker examples, Îł âź U(1.15, 1.30) and bâźâ15,...,â5. 12 A.3AUGMENTATION WITH IMAGE EDITING DIFFERENCES We additionally augment the OCR change and object position change categories in our evaluation data. Inspired by the recent advancement in high-performance image editing models, we utilize frontier commercial image editing model to help with this data augmentation process. Changing OCR Text The OCR subset contains 221 examples: 50 existing OCR/text-related pairs imported from OmniDiff, and 171 custom text edits initialized from MLT19 and TextOCR images. For the custom OCR edits, we select readable, approximately horizontal text regions with moderate size, avoiding boxes smaller than 50Ă 18 pixels or covering more than 40% of the image. We propose one of four localized textual edits: replacing one same-type character, replacing one digit, swapping adjacent characters, or replacing a word with another same-script word visible in the image. The selected crop is padded by 24 pixels, and then edited with gemini-3-pro-image which is instructed to also preserve font, color, perspective, background, lighting, and texture, and pasted back with an 8-pixel feathered boundary. We then validate that the target text appears and that non-text scene content remains unchanged. Below, we provide the prompt for the image editing model on the OCR edit task. OCR Edit Prompt You are looking at a crop from a scene image that contains text inscript script. Change the text âorigtextâ to ânewtextâ. Match the original font, color, size, alignment, perspective, and texture exactly. Keep the background, lighting, and every other element pixel-identical to the input. Only the text changes. Output the edited image. Changing Object Position We further augment the âposition changeâ category in the evaluation data. We take 150 seed images from the âswapâ task of the OmniDiff dataset to begin with; the reason for selecting the âswapâ images is because the nature of this task guarantees at least 2 clearly visible objects in images, whose positions can be cleanly changed. For each seed image, we first utilize gpt-5.4-mini to proposes 5 candidate edits, each specifying a visible, countable object, its original location, a plausible new location, an editing instruction, and a reference difference statement beginning with âIn the second image,â. The prompt explicitly disallows adding, removing, recoloring, resizing, rotating, duplicating, replacing, or deforming objects, so the intended change is restricted to spatial relocation. We use the first candidate by default and append preservation constraints requiring the same scene, camera angle, lighting, style, object identities, colors, and object counts. The selected instruction is then passed to gpt-image- 2 (OpenAI, 2026b) to produce an edited image. Below, we provide the prompts for the MLLM to propose potential positional edits, as well as the final image editing prompt template. Position Edit Proposal Prompt You are designing image-edit prompts for a rigorous visual difference benchmark. Look at the image and propose exactly 5 candidate edits where ONE visible object changes position. Each candidate must move an existing visible object to a different plausible location in the same scene. Do not add, remove, recolor, resize, restyle, rotate, deform, duplicate, or replace objects. Prefer clearly localized, countable objects whose position change would be easy to verify. Avoid vague background regions. The edit prompt must be clear and effective to instruct an image editing model to preserve everything except that single objectâs position. The differencedescription must be a short dataset annotation beginning with âIn the second image,â. Original OmniEdit swap instruction for context only: -originalinstruction1 -originalinstruction2 13 Position Edit Prompt selectedcandidateeditprompt Keep the same scene, camera angle, lighting, style, object identities, colors, and object counts. Only change the selected objectâs position. A.4ADDITIONAL DETAILS ON FALSE DIFFERENCES GENERATION We prompt the Gemini-2.5-pro model to inspect each image pair data and construct false differences that are seman- tically plausible but factually non-existent. This false difference candidates are then used as distractor options in the final multiple choice question construction, to challenge the evaluated models. Generation Prompt. For each image pair, the model receives a list of ground truth differences in them and are prompted to use these as references to create 2 to 6 statements that are factually incorrect but visually plausible, formatted similarly as the references. The prompt instructs the model to consider 4 strategies: swapping the target object, swapping the source object, changing an attribute of the target, and introducing a plausible alternative change to an entity visible in at least one image. The prompt also explicitly prohibits invented entities, indirect references to the true change, and complex relational wording. The prompt additionally requests up to five potentially missing true differences for audit; these suggestions are not treated as annotations, but any overlap with a displayed candidate alternative is conservatively flagged for visual adjudication. Fig. 5 below presents the prompt for the model to generate the false differences. False Differences Generation Prompt ### Role You are a precision image-analysis engine specialized in generating hard-negative distractors for change detection datasets. ### Context Dataset Category:category type Expected Change Logic:allowedfamilytext ### Input Data 1. Images: [Provided] 2. Ground Truth (GT) Differences: truth bullets ### Task 1: Generate False Differences (Hard Negatives) Createdistractorcount tomaxfalsedifferences statements that are FACTUALLY INCOR- RECT but visually plausible based on the scene. Each false difference should be phrased and formatted just like the GT. Use these strategies: ⢠GT Swap: Take a GT change and swap the ending object, e.g., B in âA is changed to Bâ, to another object that is also present in the image, preferably located close to A / B. ⢠Input Swap: Take a GT change and swap the beginning object, e.g., A in âA is changed to Bâ, to another object that is visible in the image, preferably located close to A / B. ⢠GT change: Take a GT change and change characteristics of the ending object, e.g., adjY B in âadjX A is changed to a adjY Bâ, to another characteristic, e.g., âadjX A is changed to a adjP Bâ, such as color, material, shape, etc. ⢠Non-Substitution changes: Randomly add false changes from non-substitution categories, such as color changes, left/right/closer/further movement, character or shape changes, illumination changes, noise changes, or RGB tone changes. Make sure the object you mention exists in at least one of the images. Constraints: ⢠Your false differences should be phrased and formatted just like the GT differences. ⢠The subject of your false statement MUST exist and be visible in at least one of the images. Do not invent new objects. 14 ⢠Preserve the main changed object from the GT in at least one but at most two generated false differences. ⢠Do NOT mention the ground truth. For example, if the ground truth is âthe white SUV has disappearedâ, do NOT describe something else as moved relative to âwhere the SUV wasâ. ⢠Do NOT use complex relational phrasing such as âbehind where X wasâ, ânext to where Y had beenâ, or âin the spot formerly occupied byâ. ⢠Do NOT generate more than two GT Swap differences that look similar to each other. ### Task 2: Identify Potential Missing True Differences List up to 5 real visual changes visible in the images that were NOT captured in the provided Ground Truth list. If the GT is exhaustive, return an empty list. ### Output Format (JSON) Return the response in this exact JSON structure: "false_differences": [ "statement": "The false sentence here.", "based_on_true_difference": "The exact GT sentence this is mimicking.", "strategy_used": "Object Swap / Spatial Error / State Error" ], "potential_missing_true_differences": ["Statement 1", "Statement 2"] Figure 5: Prompt used to generate hard-negative distractors. BEVALUATION PROMPT All main evaluations use the following textual template, with the four options substituted from the corresponding per-example inference artifact: Evaluation Prompt You are given image A and image B. Choose the option that correctly describes a real difference between image A and image B. Options: A. [option A] B. [option B] C. [option C] D. [option D] Answer ONLY one of the choice labels (e.g. âA.â) without other text: Images are supplied directly to the model in AâB order; no difference mask is used. All runs use temperature zero; Qwen3-VL-8B Thinking is run with reasoning disabled, and no separate reasoning trace is requested or recorded for the remaining models. CADDITIONAL RESULTS C.1NO-DIFFERENCE SELECTION ON LOW-LEVEL CHANGES As shown in Table 6, models differ substantially in how often they incorrectly judge an image pair as unchanged. Gemini 3.5 Flash, Gemini 3.1 Pro, GPT-5.4, and Gemini 2.5 Flash select the no-difference option on only 0.5%, 1.6%, 5.2%, and 12.1% of all questions, respectively, whereas this rate rises to 24.8â43.1% for the smaller models. Missed- change selections are particularly frequent for the low-level visual changesâincluding whole-image RGB, texture, and noise differencesâexceeding 70% in several cases. This shows that models still fail at correctly capturing visual differences that are not straightforwardly observable. 15 ModelWhole colorNoiseTextureIllum.Overall GPT-5.49.50.09.614.45.2 Gemini 2.5 Flash24.50.039.021.012.1 Gemini 3.1 Pro1.40.02.34.61.6 Gemini 3.5 Flash2.00.00.01.00.5 Grok 4.34.179.372.335.421.8 Doubao Seed 1.6 Vision0.00.04.06.21.3 Qwen3-VL-8B Thinking74.822.074.035.424.8 InternVL3.5-8B93.287.390.442.136.8 LLaVA-OneVision-Qwen2-7B93.988.772.372.843.1 Kimi K2.50.00.09.09.73.4 Kimi K315.00.010.715.95.9 Table 6: Rate (%) at which each model selects the no-difference distractor. The first four columns are category- specific; âOverallâ covers all 1,756 questions. High rates indicate a missed-change selection rather than confusion with another detailed description. C.2REPRESENTATIVE FAILURE CASES Figure 6 illustrates three distinct sources of difficulty. The smoothing operation changes local texture without altering scene semantics; the RGB shift distributes evidence across the entire image; and the surveillance example requires locating small missing entities in a low-resolution frame. The repeated no-difference choice is consistent with the aggregate behavior in Table 6. (a) Subtle smoothing (b) Global RGB shift (c) Two people disappear Figure 6: Representative no-difference selections. (a) Smoothing: Qwen, Gemini, LLaVA, and InternVL select no difference; GPT-5.4 selects the keyed answer. (b) Global RGB shift: Qwen, LLaVA, and InternVL select no difference. (c) Two people disappear: LLaVA and InternVL select no difference. C.3ADDITIONAL QUALITATIVE RESULTS Below, we provide more qualitative results on how different models perform on data from each category of our bench- mark. 16 GPT-5.43.1 Gemini 3.13.5 Gemini 3.5Kimi K2.5Grok 4.3Qwen3-VL-8B POSITION A B A: The white duck in the center is facing left in A but facing front in B. B: The pigâs tail points downward in A but curls upward in B. C: The horseâs tail changes color from brown in A to black in B. D: No difference between A and B. MODELS â3.1â3.5âââ MOTION A B A: In B, the womanâs eyes are looking down. B: The beer glass is now tilted and has a higher liquid level. C: The woman is now drinking from the glass and looking up. D: No difference between A and B. MODELS â3.1â3.5âââ REGIONAL COLOR A B A: A flower pot on the middle shelf on the left side is green in A but grey in B. B: No difference between A and B. C: The small pot on the far right is white in A and changes to blue in B. D: The overalls worn by the child in the center are light blue in A and have changed to a darker blue in B. MODELS â3.1â3.5âââ WHOLE-IMAGE COLOR A B A: No difference between A and B. B: B has a cooler blueâgreen color tone than A. C: The babyâs facial expression is different in B. D: B has a warmer redâmagenta color tone than A. MODELSâ3.1â3.5ââ APPEAR / DISAPPEAR A B A: No difference between A and B. B: The ribbon of the rabbit is missing a round pattern in B relative to A. C: The right eye of the rabbit is missing an eyelash in B relative to A. D: The bicycle is missing a pedal in B relative to A. MODELS â 3.1â3.5ââ NOISE / RESOLUTION A B A: No difference between A and B. B: B has a warmer color temperature than A. C: B is clearer than A. D: B is noisier than A. MODELS â3.1â3.5ââ Figure 7: Qualitative VDIFF-BENCH results across all ten change categories (part 1 of 2). Each panel shows the complete image pair at a common height, all four answer choices with the ground truth colored, and correctness for six representative MLLMs. Green checks and red crosses indicate correct and incorrect selections. 17 GPT-5.43.1 Gemini 3.13.5 Gemini 3.5Kimi K2.5Grok 4.3Qwen3-VL-8B TEXTURE A B A: No difference between A and B. B: The manâs brows changed to a lighter color in B. C: B is smoother than A. D: B is sharper than A. MODELSâ3.1â3.5ââ SUBSTITUTION / SIZE A B A: The large weight on the right tray of the scale in A is changed to a small weight in B. B: The large weight on the right tray of the scale in A is changed to a black package in B. C: The black package on the left tray of the scale in A is changed to a small weight in B. D: No difference between A and B. MODELSâ3.1â3.5ââ OCR / TEXT A B A: No difference between A and B. B: The Latin text âamourâ is changed to âamourâ with an extra letter. C: The Latin text âmonâ is changed to âmenâ. D: The Latin text âmonâ is changed to âmanâ. MODELSâ3.1â3.5â ILLUMINATION A B A: B is brighter than A. B: B is sharper than A. C: No difference between A and B. D: B is darker than A. MODELSâ3.1â3.5ââ Figure 7: Qualitative VDIFF-BENCH results across all ten change categories (part 2 of 2). Layout and symbols follow Figure 7. Examples are selected to illustrate mixed model outcomes rather than category-level prevalence. 18