Paper deep dive
SynDORBench: Evaluating LVLM Perceptual Robustness Under Physically Constrained Visibility Conditions
Jeremy Stephen Gabriel Yee, Zhengkui Wang, Zhiyuan Zhang, Avinash Anand, Timothy Liu, Benedict Chan, Aik Beng Ng, Simon See
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 88%
Last extracted: 9/29/2026, 11:59:41 AM
Summary
The paper introduces SynDORBench, a physically grounded benchmark for evaluating the perceptual robustness of Large Vision-Language Models (LVLMs) under DORI-calibrated visibility conditions. It utilizes a synthetic pipeline to generate over 54k question-answer pairs by systematically varying camera distance, illumination, viewpoint, and pixel density. The study evaluates 16 open-source LVLMs, a commercial baseline, and YOLO11x, revealing that LVLM performance is governed by pixel density and physical constraints rather than model scale alone, with some compact open-source models outperforming larger commercial ones and traditional detectors like YOLO11x in low-visibility scenarios.
Entities (11)
Relation Signals (9)
SynDORBench â evaluates â LVLM
confidence 95% · SynDORBench is the first physically grounded benchmark for evaluating LVLM perceptual robustness
SynDORBench â usesstandard â DORI
confidence 92% · SynDORBench... under DORI-calibrated conditions aligned with human visual capability standards
SynDORBench â evaluates â YOLO11x
confidence 90% · We evaluate 16 open-source LVLMs, a commercial LVLM baseline, and YOLO11x
LVLM â performancegovernedby â Pixel Density
confidence 90% · perceptual failure in LVLMs is strongly governed by pixel density and physical imaging constraints
SynDORBench â generatesdatausing â Blender
confidence 85% · We employ Blender as a controllable rendering environment to systematically manipulate scene geometry
Qwen2.5-VL-7B â istype â LVLM
confidence 85% · Qwen2.5-VL-7B-Instruct... Medium LVLMs
GPT-4o-mini â istype â LVLM
confidence 85% · Closed-source LVLMs gpt-4o-mini-2024-07-18
SynDORBench â collaboratedwith â NVIDIA
confidence 80% · Authors from NVIDIA AI Technology Center are listed as project leads
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large vision-language models (LVLMs) have demonstrated remarkable performance on multimodal reasoning benchmarks, yet their perceptual reliability under physically constrained imaging conditions remains poorly understood. Existing evaluations predominantly assume ideal visual inputs and therefore fail to characterize how camera distance, illumination, viewpoint, and pixel density fundamentally affect semantic recoverability. We introduce SynDORBench, the first physically grounded benchmark for evaluating LVLM perceptual robustness under DORI-calibrated conditions aligned with human visual capability standards. SynDORBench comprises over 54k question--answer pairs generated through a controllable synthetic pipeline that systematically varies viewing distance, lighting, camera geometry, and action pose according to physically interpretable pixel-density regimes. To support scalable low-visibility supervision, we further propose a discernibility annotation framework that propagates human perceptual labels using mask-conditioned statistical features and ensemble learning. We evaluate 16 open-source LVLMs, a commercial LVLM baseline, and YOLO11x across human-presence classification and action recognition tasks under progressively degraded visibility conditions. Our results reveal that perceptual failure in LVLMs is strongly governed by pixel density and physical imaging constraints rather than model scale alone. Surprisingly, several compact open-source LVLMs outperform larger commercial baselines and substantially exceed YOLO11x robustness under long-range and low-light conditions. SynDORBench establishes a new benchmark paradigm for physically grounded multimodal evaluation, enabling systematic analysis of LVLM reliability under real-world perceptual constraints and direct comparison against human visibility thresholds.
Tags
Links
- Source: https://arxiv.org/abs/2609.31823v1
- Canonical: https://arxiv.org/abs/2609.31823v1
Trouble viewing inline? Open PDF directly â
Full Text
64,825 characters extracted from source content.
Expand or collapse full text
SynDORBench: Evaluating LVLM Perceptual Robustness Under Physically Constrained Visibility Conditions Jeremy Stephen Gabriel Yee Affiliation: Singapore Institute of Technology Zhengkui Wang Affiliation: Singapore Institute of Technology Zhiyuan Zhang Affiliation: Singapore Management University Avinash Anand Affiliation: Singapore Institute of Technology Timothy Liu Affiliation: NVIDIA AI Technology Centerâ Project lead Corresponding Author Benedict Chan Affiliation: Singapore Institute of Technology Aik Beng Ng Affiliation: NVIDIA AI Technology Centerâ Project lead Corresponding Author Simon See Affiliation: NVIDIA AI Technology Centerâ Project lead Corresponding Author Abstract Large vision-language models (LVLMs) have demonstrated remarkable performance on multimodal reasoning benchmarks, yet their perceptual reliability under physically constrained imaging conditions remains poorly understood. Existing evaluations predominantly assume ideal visual inputs and therefore fail to characterize how camera distance, illumination, viewpoint, and pixel density fundamentally affect semantic recoverability. We introduce SynDORBench, the first physically grounded benchmark for evaluating LVLM perceptual robustness under DORI-calibrated conditions aligned with human visual capability standards. SynDORBench comprises over 54k questionâanswer pairs generated through a controllable synthetic pipeline that systematically varies viewing distance, lighting, camera geometry, and action pose according to physically interpretable pixel-density regimes. To support scalable low-visibility supervision, we further propose a discernibility annotation framework that propagates human perceptual labels using mask-conditioned statistical features and ensemble learning. We evaluate 16 open-source LVLMs, a commercial LVLM baseline, and YOLO11x across human-presence classification and action recognition tasks under progressively degraded visibility conditions. Our results reveal that perceptual failure in LVLMs is strongly governed by pixel density and physical imaging constraints rather than model scale alone. Surprisingly, several compact open-source LVLMs outperform larger commercial baselines and substantially exceed YOLO11x robustness under long-range and low-light conditions. SynDORBench establishes a new benchmark paradigm for physically grounded multimodal evaluation, enabling systematic analysis of LVLM reliability under real-world perceptual constraints and direct comparison against human visibility thresholds. SynDORBench is available at: Link and evaluation code is available at: Link 1 Introduction Large language models (LLMs) have rapidly evolved from language understanding systems into general multimodal foundation models capable of jointly reasoning over text, images, and video. Building upon advances in multimodal representation learning, Large VisionâLanguage Models (LVLMs) now demonstrate impressive performance across a broad spectrum of visual reasoning and perception tasks [35, 51, 64]. Existing benchmarks such as MMBench [31], MMMU [62], and SEED-Bench [27] show that LVLMs can achieve strong semantic understanding under high-quality visual conditions. However, these evaluations predominantly assume idealized inputs, where subjects are well-lit, visually salient, and captured at sufficiently high resolution. In contrast, real-world visual perception is fundamentally constrained by imaging physics. In practical deployment environments such as robotics, autonomous systems, and edge intelligence, visual evidence degrades continuously as a function of camera distance, illumination, sensor characteristics, viewpoint, and environmental interference [29]. These factors directly determine pixel density and therefore constrain the recoverability of semantic information from the scene. Consequently, an LVLM may demonstrate strong reasoning capability under ideal imaging conditions while failing catastrophically once the available visual evidence falls below perceptual thresholds. Existing evaluations primarily measure semantic competence rather than perceptual robustness, but not how far, under what visibility conditions, or under which physical constraints these systems remain operationally reliable. Figure 1: Minimum pixel density (px/m) required for human perceptual tasks under the BS EN 62676-4:2015 DORI standard [6]. The DORI framework defines physically grounded visibility thresholds for Detection, Observation, Recognition, and Identification. To address this gap, we draw inspiration from the BS EN 62676-4:2015 DORI standard [6], a widely adopted imaging framework that defines four perceptual visibility levels: Detection, Observation, Recognition, and Identification (DORI). Each level corresponds to the minimum pixel density required for a human observer to reliably perform a visual task (Fig. 1). While DORI has long served as a physically grounded model of human visual capability, it has never been used to characterize the perceptual robustness of LVLMs. Consequently, the relationship between multimodal reasoning performance and physically measurable imaging constraints remains largely unexplored. In this work, we introduce SynDORBench, the first physically grounded benchmark designed to evaluate LVLM perceptual robustness under DORI-calibrated imaging conditions. Rather than evaluating models solely on semantic correctness, SynDORBench systematically studies how multimodal perception degrades as available visual evidence deteriorates. To achieve this, we construct a controllable synthetic evaluation framework that explicitly models camera distance, illumination, viewpoint, and pixel-density regimes according to physically interpretable imaging constraints. The benchmark contains over 54k questionâanswer pairs spanning human-presence classification and action recognition tasks across progressively degraded visibility conditions. Unlike existing multimodal benchmarks that rely on uncontrolled natural variation, SynDORBench enables factorized robustness analysis by isolating individual physical degradation factors while maintaining precise control over camera geometry and perceptual visibility. Furthermore, to support scalable evaluation under low-light and night-vision conditions, we introduce a discernibility annotation framework that propagates human perceptual labels through mask-conditioned statistical features and ensemble learning. Extensive experiments across 16 open-source LVLMs, a commercial LVLM baseline, and YOLO11x reveal several important findings. First, perceptual failure in LVLMs is strongly governed by pixel density and imaging constraints rather than model scale alone. Second, modern LVLMs substantially outperform traditional detection pipelines under long-range and low-visibility conditions, suggesting that multimodal reasoning architectures exhibit greater robustness to perceptual degradation than conventional detector-based systems. Third, several compact open-source LVLMs achieve robustness comparable to or exceeding larger commercial baselines, indicating that physically grounded perceptual capability does not scale monotonically with parameter count. Our findings establish physically grounded multimodal evaluation as a critical yet largely missing dimension of LVLM assessment. By bridging camera physics, human perceptual standards, and multimodal reasoning evaluation, SynDORBench enables systematic study of LVLM reliability under real-world perceptual constraints. The main contributions of this work are summarized as follows: 1. We introduce SynDORBench, the first DORI-calibrated benchmark for evaluating LVLM perceptual robustness under physically grounded imaging conditions. Unlike existing multimodal benchmarks that rely on dataset-relative visual difficulty, SynDORBench explicitly models camera distance, illumination, viewpoint, and pixel density according to measurable imaging physics and human perceptual thresholds. 2. We propose a scalable discernibility annotation framework for low-visibility supervision under night and night-vision conditions. Using 2,000 human annotations, mask-conditioned perceptual features, and an RF+XGBoost ensemble, the framework propagates discernibility labels across a large synthetic corpus while preserving alignment with human perceptual judgments. 3. We conduct a large-scale robustness evaluation across 18 state-of-the-art models, including open-source LVLMs, commercial LVLMs, and detector-based baselines. Our analysis reveals previously underexplored relationships between pixel density, physical imaging constraints, model scale, and multimodal perceptual reliability. 4. We establish physically grounded multimodal evaluation as a new benchmark paradigm for studying LVLM robustness under real-world perceptual degradation, enabling direct comparison between algorithmic perception and human visibility capability thresholds. 2 Related Work Synthetic Datasets have become increasingly important for studying visual perception under controlled and reproducible conditions. Unlike real-world datasets, synthetic generation pipelines enable precise manipulation of scene geometry, illumination, camera pose, and environmental conditions, allowing researchers to isolate individual factors that influence model behavior. High-fidelity rendering frameworks such as Synscapes [58], SPREAD [14], and SCaRL [46] demonstrate that synthetic imagery can reproduce physically plausible degradation effects arising from adverse weather, long-range observation, and changing illumination. Human-centric synthetic datasets such as SynBody [59] and Boundless [53] further show that synthetic data can effectively support downstream perception and human understanding tasks. Despite these advances, existing synthetic benchmarks primarily focus on semantic diversity, photorealism, or downstream task performance rather than physically grounded perceptual evaluation. In most prior work, visual degradation emerges implicitly from rendering configurations and is not explicitly tied to measurable imaging properties such as focal length, sensor geometry, aperture, viewing distance, or pixel density. Consequently, benchmark difficulty is typically dataset-relative rather than physically interpretable, limiting the ability to quantify how imaging constraints affect perceptual reliability. Multimodal Benchmarking and Robustness Evaluation such as MMBench [31], MMMU [62], SEED-Bench [27], and MME [16] have become standard evaluation suites for measuring multimodal reasoning, instruction following, and semantic understanding across diverse visual domains. While these benchmarks provide important insights into multimodal competence, they predominantly evaluate models under curated, high-visibility conditions where semantic information is largely preserved. As a result, current evaluations emphasize what LVLMs can recognize under idealized settings, while providing limited understanding of how perceptual degradation affects model reliability. Critical physical factors such as viewing distance, illumination variation, viewpoint distortion, and reduced pixel density remain largely underexplored despite their importance in real-world deployment scenarios. Several recent benchmarks partially address this limitation by introducing temporally grounded evaluation settings. Video-oriented datasets such as LVBench [57] and CinePile [47] incorporate long-video reasoning, temporal dependencies, and action understanding. However, these datasets remain observational rather than physically controlled: degradation factors such as illumination, exposure, occlusion, and distance vary naturally but are not systematically parameterized according to measurable imaging constraints. Consequently, they can reveal aggregate model performance but cannot isolate the causal impact of specific physical degradation factors on multimodal perception. SynDORBench directly targets this missing dimension by introducing a physically calibrated benchmark for multimodal robustness evaluation. Rather than measuring semantic reasoning alone, SynDORBench systematically evaluates how LVLM perceptual capability degrades across DORI-calibrated visibility regimes under controlled variations in distance, lighting, and viewpoint. By connecting multimodal evaluation with camera physics and human perceptual standards, SynDORBench establishes a new benchmark paradigm for studying physically grounded perceptual robustness in LVLMs, as summarized in table 1. Table 1: Comparison of multimodal LVLM benchmarks. Benchmark Modality Answer Type # Models # QA Pairs DORI-Calibrated Dist. SEED-BENCH [27] Image, Video Multiple-choice 18 19,242 â LV-BENCH [57] Video Multiple-choice 27 1,549 â MMMU [62] Image Multiple-choice, Open-ended 24 11,550 â CinePile [47] Video Multiple-choice 24 303,828 â LOKI [60] Video, Image, 3D, Audio Multiple-choice, Open-ended 28 18,000 â SynDORBench-54k(Ours) Image Multiple-choice, Open-ended 18 54,864 â 3 SynDORBench Figure 2: Conceptual overview of SynDORBench: The benchmark systematically varies physical visibility factors (lighting, scene type, camera azimuth/elevation, action pose) and DORI-calibrated viewing distance, evaluating models on two tasks: 1) binary human-presence classification and 2) open-ended action recognition. SynDORBench is a physically grounded evaluation framework designed to systematically study how LVLM perceptual capability degrades under progressively constrained imaging conditions. Unlike conventional multimodal benchmarks that primarily evaluate semantic understanding under curated visual settings, SynDORBench explicitly models the physical factors that govern visual evidence formation, including camera distance, illumination, viewpoint, and pixel density. By anchoring synthetic image generation to DORI-calibrated visibility range thresholds, the benchmark enables controlled analysis of multimodal perceptual robustness under measurable imaging constraints rather than uncontrolled dataset variation. A central design principle of SynDORBench is factorized physical degradation. Instead of relying on naturally occurring visual variability, we explicitly parameterize and isolate individual degradation factors to study their causal effect on semantic recoverability. This allows SynDORBench to characterize not only whether an LVLM succeeds or fails, but also under which physically interpretable visibility conditions perceptual breakdown occurs. The resulting framework bridges multimodal evaluation with camera physics and human perceptual standards, enabling direct comparison between algorithmic robustness and human visibility capability. 3.1 Synthetic Image Creation This section describes the synthetic generation pipeline used to construct SynDORBench. We employ Blender as a controllable rendering environment to systematically manipulate scene geometry, camera configuration, illumination, viewpoint, and subject appearance while preserving physically interpretable imaging conditions. Controlled Single-Human Design Although real-world image capturing environments often contain multiple individuals [9], SynDORBench intentionally adopts a single-human design to isolate the relationship between physical visibility constraints and perceptual capability. Human discernibility is fundamentally governed by the pixel density allocated to the entity of interest [6]. Introducing multiple subjects would entangle distance-dependent degradation with inter-person occlusion, crowd density, and spatial ambiguity, making it difficult to attribute performance degradation to specific imaging factors. Our controlled setup therefore enables causal analysis of how distance, lighting, and viewpoint independently affect LVLM perception under minimum-information regimes. Physically Calibrated Camera Configuration The simulated camera uses a 1080p, 1/2.8" sensor, 2.8m focal length, and f/1.4 aperture, reflecting common fixed-camera specifications. Full details on specification justification is provided in section A.1.1. DORI-Calibrated Distance Modeling DORI-calibrated distances are derived from camera geometry and pixel-densities using the pinhole camera model [17] and DORI perceptual framework [6] respectively. We sample midpoint pixel-density regimes between adjacent DORI zones: 43.8, 93.8, and 187.5 px/m, which converts to calibrated distances of 23m, 10.8m, and 5.4m respectively. Full derivation is provided in section A.1.4. Camera Viewpoint Sampling To model realistic fixed-camera imaging viewing geometry, we adopt a hemispherical camera placement strategy parameterized by azimuth and elevation angles. The camera configuration consists of 8 azimuthal viewpoints and 4 elevation levels distributed around the subject. This design preserves consistent distance calibration while capturing viewpoint-dependent appearance variation. Illumination Modeling SynDORBench incorporates four lighting conditions corresponding to morning, noon, evening, and midnight scenarios. The daytime settings vary solar azimuth and elevation angles to reproduce realistic directional illumination and shadow behavior, while the midnight configuration models extreme low-light visibility degradation. This allows controlled evaluation of LVLM robustness across progressively deteriorating illumination regimes. Night-Vision Simulation Because camera systems frequently employ night-vision enhancement pipelines to improve visibility in low-light settings, SynDORBench additionally includes a night-vision subset derived from low-light renderings using a vendor-neutral image signal processing pipeline. More details found in section A.1.5. Scene Diversity To reduce scene-specific bias while preserving physical controllability, SynDORBench includes three large-scale outdoor environments: a desert landscape, an open field, and a ship wharf. Assets are sourced from BlenderKit (RF/C0) and selected to provide both environmental diversity and sufficient spatial extent for long-range DORI-calibrated rendering. The resulting environments vary substantially in texture density, background clutter, illumination interaction, and spatial composition. Action Pose Generation Human action poses are derived from the BABEl dataset [45], built upon AMASS motion-capture corpus [33]. Poses are filtered for abnormal activity perceptual relevance via a multi-stage embedding-based ranking and a final manual verification pipeline, then retargeted into Blender using SMPL-X [41]. Full details are provided in section A.1.2. Texture and Appearance Diversity To improve photometric diversity, we utilize the SMPLitex texture pack [8], which provides realistic variation in skin tone, clothing appearance, reflectance, and luminance characteristics. This diversity improves the robustness of subsequent perceptual evaluation by reducing overfitting to narrow appearance distributions and enabling richer lightâsurface interaction patterns. Factor-Wise Stratified Sampling SynDORBench adopts factor-wise stratified sampling, where balancing is performed independently over each generative factor. For every factor category (e.g., DORI distance, lighting condition, texture), samples are drawn to maintain approximately uniform marginal distributions. This strategy provides broad coverage of the physical design space while avoiding severe over-representation of dominant factor combinations. 3.2 Discernible Labels Although SynDORBench is synthetically generated, perceptual visibility is not binary. A human may be geometrically present within the image while remaining perceptually indiscernible due to insufficient pixel density, poor illumination, low contrast, or severe visibility degradation. To align presence classification labels with human perceptual capability, SynDORBench introduces discernibility labels that explicitly model whether a human annotator can reliably perceive the subject under rendered conditions. A pool of 2,000 was sampled using Cochranâs formulation [11] under a 95% confidence interval and 5% margin of error, of which 400 samples were held out as a test set. Enabled by the ground truth segmentation masks, we extracted mask-conditioned perceptual features capturing features that compared visual evidence inside and outside the mask, feature include: 1) unblocked area ratio, measuring the proportion of human pixels relative to the image. 2)Luminance Divergence, computed using the KolmogorovâSmirnov (K-S) statistic [20] between foreground and background luminance distributions in the CIE Lab color space [23]. 3)Chromatic Distribution Distance, measured using Sliced Wasserstein distance [5] to quantify differences in hue and saturation distributions between the human region and the surrounding context. 4)Edge Discernibility, computed via the intersection-over-union between Sobel edge responses [50] inside the masked region and a dilated contour around the human boundary. To improve robustness under adverse illumination, each feature is computed for both the original rendering and its corresponding night-vision-enhanced variant. This allows the annotation model to jointly reason over raw luminance structure and enhanced visibility characteristics produced by the ISP pipeline. We formulate discernibility prediction as a binary perceptual classification problem. Hyperparameter optimization is performed using grid search with 5-fold cross-validation to improve generalization across visibility conditions. We evaluate both a decision-tree classifier and an ensemble model combining Random Forest and XGBoost. As shown in table 5, the ensemble model achieves the best overall performance and is subsequently used to propagate discernibility labels across the night and night-vision subsets of SynDORBench. Importantly, the discernibility framework allows SynDORBench to distinguish between physical presence and perceptual visibility, enabling more faithful evaluation of LVLM robustness under minimum-information imaging regimes. Table 2: Overview of evaluated models by scale and modality (Custom encoders denoted by * with limited publicly disclosed details). Model Type Models # Params Vision Encoder (VE) VE Type LLM Backbone Detection-only (non-LVLM) YOLO11x [25] 56.9M â â â Small LVLMs (SVLMs) [32] SmolVLM2-256M-Video-Instruct [35, 21] 256M SigLIP-B/16-512 SigLIP SmolLM2â135M SmolVLM2-500M-Video-Instruct [35, 22] 500M SigLIP-B/16-512 SigLIP SmolLM2â360M InternVL3-1B-hf [64] 1B InternViT-300M-448px-V2.5 InternViT Qwen2.5â0.5B Perception-LM-1B [10] 1B PE L/14-448 PE Llama-3.2-1B SmolVLM2-2.2B-Instruct [35, 52] 2.2B SigLIP-So400M-patch14-384 SigLIP SmolLM2-1.7B Perception-LM-3B [10] 3B PE L/14-448 PE Llama-3.2-3B Qwen2.5-VL-3B-Instruct [3] 3B Custom ViT* Custom ViT* Qwen2.5-3B gemma-3-4b-it [51] 4B SigLIP-So400M-896* SigLIP Gemma-3-4B gemma-3n-E2B-it [51, 49] 5B (2B) MobileNet-V5-300M MobileNet Gemma-3n-E2B Medium LVLMs (MVLMs) Phi-4-multimodal-instruct [2] 6B SigLIP-So400M* SigLIP Phi-4-mini LLaVAction-7B [61] 7B SigLIP-so400m-patch14-384 SigLIP Qwen2-7B LLaVA-NeXT-Video-7B-hf [63] 7B SigLIP-so400m-patch14-384 SigLIP Qwen2-7B Molmo-7B-D-0924 [13] 7B CLIP-L/14 CLIP Qwen2-7B Qwen2.5-VL-7B-Instruct [3] 7B Custom ViT* Custom ViT* Qwen2.5-7B gemma-3n-E4B-it [51, 49] 8B (4B) MobileNet-V5-300M MobileNet Gemma-3n-E4B InternVL3-8B-hf [64] 8B InternViT-6B InternViT Qwen2.5-7B Closed-source LVLMs gpt-4o-mini-2024-07-18 [40] ⌠8B [1] â â â 3.3 Task Definition and Question Types Within this benchmark, a task is defined as a bounded perceptual objective that an LVLM must solve from a single image, under human capability-aligned imaging conditions. Each task is instantiated as a question template applied to every evaluated frame, every evaluated sample comprises one question and one expected answer. This yields a unified QA-based interface across heterogeneous LVLMs, allowing fair comparison irrespective of architectural differences, chat-template variations, or model-specific decoding protocols. The current benchmark focuses on two foundational perception tasks: 1) Human-Presence Classification (HPC), and 2) Action Recognition. These tasks were deliberately selected because they represent core long-range visual understanding capabilities commonly required in robotics, assistive technologies, and smart infrastructure. Further details of the action-recognition task and results will be found in the supplementary. For HPC, each LVLM receives the prompt: "Respond only with 1 or 0. Respond with 1 if a person is visible in the image, and 0 if not. Is there a person visible in the image?". This isolates grounded image-level binary HPC capability, and allows for evaluating sensitivity to distance, illumination, and occlusion effects without conflating perceptual capability with spatial localization. SynDORBench additionally includes mask and bounding-box metadata to support future human detection benchmarking. 3.4 Evaluation Metrics For the HPC task, we follow prior LVLM evaluation protocols that formulate existence queries as a binary classification problem [28, 24]. We report both accuracy and macro-F1. Accuracy measures overall prediction correctness, while macro-F1 accounts for class imbalance and asymmetric error patterns, making it particularly informative for evaluating hallucination behavior under degraded visibility conditions [28]. Unlike conventional benchmarks that report aggregate performance alone, SynDORBench evaluates metrics across DORI-calibrated visibility regimes to characterize perceptual degradation under varying distance, illumination, and viewpoint conditions. To contextualize deployment reliability, we additionally define an operational effectiveness threshold of 80% accuracy, motivated by prior real-world human detection literature [37, 30, 55, 39]. Performance below this threshold is treated as operationally unreliable under the corresponding imaging conditions. 4 Evaluation Results In this section, we evaluate LVLM perceptual robustness on SynDORBench under progressively degraded DORI-calibrated imaging conditions. Additional stratified analyses are provided in section A.5.2. Evaluated Models We evaluate 16 open-source LVLMs, one commercial LVLM baseline, and YOLO11x, as summarized in table 2. As a representative detection-only baseline, we adopt YOLO11x [25], which remains widely used in detection systems [26]. YOLO11x is the strongest model in the YOLOv11 family and therefore provides a competitive detector-based reference point for comparison against multimodal architectures. We additionally include GPT-4o-mini [40] as a closed-source commercial LVLM baseline. Because SynDORBench focuses on perceptual robustness rather than large-scale reasoning capacity, we restrict evaluation to models with †8B parameters to maintain practical relevance for low-latency visual understanding systems. Larger models incur substantially higher inference costs, making direct comparison with real-time detection pipelines less meaningful [54]. The evaluated open-source models include SmolVLM2 [35], InternVL3 [64], Perception-LM [10], Qwen2.5-VL [3], Gemma-3 [51], Phi-4-multimodal [2], LLaVAction [61], LLaVA-NeXT-Video [63], and Molmo [13]. Following the categorization proposed by Lu et al. [32], we group models with â€5†5B parameters as Small VisionâLanguage Models (SVLMs), while models in the 6â8B range are grouped as Medium VisionâLanguage Models (MVLMs). HPC Robustness Across Distances To isolate the effect of perceptual distance degradation, we evaluate all models across DORI-calibrated distance regimes while averaging over other generative factors such as illumination and viewpoint. We additionally report group means for SVLMs and MVLMs to analyze scale-dependent robustness trends. Results are summarized in table 3. Across all evaluated models, HPC performance exhibits clear distance sensitivity: both accuracy and macro-F1 consistently decrease as viewing distance increases from 5.4 m to 23.0 m. This confirms that reduced pixel density substantially constrains semantic recoverability under physically degraded imaging conditions. MVLMs achieve the strongest overall group performance across all distances, indicating that increased model capacity generally improves perceptual robustness. However, the relationship between parameter count and robustness is not strictly monotonic. Several compact SVLMs, particularly gemma-3-4b-it and gemma-3n-E2B-it, match or exceed the performance of larger MVLMs, suggesting that architectural design and multimodal alignment may play a more important role than scale alone under low-information regimes. Compared to LVLMs, YOLO11x exhibits substantially weaker robustness, particularly at longer viewing distances. Importantly, this degradation is not driven solely by distance. YOLO11x already falls below the 80% operational reliability threshold at 5.4 m, where visibility conditions remain relatively favorable within SynDORBench. This suggests that conventional detector-based pipelines are highly sensitive not only to pixel-density reduction, but also to heterogeneous illumination, contrast variation, and perceptual ambiguity. Overall, the results indicate that modern LVLMs substantially outperform traditional detector-based systems on DORI-calibrated HPC and remain competitive with commercial closed-source LVLMs under progressively degraded visibility conditions. Table 3: Performance degradation across DORI-calibrated distances for presence classification, with accuracy (Acc) and macro-F1 (F1) reported. Best scores are in bold. Model 5.4 m 10.8m 23.0 m Acc F1 Acc F1 Acc F1 YOLO11x 73.92 50.17 65.38 50.36 50.29 42.98 SmolVLM2-256M-Video- Instruct 10.02 9.63 13.47 12.68 22.70 18.69 SmolVLM2-500M-Video- Instruct 23.20 20.23 26.51 24.04 27.19 22.08 InternVL3-1B-hf 87.77 59.63 85.69 64.58 83.97 68.13 Perception-LM-1B 69.77 48.21 62.64 49.11 45.94 39.89 SmolVLM2-2.2B-Instruct 80.84 55.07 75.89 57.64 66.51 56.22 Perception-LM-3B 75.76 50.83 71.76 53.77 58.80 48.37 Qwen2.5-VL-3B-Instruct 66.70 45.49 64.00 48.86 54.75 45.16 gemma-3-4b-it 94.66 50.44 92.91 54.04 85.66 67.68 gemma-3n-E2B-it 87.10 75.78 86.29 65.27 88.87 72.12 SVLM (mean) 66.20 46.14 64.35 47.78 59.38 48.71 Phi-4-multimodal-instruct 84.97 56.59 83.42 62.74 79.53 65.76 LLaVAction-7B 81.80 55.96 74.70 57.33 57.88 50.01 LLaVA-NeXT-Video-7B- hf 81.99 54.84 73.15 54.76 54.90 47.17 Molmo-7B-D-0924 84.63 57.68 80.61 60.82 70.76 59.06 Qwen2.5-VL-7B-Instruct 86.13 58.41 84.75 63.86 80.24 64.99 gemma-3n-E4B-it 87.77 59.08 86.73 65.05 87.76 70.40 InternVL3-8B-hf 85.45 58.08 83.13 62.67 71.66 59.84 MVLM (mean) 84.68 57.24 80.93 61.03 71.82 59.61 gpt-4o-mini-2024-07-18 82.72 55.79 77.87 58.72 69.60 59.03 Discussion Interestingly, while MVLMs achieve the strongest overall group performance, robustness does not scale monotonically with parameter count. Several compact SVLMs remain highly competitive and, in some cases, outperform larger models under degraded visibility conditions. This suggests that multimodal robustness depends not only on model scale, but also on factors such as vision encoder design, visual alignment quality, and robustness of multimodal feature integration. These findings indicate that physically grounded perception may require architectural properties distinct from those optimized by conventional semantic reasoning benchmarks. Another notable observation is the substantial robustness gap between LVLMs and traditional detector-based systems. YOLO11x exhibits rapid degradation under long-range and heterogeneous illumination conditions, even falling below operational reliability thresholds at relatively favorable visibility regimes. In contrast, LVLMs retain stronger discriminability under the same conditions, potentially because multimodal architectures can leverage broader contextual and semantic priors rather than relying solely on local appearance cues. This suggests that modern LVLMs may offer a more resilient perception framework for low-information environments where conventional detectors become brittle. 5 Conclusion In this paper, we introduced SynDORBench, the first physically grounded benchmark for evaluating LVLM perceptual robustness under DORI-calibrated imaging conditions. Unlike existing multimodal benchmarks that primarily assess semantic reasoning under ideal visual settings, SynDORBench systematically studies how multimodal perception degrades as visual evidence deteriorates under controlled physical constraints. Through large-scale evaluation across open-source LVLMs, commercial LVLMs, and detector-based baselines, we show that pixel density and imaging conditions play a central role in determining multimodal perceptual reliability. Our experiments further demonstrate that modern LVLMs substantially outperform traditional detector-based pipelines under long-range and low-visibility conditions, while several compact open-source models achieve robustness comparable to larger commercial systems. Overall, SynDORBench establishes physically grounded multimodal evaluation as an important and previously underexplored dimension of LVLM assessment. We hope the benchmark will support future research on robust multimodal perception, physically constrained reasoning, and reliability-aware evaluation for real-world vision-language systems. Data Availability The dataset and evaluation framework will be released under the C BY-NC-SA 4.0 license for non-commercial academic research. The release includes rendered images, metadata, and evaluation code to support reproducibility and future benchmarking research, while respecting the licensing terms of upstream resources including SMPL-X, AMASS, BABEL, and SMPLitex. Limitations SynDORBench is designed as a controlled and physically grounded benchmark for studying how imaging conditions affect LVLM perceptual robustness. To enable reproducible and factorized analysis, the current benchmark focuses on single-human, frame-level scenarios under systematically calibrated visibility conditions. Broader Impact SynDORBench is intended to advance research on reliable multimodal perception under adverse imaging conditions, with potential applications in robotics, assistive systems, intelligent infrastructure, autonomous platforms, and safety-critical visual understanding. Safeguards SynDORBench consists entirely of synthetically rendered imagery and does not contain real individuals or real-world imagery, thereby avoiding privacy and identity-related concerns associated with real data collection. The benchmark is released strictly for non-commercial academic research purposes to encourage responsible use within the research community. Acknowledgments This work is supported by the SIT-NVIDIA AI Centre. References [1] A. B. Abacha, W. Yim, Y. Fu, Z. Sun, M. Yetisgen, F. Xia, and T. Lin (2024) Medec: a benchmark for medical error detection and correction in clinical notes. arXiv preprint arXiv:2412.19260. Cited by: Table 2. [2] A. Abouelenin, A. Ashfaq, A. Atkinson, H. Awadalla, N. Bach, J. Bao, A. Benhaim, M. Cai, V. Chaudhary, C. Chen, et al. (2025) Phi-4-mini technical report: compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743. Cited by: Table 2, §4. [3] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. (2025) Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. Cited by: Table 2, Table 2, §4. [4] O. Bielova, R. Hansch, A. Ley, and O. Hellwich (2019) A digital image processing pipeline for modelling of realistic noise in synthetic images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, p. 0â0. Cited by: §A.1.5. [5] N. Bonneel, J. Rabin, G. PeyrĂ©, and H. Pfister (2015) Sliced and radon wasserstein barycenters of measures. Journal of Mathematical Imaging and Vision 51 (1), p. 22â45. Cited by: §3.2. [6] British Standards Institution (2022) BS EN 62676-4:2015 video surveillance systems for use in security applications. application guidelines. BSI. Note: Standard Cited by: Figure 1, §1, §3.1, §3.1. [7] G. Buchsbaum (1980) A spatial processor model for object colour perception. Journal of the Franklin institute 310 (1), p. 1â26. Cited by: §A.1.5. [8] D. Casas and M. Comino-Trinidad (2023) Smplitex: a generative model and dataset for 3d human texture estimation from single image. arXiv preprint arXiv:2309.01855. Cited by: §3.1. [9] T. Chavdarova, P. BaquĂ©, S. Bouquet, A. Maksai, C. Jose, T. Bagautdinov, L. Lettry, P. Fua, L. Van Gool, and F. Fleuret (2018) Wildtrack: a multi-camera hd dataset for dense unscripted pedestrian detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 5030â5039. Cited by: §3.1. [10] J. H. Cho, A. Madotto, E. Mavroudi, T. Afouras, T. Nagarajan, M. Maaz, Y. Song, T. Ma, S. Hu, S. Jain, et al. (2025) Perceptionlm: open-access data and models for detailed visual understanding. arXiv preprint arXiv:2504.13180. Cited by: Table 2, Table 2, §4. [11] W. G. Cochran (1977) Sampling techniques. john wiley & sons. Cited by: §3.2. [12] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian (2007) Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on image processing 16 (8), p. 2080â2095. Cited by: §A.1.5. [13] M. Deitke, C. Clark, S. Lee, R. Tripathi, Y. Yang, J. S. Park, M. Salehi, N. Muennighoff, K. Lo, L. Soldaini, et al. (2025) Molmo and pixmo: open weights and open data for state-of-the-art vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 91â104. Cited by: Table 2, §4. [14] Z. Feng, Y. She, and S. Keshav (2025) SPREAD: a large-scale, high-fidelity synthetic dataset for multiple forest vision tasks. Ecological Informatics 87, p. 103085. Cited by: §2. [15] K. R. Fowler (2004) Automatic gain control for image-intensified camera. IEEE Transactions on Instrumentation and Measurement 53 (4), p. 1057â1064. Cited by: §A.1.5. [16] C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun, et al. (2025) MME: a comprehensive evaluation benchmark for multimodal large language models. In Neural Information Processing Systems Datasets and Benchmarks Track, Cited by: §2. [17] R. I. Hartley and A. Zisserman (2004) Multiple view geometry in computer vision. Second edition, Cambridge University Press, ISBN: 0521540518. Cited by: §A.1.4, §3.1. [18] F. Heide, M. Steinberger, Y. Tsai, M. Rouf, D. PajÄ k, D. Reddy, O. Gallo, J. Liu, W. Heidrich, K. Egiazarian, et al. (2014) Flexisp: a flexible camera image processing framework. ACM Transactions on Graphics (ToG) 33 (6), p. 1â13. Cited by: §A.1.5. [19] J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y. Choi (2021) Clipscore: a reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718. Cited by: §A.3. [20] J. Hodges Jr (1958) The significance probability of the smirnov two-sample test. Arkiv för matematik 3 (5), p. 469â486. Cited by: §3.2. [21] Hugging Face Team (2025) SmolVLM2-256M-Video-Instruct. Note: https://huggingface.co/HuggingFaceTB/SmolVLM2-256M-Video-InstructHugging Face Model Card Cited by: Table 2. [22] Hugging Face Team (2025) SmolVLM2-500M-Video-Instruct. Note: https://huggingface.co/HuggingFaceTB/SmolVLM2-500M-Video-InstructHugging Face Model Card Cited by: Table 2. [23] ISO/CIE (2019) ISO/CIE 11664-4:2019 colorimetry â part 4: CIE 1976 L* a* b* colour space. International Organization for Standardization and CIE. Cited by: §3.2. [24] P. Kaul, Z. Li, H. Yang, Y. Dukler, A. Swaminathan, C. Taylor, and S. Soatto (2024) Throne: an object-based hallucination benchmark for the free-form generations of large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 27228â27238. Cited by: §3.4. [25] R. Khanam and M. Hussain (2024) Yolov11: an overview of the key architectural enhancements. arXiv preprint arXiv:2410.17725. Cited by: Table 2, §4. [26] S. Kukreti, R. Praveen, S. Bansal, H. Raju, N. Singh, and R. Begum (2025) Object detection in real-time surveillance using deep learning-based yolo framework. In 2025 International Conference on Computational, Communication and Information Technology (ICCCIT), p. 601â606. Cited by: §4. [27] B. Li, R. Wang, G. Wang, Y. Ge, Y. Ge, and Y. Shan (2023) Seed-bench: benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125. Cited by: §1, §2, Table 1. [28] Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J. Wen (2023) Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, p. 292â305. Cited by: §3.4. [29] B. Liu, P. Qiao, M. Ma, X. Zhang, Y. Tang, P. Xu, K. Liu, and T. Yuan (2025) SurveillanceVQA-589k: a benchmark for comprehensive surveillance video-language understanding with large models. arXiv preprint arXiv:2505.12589. Cited by: §1. [30] G. Liu and Y. Sun (2012) An in-vehicle system for pedestrian detection. In 2012 11th International Symposium on Distributed Computing and Applications to Business, Engineering & Science, p. 328â331. Cited by: §A.5.2, §3.4. [31] Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, et al. (2024) Mmbench: is your multi-modal model an all-around player?. In European conference on computer vision, p. 216â233. Cited by: §1, §2. [32] Z. Lu, X. Li, D. Cai, R. Yi, F. Liu, X. Zhang, N. D. Lane, and M. Xu (2024) Small language models: survey, measurements, and insights. arXiv preprint arXiv:2409.15790. Cited by: Table 2, §4. [33] N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black (2019) AMASS: archive of motion capture as surface shapes. In International Conference on Computer Vision, p. 5442â5451. Cited by: §3.1. [34] Y. MĂ€kinen, L. Azzari, and A. Foi (2020) Collaborative filtering of correlated noise: exact transform-domain variance for improved shrinkage and patch matching. IEEE Transactions on Image Processing 29, p. 8339â8354. External Links: Document Cited by: §A.1.5. [35] A. Marafioti, O. Zohar, M. FarrĂ©, M. Noyan, E. Bakouch, P. Cuenca, C. Zakka, L. B. Allal, A. Lozhkov, N. Tazi, et al. (2025) Smolvlm: redefining small and efficient multimodal models. arXiv preprint arXiv:2504.05299. Cited by: §1, Table 2, Table 2, Table 2, §4. [36] A. Mishra (2021) Contrast limited adaptive histogram equalization (clahe) approach for enhancement of the microstructures of friction stir welded joints. arXiv preprint arXiv:2109.00886. Cited by: §A.1.5. [37] H. Mokayed, T. Z. Quan, L. Alkhaled, and V. Sivakumar (2023) Real-time human detection and counting system using deep learning computer vision techniques. Artificial Intelligence and Applications 1 (4), p. 205â213. Cited by: §A.5.2, §3.4. [38] M. Neau, Z. Falomir, C. Buche, and A. Sugimoto (2025) Measuring image-relation alignment: reference-free evaluation of vlms and synthetic pre-training for open-vocabulary scene graph generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 7614â7623. Cited by: §A.3. [39] H. H. Nguyen, T. N. Ta, N. C. Nguyen, V. T. Bui, H. M. Pham, and D. M. Nguyen (2021) Yolo based real-time human detection for smart video surveillance at the edge. In 2020 IEEE eighth international conference on communications and electronics (ICCE), p. 439â444. Cited by: §A.5.2, §3.4. [40] OpenAI (2024) GPT-4o mini: advancing cost-efficient intelligence. Note: https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/Accessed: 2025-08-12 Cited by: Table 2, §4. [41] G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. A. Osman, D. Tzionas, and M. J. Black (2019) Expressive body capture: 3D hands, face, and body from a single image. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), p. 10975â10985. Cited by: §3.1. [42] (2020) Product catalog 2020. Dahua Technology. External Links: Link Cited by: §A.1.1. [43] (2025) Product comparison tables â q2 2025. Axis Communications. Note: https://w.axis.com/dam/public/permalink/239412/product-comparison-tables--q2-2025-en-US_239412.pdf External Links: Link Cited by: §A.1.1. [44] (2023) Product guide â winter 2023â2024. Hikvision, USA. External Links: Link Cited by: §A.1.1. [45] A. R. Punnakkal, A. Chandrasekaran, N. Athanasiou, A. Quiros-Ramirez, and M. J. Black (2021) BABEL: bodies, action and behavior with english labels. In Proceedings IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), p. 722â731. External Links: Document Cited by: §A.1.2, §3.1. [46] A. N. Ramesh, A. Correas-Serrano, and M. GonzĂĄlez-Huici (2024) SCaRL-a synthetic multi-modal dataset for autonomous driving. In ICMIM 2024; 7th IEEE MTT Conference, p. 103â106. Cited by: §2. [47] R. Rawal, K. Saifullah, M. FarrĂ©, R. Basri, D. Jacobs, G. Somepalli, and T. Goldstein (2024) Cinepile: a long video question answering dataset and benchmark. arXiv preprint arXiv:2405.08813. Cited by: §2, Table 1. [48] E. Reinhard, M. Stark, P. Shirley, and J. Ferwerda (2002) Photographic tone reproduction for digital images. ACM Trans. Graph. 21 (3), p. 267â276. Note: https://doi.org/10.1145/566654.566575 External Links: ISSN 0730-0301, Link, Document Cited by: §A.1.5. [49] G. A. D. Relations (2025) Introducing gemma 3n: the developer guide. Note: https://developers.googleblog.com/en/introducing-gemma-3n-developer-guide/Developer blog post Cited by: Table 2, Table 2. [50] I. Sobel and G. Feldman (1968) An isotropic 3Ă3 image gradient operator. Technical report Stanford Artificial Intelligence Laboratory (SAIL). Note: Presentation at Stanford A.I. Project 1968 External Links: Link Cited by: §3.2. [51] G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. RamĂ©, M. RiviĂšre, et al. (2025) Gemma 3 technical report. arXiv preprint arXiv:2503.19786. Cited by: §A.1.2, §1, Table 2, Table 2, Table 2, §4. [52] H. F. Team (2025) SmolVLM2-2.2b-instruct. Note: https://huggingface.co/HuggingFaceTB/SmolVLM2-2.2B-InstructHugging Face Model Card Cited by: Table 2. [53] M. K. Turkcan, Y. Li, C. Zang, J. Ghaderi, G. Zussman, and Z. Kostic (2024) Boundless: generating photorealistic synthetic data for object detection in urban streetscapes. arXiv preprint arXiv:2409.03022. Cited by: §2. [54] Ultralytics (2024) YOLO benchmark modes and performance metrics. Note: https://docs.ultralytics.com/modes/benchmark/Accessed: 2025-08-12 Cited by: §4. [55] S. A. Velastin, R. FernĂĄndez, J. E. Espinosa, and A. Bay (2020) Detecting, tracking and counting people getting on/off a metropolitan train using a standard video camera. Sensors 20 (21), p. 6251. Cited by: §A.5.2, §3.4. [56] L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, and F. Wei (2022) Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533. Cited by: §A.1.2, §A.3. [57] W. Wang, Z. He, W. Hong, Y. Cheng, X. Zhang, J. Qi, M. Ding, X. Gu, S. Huang, B. Xu, et al. (2025) Lvbench: an extreme long video understanding benchmark. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 22958â22967. Cited by: §2, Table 1. [58] M. Wrenninge and J. Unger (2018) Synscapes: a photorealistic synthetic dataset for street scene parsing. arXiv preprint arXiv:1810.08705. Cited by: §2. [59] Z. Yang, Z. Cai, H. Mei, S. Liu, Z. Chen, W. Xiao, Y. Wei, Z. Qing, C. Wei, B. Dai, et al. (2023) Synbody: synthetic dataset with layered human models for 3d human perception and modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 20282â20292. Cited by: §2. [60] J. Ye, B. Zhou, Z. Huang, J. Zhang, T. Bai, H. Kang, J. He, H. Lin, Z. Wang, T. Wu, et al. (2025) Loki: a comprehensive synthetic data detection benchmark using large multimodal models. In ICLR, Cited by: Table 1. [61] S. Ye, H. Qi, A. Mathis, and M. W. Mathis (2025) LLaVAction: evaluating and training multi-modal large language models for action recognition. arXiv preprint arXiv:2503.18712. Cited by: Table 2, §4. [62] X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, et al. (2024) Mmmu: a massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 9556â9567. Cited by: §1, §2, Table 1. [63] Y. Zhang, B. Li, h. Liu, Y. j. Lee, L. Gui, D. Fu, J. Feng, Z. Liu, and C. Li (2024) LLaVA-next: a strong zero-shot video understanding model. Note: https://llava-vl.github.io/blog/2024-04-30-llava-next-video/ External Links: Link Cited by: Table 2, §4. [64] J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, H. Tian, Y. Duan, W. Su, J. Shao, et al. (2025) Internvl3: exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479. Cited by: §1, Table 2, Table 2, §4. Appendix A Technical Appendices and Supplementary Material A.1 Additional Synthetic Image Creation Details A.1.1 Camera Specification Survey The camera specification survey in table 4 details the typical configuration in fixed-camera systems. Across major manufacturers(Axis [43], Hikvision [44], Dahua [42]), the dominant hardware profile consists of 1/2.8" sensors paired with fixed 2.8m lenses. This configuration is used for our simulated hardware to capture the images in Blender. Table 4: Specifications of representative commercial camera models commonly used in fixed-camera systems. Manufacturer Model Resolution Sensor Lens Type Focal Length (m) Aperture Axis Q1615-LE Mk I 1920Ă1080 1/2.8â Varifocal 2.8â8.5 f/1.2 Axis Q1715 1920Ă1080 1/2.8â Varifocal 4â84.6 f/1.6â4.5 Axis P1385 1920Ă1080 1/2.8â Varifocal 2.8â13 f/1.4 Axis P1385-B 1920Ă1080 1/2.8â â â â Axis P1385-E 1920Ă1080 1/2.8â Varifocal 2.8â13 f/1.4 Axis P1385-BE 1920Ă1080 1/2.8â â â â Hikvision HX0D12Z2S 1920Ă1080 1/2.8â Varifocal 2.7â13.5 f/1.4 Hikvision HX-B12Z2S 1920Ă1080 1/2.8â Varifocal 2.7â13.5 f/1.4 Hikvision HX-D12F2S 1920Ă1080 1/2.8â Fixed 2.8 f/1.4 Hikvision HX-T12F6S 1920Ă1080 1/2.8â Fixed 2.8 f/1.4 Hikvision HX-B12F2S 1920Ă1080 1/2.8â Fixed 2.8 f/1.4 Hikvision HX-LB12F6S 1920Ă1080 1/2.8â Fixed 2.8 f/1.4 DaHua N25CL5Z 1920Ă1080 1/2.8â Varifocal 2.7â13.5 f/1.4 DaHua N24CL52 1920Ă1080 1/2.8â Fixed 2.8 f/1.6 DaHua DH-IPC-HDBW4239RN-ASE 1920Ă1080 1/2.8â Fixed 3.6 f/1.0 DaHua N24BN52 1920Ă1080 1/2.8â Fixed 2.8 f/2.0 DaHua N22AL12 1920Ă1080 1/2.7â Fixed 2.8 f/2.0 DaHua N28BL7Z 1920Ă1080 1/1.9â Varifocal 4.1â16.4 f/1.53 A.1.2 Action Pose Filtering Pipeline To identify relevant visually discernible actions from the BABEL dataset [45], a three-stage filtering process was applied. First, to reduce the excessive number of frames, frame-level BABEL action descriptions were randomly shuffled, and 100 frames were sampled to get a diverse representation of frame labeled actions. The sampled frames were embedded using E5-large-v2[56], and cosine similarity was computed against the phrase "outdoor abnormal action" to obtain an initial relevance ranking. Candidate actions are subsequently reranked by Gemma-3-13b-it [51] according to action relevance. Finally a manual verification step is performed to remove semantically inconsistent labels and to align the labels with human perception from a still frame. The resulting subset prioritizes visually distinguishable static actions suitable for single-frame perception evaluation. The selected poses are retargeted and animated within Blender. The final action set comprises: walking, kicking, sneaking, choking, dodging and covering face with hands, stepping over something and guard up. A.1.3 Discernible Label Model Ablation Table 5: Performance of discernibility annotation models on validation set (5-Fold cross-validation). Model Macro F1 (%) Accuracy (%) Decision Tree 82.41 90.25 RF + XGBoost Ensemble 83.55 90.50 A.1.4 Modeling Distance by Pixel Density Under the pinhole camera model [17], the vertical field of view (VFOV) is defined as: ΞVFOV=2âtanâ1âĄ(hsensor2âf), _VFOV=2 ^-1\! ( h_sensor2f ), (1) where hsensorh_sensor denotes the sensor height and f the focal length. At distance D, the corresponding vertical scene coverage is: hscene=2âDâtanâĄ(ΞVFOV2),h_scene=2D \! ( _VFOV2 ), (2) and the resulting vertical pixel density is given by: PâPâMvert=hreshscene,PPM_vert= h_resh_scene, (3) where hresh_res denotes the image height in pixels. Rearranging yields the viewing distance as a function of the required pixel density: D=hresPâPâMreqâ 2âtanâĄ(ΞVFOV2),D= h_resPPM_req· 2 \! ( _VFOV2 ), (4) where PâPâMreqPPM_req denotes the target DORI pixel density threshold. A.1.5 Night-Vision Simulation Real commercial night-vision systems rely on proprietary image signal processing (ISP) pipelines that are typically inaccessible [4]. To approximate these systems, we implement a vendor-neutral ISP pipeline inspired by prior computational imaging work [4, 18]. The pipeline consists of: (1) sRGB linearization, (2) automatic gain control (AGC) normalization [15], (3) color normalization [7], (4) BM3D denoising [12, 34], (5) local contrast enhancement using CLAHE [36], and (6) tone mapping [48]. This process produces physically plausible night-vision imagery while preserving consistency with the underlying DORI-calibrated visibility conditions. A.2 Additional Task Definition and Question Type Action Recognition aims to determine if a model is able to identify the specific activity performed by the human subject presented in each image. This task evaluates an LVLMâs ability to interpret human pose, motion cues, and contextual elements necessary for recognizing semantically distinct behaviors. Each image is paired with an instruction and open-ended question (âRespond only with the name of the action. What action is the person performing in the image?â) requiring a short natural-language response such as walking, running, or waving. Unlike the binary HPC task, action recognition emphasizes higher-level visual understanding, testing a modelâs ability to associate human posture and spatial configuration with the corresponding action category. It also provides insight into the modelâs robustness under DORI-calibrated conditions, where reduced visibility, distance, and illumination can degrade action-level semantic recognition. A.3 Additional Metrics For the Action Recognition task, we follow prior LVLM evaluation protocols that treat open-ended responses as text predictions and therefore evaluate semantic agreement beyond exact string matching [19]. For open-ended action recognition, our prompts explicitly instruct the model to respond concisely with the action, resulting in outputs that are typically 1-3 tokens. In this regime, lexical-overlap metrics (e.g., ROUGE-L) become unstable. Since utility depends on semantic proximity rather than exact lexical reproduction, we evaluate using E5Score [56] to quantify sentence-level semantic alignment, following the same embedding-similarity paradigm as CLIPScore [19] and SigLIPScore which is commonly used for Image-Text alignment [38], but modifying to fit our Text-to-Text setting. We use E5Score to avoid score bias due to the LVLMs using the vision encoders as the scoring metrics (e.g., CLIP, SigLIP). To improve interpretability, we report a distance-based z-score that highlights relative performance differences across models. Additionally, we obtain a human baseline from one non-expert annotator on a uniformly random sample of 300 instances. A.4 Computational Resources Raw image rendering was conducted on 4 NVIDIA A6000 GPUs, totaling approximately 535.7 GPU hours. Night-vision enhanced image creation was conducted on the CPU, totaling approximately 19 CPU hours. Experimental evaluations were conducted on 2â3 NVIDIA L40S GPUs, depending on availability. Each model required approximately 4 GPU hours across both tasks, totaling approximately 72 GPU hours for the full evaluation across 18 models. A.5 Additional Results A.5.1 Action Recognition Robustness Across Distances Table 6: Action recognition performance across DORI distances. Mean E5Score and corresponding z-score per distance. Best z-scores are in bold. Model 5.4 m 10.8 m 23.0 m Non-Expert Human 0.94 0.91 0.83 SmolVLM2-256M-Video-Instruct 0.78 (-1.45) 0.77 (-1.39) 0.77 (-1.15) SmolVLM2-500M-Video-Instruct 0.80 (-0.76) 0.79 (-0.75) 0.78 (-0.69) Internvl3-1B-hf 0.81 (-0.23) 0.80 (-0.43) 0.78 (-0.72) Perception-LM-1B 0.77 (-1.87) 0.76 (-1.78) 0.75 (-1.72) SmolVLM2-2.2B-Instruct 0.84 (0.93) 0.83 (0.98) 0.83 (1.09) Perception-LM-3B 0.77 (-1.98) 0.76 (-1.98) 0.75 (-1.83) Qwen2.5-VL-3B-Instruct 0.83 (0.69) 0.82 (0.61) 0.82 (0.65) gemma-3-4b-it 0.80 (-0.59) 0.80 (-0.52) 0.78 (-0.63) gemma-3n-E2B-it 0.84 (1.04) 0.84 (1.14) 0.83 (1.13) SVLM (mean) 0.80 (-0.79) 0.80 (-0.65) 0.79 (-0.39) Phi4-multimodal-instruct 0.84 (0.90) 0.83 (0.95) 0.83 (1.12) LLaVAction-7B 0.84 (0.89) 0.83 (0.95) 0.83 (0.98) LLaVA-NeXT-Video-7B-hf 0.82 (-0.02) 0.80 (-0.23) 0.78 (-0.64) Molmo-7B-D-0924 0.82 (0.23) 0.82 (0.39) 0.82 (0.51) Qwen2.5-VL-7B-Instruct 0.84 (0.88) 0.83 (0.88) 0.83 (0.90) gemma-3n-E4B-it 0.83 (0.57) 0.82 (0.64) 0.82 (0.59) internVL3-8B-hf 0.82 (0.34) 0.81 (0.17) 0.81 (0.18) MVLM (mean) 0.83 (0.44) 0.82 (0.59) 0.82 (0.53) gpt-4o-mini-2024-07-18 0.83 (0.44) 0.82 (0.39) 0.81 (0.23) We evaluate all LVLMs on SynDORBench-54k and then average over non-distance factor similarly to before, so that the reported scores reflect the effect of distance alone. Results are detailed in table 6. Across all three distances, we observe a monotonic decrease in performance as radius increases, indicating that action recognition is strongly distance-sensitive and degrades as pixel density falls. For group means, the MLVLMs exceed that of the SVLMs at all radii, suggesting that parameter count continues to provide a net advantage for open-ended action classification when visual detail becomes sparse. However, this effect is not purely size-deterministic, although MLVLMs have higher group means, Gemma-3n-E2B-it, despite not being the largest model, achieves the best individual performance across all three distances. This indicates that architectural and alignment choices can yield better semantic retention than scale alone. Notably, the mean MLVLM exhibits a similar level of performance to GPT-4o mini, indicating that open medium-scale architectures can approach commercial-grade action recognition at the DORI ranges considered. However, it should be noted that all LVLMs fail to surpass a non-expert human on all radii, highlighting a clear gap in capability on this task. A.5.2 HPC Performance Over Lightings To complement the main-manuscript analysis, we conduct a series of additional evaluations that probe how model performance varies across distance, lighting, and pixel-density regimes, offering a deeper view of robustness characteristics not captured in aggregate metrics. Table 7: HPC performance across DORI-calibrated distances (5.4m, 10.8m, 23.0m). We report accuracy and F1-score for all models. The best scores for each column are in bold; underlined numbers indicate the second-highest. day night night-vision radius(m) 5.4 10.8 23 avg 5.4 10.8 23 avg 5.4 10.8 23 avg metrics acc acc acc acc acc f1-score acc f1-score acc f1-score acc f1-score acc f1-score acc f1-score acc f1-score acc f1-score YOLO11x 95.06 85.42 53.49 77.99 62.80 50.51 54.29 51.65 47.00 44.96 54.70 49.04 63.90 51.25 56.43 53.36 50.39 49.11 56.91 51.24 SmolVLM2-256M-Video-Instruct 9.88 8.54 7.79 8.74 12.48 12.44 15.20 15.14 27.02 23.47 18.23 17.02 7.71 7.46 16.68 15.03 33.31 25.37 19.23 15.95 SmolVLM2-500M-Video-Instruct 23.49 26.40 11.53 20.47 7.82 7.57 16.87 15.20 33.11 24.88 19.27 15.88 38.29 34.09 36.24 36.05 36.93 31.02 37.16 33.72 Internvl3-1B-hf 99.95 99.77 95.09 98.27 78.20 61.48 74.94 68.32 71.96 71.49 75.03 67.10 85.17 67.41 82.35 75.46 84.87 84.16 84.13 75.68 Perception-LM-1B 90.80 78.79 40.33 69.97 43.88 37.99 39.05 38.65 35.92 29.44 39.62 35.36 74.64 59.07 70.07 64.61 61.58 61.50 68.76 61.73 SmolVLM2-2.2B-Instruct 95.73 92.02 71.57 86.44 64.79 51.92 57.95 54.62 54.37 53.72 59.04 53.42 82.01 64.39 77.71 70.39 73.60 73.24 77.77 69.34 Perception-LM-3B 99.33 95.42 70.20 88.31 60.82 49.23 52.96 50.57 46.53 44.37 53.44 48.06 67.14 53.43 66.91 61.92 59.67 59.50 64.58 58.29 Qwen2.5-VL-3B-Instruct 94.10 86.68 63.12 81.30 39.42 34.89 39.56 39.10 40.44 36.20 39.81 36.73 66.57 53.09 65.78 61.03 60.69 60.57 64.35 58.23 gemma-3-4b-it 99.98 99.95 100.00 99.97 92.67 51.26 90.71 57.49 75.59 48.54 86.32 52.43 91.33 50.05 88.07 54.64 81.40 54.49 86.93 53.06 gemma-3n-E2B-it 100.00 99.72 98.75 99.49 75.08 58.11 76.42 70.11 79.49 79.08 77.00 69.10 86.22 69.23 82.74 75.75 88.38 87.61 85.78 77.53 phi4-multimodal-instruct 99.93 98.51 86.10 94.85 72.85 57.54 71.59 65.89 75.70 75.44 73.38 66.29 82.13 62.25 80.16 72.71 76.79 75.57 79.70 70.18 LLaVAction-7B 97.20 89.92 58.74 81.95 65.07 52.11 56.78 53.67 47.70 45.84 56.52 50.54 83.14 66.49 77.40 70.99 67.20 67.19 75.91 68.22 LLaVA-NeXT-Video-7B-hf 96.69 87.48 56.86 80.35 68.88 54.73 59.94 56.22 52.77 51.95 60.53 54.30 80.39 60.65 72.02 61.40 55.07 53.31 69.16 58.45 Molmo-7B-D-0924 99.81 97.36 78.82 92.00 69.85 55.49 66.56 61.68 59.48 59.29 65.30 58.82 84.24 67.58 77.90 71.46 73.99 73.82 78.71 70.96 Qwen2.5-VL-7B-Instruct 99.93 99.57 93.80 97.77 81.24 64.19 79.15 72.43 75.66 75.42 78.68 70.68 77.23 61.07 75.53 69.27 71.26 71.14 74.67 67.16 gemma-3n-E4B-it 99.98 99.55 99.07 99.53 76.46 59.28 78.29 71.80 81.71 81.18 78.82 70.75 86.87 67.97 82.35 73.46 82.49 80.25 83.90 73.89 internVL3-8B-hf 99.98 98.97 79.58 92.84 72.49 57.46 70.42 64.91 62.32 62.27 68.41 61.54 83.87 66.79 80.01 73.37 73.09 72.95 78.99 71.04 gpt-4o-mini-2024-07-18 98.97 95.82 73.84 89.54 69.25 55.05 63.21 58.91 60.34 60.20 64.26 58.05 79.94 62.59 74.59 68.31 74.61 74.42 76.38 68.44 Figure 3: HPC Robustness: Accuracy across DORI-calibrated distances under different lighting conditions. The left, middle, and right subplots correspond to day, night-vision (NV), and night settings, respectively. With each subplot, distance increases from left to right, corresponding to progressively more difficult evaluation conditions. To demonstrate a deeper robustness investigation via metadata stratification analysis, we analyze how degradation patterns vary with illumination by stratifying by lighting conditions. We report human-presence classification (HPC) performance over distances separately for day, night-vision, and night settings in table 7 and fig. 3. Under day conditions, nearly all LVLMs outperform YOLO11x, with the exception of SmolVLM2-256M-Video-Instruct and SmolVLM2-500M-Video-Instruct. This indicates that, in well-lit environments, LVLMs above roughly 1B parameters are generally capable of surpassing conventional detectors in terms of HPC and suggests that in general, LVLMs may have better perception capabilities. When inspecting how parameters affect performance, as seen in fig. 3, YOLO and SVLMs degrade sharply at 23m, meaning that already in ideal lighting conditions, SVLMs and YOLO perform worse than our proposed failure threshold (80%) based on reported operational performance of models [37, 30, 55, 39]. In contrast, night conditions lead to a substantial collapse in performance. No model attains an F1-score of 0.8, levels that many models comfortably achieve during the day, highlighting the severe degradation induced by low-light environments. In such scenarios, all evaluated LVLMs become effectively non-operational. Encouragingly, applying night-vision enhancement yields partial recovery in model performance. Several models show modest but consistent gains, including gemma-3-4b-it and gemma-3n-E2B-it, suggesting that algorithmic illumination compensation is beneficial for vision-language understanding. Across all lighting conditions and distances, the Gemma3 family: Gemma3 and Gemma3n, consistently achieves the strongest HPC performance, and specifically, gemma-3-4b-it and gemma-3n-E2B-it achieve the best overall performance. Interestingly, the smaller variants tend to outperform their larger counterparts, indicating that model scale within this family does not linearly translate to better HPC robustness. However, for the night lighting conditions, the Mid-sized LVLMs show promise in beating gemma-3-4b-it and gemma-3n-E2B-it, this could be due to the higher reasoning capabilities of the larger models to understand the contextual clues.