Paper deep dive
VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models
Zaid Pervaiz Bhat, Nimra Nayyar, Arihant Jain, Lap Fung Chan, John Suchanek, Yu Wang, Varun Praveen, Tomasz Kornuta, Vidya Nariyambut Murali
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/10/2026, 2:42:52 AM
Summary
The paper introduces VANTAGE-Bench, a benchmark designed to evaluate the 'Infrastructure AI Gap' in Vision-Language Models (VLMs). Unlike existing benchmarks focused on consumer or embodied AI, VANTAGE-Bench assesses VLMs on fixed-camera infrastructure video across three domains: Logistics, Transportation, and Smart Spaces. It employs eight diverse task formulations (including dense captioning, temporal localization, and single-object tracking) to measure semantic, spatial, temporal, and spatio-temporal capabilities. The study evaluates 17 models zero-shot, finding that performance deficits are concentrated in temporal and referring expression tasks rather than being uniform, with open-weight models performing competitively in 2D localization.
Entities (13)
Relation Signals (10)
VANTAGE-Bench → coversdomain → Logistics
confidence 95% · It spans three operational domains (Logistics, Transportation, and Smart Spaces)
VANTAGE-Bench → coversdomain → Transportation
confidence 95% · It spans three operational domains (Logistics, Transportation, and Smart Spaces)
VANTAGE-Bench → coversdomain → Smart Spaces
confidence 95% · It spans three operational domains (Logistics, Transportation, and Smart Spaces)
VANTAGE-Bench → evaluates → Vision-Language Models
confidence 95% · Evaluating 17 models zero-shot, we find the shortfall relative to consumer-centric benchmarks is concentrated
VANTAGE-Bench → includestask → Temporal localization
confidence 95% · Temporal pillar is weakest in absolute terms: no system exceeds 55.7 mIoU on temporal localization
VANTAGE-Bench → includestask → Single Object Tracking
confidence 95% · It adds a single-pass trajectory protocol for Single Object Tracking
VANTAGE-Bench → includestask → Dense Video Captioning
confidence 95% · no system exceeds 55.7 mIoU on temporal localization or 37.3 SODA_c on dense video captioning
VANTAGE-Bench → createdby → NVIDIA
confidence 90% · The core footage for VANTAGE-Bench was curated by NVIDIA
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:As Vision-Language Models (VLMs) advance toward physical deployment, the focus has remained on action-oriented Embodied AI evaluated on subject-centric consumer video. This overlooks a pervasive class of Physical AI: Infrastructure AI, which relies on fixed cameras for open-loop insights like safety monitoring and operational logging. We introduce VANTAGE-Bench, a benchmark measuring this "Infrastructure AI Gap." It spans three operational domains (Logistics, Transportation, and Smart Spaces), unifies image and video evaluation across semantic, spatial, temporal, and spatio-temporal capabilities, and moves beyond multiple-choice to eight task formulations including dense captioning and spatio-temporal grounding. It adds a single-pass trajectory protocol for Single Object Tracking and, to our knowledge, the first such evaluation on fixed-camera infrastructure video, scored against specialist trackers. Annotation spans three regimes over 3,346 media assets: 3,342 video-task annotations, 4,281 image-grounding annotations, and 27,404 detection boxes. Evaluating 17 models zero-shot, we find the shortfall relative to consumer-centric benchmarks is concentrated, not general. Event verification, referring expressions, and temporal localization fall roughly 9 to 24 points at every model scale, while video question answering stays within 5.3 points of VideoMME and 2D spatial pointing shows no shortfall against BLINK. The temporal pillar is weakest in absolute terms: no system exceeds 55.7 mIoU on temporal localization or 37.3 SODA_c on dense video captioning. On tracking, frontier models come within roughly 5 points of specialist trackers over short horizons but separate as the horizon extends. Open-weight models lead 2D object localization outright, so neither scale nor proprietary access explains the pattern. Data, evaluation harness, and leaderboard: this https URL
Tags
Links
- Source: https://arxiv.org/abs/2609.09396v1
- Canonical: https://arxiv.org/abs/2609.09396v1
Trouble viewing inline? Open PDF directly →
Full Text
82,427 characters extracted from source content.
Expand or collapse full text
https://vantage-bench.org/ VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models Zaid Pervaiz Bhat 1,*,† , Nimra Nayyar 2,* , Arihant Jain 1,* , Lap Fung Chan 2,* , John Suchanek 2 , Yu Wang 1 , Varun Praveen 1 , Tomasz Kornuta 1 and Vidya Nariyambut Murali 1 1 NVIDIA 2 Clemson University Abstract As Vision-Language Models (VLMs) rapidly advance toward physical deployment, the predominant focus has re- mained on action-oriented Embodied AI evaluated on subject-centric consumer video. This trajectory overlooks a pervasive class of Physical AI: Infrastructure AI, which relies on fixed-infrastructure cameras for open-loop insights like safety monitoring and operational logging. How well VLMs process dense, fixed-camera visual data remains insufficiently explored. We introduce VANTAGE-Bench, an evaluation benchmark that measures this “Infrastructure AI Gap,” distinguished by four features: 1) Domain Relevance, spanning three operational environments (Logistics, Transportation, and Smart Spaces); 2) Modality Breadth, unifying image and video evaluation to probe semantic, spatial, temporal, and spatio-temporal capabilities; 3) Task Format Diversity, moving beyond multiple-choice to eight task formulations spanning discriminative QA, generative dense caption- ing, and spatio-temporal grounding; and 4) Evaluation Novelty, a single-pass trajectory-generation protocol for Single Object Tracking and, to our knowledge, the first such evaluation on fixed-camera infrastructure video, scored against specialist trackers. Annotation spans three regimes over 3,346 expert-annotated media assets: video tasks (3,342 annotations over 854 videos), image grounding (4,281 over 1,864 images), and dense detection (27,404 boxes over 628 images). Evaluating 17 models zero-shot, we find the shortfall relative to consumer-centric benchmarks is concentrated rather than general. Event verification, referring expressions, and temporal localization fall roughly 9 to 24 points at every model scale, while video question answering stays within 5.3 points of VideoMME and 2D spatial pointing shows no shortfall against BLINK. The temporal pillar is weakest in absolute terms: no system exceeds 55.7 mIoU on temporal localization or 37.3 SODA 푐 on dense video captioning. On tracking, frontier models come within roughly 5 points of specialist trackers over short horizons but separate as the horizon extends. Open-weight models lead 2D object localization outright, so neither parameter count nor proprietary access accounts for the pattern. The dataset is available athttps://huggingface.co/datasets/nvidia/PhysicalA I-VANTAGE-Bench, and the evaluation harness and public leaderboard at https://vantage-bench.org/. 1. Introduction As VLMs rapidly advance toward physical deployment, the prevailing research trajectory has heavily indexed on action-oriented Embodied AI and evaluations grounded in subject-centric consumer video (e.g., egocentric recordings, broadcast media). This focus overlooks a pervasive and foundational class of physical AI: Infras- tructure AI. Relying on fixed-infrastructure cameras (e.g., CCTV, elevated sensors), Infrastructure AI provides open-loop insights for safety monitoring, operational logging, and large-scale spatial reasoning. Because frontier models are predominantly trained on internet-crawled data, they are biased toward standard photographic perspectives. Consequently, their capability to process the dense, oblique, and multi-agent visual data inherent to fixed-camera environments remains insufficiently evaluated, creating a severe implicit domain shift. Under these conditions, the “internet-video prior” fails, and models must rely on pure spatial-temporal logic. A model can achieve state-of-the-art scores on internet video and still perform poorly in the environments * Equal contribution. † Project lead. Correspondence: zbhat@nvidia.com. © 2026 NVIDIA. All rights reserved. arXiv:2609.09396v1 [cs.CV] 8 Sep 2026 VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models where these systems are actually deployed. We term this the Infrastructure AI Gap: the discrepancy between a model’s performance on general video benchmarks and its reliability as an insight-generating system in dense, fixed-camera infrastructure environments. Closing this gap requires evaluation frameworks designed natively for operational reality. While comprehensive suites like VideoMME [1] and MVBench [2] evaluate multiple reasoning dimensions, they ultimately reduce evaluation to a single format: MCQ. MCQ evaluation provides the correct answer as an implicit prior and fails to test if a model can autonomously generate bounding box coordinates or predict explicit temporal boundaries. To systematically measure and close this capability deficit, we introduce VANTAGE-Bench (Video Analysis Tasks Across Generalized Environments). VANTAGE-Bench is the first multi-task benchmark specifically curated for Infrastructure AI, moving beyond single-format MCQ to evaluate eight distinct task formulations across four pillars of operational visual intelligence: Semantic, Spatial, Temporal, and Spatio-Temporal Understanding. Finding: Our zero-shot evaluation of 17 models shows that the deficit is localized rather than uniform, and the asymmetry is what makes it interpretable. Measured against the same models’ published scores on consumer- centric benchmarks, event verification, referring expressions, and temporal localization drop at every scale, while video question answering holds within 5.3 points of VideoMME [1] and 2D spatial pointing shows a surplus of 8.3 points against BLINK [4] at 32B. The same models therefore lose ground on some capabilities and gain it on others within one suite, which places the shortfall in specific capabilities rather than in overall difficulty. Section 4 tightens this with a prompt- and metric-matched control, and shows that models carrying more physical-AI training data gain 14 to 26 points on tracking and object localization while the two temporal tasks stay nearly untouched — an asymmetry we trace to the scarcity of densely timestamped video supervision rather than to a general weakness in video understanding. Those two tasks remain the weakest in absolute terms across the entire suite, at frontier scale as much as at 2B. Our main contributions are: 1.VANTAGE-Bench: The first multi-task benchmark specifically curated for Infrastructure AI, spanning three deployment domains – Logistics/Warehouse, Transportation, and Smart Spaces – addressing the structural distribution shift from egocentric video. 2.Format-Diverse Evaluation: Eight task formulations spanning discriminative QA, generative dense caption- ing, and spatio-temporal grounding. Five of the eight require the model to emit bounding boxes, temporal boundaries, or a full trajectory with no candidate set supplied, removing the implicit answer prior that MCQ evaluation provides; only two are multiple-choice and one binary. 3.Tracking Protocol: A single-pass trajectory-generation protocol for Single Object Tracking (SOT) in VLMs and, to our knowledge, the first such evaluation on fixed-camera infrastructure video spanning open-weight and proprietary models, reported against specialist-tracker, static-box, and random baselines. 4.Expert-Annotated Dataset: 3,346 media assets across three annotation regimes: 3,342 video-task an- notations over 854 videos, 4,281 image-grounding annotations over 1,864 images, and 27,404 detection boxes over 628 images. Annotation density is uneven across regimes by design, and we report the regimes separately rather than as a single aggregate. 5.Open Evaluation Ecosystem: An open-source evaluation suite and a live public leaderboard with held-out ground truth (Appendix F.3). 6.Empirical Characterization: A 17-model zero-shot evaluation establishing that the gap is localized rather than uniform — event verification, referring expressions, and temporal localization fall roughly 9 to 24 points against the same models’ published scores, while video question answering and 2D spatial pointing do not — and that the two temporal tasks hold the lowest ceilings of the eight, unclosed at frontier scale. 2 VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models 2. Related Work 2.1. General Video Benchmarks and the MCQ Bottleneck The dominant paradigm in VLM evaluation relies on general-purpose, multi-task benchmarks such as VideoMME [1], MVBench [2], and EgoSchema [3]. While these comprehensive suites have driven substantial progress in general video understanding, their relevance to operational deployment is limited by two factors. First, they rely on a “cinematic prior”—utilizing web-sourced, broadcast, or egocentric footage characterized by high-velocity optical flow, human-centric framing, and edited temporal structures. Second, they suffer from the “MCQ Bottleneck.” By reducing complex spatio-temporal reasoning to MCQ, these benchmarks provide the correct answer as an implicit prior. This format does not test the open-ended generative capabilities—such as producing precise bounding box coordinates or explicit temporal boundaries—that autonomous Infrastructure AI requires to generate reliable downstream insights. 2.2. Embodied AI Datasets vs. Classical Surveillance The physical AI community has developed extensive datasets for real-world deployment, but these efforts remain fragmented across two divides. First is the operational divide. Datasets like KITTI [5], nuScenes [6], and Waymo [7] are optimized exclusively for Embodied AI—featuring moving cameras, low-latency requirements, and closed-loop perception. On the other side is classical surveillance, where datasets like UCF-Crime [8] and VIRAT [9] capture the correct fixed-infrastructure perspective, but rely on classical bounding boxes or binary anomaly labels without a natural language interface. Second is the domain divide. Existing enterprise datasets are strictly siloed—focusing solely on either transportation (e.g., AI City Challenge [10]), generic crime (UCF-Crime), or robotic manipulation in logistics. VANTAGE-Bench bridges both divides simultaneously. It pairs the fixed-infrastructure perspective of classical surveillance corpora with a natural-language interface, and unifies evaluation across three distinct deployment domains: Transportation, Warehouse, and Smart Spaces. This multi-domain approach ensures models are evaluated on generalized physical reasoning rather than overfitted domain-specific priors. 2.3. Spatial and Temporal Grounding in Idealized Settings Grounding language to spatial and temporal coordinates is a foundational requirement for physical AI, yet existing benchmarks evaluate these capabilities under highly idealized conditions. Spatial grounding and referring expression datasets like RefCOCO [13] and BLINK utilize natural indoor and outdoor photography featuring well-lit, centered, and distinct objects. This fails to evaluate the dense, multi-instance disambiguation required when monitoring infrastructure from an elevated, oblique perspective (e.g., distinguishing between dozens of identical pallets or vehicles). Similarly, temporal grounding benchmarks like Charades-STA [11] and ActivityNet Captions [12] utilize scripted, continuous human actions where the event of interest dominates the video timeline. In contrast, operational video is characterized by long quiescent periods. VANTAGE-Bench forces models to search through extended periods of inactivity to localize brief, sparse, safety-critical events, which tests temporal attention and contextual recall more directly. 2.4. Spatio-Temporal Tracking and Grounding Tracking objects continuously through dense scenes is among the most demanding spatio-temporal tasks. We position our Single Object Tracking track against four adjacent lines of work. Classical tracking benchmarks. LaSOT [14], GOT-10k [15], and MOTChallenge [16] were designed to evaluate dedicated computer vision pipelines that maintain an explicit internal state (e.g., appearance models, motion priors, and frame-by-frame memory updates). We evaluate four specialist trackers—SAM 3 [25], SAMURAI [26], SUTrack [27], and MCITrack [28]—as anchors on our own tracking annotations: they reach approximately 80 Success AUC on the same tracklets and scorer, establishing the level a purpose-built system attains and giving the VLM scores a reference point (Appendix D). 3 VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models Infrastructure AI DomainsEvaluation PillarsTask Taxonomy Logistics (Warehouse) W Transportation T Smart Spaces S Temporal The When Spatio-Temporal The Dynamics Spatial The Where Semantic The What / Why Dense Video Captioning WTSS 717Events 104Videos Temporal Localization WTSS 1,067QA 203Videos Single Object Tracking W 200Objects 102Videos 2D Spatial Pointing T 1,005QA 361Images 2D Object Localization S 27,404Bboxes 628Images 2D Referring Expressions T 3,276QA 1,503Images Event Verification WTSS 163QA 163Videos Video Question Answering WTSS 1,195QA 282Videos 3 Domains · 4 Pillars · 8 Tasks3,346 Media35,027 Annotations Figure 1: Taxonomy and distribution of VANTAGE-Bench. The benchmark spans three operational domains (W: Warehouse, T: Transportation, and S: Smart Spaces), evaluating eight distinct tasks across Semantic, Spatial, Temporal, and Spatio-Temporal pillars. In total, the dataset comprises 3,346 media assets across three annotation regimes: 3,342 video-task annotations, 4,281 image-grounding annotations, and 27,404 detection boxes. Language-conditioned tracking. TNL2K [17] evaluates iterative language-conditioned tracking by dedicated architectures, whereas our protocol is single-pass trajectory generation by a general-purpose VLM from a visual first-frame box. SOT removes language in order to isolate visual correspondence over time. Joint spatio-temporal grounding. VidSTG [18] and HC-STVG [19] evaluate joint spatio-temporal grounding, where one score conflates language grounding, spatial localization, and temporal localization. Our pillar design decomposes exactly these: 2D Referring Expressions isolates language to space, Temporal Localization isolates language to time, and SOT removes language to isolate visual correspondence over time. Tracking with multimodal language models. Elysium [20] and Merlin [21] are purpose-trained tracking MLLMs. VideoChat-R1 [23] and R1-Track [24] address zero-shot box-initialized tracking of open VLMs. MMT-Bench [22] evaluates tracking as multiple-choice box selection, the format limitation our protocol moves past. Our protocol. The model is initialized with the target object’s bounding box visually overlaid on the first frame. Through a single instruction, it then processes the subsequent multi-frame sequence and outputs a continuous coordinate trajectory, without rolling memory updates. We therefore describe our contribution as a single-pass trajectory-generation tracking protocol for VLMs and, to our knowledge, the first such evaluation on fixed-camera infrastructure video, spanning open-weight and proprietary models. 4 VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models 3. Dataset and Benchmark Construction To construct a benchmark capable of testing the Infrastructure AI gap, we curated a multi-modal dataset combining expert-annotated real-world infrastructure footage with a limited set of high-fidelity simulated environments. The dataset is overwhelmingly human-labeled. Automated procedures are used to structure and scale one task format, 2D Spatial Pointing, where the question and its correct answer are both derived deterministically from human-labeled box coordinates (Section 3.4). VANTAGE-Bench comprises 3,346 media assets across eight tasks (see Figure 1). The video tasks and 2D Object Localization draw on fixed-infrastructure cameras: static, elevated, wide-angle viewpoints at mounting heights of 8 to 20 feet and viewing angles of 30 to 60 degrees from vertical, including low-light, night, and rain conditions. These are highway ITS and warehouse installations. The two remaining image tracks sample different geometries: 2D Referring Expressions uses oblique aerial views from RefDrone [29], and 2D Spatial Pointing is derived from object-localization boxes over vehicle-mounted footage. Single Object Tracking uses synthetic elevated-fixed warehouse sequences from PhysicalAI-SmartSpaces [30]. To support diverse physical reasoning, approximately 20% of the Video Question Answering and Temporal Localization splits feature high-fidelity synthetic simulations. VANTAGE-Bench is publicly released under the NVIDIA Evaluation Data License, which strictly restricts dataset usage to evaluation and benchmarking purposes, available athttps://huggingface.co/datasets/nvidia/PhysicalAI-VANTA GE-Bench. 3.1. Design Motivation The architecture of VANTAGE-Bench is driven by the disconnect between how VLMs are currently evaluated and how Infrastructure AI operates in production. Existing video benchmarks inadvertently provide models with two strong priors: the “cinematic prior” of dynamic, human-centric framing, and the implicit hints embedded within MCQ formats. To evaluate physical grounding, we constructed VANTAGE-Bench around three core design principles: •Ecological Validity: We strip away internet-video biases by drawing the video tasks and 2D Object Local- ization from static, elevated, wide-angle infrastructure cameras; the remaining image tracks sample other operational geometries. This forces models to disambiguate dense, multi-instance scenes without the aid of tracking shots or ideal lighting. •Format Authenticity: By moving beyond the MCQ bottleneck, we evaluate autonomous insight generation. VANTAGE-Bench requires models to predict exact temporal boundaries, 2D coordinates, and continuous spatio-temporal trajectories. •Dimensional Isolation: We design specialized task tracks to independently evaluate Semantic, Spatial, Temporal, and Spatio-Temporal intelligence, allowing researchers to isolate the exact locus of model failure. 3.2. Task Taxonomy We organize VANTAGE-Bench around four reasoning pillars, evaluating the distinct functional capabilities required for reliable Infrastructure AI. High-level descriptions are provided below, with formal mathematical formulations and evaluation metrics detailed in Appendix B. Pillar I: Semantic Understanding. This pillar evaluates high-level causal logic and operational reasoning via: (1) Event Verification (EV), which requires models to verify operational hypotheses against visual evidence, and (2) Video Question Answering (VQA), which tests multi-step logical reasoning over untrimmed sequences. Pillar I: Spatial Understanding. This pillar evaluates precise geometric grounding within infrastructure scenes through: (3) 2D Referring Expressions (RefEx), requiring dense semantic disambiguation to localize targets; (4) 2D Spatial Pointing, evaluating coordinate selection from positional prompts; and (5) 2D Object Localization, establishing a baseline for class-level spatial awareness. Pillar I: Temporal Understanding. This pillar focuses on perceiving action duration and sequence boundaries using: (6) Temporal Localization, requiring exact start/end timestamp prediction; and (7) Dense Video 5 VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models Semantic Understanding Video Question Answering Q Did both individuals use an access credential to enter? A Yes, both scanned their badges B No, only the first person scanned✓ C Neither person used a credential D The video does not show an entrance Event Verification Q Did a tailgating event occur at the entrance? A Yes Q Did an unauthorized access event occur? A Yes Temporal Understanding 5.49 s6.11 s8.02 s Temporal Localization Q When does the collision between the car and motorcycle occur? A 6.10 – 8.40 s Dense Video Captioning 5.40 – 6.11 s Car enters intersection 6.11 – 7.32 s Motorcycle hits the car 8.02 – 10.42 s Motorcycle hits the ground Spatial Understanding 2D Object Localization Locate every instance that belongs to the following categories: 'person'. Report bbox coordinates in JSON format. 2D Spatial Pointing Q Where is the leftmost car in this image? Options (coordinates [x, y] in the source frame) A [129, 896]✓ B [291, 809] C [401, 794] D [519, 804] 2D Referring Expressions Locate every instance described by the following expression: "The black motorcycle is located near the bottom right of the image." A [1768, 870, 1852, 992] Spatio-Temporal Understanding frame 1frame 3 frame 4frame 7 Single Object Tracking Given an initial object anchor, identify and track that object across the given frames. Figure 2: Visual examples of the eight evaluation tasks in VANTAGE-Bench, categorized by their primary reasoning pillar: Semantic, Spatial, Temporal, and Spatio-Temporal Understanding. Captioning (DVC), a generative task for autonomous localization and description of chronological events. Pillar IV: Spatio-Temporal Understanding. This pillar evaluates continuous visual persistence through: (8) Single Object Tracking (SOT). Unlike standard VLM tasks, SOT requires predicting a target’s coordinate trajectory across a full sequence, testing the maintenance of spatial context over time. 3.3. Source Data and Expert Human Annotation The core footage for VANTAGE-Bench was curated by NVIDIA to capture hard to source fixed-camera infrastruc- ture perspectives. Real-world footage originates from two U.S. municipalities, San Francisco, CA and Dubuque, IA, supplemented by publicly sourced footage including European footage under license. Consent provenance and the PII obfuscation and verification pipeline are detailed in Appendix E. Annotations were generated by trained professionals—not crowdsourced—using domain-specific rulebooks. Annotators train for months, only the top 10% of the pool work on benchmark data, and per-task rulebooks are stabilized over up to two months before production. Our quality pipeline uses a multi-tier validation structure: 100% of initial annotations undergo a second-expert review, with a measured 98% first-pass acceptance (a 2% correction rate) and 99.43% acceptance after revision. A third-tier QA expert then audits a random 10% subset for consistency across spatial, temporal, and semantic boundaries. 3.4. Programmatic Task Generation via Pseudo-Labeling To scale the evaluation of fine-grained spatial reasoning, the Pointing Data Engine converts human-labeled object-localization boxes over vehicle-mounted footage into 2D Spatial Pointing tasks. Both the question and its answer are generated programmatically from KITTI-style labels: the question from a template over relative 6 VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models distances between box coordinates, the correct option deterministically from the same geometry. A VLM is invoked only afterwards, to rewrite the templated question in more natural language; it cannot change which option is correct. Anchoring every pair to human-verified coordinates keeps the answer key tied to expert annotation at scale, and same-class distractors keep the multiple choice difficult. 3.5. Adaptation of Public and Synthetic Tracks To supplement our proprietary real-world data, we selectively adapted highly relevant public and synthetic sources for specific spatial reasoning tasks. For 2D Referring Expressions (RefEx), we integrated images from the RefDrone dataset [29], leveraging its dense, oblique aerial perspectives to test language-grounded object disambiguation. These queries are human-authored; the programmatic rewriting described in Section 3.4 applies only to 2D Spatial Pointing. For SOT, we adapted high-fidelity synthetic logistics sequences from the NVIDIA PhysicalAI-SmartSpaces collection [30], specifically targeting multi-camera warehouse environments. These sequences were reformatted into our VLM-native, single-pass[푥,푦,푤,ℎ]coordinate prediction protocol. Full details regarding sequence filtering, data reformatting, and temporal sampling densities are provided in Appendix C. 3.6. Safely Simulating Severe Anomalies A fundamental challenge in Infrastructure AI is evaluating safety-critical anomalies (e.g., collisions, industrial accidents) that are inherently rare and heavily restricted by privacy regulations. To evaluate these high-stakes scenarios without ethical compromise, approximately 20% of our VQA and Temporal splits utilize high-fidelity synthetic data generated using NVIDIA Omniverse DRIVE Sim [36]. This ensures VANTAGE-Bench covers severe long-tail physical anomalies that are otherwise absent from purely real-world datasets. Benchmark Scope. Unlike prior datasets, which are typically restricted to a single modality or reasoning dimension, VANTAGE-Bench jointly evaluates semantic, spatial, temporal, and spatio-temporal reasoning across both image and video domains, and moves past MCQ-only evaluation to generative captioning and coordinate-based grounding. Appendix A compares its scale and coverage against representative benchmarks. 4. Evaluation Framework and Empirical Results To quantify the Infrastructure AI capability gap, we evaluate 17 VLMs on VANTAGE-Bench across the four pillars defined in Section 3.2. 4.1. Experimental Setup We evaluate 17 models spanning four vendors and 2B to frontier scale. Twelve are open-weight: Qwen3-VL at 2B, 8B, and 32B [37]; Qwen3.5 at 9B and 27B [43]; Cosmos-Reason2 at 2B, 8B, and 32B [42]; Cosmos3 Edge, Nano, and Super [38]; and Gemma-4-E2B [39]. Five are proprietary: GPT-5.6 Sol [40], Gemini 3.6 Flash, Gemini 3.5 Flash-Lite, Gemini 3.1 Pro, and Gemini 3.1 Flash-Lite [41]. Four of the open-weight families provide a scaling series, which we use in Section 4.4. To test out-of-the-box physical reasoning and eliminate fine-tuning biases, all models were evaluated zero-shot. Inference and metric computation (e.g., Story-Oriented Dense Captioning (SODA 푐 ) [34], temporal Intersection-over-Union (IoU), spatial trajectory parsing) were conducted using an extended version of the VLMEvalKit harness [31] calibrated for VANTAGE-Bench tasks. Full details regarding our evaluation frameworks, custom metric scripts, and computational setup are provided in Appendix F. 4.2. Main Results We analyze model performance across the four pillars of operational visual intelligence (Table 1). Performance is uneven across pillars rather than uniformly low. Absolute scores are not directly comparable across tasks, since the metrics differ in what a perfect score demands; Section 4.3 therefore reads each task against an external reference measured the same way. 7 VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models SemanticSpatialTemporalSp.-Temp. ModelParams VQA EV Point. RefEx Obj Loc. Temp Loc. DVCSOT Overall (Acc.)(M-F1)(Acc.)(mIoU)(F1@.5)(mIoU)(SODA 푐 )(AUC) Open-weight models Qwen3-VL2B63.85 44.73 53.13 65.12 47.4235.2119.90 30.3841.86 Qwen3-VL8B66.44 59.39 68.56 73.29 59.9344.3229.64 33.1250.07 Qwen3-VL32B71.30 60.04 75.62 72.39 72.6746.8029.40 44.1955.38 Qwen3.59B71.63 46.78 71.74 77.16 76.8742.5327.88 23.5248.30 Qwen3.527B67.95 55.34 75.32 76.35 85.4636.8826.97 24.4049.25 Cosmos-Reason22B64.69 55.27 59.70 56.33 73.5638.9528.52 26.8545.94 Cosmos-Reason28B67.95 64.09 68.60 70.26 83.8847.3032.50 37.6954.46 Cosmos-Reason232B70.29 73.58 74.03 50.04 4.35 † 50.4730.04 33.12 47.03 † Cosmos3-Edge4B62.68 63.36 67.76 69.27 79.0239.5116.64 24.6546.94 Cosmos3-Nano16B68.95 68.88 74.83 75.57 74.1148.0431.42 59.2160.67 Cosmos3-Super64B69.46 71.28 72.94 76.23 86.9751.9029.54 64.6663.62 Gemma-4-E2B2B46.19 27.65 31.74 5.52 45.409.029.1411.5821.28 Proprietary models Gemini 3.1 Flash-Lite —68.03 63.78 63.68 52.70 71.7437.7232.54 46.6852.61 Gemini 3.1 Pro—71.46 68.57 72.04 67.17 77.2145.6935.55 67.8862.66 Gemini 3.5 Flash-Lite —65.61 70.41 43.08 74.46 65.7324.0032.29 55.7753.25 Gemini 3.6 Flash—76.82 82.00 79.20 75.78 81.5751.5036.15 75.99 69.52 GPT-5.6 Sol—78.08 76.14 81.90 69.44 69.3055.71 37.28 75.0268.04 Table 1: VANTAGE-Bench main results. Zero-shot performance of 17 models across the eight tasks, grouped by reasoning pillar. All metrics scaled 0–100; Overall is the macro-average of the four pillar scores, and bold denotes the best score per column. Pillar scores are in Table 11 and item-level 95% bootstrap intervals in Table 13. † Output-format regression on Object Localization for this checkpoint; reported unmodified, and it propagates into the aggregates that contain it. Semantic Understanding. Semantic understanding is the strongest pillar at the top of the table, though spatial grounding has the higher aggregate across the full model set (Table 11). Gemini 3.6 Flash leads Event Verification at 82.00 Macro F1 and GPT-5.6 Sol leads Video Question Answering at 78.08, with Cosmos- Reason2-32B and Cosmos3-Super leading the open-weight field on Event Verification at 73.58 and 71.28. Event Verification separates models far more sharply than VQA, spanning 27.65 to 82.00 against 46.19 to 78.08. Smaller models are conservative under uncertainty rather than uniformly weak: Qwen3-VL-8B rejects plausible negatives at a Specificity of 74.58 while reaching only 50.96 in True Event Recall, so it fails to recognize when physical interactions such as collisions or safety violations actually occur. We term this causal blindness and break it down in Appendix D; metric definitions are in Appendix B. Spatial Understanding. The spatial pillar is where open-weight models are most competitive. GPT-5.6 Sol leads 2D Spatial Pointing at 81.90, but open-weight models lead the other two spatial tasks outright. Cosmos3- Super leads 2D Object Localization at 86.97, ahead of the best proprietary model at 81.57, and Qwen3.5-9B leads 2D Referring Expressions at 77.16, with Qwen3.5-27B and Cosmos3-Super close behind at 76.35 and 76.23 against 75.78 for the best proprietary model. Single-frame semantic-to-spatial mapping is therefore maturing in open-weight architectures. Temporal Understanding. The temporal pillar is the weakest in absolute terms across the entire suite: the best scores on Temporal Localization and Dense Video Captioning, 55.71 mIoU and 37.28 SODA 푐 , sit far below the leader on every other task (Section 4.4). This is not confined to open-weight models. On Temporal Localization the best proprietary and best open-weight scores are 55.71 and 51.90, a margin of 3.81 that is the narrowest frontier advantage on any task the frontier leads; on referring expressions and object localization the margin runs the other way, with open-weight models ahead. In Dense Video Captioning, models generate accurate semantic descriptions but fail to localize them in time, which depresses the jointSODA 푐 metric; we 8 VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models separate the two components in Appendix D. Spatio-Temporal Understanding. Single Object Tracking shows the widest separation between proprietary and open-weight models of any task. Gemini 3.6 Flash reaches 75.99 Success AUC and GPT-5.6 Sol 75.02, against 64.66 for the strongest open-weight model, Cosmos3-Super. The spread within the open-weight group is itself large, from 64.66 down to 11.58, and tracks training focus more closely than parameter count. Appendix D.3 places these numbers against specialist trackers and shows that the separation grows with the tracking horizon. 4.3. The Operational Deployment Gap (Δ) To test whether the Infrastructure AI Gap is real, we compare model performance on VANTAGE-Bench against the same models’ reported baselines on standard, consumer-focused datasets (Table 2). We include every model whose developer publishes reference scores: Qwen3-VL reports all five reference benchmarks at 2B, 8B, and 32B, and the Cosmos3 report publishes RefCOCO. Frontier proprietary releases and newer open-weight families increasingly report only high-level multiple-choice cognitive benchmarks and no longer publish zero-shot baselines for granular spatial and temporal grounding, which is itself symptomatic of the evaluation gap we describe. The deficit is localized, not uniform. The deltas run in both directions, and that is what makes the pattern informative. Event verification, referring expressions, and temporal localization drop roughly 9 to 24 points across every family and scale we could match. Video question answering sits within 5.3 points, and pointing is flat at the two smaller scales (−0.7and−0.5) before turning into a surplus of+8.3at 32B. The same models therefore lose ground on some capabilities and gain it on others within one benchmark, which locates the shortfall in specific capabilities rather than in overall difficulty. Qwen3-VL TaskReference benchmark2B8B32B VQAVideoMME [1]+2.0 −5.0 −5.3 Event Verification * MLVU [32]−23.6 −18.7 −22.1 2D Spatial PointingBLINK [4]−0.7 −0.5+8.3 2D Referring Expressions RefCOCO [13]−20.5 −15.8 −19.5 Temporal LocalizationCharades-STA [11] −19.3 −11.7 −14.4 Cosmos3 on RefCOCO [13] (Edge / Nano / Super) 2D Referring Expressions RefCOCO [13]−10.8 / −8.7 / −13.3 Table 2: The Infrastructure AI Gap (∆= VANTAGE−reference), for every model whose developer publishes reference scores. Qwen3-VL reports all five reference benchmarks at each scale; the Cosmos3 report publishes RefCOCO only. * Loosest mapping: Macro F1 against accuracy. Where the shortfall sits. Three observations locate it. Models sharing one base diverge sharply by training focus (Section 4.4), so the pattern moves with what a model was trained on rather than with the suite as a whole. Proprietary access is no shortcut either: two proprietary models fall below four open-weight models on the overall pillar average (Table 1). Most directly, the split appears within a single task. On dense video captioning, BERTScore-F1 spans 54.2 to 63.9 across models while temporal IoU-F1 spans 33.2 to 39.4 on the same items and the same annotations (Table 6): models describe the events correctly and misplace them in time. The failure is in temporal grounding specifically, not in video understanding broadly. A prompt- and metric-matched control. Our temporal localization evaluation uses the prompt template the developer published for its own Charades-STA evaluation, so task, metric, and prompt are held constant. 9 VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models Qwen3-VL drops from 56.0 to 44.3 at 8B and from 61.2 to 46.8 at 32B, and the strongest frontier model reaches only 55.7, below what mid-tier open models score on Charades-STA. A capability with no external baseline. Model releases claim tracking in their demonstrations, as the Qwen3-VL release does, but none reports a quantitative tracking benchmark, and no prior VLM tracking evaluation targets fixed-camera infrastructure video. Frontier models have therefore been deployed into physical environments without a published baseline for continuous spatio-temporal attention. VANTAGE-Bench supplies one, anchored against specialist trackers in Appendix D.3. 4.4. Returns to Scale and the Temporal Lag A prevailing hypothesis is that grounding deficits will resolve through parameter scaling. The four open-weight scaling series let us test this directly (Table 3). Open-weight scaling seriesProprietary TaskQwen3-VLQwen3.5Cosmos-Reason2Cosmos3GPT-5.6 Gemini 3.6 (2B→32B)(9B→27B)(2B→32B)(Edge→Super)SolFlash VQA63.9→71.3 (+7.4) 71.6→68.0 (−3.6) 64.7→70.3 (+5.6) 62.7→69.5 (+6.8) 78.176.8 Event Verif. 44.7→60.0 (+15.3) 46.8→55.3 (+8.5) 55.3→73.6 (+18.3) 63.4→71.3 (+7.9) 76.182.0 Obj. Loc. 47.4→72.7 (+25.3) 76.9→85.5 (+8.6) 73.6→4.4 † (−69.2) 79.0→87.0 (+8.0) 69.381.6 Ref. Exp.65.1→72.4 (+7.3) 77.2→76.4 (−0.8) 56.3→50.0 (−6.3) 69.3→76.2 (+6.9) 69.475.8 Pointing53.1→75.6 (+22.5) 71.7→75.3 (+3.6) 59.7→74.0 (+14.3) 67.8→72.9 (+5.1) 81.979.2 Temp. Loc. 35.2→46.8 (+11.6) 42.5→36.9 (−5.6) 39.0→50.5 (+11.5) 39.5→51.9 (+12.4) 55.751.5 DVC19.9→29.4 (+9.5) 27.9→27.0 (−0.9) 28.5→30.0 (+1.5) 16.6→29.5 (+12.9) 37.336.2 SOT30.4→44.2 (+13.8) 23.5→24.4 (+0.9) 26.9→33.1 (+6.2) 24.7→64.7 (+40.0) 75.076.0 Table 3: Returns to scale, smallest to largest member of each open-weight family, with the change in parentheses, alongside the two newest proprietary models. Row-best in bold; markers as in Table 1. Returns to scale are task-specific, not pillar-specific. Scaling helps substantially on some tasks and not at all on others, and the pattern does not align with the four pillars. Temporal Localization improves by an average of 7.5 points from the smallest to the largest family member, so temporal grounding does respond to scale. Object Localization improves with scale in three of the four families; Cosmos-Reason2-32B is the exception, due to an output-format regression rather than a perception result. Qwen3.5 regresses on four of eight tasks despite tripling in size. The persistent finding is an absolute temporal lag. Across all 17 models, no system exceeds 55.71 mIoU on Temporal Localization or 37.28 SODA 푐 on Dense Video Captioning. The same metric on consumer footage runs higher: Qwen3-VL-32B scores 61.2 mIoU on Charades-STA against 46.8 here. Temporal grounding on operational footage is therefore unsolved at present scale, though nothing in our evidence bounds what further scaling might achieve. Physical-AI training data helps some tasks far more than others. Qwen3-VL-8B, Cosmos-Reason2-8B, and Cosmos3-Nano share a common base but carry progressively more physical-AI training data. Across that sequence tracking rises 26.1 points, object localization 14.2, and event verification 9.5, while temporal localization moves 3.7 and dense captioning 1.8. All three are evaluated zero-shot. The Cosmos models add physical-AI training over the shared base [42,38], though the exact composition of that mixture is not public. What the sequence shows is that such training transfers to this suite very unevenly, transforming tracking while barely moving the temporal tasks. Our working hypothesis is a training-mixture asymmetry: spatial-grounding supervision in the form of bounding-box datasets is abundant, while densely timestamped video is scarce. A targeted fine-tuning probe would distinguish a capability gap from an elicitation gap, and we leave this test to future work. 10 VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models 5. Qualitative Analysis: Failure Modes of Infrastructure AI While Section 4 quantifies the Infrastructure AI gap, we analyze key failure modes underlying this degradation. Additional quantitative breakdowns are provided in Appendix D. Semantic Illusion (Fluency vs. Grounding). Models exhibit strong linguistic fluency but fail to anchor predictions in space and time: they recognise what happens without resolving when it occurs (Section 4.3). The effect extends to Event Verification, where smaller models exhibit causal blindness: they correctly reject negatives but fail to detect true events, defaulting to conservative predictions in dense scenes. Spatial vs. Spatio-Temporal Breakdown. Strong single-frame spatial reasoning does not translate to temporal persistence. Weaker models often initialize correctly but fail to maintain object identity as the horizon extends. Two distinct failure modes appear: predictions that stop updating (tracking freeze) and predictions that keep updating but drift off target. Appendix D.3 separates these using freeze rate read jointly with Success AUC. Domain-Specific Vulnerabilities. Holding the model and the annotation pipeline fixed, the two pillars degrade in different domains. For Cosmos3-Super, VQA accuracy is comparable in Warehouse and Transportation, at 75.73 and 71.51, and drops sharply only in Smart Spaces, at 55.38, where human activity is least structured and intent is hardest to read from the scene alone. Dense video captioning follows a different order: strongest in Warehouse at 40.61, lower in Smart Spaces at 30.38, and collapsing in Transportation at 8.35. No single domain is uniformly hard. Each pillar has its own worst case, and a model’s aggregate score can therefore conceal a domain in which one capability has failed almost completely (Appendix D). Where the Frontier Leads. Proprietary access confers no across-the-board advantage; the two newest frontier releases do lead most tasks, but by a fairly uniform margin with tracking the one large outlier. The distinctive frontier capability is temporal persistence rather than spatial grounding, which we quantify in Appendix D.6. 6. Discussion and Limitations VANTAGE-Bench establishes a baseline for Infrastructure AI evaluation. Camera geometry. Our claims are scoped to the camera configurations actually sampled: elevated fixed cameras at 8 to 20 feet and 30 to 60 degrees from vertical for the video tasks and 2D Object Localization, oblique aerial views for 2D Referring Expressions, and vehicle-mounted footage for 2D Spatial Pointing. Real-world fixed-camera deployments vary more widely than this. Eye-level retail and low-angle industrial viewpoints are not covered; the constraint is data rights, since each deployment type requires separate licensing agreements. Geographic concentration. Real-world footage comes primarily from two U.S. municipalities, supplemented by publicly sourced and licensed European footage (Section 3). Coverage is therefore weighted toward North America, and broadening it is future work. Annotation density. Counts are uneven across the three annotation regimes because the tasks themselves differ in density. Infrastructure scenes contain many simultaneous objects such as people and vehicles, so a single detection image carries 43.6 boxes on average. The detection task therefore reflects deep labeling by the nature of its definition rather than broad sampling. We report the three regimes separately rather than as one aggregate: video-task annotations, image-grounding annotations, and dense detection boxes. Quality assurance. Our acceptance rates come from a sequential second-expert review pipeline, in which a reviewer corrects a prior annotation rather than labeling the item independently. They measure how often expert work passes review, not agreement between independent annotators. Programmatic question generation. For 2D Spatial Pointing, the correct answer is fixed by human-labeled geometry and cannot be altered by the rewriting stage. The rewritten question text is nonetheless not constrained to preserve the uniqueness of its spatial cue in dense scenes, and we do not currently verify that it does. A targeted uniqueness audit of the rewritten questions is planned. 11 VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models Tracking is synthetic only. The Single Object Tracking track is built from synthetic warehouse sequences, so its conclusions are limited to that distribution and do not yet separate tracking failure from synthetic-domain mismatch. The specialist-tracker and horizon analyses in Appendix D.3 establish that the task is well posed, but they do not substitute for real-data validation. Real tracking sequences are planned for a future release. Model coverage and reported uncertainty. Our evaluation is strictly zero-shot; domain-specific fine-tuning is left to future work. Extreme weather and lighting conditions are not systematically covered. Inference settings were tuned per model, following each developer’s published guidance where it exists and our own prompt- sensitivity checks otherwise; Appendix F.2 reports the configuration each model was finally evaluated under rather than the alternatives we tried. The intervals we report (Appendix G.1) therefore cover item sampling only, and do not capture how far a score would move under a different prompt or decoding configuration. Planned extensions. Coverage is uneven across tasks and domains: 2D Spatial Pointing and 2D Referring Expressions are currently drawn from a single domain, and the tracking track is synthetic. We are extending every task with additional real-world footage spanning all three deployment domains rather than a subset, and we are broadening the camera configurations sampled within each. We also plan a human performance baseline on a representative subset of each task, which would calibrate task difficulty independently of annotation quality. Beyond the current suite, we plan to add task tracks that operational deployments require and this release does not cover, including multi-object tracking and 3D grounding. Ethical considerations. The automation of fixed-camera monitoring carries privacy implications. All raw assets underwent a de-identification pipeline combining automated obfuscation with human verification, documented in Appendix E. VANTAGE-Bench is released under a license prohibiting biometric identification and demographic profiling. We emphasize that the benchmark evaluates physical and causal reasoning rather than identity recognition, and we advocate for human-in-the-loop oversight in downstream operational deployments. 7. Conclusion We introduced VANTAGE-Bench, an evaluation suite for VLMs in open-loop Infrastructure AI settings. Moving beyond the priors of internet video and single-format MCQ evaluation, we built an 8-task taxonomy spanning semantic, spatial, temporal, and spatio-temporal reasoning over 3,346 expert-annotated media assets, and evaluated 17 models zero-shot. The resulting picture is uneven rather than uniformly poor. Open-weight models lead 2D Object Localization outright, and they edge ahead on 2D Referring Expressions by a margin comparable to its confidence interval. The deficit against consumer-centric benchmarks concentrates in event verification, referring expressions, and temporal grounding, while pointing is flat at smaller scales and turns into a surplus at 32B. Returns to scale are task-specific rather than pillar-specific. What persists across all 17 models is an absolute lag on the two temporal tasks, which no system has yet closed, and a widening separation on tracking as the horizon extends. We hope VANTAGE-Bench provides a useful framework for evaluating physical reasoning in operational settings, and that its open leaderboard makes progress on these specific deficits measurable. Acknowledgements We thank Paris Zhang, Yilin Zhao and Zheng Liu for developing portions of several task tracks, including Event Verification; Chintan Shah, Sumeeth Nagaraja and Ratnesh Kumar for data sourcing, discussions and evaluation feedback; and Yao Xu and the NVIDIA Data Factory team for video sourcing and human annotation. Tsung-Yi Lin, Ke Ding and Ming-Yu Liu provided evaluation infrastructure and model access. We thank Professor Carrie Russell of Clemson University for collaboration throughout, and Ahmet Dokmeci for the leaderboard evaluation pipeline. This research used in part resources on the Palmetto 2 cluster at Clemson University under National Science Foundation awards MRI 1228312, I NEW 1405767, MRI 1725573, and MRI 2018069. The views expressed in this article do not necessarily represent the views of NSF or the United States government. 12 VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models A. Data Statistics and Benchmark Comparison This appendix places VANTAGE-Bench alongside the benchmarks it is most often compared against and records how its annotations are distributed. Table 4 reports scale, modality, and pillar coverage for each. BenchmarkModality# Media # Annot. PillarsAnnotation Source VideoMMEVideo9002,700 Semantic Human BLINKImage3,6831,906 SpatialHuman, Existing RefCOCO_avgImage3,98230,969 SpatialHuman, Existing ODinW13 [33]Image4,60810,966 SpatialHuman, Existing CharadesSTAVideo1,3343,720 Temporal Human, PL ActivityNet Cap. Video5,04417,750 Temporal Human VANTAGE-Bench Image + Video3,346 35,027 † All four Human, Simulated, PL, Existing Table 4: Cross-benchmark comparison. All statistics reflect available evaluation test sets. Pillars is the reasoning taxonomy each benchmark covers. † Shown as a single total for comparability with the other rows; we report our annotations by regime elsewhere. Each of the three annotation regimes listed in Section 3 is comparable to or larger than its established counterpart in Table 4: the video tasks carry more annotations than VideoMME (3,342 against 2,700), though over slightly fewer videos (854 against 900), and image grounding exceeds BLINK’s test set (4,281 against 1,906). Tracking is the one smaller track. Detection dominates the aggregate because infrastructure scenes are dense, at 43.6 boxes per image against ODinW-13’s 2.4, which reflects scene density rather than broad sampling. B. Formal Task Formulations and Evaluation Metrics In this section, we provide the rigorous definitions for the mappings and evaluation protocols used in VANTAGE- Bench. B.1. Notation and Definitions Each task is stated in Table 5 as a function from inputs to a structured output, with its primary metric. We define the following input and output spaces to standardize the task formulations: • Inputs: ℐ (Image);풱 (Video);풬 (Query); 퐵 푡=0 (Initial Box). •Outputs:휏(Temporal segment[푡 푠푡푎푟푡 ,푡 푒푛푑 ]);퐶(Caption);푎(Selection from candidates);퐵(Set of coordinates). B.2. Metric Clarifications •SODA 푐 : Evaluates narrative quality by matching predicted event sequences to ground truth based on temporal overlap and BERTScore-based [35] linguistic similarity. • Success AUC: For SOT, the Success Plot counts frames where predicted IoU exceeds threshold휎, calculated by varying 휎 ∈ [0, 1]. • Macro F1: Penalizes models that exploit majority-class bias in binary Event Verification. C. Pipeline Details and Data Adaptation C.1. Data Adaptation for SOT and RefEx 2D Referring Expressions: We utilized RefDrone because its elevated perspective mimics infrastructure cameras. We ported prompts requiring dense disambiguation between identically colored vehicles based on 13 VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models PillarTaskMathematical FormulationPrimary Metric Additional Semantic EV푓 (풱,풬 ℎ푦푝 )→0, 1Macro F1Sens., Spec. VQA푓 (풱,풬,퐴,퐵,퐶,퐷)→ 푎∈퐴,퐵,퐶,퐷Accuracy— Spatial RefEx푓 (ℐ,풬)→ [푥,푦,푤,ℎ]mIoUPrecision@0.5 Pointing 푓 (ℐ,풬,퐴,퐵,퐶,퐷)→ 푎∈퐴,퐵,퐶,퐷Accuracy— Obj Loc. 푓 (ℐ,풬 푐 )→퐵 푘 푀 푘=1 F1@0.5COCO mAP Temporal Temp Loc. 푓 (풱,풬)→ [푡 푠푡푎푟푡 ,푡 푒푛푑 ]mIoURecall@0.5 DVC푓 (풱)→(휏 푖 ,퐶 푖 ) 푁 푖=1 SODA 푐 BERTScore-F1, IoU-F1 Spatio-Temp. SOT푓 (풱,퐵 푡=0 )→퐵 푡 푇 푡=1 Success AUCMean IoU Table 5: Formal task definitions for VANTAGE-Bench. The primary metric is the one reported in Table 1 and on the public leaderboard. relative spatial relationships. Single Object Tracking: Adapted from the MTMC 2025 subset of PhysicalAI-SmartSpaces. We filtered 17 warehouse scenes featuring workers and robotic profiles (e.g., Nova Carter). Sequences include an initial “visual anchor” followed by a sequence of frames sampled at densities of 8, 16, and 32 frames. D. Extended Analysis: Deconstructing the Infrastructure AI Gap This section provides detailed quantitative and qualitative analysis of the failure modes summarized in Section 5. While Section 4 establishes the magnitude of the Infrastructure AI gap, the analyses below isolate the underlying mechanisms responsible for model failure. D.1. Semantic Illusion: Fluency vs. Grounding We observe a systemic disconnect between linguistic fluency and physical grounding, which we term the Semantic Illusion. Dense Video Captioning. Models generate semantically coherent descriptions (high BERTScore-F1) but fail to temporally localize events, resulting in low joint performance (SODA 푐 ). This indicates that models capture what happens but fail to resolve when it occurs, leading to temporal smearing across events. Table 6 separates the two components. ModelSemantic Fluency Discrete Loc. Joint Metric (BERTScore-F1)(IoU-F1)(SODA 푐 ) Gemini 3.1 Pro63.8639.4435.55 Cosmos-Reason2-8B58.0836.7032.50 Qwen3-VL-8B58.6633.2429.64 Cosmos-Reason2-32B54.7433.9530.04 Cosmos3-Super54.2433.3929.54 Table 6: Dense Video Captioning decomposed into its semantic and temporal components, with the joint metric for reference. Scaled 0–100. Event Verification: Causal Blindness. The same disconnect appears in Event Verification, though the direction of the imbalance depends on the model rather than on scale alone (Table 7). Qwen3-VL-8B fails to detect true events while maintaining strong specificity, at 50.96 against 74.58, a bias toward conservative predictions 14 VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models under uncertainty rather than an inability to reason about the scene. Cosmos-Reason2-8B shows the opposite profile, at 71.15 sensitivity against 57.63 specificity, accepting events that did not occur. Macro F1 conceals both, which is why we report the two components separately. ModelMacro F1 True Event Recall True Negative Recall (Sensitivity)(Specificity) Gemini 3.6 Flash82.0087.5076.27 GPT-5.6 Sol76.1477.8876.27 Cosmos-Reason2-32B73.5883.6562.71 Cosmos3-Super71.2872.1272.88 Gemini 3.1 Pro68.5771.1567.80 Cosmos-Reason2-8B64.0971.1557.63 Qwen3-VL-8B59.3950.9674.58 Table 7: Event Verification causal breakdown (푛 = 163; 104 positive, 59 negative). Sensitivity and specificity are reported separately, since Macro F1 alone conceals the asymmetry. D.2. Spatial vs. Spatio-Temporal Breakdown We identify a key architectural limitation: static spatial grounding does not imply temporal persistence. Table 8 contrasts the two for six representative models. The dissociation is sharpest in the open-weight group. Qwen3.5-27B reaches 85.46 on 2D Object Localization and 76.35 on 2D Referring Expressions, among the strongest spatial results in the suite, yet scores 24.40 on tracking. Cosmos3-Super posts comparable spatial numbers and reaches 64.66. Spatial grounding is therefore not sufficient for temporal persistence, and the two are not acquired together. Model SpatialSpatio-Temporal Pointing RefEx Object Loc.SOT (Accuracy) (mIoU) (F1@0.5)(AUC) GPT-5.6 Sol81.9069.4469.3075.02 Gemini 3.6 Flash79.2075.7881.5775.99 Cosmos3-Super72.9476.2386.9764.66 Qwen3.5-27B75.3276.3585.4624.40 Cosmos-Reason2-8B68.6070.2683.8837.69 Qwen3-VL-8B68.5673.2959.9333.12 Table 8: Single-frame spatial scores against Single Object Tracking, for six representative models. Metrics scaled 0–100. D.3. Tracking Anchors and Horizon Analysis Every system is reported at all three horizons in Table 9. Protocol. All evaluations are performed on the same underlying object tracklets (200 sequences); the 8-frame, 16-frame, and 32-frame benchmarks differ only in the temporal sampling and clip length. The 8f and 16f datasets share an identical frame-level prefix (stride 15), with the 16f clips extending the 8f sequences by eight additional sampled frames. The 32f benchmark is independently resampled from the same tracklets using a stride of 14, resulting in a longer temporal window rather than a strict continuation of the 16f clips. At the default sampling rate (30 FPS), the three settings correspond to approximately 3.5 s (8f), 7.5 s (16f), 15 VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models and 14.5 s (32f) of video. We additionally report a freeze rate: the fraction of predicted boxes that are nearly identical (IoU≥ 0.95) to the box predicted on the previous frame. Baselines. We evaluate four specialist trackers, a static-box floor, and a random floor on the same 200 tracklets and the exact frames the VLMs see, using the identical Success-AUC scorer. We did not add classical trackers (e.g., KCF, CSRT): on standard benchmarks they are bounded above by the modern trackers and below by these floors, so they would not change where the VLM scores sit. Success AUC↑Freeze rate↓ Model8f16f 32f8f16f 32fΔ AUC Floors Random6.55 6.27 6.12 0.00 0.00 0.00 −0.43 Static box24.64 18.28 13.75 99.08 99.35 99.20 −10.89 Specialist trackers SAM 379.92 82.25 80.47 17.58 30.66 29.83+0.55 SAMURAI80.91 81.16 79.85 18.00 30.23 30.01 −1.06 SUTrack79.12 81.30 78.50 16.33 28.43 27.77 −0.62 MCITrack77.74 79.44 76.61 16.67 27.53 27.29 −1.13 Vision-language models Gemini 3.6 Flash75.99 70.77 66.25 23.75 36.46 36.66 −9.74 GPT-5.6 Sol75.02 74.75 70.91 22.75 37.32 34.96 −4.11 Cosmos3-Super64.66 61.32 45.32 32.08 48.74 58.99 −19.34 Cosmos-Reason2-8B 37.69 30.64 17.98 77.67 84.57 92.96 −19.71 Qwen3-VL-8B33.12 23.27 15.67 38.42 47.20 48.50 −17.45 Table 9: Tracking anchors and horizon analysis. Success AUC and freeze rate at three horizons, scaled 0–100. The 8-frame column is the Success AUC reported on the public leaderboard. Reading the anchors. The specialist trackers reach approximately 80 Success AUC and hold steady as the horizon grows, establishing the level a purpose-built system reaches on these annotations. Frontier VLMs come within roughly 5 points of them at 8 frames but degrade with horizon (GPT-5.6 Sol−4.1, Gemini 3.6 Flash−9.7from 8f to 32f). Open-weight behavior is heterogeneous. Cosmos3-Super remains well above the static floor at 32 frames (45.3 versus 13.8 AUC), although it drops 19.3 points as the horizon increases. Cosmos-Reason2-8B and Qwen3-VL-8B finish much closer to the static floor, at 18.0 and 15.7 versus 13.8. Long- horizon degradation is therefore broadly observed, while convergence toward the static baseline is confined to the weaker open-weight models. The tier ordering—specialists, frontier VLMs, strongest open-weight, then 8B-class—is unchanged across all three horizons. Reading the freeze rate. Freeze rate must be interpreted jointly with AUC, since a nearly unchanged box can be correct when the target is stationary; freeze rate alone is not an error rate. The specialist trackers provide an empirical reference that itself rises with horizon: 16–18% at 8 frames against 27–31% at 16 and 32 frames. The static-box floor, which by construction never updates, sits at 99% throughout, so the metric is bracketed at both ends. Cosmos-Reason2-8B has a much higher freeze rate of 93% together with only 18.0 AUC at 32 frames, consistent with a failure to update its prediction. Qwen3-VL-8B has a lower freeze rate of 49% but only 15.7 AUC, consistent with continued box updates that drift off target. We therefore use freeze rate to distinguish these two failure modes, without interpreting the specialist rate as the true fraction of stationary targets. The track covers 200 objects across 102 videos, in line with established tracking test sets (GOT-10k validation: 180 videos; LaSOT: 280). What separates VLMs from purpose-built trackers on this track is persistence over 16 VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models time rather than box regression: at short horizons the frontier models are close to the specialists, and the gap opens as the horizon extends. D.4. Domain-Specific Vulnerabilities Model performance varies significantly across operational domains, revealing sensitivity to environmental priors (Table 10). DomainVQA (Acc %) DVC (SODA 푐 ) Warehouse75.7340.61 Smart Spaces55.3830.38 Transportation71.518.35 Overall69.4629.54 Table 10: Cosmos3-Super by operational domain, holding model and annotation pipeline fixed. Unstructured Intent. Semantic reasoning is weakest in Smart Spaces, at 55.38 VQA accuracy against 75.73 in Warehouse, which we attribute to ambiguous human behaviour and the absence of structured, task-driven interactions. Egocentric Bias is Temporal, not Semantic. Transportation leaves VQA accuracy nearly intact at 71.51 but reduces dense video captioning to 8.35 from 40.61 in Warehouse. The model still recognises what is in the scene; it loses the ability to say when events begin and end. We attribute this to the mismatch between ego-centric training footage and fixed-camera evaluation perspectives, which is most disruptive to the continuous-motion cues temporal grounding depends on. The domain effect is therefore larger than previously reported, but it falls on the temporal pillar. D.5. Pillar Scores The main results table reports the eight task scores and the Overall column, but not the intermediate pillar scores from which Overall is computed. Table 11 supplies them. Each pillar score is the unweighted mean of its constituent task scores, and Overall is the unweighted mean of the four pillar scores, so a pillar containing one task carries the same weight as a pillar containing three. This equal weighting is deliberate: it prevents the spatial pillar, which has the most tasks, from dominating the aggregate. It does mean the Spatio-Temporal pillar is identical to the Single Object Tracking column of Table 1, since tracking is its only task, and that tracking therefore carries a quarter of the Overall score. Readers who prefer a task-weighted aggregate can compute one from Table 1; it would raise the relative weight of the spatial pillar, where open-weight models are strongest. D.6. Where the Frontier Advantage Actually Lies The frontier advantage is concentrated in tracking rather than spread across spatial grounding, and this section sets out the evidence for that. There is no across-the-board proprietary advantage. Open-weight models lead 2D Object Localization outright (86.97 against 81.57) and lead 2D Referring Expressions by 1.38 points (77.16 against 75.78), and two proprietary models fall below four open-weight models on the overall pillar average (Table 1). Proprietary access alone does not confer an edge. The advantage belongs to the two newest releases, and it is mostly uniform. GPT-5.6 Sol and Gemini 3.6 Flash lead the best open-weight model on six of eight tasks, by a fairly uniform margin of 3.8 to 8.4 points across spatial, semantic, and temporal tasks alike. Three tasks depart from that band. Two are exceptions in the open-weight direction: Cosmos3-Super leads object localization by 5.4 points, and Qwen3.5-9B leads 17 VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models ModelParams Semantic Spatial Temporal Sp.-Temp. Overall (2 tasks)(3 tasks)(2 tasks)(1 task) Open-weight models Qwen3-VL2B54.2955.2227.5530.3841.86 Qwen3-VL8B62.9167.2636.9833.1250.07 Qwen3-VL32B65.6773.5638.1044.1955.38 Qwen3.59B59.2075.2635.2023.5248.30 Qwen3.527B61.6579.0431.9324.4049.25 Cosmos-Reason22B59.9863.2033.7326.8545.94 Cosmos-Reason28B66.0274.2539.9037.6954.46 Cosmos-Reason232B71.9442.81 † 40.2533.1247.03 † Cosmos3-Edge4B63.0272.0228.0724.6546.94 Cosmos3-Nano16B68.9174.8439.7359.2160.67 Cosmos3-Super64B70.3778.7140.7264.6663.62 Gemma-4-E2B2B36.9227.559.0811.5821.28 Proprietary models Gemini 3.1 Flash-Lite —65.9162.7135.1346.6852.61 Gemini 3.1 Pro—70.0172.1440.6267.8862.66 Gemini 3.5 Flash-Lite —68.0161.0928.1455.7753.25 Gemini 3.6 Flash—79.4178.8543.8375.9969.52 GPT-5.6 Sol—77.1173.5546.5075.0268.04 Table 11: Pillar scores and their aggregate. Each pillar is the unweighted mean of its tasks; Overall is the unweighted mean of the four pillars. The Spatio-Temporal pillar consists of Single Object Tracking alone. All values scaled 0–100; bold denotes the best score per column. Markers as in Table 1. referring expressions by 1.4. Single Object Tracking is the exception in the other direction, where the frontier lead is 11.3 points, roughly double the typical margin. The distinctive frontier capability is temporal persistence. The tracking gap is where the frontier separates, not spatial grounding, where open-weight models in fact lead two of the three tasks, and not temporal localization, where the edge is smallest at 3.8 points because no model of either kind exceeds 55.71 mIoU (Section 4.4). The horizon analysis in Appendix D.3 is consistent with this reading: frontier models degrade least as the tracking horizon extends, while weaker open-weight models converge toward the static-box floor. E. De-identification Pipeline What counts as PII. We treat as personally identifiable any information that can identify or locate a person: names, recognizable faces, license plates, addresses, and GPS or location data. Detection and obfuscation. Faces, license plates, and video locations were obfuscated using a proprietary detector. Detector output was then verified and corrected by human annotators, so that the released media reflect a human-in-the-loop pass rather than automated obfuscation alone. Licensing safeguards. Beyond obfuscation, the NVIDIA Evaluation Data License prohibits biometric identifi- cation and demographic profiling, restricting the released data to evaluation and benchmarking use. Consent. 70% of footage was provided by vendors who obtained explicit informed consent from recorded individuals for research and redistribution. A smaller portion was captured in limited spaces with posted notice of camera operation, where individuals incidentally captured consented to fixed-camera monitoring as a condition of site access. 18 VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models F. Reproducibility and Experimental Configuration F.1. Compute Resources and Frameworks Evaluations for open-weight models (Cosmos-Reason2, Qwen3-VL, Qwen3.5, Cosmos3, Gemma-4) were con- ducted on the Palmetto 2 cluster at Clemson University, using 7 nodes of 8×NVIDIA H100 (80GB) GPUs, as well as additional cloud compute instances. Proprietary models (GPT-5.6 Sol and the Gemini family) were accessed via public APIs. We build our evaluation harness on top of VLMEvalKit [31], extending it with custom metric scripts for automated SODA 푐 computation, temporal IoU evaluation, and spatial trajectory parsing. F.2. Task-Specific Inference Configurations All evaluations used greedy decoding (temperature = 0) with chain-of-thought disabled (enable_thinking = False). The sampling and pixel budgets used to manage visual granularity and token limits are given in Table 12. FPS ranges reflect model-specific sampling rates. PillarTaskFPSMax Frames Max Total Pixels SemanticEV / VQA1.0 – 4.025616.7× 10 6 SpatialPointing / RefEx—11.5× 10 6 TemporalTemporal Localization 2.0 – 10.025616.7× 10 6 TemporalDVC1.0 – 4.01288.4× 10 6 Spatio-Temp. SOT—8 / 16 / 320.9× 10 6 Spatial2D Object Localization—11.5× 10 6 Table 12: VANTAGE-Bench Task Inference Parameters. F.3. Code, Data Access, and Release Strategy To support reproducibility, we provide the VANTAGE-Bench dataset at huggingface.co/datasets/nvidia/PhysicalAI- VANTAGE-Bench the evaluation code at github.com/Clemson-Capstone/VANTAGE-Bench, and the public leader- board at huggingface.co/spaces/clemson-computing/VANTAGE-Bench-Leaderboard. Project information is collected at vantage-bench.org. To preserve long-term integrity against training-data contamination, we adopt a held-out evaluation protocol. We publicly release (i) all input media, (i) all prompts and questions, and (i) the evaluation harness. Ground-truth annotations are held-out to be used by the official leaderboard server. The leaderboard is live and open for submissions, which are uploaded as JSON in LLaVA format. G. Reported Uncertainty We report two kinds of variability, which answer different questions: how precisely a score is estimated from the items in the test set, and how reproducible it is across runs. G.1. Item-Level Confidence Intervals Because all evaluation uses greedy decoding at temperature 0, run-to-run variance is small (Section G.2, below); the dominant source of uncertainty is which items happen to be in the test set. We therefore report 95% percentile bootstrap confidence intervals over test items, resampling items with replacement 10,000 times per model–task cell and recomputing the metric on each resample. For Single Object Tracking the resampling unit is the object tracklet (푛 = 200) rather than the individual frame, since frames within a tracklet are strongly correlated. The half-width of each interval is given in Table 13. Interval width is governed by the number of scored items, so it varies far more across tasks than across models. Referring expressions and object localization are the tightest (median±1.00 and±1.27 respectively), while event verification is the widest by a wide margin (median 19 VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models ±7.47), reflecting its smaller item pool and binary scoring. The practical consequence is that the comparisons we draw in Section 4 should be read against very different resolutions: a two-point difference is meaningful on referring expressions and is not meaningful on event verification. SemanticSpatialTemporalSp.-Temp. ModelParamsVQAEVPoint.RefExObj Loc. Temp Loc.DVCSOT Open-weight models Qwen3-VL2B63.85±2.76 44.73±7.56 53.13±3.08 65.12±1.10 47.42±2.68 35.21±2.23 19.90±1.77 30.38±2.83 Qwen3-VL8B66.44±2.68 59.39±7.64 68.56±2.94 73.29±1.01 59.93±3.11 44.32±2.49 29.64±2.26 33.12±2.90 Qwen3-VL32B71.30±2.55 60.04±7.66 75.62±2.64 72.39±1.00 72.67±2.51 46.80±2.40 29.40±2.87 44.19±2.98 Qwen3.59B71.63±2.51 46.78±7.61 71.74±2.79 77.16±0.90 76.87±1.81 42.53±2.49 27.88±2.87 23.52±2.47 Qwen3.527B67.95±2.68 55.34±7.73 75.32±2.64 76.35±0.89 85.46±0.64 36.88±2.22 26.97±2.53 24.40±2.35 Cosmos-Reason22B64.69±2.72 55.27±7.97 59.70±3.08 56.33±1.24 73.56±1.62 38.95±2.53 28.52±3.12 26.85±2.71 Cosmos-Reason28B67.95±2.72 64.09±7.70 68.60±2.89 70.26±1.02 83.88±0.89 47.30±2.53 32.50±3.50 37.69±3.12 Cosmos-Reason232B70.29±2.59 73.58±7.00 74.03±2.74 50.04±1.05 4.35 † ±0.91 50.47±2.65 30.04±3.18 33.12±3.02 Cosmos3-Edge4B62.68±2.76 63.36±7.51 67.76±2.84 69.27±1.09 79.02±0.70 39.51±2.57 16.64±2.85 24.65±2.81 Cosmos3-Nano16B68.95±2.68 68.88±7.16 74.83±2.69 75.57±0.99 74.11±1.74 48.04±2.61 31.42±3.81 59.21±3.75 Cosmos3-Super64B69.46±2.59 71.28±7.12 72.94±2.74 76.23±0.98 86.97±0.70 51.90±2.57 29.54±3.31 64.66±3.67 Gemma-4-E2B2B46.19±2.85 27.65±4.52 31.74±2.94 5.52±0.40 45.40±0.74 9.02±1.10 9.14±1.83 11.58±1.31 Proprietary models Gemini 3.1 Flash-Lite —68.03±2.72 63.78±7.47 63.68±2.99 52.70±1.41 71.74±1.34 37.72±2.36 32.54±2.46 46.68±3.89 Gemini 3.1 Pro—71.46±2.55 68.57±7.23 72.04±2.79 67.17±1.27 77.21±1.73 45.69±2.36 35.55±2.82 67.88±3.98 Gemini 3.5 Flash-Lite —65.61±2.68 70.41±7.06 43.08±3.03 74.46±0.97 65.73±1.27 24.00±2.67 32.29±2.39 55.77±4.56 Gemini 3.6 Flash—76.82±2.38 82.00±6.26 79.20±2.54 75.78±0.87 81.57±0.61 51.50±2.67 36.15±3.16 75.99±3.44 GPT-5.6 Sol—78.08±2.38 76.14±6.72 81.90±2.39 69.44±0.91 69.30±0.72 55.71±2.65 37.28±2.88 75.02±3.34 Table 13: Scores with 95% item-level bootstrap confidence-interval half-widths (±), for every cell of Table 1, in points on the 0–100 scale (10,000 resamples; tracking resampled at the tracklet level) [44]. Scored items per task: VQA 1195; EV 163; Point. 1005; RefEx 3276; Obj Loc. 628; Temp Loc. 203; DVC 104; SOT 200. G.2. Seed Variance To confirm that reported differences are not seed-driven, we ran Qwen3-VL-8B three times with different random seeds (Table 14). Because decoding is greedy at temperature 0, this variance is expected to be small, and it is. All five tasks measured show a standard deviation below 0.5 points, with 2D Spatial Pointing and Single Object Tracking the largest at 0.42 and 0.40. This establishes that individual cells are reproducible across runs. It is not a basis for judging whether two models differ: seed variance holds the item set fixed and therefore omits the dominant source of uncertainty in benchmark scores, which is item sampling [44]. For model comparisons, the item-level intervals above are the relevant quantity. The dominant source of uncertainty is item sampling, which seed runs cannot measure. The variance reported here should therefore not be read as a confidence interval on the scores themselves; item-level bootstrap intervals for every model and task are reported separately in Appendix G.1. PillarTask (metric)Std. dev. across 3 seeds SemanticEvent Verification (Macro F1)0.31 SemanticVideo Question Answering (Accuracy)0.09 Spatial2D Spatial Pointing (Accuracy)0.42 TemporalDense Video Captioning (SODA 푐 )0.32 Spatio-Temp. Single Object Tracking (AUC)0.40 Table 14: Seed variance on VANTAGE-Bench (Qwen3-VL-8B), standard deviation across three random seeds, scaled 0–100. This reflects decoding randomness only, not item sampling. 20 VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models References [1]Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 2, 3, 9 [2]Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. MVBench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2, 3 [3] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. EgoSchema: A diagnostic benchmark for very long-form video language understanding. In Advances in Neural Information Processing Systems (NeurIPS), 2023. 3 [4]Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. BLINK: Multimodal large language models can see but not perceive. In Proceedings of the European Conference on Computer Vision (ECCV), 2024. 2, 9 [5]Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? The KITTI vision benchmark suite. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2012. 3 [6] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yuxin Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 3 [7] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo Open Dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 3 [8]Waqas Sultani, Chen Chen, and Mubarak Shah. Real-world anomaly detection in surveillance videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 3 [9]Sangmin Oh, Anthony Hoogs, Amitha Perera, Naresh Cuntoor, Chia-Chih Chen, Jong Taek Lee, Saurajit Mukherjee, J.K. Aggarwal, Hyungtae Lee, Larry Davis, et al. A large-scale benchmark dataset for event recognition in surveillance video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2011. 3 [10] Zheng Tang, Shuo Wang, David C. Anastasiu, Ming-Ching Chang, Anuj Sharma, Quan Kong, Munkhjargal Gochoo, Jun-Wei Hsieh, Tomasz Kornuta, et al. The 10th AI City Challenge. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2026. 3 [11]Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. TALL: Temporal activity localization via language query. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017. 3, 9 [12]Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017. 3 [13]Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expressions. In Proceedings of the European Conference on Computer Vision (ECCV), 2016. 3, 9 21 VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models [14]Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. LaSOT: A high-quality benchmark for large-scale single object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 3 [15]Lianghua Huang, Xin Zhao, and Kaiqi Huang. GOT-10k: A large high-diversity benchmark for generic object tracking in the wild. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(5):1562– 1577, 2021. 3 [16]Anton Milan, Laura Leal-Taixé, Ian Reid, Stefan Roth, and Konrad Schindler. MOT16: A benchmark for multi-object tracking. arXiv preprint arXiv:1603.00831, 2016. 3 [17] Xiao Wang, Xiujun Shu, Zhipeng Zhang, Bo Jiang, Yaowei Wang, Yonghong Tian, and Feng Wu. Towards more flexible and accurate object tracking with natural language: Algorithms and benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 4 [18]Zhu Zhang, Zhou Zhao, Yang Zhao, Qi Wang, Huasheng Liu, and Lianli Gao. Where does it exist: Spatio-temporal video grounding for multi-form sentences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 4 [19] Zongheng Tang, Yue Liao, Si Liu, Guanbin Li, Xiaojie Jin, Hongxu Jiang, Qian Yu, and Dong Xu. Human- centric spatio-temporal video grounding with visual transformers. IEEE Transactions on Circuits and Systems for Video Technology, 2021. 4 [20] Han Wang, Yanjie Wang, Yongjie Ye, Yuxiang Nie, and Can Huang. Elysium: Exploring object-level perception in videos via MLLM. In Proceedings of the European Conference on Computer Vision (ECCV), 2024. 4 [21]En Yu, Liang Zhao, Yana Wei, Jinrong Yang, Dongming Wu, Lingyu Kong, Haoran Wei, Tiancai Wang, Zheng Ge, Xiangyu Zhang, and Wenbing Tao. Merlin: Empowering multimodal LLMs with foresight minds. In Proceedings of the European Conference on Computer Vision (ECCV), 2024. 4 [22]Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al. MMT-Bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask AGI. In Proceedings of the International Conference on Machine Learning (ICML), 2024. 4 [23] Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. VideoChat-R1: Enhancing spatio-temporal perception via reinforcement fine-tuning. arXiv preprint arXiv:2504.06958, 2025. 4 [24] Biao Wang, Wenwen Li, and Jiawei Ge. R1-Track: Direct application of MLLMs to visual object tracking via reinforcement learning. arXiv preprint arXiv:2506.21980, 2025. 4 [25] Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, et al. SAM 3: Segment anything with concepts. In International Conference on Learning Representations (ICLR), 2026. 3 [26]Cheng-Yen Yang, Hsiang-Wei Huang, Wenhao Chai, Zhongyu Jiang, and Jenq-Neng Hwang. SAMURAI: Motion-aware memory for training-free visual object tracking with SAM 2. IEEE Transactions on Image Processing, 35:970–982, 2026. 3 [27]Xin Chen, Ben Kang, Wanting Geng, Jiawen Zhu, Yi Liu, Dong Wang, and Huchuan Lu. SUTrack: Towards simple and unified single object tracking. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025. 3 22 VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models [28]Ben Kang, Xin Chen, Simiao Lai, Yang Liu, Yi Liu, and Dong Wang. Exploring enhanced contextual information for video-level object tracking. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025. 3 [29]Zhichao Sun, Yepeng Liu, Zhiling Su, Huachao Zhu, Yuliang Gu, Yuda Zou, Zelong Liu, Gui-Song Xia, Bo Du, and Yongchao Xu. RefDrone: A challenging benchmark for referring expression comprehension in drone scenes. arXiv preprint arXiv:2502.00392, 2025. 5, 7 [30]NVIDIA. PhysicalAI-SmartSpaces: A multi-camera 3D perception dataset for smart spaces.https: //huggingface.co/datasets/nvidia/PhysicalAI-SmartSpaces, 2025. 5, 7 [31]Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. VLMEvalKit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM International Conference on Multimedia (ACM M), pages 11198–11201, 2024. 7, 19 [32]Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. MLVU: Benchmarking multi-task long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 9 [33]Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language- image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 13 [34]Soichiro Fujita, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. SODA: Story oriented dense video captioning evaluation framework. In Proceedings of the European Conference on Computer Vision (ECCV), 2020. 7 [35]Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations (ICLR), 2020. 13 [36]NVIDIA. NVIDIA Omniverse DRIVE Sim.https://developer.nvidia.com/drive/simulation, 2025. 7 [37] Qwen Team. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025. 7 [38] NVIDIA. Cosmos3: Reasoning vision-language models for physical AI.https://huggingface.co/nvidi a/Cosmos3-Super, 2026. 7, 10 [39] Google DeepMind. Gemma 4 model card.https://huggingface.co/google/gemma-4-e2b-it, 2026. 7 [40] OpenAI. GPT-5.6 Sol. https://openai.com/index/gpt-5-6/, 2026. 7 [41]Google DeepMind. Gemini 3.1 Pro model card.https://deepmind.google/models/model-cards/gem ini-3-1-pro/, 2026. 7 [42]NVIDIA. Cosmos-Reason2: An open reasoning vision-language model for physical AI and robotics. https://docs.nvidia.com/cosmos/latest/reason2/index.html, 2026. 7, 10 [43] Qwen Team. Qwen3.5 model release. https://huggingface.co/Qwen/Qwen3.5-9B, 2026. 7 [44] Evan Miller. Adding error bars to evals: A statistical approach to language model evaluations. arXiv preprint arXiv:2411.00640, 2024. 20 23