Paper deep dive
VidOmni-Bench: A Benchmark for Fine-Grained Video Understanding via Spatio-Temporal Event Verification across Complexity and Duration
Changbeen Kim, Junwon Chang, Kipyo Kim, Risa Shinoda, Kuniaki Saito, Donghyun Kim
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/21/2026, 4:30:31 AM
Summary
The paper introduces VidOmni-Bench, a benchmark for evaluating fine-grained video understanding in Video Large Language Models (Video-LLMs) through sentence-wise event verification. Unlike existing benchmarks relying on multiple-choice questions or sparse caption matching, VidOmni-Bench uses dense captions generated by Video-LLMs and human-verified labels to create hard negatives. The dataset comprises 500 videos spanning five complexity types and durations from 4 seconds to 90 minutes. Experiments reveal that Video-LLMs frequently hallucinate in dense captioning and struggle to verify plausible but incorrect events, with weaknesses varying significantly across video complexity and duration.
Entities (14)
Relation Signals (11)
VidOmni-Bench → contains → 500 videos
confidence 100% · VidOmni-Bench consists of 500 videos spanning five complexity types and diverse durations from 4 seconds to 90 minutes.
VidOmni-Bench → evaluates → Video Large Language Models
confidence 95% · VidOmni-Bench is a benchmark that requires models to verify whether each event in dense video captions is supported by the video.
Video LLMs → exhibits → Hallucination
confidence 95% · Video-LLMs frequently generate hallucinated descriptions in dense video captioning
VidOmni-Bench → stratifiedby → five complexity types
confidence 95% · Our collection is stratified into five predefined spatio-temporal complexity types: (1) Static World, (2) Body Motion, (3) Hand-Object Interaction, (4) Crowd Action, and (5) Multi-Scene Dynamics.
VidOmni-Bench → uses → dense captions
confidence 95% · we use diverse Video-LLMs to generate dense captions and obtain human-verified sentence-level labels
Video LLMs → struggleswith → event verification
confidence 90% · they also struggle as verifiers, failing to reliably detect plausible but incorrect event descriptions
InternVL3.5 → usedin → VidOmni-Bench
confidence 90% · We use five Video-LLMs... InternVL3.5... to generate dense captions
Qwen3-VL → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:While Video Large Language Models (Video-LLMs) have recently demonstrated strong performance, reliably evaluating their fine-grained video understanding remains challenging. Existing benchmarks often rely on question answering or ground-truth caption matching, where models may succeed through superficial cues and incomplete annotations. To this end, we introduce VidOmni-Bench, a benchmark that requires models to verify whether each event in dense video captions is supported by the video. VidOmni-Bench consists of 500 videos spanning five complexity types and diverse durations from 4 seconds to 90 minutes. After collecting videos along these axes, we use diverse Video-LLMs to generate dense captions and obtain human-verified sentence-level labels, where sentences containing incorrect events serve as hard negatives for evaluation. Our experiments on VidOmni-Bench reveal three key findings: (i) Video-LLMs frequently generate hallucinated descriptions in dense video captioning; (ii) they also struggle as verifiers, failing to reliably detect plausible but incorrect event descriptions; and (iii) model weaknesses vary across video complexity and duration, revealing diverse, model-specific bottlenecks in current Video-LLMs.
Tags
Links
- Source: https://arxiv.org/abs/2609.21521v1
- Canonical: https://arxiv.org/abs/2609.21521v1
Trouble viewing inline? Open PDF directly →
Full Text
75,138 characters extracted from source content.
Expand or collapse full text
VidOmni-Bench: A Benchmark for Fine-Grained Video Understanding via Spatio-Temporal Event Verification across Complexity and Duration Changbeen Kim 1 , Junwon Chang 1 , Kipyo Kim 2 , Risa Shinoda 3 , Kuniaki Saito 4 , Donghyun Kim 1 * 1 Korea University 2 Sungkyunkwan University 3 The University of Tokyo 4 OMRON SINIC X Corporation Abstract While Video Large Language Models (Video- LLMs) have recently demonstrated strong per- formance, reliably evaluating their fine-grained video understanding remains challenging. Ex- isting benchmarks often rely on question an- swering or ground-truth caption matching, where models may succeed through superfi- cial cues and incomplete annotations. To this end, we introduce VidOmni-Bench, a bench- mark that requires models to verify whether each event in dense video captions is supported by the video. VidOmni-Bench consists of 500 videos spanning five complexity types and di- verse durations from 4 seconds to 90 minutes. After collecting videos along these axes, we use diverse Video-LLMs to generate dense captions and obtain human-verified sentence- level labels, where sentences containing incor- rect events serve as hard negatives for evalu- ation. Our experiments on VidOmni-Bench reveal three key findings: (i) Video-LLMs fre- quently generate hallucinated descriptions in dense video captioning; (i) they also struggle as verifiers, failing to reliably detect plausible but incorrect event descriptions; and (i) model weaknesses vary across video complexity and duration, revealing diverse, model-specific bot- tlenecks in current Video-LLMs. 1 Introduction VideoLargeLanguageModels(Video- LLMs) (Tang et al., 2025b; Bai et al., 2025; Wang et al., 2025b; Yang et al., 2025b) have demonstrated strong capabilities in video under- standing tasks such as question answering and video retrieval.However, reliably evaluating their fine-grained video understanding remains challenging (Kong et al., 2025; Cai et al., 2024), as videos vary widely in duration and visual dynamics. A comprehensive evaluation should therefore assess whether models can interpret diverse video * Corresponding author. Short He is doing Taekwondo on a blue mat Video-LLM “Describe this video” Previous works VidOmni-Bench (Ours) A man is getting ready for Taekwondo. He punches into the air. He lifts his leg. He performs a high block. Generated Detail Caption “Incorrect” He is running Correct. The streamer is playing... Incorrect. Not other class Correct. His strategy ... Incorrect. No mana bar is visible... Video-LLM Verify Correctness Following the kick, he seamlessly transitions into a knife-hand strike. The video presents a detailed and immersive journey through a series of Hearthstone matches. [O/X] The players' strategic use of cards and their ability to adapt to different opponents' strategies are highlighted. [O/X] The player's mana bar is partially filled, showing the remaining mana available for the turn. [O/X] The streamer continues with his strategy of generating cards from other classes. [O/X] Correct. Incorrect. Correct. “Horse’s legs come into view.” Query: Describe this video. There is a leg of a horse. B. Standing. A.Sitting B. Standing C. I don’t know Minimum Temporal Evidence Coverage Correct. Incorrect. Multiple-Choice Question-Anserwing Question: What is the horse doing? Correct. Figure 1: Top: Previous benchmarks typically rely on MCQA or sparse GT captions, allowing models to succeed with limited temporal evidence. Bottom: In VidOmni-Bench, models must verify each event in dense captions against the video, enabling a more rigor- ous assessment of fine-grained video understanding. evidence across different temporal scales and visual aspects, including object states, actions, background changes, and scene transitions. Recent temporal and long-form video bench- marks mostly rely on Multiple-Choice Question Answering (MCQA) (Fu et al., 2025; Fang et al., 2024; Tan et al., 2025) or Ground-Truth (GT) cap- tion matching (Rawal et al., 2025; Liu et al., 2026; Lee et al., 2025). MCQA-based evaluations may al- low models to rely on localized evidence, or textual shortcuts without temporal reasoning (Feng et al., 2025). Caption-matching-based benchmarks can assess video understanding more rigorously, but their focus on relatively short videos and human- written GT captions limits scalable dense event- level coverage across diverse video durations. Con- sequently, existing benchmarks do not directly eval- uate whether Video-LLMs can understand fine- grained video events by integrating broad temporal evidence across diverse durations. To address these gaps, we introduce VidOmni- Bench, a sentence-wise event verification bench- arXiv:2609.21521v1 [cs.CV] 18 Sep 2026 Video-LLM Sentence 1: The video opens with a close-up of a hand holding a silver credit card, transitioning to a personin a white shirt and beige vest working at a desk with a laptop and a tablet displaying financial data. Sentence 2: The scene shifts to a man in a suit and red tie walking down a hallway, followed by a close-up of a hand writing on a document with a pen, accompanied by a calculator and financial charts. ... Sentence 12: The final scene shows a man in a dark suit and blue tie speaking in front of a red background, identified as 'Dominick Gabriele, Senior analyst, Oppenheimer & Co.' Sentence 1 is Incorrect. The video opens with a wide shot of a multi-level, modern building interior, not a close-up of a hand holding a credit card. Sentence 2 is Correct. This is accurate as the video shows a man in a suit walking, then a hand writing on a document with a calculator and charts. Sentence 12 is Incorrect. The final scene with a man in a dark suit and blue tie speaking in front of a red background, identified as 'Dominick Gabriele, Senior analyst, Oppenheimer & Co.' is not present. Order Action Success Fail Video Complexity Type Type 1 Static World ... Type 2 Body Motion ... Type 3 Hand-Object Interaction ... Type 4 Crowd Action ... Type 5 Multi-Scene Dynamics ... Illusion Correct T1 T2 T3 T4 T5 Fail Input Video-Caption Query Sentence-wise Event Verification 4 sec ~ 90 min 6 sec ~ 56 min 9 sec ~ 63 min 11 sec ~ 81 min 28 sec ~ 60 min F1 - Score Short Medium Long Complexity Weakness F1 - Score Weakness Analysis Duration Weakness Complexity Type Duration Weak in Body Motion Duration: Long Type 4: Crowd Action Weak in Long Videos Figure 2: We introduce VidOmni-Bench, a novel benchmark designed to evaluate how well Video-LLMs distinguish between correct and incorrect events across five types of spatio-temporal complexity and diverse video durations. This structured framework enables a rigorous assessment of whether Video-LLMs can truly ground their reasoning in the visual evidence across diverse scales. mark where a model verifies whether each sentence in dense video captions is supported by the video. As shown in Figure 2, VidOmni-Bench consists of 500 videos curated across five complexity types and diverse durations from 4 seconds to 90 minutes. For each video, we use a diverse set of Video-LLMs to generate dense captions, and human annotators ver- ify each sentence to produce sentence-level labels. Hallucinated or incorrect sentences are then used as hard negatives for evaluation, resulting in a bench- mark designed to assess event verification across diverse types of video complexity and duration. As shown in Table 2, VidOmni-Bench requires models to inspect substantially broader portions of the video than existing benchmarks, indicating that VidOmni-Bench cannot be solved through su- perficial cues. Furthermore, Figure 7 shows that the captions in VidOmni-Bench provide dense and complex event descriptions across the five complex- ity types. These results support VidOmni-Bench as a challenging benchmark for fine-grained video understanding. Our experiments on VidOmni-Bench reveal sev- eral critical limitations of current Video-LLMs. First, Video-LLMs frequently produce hallucinated descriptions during dense video captioning, indi- cating unreliable video-grounded generation. Sec- ond, they also struggle as verifiers, failing to reli- ably detect plausible but incorrect event descrip- tions within these captions. Third, model weak- nesses vary substantially across video complexity and duration, revealing diverse, model-specific bot- tlenecks rather than a single uniform failure mode. These findings indicate that current Video-LLMs still lack robust, temporally grounded event under- standing needed to faithfully interpret dense and diverse video content. 2 Related Works Video Large Language Models. Following the success of Large Language Models (LLMs) (Yang et al., 2025a; Grattafiori et al., 2024), multi- modal understanding has expanded from static im- ages (Liu et al., 2023; Chen et al., 2024b) to dy- namic videos (Li et al., 2024a,b; Chen et al., 2024a; Bai et al., 2025). Recent Video-LLMs can generate detailed video descriptions, but their outputs often contain subtle hallucinations (Gao et al., 2025). We use these generated descriptions for sentence-wise event verification, with human annotators labeling hallucinated sentences as hard negatives. Video Understanding Benchmarks.Recent benchmarks have advanced the evaluation of tem- poral and long-form video understanding (Ben Ami et al., 2026; Ha et al., 2026; Li et al., 2026; Cai et al., 2024; Kong et al., 2025), but many remain based on MCQA formats. Such positive-selection formats can be affected by language priors or par- tial visual evidence (Feng et al., 2025), and provide limited evidence of whether models can reject plau- sible but incorrect events. VidOmni-Bench instead evaluates dense event verification beyond sparse question-answer pairs. Hallucination in Video Understanding. Several studies evaluate video captioning and hallucination detection by comparing model outputs with human- Video CollectionDense CaptionGeneration Multi-Stage Annotation Data Filtering Public Datasets YouTube Sourced Duration Video Complexity The video begins with two men standing... The scene then shifts to a gym where... Finally, two men stand beside the gym... The video begins with two men... The scene then shifts to a gym... Finally, two men stand beside... Stage 1 : Sentence-wise Correctness Majority Vote Stage 2 : Word-Level Errors The video begins with two menstanding and facing the camera in front of BEFIT 360° sign. Error Category: Illusion “A man and a woman.” Annotations with Unknown Category Final Benchmark (VidOmni-Bench) Figure 3: Construction pipeline of VidOmni-Bench. We collect videos from public datasets and YouTube along with two axes. Then, we utilize diverse Video-LLMs for dense caption generation. These video and dense caption pairs then undergo a multi-stage annotation phase for sentence-wise error (hallucination) labeling, where annotators label sentence-level errors to construct hard negatives. Finally, the data filtering stage for a robust evaluation. 12345 Video Complexity Type 0 20 40 60 80 100 Number of Videos 31 28 23 2121 35 38 49 49 39 3434 28 30 40 ShortMediumLong Figure 4: Our dataset consists of 500 videos, with ex- actly 100 videos allocated to each of the five complexity types. Within each type, we maintain a balanced distri- bution across Short, Medium, and Long duration. written GT captions (Liu et al., 2026; Rawal et al., 2025; Lee et al., 2025). While useful for assessing factual consistency, such sparse references limit dense event-level coverage across diverse domains and durations. Motivated by Saito et al. (2025), VidOmni-Bench uses Video-LLM-generated dense descriptions and human-verified negatives to ana- lyze hallucination patterns across video complexity and duration. 3 VidOmni-Bench To evaluate the fine-grained understanding of Video-LLMs, we introduce a sentence-wise event verification task. This task requires models to act as rigorous verifiers, determining both the factual occurrence of described events and their temporal consistency within the video. To provide a granu- lar diagnosis of model capabilities, we constructed VidOmni-Bench by carefully disentangling spatio- temporal complexity from video duration. Rather than relying on human-written GT captions, we generate dense captions using diverse Video-LLMs and verify them at the sentence level. The fol- lowing sections define the task, describe the video collection and annotation process, and present sta- tistical analysis. Overall construction process of VidOmni-Bench is illustrated in Figure 3. 3.1 Task Definition Sentence-wise event verification evaluates whether each sentence in a video caption is factually sup- ported by the video. Formally, given a videoV and ann-sentence captionC = s 1 ,s 2 ,...,s n , the goal is to predict binary labelsy = y 1 ,y 2 ,...,y n . For each sentences i , we define y i ∈0, 1 as: y i = ( 0, if s i is factually supported by V, 1, otherwise. (1) Thus, a Video-LLM must verify each sentence against the video evidence and determine whether it contains hallucinated or incorrect content. 3.2 Video Collection To enable a granular diagnosis of Video-LLMs, we curate videos by controlling spatio-temporal com- plexity and video duration as two separate axes. We collect videos from diverse public datasets, covering egocentric actions (Grauman et al., 2022; Damen et al., 2018; Sener et al., 2022), anomaly events (Sultani et al., 2018; Wu et al., 2020), and long-form content (Caba Heilbron et al., 2015; Fu et al., 2025). We further supplement these sources with targeted YouTube crawling to fill underrepre- sented combinations of duration and complexity. For crawled videos, we only retain content released under Creative Commons licenses. Our collection is stratified into five predefined spatio-temporal complexity types: (1) Static World, (2) Body Motion, (3) Hand-Object Interaction, (4) Crowd Action, and (5) Multi-Scene Dynamics. As shown in Figure 4, each complexity type is bal- anced across three duration brackets: short (≤3 min), medium (4–30 min), and long (≥30 min). This design enables us to analyze the effects of complexity and duration separately. The resulting VidOmni-Bench contains 500 videos, 433 of which include audio tracks. Detailed definitions of com- plexity types and data sources are provided in the Appendix B. 3.3 Annotation Pipeline To overcome the limited coverage of human- written GT captions, we use modern Video-LLMs to generate dense descriptions and repurpose their human-verified hallucinations as hard negatives. We use five Video-LLMs with diverse architectures and parameter scales: InternVL3.5 (Wang et al., 2025b) (38B), Qwen3-VL (Bai et al., 2025) (32B), video-SALMONN 2+ (Tang et al., 2025a) (72B), Keye-VL 1.5 (Yang et al., 2025b) (8B), and Gem- ini 3 Flash (Google, 2025). For each video, one of five specialized chat templates (e.g., “Describe the full content of the video in detail.”) is ran- domly assigned to each model, producing 2,500 video-caption pairs in total. Using multiple models and prompts diversifies the generated descriptions and reduces dependence on the output of a single model. These raw captions serve as the basis for a two- stage human annotation process. In the first stage, five independent annotators verify the factual cor- rectness of each sentence and label it as Correct, Incorrect, or Unknown. Final labels are determined by majority vote, and Unknown cases are excluded to maintain reliable evaluation. In the second stage, we perform word-level error annotation for sentences labeled as Incorrect. An- notators identify the incorrect words within each sentence and assign each error to one of ten cate- gories: five related to static-frame understanding (Object, Relation, Location, Direction, and Illu- sion) and five related to temporal understanding (Action, Order, Causality, Frequency, and State Change). The resulting human-verified incorrect sentences serve as hard negatives for sentence-wise event verification. Details for dense captioning, and annotation process are provided in the Appendix B. 3.4 Data Statistics and Analysis Basic statistics. Table 1 summarizes VidOmni- Bench, which contains 64K annotated sentences. We report the Video Incorrect Rate, defined as the proportion of video-caption pairs in which the gen- erated caption contains at least one hallucinated sentence, along with the average sentence-level In- correct Rate per caption. The results show that 0 10 20 30 40 Error Type Percentage (%) InternVL 3.5 KeyeVL 1.5 video- SALMONN 2+ Qwen3-VL Gemini 3.0 Flash Object Action Relation Order Location Causality Direction Frequency Illusion State Change Figure 5: Word-level error statistics per category. The consistent high frequency of temporal errors in dense video captioning highlights a fundamental bottle- neck in video understanding. ShortMedium Long Duration 0 20 40 60 80 Incorrect / Caption (%) Type 5 12345 Complexity Type 0 20 40 60 80 Incorrect / Caption (%) Long InternVL 3.5 video-SALMONN 2+ Gemini 3.0 Flash KeyeVL 1.5 Qwen3-VL Figure 6: Effect of video complexity and duration on hallucination. Plots show the average sentence-level Incorrect Rate per caption. Left: across durations for Type 5 videos. Right: across complexity types within Long videos. Dashed lines indicate overall trends. Video-LLMs frequently generate hallucinated cap- tions, with Incorrect rates remaining high across all models, thereby providing a sufficient pool of hard negatives for sentence-wise event verification. Error category distribution. As shown in Fig- ure 5, temporal errors such as Action, Order, and Frequency account for a substantial portion of hal- lucinations, suggesting that current Video-LLMs struggle with fine-grained temporal reasoning. Fig- ure 2 further shows that a sentence can appear plau- sible in isolation but become Incorrect when veri- fied against the broader video context. Effect of complexity and duration on dense cap- tioning. By controlling video complexity and du- ration, we analyze hallucination patterns in dense captioning (Figure 6). The sentence-level Incorrect Rate per caption generally increases with complex- ity type within each duration bracket, while Type 5 remains consistently high across all durations. This suggests that hallucinations are driven not only by video length, but also by the difficulty of reasoning over complex event dynamics. Caption characteristics differ across complex- ity types. We analyze how caption properties and KeyeVL 1.5InternVL 3.5Qwen3-VLvideo-SALMONN 2+Gemini 3 Params.8B38B32B72BN/A Total Sentences18974554211868189749485 Video Incorrect Rate (%)92.685.694.092.291.1 Incorrect / Caption (%)40.639.526.544.924.1 Table 1: Statistics of incorrect sentence in generated captions. VidOmni-Bench provides a large set of annotated incorrect sentences for evaluating sentence-wise event verification. Sentences labeled as Unknown are excluded. 12345 Complexity Type 0 20 40 60 80 Percentage (%) Video Incorrect Rate (%) - Action, Order, Causality, Frequency, State Change Certificate Coverage (%)Average Word Count Per Caption 400 500 600 700 800 Average Word Count Per Caption Figure 7: Caption characteristics across complexity types. Higher-complexity videos lead to longer captions and a higher Video Incorrect Rate for temporal error categories, while also requiring broader video evidence in sentence-wise event verification of VidOmni-Bench. verification requirements vary across the five com- plexity types in VidOmni-Bench. Figure 7 shows that higher complexity is associated with longer captions and higher Video Incorrect Rates over tem- poral error categories(Action, Order, Causality, Fre- quency, and State Change). We further report Cer- tificate Coverage following Video-MME (Fu et al., 2025), showing that sentence-wise event verifica- tion generally requires broader video evidence as complexity increases. Details for calculating Cer- tificate Coverage are provided in the Appendix D. Comparison with existing benchmarks. Figure 1 and Table 2 position VidOmni-Bench among ex- isting video understanding benchmarks. Moving beyond MCQA-based evaluation (Fu et al., 2025; Ben Ami et al., 2026; Li et al., 2026), VidOmni- Bench performs sentence-wise event verification over dense descriptions. Compared with bench- marks based on human-written GT caption match- ing (Rawal et al., 2025; Lee et al., 2025; Liu et al., 2026; Ma et al., 2025), VidOmni-Bench spans broader video durations. Its higher Certificate Coverage further shows that verification requires broader video evidence, while the Appendix D shows that VidOmni-Bench captions are more com- plex than those in prior benchmarks. BenchmarkEvaluation TaskDuration Cert. Cov. (%) TimeBlind (Li et al., 2026)MCQA≤ 25s32.3 NOAH (Lee et al., 2025)GT Matching / MCQA ≤ 12m30.8 ARGUS (Rawal et al., 2025)GT Matching ≤ 3m25.0 HERBench (Ben Ami et al., 2026)MCQA≤ 6m24.8 Video-MME (Fu et al., 2025)MCQA≤ 60m3.4 CAPAbility (Liu et al., 2026)GT Matching / MCQA ≤ 50s2.0 VideoEval-Pro (Ma et al., 2025)GT Matching ≥ 30m0.9 VidOmni-Bench (Ours)Sentence Verification ≤ 90m54.0 Table 2: Comparison with other benchmarks in task design and temporal evidence. VidOmni-Bench evalu- ates sentence-wise event verification over longer videos and requires broader temporal evidence, indicated by higher Certificate Coverage (Cert. Cov.). 4 Experiments In this section, we evaluate a wide range of Video- LLMs on VidOmni-Bench to assess their fine- grained video understanding. We first present quan- titative results, followed by a detailed analysis of model performance across different dimensions. In summary, our analysis yields five key findings: (i) VidOmni-Bench reveals verification weaknesses not captured by existing benchmarks, showing that models often fail to consistently identify halluci- nated sentences; (i) scaling model size mainly im- proves verification of static-frame-related errors, whereas temporal ordering and causal reasoning re- main major bottlenecks; (i) current Video-LLMs exhibit a distinct self-preference bias, showing un- reliable verification behavior when judging their own generated sentences; (iv) audio often provides verification-relevant information complementary to visual evidence, and models that effectively in- corporate these cues achieve stronger performance; and (v) model vulnerabilities are multidimensional, with performance depending on model-specific sen- sitivity to both complexity and duration. 4.1 Experimental Setup Benchmark coverage and evaluation protocols. We evaluate Video-LLMs on VidOmni-Bench to- gether with three existing video understanding benchmarks: Video-MME (Fu et al., 2025), Video- Video-LLMModality Video-MME Video-MMMULVBenchVidOmni-Bench QA-Acc. (%) QA-Acc. (%) QA-Acc. (%) Precision (%) Recall (%) F1-Score (%) Open-Source Models Keye-VL 1.5 8Bv73.066.035.1 † 19.18.49.6 InternVL 3.5 2Bv58.441.8 † 30.5 † 6.01.82.3 InternVL 3.5 8Bv66.048.9 † 34.2 † 26.413.914.7 InternVL 3.5 14Bv67.954.8 † 35.1 † 32.318.119.4 InternVL 3.5 38Bv70.957.9 † 38.1 † 25.012.714.1 Qwen3-VL 2Bv61.941.947.45.54.23.6 Qwen3-VL 4Bv69.356.256.222.911.712.7 Qwen3-VL 8Bv71.465.358.025.913.614.7 Qwen3-VL 32Bv76.671.963.835.419.621.7 video-SALMONN 2+ 7Bv67.8 † 46.0 † 49.71.31.11.1 video-SALMONN 2+ 7Bv, a73.450.2 † 47.8 † 0.10.20.1 video-SALMONN 2+ 72Bv74.0 † 62.0 † 55.58.35.45.8 video-SALMONN 2+ 72B v, a79.760.8 † 55.2 † 5.73.43.7 Closed-Source Models Gemini 3 Flashv73.2 † 80.3 † 74.3 † 54.637.239.8 Gemini 3 Flashv, a86.1 † 75.3 † 76.9 † 54.739.941.7 GPT 5 miniv71.056.757.8 † 39.333.228.9 Table 3: Main results on VidOmni-Bench. We report verification Precision, Recall, and F1-Score for VidOmni- Bench, alongside QA Accuracy (QA-Acc) for Video-MME, Video-MMMU, and LVBench. Models are evaluated under video-only (v) or video+audio (v,a) settings. † denotes reproduced performance. Object Relation Location Direction Illusion Action Order Causality Frequency State Change 0 10 20 30 40 50 60 Verification Recall (%) InternVL 3.5 38B KeyeVL 1.5 8B video-SALMONN 2+ 72B Qwen3-VL 32B Gemini 3.0 Flash Figure 8: Performance of Video-LLMs across error categories. This visualization highlights the relative strengths and weaknesses of current Video-LLMs in detecting specific static and temporal discrepancies. MMMU (Hu et al., 2026), and LVBench (Wang et al., 2025a). Since these benchmarks are for- mulated as Question Answering (QA) tasks, we report QA accuracy following their original eval- uation protocols. In contrast, VidOmni-Bench is formulated as a sentence-wise event verification task, where sentences labeled Incorrect are treated as the detection target. For thei-th video-caption pair (i = 1,...,N), we compute Precision, Recall, and F1-Score over all sentences within the pair: Precision i = TP i TP i + FP i ,Recall i = TP i TP i + FN i , F1-Score i = 2· Precision i · Recall i Precision i + Recall i .(2) We then report the averaged scores over allNpairs. Evaluated models. To analyze how model charac- teristics affect verification performance, our model selection in evaluation is guided by three criteria. First, we include models with varying parameter scales to examine the impact of model capacity. Second, for audio-capable models, we evaluate both video-only and video+audio settings to assess the contribution of audio signals to event verifica- tion. Third, we evaluate powerful closed-source models, including GPT 5 mini (Singh et al., 2025), as strong reference points for current model per- formance. Detailed model configurations and the evaluation prompt are provided in Appendix C. 4.2 Overview of Results Table 3 presents the quantitative evaluation of di- verse Video-LLMs on VidOmni-Bench, together with their performance on existing benchmarks. Qualitative examples of Video-LLM outputs are shown in Figure 9. VidOmni-Bench enables rigorous evaluation be- yond existing benchmarks. Our sentence-wise event verification task effectively differentiates Video-LLM performance. High leaderboard scores on existing benchmarks such as Video-MME (Fu et al., 2025), Video-MMMU (Hu et al., 2026), and LVBench (Wang et al., 2025a) do not necessar- ily translate to strong performance on VidOmni- Caption Generated by Qwen3-VL Incorrect.Thecookdoesnotaddoiltoanewpan;they continuecookinginthesamepan. Incorrect.Thevegetablesbeingchoppedincludebell peppers,onions,andgreenbeans,notcarrotsand zucchini. Correct.Thechoppedvegetablesareaddedtothepan andsautéed. InternVL3.5 38B Correct. The cook returning to the stove and adding oil to a clean pan is consistent with the video. Correct. The cook chopping colorful vegetables, including bell peppers, carrots, and zucchini, on a blue cutting board is accurate. Correct. The chopped vegetables being added to the and sauteed with oil is consistent with the video. The cook adds the vegetables to the pan and stirs them. Correct.Thecookaddsoiltoacleanpanonthestove. Correct.Thecookchopsbellpeppers,carrots,and zucchinionabluecuttingboard. Correct.Thechoppedvegetablesareaddedtothepan andsautéedwithoil. Qwen3-VL 32B Keye-VL 1.5 8B GPT5 mini Correct. The cook returns to the stove and pours oil into a clean pan (visible in several frames). Incorrect. Carrots and zucchini are not clearly present in the frames, so that part is incorrect. Correct. The chopped vegetables are poured/added into the pan and then sautéed with oil –that is visible. ... Next,thecookreturnstothestoveand addsoiltoacleanpan,preparingitfor thenextsetofingredients. The cook then begins to chopa variety of colorful vegetables, including bell peppers, carrots, and zucchini,on a blue cutting board. The chopped vegetables are then added to the pan, where they are sautéed with the oil. ... Correct Correct Action,Illusion video-SALMONN 2+ 72B Correct.Correct.Correct.Thevideodescription providedisentirelyconsistentwiththevisualcontent showninthevideo.Eachsentenceaccuratelydescribes theelementsandactionsobservedinthekitchen setting,suchasthemodernkitchendesign,thecook's attire,thecookingprocessinvolvingvariousvegetables, andtheemphasisoncleanlinessandorganization.There arenodiscrepanciesorhallucinationsinthedescription; alldetailsalignwiththevisualevidencepresentedinthe video. Correct.Thesentenceisfaithful,showingoilbeing addedtothepanafterthechickenisremoved. Incorrect.Thesentencecontainsahallucination;the vegetablesbeingchoppedarebellpeppersandonions, notcarrotsorzucchini. Correct.Thesentenceisfaithful,describingthe choppedvegetables. Gemini 3 Flash Sentence-wise Event Verification Results Chop new vegetables Sautéedwith oil Add oil to a clean pan Put vegetables already chopped Incorrect description No carrot and zucchini Figure 9: Qualitative examples of sentence-wise event verification on a Qwen3-VL caption. Video-LLMs may produce the same prediction while relying on different reasoning, revealing varying degrees of video understanding. ShortMedium Long Duration 20 25 30 35 F1-Score (%) Qwen3-VL 32B Type 3 ShortMedium Long Duration 20 25 30 35 F1-Score (%) Type 4 ShortMedium Long Duration 20 25 30 35 F1-Score (%) Type 5 ShortMedium Long Duration 30 35 40 45 F1-Score (%) Gemini 3.0 Flash ShortMedium Long Duration 30 35 40 45 F1-Score (%) ShortMedium Long Duration 30 35 40 45 F1-Score (%) Figure 10: F1-Score variations by video duration under fixed complexity. Each column represents a specific complexity type, illustrating how performance changes across Short, Medium, and Long durations. Bench. We further observe a significant gap be- tween Precision and Recall across most models, indicating that they often fail to consistently iden- tify hallucinated sentences. These findings show that VidOmni-Bench provides a more challenging testbed for fine-grained video understanding than existing evaluation benchmarks. Performance comparison. Gemini 3 Flash shows the highest overall performance when using both video and audio inputs, serving as the strongest ref- erence point among evaluated settings. Among open-source models, Qwen3-VL 32B performs best, while the InternVL3.5 family also shows com- petitive results. However, Figure 8 shows that In- ternVL3.5 38B is particularly strong at verifying static-frame-related hallucinations but weaker on temporal errors. This suggests that strong frame- level understanding does not necessarily translate to robust temporal reasoning. Examples of Video-LLMs’ output. Figure 9 shows sentence-wise event verification results from multiple Video-LLMs on a Qwen3-VL-generated caption. Each model outputs a sentence-level pre- diction together with reasoning, allowing us to qual- itatively compare not only the final judgments but also the evidence used to support them. Effect of scaling model size on event verification. Performance generally improves with model scale within the same family. Larger models achieve higher F1-Scores, suggesting that increased param- eter capacity enhances their ability to detect subtle discrepancies between hard negative sentences and the corresponding video evidence. Effect of audio modality on event verification. As shown in Table 3, adding audio consistently de- creases the F1-Scores within the video-SALMONN 2+ family, indicating that audio may introduce cross-modal noise when it is not properly grounded with visual evidence. In contrast, Gemini 3 bene- fits from audio input, where audio often provides verification-relevant cues complementary to visual information. This suggests that audio can improve sentence-wise event verification when models ef- fectively identify and integrate audio evidence rele- vant to each caption sentence. 4.3 Detailed Analysis Success cases of event verification. As illustrated in Figure 9, successful verification is often tightly InternVL 3.5Qwen3-VLvideo-SALMONN 2+ Object Relation Location Direction Illusion Action Order Causality Frequency State Change 19.114.11.9 13.410.22.0 17.111.61.9 12.514.00.0 17.625.03.7 8.910.60.9 9.714.30.6 8.34.04.2 8.116.50.5 8.59.70.0 0.0 5.0 10.0 15.0 20.0 25.0 Verification Recall Gain (+p.p.) Figure 11: Recall gain by error category as model size increases. The heatmap shows the improvement in Recall for InternVL3.5, Qwen3-VL, and video- SALMONN 2+ when scaling up parameters. coupled with factually grounded reasoning. This indicates that the model’s output is based on an integrated understanding of the video content. Failure cases of event verification. We observe two primary failure patterns. First, models may pre- dict an Incorrect sentence as Correct by overlook- ing its mismatch with the video evidence. Second, even correct predictions can be accompanied by incomplete or factually flawed explanations. These cases highlight the need to examine both predic- tions and explanations when assessing whether ver- ification is grounded in the video evidence. Video-LLMs have their own weak complexity and duration. Figure 10 shows that performance trends across complexity and duration differ by model. For example, Gemini 3 Flash exhibits a con- sistent F1-Score decline as duration shortens within high-complexity types, while Qwen3-VL shows different vulnerabilities depending on the complex- ity type. These results demonstrate that VidOmni- Bench diagnoses multidimensional, model-specific weaknesses beyond a single difficulty axis. Further analyses are provided in the Appendix C. Category-wise performance gains from model size scaling. To examine where performance im- provements arise as Video-LLMs scale, we mea- sure Recall gains across error categories when in- creasing model size for InternVL3.5 (2B to 38B), Qwen3-VL (2B to 32B), and video-SALMONN 2+ (7B to 72B). Figure 11 shows that scaling con- sistently improves Recall more for static-frame- related errors than for temporal-related errors. These results suggest that increasing model size within the same family mainly improves verifica- tion of static visual attributes and spatial relations, whereas temporal errors remain challenging. InternVL 3.5 KeyeVL 1.5 Qwen3 VL video- SALMONN 2+ Gemini 3 Flash InternVL 3.5 KeyeVL 1.5 Qwen3 VL video- SALMONN 2+ Gemini 3 Flash Verifiers Caption Source 13.113.520.813.89.3 21.14.75.47.17.5 36.221.512.220.916.6 19.31.03.80.32.2 44.542.147.044.621.9 10 20 30 40 F1-Score (%) InternVL 3.5 KeyeVL 1.5 Qwen3 VL video- SALMONN 2+ Gemini 3 Flash 0 10 20 30 40 50 F1-Score (%) Cross-Source Verification Self-Verification Figure 12: Self-preference bias in self-verification. Left: The heatmap shows F1-Scores for sentence-wise verification, with the y-axis representing the evaluated models and the x-axis representing the caption source models. Right: The bar chart compares the performance on cross-source verification versus self-verification. Video-LLMs struggle to detect their own hallu- cinations. As shown in Figure 12, Video-LLMs perform worse on their own captions than on cap- tions generated by other models. This trend is clear even for Gemini 3 Flash, the best-performing model. Its F1-Score exceeds 40 on captions from other models but drops by nearly half on its own captions. This suggests a strong self-preference bias, where models tend to judge their own gener- ated sentences as Correct. Further analysis in the Appendix C shows that ensemble-based verifica- tion can mitigate this bias. These results indicate that self-evaluation remains a significant challenge for current Video-LLMs. Additional analysis in the Appendix. The Ap- pendix C provides additional robustness analyses for VidOmni-Bench. Increasing the input frames does not yield clear performance improvements, indicating that the benchmark’s difficulty cannot be resolved simply by sampling more frames. We also evaluate a subset excluding YouTube-sourced videos and observe similar trends, suggesting that the strong performance of Gemini 3 and GPT 5 is unlikely to be driven primarily by data leakage. 5 Conclusion We introduced VidOmni-Bench, a sentence-wise event verification benchmark for fine-grained video understanding in Video-LLMs. By requiring mod- els to verify dense caption sentences across video complexity and duration, VidOmni-Bench reveals model-specific vulnerabilities and self-preference bias. These findings show that even strong mod- els still struggle with reliable, temporally grounded event verification. We hope VidOmni-Bench serves as a rigorous evaluation standard for more faithful fine-grained video understanding. Limitations VidOmni-Bench has several limitations that sug- gest directions for future work. First, verifying dense video captions inevitably involves some am- biguity, especially for complex temporal events. We mitigate this issue by introducing an Unknown category and using a multi-step labeling protocol, but complete agreement can still be difficult in borderline cases. Second, our analysis reveals self-preference bias in Video-LLMs, suggesting that model-based caption generation and evaluation may benefit from more diverse caption sources and cross-verification protocols. Third, the benchmark construction depends on the generative capability of current Video-LLMs and requires human verifi- cation to ensure data quality, which may limit scal- ability. Despite these limitations, VidOmni-Bench provides a useful diagnostic resource for studying current gaps in fine-grained video understanding. Ethics Statement AI assistant usage in writing. ChatGPT (Singh et al., 2025) was used to improve readability during manuscript preparation. The authors reviewed and edited all generated text and take full responsibility for the final content. Dataset construction. VidOmni-Bench is con- structed from publicly available video datasets to- gether with supplementary videos crawled from YouTube under Creative Commons licenses. To re- spect the copyright and terms of service of the orig- inal content providers, we do not redistribute raw video files for any of the 500 videos in VidOmni- Bench, regardless of source. Instead, we will re- lease only video metadata, the generated dense captions, and the annotations upon publication. We note that VidOmni-Bench includes videos from anomaly detection datasets that may depict vio- lent or disturbing events, as well as footage that may contain identifiable individuals. We apply no additional filtering beyond the curation already per- formed by the original providers, and recommend that users exercise discretion accordingly. Acknowledgements This research was supported by the Institute of In- formation &communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No. RS-2019-I190079, Arti- ficial Intelligence Graduate School Program(Korea University), 1%; No. RS-2025-25439490, 40%), the KOCCA grant (RS-2024-00345025, 10%), the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT)(No. RS-2025-02263628, 34%) This research was sup- ported by JST PRESTO, Japan, Grant Number JP- MJPR2523 (5%). This work was partly achieved through the use of SQUID at D3 Center, The Uni- versity of Osaka (5%). This work was supported in part by the Physical AI Development Support Program by AWS Japan through the provision of computational resources (5%). References Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, and 1 others. 2025. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Dan Ben Ami, Gabriele Serussi, Kobi Cohen, and Chaim Baskin. 2026. Herbench: A benchmark for multi-evidence integration in video question answer- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4505–4514. Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961–970. Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, and 1 others. 2024. Tempo- ralbench: Benchmarking fine-grained temporal un- derstanding for multimodal video models. arXiv preprint arXiv:2410.10818. Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, and 1 others. 2024a. Sharegpt4video: Improving video understanding and generation with better captions. Advances in Neural Information Processing Systems, 37:19472–19495. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, and 1 others. 2024b. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185–24198. Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and 1 others. 2018. Scaling egocentric vision: the dataset. In European conference on com- puter vision, pages 753–771. Springer. Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. 2024. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. Advances in Neural Information Processing Systems, 37:89098–89124. Bo Feng, Zhengfeng Lai, Shiyu Li, Zizhen Wang, Simon Wang, Ping Huang, and Meng Cao. 2025. Breaking down video llm benchmarks: Knowledge, spatial perception, or true temporal understanding? arXiv preprint arXiv:2505.14321. Joseph L Fleiss. 1971. Measuring nominal scale agree- ment among many raters. Psychological bulletin, 76(5):378. Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, and 1 oth- ers. 2025. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In 2025 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 24108–24118. IEEE. Hongcheng Gao, Jiashu Qu, Jingyi Tang, Baolong Bi, Yue Liu, Hongyu Chen, Li Liang, Li Su, and Qingming Huang. 2025. Exploring hallucination of large multimodal models in video understanding: Benchmark, analysis and mitigation. arXiv preprint arXiv:2503.19622. Google. 2025.Gemini 3 developer guide (gemini api): Gemini 3 flash.https://ai.google.dev/ gemini-api/docs/gemini-3. Accessed 2026-08- 24. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, and 1 others. 2022. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18995–19012. Hyeonjeong Ha, Jinjin Ge, Bo Feng, Kaixin Ma, and Gargi Chakraborty. 2026. Narrativetrack: Evaluating entity-centric reasoning for narrative understanding. arXiv preprint arXiv:2601.01095. Matthew Honnibal, Ines Montani, Sofie Van Lan- deghem, and Adriane Boyd. 2020. spacy: Industrial- strength natural language processing in python. Brian C Hosler, Xinwei Zhao, Owen Mayer, Chen Chen, James A Shackleford, and Matthew C Stamm. 2019. The video authentication and camera identification database: A new database for video forensics. IEEE access, 7:76937–76948. Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Xi- ang Yue, Bo Li, Yuanhan Zhang, and Ziwei Liu. 2026. Video-mmmu: Evaluating knowledge acquisi- tion from multidisciplinary professional videos. In Proceedings of the 64th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 27798–27828. Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. 2014. Large-scale video classification with convolu- tional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recogni- tion, pages 1725–1732. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An open source language model specialized in evaluating other language mod- els. In Proceedings of the 2024 Conference on Empir- ical Methods in Natural Language Processing, pages 4334–4353. Fanheng Kong, Jingyuan Zhang, Hongzhi Zhang, Shi Feng, Daling Wang, Linhao Yu, Xingguang Ji, Yu Tian, Fuzheng Zhang, and 1 others. 2025. Tuna: Comprehensive fine-grained temporal understanding evaluation on dense dynamic videos. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1810–1839. Kyuho Lee, Euntae Kim, Jinwoo Choi, and Buru Chang. 2025. Noah: Benchmarking narrative prior driven hallucination and omission in video large language models. arXiv preprint arXiv:2511.06475. Baiqi Li, Kangyi Zhao, Ce Zhang, Chancharik Mi- tra, Jean de Dieu Nyandwi, and Gedas Bertasius. 2026. Timeblind: A spatio-temporal composition- ality benchmark for video llms.arXiv preprint arXiv:2602.00288. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and 1 others. 2024a. Llava- onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326. Yanwei Li, Chengyao Wang, and Jiaya Jia. 2024b. Llama-vid: An image is worth 2 tokens in large lan- guage models. In European conference on computer vision, pages 323–340. Springer. Zhengqi Li, Tali Dekel, Forrester Cole, Richard Tucker, Noah Snavely, Ce Liu, and William T Freeman. 2020. Mannequinchallenge: Learning the depths of moving people by watching frozen people. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 43(12):4229–4241. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892– 34916. Zhihang Liu, Chen-Wei Xie, Bin Wen, Feiwu Yu, Pan- deng Li, Boqiang Zhang, Nianzu Yang, Zuan Gao, Yun Zheng, and Hongtao Xie. 2026. Capability: A comprehensive visual caption benchmark for evaluat- ing both correctness and thoroughness. Advances in Neural Information Processing Systems, 38. Wentao Ma, Weiming Ren, Yiming Jia, Zhuofeng Li, Ping Nie, Ge Zhang, and Wenhu Chen. 2025. Videoeval-pro: Robust and realistic long video understanding evaluation.arXiv preprint arXiv:2505.14640. Ruchit Rawal, Reza Shirkavand, Heng Huang, Gowthami Somepalli, and Tom Goldstein. 2025. Ar- gus: Hallucination and omission evaluation in video- llms. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 20280–20290. IEEE. Kuniaki Saito, Risa Shinoda, Shohei Tanaka, Tosho Hirasawa, Fumio Okura, and Yoshitaka Ushiku. 2025.Haldec-bench: Benchmarking hallucina- tion detector in image captioning. arXiv preprint arXiv:2511.20515. Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. 2022. Assembly101: A large-scale multi-view video dataset for understanding procedural activi- ties. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21064– 21074. IEEE. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, and 1 oth- ers. 2025. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267. Lars St, Svante Wold, and 1 others. 1989. Analysis of variance (anova). Chemometrics and intelligent laboratory systems, 6(4):259–272. Waqas Sultani, Chen Chen, and Mubarak Shah. 2018. Real-world anomaly detection in surveillance videos. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6479–6488. IEEE. Xichen Tan, Yuanjing Luo, Yunfan Ye, Fang Liu, and Zhiping Cai. 2025. Allvb: All-in-one long video un- derstanding benchmark. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 7211–7219. Changli Tang, Yixuan Li, Yudong Yang, Jimin Zhuang, Guangzhi Sun, Wei Li, Zejun Ma, and Chao Zhang. 2025a. video-salmonn 2: Caption-enhanced audio- visual large language models.arXiv preprint arXiv:2506.15220. Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, and 1 others. 2025b. Video un- derstanding with large language models: A survey. IEEE Transactions on Circuits and Systems for Video Technology, 36(2):1355–1376. MaximTkachenko,MikhailMalyuk,Andrey Holmanyuk,andNikolaiLiubimov.2020- 2025.Label Studio:Data labeling soft- ware.Open source software available from https://github.com/HumanSignal/label-studio. Jiahao Wang, Yufeng Yuan, Rujie Zheng, Youtian Lin, Jian Gao, Lin-Zhuo Chen, Yajie Bao, Chang Zeng, Yanxi Zhou, Xiao-Xiao Long, and 1 others. 2026. Spatialvid: A large-scale video dataset with spatial annotations. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 42592–42603. Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xi- aohan Zhang, Ji Qi, Ming Ding, Xiaotao Gu, Shiyu Huang, Bin Xu, and 1 others. 2025a. Lvbench: An extreme long video understanding benchmark. In 2025 IEEE/CVF International Conference on Com- puter Vision (ICCV), pages 22958–22967. IEEE. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, and 1 others. 2025b. In- ternvl3. 5: Advancing open-source multimodal mod- els in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and 1 others. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38–45. Peng Wu, Jing Liu, Yujia Shi, Yujia Sun, Fangtao Shao, Zhaoyang Wu, and Zhiwei Yang. 2020. Not only look, but also listen: Learning multimodal violence detection under weak supervision. In European con- ference on computer vision, pages 322–339. Springer. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Biao Yang, Bin Wen, Boyang Ding, Changyi Liu, Chenglong Chu, Chengru Song, Chongling Rao, Chuan Yi, Da Li, Dunju Zang, and 1 others. 2025b. Kwai keye-vl 1.5 technical report. arXiv preprint arXiv:2509.01563. Appendix A Attribution of Icons The icons used in the figures are sourced from Flati- con (https://w.flaticon.com/). We grate- fully acknowledge the creators for providing these visual assets, which are utilized in accordance with their licensing terms to enhance the clarity of our technical illustrations. B Construction of VidOmni-Bench B.1 Video Collection More details of video collection. Before collect- ing videos for VidOmni-Bench, we first define a taxonomy that categorizes videos according to the complexity of their spatio-temporal dynamics. The taxonomy consists of five complexity types, where each type progressively introduces an additional source of difficulty: motion, fine-grained action, multi-agent binding, and scene transitions. This design results in five complexity types. Static World videos contain little or no temporal change, so they can largely be understood from static visual content. Body Motion videos intro- duce temporal changes caused by human or object motion, such as walking, running, or turning. Hand- Object Interaction videos further require under- standing fine-grained local actions, such as grasp- ing, placing, or manipulating an object. Crowd Action videos involve multiple agents acting simul- taneously, where the model must correctly asso- ciate each action with the corresponding subject or object. Finally, Multi-Scene Dynamics videos contain shot changes or multiple scenes, requir- ing the model to connect events across temporally discontinuous segments. We note that these types are not intended to be mutually exclusive definitions of intrinsic video complexity. Instead, our taxonomy is designed as a controlled diagnostic dimension, where each type emphasizes a different source of spatio-temporal difficulty. As shown in Figure 16, this taxonomy provides a useful axis for examining how Video- LLMs behave under different video dynamics. Ta- bles 4 and 5 summarize the complexity taxonomy and the public video sources used for each type. ComplexityMotion Fine-grained Action Multi-Agent Binding Scene Transitions Type 1✗ Type 2✓✗ Type 3✓✗ Type 4✓✗ Type 5✓ Table 4: Complexity taxonomy of VidOmni-Bench. ComplexityName Video Source (Public Dataset) Type 1Static World Video-ACID (Hosler et al., 2019) SpatialVID (Wang et al., 2026) Mannequin Challenge (Li et al., 2020) Type 2Body Motion ActivityNet (Caba Heilbron et al., 2015) Type 3Hand-Object Interaction Assembly101 (Sener et al., 2022) Epic-Kitchens (Damen et al., 2018) Ego4D (Grauman et al., 2022) Type 4Crowd Action UCF-Crime (Sultani et al., 2018) Sports-1M (Karpathy et al., 2014) Type 5Multi-Scene Dynamics Video-MME (Fu et al., 2025) XD-Violence (Wu et al., 2020) Table 5: Data sources of each complexity type in VidOmni-Bench. Videos are primarily collected from public datasets. We further supplement underrepre- sented types and duration ranges with videos sourced from YouTube. B.2 Dense Video Caption Generation Captioning setups. Table 6 summarizes the Video- LLMs used for caption generation. Open-source models are deployed with official implementations and publicly available weights. Input Frames Per Second (FPS) and the maximum number of in- put frames are set following reported configura- tions from Video-MME (Fu et al., 2025), Video- MMMU (Hu et al., 2026), and LVBench (Wang et al., 2025a) whenever available. Otherwise, we carefully set configurations that cover as much video content as possible within each model’s con- text length. We fix the generation seed for repro- ducibility. For the Video-LLMs used for dense captioning, we randomly select one of the instruc- tions in Figure 13. ModelProviderOpen/ClosedFPSMax FramesRelease Gemini 3 FlashGoogleClosed1-2026/02 Qwen3-VL 32BAlibabaOpen220482026/01 InternVL 3.5 38BOpenGVLabOpen1322025/12 video-SALMONN 2+ 72BByteDanceOpen17682025/11 Keye-VL 1.5 8BKwaiOpen110242025/03 Table 6: Details of Video-LLMs chosen as caption- ing engines. We use five Video-LLMs with diverse providers, accessibility, FPS, and maximum number of frames (Max Frames) to generate dense captions. Instruction given to Video- LLMs in captioning 1.Describe this video in detail, focusing on the spatio-temporal dynamics. Describe exactly how objects and agents move, change, and occupy space over time within the scene. 2. Give a detailed account of everything shown in the video, capturing all visible specifics. Describe events in the exact order they appear over time. Ensure that you describe the sequence of events exactly as they occur, without skipping any steps. 3.Describe the video in detail, paying special atten- tion to how objects and people interact with each other. Capture the precise timing and nature of every contact, movement, and reaction shown in the footage. 4.Thoroughly describe the video’s visual narrative, capturing every visible detail from start to end. Emphasize how actions unfold and how the scene transitions logically over time. 5.Provide a thorough description of every detail, explicitly prioritizing spatio-temporal dynamics. Capture all visual elements, and focus on their spatial positions, movement trajectories, and how the scene evolves over time. Figure 13: Prompt templates for dense video caption- ing. We use five prompt templates to generate dense cap- tions for each video, reducing the potential instruction bias that may arise from relying on a single captioning prompt. B.3 Sentence-wise Annotation Annotator. We recruit five annotators specifically for our multi-stage labeling task, which requires meticulous reading and temporal reasoning to iden- tify hallucinated sentences within dense captions. All candidates are required to demonstrate high- level English proficiency and the ability to detect subtle spatio-temporal inconsistencies. These qual- ifications are rigorously verified through a pilot annotation task involving 10 videos prior to the main annotation phase. Voting and quality control. The authors review all pilot annotations, and only those who demon- strated high accuracy in identifying hallucinated sentences are designated as trusted annotators for the main task. Each trusted annotator is assigned between 500 and 1,000 video-caption pairs. To ensure objective and robust annotation, each pair is independently evaluated by at least two annota- tors, including a minimum of one trusted annotator. Once at least two annotations are collected for a sentence, the final label is determined through ma- jority voting. Furthermore, the authors conduct sys- tematic audits for every batch of 100 video-caption pairs. If a batch fails to meet our criteria, annota- tors are required to re-evaluate the entire set before proceeding. Inter-annotator agreement. We evaluate inter- annotator agreement on VidOmni-Bench using Fleiss’ kappa (Fleiss, 1971) with labels (Correct, In- correct, Unknown). The overall agreement reaches κ= 0.50 with an observed agreement ofP bar = 0.764. While these values indicate a moderate level of agreement, we prioritize data integrity through a rigorous manual adjudication process. To ensure the absolute reliability of the benchmark, authors meticulously review and resolve all disputed labels and cases with low confidence, establishing a high- quality ground truth for subsequent evaluation. Annotation interface. We customize the Label Studio (Tkachenko et al., 2020-2025) interface for our two-stage annotation pipeline. Figure 17 shows the interface for sentence-wise correctness annota- tion (Stage 1), while Figure 18 shows the interface for word-level error category annotation (Stage 2). Word-level error categories. Table 7 provides detailed descriptions for each error categories used during our annotation process. For every sentence labeled as Incorrect, annotators are instructed to pinpoint the specific errors at the word level and classify them according to these categories. B.4 Edge Case Handling Guidelines Due to the length and complexity of generated cap- tions, we establish internal guidelines to maintain consistency, particularly for ambiguous or repet- itive sentences that are difficult to classify using our error categories. These guidelines function as a decision rules to help annotators navigate edge cases systematically. Handling repetitive sentences. A common fail- ure mode in Keye-VL 1.5 (Yang et al., 2025b) and video-SALMONN 2+ (Tang et al., 2025a) is that a model repeatedly generates the same or simi- lar sentences. In such cases, we evaluate the first occurrence based on its correctness. If the video progresses to a new event but the model continues to repeat the previous sentence, all subsequent rep- etitions are labeled Incorrect and assigned the Fre- quency error. This ensures the annotation reflects the model’s inability to track temporal progression. TypeDescriptionExample ObjectMisdescriptions of the object itself including its attributes (color, text, count, etc.). There are two blue cars, but described as “three red cars.” RelationMisdescriptions of spatial relationships be- tween objects. A cup is under the table, but described as “on the table.” LocationMisdescriptions of an object’s position. A cup is on the right side, but described as “on the left.” DirectionMisdescriptions of orientation or direction of an object. A person is looking to the right, but described as “looking to the left.” Illusion Mentioning non-existent objects, events, or actions. No bird is present, but described as “a bird flying in the sky.” ActionIncorrect or incomplete description of ac- tions over time. A person is walking, but described as “running.” OrderMisdescriptions of the temporal order of events or actions. Sat down after opening a door, but described as “door opened after sitting down.” CausalityMisdescriptions of causal relationships among events. Describing that rain was caused by people opening umbrellas. FrequencyIncorrect tracking of the frequency of re- peated actions. A person jumped once, but described as “jumped three times.” State ChangeMisdescribing an object’s state transition.Traffic light changed from red to yellow, but de- scribed as “yellow to red.” Table 7: Word-level error categories in VidOmni-Bench. We group word-level errors into five static-frame- related categories(Object, Relation, Location, Direction, and Illusion) and five temporal-understanding-related categories(Action, Order, Causality, Frequency, and State Change). Distinguishing temporal errors. We provide ex- plicit instructions to help annotators distinguish events that are temporally misplaced from those that are entirely hallucinated. For example, if a model describes an event that does not occur at the specified timestamp but happens elsewhere in the video, it is labeled as an Order error rather than an Illusion error. This distinction is crucial for diagnosing whether a model suffers from to- tal hallucination or merely temporal shuffling of frames. Resolving visual ambiguity. In cases where the vi- sual evidence is too ambiguous to make a definitive judgment, annotators utilize the Unknown category or seek final adjudication through discussions with the authors. This collaborative feedback loop en- sures that the final dataset remains a reliable ground truth for fine-grained video understanding. C Details of Experimental Setups C.1 Details of Evaluation on VidOmni-Bench Source of models. For open-weight models, we use checkpoints available on Hugging Face (Wolf et al., 2020) and base our code on the Hugging Face Transformers package. For GPT 5 mini and Gemini 3 Flash, we use their official APIs. Prompt for sentence-wise event verification. To ensure a fair and consistent evaluation, we apply a unified chat template across all evaluated Video- LLMs. The full prompt template is shown in Fig- ure 14. For each inference, the prompt remains identical except for the specific target caption pro- vided for sentence-wise verification. Computation. At most four H200 GPUs are used for inference of a single model. Evaluation configs. As in our captioning pipeline, we set FPS and the maximum number of input frames following each benchmark’s reported set- tings whenever available, and otherwise sample as many frames as possible up to each model’s native context limit, as detailed in Table 8. ModelProviderOpen/ClosedFPSMax FramesRelease Gemini 3 FlashGoogleClosed1-2026/02 GPT 5 miniOpenAIClosed-2562026/03 Qwen3-VL 32BAlibabaOpen220482026/01 InternVL 3.5 38BOpenGVLabOpen17682025/12 Video-SALMONN 2+ 72BByteDanceOpen1322025/11 Keye-VL 1.5 8BKwaiOpen110242025/03 Table 8:Details of Video-LLMs evaluated on VidOmni-Bench. We list the provider, accessibility, FPS, and maximum number of input frames (Max Frames) for each Video-LLM evaluated on VidOmni- Bench. D Additional Analysis D.1 Caption Analysis in VidOmni-Bench Details for Certificate Coverage. We follow Video-MME (Fu et al., 2025), which defines Certifi- cate Length as the minimum video span required to solve a task. For sentence-wise event verification, we annotate the minimum video intervals needed Instruction given to Video- LLMs in sentence-wise verifi- cation You are an expert in Video Hallucination Detection. You are given a video and a caption that consists of multiple sentences. Your task is to evaluate each sentence individually to determine if it contains a hallucination based on the visual evidence. For each sentence, assign a binary label (0 or 1) ac- cording to the following definition: - Label 1: The sentence contains a hallucination. This means it describes something not present or incorrect regarding the video. - Label 0: The sentence is faithful to the video. This means it correctly describes the visual content. Please first output the dictionary of labels before ex- plaining the reason for your judgment. Output format requirements: 1) First line must be a valid JSON dictionary with keys sentence_1, sentence_2, ... and values 0 or 1. 2) Then explain the reason for each sentence. Caption: Figure 14: Sentence-wise event verification instruc- tion. The prompt asks Video-LLMs to assign a binary label to each caption sentence before providing sentence- level reasoning. to determine whether each sentence is supported by the video. Certificate Coverage is then computed as the ratio of the annotated Certificate Length to the full video duration. Since annotating Certificate Length is highly labor-intensive, the Certificate Coverage reported in Figure 7 and Table 2 is com- puted on a randomly sampled subset of 30 videos for each benchmark. Captions of VidOmni-Bench are more complex than other benchmarks. We compare the informa- tiveness of captions in VidOmni-Bench with those from ARGUS (Rawal et al., 2025), NOAH (Lee et al., 2025), and CAPAbility (Liu et al., 2026). We first conduct a lexical and syntactic analysis using spaCy (Honnibal et al., 2020). As shown in Table 10, captions in VidOmni-Bench contain substantially more words, information units, noun chunks, and verbs than captions from prior bench- marks. In addition to these surface-level statistics, we further assess caption informativeness through a pairwise LLM-as-a-judge evaluation follow- ing Prometheus-2 (Kim et al., 2024).Given the predefined informativeness rubric and prompt shown in Figure 15, the judge compares each VidOmni-Bench caption with a caption from an- other benchmark and selects the more informative one. VidOmni-Bench captions win in 96.0–99.6% of pairwise comparisons, suggesting that they pro- vide richer and more detailed event descriptions for sentence-wise event verification. D.2 Additional Experiments on VidOmni-Bench VidOmni-Bench remains challenging beyond frame sampling. Table 9 reports type-wise F1- Scores of Qwen3-VL 32B under different FPS from 1 to 10. Overall, F1-Scores show no consistent im- provement as FPS increases, indicating that poor performances in Table 3 are not primarily driven by insufficient frame sampling. Video-LLMFPSType 1Type 2Type 3Type 4Type 5 Qwen3-VL 32B 127.3216.2122.1837.0825.22 215.1914.4410.2533.3621.22 418.3815.9931.7537.7122.24 818.3820.0920.1429.0424.63 1018.3823.0619.7428.4924.02 Table 9: Effect of frame sampling rate on Qwen3-VL 32B performance across complexity types. Aggregated performance across all complexity levels and durations. Figure 16 reports the F1- Score of each Video-LLM across the complexity and duration axes of VidOmni-Bench. The results show that Video-LLM performance varies substan- tially across different combinations of video dy- namics and temporal scale. This reveals multidi- mensional weaknesses, where models do not fail uniformly but instead exhibit different vulnerabili- ties depending on the required visual reasoning and temporal coverage. YouTube leakage analysis. Table 12 compares model performance on the full VidOmni-Bench test set and a subset excluding YouTube-sourced videos. F1-Scores remain consistent across the two settings, indicating that the results in Table 3 are not substantially driven by YouTube-sourced videos. This suggests that potential YouTube data leakage or source-specific bias has limited impact on the reported performances. Ensemble can mitigate self-preference bias in self-evaluation. As shown in Table 11, cross- model ensemble substantially improves F1-Scores under self-evaluation settings. In this strategy, a Benchmark Avg. Words Info Count Noun Chunks Verbs LLM Judge Win Rate (%) (vs. Ours) ARGUS (Rawal et al., 2025)477.5336.4134.363.14.0 NOAH (Lee et al., 2025)48.034.213.88.00.6 CAPAbility (Liu et al., 2026)3.22.41.10.30.4 VidOmni-Bench578.9400.6159.382.1– Table 10: Comparison of caption informativeness across benchmarks. VidOmni-Bench achieves substantially longer captions, higher information counts, and more noun chunks and verbs than prior benchmarks. The last column reports the percentage of pairwise LLM-as-a-judge comparisons in which the baseline caption is preferred over the VidOmni-Bench caption in terms of informativeness. Video-LLMEnsemble Verifier F1-Score (%) Self-VerificationCross-Ensemble (+p.p) Qwen3-VL 32B+ InternVL3.5 38B5.326.2 (+20.9) InternVL3.5 38B+ Keye-VL 1.5 8B7.724.6 (+16.9) Gemini 3 Flash+ GPT 5 mini21.925.1 (+3.2) Table 11: Effect of cross-model ensemble on self-verification. Cross-model ensemble verification consistently improves F1-Score over self-verification across caption sources. These gains show that combining judgments from other Video-LLMs can compensate for the low verification performance observed when models evaluate their own captions. generated sentence is marked as Incorrect if either the captioning model or an external verifier pre- dicts it as Incorrect. The consistent gains suggest that self-evaluation failures are largely caused by missed hallucinations, and that external model ver- ification can partially mitigate this self-bias. VidOmni-Bench SubsetVideo-LLMF1-Score (%) Full Test Set Gemini 3 Flash41.69 GPT 5 mini28.93 Qwen3-VL 32B21.74 w/o YouTube Videos Gemini 3 Flash42.46 GPT 5 mini29.15 Qwen3-VL 32B23.21 Table 12: YouTube data leakage analysis. The similar F1-Scores across the two settings suggest that the re- ported results are not driven by leakage from YouTube. Analysis of audio modality in verification. To investigate the impact of audio information on sentence-wise event verification, we conduct an ANOVA (St et al., 1989) across three audio cat- egories defined by the type of content they con- vey: Speech (meaningful spoken content), Sound Effect (semantically relevant non-speech sounds), and Background (uninformative audio). As shown in Table 13, for Gemini 3 Flash, audio input sig- nificantly improves F1-Score in both Speech and Sound Effect, while causing no significant change in the Background. In contrast, F1-Scores of video- SALMONN 2+ decline across these categories. These results suggest that while Gemini effectively leverages audio as a complementary signal, video- SALMONN 2+ tends to treat audio as a distractor. Prompt for pairwise caption informativeness judgment Task Description: Compare Response A and Response B using the rubric. Write one short feedback sentence, then end with exactly one result tag. The final line must be exactly either: [RESULT] A [RESULT] B Instruction: Describe this video in detail, focusing on the spatio-temporal dynamics. Describe exactly how objects and agents move, change, and occupy space over time within the scene. Response A: response_A Response B: response_B Rubric: Does the model’s response provide a sufficient amount of relevant and substantive information for the given query? Score 1: The response is severely lacking in information. It is vague, overly generic, or omits essential details, leaving the query largely unaddressed. Score 2: The response provides minimal information. It touches on the topic but lacks depth, with significant gaps or important aspects left uncovered. Score 3: The response provides a moderate amount of information. It covers the core of the query but may lack depth in some areas or omit secondary details that would add value. Score 4: The response is informative and substantive. It thoroughly addresses the query with relevant details, with only minor opportunities for additional depth or coverage. Score 5: The response is highly informative and comprehensive. It fully addresses the query with rich, relevant, and well-elaborated detail, leaving little need for follow-up. Feedback: Figure 15: Prompt for pairwise caption informativeness judgment. We use Prometheus-2 7B to compare captions from VidOmni-Bench and other benchmarks under a predefined informativeness rubric. Given the captioning instruction, two captions, and the rubric, the judge selects the more informative caption. Video-LLMAudio CategoryF1-Score Gain (%) tSignificant Gemini 3 Flash Speech+3.3+3.9*** Sound Effect+4.2+3.6*** Background−0.8 −0.9n.s. video-SALMONN 2+ 7B Speech−0.3 −1.7n.s. Sound Effect−1.5 −2.5* Background−0.9 −2.7** video-SALMONN 2+ 72B Speech−2.2 −4.2*** Sound Effect−2.4 −2.6* Background−1.6 −2.6** Table 13: F1-Score gain when audio is added by audio category and model. For each model and audio category, we compute the F1-Score gain (with audio - without audio) and test whether this gain is significantly different from zero using a one-sample t-test. *p < .05, **p < .01, ***p < .001, n.s.: not significant p≥ .05. ShortMedium Long 30 40 50 F1-Score (%) Gemini 3.0 Flash (v) Type 1 ShortMedium Long 30 40 50 Type 2 ShortMedium Long 30 40 50 Type 3 ShortMedium Long 30 40 50 Type 4 ShortMedium Long 30 40 50 Type 5 ShortMedium Long 20 30 40 50 60 F1-Score (%) Gemini 3.0 Flash (v, a) ShortMedium Long 20 30 40 50 60 ShortMedium Long 20 30 40 50 60 ShortMedium Long 20 30 40 50 60 ShortMedium Long 20 30 40 50 60 ShortMedium Long 10 20 30 F1-Score (%) Qwen3-VL 32B ShortMedium Long 10 20 30 ShortMedium Long 10 20 30 ShortMedium Long 10 20 30 ShortMedium Long 10 20 30 ShortMedium Long 5 10 15 20 F1-Score (%) InternVL3.5 38B ShortMedium Long 5 10 15 20 ShortMedium Long 5 10 15 20 ShortMedium Long 5 10 15 20 ShortMedium Long 5 10 15 20 ShortMedium Long 0 10 20 F1-Score (%) KeyeVL 1.5 8B ShortMedium Long 0 10 20 ShortMedium Long 0 10 20 ShortMedium Long 0 10 20 ShortMedium Long 0 10 20 ShortMedium Long 0 5 10 15 F1-Score (%) video SALMONN 2+ 72B (v) ShortMedium Long 0 5 10 15 ShortMedium Long 0 5 10 15 ShortMedium Long 0 5 10 15 ShortMedium Long 0 5 10 15 ShortMedium Long Duration 0 5 10 F1-Score (%) video SALMONN 2+ 72B (v, a) ShortMedium Long Duration 0 5 10 ShortMedium Long Duration 0 5 10 ShortMedium Long Duration 0 5 10 ShortMedium Long Duration 0 5 10 Figure 16: Performances of Video-LLMs across all complexity types and durations. The results reveal multidimensional and model-specific weaknesses, indicating that current Video-LLMs exhibit different failure patterns depending on both video dynamics and temporal scale. Figure 17: Annotation interface for sentence-wise correctness (Stage 1). Annotators label each caption sentence as Correct, Incorrect, or Unknown based on the video evidence. An additional memo field is provided to collect detailed notes and ambiguous cases during dataset construction. Figure 18: Annotation interface for word-level error category (Stage 2). For each sentence labeled as Incorrect in Stage 1, annotators identify the erroneous words or phrases and assign them to predefined error categories, enabling fine-grained analysis of hallucination types.