Paper deep dive
Long-CODE: Isolating Pure Long-Context as an Orthogonal Dimension in Video Evaluation
Zhijiang Tang, Jiaxin Qi, Bing Zhao, Jianqiang Huang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 4/27/2026, 7:29:06 AM
Summary
The paper introduces Long-CODE, a novel evaluation framework and benchmark designed to assess long-context attributes in video generation models. The authors argue that long-video evaluation (focusing on narrative richness and global causal consistency) is an orthogonal dimension to short-video evaluation (focusing on frame-level quality). Long-CODE utilizes two complementary components: Dynamic Structure Alignment (DSA), which uses Spearman rank correlation of shot-level embeddings to measure structural consistency, and an MLLM-based reasoning pipeline for deep semantic analysis. The framework is validated through a corruption testing framework (Shuffle, Replace, Edition, Synthesis) demonstrating that existing short-video metrics are insensitive to long-range structural perturbations.
Entities (12)
Relation Signals (9)
Long-CODE → comprises → Dynamic Structure Alignment
confidence 100% · The final Long-CODE metric integrates both dimensions: M_Long-CODE = alpha * M_DSA + (1-alpha) * M_MLLM
Long-CODE → introduces → Dynamic Structure Alignment
confidence 100% · we introduce Long-CODE... We first introduce a suite of long-video attribute corruption tests... we design a novel long-video metric based on shot dynamics...
VBench → isa → Short-video Metric
confidence 100% · Traditional metrics are intrinsically tailored for short-video evaluation... such as VBench
Long-CODE → uses → MLLM
confidence 100% · we integrate Multimodal Large Language Models (MLLMs) to perform “deep thinking” and provide qualitative scoring.
Dynamic Structure Alignment → uses → Spearman rank correlation
confidence 100% · We utilize the Spearman rank correlation coefficient (Spearman, 1961) to evaluate the alignment
Long-CODE → uses → MLLM
confidence 100% · we integrate Multimodal Large Language Models (MLLMs) to perform “deep thinking” and provide qualitative scoring.
Long-CODE → complements → VBench
confidence 90% · Ultimately, our metric and benchmark seamlessly complement existing short-video standards
Long-CODE → evaluates →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:As video generation models achieve unprecedented capabilities, the demand for robust video evaluation metrics becomes increasingly critical. Traditional metrics are intrinsically tailored for short-video evaluation, predominantly assessing frame-level visual quality and localized temporal smoothness. However, as state-of-the-art video generation models scale to generate longer videos, these metrics fail to capture essential long-range characteristics, such as narrative richness and global causal consistency. Recognizing that short-term visual perception and long-context attributes are fundamentally orthogonal dimensions, we argue that long-video metrics should be disentangled from short-video assessments. In this paper, we focus on the rigorous justification and design of a dedicated framework for long-video evaluation. We first introduce a suite of long-video attribute corruption tests, exposing the critical limitations of existing hort-video metrics from their insensitivity to structural inconsistencies, such as shot-level perturbations and narrative shuffling. To bridge this gap, we design a novel long-video metric based on shot dynamics, which is highly sensitive to the long-range testing framework. Furthermore, we introduce Long-CODE (Long-Context as an Orthogonal Dimension for video Evaluation), a specialized dataset designed to benchmark long-video evaluation, with human annotations isolated specifically to genuine long-range characteristics. Extensive experiments show that our proposed metrics achieve state-of-the-art correlation with human judgments. Ultimately, our metric and benchmark seamlessly complement existing short-video standards, establishing a holistic and unbiased evaluation paradigm for video generation models.
Tags
Links
- Source: https://arxiv.org/abs/2604.17428v1
- Canonical: https://arxiv.org/abs/2604.17428v1
Trouble viewing inline? Open PDF directly →
Full Text
48,351 characters extracted from source content.
Expand or collapse full text
Long-CODE: Isolating Pure Long-Context as an Orthogonal Dimension in Video Evaluation Zhijiang Tang tangzhijiang24@mails.ucas.ac.cn Hangzhou Institute for Advanced Study, UCASChina , Jiaxin Qi jxqi@cnic.cn Computer Network Information Center, CASChina , Bing Zhao zhaobingchs@gmail.com Department of AI Infrastructure, Bilibili Inc.China and Jianqiang Huang jqhuang@cnic.cn Computer Network Information Center, CASChina Hangzhou Institute for Advanced Study, UCASChina (5 June 2009) Abstract. As video generation models achieve unprecedented capabilities, the demand for robust video evaluation metrics becomes increasingly critical. Traditional metrics are intrinsically tailored for short-video evaluation, predominantly assessing frame-level visual quality and localized temporal smoothness. However, as state-of-the-art video generation models scale to generate longer videos, these metrics fail to capture essential long-range characteristics, such as narrative richness and global causal consistency. Recognizing that short-term visual perception and long-context attributes are fundamentally orthogonal dimensions, we argue that long-video metrics should be disentangled from short-video assessments. In this paper, we focus on the rigorous justification and design of a dedicated framework for long-video evaluation. We first introduce a suite of long-video attribute corruption tests, exposing the critical limitations of existing short-video metrics from their insensitivity to structural inconsistencies, such as shot-level perturbations and narrative shuffling. To bridge this gap, we design a novel long-video metric based on shot dynamics, which is highly sensitive to the long-range testing framework. Furthermore, we introduce Long-CODE (Long-Context as an Orthogonal Dimension for video Evaluation), a specialized dataset designed to benchmark long-video evaluation, with human annotations isolated specifically to genuine long-range characteristics. Extensive experiments show that our proposed metrics achieve state-of-the-art correlation with human judgments. Ultimately, our metric and benchmark seamlessly complement existing short-video standards, establishing a holistic and unbiased evaluation paradigm for video generation models. Long Video Evaluation, Human Consistency †copyright: acmlicensed†journalyear: 2026†doi: X.X†conference: Proceedings of the 34th ACM International Conference on Multimedia; November 10-14, 2026; Rio de Janeiro, Brazil†isbn: 978-1-4503-X-X/2018/06†submissionid: 4882†ccs: Computing methodologies Computer vision 1. Introduction Figure 1. Illustrations of our proposed Long-CODE benchmark. (a) Comparison between the short video metric VBench (Huang et al., 2024b) and our proposed metric, Long-CODE. The "Corruption" test involves shuffling the original shot order of a long video depicting the folklore of Newton discovering gravity (Brewster, 2010). (b) Performance benchmarking of cutting-edge long video generation models, including OpenAI’s Sora 2 (Liu et al., 2024b), Google’s Veo 3.1 (Google, 2024), ByteDance’s StoryMem (Zhang et al., 2025), Alibaba’s Wan 2.2 (Wan et al., 2025), and HKUST’s HoloCine (Meng et al., 2025) and VGoT (Zheng et al., 2024a). Please refer to Section 4 for the generation pipeline. Recent advancements in Video Generation Models (VGMs) have yielded unprecedented visual fidelity and realism (Liu et al., 2024b; Google, 2024; Wan et al., 2025). To systematically evaluate the synthesis capabilities of these models, various metrics and benchmarks have been established (Li et al., 2025), such as VBench (Huang et al., 2024b) for frame-level visual quality, VideoCLIP (Wang et al., 2024) for text-to-video semantic alignment, and LocoT2V-Bench (Zheng et al., 2025b) for localized temporal smoothness. However, existing metrics are intrinsically tailored for short-video evaluation, typically spanning only a few seconds (Huang et al., 2024b; Liu et al., 2024a). As state-of-the-art VGMs scale to generate minute-level content (Zhang et al., 2025; Zheng et al., 2024a), evaluating video quality necessitates dimensions far beyond short-term perception. Specifically, long-video generation introduces additional higher-order attributes, such as narrative richness and long-term causal consistency. For example, whether an object that exists in the camera view logically reappears later with consistent physical properties. These long-range cognitive and structural evaluations fundamentally exceed the functional scope of existing short-video metrics, which are inherently limited to capturing adjacent frame variances or localized semantic distances. As illustrated in Figure 1(a), applying temporal corruption to a long video may preserve the visual quality and local smoothness of individual shots. Consequently, traditional short-video metrics, such as VBench, fail to register a decline in performance. However, such a video is fundamentally flawed for long-form content because its underlying narrative structure and causal logic have been destroyed. To bridge this gap, we argue that long-video attributes and short-video visual perception are fundamentally orthogonal dimensions. Traditional metrics remain valid and necessary for guaranteeing the foundational visual quality of long videos, which can be easily achieved through sliding-window or aggregation strategies. Therefore, what the community urgently requires is a dedicated evaluation paradigm that is maximally decoupled from baseline visual quality, focusing exclusively on the emergent characteristics of long-context generation. To this end, we introduce Long-CODE (Long-Context as an Orthogonal Dimension in Video Evaluation), a novel framework designed to seamlessly complement existing standards and establish a holistic evaluation ecosystem. In this paper, we present a systematic approach to long-video evaluation. First, we formally define a long-context video as an organic sequence of multiple interconnected shots, where a traditional short video is merely a single constituent element of this set. Furthermore, we provide a theoretical justification demonstrating that long-video attributes are orthogonal to short-video attributes, necessitating a distinct evaluative dimension. Based on this formulation, we design a suite of long-video attribute-corruption tests (e.g., shot shuffling and replacement) and empirically demonstrate the critical insensitivity of existing short-video metrics to these long-range structural perturbations, thereby justifying their limitations for long-range video evaluation. Second, we propose Dynamic Structure Alignment, a novel evaluation metric specifically designed to capture long-range dependencies and temporal dynamics across multiple shots. Within this framework, we define structural similarity vectors to represent the inter-shot relationships. We utilize the Spearman rank correlation coefficient (Spearman, 1961) to evaluate the alignment between the generated video vector and the corresponding prompt vector. To complement the assessment of long-video semantics, we integrate Multimodal Large Language Models (MLLMs) to perform “deep thinking” and provide qualitative scoring. The final evaluation score is derived from the fusion of these two distinct sub-metrics. Finally, we construct the Long-CODE benchmark, a specialized dataset comprising complex, multi-shot prompts. Specifically, we leverage an MLLM to decompose a complex storyline into a sequence of structured shots, thereby yielding the individual shot prompts. We also introduce a rigorous human evaluation protocol strictly isolated to gauge genuine long-range characteristics. For instance, during the annotation process, human evaluators are instructed to prioritize narrative fluidity over image quality, ensuring the benchmark only reflects true long-context performance. Extensive experiments validate the effectiveness of our proposed framework. Our metric exhibits high sensitivity in the designed corruption tests, yielding orthogonal results to standard short-video metrics, validating its unique utility. Furthermore, evaluated on the Long-CODE benchmark, our proposed metric achieves state-of-the-art correlation with human judgments specifically targeted at long-video quality. Ultimately, our metric and benchmark serve as a robust complement to existing short-video standards, offering a comprehensive and unbiased evaluation paradigm for current video generation models. We summarize our primary contributions as follows: (1) We critically analyze the limitations of current video evaluation paradigms, demonstrating their systematic failure to capture long-range video. By establishing the orthogonality between short-term visual perception and long-term contextual attributes, we advocate for a decoupled evaluation framework and propose the first dedicated methodology focused exclusively on the structural integrity of long videos. (2) We introduce an attribute corruption framework that exposes the insensitivity of prevailing metrics to macro-level logical inconsistencies. To address this, we develop two novel metrics based on shot dynamics and narrative flow. Empirical results reveal a performance divergence, where our metrics capture errors that short-video benchmarks overlook, thereby validating our hypothesis regarding the dimensional independence of long-form video quality. (3) We curate and release Long-CODE, a specialized benchmark dataset designed to isolate and evaluate genuine long-range characteristics. Through extensive human experiments, we demonstrate that our proposed metrics achieve state-of-the-art alignment with human evaluation, providing a necessary and robust complement to existing standards for a truly holistic assessment of video generation models. Code: https://github.com/ZhijiangTang/Long-CODE 2. Related Work 2.1. Video Generation Models With the rapid development of diffusion models (Ho et al., 2020; Lipman et al., 2022; Peebles and Xie, 2023), Video Generation Models (VGMs) have achieved astonishing visual results. For short-video generation, the open-source community has made significant strides (Zheng et al., 2024b; Huang et al., 2024a): VDM (Ho et al., 2022) extends diffusion architectures to the video domain; CogVideoX (Yang et al., 2024) leverages 3D causal VAEs and expert transformers for efficient synthesis; the Wan series (Wan et al., 2025) provides highly competitive, open-source, high-fidelity generation capabilities. Meanwhile, closed-source models such as Sora (Liu et al., 2024b) and Veo (Google, 2024) have set new industry benchmarks by producing remarkably realistic, physically grounded short clips. Beyond short clips, there is a growing trend towards the generation of long videos (Zheng et al., 2024a; Zhou et al., 2024). Frameworks like StoryMem (Zhang et al., 2025) incorporate memory mechanisms to preserve character and scene consistency over extended durations. Similarly, HoloCine (Meng et al., 2025) proposes a holistic generation paradigm specifically designed to ensure cinematic multi-shot narrative coherence. In addition to these standalone systems, various plug-and-play modules (Cai et al., 2025; Jia et al., 2025) have been developed to seamlessly empower existing short-video models with long-horizon storytelling capabilities. 2.2. Video Benchmarks Alongside the evolution of VGMs, numerous video benchmarks have been developed to quantify their capabilities. VBench (Huang et al., 2024b, 2025) provides a comprehensive, fine-grained, multidimensional metric suite for generic visual quality. To further assess adherence to physical laws, enhanced benchmarks such as VBench-2.0 (Zheng et al., 2025a) and WorldModelBench (Li et al., 2025) have been introduced to evaluate how well models simulate real-world physical dynamics. Another emerging paradigm utilizes Multimodal Large Language Models (MLLMs) for automated scoring; for instance, VideoQA (Song et al., 2026; Lin et al., 2024), Video-Bench (Han et al., 2025), and VideoScore2 (He et al., 2025) employ advanced vision-language models to semantically evaluate generated content. Recently, new research has begun to explore long-video benchmarks, such as LocoT2V (Zheng et al., 2025b). However, the multidimensional metrics proposed in these early attempts are essentially extrapolations of short-video metrics and fundamentally fail to genuinely assess the intrinsic narrative and structural quality of long videos. In this paper, we theoretically and experimentally demonstrate that short-video metrics and long-video metrics operate on orthogonal dimensions when evaluating long-form generation. Therefore, we propose Long-CODE as a dedicated and necessary complement to existing short-video standards, with extensive experiments strongly validating the effectiveness and indispensability of our benchmark. Figure 2. The framework of Long-CODE. The shot prompts and the generated video together are processed through two parallel and complementary evaluation pipelines. The upper branch calculates the structural dynamic score (MDSAM_DSA) by extracting embeddings from both modalities, performing structure vectorization, and computing the matching correlation (i.e., via Spearman’s r) between prompt-prompt and prompt-video structural similarity vectors to assess structural consistency. Concurrently, the lower branch utilizes an MLLM to compute the semantic quality score (MLLM_MLLM) through a structured, multi-stage workflow encompassing “Captioning”, “Thinking”, and “Scoring”. Finally, MDSAM_DSA and MMLLMM_MLLM are aggregated to formulate the comprehensive long-video evaluation metric, MLong-CODEM_Long-CODE 3. Method 3.1. Preliminary Conditional Flow Matching (CFM) (Lipman et al., 2022). Modern video generation models commonly employ CFM as the training objective (Zheng et al., 2024b; Yang et al., 2024). Given a noise sample x0∼(0,I)x_0 (0,I) and a target sample x1∼x_1 from dataset D, the interpolant is defined as xt=(1−t)x0+tx1x_t=(1-t)x_0+tx_1 for t∈[0,1]t∈[0,1]. A neural network vθv_θ is trained to predict the velocity field: (1) ℒCFM(θ)=t,x0∼,x1∼[‖vθ(xt,t)−(x1−x0)‖2]L_CFM(θ)=E_t,\,x_0 ,\,x_1 [ \|v_θ(x_t,t)-(x_1-x_0) \|^2 ] At inference, the generated sample is obtained by integrating the learned ODE dxt/dt=vθ(xt,t)dx_t/dt=v_θ(x_t,t) from t=0t=0 to t=1t=1 (Lipman et al., 2022; Liu et al., 2025). Shot-Based Long Video Generation. State-of-the-art long video generation models (Zhang et al., 2025; Zheng et al., 2024a) decompose the generation task into a sequence of shot-level productions. Formally, a long video V is produced as the temporal concatenation of K shots: (2) =s1⊕s2⊕⋯⊕sK,sk=Gθ(ck,<k)V=s_1 s_2 ·s s_K, s_k=G_θ (c_k,\;S_<k ) where ckc_k denotes the text prompt for the k-th shot, <k=s1,…,sk−1S_<k=\s_1,…,s_k-1\ represents all previously generated shots that provide conditioning context (e.g., the last frame of sk−1s_k-1 serves as the initial frame for generating sks_k), and ⊕ denotes temporal concatenation. Each shot sks_k is itself a short video clip generated via a flow-matching based diffusion model conditioned on both the textual description and visual context from preceding shots. For models lacking native long-video generation capabilities (e.g., Sora and Veo), a prevalent paradigm involves a shot-by-shot synthesis approach. Specifically, each shot is generated by conditioning on the final frame of the preceding shot and the text prompt for the current shot. Ultimately, these discrete shots are temporally concatenated to synthesize a cohesive long video. 3.2. Long-Context Video Attributes Theoretical Analysis. We define a long-context video V as an ordered sequence of shots s1,s2,…,sKs_1,s_2,…,s_K, where each shot sks_k is a short video clip corresponding to a specific scene or narrative segment. Unlike short videos, which can be fully characterized by frame-level visual quality and local temporal smoothness, long videos inherently carry additional long-range structural attributes that emerge exclusively from inter-shot relationships (e.g., temporal logic and causal consistency). We denote by MSM_S a short-video metric that evaluates individual shot quality. To score a long video V, existing short-video benchmarks (e.g., VBench (Huang et al., 2024b)) typically aggregate per-shot scores through a symmetric aggregator ϕφ (e.g., arithmetic mean): (3) MS()=ϕ(q(s1),q(s2),…,q(sK)),M_S(V)=φ (q(s_1),\,q(s_2),\,…,\,q(s_K) ), where q:→ℝq maps each shot to a scalar quality score. A long-context metric MLM_L, by contrast, is structure-aware: it is sensitive to the sequential inter-shot coherence encoded by the pairwise relation matrix ∈ℝK×KR ^K× K, where Rij=r(si,sj)R_ij=r(s_i,s_j) measures the semantic or temporal coherence between shots i and j. Proposition 1. For the short-video metric MSM_S defined in Eq. (3) with a symmetric aggregator ϕφ, and a long-context metric MLM_L that depends solely on the inter-shot relation matrix R, the mutual information between MSM_S and MLM_L vanishes: I(MS;ML)=∬p(MS,ML)logp(MS,ML)p(MS)p(ML)dMSdML=0.I(M_S;M_L)= p(M_S,M_L) p(M_S,M_L)p(M_S)\,p(M_L)\,dM_S\,dM_L=0. The proof of this proposition is in the supplementary material. This result establishes that any aggregation-based short-video metric is provably insensitive to structural perturbations in the shot sequence, underscoring the need to introduce a complementary long-context evaluation dimension. We empirically validate this orthogonality using the following corruption-testing framework. Corruption Testing Framework. We design four families of corruption operators that target long-range structural attributes (influencing MLM_L) while reducing per-shot visual quality degradation (preserving MSM_S), thereby exposing the insensitivity of existing short-video metrics: (1) Shuffle. The shot ordering of V is randomly permuted according to a uniformly sampled permutation π. Each shot’s visual content remains completely intact. (2) Replace. A subset of constituent shots within a video V is replaced by exogenous shots retrieved from a comprehensive shots Bank (constructed from large-scale open-source video repositories). (3) Edition. A subset of shots in V is re-edited by a depth-guided video editing model (i.e., VACE (Jiang et al., 2025)), which replaces key semantic elements (e.g., character gender or identity) while preserving the spatial layout and motion dynamics. (4) Synthesis. A subset of shots in V is regenerated from modified descriptions: an MLLM first captions the original shot, the caption is then rewritten to be surreal, and VGM synthesizes a replacement shot from the altered prompt and the original first frame. All corruption operators are parameterized by a strength hyperparameter that controls the intensity of the perturbation (e.g., the number or fraction of shots affected). Extensive experiments in Section 4 confirm that representative short-video metrics remain largely invariant under these corruptions, empirically validating the orthogonality established in Proposition 1. 3.3. Long-CODE Dynamic Structure Alignment (DSA). DSA measures the structural alignment between intended and realized shot dynamics through a video-language embedding space. Given a video V with shot prompts =c1,…,cKC=\c_1,…,c_K\, let ft(⋅)f_t(·) and fv(⋅)f_v(·) denote the L2-normalized text and video embedder, respectively. We define two K-dimensional similarity vectors: (4) sic=ft(ci)⊤ft(),siv=ft(ci)⊤fv().s_i^c=f_t(c_i) \,f_t(C), s_i^v=f_t(c_i) \,f_v(V). Since all feature vectors are L2-normalized, these inner products equal cosine similarities. The vector c=(s1c,…,sKc)s^c=(s_1^c,…,s_K^c) captures the intended structural importance of each shot relative to the global narrative (a purely text-derived reference), while v=(s1v,…,sKv)s^v=(s_1^v,…,s_K^v) captures how well the generated video’s visual content reflects each shot’s intended contribution. We quantify the structural alignment via the Spearman rank correlation coefficient between these vectors: (5) MDSA(,)=∑i=1K(R(sic)−R¯c)(R(siv)−R¯v)∑i=1K(R(sic)−R¯c)2⋅∑i=1K(R(siv)−R¯v)2M_DSA(V,C)= _i=1^K (R(s_i^c)- R^c ) (R(s_i^v)- R^v ) _i=1^K (R(s_i^c)- R^c )^2· _i=1^K (R(s_i^v)- R^v )^2 where R(⋅)R(·) denotes the rank function and R¯c,R¯v R^c, R^v are the respective mean ranks. The Spearman correlation is particularly appropriate here: it captures whether the relative ordering of shot importances is preserved in the generated video, rather than requiring exact magnitude agreement. This makes DSA inherently sensitive to disruptions in shot ordering and structural dynamics, precisely the long-range attributes that short-video metrics fail to capture. Long-Context Thinking with MLLM. While DSA captures temporal structural alignment, it cannot reason about fine-grained semantic attributes such as causal consistency across shots. To complement DSA, we design a multi-phase MLLM reasoning pipeline that employs deep thinking to assess long-range semantic quality. As illustrated in Figure 2, our evaluation process unfolds in three key stages. (a) First, we employ an MLLM to segment the input video V into discrete shots and generate corresponding captions, denoted as C. (b) Second, we feed both the original C and the C into an LLM for deep reasoning. This step systematically analyzes C to detect potential structural errors (e.g., causal violations) and synthesizes a summary, T. (c) Finally, we re-input all the aggregated information into the MLLM to compute the final metric, averaging the scores across n independent sampling rounds: (6) MMLLM=1n∑i=1nMLLM(,,,ℬ)M_MLLM= 1n _i=1^nMLLM(V,C,T,B) where ℬB denotes a reference set of human-annotated video cases utilized for in-context alignment. The final Long-CODE metric integrates both dimensions: (7) MLong-CODE=α⋅MDSA+(1−α)⋅MMLLM,M_Long -CODE=α· M_DSA+(1-α)· M_MLLM, where DSA captures temporal-structural alignment from a shot-dynamics perspective, and MMLLMM_MLLM provides deep semantic-level analysis of long-range narrative quality. Figure 3. Correlation between metric scores and corruption strengths in the corruption tests. Since all evaluated benchmarks are higher-is-better metrics, their scores are expected to decrease as the corruption intensity increases. Consequently, if a metric exhibits a positive correlation with corruption strengths (indicated by the red lines in the figure, e.g., VideoBench in Shuffle) or yields a coefficient of determination R2<0.6R^2<0.6 (Draper and Smith, 1998) (e.g., VideoQA in Replace), it is deemed insensitive to the given corruption (denoted by the red crosses). 3.4. Dataset Long-CODE Benchmark Dataset. To rigorously evaluate long-video generation metrics, we introduce a human-annotated dataset specifically targeting long-context attributes through a three-stage construction pipeline. First, an MLLM extracts a story seed from a single representative frame of a real-world video to generate a concise cinematic outline. Second, the MLLM decomposes this narrative into a sequence of structured shots, yielding the final prompt sequence =(c1,…,cK)C=(c_1,…,c_K). Finally, multiple state-of-the-art models generate videos from C, which human annotators evaluate strictly on long-range dimensions—such as causal consistency, temporal logic, and cross-shot identity persistence. By explicitly excluding per-shot visual quality from the evaluation rubric, this protocol isolates genuine long-context attributes from short-video fidelity confounds, establishing Long-CODE as a specialized complement to existing video generation assessment standards. 4. Experiments Benchmark VGoT HoloCine Wan2.2 StoryMem Veo3.1 Sora2 Overall Spearman Pearson VBench 0.013 -0.035 -0.069 -0.090 -0.171 -0.105 -0.023 -0.104 Motion 0.012 -0.048 0.124 0.131 0.074 0.195 0.137 0.132 Subject 0.228 -0.006 0.217 0.193 0.076 0.165 0.045 0.047 Dynamic -0.152 0.097 -0.152 -0.180 -0.006 -0.092 0.014 0.022 VideoScore2 -0.023 0.061 -0.140 -0.069 -0.160 0.004 -0.010 -0.028 VideoQA 0.086 -0.055 -0.111 0.027 -0.011 0.127 -0.004 -0.006 VideoText 0.138 -0.059 0.204 0.072 0.189 0.002 0.092 0.081 VideoBench 0.069 0.152 0.042 0.186 0.210 0.108 0.373 0.381 Object 0.081 0.215 0.042 0.193 0.304 -0.016 0.299 0.294 Scene -0.010 0.186 0.026 0.092 0.131 0.148 0.300 0.292 Videotext 0.123 0.121 0.040 0.133 0.298 0.141 0.332 0.327 Long-CODE 0.708 0.571 0.835 0.831 0.796 0.733 0.765 0.754 Table 1. Spearman (Spearman, 1961) correlation coefficients between various benchmarks and human evaluations within each model. “Overall” denotes the Spearman and Pearson (Pearson, 1895) correlation coefficients calculated by aggregating all samples. For the multi-dimensional benchmarks, VBench and VideoBench, we report the three metrics that exhibit the highest correlations. Specifically, in VBench (Huang et al., 2024b), “Motion”, “Subject”, and “Dynamic” denote the dimensions of motion smoothness, subject consistency, and dynamic degree, respectively. In VideoBench (Han et al., 2025), “Object”, “Scene”, and “Videotext” correspond to the dimensions of object class, scene, and video-text consistency, respectively. Bold numbers indicate the best performance, while underlined numbers denote the runner-up. 4.1. Models and Benchmarks Long Video Generation Models. We evaluate 6 representative video generation models spanning both natively long-form generators and single-shot models extended via a sequential pipeline: • VGoT (Zheng et al., 2024a) is a training-free, modular framework. We generate each shot at 640p resolution with 25 denoising steps and a per-shot duration of 5 s. • HoloCine (Meng et al., 2025) holistically generates cinematic multi-shot narratives through Window Cross-Attention and Sparse Inter-Shot Self-Attention. Built upon the Wan2.2-T2V-A14B backbone, we produce 480×832 videos at 15 fps with 3 shots per generation chunk and 5 s per shot. • Wan2.2 (Wan et al., 2025) is Alibaba’s open-source Mixture-of-Experts video diffusion model. We employ the TI2V-5B variant at 1280×704 resolution with 25 sampling steps. • StoryMem (Zhang et al., 2025) adopts a Memory-to-Video design that maintains a dynamically updated keyframe memory bank. It fine-tunes Wan2.2 with lightweight LoRA (rank 128) and generates 832×480 videos with 5 s shots. • Veo3.1 (Google, 2024) is Google’s commercial cinematic video engine. We produce 1920×1080 clips of 8 s each. • Sora2 (Liu et al., 2024b) is OpenAI’s video generation model with synchronized audio and advanced physics simulation. We use the model configuration at 1280×720, generating 15 s clips. Among these, VGoT, HoloCine, and StoryMem natively support multi-shot long-video generation. For models that do not (i.e., Wan2.2, Veo3.1, and Sora2), we introduce a long video generation pipeline: each shot is generated conditioned on the current shot prompt and the last frame of the preceding shot, and all shots are then concatenated into a long video. Each model generates 100 long videos from the Long-CODE Dataset, yielding a total of 600. Video Benchmarks. We compare Long-CODE against 5 established video evaluation methods that represent the predominant paradigms in the field. • VBench (Huang et al., 2024b) is a comprehensive benchmark suite (CVPR 2024) that decomposes video quality into fine-grained dimensions, including subject consistency, background consistency, motion smoothness, dynamic degree, aesthetic quality, and imaging quality; we evaluate all applicable dimensions for long videos. • VideoScore2 (He et al., 2025) is a fine-tuned MLLM evaluator (based on Qwen2.5-VL-7B) that explicitly scores three dimensions (i.e., visual quality, text-to-video alignment, and physics/common-sense consistency) with chain-of-thought reasoning. • VideoQA (Song et al., 2026; Lin et al., 2024) assesses narrative comprehension through multiple-choice questions: we auto-generate 5 questions per sample from the storyline and use Qwen3-VL-32B-Instruct (Bai et al., 2025) to answer them over the generated video, reporting accuracy. • VideoText (Liu et al., 2024a; Wang et al., 2024) computes the per-shot cosine similarity between VideoCLIP-XL (Wang et al., 2024) video features and text features and averages across shots. • VideoBench (Han et al., 2025) (CVPR 2025) employs a Qwen3-VL-32B-Instruct judge to score videos across 9 quality dimensions with few-shot, chain-of-query prompting. Figure 4. A case study of different long video generation models on the Long-CODE dataset. Storyline is “At dawn, a mechanic works on the dusty engine of an old white pickup truck. While checking a hose, he remembers the day his late son helped him install it. With a sigh, he pours in fresh coolant, keeping the old truck running in honor of his son’s memory.” 4.2. Implementation Corruption Tests. To expose the limitations of current short-video metrics, we devise four long-range attribute corruption tests and apply them to 100 real videos from FineVideo (Farré et al., 2024). • Shuffle: Segments the video into 10-second blocks and randomly permutes them, with the proportion of shuffled blocks varying across 0.2, 0.4, 0.8. • Replace: Substitutes chosen blocks with the most cosine-similar 10-second segments retrieved from a 100-video candidate pool, encoded via VideoCLIP-XL (Wang et al., 2024). The number of replaced blocks varies across 2, 4, 8, 16. • Edition: Utilizes VACE (Jiang et al., 2025) (1.3B) for depth-guided editing to modify character identities (e.g., gender swapping) while maintaining the original scene layout. We randomly select 2, 4, 8, 16 blocks (81 frames each) for editing. • Synthesis: Generates captions for randomly selected blocks using Qwen3-VL-8B-Instruct, rewrites them into surreal descriptions, and resynthesizes the blocks using Wan2.2 TI2V-5B (1280×7041280× 704 resolution, 25 denoising steps). The number of resynthesized blocks varies across 2, 4, 8, 16. Long-CODE Metric. The Long-CODE metric sets α=0.5α=0.5 balances the two complementary sub-metrics. MDSAM_DSA uses VideoCLIP-XL (Wang et al., 2024) to extract features, and finally normalized to [0,1][0,1]. MMLLMM_MLLM (MLLM-as-Judge) leverages Qwen3-VL-32B-Instruct (Bai et al., 2025) describes each shot from the sampled frames; analyzes quality from four perspectives (i.e., visual richness, temporal transition, object/background consistency, and logic errors), producing a structured summary; the target video is compared against 3 reference videos (randomly drawn from a bank of 15 human-scored samples) and scored on a 1–5 scale. This process is repeated for 3 rounds with increasing temperature (t∈0.3,0.4,0.5t∈\0.3,0.4,0.5\) and different reference selections; the final score is the average across rounds, normalized to [0,1][0,1]. Long-CODE Dataset. We source 100 diverse real-world videos from the FineVideo (Farré et al., 2024) dataset as seed material. For each video, we extract a random frame and prompt Qwen3-VL-32B-Instruct (Bai et al., 2025) to produce a cinematic story outline. The outline is then expanded into a sequence of shot descriptions with duration and cut-type annotations, targeting a total duration of 60–120 s. To ensure temporal regularity, each shot is normalized to a fixed duration of 5 s. This yields a structured prompt comprising 12–24 shots per sample, each annotated with a textual description, duration, and cut indicator. Human evaluation is conducted on three long-range dimensions that are orthogonal to short-video visual quality: narrative accuracy (whether the generated video faithfully renders the intended storyline), long-range causality (whether cause and effect relationships hold across distant shots), and cross-scene identity consistency (whether characters and key objects maintain coherent appearance throughout the video). Each of the 600 generated videos (100 samples × 6 models) is rated by 8 human annotators on a 1–5 Likert scale for these three dimensions. 4.3. Result Q1. Are long-context attributes orthogonal to short-video dimensions? A1. Long-context attributes are orthogonal to short-video dimensions. As shown in Figure 3, we systematically apply long-range structural corruptions to long videos and measure the sensitivity of existing metrics. A reliable metric should exhibit a monotonic decrease as corruption intensity increases. However, established short-video metrics fail this fundamental test: VideoBench shows a positive correlation with corruption strength under the Shuffle test. Similarly, VBench, VideoText, and VideoScore2 failed the Replace, Edition, and Synthesis tests, respectively. Furthermore, although VideoQA demonstrates a negative correlation across all corruption tests, its R2<0.6R^2<0.6 in the Replace and Synthesis tests, reflecting a negligible sensitivity to content-level perturbations (Draper and Smith, 1998). Qualitative analysis further corroborates this orthogonality. As shown in the Figure 4, each individual shot can exhibit high visual quality, yet the overall long video suffers from severe cross-scene inconsistencies, including character identity drift, broken causal chains, and incoherent narrative progression. As shown in Table 1, quantitative analysis at scale confirms the above findings. When evaluated against human judgments specifically annotated for long-range attributes, short-video metrics exhibit near-zero or even negative correlations: VBench achieves an overall Spearman correlation of merely −0.023-0.023, VideoScore2 yields −0.010-0.010, and VideoQA produces −0.004-0.004. Even the best sub-dimensions of VBench, Motion (0.1370.137), Subject (0.0450.045), and Dynamic (0.0140.014), remain negligible. These results confirm that metrics designed for short-video quality, focusing on local temporal coherence, are inherently blind to long-range structural disruptions. Narrative Causality Consistency MDSAM_DSA 0.259 0.214 0.219 MMLLMM_MLLM 0.637 0.592 0.604 MLong-CODEM_Long-CODE 0.685 0.620 0.626 Table 2. Ablation studies of the proposed Long-CODE metric across different human evaluation dimensions. The human evaluation dimensions “Narrative”, “Causality”, and “Consistency” denote narrative accuracy, long-range causality, and cross-scene identity consistency, respectively (Section 4.2). Q2. Does Long-CODE align with human preferences in evaluating long videos? A2. Long-CODE align with human preferences in evaluating long videos. As reported in Table 1, Long-CODE obtains the highest Spearman correlation with human judgments across all six evaluated models without exception: VGoT (0.7080.708), HoloCine (0.5710.571), and etc.. These correlations are consistently strong (>0.57>0.57) across models spanning diverse architectures and generation paradigms, demonstrating the robustness of our metric. The overall Spearman and Pearson correlations of Long-CODE are 0.7650.765 and 0.7540.754, respectively. The strongest baseline, VideoBench, achieves only 0.3730.373 (Spearman) and 0.3810.381 (Pearson), less than half of Long-CODE’s correlation. All other baselines fall below 0.10.1 in overall correlation. This constitutes a relative improvement of over 105%105\% in Spearman correlation over the best existing metric. While certain baselines exhibit moderate per-model correlations (e.g., VideoText achieves 0.2040.204 on Wan2.2), they fail to maintain consistent performance across models. In contrast, Long-CODE’s minimum per-model correlation (0.5710.571 on HoloCine) already exceeds the maximum per-model correlation of any baseline. This underscores the superior generalizability of our metric. Q3. What specific qualities of long videos do the two metrics within Long-CODE respectively evaluate? A3. The two component metrics within Long-CODE, Dynamic Structure Alignment MDSAM_DSA and MMLLMM_MLLM, capture complementary dimensions of long-video quality, and their combination yields consistent improvements. As shown in Table 2, MMLLMM_MLLM dominates on all three semantic evaluation dimensions, achieving correlations of 0.6370.637 (Narrative), 0.5920.592 (Causality), and 0.6040.604 (Consistency). This is expected, as the MLLM component explicitly reasons about narrative content, causal relationships, and cross-scene identity. In contrast, MDSAM_DSA alone yields lower correlations (0.2590.259, 0.2140.214, 0.2190.219), since it operates on structural shot dynamics rather than semantic content. Crucially, the combined metric MLong-CODEM_Long-CODE consistently surpasses both individual components: 0.6850.685 (+0.048+0.048 over MMLLMM_MLLM on Narrative), 0.6200.620 (+0.028+0.028 on Causality), and 0.6260.626 (+0.022+0.022 on Consistency). This complementary sensitivity profile confirms the design motivation: MDSAM_DSA evaluates temporal-structural quality, while MMLLMM_MLLM evaluates semantic-content quality. Their fusion in MLong-CODEM_Long-CODE yields a comprehensive metric sensitive to both corruption categories. Q4. Why is Dynamic Structure Alignment (DSA) effective in evaluating long-video dynamics? A4. From the formulation perspective. The key insight behind DSA is a decoupling of what each shot should contribute from what it actually contributes in the generated video. Recall that cs^c encodes the intended structural importance of each shot relative to the global narrative purely from text, while vs^v reflects how the generated video’s visual content realizes each shot’s intended contribution. By comparing these two profiles via Spearman rank correlation, DSA evaluates whether the relative ordering of shot importance is faithfully preserved, rather than demanding exact magnitude agreement. DSA is inherently sensitive to long-range structural disruptions: any perturbation that alters the temporal arrangement of shots (e.g., narrative shuffling) directly rearranges vs^v while leaving cs^c unchanged, producing a measurable drop in MDSAM_DSA. From the empirical perspective. The ablation in Table 2 reveals that even though MDSAM_DSA alone achieves moderate human correlation, it consistently improves the combined metric: MLong-CODEM_Long-CODE outperforms MMLLMM_MLLM by +3.3%+3.3\%–7.5%7.5\% across all three human evaluation dimensions, confirming that DSA captures structural information complementary to semantic evaluation. 5. Conclusion In this paper, we identified a critical gap in the evaluation of Video Generation Models (VGMs) as they scale to generate longer content. We theoretically and empirically demonstrated that short-term visual perception and long-context attributes are fundamentally orthogonal dimensions. To address the insensitivity of traditional short-video metrics to structural inconsistencies, we introduced a suite of long-video attribute corruption tests. Furthermore, we developed a novel evaluation metric based on shot dynamics and the deep reasoning capabilities of Multimodal Large Language Models. Alongside this, we introduced Long-CODE, a specialized dataset designed to benchmark long-video evaluation, with human annotations isolated specifically to genuine long-range characteristics. Extensive experiments show that our proposed metrics achieve state-of-the-art correlation with human judgments. Ultimately, our metric and benchmark seamlessly complement existing short-video standards, establishing a holistic and unbiased evaluation paradigm for video generation models. References S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, et al. (2025) Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Cited by: 3rd item, §4.2, §4.2. D. Brewster (2010) Memoirs of the life, writings, and discoveries of sir isaac newton. Vol. 2, Cambridge University Press. Cited by: Figure 1. S. Cai, C. Yang, L. Zhang, Y. Guo, J. Xiao, Z. Yang, Y. Xu, Z. Yang, A. Yuille, L. Guibas, et al. (2025) Mixture of contexts for long video generation. arXiv preprint arXiv:2508.21058. Cited by: §2.1. N. R. Draper and H. Smith (1998) Applied regression analysis. Vol. 326, John Wiley & Sons. Cited by: Figure 3, §4.3. M. Farré, A. Marafioti, L. Tunstall, L. Von Werra, and T. Wolf (2024) FineVideo. Note: https://huggingface.co/datasets/HuggingFaceFV/finevideo Cited by: §4.2, §4.2. Google (2024) Veo: a text-to-video generation system. Note: https://storage.googleapis.com/deepmind-media/veo/Veo-3-Tech-Report.pdf Cited by: Figure 1, §1, §2.1, 5th item. H. Han, S. Li, J. Chen, Y. Yuan, Y. Wu, Y. Deng, C. T. Leong, H. Du, J. Fu, Y. Li, et al. (2025) Video-bench: human-aligned video generation benchmark. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 18858–18868. Cited by: §2.2, 5th item, Table 1. X. He, D. Jiang, P. Nie, M. Liu, Z. Jiang, M. Su, W. Ma, J. Lin, C. Ye, Y. Lu, et al. (2025) Videoscore2: think before you score in generative video evaluation. arXiv preprint arXiv:2509.22799. Cited by: §2.2, 2nd item. J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. Advances in neural information processing systems 33, p. 6840–6851. Cited by: §2.1. J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet (2022) Video diffusion models. Advances in neural information processing systems 35, p. 8633–8646. Cited by: §2.1. Y. Huang, W. Zheng, Y. Gao, X. Tao, P. Wan, D. Zhang, J. Zhou, and J. Lu (2024a) Owl-1: omni world model for consistent long video generation. arXiv preprint arXiv:2412.09600. Cited by: §2.1. Z. Huang, Y. He, J. Yu, F. Zhang, C. Si, Y. Jiang, Y. Zhang, T. Wu, Q. Jin, N. Chanpaisit, et al. (2024b) Vbench: comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 21807–21818. Cited by: Figure 1, §1, §1, §2.2, §3.2, 1st item, Table 1. Z. Huang, F. Zhang, X. Xu, Y. He, J. Yu, Z. Dong, Q. Ma, N. Chanpaisit, C. Si, Y. Jiang, et al. (2025) Vbench++: comprehensive and versatile benchmark suite for video generative models. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §2.2. W. Jia, Y. Lu, M. Huang, H. Wang, B. Huang, N. Chen, M. Liu, J. Jiang, and Z. Mao (2025) Moga: mixture-of-groups attention for end-to-end long video generation. arXiv preprint arXiv:2510.18692. Cited by: §2.1. Z. Jiang, Z. Han, C. Mao, J. Zhang, Y. Pan, and Y. Liu (2025) Vace: all-in-one video creation and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 17191–17202. Cited by: item 3, 3rd item. D. Li, Y. Fang, Y. Chen, S. Yang, S. Cao, J. Wong, M. Luo, X. Wang, H. Yin, J. E. Gonzalez, et al. (2025) Worldmodelbench: judging video generation models as world models. arXiv preprint arXiv:2502.20694. Cited by: §1, §2.2. Z. Lin, D. Pathak, B. Li, J. Li, X. Xia, G. Neubig, P. Zhang, and D. Ramanan (2024) Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision, p. 366–384. Cited by: §2.2, 3rd item. Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le (2022) Flow matching for generative modeling. arXiv preprint arXiv:2210.02747. Cited by: §2.1, §3.1, §3.1. J. Liu, G. Liu, J. Liang, Y. Li, J. Liu, X. Wang, P. Wan, D. Zhang, and W. Ouyang (2025) Flow-grpo: training flow matching models via online rl. arXiv preprint arXiv:2505.05470. Cited by: §3.1. Y. Liu, X. Cun, X. Liu, X. Wang, Y. Zhang, H. Chen, Y. Liu, T. Zeng, R. Chan, and Y. Shan (2024a) Evalcrafter: benchmarking and evaluating large video generation models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 22139–22149. Cited by: §1, 4th item. Y. Liu, K. Zhang, Y. Li, Z. Yan, C. Gao, R. Chen, Z. Yuan, Y. Huang, H. Sun, J. Gao, et al. (2024b) Sora: a review on background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:2402.17177. Cited by: Figure 1, §1, §2.1, 6th item. Y. Meng, H. Ouyang, Y. Yu, Q. Wang, W. Wang, K. L. Cheng, H. Wang, Y. Li, C. Chen, Y. Zeng, et al. (2025) Holocine: holistic generation of cinematic multi-shot long video narratives. arXiv preprint arXiv:2510.20822. Cited by: Figure 1, §2.1, 2nd item. K. Pearson (1895) VII. note on regression and inheritance in the case of two parents. proceedings of the royal society of London 58 (347-352), p. 240–242. Cited by: Table 1. W. Peebles and S. Xie (2023) Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, p. 4195–4205. Cited by: §2.1. Y. Song, T. Pfister, and Y. Song (2026) VQQA: an agentic approach for video evaluation and quality improvement. arXiv preprint arXiv:2603.12310. Cited by: §2.2, 3rd item. C. Spearman (1961) The proof and measurement of association between two things.. Cited by: §1, Table 1. T. Wan, A. Wang, B. Ai, B. Wen, C. Mao, C. Xie, D. Chen, F. Yu, H. Zhao, J. Yang, et al. (2025) Wan: open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314. Cited by: Figure 1, §1, §2.1, 3rd item. J. Wang, C. Wang, K. Huang, J. Huang, and L. Jin (2024) Videoclip-xl: advancing long description understanding for video clip models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 16061–16075. Cited by: §1, 4th item, 2nd item, §4.2. Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y. Yang, W. Hong, X. Zhang, G. Feng, et al. (2024) Cogvideox: text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072. Cited by: §2.1, §3.1. K. Zhang, L. Jiang, A. Wang, J. Z. Fang, T. Zhi, Q. Yan, H. Kang, X. Lu, and X. Pan (2025) StoryMem: multi-shot long video storytelling with memory. arXiv preprint arXiv:2512.19539. Cited by: Figure 1, §1, §2.1, §3.1, 4th item. D. Zheng, Z. Huang, H. Liu, K. Zou, Y. He, F. Zhang, L. Gu, Y. Zhang, J. He, W. Zheng, et al. (2025a) Vbench-2.0: advancing video generation benchmark suite for intrinsic faithfulness. arXiv preprint arXiv:2503.21755. Cited by: §2.2. M. Zheng, Y. Xu, H. Huang, X. Ma, Y. Liu, W. Shu, Y. Pang, F. Tang, Q. Chen, H. Yang, et al. (2024a) VideoGen-of-thought: step-by-step generating multi-shot video with minimal manual intervention. arXiv preprint arXiv:2412.02259. Cited by: Figure 1, §1, §2.1, §3.1, 1st item. X. Zheng, C. Wu, K. Chen, and M. Zhang (2025b) LoCoT2V-bench: benchmarking long-form and complex text-to-video generation. arXiv preprint arXiv:2510.26412. Cited by: §1, §2.2. Z. Zheng, X. Peng, T. Yang, C. Shen, S. Li, H. Liu, Y. Zhou, T. Li, and Y. You (2024b) Open-sora: democratizing efficient video production for all. arXiv preprint arXiv:2412.20404. Cited by: §2.1, §3.1. Y. Zhou, D. Zhou, M. Cheng, J. Feng, and Q. Hou (2024) Storydiffusion: consistent self-attention for long-range image and video generation. Advances in Neural Information Processing Systems 37, p. 110315–110340. Cited by: §2.1.