Paper deep dive
OSCBench: Benchmarking Object State Change in Text-to-Video Generation
Xianjing Han, Bin Zhu, Shiqi Hu, Franklin Mingzhe Li, Patrick Carrington, Roger Zimmermann, Jingjing Chen
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 99%
Last extracted: 3/22/2026, 6:26:52 AM
Summary
OSCBench is a diagnostic benchmark designed to evaluate Object State Change (OSC) in text-to-video (T2V) generation models. It categorizes instructional cooking actions into regular, novel, and compositional scenarios to test both in-distribution performance and generalization. The study evaluates six SOTA T2V models using human studies and MLLM-based Chain-of-Thought evaluation, revealing that while models achieve high semantic alignment, they struggle with accurate and temporally consistent object state transformations.
Entities (8)
Relation Signals (3)
Open-Sora-2.0 â evaluatedby â OSCBench
confidence 100% ¡ We evaluate six representative open-source and proprietary T2V models... using OSCBench
OSCBench â evaluates â T2V Models
confidence 100% ¡ OSCBench, a benchmark specifically designed to assess OSC performance in T2V models.
OSCBench â usesdatafrom â HowToChange
confidence 100% ¡ We therefore build OSCBench on the HowToChange dataset (Xue et al., 2024)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Text-to-video (T2V) generation models have made rapid progress in producing visually high-quality and temporally coherent videos. However, existing benchmarks primarily focus on perceptual quality, text-video alignment, or physical plausibility, leaving a critical aspect of action understanding largely unexplored: object state change (OSC) explicitly specified in the text prompt. OSC refers to the transformation of an object's state induced by an action, such as peeling a potato or slicing a lemon. In this paper, we introduce OSCBench, a benchmark specifically designed to assess OSC performance in T2V models. OSCBench is constructed from instructional cooking data and systematically organizes action-object interactions into regular, novel, and compositional scenarios to probe both in-distribution performance and generalization. We evaluate six representative open-source and proprietary T2V models using both human user study and multimodal large language model (MLLM)-based automatic evaluation. Our results show that, despite strong performance on semantic and scene alignment, current T2V models consistently struggle with accurate and temporally consistent object state changes, especially in novel and compositional settings. These findings position OSC as a key bottleneck in text-to-video generation and establish OSCBench as a diagnostic benchmark for advancing state-aware video generation models.
Tags
Links
- Source: https://arxiv.org/abs/2603.11698v1
- Canonical: https://arxiv.org/abs/2603.11698v1
Trouble viewing inline? Open PDF directly â
Full Text
66,446 characters extracted from source content.
Expand or collapse full text
OSCBench: Benchmarking Object State Change in Text-to-Video Generation Xianjing Han 1 * , Bin Zhu 2 * â , Shiqi Hu 1 , Franklin Mingzhe Li 3 , Patrick Carrington 3 , Roger Zimmermann 1 , Jingjing Chen 4 1 National University of Singapore 2 Singapore Management University 3 Carnegie Mellon University 4 Fudan University Correspondence: binzhu@smu.edu.sg Abstract Text-to-video (T2V) generation models have made rapid progress in producing visually high- quality and temporally coherent videos. How- ever, existing benchmarks primarily focus on perceptual quality, textâvideo alignment, or physical plausibility, leaving a critical aspect of action understanding largely unexplored: ob- ject state change (OSC) explicitly specified in the text prompt. OSC refers to the transforma- tion of an objectâs state induced by an action, such as peeling a potato or slicing a lemon. In this paper, we introduce OSCBench, a bench- mark specifically designed to assess OSC per- formance in T2V models. OSCBench is con- structed from instructional cooking data and systematically organizes actionâobject inter- actions into regular, novel, and compositional scenarios to probe both in-distribution perfor- mance and generalization. We evaluate six rep- resentative open-source and proprietary T2V models using both human user study and mul- timodal large language model (MLLM)âbased automatic evaluation. Our results show that, despite strong performance on semantic and scene alignment, current T2V models consis- tently struggle with accurate and temporally consistent object state changes, especially in novel and compositional settings. These find- ings position OSC as a key bottleneck in text- to-video generation and establish OSCBench as a diagnostic benchmark for advancing state- aware video generation models. Project page: https://hanxjing.github.io/OSCBench. 1 Introduction Text-to-video (T2V) generation models have made remarkable progress in recent years, producing videos with increasingly high visual fidelity and temporal coherence. These advances have enabled a wide range of applications, including creative 1 * Equal contribution. 2 â Corresponding author and project lead. content generation, instructional video synthesis, and simulation of real-world processes (Google DeepMind, 2025b; Ma et al., 2025). As T2V mod- els continue to scale, a central question emerges: to what extent do these models faithfully realize the consequences of actions specified in language, rather than merely producing visually appealing motion patterns? Recent benchmarks have taken important steps toward answering this question by evaluating phys- ical plausibility and commonsense constraints in generated videos, such as adherence to gravity, col- lisions, and material properties (Meng et al., 2024; Gu et al., 2025). While these evaluations probe fundamental aspects of physical realism, they over- look a critical dimension of language-grounded action understanding that is ubiquitous in everyday activities: Object State Change (OSC) explicitly specified by the prompt. In many real-world tasks, such as slicing a lemon, peeling a carrot, or mixing dough, success is defined not only by performing an action, but by transforming an object from an initial state to a specific target state (e.g., a whole lemon becoming sliced). Correctly modeling such object state change is essential for downstream ap- plications, including robotics, embodied AI, and instructional video generation. Object state change poses a particularly strin- gent test of language-grounded reasoning in T2V models. Correct OSC generation requires a model to understand the action semantics expressed in language, infer the intended object transformation, and render a continuous and coherent visual evolu- tion over time. However, despite producing visu- ally compelling videos, current T2V models often fail on this dimension: generated outputs may ap- pear realistic at a glance while exhibiting incorrect, incomplete, or inconsistent object state changes. Figure 1 (a) illustrates representative failure cases, where objects change into implausible states or the instructed action is misunderstood, revealing 1 arXiv:2603.11698v1 [cs.CV] 12 Mar 2026 Compositional Achefispeelingand slicingpearinahome kitchen. Novel Regular Amanwithawhite apronisslicinglemon onthegrass. Amaninblueshirtis mashinggrapefruitin thekitchen. (a) (b) Figure 1: Overview of OSCBench evaluation. (a) Representative failure cases from regular, novel, and compositional object state change scenarios. In the regular case, the red box marks an implausible state change of the lemon during slicing. In the novel case, the model misinterprets the instructed action, resulting in a wrong object transformation. In the compositional case, the yellow box indicates an incomplete state change where the pear remains unpeeled. (b) Human-evaluated multi-dimensional performance of T2V models on OSCBench. a gap between high-level semantic alignment and faithful realization of action consequences. De- spite the importance of OSC, it has not been sys- tematically evaluated in existing T2V benchmarks, which primarily emphasize overall perceptual qual- ity, textâvideo alignment, or physical plausibil- ity, without explicitly assessing whether an object reaches the the correct target state or whether the state transition unfolds consistently over time. To address this gap, we introduce OSCBench, a benchmark designed to evaluate object state change in text-to-video generation. We focus on instruc- tional cooking scenarios, where state changes are frequent, diverse, and well-defined, and build OS- CBench on top of the HowToChange dataset (Xue et al., 2024). To enable balanced and comprehen- sive evaluation, we abstract actions and objects into semantically meaningful categories and then con- struct three complementary evaluation regimes as shown in Figure 1 (a): regular scenarios covering common actionâobject pairs (e.g., slicing lemon), novel scenarios that test generalization to uncom- mon yet feasible state changes (e.g., mashing grape- fruit), and compositional scenarios involving mul- tiple action compositions (e.g., peeling and slic- ing pear). In total, OSCBench comprises 1,120 prompts across 140 object-state scenarios, provid- ing a specific benchmark for evaluating OSC per- formance in T2V models. In addition, we evaluate six state-of-the-art (SOTA) T2V models on OSCBench, including four widely used open-source systems (Open-Sora- 2.0 (Peng et al., 2025), HunyuanVideo (Kong et al., 2024), HunyuanVideo-1.5 (Team, 2025), Wan-2.2 (Wan et al., 2025)) and two proprietary models (Kling-2.5-Turbo (KlingAI, 2025) and Veo- 3.1-Fast (Google DeepMind, 2025b)). We conduct both human user study and automatic evaluation using the latest multimodal large language mod- els (MLLMs). Across the two evaluation methods, we design a comprehensive set of criteria cover- ing semantic adherence, OSC performance, scene alignment, and perceptual quality. In particular, rather than using MLLMs as black-box scorers, we employ Chain-of-Thought (Wei et al., 2022) evalu- ation strategy that explicitly guides the reasoning process through criteria grounding, evidence ex- traction, and score justification. We further ana- lyze the correlation between human judgments and MLLM-based evaluations to assess the reliability of automated OSC evaluation. Our results in Fig- ure 1 (b) reveal that while SOTA T2V models gen- erally perform well on high-level semantic align- ment (e.g., subject, object, and scene), object state change accuracy and consistency remain a signifi- cant challenge. These findings position OSC as a critical diagnostic dimension that complements ex- isting evaluations. By revealing how state changes deviate from intended action effects, OSCBench provides practical guidance for building video gen- eration models that reason more faithfully about actions and their consequences. In summary, our contributions are three-fold: â˘We introduce OSCBench, the first benchmark explicitly designed to evaluate object state change in text-to-video generation across reg- ular, novel and complex scenarios. â˘We design a set of criteria covering seman- tic adherence, OSC performance, scene align- ment, and perceptual quality to comprehen- sively evaluate the video generation perfor- 2 mance with both human user study and auto- matic MLLM assessment. â˘We benchmark six SOTA T2V models, sys- tematically examine their performance across different OSC scenarios, and identify key chal- lenges that persist. The results offer guidance for designing models with OSC-aware genera- tion and outline directions for future research. 2 Related Work Benchmarks for Text-to-Video Generation. The rapid advancement of T2V models has motivated the development of benchmarks for accurate and reliable assessment. A number of recent bench- marks (Huang et al., 2024; He et al., 2024) aim to provide systematic evaluation of T2V models ei- ther from a comprehensive perspective or through specific aspects of generation quality. For ex- ample, VBench (Huang et al., 2024) and Eval- Crafter (Liu et al., 2024) target holistic evaluation across multiple interpretable dimensions, includ- ing temporal consistency, motion smoothness, and textâvideo alignment. To better diagnose particular modeling challenges, several aspect-specific bench- marks have been proposed. For example, T2V- CompBench (Sun et al., 2025) evaluates composi- tional generation capabilities, while DEVIL (Liao et al., 2024) focuses on the dynamic characteris- tics of generated videos. More recently, researchers have observed that T2V models frequently generate videos that violate physical constraints. This has motivated the development of benchmarks that ex- plicitly assess physical plausibility, such as Video- Phy (Bansal et al., 2024), PhyGenBench (Meng et al., 2024), and PhyWorldBench (Gu et al., 2025), which examine whether generated videos adhere to basic physical commonsense. Despite these ad- vances, existing benchmarks pay limited attention to OSC. In this work, we introduce a benchmark specifically for object state change, providing sce- narios that require accurate state modeling and en- abling targeted evaluation of a modelâs OSC under- standing. Evaluation Methods for Text-to-Video Models. Recent video benchmarks (Huang et al., 2024; Meng et al., 2024; Gu et al., 2025) commonly adopt a hybrid evaluation protocol that combines automatic model evaluation with human user study. For automatic evaluation, CLIP (Xue et al., 2024) and ViCLIP (Wang et al., 2023) based textâvideo similarity models are widely used to assess se- mantic alignment between prompts and generated videos. More recently, MLLMs have demonstrated strong abilities in understanding complex visual content (Ouyang et al., 2025; Zhang et al., 2025; He et al., 2025). Therefore, many video bench- marks (Feng et al., 2025; Motamed et al., 2025; Han et al., 2025) employ MLLMs to evaluate the semantic consistency in generated videos. Building on this capability, PhyWorldBench (Gu et al., 2025) further leverages MLLMs to evaluate whether gen- erated videos obey physical laws, which often requires multi-step reasoning. To evaluate fine- grained OSC, we leverage the reasoning capabil- ities of MLLMs and adopt a CoT strategy (Wei et al., 2022). Unlike existing benchmarks (Gu et al., 2025), which mainly use CoT to generate textual descriptions, we use it to guide models through a structured reasoning process, encouraging careful visual inspection and more reliable state-change judgments. 3 OSCBench Construction The goal of OSCBench is to provide a structured and comprehensive benchmark for evaluating ob- ject state change in text-to-video generation. De- signing such a benchmark requires addressing three key challenges: (i) covering realistic and diverse object state changes grounded in textual prompts, (i) ensuring controlled and balanced coverage of actions and objects to reduce dataset bias, and (i) introducing varying levels of difficulty to probe both memorization and generalization. In this sec- tion, we describe how OSCBench is constructed to meet these requirements. 3.1 Data Source and Abstraction Object state change is ubiquitous in everyday ac- tivities, with cooking being a representative do- mains. Cooking tasks naturally involve diverse state transformations, such as chopping, peeling, and heating, and exhibit clear causal relationships between actions and resulting object states. We therefore build OSCBench on the HowToChange dataset (Xue et al., 2024), which is derived from in- structional cooking videos in HowTo100M (Miech et al., 2019). HowToChange contains 20 fine- grained action elements and 134 object elements, yielding 409 distinct actionâobject combinations (e.g., slicing apple). However, these combinations exhibit a strong long-tail distribution: common pairs appear frequently (e.g., chopping potato), 3 Data Source and Abstraction CuttingSlicing Heating Mincing Frying Grilling PeelingPeeling ... ... VegeRoot FruitCitrus ... DairyMilk Leafy ... ... Potato Carrot Radish Beet Ginger ... Action Elements Object Elements Regular Object Category Action Category Cutting Root Slicing Carrot + Mincing Ginger ... ... Novel Novel Object Elements Action Elements Peeling Berry + Peeling Almond ... ... OSC ScenarioDes ign Compositional Peeling and Slicing Pear + Carrot ... Action Element Pairs Object Elements Prompt Construction Video Generation <subject> <action> <object> <scene> Generated Videos Evaluation Correlation Analysis â˘Semantic Adherence â˘Object State Change â˘Scene Alignment â˘Perceptual Quality Human Evaluation Automatic Evaluation + A man is chopping lettuce in the kitchen. Regular Novel Compositional A chef is chopping and sauteing chicken at a market stall. A woman in blue shirt is peeling berry outdoors. Evaluation Dimensions Experts Object Category Action Category Human-in-the-Loop Experts + + + + + + Figure 2: Overview of the OSCBench construction and evaluation pipeline. We build unified action and object categories from instructional cooking data via a human-in-the-loop process, and construct regular, novel, and compositional OSC scenarios as text prompts for video generation. The generated videos are evaluated by humans and MLLMs across multiple criteria, and we analyze their correlations to assess automatic evaluation reliability. while many plausible ones are rare or absent (e.g., squeezing ginger). Directly sampling from this distribution would bias evaluation toward frequent patterns and limit insights into generalization. To mitigate this issue, we reorganize the raw action and object elements into high-level concep- tual categories using a human-in-the-loop abstrac- tion process. Specifically, as shown in Figure 2, guided by cooking objectives, we first use GPT- 5.2 (OpenAI, 2025) and Gemini-3 (Google Deep- Mind, 2025a) to propose candidate groupings of the 20 action elements into 9 action categories (e.g., heating), and to cluster the 134 object elements into 8 major object categories (e.g., vegetable) with 28 finer-grained subcategories (e.g., root vegetables). These groupings are then iteratively refined and validated by human experts to ensure semantic cor- rectness and practical plausibility. This abstraction enables systematic scenario construction while pre- serving semantic diversity. 3.2 OSC Scenario Design Based on the abstracted action and object taxon- omy, we construct three complementary types of OSC scenarios to evaluate different aspects of OSC- aware video generation: regularity, generalization, and compositionality. Regular OSC Scenarios. Regular scenarios are designed to cover a broad range of realistic and commonly occurring object state changes. We pair each action category with compatible object subcat- egories to form candidate scenarios. All candidates are first filtered using automated checks by Chat- GPT and then validated by human review. This process yields 108 regular OSC scenarios. For each scenario, we further enumerate concrete instances by pairing specific action elements with object el- ements and manually select eight representative actionâobject combinations (e.g., mincing ginger), ensuring diversity while maintaining feasibility. Novel OSC Scenarios. To evaluate whether mod- els can reason about unfamiliar yet plausible object state changes, we introduce novel scenarios that deliberately deviate from common action-object combinations. For each of the 20 action elements, we select 8 uncommon yet feasible objects (e.g., peeling berries), resulting in 20 novel scenarios. These scenarios cannot be reliably solved through memorization of frequent actionâobject pairs and instead require models to infer state changes from action semantics. Compositional OSC Scenarios. Real-world ac- tivities often involve multiple actions applied se- quentially, where state changes evolves over time. To assess whether models can maintain coherent intermediate and final states, we construct com- positional scenarios by composing pairs of action elements (e.g., peeling followed by slicing). We se- lect 12 common action pairs, verified by human in- spection, and combine each pair with eight suitable objects (e.g., peeling and slicing potato). These scenarios explicitly examine multiple action com- position and temporal consistency for OSC-aware video generation. 3.3 Prompt Construction For every actionâobject combination in each sce- nario, we generate prompts using a structured tem- plate: <subject><action><object><scene>. We randomly generate three candidate prompts for each combination using GPT-5.2 and manually se- lect the most natural one. Examples can be seen 4 in Figure 2 (e.g., A man is chopping lettuce in the kitchen). In addition to full prompts with subjects and scenes, we further test how models respond when only object state change cues are provided. Specifically, we randomly simplify 1-2 prompts per scenario to the minimal form, <action><object>. This variant reduces contextual cues and places greater emphasis on the modelâs ability to infer and realize OSC directly from the action description. 3.4 Benchmark Statistics OSCBench comprises 140 object state change sce- narios in total, including 108 regular scenarios, 20 novel scenarios, and 12 compositional scenar- ios. Each scenario contains 8 actionâobject com- binations, resulting in 1,120 prompts overall. The prompts are concise and descriptive, with an aver- age length of 9.2 words, providing sufficient con- text while avoiding unnecessary linguistic com- plexity. We additionally provide a word cloud for OSCBench to illustrate the word distribution in the Appendix A. By combining structured abstraction, controlled scenario design, and multiple difficulty regimes, OSCBench enables systematic analysis of object state change performance in text-to-video models, covering both common patterns and chal- lenging generalization cases. 4 Evaluation Evaluating text-to-video generation models is in- herently challenging, particularly when the goal is to assess object state change specified by the prompt. A reliable evaluation must verify not only whether a generated video aligns with the prompt at a semantic level, but also whether the prompt- implied object state transition is realized accurately and consistently over time. While human evalua- tors can naturally perform such judgments, large- scale human use study is costly and difficult to scale. Following PhyWorldBench (Gu et al., 2025), we conduct both human user study and automatic assessment using multiple large language models. 4.1 Evaluation Dimensions We comprehensively evaluate generated videos along four complementary evaluation dimensions: semantic adherence, object state change, scene alignment, and perceptual quality. Semantic Adherence. This dimension measures whether the core semantic entities described in the prompt are faithfully grounded in the generated video. Specifically, we evaluate three key compo- nents independently: Subject alignment to measure whether the acting subject (e.g., a man or a woman) is present and correct, object alignment to evalu- ate whether the manipulated object matches the prompt and action alignment to assess whether the performed action corresponds to the intended ac- tion described in the prompt. Object State Change. This is the central dimen- sion of OSCBench. Evaluating object state change requires reasoning about both the outcome and the temporal evolution of the object. We therefore de- compose OSC evaluation into two sub-dimensions: state-change accuracy, which measures whether the object reaches the correct target state implied by the prompt (e.g., a whole apple becoming sliced), and state-change consistency, which assesses whether the transformation unfolds smoothly and coher- ently over time, without abrupt jumps or unnatural object appearances or unexplained appearance or disappearance of object parts. Scene Alignment.This dimension evaluates whether the global environment in the video matches the scene description in the prompt (e.g., kitchen or market). It focuses on the background context, such as whether the video clearly occurs in a kitchen or an outdoor market, and whether the scene remains stable and coherent over time. Perceptual Quality. This dimension measures the overall visual impression of the video and includes two aspects: realism, which measures whether the video resembles real-world footage in motion, light- ing, and texture, and aesthetic quality, which re- flects how visually appealing the video appears in composition, color, and overall presentation. 4.2 Human Evaluation We first conduct human user study as a strong ref- erence to evaluate our OSCBench. As exhaustive human evaluation over all generated videos would be prohibitively costly and time-consuming, we adopt a representative sampling strategy. Specif- ically, to cover the full diversity of OSCBench, we sample one prompt from each of the 140 OSC scenarios, ensuring that all regular, novel, and com- plex scenarios are represented. For each selected prompt, we generate one video for each T2V model, resulting in 140 videos per model for human evalu- ation. Each video is independently rated by three human evaluator across the evaluation dimensions described in Section 4.1. To encourage fine-grained and consistent judgments, we provide a 1-5 Likert 5 scale for each dimension. For each textâvideo pair, we average the three evaluatorâ scores to obtain the mean opinion score for each evaluation dimension. These human scores serve both as primary bench- mark results and as a reference signal for validating automatic evaluation using MLLMs. 4.3 MLLM-Based Automatic Evaluation Automatic evaluation using textâvideo similarity models (e.g., CLIP and ViCLIP) measures coarse semantic alignment but insufficient for assessing fine-grained object state changes and perceptual quality. MLLMs have recently shown strong visual understanding and multi-step reasoning abilities, which can serve as reasoning-based evaluators for video generation (Gu et al., 2025). Rather than treating MLLMs as black-box scorers, we design a CoT evaluation strategy that structures the reason- ing process. For each video and each evaluation dimension, the MLLM follows three steps:(1) Cri- teria grounding. The model restates the scoring criterion of each evaluation dimension in its own words, ensuring it internalizes the scoring defini- tion before examining the video. (2) Evidence extraction. The model then identifies frame-level visual evidence that is relevant to the criterion and briefly explains why these observations support its assessment. (3) Score decision. Based on the ex- tracted evidence, the model assigns a discrete score from 1 to 5 and explicitly links the score to the ob- served evidence. We provide the detailed prompt used for MLLM evaluation in the Appendix C. We apply this procedure to all adopted MLLMs across all evaluation dimensions. By constraining the reasoning route, the CoT strategy encourages the model to focus on fine-grained object states and their temporal evolution, rather than being dis- tracted by salient but irrelevant visual details. 5 Evaluation Results and Analysis 5.1 Experimental Setup We evaluate six representative SOTA T2V gener- ation models, including four widely used open- source systems (Open-Sora-2.0 (Peng et al., 2025), HunyuanVideo (Kong et al., 2024), HunyuanVideo- 1.5 (Team, 2025), and Wan-2.2 (Wan et al., 2025)) and two proprietary models (Kling-2.5- Turbo (KlingAI, 2025) and Veo-3.1-Fast (Google DeepMind, 2025b)). Detailed video generation settings are provided in the Appendix B. For auto- matic evaluation, we assess the generated videos 0.5 0.6 0.7 0.8 0.9 1.0 Qwen3-VL-30BGPT-5.2Human Veo-3.1-FastKling-2.5-TurboWan-2.2 HunyuanVideo-1.5HunyuanVideoOpen-Sora-2.0 Score Figure 3: Overall performance comparison of T2V mod- els based on aggregated evaluation scores from human evaluator and MLLM-based evaluators (Qwen3-VL- 30B and GPT-5.2). using ViCLIP for semantic similarity measurement as well as MLLM, including Qwen3-VL-30B, GPT- 5-mini, and GPT-5.2. For space considerations, we present GPT-5.2âbased evaluation results in the main paper and include results from other MLLMs in the Appendix C. All human and automatic evalu- ation scores are normalized to 0â1 for comparison. 5.2 Performance Comparison Figure 3 presents the overall evaluation results by averaging the scores of all evaluation dimensions based on Qwen-VL-30B, GPT-5.2 and human judg- ment. Among the evaluated models, Veo-3.1-Fast achieves the strongest overall performance, fol- lowed by Kling-2.5-Turbo, while open-source mod- els exhibit comparatively lower performance on av- erage. Tables 1 and 2 report the human evaluation and GPT5.2-based automatic assessment scores for each model across individual evaluation dimen- sion, respectively. Despite differences in absolute scores, both evaluation methods exhibit highly con- sistent trends across models. It can be observed that most models perform well on semantic adher- ence (particularly for subject and object) and scene alignment, but exhibit substantially lower scores on OSC accuracy and consistency. This discrepancy suggests that current T2V models are generally ca- pable of grounding high-level semantics from text, yet struggle to faithfully model the consequences of actions on object states over time. Notably, real- ism also remains challenging, particularly in terms of human evaluation, suggesting that limitations in accurately modeling object state changes are of- ten accompanied by residual visual artifacts, even when aesthetic quality is relatively strong. To further illustrate these findings, Figure 4 shows example videos generated by different mod- els for the same object-state-change prompt. In 6 Semantic AdherenceObject State Change Scene Alignment Perceptual Quality Model SubjectObjectActionAccuracyConsistancyRealismAesthetics Open-source models Open-Sora-2.0 (Peng et al., 2025)0.8600.7340.5180.3800.4280.7400.4160.540 HunyuanVideo (Kong et al., 2024)0.8680.8260.4940.4020.5100.8340.5260.688 HunyuanVideo-1.5 (Team, 2025)0.9140.9020.6560.5240.6080.8760.6180.730 Wan-2.2 (Wan et al., 2025)0.9040.8420.6160.5600.6680.8940.7020.818 Proprietary models Kling-2.5-Turbo (KlingAI, 2025)0.9380.9000.8260.7260.7260.8940.7320.836 Veo-3.1-Fast (Google DeepMind, 2025b)0.9360.9160.9080.7860.7480.8900.7520.874 Table 1: Human evaluation results of different T2V models across multiple evaluation dimensions. Semantic AdherenceObject State Change Scene Alignment Perceptual Quality Model SubjectObjectActionAccuracyConsistancyRealismAesthetics Open-source models Open-Sora-2.0 (Peng et al., 2025)0.9100.7220.6160.5120.6580.8920.6340.712 HunyuanVideo (Kong et al., 2024)0.8980.7640.5620.4660.7300.9480.7520.782 HunyuanVideo-1.5 (Team, 2025)0.9820.7880.6420.5460.7080.9360.7360.778 Wan-2.2 (Wan et al., 2025)0.9500.7740.5700.5180.7100.9740.7680.798 Proprietary models Kling-2.5-Turbo (KlingAI, 2025)0.9900.7920.7420.6520.6920.9720.7720.802 Veo-3.1-Fast (Google DeepMind, 2025b)0.9760.8340.8020.7400.7020.9780.7820.802 Table 2: GPT-5.2âbased evaluation results of T2V models on OSCBench across multiple evaluation dimensions. the first three models, the object state change is incorrect, where the apple is not sliced into pieces. Although videos generated by Wan-2.2, Kling-2.5- Turbo, and Veo-3.1-Fast successfully exhibit slic- ing behavior, they still suffer from issues in state change consistency or noticeable artifacts. For in- stance, Wan-2.2 shows a half-sliced apple reverting to a whole state (red box), Kling-2.5-Turbo pro- duces unreal interactions between the knife and the bowl (yellow box), and Veo-3.1-Fast introduces an additional apple chunk in the final frame (green box). Despite these issues, most models correctly render the subject, object, and scene, reinforcing the conclusion that high-level semantic alignment is substantially easier than accurate and consistent object state change modeling. 5.3 HumanâMLLM Correlation Analysis We analyze the correlation between human and automatic evaluation results to assess the reliabil- ity of MLLM-based evaluation. We report the model correlations with human evaluation in terms of KendallâsĎand SpearmanâsĎin Table 3, and include inter-evaluator agreement among human evaluators as a reference. Overall, MLLM-based evaluators exhibit substantially higher correlation with human judgments than the textâvideo simi- larity model ViCLIP across all evaluation dimen- sions, highlighting the advantage of multimodal reasoning over similarity-based scoring. Among all evaluated MLLMs, GPT-5.2 by incorporating the CoT evaluation strategy generally achieves the strongest overall agreement with human evalua- tion, indicating that explicitly structured reasoning 1.5 Prompt: A man is slicing apple in the kitchen. Figure 4: Sampled video frames generated by different T2V models. State change consistency or noticeable artifacts are highlighted in boxes. helps the model better identify fine-grained visual cues and state transitions. Despite these strengths, we also observe noticeably weaker correlations on perceptual-quality metrics (i.e., realism and aes- thetics) for GPT-5.2 with CoT, compared to other MLLMs. This gap likely reflects the inherently sub- jective nature of such judgments and indicates that fully automating perceptual assessment remains challenging. Besides, although human-MLLM cor- relations are still lower than humanâhuman agree- ment, Figure 3 shows that the MLLM-based eval- uation produces the same overall ranking of T2V systems as human. This suggests that automatic evaluation with MLLMs, while imperfect at the fine-grained scoring, is nevertheless reliable for as- sessing overall model performance trends at scale. 7 Metrics Semantic AdherenceObject State Change Scene Alignment Perceptual Quality SubjectObjectActionAccuracyConsistancyRealismAesthetics ĎĎĎĎĎĎĎĎĎĎĎĎĎĎĎĎ ViCLIP0.1060.1320.1950.2450.2880.386--------- Qwen3-VL-30B 0.4060.4130.4120.4290.5420.6240.4260.5030.2890.3410.1450.1490.2690.2970.4070.426 GPT-5-mini 0.433 0.4390.4280.4410.4780.5430.3420.3920.2430.2590.2000.2060.3030.338 0.5140.541 GPT-5.2 (w/o CoT)0.2950.3180.4090.4440.6230.7030.4150.4930.3030.3430.4250.447 0.323 0.3550.3930.411 GPT-5.20.3690.374 0.433 0.466 0.628 0.710 0.427 0.507 0.317 0.359 0.485 0.5050.2760.3180.3670.385 Human0.4680.4720.4840.5060.6360.7350.6030.6910.5010.5980.4920.5170.6130.7110.5810.647 Table 3: Correlation between human and MLLM-based automatic evaluations in terms of KendallâsĎand Spearmanâs Ď. The last row reports the mean inter-human correlation for reference. Models Object State Change Scenario Regular NovelCompositional Open-source models Open-Sora-2.0 (Peng et al., 2025) 0.4100.3890.416 HunyuanVideo (Kong et al., 2024)0.4720.4050.437 HunyuanVideo-1.5 (Team, 2025)0.5720.5590.556 Wan-2.2 (Wan et al., 2025) 0.6350.5310.594 Proprietary models Kling-2.5-Turbo (KlingAI, 2025) 0.7440.7140.699 Veo-3.1-Fast (Google DeepMind, 2025b)0.7970.7310.805 Table 4: Human-evaluated object state change scores of T2V models across regular, novel, and compositional scenarios, averaged over accuracy and consistency. 5.4 Category Analysis Table 4 presents the object state change perfor- mance of different T2V models across regular, novel, and compositional scenarios. Regular sce- narios mainly achieve the highest scores across all models, as they largely reflect common ac- tionâobject combinations that are well represented in training data. Novel scenarios exhibit the most severe performance degradation, indicating that current T2V models struggle to generalize state- change reasoning to uncommon but feasible ac- tionâobject pairs. In contrast, compositional sce- narios generally perform better than novel ones but worse than regular ones. This suggests that composing multiple familiar actions in sequence is less challenging than reasoning about unseen com- binations, yet still requires maintaining coherent intermediate states over time. To further investigate how different actions affect object state change performance, Figure 5 reports results across action categories. Models achieve higher scores on relatively simple actions with clear and visually salient transformations, such as rolling and heating (e.g, rolling dough or heating root vegetables), where state changes are localized and temporally straightforward. In contrast, per- formance drops substantially for actions involving complex handâobject interactions or subtle visual transitions, such as peeling, coating, and pressing (e.g., peeling carrot or coating shrimp). These ac- tions require precise manipulation and often in- 0.40 0.45 0.50 0.55 0.60 0.65 0.70 Score Figure 5: Object state change performance across action categories by human evaluation. Scores are averaged over accuracy and consistency. volve gradual appearance changes, making both the state change and its visual evidence more diffi- cult for models to capture. 6 Conclusion We have presented OSCBench, a benchmark for evaluating text-to-video generation with a focus on object state change. OSCBench systematically characterizes regular, novel, and complex state- transition scenarios, covering a broad spectrum of cooking activities. Using this benchmark, we evaluate six representative T2V models using both human user study and MLLM-based automatic assessment, and analyze the correlation between the two methods to assess the reliability of auto- matic evaluation. Our experiments demonstrate that existing models generally succeed at ground- ing high-level semantics and producing visually ap- pealing content, but they struggle to accurately and consistently model object state change over time. These limitations persist across regular, novel, and compositional scenarios, and are particularly pro- nounced for actions involving subtle or complex handâobject interactions. Overall, OSCBench, to- gether with our evaluation framework and empiri- cal analyses, reveals fundamental limitations of ex- isting T2V systems in modeling object state change, and provides a diagnostic foundation for develop- ing more state-aware and robust video generation models in future work. 8 Limitations While OSCBench provides a focused benchmark for evaluating object state change in text-to-video generation, it has several limitations. First, OS- CBench primarily focuses on cooking-related ma- nipulation scenarios, which offer clear and well- defined object state changes but do not fully capture the diversity of interactions found in other domains, such as tool use, household assembly, or outdoor activities. Although cooking covers a wide range of everyday manipulations, extending OSCBench to broader domains would further improve its gen- erality and applicability. Second, our evaluation emphasizes comparative and diagnostic analysis rather than exhaustive human annotation of all gen- erated videos, due to practical cost and scalability constraints. While our sampling strategy ensures balanced coverage across regular, novel, and com- positional scenarios, larger-scale human evaluation could reveal additional fine-grained failure modes that are not fully captured in the current setting. We view these limitations as opportunities for fu- ture work and hope that OSCBench will serve as a foundation for extending object state change evalu- ation to broader domains and more comprehensive assessment protocols. Ethical Considerations Our study employs human evaluation to assess video generation quality and to serve as a reference for validating the reliability of MLLM-based auto- matic scoring. A representative subset of generated videos was rated by human according to clearly defined criteria. Participants were informed about the study and provided informed consent prior to participation. Since the task involved only the eval- uation of model-generated videos, no personal or sensitive information was collected. The evaluation tasks did not expose participants to harmful or sensitive content. All prompts used in the benchmark were reviewed by the authors to ensure that no unsafe or dangerous material was included. Our work is conducted solely for research purposes and aims to improve the reliability and transparency of multimodal evaluation, rather than to create or promote harmful applications. Acknowledgments This research/project is supported by the Ministry of Education (MOE), Singapore, under its Aca- demic Research Fund (AcRF) Tier 2 (Proposal ID: T2EP20125-0048). Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not reflect the views of the Ministry of Education, Singapore. References Hritik Bansal, Zongyu Lin, Tianyi Xie, Zeshun Zong, Michal Yarom, Yonatan Bitton, Chenfanfu Jiang, Yizhou Sun, Kai-Wei Chang, and Aditya Grover. 2024. Videophy: Evaluating physical commonsense for video generation. In The Thirteenth International Conference on Learning Representations. Weixi Feng, Jiachen Li, Michael Saxon, Tsu-Jui Fu, Wenhu Chen, and William Yang Wang. 2025. Tc- bench: Benchmarking temporal compositionality in conditional video generation. In Findings of the As- sociation for Computational Linguistics: ACL 2025, pages 4638â4662. Google DeepMind. 2025a. Gemini 3. Google DeepMind. 2025b. Veo 3.1. Jing Gu, Xian Liu, Yu Zeng, Ashwin Nagarajan, Fan- grui Zhu, Daniel Hong, Yue Fan, Qianqi Yan, Kai- wen Zhou, Ming-Yu Liu, and 1 others. 2025. Phy- worldbench: A comprehensive evaluation of physi- cal realism in text-to-video models. arXiv preprint arXiv:2507.13428. Hui Han, Siyuan Li, Jiaqi Chen, Yiwen Yuan, Yuling Wu, Yufan Deng, Chak Tou Leong, Hanwen Du, Junchen Fu, Youhua Li, and 1 others. 2025. Video- bench: Human-aligned video generation benchmark. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 18858â18868. Xuan He, Dongfu Jiang, Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chan- dra, Ziyan Jiang, Aaran Arulraj, and 1 others. 2024. Videoscore: Building automatic metrics to simulate fine-grained human feedback for video generation. In Proceedings of the 2024 Conference on Empiri- cal Methods in Natural Language Processing, pages 2105â2123. Zhitao He, Sandeep Polisetty, Zhiyuan Fan, Yuchen Huang, Shujin Wu, and Yi R Fung. 2025. Mmbound- ary: Advancing mllm knowledge boundary aware- ness through reasoning step confidence calibration. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16427â16444. Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, and 1 others. 2024. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21807â21818. 9 KlingAI. 2025. Klingai. Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, and 1 others. 2024. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603. Mingxiang Liao, Qixiang Ye, Wangmeng Zuo, Fang Wan, Tianyu Wang, Yuzhong Zhao, Jingdong Wang, Xinyu Zhang, and 1 others. 2024. Evaluation of text- to-video generation models: A dynamics perspective. Advances in Neural Information Processing Systems, 37:109790â109816. Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. 2024. Evalcrafter: Benchmarking and evaluating large video generation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22139â22149. Guoqing Ma, Haoyang Huang, Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, Changyi Wan, Ranchen Ming, Xiaoniu Song, Xing Chen, and 1 others. 2025. Step-video-t2v technical report: The practice, chal- lenges, and future of video foundation model. arXiv preprint arXiv:2502.10248. Fanqing Meng, Jiaqi Liao, Xinyu Tan, Wenqi Shao, Quanfeng Lu, Kaipeng Zhang, Yu Cheng, Dianqi Li, Yu Qiao, and Ping Luo. 2024. Towards world simulator: Crafting physical commonsense-based benchmark for video generation. arXiv preprint arXiv:2410.05363. Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019. Howto100m: Learning a text-video embed- ding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2630â2640. Saman Motamed, Laura Culp, Kevin Swersky, Priyank Jaini, and Robert Geirhos. 2025. Do generative video models understand physical principles? arXiv preprint arXiv:2501.09038. OpenAI. 2025. Gpt-5.2. Kun Ouyang, Yuanxin Liu, Shicheng Li, Yi Liu, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. 2025. Punchbench: Benchmarking mllms in multimodal punchline comprehension. In Proceedings of the 63rd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 986â1008. Xiangyu Peng, Zangwei Zheng, Chenhui Shen, Tom Young, Xinying Guo, Binluo Wang, Hang Xu, Hongxin Liu, Mingyan Jiang, Wenjun Li, Yuhui Wang, Anbang Ye, Gang Ren, Qianran Ma, Wany- ing Liang, Xiang Lian, Xiwen Wu, Yuting Zhong, Zhuangyan Li, and 13 others. 2025. Open-sora 2.0: Training a commercial-level video generation model in $200k. arXiv preprint arXiv:2503.09642. Kaiyue Sun, Kaiyi Huang, Xian Liu, Yue Wu, Zihan Xu, Zhenguo Li, and Xihui Liu. 2025. T2v-compbench: A comprehensive benchmark for compositional text- to-video generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 8406â8416. Tencent Hunyuan Foundation Model Team. 2025. Hunyuanvideo 1.5 technical report. Preprint, arXiv:2511.18870. Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jix- uan Chen, Kai Zhu, Kang Zhao, Keyu Yan, and 43 others. 2025. Wan: Open and advanced large- scale video generative models. arXiv preprint arXiv:2503.20314. Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, and 1 others. 2023. Internvid: A large- scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06942. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elic- its reasoning in large language models. Advances in neural information processing systems, 35:24824â 24837. Zihui Xue, Kumar Ashutosh, and Kristen Grauman. 2024.Learning object state changes in videos: An open-world perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 18493â18503. Zicheng Zhang, Xiangyu Zhao, Xinyu Fang, Chunyi Li, Xiaohong Liu, Xiongkuo Min, Haodong Duan, Kai Chen, and Guangtao Zhai. 2025. Redundancy principles for MLLMs benchmarks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12492â12504. Association for Computational Linguistics. 10 In this appendix, we first present a detailed overview of OSCBench in Section A and describe the video generation settings in Section B. Sec- tion C provides the evaluation procedure and ad- ditional MLLM-based results. Finally, Section D offers illustrative examples of different object state change scenarios, together with analyses that facil- itate a deeper understanding of the benchmark. A OSCBench Details Data Abstraction Results. In OSCBench con- truction, we begin with categorizing the actions and objects in the HowToChange dataset. The tax- onomy is constructed through GPT-5.2âassisted grouping, cross-checked with Gemini-3, and sub- sequently subjected to human-in-the-loop review by human experts. The experts consist of three PhD-level researchers with extensive cooking expe- rience. Based on this process, actions are organized into a two-level hierarchy, whereas objects follow a three-level hierarchical structure. The resulting tax- onomy is shown in Figure 6 (a) and (b). Building on this taxonomy, we then design a complementary set of object state change scenarios. Word Distribution in OSCBench. We visualize the word distribution of all prompts in OSCBench using a word cloud, as shown in Figure 6 (c). This provides an intuitive overview of the dominant con- cepts and highlights the diversity of objects and actions represented in the benchmark. We further summarize the number of prompts and evaluation dimensions across different T2V benchmarks in Ta- ble 5. As shown, OSCBench explicitly emphasizes object state change, complementing existing bench- marks that primarily focus on semantic adherence or physical plausibility. Benchmarks#Prompt SAPQPCOSC VBench (Huang et al., 2024)1362â EvalCrafter (Liu et al., 2024)700â T2V-CompBench (Sun et al., 2025)1400â VideoPhy (Bansal et al., 2024)688â PhyGenBench (Meng et al., 2024)160â PhyWorldBench (Gu et al., 2025)1050â OSCBench (ours)1120â Table 5: Number of prompts and evaluation dimensions in different T2V generation benchmarks. We abbrevi- ate semantic adherence (SA), perceptual quality (PQ), physical commonsense (PC), and object state change (OSC). B Video Generation Setting We generate videos for all prompts in our bench- mark for each open-source T2V model. For each proprietary T2V model, we generate videos for the selected 140 prompt used in human evaluation. We follow the official and default implementations of T2V models in evaluation. Details of the video gen- eration setting of T2V models, including resolution, total frames, frames per second (FPS), and duration are presented in Table 6. For each generated video, we uniformly sample 20 frames for MLLM-based evaluation. For ViCLIP-based semantic similarity measurement, we uniformly sample 8 frames per video to align with the model architecture. ModelsResolutionFramesFPSDuration (s) Open-Sora-2.0768Ă768129255 HunyuanVideo1280Ă720129255 HunyuanVideo-1.51280Ă720121245 Wan-2.21280Ă72081165 Kling-2.5-Turbo1920Ă1080121245 Veo-3.1-Fast1280Ă720144246 Table 6: Generation settings of T2V models in terms of resolution, total frames, FPS, and duration. C Evaluation Details In this section, we provide additional details about our evaluation protocol. We first describe the scor- ing criteria and then report the results obtained from the MLLM-based evaluation, followed by an illustrative example of how the MLLMs reason and assign scores. Scoring Criteria. We adopt a hybrid evaluation protocol that combines human user study with au- tomated MLLM-based evaluation. Across these two evaluation modes, we design a comprehensive set of evaluation dimensions covering semantic ad- herence, OSC performance, scene alignment, and perceptual quality. For each dimension, we provide detailed scoring criteria for both human evaluator and MLLM-based evaluation. The instructions and scoring rubrics in the human user study interface are shown in Figure 8, according to which human evaluators are asked to rate each video on a scale from 1 to 5. For MLLM-based evaluation, the prompts we use are presented in Table 7. MLLM-based Evaluation Results. We report the additional evaluation results of Qwen3-VL-30B and GPT-5-mini in Table 8 and Table 9, respec- tively. Although their correlations with human eval- uation are not particularly high, the OSC accuracy 11 Cutting Heating Pressing Grating Mixing Coating Rolling Peeling Melting chopping mincing shredding slicing frying sauteing grilling roasting browning crushing mashing squeezing grating zesting whipping blending coating rolling peeling melting (a) Action taxonomy. Vegetables Fruits Proteins Dairy&Fats Grains&Carbs Sweets Nuts&Seeds Seasonings Leafy Root Bulb Stem&Stalk Fruiting Cruciferous Mushroom Citrus Berries Tropical Pome Stone Meat Processed Seafood Plant Eggs Milk Cheeses Fats Grains Carb Sweeteners Dessert Snacks Nuts&Seeds Herbs Condiments vegetable spinach lettuce kale leaf potato carrot radish beet ginger onion garlic shallot leek scallion celery asparagus tomato capsicum eggplant zucchini cucumber pumpkin squash okra corn jalapeno pepper broccoli cauliflower cabbage mushroom citrus orange lemon lime clementine grapefruit strawberry berry kiwi mango pineapple banana plantain coconut avocado watermelon apple pear peach date olive meat beef pork chicken turkey steak ham bacon meatball pepperoni sausage shrimp prawn salmon fish tofu chickpea bean egg milk cream yogurt cheese mozzarella paneer parmesan mascarpone butter ghee margarine shortening rice oat dough batter crust pasta bread biscuit cracker tortilla pastry sugar jaggery honey candy cake caramel ganache buttercream frosting fondant meringue marshmallow gelatin chocolate oreo cooky nut almond walnut cashew hazelnut pecan peanut basil cilantro parsley mint thyme dill rosemary herb chive coriander chili chilies nutmeg sauce ice clay (b) Object taxonomy. Vegetable Prep SHAPECOLORSFONTSLAYOUT Import Actions Columns TextSize Repeat Kitchen489 Chef287 Woman220 Man204 Bakery149 Robot140 Slicing108 Peeling104 Grating98 Blending93 Chopping91 Apron80 Melting72 Potato70 Crushing69 Gloves67 Food65 Frying60 Sauteing60 Garlic60 Whipping59 Cafe56 Kiwi50 Mashing49 Rolling48 Roasting47 Browning46 G43 Generate Save Order Download WORDS 2026/1/4 16:02Vegetable Prep - Edit - WordArt.com https://wordart.com/edit/m7em2213slln1/1 (c) Word cloud of OSCBench. Figure 6: Data abstraction results and word cloud in OSCBench. in both models is lower than their scores on sub- ject and object semantic adherence. This suggests that Qwen3-VL-30B and GPT-5-mini can capture part of the difficulty associated with object state change. Furthermore, the rankings of all T2V mod- els produced by the MLLMs are fully consistent with human judgments, as shown in Figure 3. This result indicates that automatic evaluation is reliable for large-scale model comparison and benchmark- ing, even though fine-grained per-instance scoring remains imperfect. To further illustrate how MLLMs interpret the videos, Figure 7 presents an example of MLLM evaluation on a generated video. Although the video appears visually appealing, human evalua- tors identify an OSC error: juice is dripping from the lemon, yet the lemon itself shows no visible squeezing or deformation. For this video, we ob- serve that Qwen3-VL-30B and GPT-5-mini assign perfect scores, indicating that they fail to detect the fine-grained issues in the lemonâs state change. GPT-5.2, in contrast, is able to detect the OSC error and provides reasonable supporting evidence, not- ing that âthe lemon largely remains undeformedâ. This suggests that more advanced MLLMs can to handle state-change reasoning and identify incon- sistencies between visual appearance and expected physical outcomes. Although GPT-5.2 w/o CoT also assigns a relatively low score to OSC accuracy, it still gives action alignment and OSC accuracy the same score, which shows weaker consistency with human judgments. From the human perspective, the error in action alignment is minor, whereas the error in OSC is a major one. This indicates that using CoT to plan a reasoning route, in which the model first follows the grading guidelines to collect explicit evidence and then assigns scores, encour- ages more careful evaluation and results in scores that better align with human judgments. D Examples of Different OSC Scenarios To provide a more intuitive view of T2V perfor- mance on object state change in OSCBench, we present examples of generated videos in regular, novel, and compositional OSC scenarios in Fig- ures 9, Figure 10, and Figure 11, respectively. We also present examples of generated videos from minimal prompts of the form <action><object> in Figure 12. In the regular OSC scenario shown in Figure 9, all models generate videos in which the subject (chef), action (slicing), object (leek), and scene (street food stand) are rendered well. However, clear errors emerge in the object state change. For example, in videos generated by Open-Sora-2.0, HunyuanVideo, HunyuanVideo-1.5, and Kling-2.5- Turbo, the leek is not actually sliced into pieces. Although videos produced by Wan-2.2 and Veo-3.1- Fast exhibit correct object state changes, the state change consistency in the later frames is remains limited. In the novel OSC scenario shown in Figure 10, some models can roughly understand the peeling action, but noticeable issues remain. The hand details in Open-Sora-2.0 are blurred. In Hunyuan- Video, HunyuanVideo-1.5, and Wan-2.2, the object being peeled is incorrect. In particular, Wan-2.2 generates olives, which are more commonly as- sociated with the peeling action. This suggests that these models are strongly influenced by mem- orized training patterns when generating object state changes for uncommon actionâobject com- binations, indicating an incomplete understanding of the intended action. Besides, Kling-2.5-Turbo 12 Suppose you are an expert in judging and evaluating the quality of AI-generated videos. Such videos may exhibit anomalies such as unnatural object appearance or disappearance, physically implausible state changes, and temporal inconsistencies across frames. They may also contain visual artifacts or unnatural textures. You are given 20 frames evenly sampled from a 5-second AI-generated video. Video Prompt: âA chef with a white apron is slicing leek at a street food stand.â Your Task: Analyze these frames chronologically and evaluate the video using the following criteria. Criteria Instructions: - Evaluate each criterion INDEPENDENTLY. - For each criterion, first identify the relevant factual evidence from the frames, then assign a score. Output Format: Return the result strictly in JSON format. "Subject Alignment": "evidence": "...", "score": [1-5], "Object Alignment": "evidence": "...", "score": [1-5], "Action Alignment": "evidence": "...", "score": [1-5], "OSC Accuracy": "evidence": "...", "score": [1-5], "OSC Consistency": "evidence": "...", "score": [1-5], "Scene Alignment": "evidence": "...", "score": [1-5], "Realism": "evidence": "...", "score": [1-5], "Aesthetics": "evidence": "...", "score": [1-5] Table 7: Prompt for MLLM to generate the evidence and score for each sampled video. Criteria are the same as those used in the human-evaluation interface. Semantic AdherenceObject State Change Scene Alignment Perceptual Quality Model SubjectObjectActionAccuracyConsistancyRealismAesthetics Open-source models Open-Sora-2.0 (Peng et al., 2025)0.9780.9320.7180.6560.7220.9800.7960.816 HunyuanVideo (Kong et al., 2024)0.9660.9540.6820.6480.7520.9880.8460.858 HunyuanVideo-1.5 (Team, 2025)0.9860.9580.7800.7560.8080.9900.8640.864 Wan-2.2 (Wan et al., 2025)0.9920.9520.7380.7120.8200.9920.8820.910 Proprietary models Kling-2.5-Turbo (KlingAI, 2025)0.9980.9720.9340.8820.9060.9980.9500.952 Veo-3.1-Fast (Google DeepMind, 2025b)0.9960.9880.9700.9540.9680.9960.9940.978 Table 8: Qwan3-VL-30B-based evaluation results of different T2V models across multiple evaluation dimensions. exhibits state change consistency issues, where two berries gradually collapse into one. Veo-3.1-Fast produces an OSC that is close to correct, but the ob- ject still shows jitter and artifacts in the last frame. Overall, the novel scenario remains challenging for current T2V models. In the compositional OSC scenario shown in Fig- ure 11, Open-Sora-2.0, HunyuanVideo, Wan-2.2, and Kling-2.5-Turbo execute only one of the re- quired actions. For example, Open-Sora-2.0, Wan- 2.2, and Kling-2.5-Turbo perform only the frying action. HunyuanVideo-1.5 attempts to handle both actions by cutting the ham while it is being fried, but the action is not the intended slicing action. Although Veo-3.1-Fast successfully completes the compositional actions, the consistency of the object states is poor. For example, a spatula suddenly ap- pears in frame 4, and the ham disappears in frame 5, revealing noticeable artificial artifacts. These results suggest that compositional OSC remains challenging for current T2V models. We also provide an example of generated videos with minimal prompts, <action><object>, in Fig- ure 12. We observe that models can often gener- ate object state changes even under such minimal prompts, suggesting that contextual cues such as subjects or scene descriptions are not strictly re- quired to trigger state-change behavior. In addi- tion, minimal prompts encourage models to focus more on the specified action and object. However, with respect to producing the correct object state change, only Veo-3.1-Fast generates the intended mashing action, while the outputs produced by all other models fail to follow the prompt accurately. This observation highlights the difficulty of accu- rate OSC generation for most existing T2V models. 13 Semantic AdherenceObject State Change Scene Alignment Perceptual Quality Model SubjectObjectActionAccuracyConsistancyRealismAesthetics Open-source models Open-Sora-2.0 (Peng et al., 2025)0.9180.8380.7120.7940.8140.9300.7220.774 HunyuanVideo (Kong et al., 2024)0.9600.9420.7920.7180.9560.9960.9080.900 HunyuanVideo-1.5 (Team, 2025)0.9340.9140.7940.5820.7880.9620.8340.788 Wan-2.2 (Wan et al., 2025)0.9460.9440.7500.7440.9700.9660.9300.946 Proprietary models Kling-2.5-Turbo (KlingAI, 2025)0.9960.9820.9760.9060.9980.9960.9801.000 Veo-3.1-Fast (Google DeepMind, 2025b)0.9920.9940.9940.9660.9980.9980.9580.984 Table 9: GPT-5-mini-based evaluation results of different T2V models across multiple evaluation dimensions. Prompt: A woman in casual clothes is squeezing lemon on the grass. Evaluator Subject Alignment Object Alignment Action Alignment OSC Accuracy OSC Consistency Scene Alignment RealismAesthetics Human4.74.74.33.34.04.73.74.7 Qwen3-VL-30B5.05.05.05.05.05.05.05.0 GPT-5-mini5.05.05.05.05.05.05.05.0 GPT-5.2 w/o CoT4.04.03.03.03.05.04.04.0 GPT-5.2 5.04.04.03.03.05.04.04.0 Subject Alignment Evidence from GPT-5.2 Evaluation Object AlignmentAction Alignment Acrossallframes,a womanisclearlyvisible sittingongrassinan outdoorfield,wearing casualclothing. OSCAccuracy Alemonremainsinherright handwhileliquiddrips down,andadditional lemonssitonaplatenear thebottomoftheframe. Thewomanholdsalemonappearsto squeezeit.However,thehand compressionofthelemonisminimal andthelefthandisnotclearly assistingthesqueeze. Expectedstatechangewouldbelemon beingsqueezed/partiallydeformedorcut andjuiceexpressed.Thevideoshowsjuice dripping,butthelemonlargelyremains undeformedandnotvisiblycutisshown. OSCConsistency Juicestreampersistsfairlyconsistently frame-to-frame,buttherearecontinuity issues:thelemonstaysvisuallywholewhile producingjuice,andinlaterframesan additionallemonappearsinherlefthand withoutaclearlyshownpickupmotion. Scene Alignment Backgroundconsistently depictsanoutdoor grassyareawithplantsa brightsky,matchingthe promptâsrequirement Realism Overalllooksclosetorealfootagewith naturallightinganddepthoffield.Minor AIcuesincludetoo-smoothskin, somewhatfrozenmotion,andthe physicallyquestionablecontinuous juicefromanapparentlywholelemon. Aesthetics Pleasantcomposition andcolorharmony (greensofgrass,blue sky).Outdoorsceneryis richandwell-lit;subject isvisuallyappealing. Figure 7: Example of human and MLLM evaluation on a generated video. Original human evaluation scores averaged over three evaluators are provided for reference. The evidence is generated by GPT-5.2 when scoring with CoT. 14 Task Instructions Pl ease use the s am e evaluation criter ia to score all videos and follow the definitions bel ow. 1. Read the promp t and watch the vi deo from start to finish. 2. Evaluate each criterion in depend ently. Exam ple: when scoring A ction Alignm ent, focus onl y on the corr ectness of the action, independent of the object and other attributes. 3. Use the 1â5 s cale cons istently across al l criteria and all videos. Evaluation Criteria 1. Sem an ti c Adh erence 1a S ubject Alig nment Is the subject present and cor rect (i.e., the m ain actor, e.g., a person or a hand)? (Pleas e focus only on the subject. Pl ease select "NA" if the prompt does not specify a subject.) 1.Very po or:Subject is absent or replaced by s omething entirel y unrelated. 2.P oor:Subject is present but does not m atch the expected category. 3.Fair:Subject is of the cor rect categor y but exhibits major attr ibute err or s. 4.Good:Subject is correct and well-rendered, with only m inor attribute errors. 5.Excellen t:Subject perfectl y m atches the prom pt in category, form, and attributes. 1b M anip ulat ed Obj ect Alig nment Is the m anipulated object pres ent and corr ect (e.g., car rots or tom atoes)? 1.Very po or:Manipul ated object is absent, or a completel y different object is present. 2.P oor:Manipul ated object is of the wrong category or is severel y distorted. 3.Fair:Manipul ated object is of the correct category but shows m ajor visual inaccuracies. 4.Good:Manipul ated object is correct and r ealistic, with onl y minor visual inaccuracies. 5.Excellen t:Manipul ated object is realistic and pr ovides a perfect visual m atch. 1c Acti on Alig nmen t D oes the per for med action match the action in the prompt (e.g., slicing or roasting)? 1.Very po or:A fundam entally different action is perform ed. 2.Poor:The intended action is recognizabl e but executed in a physicall y incorrect way. 3.Fair:The correct action is perfor med but with clear physical or logical flaws. 4.Good:Action is perfor med correctl y, but m otion appears slightly unnatural. 5.Excellen t:Action is executed in a physicall y plausible, natural m anner. 2. St ate Chang e P erformance 2a Ob ject St ate Chang e Accuracy Is the object state change correct and as expected (e.g., an apple changing from whol e to sl ices)? 1.Very po or:Object state change is illogical or unrelated to the action. 2.P oor:Object state change clearl y does not match the expected outcom e. 3.Fair:Object state change is partiall y cor rect, but major inaccuracies remain. 4.Good:Object state change is generally corr ect, with m inor issues. 5.Excellen t:Object state change is accur ate and matches the expected outcome exactly. 2b O bject Chan ge Conti nui ty & Cons istency Is the object state change continuous and natur al, without any unnatur al object appear ances or disappearances? 1.Very po or:State change is highly discontinuous, with obvious jumps or objects suddenly appearing/disappearing. 2.Poor:State change is discontinuous or has noticeabl e object appearances/disappearances. 3.Fair:State change is mos tly continuous but includes s mall jumps or object inconsistencies. 4.Good:State change is continuous and natur al, with only minim al , non-disruptive inconsistencies. 5.Excellen t:State change is sm ooth and continuous, with no unnatural object appearances/disappearances. 3.SCENE 3aScen e Align ment D oes the background and environm ent match the prom pt (e.g., a kitchen or a m ar ket)? (Pleas e focus only on the scene and environment. Pl ease select "NA " if the pr ompt does not specify a scene.) 1.Very po or:Scene dir ectly contradicts the prom pt. 2.Poor:Scene is generic or ambig uous and lacks r equired detail s. 3.Fair:Scene partially matches the prompt but contains notable attribute inaccur acies. 4.Good :Scene contains correct el em ents with onl y m inor attribute inaccuracies. 5.Excellen t:Scene is a detail ed and accurate match to the prompt's s etting. 4.P erceptual Q uality 4aRealism D oes this video look like a real-world video? 1.Very po or:Video l ooks artificial, distor ted, or obvious ly fake. 2.Poor:Many vis ual artifacts; motion, lighting, or textures do not r esembl e real footage. 3.Fair:Som e el em ents look real , but noticeable artifacts reduce overal l realism . 4.Good:Video appears cl ose to real with only minor visual im perfections. 5.Excellent:Video l ooks convincingly real with natural motion, lighting, and textures. 4bAest heti c Is the video visuall y appealing? Are the colors harmonious and is the content rich? 1.Very po or:Video is visual ly unappealing, with distracting colors or dull/empty content. 2.P oor:Som e attem pt at aesthetics, but colors clas h or the content feels sparse. 3.Fair:Overal l visually fine, with moderate har mony and adequate content richness. 4.Good:Visually appeal ing, with harmonious colors and rich, engaging content. 5.Excellen t:Highly pl easing visuals, s trong color harmony, and rich, well-composed content throughout. Figure 8: Task instructions and evaluation criteria in the human evaluation interface. 15 1.5 RegularScenario: A chef with a white apron is slicing leek at a street food stand. Figure 9: Sampled videos of different models in regular OSC scenario. 1.5 NovelScenario: A man in a white coat is peeling berry at a street food stand. Figure 10: Sampled videos of different models in novel OSC scenario. 16 1.5 CompositionalScenario: A robot is slicing and frying ham in an outdoor cooking area. Figure 11: Sampled videos of different models in compositional OSC scenario. 1.5 Minimal Prompt: Mashing pumpkin. Figure 12: Sampled videos of different models with minimal OSC prompt. 17