Paper deep dive
SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation
Keyu Tu, Zhuowei Chen, Mengqi Huang, Yuxin Wang, Jiahao Zhu, Zhendong Mao, Yongdong Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/19/2026, 4:55:41 AM
Summary
The paper introduces SemComp-Bench, a benchmark for Semantic Task Completion Video Generation, which evaluates whether generated videos achieve an instructed outcome while maintaining semantic grounding with a reference image. It presents SemComp-Data, a dataset constructed via a four-stage pipeline from raw videos, and SemComp-Bench, an evaluation protocol using Vision-Language Models to measure Outcome Achievement (OA) and Generation Reliability (GR) scores.
Entities (10)
Relation Signals (7)
SemComp-Bench → uses → SemComp-Data
confidence 96% · Building on SemComp-Data, SemComp-Bench uses evidence-grounded binary VLM judgments...
SemComp-Bench → evaluates → Semantic Task Completion Video Generation
confidence 95% · SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation
SemComp-Bench → measures → OA Score
confidence 95% · SemComp-Bench reports the OA Score... for Outcome Achievement
SemComp-Bench → measures → GR Score
confidence 95% · SemComp-Bench reports... the GR Score for... Generation Reliability
SemComp-Data → derivedfrom → Koala-36M
confidence 94% · Starting from full-context videos in Koala-36M [29], we construct SemComp-Data...
HunyuanVideo → evaluatedby → SemComp-Bench
confidence 92% · Experiments on representative video generation models show... HY... [in Table 1]
Wan2.2 → evaluatedby → SemComp-Bench
confidence 92% · Experiments on representative video generation models show... Wan2.2... [in Table 1]
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We introduce Semantic Task Completion Video Generation, an outcome-oriented video generation task. Under this formulation, success requires both achievement of the intended outcome and semantic grounding. Semantic grounding characterizes the correspondence between the reference image and the generated outcome in terms of high-level semantics relevant to the task. Evaluation focuses on the generated outcome and requires neither the presentation of a complete sequence of intermediate task steps nor conventional appearance consistency with the reference image. To support systematic evaluation, we construct SemComp-Data, an evaluation dataset covering six domains. Each instance comprises a reference image, a detailed instruction, a brief instruction, and an outcome-centric video clip. A scalable four-stage curation pipeline converts raw videos into standardized SemComp-Data instances. We further introduce SemComp-Bench, an evaluation protocol that uses a vision-language model (VLM) to answer structured binary questions. SemComp-Bench reports the OA Score and the GR Score for Outcome Achievement and Generation Reliability, respectively. Experiments on representative video generation models show that achieving intended outcomes while maintaining task-relevant semantic grounding in reference images remains challenging.
Tags
Links
- Source: https://arxiv.org/abs/2608.17426v1
- Canonical: https://arxiv.org/abs/2608.17426v1
Trouble viewing inline? Open PDF directly →
Full Text
69,006 characters extracted from source content.
Expand or collapse full text
SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation Keyu Tu 1 Zhuowei Chen 1 Mengqi Huang 1,2,† Yuxin Wang 2 Jiahao Zhu 3 Zhendong Mao 1 Yongdong Zhang 1 1 University of Science and Technology of China 2 FrameX.AI 3 Sun Yat-sen University † Corresponding author Abstract We introduce Semantic Task Completion Video Generation, an outcome-oriented video generation task. Under this formulation, success requires both achievement of the intended outcome and semantic grounding. Semantic grounding characterizes the correspondence between the reference image and the generated outcome in terms of high-level semantics relevant to the task. Evaluation focuses on the generated outcome and requires neither the presentation of a complete sequence of intermediate task steps nor conventional appearance consistency with the reference image. To support systematic evaluation, we construct SemComp-Data, an evaluation dataset covering six domains. Each instance comprises a reference image, a detailed instruction, a brief instruction, and an outcome-centric video clip. A scalable four-stage curation pipeline converts raw videos into standardized SemComp-Data instances. We further introduce SemComp-Bench, an evaluation protocol that uses a vision-language model (VLM) to answer structured binary questions. SemComp-Bench reports the OA Score and the GR Score for Outcome Achievement and Generation Reliability, respectively. Experiments on representative video generation models show that achieving intended outcomes while maintaining task-relevant semantic grounding in reference images remains challenging. Date: August 19, 2026 Project Page: https://SemComp-Bench.github.io Correspondence: Mengqi Huang at huangmq@ustc.edu.cn 1 Introduction Recent video generation models [4,27,28] achieve impressive visual fidelity and temporal coherence. How- ever, their ability to achieve an instructed outcome while maintaining high-level semantic grounding remains underexplored. We formulate this problem as Semantic Task Completion Video Generation. Here, grounding preserves task-relevant semantic relationships and reference attributes while allowing unrelated attributes to change. For example, given a banknote image and the instruction “Fold the banknote into a turtle,” a successful video must visibly present the referenced banknote as turtle-shaped origami, as illustrated in Fig. 1. Intermediate folding steps are unnecessary, but the result must remain grounded in the input banknote rather than replacing it with an unrelated turtle. Existing datasets and benchmarks emphasize visual fidelity, temporal coherence, and subject consistency, but rarely assess outcome achievement jointly with high-level semantic grounding. We therefore construct SemComp-Data, an evaluation dataset of image-text-video triplets curated from full-context real-world videos. 1 Intermediate Process Not Required Semantically Ungrounded Outcome Achieved & Semantically Grounded The Reference Frame The Outcome-Centric Video Clip Instruction: Fold the banknote into a turtle. Outcome Not Achieved Figure 1 Demonstration of Semantic Task Completion Video Generation, which focuses not on the transformation process but solely on outcome achievement and semantic grounding. Turn bare eye into glitter smokey eye Assemble into complete headphones Make the deer Halloween makeup Craft heart string art Twist rubber bands into bracelets Turn skeleton into real chimp Make egg-ham rolls Assemble melon- cheese-sauce skewer Figure 2 Representative SemComp-Data instances. Each example shows a reference frame, its outcome-centric video clip, and the brief instruction for compact presentation; every instance also includes a detailed instruction. Such full-context curation naturally ensures task feasibility while enabling fine-grained visual alignment. Each instance pairs the same reference and outcome with brief and detailed instructions describing the intended result at different levels of specificity. The curation pipeline comprises Candidate Filtering, State Mining, Video Extension, and Instruction Structuring. Candidate Filtering screens raw videos, State Mining localizes and verifies reference–outcome frames, and Video Extension extracts an outcome-centric clip around the localized outcome. Instruction Structuring then identifies alignment constraints and produces instructions. Figure 2 presents representative SemComp-Data instances. Building on SemComp-Data, SemComp-Bench uses evidence-grounded binary VLM judgments to assess two complementary dimensions: Outcome Achievement and Generation Reliability, reported as the OA Score and GR Score. The evaluator answers predefined questions and supports each answer with visual evidence, yielding focused and interpretable judgments. OA measures outcome realization and semantic grounding, with safeguards for task-relevant entity consistency and global visual continuity. GR complements it by assessing physical violations, blur, rendering artifacts, local instability, and corrupted text or interface elements. Together, the scores support systematic evaluation and criterion-level failure diagnosis. The main contributions of this work are summarized as follows: •We formulate Semantic Task Completion Video Generation, a new task that requires generated videos to achieve instructed outcomes while preserving task-relevant semantic relationships with reference images. •We construct SemComp-Data through a scalable, low-cost pipeline that converts raw videos into image- text-video triplets with paired brief and detailed instructions. •We introduce SemComp-Bench, a VLM-based evaluation protocol that measures Outcome Achievement and Generation Reliability using interpretable, evidence-grounded binary judgments. 2 2 Related Work 2.1 Video Generation Benchmarks Video generation benchmarks evaluate visual fidelity, temporal coherence, and prompt adherence [10,13,14, 22,34]. Personalized and controllable benchmarks further assess subject identity and intended motion [19,32], while recent work examines physical plausibility, causal consistency, and commonsense constraints [2,3,18, 21,31,33]. Together, these benchmarks cover generation quality, identity, motion, and physical plausibility. However, they do not directly test whether a video achieves an outcome jointly defined by an instruction and reference context while preserving task-relevant semantic grounding, motivating SemComp-Bench. 2.2 Video Generation Models Current models increasingly support high-quality, instruction-following generation. Open-source systems such as Wan2.2 [28], CogVideoX [30], HunyuanVideo [16], and Pyramid Flow [15] support text- and image- conditioned generation, while LTX [11,12] emphasizes efficiency. Closed-source systems, including Sora, Veo, MovieGen, Runway, Kling, Seedance, Hailuo, and Pika, further support multimodal conditioning, multi-shot narratives, camera control, and editing [1,4,9,17,23–26]. Together, these models show strong synthesis and control, but their outcome achievement under task-relevant reference grounding remains insufficiently understood. 3 SemComp-Data Construction Starting from full-context videos in Koala-36M [29], we construct SemComp-Data as a collection of structured image-text-video tripletsx i = (r i , I i ,o i ), whereI i = (i brief i ,i detailed i ) is an aligned instruction pair describing the same intended outcome at two levels of specificity. The reference framer i defines the initial context, the paired instructions specify the desired outcome, and the outcome-centric video clipo i depicts successful completion. Bothr i ando i are drawn from the same task instance, ensuring that the visual elements are genuinely associated rather than independently collected. This full-context construction provides evidence that the intended outcome is realizable and visually verifiable for the given reference. As illustrated in Fig. 3, we develop a four-stage curation pipeline to construct standardized evaluation instances from full-context videos. The pipeline supports scalable data construction without requiring the manual design of individual tasks or the production of corresponding outcome-centric videos. The following subsections describe each stage in detail. 3.1 Stage 1: Candidate Filtering We use the Koala-36M [29] dataset as the source pool, owing to its broad thematic coverage and strong representation of real-world scenarios. Because SemComp-Data relies on visually observable outcomes, we first apply Title-Based Keyword Filtering to exclude narration-dependent videos whose titles contain lexical patterns such as talk show, interview, and news, thereby retaining candidates that are more likely to be visually self-contained, as shown in Fig. 3(a). We uniformly sample frames from each retained video and arrange them into a mosaic representation, termed the video abstract, which provides a compact visual summary for efficient categorization. Using these abstracts, an advanced VLM assigns each video to one of six domains and an associated task category; videos lacking sufficient visual evidence for reliable categorization are labeled as Uncertainand discarded. The retained domains areArtsandPrecision,BeautyandFashion,CraftsandDIY, FoodandCooking,GardeningandPets, andSportsandFitness. For each task category, we manually define the reference and outcome states according to its characteristic completion pattern. These definitions specify what constitutes the initial context and the completed outcome for downstream state mining. The complete state definitions and the keyword list used for title-based filtering are provided in the supplementary material. 3.2 Stage 2: State Mining As shown in Fig. 3(b), this stage uses the manually specified category-specific state definitions from Stage 1 to mine reliable reference–outcome frame pairs through two steps: State Grounding and Quality Checking. We formulate State Grounding as frame-level timestamp localization rather than direct outcome-segment 3 ‘Summer hat crochet tutorial’ [Title] Normalized Relation Description [Brief Instruction] [Alignment Type] Attribute Selection (Preserve vs. Discard) [Outcome-Centric Video Clip] [Reference Frame] Instruction Composition [Full-Context Video] Outcome State Definition Reference timestamp:12.0s [Reference Frame] Outcome timestamp:197.0s [Outcome Frame] State Grounding [Full-Context Video] Visually Self-Contained Videos Uniform Temporal Sampling Narration-Dependent Videos Title-Based Keyword Filtering Video Abstract Categorization Category-Specific State Definitions (a) Stage 1: Candidate Filtering Reference State Definition (b) Stage 2: State Mining Quality Checking Outcome Timestamp [Outcome-Centric Video Clip] Outcome-Centric Video Extension [Full-Context Video] (c) Stage 3: Video Extension [Attributes to Preserve] [Attributes to Discard] (d) Stage 4: Instruction Structuring Outcome-Structured Video Candidates Domain Category Reference Frame Definition Outcome Frame Definition Reference Frame Reference Timestamp Outcome Frame Outcome Timestamp Brief Instruction ‘Crochet yarn into a summer hat’ Stage Outputs Detailed Instruction ‘The main subject is a finished light green and white marbled crocheted summer hat with a wide, floppy brim and a rounded crown. A pair of hands with red-painted nails gently holds and adjusts the hat, rotating it slightly to display its shape and texture. ....’ Outcome-Centric Video Clip [Outcome-Centric Video Clip] [Reference Frame] [Instruction] Data Triplets Inputs Figure 3 Overview of the four-stage SemComp-Data curation pipeline. Panels (a)–(d) show Candidate Filtering, State Mining, Video Extension, and Instruction Structuring, respectively. The right column shows the corresponding stage outputs, and the bottom row presents the final data triplet. localization, allowing the VLM to focus on representative visual evidence of the reference and outcome states without modeling the intermediate process or resolving ambiguous segment boundaries. Given a full-context video and its state definitions, the VLM first identifies the demonstrated task and expected outcome and then locates the timestamps that best match the two state definitions. The reference and outcome frames at these timestamps are extracted for subsequent processing. The extracted frame pair is then verified through QA-based Quality Checking, which serves as a conservative consistency screen. The VLM is provided with the two unlabeled frames and the task description obtained during State Grounding and is asked to identify which frame represents the outcome. A valid pair should allow the outcome frame to be identified unambiguously. If the predicted outcome does not match the grounded outcome frame, the pair is conservatively discarded, as the disagreement may reflect an ambiguous reference–outcome relationship, insufficient visual evidence, or VLM error. The remaining reference and outcome frames, together with their timestamps, constitute the outputs of this stage. 4 3.3 Stage 3: Video Extension As shown in Fig. 3(c), this stage takes the full-context video and the verified outcome timestamp from Stage 2 as inputs. Following the shot detection and same-scene merging procedure of Panda-70M [8], we partition the full-context video into segments with consistent camera viewpoints. Using the verified timestamp as a temporal anchor, we extract an outcome-centric clip (o i ) by selecting the core segment containing the outcome frame and merging visually consistent neighboring segments until a minimum duration of 3 s is reached. Compared with directly prompting a VLM to localize the outcome interval, this timestamp-anchored strategy more effectively centers the extracted clip on the completed outcome while offering greater efficiency and lower cost. The clips have an average duration of approximately 4.03 s in SemComp-Data, providing sufficient temporal evidence of the completed outcome while minimizing intermediate processes and unrelated content. The final outcome-centric clip constitutes the output of this stage. 3.4 Stage 4: Instruction Structuring As shown in Fig. 3(d), this stage takes the reference frame and outcome-centric video clip as inputs and produces two aligned instruction variants—a brief instructioni brief i and a detailed instructioni detailed i —through Normalized Relation Description, Attribute Selection, and Instruction Composition. In preliminary experiments, directly prompting the VLM with the two inputs often produces overly verbose instructions containing incidental details, such as brand names and on-screen text. To capture the essential reference–outcome relation, Normalized Relation Description constrains the VLM to generate a brief instructioni brief i of no more than 30 words following the template [Verb] + [Main Subject in r i ] + [Preposition] + [Main State in o i ]. In parallel, Attribute Selection determines which reference attributes should be preserved and which can be discarded during outcome-centric video generation. Because the task-relevant attributes vary across categories, the VLM first selects an applicable alignment type from four options:ObjectElement,PersonIdentity,Object Appearance, andScene. For example, person identity and task-relevant appearance cues should be preserved when the reference and outcome depict the same person with different makeup or hairstyles, whereas the spatial layout and design structure should be retained when transforming a blueprint into its corresponding physical result. Based on the selected alignment type, the VLM identifies the attributes to preserve and discard. The definitions and available options for the alignment types are provided in the supplementary material. Instruction Composition expands the brief instructioni brief i into the detailed instructioni detailed i by combining (i) the selected alignment type and preserve/discard attributes, which specify the reference-grounding constraints, and (i) fine-grained visual characteristics that describe the completed outcome, including its background, lighting, colors, shapes, and component-level details. These outcome characteristics do not constitute reference- appearance constraints unless they are explicitly selected for preservation. The resulting detailed instruction jointly specifies the desired outcome and its reference-grounding requirements, making it suitable for evaluation and potentially useful for future task-specific training. The two variants describe the same task instance at different levels of specificity, enabling controlled evaluation of instruction specificity without changing the underlying reference or target outcome. The brief instruction better reflects typical user prompting habits and therefore provides a more challenging evaluation setting. Figure 4 summarizes the alignment-type and domain distributions of SemComp-Data, together with high- frequency words in the brief instructions. 5 (c) Domain Distribution (b) High-Frequency Words in Brief Instructions (a) Domain-wise Distribution of Alignment Types Figure 4 Statistics of SemComp-Data: (a) distribution of alignment types across domains, with distinct textures denoting different types; (b) high-frequency words in the instructions; and (c) distribution of evaluation instances across the six domains. ModelsA or A sg A gec A gvc OA Score Seedance 2.0 [5]0.8390.7440.4440.59420.0% Wan2.2-TI2V-5B [28]0.5890.4000.6890.92223.3% Wan2.2-I2V-A14B [28]0.8000.5280.628 0.78928.3% CogVideoX1.5-5B-I2V [30]0.5500.3890.5060.74414.4% SkyReels-V2-I2V-14B-720P [7]0.7330.4890.5220.77222.8% HY † -1.5-720P-I2V 0.8780.706 0.5830.79437.8% Phantom-1.3B [20]0.5390.3560.3220.5113.9% Table 1 SemComp-Core Outcome Achievement with detailed instructions.A or ,A sg ,A gec , andA gvc denote pass rates for outcome realization, semantic grounding, grounded entity consistency, and global visual continuity, respectively. The OA Score is the joint pass rate across four criteria. † HY denotes HunyuanVideo [27]. Bold and underlined values indicate the best and second-best values, respectively. 4 SemComp-Bench Evaluation Building on SemComp-Data, SemComp-Bench evaluates generated videos through structured binary questions, each targeting a specific criterion for interpretable assessment and failure diagnosis. The criteria are organized into two complementary dimensions: Outcome Achievement (OA), which measures outcome realization, reference grounding, and task-relevant continuity; and Generation Reliability (GR), which assesses visual, physical, and temporal reliability. Their aggregate results are reported as the OA Score and GR Score, respectively. For a single evaluation run on a set ofNsamples, leta i c ,g i c ∈ 0,1denote the binary criterion scores of sample ifor criterioncin the Outcome Achievement and Generation Reliability dimensions, respectively, where 1 indicates a pass and 0 otherwise. The full set of binary questions is provided in the supplementary material. 4.1 Outcome Achievement For each samplei, outcome achievement is evaluated using four criteria: outcome realization, semantic grounding, grounded entity consistency, and global visual continuity. Their binary criterion scores area i or , a i sg ,a i gec , anda i gvc , respectively. All four criteria are evaluated on the same temporally ordered sequence of 27 frames uniformly sampled from each generated video. Outcome realization and semantic grounding additionally use the reference image and instruction, whereas grounded entity consistency and global visual continuity use only the sampled sequence. The corresponding dataset-level pass rates areA or ,A sg ,A gec , and A gvc , where A c = 1 N P N i=1 a i c . The outcome realization criterion assesses whether the generated video visibly reaches the completed state specified by the instruction at a coarse semantic level, independent of fine-grained correspondence to the reference image. The semantic grounding criterion assesses whether the realized outcome preserves or modifies task-relevant entities and attributes according to the reference-instruction pair. The grounded entity consistency criterion assesses whether task-relevant key entities remain semantically identifiable throughout the sampled sequence, without unexplained disappearance, replacement, or unintended drift in identity, material, appearance, or task-relevant structure. The global visual continuity criterion examines the temporally ordered sampled sequence for abrupt global switches in scene, viewpoint, layout, background, or composition. A 6 ModelsG p G vc G afr G wsc G ti GR Score Seedance 2.0 [5]0.8830.9940.9940.7390.97891.8% Wan2.2-TI2V-5B [28]0.7940.978 0.8890.7220.88985.4% Wan2.2-I2V-A14B [28]0.9110.9720.9670.6720.92889.0% CogVideoX1.5-5B-I2V [30]0.9440.8110.7720.5830.82878.8% SkyReels-V2-I2V-14B-720P [7]0.7780.9220.7390.3280.78971.1% HY † -1.5-720P-I2V 0.8000.9390.8830.4720.86179.1% Phantom-1.3B [20]0.7780.9720.8560.5060.72876.8% Table 2 SemComp-Core Generation Reliability with detailed instructions. EachG c denotes the pass rate for a reliability criterion, and the GR Score is the mean across five criteria. † HY denotes HunyuanVideo [27]. Bold and underlined values indicate the best and second-best values. ModelModality Instruction Type A or A sg A gec A gvc OA Score Wan2.2-A14B [28] I2VDetailed0.800 0.528 0.628 0.78928.3% T2VDetailed0.889 0.522 0.317 0.1174.4% T2VBrief0.389 0.111 0.806 0.2500.6% CogVideoX1.5-5B [30] I2VDetailed0.550 0.389 0.506 0.74414.4% T2VDetailed0.567 0.272 0.361 0.1615.0% T2VBrief0.567 0.150 0.678 0.1610.6% HY † -1.5-720P I2VDetailed0.878 0.706 0.583 0.79437.8% T2VDetailed0.833 0.606 0.389 0.1334.4% T2VBrief0.550 0.100 0.761 0.1781.7% Table 3 Outcome Achievement on SemComp-Core across modalities and instruction settings. Within each model family, the I2V and T2V variants use checkpoints of the same parameter scale. EachA c denotes a criterion-level pass rate, and the OA Score is the joint pass rate. † HY denotes HunyuanVideo [27]. common, but not exclusive, failure case occurs when a reference-matching initial frame is followed by a visually disconnected continuation. These two criteria serve as validity safeguards for the generated outcome and do not assess the completeness or procedural correctness of intermediate task steps. All four indicators are pass-coded, with 1 denoting that the corresponding criterion is satisfied. For the two failure-oriented questions on entity inconsistency and global visual discontinuity, a “No” response is mapped to 1. Overall success is conjunctive: the sample-level Outcome Achievement indicatorA i equals 1 only when sample i passes all four criteria. The dataset-level OA Score is the mean of A i across all samples: A i = a i or a i sg a i gec a i gvc ∈ 0, 1, OA Score = 1 N N X i=1 A i . (1) 4.2 Generation Reliability For each samplei, the VLM evaluates generation reliability independently of outcome achievement and reference grounding using five binary criteria: physical plausibility, visual clarity, artifact-free rendering, within-scene spatiotemporal coherence, and text and interface integrity. Their binary criterion scores areg i p , g i vc ,g i afr ,g i wsc , andg i ti , respectively. The corresponding dataset-level pass rates areG p ,G vc ,G afr ,G wsc , and G ti , where G c = 1 N P N i=1 g i c . The physical plausibility criterion assesses visible motions and interactions for clear violations of basic physical principles; it neither requires the instructed transformation process to be shown nor treats an omitted 7 intermediate process as a failure. The visual clarity criterion identifies content that is difficult to recognize because of blur, improper exposure, insufficient resolution, or unclear scene details. The artifact-free rendering criterion detects scene-inconsistent synthetic artifacts, such as abnormal noise, corrupted regions, blank patches, or color streaks. The within-scene spatiotemporal coherence criterion detects low-level rendering instability within continuous scenes, such as flicker, transient duplication, and local geometric deformation, irrespective of task semantics. Unlike grounded entity consistency, this criterion concerns local rendering stability rather than the semantic persistence of task-relevant entities. Abrupt global switches across the sampled sequence are evaluated separately by global visual continuity in the Outcome Achievement dimension. The text and interface integrity criterion detects corrupted or unrecognizable text, numbers, icons, and interface elements. All five questions are failure-oriented, and the reported indicators are pass-coded: a “No” response is mapped to 1. The sample-level Generation Reliability scoreG i is the arithmetic mean of the five binary criterion scores, and the dataset-level GR Score is the mean of G i across all samples: G i = 1 5 g i p + g i vc + g i afr + g i wsc + g i ti ∈ [0, 1], GR Score = 1 N N X i=1 G i . (2) Together, the criterion-level pass rates identify specific sources of generation unreliability. 5 Experiments 5.1 Experimental Setup We randomly sample approximately 20K videos from Koala-36M [29] and process them using the curation pipeline, yielding 1,273 structured evaluation instances. To construct a domain-balanced evaluation subset, SemComp-Core comprises 60 instances, with 10 selected from each domain via stratified sampling to approxi- mately preserve the within-domain alignment-type distributions. Each SemComp-Core instance includes both instruction variants. In analyses of instruction specificity, the paired variants share the same reference image and outcome target, so only the level of instruction specificity varies. Representative open- and closed-source video generation models [5,7,20,27,28,30] are evaluated on this subset. All models in Tables 1 and 2 use I2V with the reference image and detailed instruction; Wan2.2-TI2V-5B runs in image-conditioned mode. Open-source models use their official default inference configurations, whereas closed-source models use their exposed default settings; a fixed seed is applied whenever explicit seed control is available. Under every setting, one 720p video is generated per instance, from which 27 frames are uniformly sampled over the full temporal extent to form a fixed-length, temporally ordered sequence for evaluation. Every video is scored through three independent VLM calls made at different times, with all criterion-level pass rates and both aggregate scores computed per call using the definitions above; the reported results are arithmetic means of the three corresponding run-level scores. We instantiate the VLM used in the curation and evaluation experiments as Doubao-Seed-1.8 through the Volcano Engine API [6], as preliminary experiments indicate that it offers a favorable trade-off between evaluation efficiency and cost. 5.2 Outcome Achievement Results Table 1 reports the OA Score under detailed instructions. HunyuanVideo-1.5-720P-I2V achieves the highest OA Score of 37.8%, followed by Wan2.2-I2V-A14B at 28.3%. Their results are consistent with relatively balanced pass rates across the four OA criteria, rather than dominance in any single criterion. Seedance 2.0 performs strongly in outcome realization and semantic grounding, but its lower pass rates on grounded entity consistency and global visual continuity reduce its joint OA Score. In contrast, Wan2.2-TI2V-5B achieves the highest pass rates on grounded entity consistency and global visual continuity while performing less strongly in outcome realization and semantic grounding. These complementary performance profiles, together with the fact that the best OA Score remains below 40%, indicate that robust outcome achievement requires both task fidelity and outcome-video validity. 8 5.3 Generation Reliability Results Table 2 reports generation reliability under detailed instructions. Seedance 2.0 achieves the highest GR Score of 91.8%. Among the evaluated open-source models, Wan2.2-I2V-A14B performs best at 89.0%. Both models perform strongly in visual clarity and artifact-free rendering. However, within-scene spatiotemporal coherence remains the primary bottleneck across all models, with pass rates ranging from only 0.328 to 0.739. This gap suggests that current models can often generate visually convincing individual frames but struggle to maintain stable visual evolution within scenes. Comparing the GR and OA rankings further indicates that generation reliability does not necessarily correspond to outcome achievement: Seedance 2.0 leads in GR but performs substantially worse in OA, whereas HunyuanVideo-1.5-720P-I2V leads in OA despite a lower GR ranking. Wan2.2-I2V-A14B ranks second on both metrics, indicating comparatively balanced performance across the two evaluation dimensions. 5.4 Conditioning Effects As shown in Table 3, I2V variants consistently outperform their T2V counterparts under detailed instructions across all three model families. The gains mainly arise from improved semantic grounding, grounded entity consistency, and global visual continuity, while outcome-realization pass rates remain broadly comparable and are higher for T2V in two of the three model families. These findings underscore the indispensable role of reference-image conditioning in semantic task completion, particularly in preserving entity identity, appearance, and structural consistency during complex object interactions and state changes. Within the T2V setting, detailed instructions consistently achieve higher OA Scores than brief instructions, primarily through better outcome realization and semantic grounding. In contrast, brief instructions often yield higher grounded entity consistency and global visual continuity, likely because their simpler and less constrained content is easier to generate coherently. These results reveal a trade-off between instruction specificity and generation difficulty: detailed instructions better define the intended compositional outcome but place greater demands on semantic understanding and coordinated generation, whereas brief instructions are associated with higher outcome-video consistency but lower task-fulfillment rates. 5.5 Visualization of Generated Videos As shown in Fig. 5, the evaluated models tend to preserve the appearance of the reference image and depict the process of task execution, yet often fail to achieve the intended outcome. Moreover, generated videos often suffer from reliability issues, including physically implausible transformations and abrupt visual transitions, as illustrated by the failure cases of Wan2.2-I2V-A14B and Phantom-1.3B shown in the figure. Additional visualizations of generated videos are provided in the supplementary material. 9 [Detailed Instruction] Generate a video that reaches the following state: a pair of hands uses two ropes—one blue with white and red speckles and the other solid orange—to tie a double sheet bend knot. The hands thread the orange rope through a loop formed by the blue rope and wrap it around the loop to form the knot. They then pull the ends of both ropes to tighten the knot, adjusting the loops and strands to ensure a secure connection. The background is a plain, neutral gray surface that keeps the focus on the ropes and the knot-tying process. The category, number, color palette, and material of the ropes should remain consistent with those in the reference image, while their initially coiled state and spatial arrangement may change as needed to form the knot. [The Reference Image] The Outcome-Centric Video Clip Seedance 2.0 Wan2.2-TI2V-5B Wan2.2-I2V-A14B CogVideoX1.5-5B-I2VSkyReels-V2-I2V-14B-720P HunyuanVideo-1.5-720P-I2V Phantom-1.3B Figure 5 Generated-video examples. The outcome-centric video clip and the reference image are extracted from the same full-context video. 6 Conclusion In this paper, we introduce Semantic Task Completion Video Generation, which requires generated videos to realize an instructed outcome while maintaining semantic grounding in the reference context. We develop SemComp-Data through a scalable four-stage curation pipeline, with each instance comprising a reference image, paired brief and detailed instructions, and an outcome-centric video clip. Crucially, the paired image and clip are extracted from the same full-context video, ensuring task authenticity and achievability as well as fine-grained attribute alignment between the reference and target, while the dataset preserves diverse real-world scenarios from the source collection. Building on this dataset, SemComp-Bench is the first benchmark to jointly evaluate Outcome Achievement and Generation Reliability. Experiments with representative video generation models reveal substantial limitations in completing instructed tasks and preserving reference-grounded semantics, indicating that this capability remains largely underexplored. The potential of SemComp-Data to improve model performance through task-specific training remains to be empirically validated. 10 References [1] Kuaishou AI. Kling ai: Ai-powered video generation model. https://klingai.com, 2025. [2] Zechen Bai, Hai Ci, and Mike Zheng Shou. Impossible videos.arXivpreprintarXiv:2503.14378, 2025. [3]Hritik Bansal, Clark Peng, Yonatan Bitton, Roman Goldenberg, Aditya Grover, and Kai-Wei Chang. Videophy-2: A challenging action-centric physical commonsense evaluation in video generation.arXivpreprintarXiv:2503.06800, 2025. [4] ByteDance. Seedance: Multimodal video generation model. https://seed.bytedance.com, 2025. [5]ByteDance Seed.Seedance 2.0 official launch.https://seed.bytedance.com/en/blog/ seedance-2-0-official-launch, 2026. [6]Bytedance Seed. Seed1.8 Model Card: Towards Generalized Real-World Agency.arXivpreprintarXiv:2603.20633, 2026. [7]Guanghui Chen et al. Skyreels-v2: Infinite-length film generative model.arXivpreprintarXiv:2504.13074, 2025. [8]Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. InProceedingsoftheIEEE/CVFConferenceonComputerVisionandPattern Recognition, pages 13320–13331, 2024. [9] Google DeepMind. Veo: A state-of-the-art video generation model.https://deepmind.google/technologies/ veo/, 2025. [10] Weixi Feng, Jiachen Li, Michael Saxon, Tsu-jui Fu, Wenhu Chen, and William Yang Wang. Tc-bench: Benchmarking temporal compositionality in text-to-video and image-to-video generation.arXivpreprintarXiv:2406.08656, 2024. [11]Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, et al. Ltx-video: Realtime video latent diffusion.arXivpreprintarXiv:2501.00103, 2024. [12]Yoav HaCohen, Benny Brazowski, Nisan Chiprut, Yaki Bitterman, Andrew Kvochko, Avishai Berkowitz, Daniel Shalem, Daphna Lifschitz, Dudu Moshe, Eitan Porat, et al. Ltx-2: Efficient joint audio-visual foundation model. arXivpreprintarXiv:2601.03233, 2026. [13]Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. InProceedingsoftheIEEE/CVFConferenceonComputerVisionandPatternRecognition, pages 21807–21818, 2024. [14]Ziqi Huang, Fan Zhang, Xiaojie Xu, Yinan He, Jiashuo Yu, Ziyue Dong, Qianli Ma, Nattapol Chanpaisit, Chenyang Si, Yuming Jiang, et al. Vbench++: Comprehensive and versatile benchmark suite for video generative models. IEEETransactionsonPatternAnalysisandMachineIntelligence, 2025. [15] Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling. InInternationalConference onLearningRepresentations, volume 2025, pages 23378–23402, 2025. [16]Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXivpreprint arXiv:2412.03603, 2024. [17] Pika Labs. Pika: Ai video generation platform. https://pika.art, 2025. [18] Dacheng Li, Yunhao Fang, Yukang Chen, Shuo Yang, Shiyi Cao, Justin Wong, Michael Luo, Xiaolong Wang, Hongxu Yin, Joseph Gonzalez, et al. Worldmodelbench: Judging video generation models as world models. AdvancesinNeuralInformationProcessingSystems, 38, 2026. [19]Xinran Ling, Chen Zhu, Meiqi Wu, Hangyu Li, Xiaokun Feng, Cundian Yang, Aiming Hao, Jiashu Zhu, Jiahong Wu, and Xiangxiang Chu. Vmbench: A benchmark for perception-aligned video motion generation. InProceedings oftheIEEE/CVFInternationalConferenceonComputerVision, pages 13087–13098, 2025. 11 [20]Lijie Liu, Tianxiang Ma, Bingchuan Li, Zhuowei Chen, Jiawei Liu, Gen Li, Siyu Zhou, Qian He, and Xinglong Wu. Phantom: Subject-consistent video generation via cross-modal alignment.arXivpreprintarXiv:2502.11079, 2025. [21]Mingxin Liu, Shuran Ma, Shibei Meng, Xiangyu Zhao, Zicheng Zhang, Shaofeng Zhang, Zhihang Zhong, Peixian Chen, Haoyu Cao, Xing Sun, et al. Rise-video: Can video generators decode implicit world rules?arXivpreprint arXiv:2602.05986, 2026. [22] Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. In ProceedingsoftheIEEE/CVFconferenceoncomputervisionandpatternrecognition, pages 22139–22149, 2024. [23] MiniMax. Hailuo ai video generation model. https://hailuoai.video, 2025. [24]OpenAI.Video generation models as world simulators.https://openai.com/research/ video-generation-models-as-world-simulators, 2024. [25]Adam Polyak, Amit Zohar, Andrew Brown, et al. Movie gen: A cast of media foundation models.arXivpreprint arXiv:2410.13720, 2024. [26] Runway. Introducing runway gen-4. https://runwayml.com, 2025. [27] Tencent Hunyuan Foundation Model Team. Hunyuanvideo 1.5 technical report, 2025. URLhttps://arxiv.org/ abs/2511.18870. [28]Wan Team. Wan: Open and advanced large-scale video generative models.arXivpreprintarXiv:2503.20314, 2025. [29]Qiuheng Wang, Yukai Shi, Jiarong Ou, Rui Chen, Ke Lin, Jiahao Wang, Boyuan Jiang, Haotian Yang, Mingwu Zheng, Xin Tao, et al. Koala-36m: A large-scale video dataset improving consistency between fine-grained conditions and video content. InProceedingsoftheComputerVisionandPatternRecognitionConference, pages 8428–8437, 2025. [30]Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. In InternationalConferenceonLearningRepresentations, volume 2025, pages 83048–83077, 2025. [31] Jianhao Yuan, Fabio Pizzati, Francesco Pinto, Lars Kunze, Ivan Laptev, Paul Newman, Philip Torr, and Daniele De Martini. Likephys: Evaluating intuitive physics understanding in video diffusion models via likelihood preference.arXivpreprintarXiv:2510.11512, 2025. [32] Shenghai Yuan, Xianyi He, Yufan Deng, Yang Ye, Jinfa Huang, Bin Lin, Jiebo Luo, and Li Yuan. Opens2v-nexus: A detailed benchmark and million-scale dataset for subject-to-video generation.arXivpreprintarXiv:2505.20292, 2025. [33] Yuke Zhao, Wangbo Zhao, Weijie Wang, Zeyu Zhang, Dakai An, Akide Liu, Yinghao Yu, Jiasheng Tang, Fan Wang, Wei Wang, et al. Worldolympiad: Can your world model survive a triathlon?arXivpreprintarXiv:2606.11129, 2026. [34] Dian Zheng, Ziqi Huang, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Lulu Gu, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng, et al. Vbench-2.0: Advancing video generation benchmark suite for intrinsic faithfulness.arXiv preprintarXiv:2503.21755, 2025. 12 Supplementary Material The supplementary material provides further details on SemComp-Data and SemComp-Bench. Specifically, it describes the title-filtering procedure (Sec. A), the domain and category taxonomy (Sec. B), category-specific definitions of reference and outcome states (Sec. C), alignment types and preservation attributes (Sec. D), and the complete evaluation prompt (Sec. E). It also presents dataset statistics and instances (Sec. F), additional generated-video comparisons (Sec. G), and the standard deviations of the reported results (Sec. H). The complete curation pipeline and its stage-specific prompts are provided in the Code folder of the Code and Data Supplementary materials, while the Data folder provides the metadata, source-video URLs, and timestamps required to reconstruct SemComp-Data. Configuration GroupKeyword Strings news_keywords (13)News; Briefing; Coverage; Current Affairs; Deep Dive; Documentary; Exclusive; Headline; Interview; Live; Live Update; Press Conference; Report. movie_keywords (13)Behind the scenes; Blooper; Casting; Movie Clip; Playback; Preview; Review; Spoiler; Teaser; TV Series; Talk; Talks; TalkShow. entertainment_keywords (19)ASMR; Celebrity; Challenge; Concert; Gaming; Highlights; Music Video; Podcast; Prank; Reaction; Reacts; Travel; Unboxing; Variety Show; Vlog; HBO; SportsCenter; NFL; NBA. Table S1 Complete title-filtering keyword configuration. A Title-Based Keyword Filtering Title-based filtering is used to remove narration-dependent or primarily informational videos whose outcomes cannot be reliably verified from visual evidence alone. Such videos often require spoken explanations, contextual knowledge, or other nonvisual information to determine whether the depicted task has been completed successfully, making them unsuitable for visually grounded evaluation. To ensure that this filtering step is transparent and reproducible, the implementation loads 45 keyword strings from the filtering configuration. Although these keywords are organized into three groups for readability and maintenance, the groups are flattened into a single list before matching. Each candidate title is checked against every keyword using direct, case-sensitive substring matching, as implemented in the released code. A video is excluded whenever its title contains at least one listed string with matching capitalization; an exact match to the entire title is not required. This simple deterministic procedure ensures that the same filtering criterion is applied consistently across all candidate videos. The complete filtering configuration is reported in Table S1. B Domain and Category Taxonomy SemComp-Data organizes its instances into six broad domains and 21 fine-grained categories. The domains characterize general application scenarios, whereas the categories distinguish task-specific reference-to-outcome patterns. Table S2 lists the complete taxonomy. 13 DomainCategoryDefinition Food and Cooking Dish MakingA dish or food product is completed from ingredients. Food TransformationThe same food item undergoes a visible cooking-related state change. Food PlatingFood elements are arranged into a complete presentation. Beauty and Fashion StylingThe appearance of the same person is visibly restyled. Tool CleaningA beauty or fashion tool is cleaned to a reusable condition. Try-onAn external appearance reference is applied to a subject. Product RevealA previously concealed product becomes visible. Sports and Fitness Gear SetupSports gear or equipment is configured into a ready-to-use state. Body PreparationThe subject’s body enters a sports-related prepared state. Sports EditsSource event footage or identity cues are transformed into a completed sports-media edit. Crafts and DIY AssemblySeparate parts are combined into a complete structure or object. RestorationThe condition of the same object is restored or refurbished. RenovationA built space is reorganized or remodeled into a new spatial state. Blueprint Construction A physical structure is built according to an abstract design specification. Material ShapingA raw material is physically reshaped into a new form. Gardening and Pets Plant TreatmentA plant is improved through care-related operations. Floral DesignFloral or plant elements are trimmed and arranged into an aesthetic design. Pet GroomingA pet’s appearance is improved through grooming. Arts and Precision Artwork CreationAn artwork is completed from an unfinished visual basis. Digital CreationA digital creative product is completed from a visual or abstract reference. SculptingA sculptable material is turned into a finished three-dimensional artwork. Table S2 Domain–category taxonomy and category definitions used in SemComp-Data. 14 CategoryReference StateOutcome State Dish MakingIngredients or an incomplete food preparation are visible. A recognizable completed dish or food product is visible. Food Transformation A food item is visible before a cooking-related state change. The same food item exhibits the intended visible state change. Food PlatingFood components are present before final arrangement. The food components form a complete plated presentation. Styling A person is visible before the target styling change. The same person visibly exhibits the completed styling result. Tool CleaningA beauty or fashion tool is visibly soiled or not ready for reuse. The same tool is visibly clean and reusable. Try-onThe subject and an external appearance reference are identifiable. The subject visibly exhibits the referenced garment, accessory, or appearance. Product RevealThe product is concealed, covered, or not yet identifiable. The product is exposed and visually identifiable. Gear SetupSports gear is unconfigured, disassembled, or not ready for use. The gear is visibly configured in a ready-to-use state. Body PreparationA subject is visible before completing a sports-related preparation. The subject visibly reaches the intended prepared state. Sports EditsSource event footage or identity cues for a sports edit are visible. A completed sports-media edit is visible. Assembly Separate or partially combined components are visible. The components form a complete structure or object. RestorationA worn, damaged, or degraded object is visible. The same object is visibly restored or refurbished. RenovationA built space is visible before reorganization or remodeling. The space exhibits the completed renovated layout or appearance. Blueprint Construction An abstract design, plan, or blueprint is visible. A corresponding completed physical structure is visible. Material ShapingRaw or incompletely shaped material is visible. The material has the intended completed form. Plant TreatmentA plant is visible before care or treatment. The plant exhibits a visibly improved state after treatment. Floral DesignUnarranged floral or plant elements are visible. The elements form a completed aesthetic arrangement. Pet GroomingA pet is visible before grooming.The same pet exhibits the completed grooming result. Artwork CreationAn unfinished artwork or visual basis is visible. A completed artwork is visible. Digital CreationA visual reference or unfinished digital artifact is visible. A completed digital creative product is visible. Sculpting Raw or partially shaped sculptable material is visible. A completed three-dimensional sculpture is visible. Table S3 Category-specific reference and outcome state definitions used for State Mining. 15 C Reference and Outcome State Definitions The construction pipeline uses category-specific state definitions to localize representative reference and outcome frames. A reference state describes the initial visual evidence required to ground the task, while an outcome state describes the visually verifiable completed result. Table S3 summarizes the definitions used for State Mining. D Alignment and Attributes Instruction Structuring first assigns one of four alignment types and then selects instance-specific attributes to preserve from the reference image. The alignment type identifies the principal form of semantic grounding in the reference; it does not require all attributes associated with that type to remain unchanged. Table S4 gives the operational definitions. The complete preservation-attribute vocabulary contains 17 options spanning object semantics, visual appearance, human identity, and scene structure. Alignment TypeOperational DefinitionTypical Relevant Attributes Object ElementGrounds the outcome in task-relevant object categories, components, ingredients, materials, quantities, or colors appearing in the reference, without requiring the exact appearance of a specific object instance. Object category, composition, count, material, and color palette. IdentityRequires the depicted person to remain identifiable across the reference and generated outcome. Person identity, face appearance, body appearance, and pose. Object AppearanceGrounds the outcome in the instance-level visual appearance of a specific reference object. Object category, color palette, material, shape structure, surface appearance, and graphic details. SceneGrounds the outcome in the organization or context of the reference scene. Scene type, scene layout, spatial relation, background context, and viewpoint. Complete preservation-attribute vocabulary (17): object_category, object_composition, object_count, material, color_palette, shape_structure, surface_appearance, graphic_details, person_identity, face_appearance, body_appearance, pose, scene_type, scene_layout, spatial_relation, background_context, and viewpoint. Table S4 Operational definitions of the four alignment types and the complete preservation-attribute vocabulary. Attribute selection remains instance-specific. E SemComp-Bench Evaluation Prompts For each generated video, 27 frames are uniformly sampled and presented in temporal order. The evaluator answers each question withyesornoand provides visual evidence for the decision. The two positive OA questions pass onyes; the remaining OA questions and all GR questions are failure-oriented and pass on no. SemComp-Bench combines a common system prompt with criterion-specific task prompts. The system prompt in Fig. S1 defines the evaluator role, restricts each binary decision toyesorno, and requires all requested question identifiers to appear in the output. Outcome Achievement is evaluated using two prompt groups. The first prompt, shown in Fig. S2, evaluates Outcome Realization and Semantic Grounding throughQ 1 OA andQ 2 OA . A brief video-frame description generated by the VLM during State Mining is supplied as auxiliary context; the binary decisions remain grounded in the visible sampled frames. The second prompt, shown in Fig. S3, evaluates Grounded Entity Consistency and Global Visual Continuity through Q 3 OA and Q 4 OA . Grounded Entity Consistency checks for unexplained disappearance, replacement, or task-relevant semantic drift of key entities, whereas Global Visual Continuity checks for abrupt whole-frame changes. These questions are evaluated from temporally ordered 16 [System Prompt] You are an expert in video understanding and visual content analysis. Analyze the video based on frame-level visual content. Answer the questions according to the specific task requirements and the actual visible video content. Strictly follow the requirements: 1) Each question ID value must be either "yes" or "no". 2) If the task requirements explicitly request additional metadata fields, include those fields using the requested value types. 3) Always include keys for the question IDs in the task requirements, such as "Q1", "Q2", "Q3", etc. Figure S1 Common system prompt used for SemComp-Bench evaluation. [Outcome Achievement Prompt] – Q1 & Q2 [Inputs] -The generated video -Generated video frames sampled in chronological order. Each generated video frame has a 1-based frame index. Use the provided frame indices when reporting evidence. If no original frame indices are provided, use the 1-based order of the sampled frames. -The reference image -The brief Instruction [Task] -Evaluate whether the generated video achieves the outcome required by the instruction. [Important rules] -Focus only on outcome achievement. -Do not evaluate general visual quality unless visual defects prevent the required outcome from being verified. -Do not penalize minor artifacts, blur, or temporal instability unless they affect whether the required outcome is achieved. -Judge only based on visible evidence in the reference image and the sampled video frames. -Use the brief video-frame description as an additional observation from the same sampled frames, but answer Q1 and Q2 according to the actual visible frame evidence. [Questions] -Q1. The task requirement is:brief instruction. Does the video contain the required target outcome entity or outcome category in its essential task-defined state, regardless of detailed appearance, style, material, structure, or fine-grained attributes? For example, if the instruction is to remove a logo's white background, Q1 is "yes" only if a logo without the white background or with a transparent background is visible, not merely because a logo is visible. -Q2. The reference image is reference image. Does the visible target outcome match the specific requirements in both the text instruction and the reference image, including object type, appearance, style, material, structure, quantity, composition, required relationships, task-specific details, and the original object attributes from the reference image that should be preserved or changed according to the instruction? [Output format] -Q1: yes/no -Q1_reason: [reason, not exceeding 30 words] -Q2: yes/no -Q2_reason: [reason, not exceeding 30 words] Figure S2 Outcome Achievement prompt for Outcome Realization (Q 1 OA ) and Semantic Grounding (Q 2 OA ). sampled frames, with the reference image and instruction additionally provided when required by the criterion. Together, the four OA criteria separate task fulfillment from outcome-video validity. Outcome Realization first establishes whether the instructed completed state is visibly present at a coarse semantic level, while Semantic Grounding checks the task-relevant correspondence between that outcome and the reference–instruction pair. Grounded Entity Consistency does not require pixel-level appearance constancy; it only flags unexplained changes that prevent key entities from remaining semantically identifiable. Global Visual Continuity instead operates at the whole-frame level and detects abrupt switches in the scene, viewpoint, layout, background, or composition. Neither safeguard requires the video to reproduce the intermediate procedure. Generation Reliability uses a dedicated standalone prompt for Physical Plausibility and a joint prompt for the remaining diagnostic criteria. Figure S4 shows the standalone prompt forQ 1 GR , and Fig. S5 shows the joint prompt for Q 2 GR –Q 5 GR , covering Visual Clarity, Artifact-Free Rendering, Within-Scene Spatiotemporal Coherence, and Text and Interface Integrity. All five GR questions are failure-oriented, so anoresponse denotes a pass. GR is assessed independently of task completion and semantic grounding. Physical Plausibility examines visible motion, contact, interaction, and material behavior across the full temporally ordered sampled sequence. The remaining questions distinguish recognizability problems, scene-inconsistent rendering artifacts, local instability within otherwise continuous scenes, and malformed text or interface elements. Because these failure types are not mutually exclusive, the evaluator answers each question independently and may cite the same visual evidence for more than one criterion. Criterion-specific answers and concise evidence keep the reported pass rates interpretable and consistent with the main-paper score definitions. 17 [Outcome Achievement Prompt] – Q3 & Q4 [Inputs] -The generated video -Generated video frames sampled in chronological order. Each generated video frame has a 1-based frame index. Use the provided frame indices when reporting evidence. -If no original frame indices are provided, use the 1-based order of the sampled frames. -The reference image -The brief instruction [Task] -Evaluate whether the generated video contains clearly visible temporal-continuity or physical-continuity errors. [Important rules] -Focus only on temporal coherence and physical continuity within the generated video frames. -Do not evaluate general visual quality unless visual defects prevent temporal continuity or physical continuity from being verified. -Do not penalize minor artifacts, blur, or small temporal instability unless they create a clearly visible continuity error. -Judge only based on visible evidence in the sampled video frames. - Answer Q3 and Q4 independently. Do not mark Q3 as "yes" only because Q4 is "yes"; do not mark Q4 as "yes" only because Q3 is "yes". [Questions] -Q3. Does the generated video clip contain any clearly visible object-level or subject-level physical-continuity violation within the scene, such as an object or subject suddenly appearing or disappearing, being replaced by another object or subject, or changing its shape, color, material, texture, pattern, identity, or spatial structure without a visible continuous transition? Focus only on local changes to objects or subjects inside the scene, not on global whole- frame switches. -Q4. Does the generated video clip contain any abrupt whole-picture switch like a slideshow page turn, where consecutive frames or segments look like completely different pages? Focus on global switches of the entire frame, such as sudden changes to the overall scene, viewpoint, layout, background, and composition. Do not mark Q4 as "yes" for a local object-level or subject-level physical-continuity violation if the whole picture remains the same scene. [Output format] Q3: yes/no Q3_reason: [reason, or none] Q4: yes/no Q4_reason: [reason, or none] Figure S3 Outcome Achievement prompt for Grounded Entity Consistency (Q 3 OA ) and Global Visual Continuity (Q 4 OA ). [Physical Plausibility Evaluation] [Question] - Does the video contain any visible physically implausible content, motion, contact, interaction, or event at any time? [Important Rules] Answer "yes" when at least one observable violation occurs, even if it is brief, local, subtle, or appears only near the edge of the video. Examine the entire sequence and check all of the following: -Gravity and support: unsupported floating, hovering, or suspension; objects or body parts that should fall but do not;unstable balance or support relationships that cannot physically hold. -Motion and forces: acceleration, stopping, rotation, or direction changes without a plausible cause; inconsistent inertia, momentum, mass, weight, speed, or trajectory; visible sliding or drifting when stable contact should prevent it. -Contact, collision, and attachment: impossible collision responses or missing reactions to contact; solid objects penetrating, passing through, merging into, or occupying the same space; hands, feet, wheels, tools, or objects losing contact, attaching incorrectly, or failing to affect what they visibly touch; support, attachment, or contact relationships that change without a plausible event. -Object and material behavior: rigid objects bending, stretching, melting, splitting, merging, or changing volume without a plausible cause; implausible behavior of cloth, hair, liquids, smoke, fire, shadows, or deformable matter; spontaneous object deformation, creation, disappearance, or transformation when it represents a physical event rather than merely a shot or interface transition. -Cause-and-effect consistency: an action has no physically plausible visible cause or expected effect; interacting objects respond before contact, fail to respond after contact, or respond in an impossible direction or magnitude; physical properties or scene rules change inconsistently over time. Answer "no" only after checking the complete video and finding no observable violation in any category above. [Output Format] Question: yes/no Figure S4 Generation Reliability prompt for Physical Plausibility (Q 1 GR ). [Generation Reliability Q2-Q5 Evaluation Prompt] [Important Rules] - Evaluate whether the video contains visual quality defects. For each question, answer "yes" if the issue appears in any frame or in the transition between any adjacent frames. Otherwise, answer "no". - Evaluate each question independently. These questions are not mutually exclusive. Do not choose only the best-matching defect category. The same visual evidence may support "yes" for multiple questions. - Focus only on observable visual quality problems in the video. Do not evaluate whether the instruction is completed. [Questions] -Q2 Visual Clarity. Does the video have poor visual clarity, such as heavy blur, low resolution, overexposure, underexposure, or unclear details, making the main scene content, objects, or important details difficult to recognize? -Q3 Artifact-Free Rendering. Does the video contain visual artifacts inconsistent with the scene content, such as unrelated bright spots, noise, color blocks, color streaks, irrelevant thick lines, abnormal blank areas, gray patches, corrupted regions, or other inconsistent elements? Answer "yes" even if the artifact appears only in a local region or only in a few frames. -Q4 Within-Scene Spatiotemporal Coherence. Does the video contain obvious temporal discontinuity between adjacent frames, such as scene content, object structure, human structure, screen layout, interface regions, or important visual elements suddenly appearing, disappearing, changing shape, or being replaced without a continuous visual transition? Answer "yes" even if the discontinuity also appears as a visual artifact in a single frame. -Q5 Text and Interface Integrity. If text, numbers, icons, or user-interface elements appear in the video, are they blurry, distorted, malformed, unreadable, or not recognizable as normal written or interface content? Answer "yes" if the text, numbers, icons, or UI elements are abnormal, even if the overall scene is still recognizable. If no text, numbers, icons, or user-interface elements appear, answer "no". [Output Format] Q2: yes/no, [reason];Q3: yes/no, [reason]; Q4: yes/no, [reason]; Q5: yes/no, [reason] Figure S5 Generation Reliability prompt for Visual Clarity, Artifact-Free Rendering, Within-Scene Spatiotemporal Coherence, and Text and Interface Integrity (Q 2 GR –Q 5 GR ). 18 F Dataset Statistics and Instances The curation pipeline yields 1,273 SemComp-Data instances. Figure S6 presents 22 additional instances spanning diverse task categories. Each example shows the reference image, three temporally ordered frames from the paired outcome-centric clip, and the brief instruction. Dress small dog in new dress Make topped cookie dough Dress doll in July-themed outfit Transform dark hair to blonde ombre Print peacock on tote bag Cook bacon to crisp strips Wrap heart frame into yarn wreath Transform to shimmer gold makeup Fry raw eggs into fried eggs Give man dramatic full makeup Turn flat paper into decorated gift bag Cook into potato tortilla Make ingredients into mini ring cakes Assemble into layered pin Form yarn into magic circle Assemble block figures Assemble bands into bracelet Make into fried sticks & sauce Print multi-color 3D object Bake apple crisp dog treats Build robot circuit from schematic Cover graffiti wall with stucco Figure S6 Additional SemComp-Data instances. Each example shows a reference image, three temporally ordered frames sampled from the paired outcome-centric video clip, and the corresponding brief instruction. 19 The arrows denote the intended semantic transitions to the com- pleted outcomes. This notation does not imply that a generated video must reproduce the intermediate procedures. The split- ting module first partitions each full-context video into shots and merges temporally adjacent, visually consistent shots into events. The event containing the verified outcome timestamp is then selected, after which duration-constrained extraction pro- duces the outcome-centric clip. Events within the nominal target range of 3–4 s are retained in full; otherwise, an outcome-centered window is extracted from the full-context video with a target duration of 4 s for an overlong event or 3 s for an underlength event, with boundary-aware temporal shifting when necessary. This procedure results in a mean clip duration of approximately 4.03 s. StatisticValue Number of instances1,273 Minimum clip duration3.020 s Median clip duration4.067 s Mean clip duration4.033 s Maximum clip duration4.125 s SemComp-Core instances60 Sampled evaluation frames27 Table S5 Additional dataset and evaluation statistics. G Qualitative Results Figure S7 provides four additional SemComp-Core comparisons spanning Sculpting, Artwork Creation, Restoration, and Food Transformation. Within each case, all seven models receive the same reference condition and detailed instruction; the paired brief instruction is displayed in the figure only for compact presentation. Their outputs are shown as temporally ordered frame sequences. These matched comparisons expose complementary failure modes that are not fully captured by a terminal frame alone. Some sequences preserve the reference scene but make limited progress toward the specified outcome, whereas others depict a partial state change while introducing entity deformation, material drift, or abrupt visual transitions. The examples therefore illustrate why Outcome Achievement and Generation Reliability provide complementary views of semantic task completion. 20 Seedance 2.0- I2V Wan2.2-TI2V- 5B Wan2.2-I2V- A14B CogVideoX1.5- 5B-I2V SkyReels-V2- I2V-14B-720P Hunyuan1.5- 720P-I2V Phantom-1.3B [Brief Instruction] Cover the exposed dragon armature with clay to form a base 3D sculpture The Reference Conditions [Brief Instruction] Paint the blank canvas into a finished artwork Seedance 2.0- I2V Wan2.2-TI2V- 5B Wan2.2-I2V- A14B CogVideoX1.5- 5B-I2V SkyReels-V2- I2V-14B-720P Hunyuan1.5- 720P-I2V Phantom-1.3B [Brief Instruction] Remove price sticker residue from the diecast car surface The Reference Conditions [Brief Instruction] Cut the whole raw chicken into pieces Figure S7 Additional generated-video comparisons on four SemComp-Core instances. Outputs from all seven I2V models in each block were generated using the shared reference condition and detailed instruction; the paired brief instruction is displayed only for compact presentation. Temporally ordered output frames illustrate differences in both outcome achievement and generation reliability. 21 H Result Variability Each generated video is evaluated by three independent VLM calls made at different times. Tables S6–S8 report the sample standard deviation of the three corresponding run-level scores. Specifically, for run-level scorex r , we computeσ(x) = q P 3 r=1 (x r − ̄x) 2 3−1 . All entries are measured in percentage points (p). The OA Score standard deviations use the conjunctive definition in which a sample must pass all four OA criteria, whereas the GR Score standard deviations are computed from the run-level mean of the five GR criterion pass rates. Thus, the aggregate-score columns follow exactly the definitions used for the results reported in the main paper. Modelσ(A or )σ(A sg )σ(A gec )σ(A gvc ) σ(OA Score) Seedance 2.00.960.962.557.524.41 Wan2.2-TI2V-5B4.194.413.850.963.33 Wan2.2-I2V-A14B1.672.555.093.472.89 CogVideoX1.5-5B-I2V3.334.192.550.960.96 SkyReels-V2-I2V-14B-720P1.673.473.472.550.96 HY † -1.5-720P-I2V2.552.552.891.923.47 Phantom-1.3B0.961.921.920.960.96 Table S6 Sample standard deviations for SemComp-Core Outcome Achievement under detailed instructions (p; three runs). † HY denotes HunyuanVideo. Modelσ(G p ) σ(G vc ) σ(G afr ) σ(G wsc ) σ(G ti ) σ(GR Score) Seedance 2.03.330.960.961.920.961.58 Wan2.2-TI2V-5B0.962.556.7412.950.963.95 Wan2.2-I2V-A14B5.850.961.671.924.191.53 CogVideoX1.5-5B-I2V2.5513.476.318.828.557.12 SkyReels-V2-I2V-14B-720P0.963.476.7411.103.854.53 HY † -1.5-720P-I2V8.821.920.006.940.961.35 Phantom-1.3B5.090.968.2215.846.947.20 Table S7 Sample standard deviations across three evaluation runs for the SemComp-Core Generation Reliability results under detailed instructions. All values are in percentage points. † HY denotes HunyuanVideo. ModelModality Instruction σ(A or ) σ(A sg ) σ(A gec ) σ(A gvc ) σ(OA Score) Wan2.2-A14B I2VDetailed1.672.555.093.472.89 T2VDetailed2.554.191.674.412.55 T2VBrief0.960.962.557.260.96 CogVideoX1.5-5B I2VDetailed3.334.192.550.960.96 T2VDetailed3.330.962.550.961.67 T2VBrief2.891.670.963.470.96 HY † -1.5-720P I2VDetailed2.552.552.891.923.47 T2VDetailed0.002.556.745.001.92 T2VBrief1.673.334.195.360.00 Table S8 Sample standard deviations across three evaluation runs for the Outcome Achievement conditioning study. All values are in percentage points. † HY denotes HunyuanVideo. 22