Paper deep dive
VOR-Bench: A Human Perception-Driven Benchmark for Video Object Removal
Haonan Huang, Tianrui Qiu, Xianghao Zang, Yinan Du, Zhixiang He, Chi Zhang, Hao Sun, Zhongjiang He, Tianwei Cao, Xuchong Zhang, Hongbin Sun, Kongming Liang, Zhanyu Ma
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Despite its crucial role in video object removal (VOR), existing evaluation paradigms face two critical limitations: questionable references and a misalignment between tradi- tional metrics and human preference. To address these challenges, we introduce VOR- Bench, which advances VOR evaluation through three integrated components. First, we present the VOR Dataset (VORD), the first benchmark dataset providing both paired edited videos and graffiti masks. Its unique strength lies in a diverse data spectrum, which encompasses model-generated, tool-rendered, and camera-captured data, ensuring robust assessment across real-world scenarios. Second, we develop rMPAF, a realistic Motion- capable Paired-video Acquisition Framework. By combining the strengths of image- based object removal and fine-tuned video generation models, rMPAF automatically generates realistic, motion-coherent paired videos. Finally, we propose three evaluation dimensions and introduce VOR-MDSM, the first perception-driven VLM-based scoring model specifically designed for mask-guided VOR. It bridges the gap between arithmetic metrics and human perception by covering the essential visual attributes and matching nuanced human judgment. Extensive experiments demonstrate that VOR-Bench yields evaluation results that align closely with human perception, achieving a remarkable cor- relation (\r{ho} > 0.9) with subjective assessments. We will release VOR-Bench along with its documentation to ensure full reproducibility.
Tags
Links
- Source: https://arxiv.org/abs/2609.16878v1
- Canonical: https://arxiv.org/abs/2609.16878v1
Trouble viewing inline? Open PDF directly →
Full Text
62,350 characters extracted from source content.
Expand or collapse full text
arXiv:2609.16878v1 [cs.CV] 15 Sep 2026 HAONAN HUANG: VOR-BENCH: A HUMAN PERCEPTION-DRIVEN BENCHMARK FOR VOR 1 VOR-Bench: A Human Perception-Driven Benchmark for Video Object Removal Haonan Huang 1,* Tianrui Qiu 2,3,* Xianghao Zang 2 Yinan Du 1 Zhixiang He 2 Chi Zhang 2 Hao Sun 2 Zhongjiang He 2 Tianwei Cao 1 Xuchong Zhang 3 Hongbin Sun 3 Kongming Liang 1,† Zhanyu Ma 1 1 Beijing University of Posts and Telecommunications 2 China Telecom Artificial Intelligence Technology Co. Ltd 3 State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University Abstract Despite its crucial role in video object removal (VOR), existing evaluation paradigms face two critical limitations: questionable references and a misalignment between tradi- tional metrics and human preference. To address these challenges, we introduce VOR- Bench, which advances VOR evaluation through three integrated components. First, we present the VOR Dataset (VORD), the first benchmark dataset providing both paired edited videos and graffiti masks. Its unique strength lies in a diverse data spectrum, which encompasses model-generated, tool-rendered, and camera-captured data, ensuring robust assessment across real-world scenarios. Second, we develop rMPAF, a realistic Motion- capable Paired-video Acquisition Framework. By combining the strengths of image- based object removal and fine-tuned video generation models, rMPAF automatically generates realistic, motion-coherent paired videos. Finally, we propose three evaluation dimensions and introduce VOR-MDSM, the first perception-driven VLM-based scoring model specifically designed for mask-guided VOR. It bridges the gap between arithmetic metrics and human perception by covering the essential visual attributes and matching nuanced human judgment. Extensive experiments demonstrate that VOR-Bench yields evaluation results that align closely with human perception, achieving a remarkable cor- relation (ρ> 0.9) with subjective assessments. We will release VOR-Bench along with its documentation to ensure full reproducibility. © 2026. The copyright of this document resides with its authors. It may be distributed unchanged freely in print or electronic forms. * Equal contribution. † Corresponding author. 2HAONAN HUANG: VOR-BENCH: A HUMAN PERCEPTION-DRIVEN BENCHMARK FOR VOR Original Ground T Output ruth SSIM: 0.9661 SSIM: 0.9717 SSIM: 0.9311 Figure 1: The limitation of traditional metrics in estimating unexpected con- tent generation. The first two rows yield higher SSIM scores since global averag- ing dilutes localized defects. DatasetSize Paired-E GraffitiVideo Types VideosMasks M-G C-C T-R DAVIS [39]90✓ YouTube-VOS [54] 508✓ ROSE-Bench [36]60✓ Movies [31]5✓ HQVI [8]10✓ VOR-Eval [11]43✓ VOR-Wild [11]195✓ VORD150✓ VORD-L300✓ * Paired-E: paired edited;M-G: model-generated; C-C: camera-captured;T-R: tool-rendered. Table 1: A comparative analysis of existing datasets for video object removal and our pro- posed VORD and larger version VORD-L. 1 Introduction Recent years have witnessed rapid advancements in the field of video editing [4, 7, 21, 23, 35, 51, 52]. As a crucial subtask within video editing, video object removal (VOR) aims to eliminate objects from video sequences and generate visually plausible content that is consistent with the surrounding background context. Given its significant challenges, this task has garnered extensive attention, catalyzing a proliferation of innovative methodologies that have evolved from early 3D convolutional neural networks [6, 20, 47] to approaches leveraging optical flow and transformers [61, 63], and, most recently, to emerging diffusion models [11, 29, 33, 36, 37, 43, 53, 57, 65]. Nevertheless, the notable progress in video removal has posed significant challenges in its comprehensive and accurate evaluation. These challenges stem from two critical aspects: datasets and evaluation metrics. On the data front, existing benchmarks lack paired edited videos, precluding direct comparisons with model removal outputs, leading to questionable evaluation references. Furthermore, they employ precise masks generated by segmentation models, which stand in contrast to the masks provided by users in real-world applications. Simultaneously, existing datasets suffer from homogeneity, being dominated by either tool- rendered data or camera-captured data, with limited diversity in scenes and objects. In terms of metrics, existing evaluation paradigms typically rely on traditional image- level quality metrics such as Peak Signal-to-Noise Ratio (PSNR) [18], Structural Similarity (SSIM) [50], and Learned Perceptual Image Patch Similarity (LPIPS) [62]. These metrics require paired edited videos to perform calculations against a ground-truth reference. How- ever, these metrics measure pixel-level or structural fidelity rather than semantic meaning, resulting in significant divergence from human evaluation. Meanwhile, as exemplified in Fig. 1, the first two outputs contain visible artifacts yet yield higher SSIM scores. This is because its global averaging logic causes localized defects within the mask area to be diluted by the large, defect-free background. Consequently, the models can obtain high scores by optimizing background preservation while neglecting the actual quality of object removal. Even when restricted to non-masked areas for measuring background preservation, these metrics treat images as mere collections of independent pixels rather than coherent and structured wholes [5]. This flaw prevents it from accurately capturing changes in perception. Therefore, such limitations highlight the need for a more perceptually convincing evaluation benchmark for the VOR field. HAONAN HUANG: VOR-BENCH: A HUMAN PERCEPTION-DRIVEN BENCHMARK FOR VOR 3 To remedy the data limitations, we introduce VORD, the first VOR evaluation dataset that uniquely features both paired edited videos and graffiti masks. VORD comprises 150 paired video sequences, a scale determined through careful consideration to balance evaluation re- liability and computational cost. We further introduce a larger-scale version, VORD-L, to provide an option for broader evaluation and to serve as a supplementary benchmark for com- parison; detailed ablation experiments are presented in Section 4.3. As illustrated in Fig. 2, VORD covers three data types: model-generated, camera-captured, and tool-rendered. It offers a fine-grained categorization across 8 removal object types, 3 real-world scenarios, and 3 object association effects. This dataset also encompasses varying video lengths and frame rates, thereby enhancing its complexity and practicality. As shown in Table 1, VORD distinguishes itself from existing datasets by providing not only reliable paired edited videos but also user-realistic graffiti masks. The masks for the first frames are manually painted, while a segmentation model generates the remaining masks. This dataset not only provides diverse data categories but also allows for assessing models’ ability to precisely segment target objects. Half-body Full-bodyClose-up Dark Outdoor Rigid Motion Animals Cartoon ObscuredReflectiveShadow Casting Transparent Bright Outdoor Indoor Unobscured Cartoon Character Obscured Still Life Unobscured Still Life Reflective Animals Rigid Motion Close-up of Person Half-body of Person Shadow Casting Transparent Indoor Bright Outdoor Dark Outdoor Full-body of Person M-G C-C T-R Figure 2: Diverse categories of VORD, with repre- sentative visual examples. "M-G": model-generated data; "C-C": camera-captured data; "T-R": tool- rendered data. To address the critical lack of paired edited videos in existing eval- uation sets, we present a realistic Motion-capable Paired-video Ac- quisition Framework, rMPAF. Ac- quiring such pairs is highly chal- lenging, as real-world videos inher- ently lack removal videos, while static-camera or rendered videos suffer from limited scene diver- sity and restricted motion patterns. Building upon an image removal model and a trained video gener- ation model, our framework gen- erates high-quality paired edited videos that highly simulate real- world physical motion and main- tain alignment in non-edited re- gions. These videos form the foun- dational component of our dataset, enabling precise and comprehensive model evaluation. To supplement the current field of VOR evaluation, we innovatively propose three evaluation dimensions that comprehensively cover the core aspects of the task: object removal completeness, mask-background consistency, and log- ical plausibility. To quantitatively evaluate these dimensions with close alignment to hu- man perception, we introduce VOR-MDSM, a novel VLM-based multidimensional scoring model. Built upon the strong multimodal understanding of the VLM and supervised by hu- man ratings, VOR-MDSM formulates VOR evaluation as a question-answering task. By leveraging rich temporal-spatial video information, it outputs perceptually aligned scores across the three aforementioned dimensions, enabling a highly reliable assessment of edited 4HAONAN HUANG: VOR-BENCH: A HUMAN PERCEPTION-DRIVEN BENCHMARK FOR VOR videos. Extensive experiments demonstrate that VOR-MDSM significantly complements existing evaluation methodologies, exhibiting a stronger correlation with human perceptual judgments. Our contributions can be summarized as follows: • We propose VORD, the first VOR quality evaluation dataset featuring both paired edited videos and graffiti masks, characterized by diverse data types and varied video lengths. This dataset better aligns with practical application scenarios, providing com- prehensive data categories for the VOR evaluation. • We design a novel data framework rMPAF to address the lack of paired edited videos and the homogeneity of scenarios and motion patterns in existing datasets. This frame- work provides a viable solution for constructing high-quality video pairs. • We present VOR-MDSM, a human perception-driven video removal quality scoring model based on VLM. Through deliberate designs and high-quality data, it provides a comprehensive evaluation framework encompassing three dimensions. Extensive experiments validate its superior alignment with human preference. 2 Related Work 2.1 Video Object Removal Models The VOR task aims to seamlessly fill masked regions with content consistent with the back- ground context. Early approaches typically relied on mechanisms such as 3D convolutional neural networks [6, 20, 25, 47], optical flow [28, 55, 66], and transformer [3, 26, 30, 41, 59] to achieve object removal. ProPainter [63] combines optical flow with the transformer, al- lowing for the propagation of known pixels and the generation of unknown content. In recent years, with the rapid emergence and impressive generative capabilities of diffusion models [9, 16, 17, 32, 42, 44], a new paradigm for VOR has emerged [2, 22, 43, 53, 57]. MiniMax-Remover [65] focuses on developing a lightweight architecture based on the Diffu- sion Transformer (DiT) coupled with a two-stage training strategy, enabling efficient genera- tion. Beyond simple object removal, Casper [27], ROSE [36], and EffectErase [11] tackle the more complex problem of removing objects along with their associated effects by construct- ing specialized paired datasets. VOID [37] further highlights the importance of interaction- aware object removal by focusing on objects and their induced visual interactions in realistic video scenarios. Despite these advancements, the fundamental evaluation of VOR remains underexplored: whether the target object is truly removed, whether the background is plausibly restored, and whether the resulting video appears natural to human observers. Existing metrics are insuf- ficient to effectively compare diverse VOR models from a human perception perspective. To address this gap, VOR-Bench provides a diverse benchmark dataset and a human perception- driven multidimensional scoring model for evaluating the core quality of VOR. 2.2 Video Object Removal Evaluation Datasets The video object segmentation datasets such as DAVIS [39] and YouTube-VOS [54] are widely adopted for evaluating VOR models. However, these existing evaluation datasets lack the necessary paired edited videos for reasonable evaluation reference. This absence HAONAN HUANG: VOR-BENCH: A HUMAN PERCEPTION-DRIVEN BENCHMARK FOR VOR 5 inherently hinders the objective assessment of removal quality. Although datasets such as ROSE-Bench [36], Movies [31], HQVI [8] and VOR-Eval [11] include paired edited videos, they are restricted to a singular generation paradigm, such as pure rendering, simple mask pasting, or direct camera capture. This limitation naturally leads to homogeneous data types, which, when coupled with a limited data scale, fundamentally fail to satisfy diverse and ro- bust evaluation demands. To address these limitations, we construct VORD with diverse data sources and design rMPAF to generate realistic model-generated paired videos. VORD com- prises 150 video pairs sourced from model-generated, tool-rendered, and camera-captured data, featuring various removal objects, representative real-world scenes, and object associ- ation effects. 2.3 Video Object Removal Assessment Current VOR quality assessment typically relies on generic image similarity metrics. For instance, PSNR [18] calculates the mean squared error (MSE) between real video frames and generated video frames. SSIM [50] measures the similarity across luminance, contrast, and structure. LPIPS [62] employs deep neural networks to extract features from video frames and compute feature distances to determine similarity. However, these metrics are limited to measuring pixel-level or structural consistency and often diverge from human perceptual preferences. Moreover, their calculations strictly depend on authentic reference images. Due to the limitations of traditional metrics, researchers have explored evaluation meth- ods better aligned with human perception. Studies on forgery detection highlight that accu- rately identifying perceptual artifacts is crucial for judging visual authenticity [12, 13]. Re- cent works therefore leverage VLMs for perceptual quality assessment. Video-Bench [15] combines few-shot scoring with chain-of-query techniques for video generation assessment, while ImgEdit-Judge [58] and TDVE-Assessor [48] fine-tune Qwen2.5-VL-7B [1] for im- age editing and text-driven video editing evaluation, respectively. In VOR, VOID [37] prompts closed-source VLMs to assess interaction-aware removal quality. Nevertheless, such prompt-based evaluation depends on proprietary models and is not explicitly optimized for human perception-aligned VOR assessment. Therefore, our VOR-Bench fills this gap by introducing a locally deployable, cost-efficient multidimensional scoring model supervised by human annotations, offering a benchmark consistent with human perception for this task. 3 VOR-Bench 3.1 Dataset Construction To address the data homogeneity and questionable evaluation references of existing VOR benchmarks, VORD employs four key strategies: 1) Constructing precise paired edited videos, thereby establishing a unified and fair benchmark for VOR quality assessment. 2) Acquiring diverse data types, ensuring comprehensive coverage of model-generated data, tool-rendered data, and camera-captured data. 3) Incorporating graffiti masks, which simu- late the practical user interactions common in real video editing scenarios. 4) Determining an optimal dataset scale, which balances statistical significance and evaluation efficiency. VORD finally comprises 150 video pairs with 14 different visual types, and three data types in equal proportion. More visual samples have been shown in Fig. 2. Model-generated data construction. Obtaining pairs of original and removal videos in 6HAONAN HUANG: VOR-BENCH: A HUMAN PERCEPTION-DRIVEN BENCHMARK FOR VOR VOR tasks poses a challenge for existing evaluation systems. To address this challenge, we propose a novel data framework, rMPAF, that leverages the 2D object removal model Geo- Remover [64] and the video generation model Wan-2.1 [46]. Each generated video contains 41 frames at a frame rate of 15 fps. For specific details, please refer to Section 3.2. Camera-captured data acquisition. To acquire camera-captured data that captures authen- tic object environment interaction effects, we utilized fixed cameras to capture 50 pairs of video sequences. Each collected video pair contains objects entering and exiting the frame and each video contains 150 frames at 30 fps. This design facilitates the generation of pseudo-tuples comprising an object present video and its object removed counterpart. The camera-captured data cover diverse lighting variations, but they also have limitations in cap- turing motion due to fixed camera positions. Our model-generated data compensates for this shortcoming. Tool-rendered data collection. For our tool-rendered data, we considered the challenges of long duration and effects removal, and selected a total of 50 items from the open-source ROSE-Bench [36] and Movies [31]. The tool-rendered data encompasses three different ef- fects: transparent, shadow casting and reflective. Fig. 2 shows the specific samples. Mask generation strategy. To better accommodate irregular masks that users may provide, we manually painted a graffiti mask for the first frame of each video, while extracting seg- mentation masks for subsequent frames using Grounded SAM2 [40]. Dataset scale selection. The evaluation benchmark requires a careful trade-off among sce- nario diversity, statistical significance, and inference efficiency. Considering that some ad- vanced VOR models incur substantial computational costs during inference, we deliberately select 150 video pairs as the evaluation scale. While ensuring stable and discriminative eval- uation rankings, the current benchmark provides extensive coverage of practical scenarios without introducing unnecessary computational overhead. Correctness of data. Following CineTechBench [49], we used MonST3R to estimate cam- era trajectories and computed Rotation Error (RotErr), scene-scale-normalized Translation Error (TransErr), and Camera Motion Consistency (CamMC), which measure rotational, translational, and joint camera-motion discrepancies, respectively. TransErr is not mea- sured in metres; these metrics evaluate camera-motion consistency rather than pixel identity. ModelsRotError↓ TransError↓CamMC↓ Rel. * Abs. * Rel.Abs. Klingv1.621.6848.49196.1462.57207.65 Wan2.1 [46]27.8048.3199.6167.82115.76 Ours0.4023.4864.0523.6264.16 * Rel. represents the relative score of the metric. Abs. represents the absolute score of the metric. Table 2: Proof of model-generated data correctness. As depicted in Table 2, our model- generated data achieved the lowest values for the three metrics respec- tively. These values are significantly lower than the corresponding metrics reported within the CineTechBench dataset. This demonstrates the cor- rectness and reliability of our model- generated data, laying the foundation for subsequent research. For camera- captured data, due to the complex changes in real-world scenarios, we chose a fixed per- spective to ensure that qualified video pairs can be obtained in adjacent time periods, thus guaranteeing background consistency. The construction process of tool-rendered data is al- ready systematic, further details are omitted. HAONAN HUANG: VOR-BENCH: A HUMAN PERCEPTION-DRIVEN BENCHMARK FOR VOR 7 Selecter Source Dataset Selection 2D-Removal Model Prompt: Realistic scene style, do not add any additional objects, camera shifts guided by the first and last... VAE Encoder Image Encoder Text Encoder DiT Blocks VAE Decoder First-Frame Last-Frame First-Frame Last-Frame Trainable Frozen FLF2V-Model Removal Video Graffiti-Mask Original Video Standard (1)Good insertion quality (2)Consistency of movt. (3)Scene richness (4)Moderate mask size Model-generated Video-Pairs Aesthetic Predictor Score: 6.52Score: 1.43 Human Painting Figure 3: A realistic Motion-capable Paired-video Acquisition Framework, rMPAF. It com- prises five stages: (1) source data collection (2) aesthetic filtering (3) first & last frame object removal (4) frame interpolation by FLF2V-Model (5) final selection. Specifically, FLF2V- Model is used to generate intermediate frames by first frame and last frame. 3.2 Realistic Motion-capable Paired-video Acquisition Framework To overcome the critical scarcity of paired videos with diverse scenes and natural dynam- ics, we present the rMPAF. As depicted in Fig. 3, rMPAF is systematically divided into five stages: source data collection, aesthetic filtering, first & last frame object removal, video gen- eration via frame interpolation, and final selection. We utilize Vchitect-T2V-Dataverse [10] as our source data, given its diverse and high-quality video content. To filter out higher- quality data, we use the open-source Aesthetic Predictor to remove data with low scores for the first frame. Subsequently, we extract the first and last frames from these source videos. These frames are then processed by the 2D-removal model GeoRemover to generate object removed first and last frames. The two pairs of first and last frames are then fed into the fine-tuned Wan-2.1-FLF2V model for frame interpolation, yielding the target video pairs. Specifically, we fine-tune Wan-2.1-FLF2V using low-rank adaptation (LoRA) [19] with 100 videos mainly selected from DAVIS and HQVI. We choose them because they contain diverse real-world scenes, high-quality video content, and natural camera motion, which are important for learning realistic interpolation between the first and last frames. Such data provides a suitable basis for generating paired edited videos with realistic scene appearance and camera movement. This targeted fine-tuning aims to ensure that the interpolation qual- ity is optimized for the specific data requirements of the VOR task, especially maintaining consistency in the removed regions. To ensure the reliability of generated pairs, the original and removed videos are not independently generated. Instead, they share identical first and last frame anchors, and the same fine-tuned FLF2V model is applied under identical settings to interpolate intermediate frames. We further manually filter out samples with temporal inconsistency, removal failures, visual artifacts, or invalid masks. Furthermore, compared with the existing tool-rendered data with fixed objects, the data constructed by this framework not only has a unified style but also fits real scenarios better, conforms to real motion laws without being limited to rigid objects. It also yields more diverse shot-trajectory data with lower cost than camera-captured data. 8HAONAN HUANG: VOR-BENCH: A HUMAN PERCEPTION-DRIVEN BENCHMARK FOR VOR 3.3 Evaluation Metrics Aiming to establish an evaluation framework that captures the entire visual spectrum of VOR tasks while aligning with human perception, we introduce three new evaluation dimensions: object removal completeness, mask-background consistency and logical plausibility. All three dimensions are scored from 1 to 5 using our multidimensional scoring model. Object removal completeness (ORC). As the core objective of VOR task, object removal completeness primarily evaluates whether the masked object has been completely elimi- nated. The score ranges from 1 to 5, reflecting the removal proportion of the masked object, with higher scores indicating more complete elimination. Mask-background consistency (MBC). We focus on whether the content generated within the masked region blends naturally with the surrounding background. Mask-background consistency aims to evaluate the consistency in details such as color, texture, material and patterns. It is an important measure of whether a video removal model could achieve seam- less inpainting. Logical plausibility (LP). It assesses whether the content generated by the model in the masked region is semantically related to the background and adheres to physical laws and common sense. As shown in (a) of Fig. 4, the original structure of the sofa in the removal video has been almost preserved with flaws only visible when zoomed in; (b) shows the re- dundant object in the masked area and the extremely poor removal effect. Temporal consistency. Temporal consistency is considered within the video-level MBC/LP criteria rather than treated as an independent dimension. MBC penalizes temporally unstable blending in the removed region, while LP penalizes cross-frame violations of scene logic and physical plausibility. In the dimension of LP, we have treated unreasonable motion of a still object as a low-score case, directly covering a temporal-consistency failure. Thus, temporal consistency is not ignored, but incorporated into MBC/LP. We did not add a separate TC dimension because temporal failures in VOR are highly coupled with MBC/LP, which could cause duplicated penalties and annotation ambiguity. The scoring criteria for the above dimensions will be explained in the supplementary materials (Section 6). 3.4 VOR-MDSM Training dataset construction. To construct our training dataset, source videos were col- lected using the same data categories and construction methods as described for the test dataset. To ensure a diverse and extensive training corpus, we employed five different open- OriginalLogical Plausibility RemovalMask Score: 1 (b) (a) Score: 4 Figure 4: Visual examples of the logical plausibility score. HAONAN HUANG: VOR-BENCH: A HUMAN PERCEPTION-DRIVEN BENCHMARK FOR VOR 9 (a) Data Collection Qwen3-VL Object Removal Completeness: 5 Mask-Background Consistency: 4 Logical Plausibility: 4 445 Score table Tool- rendered Data Model- generated Data Camera- captured Data 5 Video Object Removal Models 3k+ Training Samples Removal Comp. M-Bg Consist. Logical Plaus. 1 1 1 2 3 45 2 2 3 3 4 4 5 5 Evaluate the video removal quality on a 1-5 scale, focusing on three dimensions: (1) Object Removal Completeness (2) Mask-Background Consistency (3) Logical Plausibility (b) Human Annotation Figure 5: Overview of VOR-MDSM. (a) We utilize rMPAF (Section 3.2) to gather three categories of source data and employ various VOR models to yield the final training dataset; (b) Vision experts annotate this training data across three distinct dimensions. Finally, the VLM is subsequently fine-tuned using the annotated training data. source VOR models to process these source videos. This process ultimately yielded 3,299 edited video pairs. Subsequently, we recruited 15 vision experts, organized into three groups, to conduct human annotation on these video pairs. For each video pair, annotators were tasked with evaluating the quality of object removal in the generated video across three di- mensions. Prior to the formal annotation process, volunteers received comprehensive train- ing on the scoring criteria for each of the three dimensions. They were also provided with a detailed annotation guideline for reference. Each video pair received ratings from five independent annotators. The final annotation score for each pair was then determined by av- eraging these five ratings. These human-annotated scores are used as supervisory signals to train the VOR-MDSM, enabling it to learn human-aligned quality assessment. The detailed information of the experts will be explained in the supplementary materials (Section 6). Multidimensional scoring model. Considering the scoring requirements for associating vi- sual information and text semantics in multi-modal input, we have selected a VLM as the evaluator, termed VOR-MDSM. To address the current void in open-source VLM evaluators specifically designed for VOR tasks, we employ LoRA for parameter-efficient fine-tuning of the VLM using the previously constructed training corpus. Fig. 5 depicts the overall ar- chitecture of the VOR-MDSM. It provides a three-dimensional rating directly from a pair of original and edited videos. This evaluation paradigm assesses editing results without requir- ing removal ground truth. Trained under the supervision of human-annotated data, VOR- MDSM requires only video pairs to generate scores aligned with human perception. Unlike conventional image quality assessment (IQA) metrics that rely on low-level visual similarity, VOR-MDSM performs task-specific comparison between original and edited videos, focus- ing on object removal completeness, mask-background consistency, and logical plausibility. 10HAONAN HUANG: VOR-BENCH: A HUMAN PERCEPTION-DRIVEN BENCHMARK FOR VOR Based on the comparative experiments described in Section 4.3, we select the opti- mal training data input format. Specifically, each training sample is formulated as x i = (v orig i ,v edit i ,q i ), where q i is the scoring instruction. The target is a structured textual answer containing three scores s i = (s orc i ,s mbc i ,s lp i )∈1,..., 5 3 , e.g., “ORC: 3, MBC: 3, LP: 4”. VOR-MDSM is trained by supervised fine-tuning Qwen3-VL with LoRA to generate this scoring text, rather than by three-dimensional score regression. The objective is the standard token-level causal language modeling loss: L(θ) =− 1 N ∑ i ∑ t∈A i log p θ (y i,t | x i ,y i,<t ),(1) whereθ denotes the trainable model parameters, N is the number of training samples, A i denotes the set of assistant-answer token positions for the i-th sample, y i,t is the target token at position t, and y i,<t represents the preceding target tokens. Only assistant-answer tokens are supervised, while prompt and video tokens are ignored in the loss computation. Additionally, the perception alignment evaluation and cross-dataset ablation in Section 4.3 further validate the accuracy and robustness of VOR-MDSM. Implementation details. Using Qwen3-VL-8B [56] as the backbone, we performed training for 10K steps on 4 NVIDIA A100 GPUs with LoRA rank of 64. The input resolution for the training data was 992×560, with a batch size of 2 and a learning rate of 1e-4. 4 Experiments 4.1 Experimental Setup For comparative analysis, we choose ten open-source representative VOR models: Fuse- Former [34], FGT [61], ProPainter [63], FloED [14], Casper [27], MiniMax-Remover [65], DiffuEraser [29], ROSE [36], EffectErase [11], VOID [37]. This selection intentionally spans a wide range of methodologies, encompassing approaches from foundational works to recent state-of-the-art advancements in the field. We run VORD on these models, with the same input and output resolution. Then we score them by three distinct dimensions: ob- ject removal completeness, mask-background consistency, and logical plausibility, using our scoring model. 4.2 Evaluation Results Quantitative evaluation. Table 3 compares ten representative VOR methods. ROSE achieves the best overall performance across the three VOR-MDSM dimensions, followed by Ef- fectErase and VOID. The relatively lower scores of other methods demonstrate the capabil- ity of VOR-MDSM in distinguishing different removal qualities. Notably, mask-background consistency remains the weakest dimension across most methods, indicating that background restoration is still a major challenge for current VOR models and motivating future efforts toward improving this aspect. Furthermore, we evaluate traditional metrics on non-masked regions to examine whether they can better reflect background preservation. However, the resulting rankings differ from both VOR-MDSM and qualitative comparisons, suggesting limited alignment with human perception. Specifically, ProPainter achieves higher traditional metric scores than ROSE, whereas ROSE obtains substantially better VOR-MDSM scores and qualitative results. This HAONAN HUANG: VOR-BENCH: A HUMAN PERCEPTION-DRIVEN BENCHMARK FOR VOR 11 ModelsYear Background PreservationVOR-MDSM Score PSNR↑SSIM↑LPIPS↓ORCMBCLPAvg FuseFormer [34]202127.9410.8900.0703.253.223.713.39 FGT [61]202228.6300.8820.0833.573.394.053.67 ProPainter [63]202329.6440.9140.053 3.313.143.693.38 FloED [14]202427.8320.8910.0693.203.173.613.33 Casper [27]202529.4490.8930.0574.034.034.384.15 DiffuEraser [29]202529.3940.9100.0563.773.694.203.89 MiniMax-Remover [65]202528.8800.8980.0603.993.914.374.09 ROSE [36]202530.4120.9000.0534.274.114.614.33 EffectErase [11]202628.7230.8890.0644.194.024.594.27 VOID [37]202630.9880.9120.0454.044.044.574.22 Table 3: Overall quantitative comparisons. It comprises two parts: background preserva- tion and three VLM-evaluated dimensions. Bold indicates the best; underlined indicates the second-best. ROSEDiffuEraser ProPainter Ground Truth MiniMax-RemoverCasperFGT FuseFormer FloED Original Video EffectErase VOID Figure 6: The visual results of different models. The video frames on the left depict the original video and ground-truth video. indicates that pixel-level similarity alone cannot fully capture VOR-specific quality factors, as better background preservation may not necessarily correspond to better removal quality. Therefore, we report model rankings based on VOR-MDSM rather than traditional metrics. Qualitative evaluation. Figure 6 supports the quantitative ranking: ROSE produces the cleanest result, whereas other methods exhibit more visible artifacts or removal boundaries. Logical errors are relatively rare, consistent with the generally high LP scores in Table 3. Additional visual results are provided in the supplementary material (Section 8). 4.3 Ablation Study Perception alignment ablation. To verify the alignment of our scoring model with human judgment, we assessed 1500 generated videos from ten different object removal methods on our VORD benchmark. These videos were divided into three parts and rated by 15 human annotators, with five people in each group responsible for one part. Statistical analysis of human annotations (Fig. 7) confirms their reliability for supervi- sion. (1) Minor Divergence. The median standard deviations (Med. Std) for ORC, MBC, and LP are 0.5477, 0.7071, and 0.8367 (<1.0). The rating disagreements are minor and largely confined to adjacent scale levels. (2) High Consistency. The Intraclass Correlation Coefficient ICC(2,5) reaches 0.8931, 0.8913, and 0.8576. These high correlations (>0.85) 12HAONAN HUANG: VOR-BENCH: A HUMAN PERCEPTION-DRIVEN BENCHMARK FOR VOR Dim. Med. Std↓ ICC(2,5)↑ ORC0.54770.8931 MBC0.70710.8913 LP0.83670.8576 (a) 29.3 28.2 42.4 ORC 51.3 23.1 25.6 MBC 44.9 27.1 27.9 LP (b) Low [1,3)Medium [3,4)High [4,5] Figure 7: Annotation reliability and score distribution. (a) Annotation consistency across three dimensions. (b) Score distributions for ORC, MBC, and LP. 0.4 0.6 0.8 1.0 Video LLaMA3 Qwen3- VL-8B Gemini- 3.1-pro Ours rρτ 0.4 0.6 0.8 1.0 Video LLaMA3 Qwen3- VL-8B Gemini- 3.1-pro Ours rρτ Mask-Background Consistency Object Removal Completeness 0.4 0.6 0.8 1.0 Video LLaMA3 Qwen3- VL-8B Gemini- 3.1-pro Ours rρτ Logical Plausibility 0.91 0.91 0.94 Figure 8: Comparative preference consistency of four models across three dimensions. r, ρ , andτ denote Pearson, Spearman, and Kendall correlation coefficients, respectively. demonstrate a strong consensus among the experts, confirming the averaged labels are highly reliable for supervision. (3) Diverse coverage. The score distribution spans low, medium, and high quality ranges instead of concentrating on easy high-score cases. MBC has the highest low-score proportion, 51.3%, matching our observation that it is the most challeng- ing dimension. Leveraging reliable human ratings, we choose VideoLLaMA3 [60], Gemini-3.1-pro and original Qwen3-VL-8B, which support multi-video inputs, to calculate the scores of these videos together with our VOR-MDSM, while ensuring strict input-output consistency. The average scores across the three defined evaluation dimensions were then calculated for all videos. Subsequently, to quantify the correlation, we follow TDVE-Assessor’s work [48] and calculate the Pearson r [38], Spearmanρ [45], and Kendallτ [24] correlation coeffi- cients between the human scores and the scores from each of the four models on the three dimensions. The results in Fig. 8 show that VOR-MDSM achieves r,ρ,τ> 0.9 on all three dimensions and outperforms VideoLLaMA3, Qwen3-VL-8B, and Gemini-3.1-pro, demon- strating strong alignment with human ratings. At the method level, the Spearman corre- lations between human rankings and rankings induced by aggregated traditional metrics, ReMOVE [5], and VOR-MDSM are 0.382, 0.697, and 0.976, respectively. Detailed model rankings are provided in the supplementary material (Section 7). These findings conclusively demonstrate that VOR-MDSM effectively achieves a human perception-driven evaluation of the removal quality of video objects. VOR-MDSM training ablation. Currently, Qwen3-VL supports a multi-video parallel training approach. Meanwhile, inspired by TDVE’s method of temporally concatenating videos, we divide the training input modes into four types: (1) Temporal concatenation: HAONAN HUANG: VOR-BENCH: A HUMAN PERCEPTION-DRIVEN BENCHMARK FOR VOR 13 ORCMBCLP ModelInput Format rρτrρτrρτ Alignment Scores Temp. concate. 0.1612 0.0958 0.0364 0.4340 0.1905 0.1429 0.4438 0.0952 0.07140.1857 Spat. concate.0.9691 0.9758 0.9436 0.8683 0.8796 0.7412 0.8680 0.9157 0.81540.8863 Triple input0.7338 0.6627 0.5185 0.4891 0.5270 0.3273 0.6306 0.6190 0.42860.5485 Ori Double input0.7668 0.5522 0.4619 0.7376 0.8503 0.6910 0.8354 0.8503 0.69100.7152 Temp. concate. 0.9789 0.9940 0.9820 0.9244 0.9286 0.7857 0.9146 0.9524 0.85710.9242 Spat. concate.0.9885 0.9940 0.9820 0.9134 0.9048 0.7857 0.9033 0.9048 0.78570.9069 Triple input0.9903 0.9940 0.9820 0.9652 0.9701 0.9092 0.8938 0.9581 0.90920.9524 Trained Double input0.9919 0.9701 0.9092 0.9399 0.9701 0.9092 0.9429 0.9940 0.98200.9566 Table 4: Ablation studies on the input format. r: Pearson r;ρ : Spearmanρ ;τ : Kendallτ . Figure 9: Ablation studies on graffiti masks across three dimensions. Original video and removal video concatenated along the time dimension to form a single video; (2) Spatial concatenation: Original video, mask video, and removal video are concate- nated into a single video from left to right in the spatial dimension; (3) Triple input: Original video, mask video and removal video; (4) Double input: Original video and removal video. This ablation experiment is conducted with strictly consistent parameters across all four types. As shown in Table 4, the performance is characterized by calculating the correlation with human ratings. The Alignment Score is the overall mean correlation across all dimen- sions and metrics, computed as 1 3 ∑ d∈ORC,MBC,LP r d +ρ d +τ d 3 . Ultimately, the double input that exhibits the best performance is established as the training input for the scoring model. Dataset scale ablation. Some existing video object removal models require high inference costs. For instance, Casper takes more than 30 minutes to process a 150-frame video on a single GPU. Therefore, a practical benchmark must strike a careful balance between statis- tical significance and evaluation efficiency. To rigorously determine the dataset scale and verify if 150 pairs are sufficient to draw stable evaluation conclusions, we conduct an ab- lation study on the dataset scale. We construct an extended set containing 300 video pairs 14HAONAN HUANG: VOR-BENCH: A HUMAN PERCEPTION-DRIVEN BENCHMARK FOR VOR and intentionally increase the proportion of camera-captured data to 50% in order to intro- duce more extreme physical-world diversity. We then re-evaluate all baseline models on this extended set to observe if the capability rankings fluctuate with increased data volume. The evaluation shows an unchanged tier structure from VORD (150 pairs) to VORD-L (300 pairs): a) Tier-1 (score≥ 4.2): ROSE, EffectErase, VOID; b) Tier-2 (3.9 – 4.2): Casper, MiniMax-Remover; c) Tier-3 (3.6 – 3.9): DiffuEraser, FGT; d) Tier-4 (≤ 3.6): Fuse- Former, FloED, ProPainter. The model scoring tier is strictly consistent with the Table 3, confirming that 150 pairs already suffice to reliably separate model performance. VORD provides a highly discriminative and unbiased testing platform, while also saving significant inference time and computational resources for the research community. Graffiti masks ablation. To assess whether existing removal models can segment target ob- jects, we replace the first-frame graffiti masks, which better reflect practical user input, with finely segmented masks and compare the scores of the two resulting output sets (Fig. 9). VOID is excluded because its mandatory four-value mask prevents a fair comparison. With finely segmented masks, the models achieve higher mask-background consistency scores, and all except FGT also improve in logical plausibility. In contrast, object removal com- pleteness remains largely unchanged for most models, consistent with the nature of this dimension. Casper changes more than the other models, possibly because it was originally designed for layer separation. These results indicate that target segmentation in current re- moval models can still be improved. DatasetPearson rSpearmanρKendallτ DAVIS0.800.910.82 ROSE-Bench0.930.970.92 VORD (Ours)0.960.980.93 Table 5: Cross-dataset correlation with human preference. Cross-dataset ablation. To evaluate the generalization of VOR-MDSM, we conduct cross-dataset validation on the widely used DAVIS and ROSE-Bench datasets under identical experimental settings. We collect human ratings and calculate their correla- tions with VOR-MDSM scores. As shown in Table 5, the high correlations on both datasets confirm the cross-dataset robustness of VOR-MDSM. In contrast, traditional met- rics produce substantially different model rankings across benchmarks, demonstrating their sensitivity to evaluation references. Detailed traditional-metric results and the cross-axis statistics of VOR-MDSM are provided in the supplementary material (Section 7). Inference time. Under identical full-video inputs, VOR-MDSM averages 0.99s per video, compared with 1.22s for VideoLLaMA3 and 21.50s for the three traditional metrics, demon- strating practical benchmark-scale efficiency. 5 Conclusion This work introduces VOR-Bench to evaluate video removal models. VOR-Bench over- comes the limitations of existing VOR evaluation sets and metrics, providing more credible evaluation results. The proposed VOR dataset, VORD, with paired edited videos, diverse data types and graffiti masks, better aligns with practical application scenarios. Our novel data framework, rMPAF, provides a viable solution for generating paired edited videos in various scenarios. In addition, we present a specialized human perception-driven multi- modal evaluator for VOR, VOR-MDSM. This evaluator assesses three key dimensions: ob- ject removal completeness, mask-background consistency and logical plausibility. Overall, VOR-Bench provides references for the performance improvement of video removal models. HAONAN HUANG: VOR-BENCH: A HUMAN PERCEPTION-DRIVEN BENCHMARK FOR VOR 15 Acknowledgements This work was supported by the National Nature Science Foundation of China (Grant 62476029, 62225601, 62506043), and sponsored by Beijing Nova Program and the Beijing Key Labo- ratory of Multimodal Data Intelligent Perception and Governance. References [1] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. [2] Yuxuan Bian, Zhaoyang Zhang, Xuan Ju, Mingdeng Cao, Liangbin Xie, Ying Shan, and Qiang Xu. Videopainter: Any-length video inpainting and editing with plug-and- play context control. In Proc. SIGGRAPH, pages 1–12, 2025. [3] Jiayin Cai, Changlin Li, Xin Tao, Chun Yuan, and Yu-Wing Tai. Devit: Deformed vision transformers in video inpainting. In Proc. ACM M, pages 779–789, 2022. [4] Duygu Ceylan, Chun-Hao P Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. In Proc. ICCV, pages 23206–23217, 2023. [5] Aditya Chandrasekar, Goirik Chakrabarty, Jai Bardhan, Ramya Hebbalaguppe, and Prathosh AP. Remove: A reference-free metric for object erasure. In Proc. CVPR, pages 7901–7910, 2024. [6] Ya-Liang Chang, Zhe Yu Liu, Kuan-Ying Lee, and Winston Hsu. Free-form video inpainting with 3d gated convolution and temporal patchgan. In Proc. ICCV, pages 9066–9075, 2019. [7] Yuqing Chen, Junjie Wang, Lin Liu, Ruihang Chu, Xiaopeng Zhang, Qi Tian, and Yujiu Yang. O-disco-edit: Object distortion control for unified realistic video editing. arXiv preprint arXiv:2509.01596, 2025. [8] Suhwan Cho, Seoung Wug Oh, Sangyoun Lee, and Joon-Young Lee. Elevating flow- guided video inpainting with reference generation. In Proc. AAAI, volume 39, pages 2527–2535, 2025. [9] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthe- sis. NeurIPS, 34:8780–8794, 2021. [10] Weichen Fan, Chenyang Si, Junhao Song, Zhenyu Yang, Yinan He, Long Zhuo, Ziqi Huang, Ziyue Dong, Jingwen He, Dongwei Pan, et al. Vchitect-2.0: Parallel trans- former for scaling up video diffusion models. arXiv preprint arXiv:2501.08453, 2025. [11] Yang Fu, Yike Zheng, Ziyun Dai, and Henghui Ding. Effecterase: Joint video object removal and insertion for high-quality effect erasing. arXiv preprint arXiv:2603.19224, 2026. [12] Yueying Gao, Weiguo Lin, Junfeng Xu, Wanshan Xu, and Peibin Chen. Self-supervised adversarial training for robust face forgery detection. In Proc. BMVC, page 718, 2023. 16HAONAN HUANG: VOR-BENCH: A HUMAN PERCEPTION-DRIVEN BENCHMARK FOR VOR [13] Yueying Gao, Dongliang Chang, Bingyao Yu, Haotian Qin, Muxi Diao, Lei Chen, Kongming Liang, and Zhanyu Ma. Toward generalizable forgery detection and reason- ing. IEEE TIP, 35:3395–3410, 2026. [14] Bohai Gu, Hao Luo, Song Guo, Peiran Dong, and Qihua Zhou. Coherent video inpaint- ing using optical flow-guided efficient diffusion. arXiv preprint arXiv:2412.00857, 2024. [15] Hui Han, Siyuan Li, Jiaqi Chen, Yiwen Yuan, Yuling Wu, Yufan Deng, Chak Tou Leong, Hanwen Du, Junchen Fu, Youhua Li, et al. Video-bench: Human-aligned video generation benchmark. In Proc. CVPR, pages 18858–18868, 2025. [16] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 33:6840–6851, 2020. [17] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. NeurIPS, 35:8633–8646, 2022. [18] Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In Proc. ICPR, pages 2366–2369. IEEE, 2010. [19] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022. [20] Yuan-Ting Hu, Heng Wang, Nicolas Ballas, Kristen Grauman, and Alexander G Schwing. Proposal-based video completion. In Proc. ECCV, pages 38–54. Springer, 2020. [21] Zeyinzi Jiang, Zhen Han, Chaojie Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu. Vace: All-in-one video creation and editing. In Proc. ICCV, pages 17191–17202, October 2025. [22] Xuan Ju, Xian Liu, Xintao Wang, Yuxuan Bian, Ying Shan, and Qiang Xu. Brushnet: A plug-and-play image inpainting model with decomposed dual-branch diffusion. In Proc. ECCV, pages 150–168. Springer, 2024. [23] Xuan Ju, Tianyu Wang, Yuqian Zhou, He Zhang, Qing Liu, Nanxuan Zhao, Zhifei Zhang, Yijun Li, Yuanhao Cai, Shaoteng Liu, et al.Editverse: Unifying im- age and video editing and generation with in-context learning. arXiv preprint arXiv:2509.20360, 2025. [24] Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1-2):81–93, 1938. [25] Dahun Kim, Sanghyun Woo, Joon-Young Lee, and In So Kweon. Deep video inpaint- ing. In Proc. CVPR, pages 5792–5801, 2019. [26] Sungho Lee, Seoung Wug Oh, DaeYeun Won, and Seon Joo Kim. Copy-and-paste networks for deep video inpainting. In Proc. ICCV, pages 4413–4421, 2019. HAONAN HUANG: VOR-BENCH: A HUMAN PERCEPTION-DRIVEN BENCHMARK FOR VOR 17 [27] Yao-Chih Lee, Erika Lu, Sarah Rumbley, Michal Geyer, Jia-Bin Huang, Tali Dekel, and Forrester Cole. Generative omnimatte: Learning to decompose video into layers. In Proc. CVPR, pages 12522–12532, 2025. [28] Ang Li, Shanshan Zhao, Xingjun Ma, Mingming Gong, Jianzhong Qi, Rui Zhang, Dacheng Tao, and Ramamohanarao Kotagiri. Short-term and long-term context aggre- gation network for video inpainting. In Proc. ECCV, pages 728–743. Springer, 2020. [29] Xiaowen Li, Haolan Xue, Peiran Ren, and Liefeng Bo. Diffueraser: A diffusion model for video inpainting. arXiv preprint arXiv:2501.10018, 2025. [30] Zhen Li, Cheng-Ze Lu, Jianhua Qin, Chun-Le Guo, and Ming-Ming Cheng. Towards an end-to-end framework for flow-guided video inpainting. In Proc. CVPR, pages 17562–17571, 2022. [31] Geng Lin, Chen Gao, Jia-Bin Huang, Changil Kim, Yipeng Wang, Matthias Zwicker, and Ayush Saraf. Omnimatterf: Robust omnimatte with 3d background modeling. In Proc. ICCV, pages 23471–23480, 2023. [32] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022. [33] Jie Liu and Zheng Hui. Eraserdit: Fast video inpainting with diffusion transformer model. arXiv preprint arXiv:2506.12853, 2025. [34] Rui Liu, Hanming Deng, Yangyi Huang, Xiaoyu Shi, Lewei Lu, Wenxiu Sun, Xiaogang Wang, Jifeng Dai, and Hongsheng Li. Fuseformer: Fusing fine-grained information in transformers for video inpainting. In Proc. ICCV, pages 14040–14049, 2021. [35] Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. Video-p2p: Video editing with cross-attention control. In Proc. CVPR, pages 8599–8608, 2024. [36] Chenxuan Miao, Yutong Feng, Jianshu Zeng, Zixiang Gao, Hantang Liu, Yunfeng Yan, Donglian Qi, Xi Chen, Bin Wang, and Hengshuang Zhao. Rose: Remove objects with side effects in videos. NeurIPS, 38:149140–149162, 2026. [37] Saman Motamed, William Harvey, Benjamin Klein, Luc Van Gool, Zhuoning Yuan, and Ta-Ying Cheng. Void: Video object and interaction deletion. arXiv preprint arXiv:2604.02296, 2026. [38] Karl Pearson. Vii. note on regression and inheritance in the case of two parents. pro- ceedings of the royal society of London, 58(347-352):240–242, 1895. [39] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine- Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675, 2017. [40] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 18HAONAN HUANG: VOR-BENCH: A HUMAN PERCEPTION-DRIVEN BENCHMARK FOR VOR [41] Jingjing Ren, Qingqing Zheng, Yuanyuan Zhao, Xuemiao Xu, and Chen Li. Dlformer: Discrete latent transformer for video inpainting. In Proc. CVPR, pages 3511–3520, 2022. [42] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Om- mer. High-resolution image synthesis with latent diffusion models. In Proc. CVPR, pages 10684–10695, 2022. [43] Dvir Samuel, Matan Levy, Nir Darshan, Gal Chechik, and Rami Ben-Ari. Omnimat- tezero: Fast training-free omnimatte with pre-trained video diffusion models. In Proc. SIGGRAPH Asia, pages 1–11, 2025. [44] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit mod- els. arXiv preprint arXiv:2010.02502, 2020. [45] Charles Spearman. The proof and measurement of association between two things. The American journal of psychology, 100(3/4):441–471, 1987. [46] Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025. [47] Chuan Wang, Haibin Huang, Xiaoguang Han, and Jue Wang. Video inpainting by jointly learning temporal structure and spatial details. In Proc. AAAI, volume 33, pages 5232–5239, 2019. [48] Juntong Wang, Jiarui Wang, Huiyu Duan, Guangtao Zhai, and Xiongkuo Min. Tdve- assessor: Benchmarking and evaluating the quality of text-driven video editing with lmms. arXiv preprint arXiv:2505.19535, 2025. [49] Xinran Wang, Songyu Xu, Xiangxuan Shan, Yuxuan Zhang, Muxi Diao, Xueyan Duan, Yanhua Huang, Kongming Liang, and Zhanyu Ma. Cinetechbench: A bench- mark for cinematographic technique understanding and generation. arXiv preprint arXiv:2505.15145, 2025. [50] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 13(4):600–612, 2004. [51] Shuchen Weng, Haojie Zheng, Peixuan Zhang, Yuchen Hong, Han Jiang, Si Li, and Boxin Shi. Vires: Video instance repainting via sketch and text guided generation. In Proc. CVPR, pages 28416–28425, 2025. [52] Jianzong Wu, Xiangtai Li, Chenyang Si, Shangchen Zhou, Jingkang Yang, Jiangning Zhang, Yining Li, Kai Chen, Yunhai Tong, Ziwei Liu, et al. Towards language-driven video inpainting via multimodal large language models. In Proc. CVPR, pages 12501– 12511, 2024. [53] Xian Wu and Chang Liu. Ditpainter: Efficient video inpainting with diffusion trans- formers. arXiv preprint arXiv:2504.15661, 2025. HAONAN HUANG: VOR-BENCH: A HUMAN PERCEPTION-DRIVEN BENCHMARK FOR VOR 19 [54] Ning Xu, Linjie Yang, Yuchen Fan, Jianchao Yang, Dingcheng Yue, Yuchen Liang, Brian Price, Scott Cohen, and Thomas Huang. Youtube-vos: Sequence-to-sequence video object segmentation. In Proc. ECCV, pages 585–601, 2018. [55] Rui Xu, Xiaoxiao Li, Bolei Zhou, and Chen Change Loy. Deep flow-guided video inpainting. In Proc. CVPR, pages 3723–3732, 2019. [56] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [57] Shiyuan Yang, Zheng Gu, Liang Hou, Xin Tao, Pengfei Wan, Xiaodong Chen, and Jing Liao. Mtv-inpaint: Multi-task long video inpainting. arXiv preprint arXiv:2503.11412, 2025. [58] Yang Ye, Xianyi He, Zongjian Li, Bin Lin, Shenghai Yuan, Zhiyuan Yan, Bohan Hou, and Li Yuan. Imgedit: A unified image editing dataset and benchmark. arXiv preprint arXiv:2505.20275, 2025. [59] Yanhong Zeng, Jianlong Fu, and Hongyang Chao. Learning joint spatial-temporal transformations for video inpainting. In Proc. ECCV, pages 528–543. Springer, 2020. [60] Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Fron- tier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106, 2025. [61] Kaidong Zhang, Jingjing Fu, and Dong Liu. Flow-guided transformer for video in- painting. In Proc. ECCV, pages 74–90. Springer, 2022. [62] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proc. CVPR, pages 586–595, 2018. [63] Shangchen Zhou, Chongyi Li, Kelvin CK Chan, and Chen Change Loy. Propainter: Improving propagation and transformer for video inpainting. In Proc. ICCV, pages 10477–10486, 2023. [64] Zixin Zhu, Haoxiang Li, Xuelu Feng, He Wu, Chunming Qiao, and Junsong Yuan. Georemover: Removing objects and their causal visual artifacts. arXiv preprint arXiv:2509.18538, 2025. [65] Bojia Zi, Weixuan Peng, Xianbiao Qi, Jianan Wang, Shihao Zhao, Rong Xiao, and Kam-Fai Wong. Minimax-remover: Taming bad noise helps video object removal. arXiv preprint arXiv:2505.24873, 2025. [66] Xueyan Zou, Linjie Yang, Ding Liu, and Yong Jae Lee. Progressive temporal feature alignment network for video inpainting. In Proc. CVPR, pages 16448–16457, 2021. HAONAN HUANG: VOR-BENCH: A HUMAN PERCEPTION-DRIVEN BENCHMARK FOR VOR 1 Supplementary Material 6 More Details of Scoring Information 6.1 Detailed Information of Vision Experts There are 15 visual experts, each of whom has practical experience in video editing and possesses a professional background in the field. They have participated in subjective eval- uation experiments for text-to-image and text-to-video models before, gaining a thorough understanding of aspects such as image quality, aesthetics, distortion in the spatial domain, and issues like multi-frame artifacts and frame-to-frame deformation consistency in the tem- poral domain. For this work, they all underwent thorough training prior to the evaluation. 6.2 Detailed Information of Scoring Criteria Object removal completeness. The scoring rules are as follows: Score 1: object removal completion degree 0%∼50%; Score 2: object removal completion degree 50%∼70%; Score 3: object removal completion degree 70%∼90%; Score 4: object removal completion degree 90%∼100%; Original Object Removal Comp. RemovalMask (a) Score: 5 (b) Score: 4 (c) Score: 3 (d) Score: 2 (e) Score: 1 Figure 10: Visual examples of the object removal completeness. Zoom in to observe a more pronounced effect. 2HAONAN HUANG: VOR-BENCH: A HUMAN PERCEPTION-DRIVEN BENCHMARK FOR VOR Original Mask-Background Consist. RemovalMask (a) Score: 5 (b) Score: 4 (c) Score: 3 (d) Score: 2 (e) Score: 1 Figure 11: Visual examples of the mask-background consistency. Zoom in to observe a more pronounced effect. Score 5: object removal completion degree 100%. Specific examples are shown in Fig. 10. Mask-background consistency. The scoring rules are as follows: Score 1: the removed area does not blend with the background, and there are large areas of holes, artifacts, and color differences in the removed area, which affect the visual quality; Score 2: the transition between the removed area and the background is poor, with obvi- ous boundaries and a sense of disharmony; Score 3: moderate fusion, with occasional visible boundaries or seams, and inconsistent textures or styles; Score 4: the removed area blends well with the background, with slight differences in boundaries or textures; Score 5: the removed area blends perfectly with the background. Specific examples are shown in Fig. 11. Logical plausibility. The scoring rules are as follows: Score 1: the generation of hallucinated objects; Score 2: obvious logical deviations, such as deformed shapes and moving still objects; Score 3: related generation to the background, but there are slight logical issues; Score 4: more subtle logical issues can only be seen by zooming in on the video; Score 5: No issues with any physical laws or scene logic. Specific examples are shown in Fig. 12. In addition, the sample (d) in Fig. 12 shows the unreasonable movement of a static pillow, and please refer to the specific video for more intuitive display. HAONAN HUANG: VOR-BENCH: A HUMAN PERCEPTION-DRIVEN BENCHMARK FOR VOR 3 Original Logical Plaus. RemovalMask (a) Score: 5 (b) Score: 4 (c) Score: 3 (d) Score: 2 (e) Score: 1 Figure 12:Visual examples of the logical plausibility. Zoom in to observe a more pro- nounced effect. All the examples above are accompanied by corresponding videos. 7 Additional Quantitative Results 7.1 Metric-Induced Model Rankings Since traditional metrics, ReMOVE, and VOR-MDSM operate on different score scales, we compare their induced model rankings with the human preference ranking. As shown in Table 6, VOR-MDSM achieves a substantially higher Spearman correlation with the human ranking than traditional metrics and ReMOVE, further demonstrating its alignment with hu- man perception. 7.2 Traditional Metrics Across Benchmarks We further evaluate traditional metrics on DAVIS and ROSE-Bench. As shown in Table 7, the resulting model rankings vary substantially across benchmarks, indicating their sensitiv- ity to evaluation references. The VORD column additionally reports the mean and standard deviation across the three VOR-MDSM dimensions. The relatively small standard deviations indicate balanced cross-axis evaluation. 4HAONAN HUANG: VOR-BENCH: A HUMAN PERCEPTION-DRIVEN BENCHMARK FOR VOR FuseF. FGT ProP. FloED Casp. DiffE. MiniM. ROSE EffE. VOID ReMOVE↑0.884 0.9010.886 0.841 0.887 0.8920.8970.902 0.898 0.895 ReMOVE rank92810764135 Trad. rank † 91028546371 VOR-MDSM rank87910465123 Human rank78910564123 Spearmanρ w.r.t. human ranking: Trad. 0.382; ReMOVE 0.697; VOR-MDSM 0.976 † The traditional ranking aggregates PSNR, SSIM, and LPIPS. FuseF.: FuseFormer; ProP.: ProPainter; Casp.: Casper; DiffE.: DiffuEraser; MiniM.: MiniMax-Remover; EffE.: EffectErase. Table 6: Comparison of metric-induced model rankings on VORD. DAVISROSE-BenchVORD ModelPSNR↑ SSIM↑ LPIPS↓PSNR↑ SSIM↑ LPIPS↓Avg±Std FuseFormer33.9370.9600.03328.6730.9210.0653.393± 0.222 FGT32.4050.9430.04228.8100.9200.0673.669± 0.278 ProPainter36.6740.9800.01529.4280.9300.0583.380± 0.228 FloED21.7420.6560.11027.7280.9100.0703.327± 0.199 Casper27.8030.8620.08229.7250.9230.0634.149± 0.164 DiffuEraser33.3070.9580.03228.9440.9250.0623.887± 0.224 MiniMax-Remover 29.7980.8880.05928.5650.9230.0654.091± 0.198 ROSE27.4010.8660.07132.7990.9390.0524.327± 0.208 EffectErase22.9120.7730.09530.6070.9370.0564.269± 0.238 VOID28.0290.8800.06331.3930.9330.0384.218± 0.251 Table 7: Traditional metrics on DAVIS and ROSE-Bench, together with the cross-axis statis- tics of VOR-MDSM on VORD. Bold and underlined values denote the best and second-best results, respectively. 8 More Visualization Samples More visualization samples are shown in Fig. 13. We have also provided the corresponding video files, which can be found in the attachments as tool-rendered-visual-samples, model-generated-visual-samples and camera-captured-visual-samples. HAONAN HUANG: VOR-BENCH: A HUMAN PERCEPTION-DRIVEN BENCHMARK FOR VOR 5 Original Video Original Video MiniMax-Remover Model-generated Camera-captur ed T ool-r ender ed DiffuEraserROSEProPainter Ground TruthMiniMax-RemoverCasper FGT FuseFormer FloED ROSEDiffuEraserProPainter Ground Truth Casper FGT FuseFormer FloED Original Video Ground Truth MiniMax-Remover DiffuEraser ProPainter CasperFGT FuseFormer FloED ROSE VOID EffectErase VOID EffectErase VOID EffectErase Figure 13: The visual results of different models across three data types: model-generated data, camera-captured data and tool-rendered data.