Paper deep dive

RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning

Seungku Kim, Suhyeok Jang, Byungjun Yoon, Dongyoung Kim, John Won, Jinwoo Shin

Year: 2026Venue: arXiv preprintArea: cs.ROType: PreprintEmbeddings: 91

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 7/20/2026, 9:27:04 PM

Summary

The paper introduces RoboCurate, a framework for generating high-quality synthetic robot data using video generative models. It addresses the issue of inconsistent action quality in synthetic videos by replaying predicted actions in a simulator and filtering samples based on motion consistency between the generated video and the simulator rollout. The framework also employs image-to-image and video-to-video editing to enhance visual diversity. Experiments show significant improvements in success rates for robot learning tasks compared to using real data alone or previous synthetic methods.

Entities (10)

RoboCurate · framework · 98%ALLEX · robot · 95%DexMimicGen · benchmark · 95%GR-1 Tabletop · benchmark · 95%Image-to-Image Editing · technique · 90%Inverse Dynamics Models · model · 90%Simulator Replay · process · 90%Video-to-Video Transfer · technique · 90%Vision-Language Models · model · 90%Attentive Probe · model-component · 85%

Relation Signals (8)

RoboCurate → improves → GR-1 Tabletop

confidence 95% · achieving +70.1% on GR-1 Tabletop (300 demos)

RoboCurate → improves → ALLEX

confidence 95% · +179.9% in the challenging real-world ALLEX humanoid dexterous manipulation setting

RoboCurate → improves → DexMimicGen

confidence 95% · +16.1% on DexMimicGen in the pre-training setup

RoboCurate → uses → Simulator Replay

confidence 95% · RoboCurate replays the predicted actions in a simulator and assesses action quality by measuring the consistency of motion

RoboCurate → uses → Video-to-Video Transfer

confidence 90% · apply action-preserving video-to-video transfer to further augment appearance

RoboCurate → uses → Image-to-Image Editing

confidence 90% · unlock observation diversity beyond the available dataset via image-to-image editing

RoboCurate → employs → Attentive Probe

confidence 85% · train a lightweight attentive probe on top of a pre-trained video encoder to classify whether a video pair exhibits consistent motion

Vision-Language Models → limitations → RoboCurate

confidence 80% · VLMs have limitations in distinguishing physically accurate videos and cannot directly evaluate the generated actions themselves. To tackle this issue, we introduce RoboCurate

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Synthetic data generated by video generative models has shown promise for robot learning as a scalable pipeline, but it often suffers from inconsistent action quality due to imperfectly generated videos. Recently, vision-language models (VLMs) have been leveraged to validate video quality, but they have limitations in distinguishing physically accurate videos and, even then, cannot directly evaluate the generated actions themselves. To tackle this issue, we introduce RoboCurate, a novel synthetic robot data generation framework that evaluates and filters the quality of annotated actions by comparing them with simulation replay. Specifically, RoboCurate replays the predicted actions in a simulator and assesses action quality by measuring the consistency of motion between the simulator rollout and the generated video. In addition, we unlock observation diversity beyond the available dataset via image-to-image editing and apply action-preserving video-to-video transfer to further augment appearance. We observe RoboCurate's generated data yield substantial relative improvements in success rates compared to using real data only, achieving +70.1% on GR-1 Tabletop (300 demos), +16.1% on DexMimicGen in the pre-training setup, and +179.9% in the challenging real-world ALLEX humanoid dexterous manipulation setting.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

89,656 characters extracted from source content.

Expand or collapse full text

RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning Seungku Kim * 1 Suhyeok Jang * 1 Byungjun Yoon 1 Dongyoung Kim 1 2 John Won 1 Jinwoo Shin 1 2 Abstract Synthetic data generated by video generative mod- els has shown promise for robot learning as a scalable pipeline, but it often suffers from incon- sistent action quality due to imperfectly gener- ated videos. Recently, vision-language models (VLMs) have been leveraged to validate video quality, but they have limitations in distinguishing physically accurate videos and, even then, cannot directly evaluate the generated actions themselves. To tackle this issue, we introduce RoboCurate, a novel synthetic robot data generation framework that evaluates and filters the quality of annotated actions by comparing them with simulation replay. Specifically, RoboCurate replays the predicted ac- tions in a simulator and assesses action quality by measuring the consistency of motion between the simulator rollout and the generated video. In addition, we unlock observation diversity beyond the available dataset via image-to-image editing and apply action-preserving video-to-video trans- fer to further augment appearance. We observe RoboCurate’s generated data yield substantial rel- ative improvements in success rates compared to using real data only, achieving +70.1% on GR-1 Tabletop (300 demos), +16.1% on DexMimicGen in the pre-training setup, and +179.9% in the chal- lenging real-world ALLEX humanoid dexterous manipulation setting. 1. Introduction Robot foundation models (RFMs; (Kim et al., 2024; Zitkovich et al., 2023; Bjorck et al., 2025)) have achieved strong performance across diverse robotics domains, e.g., manipulation, locomotion, and navigation (Hirose et al., 2025; Zhang et al., 2025). The key component of this suc- cess is large-scale robotics datasets (O’Neill et al., 2024; Bu et al., 2025) spanning diverse tasks and embodiments, * Equal contribution 1 KAIST 2 RLWRLD. Correspondence to: Jinwoo Shin <jinwoos@kaist.ac.kr>. Preprint. February 24, 2026. often leveraging foundation models like vision-language models (VLMs) and video generative models (Cheang et al., 2024; Shen et al., 2025) for action generation by bridging world knowledge and commonsense with robotic actions. This serves as a form of pre-training for action data, en- abling RFMs to adapt to downstream tasks with little or no task-specific fine-tuning. Despite its success, robotics data remains far more lim- ited than vision and language data due to costly and labor- intensive collection processes (Khazatsky et al., 2024; Walke et al., 2023), which fundamentally constrain large- scale pre-training. To address this problem, prior research has focused on simulation-based synthetic data as a substi- tute for real data, but simulation suffers from visual discrep- ancies with real-world environments, limited knowledge transfer to real-world settings (i.e., sim-to-real gap), and requires substantial engineering effort to generate diverse environments and action data. Meanwhile, neural trajectory, i.e., synthetic data generated by video generative models, have emerged as a promising alternative, as they are visu- ally similar to real-world data than simulation. Moreover, those models can generate diverse task videos conditioned on text prompts (Zhou et al., 2024; Fu et al., 2025). This approach annotates episodes by mapping videos to actions through inverse dynamics models (IDMs) trained on real data. Several studies have experimentally shown that neural trajectory can improve generalization and enable learning novel actions beyond the training dataset (Jang et al., 2025; Team et al., 2025). However, neural trajectory pipelines often face quality is- sues. Unlike in simulation, the video generation stage can fail to follow input instructions or produce physically im- plausible videos (e.g., objects overlapping or deforming unnaturally), which leads to incorrect action annotations. Moreover, even when videos are accurate, relying on learned models such as IDMs instead of ground-truth annotators can produce low-quality action labels, degrading overall data quality and ultimately leading to suboptimal downstream policy performance. To address this problem, recent work has employed VLMs to judge instruction following or video plausibility (Motamed et al., 2025; Bansal et al., 2024). However, such judgments are typically too coarse to capture 1 arXiv:2602.18742v1 [cs.RO] 21 Feb 2026 RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning Stage 2: Filtering Inverse Dynamic Model Video Encoder Video Encoder Simulator Cross Attention Classificaion Head Synthetic Video Real Robot Demo Synthetic Video I2I + I2V Pipeline I2V + V2V Pipeline Initial Frame I2I I2V I2V V2V Stage 1: Generation Simulator Rollout 푎 ["..."$%] Curated Data DiscardedData VLA Training Motion Consistent Motion Inconsistent Figure 1. Overview of RoboCurate. (1) We generate diverse neural trajectory by applying image-to-image (I2I) model for scene diversity and video-to-video (V2V) model for appearance diversity, respectively. (2) We then filter neural trajectory using simulator-replay consistency, retaining only those for which a classifier predicts the motion in the generated video matches the simulator rollout. task-critical motion for policy learning (e.g., whether the arm moves far enough to reach the object) and often stop at superficial compliance with basic physics without directly evaluating the actions themselves. This highlights the ne- cessity for a new method that can verify actions with visual correspondence and physical grounding. Contributions. To overcome this limitation, we propose RoboCurate, a unified framework that (1) generates syn- thetic robot data and (2) filters neural trajectory by quality via simulator-replay consistency, while supporting diverse scene and task generation. The core idea is to leverage a simulator to replay the generated action sequences, produc- ing proxy robot videos with guaranteed correspondence to the generated actions. By measuring visual motion con- sistency between these simulator-replayed videos and the synthesized videos, we can evaluate both the visual quality of the generated data and the correctness of action labels. Concretely, we replay IDM-predicted actions in a simula- tor to render a rollout video, and reformulate alignment as motion matching between the generated video and the sim- ulator replay. To assess this match, we train a lightweight attentive probe on top of a pre-trained video encoder to classify whether a video pair exhibits consistent motion pat- terns and robot geometry. In addition, we introduce a con- trollable visual diversification pipeline that expands initial scenes and augments generated videos using recent diffu- sion models. Specifically, we employ (1) image-to-image (I2I) editing to substantially increase scene diversity, and (2) action-preserving video-to-video (V2V) transfer to augment appearance while preserving motion dynamics. Together, such pipeline generates high-quality data that reflects a wide range of realistic environmental variations. We verify the effectiveness of RoboCurate across diverse scenarios. We pre-train vision-language-action models (VLAs) on Fourier ActionNet (Fourier ActionNet Team, 2025) and evaluate on GR-1 Tabletop (NVIDIA et al., 2025) and DexMimicGen (Jiang et al., 2025) benchmarks. We fur- ther perform co-finetuning with real-robot evaluation on the ALLEX humanoid. In the pre-training setup, RoboCurate yields remarkable relative gains compared to real-data-only baseline, achieving a +70.1% on GR-1 Tabletop (300 de- mos) and a +16.1% on DexMimicGen. In contrast, the base synthetic data generation framework (Jang et al., 2025) yields only marginal gains over the same real-data-only baseline of +26.6% and +4.0% on the same benchmarks, re- spectively. We observe that this trend generalizes to the real ALLEX humanoid in the co-finetuning setup: RoboCurate demonstrates a +179.9% relative improvement in success rate, whereas Jang et al. (2025) achieves a +100.0% gain. Notably, RoboCurate further demonstrates strong out-of- distribution generalization on the challenging real-world ALLEX humanoid dexterous manipulation environment, 2 RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning achieving a +162.3% relative improvement on novel ob- ject pick-and-place tasks and enabling emergent success on novel action tasks (from 0.0% to 25.0%). 2. Preliminaries Video diffusion models.Recent video generative models (Blattmann et al., 2023; Kong et al., 2024; Yang et al., 2025; Wan et al., 2025) typically operate in a compressed latent space to reduce computational complexity. Given a videow in pixel space, an encoderEmaps it to a latent representation x =E(w). To learn the video distribution within this space, we employ the Flow Matching (FM) framework (Lipman et al., 2022), which defines a probability pathx t linearly interpolating between the dataxand Gaussian noiseε, with timestep t∈ [0, 1]: x t = (1− t)x + tε. The diffusion model performs as a velocity predictorv θ , trained to regress the target velocity fieldu t =ε− x. This is achieved by minimizing the mean squared error (MSE) objective: L(θ) =E t,x,ε,c ∥v θ (x t ,t,c)− (ε− x)∥ 2 2 , where c denotes conditioning information. Inverse dynamics models. Inverse dynamics models (IDMs) (Baker et al., 2022) predict actions between a cur- rent observationx t and a future observationx t+H , whereH denotes the action horizon. Formally, IDMs estimate the in- termediate action sequence asa t:t+H−1 = IDM(x t ,x t+H ). We use Diffusion Transformer (DiT) (Peebles & Xie, 2023) as our inverse dynamics model, trained with a flow-matching objective. At inference time, the model denoises the entire action sequence a t:t+H−1 conditioned on the input frames. As a result, IDMs serve as a pseudo-action labeler that con- verts action-free video into action-labeled trajectory. Con- cretely, using a pre-trained image-to-video (I2V) diffusion model (Wan et al., 2025; Ali et al., 2025), we generate a synthetic task-execution video from an initial frame and a task instruction. We then apply IDMs to current and future frame pairs along the video to infer actions. This yields neural trajectory, consisting of paired synthetic videos and pseudo-labeled action sequences, enabling policy training on action-free data. Imitation learning for policy training. Our goal is to learn a policyπ θ for robot manipulation from demonstration data. We follow an imitation learning (IL) framework, where the policy predicts actions conditioned on observations and proprioceptive states. We assume access to a limited real- world datasetD real and neural trajectoryD syn , and train the policy on the combined datasetD =D real ∪D syn . Specifically, given observationo t , task instructionI, and proprioceptive stateq t , the policy outputs an action chunk A t = a t:t+H−1 . In our experiments, we use VLA with diffusion-based action head as the base policy, and optimize it with the following flow-matching objective: L IL (θ;D) =E t,A t ,ε,τ h v θ (A τ t ,τ | o t ,q t ,I)−(ε−A t ) 2 2 i , whereτ ∼U(0, 1),ε∼N(0,I), andA τ t = τA t +(1−τ)ε. We note thatD syn has zero-padded proprioceptive states, since IDMs do not predict state information. 3. Method We present RoboCurate, a novel neural trajectory generation framework that increases diversity via controllable video generation and filters low-quality samples by evaluating motion similarity between generated video and simulator replay. In Section 3.1, we introduce our video-level neural trajectory generation stage and the approaches to promote diversity. In Section 3.2, we propose a filtering strategy that verifies IDM-predicted actions by checking their con- sistency with the generated video through simulator rollout. In Section 3.3, we show that our filtering strategy can also be applied directly during generation stage via Best-of-N sampling. We provide overview of RoboCurate in Figure 1. 3.1. Generating Plausible Manipulation Scenarios To generate diverse robot synthetic videos using a video generative model, we control two factors: scene visuals and task instructions. For scene visuals, we inject visual variance with two components: (1) image-to-image (I2I) editing on the initial frame for scene-level variation, and (2) video-to-video (V2V) transfer for appearance diversity while preserving initial motion. For task instructions, we use VLM to identify plausible novel instructions for a given initial scene, and condition video generation on each gener- ated task instruction. We detail each diversification factor below. Expanding visual diversity. We perform instruction- guided image editing to generate diverse observations while preserving the underlying manipulation setup. Since the edited image should remain a valid starting state for video generative model (e.g., target object and placement should remain physically plausible), we additionally employ a Canny edge map as a condition to preserve original scene structure. We then carefully design systematic prompts to in- duce controlled visual variations. Specifically, we first query VLM to produce a detailed description of the initial image, and combine it with the original task instruction to identify the target object and relevant manipulation context. Using these as context, we generate multiple edited variants along four axes: (1) table appearance, (2) target object identity 3 RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning Figure 2. Examples of neural trajectory. (Top): original videos, (Bottom): visually augmented neural trajectory. The two bottom-left frames indicate a video whose initial frame is edited by I2I model, while the two bottom-right frames indicate a video processed with V2V transfer. Positive (+) Pair: •Episode aligned ✔ •Time aligned ✔ Negative (-) Pair: •Episode aligned ✔ •Time aligned ✘ Negative (-) Pair: •Episode aligned ✘ •Time aligned ✔ Negative (-) Pair: •Episode aligned ✘ •Time aligned ✘ Figure 3. Examples of negative pairs for attentive probe train- ing. We construct negative pairs from real-world dataset by induc- ing temporal shifts or sampling video from different episodes. and appearance, (3) lighting, and (4) background. Finally, we feed each edited initial frame into an image-to-video diffusion model with plausible task scenarios. We provide additional prompt details for image editing in Appendix C.3. While I2I editing can substantially expand visual diversity at the image level, we further augment diversity at the video level. Specifically, we apply video-to-video (V2V) transfer to successful synthetic videos to diversify their appearance while preserving motion dynamics. Since the transferred video typically retains the robot motion, we reuse the action annotations labeled by IDMs. Concretely, we condition V2V transfer on Canny edge videos to preserve the original video structure, and use system prompts analogous to our image editing pipeline along the same four axes. To ensure action reuse remains valid, we keep object identity and shape unchanged and only modify texture and color. We additionally prepend an instruction to maintain the robot’s color, as the embodiment should remain unchanged. See Appendix C.4 for V2V transfer prompt details. Expanding task instructions. We generate synthetic robot videos conditioned on an initial frame and a language instruction. For effective policy learning, it is crucial that the videos contain meaningful robot-object interactions. To achieve this, we use a proprietary VLM to generate plausi- ble task instructions conditioned on the initial frame. Since naive VLM querying may produce wrong instruction tem- plates or physically infeasible robot actions, we employ few-shot prompting with examples from the original dataset to ensure consistency. Overall, we design novel task instruc- tions along four axes: (1) behavior, (2) target object, (3) placement, and (4) robot hand type. We explicitly specify the active hand to maintain a 1:1 ratio, since we consider bimanual manipulation in downstream tasks. We provide additional prompt details for task instruction generation in Appendix C.1. 3.2. Action-level Filtering of Neural Trajectory While we can generate diverse neural trajectory by expand- ing scene visuals and task instructions, they can still contain noisy action labels: physically implausible video motion or IDM prediction errors can make the predicted action inconsistent with the video. To ensure the quality of the training data, it is necessary to identify and filter out such invalid samples. Specifically, we represent each neural- trajectory sample as a generated video paired with an IDM- predicted action,(w gen ,a IDM ). To validatea IDM , RoboCu- rate replaysa IDM in simulator and renders the corresponding rollout videow sim (a IDM ), whose robot motion is consistent witha IDM . This converts the action verification problem into a motion-consistency comparison between two videos, (w gen ,w sim (a IDM )). To solve this motion-consistency com- parison problem, we propose an an attentive probe on top of a frozen video encoder. First, we carefully construct positive and negative pairs (see Figure 3) from real-world demonstrationsT = (w real ,a real )to train the probe to detect subtle motion mismatches. Since neural trajectory can contain noisy ac- tion labels and may mislead supervision, we do not use them to train our attentive probe. Concretely, for each real action 4 RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning a real , we render a simulator rollout videow sim (a real )and build positive (aligned) pairs: P + = n w t:t+H real , w sim (a real ) t:t+H o t , wheretis the starting time of video clip andHis fixed clip length. We also design negative pairsP − =P − shift ∪P − cross of two types: (1) Temporally shifted negatives. These negatives deliber- ately misalign time within the same episode: P − shift = n w t:t+H real , w sim (a real ) t ′ :t ′ +H t ′ ̸= t o . (2) Cross-episode negatives. These negatives pair a real clip with a simulator rollout from a different episode: P − cross = n w t:t+H real , w sim (a ′ real ) t:t+H a ′ real ̸= a real o . LetP =P + ∪P − denote the training dataset for the atten- tive probe. For a sampled pair(w t 1 :t 1 +H 1 ,w t 2 :t 2 +H 2 )∼P, we encode each clip using a pre-trained video encoder f φ : z 1 = f φ (w t 1 :t 1 +H 1 ), z 2 = f φ (w t 2 :t 2 +H 2 ). Next, we concatenate the embeddings and feed them to an attention-based probeg θ (·)to predict alignment. The attentive probe consists of a single cross-attention layer with a learnable query token that attends to the concatenated video embeddings, followed by a linear head that outputs an alignment logit ℓ = g θ ([z 1 ,z 2 ]). Finally, we traing θ using binary cross-entropy loss. Let the alignment probabilityp = σ(ℓ), whereσ(·)denotes the sigmoid function. The training objective is L(θ;P) =E ((w 1 ,w 2 ),y)∼P h −y logp− (1− y) log(1− p) i . wherey = 1only if w 1 ,w 2 ) ∈ P + andy = 0other- wise. At inference time, given a neural trajectory sample (w gen ,a IDM ), we form the video pair(w gen ,w sim (a IDM ))and feed it to the trained attentive probeg θ . We retain the sam- ple only if the alignment probabilitypexceeds a threshold c, indicating the two videos are motion-consistent. Further details are provided in Appendix A.3. 3.3. Improve Neural Trajectory via Best-of-N Sampling Our filtering method can be used not only to select benefi- cial synthetic data but also to improve the neural trajectory by serving as a critic for the video generative model at in- ference time. Concretely, we adopt a Best-of-N sampling strategy: (1) we sampleNcandidate videos along with their IDM-predicted actions, and (2) we select the video-action pair with the highest critic score. As the critic score, we Phase 2: Co-finetuning Real Data Neural Trajectory Real Data Neural Trajectory Simulation Data VLA Co-finetuning VLA Pretraining Phase 1: Two Stage Training VLA Finetuning Figure 4. An overview of experimental design for RoboCurate. We conduct two-phase experiments: (1) pre-training on real data and neural trajectory followed by fine-tuning on simulation data, and (2) co-finetuning on real data and neural trajectory. use the alignment probabilitypintroduced in Section 3.2, which measures whether the motion in the synthetic video is consistent with the simulator rollout. A higher score in- dicates better alignment between the generated video and the IDM-inferred actions, making the selected pair more suitable for policy learning. This strategy is particularly appealing in fine-tuning settings, where data collection is typically constrained to a specific target task. Importantly, Best-of-N sampling improves the quality of neural trajectory without discarding samples, en- abling more effective use of our neural trajectory generation framework in data-scarce settings. 4. Experiments We design experiments to answer the following research questions: •Does data generated by RoboCurate effectively improve vision-language-action models (VLAs) when applied in the pre-training stage (see Tables 1 and 2)? • Does data generated by RoboCurate effectively improve VLAs when applied in the co-finetuning stage (see Ta- ble 3)? •How effective is the diversity augmentation strategy in RoboCurate (see Tables 1, 2, and 5)? •How effective is the filtering strategy proposed in RoboCu- rate (see Tables 1, 2, 4, and 6)? We evaluate RoboCurate and answer these questions under two training regimes: (1) two-stage training: pre-training followed by fine-tuning, and (2) co-finetuning on real data and neural trajectory. We provide a brief overview of our experimental design in Figure 4. 5 RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning Table 1. Performance comparison on GR-1 Tabletop (NVIDIA et al., 2025). We report the average success rate (%) over 50 trials across 24 tasks (18 rearrangement and 6 articulated), trained with varying number of demonstrations. DreamGen (Jang et al., 2025) denotes neural trajectory generated by I2V model and labeled with IDM-predicted actions, without our visual diversity pipeline. For a fair comparison, all methods that leverage synthetic data use the same pre-filtering dataset size of 10K neural trajectory. MethodSynth.Filtering 300 Demos1000 Demos RearrangementArticulatedAvg.RearrangementArticulatedAvg. Real✗16.113.315.429.732.330.3 w/ DreamGen✓✗21.114.719.531.733.732.2 w/ RoboCurate (Ours)✓✗23.221.022.733.239.334.8 w/ RoboCurate (Ours)✓25.428.726.238.237.037.9 Table 2. Performance comparison on DexMimicGen (Jiang et al., 2025). We report the average success rate (%) over 50 trials across 6 tasks (3 GR-1 Humanoid and 3 Bimanual Panda Arms (Dexterous Hands)), trained with 100 demonstrations per task. Results for tables are averaged over 3 random seeds. DreamGen (Jang et al., 2025) denotes neural trajectory generated by I2V model and labeled with IDM-predicted actions, without our visual diversity pipeline. For a fair comparison, all methods that leverage synthetic data use the same pre-filtering dataset size of 10K neural trajectory. MethodSynth.FilteringGR-1 HumanoidBimanual Panda Arms (Dexterous Hands)Avg. Real✗56.932.244.6 w/ DreamGen✓✗57.135.646.4 w/ RoboCurate (Ours)✓✗59.339.349.3 w/ RoboCurate (Ours)✓62.740.951.8 Figure 5. Visualization of benchmarks. We visualize our bench- mark settings (from left to right): (1) GR-1 Tabletop (NVIDIA et al., 2025), (2) DexMimicGen (Jiang et al., 2025) with bimanual Panda arms with dexterous hands, (3) DexMimicGen with GR-1 humanoid, and (4) a real-robot benchmark on dexterous-hand hu- manoid robot ALLEX. 4.1. Pre-training Experiments Datasets. For the pre-training stage, we use Action- Net (Fourier ActionNet Team, 2025) as the default pre- training dataset. ActionNet consists of bimanual tabletop manipulation trajectories collected on Fourier GR1-T1 hu- manoid, captured from an ego-view camera. GR1-T1 hu- manoid is equipped with 6-DoF dexterous hands, resulting in a total of 44-dimensional joint-space states and actions. While the original dataset has 30K teleoperated episodes, we use a 3K subset for pre-training by applying unique prompt filtering to reduce redundant episodes with identical task instructions. We use this dataset for both video generative model training and policy learning. For the fine-tuning stage, we fine-tune and evaluate models on two simulated dexterous manipulation benchmarks: GR- 1 Tabletop (NVIDIA et al., 2025), and DexMimicGen (Jiang et al., 2025). GR-1 Tabletop focuses on dexterous hand con- trol for tabletop manipulation and includes a variety of ob- jects and placements. DexMimicGen focuses on bimanual manipulation and supports cross-embodiment evaluation, such as Panda arms and GR-1 humanoid. Baselines. We primarily use GR00T N1.5 (Bjorck et al., 2025) as the base policy and do not load pre-trained weights for the action head. We design our baselines along two axes: (1) the choice of pre-training data, and (2) whether action- level filtering is applied. For pre-training data, we consider three settings: (1) real data only, (2) real data combined with neural trajectory following the pipeline of Jang et al. (2025), and (3) real data combined with our visually augmented neural trajectory via I2I and V2V pipeline. In particular, the neural trajectory in (2) consists of image-to-video gen- eration and pseudo-action labeling by IDM, without our visual diversity strategy. We further compare uncurated and curated versions of visually augmented neural trajectory in (3) to evaluate the effect of our filtering strategy. We provide full implementation details for policy training in Appendix A.1. We also provide details on neural trajectory generation in Appendix B. Experimental results. Table 1 and 2 show that pre- training with our synthetic data generation framework con- sistently outperforms all other baselines. First, we observe that our visual augmentation pipeline (e.g., I2I editing and V2V transfer) substantially improves downstream task per- formance. Second, our action-level filtering is effective for curating neural trajectory and further enhances VLA per- formance, even with a smaller amount of data. Finally, the effectiveness of our neural trajectory is not limited to a sin- gle embodiment; neural trajectory’s prior can be transferred across embodiments, e.g., a policy pre-trained with GR-1 6 RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning Table 3. Performance comparison on ALLEX humanoid. We report the average success rate (%) over 24 trials across 3 tasks (1 seen task and 2 unseen tasks). We collect 48 demonstrations only for the seen task. We generate synthetic data for both seen and unseen tasks (48 episodes for the seen task and 50 episodes for each unseen task) and co-finetune with real data. DreamGen (Jang et al., 2025) denotes neural trajectory generated by I2V model and labeled with IDM-predicted actions, without our visual diversity pipeline. * denotes that our filtering strategy is applied during the generation stage via Best-of-N sampling, and visual augmentation is not used in this case. MethodSynth.Filtering In-distributionNovel ObjectNovel Behavior Avg. Pick and Place CanPick and Place CupPour Can Real✗25.016.70.013.9 w/ DreamGen✓✗37.533.312.527.8 w/ RoboCurate* (Ours)✓47.943.825.038.9 Table 4. Comparison with other filtering strategies on GR-1 Tabletop (NVIDIA et al., 2025). We report the average success rate (%) over 50 trials across 24 tasks (18 rearrangement and 6 articulated), trained with 1,000 demonstrations per task. Dream- GenBench (Jang et al., 2025) and VideoCon-Physics (Bansal et al., 2024) are physical plausibility benchmarks for generated videos. For a fair comparison, all methods use the same 10K pre-filtering neural trajectory. MethodFilteringRearrangementArticulatedAvg. Real + Neural✗30.736.332.1 w/ DreamGenBench✓33.740.735.4 w/ VideoCon-Physics✓33.939.035.2 w/ RoboCurate (Ours)✓37.740.338.3 humanoid neural trajectory can be successfully fine-tuned for bimanual Panda arms with dexterous hands. 4.2. Co-finetuning Experiments Datasets. We design real-world tasks with Allex hu- manoid to validate that neural trajectory can improve real- world generalization without collecting additional real data. Allex humanoid is equipped with an ego-view camera and two 15-DoF dexterous hands, yielding 48-dimensional joint- space states and actions. Specifically, we conduct experi- ments on 3 tabletop manipulation tasks: (1) pick-and-place cube, an in-distribution (ID) task for which we manually collect 48 teleoperated demonstrations; (2) pick-and-place cup, an out-of-distribution (OOD) task involving a novel object; and (3) pour can, an OOD task involving a novel behavior. We note that we do not collect any real data for these OOD tasks. Baselines. We primarily use a pre-trained GR00T N1.5 (Bjorck et al., 2025) as the base policy and fine-tune it for ALLEX humanoid. We design baselines to validate that our filtering strategy can directly improve the quality of neural trajectory. For co-finetuning data, we consider three settings: (1) real data only, (2) real data combined with neural trajectory following the pipeline of Jang et al. (2025), and (3) real data combined with Best-of-N sampled neural trajectory selected by our filtering strategy. Since the co- finetuning phase requires task-specific data, we generate Table 5. Diversity analysis of RoboCurate on GR-1 Tabletop (NVIDIA et al., 2025). We report the average success rate (%) over 50 trials across 24 tasks (18 rearrangement and 6 articulated), trained with 300 demonstrations per task. For a fair comparison, we fix the dataset size to 10K neural trajectory across all diversity setups. Diversity300 Demos VisualTaskRearrangementArticulatedAvg. ✗25%12.712.012.5 ✗50%18.214.717.3 ✗100%18.922.019.7 ✓100%22.226.723.3 neural trajectory using the same fixed task instruction for each task, following the pipeline in Appendix B. For Best-of- N sampling in (3), we sampleNcandidates with different random seeds for Gaussian noise, compute the filtering score (see Section 3.3), and keep the highest-scoring one; in (2), we generate a single sample using a fixed seed. We provide full implementation details for policy training in Appendix A.2. Experimental results. Table 3 shows that RoboCurate remains effective when co-finetuning policies on real data combined with neural trajectory. Notably, even in the ab- sence of real-world data for out-of-distribution (OOD) tasks (e.g., pour can), policies trained solely on neural trajectory can successfully perform OOD behaviors (0%→12.5% on pour can). We note that we generate 48 episodes for ID task and 50 episodes for each OOD task in our experiments. Our action-level filtering is applied during video generation, where it serves as a critic in a Best-of-N sampling proce- dure (see Section 3.3). By selecting action-verified video candidates, this strategy yields an additional∼40% relative gain over using unfiltered neural trajectory. Overall, these results indicate that our action-quality metric can effectively identify beneficial samples for policy learning, leading to improved downstream task performance. 4.3. Ablation Studies and Analyses We investigate the effectiveness of the proposed components of RoboCurate under the two-stage training setup shown in Figure 4. We use ActionNet (Fourier ActionNet Team, 7 RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning Table 6. Component-wise ablation of RoboCurate’s filtering strategy on GR-1 Tabletop (NVIDIA et al., 2025). We report the average success rate (%) over 50 trials across 24 tasks (18 rearrangement and 6 articulated), trained with 300 demonstrations per task. With attentive probe, we discard video pairs classified as motion-inconsistent; without the attentive probe, we discard pairs whose video embedding cosine similarity falls below a fixed threshold. With human labels, we train the attentive probe on 2K manually binary-labeled video pairs; without human labels, we train it using our proposed positive and negative pairs constructed from the real-world dataset (see Figure 3). For a fair comparison, all methods use the same 10K pre-filtering neural trajectory and use V-JEPA2 (Assran et al., 2025) as the pre-trained video encoder for filtering. MethodFiltering Attentive Probe Human Label RearrangementArticulatedAvg. Real + Neural✗--23.221.022.7 w/ V-JEPA2✓✗-26.117.023.8 w/ V-JEPA2✓24.819.723.5 w/ RoboCurate (Ours)✓✗25.428.726.2 GR-1 TabletopDexMimicGenReal-world 0 10 20 30 40 50 Success Rate (%) Real Real + DreamGen Real + RoboCurate Figure 6. Aggregation of RoboCurate performance. Com- parison of VLA performance across different pre-training or co- finetuning settings on GR-1 Tabletop (NVIDIA et al., 2025), DexMimicGen (Jiang et al., 2025), and real-world benchmark. RoboCurate shows strong performance across all settings. 2025) as the default pre-training dataset and GR00T N1.5 (Bjorck et al., 2025) as the base policy, without loading pre-trained weights for the action head. Across all ablation experiments, we fix the pre-filtering dataset size to 10K neural trajectory for pre-training. Unless otherwise stated, we follow the same neural trajectory generation pipeline and policy training setup as in the main experiments. Comparison with other filtering strategies. To verify the effectiveness of our filtering strategy, we conduct ex- periments comparing our method with other methods that assess the physical plausibility of generated videos. Table 4 shows that our action-level filtering outperforms video-only physical plausibility filtering methods. In particular, Dream- GenBench (Jang et al., 2025) queries VLM with a fixed prompt to judge whether a generated video violates basic physics. Similarly, VideoCon-Physics (Bansal et al., 2024) queries VideoCon, a 7B VLM, to predict whether a video follows physical laws. Our results demonstrate that veri- fying action quality is crucial for policy learning, whereas relying solely on VLM-based video-level plausibility as- sessments is insufficient for curating beneficial synthetic data. Effect of attentive-probe training.To better understand the mechanism for identifying visual motion consistency, we conduct a component-wise ablation of the action verifier. Table 6 shows that the attentive probe trained on V-JEPA2 (Assran et al., 2025) features using our proposed strategy yields the best VLA performance, outperforming other base- lines. As a simple baseline, we filter samples by thresh- olding the cosine similarity between frozen video encoder embeddings of a generated video and its simulator replay. However, this similarity-based filter yields limited gains, as the pre-trained encoder may not generalize well to simulator rollouts and instead rely on appearance cues rather than motion. In contrast, our probe learns a motion-consistency classifier on fused video embeddings, enabling the detection of subtle motion discrepancies between a generated video and its simulator replay. Moreover, obtaining reliable su- pervision is non-trivial: human labels provide little benefit and can even underperform the naive baseline, likely due to noisy judgments on subtle action mismatches. By contrast, our automatic positive/negative pair construction (see Fig- ure 3) offers consistent supervision for fine-grained motion differences, leading to improved downstream performance. Effect of diversity in neural trajectory. To evaluate the effect of diversity in our neural trajectory, we analyze two factors: (1) task diversity and (2) visual diversity. Table 5 in- dicates that VLA performance increases monotonically with higher task diversity under a fixed dataset size of 10K neural trajectory. Specifically, we vary task diversity by changing the number of distinct tasks, where each task corresponds to a unique combination of skill, target object, placement, and hand type (i.e., combinatorial coverage). These results highlight the importance of carefully designing prompts for VLM to generate diverse task instructions. Moreover, Table 5 demonstrates that RoboCurate’s visual diversity pipeline further improves performance while keeping the same task diversity ratio, underscoring the value of visually diverse observations beyond task coverage alone. 8 RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning 5. Related Work Video generative models for robot learning.Video gen- erative models were originally developed for general human video generation, and prior work has explored using such videos to provide novel scenes or motions for robot policy learning (Bharadhwaj et al., 2024), sometimes combined with simple tracking signals (Liang et al., 2024). Another line of work leverages text-to-video (T2V) diffusion models to generate synthetic trajectories and infer pseudo actions via inverse dynamics models (IDMs) for task generalization (Luo et al., 2025). More recent studies further explore using video generative models as the robot policy itself, treating video prediction as a proxy for policy learning (Wu et al., 2023). Alongside these trends, Jang et al. (2025) adopt state-of-the-art video generative models as a data generation pipeline that synthesizes robot videos and extracts pseudo actions for policy training. Our method also follows this data-centric pipeline, using video generation as a source of synthetic experience for learning from limited robot data. Synthetic data generation for robot policies. Collect- ing large-scale real-world robot data is costly and labor- intensive, which has led many works to rely on simulation datasets as a primary alternative (Mandlekar et al., 2023; Nasiriany et al., 2024). While simulation enables scalable data collection, it suffers from several limitations, includ- ing the sim-to-real gap, inaccurate physical modeling, and difficulties in handling articulated objects, deformable mate- rials, or complex tool interactions (Zhu et al., 2022). As an alternative, recent works explore generating synthetic robot data using generative models, aiming to bypass simulation constraints and improve scalability (Mandi et al., 2022; Al- haija et al., 2025). However, such approaches often suffer from limited visual and scene diversity. For example, Jang et al. (2025) first samples plausible task instructions from an initial image, and then generates synthetic robot videos using image-to-video (I2V) models, but diversity remains constrained by the manually collected initial visual context. In contrast, our method explicitly increases visual diversity by combining image-to-image (I2I) editing and video-to- video (V2V) transfer, enabling broader variation in scenes and appearances for synthetic data generation. 6. Conclusion We present RoboCurate, a synthetic robot data generation framework that improves neural trajectory by (i) verifying IDM-predicted actions through simulator-replay consistency and (i) expanding observation diversity via I2I editing and action-preserving V2V transfer. Our results show that in- creasing the visual diversity of neural trajectory is a key driver of downstream policy performance, and that our I2I and V2V pipeline yields more beneficial synthetic data than naive I2V generation. Moreover, our action-consistency measure is effective both for filtering training data for policy learning and as a critic for Best-of-N sampling in video generation, consistently boosting performance across sim- ulation and real-world benchmarks. We hope RoboCurate encourages principled quality evaluation for synthetic robot data and enables more effective policy learning from curated neural trajectory. Impact Statement This paper presents RoboCurate, a framework to expand the capabilities of robot policies using synthetic robot data. The framework suggests that a policy can learn novel actions present in synthetic robot data. Careful consideration is required to ensure that synthetic robot data and resulting policies align with social values. Acknowledgements This work, including the ALLEX humanoid platform, was supported by RLWRLD. References Alhaija, H. A., Alvarez, J., Bala, M., Cai, T., Cao, T., Cha, L., Chen, J., Chen, M., Ferroni, F., Fidler, S., et al.Cosmos-transfer1: Conditional world genera- tion with adaptive multimodal control. arXiv preprint arXiv:2503.14492, 2025. Ali, A., Bai, J., Bala, M., Balaji, Y., Blakeman, A., Cai, T., Cao, J., Cao, T., Cha, E., Chao, Y.-W., et al. World simulation with video foundation models for physical ai. arXiv preprint arXiv:2511.00062, 2025. Assran, M., Bardes, A., Fan, D., Garrido, Q., Howes, R., Muckley, M., Rizvi, A., Roberts, C., Sinha, K., Zholus, A., et al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985, 2025. Baker, B., Akkaya, I., Zhokov, P., Huizinga, J., Tang, J., Ecoffet, A., Houghton, B., Sampedro, R., and Clune, J. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639–24654, 2022. Bansal, H., Lin, Z., Xie, T., Zong, Z., Yarom, M., Bitton, Y., Jiang, C., Sun, Y., Chang, K.-W., and Grover, A. Videophy: Evaluating physical commonsense for video generation. arXiv preprint arXiv:2406.03520, 2024. Bharadhwaj, H., Dwibedi, D., Gupta, A., Tulsiani, S., Do- ersch, C., Xiao, T., Shah, D., Xia, F., Sadigh, D., and Kirmani, S. Gen2act: Human video generation in novel 9 RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning scenarios enables generalizable robot manipulation. arXiv preprint arXiv:2409.16283, 2024. Bjorck, J., Blukis, V., Casta ̃ neda, F., Cherniadev, N., Da, X., Ding, R., Fan, L., Fang, Y., Fox, D., Hu, F., Huang, S., et al.Gr00t n1.5:An im- proved open foundation model for generalist hu- manoid robots.https://research.nvidia. com/labs/gear/gr00t-n1_5/, June 2025. Ac- cessed: 2025-09-09. Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. Bu, Q., Cai, J., Chen, L., Cui, X., Ding, Y., Feng, S., Gao, S., He, X., Hu, X., Huang, X., et al. Agibot world colosseo: A large-scale manipulation platform for scal- able and intelligent embodied systems. arXiv preprint arXiv:2503.06669, 2025. Cheang, C.-L., Chen, G., Jing, Y., Kong, T., Li, H., Li, Y., Liu, Y., Wu, H., Xu, J., Yang, Y., et al. Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation. arXiv preprint arXiv:2410.06158, 2024. Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. Fourier ActionNet Team, Y. M. Actionnet: A dataset for dexterous bimanual manipulation. 2025. Fu, X., Wang, X., Liu, X., Bai, J., Xu, R., Wan, P., Zhang, D., and Lin, D. Learning video generation for robotic manipulation with collaborative trajectory control. arXiv preprint arXiv:2506.01943, 2025. Google.A new era of intelligence with Gemini 3 — blog.google.https://blog.google/ products-and-platforms/products/ gemini/gemini-3/, 2025. Hirose, N., Glossop, C., Shah, D., and Levine, S. Omnivla: An omni-modal vision-language-action model for robot navigation. arXiv preprint arXiv:2509.19480, 2025. Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang, Y., Huang, Y., Zhao, W., et al. Minicpm: Unveiling the potential of small language models with scalable train- ing strategies. arXiv preprint arXiv:2404.06395, 2024. Jang, J., Ye, S., Lin, Z., Xiang, J., Bjorck, J., Fang, Y., Hu, F., Huang, S., Kundalia, K., Lin, Y.-C., et al. Dreamgen: Unlocking generalization in robot learning through video world models. arXiv preprint arXiv:2505.12705, 2025. Jiang, Z., Xie, Y., Lin, K., Xu, Z., Wan, W., Mandlekar, A., Fan, L. J., and Zhu, Y. Dexmimicgen: Automated data generation for bimanual dexterous manipulation via imi- tation learning. In 2025 IEEE International Conference on Robotics and Automation (ICRA), p. 16923–16930. IEEE, 2025. Khazatsky, A., Pertsch, K., Nair, S., Balakrishna, A., Dasari, S., Karamcheti, S., Nasiriany, S., Srirama, M. K., Chen, L. Y., Ellis, K., et al.Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945, 2024. Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakr- ishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., San- keti, P., et al. Openvla: An open-source vision-language- action model. arXiv preprint arXiv:2406.09246, 2024. Kong, W., Tian, Q., Zhang, Z., Min, R., Dai, Z., Zhou, J., Xiong, J., Li, X., Wu, B., Zhang, J., et al. Hunyuan- video: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024. Labs, B. F. FLUX.2: Frontier Visual Intelligence.https: //bfl.ai/blog/flux-2, 2025. Liang, J., Liu, R., Ozguroglu, E., Sudhakar, S., Dave, A., Tokmakov, P., Song, S., and Vondrick, C. Dreamitate: Real-world visuomotor policy learning via video genera- tion. arXiv preprint arXiv:2406.16862, 2024. Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022. Luo, C., Zeng, Z., Du, Y., and Sun, C. Solving new tasks by adapting internet video knowledge. arXiv preprint arXiv:2504.15369, 2025. Mandi, Z., Bharadhwaj, H., Moens, V., Song, S., Ra- jeswaran, A., and Kumar, V. Cacti: A framework for scalable multi-task multi-scene visual imitation learning. arXiv preprint arXiv:2212.05711, 2022. Mandlekar, A., Nasiriany, S., Wen, B., Akinola, I., Narang, Y., Fan, L., Zhu, Y., and Fox, D. Mimicgen: A data gen- eration system for scalable robot learning using human demonstrations. arXiv preprint arXiv:2310.17596, 2023. Motamed, S., Chen, M., Van Gool, L., and Laina, I. Travl: A recipe for making video-language models better judges of physics implausibility. arXiv preprint arXiv:2510.07550, 2025. 10 RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning Nasiriany, S., Maddukuri, A., Zhang, L., Parikh, A., Lo, A., Joshi, A., Mandlekar, A., and Zhu, Y. Robocasa: Large- scale simulation of everyday tasks for generalist robots. arXiv preprint arXiv:2406.02523, 2024. Nvidia. GitHub - nvidia-cosmos/cosmos-predict2: Cosmos- Predict2 is a collection of general-purpose world foun- dation models for Physical AI that can be fine-tuned into customized world models for downstream appli- cations. — github.com.https://github.com/ nvidia-cosmos/cosmos-predict2, 2026. [Ac- cessed 29-01-2026]. NVIDIA, Bjorck, J., Casta ̃ neda, F., Cherniadev, N., Da, X., Ding, R., Fan, L. J., Fang, Y., Fox, D., Hu, F., Huang, S., Jang, J., Jiang, Z., Kautz, J., Kundalia, K., Lao, L., Li, Z., Lin, Z., Lin, K., Liu, G., Llontop, E., Magne, L., Mandlekar, A., Narayan, A., Nasiriany, S., Reed, S., Tan, Y. L., Wang, G., Wang, Z., Wang, J., Wang, Q., Xiang, J., Xie, Y., Xu, Y., Xu, Z., Ye, S., Yu, Z., Zhang, A., Zhang, H., Zhao, Y., Zheng, R., and Zhu, Y. Gr00t n1: An open foundation model for generalist humanoid robots, 2025. URL https://arxiv.org/abs/2503.14734. O’Neill, A., Rehman, A., Maddukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., Jain, A., et al. Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collab- oration 0. In 2024 IEEE International Conference on Robotics and Automation (ICRA), p. 6892–6903. IEEE, 2024. Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, p. 4195–4205, 2023. Shen, Y., Wei, F., Du, Z., Liang, Y., Lu, Y., Yang, J., Zheng, N., and Guo, B. VideoVLA: Video generators can be generalizable robot manipulators. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URLhttps://openreview.net/ forum?id=UPHlqbZFZB. Team, G., Ye, A., Wang, B., Ni, C., Huang, G., Zhao, G., Li, H., Li, J., Zhu, J., Feng, L., et al. Gigabrain-0: A world model-powered vision-language-action model. arXiv preprint arXiv:2510.19430, 2025. Tschannen, M., Gritsenko, A., Wang, X., Naeem, M. F., Alabdulmohsin, I., Parthasarathy, N., Evans, T., Beyer, L., Xia, Y., Mustafa, B., et al. Siglip 2: Multilingual vision-language encoders with improved semantic under- standing, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025. Walke, H. R., Black, K., Zhao, T. Z., Vuong, Q., Zheng, C., Hansen-Estruch, P., He, A. W., Myers, V., Kim, M. J., Du, M., et al. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning, p. 1723–1736. PMLR, 2023. Wan, T., Wang, A., Ai, B., Wen, B., Mao, C., Xie, C.-W., Chen, D., Yu, F., Zhao, H., Yang, J., et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025. Wu, H., Jing, Y., Cheang, C., Chen, G., Xu, J., Li, X., Liu, M., Li, H., and Kong, T. Unleashing large-scale video generative pre-training for visual robot manipula- tion. arXiv preprint arXiv:2312.13139, 2023. Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang, Y., Hong, W., Zhang, X., Feng, G., et al. Cogvideox: Text-to-video diffusion models with an ex- pert transformer. In International Conference on Learn- ing Representations, 2025. Zhang, S., Qiao, Y., Wang, Q., Guo, L., Wei, Z., and Liu, J. Flexvln: Flexible adaptation for diverse vision-and-language navigation tasks. arXiv preprint arXiv:2503.13966, 2025. Zhou, S., Du, Y., Chen, J., Li, Y., Yeung, D.-Y., and Gan, C. Robodreamer: Learning compositional world models for robot imagination. arXiv preprint arXiv:2404.12377, 2024. Zhu, J., Cherubini, A., Dune, C., Navarro-Alarcon, D., Alambeigi, F., Berenson, D., Ficuciello, F., Harada, K., Kober, J., Li, X., et al. Challenges and outlook in robotic manipulation of deformable objects. IEEE Robotics & Automation Magazine, 29(3):67–77, 2022. Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., Wu, J., Wohlhart, P., Welker, S., Wahid, A., et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, p. 2165–2183. PMLR, 2023. 11 RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning A. Implementation Details A.1. Pre-training Experiments For the pre-training stage, we adopt Warmup-Stable-Decay (WSD; (Hu et al., 2024)) learning rate scheduler for policy training. Specifically, we pre-train all models for 60K gradient steps, maintaining a constant high learning rate for the first 50K steps, followed by a sharp decay over the last 10K steps. We hypothesize that the quality of neural trajectory becomes more critical after the model has observed a sufficient amount of data. Therefore, we use all neural trajectory for the first 50K steps and switch to only curated neural trajectory (by RoboCurate) after 50K steps. Second, we use separate embodiment tags for real (from ActionNet (Fourier ActionNet Team, 2025)) and neural trajectory, even though they share the same GR-1 embodiment, since the statistics of IDM actions differ from those of ActionNet. During training, we sample real and neural trajectory in a 1:1 ratio by default. Otherwise, we follow the original training configuration of GR00T N1.5 (Bjorck et al., 2025), except that we set the global batch size to 512 and train for 60K global steps. For the fine-tuning stage, we fine-tune GR00T N1.5 from the pre-trained checkpoint corresponding to each experimental setup (see Section 4.1). We note that the fine-tuning dataset is identical for the same fine-tuning task. We follow the original GR00T N1.5 fine-tuning configurations, except that we set the global batch size to 256 and tune the number of gradient steps per downstream task dataset (up to 60K gradient steps) to optimize performance. We train separate models for different embodiments and observation viewpoints (e.g., GR-1 humanoid in ego view, GR-1 humanoid in front view, and bimanual Panda arms with dexterous hands) on DexMimicGen (Jiang et al., 2025). A.2. Co-finetuning Experiments We follow the training setup of GR00T N1.5 (Bjorck et al., 2025) and fine-tune a pretrained GR00T N1.5 model for 10K gradient steps. During co-finetuning, we mix real and neural data and maintain a 1:1:1 sampling ratio across ”Pick and place can”, ”Pick and place cup”, and ”Pour can” tasks. We follow the original GR00T N1.5 fine-tuning configurations, except that we set the global batch size to 128 and the number of gradient steps to 10K. A.3. Attentive Probe Training We train an attentive probe on top of V-JEPA2 (Assran et al., 2025). We follow the V-JEPA2 implementation and training strategy, and attach the probe to a pre-trained 0.3B V-JEPA2 large model wtih the backbone frozen during training. The hyperparameters for attentive probe training are as follows. Table 7. Attentive probe training hyperparameters. HyperparameterValue Cross-attn heads (n heads ) 8 Cross-attn layers (L) 1 Clip length (H ) 16 Temporal stride (frames) 4 Input resolution 256×256 OptimizerAdamW Learning rate 1e-4 Batch size (pairs) 32 Maximum epochs 50 12 RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning B. Neural Trajectory Generation Video generation. We use Cosmos-Predict2-14B (Nvidia, 2026) as the base video generative model and fine-tune it from the pre-trained checkpoint. Specifically, we fine-tune on ActionNet (Fourier ActionNet Team, 2025) and GR00T-GR1-100 1 (Jang et al., 2025) for the pre-training experiments, and on a manually collected ALLEX dataset for the co-finetuning experiments. Unless otherwise stated, we generate approximately 10K tabletop manipulation videos. For the co-finetuning setup, we instead generate 48/50/50 videos per one ID task and two OOD tasks, respectively. All videos are conditioned on initial images from the fine-tuning datasets and plausible task instructions. We use Gemini 3 Pro (Google, 2025) to generate instructions for each initial image using a carefully designed system prompt. We provide more details in Appendix C.1. In addition, we use FLUX.2-dev (Labs, 2025) for image editing and Cosmos-Transfer2.5-2B (Ali et al., 2025) for video-to- video transfer as visual augmentation (see Section 3.1), except in the co-finetuning experiments. For image editing, we use initial images from the same dataset used for video generation, and for V2V transfer, we use the subset of previously generated videos as input. We note that we generate approximately 10K augmented tabletop manipulation videos, with an I2I:V2V ratio of 2:1 for efficiency given the generation cost. Post-processing. We apply post-processing to our neural trajectory along two axes, inspired by NVIDIA et al. (2025): (1) instruction following and (2) trajectory plausibility. First, we filter out videos conditioned on implausible tasks, as the initial instructions generated by VLM may be ill-posed. Second, we check whether the generated videos correctly follow instructions, and re-caption videos that do not follow the instructions. Since instruction following directly determines task success, we apply this post-processing to all neural trajectory dataset by default (resulting in exactly 10K neural trajectory after this step, except in the co-finetuning stage). Finally, we check whether robot hand trajectories in the generated videos are plausible, exhibiting correct sense of depth and spatial orientation. We focus on this aspect due to common failures in neural trajectory involving incorrect approach distances and object interactions. We prompt VLM to evaluate the video on a Likert scale from 1 to 5 and select videos with score greater than or equal to 3. We use Gemini 3 Pro (Google, 2025) for instruction following post-processing and Gemini 2.5 Flash (Comanici et al., 2025) for trajectory plausibility post-processing, by feeding 8 video frames as input. Details of the specific prompts used are provided in Appendix C.2. We note that trajectory plausibility post-processing is applied only prior to out action-level filtering stage. Pseudo-action labeling. We use diffusion transformer with SigLIP-2 vision encoder (Tschannen et al., 2025) as our base IDM, following Jang et al. (2025). For pre-training setup, we use an open-source pre-trained checkpoint 2 for GR-1 embodiment. For fine-tuning setup, we train the IDM from scratch for ALLEX embodiment using manually collected data (the same data used to fine-tune the video model), with a global batch size of 64, and 20K gradient steps. 1 https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-GR00T-GR1 2 https://huggingface.co/seonghyeonye/IDM_gr1 13 RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning C. Prompt for VLM C.1. Task Instruction for Image-to-Video Generative Model Task instruction generation <system> You are an expert at analyzing robot manipulation scenes and generating diverse, realistic task instructions. Your task is to generate exactly 5 different plausible robot manipulation tasks that could realistically be performed in the scene. IMPORTANT: Each instruction must use a SPECIFIC hand as follows: - Instruction 1: Use the hands[0] hand - Instruction 2: Use the hands[1] hand - Instruction 3: Use the hands[2] hand - Instruction 4: Use the hands[3] hand - Instruction 5: Use the hands[4] hand Each instruction MUST: 1. Follow this exact format: "The robot arm is performing a task. Use the [specified hand] hand to [specific action]." 2. Be based on what you observe in the frame (objects, positions, colors, spatial relationships) 3. Be actionable and realistic for a robot arm 4. Represent a primitive manipulation task doable in 10 seconds 5. Be independent and novel - not variations of each other 6. Use specific object names and descriptive details visible in the scene 7. Have a clear goal condition <user> Analyze the image and generate 5 robot manipulation instructions. IMPORTANT HAND CONSTRAINTS: - Instruction 1: Must use the hands[0] hand - Instruction 2: Must use the hands[1] hand - Instruction 3: Must use the hands[2] hand - Instruction 4: Must use the hands[3] hand - Instruction 5: Must use the hands[4] hand FOLLOW THESE STEPS: 1. First, let’s think step by step through the problem: - Identify all visible objects (names, colors, positions) - Describe spatial relationships between objects - List plausible manipulation tasks that can be done with either hand - Plan 5 diverse, novel instructions (each using its specified hand) 2. Generate the response with exactly 5 instructions in the specified format. REFERENCE EXAMPLES OF INSTRUCTION FORMAT (from similar datasets): These are examples of high-quality instructions to use as reference: <example1> <example2> <example3> <example4> <example5> Use these examples as style and format reference, but generate COMPLETELY NOVEL instructions based on what you see in the image. Generate exactly 5 instructions, each following the format: "The robot arm is performing a task. Use the [specified hand] hand to [action]." REMEMBER: Each instruction must use its assigned hand from the list above. Make sure all instructions are diverse and use specific object names from the scene. 14 RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning C.2. Post-processing for Neural Trajectory Filtering wrong instruction You are evaluating whether a video correctly follows a given instruction. Original Instruction: "instruction" The video is shown as a sequence of frames. Please carefully analyze the frames and determine if the video demonstrates the action described in the instruction. Answer with ONLY "YES" or "NO". Your answer: Recaptioning wrong instruction You are generating a natural language instruction that describes the action shown in a video of a robot performing a task. The video is shown as a sequence of frames. Please carefully analyze the frames and generate a concise, clear instruction that describes what action is being performed. Here are some examples of the format we’re looking for: examples_text Now, based on the video frames provided, generate a similar instruction that describes the action shown in the video. Your instruction (one sentence): Evaluating physical plausibility of video Evaluate the robot hand/end-effector for depth- and geometry-consistent motion across three phases. Judge whether the hand’s approach/contacts/transport align with the object’s apparent depth, size, and shape (including toward/away-from-camera motion). (1) Move-to-object (approach distance) Does the hand move a plausible distance to reach the object’s depth/position? No tele-grab: grasp/contact must not occur while the hand is visibly short in depth or with near-zero approach motion. (2) Grasp (contact precision) At grasp/contact, is the hand spatially precise relative to the object’s size/shape? Penalize offset grasps, clipping/penetration, or sloppy contact unlikely to stably hold the object. (3) Move-to-target (transport distance) After grasping, does the hand (with the object) move a plausible distance/direction toward the target location? Penalize large object displacement with little hand motion, drifting, or reaching the target without a clear transport trajectory. Rate plausibility (1-5) with these guidelines: 5 = All three phases look consistent; no noticeable depth/contact/transport issues. 4 = Mostly consistent; at most ONE minor issue (small offset/jitter) that doesn’t change the action outcome. 3 = Mixed; ONE clear issue OR multiple minor issues (e.g., slightly short reach, noticeably offset grasp, weakly justified transport). 2 = Largely implausible; ONE major issue or >=2 clear issues (tele-grab, clear mis-grasp, object moves without matching hand motion). 1 = Clearly implausible; >=2 major issues or a severe violation that dominates (obvious tele-grab and/or heavy clipping/drift) . Reply with a single integer (1-5). 15 RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning C.3. System Prompts for Image-to-Image Editing Initial Scene Description Prompt You are given a single image from an indoor tabletop robot manipulation scene. Describe the visible scene in one coherent paragraph from the robot’s first-person perspective. Mention the table or main support surface. Mention the main objects on or near the table, their colors, and approximate positions. Mention the surrounding room context briefly (e.g., kitchen, lab, cabinets). Focus only on what is currently visible; do NOT describe actions, motion, or intent. Use natural English in paragraph form (no bullet points) - INITIAL_SCENE_DESCRIPTION: Build Variation Prompt You are given: 1) An initial scene description of an indoor tabletop manipulation scenario. 2) A reference instruction describing what interaction happens in the scene. Use it as context to understand the type of manipulation, but you are NOT required to keep the same object. Your goal is to produce structured visual variation candidates for: [TABLE], [TARGET_OBJECT_VARIANTS], [SENTENCE_LIGHT], and [SENTENCE_BACKGROUND]. All variations must remain visually plausible for an indoor tabletop setup. IMPORTANT: Generate MULTIPLE diverse candidates for each category to enable varied counterfactual generation. MAXIMIZE DIVERSITY - avoid similar or repetitive descriptions: - [TABLE]: Generate 2-3 DIFFERENT table/surface descriptions with VARIED patterns, textures, materials, and finishes. CRITICAL: Each table description must be DISTINCTLY different from others. Include: - Material variety: wood (oak, pine, walnut, bamboo), metal (steel, aluminum, brushed), glass, laminate, marble,... - Pattern variety: solid colors, stripes, checkered, wood grain, marble veins, geometric patterns, plain surfaces - Texture variety: smooth, rough, matte, glossy, polished, brushed, textured, grainy, satin finish Examples of DIVERSE table descriptions: - "a dark walnut wooden table with visible grain patterns and a matte finish", -... AVOID: Similar descriptions (e.g., don’t generate "wooden table" and "oak table" - they’re too similar). - [TARGET_OBJECT_VARIANTS]: Generate 2-3 object variants with varied shapes, colors, and materials. CRITICAL: Variants can include BOTH the original target object (with different shape/color/material) AND completely different object types. IMPORTANT VARIANT OPTIONS: 1. Original target object with different attributes: Keep the SAME object TYPE as the reference instruction, but change SHAPE, COLOR, and/or MATERIAL/FINISH. Example: If original is "potato", you can generate "a round red potato with a glossy finish" or "a cylindrical brown potato with a matte texture" 2. Completely different object types: Use DIFFERENT object categories for maximum diversity. Example: If original is "potato", you can generate "a cylindrical blue ceramic mug" or "a rectangular green glass bottle" CRITICAL - RANDOM COMBINATION FOR EACH VARIANT: - For EACH variant, you MUST RANDOMLY combine: TYPE x SHAPE x COLOR x MATERIAL/FINISH - TYPE can be: (1) the original target object type, OR (2) a completely different object type - If using original target object type: Keep TYPE the same, but RANDOMLY change SHAPE, COLOR, and MATERIAL/FINISH - If using different object type: RANDOMLY select TYPE, SHAPE, COLOR, and MATERIAL/FINISH - Do NOT use predictable patterns (e.g., don’t always use round + red, or cylindrical + blue) - RANDOMLY select one type, one shape, one color, and one material/finish for each variant - Each variant should have a UNIQUE and RANDOM combination - Example of RANDOM combinations (if original is "potato"): * Variant 1: potato(type)+round(shape)+red(color)+glossy(finish) = "a round red potato with a glossy finish", * ... Examples of HIGHLY DIVERSE object descriptions: - "a round red apple with a glossy finish", -... IMPORTANT: Each variant must be a single, natural English phrase that includes: [object type] + [shape] + [color] + [ material/finish]. DO NOT include size descriptions (e.g., "small", "large", "medium"). AVOID: Predictable patterns in type x shape x color combinations. - [SENTENCE_LIGHT]: Generate 2-3 DIFFERENT lighting descriptions with varied brightness levels and qualities. CRITICAL: Only describe the general brightness and lighting quality that illuminates the table surface. IMPORTANT: Maximize diversity across brightness levels and qualities: - Brightness levels: bright, dim, soft, harsh, intense, subtle, muted, vibrant, subdued, gentle - Lighting qualities: even illumination, diffused light, natural daylight, warm ambient, ... Examples of DIVERSE lighting: - "bright and even illumination", -... DO NOT mention any specific light sources, lamps, fixtures, or objects. DO NOT mention windows, ceiling fixtures, or any physical items. Only describe the overall brightness and lighting quality that affects the table surface. AVOID: Similar descriptions (e.g., don’t generate "bright light" and "bright illumination" - they’re too similar). - [SENTENCE_BACKGROUND]: Generate 2-3 COMPLETELY DIFFERENT background descriptions with varied environments and styles. CRITICAL: Describe ONLY the room environment, walls, and space around/behind the table. You can add distractors on the table OR describe the background environment, but do NOT affect the TABLE or the main target objects. IMPORTANT: Maximize diversity across different background types: 1. Room environments: kitchen, lab, workshop, office, living room, garage, studio, classroom, warehouse, storage room 2. Wall styles: smooth white walls, textured beige walls, brick walls, painted walls, concrete walls, ... 3. Distractors on table: fruits/vegetables, tools, books, utensils, random objects (must be non-intrusive) 4. Background elements: furniture, equipment, shelves, cabinets, windows, doors, artwork, plants, storage items 16 RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning Build Variation Prompt - Continued Examples of HIGHLY DIVERSE backgrounds: - "sterile lab room with smooth white walls and a faint grid pattern on the floor behind the table", -... CRITICAL CONSTRAINTS: - The TABLE description must remain unchanged and unaffected by background descriptions. - The distractors or background should NOT interfere with, replace, or modify the TABLE or the main target objects. - Distractors should be additional small objects that do NOT touch, overlap, or interfere with the primary manipulation objects. - They should only add visual diversity in the background or as non-intrusive elements on the table surface, without affecting the TABLE itself or the target objects. AVOID: Similar backgrounds (e.g., don’t generate "white walls" and "smooth white walls" - they’re too similar). ------------------------------------------------------------ INITIAL_SCENE: initial_scene REFERENCE_INSTRUCTION (object identity only): reference_instruction ------------------------------------------------------------ Hard constraints (must follow) - Output MUST contain ALL labeled blocks exactly once. - Each list item MUST start with "- ", - [TARGET_OBJECT_VARIANTS] MUST contain exactly 2-3 complete object descriptions. General rules - Indoor tabletop environment only. - Do NOT invent new actions, motion, or outcomes. - [TARGET_OBJECT_VARIANTS]: Variants can include BOTH the original target object (with different shape/color/material) AND completely different object types. For example, if original is "potato", you can generate: (1) "a round red potato with a glossy finish" (same type, different attributes), (2) "a cylindrical blue ceramic mug" (different type), (3) "a rectangular green glass bottle" (different type). This maximizes diversity while allowing editing - [TARGET_OBJECT_VARIANTS] - RANDOM COMBINATION: For each variant, RANDOMLY combine TYPE x SHAPE x COLOR x MATERIAL/FINISH. TYPE can be the original target object OR a different object type. Do NOT use predictable or sequential patterns. Each combination must be unique and randomly selected. - Each variant must be a complete, natural English description. DO NOT include size descriptions. - [TABLE]: Ensure each table description differs significantly in material, pattern, or texture. Avoid generating multiple wooden tables with only slight variations. RANDOMLY select different material/pattern/texture combinations. - [SENTENCE_BACKGROUND]: Ensure each background differs significantly in environment type, wall style, or distractor type. Avoid generating multiple "white wall" variations. RANDOMLY select different environment/wall/distractor combinations. - [SENTENCE_LIGHT]: RANDOMLY select different brightness levels and lighting qualities. Avoid predictable patterns. - Focus only on scene elements: table, objects, lighting, background. - Do NOT mention perspective, viewpoint, or robot hands (these will be preserved from the original image). - Fluent natural English only. - MAXIMIZE DIVERSITY: When in doubt, choose the more diverse option. The goal is to generate counterfactual scenarios that are visually distinct from each other. - RANDOM SELECTION: All combinations must be RANDOM - no sequential patterns, no predictable rotations, no fixed orders. ------------------------------------------------------------ Return exactly: INITIAL_SCENE_REWRITE: ... [TABLE]: - ... [TARGET_OBJECT_VARIANTS]: - a round red apple with a glossy finish - a cylindrical blue ceramic mug with a matte texture - a rectangular green glass bottle with a smooth surface - a metallic silver wrench with brushed finish - ... [SENTENCE_LIGHT]: - .. [SENTENCE_BACKGROUND]: - ... 17 RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning Build Counterfactuals Prompt You are generating scene-only counterfactual prompts for an image-to-image editing model in STRUCTURED JSON format. The original image is provided as input to the image-to-image model, so the model will preserve the existing scene structure, robot view, and hands while applying counterfactual changes. The original image’s perspective, viewpoint, and robot hands will be preserved automatically. You only need to describe the scene elements that should be changed. You are given: 1) ORIGINAL_SCENE_CAPTION (for context only): initial_scene_caption 2) REFERENCE_INSTRUCTION (this has been parsed to extract target object): reference_instruction 3) VARIANTS_JSON (candidates): variants_json target_object_note ------------------------------------------------------------ Your task Generate exactly num_instructions counterfactual prompts in JSON format. Each prompt must be a valid JSON object following this EXACT structure: "scene": "Image 1 is the original scene image. Image 2 is a Canny edge map that defines the strict structural layout and boundaries of all objects in the scene. You MUST strictly follow the edge map structure from Image 2, but ignore the background edges from Image 2 and paint them in as a realistic image. Realistic image of a first-person perspective, from which a robot faces a [table description from TABLE candidates]. All objects on the table must be physically plausible - they must be properly placed on the table surface, not floating in the air, not overlapping with each other , and must follow realistic physics and gravity.", "subjects": [ "description": "The original target_object_from_ref if target_object_from_ref else ’target object’ has been edited to [target_object_variant from TARGET_OBJECT_VARIANTS]. The edited object must be a hand-sized object (appropriate size for robot manipulation, not too large or too small). The object must be placed on the table surface, resting naturally without floating or overlapping with other objects. The object must NOT be in the process of being manipulated by the robot - it must be in a static, resting state on the table.", "color_palette": ["extract and list colors from the target_object_variant description"] ], "style": "Ultra-realistic image. The output must NOT be a Canny edge map - it must be a realistic, photorealistic image with proper textures, colors, shading, and lighting. It must look like a real photograph, NOT like an edge map or line drawing.", "lighting": "[ONE lighting sentence from SENTENCE_LIGHT candidates]", "background": "[ONE background sentence from SENTENCE_BACKGROUND candidates]" IMPORTANT - RANDOM COMBINATION RULE (CRITICAL - MUST FOLLOW): - For each of the num_instructions prompts, you MUST randomly select DIFFERENT combinations from each category. - Do NOT select sequentially (e.g., don’t always use the first item from each list). - Do NOT use any predictable pattern (e.g., don’t rotate through items in order). - For each prompt, RANDOMLY pick one item from [TABLE], one from [TARGET_OBJECT_VARIANTS], one from [SENTENCE_LIGHT], and one from [SENTENCE_BACKGROUND]. - Each prompt should have a UNIQUE combination of all four categories to maximize diversity. - CRITICAL - TARGET OBJECT RANDOMIZATION: - When selecting from [TARGET_OBJECT_VARIANTS], treat each variant as a complete unit and select it randomly. - However, if you need to create NEW target object descriptions, you MUST randomly combine: * Object TYPE (e.g., apple, mug, bottle, wrench, banana, carrot, book, tool, container, etc.) - RANDOM selection * Object SHAPE (e.g., round, cylindrical, rectangular, square, oval, triangular, elongated, ...) - RANDOM selection * Object COLOR (e.g., red, blue, green, yellow, orange, black, white, gray, transparent, ...) - RANDOM selection * Material/Finish (e.g., ceramic, glass, metal, plastic, wood, rough, polished, brushed, ...) - RANDOM selection - The combination of TYPE x SHAPE x COLOR x MATERIAL must be RANDOM for each prompt - do NOT use predictable patterns. - Example of RANDOM combinations: * Prompt 1: round (shape) + red (color) + apple (type) + glossy (finish) = "a round red apple with a glossy finish" * ... - Each combination should be UNIQUE and RANDOMLY selected, not following any sequential or predictable pattern. CRITICAL JSON FORMATTING RULES: - Each JSON object must be valid JSON (proper quotes, commas, brackets). - The "scene" field must start with "Image 1 is the original scene image. Image 2 is a Canny edge map that defines the strict structural layout and boundaries of all objects in the scene. You MUST strictly follow the edge map structure from Image 2, but ignore the background edges from Image 2 and paint them in as a realistic image. Realistic image of a first-person perspective, from which a robot faces a [table description]". It MUST also include: "All objects on the table must be physically plausible - they must be properly placed on the table surface, not floating in the air, not overlapping with each other, and must follow realistic physics and gravity." - The "subjects" field must be an array with exactly one object containing: - "description": MUST explicitly state that "The original [target_object] has been edited to [target_object_variant]". It MUST also include: (1) "The edited object must be a hand-sized object (appropriate size for robot manipulation, not too large or too small)", (2) "The object must be placed on the table surface, resting naturally without floating or overlapping with other objects", (3) "The object must NOT be in the process of being manipulated by the robot - it must be in a static, resting state on the table" - "color_palette": An array of color strings extracted from the target object variant description - The "style" field must state that the output must be a realistic, photorealistic image and NOT a Canny edge map - The "lighting" field must be a simple string describing overall brightness/illumination quality (from [SENTENCE_LIGHT]) - The "background" field must be a simple string describing room environment or distractors (from [SENTENCE_BACKGROUND]) 18 RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning Build Counterfactuals Prompt - Continued CRITICAL CONTENT RULES: - The original target object ’target_object_from_ref if target_object_from_ref else ’target object’ has been edited to the variant in subjects[0].description - SCENE PHYSICAL PLAUSIBILITY RULE: All objects on the table must be physically plausible. Objects must be properly placed on the table surface, not floating in the air, not overlapping with each other, and must follow realistic physics and gravity. No objects should appear to be levitating or intersecting with other objects. - SUBJECTS DESCRIPTION RULE: The "description" field in subjects[0] MUST explicitly state: "The original target_object_from_ref if target_object_from_ref else ’target object’ has been edited to [variant description]". It MUST also include: (1) The edited object must be a hand-sized object (appropriate size for robot manipulation, not too large or too small), (2) The object must be placed on the table surface, resting naturally without floating or overlapping with other objects, (3) The object must NOT be in the process of being manipulated by the robot - it must be in a static, resting state on the table. - LIGHTING RULE: Only describe overall brightness and lighting quality. DO NOT mention light sources, lamps, fixtures, windows , or any physical objects. - BACKGROUND RULE: Either describe room environment/walls/space behind table, OR add distractors. Must NOT affect TABLE or target objects. - Do NOT mention robot hands, motion, actions, tasks, or intent. - Indoor tabletop scene only. Return ONLY valid JSON objects, one per line, under [COUNTERFACTUAL_INSTRUCTIONS]: [COUNTERFACTUAL_INSTRUCTIONS]: "scene": "Image 1 is the original scene image. Image 2 is a Canny edge map that defines the strict structural layout and boundaries of all objects in the scene. You MUST strictly follow the edge map structure from Image 2, but ignore the background edges from Image 2 and paint them in as a realistic image. Realistic image of a first-person perspective, from which a robot faces a [table description]. All objects on the table must be physically plausible - they must be properly placed on the table surface, not floating in the air, not overlapping with each other, and must follow realistic physics and gravity.", "subjects": [ "description": "The original target_object_from_ref if target_object_from_ref else ’target object’ has been edited to a round red apple with a glossy finish. The edited object must be a hand-sized object (appropriate size for robot manipulation, not too large or too small). The object must be placed on the table surface, resting naturally without floating or overlapping with other objects. The object must NOT be in the process of being manipulated by the robot - it must be in a static, resting state on the table.", "color_palette": ["red"] ], "style": "Ultra-realistic image. The output must NOT be a Canny edge map - it must be a realistic, photorealistic image with proper textures, colors, shading, and lighting. It must look like a real photograph, NOT like an edge map or line drawing.", "lighting": "[ONE lighting sentence from SENTENCE_LIGHT candidates]", "background": "[ONE background sentence from SENTENCE_BACKGROUND candidates]" "scene": "Image 1 is the original scene image. Image 2 is a Canny edge map that defines the strict structural layout and boundaries of all objects in the scene. You MUST strictly follow the edge map structure from Image 2, but ignore the background edges from Image 2 and paint them in as a realistic image. Realistic image of a first-person perspective, from which a robot faces a [table description]. All objects on the table must be physically plausible - they must be properly placed on the table surface, not floating in the air, not overlapping with each other, and must follow realistic physics and gravity.", "subjects": [ "description": "The original target_object_from_ref if target_object_from_ref else ’target object’ has been edited to a cylindrical blue ceramic mug with a matte texture. The edited object must be a hand-sized object (appropriate size for robot manipulation, not too large or too small). The object must be placed on the table surface, resting naturally without floating or overlapping with other objects. The object must NOT be in the process of being manipulated by the robot - it must be in a static, resting state on the table.", "color_palette": ["blue"] ], "style": "Ultra-realistic image. The output must NOT be a Canny edge map - it must be a realistic, photorealistic image with proper textures, colors, shading, and lighting. It must look like a real photograph, NOT like an edge map or line drawing.", "lighting": "[ONE lighting sentence from SENTENCE_LIGHT candidates]", "background": "[ONE background sentence from SENTENCE_BACKGROUND candidates]" ... 19 RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning C.4. System Prompts for Video-to-Video Transfer Initial Scene Description Prompt You are given a single image from an indoor tabletop robot manipulation scene. Describe the visible scene in one coherent paragraph from the robot’s first-person perspective. Mention the table or main support surface. Mention the main objects on or near the table, their colors, and approximate positions. Mention the surrounding room context briefly (e.g., kitchen, lab, cabinets). Focus only on what is currently visible; do NOT describe actions, motion, or intent. Use natural English in paragraph form (no bullet points) - INITIAL_SCENE_DESCRIPTION: Build Variation Prompt You are given: 1) An initial scene description of an indoor tabletop manipulation scenario. 2) A reference instruction describing what interaction happens in the scene. Your goal is to produce structured visual variation candidates for: [TABLE], [TARGET_OBJECT_VARIANTS], [SENTENCE_LIGHT], and [SENTENCE_BACKGROUND]. All variations must remain visually plausible for an indoor tabletop setup. IMPORTANT: Generate MULTIPLE diverse candidates for each category. - [TABLE]: Generate 2-3 DIFFERENT table/surface descriptions with VARIED patterns, textures, materials, and finishes. CRITICAL: Each table description must be DISTINCTLY different from others. Include: - Material variety: wood (oak, pine, walnut, bamboo), metal (steel, aluminum), glass, laminate, marble, ... - Pattern variety: solid colors, stripes, checkered, wood grain, marble veins, geometric patterns, ... - Texture variety: smooth, rough, matte, glossy, polished, brushed, textured, grainy, satin finish Examples of DIVERSE table descriptions: - "a dark walnut wooden table with visible grain patterns and a matte finish", - ... AVOID: Similar descriptions (e.g., don’t generate "wooden table" and "oak table" - they’re too similar). - [TARGET_OBJECT_VARIANTS]: Generate 2-3 object variants that STRICTLY PRESERVE the object TYPE and SHAPE from the reference instruction, but vary in COLOR, MATERIAL, and FINISH. CRITICAL: You must identify the target object in the reference instruction and keep its TYPE and SHAPE exactly the same. WHAT TO KEEP (FIXED): - Object Identity (e.g., if it’s a "bottle", it must stay a "bottle") - Object Shape (e.g., if it’s "cylindrical", it must stay "cylindrical") WHAT TO VARY (RANDOMIZED): - Color (e.g., red, blue, transparent, ...), - Material/Finish (e.g., plastic, glass, brushed, ...) CRITICAL - RANDOM COMBINATION FOR EACH VARIANT: - For EACH variant, you MUST RANDOMLY combine: COLOR x MATERIAL/FINISH - Do NOT use predictable patterns. RANDOMLY select one color and one material/finish for each variant. - Example: If original is "potato" (irregular round shape): * Variant 1: potato + red (color) + glossy (finish) = "a round red potato with a glossy finish", * ... IMPORTANT: Each variant must be a single, natural English phrase that includes: [shape/type] + [color] + [material/finish]. DO NOT include size descriptions. - [SENTENCE_LIGHT]: Generate 2-3 DIFFERENT lighting descriptions with varied brightness levels and qualities. CRITICAL: Only describe the general brightness and lighting quality that illuminates the table surface. IMPORTANT: Maximize diversity across brightness levels and qualities: - Brightness levels: bright, dim, soft, harsh, intense, subtle, muted, vibrant, subdued, gentle - Lighting qualities: even illumination, diffused light, natural daylight, warm ambient, ... Examples of DIVERSE lighting: - "bright and even illumination", - ... DO NOT mention any specific light sources, lamps, fixtures, or objects. Only describe the overall brightness and lighting. - [SENTENCE_BACKGROUND]: Generate 2-3 COMPLETELY DIFFERENT background descriptions with varied environments and styles. CRITICAL: Describe ONLY the room environment, walls, and space around/behind the table. IMPORTANT: Maximize diversity across different background types: 1. Room environments: kitchen, lab, workshop, office, living room, garage, studio, classroom, warehouse, storage room 2. Wall styles: smooth white walls, textured beige walls, brick walls, painted walls, concrete walls, ... 3. Distractors on table: fruits/vegetables, tools, containers, utensils, random objects (must be non-intrusive) Examples of HIGHLY DIVERSE backgrounds: - "sterile lab room with smooth white walls and a faint grid pattern on the floor behind the table", - ... CRITICAL CONSTRAINTS: - The TABLE description must remain unchanged. - Distractors must NOT touch or interfere with the primary manipulation objects. INITIAL_SCENE: initial_scene REFERENCE_INSTRUCTION (object identity only): reference_instruction Hard constraints (must follow) - Output MUST contain ALL labeled blocks exactly once. - Each list item MUST start with "- ". - [TARGET_OBJECT_VARIANTS] MUST contain exactly 2-3 complete object descriptions. General rules - Indoor tabletop environment only. - Do NOT invent new actions, motion, or outcomes. - [TARGET_OBJECT_VARIANTS]: STRICT CONSTRAINT: The Object TYPE and SHAPE must match the reference instruction exactly. - [TARGET_OBJECT_VARIANTS] - RANDOM COMBINATION: For each variant, RANDOMLY combine COLOR x MATERIAL/FINISH. - Each variant must be a complete, natural English description. DO NOT include size descriptions. - [TABLE]: Ensure each table description differs significantly in material, pattern, or texture. - [SENTENCE_BACKGROUND]: Ensure each background differs significantly. - [SENTENCE_LIGHT]: RANDOMLY select different brightness levels and lighting qualities. - Focus only on scene elements: table, objects, lighting, background. - Do NOT mention perspective, viewpoint, or robot hands. - Fluent natural English only. - RANDOM SELECTION: All combinations must be RANDOM - no sequential patterns. Return exactly: INITIAL_SCENE_REWRITE: ..., [TABLE]: ..., [TARGET_OBJECT_VARIANTS]: ..., [SENTENCE_LIGHT]: ..., [SENTENCE_BACKGROUND]: ... 20

Instant research discovery