Paper deep dive
KnowDemo: Knowledge-Guided Robot Demonstration Generation from Human Videos
Zhiyuan Gao, Yanxiang Zhan, Mohammad Khoshnazar, Jeroen Schäfer, Michael Beetz
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 88%
Last extracted: 9/21/2026, 4:10:04 AM
Summary
The paper introduces KnowDemo, a framework that generates diverse robot manipulation demonstrations from human videos by extracting structured manipulation knowledge using a Vision-Language Model (VLM). This knowledge, including task conditions and permissible variations, is grounded in a digital twin of the target workspace to guide candidate generation and screening before motion planning. The approach improves behavioral diversity and planning success rates, enabling effective sim-to-real transfer when fine-tuning the pi_0.5 policy model.
Entities (8)
Relation Signals (7)
KnowDemo → generates → robot demonstrations
confidence 95% · KnowDemo, a framework that uses structured manipulation knowledge from human videos to generate diverse robot demonstrations
KnowDemo → uses → Vision-Language Model
confidence 95% · we develop a knowledge extraction and reasoning module based on a vision-language model (VLM)
KnowDemo → enables → Sim-to-real transfer
confidence 90% · achieving sim-to-real transfer across three tasks
$\\pi_{0.5}$ → isfinetunedon → KnowDemo
confidence 90% · fine-tune the pretrained π0.5 model on simulation data
KnowDemo → uses → Isaac Sim
confidence 85% · construct a digital twin ℰR of the target workspace in Isaac Sim
KnowDemo → adoptsmechanismfrom → CRAM
confidence 80% · We adopt CRAM’s designators and resolution mechanism
KnowDemo → uses → cuRobo
confidence 80% · Planning cuRobo [47] receives the joint state
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Learning robot manipulation policies typically requires substantial demonstration data, which are costly to collect on real robots. Recent methods generate robot demonstrations from human videos by adapting recovered motion and validating the resulting trajectories in simulation. However, methods centered on motion-reference adaptation can limit behavioral diversity by retaining the demonstrated contact strategies and subtask orders, while insufficient understanding of task requirements and scene relations can reduce demonstration generation efficiency by generating invalid candidates. To address these limitations, we propose KnowDemo, a framework that uses structured manipulation knowledge from human videos to generate diverse robot demonstrations for a target workspace. To distinguish task requirements from demonstration-specific choices, we develop a knowledge extraction and reasoning module based on a vision-language model (VLM) that associates object and action descriptions with inferred task conditions, demonstration references, and permissible execution variations. To translate this knowledge into executable demonstrations, we resolve the descriptions against target-scene entities and geometry to guide candidate generation and screening before motion planning and simulation. The resulting demonstrations exhibit multimodal behavior through alternative contact strategies and valid subtask orders, with structured execution labels. Experiments demonstrate additional verified execution modes beyond a reference-only configuration and improved candidate planning success through task-guided grasp sampling. To validate the generated data for policy learning, we fine-tune the pretrained $\pi_{0.5}$ model on simulation data, achieving sim-to-real transfer across three tasks. Project page: this https URL
Tags
Links
- Source: https://arxiv.org/abs/2609.21229v1
- Canonical: https://arxiv.org/abs/2609.21229v1
Trouble viewing inline? Open PDF directly →
Full Text
47,531 characters extracted from source content.
Expand or collapse full text
KnowDemo: Knowledge-Guided Robot Demonstration Generation from Human Videos Thanks: Email addresses, in author order: gao@uni-bremen.de, yanxiang@uni-bremen.de, khoshnam@uni-bremen.de, jeroen.schaefer@uni-bremen.de, and beetz@informatik.uni-bremen.de. Zhiyuan Gao Affiliation: University of Bremen Yanxiang Zhan Affiliation: University of Bremen Mohammad Khoshnazar Affiliation: University of Bremen Jeroen Schäfer Affiliation: University of Bremen Michael Beetz Affiliation: University of Bremen Affiliation: Robotics Institute Germany (RIG) Abstract Learning robot manipulation policies typically requires substantial demonstration data, which are costly to collect on real robots. Recent methods generate robot demonstrations from human videos by adapting recovered motion and validating the resulting trajectories in simulation. However, methods centered on motion-reference adaptation can limit behavioral diversity by retaining the demonstrated contact strategies and subtask orders, while insufficient understanding of task requirements and scene relations can reduce demonstration generation efficiency by generating invalid candidates. To address these limitations, we propose KnowDemo, a framework that uses structured manipulation knowledge from human videos to generate diverse robot demonstrations for a target workspace. To distinguish task requirements from demonstration-specific choices, we develop a knowledge extraction and reasoning module based on a vision-language model (VLM) that associates object and action descriptions with inferred task conditions, demonstration references, and permissible execution variations. To translate this knowledge into executable demonstrations, we resolve the descriptions against target-scene entities and geometry to guide candidate generation and screening before motion planning and simulation. The resulting demonstrations exhibit multimodal behavior through alternative contact strategies and valid subtask orders, with structured execution labels. Experiments demonstrate additional verified execution modes beyond a reference-only configuration and improved candidate planning success through task-guided grasp sampling. To validate the generated data for policy learning, we fine-tune the pretrained π0.5 _0.5 model on simulation data, achieving sim-to-real transfer across three tasks. Project page: https://zhiyuan-gao.github.io/knowdemo/. Fig. 1: Overview of KnowDemo. (a) Pick, place, and close frames from a human demonstration, the target workspace, and its digital twin. (b) A drawer-closure example links an inferred task goal to required conditions, the video reference, and allowed alternatives. (c) Colors link alternatives to grounded regions and candidate contacts. Rendered robots and objects are composited onto the real background to form policy observations. (d) Planning and simulation validate candidates, yielding demonstrations with shared task instructions and execution labels for π0.5 _0.5 fine-tuning and real-world deployment, illustrated by banana placement during drawer storage. I Introduction Robot demonstration datasets support learning manipulation policies across tasks and environments [1, 2]. However, real-robot demonstration collection remains costly [3, 4]. Human videos offer accessible task knowledge and motion evidence for generating robot demonstrations [5, 6, 7]. Turning these videos into robot training data requires generating observations and actions for the target robot and its workspace. Video-to-robot approaches extract task plans or recover motion to guide execution [8, 9, 10]. Demonstration-generation methods expand datasets by adapting reference motions to new object poses and geometries [3, 11]. Task and scene knowledge further guide execution by grounding abstract action descriptions in the current environment [12] and incorporating semantic and geometric constraints into motion planning [13, 14, 11]. Simulation subsequently tests the resulting trajectories for physical feasibility [10, 15]. These efforts extend demonstration generation across scene and object variations. However, adapting recovered object motions to new scene configurations can retain the demonstrated contact strategy and subtask order. The resulting data may cover different object configurations while capturing only a limited range of ways to complete the task. A single task can admit multimodal behavior [16, 17, 18]. For example, the person in the video may close a drawer by holding its handle, whereas a robot could achieve the same goal by pushing the front panel. Generating such alternative behaviors requires distinguishing essential task conditions from demonstration-specific choices. Candidate generation must also account for subsequent subtasks and target-scene geometry. A grasp that permits object pickup may obstruct later placement or gripper withdrawal. Ignoring these requirements can lead to failed planning attempts. Grounding task knowledge in the target scene allows these requirements to guide candidate generation before motion planning and simulation. To address these challenges, we propose KnowDemo, a framework that uses structured manipulation knowledge from human videos to generate diverse robot demonstrations for a target workspace (Fig. 1). To distinguish task requirements from demonstration-specific choices, our knowledge extraction and reasoning module based on a vision-language model (VLM) associates structured object and action descriptions with inferred task conditions, observed contacts and motions, and permissible execution variations. To translate this knowledge into executable demonstrations, we resolve the descriptions against target-scene entities and geometry, using the grounded task conditions to guide candidate sampling and screening before motion planning and simulation. Each accepted demonstration retains the task instruction inferred from the video and structured execution labels, allowing data to be organized and selected by execution mode. The resulting simulation demonstrations support policy fine-tuning for deployment in the target workspace. We quantify behavioral diversity by comparing the number of verified execution modes with a reference-only configuration, supported by examples of alternative contact strategies and subtask orders from the same human video. A shared-backend ablation measures the effect of task-guided grasp sampling on candidate planning success. To validate the generated data for robot learning, we fine-tune the pretrained π0.5 _0.5 vision-language-action (VLA) model [19] on the generated simulation demonstrations and demonstrate sim-to-real transfer. In summary, our main contributions are: • To generate robot training data from human videos, we introduce KnowDemo, a framework that uses structured manipulation knowledge to generate diverse demonstrations for a target workspace. • To distinguish task requirements from demonstration-specific choices, we develop a knowledge extraction and reasoning method that associates task conditions, demonstration references, and permissible variations with structured object and action descriptions. To translate this knowledge into executable demonstrations, we ground object and action descriptions in target-scene entities and geometry to guide candidate sampling and screening before motion planning and simulation, retaining structured execution labels with the generated demonstrations. • To evaluate the framework, we quantify gains in verified execution modes over a reference-only configuration, measure the improvement in candidate planning success, and validate sim-to-real transfer using policies fine-tuned on the generated data. I Related Work I-A Manipulation Knowledge from Human Videos Human videos support robot manipulation through two common approaches: training policies that map robot observations to actions, and extracting explicit knowledge representations of goals, object relations, and motion references for planning. In the first approach, MimicPlay learns latent plans from human play to guide a policy trained on robot demonstrations [20], while Vid2Robot conditions actions on demonstration videos and current observations [5]. Explicit representations expose task structure and motion for reasoning and adaptation. SeeDo converts videos into action plans through vision-language reasoning [8]; ORION represents object states, contacts, and motion with Open-World Object Graphs [21]; DITTO transforms object-relative motion for the current scene [9]. DemoDiffusion combines motion references with learned action priors, refining retargeted human motion using a pretrained diffusion policy [22]. Our structured knowledge separates task requirements, demonstration references, and permissible variations to guide target-scene demonstration generation. I-B Knowledge-Guided Robot Demonstration Generation Task and scene knowledge guides contact selection and motion parameters. CRAM represents objects, locations, and actions through designators; perception and reasoning resolve their unspecified properties to contextualize generalized plans [12]. KnowRob combines symbolic knowledge, perception, geometry, and simulation for manipulation reasoning [23]. Spatial constraints and value maps provide geometric guidance for planning [24, 25, 26]. Task-oriented grasping connects contact selection to the intended operation: OVAL-Grasp uses task-relevant part heatmaps to score and filter grasp candidates [27], and DemoFunGrasp conditions grasps on affordance regions and grasping styles [28]. Demonstration generators make these decisions across scene configurations. MimicGen and DemoGen adapt references to new object poses [3, 4]; SkillMimicGen combines skill-level generation with planning [13]. CP-Gen preserves keypoint-trajectory constraints across geometries [11], HybridGen uses VLM segment classification and constraint-guided replanning [14], and AffordGen transfers affordance keypoints across meshes [29]. Using human videos, DexImit, PSI, and Real2Gen combine motion synthesis with simulation [10, 15, 30]. Pegasus combines task, affordance, and constraint graphs with physical verification [31]. We adopt CRAM’s designators and resolution mechanism to link video-derived task conditions, references, and permissible variations to action descriptions. Target-scene resolution specifies alternative contact regions, interaction modes, grasp strategies, and valid subtask orders. Generated demonstrations retain these choices as execution labels under the shared task instruction. I-C Scene Reconstruction for Robot Simulation Video2Policy, Human2Sim2Robot, and X-Sim reconstruct interactive environments for learning from human videos [32, 33, 7]. Articulated-object reconstruction supplies geometry and joints [34]. For visual matching, SIMPLER composites simulated assets onto real backgrounds [35], while SplatSim and RoboSimGS use Gaussian splatting [36, 37]. Our target-workspace digital twin supplies entity references and geometry for knowledge grounding and demonstration generation across tasks. I Method KnowDemo comprises four components: digital-twin construction (Section I-A), manipulation knowledge extraction and reasoning (Section I-B), knowledge-guided execution generation (Section I-C), and simulation validation and policy learning (Section I-D). Figure 1 shows the framework. I-A Problem Setup and Digital-Twin Construction Given a human video V, calibrated target-workspace RGB observations YRY_R (optionally with depth), and a robot model MRM_R, we generate an observation–action dataset DRD_R. Demonstrations satisfy the inferred task requirements while allowing alternatives to the observed contacts and subtask order. We construct a digital twin ℰRE_R of the target workspace in Isaac Sim, using an official Franka model, reconstructed task objects, and an aligned collision table. The table provides physical support and collision geometry but is excluded from the rendered RGB foreground. Policy observations combine robot and object renderings with a fixed-camera workspace image after masking and inpainting movable entities [35]. SAM3 [38] segments images, SAM3D Objects [39] reconstructs rigid objects, and URDF-Anything+ [40] reconstructs articulated task objects, including drawer cabinets and microwave doors, with their part geometry and joints. Assets are scaled using known dimensions or depth, calibrated to the robot base, and converted to USD with joint and physical properties. Collision meshes preserve cavities and contact surfaces. A queryable USD index [41] stores entity IDs, prim paths, poses, meshes, part hierarchies, and joint types, axes, and limits. Rendered masks link IDs to image regions; back-projected part masks and fitted interiors define contact patches and containment volumes for grounding. I-B Manipulation Knowledge Extraction and Reasoning To distinguish task requirements from demonstration-specific choices, our knowledge extraction and reasoning module combines VLM-based parsing of object roles and subtask structure with recovered motion evidence. A pretrained VLM with frozen parameters reasons over this information and target-scene descriptions, using commonsense knowledge to infer task conditions and permissible execution variations. Video parsing The VLM infers a task instruction from the video and identifies subtask boundaries from changes in manipulated objects, contacts, and object relations. Each subtask records its interval, object roles, initial and goal relations, contact/release events, and supporting frames. SAM3.1 [42] tracks object masks; SpatialTracker V2 [43] estimates depth and camera parameters; FoundationPose++ [44] uses these estimates and aligned SAM3D Objects meshes to track object and rigid-part poses. Motion relative to reference object b is Tob(t)=(TbC(t))−1ToC(t)^bT_o(t)=(^CT_b(t))^-1^CT_o(t), where C is the source camera frame and TBA^AT_B maps B coordinates to A. Without known source dimensions, translations are normalized by source extent and scaled to the corresponding target dimension. Motion intervals are linked to their subtasks. Structured knowledge We adopt CRAM’s object, location, and action designators [12]. Object designators describe properties and parts; location designators specify spatial relations and pose requirements; action designators reference these descriptions and contain execution parameters to be resolved in the target scene. The VLM organizes the linked knowledge as =(,,,ℛ,)K=(O,S,C,R,A): O contains object and part descriptions, S contains subtasks and their dependencies, C specifies required conditions, ℛR stores contact and motion references, and A specifies permissible variations. Records link to the relevant designators and subtasks; target-scene resolution instantiates alternatives using available parts, geometry, and execution operators. Conditions specify predicates, role arguments, and whether they apply before, during, or after a subtask; variations identify candidate contact parts and supported interaction modes. Predicates cover containment, relative placement, orientation, joint state, release, and gripper clearance. Observed order is recorded in ℛR, while precedence edges in S are inferred when one subtask produces a state required by another. Video observations carry evidence links and are distinguished from commonsense deductions. Structural checks verify role references, acyclic subtask dependencies, and supported predicates and modes. When ambiguities or inconsistencies arise, the VLM re-examines relevant clips and target-scene views; unresolved critical fields stop generation. I-C Knowledge-Guided Execution Generation in the Target Workspace We use CRAM’s designator resolution mechanism to instantiate descriptions in the digital twin. Our resolution procedures combine VLM-based semantic matching, USD queries, and geometric sampling (Algorithm 1). Fig. 2: Real-robot workspace with a Franka arm, parallel-jaw gripper, and fixed external camera. Entity and region grounding The VLM uses the digital twin’s entity and part descriptions, spatial relations, joint states, geometric queries, and rendered views with ID overlays to match object references to scene entities. Each reference r is bound to an entity and region, β(r)=(eR,ρR)β(r)=(e_R, _R). The scene index verifies entity existence, part membership, and joint compatibility. The location resolver converts spatial relations and pose requirements into sampling conditions; concrete poses are sampled after each layout update. Execution selection Given the inferred task conditions and permissible variations, the VLM uses digital-twin information to select candidate contact parts, interaction modes, and grasp strategies supported by the system. For grasping, it also selects a coarse approach direction compatible with subsequent placement, release, and withdrawal. Each alternative is ξ=(p,m,h)ξ=(p,m,h), with contact part/region p, interaction mode m, and grasp strategy h; surface pushing uses h=∅h= . For grasping alternatives, an associated approach-direction rule constrains geometric grasp sampling. Supported strategies include rim grasping, body grasping, and handle grasping, corresponding to contacts on the rim, outer body, and handle. The available parts determine which strategies apply to an object. The VLM retains multiple task-compatible alternatives. A task-constraint adapter maps modes to grasp-and-transport, placement/release, grasped-articulation, or surface-pushing operators. Each operator provides geometric inputs, gripper states, preconditions/effects, and a waypoint constructor. Task dependencies and operator preconditions determine admissible subtask orders. Typed predicates check preconditions, propagated effects, contact regions, and goal conflicts before planning; continuous conditions remain attached to their subtasks for geometric and physical validation. Geometric sampling and motion construction The adapter obtains contact regions from bound parts and resolves admissible approach directions from their geometry. A grasp strategy may admit multiple approach directions. We represent each admissible approach direction using the approach-cone formulation in CAPGrasp [45]. The adapter resolves a unit direction d in the robot-base frame from gravity, a part frame, or a surface normal. With angular tolerance α, a grasp g must satisfy (g)≥cosα,u(g) Td≥ α, (1) where (g)u(g) is the unit pre-grasp-to-contact direction. The tolerance is specified by the geometric rule for that approach. Top-down and side describe approach directions; rim, body, and handle describe the grasp strategies whose contacts are constrained by the bound regions. After each layout update, the Isaac Sim Grasping SDG antipodal sampler [46] generates concrete grasp poses from object meshes, with opposing surface contacts within the gripper aperture. The adapter retains poses satisfying the selected strategy’s grounded contact-region, approach-direction, and clearance constraints. Rotation about the approach axis remains available. Placement sampling accounts for destination boundaries, object footprint, support height, and orientation. For compatible interaction modes, we express the recovered object motion relative to the task-relevant reference object or part. We map this motion into the corresponding target frame, scale translations to target dimensions, and adapt it to the sampled placement pose. For candidate a, its object waypoints and sampled grasp gag_a determine the end-effector targets: TeWR(t,a)=ToWR(t,a)Teo(ga),^W_RT_e(t;a)=^W_RT_o(t;a)\,^oT_e(g_a), (2) where WRW_R is the robot-base frame and Teo(ga)^oT_e(g_a) is the grasp’s object-relative gripper pose. Alternative interaction modes construct waypoints from sampled contacts, goals, and operator geometry. Each candidate includes release and withdrawal. Drawer example In Fig. 1(b)–(c), the required outcome is a closed drawer with the banana inside. The human’s handle grasp is a reference; resolution yields handle-grasp and panel-push alternatives. Handle-based motion adapts the reference to the target joint geometry. Panel pushing uses the sampled contact, panel normal, joint axis, and drawer displacement. Both follow insertion and gripper withdrawal and satisfy the same containment and joint-state conditions (Fig. 3(a)). Planning cuRobo [47] receives the joint state, collision world, carried-object geometry, and targets. Free-space segments use motion generation; interaction waypoints use sequential inverse kinematics initialized from the preceding solution. Interpolated paths are checked for joint limits, collisions, and orientation/region conditions, with intended contacts specified per subtask. Algorithm 1 Knowledge-guided demonstration generation 0: Knowledge K, indexed digital twin ℰRE_R, attempt budget B 1: Resolve designators to bindings β, sampling conditions, and alternatives ξ=(p,m,h)ξ=(p,m,h) with grasp approach rules 2: Enumerate admissible alternative/order combinations; initialize DR←∅D_R← 3: for i=1i=1 to B do 4: Select the next combination in round-robin order; sample x0x_0 and update grounded geometry 5: if symbolic conditions fail then continue 6: Sample contacts, grasps, and goals under grounded constraints 7: if geometric sampling fails then continue 8: Construct waypoints from compatible references or operator geometry; plan the complete execution 9: if planning fails then continue 10: Reset to x0x_0; execute and monitor subtask conditions 11: Verify execution labels against the rollout 12: if all subtask conditions, final goals, and label checks pass then 13: Store the task instruction, verified labels, observations, and actions in DRD_R 14: end if 15: end for 16: return DRD_R I-D Simulation Validation and Policy Learning Isaac Sim executes joint/gripper commands and checks subtask predicates using relative gripper–object motion, placement stability, joint states, and destination geometry. Acceptance requires all subtask conditions and final goals, including release and withdrawal; failures are logged by subtask. Accepted records synchronize RGB, proprioception, actions, knowledge, and bindings. They retain the shared task instruction and execution labels: target entities, contact parts, interaction modes, grasp strategies, and subtask order. Execution traces and contact checks verify labels initialized from candidate choices. An execution mode is defined by a contact strategy and a valid subtask order. Continuous contact positions and target-plate choices are not counted as separate modes. Labels support selection by execution mode and optional conditioning of policy training. Training actions comprise end-effector pose increments computed from consecutive executed poses and the corresponding binary gripper commands. Policy fine-tuning and real deployment use the same action representation. Section IV-D specifies the learning and deployment settings. IV Experiments We evaluate three aspects of the framework: behavioral diversity from the same human video, candidate planning success through task-guided grasp sampling, and the use of generated data for policy fine-tuning and sim-to-real transfer. (a) Drawer Storage: contact strategy Put the banana in the drawer and close it. Grasp handle and push Push front panel (b) Two-Fruit Placement: subtask order Place the banana and orange on the plate. 1. Place orange 2. Place banana 1. Place banana 2. Place orange Fig. 3: Successful simulation executions under shared task instructions. For each task, alternatives use the same reference video and initial state. (a) Drawer closure by grasping the handle and pushing, or by pushing the front panel. (b) Both fruit-placement orders on the same plate, shown after each placement. IV-A Experimental Setup Platform and inputs We use a Franka arm with a parallel-jaw gripper, a fixed external RGB camera, and the digital twin described in Section I-A (Fig. 2). We run the pretrained Qwen3.8-27B [48] with frozen parameters on a single NVIDIA A100 GPU. Videos are sampled at 2 FPS with timestamps; ambiguous contacts or subtask boundaries trigger VLM re-analysis of more densely sampled clips. Each task uses one reference video, recovered motion evidence, and target assets. Layouts vary object poses and include distractors. Tasks Drawer Storage opens a drawer, inserts a banana, withdraws the gripper from the interior, and closes the drawer with the banana inside; closure permits handle grasping or panel pushing. Two-Fruit Placement places an orange and a banana on a plate in either order, leaving both stable within their placement regions. Close Microwave requires full door closure and gripper withdrawal, allowing different pushing contacts. Real-policy evaluation uses these three tasks, with Banana Drawer Storage starting from an already open drawer. Grasp-sampling evaluation uses the two tasks in Fig. 4: Cup-in-Drawer moves a cup from the cabinet top into an open drawer, and Cup-to-Cabinet moves a cup from beneath an extended drawer onto the cabinet top. Both require stability after release and gripper withdrawal. Success criteria are fixed independently of inferred knowledge. Fig. 4: Human grasp and placement references with corresponding robot grasps in simulation: (a) cabinet top to drawer, using a top-down robot approach; (b) beneath an extended drawer to cabinet top, pairing a palm-down human grasp with a side approach by the robot. Robot grasps accommodate placement and withdrawal in (a), and access beneath the drawer in (b). Generation backend The backend combines video-derived motion references, geometric grasp sampling, and planning, following the general workflow of Real2Gen [30], with task adapters for articulated motion and execution of multiple subtasks. The grasp-sampling comparison changes grasp-strategy selection within this shared backend, as detailed below. IV-B Behavioral Diversity of Generated Demonstrations Table I reports execution modes verified by complete successful demonstrations. Reference-only retains the demonstrated contact strategy and subtask order using the same backend for target-scene adaptation. Modes follow the definition in Section I-D. TABLE I: Verified execution modes. Task Reference-only KnowDemo Drawer Storage 1 2 Two-Fruit Placement 1 2 Close Microwave∗ 1 1 ∗Close Microwave: both use one pushing mode; KnowDemo varies contact positions within that mode. Figure 3 shows successful alternatives from matched initial states. Drawer Storage uses handle grasping or panel pushing for closure after banana insertion and gripper withdrawal, preserving containment and the final closed state (Fig. 3(a)). For Two-Fruit Placement, KnowDemo varies placement order and, when task conditions permit, target plate; both are retained as execution labels. Figure 3(b) shows banana-first and orange-first sequences using the same plate, with frames after each placement. For Close Microwave, the generated demonstrations use different door contact positions within the same pushing mode, while preserving the task instruction and requiring no additional human videos. TABLE I: Grasp-sampling ablation. Task Without task guidance KnowDemo Cup-in-Drawer 29/50 50/50 Cup-to-Cabinet 23/50 45/50 Entries report successful complete-task plans / sampled candidates. Banana Drawer Storage Simulation: initial Real: initial Real: intermediate Real: final Close Microwave Simulation: initial Real: initial Real: intermediate Real: final Two-Fruit Placement Simulation: initial Real: initial Real: intermediate Real: final Fig. 5: Simulation observations and real-robot executions. For each task, the upper-left image combines rendered robots and task objects with a real background. The other three images show initial, intermediate, and final frames of autonomous real-robot executions using π0.5 _0.5 fine-tuned on generated simulation demonstrations. IV-C Task-Guided Grasp Sampling Human grasp strategies need not transfer directly to a robot gripper. Figure 4 pairs grasp and placement frames from each reference video. In (a), the cup moves from the cabinet top into a drawer, where placement and gripper withdrawal constrain grasp choice. The human reorients the cup within the hand before insertion, so the grasp and placement frames show different hand–object configurations. In (b), the human uses a palm-down grasp to move the cup from beneath an extended drawer to the cabinet top; the overhang constrains robot approach and extraction. The framework preserves the task goals while selecting robot contact regions and approach directions from target geometry. Planning and simulation check the complete execution. Task-oriented grasping connects contact selection to operation requirements [27]. Our quantitative ablation uses Cup-in-Drawer and Cup-to-Cabinet to measure whether task-guided strategy selection increases complete-task planning success under a fixed budget. Comparison We isolate task-guided grasp selection within the same generation backend. Without task guidance, a grasp strategy and its associated approach rule are sampled from the object’s applicable set independently of task-specific approach and placement requirements. KnowDemo selects strategies using task requirements and target-scene information. For the handleless cups, both configurations use the rim- and body-grasp strategies from Section I-C. The adapter maps each strategy to contact-region and approach-direction constraints. KnowDemo selects a top-down approach for Cup-in-Drawer to accommodate placement and withdrawal inside the drawer, and a side approach for Cup-to-Cabinet to accommodate access beneath the extended drawer. Both configurations use Grasping SDG with the same scene assets, grounded geometry, placement goals, execution operators, and per-candidate planning budget; task conditions and geometric checks remain active during planning and simulation validation. Initial positions vary within a small neighborhood of each setup, with paired states across methods. Each method submits 50 candidates per task; a plan must cover grasp, transfer, placement, and withdrawal. Results Table I reports complete-task plans out of 50 candidates per task and method, evaluated before simulation execution. Task-guided grasp selection improves planning success from 58% to 100% for Cup-in-Drawer and from 46% to 90% for Cup-to-Cabinet. Failed candidates include grasps that permit pickup but prevent the planner from finding a collision-free trajectory for subsequent placement or gripper withdrawal. Task-guided selection accounts for these downstream requirements when choosing the grasp strategy. TABLE I: Real-robot policy success rates. Task Simulation demos Successes/ trials Success rate (%) Banana Drawer Storage 200 10/20 50 Close Microwave 100 14/20 70 Two-Fruit Placement 200 11/20 55 IV-D Policy Learning and Real-Robot Evaluation We fine-tune the pretrained π0.5 _0.5 vision-language-action model [19] for each task evaluated on the real robot, using only generated simulation demonstrations. Policies receive RGB images, proprioceptive states, and the shared task instructions. Execution labels are retained as dataset metadata. We perform full-parameter fine-tuning with openpi for 15,000 steps, using a batch size of 256 and a learning rate of 5×10−55× 10^-5. The action chunk length is 10, with all 10 actions executed before the next query. Table I reports dataset sizes and complete-task success over 20 real trials per task with layout variations and no human intervention. Success rates of 50–70% demonstrate sim-to-real transfer for door pushing, sequential fruit placement, and object placement followed by drawer closure. Figure 5 pairs simulation initial states with autonomous real executions. The learned policies also exhibit behavioral diversity: the same Two-Fruit Placement policy successfully uses both placement orders, and Close Microwave executions use different door contact positions. Failures occur when target objects are occluded or when the microwave and drawer cabinet positions differ substantially from those represented in the generated demonstrations. These observations indicate sensitivity to occlusion and spatial layout changes in the deployed policies. V Limitations The framework depends on the quality of inferred knowledge and the target scene model. Task ambiguity, occlusion, and motion-recovery errors can lead to incorrect task conditions, excluding valid alternatives or admitting inappropriate ones. Geometric and physical validation checks execution against the inferred conditions but cannot establish whether those conditions capture the intended task. Errors in target geometry, articulation, and contact parameters, together with visual and dynamic differences between simulation and hardware, can degrade generated demonstrations and real-world execution. Extending the framework to new tasks requires compatible assets, support for the required interactions, and policy fine-tuning. Multiple demonstrations and real execution feedback could improve condition inference and scene-model accuracy. VI Conclusion We presented KnowDemo, a framework that uses structured manipulation knowledge from human videos to guide robot demonstration generation in a target workspace. To expand behavioral diversity while preserving task requirements, our VLM-based knowledge extraction and reasoning module distinguishes required conditions, demonstration references, and permissible variations within CRAM action descriptions. To reduce invalid generation attempts, we ground these descriptions in target-scene entities and geometry to guide candidate sampling and screening before planning and simulation. The resulting demonstrations share the inferred task instruction and retain structured execution labels. Experiments demonstrate additional verified execution modes beyond a reference-only configuration, improved candidate planning success through task-guided grasp sampling, and sim-to-real transfer using π0.5 _0.5 fine-tuned on generated demonstrations. References [1] Open X-Embodiment Collaboration (2024) Open X-Embodiment: robotic learning datasets and RT-X models. In Proc. IEEE Int. Conf. Robot. Autom. (ICRA), p. 6892–6903. External Links: Document Cited by: §I. [2] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, P. D. Fagan, J. Hejna, M. Itkina, M. Lepert, Y. J. Ma, P. T. Miller, J. Wu, S. Belkhale, S. Dass, H. Ha, A. Jain, A. Lee, Y. Lee, M. Memmel, S. Park, I. Radosavovic, K. Wang, A. Zhan, K. Black, C. Chi, K. B. Hatch, S. Lin, J. Lu, J. Mercat, A. Rehman, P. R. Sanketi, A. Sharma, C. Simpson, Q. Vuong, H. R. Walke, B. Wulfe, T. Xiao, J. H. Yang, A. Yavary, T. Z. Zhao, C. Agia, R. Baijal, M. G. Castro, D. Chen, Q. Chen, T. Chung, J. Drake, E. P. Foster, J. Gao, V. Guizilini, D. A. Herrera, M. Heo, K. Hsu, J. Hu, M. Z. Irshad, D. Jackson, C. Le, Y. Li, K. Lin, R. Lin, Z. Ma, A. Maddukuri, S. Mirchandani, D. Morton, T. Nguyen, A. O’Neill, R. Scalise, D. Seale, V. Son, S. Tian, E. Tran, A. E. Wang, Y. Wu, A. Xie, J. Yang, P. Yin, Y. Zhang, O. Bastani, G. Berseth, J. Bohg, K. Goldberg, A. Gupta, A. Gupta, D. Jayaraman, J. J. Lim, J. Malik, R. Martín-Martín, S. Ramamoorthy, D. Sadigh, S. Song, J. Wu, M. C. Yip, Y. Zhu, T. Kollar, S. Levine, and C. Finn (2024) DROID: a large-scale in-the-wild robot manipulation dataset. In Proc. Robot.: Sci. Syst. (RSS), Cited by: §I. [3] A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y. Narang, L. Fan, Y. Zhu, and D. Fox (2023) MimicGen: a data generation system for scalable robot learning using human demonstrations. In Proc. Conf. Robot Learn. (CoRL), Proc. Mach. Learn. Res., Vol. 229, p. 1820–1864. Cited by: §I, §I-B. [4] Z. Xue, S. Deng, Z. Chen, Y. Wang, Z. Yuan, and H. Xu (2025) DemoGen: synthetic demonstration generation for data-efficient visuomotor policy learning. In Proc. Robot.: Sci. Syst. (RSS), External Links: Document Cited by: §I, §I-B. [5] V. Jain, M. Attarian, N. J. Joshi, A. Wahid, D. Driess, Q. Vuong, P. R. Sanketi, P. Sermanet, S. Welker, C. Chan, I. Gilitschenski, Y. Bisk, and D. Dwibedi (2024) Vid2Robot: end-to-end video-conditioned policy learning with cross-attention transformers. Note: arXiv:2403.12943 External Links: Document Cited by: §I, §I-A. [6] J. Shi, Z. Zhao, T. Wang, I. Pedroza, A. Luo, J. Wang, J. Ma, and D. Jayaraman (2025) ZeroMimic: distilling robotic manipulation skills from web videos. In Proc. IEEE Int. Conf. Robot. Autom. (ICRA), External Links: Link Cited by: §I. [7] P. Dan, K. Kedia, A. Chao, E. W. Duan, M. A. Pace, W. Ma, and S. Choudhury (2025) X-Sim: cross-embodiment learning via real-to-sim-to-real. Note: arXiv:2505.07096 External Links: Document Cited by: §I, §I-C. [8] B. Wang, J. Zhang, S. Dong, I. Fang, and C. Feng (2024) VLM see, robot do: human demo video to robot action plan via vision language model. Note: arXiv:2410.08792 External Links: Document Cited by: §I, §I-A. [9] N. Heppert, M. Argus, T. Welschehold, T. Brox, and A. Valada (2024) DITTO: demonstration imitation by trajectory transformation. In Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), p. 7565–7572. External Links: Document Cited by: §I, §I-A. [10] J. Mu, S. Yang, Y. Bao, H. Bae, T. Wei, L. Xu, B. Li, H. Xu, and J. Pang (2026) DexImit: learning bimanual dexterous manipulation from monocular human videos. Note: arXiv:2602.10105 External Links: Document Cited by: §I, §I-B. [11] K. Lin, V. Ragunath, A. McAlinden, A. Prasad, J. Wu, Y. Zhu, and J. Bohg (2025) Constraint-preserving data generation for one-shot visuomotor policy generalization. In Proc. Conf. Robot Learn. (CoRL), Proc. Mach. Learn. Res., Vol. 305, p. 3631–3646. Cited by: §I, §I-B. [12] M. Beetz, G. Kazhoyan, and D. Vernon (2025) Robot manipulation in everyday activities with the CRAM 2.0 cognitive architecture and generalized action plans. Cogn. Syst. Res. 92, p. 101375. External Links: Document Cited by: §I, §I-B, §I-B. [13] C. R. Garrett, A. Mandlekar, B. Wen, and D. Fox (2025) SkillMimicGen: automated demonstration generation for efficient skill learning and deployment. In Proc. Conf. Robot Learn. (CoRL), Proc. Mach. Learn. Res., Vol. 270, p. 2750–2790. Cited by: §I, §I-B. [14] W. Wang and N. Tan (2025) HybridGen: VLM-guided hybrid planning for scalable data generation of imitation learning. Note: arXiv:2503.13171 External Links: Document Cited by: §I, §I-B. [15] A. J. Zhai, K. Zeng, J. Lu, A. Farhadi, S. Wang, and W. Ma (2026) Imitating what works: simulation-filtered modular policy learning from human videos. Trans. Mach. Learn. Res.. External Links: Link Cited by: §I, §I-B. [16] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song (2023) Diffusion policy: visuomotor policy learning via action diffusion. In Proc. Robot.: Sci. Syst. (RSS), External Links: Document Cited by: §I. [17] N. M. M. Shafiullah, Z. J. Cui, A. Altanzaya, and L. Pinto (2022) Behavior transformers: cloning k modes with one stone. In Adv. Neural Inf. Process. Syst., Vol. 35. Cited by: §I. [18] X. Jia, D. Blessing, X. Jiang, M. Reuss, A. Donat, R. Lioutikov, and G. Neumann (2024) Towards diverse behaviors: a benchmark for imitation learning with human demonstrations. In Proc. Int. Conf. Learn. Represent. (ICLR), Cited by: §I. [19] Physical Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, M. Y. Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke, A. Walling, H. Wang, L. Yu, and U. Zhilinsky (2025) π0.5 _0.5: a vision-language-action model with open-world generalization. Note: arXiv:2504.16054 Cited by: §I, §IV-D. [20] C. Wang, L. Fan, J. Sun, R. Zhang, L. Fei-Fei, D. Xu, Y. Zhu, and A. Anandkumar (2023) MimicPlay: long-horizon imitation learning by watching human play. In Proc. Conf. Robot Learn. (CoRL), Proc. Mach. Learn. Res., Vol. 229, p. 201–221. Cited by: §I-A. [21] Y. Zhu, A. Lim, P. Stone, and Y. Zhu (2026) Vision-based manipulation from single human video with open-world object graphs. Auton. Robots 50. Note: Art. no. 27 External Links: Document Cited by: §I-A. [22] S. Park, H. Bharadhwaj, and S. Tulsiani (2026) DemoDiffusion: one-shot human imitation using pre-trained diffusion policy. In Proc. IEEE Int. Conf. Robot. Autom. (ICRA), Cited by: §I-A. [23] M. Beetz, D. Beßler, A. Haidu, M. Pomarlan, A. K. Bozcuoğlu, and G. Bartels (2018) KnowRob 2.0—a 2nd generation knowledge processing framework for cognition-enabled robotic agents. In Proc. IEEE Int. Conf. Robot. Autom. (ICRA), p. 512–519. External Links: Document Cited by: §I-B. [24] L. Manuelli, W. Gao, P. Florence, and R. Tedrake (2019) kPAM: KeyPoint affordances for category-level robotic manipulation. Note: arXiv:1903.06684 External Links: Document Cited by: §I-B. [25] W. Huang, C. Wang, Y. Li, R. Zhang, and L. Fei-Fei (2025) ReKep: spatio-temporal reasoning of relational keypoint constraints for robotic manipulation. In Proc. Conf. Robot Learn. (CoRL), Proc. Mach. Learn. Res., Vol. 270, p. 4573–4602. Cited by: §I-B. [26] W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei (2023) VoxPoser: composable 3D value maps for robotic manipulation with language models. In Proc. Conf. Robot Learn. (CoRL), Proc. Mach. Learn. Res., Vol. 229, p. 540–562. Cited by: §I-B. [27] E. Tong, A. Balaji, A. Opipari, S. Lewis, Z. Zeng, and O. C. Jenkins (2025) OVAL-Grasp: open-vocabulary affordance localization for task oriented grasping. In Proc. Int. Symp. Exp. Robot. (ISER), External Links: Link Cited by: §I-B, §IV-C. [28] C. Mao, H. Yuan, Z. Huang, C. Xu, K. Ma, and Z. Lu (2026) DemoFunGrasp: universal dexterous functional grasping via demonstration-editing reinforcement learning. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), p. 986–995. Cited by: §I-B. [29] J. Zhang, K. Hu, Y. Huang, Y. Ju, Z. Xue, and H. Xu (2026) AffordGen: generating diverse demonstrations for generalizable object manipulation with affordance correspondence. Note: arXiv:2604.10579 External Links: Document Cited by: §I-B. [30] N. Heppert, M. Q. Nguyen, and A. Valada (2026) Scaling single human demonstrations for imitation learning using generative foundational models. In Proc. IEEE Int. Conf. Robot. Autom. (ICRA), Cited by: §I-B, §IV-A. [31] J. Luo (2026) From passive video to editable experience: physically grounded experience synthesis for embodied intelligence. Note: arXiv:2607.26903 External Links: Document Cited by: §I-B. [32] W. Ye, F. Liu, Z. Ding, Y. Gao, O. Rybkin, and P. Abbeel (2025) Video2Policy: scaling up manipulation tasks in simulation through internet videos. Note: arXiv:2502.09886 External Links: Document Cited by: §I-C. [33] T. G. W. Lum, O. Y. Lee, C. K. Liu, and J. Bohg (2025) Crossing the human-robot embodiment gap with sim-to-real RL using one human demonstration. Note: arXiv:2504.12609 External Links: Document Cited by: §I-C. [34] Y. Weng, B. Wen, J. Tremblay, V. Blukis, D. Fox, L. Guibas, and S. Birchfield (2024) Neural implicit representation for building digital twins of unknown articulated objects. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), p. 3141–3150. Cited by: §I-C. [35] X. Li, K. Hsu, J. Gu, K. Pertsch, O. Mees, H. R. Walke, C. Fu, I. Lunawat, I. Sieh, S. Kirmani, S. Levine, J. Wu, C. Finn, H. Su, Q. Vuong, and T. Xiao (2024) Evaluating real-world robot manipulation policies in simulation. Note: arXiv:2405.05941 External Links: Document Cited by: §I-C, §I-A. [36] M. N. Qureshi, S. Garg, F. Yandun, D. Held, G. Kantor, and A. Silwal (2025) SplatSim: zero-shot Sim2Real transfer of RGB manipulation policies using gaussian splatting. In Proc. IEEE Int. Conf. Robot. Autom. (ICRA), p. 6502–6509. External Links: Document Cited by: §I-C. [37] H. Zhao, C. Zeng, L. Zhuang, Y. Zhao, S. Xue, H. Wang, X. Zhao, Z. Li, K. Li, S. Huang, M. Chen, X. Li, D. Zhao, and H. Zou (2025) High-fidelity simulated data generation for real-world zero-shot robotic manipulation learning with gaussian splatting. Note: arXiv:2510.10637 Cited by: §I-C. [38] N. Carion, L. Gustafson, Y. Hu, S. Debnath, R. Hu, D. Suris, C. Ryali, K. V. Alwala, H. Khedr, A. Huang, J. Lei, T. Ma, B. Guo, A. Kalla, M. Marks, J. Greer, M. Wang, P. Sun, R. Rädle, T. Afouras, E. Mavroudi, K. Xu, T. Wu, Y. Zhou, L. Momeni, R. Hazra, S. Ding, S. Vaze, F. Porcher, F. Li, S. Li, A. Kamath, H. K. Cheng, P. Dollár, N. Ravi, K. Saenko, P. Zhang, and C. Feichtenhofer (2025) SAM 3: segment anything with concepts. Note: arXiv:2511.16719 Cited by: §I-A. [39] SAM 3D Team, X. Chen, F. Chu, P. Gleize, K. J. Liang, A. Sax, H. Tang, W. Wang, M. Guo, T. Hardin, X. Li, A. Lin, J. Liu, Z. Ma, A. Sagar, B. Song, X. Wang, J. Yang, B. Zhang, P. Dollár, G. Gkioxari, M. Feiszli, and J. Malik (2025) SAM 3D: 3Dfy anything in images. Note: arXiv:2511.16624 Cited by: §I-A. [40] Z. Wu, Y. Xin, C. Hou, M. Chen, Y. Lyu, J. Zhang, and S. Zhang (2026) URDF-Anything+: end-to-end generation for simulation-ready articulated assets. Note: arXiv:2603.14010 Cited by: §I-A. [41] G. H. Nguyen, D. Beßler, S. Stelter, M. Pomarlan, and M. Beetz (2024) Translating universal scene descriptions into knowledge graphs for robotic environment. In Proc. IEEE Int. Conf. Robot. Autom. (ICRA), p. 9389–9395. External Links: Document Cited by: §I-A. [42] SAM 3 Team (2026) SAM 3.1: release notes. Note: GitHubAccessed: Sep. 14, 2026 External Links: Link Cited by: §I-B. [43] Y. Xiao, J. Wang, N. Xue, N. Karaev, Y. Makarov, B. Kang, X. Zhu, H. Bao, Y. Shen, and X. Zhou (2025) SpatialTrackerV2: advancing 3D point tracking with explicit camera motion. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), p. 6726–6737. Cited by: §I-B. [44] teal024 (2025) FoundationPose++: simple tricks boost FoundationPose performance in high-dynamic scenes. Note: GitHubAccessed: Sep. 14, 2026 External Links: Link Cited by: §I-B. [45] Z. Weng, H. Lu, J. Lundell, and D. Kragic (2024) CAPGrasp: an ℝ3×SO(2)R^3×SO(2)-equivariant continuous approach-constrained generative grasp sampler. IEEE Robot. Autom. Lett. 9 (4), p. 3641–3647. External Links: Document Cited by: §I-C. [46] NVIDIA Grasping synthetic data generation. Note: Isaac Sim 5.1 DocumentationAccessed: Sep. 15, 2026 External Links: Link Cited by: §I-C. [47] B. Sundaralingam, S. K. S. Hari, A. Fishman, C. Garrett, K. Van Wyk, V. Blukis, A. Millane, H. Oleynikova, A. Handa, F. Ramos, N. Ratliff, and D. Fox (2023) CuRobo: parallelized collision-free robot motion generation. In Proc. IEEE Int. Conf. Robot. Autom. (ICRA), p. 8112–8119. External Links: Document Cited by: §I-C. [48] Qwen Team (2026) Qwen3.8-27B. Note: Model card. Accessed: Sep. 14, 2026Hugging Face External Links: Link Cited by: §IV-A.