Paper deep dive
HIGenNTO: Scalable Humanoid Interaction Generation via Noise-Space Trajectory Optimization
Lalit Jayanti, Kashu Yamazaki, Yuto Shibata, Kotaro Amaya, Katerina Fragkiadaki
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/22/2026, 5:00:07 AM
Summary
The paper introduces HIGenNTO, a framework for generating scalable, contact-rich humanoid interaction motions by optimizing the initial noise of a pretrained text-conditioned motion diffusion model under sparse spatiotemporal and scene constraints. This approach synthesizes physically executable motion references that avoid collisions and maintain stable support, which are then used to train depth-conditioned visuomotor policies. These policies are successfully deployed on a Unitree G1 robot. Additionally, the paper demonstrates that a coding agent can automate the creation of task specifications, further scaling the system.
Entities (13)
Relation Signals (12)
Lalit Jayanti → affiliatedwith → Carnegie Mellon University
confidence 95% · Lalit Jayanti1 ... 1Carnegie Mellon University
Kotaro Amaya → affiliatedwith → Keio University
confidence 95% · Kotaro Amaya2,3 ... 3Keio University
Yuto Shibata → affiliatedwith → Keio University
confidence 95% · Yuto Shibata2,3 ... 3Keio University
Katerina Fragkiadaki → affiliatedwith → Carnegie Mellon University
confidence 95% · Katerina Fragkiadaki1 1Carnegie Mellon University
HIGenNTO → deployson → Unitree G1
confidence 95% · We deploy these policies on a Unitree G1 across four contact-rich tasks.
HIGenNTO → uses → Noise-Space Trajectory Optimization
confidence 95% · We present HIGenNTO, a framework that synthesizes humanoid-scene interaction motion references by optimizing the initial noise of a pretrained text-conditioned motion model
HIGenNTO → developedby → Keio University
confidence 90% · Yuto Shibata2,3, Kotaro Amaya2,3 ... 3Keio University
HIGenNTO → developedby → Carnegie Mellon University
confidence 90% · Lalit Jayanti1, Kashu Yamazaki1, Katerina Fragkiadaki1 1Carnegie Mellon University
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Humanoid robots can acquire complex skills by imitating kinematic humanoid motion references, yet reliable references for contact-rich interactions remain difficult to obtain: motion capture deteriorates under occlusion and close physical contact, while retargeting introduces additional contact and geometric inconsistencies. We present HIGenNTO, a framework that synthesizes humanoid-scene interaction motion references by optimizing the initial noise of a pretrained text-conditioned motion model under sparse spatiotemporal and scene constraints. The same formulation satisfies desired contacts, avoids collisions, and maintains stable support while retaining the prior's realism and temporal coherence, generating interaction motions from scratch and composing long-horizon behaviors stage-wise. Across robot-environment and robot-object tasks, HIGenNTO produces motions that can be executed by tracking policies in simulation and used to train depth-conditioned visuomotor policies operating solely from onboard sensing. We deploy these policies on a Unitree G1 across four contact-rich tasks. Finally, the task specifications themselves can be written by a coding agent, which proposes interaction tasks and compiles them into prompt, constraint, and scene programs, authoring three of our eight evaluated tasks and four further behaviors. Together, these results establish a scalable path from high-level task descriptions to physically executable humanoid interactions.
Tags
Links
- Source: https://arxiv.org/abs/2609.22611v1
- Canonical: https://arxiv.org/abs/2609.22611v1
Trouble viewing inline? Open PDF directly →
Full Text
75,134 characters extracted from source content.
Expand or collapse full text
HIGenNTO: Scalable Humanoid Interaction Generation via Noise-Space Trajectory Optimization Lalit Jayanti1, Kashu Yamazaki1, Yuto Shibata2,3, Kotaro Amaya2,3, Katerina Fragkiadaki1 1Carnegie Mellon University 2Keio AI Research Center 3Keio University Humanoid robots can acquire complex skills by imitating kinematic humanoid motion references, yet reliable references for contact-rich interactions remain difficult to obtain: motion capture deteriorates under occlusion and close physical contact, while retargeting introduces additional contact and geometric inconsistencies. We present HIGenNTO, a framework that synthesizes humanoid–scene interaction motion references by optimizing the initial noise of a pretrained text-conditioned motion model under sparse spatiotemporal and scene constraints. The same formulation satisfies desired contacts, avoids collisions, and maintains stable support while retaining the prior’s realism and temporal coherence, generating interaction motions from scratch and composing long-horizon behaviors stage-wise. Across robot–environment and robot–object tasks, HIGenNTO produces motions that can be executed by tracking policies in simulation and used to train depth-conditioned visuomotor policies operating solely from onboard sensing. We deploy these policies on a Unitree G1 across four contact-rich tasks. Finally, the task specifications themselves can be written by a coding agent, which proposes interaction tasks and compiles them into prompt, constraint, and scene programs, authoring three of our eight evaluated tasks and four further behaviors. Together, these results establish a scalable path from high-level task descriptions to physically executable humanoid interactions. Website: https://higennto.github.io 1 Introduction Figure 1: HIGenNTO synthesizes humanoid–scene interaction motion references through constrained optimization over a generative motion prior. Stair traversal, sitting and standing, and box pickup and placement across randomized scenes, without any motion capture of joint human–scene interaction. Humanoid robots have acquired remarkably complex skills by imitating kinematic 3D trajectories recovered from motion capture or monocular video [28, 23, 6], and inherit the limitations of that reference data. Most large-scale datasets record the body in isolation, omitting the scene geometry, object motion, and contacts that constitute interaction [24, 12]. Interaction is also the hardest case to capture: occlusion and close contact degrade pose estimation where precision matters most, so even datasets built for it [47, 19] contain missed contacts, interpenetrations, and temporal inconsistencies, which retargeting compounds. We introduce HIGenNTO (Humanoid Interaction Generation via Noise-Space Trajectory Optimization), which synthesizes contact-rich references without interaction motion capture (Figure 1). We cast interaction synthesis as constrained optimization over a generative motion prior: HIGenNTO optimizes the initial noise of a pretrained text-conditioned motion model subject to sparse spatiotemporal constraints and differentiable scene objectives that specify what text and keyframes cannot: which body part contacts which object, where and when, and which regions to keep clear. Gradients traverse the whole sampling chain, so the trajectory adapts to contact, collision, support, and task geometry while staying on the prior’s learned manifold. A single formulation yields motions from a task specification and scene geometry alone, and can extend past the pretrained motion prior’s prediction horizon. Optimizing diffusion noise has previously enabled motion editing and control of human characters, including navigation in a static scene [17, 53]. We extend this mechanism to humanoid interaction synthesis, where contacts must not only be avoided but also deliberately established and maintained with support surfaces, manipulated objects, and other agents. The specification for generating motions itself can be automated: from a short brief and the corpus annotations, a coding agent sizes the scene and writes the program to generate the spatiotemporal constraints. Further, these generated references can be executed by tracking policies and used to train depth-conditioned visuomotor policies operating solely from onboard sensing. Contributions. (1) Interaction synthesis as optimization in noise space. We make a pretrained, scene-unaware motion prior contact- and geometry-aware without retraining it, achieving lower scene penetration across all three interaction categories and lower object-task hand-target error than both kinematic-constraint conditioning and classifier guidance. (2) References that physical policies execute. Across eight tasks, scene-aware teachers track the generated motion and distill into depth-only students that complete 57.457.4–98.3%98.3\% of motions, where methods based on scene-unaware tracking formulations [23] largely fail. (3) Task specification by a coding agent. Agent-written programs supply three of our eight evaluated tasks and four further behaviors, with no per-task demonstration and no hand-tuned constraints. 2 Related Work Human Motion Data and Human-to-Humanoid Retargeting. Large-scale motion datasets [24, 12, 13] underpin humanoid motion imitation and control [28, 30, 7], with collection scaled by video reconstruction [20, 1]. Most model humans in isolation, and those that do model interactions with the scene [43, 41, 14, 52, 47, 42, 25] still yield penetrations, contact inconsistencies, and implausible artifacts. Retargeting these motions to robots is an effective route to agile behaviors [28, 3, 34], via inverse kinematics, trajectory optimization, and learned retargeting [9, 11, 10, 3], including interaction-aware objectives [50, 45, 49, 37, 8, 15], but all depend on high-quality interaction motion that remains hard to obtain at scale. We instead synthesize such references from generative motion priors under sparse, semantically grounded constraints. Generative Motion Priors for Humanoid Control. Diffusion-based motion models generate realistic, controllable motion conditioned on text, goals, or sparse constraints [39, 32, 35], and several works pair them with RL tracking controllers for physical realism at execution [38, 34, 48]. Adversarial motion priors such as AMP [31] and ASE [29] instead regularize policies toward realistic motion distributions without explicit trajectory tracking. Controllable and Constraint-Guided Motion Generation. Diffusion models achieve spatial and semantic control through training-time conditioning on paired control signals such as trajectories or sparse constraints [39, 35, 46, 18, 32], through inference-time guidance by gradients of rewards, constraints, or value functions during denoising [16, 44, 54], or by optimizing the initial noise itself, backpropagating a differentiable objective through the entire sampling chain. We adopt the last: DNO [17] establishes it for editing, refinement, and obstacle avoidance, and DartControl [53] applies it to human–scene contact. Both operate on kinematic human characters against static geometry; we target humanoid robots, add objectives for contact held with movable objects and other agents, and train physical policies on the resulting references. 3 Method Figure 2: System overview. (A) Task specification: a text prompt, sparse spatiotemporal constraints on end-effector and root goals, and a scene representation (SDF / height-field). (B) Noise-space optimization: differentiable scene and contact losses (goal, collision, foot, edge) are backpropagated through the denoising chain of a frozen motion prior into its input latent noise, yielding a scene-aware interaction dataset. (C) Long-horizon generation: temporal windows optimized in sequence, with key poses transferred across boundaries. (D) Motion repair: an inconsistent retargeted motion sequence is inverted to latent noise and re-optimized. (E) Deployment: the dataset trains a PPO tracking teacher, distilled into a depth- and proprioception-conditioned student. Training a scene-aware interaction generator directly would require physically consistent interaction paired with time-varying scene geometry, which is hard to collect at scale. We instead inject scene-awareness at optimization time, optimizing the latent noise of a frozen text-conditioned motion diffusion model under differentiable contact, and scene objectives (Figure 2). The synthesized motions stand in for motion-capture interaction data [40, 42], supervising a privileged tracking teacher that is distilled into a policy running from onboard sensing alone. 3.1 Scene-Aware Interaction Synthesis via Constrained Latent Optimization We consider an articulated humanoid acting in a scene. A task is specified by: (i) a text prompt y; (i) a set of sparse spatiotemporal constraints =(ji,ti,pi)i=1NcC=\(j_i,t_i,p_i)\_i=1^N_c that constrain keypoint jij_i to a target 3D position pi∈ℝ3p_i ^3 at time tit_i; and (i) a scene representation S given by differentiable signed-distance functions (SDFs) for static and dynamic geometry, manipulated objects, and terrain height fields. Constraints are authored per task (by hand or, as we show in Sec. 4.3, by a coding agent), except for repair, where contact candidates proposed from the reference are filtered by a vision-language model (Appendix D). The goal is a motion trajectory ∈ℝT×J×3x ^T× J× 3 that is semantically consistent with y, satisfies C, respects scene geometry, and remains temporally smooth. Scene-Unaware Humanoid Motion Prior. We use a pretrained text-conditioned humanoid motion diffusion model [32] as a differentiable motion prior, Mθ(∣y,)=M_θ(z y,C)=x, mapping latent noise z, a text prompt y, and optional sparse kinematic constraints C to a trajectory x. It is trained on over 700 hours of human motion retargeted to humanoid embodiments, without explicit scene or interaction input. The prior can be conditioned on kinematic constraints (full-body keyframes, sparse joint positions/rotations, and dense root paths) passed to the model as inputs. But interaction requirements such as non-penetration, stable contact, and collision avoidance are implicit: they depend on scene geometry and whole-body pose, cannot be written as keypoint targets in advance, and conditioning on these constraints alone does not yield contact-consistent motion (Sec. 4). Noise-Space Optimization. We optimize over the prior’s latent variables rather than the 3D joint-position trajectory x directly, so every candidate remains a sample the prior can produce, retaining its learned realism and temporal coherence. Given latent noise ∈ℝT×Dz ^T× D unrolled through a differentiable DDIM sampling process, we solve minℒ(Mθ(∣y,),,)=wgℒgoal+wcℒcoll+wfℒfoot+whℒhand+weℒedge, _z\;L\! (M_θ(z y,C),\,C,S )=w_g\,L_goal+w_c\,L_coll+w_f\,L_foot+w_h\,L_hand+w_e\,L_edge, (1) The goal term ℒgoalL_goal is a masked loss on the constrained keypoints; the remaining terms are the scene-dependent collision, foot-contact, hand-contact, and edge-safety objectives; the last two are active only on tasks with grasps or terrain. We backpropagate through the entire denoising trajectory and optimize z with Adam; loss definitions and other details are in Appendix G. Behavior Diversity through Augmentations. For a fixed task specification, we generate diverse motions by randomizing initial poses, approach trajectories, and latent noise seeds (Figure 3). 3.2 Long-Horizon Multi-Stage Interaction Generation Figure 3: Generated interaction motions across the task suite. Each panel overlays frames from a single generated motion, labeled in-panel: the eight tasks we evaluate, together with four further behaviors whose task programs were written by the coding agent of Sec. 4.3. Single-window optimization is limited by the prior’s temporal horizon and by GPU memory. We therefore decompose long behaviors into temporal windows (yw,w)w=1W\(y_w,C_w)\_w=1^W optimized in sequence: the decoded suffix of window w is injected as a hard pose constraint into the prefix of window w+1w+1 after re-canonicalization to the new heading and origin, composing long-horizon behavior without global optimization. 3.3 Teacher-Student Visuomotor Learning Motion-Tracking Teacher. For each task we train a privileged teacher on the 2929-DoF Unitree G1 with PPO [33], through the training stack of [2], tracking the full batch of motions our framework generates for that task (Figure 2E). It observes proprioceptive state together with privileged reference and scene information and outputs 2929-D joint-angle residuals executed through PD control. Depth-Conditioned Flow Student. We distill the teacher into a student operating only from onboard observations: a history of head-mounted depth images and proprioception, encoded into a sensor context c. Because the same context can be consistent with several valid interaction behaviors, we model the teacher’s action distribution rather than its conditional mean, regressing a conditional velocity field with flow matching [21, 22]. Distillation is on-policy and at test time the field is integrated with a few fixed Euler steps; objective, architecture, and training details for both policies are in Appendix H. 4 Experiments We evaluate whether HIGenNTO can generate humanoid–scene interaction motion references and whether those references provide effective supervision for humanoid control. Our experiments address four questions: (1) Does noise-space optimization produce more contact- and scene-consistent motions than alternative methods of controlling the same prior (Sec. 4.1)? (2) Can the generated references be executed by physical policies and distilled to policies that use onboard sensing (Sec. 4.2)? (3) Does tracking the generated references supervise better than task-based rewards, hierarchical control, or adversarial imitation (Sec. 4.2)? (4) Can a coding agent reduce the engineering of extending HIGenNTO to new interactions (Sec. 4.3)? We additionally present qualitative real-world deployments in videos available on the project page. Figure 4: Humanoid–scene interaction synthesis with HIGenNTO and baseline methods. Rows, top to bottom: HIGenNTO (ours, blue), kinematic-constraint conditioning (Kimodo, yellow), classifier guidance (Kimodo-CG, green), and best-of-ten sampling (Kimodo best-of-10, purple); each column shows the identical scene, spawn, and constraint set. Stair and step panels overlay time samples of one motion sequence (opacity increases with time); chair panels show the final seated frame, and box panels the frame after grasping. 4.1 Humanoid–Scene Interaction Generation Tasks. We evaluate motion generation on eight tasks spanning three forms of interaction (Table 1, Appendix C): free-space locomotion (slalom walking among obstacles), object interaction (picking up, carrying, and placing boxes on the floor or a table), and terrain interaction (climbing and descending stairs, stepping onto and off blocks, sitting on and standing from chairs), randomizing scene geometry, object placement, interaction timing, and initial conditions throughout. Baselines. We compare HIGenNTO with three alternative mechanisms for controlling the same pretrained motion prior. All methods share prompts, random seeds, initial conditions, scene representation S, and kinematic constraints C. Kimodo samples the pretrained text-conditioned prior [32] conditioned on the kinematic constraints C, without access to the scene-dependent objectives defined by S. Kimodo-CG receives the same constraints and scene information as HIGenNTO, but applies the complete interaction objective as classifier guidance at each denoising step. HIGenNTO instead optimizes the initial noise by backpropagating the objective through the complete denoising trajectory. Kimodo best-of-10 draws ten independent samples per task specification and retains the one with the lowest interaction objective. Metrics. We measure target satisfaction (root path error, hand target error at constrained frames) and geometric consistency (scene penetration over body query points, foot–support gap at the lower foot sole). The last two measure separate failure modes: a motion can close the gap by penetrating, or avoid penetration by floating. We report mean ± standard deviation over motion sequences. Results. Per-task results are in Table 1, with full metric definitions in Appendix C. HIGenNTO achieves lower scene penetration across all three interaction categories, and lowers object-task hand-target error compared to the baselines. Kimodo-CG is an informative comparison because it receives the same constraints, scene representation, and interaction objective as HIGenNTO. Classifier guidance improves collision avoidance in free-space and terrain tasks but barely helps on object interaction, where dense root and hand conditioning leaves little room for the local corrections guidance can make. Best-of-ten selection barely narrows the gap: matched sampling compute does not help, because the prior rarely samples a candidate satisfying every interaction requirement at once, which noise-space optimization instead constructs. The box-pickup task shows this most clearly (Fig. 4). Across repeated samples and varying box sizes, the pretrained prior produces nearly identical wrist configurations and grasp widths, while HIGenNTO adapts the arm trajectory, wrist configuration, and hand separation to each object’s geometry. Long-Horizon Multi-Stage Generation. We apply the multi-stage procedure from Sec. 3 to generate a four-stage pick-and-place sequence (Figure 2C): each stage is optimized with its own prompt and constraint set, with terminal poses transferred between consecutive windows. Motion Repair. The same optimization can repair imperfect retargeted human–humanoid interaction references when initialized from DDIM-inverted noise (Figure 2D); setup and qualitative results are in Appendix D. 4.2 Learning Interaction Humanoid Policies from Generated References Do generated references supervise humanoid control better than task-level reinforcement-learning objectives? HIGenNTO learns humanoid control by training privileged, scene-aware teacher policies to track its generated interaction motion references, which are then distilled into depth-conditioned students (Sec. 4.2). Prior approaches instead optimize humanoid actions directly by reinforcement learning under task rewards, regularizing exploration with adversarial scene-unaware motion priors [31] or building on pretrained low-level controllers [4]. We compare these paradigms on Sit on a Chair using a shared set of 2020 HIGenNTO-generated motions. Baselines. Figure 5: Policy learning on Sit on a Chair. Task success over reinforcement-learning iterations for generated reference-based motion tracking by HIGenNTO-Teacher policies and four alternative training objectives. Each curve represents one training run. We compare HIGenNTO-Teacher policies that track the complete generated trajectories, with four approaches: TaskReward optimizes task rewards adapted from the Sit Down task of [40], with no motion prior, reference supervision, or contact and temporal supervision. HR-TaskReward trains a hierarchical policy under the same task rewards while using a frozen Homie controller [4] as its low-level policy. AMP [31] augments the task reward with an adversarial motion objective. Its discriminator is trained on the 2020 generated motion references excluding the scene. HIGenNTO-AMPScene additionally conditions the discriminator on chair-relative state from the generated references, providing interaction-aware adversarial supervision without trajectory tracking. All methods use task goals and initial states derived from the same generated motions; AMP and HIGenNTO-AMPScene additionally train their discriminators on them, and HIGenNTO-Teacher tracks them directly. Results. Figure 5 shows that the HIGenNTO-Teacher policy reaches high task success in approximately one third of the training iterations required by HIGenNTO-AMPScene, the strongest baseline. TaskReward fails to reliably discover the approach–contact–stabilization sequence from the task objective alone; HR-TaskReward and AMP add a pretrained controller and motion realism, respectively, but neither receives explicit supervision for the humanoid–chair interaction. Conditioning the discriminator on chair-relative state substantially improves HIGenNTO-AMPScene, confirming that the generated references are also useful as a scene-aware motion prior; it still learns more slowly than HIGenNTO-Teacher, whose tracking objective supplies a time-aligned target throughout the interaction. The generated references thus carry temporal and geometric interaction structure that shaped task rewards and motion-only regularization do not recover. Fig. 5 reports one run per method, and shows evidence of improved learning efficiency in this setting. Can the generated interaction motions be executed without privileged information at test time? We compare HIGenNTO privileged teacher and depth-based student policies in Table 2 (Appendix E). For context, we also evaluate SONIC [23], a general-purpose motion tracker, zero-shot from its released weights: SONIC receives the motion reference but cannot observe the surrounding geometry, whereas the HIGenNTO student observes the scene through onboard sensing but receives no motion reference, testing whether reference tracking alone suffices for interaction. All policies face the same eight tasks, scene configurations, and task-success criteria; the student, having no reference, is scored on task success only. Per-task results and complete metric definitions are provided there. Results. The privileged teachers execute the generated references reliably, and after distillation the students retain high success across all three interaction categories (Table 2) while operating without privileged state or motion references; that interaction structure transfers to policies acting from onboard sensing alone. The student’s long observation history matters on longer-horizon tasks: on Pick and Place on Table, the table is often occluded once the box is lifted, and the history lets the student retain the table position observed earlier in the rollout. SONIC occasionally attains lower joint or root-orientation error, but its high root-position error and low task success show that scene-blind tracking reproduces plausible pose sequences without aligning them to the relevant object or support surface. 4.3 Scaling HIGenNTO by Automating Task Specifications with a Coding Agent Each task in Sec. 3.1 is defined by a prompt y, constraints C, and a scene representation S; authoring and tuning these for every new interaction is a bottleneck. We investigate whether a coding agent can compile a natural-language task description into an executable HIGenNTO specification. Setup. Our implementation exposes a single task-level interface: every task is a function that takes a few randomization parameters (e.g. chair height, box size) and returns (y,,)(y,C,S), while the optimizer and the losses of Eq. 1 stay fixed. Given a short brief and the prior’s training-corpus annotations, the agent reads the corpus and writes a prompt y framed in a similar way to the text used to train the motion prior, probes the frozen prior to size the scene S and constraints C, drafts the task function, and renders a small batch to verify and revise it; a human only reviews the final motions with free-form feedback. The full procedure is detailed in Appendix F.2. Results. Three of the eight tasks evaluated in Sec. 4.1 (Slalom Walking, Step Up/Down, and Pick and Place on Table) were specified through this procedure. In a more open-ended setting with no target task, the agent identifies capabilities represented in the corpus annotations and proposes compatible scene interactions; Figure 3 shows four resulting tasks (Crawl Under, Duck Under, Squeeze Through Gap, Push a Box), spanning whole-body and non-prehensile object interaction with neither task-specific demonstrations nor manually authored constraints. Additional task definitions and qualitative results are provided in Appendix F.2. 4.4 Real-World Deployment We deploy depth-conditioned student policies for Sit on a Chair, Standing Up, Pick and Place on Table, and Pick and Place Box on a Unitree G1 using onboard depth and proprioception. Videos of the deployments and qualitative simulation results are available on the project page; Appendix A shows the rollouts. 5 Conclusion and Limitations We presented HIGenNTO, a framework for scalable humanoid interaction synthesis through constraint-guided latent optimization of pretrained motion priors. Rather than relying on large-scale interaction motion capture, HIGenNTO generates interaction-rich humanoid motions from sparse differentiable objectives encoding contacts, geometry, and temporal interaction structure, and the resulting motions serve both as interaction motion references for humanoid control and as supervision for deployable visuomotor policies. Our results suggest that interaction-rich behaviors need not be explicitly represented in the training distribution of a motion prior; they can emerge through optimization-time interaction constraints. Our limitations suggest directions for future work. First, the generation objectives enforce kinematic and geometric consistency rather than dynamic feasibility; incorporating dynamics could directly produce motions satisfying balance, contact-force, and actuation constraints. Second, noise-space optimization remains computationally expensive and thus suited to offline data generation; flow-map-based optimization or amortized generation could reduce this cost. Finally, the framework addresses whole-body motion with simplified hand–object contact; dexterous manipulation will require richer hand representations, contact models, and object-dynamics objectives. References [1] A. Allshire, H. Choi, J. Zhang, D. McAllister, A. Zhang, C. M. Kim, T. Darrell, P. Abbeel, J. Malik, and A. Kanazawa (2025) Visual imitation enables contextual humanoid control. External Links: 2505.03729, Link Cited by: §2. [2] Amazon FAR, P. Abbeel, J. Chen, R. Duan, A. Escontrela, M. Gandhi, S. Gundry, X. Huang, A. Kanazawa, T. Lewicki, J. Li, K. Liu, C. Rosenthal, Y. Seo, C. Sferrazza, G. Shi, L. Shih, J. Tseng, Z. Wu, L. Yang, B. Yi, and Y. Zhang (2025) Holosoma. Note: https://github.com/amazon-far/holosoma Cited by: §H.1, §H.2, §3.3. [3] J. P. Araujo, Y. Ze, P. Xu, J. Wu, and C. K. Liu (2025) Retargeting matters: general motion retargeting for humanoid motion tracking. External Links: 2510.02252, Link Cited by: §2. [4] Q. Ben, F. Jia, J. Zeng, J. Dong, D. Lin, and J. Pang (2025) HOMIE: humanoid loco-manipulation with isomorphic exoskeleton cockpit. External Links: 2502.13013, Link Cited by: §4.2, §4.2. [5] Bones Studio (2026) BONES-SEED: skeletal everyday embodiment dataset. Hugging Face. Note: https://huggingface.co/datasets/bones-studio/seed Cited by: §F.2. [6] Z. Chen, M. Ji, X. Cheng, X. Peng, X. B. Peng, and X. Wang (2025) GMT: general motion tracking for humanoid whole-body control. External Links: 2506.14770, Link Cited by: §1. [7] X. Cheng, Y. Ji, J. Chen, R. Yang, G. Yang, and X. Wang (2024) Expressive whole-body control for humanoid robots. External Links: 2402.16796, Link Cited by: §2. [8] T. Cheynel, T. Rossi, B. Bellot-Gurlet, D. Rohmer, and M. Cani (2025) ReConForM : real-time contact-aware motion retargeting for more diverse character morphologies. External Links: 2502.21207, Link Cited by: §2. [9] S. Choi and J. Kim (2019) Towards a natural motion generator: a pipeline to control a humanoid based on motion data. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vol. , p. 4373–4380. External Links: Document Cited by: §2. [10] S. Choi, M. J. Song, H. Ahn, and J. Kim (2021) Self-supervised motion retargeting with safety guarantee. External Links: 2103.06447, Link Cited by: §2. [11] W. Gomes, V. Radhakrishnan, L. Penco, V. Modugno, J. Mouret, and S. Ivaldi (2019) Humanoid whole-body movement optimization from retargeted human motions. In 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids), Vol. , p. 178–185. External Links: Document Cited by: §2. [12] C. Guo, S. Zou, X. Zuo, S. Wang, W. Ji, X. Li, and L. Cheng (2022) Generating diverse and natural 3d human motions from text. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 5142–5151. External Links: Document Cited by: §1, §2. [13] F. G. Harvey, M. Yurick, D. Nowrouzezahrai, and C. Pal (2020) Robust motion in-betweening. ACM Transactions on Graphics 39 (4). External Links: ISSN 1557-7368, Link, Document Cited by: §2. [14] M. Hassan, D. Ceylan, R. Villegas, J. Saito, J. Yang, Y. Zhou, and M. Black (2021) Stochastic scene-aware motion prediction. External Links: 2108.08284, Link Cited by: §2. [15] I. Jang, S. Choi, S. Hong, C. Kim, and J. Noh (2024) Geometry-aware retargeting for two-skinned characters interaction. ACM Trans. Graph. 43 (6). External Links: ISSN 0730-0301, Link, Document Cited by: §2. [16] M. Janner, Y. Du, J. B. Tenenbaum, and S. Levine (2022) Planning with diffusion for flexible behavior synthesis. External Links: 2205.09991, Link Cited by: §2. [17] K. Karunratanakul, K. Preechakul, E. Aksan, T. Beeler, S. Suwajanakorn, and S. Tang (2024) Optimizing diffusion noise can serve as universal motion priors. External Links: 2312.11994, Link Cited by: §1, §2. [18] K. Karunratanakul, K. Preechakul, S. Suwajanakorn, and S. Tang (2023) Guided motion diffusion for controllable human motion synthesis. External Links: 2305.12577, Link Cited by: §2. [19] J. Li, J. Wu, and C. K. Liu (2023) Object motion guided human motion synthesis. External Links: 2309.16237, Link Cited by: §1. [20] J. Lin, A. Zeng, S. Lu, Y. Cai, R. Zhang, H. Wang, and L. Zhang (2024) Motion-x: a large-scale 3d expressive whole-body human motion dataset. External Links: 2307.00818, Link Cited by: §2. [21] Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le (2023) Flow matching for generative modeling. External Links: 2210.02747, Link Cited by: §3.3. [22] X. Liu, C. Gong, and Q. Liu (2022) Flow straight and fast: learning to generate and transfer data with rectified flow. External Links: 2209.03003, Link Cited by: §3.3. [23] Z. Luo, Y. Yuan, T. Wang, C. Li, F. Castañeda, S. Chen, Z. Cao, J. Li, D. Minor, Q. Ben, J. Park, D. Sami, Z. Wang, X. Da, R. Ding, C. Hogg, L. Song, E. Lim, E. Jeong, T. He, H. Xue, W. Xiao, S. Yuen, J. Kautz, Y. Chang, U. Iqbal, L. ". Fan, and Y. Zhu (2026) SONIC: supersizing motion tracking for natural humanoid whole-body control. External Links: 2511.07820, Link Cited by: §1, §1, §4.2. [24] N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black (2019) AMASS: archive of motion capture as surface shapes. External Links: 1904.03278, Link Cited by: §1, §2. [25] J. Mao, S. Zhao, S. Song, T. Shi, J. Ye, M. Zhang, H. Geng, J. Malik, V. Guizilini, and Y. Wang (2024) Learning from massive human videos for universal humanoid pose control. External Links: 2412.14172, Link Cited by: §2. [26] M. Mihajlovic, S. Zhang, G. Li, K. Zhao, L. Müller, and S. Tang (2025) VolumetricSMPL: a neural volumetric body model for efficient interactions, contacts, and collisions. External Links: 2506.23236, Link Cited by: Appendix D. [27] G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. A. Osman, D. Tzionas, and M. J. Black (2019) Expressive body capture: 3d hands, face, and body from a single image. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Cited by: Appendix D. [28] X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne (2018) DeepMimic: example-guided deep reinforcement learning of physics-based character skills. ACM Transactions on Graphics 37 (4), p. 1–14. External Links: ISSN 1557-7368, Link, Document Cited by: §1, §2. [29] X. B. Peng, Y. Guo, L. Halper, S. Levine, and S. Fidler (2022) ASE: large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Transactions on Graphics 41 (4), p. 1–17. External Links: ISSN 1557-7368, Link, Document Cited by: §2. [30] X. B. Peng, A. Kanazawa, J. Malik, P. Abbeel, and S. Levine (2018) SFV: reinforcement learning of physical skills from videos. External Links: 1810.03599, Link Cited by: §2. [31] X. B. Peng, Z. Ma, P. Abbeel, S. Levine, and A. Kanazawa (2021) AMP: adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics 40 (4), p. 1–20. External Links: ISSN 1557-7368, Link, Document Cited by: §2, §4.2, §4.2. [32] D. Rempe, M. Petrovich, Y. Yuan, H. Zhang, X. B. Peng, Y. Jiang, T. Wang, U. Iqbal, D. Minor, M. de Ruyter, J. Li, C. Tessler, E. Lim, E. Jeong, S. Wu, E. Hassani, M. Huang, J. Yu, C. Chung, L. Song, O. Dionne, J. Kautz, S. Yuen, and S. Fidler (2026) Kimodo: scaling controllable human motion generation. External Links: 2603.15546, Link Cited by: §F.1, §F.2, §2, §2, §3.1, §4.1. [33] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. External Links: 1707.06347, Link Cited by: §H.2, §H.3, §3.3. [34] A. Serifi, R. Grandia, E. Knoop, M. Gross, and M. Bächer (2024) Robot motion diffusion model: motion generation for robotic characters. In SIGGRAPH Asia 2024 Conference Papers, SA ’24, New York, NY, USA. External Links: ISBN 9798400711312, Link, Document Cited by: §2, §2. [35] Y. Shafir, G. Tevet, R. Kapon, and A. H. Bermano (2023) Human motion diffusion as a generative prior. External Links: 2303.01418, Link Cited by: §2, §2. [36] J. Song, C. Meng, and S. Ermon (2022) Denoising diffusion implicit models. External Links: 2010.02502, Link Cited by: Appendix D. [37] C. Tessler, Y. Jiang, E. Coumans, Z. Luo, G. Chechik, and X. B. Peng (2025) MaskedManipulator: versatile whole-body manipulation. External Links: 2505.19086, Link Cited by: §2. [38] G. Tevet, S. Raab, S. Cohan, D. Reda, Z. Luo, X. B. Peng, A. H. Bermano, and M. van de Panne (2024) CLoSD: closing the loop between simulation and diffusion for multi-task character control. External Links: 2410.03441, Link Cited by: §2. [39] G. Tevet, S. Raab, B. Gordon, Y. Shafir, D. Cohen-Or, and A. H. Bermano (2022) Human motion diffusion model. External Links: 2209.14916, Link Cited by: §2, §2. [40] H. Wang, W. Zhang, R. Yu, T. Huang, J. Ren, F. Jia, Z. Wang, X. Niu, X. Chen, J. Chen, Q. Chen, J. Wang, and J. Pang (2025) PhysHSI: towards a real-world generalizable and natural humanoid-scene interaction system. External Links: 2510.11072, Link Cited by: §3, §4.2. [41] Y. Wang, J. Lin, A. Zeng, Z. Luo, J. Zhang, and L. Zhang (2023) PhysHOI: physics-based imitation of dynamic human-object interaction. External Links: 2312.04393, Link Cited by: §2. [42] Y. Wang, Q. Zhao, Y. F. Lau, R. Yu, H. W. Tsui, Q. Chen, J. Wang, J. Pang, and P. Tan (2026) HumanX: toward agile and generalizable humanoid interaction skills from human videos. External Links: 2602.02473, Link Cited by: §2, §3. [43] Y. Wang, Q. Zhao, R. Yu, H. W. Tsui, A. Zeng, J. Lin, Z. Luo, J. Yu, X. Li, Q. Chen, J. Zhang, L. Zhang, and P. Tan (2025) SkillMimic: learning basketball interaction skills from demonstrations. External Links: 2408.15270, Link Cited by: §2. [44] Z. Wang, J. J. Hunt, and M. Zhou (2023) Diffusion policies as an expressive policy class for offline reinforcement learning. External Links: 2208.06193, Link Cited by: §2. [45] H. Weng, Y. Li, N. Sobanbabu, Z. Wang, Z. Luo, T. He, D. Ramanan, and G. Shi (2025) HDMI: learning interactive humanoid whole-body control from human videos. External Links: 2509.16757, Link Cited by: §2. [46] Y. Xie, V. Jampani, L. Zhong, D. Sun, and H. Jiang (2024) OmniControl: control any joint at any time for human motion generation. External Links: 2310.08580, Link Cited by: §2. [47] L. Xu, X. Lv, Y. Yan, X. Jin, S. Wu, C. Xu, Y. Liu, Y. Zhou, F. Rao, X. Sheng, Y. Liu, W. Zeng, and X. Yang (2023) Inter-x: towards versatile human-human interaction analysis. External Links: 2312.16051, Link Cited by: Appendix D, §1, §2. [48] M. Xu, Y. Shi, K. Yin, and X. B. Peng (2025) PARC: physics-based augmentation with reinforcement learning for character controllers. In Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers, SIGGRAPH Conference Papers ’25, p. 1–11. External Links: Link, Document Cited by: §2. [49] S. Xu, H. Y. Ling, Y. Wang, and L. Gui (2026) InterMimic: towards universal whole-body control for physics-based human-object interactions. External Links: 2502.20390, Link Cited by: §2. [50] L. Yang, X. Huang, Z. Wu, A. Kanazawa, P. Abbeel, C. Sferrazza, C. K. Liu, R. Duan, and G. Shi (2026) OmniRetarget: interaction-preserving data generation for humanoid whole-body loco-manipulation and scene interaction. External Links: 2509.26633, Link Cited by: Appendix D, §2. [51] S. Yin, Y. Ze, H. Yu, C. K. Liu, and J. Wu (2025) VisualMimic: visual humanoid loco-manipulation via motion tracking and generation. External Links: 2509.20322, Link Cited by: §H.4. [52] X. Zhang, B. L. Bhatnagar, V. Guzov, S. Starke, and G. Pons-Moll (2022) COUCH: towards controllable human-chair interactions. External Links: 2205.00541, Link Cited by: §2. [53] K. Zhao, G. Li, and S. Tang (2025) DartControl: a diffusion-based autoregressive motion model for real-time text-driven motion control. External Links: 2410.05260, Link Cited by: §1, §2. [54] Y. Zheng, R. Liang, K. Zheng, J. Zheng, L. Mao, J. Li, W. Gu, R. Ai, S. E. Li, X. Zhan, and J. Liu (2025) Diffusion-based planning for autonomous driving with flexible guidance. External Links: 2501.15564, Link Cited by: §2. Appendix A Real-world deployment Figure 6: Real-world deployment of the distilled students, using only onboard sensor observations. The robot approaches a chair and sits down (Sit on a Chair); stands back up from the seated state (Standing Up); lifts a box from the floor onto a table (Pick and Place on Table); and picks up and carries a box (Pick and Place Box). Figure 6 shows the distilled student running on the physical robot for four tasks, from onboard sensing alone. Appendix B Student evaluation in simulation Figure 7: Evaluation of the trained student policies in simulation. The inset in each frame is the depth image the policy receives at that instant. Figure 7 shows the qualitative results for the trained students in simulation on all eight tasks, with depth and proprioception the policy’s only inputs. Appendix C Motion-generation results C.1 Per-task results Comparison setup. Table 1 reports the motion-generation results per task. All four methods are tested on an identical scene and constraint set. The baselines are sampled with 5050 DDIM denoising steps, while HIGenNTO uses 1010 steps to limit the memory and compute required for backpropagation through the denoising process; Kimodo best-of-10 draws ten samples per motion sequence from the motion prior and keeps the one scoring lowest under the weighted objective of Appendix G.2. Runtime and compute. One noise-optimization iteration of HIGenNTO backpropagates through a 1010-step DDIM process (Alg. 1). On a single RTX A6000 it takes 9.79.7–2626 s at batch size 2020 per iteration. The motions are optimized for K=50K=50–100100 iterations. Free Object Terrain Method Slalom Pick and Pick and Place Climbing Descending Step Sit on Standing Walking Place Box on Table Stairs Stairs Up/Down a Chair Up Root path error (cm) ↓ Ours 4.53 ± 0.82 2.65 ± 0.26 2.73 ± 0.53 2.78 ± 1.63 6.70 ± 3.40 3.89 ± 0.61 3.86 ± 0.74 2.87 ± 0.29 Kimodo 7.23 ± 3.20 2.93 ± 0.65 2.29 ± 0.51 34.00 ± 27.82 16.41 ± 7.50 2.86 ± 0.33 4.61 ± 1.15 2.96 ± 0.23 Kimodo-CG 7.23 ± 2.84 2.58 ± 0.32 2.10 ± 0.35 10.29 ± 8.85 12.81 ± 5.57 2.90 ± 0.66 4.49 ± 0.99 2.95 ± 0.26 Kimodo best-of-10 7.22 ± 3.43 2.48 ± 0.48 2.30 ± 0.47 28.25 ± 19.41 17.06 ± 4.38 2.76 ± 0.20 4.46 ± 1.35 3.14 ± 0.19 Hand target error (cm) ↓ Ours – 6.36 ± 2.26 6.61 ± 2.67 – – – – – Kimodo – 18.93 ± 2.18 21.25 ± 1.78 – – – – – Kimodo-CG – 19.18 ± 2.23 20.60 ± 1.48 – – – – – Kimodo best-of-10 – 18.29 ± 2.37 19.97 ± 1.59 – – – – – Scene penetration (cm) ↓ Ours 0.01 ± 0.02 0.58 ± 0.46 0.51 ± 0.47 1.10 ± 0.49 1.94 ± 0.62 0.08 ± 0.08 1.25 ± 0.57 1.38 ± 1.63 Kimodo 33.96 ± 4.11 7.07 ± 6.74 10.35 ± 7.51 0.06 ± 0.08 232.79 ± 164.20 8.45 ± 3.39 9.42 ± 10.17 115.73 ± 118.03 Kimodo-CG 2.99 ± 0.95 7.95 ± 6.50 10.62 ± 7.09 0.01 ± 0.02 1.63 ± 0.88 1.97 ± 2.32 6.04 ± 7.22 107.59 ± 111.51 Kimodo best-of-10 27.37 ± 2.84 4.99 ± 6.08 6.68 ± 5.70 0.04 ± 0.05 149.14 ± 103.53 4.63 ± 2.32 5.39 ± 6.26 86.75 ± 97.79 Foot–support gap (cm) ↓ Ours 0.01 ± 0.04 2.60 ± 0.22 2.82 ± 0.22 2.31 ± 0.30 3.17 ± 1.21 0.46 ± 0.59 0.21 ± 0.14 0.19 ± 0.10 Kimodo 0.00 ± 0.01 3.77 ± 0.80 3.24 ± 1.51 51.22 ± 14.42 0.42 ± 0.27 2.69 ± 2.06 0.75 ± 1.18 1.62 ± 2.93 Kimodo-CG 0.01 ± 0.06 3.17 ± 0.73 2.96 ± 0.56 24.57 ± 4.92 18.78 ± 5.65 3.48 ± 1.90 0.47 ± 0.24 1.66 ± 2.91 Kimodo best-of-10 0.00 ± 0.00 5.25 ± 1.02 4.94 ± 1.27 39.46 ± 12.92 0.59 ± 0.30 2.99 ± 1.77 1.12 ± 1.21 1.92 ± 2.51 Table 1: Motion-generation ablation, per task, on our dataset. All four methods are provided identical scene and constraints for each sequence. Bold == best mean per metric. Metric definitions in Appendix C.2. C.2 Evaluation metrics This section defines the four metrics of Table 1. All are reported in centimeters and lower is better. jt∈ℝ3x^t_j ^3 is the world position of joint j at frame t of a generated motion sequence, over T frames and J skeleton joints. The contact threshold τ is the clearance within which a query point counts as touching a surface. The skin distance δj _j is the clearance each joint is expected to keep clear of static geometry. Against a moving obstacle the corresponding clearance is δjmovδ^mov_j. Root path error. The mean Euclidean distance, in the world ground plane, between the generated smoothed root position tr^t and its target ^t r^t, evaluated only at the constrained frames rT_r: Eroot=1|r|∑t∈r‖Πxyt−^t‖2,E_root= 1|T_r| _t _r \| _xyr^t- r^t \|_2, (2) with Πxy _xy the projection onto the ground plane. Every task carries a root path constraint, so this is defined throughout; because that path is handed to all four methods, the metric reports how well each preserves a constraint it was given. Hand target error. The mean Euclidean distance between a constrained hand joint jej_e and its keyframe target ^jet x^t_j_e over the specified keyframes eT_e: Ehand=1|e|∑t∈e‖jet−^jet‖2.E_hand= 1|T_e| _t _e \|x^t_j_e- x^t_j_e \|_2. (3) It is defined only for tasks whose constraint set specifies hand end-effector targets, and reported as “–” elsewhere. Scene penetration. The per-frame penetration depth into scene geometry, summed over body query points and averaged over the sequence. Writing s(⋅)s(·) for the signed distance to the static scene and so(⋅)s_o(·) for the signed distance to moving obstacle o, Epen=1T∑t=1T[ E_pen= 1T _t=1^T [ ∑jmax(0,δj−s(jt)) _j \! (0,\ _j-s(x^t_j) ) (4) + + ∑p∈max(0,τ−s(pt)) _p \! (0,\ τ-s(x^t_p) ) + + ∑o∑jmax(0,δmovj−so(tj))], _o _j \! (0,\ δ^mov_j-s_o(x^t_j) ) ], where P is the set of sole points of both feet; querying the sole rather than the ankle joint captures the actual surface of the feet. Joints gripping a moving obstacle are excluded while the object is held, so an intended grasp is not penalized, but other joints are penalized for penetrating the obstacle. Foot–support gap. The mean height of the lower foot’s nearest sole point above the support surface, beyond a contact threshold: Egap=1T∑t=1Tmax(0,minp∈s(pt)−τ).E_gap= 1T _t=1^T \! (0,\ _p s(x^t_p)\;-\;τ ). (5) The minimum runs over the same P, so it selects the lowest sole point of whichever foot is lower. Why the last two are reported together. Scene penetration and foot–support gap are the two one-sided halves of a single signed clearance: a motion can drive penetration to zero by hovering above the scene entirely, and can close the support gap by penetrating through it. Both failure modes occur among the baselines (Table 1), which is why we report the pair. Appendix D Repairing retargeted interaction motions Figure 8: Motion reference repair. HIGenNTO initialized with DDIM-inverted noise. The same optimization framework can repair an existing interaction trajectory by initializing from its inverted latent noise rather than from random noise. We demonstrate this on two Inter-X sequences [47], Hug and HighFive, retargeted with OmniRetarget [50]: in the resulting references the hands fail to meet during the high-five, while the hug shows body interpenetration without the intended surface contact (Figure 8, first row). To construct the repair constraints we threshold inter-body distances in the original human–human sequence for candidate contact locations and times, then use a vision-language model to retain the interaction-relevant ones as C. The human is represented by the SMPL-X [27] body SDF from VolumetricSMPL [26], and the latent is initialized by DDIM inversion [36] of the retargeted trajectory. HIGenNTO then restores the intended contacts in both interactions while preserving the structure of the original motion (Figure 8, last row). Random initialization suffices for the simpler high-five but fails on the more tightly constrained hug (second row), indicating that inversion provides an important initialization when repairing interactions with dense contact requirements. Appendix E Motion-tracking results E.1 Per-task results Table 2 reports the motion-tracking results per task, aggregated by interaction type (free-space, object, terrain) in the main text. The two trackers are scored on every metric and the student, which receives no motion reference, is scored on task success alone. Free Object Terrain Method Slalom Pick and Pick and Place Climbing Descending Step Sit on Standing Walking Place Box on Table Stairs Stairs Up/Down a Chair Up Joint error (rad) ↓ Teacher (ours) 0.141 ± 0.014 0.182 ± 0.022 0.296 ± 0.065 0.087 ± 0.008 0.107 ± 0.011 0.099 ± 0.010 0.087 ± 0.008 0.068 ± 0.012 SONIC 0.231 ± 0.059 0.114 ± 0.009 0.100 ± 0.025 0.326 ± 0.128 0.220 ± 0.104 0.124 ± 0.050 0.115 ± 0.025 0.388 ± 0.138 Root position error (m) ↓ Teacher (ours) 0.131 ± 0.047 0.076 ± 0.010 0.128 ± 0.069 0.056 ± 0.008 0.060 ± 0.009 0.059 ± 0.007 0.058 ± 0.008 0.062 ± 0.016 SONIC 1.153 ± 0.387 0.462 ± 0.080 0.526 ± 0.151 1.399 ± 0.297 0.763 ± 0.278 1.538 ± 0.670 0.341 ± 0.124 1.006 ± 0.174 Root orientation error (rad) ↓ Teacher (ours) 0.221 ± 0.047 0.193 ± 0.033 0.301 ± 0.119 0.144 ± 0.021 0.168 ± 0.031 0.127 ± 0.022 0.145 ± 0.020 0.140 ± 0.034 SONIC 0.453 ± 0.167 0.153 ± 0.009 0.154 ± 0.060 0.747 ± 0.324 0.485 ± 0.269 0.224 ± 0.107 0.198 ± 0.041 0.864 ± 0.368 Task success (%) ↑ Teacher (ours) 97.7 ± 2.1 99.5 ± 0.8 71.1 ± 34.8 99.7 ± 0.2 99.5 ± 0.5 100.0 ± 0.0 99.9 ± 0.1 97.6 ± 2.2 Student (ours) 65.9 ± 25.4 80.9 ± 18.0 57.4 ± 36.7 96.5 ± 5.2 95.2 ± 4.9 98.3 ± 1.6 88.5 ± 20.1 96.3 ± 4.2 SONIC 3.1 ± 4.9 8.0 ± 8.3 0.0 ± 0.0 0.2 ± 0.5 28.8 ± 23.0 0.2 ± 0.4 19.5 ± 22.6 3.9 ± 7.0 Table 2: Motion tracking, per task. Bold == best mean per metric. Setup and rubrics in Appendix E.2. E.2 Evaluation metrics and setup Metrics. The three error metrics are accumulated once per control step and averaged over the attempt. Joint error is the mean absolute difference between reference and measured joint angles over the 2929 actuated degrees of freedom, in radians. Root position error is the Euclidean distance between reference and measured root positions in the world frame, in meters, and root orientation error the geodesic angle between the two root quaternions, in radians. Uncensored attempts. Every attempt plays its reference from the first frame to the last: terminations are recorded, but the environment keeps running rather than resetting. This avoids the bias of censored measurements, under which a policy that terminates early is scored almost entirely on the easy opening of every sequence. Task success. Task success measures how well the expected behavior is executed. It combines a small set of criteria selected per task, derived from the reference motions: terminal root height and uprightness for stair climbing; the same plus terminal root location for stair descent and standing up; terminal root location and height for sitting; terminal root height, uprightness, and peak height for stepping onto a box; terminal root location and uprightness for slalom walking; and terminal object position for the box-manipulation tasks. Evaluation setup. Each policy is rolled with 40964096 environments for the teacher and SONIC and 384384 for the students, which are limited by the memory requirements of the simulated depth camera. Students are evaluated on clean rendered depth with no augmentation. Appendix F Task specifications F.1 Per-task specifications Every task is specified through the three components of Sec. 3.1: a prompt y for the frozen Kimodo prior [32], sparse spatiotemporal constraints C on a few keypoints (typically root/pelvis and hands, plus a heading direction), and a scene S of SDFs and terrain heightfields. It needs no additional data collection. The feet are never given targets: foot placement comes entirely from the contact and edge losses against the scene. The chair, object, and slalom tasks sample an initial root pose and an obstacle-avoiding path, which is their main source of behavioral diversity; the stair and step tasks prescribe the root trajectory from the geometry instead, varying the terrain and the sampled noise. Per-task loss weights are in Appendix G.2. • Climbing Stairs: prompt “A person climbs up stairs.” The root follows a stepped path up the staircase while the foot-contact and edge losses place the feet on the treads. Scene: a closed-form staircase SDF over a ground plane, with a heightfield obtained from the same geometry, which is used to compute the edges for ℒedgeL_edge. Randomization: tread (step depth) ∈0.3,0.4∈\0.3,0.4\ m × rise (step height) ∈0.1,0.2∈\0.1,0.2\ m over five steps. • Descending Stairs: prompt “A person climbs down stairs.” Scene and randomization as for Climbing Stairs. • Step Up/Down: prompts “A person climbs up a box.” then “A person climbs down a box.”, optimized as two overlapping windows. The root follows a prescribed trajectory: a walk in, a rise onto the box, a pause on top, and a drop. Scene: a closed-form box SDF over a ground plane. Randomization: box heights 0.3,0.4,0.5\0.3,0.4,0.5\ m against depths 0.3,0.6\0.3,0.6\ m. • Sit on a Chair: prompt “A person walks for sometime and sits down on a chair.” The root follows an approach path and is then held at the seat, with the robot turned to face away from the backrest, as it must be to sit. Scene: a parametric chair SDF consisting of 22 cuboids (seat and backrest) on a ground plane. Randomization: seat height ∈0.2,0.3,0.4,0.5∈\0.2,0.3,0.4,0.5\ m, with start positions drawn within a 2.02.0 m radius. • Standing Up: prompt “A person sitting on a chair stands up and walks forward.”, which starts the motion seated without any explicit initialization. The root is held at the seat, with its heading held in the same seated orientation as Sit on a Chair; it is then constrained along a prescribed path 0.50.5 m out from the chair’s front edge. There is no explicit stand-up constraint, the rise results from the prompt. Scene: the same chair SDF as Sit on a Chair. Randomization: seat height ∈0.2,0.3,0.4,0.5∈\0.2,0.3,0.4,0.5\ m. • Slalom Walking: prompt “A person walks forward, turning left and right to avoid obstacles.” A path weaving between the pillars is prescribed and paced at 1.0m/s1.0\,m/s; the root tracks it and the pelvis height is held at its nominal standing value. Scene: a union of SDFs of pillars which act as obstacles over a ground plane. Randomization: pillar shape ∈box,cone∈\box,cone\ × count ∈3,4,5∈\3,4,5\ × spacing ∈0.9,1.2,1.5∈\0.9,1.2,1.5\ m. • Pick and Place Box: prompts “A person walks.”, “A person lifts a crate with both hands.”, “A person carrying a crate with both hands walks forward.”, and “A person places a crate on the floor with both hands.”, optimized as four overlapping windows. Both hand end effectors are constrained to the box’s side faces and the pelvis height profile is prescribed for the lift and the set-down. Scene: a ground plane and a moving box SDF driven by an object trajectory. Randomization: boxes with side lengths in [0.15,0.50][0.15,0.50] m; the start pose is drawn from a wedge-shaped region 0.550.55–1.01.0 m behind the box. • Pick and Place on Table: the same four windows as Pick and Place Box, with the fourth prompt “A person puts a crate down with both hands.” Only the destination for the constraints differs: the hands settle at the table’s height and the pelvis height is constrained to remain near standing height rather than dropping into a squat. Scene: adds a static table cuboid. Randomization: boxes with side lengths in [0.15,0.50][0.15,0.50] m, with table height ∈0.2,0.4,0.6∈\0.2,0.4,0.6\ m and table width ∈0.3,0.5∈\0.3,0.5\ m at a fixed 0.60.6 m length. F.2 Agent-authored task specification Figure 9: Agent-authored behaviors across their randomization settings. Each panel overlays frames from a single generated motion. The three rows are different scene configurations, and the columns are (a) Crawl Under, varying slab clearance and length; (b) Duck Under, varying slab clearance and depth; (c) Squeeze Through Gap, varying gap width; and (d) Push a Box, varying box path. The prompt and constraints behind every panel were written by the coding agent, with no task-specific demonstrations and no hand-authored constraints. This section details the procedure of Sec. 4.3, by which a frontier coding agent (Claude Code running Claude Fable 5) authors the task programs consumed by HIGenNTO. Task interface. The agent writes a single function per task. This function takes a few task-specific parameters as input and returns the prompt y, the constraints C, and the scene S of Sec. 3.1, together with the object trajectory and the frames in which contact is required. Varying those parameters is what yields the diverse scene configurations and initial poses for each task. The agent is instructed not to modify the optimizer or the loss terms. Procedure. Given a short natural-language brief and a pointer to the text annotations of the corpus the prior was trained on, the agent: (i) mines the annotations for the group describing the requested behavior and adopts the corpus’s own phrasing as the prompt y; (i) probes the frozen prior by running the candidate prompts with no scene and no constraints, then measures the resulting motion for an initial estimate of plausible scene sizes and constraints; (i) drafts the first iteration of the function that builds the scene and constraints for the task; (iv) generates a batch of motions (22 settings × 55 motions × 100100 optimization steps), reports per-term losses, renders the motions as videos to check that they follow the intended behavior and look natural, and iterates if needed. Steps (i) and (iv) close the loop: the agent verifies that a task works before involving a human. Agent-authored behaviors. An agent can author tasks in two ways. In the user-specified mode the human names the behavior (Pick and Place on Table, Slalom Walking, Step Up/Down). In the discovery mode (Crawl Under, Duck Under, Squeeze Through Gap, Push a Box) the human names no task at all: given only the corpus annotations, the agent surveys the prior’s vocabulary and returns grounded task proposals, each citing the annotation group that makes it in-distribution, sketching an implementation, and proposing axes along which to randomize the motions. We use the annotations from the BONES-SEED dataset [5], used in Kimodo [32]. Figure 9 shows generated motions for each of these discovered tasks, with three scene configurations per task. All four use the same interface as the tasks of Appendix F.1, and their optimizer settings and loss weights are listed alongside them in Table 3. • Crawl Under: prompts “A person walks forward and crouches down.”, “A person crawls forward on hands and knees.”, and “A person stands up and walks forward.”, generated over three windows. The root follows a prescribed path whose height descends from standing to a hands-and-knees pelvis height (this height is found by probing the prior as described in step (i)) before the slab’s near face, is held through the slab span, and rises only once the far face is cleared. Each window boundary therefore falls where the body is momentarily at rest. Scene: a horizontal slab raised on four corner posts over a ground plane (55 cuboid SDFs), where the clearance is the free gap between the ground and the slab’s underside. Randomization: clearance ∈0.65,0.75∈\0.65,0.75\ m and slab length ∈1.2,1.8∈\1.2,1.8\ m, with start positions randomized up to 0.250.25 m back along the approach path. • Duck Under: prompt “A person walks forward and bends down to avoid an obstacle.” The root follows a straight path at a constant speed, with a pelvis-height constraint that dips across the slab span and returns to standing once past it; the SDF-based loss against the slab forces the head to clear it. Scene: an overhead slab spanning the path on two side posts over a ground plane (33 cuboid SDFs), where the clearance is the gap between the ground and the slab’s underside. Randomization: clearance ∈1.00,1.15∈\1.00,1.15\ m and slab depth ∈0.20,0.60∈\0.20,0.60\ m, with start positions randomized up to 0.300.30 m back along the approach path. • Squeeze Through Gap: prompt “A person turns sideways and walks through a narrow gap.” The root path runs straight through the gap center, while a ±π/2±π/2 heading constraint on the frames at the narrowest point turns the body sideways for the squeeze. Scene: two facing wall blocks (22 cuboid SDFs, 2.02.0 m tall) on a ground plane, leaving a corridor of gap width across the path. Randomization: gap width ∈0.42,0.48,0.55∈\0.42,0.48,0.55\ m, the leading shoulder sampled left or right per motion sequence, and start positions randomized up to 0.250.25 m back along the approach path. • Push a Box: prompts “A person walks.”, “A person strains while slowly moving forward, pushing a large box.”, and “A person stands still.”, generated over three windows. The path of the box is prescribed first as a constant-curvature arc, and both hand targets are derived from it as rigid offsets on the box’s rear face. The root is constrained to trail the rear face at a fixed distance, with its heading held to the path tangent and its pelvis lowered through the pushing phase; it steps back once the box is released. Scene: a ground plane and a rigid box represented as a moving box SDF following the prescribed object trajectory. Randomization: box edge ∈0.5,0.6∈\0.5,0.6\ m × path ∈\straight, left arc, right arc\ with curvature ±0.15m−1± 0.15\,m^-1, pushed 1.751.75–2.02.0 m, with start positions drawn from a wedge-shaped region 0.90.9–1.81.8 m behind the box. Appendix G Constraint-guided motion optimization G.1 Optimization algorithm Algorithm 1 Constraint-guided motion optimization. 1: prompt y, constraints C, scene S, outer steps K, DDIM steps N, batch size B 2: Initialize ∼(0,σinit2)z (0, _init^2I) for B samples 3: for k=1,…,Kk=1,…,K do 4: ←Mθ(∣y,)x← M_θ(z y,C) ⊳ full N-step DDIM denoise with gradients enabled 5: Align x to the global coordinate frame at the desired frame-0 root position and orientation 6: ℒ←wgℒgoal(,)+wcℒcoll(,)+wfℒfoot(,)+whℒhand(,)+weℒedge(,)L← w_gL_goal(x,C)+w_cL_coll(x,S)+w_fL_foot(x,S)+w_hL_hand(x,S)+w_eL_edge(x,S) 7: ←∇ℒg← _zL 8: ←Adam(,)z← Adam(z,g); record per-sample best b⋆x _b at the lowest ℒbL_b seen so far 9: end for 10: return b⋆b=1B\x _b\_b=1^B G.2 Loss terms We detail the loss terms summed in line 6 of Alg. 1. Let jt∈ℝ3x^t_j ^3 denote the world-frame position of joint j at frame t for a single sample. The scene SDF s(⋅)s(·) (negative inside geometry), the contact threshold τ, and the skin distances δj _j and δjmovδ^mov_j are as defined in Appendix C.2, as is the set of sole points P; ℱ=left,rightF=\left,\,right\ indexes the two feet. Goal (ℒgoalL_goal). A mask-normalized Huber loss between the denoised joints and the targets ^jt x^t_j obtained from C, gated by a per-frame mask mjt∈0,1m^t_j∈\0,1\ that selects only the constrained (t,j)(t,j) entries: ℒgoal=∑t,jmjtH1(jt,^jt)∑t,jmjt,L_goal= _t,jm^t_j\,H_1\! (x^t_j,\, x^t_j ) _t,jm^t_j, (6) where H1H_1 is the Huber loss (δH=1 _H=1) summed over the three coordinates. This single term handles the constraints on the root, hand, and full-body; the mask is the only thing that changes across tasks. Foot contact (ℒfootL_foot). Penalizes the foot nearest to the ground from floating, encouraging at least one foot to stay in contact at all times. It is queried at points on the sole of the feet P. ℒfoot=1T∑t=1Tmax(0,minp∈s(pt)−τ).L_foot= 1T _t=1^T \! (0,\; _p s(x^t_p)-τ ). (7) Hand contact (ℒhandL_hand). Holds the hands at a fixed palm clearance dpalmd_palm off the object’s faces, measured against the object’s moving SDF over the grip window G and the hand keypoints ℋH. Unlike the terms above it is two-sided (a hand drifting off the object is penalized as much as one sinking into it) and tolerates a deviation of up to τ from the target clearance: ℒhand=1|||ℋ|∑t∈∑j∈ℋmax(0,|so(jt)−dpalm|−τ).L_hand= 1|G|\,|H| _t _j \! (0,\ |s_o(x^t_j)-d_palm |-τ ). (8) It is active on the tasks involving object manipulation. Collision (ℒcollL_coll). Penalizes penetration of any joint into scene geometry up to its skin distance: ℒcoll=1T∑t=1T∑j=1Jmax(0,δj−s(jt)).L_coll= 1T _t=1^T _j=1^J \! (0,\; _j-s(x^t_j) ). (9) Each moving obstacle contributes an additional term of the same form, queried against its per-frame (moving) SDF. Edge contact (ℒedgeL_edge). This term keeps the planted foot away from sharp terrain edges. Each foot f carries a query set fQ_f of its ankle and toe points, and we define a per-point risk as a linear ramp in the distance d(q)d(q) to the nearest edge, ℓ(q)=1−clamp(d(q)/ρ, 0, 1)∈[0,1], (q)=1-clamp\! (d(q)/ρ,\,0,\,1 )∈[0,1], (10) where ρ is a safety radius and d(q)d(q) is the distance from an edge precomputed from the terrain heightfield. The planted foot f⋆f is the one whose lowest query point sits closest to the terrain, f⋆=argminf∈ℱminq∈f(z(qt)−h(qt)),f = _f \; _q _f (z(q^t)-h(q^t) ), (11) with z(⋅)z(·) the height of a query point and h(⋅)h(·) the terrain height beneath it. The term is the planted foot’s mean point risk, averaged over the sequence, ℒedge=1T∑t=1T1|f⋆|∑q∈f⋆ℓ(qt).L_edge= 1T _t=1^T 1|Q_f | _q _f \! (q^t ). (12) The choice of f⋆f is treated as a constant in the backward pass, so gradients flow through the selected foot’s risk but not through which foot is selected. Loss weights. The goal weight is wg=1.0w_g=1.0 throughout; the remaining weights and the optimizer settings are per task, and are listed in Table 3. The shared constants are the contact threshold τ=0.01mτ=0.01\,m; the skin distance δj=0.05m _j=0.05\,m (0.01m0.01\,m on the ankles and toes) and δjmov=0.05mδ^mov_j=0.05\,m against moving obstacles (0.03m0.03\,m on the legs and toes); the safety radius ρ=0.10mρ=0.10\,m; and Huber δH=1 _H=1. The latent is initialized at σinit=0.1 _init=0.1 (Alg. 1), which we observe to produce more natural-looking results when using noise optimization; the baselines use σ=1σ=1. The edge term is enabled only where the scene carries a heightfield, and the hand-contact term only where the task carries a grip. Task K η wcw_c wfw_f wew_e whw_h Chair (sit, stand) 50 0.01 0.1 0.1 0 0 Stairs (up, down) 50 0.05 1.0 1.5 1.0 0 Step up/down 50 0.05 2.0 1.5 1.0 0 Slalom 100 0.05 2.0 1.5 0 0 Pick and place 100 0.01 0.5 0.1 0 1.0 Agent-authored (Appendix F.2) Crawl under 100 0.05 2.0 1.5 0 0 Duck under 100 0.05 2.0 1.5 0 0 Squeeze through gap 100 0.05 2.0 1.5 0 0 Push a box 100 0.05 2.0 1.5 0 1.0 Table 3: Optimizer settings and loss weights per task family, for the eight evaluated tasks and the four agent-authored behaviors. K is the number of outer Adam iterations of Alg. 1 and η its learning rate; wg=1.0w_g=1.0 throughout. The two pick tasks share one recipe, as do the two chair tasks and the two stair tasks. The agent-authored tasks were all generated with a single recipe, differing only in whether the task carries a grip. Appendix H Policy learning H.1 Simulation setup and dynamics randomization Simulator and robot. Teachers and students are trained in Isaac Sim through the Holosoma training stack [2]. Physics is stepped at 200Hz200\,Hz with a control decimation of four, giving the 50Hz50\,Hz control rate. The robot is a Unitree G1 with 2929 actuated degrees of freedom over 3232 bodies, driven by joint-space PD control with per-joint gains and the manufacturer’s torque limits; the policy output is a residual on a fixed default pose. Episodes and termination. Teachers are trained with 40964096 parallel environments and students with 384384. Episodes last 1010–15s15\,s, drawing a reference motion sequence uniformly from the task’s batch at each reset. An episode terminates when the root drifts more than 0.5m0.5\,m from the reference, when the projected gravity of the root differs from the reference by more than 0.80.8, when any of four bodies (the two ankles and the two wrists) exceeds 0.25m0.25\,m of tracking error, or, on object tasks, when the object’s pose relative to the robot deviates by more than 0.25m0.25\,m or 0.8rad0.8\,rad. Dynamics randomization. The following are resampled per environment: robot friction (0.30.3–1.61.6 static, 0.30.3–1.21.2 dynamic) and restitution (0.00.0–0.50.5); object friction (0.10.1–0.60.6), restitution (0.00.0–1.01.0), mass (11–4×4×) and principal inertia (0.50.5–1.5×1.5×); base center-of-mass offset (±0.025± 0.025 m longitudinally, ±0.05± 0.05 m laterally and vertically); and a ±0.01rad± 0.01\,rad joint-position bias. Random pushes are applied every 11–3s3\,s at up to 0.5m/s0.5\,m/s linear and 0.78rad/s0.78\,rad/s angular velocity. Observations carry Gaussian noise on base angular velocity (0.20.2), joint position (0.010.01), joint velocity (0.50.5), and reference orientation (0.050.05). The teacher and student stages randomize identically; the student adds the depth randomization (Appendix H.4). H.2 Motion-tracking teacher A single teacher (Sec. 3.3) is trained per task with PPO [33] on the full batch of motions generated for that task: at each episode reset, a reference trajectory is sampled uniformly from the batch and the policy is required to track it. The actor consumes the robot’s proprioception, the kinematic reference over a short horizon of future steps, and a privileged terrain height scan; on object tasks it additionally receives the object’s pose, velocity, and sampled surface points in the robot frame. It outputs a 2929-D joint-angle residual over the default pose, applied through PD control. The critic is further privileged with reference-body and tracked-body poses and the base linear velocity. Rewards combine standard motion-tracking terms (global and per-body pose, position, and velocity tracking, plus object-pose tracking on object tasks) with regularizers on action rate, DoF limits, and undesired contacts. The actor-critic architecture and PPO hyperparameters are the default values from [2]. H.3 Depth-conditioned flow student Objective. Along the straight-line path t=(1−t)tea+tϵa_t=(1-t)\,a^tea+t\, ε that connects the teacher action teaa^tea at t=0t=0 to Gaussian noise ϵ∼(,) ε (0,I) at t=1t=1, the student (Sec. 3.3) regresses a conditional velocity field vθv_θ onto the velocity of that path, given the onboard sensor context c: ℒstudent=t,ϵ‖vθ(t,t,)−(ϵ−tea)‖2.L_student=E_t, ε \|\,v_θ(a_t,t,c)- ( ε-a^tea ) \|^2. (13) At test time an action is obtained by integrating vθv_θ from t=1t=1 to t=0t=0. Observations. The student (Sec. 3.3) observes only quantities the robot carries: a short history of head-mounted depth frames, each encoded by a shared depth CNN into a compact embedding, and a proprioception stream (base angular velocity, projected gravity, joint-angle deviations from the default pose, joint velocities, end-effector positions in the base frame, and the previous action) kept as both a short recent window and a longer buffer compressed by a temporal 1-D CNN. Shorter-horizon tasks use a smaller memory. It outputs the same 2929-D joint-angle residual as the teacher. Architecture. The velocity field vθv_θ of Eq. 13 is a small transformer that attends over tokens for the proprioception history, the depth frames, the current action ta_t, and the flow time t (alongside learnable register tokens), and projects the action token’s output back to the 2929-D velocity. Training. Distillation is on-policy behavior cloning: the student rolls out in simulation and the frozen teacher relabels every visited state with its action, the regression target of Eq. 13; the teacher itself never acts in the environment (β=0β=0 throughout). At evaluation and deployment the field is integrated with 55 fixed Euler steps. A value critic is trained alongside by standard value regression [33] but contributes no gradient to the student. Training uses AdamW (learning rate 5×10−45×10^-4) for 2525k iterations across 384384 parallel environments; depth randomization is applied throughout. H.4 Depth randomization for sim-to-real Rendered depth is clean, whereas the on-board camera returns holes, invalid regions, self-occlusion from the robot’s body, and slight mount misalignment. To close this gap we add depth randomization during distillation, following [51]. It acts on the preprocessed frame stack of Appendix H.3 (normalized so that +0.5+0.5 is far or missing and −0.5-0.5 near) per minibatch, in two stages. Camera-rotation jitter. Each sample draws one in-plane rotation α∼(−5∘,5∘)α (-5 ,5 ), applied to every frame of that sample’s stack, modeling a fixed but uncertain mount orientation rather than per-frame shake. Random masking. Two families of occlusion are added. Fixed blocks anchored to the bottom corners of the frame model self-occlusion by the robot’s own body; they are drawn per sample and held fixed across the stack, since the body stays in view over the whole window. Random rectangles model transient holes and dropouts; these are drawn independently per frame. Randomization is applied only during training and not during deployment.