Paper deep dive
HiPHI: A Large-Scale Benchmark for High-Precision Human Motion and Object-Interaction
Jiahao Ji, Ji Ma, Runhan Zhang, Runyi Yu, Wenjia Wang, Weiheng Chi, Qianqian Peng, Weichao Yan, Yongfei Gu, Ye Tian, Ting Wu, Longwei Li, Chun Yuan, Ruoli Dai, Lei Han
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/23/2026, 1:42:41 AM
Summary
The paper introduces HiPHI, a large-scale (617.5 hours), high-fidelity human motion and object-interaction dataset designed for humanoid robot learning. Guided by FrameNet semantics, HiPHI captures sub-millimeter precision whole-body motion and synchronized object states (meshes/trajectories) for 245.7 hours of interaction data. It significantly expands motion coverage compared to existing datasets like AMASS and BONES-SEED, providing a scalable foundation for training and evaluating humanoid policies in physically grounded tasks.
Entities (7)
Relation Signals (6)
HiPHI → createdby → Noitom Robotics
confidence 95% · Corresponding authors... Noitom Robotics
HiPHI → usesframework → FrameNet
confidence 95% · HiPHI is theoretically guided by FrameNet, a linguistic framework organizing human primitives.
HiPHI → usestechnology → optical motion capture
confidence 95% · Created using an optical motion capture pipeline, HiPHI provides sub-millimeter spatial marker tracking accuracy
Jiahao Ji → affiliatedwith → National University of Singapore
confidence 90% · Jiahao Ji Affiliation: National University of Singapore
HiPHI → outperforms → AMASS
confidence 90% · Our analyses demonstrate that HiPHI significantly expands motion coverage compared to existing motion datasets
HiPHI → outperforms → BONES-SEED
confidence 90% · HiPHI’s support nearly encloses BONES-SEED... HiPHI covers more grid cells than the closest baseline BONES-SEED
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Humanoid intelligence requires learning over an extremely diverse space of whole-body motions and physically grounded interactions. However, existing embodied datasets remain fundamentally limited: internet-scale video data lack precise physical states and interaction grounding, while laboratory motion datasets provide high fidelity but only narrow behavioral coverage. This mismatch creates a critical bottleneck for scalable humanoid policy learning. We present HiPHI, a 600+ hour scale high-fidelity whole-body human motion dataset designed to systematically maximize coverage of the human motion and interaction manifold. HiPHI is theoretically guided by FrameNet, a linguistic framework organizing human primitives. Created using an optical motion capture pipeline, HiPHI provides sub-millimeter spatial marker tracking accuracy for full-body human motion and mesh-level object trajectories. We further introduce a benchmark suite evaluating motion-space diversity, interaction grounding, object consistency, and physical AI applications. Our analyses demonstrate that HiPHI significantly expands motion coverage compared to existing motion datasets while maintaining high-fidelity interaction quality, and establishes a scalable data foundation for training, evaluating, and generalizing humanoid policies in real-world embodied tasks, where similar extensions are also applicable to motion prior models in computer graphics.
Tags
Links
- Source: https://arxiv.org/abs/2608.16222v1
- Canonical: https://arxiv.org/abs/2608.16222v1
Trouble viewing inline? Open PDF directly →
Full Text
75,293 characters extracted from source content.
Expand or collapse full text
HiPHI: A Large-Scale Benchmark for High-Precision Human Motion and Object-Interaction Jiahao Ji Affiliation: National University of Singapore Ji Ma Affiliation: The University of Hong Kong Runhan Zhang Affiliation: SIGS, Tsinghua University Runyi Yu Affiliation: The Hong Kong University of Science and Technology Wenjia Wang Affiliation: The University of Hong Kong Weiheng Chi Qianqian Peng Weichao Yan Yongfei Gu Ye Tian Ting Wu Longwei Li Chun Yuan Affiliation: SIGS, Tsinghua University Ruoli Dai Affiliation: Corresponding authors.[tristan@noitomrobotics.com], [lhan@noitomrobotics.com] Lei Han Affiliation: Corresponding authors.[tristan@noitomrobotics.com], [lhan@noitomrobotics.com] [4pt] Noitom Robotics [4pt] Equal contribution Abstract Humanoid intelligence requires learning over an extremely diverse space of whole-body motions and physically grounded interactions. However, existing embodied datasets remain fundamentally limited: internet-scale video data lack precise physical states and interaction grounding, while laboratory motion datasets provide high fidelity but only narrow behavioral coverage. This mismatch creates a critical bottleneck for scalable humanoid policy learning. We present HiPHI, a 600+ hour scale high-fidelity whole-body human motion dataset designed to systematically maximize coverage of the human motion and interaction manifold. HiPHI is theoretically guided by FrameNet, a linguistic framework organizing human primitives. Created using an optical motion capture pipeline, HiPHI provides sub-millimeter spatial marker tracking accuracy for full-body human motion and mesh-level object trajectories. We further introduce a benchmark suite evaluating motion-space diversity, interaction grounding, object consistency, and physical AI applications. Our analyses demonstrate that HiPHI significantly expands motion coverage compared to existing motion datasets while maintaining high-fidelity interaction quality, and establishes a scalable data foundation for training, evaluating, and generalizing humanoid policies in real-world embodied tasks, where similar extensions are also applicable to motion prior models in computer graphics. Project page: https://noitom-robotics.github.io/hiphi Figure 1: Overview of HiPHI. A large-scale, high-fidelity dataset for humanoid learning. Keywords: Large-scale Motion Capture, Humanoid Robot Learning, Human-Object Interaction, Reinforcement Learning, Motion Tracking 1 Introduction Humanoid learning requires reference motion that is both broad in coverage and physically faithful. Robot policies must acquire balance, posture transitions, limb coordination, contact, and load-dependent strategies. However, most existing data sources are not designed to capture high-precision human behavior for humanoid learning. Robot demonstrations and teleoperation data are limited by scale and cost, and are tied to specific embodiments [30, 4]. Human videos and egocentric data provide broad behavioral diversity at a large scale, but are limited to visual observations and therefore need to rely on reconstruction or proxy signals rather than directly measured physical states [6]. MoCap and HOI datasets provide accurate motion and sometimes object state, but are primarily designed for motion synthesis, language grounding, reconstruction, or interaction understanding [27, 7, 2, 23]. Humanoid policy learning, therefore, calls for data that are broad, precise, and grounded in object-constrained whole-body control. We formulate human data collection for humanoid learning as a problem of motion-space design. Rather than manually curating behavioral scripts, HiPHI uses FrameNet [1, 36] as a semantic scaffold, where frames represent event types and lexical units (LUs) identify the word senses that evoke them. We select frames and LUs most relevant to embodied intelligence, and use each LU under a specific frame as a seed for performer-facing capture scripts. We then expand each seed along multiple dimensions, including direction, speed, amplitude, body posture, body-part involvement, and object/contact conditions, producing a diverse set of repeatable motion instances for systematic capture. This turns handcrafted script authoring into a structured and scalable process for expanding motion-space coverage. Using this pipeline, we present HiPHI, a 600-hour-scale, high-precision human motion and object-interaction benchmark, all captured from 132 performers using a large optical motion capture (MoCap) system with sub-millimeter spatial tracking accuracy. Specifically, HiPHI combines 371.8 hours of diverse short-horizon whole-body motion and 245.7 hours of real-object interaction data, spanning 214 Frame–LU motion units across 22 frames. The interaction subset covers 40 distinct real-world objects from 12 categories. Each clip is indexed by a Frame–LU label and paired with a natural-language description. For human-object interaction sequences, synchronized object trajectories and meshes are captured together with human motion, making object state an integral part of the motion record. Capturing real objects is essential, as load, friction, inertia, resistance, and contact substantially shape control strategies in ways that pantomime motion cannot reflect. We evaluate HiPHI in terms of motion-space coverage, data quality, interaction consistency, and humanoid policy learning performance. Motion coverage is measured using label-free kinematic embeddings; data quality is assessed through smoothness, ground contact, and interaction geometry; and downstream applications are tested through humanoid motion tracking and real-world deployment. Together, these analyses show that HiPHI is diverse, physically stable, interaction-grounded, and usable as executable reference data in humanoid learning, substantially surpassing existing datasets. Our contributions are threefold: (i) a FrameNet-guided motion-space construction pipeline for systematic and scalable coverage of whole-body motion and interaction; (i) a 617.5-hour, high-precision dataset including 245.7 hours of human–object interaction, synchronized object trajectories and meshes, Frame–LU indexing, and natural-language descriptions; and (i) a comprehensive evaluation protocol for humanoid robot learning, including motion coverage, data quality, interaction consistency, motion tracking, and real-robot deployment. 2 Related Work Robot Teleoperation and Egocentric Vision Data. Embodied intelligence increasingly demands large and diverse data sources. One major category is real-robot data, which avoids the embodiment gap, including teleoperated robot data [30, 14, 40], manipulation benchmarks [12, 16, 28, 21], and low-cost interfaces such as UMI [4]. These data sources avoid gaps in robot embodiment or end-effector configuration, but are often limited by collection cost, data scale, behavioral diversity, and dependence on specific robots, hardware, sensors, and viewpoints. Recently, egocentric visual data have become increasingly popular, as they provide life-scale observations of long-tail human behaviors in the real world [6, 13, 10, 17, 44]. However, such data are typically limited to visual modalities, making it difficult to recover precise human motion and physical interaction states. Human Motion Capture Data. Motion capture data can accurately record human motion, with widely used datasets including AMASS [27], LAFAN1 [9], BABEL/KIT/HumanML3D [35, 34, 7], Motion-X/Motion-X++ [20, 43], and the recent BONES-SEED [3]. These datasets were originally designed mainly for motion synthesis, annotation, generation, or animation, but have recently become widely used in humanoid learning. However, most were not systematically designed for embodied intelligence: few simultaneously provide large scale, high-precision motion, systematic motion-space coverage, and synchronized object states. In contrast, HiPHI is a large-scale dataset specifically designed for embodied intelligence and humanoid learning. It uses the well-established FrameNet theory to structure data collection and directly evaluates motion-space coverage. Human-Object Interaction. Similarly, many HOI datasets have been introduced for motion synthesis, animation, and interaction understanding [37, 2, 11, 22, 18, 26, 23, 41]. However, these datasets are typically small in scale, often only a few hours long, and their contact-level accuracy and interaction diversity remain limited. In contrast, our dataset contains 245.7 hours of HOI data and explicitly emphasizes the physical accuracy of contact-rich interactions. This design supports efficient humanoid learning based on motion imitation methods [31, 33, 32, 39, 24, 45, 8, 19, 25, 42, 38]. 3 FrameNet-Guided Motion-Space Construction Figure 2: From FrameNet to HiPHI data construction. Motion-relevant frames and lexical units define motion seeds, which are expanded through controlled factors such as speed, direction, amplitude, body-part involvement, and object/contact conditions. 3.1 Motion Coverage The most suitable data for humanoid robot learning is not necessarily the full distribution of motions that appear in everyday human life. For example, we do not expect robots to over-learn behaviors such as sleeping, eating, or watching television, which robots do not need to perform. Instead, the data most relevant to humanoid learning should cover control-centric motion patterns, including locomotion, posture transitions, limb coordination, dynamic balance, and responses to external constraints. The goal is therefore not to reproduce the full catalog of human activities, but to cover the motion space that matters for humanoid learning. On the other hand, the human behaviors or tasks in existing motion datasets are often derived from manually designed scripts. These scripts usually describe semantically meaningful daily actions, which makes the resulting motion clips natural and interpretable. However, because the tasks are manually designed at the semantic level, they can lead to substantial overlap in motion space. For example, “raising a hand to wipe sweat from the forehead” and “raising a hand to block sunlight” are two different scripts at the task level, but may produce nearly identical motions. Counting them as two different tasks does not give us two genuinely different motions. The reverse is also true: one simple action such as walking can produce many different whole-body motions when its direction, speed, stride length, turning pattern, or posture changes. A growing list of manually designed scripts therefore provides no clear way to tell which parts of the motion space have already been covered and which are still missing. For humanoid learning, we instead aim to enable robots to span the full reachable human motion space, so that learned policies can better adapt to diverse downstream tasks. This requires a structured set of motion units that can be systematically enumerated and expanded, rather than an open-ended list of task descriptions. Finally, existing datasets often contain many human-object interaction motions without capturing the corresponding object states, leading to, for example, “a person sitting in the air”. In this example, the body trajectory is recorded, but the chair that provides the support surface and contact relation is missing, leaving the physical interaction incomplete. Data for object-constrained humanoid learning should therefore capture the relevant object states together with the human motion. For instance, pushing, carrying, dragging, or leaning can substantially change posture, foot placement, and center-of-mass motion through the object’s geometry, friction, load, and trajectory. These factors are essential for learning physically grounded humanoid behaviors. HiPHI records these objects together with the human body, preserving the complete interaction as physically executable motion reference. 3.2 A FrameNet-Guided Collection Pipeline To systematically address the problem of motion-space coverage, we use FrameNet as a semantic foundation for motion construction. FrameNet organizes word meanings according to the events they describe. A frame represents a type of event, while a lexical unit (LU) represents a word used in one particular meaning within that frame. For example, walk, jog, and run under the Self_motion frame describe related forms of self-propelled movement. This distinction is important because the same word may describe completely different events: “run across the field” refers to body motion, whereas “run a company” does not. A Frame–LU pair identifies the first meaning precisely instead of treating every use of the term “run” as the same motion. This follows the same organizing idea as ImageNet [5], where WordNet [29] provided a structured set of visual concepts to collect: FrameNet provides HiPHI with a structured set of motion meanings to collect. The goal is to avoid relying on manually designed task scripts and instead enumerate the motion space from a theoretically grounded and well-structured system. The action lexical units in FrameNet naturally serve this purpose, making them suitable seeds for generating a broad motion space. Figure 2 shows this complete pipeline from the FrameNet scaffold to the captured HiPHI motion space. We select frames and lexical units that are most relevant to humanoid robots, including those related to body motion, posture change, directional movement, body-part motion, object actuation, and human-object interaction. Starting from these units, we further expand them along dimensions such as path, direction, speed, rhythm, amplitude, body posture, body-part involvement, support relation, and object/contact conditions. For example, a walking seed can be expanded with different routes, directions, speeds, stride lengths, turning patterns, and postures. A pushing seed can vary the object, load, contact point, pushing direction, and object trajectory. These variations change the actual human or object motion, rather than merely changing the story attached to it. These expanded scripts are then provided to motion-capture performers for execution. In this way, each Frame–LU pair maps to a family of capturable motions rather than a single clip. Figure 3: Manual scripting versus FrameNet-guided construction. Manual collection grows by adding scripts one by one, whereas HiPHI organizes collection around Frame-LU motion seeds and expands them through shared factors. This turns dataset growth into a structured and scalable expansion of the motion space. Figure 3 makes the difference from manual script design explicit. In a conventional workflow, scripts are typically written case by case, guided mainly by the designers’ experience and intuition, without an explicit taxonomy or coverage criterion. Dataset growth therefore becomes an ad hoc, trial-and-error process: there is no principled way to determine which motions are still missing, whether a new script genuinely expands the motion space, or whether it merely wraps an already collected motion in a different story. HiPHI instead starts from Frame-LU motion seeds and expands them through a shared set of factors, such as intensity, route, speed, and object conditions. This replaces the blind accumulation of disconnected scripts with a systematic and traceable expansion of motion units and their physical variations. To our knowledge, HiPHI is the first MoCap dataset for robot learning to prospectively adopt such a linguistic scaffold for data construction. 4 Dataset Composition and Statistics Following this construction procedure, HiPHI is created as a LU-indexed motion dataset containing body-only and object-interaction sequences. This section summarizes its scale, relationship to existing datasets, Frame-LU composition, and object-interaction subset. The file organization and metadata are detailed in Appendix B. 4.1 Data Characteristics HiPHI contains 617.5 hours of high-fidelity optical motion capture data in BVH format, corresponding to approximately 200.1 million frames. This is obtained by applying left-right mirroring to 308.7 hours of original motion captured with a high-precision optical MoCap system, following the convention used by BONES-SEED [3]. Human motion is captured at 90 Hz from 132 distinct performers, with detailed statistics in Appendix B. For object-interaction sequences, the data include synchronized human motion, object trajectories, and object meshes, making the object state an integral part of the motion record. Table 1 compares HiPHI with existing representative motion and human-object interaction datasets. Existing data often emphasize either large-scale body motion without object state or smaller-scale object-centric interaction. In contrast, HiPHI combines large-scale high-precision optical MoCap, Frame-LU indexing, synchronized object state, and a benchmark protocol designed for humanoid robot learning. Table 1: Comparison with representative human motion and interaction datasets. HiPHI offers 617.5 hours of MoCap data, including 245.7 hours of object-state-aligned motion, that exceeds representative human motion and interaction datasets in scale and interaction coverage. Dataset Hours Capture Format Motion index Object motion HiPHI 617.5 MoCap BVH Frame-LU 245.7 h AMASS >>40 MoCap SMPL – – BONES-SEED 288.3 MoCap SOMA/G1 NL segments – Motion-X++ (orig.)† 40.4 Video SMPL-X text + pose – LAFAN1 4.6 MoCap BVH action themes – GRAB 3.8 MoCap SMPL-X intent labels 3.8 h OMOMO 9.8 MoCap SMPL-X – 9.8 h HIMO 9.4 MoCap SMPL-X text seg. 9.4 h † Original Motion-X++ data only, where its integrated third-party sources are excluded. NL indicates Natural Language. 4.2 Frame-LU Composition and Long-Tail Coverage Each clip in HiPHI is paired with a Frame-LU label and a natural-language description. The Frame-LU labels also serve as indices for retrieval, sampling, and analysis. The 214 Frame-LU labels across 22 FrameNet frames, placing each motion meaning within its event context rather than relying on ambiguous motion names. As shown in Figure 4, the dataset spans both frequent motion units, such as locomotion, posture changes, and body-part movements, and a long tail of more dynamic, irregular, and constrained patterns. The top 50 Frame-LUs account for 53.7% of the released duration, leaving 46.3% distributed across the remaining long-tail motion units. Motion scripts generated from each LU are performed by approximately 24 different actors on average, and 154 Frame-LUs are performed by at least 10 actors, providing substantial performer variation within individual motion units. Figure 4: Frame-LU composition and long-tail structure. The figure shows duration by FrameNet frame, the Frame-LU duration distribution, and cumulative duration share, illustrating that HiPHI combines common motion units with a broad long tail of whole-body patterns. 4.3 Body-Only and Object-Interaction Motions HiPHI includes 371.8 hours of body-only motion data and 245.7 hours of human-object interaction motion data. The body-only subset covers self-motion, posture changes, body-part movement, dynamic motion, and coupled whole-body coordination. The object-interaction subset records motions shaped by real geometry, contact, load, and object motion, such as sitting, leaning, supporting, pushing, pulling, and carrying. It contains 40 objects across 12 categories, ranging from furniture and containers to cleaning tools and sports equipment, with masses from 0.45 to 6.25 kg. The subset spans 90 Frame-LUs across 15 FrameNet frames. Together, HiPHI provide a large-scale reference base for physically grounded interaction analysis and object-constrained humanoid learning. 5 Experiments In this section, we create the HiPHI benchmark and evaluate HiPHI along two axes: dataset-level coverage and quality (Secs. 5.1, 5.2), and downstream utility through humanoid tracking in simulation and sim-to-real deployment (Secs. 5.3, 5.4). Figure 5: Full kinematic motion-space visualization. All datasets are encoded by the same unsupervised body-motion encoder and projected into one shared t-SNE space. (a) Global projection with balanced sampled points per dataset. (b) Support-envelope comparison between HiPHI and BONES-SEED, the closest large-scale baseline; HiPHI’s support nearly encloses BONES-SEED. (c) Grid-based local-coverage statistics on the same embedding using a 55×5555×55 grid. 5.1 Motion-Space Diversity We use a dataset-balanced protocol to compare kinematic motion-space coverage. All datasets are first mapped to a unified 23-keypoint, 30 FPS, root-aligned, and body-scale-normalized representation, and each sequence is divided into one-second windows. For shared-encoder training, we randomly sample 5,000 clips from each dataset and draw one window from each sampled clip. One exception is LAFAN1 that its full dataset contains fewer clips than 5,000, and thus we use its all available clips for comparison. We then train a shared temporal-convolutional autoencoder using only a reconstruction objective and project its 16-D latent codes into a common t-SNE space. The visualization and occupancy analysis likewise use at most 5,000 windows per dataset (see Appendix E). Figure 5 shows that HiPHI spans the broadest kinematic region among the compared datasets, covering most regions occupied by the other datasets while extending into additional parts of the motion space. We further discretize the embedding and report three statistics (defined in Appendix E): occupied cells (how wide the coverage is), effective occupancy (how uniform the coverage is across occupied cells), and long-tail share (the fraction of samples in globally rare cells). HiPHI covers more grid cells than the closest baseline BONES-SEED (1620 vs. 1438), with larger effective occupancy (1443 vs. 1114) and higher long-tail share (14.1% vs. 10.7%), indicating broader and more uniform coverage rather than denser sampling of common motions. In other words, HiPHI not only reaches more regions of the kinematic space, but also distributes its motions more broadly across those regions, under the same randomly sampled size. We further repeat the coverage analysis across multiple random seeds, t-SNE perplexities, and grid resolutions, and refer the detailed results to Appendix E. HiPHI retains a positive occupied-cell margin over the strongest baseline in every tested setting, confirming that its coverage advantage is consistent across projection and discretization choices. 5.2 Data Quality Body-motion precision. We measure five quantities (lower is better; aggregation defined in Appendix F) that capture common failure modes in humanoid imitation: jerk τJ_τ (m/s3), acceleration A (m/s2), ground penetration δground _ground (m), unsupported-floating share ϕfloat _float (%), and support-point drift νfoot _foot (m/s). ℚτQ_τ and E denote the upper-tail quantile and mean (τ=0.95τ=0.95); ∇t _t is a finite-difference operator divided by the source-specific frame interval Δt t, so that ∇tkx _t^kx has units of m/sk; x¯j x_j and xjx_j are smoothed and raw joint positions; C is the core-body joint set; ℱF is the support-point set; =(t,f):hf(t)−g<ϵK=\(t,f):h_f(t)-g<ε\ is the contact set of (frame, support-point) pairs within ϵ=30ε=30 m of the ground at height g; hfh_f and pfp_f are the height and position of support point f; u(t)u(t) is the unsupported-floating indicator; and Πxy(⋅) _xy(·) is an operator that projects the input onto the ground plane. Formally, we have τ=ℚτ‖∇t3x¯j(t)‖2:j∈,=ℚ1/2‖∇t2xj(t)‖2:j∈,ϕfloat=[u(t)], _τ=Q_τ\! \ \| _t^3 x_j(t) \|_2:j \,\,\,A=Q_1/2\! \ \| _t^2x_j(t) \|_2:j \,\,\, _float=E[u(t)], (1) δground=ℚτmax(g−hf(t),0):f∈ℱ,νfoot=[‖Πxy(∇tpf(t))‖2:(t,f)∈]. _ground=Q_τ\! \ \! (g-h_f(t),0 ):f \, _foot=E\! [ \| _xy ( _tp_f(t) ) \|_2:(t,f) ]. Floor-related metrics (δground _ground, ϕfloat _float, νfoot _foot) require a comparable absolute ground convention. Motion-X++ does not provide one and is therefore excluded from these three metrics in Table 2(a). Table 2: Data quality comparison. (a) Body-motion smoothness and ground-contact metrics, lower is better: jerk τJ_τ, acceleration A, below-ground depth δground _ground, unsupported-floating share ϕfloat _float, and support-point drift νfoot _foot. (b) Human-object geometric consistency, higher is better: non-conflict fraction ηnc _nc and near-surface grounding within 20 cm ρnear _near. (a) Body motion precision Dataset Duration (h) τ J_τ (m/s3) A (m/s2) ground δ_ground (m) ϕfloat φ_float (%) foot ν_foot (m/s) HiPHI 617.5 173.9 10.7 8 0.015 64 AMASS 62.9 529.0 14.9 111 0.76 87 BONES-SEED 288.3 294.2 11.3 18 1.02 86 Motion-X++† 26.0 709.0 23.8 – – – LAFAN1 4.6 383.1 23.4 29 7.22 157 (b) Object interaction geometry Dataset Duration (h) nc η_nc (%) near ρ_near (%) HiPHI 245.7 98.1 95.7 HIMO∗ 21.6 97.6 79.0 OMOMO 9.8 90.6 50.4 HUMOTO‡ 0.81 99.9 82.9 All metrics are aggregated over the full evaluated duration. ∗HIMO contains 9.4 h of unique multi-object sequences. Since each object in a multi-object sequence is evaluated separately, the reported duration becomes 21.6 object-track hours. †Motion-X++ floor metrics require a comparable absolute floor convention and are therefore omitted. ‡HUMOTO is evaluated on its public subset; the full licensed dataset is not directly downloadable as public data. Among datasets sharing the same floor-plane convention, HiPHI achieves the best value on every reported body-motion metric in Table 2(a). These metrics correspond to important physical failure modes in humanoid imitation: τJ_τ measures abrupt motion changes, A captures excessive acceleration, δground _ground indicates ground penetration, ϕfloat _float reflects unsupported floating, and νfoot _foot measures foot sliding during contact. Object-interaction consistency. Two quantities (higher is better) measure whether human motion and object geometry remain physically consistent in a shared frame: non-conflict fraction ηnc _nc (%) and near-surface grounding ρnear _near (%). Let (t)S(t) be a set of points sampled along the human-skeleton segment centerlines (uniformly along each segment), Ω(t) (t) the posed object as a closed 3D region, and d(,∂Ω)d(S,∂ ) the minimum point-to-surface distance from S to the object boundary ∂Ω∂ . Below, [⋅]1[·] is the indicator function (1 if the condition holds, 0 otherwise): ηnc=[[(t)∩Ω(t)=∅]],ρnear=[[d((t),∂Ω(t))<0.20m]]. _nc=E\! [1[\,S(t)∩ (t)= \,] ], _near=E\! [1[\,d(S(t),∂ (t))<0.20\,m\,] ]. (2) Table 2(b) reports ηnc _nc and ρnear _near. HiPHI provides substantially longer object-interaction duration than existing full-body HOI datasets (245.7 h vs. 21.6 h for HIMO and 9.8 h for OMOMO) while maintaining strong geometric consistency. In particular, HiPHI achieves 98.1% non-conflict rate and 95.7% near-surface grounding, substantially improving interaction grounding over HIMO (79.0%) and OMOMO (50.4%). 5.3 Humanoid Learning We evaluate the physical executability of HiPHI through physics-based humanoid tracking. For whole-body motion, we compare HiPHI with AMASS [27], LAFAN1 [9], Motion-X++ [43], and BONES-SEED [3]. All motions are retargeted to the Unitree G1 robot and trained with the same DeepMimic imitation learning pipeline [31]. For motion-with-object tracking, we compare with HUMOTO [23] and OMOMO [18], using Omni-Retarget [42] and BeyondMimic-style [19] to handle synchronized human-object demonstrations. This unified setup allows performance differences to mainly reflect the usability of the motion data. All experiments are conducted from unmirrored original motions. 5.3.1 Whole-Body Motion Tracking Whole-body tracking tests how well each motion source can be physically imitated without the additional complexity of object interaction. Due to the limited usable duration of LAFAN1, we construct a balanced 3-hour subset for each data source and additionally evaluate a 20-hour setting for larger datasets, to ensure a fair comparison independent of HiPHI’s data-scale advantage. For each setting, subsets are sampled using the same random sampling procedure, and all policies are trained under the same optimization budget, with additional details provided in Appendix G. A rollout is considered successful only if its mean full-body position error remains below 0.5 m at every simulation step; otherwise, it is marked as failed. As shown in Figure 6, HiPHI achieves the highest success rates and fastest convergence in both the 3-hour and 20-hour settings. The advantage remains consistent across five independent training runs, with HiPHI maintaining lower failure rates and smaller variance than the compared datasets. These results show that HiPHI provides motion references that are not only larger in scale, but also more directly executable for humanoid policy learning. Figure 6: Whole-body humanoid tracking performance across different motion data sources. (a) Success rates under matched 3-hour and 20-hour data budgets. (b) Training failure-rate curves averaged over five independent runs, with shaded regions indicating variance. HiPHI achieves faster convergence and lower final failure rates across different data scales. 5.3.2 Data Scaling Beyond the matched-budget comparison, we further investigate how the scale of HiPHI affects downstream humanoid tracking. We progressively increase the amount of unmirrored HiPHI training data from 3 to 300 hours and evaluate the resulting policies on AMASS, BONES-SEED, Motion-X++, and LAFAN1. Consistent with the experiments above, we randomly sample 20 hours from each dataset as the evaluation set, except for LAFAN1, for which 3-hour setting is used. All experiments are repeated 10 times, and we report the mean curves with variance bands. As shown in Figure 7, increasing the amount of training data consistently reduces cross-dataset MPJPE, demonstrating that the large scale of HiPHI translates into continuous downstream performance gains. Figure 7: Scaling behavior of HiPHI for humanoid tracking. Increasing the amount of unmirrored HiPHI training data from 3 to 300 hours consistently reduces cross-dataset MPJPE on AMASS, BONES-SEED, Motion-X++, and LAFAN1, demonstrating continuous performance gains with increasing data scale. 5.3.3 Whole-Body Motion Tracking with Object We further evaluate human-object interaction tracking on four categories: kick, carry, push, and lean. All datasets are processed using the same retargeting and physics-based tracking pipeline, allowing us to evaluate how effectively each dataset supports physically grounded human-object imitation. Some categories are unavailable in existing datasets: HUMOTO does not contain push motions, and OMOMO does not provide lean motions. These cases are marked as “N/A” in Table 3. Table 3: Whole-body motion tracking with object comparison (a) Body tracking errors are measured by MPJPE in m and Vel. in m/frame. (b) Object tracking errors are measured by Obj-Pos. in m and Obj-Ori. in degrees. (a) Body tracking errors Dataset Kick Carry Push Lean MPJPE ↓ Vel. ↓ MPJPE ↓ Vel. ↓ MPJPE ↓ Vel. ↓ MPJPE ↓ Vel. ↓ HUMOTO [23] 47.77 4.75 55.37 11.67 N/A N/A 38.83 4.67 OMOMO [18] 85.31 7.05 75.57 6.83 66.09 11.28 N/A N/A HiPHI (ours) 26.31 3.68 50.33 4.14 99.20 6.26 30.76 2.12 (b) Object tracking errors Dataset Kick Carry Push Lean Obj-Pos. ↓ Obj-Ori. ↓ Obj-Pos. ↓ Obj-Ori. ↓ Obj-Pos. ↓ Obj-Ori. ↓ Obj-Pos. ↓ Obj-Ori. ↓ HUMOTO [23] 214.91 35.92 59.20 5.39 N/A N/A 136.71 4.81 OMOMO [18] 128.62 104.28 62.46 4.87 257.82 46.24 N/A N/A HiPHI (ours) 68.17 3.27 89.13 2.64 60.51 2.23 69.16 9.63 We report both body-level and object-level tracking errors. Mean per-joint position error (MPJPE) and velocity error (Vel.) measure humanoid pose and motion tracking quality, while object position error (Obj-Pos.) and object orientation error (Obj-Ori.) measure object trajectory and rotation accuracy. As shown in Table 3, HiPHI achieves the best overall body tracking performance across most categories. For object tracking, HiPHI obtains the best position and orientation accuracy on kick and push, and the best orientation accuracy on carry, indicating better interaction consistency between the humanoid and manipulated objects. Additional experimental details are provided in Appendix H. 5.4 Real-World Evaluation of HiPHI To evaluate the practical usability of HiPHI beyond simulation, we deploy policies trained with HiPHI on Unitree G1. Despite actuation limits, sensing noise, and sim-to-real discrepancies, the robot successfully performs diverse behaviors, including locomotion, sitting, crawling, carrying, flipping, and pulling motions. As shown in Figure 8, these results demonstrate that HiPHI supports stable and physically plausible whole-body and interaction-rich control on real hardware. Figure 8: Real-world deployment of policies trained with HiPHI. The humanoid successfully performs diverse whole-body and object-interaction behaviors, including (a) running, (b) sitting, (c) crawling, (d) carrying a box, (e) flipping, and (f) pulling a suitcase, showing the physical executability of our motion data. 6 Conclusion We introduced HiPHI, a 617.5-hour high-precision optical MoCap dataset and benchmark for humanoid learning, including 245.7 hours of synchronized human-object interaction with object trajectories and meshes. Built upon a Frame-LU-guided motion-space construction pipeline, HiPHI provides a systematic and scalable approach for collecting broad whole-body motion and physically grounded interactions. Through comprehensive evaluations, HiPHI demonstrates broader motion-space coverage, higher motion quality, stronger human-object geometric consistency, and improved downstream humanoid tracking performance. Policies trained with HiPHI achieve strong matched-budget tracking performance, continue to improve with increasing data scale, and successfully transfer to real humanoid hardware across diverse whole-body and object-interaction behaviors. Together, these results establish HiPHI as a large-scale data foundation for learning physically grounded humanoid skills, bridging high-precision motion capture, object-aware interaction, and scalable policy learning. 7 Limitations HiPHI currently captures only single-person motion. Multi-person interaction and human-human contact remain outside its scope and require complementary efforts. In addition, although HiPHI provides high-precision full-body motion and object trajectories, it focuses on kinematics rather than directly measuring contact forces or tactile signals. Finally, the dataset is collected in studio, and capturing accurate motion and object interaction with ego-vision in-the-wild could be a promising future data collection direction. Acknowledgments We sincerely thank Roch Nakajima for generously sharing his extensive experience in motion capture and for his invaluable support in interpreting FrameNet lexical units and assessing their suitability for motion capture. We thank Alex Martinez for developing a practical data visualization tool that greatly facilitated data inspection and quality control. We also thank Baoze Du for his valuable advice on standardizing the BVH skeleton specification. Finally, we thank the many other staff members at Noitom Robotics who operated and maintained the motion-capture facilities and equipment, coordinated data-collection logistics and infrastructure, and supported the performers throughout the capture process. We are also deeply grateful to all performers who contributed their time and effort to the creation of HiPHI. References [1] C. F. Baker, C. J. Fillmore, and J. B. Lowe (1998) The berkeley framenet project. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics - Volume 1, ACL ’98/COLING ’98, USA, p. 86–90. External Links: Link, Document Cited by: §1. [2] B. L. Bhatnagar, X. Xie, I. A. Petrov, C. Sminchisescu, C. Theobalt, and G. Pons-Moll (2022) Behave: dataset and method for tracking human object interactions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 15935–15946. Cited by: §1, §2. [3] Bones Studio (2026) BONES-SEED: Skeletal Everyday Embodiment Dataset. Note: Hugging Face dataset External Links: Link Cited by: §2, §4.1, §5.3. [4] C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song (2024) Universal manipulation interface: in-the-wild robot teaching without in-the-wild robots. arXiv preprint arXiv:2402.10329. Cited by: §1, §2. [5] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei (2009) Imagenet: a large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, p. 248–255. Cited by: §3.2. [6] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al. (2022) Ego4d: around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 18995–19012. Cited by: §1, §2. [7] C. Guo, S. Zou, X. Zuo, S. Wang, W. Ji, X. Li, and L. Cheng (2022) Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 5152–5161. Cited by: §1, §2. [8] L. Han, Q. Zhu, J. Sheng, C. Zhang, T. Li, Y. Zhang, H. Zhang, Y. Liu, C. Zhou, R. Zhao, et al. (2024) Lifelike agility and play in quadrupedal robots using reinforcement learning and generative pre-trained models. Nature Machine Intelligence 6 (7), p. 787–798. Cited by: §2. [9] F. G. Harvey, M. Yurick, D. Nowrouzezahrai, and C. Pal (2020) Robust motion in-betweening. ACM Transactions on Graphics (TOG) 39 (4), p. 60–1. Cited by: §2, §5.3. [10] R. Hoque, P. Huang, D. J. Yoon, M. Sivapurapu, and J. Zhang (2025) Egodex: learning dexterous manipulation from large-scale egocentric video. arXiv preprint arXiv:2505.11709. Cited by: §2. [11] Y. Huang, O. Taheri, M. J. Black, and D. Tzionas (2024) InterCap: joint markerless 3d tracking of humans and objects in interaction from multi-view rgb-d images. International Journal of Computer Vision 132 (7), p. 2551–2566. Cited by: §2. [12] S. James, Z. Ma, D. R. Arrojo, and A. J. Davison (2020) Rlbench: the robot learning benchmark & learning environment. IEEE Robotics and Automation Letters 5 (2), p. 3019–3026. Cited by: §2. [13] S. Kareer, D. Patel, R. Punamiya, P. Mathur, S. Cheng, C. Wang, J. Hoffman, and D. Xu (2025) Egomimic: scaling imitation learning via egocentric video. In 2025 IEEE International Conference on Robotics and Automation (ICRA), p. 13226–13233. Cited by: §2. [14] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, et al. (2024) Droid: a large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945. Cited by: §2. [15] C. M. Kim, B. Yi, H. Choi, Y. Ma, K. Goldberg, and A. Kanazawa (2025) Pyroki: a modular toolkit for robot kinematic optimization. In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 1312–1319. Cited by: Appendix G. [16] C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Martín-Martín, C. Wang, G. Levine, M. Lingelbach, J. Sun, et al. (2023) Behavior-1k: a benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference on Robot Learning, p. 80–93. Cited by: §2. [17] G. Li, Y. Lyu, Z. Liu, C. Hou, J. Zhang, and S. Zhang (2025) H2r: a human-to-robot data augmentation for robot pre-training from videos. arXiv preprint arXiv:2505.11920. Cited by: §2. [18] J. Li, J. Wu, and C. K. Liu (2023) Object motion guided human motion synthesis. ACM Transactions on Graphics (TOG) 42 (6), p. 1–11. Cited by: §2, §5.3, Table 3, Table 3. [19] Q. Liao, T. E. Truong, X. Huang, Y. Gao, G. Tevet, K. Sreenath, and C. K. Liu (2025) Beyondmimic: from motion tracking to versatile humanoid control via guided diffusion. arXiv preprint arXiv:2508.08241. Cited by: Appendix H, §2, §5.3. [20] J. Lin, A. Zeng, S. Lu, Y. Cai, R. Zhang, H. Wang, and L. Zhang (2023) Motion-x: a large-scale 3d expressive whole-body human motion dataset. Advances in Neural Information Processing Systems 36, p. 25268–25280. Cited by: §2. [21] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone (2023) Libero: benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems 36, p. 44776–44791. Cited by: §2. [22] Y. Liu, Y. Liu, C. Jiang, K. Lyu, W. Wan, H. Shen, B. Liang, Z. Fu, H. Wang, and L. Yi (2022) Hoi4d: a 4d egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 21013–21022. Cited by: §2. [23] J. Lu, C. P. Huang, U. Bhattacharya, Q. Huang, and Y. Zhou (2025) HUMOTO: a 4d dataset of mocap human object interactions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 10886–10897. Cited by: §1, §2, §5.3, Table 3, Table 3. [24] Z. Luo, J. Cao, K. Kitani, W. Xu, et al. (2023) Perpetual humanoid control for real-time simulated avatars. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 10895–10904. Cited by: §2. [25] Z. Luo, Y. Yuan, T. Wang, C. Li, S. Chen, F. Castaneda, Z. Cao, J. Li, D. Minor, Q. Ben, et al. (2025) Sonic: supersizing motion tracking for natural humanoid whole-body control. arXiv preprint arXiv:2511.07820. Cited by: §2. [26] X. Lv, L. Xu, Y. Yan, X. Jin, C. Xu, S. Wu, Y. Liu, L. Li, M. Bi, W. Zeng, et al. (2024) Himo: a new benchmark for full-body human interacting with multiple objects. In European Conference on Computer Vision, p. 300–318. Cited by: §2. [27] N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black (2019) AMASS: archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF international conference on computer vision, p. 5442–5451. Cited by: §1, §2, §5.3. [28] O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard (2022) Calvin: a benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters 7 (3), p. 7327–7334. Cited by: §2. [29] G. A. Miller (1994) WordNet: a lexical database for English. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994, External Links: Link Cited by: §3.2. [30] A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, et al. (2024) Open x-embodiment: robotic learning datasets and rt-x models: open x-embodiment collaboration 0. In 2024 IEEE International Conference on Robotics and Automation (ICRA), p. 6892–6903. Cited by: §1, §2. [31] X. B. Peng, P. Abbeel, S. Levine, and M. Van de Panne (2018) Deepmimic: example-guided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG) 37 (4), p. 1–14. Cited by: §2, §5.3. [32] X. B. Peng, Y. Guo, L. Halper, S. Levine, and S. Fidler (2022) Ase: large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Transactions On Graphics (TOG) 41 (4), p. 1–17. Cited by: §2. [33] X. B. Peng, Z. Ma, P. Abbeel, S. Levine, and A. Kanazawa (2021) Amp: adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics (ToG) 40 (4), p. 1–20. Cited by: §2. [34] M. Plappert, C. Mandery, and T. Asfour (2016) The kit motion-language dataset. Big data 4 (4), p. 236–252. Cited by: §2. [35] A. R. Punnakkal, A. Chandrasekaran, N. Athanasiou, A. Quiros-Ramirez, and M. J. Black (2021) BABEL: bodies, action and behavior with english labels. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 722–731. Cited by: §2. [36] J. Ruppenhofer, M. Ellsworth, M. Schwarzer-Petruck, C. R. Johnson, and J. Scheffczyk (2016) FrameNet i: extended theory and practice. Technical report International Computer Science Institute. Cited by: §1. [37] O. Taheri, N. Ghorbani, M. J. Black, and D. Tzionas (2020) GRAB: a dataset of whole-body human grasping of objects. In European conference on computer vision, p. 581–600. Cited by: §2. [38] C. Tessler*, Y. Jiang*, X. B. Peng, E. Coumans, Y. Shi, H. Zhang, D. Rempe, G. Chechik†, and S. Fidler† (2025) ProtoMotions3: an open-source framework for humanoid simulation and control. GitHub. Note: https://github.com/NVLabs/ProtoMotions/ Cited by: §2. [39] N. Wagener, A. Kolobov, F. Vieira Frujeri, R. Loynd, C. Cheng, and M. Hausknecht (2022) Mocapact: a multi-task dataset for simulated humanoid control. Advances in Neural Information Processing Systems 35, p. 35418–35431. Cited by: §2. [40] H. R. Walke, K. Black, T. Z. Zhao, Q. Vuong, C. Zheng, P. Hansen-Estruch, A. W. He, V. Myers, M. J. Kim, M. Du, et al. (2023) Bridgedata v2: a dataset for robot learning at scale. In Conference on Robot Learning, p. 1723–1736. Cited by: §2. [41] S. Xu, D. Li, Y. Zhang, X. Xu, Q. Long, Z. Wang, Y. Lu, S. Dong, H. Jiang, A. Gupta, et al. (2025) Interact: advancing large-scale versatile 3d human-object interaction generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 7048–7060. Cited by: §2. [42] L. Yang, X. Huang, Z. Wu, A. Kanazawa, P. Abbeel, C. Sferrazza, C. K. Liu, R. Duan, and G. Shi (2025) Omniretarget: interaction-preserving data generation for humanoid whole-body loco-manipulation and scene interaction. arXiv preprint arXiv:2509.26633. Cited by: Appendix H, §2, §5.3. [43] Y. Zhang, J. Lin, A. Zeng, G. Wu, S. Lu, Y. Fu, Y. Cai, R. Zhang, H. Wang, and L. Zhang (2025) Motion-x++: a large-scale multimodal 3d whole-body human motion dataset. arXiv preprint arXiv:2501.05098. Cited by: §2, §5.3. [44] L. Y. Zhu, P. Kuppili, R. Punamiya, P. Aphiwetsa, D. Patel, S. Kareer, S. Ha, and D. Xu (2026) Emma: scaling mobile manipulation via egocentric human data. IEEE Robotics and Automation Letters. Cited by: §2. [45] Q. Zhu, H. Zhang, M. Lan, and L. Han (2023) Neural categorical priors for physics-based character control. ACM Transactions on Graphics (TOG). Cited by: §2. Appendix The appendix provides additional details that complement the main text. Appendix A-B describe FrameNet-guided motion-unit construction and the release composition, metadata, and demographics. Appendix C states the release license and ethics. Appendix D shows representative sample sequences. Appendix E-F provide the unsupervised motion-space embedding setup and the full-run quality metrics. Appendix G-H document training details for whole-body and motion-with-object tracking. Appendix A Additional Details for Dataset Construction FrameNet-guided motion units. HiPHI uses FrameNet as a semantic scaffold rather than a closed action taxonomy. We identify motion-relevant frames and lexical units (LUs), and use each Frame-LU pair as a seed for a motion unit that can be translated into performer-facing capture instructions. This keeps the public index compact while allowing each semantic seed to expand into multiple concrete realizations. Table 4: Examples of FrameNet-guided motion-unit construction. Frame-LU seeds are expanded by controlled realization factors rather than treated as single fixed actions. Motion family Frame-LU examples Realization factors Locomotion and direction Self_motion: walk, run, jog, stride path, direction, speed, turning radius, step rhythm Posture and support Change_posture, Posture: crawl, kneel, lean body height, support relation, amplitude, transition speed Body-part motion Body_movement: bend, toss, shake, clap body part, range of motion, rhythm, unilateral/bilateral pattern Object actuation and transfer Cause_motion, Bringing: push, pull, lift, carry object category, contact mode, load condition, object trajectory Object-local dynamics Cause_to_move_in_place: rotate, shake, swing axis of motion, contact point, amplitude, repetition pattern Performer-facing realization protocol. Each selected Frame-LU seed is converted into short, repeatable capture instructions by varying factors that are directly observable during MoCap: direction, path, speed, intensity, amplitude, body height, body-part involvement, object category, contact mode, and load condition. For instance, a push seed is realized with different object categories, loads, and directions; a lean seed varies support surface and body height; and a crawl seed varies path shape, speed, and limb coordination. This protocol turns a compact semantic index into a broader set of controllable whole-body motion realizations. Appendix B Additional Details for Dataset Composition and Metadata Release duration convention. All release-scale durations in the paper follow the public mirrored-release convention, where left-right mirrored sequences are included as normal released data. Under this convention, HiPHI contains 617.5 hours of motion, corresponding to approximately 200.1M frames at 90 Hz. The release contains 371.8 hours of body-only motion and 245.7 hours of strict object-interaction motion, where strict object-interaction denotes explicit physical interaction with a tracked object. Table 5: Release-level composition used throughout the paper. Durations follow the mirrored-release convention. Statistic Value Total release duration 617.5 h Total frames 200.1M Capture frame rate 90 Hz Total performers 132 Body-only duration 371.8 h Human object-interaction duration 245.7 h Human object-interaction share 39.8% FrameNet frames 22 Frame-LU labels 214 Median actors per Frame-LU 24 Frame-LUs with at least 10 actors 154 Object categories in strict object-interaction subset 12 Frame-LU composition. The released semantic layer contains 22 FrameNet frames and 214 Frame-LU motion-unit labels. Its duration distribution follows a common-to-long-tail structure: the top 10 Frame-LUs account for 20.4% of duration, the top 20 for 31.8%, and the top 50 for 53.7%. Thus, nearly half of the release duration remains outside the top 50 Frame-LUs, supporting retrieval and analysis beyond a small set of frequent actions. Performer demographics. HiPHI contains 132 anonymized performer profiles, including 76 male and 56 female profiles, with heights from 155 to 185 cm and weights from 40 to 82 kg.Figure 9 summarizes the distribution. Figure 9: Performer demographics of HiPHI. (a) Gender split among 132 performers. (b) Height distribution. (c) Weight distribution. Table 6: Metadata fields exposed by the HiPHI release. Fields are provided through the global motion manifest, performer metadata, and per-package metadata.json files. Field Scope Meaning Global motion metadata motion_id Global / package Unique motion identifier. Mirrored motions use the corresponding ID with a __mirror suffix. frame Global / package FrameNet frame label. lu Global / package FrameNet lexical-unit label. frame_lu Global / package Frame-LU motion-unit label. duration_sec Global / package Temporal duration of the motion, in seconds. frame_count Global / package Number of motion frames in the sequence. actor_id Global / package Anonymized performer identifier in the release. text_annotation Global / package Natural-language annotation aligned with the motion sequence. is_hoi Global / package Whether the sequence contains tracked human-object interaction. object_categories Global Object-category information associated with the sequence, when applicable. mirrored Global / package Boolean indicating whether the motion is the left-right mirrored counterpart of an original sequence. Performer metadata actor_id Actor metadata Anonymized performer identifier used to link performer metadata to motion sequences. height_cm Actor metadata Performer height in centimeters. weight_kg Actor metadata Performer weight in kilograms. gender Actor metadata Performer gender metadata. Additional per-package metadata dataset Package only Dataset identifier stored in the package metadata. fps Package only Frame rate associated with the released motion package. actor_metadata Package only Nested performer metadata associated with the sequence. objects Package only List of tracked objects associated with the sequence; empty for body-only motion. objects[].object_id Package only Identifier of a tracked object instance. objects[].object_category Package only Semantic category of the tracked object. objects[].trajectory_path Package only Package-relative path to the synchronized object trajectory. objects[].mesh_id Package only Identifier of the corresponding object mesh. objects[].mesh_path Package only Package-relative path to the corresponding object mesh. Object-interaction metadata. The HiPHI object-interaction subset contains 40 real-world objects from 12 categories, covering a mass range of 0.45-6.25 kg and spanning 15 FrameNet frames and 90 Frame-LU labels (Table 7). For strict object-interaction sequences, the release provides synchronized object trajectories and object meshes together with the human BVH motion. Table 7: Object statistics of the HiPHI human-object interaction subset. The subset contains diverse real-world objects with synchronized human motion, object trajectories, and object meshes. Statistic Value Object instances 40 Object categories 12 Mass range 0.45-6.25 kg Covered FrameNet frames 15 Covered Frame-LU labels 90 Interaction duration 245.7 h Object state modalities Trajectory + Mesh Appendix C Release Terms, License, and Ethics License. HiPHI will be released under the ModalityNet Open Research License v1.0, a custom non-commercial license for scientific research, education, and evaluation. The complete license terms will accompany the public release of HiPHI. Release schedule. The full dataset, including BVH motions, object trajectories, object meshes, and the Frame-LU index, will be released publicly soon. The supplementary material contains four representative sample sequences (three body-only and three object-interaction) that illustrate the release format and quality. Ethics. Data collection was reviewed and approved by an institutional ethics review board before capture. All performers were adults who provided written informed consent describing the capture protocol, the intended research use, and the planned public release of the motion data and anonymized demographic attributes. Performers were compensated for each capture session and may request withdrawal of their data at any time. During capture, RGB video and audio were recorded as references for production review; the public release of HiPHI contains only BVH motion, object trajectories, and object meshes, and does not include any RGB video, audio, facial imagery, or voice recordings. Performer identities are released only as numeric actor identifiers, with anonymized height, weight, and gender attributes. Specific institutional and protocol identifiers are withheld for double-blind review and will be provided in the camera-ready version. Appendix D Representative Sample Sequences We show six representative sequences from HiPHI. Per-sequence renderings are shown in Figure 10, and full motion playback is provided in the supplementary video. Figure 10: Representative sample sequences from HiPHI. We show six representative sequences from HiPHI: (a)-(c) body-only motion sequences and (d)-(f) object-interaction motion sequences. Appendix E Additional Details for Motion-Space Diversity Unified motion representation. For the motion-space analysis in Sec. 5.1, all datasets are first mapped to a common body representation with 23 keypoints in meters, resampled to 30 FPS in a z-up coordinate frame. The keypoints are pelvis, spine, chest, neck, head, left/right hip, knee, ankle, foot, toe, shoulder, elbow, wrist, and hand. Each sequence is divided into 30-frame windows with a stride of 25 frames. The per-frame input contains heading-local, root-relative joint positions and root-relative joint velocities. We further apply body-scale normalization and subtract the per-window median pose, which reduces scale and skeleton-layout differences across datasets. The resulting feature dimension is 23×3×2=13823× 3× 2=138 per frame. Dataset-balanced sampling. To prevent dataset size from dominating either representation learning or coverage evaluation, we construct a balanced sample for each dataset. We randomly sample 5,000 motion clips from each dataset when available and draw one valid 30-frame window from each selected clip, yielding 5,000 training windows per dataset. LAFAN1 uses all available clips because it contains fewer than 5,000 samples. The t-SNE projection and grid-based occupancy analysis follow the same cap of at most 5,000 windows per dataset. Therefore, both the shared representation and the reported coverage statistics are dataset-balanced rather than proportional to raw dataset size. Shared unsupervised encoder. We train one temporal convolutional autoencoder jointly on the balanced samples from all datasets and use its encoder as the motion descriptor. The model takes a tensor of size 30×13830× 138, uses temporal Conv1D layers with residual dilated blocks, and outputs a 16-D latent code. It is trained only with reconstruction loss; dataset identity, text labels, Frame-LU labels, and object labels are not used. The default analysis uses the epoch-20 checkpoint. Before t-SNE, latent features are standardized and reduced to 5 dimensions with PCA. We then run t-SNE with Euclidean distance, perplexity 30, PCA initialization, automatic learning rate, 1000 iterations, and random seed 42. Table 8: Embedding setup for the motion-space coverage analysis. Item Setting Keypoints 23 body keypoints, meter scale, z-up Frame rate 30 FPS Window / stride 30 frames / 25 frames Feature type Heading-local normalized joint dynamics and relative velocities Feature dimension 138 per frame Balanced sampling Up to 5,000 clips and 5,000 windows per dataset Encoder Shared unsupervised temporal Conv1D autoencoder Latent dimension 16 Objective Reconstruction MSE Checkpoint Epoch 20 t-SNE PCA dim. 5, perplexity 30, PCA init., auto learning rate, 1000 iterations, seed 42 Grid statistics 55×5555× 55 grid on the default 2-D embedding Grid-based coverage statistics. Let G be the 55×5555× 55 grid in the shared t-SNE plane and nD(c)n_D(c) the number of sampled points from dataset D in cell c∈c . Let D+=c∈:nD(c)>0G_D^+=\c :n_D(c)>0\ be the cells occupied by D. We report occupied grid cells, entropy-based effective grid cells, and long-tail share: mD m_D =|D+|,pD(c)=nD(c)∑c′∈nD(c′), = _D^+ , p_D(c)= n_D(c) _c n_D(c ), (3) eD e_D =exp(−∑c∈D+pD(c)logpD(c)),ℓD=∑c∈pD(c). = \! (- _c _D^+p_D(c) p_D(c) ), _D= _c p_D(c). Here, mDm_D is the number of occupied cells, measuring how broadly a dataset spans the shared motion space. The effective occupancy eDe_D measures how evenly its samples are distributed across the occupied cells. The long-tail share ℓD _D measures the fraction of samples located in globally rare regions. We use the convention 0log0=00 0=0. The set ⊆T contains non-empty cells whose global occupancy is at or below the 25th percentile of all non-empty global cells; under the default 55×5555× 55 setting, this threshold corresponds to 7 points per cell. Coverage robustness analysis. We further test whether the occupied-region advantage depends on a particular random seed, t-SNE perplexity, or grid resolution. Starting from the default configuration in Table 8, we repeat the analysis using random seeds 55, 73, and 112; perplexities 40, 55, and 65; and grid resolutions of 55×5555× 55, 88×8888× 88, and 111×111111× 111. For every configuration, we report the occupied-cell margin ΔOcc=mHiPHI−maxD≠HiPHImD, =m_HiPHI- _D m_D, (4) where the second term is the occupied-cell count of the strongest baseline under the same configuration. A positive value means that HiPHI covers more cells than every compared dataset. Because occupied-cell counts naturally change with grid resolution, the margin is reported separately for each resolution. As shown in Figure 11, HiPHI maintains a positive occupied-cell margin in every tested configuration. The coverage advantage therefore remains consistent across resampling, projection, and discretization choices. Figure 11: Robustness of motion-space coverage. We repeat the coverage analysis using different random seeds, t-SNE perplexities, and grid resolutions. Each entry reports the occupied-cell margin between HiPHI and the strongest baseline under the corresponding setting. HiPHI maintains a positive margin across all tested configurations. Appendix F Additional Details for Data Quality and Precision Aggregation across sequences. All quality statistics in Sec. 5.2 are aggregated over the full evaluated duration of each dataset, so that every recorded second contributes equally and dataset size does not bias the result. Let rkr_k index any observation at the relevant level (frame, window, or sequence aggregate) with non-negative duration weight wkw_k (i.e., wk=Δtw_k= t of the source for per-frame observations, the window length for per-window observations, and the sequence duration TiT_i for per-sequence aggregates). We report [r]=∑kwkrk∑kwk,ℚτ(r)=infa:∑kwk 1[rk≤a]∑kwk≥τ,E[r]= _kw_k\,r_k _kw_k, _τ(r)= \! \a: _kw_k\,1[r_k≤ a] _kw_k≥τ \, (5) i.e., the time-weighted mean and quantile, with τ=0.95τ=0.95 for upper-tail statistics. For ratio-type metrics (ηnc _nc, ρnear _near, ϕfloat _float), each rir_i is itself a frame-level ratio inside sequence i, and weighting by TiT_i recovers the frame-level aggregation across the dataset. For distribution-type metrics (τJ_τ, A, δground _ground, νfoot _foot), rkr_k is the per-frame (or per-pair) observation entering the corresponding equation in Sec. 5.2. Body-motion metric sets and smoothing. Equation 1 uses one joint set, one point set, and one pair set. C is the core-body joint set, excluding fingers and face joints when present. ℱF is the support-point set: foot, toe, and ankle proxies when available. =(t,f):hf(t)−g<ϵK=\(t,f):h_f(t)-g<ε\ is the contact set of (frame, support-point) pairs within ϵ=30ε=30 m of the ground at height g; speed is then used to measure drift over those pairs. The finite-difference operator ∇t _t is divided by the source-specific frame interval Δt t, so ∇tkx _t^kx has units of m/sk. For jerk, trajectories are smoothed by a fixed physical window of 1/61/6 s, implemented as an odd-length moving average with a minimum size of three frames; raw positions are used for acceleration. Floor-related metrics require a comparable absolute ground convention, which is why Motion-X++ is excluded from δground _ground, ϕfloat _float, and νfoot _foot in Table 2. Object-interaction geometry. Equation 2 uses a sparse skeleton-point proxy. The human body is represented by canonical skeleton segments covering hands, arms, legs, torso, pelvis, and head/neck regions. For each evaluated frame, segment centerlines are uniformly sampled (32 points per segment in the released code) to produce (t)S(t). The object is represented as a closed 3D region Ω(t) (t) obtained by posing the released mesh; ∂Ω(t)∂ (t) is its boundary surface. Distances d(,∂Ω)d(S,∂ ) are computed in the shared coordinate frame. HUMOTO is evaluated on its publicly released GLB subset; the full licensed dataset is not directly downloadable as public data. Table 9: Summary of quality metrics used in Sec. 5.2. Symbol Meaning Unit Better τJ_τ Upper-tail fixed-window jerk over core-body joints m/s3 lower A Median core-body acceleration m/s2 lower δground _ground Upper-tail below-ground support-point depth m lower ϕfloat _float Unsupported-floating duration share % lower νfoot _foot Support-point drift over contact frames m/s lower ηnc _nc Non-conflict fraction under skeleton-point / object-volume proxy % higher ρnear _near Near-surface fraction within 20 cm of the object surface % higher Applicability. The body-motion smoothness metrics require only temporally aligned human joint trajectories and native timing. Ground-contact metrics additionally require a comparable up-axis and absolute ground convention. Object-interaction metrics require synchronized human motion, object pose, and object mesh in a shared coordinate frame. HUMOTO is evaluated on its publicly released GLB subset; the full licensed dataset is not directly downloadable as public data. Appendix G Training Details for Whole-Body Motion Tracking For whole-body motion tracking, we evaluate all datasets under the same physics-based imitation learning pipeline. All source motions are first retargeted to the Unitree G1 humanoid using PyRoki [15], and the resulting robot motions are then used as reference trajectories for policy learning. We use the same observation design, reward formulation, optimization budget, and evaluation protocol for all data sources so that performance differences mainly reflect the physical usability of the underlying motion data. Each tracking experiment is repeated five times with different random seeds. We report the mean curve and variance across runs. The detailed observation and reward are summarized in Tables 10 and 11. Figure 12 shows qualitative whole-body tracking results of HiPHI on the Unitree G1 humanoid. The learned policy produces stable and physically plausible motions across diverse behaviors, demonstrating that the motions in HiPHI can serve as effective executable references for humanoid control. Table 10: Observation terms for whole-body motion tracking. State Dim. (a) Motion Command Reference Body Position (heading frame) 9999 Reference Body Position (relative) 9999 Reference Body Orientation (heading frame) 198198 Reference Body Orientation (relative) 198198 Reference Body Linear Velocity (relative) 9999 Reference Body Angular Velocity (relative) 9999 (b) Proprioceptive State Root Height (above terrain) 11 Body Position (non-root, heading frame) 9696 Body Orientation (heading frame, tan-norm) 198198 Body Linear Velocity (heading frame) 9999 Body Angular Velocity (heading frame) 9999 Last Action 2929 Figure 12: Qualitative visualization of diverse humanoid motion skills trained with HiPHI. The motions cover diverse whole-body behaviors, including standing, walking, sitting, lying down, crawling, kneeling, and recovery motions, demonstrating the coverage and diversity of the training data. Table 11: Reward terms used for whole-body motion tracking. Term Expression Weight Remarks Tracking Rewards Global Translation (gt) exp(cgt1nb∑i‖pi−p^i‖22) \! (c_gt\, 1n_b _i\|p_i- p_i\|_2^2 ) 0.50.5 cgt=−25c_gt=-25 Global Rotation (gr) exp(cgr1nb∑idR(qi,q^i)2) \! (c_gr\, 1n_b _id_R(q_i, q_i)^2 ) 0.30.3 cgr=−5c_gr=-5 Global Linear Velocity (gv) exp(cgv1nb∑i‖vi−v^i‖22) \! (c_gv\, 1n_b _i\|v_i- v_i\|_2^2 ) 0.10.1 cgv=−0.5c_gv=-0.5 Global Angular Velocity (gav) exp(cgav1nb∑i‖ωi−ω^i‖22) \! (c_gav\, 1n_b _i\| _i- ω_i\|_2^2 ) 0.20.2 cgav=−0.1c_gav=-0.1 Root Height (rh) exp(crh(zroot−z^root)2) \! (c_rh\,(z_root- z_root)^2 ) 0.20.2 crh=−100c_rh=-100 Regularization and Safety Penalties Action Smoothness ‖a~t−a~t−1‖2\| a_t- a_t-1\|_2 −0.02-0.02 – Power Consumption ∑j|τjq˙j| _j| _j\, q_j| −10−5-10^-5 – Here dR(⋅,⋅)d_R(·,·) denotes quaternion orientation error, nbn_b is the number of tracked body links. Appendix H Training Details for Motion-with-Object Tracking For motion-with-object tracking, we first process synchronized human-object demonstrations using Omni-Retarget [42], which retargets human motion to the Unitree G1 humanoid while preserving the corresponding object trajectories. Because existing open-source human-object motion datasets usually do not specify object physical parameters, we use a unified object configuration across datasets for fair comparison. Unless otherwise specified, each object is modeled as a free-joint mesh object with a total mass of 2.02.0 kg. We then use a BeyondMimic-style tracking setup [19], following a similar observation design and reward formulation while extending the framework to explicitly model object interactions. Since the task requires the humanoid to reproduce both whole-body motion and the associated object trajectory, we additionally include the current object state in the policy observation. We also introduce object-tracking rewards by measuring the position and orientation errors between the simulated object and its reference trajectory. To improve policy robustness, we apply domain randomization to both the humanoid and the object, including object mass, friction, inertial properties, contact parameters, and external perturbations. The detailed observation, reward, and randomization settings are summarized in Tables 12, 13, and 14. Table 12: Observation terms for motion-with-object tracking. State Dim. Policy Critic (a) Motion Command Reference Joint Pos 2929 ✓ ✓ Reference Joint Vel 2929 ✓ ✓ Reference Anchor Position 33 ✓ ✓ Reference Anchor Orientation 66 ✓ ✓ (b) Proprioceptive State Base Linear Velocity 33 ✗ ✓ Base Angular Velocity 33 ✓ ✓ Joint Position 2929 ✓ ✓ Joint Velocity 2929 ✓ ✓ Last Action 2929 ✓ ✓ (c) Object State Object position in robot frame 33 ✓ ✓ Object orientation in robot frame 66 ✓ ✓ (d) Critic-Only Privileged State Body position 14×314× 3 ✗ ✓ Body orientation 14×614× 6 ✗ ✓ Table 13: Reward terms used for motion-with-object tracking. Term Expression Weight Remarks Tracking Rewards Anchor position exp(−∥pa−p^a∥22/σ2) \! (-\|p_a- p_a\|_2^2/σ^2 ) 0.50.5 σ=0.3σ=0.3 Anchor orientation exp(−dR(qa,q^a)2/σ2) \! (-d_R(q_a, q_a)^2/σ^2 ) 0.50.5 σ=0.4σ=0.4 Body position exp(−1nb∑i∥pi−p^i∥22/σ2) \! (- 1n_b _i\|p_i- p_i\|_2^2/σ^2 ) 1.01.0 σ=0.3σ=0.3 Body orientation exp(−1nb∑idR(qi,q^i)2/σ2) \! (- 1n_b _id_R(q_i, q_i)^2/σ^2 ) 1.01.0 σ=0.3σ=0.3 Body linear velocity exp(−1nb∑i∥vi−v^i∥22/σ2) \! (- 1n_b _i\|v_i- v_i\|_2^2/σ^2 ) 0.50.5 σ=1.0σ=1.0 Body angular velocity exp(−1nb∑i∥ωi−ω^i∥22/σ2) \! (- 1n_b _i\| _i- ω_i\|_2^2/σ^2 ) 0.50.5 σ=3.14σ=3.14 Object position exp(−∥po−p^o∥22/σ2) \! (-\|p_o- p_o\|_2^2/σ^2 ) 1.01.0 σ=0.3σ=0.3 Object orientation exp(−dR(qo,q^o)2/σ2) \! (-d_R(q_o, q_o)^2/σ^2 ) 1.01.0 σ=0.3σ=0.3 Regularization and Safety Penalties Action rate ‖at−at−1‖22\|a_t-a_t-1\|_2^2 −0.1-0.1 Smooth action penalty Joint limit (q∉[qmin,qmax])1(q∉[q_ ,q_ ]) −10.0-10.0 Joint limit violation Here dR(⋅,⋅)d_R(·,·) denotes quaternion orientation error, nbn_b is the number of tracked body links, c and c c denote measured and reference contact states, and hatted variables denote reference motion states. Table 14: Domain randomization and disturbance settings for motion-with-object tracking. Term Value External Disturbances Push robot interval =1=1–33s, vx,vy∼U[−0.5,0.5]v_x,v_y U[-0.5,0.5] m/s, vz∼U[−0.2,0.2]v_z U[-0.2,0.2] m/s Robot Dynamics Randomization Torso COM offset x∼U[−0.025,0.025]x U[-0.025,0.025] m, y,z∼U[−0.05,0.05]y,z U[-0.05,0.05] m Encoder bias U[−0.01,0.01]U[-0.01,0.01] Default joint position offset U[−0.01,0.01]U[-0.01,0.01] rad Rigid-body material static friction U[0.3,1.6]U[0.3,1.6], dynamic friction U[0.3,1.2]U[0.3,1.2], restitution U[0.0,0.5]U[0.0,0.5] Object Dynamics Randomization Object mass U[0.3,2.0]×U[0.3,2.0]× default mass Object COM offset U[−0.02,0.02]U[-0.02,0.02] m Figure 13 presents qualitative comparisons of motion-with-object tracking across three datasets: HiPHI, OMOMO, and HUMOTO. We visualize two representative interaction categories, including carry (first row) and kick interactions (bottom row). Existing datasets exhibit limited interaction coverage and tracking robustness in certain categories. For example, HUMOTO fails to reproduce stable object-aware behaviors in the carry scenario, while OMOMO suffers from reduced interaction fidelity. In contrast, HiPHI produces more stable whole-body motions and more consistent human-object interaction behaviors across diverse tasks. Figure 13: Qualitative comparisons of motion-with-object tracking across different datasets. The top row shows carry interactions, while the bottom row presents kick interaction behaviors. HiPHI demonstrates more stable whole-body tracking and better human-object interaction consistency across diverse tasks.