Paper deep dive
Weave: Learning Whole-Body Dexterous Loco-Manipulation from Human-Object Interactions
Liu Cao, Xingze Wu, Jingzhi Cui, Botian Xu, Mingzhi Pei, Ruoqu Chen, Mengdi Xu
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Learning humanoid-object interaction requires coordinating whole-body balance, locomotion, and dexterous hand contact to control both robot and object motion. Human demonstrations provide examples of coordinated interaction, but transferring these behaviors to humanoid robots requires learning how to establish and maintain effective contacts under different embodiments and dynamics. We present Weave, a unified framework for learning whole-body dexterous humanoid-object interaction from captured human demonstrations. Weave first converts captured human-object interactions into executable robot-object references through contact-aware retargeting and approach-motion completion. At its core is a contact- and geometry-aware policy that jointly commands 29 body joints and 12 actuated finger joints across multiple objects and interaction sequences. Evaluation across nine objects yields a 92.5% success rate on trained interactions and, without any additional training, 65.0% on sequences never seen during training. We additionally release ~9,000 physically executed rollouts spanning ~23 hours, providing robot-object trajectories with contact annotations for downstream interaction-policy learning and physically consistent HOI motion generation. Project website: this https URL
Tags
Links
- Source: https://arxiv.org/abs/2609.16683v1
- Canonical: https://arxiv.org/abs/2609.16683v1
Trouble viewing inline? Open PDF directly →
Full Text
50,386 characters extracted from source content.
Expand or collapse full text
Weave: Learning Whole-Body Dexterous Loco-Manipulation from Human–Object Interactions Liu Cao1,† , Xingze Wu3,† , Jingzhi Cui1, Botian Xu4, Mingzhi Pei2, Ruoqu Chen1,† , Mengdi Xu1 Affiliation: 1Tsinghua IIIS 2Tsinghua College AI 3Dalian University of Technology 4The Chinese University of Hong Kong Abstract Learning humanoid–object interaction requires coordinating whole-body balance, locomotion, and dexterous hand contact to control both robot and object motion. Human demonstrations provide examples of coordinated interaction, but transferring these behaviors to humanoid robots requires learning how to establish and maintain effective contacts under different embodiments and dynamics. We present Weave, a unified framework for learning whole-body dexterous humanoid–object interaction from captured human demonstrations. Weave first converts captured human–object interactions into executable robot–object references through contact-aware retargeting and approach-motion completion. At its core is a contact- and geometry-aware policy that jointly commands 2929 body joints and 1212 actuated finger joints across multiple objects and interaction sequences. Evaluation across nine objects yields a 92.5%92.5\% success rate on trained interactions and, without any additional training, 65.0%65.0\% on sequences never seen during training. We additionally release ∼9,000 9,000 physically executed rollouts spanning ∼ 23 hours, providing robot–object trajectories with contact annotations for downstream interaction-policy learning and physically consistent HOI motion generation. Our project website: https://xiaohu-art.github.io/Weave/. Figure 1: Policy rollout examples with Weave. We learn a unified policy for whole-body dexterous loco-manipulation from captured human–object interaction. Parallel simulation rollouts showcase the one policy coordinating locomotion and dexterous manipulation across diverse everyday objects. † partially done at CocoMatrix. I Introduction Humanoid robots are expected to operate in environments built for humans, where many everyday tasks require the robot to manipulate objects—approaching, grasping, and transporting them while maintaining balance. Such contact-rich loco-manipulation is a core capability for general-purpose humanoids. Even a task as simple as moving a chair requires the robot to establish a stable multi-finger grasp, generate sufficient contact forces, and continuously adapt its whole-body posture to the resulting load. However, learning such behaviors with a unified policy remains challenging, as balance, locomotion, object motion, and dexterous grasping are tightly coupled: an inaccurate grasp alters the forces acting on the upper body, while poor whole-body coordination can destabilize both the robot and the object. A primary challenge is learning contact-rich robot-object interactions from human demonstrations. Reference motion tracking [1, 2, 3, 4, 5] offers a route from demonstrations to physically feasible behavior, but supervises robot and object motion at the body and arm level and leaves finger-level hand–object contact unmodeled; dexterous interaction learning [6, 7, 8, 9, 10, 11] does model such contact, yet targets simulated characters or fixed-base hands whose kinematics and actuation differ from a humanoid with underactuated hands. Human demonstrations describe coordinated body and object motion, however, differences in morphology and actuation change how the humanoid must establish hand–object contact, support the object, and maintain balance. The resulting control problem is also high-dimensional and heterogeneous: body joints govern balance and load bearing on timescales set by locomotion, while actuated finger joints govern contacts whose outcome turns on millimeter-scale placement, and both must be commanded by one policy. Even for a single interaction, matching demonstrated poses does not ensure that the robot can sustain the force and torque required to move the object. Learning therefore requires coupling robot–object motion tracking with the acquisition of contact-aware, closed-loop control under the robot’s dynamics. A secondary challenge is learning diverse object interactions within a unified policy. Across objects, differences in geometry affect the feasible contact locations and grasp configurations. Across interaction sequences, variations in approach direction, body posture, and object motion require different control behaviors, even for the same object. Prior dexterous interaction methods largely train one policy per object or per clip [9], which sidesteps this variation. The challenge is to learn a shared control strategy that captures common coordination patterns across interactions while adapting whole-body motion and finger-level contact to the specific object and reference sequence. In this work, we investigate how to equip humanoids with whole-body dexterous interaction capability from captured human demonstrations. Our key idea is to make object geometry and hand–object contact explicit throughout the pipeline: references are constructed to preserve task-relevant contacts across the morphology gap, and a unified policy, conditioned on object geometry and trained jointly across objects and interaction clips, tracks robot and object motion while explicitly learning whole-body coordination and finger-level contact. We present Weave, an end-to-end framework for learning whole-body dexterous loco-manipulation from human–object interaction. Given captured human–object interactions, which records full-body human motion while manipulating everyday objects such as chairs, boxes, and tables, Weave completes the missing approach and transition motions with whole-body motion generation [12], retargets the interaction to the robot with contact-aware retargeting, and trains the contact- and geometry-aware policy in simulation. As a result, the robot walks up to an object, grasps it with multi-fingered hands, lifts it, and carries it to a target configuration. We evaluate the learned policy and publicly release a curated dataset of simulation rollouts to support research on whole-body dexterous interaction and learning from robot demonstrations. In summary, our contributions are as follows: • Contact- and geometry-aware whole-body dexterous policy. A unified policy, trained jointly across multiple objects and interaction clips, that tracks robot and object motion while explicitly learning hand–object contact, reproducing whole-body coordination, object transport, and finger-level grasping within one policy. • Interaction-preserving trajectory construction. A pipeline that converts captured human–object interactions into executable robot–object trajectories through approach-motion completion and contact-aware retargeting, preserving object motion and hand–object contacts across the human–robot morphology gap. • Systematic evaluation and physically plausible dataset. We conduct extensive experiments to assess interaction execution, quantify the benefits of joint multi-object learning, and examine how policy architecture and optimization affect learning efficiency, together with the resulting rollout dataset released for downstream policy learning and humanoid-object interaction modeling. I Related Works I-A Interaction-Preserving Retargeting and Motion Generation Motion retargeting constructs reference motions from human demonstrations, conventionally by optimizing joint configurations to match Cartesian keypoints [13, 14, 15]. For human–object interaction, however, pose similarity alone is insufficient: embodiment differences can alter hand–object contacts, break the relative interaction geometry, or produce configurations that cannot exert the intended effect on the object. Interaction-preserving retargeting therefore incorporates contact constraints: OmniRetarget [3] preserves whole-body interaction structure for humanoids, TopoRetarget [16] and REGRIND [17] encode local hand–object topology for dexterous hands, and other works [18, 19, 20, 21] further enforce force consistency of the retargeted contacts. In parallel, human motion generation has progressed from parametric motion synthesis [22] to more scalable and controllable models: Kimodo [12] handles heterogeneous spatial and temporal constraints for both human and humanoid embodiments, and ARDY [23] extends to autoregressive diffusion for streaming generation. Weave combines contact-aware retargeting with Kimodo-based approach-motion completion to construct robot–object reference trajectories. These kinematic references then supervise reinforcement learning, which converts them into physically executable interactions in simulation. I-B Humanoid Loco-Manipulation Humanoid loco-manipulation requires coordinating locomotion, balance, and object interaction over extended task horizons. Modular approaches organize navigation, locomotion, reaching, and manipulation into separate components [24, 25, 26, 27]. Such decomposition provides structured interfaces, while coordinating balance and hand–object contact across these interfaces remains an important consideration. Reference-tracking approaches instead learn a physics-based policy that tracks diverse humanoid motions and object interactions [1, 2, 3, 28, 4, 5]. Physics-based grasping and human–object interaction methods [6, 8, 9, 10, 11] highlight the importance of coupling motion tracking with hand–object contact, motivating unified control across objects and interaction sequences. A parallel line of work introduces egocentric depth or RGB observations for visually guided loco-manipulation [29, 30, 31, 32, 33]. Data-generation approaches further diversify humanoid interactions: GRAIL [34] synthesizes 4D humanoid–object interactions with a video generation model [35], HumanoidMimicGen [36] augments a few teleoperated demonstrations through whole-body planning, and VLK [37] synthesizes paired vision–language–kinematics data in reconstructed scenes. Relative to this literature, Weave supervises finger-level contact rather than body- and arm-level motion alone, learns a single policy across objects and interaction clips rather than per-object controllers, and releases the executed rollouts as reusable interaction data. I-C Humanoid Vision–Language–Action Models VLA models connect semantic task descriptions and visual observations with executable robot actions. General-purpose systems [38, 39, 40, 41] leverage large-scale robot datasets and pretrained VLM representations to generalize across scenes and tasks. Most of these models were initially developed for fixed-base manipulators, whose action spaces and stability requirements differ substantially from those of humanoid robots. Extending VLA models to humanoids requires reasoning over high-dimensional whole-body motion while maintaining balance and physical contact. Recent humanoid-specific systems [42, 43, 44, 45], explore unified or hierarchical architectures that connect semantic reasoning with whole-body motor control. Nevertheless, acquiring large-scale data that jointly contain locomotion, precise object interaction, and articulated finger motion remains substantially more difficult than collecting arm-level manipulation trajectories. Therefore, Weave is complementary rather than competing: instead of collecting interactions through teleoperation or generation, we acquire the skills through physics-based reinforcement learning on human-object motion, producing grounded, finger-level behavior. Such physically grounded skills can serve as reusable low-level capabilities for future humanoid VLA systems, while VLA models can provide the semantic task specifications and high-level motion commands needed to select and compose them. Figure 2: Overview of Weave. (a) Captured human–object interactions provide paired SMPL-X body motion and object trajectories. (b) Whole-body inverse kinematics retargets each interaction while preserving the hand–object contacts and the object motion. (c) Kimodo [12] synthesizes locomotion prefixes that approach the initial interaction pose from different azimuths, producing several complete references from one captured sequence. (d) A unified reference-conditioned policy is trained in simulation across all objects and interaction clips. I Problem Formulation We study whole-body dexterous loco-manipulation, in which a humanoid equipped with dexterous hands must approach an object, establish a stable grasp, and transport the object toward a desired configuration while maintaining whole-body balance. We formulate the closed-loop control problem as a partially observable Markov decision process (POMDP) ℳ=(,,,,r,γ)M=(S,A,O,T,r,γ), where S and O denote the state and observation spaces, A is the action space, T is the state-transition function, r is the reward function, and γ is the discount factor. At each time step t, the policy outputs an action at∼πθ(⋅∣ot)a_t _θ(· o_t), where ata_t specifies desired joint positions for low-level PD controller. Policy learning is bootstrapped from captured human–object interaction data [46]. After retargeting and motion completion, each sequence provides a robot–object reference trajectory τref=s^treft=0T−1,s^tref=(x^trobot,x^tobj),τ^ref= \ s^ref_t \_t=0^T-1, s^ref_t= ( x^robot_t, x^obj_t ), (1) where x^trobot x^robot_t specifies the reference humanoid configuration and x^tobj x^obj_t specifies the corresponding object pose. The reference trajectory is used to initialize state in simulation and construct the tracking objectives during policy training. IV Method IV-A Overview Weave transforms captured human–object interactions into whole-body dexterous loco-manipulation skills through two main stages. First, we construct reference trajectories using whole-body retargeting, contact-aware hand refinement, and approach-motion completion. Then we train a unified contact- and geometry-aware policy through reference-tracking, jointly coordinating the humanoid body, articulated fingers, and object motion across multiple objects and interaction clips. IV-B Interaction-Preserving Reference Construction Before policy learning, we convert captured human–object interaction sequences into reference motions. Starting from the paired SMPL-X motion and object trajectory, our pipeline first retargets the interaction segment to the humanoid through whole-body inverse kinematics and contact-aware hand refinement. We then use Kimodo [12] to synthesize a locomotion prefix that approaches the initial interaction pose. This process produces a synchronized robot–object trajectory spanning approach, grasping, and object transport, together with contact annotations used for policy training. Whole-body retargeting. We first obtain an initial robot trajectory x~trobott=0T−1\ x^robot_t\_t=0^T-1 using a GMR-style whole-body inverse-kinematics objective [47], which aligns selected robot links with their SMPL-X counterparts. To account for morphological differences, we constrain the pelvis only in the horizontal plane and optimize its height based on ground contact in place of scale calibration, while joint-limit, velocity, and acceleration penalties promote feasible and temporally smooth motion. TABLE I: Reward terms for reference tracking. Hatted quantities denote references. Tracking terms contribute wiexp(−et,i/σi2)w_i (-e_t,i/ _i^2). Reward term Computation Definition Weight w Scale σ Robot and object tracking Pelvis position ∥ptr−p^tr∥22 p^r_t- p^r_t _2^2 ptr,p^trp^r_t, p^r_t: current/reference pelvis position 1.01.0 0.30.3 Pelvis orientation ∠(Rtr,R^tr)2 (R^r_t, R^r_t)^2 Rtr,R^trR^r_t, R^r_t: pelvis orientation; ∠ : geodesic error 1.01.0 0.40.4 Body positions 1|ℬ|∑b∈ℬ∥pt,b−p^t,b∥22 1|B| _b p_t,b- p_t,b _2^2 ℬB: tracked robot links 1.01.0 0.30.3 Body orientations 1|ℬ|∑b∈ℬ∠(Rt,b,R^t,b)2 1|B| _b (R_t,b, R_t,b)^2 Rt,b,R^t,bR_t,b, R_t,b: current/reference link orientation 1.01.0 0.40.4 Object position ∥pto−p^to∥22 p^o_t- p^o_t _2^2 pto,p^top^o_t, p^o_t: current/reference object position 2.02.0 0.20.2 Object orientation ∠(Rto,R^to)2 (R^o_t, R^o_t)^2 Rto,R^toR^o_t, R^o_t: current/reference object orientation 2.02.0 0.30.3 Grasp and contact Hand opposition ∑hηt,h(1|h|∑k∈h1−ut,h,0⊤ut,h,k2)∑hηt,h+ϵ _h _t,h ( 1|K_h| _k _h 1-u_t,h,0 u_t,h,k2 ) _h _t,h+ε ut,h,ku_t,h,k: surface-to-fingertip unit vector; k=0k=0: thumb; ηt,h _t,h: expected-contact gate 2.02.0 – Contact matching ∑h∈ℋmt,h(1−|yt,h−c~t,h|)∑h∈ℋmt,h+ϵ _h m_t,h(1-|y_t,h- c_t,h|) _h m_t,h+ε mt,h=[c^t,h≠0]m_t,h=1[ c_t,h≠ 0]; yt,h=[c^t,h>0]y_t,h=1[ c_t,h>0]; c~t,h=min(∥ft,h∥/f¯,1) c_t,h= ( f_t,h / f,1) 2.02.0 – Regularization Foot sliding ∑f∈ℬfootχt,fground∥vt,fxy∥2 _f _footχ^ground_t,f v^xy_t,f _2 χt,fgroundχ^ground_t,f: foot–ground contact indicator −1.0-1.0 – Action rate ∥at−at−1∥22 a_t-a_t-1 _2^2 at,at−1a_t,a_t-1: current/previous joint command −0.1-0.1 – Joint limits ∑j∈body([qt,j−q¯j]++[q¯j−qt,j]+) _j _body ([q_t,j- q_j]_++[ q_j-q_t,j]_+ ) bodyJ_body: non-finger joints; [q¯j,q¯j][ q_j, q_j]: soft joint limits −10.0-10.0 – Contact-aware hand refinement. Whole-body IK captures the arm and wrist motion but does not ensure a stable fingertip grasp. We therefore jointly refine the arm and finger configurations of both limbs, ξt=(qtarm,qthand) _t=(q_t^arm,q_t^hand), initialized from x~trobott=0T−1\ x^robot_t\_t=0^T-1 and held fixed elsewhere: the pelvis pose and the leg configuration remain at their IK solution, so the refinement does not alter the retargeted whole-body motion. For fingertip k at frame t, let pt,k(ξt)p_t,k( _t) be its forward-kinematics position, p¯t,k p_t,k the object-surface point nearest to the corresponding human fingertip, nt,kn_t,k its outward normal, and ct,k∈0,1c_t,k∈\0,1\ the contact label. Following [19], we minimize ℒhand= _hand= λa∑t,kct,k∥pt,k−p¯t,k∥⏟contact attraction−λq∑tQ^FC(qth)⏟force-closure objective _a _t,kc_t,k p_t,k- p_t,k _contact attraction- _q _t Q_FC(q_t^h)_force-closure objective (2) +λp∑t,k[−(pt,k−p¯t,k)⊤nt,k]+⏟penetration penalty+ℒreg. + _p _t,k [-(p_t,k- p_t,k) n_t,k ]_+_penetration penalty+\ L_reg. where [z]+=max(z,0).[z]_+= (z,0). The attraction term alone only places fingertips on the object surface, which admits configurations that touch the object without being able to hold it. We therefore additionally require the contacts to resist arbitrary disturbance wrenches, and maximize a differentiable approximation Q^FC Q_FC of the force-closure quality [48], QFC(ξt)=min∥w∥2=1maxf∈ℱ(ξt)∥f∥1≤1w⊤G(ξt)f.Q_FC( _t)= _ w _2=1 _ subarraycf ( _t)\\ f _1≤ 1 subarrayw G( _t)f. (3) Here, G maps feasible contact forces f to object-centric wrenches, and ℱF is the linearized friction cone with soft-finger torsion. Intuitively, QFCQ_FC measures the weakest disturbance wrench that the grasp can resist; QFC>0Q_FC>0 indicates force closure. We approximate it by a fixed set of wrench directions. Approach motion completion. The retargeted interaction begins near the object and therefore lacks the locomotion required to approach it. We use Kimodo [12] to generate an approach prefix τpreτ^pre for the refined interaction trajectory τintτ^int. The initial frames of τintτ^int provide terminal whole-body constraints, while the robot follows a path ending at the initial pose and heading. We concatenate the two segments: τref=τpre⊕τint,T=Tpre+Tint.τ^ref=τ^pre τ^int, T=T_pre+T_int. (4) Sampling different approach azimuths produces multiple complete references from the same interaction sequence. We sample K=3K=3 for training set and K=5K=5 for test set, which is the main source of the trajectory counts reported in V-A. Reference annotations. After concatenation, the completed reference is re-indexed as τref=s^treft=0T−1τ^ref=\ s^ref_t\_t=0^T-1, where s^tref=(x^trobot,x^tobj)=(p^tr,R^tr,q^t,p^to,R^to). s^ref_t= ( x^robot_t, x^obj_t )= ( p^r_t, R^r_t, q_t, p^o_t, R^o_t ). (5) Here, p^tr p^r_t, R^tr R^r_t, and q^t q_t denote the humanoid root position, root orientation, and joint configuration, while p^to p^o_t and R^to R^o_t denote the object pose. For each robot link ℓ , we additionally assign c^t,ℓ=[dt,ℓo<δcontact]−[dt,ℓo>δfar], c_t, =1 [d^o_t, < _contact ]-1 [d^o_t, > _far ], (6) where dt,ℓod^o_t, is the distance from the link position to the object surface. Thus, c^t,ℓ∈−1,0,1 c_t, ∈\-1,0,1\ denotes separated, neutral, and contact states, respectively. The completed trajectories provide initialization states and reference motion for policy learning, while the contact annotations specify the desired interaction pattern. Together, they guide a policy that learns to realize the reference interactions under the simulator dynamics. IV-C Contact-Rich Skill Acquisition via Reference Tracking We train a unified reference-conditioned policy in simulation, jointly across multiple objects and interaction clips. The policy uses robot proprioception, object state and geometry features, and reference motion to coordinate whole-body control with finger-level interaction: at∼πtrack(⋅∣otprop,otobj,x^t:t+Hrobot,x^t:t+Hobj,),a_t _track (· o^prop_t,o^obj_t, x^robot_t:t+H, x^obj_t:t+H, ), (7) where otpropo^prop_t is the robot proprioception, otobjo^obj_t is the simulated object observation, and x^t:t+Hrobot,x^t:t+Hobj\ x^robot_t:t+H, x^obj_t:t+H\ is a short-horizon command extracted from τrefτ^ref. We optimize the policy with PPO [49], using reference state initialization and reference-based early termination following [50]. Observation and action. All relative poses and geometric features are expressed in robot local frame. The proprioceptive observation contains the base angular velocity, projected gravity, joint positions and velocities, and the previous action. The object observation contains its ground-truth relative pose, fingertip-to-surface vectors, binary object-contact flags, and BPS-SDF descriptor encoding object geometry. The reference command x^t:t+Hrobot,x^t:t+Hobj\ x^robot_t:t+H, x^obj_t:t+H\ contains the joint configuration, pelvis pose, object pose, and contact labels of a short horizon. We use an asymmetric actor–critic: the actor receives (otprop,otobj,x^t:t+Hrobot,x^t:t+Hobj)(o^prop_t,o^obj_t, x^robot_t:t+H, x^obj_t:t+H), whereas the critic replaces otpropo^prop_t with a privileged robot observation that additionally contains the base linear velocity and current tracked-body poses. The action ata_t specifies joint-position targets tracked by PD controllers at 5050 Hz. For the underactuated Inspire hands, the policy commands only the proximal finger joints, while the intermediate and distal joints follow fixed mimic couplings. Tracking objective. The reward combines robot and object tracking, geometry-aware grasping, contact matching, and motion regularization: rt= r_t= ∑i∈ℐwiexp(−et,i/σi2)+wopproppt+wcrcontactt _i w_i (-e_t,i/ _i^2 )+w_oppr^opp_t+w_cr^contact_t (8) +wslidertslide+wactrtact+wlimrtlim, +w_slider^slide_t+w_actr^act_t+w_limr^lim_t, where ℐI includes pelvis pose, tracked-body pose over the body set ℬB, and object pose. The hand-opposition reward rtoppr^opp_t encourages the thumb and opposing fingers to lie on different sides of the object surface whenever the reference specifies a grasp following [10]. For hand link h∈ℋh , let c^t,h c_t,h be the reference contact label and c~t,h=min(∥ft,h∥/f¯,1) c_t,h= ( f_t,h / f,1) the normalized contact-force magnitude. Contact matching is computed only for non-neutral labels: rtcontact=∑h∈ℋmt,h(1−|yt,h−c~t,h|)∑h∈ℋmt,h+ϵ,r^contact_t= _h m_t,h (1- |y_t,h- c_t,h | ) _h m_t,h+ε, (9) where mt,h=[c^t,h≠0]m_t,h=1[ c_t,h≠ 0] and yt,h=[c^t,h>0]y_t,h=1[ c_t,h>0]. The remaining terms penalize foot sliding, action variation, and non-finger joint-limit violations as showed in Table I. Figure 3: Reference-motion dataset distribution. Per-object composition of the training and test splits in terms of (a) the number of reference trajectories and (b) total motion duration. Randomization and termination. Across parallel environments, we randomize robot and object friction and restitution, torso center of mass, and finger actuator properties. Episodes terminate when the pelvis-position error exceeds 0.250.25 m, the object-position error exceeds 0.300.30 m, the projected-gravity error of the pelvis or object exceeds 0.80.8 or 0.30.3, respectively, or the vertical tracking error of an ankle or wrist exceeds 0.250.25 m. We also terminate after all expected hand–object contacts are absent for ten consecutive control steps. These conditions keep the on-policy state distribution close to the reference [51, 9]. Policy optimization. Each observation group is encoded and projected to latent representation. The concatenated features are processed by SimBaV2 networks [52]. Two-dimensional weight matrices are optimized with Muon [53], while biases and other parameters use AdamW. We train a unified policy jointly over all objects and reference clips under a 200200 Hz simulation frequency and a 5050 Hz control frequency. Figure 4: Evaluation against training iterations. Success rate (top) and tracking errors (bottom) for πtrainmultiπ^multi_train evaluated on trainD_train and on testD_test. The dashed line is πtestmultiπ^multi_test evaluated on testD_test, which serves as an oracle for measuring generalization. V Experiments We evaluate Weave to answer the following questions: Q1: How effectively does the learned policy execute whole-body dexterous loco-manipulation across objects and interaction sequences? Q2: How does joint multi-object training compare with single-object specialist in interaction completion and tracking accuracy? Q3: How do the policy architecture and optimizer affect learning efficiency? V-A Experimental Setup Embodiment and dataset. We instantiate Weave on Unitree G1 humanoid with 29 actuated body DoFs and two Inspire dexterous hands with 12 actuated finger DoFs. We construct two disjoint reference splits, trainD_train and testD_test, containing 7,869 (19.56h) and 1,605 (3.67h) reference trajectories respectively as showed in Figure 3. Evaluation metrics. For N evaluation clips, let TiT_i denote the number of executed frames in clip i. We define the clip-balanced average ⟨zi,t⟩=1N∑i=1N1Ti∑t=0Ti−1zi,t. z_i,t = 1N _i=1^N 1T_i _t=0^T_i-1z_i,t. The tracking errors are then defined as Eanchorp=⟨∥pi,ta−p^i,ta∥2⟩,EanchorR=⟨∠(Ri,ta,R^i,ta)⟩,Ebodyp=⟨1|ℬ|∑b∈ℬ∥pi,t,b−p^i,t,b∥2⟩,EbodyR=⟨1|ℬ|∑b∈ℬ∠(Ri,t,b,R^i,t,b)⟩. gatheredE_anchor^p= p_i,t^a- p_i,t^a _2 , E_anchor^R= (R_i,t^a, R_i,t^a) ,\\[2.84526pt] E_body^p= 1|B| _b p_i,t,b- p_i,t,b _2 ,\\[2.84526pt] E_body^R= 1|B| _b (R_i,t,b, R_i,t,b) . gathered (10) For joint and object tracking, we report Ejointq=⟨∥qi,t−q^i,t∥2⟩, E_joint^q= q_i,t- q_i,t _2 , Ejointq˙=⟨∥q˙i,t−q˙^i,t∥2⟩, E_joint q= q_i,t- q_i,t _2 , (11) Eobjp=⟨∥pi,to−p^i,to∥2⟩, E_obj^p= p_i,t^o- p_i,t^o _2 , EobjR=⟨∠(Ri,to,R^i,to)⟩. E_obj^R= (R_i,t^o, R_i,t^o) . Here, a denotes the pelvis anchor, ℬB is the set of tracked robot bodies, and ∠(⋅,⋅) (·,·) denotes the geodesic rotation error. V-B Whole-Body Dexterous Loco-Manipulation Protocol. We train a multi-object tracking policy πtrainmultiπ^multi_train on trainD_train and evaluate it on both trainD_train and testD_test to measure generalization to unseen interaction. We additionally train πtestmultiπ^multi_test directly on testD_test to serve as a comparison. The three evaluations are complementary. The first measures how well a single policy fits the references it is trained on, the second measures transfer to unseen interaction sequences of the same objects, and the third serves as an oracle. Results. Figure 4 reports both evaluations against training iterations. Training on trainD_train progresses steadily. The success rate rises from 78.0%78.0\% at 10k iterations to 92.5%92.5\% at 100k and the progress rate from 89.5%89.5\% to 96.3%96.3\%, while the tracking errors decrease over the same span, with object rotation falling from 11.17∘11.17 to 7.18∘7.18 and object position from 5.175.17 cm to 4.274.27 cm. A single policy thus absorbs 7,869 reference trajectories spanning nine objects, and performance is still improving at the end of training. The policy transfers to unseen interactions, reaching 65.0%65.0\% SR and 84.5%84.5\% PR on testD_test with tracking errors close to those on trainD_train, but the gap in completion is substantial. A likely cause is that the two splits are not identically distributed: approach-motion completion generates K=3K=3 variations per captured interaction for trainD_train and K=5K=5 for testD_test as introduced in IV-B, so the test split spans a wider range of approach directions and generated locomotion prefixes than the policy encounters during training, leading to a mismatch of data distribution. V-C Joint Multi-Object Skill Learning Figure 5: Joint multi-object training versus object-specific specialists. Top: per-object and pooled success rates on testD_test, where the pooled result is weighted by the number of evaluation trajectories. Bottom: tracking errors computed over rollouts. Protocol. We compare a single multi-object policy πtestmulti _test^multi, trained on all objects in testD_test, with nine single-object specialist πtestjj=19\π^j_test\_j=1^9, each trained on the corresponding object’s trajectories. Each single-object specialist is trained for 3k iterations. The unified policy is trained for Nobj×3N_obj× 3k iterations, where Nobj=9N_obj=9, giving 27k iterations in total. Thus, the unified policy and the collection of nine specialists receive the same aggregate number of training iterations. This comparison measures joint skill acquisition on a shared reference collection. All policies use the same observations, actions, reward, network architecture, and PPO configuration. We do not tune hyperparameters for either the specialists or the unified policy. Results. As shown in Figure 5, the unified policy achieves a pooled success rate of 95.3%95.3\%, compared with 91.5%91.5\% for the specialists. Joint training matches specialist success on three objects and improves it on the other six, so the benefit is distributed across multiple objects rather than driven by a single easy category, and no object regresses. The pooled rates are weighted by the number of evaluation trajectories and therefore give greater weight to objects with more clips. Success and imitation fidelity dissociate. The specialists often attain slightly lower tracking errors while completing fewer interactions. A specialist fits a single object’s reference closely, whereas the unified policy allocates capacity across nine contact patterns and converges to a more conservative solution that trades pose fidelity for recovery margin. Lower imitation error therefore does not imply more reliable interaction completion, which argues against using tracking error alone as the primary metric for contact-rich interaction. Why joint training helps. We offer two mechanisms consistent with the evidence. First, the approach and transport phases share whole-body structure across objects, such as walking to a pose, balancing the torso, and bearing a load, so clips from different objects act as mutual augmentation. Second, contact patterns are shared across objects of similar shape: a small table and a large table are grasped in much the same way, as are a clothes stand and a floor lamp. A specialist sees one geometry and can memorize a single grasp that fits it, whereas joint training exposes the policy to families of related shapes and pushes it toward selecting a grasp according to the object’s geometry, which benefits to unseen interaction generalization. V-D Architecture and Optimizer Ablation Figure 6: Sample-efficiency ablation on the small-table task. We compare four combinations under a common training budget of 3,000 iterations. Light curves show the raw mean episode length and reward, while dark curves show exponential moving averages with a span of 50 iterations. Protocol. We evaluate all four combinations of MLP and SimBaV2 with AdamW and Muon on the small-table task under a common budget of 3,000 training iterations. Figure 6 reports mean episode length and reward, with raw curves and exponential moving averages. Results. Both Muon configurations improve earlier than their AdamW counterparts, suggesting that the optimizer contributes the larger gain in sample efficiency in this comparison. A plausible explanation is that Muon’s orthogonalized momentum updates, so no single direction dominates a gradient update step. This matters because the reward couples body tracking, object tracking, and finger contact, whose gradients differ substantially in scale: an update dominated by the body tracking term yields a policy that follows the reference pose without closing fingers, a local optimum that early training falls into easily. SimBaV2 provides a smaller additional improvement. Its hyperspherical normalization constrains weight and feature norms, which limits how far the policy moves in a single updates as the state distribution shifts at the onset of the contact, where the reward landscape changes most abruptly. The two effects are complementary: the optimizer governs how the gradient is distributed across the coupled objectives, while the architecture governs how much the policy is allowed to change when the state distribution moves. VI Conclusions and Limitations We present Weave, a framework that converts captured human-object interactions into whole-body dexterous loco-manipulation skills. A reference-construction pipeline preserves task-relevant hand-object contacts across the morphology gap and completes the missing approach phases, and a unified contact- and geometry-aware policy, trained jointly over diverse object and multiple interaction clips, coordinates locomotion, whole-body balance, and finger-level grasping. The unified policy completes 92.5%92.5\% success rate on trained interactions, and without any additional training, 65.0%65.0\% on sequences beyond training data distribution, overall produces ∼ 23h of physically executed rollouts. However, several limitations remain. Evaluation is carried out entirely in simulation: the policy consumes ground-truth odometry and object pose, so real-world deployment requires either onboard state estimation or distillation into a perception-based student, and the sim2real gap for contact-rich interaction is not accessed here. Concern for embodiment, the Inspire hand is underactuated and the policy commands proximal joints only, which bounds the achievable finger-level fidelity, and heavy or highly articulated objects are outside the range covered by our references and domain randomizations. Finally, the behaviors are reference-conditioned: the policy executes an interaction that must be supplied to it, and pairing it with a higher-level planner that selects and composes interaction, for instance, the humanoid VLA systems discussed in I-C, remains future work. References [1] F. Liu, Z. Gu, Y. Cai, Z. Zhou, H. Jung, J. Jang, S. Zhao, S. Ha, Y. Chen, D. Xu et al., “Opt2skill: Imitating dynamically-feasible whole-body trajectories for versatile humanoid loco-manipulation,” IEEE Robotics and Automation Letters, 2025. [2] H. Weng, Y. Li, N. Sobanbabu, Z. Wang, Z. Luo, T. He, D. Ramanan, and G. Shi, “Hdmi: Learning interactive humanoid whole-body control from human videos,” arXiv preprint arXiv:2509.16757, 2025. [3] L. Yang, X. Huang, Z. Wu, A. Kanazawa, P. Abbeel, C. Sferrazza, C. K. Liu, R. Duan, and G. Shi, “Omniretarget: Interaction-preserving data generation for humanoid whole-body loco-manipulation and scene interaction,” arXiv preprint arXiv:2509.26633, 2025. [4] S. Zhao, Y. Ze, Y. Wang, C. K. Liu, P. Abbeel, G. Shi, and R. Duan, “Resmimic: From general motion tracking to humanoid whole-body loco-manipulation via residual learning,” arXiv preprint arXiv:2510.05070, 2025. [5] S. Chen, S. Zhao, Z. Wu, J. Li, G. Shi, and C. K. Liu, “Scenebot: Contact-prompted general humanoid whole body tracking with scene-interaction,” arXiv preprint arXiv:2606.27581, 2026. [6] Z. Luo, J. Cao, S. Christen, A. Winkler, K. Kitani, and W. Xu, “Omnigrasp: Grasping diverse objects with simulated humanoids,” Advances in Neural Information Processing Systems, vol. 37, p. 2161–2184, 2024. [7] Z. Luo, C. Tessler, T. Lin, Y. Yuan, T. He, W. Xiao, Y. Guo, G. Chechik, K. Kitani, L. Fan et al., “Emergent active perception and dexterity of simulated humanoids from visual reinforcement learning,” arXiv preprint arXiv:2505.12278, 2025. [8] C. Tessler, Y. Jiang, E. Coumans, Z. Luo, X. B. Peng, and G. Chechik, “Maskedmanipulator: Versatile whole-body control for loco-manipulation,” in Proceedings of the SIGGRAPH Asia 2025 Conference Papers, 2025, p. 1–11. [9] S. Xu, H. Y. Ling, Y.-X. Wang, and L.-Y. Gui, “Intermimic: Towards universal whole-body control for physics-based human-object interactions,” in Proceedings of the Computer Vision and Pattern Recognition Conference, 2025, p. 12 266–12 277. [10] S. Xu, S. Schulter, M. Ziyadi, X. He, X. Fei, Y.-X. Wang, and L. Gui, “Interprior: Scaling generative control for physics-based human-object interactions,” arXiv preprint arXiv:2602.06035, 2026. [11] Z. Wu, J. Li, P. Xu, and C. K. Liu, “Human-object interaction from human-level instructions,” in 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025, p. 11 176–11 186. [12] D. Rempe, M. Petrovich, Y. Yuan, H. Zhang, X. B. Peng, Y. Jiang, T. Wang, U. Iqbal, D. Minor, M. de Ruyter et al., “Kimodo: Scaling controllable human motion generation,” arXiv preprint arXiv:2603.15546, 2026. [13] Z. Luo, J. Cao, K. Kitani, W. Xu et al., “Perpetual humanoid control for real-time simulated avatars,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, p. 10 895–10 904. [14] K. Zakka, “Mink: Python inverse kinematics based on MuJoCo,” Feb. 2026. [Online]. Available: https://github.com/kevinzakka/mink [15] T. He, Z. Luo, X. He, W. Xiao, C. Zhang, W. Zhang, K. Kitani, C. Liu, and G. Shi, “Omnih2o: Universal and dexterous human-to-humanoid whole-body teleoperation and learning,” arXiv preprint arXiv:2406.08858, 2024. [16] J. Wu, S. Yao, G. He, X. Liu, Z. Zeng, X. Jiang, H. Yang, W. Zhang, and H. Zhao, “Toporetarget: Interaction-preserving retargeting for dexterous manipulation,” arXiv preprint arXiv:2606.16272, 2026. [17] Y. Feng, N. Leung, J. Wang, L. Yang, H. Qi, and P. Culbertson, “A minimalist retargeting-guided reinforcement learning recipe for dexterous manipulation,” arXiv preprint arXiv:2607.11874, 2026. [18] Y. Wu, L. Zeng, C. Jing, J. Ye, and X. Wang, “Reforce: Learning force-aware retargeting for dexterous manipulation,” arXiv preprint arXiv:2608.15560, 2026. [19] Y. Shao, Z. Chen, W. Lin, M. Zhou, T. Chen, X. Yang, Y. Chi, and Y. Mu, “Synmandex: Synthesizing human-like dexterous grasps from synthetic human pre-grasps,” arXiv preprint arXiv:2606.09798, 2026. [20] X. Zhu, Z. Liu, S. Jain, C. Li, M. Noori, M. A. Lin, H. Zhao, J. Welsh, M. Verghese, W. Liu et al., “Learning dexterous manipulation using contact wrench guidance from human demonstration,” arXiv preprint arXiv:2607.00033, 2026. [21] R. Chen, F. Ruan, L. Cao, Z. Wang, B. Xu, S. Tong, J. Liu, M. Pei, C. Zhang, W. Xing et al., “Dex-x: Learning visual-tactile dexterous manipulation from human videos with simulated interaction,” challenge, vol. 1, no. 29, p. 30. [22] T. Wang, O. Dionne, M. De Ruyter, D. Minor, D. Rempe, K. Zhao, M. Petrovich, Y. Yuan, C. Li, Z. Luo et al., “Motionbricks: Scalable real-time motions with modular latent generative model and smart primitives,” ACM Transactions on Graphics (TOG), vol. 45, no. 4, p. 1–22, 2026. [23] K. Zhao, M. Petrovich, H. Zhang, T. Wang, S. Tang, and D. Rempe, “Ardy: Autoregressive diffusion with hybrid representation for interactive human motion generation,” arXiv preprint arXiv:2607.08741, 2026. [24] S. Chen, Y. Ye, Z.-a. Cao, J. Lew, P. Xu, and C. K. Liu, “Hand-eye autonomous delivery: Learning humanoid navigation, locomotion and reaching,” arXiv preprint arXiv:2508.03068, 2025. [25] S. Arnaud, P. McVay, A. Martin, A. Majumdar, K. M. Jatavallabhula, P. Thomas, R. Partsey, D. Dugas, A. Gejji, A. Sax et al., “Locate 3d: Real-world object localization via self-supervised learning in 3d,” arXiv preprint arXiv:2504.14151, 2025. [26] W. Liu, H. Zhao, C. Li, Y. Deng, J. Biswas, S. Pouya, and Y. Chang, “Compass: Cross-embodiment mobility policy via residual rl and skill synthesis,” arXiv preprint arXiv:2502.16372, 2025. [27] T. He, Z. Wang, H. Xue, Q. Ben, Z. Luo, W. Xiao, Y. Yuan, X. Da, F. Castañeda, S. Sastry et al., “Viral: Visual sim-to-real at scale for humanoid loco-manipulation,” arXiv preprint arXiv:2511.15200, 2025. [28] I. Taouil, M. Ciebelski, S. Omar, H. Zhao, A. Dai, A. M. Johnson, and M. Khadiv, “Motiondisco: Motion discovery for extreme humanoid loco-manipulation,” 2026. [29] D. Li, Q. Wu, X. Chen, L. Li, Y. Lin, S. Wu, G. Zhang, M. Zhou, D. Xiang, Q. Zhang et al., “Vaic: Vision-guided humanoid agile object interaction control via decoupled commands,” arXiv preprint arXiv:2606.09286, 2026. [30] H. Wang, W. Zhang, R. Yu, T. Huang, J. Ren, F. Jia, Z. Wang, X. Niu, X. Chen, J. Chen et al., “Physhsi: Towards a real-world generalizable and natural humanoid-scene interaction system,” arXiv preprint arXiv:2510.11072, 2025. [31] X. He, S. Xu, X. Li, R. Dong, L. Bian, Y.-X. Wang, and L.-Y. Gui, “Ultra: Unified multimodal control for autonomous humanoid whole-body loco-manipulation,” arXiv preprint arXiv:2603.03279, 2026. [32] M. Byrd, D. Baek, K. Garg, H. Jung, D. Cho, M. Sorokin, R. Wright, and S. Ha, “Adaptmanip: Learning adaptive whole-body object lifting and delivery with online recurrent state estimation,” arXiv preprint arXiv:2602.14363, 2026. [33] S. Yin, Y. Ze, H.-X. Yu, C. K. Liu, and J. Wu, “Visualmimic: Visual humanoid loco-manipulation via motion tracking and generation,” arXiv preprint arXiv:2509.20322, 2025. [34] T. Xie, H. Zhang, J. Park, Z. Wang, B. Wen, J. Li, X. Li, Q. Ben, H. Weng, Y. Ye, D. Minor, T. Wang, C. Jiang, S. Fidler, J. Kautz, L. Fan, Y. Zhu, Z. Luo, U. Iqbal, and Y. Yuan, “Grail: Generating humanoid loco-manipulation from 3d assets and video priors,” 2026. [Online]. Available: https://arxiv.org/abs/2606.05160 [35] K. Team, J. Chen, Y. Ding, Z. Fang, K. Gai, K. He, X. He, J. Hua, M. Lao, X. Li et al., “Kling-motioncontrol technical report,” arXiv preprint arXiv:2603.03160, 2026. [36] K. Lin, A. Mandlekar, C. R. Garrett, N. Chernyadev, Y. Fang, R. Ding, Y. Xie, J. Tran, L. Fan, and Y. Zhu, “Humanoidmimicgen: Data generation for loco-manipulation via whole-body planning,” arXiv preprint arXiv:2605.27724, 2026. [37] Y.-J. Wang, J. Li, S. Chen, T. E. Truong, P. Xu, P. Abbeel, R. Duan, K. Sreenath, A. Kanazawa, C. Sferrazza et al., “Vlk: Learning humanoid loco-manipulation from synthetic interactions in reconstructed scenes,” arXiv preprint arXiv:2606.30645, 2026. [38] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi et al., “Openvla: An open-source vision-language-action model,” arXiv preprint arXiv:2406.09246, 2024. [39] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter et al., “π0 _0: A vision-language-action flow model for general robot control,” arXiv preprint arXiv:2410.24164, 2024. [40] P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai et al., “π0.5 _0.5: a vision-language-action model with open-world generalization,” arXiv preprint arXiv:2504.16054, 2025. [41] J. Bjorck, F. Castañeda, N. Cherniadev, X. Da, R. Ding, L. Fan, Y. Fang, D. Fox, F. Hu, S. Huang et al., “Gr00t n1: An open foundation model for generalist humanoid robots,” arXiv preprint arXiv:2503.14734, 2025. [42] H. Jiang, J. Chen, Q. Bu, L. Chen, M. Shi, Y. Zhang, D. Li, C. Suo, C. Wang, Z. Peng et al., “Wholebodyvla: Towards unified latent vla for whole-body loco-manipulation control,” arXiv preprint arXiv:2512.11047, 2025. [43] S. Bai, M. Li, X. Lv, J. Wang, X. Wang, F. Liao, C. Hou, L. Gu, W. Zhou, K. Wu et al., “Hex: Humanoid-aligned experts for cross-embodiment whole-body manipulation,” arXiv preprint arXiv:2604.07993, 2026. [44] S. Wei, H. Jing, B. Li, Z. Zhao, J. Mao, Z. Ni, S. He, J. Liu, X. Liu, K. Kang et al., “Ψ0 _0: An open foundation model towards universal humanoid loco-manipulation,” arXiv preprint arXiv:2603.12263, 2026. [45] Z. Li, Z. Zhang, Y. Wei, W. Zhang, X. Yuan, P. Zhi, G. Li, X. Guo, F. Gao, J. Yang et al., “omegaomega-0: A latent predictive world action model for concurrent humanoid loco-manipulation,” arXiv preprint arXiv:2608.06375, 2026. [46] J. Li, J. Wu, and C. K. Liu, “Object motion guided human motion synthesis,” 2023. [Online]. Available: https://arxiv.org/abs/2309.16237 [47] J. P. Araujo, Y. Ze, P. Xu, J. Wu, and C. K. Liu, “Retargeting matters: General motion retargeting for humanoid motion tracking,” arXiv preprint arXiv:2510.02252, 2025. [48] C. Ferrari, J. Canny et al., “Planning optimal grasps,” in Proceedings., 1992 IEEE International Conference on Robotics and Automation, 1992., vol. 3. IEEE, 1992, p. 2290–2295. [49] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017. [50] X. B. Peng, P. Abbeel, S. Levine, and M. Van de Panne, “Deepmimic: Example-guided deep reinforcement learning of physics-based character skills,” ACM Transactions On Graphics (TOG), vol. 37, no. 4, p. 1–14, 2018. [51] Q. Liao, T. E. Truong, X. Huang, Y. Gao, G. Tevet, K. Sreenath, and C. K. Liu, “Beyondmimic: From motion tracking to versatile humanoid control via guided diffusion,” arXiv preprint arXiv:2508.08241, 2025. [52] H. Lee, D. Hwang, D. Kim, H. Kim, J. J. Tai, K. Subramanian, P. Wurman, J. Choo, P. Stone, and T. Seno, “Simba: Simplicity bias for scaling up parameters in deep reinforcement learning,” in International Conference on Learning Representations, vol. 2025, 2025, p. 46 816–46 848. [53] J. Liu, J. Su, X. Yao, Z. Jiang, G. Lai, Y. Du, Y. Qin, W. Xu, E. Lu, J. Yan et al., “Muon is scalable for llm training,” arXiv preprint arXiv:2502.16982, 2025.