Paper deep dive
DexMani: Human-Derived Manipulability Guidance for Dexterous Rotation
Xiaoyang Chen, Shengcheng Luo, Haoran Guo, Jiaming Jiang, Wanlin Li, Ziyuan Jiao, Chenxi Xiao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/4/2026, 4:38:31 AM
Summary
The paper introduces DexMani, a framework that transfers human demonstrations as contact-conditioned manipulability evolution to guide reinforcement learning for dexterous object rotation. By learning an energy-based prior from human visual-tactile data, DexMani guides downstream RL policies to achieve higher success rates across various robotic hand embodiments (Shadow Hand, Allegro Hand, XHand, LEAP Hand) compared to baselines like PPO and VT Pretraining.
Entities (12)
Relation Signals (9)
DexMani → evaluatedon → Allegro Hand
confidence 95% · Across the Shadow Hand, Allegro Hand, and XHand, DexMani achieves the highest success rates
DexMani → evaluatedon → LEAP Hand
confidence 95% · DexMani reaches an average success rate of 57.5% on LEAP Hand
DexMani → evaluatedon → XHand
confidence 95% · Across the Shadow Hand, Allegro Hand, and XHand, DexMani achieves the highest success rates
DexMani → evaluatedon → Shadow Hand
confidence 95% · Across the Shadow Hand, Allegro Hand, and XHand, DexMani achieves the highest success rates
DexMani → uses → human demonstrations
confidence 95% · DexMani, a framework that transfers human demonstrations as contact-conditioned manipulability evolution.
contact-conditioned manipulability → captures → rotational capability evolution
confidence 90% · contact-conditioned manipulability, a human-to-robot representation that captures how contact transitions reshape task-space rotational capabilities.
energy-based prior → guides → Reinforcement Learning
confidence 90% · DexMani then learns this manipulability evolution and uses it to guide downstream reinforcement learning
DexMani → outperforms → VT Pretraining
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Dexterous object rotation is a sequential contact problem: each support, release, and re-contact decision must both produce the desired object motion, and prepare the hand configuration for continued rotation. Existing reinforcement learning methods discover such movement patterns through trial and error on specific robotic hand embodiments, without explicitly accounting for how each contact transition affects the hand's ability to sustain object rotation in subsequent steps. We introduce DexMani, a framework that transfers human demonstrations as contact-conditioned manipulability evolution. This prior captures how successful human contact transitions reshape the object-rotation directions available to the hand. DexMani then learns this manipulability evolution and uses it to guide downstream reinforcement learning, enabling rotation skills to be acquired across robot embodiments with distinct kinematics and active-contact configurations. Across the Shadow Hand, Allegro Hand, and XHand, DexMani achieves the highest success rates in every evaluated setting for both seen and unseen objects. DexMani reaches an average success rate of 57.5% on LEAP Hand, outperforming other baselines and producing smoother rotatory motions. Project site: this https URL
Tags
Links
- Source: https://arxiv.org/abs/2608.00554v1
- Canonical: https://arxiv.org/abs/2608.00554v1
Trouble viewing inline? Open PDF directly →
Full Text
78,637 characters extracted from source content.
Expand or collapse full text
DexMani: Human-Derived Manipulability Guidance for Dexterous Rotation Xiaoyang Chen1,2,3 , Shengcheng Luo2,3 , Haoran Guo2, Jiaming Jiang2,3, Wanlin Li3, Ziyuan Jiao3,4 , Chenxi Xiao2 Abstract Dexterous object rotation is a sequential contact problem: each support, release, and re-contact decision must both produce the desired object motion, and prepare the hand configuration for continued rotation. Existing reinforcement learning methods discover such movement patterns through trial and error on specific robotic hand embodiments, without explicitly accounting for how each contact transition affects the hand’s ability to sustain object rotation in subsequent steps. We introduce DexMani, a framework that transfers human demonstrations as contact-conditioned manipulability evolution. This prior captures how successful human contact transitions reshape the object-rotation directions available to the hand. DexMani then learns this manipulability evolution and uses it to guide downstream reinforcement learning, enabling rotation skills to be acquired across robot embodiments with distinct kinematics and active-contact configurations. Across the Shadow Hand, Allegro Hand, and XHand, DexMani achieves the highest success rates in every evaluated setting for both seen and unseen objects. DexMani reaches an average success rate of 57.5% on LEAP Hand, outperforming other baselines and producing smoother rotatory motions. Project site — https://dexmani.github.io/ 1 Introduction Dexterous rotation tasks, such as unscrewing a cap, turning a valve, or spinning an object, often require the fingers to repeatedly break and re-establish contact (Morgan et al. 2022). Each contact transition must not only advance the object’s rotation but also leave the hand in a new configuration that support continued rotation about the desired axis. We refer to this configuration-dependent capability as contact-conditioned rotational manipulability. Sustained rotation therefore depends not only on the immediate rotational progress of each transition, but also on how this manipulability is maintained and reshaped over successive transitions. Despite its importance for generating rotation motion, the evolution of contact-conditioned rotational manipulability (hereafter referred to simply as manipulability) has received little attention in existing learning-based methods. For instance, reinforcement learning methods can discover effective contact transitions through extensive trial and error, but generally do not explicitly consider how manipulability evolves with the hand’s contact state (Qi et al. 2023; Yang et al. 2024). Similarly, imitation via demonstration-retargeting approaches transfer human motions into embodiment-specific robot trajectories (Qin et al. 2022; Li et al. 2025a), but have not yet incorporated contact-conditioned manipulability evolution as an explicit guidance or conditioning signal. Figure 1: DexMani overview. Human visual–tactile rotation demonstrations are used to pretrain an energy-based prior over contact-conditioned manipulability evolution. The frozen prior then guides downstream reinforcement learning across rotation tasks and dexterous hand embodiments. To bridge the gap, we propose DexMani, a framework that distills successful human rotation demonstrations into a prior over the short-horizon evolution of rotational manipulability, and uses it to guide downstream robot learning (Fig. 1). DexMani first learns an energy model that evaluates changes in manipulability based on patterns observed in human demonstrations. During robot control, the model scores the robot manipulability changes induced by candidate actions, and then generate residual action guidance that favors human-like evolution. Because this guidance is expressed through a manipulability representation that is transferable across embodiments, DexMani can generalize across rotation tasks and dexterous hands. To evaluate its performance, we conducted comprehensive evaluations of DexMani on three contact-rich rotation tasks: Unscrew Cap, Rotate Object, and Turn Faucet. The results demonstrate strong cross-embodiment and cross-task generalization, with DexMani achieving the highest success rates on both seen and unseen objects in every evaluated setting. It achieves an average success rate of 57.5% across the three LEAP Hand tasks and 43.4% for cap unscrewing across the Shadow, Allegro, and XHand embodiments. We additionally demonstrate closed-loop deployment on a physical dexterous hand, highlighting the practical applicability of the proposed approach to real-world robotic systems. In summary, our main contributions are: • We introduce contact-conditioned manipulability, a human-to-robot representation that captures how contact transitions reshape task-space rotational capabilities. • We develop an energy-based prior that evaluates whether changes in manipulability are consistent with human demonstrations and conducive to sustained rotation. • We propose and evaluate a reinforcement learning framework that transfers this shared prior across dexterous hands with different kinematics. 2 Related Work Learning-Based Dexterous In-Hand Rotation Learning-based methods generally acquire dexterous rotation skills from robot-generated interaction data, human demonstrations, or a combination of both. For instance, reinforcement learning has been applied to a variety of rotation tasks, including in-hand reorientation (Qi et al. 2023), tactile-only control (Yin et al. 2023), rapid finger gaiting (Wang et al. 2024), and articulated-object rotation (Yang et al. 2024). More recent work has explored transferable interaction representations learned from large-scale visual–tactile data (Ye et al. 2026; Huang et al. 2026; Zhang et al. 2026), leveraging privileged simulation information, policy distillation, and tactile perception to enable effective rotation skills. However, these methods must still discover effective contact transitions through either trial-and-error exploration or imitation of human demonstrations. They lack an explicit prior that encodes the desired motion trend throughout the rotation process. Human Demonstration Priors and Motion Transfer Recent research has increasingly used human demonstrations to guide robot learning. These approaches often convert demonstrations into hand-specific actions that a robot can reproduce. For example, a commonly used method is to retarget human motions into robot-specific joint commands, or fingertip trajectories (Qin et al. 2022; Li et al. 2025a). Recent work has further improved retargeting through contact priors, or by models for correcting retarget errors (Wu et al. 2026; Pan et al. 2026). Beyond retargeting, demonstration-guided reinforcement learning incorporates human-motion priors through policy pretraining, or additional reward objectives (Rajeswaran et al. 2018; Peng et al. 2021; Lum et al. 2025). However, these representations are often embodiment-specific: differences in hand kinematics and workspace necessitate retargeting or policy relearning for each new hand. In contrast, DexMani transfers an embodiment-agnostic prior over the desired evolution of manipulability, allowing each hand to realize this trend through its own actions and contact transitions. Manipulability and Cross-Embodiment Guidance Manipulability characterizes the directions of motion enabled by a hand configuration (Friedman and Flash 2007; Yokokohji et al. 2009). Prior work has investigated the geometry of manipulability ellipsoids (Jaquier et al. 2021), transferred desired manipulability profiles across embodiments (Jaquier et al. 2020; Reithmeir et al. 2022), and incorporated manipulability objectives into policy learning (Sun and Liao 2024; Li et al. 2025b). However, these methods do not explicitly account for how object contacts constrain manipulability during rotation. DexMani addresses this limitation with a contact-conditioned manipulability measure that captures a hand’s ability to sustain rotational motion under contact constraints, providing embodiment-agnostic guidance for contact-rich manipulation. 3 Methods Figure 2: DexMani pipeline. DexMani learns a contact-conditioned manipulability-evolution energy prior from human visual–tactile demonstrations (left). During robot reinforcement learning, the frozen prior scores action-induced manipulability changes and provides hand-specific guidance to a residual policy (right). DexMani aims to transfer a preference over how rotational capability evolves during manipulation. As shown in Fig. 2, it first learns such a preference prior from human demonstrations. Then, during downstream reinforcement learning, the learned prior is used to provide residual action guidance, encouraging the policy to follow the learned preference. 3.1 Human Demonstration Collection High-quality robot demonstrations of dexterous rotation are difficult to collect due to the intractability of accurately conveying contact perception during teleoperation (Qin et al. 2023; Luo et al. 2025). We therefore collect human demonstrations instead by having participants directly rotate the target objects via a tactile glove. For this purpose, we develop the data-collection system shown in Fig.3. A Meta Quest 3 controller (Meta 2023) attached to the back of the hand tracks the global hand pose. A Manus Quantum MetaGlove (Manus Meta 2024) captures finger joint poses, while a JQ-Industries tactile glove (Juqiao Industrial 2024) measures contact pressure across the fingertips using a piezoresistive pressure array. Using this device, We concurrently extract fingertip pressure measurements, as well as the hand pose, and RGB observations. In total, we collect over 100,000100,000 frames of human rotation data. These synchronized recordings allow us to determine active contact states and compute the corresponding contact-conditioned manipulability labels used for pretraining. Figure 3: Multimodal data collection system. We collect synchronized hand pose and tactile pressure using a Manus Quantum MetaGlove and a JQ-Industries tactile glove. 3.2 Analytic Manipulability from Human Using collected human data, we calculate the contact-conditioned manipulability introduced as follows. The overall pipeline for such calculation comprises three components: (1) a visual–tactile encoder that fuses RGB and tactile histories into a compact representation, (2) contact-conditioned manipulability, an analytic, tactile-gated manipulability descriptor that provides supervision labels from demonstrations, and (3) an auxiliary prediction decoder that is discarded after pretraining. We describe each component below. Visual–Tactile Encoder. Given a human demonstration sequence, the encoder maps these observations to (ztVT,hv,hc)=Fη(Vt−H+1:t,Tt−H+1:t),(z_t^VT,h_v,h_c)=F_η(V_t-H+1:t,T_t-H+1:t), (1) where visual and tactile histories are denoted as Vt−H+1:tV_t-H+1:t and Tt−H+1:tT_t-H+1:t, respectively, H is the history length, ztVTz_t^VT is the fused visual–tactile representation, and hvh_v and hch_c are modality-specific token features. Using this encoder, the RGB observations are embedded as visual patch tokens, while the tactile readings are projected into contact tokens. A learnable transformer-based (Vaswani et al. 2017) fusion network FηF_η then integrates the two modalities. Thus, ztVTz_t^VT compactly describes both the visually observed hand–object interaction and where contact is currently occurring. Contact-conditioned Manipulability. We introduce contact-conditioned manipulability as a cross-embodiment representation of rotational capability evolution. Conventional configuration based manipulability indices characterize a hand’s kinematic capabilities but do not explicitly capture how active contacts constrain object motion. To address this limitation, we introduce contact-conditioned manipulability defined as following. Let Jc,thumJ_c,t^hum denote the stacked Jacobian of the active fingertip contacts, where WtW_t denote the diagonal matrix of tactile-confidence weights, HhH_h is a diagonal joint-space scaling matrix, and Gt+G_t^+ the damped pseudoinverse of the grasp matrix. Extending the conventional manipulability formulation of (Chiacchio et al. 1991) with tactile-induced kinematic constraints, we further define the contact capability Cc,thumC_c,t^hum and rotational manipulability matrix Mω,thumM_ω,t^hum as Cc,thum C_c,t^hum =Wt1/2Jc,thumHh(Jc,thum)⊤Wt1/2, =W_t^1/2J_c,t^humH_h(J_c,t^hum) W_t^1/2, (2) Mω,thum M_ω,t^hum =Pω(Gt+)⊤Cc,thumGt+Pω⊤+ϵmI3. =P_ω(G_t^+) C_c,t^humG_t^+P_ω + _mI_3. where PωP_ω selects the rotational component and ϵmI3 _mI_3 ensures positive definiteness. We represent Mω,thumM_ω,t^hum with the log-Euclidean coordinate mthum=vech(logMω,thum)∈ℝ6.m_t^hum=vech ( M_ω,t^hum ) ^6. Although mtm_t is computed using each hand’s own contact Jacobian, we define the wrist-frame axes consistently across hands, so that its dimensions describe rotational capabilities about the same x-, y-, and z-directions. Contact positions are measured relative to the weighted center of the tactilely detected contacts, avoiding the need to estimate the object center. This shared task-space definition allows human evolution prior to be transferred to robot-induced changes without joint-level correspondence. 3.3 Manipulability-Evolution Energy Prior Using the contact-conditioned manipulability descriptors extracted from human demonstrations, we learn a prior over the evolution of manipulability during successful object rotation. The model is capable of evaluating candidate directions of manipulability change, assigning low energy to directions consistent with human demonstrations and high energy to incompatible directions. To train this prior model, we use three complementary components: (1) a contrastive energy objective, (2) structured negative sampling, and (3) an auxiliary objective for predicting the magnitude of the change. All these components are detailed as following. Contrastive Manipulability-Evolution Energy. We define a conditional energy model Eθ(v^∣ct)E_θ( v c_t) that measures the compatibility between a candidate manipulability-change direction v v and the current interaction context ct=[ztVT,mt,dt]c_t=[z_t^VT,m_t,d_t], where ztVTz_t^VT encodes the visual–tactile observations, mtm_t denotes the current manipulability state, and dtd_t specifies the desired rotation direction. A lower energy indicates that the candidate direction is more consistent with the manipulability evolution observed in human demonstrations. We obtain the positive direction from the demonstrated short-horizon change: v^t+=mt+Δ−mt∥mt+Δ−mt∥2+ϵnum. v_t^+= m_t+ -m_t m_t+ -m_t _2+ _num. (3) This produces a unit vector in six-dimensional manipulability space, allowing the energy model to focus on the direction in which manipulability should evolve. We exclude near-stationary frames from the directional objective because their normalized directions are dominated by measurement noise. Structured Negatives and Energy Loss. To teach the model which directions are incompatible with the current context, we follow the sampling-based contrastive strategy of IBC (Florence et al. 2021). We contrast each demonstrated direction with K structured negatives. These negatives comprise random directions, reversed directions, and directions drawn from mismatched coordinate frames. Random negatives provide broad coverage of the direction space, whereas reversed and mismatched directions provide incorrect instances within interaction context. Let t=v^t+∪v^t,k−k=1KV_t=\ v_t^+\∪\ v_t,k^-\_k=1^K denote the resulting candidate set. We optimize the contrastive energy loss ℒE=−logexp[−Eθ(v^t+∣ct)/τ]∑v^∈texp[−Eθ(v^∣ct)/τ],L_E=- \! [-E_θ( v_t^+ c_t)/τ ] _ v _t \! [-E_θ( v c_t)/τ ], (4) where τ is the contrastive temperature. This objective encourages the demonstrated direction to have lower energy than incompatible alternatives, thereby shaping an energy landscape that captures context-dependent manipulability evolution. Magnitude Prediction and Full Pretraining Loss. In addition to the aforementioned directional target (normalized, it does not preserve the magnitude of desired motion), we introduce an additional auxiliary head that predicts the log-scaled magnitude: αt=log(1+∥mt+Δ−mt∥2),ℒα=|α^t−αt|. _t= \! (1+ m_t+ -m_t _2 ), _α= α_t- _t . (5) Thus, the complete pretraining objective is ℒpre=λrecℒrec+λpredℒpred+λEℒE+λαℒα,L_pre= _recL_rec+ _predL_pred+ _EL_E+ _αL_α, (6) where ℒrecL_rec and ℒpredL_pred train the auxiliary decoder and learned representation, respectively, while ℒEL_E and ℒαL_α supervise the direction and magnitude of manipulability evolution. After pretraining, the visual–tactile encoder FηF_η and the energy model EθE_θ are frozen and subsequently used during robot reinforcement learning. 3.4 Manipulability Energy-Guided Residual Policy Finally, we transfer the manipulability-evolution preference learned from human demonstrations to robot control. Rather than directly executing the direction preferred by the energy model, we use it as a local action-space hint for a residual policy. This design preserves the task-solving capability of the base policy while biasing exploration toward actions that induce human-like, contact-aware manipulability evolution. Predicting Action-Induced Manipulability Change. At time t, the robot receives an RGB observation VtrV_t^r, tactile signals TtrT_t^r, and joint state qtq_t. The frozen visual–tactile encoder produces ztr=Fη(Vtr,Ttr),z_t^r=F_η(V_t^r,T_t^r), (7) and the base policy proposes a nominal continuous action at0∼πbase,ψ(⋅∣ztr,qt).a_t^0 _base,ψ (· z_t^r,q_t ). (8) We represent the robot’s current rotational manipulability using the same descriptor as for the human demonstrations: mtr=vech(logMω,tr)∈ℝ6,m_t^r=vech ( M_ω,t^r ) ^6, where vech(⋅)vech(·) stacks the six unique entries of a symmetric 3×33× 3 matrix. To evaluate how a candidate action changes manipulability, let ut(a,qt)u_t(a,q_t) denote the local joint displacement induced by action a under the robot controller. We define the resulting normalized manipulability-change direction as v^tr(a)=mr(qt+ut(a,qt))−mtr‖mr(qt+ut(a,qt))−mtr‖2+ϵ. v_t^r(a)= m^r\! (q_t+u_t(a,q_t) )-m_t^r \|m^r\! (q_t+u_t(a,q_t) )-m_t^r \|_2+ε. (9) This one-step estimate keeps the current contact geometry fixed, providing an efficient local approximation without requiring contact-dynamics rollouts. Contact changes are incorporated at the next control step through the updated tactile observations and robot state. Energy-Guided Action Bias. We define the robot-side energy context as ctr=[ztr,mtr,dt]c_t^r=[z_t^r,m_t^r,d_t], where dtd_t denotes the desired task rotation axis. To identify a locally preferred action without replacing the base policy, we construct a candidate set around its nominal action: t=at0∪clip(at0+ξt,k)k=1K,ξt,k∼(0,σa2I). gatheredA_t=\a_t^0\∪ \clip_A(a_t^0+ _t,k) \_k=1^K,\\ _t,k (0, _a^2I). gathered (10) Each candidate is mapped to its induced manipulability-change direction through Eq. (9) and evaluated by the frozen energy model. The lowest-energy candidate then defines a local action-space bias: at⋆=argmina∈tEθ(v^tr(a)∣ctr),btE=sg[clipbmax(at⋆−at0)]. gathereda_t = _a _tE_θ ( v_t^r(a) c_t^r ),\\ b_t^E=sg [clip_b_ (a_t -a_t^0 ) ]. gathered (11) Here, sg(⋅)sg(·) denotes the stop-gradient operator, and btEb_t^E indicates which local deviation from the nominal action is predicted to yield a more desirable manipulability evolution. Clipping limits the influence of this guidance and prevents the energy model from overriding the base policy. Residual Policy and Training. Because the one-step energy score captures only a local preference and does not account for long-horizon task outcomes, we do not execute at⋆a_t directly. Instead, we condition a learned residual policy on the energy-guided bias: Δat∼πres,ϕ(⋅∣ztr,qt,at0,mtr,sg(btE)),at=clip(at0+λRΔat). gathered a_t _res,φ (· z_t^r,q_t,a_t^0,m_t^r,sg(b_t^E) ),\\ a_t=clip_A (a_t^0+ _R a_t ). gathered (12) where λR _R controls the magnitude of the residual correction, and clip(⋅)clip_A(·) projects the final command onto the valid action space A. During robot learning, the encoder FηF_η and energy model EθE_θ remain frozen, while the base and residual policies are optimized solely using the original task reward. Notably, our framework guides policy learning through an embodiment-agnostic preference over manipulability evolution, which differs from conventional approaches that use human demonstrations for hand-specific action imitation, reward shaping, or online optimization. Additional definitions, hyperparameters, and implementation details are provided in the supplementary material. Method Unscrew Cap Rotate Object Turn Faucet Avg. SR Seen Unseen Seen Unseen Seen Unseen PPO (Schulman et al. 2017) 11.8±2.111.8± 2.1 5.0±1.95.0± 1.9 26.5±0.726.5± 0.7 21.5±0.621.5± 0.6 33.2±3.533.2± 3.5 20.5±1.020.5± 1.0 19.8 VT Pretraining 29.0±1.529.0± 1.5 12.7±3.112.7± 3.1 42.6±1.442.6± 1.4 34.0±4.634.0± 4.6 62.4±4.762.4± 4.7 53.5±4.853.5± 4.8 39.0 VTM 47.0±0.947.0± 0.9 27.8±1.827.8± 1.8 49.5±3.049.5± 3.0 37.9±3.437.9± 3.4 79.0±1.279.0± 1.2 70.1±3.270.1± 3.2 51.9 VTA 40.1±0.840.1± 0.8 25.1±1.725.1± 1.7 37.4±5.837.4± 5.8 20.3±1.820.3± 1.8 77.2±2.277.2± 2.2 63.5±7.763.5± 7.7 43.9 VTA-E 24.2±2.024.2± 2.0 11.6±3.611.6± 3.6 29.0±6.329.0± 6.3 19.1±3.019.1± 3.0 67.6±1.967.6± 1.9 42.0±5.142.0± 5.1 32.3 DexMani 65.1±2.465.1± 2.4 39.3±2.539.3± 2.5 50.7±2.250.7± 2.2 39.2±3.139.2± 3.1 80.2±2.080.2± 2.0 70.3±2.970.3± 2.9 57.5 Table 1: Peformance of generalization across tasks and objects. Using a single frozen human-derived prior, DexMani achieves the highest mean SR in all six task–split settings, including Turn Faucet, which is absent from human pretraining. Method Shadow Hand Allegro Hand XHand Avg. SR Seen Unseen Seen Unseen Seen Unseen PPO (Schulman et al. 2017) 20.8±2.620.8± 2.6 6.8±2.66.8± 2.6 11.8±1.611.8± 1.6 5.2±2.45.2± 2.4 32.5±3.332.5± 3.3 17.2±1.617.2± 1.6 15.7 VTM 66.4±4.466.4± 4.4 40.9±9.240.9± 9.2 24.3±3.924.3± 3.9 12.5±4.212.5± 4.2 58.0±1.458.0± 1.4 22.7±8.422.7± 8.4 37.5 VTA-E 52.7±3.252.7± 3.2 24.5±6.724.5± 6.7 17.6±3.517.6± 3.5 9.8±1.79.8± 1.7 45.2±2.845.2± 2.8 18.2±7.518.2± 7.5 28.0 DexMani 70.0±3.170.0± 3.1 48.2±4.848.2± 4.8 34.6±4.034.6± 4.0 14.8±2.314.8± 2.3 63.4±4.563.4± 4.5 29.5±1.929.5± 1.9 43.4 Table 2: Performance of Cross-hand generalization on cap-unscrewing task. The same human-derived DexMani prior is reused across the Shadow, Allegro, and XHand platforms, with a separate policy trained in each hand-specific action space. 4 Experiments We evaluate whether contact-conditioned manipulability evolution provides effective and reusable guidance for dexterous rotation. We first measure task success and generalization to held-out objects across three LEAP Hand tasks, as well as cap unscrewing with the Shadow, Allegro, and XHand, using the same frozen human-derived prior. We then conduct controlled ablations to isolate the benefit of state-aligned evolution guidance. Finally, we assess execution quality and demonstrate closed-loop deployment in the real world. 4.1 Experimental Setup We evaluate our approach on three contact-rich rotation tasks: (1) Unscrew Cap, (2) Rotate Object, and (3) Turn Faucet. A trial is considered successful if the cap, object, or faucet handle rotates by at least 2π2π about the target axis. We pretrain the human-derived preference prior using demonstrations of Unscrew Cap and Rotate Object. To assess cross-task generalization, we exclude Turn Faucet from prior pretraining. We further enforce two criteria to evaluate object-level generalization rigorously: (1) objects appearing in the human demonstrations are excluded from robot-policy training and evaluation; and (2) the objects for each robot task are divided into Seen objects used for policy training and Unseen objects reserved exclusively for evaluation. We conduct three categories of evaluation. (1) First, to determine whether the proposed prior improves task learning, we train a separate LEAP-hand policy for each task and evaluate its performance. (2) Second, to assess cross-task transfer, we train and evaluate a policy on Turn Faucet using the same frozen prior (this task is excluded from prior pretraining). (3) Third, to examine cross-hand transfer, we train Shadow, Allegro, and XHand policies for Unscrew Cap using the same task definition, reward function, and frozen preference prior. These evaluations collectively assess whether a single human-derived prior can facilitate policy learning across tasks, objects, and hand morphologies. We evaluate three aspects of performance: (1) task success, (2) generalization to unseen objects, and (3) execution quality. We report the success rate (SR, %) separately for Seen and Unseen objects. Avg. SR denotes the unweighted mean across all conditions. All results are reported as the mean ± standard deviation over three independent evaluation seeds. For each seed, we evaluate 1,0001,000 episodes per object with randomized initial hand poses. Method Unscrew Faucet Avg. SR Seen Unseen Seen Unseen Zero Guidance 18.618.6 9.49.4 48.648.6 20.220.2 24.2 Context-Shuffled 14.014.0 5.35.3 22.522.5 7.37.3 12.3 Greedy-M 54.554.5 36.336.3 67.867.8 44.744.7 50.8 DexMani 65.165.1 39.339.3 80.280.2 70.370.3 63.763.7 Table 3: Ablation Studies. All methods use the same training setup, differing only in the guidance signal. Method Unscrew Cap Rotate Object Turn Faucet TCI ↑ LDLJ ↑ SPARC ↑ TCI ↑ LDLJ ↑ SPARC ↑ TCI ↑ LDLJ ↑ SPARC ↑ PPO 0.29 ±0.11± 0.11 -25.6 ±1.4± 1.4 -6.0 ±0.5± 0.5 0.43±0.110.43± 0.11 -14.5 ±0.1± 0.1 -2.6 ±0.2± 0.2 0.22 ±0.05± 0.05 -23.2 ±2.3± 2.3 -4.8 ±0.8± 0.8 VT Pretrain 0.41 ±0.13± 0.13 -24.5 ±1.1± 1.1 -5.5 ±0.6± 0.6 0.41 ±0.23± 0.23 -14.4 ±0.1± 0.1 -2.8 ±0.2± 0.2 0.23 ±0.06± 0.06 -23.2 ±2.0± 2.0 -4.5 ±0.9± 0.9 VTM 0.44 ±0.17± 0.17 -24.1 ±0.8± 0.8 -5.3 ±0.7± 0.7 0.26 ±0.18± 0.18 -15.5 ±0.8± 0.8 -3.3 ±0.3± 0.3 0.27 ±0.15± 0.15 -23.4 ±1.8± 1.8 -4.4 ±0.4± 0.4 VTA 0.45 ±0.12± 0.12 -23.7 ±1.6± 1.6 -5.0 ±0.1± 0.1 0.29 ±0.09± 0.09 -15.6 ±0.9± 0.9 -3.1 ±0.2± 0.2 0.23 ±0.04± 0.04 -23.9 ±2.4± 2.4 -4.1 ±1.2± 1.2 VTA-E 0.43 ±0.05± 0.05 -23.6 ±0.4± 0.4 -5.0 ±0.3± 0.3 0.23 ±0.10± 0.10 -13.9 ±0.2± 0.2 -2.8 ±0.5± 0.5 0.20 ±0.02± 0.02 -24.8 ±0.4± 0.4 -4.5 ±0.8± 0.8 DexMani 0.50±0.130.50± 0.13 −23.1±0.5-23.1± 0.5 −4.5±0.4-4.5± 0.4 0.36 ±0.05± 0.05 −11.9±1.8-11.9± 1.8 −2.3±0.7-2.3± 0.7 0.28±0.070.28± 0.07 −22.4±0.5-22.4± 0.5 −3.9±0.1-3.9± 0.1 Table 4: Quantitative comparison of execution quality and efficiency across three dexterous rotation tasks on the LEAP hand. 4.2 Performance of the Proposed Approach Generalization across tasks and objects. We compare DexMani against five baselines: (1) PPO, which learns solely from the task reward; (2) VT Pretraining, which learns a visual–tactile policy representation from human demonstrations via pretraining, without manipulability supervision or online guidance; (3) VTM, which additionally predicts contact-conditioned manipulability during pretraining, but does not utilize manipulability during policy learning; (4) VTA, which additionally guides policy learning using retargeted human actions; and (5) VTA-E, which additionally learns an energy model over retargeted actions and provides guidance through the same residual interface as DexMani. The details, and additional differences between these baselines are reported in the supplementary material. The results are reported in Table 1 and reveal three main findings. First, DexMani achieves the highest SR across all six task–split conditions, attaining an Avg. SR of 57.5% and outperforming VTM, the strongest baseline, by 5.6 percentage points. Second, the improvements are particularly pronounced on Unseen objects, indicating that the learned guidance generalizes beyond the object instances and categories encountered during policy training. Third, DexMani performs strongly on Turn Faucet, despite this task being excluded from human-prior pretraining. This result demonstrates that the same frozen prior can transfer to a previously unseen rotation task. The baseline comparisons provide two additional insights. First, the limited performance of VT Pretraining and VTM indicates that representation pretraining on human demonstrations alone, even with manipulability supervision, is insufficient to fully exploit the transferred knowledge without online policy guidance. Second, the performance of VTA and VTA-E suggests that retargeted human actions transfer less effectively across embodiments than the proposed manipulability-based representation. In contrast, DexMani transfers a preference over how rotational manipulability should evolve, allowing each robot to realize this preference according to its own morphology, and kinematic constraints. Consequently, a single prior can guide multiple rotation tasks while generalizing to previously unseen objects. Generalization across hands. To assess whether the learned prior transfers across embodiments, we evaluate DexMani on three dexterous hands: (1) Shadow, (2) Allegro, and (3) XHand. For all three embodiments, we use the same Unscrew Cap task, reward function, and preference prior model, while training a separate policy in each hand’s native action space. As shown in Table 2, DexMani achieves the highest SR on both Seen and Unseen objects for all three hands. These consistent improvements demonstrate that manipulability-based guidance transfers across distinct hand kinematics. Figure 4: Learning and energy analysis. Left: Turn Faucet learning curves. Right: Projected action-energy landscape and selected actions under different guidance strategies. 4.3 Ablation Study To understand why the manipulability preference prior improves policy performance, we investigate three questions: (1) Does manipulability-based guidance improve action selection? (2) If so, must this guidance be conditioned on the correct interaction context? (3) Is guiding a residual policy using the short-horizon evolution of human manipulability preferences more effective than greedily maximizing the robot’s instantaneous rotational manipulability? We address these questions using three ablations: Zero Guidance, which removes guidance from the residual policy’s input; Context-Shuffled, which replaces the correct guidance with mismatched interaction context; and Greedy-M, replaces the learned evolution guidance with a bias toward the candidate that maximizes instantaneous rotational manipulability, while retaining the same residual-policy interface. As shown in Table 3, DexMani achieves the highest success rate across all conditions. Its improvements over Zero Guidance and Context-Shuffled confirm the benefits of manipulability-based guidance matched to the interaction context. The advantage over Greedy-M shows that the gain does not come from simply increasing instantaneous rotational capability. Instead, guidance learned from successful manipulability evolution provides a more effective signal for sustained rotation. Consistently, Figure 4 shows that DexMani learns faster and performs best on Turn Faucet. The action–energy landscape further reveals that DexMani selects an action in a low-energy region, whereas Greedy-M is only partially aligned with the human-derived prior. These results demonstrate the importance of interaction-conditioned, temporally informed guidance. 4.4 Motion Quality and Deployment Beyond task success, we further assess execution quality in terms of rotational capability and motion smoothness. We quantify rotational capability using the Task Compatibility Index (TCI) (Chiu 1988), which measures the ability of the hand–object system to generate motion about the target rotation axis; a higher TCI indicates greater task-aligned rotational capability. We evaluate motion smoothness in both the time and frequency domains using log dimensionless jerk (LDLJ) (Hogan and Sternad 2009) and spectral arc length (SPARC) (Balasubramanian et al. 2015), respectively. Higher LDLJ and SPARC values indicate smoother motion. As shown in Table 4, DexMani achieves the best LDLJ and SPARC scores across all three tasks, indicating smoother and more coordinated object motions. It also obtains the highest TCI on Unscrew Cap and Turn Faucet, demonstrating a stronger ability to generate rotational motion aligned with the target axis. Figure 5: DexMani real-world deployment. The sequence shows closed-loop execution at 20Hz20\,Hz in the real world. Method Unscrew Cap Rotate Object Turn Faucet DexMani 6/10 3/10 1/10 Table 5: Real-world DexMani deployment results. Next, we deploy DexMani on a physical system consisting of a 16-DoF LEAP Hand (Shaw et al. 2023) mounted on a 6-DoF xArm, as shown in Figure 5. The fingertips are equipped with TwinTac tactile sensors (Huang et al. 2025), whose measurements are binarized to indicate contact. At each control step, the policy integrates these tactile signals with visual and proprioceptive observations to predict joint-position targets. To facilitate sim-to-real transfer, we apply domain randomization (Tobin et al. 2017) during simulation training, improving robustness to variations in physical dynamics and sensory observations. As reported in Table 5, DexMani successfully performs closed-loop rotation tasks on the physical system, demonstrating its ability to transfer from simulation to the real world despite discrepancies in sensing and dynamics. 5 Conclusion We introduced DexMani, a framework that transfers human rotational manipulation experience through contact-conditioned manipulability evolution. By representing human preferences as manipulability evolution rather than hand-specific joint trajectories, DexMani improves task success, motion smoothness, and rotational capability while generalizing across tasks and objects. We also demonstrated its successful deployment on a physical LEAP Hand. These results establish manipulability evolution as a transferable representation of human experience across tasks, objects, and hand morphologies. One main limitation of this work is that, although the prior is shared, each task–hand configuration still requires a separately trained policy, precluding direct transfer of a unified policy across embodiments. Future work will focus on improving Sim2Real performance and developing unified policies that support direct cross-embodiment transfer. Acknowledgment This work was supported by the National Natural Science Foundation of China (Grant No. 52305007), the Natural Science Foundation of Shanghai (Grant No. 25ZR1402370), the Artificial Intelligence Project of the State Key Laboratory of General Artificial Intelligence, BIGAI, Peking University, Beijing, China (Project No. SKLAGI2025OP19), the State Key Laboratory of Mechanical System and Vibration (Grant No. MSV202519) and the MoE Key Laboratory of Intelligent Perception and Human-Machine Collaboration (KLIP-HuMaCo). References S. Balasubramanian, A. Melendez-Calderon, A. Roby-Brami, and E. Burdet (2015) On the analysis of movement smoothness. Journal of neuroengineering and rehabilitation 12 (1), p. 112. Cited by: 4.4 Motion Quality and Deployment. B. Calli, A. Walsman, A. Singh, S. Srinivasa, P. Abbeel, and A. M. Dollar (2015) Benchmarking in manipulation research: using the yale-cmu-berkeley object and model set. IEEE Robotics Automation Magazine 22 (3), p. 36–52. External Links: ISSN 1070-9932, Link, Document Cited by: Rotate Object.. P. Chiacchio, S. Chiaverini, L. Sciavicco, B. Siciliano, et al. (1991) Global task space manipulability ellipsoids for multiple-arm systems. IEEE Transactions on Robotics and Automation 7 (5), p. 678–685. Cited by: Contact-conditioned Manipulability.. S. L. Chiu (1988) Task compatibility of manipulator postures. The international journal of robotics research 7 (5), p. 13–21. Cited by: 4.4 Motion Quality and Deployment. P. Florence, C. Lynch, A. Zeng, O. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson (2021) Implicit behavioral cloning. External Links: 2109.00137, Link Cited by: Structured Negatives and Energy Loss.. J. Friedman and T. Flash (2007) Task-dependent selection of grasp kinematics and stiffness in human object manipulation. Cortex 43 (3), p. 444–460. Cited by: Manipulability and Cross-Embodiment Guidance. N. Hogan and D. Sternad (2009) Sensitivity of smoothness measures to movement duration, amplitude and arrests. Journal of motor behavior 41, p. 529 – 534. External Links: Link Cited by: 4.4 Motion Quality and Deployment. P. Hsu, Z. Li, and S. Sastry (1988) On grasping and coordinated manipulation by a multifingered robot hand. In Proceedings. 1988 IEEE International Conference on Robotics and Automation, Vol. , p. 384–389 vol.1. External Links: Document Cited by: Tactile processing.. X. Huang, Z. Xu, and C. Xiao (2025) TwinTac: a wide-range, highly sensitive tactile sensor with real-to-sim digital twin sensor model. In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vol. , p. 12286–12292. External Links: Document Cited by: 4.4 Motion Quality and Deployment. Y. Huang, J. Wu, J. Jiang, H. Lin, A. Aierken, Y. Wang, K. Cheng, Z. Jiao, and Y. Zhong (2026) HT-bench: benchmarking and learning dexterous full-hand tactile representations with egocentric vision. arXiv preprint arXiv:2606.19161. Cited by: Learning-Based Dexterous In-Hand Rotation. N. Jaquier, L. Rozo, D. G. Caldwell, and S. Calinon (2021) Geometry-aware manipulability learning, tracking, and transfer. The International Journal of Robotics Research 40 (2-3), p. 624–650. Cited by: Manipulability and Cross-Embodiment Guidance. N. Jaquier, L. Rozo, and S. Calinon (2020) Analysis and transfer of human movement manipulability in industry-like activities. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 11131–11138. Cited by: Manipulability and Cross-Embodiment Guidance. Juqiao Industrial (2024) Juqiao Tactile Glove. Note: Flexible piezoresistive fiber sensors for robotics. https://jq-industries.com/ Cited by: 3.1 Human Demonstration Collection. Y. LeCun, S. Chopra, R. Hadsell, A. Ranzato, and F. J. Huang (2006) A tutorial on energy-based learning. External Links: Link Cited by: 2.2 Robot-Side Residual Action. K. Li, P. Li, T. Liu, Y. Li, and S. Huang (2025a) Maniptrans: efficient dexterous bimanual manipulation transfer via residual learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, Cited by: 1 Introduction, Human Demonstration Priors and Motion Transfer. Z. Li, J. Liu, D. Li, T. Teng, M. Li, S. Calinon, D. Caldwell, and F. Chen (2025b) ManiDP: manipulability-aware diffusion policy for posture-dependent bimanual manipulation. In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vol. , p. 9956–9962. External Links: Document Cited by: Manipulability and Cross-Embodiment Guidance. Q. Liu, Y. Cui, Z. Sun, G. Li, J. Chen, and Q. Ye (2025) VTDexManip: a dataset and benchmark for visual-tactile pretraining and dexterous manipulation with reinforcement learning. In ICLR, Cited by: 4.1 Task Details, VT pretrain.. T. G. W. Lum, O. Y. Lee, K. Liu, and J. Bohg (2025) Crossing the human-robot embodiment gap with sim-to-real rl using one human demonstration. In Conference on Robot Learning, p. 4418–4441. Cited by: Human Demonstration Priors and Motion Transfer. S. Luo, Q. Peng, J. Lv, K. Hong, K. R. Driggs–Campbell, C. Lu, and Y. Li (2025) Human-agent joint learning for efficient robot manipulation skill acquisition. In 2025 IEEE International Conference on Robotics and Automation (ICRA), Vol. , p. 1370–1377. External Links: Document Cited by: 3.1 Human Demonstration Collection. V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State (2021) Isaac gym: high performance gpu-based physics simulation for robot learning. External Links: 2108.10470, Link Cited by: 4.1 Task Details. Manus Meta (2024) Manus VR Gloves. Note: https://w.manus-meta.com/Accessed: 2026-04 Cited by: 3.1 Human Demonstration Collection. Meta (2023) Meet meta quest 3, our mixed reality headset. Note: https://about.fb.com/news/2023/09/meet-meta-quest-3-mixed-reality-headset/Accessed: 2026-07-27 Cited by: 3.1 Human Demonstration Collection. A. S. Morgan, K. Hang, B. Wen, K. Bekris, and A. M. Dollar (2022) Complex in-hand manipulation via compliance-enabled finger gaiting and multi-modal planning. IEEE Robotics and Automation Letters 7 (2), p. 4821–4828. Cited by: 1 Introduction. C. Pan, C. Wang, H. Qi, Z. Liu, H. Bharadhwaj, A. Sharma, T. Wu, G. Shi, J. Malik, and F. Hogan (2026) SPIDER: scalable physics-informed dexterous retargeting. External Links: 2511.09484, Link Cited by: Human Demonstration Priors and Motion Transfer. X. B. Peng, Z. Ma, P. Abbeel, S. Levine, and A. Kanazawa (2021) AMP: adversarial motion priors for stylized physics-based character control. ACM Trans. Graph. 40 (4). External Links: Link, Document Cited by: Human Demonstration Priors and Motion Transfer. H. Qi, B. Yi, S. Suresh, M. Lambeta, Y. Ma, R. Calandra, and J. Malik (2023) General In-Hand Object Rotation with Vision and Touch. In Conference on Robot Learning (CoRL), Cited by: 1 Introduction, Learning-Based Dexterous In-Hand Rotation. Y. Qin, Y. Wu, S. Liu, H. Jiang, R. Yang, Y. Fu, and X. Wang (2022) DexMV: imitation learning for dexterous manipulation from human videos. In Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIX, Berlin, Heidelberg, p. 570–587. External Links: ISBN 978-3-031-19841-0, Link, Document Cited by: 1 Introduction, Human Demonstration Priors and Motion Transfer. Y. Qin, W. Yang, B. Huang, K. Van Wyk, H. Su, X. Wang, Y. Chao, and D. Fox (2023) AnyTeleop: a general vision-based dexterous robot arm-hand teleoperation system. In Robotics: Science and Systems, Cited by: 3.1 Human Demonstration Collection, VTA-E.. A. Rajeswaran, V. Kumar, A. Gupta, 2. Vezzani, J. Schulman, E. Todorov, and S. Levine (2018) Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. In Proceedings of Robotics: Science and Systems (RSS), Cited by: Human Demonstration Priors and Motion Transfer. A. Reithmeir, L. Figueredo, and S. Haddadin (2022) Human-to-robot manipulability domain adaptation with parallel transport and manifold-aware icp. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vol. , p. 5218–5225. External Links: Document Cited by: Manipulability and Cross-Embodiment Guidance. J. Romero, D. Tzionas, and M. J. Black (2017) Embodied hands: modeling and capturing hands and bodies together. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia) 36 (6). Cited by: Visual processing.. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: Table 1, Table 2. K. Shaw, A. Agarwal, and D. Pathak (2023) LEAP hand: low-cost, efficient, and anthropomorphic hand for robot learning. In Robotics: Science and Systems, External Links: Document Cited by: 4.4 Motion Quality and Deployment. D. Sun and Q. Liao (2024) A framework of robot manipulability learning and control and its application in telerobotics. IEEE Transactions on Fuzzy Systems 32 (1), p. 266–280. External Links: Document Cited by: Manipulability and Cross-Embodiment Guidance. J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel (2017) Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), p. 23–30. Cited by: 4.4 Motion Quality and Deployment, Domain Randomization.. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, p. 6000–6010. External Links: ISBN 9781510860964 Cited by: Visual–Tactile Encoder.. J. Wang, Y. Yuan, H. Che, H. Qi, Y. Ma, J. Malik, and X. Wang (2024) Lessons from learning to spin “pens”. In CoRL, Cited by: Learning-Based Dexterous In-Hand Rotation. J. Wu, S. Yao, G. He, X. Liu, Z. Zeng, X. Jiang, H. Yang, W. Zhang, and H. Zhao (2026) TopoRetarget: interaction-preserving retargeting for dexterous manipulation. External Links: 2606.16272, Link Cited by: Human Demonstration Priors and Motion Transfer. F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, L. Yi, A. X. Chang, L. J. Guibas, and H. Su (2020) SAPIEN: a simulated part-based interactive environment. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: Turn Faucet.. M. Yang, chenghua lu, A. Church, Y. Lin, C. J. Ford, H. Li, E. Psomopoulou, D. A.W. Barton, and N. F. Lepora (2024) AnyRotate: gravity-invariant in-hand object rotation with sim-to-real touch. In 8th Annual Conference on Robot Learning, External Links: Link Cited by: 1 Introduction, Learning-Based Dexterous In-Hand Rotation. Q. Ye, Q. Liu, S. Wang, J. Chen, Y. Cui, K. Jin, H. Chen, X. Cai, G. Li, and J. Chen (2026) Visual-tactile pretraining and online multitask learning for humanlike manipulation dexterity. Science Robotics 11 (110). External Links: Document, Link, https://w.science.org/doi/pdf/10.1126/scirobotics.ady2869 Cited by: Learning-Based Dexterous In-Hand Rotation. Z. Yin, B. Huang, Y. Qin, Q. Chen, and X. Wang (2023) Rotating without seeing: towards in-hand dexterity through touch.. In Robotics: Science and Systems, External Links: ISBN 978-0-9923747-9-2, Link Cited by: Learning-Based Dexterous In-Hand Rotation. Y. Yokokohji, J. San Martin, and M. Fujiwara (2009) Dynamic manipulability of multifingered grasping. IEEE Transactions on Robotics 25 (4), p. 947–954. Cited by: Manipulability and Cross-Embodiment Guidance. C. Zhang, P. Cai, Z. Xi, H. Yuan, H. Luo, W. Zhang, S. Zheng, C. Xu, and Z. Lu (2026) Human-centric transferable tactile pre-training for dexterous robotic manipulation. External Links: 2607.01067, Link Cited by: Learning-Based Dexterous In-Hand Rotation. Supplementary Material Supplementary Material for DexMani Written by AAAI Press Staff1With help from the AAAI Publications Committee. AAAI Style Contributions by Peter Patel Schneider, Sunil Issar, J. Scott Penberthy, George Ferguson, Hans Guesgen, Francisco Cruz , Marc Pujol-Gonzalez 1 Human Dataset and Hardware Details Human demonstrations provide supervision for both the visual–tactile encoder and the prior model of contact-conditioned rotational manipulability evolution. To explain how these data are collected, we briefly describe the data collection system and the processing procedure of the dataset. Data collection system. We collect synchronized multimodal recordings of human object rotation using a wearable acquisition system. The system captures RGB video, right-hand kinematics, and dense tactile measurements at 30 Hz. Hand kinematics are recorded using a Manus Quantum MetaGlove (Fig. 6). The Manus hand model provides the three-dimensional positions of 21 hand joints. Contact signals are captured using a 256-channel Juqiao piezoresistive tactile glove. At each frame, the tactile glove outputs a pressure vector ut∈0,…,255u_t∈\0,…,255\, where each element corresponds to a tactile taxel and larger values indicate stronger local pressure responses. Two RGB cameras record the hand–object interaction at a resolution of 640×480640× 480. All sensor streams are temporally aligned using timestamps. Figure 6: Hardware for human data collection. Left: the Juqiao tactile glove. Right: the Manus Quantum MetaGlove. As shown in Fig. 7, the human-demonstration dataset contains 27 objects, including common geometric primitives and everyday household items. This collection encompasses diverse object geometries and contact patterns for learning the evolution of rotational manipulability. Figure 7: Objects used for human demonstration collection. Visual processing. We use the recorded human-hand trajectories to animate a MANO hand model (Romero et al. 2017). The resulting hand motions are rendered in simulation to provide additional RGB observations. During pretraining, both the captured RGB images and the MANO-rendered images are used as visual inputs. Tactile processing. We partition the raw tactile taxels into five semantic fingertip regions corresponding to the thumb, index, middle, ring, and little fingers. For each finger i, we aggregate the responses of the taxels assigned to its fingertip region. The corresponding taxel set, iS_i, is determined by the physical layout of the Juqiao tactile glove. The finger-level tactile response is defined as fi,t=∑ℓ∈i|uℓ,t|.f_i,t= _ _i |u_ ,t |. (13) Figure 8: Human Dataset Visualization. Each pair shows the RGB observation and the corresponding tactile-pressure map. We compute a contact threshold separately for each recording r: τr=0.15Percentile95(fi,t∣i∈1,…,5,t∈r). _r=0.15\,Percentile_95 ( \f_i,t i∈\1,…,5\,\,t∈ r \ ). (14) The binary contact state is then given by ci,t=[fi,t>τr].c_i,t=I [f_i,t> _r ]. (15) Only fingertips for which ci,t=1c_i,t=1 are included when constructing the contact Jacobian and grasp matrix (Hsu et al. 1988). For visual–tactile pretraining, the tactile input to the encoder is Tt=[(c1,t),…,(c5,t)].T_t= [(c_1,t),…,(c_5,t) ]. (16) Thus, the encoder receives five finger-level tokens rather than the raw tactile vector. During RL training, we do not match individual taxels across different hands. Instead, we group the tactile sensors by finger and convert each finger’s response into a binary contact state. We use the same contact threshold for all robot hands, setting the simulated force threshold to 0.01N0.01\,N. For four-fingered hands, the little-finger entry is set to zero. Consequently, the method requires only finger-level contact information and does not rely on identical sensor layouts, taxel-level correspondence, or contact-normal estimation. 2 Additional Method Details This section provides the mathematical details omitted from the main text for clarity. We first describe how human manipulability labels are analytically computed from hand kinematics and tactile contacts. We then detail the manipulability-evolution energy objective and the robot-side realization of the residual action guidance. 2.1 Contact-conditioned Manipulability We compute contact-conditioned manipulability from the human hand configuration and the tactile contact state. The computation contains three steps. We first construct the human hand Jacobian from the Manus hand model. Then we modulate the Jacobians with the active contact information. Finally, we map the resulting motion capability to the object rotational space. Human hand Jacobian. The Manus glove provides hand poses at each frame. Each finger is represented as a serial kinematic chain. For an active fingertip contact i, its positional Jacobian is obtained by differentiating the forward kinematics of the corresponding finger: Ji,t=∂pi,t∂qt,J_i,t= ∂ p_i,t∂ q_t, (17) where qtq_t denotes the human hand configuration and pi,tp_i,t denotes the fingertip position. We stack the Jacobians of all active contacts into Jc,thum=[J1,t⋮JNt,t],p˙t=Jtq˙t,J_c,t^hum= bmatrixJ_1,t\\ \\ J_N_t,t bmatrix, p_t=J_t q_t, (18) where NtN_t is the number of active contacts. We compute the human contact-conditioned rotational manipulability as Cc,thum C_c,t^hum =Wt1/2Jc,thumHh(Jc,thum)⊤Wt1/2, =W_t^1/2J_c,t^humH_h(J_c,t^hum) W_t^1/2, (19) Mω,thum M_ω,t^hum =Pω(Gt+)⊤Cc,thumGt+Pω⊤+ϵmI3. =P_ω(G_t^+) C_c,t^humG_t^+P_ω + _mI_3. The terms in Eq. (19) are defined below. Contact activation matrix WtW_t. In the general formulation, WtW_t is a block-diagonal matrix that removes inactive fingertip contacts. In our implementation, inactive contacts are removed before constructing Jc,thumJ_c,t^hum and GtG_t. Therefore, Wt=I3Nt,W_t=I_3N_t, (20) and no additional contact masking is applied. Joint-space scaling matrix HhH_h. The matrix Hh=diag(h1,…,hnh)H_h=diag(h_1,…,h_n_h) controls the contribution of each human joint direction. A larger hjh_j gives the j-th joint direction a larger contribution to Cc,thumC_c,t^hum. In our implementation, we set Hh=InhH_h=I_n_h and weight all joint directions equally. Grasp matrix GtG_t. Let pi,tp_i,t be the position of active contact i∈ti _t in the wrist frame. We use the center of the active contacts as the local reference point: p¯t=1Nt∑i∈tpi,t. p_t= 1N_t _i _tp_i,t. (21) The relative contact position is ri,t=pi,t−p¯t.r_i,t=p_i,t- p_t. (22) The standard grasp matrix is Gt=[I3⋯I3[r1,t]×⋯[rNt,t]×],G_t= bmatrixI_3&·s&I_3\\ [r_1,t]_×&·s&[r_N_t,t]_× bmatrix, (23) where [⋅]×[·]_× denotes the skew-symmetric matrix. We use the damped pseudoinverse Gt+=Gt⊤(GtGt⊤+λGI6)−1.G_t^+=G_t (G_tG_t + _GI_6 )^-1. (24) The term (Gt+)⊤Cc,thumGt+(G_t^+) C_c,t^humG_t^+ maps the contact-space motion capability into the object-motion space. The grasp mapping first produces a 6×66× 6 object-space manipulability matrix. Its first three dimensions describe translation, and its last three dimensions describe rotation. We use Pω=[03×3I3]P_ω= bmatrix0_3× 3&I_3 bmatrix (25) to select the rotational component. The resulting Mω,thum∈ℝ3×3M_ω,t^hum ^3× 3 describes the available object rotation about the three wrist-frame axes. Finally, we convert the rotational manipulability matrix into a six-dimensional descriptor: mthum=vech(logMω,thum)∈ℝ6.m_t^hum=vech ( M_ω,t^hum ) ^6. (26) DexMani learns the normalized short-horizon changes of mthumm_t^hum. These changes describe how rotational capability evolves with the hand configuration and active contacts. Cross-embodiment convention. Here, we explain why the contact-conditioned manipulability enables cross-embodiment applications. Specifically, the descriptor mt=vech(logMω,t)m_t=vech( M_ω,t) is not invariant to arbitrary coordinate changes. We instead use a shared wrist-frame convention for human demonstrations and robot rollouts. Fingertip positions and Jacobians are expressed in wrist frame before manipulability is computed. In all experiments, the target rotation direction is represented by the hand-frame z-axis, i.e., d=[0,0,1]d=[0,0,1]. Tactile readings are binarized and used only to select active fingertip contacts. After this alignment, the components of mtm_t refer to the same wrist-frame rotation axes for both the human and the robot. The human-trained energy model can therefore score the manipulability changes induced by robot actions in the same coordinate system. No joint-level correspondence or action retargeting is required. Each robot produces a preferred change through its own joint motion and contact transitions. 2.2 Robot-Side Residual Action Algorithm 1 Robot-side update at one control step. 0: Frozen encoder FηF_η and energy model EθE_θ 0: Robot observation (Vtr,Ttr,qt)(V_t^r,T_t^r,q_t) and direction dtd_t 1: Compute the visual–tactile feature ztr=Fη(Vtr,Ttr)z_t^r=F_η(V_t^r,T_t^r). 2: Compute the current descriptor mtr=vech(logMω,tr)m_t^r=vech( M_ω,t^r). 3: Sample the nominal action at0∼πbase,ψ(⋅∣ztr,qt)a_t^0 _base,ψ(· z_t^r,q_t). 4: Construct the local candidate set t=at0∪clip(at0+ξk)k=1K.A_t=\a_t^0\∪ \clip_A(a_t^0+ _k) \_k=1^K. 5: Estimate v^tr(a) v_t^r(a) for each candidate using the current contact geometry. 6: Discard candidates with near-zero predicted manipulability change. 7: Set the energy context ctr=[ztr,mtr,dt]c_t^r=[z_t^r,m_t^r,d_t]. 8: Score each remaining candidate by St(a)=Eθ(v^tr(a)∣ctr)S_t(a)=E_θ( v_t^r(a) c_t^r). 9: Select at⋆=argmina∈tSt(a)a_t = _a _tS_t(a). 10: Compute the stop-gradient bias btE=sg[clipbmax(at⋆−at0)].b_t^E=sg [clip_b_ (a_t -a_t^0) ]. 11: Sample the residual action Δat∼πres,ϕ(⋅∣ztr,qt,at0,mtr,btE). a_t _res,φ(· z_t^r,q_t,a_t^0,m_t^r,b_t^E). 12: Execute at=clip(at0+λRΔat).a_t=clip_A(a_t^0+ _R a_t). 13: Update πbase,ψ _base,ψ and πres,ϕ _res,φ with PPO. This section provides the implementation details used in our methods. At each control step, the base policy produces a normalized action at0a_t^0. We sample eight nearby actions around at0a_t^0. Each candidate action is rescaled to a target joint configuration in Isaac Gym. For a candidate a, we compute the joint displacement ut(a,qt)=qttar(a)−qt,u_t(a,q_t)=q_t^tar(a)-q_t, (27) where qtq_t is the current joint configuration and qttar(a)q_t^tar(a) is the target specified by a. We use the robot fingertip Jacobian to calculate the resulting fingertip motion: Jtip,t=∂ptip(qt)∂qt,Δpt(a)=Jtip,tut(a,qt). gatheredJ_tip,t= ∂ p_tip(q_t)∂ q_t,\\ p_t(a)=J_tip,t\,u_t(a,q_t). gathered (28) The candidate contact positions are obtained by adding Δpt(a) p_t(a) to the current positions of the active fingertips, the contact Jacobian is recomputed at the candidate joint configuration qttar(a)q_t^tar(a), these contact positions and Jacobian are then used to construct the grasp matrix and the corresponding rotational manipulability Mω,tr(a)M_ω,t^r(a). Finally, we convert Mω,tr(a)M_ω,t^r(a) into its log-Euclidean descriptor and normalize its difference from the current descriptor to obtain v^tr(a) v_t^r(a), following Eq. (9) in the main paper. The frozen energy-based model (EBM) (LeCun et al. 2006) scores the manipulability change induced by each of the nine candidates, including the nominal action. A lower energy indicates better agreement with the human-derived manipulability-evolution prior. 2.3 Computational Cost. We profile one training iteration on an NVIDIA RTX 4090. PPO takes 5.775.77 s per iteration, while DexMani takes 11.5011.50 s. The detailed runtime breakdown is shown in Table 6. The local manipulability computation and EBM scoring together take 0.4340.434 s. Method Time (s) PPO 5.773 VT Pretraining 9.256 VTM 10.447 VTA 9.436 VTA-E 10.823 DexMani 11.497 DexMani components Local manipulability computation 0.410 EBM context and candidate scoring 0.024 Total local guidance computation 0.434 Table 6: Runtime profiling. 3 Training Details. We implement all baseline methods under the same task environments, object splits, reward functions, and PPO settings. The methods differ only in the human supervision used during pretraining and whether online guidance is provided during RL. VT pretrain. We implement VT following the visual–tactile joint pretraining method of (Liu et al. 2025). As shown in Fig. 9, each training sample contains a visual–tactile history,(Vt−H+1:t,Tt−H+1:t)(V_t-H+1:t,T_t-H+1:t), with H=8H=8. Each RGB frame is divided into image patches, while each tactile frame is represented by finger level tokens. Tokens from the temporal window are randomly masked, and a transformer encoder fuses the remaining visual and tactile tokens. A lightweight decoder reconstructs the masked inputs.The network and training settings are listed in Table 7.After pretraining, the decoder is discarded and the encoder is frozen. The fused visual-tactile feature is concatenated with robot proprioception and used as input to the PPO policy. Figure 9: VT pretraining pipeline. Hyperparameter All Models History length H 88 Image resolution per frame (Hv,Wv)(H_v,W_v) (224,224)(224,224) Image patches per frame NvN_v 196196 Tactile tokens per frame NcN_c 55 Patch resolution (P,P)(P,P) (16,16)(16,16) Encoder embedding dimension dend_en 384384 Image mask ratio γv _v 0.750.75 Tactile mask ratio γc _c 0.500.50 Fusion dimension d 384384 Decoder embedding dimension dded_de 192192 Image loss weight λv _v 11 Tactile loss weight λc _c 1010 Learning rate 1.5×10−41.5× 10^-4 Batch size 10241024 Table 7: Hyperparameters of visual–tactile pretraining. VTM. As shown in Fig. 10, VTM extends VT with manipulability prediction during pretraining. The current contact-conditioned rotational manipulability and task axis are added to the context ctc_t. predicts four future manipulability descriptors sampled at a stride of four raw frames. The model is trained with manipulability, direction, and magnitude prediction losses. The energy-loss weight is set to zero, so VTM does not learn an energy field. The detailed settings are listed in Table 8. Figure 10: VTM pretraining pipeline. Hyperparameters Value Future prediction horizon 44 Manipulability loss weight 1.01.0 Direction loss weight 0.20.2 Magnitude loss weight 0.10.1 Energy loss weight 0.00.0 Batch size 10241024 Learning rate 1.5×10−41.5× 10^-4 Table 8: Hyperparameters of VTM. VTA. As shown in Fig. 11, VTA follows the same pretraining and downstream protocol as VTM. The main difference is the supervision used during pretraining. VTA replaces the manipulability token in the context ctc_t with the human action recorded in the dataset. The prediction head also predicts future human actions instead of future manipulability. All other pretraining settings, including the prediction horizon, loss weights, training epochs, batch size, and learning rate, are the same as those of VTM. Figure 11: VTA pretraining pipeline. VTA-E. As shown in Fig. 12, VTA-E follows the same energy-guided training procedure as VTM-E. The main difference is the representation used by the energy model. During pretraining, VTA-E learns an energy field over human actions. Its auxiliary prediction head also predicts future human actions instead of future manipulability. During downstream RL, both the visual-tactile encoder and the human-action energy model remain frozen. The current robot-hand pose and the poses induced by candidate actions are retargeted to the human-hand space using AnyTeleop (Qin et al. 2023). The energy model then scores the corresponding human-action changes, following the same procedure as VTM-E. Figure 12: VTA-E pretraining pipeline. Manipulability Evolution Energy Learning. During pretraining, manipulability descriptors are sampled every four raw frames. We use the normalized change from mtm_t to mt+4m_t+4 as the positive evolution direction. Each positive sample is paired with eight negatives, including its reversed direction, and randomly sampled directions. Conditioned the context ctc_t, the energy head assigns a scalar energy to each candidate direction, with lower values indicating better agreement with the human demonstrations. After pretraining, the visual-tactile encoder, context encoder, and energy head are frozen for downstream RL The detailed settings are listed in Table 9 Hyperparameter Value Temporal offset Δ 4 Prediction horizon 4 Number of negatives K 8 Negative composition 8 Stationary threshold 10−410^-4 Temperature τ 0.1 Energy loss weight λE _E 0.1 Magnitude loss weight λα _α 0.1 Learning rate 1.5×10−41.5× 10^-4 Batch size 1024 Training epochs 400 Table 9: Hyperparameters for energy pretraining. PPO Training Details. All downstream policies are trained with PPO using the same optimization settings. We run 200 parallel environments and each policy update uses rollouts of 32 steps. The collected samples are divided into four mini-batches and optimized for ten epochs. The actor and critic use multilayer perceptrons with hidden dimensions [1024,1024,512][1024,1024,512]. ELU is used as the activation function. The learning rate is 3×10−43× 10^-4. The PPO clipping range is 0.20.2, and the gradient norm is clipped at 1.01.0. We use a discount factor of γ=0.96γ=0.96 and a GAE parameter of λGAE=0.95 _GAE=0.95. The initial policy noise standard deviation is 0.80.8. The desired KL divergence is 0.0160.016, and no entropy bonus is applied. The maximum episode length is 500 steps for Unscrew Cap and Turn Faucet, and 600 steps for Rotate Object. The same PPO settings are used for all methods. The pretrained encoders and energy models remain frozen during downstream policy training. Hyperparameter Value Parallel environments 200 Rollout length 32 Number of mini-batches 4 Optimization epochs 10 Hidden dimensions [1024,1024,512][1024,1024,512] Activation ELU Learning rate 3×10−43× 10^-4 PPO clip range 0.20.2 Maximum gradient norm 1.01.0 Discount factor γ 0.960.96 GAE parameter λGAE _GAE 0.950.95 Initial action-noise std. 0.80.8 Desired KL divergence 0.0160.016 Entropy coefficient 0 Table 10: PPO training hyperparameters. Domain Randomization. We apply domain randomization (Tobin et al. 2017) during simulation training to reduce the gap between simulated and real observations. The randomization covers proprioception, appearance, camera configuration, and tactile sensing. Zero-mean Gaussian noise is added to the joint angles and joint velocities at each observation step. Object color and light direction are randomized when the environment is reset. We also perturb the camera position and look-at point at each reset. Image brightness, contrast, and pixel values are randomized independently for every frame. Finally, Gaussian noise is added to the tactile force readings before contact binarization. The complete settings are listed in Table 11. Parameter Randomization strategy Joint angle +(0,0.05)+N(0,0.05), per observation Joint velocity +(0,0.05)+N(0,0.05), per observation Object color Randomized at environment reset Light direction Randomized at environment reset Camera eye position [−4cm,4cm]U[-4\,cm,4\,cm], at reset Camera look-at position [−4cm,4cm]U[-4\,cm,4\,cm], at reset Image brightness [−20,20]U[-20,20], per frame Image contrast [−8,8]U[-8,8], per frame Image pixel noise [−5,5]U[-5,5], per frame Tactile force +(0,0.1)+N(0,0.1), before binarization Table 11: Domain randomization settings. N and U denote Gaussian and uniform distributions, respectively. At deployment, the real visual, tactile, and proprioceptive observations are processed using the same input format as in simulation. The tactile signals are binarized before being passed to the policy. No real-world policy fine-tuning is performed. 4 Experiment Details DexMani uses human demonstrations to pretrain the visual–tactile encoder and the prior over contact-conditioned rotational manipulability evolution. 4.1 Task Details We evaluate DexMani on three contact-rich object rotation tasks: Unscrew Cap, Rotate Object, and Turn Faucet. These tasks require different contact patterns and finger coordination strategies, while sharing the goal of sustained object rotation. Figure 13 shows the three LEAP Hand environments, and the setup used for cross-hand evaluation on the Shadow Hand, Allegro Hand, and XHand in IsaacGym (Makoviychuk et al. 2021). Figure 14 shows the training and evaluation objects used in the simulation tasks. Objects inside the black dashed boxes are excluded from policy training and used only for unseen-object evaluation. The simulation environments and reward functions are adapted from VTDexManip (Liu et al. 2025). We adjust the task settings to match our evaluation protocol. The setup, success criterion, object split, and reward function of each task are described below. Figure 13: Simulated task settings. Top: Unscrew Cap, Rotate Object, and Turn Faucet on the LEAP Hand. Bottom: cap unscrewing with the Shadow, XHand, and Allegro Hand. Unscrew Cap. Figure 14: Objects used in the three simulation tasks. Objects inside the black dashed boxes are held out for unseen-object evaluation. This task evaluates coordinated finger motion during bottle-cap rotation. Each bottle is fixed to the table. The cap has one rotational degree of freedom about its local z-axis. The goal is to rotate the cap counterclockwise by one full revolution within an episode. The reward is r=λ1rp+λ2rv+λ3rd+λ4rs.r= _1r_p+ _2r_v+ _3r_d+ _4r_s. (29) The rotation term is rp=min(θjoint,7.0)r_p= ( _joint,7.0), where θjoint _joint is the cap rotation angle. The velocity term is rv=clamp(vjoint,−10,10)r_v=clamp(v_joint,-10,10). The distance term is rd=exp(−10d)r_d= (-10d), where d is the summed distance from the fingertips to a reference point located 2cm2\,cm below the cap. The success term is rs=5r_s=5. The reward weights are listed in Table 12. Reward term Weight Rotation position λ1=0.5 _1=0.5 Angular velocity λ2=1 _2=1 Fingertip distance λ3=0.5 _3=0.5 Success λ4=1 _4=1 Table 12: Reward weights for Unscrew Cap. The velocity reward is disabled when the cap has positive angular velocity but no tactile contact is active. This encourages the hand to rotate the cap through fingertip contact. Rotate Object. In this task, the dexterous hand rotates an object on a table without toppling it. The task requires coordination between the thumb and the other fingers. The goal is to rotate the object by 2π2π while keeping it upright. We use thirty YCB objects (Calli et al. 2015) for policy training and ten held-out objects for evaluation. The reward is r=λ1⊤rd+λ2rrot+λ3ra+λ4rv+rb.r=λ_1 r_d+ _2r_rot+ _3r_a+ _4r_v+r_b. (30) The distance term is rd=[exp(−10d1),d2]⊤r_d=[ (-10d_1),\,d_2] . Here, d1d_1 is the vertical distance between the fingertips and the object. The term d2d_2 is the planar distance between the object and its target position. The term rrotr_rot measures the difference between the current and target object poses. The action penalty is ra=∥a∥2r_a= a _2. The velocity term is rv=clamp(ωz,−10,10)r_v=clamp( _z,-10,10), where ωz _z is the object angular velocity about the z-axis. A success bonus of rb=250r_b=250 is added when the task is completed. The reward weights are listed in Table 13. Reward term Weight Distance 1=[0.25,−10]⊤ λ_1=[0.25,-10] Pose difference λ2=1 _2=1 Action penalty λ3=−2×10−4 _3=-2× 10^-4 Angular velocity λ4=1 _4=1 Success bonus 11 Table 13: Reward weights for Rotate Object. Turn Faucet. This task requires the dexterous hand to rotate a faucet handle. Unlike Unscrew Cap, the fingers must push against the handle rather than twist a cylindrical cap. The goal is to rotate the handle clockwise by one full revolution. We use five faucet models from the SAPIEN dataset (Xiang et al. 2020) for policy training. All models have a rotation axis perpendicular to the ground. For evaluation, we use three unseen instances. The reward for faucet turning follows the same formulation as Unscrewing cap. The main difference is that the velocity reward is always applied and is not gated by tactile contact. In addition, we encourage the fingertips to approach task-specific target positions around the faucet handle. The distance term is computed as the sum of the distances from each fingertip to its corresponding target position. 4.2 Additional Analysis Learning efficiency. Figures 15 and 16 compare learning progress in simulation. On the LEAP Hand, DexMani reaches strong cap-unscrewing performance earlier and converges to a higher final success rate than the baselines. The same trend holds across the Shadow, Allegro, and XHand embodiments, despite their different kinematics and action spaces. These results show that the human-derived guidance improves both learning efficiency and final policy performance, and that this benefit remains consistent across hand morphologies. Figure 17 further shows six representative simulated rollouts. Across different settings, DexMani maintains object progress while releasing and re-establishing finger contacts, producing sustained rotation without prescribing a fixed contact sequence. Together with the learning curves, these examples illustrate that the human-derived guidance improves both policy learning and execution. Figure 15: Learning curves of unscrew cap Task. Figure 16: Learning curves of cross-hand unscrew cap Task. Figure 17: Representative simulated rollouts. Six sequences illustrate sustained rotation through repeated contact transitions across the evaluated settings. Real-world experiment. The Faucet task presents the largest distribution shift in our real-world evaluation. Neither faucet-turning demonstrations nor the physical faucet object appear in the human pretraining data, and the physical faucet also differs from the simulated objects used for robot-policy training. The policy must therefore handle a new task-specific contact pattern, unseen object geometry, and the sim-to-real sensing and dynamics gap simultaneously. We attribute the low success rate mainly to this combined task and object distribution shift.