Paper deep dive
GOTT: Object-centric Dexterous Manipulation with a Reusable Cross-Embodiment Primitive
Yulin Liu, Lai Wei, Yen-Jen Wang, Akash Sharma, Pieter Abbeel, Henrik I. Christensen, Haozhi Qi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 10/6/2026, 10:24:37 AM
Summary
The paper introduces GOTT (General Object Trajectory Tracking), a framework for dexterous manipulation that uses a reusable, cross-embodiment contact-acquisition primitive. GOTT decomposes manipulation into reach, acquire, and move stages, allowing high-level intent (object trajectories and reach specifications) to be translated into robust low-level robot behavior across diverse objects and hand morphologies.
Entities (8)
Relation Signals (6)
GOTT â decomposes â Reach-Acquire-Move
confidence 95% · GOTT implements this interface through a reach, acquire, and move decomposition.
GOTT â uses â Contact-Acquisition Primitive
confidence 95% · GOTT is a reach-acquire-move framework built around a single cross-embodiment contact-acquisition primitive.
GOTT â supports â Cross-embodiment
confidence 90% · GOTT is able to establish robust contact across diverse objects, arm-hand platforms, and seen and unseen hand morphologies.
Franka Research 3 â usedin â GOTT
confidence 90% · We evaluate GOTT on multiple arm-hand platforms, including Franka Research 3
VLM â usedby â GOTT
confidence 85% · We therefore use a VLM to identify a task-relevant contact region.
SAM2 â usedby â GOTT
confidence 85% · During real-world deployment, we use SAM2 to segment the target object
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Foundation models and large-scale human data provide rich sources of manipulation intent, but translating this intent into multi-fingered robot behavior remains difficult. Dexterous hands still lack a reusable low-level primitive that reliably establishes contact across tasks and embodiments. We propose GOTT, a reach-acquire-move framework built around a single cross-embodiment contact-acquisition primitive. Given a robot-agnostic object trajectory and a reach specification, GOTT first brings the hand near a task-relevant contact region. The shared closed-loop primitive then establishes stable contact from this approximate initialization, and a pose-conditioned controller tracks the desired object motion. Reach specifications may come from future-aware planning, external models, or human demonstrations, while the primitive and tracking backend remain unchanged. Simulation and real-world experiments show that GOTT is able to establish robust contact across diverse objects, arm-hand platforms, and seen and unseen hand morphologies. It also consistently improves end-to-end task success over open-loop grasp execution.
Tags
Links
- Source: https://arxiv.org/abs/2610.03861v1
- Canonical: https://arxiv.org/abs/2610.03861v1
Trouble viewing inline? Open PDF directly â
Full Text
91,293 characters extracted from source content.
Expand or collapse full text
GOTT: Object-centric Dexterous Manipulation with a Reusable Cross-Embodiment Primitive Yulin Liu Affiliation: Amazon FAR Affiliation: UC San Diego Lai Wei Affiliation: Amazon FAR Affiliation: University of Chicago Yen-Jen Wang Affiliation: Amazon FAR Affiliation: UC Berkeley Akash Sharma Affiliation: Amazon FAR Pieter Abbeel Affiliation: Amazon FAR Affiliation: UC Berkeley Henrik I. Christensen Affiliation: UC San Diego Haozhi Qi Affiliation: Amazon FAR Affiliation: University of Chicago Abstract Foundation models and large-scale human data provide rich sources of manipulation intent, but translating this intent into multi-fingered robot behavior remains difficult. Dexterous hands still lack a reusable low-level primitive that reliably establishes contact across tasks and embodiments. We propose GOTT, a reach-acquire-move framework built around a single cross-embodiment contact-acquisition primitive. Given a robot-agnostic object trajectory and a reach specification, GOTT first brings the hand near a task-relevant contact region. The shared closed-loop primitive then establishes stable contact from this approximate initialization, and a pose-conditioned controller tracks the desired object motion. Reach specifications may come from future-aware planning, external models, or human demonstrations, while the primitive and tracking backend remain unchanged. Simulation and real-world experiments show that GOTT is able to establish robust contact across diverse objects, arm-hand platforms, and seen and unseen hand morphologies. It also consistently improves end-to-end task success over open-loop grasp execution. 1 Introduction Foundation models [16, 40] and large-scale human data [60, 3] provide rich sources of manipulation intent. For parallel-jaw grippers, such intent can be translated into robot actions through mature low-level primitives such as grasping, placing, and trajectory following [24, 23, 31]. However, the same approach does not directly extend to dexterous hands, which lack a comparable set of general-purpose low-level skills. These skills must establish complex contact configurations, coordinate high-dimensional actions, and adapt to different hand morphologies. Existing dexterous manipulation methods usually address these challenges through task-specific sim-to-real learning [39] or hand-motion retargeting [30]. However, these approaches are not well suited to converting high-level intent into robot behavior. Task-specific policies [44, 45, 9, 19] require retraining for each new behavior, making them difficult to scale. Retargeting recovers kinematic targets from demonstrations [37], often without incorporating perceptual feedback [41]. This lack of feedback makes contact-rich tasks difficult to execute in the real world. This raises a key question: what is the right interface for translating high-level manipulation intent into low-level dexterous execution? Object-centric motion representations provide a promising answer to this question. Prior work has shown that many rigid-object manipulation tasks can be described by the desired object motion [27, 43, 20], abstracting away embodiment-specific robot motion. However, an object trajectory specifies how the object should move, but not where and how the hand should establish contact. This distinction is critical for functional manipulation: a geometrically stable grasp may still be unsuitable for pouring, sweeping, or tool use. Prior methods address this gap by supplying a pre-grasp or hand reference [10, 32], or by learning contact acquisition jointly with trajectory execution [26, 28]. While effective, these designs do not expose contact acquisition as a reusable skill across high-level sources and robot embodiments. We propose General Object Trajectory Tracking (GOTT), a framework for executing diverse object motions across tasks and robot embodiments using a shared contact-acquisition primitive. We represent high-level manipulation intent using a desired object trajectory and a reach specification, which specifies how to bring the hand near a task-relevant contact region. These components may come from different sources. For example, the object trajectory may be extracted from a video, while the reach specification may be generated through affordance-aware planning or obtained by retargeting a human trajectory. GOTT implements this interface through a reach, acquire, and move decomposition. During reach, the robot follows the reach specification toward the selected contact region. During acquire, a learned closed-loop contact-acquisition primitive establishes stable contact from this approximate initialization. During move, an object-pose-conditioned controller tracks the desired object trajectory. The primitive uses only local hand-object geometry and proprioceptive history, while a canonical per-link representation allows the same policy to be shared across hands. Reach specifications can come from VLM-guided, future-aware planning, external models, or demonstrations, while the contact-acquisition primitive remains unchanged. We evaluate GOTT in simulation and the real world. First, we test whether the contact-acquisition primitive can robustly grasp diverse objects across different hands and transfer zero-shot to unseen hand morphologies. We then evaluate the full pipeline on object-trajectory tracking tasks, including trash cleanup, pouring, whiteboard erasing, and tool use. We compare against end-to-end trajectory learning, showing that our decomposition enables robustness to workspace shifts and reuse across robot arms. We also show that the same primitive supports high-level specifications from generated videos, keyframes, human demonstrations, and hand-object interaction datasets. Finally, we conduct ablations on the primitiveâs observation and reward design and the systemâs planning modules. 2 Related Work High-level task interfaces. Foundation models and human demonstrations can specify robot manipulation tasks through language, visual constraints [23], code [31], or generated videos [43]. ManipGen [12] combines reusable local policies with foundation models and motion planning. Another line of work learns from human videos by extracting intermediate motion representations, such as point trajectories, object flow, or object-centric motion plans [4, 56, 58, 65, 20, 29]. SPOT [21] learns object-pose trajectory plans from human demonstrations and uses them to condition a robot policy. VLM-Scaffold [13] adapts a similar pipeline to dexterous hands but relies on task-specific residual RL, making it challenging to scale across tasks and embodiments. LUCID [18] learns embodiment-agnostic intent from unstructured human videos and connects it to embodiment-specific sensorimotor policies through object flow and a rough palm pose. In contrast, GOTT represents high-level intent using an object trajectory and a flexible reach specification, and connects these inputs to robot execution through a reusable dexterous contact-acquisition primitive. Trajectory-based dexterous manipulation. D-Grasp [11] combines reference-conditioned grasp control with global motion synthesis to move objects to target poses. Methods that start from a reference trajectory typically follow one of two strategies. The first performs trajectory-specific transfer, converting a human or object trajectory into robot motion through kinematic optimization, sampling [41], or reinforcement learning [30, 37]. Human2Sim2Robot [33] uses a single object trajectory and a pre-grasp to learn a task-specific robot policy. ManipTrans [30] and REGRIND [15] transfer hand-object interaction trajectories through residual learning, while SPIDER [41] performs fast physics-informed retargeting. These transfer methods can reproduce a given demonstration but are often tied to a specific trajectory or embodiment. The second strategy trains policies conditioned on multiple reference trajectories. ObjDex [10] and DexTrack [32] condition policies on object or hand trajectories. AdaDexTrack [1] adds online modulation to improve tracking of noisy references. OmniGrasp [36] learns to grasp diverse objects and follow object trajectories with a simulated humanoid. SimToolReal [26] and Dex4D [28] learn general object-centric policies conditioned on target object poses or point tracks, jointly handling contact acquisition and motion execution. Grasp to Act [17] combines human-guided grasp optimization with residual RL for online finger adaptation. In contrast, GOTT factors execution into reach, acquire, and move: high-level inputs specify the desired object motion and a task-relevant reach, while a trajectory-agnostic primitive establishes contact before trajectory tracking begins. This decomposition enables the same contact-acquisition primitive to be reused across trajectories, objects, and robot embodiments. Dexterous grasping. Existing learning-based methods develop generalizable tabletop grasping policies [59, 50, 35, 64]. Grasp synthesis and evaluation methods instead generate or refine grasp targets [7, 53, 51, 8, 34, 14]. GraspXL [62] and ConGraspXL [63] condition grasping motions on approach, contact-region, or pose constraints. Functional grasping methods use affordances, human grasp references, or language to select task-relevant grasps [2, 22, 52]. Cross-embodiment methods further share grasping policies or representations across hands: CrossDex [61] uses a unified observation space and retargeted eigengrasp actions, while OHRA [54] uses canonical hand representations and UniMorphGrasp [57] conditions grasp generation on morphology. GET-Zero [42] studies zero-shot transfer to unseen hand structures for in-hand rotation. These works advance grasping and cross-hand transfer, while primarily focusing on grasp acquisition itself. In contrast, GOTT treats contact acquisition as a reusable intermediate primitive that refines an approximate, task-relevant reach into stable contact before downstream object-trajectory tracking. This enables the same primitive to be reused across objects, tasks, trajectories, and hand embodiments. 3 Dexterous Manipulation via General Object Trajectory Tracking We propose GOTT, a reach-acquire-move framework (Section 3.1) for executing rigid-object motions with dexterous hands. Its high-level interface consists of a desired object trajectory Ïo _o and a reach specification Ï that brings the hand near a task-relevant contact region. At its core, a reusable cross-embodiment contact-acquisition primitive grounds this interface by turning an approximate reach into stable contact (Section 3.2). In the object-centric setting, a future-aware planner constructs Ï from the object mesh, desired trajectory, and task description (Section 3.3). Alternatively, external models or human demonstrations can provide Ï (Section 3.4), while the primitive and tracking backend remain unchanged (Section 3.5). 3.1 Reach-Acquire-Move Decomposition Given a desired object trajectory Ïo=TO,tt=0T _o=\T_O,t\_t=0^T where TO,tT_O,t denotes the object pose at timestep t, and a reach specification Ï=(Οapp,ginit)Ï=( _app,g_init), GOTT decomposes dexterous execution into three stages. During reach, the robot follows the approach trajectory Οapp _app to an approximate initial configuration ginitg_init. During acquire, the contact-acquisition primitive uses closed-loop feedback to establish stable contact. During move, the pose-conditioned tracker moves the object along Ïo _o. This decomposition separates high-level reasoning about functional contact selection, collision-free approach, and long-horizon reachability from local contact-rich control. Consequently, the high-level source needs to provide only a feasible approach to an approximate initial configuration, rather than an exactly executable final grasp. 3.2 General-Purpose Contact Acquisition Primitive Establishing stable contact from an initial hand configuration near the object remains challenging. State-of-the-art grasp-generation methods either execute grasps open loop [7, 53], do not account for the functional suitability of the selected initial grasp [35, 64], or do not generalize across embodiments. We therefore train a local, closed-loop contact-acquisition policy using reinforcement learning in simulation and transfer it to the real world. The policy uses local hand-object geometry and proprioceptive history, while a canonical representation enables sharing across hands. This design allows the same primitive to be reused across tasks, high-level sources, objects, and hand embodiments. We first describe the single-hand control policy and then its extension to cross-embodiment control. Environment setup. We select 16 YCB objects [5] as our training set and collect several stable resting poses for each object. For each objectâpose pair, we use BODex to generate 4,000 initial hand configurations of the type described in Section 3.3 and shown in Figure 1A. Each episode starts from one of these initial hand configurations. The policy must establish stable contact and lift the object. Observations and actions. The observation contains three components: (1) the link-distance representation from [64], in which each hand link is represented by its displacement to the nearest point in the object point cloud; (2) hand-link positions; and (3) joint positions and previous joint targets. Components (2) and (3) are stacked over the five most recent timesteps. The policy outputs a 6D wrist-delta command and a vector of hand-joint deltas: at=ÏΞâ(ot)a_t= _Ξ(o_t). Rewards. Our reward function is defined as: r=wdârdist+wcârcontact+wlârlift+woârobj.r=w_dr_dist+w_cr_contact+w_lr_lift+w_or_obj. The four terms encourage link-object proximity, opposing contact between the thumb and another finger, lifting after contact, and object stability, respectively. Additional definitions and weights are provided in the appendix. Figure 1: Contact-acquisition training and cross-embodiment policy. (A) The RL policy starts from randomized initial configurations and learns to establish stable contact using local hand-object observations. (B) Padding and binary masks map different hands into a shared observation and action space, allowing one policy to operate across hand embodiments. Domain randomization. During training, we randomize object scale, friction, and mass; robot-link friction; and actuator stiffness and damping. We also apply random force and torque perturbations once the object is held to improve grasp robustness. Contact prediction. During deployment, we use a proprioception-based contact predictor to determine when to switch from acquisition to tracking. Because proprioceptive and action histories can reveal contact events [44], we train a separate network after the policy converges: c^t=fÏ(htâk+1:tprop)â[0,1]5 c_t=f_Ï(h^prop_t-k+1:t)â[0,1]^5, where hproptâk+1:th^prop_t-k+1:t contains recent joint positions, joint targets, and hand-link positions, and c^t c_t contains the per-finger contact probabilities. We train the network in simulation using ground-truth per-finger contact labels. Acquisition terminates when the predicted contact probabilities for the thumb and at least one other finger exceed Ïc _c for k consecutive timesteps. Unified cross-embodiment representation. To train one controller across multiple multi-fingered hands, we map hand-specific observations and actions into a canonical representation, as shown in Figure 1B. For link-level observations, we map each hand onto a 21-keypoint topology consisting of one palm keypoint and four keypoints per finger. Missing keypoints are filled using the nearest available links, and a binary link mask distinguishes valid entries from padded ones. Joint positions and hand actions are similarly mapped to a fixed 26-slot canonical joint order. Missing joint slots are zero-padded, and a binary joint mask identifies the active slots. The policy outputs a 6D end-effector delta and a canonical joint-delta vector. Before execution, padded joints are masked out, and the remaining entries are remapped to the robotâs native joint indices. This representation allows one policy to operate across hands with different joint counts and kinematic structures without hand-specific encoder or decoder networks. 3.3 Reach Specifications from Future-Aware Planning The reach stage can be instantiated from different high-level inputs. We first consider the default object-centric setting, in which the inputs are an object mesh âłM, a desired object trajectory Ïo _o, and a task description. The planner must determine where to contact the object and select an initial hand configuration that allows a feasible approach and downstream arm motion. Existing geometry-based grasp synthesis methods [7, 53] can generate locally stable candidates but do not consider task affordances or the full object trajectory. A candidate may therefore be functionally unsuitable, or it may be initially reachable but become infeasible later because of joint limits or collisions (Figure 2). We address these issues through VLM-based affordance reasoning and future-aware IK selection to select a task-relevant initial configuration for which the induced arm motion remains feasible along Ïo _o. VLM-based affordance reasoning. Geometric stability alone does not determine whether a grasp is suitable for the task. For example, grasping a drill by its chuck may be stable but does not support functional tool use. We therefore use a VLM to identify a task-relevant contact region. We render the object from four viewpoints and prompt a VLM to identify the task-relevant grasping part and predict keypoints spanning its extent. The selected 2D keypoints are back-projected onto the mesh and aggregated across views by voting, yielding a task-relevant contact region ââââłRâ . Figure 2: Future-aware planning. We roll out the hand motion induced by each candidate initial configuration over the full object trajectory and solve IK along this motion. Red segments indicate joint jumps from discontinuous IK solutions. We reject candidates that are initially reachable but become infeasible later and select the remaining candidate with the least cumulative joint motion. Region-conditioned candidate generation. We use BODexâs lightweight initialization stage [7] to sample candidate hand poses and joint configurations from object geometry within âR, yielding =gii=1NG=\g_i\_i=1^N (N=10N=10). Each gig_i is an approximate initialization rather than a final grasp. The learned contact-acquisition primitive then establishes contact more robustly, as shown in our experiments. Trajectory-aware selection. For each gig_i, we first plan a collision-free approach trajectory Οapp _app from the robotâs current configuration to the candidate initialization. We then roll out the nominal hand motion induced by keeping the hand-object transform fixed while the object follows Ïo _o. We then solve inverse kinematics (IK) along this motion and discard candidates for which the approach is infeasible, the IK solutions violate joint limits, or the solver fails to converge. Among the remaining candidates, we select gâg with the least cumulative joint motion: gâ=argâĄminâĄât=1TgiâvalidâĄâqi,tarmâqi,tâ1armâ2,g = _g_i _valid _t=1^T \|q^arm_i,t-q^arm_i,t-1 \|_2, where validG_valid is the set of remaining candidates and qi,tarmq^arm_i,t is the arm IK solution for candidate gig_i at timestep t. Implementation details are provided in the Appendix B. 3.4 Reach Specifications from Human Guidance and External Models Section 3.3 constructs the reach specification Ï from object-side information. Alternatively, human demonstrations or external models can supply contact regions, initial hand configurations, or approach trajectories directly, replacing the corresponding planning steps (Figure 3). The resulting reach specification is executed by the same contact-acquisition primitive and trajectory tracker. Figure 3: Alternative sources of reach specifications. Given a desired object trajectory, GOTT can construct a reach specification through future-aware planning or use additional guidance from reference grasps or human approach trajectories. The shared contact-acquisition primitive refines the resulting approximate initializations into stable contact. Contact regions or initial configurations. External models or human demonstrations can provide a task-relevant contact region or an initial hand configuration. Such a contact region replaces VLM-based affordance reasoning, while an initial configuration also bypasses candidate generation. The remaining planning steps construct a feasible approach and check downstream arm feasibility to form Ï. Retargeted human approaches. Human-object datasets such as DexYCB [6] and OakInk [60] provide synchronized hand and object motions. We retarget the human approach segment to the robot hand to obtain Οapp _app and its terminal configuration ginitg_init, which together define Ï. This supplied approach replaces approach generation. Its endpoint need not be an exact final grasp: the contact-acquisition primitive establishes stable contact, after which the trajectory tracker follows Ïo _o. Richer guidance replaces more planning steps, but may not account for robot-specific reachability or collisions. We therefore retain future-aware planning as the default for arm-hand execution. Qualitative examples of these alternative compositions are available on the project website. 3.5 Execution from High-Level Specifications Regardless of the high-level source, the execution backend receives a desired object trajectory Ïo _o and a reach specification Ï. During real-world deployment, we use SAM2 [46] to segment the target object and FoundationPose [55] to estimate and track its pose. GOTT executes the task in three stages: reach, acquire, and move. Reach. The robot follows the approach trajectory specified by Ï. The system switches to the contact-acquisition primitive once the minimum distance between the hand links and the object surface falls below Ï”approach _approach. Contact acquisition. At this transition, the contact-acquisition primitive takes control of the wrist and fingers. It locally adjusts the hand to compensate for object pose errors, geometry mismatch, and contact uncertainty. During this phase, we run the proprioception-based contact predictor described in Section 3.2. Once the predictor detects stable contact, the system proceeds to the move stage. Pose-conditioned end-effector tracking. After stable contact is detected, the robot performs a short lift before tracking the desired object trajectory. We approximate the hand-object transform as fixed during tracking. At each timestep, we compute an end-effector target from the desired object pose and the estimated hand-object transform, correct the target using the observed object pose error, and send it to the low-level position controller. 4 Experiments We evaluate the contact-acquisition primitiveâs robustness, cross-embodiment transfer, and contribution to end-to-end task success. We further examine its reuse across workspaces and with different high-level sources, and evaluate key design choices through ablations. Figure 4: Real-world evaluation setup. (a) in-distribution (blue) and out-of-distribution (purple) test objects; (b) three robot embodiments used for evaluation. Hardware setup. We evaluate GOTT on multiple arm-hand platforms, including Franka Research 3 (7-DoF) and Tianji (7-DoF) arms paired with XHand (12-DoF) and Sharpa Wave (22-DoF) hands, as shown in Figure 4. Object assets and perception stack. We obtain object meshes using AR Code11 1 https://ar-code.com/ on an iPhone and use them for planning and pose tracking. During execution, SAM2 [46] segments the target object in the initial frame, and FoundationPose [55] estimates and tracks its 6D pose using the mesh and segmentation mask. FoundationPose runs at 45 Hz, while the policy runs at 10 Hz and uses the latest pose estimate at each control step. Table 1: Simulation lifting success on three left hands (%). BODex and DRO are executed as open-loop grasp baselines. CrossDex and GOTT are evaluated as specialist policies trained separately for each hand and unified policies trained jointly across all three hands. Learned-policy results report mean ± standard deviation across three seeds. Best results per column are bolded. Method Setting XHand Sharpa Allegro Avg. BODex [7] - 45.4 27.2 43.6 38.7 DRO [53] - 31.3 47.0 43.2 40.5 CrossDex [61] Specialist 30.4 ± 10.9 74.9 ± 1.5 34.5 ± 3.3 46.6 ± 4.1 Unified 44.7 ± 2.9 52.8 ± 3.7 57.6 ± 1.0 51.7 ± 1.5 Ours Specialist 64.3 ± 0.7 81.4 ± 1.2 80.4 ± 1.8 75.4 ± 1.0 Unified 74.1 ± 1.2 75.0 ± 1.1 87.1 ± 4.4 78.7 ± 1.1 Simulation setup. We train a unified policy for XHand, Sharpa, and Allegro in IsaacLab [38] using PPO [48]. We train for 2,000 epochs using 4,096 parallel environments. The training set includes 16 YCB objects [5], with 4,000 initial configurations generated using BODex for each objectâresting-pose pair. 4.1 Is It a Reliable General-Purpose Contact Primitive? We isolate the low-level contact-acquisition primitive by evaluating object lifting in simulation and the real world, without full-arm planning. 4.1.1 Simulation grasping evaluation We use a floating-hand tabletop setup and evaluate on DexGraspNet objects [51], following the preprocessing protocol of BODex [7]. After filtering out flat objects and rescaling the remaining objects to four sizes, the evaluation set contains 9,588 assets and 54,149 object-scale-pose instances. We run simulations in IsaacLab with the friction coefficient set to 1.5 and the object mass set to 100 g. All physics parameters are fixed across evaluations. Baselines. We compare against two representative grasp synthesis methods. BODex [7] optimizes grasp poses for each robot-object pair using bilevel optimization. DRO [53] is a cross-embodiment grasp prediction network, for which we use the authorsâ released checkpoints. Since both methods output grasp targets rather than closed-loop acquisition policies, we execute their outputs open-loop using their original execution controllers. Evaluation protocol. For each object-scale-pose instance, we generate 20 initial poses using BODexâs first stage (Section 3.3) and share them across methods. BODex refines them via bilevel optimization, DRO uses them to specify initial approach directions, and our policy starts directly from them for closed-loop contact acquisition. In the same tabletop setup, BODex and DRO execute open-loop from predicted pre-grasps, effectively assuming successful motion planning to these poses. Our method additionally handles the approach from the raw initial poses. Metrics. We report lifting success rates across hand embodiments, objects, and initial object poses. A trial is successful if the hand establishes contact, lifts the object by at least 5 cm, and keeps it stable for the full evaluation horizon. Results. The results are shown in Table 1. GOTT consistently outperforms BODex and DRO in lifting success across all three hands. Notably, although the policy is trained on only 16 YCB objects, it generalizes to the full evaluation set of 9,588 DexGraspNet assets. The success rates for BODex and DRO are lower than those reported in their original papers due to our stricter tabletop protocol. Prior work initializes objects without gravity and applies gravity only after contact is established, whereas we place objects on a table with gravity enabled from the start and execute directly. Appendix E details the baseline implementations and evaluation protocols and presents controlled comparisons. 4.1.2 Real-world grasping experiments We evaluate the contact-acquisition primitive using the same checkpoint on three arm-hand platforms: Franka-Sharpa, Franka-XHand, and Tianji-XHand. This tests both cross-hand generalization and transfer across arm platforms through the end-effector control interface. The evaluation setup is shown in Figure 4. We run 5 trials on each of 10 objects, for 50 rollouts per platform. A trial is successful if the hand establishes contact, lifts the object by at least 5 cm, and keeps it stable for at least 3 seconds. GOTT achieves success rates of 90%, 90%, and 96% on Franka-Sharpa, Franka-XHand, and Tianji-XHand, respectively. The slightly lower success rates on Franka are mainly due to the more constrained reachable workspace in our Franka setup. 4.2 Does It Generalize across and beyond Training Hands? 4.2.1 Cross-hand evaluation and representation comparison The lower portion of Table 1 compares GOTT with CrossDex [61], an RL baseline that uses a shared action space through human-hand retargeting. To isolate the effect of the observationâaction representation, we reimplement CrossDex within our framework. We adopt the observation and action design from its released code while keeping the reward, PPO hyperparameters, and training and evaluation protocol the same as those used for GOTT. Specialist policies are trained separately for each hand, while unified policies are trained jointly on XHand, Sharpa, and Allegro. Our unified policy with the per-link canonical representation achieves 78.7% average success versus 51.7% for CrossDex. Joint training also improves average success over our specialist policies, supporting positive transfer across morphologies. 4.2.2 Zero-shot transfer to unseen Allegro variants Table 2: Zero-shot generalization to unseen Allegro variants. Lifting success rate (%), reported as mean ± standard deviation over three seeds. xyzw denotes per-finger link counts (4444 = original Allegro). Oracle trains one policy per variant; Allegro-only trains on the original Allegro; Three-hand is our unified policy over XHand+Sharpa+original Allegro, evaluated zero-shot. Variant 4433 4333 4324 Avg. Oracle (per-variant) 69.0 ± 6.8 71.6 ± 1.1 76.0 ± 4.0 72.2 ± 1.2 Allegro-only (zero-shot) 29.4 ± 19.7 12.9 ± 8.6 9.4 ± 8.1 17.2 ± 12.1 Three-hand (zero-shot) 74.4 ± 6.5 72.2 ± 4.8 69.6 ± 9.6 72.1 ± 6.9 To further probe cross-embodiment generalization, we test unseen Allegro variants generated by removing finger links. We evaluate our three-hand unified policy, trained on XHand, Sharpa, and the original Allegro hand, zero-shot on these variants. We compare it with (i) per-variant oracle policies trained directly on each variant and (i) a policy trained only on the original Allegro hand. As shown in Table 2, our three-hand policy averages 72.1% success across the three variants, matching the average of the per-variant oracle policies (72.2%), whereas the Allegro-only policy achieves 17.2%. Transfer therefore comes from multi-morphology training, not the canonical representation alone. Figure 5 further shows that the policy produces coordinated grasp configurations on these unseen variants. Figure 5: Qualitative grasps on training and unseen embodiments. We compare the full GOTT (top) with the variant without rdistr_dist (bottom) on three training hands and three unseen Allegro variants. 4.3 Does the Primitive Improve End-to-End Task Success? Figure 6: Qualitative illustration of four real-world tasks. We evaluate whether the contact-acquisition primitive improves end-to-end execution on four real-world manipulation tasks using the Franka-Sharpa platform. For each task, the robot must approach the object, acquire stable contact, and move the object along the specified trajectory. We compare GOTT with two matched open-loop variants. Each replaces our region-conditioned candidate generation with the full grasp generation procedure of BODex [7] or DRO [53] and executes the predicted grasp open-loop. All other components, including perception, VLM-based affordance reasoning, future-aware IK selection, and trajectory tracking, are kept fixed, isolating the contribution of closed-loop contact acquisition. Real-world tasks. We evaluate Trash Cleanup (grasp a brush and sweep trash), Mustard Pouring (grasp a mustard bottle and rotate it to a pouring orientation), Whiteboard Erasing (grasp an eraser and sweep across the board), and Drill Use (grasp a power drill and rotate it to a working orientation), as illustrated in Figure 6. These tasks require both translation and rotation, and some require functional contacts, such as grasping the drill near its handle. Each task is repeated 10 times across varied initial poses. Table 3: Real-world task results. Per-task grasp success rate (gSR) and task success rate (tSR) across four manipulation tasks on Franka-Sharpa platform, 10 trials each. Best per column in bold. Trash Cleanup Mustard Pouring Whiteboard Erasing Drill Use Avg. (%) Method gSR tSR gSR tSR gSR tSR gSR tSR gSR tSR DRO 6/10 0/10 5/10 5/10 5/10 3/10 7/10 6/10 57.5 35.0 BODex 5/10 2/10 4/10 2/10 7/10 3/10 7/10 7/10 57.5 35.0 Ours 8/10 7/10 8/10 7/10 10/10 9/10 9/10 9/10 87.5 80.0 Metrics. We report grasp success rate and task success rate. Grasp success measures whether the robot acquires stable contact, while task success measures whether it completes the downstream trajectory-tracking task. Results. Table 3 shows that GOTT achieves the highest grasp and task success rates on every task, with an average task success rate of 80.0% versus 35.0% for both baselines. Baseline failures include unsuitable contact regions, object displacement during approach, and cage-like grasps lacking firm opposing contact between the fingers and thumb. The last failure mode explains DROâs 0/10 task success on Trash Cleanup: loose contact around the brush handle causes the brush to slip during sweeping. While the open-loop baselines are sensitive to planning errors, our closed-loop controller adjusts the wrist and fingers to recover from imperfect initializations. 4.4 Does Decomposition Improve Robustness to Workspace Variations? SimToolReal [26] and Dex4D [28] jointly learn contact acquisition and object tracking with embodiment-specific policies. In contrast, GOTT uses a reusable contact primitive, arm-specific IK, and object-pose-conditioned tracking to separate contact acquisition from robot-specific reachability. We evaluate GOTT on the eraser object from SimToolRealâs DexToolBench, using a 55\,cm lifting trajectory, over 600 simulated episodes. We vary the objectâs planar position and the tableâs height offset z relative to the robot base (Figure 7). Figure 7: Success rate versus table height offset z on the SimToolReal eraser object. Near SimToolRealâs training configuration, its success rate is 0.983, compared with 0.792 for GOTT. Our failures are mainly due to IK singularities under a constrained end-effector orientation. Below z=0.4z=0.4\,m, SimToolRealâs success rate drops to 0.002, while GOTT achieves 0.914. In this comparison, we reuse our contact primitive on SimToolRealâs iiwa-Sharpa platform and evaluate SimToolReal using its released checkpoint. Adapting SimToolReal to a different arm embodiment would instead require roughly seven days of training, whereas our primitive is arm-agnostic by design. These results support separating workspace-specific reachability from reusable contact control. 4.5 Does It Support High-Level Specifications from Different Sources? We test whether GOTT accepts high-level specifications from diverse sources. These include object trajectories obtained from different sources and unseen human demonstrations that also specify the handâs approach. Across all sources, the contact-acquisition primitive remains fixed, without task- or trajectory-specific fine-tuning. 4.5.1 Object trajectories from diverse sources Figure 8: Visualization of three sources of object trajectories. Figure 8 shows that object trajectories can be obtained from human demonstrations, generated videos, and manually specified object-pose keypoints. For human demonstrations, we track object SE(3) trajectories using FoundationPose [55]. For generated videos, we follow RigVid [43], estimating per-frame depth with RollingDepth [25] and recovering object trajectories with FoundationPose [55]. For keypoints, we interpolate between user-specified object poses. The resulting trajectories from all three sources can be used directly by the GOTT pipeline. 4.5.2 Zero-shot execution of unseen human demonstrations We evaluate all 499 left-hand DexYCB sequences [6], each containing human-hand motion and an object trajectory. We kinematically retarget the hand motion to XHand for the approach, acquire stable contact with our policy, and then track the object trajectory. We use the same policy for all sequences without fine-tuning. We compare against kinematic replay, which directly executes the retargeted trajectories. Neither kinematic replay nor GOTT performs per-demonstration test-time optimization. We additionally report SPIDER [41], which generates robot controls via per-demonstration physics-informed optimization, as a reference with an additional test-time optimization stage. We use SPIDERâs task-success criterion and released MuJoCo implementation. Our reimplementations of SPIDER in IsaacLab and ManiSkill failed to achieve stable execution, so we omit their results. Our policy is trained in IsaacLab and evaluated zero-shot in ManiSkill and MuJoCo to assess sim-to-sim transfer. On these 499 demonstrations, kinematic replay achieves a success rate of 12.6% in MuJoCo, whereas GOTT achieves 71.5% in IsaacLab, 72.9% in ManiSkill, and 69.7% in MuJoCo, demonstrating sim-to-sim transfer. SPIDER achieves 76.0% in MuJoCo but requires per-demonstration optimization and reoptimization whenever nominal object states or trajectories change. Figure 9 compares human demonstrations, kinematic replay, and executions by GOTT. Figure 9: Execution of unseen DexYCB demonstrations. Transparent hands indicate motion over time. (a) Annotated human hand-object motion. (b) Open-loop kinematic replay fails to maintain contact in physics simulation. (c) Our reactive primitive acquires stable contact and executes the demonstrated object trajectory. Table 4: Ablation results. Success rate (%) on three left hands. Variant XHand Sharpa Allegro Avg. Full GOTT 74.1 ± 1.2 75.0 ± 1.1 87.1 ± 4.4 78.7 ± 1.1 w/o link position obs. 49.4 ± 6.4 57.9 ± 10.3 75.7 ± 10.6 61.0 ± 4.4 proprio history length =1=1 56.5 ± 5.1 65.0 ± 2.1 67.5 ± 3.0 63.0 ± 3.3 OHRA-style [54] obs. 24.9 ± 22.8 31.2 ± 25.2 20.9 ± 15.6 25.7 ± 18.3 w/o rcontactr_contact 0.0 ± 0.0 0.0 ± 0.0 0.0 ± 0.0 0.0 ± 0.0 w/o rliftr_lift 0.8 ± 0.1 2.7 ± 1.3 1.0 ± 1.2 1.5 ± 0.8 w/o rdistr_dist 70.6 ± 3.0 76.9 ± 4.2 80.6 ± 7.0 76.0 ± 1.6 4.6 Ablations and Analysis We analyze two aspects of GOTT: the primitiveâs observation and reward design (Section 3.2) and the role of affordance reasoning and trajectory-aware selection in generating functional, feasible high-level specifications (Section 3.3). 4.6.1 Contact primitive design We vary one observation or reward setting at a time, holding all others fixed (Table 4). Removing link-position observations degrades success across hands. Reducing the proprioceptive history to one timestep removes temporal contact cues, lowering average success by 15.7 percentage points. We also compare with an OHRA-style [54] observation representation. In its original form, OHRA encodes only the object point cloud with a heavyweight point-cloud encoder, concatenates the resulting feature with a morphology VAE embedding, and uses the combined representation to predict grasps offline. This encoder is too large to train jointly with PPO at our training scale, so we adapt the representation to RL by (i) replacing the encoder with a lighter PointNet and (i) adding the hand point cloud as input, while preserving OHRAâs morphology VAE conditioning. Even after this adaptation, average success drops to 25.7%, suggesting that per-link contact geometry provides a better representation for a closed-loop primitive. Removing either rcontactr_contact or rliftr_lift reduces average success to below 2%, confirming their necessity. While rdistr_dist modestly improves lifting on training hands on average, it suppresses finger inversion and distorted joint bending (Figure 5). Under morphology shift, removing this regularization lowers average success on three unseen Allegro variants from 72.1% to 55.1%. 4.6.2 Affordance- and Future-Aware Grasp Selection Table 5: Affordance- and Future-aware selection on Drill Use (20 scenes). Random and IK-only use affordance-agnostic candidates and select by random sampling and trajectory feasibility, respectively. VLM variants first generate candidates from a task-relevant contact region and then apply future-aware IK selection. Selection Affordance Future-aware gSR (%) tjSR (%) func. (%) tSR (%) Random â â 100 80 35 20 IK-only â â 95 90 30 30 VLM (gpt-5.6-sol) â â 90 70 80 55 VLM (gemini-flash-latest) â â 70 50 65 50 VLM (claude-opus-4-7) â â 80 70 85 65 VLM (claude-opus-5) â â 95 75 85 60 We evaluate affordance-conditioned candidate generation and future-aware selection on 20 matched Drill Use scenes, which require grasping the handle instead of the chuck. All methods use the same scene initializations, with position randomized by ±0.10± 0.10\,m along x and y and orientation by ±30â± 30 . Table 5 reports grasp success rate (gSR), task trajectory success rate (tjSR), functional correctness (func.), and overall task success rate (tSR). Trajectory success requires the drill to rotate by 45â45 about the specified axis while remaining grasped by the robot. Overall task success requires grasp success, trajectory success, and an affordance-correct grasp to all hold in the same trial. All rates are computed over the same 20 scenes. Scenes with no valid candidate terminate without execution and count as failures for all metrics. Random selection achieves 100% gSR but only 20% tSR: an initially valid grasp may not support the task. Future-aware IK selection raises tjSR from 80% to 90%, but functional correctness and overall task success remain at 30%: successful trajectory execution alone does not ensure a task-appropriate grasp. Across VLM backbones, affordance conditioning improves func. to 65â85% and tSR to 50â65%. Restricting candidates to task-relevant regions lowers tjSR to 50â75% but improves overall task success. 5 Conclusion We presented GOTT, a reach-acquire-move framework for dexterous manipulation from object trajectories and flexible reach specifications. At its core, a cross-embodiment contact-acquisition primitive establishes stable contact from approximate reach specifications, decoupling local contact control from arm-specific reaching and downstream tracking. Simulation and real-world experiments demonstrate the primitiveâs reuse across objects, embodiments, tasks, workspaces, and high-level input sources. Limitations and future work. First, high-level planning and contact acquisition operate sequentially. Tighter feedback between them could enable online recovery from poor reach specifications. Second, trajectory tracking relies on visual object pose estimation, which can fail under severe hand-object occlusion. Incorporating tactile sensing or visuotactile state estimation is a natural next step. Finally, we treat the hand-object transform as approximately fixed after grasping. Extending GOTT to handle changes in this transform would enable more precise in-hand manipulation tasks, such as unscrewing. Acknowledgements.We thank Tingfan Wu for his help with hardware support. We are grateful to Shuo Chen for proposing to use a vision-language model to query task-relevant keypoints, as well as for insightful discussions and suggestions on VLM-based affordance reasoning. We thank Qiyang Yan for substantially accelerating the FoundationPose tracking speed, and Jiayuan Mao and Ruoshi Liu for helpful discussions and suggestions. References [1] J. Adalibieke, Q. Han, X. Liu, Y. Qin, and L. Yi (2026) AdaDexTrack: dynamic modulation for adaptive and generalizable dexterous manipulation tracking. In CVPR, Cited by: §2. [2] A. Agarwal, S. Uppal, K. Shaw, and D. Pathak (2023) Dexterous functional grasping. In CoRL, Cited by: §2. [3] P. Banerjee, S. Shkodrani, P. Moulon, S. Hampali, F. Zhang, J. Fountain, E. Miller, S. Basol, R. Newcombe, R. Wang, Jakob Julian Engel, and Tomas Hodan (2024) Introducing hot3d: an egocentric dataset for 3d hand and object tracking. arXiv:2406.09598. Cited by: §1. [4] H. Bharadhwaj, R. Mottaghi, A. Gupta, and S. Tulsiani (2024) Track2act: predicting point tracks from internet videos enables generalizable robot manipulation. In ECCV, Cited by: §2. [5] B. Calli, A. Singh, J. Bruce, A. Walsman, K. Konolige, S. Srinivasa, P. Abbeel, and A. M. Dollar (2017) Yale-cmu-berkeley dataset for robotic manipulation research. IJRR. Cited by: §3.2, §4. [6] Y. Chao, W. Yang, Y. Xiang, P. Molchanov, A. Handa, J. Tremblay, Y. S. Narang, K. Van Wyk, U. Iqbal, S. Birchfield, J. Kautz, and D. Fox (2021) Dexycb: a benchmark for capturing hand grasping of objects. In CVPR, Cited by: §3.4, §4.5.2. [7] J. Chen, Y. Ke, and H. Wang (2025) Bodex: scalable and efficient robotic dexterous grasp synthesis using bilevel optimization. In ICRA, Cited by: §A.8, Appendix B, §E.2, Appendix E, §2, §3.2, §3.3, §3.3, §4.1.1, §4.1.1, §4.3, Table 1. [8] S. Chen, J. Bohg, and C. K. Liu (2024) Springgrasp: synthesizing compliant, dexterous grasps under shape uncertainty. In RSS, Cited by: §2. [9] T. Chen, M. Tippur, S. Wu, V. Kumar, E. Adelson, and P. Agrawal (2023) Visual dexterity: in-hand reorientation of novel and complex object shapes. Science Robotics. Cited by: §1. [10] Y. Chen, C. Wang, Y. Yang, and C. K. Liu (2024) Object-centric dexterous manipulation from human motion data. In CoRL, Cited by: §1, §2. [11] S. Christen, M. Kocabas, E. Aksan, J. Hwangbo, J. Song, and O. Hilliges (2022) D-Grasp: physically plausible dynamic grasp synthesis for hand-object interactions. In CVPR, Cited by: §2. [12] M. Dalal, M. Liu, W. Talbott, C. Chen, D. Pathak, J. Zhang, and R. Salakhutdinov (2025) Local policies enable zero-shot long-horizon manipulation. In ICRA, Cited by: §2. [13] V. de Bakker, J. Hejna, T. Lum, O. Celik, A. Taranovic, D. Blessing, G. Neumann, J. Bohg, and D. Sadigh (2026) Scaffolding dexterous manipulation with vision-language models. In NeurIPS, Cited by: §2. [14] H. Fang, H. Yan, Z. Tang, H. Fang, C. Wang, and C. Lu (2025) AnyDexGrasp: general dexterous grasping for different hands with human-level learning efficiency. arXiv:2502.16420. Cited by: §2. [15] Y. Feng, N. Leung, J. Wang, L. Yang, H. Qi, and P. Culbertson (2026) A minimalist retargeting-guided reinforcement learning recipe for dexterous manipulation. In CoRL, Cited by: §2. [16] Google DeepMind (2026) Gemini 3.1 pro model card. Note: https://deepmind.google/models/model-cards/gemini-3-1-pro/ Cited by: §1. [17] H. Gupta, M. A. Mirzaee, and W. Yuan (2026) Grasp to act: dexterous grasping for tool use in dynamic settings. RA-L. Cited by: §2. [18] H. Gupta, G. Shi, and W. Yuan (2026) LUCID: learning embodiment-agnostic intent models from unstructured human videos for scalable dexterous robot skill acquisition. In CoRL, Cited by: §2. [19] A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makoviichuk, K. Van Wyk, A. Zhurkevich, B. Sundaralingam, Y. Narang, J. Lafleche, D. Fox, and G. State (2023) DeXtreme: transfer of agile in-hand manipulation from simulation to reality. In ICRA, Cited by: §1. [20] N. Heppert, M. Argus, T. Welschehold, T. Brox, and A. Valada (2024) Ditto: demonstration imitation by trajectory transformation. In IROS, Cited by: §1, §2. [21] C. Hsu, B. Wen, J. Xu, Y. Narang, X. Wang, Y. Zhu, J. Biswas, and S. Birchfield (2025) SPOT: SE(3) pose trajectory diffusion for object-centric manipulation. In ICRA, Cited by: §2. [22] L. Huang, H. Zhang, Z. Wu, S. Christen, and J. Song (2025) FunGrasp: functional grasping for diverse dexterous hands. RA-L. Cited by: §2. [23] W. Huang, C. Wang, Y. Li, R. Zhang, and L. Fei-Fei (2024) ReKep: spatio-temporal reasoning of relational keypoint constraints for robotic manipulation. In CoRL, Cited by: §1, §2. [24] W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei (2023) VoxPoser: composable 3d value maps for robotic manipulation with language models. In CoRL, Cited by: §1. [25] B. Ke, D. Narnhofer, S. Huang, L. Ke, T. Peters, K. Fragkiadaki, A. Obukhov, and K. Schindler (2025) Video depth without video models. In CVPR, Cited by: §4.5.1. [26] K. Kedia, T. G. W. Lum, J. Bohg, and C. K. Liu (2026) SimToolReal: an object-centric policy for zero-shot dexterous tool manipulation. In RSS, Cited by: §1, §2, §4.4. [27] J. Kerr, C. M. Kim, M. Wu, B. Yi, Q. Wang, K. Goldberg, and A. Kanazawa (2024) Robot see robot do: imitating articulated object manipulation with monocular 4d reconstruction. In CoRL, Cited by: §1. [28] Y. Kuang, S. Park, K. Fragkiadaki, and S. Tulsiani (2026) Dex4D: task-agnostic point track policy for sim-to-real dexterous manipulation. In RSS, Cited by: §1, §2, §4.4. [29] J. Li, Y. Zhu, Y. Xie, Z. Jiang, M. Seo, G. Pavlakos, and Y. Zhu (2024) Okami: teaching humanoid robots manipulation skills through single video imitation. In CoRL, Cited by: §2. [30] K. Li, P. Li, T. Liu, Y. Li, and S. Huang (2025) Maniptrans: efficient dexterous bimanual manipulation transfer via residual learning. In CVPR, Cited by: §1, §2. [31] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng (2023) Code as policies: language model programs for embodied control. In ICRA, Cited by: §1, §2. [32] X. Liu, J. Adalibieke, Q. Han, Y. Qin, and L. Yi (2025) Dextrack: towards generalizable neural tracking control for dexterous manipulation from human references. In ICLR, Cited by: §1, §2. [33] T. G. W. Lum, O. Y. Lee, C. K. Liu, and J. Bohg (2025) Crossing the human-robot embodiment gap with sim-to-real rl using one human demonstration. In CoRL, Cited by: §2. [34] T. G. W. Lum, A. H. Li, P. Culbertson, K. Srinivasan, A. D. Ames, M. Schwager, and J. Bohg (2024) Get a grip: multi-finger grasp evaluation at scale enables robust sim-to-real transfer. In CoRL, Cited by: §2. [35] T. G. W. Lum, M. Matak, V. Makoviychuk, A. Handa, A. Allshire, T. Hermans, N. D. Ratliff, and K. Van Wyk (2024) Dextrah-g: pixels-to-action dexterous arm-hand grasping with geometric fabrics. In CoRL, Cited by: §2, §3.2. [36] Z. Luo, J. Cao, S. J. Christen, A. Winkler, K. Kitani, and W. Xu (2024) Omnigrasp: grasping diverse objects with simulated humanoids. In NeurIPS, Cited by: §2. [37] Z. Mandi, Y. Hou, D. Fox, Y. Narang, A. Mandlekar, and S. Song (2025) Dexmachina: functional retargeting for bimanual dexterous manipulation. arXiv:2505.24853. Cited by: §1, §2. [38] M. Mittal, P. Roth, J. Tigue, A. Richard, O. Zhang, P. Du, A. Serrano-Muñoz, X. Yao, R. ZurbrĂŒg, N. Rudin, L. Wawrzyniak, M. Rakhsha, A. Denzler, E. Heiden, A. Borovicka, O. Ahmed, I. Akinola, A. Anwar, M. T. Carlson, J. Y. Feng, A. Garg, R. Gasoto, L. Gulich, Y. Guo, M. Gussert, A. Hansen, M. Kulkarni, C. Li, W. Liu, V. Makoviychuk, G. Malczyk, H. Mazhar, M. Moghani, A. Murali, M. Noseworthy, A. Poddubny, N. Ratliff, W. Rehberg, C. Schwarke, R. Singh, J. L. Smith, B. Tang, R. Thaker, M. Trepte, K. Van Wyk, F. Yu, A. Millane, V. Ramasamy, R. Steiner, S. Subramanian, C. Volk, C. Chen, N. Jawale, A. V. Kuruttukulam, M. A. Lin, A. Mandlekar, K. Patzwaldt, J. Welsh, H. Zhao, F. Anes, J. Lafleche, N. MoĂ«nne-Loccoz, S. Park, R. Stepinski, D. Van Gelder, C. Amevor, J. Carius, J. Chang, A. H. Chen, P. d. H. Ciechomski, G. Daviet, M. Mohajerani, J. von Muralt, V. Reutskyy, M. Sauter, S. Schirm, E. L. Shi, P. Terdiman, K. Vilella, T. Widmer, G. Yeoman, T. Chen, S. Grizan, C. Li, L. Li, C. Smith, R. Wiltz, K. Alexis, Y. Chang, D. Chu, L. J. Fan, F. Farshidian, A. Handa, S. Huang, M. Hutter, Y. Narang, S. Pouya, S. Sheng, Y. Zhu, M. Macklin, A. Moravanszky, P. Reist, Y. Guo, D. Hoeller, and G. State (2025) Isaac lab: a gpu-accelerated simulation framework for multi-modal robot learning. arXiv:2511.04831. Cited by: §4. [39] OpenAI, M. Andrychowicz, B. Baker, M. Chociej, R. JĂłzefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba (2020) Learning dexterous in-hand manipulation. IJRR. Cited by: §1. [40] OpenAI (2026) GPT-5.5 system card. Note: https://openai.com/index/gpt-5-5-system-card/ Cited by: §1. [41] C. Pan, C. Wang, H. Qi, Z. Liu, H. Bharadhwaj, A. Sharma, T. Wu, G. Shi, J. Malik, and F. Hogan (2026) SPIDER: scalable physics-informed dexterous retargeting. In IROS, Cited by: §1, §2, §4.5.2. [42] A. Patel and S. Song (2025) GET-Zero: graph embodiment transformer for zero-shot embodiment generalization. In ICRA, Cited by: §2. [43] S. Patel, S. Mohan, H. Mai, U. Jain, S. Lazebnik, and Y. Li (2026) Robotic manipulation by imitating generated videos without physical demonstrations. In ICLR, Cited by: §1, §2, §4.5.1. [44] H. Qi, A. Kumar, R. Calandra, Y. Ma, and J. Malik (2022) In-hand object rotation via rapid motor adaptation. In CoRL, Cited by: §1, §3.2. [45] H. Qi, B. Yi, S. Suresh, M. Lambeta, Y. Ma, R. Calandra, and J. Malik (2023) General in-hand object rotation with vision and touch. In CoRL, Cited by: §1. [46] N. Ravi, V. Gabeur, Y. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. RĂ€dle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V. Alwala, N. Carion, C. Wu, R. Girshick, P. DollĂĄr, and C. Feichtenhofer (2025) SAM 2: segment anything in images and videos. In ICLR, Cited by: §3.5, §4. [47] J. Romero, D. Tzionas, and M. J. Black (2022) Embodied hands: modeling and capturing hands and bodies together. arXiv preprint arXiv:2201.02610. Cited by: §A.2. [48] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv:1707.06347. Cited by: §4. [49] B. Sundaralingam, S. K. S. Hari, A. Fishman, C. Garrett, K. Van Wyk, V. Blukis, A. Millane, H. Oleynikova, A. Handa, F. Ramos, N. Ratliff, and D. Fox (2023) Curobo: parallelized collision-free robot motion generation. In ICRA, Cited by: Appendix B. [50] W. Wan, H. Geng, Y. Liu, Z. Shan, Y. Yang, L. Yi, and H. Wang (2023) UniDexGrasp++: improving dexterous grasping policy learning via geometry-aware curriculum and iterative generalist-specialist learning. In ICCV, Cited by: §2. [51] R. Wang, J. Zhang, J. Chen, Y. Xu, P. Li, T. Liu, and H. Wang (2023) Dexgraspnet: a large-scale robotic dexterous grasp dataset for general objects based on simulation. In ICRA, Cited by: Table 12, Table 12, §2, §4.1.1. [52] Y. Wei, M. Lin, Y. Lin, J. Jiang, X. Wu, L. Zeng, and W. Zheng (2025) AffordDexGrasp: open-set language-guided dexterous grasp with generalizable-instructive affordance. In ICCV, Cited by: §2. [53] Z. Wei, Z. Xu, J. Guo, Y. Hou, C. Gao, Z. Cai, J. Luo, and L. Shao (2025) âĄ(â,)D(R,O) Grasp: a unified representation of robot and object interaction for cross-embodiment dexterous grasping. In ICRA, Cited by: §E.4, §E.5, Appendix E, §2, §3.2, §3.3, §4.1.1, §4.3, Table 1. [54] Z. Wei, Y. Yao, and M. Ding (2026) One hand to rule them all: canonical representations for unified dexterous manipulation. In RSS, Cited by: §2, §4.6.1, Table 4. [55] B. Wen, W. Yang, J. Kautz, and S. Birchfield (2024) Foundationpose: unified 6d pose estimation and tracking of novel objects. In CVPR, Cited by: §3.5, §4.5.1, §4. [56] C. Wen, X. Lin, J. So, K. Chen, Q. Dou, Y. Gao, and P. Abbeel (2024) Any-point trajectory modeling for policy learning. In RSS, Cited by: §2. [57] Z. Wu, X. Zhang, Z. Chen, J. Deng, R. A. Potamias, and S. Luo (2026) UniMorphGrasp: diffusion model with morphology-awareness for cross-embodiment dexterous grasp generation. arXiv preprint arXiv:2602.00915. Cited by: §2. [58] M. Xu, Z. Xu, Y. Xu, C. Chi, G. Wetzstein, M. Veloso, and S. Song (2024) Flow as the cross-domain manipulation interface. In CoRL, Cited by: §2. [59] Y. Xu, W. Wan, J. Zhang, H. Liu, Z. Shan, H. Shen, R. Wang, H. Geng, Y. Weng, J. Chen, T. Liu, L. Yi, and H. Wang (2023) UniDexGrasp: universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy. In CVPR, Cited by: §2. [60] L. Yang, K. Li, X. Zhan, F. Wu, A. Xu, L. Liu, and C. Lu (2022) Oakink: a large-scale knowledge repository for understanding hand-object interaction. In CVPR, Cited by: §1, §3.4. [61] H. Yuan, B. Zhou, Y. Fu, and Z. Lu (2025) Cross-embodiment dexterous grasping with reinforcement learning. In ICLR, Cited by: §2, §4.2.1, Table 1. [62] H. Zhang, S. Christen, Z. Fan, O. Hilliges, and J. Song (2024) GraspXL: generating grasping motions for diverse objects at scale. In ECCV, Cited by: §2. [63] H. Zhang, M. Meboldt, and J. Song (2026) ConGraspXL: controllable constraint-conditioned dexterous grasping motion synthesis. arXiv:2609.16319. Cited by: §2. [64] H. Zhang, Z. Wu, L. Huang, S. Christen, and J. Song (2025) Robustdexgrasp: robust dexterous grasping of general objects. In CoRL, Cited by: §A.3, §2, §3.2, §3.2. [65] Y. Zhu, A. Lim, P. Stone, and Y. Zhu (2026) Vision-based manipulation from single human video with open-world object graphs. Autonomous Robots. Cited by: §2. Appendix A Reinforcement Learning Details A.1 Reward Function Details The per-step reward is r= r\;=\; wdârdist+wcârcontact+wlârlift+wopârobjp+worârobjr, w_d\,r_dist+w_c\,r_contact+w_l\,r_lift+w_o^p\,r^p_obj+w_o^r\,r^r_obj, (1) Reward weights are listed in Table 6. Distance reward rdistr_dist. A negative weighted sum of link-to-surface distances, each clipped at 0.10.1 m: rdist=ââlÏlmin(â„lâ„, 0.1),r_dist=- _l _l\, \! ( _l ,\,0.1 ), with unnormalized per-link weights Ïthumb-tip=8 _thumb-tip=8, Ïother fingertips=4 _other fingertips=4, Ïpalm=0 _palm=0, and Ïl=1 _l=1 for all remaining links. In cross-embodiment training, the weights are masked to active links and normalized to sum to 11 per environment. For each canonical link l at world position lp_l, la_l is the displacement to the nearest of M=512M=512 sampled object-surface points, and â„lâ„ _l is its Euclidean norm. The point cloud is stored in the object frame and transformed to the world frame using the tracked object pose. Contact reward rcontactr_contact. Encourages both thumb contact and an opposing grasp, in which the thumb and at least one non-thumb finger simultaneously contact the object: rcontact=â[thumb in contact]+â[is_grasping],r_contact=1[thumb in contact]+1[is\_grasping], where a finger is considered to be in contact if its contact force with the object exceeds Ï”F=0.05 _F=0.05 N, and is_grasping=(thumb in contact)â§(ânon-thumb finger in contact)is\_grasping=(thumb in contact) (â\,non-thumb finger in contact). Lifting reward rliftr_lift. A grasp-gated, scaled reward proportional to the objectâs height above its post-reset position: rlift=â[is_grasping]â clipâĄ(100âÎâh, 0, 100âhthr),Îâh=maxâĄ(0,zâz0),r_lift=1[is\_grasping]·clip\! (100\, h,\ 0,\ 100\,h_thr ), h= (0,\ z-z_0), where z is the objectâs current height, z0z_0 is its height immediately after reset, and hthr=0.05h_thr=0.05 m (so the clip ceiling is 5.05.0). The indicator â[is_grasping]1[is\_grasping] ensures that lifting is rewarded only when the object is grasped. Clipping prevents additional reward from throwing the object upward. Object stability reward robjr_obj. Exponential âstay-putâ bonuses on planar displacement and rotation from the post-reset pose: robjp=eâ10ââ„Îââ„,robjr=eâÎâΞâ(,0),r^p_obj=e^-10\, , r^r_obj=e^- Ξ(q,q_0), where Îâââ2 ^2 is the planar displacement from the objectâs post-reset position, and ÎâΞâ(,0)=2âarccosâĄ(|âš,0â©|) Ξ(q,q_0)=2 (| ,q_0 |) is the geodesic angle between its current and post-reset orientations, represented by unit quaternions. These rewards discourage sliding or rotating the object during contact acquisition, which we found important for stable real-world execution. Curriculum / annealing. The object-stability weights are linearly annealed from an initial value to the final value over TcurT_cur per-environment control steps: wâĄ(t)=winitâ(1âÏ)+wfinalâÏw(t)=w_init(1-Ï)+w_finalÏ, Ï=minâĄ(1,t/Tcur)Ï= (1,t/T_cur). We anneal wop,worw_o^p,w_o^r from winit=2.0w_init=2.0 to wfinal=10.0w_final=10.0 over Tcur=105T_cur=10^5 steps. The distance, contact, and lifting weights are not annealed. Table 6: Reward weights. Symbol Term Value wdw_d distance 16.016.0 wcw_c contact 4.04.0 wopw_o^p object position stability 10.010.0 (init. 2.02.0) worw_o^r object rotation stability 10.010.0 (init. 2.02.0) wlw_l lifting 1.01.0 TcurT_cur annealing steps 10510^5 A.2 Cross-Embodiment Unified Policy Representation Canonical joint space. We map every hand onto a fixed canonical joint layout consisting of Jcan=26J_can=26 slots partitioned into five contiguous per-finger blocks (6 for the thumb and 5 for each other finger). The wrist is excluded from this layout and controlled separately through a shared 6-DoF floating-base interface. Each hand fills only the slots corresponding to its native joints, and the remaining slots are masked. The number of active slots equals the handâs DoF: XHand 1212, Allegro 1616, and Sharpa Wave 2222 (Table 7). Table 7: Canonical 26-DoF joint layout and per-hand instantiation. Each finger is allocated a fixed number of slots in the canonical layout (6 for the thumb, 5 for each non-thumb finger); slots are filled by a handâs native joint when present, and masked ([]) otherwise. F, A, and R denote flexion, abduction/adduction, and axial rotation respectively. Reading order within each finger is base â tip. Hand DoF Thumb (6) Index (5) Middle (5) Ring (5) Pinky (5) Allegro 16 RAF[ ]F[ ] R[ ]F R[ ]F R[ ]F [ ][ ][ ][ ][ ] Sharpa Wave 22 [ ]AFAFF [ ]AFFF [ ]AFFF [ ]AFFF FAFFF XHand 12 [ ]AF[ ]F[ ] [ ]AFF[ ] [ ][ ]F[ ] [ ][ ]F[ ] [ ][ ]F[ ] Canonical link space. For the canonical link ordering, we map each robotâs native link positions (obtained via forward kinematics from its URDF) onto a fixed Lcan=21L_can=21 keypoint skeleton following the standard hand topology used in MANO [47]: one palm keypoint (canonical index 00) plus four keypoints per finger (5Ă45Ă 4) ordered base-to-tip, with fingertips at canonical indices 4,8,12,16,20\4,8,12,16,20\. Link observations and link-based rewards use this canonical ordering. Per-robot canonical mapping. Each robot r stores an index map Ïr _r and a sign vector srs_r with entries in â1,+1\-1,+1\ that translate between its native joints and the canonical slots. The policy reads and writes in the canonical space. At actuation, the canonical hand command ahandââJcana^hand\!â\!R^J_can is gathered into native joint order and sign-corrected to account for differences in native joint-axis conventions: arnative=ahandâ[Ïr]âsr.a^native_r\;=\;a^hand[ _r] s_r. Masks and padding. For each robot we form binary masks ÎŽjointâ0,126ÎŽ^joint\!â\!\0,1\^26 and ÎŽlinkâ0,121ÎŽ^link\!â\!\0,1\^21 that flag occupied canonical slots. Both masks are appended to the observation so that the policy can identify active dimensions. Unoccupied joint slots are filled with zeros, and missing link positions are filled using the nearest existing link. A.3 Observations The observation concatenates three blocks: a 5-frame proprioceptive history, single-frame hand-object displacement vectors, and joint and link masks. The policy does not receive the object pose explicitly. Object information enters only through the per-link nearest-surface displacement vectors la_l defined in the distance reward above. Within the history buffer, frames are ordered from oldest to newest, [otâK+1,âŠ,ot][o_t-K+1,âŠ,o_t] with history length K=5K=5. At episode start, the buffer is padded with copies of the first observed frame. The full observation breakdown is given in Table 8. Table 8: Observation components. K=5K\!=\!5. Component Composition Dim Joint positions 2626 canonical ĂKĂ\,K frames 26âK26K Joint targets 2626 canonical ĂKĂ\,K frames 26âK26K Link positions 2121 links Ă 3ĂKĂ\,3Ă\,K frames 63âK63K Hand-object vectors 2121 links Ă 1Ă\,1 nearest pt Ă 3Ă\,3 6363 Joint mask active canonical joints 2626 Link mask active canonical links 2121 Joint positions and joint targets. Joint positions and previous-step joint targets are reordered into the canonical 26-joint layout and padded with zeros for joints not present on the active hand. Link positions. We use forward kinematics to compute the 3D world positions of links in the canonical 21-keypoint topology, discarding their orientations. Missing keypoints are padded using the nearest existing link, as described in Section A.2. Hand-object vectors (following [64]). For each of the 2121 canonical links we compute the vector to its nearest point on the object surface point cloud (N=512N\!=\!512 points sampled per object), giving a 21Ă321Ă 3 tensor. Normalization. There is no in-environment normalization. rl_games normalizes observations and value targets using running means and standard deviations. Observations are clipped to [â5,5][-5,5]. A.4 Policy Action Action space. The action is a=[aEEâ(6),ahandâ(Jcan)]a=[\,a^E(6),\ a^hand(J_can)\,], i.e. 6+26=326+26=32 dimensions. The first 66 dimensions specify an end-effector delta (3 translation components and 3 Euler angles) for the shared floating base, converted to the 66 virtual base-joint targets. The hand block is in canonical space and remapped per robot as in Section A.2. Range, scaling, and clipping. Raw policy actions are clipped to [â1,1][-1,1] before being rescaled into physical commands. All controllers operate in residual (delta-target) mode: each scaled delta is added to the previous control target, which is then clamped to the corresponding physical limit. For the hand, per-joint deltas are rescaled using each jointâs upper and lower limits so that an action of ±1± 1 corresponds to a full-range increment. The end-effector deltas are scaled to ±0.008± 0.008 m per step for translation and ±0.052± 0.052 rad per step for rotation, then composed with the previous target pose. Control frequency. Simulation uses a timestep of dt=1/120dt=1/120 s. With decimation=12=12, each action is held for 1212 physics substeps, giving a control rate of 1010 Hz. A.5 Reset and Initialization Length and termination. We use a fixed episode horizon of H=60H=60 control steps (corresponding to 6.06.0 s at 1010 Hz). An episode terminates when the step counter reaches H. At the start of training, episode step counters across the 4,096 parallel environments are initialized uniformly at random in 0,1,âŠ,Hâ1\0,1,âŠ,H-1\ rather than synchronously at zero. This staggers episode resets so that rollout batches contain transitions from different phases (approach, contact, and lifting), improving sample diversity and stabilizing early training. Reset distribution. At scene initialization, each environment randomly selects a robot embodiment and an object. At each reset, it samples a valid pair (object resting pose, initial hand configuration) from the per-object grasp dataset. We further augment the initial state with a random planar translation ÎâxâyâŒâĄ([â0.05,0.05]2) _xy ([-0.05,0.05]^2) m and a random yaw ÏâŒâĄ(âÏ,Ï)Ï (-Ï,Ï) about the world z-axis. The same transformation is applied to both components, preserving their relative geometry. A.6 Domain Randomization We apply domain randomization to encourage robustness and sim-to-real transfer. All physics parameters are resampled at every environment reset. Friction and restitution are set directly to values sampled from the listed ranges. Mass, scale, and actuator gains are scaled relative to their nominal values for each robot or object. The nominal object density is Ï=200Ï=200\,kg/m3, which determines each objectâs baseline mass before the Ă[0.5,3.0]Ă[0.5,3.0] mass scale is applied. Table 9: Domain randomization ranges. All distributions are uniform. Multiplicative ranges (Ă) scale the per-object or per-robot nominal value; absolute ranges directly set the parameter value. Group Parameter Range Type Object Scale [0.96,1.04][0.96,1.04] Ă nominal Mass [0.5,3.0][0.5,3.0] Ă nominal (from Ïâ Vâ scale3Ï· V·scale^3) Static/dynamic friction [0.5,1.5][0.5,1.5] absolute Restitution [0.0,0.1][0.0,0.1] absolute Robot links Static/dynamic friction [0.5,2.0][0.5,2.0] absolute Restitution [0.1,0.5][0.1,0.5] absolute Actuators Stiffness (kpk_p) [0.5,2.0][0.5,2.0] Ă nominal Damping (kdk_d) [0.3,3.0][0.3,3.0] Ă nominal Perturbation Push force (max) 2.02.0 N sampled uniformly per axis Application offset ±0.05± 0.05 m from object center Nominal actuator gains. The floating arm base has kp=[1000,1000,1000,300,300,300]k_p=[1000,1000,1000,300,300,300] and kd=[100,100,100,15,15,15]k_d=[100,100,100,15,15,15] for its six virtual joints. The per-hand kpk_p and kdk_d values are obtained through system identification on the physical hardware. A.7 Contact Prediction Details We train a separate contact predictor using rollouts from the frozen policy after convergence. The predictor is a 2-layer MLP with a hidden size of 256 and ELU activation. It takes joint positions, joint targets, and link positions over the most recent k=5k=5 steps and outputs five per-finger contact logits. We train it with binary cross-entropy. The ground-truth label for each finger is 1 if the contact-sensor force exceeds Ï”F=0.05 _F=0.05 N. Training. At each epoch, we collect 120 steps using deterministic actions from the frozen policy across 4,096 parallel environments. We train the predictor on the resulting in-memory buffer in shuffled mini-batches of 4,096 with Adam at a learning rate of 3Ă10â43Ă 10^-4. We train for 200 epochs in total. Deployment. At inference, contact probabilities are obtained by sigmoid activation and thresholded at 0.50.5. We set is_grasping to true when the thumb and at least one non-thumb finger are predicted to be in contact. We switch to tracking when is_grasping remains true for five consecutive control steps. A.8 Additional Training Details Network architecture. We use a feed-forward MLP with hidden sizes [512,512,256,128][512,512,256,128]. Training hyperparameters are listed in Table 10. Table 10: PPO hyperparameters. Parameter Value Parallel environments 40964096 Horizon (rollout) length 6060 Minibatch size 81928192 Mini-epochs per update 55 Discount Îł 0.950.95 GAE λ (tau) 0.900.90 Learning rate 3Ă10â43Ă 10^-4, adaptive (KL) KL threshold 0.0160.016 PPO clip ϔΔ 0.20.2 Value loss coef. (critic_coef) 44 Value clipping enabled Entropy coef. 0.00.0 Bounds-loss coef. 1Ă10â41Ă 10^-4 Gradient norm clip 1.01.0 Normalize input / value / advantage yes / yes / yes Mixed precision no Max epochs 20002000 Seed 42,84,12642,84,126 Figure 10: Simulation training assets. Object set. We use 1616 YCB objects, as illustrated in Figure 10. Each object has 22â77 distinct stable resting poses on the table (âŒ100 \!100 objectâpose configurations per hand). Grasp data generation and filtering. For each (object, resting-pose) pair, we generate grasp candidates with BODex [7]. BODex uses a heuristic seed sampler to propose an initial hand configuration, which we keep for the reset distribution (Section A.5). The full bilevel grasp optimization then refines this seed into a converged reference grasp. We validate the reference grasps with BODexâs official evaluation code and retain 4,000 successful (initial configuration, reference grasp) pairs per (object, resting-pose) pair. Hardware and training duration. We train each policy in IsaacLab on a single NVIDIA L40S GPU for 2000 epochs, taking 24 hours. This corresponds to approximately 491.5 million environment steps, with 60 steps across 4096 parallel environments per epoch. Appendix B Future-Aware Planning Details At deployment, the planner identifies a task-relevant contact region and then generates candidate initializations within this region. A vision-language model (VLM) provides the task-aware affordance selection, while candidate generation and trajectory selection are performed geometrically and kinematically. Task-aware region selection. We render the object at its observed pose on the table from four side views (Figure 11 (a)). The views use textured rendering with a low camera elevation and are overlaid with a light pixel grid for image-coordinate reference. The VLM is prompted to identify the objectâs functional parts, mark each visible part in each view, and designate the single part to be grasped for the specified task (Figure 11 (b)). For the designated part, the VLM additionally predicts multiple points spanning its extent. All predicted points are back-projected onto the object mesh by ray casting. For each functional part, its annotations across views are clustered in the object frame, and the mean of the largest cluster is used as the part location. Predicted grasp points whose back-projected locations are inconsistent with the designated part are discarded. We define the task-relevant contact region as the union of 33\,cm surface neighborhoods around the remaining grasp points (Figure 11 (c)). Candidate generation. Following the initialization stage of BODex [7], we sample candidate hand poses and joint configurations from surface points within the selected contact region. We include the table in the collision world and reject candidate initializations that collide with the table or other scene geometry, as well as ill-posed poses. We retain N=10N=10 valid candidate initializations by default. These candidates serve as approximate initial configurations for the contact-acquisition primitive rather than final grasps. Trajectory expansion and selection. For each remaining candidate, we plan a collision-free approach from the robotâs initial configuration to the initial grasp pose with cuRobo [49]. We then assume a rigid hand-object attachment and compute the hand trajectory along the desired object motion Ïo _o, solving IK along this trajectory to recover the full arm-hand motion. Candidates whose approach or tracking segment is infeasible are discarded. Among the remaining candidates, we select the trajectory with the smallest arm motion, ciik=ât=1Tâ„qi,tarmâqi,tâ1armâ„2,c^ik_i\;=\; _t=1^T \,q^arm_i,t-q^arm_i,t-1\, _2, where qi,tarmq^arm_i,t denotes the arm configuration obtained by IK for candidate i at timestep t. Prompt and example. We use Claude Opus 4.7 or Claude Opus 5 via the Anthropic API. The prompt specifies the manipulation task and provides generic guidance distinguishing the holding region from the working end of an object. Given the four rendered views, the model returns a JSON list of labeled object parts, with one pixel coordinate for each visible part in each view and a binary grasp flag indicating the part selected for grasping. Figure 11 shows an example from a Drill Use trial. The VLM identifies the pistol grip as the task-relevant holding region, and the subsequent candidate generation is restricted to this region. You are labeling the functional KEYPOINTS of an object for a dexterous robot, then choosing which one to grasp. Task: task_name The object may have a designated HOLDING REGION (shaped to fit a hand) vs a WORKING END (does the task). Grasp the HOLDING REGION. [...] You are given 4 rendered views of the SAME object standing on a table, from different angles. Each image is 640x480 px with a light coordinate grid [...] Step 1 --- Identify the objectâs distinct FUNCTIONAL KEYPOINTS (the named parts a person would talk about). Give each a short label. Step 2 --- For EACH keypoint, mark its pixel in EVERY view where it is clearly visible. All picks for one keypoint must be the SAME physical point, so we can triangulate it in 3D. Step 3 --- Set "grasp": true on the ONE keypoint the hand should converge on to hold the object for this task, and false on the rest. Step 4 --- For that grasp keypoint ONLY, also give "extent": 2--4 extra pixel picks that span the full length of the holding part (e.g. both ends and the middle of a grip or handle). Together with the keypoint they define WHERE the fingers may land, so cover the whole holdable part but do NOT extend into the working end or onto other parts. Respond with ONLY a JSON object: "keypoints": ["label": <str>, "grasp": <bool>, "picks": ["view": <int>, "u": <int>, "v": <int>, ...], "extent": ["view": <int>, "u": <int>, "v": <int>, ...], ...], "reason": "<why>" ("extent" only on the grasp keypoint; omit it elsewhere.) Figure 11: Task-aware region selection for Drill Use. (a) Four rendered views provided to the VLM. (b) The VLM identifies functional parts and selects the pistol grip as the task-relevant holding region, with multiple grasp points spanning the grip. (c) Back-projected grasp points define the contact region (yellow), from which candidate initializations are sampled. Appendix C Real-World Grasping Experiment Details This section provides per-object real-world grasping results across the three arm-hand platforms (Table 11). Tianji-XHand (48/50, 96%) achieves higher overall success than both Franka platforms (45/50 each, 90%). We attribute this to the larger reachable workspace of the Tianji arm relative to our Franka setup. Table 11: Per-object grasp success across embodiments (out of 5 trials). Objects are abbreviated as O1-O10: O1 mustard-bottle, O2 bleach-cleanser, O3 drill-red, O4 tomato-soup, O5 meat-can, O6 red-bottle, O7 drill-green, O8 apple-tea, O9 eraser, O10 maize. Objects O1âO5 are seen during training; O6âO10 are unseen. Embodiment O1 O2 O3 O4 O5 O6 O7 O8 O9 O10 Total (%) Franka-Sharpa 4 4 5 5 5 5 4 3 5 5 90 Franka-XHand 5 5 5 4 5 3 5 4 5 4 90 Tianji-XHand 5 5 4 4 5 5 5 5 5 5 96 Appendix D Real-World Full-Pipeline Experiment Visualizations Figure 12: Qualitative comparison on real-world tasks. Subfigures (a)-(b) are from BODex, and (c)-(d) are from DRO. We present qualitative comparisons of the full pipeline against the DRO and BODex baselines in Figure 12, illustrating three recurring failure modes: (i) grasping at an unintended contact region that does not support the downstream trajectory (Figure 12(d)), (i) displacing the object during approach (Figure 12(a, b)), and (i) forming a cage-like grasp without firm opposing finger-thumb contact (Figure 12(c)). In contrast, GOTT addresses these failure modes by (i) adjusting the wrist and fingers locally to recover from imperfect pre-grasp poses produced by upstream planning, (i) reacting to object displacement during approach, and (i) establishing firm opposing finger-thumb contact rather than caging the object loosely. We refer the reader to the supplementary video or our project website for video demonstrations of real-world performance. Appendix E Simulation Grasping Evaluation Protocol and Baseline Details The grasp success rates we report for BODex [7] and DRO [53] in Table 1 of the main paper are lower than those reported in their original publications. This section clarifies the source of this gap and presents a controlled reproduction. E.1 Protocol Comparison with BODex BODexâs original evaluation places each scaled object in a floating state without a table. Ten grasps are synthesized per (object, scale, pose) instance. The hand is initialized at the pre-grasp pose px_p and driven to the squeezed pose sx_s. Gravity is then applied along the positive and negative directions of each of the three coordinate axes. A grasp counts as successful only if the objectâs translation and rotation deviations remain within 55\,cm and 15â15 for at least 33\,s under all six directions. In contrast, our protocol places objects on a tabletop under gravity from the start of the trial. Both seed-pose initialization (used identically across all methods) and grasp execution must avoid hand-table collisions throughout the trial. This setting is closer to real-world deployment, where objects rest on a workspace surface and the robot must operate within the resulting constraints. In summary, the two protocols differ in two respects: âą Generation phase. BODex synthesizes grasps without considering a table. Our setup adds hand-table collision constraints to both initial-pose sampling and the bilevel optimization. âą Evaluation phase. BODex evaluates grasps in a floating-hand state with gravity applied after squeezing along all six directions. Our setup places the object on a table under âz-z gravity from the start of the trial. E.2 Reproducing BODex under Its Evaluation Protocol To isolate the effect of the generation procedure, we use BODexâs officially released code with its default parameters (in the main branch), synthesize 20 initial poses per object under two conditions, and evaluate both under BODexâs original protocol: âą Floating generation (BODex default): initial-pose sampling and bilevel optimization ignore the table. âą Tabletop generation (ours): initial-pose sampling and bilevel optimization avoid hand-table collisions. Table 12: Effect of generation-time table constraints on BODex grasp success rate (%). Allegro right hand, 4 object scales, evaluated under BODexâs original protocol. Only the generation phase differs between the two columns. Objects are taken from DexGraspNet [51], normalised so that each objectâs bounding-box diagonal measures 11\,m, and then rescaled to four target diagonals (@006=0.060.06\,m, @008=0.080.08\,m, @010=0.100.10\,m, @012=0.120.12\,m). Scale @006 @008 @010 @012 avg. Floating 69.79 73.79 72.09 67.26 70.73 Tabletop 53.29 60.90 60.05 54.28 57.13 Table 12 shows that tabletop-aware generation alone, with evaluation held fixed, drops the success rate from 70.7%70.7\% to 57.1%57.1\%. This degradation is attributable purely to the harder generation problem: the init-pose distribution changes and the bilevel optimization becomes more constrained. The floating-generation result is below the 79.9%79.9\% reported in [7] because the released codeâs main branch differs from the paperâs configuration (preserved in the baseline branch) in object mass (100100\,g vs. 3030\,g) and joint stiffness (kp=5k_p=5 vs. kp=2k_p=2); rerunning floating evaluation with the paperâs configuration (object mass 3030\,g and kp=2k_p=2) yields 77.2%77.2\%, with the residual gap likely due to seed variance. Following the authorsâ recommendation, Table 12 and Figure 13 use the main-branch defaults; this choice does not affect the conclusions above, since both rows of Table 12 share the same configuration. E.3 Evaluation-Phase Comparison We further evaluate the same set of BODex-generated grasps under both protocols. Figure 13 plots, for each (embodiment, object set) pair, BODexâs success rate under its original protocol (x-axis) against the success rate of the same grasps under our tabletop protocol (y-axis). The points cluster close to the diagonal, indicating that the two evaluation protocols give similar success-rate estimates. Reference lines mark GOTTâs tabletop success rates of 0.74 (XHand), 0.75 (Sharpa), and 0.87 (Allegro), which lie well above BODexâs results. Even when BODex grasps are evaluated under its original floating-hand protocol, GOTT outperforms BODex on every embodiment, confirming that the gap in Table 1 of the main paper is not an artifact of the evaluation choice. Figure 13: Evaluation protocol comparison. The same BODex grasps evaluated under BODexâs original protocol and our tabletop protocol. E.4 Protocol Comparison with DRO DRO [53] evaluates its grasps in Isaac Gym, where a controller closes the hand around the object. The grasp is then perturbed by sequential forces along the positive and negative directions of each coordinate axis, with a magnitude corresponding to an acceleration of 0.5âm/s20.5\,m/s^2 (i.e., F=mobjâ(0.5âm/s2)F=m_obj(0.5\,m/s^2)). A grasp is successful if the objectâs displacement stays below 2 cm. Our protocol differs in two ways. First, we require the grasp to lift the object against gravity rather than resist a brief perturbation. The gravitational load Fg=mobjâ(9.8âm/s2)F_g=m_obj(9.8\,m/s^2) is roughly 20Ă the perturbation force F used in DROâs protocol and is sustained throughout the trial. Second, our protocol places the object on a table from the start, whereas DROâs grasp synthesis does not account for table collisions, which limits its performance under our protocol. E.5 DRO Baseline Variants Our main table (Table 1) uses DROâs [53] released checkpoint. DRO is trained across embodiments and transfers zero-shot to new hands given their URDFs, but XHand and Sharpa Wave are not part of its training set. For a fair comparison, we additionally fine-tune or retrain DRO on our BODex-generated YCB grasp set. Specifically, we run BODexâs tabletop-aware bilevel optimization from the same initial hand configurations used by GOTT and keep the 4,000 grasps that pass BODexâs own validity check as the training set. All variants are evaluated under our tabletop protocol (Table 13). All fine-tuning and retraining variants underperform the released checkpoint on average across hands. We attribute this to our smaller 16-object training set with tabletop-constrained grasps, which differs from DROâs original training distribution. We therefore keep the released checkpoint in the main table as the strongest DRO variant overall. Notably, even on the natively supported Allegro hand, DRO reaches 43.2% versus our 87.1%, indicating that the gap primarily reflects the benefit of closed-loop contact acquisition over open-loop grasp execution under the same tabletop protocol. Table 13: DRO variants. Lifting success rate (%) under our tabletop protocol; best per column in bold. DRO variant Allegro Sharpa XHand Av. Released ckpt 43.2 47.0 31.3 40.5 Fine-tune 1 ep. 21.6 52.2 31.1 35.1 Fine-tune 5 ep. 20.0 37.9 32.0 30.0 Retrain 5 ep. 19.8 40.6 32.6 31.0 Retrain 40 ep. 18.6 35.5 31.3 28.5 Appendix F Additional Experiments F.1 Runtime Breakdown Table 14 reports the per-stage runtime averaged over 10 rollouts. One-time initialization (4-view rendering, VLM-based affordance reasoning, candidate generation, IK filtering, and cuRobo planning) totals about 29.6 s. Online execution is real-time: FoundationPose runs at 45 Hz on an RTX 4090, and the contact-acquisition and contact-prediction networks take 0.752 ms and 0.085 ms per step, respectively, supporting the 10 Hz control loop. Table 14: Per-stage runtime, averaged over 10 rollouts. Stage Time 4-view rendering 0.11 ± 0.01 s VLM-based affordance reasoning 14.9 ± 0.9 s Candidate generation 0.76 ± 0.01 s IK filtering 3.95 ± 0.06 s cuRobo planning (N=10N=10) 9.6 ± 0.5 s FoundationPose 45 Hz Contact acquisition (per step) 0.752 ms Contact predictor (per step) 0.085 ms F.2 Contact Predictor Evaluation We evaluate the contact predictor described in Section A.7 on unseen rollouts of the frozen contact-acquisition primitive. We use the 16 training YCB objects under the same domain randomization, with 2560 episodes of 60 steps per hand. Evaluation protocol. At each control step, is_grasping is set according to the per-finger contact predictions. We compare this signal against the simulatorâs ground-truth grasp state. A false positive is a step at which is_grasping is true while the simulator indicates no grasp, and a false negative is the reverse. At the episode level, a switch is considered correct if the predictor triggers tracking while the hand is actually holding the object. Results. At a contact-probability threshold of Ïc=0.5 _c=0.5, the per-step false-positive rate is 0.066â0.125 and the false-negative rate is 0.036â0.047 across the three hands. Per-episode switch accuracy is 0.953â0.960, with a median lag of at most one step relative to the simulatorâs switch. Switch decisions are insensitive to the threshold: switch accuracy remains 0.94â0.98 for Ïcâ[0.4,0.8] _câ[0.4,0.8] and degrades only for Ïcâ€0.3 _c†0.3. Figure 14: Contact-predictor accuracy under observation noise. Per-step accuracy of the debounced switch signal, averaged over the three hands, as the predictor input is perturbed along one axis at a time (solid: object position, 0â5 cm; dashed: joint angle, 0â0.5 rad). Our proprioceptive predictor is unaffected by object-pose noise and degrades least under joint noise. Robustness to observation noise. We train a separate proprioception-only predictor because it is more robust to observation noise than a contact head coupled to the policy. To verify this, we compare against two alternatives: a head inside the policy network trained jointly with PPO from the full policy observation, and per-embodiment proprioceptive predictors trained alongside the policy instead of after it. All three are evaluated offline on the same recorded rollouts using their respective inputs and the same ground-truth labels. Noise is added only to the predictor input, one axis at a time (object position up to 5 cm, joint angles up to 0.5 rad), while the rollouts themselves are left unchanged. As shown in Figure 14, our predictor is unaffected by object-pose noise by construction and loses at most 0.01 in accuracy under 0.5 rad joint noise, whereas the in-policy head loses on average 0.16 under object-position noise and the jointly trained heads lose on average 0.35 under joint noise.