Paper deep dive

EMPIRE: Explicit Manipulation Planning as a Learnable Intermediate Representation for Egocentric Hand-Motion Forecasting

Wen Wang, Ruibing Hou, Hong Chang, Shiguang Shan, Xilin Chen

Year: 2026Venue: arXiv preprintArea: cs.ROType: PreprintEmbeddings: 70

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Forecasting dexterous hand motions from egocentric observations is fundamental to intelligent interactive systems. Existing VLM-based methods typically map observations directly to future motions, overlooking the underlying manipulation process that governs hand-object interactions. Moreover, end-to-end optimization couples manipulation learning with motion synthesis, causing motion-generation gradients to interfere with the pre-learned manipulation-aware representations. To overcome these limitations, we propose EMPIRE, a two-stage framework that introduces Explicit Manipulation Planning as an Intermediate Representation for Egocentric hand-motion forecasting. Stage I: Learn to Plan. EMPIRE first learns explicit manipulation plans from multimodal context to capture the progression of hand-object interactions. Stage II: Learn to Act. A motion generator synthesizes future bimanual hand motions conditioned on frozen planner representations, preventing motion-generation gradients from affecting manipulation planning. To support our method, we further construct EMPIRE-651K, a bimanual hand-motion forecasting dataset comprising 650,910 training windows across 111 tasks, each paired with an explicit per-hand manipulation plan. Under identical training and evaluation protocols, EMPIRE achieves state-of-the-art forecasting accuracy, with an MPJPE of 84.53 mm and a finger-relative error of 38.97mm. We release the code and dataset at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

67,192 characters extracted from source content.

Expand or collapse full text

MPIRE: Explicit Manipulation Planning as a Learnable Intermediate Representation for Egocentric Hand-Motion Forecasting Wen Wang Ruibing Hou Hong Chang Shiguang Shan Xilin Chen Abstract Forecasting dexterous hand motions from egocentric observations is fundamental to intelligent interactive systems. Existing VLM-based methods typically map observations directly to future motions, overlooking the underlying manipulation process that governs hand-object interactions. Moreover, end-to-end optimization couples manipulation learning with motion synthesis, causing motion-generation gradients to interfere with the pre-learned manipulation-aware representations. To overcome these limitations, we propose EMPIRE, a two-stage framework that introduces Explicit Manipulation Planning as an Intermediate Representation for Egocentric hand-motion forecasting. Stage I: Learn to Plan. EMPIRE first learns explicit manipulation plans from multimodal context to capture the hand-object interactions progression. Stage I: Learn to Act. A motion generator synthesizes future bimanual-hand motions conditioned on frozen plan’s representations, preventing motion-generation gradients from affecting manipulation planning To support our method, we further construct EMPIRE-651K, a bimanual hand-motion forecasting dataset comprising 650,910650,910 training windows across 111111 tasks, each paired with an explicit per-hand manipulation plan. Under identical training and evaluation protocols, EMPIRE achieves state-of-the-art forecasting accuracy,, with an MPJPE of 84.5384.53 m and a finger-relative error of 38.9738.97 m. We release the code and Dataset at https://github.com/wangwen-banban/EMPIRE. Introduction Forecasting future dexterous hand movements from egocentric observations is fundamental to intelligent interactive systems. It has broad applications in robot learning, virtual and augmented reality, and human–robot collaboration. With the growing availability of large-scale human videos, recent studies have explored their potential as scalable sources of semantic and physical knowledge for embodied manipulation learning (13). Unlike holistic human-motion prediction, egocentric hand-motion forecasting focuses on fine-grained bi-manual coordination and complex finger articulation during object manipulation. Such intricate dynamics make accurate long-horizon forecasting particularly challenging. Nevertheless, recent advances in large-scale egocentric datasets and vision–language foundation models have driven significant progress in this field (15; 16; 32; 1). Figure 1: EMPIRE at a glance. Stage I learns to plan; Stage I learns to act with the planner frozen. Hand–object motion generation has consequently shifted from text-conditioned synthesis (33; 14; 7; 10; 17) towards vision–language-guided forecasting (23; 27; 40), where egocentric observations provide the scene, object, and viewpoint cues which are unavaiable from language alone. These approaches mainly follow two paradigms. Diffusion-based methods such as VITRA generate continuous hand motion conditioned on a VLM representation, whereas autoregressive models such as Being-H0 jointly model multimodal observations and discretized future hand motion in a unified sequence. Despite their architectural differences, these methods share a common design: directly mapping VLM representations to future hand motion without explicitly modeling the intermediate manipulation process that drives hand-motion generation. While recent progress, existing methods remain fundamentally constrained by two overlooked design limitations. 1) Implicit supervision of manipulation planning. Future hand movements are generated through a sequence of manipulation decisions, including object interaction, hand coordination, and task progression. Accurate forecasting therefore requires understanding not only how the hand moves but also what manipulation process should occur next. Existing methods, however, supervise the final trajectory or motion-token sequence, leaving manipulation planning to be inferred implicitly during motion prediction. Without an explicit representation of the manipulation process, generated motion may remain locally plausible while deviating from the intended interaction progression, particularly in long-horizon tasks where early planning errors accumulate over time. 2) Coupled optimization of semantic learning and motion generation. Existing egocentric hand forecasting methods typically jointly optimize the VLM backbone and the motion generation model (6; 20; 5). Consequently, motion-generation gradients continually update the semantic representation used for conditioning, causing the generator to adapt to a dynamically changing feature space. Meanwhile, optimizing directly for spatial motion objectives may overwrite manipulation-aware structure acquired during VLM pretraining, weakening the representation required for scene understanding and interaction anticipation. Therefore, this coupled optimization may limit the ability of VLMs to provide stable semantic guidance and ultimately constrain forecasting performance. As summarized in Figure 1, we introduce EMPIRE, a two-stage method that first learns Explicit Manipulation Planning as an Intermediate Representation and then leverages it to forecast future hand motion from Egocentric-view. In Stage I: Learn to Plan, a VLM predicts an explicit, motion-oriented manipulation plan from an egocentric RGB observation, RGB-derived monocular depth, a coarse instruction, and camera field of view. Monocular depth provides complementary geometric context because general-purpose VLMs remain unreliable at metric distance and complex 3D spatial reasoning (8). Rather than predicting hand motion, the plan decomposes the anticipated interaction into temporally ordered steps for both hands, providing an interpretable intermediate representation. In Stage I: Learn to Act, the entire planner is frozen, and a flow-matching DiT (25) predicts future two-hand motion from the current hand state and the hidden representation of the predicted plan. Training on plans predicted by Stage I rather than ground-truth plans matches the condition available at inference and reduces the train–deployment gap. By preventing motion gradients from updating the planner, this decoupled design preserves manipulation-aware representations, provides stable semantic conditioning. For training, we construct EMPIRE-651K from EgoDex (16) by augmenting 650,910650,910 windows across 111111 manipulation tasks with explicit per-hand manipulation plans. Extensive experiments demonstrate that EMPIRE achieves accurate and efficient egocentric hand-motion forecasting, improving both global hand placement and fine-grained finger articulation. It consistently outperforms existing methods under a unified evaluation protocol, reducing MPJPE by 19.8% compared with VITRA while decreasing optimization time by 38.8%. Moreover, EMPIRE achieves comparable accuracy to the much larger Being-H0-14B model with 83.5× faster end-to-end inference. Related Work Vision–Language Models Vision–language models (VLMs), enabled by visual instruction tuning and large-scale multimodal pretraining, have demonstrated strong open-vocabulary scene understanding from visual and linguistic inputs (26; 32; 9; 2; 1). However, general-purpose VLMs remain limited in metric-scale perception and fine-grained 3D spatial reasoning, which are critical for spatial interaction (8; 37; 35). To bridge this gap, recent embodied systems leverage VLMs as high-level planners. For example, SayCan grounds language-model plans with environmental affordances (18), PaLM-E enables multimodal sequential planning for embodied tasks (11), RT-H predicts language-conditioned motion intentions before execution (4), and Embodied Chain-of-Thought introduces intermediate reasoning over plans, subtasks, motions, and visual grounding (38). These studies demonstrate the potential of VLMs for embodied planning. Hand Motion Generation and Forecasting Hand Motion Generators connect multimodal perception to hand actions. RT-2 and OpenVLA predict robot actions from visual-language inputs (6; 20), Octo learns a generalist policy across heterogeneous embodiments (28), and π0 _0 integrates a pretrained VLM with a flow-based action expert for action generation (5). XL-VLA further extends this paradigm by adopting a π0 _0-style flow-based action expert in a shared latent action space across heterogeneous dexterous robotic hands (19). For dexterous hand-motion generation, VITRA conditions a DiT-based motion generator on learned VLM representations (23), Being-H0 autoregressively predicts discretized motion tokens conditioned on multimodal observations (27), and MEgoHand combines VLM-derived motion priors, monocular depth, and a flow-matching policy for egocentric hand control (40). These advances are enabled by increasingly large-scale interaction datasets, including GRAB with whole-body grasping and object meshes (34), H2O and ARCTIC with hand–object manipulation sequences (22; 12), HOT3D with multi-view egocentric 3D hand-object tracking (3), and EgoDex with large-scale egocentric manipulation videos and tracked 3D hands (16). Despite these advances, existing hand-motion forecasting approaches typically generate future motions directly from implicit multimodal representations, without explicitly modeling the intermediate manipulation process that governs hand-object interactions. Method Figure 2: Overview of EMPIRE. Stage I (left): RGB and inferred monocular-depth features, together with the caption and camera FoV, form a multimodal prefix that conditions explicit motion-plan generation. Stage I (right): The entire Stage-I VLM is frozen, while the motion generator is trained with flow matching by conditioning on the current hand state and cross-attending to the hidden states HplanH_plan of Stage-I prediction. Bottom: During inference, the frozen VLM generates a plan with hidden states, which directly condition the DiT to generate future motion. As shown in Figure 2, EMPIRE follows a two-stage plan-then-act pipeline composed of a Planner and an Actor. The Planner generates an explicit manipulation plan from multimodal egocentric observations, and the Actor transforms the plan hidden states into future two-hand motion. Problem Formulation We formulate egocentric dexterous hand-motion forecasting as predicting future bimanual motion from multimodal egocentric observations. Let O=(I,c,f)O=(I,c,f) denote an egocentric RGB observation I, a coarse action caption c, and the camera field of view f. Given O and the current two-hand state s0s_0, the goal is to generate a future trajectory A=(a(1),…,a(T))∈ℝT×daA=(a^(1),…,a^(T)) ^T× d_a, where T is the prediction horizon and dad_a is the dimension of the two-hand pose at each step. The generated trajectory should remain continuous with the current state s0s_0 while consistent with the manipulation context described by O. Directly learning p⁡(A∣O,s0)p(A O,s_0) leaves the manipulation process that connects perception to motion implicitly. We instead introduce a structured motion plan P as an intermediate representation. The plan describes the anticipated interaction as temporally ordered, hand-specific sub-actions, while HPH_P denotes the hidden states of the Planner’s tokens after the Planner has conditioned on O. These hidden states serve as the interface between planning and motion generation, yielding the following two-stage decomposition: p⁡(A∣O,s0)=∑Ppθ(A∣HP,s0)pϕ(P∣O),p(A O,s_0)= _Pp_θ(A H_P,s_0)\,p_φ(P O), (1) where pϕ(P∣O)p_φ(P O) denotes the Planner learned in Stage I and pθ(A∣HP,s0)p_θ(A H_P,s_0) denotes the Actor learned in Stage I. During deployment, the latent plan variable is instantiated by the Planner prediction, whose hidden states are subsequently consumed by the Actor. Stage I: Learn to Plan Stage I instantiates the Planner pϕ(P∣O)p_φ(P O) with a PaliGemma-2 VLM (32). Given the observation O, we construct a multimodal prefix from RGB appearance (39), inferred monocular depth feature (24), caption, and horizontal and vertical FoV: Z=[Zrgb;Zdepth;Zcap;Zfov],Z=[Z_rgb;Z_depth;Z_cap;Z_fov], (2) in a fixed order. RGB tokens provide semantic appearance cues, while depth tokens expose complementary scene geometry. Caption and FoV tokens specify the task and camera configuration, respectively. The supervision target is a motion-oriented plan P=(P1,…,P|P|)P=(P_1,…,P_|P|) which decomposes the coarse action caption into temporally ordered, hand-specific sub-actions, such as reaching, stabilizing, grasping, lifting, and placing. We mask the multimodal prefix from the language loss and optimize autoregressive next-token prediction only over the plan: ℒplan=−∑i=1|P|logpϕ(Pi∣Z,P<i).L_plan=- _i=1^|P| p_φ(P_i Z,P_<i). (3) Here, pϕp_φ denotes the Planner’s next-token distribution. During training, the visual and depth encoders remain frozen, while the planner and lightweight modality adapters are optimized. Therefore, Stage I learns both an explicit manipulation plan and its corresponding hidden representation, which serves as the planning interface for the Actor. Stage I: Learn to Act Stage I instantiates the Actor pθ(A∣HP,s0)p_θ(A H_P,s_0) with a flow-matching diffusion transformer. Before Actor training, the Stage-I Planner generates a predicted plan P P for each training window which is cached offline as the planning condition. The ground-truth plan P is only used for Stage-I supervision. Conditioning Stage I on planner-generated plans rather than ground-truth plans eliminates the mismatch between training-time and deployment-time conditions (see the supplementary analysis of Stage I training with predicted versus ground-truth plans). We initialize the Planner with the Stage-I learned parameters and freeze it during Stage-I training. For each Stage-I example, the frozen Planner processes the multimodal prefix together with the cached predicted plan: H=[hi]i=1|Z|+|P^|=ℳϕ(Z,P^),H=[h_i]_i=1^|Z|+| P|=M_φ(Z, P), (4) where ℳϕM_φ denotes the VLM planner. We extract the hidden states corresponding to the generated plan span: Hplan=[hi]i=|Z||Z|+|P^|H_plan=[h_i]_i=|Z|^|Z|+| P| (5) as the manipulation representation. A lightweight projector ψG_ψ maps this variable-length span to the Actor conditioning space, while ℰstateE_state encodes the current hand state: Zstate=ℰstate(s0),Zplan=ψ(Hplan).Z_state=E_state(s_0), Z_plan=G_ψ(H_plan). (6) The DiT action expert treats noisy motion tokens as queries and cross-attends to ZplanZ_plan as keys and values; ZstateZ_state supplies the initial-pose condition. Thus, the Actor generates future motion conditioned on the Planner’s manipulation-aware representation rather than directly relying on an entangled image-caption feature. We train the DiT-based Actor (30) and its condition projector with flow matching (25). Given a ground-truth trajectory A⋆A , Gaussian noise ϵ∼⁡(0,)ε (0,I), and τ∼⁡(0,1)τ (0,1), we construct the linear interpolation Aτ=(1−τ)ϵ+τA⋆A_τ=(1-τ)ε+τ A (7) and optimize the Actor to predict the velocity field: ℒact=Eϵ,A⋆,τ[‖θ(Aτ,τ,Zstate,Zplan)−(A⋆−ϵ)‖22],L_act=E_ε,A ,τ [ \|V_θ(A_τ,τ,Z_state,Z_plan)-(A -ε) \|_2^2 ], (8) where θV_θ denotes the Actor. The Actor predicts the velocity field over the full two-hand trajectory instead of separately modeling the two hands. Freezing the Planner prevents the motion generation objective from altering the learned planning representation, while avoiding back-propagation through the Planner. Deployment: Plan Then Act At deployment, the frozen Stage-I Planner first predicts the structured motion plan P P online. We prefill the VLM with Z and retain the final-layer hidden state of each generated plan token, where the predefined plan delimiters identify the span used for extracting HplanH_plan. The Stage-I Actor then conditions on this cached span together with s0s_0. The explicit plan is therefore generated once, and provides a stable, manipulation-aware representation for motion generation. Figure 3: The EMPIRE-651K construction and plan-supervision pipeline. (A) Raw ARKit skeletons are converted into wrist-aligned MANO motion targets. (B) Multi-task EgoDex videos are split into task-specific episodes, which are then segmented and resampled into aligned forecasting windows. (C) Qwen2.5-VL first captions each video segment and then generates a caption-grounded motion plan from the segment and caption; Qwen3 labels caption–plan consistency, and inconsistent samples are discarded. Dataset: EMPIRE-651K To enable long-horizon egocentric bimanual motion forecasting with explicit manipulation plans, we construct EMPIRE-651K from the EgoDex dataset (16). EgoDex contains 829829 hours of egocentric Apple Vision Pro recordings at 3030 FPS across 194194 tabletop manipulation tasks, with synchronized RGB observations, camera calibration, language task descriptions, and two-hand ARKit skeleton annotations. Following the official split, we use the five training partitions for dataset construction and keep the test partition completely held out. After processing below, EMPIRE-651K contains 650,910650,910 five-second forecasting windows across 111111 manipulation tasks. The held-out test partition contains an additional 6,8366,836 windows covering the same task vocabulary. Each sample consists of a current egocentric RGB observation, the initial two-hand state, a task caption, an explicit manipulation plan, and the future two-hand trajectory. Figure 3 shows the dataset construction. (A) Skeleton-to-MANO conversion. EgoDex provides 2121 tracked 3D hand joints but does not include a parametric hand representation required for motion forecasting. We therefore convert the original skeleton annotations into MANO-based hand representations. Specifically, we transform finger joints into the wrist coordinate frame, resolve left-hand chirality inconsistencies, and perform sequence-level MANO fitting (31). The fitting optimizes a 1515-dimensional PCA pose representation with a neutral mean shape β=0β=0, guided by joint reconstruction and first-order temporal smoothness constraints. We set the weights for the reconstruction and smoothness terms to 100100 and 0,20,2, respectively, and perform optimization using Adam (21) for 500 iterations with a learning rate of 0.01. The resulting MANO finger articulations are combined with the original ARKit wrist SE(3)SE(3) transformations, preserving both global hand motion and fine-grained finger articulation to obtain temporally consistent two-hand motion targets. (B) Temporal window construction. EgoDex recordings may contain multiple manipulation episodes within a single video. Directly treating an entire recording as one sequence would introduce task transitions unrelated to the target action, and weaken the correspondence between observation, instruction, and future motion. We therefore first divide each recording into task-specific episodes and further partition each episode into non-overlapping five-second windows. Each window contains 150 frames at the original 30 FPS and is resampled to 12 FPS for motion forecasting. Under our forecasting setting, the model receives only the current egocentric observation and initial two-hand state, while predicting the following 60 frames of bimanual motion. Each example therefore requires forecasting a full five seconds of future motion, enabling the study of long-horizon manipulation evolution rather than short-term motion continuation. (C) Caption and motion-plan annotation. To supervise explicit manipulation planning, we construct task captions and manipulation plans through a two-stage VLM-assisted annotation pipeline. Given each five-second forecasting segment, the pipeline first identifies the overall manipulation intent and then decomposes it into temporally ordered, hand-specific sub-actions. i) Caption generation. For each five-second segment, Qwen2.5-VL-7B-Instruct (2) observes video frames sampled at 33 FPS and generates a concise caption describing the dominant manipulation intent and involved interactions. The caption provides coarse semantic guidance without requiring detailed descriptions of hand motion evolution. i) Manipulation plan generation. Conditioned on the video segment and generated caption, Qwen2.5-VL produces a structured manipulation plan that decomposes the interaction into temporally ordered, hand-specific sub-actions, such as reaching, grasping, stabilizing, lifting, and placing. The caption acts as a semantic constraint to maintain consistency with the observed task and reduce unsupported actions. i) Annotation filtering. To improve annotation reliability, Qwen3-30B-A3B-Instruct-2507 (36) evaluates the consistency between each caption and its corresponding manipulation plan using binary labels. Samples with inconsistent annotations are removed. The detailed prompts for caption generation, plan construction, and consistency evaluation are provided in the supplementary material under Data-Annotation Prompts. Human annotation audit. To evaluate annotation quality, we conduct a human audit over randomly sampled training windows. We select 20 samples from each EgoDex training partition, resulting in 100 manually inspected cases from distinct manipulation episodes. Each sample is evaluated according to six criteria, including caption grounding, hand attribution, plan grounding, plan coverage, temporal coordination, and usefulness for motion forecasting. All sampled videos are successfully interpretable. As shown in Table 1, the final annotation quality reaches 90.41/100, and 84.0% of caption-plan pairs are considered suitable for downstream learning. These results indicate that the generated captions and manipulation plans are sufficiently grounded in the observed interactions and provide reliable supervision for learning the planning interface between perception and motion generation. Detailed evaluation protocols and criterion definitions are provided in the supplementary material Metric Score Caption quality 4.574.57 Motion-plan quality 4.664.66 Overall quality 4.624.62 Usable pairs 84.0%84.0\% Table 1: Human audit of EMPIRE-651K annotations. Final results over 100100 stratified training windows. Quality scores use a 11–55 scale; A pair is usable when both its caption and plan means are at least 4/54/5. Method Params MPJPE ↓ Wrist MPJPE ↓ Finger- relative MPJPE ↓ Infer.(s/window) ↓ VITRA re-impl. (23) 3.23.2B 105.37105.37 91.4291.42 45.37¯ 45.37 0.40.4 Being-H0-1B (27) 1.21.2B 115.61115.61 100.23100.23 66.8766.87 15.615.6 Being-H0-8B (27) 8.28.2B 85.3985.39 68.61¯ 68.61 50.9550.95 28.328.3 Being-H0-14B (27) 15.415.4B 84.90¯ 84.90 66.5466.54 51.6551.65 83.583.5 EMPIRE (ours) 3.23.2B 84.5384.53 73.2573.25 38.9738.97 1.0¯ 1.0 Table 2: Main results on EMPIRE-651K. EMPIRE is compared with baselines on the same held-out test set. See the Evaluation subsection for protocol details and metric definitions. Experiment Training Data splits. The final EMPIRE model and our re-implemented VITRA baseline are trained on all five training partitions of EMPIRE-651K, comprising 650,910650,910 forecasting windows from 111111 manipulation tasks. The held-out test partition is excluded from all training runs. Implementation Details. Our backbone is PaliGemma-2-3B with a 182182M DiT-Base flow-matching generator. The model is trained following the two-stage framework. In Stage I, the planner model is fine-tuned for 11 epoch. In Stage I, the VLM is frozen and only the DiT action generator is optimized for 44 epochs. Training is performed with a total batch size of 6464 across 8 A40 GPUs, more details in the Supplementary material. Evaluation Protocol. All models are evaluated on the held-out test partition containing 6,836 five-second forecasting windows from 111 manipulation tasks. The same test set is used for models trained on all five partitions and for all Part 1 ablations. For Part 1 ablations, test windows are further categorized as seen or unseen based on task overlap with the training set. Specifically, 1,247 windows from the 26 training tasks are labeled as seen, while 5,589 windows from 85 unseen tasks are labeled as unseen. This split is only used for Part 1 ablations due to their limited training task coverage. And since the motion generator is stochastic, we sample K=8K=8 trajectories per input and report best-of-K performance following prior motion-generation works (23; 27). Each trajectory is generated with 44 Euler steps, using the same sampling protocol for all models. Metrics. All methods are evaluated using MPJPE and its variants. Errors are reported in millimeters, with lower values indicating better performance. Unless otherwise specified, global-coordinate metrics are computed in the absolute camera coordinate frame. i) MPJPE measures the average Euclidean distance over both hands, all valid prediction frames, and all MANO joints. Under the best-of-8 setting, we select the trajectory with the lowest MPJPE for each test sample and compute all other metrics based on the selected trajectory. i) Wrist MPJPE measures global hand placement accuracy by averaging the error of wrist joints over time. i) Finger-relative MPJPE removes wrist translation before computing joint errors to evaluate finger articulation independent of global hand motion. Main Results Forecasting accuracy. Table 2 compares EMPIRE with VITRA (23) and Being-H0 (27). EMPIRE achieves the best overall performance among the evaluated methods, reducing MPJPE by 19.8% compared with VITRA while using a substantially smaller model than the larger Being-H0 variants. Although the largest Being-H0 model achieves better wrist localization, EMPIRE obtains substantially more accurate finger articulation, demonstrating the advantage of explicit manipulation planning for fine-grained hand motion forecasting over long horizons. Figure 4 shows the same trend qualitatively: EMPIRE remains stable over the long horizon, while the other methods are relatively stable early but degrade at later steps. Additional cases are provided in the supplementary material under Qualitative Hand-Motion Forecasts. Training and inference efficiency. Beyond forecasting accuracy, EMPIRE also achieves improved efficiency. Training the complete two-stage framework requires 71 hours on 8 A40 GPUs, compared with 116 hours for the VITRA re-implementation, reducing training time by 38.8%. During inference, EMPIRE generates each forecasting window in 1.0 second, substantially faster than Being-H0 models. In particular, EMPIRE is 83.5×83.5× faster than Being-H0-14B in inference, while achieving comparable overall MPJPE and substantially better finger articulation accuracy. Detailed latency analysis is provided in the supplementary material under Inference Cost. When does motion planning help? To investigate when explicit planning is most beneficial, we analyze planning gains across different task difficulties. Table 3 groups test tasks according to their baseline MPJPE and shows that the benefit of planning increases with task complexity. The average MPJPE reduction grows from 4.6 m on easy tasks to 49.7 m on hard tasks, indicating that explicit plans are particularly effective for challenging manipulation scenarios. Overall, planning improves 89 out of 111 tasks, achieving an average MPJPE reduction of 22.3 m. This trend is further supported by the negative correlation between baseline MPJPE and planning gain (r=−0.567r=-0.567) (29), suggesting that tasks with larger initial errors benefit more from explicit planning. Furthermore, planning generalizes beyond the training task categories, improving 74 out of 85 unseen tasks and 15 out of 26 seen tasks. These results suggest that explicit manipulation plans provide structured temporal guidance, which is particularly valuable for long-horizon and compositional manipulation tasks. Difficulty (baseline MPJPE) #tasks Mean Δ Plan helps Easy (<100<100) 4343 −4.6-4.6 25/4325/43 Medium (100100–175175) 4949 −26.2-26.2 47/4947/49 Hard (≥175≥ 175) 1919 −49.7-49.7 17/1917/19 All 111111 −22.3-22.3 89/11189/111 Table 3: Planning benefit by task difficulty. Δ means that ours minus baseline; Negative means improvement. Figure 4: Qualitative comparison of motion predictions against baselines. Task: peel_place_sticker. Ablation and Discussion To ensure computationally efficient and controlled ablation studies, we conduct all ablations on 2626-task Part 1 subset. This setting enables evaluation of both in-distribution performance and generalization to the 85 tasks excluded from ablation training. Unless otherwise specified, each ablation modifies only the component indicated by its name, while keeping the input representation, motion targets, optimization schedule, and evaluation protocol fixed. Model components. Table 4 evaluates the contribution of explicit motion plans and monocular depth features. Starting from the VITRA re-implementation with caption-only conditioning, introducing the explicit motion plan reduces overall MPJPE from 124.52 m to 102.26 m. The improvement is particularly large on unseen tasks, where the error decreases from 129.73 m to 104.80 m, suggesting that the plan provides transferable temporal guidance beyond the semantic information contained in the caption alone. This result highlights the importance of explicitly modeling the intermediate manipulation process rather than directly mapping observations and instructions to future hand motions. We further evaluate different strategies for incorporating depth information. Direct depth concatenation achieves the best performance, reducing overall MPJPE to 95.42 m, while cross-attention over the same depth features does not improve over the plan-only model. This indicates that geometric cues are more effective when directly integrated into the multimodal representation. Interestingly, the depth-augmented model mainly benefits unseen tasks, suggesting a trade-off between cross-task generalization and fitting to the smaller ablation training subset. Configuration Overall ↓ Seen ↓ Unseen ↓ VITRA re-impl. (caption only) 124.52124.52 92.7192.71 129.73129.73 ++ motion plan 102.26102.26 90.8790.87 104.80104.80 ++ depth (cross-attention) 104.71104.71 110.10110.10 103.51103.51 ++ depth (concatenation) 95.4295.42 99.3399.33 94.5594.55 Table 4: Model-component ablation. All results in the table are reported using MPJPE. Preventing motion gradients from updating the VLM. Table 5 investigates whether isolating motion gradients is necessary beyond the two-stage training strategy. In the two-stage trainable setting, updating the VLM with motion gradients results in an MPJPE of 118.44 m. In contrast, freezing the VLM after Stage I and optimizing only the motion generator reduces MPJPE to 94.67 m, improving by 23.77 m (20.1%). This result indicates that freezing is not merely a constraint, but preserves stable semantic representations and provides a consistent interface for motion generation. Setting VLM grad.? LM loss MPJPE ↓ Single-stage Yes – 124.52124.52 Two-stage, trainable Yes 0.50.5 118.44118.44 Two-stage, frozen (ours) No 00 94.6794.67 Table 5: Motion-gradient isolation in Stage I. The plan loss coefficient is set to 0.5. DiT capacity. Table 6 studies the effect of actor capacity while fixing plan-span conditioning, concatenated depth features, and a frozen VLM. DiT-M consistently underperforms across all evaluation settings, indicating insufficient capacity to capture complex bimanual hand motion dynamics. Increasing the capacity to DiT-L improves performance on seen tasks but leads to degraded generalization on unseen tasks, suggesting that excessive model capacity may overfit the observed manipulation patterns under the available training data scale. In contrast, DiT-B achieves the best overall and unseen-task performance, providing a better balance between motion modeling capability and generalization. We therefore select DiT-B (182M parameters) as the default actor. Action head Params Overall ↓ Seen ↓ Unseen ↓ DiT-M 4646M 107.93107.93 114.15114.15 106.55106.55 DiT-B (default) 182182M 95.4295.42 99.3399.33 94.5594.55 DiT-L 637637M 100.08100.08 95.3795.37 101.13101.13 Table 6: DiT-capacity ablation. All results in the table are reported using MPJPE Conclusion We presented EMPIRE, a two-stage framework that introduces an explicit motion plan as an intermediate interface between vision-language understanding and long-horizon bimanual hand-motion forecasting. Stage I learns structured, manipulation-aware plans for hand-object interactions, while Stage I trains a flow-matching motion generator conditioned on the learned plans with the VLM frozen. This decoupled optimization prevents gradients from low-level motion generation objectives from disrupting the manipulation-aware representations learned during planning. To support this task, we introduced EMPIRE-651K, a large-scale benchmark constructed from EgoDex by converting skeleton annotations into temporally aligned MANO trajectories, coarse instructions, and motion-plan supervision, covering 650,910 training windows across 111 manipulation tasks. Extensive experiments demonstrate that EMPIRE achieves state-of-the-art forecasting accuracy. These results validate explicit manipulation planning combined with decoupled motion learning as an effective paradigm for accurate and efficient egocentric dexterous-motion forecasting. References Bai et al. (2025a) S. Bai, Y. Cai, R. Chen, et al. Qwen3-VL technical report. External Links: 2511.21631, Link Cited by: Introduction, Vision–Language Models. Bai et al. (2025b) S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. Qwen2.5-vl technical report. External Links: 2502.13923, Link Cited by: Vision–Language Models, (C) Caption and motion-plan annotation.. Banerjee et al. (2025) P. Banerjee, S. Shkodrani, P. Moulon, S. Hampali, S. Han, F. Zhang, L. Zhang, J. Fountain, E. Miller, S. Basol, et al. HOT3D: hand and object tracking in 3d from egocentric multi-view videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 7061–7071. External Links: Link Cited by: Hand Motion Generation and Forecasting. Belkhale et al. (2024) S. Belkhale, T. Ding, T. Xiao, P. Sermanet, Q. Vuong, J. Tompson, Y. Chebotar, D. Dwibedi, and D. Sadigh RT-H: action hierarchies using language. In Proceedings of Robotics: Science and Systems, External Links: Document, Link Cited by: Vision–Language Models. Black et al. (2025) K. Black, N. Brown, D. Driess, A. Esmail, M. R. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, L. Smith, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky π0 _0: a vision-language-action flow model for general robot control. In Proceedings of Robotics: Science and Systems, External Links: Document, Link Cited by: Introduction, Hand Motion Generation and Forecasting. Brohan et al. (2023) A. Brohan, N. Brown, J. Carbajal, et al. RT-2: vision-language-action models transfer web knowledge to robotic control. In Proceedings of The 7th Conference on Robot Learning, Proceedings of Machine Learning Research, Vol. 229, p. 2165–2183. External Links: Link Cited by: Introduction, Hand Motion Generation and Forecasting. Cha et al. (2024) J. Cha, J. Kim, J. S. Yoon, and S. Baek Text2hoi: text-guided 3d motion generation for hand-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 1577–1585. Cited by: Introduction. Chen et al. (2024a) B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Sadigh, L. Guibas, and F. Xia SpatialVLM: endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 14455–14465. Cited by: Introduction, Vision–Language Models. Chen et al. (2024b) Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. Intern vl: scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 24185–24198. Cited by: Vision–Language Models. Christen et al. (2024) S. Christen, S. Hampali, F. Sener, E. Remelli, T. Hodan, E. Sauser, S. Ma, and B. Tekin DiffH2O: diffusion-based synthesis of hand-object interactions from textual descriptions. In SIGGRAPH Asia Conference Papers, Cited by: Introduction. Driess et al. (2023) D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. PaLM-E: an embodied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, Vol. 202, p. 8469–8488. External Links: Link Cited by: Vision–Language Models. Fan et al. (2023) Z. Fan, O. Taheri, D. Tzionas, M. Kocabas, M. Kaufmann, M. J. Black, and O. Hilliges ARCTIC: a dataset for dexterous bimanual hand-object manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 12943–12954. External Links: Link Cited by: Hand Motion Generation and Forecasting. Feng et al. (2026) Z. Feng, Q. Li, H. Liang, R. Yang, Y. Shen, Z. Du, Z. Zhang, Y. Deng, L. Zhao, H. Zhao, Z. Lu, O. Mees, M. Pollefeys, J. Yang, and B. Guo From human videos to robot manipulation: a survey on scalable vision-language-action learning with human-centric data. External Links: 2606.00054, Link Cited by: Introduction. Ghosh et al. (2023) A. Ghosh, R. Dabral, V. Golyanik, C. Theobalt, and P. Slusallek IMoS: intent-driven full-body motion synthesis for human-object interactions. 42 (2). Cited by: Introduction. Grauman et al. (2022) K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, et al. Ego4D: around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 18995–19012. External Links: Link Cited by: Introduction. Hoque et al. (2026) R. Hoque, P. Huang, D. J. Yoon, M. Sivapurapu, and J. Zhang EgoDex: learning dexterous manipulation from large-scale egocentric video. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: Introduction, Introduction, Hand Motion Generation and Forecasting, Dataset: EMPIRE-651K. Huang et al. (2025) M. Huang, F. Chu, B. Tekin, K. J. Liang, H. Ma, W. Wang, X. Chen, P. Gleize, H. Xue, S. Lyu, K. Kitani, M. Feiszli, and H. Tang HOIGPT: learning long-sequence hand-object interaction with language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 7136–7146. Cited by: Introduction. Ichter et al. (2023) B. Ichter, A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, et al. Do as i can, not as i say: grounding language in robotic affordances. In Proceedings of the 6th Conference on Robot Learning, Proceedings of Machine Learning Research, Vol. 205, p. 287–318. External Links: Link Cited by: Vision–Language Models. Jiang et al. (2026) G. Jiang, Y. Liang, J. Ye, J. Huang, C. Jing, R. Duan, P. Abbeel, X. Wang, and X. Zou Cross-hand latent representation for vision-language-action models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 13496–13507. Cited by: Hand Motion Generation and Forecasting. Kim et al. (2025) M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. P. Foster, P. R. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn OpenVLA: an open-source vision-language-action model. In Proceedings of The 8th Conference on Robot Learning, Proceedings of Machine Learning Research, Vol. 270, p. 2679–2713. External Links: Link Cited by: Introduction, Hand Motion Generation and Forecasting. Kingma and Ba (2015) D. P. Kingma and J. Ba Adam: a method for stochastic optimization. In International Conference on Learning Representations (ICLR), Cited by: (A) Skeleton-to-MANO conversion.. Kwon et al. (2021) T. Kwon, B. Tekin, J. Stühmer, F. Bogo, and M. Pollefeys H2O: two hands manipulating objects for first person interaction recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 10138–10148. External Links: Link Cited by: Hand Motion Generation and Forecasting. Li et al. (2025) Q. Li, Y. Deng, Y. Liang, L. Luo, L. Zhou, C. Yao, L. Zeng, Z. Feng, H. Liang, S. Xu, Y. Zhang, X. Chen, H. Chen, L. Sun, D. Chen, J. Yang, and B. Guo Scalable vision-language-action model pretraining for robotic manipulation with real-life human activity videos. External Links: 2510.21571, Link Cited by: Appendix C, Appendix D, Introduction, Hand Motion Generation and Forecasting, Table 2, Evaluation Protocol., Forecasting accuracy.. Lin et al. (2026) H. Lin, S. Chen, J. H. Liew, D. Y. Chen, Z. Li, Y. Zhao, S. Peng, H. Guo, X. Zhou, G. Shi, J. Feng, and B. Kang Depth anything 3: recovering the visual space from any views. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: Stage I: Learn to Plan. Lipman et al. (2023) Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le Flow matching for generative modeling. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: Introduction, Stage I: Learn to Act. Liu et al. (2023) H. Liu, C. Li, Q. Wu, and Y. J. Lee Visual instruction tuning. In Advances in Neural Information Processing Systems, Vol. 36, p. 34892–34916. External Links: Document, Link Cited by: Vision–Language Models. Luo et al. (2025) H. Luo, Y. Feng, W. Zhang, S. Zheng, Y. Wang, H. Yuan, J. Liu, C. Xu, Q. Jin, and Z. Lu Being-h0: vision-language-action pretraining from large-scale human videos. External Links: 2507.15597, Link Cited by: Appendix C, Appendix D, Introduction, Hand Motion Generation and Forecasting, Table 2, Table 2, Table 2, Evaluation Protocol., Forecasting accuracy.. Octo Model Team et al. (2024) Octo Model Team, D. Ghosh, H. R. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, J. Luo, et al. Octo: an open-source generalist robot policy. In Proceedings of Robotics: Science and Systems, External Links: Document, Link Cited by: Hand Motion Generation and Forecasting. Pearson (1895) K. Pearson VII. note on regression and inheritance in the case of two parents. Proceedings of the Royal Society of London 58, p. 240–242. External Links: Document, Link Cited by: When does motion planning help?. Peebles and Xie (2023) W. Peebles and S. Xie Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 4195–4205. External Links: Link Cited by: Stage I: Learn to Act. Romero et al. (2017) J. Romero, D. Tzionas, and M. J. Black Embodied hands: modeling and capturing hands and bodies together. ACM Transactions on Graphics (ToG) 36 (6). Cited by: (A) Skeleton-to-MANO conversion.. Steiner et al. (2024) A. Steiner, A. Susano Pinto, M. Tschannen, D. Keysers, X. Wang, Y. Bitton, A. Gritsenko, M. Minderer, A. Sherbondy, S. Long, S. Qin, R. Ingle, E. Bugliarello, S. Kazemzadeh, T. Mesnard, I. Alabdulmohsin, L. Beyer, and X. Zhai PaliGemma 2: a family of versatile vlms for transfer. External Links: 2412.03555, Link Cited by: Introduction, Vision–Language Models, Stage I: Learn to Plan. Taheri et al. (2022) O. Taheri, V. Choutas, M. J. Black, and D. Tzionas GOAL: generating 4d whole-body motion for hand-object grasping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 13253–13263. Cited by: Introduction. Taheri et al. (2020) O. Taheri, N. Ghorbani, M. J. Black, and D. Tzionas GRAB: a dataset of whole-body human grasping of objects. In European Conference on Computer Vision (ECCV), Cited by: Hand Motion Generation and Forecasting. Wang et al. (2025) X. Wang, W. Ma, T. Zhang, C. M. de Melo, J. Chen, and A. Yuille Spatial457: a diagnostic benchmark for 6D spatial reasoning of large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 24669–24679. Cited by: Vision–Language Models. Yang et al. (2025a) A. Yang, A. Li, B. Yang, et al. Qwen3 technical report. External Links: 2505.09388, Link Cited by: (C) Caption and motion-plan annotation.. Yang et al. (2025b) J. Yang, S. Yang, A. W. Gupta, R. Han, L. Fei-Fei, and S. Xie Thinking in space: how multimodal large language models see, remember, and recall spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 10632–10643. Cited by: Vision–Language Models. Zawalski et al. (2025) M. Zawalski, W. Chen, K. Pertsch, O. Mees, C. Finn, and S. Levine Robotic control via embodied chain-of-thought reasoning. In Proceedings of The 8th Conference on Robot Learning, Proceedings of Machine Learning Research, Vol. 270, p. 3157–3181. External Links: Link Cited by: Vision–Language Models. Zhai et al. (2023) X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 11975–11986. External Links: Link Cited by: Stage I: Learn to Plan. Zhou et al. (2026) B. Zhou, Y. Zhan, Z. Zhang, and Z. Lu Megohand: multimodal egocentric hand-object interaction motion generation. Vol. 38, p. 49464–49490. Cited by: Introduction, Hand Motion Generation and Forecasting. Appendix A Data-Annotation Prompts We reproduce the three prompts used to construct and filter the caption–plan annotations in EMPIRE-651K. Specifically, Qwen2.5-VL first generates a coarse caption from the five-second egocentric video. The video and generated caption are then jointly provided to the model in a second pass to derive the corresponding motion plan. Finally, Qwen3 evaluates each caption–plan pair to assess semantic consistency and filters out inconsistent annotations. Caption generation (user prompt). Describe hand actions in this video sequence. Focus on what left hand and right hand are doing. Use very simple sentences. Output format must be exactly: ‘Left hand: [action description]. Right hand: [action description or None].’ For example: ‘Left hand: Pick up the cup. Right hand: None.’ ‘Left hand: Pour water into the glass. Right hand: Hold the glass.’ ‘Left hand: None. Right hand: Close the door.’ What are the hands doing? Caption-grounded motion-plan construction (user prompt). You are given an egocentric hand-object manipulation video and a coarse caption. Caption: caption Generate one fine-grained action decomposition of the hand actions in this clip. The decomposition must be consistent with both the video and the caption. Output exactly one XML-like tag: <cot>1. [first visible hand-action step] 2. [next hand-action step] 3. [next or final hand-action step]</cot> Rules: -- Use a numbered list inside the <cot> tag: 1., 2., 3. and optionally 4. or 5. -- Each numbered step should describe one clear sub-action in temporal order. -- Mention left hand and right hand when they are visible or active. -- Use temporal words such as first, then, while, next, finally when helpful. -- Describe low-level hand motion, contact, grasp, release, lift, place, open, close, rotate, or stabilize events when visible. -- If a hand is inactive, missing, or captioned as None, state that briefly. -- Do not invent objects or actions that contradict the caption. -- Keep the whole content 40 to 120 words. -- Do not use markdown bullets, headings, or text outside <cot>. Caption–plan consistency labeling (user prompt). Determine whether the generated caption and motion plan describe the same hand-object manipulation task. Caption: caption Motion plan: motion plan Label the pair CONSISTENT when the plan preserves the caption’s task, object, and hand roles while providing compatible fine-grained steps. Label it INCONSISTENT when the plan changes the task or object, conflicts with the stated hand roles, or introduces an incompatible goal. Output exactly one label: CONSISTENT or INCONSISTENT. Samples labeled INCONSISTENT are removed before constructing the final aligned supervision tuples. Appendix B Human Audit of Annotation Quality Sampling and review. The audit evaluates the quality of the training annotations. Using random seed 20260725, we randomly sample 2020 windows from each of the five training partitions. The resulting 100100 cases are drawn from 100100 different episodes, preventing the evaluation from being biased toward adjacent windows within the same episode. Each case follows the same temporal configuration as training, consisting of a 6060-frame, 1212 FPS, five-second egocentric video window. A human reviewer jointly examines the source video, caption, and ground-truth motion plan, while predicted plans are withheld during evaluation. All sampled videos contain sufficient visual evidence for reliable assessment. Target Criterion Definition Score Caption Grounding Described actions and objects are consistent with the visual evidence in the video. 4.38/54.38/5 Caption Hand attribution Left/right roles, hand activities, and None assignments are correctly identified. 4.55/54.55/5 Plan Grounding Described actions, objects, and contact states are supported by the visual evidence. 4.69/54.69/5 Plan Coverage Key sub-actions and all visibly active hands are covered. 4.68/54.68/5 Plan Temporal coordination Step order, simultaneous motion, and bimanual coordination are correct. 4.60/54.60/5 Plan Conditioning utility The plan is specific, non-redundant, complete, and useful for motion generation. 4.35/54.35/5 Table 7: Human-audit criteria and final scores. Criteria and quality measurement. Each criterion is rated on a scale from 11 (highly inconsistent with the video) to 55 (fully consistent and directly usable). A score of 44 indicates that the main semantics are correctly captured with only minor issues, whereas a score of 33 indicates an evident discrepancy requiring correction. Table 7 summarizes the six human-audit criteria and their corresponding average scores. Let C denote the average score of the two caption-related criteria and P denote the average score of the four plan-related criteria. We compute the overall audit quality as: Qaudit=C+P2.Q_audit= C+P2. An annotation pair is considered usable if both C≥4C≥ 4 and P≥4P≥ 4. Table 8 shows that the overall audit quality reaches 4.62/54.62/5, with a usable-pair rate of 84.0%84.0\%. The quality remains consistently high across all five training partitions, with overall scores ranging from 4.464.46 to 4.764.76, indicating that the audit results are not dominated by any single subset. The structured error analysis further shows that the remaining annotation issues are primarily localized to hand attribution and plan details. The most frequent error categories correspond to caption hand/None attribution errors (9/1009/100 cases), plan hand-role assignment errors (8/1008/100 cases), and hallucinated plan steps (6/1006/100 cases). No cases are identified as source-video ambiguous, containing unsupported caption details, or involving coarse, repetitive, or truncated motion plans. Partition Caption Plan Overall Usable (%\%) Part 1 4.454.45 4.514.51 4.484.48 75.0%75.0\% Part 2 4.634.63 4.724.72 4.684.68 85.0%85.0\% Part 3 4.804.80 4.714.71 4.764.76 95.0%95.0\% Part 4 4.354.35 4.564.56 4.464.46 75.0%75.0\% Part 5 4.634.63 4.804.80 4.714.71 90.0%90.0\% Overall 4.574.57 4.664.66 4.624.62 84.0%84.0\% Table 8: Human-audit quality scores across training partitions. Each partition contains 20 cases sampled from distinct episodes. The overall row reports the aggregate results over all 100 audited cases. Qualitative examples. Figure 5 visualizes three cases sampled from distinct training partitions. The ordered ground-truth frames demonstrate that the captions correctly identify the visible hand roles and manipulation task, while the corresponding motion plan decomposes the same interaction into temporally ordered and semantically consistent steps. (a) Folding a cloth (Part 2; audit case #11). Caption. Left hand: Place the folded cloth on the red surface. Right hand: Fold the cloth in half vertically. Plan. 1. The person places a folded piece of cloth on a red surface. 2. They fold the cloth in half vertically with both hands. 3. They then fold the cloth in half horizontally, completing the folding process. (b) Scooping ice into a cup (Part 4; audit case #43). Caption. Left hand: Pick up the yellow cup. Right hand: Scoop ice cubes into the cup. Plan. 1. The person reaches for a yellow cup with their left hand. 2. They pick up a purple scoop with their right hand. 3. The person scoops ice cubes from a container into the yellow cup. 4. After filling the cup, they place the scoop back into the container. 5. The person adjusts the position of the cup on the table. (c) Connecting an AirPods charging cable (Part 1; audit case #62). Caption. Left hand: Insert the USB cable into the white device. Right hand: Hold the white device steady. Plan. 1. The left hand picks up the USB cable from the bed. 2. The left hand inserts the USB cable into the white device held by the right hand. 3. The right hand holds the white device steady while the left hand completes the insertion of the USB cable. Figure 5: Some annotation examples from the human audit. Each row shows four uniformly spaced ground-truth frames from an audited five-second window, followed by its caption and temporally ordered motion-plan steps. Configuration Stage I: Learn to Plan Stage I: Learn to Act Trainable modules Gemma-2 decoder, multimodal projector, depth mapper, and FoV encoder; SigLIP and DA3 encoders are frozen DiT-B and conditioning projectors; the Planner and all condition encoders are frozen Training objective Autoregressive plan-language cross-entropy Flow matching with t∼Beta⁡(1.5,1.0)t\! \!Beta(1.5,1.0) on [0,1][0,1] Training length 11 epoch (10,17110,171 optimizer steps) 44 epochs (40,68440,684 optimizer steps) Optimizer AdamW, (β1,β2)=(0.9,0.999)( _1, _2)=(0.9,0.999), weight decay 0.10.1 Same as Stage I Learning rate 1×10−51×10^-5 1×10−41×10^-4 Schedule / clipping Constant learning rate, no warmup; gradient-norm clipping at 1.01.0 Same as Stage I Batch size Global batch 6464; per-device batch 22; 44 gradient-accumulation steps Global batch 6464; per-device batch 44; 22 gradient-accumulation steps Compute 8×8×NVIDIA A40; bfloat16; FSDP full sharding with activation checkpointing Same as Stage I Table 9: Training details. Stage-specific settings from the final Stage I and Stage I checkpoint configurations. Appendix C Training and Evaluation Evaluation Protocol All models are evaluated on the same EgoDex test set, which contains 6,8366,836 five-second forecasting windows covering all 111111 tasks. Given each observation window, the model predicts 6060 future frames at 1212 FPS for both hands. For each test window, we generate K=8K=8 stochastic predictions and recover the corresponding joint trajectories for evaluation. We report the best-of-8 performance by selecting the prediction with the lowest overall MPJPE. The MPJPE is computed as the mean Euclidean distance over valid frames and joints across both hands. Wrist error is measured using joint 00, while finger-relative error is computed after subtracting the wrist position from all joints before comparison. All errors are reported in millimeters in the absolute camera coordinate frame and averaged across all test windows. Baseline Implementations VITRA re-implementation. We adopt a re-implementation of VITRA (23) as the no-plan baseline. Since the released VITRA model is trained under a different data configuration and predicts short action chunks rather than five-second future trajectories, we re-train the model under our experimental setting for a fair comparison. Our implementation follows the same architecture and training configuration as our Stage I model, including the PaliGemma-2-3B backbone, DiT action expert, EMPIRE-651K training data, four-epoch schedule, and flow-matching objective. This baseline is trained in a single stage directly from coarse captions, without explicit motion plans or monocular depth inputs, and jointly optimizes the VLM and DiT modules. Being-H0 adaptation. We evaluate the released Being-H0 models (1B, 8B, 14B) (27) using their official inference protocol. Motion generation follows the original sampling procedure with fixed block-length and block-count control. The input consists of the current EgoDex RGB frame, warped from the original camera intrinsics to Being-H0’s canonical camera space, and the same caption instruction used by our models. Since Being-H0 does not use the current hand state as input, no pose initialization is provided. We generate 55 seconds of motion generation, (75 frames at 15 FPS). The generated wrist and finger tokens are decoded into camera-frame MANO trajectories using the GRVQ-8K tokenizer, and linearly resampled to our 60-frame, 12 FPS evaluation protocol. A same-pose alignment test verifies the consistency between Being-H0 and our MANO joint conventions, with an average residual of approximately 5 m caused by different fingertip definitions. To compensate for the lack of initial hand state, we additionally evaluate a wrist-anchored variant by aligning the first-frame wrist position with the ground truth. This improves Being-H0-1B from 115.61 m to 96.55 m, Being-H0-8B from 85.39 m to 81.22 m, and Being-H0-14B from 84.90 m to 81.21 m. While larger Being-H0 models benefit from improved global wrist localization, our lower finger-relative error demonstrates the advantage of explicit per-hand plans for fine-grained long-horizon dexterous motion forecasting. Run Trainable Epoch Steps s/step Hours Baseline (single stage) VLM ++ DiT 44 40,68440,684 10.2610.26 116116 Ours Stage-I (plan SFT) VLM 11 10,17110,171 12.1412.14 3434 Ours Stage-I (DiT) DiT 44 40,68440,684 3.233.23 3737 Ours total — 55 50,85550,855 — 71 Table 10: Gradient-optimization cost on EMPIRE-651K. Training Cost The proposed two-stage training strategy reduces optimization cost compared with the single-stage baseline. Table 10 reports the gradient-optimization time over all five EMPIRE-651K partitions. All runs use the same hardware setup (8 A40 GPUs) and batch size (64 windows per optimizer step). The offline generation of Stage-I predicted plans is treated as preprocessing and excluded. The single-stage baseline jointly fine-tunes the 3B VLM and DiT for four epochs, requiring 40,684 steps at 10.26 s/step and 116 hours in total. In contrast, our method separates plan learning and motion generation. Stage-I updates only the 3B VLM for one epoch (10,171 steps, 34 hours), while Stage-I freezes the VLM and trains the 182M DiT for four epochs (40,684 steps, 37 hours). As a result, our complete training pipeline requires 71 hours, reducing the optimization cost by 39% compared with the baseline, despite using one additional training epoch. The efficiency gain comes from decoupling semantic learning from motion generation: the expensive VLM backbone is optimized only during plan learning, while motion synthesis training updates only the lightweight DiT. Inference Cost Table 11 reports the end-to-end wall-clock time per test window under the same protocol used for the main-paper results. For each window, a single GPU generates the complete best-of-88 prediction, including preprocessing, model inference, and trajectory decoding. As shown in Table 11, our model requires only 1.01.0 s per window, substantially faster than Being-H0 variants (15.615.6–83.583.5 s). The efficiency advantage comes from the different generation paradigms. Our model generates a compact manipulation plan with at most 9696 autoregressive tokens, whose hidden states are reused to condition all 88 DiT samples. The VLM is therefore executed only once, while motion synthesis is performed through horizon-parallel flow matching with 44 Euler steps. In contrast, Being-H0 directly generates motion tokens autoregressively, requiring approximately 10,40010,400 sequential token generations under the best-of-88 setting. Since each token depends on previous outputs, its inference cost scales with both the forecasting horizon and the number of samples. The caption-only baseline requires only 0.40.4 s per window because it entirely removes autoregressive plan generation. Compared with this baseline, our additional cost is limited to generating a short manipulation plan, while avoiding the expensive autoregressive motion synthesis. These results demonstrate that explicit planning provides an effective balance between semantic guidance and inference efficiency. Model AR tokens s / window Baseline (no plan) 00 0.40.4 EMPIRE (plan ++ depth) ≤96≤96 1.01.0 Being-H0-1B ∼10,400 10,400 15.615.6 Being-H0-8B ∼10,400 10,400 28.328.3 Being-H0-14B ∼10,400 10,400 83.583.5 Table 11: End-to-end inference cost. Appendix D Ablation Analysis Stage I Training on Predicted Motion Plans During deployment, Stage I is conditioned on motion plans generated by the frozen Stage I planner. Training the motion generator with ground-truth plans would introduce a train–test mismatch by providing cleaner conditioning signals than those available during inference. Table 12 compares training Stage I with ground-truth plans and with plans predicted by Stage I. For predicted-plan training, Stage I generates motion plans for training windows using the same inference setting, and Stage I is trained with these predicted plans as conditioning inputs. Since Stage I remains frozen, this only changes the conditioning data without introducing additional optimization cost. Table 12 shows that predicted-plan training consistently improves performance. On the Part 1 2626-task subset, MPJPE decreases from 94.6794.67 to 93.4093.40 m. When trained on all 111111 tasks, the improvement increases from 90.2290.22 to 84.5384.53 m. These results demonstrate that predicted-plan training better matches the inference-time conditioning distribution, and we therefore adopt Stage-I predicted plans for Stage I in the final model. Stage-I plan condition 26-task subset Train Part 1–5 Ground-truth plan 94.6794.67 90.2290.22 Stage-I predicted plan (ours) 93.4093.40 84.5384.53 Table 12: Stage I training with predicted versus ground-truth plans. Detailed Analysis of When Motion Planning Helps? Table 3 of the main paper compares the deployable model with self-generated-plan against the no-plan baseline on the EgoDex test set. As shown, explicit motion planning improves forecasting performance on average, but the benefit is not uniform across tasks. Let Δ denote the per-task MPJPE difference between the planning model and the baseline (negative values indicate improvement). As shown in Table 3 of the main paper, the self-generated plan improves performance on 89/11189/111 tasks (80%80\%), achieving a sample-weighted average gain of 22.322.3 m and a median per-task improvement of 14.314.3 m. Moreover, the benefit increases with task difficulty: baseline MPJPE and Δ exhibit a Pearson correlation of −0.567-0.567, indicating larger improvements for more challenging tasks. Consistent with the difficulty analysis in the main paper, planning provides limited gains on easy tasks (Δ=−4.6 =-4.6 m, improving 25/4325/43 tasks), but substantially larger improvements on medium (−26.2-26.2 m, 47/4947/49 tasks) and hard tasks (−49.7-49.7 m, 17/1917/19 tasks). Planning also provides stronger benefits for unseen tasks. Compared with the Part 1 training setting, the self-generated plan improves 15/2615/26 seen tasks (58%58\%, weighted Δ=−10.9 =-10.9 m) and 74/8574/85 unseen tasks (87%87\%, weighted Δ=−24.8 =-24.8 m), yielding more than twice the improvement on unseen scenarios. The largest gains are observed on complex, long-horizon manipulation tasks, including wash_fruit (343→154343→154 m), stock_unstock_fridge (303→185303→185 m), wash_put_away_dishes (234→126234→126 m), and flip_pages (164→58164→58 m). Performance degradation is relatively rare and is primarily observed in two scenarios: simple tasks with low baseline errors, where additional planning constraints may limit motion flexibility, and highly challenging tasks where the predicted plans may still contain inaccuracies. Qualitative Hand-Motion Forecasts Figures 6 and 7 present qualitative comparisons of future hand-motion forecasts on two representative tasks. All predictions are rendered on the same five future RGB frames sampled from the 6060-frame, 1212 fps forecasting horizon. The ground-truth row uses the corresponding EgoDex MANO annotations, and all predictions are transformed into the corresponding future camera views before rendering. Blue and red denote the left and right hands, respectively. As shown in the figures, EMPIRE better captures the temporal evolution of bimanual manipulation compared with existing baselines. In particular, it preserves more accurate hand-object interactions and finger articulations over long horizons, while VITRA and Being-H0 exhibit larger deviations in hand placement and coordination. These results demonstrate that explicit manipulation plans provide effective guidance for long-horizon hand-motion forecasting. Figure 6: Qualitative comparison of motion predictions against baselines. Task: insert_remove_bookshelf. Figure 7: Qualitative comparison of motion predictions against baselines. Task: dry_hands. 2327

Instant research discovery