Paper deep dive
From Where to How: Continuous 4D Interaction Forecasting from Egocentric Video
Qiaohui Chu, Haoyu Zhang, Meng Liu, Haoxiang Shi, Dongmei Jiang, Liqiang Nie
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/14/2026, 5:28:49 AM
Summary
The paper introduces Coherent4D, a large-scale egocentric dataset for continuous 4D interaction forecasting, and HIGFlow, a framework that predicts future 3D interaction locations and full-body poses jointly. HIGFlow uses a cascaded approach where semantic and visual dynamics predict interaction locations, which then condition a residual flow matching model for diverse and structurally consistent full-body motion.
Entities (7)
Relation Signals (6)
Coherent4D → derivedfrom → Ego-Exo4D
confidence 95% · Coherent4D is constructed from Ego-Exo4D [53] to support continuous 4D interaction forecasting from egocentric video.
Semantic-Dynamic Location Forecasting → partof → HIGFlow
confidence 95% · Building on this formulation, we propose HIGFlow... For location forecasting, we develop Semantic-Dynamic Location Forecasting
Hand-Conditioned Residual Flow Matching → partof → HIGFlow
confidence 95% · Building on this formulation, we propose HIGFlow... For full-body pose forecasting, we introduce Hand-Conditioned Residual Flow Matching
HIGFlow → uses → Semantic-Dynamic Location Forecasting
confidence 95% · HIGFlow first forecasts continuous future interaction locations by combining semantic grounding with short-horizon visual dynamics... For location forecasting, we develop Semantic-Dynamic Location Forecasting
HIGFlow → uses → Hand-Conditioned Residual Flow Matching
confidence 95% · HIGFlow... uses the predicted location sequence to condition a deterministic motion anchor and residual Flow Matching... For full-body pose forecasting, we introduce Hand-Conditioned Residual Flow Matching
Coherent4D → uses → SMPL
confidence 90% · Coherent4D fills this gap by pairing ordered 3D interaction locations with synchronized full-body poses parameterized by the Skinned Multi-Person Linear (SMPL) [25] model
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Egocentric 4D interaction forecasting aims to anticipate both where future interactions will occur in 3D and how the human body will move to realize them, providing an important capability for assistive robotics and human-computer interaction. Existing methods struggle to translate semantic understanding into precise continuous 3D localization and to balance motion diversity with structural consistency in pose forecasting. More fundamentally, these tasks are often modeled separately, leaving the continuous geometric and temporal correspondence between interaction locations and body motion insufficiently captured. To address these challenges, we introduce Coherent4D, a large-scale egocentric dataset for continuous 4D interaction forecasting, comprising approximately 233K samples across three domains. Each sample pairs a sequence of future 3D interaction locations with corresponding full-body poses, aligned in time and expressed in a shared coordinate system. We also provide evaluation metrics in continuous space. Building on this formulation, we propose HIGFlow, a Hand Interaction Guided Residual Flow framework that models forecasting as a cascaded where-to-how process. HIGFlow first forecasts continuous future interaction locations by combining semantic grounding with short-horizon visual dynamics, and then uses the predicted location sequence to condition a deterministic motion anchor and residual Flow Matching for diverse yet structurally consistent full-body motion forecasting. Extensive experiments across all three domains demonstrate consistent improvements over representative baselines on both location and pose forecasting, while ablations validate the contributions of the proposed components. The project page is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.08636v1
- Canonical: https://arxiv.org/abs/2609.08636v1
Trouble viewing inline? Open PDF directly →
Full Text
93,858 characters extracted from source content.
Expand or collapse full text
From Where to How: Continuous 4D Interaction Forecasting from Egocentric VideoThanks: Qiaohui Chu, Haoyu Zhang and Haoxiang Shi are with the School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen 518055, China, and Pengcheng Laboratory, Shenzhen 518000, China (e-mail: qiaohuichu8599@gmail.com; zhang.hy.2019@gmail.com; shihaoxiang1999@gmail.com).Thanks: Meng Liu is with the School of Software, Shandong University, Jinan, 250101, China (e-mail: mengliu.sdu@gmail.com).Thanks: Dongmei Jiang is with Pengcheng Laboratory, Shenzhen 518000, China (e-mail: jiangdm@pcl.ac.cn).Thanks: Liqiang Nie is with the School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen 518055, China (e-mail: nieliqiang@gmail.com).Thanks: Corresponding authors: Meng Liu and Liqiang Nie. Qiaohui Chu Haoyu Zhang Meng Liu Haoxiang Shi Affiliation: Dongmei Jiang, and Liqiang Nie, Abstract Egocentric 4D interaction forecasting aims to anticipate both where future interactions will occur in 3D and how the human body will move to realize them, providing an important capability for assistive robotics and human-computer interaction. Existing methods struggle to translate semantic understanding into precise continuous 3D localization and to balance motion diversity with structural consistency in pose forecasting. More fundamentally, these tasks are often modeled separately, leaving the continuous geometric and temporal correspondence between interaction locations and body motion insufficiently captured. To address these challenges, we introduce Coherent4D, a large-scale egocentric dataset for continuous 4D interaction forecasting, comprising approximately 233K samples across three domains. Each sample pairs a sequence of future 3D interaction locations with corresponding full-body poses, aligned in time and expressed in a shared coordinate system. We also provide evaluation metrics in continuous space. Building on this formulation, we propose HIGFlow, a Hand Interaction Guided Residual Flow framework that models forecasting as a cascaded where-to-how process. HIGFlow first forecasts continuous future interaction locations by combining semantic grounding with short-horizon visual dynamics, and then uses the predicted location sequence to condition a deterministic motion anchor and residual Flow Matching for diverse yet structurally consistent full-body motion forecasting. Extensive experiments across all three domains demonstrate consistent improvements over representative baselines on both location and pose forecasting, while ablations validate the contributions of the proposed components. The project page is available at https://corrineqiu.github.io/from-where-to-how/. Index Terms: 4D interaction forecasting, egocentric video, interaction location forecasting, full-body pose forecasting. I Introduction Egocentric 4D interaction forecasting aims to jointly anticipate where future interactions will occur in 3D space and how the human body will move to execute them. This capability is fundamental to proactive embodied intelligence, where effective anticipation, planning, and assistance require a temporally continuous and geometrically consistent understanding of future locations and their associated body motions. It can support applications such as task assistance [1], risk warning [2], and human-robot collaboration [3]. Despite the inherently coupled nature of interaction location and body motion, existing research has largely approached them as two separate forecasting problems. For interaction location forecasting, existing methods predict the spatial targets or trajectories of upcoming interactions from egocentric observations, including continuous hand motion regression using temporal or state-space models [4], generative modeling of multiple plausible hand trajectories [5], and semantic or language-guided forecasting for object grounding and procedural reasoning [6, 7, 8, 9]. In parallel, full-body pose forecasting predicts temporally evolving human poses from motion history and contextual cues. Existing methods leverage pose history, egocentric observations, scene context, robot-view cues, or map-aware representations [10, 11, 12, 13, 14], while stochastic diffusion- or flow-based methods model multiple feasible future motions [15, 16, 17] and skeleton-aware approaches incorporate structural or kinematic priors to preserve valid body configurations [18]. Together, these two lines of research provide complementary capabilities for predicting where future interactions occur and how the body moves to realize them, but do not explicitly model their temporal and geometric correspondence. FIction [19] represents an early effort to bridge this gap by connecting interaction localization and pose forecasting within a unified dataset and task formulation. It represents future interaction locations as discrete voxel occupancy and predicts poses conditioned on candidate locations, thereby introducing an explicit dependency between the two predictions. However, interaction locations and full-body poses are not modeled as continuous, temporally aligned sequences in a shared metric space. As a result, the stepwise correspondence between the evolving interaction target and the body motion that realizes it remains unresolved. Fig. 1: Task distribution of Coherent4D. Sequences are associated with the corresponding task annotations to summarize the procedural coverage of the dataset. This limitation exposes a more fundamental gap: existing studies have not yet established a continuous and jointly grounded 4D forecasting formulation that preserves the temporal and geometric correspondence between future interaction locations and full-body motion. Addressing this gap involves four key challenges: ❶ Decoupled task formulation. Even when location and motion cues are jointly available, existing predictors typically treat interaction localization and pose forecasting as independent or loosely connected objectives. Consequently, continuous predictions of interaction locations are not explicitly propagated as geometric constraints for subsequent motion forecasting, limiting coordination between where an interaction develops and how it is physically realized. ❷ Spatiotemporal pairing gap. Existing datasets rarely provide ordered, continuous 3D hand interaction locations paired with full-body poses at matched future timestamps in a shared metric coordinate system. Without such temporally synchronized and spatially co-registered targets, models cannot directly learn or evaluate how interaction locations and body motion co-evolve over time. ❸ Semantic-dynamic localization gap. Accurate location forecasting requires both task-level semantic grounding and precise continuous 3D localization over future time steps. However, representations from vision-language models (VLMs) are primarily optimized for semantic reasoning rather than metric coordinate regression and provide limited modeling of short-horizon visual dynamics. This mismatch can produce semantically plausible yet spatially inaccurate location predictions. ❹ Pose diversity-structure trade-off. Deterministic pose forecasting tends to preserve structural consistency but suppresses alternative feasible executions, while stochastic generation captures multiple feasible futures but may compromise skeletal structure and joint consistency. Effective forecasting therefore requires motion diversity to be modeled without sacrificing structural plausibility. To address these challenges, we introduce Coherent4D, a large-scale egocentric dataset for continuous 4D interaction forecasting. As shown in Fig. 1, Coherent4D contains approximately 233K samples spanning Cooking, Health, and Bike Repair. Unlike existing datasets that provide location and motion supervision separately or through discrete spatial representations, each sample in Coherent4D contains two temporally synchronized targets in a shared 3D coordinate system: an ordered sequence of continuous future interaction locations describing where the interaction evolves over time, and a corresponding full-body pose sequence describing how the body realizes it. This paired formulation establishes explicit temporal and geometric correspondence between interaction localization and motion realization. We further introduce continuous-space evaluation metrics to assess both forecasts at corresponding future time steps. Based on this coupled formulation, we propose HIGFlow, a cascaded framework for continuous 4D interaction forecasting that follows a structured where-to-how process. HIGFlow first forecasts the continuous spatial progression of future locations and then uses these forecasts to condition the corresponding full-body motion, thereby explicitly coupling interaction localization with motion realization. For location forecasting, we develop Semantic-Dynamic Location Forecasting, which decouples task-level semantic grounding from continuous metric localization while incorporating short-horizon latent visual dynamics to improve future location prediction. For full-body pose forecasting, we introduce Hand-Conditioned Residual Flow Matching, which first constructs a deterministic motion anchor conditioned on the predicted future interaction locations and then models bounded stochastic residuals around this anchor. This design combines a geometrically grounded and structurally stable motion estimate with stochastic variation, enabling diverse future pose predictions while preserving skeletal structure and joint consistency. Extensive experiments validate the soundness of the Coherent4D dataset and the effectiveness of HIGFlow in jointly modeling interaction localization and full-body motion realization. Our main contributions are summarized as follows: • We introduce Coherent4D, a large-scale egocentric dataset with approximately 233K samples, pairing ordered future 3D interaction locations with temporally aligned full-body poses in a shared coordinate system. • We propose HIGFlow, a cascaded where-to-how framework that uses predicted interaction locations as geometric conditions for full-body pose forecasting, explicitly coupling interaction localization with motion realization. • We develop Semantic-Dynamic Location Forecasting for accurate continuous interaction localization and Hand-Conditioned Residual Flow Matching for diverse yet structurally consistent full-body motion forecasting. • Extensive experiments demonstrate the effectiveness of Coherent4D and HIGFlow, as well as the benefit of jointly modeling future interaction locations and full-body motion. I Related Work Fig. 2: Representative subset of interaction events from a Coherent4D sample. Synchronized exocentric source frames on both sides provide visual references for the selected events. The center shows the scene geometry, object bounding boxes, and temporally ordered SMPL body states in the shared sample-local coordinate frame. The action timeline summarizes the corresponding interaction objects, while colors associate each event with its rendered body state and reference frame. I-A Datasets for 4D Interaction Forecasting Large-scale egocentric datasets such as EPIC-KITCHENS and Ego4D provide broad supervision for action understanding and anticipation [20, 21], but do not pair ordered continuous 3D interaction locations with temporally synchronized full-body poses. Hand-centric datasets support forecasting future hand motion or interaction targets from egocentric observations. EgoPAT3D [22] and EgoPAT3Dv2 [23] focus on 3D action target forecasting, while EgoHandTrajPred, introduced with USST [4], provides ordered annotations for future 3D hand trajectory prediction. More recent datasets incorporate richer semantic and state information: EgoH4 [5] supports forecasting future 3D trajectories and poses of both hands, EgoHaFL [6] provides language-guided annotations of future hand states, poses, and trajectories, and EgoMAN [7] supports 6-DoF hand trajectory forecasting with location awareness, using structured semantic, spatial, and motion cues. In contrast, pose-centric datasets focus on future full-body motion. MoGaze [24] and GIMO [10] incorporate workspace geometry, scene scans, egocentric observations, and gaze, while HARPER [11] and Real-IM [12] extend pose forecasting to robot-centric and map-aware settings. However, hand-centric datasets generally lack temporally aligned full-body pose sequences, whereas pose-centric datasets rarely annotate future interaction moments. Consequently, existing datasets cannot directly associate future interaction locations with the corresponding full-body motion or evaluate whether the predicted body reaches the intended interaction target at the correct time. FIction [19] is the closest prior dataset linking future interaction localization with body pose forecasting. However, it represents locations as voxel occupancy and conditions pose prediction on individual candidate locations, rather than providing continuous, temporally aligned sequences of interaction locations and full-body poses. Coherent4D fills this gap by pairing ordered 3D interaction locations with synchronized full-body poses parameterized by the Skinned Multi-Person Linear (SMPL) [25] model in a shared coordinate system, enabling joint evaluation of interaction localization and motion execution. I-B 3D Interaction Location Forecasting 3D interaction location forecasting predicts ordered metric hand interaction locations from egocentric observations, specifying where upcoming hand-environment interactions will occur. Related egocentric studies address long-term anticipation and next active object prediction [26, 27, 28]. Early hand trajectory forecasting methods mainly operate in the 2D image space. OCT predicts future 2D hand trajectories with interaction hotspots, while Diff-IP2D and MADiff use diffusion- or dynamics-aware modeling to capture future uncertainty and ego-motion effects [29, 30, 31]. Recent works extend hand forecasting to metric 3D space: USST establishes egocentric 3D hand trajectory forecasting from RGB observations, MMTwin extends the MADiff-style diffusion paradigm to multimodal 3D hand trajectory forecasting, and Uni-Hand further unifies 2D/3D hand waypoint forecasting with richer hand motion and interaction targets [4, 32, 33]. Although these methods advance hand motion forecasting from image-space trajectories to metric 3D waypoints, they still primarily model future hand motion itself, without sufficiently integrating task-level semantic grounding, continuous metric regression, and short-horizon visual dynamics for ordered future interaction location forecasting. Semantic and VLM-assisted methods introduce object-centric grounding, action semantics, and procedural context [34, 35, 36, 37]. HandsOnVLM shows that directly serializing coordinates as language is insufficient, and instead uses dedicated hand tokens with a trajectory decoder [38]. Predictive video models and latent world representations further capture short-horizon spatiotemporal dynamics [39, 40]. However, semantic reasoning, continuous metric regression, and predictive visual dynamics are still rarely integrated in a unified interaction location predictor. To address this limitation, our Semantic-Dynamic Location Forecasting separates task-level VLM grounding from coordinate decoding and augments the future position representation with short-horizon visual dynamics, improving goal consistency while preserving deterministic continuous 3D regression. I-C Full-Body Pose Forecasting Full-body pose forecasting predicts future global motion and articulated body configurations from observed pose histories and contextual cues [41, 42]. Prior methods improve motion stability using gaze, workspace context, spatiotemporal anchors, global trajectories, or affordance cues. MoGaze incorporates gaze and scene context, STARS separates deterministic anchors from within-mode variation, T2P conditions local pose forecasting on predicted global trajectories, and GAP3DS uses gaze-informed affordance cues in 3D scenes [24, 43, 44, 45]. These methods improve scene consistency and stable coarse motion, but deterministic or anchor-based forecasting often suppresses alternative feasible executions when the same interaction can be realized by different body motions. Other methods improve motion diversity through latent variables, diffusion, flow matching, motion fields, skeleton-aware generation, or residual refinement [46, 47, 48]. SLD-HMP learns controllable semantic latent directions, BeLFusion and CoMusion improve diverse motion generation and history consistency, SkeletonDiffusion incorporates structural priors for anatomical plausibility, and PrediFlow refines coarse motion forecasts with Flow Matching residuals [49, 50, 51, 18, 52]. While these methods enhance diversity or realism, noise-initialized generation can still introduce inefficient sampling, temporal instability, or implausible articulation when stochastic variation is not sufficiently constrained. Our Hand-Conditioned Residual Flow Matching addresses this diversity-stability trade-off by first establishing a hand-conditioned deterministic SMPL motion anchor and then modeling bounded stochastic residuals around it. I Coherent4D Dataset Coherent4D is constructed from Ego-Exo4D [53] to support continuous 4D interaction forecasting from egocentric video. Here, continuous refers to both the temporal continuity of motion sequences and the use of continuous 3D coordinates rather than discrete spatial grids. It pairs ordered 3D interaction locations with SMPL-parameterized full-body pose sequences at matched future timestamps in a shared sample-local coordinate frame, jointly capturing the spatial progression of future locations and the corresponding body motion. Each sample contains 30 egocentric frames uniformly sampled from a 30-second observation window, structured object and environment descriptors, histories of observed interaction locations and SMPL poses, and the corresponding future interaction location and pose targets. Fig. 2 illustrates representative interaction events from a Coherent4D sample. Each event associates a continuous 3D interaction location with the corresponding SMPL body state at the same timestamp, explicitly preserving their temporal and spatial correspondence. I-A Annotation Pipeline We construct Coherent4D from synchronized procedural takes in Ego-Exo4D [53]. The Aria egocentric stream serves as the forecasting input, while synchronized exocentric views are used only for offline annotation construction and refinement. To establish temporally and geometrically coupled supervision, we organize the source annotations into ordered interaction location and SMPL pose sequences in a shared sample-local coordinate frame. As illustrated in Fig. 3, the pipeline consists of five stages: scene object grounding, shared coordinate construction, location sequence construction, SMPL state attachment, and forecast sample generation. Fig. 3: Overview of the Coherent4D data construction pipeline. We (1) ground scene objects with semantic labels and 3D bounding boxes, (2) transform scene, location, and body annotations into a shared sample-local coordinate frame, (3) construct temporally ordered 3D interaction location sequences, (4) attach time-aligned SMPL states, and (5) organize them into forecasting samples with location-pose histories, future targets, and relative timestamps. I-A1 Scene Object Grounding We construct object-level semantic and geometric context for interaction location forecasting. We apply Detic [54] with the LVIS vocabulary [55] to detect candidate objects in the egocentric frames. The detected regions are lifted into 3D using the Ego-Exo4D SLAM reconstruction and subsequently clustered to obtain oriented 3D bounding boxes. For each object, we retain its semantic category together with continuous 3D bounding box attributes, including center, size, and orientation. The spatial attributes are subsequently expressed in the shared sample-local coordinate frame defined below, while the box dimensions retain their metric scale. This representation preserves both object semantics and continuous scene geometry without discretizing the environment into voxels. I-A2 Shared Coordinate Construction Geometric coupling between interaction location, body motion, and scene geometry requires all spatial quantities to be represented in a common coordinate frame. For each sample n, we define a stable egocentric reference pose (nref,nref)(R_n^ref,t_n^ref). Given a point wp^w in the world coordinate frame, its representation in the shared sample-local coordinate frame is defined as: nloc=(nref)⊤(w−nref).p^loc_n= (R_n^ref ) (p^w-t_n^ref ). (1) We apply this transformation to interaction locations, object centers, SMPL root translations, and 3D body joints. Global orientations, including the SMPL root orientation and object orientations, are rotated by the same reference rotation, whereas the 23 SMPL body joint rotations remain defined relative to their parent joints. For model input and supervision, positional quantities are divided by s=5ms=5\,m and then clipped to [−1,1][-1,1] for each coordinate. Division by s rescales the coordinates without changing the reference frame, whereas clipping limits values outside this range to the corresponding boundary. For model input and supervision, coordinates are normalized by dividing by s=5ms=5\,m and clipping to [−1,1][-1,1], which rescales their values without changing the reference frame. Root translations and translation residuals remain in meters for pose residual computation, clipping, and composition. I-A3 Location Sequence Construction We convert sparse hand-object interaction annotations into temporally ordered continuous 3D interaction location sequences. Following FIction [19], candidate interaction events are identified by combining narration timestamps, Llama 3-based object matching [56], and geometric consistency between the hand and the corresponding object. Each valid event retains its timestamp, interaction object when available, and continuous 3D interaction location recovered from the corresponding hand mesh. Interactions involving the right hand or both hands are mapped into a unified interaction stream. Temporally adjacent events with redundant spatial locations are merged according to their timestamps and normalized spatial displacement. The resulting sequence therefore contains distinct interaction locations while preserving their chronological order and continuous spatial evolution. I-A4 SMPL State Attachment We then associate each interaction location with the full-body state at the corresponding timestamp. We reconstruct human motion using WHAM [57], select the primary actor track, and align its SMPL trajectory with the Ego-Exo4D scene coordinate system. For every observed and future interaction timestamp, we retrieve the nearest valid SMPL state and transform its global components into the sample-local frame. Each SMPL state contains the root translation, root orientation, body joint rotations, and 3D joint positions. The root translation and 3D joints are thus spatially co-registered with the corresponding interaction location, while body joint rotations remain relative to their parent joints. This step yields temporally paired interaction location and full-body pose sequences for subsequent forecasting. TABLE I: Statistics of Coherent4D. Train, Val, Test, and Total report the numbers of forecasting samples. Targets denote valid future interaction locations, excluding padded steps. Obj. denotes interaction object labels, while the Total entry for Obj. reports the number of unique labels across the dataset. Domain Horizon Takes Train Val Test Total Targets Obj. Cooking 10 331 136,079 15,435 14,527 166,041 1,337,589 428 Health 5 205 21,532 1,899 4,167 27,598 114,964 170 Bike Repair 4 251 35,987 3,150 1,052 40,189 141,633 197 Total – 787 193,598 20,484 19,746 233,828 1,594,186 535 Fig. 4: Narration verb distribution over future interaction locations. Bars show the share of all future interaction targets for each normalized first narration verb, and colors indicate the contribution from each domain. I-A5 Forecast Sample Generation Finally, we convert the aligned annotation streams into forecasting samples while retaining the nonuniform timing of interaction events. The ordered sequences are partitioned according to the domain-specific forecasting horizons reported in Table I. For each sample n, we take the first future interaction timestamp as the time origin and denote it by tn,1t_n,1. The preceding 30 seconds constitute the observation window and provide the egocentric frames, location history, and pose history. For each future step k∈1,…,Kk∈\1,…,K\, we retain its absolute timestamp tn,kt_n,k, relative timestamp Δtn,k=tn,k−tn,1 t_n,k=t_n,k-t_n,1, continuous interaction location, and temporally aligned SMPL state. The relative timestamps preserve the nonuniform intervals between interaction events, while a fixed number of target steps provides a consistent sequence interface for training and evaluation. Incomplete tail sequences are retained using padding and validity masks. Samples with invalid timestamp alignment, object grounding, coordinate transformation, interaction locations, or SMPL attachment are discarded. We split the dataset at the take level to prevent overlap of environments and procedural sequences across the training, validation, and test sets. I-B Dataset Statistics Fig. 5: Overview of HIGFlow. The first stage forecasts the locations of future interactions from egocentric context, and the second stage forecasts temporally aligned full-body poses conditioned on the predicted location sequence. As summarized in Table I, Coherent4D contains 233,828 samples derived from 787 unique takes across three domains: Cooking, Health, and Bike Repair. The corresponding forecasting horizons are 10, 5, and 4 interaction steps, respectively. The dataset further covers 535 interaction object categories, providing diverse object-centric procedural activities across the three domains. To characterize action-level diversity, we associate each valid future interaction target with its corresponding narration. All 1,594,186 valid future targets are aligned with narration annotations. Fig. 4 shows the normalized distribution of the first verb in each narration, covering common manipulation primitives such as pick, place, hold, drop, move, pour, pass, and turn. Cooking accounts for most high-frequency manipulation actions, while Bike Repair and Health contribute complementary domain-specific interaction patterns. I-C Continuous Metrics Voxel-based accuracy measures whether a discrete spatial cell is correctly activated, but does not quantify continuous 3D localization or motion errors. We therefore introduce continuous-space metrics for both interaction location and full-body pose forecasting. For evaluation, normalized interaction locations are converted back to metric coordinates in the shared sample-local frame using the position scale s. SMPL root translations and 3D joints are evaluated in the same metric frame. Positional and rotational errors are reported in millimeters and degrees, respectively, with lower values indicating better performance. Let D denote the evaluation set and K the forecasting horizon. For sample n and future step k, let mn,k∈0,1m_n,k∈\0,1\ indicate whether the paired interaction location and pose targets are valid. I-C1 Interaction Location Forecasting Metrics To characterize overall sequence accuracy, upper-tail localization error, and endpoint accuracy, we report average displacement error (ADE), ADE90ADE_90, and final displacement error (FDE), respectively. For sample n and future step k, let n,k∈ℝ3y_n,k ^3 and ^n,k∈ℝ3 y_n,k ^3 denote the ground-truth and predicted interaction locations, respectively. The localization error at each future step is dn,kloc=‖^n,k−n,k‖2d^loc_n,k= \| y_n,k-y_n,k \|_2. We compute the ADE by first averaging over the valid future steps of each sample and then across samples: ADE=1||∑n∈∑k=1Kmn,kdn,kloc∑k=1Kmn,k.ADE= 1|D| _n _k=1^Km_n,kd^loc_n,k _k=1^Km_n,k. (2) We additionally report ADE90ADE_90, defined as the 90th percentile of the per-sample ADE values, to characterize upper-tail localization error. FDE measures localization accuracy at the last valid future step. Let κn=maxk:mn,k=1 _n= \k:m_n,k=1\ denote the last valid step of sample n. FDE is then defined as: FDE=1||∑n∈dn,κnloc.FDE= 1|D| _n d^loc_n, _n. (3) I-C2 Full-body Pose Forecasting Metrics To assess absolute joint position accuracy, articulated pose accuracy after similarity alignment, global body displacement, and local body rotation, we report mean per-joint position error (MPJPE), Procrustes-aligned mean per-joint position error (PA-MPJPE), root translation error (Root Trans.), and body geodesic error (Body Geo.), respectively. Following FIction [19], we evaluate each SMPL state using the same set of Jpos=19J_pos=19 body joints. Let n,k,^n,k∈ℝ3×JposP_n,k, P_n,k ^3× J_pos denote the ground-truth and predicted joint coordinate matrices for sample n at future step k, with n,k,j,^n,k,j∈ℝ3p_n,k,j, p_n,k,j ^3 denoting the corresponding j-th joint coordinates. For compact notation, let Nvalid=∑n∈∑k=1Kmn,kN_valid= _n _k=1^Km_n,k denote the total number of valid future steps. MPJPE measures the average Euclidean distance between corresponding predicted and ground-truth joints over all valid future poses: MPJPE=1JposNvalid∑n∈∑k=1K∑j=1Jposmn,k‖^n,k,j−n,k,j‖2.MPJPE= 1J_posN_valid _n _k=1^K _j=1^J_posm_n,k \| p_n,k,j-p_n,k,j \|_2. (4) PA-MPJPE evaluates pose accuracy after removing global similarity misalignment. For each valid pose of sample n at future step k, we align the predicted joints to the ground truth using the optimal similarity transformation: (sn,k⋆,n,k⋆,n,k⋆)=ProcAlign(^n,k,n,k),~n,k=sn,k⋆n,k⋆^n,k+n,k⋆⊤, gathered (s _n,k,Q _n,k,b _n,k )=ProcAlign ( P_n,k,P_n,k ),\\[2.0pt] P_n,k=s _n,kQ _n,k P_n,k+b _n,k1 , gathered (5) where sn,k⋆∈ℝ+s _n,k _+, n,k⋆∈SO(3)Q _n,k (3), and n,k⋆∈ℝ3b _n,k ^3 denote the optimal scale, alignment rotation, and translation, respectively, and ~n,k P_n,k denotes the aligned prediction. The vector ∈ℝJpos1 ^J_pos broadcasts the translation to all joints. Let ~n,k,j p_n,k,j denote the j-th column of ~n,k P_n,k. PA-MPJPE is then defined as: PA-MPJPE=∑n∈∑k=1K∑j=1Jposmn,k‖~n,k,j−n,k,j‖2JposNvalid.PA -MPJPE= _n _k=1^K _j=1^J_posm_n,k \| p_n,k,j-p_n,k,j \|_2J_posN_valid. (6) Root Trans. measures the Euclidean distance between predicted and ground-truth SMPL root translations, reflecting the accuracy of global body displacement. Let n,k,^n,k∈ℝ3t_n,k, t_n,k ^3 denote the ground-truth and predicted SMPL root translations for sample n at future step k, respectively. It is defined as: Root Trans.=1Nvalid∑n∈∑k=1Kmn,k‖^n,k−n,k‖2.Root Trans.= 1N_valid _n _k=1^Km_n,k \| t_n,k-t_n,k \|_2. (7) Body Geo. measures the rotational discrepancy of the Jrot=23J_rot=23 non-root body joints. Let n,k,j,^n,k,j∈SO(3)R_n,k,j, R_n,k,j (3) denote the ground-truth and predicted local rotation matrices of body joint j, respectively. Their geodesic angular distance is defined as: dn,k,jrot=arccos(tr(^n,k,j⊤n,k,j)−12),d^rot_n,k,j= ( tr ( R_n,k,j R_n,k,j )-12 ), (8) where the argument of arccos(⋅) (·) is clipped to [−1,1][-1,1] for numerical stability. The mean body geodesic error, excluding the root joint, is Body Geo.=180πJrotNvalid∑n∈∑k=1K∑j=1Jrotmn,kdn,k,jrot.Body Geo.= 180π J_rotN_valid _n _k=1^K _j=1^J_rotm_n,kd^rot_n,k,j. (9) To account for the inherent multimodality of future body motion, the model generates multiple plausible pose sequences for each observation. We therefore report Single and Best-5 to evaluate the first candidate and the best candidate among five forecasts, respectively. For the q-th candidate, let ^n(q)=^n,k(q)k=1K X^(q)_n=\ X^(q)_n,k\_k=1^K denote the predicted SMPL sequence. Let ^n,k(q)∈ℝ3×Jpos P^(q)_n,k ^3× J_pos and ^n,k,j(q)∈ℝ3 p^(q)_n,k,j ^3 denote its joint coordinate matrix and j-th joint coordinate, respectively. For Best-5, we select for each sample the candidate with the lowest joint position error over all valid future steps: qn⋆=argminq∈1,…,5∑k=1K∑j=1Jposmn,k‖^n,k,j(q)−n,k,j‖2.q _n= _q∈\1,…,5\ _k=1^K _j=1^J_posm_n,k \| p^(q)_n,k,j-p_n,k,j \|_2. (10) All Best-5 pose metrics are computed from the same selected SMPL sequence ^n(qn⋆) X^(q _n)_n. Its joint coordinates are used for MPJPE and PA-MPJPE, while its root translations and body joint rotations are used for Root Trans. and Body Geo., respectively. This ensures that all Best-5 metrics evaluate a consistent motion forecast selected according to joint position accuracy. IV Method Algorithm 1 Training and inference of HIGFlow 0: Observed contexts (loc,ℋpose)(C_loc,H_pose), ground-truth sequences (,)(Y,X), validity masks (ℳY,ℳX)(M_Y,M_X), forecasting horizon K, Heun steps NODEN_ODE, pose candidates S. 0: Interaction location sequence Y and pose sequences ^(q)q=1S\ X^(q)\_q=1^S. 0: // Training 0: Stage 1: Interaction location training 1: Encode locC_loc with Qwen3-VL and extract kQk=1K\h_k^Q\_k=1^K at the indexed <hand_traj_k> positions. 2: Decode the interaction locations and train the location predictor by minimizing ℒlocL_loc in Eq. (16) using ℳYM_Y. 3: Encode the observed frames with V-JEPA. Fuse the dynamic features with kQk=1K\h_k^Q\_k=1^K and update the V-JEPA adapter by minimizing λADEℒADE+λS1ℒS1 _ADEL_ADE+ _S1L_S1 using ℳYM_Y. 3: Stage 2: Full-body pose training 4: Encode (ℋpose,)(H_pose,Y) and obtain the anchor sequence ^kak=1K\ X^a_k\_k=1^K by Eq. (18). 5: Train the anchor predictor by minimizing ℒanchorL_anchor in Eq. (19) using ℳXM_X. 6: Compute the residual targets k∗k=1K\r_k \_k=1^K by Eq. (21) and construct kflow,ℛkk=1K\c_k^flow,R_k\_k=1^K by Eqs. (22) and (23). 7: for k=1k=1 to K do 8: Sample τ∼(0,1)τ (0,1) and k∼(,σr275)z_k (0, _r^2I_75). 9: Compute the flow state τ,kx_τ,k as (1−τ)k+τℬ(k∗)(1-τ)z_k+ _ ρ(r_k ). 10: Accumulate ℒFML_FM according to Eq. (25) and compute the direct and Heun rollout residuals. 11: end for 12: Compose the residuals with the anchors and train the flow model by minimizing ℒposeL_pose in Eq. (30) using ℳXM_X. 12: // Inference 13: Apply the trained location predictor to locC_loc and obtain ^=^kk=1K Y=\ y_k\_k=1^K. 14: Condition on (ℋpose,^)(H_pose, Y) to obtain the anchor sequence and residual flow conditions ^ka,kflow,ℛkk=1K\ X^a_k,c_k^flow,R_k\_k=1^K. 15: for q=1q=1 to S do 16: Independently sample the residual priors k(q)k=1K\z_k^(q)\_k=1^K. 17: Integrate the residual flows with NODEN_ODE Heun steps and compose them with the anchors to obtain ^k(q)k=1K\ X^(q)_k\_k=1^K. 18: Collect the K predicted body states to form ^(q)=^k(q)k=1K X^(q)=\ X^(q)_k\_k=1^K. 19: end for 20: return Y and ^(q)q=1S\ X^(q)\_q=1^S. IV-A Problem Formulation Given an observed egocentric context, we formulate continuous 4D interaction forecasting as a coupled where-to-how prediction problem: first forecasting an ordered sequence of future interaction locations and then predicting the temporally aligned full-body motion conditioned on these locations. For interaction location forecasting, the observed context is loc=(obs,ℰ,obs,)C_loc= (V_obs,E,O_obs,T ), where obsV_obs, ℰE, obsO_obs, and T denote the observed egocentric frames, structured environment descriptors, observed location history, and task prompt, respectively. The future location sequence is predicted as ^=floc(loc)=^kk=1K, Y=f_loc(C_loc)=\ y_k\_k=1^K, (11) where K is the forecasting horizon and ^k∈ℝ3 y_k ^3 denotes the predicted interaction location at future step k in the normalized coordinate representation. The corresponding ground-truth location sequence is denoted by =kk=1KY=\y_k\_k=1^K. For full-body pose forecasting, the model takes the observed pose history ℋposeH_pose together with a future interaction location sequence ~ Y: ^=fpose(ℋpose,~)=^kk=1K. X=f_pose (H_pose, Y )=\ X_k\_k=1^K. (12) During training, ~= Y=Y is the ground-truth location sequence, whereas during inference, ~= Y= Y is predicted by the first stage. The corresponding ground-truth pose sequence is denoted by =kk=1KX=\X_k\_k=1^K, with the same validity mask as the interaction location targets, i.e., ℳX=ℳYM_X=M_Y. Each ^k∈ℝ147 X_k ^147 represents an SMPL body state comprising a 6D root orientation, a 3D root translation, and 23 local body joint rotations represented in continuous 6D form. IV-B HIGFlow Framework HIGFlow instantiates the above where-to-how formulation with the cascaded architecture shown in Fig. 5. It consists of two specialized components: Semantic-Dynamic Location Forecasting for predicting continuous future interaction locations, and Hand-Conditioned Residual Flow Matching for generating the corresponding full-body motion. The predicted location sequence serves as an explicit geometric condition for motion forecasting, establishing a direct dependency between location progression and body motion realization. Algorithm 1 summarizes the training and inference procedures. IV-B1 Semantic-Dynamic Location Forecasting The interaction location stage forecasts an ordered sequence of continuous 3D interaction locations by combining high-level semantic grounding with short-horizon visual dynamics. Specifically, Qwen3-VL [58] encodes the observed egocentric context and provides semantic representations for future steps, while a frozen V-JEPA [59] encoder supplies complementary motion-sensitive features for dynamic refinement. Semantic context encoding. Qwen3-VL receives the sampled egocentric frames, task prompt, observed location history, and structured environment descriptors. The location and environment encoders transform obsO_obs and ℰE into dense features, which replace their corresponding placeholder embeddings before the Qwen3-VL forward pass. For each future step k, we extract the hidden state at the indexed placeholder <hand_traj_k> as the step-specific semantic representation kQh_k^Q. Dynamic feature augmentation. Semantic representations provide task- and object-level grounding but may not sufficiently capture short-term hand-object dynamics that are important for precise spatial forecasting. We therefore encode the observed video with V-JEPA [59] and project its latent features into the Qwen hidden space, followed by resampling into M dynamic memory tokens. Each future step representation kQh_k^Q, augmented with a learned step embedding klocs_k^loc, attends to this dynamic memory to obtain the motion context kc_k. We then inject the dynamic information through a gated residual adapter: kF=kQ+gkΔk,Δk=αtanh(D([kQ,k,kloc])),gk=sigmoid(G([kQ,k,kloc])), \ aligned h_k^F&=h_k^Q+g_k _k,\\ _k&=α (D\! ([h_k^Q,c_k,s_k^loc] ) ),\\ g_k&=sigmoid (G\! ([h_k^Q,c_k,s_k^loc] ) ) aligned ., (13) where D and G denote lightweight residual and gating heads, respectively. The scalar gk∈(0,1)g_k∈(0,1) adaptively controls the contribution of the dynamic residual, while α>0α>0 bounds its magnitude. Continuous coordinate decoding. A coordinate decoder maps each fused representation kFh_k^F directly to the normalized continuous 3D interaction location ^k y_k. This explicit regression head separates continuous spatial prediction from the language generation interface, avoiding the need to represent metric coordinates as text tokens. Let B denote the mini-batch size and b∈1,…,Bb∈\1,…,B\ index the samples. Let (ℳY)b,k∈0,1(M_Y)_b,k∈\0,1\ indicate whether the interaction target b,ky_b,k is valid. We define the mask-aware ADE loss as: ℒADE=1B∑b=1B∑k=1K(ℳY)b,k‖^b,k−b,k‖2∑k=1K(ℳY)b,k.L_ADE= 1B _b=1^B _k=1^K(M_Y)_b,k \| y_b,k-y_b,k \|_2 _k=1^K(M_Y)_b,k. (14) The Smooth L1 location loss is defined as: ℒS1=1B∑b=1B∑k=1K(ℳY)b,kSmoothL1(^b,k,b,k)∑k=1K(ℳY)b,k.L_S1= 1B _b=1^B _k=1^K(M_Y)_b,kSmoothL1 ( y_b,k,y_b,k ) _k=1^K(M_Y)_b,k. (15) The overall location objective is: ℒloc=λADEℒADE+λS1ℒS1+λCEℒCE,L_loc= _ADEL_ADE+ _S1L_S1+ _CEL_CE, (16) where ℒCEL_CE denotes the auxiliary language modeling loss. The SmoothL1SmoothL1 loss is computed coordinate-wise and averaged over the three spatial dimensions. Both location regression losses are evaluated in the normalized coordinate space. IV-B2 Hand-Conditioned Residual Flow Matching Given the observed pose history and ordered future interaction locations, the pose stage first forecasts a location-conditioned deterministic anchor and then models stochastic residuals around it using conditional flow matching. TABLE I: Interaction location forecasting results on the Coherent4D dataset. HIGFlow and all baselines are evaluated under the same 3D setting, following the MMTwin 3D setting. All values are reported in millimeters, with lower values being better. Bold and underline denote the best and second-best results, respectively. Model Health Bike Repair Cooking ADE↓ ADE90↓_90 FDE↓ ADE↓ ADE90↓_90 FDE↓ ADE↓ ADE90↓_90 FDE↓ FIction 40.30 62.62 40.58 88.55 143.21 98.61 100.61 184.63 107.11 Qwen3-VL 57.53 89.54 82.04 90.71 156.55 103.81 103.95 199.20 112.22 V-JEPA 45.18 69.49 46.26 86.50 142.43 96.33 102.33 194.80 107.35 Diff-IP3D 292.84 433.26 294.25 525.63 842.93 561.19 647.23 1268.28 662.71 MMTwin 290.45 437.88 285.21 429.71 690.80 480.29 513.75 957.17 556.40 HIGFlow 40.21 59.52 41.44 80.91 131.87 93.78 93.46 178.20 104.26 Spatiotemporal conditioning and anchoring. To preserve the kinematic structure of the human body, each observed SMPL state is represented as a 24-node graph consisting of one root joint and 23 articulated body joints, with skeletal connections defining the graph edges. Graph propagation captures dependencies among physically connected body parts while preserving the SMPL topology. The resulting node features are pooled into frame-level pose tokens and processed by a pose history Transformer, whose final classification token state h summarizes the observed motion history. Given the interaction location sequence ~ Y defined in the problem formulation, we set Δ~1= y_1=0 and Δ~k=~k−~k−1 y_k= y_k- y_k-1 for k=2,…,Kk=2,…,K. The future condition encoder represents each future step as: (k)k=1K=Transformerfut((k)k=1K),k=MLPc([~k,Δ~k,k/K]), \ aligned &(f_k)_k=1^K=Transformer_fut (( η_k)_k=1^K ),\\[2.0pt] & η_k=MLP_c ([ y_k, y_k,k/K] ) aligned ., (17) where k η_k encodes the interaction location, displacement, and relative future step, and kf_k denotes the corresponding context-aware representation. For each future step, the deterministic anchor is predicted by combining the global pose history representation with the corresponding future location representation: ^ka=MLPa([,k]). X_k^a=MLP_a ([h,f_k] ). (18) The anchor estimates the root translation, root orientation, and articulated body configuration, providing a reference grounded in the interaction location for subsequent residual motion generation rather than serving as the final prediction. The anchor predictor is optimized with ℒanchor= _anchor= λ6Dℒ6D+λgeoℒgeo+λrootgeoℒrootgeo _6DL_6D+ _geoL_geo+ _rootgeoL_rootgeo (19) +λbodygeoℒbodygeo+λtransℒtrans + _bodygeoL_bodygeo+ _transL_trans +λvelℒvel+λjointℒjoint, + _velL_vel+ _jointL_joint, where ℒ6DL_6D is the coordinate-wise ℓ1 _1 loss over the 24 continuous 6D rotations. ℒgeoL_geo measures the mean geodesic error over all 24 rotations, while ℒrootgeoL_rootgeo and ℒbodygeoL_bodygeo separately supervise the root and 23 non-root body rotations. All framewise losses are evaluated only at future steps marked valid by ℳXM_X. Let ^b,ka t^a_b,k and b,kt_b,k denote the predicted and ground-truth root translations, and let ^b,k,ja p^a_b,k,j and b,k,jp_b,k,j denote the corresponding 3D joint positions. The translation and joint position losses are ℒtrans=∑b=1B∑k=1K(ℳX)b,k‖^b,ka−b,k‖13∑b=1B∑k=1K(ℳX)b,k,ℒjoint=∑b=1B∑k=1K(ℳX)b,k∑j=1Jpos‖^b,k,ja−b,k,j‖2Jpos∑b=1B∑k=1K(ℳX)b,k. gatheredL_trans= _b=1^B _k=1^K(M_X)_b,k \| t^a_b,k-t_b,k \|_13 _b=1^B _k=1^K(M_X)_b,k,\\[2.0pt] L_joint= _b=1^B _k=1^K(M_X)_b,k _j=1^J_pos \| p^a_b,k,j-p_b,k,j \|_2J_pos _b=1^B _k=1^K(M_X)_b,k. gathered (20) The velocity loss ℒvelL_vel further penalizes the masked coordinate-wise ℓ1 _1 error between consecutive differences of the predicted and ground-truth 147-dimensional SMPL states, using only valid consecutive-step pairs. After anchor pretraining, the pose history encoder, future condition encoder, and anchor predictor are frozen. Residual Flow Matching. To capture multiple feasible motions without deviating excessively from the deterministic anchor, we model stochastic variations in an anchor-relative residual space using conditional Flow Matching [17]. For each future step k, the target residual is defined as: k∗=Residual(^ka,k),r_k =Residual ( X^a_k,X_k ), (21) where ResidualResidual computes rotational corrections for the root and body joints via the logarithmic map of relative rotations, and the root translation correction by subtraction. Concatenating the 3D root rotation correction, 3D root translation correction, and 23×323× 3 body rotation corrections yields a 7575-dimensional residual vector. Root translation residuals are computed and clipped in meters, and are added to the anchor root translations in meters during pose composition. To regulate the magnitude of stochastic motion variations, we derive a step-specific residual gate from the pose history, future location condition, and deterministic anchor: kflow=[,k,Pa(^ka)],k=max⊙sigmoid(Γ(kflow)). gatheredc_k^flow= [h,\,f_k,\,P_a ( X^a_k ) ],\\[2.0pt] γ_k= γ_ ( (c_k^flow ) ). gathered (22) Here, PaP_a projects the anchor state and Γ predicts three gate values corresponding to root rotation, root translation, and body rotation. The vector max∈ℝ+3 γ_ _+^3 sets their maximum strengths. We further impose component-specific residual bounds. Let =concat(ρR3,ρt3,ρB69) ρ=concat( _R1_3, _t1_3, _B1_69), where ρR _R, ρt _t, and ρB _B bound the root rotation, root translation, and body rotation residuals, respectively. Denoting element-wise clipping to [−,][- ρ, ρ] by ℬB_ ρ, we define the step-specific regulation operator as: ℛk()=ℬ(bcast(k)⊙),R_k(x)=B_ ρ (bcast( γ_k) ), (23) where bcastbcast expands the three gate values over the corresponding 33, 33, and 6969 residual dimensions. The training target is bounded as ¯k∗=ℬ(k∗) r_k =B_ ρ(r_k ), while generated residuals are additionally modulated by ℛkR_k. This regulation constrains stochastic deviations from the anchor while allowing their magnitude to adapt to each future interaction step. For τ∼(0,1)τ (0,1) and k∼(,σr275)z_k (0, _r^2I_75), we construct the linear probability path: τ,k=(1−τ)k+τ¯k∗.x_τ,k=(1-τ)z_k+τ r_k . (24) The conditional velocity field is trained with ℒFM=[‖v(τ,k,τ,kflow)−(¯k∗−k)‖22],L_FM=E [ \|v (x_τ,k,τ,c_k^flow )- ( r_k -z_k ) \|_2^2 ], (25) where v is the conditional residual velocity field. The expectation is taken over valid future steps, flow times, and Gaussian prior samples. For training-time endpoint supervision, we obtain a direct residual estimate from an intermediate flow state as: ~kdirect=ℛk(τ,k+(1−τ)v(τ,k,τ,kflow)). r^direct_k=R_k (x_τ,k+(1-τ)v (x_τ,k,τ,c_k^flow ) ). (26) We additionally perform a differentiable NODEN_ODE-step Heun rollout from the Gaussian prior and denote the regulated terminal residual by ~kroll r^roll_k. Finally, kinematic pose reconstruction composes the direct and rollout residuals with the deterministic anchor through ComposeCompose, which applies rotational corrections via the exponential map and adds the root translation correction. For u∈direct,rollu∈\direct,roll\, we reconstruct ^ku=Compose(^ka,~ku),^u=^kuk=1K. X_k^u=Compose ( X_k^a, r_k^u ), X^u=\ X_k^u\_k=1^K. (27) The corresponding endpoint pose loss is: ℒu=ℰpose(^u,),u∈direct,roll,L_u=E_pose ( X^u,X ), u∈\direct,roll\, (28) where ℰposeE_pose uses the same masked 6D rotation, geodesic rotation, root translation, temporal velocity, and 3D joint position terms as the anchor objective. We further impose a residual alignment loss that encourages the direct endpoint residual to match the clipped target residual: ℒres=∑b∑k=1K(ℳX)b,k‖~b,kdirect−¯b,k∗‖2275∑b∑k=1K(ℳX)b,k.L_res= _b _k=1^K(M_X)_b,k \| r^direct_b,k- r_b,k \|_2^275 _b _k=1^K(M_X)_b,k. (29) The overall pose objective is: ℒpose=λFMℒFM+λdirectℒdirect+λrollℒroll+λresℒres.L_pose= _FML_FM+ _directL_direct+ _rollL_roll+ _resL_res. (30) At inference, we independently sample S residual priors and integrate each using NODEN_ODE Heun steps. The resulting regulated terminal residuals are composed with the deterministic anchor to produce S plausible future motion sequences, ^(q)q=1S\ X^(q)\_q=1^S. V Experiments V-A Experimental Settings TABLE I: Full-body pose forecasting results on the Coherent4D dataset. The left side reports pose forecasting conditioned on ground-truth future interaction locations, while the right side reports pose forecasting conditioned on predicted future interaction locations. MPJPE, PA-MPJPE, and Root Trans. are reported in millimeters, while Body Geo. is reported in degrees. Single uses the first candidate, whereas Best-5 reports all metrics for the candidate selected by sample-level MPJPE. Lower values are better. Bold and underline denote the best and second-best results, respectively. Model MPJPE↓ PA-MPJPE↓ Root Trans.↓ Body Geo.↓ MPJPE↓ PA-MPJPE↓ Root Trans.↓ Body Geo.↓ Single Best-5 Single Best-5 Single Best-5 Single Best-5 Single Best-5 Single Best-5 Single Best-5 Single Best-5 Health FIction 117.17 115.91 42.12 42.01 104.09 102.91 9.03 9.02 124.59 123.43 50.82 49.73 108.67 107.52 9.01 9.00 SkeletonDiffusion 166.32 137.39 53.18 52.19 158.85 138.47 10.55 10.45 184.97 155.55 55.01 54.20 172.37 151.19 10.89 10.81 SLD-HMP 421.56 88.38 44.79 32.50 403.30 74.40 8.76 5.63 402.76 92.15 48.66 33.22 381.66 89.37 8.56 5.72 HIGFlow 95.25 93.45 43.79 43.66 80.93 79.06 8.80 8.79 119.96 118.14 47.06 46.99 94.19 92.29 9.33 9.32 Bike Repair FIction 295.34 284.52 78.95 78.24 292.07 279.21 13.60 13.50 351.64 341.06 86.75 85.85 325.06 312.04 14.14 14.03 SkeletonDiffusion 453.52 370.26 113.22 109.63 405.49 331.24 17.69 17.44 469.82 398.82 115.09 111.77 408.17 343.07 17.89 17.74 SLD-HMP 269.06 234.95 74.34 70.03 261.00 218.63 11.97 11.12 358.20 314.88 83.82 81.76 311.54 296.01 12.39 11.76 HIGFlow 213.65 210.09 70.07 70.03 224.19 220.18 15.05 15.05 306.92 303.07 82.26 82.25 287.24 282.98 16.07 16.07 Cooking FIction 237.86 226.62 43.81 43.76 227.23 214.75 8.34 8.34 374.89 364.15 46.30 46.28 348.70 336.89 8.51 8.51 SkeletonDiffusion 310.39 260.00 51.43 51.25 278.05 229.97 11.42 11.41 421.75 382.35 53.09 52.91 377.64 339.03 11.89 11.88 SLD-HMP 184.42 167.19 43.41 41.21 174.33 156.25 7.67 7.25 358.92 342.11 44.86 43.13 332.74 313.26 7.83 7.53 HIGFlow 143.98 143.41 39.45 39.42 139.26 138.73 7.53 7.53 340.72 340.24 45.65 45.64 312.56 312.11 8.50 8.50 V-A1 Baselines We compare HIGFlow with two groups of baselines corresponding to the two forecasting stages. For interaction location forecasting, we consider FIction [19], Qwen3-VL-2B [58], V-JEPA 2.1 [59], Diff-IP3D [30], and MMTwin [32]. FIction is the closest prior method coupling interaction localization with pose forecasting, while Qwen3-VL and V-JEPA 2.1 provide semantic- and dynamics-oriented baselines, respectively. Following MMTwin [32], which extends Diff-IP2D [30] and MADiff [31] from 2D to 3D, we extend Diff-IP2D to predict continuous 3D interaction locations and refer to this variant as Diff-IP3D. MMTwin is reproduced in its 3D configuration with its prediction targets matched to Coherent4D. All methods are evaluated in the shared sample-local coordinate frame using the corrected USST evaluation implementation [4].11 1 See the USST erratum commit: https://github.com/oppo-us-research/USST/commit/beebdb963a702b08de3a4cf8d1ac9924b544abc4. For full-body pose forecasting, we compare HIGFlow with FIction, SkeletonDiffusion, and SLD-HMP under the same interaction location conditioning protocol. During training, all methods receive ground-truth future interaction locations. FIction follows its original location-conditioned formulation, whereas SkeletonDiffusion and SLD-HMP retain their original forecasting architectures with only the input and output interfaces adapted to our SMPL representation. We report both Single and Best-5 results following the evaluation protocol defined above. V-A2 Implementation Details Throughout this paper, V-JEPA refers to V-JEPA 2.1 ViT-Giant/384, and Qwen3-VL to Qwen3-VL-2B. For interaction location forecasting, we use M=64M=64 resampled V-JEPA memory tokens, set α=0.1α=0.1, and employ a three-layer coordinate decoder. The loss weights are (λADE,λS1,λCE)=(1.5,0.6,0.01)( _ADE, _S1, _CE)=(1.5,0.6,0.01). The base predictor and JEPA adapter are trained sequentially with AdamW and cosine learning rate schedules, using learning rate/warmup ratio pairs of (1×10−5,0.10)(1× 10^-5,0.10) and (3×10−6,0.05)(3× 10^-6,0.05), respectively, while the base predictor, structured encoders, and coordinate decoder remain frozen during adapter training. The pose history and future condition Transformers contain four and two layers, respectively, while MLPcMLP_c and MLPaMLP_a use two and three linear layers. The anchor predictor is pretrained with AdamW for up to 60 epochs, using a learning rate of 8×10−58× 10^-5, weight decay 10−410^-4, gradient clipping at 1.01.0, and an early stopping patience of six epochs. For Health and Cooking, we set (λ6D,λgeo,λrootgeo,λbodygeo,λvel,λjoint)=(1,0.1,0.5,1,1,10)( _6D, _geo, _rootgeo, _bodygeo, _vel, _joint)=(1,0.1,0.5,1,1,10), with λtrans=12 _trans=12, batch size 11, and dropout 0.100.10. For Bike Repair, we set λrootgeo=λbodygeo=0 _rootgeo= _bodygeo=0 and λtrans=10 _trans=10, with the remaining loss weights unchanged, a batch size of 1212, and dropout 0.150.15. Residual flow matching freezes the pose history encoder, future condition encoder, and anchor predictor. We use σr=0.003 _r=0.003, batch size 1616, and AdamW with weight decay 10−410^-4. The learning rates are 8×10−58× 10^-5 for Health and Cooking and 5×10−55× 10^-5 for Bike Repair. The loss weights are set to λFM=λdirect=λroll=1.0 _FM= _direct= _roll=1.0 and λres=0.5 _res=0.5. For Health and Cooking, we set max=(1.0,1.0,0.7) γ_ =(1.0,1.0,0.7) and (ρR,ρt,ρB)=(0.174533rad,0.12m,0.12rad)( _R, _t, _B)=(0.174533\,rad,0.12\,m,0.12\,rad). For Bike Repair, the corresponding settings are max=(0.8,1.0,0.35) γ_ =(0.8,1.0,0.35) and (ρR,ρt,ρB)=(0.25rad,0.25m,0.08rad)( _R, _t, _B)=(0.25\,rad,0.25\,m,0.08\,rad). The bound ρt _t is applied directly to each coordinate of the root translation residual in meters, without division by s. Both training and inference use NODE=4N_ODE=4 Heun steps. At inference, we generate S=5S=5 candidates for Best-5 evaluation. All experiments are conducted on six NVIDIA A100 GPUs. TABLE IV: Ablation of V-JEPA residual fusion for interaction location forecasting. LocEnc, EnvEnc, CoordDec, and sampled video frames are fixed. All values are reported in millimeters, and lower values are better. Domain V-JEPA injection ADE↓ ADE90↓_90 FDE↓ Health – 40.28 59.34 41.47 √ 40.21 59.52 41.44 Bike Repair – 81.18 134.10 94.03 √ 80.91 131.87 93.78 Cooking – 94.20 178.21 104.83 √ 93.46 178.20 104.26 V-B Quantitative Analysis We organized the quantitative analysis around the following questions. 1) Can HIGFlow jointly forecast interaction locations and full-body poses while outperforming specialized baselines? We evaluated the two stages of HIGFlow against methods designed specifically for the corresponding forecasting tasks. Table I reports interaction location forecasting results, while Table I reports full-body pose forecasting results under two location-conditioning settings: the left part uses ground-truth future interaction locations, whereas the right part uses predicted future interaction locations and represents the standard forecasting setting. For each task and conditioning setting, HIGFlow and all corresponding baselines are evaluated under the same evaluation protocol, enabling direct comparison of their forecasting performance. HIGFlow achieves leading performance in both interaction location forecasting and full-body pose forecasting, with particularly strong results on Cooking and Bike Repair and competitive performance on Health. Overall, HIGFlow demonstrates consistent performance on the interaction forecasting task. TABLE V: Domain-wise motion statistics in the shared sample-local coordinate frame. Loc. Disp. and Root Disp. measure the mean displacement between consecutive valid future steps for interaction locations and SMPL root translations, respectively. Values are reported in millimeters. Domain Loc. Disp. Root Disp. Spatial pattern Health 153.86 59.45 Compact spatial movement Bike Repair 293.53 195.71 Tool-centric reaching and repair Cooking 318.69 171.48 Broad object-centric movement TABLE VI: Effect of the V-JEPA memory token budget on interaction location forecasting. M denotes the number of resampled V-JEPA memory tokens. The shaded entries indicate the token budget selected for the final model. All values are reported in millimeters, and lower values are better. Bold denotes the best result for each domain and metric. Domain M ADE↓ ADE90↓_90 FDE↓ Health 16 40.23 59.61 41.46 32 40.23 59.49 41.46 64 40.21 59.52 41.44 128 40.23 59.44 41.42 Bike Repair 16 88.47 138.75 98.59 32 87.08 136.32 97.56 64 80.91 131.87 93.78 128 86.94 135.53 97.58 Cooking 16 93.36 178.45 104.20 32 93.04 177.63 104.33 64 93.46 178.20 104.26 128 93.36 178.03 104.22 TABLE VII: Ablation of input representations and output decoding for interaction location forecasting. LocEnc, EnvEnc, and CoordDec denote the location encoder, environment encoder, and coordinate decoder, respectively. A dash in the LocEnc or EnvEnc column indicates that the corresponding input is represented as text rather than processed by a dedicated encoder. A dash in the CoordDec column indicates that interaction locations are generated as text rather than decoded by a dedicated coordinate decoder. Gray rows include V-JEPA residual fusion. All values are reported in millimeters, with lower values indicating better performance and the best results highlighted in bold. Variant LocEnc EnvEnc CoordDec Health Bike Repair Cooking ADE↓ ADE90↓_90 FDE↓ ADE↓ ADE90↓_90 FDE↓ ADE↓ ADE90↓_90 FDE↓ Qwen3-VL – – – 57.53 89.54 82.04 90.71 156.55 103.81 103.95 199.20 112.22 w/o CoordDec √ √ – 57.11 89.31 59.71 99.64 161.61 113.42 113.87 222.51 119.68 w/o LocEnc – √ √ 42.57 63.41 44.88 87.78 143.79 98.04 97.29 182.26 105.15 w/o LocEnc + V-JEPA – √ √ 42.18 63.11 44.20 86.40 139.00 96.76 92.14 179.32 101.24 w/o EnvEnc √ – √ 42.17 65.26 43.32 87.65 137.12 97.31 100.13 184.76 106.65 w/o EnvEnc + V-JEPA √ – √ 41.86 63.68 42.83 85.94 133.97 95.19 96.24 182.77 103.39 HIGFlow (Full model) 40.21 59.52 41.44 80.91 131.87 93.78 93.46 178.20 104.26 2) What is the effect of V-JEPA residual fusion? We fixed the location encoder, environment encoder, coordinate decoder, frame input, and training configuration, and varied only the JEPA-guided residual adapter. When enabled, the resampled V-JEPA memory tokens are fused with the hidden states at the future step readout positions after the Qwen3-VL decoder forward pass. Table IV shows that V-JEPA residual fusion generally improves interaction location forecasting, with the most pronounced gains on Bike Repair. These results indicate that V-JEPA provides complementary motion-sensitive visual cues that help localize future locations beyond the semantic and contextual information captured by the base predictor. 3) Why do HIGFlow’s gains vary across domains? HIGFlow achieves larger improvements on Cooking and Bike Repair than on Health in both interaction location and pose forecasting. As shown in Table V, Cooking and Bike Repair exhibit substantially greater location and root displacements, providing richer geometric and dynamic cues for modeling future locations and full-body motion. In contrast, Health involves more compact spatial movements, for which the competing methods already perform relatively well, leaving less room for improvement. These observations suggest that HIGFlow benefits particularly from domains involving larger and more complex motion variations. TABLE VIII: Input ablation for interaction location forecasting across the three domains. Loc. and Env. denote location history and environment context, respectively. Frames denotes sampled video frames provided to Qwen3-VL. Gray rows include V-JEPA residual fusion, which retains features extracted by V-JEPA from the observed video. All values are reported in millimeters, with lower values indicating better performance and the best results highlighted in bold. Input Loc. Env. Frames Health Bike Repair Cooking ADE↓ ADE90↓_90 FDE↓ ADE↓ ADE90↓_90 FDE↓ ADE↓ ADE90↓_90 FDE↓ w/o Env. √ – √ 37.40 58.16 38.65 114.51 161.50 124.01 97.92 189.73 106.63 w/o Env. + V-JEPA √ – √ 37.32 58.27 38.68 91.79 141.30 101.61 94.60 186.08 104.71 w/o Loc. – √ √ 69.48 103.51 67.04 135.71 199.00 144.62 143.82 267.09 148.47 w/o Loc. + V-JEPA – √ √ 55.59 85.30 55.53 162.59 250.96 168.90 127.97 246.60 130.92 w/o Frames √ √ – 51.91 75.49 51.98 143.90 224.01 143.32 145.86 221.17 143.07 w/o Frames + V-JEPA √ √ – 50.61 75.93 51.78 117.27 175.46 125.37 119.60 225.82 123.65 HIGFlow (Full model) 40.21 59.52 41.44 80.91 131.87 93.78 93.46 178.20 104.26 4) How do location stage design choices affect interaction location forecasting? We examined three design aspects of the location stage: structured representation and coordinate decoding, input composition, and V-JEPA memory capacity. The results collectively support the use of structured multimodal cues, explicit continuous coordinate regression, and a moderate dynamic memory budget. a) Representation and coordinate decoding. We ablated the location encoder, environment encoder, and coordinate decoder while keeping the remaining components fixed. Without LocEnc or EnvEnc, the corresponding structured inputs are serialized as text. Without CoordDec, interaction locations are generated through the language interface rather than continuous coordinate regression. We also included Qwen3-VL as a baseline without these three components. As shown in Table VII, CoordDec provides the clearest and most consistent gains, highlighting the benefit of directly regressing continuous 3D coordinates. LocEnc and EnvEnc generally improve cross-domain consistency by preserving structured location and scene information, although Cooking shows mixed sensitivity. Overall, the full model achieves the most balanced performance across domains. (a) Bike Repair (b) Cooking Fig. 6: Qualitative pose forecasting comparison on Bike Repair and Cooking samples. For each example, the upper panel shows the observed SMPL pose history with synchronized exocentric source frames, while the lower panel presents future exocentric reference frames together with the corresponding ground-truth, HIGFlow, and FIction pose sequences. The exocentric frames are included only for visualization and are not used as inputs to the pose models. b) Input composition. We separately removed location history, environment context, or the sampled video frames provided to Qwen3-VL, while retaining the remaining inputs and coordinate decoding architecture. Each configuration was evaluated with and without V-JEPA residual fusion, which still provided features extracted from the observed video when Qwen3-VL received no frames. As shown in Table VIII, removing location history or frame input substantially degrades forecasting performance, while environment context is particularly important for Bike Repair. V-JEPA mitigates the degradation in most cases, suggesting that the visual context encoded by Qwen3-VL and the dynamic features provided by V-JEPA offer complementary information for future location forecasting. Fig. 7: Pose stage component ablation across Health, Bike Repair, and Cooking. We compared HIGFlow with variants without future interaction location conditioning, without residual flow matching, and without the deterministic anchor. Fig. 8: Effect of ODE integration steps on Best-5 pose forecasting. Regret is computed relative to the best tested step count for each domain and metric. The shaded band marks the selected setting of four integration steps. Lower values are better. c) V-JEPA token budget. We varied the number of resampled V-JEPA memory tokens over M∈16,32,64,128M∈\16,32,64,128\ while keeping all other settings fixed. Table VI shows that performance does not improve monotonically with increasing memory size. M=64M=64 achieves the best results on Bike Repair and remains competitive on Health and Cooking, providing the strongest overall trade-off across domains. We therefore adopted M=64M=64 as the default setting. 5) How do pose stage design choices and inference settings affect pose forecasting? We examined the contributions of location conditioning, the deterministic anchor, and residual flow matching, together with the effect of ODE integration steps. a) Location source ablation. The left part of Table I provides a controlled evaluation using ground-truth future interaction locations while keeping the same trained pose models, remaining inputs, and inference settings. Compared with the predicted-location setting on the right, ground-truth location conditioning generally improves pose forecasting accuracy, indicating that localization quality directly affects downstream pose prediction. HIGFlow consistently benefits from more accurate location guidance while maintaining strong overall performance under both location settings. b) Component ablation. We removed future interaction location conditioning, residual flow matching, or the deterministic anchor while keeping the observed pose history and Best-5 protocol fixed, yielding location-independent, anchor-only, and Flow-Matching-only variants, respectively. As shown in Fig. 7, removing future interaction location conditioning causes the largest overall degradation, demonstrating the importance of interaction locations as geometric guidance for full-body motion forecasting. The deterministic anchor improves structural stability, while residual flow matching captures additional motion variation and improves forecasting accuracy. Their combination achieves the strongest and most balanced performance across domains and metrics. c) Integration step analysis. We evaluated NODE∈1,2,4,8,16N_ODE∈\1,2,4,8,16\ under the same model configuration and Best-5 protocol. Because the pose metrics have different units and scales, we computed the percentage increase in error relative to the best tested result for each domain-metric pair and averaged it over all 12 pairs. As shown in Fig. 8, increasing the number of integration steps does not consistently reduce forecasting error. Four steps achieve the best result in 9 of the 12 comparisons and the lowest mean regret, whereas fewer steps provide insufficient integration accuracy and additional steps increase inference cost without consistent performance gains. We therefore used NODE=4N_ODE=4 by default. V-C Qualitative Analysis Fig. 6 compares HIGFlow with FIction in two representative domains, Bike Repair and Cooking. For each example, the upper part shows the observed pose history with synchronized exocentric frames, while the lower part presents future reference frames together with the ground truth and the predictions of both methods. The exocentric frames are used only for visualization. In Bike Repair, HIGFlow preserves the bent working pose during the early forecast and transitions to standing at a time closer to the ground truth, whereas FIction becomes upright too early. In Cooking, HIGFlow more often places the predicted interaction location in the correct workspace and produces body displacement and reaching poses that are closer to the reference motion. Together with the ablation results, these examples support the role of future interaction locations as geometric guidance, while the deterministic anchor and residual flow matching improve structural stability and refine articulated motion. The examples also reveal two limitations. In the later Bike Repair frames, interaction location errors may grow over the forecasting horizon and propagate through cascaded inference to the pose stage. As a result, the predicted body remains too upright instead of following the reference leaning and reaching motion. In Cooking, HIGFlow often identifies the correct interaction region, but the contacting hand, contact height, arm configuration, or torso orientation may still differ from the ground truth. These cases suggest that interaction location conditioning improves coarse spatial alignment but does not fully capture human intent, object affordances, or detailed contact constraints. VI Conclusion We introduced the Coherent4D dataset, which provides temporally aligned interaction location and full-body pose sequences in a shared coordinate system for continuous supervision and evaluation. Building on this formulation, we proposed the HIGFlow framework, a cascaded where-to-how model that couples future interaction location and full-body pose forecasting by using future interaction locations as geometric conditions for pose prediction. Within HIGFlow, Semantic-Dynamic Location Forecasting improves continuous interaction localization, while Hand-Conditioned Residual Flow Matching produces diverse yet structurally consistent poses. Experiments validate HIGFlow on both forecasting tasks, while ablations confirm the contributions of its key components. Future work will explore human intent modeling and contact-aware conditioning to enable more physically consistent forecasts over longer horizons. References [1] C. Li, G. Wu, G. Y. Chan, D. G. Turakhia, S. Castelo Quispe, D. Li, L. Welch, C. Silva, and J. Qian (2025) Satori: towards proactive ar assistant with belief-desire-intention user modeling. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, p. 1–24. Cited by: §I. [2] Y. Pei, R. Huang, M. Zha, G. Wang, P. Wang, Q. Kang, Y. Yang, and H. T. Shen (2025) AttentionAR: ar adaptation and warning for real-world safety via attention modeling and mllm reasoning. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology, p. 1–19. Cited by: §I. [3] A. Noormohammadi-Asl, S. L. Smith, and K. Dautenhahn (2025) To lead or to follow? adaptive robot task planning in human–robot collaboration. IEEE Transactions on Robotics 41, p. 4215–4235. Cited by: §I. [4] W. Bao, L. Chen, L. Zeng, Z. Li, Y. Xu, J. Yuan, and Y. Kong (2023) Uncertainty-aware state space transformer for egocentric 3D hand trajectory forecasting. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), p. 13656–13665. Cited by: §I, §I-A, §I-B, §V-A1. [5] M. Hatano, Z. Zhu, H. Saito, and D. Damen (2025) The invisible EgoHand: 3D hand forecasting through egobody pose estimation. arXiv preprint arXiv:2504.08654. Cited by: §I, §I-A. [6] R. Liu, Y. Huang, L. Ouyang, C. Kang, and Y. Sato (2025) SFHand: a streaming framework for language-guided 3d hand forecasting and embodied manipulation. arXiv preprint arXiv:2511.18127. Cited by: §I, §I-A. [7] M. Chen, Y. Wang, Z. Li, H. Bharadhwaj, Y. Chen, C. Qin, Z. Kou, Y. Tian, E. Whitmire, R. Sodhi, et al. (2025) Flowing from reasoning to motion: learning 3D hand trajectory prediction from egocentric human interaction videos. arXiv preprint arXiv:2512.16907. Cited by: §I, §I-A. [8] L. Seminara, G. M. Farinella, and A. Furnari (2026) Task graph maximum likelihood estimation for procedural activity understanding in egocentric videos. IEEE Transactions on Pattern Analysis and Machine Intelligence 48 (9), p. 10518–10534. Cited by: §I. [9] T. Liu and B. Bao (2026) Goal-guided prompting with adaptive modality selection for efficient assembly activity anticipation in egocentric videos. IEEE Transactions on Pattern Analysis and Machine Intelligence 48 (5), p. 5945–5962. Cited by: §I. [10] Y. Zheng, Y. Yang, K. Mo, J. Li, T. Yu, Y. Liu, C. K. Liu, and L. J. Guibas (2022) GIMO: gaze-informed human motion prediction in context. In European Conference on Computer Vision, p. 676–694. Cited by: §I, §I-A. [11] A. Avogaro, A. Toaiari, F. Cunico, X. Xu, H. Dafas, A. Vinciarelli, E. Li, and M. Cristani (2024) Exploring 3d human pose estimation and forecasting from the robot’s perspective: the HARPER dataset. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 5828–5835. Cited by: §I, §I-A. [12] Q. Jiang, B. Susam, J. Chao, and V. Isler (2024) Map-aware human pose prediction for robot follow-ahead. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 13031–13038. Cited by: §I, §I-A. [13] Z. Liu, S. Wu, S. Jin, S. Ji, Q. Liu, S. Lu, and L. Cheng (2022) Investigating pose representations and motion contexts modeling for 3d motion prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (1), p. 681–697. Cited by: §I. [14] X. Shu, L. Zhang, G. Qi, W. Liu, and J. Tang (2021) Spatiotemporal co-attention recurrent neural networks for human-skeleton motion prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence 44 (6), p. 3300–3315. Cited by: §I. [15] D. Wei, H. Sun, B. Li, J. Lu, W. Li, X. Sun, and S. Hu (2023) Human joint kinematics diffusion-refinement for stochastic motion prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, p. 6110–6118. Cited by: §I. [16] L. Chen, J. Zhang, Y. Li, Y. Pang, X. Xia, and T. Liu (2023) HumanMAC: masked motion completion for human motion prediction. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), p. 9510–9521. Cited by: §I. [17] Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le (2022) Flow matching for generative modeling. arXiv preprint arXiv:2210.02747. Cited by: §I, §IV-B2. [18] C. Curreli, D. Muhle, A. Saroha, Z. Ye, R. Marin, and D. Cremers (2025) Nonisotropic Gaussian diffusion for realistic 3D human motion prediction. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 1871–1882. Cited by: §I, §I-C. [19] K. Ashutosh, G. Pavlakos, and K. Grauman (2025) FICTION: 4D future interaction prediction from video. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 17613–17625. Cited by: §I, §I-A, §I-A3, §I-C2, §V-A1. [20] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, et al. (2020) The EPIC-KITCHENS dataset: collection, challenges and baselines. IEEE Transactions on Pattern Analysis and Machine Intelligence 43 (11), p. 4125–4141. Cited by: §I-A. [21] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al. (2022) Ego4D: around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 18995–19012. Cited by: §I-A. [22] Y. Li, Z. Cao, A. Liang, B. Liang, L. Chen, H. Zhao, and C. Feng (2022) Egocentric prediction of action target in 3D. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 20971–20980. Cited by: §I-A. [23] I. Fang, Y. Chen, Y. Wang, J. Zhang, Q. Zhang, J. Xu, X. He, W. Gao, H. Su, Y. Li, et al. (2024) EgoPAT3Dv2: predicting 3D action target from 2D egocentric vision for human-robot interaction. In 2024 IEEE International Conference on Robotics and Automation (ICRA), p. 3036–3043. Cited by: §I-A. [24] P. Kratzer, S. Bihlmaier, N. B. Midlagajni, R. Prakash, M. Toussaint, and J. Mainprice (2020) MoGaze: a dataset of full-body motions that includes workspace geometry and eye-gaze. IEEE Robotics and Automation Letters 6 (2), p. 367–373. Cited by: §I-A, §I-C. [25] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black (2023) SMPL: a skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, p. 851–866. Cited by: §I-A. [26] Z. Qi, S. Wang, W. Zhang, and Q. Huang (2024) Uncertainty-boosted robust video activity anticipation. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (12), p. 7775–7792. Cited by: §I-B. [27] S. A. Peirone, F. Pistilli, A. Alliegro, T. Tommasi, and G. Averta (2026) Hier-egopack: hierarchical egocentric video understanding with diverse task perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence 48 (2), p. 1917–1931. Cited by: §I-B. [28] L. Mur-Labadia, R. Martinez-Cantin, J. J. Guerrero, G. M. Farinella, and A. Furnari (2026) Integrating affordances and attention models for short-term object interaction anticipation. IEEE Transactions on Pattern Analysis and Machine Intelligence 48 (5), p. 5425–5441. Cited by: §I-B. [29] S. Liu, S. Tripathi, S. Majumdar, and X. Wang (2022) Joint hand motion and interaction hotspots prediction from egocentric videos. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 3272–3282. Cited by: §I-B. [30] J. Ma, X. Chen, J. Xu, and H. Wang (2025) Diff-IP2D: diffusion-based hand-object interaction prediction on egocentric videos. In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 4291–4298. Cited by: §I-B, §V-A1. [31] J. Ma, X. Chen, W. Bao, J. Xu, and H. Wang (2026) MADiff: motion-aware mamba diffusion models for hand trajectory prediction on egocentric videos. IEEE Transactions on Pattern Analysis and Machine Intelligence 48 (3), p. 3250–3267. Cited by: §I-B, §V-A1. [32] J. Ma, W. Bao, J. Xu, G. Sun, X. Chen, and H. Wang (2025) Novel diffusion models for multimodal 3D hand trajectory prediction. In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 2408–2415. Cited by: §I-B, §V-A1. [33] J. Ma, W. Bao, J. Xu, G. Sun, Y. Zheng, E. Zhang, X. Chen, and H. Wang (2026) Uni-Hand: universal hand motion forecasting in egocentric views. IEEE Transactions on Pattern Analysis and Machine Intelligence. Note: Early Access Cited by: §I-B. [34] Y. Zeng, X. Zhang, H. Li, J. Wang, J. Zhang, and W. Zhou (2023) X2^2-VLM: all-in-one pre-trained model for vision-language tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (5), p. 3156–3168. Cited by: §I-B. [35] S. Tian, R. Wang, H. Guo, P. Wu, Y. Dong, X. Wang, J. Yang, H. Zhang, H. Zhu, and Z. Liu (2026) Ego-r1: agentic chain-of-tool-thought for ultra-long egocentric video reasoning. IEEE Transactions on Pattern Analysis and Machine Intelligence. Note: Early Access Cited by: §I-B. [36] L. Chen, S. Lu, A. Zeng, H. Zhang, B. Wang, R. Zhang, and L. Zhang (2025) MotionLLM: understanding human behaviors from human motions and videos. IEEE Transactions on Pattern Analysis and Machine Intelligence. Note: Early Access External Links: Document Cited by: §I-B. [37] Y. Liu, D. Yang, M. Zheng, and M. Yang (2026) Anticipating object interactions via aggregation and distillation of spatio-temporal knowledge from vision language models. IEEE Transactions on Pattern Analysis and Machine Intelligence. Note: Early Access External Links: Document Cited by: §I-B. [38] C. Bao, J. Xu, X. Wang, A. Gupta, and H. Bharadhwaj (2024) HandsOnVLM: vision-language models for hand-object interaction prediction. arXiv preprint arXiv:2412.13187. Cited by: §I-B. [39] Z. Chang, X. Zhang, S. Wang, S. Ma, and W. Gao (2025) STAU: a spatiotemporal-aware unit for video prediction and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence 47 (9), p. 7916–7929. Cited by: §I-B. [40] L. Maes, Q. L. Lidec, D. Scieur, Y. LeCun, and R. Balestriero (2026) LeWorldModel: stable end-to-end joint-embedding predictive architecture from pixels. arXiv preprint arXiv:2603.19312. Cited by: §I-B. [41] T. Fernando, H. Gammulle, S. Sridharan, S. Denman, and C. Fookes (2024) Remembering what is important: a factorised multi-head retrieval and auxiliary memory stabilisation scheme for human motion prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence 47 (3), p. 1941–1957. Cited by: §I-C. [42] J. Tang, J. Hu, T. Liang, X. Lin, J. Sun, W. Zheng, and J. Lai (2026) Human motion prediction via continual prior compensation. IEEE Transactions on Pattern Analysis and Machine Intelligence 48 (5), p. 5131–5146. Cited by: §I-C. [43] S. Xu, Y. Wang, and L. Gui (2022) Diverse human motion prediction guided by multi-level spatial-temporal anchors. In European Conference on Computer Vision, p. 251–269. Cited by: §I-C. [44] J. Jeong, D. Park, and K. Yoon (2024) Multi-agent long-term 3D human pose forecasting via interaction-aware trajectory conditioning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 16975–16984. Cited by: §I-C. [45] T. Yu, Y. Lin, J. Yu, Z. Lou, and Q. Cui (2025) Vision-guided action: enhancing 3D human motion prediction with gaze-informed affordance in 3D scenes. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 12335–12346. Cited by: §I-C. [46] W. Zhu, X. Ma, D. Ro, H. Ci, J. Zhang, J. Shi, F. Gao, Q. Tian, and Y. Wang (2023) Human motion generation: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (4), p. 2430–2449. Cited by: §I-C. [47] M. Zhang, Z. Cai, L. Pan, F. Hong, X. Guo, L. Yang, and Z. Liu (2024) MotionDiffuse: text-driven human motion generation with diffusion model. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (6), p. 4115–4128. Cited by: §I-C. [48] Y. Yang, Z. Huang, C. Xu, and S. He (2026) Lagrangian motion fields for long-term motion generation. IEEE Transactions on Pattern Analysis and Machine Intelligence 48 (2), p. 1171–1184. Cited by: §I-C. [49] G. Xu, J. Tao, W. Li, and L. Duan (2024) Learning semantic latent directions for accurate and controllable human motion prediction. In European Conference on Computer Vision, p. 56–73. Cited by: §I-C. [50] G. Barquero, S. Escalera, and C. Palmero (2023) BeLFusion: latent diffusion for behavior-driven human motion prediction. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), p. 2317–2327. Cited by: §I-C. [51] J. Sun and G. Chowdhary (2024) CoMusion: towards consistent stochastic human motion prediction via motion diffusion. In European conference on computer vision, p. 18–36. Cited by: §I-C. [52] S. Tian, M. Zheng, and X. Liang (2025) PrediFlow: a flow-based prediction-refinement framework for real-time human motion prediction in human-robot collaboration. arXiv preprint arXiv:2512.13903. Cited by: §I-C. [53] K. Grauman, A. Westbury, L. Torresani, K. Kitani, J. Malik, T. Afouras, K. Ashutosh, V. Baiyya, S. Bansal, B. Boote, et al. (2024) Ego-Exo4D: understanding skilled human activity from first -and third-person perspectives. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 19383–19400. Cited by: §I-A, §I. [54] X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, and I. Misra (2022) Detecting twenty-thousand classes using image-level supervision. In European conference on computer vision, p. 350–368. Cited by: §I-A1. [55] A. Gupta, P. Dollar, and R. Girshick (2019) LVIS: a dataset for large vocabulary instance segmentation. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 5351–5359. Cited by: §I-A1. [56] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. (2024) The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §I-A3. [57] S. Shin, J. Kim, E. Halilaj, and M. J. Black (2024) WHAM: reconstructing world-grounded humans with accurate 3D motion. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 2070–2080. Cited by: §I-A4. [58] S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, et al. (2025) Qwen3-VL technical report. arXiv preprint arXiv:2511.21631. Cited by: §IV-B1, §V-A1. [59] L. Mur-Labadia, M. Muckley, A. Bar, M. Assran, K. Sinha, M. Rabbat, Y. LeCun, N. Ballas, and A. Bardes (2026) V-JEPA 2.1: unlocking dense features in video self-supervised learning. arXiv preprint arXiv:2603.14482. Cited by: §IV-B1, §IV-B1, §V-A1.