Paper deep dive
4D-HOF: Hand-Object Flow Matching for Feed-Forward 4D Interaction Reconstruction
Shiqi Li, Sean Cho, Yijie Li, Fengzhi Guo, Bowen Wen, Cheng Zhang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Existing methods for 4D hand-object reconstruction often rely on costly per-sequence optimization, while generative approaches typically synthesize interactions from random noise, which can lead to unstable interaction prediction. We introduce 4D-HOF, a feed-forward framework that reconstructs 4D hand-object interactions from coarse but informative estimates produced by vision foundation models. Concretely, we learn a conditional flow matching model that transports foundation-model-derived hand-object states toward an interaction manifold, allowing the model to correct errors in translation, rotation, and alignment in a feed-forward manner. A key advantage of our generative formulation is that it naturally enables test-time guidance within the transport process. Rather than applying a separate post-hoc optimization after reconstruction, we directly steer the evolving generative states using physical interaction constraints and observed 2D evidence, allowing the reconstruction to be refined as part of the generative process itself. By training the generative model on diverse datasets, 4D-HOF generalizes robustly to challenging in-the-wild scenarios. Experiments on out-of-domain benchmarks show that 4D-HOF achieves state-of-the-art performance, producing more stable and accurate 4D hand-object reconstructions.
Tags
Links
- Source: https://arxiv.org/abs/2610.08782v1
- Canonical: https://arxiv.org/abs/2610.08782v1
Trouble viewing inline? Open PDF directly →
Full Text
78,246 characters extracted from source content.
Expand or collapse full text
4D-HOF: Hand-Object Flow Matching for Feed-Forward 4D Interaction Reconstruction Shiqi Li Sean Cho Yijie Li Fengzhi Guo Bowen Wen Affiliation: NVIDIA[3pt] https://tamu-visual-ai.github.io/4D-HOF Cheng Zhang [3pt] Texas A&M University Abstract Existing methods for 4D hand-object reconstruction often rely on costly per-sequence optimization, while generative approaches typically synthesize interactions from random noise, which can lead to unstable interaction prediction. We introduce 4D-HOF, a feed-forward framework that reconstructs 4D hand-object interactions from coarse but informative estimates produced by vision foundation models. Concretely, we learn a conditional flow matching model that transports foundation-model-derived hand-object states toward an interaction manifold, allowing the model to correct errors in translation, rotation, and alignment in a feed-forward manner. A key advantage of our generative formulation is that it naturally enables test-time guidance within the transport process. Rather than applying a separate post-hoc optimization after reconstruction, we directly steer the evolving generative states using physical interaction constraints and observed 2D evidence, allowing the reconstruction to be refined as part of the generative process itself. By training the generative model on diverse datasets, 4D-HOF generalizes robustly to challenging in-the-wild scenarios. Experiments on out-of-domain benchmarks show that 4D-HOF achieves state-of-the-art performance, producing more stable and accurate 4D hand-object reconstructions. Figure 1: 4D Hand-Object Flow Matching (4D-HOF). Given a monocular video of a hand interacting with an unseen object, we learn a flow matching model for accurate and efficient 4D hand-object interaction reconstruction. Our method generalizes across diverse and challenging scenarios, including ambiguous depth (headset), occlusion (kettle), complex articulated hand poses (bar), and large object rotation (camera). Moreover, 4D-HOF is substantially faster than prior optimization-based methods due to our feed-forward design. 1 Introduction Reconstructing 4D hand-object interaction (HOI) from monocular video is a fundamental problem for understanding human behavior and enabling a variety of applications such as robot learning (He et al., 2025; Zhong et al., 2025; Paliwal et al., 2026), extended reality (Marchand et al., 2015), and human-computer interaction (Pavlovic et al., 2002). Despite recent progress (Fan et al., 2024; Wang et al., 2025b; Shi et al., 2026; Xu et al., 2026), recovering accurate hand-object geometry and interaction remains challenging due to diverse object geometry and pose, severe occlusion, and the inherent ambiguity of monocular observations. To address these challenges, existing approaches mainly fall into two lines. Optimization-based methods (Fan et al., 2024; Wang et al., 2025b; Shi et al., 2026; Aytekin et al., 2026b; Zhu et al., 2025) formulate HOI reconstruction as a per-sequence fitting problem, enforcing interaction constraints through manually designed objectives such as contact, rendering, and interaction stability. While effective, these methods are computationally expensive and often brittle under imperfect initialization or occlusion. Generative methods (Ye et al., 2023; Ye et al., 2024; Christen et al., 2024; Li et al., 2025; Ye et al., 2026) learn priors over HOI distributions from large-scale data, yet often start from random noise or rely on category-level priors with closed-set vocabularies, making it difficult to preserve instance-specific geometry and pose cues. This raises a key question: can we retain and learned priors of generative modeling while starting from informative observations rather than random noise? Vision foundation models provide a natural starting point. Recent large-scale pretrained models can estimate hand pose, object geometry, depth, and scene structure from monocular inputs (Carion et al., 2026; Oquab et al., 2023; Piccinelli et al., 2025; Lin et al., 2025b; Wen et al., 2024; Pavlakos et al., 2024; Potamias et al., 2025). These models produce coarse yet informative estimates that capture rich geometric cues. Indeed, recent works have begun to incorporate foundation models into HOI reconstruction (Liu et al., 2025b; Wang et al., 2026; Lin et al., 2025a; Xu et al., 2026). However, they typically treat these estimates as initialization and rely on downstream heuristic optimization to enforce consistency, which reintroduces inefficiency and instability. In this work, we instead couple foundation-model perception directly with generative modeling. We use them to define the source distribution of a conditional generative transport process. Starting from coarse but informative hand-object states, we learn how these states should evolve toward physically plausible interactions. Specifically, we adopt flow matching and stochastic interpolates (Liu et al., 2022b; Shi et al., 2023; Albergo et al., 2025) to construct a generative bridge that transports the initial estimates toward an HOI manifold. To ground refinement in observed evidence and exploit interaction dynamics, the bridge is conditioned on both the input video frames and the temporal context of the initial 4D hand-object sequence. This formulation enables unified correction of geometry, pose, and alignment in a feed-forward manner, while preserving instance-specific cues from the input. Importantly, the generative formulation also gives us a natural mechanism for test-time control to further enhance fine-grained physical plausibility and consistency with the 2D observation. Our method, termed 4D-HOF, is a feed-forward framework that takes foundation-model estimates as input and jointly refines hand-object interactions. We summarize our contributions as follows: • We formulate 4D HOI reconstruction as conditional generative transport from foundation-model-derived hand-object state initialization to physically plausible interactions, avoiding random noise as initialization to flow matching and costly per-sequence optimization. • We design a conditional flow matching model that jointly refines hand-object geometry, pose, and alignment using visual, geometric, and temporal cues. • We introduce test-time guidance enabled by the generative transport process, which steers evolving states with image, interaction, and temporal constraints without separate post-hoc optimization. • Experiments on 4D HOI benchmarks and challenging in-the-wild scenarios demonstrate state-of-the-art performance in hand estimation, object reconstruction, interaction consistency, and temporal stability, while running over 5 times faster than prior methods. 2 Related Work Hand-Object Reconstruction. Early image-based methods jointly regress hand and object meshes using parametric hand priors and contact constraints (Hasson et al., 2019; Ye et al., 2022). Video-based methods typically optimize each sequence with hand-crafted geometric and interaction objectives (Hasson et al., 2021; Wen et al., 2023; Fan et al., 2024; Wang et al., 2025b; Lin et al., 2025a; Shi et al., 2026; Aytekin et al., 2026a; On et al., 2025; Liu et al., 2025a; Aboukhadra et al., 2026), incurring high computational costs and sensitivity to initialization. Feed-forward approaches accelerate inference (Chen et al., 2026b) but emphasize geometry without explicitly modeling interaction dynamics. Our method instead learns an interaction prior that maps coarse estimates to physically plausible interactions for feed-forward reconstruction. Foundation Models for 3D Understanding. Foundation models provide strong priors for segmentation (Carion et al., 2026), visual representations (Oquab et al., 2023), monocular and multi-view geometry (Kong et al., 2026; Piccinelli et al., 2025; Lin et al., 2025b; Wen et al., 2025; Wen et al., 2026; Wang et al., 2025a), object pose (Wen et al., 2024), hand reconstruction (Pavlakos et al., 2024; Potamias et al., 2025), and 3D generation (Chen et al., 2026a; Xiang et al., 2026). Recent HOI reconstruction pipelines use these models for initialization (Wang et al., 2026; Lin et al., 2025a; Xie et al., 2026), but independently estimated components often have inconsistent scale, depth, and orientation. Existing methods resolve these discrepancies through additional optimization (Shi et al., 2026; Aytekin et al., 2026a); we learn to jointly refine the estimates without per-sequence optimization. Generative Priors for Hand-Object Interaction. Generative models capture HOI distributions through diffusion (Ho et al., 2020; Christen et al., 2024; Li et al., 2025) or language modeling (Huang et al., 2025a), primarily for interaction synthesis. Complementary denoising methods learn spatio-temporal priors to refine noisy HOI trajectories (Zhou et al., 2022; Liu and Yi, 2024; Luo et al., 2024), demonstrating the potential of learned priors for HOI correction. For reconstruction, Diff-HOI (Ye et al., 2023) and G-HOP (Ye et al., 2024) incorporate generative priors, but they rely on noise-driven generation or category-level priors with closed-set vocabulary. Most closely related, CHOIR (Xu et al., 2026) learns a flow-matching prior to correct relative hand-object 1D depth displacement along the viewing ray from initializations, while contact-aware optimization refines the full interaction. Building on flow matching and stochastic interpolants (Liu et al., 2022b; Shi et al., 2023; Albergo et al., 2025), we instead learn a conditional bridge that jointly refines entire HOI states, making the learned prior the primary reconstruction mechanism, complemented by test-time guidance. Agentic Real-to-Sim. Recent advances in agentic AI motivate its integration into 3D reconstruction pipelines. AgentSTAR (Mazur et al., 2026) couples visual reasoning with numerical optimization in a render-and-compare loop to recover object geometry, articulation, and motion, but does not explicitly model hand–object interactions. Inspired by GPT-6 Astra’s capabilities, we conduct pilot experiments on agentic HOI reconstruction, detailed in the Appendix B.4. Our preliminary findings suggest that this approach is promising but incurs substantial runtime and costs. Our proposed feed-forward method offers an efficient reconstruction module that agents could use to accelerate this process. 3 Approach 3.1 Problem Formulation 4D HOI reconstruction. Given an N-frame monocular RGB video ℐ=ii=1NI=\I^i\_i=1^N, where a hand interacts with an object of arbitrary shape, our goal is to reconstruct the underlying 4D hand-object interaction. We aim to reconstruct the object mesh om_o with corresponding per-frame 6-DoF poses i=(o,o)o^i=(R^o,t^o), where o∈SO(3)R^o∈ SO(3) and translation o∈ℝ3t^o ^3. For the hand, we estimate the per-frame MANO (Romero et al., 2022) parameters, consisting of shape β and pose i=(h,h)h^i=(R^h,t^h). We denote the full interaction trajectory as =ii=1NS=\s^i\_i=1^N, where i=[i,i]s^i=[h^i,o^i]. Figure 2: 4D-HOF overview. Given a monocular video, we construct initial hand-object states by leveraging foundation models, including (a) contextual scene parsing, (b) object reconstruction, (d) hand reconstruction, and (c) depth alignment for recovering metric geometry, producing coarse HOI initialization. We then formulate HOI refinement as a conditional generative bridge matching problem that takes the HOI initialization together with RGB and 3D cues as input and outputs the refined HOI reconstruction. At inference time, test-time guidance adjusts the evolving states to better satisfy physical and image-space constraints. Flow matching for HOI reconstruction. Flow matching (Lipman et al., 2022; Liu et al., 2022a) formulates generative modeling as learning a continuous transport from a source distribution p(0)p(x_0) to a target data distribution p(1)p(x_1). Let 0∼p(0)x_0 p(x_0) and 1∼p(1)x_1 p(x_1). A stochastic interpolant defines intermediate states t=(1−t)0+t1+γ(t)ϵ,x_t=(1-t)x_0+tx_1+γ(t) ε, where t∈[0,1]t∈[0,1], ϵ∼(0,) ε (0,I), and γ(t)γ(t) is a noise schedule. Flow matching learns a velocity field θ(,t)v_θ(x,t) such that the trajectory follows d(t)/dt=θ((t),t)dx(t)/dt=v_θ(x(t),t), transporting samples from p(0)p(x_0) to p(1)p(x_1). In practice, θv_θ can be a neural network (e.g., DiT (Peebles and Xie, 2023)) trained to match the conditional drift induced by the interpolate via the following objective: ℒθ _θ =t,0,1,ϵ[‖θ(t,t)−(1−t)/(1−t)‖22]. =E_t,x_0,x_1, ε [ \|v_θ(x_t,t)-(x_1-x_t)/(1-t) \|_2^2 ]. (1) In the context of HOI, the target distribution p(1)p(x_1) corresponds to physically plausible hand-object interactions, which can be learned from large-scale HOI datasets (Wang et al., 2024; Yang et al., 2022; Swamy et al., 2023; Hampali et al., 2020). While several recent works adopt generative models for HOI, they often start from random noise (Ye et al., 2022; Ye et al., 2026), depend on object templates (Ye et al., 2024), or target the task of grasp synthesis (Wu et al., 2026) instead of full 4D HOI reconstruction. Bridge matching. A key limitation of vanilla flow matching is that it assumes a fixed Gaussian prior, whose stochastic nature can hinder physical accuracy in many deterministic tasks. To address this, the Bridge Matching (Albergo et al., 2025) framework alleviates this by learning transport between arbitrary distributions π0 _0 and π1 _1. Given 0∼π0x_0 _0 and 1∼π1x_1 _1, the trajectory can be written as dt=θ(t,t)dt+σ(t)dt, dx_t=v_θ(x_t,t)\,dt+σ(t)\,dW_t, (2) where θv_θ denotes the learnable drift and σ(t)σ(t) controls the noise level. A common instantiation is the Brownian bridge with σ(t)=σt(1−t)σ(t)=σ t(1-t). The velocity field is trained by matching the drift: ℒbridge=t,0,1[‖θ(t,t)−(1−t)/(1−t)‖22]. _bridge=E_t,x_0,x_1 [ \|v_θ(x_t,t)-(x_1-x_t)/(1-t) \|_2^2 ]. (3) When σ(t)=0σ(t)=0, Eq (3) reduces to deterministic flow matching. By replacing the Gaussian prior with a more plausible initial distribution, it reduces the stochasticity inherent in noise-driven generation. A key remaining question therefore is the choice of the source distribution. In this work, we define it using foundation-model-derived hand and object states, which will be described in the next section. 3.2 Hand and Object States Initialization The goal is to leverage foundation models to obtain initial hand and object geometry and their per-frame interaction initialization 0i=[0i,0i]s_0^i=[h_0^i,o_0^i], where i indexes video frames. For brevity, we will omit the frame index i. These estimates provide coarse yet informative hand and object pose, which serve as the starting point of our conditional generative bridge. While foundation models has made significant progress, they treat hand, object, and the context separately and do not consider their intricate interactions. To address this, we introduce the following methods for reconstructing hand, object, and their interaction. Please refer to Appendix for the detailed process. Contextual scene parsing. We first perform holistic 3D scene understanding to reduce ambiguity in monocular reconstruction and generate outputs for object and hand states estimation. Specifically, for each frame, we obtain the object mask iM_i via vision language grounding (Comanici et al., 2025) and segmentation models (Carion et al., 2026). We also estimate depth iD_i and camera parameters using MoGe-3 (Kong et al., 2026), which provide geometric cues for both hand and object reconstruction. These outputs establish spatial consistency across frames. Figure 2-(a) illustrates the main idea. Hand reconstruction and state estimation. For hand initialization, we employ WiLoR (Potamias et al., 2025) to estimate per-frame MANO (Romero et al., 2022) parameters and 2D keypoints, with hand bounding boxes extracted from masks. While WiLoR provides reliable hand articulation and rotation estimates, its translation prediction is limited by the weak-perspective camera assumption, which cannot accurately recover metric camera-space depth. To address this, we solve a lightweight closed-form least-squares to estimate the hand translation. Figure 2-(d) shows the pipeline. Object reconstruction and state estimation. We estimate a canonical object mesh om_o and temporally consistent per-frame poses ii=1N\o^i\_i=1^N. We sample sparse keyframes with diverse viewpoints and reconstruct the mesh in normalized space using pre-trained multi-view generative models (Comanici et al., 2025; Zhao et al., 2025). Compared with single-view generation (Chen et al., 2026a), this reduces ambiguity under occlusion and improves texture quality. We then align the object with the scene using depth cues and an enhanced FoundationPose (Wen et al., 2024), as shown in Figure 2-(b). The resulting states provide plausible structural alignment but remain susceptible to occlusion-induced instability and generative artifacts, motivating subsequent bridge matching refinement. 3.3 4D HOI Flow Matching After we obtained initialized hand-object states 0s_0 from the input video sequence, we learn a conditional generative model that transports these states toward physically plausible interactions. We parameterize the conditional velocity field θ(t,t)v_θ(s_t,t) using a neural network that takes as input the current state ts_t, flow time t, and multi-modal conditions derived from the input video. Conditional bridge matching. As discussed in Section 3.1, directly applying Eq. (1) to HOI reconstruction is challenging due to the large solution space and ambiguity of monocular observations. To address this issue, we inject an initialization state 0s_0 through the bridge matching formulation in Eq. (3), which guides the transport process toward physically plausible hand-object configurations. We further encode geometric structure by representing both the hand and object as point clouds. Specifically, we sample points from the object mesh om_o and use MANO vertices parameterized by shape coefficients β for the hand representation. These point clouds are processed using a Point Transformer V3 backbone (Wu et al., 2024) to capture spatial relationships between the hand and object. To incorporate visual context, we extract image features from each frame using a pretrained DINOv2 encoder (Oquab et al., 2023). For efficient 4D modeling, we aggregate multi-frame features and feed them into a DiT backbone for temporal reasoning and conditional flow prediction. Detailed architecture is shown in Figure S1. The bridge matching objective in Eq. (3) can be written as: ℒhof=0,1,t[‖θ(t,t,o,,ℐ)−(1−t)/(1−t)‖22], _hof=E_s_0,s_1,t [ \|v_θ(s_t,t,m_o, β,I)-(s_1-s_t)/(1-t) \|_2^2 ], (4) where om_o denotes the object mesh, β the hand shape, and ℐI the sequence of input images. This objective learns to transport the initialized states 0s_0 toward the target interaction distribution. Training objectives. We introduce three auxiliary losses to improve reconstruction quality: 3D position loss ℒ3DL_3D, interaction loss ℒinterL_inter, and regularization loss ℒregL_reg. We apply regression-based (Pavlakos et al., 2024) 3D position losses ℒ3DL_3D to the terminal estimate 1s_1, recovered from the generated trajectory since flow matching predicts velocities rather than states. The 3D loss consists of three components: (1) a hand loss supervising the 3D MANO joints 3DJ_3D, (2) an object loss supervising the transformed object point cloud P, and (3) a relative loss supervising the spatial relationship between the hand and the object through their relative Euclidean distances: ℒ3D=‖3D−3D∗‖1+‖−∗‖1+‖(,3D)−(∗,3D∗)‖1, _3D= \|J_3D-J^*_3D \|_1+ \|P-P^* \|_1+ \|D(P,J_3D)-D(P^*,J_3D^*) \|_1, (5) where P is the point cloud sampled from the transformed object mesh om_o, (,3D)D(P,J_3D) is the distance matrix for each transformed object point-hand joint pair. The asterisk denotes the ground truth. To encourage physically plausible interactions, we further incorporate contact and penetration-based constraints (Jiang et al., 2021), denoted by ℒinterL_inter, between the hand mesh and object point cloud: ℒinter=∑j∈Ωd(pj,ℋ)+∑j∈d(pj,ℋ), _inter=Σ _j∈ d(p_j,H)+Σ _j d(p_j,H), (6) where d(pj,ℋ)d(p_j,H) is the distance between object point pjp_j to MANO mesh surface ℋH, Ω is the contact set derived from ground-truth pose and P is the object point set that penetrates into the hand mesh. The first term encourages contact at ground-truth contact regions, while the second penalizes penetration. In addition, we empirically observe that the hand estimates provided by the initialization are accurate in most cases. Therefore, the refinement process should make only subtle adjustments to improve physical plausibility, rather than performing unconstrained optimization that may introduce undesirable artifacts. To this end, we introduce a regularization term, ℒregL_reg, on hand articulation related channels θ,handv_θ,hand to preserve the initialized hand motion during refinement: ℒreg=‖θ,hand(t,t,o,,ℐ)‖22. _reg= \|v_θ,hand(s_t,t,m_o, β,I) \|_2^2. (7) The overall training objective combines the bridge matching loss with the auxiliary terms: ℒtotal=ℒhof+ℒ3D+ℒinter+ℒreg. _total=L_hof+L_3D+L_inter+L_reg. (8) Test-time guidance. An additional benefit of our bridge matching formulation is that the continuous transport process provides a flexible interface for test-time control. During inference, task-specific constraints can be directly incorporated by steering intermediate states along the learned transport trajectory, without performing expensive per-sequence optimization. This allows us to efficiently impose additional geometric and interaction constraints, such as contact, temporal consistency, and image-space alignment, while preserving the interaction prior learned by the model. Formally, we incorporate the following lightweight test-time guidance G(t)G(s_t) into the bridge matching process: t=t−∇tG(t). _t=s_t- _s_tG(s_t). (9) The guidance function G combines three complementary objectives: silhouette guidance GsilG_sil for image alignment, interaction guidance GinterG_inter for physical correctness, and temporal smoothness guidance GtempG_temp for motion consistency. The silhouette guidance GsilG_sil aligns the projected object geometry with the observed segmentation. We sample object surface points and project them onto the image plane to obtain U. Let ℬB denote sampled boundary pixels of the visible object mask, and let ℳM be the union of the object mask and the dilated hand mask. The guidance is defined as Gsil=12||∑∈ρ(Dℳ()+DΨ())+12|ℬ|∑∈ℬρ(D()), G_sil= 12|U| _u ρ\! (D_M(u)+D_ (u) )+ 12|B| _b ρ\! (D_U(b) ), (10) where Dℳ()D_M(u) is the distance from u to the admissible region ℳM, DΨ()D_ (u) projections outside the image, and D()D_U(b) is the distance from b to its nearest projected point in U. Here, ρ denotes the smooth L1 penalty. The first term penalizes projections outside the object or hand-occluded regions, while the second term encourages coverage of the visible object boundary. Our interaction guidance GinterG_inter follows the interaction loss ℒinterL_inter used during training, encouraging contact while penalizing interpenetration. Since ground-truth contact annotations are unavailable at test time, we use the learned contact prediction model HACO (Jung and Lee, 2026) to infer the contact index set Ω . This allows guidance to adapt to diverse hand-object interaction patterns without relying on manually predefined fixed contact regions (Fan et al., 2024; Hasson et al., 2019). To further reduce temporal jitter, we introduce temporal smoothness guidance GtempG_temp that penalizes the velocities and accelerations of the hand and object: Gtemp=‖∇i‖22+‖∇2i‖22, G_temp=\| ^i\|_2^2+\|∇^2s^i\|_2^2, (11) where ∇ and ∇2∇^2 denote first- and second-order temporal finite differences, respectively. Translational increments are computed as differences between consecutive positions, while rotational increments are represented by the axis-angle vectors of the corresponding relative rotations. Second-order differences are computed from consecutive increments. We apply guidance only during the later stages of flow-matching integration, when the predicted state is already close to a plausible interaction and requires only minor corrections. By concentrating refinement on these later steps, the guidance makes targeted adjustments that improve image alignment, interaction validity, and temporal consistency while introducing limited computational overhead. 3.4 Data Preparation and Model Training We train our model on multiple large-scale HOI datasets, including HO-Cap (Wang et al., 2024), OakInk (Yang et al., 2022), and SHOWMe (Swamy et al., 2023), which covering diverse interactions and object categories. Training excludes all evaluation benchmarks, and we use identical model weights across test datasets to assess cross-dataset generalization. To efficiently leverage the available data and prevent overfitting to specific dataset pattern, we construct training pairs by perturbing the ground-truth HOI states 1s_1 with a randomly sampled noise (see Appendix for details). Unlike conventional generative modeling that transports samples from pure Gaussian noise, our formulation learns a conditional transport process between imperfect yet structured initialization states and physically plausible HOI states. This matches the practical inference setting, where modern foundation models already provide strong but imperfect estimates. 4 Experiments 4.1 Setup Datasets. We evaluate our approach on two standard benchmarks: HO3D-v3 (Hampali et al., 2020) and DexYCB (Chao et al., 2021). HO3D-v3 features complex interactions resulting in severe occlusions, and DexYCB emphasizes tabletop grasping across diverse subjects. Following AGILE’s evaluation protocol (Shi et al., 2026), we use 18 HO3D-v3 and 20 DexYCB sequences; we refer readers to AGILE for sequence selection and evaluation details. Baselines. We compare with state-of-the-art monocular HOI reconstruction methods: HOLD (Fan et al., 2024) jointly optimizes implicit hand-object fields; MagicHOI (Wang et al., 2025b) improves object geometry with novel-view synthesis priors. AGILE (Shi et al., 2026) combines agentic initialization with heuristic optimization, and CHOIR (Xu et al., 2026) integrates foundation-model initialization, learned relative-depth correction, and contact-aware joint optimization. Metrics. We evaluate five complementary aspects. Hand accuracy: mean per-joint position error (MPJPE) [m] for local articulation. Object accuracy: bidirectional Chamfer distance (CD) [cm2] and F-scores at 5/10 m (F5/F10) [%] after ICP alignment of scale, rotation, and translation. Interaction consistency: hand-relative Chamfer distance (CDh) [cm2] (Fan et al., 2024), computed in the hand-root frame; interpenetration volume (IV) [cm3], measuring hand-object overlap; and contact accuracy (CA) [%] (Jiang et al., 2021), the fraction of frames whose contact states match the ground truth. Temporal stability: hand-joint trajectory jerk magnitude (JM) [m/s3], with lower values indicating smoother motion. Robustness: per-sequence success rate (SR) [%] (Shi et al., 2026), requiring both hand and object reconstruction to complete without failure. 4.2 4D HOI Reconstruction Comparison We compare 4D-HOF against state-of-the-art methods on HO3D-v3 (Table 1) and DexYCB (Table 3), and show qualitative results in Figure 3 and the supplementary video. Table 1: Results on HO3D-v3. Our method shows strong improvements on interaction metrics, which are often underreported in previous methods (Wang et al., 2025b; Shi et al., 2026). Hand and object accuracy are determined by foundation models and our method performs on par with AGILE (code not available). Method Interaction Hand Object Robustness CDh ↓ IV ↓ CA ↑ JM ↓ MPJPE ↓ CD ↓ F5 ↑ F10 ↑ SR ↑ HOLD 18.66 18.76 89.50 10.43 22.09 1.11 81.75 92.42 100.0 MagicHOI 21.81 17.41 83.00 21.66 7.38 0.90 76.74 91.59 83.3 AGILE 15.81 – – – 3.92 0.27 86.63 97.77 100.0 CHOIR 42.37 22.90 90.99 0.92 21.12 0.84 63.87 90.29 94.4 4D-HOF (ours) 5.98 13.44 97.27 1.61 4.40 0.28 89.11 97.13 100.0 Table 2: Results on DexYCB (Chao et al., 2021) dataset. Method Hand Object Interaction Robustness MPJPE ↓ CD ↓ F5 ↑ F10 ↑ CDh ↓ SR ↑ HOLD 30.86 19.30 33.20 54.94 170.90 45.0 MagicHOI 21.20 2.05 45.67 67.14 661.90 25.0 AGILE 19.06 0.52 83.21 95.43 94.60 100.0 CHOIR 27.24 0.84 79.46 93.09 134.74 100.0 4D-HOF (ours) 6.84 0.45 83.62 95.81 48.85 100.0 Table 3: Computational cost. Method Preproc. Optim. Total 4D-HOF ∼ 23 min ∼ 1 min ∼ 24 min HOLD ∼ 65 min ∼ 24 h ∼ 25.1 h MagicHOI ∼ 25 min ∼ 2 h ∼ 2.4 h AGILE 21–29 min ∼ 1.75 h 2.1–2.2 h CHOIR ∼ 11min ∼ 19min ∼ 30min Results on HO3D-v3. 4D-HOF attains the lowest hand-to-object Chamfer distance and interpenetration volume among all methods, improving over the next best baseline by 62.2% in CDh. The jerk magnitude is reduced to a level 6.4×6.4× lower than HOLD (Fan et al., 2024) and 13.4×13.4× lower than MagicHOI (Wang et al., 2025b), confirming that our learned interaction prior produces temporally stable trajectories without explicit smoothness regularization. Our method further achieves the highest contact ratio among reporting methods and the strongest F5 among all baselines. Results on DexYCB. DexYCB contains diverse subjects performing tabletop grasps, where prior baselines suffer from large scale and translation errors. 4D-HOF yields a particularly pronounced margin across all the metrics comparing with other baselines. Figure 3: Qualitative results on HO3D-v3 (Hampali et al., 2020) and DexYCB (Chao et al., 2021). 4D-HOF consistently outperforms HOLD (Fan et al., 2024) and MagicHOI (Wang et al., 2025b) in preserving accurate geometry and hand-object interactions. Please see our supplementary video for detailed comparisons. Computational cost. We compare the inference time in Table 3. Existing methods rely on expensive test-time optimization and typically require hours to process a 200-frame sequence. In contrast, 4D-HOF performs reconstruction through a feed-forward generative transport process, reducing the refinement stage to approximately one minute while maintaining competitive reconstruction quality. 4.3 Ablation Study and Analysis We conduct ablation studies on HO3D-v3 (Hampali et al., 2020) to validate the design choices. Table 4: Ablation of vision foundation models. Method Variants Interaction Hand CDh ↓ IV ↓ CA ↑ JM ↓ MPJPE ↓ 4D-HOF (full model) 5.98 13.44 97.27 1.61 4.40 (a) SAM3D object recon. 262.19 9.87 64.40 1.75 6.09 (b) FPose default tracking 15646.78 9.39 75.74 2.07 5.00 (c) VFM only 46.58 11.06 78.07 3.95 4.03 (d) VFM + guidance 41.75 11.89 81.53 3.71 4.03 Table 5: Ablation of the bridge matching model. Method Variants Interaction Hand CDh ↓ IV ↓ CA ↑ JM ↓ MPJPE ↓ 4D-HOF (full model) 5.98 13.44 97.27 1.61 4.40 (a) w/o temporal design 18.93 14.23 95.64 4.26 4.15 (b) w/o explicit encoding 11.92 14.76 96.41 1.62 4.40 (c) w/o object optimization 90.49 9.63 62.23 2.31 5.06 (d) w/o hand optimization 12.34 13.05 96.02 3.95 4.03 (e) w/o guidance 8.03 13.39 96.62 1.71 4.37 How to use vision foundation models? We first study different strategies for constructing the initial hand-object states (Table 5). (a) Replacing our multi-view object generation with SAM3D (Chen et al., 2026a) significantly degrades interaction quality, showing the importance of temporally consistent object reconstruction. (b) Using FoundationPose (Wen et al., 2024) in pure tracking mode also fails under severe occlusions, where a single mistracked frame can corrupt the entire sequence. (c) Directly using vision foundation model outputs without our generative bridge yields poor alignment and contact. (d) Adding guidance (but without bridge matching model) offers only modest improvements, suggesting the need for a learned interaction prior to refine independently estimated HOI states. How to design the HOI flow matching model? We next analyze the design of the conditional bridge (Table 5). (a): Removing temporal modeling produces unstable 4D trajectories, suggesting that independent frame-wise refinement cannot capture interaction dynamics. (b): Removing the explicit hand-object structural encoding consistently degrades interaction metrics, highlighting the importance of explicitly modeling relative hand-object geometry. (c) and (d): Optimizing only the hand or only the object also leads to worse interaction quality, indicating that hand and object states must be refined jointly to resolve their mutual ambiguity. (e): Removing test-time guidance degrades contact and interaction quality, showing that guidance provides effective test-time control toward geometrically and physically consistent HOI states. Notably, these variants leave the object geometry unchanged, while contact quality and physical interaction degrade significantly. Table 6: Ablation studies on loss functions. ℒL Variants Interaction Hand CDh ↓ IV ↓ CA ↑ JM ↓ MPJPE ↓ 4D-HOF 5.98 13.44 97.27 1.61 4.40 w/o ℒ3DL_3D 7.68 15.94 96.27 2.93 3.98 w/o ℒinterL_inter 6.02 13.78 97.00 1.63 4.60 w/o ℒregL_reg 6.71 13.78 97.15 1.10 5.64 Table 7: Ablation studies on guidance terms. G Variants Interaction Hand CDh ↓ IV ↓ CA ↑ JM ↓ MPJPE ↓ 4D-HOF 5.98 13.44 97.27 1.61 4.40 w/o GsilG_sil 7.51 13.85 97.06 1.72 4.75 w/o GinterG_inter 7.81 16.00 95.07 1.63 4.74 w/o GtempG_temp 6.35 14.53 97.29 1.91 4.74 How to supervise 4D-HOF training? We ablate each auxiliary losses in Table 7. Removing ℒ3DL_3D causes severe degradation across most metrics, showing that explicit 3D supervision is critical for HOI reconstruction. Removing ℒinterL_inter reduces contact quality and hand pose accuracy, indicating that the contact loss encourages meaningful physical interaction. Removing ℒregL_reg leads to the largest drop in hand pose, suggesting that regularization helps preserve the hand state from foundation models. How to guide 4D-HOF inference? We ablate each test-time guidance term in Table 7. Removing GsilG_sil degrades all metrics, suggesting that image-space alignment helps refine the reconstructed interaction. Removing GinterG_inter leads to the largest increase in interpenetration and the largest drop in contact accuracy, demonstrating its importance for plausible interactions. Removing GtempG_temp causes the largest increase in motion jitter, confirming that temporal guidance improves motion consistency. 5 Conclusion and Future Works We propose 4D-HOF, a novel feed-forward framework for 4D hand-object interaction reconstruction from monocular video. Our key innovation is to replace noise-driven generation with initialization derived from foundation models, and to learn a conditional generative bridge that transports these initial states toward physically plausible interactions. Our formulation enables joint refinement of hand, object, and their interactions in a feed-forward manner. Experiments show that our approach achieves strong performance in interaction quality while maintaining competitive hand and object accuracy, with significantly improved inference efficiency. One limitation, shared by many foundation model-based methods, is its sensitivity to initialization failures (Figure S6). Our current formulation focuses on single-hand interactions with rigid objects. Extending 4D-HOF to more complex settings, such as bimanual interactions and articulated or deformable objects, is an important future direction. References Aboukhadra et al. (2026) A. T. Aboukhadra, M. Rogge, N. Robertini, A. Arafa, J. Malik, A. Elhayek, and D. Stricker GHOST: fast category-agnostic hand-object interaction reconstruction from rgb videos using gaussian splatting. arXiv preprint arXiv:2603.18912. Cited by: §2. Albergo et al. (2025) M. Albergo, N. M. Boffi, and E. Vanden-Eijnden Stochastic interpolants: a unifying framework for flows and diffusions. Journal of Machine Learning Research 26 (209), p. 1–80. Cited by: §1, §2, §3.1. Aytekin et al. (2026a) A. I. Aytekin, X. Chen, Z. Shen, T. Beeler, H. Rhodin, R. Dabral, and C. Theobalt Grasp in gaussians: fast monocular reconstruction of dynamic hand-object interactions. arXiv preprint arXiv:2604.12929. Cited by: §2, §2. Aytekin et al. (2026b) A. I. Aytekin, H. Rhodin, R. Dabral, and C. Theobalt Follow my hold: hand-object interaction reconstruction through geometric guidance. In Thirteenth International Conference on 3D Vision, Cited by: §1. Blender Online Community (2026) Blender Online Community Blender: a 3d modelling and rendering package. Stichting Blender Foundation, Amsterdam, Netherlands. External Links: Link Cited by: §B.4. Carion et al. (2026) N. Carion, L. Gustafson, Y. Hu, S. Debnath, R. Hu, D. Suris Coll-Vinent, C. Ryali, K. V. Alwala, H. Khedr, A. Huang, et al. Sam 3: segment anything with concepts. In International conference on learning representations, Vol. 2026, p. 138846–138923. Cited by: §A.1.1, §1, §2, §3.2. Chao et al. (2021) Y. Chao, W. Yang, Y. Xiang, P. Molchanov, A. Handa, J. Tremblay, Y. S. Narang, K. Van Wyk, U. Iqbal, S. Birchfield, et al. Dexycb: a benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 9044–9053. Cited by: Figure 3, §4.1, Table 3. Chen et al. (2026a) X. Chen, F. Chu, P. Gleize, K. J. Liang, A. Sax, H. Tang, W. Wang, M. Guo, T. Hardin, X. Li, et al. Sam 3d: 3dfy anything in images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 7220–7232. Cited by: Figure S3, §B.2, §2, §3.2, §4.3. Chen et al. (2026b) Y. Chen, J. Chang, C. Ye, C. Zhang, Z. Fang, C. Li, and X. Han ForeHOI: feed-forward 3d object reconstruction from daily hand-object interaction videos. arXiv preprint arXiv:2602.06226. Cited by: §2. Christen et al. (2024) S. Christen, S. Hampali, F. Sener, E. Remelli, T. Hodan, E. Sauser, S. Ma, and B. Tekin Diffh2o: diffusion-based synthesis of hand-object interactions from textual descriptions. In SIGGRAPH Asia 2024 Conference Papers, p. 1–11. Cited by: §1, §2. Comanici et al. (2025) G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Cited by: §A.1.1, §3.2, §3.2. Fan et al. (2024) Z. Fan, M. Parelli, M. E. Kadoglou, X. Chen, M. Kocabas, M. J. Black, and O. Hilliges HOLD: category-agnostic 3d reconstruction of interacting hands and objects from video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 494–504. Cited by: §1, §1, §2, §3.3, Figure 3, §4.1, §4.1, §4.2. Hampali et al. (2020) S. Hampali, M. Rad, M. Oberweger, and V. Lepetit Honnotate: a method for 3d annotation of hand and object poses. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 3196–3206. Cited by: Figure S2, Table S1, §3.1, Figure 3, §4.1, §4.3. Hasson et al. (2021) Y. Hasson, G. Varol, C. Schmid, and I. Laptev Towards unconstrained joint hand-object reconstruction from rgb videos. In 2021 International Conference on 3D Vision (3DV), p. 659–668. Cited by: §2. Hasson et al. (2019) Y. Hasson, G. Varol, D. Tzionas, I. Kalevatykh, M. J. Black, I. Laptev, and C. Schmid Learning joint reconstruction of hands and manipulated objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 11807–11816. Cited by: §2, §3.3. He et al. (2025) J. He, D. Li, X. Yu, Z. Qi, W. Zhang, J. Chen, Z. Zhang, Z. Zhang, L. Yi, and H. Wang Dexvlg: dexterous vision-language-grasp model at scale. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 14248–14258. Cited by: §1. Ho et al. (2020) J. Ho, A. Jain, and P. Abbeel Denoising diffusion probabilistic models. Advances in neural information processing systems 33, p. 6840–6851. Cited by: §2. Huang et al. (2025a) M. Huang, F. Chu, B. Tekin, K. J. Liang, H. Ma, W. Wang, X. Chen, P. Gleize, H. Xue, S. Lyu, et al. Hoigpt: learning long-sequence hand-object interaction with language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 7136–7146. Cited by: §2. Huang et al. (2025b) Z. Huang, Y. Guo, H. Wang, R. Yi, L. Ma, Y. Cao, and L. Sheng Mv-adapter: multi-view consistent image generation made easy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 16377–16387. Cited by: §A.1.3. Jiang et al. (2021) H. Jiang, S. Liu, J. Wang, and X. Wang Hand-object contact consistency reasoning for human grasps generation. In Proceedings of the IEEE/CVF international conference on computer vision, p. 11107–11116. Cited by: §3.3, §4.1. Jung and Lee (2026) D. Jung and K. M. Lee Learning dense hand contact estimation from imbalanced data. Advances in Neural Information Processing Systems 38, p. 120351–120384. Cited by: §3.3. Kong et al. (2026) L. Kong, R. Li, R. Wang, S. Xu, C. Yao, J. Xiang, and J. Yang MoGe-3: fine-detail monocular geometry estimation with self-guided sparse volumetric refinement. arXiv e-prints, p. arXiv–2607. Cited by: §A.1.2, §B.4, Appendix C, §2, §3.2. Li et al. (2025) M. Li, S. Christen, C. Wan, Y. Cai, R. Liao, L. Sigal, and S. Ma LatentHOI: on the generalizable hand object motion generation with latent hand diffusion.. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 17416–17425. Cited by: §1, §2. Lin et al. (2025a) D. Lin, T. Wang, Z. Pan, Y. Wang, L. Liu, and K. Daniilidis Zero-shot reconstruction of in-scene object manipulation from video. arXiv preprint arXiv:2512.19684. Cited by: §1, §2, §2. Lin et al. (2025b) H. Lin, S. Chen, J. Liew, D. Y. Chen, Z. Li, G. Shi, J. Feng, and B. Kang Depth anything 3: recovering the visual space from any views. arXiv preprint arXiv:2511.10647. Cited by: §1, §2. Lipman et al. (2022) Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le Flow matching for generative modeling. arXiv preprint arXiv:2210.02747. Cited by: §3.1. Liu et al. (2022a) X. Liu, C. Gong, and Q. Liu Flow straight and fast: learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003. Cited by: §3.1. Liu et al. (2022b) X. Liu, L. Wu, M. Ye, and Q. Liu Let us build bridges: understanding and extending diffusion generative models. arXiv preprint arXiv:2208.14699. Cited by: §1, §2. Liu et al. (2025a) X. Liu, P. Ren, Q. Qi, H. Sun, Z. Zhuang, J. Wang, J. Liao, and J. Wang Generalizable hand-object modeling from monocular rgb images via 3d gaussians. Advances in Neural Information Processing Systems 38, p. 127318–127340. Cited by: §2. Liu and Yi (2024) X. Liu and L. Yi Geneoh diffusion: towards generalizable hand-object interaction denoising via denoising diffusion. arXiv preprint arXiv:2402.14810. Cited by: §2. Liu et al. (2025b) Y. Liu, X. Long, Z. Yang, Y. Liu, M. Habermann, C. Theobalt, Y. Ma, and W. Wang Easyhoi: unleashing the power of large models for reconstructing hand-object interactions in the wild. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 7037–7047. Cited by: §1. Luo et al. (2024) H. Luo, Y. Liu, and L. Yi Physics-aware hand-object interaction denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 2341–2350. Cited by: §2. Marchand et al. (2015) E. Marchand, H. Uchiyama, and F. Spindler Pose estimation for augmented reality: a hands-on survey. IEEE transactions on visualization and computer graphics 22 (12), p. 2633–2651. Cited by: §1. Mazur et al. (2026) K. Mazur, N. Karaev, M. Chang, J. Malik, and N. M. Shafiullah AgentSTAR: agentic shape tracking and reconstruction from monocular videos. External Links: 2609.24487, Link Cited by: §2. On et al. (2025) J. On, K. Gwak, G. Kang, J. Cha, S. Hwang, H. Hwang, and S. Baek Bigs: bimanual category-agnostic interaction reconstruction from monocular videos via 3d gaussian splatting. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 17437–17447. Cited by: §2. OpenAI (2026) OpenAI ChatGPT (6-astra) [large language model]. Note: https://chatgpt.comAccessed: 2026-09-25 Cited by: §B.4. Oquab et al. (2023) M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. Dinov2: learning robust visual features without supervision. arXiv preprint arXiv:2304.07193. Cited by: §A.1.3, §1, §2, §3.3. Paliwal et al. (2026) B. Paliwal, H. Etukuru, W. Liang, P. Abbeel, N. M. M. Shafiullah, and J. Malik Do as i do: dexterous manipulation data from everyday human videos. arXiv preprint arXiv:2606.19333. Cited by: §1. Pavlakos et al. (2024) G. Pavlakos, D. Shan, I. Radosavovic, A. Kanazawa, D. Fouhey, and J. Malik Reconstructing hands in 3d with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 9826–9836. Cited by: §1, §2, §3.3. Pavlovic et al. (2002) V. I. Pavlovic, R. Sharma, and T. S. Huang Visual interpretation of hand gestures for human-computer interaction: a review. IEEE Transactions on pattern analysis and machine intelligence 19 (7), p. 677–695. Cited by: §1. Peebles and Xie (2023) W. Peebles and S. Xie Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, p. 4195–4205. Cited by: §3.1. Piccinelli et al. (2025) L. Piccinelli, C. Sakaridis, Y. Yang, M. Segu, S. Li, W. Abbeloos, and L. Van Gool Unidepthv2: universal monocular metric depth estimation made simpler. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §1, §2. Potamias et al. (2025) R. A. Potamias, J. Zhang, J. Deng, and S. Zafeiriou Wilor: end-to-end 3d hand localization and reconstruction in-the-wild. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 12242–12254. Cited by: §A.1.2, §B.4, §1, §2, §3.2. Romero et al. (2022) J. Romero, D. Tzionas, and M. J. Black Embodied hands: modeling and capturing hands and bodies together. arXiv preprint arXiv:2201.02610. Cited by: §A.1.2, §3.1, §3.2. Shi et al. (2026) J. Shi, B. Ye, T. Liu, X. Liu, Y. Xu, J. He, Z. Li, H. Chen, and C. Shen Agile: hand-object interaction reconstruction from video via agentic generation. In Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers, p. 1–11. Cited by: §1, §1, §2, §2, §4.1, §4.1, §4.1, Table 1. Shi et al. (2023) Y. Shi, V. De Bortoli, A. Campbell, and A. Doucet Diffusion schrödinger bridge matching. Advances in neural information processing systems 36, p. 62183–62223. Cited by: §1, §2. Su et al. (2024) J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu Roformer: enhanced transformer with rotary position embedding. Neurocomputing 568, p. 127063. Cited by: §A.2.1. Swamy et al. (2023) A. Swamy, V. Leroy, P. Weinzaepfel, F. Baradel, S. Galaaoui, R. Brégier, M. Armando, J. Franco, and G. Rogez SHOWMe: benchmarking object-agnostic hand-object 3d reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, p. 1935–1944. Cited by: §3.1, §3.4. Valassakis and Garcia-Hernando (2024) E. Valassakis and G. Garcia-Hernando Handdgp: camera-space hand mesh prediction with differentiable global positioning. In European Conference on Computer Vision, p. 479–496. Cited by: §A.1.2. Wang et al. (2025a) J. Wang, M. Chen, N. Karaev, A. Vedaldi, C. Rupprecht, and D. Novotny Vggt: visual geometry grounded transformer. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 5294–5306. Cited by: §2. Wang et al. (2024) J. Wang, Q. Zhang, Y. Chao, B. Wen, X. Guo, and Y. Xiang Ho-cap: a capture system and dataset for 3d reconstruction and pose tracking of hand-object interaction. arXiv preprint arXiv:2406.06843. Cited by: §3.1, §3.4. Wang et al. (2025b) S. Wang, H. He, M. Parelli, C. Gebhardt, Z. Fan, and J. Song Magichoi: leveraging 3d priors for accurate hand-object reconstruction from short monocular video clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 5957–5968. Cited by: §B.1, §1, §1, §2, Figure 3, §4.1, §4.2, Table 1. Wang et al. (2026) Z. Wang, Z. Zhang, Y. Wang, H. Li, and W. Zuo ArtHOI: taming foundation models for monocular 4d reconstruction of hand-articulated-object interactions. arXiv preprint arXiv:2603.25791. Cited by: §1, §2. Wen et al. (2026) B. Wen, S. Dewan, and S. Birchfield Fast-foundationstereo: real-time zero-shot stereo matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 7513–7524. Cited by: §2. Wen et al. (2023) B. Wen, J. Tremblay, V. Blukis, S. Tyree, T. Müller, A. Evans, D. Fox, J. Kautz, and S. Birchfield Bundlesdf: neural 6-dof tracking and 3d reconstruction of unknown objects. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 606–617. Cited by: §2. Wen et al. (2025) B. Wen, M. Trepte, J. Aribido, J. Kautz, O. Gallo, and S. Birchfield Foundationstereo: zero-shot stereo matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 5249–5260. Cited by: §2. Wen et al. (2024) B. Wen, W. Yang, J. Kautz, and S. Birchfield Foundationpose: unified 6d pose estimation and tracking of novel objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 17868–17879. Cited by: §A.1.3, §B.4, §1, §2, §3.2, §4.3. Wimmer et al. (2026) T. Wimmer, P. Truong, M. Rakotosaona, M. Oechsle, F. Tombari, B. Schiele, and J. E. Lenssen Anyup: universal feature upsampling. In International Conference on Learning Representations, Vol. 2026, p. 140700–140720. Cited by: §A.1.3. Wu et al. (2026) X. Wu, Y. Zhang, Y. Liu, Y. Ma, Y. Shi, and X. He AffordGrasp: cross-modal diffusion for affordance-aware grasp synthesis. arXiv preprint arXiv:2603.08021. Cited by: §3.1. Wu et al. (2024) X. Wu, L. Jiang, P. Wang, Z. Liu, X. Liu, Y. Qiao, W. Ouyang, T. He, and H. Zhao Point transformer v3: simpler faster stronger. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 4840–4851. Cited by: §A.2.1, §3.3. Xiang et al. (2026) J. Xiang, X. Chen, S. Xu, R. Wang, Z. Lv, Y. Deng, H. Zhu, Y. Dong, H. Zhao, N. J. Yuan, et al. Native and compact structured latents for 3d generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 14419–14429. Cited by: §2. Xie et al. (2026) X. Xie, B. Wen, C. Yan, H. Rabeti, J. Li, Y. Yuan, G. Pons-Moll, and S. Birchfield CARI4D: category agnostic 4d reconstruction of human-object interaction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Cited by: §2. Xu et al. (2026) H. Xu, Y. Liu, Y. Wang, C. Fu, and N. J. Mitra CHOIR: contact-aware 4d hand-object interaction reconstruction. arXiv preprint arXiv:2605.20992. Cited by: §1, §1, §2, §4.1. Yang et al. (2022) L. Yang, K. Li, X. Zhan, F. Wu, A. Xu, L. Liu, and C. Lu Oakink: a large-scale knowledge repository for understanding hand-object interaction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 20953–20962. Cited by: §3.1, §3.4. Ye et al. (2024) Y. Ye, A. Gupta, K. Kitani, and S. Tulsiani G-hop: generative hand-object prior for interaction reconstruction and grasp synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 1911–1920. Cited by: §1, §2, §3.1. Ye et al. (2022) Y. Ye, A. Gupta, and S. Tulsiani What’s in your hands? 3d reconstruction of generic objects in hands. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 3895–3905. Cited by: §2, §3.1. Ye et al. (2023) Y. Ye, P. Hebbar, A. Gupta, and S. Tulsiani Diffusion-guided reconstruction of everyday hand-object interaction clips. In Proceedings of the IEEE/CVF international conference on computer vision, p. 19717–19728. Cited by: §1, §2. Ye et al. (2026) Y. Ye, J. Li, R. Rong, and C. K. Liu Whole: world-grounded hand-object lifted from egocentric videos. arXiv preprint arXiv:2602.22209. Cited by: §1, §3.1. Zhao et al. (2025) Z. Zhao, Z. Lai, Q. Lin, Y. Zhao, H. Liu, S. Yang, Y. Feng, M. Yang, S. Zhang, X. Yang, et al. Hunyuan3d 2.0: scaling diffusion models for high resolution textured 3d assets generation. arXiv preprint arXiv:2501.12202. Cited by: §A.1.3, §3.2. Zhong et al. (2025) Y. Zhong, Q. Jiang, J. Yu, and Y. Ma Dexgrasp anything: towards universal robotic dexterous grasping with physics awareness. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 22584–22594. Cited by: §1. Zhou et al. (2022) K. Zhou, B. L. Bhatnagar, J. E. Lenssen, and G. Pons-Moll Toch: spatio-temporal object-to-hand correspondence for motion refinement. In European Conference on Computer Vision, p. 1–19. Cited by: §2. Zhu et al. (2025) Z. Zhu, S. Bansal, S. Tripathi, and D. Damen Reconstructing objects along hand interaction timelines in egocentric video. arXiv preprint arXiv:2512.07394. Cited by: §1. Table of Contents in Appendix Appendix A Implementation Details In this section, we describe additional details of our method. We first present the hand and object state initialization pipeline based on various foundation models in Section A.1, followed by details of the flow-matching model in Section A.2. A.1 Hand and Object State Initialization A.1.1 Contextual scene parsing. Given an input video ℐI, we first employ a VLM (Comanici et al., 2025) to understand the interaction and identify the object manipulated by the human hand. Since processing all video frames is computationally prohibitive, we uniformly sample five frames from the video and prompt the VLM to describe the target object. The resulting object description, together with the prompt “hand,” is then provided to SAM 3 (Carion et al., 2026) for segmentation. A.1.2 Hand reconstruction and state estimation. For hand estimation, we use WiLoR (Potamias et al., 2025) to predict the MANO (Romero et al., 2022) parameters and 2D keypoints. We then recover the hand translation in camera coordinates using the camera intrinsics from MoGe-3 (Kong et al., 2026; Valassakis and Garcia-Hernando, 2024). Specifically, for each joint, the perspective projection yields the 2D position (u,v)(u,v) of each 3D joint following equations: u=fxx+xz+z+cx, u=f_x J_x+t_xJ_z+t_z+c_x, (S1) v=fyy+yz+z+cy, v=f_y J_y+t_yJ_z+t_z+c_y, (S2) where (x,y,z)(J_x,J_y,J_z) denotes the position of 3D joint in the canonical hand space, (x,y,z)(t_x,t_y,t_z) represents the global hand translation, and fx,fy,cx,cyf_x,f_y,c_x,c_y are the camera intrinsic parameters. By aggregating constraints from all joints, we construct a linear system and solve it using least squares. A.1.3 Object reconstruction and state estimation. In order to automatically select a compact yet informative set of observations for the object from the video, we propose a visual feature–based frame selection strategy. Specifically, we extract image features using a pre-trained DINOv2 (Oquab et al., 2023) image encoder. Since DINO only yields patch-level embeddings, we also apply AnyUp (Wimmer et al., 2026) to enhance their spatial resolution. We then pool the upsampled features within the object mask region to obtain an object descriptor for each frame. After having the object descriptors, the next step is picking the frames that are visually different. The frame with the largest object mask is first chosen as the anchor, and Farthest Point Sampling (FPS) is performed in the descriptor space to select a visual diverse set of frames. These selected frames are used as context prompts for canonical view generation, and the synthesized views are subsequently fed into a multi-view-to-3D model (Zhao et al., 2025) to recover the object mesh. Although the object geometry is generally reliable, textures produced by generative models are often unsatisfactory, preserving coarse color spots but failing on fine-grained patterns and text. To address this limitation, we construct two types of texture maps: one obtained directly from the generative model, which is suitable for unstructured objects with simple appearance, and another derived by projecting the canonical views (Huang et al., 2025b), which better preserves detailed patterns for structured objects. We render the object under multiple views using both texture types and employ a VLM to automatically select the optimal variant. To better leverage FoundationPose (Wen et al., 2024) for object pose estimation, we initialize the pose on the frame with the largest object mask and propagate it bidirectionally throughout the sequence. Instead of relying on the default tracking mode of FoundationPose, in which a single occluded frame may lead to catastrophic error accumulation, we perform independent per-frame registration while explicitly rejecting pose hypotheses that deviate significantly from the previous estimate. Specifically, we use the geodesic distance between adjacent frame rotations as the filtering criterion and set the threshold to 60∘. We further employ silhouette-based filtering to further enforce visual consistency. For each pose hypothesis, we render an object mask o~ M^o and compute its intersection-over-union (IoU) with the segmentation mask oM^o. To reduce the influence of hand occlusion, we exclude the hand mask hM^h region before computing the IoU: IoU=(~o∩∼h)∩(~o∩∼h)(~o∩∼h)∪(~o∩∼h). = ( M^o∩ ^h)∩( M^o∩ ^h)( M^o∩ ^h)∪( M^o∩ ^h). (S3) We retain only the top 30 pose hypotheses with the highest IoU scores. To further improve robustness, we enable both RGB and RGB-D modes in the pose refinement and scoring stages of FoundationPose, and compute the final pose score as the average of the two modalities. Figure S1: Model architecture. Our model adopts a DiT-style architecture for conditional flow matching. The latent hand-object state is processed through temporal self-attention, point-stream cross-attention using Point Transformer V3 features, and visual cross-attention using DINO features. Adaptive layer normalization conditioned on the timestep is applied throughout the network to modulate the generative transport process. A.2 4D HOI Flow Matching A.2.1 Model architecture Our flow matching model transports the noisy hand and object pose from the foundation model-based initialization to a plausible interaction pose manifold. Since we aim to reconstruct the 4D hand object interaction, a natural design is directly handling the sequence data, which benefits from the temporal information interaction. Specifically, our model takes a hand and object pose sequence as input. In each frame, the hand pose and object pose are concatenated together and tokenized by a linear layer. This pose token sequence is then passed thought several stacked Transformer blocks, after that the hand and object pose in each frame are decoded from the corresponding pose token. There are three attention operations in each Transformer block. Our block starts with self-attention operation along the temporal dimension with RoPE (Su et al., 2024) for temporal embedding. This role of this layer is to utilize the temporal corrections in the sequence. In a motion sequence, the pose divergence usually small for nearby frames and large for far away frames, besides the motion evolution in the sequence should be smooth. By learning large scale data, this layer can make the pose sequences coherent and smooth. The second layer is a cross-attention layer between the 3D HOI point cloud sequence and pose token. The motivation of this operation is to inject explicit 3D spatial information to the model, since it’s impossible to understand the hand object interaction solely from a set of bare rotations and translation. To prepare 3D cues, we use the pose results from the initialization stage to transform the MANO hand and object, then use a Point Transformer V3 (Wu et al., 2024) to encode them. In each frame, we use MANO vertices as hand point cloud and sample points on the object surface as the object point cloud. We treat the two components as a single point cloud and add a class label to each point. After feature extraction, a max pooling operation is applied to construct the final HOI point cloud token sequence. By this cross attention, the model can obtain explicit 3D information which boosts joint optimization. The third one is another cross attention with the 2D image features, since we hope the reconstruction results respect the observation. For each video frame, we use a frozen DINOv2-B to extract visual features and use Plucker ray to add camera information to each patch embedding. To balance the computational cost, we only apply cross attention within each frame. A standard feed-forward layer is added at the end of block, and adaptive layer norm is used before each attention and FFN later to inject flow-step information. A.2.2 Training data curation We initially explored training with initializations produced by our foundation model pipeline, but observed two limitations. First, object symmetries introduce ambiguities in the estimated rotations, complicating the construction of interpolation paths for bridge matching and subsequent training. Second, initialization errors exhibit dataset-specific biases, limiting the effectiveness of these initializations as a training source. We therefore adopt a dataset-agnostic perturbation strategy that synthetically corrupts ground-truth HOI states to emulate imperfect initializations. We introduce synthetic noise to the ground-truth HOI states to emulate the imperfect initializations generated by the foundation models. To perturb the object pose, we sample a large offset (3.14 rad, 0.2 m) across 20% of the frames to simulate catastrophic pose estimation failures, while applying a smaller offset (0.1 rad, 0.02 m) to the remaining 80%. For the hand parameters, the translation vector is perturbed using noise sampled from a uniform spatial distribution (0.02, 0.02, 0.1). Meanwhile, the global orientation and the local pose of the hand are perturbed by adding Gaussian noise with a standard deviation of 0.1. A.2.3 Training details We optimize our model using AdamW with a learning rate of 0.00010.0001 and a weight decay of 0.00010.0001, regulated by a cosine learning rate scheduler with a warmup phase. The training process spans 20 epochs with a batch size of 16. During feature extraction, input images are resized to 224×224 before being encoded by DINOv2. To establish an explicit 3D encoding, we uniformly sample 4096 points from the object mesh surface. Finally, the temporal sequence length is set to 16 frames during training and extended to 128 frames during inference. The entire training process takes approximately 30 hours on a single NVIDIA H100 GPU. Figure S2: Qualitative results on HO3D-v3 (Hampali et al., 2020) following MagicHOI’s setting. Table S1: Results on HO3D-v3 (Hampali et al., 2020) following MagicHOI’s setting. Method Hand Object Interaction MPJPE ↓ CD ↓ F5 ↑ F10 ↑ RS ↓ CDh ↓ iHOI 27.75 2.37 35.78 62.11 0.17 25.45 DiffHOI 16.02 2.30 39.59 64.49 0.13 33.33 EasyHOI 16.69 1.86 46.10 70.92 0.28 19.55 HOLD 30.79 1.31 57.20 80.23 0.62 21.28 MagicHOI 4.62 0.87 69.72 92.15 0.11 2.39 CHOIR 5.55 0.77 72.33 96.03 0.10 2.41 4D-HOF (ours) 4.60 0.24 90.24 98.13 0.13 2.17 Appendix B More Experimental Results B.1 HO3D-v3 MagicHOI’s setting We empirically observe that some methods perform better on shorter clips, motivating an additional evaluation using the same sequences and frames as MagicHOI (Wang et al., 2025b). Table S1 and Figure S2 present the quantitative and qualitative results, respectively. Under this protocol, 4D-HOF achieves the best results on five of the six metrics, with substantial improvements in object reconstruction accuracy. B.2 Comparison of Object Generation We compare our multiview-based object generation pipeline with SAM3D (Chen et al., 2026a), with quantitative and qualitative results presented in Table S2 and Figure S3, respectively. Limited to a single input image, SAM3D produces less reliable textures, which can compromise downstream pose estimation with FoundationPose, where rendered object appearance provides a key cue for pose evaluation. By combining complementary observations from multiple views with a powerful image generation model, our pipeline produces objects with higher geometric accuracy and texture fidelity. Table S2: Object generation comparison. Our multiview-based pipeline outperforms SAM3D across all metrics, demonstrating the benefit of complementary visual observations for object reconstruction. Methods CD↓ F5↑ F10↑ RS↓ SAM3D 0.47 77.54 96.20 0.17 4D-HOF (Ours) 0.28 89.11 97.13 0.12 Figure S3: Qualitative comparison of object generation. We show two views of each object reconstructed by 4D-HOF and SAM3D (Chen et al., 2026a) alongside the ground truth. Our pipeline better preserves object geometry and texture details, including labels and fine structures. Ground-truth object widths are provided for scale reference. B.3 Do more HOI data help? To investigate the impact of training data volume, we evaluate our model’s performance on the HO3D dataset using the CDh metric (Figure S4). The results indicate that increasing the size of the training dataset yields consistent performance gains, demonstrating the scalability of our approach. Figure S4: Dataset scaling analysis for bridge matching model training. B.4 Comparison with GPT 6 Astra Ultra (as of Sep 20, 2026) Recent GPT-6 (OpenAI, 2026) models have demonstrated strong capabilities in 3D reasoning and tool use. To provide a broader and more rigorous evaluation of our model, we constructed an agentic baseline by prompting GPT-6-Astra with Ultra reasoning effort and providing access to FoundationPose (Wen et al., 2024), WiLoR (Potamias et al., 2025), MoGe-3 (Kong et al., 2026), Blender (Blender Online Community, 2026), and related tools. We tasked the agent with reconstructing 4D hand–object interactions from the ABF12 and SM2 sequences of HO3D dataset , without imposing execution-time or token-budget limits. Our prompt is illustrated in B.4. Qualitative comparisons are shown in Figure S5. Although GPT-6 recovers plausible coarse hand and object poses, it exhibits noticeable hand–object separation and weaker cross-view contact consistency. In contrast, our method better preserves instance-specific object geometry and produces tighter, more physically plausible hand–object configurations: in SM2, the fingers more consistently wrap around the bottle, while in ABF12, our reconstruction better captures the complex fingertip contact and provide consistent result across viewpoints. These results suggest that, despite GPT-6’s strong general-purpose 3D reasoning ability, the proposed pipeline provides more accurate geometry and more coherent hand–object interactions. Beyond reconstruction quality, the agentic baseline incurs substantial runtime and resource overhead. In our experiments, a single run typically required several hours and consumed approximately 35% of the weekly usage allowance associated with our $100 subscription plan, making large-scale evaluation impractical under this budget. These observations highlight the challenges of applying general-purpose agentic systems to efficient, contact-consistent HOI reconstruction. Our feed-forward module could serve as a specialized component within such systems to accelerate reconstruction while improving geometric and interaction consistency. Figure S5: Qualitative comparison between GPT6-Astra and our 4D-HOF. GPT6 HOI Reconstruction Prompt Two third-person-view human hand–object interaction videos is available at: data/ABF12.mp4, data/SM2.mp4 For each video, reconstruct: 1. A temporally consistent MANO hand sequence. 2. A textured mesh for each manipulated object. 3. A per-frame 6DoF object pose sequence. The hand and object should share a consistent metric coordinate system. Minimize hand–object penetration and separation, keep motion smooth, preserve plausible physical contact, and ensure rendered hand/object meshes align closely with the original RGB frames in 2D. Handle occlusion and tracking loss without large pose jumps, scale changes, or identity switches. If additional Python packages are needed, create a virtual environment in the current directory rather than modifying the system environment. Blender and Blender-mcp are installed and enabled. - Use MoGe-3 (https://github.com/microsoft/MoGe) for monocular depth estimation. - Use WiLoR (https://github.com/rolpotamias/WiLoR) for MANO hand reconstruction, plus additional optimization procedures. - Use FoundationPose (https://github.com/NVlabs/FoundationPose) for object 6DoF pose estimation, and make suitable modifications if you need. Create textured mesh and adjust mesh scale before running the model. You can use the depth from MoGe 3 as input. Output: - MANO parameters, joints, and vertices. - Textured object meshes. - Per-frame object poses and confidence. - Camera parameters and coordinate-system documentation. - RGB overlay videos and necessary 3D/diagnostic visualizations. - README and other documents containing setup, commands, methods, optimization details, output paths, validation results, and known limitations. Appendix C Failure Case Analysis Our pipeline is reliant on several vision foundation models, with the monocular depth estimator, MoGe-3, acting as a critical geometric anchor. MoGe-3 (Kong et al., 2026) recovers the object’s metric scale to match the hand and coarsely aligns both components within a unified coordinate space. Consequently, if the predicted point map exhibits significant artifacts (Figure S6), these inaccuracies cascade directly into the object scale and pose estimation modules. Such highly corrupted, out-of-distribution initializations ultimately misguide the flow-matching refinement model, resulting in catastrophic reconstruction failures. Figure S6: Failure analysis. When the estimated point map contains severe artifacts or out-of-distribution predictions, the resulting initialization may become unreliable, which can affect the flow-matching refinement process and lead to reconstruction failures. Appendix D VLM Prompt Our hand and object states initialization pipeline employs VLM for multiple purposes, including identifying the manipulated object, generating multi-view images, and selecting object textures. We describe each task below and provide the corresponding prompts. D.1 Interaction object reasoning Given an input video, the VLM identifies the object manipulated by the hand and generates a textual description. This description is then passed to SAM3 as a prompt for automatic object segmentation. Object Identification Prompt Describe the object interaction with hand, only return in [color object_name] format. D.2 Multi-view image generation We select object keyframes from the input video and provide them to an image generation model to synthesize canonical views of the object. The placeholder description in the prompt is filled with the object description obtained in the preceding reasoning step. Canonical View Generation Prompt Generate the front, back, left, right, top, bottom canonical views of this description. The generated views MUST perfectly match the exact shape, texture, color, and geometric details of the object in the reference images. Complete the occluded regions. Present as a 2*3 grid, front, back, left in the upper row, right, top, bottom in the lower row. Pure white background. The image must not contain text indicating viewpoint direction. After each generation attempt, the VLM acts as a critic to assess whether the generated image satisfies the specified requirements. If the image fails this evaluation, we regenerate it. Generated View Evaluation Prompt You are an expert evaluator. Look at the provided generated image at the very end. It is supposed to be a 2x3 image grid showing the front, back, left, right, top, and bottom canonical views of the object in the reference images. Does the generated image strictly follow these requirements: 1. It is exactly a 2x3 grid. 2. Background is pure white. 3. No text indicating viewpoint direction. 4. Identity and texture are preserved. Answer EXACTLY ’Yes’ or ’No’ without any explanation. If both consecutive attempts fail, the VLM selects the better of the two candidates, allowing the pipeline to proceed. Candidate Image Selection Prompt I have two generated images at the end, each supposed to be a 2x3 grid showing canonical views of the same object from the preceding reference images. Image 1 is the previous best. Image 2 is the newly generated one. Compare them based on these criteria: 1. Exactly a 2x3 grid layout. 2. Pure white background. 3. No text or labels. 4. High visual quality and accurate canonical views. Which image strictly follows these instructions better and has better quality? Return EXACTLY ’Image 1’ or ’Image 2’ without any explanation or markdown formatting. D.3 Texture selection Given the reconstructed object geometry, we obtain two candidate texture maps through direct generation and image projection, respectively. We apply each texture map to the geometry and render the resulting textured object from multiple viewpoints. The VLM then compares these multi-view renderings and selects the better texture. Texture Selection Prompt I have two 2x3 image grids showing canonical views of the same 3D object’s geometry. Image 1 represents Model A. Image 2 represents Model B. Important: The 3D geometry is identical in both models, but the poses or viewpoints might differ between the two images. Please act as a 3D texture quality judge. Ignore any pose or orientation differences, and focus strictly on comparing the texture quality. Look for detail, sharpness, realism, and a lack of noise or artifacts. Which model has better texture quality? Return EXACTLY ’Model A’ or ’Model B’ without any explanation or markdown formatting.