Paper deep dive
VOMMI: Collecting and Leveraging Portable Demonstrations for Mobile Manipulation
Yutian Zhang, Xingrui Xiong, Siyuan Ma, Yang Li, Jiawen Wen, Jiaqi Zhai, Liwen Yang, Ce Hao, Haozhen Chi, Yangkun Zhu, Yifan Zhu, Xiaowen Chu, Dong Wei, Qiaojun Yu, Dibo Hou
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Portable mobile-manipulation demonstrations can help alleviate data scarcity for embodied intelligence, but obtaining reliable, low-cost, and robot-free motion supervision from RGB observations remains challenging. Existing approaches often rely on teleoperation or specialized devices equipped with additional sensing hardware, while directly using estimated visual odometry (VO) trajectories can introduce inconsistencies due to accumulated drift and imperfect motion supervision. We present the Visual-Odometry-Conditioned Mobile Manipulation Interface (VOMMI), a portable demonstration collection and learning framework that connects portable RGB demonstrations to vision-language-action (VLA) post-training through offline trajectory reconstruction and online visual-motion conditioning. VOMMI synchronizes body and hand views to capture navigation context and local object interactions without requiring human-robot kinematic correspondence calibration. R2-VO refines offline demonstration trajectories using sparse geometric anchors and produces causal local-motion tokens over multiple prediction horizons for online policy conditioning. An action-group residual adapter incorporates these tokens only into the base branch. Experiments use a 500-trajectory portable for each task, with 75 trajectories held out for RGB-VO evaluation, and 200 robot demonstrations as references. Our policy, post-trained only on portable demonstrations, achieves 18.2% lower base-velocity error than a policy trained with robot-collected demonstrations, while maintaining comparable end-effector translation accuracy. Offline reconstruction reduces absolute trajectory errors for the body and hand streams by 24.6% on average relative to the best evaluated baseline for each stream. The complete system improves the mean success rate by 8.3 percentage points over OpenPI 0.5 across three real-robot tasks.
Tags
Links
- Source: https://arxiv.org/abs/2610.08220v1
- Canonical: https://arxiv.org/abs/2610.08220v1
Trouble viewing inline? Open PDF directly →
Full Text
44,102 characters extracted from source content.
Expand or collapse full text
VOMMI: Collecting and Leveraging Portable Demonstrations for Mobile Manipulation Yutian Zhang1,∗, Xingrui Xiong4,∗, Siyuan Ma5,∗, Yang Li2, Jiawen Wen7, Jiaqi Zhai3, Liwen Yang1, Ce Hao6, Haozhen Chi8, Yangkun Zhu2, Yifan Zhu2, Xiaowen CHU7, Dong Wei6,†, Qiaojun Yu2,†, Dibo Hou1,† Affiliation: 1ZJU 2Shanghai AI Lab 3DeepRobotics 4Yale 5THU 6ZGCA 7HKUST(GZ) 8ZUST Equal contribution∗. Corresponding authors† Abstract Portable mobile-manipulation demonstrations can help alleviate data scarcity for embodied intelligence, but obtaining reliable, low-cost, and robot-free motion supervision from RGB observations remains challenging. Existing approaches often rely on teleoperation or specialized devices equipped with additional sensing hardware, while directly using estimated visual odometry (VO) trajectories can introduce inconsistencies due to accumulated drift and imperfect motion supervision. We present the Visual-Odometry-Conditioned Mobile Manipulation Interface (VOMMI), a portable demonstration collection and learning framework that connects portable RGB demonstrations to vision-language-action (VLA) post-training through offline trajectory reconstruction and online visual-motion conditioning. VOMMI synchronizes body and hand views to capture navigation context and local object interactions without requiring human–robot kinematic correspondence calibration. R2-VO refines offline demonstration trajectories using sparse geometric anchors and produces causal local-motion tokens over multiple prediction horizons for online policy conditioning. An action-group residual adapter incorporates these tokens only into the base branch. Experiments use a 500-trajectory portable for each task, with 75 trajectories held out for RGB-VO evaluation, and 200 robot demonstrations as references. Our policy, post-trained only on portable demonstrations, achieves 18.2% lower base-velocity error than a policy trained with robot-collected demonstrations, while maintaining comparable end-effector translation accuracy. Offline reconstruction reduces absolute trajectory errors for the body and hand streams by 24.6% on average relative to the best evaluated baseline for each stream. The complete system improves the mean success rate by 8.3 percentage points over OpenPI 0.5 across three real-robot tasks. I Introduction Mobile manipulation is a foundational capability for embodied intelligence, enabling robots to interact with the physical world in applications ranging from domestic assistance and infrastructure inspection to industrial automation. Mobile manipulation data collection typically involves long-horizon tasks, composed of multiple sequential stages, requiring a generalizable policy that can coordinate precise whole-body motion with fine-grained end-effector (E) manipulation. Recent whole-body vision-language-action models (VLAs) [1, 2] and world action models (WAMs) [3, 4] advocate learning mobile manipulation as an integrated whole-body behavior rather than as separate navigation and arm-manipulation skills, but scaling such learning remains challenging due to the limited availability of demonstration data. Existing demonstration-collection paradigms often rely on teleoperation, which is expensive, slow, and difficult to scale for scene generalization. Cross-embodiment collection methods, such as UMI [5] and its variants [6, 7, 8, 9], use handheld grippers with mounted cameras to collect manipulation demonstrations, while egocentric approaches use head-mounted cameras to capture broader scene context [10]. For mobile manipulation tasks, existing methods have shown that portable demonstration collection is feasible [11, 9, 8]. However, these approaches commonly depend on special devices, such as high-quality IMU sensors, GoPro cameras, iPhones, and VR headsets, which raise the hardware barrier for more accessible data collection and limit the number of demonstrators who can participate. Using visual odometry (VO) can reduce the complexity of the data-collection hardware, but it remains vulnerable to drift over long trajectories and large rotations, which can degrade data quality or even make the data unusable. Existing portable collection pipelines typically require accurate device calibration to recover reliable trajectories, placing strict demands on data post-processing, especially for mobile manipulation tasks. Fig. 1: VOMMI. We build upon portable manipulation interfaces with a low-cost RGB-based perception system to enable accessible mobile demonstration collection and integration with VLA frameworks. As shown in Figure 1, we propose a VO-conditioned mobile-manipulation interface (VOMMI) to establish the data pipeline for training whole-body VLA policies from portable demonstrations. VOMMI records RGB videos from body and hand views and uses VO to recover local motion cues, enabling low-cost demonstration collection. Collection-time pose references are used only for offline metric supervision and evaluation, while deployment relies solely on RGB observations, camera intrinsics, timestamps, and robot proprioception. To improve trajectory quality during offline data processing, VOMMI can leverage pretrained visual geometry for base movement. These tokens are fed into the VLA post-training interface and connected only to the base-action residual adapter, while the manipulation residual remains blind to body motion. This design provides the missing local visual proprioception for navigation-conditioned mobile manipulation without claiming global localization or a new general-purpose VO algorithm. VOMMI is designed to test whether portable RGB demonstrations can improve long-horizon mobile-manipulation competence in a VLA policy, while making the answer auditable by separating portable-data supervision from robot-data supervision. Our contributions are summarized as follows: • We propose a portable data-collection protocol for mobile manipulation with separate body and hand views, requiring no access to the target robot or human–robot kinematic correspondence calibration. • We present R2-VO, a real-time and refined RGB motion reconstruction module that produces low-latency local motion tokens for online VLA inference and anchor-refined trajectories for offline dataset processing. • We define an action interface with dimension-specific reference pose supervision and action-group residuals, so that the manipulation residual remains independent of the body-motion token. I Related Work I-A Portable Demonstration Interfaces Portable data collection is fast, cheap, and easy to deploy for mobile manipulators in diverse environments. UMI [5] records in-the-wild manipulation with a portable handheld interface and transfers relative E actions to a robot. To enhance mobility, UMI-on-Legs combines this interface with a manipulation-centric whole-body controller [11], while Mobile UMI [9] introduces multiple views and decoupled planar-base and E representations for mobile manipulation. HoMMI [8] and EgoHumanoid [10] further use egocentric human demonstrations to capture whole-body task context. These systems demonstrate the value of extending portable whole-body data from fixed-base to mobile manipulation tasks, but they require customized, dedicated hardware. In contrast, VOMMI addresses a complementary sensing question: how to recover the demonstrator’s motion from RGB observations without requiring metric pose or IMU measurements as policy inputs. I-B Robot Learning for Mobile Manipulation Mobile manipulation requires coordinated locomotion and manipulation over extended workspaces. HomeRobot integrates perception, navigation, and manipulation in realistic environments [12]. Recent whole-body learning methods address complementary control challenges, from force-aware compensation for contact-rich loco-manipulation to latent policies for unified loco-manipulation [1, 13]. In addition, bimanual manipulation learning uses egocentric demonstrations to learn active perception behaviors [14]. DECOWAM [15] trains adapters by separating base and arm latents and conditioning on base velocity to improve both future-video and action prediction. Despite these advances, existing portable-demonstration-based mobile manipulation policies often lack an explicit representation of recent ego-motion when learning from portable RGB demonstrations. VOMMI addresses this gap by incorporating visual-odometry (VO) observations directly into the VLA and introducing separate residual pathways for body and manipulation actions. This design enables motion-aware whole-body correction without requiring metric pose or IMU measurements as policy inputs at deployment. I-C Visual Body Motion for Policy State Learning-based VO systems such as DROID-SLAM and DPVO improve sequential motion estimation, while VGGT and VGGT-SLAM 2.0 extend feed-forward geometry toward camera and real-time scene reconstruction [16, 17, 18, 19]. Beyond geometric reconstruction, recent robot learning studies also explore recovering actionable trajectories from human or egocentric demonstrations, such as observation-only imitation and scalable egocentric data collection [20, 21]. However, these approaches either optimize geometric consistency or focus on embodiment alignment, rather than extracting policy-oriented motion for mobile manipulation. VOMMI considers visual motion from a policy-oriented perspective. This causal local representation reduces sensitivity to constant world-frame offsets while preserving short-term ego-motion cues relevant for whole-body control. The resulting token provides a compact visual proprioceptive state for policy conditioning rather than a globally consistent trajectory or localization memory. I Data Collection System I-A Dual-view capture and temporal alignment VOMMI is designed around a specific robot-free data-collection requirement: each demonstration must capture both the demonstrator’s locomotion and hand–object interaction using only RGB observations, without relying on additional sensing or target-robot instrumentation. Fig. 2 summarizes the data flow. The demonstrator wears a chest-mounted fisheye RGB camera for scene context and body motion and carries a second fisheye RGB camera on the UMI-style gripper module for interaction. These two views are recorded as complementary streams and reconciled later through local timestamps and trajectory-based compensation, resampled at 15 Hz after validity checks. Each episode stores the language instruction, both videos and intrinsics, tool center point (TCP) and gripper signals, and validity masks. The resulting episode contains both the observations required for RGB reconstruction and the action targets necessary to train a whole-body policy. Collection-time sensor-derived trajectories serve as privileged offline references for R2-VO supervision and evaluation, but are not policy inputs at deployment. As shown in Table I, the cost of our VOMMI’s hardware is significantly lower than that of current portable data acquisition methods. Fig. 2: Portable data collection interface. Independent Body and Hand RGB views are timestamped into a robot-compatible episode. TABLE I: Representative hardware requirements and estimated acquisition cost for portable data collection methods Method Camera(s) View Price (USD) UMI [5] GoPro Wrist $350–450 iPhUMI [7] iPhone Wrist $1,000–1,200 FastUMI [6] GoPro+RealSense T265 Wrist $650–900 Mobile UMI [9] Dual RGB Wrist+ego $800–1,200 VOMMI (ours) 2 fisheye RGB Wrist+ego $100–200 I-B Episode schema and split protocol The collection contains 500 portable episodes per evaluated task. For each task, 75 complete episodes form a fixed held-out RGB-VO evaluation split and are excluded from estimator fitting, canonicalizer fitting, and model selection. The remaining 425 episodes support model development and downstream VLA training under the experimental protocol. For each valid frame in the Hand stream, let TEWH(t)∈SE(3)T^W_H_E(t) (3) denote the TCP pose at time t, expressed in the Hand-stream local world frame WHW_H. The current-relative manipulation target at future step k is then defined as: At,ktcp=(TEWH(t))−1TEWH(t+k+1),A^tcp_t,k= (T^W_H_E(t) )^-1T^W_H_E(t+k+1), (1) which is the tracking-derived current-TCP-relative SE(3)SE(3) reference used for Hand supervision. The Body stream supplies the corresponding planar reference [Δx,Δy,Δψ][ x, y, ψ]. During RGB source replacement, the canonicalized Body trajectory is aligned with the Hand timestamps and used to compensate the Hand TCP motion. Section V defines this compensation and the resulting action targets. IV Problem Formulation The central data-construction problem is to recover a robot-compatible local motion representation from two unsynchronized RGB streams, enabling the transfer of portable human demonstrations to robot policy learning. Let IB0:TI^B_0:T and IH0:TI^H_0:T denote the Body and Hand videos. The Body trajectory is represented by planar motion as: ξt,kB=[Δxt,k,Δyt,k,Δψt,k]∈(2),ξ^B_t,k=[ x_t,k, y_t,k, _t,k]∈ se(2), (2) whereas the Hand trajectory captures fine-grained manipulation motion through current-relative TCP motion, yielding a local SE(3) representation that reduces dependence on the global reference frame: ξt,kE=log(E(t)−1E(t+k+1))=[Δpt,k,ϕt,k]∈(3).ξ^E_t,k= \! (T_E(t)^-1T_E(t+k+1) )=[ p_t,k, _t,k]∈ se(3). (3) We estimate the motions in Eqs. (2) and (3) from past RGB observations and compensate the Hand motion using the planar Body estimate. The Body trajectory is canonicalized and aligned with the Hand timestamps, yielding Eq. (4): TE|Bcomp(t)=(TB(t))−1TE(t),T^comp_E|B(t)= (T_B(t) )^-1T_E(t), (4) where TB(t)T_B(t) and TE(t)T_E(t) are expressed in the same reference frame. Reference trajectories provide privileged supervision for offline data construction and evaluation, while deployment relies only on RGB observations. IV-A Unified Observation and Action Space The source-conditioned observation is: ts=(ℓ,It−hB,s,It−hH,s,sts,Mto),O_t^s=( ,I_t-h^B,s,I_t-h^H,s,s_t^s,M_t^o), (5) In Eq. (5), MtoM_t^o masks unavailable source-specific observation components. Here, t denotes the current policy frame, k denotes a future step within the action chunk, and H denotes the chunk horizon. Superscripts B, H, and E denote Body-camera, Hand-camera, and TCP quantities, respectively. Superscripts ref” and rgb” denote collection-time reference and RGB-reconstructed trajectories, respectively. The policy predicts a horizon-H action chunk: At+H−1∈ℝH×16,H=50,A_t+H-1 ^H× 16, H=50, (6) with each step represented canonically as: at,k a_t,k =[vx,vy,ψ˙,ξEt,k,q0:5,g], =[v_x,v_y, ψ,\ ξ^E_t,k,\ q_0:5,g], (7) ξt,kE ξ^E_t,k =log(TEt+k+1Et)=[Δpt,k,ϕt,k]∈ℝ6. = \! (T^E_t_E_t+k+1 )=[ p_t,k, _t,k] ^6. where k=0,…,H−1k=0,…,H-1, ϕt,k _t,k is the rotation-vector logarithm, q0:5q_0:5 are the six arm-joint targets, and g is the gripper target. Eq. (6) sets a 50-step chunk at the 15-Hz policy clock, and Eq. (7) expresses base velocities in the current Body frame and ξt,kEξ^E_t,k in the current TCP frame. The action mask Mta∈0,1H×16M_t^a∈0,1^H× 16 indicates the valid action dimensions. The same masks are used in Sec. V and exclude missing values rather than treating them as zeros. V VOMMI Fig. 3: Overview of VOMMI. The online branch estimates multi-horizon local motion from RGB and conditions the base-action residual. The offline branch combines the causal estimator with frozen VGGT anchors, initializes the Body trajectory at its first valid pose, and applies planar Body compensation to the Hand stream before forming demonstration targets. Reference pose supervision available only during data collection. VOMMI converts portable RGB demonstrations into a unified whole-body policy through the pipeline shown in Fig. 3. Local base and E actions provide a common representation across demonstrator and robot embodiments. R2-VO refines demonstration trajectories offline using sparse geometric anchors and provides causal body-motion estimates from RGB observations online. We encode these online estimates as a multi-horizon motion token and integrate it into the policy through an action-group residual adapter, restricting motion-conditioned updates to the body actions. V-A Local Cross-Embodiment Action Representation We express body motion commands in the current body frame and E targets in the current TCP frame. This representation provides a common action interface across portable and robot data while avoiding supervision tied to an arbitrary world origin. A per-sample mask, Mia∈0,1H×16M_i^a∈\0,1\^H× 16, selects the action dimensions available from each source, and the flow loss is normalized over the valid dimensions before being averaged across the batch. Local actions reduce sensitivity to global pose errors. Let TtT_t be the true pose and T^t=GtTt T_t=G_tT_t its estimate, where GtG_t is a time-varying world-frame error. We define the current-frame relative target as: ΔT^t,i=T^t−1T^t+i=Tt−1Gt−1Gt+iTt+i, T_t,i= T_t^-1 T_t+i=T_t^-1G_t^-1G_t+iT_t+i, (8) Eq. (8) cancels constant global translation and yaw offsets when Gt=Gt+iG_t=G_t+i. Otherwise, only the local error increment Gt−1Gt+iG_t^-1G_t+i enters the relative target. The same property applies independently to the Body and Hand streams, even though their tracking frames are not mutually registered. Accordingly, we formulate relative targets independently within each stream. V-B Body-Hand Alignment and Planar Compensation A body-relative hand trajectory requires the hand TCP and body pose to be expressed in a common coordinate system. Let WHW_H and WBW_B denote the tracking worlds of the hand and Body streams, CHC_H and CBC_B their camera frames, E the hand TCP, and B the demonstrator body frame. For a setting with a verified rigid cross-device registration ABH=TWHWBA_BH=T^W_B_W_H and fixed camera extrinsics XH=TECHX_H=T^C_H_E, XB=TBCBX_B=T^C_B_B, the trajectories would satisfy: TEWH(t) T^W_H_E(t) =TCHWH(t)XH, =T^W_H_C_H(t)X_H, (9) TBWB(t) T^W_B_B(t) =TCBWB(t)XB, =T^W_B_C_B(t)X_B, (10) T^EWB(t) T^W_B_E(t) =A^BHTEWH(t). = A_BH\,T^W_H_E(t). (11) Applying Eq. (4) to the registered poses in Eqs. (9)–(11) gives the body-relative TCP pose: TEB(t)=(TBWB(t))−1T^EWB(t).T^B_E(t)= (T^W_B_B(t) )^-1 T^W_B_E(t). (12) For RGB-based Body trajectory reconstruction, we refine the planar motion estimated from RGB observations using a frozen VGGT+anchor branch. Let T~BR(t)∈SE(2) T^R_B(t) (2) denote the resulting canonicalized Body trajectory, where VGGT provides global geometric constraints and sparse anchors recover metric consistency to mitigate long-term VO drift. We initialize this trajectory at its first valid timestamp t0t_0 as: Ginit=T^BR(t0)(T~BR(t0))−1,T^BR(t)=GinitT~BR(t).G_init= T^R_B(t_0) ( T^R_B(t_0) )^-1, T^R_B(t)=G_init T^R_B(t). (13) The trajectory initialized in Eq. (13) supplies planar displacement for compensation. Under a constant left gauge, its relative increments are invariant as shown in Eq. (14): T~B(t)−1T~B(t+k)=TB(t)−1TB(t+k). T_B(t)^-1 T_B(t+k)=T_B(t)^-1T_B(t+k). (14) Planar compensation projects the Body motion to SE(2)SE(2) and lifts it to SE(3)SE(3): TB,∥WB(t)=[Rz(ψB(t))[xB(t),yB(t),0]⊤1],T^W_B_B, (t)= bmatrixR_z( _B(t))&[x_B(t),y_B(t),0] \\ 0 &1 bmatrix, (15) where (xB,yB,ψB)(x_B,y_B, _B) are the body translation and yaw in the common reference. Using Eq. (15) in Eq. (12) gives the planar-compensated Hand pose: T¯E|B∥(t)=(TB,∥WB(t))−1T^EWB(t), T_E|B (t)= (T^W_B_B, (t) )^-1 T^W_B_E(t), (16) and the current-relative future target is: At,kee,∥=(T¯E|B∥(t))−1T¯E|B∥(t+k+1).A^e, _t,k= ( T_E|B (t) )^-1 T_E|B (t+k+1). (17) For q∈ref,rgbq∈\ref,rgb\, let Bq(t)p_B^q(t) and ψBq(t) _B^q(t) denote the canonical Body translation and heading. Here, q=refq=ref denotes the collection-time reference and q=rgbq=rgb denotes the RGB reconstruction. After interpolation to hand timestamps, we form the planar body transform: TB,qWB(t)=[Rz(ψBq(t))[pB,xq(t),pB,yq(t),0]⊤1].T^W_B_B,q(t)= bmatrixR_z( _B^q(t))&[p^q_B,x(t),p^q_B,y(t),0] \\ 0 &1 bmatrix. (18) The body-normalized TCP pose and relative target are: T¯E|Bq(t) T_E|B^q(t) =(TB,qWB(t))−1T^EWB(t), = (T^W_B_B,q(t) )^-1 T^W_B_E(t), (19) At,kee,q A^e,q_t,k =(T¯E|Bq(t))−1T¯E|Bq(t+k+1). = ( T_E|B^q(t) )^-1 T_E|B^q(t+k+1). Portable RGB source-replacement and VLA experiments use Eqs. (18)–(19) with q=rgbq=rgb to compensate Body translation in x and y and Body yaw. These equations apply the planar-compensation rule in Eqs. (16)–(17) to the selected Body trajectory source. During deployment, the robot-side adapter composes the predicted base transform TB,cmdWR(t+k)T^W_R_B,cmd(t+k) and emits: TEt+k,cmdEt=(TEWR(t))−1TB,cmdWR(t+k)T¯E|Bdes(t+k),T^E_t_E_t+k,cmd= (T^W_R_E(t) )^-1T^W_R_B,cmd(t+k)\, T_E|B^des(t+k), (20) where T¯E|Bdes T_E|B^des denotes the planar-compensated target. Eq. (20) yields the six current-TCP-relative E dimensions. V-C R2-VO R2-VO provides local metric motion from RGB in two complementary modes. The causal branch produces motion tokens for deployment, while the offline branch refines trajectories during demonstration construction. Both branches share the same frozen motion estimator. V-C1 Causal Local Motion Estimation For consecutive Body-camera frames, the collection-time tracker provides the planar motion target Δξt,1=[Δx,Δy,Δψ] _t,1=[ x, y, ψ]. We extract visual motion descriptors from optical flow, geometric cues, and track quality statistics, resulting in a 206-dimensional descriptor. A seven-frame temporal window augments this descriptor to 317 dimensions by incorporating temporal motion information. An extremely randomized tree regressor with 32 trees predicts the one-step motion target. During inference, the estimator uses only past RGB frames, camera intrinsics, and timestamps. Atomic predictions are resampled to the 15-Hz clock and composed over horizons of k∈2,3,6,12k∈\2,3,6,12\ frames, corresponding to 0.13,0.20,0.40,0.800.13,0.20,0.40,0.80 s. Starting from (x,y,θ)=(0,0,0)(x,y,θ)=(0,0,0), each step integrates the body-frame velocities [vx,vy,ω][v_x,v_y,ω] as follows: θj+12 _j+ 12 =θj+12ωjΔt, = _j+ 12 _j t, (21) xj+1 x_j+1 =xj+(vx,jcosθj+12−vy,jsinθj+12)Δt, =x_j+(v_x,j _j+ 12-v_y,j _j+ 12) t, (22) yj+1 y_j+1 =yj+(vx,jsinθj+12+vy,jcosθj+12)Δt, =y_j+(v_x,j _j+ 12+v_y,j _j+ 12) t, (23) θj+1 _j+1 =θj+ωjΔt, = _j+ _j t, (24) where Δt=1/15 t=1/15. Each horizon is represented by: [x,y,sinθ,cosθ,logσx,logσy,logσθ],[x,y, θ, θ, _x, _y, _θ], (25) with σx=σy=σθ=0.02k/2 _x= _y= _θ=0.02 k/2. Eqs. (21)–(24) compose source-Body-frame increments in each horizon’s window-start frame. Concatenating four encodings from Eq. (25) with a history-validity bit yields νt∈ℝ29 _t ^29. The incomplete history is assigned νt=0 _t=0 and qt=0q_t=0. The resulting token encodes causal short-horizon Body motion. V-C2 Offline Drift Refinement For offline action construction, R2-VO combines the integrated causal trajectory with 1-Hz VGGT-1B anchors [18]. A low-capacity canonicalizer C maps VGGT coordinates to the metric initial-camera frame. Let gt,xyg_t,xy denote the sparse metric anchors obtained from canonicalized VGGT geometry, ct,xyc_t,xy denote the integrated causal VO trajectory, and L(cxy)tL(c_xy)_t denote the causal trajectory evaluated at the anchor timestamps. The refined trajectory preserves short-term VO motion while correcting accumulated drift through sparse geometric constraints: p^t,xyoff=gt,xy+ct,xy−L(cxy)t. p^off_t,xy=g_t,xy+c_t,xy-L(c_xy)_t. (26) The canonicalizer is a weighted Ridge map from first-frame-centered VGGT positions and temporal features to metric reference positions in the initial-camera frame: min∑eΘ1Ne∑t‖[(ϕ(ue,t)−μ)/s1]Θ−pe,tref‖22+λ‖Θ‖F2, _ _e 1N_e _t \| bmatrix(φ(u_e,t)-μ)/s\\ 1 bmatrix -p^ref_e,t \|_2^2+λ\| \|_F^2, (27) In Eq. (27), ue,tu_e,t is the normalized VGGT position, and ϕφ supplies linear, time-augmented, or quadratic features for metric translation prediction. For the Hand stream, sparse anchors refine the vertical component during offline action construction, while the Hand VO branch predicts the complete local TCP SE(3)SE(3) increment, including rotation, from RGB observations. The TCP-relative pose in Eq. (1) serves as the training and evaluation target. For source-replaced demonstrations, the offline Body trajectory from Eq. (26) is interpolated to the Hand timestamps and inserted into Eq. (18) to construct demonstration labels. Online inference uses causal local increments. V-D VO-Conditioned Action-Group Adaptation To condition the base actions on visual motion, we augment the pretrained action expert with separate residual branches for base and manipulation actions. Let ht,k∈ℝ1024h_t,k ^1024 and ut,k0∈ℝ32u^0_t,k ^32 denote the expert hidden state and flow prediction at step (t,k)(t,k). A LayerNorm-MLP P projects the motion token, centered and gated by a validity indicator: zt=qt[P(νt)−P(0)].z_t=q_t[P( _t)-P(0)]. (28) After lifting ztz_t from Eq. (28) to the hidden dimension, the base residual is: rt,kb=Rb([ht,k⊙Wzzt,zt])−Rb(0),r^b_t,k=R_b([h_t,k W_zz_t,z_t])-R_b(0), (29) while the manipulation residual rt,km=Rm(ht,k)r^m_t,k=R_m(h_t,k) does not receive body motion. Independent gates (gb,gm)=σ(Wrht,k)(g^b,g^m)=σ(W_rh_t,k) combine the two paths: u^t,k=ut,k0+gt,kbrt,kbMb+gt,kmrt,kmMm, u_t,k=u^0_t,k+g^b_t,kr^b_t,kM_b+g^m_t,kr^m_t,kM_m, (30) where MbM_b selects dimensions 0:30:3, MmM_m selects 3:163:16, and both are zero on padded dimensions. Zero-initialized outputs preserve the pretrained policy at initialization. Eqs. (29)–(30) restrict motion-conditioned updates to the base group. Router targets are derived from base and manipulation activity. The training objective is: ℒ=1B∑iℒflow(i)+0.1ℒrouter,L= 1B _iL_flow^(i)+0.1\,L_router, (31) where Eq. (31) averages the validity-normalized per-sample flow losses. Each flow loss applies the corresponding action masks. VI Experiments In this section, we evaluate the impact of R2-VO post-processing on data quality, the performance of the VO-conditioned VLA policy, and real-world deployment across diverse mobile manipulation tasks. The experiments address the following research questions: • Q1: How accurately can R2-VO reconstruct tracking-derived planar SE(2)SE(2) and local TCP SE(3)SE(3) pose references from RGB data? • Q2: Do VO-refined data improve real-world rollouts? • Q3: How does the local motion token affect long-horizon mobile manipulation? • Q4: Which components contribute most significantly to the R2-VO and VOMMI? VI-A Experimental Setting Tasks and evaluation metrics. Our benchmark contains three mobile-manipulation tasks: moving across barriers to pick up different fruits, moving in and out of a room to manipulate a bottle, and opening a cabinet door to place a folded box. We use collection-time tracking-derived references for both streams: a planar metric motion reference for Body and a tracking-derived local TCP SE(3)SE(3) pose reference for Hand. For data post-processing, we report trajectory error against these references. For real-world rollout, we evaluate the wheel-legged DeepRobotics M20S equipped with a CM1 manipulator over 20 independent trials per condition, reporting task success rate and action deviation from the reference commands. Baselines. To evaluate VOMMI, we compare it with DROID-W [16], DPVO [17], MASt3R-SLAM [22], and CUT3R [23], and VGGT-SLAM 2.0 [19] for VO performance, as well as OpenPI 0.5 [24], GR00T N1.6 [25], and X-VLA [26] for real-world evaluation. VI-B Main Experiment Results VO motion accuracy (Q1). We evaluate the frozen RGB-based R2-VO on the fixed 75-episode held-out test split defined above. Table I reports the per-step relative-motion reconstruction error for the Body and Hand streams in their local references. Source-replacement and VLA use the RGB-source planar compensation (q=rgbq=rgb) defined in Sec. V-B. Body-span uses the same held-out split. TABLE I: Local and accumulated motion errors on VOMMI trajectories. Lower is better. Body SE(2)SE(2) Hand SE(3)SE(3) Local (mean/P90) Accumulated Local (mean/P90) Accumulated Method Trans. (cm) Yaw (∘) XY ATE (cm) XY End (cm) Trans. (cm) Rot. (∘) 3D ATE (cm) 3D End (cm) XY ATE (cm) XY End (cm) Ours (online) 0.26/0.54 0.092/0.227 15.65 19.29 0.60/1.08 0.346/0.717 71.82 60.42 71.51 60.30 Ours (offline) 0.24/0.58 0.093/0.224 5.97 5.33 0.60/1.08 0.340/0.704 50.47 56.68 50.23 56.66 DPVO 1.24/2.86 0.412 0.537 28.09 27.86 14.48/21.90† 3.887/8.591† 220.28 219.21 220.16 219.21 DROID-SLAM 1.50/3.21 0.309/0.662 83.86 62.67 2.85/4.62† 2.358/4.834† 124.54 109.75 124.16 109.45 DROID-W 0.45/1.07 0.140/0.345 8.54 8.02 4.91/7.83† 3.202/6.665† 126.12 138.17 125.80 137.86 VGGT-SLAM 2.0 0.86/1.63 0.227/0.463 6.88 9.95 10.95/19.44† 6.100/10.443† 95.41 83.90 94.78 83.68 CUT3R 24.52/52.81 0.906/1.954 23.91 25.01 23.69/41.60† 10.748/23.975† 83.35 103.57 83.02 103.57 MASt3R-SLAM 63.71/106.80 0.581/1.137 66.03 57.43 165.25/235.28† 35.939/59.034† 78.88 84.80 78.42 85.31 At the deployed 15-Hz step, the causal Ours branch gives 0.26/0.54 cm Body translation error (mean/P90). Its TCP-relative Hand-stream evaluator gives 0.601/1.081 cm translation and 0.346/0.717∘ SO(3)SO(3) rotation. With complete-sequence access for demonstration conversion, the Ours Offline (VGGT+anchor) branch gives 0.244/0.592 cm and 0.093/0.224∘ for Body, and 0.600/1.082 cm and 0.340/0.704∘ for hand at the same 67-ms reference span. On an Intel Core i5-9600KF CPU, the causal Body branch takes 10.02 ms on average (P95: 13.76 ms), excluding image capture and decoding. Table I uses local increments: Body is planar SE(2)SE(2) and Hand is TCP-relative SE(3)SE(3). Ours and DPVO use matched stored-frame spans, whereas CUT3R and MASt3R-SLAM retain native sparse steps from 1-Hz RGB bundles. Thus, the baseline rows are not all 15-Hz runs. At the deployed 15-Hz step, R2-VO lowers mean translation error by 80.4% and 83.8% relative to DPVO and DROID-SLAM, respectively. At 267 ms, it retains 0.878 cm mean error, corresponding to reductions of 79.9% and 84.1%. The measured latency is compatible with online 15-Hz inference. Accumulated trajectory drift is reported in the accumulated-error columns of Table I. Table I groups local and accumulated errors by stream. Bold entries indicate minima before rounding. Integrating the causal Body branch gives 15.65 cm ATE and 19.29 cm endpoint error. The offline VGGT+anchor variant reduces these to 5.97 cm and 5.33 cm. The accumulated metrics are therefore trajectory diagnostics rather than deployment-rate runtime rankings. Offline reconstruction reduces mean absolute Body/Hand trajectory error by 24.6% versus the best baseline for each stream. Fig. 4: Camera trajectories reconstruction on held-out episode trail. The online stream and reference are projected to the ground plane. Offline Body scores in Table I use the common x–y state. Fig. 5: Qualitative M20S rollouts on three long-horizon mobile-manipulation tasks. The upper rows show representative stages from navigation and approach to object interaction and task completion. The bottom row shows representative failure cases, marked with red crosses. TABLE I: Complete-task success rate (%) over 20 trials per task. Method Barriers–fruit Room–bottle Cabinet–box OpenPI 0.5 35 50 65 GR00T N1.6 20 25 10 X-VLA 25 35 30 VOMMI (ours) 65 60 50 Overall performance on mobile manipulation tasks (Q2). Figure 5 shows representative rollouts. Across the tasks in Table I, mean success is 58.3% for VOMMI versus 50.0% for OpenPI 0.5, an 8.3-percentage-point gain. Table IV evaluates policy predictions against reference actions on 15 real-robot trajectories. M20-trained uses robot demonstrations. VOMMI-R/A use portable demonstrations with compensated current-TCP-relative targets and uncompensated absolute targets. Metrics pool valid actions over 50-step horizons: Base MAE averages |vx−vx∗||v_x-v_x^*| and |vy−vy∗||v_y-v_y^*|. Yaw MAE averages |ψ˙−ψ˙∗|| ψ- ψ^*|, where stars denote reference commands. E errors compare forward-kinematic TCP poses in the current TCP frame. VOMMI-R reduces base MAE from 0.044 to 0.036 m/s (18.2%) relative to M20-trained. Compared with VOMMI-A, E translation/rotation errors fall from 0.569 cm/1.091∘1.091 to 0.170 cm/0.453∘0.453 . TABLE IV: Open-loop action prediction errors. Policy Base MAE (m/s) Yaw MAE (rad/s) E trans. mean (cm) E rot. mean (∘) E trans. max (cm) M20-trained 0.044 0.107 0.155 0.372 4.32 VOMMI-R 0.036 0.104 0.170 0.453 4.55 VOMMI-A 0.079 0.204 0.569 1.091 13.96 VI-C Ablation Study TABLE V: Residual ablation: flow loss (×10−2× 10^-2). Variant Residual target Overall Navigation Port. manip. Parent None ±0.063.86\!±\!0.06 ±0.063.82\!±\!0.06 ±0.193.99\!±\!0.19 Base-VO Base (VO) ±0.033.91\!±\!0.03 ±0.073.90\!±\!0.07 ±0.163.95\!±\!0.16 All-Action-VO All actions (VO) ±0.053.83\!±\!0.05 ±0.113.82\!±\!0.11 ±0.223.88\!±\!0.22 VOMMI Base (VO) + manip. ±0.023.79\!±\!0.02 ±0.073.76\!±\!0.07 ±0.203.89\!±\!0.20 Fig. 6: Open-loop comparison using global observations and first-step base actions (vx,vy,yaw)(v_x,v_y,yaw), with the residual VO turn highlighted at t=3.73t=3.73 s. Local-action robustness (Q3). Figure 6 further examines how motion conditioning changes the temporal action response on one real-robot episode. At the highlighted observation, the OpenPI baseline has returned to near-zero yaw, whereas the VO-conditioned policy retains a positive yaw command. This case study shows that local-motion conditioning materially changes policy dynamics. VO conditioning and residual design (Q4). Table V compares Parent (no VO or residual), Base-VO (a VO-conditioned base residual), All-Action-VO (VO-conditioned residuals for all actions), and VOMMI (a VO-conditioned base residual plus a manipulation residual driven only by ht,kh_t,k). We report overall, navigation, and portable-manipulation flow losses. Within-source shuffled VO provides a paired motion-content check for the VO variants. VII Limitation and Conclusion Portable VOMMI demonstrations support local-motion prediction and produce actions that closely match the teleoperation references in in-domain real-world tasks, demonstrating effectiveness on long-horizon mobile manipulation tasks. These benefits are most pronounced in settings with substantial navigation demands, while gains are relatively limited in tasks dominated by manipulation. Our method has been validated on the M20S platform, while further cross-embodiment evaluation remains necessary. VOMMI provides a portable RGB framework for learning long-horizon mobile manipulation from robot-free demonstrations. It combines provenance-aware masked observations, causal body-frame motion, planar Body-to-Hand compensation, and action-group residuals that condition base actions on visual motion. Collection-time tracking poses supervise the RGB motion estimator and the local TCP SE(3)SE(3) predictor during data construction. Together, these components form a unified pipeline from portable demonstrations to whole-body policy learning. Acknowledgments AI-Use Disclosure. OpenAI Codex assisted with drafting and revising selected manuscript text, refining figure captions, checking reference consistency, formatting, and generating plotting code from author-provided statistics. The authors reviewed and verified all AI-assisted content and take full responsibility for the final manuscript. References [1] H. Jiang, J. Chen, Q. Bu, L. Chen, M. Shi, Y. Zhang, D. Li, C. Suo, C. Wang, Z. Peng, and H. Li (2026) WholeBodyVLA: towards unified latent VLA for whole-body loco-manipulation control. In International Conference on Learning Representations, Cited by: §I, §I-B. [2] Y. Hu, H. Zhu, B. Zheng, Y. Hu, T. Zhang, Z. Chen, J. Zhao, R. Nai, and Y. Gao (2026) OpenHLM: an empirical recipe for whole-body humanoid loco-manipulation. arXiv preprint arXiv:2606.22174. External Links: 2606.22174, Link Cited by: §I. [3] Z. Li, Z. Zhang, Y. Wei, W. Zhang, X. Yuan, P. Zhi, G. Li, X. Guo, F. Gao, J. Yang, and S. Zhang (2026) ω-0: a latent predictive world action model for concurrent humanoid loco-manipulation. arXiv preprint arXiv:2608.06375. External Links: 2608.06375, Link Cited by: §I. [4] J. Zheng, T. Ma, Y. Fan, Z. Wang, S. Yang, and J. Liang (2026) MotionWAM: towards foundation world action models for real-time humanoid loco-manipulation. arXiv preprint arXiv:2606.09215. External Links: 2606.09215, Link Cited by: §I. [5] C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song (2024) Universal manipulation interface: in-the-wild robot teaching without in-the-wild robots. arXiv preprint arXiv:2402.10329. External Links: Link Cited by: §I, §I-A, TABLE I. [6] Zhaxizhuoma, K. Liu, C. Guan, Z. Jia, Z. Wu, X. Liu, T. Wang, S. Liang, P. Chen, P. Zhang, H. Song, D. Qu, D. Wang, Z. Wang, N. Cao, Y. Ding, B. Zhao, and X. Li (2025) FastUMI: a scalable and hardware-independent universal manipulation interface with dataset. arXiv preprint arXiv:2409.19499. External Links: 2409.19499, Link Cited by: §I, TABLE I. [7] A. Patel, B. Pekarek, J. E. C. Hernandez, and S. Song (2026) Behavior prompting policy: demonstrations as prompts for manipulation. arXiv preprint arXiv:2606.30457. External Links: Link Cited by: §I, TABLE I. [8] X. Xu, J. Park, H. Zhang, E. Cousineau, A. Bhat, J. Barreiros, D. Wang, and S. Song (2026) HoMMI: learning whole-body mobile manipulation from human demonstrations. arXiv preprint arXiv:2603.03243. External Links: Link Cited by: §I, §I-A. [9] H. Huang, H. Dong, and H. Dong (2026) Mobile UMI: cross-view diffusion policy with decoupled kinematics for mobile manipulation. arXiv preprint arXiv:2605.20894. External Links: Link Cited by: §I, §I-A, TABLE I. [10] M. Shi, S. Peng, J. Chen, H. Jiang, Y. Li, D. Huang, P. Luo, H. Li, and L. Chen (2026) EgoHumanoid: unlocking in-the-wild loco-manipulation with robot-free egocentric demonstration. arXiv preprint arXiv:2602.10106. External Links: Link Cited by: §I, §I-A. [11] H. Ha, Y. Gao, Z. Fu, J. Tan, and S. Song (2024) UMI on legs: making manipulation policies mobile with manipulation-centric whole-body controllers. arXiv preprint arXiv:2407.10353. External Links: Link Cited by: §I, §I-A. [12] S. Yenamandra, A. Ramachandran, K. Yadav, A. Wang, M. Khanna, et al. (2023) HomeRobot: open-vocabulary mobile manipulation. arXiv preprint arXiv:2306.11565. External Links: Link Cited by: §I-B. [13] P. Ding, H. Zhao, W. Zhang, W. Song, M. Zhang, S. Huang, N. Yang, and D. Wang (2025) QUAR-VLA: vision-language-action model for quadruped robots. arXiv preprint arXiv:2312.14457. External Links: 2312.14457, Link Cited by: §I-B. [14] I. Chuang, J. Zou, A. Lee, D. Gao, and I. Soltani (2026) Look, Focus, Act: efficient and robust robot learning via human gaze and foveated vision transformers. arXiv preprint arXiv:2507.15833. External Links: 2507.15833, Link Cited by: §I-B. [15] S. Ma, B. Zhang, Y. Zhang, Q. Wu, J. Zhai, D. Wei, and Q. Yu (2026) DECOWAM: decoupled whole-body world-action model for legged mobile manipulation. arXiv preprint arXiv:2608.20114. External Links: 2608.20114, Link Cited by: §I-B. [16] Z. Teed and J. Deng (2021) DROID-SLAM: deep visual SLAM for monocular, stereo, and RGB-D cameras. In Advances in Neural Information Processing Systems, Cited by: §I-C, §VI-A. [17] Z. Teed, L. Lipson, and J. Deng (2022) Deep patch visual odometry. In European Conf. Computer Vision, Cited by: §I-C, §VI-A. [18] J. Wang, M. Chen, N. Karaev, A. Vedaldi, C. Rupprecht, and D. Novotny (2025) VGGT: visual geometry grounded transformer. In Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, Cited by: §I-C, §V-C2. [19] D. Maggio and L. Carlone (2026) VGGT-SLAM 2.0: real-time dense feed-forward scene reconstruction. arXiv preprint arXiv:2601.19887. External Links: Document, Link Cited by: §I-C, §VI-A. [20] H. Karnan, G. Warnell, X. Xiao, and P. Stone (2021) VOILA: visual-observation-only imitation learning for autonomous navigation. arXiv preprint arXiv:2105.09371. External Links: 2105.09371, Link Cited by: §I-C. [21] S. Kareer, D. Patel, R. Punamiya, P. Mathur, S. Cheng, C. Wang, J. Hoffman, and D. Xu (2024) EgoMimic: scaling imitation learning via egocentric video. arXiv preprint arXiv:2410.24221. External Links: 2410.24221, Link Cited by: §I-C. [22] R. Murai, E. Dexheimer, and A. J. Davison (2025) MASt3R-SLAM: real-time dense SLAM with 3D reconstruction priors. arXiv preprint arXiv:2412.12392. External Links: 2412.12392, Link Cited by: §VI-A. [23] Q. Wang, Y. Zhang, A. Holynski, A. A. Efros, and A. Kanazawa (2025) Continuous 3D perception model with persistent state. arXiv preprint arXiv:2501.12387. External Links: 2501.12387, Link Cited by: §VI-A. [24] Physical Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, et al. (2025) π0.5 _0.5: A vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054. External Links: Link Cited by: §VI-A. [25] J. Bjorck, F. Castaneda, N. Cherniadev, X. Da, R. Ding, L. Fan, Y. Fang, D. Fox, F. Hu, S. Huang, J. Jang, et al. (2025) GR00T N1: an open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734. External Links: 2503.14734, Link Cited by: §VI-A. [26] J. Zheng, J. Li, Z. Wang, D. Liu, X. Kang, Y. Feng, Y. Zheng, J. Zou, Y. Chen, J. Zeng, Y. Zhang, J. Pang, J. Liu, T. Wang, and X. Zhan (2025) X-VLA: soft-prompted transformer as scalable cross-embodiment vision-language-action model. arXiv preprint arXiv:2510.10274. External Links: 2510.10274, Link Cited by: §VI-A.