Paper deep dive

DepthWorld: 3D World Model for Robot Manipulation

Jai Bardhan, Josef Sivic, Vladimir Petrik

Year: 2026Venue: arXiv preprintArea: cs.ROType: PreprintEmbeddings: 107

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:World models offer a data-driven alternative to traditional simulators for robotics, with applications spanning policy evaluation, improvement, and planning. All of these uses depend on faithful 3D geometry, yet current video-based world models are trained on RGB alone and produce rollouts that look correct frame-by-frame but do not compose into a consistent 3D world. Closing this gap requires progress on two fronts: large-scale 3D supervision for manipulation, and an architecture that can absorb it without disturbing strong pretrained video priors. We introduce a calibration pipeline that combines learned stereo depth with a joint factor graph, pooling all episodes collected from the same physical robot to recover its shared kinematic parameters alongside per-scene extrinsics. Applied to the DROID dataset, this yields DROID-3D, a calibrated 3D dataset providing dense metric depth and recalibrated multi-view extrinsics (achieving <0.7 px reprojection error on 90% of episodes for external cameras). We then train DepthWorld, a Stable Video Diffusion-based world model that jointly predicts multi-view RGB and depth via spatial latent tiling, leaving the pretrained Variational Autoencoder (VAE) unchanged. Depth supervision improves RGB prediction itself by +1.48 dB PSNR over an identical RGB-only baseline at equal training budget, while simultaneously yielding accurate metric depth for downstream geometric reasoning.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

104,189 characters extracted from source content.

Expand or collapse full text

DepthWorld: 3D World Model for Robot Manipulation Jai Bardhan Email: jai.bardhan@cvut.cz Josef Sivic Email: josef.sivic@cvut.cz Vladimir Petrik Email: vladimir.petrik@cvut.cz Czech Institute of Informatics Robotics Cybernetics Affiliation: Czech Technical University in Prague w.jaibardhan.com/depthworld Abstract World models offer a data-driven alternative to traditional simulators for robotics, with applications spanning policy evaluation, improvement, and planning. All of these uses depend on faithful 3D geometry, yet current video-based world models are trained on RGB alone and produce rollouts that look correct frame-by-frame but do not compose into a consistent 3D world. Closing this gap requires progress on two fronts: large-scale 3D supervision for manipulation, and an architecture that can absorb it without disturbing strong pretrained video priors. We introduce a calibration pipeline that combines learned stereo depth with a joint factor graph, pooling all episodes collected from the same physical robot to recover its shared kinematic parameters alongside per-scene extrinsics. Applied to the DROID dataset, this yields DROID-3D, a calibrated 3D dataset providing dense metric depth and recalibrated multi-view extrinsics (achieving <0.7<0.7px reprojection error on 90% of episodes for external cameras). We then train DepthWorld, a Stable Video Diffusion-based world model that jointly predicts multi-view RGB and depth via spatial latent tiling, leaving the pretrained Variational Autoencoder (VAE) unchanged. Depth supervision improves RGB prediction itself by +1.48 dB PSNR over an identical RGB-only baseline at equal training budget, while simultaneously yielding accurate metric depth for downstream geometric reasoning. Keywords: 3D World Models, Robots, Learning 1 Introduction World models are a promising data-driven alternative to traditional simulators for robot learning, with the potential to simulate manipulation tasks that are difficult to capture with physics-based engines. Recent work has shown that finetuning pretrained video diffusion backbones with action conditioning on large-scale robotics datasets yields models that generate faithful action-consistent rollouts [12, 2]. Such rollouts are increasingly used for policy evaluation [12, 2], policy improvement [12, 8], and planning [49, 34]—uses in which the geometric fidelity of the rollout determines whether the signal is usable. However, current video world models miss this geometric structure entirely. They are trained on RGB alone, and the training loss provides no signal to resolve the underlying 3D ambiguity of pixel sequences. The resulting models produce visually plausible rollouts whose geometry is internally incoherent: predicted depth disagrees across views of the same scene, the model struggles with geometric reasoning about occlusion and contact (Figure 4), and object shape and identity degrade over the course of a rollout (Figure 1). For downstream uses that read off the rollout, this incoherence is the limiting factor. The natural fix is to supervise geometry directly during training, the recipe behind recent geometric foundation models [40, 19, 39, 41, 21, 17]. However, this approach does not directly apply to robot world models for two reasons. First, geometric foundation models earn their generalization by training on datasets that already carry metric depth and calibrated poses [45, 4, 33, 20, 44, 31]. Robot teleoperation datasets such as DROID [18] capture the appearance and dynamics of real manipulation at scale, but their geometric annotations are secondary outputs of the collection pipeline and not at the quality geometric pretraining demands. Lifting them into supervision-quality 3D is non-trivial: manipulation scenes are dominated by textureless and specular surfaces, and even a rig that is internally self-consistent under bundle adjustment can be kinematically decoupled from from the robot’s own coordinate frame, severing the critical link to the robot’s proprioception and end-effector actions. Second, current video world models rely on delicate, pretrained image and video diffusion priors. Conventional architectural modifications for incorporating spatial modalities—such as expanding the input channels of the VAE or introducing parallel dual-branch U-Nets—require destructive weight re-initializations. These interventions risk corrupting the strong visual and motion priors that make these foundation backbones effective in the first place. We address both through the following contributions: 1. We introduce a calibration pipeline for recovering metric depth and accurate multi-view extrinsics from any multi-view stereo teleoperation collection with a known URDF. The pipeline combines learned stereo depth with a joint factor graph optimization that pools all episodes of the same physical robot to recover its shared kinematic parameters (joint offsets, hand-eye calibration) alongside per-scene extrinsics. Applied to DROID [18], this yields DROID-3D—a calibrated 3D corpus providing dense metric depth and recalibrated multi-view extrinsics for over 70,000 episodes. 2. We train DepthWorld, a Stable Video Diffusion-based world model that jointly predicts multi-view RGB and depth via spatial latent tiling that leaves the pretrained VAE unchanged. 3. We show that depth supervision improves RGB prediction: at equal training budget, DepthWorld gains +1.48+1.48 dB PSNR over an RGB-only baseline of identical architecture and data. Figure 1: DepthWorld: The RGB-only baseline fails to maintain the shape of the cutlery, and the object disappears. DepthWorld capitalizes on the depth inputs to accurately maintain the cutlery’s shape throughout the rollout while producing high-quality metric depth maps. 2 Related Work Robotic World Models. Action-conditioned video models have emerged as a learned substitute for physics-based simulators in manipulation, with applications spanning policy evaluation [12, 2], policy improvement [12, 8], and visual planning [49, 34]. Recent works finetune pretrained video diffusion backbones with proprioceptive action conditioning: Ctrl-World [12], an SVD-based controllable world model trained on DROID [18] forms our backbone; IRASim [51] along with other works [15] systematize the policy-evaluation use case. All these uses presuppose geometrically faithful rollouts. Current video world models are trained on RGB alone, which does not generally serve as a strong geometric prior. We close this gap by jointly predicting depth and supervising with point maps. Geometric Foundation Models. A recent line of work regresses dense metric geometry directly from images: DUSt3R [40] and MASt3R [19] predict point maps from image pairs, VGGT [39] scales this to many-view inputs through intermediate camera and DPT [32] heads, and MapAnything [17] unifies a broader class of supervision targets. These models inherit their generalization from large datasets [45, 4, 33, 20, 44, 31, 50] carrying dense metric depth and calibrated poses, drawn from structured light, LiDAR, and offline reconstruction. None of these covers the visual distribution of real manipulation, with its indoor setting and extreme viewpoint differences between fixed external and wrist-mounted cameras. DROID-3D is our contribution at this distribution and scale. Depth-Aware Video Diffusion. Stable Video Diffusion (SVD) [5] provides strong appearance and motion priors but is trained on RGB alone. Three approaches have been explored to add depth into this prior. Marigold [16] showed that an image diffusion VAE encodes depth maps essentially without loss, repurposing the model as a depth predictor; DepthCrafter [13] extends this to video. Joint RGB–depth prediction has been pursued through channel expansion [49, 47], which adds depth channels to the VAE’s input and output convolutions and through dual-branch architectures [30] that run a parallel U-Net for depth with cross-connections to the RGB branch. Both modify the pretrained backbone substantially, preventing weight transfer. Our spatial latent tiling extends Marigold’s observation from single-image depth prediction to multi-view RGB–depth video prediction. Robot Datasets and Their Calibration. Large-scale teleoperation datasets have grown rapidly in size and diversity—BridgeData V2 [38], RT-1 [6] and the Open X-Embodiment aggregation [29], RH20T [10], and DROID [18]—however; almost all carry geometry as a byproduct of the capture pipeline: monocular RGB in Open X, single RGB-D in Bridge, and ZED stereo in DROID. Simulation-based corpora such as RoboCasa [26] and ARNOLD [11] carry perfect geometry but inherit the sim-to-real gap that motivates training world models on real data in the first place. DROID is unique among real-world manipulation datasets at this scale in providing synchronized multi-view stereo with known baselines and a publicly released URDF, but its shipped geometric metadata is not at the quality level downstream geometric learning depends on. Post-hoc recalibration of DROID has been attempted: PointWorld [14] initializes per-scene extrinsics from VGGT and refines them against robot depth observations, and serves as our baseline in §5. 3 Constructing Large-scale 3D Datasets for Robot Manipulation Figure 2: DROID-3D calibration pipeline. Given raw stereo from two external cameras and a wrist-mounted camera together with the robot’s URDF and joint encoder readings, Stage 1 recovers per-view metric depth from each stereo pair via a learned global-matching network and uses dense correspondences to enforce multi-view consistency. Stage 2 renders the URDF into each external view to ground the rig to the robot frame, then forms a joint factor graph that pools all episodes of a robot to recover per-scene extrinsic corrections (δText1sδ T_ext1^s, δText2sδ T_ext2^s) alongside shared kinematic parameters (hand-eye correction δTwristδ T_wrist, joint encoder offsets δδ q). The recent progress in learning-based geometric vision models rests heavily on large-scale datasets providing dense metric depth and calibrated camera poses. The manipulation domain lacks these resources at a comparable scale. While large teleoperation datasets like DROID [18] provide the necessary visual diversity and embodied dynamics, their raw geometric annotations are not at the supervision quality required for geometric pretraining. We propose an approach that takes a multi-view stereo teleoperation dataset with a known kinematic model (URDF) as input and produces a calibrated RGB-D corpus in the robot’s coordinate frame. Our approach addresses the domain’s unique visual and calibration challenges in two distinct stages, shown in Figure 2 and described in the following sections. 3.1 Stage 1: Initial Multi-View Calibration Stage 1 turns the independently calibrated cameras of each scene into an internally consistent rig placed approximately in the robot’s coordinate frame, good enough that the robot’s URDF can be rendered into each view and matched against the real image in Stage 2. Per-View Metric Depth. Each camera is a calibrated stereo pair with a known baseline b and focal length f, so metric depth reduces to predicting per-pixel disparity d, with z=fb/dz=fb/d. Manipulation scenes are dominated by textureless tabletops and specular robot arms, causing classical stereo SDKs to fail. To overcome this visual ambiguity, we recover per-view metric depth from each stereo pair using a learned global-matching network, S2M2S^2M^2 [25]. This provides the boundary-sharp metric signal necessary to turn cross-view pixel correspondences into reliable 3D constraints (see Appendix A.1 for an ablation against monocular and SDK baselines). Multi-View Consistency. The initial camera calibrations lack full multi-view consistency, and the extreme viewpoint differences between external and wrist-mounted cameras defeat sparse keypoint matchers. We utilize the metric depth above alongside dense feature matching via RoMa v2 [9] to perform robust bundle adjustment. Because wrist–external matches are markedly noisier than external–external ones, a joint solve over all three cameras settles in poor local minima, so we optimize in two passes. Pass 1 aligns the two external cameras (E1,E2E_1,E_2) into a consistent pair by minimizing their mutual Cauchy photometric residual. Pass 2 places this pair in the robot frame: the wrist camera pose is held fixed from forward kinematics and the factory hand-eye prior, and a single global SE(3)SE(3) transform of the external pair is solved to best explain the wrist–external correspondences. The output is a per-scene rig that is only approximately in the robot’s base frame, since it still inherits the systematic hand-eye and joint-encoder errors of the factory priors; removing these is the role of Stage 2. Details of the two-pass formulation, correspondence filtering, and sparse-matcher failure modes are in Appendix A.2. 3.2 Stage 2: Robot-Grounded Joint Optimization The input to this stage comprises the visually consistent per-scene multi-camera rigs produced in Stage 1, alongside the robot’s URDF and raw joint encoder readings. The output is a set of refined camera extrinsics firmly anchored to the robot’s physical base frame, coupled with globally optimized, robot-specific kinematic parameters (hand-eye transform and joint encoder offsets). Achieving this physical grounding presents two primary difficulties. First, visual reprojection consistency is invariant to rigid drift. A camera rig can be internally self-consistent but physically misaligned with the robot’s coordinate frame, which breaks the object-arm spatial relationships required for manipulation. Second, it is notoriously difficult to disentangle scene-specific camera calibration noise from systematic, robot-wide errors, such as constant joint encoder biases or gradual hand-eye mount drift. Optimizing scenes independently simply absorbs these hidden, systematic errors into the per-scene extrinsics. We overcome these challenges through a joint factor graph optimization that pools visual and kinematic signals across all episodes of a given robot. By rendering the robot arm into the camera views using the URDF and matching it to the real images, we tie the extrinsics to the real world. By optimizing all scenes simultaneously, we cleanly separate transient, scene-specific extrinsic noise from the shared kinematic parameters, multiplying the correction signal for the latter by the number of episodes. Details are given next. Kinematic Anchoring via Rendering. To supply the absolute anchor to the robot frame that cross-view consistency alone cannot provide (the first difficulty above), we render the robot arm into each external view utilizing the URDF, the corrected joint angles, and the Stage 1 extrinsic estimates, and use LoMA-G [27] to extract 2D-3D correspondences between the rendered template and the real image. Each match links a rendered surface point, whose 3D position in the robot frame is known from the URDF and forward kinematics, to a real-image pixel, yielding a robot-frame ground-truth signal independent of any cross-view photometric cue. Simultaneously, patch-level features [28] extracted by LoMA-G provide a featuremetric alignment term that yields sub-pixel signals in textureless regions. Rendering details (cropping, non-PBR shading) are given in Appendix B.1. Joint Factor Graph Construction. We build a single factor graph that spans all NsN_s episodes of a given robot (indexed s∈1,…,Nss∈\1,…,N_s\). Two kinds of parameters enter the graph: per-episode corrections to the external camera poses, which absorb scene-specific calibration noise, and shared kinematic parameters of the robot itself, which are tied across every episode of the robot. Concretely, each episode contributes a 6-DoF SE(3)SE(3) correction δText1sδ T_ext1^s and δText2sδ T_ext2^s for each external camera (12 DoF per episode), and all episodes share a 6-DoF hand-eye correction δTwristδ T_wrist to the CAD-specified gripper-to-wrist-camera mount, together with 7-DoF joint encoder offsets δδ q. The wrist camera is rigidly mounted to the gripper, so its pose has no per-episode degrees of freedom — at frame t it is fully determined by the kinematic chain: Tworld,wrist(t)=Tworld,gripper(⁡(t)+δ)⋅Tgripper,wristCAD⋅δTwrist,T_world,wrist(t)=T_world,gripper ( q(t)+δ q )· T_gripper,wrist^CAD·δ T_wrist, (1) where Tworld,gripper()T_world,gripper( q) is the base-to-gripper forward kinematics evaluated at the corrected joint angles, Tgripper,wristCADT_gripper,wrist^CAD is the CAD-specified gripper-to-wrist-camera mount, and δTwristδ T_wrist is the optimized correction to that mount. Robust Objective. The objective unifies these constraints into a single robust loss over the tens to ∼100,000 100,000 parameters of one robot’s graph (varying with episode count): ℒ=∑k∈2d,3d,l,e,fλk∑s,iρ⁡(‖rs,ik‖2,ck)+λq‖δ‖2,L= _k∈\2d,\,3d,\,l,\,e,\,f\ _k _s,iρ\! (\|r_s,i^k\|^2;\,c_k )+ _q\|δ q\|^2, (2) where ρ⁡(⋅,c)ρ(\,·\,;\,c) is the Cauchy loss with scale c, s indexes episodes, and i indexes residuals within an episode. The five data terms each constrain a different part of the graph: Mahalanobis-whitened wrist–external 2D reprojections (2d) and 3D metric lifting residuals (3d) tie the kinematic chain of Eq. (1), and thus δTwristδ T_wrist and δδ q, to the external cameras; URDF-rendered robot reprojections (l) ground the external extrinsics to the robot frame; cross-camera constraints (e) keep the external pair consistent; and featuremetric alignment (f) adds sub-pixel signal in textureless regions. The Tikhonov term λq‖δ‖2 _q\|δ q\|^2 regularizes under-constrained components of δδ q, letting the more flexible per-episode parameters explain residuals in those directions. Scalable Optimization. Solving this massive graph is bottlenecked by scale, but the Hessian exhibits a highly sparse block structure. We leverage a GPU-accelerated Levenberg-Marquardt solver using the Schur complement. This effectively eliminates the per-scene blocks, reducing the bottleneck to a highly efficient 13×1313× 13 global solve per iteration. This drops the computational cost from cubic-in-NsN_s to O⁡(Ns⋅123+133)O(N_s· 12^3+13^3), keeping typical wall time to roughly 12.5 minutes per robot group. 3.3 DROID-3D We run this pipeline on DROID’s raw release [18], the 71k-trajectory split that ships with the factory camera parameters. The recordings come from 13 institutions on Franka Panda arms, each with two table-mounted ZED 2 external cameras and a wrist-mounted ZED Mini. After per-robot filtering for broken metadata or unrecoverable initialization, we run across 28 robots and 13 labs covering 71,100 scenes. Calibration quality against a per-scene baseline is reported in §5; hyperparameters and solver wall-time in Appendix B.4. 4 Building a 3D-Consistent World Model Figure 3: DepthWorld Architecture. Multi-view RGB and depth are independently encoded into a spatially tiled latent grid (72×8072× 80) for joint U-Net denoising, preserving pretrained SVD priors. Predicted future latents (x0,fx_0,f) pass through a DPT head to predict a 3D point map (X,Y,ZX,Y,Z) and confidence logit in the robot base frame, supervised by a robust geometric loss. The objective of DepthWorld is to generate video rollouts where predicted depth and RGB compose into a single, geometrically consistent 3D world, rather than just independent, visually plausible frames. We achieve this by consuming the dense metric depth and calibrated extrinsics of DROID-3D as a direct training signal. Integrating geometric prediction into state-of-the-art video world models presents two primary difficulties. First, modifying a pretrained backbone like Stable Video Diffusion (SVD) (e.g., via channel expansion or dual-branch networks) destroys the strong appearance and motion priors that make the model useful in the first place. Second, supervising depth independently per-view does not directly encourage 3D consistency across different cameras. We overcome these challenges through two mechanisms. First, rather than altering the U-Net architecture, we introduce spatial latent tiling (§4.1) to exploit the Variational Autoencoder’s (VAE) natural ability to encode depth as an image, placing RGB and depth side-by-side in a wider latent grid. Second, to enhance cross-view consistency, we apply robot-frame auxiliary supervision (§4.2) through a lightweight point-map head that projects predictions into the robot’s physical base frame and supervises them against unified 3D point maps. 4.1 Joint RGB-Depth Prediction via Spatial Tiling Marigold [16] showed that an image-diffusion VAE encodes and decodes depth maps faithfully. Building on this, we encode depth as a separate image alongside RGB and tile the two in latent space: each is encoded by the unmodified SVD VAE and placed side-by-side, so the U-Net sees a wider latent grid covering both modalities. The only network change is extending the spatial position embeddings to cover the wider grid. At each timestep, the model processes images of resolution 192×320192× 320 for V=3V=3 cameras (two external, one wrist) and two modalities (RGB, depth). Since the SVD VAE downsamples spatially by 8×8×, each view-modality tile is independently encoded into a latent of dimensions Hlat×WlatH_lat× W_lat (where Hlat=24,Wlat=40H_lat=24,W_lat=40). We arrange the six tiles in a 2D grid: RGB and depth tiles for each view are concatenated horizontally (yielding width 2⋅Wlat2· W_lat), and the V views are stacked vertically (yielding height V⋅HlatV· H_lat). This produces a single joint latent of shape 72×8072× 80 per timestep (see Figure 3). The U-Net denoises this full grid as one tensor across a temporal window of 11 frames (6 history, 5 future) at 5 Hz. (See Appendix C.1 for the depth preprocessing pipeline required to conform the raw depth to the VAE distribution.) 4.2 Robot-Frame Point-Map Supervision Per-view depth denoising trains each predicted depth map against its own target independently, with no explicit penalty when predicted depths disagree across different camera views of the same scene. We make cross-view coherence explicit by lifting predictions into the robot base frame and computing the loss there. We do so by attaching a lightweight DPT [32] point-map head (initialized from VGGT [39]) to the SVD U-Net’s final predicted latent x^0 x_0. For the predicted future frames x^0,f x_0,f, we pass the (RGB, depth) latent pairs to the DPT head, which predicts a 3D point map in robot base coordinates for each view. The ground-truth point map Pv,t∗∈ℝH×W×3P_v,t^* ^H× W× 3 for view v at frame t is obtained by backprojecting the GT depth through the camera intrinsics and transforming it to the robot base frame via the calibrated extrinsics from §3. The head outputs four channels per frame and view: spatial coordinates X,Y,ZX,Y,Z (the point-map prediction Pv,tP_v,t in robot world frame) and a per-pixel confidence logit. We pass the logit through a softplus to obtain the positive confidence Cv,t(u)>0C_v,t(u)>0. Following DUSt3R [40] and MapAnything [17], the loss is confidence-weighted in log space, with Barron’s robust ρ [3]. Training Schedule We initialize from the SVD checkpoint used by Ctrl-World [12], utilizing identical action conditioning (per-frame end-effector pose). The combined training objective is ℒ=ℒdenoise+λpmℒpmL=L_denoise+ _pmL_pm. We train in two stages for stability. Stage 1 trains the joint RGB-depth model alone for 40,000 steps (λpm=0 _pm=0), allowing the model to learn the joint latent distribution. Stage 2 initializes the DPT head from VGGT and trains all components jointly for an additional 50,000 steps (λpm=0.005 _pm=0.005). 5 Experiments We evaluate both contributions empirically: the calibration quality of DROID-3D against a per-scene baseline, and the world model prediction accuracy on held-out trajectories. DROID-3D evaluation setup and baseline. We evaluate calibration on 5 DROID labs (∼ 1,000 scenes total). Our baseline is a per-scene refinement procedure modeled on PointWorld [14]: VGGT [39] extrinsic initialization followed by per-scene optimization against robot depth, with S2M2 [25] substituted as the depth source to match ours. We use four geometric proxy metrics: ext–ext (E) and wrist–ext (WE) reprojection error on MAGSAC [1] gold inliers, robot depth error (RD) between S2M2 and URDF-rendered depth, and Mask IoU between the URDF-rendered robot mask and a RoboEngine [46] segmentation. Full definitions are in Appendices D.1–D.2. Additionally, we replicated the DROID setup in our lab and recorded 55 ground truth (GT) poses of the cameras as well as the robot joint offsets using a ChArUco board. Using this GT setup, we measure the average rotation error and translation error in the optimized poses. We test the pose accuracy using 5050 synthetic scenes from the REALM simulator [37]. DROID-3D calibration accuracy. Table 1 reports results on the full evaluation set and on the subset where the baseline converges. The gap is large on every axis: 23× improvement on E, 13× on WE, 8× on RD, and +37 absolute points on IoU; This substantial gap remains even when restricting evaluation to the subset of scenes where the baseline successfully converges. Beyond accuracy, the baseline fails to converge on ∼ 18% of scenes (180 of 995); our coupled optimization has no analogous per-scene failure mode (per-factor ablations: Table A1; qualitative: Figure A10). Accuracy to GT poses. On our GT setup, from teleoperation episodes alone, our approach recovers extrinsics to 4.9 m/0.39∘ median error and hand-eye to 3 m/0.3∘ across all five capture sets; joint offsets are recovered to 0.14∘ mean error across all seven joints. On the same set-up, the PointWorld-style per-scene baseline obtains 72 m / 3.0∘ median error, an order of magnitude worse than ours. In REALM simulation with exact GT poses (50 configurations, varied viewpoints), our approach recovers extrinsics with 6.4 m/0.3∘ median error. Table 1: Calibration quality on ∼ 1,000 evaluation scenes. Blue rows: 2D multi-view consistency (medians on joint-MAGSAC gold inliers). Amber rows: 3D grounding to the robot frame. The right block restricts to the subset where the baseline’s per-scene refinement converges. All scenes (n=995n=995) PointWorld-converged subset (n=815n=815) Metric PointWorld [14] Ours PointWorld [14] Ours E median (px) ↓ 5.84 0.25 5.67 0.25 WE median (px) ↓ 13.19 1.03 12.52 1.02 RD median (m) ↓ 111.3 14.0 104.6 13.9 Mask IoU mean ↑ 0.440 0.810 0.458 0.819 Figure 4: Qualitative Result: We show two rollouts by the baseline RGB model and DepthWorld, which also produces high-quality depth. Left: The RGB-only model fails to simulate the robot grasping the red cup as in the GT trajectory. DepthWorld accurately simulates the interaction and produces an aligned depth map. Right: Without depth supervision, the RGB-only world model cannot distinguish the open wardrobe door from a solid surface and simulates the arm colliding with it. DepthWorld accurately generates the arm going inside the wardrobe by accounting for the depth of the arm and the door. See the project website for additional results, including videos. World model evaluation setup and metrics. We evaluate on 256 held-out trajectories from DROID with 10 consecutive autoregressive rollouts. We compare three variants: RGB is the Ctrl-World [12] backbone (multi-view RGB only); RGB+D is our joint RGB-depth model; RGB+D PM-DPT adds the point-map auxiliary head. We report PSNR, SSIM [42], and LPIPS [48] for RGB, and AbsRel, RMSE, and δ1 _1 for depth, each computed separately for external and wrist views. Depth metrics are against the depth obtained in Sec. 3. Full protocol is in Appendix D.3. Joint depth prediction improves RGB. Table 2 reports both modalities on the held-out set, with qualitative rollouts shown in Figure 4. Adding depth as a joint prediction target through spatial latent tiling delivers a clear gain on RGB quality: RGBD models outperform the RGB-only baseline on all three RGB metrics with +1.48 dB PSNR on external views and +1.02 dB on wrist. Point-map supervision and DROID-3D’s downstream value. Adding the point-map auxiliary loss (PM-DPT) further refines depth accuracy—especially on the challenging wrist view – while fully maintaining RGB generation quality. This suggests that mapping RGB and depth into a shared, spatially tiled latent space already encourages the U-Net to learn strong cross-modality constraints. Consequently, the robot-frame auxiliary loss acts primarily as a fine-grained 3D regularizer rather than the sole driver of geometric learning. More broadly, PM-DPT serves as a working instance of how DROID-3D’s robot-grounded annotations can drive spatial supervision beyond independent per-view depth. This framework opens the door for future extensions, such as enforcing strict spatiotemporal (4D) consistency or enabling downstream policies to extract unified 3D representations directly from the world model’s internal latents. Cross-view consistency improvements. We measure two quantities: (i) Cross-view reprojection: We warp one generated external view into the other using GT depth and poses. RGB+D model beats the RGB baseline model by +1.0 dB PSNR (17.60 vs. 16.60), and our PM-DPT head always improves by +0.05+0.05–0.080.08 dB over RGB+D. (i) Cross-view generated-depth agreement: Our PM-DPT model reduces external-view projected depth disagreement by 2.62.6 m compared to our RGBD model. Comparison with other baselines. We compare against the existing world models for manipulation that predict future 3D geometry rather than RGB alone: PointWorld (PW) [14], which predicts 3D scene flow and is the only action-conditioned 3D world model trained on DROID, and TesserAct [49], which predicts RGB-D-normal video; iMoWM [47] has no public code. TesserAct is text-conditioned and single-view, so it is incomparable as is; we adapted it to action conditioning and trained it on the same DROID split. At 50k steps it trails our model: external PSNR 19.43 vs 24.07 (ours), AbsRel 0.154 vs 0.074 (ours); see Appendix D.5 for the adaptation and protocol. For PW, whose output is a point cloud rather than depth images, we compare predicted 3D trajectories of scene points (robot excluded), each model scored on its own depth target; see Appendix D.4, Table A5, and Figure A11. Tracked-point ℓ2 _2 (2→ 8 s): PW 23→ 56 m, ours 31→ 46 m. Chamfer: whole scene PW 8→ 18 m vs. ours 8→ 9 m; moving region PW 21→ 58 m vs. ours 25→ 32 m. Appendix C ablates the training depth source (Table A3), the point-map supervision extrinsics (Table A4), and the depth-integration architecture (Table A2). Depth quality is strongly view-dependent across both depth-augmented variants: δ1>0.94 _1>0.94 on external views but ≈0.82≈ 0.82 on wrist due to the wrist camera’s rapid motion, proximity to the manipulated object, and frequent self-occlusion by the arm. Limitations. As an autoregressive rollout model, DepthWorld inherits the exposure bias and quality degradation common to such models [12]. However, recent works like PersistWorld [2] have shown that RL post-training can significantly improve rollout stability. The availability of high-quality 3D annotations in DROID-3D, combined with the native 3D-consistent outputs of DepthWorld, opens exciting new frontiers for geometric reward design in future world modelling. Table 2: Ablation of world model prediction quality on held-out trajectories. Depth metrics are against obtained depth from Sec. 3. The RGB-only baseline does not predict depth (—). Joint depth prediction (RGB+D) improves all RGB metrics on both view classes; the point-map auxiliary loss (PM-DPT) further refines depth accuracy while maintaining RGB generation quality, specially on wrist metrics. RGB quality Depth quality View Model PSNR↑ SSIM↑ LPIPS↓ AbsRel↓ RMSE↓ δ1↑ _1 External RGB 22.63 0.8170 0.0930 — — — RGB+D 24.09 0.8333 0.0889 0.0765 0.220 0.9432 RGB+D PM-DPT 24.11 0.8333 0.0891 0.0782 0.219 0.9434 Wrist RGB 16.98 0.5737 0.3228 — — — RGB+D 17.96 0.6106 0.3137 0.2230 0.150 0.8182 RGB+D PM-DPT 18.00 0.6129 0.3131 0.2182 0.148 0.8210 6 Conclusion We introduced two contributions toward 3D-consistent world models for robot manipulation. DROID-3D recalibrates the DROID dataset into a 3D corpus with dense metric depth and robot-grounded multi-view extrinsics across more than 70,000 episodes, through a factor graph that pools each robot’s episodes against shared kinematic parameters. DepthWorld consumes this signal in a Stable Video Diffusion-based world model that jointly predicts multi-view RGB and depth via spatial latent tiling. The central empirical finding is that at equal training budget, joint RGB-depth prediction improves RGB by +1.48 dB PSNR over an RGB-only baseline, while simultaneously yielding metric depth for downstream geometric reasoning. Together, DROID-3D and DepthWorld open a path toward world models whose rollouts compose into a single coherent 3D scene—the property all downstream uses, policy evaluation, improvement, and planning, depend on. Acknowledgments This work was supported by the European Union’s Horizon Europe projects AGIMUS (No. 101070165), euROBIN (No. 101070596), ERC FRONTIER (No. 101097822), ELIAS (No. 101120237), ELLIOT (No. 101214398), ČVUT Starting grant ”DREAM-ACT” (Project ID CVUT-StG-26-089), and CTU Future Fund (Project ID: CVUT-BrF-26-22825M). This work was also supported by the EU’s Horizon Europe Programme under the Grant agreement No. 10113667 (CLARA Project), and was co-funded by the EU from the Operational Programme Jan Amos Komenský (OP JAK) (project “Center for Artificial Intelligence and Quantum Computing in System Brain Research”, reg. no. CZ.02.01.01/00/23_029/0008437). Compute resources and infrastructure were supported by the Ministry of Education, Youth and Sports of the Czech Republic through the e-INFRA CZ (ID:90254). References [1] D. Barath, J. Noskova, M. Ivashechkin, and J. Matas (2020) MAGSAC++, a fast, reliable and accurate robust estimator. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 1304–1312. Cited by: Figure A4, §A.2, §5. [2] J. Bardhan, P. Drozdik, J. Sivic, and V. Petrik (2026) Persistent robot world models: stabilizing multi-step rollouts via reinforcement learning. arXiv preprint arXiv:2603.25685. Cited by: §1, §2, §5. [3] J. T. Barron (2019) A general and adaptive robust loss function. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 4331–4339. Cited by: §C.5, §4.2. [4] G. Baruch, Z. Chen, A. Dehghan, T. Dimry, Y. Feigin, P. Fu, T. Gebauer, B. Joffe, D. Kurz, A. Schwartz, et al. (2021) Arkitscenes: a diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data. arXiv preprint arXiv:2111.08897. Cited by: §1, §2. [5] A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English, V. Voleti, A. Letts, et al. (2023) Stable video diffusion: scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127. Cited by: §2. [6] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al. (2022) Rt-1: robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817. Cited by: §2. [7] D. DeTone, T. Malisiewicz, and A. Rabinovich (2018) Superpoint: self-supervised interest point detection and description. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, p. 224–236. Cited by: 4(c), 5(c), 6(c), §A.2. [8] Y. Du, S. Yang, B. Dai, H. Dai, O. Nachum, J. Tenenbaum, D. Schuurmans, and P. Abbeel (2023) Learning universal policies via text-guided video generation. Advances in neural information processing systems 36, p. 9156–9172. Cited by: §1, §2. [9] J. Edstedt, D. Nordström, Y. Zhang, G. Bökman, J. Astermark, V. Larsson, A. Heyden, F. Kahl, M. Wadenbäck, and M. Felsberg (2025) RoMa v2: harder better faster denser feature matching. arXiv preprint arXiv:2511.15706. Cited by: 4(a), 5(a), 6(a), §A.2, §3.1. [10] H. Fang, H. Fang, Z. Tang, J. Liu, C. Wang, J. Wang, H. Zhu, and C. Lu (2023) Rh20t: a comprehensive robotic dataset for learning diverse skills in one-shot. arXiv preprint arXiv:2307.00595. Cited by: §2. [11] R. Gong, J. Huang, Y. Zhao, H. Geng, X. Gao, Q. Wu, W. Ai, Z. Zhou, D. Terzopoulos, S. Zhu, et al. (2023) Arnold: a benchmark for language-grounded task learning with continuous states in realistic 3d scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 20483–20495. Cited by: §2. [12] Y. Guo, L. X. Shi, J. Chen, and C. Finn (2025) Ctrl-world: a controllable generative world model for robot manipulation. arXiv preprint arXiv:2510.10125. Cited by: §C.6, §C.7, §D.3, §1, §2, §4.2, §5, §5. [13] W. Hu, X. Gao, X. Li, S. Zhao, X. Cun, Y. Zhang, L. Quan, and Y. Shan (2025) Depthcrafter: generating consistent long depth sequences for open-world videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 2005–2015. Cited by: §2. [14] W. Huang, Y. Chao, A. Mousavian, M. Liu, D. Fox, K. Mo, and L. Fei-Fei (2026) PointWorld: scaling 3d world models for in-the-wild robotic manipulation. arXiv preprint arXiv:2601.03782. Cited by: Figure A10, §D.4, Table A5, §2, Table 1, Table 1, §5, §5. [15] F. Jiang, Y. Chen, K. Xu, Y. Liu, H. Wang, Z. Shen, J. Lu, S. Huang, Y. Wang, C. Xie, et al. (2026) RoboWM-bench: a benchmark for evaluating world models in robotic manipulation. arXiv preprint arXiv:2604.19092. Cited by: §2. [16] B. Ke, A. Obukhov, S. Huang, N. Metzger, R. C. Daudt, and K. Schindler (2024) Repurposing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 9492–9502. Cited by: §2, §4.1. [17] N. Keetha, N. Müller, J. Schönberger, L. Porzi, Y. Zhang, T. Fischer, A. Knapitsch, D. Zauss, E. Weber, N. Antunes, et al. (2025) Mapanything: universal feed-forward metric 3d reconstruction. arXiv preprint arXiv:2509.13414. Cited by: §1, §2, §4.2. [18] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, et al. (2024) Droid: a large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945. Cited by: item 1, §1, §2, §2, §3.3, §3. [19] V. Leroy, Y. Cabon, and J. Revaud (2024) Grounding image matching in 3d with mast3r. In European conference on computer vision, p. 71–91. Cited by: §C.4, §1, §2. [20] Z. Li and N. Snavely (2018) Megadepth: learning single-view depth prediction from internet photos. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 2041–2050. Cited by: §1, §2. [21] H. Lin, S. Chen, J. Liew, D. Y. Chen, Z. Li, G. Shi, J. Feng, and B. Kang (2025) Depth anything 3: recovering the visual space from any views. arXiv preprint arXiv:2511.10647. Cited by: §A.1, §1. [22] P. Lindenberger, P. Sarlin, and M. Pollefeys (2023) Lightglue: local feature matching at light speed. In Proceedings of the IEEE/CVF international conference on computer vision, p. 17627–17638. Cited by: 4(c), 4(d), 5(c), 5(d), 6(c), 6(d), §A.2. [23] D. G. Lowe (1999) Object recognition from local scale-invariant features. In Proceedings of the seventh IEEE international conference on computer vision, Vol. 2, p. 1150–1157. Cited by: 4(d), 5(d), 6(d), §A.2. [24] A. Š. Mikeštíková, M. Fourmy, M. Cifka, J. Sivic, and V. Petrik (2026) AlignPose: generalizable 6d pose estimation via multi-view feature-metric alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 14626–14636. Cited by: §B.1. [25] J. Min, Y. Jeon, J. Kim, and M. Choi (2025) S2M2: scalable stereo matching model for reliable depth estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 26729–26739. Cited by: §A.1, §3.1, §5. [26] S. Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y. Zhu (2024) Robocasa: large-scale simulation of everyday tasks for generalist robots. arXiv preprint arXiv:2406.02523. Cited by: §2. [27] D. Nordström, J. Edstedt, G. Bökman, J. Astermark, A. Heyden, V. Larsson, M. Wadenbäck, M. Felsberg, and F. Kahl (2026) LoMa: local feature matching revisited. arXiv preprint arXiv:2604.04931. Cited by: 4(b), 5(b), 6(b), §A.2, §B.1, §3.2. [28] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. (2023) Dinov2: learning robust visual features without supervision. arXiv preprint arXiv:2304.07193. Cited by: §B.1, §3.2. [29] A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, et al. (2024) Open x-embodiment: robotic learning datasets and rt-x models: open x-embodiment collaboration 0. In 2024 IEEE International Conference on Robotics and Automation (ICRA), p. 6892–6903. Cited by: §2. [30] E. Pallotta, S. M. Azar, S. Li, O. Zatsarynna, and J. Gall (2025) SyncVP: joint diffusion for synchronous multi-modal video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 13787–13797. Cited by: §C.2, §2. [31] S. K. Ramakrishnan, A. Gokaslan, E. Wijmans, O. Maksymets, A. Clegg, J. Turner, E. Undersander, W. Galuba, A. Westbury, A. X. Chang, et al. (2021) Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. arXiv preprint arXiv:2109.08238. Cited by: §1, §2. [32] R. Ranftl, A. Bochkovskiy, and V. Koltun (2021) Vision transformers for dense prediction. In Proceedings of the IEEE/CVF international conference on computer vision, p. 12179–12188. Cited by: §C.4, §2, §4.2. [33] J. Reizenstein, R. Shapovalov, P. Henzler, L. Sbordone, P. Labatut, and D. Novotny (2021) Common objects in 3d: large-scale learning and evaluation of real-life 3d category reconstruction. In Proceedings of the IEEE/CVF international conference on computer vision, p. 10901–10911. Cited by: §1, §2. [34] L. Russell, A. Hu, L. Bertoni, G. Fedoseev, J. Shotton, E. Arani, and G. Corrado (2025) Gaia-2: a controllable multi-view generative world model for autonomous driving. arXiv preprint arXiv:2503.20523. Cited by: §1, §2. [35] C. Ryali, Y. Hu, D. Bolya, C. Wei, H. Fan, P. Huang, V. Aggarwal, A. Chowdhury, O. Poursaeed, J. Hoffman, et al. (2023) Hiera: a hierarchical vision transformer without the bells-and-whistles. In International conference on machine learning, p. 29441–29454. Cited by: §A.1. [36] P. Sarlin, A. Unagar, M. Larsson, H. Germain, C. Toft, V. Larsson, M. Pollefeys, V. Lepetit, L. Hammarstrand, F. Kahl, et al. (2021) Back to the feature: learning robust camera localization from pixels to pose. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 3247–3257. Cited by: §B.1. [37] M. Sedlacek, P. Yefanov, G. Ponimatkin, J. Bardhan, S. Pilc, M. Fourmy, E. Kazakos, C. G. Snoek, J. Sivic, and V. Petrik (2026) Realm: a real-to-sim validated benchmark for generalization in robotic manipulation. IEEE Robotics and Automation Letters. Cited by: §5. [38] H. R. Walke, K. Black, T. Z. Zhao, Q. Vuong, C. Zheng, P. Hansen-Estruch, A. W. He, V. Myers, M. J. Kim, M. Du, et al. (2023) Bridgedata v2: a dataset for robot learning at scale. In Conference on Robot Learning, p. 1723–1736. Cited by: §2. [39] J. Wang, M. Chen, N. Karaev, A. Vedaldi, C. Rupprecht, and D. Novotny (2025) Vggt: visual geometry grounded transformer. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 5294–5306. Cited by: §C.4, §1, §2, §4.2, §5. [40] S. Wang, V. Leroy, Y. Cabon, B. Chidlovskii, and J. Revaud (2024) Dust3r: geometric 3d vision made easy. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 20697–20709. Cited by: §C.4, §1, §2, §4.2. [41] Y. Wang, J. Zhou, H. Zhu, W. Chang, Y. Zhou, Z. Li, J. Chen, J. Pang, C. Shen, and T. He (2025) Pi3: permutation-equivariant visual geometry learning. arXiv preprint arXiv:2507.13347. Cited by: §1. [42] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli (2004) Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13 (4), p. 600–612. Cited by: §D.3, §5. [43] B. Wen, M. Trepte, J. Aribido, J. Kautz, O. Gallo, and S. Birchfield (2025) Foundationstereo: zero-shot stereo matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 5249–5260. Cited by: §A.1. [44] Y. Yao, Z. Luo, S. Li, J. Zhang, Y. Ren, L. Zhou, T. Fang, and L. Quan (2020) Blendedmvs: a large-scale dataset for generalized multi-view stereo networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 1790–1799. Cited by: §1, §2. [45] C. Yeshwanth, Y. Liu, M. Nießner, and A. Dai (2023) Scannet++: a high-fidelity dataset of 3d indoor scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 12–22. Cited by: §1, §2. [46] C. Yuan, S. Joshi, S. Zhu, H. Su, H. Zhao, and Y. Gao (2025) Roboengine: plug-and-play robot data augmentation with semantic robot segmentation and background generation. In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 7622–7629. Cited by: item 4, §5. [47] C. Zhang, Z. Wu, G. Lu, Y. Tang, and Z. Wang (2025) Imowm: taming interactive multi-modal world model for robotic manipulation. arXiv preprint arXiv:2510.09036. Cited by: §2, §5. [48] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang (2018) The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 586–595. Cited by: §D.3, §5. [49] H. Zhen, Q. Sun, H. Zhang, J. Li, S. Zhou, Y. Du, and C. Gan (2025) Tesseract: learning 4d embodied world models. arXiv preprint arXiv:2504.20995. Cited by: §D.5, §1, §2, §2, §5. [50] T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely (2018) Stereo magnification: learning view synthesis using multiplane images. arXiv preprint arXiv:1805.09817. Cited by: §2. [51] F. Zhu, H. Wu, S. Guo, Y. Liu, C. Cheang, and T. Kong (2025) Irasim: a fine-grained world model for robot manipulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 9834–9844. Cited by: §2. Supplementary Material for DepthWorld: 3D World Model for Robot Manipulation 1 Introduction 2 Related Work 3 Constructing Large-scale 3D Datasets for Robot Manipulation 3.1 Stage 1: Initial Multi-View Calibration 3.2 Stage 2: Robot-Grounded Joint Optimization 3.3 DROID-3D 4 Building a 3D-Consistent World Model 4.1 Joint RGB-Depth Prediction via Spatial Tiling 4.2 Robot-Frame Point-Map Supervision 5 Experiments 6 Conclusion References A Stage 1: Initial Multi-View Calibration A.1 Per-View Metric Depth A.2 Robust Multi-View Consistency Optimization B Stage 2: Robot-Grounded Joint Optimization B.1 Kinematic Anchoring via Rendering B.2 Joint Factor Graph: Residual Definitions B.3 Ablations on the Pose Estimation Pipeline B.4 Hyperparameters and Solver B.5 Qualitative results of DROID-3D Calibration Dataset vs Baseline C DepthWorld Architecture and Training Details C.1 Depth Preprocessing Rationale (Supplement to Section 4.1) C.2 Spatial Latent Tiling vs. Other Depth Integration Architectures C.3 Effect of the Depth Source on DepthWorld C.4 Auxiliary Point-Map Head (Supplement to Section 4.2) C.5 Training Objective C.6 Training Schedule and Optimization C.7 Inference and Autoregressive Rollout C.8 Depth Reference and De-normalization for Metric Evaluation D Experimental Setup and Metrics D.1 Calibration Baseline Modifications (Supplement to Section 5.1) D.2 Geometric Calibration Metrics D.3 World-Model Evaluation Protocol D.4 Comparison with PointWorld (Supplement to Section ) D.5 Comparison with TesserAct (Supplement to Section ) This supplementary material is organized as follows. Appendix A and Appendix B detail the two stages of the DROID-3D calibration pipeline: per-view metric depth and multi-view consistency (Stage 1), and the robot-grounded joint factor graph with its explicit residual definitions, hyperparameters, and discarded design choices (Stage 2). Appendix C covers the DepthWorld architecture and training: depth preprocessing, spatial latent tiling, the point-map head, the training objective, the optimization schedule, and the inference rollout. Appendix D specifies the calibration baseline, the geometric calibration metrics, the world-model evaluation protocol, and the protocols for the comparisons with PointWorld and TesserAct. Appendix A Stage 1: Initial Multi-View Calibration A.1 Per-View Metric Depth Each DROID camera (two table-mounted ZED 2 external cameras and a wrist-mounted ZED Mini) is a calibrated stereo pair with a known baseline, so metric depth reduces to predicting per-pixel disparity. For a pixel u with predicted disparity d⁡(u)d(u) we obtain metric depth by direct triangulation, z⁡(u)=fbd⁡(u),z(u)= f\,bd(u), (A1) with focal length f and baseline b from the factory stereo calibration. The choice of disparity estimator critically impacts the downstream photometric reprojection losses, as sharper depth maps produce tighter geometric correspondences. We use the global-matching stereo network S2M2S^2M^2 [25], and evaluated three families of alternatives on representative manipulation scenes: Stereo SDKs (ZED Ultra, Neural). Classical semi-global matching (SGM) algorithms yield metric scale but fail on the texture-poor tabletops and specular surfaces of the robot arm, resulting in sparse depth maps with characteristic holes. The learned ZED Neural mode is denser but remains semi-dense and is outperformed by S2M2S^2M^2 in both coverage and boundary fidelity. See Figure A1 for qualitative examples of the performance. Additionally, we found that on our GT setup with GT poses, S2M2 (ours) beats ZED NEURAL on robot-surface depth against the URDF render on all sets (median errors 11.2–15.3 m vs. 12.6–16.9 m). Feed-Forward Monocular Depth. Feed-forward monocular models, such as [21], provide dense, visually pleasing depth but suffer from an inherent affine scale ambiguity. This lack of reliable metric scale causes severe misalignment during multi-view triangulation. In our specific case with Depth Anything 3 (DA3), we find that it generates noisy and overly smoothened depth maps, which fails to capture the details of the objects (see Figure A2). We further note that DA3 is itself distilled from a stereo teacher, and its own evaluation reports the stereo teacher to be the stronger depth predictor—consistent with our preference for a stereo source. Other Stereo Disparity methods. We also experimented with using FoundationStereo (FS) [43] for recovering the stereo disparity maps. However, we found that S2M2 more reliably produced sharper results. Furthermore, S2M2 was significantly faster than FS at the 1280×7201280× 720p high resolution, and did not require special Hierarchical models (as opposed to FS requiring Hiera [35] models). For a qualitative comparison see Figure A3. Figure A1: ZED SDK vs S2M2. The classical ZED ULTRA algorithm fails completely in the wrist cam view and produces sparse depth for the external views. ZED NEURAL produces comparatively denser results but fails at the boundaries and edges of objects. S2M2 produces denser and sharper results and outperforms both the alternatives. Figure A2: Depth Anything 3 (DA3) vs S2M2. DA3 produces generates noisy and overly smoothed depth maps, which fail to capture the details of the objects. The two methods are visualized with a different colormap (depth for DA3 vs disparity for S2M2). Figure A3: Foundation Stereo vs S2M2. We find that S2M2 produces more stable results. Notice the missing gripper on the left image and the higher quality details of the objects within the cup in the right image. A.2 Robust Multi-View Consistency Optimization Correspondences and outlier rejection. Standard sparse keypoint detectors routinely fail to extract sufficient correspondences in this domain due to the extreme viewpoint differences between the fixed external cameras and the dynamic wrist camera, compounded by textureless surfaces: SIFT [23], SuperPoint [7] + LightGlue [22], and LoMa [27] all return too few reliable matches for bundle adjustment to converge, particularly on wrist–external pairs (Figures A4–A6). We instead rely on RoMa v2 [9] to extract dense, pixel-wise correspondences even in textureless regions, robustly filtered using MAGSAC [1] fundamental-matrix estimation. Figure A7 shows the count of correspondences and the coverage across the frames for a small set of 2525 samples. (a) RoMa-v2 [9] (b) LoMa [27] (c) SuperPoint [7] + LightGlue [22] (d) SIFT [23] + LightGlue [22] Figure A4: Correspondence quality across feature matchers. Each panel overlays the correspondences recovered by one matcher on a wrist–external camera pair; green points are MAGSAC [1] fundamental-matrix inliers. The dense matcher RoMa-v2 (a) recovers accurate correspondences across the extreme viewpoint change, whereas the sparse matchers (b–d) collapse to a handful of matches or fail outright on the textureless and specular regions that dominate the scene. (a) RoMa-v2 [9] (b) LoMa [27] (c) SuperPoint [7] + LightGlue [22] (d) SIFT [23] + LightGlue [22] Figure A5: Correspondence quality across feature matchers (continued). Panels and color coding as in Fig. A4: (a) RoMa-v2, (b) LoMa, (c) SuperPoint + LightGlue, (d) SIFT + LightGlue; green points are MAGSAC inliers. RoMa-v2 again yields dense, accurate matches where the sparse matchers largely fail on the wrist–external pair. (a) RoMa-v2 [9] (b) LoMa [27] (c) SuperPoint [7] + LightGlue [22] (d) SIFT [23] + LightGlue [22] Figure A6: Correspondence quality across feature matchers (continued). Panels and color coding as in Fig. A4: (a) RoMa-v2, (b) LoMa, (c) SuperPoint + LightGlue, (d) SIFT + LightGlue; green points are MAGSAC inliers. The gap is consistent across scenes—RoMa-v2 sustains coverage in the wrist view while the other matchers degrade. Figure A7: RoMa-v2 vs others summary. Left: We show the number of usable correspondences (inliers from MAGSAC) for each method. RoMa-v2 consistently produces an order of magnitude more correspondences. Right: We show the coverage of each method. RoMa-v2 has approximately 2×2× the coverage on the extreme viewpoint pair of wrist and external cameras. Two-pass bundle adjustment. To avoid suboptimal local minima caused by the noise disparity between external–external and wrist–external view pairs, the bundle adjustment is solved in two passes. Pass 1 locks the two external cameras (E1,E2)(E_1,E_2) into a mutually consistent pair: using the metric depth to lift matched pixels, we reproject depth from one external view into the other and minimize the Cauchy photometric residual. This reliably reaches sub-pixel triangulation error on the external pair for the large majority of scenes. The resulting pair is internally consistent but its global pose in the robot frame remains undetermined (a 66-DoF gauge freedom; metric scale is fixed by stereo depth). Pass 2 fixes this external pair internally and resolves its global 66-DoF pose by anchoring it to the wrist camera’s forward-kinematic trajectory. The wrist pose at each frame is not a free variable: it is taken from the robot’s kinematic chain—forward kinematics on the raw joint angles composed with DROID’s factory hand-eye prior—and held fixed in this pass. We solve for the single global SE(3)SE(3) transform of the locked external pair that best explains the wrist–external correspondences, using a looser Cauchy scale to reflect the noisier wide-baseline matches. The output is a per-scene rig approximately in the robot’s base frame—approximate because it still inherits the systematic hand-eye and joint-encoder errors of the factory priors, which Stage 2 (Appendix B) removes. Appendix B Stage 2: Robot-Grounded Joint Optimization The input to this stage is the set of visually consistent per-scene rigs from Stage 1, together with the robot’s URDF and raw joint encoder readings. The output is a set of refined extrinsics anchored to the robot’s base frame, coupled with globally optimized robot-specific kinematic parameters (hand-eye transform and joint encoder offsets). B.1 Kinematic Anchoring via Rendering To tie the rig to the physical robot, we render the arm into each external view with pyrender, using the URDF, the current joint angles, and the Stage 1 extrinsics. We extract 22D–33D correspondences between the rendered template and the real image with LoMa-G [27]: each match links a rendered surface point—whose 33D location in the robot frame is known from the URDF and forward kinematics—to a real-image pixel, yielding a robot-frame ground-truth signal independent of any cross-view photometric cue. LoMa-G internally computes DINO [28] patch features; we store these at the correspondences and PCA-reduce them to 6464 dimensions to support a featuremetric alignment term [36, 24] (the f term in Eq. (2)) that provides sub-pixel gradients in textureless regions. Two practical details matter. First, we crop both the render and the real image tightly around the rendered arm before matching: this raises both the match count and the effective patch resolution (92→27792→ 277 matches in Fig. A8). Second, we render with well-lit, non-physically-based (non-PBR) shading, which produces more stable and dense discrete correspondences for LoMa-G than physically-based rendering. Figure A9 shows the patch level features for rendered robot and the input image. We see that the patches between the render and the actual image align nicely. Note that the features produced by LoMa are higher resolution and align much better than raw DINOv2 features. Therefore, we use the LoMa generated features (which builds upon the DINOv2 features). Figure A8: Effect of cropping the render and the real image before matching with LoMa-G. The yellow box indicates the crop region. The number of matches goes up from 92→27792→ 277. Green dots indicate matches that are on the surface of the robot, red dots are for points that do not lie on the surface. Figure A9: Visualization of the PCA of the DINOv2 features and the LoMa features. Here we show a comparison between the patch level features of the actual image, cropped around the robot arm, and a similar render of the robot URDF. The patch level features for both DINOv2 and LoMa match at the robot arm. However, the patch level features from LoMa contain more fine-grained details, since LoMa was specifically tuned for matching. B.2 Joint Factor Graph: Residual Definitions A single factor graph spans all NsN_s episodes of one physical robot (NsN_s ranges from ∼1,000 1,000 to ∼9,000 9,000). Each episode contributes per-scene 66-DoF SE(3)SE(3) corrections δText1s,δText2sδ T^s_ext1,δ T^s_ext2 to its two external cameras (1212 DoF/episode), which absorb scene-specific calibration noise. Shared across all episodes are the 66-DoF hand-eye correction δTwristδ T_wrist and the 77-DoF joint encoder offsets δδ q (1313 DoF total). The wrist camera carries no per-episode degrees of freedom; at frame t its pose is fully determined by the kinematic chain of Eq. (1). For the largest robot groups this gives 12Ns+13≈1.08×10512N_s+13≈ 1.08× 10^5 parameters. Notation. Let KcK_c and Tcs∈SE(3)T^s_c∈ SE(3) (camera-to-world) be the intrinsics and extrinsics of camera c in scene s, with Tcs=δTcsT^csT^s_c=δ T^s_c\, T^s_c for the Stage-1 initialization T^cs T^s_c. For pixel u (homogeneous ~ u) with metric depth Dc()D_c(u), the back-projected world point is =TcsDc()Kc−1~X=T^s_c\,D_c(u)\,K_c^-1 u, and π⁡(⋅)π(·) denotes perspective projection, π⁡([x,y,z]⊤)=[x/z,y/z]⊤π([x,y,z] )=[x/z,\,y/z] . All data terms enter the scaled-Cauchy loss ρ⁡(∥r∥2,c)=c2log⁡(1+∥r∥2/c2)ρ( r ^2;c)=c^2 \! (1+ r ^2/c^2 ) of Eq. (2). The five data terms. • Cross-camera, external–external (e). For a RoMa-v2 match (,′)(u,u ) between E1E_1 and E2E_2 in scene s, lifted via S2M2S^2M^2 depth DE1()D_E_1(u): rs,iee=π⁡(KE2(TE2s)−1TE1sDE1()KE1−1~)−′.r^e_s,i=π\! (K_E_2\,(T^s_E_2)^-1\,T^s_E_1\,D_E_1(u)\,K_E_1^-1 u )-u . (A2) • Wrist–external 2D reprojection (2d). For a match (W,E)(u_W,u_E) between the wrist W at frame t and external EcE_c, with TW(t)T_W(t) given by the chain in Eq. (1): rs,i2d=π⁡(KEc(TEcs)−1TW(t)DW(W)KW−1~W)−E.r^2d_s,i=π\! (K_E_c\,(T^s_E_c)^-1\,T_W(t)\,D_W(u_W)\,K_W^-1 u_W )-u_E. (A3) This term is Mahalanobis-whitened: the residual entering ρ is Σi−1/2r2ds,i _i^-1/2\,r^2d_s,i, where Σi _i is the per-match covariance derived from RoMa v2’s predicted certainty, so noisier wrist–external matches contribute proportionally less. • Wrist–external 3D metric lifting (3d, written rwe-3dr^we-3d). We use a metric 33D residual lifting the same wrist–external match into the world frame on both sides: rs,i3d=TW(t)DW(W)KW−1~W−TEcsDEc(E)KEc−1~E∈ℝ3(m).r^3d_s,i=T_W(t)\,D_W(u_W)\,K_W^-1 u_W-T^s_E_c\,D_E_c(u_E)\,K_E_c^-1 u_E ^3\ (m). (A4) • URDF-rendered robot reprojection (l). For a LoMa-G 22D–33D match between a rendered robot-surface point s,irob(δ)X^rob_s,i(δ q) (in the world frame, depending on δδ q through forward kinematics) and a real-image pixel s,iu_s,i in EcE_c: rs,il=π⁡(KEc(TEcs)−1s,irob(δ))−s,i.r^l_s,i=π\! (K_E_c\,(T^s_E_c)^-1\,X^rob_s,i(δ q) )-u_s,i. (A5) While robX^rob moves with δδ q, in practice we use δ=0δ q=0 for rendering the mesh, since the correction is generally very minor. • Featuremetric alignment (f). With ϕi∈ℝ64 _i ^64 the PCA-reduced DINO feature of rendered point s,irobX^rob_s,i and FEc:Ω→ℝ64F_E_c: ^64 the dense real-image feature field (Fig. A9): rs,if=FEc(π⁡(KEc(TEcs)−1s,irob))−ϕi.r^f_s,i=F_E_c\! (π (K_E_c\,(T^s_E_c)^-1\,X^rob_s,i ) )- _i. (A6) All reprojection terms operate on the MAGSAC-filtered inlier sets described in Appendix A.2. B.3 Ablations on the Pose Estimation Pipeline Table A1: Ablation relative to the full model on ∼ 1,000 scenes. All rows are the full model with the listed modification. Blue: 2D consistency (gold-inlier median px). Amber: 3D grounding. ++AMP 3D, −-FM is the deployed model. Full model ± E ↓ WE ↓ RD ↓ IoU ↑ (px) (px) (m) (mean) Full (all factors) 0.251 1.032 14.15 0.808 Weight tweaks ++ amplify 3D 0.251 1.027 14.02 0.808 ++ amplify FM 0.250 1.038 14.09 0.804 Leave-one-out (drop one factor) −- featuremetric 0.251 1.031 14.03 0.808 −- wrist-3D 0.251 1.032 14.05 0.808 −- featuremetric & 3D 0.251 1.032 14.07 0.809 −- split Text,ksT^s_ext,k 0.320 1.082 14.12 0.804 −- ext–ext factor 0.662 0.982 14.58 0.804 −- joint offset δqδ q 0.251 1.214 14.68 0.806 −- δTwristδ T_wrist 0.252 2.067 17.11 0.786 −- δTwristδ T_wrist & δqδ q 0.252 2.217 17.10 0.786 ++ amplify 3D, −- FM (ours) 0.251 1.027 14.01 0.809 We ablate the factors of the joint optimization in Table A1. From the table, it is clear that our joint factor graph-based optimization of the shared robot parameters (such as hand eye calibration and robot joint offsets) significantly improves the quality of the obtained poses. B.4 Hyperparameters and Solver All data terms use the robust Cauchy loss with empirically-set scales c2d=3σc_2d=3σ (whitened distance), c3d=10c_3d=10 m, cl=50c_l=50 px, and cee=5c_e=5 px. The 3D metric-lifting factor is up-weighted by λ3d=104 _3d=10^4 to rebalance the unit mismatch between its metre-scale residual (O⁡(10−2)O(10^-2)) and the pixel-scale factors (O⁡(1–50)O(1--50)); the remaining λk _k are 11. The Tikhonov regularization weight on the joint offsets is λq=10−2 _q=10^-2. The deployed (“ours”) configuration amplifies the 33D term and omits the featuremetric term (−-FM), corresponding to the “++amplify 3D, −-FM” row of Table A1. While the featuremetric term improved the robot depth by itself, we found that it was a bit redundant after amplifying the 3D loss term. However, we still include the term in our description since our framework is general and computes the feature metric loss, which could be applied to further refine after optimization. The graph is large but its Hessian is highly sparse: each episode’s 1212-DoF block couples only to the 1313 shared parameters, never to other episodes. We exploit this with a custom GPU-accelerated Levenberg–Marquardt solver. Each iteration applies the Schur complement to marginalize all per-scene blocks, reducing the step to a dense 13×1313× 13 global solve over the shared kinematic parameters, followed by parallel per-scene back-substitution for the extrinsics. This drops the computational cost from cubic in NsN_s to O⁡(Ns⋅123+133)O(N_s· 12^3+13^3), with a typical convergence wall time of ≈12.5≈12.5 minutes per robot group; in practice data loading, not the linear algebra, is the bottleneck. B.5 Qualitative results of DROID-3D Calibration Dataset vs Baseline We present some additional qualitative comparisons of the obtained poses by our method and the Point World based baseline method in Figure A10. Figure A10: Qualitative Results of the pose estimation of PointWorld [14] based baseline vs Ours. In all the cases our method produces significantly more aligned poses. The average robot depth error for PointWorld based approach on these examples is around 110-130m while ours is around 15-20m. We show only the points corresponding to segmented robot pixels in the image. In all the cases, it is clear that poses from the baseline method do not align, while they do for ours. Minor cleaning has been done to remove floaters due to imperfect segmentation. Appendix C DepthWorld Architecture and Training Details C.1 Depth Preprocessing Rationale (Supplement to Section 4.1) Raw depth maps are converted to VAE-encodable grayscale images through a strict four-step pipeline designed to keep the depth inside the distribution the Stable Video Diffusion (SVD) VAE was pretrained on: (1) We downsample to the 192×320192× 320 working resolution using unprojection and z-buffer min-pooling rather than bilinear interpolation, which preserves foreground geometry and prevents “flying pixel” artifacts at object boundaries. (2) Depths are clipped to the 95th percentile of the per-camera distribution (≈3≈ 3 m for external, ≈2≈ 2 m for wrist) to bound the VAE input while preserving manipulation-relevant content. (3) A log transform compresses the dynamic range, allocating more representational capacity to close-range objects where manipulation precision matters most. (4) Values are normalized to [0,1][0,1] to match the VAE’s expected input range, and the single channel is replicated to three channels. C.2 Spatial Latent Tiling vs. Other Depth Integration Architectures We opted for spatial latent tiling over dual-branch and channel expansion architectures. Dual-branch methods require running a parallel U-Net for depth, linked via cross-connections to the RGB branch, effectively doubling the parameter count and complicating the training dynamics. Channel expansion, on the other hand, increase the number of channels in the input and output layers of the model to incorporate the added modality. Our tiling approach leverages the observation that the SVD VAE naturally encodes replicated single-channel depth, allowing us to place RGB and depth side-by-side in a wider latent grid. This forces the single, unmodified U-Net to jointly denoise both modalities, learning geometric consistency without major changes to the architecture. We ablated this choice empirically. Table A2 compares spatial latent tiling against a dual-branch model—a parallel U-Net for depth with cross-connections to the RGB branch, in the style of SyncVP [30] trained for around 7070k steps and evaluated under the autoregressive rollout protocol of Appendix D.3. Different from [30], we do not use the same noise for both the modalities as our earlier experiments showed that it allowed the model to cheat in the EDM diffusion formulation. Furthermore, the model was first trained on RGB-only data for 50k steps and then continued for 20k steps with the dual branch. We would like to note that this is not a 1-1 comparison, but was done due to the additional compute requirements of the dual branch setup, which required more GPU hours to run through extensive FSDP parallelism. For the channel expansion architecture, we simply double the number of channels in the input and output layers of the model to incorporate the added modality. We train this in a single stage for 70k steps. Spatial tiling performs better: it improves every RGB metric on both views (e.g. +1.7+1.7 dB PSNR on external and +1.5+1.5 dB on wrist), and matches or improves depth quality, with the dual-branch only marginally ahead on external RMSE. Therefore, this influenced our decision to avoid changing the architecture. Table A2: Ablation: Spatial latent tiling vs. other integration architectures, at ∼70 70k training steps under the autoregressive rollout protocol (Appendix D.3). All are joint RGB–depth models; depth metrics are against the VAE-round-tripped GT (Appendix C.8). Spatial tiling matches or beats both the baseline integration architectures architecture on nearly every metric and view despite leaving the U-Net unmodified. Best per view/metric in bold, second best underlined. RGB quality Depth quality View Model PSNR ↑ SSIM ↑ LPIPS ↓ AbsRel ↓ RMSE ↓ δ1 _1 ↑ External Dual-branch 21.94 0.8039 0.1010 0.0861 0.232 0.9383 Channel expansion 22.22 0.8017 0.1021 0.0796 0.220 0.9414 Spatial tiling (ours) 23.66 0.8252 0.0945 0.0854 0.235 0.9383 Wrist Dual-branch 16.05 0.5441 0.3554 0.2523 0.161 0.7685 Channel expansion 16.34 0.5593 0.3422 0.2801 0.158 0.7930 Spatial tiling (ours) 17.58 0.5954 0.3276 0.2420 0.159 0.8001 C.3 Effect of the Depth Source on DepthWorld DepthWorld is trained on S2M2 depth (Appendix A.1). To quantify how much the choice of depth source matters, we train two further models with identical architecture and training protocol that instead use the ZED Ultra and ZED Neural depth of the wrist and external cameras. All three models are trained for 5050k steps and evaluated under the autoregressive rollout protocol of Appendix D.3; Table A3 reports the results. Two caveats apply when reading the table. The RGB metrics are directly comparable across rows, since every model predicts the same RGB frames. The depth metrics are not: each model is scored against the depth source it was trained on, so they measure how faithfully a model reproduces its own supervision rather than its absolute metric accuracy. The depth source has a large effect. The model trained on ZED Ultra depth is worst on every metric of both views: AbsRel rises from 0.0740.074 to 0.1940.194 on the external view and from 0.2660.266 to 0.4610.461 on the wrist view, where Ultra depth is particularly sparse and noisy (Figure A1). Notably, the poor depth target also degrades RGB prediction, by 2.02.0 dB PSNR on the external view and 1.41.4 dB on the wrist view, even though the RGB supervision is unchanged. Sparse and noisy depth is therefore not a neutral auxiliary signal but actively harms the joint RGB–depth model, and a high-quality depth source is crucial for training it. S2M2 and ZED Neural depth, in contrast, yield very similar models. The RGB metrics agree to within 0.20.2 dB PSNR on both views, and the depth metrics are close, with S2M2 slightly ahead on the external view and on wrist δ1 _1, and ZED Neural slightly ahead on wrist AbsRel. This is expected: the two depth sources are highly correlated (∼0.97 0.97 correlation between their depth values), so they provide nearly the same training signal; where they differ is in coverage and boundary sharpness (Appendix A.1). Table A3: Ablation: Depth source used to train DepthWorld, at 5050k training steps under the autoregressive rollout protocol (Appendix D.3). All are joint RGB–depth models with identical architecture and training protocol; only the depth maps used for training differ. Best per view/metric in bold, second best underlined. External Wrist K=10K=10 rollout PSNR↑ LPIPS↓ AbsRel↓ δ1↑ _1 PSNR↑ LPIPS↓ AbsRel↓ δ1↑ _1 RGBD, S2M2 depth (ours) 24.07 0.0877 0.074 0.946 17.68 0.3270 0.266 0.797 RGBD, ZED NEURAL depth 24.26 0.0881 0.079 0.939 17.77 0.3260 0.254 0.794 RGBD, ZED ULTRA depth 22.11 0.0947 0.194 0.830 16.28 0.3395 0.461 0.655 C.4 Auxiliary Point-Map Head (Supplement to Section 4.2) The only auxiliary module is a single DPT [32] point-map decoder. It operates on the model’s predicted clean latent for the future frames, x^0,f x_0,f, not on intermediate U-Net activations. For each future frame and view, the horizontal tiling is split back into its RGB and depth halves, and the two 44-channel latents are concatenated per view into an 88-channel input of shape (B⋅V,Tfuture, 8, 24, 40)(B·V,\,T_future,\,8,\,24,\,40). A learned adapter then maps this latent into the DPT feature space, building a four-scale feature pyramid from the 24×4024× 40 latent: 24×40→12×20→6×10→3×524× 40→ 12× 20→ 6× 10→ 3× 5. We reuse VGGT’s [39] DPT fusion trunk and final regressor: the four RefineNet fusion blocks (refinenet1--4) and the two output convolutions (output_conv1, 2) of VGGT’s point head are transferred and produce the per-pixel (X,Y,Z)(X,Y,Z) point map in the robot base frame and a raw confidence channel. Everything that adapts the Ctrl-World latents into that feature space is trained from scratch: the input-clamp/latent input path, the stem, the three pyramid stages, and the scale-projection convolutions (layer1--4_rn) wherever VGGT’s shapes do not match. The decoder upsamples to full image resolution, outputting four channels at 192×320192× 320 (predicted tensor (B,V,Tfuture, 4, 192, 320)(B,\,V,\,T_future,\,4,\,192,\,320)). The raw confidence is mapped to a strictly positive confidence via Cv,t()=1+exp⁡(⋅)C_v,t(u)=1+ (·), following DUSt3R [40]/MASt3R [19]. Table A4 isolates the contribution of the recalibrated extrinsics to this head: training PM-DPT with point-map targets built from DROID’s factory extrinsics instead of the refined DROID-3D extrinsics leaves RGB quality unchanged but gives worse wrist-view depth (AbsRel 0.2330.233 vs. 0.2200.220). Table A4: Ablation: Extrinsics used to build the point-map targets, under the autoregressive rollout protocol (Appendix D.3). Both are RGB+D PM-DPT models trained identically; the robot-frame point-map targets are backprojected either with DROID’s factory extrinsics or with the refined DROID-3D extrinsics of Appendix B. Best per view/metric in bold. External Wrist K=10K=10 rollout PSNR↑ LPIPS↓ AbsRel↓ δ1↑ _1 PSNR↑ LPIPS↓ AbsRel↓ δ1↑ _1 PM-DPT, factory extrinsics 24.10 0.0891 0.075 0.945 18.01 0.3131 0.233 0.817 PM-DPT, refined extrinsics 24.13 0.0890 0.075 0.945 18.03 0.3124 0.220 0.819 C.5 Training Objective The combined objective is ℒ=ℒdenoise+λpmℒpm,λpm=0.005,L=L_denoise+ _pm\,L_pm, _pm=0.005, (A7) where ℒdenoiseL_denoise is the standard Stable Video Diffusion denoising objective applied to the joint tiled latent, with SVD’s noise schedule and loss weighting inherited unchanged. The point-map term ℒpmL_pm supervises the head’s predicted point maps Pv,tP_v,t against the ground-truth point maps Pv,t∗P^*_v,t (the world-frame / robot-base XYZ point maps obtained by backprojecting the GT depth through the camera intrinsics and the calibrated extrinsics of Appendix B), with a per-pixel validity mask mv,t()∈0,1m_v,t(u)∈\0,1\. Write =Pv,t()p=P_v,t(u) and =Pv,t∗()g=P^*_v,t(u) for the predicted and target 33D points at a pixel. We compress the large dynamic range of metric coordinates with a direction-preserving log map: ϕ⁡()=∥log⁡(1+∥),φ(x)= x \, \! (1+ ), (A8) and measure the residual in this space: =ϕ⁡()−ϕ⁡(),s=∥2c2,c=0.05.d=φ(p)-φ(g), s= ^2c^2, c=0.05. (A9) The scaled residual s is passed through Barron’s general robust loss [3] with shape α=0.5α=0.5, ρα(s)=|α−2|α[(s|α−2|+1)α/2−1], _α(s)= |α-2|α [ ( s|α-2|+1 )^\!α/2-1 ], (A10) and combined with the confidence C=Cv,t()=1+exp⁡(⋅)C=C_v,t(u)=1+ (·) in the DUSt3R confidence-weighted form kv,t()=Cρα(s)−βlog⁡C,β=0.2.k_v,t(u)=C\, _α(s)-β C, β=0.2. (A11) Two further operations are applied before aggregation. First, the largest 5%5\% of per-pixel terms k within each (batch element, view) are dropped as outliers. Second, each future-frame sample is weighted by its EDM loss weight, clamped by a min-SNR-γ of 55. The point-map loss is the masked, weighted mean over valid pixels, ℒpm=∑mv,t()σwkv,t()∑mv,t(),L_pm= Σ m_v,t(u)\, _w\,k_v,t(u)Σ m_v,t(u), (A12) with the sum over the predicted future frames only. C.6 Training Schedule and Optimization We initialize from the Stable Video Diffusion checkpoint used by Ctrl-World [12] and inherit its action conditioning unchanged: a per-frame 77-dimensional end-effector action (66-DoF pose plus 11 gripper state). Training proceeds in two stages for stability, for 90,00090,000 steps in total. Stage 1 trains the joint RGB–depth model alone for 40,00040,000 steps with λpm=0 _pm=0, letting the model learn the joint tiled-latent distribution. Stage 2 attaches the DPT head (VGGT fusion trunk and regressor reused, latent adapter learned from scratch; Appendix C.4) and trains all components jointly for a further 50,00050,000 steps with λpm=0.005 _pm=0.005. We optimize with AdamW at a constant learning rate of 10−510^-5 (no schedule) and a global batch size of 6464. We maintain an exponential moving average (EMA) of the weights with decay 0.99990.9999, and all reported results use the EMA model. Total training takes approximately two days on two H200 nodes. C.7 Inference and Autoregressive Rollout At inference the model denoises the joint tiled latent over a temporal window of 1111 frames (66 history, 55 future) at 55 Hz, using 5050 denoising steps and a classifier-free guidance scale of 22 (matching Ctrl-World [12]). The history frames are sampled sparsely over the past, following Ctrl-World [12], rather than as the six immediately preceding frames. Each rollout starts from the first frame, with the history buffer initialized by repeating it; the model then generates 1010 consecutive future chunks autoregressively, feeding each predicted chunk back as history for the next window. Reported metrics therefore reflect compounding prediction error from a single-frame start, rather than teacher-forced single-step error. C.8 Depth Reference and De-normalization for Metric Evaluation To recover metric depth from a predicted depth tile we invert the four-step preprocessing of Appendix C.1: the depth tile is VAE-decoded, the three replicated channels are averaged back to one, values are mapped from [0,1][0,1] back through the inverse of the normalization and the inverse log transform, and the per-camera percentile clip is undone. The resulting depth is metric. The reference is the same S2M2S^2M^2 ground-truth depth of Appendix B, but passed through the VAE encode–decode round-trip before scoring. Measuring against the VAE reconstruction of the GT depth, rather than the raw GT, factors out the irreducible round-trip error that bounds any latent-space predictor, so the depth metrics reflect prediction quality relative to the best depth the latent representation can represent. Appendix D Experimental Setup and Metrics D.1 Calibration Baseline Modifications (Supplement to Section 5.1) To ensure a fair evaluation of our joint factor graph optimization against the per-scene refinement baseline (modeled on PointWorld), we modified the baseline’s depth source. Instead of using FoundationStereo as in the original PointWorld pipeline, we substituted our global-matching S2M2S^2M^2 depth. This isolates the performance delta specifically to the structural difference in the optimization (per-scene vs. robot-coupled joint graph) rather than a difference in depth predictors. D.2 Geometric Calibration Metrics Because ground-truth extrinsics are unavailable for DROID, we define four geometric proxy metrics to evaluate calibration quality: 1. Ext-Ext (E) Reprojection Error. The median pixel reprojection error between the two external cameras, computed on a gold-inlier set extracted via MAGSAC enforcing epipolar and Kabsch-based 3D alignment. 2. Wrist-Ext (WE) Reprojection Error. The analogous median pixel reprojection error measured between the dynamic wrist camera and the fixed external cameras. 3. Robot Depth Error (RD). The median absolute difference (in millimeters) between the predicted S2M2S^2M^2 depth map and the depth obtained by rendering the robot URDF using the optimized extrinsics and joint angles. 4. Mask IoU. The Intersection-over-Union between the URDF-rendered robot silhouette mask and a ground-truth robot segmentation mask produced by RoboEngine [46] on the real image. D.3 World-Model Evaluation Protocol Training set and split. DepthWorld is trained on the calibrated episodes that constitute DROID-3D. Starting from the publicly released raw DROID split (∼75 75k trajectories with high-quality RGB, raw ZED SVO stereo, and factory camera parameters), per-robot filtering for broken metadata or unrecoverable calibration leaves the 71,10071,100 episodes across 2828 robots and 1313 labs reported in Appendix B. DepthWorld trains on exactly these 71,10071,100 episodes. We hold out 1%1\% as a validation set and report world-model metrics on 256256 trajectories sampled at random from it. Because the holdout is random at the episode level, the held-out trajectories come from the same robots and labs seen in training; these numbers therefore measure held-out-trajectory prediction quality, not cross-embodiment or cross-lab generalization. Baseline. The RGB baseline (RGB in Tables 1–2) is the Ctrl-World [12] backbone (multi-view RGB only), which we retrain rather than use off the shelf. The released Ctrl-World checkpoint is trained on the 9595k DROID split stored in RLDS format, whereas DROID-3D is constructed over the ∼75 75k publicly released raw split. Retraining the RGB model on the same 71,10071,100-episode set, split, architecture, and training budget as DepthWorld makes RGB and RGB+D a controlled comparison in which the only difference is the depth signal. Metrics. For RGB we report PSNR, SSIM [42], and LPIPS [48]. For depth we report AbsRel, RMSE, and δ1 _1, where AbsRel=1N∑|d−d∗|/d∗AbsRel= 1NΣ|d-d^*|/d^*, RMSE is in meters, and δ1 _1 is the fraction of valid pixels with max⁡(d/d∗,d∗/d)<1.25 (d/d^*,\,d^*/d)<1.25. Depth metrics are evaluated in metric units without any per-frame scale-and-shift alignment, against the VAE decoded GT depth reference of Appendix C.8, and are masked to pixels with a valid depth reference. All metrics are reported separately for external and wrist views, and are averaged over all rollout frames. D.4 Comparison with PointWorld (Supplement to Section 5) PointWorld [14] and DepthWorld predict different things. PointWorld is a 3D scene-flow model: given a set of seed points in the robot frame at the anchor frame, it rolls out the future 3D position of every seed point. DepthWorld predicts future depth images per camera, from which 3D points are obtained by sampling the predicted depth at a pixel and unprojecting it with the calibrated intrinsics and extrinsics. Neither output can be scored directly against the other, so the protocol below lifts both to 3D points in the robot frame at the same tracked scene locations and scores them there. Two consequences of this design should be kept in mind. First, DepthWorld is only ever asked for the depth of a known pixel, which is an easier question than PointWorld’s task of predicting where an entire point set goes. Second, DepthWorld predicts depth in the latent space of the SVD VAE, whose encode–decode round trip alone has an error floor that PointWorld’s point representation does not have; how we account for this is described below. Shared validation split. We use exactly the 256256 held-out windows of Table 2, recovered from the same seeded random permutation of the validation set, so both models are evaluated on identical clips. Each window consists of an anchor frame followed by 4040 future frames at 55 Hz, i.e. an 88 s horizon. Of these windows, 141141 contain moving scene points that survive the persistent point-set criterion described below and are used for the moving-point metrics, and 147147 are used for the whole-scene metric. We report a short horizon of 22 s (frame 1010) and the full horizon of 88 s (frame 4040). Tracker-based pseudo ground truth. DROID has no ground-truth 3D trajectories, so we build them from point tracks, following the procedure of the PointWorld repository exactly. On the anchor frame of each external view we place a 64×6464× 64 grid of query points and track them through all 4141 frames with CoTracker3 [karaev2024cotracker3], the tracker used by PointWorld’s authors, at the native 192×320192× 320 resolution, which matches the depth resolution and avoids rescaling. At every frame each tracked pixel is lifted to 3D by sampling the ground-truth S2M2S^2M^2 depth of that frame and unprojecting it with the camera intrinsics and the calibrated camera-to-world pose of DROID-3D. Points with invalid depth (z≤0z≤ 0 or z>4z>4 m) are discarded. We then keep only points inside the robot workspace, a box of [0,0.7]×[−0.4,0.4]×[−0.3,1.2][0,0.7]×[-0.4,0.4]×[-0.3,1.2] m in the robot base frame; this removes floor, wall, and background tracks, which otherwise dominate the error. Remaining spatial outliers, mostly tracks that jump between depth layers, are removed with a multi-scale DBSCAN [ester1996dbscan] filter at radii 0.20.2, 0.50.5, and 1.01.0 m, dropping any track that is an outlier in a large fraction of frames. A point is labelled moving if its ground-truth displacement from the anchor frame exceeds 33 cm. Points on the robot arm, identified by the URDF silhouette dilated by 33 px, are excluded from all scored sets, for the reasons given next. Why robot points are excluded. In PointWorld the robot is an input, not a prediction target: the arm enters as points computed from the joint states through the URDF (the gripper only, in the released checkpoint), and its training clouds are built with the robot cut out. Seeding it with arm pixels therefore asks it to predict the arm as if it were scenery, a case it never saw in training, and the arm disintegrates within a few frames. Excluding the robot also matches PointWorld’s own evaluation, whose scene flows exclude it by construction. More fundamentally, both models are conditioned on the robot’s actions, so for PointWorld the arm’s future position is simply forward kinematics of the given joints, and scoring arm points would test calibration rather than world modelling; the question worth comparing is how the scene responds, i.e. objects, cloth, and doors. Finally, the arm would otherwise dominate the metric: with the robot kept, about 178178 points per window qualify as moving, against about 4545 without it, so roughly three quarters of a robot-inclusive moving score would be arm motion that both models are handed. The exclusion has costs. DepthWorld receives no credit for rendering the arm, which a video model must do correctly. Because the mask is the dilated URDF silhouette, points on a grasped object right at the gripper are dropped as well, as in PointWorld’s own pipeline. Cables remain in the scene, since they move with the arm but are not part of the URDF. Persistent point set. Tracked points become occluded over the horizon. Scoring whatever is valid at each frame shrinks the scored set over time and biases it toward easy, unoccluded points, which makes the error appear to decrease with horizon. We therefore score a fixed persistent set per window: the moving points that are valid in every frame of the horizon and that could be matched to a PointWorld seed (next paragraph), about 4545 points per window. Error then increases with horizon, as it should. The set is defined from the ground truth and PointWorld only, never from DepthWorld’s predictions, so PointWorld is a fixed reference when different DepthWorld checkpoints are compared. Prediction clouds. PointWorld. PointWorld was trained with up to 12,00012,000 scene points, and seeding it with only the few hundred tracked points is out of distribution and understates it. We therefore seed it in distribution with 12,00012,000 scene points sampled from the workspace-filtered, robot-masked anchor-frame point map of both external views, supply the robot as URDF points from the joint states as in its training, roll out the dense cloud with the released large DROID checkpoint, and read off the trajectory of each tracked point from its nearest dense seed (nearest-neighbour match on the anchor frame, accepted if closer than 11 cm). PointWorld thus receives full multi-view context but is scored on exactly the tracked points. DepthWorld. For each tracked pixel and frame we bilinearly sample the predicted depth of the corresponding external view, de-normalised to metric depth as in Appendix C.8, and unproject it with the same intrinsics and pose used for the ground truth. Depth references and the own-ceiling comparison. For every window three depth stacks are available: the raw ground-truth S2M2S^2M^2 depth, the same depth after an encode–decode round trip through the SVD VAE (VAE-GT, the reference of Appendix C.8), and DepthWorld’s prediction. The round trip is not lossless: on moving points it alone displaces the ground truth by about 4545 m (median), and because DepthWorld must operate in the VAE’s latent space, this is a floor its network cannot go below. Scoring DepthWorld against raw depth therefore charges it for the VAE floor, whereas scoring PointWorld against VAE-GT would penalise it for a smear it never produced. We compute the full 2×22× 2 of PointWorld, DepthWorld×raw, VAE-GT and report the diagonal as the fair comparison: PointWorld against raw depth, and DepthWorld against VAE-GT. Each model is thus measured against the best target its own representation can reach. The off-diagonal entries are unfair in opposite directions and reverse the winner depending on which is chosen. As a sanity check, the raw depth stored with DepthWorld’s outputs lifts to exactly the same 3D points as our own ground-truth pipeline, ruling out any scale or alignment mismatch between the two pipelines. Figure A11: Qualitative comparison with PointWorld on a held-out window in which the robot opens a door. Top: observed external RGB at the input frame and at 22, 44, 66, and 88 s. Rows 2–5: workspace point clouds in the robot frame at the same times: the ground truth lifted from S2M2S^2M^2 depth; PointWorld’s dense rollout seeded at t=0t=0; the ground truth after the SVD-VAE round trip, which is DepthWorld’s own reference in Table A5; and DepthWorld’s predicted depth unprojected into the robot frame. PointWorld follows the arm at short horizon, but from about 44 s its cloud scatters and drifts, consistent with its rising whole-scene Chamfer distance. DepthWorld stays coherent over the full 88 s and reconstructs the opening door in agreement with its VAE reference, at the cost of thin structures and some foreground geometry that are present in the raw ground truth. Table A5: Comparison with PointWorld on the shared 256256-window split (141141 windows for the moving-point metrics, 147147 for the whole scene), fair diagonal of the own-ceiling protocol: PointWorld is scored against raw ground-truth depth, DepthWorld (the RGB+D PM-DPT variant) against the VAE round-tripped ground truth. Medians over windows in m at horizons of 22 s and 88 s; ℓ2 _2 is averaged over the frames up to the horizon, Chamfer is evaluated at the horizon frame. Lower is better; best per column in bold. Tracked-point ℓ2 _2, moving Chamfer, moving Chamfer, whole scene Model 2 s 8 s 2 s 8 s 2 s 8 s PointWorld [14] vs. raw GT 23 56 21 58 8 18 DepthWorld vs. VAE-GT 31 46 25 32 8 9 Metrics and scoring regions. We report two metrics in two regions, as medians over windows; the ℓ2 _2 error is averaged over the frames up to the horizon and the Chamfer distance is evaluated at the horizon frame. The tracked-point ℓ2 _2 error is the Euclidean distance between the predicted and target 3D position of the same point, averaged over the persistent moving set; it penalises a point that lands on the wrong part of the scene and is only possible because the persistent set provides stable correspondences. The symmetric Chamfer distance, 12(meanN(pred→target)+meanN(target→pred)) 12 (mean\,N(pred )+mean\,N(target ) ), is correspondence-free and measures whether the predicted cloud occupies the right space, regardless of point identity. The moving region scores only moving points and tests whether the model predicts the manipulation itself. The whole scene scores all workspace points, subsampled to 8,0008,000 per cloud for equal density, and is dominated by static structure: it mostly measures whether the scene is reconstructed without drift. A Chamfer distance much smaller than the ℓ2 _2 error indicates that the predicted cloud has roughly the right shape but that correspondences have drifted. Results. Table A5 reports the fair diagonal at 22 s and 88 s, and Figure A11 shows a representative window. PointWorld is more accurate on tracked moving points at short horizon (2323 vs. 3131 m at 22 s) but degrades faster, and DepthWorld is ahead at 88 s (4646 vs. 5656 m). On the whole scene the two are equal at 22 s (88 m), after which DepthWorld’s Chamfer distance stays flat (99 m at 88 s) while PointWorld’s grows to 1818 m: the depth-image representation reconstructs the static scene without drift, whereas PointWorld’s point cloud drifts as the rollout proceeds. In the moving region PointWorld is slightly ahead at 22 s (2121 vs. 2525 m) and DepthWorld degrades far less by 88 s (3232 vs. 5858 m). The DepthWorld model in this comparison is the RGB+D PM-DPT variant of Table 2. We stress the asymmetries noted above: DepthWorld answers an easier query and is scored above its VAE floor. Against raw depth, PointWorld’s point representation retains a large advantage on moving objects, a representational cost of latent-space depth prediction that we consider an important direction for future work. D.5 Comparison with TesserAct (Supplement to Section 5) TesserAct [49] is a text-conditioned, single-view RGB–depth–normal (RGBDN) video model built on CogVideoX-5b-I2V [yang2024cogvideox], a 5B-parameter image-to-video diffusion transformer. It is not comparable off the shelf: it has no action conditioning, sees a single camera, and was trained on different data. We therefore adapt it minimally to our setting and retrain it on the same episodes as DepthWorld at an equal step budget. Model and adaptation. The TesserAct architecture is kept unchanged. Its patch embedding splits the latent into three modality streams (RGB, depth, normal), projects each, and sums them into one shared token sequence, and two additional output heads produce the depth and normal latents. The transformer is initialised from the public CogVideoX-5b-I2V weights rather than from TesserAct’s released RGBDN checkpoint, mirroring DepthWorld, which starts from the generic SVD checkpoint rather than from a depth-pretrained model. The single conceptual change is the conditioning signal. CogVideoX conditions on T5 text tokens passed as the cross-modal context of its joint-attention blocks. We replace these with action tokens: for each of the 99 frames of a clip, the 7-D end-effector pose and gripper state is mapped by a small MLP (7→1024→40967→ 1024→ 4096 with a SiLU non-linearity), added to a learned per-frame temporal embedding, and layer-normalised, giving 99 tokens of width 40964096 that occupy the T5 slot. Because CogVideoX-5b-I2V uses rotary position embeddings, the context length is free, so 99 action tokens replace the 226226 padded text tokens without touching any transformer block. This mirrors DepthWorld’s per-frame end-effector action conditioning. For classifier-free guidance, the whole trajectory is replaced by an all-zero null trajectory with probability 0.050.05, and the conditioning frame is dropped with the same probability. Data. We use the same DROID-3D training episodes as DepthWorld, restricted to the two external cameras; the wrist camera is excluded, and each external camera is treated as an independent single-view sample, since TesserAct has no notion of a multi-view rig. Clips are contiguous 99-frame windows at 55 Hz and 192×320192× 320 resolution, with the per-frame actions aligned to the same timestamps. Depth is the DROID-3D metric depth mapped to TesserAct’s log-grayscale encoding over 0.300.30–3.303.30 m and replicated to three channels; normals are computed geometrically from the depth; actions are percentile-normalised per dimension. This yields 144,436144,436 training clips and 1,5341,534 validation clips. Training. The training objective is TesserAct’s own: given the VAE-encoded first RGBDN frame and the action trajectory, denoise the 99-frame RGBDN clip with CogVideoX’s velocity target, with the loss being the sum of three separately weighted per-modality MSE terms. Each modality is encoded separately by the frozen CogVideoX VAE. Both the transformer and the action encoder are trained. We train on 1616 H200 GPUs with data-parallel training in bf16 with gradient checkpointing, full-precision AdamW (β1=0.9 _1=0.9, β2=0.95 _2=0.95, weight decay 10−310^-3, gradient clipping at 1.01.0), a global batch of 6464 clips, and a constant learning rate of 5×10−55× 10^-5 after a 200200-step warm-up. We train for 5050k steps (about 2020 hours), matching the 5050k-step budget of the DepthWorld model it is compared against. Evaluation. We mirror the DepthWorld protocol of Appendix D.3. Generation uses the released TesserAct pipeline without its text encoder: the action tokens are passed as the prompt embedding and the null trajectory as the negative embedding, with classifier-free guidance scale 6.06.0 and 5050 DPM-Solver steps at 192×320192× 320. Rollouts are autoregressive and closed-loop: five chained 99-frame generations, each re-anchored on the last generated RGBDN frame (ground truth is never re-injected) and given the next 88-frame action segment, producing 1+5×8=411+5× 8=41 frames or 8.28.2 s, the same horizon as DepthWorld, which reaches it in ten steps of four new frames. The evaluation set is 128128 held-out samples, each an episode with a fixed anchor frame scored on both external views; the two views are averaged with equal weight. As for DepthWorld, metrics are computed against the ground truth after an encode–decode round trip through the model’s own VAE (here the CogVideoX VAE), so that each model is scored above its own representational floor (Appendix C.8); we additionally record the error against raw depth and the VAE floor itself. Metric depth is recovered by inverting the log encoding, the conditioning frame is excluded, and the RGB and depth metrics of Appendix D.3 are averaged over the 4040 generated frames. Results. At the same 5050k steps, averaged over the 4040 generated frames on the external views, the adapted TesserAct reaches 19.4319.43 dB PSNR and 0.1540.154 AbsRel, against 24.0724.07 dB and 0.0740.074 for the DepthWorld RGB+D model (the S2M2 row of Table A3): it trails by 4.64.6 dB and about 2×2× in AbsRel. The gap is mostly one of rollout stability rather than single-step quality: TesserAct’s first generated chunk is reasonable (PSNR 23.5523.55, LPIPS 0.0650.065, AbsRel 0.0820.082, δ1 _1 0.950.95 over its eight future frames), but quality decays steadily along the rollout, from PSNR 27.927.9, AbsRel 0.0480.048, and δ1 _1 0.970.97 at the first generated frame to 16.816.8, 0.2110.211, and 0.810.81 at the fortieth, with no visible seams at the re-anchoring boundaries. Qualitatively, the generated arm follows the commanded trajectory over the full 88 s, indicating that the action conditioning is used. Two differences should be kept in mind when reading these numbers. TesserAct sees a single view and so has no cross-view context, whereas DepthWorld predicts all cameras jointly. Its depth also passes through the CogVideoX VAE, whose floor on raw depth is large (AbsRel 0.280.28 on one external camera), so its raw-depth error is dominated by the representation rather than the network; scoring both models above their own VAE floor, as we do, removes this effect from the comparison. The guidance scale was not tuned.

Instant research discovery