Paper deep dive
WildHSR: Metric Feed-Forward 4D People-Scene Reconstruction from a 3D Foundation Model
Jerrin Bright, John Zelek
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/25/2026, 4:51:28 AM
Summary
The paper introduces WildHSR, a feed-forward system for metric 4D people-scene reconstruction from monocular video using a 3D foundation model (VGGT-Ω). It addresses the lack of metric scale and persistent person identity in up-to-scale models by introducing a Scale Readout (pretrained via pseudo-labels from web video and fine-tuned on real data) and an identity readout leveraging intermediate query-key features for person association. WildHSR achieves state-of-the-art performance on EMDB-2 and RICH datasets.
Entities (9)
Relation Signals (7)
Pelvis Readout → partof → WildHSR
confidence 95% · WildHSR combines both readouts... Pelvis Readout... Scale Readout
Scale Readout → partof → WildHSR
confidence 95% · WildHSR combines both readouts to reconstruct metric cameras, scene and people
WildHSR → uses → VGGT
confidence 95% · A window of video passes once through a 3D foundation model (VGGT-Ω...)
WildHSR → leads → RICH
confidence 90% · On RICH, it leads feed-forward people-and-scene methods on WA-MPJPE and W-MPJPE.
WildHSR → outperforms → RTE
confidence 90% · On EMDB-2, WildHSR is the first feed-forward method... to beat the best optimization-based WA-MPJPE and RTE
WildHSR → outperforms → WA-MPJPE
confidence 90% · On EMDB-2, WildHSR is the first feed-forward method... to beat the best optimization-based WA-MPJPE
VGGT → successorof → VGGT
confidence 90% · VGGT-Ω (Wang et al., 2026), the successor of VGGT (Wang et al., 2025a)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:3D foundation models recover video cameras and geometry in one forward pass, but some of the strongest are up to scale. Joint people-scene reconstruction then requires two missing outputs: metric scale and persistent person identity. We ask whether one up-to-scale foundation representation can support both through lightweight adaptation. Exact metric labels are scarce, but unlabeled in-the-wild video is abundant. We use people in curated web video to initialise the solution: a posed metric body and 2D keypoints give an approximate, closed-form scale pseudo-label. These pseudo-labels pretrain a Scale Readout, which is then fine-tuned together with a lightweight adapter using exact metric supervision from standard real-video training splits. At inference the head predicts metric scale from foundation-model tokens, without the ruler or its teachers. For person identity, we probe the pretrained foundation model alone and find evidence that its intermediate query-key features encode person correspondence across frames. In most evaluated moving-person clips, a mid-layer token prefers that person over the vacated location and other people. A tiny projection reads this correspondence; together with metric pelvis motion and proposal confidence, it drives dustbin-aware Sinkhorn association of per-frame bodies. WildHSR combines both readouts to reconstruct metric cameras, scene and people from monocular video. Each window is predicted feed-forward; analytic association and Sim(3) composition connect windows. On EMDB-2, WildHSR is the first feed-forward method in the published comparison to beat the best optimization-based WA-MPJPE and RTE while leading feed-forward methods on all three world-frame metrics. On RICH, it leads feed-forward people-and-scene methods on WA-MPJPE and W-MPJPE. The complete pipeline runs at 10.1 fps on one GPU.
Tags
Links
- Source: https://arxiv.org/abs/2609.29106v1
- Canonical: https://arxiv.org/abs/2609.29106v1
Trouble viewing inline? Open PDF directly →
Full Text
68,884 characters extracted from source content.
Expand or collapse full text
WILDHSR: METRIC FEED-FORWARD 4D PEOPLE-SCENE RECONSTRUCTION FROM A 3D FOUNDATION MODEL Jerrin Bright, John Zelek Vision and Image Processing Lab, University of Waterloo, Canada j3bright, jzelek@uwaterloo.ca Long walk, 33 s, one person (EMDB) Running in circles, 10 s (EMDB)Crowded street, 14 people (3DPW)Ramp and stairs (3DPW)Indoor walk, first 17 s (EMDB) person, one colour per tracksame person, earliercamera path and views1 m floor grid Figure 1: Metric people, scene and camera from one moving camera. Each panel overlays persistent people, the person-free scene and the camera path on a 1 m grid. The large EMDB example spans 33 s; its insets show pelvis-token attention and a top view against ground truth after rigid rotation-and-translation alignment, preserving the predicted scale. The remaining clips cover cyclic motion, a 14-person street, stairs and an indoor walk. Inference receives a single monocular video. ABSTRACT 3D foundation models recover video cameras and geometry in one forward pass, but some of the strongest are up to scale. Joint people-scene reconstruction then requires two missing outputs: metric scale and persistent person identity. We ask whether one up-to-scale foundation representation can support both through lightweight adaptation. Exact metric labels are scarce, but unlabeled in-the-wild video is abundant. We use people in curated web video to initialise the solution: a posed metric body and 2D keypoints give an approximate, closed-form scale pseudo-label. These pseudo-labels pretrain a Scale Readout, which is then fine- tuned together with a lightweight adapter using exact metric supervision from standard real-video training splits. At inference the head predicts metric scale from foundation-model tokens, without the ruler or its teachers. For person iden- tity, we probe the pretrained foundation model alone and find evidence that its intermediate query-key features encode person correspondence across frames. In most evaluated moving-person clips, a mid-layer token prefers that person over the vacated location and other people. A tiny projection reads this correspondence; together with metric pelvis motion and proposal confidence, it drives dustbin- aware Sinkhorn association of per-frame bodies. WildHSR combines both read- outs to reconstruct metric cameras, scene and people from monocular video. Each window is predicted feed-forward; analytic association and Sim(3) composition connect windows. On EMDB-2, WildHSR is the first feed-forward method in the published comparison to beat the best optimization-based WA-MPJPE and RTE while leading feed-forward methods on all three world-frame metrics. On RICH, it leads feed-forward people-and-scene methods on WA-MPJPE and W-MPJPE. The complete pipeline runs at 10.1 fps on one GPU. 1 arXiv:2609.29106v1 [cs.CV] 24 Sep 2026 1INTRODUCTION Reconstructing people together with their surroundings from monocular video requires bodies, cam- eras and scene geometry in one metric world. 3D foundation models now recover strong relative geometry and camera motion in a single forward pass (Wang et al., 2024a; 2025b;a; 2026), in- cluding video whose content moves (Zhang et al., 2025). Some learn metric output from metric supervision (Wang et al., 2025b; Keetha et al., 2026; Ma et al., 2026); some of the strongest re- ported camera estimates instead come from models whose output is up to scale (Wang et al., 2025a; 2026), and it is such a representation we study. Joint people-scene reconstruction then lacks two quantities. First, monocular geometry is ambiguous up to a global metric scale (Eigen et al., 2014), and an up-to-scale model normalises each window to an arbitrary unit. Second, its geometric output has no persistent person identity: it does not say which body in one frame is the same person later. Over time, independently scaled windows distort trajectories, while identity switches splice different people into one path. The prevailing answer is to obtain both outside the up-to-scale representation: metric scale from a metric-native backbone (Chen et al., 2025b; Wang et al., 2025b), from metric depth priors (Wang et al., 2024b), or from synthetic metric supervision and a depth teacher (Li et al., 2026; Shi et al., 2026), and person identity from external detectors and trackers (Ye et al., 2023; Shin et al., 2024; Shen et al., 2024). These choices introduce separate models, supervision sources or sequence- processing stages. We ask whether one up-to-scale foundation representation can support both capabilities without an external scale network or separately trained tracker. Scale requires supervi- sion. Person correspondence presents a different opportunity: 3D foundation models learn matching and geometric consistency across views, so their intermediate representations may retain correspon- dence even on moving people. Large vision models often encode structure beyond their explicit outputs: object segmentation emerges in the attention of self-supervised transformers (Caron et al., 2021), 3D structure can be probed out of image foundation models (El Banani et al., 2024), and the attention of a pairwise geometry model separates moving objects from the static scene with no training at all (Chen et al., 2025a). We therefore transfer automatic human-derived scale supervi- sion into its scene tokens and test whether person correspondence can be read from its pretrained representation (Alain & Bengio, 2017). P1P2P3 patches on people, frame t 0 P1 P2 P3 frame t Figure 2: Person-patch matching at layer 13. Rows are patches on three people at t; columns are patches on the same people at t ′ . Color shows pre-rotation query-key probe scores, normalized only over these candidates per head and then av- eraged, not scene-wide attention. Dashed blocks mark same-person pairs; 82% of top matches have the right identity here. Reading person correspondence from the representation. Reconstructing a body in each frame does not establish which bodies belong to the same person over time. Rather than import- ing a separate visual tracker, we ask whether the 3D backbone itself carries that correspondence. It learns geometric matching across views, yet moving people are excluded from its matching supervision (Wang et al., 2026). We probe the unmodified backbone before training an identity readout and find a correspondence signal on peo- ple in its intermediate query-key features. Fig- ure 2 illustrates the finding: patches on each per- son in one frame preferentially match patches on that person later, forming same-person blocks. A retrieval probe validates the layer choice: in- termediate query-key features retrieve the per- son even after motion, while early and late fea- tures are nearer chance. This identifies a use- ful layer for reading identity. A small projec- tion extracts person features from the interme- diate representation; metric pelvis motion and proposal confidence then help resolve ambigu- ous links through dustbin-aware analytic assign- ment. Thus the same 3D representation that sup- ports people and scene reconstruction also sup- plies the correspondence cue for persistent tracks, without a separately trained visual tracker. 2 1 Label curated web video with the ruler no ground truth read · every model frozen t 1 2 n unlabeled web video 100,000 clips, ~300 h 3D Foundation Model ❄ ViTPose++ 2D keypoints ❄ 4DHumans metric SMPL ❄ z̃ k : depth at the hip f: focal length e 2 : torso length in pixels e 3 : torso length in metres Body Ruler closed form, per frame z k = f · e 3 / e 2 σ k = z k / z̃ k metric depth over backbone depth Agreement gate, per window reading: torso ≥ 8 px window: ≥ 8 readings agree, log-std(σ k ) ≤ 0.25 frames disagree: drop frames agree: keep σ = median k σ k 2 Pretrain the Scale Readout BEDLAM2-initialized · pseudo-labels only · backbone frozen pseudo-label σ kept windows 3D Foundation Model ❄ backbone tokens Scale Readout init: BEDLAM2 × 4 K,V Q scale query predicted log σ Huber loss in log σ against the ruler's pseudo-label gradient the Scale Readout alone is trained here 3 Exact metric fine-tuning RICH and 3DPW training splits pretrained readout RICH + 3DPW train splits GT pelvis depth 3D Foundation Model backbone LoRA Scale Readout loss against exact metric σ ground truth is used at this stage only gradient readout and backbone LoRA, jointly 4 Deploy no ruler, no ViTPose++, no 4DHumans 1 n new video 3D Foundation Model + LoRA up to scale Scale Readout Scale σ × metric geometry σ comes from the scene: nobody needs to be visible Figure 3: Training and deploying the Scale Readout. (1) A body ruler combines observed 2D torso extent e 2 , the posed body’s metric in-plane extent e 3 , backbone depth ̃z k and focal length f to estimate σ k = (fe 3 /e 2 )/ ̃z k ; an agreement gate yields window pseudo-labels. (2) These labels pretrain the readout on web clips. (3) Exact RICH and 3DPW targets fine-tune the readout and back- bone LoRA. (4) Deployment predicts σ from backbone tokens alone, without the ruler or teachers. Teaching metric scale to scene tokens. Exact metric labels for ordinary real video are scarce. We instead use people as offline rulers: a posed body supplies a metric torso extent in the image plane (Loper et al., 2015; Pavlakos et al., 2019), which we compare with 2D keypoints to obtain an approximate, closed-form scale pseudo-label (Fig. 3). Repeated readings expose inconsistent labels, enabling pretraining on curated unlabeled web video before exact metric fine-tuning on standard real-video training splits. The resulting Scale Readout predicts from scene-level foundation-model tokens rather than repeating the body measurement at inference. This transfers human-derived su- pervision into a scale estimate that is independent of a visible person at deployment. WildHSR is the system these two mechanisms make possible: metric cameras, scene and people from monocular video, with no external scale network or separately trained external tracker (Fig. 4). Each window is reconstructed by a feed-forward network. Fixed analytic association and Sim(3) composition connect the window predictions. Our contributions are: • WildHSR, a system that jointly reconstructs people, cameras and scene geometry in one metric world from monocular video while maintaining person tracks. • Emergent person correspondence in a 3D foundation model: a controlled probe locates motion-robust identity cues in intermediate query-key features; a lightweight projection reads them for association without an external tracker (§3.2). • Human-derived scale supervision for scene tokens: closed-form pseudo-labels from peo- ple in unlabeled video initialise a Scale Readout before exact metric adaptation (§4.2). • State-of-the-art global motion: on EMDB-2, the first feed-forward method in our com- parison to beat the best optimization method on WA-MPJPE and RTE; on RICH, the best feed-forward people-and-scene method on WA-MPJPE and W-MPJPE. 2RELATED WORK 3D foundation models, and what they encode. Feed-forward geometry models recover cameras and dense structure from unposed images in one pass (Wang et al., 2024a; 2025a), online with a persistent state (Wang et al., 2025b) and in the presence of motion (Zhang et al., 2025). Those that output metres learn them from metric supervision (Wang et al., 2025b; Keetha et al., 2026; Ma et al., 2026); some of the strongest reported camera estimates come from a model that is up to scale and re-normalised per inference (Wang et al., 2025a; 2026). We build on that representation and learn its metric conversion. What such models encode beyond their outputs is less studied. Image foundation models carry 3D structure that probes can read (El Banani et al., 2024), segmentation emerges in self-supervised attention (Caron et al., 2021), DUSt3R’s attention separates moving from static content without training (Chen et al., 2025a), and the authors of VGGT-Ω report that clustering its intermediate tokens isolates a moving dancer, and that auxiliary quantities, metric scale among them, can be decoded from its register tokens in preliminary experiments (Wang et al., 2026). We test two readouts: metric scale from in-the-wild pseudo-labels followed by exact adaptation, and person identity from cross-frame correspondence in intermediate features. 3 Supervision for metric scale. Systems that reconstruct people and scenes in metres obtain metric scale from a metric-native backbone (Chen et al., 2025b), from metric depth priors (Wang et al., 2024b), from synthetic metric supervision with an expert depth teacher (Li et al., 2026), or by train- ing the body’s scale prior into point-map prediction (Shi et al., 2026). Anthropometric size is an older cue. People as Scene Probes (Wang et al., 2020) reads depth, occlusion and lighting from passing pedestrians, SLAHMR and PACE use body priors inside their objectives (Ye et al., 2023; Kocabas et al., 2024), HAMSt3R (Rojas et al., 2025) distils a mesh recovery encoder into a stereo network, HSfM (Müller et al., 2025) recovers approximate metric scale by per-scene optimization and, closest in mechanism, HAC (Yang et al., 2025) calibrates a SLAM reconstruction against the metric depth of contact joints from a mesh recovery model. In these uses, the body measurement remains part of scale recovery at inference. We instead use the reading offline to pretrain on cu- rated unlabeled video, in the tradition of self-training (Lee, 2013; Xie et al., 2020) and consistency supervision (Godard et al., 2019), then refine the predictor with exact metric labels from real-video training splits. UniSH also trains on unlabeled in-the-wild video, but uses an external metric teacher; our pseudo-label is produced by the geometric body ruler. Humans and scenes from video. Global human mesh recovery places SMPL (Loper et al., 2015) bodies in world coordinates, by optimization over SLAM and motion priors (Ye et al., 2023; Ko- cabas et al., 2024) or by regressing world trajectories (Yuan et al., 2022; Shin et al., 2024; Wang et al., 2024b; Shen et al., 2024; Wang et al., 2025c). Closest to us, Human3R (Chen et al., 2025b), UniSH (Li et al., 2026), SHOW (Shi et al., 2026) and GUSH3R (Abe et al., 2026) attach human de- coders to geometry foundation models and reconstruct people and scene in one feed-forward pass, the last as Gaussians; MetricHMSR (Song et al., 2025) does so metrically from one image, and JOSH3R (Liu et al., 2026) is trained from the pseudo-labels of a per-sequence optimization. These systems differ in where the human branch reads the backbone. Human3R decodes SMPL-X param- eters at a detected head cell, which becomes ambiguous when two people share that token. We read each person at the pelvis, a geometric anchor for body placement (§3.2). 3METHOD 3.1PROBLEM FORMULATION AND SYSTEM OVERVIEW A window of video passes once through a 3D foundation model (VGGT-Ω (Wang et al., 2026), the successor of VGGT (Wang et al., 2025a)), adapted with a small LoRA for metric transfer, which predicts per-frame cameras with focal length f and depth ̃z, from which point maps are obtained by unprojection, together with the tokens used to decode these outputs (Fig. 4, top). All of it is correct up to one unknown scale. We write σ = z metric /z backbone for the metres-per-unit conversion of that inference. The backbone re-normalises every pass, so σ must be predicted per window rather than pooled over a sequence. Cameras and scene are in backbone units, whereas SMPL-X dimensions are metric; σ puts both in one world. The body prior supplies shape, not placement; VGGT supplies cameras and depth, not identity. Cross-attention grounds proposals; scale and association connect windows. Two pretrained networks supply scene and human tokens, while our Scale Readout, Pelvis Readout, cross-attention fusion and identity projection expose the quantities needed for joint reconstruction. Their released weights remain frozen; small LoRA adapters support metric and body adaptation. Appendix B gives the exact modules, parameter counts and training configuration. We next describe how the system reads and associates people, learns scale, and composes one metric world. 4 Monocular video t 1 2 n 3D Foundation Model LoRA HMR Encoder LoRA scene tokens HMR tokens Camera head Camera (up to scale) Dense (DPT) head Depth maps Point maps unproject up to scale Cross-attention × L K,V Pelvis Readout 2D pelvis + query from proposal tokens Q pelvis query what the pelvis query attends to person token HMR Decoder SMPL-X body Scale Readout Scale σ × Sim(3) composition windows into one world Metric scene + people Camera (metric) Figure 4: Overview of WildHSR. A 3D foundation model produces up-to-scale cameras, geometry and scene tokens. A mesh branch proposes people; a Pelvis Readout forms a body-localized query, and cross-attention decodes a metric SMPL-X body. The Scale Readout supplies σ; analytic associ- ation and Sim(3) composition link people and windows in one metric world. 3.2PERSON RECONSTRUCTION AND TEMPORAL ASSOCIATION Per-frame proposals and bodies. Multi-HMR (Baradel et al., 2024), adapted with a 0.75M LoRA, scores person-centre patches and proposes initial SMPL-X bodies per frame; VGGT-Ω indepen- dently supplies scene tokens for the video window. For proposal n at time t, a lightweight Pelvis Readout takes its HMR tokens and produces a 2D pelvis location and person query. During training, the location is supervised by the pelvis joint of the ground-truth SMPL-X body projected into the image; the target does not come from the model’s own final prediction. We then fuse the readout query with the two token streams: (u t,n ,q pel t,n ) = R pel (H t,n ), h t,n = CrossAttn q pel t,n , [S 1:T ;H t ] , B t,n = D HMR (h t,n ). (1) Here H t,n denotes the HMR tokens of proposal n, u t,n is the readout’s image-space pelvis location, q pel t,n its query, S 1:T the VGGT scene tokens, and H t the frame’s HMR tokens; the latter two supply keys and values. At inference, the Pelvis Readout needs no ground-truth body or external keypoint model. No external detector runs in the deployed path. The pelvis anchor. Placement unprojects the Pelvis Readout’s pixel u t,n using the median backbone depth ̃z t,n over the body’s projected torso. With backbone-unit camera centre C t , camera-to-window rotation R t , intrinsics K t and ̄u t,n = [u T t,n , 1] T , the metric pelvis in window w is p (w) t,n = σ C t + R t ( ̃z t,n K −1 t ̄u t,n ) .(2) The readout is supervised at SMPL’s hip midpoint, so p (w) t,n places the body’s root. The torso me- dian avoids a body–ground depth discontinuity at the anchor. Direct metric-translation regression performs worse; a head anchor introduces an orientation-sensitive lever arm of roughly 0.6 m. Identity readout and association. The probe of Fig. 5 peaks in the layer-13 query-key space. We pool those vectors around each pelvis and map them to a 128-dimensional unit descriptor with a two-layer projection trained by supervised contrastive loss. For track i and proposal j, let m i ,e j be unit descriptors, ˆp i ,p j their predicted and observed metric pelvis positions, c j proposal confidence and g i a time-dependent motion gate. Their assignment cost is C ij = 1 2 (1−⟨m i ,e j ⟩) + min ∥p j − ˆp i ∥ 2 g i , 1 + 0.25(1− c j ).(3) Pairs beyond g i are invalid; Appendix I specifies the gate and track state. A dustbin and Sinkhorn optimal transport (Cuturi, 2013) give a soft assignment, which is hardened one-to-one with Hungar- ian matching. The projection also receives a ground-truth assignment loss through the soft Sinkhorn matrix; Hungarian selection and confidence-gated memory updates remain outside backpropaga- tion. Appendix I gives the full specification. Learned inference is feed-forward within each window; fixed-step association and memory updates require no test-time gradient-based fitting. 5 1 2 5 10 attention / uniform (log) (a) within a frame film clip, left dancerfilm clip, right dancermoving camera (b) across frames, one query t-20t (query)t+20 Figure 5: Pretrained person correspondence. At layer 13, a VGGT-Ω pelvis token attends to its own person within a frame (3.8× uniform over four clips) and across the window (5.8×). When the person moves, attention follows the person (5.0×), not the vacated location (1.2×). This probe runs the backbone alone; Appendix I gives controls and the complete sample accounting. 3.3SCALE READOUT AND METRIC ADAPTATION Ruler pseudo-labels. For each visible person, ViTPose++-H (Xu et al., 2024) at 256× 192 gives the observed image-plane extent e 2 from shoulder midpoint to hip midpoint. 4DHumans (Goel et al., 2023) supplies the posed metric torso; e 3 is the in-plane extent between its corresponding midpoints. With the backbone’s focal length f and estimated depth ̃z k , a local weak-perspective approximation gives the ruler reading (Fig. 3, stage 1) z k = f e 3 e 2 , σ k = z k ̃z k ,(4) without a metric label or sensor for the web clip. The estimate can be biased when the torso end- points have different depths. We retain confident detections with e 2 ≥ 8 px; a window needs at least eight readings and std(logσ k ) ≤ 0.25. Its pseudo-label is the median across eligible frames and people; agreement tests consistency, not absolute correctness. Pretraining and adaptation. These labels pretrain the Scale Readout on 100,000 curated web- video clips (about 300 hours). The clips include publicly accessible video such as YouTube; YOLO person counts guide sampling across crowd sizes. A learned query reads the backbone’s camera and register tokens to regress logσ with a log-space Huber loss while the backbone remains fixed (architecture in Appendix B). After synthetic initialization and ruler pretraining, we fine-tune the readout and backbone LoRA with exact metric targets from standard RICH (Huang et al., 2022) and 3DPW (von Marcard et al., 2018) training splits (Chen et al., 2025b). Camera calibration and body translation provide reference pelvis depth; no scene point cloud is needed. At inference, the readout needs only backbone tokens, with no ruler, teacher or visible person. 3.4COMPOSING ONE METRIC WORLD Given σ, the backbone’s camera centres, depth and point maps are multiplied by it, while the metric body is not, and one forward pass per window yields the metric scene, the metric camera path and the people placed in it (Fig. 4). Long video needs one continuous world frame, but every window is an independent inference with its own frame and unit. We use 100-frame windows with stride 50 and require at least eight shared frames. For consecutive windows a and b, the relative rotation R ba is the chordal mean of the framewise rotations between their shared camera orientations. With R ba fixed, we solve (s ba ,t ba ) = arg min s,t X k∈a∩b C a k − (sR ba C b k + t) 2 2 ,(5) in closed form on their metric camera centres. If fewer than eight correspondences survive or the centred camera trajectory is degenerate, we retain the previous cumulative similarity. All shared frames have equal weight and no outlier trimming is applied. We compose valid relative similarities in temporal order. If S w is the cumulative scale of window w, we divide every cumulative similarity byg = exp |W| −1 P w logS w , preserving the sequence’s aggregate metric scale rather than letting one window set it. The similarity acts fully on camera centres, point maps and pelvis translations; its rotation also acts on camera and SMPL-X global orientations, while body dimensions remain unscaled and no network or scene parameters are optimized at test time. 6 Input frameGround truthHuman3RUniSHJOSH3RWildHSR Long walk EMDB 24 Running in circles EMDB 37 Stairs up EMDB 29 Figure 6: Scene and people in one metric world (three EMDB-2 clips), each method at its own metric scale; labels give placement error and, for UniSH and JOSH3R, view turn. WildHSR is 0.2 to 0.3 m off, against 0.3 to 1.6 m (Human3R), 0.9 to 3.9 m (UniSH) and 1.3 to 5.8 m (JOSH3R). 3.5TRAINING OBJECTIVES The scale pathway uses the log-space Huber objective above; the Pelvis Readout’s location is super- vised by projected ground-truth SMPL-X pelvis joints, while its query and the fusion receive gra- dients through the decoded body. No target is imposed on the cross-attention weights themselves. Body losses cover SMPL-X parameters, mesh and reprojection; identity losses use contrastive and soft assignment, while scene contact regularizes placement. Feet on the reconstructed ground. A scene-contact consistency term encourages supporting feet to agree with the local reconstructed surface during training. The body branch predicts its placement directly at inference, without a manual vertical shift (Appendix G). 4EXPERIMENTS Protocol.We follow the published EMDB-2 (Kaufmann et al., 2023) 25-sequence and RICH (Huang et al., 2022) protocols used by the compared methods (Chen et al., 2025b; Li et al., 2026; Shi et al., 2026; Liu et al., 2026; Ying et al., 2025; Sun et al., 2023; Li et al., 2024). WA- MPJPE uses similarity alignment, W-MPJPE aligns the first two frames, and RTE aligns trajectories by rotation and translation only; RTE therefore preserves scale error. 4.1METRIC CAMERAS, SCENE AND PEOPLE EMDB-2. Table 1 compares WildHSR with previously reported EMDB-2 results. Trajectory error is the best published: RTE 0.9 improves on the prior best (1.3) by 31%. On the joint metrics only JOSH (Liu et al., 2026), which optimizes each sequence jointly over scene and body, is ahead on W-MPJPE (174.7 against 193.6). WildHSR has the lowest WA-MPJPE in the full table (66.3). Among feed-forward methods WildHSR leads all three metrics, by 38%, 26% and 47% over the best of the others on each. What differs is what each system spends to become metric: metric depth priors, large-scale supervised pretraining or a metric-native backbone, where WildHSR learns scale from the people. Fig. 6 shows the reconstructions behind these numbers, and Appendix F the paths themselves (Fig. 8) and six more clips. The visual comparison is restricted to systems whose neural reconstruction networks jointly predict people and scene: Human3R, UniSH and JOSH3R (Liu et al., 2026); SHOW (Shi et al., 2026) has not released weights and appears in the tables only (alignment and re-run details in Appendix F). Because RTE preserves scale error, this comparison tests the shared metric scene and pelvis-based body placement, not only local pose quality. 7 Table 1: EMDB-2 global motion. WA/W: global joint error; RTE: trajectory error.best2nd3rd applies throughout. methodWA↓W↓RTE↓ Optimization-based SLAHMR326.9776.110.2 COIN152.8407.33.5 TRAM76.4222.41.4 PromptHMR-vid 71.0216.51.3 JOSH68.9174.71.3 Feed-forward WHAM135.6354.86.0 GVHMR111.0276.52.0 WATCH106.4269.31.7 JOSH3R220.0661.713.1 Human3R112.2267.92.2 UniSH118.5270.15.8 SHOW109.1262.32.1 WildHSR 66.3193.60.9 ∆ vs best-3.8%+10.8%-30.8% Table 2: RICH global motion. WA/W: global joint error; RTE: per-segment tra- jectory error. methodWA↓W↓RTE↓ Optimization-based TRAM127.8238.06.0 JOSH89.0132.53.0 Feed-forward, people and scene Human3R 110.0184.93.3 UniSH118.1183.24.8 SHOW 107.3172.72.2 WildHSR73.2163.12.7 ∆ vs best-17.8%+23.1%+22.7% Table 3: Training-data ablation on EMDB-2. End-to-end joint-position (WA, W) and trajectory (RTE) errors for checkpoints trained with the indicated data sources. training dataEMDB-2 checkpointBEDLAM2curated ITWRICH+3DPWWA↓W↓RTE↓ synthetic base✓182.21009.211.3 ITW pretrained✓136.7418.64.63 no ITW pretraining✓116.3255.93.7 metric fine-tuned✓66.3193.60.90 Table 4: Held-out 3DPW body reconstruc- tion. 14-joint test errors in m; published base- lines from SHOW (Shi et al., 2026). methodPA-MPJPE↓ MPJPE↓ PVE↓ Human3R44.171.284.9 UniSH48.875.688.8 SHOW41.067.778.7 WildHSR38.765.274.4 Local body accuracy.Table 4 evaluates held- out 3DPW. PA-MPJPE removes per-pose similarity; MPJPE and PVE measure camera-frame joints and mesh. WildHSR improves on SHOW, the strongest baseline, by 2.3, 2.5 and 4.3 m, respectively. These metrics do not assess world placement. Together with Tables 1 and 2, they show that WildHSR gains global accuracy without sacrificing local body fidelity. RICH. WildHSR has the best WA-MPJPE of Table 2 and leads the feed-forward people-and-scene methods on W-MPJPE, where only JOSH, an optimization, is ahead. Its median multiplicative scale error is 7.2%. Its per-segment RTE is second among feed-forward people-and-scene systems (2.7), behind SHOW at 2.2; subjects are near-stationary and RTE divides by a median displacement of 0.52 m. Our model is fine-tuned with exact labels from RICH-train and 3DPW-train; the RICH test partition is held out. Table 2 uses the same test split and evaluation protocol as the baselines, not necessarily the same training data. With camera motion largely removed, the joint results are consistent with the Scale Readout and pelvis-to-scene fusion placing bodies in a shared metric frame. Pretrained person correspondence. Figures 2 and 5 show intermediate features matching a moving person across frames. Tested on 1,500 clips, same-person retrieval reaches 80 to 82% in layers 11 to 15 against 46% chance. This retrieval accuracy is distinct from the conditional match weights in Fig. 2 and the attention lift in Fig. 5. The identity projection reads these mid-depth features and combines them with metric pelvis motion for cross-frame assignment (Appendix I). Long video and runtime. Over 1000 frames (33 s), WildHSR’s camera-to-person distance error is 5.4 to 6.0% (9.4 to 12.9% for Human3R); without alignment, its median range error is 14.1 cm and its proxemic zone is correct on 100% of frames. The complete pipeline runs at 10.1 fps on one RTX PRO 6000 Blackwell. Fixed association and Sim(3) composition link feed-forward windows without per-video fitting; Appendix B gives the matched runtime breakdown. 8 Table 5: Association on 13 two-person 3DPW test sequences. WildHSR variants share fixed Multi-HMR outputs; Human3R uses its native pipeline. Assignment thresholds were tuned on four validation sequences. methodIDF1↑HOTA↑ID switches↓fragments↓MOTA↑ Human3R (native)87.472.316828787.1 motion + confidence DP66.956.9977787.5 raw VGGT query/key51.853.4806586151.7 projected identity60.254.566428473.7 projection + motion + confidence 82.174.5397396.5 Temporal association. Table 5 compares association cues on 13 two-person 3DPW test sequences with fixed Multi-HMR proposals and bodies. The combined cost reaches 74.5 HOTA and 39 ID switches, versus 72.3 and 1682 for native Human3R. Human3R leads IDF1 (87.4 versus 82.1) and nine sequences; one crowded clip dominates its switch count. The weaker single-cue variants sup- port combining projected identity with motion, but do not isolate confidence. This controlled com- ponent test does not measure the final end-to-end pipeline (Appendix A). 4.2EFFECT OF THE SCALE-TRAINING STAGES Table 3 tests web-video pseudo-labeling within the final recipe. Removing it while keeping BED- LAM2 initialization and exact RICH/3DPW fine-tuning worsens WA/W/RTE from 66.3/193.6/0.90 to 116.3/255.9/3.7. The pseudo-label stage contributes beyond exact labels. 5CONCLUSION WildHSR transfers human-derived scale pseudo-labels into an up-to-scale 3D foundation model and reads person correspondence from its pretrained intermediate query-key features. The Scale Readout is initialized on real-video pseudo-labels before exact metric adaptation; a small identity projection combines the correspondence signal with metric motion for association. Together they enable joint metric reconstruction of people, cameras and scene in feed-forward windows, linked by analytic association and Sim(3) composition without test-time optimization. Limitations. Identity weakens under interaction and occlusion, and its specificity to people is untested. Sequence composition is offline; RICH training mixtures may differ across methods. REPRODUCIBILITY STATEMENT VGGT-Ω and Multi-HMR retain their released base weights; small LoRAs adapt them (Hu et al., 2022). The Scale Readout starts on BEDLAM2, learns from offline ruler pseudo-labels, then re- ceives exact metric supervision from RICH- and 3DPW-train; EMDB-2 is held out. The ruler, agreement gate, ViTPose++-H and 4DHumans are absent at inference. The appendix details objec- tives, splits, settings and evaluation; code, models and outputs will be released. ETHICS AND WEB-DATA STATEMENT Public web video supplies 100,000 clips for non-identifying scale pseudo-labels; we do not redis- tribute it. Persistent reconstruction may enable surveillance. Track IDs do not identify people, and the system should not be used for biometric or high-stakes decisions. Web video and parametric body priors may introduce demographic, body-shape, clothing, mobility and visibility biases. AI USE STATEMENT Large language models assisted manuscript drafting, restructuring and editing. The authors directed and revised the generated text and take full responsibility for this paper. 9 REFERENCES Keito Abe, Kaede Shiohara, Takashi Otonari, and Toshihiko Yamasaki. Gush3r: Everyone every- where all at once as gaussians. arXiv preprint arXiv:2607.05243, 2026. Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. In International Conference on Learning Representations (ICLR), Workshop Track, 2017. Fabien Baradel, Matthieu Armando, Salma Galaaoui, Romain Brégier, Philippe Weinzaepfel, Gré- gory Rogez, and Thomas Lucas. Multi-hmr: Multi-person whole-body human mesh recovery in a single shot. In ECCV, 2024. Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In IEEE/CVF Inter- national Conference on Computer Vision (ICCV), 2021. Xingyu Chen, Yue Chen, Yuliang Xiu, Andreas Geiger, and Anpei Chen. Easi3R: Estimating dis- entangled motion from DUSt3R without training. In IEEE/CVF International Conference on Computer Vision (ICCV), 2025a. Yue Chen, Xingyu Chen, Yuxuan Xue, Anpei Chen, Yuliang Xiu, and Gerard Pons-Moll. Human3r: Everyone everywhere all at once. arXiv preprint arXiv:2510.06219, 2025b. Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems, 2013. David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In Advances in Neural Information Processing Systems (NeurIPS), 2014. Mohamed El Banani, Amit Raj, Kevis-Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Ru- binstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3D awareness of visual foundation models. In IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), 2024. Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J. Brostow. Digging into self- supervised monocular depth estimation. In IEEE/CVF International Conference on Computer Vision (ICCV), 2019. Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa, and Jitendra Malik. Humans in 4D: Reconstructing and tracking humans with transformers. In IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), 2023. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations (ICLR), 2022. Chun-Hao P. Huang, Hongwei Yi, Markus Höschle, Matvey Safroshkin, Tsvetelina Alexiadis, Senya Polikovsky, Daniel Scharstein, and Michael J. Black. Capturing and inferring dense full-body human-scene contact. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. Manuel Kaufmann, Jie Song, Chen Guo, Kaiyue Shen, Tianjian Jiang, Chengcheng Tang, Juan Zarate, and Otmar Hilliges. EMDB: The electromagnetic database of global 3d human pose and shape in the wild. In ICCV, 2023. Nikhil Keetha, Norman Müller, Johannes Schönberger, Lorenzo Porzi, Yuchen Zhang, Tobias Fis- cher, Arno Knapitsch, Duncan Zauss, Ethan Weber, Nelson Antunes, Jonathon Luiten, Manuel Lopez-Antequera, Samuel Rota Bulò, Christian Richardt, Deva Ramanan, Sebastian Scherer, and Peter Kontschieder. Mapanything: Universal feed-forward metric 3D reconstruction. In Interna- tional Conference on 3D Vision (3DV), 2026. Muhammed Kocabas, Ye Yuan, Pavlo Molchanov, Yunrong Guo, Michael J. Black, Otmar Hilliges, Jan Kautz, and Umar Iqbal. PACE: Human and camera motion estimation from in-the-wild videos. In 3DV, 2024. 10 Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICML Workshop on Challenges in Representation Learning, 2013. Jiefeng Li, Ye Yuan, Davis Rempe, Haotian Zhang, Pavlo Molchanov, Cewu Lu, Jan Kautz, and Umar Iqbal. COIN: Control-inpainting diffusion prior for human and camera motion estimation. In European Conference on Computer Vision (ECCV), 2024. Mengfei Li, Peng Li, Zheng Zhang, Jiahao Lu, Chengfeng Zhao, Wei Xue, Qifeng Liu, Sida Peng, Wenxiao Zhang, Wenhan Luo, Yuan Liu, and Yike Guo. UniSH: Unifying scene and human reconstruction in a feed-forward pass, 2026. Zhizheng Liu, Joe Lin, Wayne Wu, and Bolei Zhou. Joint optimization for 4D human-scene recon- struction in the wild. In International Conference on Learning Representations (ICLR), 2026. Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. In ACM TOG (SIGGRAPH Asia), 2015. Baorui Ma, Jiahui Yang, Donglin Di, Xuancheng Zhang, Jianxun Cui, Hao Li, Yan Xie, and Wei Chen. Metricanything: Scaling metric depth pretraining with noisy heterogeneous sources. In European Conference on Computer Vision (ECCV), 2026. Lea Müller, Hongsuk Choi, Anthony Zhang, Brent Yi, Jitendra Malik, and Angjoo Kanazawa. Re- constructing people, places, and cameras. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dim- itrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from a single image. In CVPR, 2019. Sara Rojas, Matthieu Armando, Bernard Ghanem, Philippe Weinzaepfel, Vincent Leroy, and Gre- gory Rogez. HAMSt3R: Human-aware multi-view stereo 3D reconstruction. In IEEE/CVF Inter- national Conference on Computer Vision (ICCV), 2025. Zehong Shen, Huaijin Pi, Yan Xia, Zhi Cen, Sida Peng, Zechen Hu, Hujun Bao, Ruizhen Hu, and Xiaowei Zhou. World-grounded human motion recovery via gravity-view coordinates. In SIGGRAPH Asia, 2024. Boao Shi, Qiao Feng, Yiming Huang, and Lingjie Liu. Scene and human in one world: Reconstruc- tion in a feedforward pass, 2026. Soyong Shin, Juyong Kim, Eni Halilaj, and Michael J. Black. WHAM: Reconstructing world- grounded humans with accurate 3d motion. In CVPR, 2024. Chentao Song, He Zhang, Haolei Yuan, Haozhe Lin, Jianhua Tao, Hongwen Zhang, and Tao Yu. Metrichmsr: Metric human mesh and scene recovery from monocular images. arXiv preprint arXiv:2506.09919, 2025. Yu Sun, Qian Bao, Wu Liu, Tao Mei, and Michael J. Black. TRACE: 5d temporal regression of avatars with dynamic cameras in 3d environments. In CVPR, 2023. Timo von Marcard, Roberto Henschel, Michael J. Black, Bodo Rosenhahn, and Gerard Pons-Moll. Recovering accurate 3d human pose in the wild using IMUs and a moving camera. In European Conference on Computer Vision (ECCV), p. 601–617, 2018. Hengyi Wang and Lourdes Agapito. AMB3R: Accurate feed-forward metric-scale 3d reconstruction with backend. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026. Jianyuan Wang, Minghao Chen, Nikita Karaev, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In CVPR, 2025a. Jianyuan Wang, Minghao Chen, Shangzhan Zhang, Nikita Karaev, Johannes Schönberger, Patrick Labatut, Piotr Bojanowski, David Novotny, Andrea Vedaldi, and Christian Rupprecht. VGGT-ω. arXiv preprint arXiv:2605.15195, 2026. 11 Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A. Efros, and Angjoo Kanazawa. Con- tinuous 3D perception model with persistent state. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025b. Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jérôme Revaud. DUSt3R: Geometric 3D vision made easy. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024a. Yifan Wang, Brian Curless, and Steve Seitz. People as scene probes. In European Conference on Computer Vision (ECCV), 2020. Yufu Wang, Ziyun Wang, Zhicheng Liu, and Kostas Daniilidis. TRAM: Global trajectory and motion of 3d humans from in-the-wild videos. In ECCV, 2024b. Yufu Wang, Yu Sun, Priyanka Patel, Kostas Daniilidis, Michael J. Black, and Muhammed Kocabas. PromptHMR: Promptable human mesh recovery. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025c. arXiv:2504.06397; video variant numbers as reported in Ying et al. (2025). Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V. Le. Self-training with noisy student improves ImageNet classification. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. ViTPose++: Vision transformer for generic body pose estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(2): 1212–1230, 2024. Fengyuan Yang, Kerui Gu, Ha Linh Nguyen, Tze Ho Elden Tse, and Angela Yao. Humans as checkerboards: Calibrating camera motion scale for world-coordinate human mesh recovery. In IEEE/CVF International Conference on Computer Vision (ICCV), 2025. Vickie Ye, Georgios Pavlakos, Jitendra Malik, and Angjoo Kanazawa. Decoupling human and camera motion from videos in the wild. In CVPR, 2023. Qijun Ying, Zhongyuan Hu, Rui Zhang, Ronghui Li, Yu Lu, and Zijiao Zeng. WATCH: World-aware allied trajectory and pose reconstruction for camera and human, 2025. Ye Yuan, Umar Iqbal, Pavlo Molchanov, Kris Kitani, and Jan Kautz. GLAMR: Global occlusion- aware human mesh recovery with dynamic cameras. In CVPR, 2022. Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, De- qing Sun, and Ming-Hsuan Yang. MonST3R: A simple approach for estimating geometry in the presence of motion. In International Conference on Learning Representations (ICLR), 2025. 12 Supplementary Material WildHSR: Metric Feed-Forward 4D People-Scene Reconstruction from a 3D Foundation Model Appendices A–E cover the association protocol, runtime, RICH evaluation, scale calibration, and scene reconstruction. Appendix F presents qualitative results. Appendices G–K cover ablations, people-scene consistency, identity and pelvis probes, robustness, and failure modes. ATEMPORAL ASSOCIATION PROTOCOL Table 5 reports a controlled component experiment, not the final end-to-end pipeline. WildHSR variants use fixed Multi-HMR proposals and frozen bodies, camera-frame pelvis positions, non- overlapping windows, and an identity projection trained on 3DPW only. Assignment thresholds were tuned on four validation sequences. Human3R retains its native pipeline; the comparison therefore tests tracking behavior, not identical upstream reconstructions. BCOMPUTATIONAL COST Training configuration. The released VGGT-Ω and Multi-HMR base weights remain frozen. The backbone LoRA uses rank 16, α = 32 and dropout 0.05 on the linear layers of the last four frame- attention and last four global-attention blocks (2.10M parameters); the mesh adapter has 0.75M parameters. Every Scale Readout in Table 3 uses a 512-dimensional query, four decoder layers, eight attention heads, batch size 16 and 6,000 pretraining steps. It is initialized on BEDLAM2, pretrained from ruler labels with the backbone frozen, then jointly fine-tuned with the backbone LoRA on the standard RICH and 3DPW training splits using AdamW at 10 −4 . Standard valida- tion partitions are used for model selection. The Pelvis Readout uses each proposal’s HMR tokens to produce the image-space hip-midpoint location and query. The location is trained against the projected ground-truth SMPL-X pelvis; decoded-body losses train the query and fusion, without an attention-map target. The identity readout pools layer-13 query and key vectors around each Pelvis Readout location, maps them to a 128-dimensional unit vector with a two-layer MLP, and is trained with supervised contrastive and dustbin-aware soft-assignment losses on the BEDLAM2, RICH- train and 3DPW-train identities while both base networks remain frozen. Training windows contain 17 frames for BEDLAM2 and 32 frames for real video. These are training clip lengths, not the infer- ence window: the reported evaluations use 100-frame windows with stride 50. No window-length ablation is reported. Table 6: Compute on one GPU, same 60-frame clip. End-to-end throughput and peak memory; FLOPs count traced model stages. methodparams run↓TFLOPs/frame↓fps↑s / 60 frames↓peak GPU↓ Human3R1.17Bn/a8.86.85.95 GB UniSH1.86B 7.294.912.27.06 GB JOSH3R 1.50B3.776.19.87.55 GB WildHSR (512× 288)1.52B6.9412.64.88.85 GB WildHSR (688× 384, ours) 1.52B11.4310.16.011.22 GB Table 6 reports throughput, peak GPU memory and traced model FLOPs on the first 60 frames of EMDB-2 24_outdoor_long_walk. We use one RTX PRO 6000 Blackwell per method, two warmup passes and three timed passes, excluding model loading. Each system uses its evaluated resolution. FPS measures the full pipeline; FLOPs cover traced neural stages only. The parameter count includes every loaded model in the inference path. At its main 688× 384 resolution, WildHSR processes 10.1 frames per second, the highest end-to- end throughput among the configurations in Table 6. Its body encoder accounts for 62% of runtime, making that stage the clearest target for further speedups. 13 CRICH EVALUATION PROTOCOL Identifying the target subject. RICH annotates one subject while up to 7 people are visible, so every method must decide which reconstruction to score; 18 of our 46 segments contain more than one person. Motion and centre confidence alone do not reliably separate a smoothly walking, confi- dently detected bystander from the annotated subject. Our association instead matches the projected VGGT identity descriptor against the track memory and uses metric pelvis motion only as a comple- mentary cue. A dustbin permits the annotated person to be temporarily missing rather than forcing a match to a bystander. No image-centre prior or capture-rig-specific rule enters the method. Scale-error metric. Throughout the paper, “median multiplicative scale error (%)” denotes E σ = 100 " exp median i log σ pred i σ gt i ! − 1 # .(6) This is symmetric in log scale before conversion to a percentage: predicting either twice or half the ground-truth scale gives E σ = 100%. We use the published RICH test split and evaluator, though training may differ. Median segment displacement is 0.52 m, so RTE magnifies small trajectory errors. DHELD-OUT RULER CALIBRATION ON RICH Table 7:Ruler calibration on held-out RICH. Median multiplicative scale error (%). scale sourceerror↓ body-ruler pseudo-label14.1 Scale Readout, ruler-pretrained8.4 Scale Readout, metric fine-tuned 7.2 Table 8: 7Scenes scene accuracy. Mean point- map accuracy in cm (↓); methodAcc. (cm)↓ Spann3R4.81 MapAnything3.48 CUT3R2.88 VGGT 2.32 AMB3R1.75 WildHSR1.66 ∆ vs best-5.1% The body ruler can inherit systematic error from the metric body teacher. Table 7 compares the scale pathway’s stages on the same held-out RICH examples using the median multiplicative scale error of Eq. (6). Error falls from 14.1% for the pseudo-label to 8.4% after ruler pretraining and 7.2% after exact metric fine-tuning. EPERSON-FREE SCENE RECONSTRUCTION Table 8 compares published 7Scenes point-map accuracy under the same scale-aligned proto- col (Wang & Agapito, 2026). WildHSR has the lowest error at 1.66 cm versus 1.75 cm for AMB3R. This tests scene geometry, not metric scale without visible people. FQUALITATIVE RESULTS Figures 7–11 show metric people, cameras and scene geometry across varied motion, longer videos and crowds. Rigid camera-path alignment preserves predicted scale in the people-scene views; similarity alignment isolates trajectory shape and drift. 14 Input frameGround truthHuman3RUniSHJOSH3RWildHSR Rectangle walk EMDB 79 Outdoor walk EMDB 09 Walking lunges EMDB 28 Stairs down EMDB 30 Stairs up EMDB 78 Long walk EMDB 36 Indoor circle EMDB 40 Stairs up EMDB 77 Big circle walk EMDB 80 Figure 7: Metric people-scene placement on nine EMDB-2 clips. Each method’s body history and scene retain predicted scale after rigid camera-path alignment. Labels report pelvis offset in metres; UniSH and JOSH3R also show the view rotation required for alignment. What the alignment shows. Across walks, stairs and lunges, WildHSR has the lowest pelvis offset on seven of nine clips. Human3R is closer on two stairs-up clips (0.3 versus 0.5 and 0.8 m). The body history stays positioned relative to its reconstructed ground, linking human placement to a coherent metric people-scene reconstruction throughout each motion. 15 0246 m −3 −2 −1 0 1 2 3 4 Long walk (EMDB 24) 0.59 / 1.04 / 0.10 m 0.02.55.07.510.0 m −4 −2 0 2 4 6 Rectangle walk (EMDB 79) 0.52 / 3.75 / 0.11 m 024 along walk (m) −2 −1 0 1 2 3 height (m) Stairs up (EMDB 29) 0.82 / 3.55* / 0.21 m 024 m −3 −2 −1 0 1 2 3 Running in circles (EMDB 37) 0.96 / 2.45 / 0.28 m Ground truthHuman3RUniSHWildHSR Figure 8: Global human trajectories on four EMDB-2 clips after WA-MPJPE alignment. Stairs plot height; other panels are top-down. Numbers give mean path error (Human3R / Un- iSH / WildHSR); dots mark 2 s intervals and ∗ a path leaving the plot. What the paths show. Figure 8 tests whether placement stays coherent through long walks, turns and stairs. WildHSR follows the route and ascent, while Human3R drifts on the long walk and UniSH loses the turning path. Mean path er- ror is 0.10 to 0.28 m for WildHSR, 0.52 to 0.96 m for Human3R and 1.04 to 3.75 m for UniSH. These paths are similarity-aligned; Fig. 9 sepa- rately tests metric scale. Human3R Stairs down EMDB 30 x1.14 0.24 m 2 m Indoor circle EMDB 40 x1.00 0.06 m Long walk EMDB 36 x1.21 0.18 m Long walk EMDB 24 x0.93 0.18 m Running in circles EMDB 37 x1.52 0.15 m UniSH x0.29 1.23 m x0.69 0.67 m x0.30 0.60 mx0.69 0.88 mx5.20 1.09 m WildHSR x0.99 0.02 m x1.00 0.02 m x0.97 0.03 mx0.97 0.09 mx0.95 0.03 m 0246 along walk (m) −1 0 1 fl oor height (m) 9 / 40 / 2 cm 0246 along walk (m) −0.6 0.0 0.6 17 / 13 / 5 cm 024 along walk (m) −0.8 0.0 0.8 26 / 17 / 8 cm 0246 along walk (m) −0.6 0.0 0.6 13 / 46 / 5 cm 0369 along walk (m) −0.6 0.0 0.6 21 / 15 / 7 cm maps: world size vs ground truth, camera error (fixed scale) · floor row: floor-gap under the real feet, Human3R / UniSH / WildHSR ground-truth cameraHuman3RUniSHWildHSRground-truth feet Figure 9: Metric scene, camera and ground. Top three rows: person-free scene and camera path after rigid alignment, preserving predicted scale; labels give world-size ratio and mean camera error. Bottom: reconstructed floor height along the walk against the ground-truth feet; values are median foot-floor gap for Human3R / UniSH / WildHSR. What the scene shows. WildHSR reconstructs all five worlds at 0.95 to 1.00× ground-truth size with 0.02 to 0.09 m mean camera error. Its median foot-floor gap is 2 to 8 cm, compared with 9 to 26 cm for Human3R and 4 to 46 cm for UniSH. The scene scale, camera path and ground beneath the person remain mutually consistent, giving the human trajectory a metric reference in the reconstructed environment. 16 1003001000 Number of input frames 0.0 0.1 0.2 0.3 0.4 Camera trajectory error (m) (a) Camera pose, Sim(3) fit 1003001000 Number of input frames 0 2 4 6 8 10 12 Distance error (%) (b) Camera-to-person distance, metric, no alignment 1003001000 Number of input frames 0.0 0.1 0.2 0.3 0.4 0.5 Person position error (m) (c) Person path in the world, no scale fit Human3RWildHSR Figure 10: Accuracy against video length, averaged over six EMDB-2 clips of at least 1000 frames. (a) Camera trajectory error after Sim(3) alignment. (b) Metric camera-to-person distance error with- out alignment. (c) World-frame person-path error after rigid alignment, retaining scale error. frame 490frame 600frame 720 11 people11 people10 people Input WildHSR 3DPW runForBus camera path, start → shown frame meshes and boxes coloured by track ID: Figure 11: Crowded in-the-wild reconstruction on 3DPW downtown_runForBus. Top: input and persistent tracks. Bottom: metric scene, camera trajectory and bodies colored by identity. The sequence contains 14 tracks over 10 s, with 10 to 11 people reconstructed per frame. What the crowd shows. WildHSR maintains 14 tracks over 10 s and reconstructs 10 to 11 people per frame in one scene. For the two people with 3DPW annotations, hip-depth error is 0.29 m for WildHSR and 1.20 m for Human3R, while local pose is similar (PA-MPJPE 55.3 versus 51.7 m). The larger improvement in placement shows the value of relating each person’s body to the shared scene and camera estimate. What longer videos show. From 100 to 1000 frames in Fig. 10, WildHSR accumulates less camera and human-path error than Human3R, while its metric camera-to-person distance error stays near 5 to 6%. The method maintains person placement in its reconstructed world as the observed path grows. WildHSR processes each complete prefix; Human3R runs causally. GADDITIONAL ABLATIONS Table 9: Ablations on EMDB-2. Each row sub- stitutes one component of the full system. EMDB-2 configurationWA↓W↓RTE↓ WildHSR (full)66.3193.60.90 synthetic-only scale182.21009.211.3 − scene contact72.2203.40.91 Scale pathway. Table 9 shows the largest loss when the final two-stage Scale Readout is re- placed by the best regressor trained on syn- thetic metric ground truth: 2.7× WA and 13× RTE. Because the final head combines ruler pretraining with ground-truth fine-tuning, this substitution does not isolate their separate con- tributions. 17 HPLAUSIBILITY AND METRIC SCALE ON EMDB-2 Table 10 measures people-scene consistency on all 25 EMDB-2 sequences. Foot gap compares the sole with each method’s reconstructed floor, and sliding measures toe speed during ground-truth contact. World size compares camera-trajectory scale with ground truth (ideal: one), while camera error aligns rigidly without rescaling. WildHSR reduces the foot gap to 3.6 cm versus 12.7 cm for Human3R and 15.5 cm for UniSH. Its 6% world-size error and 0.13 m camera error support placing bodies and scene in a shared metric frame; contact is a training term, not an inference correction. Table 10: People-scene consistency on EMDB-2. Five diagnostics use the first 300 frames of each sequence; WA-MPJPE uses full sequences. Shades rank methods per column, with world size ranked by distance from one. methodfoot gap (cm)↓ sliding (cm/s)↓ world size (×GT) size err.↓ camera (m)↓ WA (m)↓ Human3R12.722.61.0315%0.20112.2 UniSH15.528.10.4773%0.98118.5 WildHSR3.617.40.996%0.1366.3 02468 time (s) −40 −20 0 20 40 sole above own ground (cm) median |sole - own ground| Ground truth Human3R 19 cm UniSH 13 cm JOSH3R 18 cm WildHSR 2 cm GT foot planted (strip) Ground truth Human3R UniSH JOSH3R WildHSR t = 1.3 st = 3.7 st = 6.3 st = 8.7 s Figure 12: Feet on the reconstructed ground (EMDB 79), each method on its own scene, at four moments with the same viewing direction and zoom. Fig. 12 shows feet relative to each method’s re- constructed ground; Table 10 extends the mea- surement to every EMDB-2 sequence. How Table 10 is measured.Feet-to-ground and sliding are read at each method’s own met- ric scale: the method is placed in the ground- truth world by a rotation and translation only alignment of its camera trajectory, so no ground- truth scale is imposed on its body or on its scene. Feet-to-ground is, per sequence, the me- dian over frames of the absolute height gap be- tween the lower foot sole and the floor of the method’s own reconstructed scene directly be- neath it, where that floor is the mode of scene- point heights within 25 cm horizontally, gath- ered from frames within ±1 s. EMDB provides no ground-truth scene, so the quantity measures how self-consistent a method’s human and scene are, not how accurate either one is. The fraction of frames carrying a floor estimate is 95% for Human3R, 100% for UniSH and 98% for WildHSR; admitting floors up to 2.5 m above the sole instead of 0.5 m moves the means to 12.7, 16.9 and 3.6 cm, so the ordering is not an artefact of that threshold. Other diagnostics in Table 10. Sliding is median horizontal toe speed on contact frames. A ground- truth toe counts as contacting when its speed is below 0.25 m/s and its height is within 5 cm of its local minimum, after five-frame smoothing. World size is the median ratio of reconstructed to ground-truth camera-trajectory scale (ideal: one). Size error averages |1− s| over sequences, so a near-one median can still conceal large errors on individual sequences. Camera error is mean camera-centre distance after rotation and translation alignment only. WA-MPJPE repeats the full- benchmark values of Table 1, not a first-300-frame recomputation. 18 Physical plausibility. Table 9 shows that contact regularization improves WA while leaving W- MPJPE and RTE nearly unchanged. These metrics do not directly test body-ground consistency. Measured against the backbone’s own reconstructed floor on all 25 EMDB-2 sequences, the lowest body vertex floats 9.8 cm above the floor without the term and 1.1 cm with it, with 1.2% of frames penetrating the floor by more than 5 cm; foot skate and vertex jitter are unchanged (1.45 cm/frame, 3.9 cm/frame 2 ). These measurements complement the benchmark metrics by testing people-scene consistency directly. IIDENTITY FEATURES AND PELVIS LOCALIZATION Probe setup. We run the unmodified pretrained VGGT-Ω alone; the mesh model, fusion and adapters are absent. Person boxes and tracks select evaluation patches and provide identity labels, but are not inputs to the backbone. VGGT-Ω learns geometric matching on static scene points, with movable content excluded from that supervision (Wang et al., 2026). The question is whether its matching representation also links moving people, even though no training target associates their identities. Position control. A stationary person can be matched by image location alone, so the probe sepa- rately tests pairs where the person moves (box IoU below 0.2). It compares the person’s new position with the vacated position and other people. The aggregate result below, rather than a selected frame pair, motivates the intermediate-depth identity features. Retrieval over 1,500 clips. For each 100-frame multi-person window, a patch on one person is matched to on-person patches in a later frame. A match is correct when its top candidate belongs to the same person. We evaluate output-token cosine similarity and pre-rotation query-key logits, including hard pairs where the nearest image-space candidate is the wrong person. Chance is 46% overall and 40% on hard pairs; moved-person pairs (box IoU below 0.2) have 38% chance. In query-key space, retrieval peaks at 80 to 82% in layers 11 to 15, including 70% after motion and 60% on hard pairs, then returns to chance in late layers. Figure 13(a) shows the intermediate-depth advantage on hard pairs, where matching by image location fails. Output tokens already retrieve at 85% at layer 0, consistent with appearance information from the image encoder, but their retrieval decays with depth. This contrast motivates reading identity from intermediate query-key features rather than final tokens. After motion. At layer 13, attention lift on the person’s new position averages 5.0× uniform, compared with 1.2× on the vacated position and 1.5× on other people among clips with sufficient moved pairs. The new position wins in 91% of those clips, although overlap and small people weaken the effect. This control separates person correspondence from simply revisiting an image location; it does not by itself measure the trained tracker’s accuracy. What the probe does not show. We have not run the same test on objects other than people, so we do not claim the behaviour is specific to people: a model that matches any moving surface across frames would pass it, and for our purpose that would serve equally. The probe alone also cannot establish that its correspondence improves tracking. The association-component experiment in Table 5 tests how descriptor and motion cues combine when upstream proposals and bodies are held fixed; it does not isolate their effect in the final end-to-end pipeline. Identity readout and training. The pelvis is the body root, giving a smoother motion anchor than distal joints and avoiding a head-to-root translation lever arm. Lower-body occlusion is examined in Appendix J. For proposal n in frame t, we take the QK-normalised layer-13 query and key vectors before positional rotation, average each head over the 3× 3 token neighbourhood around the pelvis patch, concatenate all heads and both roles, and compute e t,n = norm 2 P [ ̄q t,n ; ̄ k t,n ] , P = Linear◦ GELU◦ Linear◦ LayerNorm,(7) where e t,n ∈R 128 . VGGT and the mesh branch remain frozen. We first train only P with supervised contrastive temperature 0.07: observations of the same training identity at different times are posi- tives and co-occurring people are hard negatives. We then fine-tune P through the soft assignment defined below using negative log likelihood of the ground-truth match, birth and miss decisions. Identity groups are disjoint between training and validation, and only BEDLAM2, RICH-train and 3DPW-train identities are used. 19 05101520 layer 0 20 40 60 80 100 same person retrieved (%) chance nearest in image: 0 by construction (a) retrieval where position gives the wrong answer attention's matching space (query to key) token content (cosine) 05101520 layer 1 2 3 4 5 attention over uniform (b) after the person has moved the person, where they are now the place they left other people Figure 13: Position or person? Pretrained VGGT-Ω by itself. Top: a query token on a walking person (white box) and its global attention on a later frame, at layers 4, 8, 13 and 21; dashed box, where the person was; green box, where they are. The example is the clip’s median, not its best. Bottom, means over clips, layers 10 to 16 shaded: (a) same-person retrieval on pairs where the nearest candidate in the image is the wrong person, in the attention’s query and key space and in token content; (b) attention lift after the person has moved. State and dimensionless cost. Association is applied after each window has been composed into the common world frame of §3.4. A live track i stores a unit descriptor m i , its last metric pelvis p i , an exponentially smoothed velocity v i , last-seen time and track shape. At elapsed time ∆t, its predicted pelvis is ˆp i = p i + v i ∆t. Proposal j has descriptor e j , world pelvis p j and confidence c j . We use d id ij = 1 2 (1−⟨m i ,e j ⟩), g i = 0.5 m + min(3 m s −1 ∆t, 3 m), d mot ij = min(∥p j − ˆp i ∥ 2 /g i , 1), d conf j = 1− c j , C ij = d id ij + d mot ij + 0.25d conf j . (8) The three component costs lie in [0, 1], while C ij ∈ [0, 2.25]. A real pair outside the motion gate, ∥p j − ˆp i ∥ 2 > g i , is invalid. These weights and gates are fixed before benchmark evaluation. Births, misses and one-to-one hardening. With M tracks and N proposals of confidence at least 0.5, we construct a square (M +N )× (N +M ) cost matrix. Its top-left block is C. Track i has one dedicated miss column of cost 0.7, proposal j has one dedicated birth row of cost 0.7, invalid dummy edges have infinite cost, and the dummy-to-dummy block has zero cost. Log-domain Sinkhorn with temperature 0.1 and 20 normalisation iterations produces the soft matrix used by the assignment loss. At inference, Hungarian matching on its negative log probabilities gives exactly one decision per real track and proposal. A matched real pair continues a track, a matched birth starts one, and a matched miss preserves the track without fabricating an observation. Safe memory update. A match updates memory only when its Sinkhorn probability is at least 0.7: m i ← norm 2 (0.9m i + 0.1e j ), while velocity uses the analogous 0.8/0.2 update. Lower- confidence matches keep the identity but do not change descriptor, velocity or shape memory. A missed track can be reactivated for one second and is then terminated. Track shape is the median of confident matched observations only. Proposal selection, Hungarian hardening, births, misses, memory updates and the shape median receive no gradient. Sinkhorn is differentiable during training and analytic at inference; no test-time model fitting or backpropagation is used. 20 JROBUSTNESS TO TRUNCATION AND OCCLUSION WA 54 m coverage 99% WA 140 m coverage 100% 0100200300 frame 30 100 300 1000 WA (m) WA 106 m coverage 100% WA 451 m coverage 100% 0100200300 frame WA 86 m coverage 43% WA 92 m coverage 53% 0100200300 frame WA 63 m coverage 82% WA 172 m coverage 82% 0100200300 frame no body output WA 112 m coverage 80% no body output WA 267 m coverage 80% 0100200300 frame CleanLegs cut (below hips)Head cut (above shoulders)Moving occluderFrame dropout Input WildHSR Human3R White outline / dark dots = ground-truth body. 3D bodies after the WA alignment of their 100-frame chunk. Error plots: per-frame WA-MPJPE, log scale; grey band = frames with the mask on the person; dotted line = shown frame; gaps = no body output. Figure 14: Truncation and occlusion on a 300-frame EMDB-2 clip. Top: masked input and ground- truth outline. Middle: reconstructed bodies after WA alignment, with ground truth as dots. Bottom: per-frame WA-MPJPE on a log scale; shading marks masked frames. Table 11: Six-clip truncation and occlusion. Median WA-MPJPE (m); body coverage is WildHSR / Human3R. Shading compares meth- ods within each condition. conditionWildHSR↓ Human3R↓coverage clean63.689.1100/100% legs cut119.5238.1100/100% head cut80.9131.848/52% moving occluder71.3181.682/82% frame dropout183.6254.480/80% Six-sequence evaluation. Table 11 reports the median over six EMDB-2 clips under identi- cal synthetic masks. WildHSR has lower er- ror in 29 of 30 clip-condition pairs. Leg re- moval mainly worsens placement; head removal roughly halves detection coverage for both sys- tems. Blank frames are hardest because camera and scale are shared across the window. The crop-based mesh branch preserves local articulation under these masks; the larger changes are in person detection and placement. The masks are synthetic, so the table tests sensitivity to missing input rather than natural-occlusion frequency. 21 KMEASURED FAILURE MODES EMDB 28. WildHSR's knee-flexion estimate deviates from ground truth during deep lunges. RICH burpee. No WildHSR body is reconstructed in frames 100-149; the person returns under a new ID. (b) Wrong knee bend in deep lunges (c) Person lost on the ground, back under a new ID EMDB 28. WildHSR's knee-flexion estimate deviates from ground truth during deep lunges. RICH burpee. No WildHSR body is reconstructed in frames 100-149; the person returns under a new ID. (b) Wrong knee bend in deep lunges (c) Person lost on the ground, back under a new ID EMDB 28. WildHSR's knee-flexion estimate deviates from ground truth during deep lunges. RICH burpee. No WildHSR body is reconstructed in frames 100-149; the person returns under a new ID. (b) Wrong knee bend in deep lunges (c) Person lost on the ground, back under a new ID EMDB 28. WildHSR's knee-flexion estimate deviates from ground truth during deep lunges. RICH burpee. No WildHSR body is reconstructed in frames 100-149; the person returns under a new ID. Figure 15: Measured failure modes. (a) Stair contact, (b) knee articulation, and (c) identity conti- nuity during a burpee. Distances use each method’s own metric scale. What the failures show. (a) On EMDB 77, the soles sink below the reconstructed step while climbing: a median of−9 cm over the clip and about−20 cm on the steepest stretch, versus−3 cm on a flat walk; Human3R also sinks (−17 cm). Scene contact is a training term, not an inference- time correction. (b) On EMDB 28, full-clip knee-flexion error is 7.8 ◦ for WildHSR versus 5.4 ◦ for Human3R; the plotted window reports its own error. (c) In a RICH burpee, no WildHSR body is reconstructed for frames 100–149, and the person returns under a new track ID. These examples expose limits in stair placement, body articulation and track continuity. 22