Paper deep dive
Syn2RealTrack: Bridging the Gap Between Synthetic and Real-World Datasets for Online Multi-View Multi-Target Tracking
Duong Nguyen-Ngoc Tran, Ngoc Doan-Minh Huynh, Cu Quoc Le, Hoang-Khang Nguyen, Long Hoang Pham, Huy-Hung Nguyen, Quoc Pham-Nam Ho, Trinh Le Ba Khanh, Chi Dai Tran, Duong Khac Vu, Son Hong Phan, Hyung-Min Jeon, Jae Wook Jeon
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/29/2026, 4:01:29 AM
Summary
The paper introduces Syn2RealTrack, an online multi-view multi-target tracking pipeline designed to bridge the synthetic-to-real gap in warehouse 3D perception. Instead of using a single domain adaptation module, the system addresses three specific sources of discrepancy: camera calibration (lens distortion), object shape priors (height estimation), and the assumption of known object census. Key technical contributions include distortion-aware camera grouping using AnyCalib, visibility-weighted part-based cross-view fusion that abstains on occluded parts, closed-form person height estimation from calibration, and a causal filter to remove phantom boxes based on a closed-world cardinality prior. The system achieved a 3D HOTA of 52.0118% on the AI City Challenge 2026 Track 1 evaluation.
Entities (9)
Relation Signals (10)
Syn2RealTrack → achievesscoreon → AI City Challenge 2026 Track 1
confidence 95% · On the AI City Challenge 2026 Track~1 evaluation server it reaches a 3D Higher Order Tracking Accuracy (HOTA) of 52.0118%.
Syn2RealTrack → uses → AnyCalib
confidence 92% · We use AnyCalib [35] only for distortion analysis, grouping, and initialization
Syn2RealTrack → addressesgapat → object shape prior
confidence 90% · we argue instead that it enters the pipeline at three separable points: the camera calibration, the object shape prior, and the assumption that the object census is known
Syn2RealTrack → addressesgapat → camera calibration
confidence 90% · we argue instead that it enters the pipeline at three separable points: the camera calibration, the object shape prior, and the assumption that the object census is known
AnyCalib → estimates → lens distortion
confidence 90% · AnyCalib [35] predicts a dense per-pixel ray and field-of-view map and fits a pinhole baseline, a radial Brown–Conrady model, and the Unified Camera Model
Syn2RealTrack → uses → RF-DETR
confidence 90% · We train the RF-DETR 2x-large detector [25] for 200 epochs
Syn2RealTrack → uses → ViTPose
confidence 90% · We apply pretrained ViTPose++ [46] to each detection crop.
→ →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multi-camera 3D perception systems for warehouse scenes are trained largely on synthetic data and evaluated on physically captured environments. The resulting synthetic-to-real gap, which corrupts ground-plane localization and cross-camera identity association, is usually treated as one deficiency for a single domain-adaptation module to absorb; we argue instead that it enters the pipeline at three separable points: the camera calibration, the object shape prior, and the assumption that the object census is known, each admitting a different local remedy. Our online pipeline, Syn2RealTrack, follows this decomposition: lens distortion is recovered from images alone under a calibration that provides none, detections are fused across views by a visibility-weighted part-based descriptor that abstains on occluded parts rather than guessing, person height is measured in closed form from calibration instead of copied from a synthetic prior, and a closed-world cardinality prior is paired with a causal filter that removes the phantom boxes the prior manufactures. The system therefore adapts by reallocating trust between geometry and appearance without retraining a feature extractor. On the AI City Challenge 2026 Track~1 evaluation server it reaches a 3D Higher Order Tracking Accuracy (HOTA) of 52.0118%. The code will be released at this https URL
Tags
Links
- Source: https://arxiv.org/abs/2608.24130v1
- Canonical: https://arxiv.org/abs/2608.24130v1
Trouble viewing inline? Open PDF directly →
Full Text
65,514 characters extracted from source content.
Expand or collapse full text
Syn2RealTrack: Bridging the Gap Between Synthetic and Real-World Datasets for Online Multi-View Multi-Target Tracking Duong Nguyen-Ngoc Tran∗ Ngoc Doan-Minh Huynh∗ Cu Quoc Le∗ Khang Nguyen Hoang∗ Long Hoang Pham Huy-Hung Nguyen Quoc Pham-Nam Ho Trinh Le Ba Khanh Chi Dai Tran Duong Khac Vu Son Hong Phan Hyung-Min Jeon Jae Wook Jeon△ Affiliation: Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, South Korea E-mail duongtran, ngochdm, lequoccu2003, khangnguyen, jwjeon@skku.edu Abstract Multi-camera 3D perception systems for warehouse scenes are trained largely on synthetic data and evaluated on physically captured environments. The resulting synthetic-to-real gap, which corrupts ground-plane localization and cross-camera identity association, is usually treated as one deficiency for a single domain-adaptation module to absorb; we argue instead that it enters the pipeline at three separable points: the camera calibration, the object shape prior, and the assumption that the object census is known, each admitting a different local remedy. Our online pipeline, Syn2RealTrack, follows this decomposition: lens distortion is recovered from images alone under a calibration that provides none, detections are fused across views by a visibility-weighted part-based descriptor that abstains on occluded parts rather than guessing, person height is measured in closed form from calibration instead of copied from a synthetic prior, and a closed-world cardinality prior is paired with a causal filter that removes the phantom boxes the prior manufactures. The system therefore adapts by reallocating trust between geometry and appearance without retraining a feature extractor. On the AI City Challenge 2026 Track 1 evaluation server it reaches a 3D Higher Order Tracking Accuracy (HOTA) of 52.0118%52.0118\%. The code will be released at https://github.com/SKKUAutoLab/aic26_mc3dp. Keywords: Multi-target multi-camera tracking 3D object perception Synthetic-to-real transfer **footnotetext: These authors contributed equally to this work.$ $$ $footnotetext: Corresponding author. 1 Introduction Warehouse automation depends on knowing where every person and every piece of equipment is, in metric floor coordinates, at every instant. A single camera cannot provide this information: it recovers a bearing but not a range, and, among tall racks, a target may disappear behind an obstruction within a few frames. Fixed multi-camera rigs with overlapping fields of view are the standard remedy, and the task they define is multi-target multi-camera (MTMC) 3D perception: recovering each object’s world-frame position, footprint, and orientation, together with an identity that persists through occlusions and handovers between cameras [32]. The obstacle is supervision. Dense 3D boxes and cross-camera identities are expensive to annotate at scale in a working facility [44], so the practical solution has been to train in simulation, where exact ground truth is freely available. What simulation does not capture is the collection of small physical irregularities that accumulate in a real installation: a lens that does not conform to the calibration on file, an object whose true dimensions differ from those in the catalogue, and a floor whose occupancy has not been counted. (a) Synthetic scene (b) Real-world scene Figure 1: Representative point-cloud-guided 3D box refinement examples on (a) a synthetic and (b) a real-world scene of the AI City Challenge 2026 Track 1 dataset [31]. Red boxes denote the original multi-view tracking prior, green boxes denote the refined 3D boxes, and the colored points denote the local cloud reconstructed by Depth Anything 3 (DA3) [15]. We present Syn2RealTrack, our AI City Challenge 2026 Track 1 system: an online multi-camera 3D perception pipeline trained mainly on synthetic data and evaluated partly on real scenes [31]; Fig. 1 illustrates the two domains and our point-cloud-guided box refinement on both. Rather than treating the synthetic-to-real gap as one domain-adaptation problem, we localize it to three interfaces (camera calibration, object-shape priors, and known object counts) and address each where it arises. Fig. 2 summarizes the framework; our contributions are as follows. • Localizing the synthetic-to-real gap. With feature extractors trained once and then frozen across domains, we expose calibration, camera overlap, and object counts as a configuration surface that shifts trust from simulation-specific geometry to appearance when these cues are unreliable in real scenes (Tab. 1). • Distortion-aware camera grouping. As the calibration omits lens distortion, we estimate it with AnyCalib [35], split reference and fisheye views by the Unified Camera Model parameter ξ, undistort footpoints while keeping the provided geometry, and mask fisheye-only zones to avoid invalid matches. • Cross-view fusion that can abstain. We ground detections, fuse views by visibility-weighted part similarity, and propagate single-camera identities before geometric assignment to reject inconsistent world placements; this fusion adds 0.060.06 HOTA on top of single-view association. • Measured rather than synthetic geometry. For people, a closed-form estimate from calibration and ankle ground points replaces the synthetic height prior without depth or extra supervision; an RGB-only Depth Anything 3 (DA3) point cloud [15] then refines footprint and yaw while preserving identity and class-specific size. • A contained closed-world prior. We apply exact per-class counts only in suitable closed-world scenes and suppress unsupported boxes with a causal filter requiring visible-ankle confirmation after reprojection into covering cameras; this filter yields a detection-driven gain at negligible association cost. 2 Related Work Synthetic-to-Real Generalization and Calibration. Multi-camera domain shift is both photometric and geometric. Prior work improves transfer through view-consistent augmentation, depth- and pose-robust BEV features, or perspective-invariant rendering [6, 42, 17, 10]; calibration methods instead recover camera geometry from person correspondences, learned perspective cues, or predicted pixel rays [45, 39, 35, 37]. Our pipeline keeps its feature extractors and the provided intrinsics and extrinsics fixed; it adapts only at explicit downstream interfaces and estimates each camera’s omitted lens distortion before ground-plane projection. Geometry, Visibility, and Multi-View Association. Prior work localizes people and recovers metric shape through calibrated pose, projective metrology, RGB geometry, or point-supported box refinement [14, 4, 41, 19, 15, 26]. Association methods use visible-part features [29, 28], probabilistic occupancy, target-count, and camera models [40, 21, 22], or learned BEV and trajectory relationships [34, 43, 47]. Our causal pipeline combines these ideas: class-specific anchors and ankle keypoints recover ground contact and person height; RGB point clouds refine footprint and yaw; shared visible parts, local-identity memory, motion, and a bounded gallery guide sequential association. An external class census conditions assignment when available; otherwise, adaptive tracking and a camera-coverage check suppress unsupported person boxes. The key difference from these works is that our association can abstain: parts occluded in either view are excluded from similarity instead of imputed, and single-camera identities take precedence over geometric matching. Multi-View Fusion and Tracking. Multi-view detectors aggregate projected features in BEV using convolution, transformers, or multi-height homographies, while CaMuViD exchanges features directly across uncalibrated views [8, 9, 30, 5, 36]. AI City 2025 systems similarly combine global point-cloud detection, cluster-to-tracklet mapping, or late multi-view tracking and refinement [32, 13, 38, 23], but rely on benchmark-provided depth and calibration. The 2026 protocol adds real scenes and withholds depth at inference [31]. Our pipeline therefore preserves modular per-view perception and fuses sparse object observations late rather than learning dense BEV aggregation. 3 Methodology We address online 3D multi-view multi-target tracking in warehouse scenes [31, 38], each observed by C static, calibrated RGB cameras over frames t∈0,…,T−1t∈\0,…,T-1\ across seven classes =0,…,6 K=\0,…,6\ (Person, Forklift, NovaCarter, Transporter, FourierGR1T2, AgilityDigit, PalletTruck). Per frame we emit oriented, class-labeled 3D boxes with time- and view-consistent identities. The synthetic-to-real gap surfaces at calibration, the shape prior, and the closed-world assumption. Figure 2: Overview of the proposed framework, read left to right. (A) Data preprocessing. AnyCalib [35] estimates per-camera distortion from 3131 sampled frames, and the Unified Camera Model (UCM) parameter ξ separates reference (ξ≤0.3ξ≤ 0.3, blue) and fisheye-candidate (red) views. (B) Per-view perception. RF-DETR [25], ViTPose++ [46], and Keypoint Promptable Re-Identification (KPR) [28] provide detections, keypoints, and visibility-aware part descriptors, which an appearance–IoU Kalman tracker links into local identities. (C) Multi-view fusion. Class-specific ground anchors and visibility-weighted part similarity merge observations within per-class bird’s-eye-view (BEV) gates; local-identity carry-forward and gated Hungarian assignment then form global tracks under cap NkN_k. (D) 3D box construction and refinement. Trajectory-derived yaw, monocular person height, footprint and yaw refinement guided by Depth Anything 3 (DA3) [15], and BEV visibility filtering produce the final boxes. Numbered badges identify the three synthetic-to-real gaps (calibration, shape priors, and closed-world cardinality), and dashed links show cross-stage dependencies between measured height, ground anchoring, and BEV filtering. Notation. Camera c∈1,…,Cc∈\1,…,C\ has intrinsics c∈ℝ3×3K_c ^3× 3 and world-to-camera extrinsics c=[c∣c]∈ℝ3×4E_c=[R_c _c] ^3× 4. An image pixel =(u,v)⊤u=(u,v)^\! has homogeneous form ~=(u,v,1)⊤ u=(u,v,1)^\! ; =(X,Y)⊤x=(X,Y)^\! is a metric world point on the ground plane Z=0Z=0. A detection carries a normalized center-size box =(bx,by,bw,bh)b=(b_x,b_y,b_w,b_h), confidence s∈[0,1]s∈[0,1], class k∈k∈ K, 1717 COCO [16] keypoints (j,sj)j=016\(u_j,s_j)\_j=0^16 for humanoid classes, a part-based appearance descriptor ∈ℝ6×512F ^6× 512 whose rows pf_p are ℓ2 _2-normalized per body part, and per-part visibility ∈[0,1]6 ν∈[0,1]^6. Ground-plane projection. A ground point has Z=0Z=0, so the third column of the world-to-image matrix c=cc∈ℝ3×4P_c=K_cE_c ^3× 4 drops out of the projection. For ¯c∈ℝ3×3 P_c ^3× 3 formed by columns 11, 22, and 44 of cP_c, the ground-plane projection is λ~=¯c(X,Y,1)⊤λ u= P_c(X,Y,1)^\! at projective depth λ≠0λ≠ 0, inverted by the homography c≜¯c−1H_c P_c^-1: =c~,πc()=(q1q3,q2q3)⊤,q\;=\;H_c\, u, _c(u)\;=\; ( q_1q_3,\; q_2q_3 )^\! , -11.38092pt (1) with homogeneous =(q1,q2,q3)⊤q=(q_1,q_2,q_3)^\! and ground position πc _c. We reject back-projections with |q3|<εh|q_3|< _h, where εh=10−9 _h=10^-9 (the pixel lies on or beyond the horizon, where its viewing ray never meets the ground), or with ‖πc()‖2>103m\| _c(u)\|_2>10^3\,m, which bounds the world extent and discards numerically degenerate rays. 3.1 Distortion-Aware Camera Grouping Figure 3: Single-view distortion estimation and camera grouping. (1) Frames are evenly sampled from every camera of a scene. (2) AnyCalib [35] predicts a dense per-pixel ray and field-of-view map and fits a pinhole baseline, a radial Brown–Conrady model, and the Unified Camera Model (UCM), whose parameter ξ is used as a fisheye score; the estimates are used for distortion analysis and grouping only, and the dataset calibration is kept fixed for bird’s-eye-view (BEV) projection. (3) Cameras with ξ below the threshold become reference views (blue), and the rest become fisheye candidates (red), whose reprojection displacements grow toward the image borders. The dataset calibration [31] is pinhole-only: intrinsics, extrinsics, and a ground-plane homography, but no distortion coefficients. With noticeable lens distortion, projected foot points land at the wrong BEV positions, and cross-view association suffers. This step identifies the offending cameras and pulls the projected locations back (Fig. 3). We use AnyCalib [35] only for distortion analysis, grouping, and initialization; the dataset pinhole calibration remains the reference geometry for BEV projection. Single-View Distortion Estimation (steps 1–2 in Fig. 3): We run pretrained AnyCalib on 31 evenly sampled frames per camera and take the per-parameter median. It predicts a dense pixel-to-ray field and fits a pinhole baseline, a radial Brown–Conrady model [2], and the Unified Camera Model (UCM) [7, 20]; we use the UCM parameter ξ as a fisheye score. Camera Grouping (step 3): Cameras with ξ≤0.3ξ≤ 0.3 are reference cameras; the rest are fisheye candidates. This distortion-based split (not a physical lens label) agrees with the estimated radial coefficients and is confirmed by reprojection quiver plots: displacements stay small for reference cameras and grow toward the borders for fisheye candidates. 3.2 2D Object Detection Because noisy 2D annotations can slow convergence, we classify each category as static, fixed-shape, or dynamic-shape from its per category dimension statistics, providing fallback shapes when matching fails, and remove redundant, tiny, unseen, or invalid boxes. Training scenes are matched to each test scenario, including the otherwise absent ‘AgilityDigit’ and ‘FourierGR1T2’ categories, with real scenes drawn from AI City [32] and MTMMC [44]. We train the RF-DETR 2x-large detector [25] for 200 epochs at 19201920 pixels (batch size 4). Inference uses the same resolution with NMS at IoU 0.5 and a 0.1 confidence threshold. Further details are in the supplementary material (Appendix 0.A). 3.3 Single-View Re-Identification and Tracking For each detected person, we estimate body keypoints and use them to extract pose-aware descriptors for identity association. Within each camera, motion and these appearance cues then link detections over time into local tracklets, before any cross-view reasoning. Pose Estimation. We apply pretrained ViTPose++ [46] to each detection crop. Its transformer encoder and lightweight decoder provide keypoints under changes in pose, scale, and occlusion. Re-Identification. Keypoint Promptable Re-Identification (KPR) [28] uses these keypoints (positive only) to split the body into six regions and extract region-level descriptors, reducing background interference and occlusion sensitivity. As identities are scene-specific in AI City 2026 Track 1, we merge and relabel the train/val splits into 155 identities and train KPR for 110 epochs (≈ 3 h 7 min). These descriptors feed both the single-view tracker below and the multi-view associator of Sec. 3.4; further details are given in Appendix 0.B. Single-View Tracking. Following tracking-by-detection [1], we pair an appearance-augmented IoU tracker with a constant-velocity Kalman filter [11]: box overlap and predicted motion supply the association affinity, and the part-based re-identification features disambiguate crossings and brief occlusions. Each surviving tracklet gets a local identity ℓ , and per camera and frame the stage emits instances (ℓ,k,,s,(j,sj),,)( ,k,b,s,\(u_j,s_j)\,F, ν). The global associator of Sec. 3.4 consumes these local identities as a carry-forward cue, since single-camera identity proved more reliable than cross-frame geometric matching alone. 3.4 Multi-View Tracking Given the per-camera tracklets, multi-view tracking fuses observations of the same object across cameras into a globally consistent identity, in three steps: each detection is anchored to a ground-plane position by a class-adaptive rule, the resulting observations are merged across views per frame, and the merged world observations are bound to persistent global tracks. 3.4.1 Class-Adaptive Ground-Contact Anchoring. Back-projection through cH_c is exact only for a pixel that genuinely lies on the floor, so the accuracy of the entire bird’s-eye-view (BEV) representation reduces to choosing that pixel. A single rule cannot serve all seven classes: a person’s feet are visible and semantically well defined, a low, flat robot has no meaningful feet, and a tall vehicle seen from a steep overhead angle has its base occluded by its own body. We therefore select an anchoring strategy per class through a configurable map, with three strategies. Skeleton. For humanoid classes with a reliable pose, the anchor is the midpoint of the two ankle keypoints (COCO indices 1515 and 1616) whenever at least one clears the ankle-confidence threshold θank _ank. When neither does, we extrapolate down the leg’s kinematic chain as ankle=knee+ρ(knee−hip)u_ankle=u_knee+ρ (u_knee-u_hip ) with ρ=1ρ=1, where hipu_hip and kneeu_knee are the hip and knee keypoints of one leg and ρ is the shank-to-thigh length ratio, taken as unity for a straight leg; the anchor is the mean over whichever legs admit the construction. This extrapolation fires only when more than nkpn_kp keypoints clear the pose-confidence threshold θpose _pose, so that a fragmentary skeleton never fabricates a ground contact. Center point. For low, flat robots whose bounding box is effectively their footprint, the anchor is the box center (bxWimg,byHimg) (b_xW_img,\;b_yH_img ), with WimgW_img and HimgH_img the image width and height in pixels. Top–bottom. For tall or self-occluding objects the anchor depends on where the object sits in the frame. A box in the upper image half is viewed near-horizontally, and its bottom edge is a trustworthy ground contact, so the anchor is the bottom-center (bxWimg,(by+bh/2)Himg) (b_xW_img,\;(b_y+b_h/2)H_img ). A box in the lower image half is close to the camera under a steep view, where the base is routinely occluded or clipped while the top of the object stays crisp; there we anchor on the reliable top edge and descend by the object’s projected height. Let ~h=(X,Y,h,1)⊤ X_h=(X,Y,h,1)^\! be the world point at height h above the ground point (X,Y)(X,Y), and let c(2)p^(2)_c and c(3)p^(3)_c denote the second and third rows of cP_c. The image row of that point, and the apparent pixel height of a vertical segment of length h, are v(h)=c(2)~hc(3)~h,Δvc(X,Y,h)=|v(0)−v(h)|,v(h)\;=\; p^(2)_c X_hp^(3)_c X_h, v_c(X,Y,h)\;=\; |\,v(0)-v(h)\, |, -8.5359pt (2) where v(h)v(h) is the image row of the top of that segment standing at (X,Y)(X,Y) and Δvc v_c its apparent pixel height. Seeding the footprint at the bottom-center back-projection and taking h=Hkh=H_k, the constant world height prior of class k (defined with the other class extents in Sec. 3.5), the anchor row becomes vanchor=max((by−bh/2)Himg+Δvc(πc(bot),Hk),(by+bh/2)Himg),v_anchor\;=\; (\,(b_y-b_h/2)\,H_img\;+\; v_c ( _c(u_bot),\,H_k ),\;\;(b_y+b_h/2)\,H_img\, ), -8.5359pt (3) where botu_bot is the box bottom-center pixel and HkH_k the class height prior of Eq. 6 in Sec. 3.5. The max enforces that the anchor never rises above the visible box bottom, while permitting it to fall below that edge, exactly the intended correction when the true base is clipped out of the box. Every anchor pixel is back-projected with cH_c through Eq. 1 and discarded if it is degenerate or falls outside the camera’s ground-coverage zone. This third strategy is powerful but circular: it derives the ground contact from the class height prior. The estimator in Sec. 3.5 breaks that circularity for persons by estimating height from a ground point constructed independently of it. 3.4.2 Cross-View Association on the Ground Plane At each frame, projected observations oi=(ci,ki,ℓi,i,si,i,i)o_i=(c_i,k_i, _i,x_i,s_i,F_i, ν_i), encoding camera, class, local identity, ground position, confidence, descriptor, and part visibility, are associated within each class before temporal tracking. Appearance is compared by a visibility-weighted cosine distance that ignores parts occluded in either view: dapp(oa,ob)=∑p=16νa,pνb,p(1−⟨a,p,b,p⟩)/∑p=16νa,pνb,p,d_app(o_a,o_b)\;=\; . _p=1^6 _a,p\, _b,p\, (1- _a,p,\,f_b,p )\; /\; _p=1^6 _a,p\, _b,p ., -11.38092pt (4) where p indexes six body parts; since the descriptor rows are unit-norm, dapp∈[0,2]d_app∈[0,2], while no co-visible part gives +∞+∞. Observations from different cameras are admissible when ‖i−j‖2<θkbev\|x_i-x_j\|_2<θ^bev_k, a per-class BEV merge radius, and, for classes in reid K_reid (the classes with part-based appearance descriptors), dapp(oi,oj)<θgrpd_app(o_i,o_j)< _grp, an appearance grouping gate. Single-linkage clustering [27] over a union–find structure [33] processes pairs by increasing ground distance and rejects merges between clusters already containing the same camera. Each cluster ω G_ω becomes ω=(kω,ω,ω,ω,ω)ω=(k_ω,x_ω,F_ω, ν_ω, G_ω), with confidence-weighted position ω=∑m∈ωsmm∑m∈ωsmx_ω= _m∈ G_ωs_mx_m _m∈ G_ωs_m (or an unweighted mean if the denominator is zero). Part descriptors are fused as ω,p=∑m∈ωνm,pm,p‖∑m∈ωνm,pm,p‖2f_ω,p= _m∈ G_ω _m,pf_m,p\| _m∈ G_ω _m,pf_m,p\|_2 with νω,p=maxm∈ωνm,p _ω,p= _m∈ G_ω _m,p; this combines complementary views without imputing occluded parts, and when no member sees part p the fused weight is νω,p=0 _ω,p=0, so the part is ignored by Eq. 4. 3.4.3 Global Multi-View Tracking A global track τ holds a class kτk_τ, a ground position τx_τ, a velocity τv_τ (mframe−1m\,frame^-1), a member map ℳτ:c↦ℓ M_τ:c of the local tracklets feeding it, an appearance gallery τ A_τ of up to NgalN_gal recent descriptor–visibility pairs, and a missing age aτa_τ of consecutive unmatched frames. Each frame’s world observations bind to tracks in two passes. Pass A: carry-forward by local identity. Per observation ω and same-class track τ we count ω’s members (c,ℓ)(c, ) already in ℳτ M_τ and bind pairs greedily in descending count, each observation and track consumed once. This pass is the primary defense against identity switches: a track survives while any one of its cameras holds it. Pass B: gated assignment for the remainder. Whatever Pass A leaves is resolved per class. The prediction ^τ=τ+τ x_τ=x_τ+v_τ gives dbev(ω,τ)=‖ω−^τ‖2d_bev(ω,τ)=\|x_ω- x_τ\|_2, gated by g(τ)=θkbev+Vmaxfraτg(τ)=θ^bev_k+ V_ f_ra_τ, exactly the distance coverable while unobserved at maximum plausible speed VmaxV_ (ms−1m\,s^-1) and frame rate frf_r, so a track survives long occlusions without an indiscriminate gate. Appearance uses the nearest gallery distance dgal(ω,τ)=min(′,′)∈τdapp((ω,ω),(′,′))d_gal(ω,τ)= _(F , ν )∈ A_τ d_app ((F_ω, ν_ω),\, (F , ν ) ), where dappd_app of Eq. 4 depends on its arguments only through the descriptor–visibility pair and is +∞+∞ absent a gallery or co-visible part; the minimum, not the mean, lets one confident past view re-identify across pose or illumination change. Pairs are gated out by dbev>g(τ)d_bev>g(τ), or by k∈reid∧θreid<dgal<∞k∈ K_reid _reid<d_gal<∞ for appearance gate θreid _reid. Survivors cost [ω,τ]=βdbev(ω,τ)g(τ)+(1−β)d¯gal(ω,τ)C[ω,τ]=β d_bev(ω,τ)g(τ)+ (1-β) d_gal(ω,τ) for k∈reidk∈ K_reid and [ω,τ]=dbev(ω,τ)C[ω,τ]=d_bev(ω,τ) otherwise, with d¯gal=dgal d_gal=d_gal when finite and a neutral 11 otherwise. Normalizing by g(τ)g(τ) puts geometry on a bounded scale comparable to the appearance term’s [0,2][0,2] range, so a single weight β∈[0,1]β∈[0,1] balances the two across classes with very different gate radii. The Hungarian algorithm [12] minimizes ∑ exactly; ∞-sentinel pairs are discarded after the assignment is solved. Trajectory Lifecycle Dynamics. An unmatched observation spawns a track only when minτ:kτ=kω∥ω−τ∥2≥δnew _τ:\,k_τ=k_ω\|x_ω-x_τ\|_2≥ _new, the minimum birth separation (m); this blocks the rebirth of a momentarily unmatched track beside itself under a fresh identity. An unmatched track instead dead-reckons, τ←τ+τx_τ _τ+v_τ, aτ←aτ+1a_τ← a_τ+1, and retires once aτ>Amaxa_τ>A_ . A matched track takes τ+=αω+(1−α)τx^+_τ= _ω+(1-α)x_τ for smoothing factor α∈(0,1]α∈(0,1] and τ=τ+−τmax(1,t−tτlast)v_τ= x^+_τ-x_τ (1,\;t-t^last_τ) from its previous sighting frame tτlastt^last_τ, then τ←τ+x_τ ^+_τ; its members refresh ℳτ M_τ, its fused descriptor enters τ A_τ, and aτa_τ resets to zero. A closed-world cardinality prior. In a controlled warehouse the per-class count NkN_k is often known a priori: trivially in simulation, by inspection for a staged capture. A fixed-cardinality variant then constrains each class’s live track set k T_k, |k|≤Nk∀k,Nk=∞ if k is uncapped,| T_k|\;≤\;N_k ∀ k, N_k=∞\; if k is uncapped, -5.69046pt (5) under which capped tracks coast rather than retire, so the count converges to exactly NkN_k; contested slots go to the most confident observations, and a stale slot is reclaimed past a missing-age threshold. Association thus becomes assigning a fixed identity set to the frame’s evidence rather than managing tracks from detections. This regime is sharpest with the Pass-B gates dropped: the solve force-assigns every track to its best observation, and a track coasts only when the frame yields fewer observations than its class has tracks. The prior is exact in simulation, where the census is a fact of the scene description, and precisely what a real deployment cannot assume; the real scene therefore runs the adaptive tracker with all gates active. Finalization. In an offline post-processing pass after the last frame (the only acausal step, applied once per sequence before submission), each track’s sightings become a per-frame position map; gaps under GmaxG_ frames are linearly interpolated between bracketing sightings, and longer ones are left unfilled instead of being bridged through space the object may not have occupied. Table 1: The synthetic-to-real configuration surface: parameter settings used in the synthetic and real regimes. Parameter Synthetic regime Real regime Cameras per scene 1010–2020 44–77 Intrinsics ideal pinhole, cloned per-camera, fx≠fyf_x≠ f_y Person ground anchor bbox ++ height pole ankle skeleton Ankle confidence θank _ank 0.50.5 0.80.8 Skeleton gate nkpn_kp 33 88 Position smoothing α 1.01.0 (none) 0.60.6 Appearance gates θgrp,θreid _grp,\, _reid >2>2 (inactive) 0.80.8, 1.61.6 (active) Object cardinality NkN_k known and enforced unknown; adaptive Gap interpolation GmaxG_ 00 frames 6060 frames 3.4.4 The Synthetic-to-Real Configuration Surface The domain gap shifts the relative reliability of geometry and appearance (Tab. 1): synthetic scenes (exact calibration, broad overlap, known counts) let geometry and the closed-world prior suffice, while real scenes (estimated calibration, sparse overlap, unknown counts) need smoothing and active appearance gating (toggled by the cosine threshold of Eq. 4). The pipeline thus adapts by shifting trust between geometry and appearance, not by retraining the feature extractor. 3.5 Height and Yaw Angle Estimation Each track position (X,Y)(X,Y) at frame t is lifted to an axis-parameterized 3D box (X,Y,Z,w,l,h,ψ)(X,Y,Z,w,l,h,ψ). For yaw ψ, we fit a second-degree polynomial to the previous 45 track positions, differentiate at the current point, and take the arctangent. The baseline lift assigns constant class extents, (w,l,h)=(Wk,Lk,Hk),Z=h2,(w,l,h)\;=\;(W_k,\,L_k,\,H_k), Z\;=\; h2, -5.69046pt (6) where (Wk,Lk,Hk)(W_k,L_k,H_k) are class k’s width, length, and height priors, the per-class medians of the previous year’s warehouse ground-truth 3D box scales (the one class lacking a counterpart there is given a manual estimate), and Z=h/2Z=h/2 places a floor-standing object’s centroid at half its height. Imported wholesale, this synthetic prior is the second gap point: the shape prior. Monocular metric person height. For persons this prior becomes a per-frame calibration estimate. To break the circularity of the HkH_k-dependent anchor in Eq. 3, we form an independent ground point ~0=(X,Y,0,1)⊤ X_0=(X,Y,0,1)^\! from the ankle midpoint; with av=c(2)~0a_v=p^(2)_c X_0, bv=(c)23b_v=(P_c)_23, γv=c(3)~0 _v=p^(3)_c X_0, and dv=(c)33d_v=(P_c)_33, v(h)=(av+bvh)/(γv+dvh)v(h)=(a_v+b_vh)/( _v+d_vh), and matching the detection’s top row vtopv_top yields h^=vtopγv−avbv−vtopdv, h\;=\; v_top\, _v\;-\;a_vb_v\;-\;v_top\,d_v, -8.5359pt (7) which inverts Eq. 2 without learned depth or 3D ground truth. A sample is accepted only if the detection is untruncated, its confidence exceeds the track-confidence threshold θtrk _trk, both ankles exceed θank _ank, a head keypoint exceeds θpose _pose, the back-projection is valid and in-zone, and h^∈[hmin,hmax] h∈[h_ ,h_ ]; failing samples are rejected, not clamped, which leaves a small positive bias from ankle elevation. Track τ takes the cross-camera median h^τ,t=medianh^m∣m∈ω,h^maccepted h_τ,t=median\ h_m m∈ G_ω,\ h_m\ accepted\ over its members ω G_ω, else HkH_k with no cross-frame propagation. The box then scales as (w,l,h,Z)=(WkHkh^τ,t,LkHkh^τ,t,h^τ,t,h^τ,t2)(w,l,h,Z)= ( W_kH_k h_τ,t,\, L_kH_k h_τ,t,\, h_τ,t,\, h_τ,t2 ), which preserves the class aspect ratio. (a) Synthetic scene (b) Real-world scene Figure 4: Multi-view RGB inputs and DA3-estimated point clouds for (a) the synthetic scene Warehouse 025 and (b) the real-world scene Warehouse 027 at frame 0. 3.6 Point-Cloud-Guided 3D Box Refinement After multi-view tracking, each object has a stable identity and size but a coarse center, obtained by projecting a 2D image point to the floor via intrinsics cK_c and extrinsics cE_c. This projection fails when the point is not a true ground contact (elevated Transporters, occluded feet, distortion, calibration residuals), producing planar position and yaw errors that lower 3D IoU despite correct association. Since Track 1 withholds depth at inference [31], we refine footprint and yaw from an RGB-estimated point cloud, preserving identity and size; Fig. 1 shows typical pre-refinement misalignments. DA3-Based Metric Point Cloud Construction. We run Depth Anything 3 (DA3) [15] in a pose-conditioned multi-view setting with the Nested Giant-Large 1.1 checkpoint, which couples any-view geometry with metric-depth scaling. Per synchronized frame, all camera images are processed with the provided cK_c and cE_c, and predicted depths are back-projected into a common world frame and fused into a colored point cloud (Fig. 4; details in Appendix 0.D). Synthetic Scene Refinement. Reliable calibration and controlled categories keep the class-size priors of Eq. 6 valid, so DA3 refines only footprint and yaw. For fixed-shape objects, we crop points around the tracked box, project them to a BEV density grid, drop floor, background, and weak components, and fit the prior-sized footprint near the tracked pose, reverting to the tracker output when support is weak (Fig. 5). For dynamic-shape objects (persons, humanoid robots), the tracker center is steadier than the sparse articulated cloud, so DA3 corrects only standing yaw or one-frame jumps and keeps the prior (Fig. 1(a)). Real-World Scene Refinement. Moving targets are mainly persons, whose centers are sensitive to distorted rays, calibration residuals, and occluded feet; parked vehicles keep fixed hand-verified poses. After removing static background via temporal depth statistics, each person track’s BEV window is resolved to its densest mode by seeded coarse-to-fine mean-shift [3]: the tracker seeds the first frame, later frames reuse the refined state, and a causal temporal state bridges short losses and suppresses jitter. DA3 corrects the center while keeping the class-size prior (Fig. 1(b)). 3.7 Filtering Predictions Based on the BEV Map A per-class cap (Eq. 5) or weak detections can emit a box no camera genuinely supports: a phantom that inflates false positives. We test every emitted box against the cameras whose ground-plane coverage includes it, via per-camera visibility maps on the BEV floor map. The filter runs per frame on the emitted box set only and never alters a box’s identity, size, or pose; it proceeds in three steps: Definition of Camera Visibility Maps: For each camera c we record the ground region it observes: its projected field of view intersected with the hand-annotated coverage polygons of the BEV floor map (Fig. 6), which also mark the fisheye-only zones. Rasterizing over all cameras gives, per BEV cell, the set of cameras that see it: a visibility map () V(x) indexed by ground position x. Object Verification and Filtering: Each emitted box at τx_τ gets covering cameras (τ,t)=(τ) C(τ,t)= V(x_τ); when this set is empty, all cameras the box geometrically projects onto are used instead. A person box is re-projected into these views and kept only if some re-extracted crop shows a visible ankle, keep(τ,t)⇔(τ,t)≠∅∧maxc∈(τ,t)maxj∈15,16sc,j>θvis(kτa person class),keep(τ,t) C(τ,t)≠ \; \; _c\,∈\, C(τ,t) _j\,∈\,\15,16\s_c,j> _vis (k_τ\ a person class), -14.22636pt (8) where sc,js_c,j is the score of ankle keypoint j in camera c and θvis _vis the visibility threshold. Unconfirmed person boxes are removed, and boxes of all other classes pass unchanged; this causal per-frame check suppresses forced-assignment phantoms while leaving every surviving box untouched. Figure 5: Local fixed-shape object refinement pipeline for the synthetic scene. Starting from a colored DA3 .ply crop around a tracked object, we project the crop to a BEV density grid, keep density-excess cells after floor and ghost filtering, and fit a fixed-size box to the remaining support. The red dashed box is the multi-view tracking prior, and the green box is the refined result. (a) BEV floor map: cameras and fisheye zones (b) BEV consistency: one matched object Figure 6: BEV use of camera grouping. (a) Floor map of reference (blue) and fisheye (red) cameras; dashed violet marks fisheye-only zones, where objects retain their fisheye BEV positions. (b) For a matched object, the reference view anchors the fused target, and each fisheye projection’s error is measured in ground-plane meters. Fisheye BEV Consistency (Fig. 6): Some ground regions are seen only by fisheye cameras and have no reference anchor; these blind spots are manually annotated as BEV polygons (dashed violet in Fig. 6(a)), and objects inside keep their fisheye-detection BEV position instead of being refined toward an unsupported target. Where both groups overlap (Fig. 6(b)), matched observations are projected to BEV and the fisheye-versus-reference disagreement is measured in meters, the error BEV tracking consumes; a foot point is then corrected by undistorting it before floor projection, refining only the distortion while intrinsics and extrinsics stay fixed. 4 Experiments & Discussion 4.1 Dataset and Evaluation Metrics MTMC Tracking 2026 [31] provides synchronized 1080p/30-fps RGB videos from calibrated cameras, top-down maps, and 2D/3D annotations. Under its RGB-only protocol, we evaluate Syn2RealTrack on expanded synthetic and hidden real scenes using HOTA [18], DetA, AssA, and LocA (higher is better). Table 2: Leaderboard of Multi-Camera 3D Perception, AI City Challenge 2026 Track 1, ranked by 3D HOTA (%). Our entry (shaded, bold) ranks second. Rank ID Name 3D HOTA (%) DetA (%) AssA (%) LocA (%) 1 289 EVA 56.5447 55.6444 49.3929 79.5558 2 34 SKKU-AL-T1 52.0118 45.3056 56.5047 76.2410 3 130 Playbox 38.0105 40.2592 31.0978 75.1778 4 4 QDTers 34.1845 29.4663 33.1122 17.6841 5 149 Calix 33.7654 30.8748 30.9816 50.4720 4.2 Quantitative Results Tab. 2 reports the official Track 1 leaderboard, ranked by 3D HOTA. Our team, SKKU-AL-T1, ranks second at 52.0118%, trailing first-place EVA by 4.53 points but leading third place by 14.00. The field thus splits into two tiers: our gap to the rest is roughly three times our gap to the top. Table 3: Impact of the detection backbone and training resolution on the Track 1 evaluation server. The shaded row is the setting adopted in Sec. 3.2. Model Train Size Infer Size HOTA (%) YOLOv26 1600 1920 50.5285 1920 1920 50.7207 RF-DETR 1600 1920 50.5913 1920 1920 50.8078 Table 4: Impact of the association cues used for single-view tracking on the Track 1 evaluation server. The shaded row is the setting adopted in Sec. 3.3. Re-ID Pose gIoU dIoU HOTA (%) ✗ ✗ ✗ ✗ 50.8078 ✗ ✗ ✓ ✗ 51.1145 ✗ ✗ ✗ ✓ 50.9577 ✗ ✗ ✓ ✓ 50.8204 ✓ ✗ ✓ ✓ 51.2906 ✗ ✓ ✓ ✓ 51.3205 ✓ ✓ ✓ ✓ 51.8568 Table 5: Impact of monocular metric person height estimation (Eq. 7); the shaded row is the setting adopted in Sec. 3.5. Person height source HOTA (%) Constant class prior (Eq. 6) 51.8568 ++ monocular estimate, Warehouse 023 & 024 51.9546 ++ monocular estimate, Warehouse 025 51.9898 Table 6: Impact of BEV-map-based prediction filtering (Sec. 3.7). Filtering stage HOTA (%) No filtering 51.8813 ++ visibility-zone & static-object filtering 52.0100 ++ 3D-box coverage verification 52.0118 4.3 Ablation Study Since hidden-test annotations are unavailable, each ablation is a single evaluation-server submission, comparable with Tab. 2. Sequential configurations with minor scene-specific tuning give stage-wise, not independent, effects; we keep four-decimal precision to expose small differences. 2D Detection Backbone and Input Resolution. This experiment selects the detection backbone and training resolution. At an inference resolution of 19201920, Tab. 4 shows RF-DETR beating YOLOv26 at both training resolutions; matching the training to the inference resolution adds 0.190.19 and 0.220.22 HOTA, motivating the 19201920/19201920 setting (Sec. 3.2). The backbones differ by only 0.090.09 HOTA, so the detector choice is not decisive. Single-View Association Cues. Tab. 4 isolates the single-view affinity terms of Sec. 3.3. Over the 50.807850.8078 baseline, generalized IoU (gIoU) [24] and distance IoU (dIoU) [48] add +0.31+0.31 and +0.15+0.15 HOTA alone, but combined without appearance reach only 50.820450.8204. Re-identification or pose guidance raises HOTA to 51.290651.2906 and 51.320551.3205; both together give the best 51.856851.8568 (+1.05+1.05 over baseline), the final configuration. Monocular Person Height Estimation. Tab. 6 replaces the constant prior in Eq. 6 with the per-frame estimate of Eq. 7, enabled scene by scene. HOTA improves monotonically, by +0.13+0.13 in total over the constant-prior baseline: calibration alone beats the synthetic prior without depth or supervision. The estimator adjusts box height and centroid, not identity, so association is essentially unaffected. A residual scene-dependent ground-plane error remains, which the point-cloud refinement of Sec. 3.6 targets. BEV-Map Prediction Filtering. Tab. 6 evaluates the false-positive filter of Sec. 3.7, which removes boxes induced by the closed-world prior of Eq. 5 at camera-unsupported locations. Disabling it costs 0.130.13 HOTA; in the cumulative decomposition of Tab. 7, the same stage adds +0.23+0.23 DetA while lowering AssA by only 0.010.01. This asymmetry supports the causal, per-frame test of Eq. 8, which suppresses phantoms without terminating identities held elsewhere. The final coverage check adds +0.0018+0.0018 HOTA at negligible overhead. Table 7: Stage-wise ablation of the full pipeline on the Track 1 evaluation server. Each row adds one component to the row above it; the shaded row 6 is the submitted system of Tab. 2. # Configuration HOTA (%) DetA (%) AssA (%) LocA (%) Δ HOTA 1 RF-DETR detector only 50.8078 42.4111 56.2599 74.1915 – 2 ++ gIoU association 51.1145 43.6031 55.8617 75.0459 +0.31+0.31 3 ++ all association cues 51.8568 44.7921 56.4608 76.2410 +0.74+0.74 4 ++ cross-view ground-plane clustering 51.9205 45.1611 56.4601 76.2053 +0.06+0.06 5 ++ monocular person height estimation 51.9898 45.0712 56.5190 76.2397 +0.07+0.07 6 ++ BEV-map prediction filtering 52.0118 45.3056 56.5047 76.2410 +0.02+0.02 Contribution of Each Pipeline Stage. The cumulative ablation in Tab. 7 raises HOTA from 50.807850.8078 to 52.011852.0118 (+1.20+1.20), of which single-view association supplies +1.05+1.05 (87.1%87.1\%). The remaining stages add smaller, detection-driven gains: cross-view ground-plane clustering +0.06+0.06, monocular person height +0.07+0.07, and BEV-map filtering +0.02+0.02, the last through a +0.23+0.23 DetA gain. Across all stages DetA rises from 42.4142.41 to 45.3145.31 (+2.90+2.90), while AssA stays within 56.4656.46–56.5256.52 once all association cues are enabled. 5 Conclusion Syn2RealTrack addresses synthetic-to-real gaps in calibration, shape priors, and object census, achieving 52.0118%52.0118\% 3D HOTA and second place in AI City Challenge 2026 Track 1. It remains sensitive to calibration and RGB-derived depth, severe occlusion, frequent entry/exit, and closed-world census assumptions; future work will target uncertainty-aware calibration and depth, stronger temporal prediction, and domain adaptation. Acknowledgements This work was supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2021-0-01364, An intelligent system for 24/7 real-time traffic surveillance on edge devices) References [1] N. Aharon, R. Orfaig, and B. Bobrovsky (2022) BoT-sort: robust associations multi-pedestrian tracking. arXiv preprint arXiv:2206.14651. Cited by: §3.3. [2] D. C. Brown (1966) Decentering distortion of lenses. Photogrammetric Engineering 32 (3), p. 444–462. Cited by: §3.1. [3] D. Comaniciu and P. Meer (2002) Mean shift: a robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 24 (5), p. 603–619. External Links: Document Cited by: §3.6. [4] A. Criminisi, I. Reid, and A. Zisserman (2000) Single view metrology. International Journal of Computer Vision 40 (2), p. 123–148. External Links: Document Cited by: §2. [5] A. E. Daryani, M. U. M. Bhutta, B. Hernandez, and H. Medeiros (2025) CaMuViD: calibration-free multi-view detection. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), p. 1220–1229. External Links: Document Cited by: §2. [6] M. Engilberge, H. Shi, Z. Wang, and P. Fua (2023) Two-level data augmentation for calibrated multi-view detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), p. 128–136. External Links: Document Cited by: §2. [7] C. Geyer and K. Daniilidis (2000) A unifying theory for central panoramic systems and practical implications. In Computer Vision – ECCV 2000, p. 445–461. Cited by: §3.1. [8] Y. Hou, L. Zheng, and S. Gould (2020) Multiview detection with feature perspective transformation. In Computer Vision – ECCV 2020, p. 1–18. External Links: Document Cited by: §2. [9] Y. Hou and L. Zheng (2021) Multiview detection with shadow transformer (and view-coherent data augmentation). In Proceedings of the 29th ACM International Conference on Multimedia, p. 1673–1682. External Links: Document Cited by: §2. [10] N. D. Huynh, D. N. Tran, L. H. Pham, T. H. Tran, H. Jeon, H. Nguyen, D. Khac Vu, H. Jeon, S. H. Phan, Q. Pham-Nam Ho, C. D. Tran, T. L. B. Khanh, and J. W. Jeon (2026) TSBOW – traffic surveillance benchmark for occluded vehicles under various weather conditions. Proceedings of the AAAI Conference on Artificial Intelligence 40 (7), p. 5239–5247. External Links: Document Cited by: §2. [11] R. E. Kalman (1960) A new approach to linear filtering and prediction problems. Journal of Basic Engineering 82 (1), p. 35–45. External Links: Document Cited by: §3.3. [12] H. W. Kuhn (1955) The Hungarian method for the assignment problem. Naval Research Logistics Quarterly 2 (1–2), p. 83–97. External Links: Document Cited by: §3.4. [13] J. Lee, H. Kim, D. Lee, and K. Lee (2025) Multi-camera 3D object tracking via 3D point clouds and re-identification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, p. 5417–5424. External Links: Document Cited by: §2. [14] J. P. Lima, R. Roberto, L. Figueiredo, F. Simões, and V. Teichrieb (2021) Generalizable multi-camera 3D pedestrian detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, p. 1232–1240. External Links: Document Cited by: §2. [15] H. Lin, S. Chen, J. H. Liew, D. Y. Chen, Z. Li, Y. Zhao, S. Peng, H. Guo, X. Zhou, G. Shi, J. Feng, and B. Kang (2026) Depth Anything 3: recovering the visual space from any views. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: Appendix 0.D, Figure 1, Figure 1, 4th item, §2, Figure 2, §3.6. [16] T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick (2014) Microsoft COCO: common objects in context. In Computer Vision – ECCV 2014, p. 740–755. External Links: Document Cited by: §3. [17] H. Lu, Y. Zhang, G. Wang, Q. Lian, D. Du, and Y. Chen (2025) Towards generalizable multi-camera 3D object detection via perspective rendering. Proceedings of the AAAI Conference on Artificial Intelligence 39 (6), p. 5811–5819. External Links: Document Cited by: §2. [18] J. Luiten, A. Osep, P. Dendorfer, P. Torr, A. Geiger, L. Leal-Taixé, and B. Leibe (2021) HOTA: a higher order metric for evaluating multi-object tracking. International Journal of Computer Vision 129 (2), p. 548–578. External Links: Document Cited by: §4.1. [19] J. Ma, T. Wang, M. Liu, D. Ahmedt-Aristizabal, and C. Nguyen (2025) DCHM: depth-consistent human modeling for multiview detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 7731–7740. External Links: Document Cited by: §2. [20] C. Mei and P. Rives (2007) Single view point omnidirectional camera calibration from planar grids. In IEEE International Conference on Robotics and Automation (ICRA), p. 3945–3950. External Links: Document Cited by: §3.1. [21] J. Ong, B. Vo, B. Vo, D. Y. Kim, and S. Nordholm (2022) A Bayesian filter for multi-view 3D multi-object tracking with occlusion handling. IEEE Transactions on Pattern Analysis and Machine Intelligence 44 (5), p. 2246–2263. External Links: Document Cited by: §2. [22] L. H. Pham, Q. P. Ho, D. K. Vu, H. Nguyen, C. D. Tran, D. N. Tran, T. H. Tran, N. D. Huynh, H. J. Jeon, H. Jeon, S. H. Phan, T. Le Ba Khanh, and J. W. Jeon (2025) Data augmentation is all you need for robust fisheye object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, p. 5393–5401. External Links: Document Cited by: §2. [23] T. Phan, D. Dinh, T. Huynh, Q. Le, H. Dang, V. Tran, V. Luu, and C. Huang (2025) VGCRTrack: multi-camera 3D tracking with view-aware geometric center refinement. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, p. 5434–5440. External Links: Document Cited by: §2. [24] H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, and S. Savarese (2019) Generalized intersection over union: a metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 658–666. External Links: Document Cited by: §4.3. [25] I. Robinson, P. Robicheaux, M. Popov, D. Ramanan, and N. Peri (2026) RF-DETR: neural architecture search for real-time detection transformers. In International Conference on Learning Representations (ICLR), Cited by: §0.A.1, Figure 2, §3.2. [26] S. Shi, X. Wang, and H. Li (2019) PointRCNN: 3D object proposal generation and detection from point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 770–779. External Links: Document Cited by: §2. [27] R. Sibson (1973) SLINK: an optimally efficient algorithm for the single-link cluster method. The Computer Journal 16 (1), p. 30–34. External Links: Document Cited by: §3.4. [28] V. Somers, A. Alahi, and C. De Vleeschouwer (2024) Keypoint promptable re-identification. In Computer Vision – ECCV 2024, p. 216–233. External Links: Document Cited by: Appendix 0.B, §2, Figure 2, §3.3. [29] V. Somers, C. De Vleeschouwer, and A. Alahi (2023) Body part-based representation learning for occluded person re-identification. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), p. 1613–1623. External Links: Document Cited by: §2. [30] L. Song, J. Wu, M. Yang, Q. Zhang, Y. Li, and J. Yuan (2021) Stacked homography transformations for multi-view pedestrian detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 6049–6057. External Links: Document Cited by: §2. [31] Z. Tang, S. Wang, D. C. Anastasiu, M. Chang, et al. (2026) The 10th AI City Challenge. In ECCV Workshops, Malm"o, Sweden. Cited by: Figure 1, Figure 1, §1, §2, §3.1, §3.6, §3, §4.1. [32] Z. Tang, S. Wang, D. C. Anastasiu, M. Chang, A. Sharma, Q. Kong, N. Kobori, M. Gochoo, G. Batnasan, M. Otgonbold, F. Alnajjar, J. Hsieh, T. Kornuta, X. Li, Y. Zhao, H. Zhang, S. Radhakrishnan, A. Jain, R. Kumar, V. N. Murali, Y. Wang, S. S. Pusegaonkar, Y. Wang, S. Biswas, X. Wu, Z. Zheng, P. Chakraborty, and R. Chellappa (2025) The 9th AI City Challenge. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, p. 5526–5535. External Links: Document Cited by: §0.A.1, §1, §2, §3.2. [33] R. E. Tarjan (1975) Efficiency of a good but not linear set union algorithm. Journal of the ACM 22 (2), p. 215–225. External Links: Document Cited by: §3.4. [34] T. Teepe, P. Wolters, J. Gilg, F. Herzog, and G. Rigoll (2024) EarlyBird: early-fusion for multi-view tracking in the Bird’s Eye View. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops, p. 102–111. External Links: Document Cited by: §2. [35] J. Tirado-Garín and J. Civera (2025) AnyCalib: on-manifold learning for model-agnostic single-view camera calibration. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 8044–8055. External Links: Document Cited by: 2nd item, §2, Figure 2, Figure 3, Figure 3, §3.1. [36] D. N. Tran, L. H. Pham, H. Jeon, H. Nguyen, H. Jeon, T. H. Tran, and J. Wook Jeon (2022) A robust traffic-aware city-scale multi-camera vehicle tracking of vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, p. 3149–3158. External Links: Document Cited by: §2. [37] D. N. Tran, L. H. Pham, H. Nguyen, T. H. Tran, H. Jeon, and J. W. Jeon (2021) A region-and-trajectory movement matching for multiple turn-counts at road intersection on edge device. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, p. 4082–4089. External Links: Document Cited by: §2. [38] T. H. Tran, D. N. Tran, N. D. Huynh, C. D. Tran, L. H. Pham, Q. P. Ho, H. Nguyen, D. K. Vu, H. Jeon, H. Jeon, S. H. Phan, T. Le Ba Khanh, and J. W. Jeon (2025) DepthTrack: cluster meets BEV for multi-camera multi-target 3D tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, p. 5348–5357. External Links: Document Cited by: §2, §3. [39] A. Veicht, P. Sarlin, P. Lindenberger, and M. Pollefeys (2024) GeoCalib: learning single-image calibration with geometric optimization. In Computer Vision – ECCV 2024, Lecture Notes in Computer Science, Vol. 15098, p. 1–20. External Links: Document Cited by: §2. [40] B. Vo, B. Vo, and A. Cantoni (2007) Analytic implementations of the cardinalized probability hypothesis density filter. IEEE Transactions on Signal Processing 55 (7), p. 3553–3567. External Links: Document Cited by: §2. [41] J. Wang, M. Chen, N. Karaev, A. Vedaldi, C. Rupprecht, and D. Novotny (2025) VGGT: visual geometry grounded transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 5294–5306. External Links: Document Cited by: §2. [42] S. Wang, X. Zhao, H. Xu, Z. Chen, D. Yu, J. Chang, Z. Yang, and F. Zhao (2023) Towards domain generalization for multi-view 3D object detection in bird-eye-view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 13333–13342. External Links: Document Cited by: §2. [43] Y. Wang, T. Meinhardt, O. Cetintas, C. Yang, S. Pusegaonkar, B. Missaoui, S. Biswas, Z. Tang, and L. Leal-Taixé (2025) MCBLT: multi-camera multi-object 3D tracking in long videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, p. 5304–5313. External Links: Document Cited by: §2. [44] S. Woo, K. Park, I. Shin, M. Kim, and I. S. Kweon (2024) MTMMC: a large-scale real-world multi-modal camera tracking benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 22335–22346. External Links: Document Cited by: §0.A.1, §1, §3.2. [45] Y. Xu, Y. Li, X. Weng, and K. Kitani (2021) Wide-baseline multi-camera calibration using person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 13129–13138. External Links: Document Cited by: §2. [46] Y. Xu, J. Zhang, Q. Zhang, and D. Tao (2024) ViTPose++: vision transformer for generic body pose estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (2), p. 1212–1230. External Links: Document Cited by: Figure 9, Figure 9, Appendix 0.B, Figure 2, §3.3. [47] T. Yamane, R. Masumura, S. Suzuki, and S. Orihashi (2025) MVTrajecter: multi-view pedestrian tracking with trajectory motion cost and trajectory appearance cost. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 13270–13280. External Links: Document Cited by: §2. [48] Z. Zheng, P. Wang, W. Liu, J. Li, R. Ye, and D. Ren (2020) Distance-IoU loss: faster and better learning for bounding box regression. Proceedings of the AAAI Conference on Artificial Intelligence 34 (7), p. 12993–13000. External Links: Document Cited by: §4.3. Syn2RealTrack: Bridging the Gap Between Synthetic and Real-World Datasets for Online Multi-View Multi-Target Tracking Supplementary Material The supplementary material is organized as follows: ⊳ Appendix 0.A: 2D Object Detection. ⊳ Appendix 0.B: Re-Identification and Pose Estimation. ⊳ Appendix 0.C: 2D Single-View Tracking. ⊳ Appendix 0.D: Point Cloud Construction. Appendix 0.A 2D Object Detection 0.A.1 Dataset Preprocessing The detection ground truth provides both 3D bounding box annotations and their corresponding 2D bounding box annotations. Since the number of 2D bounding boxes is relatively large, directly training on all available annotations may require a longer convergence time, and using the entire training and validation sets without scenario-specific selection may reduce the model’s generalization to the test scenarios. The raw annotations also include redundant, small, or invalid 2D bounding boxes, which may increase false-positive detections. To address these issues, we apply three preprocessing steps before training: lookup-table mapping that summarizes per-object-type shape statistics for fallback object typing, scenario-specific selection that matches the training data to each test scenario, and filtering of redundant, small, or invalid 2D bounding boxes. Each step is described as follows: Lookup Table Mapping From the provided ground-truth annotations, we derive per-object-type shape statistics by measuring the minimum, maximum, and mean size of every object category. On this basis, objects are grouped into three types: • Static Objects: objects that stay in place for the entire observation period. • Fixed-Shape Objects: objects whose shape stays constant whether they are moving or at rest. • Dynamic-Shape Objects: objects whose shape changes as they move. These statistics allow the framework to seed each object with a default shape, so that the evaluation still yields meaningful results even when later matching steps fail. Scenario-Specific Training-Set Selection: The framework aims to reduce the number of training bounding boxes while preserving detection accuracy as much as possible. Accordingly, we select the training data according to the specific conditions of each test scenario, which shortens training without sacrificing reliability. Specific scenarios are also selected to include object categories that are absent from other scenes, such as ‘AgilityDigit’ and ‘FourierGR1T2’. For real-world scenes, beyond the AI City training dataset [32], the MTMMC dataset [44] is used to increase the amount and diversity of training data. Filtering Small or Invalid 2D Bounding Boxes: Redundant bounding boxes, as illustrated in Fig. 7, may increase the likelihood of false-positive detections and consequently degrade the reliability of the detection results. To mitigate this issue, we filter out irrelevant annotations: bounding boxes with excessively small spatial extent, as well as those covering unseen or invalid objects, are removed from the training annotations. Scene without filtering Scene with filtering Figure 7: Visualization of the 2D bounding box filtering process. The bounding boxes removed from the training annotations are indicated by red ovals. 2D Object Detection Training Step: We adopt RF-DETR [25], a real-time detection transformer, as the object detector. Specifically, we train the 2x-large variant for 200 epochs on input images with a resolution of 19201920 pixels and a batch size of 4; this setting balances memory usage and training stability. 2D Object Detection Inference Step: During inference, the trained RF-DETR model processes images at the same 19201920-pixel resolution with a batch size of 16 to improve throughput. We perform non-maximum suppression at an Intersection over Union (IoU) threshold of 0.5 to reduce redundant overlapping bounding boxes and discard detections with confidence scores below 0.1. Qualitative detection results for the five test scenes are shown in Fig. 8. Figure 8: Qualitative 2D detection results on scenes Warehouse 023–027 of the AI City Challenge 2026 Track 1 test set. Appendix 0.B Re-Identification and Pose Estimation Pose Estimation We use the pretrained ViTPose++ model [46] for pose estimation to extract keypoints from detected person regions (as shown in Fig. 9). ViTPose++ adopts a plain, non-hierarchical vision transformer encoder with a lightweight keypoint decoder, which localizes body keypoints effectively with a simple, scalable architecture. In addition, the model introduces knowledge factorization through task-agnostic and task-specific feed-forward networks, which allows it to handle heterogeneous body keypoint categories across different pose estimation tasks. The pretrained model supplies the pose information used by the subsequent feature extraction, association, and refinement steps. Re-Identification For person re-identification (Re-ID), which relies on deep visual features, we adopt Keypoint Promptable Re-Identification (KPR) [28] for feature extraction. KPR uses pose keypoints to divide the human body into six regions and extracts region-level features from each part. This design emphasizes human-centric cues, reduces background interference, and improves robustness under occlusion. Although KPR supports negative keypoints for modeling occluded or unreliable body regions, only positive keypoints are used during training and inference in our pipeline. Since identity labels differ across scenes in the AI City 2026 Track 1 dataset, the training and validation sets are merged and relabeled into a unified dataset containing 155 identities. The model is trained for 110 epochs, with a total training time of approximately 3 hours and 7 minutes. Figure 9: Qualitative pose-estimation results (ViTPose++ [46] keypoint overlays) on scenes Warehouse 023–027. Appendix 0.C 2D Single-View Tracking The result of 2D single-view tracking is visualized in Fig. 10. In the shown frames, the method tracks multiple objects in a single camera view through occlusions and close object interactions: each object keeps its identity across frames, and the bounding boxes follow the objects throughout the sequence. Figure 10: Qualitative 2D single-view tracking results on scenes Warehouse 023–027; each tracked object carries a persistent identity. Appendix 0.D Point Cloud Construction Since depth maps are unavailable at test time, we generate point-cloud guidance using Depth Anything 3 (DA3) [15] in its pose-conditioned multi-view setting. Specifically, we adopt the updated DA3 Nested Giant-Large 1.1 checkpoint, which integrates any-view geometry with metric-depth scaling. At each synchronized time step, images from all cameras are processed together with the provided intrinsic and extrinsic parameters, K and E. The resulting depth estimates are back-projected into a shared world coordinate system and merged to form a colored point cloud. Fig. 4 presents the multi-view RGB inputs alongside their DA3-estimated point clouds, while Fig. 11 illustrates the corresponding BEV projections for two real-world scenes. (a) Warehouse 026 (b) Warehouse 027 Figure 11: Scene-level BEV visualizations constructed from DA3-estimated point clouds for (a) Warehouse 026 and (b) Warehouse 027.