Paper deep dive
Segment Any Motion with Radar: Robust Multimodal Moving-Object Segmentation and Tracking
Jue Wang, Xuan Wang, Hao Zhou, Ruixiang Zhou, Yixuan Zhou, Tianshuo Yuan, Jieming Ma, Jie Zhang, Fei Luo
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/14/2026, 5:07:26 AM
Summary
The paper introduces RGBTR-Motion, a synchronized fixed-camera benchmark providing RGB, thermal, and radar streams with dense instance masks and temporally consistent identities for surveillance. It also presents SAM-Radar, a segmentation and tracking framework built on SAM 3 that fuses calibrated RGBT features with spatially anchored radar returns. The method uses motion supervision to reject clutter and a radar-conditioned tracker with Hold-Lost memory to maintain identities during occlusions or low visibility, achieving state-of-the-art performance on the new benchmark.
Entities (14)
Relation Signals (13)
SAM-Radar → achievesmetric → IoU
confidence 95% · SAM-Radar attains 0.7027 IoU
SAM-Radar → achievesmetric → IDF1
confidence 95% · raises IDF1 by 0.2857
SAM-Radar → achievesmetric → HOTA
confidence 95% · raises HOTA... by 0.1603
SAM-Radar → achievesmetric → MOTA
confidence 95% · raises MOTA... by 0.2977
SAM-Radar → builton → SAM 3
confidence 95% · SAM-Radar, an RGB, thermal, and radar-based segmentation and tracking framework built on SAM 3.
RGBTR-Motion → containssensordata → RGB
confidence 95% · RGBTR-Motion... pairs RGB, thermal, and radar streams
RGBTR-Motion → containssensordata → Thermal
confidence 95% · RGBTR-Motion... pairs RGB, thermal, and radar streams
RGBTR-Motion → containssensordata → RADAR
confidence 95% · RGBTR-Motion... pairs RGB, thermal, and radar streams
SAM-Radar → evaluatedon → RGBTR-Motion
confidence 95% · All models are fine-tuned on RGBTR-Motion... SAM-Radar (ours)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Moving-object perception must decide which image regions correspond to real motion and keep every instance identified over time. Methods that read motion from appearance, optical flow, or estimated trajectories lose that evidence under poor illumination, adverse weather, reflections, and occlusion. Radar is a natural remedy because it measures radial velocity directly instead of inferring it from photometric correspondence. However, existing benchmarks do not jointly provide radar measurements, dense moving-instance masks, and temporally consistent identities for surveillance. We therefore introduce RGBTR-Motion, a synchronized and calibrated fixed-camera benchmark that pairs RGB, thermal, and radar streams with dense instance masks and temporally consistent identities across diverse surveillance scenes. We also develop SAM-Radar, an RGB, thermal, and radar-based segmentation and tracking framework built on SAM 3. SAM-Radar's radar-aware detector fuses calibrated RGBT features with radar returns that are grounded at their projected image locations, and motion supervision, implemented as foreground classification of those projected returns, teaches the detector to reject clutter without any text prompt. The tracker associates accepted radar returns with individual trajectories and uses them as physical evidence that a visually degraded target remains present. This allows it to bridge short periods of low visibility or occlusion and reconnect a reappearing target to its existing identity instead of starting a new track. SAM-Radar attains 0.7027 IoU and 0.8090 F1-50, and raises MOTA, HOTA, and IDF1 by 0.2977, 0.1603, and 0.2857 over the strongest competing values.
Tags
Links
- Source: https://arxiv.org/abs/2609.08346v1
- Canonical: https://arxiv.org/abs/2609.08346v1
Trouble viewing inline? Open PDF directly →
Full Text
39,155 characters extracted from source content.
Expand or collapse full text
Segment Any Motion with Radar: Robust Multimodal Moving-Object Segmentation and Tracking Jue Wang Xuan Wang Hao Zhou Ruixiang Zhou Yixuan Zhou Tianshuo Yuan Jieming Ma Jie Zhang Fei Luo Abstract Moving-object perception must decide which image regions correspond to real motion and keep every instance identified over time. Methods that read motion from appearance, optical flow, or estimated trajectories lose that evidence under poor illumination, adverse weather, reflections, and occlusion. Radar is a natural remedy because it measures radial velocity directly instead of inferring it from the photometric correspondence. However, existing benchmarks do not jointly provide radar measurements, dense moving-instance masks, and temporally consistent identities for surveillance. We therefore introduce RGBTR-Motion, a synchronized and calibrated fixed-camera benchmark that pairs RGB, thermal, and radar streams with dense instance masks and temporally consistent identities across diverse surveillance scenes. We also develop SAM-Radar, a RGB, thermal and radar-based segmentation and tracking framework built on SAM 3. SAM-Radar’s radar-aware detector fuses calibrated RGBT features with radar returns that are grounded at their projected image locations, and motion supervision, implemented as foreground classification of those projected returns, teaches the detector to reject clutter without any text prompt. The tracker associates accepted radar returns with individual trajectories and uses them as physical evidence that a visually degraded target remains present. This allows it to bridge short periods of low visibility or occlusion and reconnect a reappearing target to its existing identity instead of starting a new track. SAM-Radar attains 0.7027 IoU and 0.8090 F150, and raises MOTA, HOTA, and IDF1 by 0.2977, 0.1603, and 0.2857 over the strongest competing values. 1Harbin Institute of Technology, Shenzhen, China 2Great Bay University, Dongguan, China 3Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China 1 Introduction Figure 1: Overview of the acquisition and SAM-Radar pipeline. A rigid rig records calibrated RGB, thermal, and radar streams. The detector fuses visual features with spatially anchored radar tokens trained by motion supervision, and the tracker combines radar association with Hold-Lost memory to produce temporally consistent masks and identities. Reliable moving-object perception must recover accurate masks and preserve the identities of physical movers over time. In fixed-camera surveillance, weak appearance corrupts boundaries, while crossings and occlusions disrupt identity over long recordings. We target physical agents, predominantly pedestrians, cyclists, and vehicles, rather than photometric or background changes. The task is therefore stricter than generic change detection but broader than closed-vocabulary detection. Most established approaches derive motion from images. Optical-flow methods group short-term displacements or learn masks from flow fields (Xie et al. 2022; Teed and Deng 2020); OCLR (Xie et al. 2022), for example, maintains layered object masks through mutual occlusion. SegAnyMo (Huang et al. 2025) instead classifies long-range visual trajectories and prompts SAM 2 (Ravi et al. 2025) to recover dense masks. Yet their motion evidence still comes from visual correspondence: illumination changes corrupt photometric evidence, thermal crossover reduces contrast, occlusion removes correspondences, and fast or subtle motion corrupts estimated tracks. Camera and radar fusion is well established in autonomous driving (Caesar et al. 2020; Nabati and Qi 2021; Kim et al. 2023), waterways (Guan et al. 2024), and other platform-centric perception. Our contribution is therefore not the first RGBT-radar dataset, but a surveillance-specific viewpoint, motion-centric task, and dense temporal annotation. Mobile-platform datasets emphasize ego-centric geometry and category-level perception; fixed-camera surveillance instead requires long-duration observation from a stationary viewpoint, together with pixel-level masks and persistent identities. Existing multimodal corpora do not directly support this setting. Radar measures motion rather than inferring it from photometric correspondence, so radial velocity can remain informative when appearance is weak. Radar alone is nevertheless sparse and noisy, and its velocity cue vanishes when a target stops or moves tangentially to the line of sight. It therefore complements rather than replaces appearance. When visual evidence becomes unreliable, associated radar measurements remain available. Thermal imagery complements RGB by providing clearer boundaries in low light. The central challenge is to ground sparse radar evidence in the image and assign it consistently to object tracks. We build SAM-Radar (Segment Any Motion with Radar) on SAM 3’s decoupled detector and tracker (Carion et al. 2025) and train them in two separate stages. In Stage 1, the detector combines complementary RGB and thermal features with radar returns anchored to their projected image neighborhoods, allowing it to locate movers using both visual structure and measured motion. Motion supervision teaches the detector to distinguish target returns from multipath reflections and background clutter, reducing clutter-induced predictions. In Stage 2, the detector and radar encoder are frozen, and the tracker matches each return to at most one active track based on spatial, motion, and identity consistency. Confident matches reinforce the corresponding track when visual evidence weakens. To bridge short occlusions, Hold-Lost memory preserves the last reliable visual state while radar continues to support the target and releases the identity only after both visual and radar evidence remain absent. We also organize RGBTR-Motion, a synchronized and calibrated fixed-camera benchmark with 107 sequences and 8,537 annotated frames across daytime, nighttime, rainy, and indoor conditions. It provides frame-level masks and persistent identities, with an acquisition-grouped train/test split that prevents temporal leakage. Representative motion-segmentation and multimodal baselines are adapted to the same data and evaluation protocol. Our contributions are threefold: • We construct RGBTR-Motion, a synchronized and calibrated fixed-camera surveillance benchmark with dense masks and persistent identities, addressing a task and annotation setting underrepresented by existing automotive-oriented tri-modal data. • We introduce SAM-Radar, whose radar-aware detector combines calibrated RGBT features with spatially anchored radar tokens, while tracker uses radar association, confidence-gated radar residual injection, and Hold-Lost memory to preserve identities through temporary appearance failures. • We develop a radar-aware two-stage training strategy. Stage 1 trains the detector to discover and segment moving objects from the aligned RGBT-radar inputs. Stage 2 freezes the detector and trains the tracker to preserve object identities over time, especially when visual evidence weakens. 2 Related Work Moving-Object Segmentation Classical motion segmentation groups pixels or trajectories by geometric consistency (Brox and Malik 2010; Ochs et al. 2014). In fixed-camera surveillance, background subtraction remains vulnerable to nuisance appearance changes (Stauffer and Grimson 1999; Zivkovic 2004; Barnich and Van Droogenbroeck 2011; Bouwmans 2014), as systematically benchmarked by CDnet (Wang et al. 2014). Modern systems instead use optical flow (Teed and Deng 2020); OCLR (Xie et al. 2022), for example, predicts depth-ordered object layers and learns temporal shape consistency from synthetic compositions. Instance-level video tasks such as MOTS (Voigtlaender et al. 2019) and video instance segmentation (Yang et al. 2019) couple masks with identities but assume a closed vocabulary. SAM (Kirillov et al. 2023) established promptable segmentation, SAM 2 (Ravi et al. 2025) introduced streaming video memory, and SAM 3 (Carion et al. 2025) unified open-vocabulary detection and tracking. SegAnyMo (Huang et al. 2025) classifies long-range point tracks and prompts SAM 2 for dense masks. Motion-oriented uses of these models still rely on visual correspondence and fail when that evidence disappears. SAM-Radar instead uses measured radar motion, spatially anchored radar tokens, and Hold-Lost memory. Vision-Radar Perception RGB supplies texture and color, while thermal imagery complements it in low light (Hwang et al. 2015; Li et al. 2019; Li et al. 2022). Condition-aware fusion improves robustness when either stream degrades (Brödermann et al. 2025), but both remain appearance based. Radar adds independent range and radial velocity, motivating camera-radar systems for driving and corresponding adverse-weather and 4D-radar datasets (Caesar et al. 2020; Nabati and Qi 2021; Kim et al. 2023; Sheeny et al. 2021; Palffy et al. 2022; Paek et al. 2022). ASY-VRNet (Guan et al. 2024) combines visual features and 4D radar maps for mobile waterway perception. RADCI (Yu et al. 2025) is a fixed-tripod RGBT-radar benchmark for detection and tracking, with 2D boxes and target IDs. These works establish the value of multimodal fusion, but neither provides motion-centric dense instance masks with sequence-level identity annotations. We instead align thermal features and sparse returns to the visible image, using radar as track-specific motion evidence. 3 The RGBTR-Motion Benchmark Figure 2: Composition of the benchmark RGBTR-Motion. Example panels show calibrated RGBT fusion, sequence-local moving-instance masks and identities, and projected radar returns for daytime, nighttime, rainy, and indoor monitoring conditions. Data Collection RGBTR-Motion is designed specifically for motion perception from stationary surveillance viewpoints. Data are acquired with a rigid tri-modal rig comprising a ×7201280\!×\!720 visible-light camera, a ×512640\!×\!512 long-wave infrared camera, and a ZF 4D millimeter-wave radar with 12 transmit and 16 receive channels. Both cameras and radar record at 10 fps. All three sensors are mounted on a common plate and remain fixed throughout each acquisition, providing consistent geometry across daytime, nighttime, rainy, and indoor recordings. Complementing the aligned visual streams, the radar records each return’s 3D position, range-related geometry, radial velocity, signal-to-noise ratio (SNR), and timestamp. The benchmark targets the sustained, scene-traversing movers described in Section 1, chiefly pedestrians, cyclists, and vehicles, and preserves identity through surveillance events such as entry, target crossing, occlusion, and exit. The audited benchmark contains 107 sequences, 8,537 annotated frames, 15,475 instance masks, and 274 sequence-local identities. Table 1 focuses on RADCI because it provides the closest sensor and acquisition setting. RADCI reports seven road and square scenarios recorded by day and night. RGBTR-Motion expands the coverage to outdoor and indoor surveillance spanning daytime, nighttime, and rain. More importantly, it goes beyond category-level box annotation by linking motion-centric instance masks to sequence-local identities, enabling pixel-level segmentation and tracking through entry, crossing, occlusion, and exit. Table 1: Comparison with RADCI (Yu et al. 2025), the closest synchronized RGBT-radar benchmark. The two share a sensor configuration; RGBTR-Motion differs in condition coverage, annotation granularity, and how a target is defined. RADCI RGBTR-Motion (ours) Modalities RGB, thermal, radar RGB, thermal, radar Environment road, square road, gym, square, indoor Conditions Day, night Day, night, rain Sequences 94 107 Annotation 2D boxes with IDs Instance masks with IDs Target Predefined categories Any sustained mover Task Detection, tracking Segmentation, tracking Figure 2 visualizes this composition together with calibrated RGBT-radar surveillance examples. Figure 3: For each projected radar return, the radar-aware detector combines local RGBT context, image position, and radar attributes to form a spatially anchored token. Motion supervision teaches it to distinguish target-associated returns from clutter, and the segmentation decoder combines these sparse tokens with dense visual features to predict moving instances. Temporal Synchronization and Spatial Calibration All sensor streams are mapped to the visible-camera timeline. Visible and infrared frames are undistorted, and infrared frames are warped into visible-image coordinates using the calibrated cross-camera transformation. For a radar point r∈ℝ3x^r ^3, calibrated rotation rvR_rv and translation rvt_rv transform it into the visible-camera coordinate system: v=rvr+rv.x^v=R_rvx^r+t_rv. (1) The visible camera matrix and distortion model then project vx^v to pixel coordinates (u,v)(u,v). Returns outside the image plane are discarded. Video frame t is paired with the t-th radar timestamp and groups returns in a small temporal neighborhood. We retain reliable moving points together with high-SNR static or slow points, then convert projected coordinates into the undistorted visible-image system, producing ℛt=(uj,vj,dj,d˙j,sj)j=1Nt,R_t=\(u_j,v_j,d_j, d_j,s_j)\_j=1^N_t, (2) where djd_j, d˙j d_j, and sjs_j denote range, radial velocity, and SNR, respectively. Annotations and Tasks Each visible frame is annotated in COCO-compatible format with instance masks, boxes, categories, and sequence-local persistent identities. The labels are motion-centric rather than restricted to a predefined surveillance object taxonomy. They support two tasks under a common annotation set: frame-level moving-object instance segmentation and video-level segmentation and tracking. The annotation policy covers temporary full occlusion, targets leaving the field of view, and stationary intervals of otherwise moving objects. 4 Method Overview Given synchronized RGB frames IR1:TI^R_1:T, aligned thermal frames IT1:TI^T_1:T, and projected radar sets ℛ1:TR_1:T, the model predicts instance masks and persistent identities for physical movers. We retain the decoupled detector and memory tracker of SAM 3 (Carion et al. 2025) and train them in two stages. The detector first learns frame-level discovery and segmentation from all three modalities. Its weights and radar encoder are then frozen while the tracker learns radar association, feature updates, and Hold-Lost memory. This separation prevents temporal training from changing the learned object discovery process. Figures 3 and 4 show the data flow. Radar-Aware Multimodal Detector Calibrated RGBT fusion Directly stacking thermal and RGB images at the input assumes compatible channel statistics and exposes the pretrained RGB representation to thermal noise. We instead process the aligned images with separate vision trunks initialized from the same pretrained encoder. Let FRF^R and FTF^T denote their features, and let ARA_R and ATA_T be lightweight adapters. Their detector-level fusion is F¯m F^m =Am(Fm),m∈R,T, =A_m(F^m), m∈\R,T\, (3) FRT F^RT =FR+Φ(F¯R,F¯T). =F^R+ ( F^R, F^T). Here, Φ is the learned fusion module. It predicts independent sigmoid channel weights from globally pooled adapted features, combines the weighted and spatial-fusion branches with a learned scalar initialized to 0.50.5, and returns a residual. Consequently, FRTF^RT retains a direct RGB path while incorporating thermal evidence. Figure 4: The radar-conditioned tracker receives the current RGBT observations and projected radar returns. Radar association combines projected position, radar attributes, and descriptor similarity to assign each return to at most one active track. When visual evidence weakens but radar continues to support the target, Hold-Lost memory preserves the last reliable visual state. Spatially anchored radar tokens A direct approach is to encode only the projected coordinates and physical attributes of each radar return with an MLP. This retains range and radial velocity but omits the visual structure at the projected location. For return rj=(uj,vj,dj,d˙j,sj)r_j=(u_j,v_j,d_j, d_j,s_j), let pjp_j encode position (uj,vj)(u_j,v_j), let ℓj _j be the locally pooled RGBT feature, and let fjattrf_j^attr encode the normalized attributes (dj,d˙j,sj)(d_j, d_j,s_j). We construct its token as bj b_j =pj+Wℓℓj, =p_j+W_ _j, (4) gj g_j =σ(Wgfjattr), =σ(W_gf_j^attr), zj z_j =LN(bj+gj⊙fjattr+erad+ejfg). =LN(b_j+g_j f_j^attr+e_rad+e_j^fg). WℓW_ and WgW_g are learned projections, erade_rad identifies the radar modality, and ejfge_j^fg is a soft foreground embedding predicted for the return. The gate gjg_j controls the attribute residual, while layer normalization produces the detector token zjz_j. This construction links each sparse physical measurement to its local image context. Motion supervision Standard mask and box losses supervise the final object predictions but do not indicate which radar returns correspond to targets. We therefore assign label yj∈0,1y_j∈\0,1\ to each projected return and predict its foreground probability y^j y_j. The corresponding focal objective (Lin et al. 2017) and total detector objective are qj q_j =yjy^j+(1−yj)(1−y^j), =y_j y_j+(1-y_j)(1- y_j), (5) ℒrad _rad =−1Nt∑j=1Ntβyj(1−qj)γlogqj, =- 1N_t _j=1^N_t _y_j(1-q_j)^γ q_j, ℒdet _det =ℒSAM3+λradℒrad. =L_SAM3+ _radL_rad. Here, NtN_t is the number of returns, βyj _y_j is the class weight, and γ is the focal exponent. ℒSAM3L_SAM3 collects the classification, box, generalized-IoU (Rezatofighi et al. 2019), and mask losses, while λrad _rad weights motion supervision. Radar-Conditioned Tracker SAM 3 uses separate feature pyramids for detection and tracking, so detector-side fusion does not expose the high-resolution tracking path to thermal input. We therefore fuse thermal features at every tracker scale while freezing the spatial encoder, detector, and memory writer; stage two updates only temporal multimodal components. Radar association A hard previous-mask gate cannot recover after drift, whereas independent matching can assign one return to several tracks. We instead multiply four pairwise terms: spatial support SijtS_ij^t, SNR reliability QjtQ_j^t, range-velocity continuity DijtD_ij^t, and descriptor similarity CijtC_ij^t. They test whether a return is spatially close, reliable, consistent with the track’s range-velocity history, and similar to its accumulated radar descriptor: wijt w_ij^t =Sijt⋅Qjt⋅Dijt⋅Cijt, =S_ij^t· Q_j^t· D_ij^t· C_ij^t, (6) aijt a_ij^t =wijtw∅jt+∑k∈twkjt, = w_ij^tw_ j^t+ _k _tw_kj^t, ij∗ i_j^* =argmaxi∈taijt, = _i _ta_ij^t, j j ↦ij∗ifaij∗jt≥τown. i_j^* a_i_j^*j^t≥ _own. Here, tT_t is the active-track set, w∅jt=0.25w_ j^t=0.25 is the fixed null evidence for clutter, and τown=0.35 _own=0.35 is the ownership threshold. Each return is assigned to at most one track and remains unassigned when the null option or confidence test rejects it. Once a track has accepted a radar measurement, we call it radar initialized. Its spatial prior is then floored at 0.050.05 outside the previous visual box so that a displaced return can still compete for association. Accepted returns are pooled into a current measurement and a track-specific descriptor. Radar feature updates Only observed returns update the persistent radar state. If a measurement is missing, the last descriptor and radial velocity are retained for at most four frames with decaying confidence, and range is extrapolated under constant radial velocity. Rendering this predicted state as a spatial return would reuse an old image location even though range extrapolation does not determine image-plane motion. We therefore allow only current measurements to populate the sparse spatial residual; propagated descriptors can affect the global object representation and pointer, but not the spatial radar map. Let MitM_i^t be the current sparse radar map, hith_i^t the current or propagated track descriptor, citc_i^t its confidence, FitF_i^t the visual tracking feature, and oito_i^t the object pointer. Radar is injected after memory attention and before mask decoding as ΔFit F_i^t =tanh(gs)As(Mit)+tanh(gp)Ap(hit), = (g_s)A_s(M_i^t)+ (g_p)A_p(h_i^t), (7) F~it F_i^t =Fit+citΔFit, =F_i^t+c_i^t F_i^t, o~it o_i^t =oit+cittanh(gp)Ap(hit). =o_i^t+c_i^t (g_p)A_p(h_i^t). AsA_s and ApA_p are learned residual projections, and gsg_s and gpg_p are scalar gates. Hold-Lost memory Immediately discarding a visually weak track loses objects during short occlusions, whereas writing every low-confidence prediction into memory can corrupt later propagation. Hold-Lost memory uses three states to separate these cases. TRACKED denotes a visually reliable prediction and refreshes the most recent reliable frame. If visual confidence becomes unreliable while a current radar measurement remains associated, the track enters HOLD and keeps that reliable frame in the seven-slot memory selection window. Visual recovery returns the track to TRACKED. If neither vision nor current radar supports the track for four consecutive frames, it becomes LOST; its mask and score are suppressed and its active anchor is released. 5 Experiments Table 2: Main comparison on the RGBTR-Motion. IoU and F150 evaluate foreground and instance moving-object segmentation. MOTA, MOTP, HOTA, and IDF1 evaluate tracking. All models are fine-tuned on RGBTR-Motion; ∗ denotes methods additionally adapted to accept aligned RGB, thermal, and radar inputs. Best results are in bold and second-best results are underlined. Method Modalities IoU ↑ F150 ↑ MOTA ↑ MOTP ↑ HOTA ↑ IDF1 ↑ OCLR Hybrid (Xie et al. 2022) RGB 0.4669 0.5885 0.2603 0.7349 0.3583 0.3449 SegAnyMo DINO+SAM (Huang et al. 2025) RGB 0.6103 0.6070 0.3074 0.7568 0.4166 0.4756 ASY-VRNet∗ (Guan et al. 2024) RGB+T+R 0.4884 0.4805 0.2990 0.7716 0.3390 0.2808 CAFuser∗ (Brödermann et al. 2025) RGB+T+R 0.6489 0.6111 0.3188 0.7650 0.4308 0.3750 RADCI-RCINet∗ (Yu et al. 2025) RGB+T+R 0.6574 0.6243 0.3259 0.7666 0.4415 0.3931 SAM-Radar (ours) RGB+T+R 0.7027 0.8090 0.6236 0.7559 0.6018 0.7613 Experimental Setup Baselines We evaluated five representative baselines using the same acquisition-grouped split and annotation protocol. OCLR Hybrid (Xie et al. 2022) and SegAnyMo DINO+SAM (Huang et al. 2025) retained their native RGB motion inputs and predicted class-agnostic moving-instance masks. For ASY-VRNet, separately encoded RGB and thermal features formed the visual input to its original visual-radar fusion, and projected returns were rasterized in the camera plane. CAFuser received RGB, thermal, and rasterized radar through modality-specific adapters while retaining its RGB-derived condition token. RADCI-RCINet retained its RGBT-radar concatenation and attention front end, followed by a class-agnostic mask decoder (Guan et al. 2024; Brödermann et al. 2025; Yu et al. 2025). We trained the adapted mask outputs with the same binary moving-instance targets and applied a common online Hungarian linker based on mask overlap and appearance similarity to methods without native identities. All compared methods used the same inputs and annotations. Implementation details Training was performed in two stages. The detector was trained for 60 epochs with AdamW (Loshchilov and Hutter 2019), an empty text prompt, and randomized static negatives with per-instance probability 0.50.5 and box scale 1.51.5. The tracker was trained for 1,000 updates on two RTX A6000 GPUs using bfloat16 and 32-frame clips split into four-frame graphs, with at most four identities per clip. The learning rates were 10−610^-6 for the tracker and ×10−52\!×\!10^-5 for the radar adapter, and radar dropout was 0.150.15. The detector and radar encoder remained frozen during tracker training. Metrics For segmentation, the IoU in Table 2 is the macro-average of per-frame foreground overlap after merging all instances. F150 evaluates instance recovery via Hungarian matching with an IoU≥0.5≥ 0.5 correctness threshold. For tracking, MOTA penalizes missed targets, false positives, and identity switches, whereas MOTP measures the localization quality of successfully matched predictions (Bernardin and Stiefelhagen 2008). HOTA balances detection accuracy with association accuracy (Luiten et al. 2021), while IDF1 is the identity-level F1 score and reflects how consistently detections retain the correct identity over time (Ristani et al. 2016). Higher is better for all reported metrics. Main Results Table 2 reveals two gain regimes. Against the strongest competing score in each column, SAM-Radar raises IoU from 0.6574 to 0.7027, an absolute gain of 0.0453, whereas F150 rises from 0.6243 to 0.8090, a gain of 0.1847. The larger F150 gain shows that the segmentation advantage is more pronounced for recovering matchable moving instances than for improving frame-averaged foreground overlap. The contrast is larger for identity: MOTA, HOTA, and IDF1 improve by 0.2977, 0.1603, and 0.2857 over their respective strongest competing values. IoU is computed independently per frame, so preserving a track through an occlusion need not alter mask overlap on frames where the object is already visible. IDF1 instead measures correctly identified detections over a sequence, so fragmentation or reassignment can reduce the score across multiple frames. Radar association is designed to resolve ambiguous ownership, while Hold-Lost memory protects the last reliable state during radar-supported visual degradation. The fact that both components improve IDF1 in Table 4 is consistent with this continuity-based explanation for the larger identity gain. Consistently, SAM-Radar obtains 0.7559 MOTP, below ASY-VRNet at 0.7716, showing that the main benefit is temporal continuity rather than tighter localization of already matched masks. The multimodal baselines also do not uniformly outperform the RGB methods, showing that additional sensor inputs alone do not guarantee a gain on this benchmark. Figure 5: Qualitative comparison under multi-distance targets, low illumination, and occlusion. Frames progress from left to right within each condition; rows compare the ground truth, five baselines, and SAM-Radar. Ablation Studies Detector components Table 3 reports the detector ablations. Adding motion supervision to radar attributes raises Pixel IoU from 0.4677 to 0.5850, Pixel Dice from 0.6373 to 0.7382, and F150 from 0.5599 to 0.7927. The largest absolute change is in F150, showing that return-level foreground classification improves instance recovery under the matching criterion, while the simultaneous pixel-metric gains show better foreground estimation. The full detector achieves the best overall values of 0.7540, 0.8598, and 0.8389. Table 3: Detector configurations and results. All rows include RGB; T, R, and M denote thermal fusion, radar attributes, and motion supervision. P-IoU and P-Dice are dataset-level pixel metrics. Variant T R M P-IoU P-Dice F150 RGB only - - - 0.4064 0.5779 0.6517 Thermal fusion ✓ - - 0.4128 0.5844 0.7581 Radar attributes - ✓ - 0.4677 0.6373 0.5599 Motion supervision - ✓ ✓ 0.5850 0.7382 0.7927 Full detector ✓ ✓ ✓ 0.7540 0.8598 0.8389 Tracker components Table 4 reports complementary gains from the two tracker mechanisms. Relative to the visual tracker, radar association raises MOTA/HOTA/IDF1 by 0.1291/0.0530/0.0903, while Hold-Lost memory raises them by 0.1190/0.0444/0.0755. Radar association is the stronger individual component, consistent with competitive ownership resolving which track should receive each return. Hold-Lost memory still raises IDF1 from 0.6233 to 0.6988, consistent with its role in protecting the stored identity state when visual evidence temporarily degrades. Combining the mechanisms gives the largest gains, 0.2229/0.1053/0.1380, and the best value for every reported metric. This result is consistent with their distinct roles: radar association selects the recipient track, whereas Hold-Lost memory decides whether an uncertain visual update may overwrite its state. Table 4: Tracker ablation. Radar association and Hold-Lost memory are each added to the visual tracker. Variant MOTA HOTA IDF1 SAM 3 visual tracker 0.4007 0.4965 0.6233 SAM 3 + radar association 0.5298 0.5495 0.7136 SAM 3 + Hold-Lost memory 0.5197 0.5409 0.6988 Full tracker 0.6236 0.6018 0.7613 Qualitative Results Figure 5 complements the aggregate metrics with three surveillance conditions. In the multi-distance sequence, SAM-Radar follows the annotated movers with compact masks while several baselines produce fragmented or extraneous regions. Under low illumination, it recovers a coherent silhouette in Frames 43 and 52 and correctly produces no mask in Frame 62. In the occlusion sequence, it recovers the same track identity after the obstruction in Frames 10 and 25, whereas several competing rows miss the target or assign a different identity after it reappears. These examples mirror the two quantitative gain regimes: instance recovery improves F150, while continuity through degraded visual evidence improves IDF1 without requiring a comparable increase in framewise IoU. 6 Conclusion We introduced RGBTR-Motion, an RGBT-radar benchmark designed for motion segmentation and tracking under various surveillance conditions, together with SAM-Radar, a framework that uses radar measurements to strengthen object discovery and temporal association. In the detector, a return’s projected location identifies the image region it can support, while its range, radial velocity, and SNR provide physical motion and reliability cues beyond local RGBT appearance. Motion supervision then teaches the detector to distinguish returns associated with actual movers from multipath reflections and static clutter, allowing these radar attributes to contribute to more reliable instance predictions. During tracking, radar association connects each measurement to an individual trajectory, while Hold-Lost memory preserves the last reliable visual state when radar confirms that a visually degraded target remains present. The experiments consistently show that these designs improve instance recovery and identity continuity. These findings indicate that radar is most effective when its sparse measurements are explicitly grounded, assigned to individual tracks, and coupled to controlled visual memory. References Barnich and Van Droogenbroeck (2011) O. Barnich and M. Van Droogenbroeck ViBe: a universal background subtraction algorithm for video sequences. IEEE Transactions on Image Processing 20 (6), p. 1709–1724. Cited by: §2. Bernardin and Stiefelhagen (2008) K. Bernardin and R. Stiefelhagen Evaluating multiple object tracking performance: the CLEAR MOT metrics. EURASIP Journal on Image and Video Processing 2008, p. 246309. Cited by: §5. Bouwmans (2014) T. Bouwmans Traditional and recent approaches in background modeling for foreground detection: an overview. Computer Science Review 11–12, p. 31–66. Cited by: §2. Brödermann et al. (2025) T. Brödermann, C. Sakaridis, Y. Fu, and L. Van Gool CAFuser: condition-aware multimodal fusion for robust semantic perception of driving scenes. IEEE Robotics and Automation Letters. Cited by: §2, §5, Table 2. Brox and Malik (2010) T. Brox and J. Malik Object segmentation by long term analysis of point trajectories. In European Conference on Computer Vision, Cited by: §2. Caesar et al. (2020) H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom nuScenes: a multimodal dataset for autonomous driving. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §1, §2. Carion et al. (2025) N. Carion, L. Gustafson, Y. Hu, S. Debnath, R. Hu, D. Suris, C. Ryali, K. V. Alwala, H. Khedr, A. Huang, et al. SAM 3: segment anything with concepts. External Links: 2511.16719 Cited by: §1, §2, §4. Guan et al. (2024) R. Guan, S. Yao, K. L. Man, X. Zhu, Y. Yue, J. Smith, E. G. Lim, and Y. Yue ASY-VRNet: waterway panoptic driving perception model based on asymmetric fair fusion of vision and 4D mmwave radar. External Links: 2308.10287 Cited by: §1, §2, §5, Table 2. Huang et al. (2025) N. Huang, W. Zheng, C. Xu, K. Keutzer, S. Zhang, A. Kanazawa, and Q. Wang Segment any motion in videos. External Links: 2503.22268 Cited by: §1, §2, §5, Table 2. Hwang et al. (2015) S. Hwang, J. Park, N. Kim, Y. Choi, and I. S. Kweon Multispectral pedestrian detection: benchmark dataset and baseline. In IEEE Conference on Computer Vision and Pattern Recognition, Cited by: §2. Kim et al. (2023) Y. Kim, S. Kim, J. W. Choi, and D. Kum CRAFT: camera-radar 3D object detection with spatio-contextual fusion transformer. In AAAI Conference on Artificial Intelligence, Cited by: §1, §2. Kirillov et al. (2023) A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W. Lo, P. Dollár, and R. Girshick Segment anything. In IEEE/CVF International Conference on Computer Vision, Cited by: §2. Li et al. (2019) C. Li, X. Liang, Y. Lu, N. Zhao, and J. Tang RGB-T object tracking: benchmark and baseline. Pattern Recognition 96, p. 106977. Cited by: §2. Li et al. (2022) C. Li, W. Xue, Y. Jia, Z. Qu, B. Luo, J. Tang, and D. Sun LasHeR: a large-scale high-diversity benchmark for RGBT tracking. IEEE Transactions on Image Processing 31, p. 392–404. Cited by: §2. Lin et al. (2017) T. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár Focal loss for dense object detection. In IEEE International Conference on Computer Vision, Cited by: §4. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In International Conference on Learning Representations, Cited by: §5. Luiten et al. (2021) J. Luiten, A. Osep, P. Dendorfer, P. Torr, A. Geiger, L. Leal-Taixé, and B. Leibe HOTA: a higher order metric for evaluating multi-object tracking. International Journal of Computer Vision 129 (2), p. 548–578. Cited by: §5. Nabati and Qi (2021) R. Nabati and H. Qi CenterFusion: center-based radar and camera fusion for 3D object detection. In IEEE/CVF Winter Conference on Applications of Computer Vision, Cited by: §1, §2. Ochs et al. (2014) P. Ochs, J. Malik, and T. Brox Segmentation of moving objects by long term video analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 36 (6), p. 1187–1200. Cited by: §2. Paek et al. (2022) D. Paek, S. Kong, and K. T. Wijaya K-Radar: 4D radar object detection for autonomous driving in various weather conditions. In Advances in Neural Information Processing Systems Datasets and Benchmarks Track, Cited by: §2. Palffy et al. (2022) A. Palffy, E. Pool, S. Baratam, J. F. P. Kooij, and D. M. Gavrila Multi-class road user detection with 3+1D radar in the View-of-Delft dataset. IEEE Robotics and Automation Letters 7 (2), p. 4961–4968. Cited by: §2. Ravi et al. (2025) N. Ravi, V. Gabeur, Y. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, et al. SAM 2: segment anything in images and videos. In International Conference on Learning Representations, Cited by: §1, §2. Rezatofighi et al. (2019) H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, and S. Savarese Generalized intersection over union: a metric and a loss for bounding box regression. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §4. Ristani et al. (2016) E. Ristani, F. Solera, R. Zou, R. Cucchiara, and C. Tomasi Performance measures and a data set for multi-target, multi-camera tracking. In European Conference on Computer Vision Workshops, Cited by: §5. Sheeny et al. (2021) M. Sheeny, E. De Pellegrin, S. Mukherjee, A. Ahrabian, S. Wang, and A. Wallace RADIATE: a radar dataset for automotive perception in bad weather. In IEEE International Conference on Robotics and Automation, Cited by: §2. Stauffer and Grimson (1999) C. Stauffer and W. E. L. Grimson Adaptive background mixture models for real-time tracking. In IEEE Conference on Computer Vision and Pattern Recognition, Cited by: §2. Teed and Deng (2020) Z. Teed and J. Deng RAFT: recurrent all-pairs field transforms for optical flow. In European Conference on Computer Vision, p. 402–419. Cited by: §1, §2. Voigtlaender et al. (2019) P. Voigtlaender, M. Krause, A. Osep, J. Luiten, B. B. G. Sekar, A. Geiger, and B. Leibe MOTS: multi-object tracking and segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, Cited by: §2. Wang et al. (2014) Y. Wang, P. Jodoin, F. Porikli, J. Konrad, Y. Benezeth, and P. Ishwar CDnet 2014: an expanded change detection benchmark dataset. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, Cited by: §2. Xie et al. (2022) J. Xie, W. Xie, and A. Zisserman Segmenting moving objects via an object-centric layered representation. In Advances in Neural Information Processing Systems, Vol. 35. Cited by: §1, §2, §5, Table 2. Yang et al. (2019) L. Yang, Y. Fan, and N. Xu Video instance segmentation. In IEEE/CVF International Conference on Computer Vision, p. 5187–5196. Cited by: §2. Yu et al. (2025) H. Yu, R. Zhang, H. Sun, Z. Cao, B. Yang, J. Zhang, and G. Liu RADCI: a synchronized radar-RGBT object detecting-tracking dataset and a benchmark. In IEEE International Conference on Acoustics, Speech and Signal Processing, External Links: Document Cited by: §2, Table 1, §5, Table 2. Zivkovic (2004) Z. Zivkovic Improved adaptive gaussian mixture model for background subtraction. In International Conference on Pattern Recognition, Cited by: §2.