Paper deep dive
CDIS: Cross-Dimensional Class-Agnostic 3D Instance Segmentation via 2D Mask Tracking and 3D-2D Projection Merging
Juno Kim, Hye-Jung Yoon, Yesol Park, Byoung-Tak Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/21/2026, 5:40:43 AM
Summary
The paper introduces CDIS, a zero-shot, class-agnostic 3D instance segmentation framework that improves upon existing projection-based methods by establishing a feedback loop between 2D instance tracking and 3D superpoint association. By tracking 2D masks across frames and associating them with geometrically coherent 3D superpoints, CDIS produces globally consistent 3D instance labels without specific 3D training. Experiments on ScanNet200 and ScanNet++ benchmarks demonstrate that CDIS achieves higher accuracy and consistency than state-of-the-art zero-shot methods, making it suitable for robotic systems in unknown environments.
Entities (10)
Relation Signals (9)
CDIS → evaluatedon → ScanNet200
confidence 98% · We evaluated the robustness of our method on two benchmark datasets: ScanNet200 and ScanNet++.
CDIS → evaluatedon → ScanNet++
confidence 98% · We evaluated the robustness of our method on two benchmark datasets: ScanNet200 and ScanNet++.
CDIS → enables → Class-agnostic 3D Instance Segmentation
confidence 95% · Class-agnostic 3D instance segmentation is critical for robotic systems... We introduce... CDIS
CDIS → uses → 3D Superpoints
confidence 95% · These 2D tracks are then aligned with precomputed 3D superpoints, which serve as geometrically coherent anchors
CDIS → uses → 2D Instance Tracking
confidence 95% · CDIS first establishes temporally consistent 2D instance tracks from sequential RGB frames via geometric warping and multi-frame matching.
CDIS → outperforms → Open3DIS
confidence 90% · Our CropFormer-based model achieves an AP of 33.2, surpassing OV-MAP and Open3DIS
CDIS → outperforms → SAM3D
confidence 90% · CDIS demonstrates improved segmentation accuracy... compared to state-of-the-art zero-shot methods
CDIS → usesasinput → SAM
confidence 90% · We experiment with both SAM and CropFormer as the 2D mask generator.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Class-agnostic 3D instance segmentation is critical for robotic systems operating in unknown environments, enabling perception of previously unseen objects for reliable manipulation and navigation. Existing approaches typically project per-frame 2D instance masks into 3D and merge them, which often breaks object identities across time and yields fragmented 3D instances. We introduce Cross-Dimensional Class-Agnostic 3D Instance Segmentation (CDIS), a zero-shot framework that explicitly tracks 2D instance masks across frames and associates them with 3D superpoints, creating a feedback loop between 2D and 3D. This cross-dimensional reasoning links temporally stable 2D tracks with spatially coherent 3D regions, producing globally consistent 3D instance labels without any 3D-specific training. Experiments on benchmark datasets demonstrate that CDIS achieves higher accuracy and consistency than state-of-the-art zero-shot methods, while remaining efficient and scalable to diverse real-world environments.
Tags
Links
- Source: https://arxiv.org/abs/2607.17778v1
- Canonical: https://arxiv.org/abs/2607.17778v1
Trouble viewing inline? Open PDF directly →
Full Text
31,200 characters extracted from source content.
Expand or collapse full text
CDIS : Cross-Dimensional Class-Agnostic 3D Instance Segmentation via 2D Mask Tracking and 3D-2D Projection Merging Juno Kim1∗ Hye-Jung Yoon1∗ Yesol Park1∗ Byoung-Tak Zhang1 This work was partly supported by the IITP (RS-2021-I212068-AIHub/10%, RS-2021-I211343-GSAI/15%, RS-2022-I220951-LBA/15%, RS-2022-I220953-PICA/20%), NRF (RS-2024-00353991-SPARC/20%, RS-2023-00274280-HEI/10%), and KEIT (RS-2024-00423940/10%) grant funded by the Korean government.*Authors have equal contributions1Interdisciplinary Program in AI, Seoul National University Abstract Class-agnostic 3D instance segmentation is critical for robotic systems operating in unknown environments, enabling perception of previously unseen objects for reliable manipulation and navigation. Existing approaches typically project per-frame 2D instance masks into 3D and merge them, which often breaks object identities across time and yields fragmented 3D instances. We introduce Cross-Dimensional Class-Agnostic 3D Instance Segmentation (CDIS), a zero-shot framework that explicitly tracks 2D instance masks across frames and associates them with 3D superpoints, creating a feedback loop between 2D and 3D. This cross-dimensional reasoning links temporally stable 2D tracks with spatially coherent 3D regions, producing globally consistent 3D instance labels without any 3D-specific training. Experiments on benchmark datasets demonstrate that CDIS achieves higher accuracy and consistency than state-of-the-art zero-shot methods, while remaining efficient and scalable to diverse real-world environments. I Introduction Understanding the 3D environment is essential for robotic manipulation and navigation in real-world scenarios. Class-agnostic 3D instance segmentation is particularly crucial when robots encounter previously unseen objects, enabling robust manipulation, path planning, and obstacle avoidance without requiring prior knowledge of object categories. Traditional approaches to 3D instance segmentation rely heavily on large, annotated 3D datasets [19, 27, 1, 22]. This dependency severely limits generalization to novel objects and reduces applicability in real-world robotic environments [21, 23, 11]. To overcome these limitations, recent research has shifted toward leveraging 2D instance segmentation models within 3D perception pipelines [13, 20, 24, 25, 26, 5, 9, 28, 16, 7, 8]. These projection-based methods capitalize on the generalization capabilities of 2D models trained on large-scale annotated datasets due to advancements in large 2D segmentation models [3, 2, 10, 18, 17]. For instance, models like SAM3D [26] project 2D segmentation masks from the Segment Anything Model (SAM) [10] and CropFormer [18] into 3D space, then iteratively merge the projected masks based on geometric overlap. However, projection-based approaches still face significant challenges in real-world robotic scenarios. Depth data often suffer from noise, missing regions, or calibration imperfections, especially for reflective or distant surfaces, thereby amplifying segmentation errors in individual frames. When these frame-level errors are directly projected into 3D and merged, they frequently break object identities across time and produce fragmented 3D instances. Although some methods attempt to refine depth estimates using reconstructed point clouds [5], these pipelines still struggle to maintain consistent instance labels across frames. Figure 1: Overview of CDIS. Top image illustrates a reconstructed real-world 3D scene generated from sequentially captured RGB-D frames at timestamps t1,t2,t3t_1,t_2,t_3. The bottom images show the progression of 2D instance segmentation results over time, demonstrating our method, which improves segmentation quality across frames through mask tracking. CDIS integrates segmentation cues from both 2D and 3D spaces to enhance class-agnostic 3D instance segmentation and improve alignment across multiple views. Figure 2: Overall Framework of CDIS. The proposed pipeline for zero-shot, class-agnostic 3D instance segmentation. Posed RGB-D frames are input into a 2D mask predictor to generate instance masks for each frame, which are tracked over time using depth-based projection, frame warping, and 2D IoU matching. Tracked masks are then associated with pre-computed 3D superpoints, enabling spatio-temporal merging of instances across frames based on geometric consistency. Finally, duplicate superpoint assignments are resolved through overlap and temporal co-occurrence analysis, resulting in a unified and consistent 3D instance segmentation. In this paper, we present Cross-Dimensional Class-Agnostic 3D Instance Segmentation (CDIS), a zero-shot framework designed to overcome the limitations of projection-based pipelines by jointly reasoning across 2D and 3D domains, as illustrated in Fig. 1. CDIS first establishes temporally consistent 2D instance tracks from sequential RGB frames via geometric warping and multi-frame matching. This reduces short-term segmentation errors that often arise when processing frames independently. These 2D tracks are then aligned with precomputed 3D superpoints, which serve as geometrically coherent anchors to consolidate fragmented predictions. Through iterative refinement that alternates between 2D appearance cues and 3D structural constraints, CDIS corrects transient segmentation failures, prevents drift and produces globally consistent 3D instance labels without any 3D-specific training. We refer to this bidirectional reasoning paradigm as cross-dimensional processing. Unlike previous approaches that commit early to a fixed 3D representation obtained from single-frame projections, CDIS maintains a feedback loop between 2D and 3D: temporal cues from 2D tracking correct short-term errors, while 3D superpoint structure resolves spatial ambiguities and prevents long-term drift. This design mitigates segmentation fragmentation and improves the robustness of zero-shot 3D instance segmentation. We evaluate CDIS on publicly available benchmark datasets [19, 27], demonstrating superior accuracy and consistency compared to existing zero-shot methods. The results highlight CDIS as a scalable solution for autonomous robotic systems that require reliable perception of novel objects in diverse real-world environments. I Related Works I-A Learning-Based 3D Instance Segmentation Traditional 3D instance segmentation approaches rely heavily on large-scale annotated datasets [19, 27]. Fully supervised models [21, 11, 6, 15] typically adopt 3D convolutional neural networks or transformer-based backbones to process point clouds or voxelized inputs, producing detailed per-instance predictions. Although these techniques achieve strong performance in controlled, closed-set environments, they require labor-intensive 3D annotations and often generalize poorly to previously unseen object categories, limiting their use in dynamic or open-world settings. I-B 2D-Driven 3D Segmentation To reduce this dependency on 3D labels, several methods incorporate pretrained 2D segmentation models into 3D pipelines. These 2D-driven approaches [26, 28, 25] project per-frame 2D instance masks into 3D using depth maps and merge them based on geometric overlap or clustering. Such pipelines effectively leverage strong 2D visual priors but often remain tied to specific category sets or predefined taxonomies, limiting their scalability to open-world scenarios. Building on these ideas, class-agnostic and training-free methods have emerged to segment all objects without relying on semantic labels or 3D-specific training. For example, OVIR-3D [13], SAMPro3D [24], and Open3DIS [16] densely segment each RGB-D frame using general-purpose 2D models such as SAM [10] or CropFormer [18], then lift the resulting masks into 3D and merge them using superpoint associations or clustering strategies. These training-free, class-agnostic pipelines remove the need for labeled 3D data but still process frames independently, making them susceptible to segmentation drift and fragmented object identities over time. Our CDIS framework belongs to this family of training-free, class-agnostic approaches but addresses the temporal inconsistency issue by explicitly tracking 2D instance masks across frames and using 3D superpoints as geometric anchors, creating a feedback loop between 2D and 3D that yields globally consistent 3D instance labels. I Method We present a zero-shot, class-agnostic 3D instance segmentation framework that leverages a 2D entity-level segmentation model [18], pre-trained on large-scale datasets. Given an RGB-D scene represented by a sequence of T posed frames It,Dt,Ctt=1T\I_t,D_t,C_t\_t=1^T, where t denotes the frame index, It∈ℝH×W×3I_t ^H× W× 3 is the RGB image, Dt∈ℝH×WD_t ^H× W is the depth map, and Ct∈ℝ4×4C_t ^4× 4 is the camera pose, our objective is to generate 3D instance masks that accurately delineate distinct objects within the scene. Overview. Our approach generates 3D instance segmentation in three stages (Fig. 2). First, we perform 2D instance tracking to obtain temporally consistent 2D instance labels across frames by projecting 2D instance masks into 3D space using the corresponding depth maps, warping past frames, and matching instances using 2D mask IoU. Second, we perform 3D-guided 2D instance merging, where projected 3D superpoints are used to associate 2D instances across all frames based on their geometric consistency, producing spatio-temporally consistent 2D instance IDs. Finally, we perform 3D instance consolidation to resolve duplicate assignments of superpoints to multiple instance IDs by analyzing 3D overlaps and temporal co-occurrence, producing a unique and unified mapping of superpoints to instance IDs. The result is a fully spatio-temporally consistent 3D instance segmentation without requiring any task-specific training. I-A 2D Instance Tracking For each frame t, a 2D instance mask MtM_t is generated using the 2D segmentation model. These masks are then projected into 3D space using the depth map DtD_t and camera intrinsic matrix K, producing a set of 3D points Pt∈ℝNt×3P_t ^N_t× 3 with corresponding instance IDs Gt∈ℤNtG_t ^N_t, where NtN_t is the number of valid points in frame t. To ensure temporal continuity, we maintain a frame queue t=tp∣t−qmax<tp≤t Q_t=\t_p t-q_max<t_p≤ t\, storing up to qmaxq_max previous frames for comparison. Frame Warping. To track instances, we warp the 3D points from a past frame tp∈t_p∈ Q_t into the current frame t using the relative transformation: Ctp→t=Ct−1Ctp,Ptp→t=Ctp→tPtpC_t_p→ t=C_t^-1C_t_p, P_t_p→ t=C_t_p→ tP_t_p (1) where CtC_t and CtpC_t_p are the respective pose matrices representing the camera transformations for frames t and tpt_p, and PtpP_t_p represents the homogeneous 3D points from frame tpt_p. The transformation Ctp→tC_t_p→ t maps these points into the coordinate system of the current frame. The warped 3D points are then projected back into 2D space: ptp→t2D=KPtp→t⊤(KPtp→t⊤)zp_t_p→ t^2D= KP_t_p→ t (KP_t_p→ t )_z (2) ptp→t2Dp_t_p→ t^2D denotes the normalized 2D projections, with (⋅)z(·)_z indicating the Z-coordinate (depth) in camera space. A new mask Mtp→tM_t_p→ t is initialized and updated based on valid 2D coordinates within the image boundaries: Mtp→t(xi,yi)=Gtp[i]if(xi,yi)=ptp→t2D[i],M_t_p→ t(x_i,y_i)=G_t_p[i] (x_i,y_i)=p_t_p→ t^2D[i], (3) with (xi,yi)(x_i,y_i) representing the projected pixel coordinate of the i-th point and Gtp[i]G_t_p[i] indicating its associated instance label. Figure 3: Handling 2D Segmentation Failures with Multi-Frame Tracking. When generating the mask for frame t, using only the warped mask from frame t−1t\!-\!1 propagates an error where the chair and table are merged (red box). CDIS prevents this by also referencing frame t−2t\!-\!2, where the objects are correctly separated, and restores proper instance IDs in frame t. By leveraging multiple past frames rather than only the adjacent frame, CDIS mitigates temporary segmentation errors and prevents drift in object identities. Mask Matching. To maintain consistent instance labels across frames, we compare the current mask MtM_t with the warped previous mask Mtp→tM_t_p→ t at the instance level. For each instance label u in MtM_t and each instance label v in Mtp→tM_t_p→ t, we compute their intersection-over-union (IoU): IoU(u,v)=|Mt=u∩Mtp→t=v||Mt=u∪Mtp→t=v|,IoU(u,v)= |\M_t=u\∩\M_t_p→ t=v\||\M_t=u\∪\M_t_p→ t=v\|, (4) where Mt=u\M_t=u\ denotes the set of pixels in mask MtM_t labeled u. If any label v exceeds the threshold τIoU2D _IoU2D, we set the label of instance u to v∗=argmaxvIoU(u,v)v^*= _vIoU(u,v). If no label satisfies the threshold, a new unique instance label is assigned. This remapping is applied to all pixels of instance u, ensuring temporal consistency across frames (see Fig. 3). After this process, each frame MtM_t contains instance labels that are temporally matched to previous frames, resulting in a sequence Mtt=1T\M_t\_t=1^T with consistent instance identities across time. I-B 3D-Guided 2D Instance Merging We next leverage 3D geometry to improve instance consistency. A set of pre-computed 3D superpoints is used, where each superpoint Ss=pi∣pi∈ℝ3S_s=\p_i p_i ^3\ is a cluster of 3D points obtained from [4]. 3D Superpoint Projection. Each superpoint is projected onto all frames, producing its corresponding 2D footprint, denoted as Πt(s) _t(s). Each superpoint is then associated with the instance label in MtM_t that maximizes overlap: At(s)=argmaxu|Πt(s)∩(Mt=u)||Πt(s)|.A_t(s)= _u | _t(s)∩(M_t=u)|| _t(s)|. (5) Instance Association and Merging. For instance labels u (frame t) and v (frame t+1t+1), their similarity is computed directly from their associated superpoints: IoU3D(u,v)=|s∣At(s)=u∩s∣At+1(s)=v||s∣At(s)=u∪s∣At+1(s)=v|.IoU_3D(u,v)= |\s A_t(s)=u\∩\s A_t+1(s)=v\||\s A_t(s)=u\∪\s A_t+1(s)=v\|. (6) If IoU3D(u,v)>τIoU3DIoU_3D(u,v)> _IoU3D, the labels u and v are merged and assigned the same identity. This merging is applied hierarchically, pairwise across neighboring frames, reducing the number of frame groups by half per iteration, until a single consistent set of instance identities is obtained for the entire sequence. As a result, each frame MtM_t contains 2D instance IDs that are tracked across time and merged based on geometrical information, producing spatially and temporally consistent instance labels. I-C 3D Instance Consolidation Following the previous stage, we obtain spatio-temporally matched instances, where each 2D instance ID is associated with one or more 3D superpoints. However, duplicate assignments may occur, as multiple instance IDs can share the same superpoints. To establish a unique and consistent instance ID for each superpoint, we perform a consolidation process. Detection of Overlapping Instances. For each pair of instances (u,v)(u,v), we define their associated superpoint sets as uS_u and vS_v, where u=s∣At(s)=uS_u=\s A_t(s)=u\ denotes the set of superpoints assigned to instance u. We then compute the 3D intersection-over-minimum (IoMin): IoMin3D(u,v)=|u∩v|min(|u|,|v|).IoMin_3D(u,v)= |S_u _v| (|S_u|,|S_v|). (7) Pairs satisfying IoMin3D(u,v)>τIoMin3DIoMin_3D(u,v)> _IoMin3D are considered overlapping and evaluated further. Temporal Co-Occurrence Analysis. To distinguish between different objects that are spatially adjacent and identical objects observed at different times, we examine the temporal co-occurrence of the two instances: CO(u,v)=|Tu∩Tv|min(|Tu|,|Tv|),CO(u,v)= |T_u∩ T_v| (|T_u|,|T_v|), (8) where TuT_u and TvT_v are the sets of frames in which instances u and v are observed. If CO(u,v)>τcoCO(u,v)> _co, the two instances are likely distinct objects whose overlap originates from segmentation noise, and the overlapping superpoints are removed from the larger instance. Otherwise, the two instances are considered views of the same object and are merged. Iterative Refinement and Final Assignment. This process is repeated until no instance pair exceeds the τIoMin3D _IoMin3D threshold. Finally, each superpoint is assigned the instance ID with which it has been associated most frequently across all frames. This voting-based assignment ensures a one-to-one mapping between superpoints and instances, producing a unified and consistent 3D instance segmentation. IV Experiments IV-A Experimental Setup Datasets. We evaluated the robustness of our method on two benchmark datasets: ScanNet200 [19] and ScanNet++ [27]. These datasets offer a diverse range of environments for testing 3D instance segmentation. For our experiments, we used the validation sets of ScanNet200 and ScanNet++. Metrics. We assess the class-agnostic instance segmentation quality using the average precision (AP) metric, which is standard for instance segmentation evaluation. This metric disregards semantic labels and focuses solely on mask quality. AP is computed at different mask overlap thresholds—50% and 25%—and averaged across IoU thresholds from 0.5 to 0.95 in 0.05 increments. This provides a comprehensive assessment of segmentation accuracy across varying overlap conditions. Implementation Details. Our method processes posed RGB-D frames from each scene sequence. We experiment with both SAM [10] and CropFormer [18] as the 2D mask generator. For temporal tracking, we set the frame queue length to qmax=5q_max=5 and use a 2D IoU threshold of τIoU2D=0.8 _IoU2D=0.8 for mask matching. 3D superpoints are computed once for the entire scene using the method of Felzenszwalb and Huttenlocher [4] and projected into each frame for association with 2D instance masks. Merging of instances across frames is performed based on a 3D IoU threshold of τIoU3D=0.6 _IoU3D=0.6, followed by refinement using a 3D IoMin threshold of τIoMin3D=0.8 _IoMin3D=0.8 to resolve overlapping assignments. Temporal co-occurrence thresholding is set to τco=0.5 _co=0.5 to distinguish overlapping but distinct objects from partially duplicated observations. TABLE I: Class-agnostic instance segmentation on ScanNet200 [19]. Model 2D Model AP AP50 AP25 Mask3D† [21] - 39.7 53.6 62.5 SAMPro3D [24] SAM [10] 18.0 32.8 56.1 SAM3D [26] SAM [10] 20.2 35.7 55.5 Open3DIS [16] SAM [10] 31.5 45.3 51.1 MaskClustering [25] CropFormer [18] 19.2 36.6 51.7 OV-MAP [9] CropFormer [18] 29.9 49.4 67.5 Ours SAM [10] 30.1 44.9 57.8 CropFormer [18] 33.2 52.1 69.2 † : Fully supervised model with 3D ground truth masks IV-B Quantitative Results Figure 4: Comparison of Class-Agnostic 3D Instance Segmentation on ScanNet200. Visualization of 3D instance segmentation results. From left to right: input, ground truth (GT), Mask3D [21], SAM3D [26], and our method (CDIS). CDIS demonstrates improved segmentation accuracy, particularly in distinguishing object boundaries and reducing over-segmentation, as highlighted with red circles. ScanNet200 Evaluation. Table I presents the performance of our method on the ScanNet200 benchmark. Our CropFormer-based model achieves an AP of 33.2, surpassing OV-MAP [9] and Open3DIS [16], while also reporting the highest AP50 (52.1) and AP25 (69.2) among 2D-3D approaches. Compared to its SAM-based counterpart, our CropFormer-based model yields more coherent instance groupings in 3D, attributed to its stronger entity-level segmentation capability. Both versions benefit from CDIS’s cross-dimensional reasoning, where temporally tracked 2D masks are used to associate and consolidate 3D superpoints, resulting in stable and consistent instance labels across frames. While Mask3D [21], a fully supervised model, achieves the highest AP (39.7), it relies on dense 3D annotations. In contrast, our model achieves 33.2 AP without any 3D supervision, highlighting its practicality in annotation-scarce settings and its generalization to novel object categories. ScanNet++ Evaluation. On the more challenging ScanNet++ dataset (Table I), our CropFormer-based model achieves an AP of 28.2 and AP50 of 43.7, demonstrating strong performance comparable to state-of-the-art methods such as MaskClustering [25] and Open3DIS [16]. In AP25, our model scores 54.3, which is on par with MaskClustering [25] (54.7), showing that both approaches perform similarly at this level. These results validate the effectiveness of our cross-dimensional segmentation framework in handling complex 3D environments, particularly in improving instance-level consistency. Moreover, the consistent performance across datasets demonstrates CDIS’s robustness to scene variation, making it well-suited for robotic tasks in unfamiliar environments. IV-C Qualitative Results As shown in Fig. 4, we present qualitative examples of our class-agnostic 3D instance segmentation on ScanNet200. By leveraging 2D instance segmentation models, CDIS delivers high-quality 3D instance segmentation without requiring 3D-annotated training data. Mask3D [21], trained on ScanNet200 with full 3D supervision, achieves strong results but struggle with fragmented instances and occluded objects. Despite not being trained on 3D-annotated data, CDIS performs comparably or even outperforms Mask3D in challenging scenarios, particularly in preserving object boundaries and handling occlusions. Additionally, unlike SAM3D [26], which merges projected 2D masks in 3D space, CDIS tracks and refines 2D masks, which are then used to label 3D superpoints through projection overlap, reducing segmentation errors such as over-segmentation and object merging. As highlighted with red circles in Fig. 4, CDIS produces results that closely match the ground truth, demonstrating its robustness in complex indoor scenes. TABLE I: Class-agnostic instance segmentation on ScanNet++ [27]. Model 2D Model AP AP50 AP25 Mask3D† [21] - 22.8 33.3 45.7 SAM3D [26] SAM [10] 7.2 14.2 29.4 Segment3D [7] SAM [10] 19.0 29.7 41.6 Open3DIS [16] SAM [10] 20.7 38.6 47.1 MaskClustering [25] CropFormer [18] 27.9 42.8 54.7 Ours SAM [10] 22.9 35.4 46.7 CropFormer [18] 28.2 43.7 54.3 † : Fully supervised model with 3D ground truth masks IV-D Real-World Experiments To validate the practical applicability of CDIS, we performed real-world experiments using RGB-D data captured in indoor environments (home and office settings). Camera poses and synchronized RGB-D frames were obtained using the RTAB-Map visual SLAM method [12], and the fused scene geometry was reconstructed using a Truncated Signed Distance Function volume integration [14]. The captured environments contained various objects, such as chairs, tables, shelves with small items, and reflective surfaces under different lighting conditions. CDIS successfully segmented most objects, particularly small items on shelves, which are essential for robotic grasping and manipulation. However, segmentation accuracy decreased near windows with strong reflections due to inconsistent 2D mask predictions. Fig. 5 illustrates successful segmentations in challenging areas. These experiments demonstrate CDIS’s potential for integration into robotic perception systems for object manipulation, autonomous navigation, and mapping, enabling robust scene understanding without requiring 3D-specific training. Figure 5: Example of Real-World Class-Agnostic 3D Instance Segmentation. CDIS performance on real-world data, demonstrating class-agnostic instance segmentation across large and diverse indoor environments, including an office (red), a bedroom (yellow), and a kitchen (blue). Our method effectively segments cluttered and complex scenes without 3D training. References [1] A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, S. Song, A. Zeng, and Y. Zhang (2017) Matterport3d: learning from rgb-d data in indoor environments. arXiv preprint arXiv:1709.06158. Cited by: §I. [2] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar (2022) Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 1290–1299. Cited by: §I. [3] B. Cheng, A. Schwing, and A. Kirillov (2021) Per-pixel classification is not all you need for semantic segmentation. Advances in neural information processing systems 34, p. 17864–17875. Cited by: §I. [4] P. F. Felzenszwalb and D. P. Huttenlocher (2004) Efficient graph-based image segmentation. International journal of computer vision 59 (2), p. 167–181. Cited by: §I-B, §IV-A. [5] H. Guo, H. Zhu, S. Peng, Y. Wang, Y. Shen, R. Hu, and X. Zhou (2023) Sam-guided graph cut for 3d instance segmentation. arXiv preprint arXiv:2312.08372. Cited by: §I, §I. [6] J. Hou, A. Dai, and M. Nießner (2019) 3d-sis: 3d semantic instance segmentation of rgb-d scans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 4421–4430. Cited by: §I-A. [7] R. Huang, S. Peng, A. Takmaz, F. Tombari, M. Pollefeys, S. Song, G. Huang, and F. Engelmann (2023) Segment3d: learning fine-grained class-agnostic 3d segmentation without manual labels. arXiv preprint arXiv:2312.17232. Cited by: §I, TABLE I. [8] Z. Huang, X. Wu, X. Chen, H. Zhao, L. Zhu, and J. Lasenby (2024) Openins3d: snap and lookup for 3d open-vocabulary instance segmentation. In European Conference on Computer Vision, p. 169–185. Cited by: §I. [9] J. Kim, Y. Park, H. Yoon, and B. Zhang (2024) OV-map: open-vocabulary zero-shot 3d instance segmentation map for robots. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 13780–13786. Cited by: §I, §IV-B, TABLE I. [10] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W. Lo, et al. (2023) Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 4015–4026. Cited by: §I, §I-B, §IV-A, TABLE I, TABLE I, TABLE I, TABLE I, TABLE I, TABLE I, TABLE I, TABLE I. [11] M. Kolodiazhnyi, A. Vorontsova, A. Konushin, and D. Rukhovich (2024) Oneformer3d: one transformer for unified point cloud segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 20943–20953. Cited by: §I, §I-A. [12] M. Labbé and F. Michaud (2019) RTAB-map as an open-source lidar and visual simultaneous localization and mapping library for large-scale and long-term online operation. Journal of field robotics 36 (2), p. 416–446. Cited by: §IV-D. [13] S. Lu, H. Chang, E. P. Jing, A. Boularias, and K. Bekris (2023) Ovir-3d: open-vocabulary 3d instance retrieval without training on 3d data. In Conference on Robot Learning, p. 1610–1620. Cited by: §I, §I-B. [14] R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohi, J. Shotton, S. Hodges, and A. Fitzgibbon (2011) Kinectfusion: real-time dense surface mapping and tracking. In 2011 10th IEEE international symposium on mixed and augmented reality, p. 127–136. Cited by: §IV-D. [15] T. D. Ngo, B. Hua, and K. Nguyen (2023) Isbnet: a 3d point cloud instance segmentation network with instance-aware sampling and box-aware dynamic convolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 13550–13559. Cited by: §I-A. [16] P. Nguyen, T. D. Ngo, E. Kalogerakis, C. Gan, A. Tran, C. Pham, and K. Nguyen (2024) Open3dis: open-vocabulary 3d instance segmentation with 2d mask guidance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 4018–4028. Cited by: §I, §I-B, §IV-B, §IV-B, TABLE I, TABLE I. [17] Y. Park, H. Yoon, J. Kim, and B. Zhang (2025) DA-Fusion: deformable attention-based rgb-d fusion transformer for unseen object instance segmentation. In 2025 IEEE International Conference on Robotics and Automation (ICRA), p. 7490–7496. Cited by: §I. [18] L. Qi, J. Kuen, W. Guo, T. Shen, J. Gu, J. Jia, Z. Lin, and M. Yang (2022) High-quality entity segmentation. arXiv preprint arXiv:2211.05776. Cited by: §I, §I-B, §I, §IV-A, TABLE I, TABLE I, TABLE I, TABLE I, TABLE I. [19] D. Rozenberszki, O. Litany, and A. Dai (2022) Language-grounded indoor 3d semantic segmentation in the wild. In Proceedings of the European Conference on Computer Vision (ECCV), Cited by: §I, §I, §I-A, §IV-A, TABLE I. [20] D. Rozenberszki, O. Litany, and A. Dai (2024) Unscene3d: unsupervised 3d instance segmentation for indoor scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 19957–19967. Cited by: §I. [21] J. Schult, F. Engelmann, A. Hermans, O. Litany, S. Tang, and B. Leibe (2023) Mask3d: mask transformer for 3d semantic instance segmentation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), p. 8216–8223. Cited by: §I, §I-A, Figure 4, §IV-B, §IV-C, TABLE I, TABLE I. [22] J. Straub, T. Whelan, L. Ma, Y. Chen, E. Wijmans, S. Green, J. J. Engel, R. Mur-Artal, C. Ren, S. Verma, et al. (2019) The replica dataset: a digital replica of indoor spaces. arXiv preprint arXiv:1906.05797. Cited by: §I. [23] A. Takmaz, E. Fedele, R. W. Sumner, M. Pollefeys, F. Tombari, and F. Engelmann (2023) Openmask3d: open-vocabulary 3d instance segmentation. arXiv preprint arXiv:2306.13631. Cited by: §I. [24] M. Xu, X. Yin, L. Qiu, Y. Liu, X. Tong, and X. Han (2023) Sampro3d: locating sam prompts in 3d for zero-shot scene segmentation. arXiv preprint arXiv:2311.17707. Cited by: §I, §I-B, TABLE I. [25] M. Yan, J. Zhang, Y. Zhu, and H. Wang (2024) Maskclustering: view consensus based mask graph clustering for open-vocabulary 3d instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 28274–28284. Cited by: §I, §I-B, §IV-B, TABLE I, TABLE I. [26] Y. Yang, X. Wu, T. He, H. Zhao, and X. Liu (2023) Sam3d: segment anything in 3d scenes. arXiv preprint arXiv:2306.03908. Cited by: §I, §I-B, Figure 4, §IV-C, TABLE I, TABLE I. [27] C. Yeshwanth, Y. Liu, M. Nießner, and A. Dai (2023) Scannet++: a high-fidelity dataset of 3d indoor scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 12–22. Cited by: §I, §I, §I-A, §IV-A, TABLE I. [28] Y. Yin, Y. Liu, Y. Xiao, D. Cohen-Or, J. Huang, and B. Chen (2024) Sai3d: segment any instance in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 3292–3302. Cited by: §I, §I-B.