Paper deep dive

Seg2Grasp: A Robust Modular Suction Grasping in Bin Picking

Hye-Jung Yoon, Juno Kim, Yesol Park, Jun-Ki Lee, Byoung-Tak Zhang

Year: 2026Venue: arXiv preprintArea: cs.ROType: PreprintEmbeddings: 39

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 7/21/2026, 5:34:48 AM

Summary

The paper introduces Seg2Grasp, a modular robotic pipeline for robust suction-based bin picking in unstructured environments. It combines Transformer-based class-agnostic segmentation, surface normal-based grasp pose estimation, and fine-tuned open-vocabulary Mask-CLIP classification. The system outperforms end-to-end learning methods in success rates and adaptability across dynamic and cluttered scenarios.

Entities (8)

Seg2Grasp · system · 98%bin picking · task · 96%Mask-CLIP · model · 94%suction grasping · technique · 93%Mask2Former · model · 92%UR5e · robot · 90%Azure Kinect DK · sensor · 88%UOAIS-SIM · dataset · 85%

Relation Signals (7)

Seg2Grasp → performstask → bin picking

confidence 96% · Seg2Grasp, a modular pipeline designed for robust suction grasping in dynamic and cluttered bin scenarios

Seg2Grasp → usesmodule → Mask-CLIP

confidence 94% · Finally, the Classification module leverages fine-tuned open-vocabulary Mask-CLIP

Seg2Grasp → employstechnique → suction grasping

confidence 93% · robust suction grasping in dynamic and cluttered bin scenarios

Seg2Grasp → usesmodule → Mask2Former

confidence 92% · Our approach adopts Mask2Former for its robust segmentation capabilities

Seg2Grasp → runsonhardware → UR5e

confidence 90% · A UR5e robotic arm equipped with a Robotiq AirPick Vacuum Gripper was used

Seg2Grasp → usessensor → Azure Kinect DK

confidence 88% · RGB-D data was captured using an Azure Kinect DK Camera

Seg2Grasp → trainedon → UOAIS-SIM

confidence 85% · we utilized the UOAIS-SIM dataset... for training robust models

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Current bin picking methods that rely heavily on end-to-end learning often falter when confronted with unfamiliar or complex objects in unstructured environments. To overcome these limitations, we introduce Seg2Grasp, a modular pipeline designed for robust suction grasping in dynamic and cluttered bin scenarios. Seg2Grasp is built on a three-step process: Segmentation, Grasping, and Classification. The Segmentation module employs a Transformer-based model to generate class-agnostic object masks from RGB-D images, ensuring accurate detection across various conditions. The Grasping module uses surface normals and mask proposals to determine the optimal suction points, enhancing grasp success. Finally, the Classification module leverages fine-tuned open-vocabulary Mask-CLIP for precise object identification, enabling versatile handling of diverse objects. Real-world robotic experiments demonstrate that Seg2Grasp outperforms existing methods in success rates and adaptability, establishing it as a powerful tool for automated bin picking in industrial settings.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

37,867 characters extracted from source content.

Expand or collapse full text

Seg2Grasp: A Robust Modular Suction Grasping in Bin Picking Hye-Jung Yoon1∗ Juno Kim1∗ Yesol Park1∗ Jun-Ki Lee2 Byoung-Tak Zhang1,2,3 *Authors have equal contributions1Interdisciplinary Program in AI, Seoul National University2Artificial Intelligence Institute, Seoul National University3Department of Computer Science, Seoul National UniversityThis work was partly supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korean government (MSIT) [RS-2021-I211343, Artificial Intelligence Graduate School Program (Seoul National University)] and (RS-2021-I212068-AIHub/10%, RS-2021-I211343-GSAI/15%, 2022-0-00951-LBA/15%, 2022-0-00953-PICA/20%), NRF (RS-2024-00353991/20%, RS-2023-00274280/10%), and KEIT (RS-2024-00423940/10%). Abstract Current bin picking methods that rely heavily on end-to-end learning often falter when confronted with unfamiliar or complex objects in unstructured environments. To overcome these limitations, we introduce Seg2Grasp, a modular pipeline designed for robust suction grasping in dynamic and cluttered bin scenarios. Seg2Grasp is built on a three-step process: Segmentation, Grasping, and Classification. The Segmentation module employs a Transformer-based model to generate class-agnostic object masks from RGB-D images, ensuring accurate detection across various conditions. The Grasping module uses surface normals and mask proposals to determine the optimal suction points, enhancing grasp success. Finally, the Classification module leverages fine-tuned open-vocabulary Mask-CLIP for precise object identification, enabling versatile handling of diverse objects. Real-world robotic experiments demonstrate that Seg2Grasp outperforms existing methods in success rates and adaptability, establishing it as a powerful tool for automated bin picking in industrial settings. I INTRODUCTION In the field of industrial automation, bin picking is a critical yet challenging task, especially when applied to environments that are dynamic and contain unknown objects. Such settings are characterized by a variety of unpredictable factors, including fluctuating lighting, different camera viewpoints, and the presence of objects that the system has not previously encountered. These challenges demand a solution that is both precise and adaptable, capable of handling the complexities of real-world industrial scenarios. Existing bin picking systems predominantly utilize end-to-end learning-based methods [12, 13, 14, 3, 24], which aim to directly map sensory inputs to outputs. Although these approaches have shown success in controlled environments, they frequently fall short in terms of adaptability—the ability to function effectively across diverse conditions—and robustness, which refers to maintaining performance despite variations in the environment. The relatively simplistic nature of these end-to-end methods limits their generalization capabilities, particularly when faced with novel objects or unfamiliar configurations. To address these limitations, we propose Seg2Grasp, a novel pipeline designed to enhance the robustness and adaptability of suction-based bin picking systems in dynamic environments. Unlike traditional end-to-end models, Seg2Grasp leverages a modular architecture comprising three core components: Segmentation, Grasping, and Classification. This modular design, depicted in Fig. 1, allows each component to specialize in a specific task, providing greater flexibility and improved performance across a wider range of scenarios. The Segmentation module utilizes a Transformer-based model to generate class-agnostic object masks from depth-weighted RGB images. This method enables accurate object segmentation across diverse conditions, overcoming one of the primary challenges in bin picking—reliable detection and segmentation of objects regardless of environmental variability. These object masks are then passed to the Grasping module, which, combined with surface normal data, identifies optimal grasping points. This process significantly improves the precision of object manipulation, enabling the system to successfully grasp a wide range of objects with varying geometries and orientations. Figure 1: Illustration of proposed system. Our bin picking system can segment variously shaped, class-agnostic objects in a dynamic environment and proceed with grasping and classifying them. Finally, the Classification module employs an open-vocabulary classification system to categorize a wide array of objects. This capability allows the system to adapt to and handle objects without prior explicit knowledge of each object, greatly expanding the operational versatility of the robotic system. Figure 2: Overview of our modular bin picking system. The system comprises three modules: class-agnostic object segmentation, object grasp pose estimation, and open-vocabulary object classification. The black arrow indicates the flow of operations within the entire system. To validate the effectiveness of Seg2Grasp, we conducted a series of robotic experiments in dynamic environments, utilizing various bins and camera configurations to simulate realistic bin picking scenarios. Our approach consistently outperformed existing end-to-end methods [3, 14] in terms of success rates, particularly in unstructured and dynamic environments. In summary, Seg2Grasp addresses the significant gaps in adaptability and robustness found in current bin picking technologies. By integrating specialized modules for segmentation, grasping, and classification, Seg2Grasp not only enhances performance in complex environments but also sets a new benchmark for the potential of modular approaches in industrial automation. I Related Work In robotics, bin picking in unstructured environments is a complex task that requires robust object recognition, accurate grasp pose estimation, and flexible object classification. This section reviews the recent advances in these areas, which are essential to the development of our Seg2Grasp framework. I-A Class-agnostic Object Instance Segmentation Class-agnostic Object Instance Segmentation is critical for enabling robots to identify and segment objects that have not been previously encountered. Traditional segmentation methods often struggle in unstructured and cluttered environments [20, 15]. Unlike category-based instance segmentation, which focuses on known categories [26, 5, 19], Class-agnostic Object Instance Segmentation aims to generalize segmentation to arbitrary objects [22, 23, 21]. Recent advancements include UOAIS-Net [2], which addresses occlusion and amodal perception in robotic manipulation but faces challenges with occluded areas, leading to potential misinterpretations in cluttered scenarios. Another approach, MSMFormer [11], improves segmentation accuracy by integrating a clustering method with Mask2Former [5]. However, this comes at the cost of increased computational requirements, highlighting a precision-efficiency trade-off. Our approach adopts Mask2Former for its robust segmentation capabilities, particularly its ability to maintain high accuracy without the need for clustering. This selection strikes a balance between efficiency and performance, making it suitable for the dynamic environments typical of bin picking tasks. I-B Grasp Pose Estimation Grasp pose estimation is fundamental in robotic manipulation, where accurate identification of grasp points is necessary for successful object handling. This process typically relies on RGB-D and point cloud data [6]. There are two main approaches: learning-free methods [18] and learning-based methods [7, 9, 13, 3, 25, 24]. Learning-free approaches often depend on CAD models to determine grasp candidates for recognized objects [18, 17]. On the other hand, learning-based methods, particularly those utilizing deep learning, have gained prominence due to their improved accuracy and reliability in controlled environments [4]. These models are generally trained on depth or RGB-D datasets [14, 3], which allows for precise and consistent grasping. However, their performance tends to degrade in unstructured environments where the conditions differ from the training data. To address the limitations of traditional methods, our grasping algorithm is specifically designed to maintain robustness in diverse and unpredictable environments. This approach ensures that grasp performance remains effective even under the variable conditions typical of real-world industrial settings, thus filling a significant gap in current robotic manipulation strategies. I-C Open-Vocabulary Classification Open-vocabulary classification represents a significant advancement in image classification, allowing systems to recognize a broader range of visual concepts beyond fixed label sets, enhanced by natural language processing. Language-Image Pre-training models, such as CLIP [16], have been instrumental in bridging the gap between visual data and textual descriptions. Despite its strengths, CLIP and similar models often experience reduced performance when applied to images that deviate from their training distributions. OVSeg [10] addresses this issue by extracting CLIP features directly from segmentation masks, excluding background elements to enhance accuracy. This refined approach improves the model’s performance, particularly when dealing with images significantly different from those in the training set. In our Seg2Grasp framework, we utilize an open-vocabulary classification system enhanced by Mask-CLIP, which enables effective object recognition across a wide variety of scenarios. This flexibility further expands the operational capabilities of the system in dynamic bin picking environments. Figure 3: Mask proposal network input. The input I to the mask proposal network consists of a fusion of the color image SCS_C and the inverted depth image SD′S _D. I Problem Statement In the context of bin picking, our primary objective is to develop a system capable of accurately predicting a set of feasible grasp poses and their corresponding object labels. This system must effectively transfer arbitrary objects from a source bin to a target bin, even in unstructured and cluttered environments. Achieving this goal requires a comprehensive understanding of the scene, including precise segmentation, grasp pose estimation, and object classification. Given a scene image S, the task is to segment the image into distinct objects, represented by a set of masks Mi∣1≤i≤N\M_i 1≤ i≤ N\, where N is the number of objects identified in the scene. Each mask MiM_i corresponds to an individual object, facilitating further analysis. The target object in the scene is denoted by T. The grasp configuration for the target object T is defined as g=(SPopt,NVopt)g=(SP_opt,NV_opt), where SPoptSP_opt represents the optimal suction point on the object, and NVoptNV_opt is the associated normal vector at that point. Additionally, each object is assigned a label l, which indicates its categorical identity and is essential for its correct placement. The core challenge we address is to develop a predictive model f that, for each input mask MiM_i, accurately maps to both a grasp configuration gi=(SPopt,i,NVopt,i)g_i=(SP_opt,i,NV_opt,i) and an object label lil_i. This mapping is expressed as: f:Mi⟼(gi,li),for i=1,2,…,N.f:M_i (g_i,l_i), i=1,2,…,N. This formulation is critical for determining not only the optimal grasping approach but also for understanding the object’s identity, ensuring its correct transfer and placement. To solve this problem, we propose a three-step method: 1. Object Segmentation: A Transformer-based model processes the scene image S to generate class-agnostic object masks, producing the set Mi∣1≤i≤N\M_i 1≤ i≤ N\. The target object T is selected from this set. 2. Object Grasping: For the selected target object T, surface normals are used to calculate the optimal grasp pose g=(SPopt,NVopt)g=(SP_opt,NV_opt), ensuring the most suitable suction point is identified. 3. Object Classification: An open-vocabulary classification module assigns a label l to each object, facilitating accurate identification and enabling precise placement. Our modular approach, which separates segmentation, grasping, and classification into distinct yet interconnected components, offers enhanced adaptability and robustness. This flexibility is particularly advantageous in dynamic and variable environments, where conditions may differ significantly from the controlled settings typically assumed by other methods. IV METHOD Seg2Grasp is structured into three main modules: (1) Object Segmentation, (2) Object Grasping, and (3) Object Classification. The overall framework is illustrated in Fig. 2 IV-A Object Segmentation The object segmentation module processes the scene image S, and distinguishes individual objects within the scene. The target object selected for grasping is denoted as T. Input Preparation. The input to the network, denoted as I, is constructed by combining the RGB image SCS_C with a modified depth image SD′S _D. To enhance the depth information, the depth image SDS_D is normalized and inverted: SD′=1−(SD−SDminSDmax−SDmin),S _D=1- ( S_D-S_D_minS_D_max-S_D_min ), (1) where SDminS_D_min and SDmaxS_D_max are the minimum and maximum values in SDS_D. This inversion highlights closer objects, improving the contrast between objects at different depths. The final input I is the product of the color image SCS_C and the inverted depth image SD′S _D, which enhances object differentiation. This process is shown in Fig. 3 Algorithm 1 Mask Filtering 0: Set of masks M 0: Target object T, Largest planar area ATA_T 1: for each mask MiM_i in M do 2: Initialize: LPi←LP_i← null, Best inlier count Inbest,i←0In_best,i← 0 3: for iteration k do 4: Select a random set X from MiM_i 5: Estimate plane PlkPl_k from X 6: Determine inliers InkIn_k: Points in MiM_i fitting PlkPl_k within tolerance ϵε 7: if |Ink|>Inbest,i|In_k|>In_best,i then 8: LPi←PlkLP_i← Pl_k, Inbest,i←|Ink|In_best,i←|In_k| 9: end if 10: end for 11: Calculate centroid Ci=(xi,yi,zi)C_i=(x_i,y_i,z_i) of Inbest,iIn_best,i 12: end for 13: T=arg⁡maxi⁡(zi)T= _i(z_i) 14: AT=Area of LP for TA_T=Area of LP for T 15: return T, ATA_T Mask Proposal. We developed a model for proposing class-agnostic instance masks using the standard Mask Transformer model [5]. The model outputs a set of masks M=Mi∣1≤i≤100M=\M_i 1≤ i≤ 100\ for each input image, where each mask captures an individual object. By using the fused input I and focusing on class-agnostic outputs, the model generalizes well across diverse objects. Mask Filtering. To identify the target object T, we refine the set of generated masks M using a filtering process. This process leverages the RANSAC algorithm to estimate the largest planar area (LPiLP_i) within each mask MiM_i. For each mask, RANSAC iteratively selects a subset of points to estimate a plane and identifies inliers—points that fit the plane within a specified tolerance. The plane with the highest inlier count is selected, and the centroid Ci=(xi,yi,zi)C_i=(x_i,y_i,z_i) of these inliers is computed. The mask with the highest z-coordinate centroid is then chosen as the target object T, ensuring that the most elevated object in the scene, and thus the most accessible for grasping, is selected. The complete mask filtering process is detailed in Alg. 1. Figure 4: Grasp pose estimation. The cyan dot represents the optimal grasp point on the target object, identified by the highest final score (detailed in Alg. 2). IV-B Object Grasping The object grasping module processes the selected target object T to determine the optimal grasp point g=(SPopt,NVopt)g=(SP_opt,NV_opt), as depicted in Fig. 4. This process, detailed in Alg. 2, utilizes surface normals derived from the object to identify the most suitable suction pose for grasping. Our approach differs from traditional methods [3, 14], which often depend on extensive training data and specific environmental setups. Instead, we rely solely on RGB-D imagery to compute the optimal suction point, providing greater flexibility across various objects and scenarios. This is particularly advantageous when using suction grippers, which can attach to suitable surface points on diverse objects. Suction Point Evaluation. The optimal suction point is determined by analyzing the preprocessed point cloud PTP_T. The process involves navigating through PTP_T to identify potential suction areas, with the region of interest adjusted to match the suction cup’s dimensions. Each potential grasp point is evaluated based on its alignment with the normal vector NVTNV_T of the central point, considering a tolerance angle ϕφ. To determine the most suitable suction point g=(SPopt,NVopt)g=(SP_opt,NV_opt), each candidate site is assessed using three metrics: surface angle θ, distance d from the centroid CTC_T, and the count of graspable points G. These metrics are normalized and combined into a composite score that prioritizes angular alignment, proximity, and point count: FinalScore(g)=wθSθ(θ)+wdSd(d)+wgSg(||)FinalScore(g)=w_θS_θ(θ)+w_dS_d(d)+w_gS_g(|G|) (2) Here, SθS_θ, SdS_d, and SgS_g are the normalized score functions for each metric, with wθw_θ, wdw_d, and wgw_g as their respective weights, satisfying the condition wθ+wd+wg=1w_θ+w_d+w_g=1. Algorithm 2 Pose Estimation for Optimal Suction Point 0: Point cloud PTP_T, 2D mask MTM_T, centroid CTC_T, suction radius rsuctionr_suction, angle tolerance ϕφ, min grasped points ψ, angle threshold δ 0: Optimal suction point SPoptSP_opt, normal vector NVoptNV_opt at SPoptSP_opt 1: Preprocessing: 2: Filter and downsample PTP_T using MTM_T. 3: Calculate surface normals NVTNV_T and angles ΘT _T. 4: Point Evaluation: 5: for each point p in PTP_T do 6: if angle ΘT(p) _T(p) >> δ then 7: Define region RnearR_near around p within rsuctionr_suction. 8: Filter points in RnearR_near with normals within ϕφ of NVT(p)NV_T(p). 9: if count of filtered points >ψ>ψ then 10: Determine plane normal NplaneN_plane for these points. 11: Aggregate NplaneN_plane, its angle, distance to CTC_T, and inlier count. 12: end if 13: end if 14: end for 15: Aggregate and weight selection criteria Sθ,Sd,SgS_θ,S_d,S_g with weights wθ,wd,wgw_θ,w_d,w_g. 16: Identify SPoptSP_opt and NVoptNV_opt using weighted criteria. 17: return SPopt,NVoptSP_opt,NV_opt IV-C Object Classification The classification module operates in parallel with the grasping mechanism, identifying the target object T and assigning it a corresponding label l, thereby enabling precise placement. Preparation for CLIP. We employ a fine-tuned Mask-CLIP model for open-vocabulary object classification, following the methodology outlined in previous research [10]. The model processes the mask proposals generated earlier, with a prediction branch specifically tailored for masked inputs. A CLIP-based branch then computes the class probabilities p^i,k p_i,k for each mask. A key challenge arises from the fact that CLIP, being trained primarily on images with natural backgrounds, exhibits reduced effectiveness when working with masked inputs that contain large areas of blank space. Fine-Tuning Mask-CLIP. Masked images, when tokenized for CLIP, often lead to zero tokens due to the extensive blank areas, indicating a domain shift. To address this, we apply mask prompt tuning. This involves enhancing the tensorized masked images with learnable prompt tokens derived from a binary mask, helping to preserve essential boundary information and improve classification accuracy. Feature Matching. After fine-tuning the Mask-CLIP model, we measure the cosine similarity between the features of the masked images and the text descriptors of the categories. This approach enables precise object classification by effectively matching the visual features to the corresponding textual descriptions. V EXPERIMENTS This section presents the experimental setup and results, which demonstrate the efficacy of the proposed method in addressing our problem context. V-A Implementation Detail V-A1 Segmentation Module Accurate object segmentation within cluttered bin environments, especially when encountering class-agnostic items, requires a model that generalizes well across different scenarios. This capability is crucial for successful bin picking tasks. Dataset. Given the scarcity of comprehensive real-world datasets for bin picking, we utilized the UOAIS-SIM dataset [2], which includes 50,000 photorealistic RGB-D images of objects in bin settings. This dataset bridges the gap between simulation and reality, offering extensive exposure to various object scenarios critical for training robust models. Training. We optimized our segmentation module using the Tversky loss function, which addresses class imbalance by focusing on class-agnostic object detection. The final loss combines Tversky loss and a classification loss component to ensure comprehensive learning. Training was conducted on the UOAIS-SIM dataset using the AdamW optimizer with a learning rate of 1e−41e-4 and a batch size of 4 over five epochs. Figure 5: Experimental setup. (a) Examples of objects used in the bin picking experiments. (b) Experimental environment using the UR5e robot. V-A2 Classification Module To enhance the accuracy of open-vocabulary object classification with mask-only inputs, fine-tuning is imperative. This adjustment ensures the system can effectively recognize objects across a wide range of categories. Dataset. For our experiment, we constructed a product database using the ‘Product Image Dataset’ [1]. Given the variability in object orientations within bins, we developed a ‘mask-category’ dataset. This dataset encompasses 720,000 product mask images across 53 main categories (e.g., snack, drink, dairy), generated from photographs of 10,000 objects captured from 72 different angles. This comprehensive approach allows our model to better recognize objects from any orientation, enhancing the robustness of our classification module. Training. The OpenCLIP framework [8] was fine-tuned using our specialized dataset. We used 90% of the data for training and 10% for evaluation, ensuring balanced exposure across all categories. The model was optimized for accurate classification of masked images, with training conducted over 10 epochs using a ViT-L/14 CLIP variant. V-B Real-robot Experiments To demonstrate the robustness of our system, we designed three distinct experimental setups: 1) Optimal Conditions, to establish a baseline for performance; 2) Varying Camera Parameters, to examine how changes in visual input affect system performance; and 3) Different Bin Environments, to assess the system’s adaptability to changes in the surrounding environment. Experimental Setup. For our experiments, we selected a diverse range of objects, including boxes, cylinders, spheres, and various irregular shapes, as shown in Fig. 5(a). This selection ensured a comprehensive evaluation of our system’s adaptability. A UR5e robotic arm equipped with a Robotiq AirPick Vacuum Gripper was used for object manipulation, while RGB-D data was captured using an Azure Kinect DK Camera, strategically positioned at an elevated angle, as shown in Fig. 5(b). The experiment was designed to focus on objects compatible with the vacuum gripper, allowing us to optimize the evaluation of the system’s performance in real-world scenarios. Figure 6: Experimental results. (a) Failure cases highlighting the limitations of different models. (b) Setup and outcomes with different camera parameters. (c) Setup and outcomes with different bin environments. Evaluation Metrics. Our bin picking task focuses on achieving the complete removal of items from a bin, evaluated through three principal metrics: The pick success rate, pr=NgNtpr= N_gN_t, quantifies the effectiveness of grasps, where NgN_g represents the number of successful grasps and NtN_t the total number of attempts. The object success rate, or=NgNoor= N_gN_o, measures the accuracy of object handling, with NgN_g indicating the number of objects successfully handled and NoN_o the total count of objects. The segmentation success rate, sr=NsNosr= N_sN_o, assesses the precision of segmentation, where NsN_s signifies successfully segmented items, accurate within a tolerance of ±20%± 20\% of the ground truth area. Together, these critical metrics facilitate a comprehensive evaluation of our system’s performance. V-B1 Optimal Conditions Experiments We conducted experiments under conditions optimized for each method, ensuring that the environment was adjusted to provide the best possible performance. In this idealized setting, the objects and scenarios used in the experiments closely matched the data on which each model was trained. This approach allowed us to fairly assess the capabilities of each system in an environment that reflects its training conditions. The evaluation was conducted across three levels of object distribution and complexity, progressively increasing the challenge to each model, as presented in Tab. I. In the simplest conditions, all methods performed comparably, with Seg2Grasp achieving a prpr of 89%, slightly higher than DexNet 4.0 at 85% and SuctionNet at 72%. As the complexity increased, Seg2Grasp’s performance advantage became more pronounced. Under the most challenging conditions, Seg2Grasp maintained a prpr of 79%, significantly outperforming DexNet 4.0 at 28% and SuctionNet at 29%. Similarly, Seg2Grasp excelled in object handling and segmentation accuracy. It achieved an oror of 86% and a srsr of 83% in the hardest scenarios, whereas DexNet 4.0 and SuctionNet showed considerable drops in performance, with oror and srsr both falling below 31%. Both DexNet 4.0 and SuctionNet exhibit notable weaknesses under certain conditions. DexNet 4.0 struggles particularly with object overlap, making it difficult to grasp objects when heavier items are stacked on top. On the other hand, SuctionNet is highly sensitive to its environment and the specific objects it encounters, often misidentifying the yellow background as a suction point, which creates significant bottlenecks in the process. These issues are illustrated in Fig. 6 (a). These results underscore Seg2Grasp’s superior robustness and adaptability, particularly under complex and variable conditions. The ability to maintain high accuracy across different levels of difficulty highlights its potential for real-world applications where precision and adaptability are critical. V-B2 Varying Camera Parameters Experiments To evaluate the adaptability of the models under varying visual conditions, we conducted experiments focusing on the impact of different camera heights. The camera heights were altered to 50 cm (optimal), 100 cm, and 150 cm to assess how each model handles significant variations in the depth data, which are crucial for reliable object detection and grasping. Over 10 experimental sets, each model was allowed up to 75 attempts per set to pick objects from the bin. As depicted in Fig. 6(b), SuctionNet and DexNet 4.0 showed considerable performance degradation at higher camera positions. DexNet 4.0, in particular, struggled due to increased noise in depth perception, successfully picking only 10 to 15 objects. SuctionNet fared slightly better but still demonstrated a significant drop in performance. In contrast, Seg2Grasp maintained high performance across all tested heights, showcasing remarkable resilience and consistency. These results highlight Seg2Grasp’s robustness in adapting to changes in camera parameters, a critical factor in real-world robotic applications where environmental conditions can vary. TABLE I: Results of Optimal Conditions Experiments Across Three Levels of Difficulty. Level Setup (Layer, Item) Method prpr oror srsr EASY Single, Trained DexNet 4.0 [14] 0.85 0.94 - SuctionNet [3] 0.72 0.92 0.93 Ours 0.89 0.96 0.91 MEDIUM Double, Mixed DexNet 4.0 [14] 0.41 0.61 - SuctionNet [3] 0.51 0.53 0.43 Ours 0.87 0.91 0.89 HARD Complex, Novel DexNet 4.0 [14] 0.28 0.31 - SuctionNet [3] 0.29 0.23 0.26 Ours 0.79 0.86 0.83 V-B3 Different Bin Environments Experiments We also examined the models’ performance across different bin types to further test their adaptability. Starting with a large yellow bin as the baseline, we varied the bin types, including a shopping basket and a small white box, to simulate different real-world conditions. Each experimental set included 25 objects, none of which were encountered during training, with a maximum of 75 attempts allowed per set. The outcomes, as shown in Fig. 6(c), reveal that both SuctionNet and DexNet 4.0 faced significant challenges when the bin type was changed. DexNet 4.0’s reliance on depth data proved advantageous in scenarios with consistent lighting and minimal RGB variations, slightly outperforming SuctionNet in certain cases. However, neither model succeeded in fully clearing the bin under all conditions. On the other hand, Seg2Grasp consistently achieved high success rates regardless of the bin type, underscoring its superior adaptability and effectiveness in diverse environments. This consistency reinforces Seg2Grasp’s capability to handle varying operational settings, making it a robust solution for industrial bin picking tasks. V-C Open-Vocabulary Classification Experiment We conducted an experiment to compare the performance of Mask-CLIP against the standard CLIP model using our ‘mask-category’ evaluation dataset. This involved computing CLIP feature vectors for evaluation mask images with both models and calculating their cosine similarity to 53 predefined categories. The results, summarized in Tab. I, indicate that Mask-CLIP outperforms the standard CLIP model in open-vocabulary classification tasks for mask-based inputs, achieving a Top-1 accuracy of 73.8% and a Top-3 accuracy of 84.7%, compared to CLIP’s 66.1% and 78.9%, respectively. TABLE I: Results of the open-vocabulary classification. METHOD Top-1 Acc. Top-3 Acc. CLIP [16] 66.1% 78.9% MASK-CLIP [10] 73.8% 84.7% VI CONCLUSION This paper presented Seg2Grasp, a modular pipeline for enhancing suction grasping in dynamic, unstructured bin environments. The approach consists of three key modules: (1) Class-Agnostic Object Segmentation, isolating class-agnostic objects; (2) Grasp Pose Estimation using Surface Normals, identifying optimal suction points; and (3) Open-Vocabulary Object Classification, utilizing fine-tuned Mask-CLIP models for accurate identification. Real-robot experiments demonstrated that Seg2Grasp outperforms existing end-to-end bin picking models in both robustness and accuracy, highlighting the effectiveness of a modular strategy for automated bin picking. References [1] Note: AI-Hub, S. Korea, The Open AI Dataset Project. All data information can be accessed through ‘w.aihub.or.kr’. Cited by: §V-A2. [2] S. Back, J. Lee, T. Kim, S. Noh, R. Kang, S. Bak, and K. Lee (2022) Unseen object amodal instance segmentation via hierarchical occlusion modeling. In 2022 International Conference on Robotics and Automation (ICRA), p. 5085–5092. Cited by: §I-A, §V-A1. [3] H. Cao, H. Fang, W. Liu, and C. Lu (2021) Suctionnet-1billion: a large-scale benchmark for suction grasping. IEEE Robotics and Automation Letters 6 (4), p. 8718–8725. Cited by: §I, §I, §I-B, §I-B, §IV-B, TABLE I, TABLE I, TABLE I. [4] J. Chen, L. Zhang, Y. Liu, and C. Xu (2020) Survey on 6d pose estimation of rigid object. In 2020 39th Chinese Control Conference (C), p. 7440–7445. Cited by: §I-B. [5] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar (2022) Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 1290–1299. Cited by: §I-A, §I-A, §IV-A. [6] A. Cordeiro, L. F. Rocha, C. Costa, P. Costa, and M. F. Silva (2022) Bin picking approaches based on deep learning techniques: a state-of-the-art survey. In 2022 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), Vol. , p. 110–117. External Links: Document Cited by: §I-B. [7] Z. Dong, S. Liu, T. Zhou, H. Cheng, L. Zeng, X. Yu, and H. Liu (2019) PPR-net: point-wise pose regression network for instance segmentation and 6d pose estimation in bin-picking scenarios. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 1773–1780. Cited by: §I-B. [8] G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt (2021-07) OpenCLIP. Zenodo. External Links: Document, Link Cited by: §V-A2. [9] X. Li, R. Cao, Y. Feng, K. Chen, B. Yang, C. Fu, Y. Li, Q. Dou, Y. Liu, and P. Heng (2022) A sim-to-real object recognition and localization framework for industrial robotic bin picking. IEEE Robotics and Automation Letters 7 (2), p. 3961–3968. Cited by: §I-B. [10] F. Liang, B. Wu, X. Dai, K. Li, Y. Zhao, H. Zhang, P. Zhang, P. Vajda, and D. Marculescu (2023) Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 7061–7070. Cited by: §I-C, §IV-C, TABLE I. [11] Y. Lu, Y. Chen, N. Ruozzi, and Y. Xiang (2022) Mean shift mask transformer for unseen object instance segmentation. arXiv preprint arXiv:2211.11679. Cited by: §I-A. [12] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg (2017) Dex-net 2.0: deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. arXiv preprint arXiv:1703.09312. Cited by: §I. [13] J. Mahler, M. Matl, X. Liu, A. Li, D. Gealy, and K. Goldberg (2018) Dex-net 3.0: computing robust vacuum suction grasp targets in point clouds using a new analytic model and deep learning. In 2018 IEEE International Conference on robotics and automation (ICRA), p. 5620–5627. Cited by: §I, §I-B. [14] J. Mahler, M. Matl, V. Satish, M. Danielczuk, B. DeRose, S. McKinley, and K. Goldberg (2019) Learning ambidextrous robot grasping policies. Science Robotics 4 (26), p. eaau4984. Cited by: §I, §I, §I-B, §IV-B, TABLE I, TABLE I, TABLE I. [15] M. Q. Mohammed, L. C. Kwek, S. C. Chua, A. Al-Dhaqm, S. Nahavandi, T. A. E. Eisa, M. F. Miskon, M. N. Al-Mhiqani, A. Ali, M. Abaker, et al. (2022) Review of learning-based robotic manipulation in cluttered environments. Sensors 22 (20), p. 7938. Cited by: §I-A. [16] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §I-C, TABLE I. [17] P. Schillinger, M. Gabriel, A. Kuss, H. Ziesche, and N. A. Vien (2023) Model-free grasping with multi-suction cup grippers for robotic bin picking. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 3107–3113. Cited by: §I-B. [18] F. Spenrath and A. Pott (2017) Gripping point determination for bin picking using heuristic search. Procedia CIRP 62, p. 606–611. Cited by: §I-B, §I-B. [19] Z. Tian, C. Shen, and H. Chen (2020) Conditional convolutions for instance segmentation. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, p. 282–298. Cited by: §I-A. [20] A. Ückermann, R. Haschke, and H. Ritter (2012) Real-time 3d segmentation of cluttered scenes for robot grasping. In 2012 12th IEEE-RAS International Conference on Humanoid Robots (Humanoids 2012), p. 198–203. Cited by: §I-A. [21] Y. Xiang, C. Xie, A. Mousavian, and D. Fox (2021) Learning rgb-d feature embeddings for unseen object instance segmentation. In Conference on Robot Learning, p. 461–470. Cited by: §I-A. [22] C. Xie, Y. Xiang, A. Mousavian, and D. Fox (2020) The best of both modes: separately leveraging rgb and depth for unseen object instance segmentation. In Conference on robot learning, p. 1369–1378. Cited by: §I-A. [23] C. Xie, Y. Xiang, A. Mousavian, and D. Fox (2021) Unseen object instance segmentation for robotic environments. IEEE Transactions on Robotics 37 (5), p. 1343–1359. Cited by: §I-A. [24] A. Zeng, S. Song, K. Yu, E. Donlon, F. R. Hogan, M. Bauza, D. Ma, O. Taylor, M. Liu, E. Romo, et al. (2022) Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. The International Journal of Robotics Research 41 (7), p. 690–705. Cited by: §I, §I-B. [25] H. Zhang, J. Peeters, E. Demeester, and K. Kellens (2021) A cnn-based grasp planning method for random picking of unknown objects with a vacuum gripper. Journal of Intelligent & Robotic Systems 103, p. 1–19. Cited by: §I-B. [26] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai (2020) Deformable detr: deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159. Cited by: §I-A.

Instant research discovery