Paper deep dive
UniPart: Towards Zero-shot Language-Grounded 3D Part Segmentation for Embodied Interaction
Xinqiang Yu, Zekun qi, Jiawei He, Wenyao Zhang, Xuchuan Chen, Guaocai Yao, Li Yi, Zhaoxiang Zhang, He Wang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/20/2026, 2:35:02 AM
Summary
The paper introduces UniPart, a feed-forward cross-modal 3D Transformer for zero-shot, language-grounded 3D part segmentation. It addresses the limitations of existing models that are either object-aware or limited to closed-set part taxonomies. The authors propose LangPart-1M, a large-scale dataset with 160K+ Objaverse assets and 8M text-to-part pairs generated via a multi-view consistent pipeline, and LangPart-4K, a manually verified subset for fine-tuning. UniPart injects frozen CLIP text embeddings into a 3D Transformer to predict part masks from point clouds and natural language instructions, achieving strong zero-shot performance and transferring to real-world language-conditioned robotic grasping.
Entities (10)
Relation Signals (7)
UniPart → performstask → 3D Part Segmentation
confidence 97% · We study text-conditioned 3D part segmentation... We introduce UniPart
UniPart → trainedon → LangPart-1M
confidence 96% · UniPart is a feed-forward text-conditioned 3D part segmentation model trained on this dataset.
UniPart → uses → CLIP
confidence 95% · UniPart, a feed-forward cross-modal 3D Transformer that conditions CLIP text embedding.
LangPart-1M → derivedfrom → Objaverse
confidence 94% · we build LangPart-1M with 160K+ Objaverse assets
LangPart-4K → subsetof → LangPart-1M
confidence 93% · We further manually label a high-quality subset, LangPart-4K
UniPart → enables → Embodied Interaction
confidence 91% · transfers to language-conditioned part grasping in real world.
LangPart-1M → usesmethod → SAM
confidence 90% · Our pipeline combines multi-view rendering, SAM region proposals... to generate 8M text to part pairs
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Fine-grained robotic manipulation depends on understanding parts, not only whole objects. Existing 3D foundation models tend to be either generalized but object-aware, or part-aware but limited to closed-set taxonomies, which weakens zero-shot transfer. We study text-conditioned 3D part segmentation, where a free-form phrase selects a functional part on point cloud. We introduce UniPart, a feed-forward cross-modal 3D Transformer that conditions CLIP text embedding. To scale supervision, we build LangPart-1M with 160K+ Objaverse assets and 8M text to part pairs using multi-view consistent part generation. We further manually label a high-quality subset, LangPart-4K, for fine-tuning and evaluation. UniPart achieves strong zero-shot results on open-vocabulary part benchmarks and transfers to language-conditioned part grasping in real world.
Tags
Links
- Source: https://arxiv.org/abs/2609.12898v1
- Canonical: https://arxiv.org/abs/2609.12898v1
Trouble viewing inline? Open PDF directly →
Full Text
59,714 characters extracted from source content.
Expand or collapse full text
UniPart: Towards Zero-shot Language-Grounded 3D Part Segmentation for Embodied Interaction Xinqiang Yu 1,4,6,∗ , Zekun Qi 3,4,∗ , Jiawei He 6 , Wenyao Zhang 5 , Xuchuan Chen 3,4 , Guocai Yao 6 , Li Yi 3,† , Zhaoxiang Zhang 1,† , He Wang 2,4,† 1 NLPR, Institute of Automation, Chinese Academy of Sciences, China 2 Peking University, China 3 Tsinghua University, China 4 Galbot Inc., China 5 Shanghai Jiao Tong University, China 6 Beijing Academy of Artificial Intelligence, China ∗ Equal contribution, † Corresponding authors Fine-grained robotic manipulation depends on understanding parts, not only whole objects. Existing 3D foundation models tend to be either generalized but object-aware, or part-aware but limited to closed-set taxonomies, which weakens zero-shot transfer. We study text-conditioned 3D part segmentation, where a free-form phrase selects a functional part on point cloud. We introduce UniPart, a feed-forward cross-modal 3D Transformer that conditions CLIP text embedding. To scale supervision, we build LangPart-1M with 160K+ Objaverse assets and 8M text to part pairs using multi-view consistent part generation. We further manually label a high-quality subset, LangPart-4K, for fine-tuning and evaluation. UniPart achieves strong zero-shot results on open-vocabulary part benchmarks and transfers to language-conditioned part grasping in real world. Keywords: Zero-Shot; 3D Part Segmentation; Embodied Interaction Email: yuxinqiang,qizekun,chenxuchuan,yili,wanghe@galbot.com jwhe2024,wyzhang2022@gmail.com gcyao1@baai.ac.cn; zhaoxiang.zhang@ia.ac.cn Project Page: https://xinqiangyu.github.io/UniPart/ 1 Introduction The quest for truly general-purpose embodied in- telligence hinges on a robot’s ability to not merely perceive objects, but to comprehend their function and composition at a granular level [18,44,52,54]. In open-world settings, manipulation depends on gener- alizing to unseen objects and to the parts that matter for an action. A robot must identify the semantically relevant object part on a previously unseen instance, conditioned on a human instruction. Foundation models such as SAM [29] and CLIP [57] have made 2D perception generalizable, and several lines of work extend these priors to 3D. Yet fine- grained part understanding remains difficult [54,4, 52]. Existing approaches often fall into two patterns that leave a gap for dexterous manipulation. • Promptable but coarse Models that inherit SAM- style prompting in 3D, such as Point-SAM [87], are strong at separating object instances but do not resolve internal semantically relevant parts within one object. • Part-aware but closed-set Part and affordance models trained on datasets such as ShapeNet- Part [72] provide fine granularity, yet they rely on fixed taxonomies and struggle with open- vocabulary generalization. We frame the missing capability as a semantic in- terface for fine-grained 3D geometry. Instead of pre- dicting from a fixed label set, a robot should take a natural language instruction and return the corre- sponding 3D part mask. We study language-grounded zero-shot 3D part segmentation and tackle it with a paired design of supervision and model. On the supervision side, we build LangPart-1M. The key challenge is not only scale, but also multi-view consistency of part masks when lifting from 2D to 3D. Our pipeline combines multi-view rendering, SAM arXiv:2609.12898v1 [cs.CV] 11 Sep 2026 160K Objaverse Assets 8M Text-Part Pairs Handle Base Bit LangPart-1M Embodied Interaction Multi-View Consistent Part Generation Figure 1 Overview. We construct LangPart-1M from 160K+ Objaverse assets and generate 8M text to part pairs using a scalable multi-view consistent part generation pipeline. UniPart is a feed-forward text-conditioned 3D part segmentation model trained on this dataset. The predicted part masks provide fine-grained geometric cues that support embodied interaction. region proposals, Set-of-Mark prompting, and a text- guided merging step followed by lifted to 3D. This produces point-level part masks aligned to language at scale. We then curate LangPart-4K with man- ual verification so that training can benefit from both breadth and high-fidelity corrections. On the model side, we introduce UniPart, which is a feed- forward cross-modal 3D Transformer that injects a frozen CLIP text embedding into every layer through simple addition. This choice keeps the architecture lightweight while maintaining strong language con- ditioning. We also include a controlled study that separates the effect of the dataset from the effect of the model, which is critical for positioning against closely related work. In summary, our key contributions are: • LangPart-1M and LangPart-1K We introduce a large-scale dataset with 160K objects and 8M text-to-part pairs, and a benchmark split with 1K objects and more than 4K annotations for open-vocabulary evaluation. •Scalable data engine and high-quality subset We present a multi-view pipeline that aligns language to 3D part masks using SAM proposals, Set-of- Mark prompting, and text-guided merging for view consistency. We additionally build LangPart- 4K with ten annotators and assign text labels to part masks, providing high-fidelity language- grounded 3D part supervision. •UniPart model and controlled positioning We pro- pose a feed-forward cross-modal 3D Transformer with layer-wise additive text injection, and we report a matched supervision study that disentan- gles the gains from data and from architecture. • Embodied validation We show that UniPart can produce part masks on real sensor point clouds that are useful for language-conditioned grasping. 2 Related Works 2.1 3D Part Segmentation Classic 3D part segmentation relies on closed-set supervision, such as ShapeNetPart [72] and Part- Net [43], and achieves fine granularity within fixed taxonomies. More recent work leverages 2D vision language priors to reduce labeling cost, yet open- vocabulary generalization is still limited [34]. In parallel, promptable and open-vocabulary 3D seg- mentation extends SAM [28] and CLIP [56] into 3D through multi-view lifting or 3D scene pipelines, and it is mainly evaluated on instance or scene semantics rather than object internal parts [55,86,17]. Recent efforts also explore open-vocabulary part segmenta- tion in narrower domains or synthetic settings [65]. Our work is closest to concurrent text-prompted part segmentation such as FIND3D [38]. We share the goal of querying arbitrary parts with language, while dif- fering in two aspects that we evaluate under matched supervision. First, we construct LangPart-1M using a multi-view pipeline that enforces 3D consistency through text guided mask merging. Second, we train a feed-forward point-based model with layer-wise additive text injection, which provides dense part masks without multi-view querying at inference. We include a controlled study that isolates the effects of supervision and architecture. Related but distinct, language-guided 3D affordance segmentation focuses on action regions rather than semantic parts and often depends on task-specific definitions [66,37]. UniPart targets category-agnostic part masks as a perception primitive that can be composed into downstream manipulation. 2.2Language-Grounded Robot Manipulation Language-grounded robot Manipulation adopts the human language as a general instruction interface. Existing works can be categorized into two groups: (i) End-to-end models like RT-series [3,89,2] built upon unified cross-modal Transformers with tokenized ac- tions [58,33,82], large vision-language-action mod- els built from VLMs [27,81], or 3D representa- tions [83,75]. Training on robot data such as Open X-Embodiment [5] and DROID [26], a remarkable process has been made. However, the data scale is still limited compared to in-the-wild data for training VLMs. (i) Decoupled high-level reasoning and low- level actions in large VLMs and small off-the-shelf policy models, primitives [25,32,24,22,16,71,76, 40,20,85], or articulated priors [23,31]. Our model lies in this group, where an open-world generaliza- tion property emerges from VLMs and our proposed UniPart is empowered by fine-grained understanding. 2.3 3D Representation Learning Research on 3D Representation Learning encom- passes various methods, including point-based [48, 49], voxel-based [41], and multiview-based ap- proaches [60,19]. Point-based methods [53,14] have gained prominence in object classification [68,62] due to their sparsity yet geometry-informative rep- resentation. On the other hand, voxel-based meth- ods [8,73,51] offer dense representation and transla- tion invariance, leading to a remarkable performance in object detection [6] and segmentation [72,1]. The evolution of attention mechanisms [63] has also con- tributed to the development of effective representa- tions for downstream tasks, as exemplified by the emergence of 3D Transformers [14,36,39]. Notably, 3D self-supervised representation learning has gar- nered significant attention in recent studies. Point- Contrast [69] utilizes contrastive learning across dif- ferent views to acquire discriminative 3D scene repre- sentations. Innovations such as Point-BERT [74] and Point-MAE [45] introduce masked modeling [21,9] pretraining into the 3D domain. ACT [12] pioneers cross-modal geometry understanding through 2D or language foundation models such as CLIP [56] or BERT [9]. Following ACT, ReCon [50] further pro- poses a learning paradigm that unifies generative and contrastive learning. PPT [79] highlights the significance of positional encoding in 3D represen- tation learning. Additionally, leveraging foundation vision-language models like CLIP [12,56] has spurred the exploration of a new direction in open-world 3D representation learning. This line of work seeks to extend the applicability and adaptability of 3D rep- resentations in diverse and open-world/vocabulary scenarios [46, 78, 11, 10, 15, 80]. 3UniPart: Language-Grounded 3D Part Segmentation for Embodied Interaction 3.1 Datasets Scaling text-part alignment dataset to hundreds of thousands of 3D assets requires balancing annotation scalability and quality. Fully manual labeling at this scale is impractical, whereas fully automated anno- tation inevitably introduces noise. To address this Table 1 Properties and inference time of UniPart and baselines. We report whether each method is open-world, supports cross-category transfer, predicts part-level masks, and runs as a feed-forward 3D model without multi-view rendering and backprojection at inference. We also report a representative runtime and the number of training objects when applicable. MethodTime Open-world Cross-category Part-level Feed-forward Num. Object PointCLIPV25.4s✓✗- PartSLIP++174.3s✓✗✓✗- OpenMask3D 296.5s✓✗- PartDistill0.7s✗✓- PointNext1.4s✗✓- FIND3D0.9s✓36k UniPart(Ours)0.4s✓160k+ trade-off, we construct our dataset in two complemen- tary stages. The first subset LangPart-1M is generated through a fully automated pipeline leveraging large- scale foundation models, providing broad coverage and diversity across object categories. While scal- able, this subset may contain minor inconsistencies due to model-induced noise. To provide high-fidelity semantic supervision, we additionally curate a man- ually annotated subset LangPart-4K with carefully verified part labels, which is splited into LangPart-3K for training and LangPart-1K as benchmark. During training, we adopt a three-stage curriculum. We first align point-cloud representations with RGB features to inject rich cross-modal semantic cues into the 3D backbone, establishing an initialization that is both geometrically grounded and semantically aware. We then pretrain the model on large-scale data to acquire broad structural and semantic priors at scale. Finally, we fine-tune on the manually annotated sub- set to correct the noise of automatic supervision and sharpen part-level localization accuracy. This staged design progressively transfers knowledge from cross- modal semantic alignment to scalable representation learning, and ultimately to high-fidelity part under- standing. 3.1.1 Dataset Statistics To facilitate rigorous evaluation and training, LangPart-1M is curated with extensive coverage and fine-grained annotations. Specifically, the dataset contains over 160K+ unique 3D objects spanning more than 250 distinct object categories (e.g., furni- ture, tools, appliances, vehicles), as shown in table 1. Within these objects, we annotate over 1,000 unique part types (e.g., handle, button, spout, screen), yield- ing a total of approximately 8 million text-part pairs. We train on the remaining data after holding out the benchmark split, and report results on our proposed benchmark. 3.1.2 Data Source To enable large-scale training, we construct the LangPart-1M dataset based on Objaverse [7], which contains approximately 800K Internet-sourced 3D models across diverse categories. However, Internet- sourced models often exhibit noisy annotations and inconsistent visual quality. To obtain clean and con- sistent supervision, we re-render the assets in Blender with controlled lighting, generating over 8M high- fidelity images. 3.1.3 Data Filtering To clean the data that can be better used for gen- erating fine-grained part annotations, we first clean the data by using a dedicated filtering strategy that filters data to preserve samples that satisfy the fol- lowing six requirements.❶Clean objects without the ground for auxiliary visualization.❷Reasonable objects that have sufficient spatial reasoning poten- tials.❸High-quality objects. Low-quality objects such as blurry and wrong samples are filtered.❹ Distinguishable objects. Abstract objects such as meaningless solids are filtered.❺Non-scene objects. Samples that describe a 3D scene are filtered for object-centric understanding purposes.❻No test leakage. We proactively remove all asset IDs from evaluation benchmarks, including Objaverse-General, from the training pool. Filtering at this scale is impractical to do fully by hand. Inspired by recent work on using VLMs as image-based judges [84,67,47], we use GPT-5.2- Pro [59] with a fixed prompt and deterministic de- coding to decide whether an asset passes the cri- teria. For reproducibility, we will release the full filtering prompt, the list of retained asset IDs, and the per-asset decisions so that the training set can be reconstructed without re-running the VLM. Figure 2 Pipeline of LangPart-1M dataset generation, including Part Querying, Multi-View Segmentation and Text-Mask Alignment. 3.1.4 LangPart-1M Dataset Generation To generate the fine-grained annotations for our dataset, we devise a novel pipeline that systemat- ically leverages large-scale foundation models. As shown in figure 2, this process follows four stages: (i) Part-List Querying: For each cleaned 3D object, we first employ GPT-5.2-Pro [59] to query and gen- erate its object name and a comprehensive list of its potential functional or meaningful parts. (i) 2DUnsupervisedSegmentation For each object, we render twelve randomly sampled single-view images in Blender. For each view, we apply SAM [28] to obtain all possible segmentations, producing multiple fine-grained part candidates per image. To enable large language models to reason about these spatial regions, we employ the Set-of-Mark [70] prompting technique, assigning a unique visual identifier to each generated segment. (i) Multi-View Consistent Part Generation Lifting 2D masks into 3D often produces view-dependent inconsistencies. We address this with a text guided merging step followed by lifted to 3D. For each view, we provide the SoM labeled image to GPT-5.2-Pro and ask it to select the visible parts from the queried part list and to merge the region IDs that belong to the same part. This step improves cross-view coher- ence, merges over-segmented regions, and suppresses isolated outliers. (iv) Text to Point Cloud Alignment Finally, using the camera intrinsics, extrinsics, and depth maps from our renderer, we aggregate the back-projected points and obtain a unified point cloud with per-point part labels. To support reproducibility, we will release the intermediate artifacts needed to replay the pipeline, including render metadata, depth maps, SAM masks, SoM overlays, and the merged region sets produced during alignment. 3.1.5 Manual Annotation for Quality Improvement After data filtering in Sec. 3.1.3, we obtain over 160K 3D assets from Objaverse. However, not all objects are equally relevant for functional robotic manipula- tion. We therefore manually select 4K objects and manually construct and label a high-quality subset for post-training, named LangPart-4K by two stages. (i)Part-Aware Object Filtering In practice, robots pri- marily interact with objects that exhibit clear func- tional or structurally meaningful parts. Therefore, from the 160K objects, we further select 4K objects that contain explicit functional components or well- defined part structures. The subset is manually an- notated to provide high-fidelity supervision for the third training stage. (i)Refinement of Auto Annotation For each object in LangPart-4K, we start from the automatically gener- ated multi-view masks from Sec. 3.1.4. Annotators inspect the correspondence between each part name and its regions in every rendered view and correct mismatches when needed. This yields a verified set of part masks that is used for fine-tuning and for Figure 3 UniPart architecture for part segmentation. During the pretraining stage, UniPart is trained to align image feature and point feature. And for the finetuning stage, UniPart takes 3D & text embeddings as inputs of aN-layer plain Transformer model, and uses a token-level sum as cross-modal fusion in all layers. A PointNet decoder predicts the logits per point. constructing our benchmark. 3.1.6 Text-Alignment Part Segmentation Benchmark Existing 3D part segmentation benchmarks are lim- ited in diversity and flexibility. They cover only a small set of object categories, rely on closed-set part taxonomies, and often contain canonicalized CAD models. ShapeNetPart has 16 categories and 41 part labels [72]. PartNet-E expands to 45 categories but still uses a fixed vocabulary shared across cate- gories [43]. After manual verification in Sec. 3.1.5, we split LangPart-4K into LangPart-3K for train- ing and LangPart-1K with 1K objects for evaluation. LangPart-1K is designed to evaluate both seen and unseen categories and part types, and we will release the split specification together with the asset IDs and text prompts used for evaluation. 3.2A Cross-Modal 3D Transformer for Language-Grounded Part Segmentation We introduce UniPart, a simple Transformer-based ar- chitecture [63] with cross-modal point cloud–language fusion for open-vocabulary text-conditioned part segmentation. As illustrated in figure 3, UniPart takes an object’s point cloud and a language de- scription as input and predicts the corresponding text-conditioned segmentation mask. 3.2.1 Language and Point Cloud Embeddings Given an object point cloudX=x i ∈R 3 |i = 1,2,...,Nin Cartesian coordinates and an arbitrary language descriptionℓ, we first tokenize both modal- ities into discrete embeddings. For the 3D input, following [12,74,50], we sampleN s seed points via Farthest Point Sampling (FPS) and form local neigh- borhoods usingk-nearest Neighbors (kNN). Each neighborhood is encoded into a local geometric to- ken using a lightweight PointNet-style feature ex- tractor [48,49], yielding a sequence of 3D tokens. For language, we adopt OpenAI CLIP [56] and use its global text embedding as the conditioning sig- nal for cross-modal fusion with the 3D tokens. The fused representations are then processed by a Trans- former encoder. Finally, a MLP-style decoder head upsamples [CLS] token features back to per-point pre- dictions and outputs text-conditioned binary mask over the input points, producing an open-vocabulary part segmentation mask corresponding toℓ. Similar to Vision Transformer [13], we adopt a Large config- uration with more Transformer layers, larger hidden dimensions, and a stronger CLIP text backbone. 3.2.2 Cross-Modal Fusion We adopt a dense, layer-wise fusion strategy that injects the global text embedding into each layer of the 3D Transformer. In principle, cross-modal fusion can be implemented in many ways, e.g., cross- attention, adapters, or feature concatenation along the channel dimension. Empirically, we find that a simple additive injection—adding the text feature to every point token at each layer—yields the best trade-off between simplicity and effectiveness. We hypothesize this is because the language input is relatively short, and per-token addition strengthens language conditioning throughout the network. 3.3 Optimization LetF UP represent the UniPart model parameter- ized byθ UP (the CLIP is kept frozen and thus its parameters are not included). Given every object point cloudX i ∈ D LangPart-1M in the LangPart- 1M dataset, where each object is labeled with a part text setL i =ℓ i j ,j= 1,2,...,P i where P i is the part number of point cloudX i . And the corresponding ground truth 3D part segmenta- tion maskS i =s i j ,j= 1,2,...,P i whereS i is an one-hot label. For the second and third fine- tune stage, the optimization is to minimize the Bi- nary Cross-Entropy (BCE) loss between predicted segmentation mask and ground truth label, where L BCE ( ˆ S i j ,S i j ) =−[S i j log( ˆ S i j ) + (1− S i j )log(1− ˆ S i j )]. And the final optimization function is min θ UP P X i ∈D LangPart-1M P S i j ∈S i L BCE ( ˆ S i j ,S i j ) (1) During the first pretraining stage, for each single view point cloudX sin i ∈D LangPart-1M , and a correspond- ing RGB imageI i rendered from the same view, and a mappingπ(i)∈1,...,Mthat assigns each point to an image patch index, whereMis the image patch number. We extract patch-level CLIP features from the imagec j M j=1 and align them with per-point fea- tures produced by the point encoder, encouraging the 3D representation to inherit CLIP’s semantic structure. This initialization significantly facilitates subsequent text-guided part understanding during fine-tuning. And the loss function is L align = max 0, 1− 1 N P N i=1 ˆ z i , ˆ c π(i) (2) where ˆ z i is the normalized point feature, ˆ c π(i) is its corresponding normalized CLIP patch feature,Nis the point number, and⟨·,·⟩denotes the inner product. 3.4 Comparison with Related Works Recent work such as FIND3D [38] also studies text- prompted 3D part segmentation. We focus on two differentiators that we evaluate under matched super- vision. LangPart-1M provides large-scale point-level supervision constructed with a multi-view consistency mechanism based on text-guided merging. We report a controlled study that isolates the contribution of the supervision engine and the contribution of the architecture, and we test the resulting part masks in a real robotic grasping pipeline. 4 Experiments 4.1 Experimental Settings Benchmarks We evaluate on three settings. We re- port results on Objaverse-General [38] to compare with prior open-world baselines. We additionally evaluate on ShapeNetPart [72] and PartNet-E [43] to test transfer to classic part benchmarks. Finally, we evaluate on LangPart-1K, which is constructed from LangPart-4K with manual verification. Metric We report mean Intersection over Union. For each input point cloud, we compute the mean IoU across its annotated parts, then average over the point cloud. 4.2 Part Segmentation Baselines 4.2.1 Open-world Baselines To ensure a consistent and fair comparison, we fol- low the baseline selection and evaluation protocol in FIND3D [38] and detail the adopted methods below. We use the same two prompt formats, a composi- tional query of the formpart of a objectand a short query part. • PointCLIPV2 [88] is an open-world 2D-to-3D frame- work that repeatedly queries CLIP [56] across ren- dered views. It relies on a prompt-selection procedure that retains the top-ktextual prompts for each ob- ject, withk= 1400× n parts , where the prompt set is chosen using the ShapeNetPart test split. • PartSLIP++ [35] follows a detection-driven design, leveraging GLIP [30] together with a dedicated su- perpoint generation module to transfer 2D detections back to 3D. The method additionally trains category- specific models on PartNet-E [42]; for consistency and fairness, we report results using its released zero-shot checkpoint. • OpenMask3D [61] is an open-vocabulary 2D-to-3D approach originally trained on 3D scenes. Both Point- CLIPV2 and OpenMask3D produce dense predictions by assigning a label to every point. When the bench- mark includes unlabeled regions, we add an “other” text query to represent the no-label option. Table 2 Performance comparison with open-world methods on Objaverse-General. UniPart performs best on Objaverse- General benchmark. We show results evaluated with 2 common query prompts: “part of a object" and “part" for all methods following FIND3D [38]. MethodsObjaverse-General Seen CategoriesUnseen Categories part of a object part part of a object part PointCLIPV29.8111.2710.2711.09 PartSLIP++2.6915.030.5710.43 OpenMask3D11.8111.937.0110.31 FIND3D33.7834.1026.2127.41 UniPart(ours)46.4349.2741.4645.23 Table 3 Performance comparison with open-world methods on ShapeNetPart. UniPart also performs best on this benchmark. And we also evaluate with prompts “part of a object" and “part". MethodsShapeNetPart CanonicalRotatedShapeNetPart-V2 part of a object part part of a object part part of a object part PointCLIPV216.9120.2216.8818.1915.1417.11 PartSLIP++1.436.460.946.031.5411.62 OpenMask3D8.9410.376.7514.5615.8713.77 FIND3D28.3924.0929.6423.7142.1530.02 UniPart(ours)40.0741.7542.4339.5655.6351.29 Table 4 Comparison on LangPart-1K benchmark. MethodsLangPart-1K benchmark Single-View point cloudWhole Object point cloud part of a object part part of a object part FIND3D 10.39 8.97 14.36 17.28 UniPart(ours)31.2333.5628.6927.17 4.3 Experimental Results Results on Objaverse-General, ShapeNetPart, and PartNet-E Table 2 and Table 3 report mIoU on Objaverse-General and ShapeNetPart. As shown in this table, our method UniPart can perform bet- ter than others. As shown in Table 5, we compare open-world part segmentation methods on PartNet- E. Among these approaches, Our method has best performance. Qualitative results Figure 4 shows qualitative exam- ples from our model predictions. Our benchmark and dataset has more objects and more parts than FIND3D. And our datasets include single-view and whole point cloud. Results on LangPart-1K Our dataset provides both single-view point clouds and complete 3D point clouds. Accordingly, we train two separate mod- els—one tailored to partial observations and the other operating on full geometry. For a fair and compre- hensive evaluation, we further report results under both test protocols, as shown in Table 4. And Our method has best performance. 4.4 Fine-Grained Robotic Manipulation 4.4.1 Real World Robot Setups We perform object manipulation tasks using the Franka Panda equipped with a gripper and the UR robot arm with a Shadow Hand. All the robot arms mount a RealSense D415 camera at their end for im- age capturing. In figure 5, we present the workspace and robotics for real-world object manipulation. We utilize only a single D415 camera. This setup signif- icantly reduces the additional overhead associated with environmental setup and multi-camera calibra- tion, and it is more readily reproducible. Figure 4 Qualitative results and our dataset. a) Examples of our dataset. b) Our benchmark has more objects and parts than FIND3D. c) Our datasets has single-view or whole point cloud, more objects and more pairs. Table 5 Comparison of open-world methods on PartNet-E. MethodsCanonical OrientationRotated part of a object part part of a object part Find3D16.8616.3817.6217.16 PartSLIP++5.1232.713.8723.03 PointCLIPV211.289.7010.3210.22 OpenMask3D12.5411.2411.9311.67 UniPart(ours)23.1522.9024.2821.72 4.4.2Language-Conditioned Grasping in Real-World Scenarios We evaluate robotic grasping tasks to test whether UniPart produces usable part masks on real sensor point clouds. We integrate UniPart with off-the- shelf grasp generators, GraspNet [64] and DexGrasp- Net2.0 [77]. Given a language query, UniPart predicts a target part point cloud, which is then used as the input to the grasp generator to execute a grasp. Quantitative Results We evaluate 20 real objects with language prompts that specify a target part. We report part segmentation accuracy by inspection of the predicted part, and we report grasp success as the fraction of successful grasps that land on the intended part. UniPart achieves 90.5% part segmentation accuracy and the downstream pipeline achieves 85.0% grasp success. Qualitative Results UniPart delineates parts such as handles and grips under sensor noise and occlusion, which supports its use as a perception module for language-conditioned grasping. Table 6 Controlled study evaluated on Objaverse-General (zero-shot mIoU). Both our dataset and architecture con- tribute significantly to the final performance. Model Architecture Training Data mIoU FIND3DFIND3D Data30.75 UniPart(Ours)FIND3D Data35.27 FIND3DLangPart-1M (Ours) 38.42 UniPart(Ours)LangPart-1M (Ours) 47.25 4.5 Ablation Study 4.5.1 Controlled Study To strictly isolate the performance gains brought by our proposed architecture and our large-scale dataset, we conduct a controlled cross-evaluation against the FIND3D [38] baseline on the Objaverse-General zero- shot benchmark. We evaluate four settings: (1) FIND3D model trained on FIND3D data, (2) Uni- Part trained on FIND3D data, (3) FIND3D model trained on LangPart-1M, and (4) UniPart trained on LangPart-1M. As shown in table 6, when restricted to the same Franka RealSense D415 UR5 ShadowHand Figure 5 Object Manipulation robot setup. Table 7 Ablation study of pretraining for UniPart. mIoU with pretrain + partial PC33.56 with pretrain + complete PC27.17 without pretrain + partial PC26.34 without pretrain + complete PC 23.39 Table 8 Ablation study of multi-modal fusion. Fusion Method mIoU Cross-attn29.22 Multiplication 28.13 Concat30.31 Addition33.56 FIND3D training data, UniPart still outperforms the FIND3D model by 14.69% mIoU, demonstrating the superior cross-modal alignment capability of our ad- ditive text-injection Transformer. Conversely, when the FIND3D model is trained on our LangPart-1M dataset, its performance improves by 24.94% mIoU, validating the high quality and scalability of our au- tomated data engine. Ultimately, UniPart trained on LangPart-1M achieves the best performance, proving that our architecture is uniquely suited to absorb the rich supervision from large-scale text-part data. 4.5.2 Ablation study on Pretraining To isolate the contribution of the proposed cross- modal pretraining, we conduct ablations under both partial and complete point-cloud settings. Although the pretraining stage is performed only on single-view partial point clouds (as it relies on point-to-image patch correspondences), we evaluate its effect on models trained and tested with partial inputs as well as on models operating on complete geometry. As shown in table 7, pretraining on single-view partial point clouds still improves fine-tuning on complete point clouds by transferring CLIP semantics into the 3D encoder. Gains are typically larger for partial inputs but remain consistent on complete geometry. 4.5.3 Cross-Modal Fusion Choices We further conduct an ablation study on the multi- modal fusion methods in UniPart, testing commonly used feature fusion techniques such as cross-attention, multiplication, addition, and concatenation, as shown in table 8. The results indicate that simple addi- tion achieves the best performance. This may be attributed to the fact that instructions in the seman- tic domain are typically composed of short phrases or sentences, and the text CLS token already encodes sufficiently high-level semantic information. 5 Conclusion and Future Work We introduced UniPart for zero-shot, language- grounded 3D part segmentation. Our goal is to provide a simple semantic interface that maps a nat- ural language phrase to a fine-grained 3D part mask, which is a useful primitive for embodied interaction. A key enabler is LangPart-1M, a large-scale resource with 8 million text to part alignments on 160K+ ob- jects, constructed with a multi-view pipeline that improves 3D consistency through text guided merg- ing. We further curate LangPart-4K with manual verification and establish LangPart-1K for evaluation. UniPart, a feed-forward cross-modal 3D Transformer with layer-wise additive text conditioning, achieves strong generalization across open-vocabulary bench- marks and produces usable part masks on real sensor point clouds for language-conditioned grasping. Fu- ture work will extend part understanding beyond isolated objects and strengthen closed-loop integra- tion with robotic control. References [1] Iro Armeni, Ozan Sener, Amir R. Zamir, Helen Jiang, Ioannis K. Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 1534–1543. IEEE Computer Society, 2016. [2]Suneel Belkhale, Tianli Ding, Ted Xiao, Pierre Ser- manet, Quon Vuong, Jonathan Tompson, Yevgen Chebotar, Debidatta Dwibedi, and Dorsa Sadigh. RT-H: action hierarchies using language. CoRR, abs/2403.01823, 2024. [3] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexan- der Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J. Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mor- datch, Ofir Nachum, Carolina Parada, Jodilyn Per- alta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael S. Ryoo, Grecia Salazar, Pan- nag R. Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong T. Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-1: robotics transformer for real-world control at scale. In Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023. [4]Hengshuo Chu, Xiang Deng, Qi Lv, Xiaoyang Chen, Yinchuan Li, Jianye Hao, and Liqiang Nie. 3d- affordancellm: Harnessing large language models for open-vocabulary affordance detection in 3d worlds. arXiv preprint arXiv:2502.20041, 2025. [5]Open X.-Embodiment Collaboration, Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bew- ley, Alexander Herzog, Alex Irpan, Alexander Khaz- atsky, Anant Raj, Anikait Singh, Anthony Bro- han, Antonin Raffin, Ayzaan Wahid, Ben Burgess- Limerick, Beomjoon Kim, Bernhard Schölkopf, Brian Ichter, Cewu Lu, Charles Xu, Chelsea Finn, Chen- feng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Chuer Pan, Chuyuan Fu, Coline Devin, Danny Driess, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Federico Ceola, Fei Xia, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Giulio Schiavi, Gregory Kahn, Hao Su, Haoshu Fang, Haochen Shi, Heni Ben Amor, Henrik I. Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Igor Mordatch, Ilija Radosavovic, and et al. Open x- embodiment: Robotic learning datasets and RT-X models. CoRR, abs/2310.08864, 2023. [6] Angela Dai, Angel X. Chang, Manolis Savva, Ma- ciej Halber, Thomas A. Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstruc- tions of indoor scenes. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 2432–2443. IEEE Computer Society, 2017. [7]Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Van- couver, BC, Canada, June 17-24, 2023, pages 13142– 13153. IEEE, 2023. [8]Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel R-CNN: towards high performance voxel-based 3d object detection. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 1201–1209. AAAI Press, 2021. [9]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguis- tics, 2019. [10]Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. PLA: language- driven open-vocabulary 3d scene understanding. In CVPR, 2023. [11] Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. Lowis3d: Language-driven open-world instance-level 3d scene understanding. IEEE Trans. Pattern Anal. Mach. Intell., 46(12):8517–8533, 2024. [12] Runpei Dong, Zekun Qi, Linfeng Zhang, Junbo Zhang, Jianjian Sun, Zheng Ge, Li Yi, and Kaisheng Ma. Autoencoders as cross-modal teachers: Can pre- trained 2d image transformers help 3d representation learning? In ICLR, 2023. [13]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. [14]Nico Engel, Vasileios Belagiannis, and Klaus Diet- mayer. Point transformer. IEEE Access, 9:134826– 134840, 2021. [15]Guofan Fan, Zekun Qi, Wenkai Shi, and Kaisheng Ma. Point-gcc: Universal self-supervised 3d scene pre-training via geometry-color contrast. In Jian- fei Cai, Mohan S. Kankanhalli, Balakrishnan Prab- hakaran, Susanne Boll, Ramanathan Subramanian, Liang Zheng, Vivek K. Singh, Pablo César, Lexing Xie, and Dong Xu, editors, Proceedings of the 32nd ACM International Conference on Multimedia, M 2024, Melbourne, VIC, Australia, 28 October 2024 - 1 November 2024, pages 4709–4718. ACM, 2024. [16]Kuan Fang, Fangchen Liu, Pieter Abbeel, and Sergey Levine. MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting. In Pro- ceedings of Robotics: Science and Systems, Delft, Netherlands, July 2024. [17]Marco Garosi, Riccardo Tedoldi, Davide Boscaini, Massimiliano Mancini, Nicu Sebe, and Fabio Poiesi. 3d part segmentation via geometric aggregation of 2d visual features. In 2025 IEEE/CVF Winter Confer- ence on Applications of Computer Vision (WACV), pages 3257–3267. IEEE, 2025. [18]James J Gibson. The theory of affordances:(1979). In The people, place, and space reader, pages 56–60. Routledge, 2014. [19]Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem. MVTN: multi-view transformation net- work for 3d shape recognition. In ICCV, pages 1–11. IEEE, 2021. [20]Jiawei He, Danshi Li, Xinqiang Yu, Zekun Qi, Wenyao Zhang, Jiayi Chen, Zhaoxiang Zhang, Zhizheng Zhang, Li Yi, and He Wang. Dexvlg: Dex- terous vision-language-grasp model at scale. CoRR, abs/2507.02747, 2025. [21]Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022. [22] Haoxu Huang, Fanqi Lin, Yingdong Hu, Shengjie Wang, and Yang Gao. Copa: General robotic ma- nipulation through spatial constraints of parts with foundation models. CoRR, abs/2403.08248, 2024. [23]Siyuan Huang, Haonan Chang, Yuhan Liu, Yimeng Zhu, Hao Dong, Abdeslam Boularias, Peng Gao, and Hongsheng Li. A3VLM: actionable articulation- aware vision language model. In Pulkit Agrawal, Oliver Kroemer, and Wolfram Burgard, editors, Con- ference on Robot Learning, 6-9 November 2024, Mu- nich, Germany, volume 270 of Proceedings of Ma- chine Learning Research, pages 1675–1690. PMLR, 2024. [24]Wenlong Huang, Chen Wang, Ruohan Zhang, Yun- zhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Com- posable 3d value maps for robotic manipulation with language models. In CoRL, 2023. [25]Brian Ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, Dmitry Kalashnikov, Sergey Levine, Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanet, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Mengyuan Yan, Noah Brown, Michael Ahn, Omar Cortes, Nicolas Sievers, Clayton Tan, Sichun Xu, Diego Reyes, Jarek Retting- house, Jornell Quiambao, Peter Pastor, Linda Luu, Kuang-Huei Lee, Yuheng Kuang, Sally Jesmonth, Nikhil J. Joshi, Kyle Jeffrey, Rosario Jauregui Ru- ano, Jasmine Hsu, Keerthana Gopalakrishnan, By- ron David, Andy Zeng, and Chuyuan Kelly Fu. Do as I can, not as I say: Grounding language in robotic affordances. In Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, volume 205 of Proceedings of Machine Learning Re- search, pages 287–318. PMLR, 2022. [26]Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ash- win Balakrishna, Sudeep Dasari, Siddharth Karam- cheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Youngwoon Lee, Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Al- bert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pannag R. Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Bai- jal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, and et al. DROID: A large-scale in-the-wild robot manipulation dataset. CoRR, abs/2403.12945, 2024. [27]Moo Jin Kim, Karl Pertsch, Siddharth Karam- cheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Paul Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Ben- jamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open-source vision-language-action model. CoRR, abs/2406.09246, 2024. [28]Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 3992–4003. IEEE, 2023. [29] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023. [30]Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Li- juan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10965–10975, 2022. [31]Xiaoqi Li, Mingxu Zhang, Yiran Geng, Haoran Geng, Yuxing Long, Yan Shen, Renrui Zhang, Jiaming Liu, and Hao Dong. Manipllm: Embodied multimodal large language model for object-centric robotic ma- nipulation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seat- tle, WA, USA, June 16-22, 2024, pages 18061–18070. IEEE, 2024. [32]Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model pro- grams for embodied control. In IEEE International Conference on Robotics and Automation, ICRA 2023, London, UK, May 29 - June 2, 2023, pages 9493– 9500. IEEE, 2023. [33]Hao Liu, Lisa Lee, Kimin Lee, and Pieter Abbeel. Instruction-following agents with jointly pre-trained vision-language models. CoRR, abs/2210.13431, 2022. [34]Minghua Liu, Yinhao Zhu, Hong Cai, Shizhong Han, Zhan Ling, Fatih Porikli, and Hao Su. Partslip: Low-shot part segmentation for 3d point clouds via pretrained image-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 21736–21746, 2023. [35]Minghua Liu, Yinhao Zhu, Hong Cai, Shizhong Han, Zhan Ling, Fatih Porikli, and Hao Su. Partslip: Low-shot part segmentation for 3d point clouds via pretrained image-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 21736–21746, 2023. [36]Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-free 3d object detection via trans- formers. In 2021 IEEE/CVF International Con- ference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 2929–2938. IEEE, 2021. [37] Dongyue Lu, Lingdong Kong, Tianxin Huang, and Gim Hee Lee. Geal: Generalizable 3d affordance learning with cross-modal consistency. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 1680–1690, 2025. [38]Ziqi Ma, Yisong Yue, and Georgia Gkioxari. Find any part in 3d. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7818–7827, 2025. [39]Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng, Xiaodan Liang, Hang Xu, and Chunjing Xu. Voxel transformer for 3d object detection. In 2021 IEEE/CVF International Conference on Com- puter Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 3144–3153. IEEE, 2021. [40]Weixin Mao, Weiheng Zhong, Zhou Jiang, Dong Fang, Zhongyue Zhang, Zihan Lan, Fan Jia, Tiancai Wang, Haoqiang Fan, and Osamu Yoshie. Robo- matrix: A skill-centric hierarchical framework for scalable robot task planning and execution in open- world. CoRR, abs/2412.00171, 2024. [41]Daniel Maturana and Sebastian A. Scherer. Voxnet: A 3d convolutional neural network for real-time ob- ject recognition. In IROS, pages 922–928. IEEE, 2015. [42]Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pages 909–918, 2019. [43]Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understand- ing. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 909–918. Computer Vision Foundation / IEEE, 2019. [44]Soroush Nasiriany, Sean Kirmani, Tianli Ding, Laura Smith, Yuke Zhu, Danny Driess, Dorsa Sadigh, and Ted Xiao. Rt-affordance: Affordances are versatile intermediate representations for robot manipulation. CoRR, abs/2411.02704, 2024. [45]Yatian Pang, Wenxiao Wang, Francis E. H. Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supervised learning. In ECCV, 2022. [46]Songyou Peng, Kyle Genova, Chiyu "Max" Jiang, Andrea Tagliasacchi, Marc Pollefeys, and Thomas A. Funkhouser. Openscene: 3d scene understanding with open vocabularies. In CVPR, 2023. [47] Yuang Peng, Yuxin Cui, Haomiao Tang, Zekun Qi, Runpei Dong, Jing Bai, Chunrui Han, Zheng Ge, Xiangyu Zhang, and Shu-Tao Xia. Dreambench++: A human-aligned benchmark for personalized image generation. CoRR, abs/2406.16855, 2024. [48]Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, pages 77–85, 2017. [49]Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In NIPS, pages 5099–5108, 2017. [50] Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge, Xi- angyu Zhang, Kaisheng Ma, and Li Yi. Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining. In ICML, 2023. [51]Zekun Qi, Muzhou Yu, Runpei Dong, and Kaisheng Ma. VPP: efficient conditional 3d generation via voxel-point progressive representation. In NeurIPS, 2023. [52]Zekun Qi, Runpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge, Li Yi, and Kaisheng Ma. Shapellm: Universal 3d object understand- ing for embodied interaction. In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part XLIII, volume 15101 of Lecture Notes in Com- puter Science, pages 214–238. Springer, 2024. [53]Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Abed Al Kader Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisit- ing pointnet++ with improved training and scaling strategies. In NeurIPS, 2022. [54]Shengyi Qian, Weifeng Chen, Min Bai, Xiong Zhou, Zhuowen Tu, and Li Erran Li. Affordancellm: Grounding affordance from vision language mod- els. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7587–7597, 2024. [55]Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d lan- guage gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20051–20060, 2024. [56] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language su- pervision. In ICML, volume 139 of Proceedings of Ma- chine Learning Research, pages 8748–8763. PMLR, 2021. [57] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natu- ral language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021. [58]Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, volume 205 of Proceedings of Machine Learning Research, pages 785–799. PMLR, 2022. [59]Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Anan- thram, et al. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267, 2025. [60]Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik G. Learned-Miller. Multi-view convolu- tional neural networks for 3d shape recognition. In ICCV, 2015. [61]Ayça Takmaz, Elisabetta Fedele, Robert W. Sumner, Marc Pollefeys, Federico Tombari, and Francis Engel- mann. OpenMask3D: Open-Vocabulary 3D Instance Segmentation. In Advances in Neural Information Processing Systems (NeurIPS), 2023. [62]Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Duc Thanh Nguyen, and Sai-Kit Yeung. Re- visiting point cloud classification: A new benchmark dataset and classification model on real-world data. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 1588–1597. IEEE, 2019. [63]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998–6008, 2017. [64] Chenxi Wang, Haoshu Fang, Minghao Gou, Hongjie Fang, Jin Gao, and Cewu Lu. Graspness discovery in clutters for fast and accurate grasp detection. CoRR, abs/2406.11142, 2024. [65]Hongyu Wu, Pengwan Yang, Yuki M Asano, and Cees GM Snoek. Segment any 3d-part in a scene from a sentence. arXiv preprint arXiv:2506.19331, 2025. [66]Hongyu Wu, Pengwan Yang, Yuki M Asano, and Cees GM Snoek. Segment any 3d-part in a scene from a sentence. arXiv preprint arXiv:2506.19331, 2025. [67]Tong Wu, Guandao Yang, Zhibing Li, Kai Zhang, Ziwei Liu, Leonidas J. Guibas, Dahua Lin, and Gor- don Wetzstein. Gpt-4v(ision) is a human-aligned evaluator for text-to-3d generation. In CVPR, 2024. [68]Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 1912–1920. IEEE Com- puter Society, 2015. [69]Saining Xie, Jiatao Gu, Demi Guo, Charles R. Qi, Leonidas J. Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud under- standing. In ECCV, volume 12348 of Lecture Notes in Computer Science, pages 574–591. Springer, 2020. [70]Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in GPT-4V. CoRR, abs/2310.11441, 2023. [71]Sherry Yang, Yilun Du, Seyed Kamyar Seyed Ghasemipour, Jonathan Tompson, Leslie Pack Kael- bling, Dale Schuurmans, and Pieter Abbeel. Learn- ing interactive real-world simulators. In The Twelfth International Conference on Learning Representa- tions, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [72]Li Yi, Vladimir G. Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas J. Guibas. A scalable active framework for region annotation in 3d shape collections. ACM Trans. Graph., 35(6):210:1–210:12, 2016. [73]Li Yi, Hao Su, Xingwen Guo, and Leonidas J. Guibas. Syncspeccnn: Synchronized spectral CNN for 3d shape segmentation. In CVPR, 2017. [74]Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point model- ing. In CVPR, 2022. [75]Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: A vision-language model for spatial affordance predic- tion for robotics. CoRR, abs/2406.10721, 2024. [76]Zhecheng Yuan, Tianming Wei, Shuiqi Cheng, Gu Zhang, Yuanpei Chen, and Huazhe Xu. Learn- ing to manipulate anywhere: A visual generalizable framework for reinforcement learning. In Pulkit Agrawal, Oliver Kroemer, and Wolfram Burgard, ed- itors, Conference on Robot Learning, 6-9 November 2024, Munich, Germany, volume 270 of Proceed- ings of Machine Learning Research, pages 1815–1833. PMLR, 2024. [77]Jialiang Zhang, Haoran Liu, Danshi Li, XinQiang Yu, Haoran Geng, Yufei Ding, Jiayi Chen, and He Wang. Dexgraspnet 2.0: Learning generative dexterous grasping in large-scale synthetic cluttered scenes. In 8th Annual Conference on Robot Learning, 2024. [78] Junbo Zhang, Runpei Dong, and Kaisheng Ma. CLIP-FO3D: learning free open-world 3d scene rep- resentations from 2d dense CLIP. In IEEE/CVF In- ternational Conference on Computer Vision, ICCV 2023 - Workshops, Paris, France, October 2-6, 2023, pages 2040–2051. IEEE, 2023. [79] Shaochen Zhang, Zekun Qi, Runpei Dong, Xi- uxiu Bai, and Xing Wei. Positional prompt tun- ing for efficient 3d representation learning. CoRR, abs/2408.11567, 2024. [80]Wenyao Zhang, Hongsi Liu, Bohan Li, Jiawei He, Zekun Qi, Yunnan Wang, Shengyang Zhao, Xinqiang Yu, Wenjun Zeng, and Xin Jin. Hybrid-grained fea- ture aggregation with coarse-to-fine language guid- ance for self-supervised monocular depth estimation. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV), pages 6678– 6692, October 2025. [81]Wenyao Zhang, Hongsi Liu, Zekun Qi, Yunnan Wang, Xinqiang Yu, Jiazhao Zhang, Runpei Dong, Jiawei He, He Wang, Zhizheng Zhang, Li Yi, Wenjun Zeng, and Xin Jin. Dreamvla: A vision-language-action model dreamed with comprehensive world knowledge. CoRR, abs/2507.04447, 2025. [82]Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manip- ulation with low-cost hardware. In Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023. [83]Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: A 3d vision-language-action generative world model. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. [84]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm- as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Pro- cessing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. [85] Enshen Zhou, Qi Su, Cheng Chi, Zhizheng Zhang, Zhongyuan Wang, Tiejun Huang, Lu Sheng, and He Wang. Code-as-monitor: Constraint-aware vi- sual programming for reactive and proactive robotic failure detection. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 6919–6929, 2025. [86]Yuchen Zhou, Jiayuan Gu, Tung Yen Chiang, Fanbo Xiang, and Hao Su. Point-sam: Promptable 3d segmentation model for point clouds. arXiv preprint arXiv:2406.17741, 2024. [87]Yuchen Zhou, Jiayuan Gu, Tung Yen Chiang, Fanbo Xiang, and Hao Su. Point-sam: Promptable 3d segmentation model for point clouds. arXiv preprint arXiv:2406.17741, 2024. [88]Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyu Guo, Ziyao Zeng, Zipeng Qin, Shanghang Zhang, and Peng Gao. Pointclip V2: prompting CLIP and GPT for powerful 3d open-world learning. In IEEE/CVF In- ternational Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 2639– 2650. IEEE, 2023. [89] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Ste- fan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong T. Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, Michael S. Ryoo, Krista Rey- mann, Kanishka Rao, Karl Pertsch, Igor Mordatch, Henryk Michalewski, Yao Lu, Sergey Levine, Lisa Lee, Tsang-Wei Edward Lee, Isabel Leal, Yuheng Kuang, Dmitry Kalashnikov, Ryan Julian, Nikhil J. Joshi, Alex Irpan, Brian Ichter, Jasmine Hsu, Alexan- der Herzog, Karol Hausman, Keerthana Gopalakr- ishnan, Chuyuan Fu, Pete Florence, Chelsea Finn, Kumar Avinava Dubey, Danny Driess, Tianli Ding, Krzysztof Marcin Choromanski, Xi Chen, Yevgen Chebotar, Justice Carbajal, Noah Brown, Anthony Brohan, Montserrat Gonzalez Arenas, and Kehang Han. RT-2: vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, CoRL 2023, 6-9 November 2023, Atlanta, GA, USA, volume 229 of Proceedings of Ma- chine Learning Research, pages 2165–2183. PMLR, 2023.