Paper deep dive
GROW$^2$: Grounding Which and Where for Robot Tool Use
Yuhong Deng, Yuyao Liu, David Hsu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 99%
Last extracted: 7/5/2026, 4:33:37 AM
Summary
GROW^2 (GROunding Which and Where) is a hierarchical framework for open-world affordance grounding in robot tool use. It decomposes the problem into two levels: a semantic level using Vision-Language Models (VLMs) to select the appropriate tool and task-relevant parts (e.g., blade, handle), and a geometric level using vision foundation models to ground these parts into precise 3D regions from a single RGB-D view. The method utilizes 3D reconstruction (via SAM3D) and multi-view 2D segmentation/fusion to handle occlusions and achieve zero-shot generalization. Experiments demonstrate that GROW^2 outperforms state-of-the-art baselines on affordance prediction benchmarks (AGD20K, PIAD) and achieves higher success rates in simulated and real-world robot tool-use tasks.
Entities (8)
Relation Signals (7)
GROW^2 â decomposesinto â semantic level
confidence 100% ¡ decompose affordance grounding into two levels: (i) semantic selection... (ii) geometric grounding
GROW^2 â decomposesinto â geometric level
confidence 100% ¡ decompose affordance grounding into two levels: (i) semantic selection... (ii) geometric grounding
GROW^2 â evaluatedon â AGD20K
confidence 100% ¡ We evaluate GROW 2 on both existing affordance prediction benchmarks... AGD20K
GROW^2 â evaluatedon â PIAD
confidence 100% ¡ and PIAD for 3D point clouds
GROW^2 â uses â SAM3
confidence 100% ¡ we use SAM3 to segment these objects conditioned on their names
GROW^2 â uses â SAM3D
confidence 100% ¡ feed the RGB-D crop into SAM3D for reconstruction
GROW^2 â uses â Vision-Language Models
confidence 100% ¡ At the semantic level, GROW 2 leverages the reasoning capabilities of vision-language models (VLMs)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Can the robot use a plate to cut a cake if no knife is available? Tool use greatly expands robot capabilities, but to use tools creatively beyond their intended functions, the robot faces the challenge of $\textit{open-world affordance grounding}$: select an open-category object to act as a tool and localize its specific region of action. To this end, we introduce GROW$^2$ (GROunding Which and Where), which leverages object parts as a natural abstraction to split the grounding process hierarchically into semantic and geometric levels, thus bypassing the need for data-heavy, end-to-end training. Semantically, GROW$^2$ harnesses the commonsense reasoning of Vision-Language Models (VLMs) to parse a natural-language task instruction, select a suitable object as the tool, and identify task-relevant parts on the tool and the target object. Geometrically, vision foundation models then ground the selected parts into precise 3D regions from a single RGB-D image. Experiments on established benchmarks show that GROW$^2$ outperforms state-of-the-art baselines on affordance prediction benchmarks. Further, it achieves zero-shot generalization over open-category objects and outperforms baselines in both simulated and real-world robot tool use experiments.
Tags
Links
- Source: https://arxiv.org/abs/2606.30632v1
- Canonical: https://arxiv.org/abs/2606.30632v1
Trouble viewing inline? Open PDF directly â
Full Text
75,144 characters extracted from source content.
Expand or collapse full text
GROW 2 : Grounding Which and Where for Robot Tool Use Yuhong Deng 1 * Yuyao Liu 12 â David Hsu 1 1 National University of Singapore 2 Massachusetts Institute of Technology Abstract: Can the robot use a plate to cut a cake if no knife is available? Tool use greatly expands robot capabilities, but to use tools creatively beyond their in- tended functions, the robot faces the challenge of open-world affordance ground- ing: select an open-category object to act as a tool and localize its specific region of action. To this end, we introduce GROW 2 (GROunding Which and Where), which leverages object parts as a natural abstraction to split the grounding process hierarchically into semantic and geometric levels, thus bypassing the need for data-heavy, end-to-end training. Semantically, GROW 2 harnesses the common- sense reasoning of Vision-Language Models (VLMs) to parse a natural-language task instruction, select a suitable object as the tool, and identify task-relevant parts on the tool and the target object. Geometrically, vision foundation models then ground the selected parts into precise 3D regions from a single RGB-D image. Experiments on established benchmarks show that GROW 2 outperforms state-of- the-art baselines on affordance prediction benchmarks. Further, it achieves zero- shot generalization over open-category objects and outperforms baselines in both simulated and real-world robot tool use experiments. Keywords: 3D affordance, Robot tool use 1 Introduction Humans possess a remarkable capacity to use tools in improvised ways: from a cluttered scene of everyday objects, we can identify which object can serve as the needed tool and apply it through the right interaction [1, 2, 3] (Fig. 1). For example, when a knife is unavailable, we may cut a cake with the edge of a plate. We aim to bring this capability to robots, expanding their feasible manipulation strategies and improving robustness in unstructured environments such as homes and warehouses. To improvise tool use, the robot has to reason about the environment and figure out which object can serve as the tool and where the relevant interaction regions are on that object in order to perform the desired action. Unlike previous works on robot tool use, we do not assume that the tool object is already specified [4, 5, 6], or that we have a fixed set of tools with predefined tool-function mappings [7, 8, 9]. Instead, this work addresses the problem of open-world affordance grounding: selecting a tool from a set of open-category objects in the scene and predicting its affordance regions. There are two main challenges in open-world affordance grounding. (i) The system needs to seman- tically understand affordances of open-category objects. For example, to cut a cake, a robot should identify an object with a sharp edge, while preferring a plate over a dustpan despite their similar local geometry. (i) The affordances must be grounded to precise geometric regions for downstream manipulation. This grounding should be conditioned on the task and the environment, and should generalize to open-category objects. Traditionally, people build a large-scale dataset that maps tasks to affordance regions and train an end-to-end neural network [10, 11, 12], which is costly and strug- gles to generalize to unseen task-object pairs. Our key insight is that object parts, such as a blade, rim, and tip, provide a useful abstraction that captures both affordance semantics and the corresponding geometric structure. Using parts as an â Equal contribution. yuhongdeng@u.nus.edu. arXiv:2606.30632v1 [cs.RO] 29 Jun 2026 grasp pound âpound the bug.â 3D Affordance GroundingSemantic Selection âcutthe cake.ââliftthe mug.ââsweepthe trash.â âpour into the bowl.â Observation & Instruction ... bugslipperrobot hand solestrap poundgrasp paintbrush bristles ferrulehandle screwdriver shafthandle Figure 1: GROW 2 (GROunding Which and Where) decomposes open-world affordance ground- ing into two levels: (i) semantic selection of an appropriate tool and the task-relevant parts, and (i) grounding the selected parts into 3D affordance regions. This decomposition enables GROW 2 to leverage a series of foundation models and generalize to open-category objects. intermediate representation, we propose GROW 2 (GROunding Which and Where) and decompose affordance grounding into two levels. At the semantic level, GROW 2 leverages the reasoning ca- pabilities of vision-language models (VLMs) to select an appropriate tool and task-relevant parts. At the geometric level, GROW 2 grounds the selected parts into 3D affordance regions using vision foundation models. This decomposition eliminates dependence on large-scale affordance annota- tions and lets GROW 2 generalize to open-category objects. Grounding these selected parts in 3D from a single view remains challenging because object parts are often only partially visible or are occluded by other objects or by the object itself. Although multi- view camera setups can alleviate this issue, they are often costly [13, 14, 15]. GROW 2 addresses this challenge by combining 2D part grounding with 3D reconstruction. Given a single-view RGB- D observation, it reconstructs a high-fidelity object mesh, registers the mesh to the observed scene, renders the mesh from multiple viewpoints, segments the selected parts in each rendered view, and fuses the resulting masks into complete 3D affordance regions. This enables GROW 2 to recover function-faithful 3D affordances from only a single RGB-D view. We evaluate GROW 2 on both existing affordance prediction benchmarks and a newly constructed benchmark that introduces the additional challenge of tool selection. GROW 2 outperforms state-of- the-art baselines and generalizes well to open-category objects. We further evaluate the predicted affordances in downstream robot tool-use experiments, including a simulation benchmark with five task types, 85 object assets, and 100 scenes, as well as real-world experiments on a Franka arm. Across these settings, GROW 2 achieves higher task success rates than the baselines, demonstrating that its grounded affordance regions effectively support robot tool use. 2 Related Work 2.1 Affordance Prediction for Robot Manipulation Understanding affordances [16, 17, 18] is fundamental for robot interaction in unstructured envi- ronments. Keypoint-based methods represent affordances as sparse function points [19, 20, 21, 22], which are efficient for learning and planning but limited in expressiveness, whereas 2D affordance- region methods [23, 12] integrate well with vision foundation models [24, 25, 26] but lack the geo- metric information needed for manipulation. We instead ground 3D affordance regions. End-to-end 2 3D affordance prediction methods [11, 27] often require substantial 3D training data and generalize poorly to open-category objects. Moreover, most of them assume complete 3D observations [28], which is unrealistic. Multi-view fusion methods [14, 15] aggregate 2D affordance predictions across views to leverage foundation models for generalization and to mitigate occlusion. However, the multi-view setting remains inefficient and impractical for real-world deployment. GROW 2 uses part representations to decompose affordance grounding, allowing us to take advantage of different foundation models for generalization. In addition, it can produce complete and function-faithful 3D affordance regions from a single-view observation, without requiring multi-view sensing. 2.2 Robot Tool Use Tool use has been widely studied in robotics and is crucial for enabling robots to extend their ca- pabilities. Prior work equips robots with advanced tool-use skills such as cutting [29, 6], fastening a nut [30], spreading sauce [31], and using chopsticks [32]. Despite this progress, most existing approaches assume that the tool and its function are pre-specified and focus primarily on generating actions under constraints [33, 5, 34, 35]. However, in unstructured environments such as homes and warehouses, effective tool use requires the ability to first identify an appropriate object as a tool and then use it to complete the desired task [1, 36]. For tool selection, researchers have attempted to harness geometric information to guide the selection process via mathematical analysis [37, 7, 38] and data-driven approaches [39, 8]. However, these methods often rely on heavy supervision or in-domain data, which limits open-world generalization, and they may overlook the semantic under- standing needed to infer tool function. In contrast, our method leverages the commonsense knowl- edge of VLMs to select tools and task-relevant parts in semantic space, and then grounds the selected parts into 3D affordances, supporting effective robot tool use. 3 Problem Statement We address the problem of open-world affordance grounding in robot tool use. An affordance de- scribes an action possibility arising from the complementarity between an agent and its environ- ment [40]. For robot tool use, we extend this notion beyond robot-object interactions to capture object-object interactions between a tool and a target. We define an affordance as an asymmetric binary relation between two entities that specifies how they interact to realize a particular action. To ground this relation spatially, we localize corresponding interaction regions on the two entities. For robot tool use, we ground two affordances: the relation between the robot hand and the tool ob- ject o A , which enables grasping, and the relation between the tool object o A and the target object o B , which enables the tool function. The input consists of (i) a single-view RGB-D observation (I,D) containing objectso i N i=1 and (i) a language instruction l describing the task and the target object o B (e.g., cut the cake). The output consists of the selected tool object o A and the corresponding grounded 3D interaction regions. 4 Method We propose GROW 2 (GROunding Which and Where), a novel approach for open-world affordance grounding. The key challenge is the generalization over open-category objects, where end-to-end mapping from tasks to affordances often struggles [27, 26]. To address this, we introduce object parts such as blades, rims, and handles that serve as intermediate cues. Using object parts, we decompose the affordance grounding into two levels (Fig. 1). Specifically, given a language instruction l and a single-view RGB-D observation (I,D) of the environment, GROW 2 first selects an appropriate tool object and the target object, along with the names of task-relevant parts. It then grounds these selected part names into 3D affordance regions using existing vision foundation models, enabling GROW 2 to generalize across open-category objects. 3 4.1 Tool and Part Selection GROW 2 starts with a cluttered scene of multiple objects and a task specified with a language instruc- tion l. Instead of finding an appropriate object as the tool solely through geometric features, GROW 2 selects the tool in semantic space with the commonsense knowledge of VLMs. Specifically, GROW 2 first extracts the names of all objects and their parts as candidates. After that, it performs tool and part selection conditioned on the task, the observation, and the extracted object-part candidates. Object and part extraction. GROW 2 first leverages a VLM to enumerate all objects in the RGB observation I , yielding object nameso i N i=1 . Then, to enhance the VLMâs understanding of object structure and reduce hallucinations on nonexistent parts, we use SAM3 [41] to segment these objects conditioned on their names, and then extract the corresponding image crops I i N i=1 . For each object name and the corresponding image crop (o i ,I i ), we query the VLM to generate a set of part candidates P i =p i,1 ,p i,2 ,...,p i,j i , leveraging the VLMâs knowledge of object affordances. Tool and part selection. Given the RGB image I and the task instruction l, we prompt the VLM to select the tool object o A , the target object o B , and task-relevant parts from the extracted object-part candidates (o i ,P i ) N i=1 , where P i = p i,1 ,p i,2 ,...,p i,j i denotes the part candidates generated for object o i . We consider two categories of task-relevant parts: (1) a part p G on the tool, indicating where the robot should grasp o A , and (2) a part p A on the tool o A and a part p B on the target o B , indicating where the two objects should interact to realize the intended function. Concretely, we select p G ,p A â P A and p B â P B in semantic space, and obtain: [o A , (p G ,p A )], [o B ,p B ]. To improve robustness, we use in-context learning by including a small set of tool and part selection examples in the prompt. These examples specify the expected input-output format and provide chain-of-thought guidance illustrating how to choose appropriate tools and parts step-by-step. By combining the extracted object-part candidates, few-shot prompting, and the VLMâs commonsense knowledge, GROW 2 enables controllable and interpretable tool and part selection for tool use. All prompts related to tool and part selection are provided in the appendix. 4.2 3D Affordance Grounding Now that we have selected the tool object o A , the target object o B , and the task-relevant parts (p G ,p A ,p B ), the next step is to ground each part from the semantic space into a 3D affordance region for manipulation. In practice, these affordance regions may be occluded by other objects in the scene or self-occluded due to the camera viewpoint. Prior work mitigates this by relying on multi-camera setups, which are often costly and slow. Instead, we propose a novel affordance detection approach that produces complete, function-faithful 3D affordance regions from a single- view RGB-D observation. As illustrated in Fig. 2, the proposed approach consists of two stages: (1) 3D reconstruction and registration, and (2) multi-view 2D segmentation and fusion. SAM3 SAM3D Multi-View Rendering Aggregate SAM3 Part Seg 3D Affordance Region Object Mesh Object Segmentation Registered Object Mesh 3D Reconstruction and RegistrationMulti-View2D Segmentation and Fusion Single-View RGB-D Selected tool: brush & part: bristles Figure 2: 3D affordance grounding. Our method consists of two stages: (a) reconstructing and registering an object mesh from a single-view RGB-D observation and (b) rendering the registered mesh from multiple viewpoints, segmenting the selected parts in each view, and fusing the resulting masks into 3D affordance regions. 4 3D reconstruction and registration. We first leverage 3D reconstruction for the selected tool object o A and target object o B to alleviate occlusions inherent in single-view observations, as shown on the left of Fig. 2. For each object o i â o A ,o B , given the single-view RGB-D observation (I,D), we first use SAM3 to segment the object and produce a masked RGB-D crop (I i ,D i ). We then feed the RGB-D crop into SAM3D [42] for reconstruction. To preserve the objectâs pose and scale in the camera coordinate frame rather than reconstructing an object with arbitrary pose and scale, we convert the depth crop D i into a point map X i and use it as a geometric condition during reconstruction. Finally, to further improve geometric accuracy, we align the reconstructed mesh M to the original point map X i . Specifically, since X i is only from a partial observation, we render M from the same camera view to produce a partial point map X M and perform ICP-based rigid registration to obtain the transformation T from X M to X i . Applying the transformation T to the reconstructed mesh M yields the registered object mesh c M , which is used in the subsequent stage. Multi-view 2D segmentation and fusion. After acquiring the registered meshes of the tool object and the target object, the next step is to ground the task-relevant parts p G ,p A ,p B into 3D affordance regions. Since grounding parts on open-category 3D objects remains a challenge [11], we aim to leverage the strong 2D grounding capabilities of state-of-the-art vision foundation models. For each pâp G ,p A ,p B , we first render the corresponding mesh c M from multiple viewpoints and produce a set of 2D images b I k K k=1 . This multi-view rendering addresses self-occlusion and ensures the task-relevant part p is visible and can be grounded. In practice, we choose K = 8. For each b I k â b I k K k=1 , we ground the description p of the task-relevant part into a 2D segmentation mask s k using SAM3. However, SAM3 may fail to understand some complex text queries [41]. In this scenario, to improve the reliability of segmentation, we use a VLM [43] as a fallback to translate such descriptions into bounding boxes. Conditioned on these bounding boxes, SAM3 then produces fine-grained segmentations. This design leverages the VLMâs visual-semantic understanding while improving efficiency by minimizing costly VLM queries. Finally, we aggregate the multi-view 2D segmentations s k K k=1 into a 3D affordance region. Specifically, for each s k â s k K k=1 , we back-project the mask onto the reconstructed mesh c M using the corresponding camera parameters, yielding a point set on the object surface. We merge the point sets across views by taking the union, yielding an aggregated point set on the object sur- face. Since the 2D segmentations may be imperfect, the aggregated 3D points could contain noise. We therefore apply DBSCAN [44] to cluster the points, discard outliers, and retain only the largest cluster as the predicted affordance region. 5 Experiments In this section, we aim to answer three questions: (1) How does GROW 2 perform compared to baselines on affordance prediction benchmarks? (2) How effectively do the affordances predicted by GROW 2 support robot tool use? (3) How well does GROW 2 perform in real robot experiments? Table 1: Affordance prediction results on AGD20K unseen test split. MethodsKLD (â)SIM (â)NSS (â) Cross-View-AG [12]1.7870.2850.829 LOCATE [45]1.4050.3721.157 3DOI [46]3.5650.2270.657 AffordanceLLM [47]1.4630.3771.070 UAD [26]1.8780.4071.092 GROW 2 1.5060.4181.334 Table 2: Affordance prediction results on PIAD unseen test split. MethodsaIoU (â)AUC (â)MAE (â) PFusion [48]5.361.90.193 XMF [49]5.762.60.188 IAGNet [50]8.071.80.127 LASO [11]8.069.20.118 GEAL [27]8.772.50.102 GROW 2 9.074.20.102 5.1 Affordance Prediction Experiments GROW 2 achieves superior performance on existing 2D and 3D affordance prediction bench- marks. We evaluate GROW 2 on two standard affordance prediction benchmarks: AGD20K [12] 5 PoundCutLiftSweepPourScoopSpreadStirOpenPeelAverage 0.0 0.1 0.2 0.3 0.4 0.5 0.6 IoU DINOv3SD-DINOMOKA GROW 2 0.080.08 0.11 0.15 0.13 0.08 0.19 0.21 0.08 0.13 0.12 0.09 0.05 0.12 0.06 0.16 0.10 0.18 0.28 0.11 0.15 0.13 0.25 0.27 0.00 0.31 0.18 0.33 0.40 0.37 0.23 0.32 0.26 0.27 0.45 0.15 0.35 0.23 0.38 0.48 0.41 0.20 0.35 0.33 Figure 3: Affordance prediction results on GROW 2 Bench. We evaluate alignment between pre- dicted affordances and human annotations with IoU across 10 task types. The results show that GROW 2 aligns better with human expertsâ tool selection and affordance annotations. for 2D images and PIAD [50] for 3D point clouds. In both benchmarks, the model is required to predict affordance regions conditioned on a task instruction. We evaluate GROW 2 on the unseen test split of each benchmark and compare it against the corresponding baselines. Following the standard evaluation protocols, we report Kullback-Leibler divergence (KLD), similarity metric (SIM), and normalized scanpath saliency (NSS) for AGD20K, and average intersection over union (aIoU), area under the ROC curve (AUC), and mean absolute error (MAE) for PIAD. Details of the baselines and evaluation metrics are provided in the appendix. Tables 1 and 2 show the results on AGD20K and PIAD, respectively. Despite not being trained on these benchmarks, our method achieves the best overall performance across most metrics, attain- ing the highest SIM and NSS on AGD20K, as well as the highest aIoU and AUC and the lowest MAE on PIAD. The higher KLD on AGD20K arises because it penalizes mismatches across the full background distribution, while GROW 2 produces fine-grained predictions that may diverge from AGD20Kâs more diffuse ground-truth distributions. These baselines directly map task instructions to affordance regions, requiring substantial training data and limiting generalization to open-category objects and novel functions. In contrast, GROW 2 decomposes grounding into two stages, simplify- ing reasoning and better leveraging existing VLMs and visual foundation models. GROW 2 can effectively ground affordances in multi-object settings. Existing benchmarks, in- cluding AGD20K and PIAD, either provide the tool object or involve relatively simple scenes. To evaluate affordance grounding in multi-object settings, we introduce GROW 2 Bench, where the robot must first identify which object to use as the tool and then predict its affordance region. Each in- stance in GROW 2 Bench consists of a task instruction and a scene image containing at least five can- didate objects, with ground-truth affordance regions annotated by human experts. GROW 2 Bench comprises ten task types, with 50 instances each. More details are provided in the appendix. Fig. 3 shows the results on GROW 2 Bench. We compare GROW 2 with two types of training-free baselines: (1) one-shot baselines, which select a tool from the scene and transfer the affordance region from a demonstrated tool based on the cosine similarity of pretrained image features, such as DINOv3 [51] and SD-DINO [52]; and (2) MOKA [53], a zero-shot baseline that uses a VLM for tool selection and predicts affordances through keypoint-based visual prompting. Implementation details are provided in the appendix. GROW 2 achieves the highest IoU across ten task types. One- shot baselines often fail to select an appropriate tool because pretrained image features lack global semantic understanding. While MOKA uses a VLM for tool selection and affordance prediction, its reliance on visual prompting requires the VLM to reason spatially, making it prone to hallucinations. 5.2 Simulation Experiments Experiment setup. To evaluate whether GROW 2 can facilitate downstream robot tool use, we con- struct a simulation benchmark in SAPIEN 3 [55] using a Franka Emika Panda arm. The benchmark includes five task types: Pound, Cut, Lift, Sweep, and Pour, with 20 test cases per task type. Each test case consists of a task instruction and a scene containing one target object and three to five candidate tool objects. For GROW 2 and all baselines, we implement low-level skills conditioned 6 Table 3: Simulation experiments on robot tool use. We report the manipulation success rates (%) and highlight the best performance in bold. For VLM-based methods, we report std across 3 trials. MethodPoundCutLiftSweepPourAverage ICP [54]35.040.05.055.020.031.0 DINOv3 [51]25.015.00.055.010.021.0 SD-DINO [52]15.030.00.070.065.036.0 UAD [26] (given tool)45.070.00.085.035.047.0 GEAL [27] (given tool)35.055.020.065.035.042.0 MOKA [53]36.7Âą4.461.7Âą5.60.0Âą0.050.0Âą6.716.7Âą2.233.0 GROW 2 w/o multi-view56.7Âą2.265.0Âą3.360.0Âą6.771.7Âą4.460.0Âą3.362.7 GROW 2 83.3Âą5.688.3Âą5.681.7Âą5.688.3Âą4.491.7Âą4.486.7 on the predicted affordance regions, using Contact-GraspNet [56] for grasp pose sampling together with motion planning. Details of the low-level skills are provided in the appendix. Baselines. We compare GROW 2 with the same baselines as in GROW 2 Bench: DINOv3, SD- DINO, and MOKA. We additionally include ICP [54], a one-shot 3D matching baseline, as well as UAD [26] and GEAL [27], the best-performing methods on AGD20K and PIAD, respectively. Since UAD and GEAL are not designed for multi-object scenes, we provide them with the ground-truth tool. We also evaluate GROW 2 w/o multi-view, which ablates the 3D reconstruction and multi-view grounding stage. Additional baseline and implementation details are provided in the appendix. Results. Table 3 reports the results. GROW 2 achieves the highest success rate across all five task types, demonstrating strong generalizability across diverse tasks and open-category objects. In par- ticular, Lift is the most challenging task for the baselines, as success requires identifying matching affordance regions on both the tool and target objects, with little tolerance for imprecise target af- fordance localization. In contrast, GROW 2 successfully handles this task through precise semantic reasoning and 3D geometric grounding. Moreover, the performance drop of GROW 2 w/o multi-view highlights the importance of 3D reconstruction and multi-view grounding for resolving occlusions and improving affordance localization. Intel RealSense L515 Camera FrankaResearch3RobotArm Objects Presented in the Experimental Scenes Figure 4: Real-world setup. PoundCutLiftSweepPourAverage 0 20 40 60 80 100 Success rate (%) MOKA GROW 2 40.0 30.0 0.0 60.0 20.0 30.0 80.0 70.0 80.080.080.0 78.0 Figure 5: Average success rates (%) of robot tool use in the real world. We evaluate on 10 scenes for each task type. 5.3 Real-World Experiments To demonstrate that GROW 2 can work in a real robotic system, we conduct real-world experiments on five types of manipulation tasks. As illustrated in Fig. 4, our real-world setup consists of a Franka Research 3 robot arm equipped with a parallel gripper and an Intel RealSense L515 camera for RGB-D observations. We compare GROW 2 against MOKA [53], the strongest baseline in our affordance prediction and simulation experiments. For each task type, we evaluate GROW 2 and the baseline on 10 scenes and report success rates under the same low-level skill implementation. Each test scene contains a target object and at least five other objects as the tool candidates. The results in Fig. 5 indicate that GROW 2 can be deployed reliably in real-world settings. Since GROW 2 does not require extra in-domain data for training, it can be directly deployed on real robots while achieving performance comparable to that in simulation. Fig. 6 visualizes the predicted affordance regions and the executed trajectories of GROW 2 . Notably, GROW 2 is robust to noisy real-world depth observations, as it computes affordance regions from a reconstructed mesh through accurate 7 multi-view rendering, whereas the baseline is adversely affected by missing or erroneous depth measurements. tool function grasp ExecutionAffordance RegionTask instructionInitial StateTool & Part Selection âCut the cake.â plate (tool) ⢠cut: rim ⢠grasp:rim cake (target) ⢠cut: cake âPound the bug.â âSweep the trash.â brush (tool) ⢠sweep: bristles ⢠grasp:handle trash (target) ⢠sweep:trash âPour into the bowl.â hat (tool) ⢠pour: brim ⢠grasp: brim bowl (target) ⢠pour: rim âLift the teapot.â screwdriver (tool) ⢠lift: shaft ⢠grasp:handle teapot (target) ⢠lift : handle slipper (tool) ⢠pound: sole ⢠grasp:strap bug (target) ⢠pound:bug Figure 6: Qualitative results of real-world experiments. From left to right, the figure shows the task description, initial state, tool and part selection results, predicted affordance regions, and the real robot execution process. 6 Limitations Failure cases. Fig. 7 presents a breakdown of failure cases across all 50 real-world trials of GROW 2 . Failures arise across four stages: the VLM may select an inappropriate tool or part; SAM3D may produce a mesh with inaccurate scale or geometry; SAM3 may produce an invalid mask for a se- lected part name; and execution may fail due to improper grasping. Tool and part selection ac- count for the largest share of failures. A promising direction is to wrap the VLM in an agentic framework that supports iterative scene inspection, which should mitigate single-shot hallucinations. Figure 7: Failure analysis. Efficiency. The high latency of querying VLMs and running inference with vision foundation models makes it challenging for the robot to perform dynamic manipulation. In our setup, processing an entire scene takes approximately 16.6 seconds. We provide a detailed breakdown of the runtime and computational cost in the ap- pendix. Future work could distill GROW 2 into a lightweight affordance prediction model [26]. 7 Conclusion In this paper, we introduce GROW 2 , a zero-shot method for grounding affordances in robot tool use. By decomposing the problem with a part representation, GROW 2 reformulates open-world affordance grounding as part grounding, enabling existing vision foundation models to address it effectively. GROW 2 can predict complete and functionally faithful 3D affordance regions from a single-view observation. Experiments on affordance prediction benchmarks demonstrate the effec- tiveness and generalization ability of GROW 2 . Furthermore, tool-use experiments in both simulation and the real world show that the affordance regions predicted by GROW 2 can effectively support downstream robot tool use. 8 References [1] T. Fitzgerald, A. Goel, and A. Thomaz. Modeling and learning constraints for creative tool use. Frontiers in Robotics and AI, 8:674292, 2021. [2] C. P. Van Schaik and G. R. Pradhan. A model for tool-use traditions in primates: implications for the coevolution of culture and cognition. Journal of Human Evolution, 44(6):645â664, 2003. [3] A. Xie, F. Ebert, S. Levine, and C. Finn. Improvisation through physical understanding: Using novel objects as tools with visual foresight. In Proceedings of Robotics: Science and Systems, 2019. [4] K. Fang, Y. Zhu, A. Garg, A. Kurenkov, V. Mehta, L. Fei-Fei, and S. Savarese. Learning task- oriented grasping for tool manipulation from simulated self-supervision. The International Journal of Robotics Research, 39(2-3):202â216, 2020. [5] R. Holladay, T. Lozano-P Ě erez, and A. Rodriguez. Force-and-motion constrained planning for tool use. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2019. [6] Z. Xu, Z. Xian, X. Lin, C. Chi, Z. Huang, C. Gan, and S. Song. Roboninja: Learning an adaptive cutting policy for multi-material objects. In Proceedings of Robotics: Science and Systems, 2023. [7] C. G. Jensen, W. E. Red, and J. Pi. Tool selection for five-axis curvature matched machining. Computer-Aided Design, 34(3):251â266, 2002. [8] H. Shi, H. Xu, S. Clarke, Y. Li, and J. Wu. Robocook: Long-horizon elasto-plastic object manipulation with diverse tools. In Conference on Robot Learning, 2023. [9] C. Tang, A. Xiao, Y. Deng, T. Hu, W. Dong, H. Zhang, D. Hsu, and H. Zhang. Mimicfunc: Imitating tool manipulation from a single human video via functional correspondence. In Conference on Robot Learning, 2025. [10] S. Huang, I. Ponomarenko, Z. Jiang, X. Li, X. Hu, P. Gao, H. Li, and H. Dong. Manipvqa: Injecting robotic affordance and physically grounded information into multi-modal large lan- guage models. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2024. [11] Y. Li, N. Zhao, J. Xiao, C. Feng, X. Wang, and T.-s. Chua. Laso: Language-guided affordance segmentation on 3d object. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. [12] H. Luo, W. Zhai, J. Zhang, Y. Cao, and D. Tao. Learning affordance grounding from exo- centric images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. [13] W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. In Conference on Robot Learning, 2023. [14] W. Shen, G. Yang, A. Yu, J. Wong, L. P. Kaelbling, and P. Isola. Distilled feature fields enable few-shot language-guided manipulation. In Conference on Robot Learning, 2023. [15] Y. Wang, M. Zhang, Z. Li, T. Kelestemur, K. Driggs-Campbell, J. Wu, L. Fei-Fei, and Y. Li. D 3 fields: Dynamic 3d descriptor fields for zero-shot generalizable rearrangement. In Conference on Robot Learning, 2024. [16] S. Bahl, R. Mendonca, L. Chen, U. Jain, and D. Pathak. Affordances from human videos as a versatile representation for robotics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 9 [17] E. S ̧ ahin, M. Cakmak, M. R. Do Ě gar, E. U Ě gur, and G. Ě Uc ̧oluk. To afford or not to afford: A new formalization of affordances toward affordance-based robot control. Adaptive Behavior, 15(4):447â472, 2007. [18] W. Yuan, J. Duan, V. Blukis, W. Pumacay, R. Krishna, A. Murali, A. Mousavian, and D. Fox. Robopoint: A vision-language model for spatial affordance prediction for robotics. In Confer- ence on Robot Learning, 2024. [19] Y. Deng and D. Hsu. General-purpose clothes manipulation with semantic keypoints. In IEEE International Conference on Robotics and Automation, 2025. [20] J. Gao, Z. Tao, N. Jaquier, and T. Asfour. K-vil: Keypoints-based visual imitation learning. IEEE Transactions on Robotics, 39(5):3888â3908, 2023. [21] L. Manuelli, Y. Li, P. Florence, and R. Tedrake. Keypoints into the future: Self-supervised correspondence in model-based reinforcement learning. In Conference on Robot Learning, 2021. [22] D. Turpin, L. Wang, S. Tsogkas, S. Dickinson, and A. Garg. Gift: Generalizable interaction- aware functional tool affordances without labels. In Proceedings of Robotics: Science and Systems, 2021. [23] Y. Ju, K. Hu, G. Zhang, G. Zhang, M. Jiang, and H. Xu. Robo-abc: Affordance general- ization beyond categories via semantic correspondence for robot manipulation. In European Conference on Computer Vision, 2024. [24] G. Li, D. Sun, L. Sevilla-Lara, and V. Jampani. One-shot open affordance learning with foun- dation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. [25] S. Stojanov, L. Zhao, Y. Zhang, D. L. Yamins, and J. Wu. Weakly-supervised learning of dense functional correspondences. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025. [26] Y. Tang, W. Huang, Y. Wang, C. Li, R. Yuan, R. Zhang, J. Wu, and L. Fei-Fei. Uad: Unsuper- vised affordance distillation for generalization in robotic manipulation. In IEEE International Conference on Robotics and Automation, 2025. [27] D. Lu, L. Kong, T. Huang, and G. H. Lee. Geal: Generalizable 3d affordance learning with cross-modal consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025. [28] J. Zhu, Y. Ju, J. Zhang, M. Wang, Z. Yuan, K. Hu, and H. Xu. Densematcher: Learning 3d semantic correspondence for category-level manipulation from a single demo. In International Conference on Learning Representations, 2025. [29] J. Barreiros, A. Beaulieu, A. Bhat, R. Cory, E. Cousineau, H. Dai, C.-H. Fang, K. Hashimoto, M. Z. Irshad, M. Itkina, et al. A careful examination of large behavior models for multitask dexterous manipulation. arXiv preprint arXiv:2507.05331, 2025. [30] K. Pfeiffer, A. Escande, and A. Kheddar. Nut fastening with a humanoid robot. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2017. [31] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 44(10-11):1684â1704, 2025. [32] L. Ke, J. Wang, T. Bhattacharjee, B. Boots, and S. Srinivasa. Grasping with chopsticks: Com- bating covariate shift in model-free imitation learning for fine manipulation. In IEEE Interna- tional Conference on Robotics and Automation, 2021. 10 [33] K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. R. Equi, C. Finn, N. Fusai, M. Y. Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke, A. Walling, H. Wang, L. Yu, and U. Zhilinsky. Ď 0.5 : a vision-language-action model with open-world generalization. In Conference on Robot Learning, 2025. [34] A. Z. Ren, B. Govil, T.-Y. Yang, K. R. Narasimhan, and A. Majumdar. Leveraging language for accelerated learning of tool manipulation. In Conference on Robot Learning, 2022. [35] D. Seita, Y. Wang, S. J. Shetty, E. Y. Li, Z. Erickson, and D. Held. Toolflownet: Robotic manipulation with tools via predicting tool flow from point clouds. In Conference on Robot Learning, 2022. [36] M. Xu, P. Huang, W. Yu, S. Liu, X. Zhang, Y. Niu, T. Zhang, F. Xia, J. Tan, and D. Zhao. Creative robot tool use with large language models. arXiv preprint arXiv:2310.13065, 2023. [37] H. Asada and M. Brady. The curvature primal sketch. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8(1):2â14, 1986. [38] M. A. Toussaint, K. R. Allen, K. A. Smith, and J. B. Tenenbaum. Differentiable physics and stable modes for tool-use and manipulation planning. In Proceedings of Robotics: Science and Systems, 2018. [39] N. Saito, K. Kim, S. Murata, T. Ogata, and S. Sugano. Tool-use model considering tool se- lection by a robot using deep learning. In IEEE-RAS International Conference on Humanoid Robots, 2018. [40] J. J. Gibson. The theory of affordances. Hilldale, USA, 1(2):67â82, 1977. [41] N. Carion, L. Gustafson, Y.-T. Hu, S. Debnath, R. Hu, D. Suris, C. Ryali, K. V. Alwala, H. Khedr, A. Huang, et al.Sam 3: Segment anything with concepts. arXiv preprint arXiv:2511.16719, 2025. [42] X. Chen, F.-J. Chu, P. Gleize, K. J. Liang, A. Sax, H. Tang, W. Wang, M. Guo, T. Hardin, X. Li, et al. Sam 3d: 3dfy anything in images. arXiv preprint arXiv:2511.16624, 2025. [43] S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, W. Ge, Z. Guo, Q. Huang, J. Huang, F. Huang, B. Hui, S. Jiang, Z. Li, M. Li, M. Li, K. Li, Z. Lin, J. Lin, X. Liu, J. Liu, C. Liu, Y. Liu, D. Liu, S. Liu, D. Lu, R. Luo, C. Lv, R. Men, L. Meng, X. Ren, X. Ren, S. Song, Y. Sun, J. Tang, J. Tu, J. Wan, P. Wang, P. Wang, Q. Wang, Y. Wang, T. Xie, Y. Xu, H. Xu, J. Xu, Z. Yang, M. Yang, J. Yang, A. Yang, B. Yu, F. Zhang, H. Zhang, X. Zhang, B. Zheng, H. Zhong, J. Zhou, F. Zhou, J. Zhou, Y. Zhu, and K. Zhu. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025. [44] M. Ester, H.-P. Kriegel, J. Sander, and X. Xu. A density-based algorithm for discovering clusters in large spatial databases with noise. In KDD, 1996. [45] G. Li, V. Jampani, D. Sun, and L. Sevilla-Lara. Locate: Localize and transfer object parts for weakly supervised affordance grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. [46] S. Qian and D. F. Fouhey. Understanding 3d object interaction from a single image. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, 2023. [47] S. Qian, W. Chen, M. Bai, X. Zhou, Z. Tu, and L. E. Li. Affordancellm: Grounding affordance from vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 11 [48] D. Xu, D. Anguelov, and A. Jain. Pointfusion: Deep sensor fusion for 3d bounding box estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018. [49] E. Aiello, D. Valsesia, and E. Magli. Cross-modal learning for image-guided point cloud shape completion. In Advances in Neural Information Processing Systems, 2022. [50] Y. Yang, W. Zhai, H. Luo, Y. Cao, J. Luo, and Z.-J. Zha. Grounding 3d object affordance from 2d interactions in images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023. [51] O. Sim Ě eoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508.10104, 2025. [52] J. Zhang, C. Herrmann, J. Hur, E. Chen, V. Jampani, D. Sun, and M.-H. Yang. Telling left from right: Identifying geometry-aware semantic correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. [53] F. Liu, K. Fang, P. Abbeel, and S. Levine. Moka: Open-world robotic manipulation through mark-based visual prompting. In Proceedings of Robotics: Science and Systems, 2024. [54] P. Besl and N. D. McKay. A method for registration of 3-d shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2):239â256, 1992. [55] F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, L. Yi, A. X. Chang, L. J. Guibas, and H. Su. SAPIEN: A simulated part-based interactive environ- ment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, 2020. [56] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. Contact-graspnet: Efficient 6-dof grasp generation in cluttered scenes. In IEEE International Conference on Robotics and Au- tomation, 2021. [57] OpenAI. Gpt-5.2: Enhanced reasoning and general-purpose foundation model, 2025. URL https://openai.com/index/introducing-gpt-5-2/. [58] Shakker Labs and InstantX Team. Flux.1-dev-controlnet-depth. https://huggingface.co/ Shakker-Labs/FLUX.1-dev-ControlNet-Depth, 2024. Hugging Face model repository. [59] Google. Gemini 3 pro, 2026. URL https://deepmind.google/models/gemini/. [60] Google.Nano banana pro, 2026.URL https://deepmind.google/models/ gemini-image/. [61] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image syn- thesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. [62] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haz- iza, F. Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. [63] T. Dekel, S. Oron, M. Rubinstein, S. Avidan, and W. T. Freeman. Best-buddies similarity for robust template matching. In Proceedings of the IEEE/CVF Computer Vision and Pattern Recognition Conference, 2015. [64] X. Fang, B.-R. Huang, J. Mao, J. Shone, J. B. Tenenbaum, T. Lozano-P Ě erez, and L. P. Kael- bling. Kalm: Keypoint abstraction using large models for object-relative imitation learning. In IEEE International Conference on Robotics and Automation, 2025. 12 [65] O. Y. Lee, A. Xie, K. Fang, K. Pertsch, and C. Finn. Affordance-guided reinforcement learn- ing via visual prompting. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2025. [66] W. Liu, J. Mao, J. Hsu, T. Hermans, A. Garg, and J. Wu. Composable part-based manipulation. In Conference on Robot Learning, 2023. [67] G. Yin, Y. Li, Y. Wang, D. McConachie, P. Shah, K. Hashimoto, H. Zhang, K. Liu, and Y. Li. Codediffuser: Attention-enhanced diffusion policy via vlm-generated code for instruction am- biguity. In Proceedings of Robotics: Science and Systems, 2025. [68] W. Gao and R. Tedrake. kpam-sc: Generalizable manipulation planning using keypoint affor- dance and shape completion. In IEEE International Conference on Robotics and Automation, 2021. [69] Y. Liu, J. Mao, J. B. Tenenbaum, T. Lozano-P Ě erez, and L. P. Kaelbling. One-shot manipulation strategy learning by making contact analogies. In IEEE International Conference on Robotics and Automation, 2025. [70] L. Manuelli, W. Gao, P. Florence, and R. Tedrake. kpam: Keypoint affordances for category- level robotic manipulation. In The International Symposium of Robotics Research, 2019. [71] Z. Qin, K. Fang, Y. Zhu, L. Fei-Fei, and S. Savarese. Keto: Learning keypoint representations for tool manipulation. In IEEE International Conference on Robotics and Automation, 2020. [72] S. M. LaValle and J. J. Kuffner Jr. Randomized kinodynamic planning. The International Journal of Robotics Research, 20(5):378â400, 2001. [73] R. B. Rusu, N. Blodow, and M. Beetz. Fast point feature histograms (fpfh) for 3d registration. In IEEE International Conference on Robotics and Automation, 2009. 13 Appendices A Prompts15 B Details of Affordance Prediction Experiments16 B.1 AGD20K . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 B.1.1Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 B.1.2Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 B.1.3Qualitative Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 B.2 PIAD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 B.2.1Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 B.2.2Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 B.2.3Qualitative Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 B.3 GROW 2 Bench . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 B.3.1Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 B.3.2Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21 B.3.3Qualitative Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21 C Details of Robot Tool Use Experiments22 C.1 Low-Level Skill Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . .22 C.2 Simulation Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23 C.2.1Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .24 C.2.2Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25 C.2.3Qualitative Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25 C.3 Real-World Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25 C.4 Breakdown Analysis of Runtime and Computational Cost . . . . . . . . . . . . . .25 14 A Prompts We use GPT-5.2 [57] (without thinking) as the vision language model (VLM) for all simulation and real-world experiments. Below, we present the system prompts provided to GPT-5.2. Object Names Parsing You are a vision assistant. I will provide you with an image containing a table. Analyze the image and identify all objects located on top of the table surface. The response should be a dictionary in JSON form: âobjectsontableâ: [ âobject1â, âobject2â, â...â, ] Rules: ⢠include all objects that are on the table. ⢠do not miss any objects. ⢠do not include explanations, descriptions, or extra text. Output only the JSON object. Part Name Parsing You are an expert robotic affordance-analysis assistant. You will receive: ⢠an image of an object ⢠a text label for the object: <object-name> Your task is to list the meaningful physical parts of the object. Output format: âobject-nameâ: [ âpart-1â, âpart-2â ] Rules: ⢠use simple, concrete part names (1â2 words). ⢠parts must be functional object components (e.g., âhandleâ, ârimâ, âtipâ, âbladeâ, âshaftâ). ⢠all parts must be unique (no duplicates). ⢠avoid vague and geometry-only terms, such as body, area, wall, edge, surface, base, upper, and interior. ⢠avoid long or descriptive phrases If the object does not have meaningful separable parts, return an empty list. After generating the parts, verify that none of them contain any forbidden words. If any do, remove them. Examples: ⢠âpaintbrushâ: [âbristlesâ, âhandleâ] ⢠ârulerâ: [] ⢠âhookâ: [âshaftâ, âtipâ] ⢠âpanâ: [âhandleâ, ârimâ, âcooking surfaceâ] Tool and Part Selection You are an expert robot assistant. You will receive: ⢠an image of a tabletop scene ⢠a task instruction describing a manipulation action ⢠an objectâpart dictionary (each object includes a list of parts) First, reason carefully about the task using: ⢠geometry â object shapes, orientations, distances, contact regions, and alignment between tool and target ⢠semantics â object categories, functions, and affordances relevant to the action ⢠physics â stability, friction, force transfer, leverage, support, and motion constraints Use this reasoning to decide: ⢠target object: identify the object explicitly mentioned or implied as being acted upon in the task instruction ⢠tool object: select a different object that can perform the required action on the target object, based on its shape and affordances. ⢠target function part: the part of the target object that will be acted on. ⢠tool grasp part: the part of the tool object that the robot should grasp. ⢠tool function part: the part of the tool that performs the action on the target. For some objects that cannot be decomposed, the grasp part and action part can be the object itself, like a ruler. The target and tool objects must be different objects. Your response must contain two parts: (i) explain your reasoning and selection based on geometry, semantics, and physics. You should make use of object parts for reasoning. (i) A JSON block containing only the final selection results, placed strictly between a pair of *** markers. Return your answer only in the following format: reason: reasoning for tool and part selection **** âtarget objectâ: âtarget-object-nameâ, âtarget object infoâ: âtarget function partâ: âtarget-partâ, âtool objectâ: âtool-object-nameâ, âtool object infoâ: âtool grasp partâ: âtool-part-to-graspâ, âtool action partâ: âtool-part-used-to-actâ **** 15 B Details of Affordance Prediction Experiments B.1 AGD20K AGD20K [12] is a 2D affordance grounding benchmark, consisting of 20,061 exocentric images and 6,060 egocentric images from 36 affordance categories. We follow the evaluation procedure defined in AGD20K [12] and compare GROW 2 with baselines on the unseen test split using the same metrics as in previous work. B.1.1 Baselines Cross-View-AG [12] learns affordance cues from diverse exocentric interaction images and trans- fers them to the egocentric view. It further improves affordance localization by preserving correla- tions among affordance regions. LOCATE [45] learns affordance grounding from weak supervision by identifying the object part involved in exocentric human-object interactions and transferring this part-level cue to egocentric object images. 3DOI [46] predicts affordance regions from a single RGB image by identifying interactive objects and localizing where human-object interactions are likely to occur. AffordanceLLM [47] uses LLaVA as a vision-language backbone to understand the affordance, then predicts the affordance region through an added mask decoder and mask token. UAD [26] extracts knowledge from foundation models and distills it into a lightweight affordance prediction model without manual annotation. B.1.2 Metrics Kullback-Leibler Divergence (KLD) measures the distributional difference between the predicted affordance map (M ) and the ground truth (M Ⲡ), which is KLD (M,M Ⲡ) = X i M Ⲡi log Îľ + M Ⲡi Îľ + M i ,(1) where Îľ is a small positive constant added for numerical stability. Similarity Metric (SIM) is also called histogram intersection, which measures the intersection be- tween the predicted affordance map (M ) and the ground truth (M Ⲡ). The score ranges from 0 to 1. It is given by SIM (M,M Ⲡ) = X i min (M i ,M Ⲡi ),(2) where P i M i = P i M Ⲡi = 1. Normalized Scanpath Saliency (NSS) measures the correspondence between the prediction map (M ) and the ground truth (M Ⲡ). It is given by NSS (M,M Ⲡ) = 1 N X i Ë M Ă M Ⲡi ,(3) where N = P i M Ⲡi , Ë M = MâÎź(M) Ď(M) . Îź (M) and Ď (M) are the mean and standard deviation, respectively. B.1.3 Qualitative Results In Fig. 8, we show representative qualitative results on the unseen split of AGD20K. Across diverse affordance categories such as cutting, drinking, opening, and typing, GROW 2 consistently highlights task-relevant image regions, indicating its ability to perform affordance grounding on open-category objects. 16 cut withdrink withhitholdlie on throwpeelopentake phototype on Figure 8: Representative examples from the AGD20K benchmark. We visualize the affordances predicted by GROW 2 using the orange regions in the images. B.2 PIAD PIAD [50] is a 3D affordance grounding benchmark, consisting of 7,012 point clouds across 23 object classes and 17 affordance categories. We follow the evaluation protocol defined in PIAD and compare GROW 2 with baselines on the unseen test split using the same metrics as in previous work. Since the point clouds in PIAD do not contain color information, it is difficult to leverage semantic cues for affordance grounding. In realistic settings, however, RGB images are typically available when capturing point clouds. To evaluate GROW 2 on this benchmark, we make the following modi- fication. After multi-view rendering, we obtain a set of depth images. We then apply ControlNet [58] to generate RGB images conditioned on these depth images, and perform part segmentation on the generated RGB images. The rest of the pipeline remains unchanged. B.2.1 Baselines PFusion [48] is a multimodal fusion baseline that combines global image features, global point- cloud features, and point-wise 3D features for dense 3D prediction. XMF [49] performs cross-modal feature fusion with modality-specific feature extraction and attention-based interaction between 2D and 3D representations. IAGNet [50] is the original PIAD baseline, which grounds 3D affordance regions by aligning interaction-aware image features with point-cloud features. LASO [11] uses an adaptive fusion module to integrate language and point-cloud features for language-conditioned 3D affordance region prediction. GEAL [27] predicts 3D affordance regions from semantic cues by bridging sparse 3D point clouds with 2D representations via Gaussian splatting. B.2.2 Metrics Area Under the ROC Curve (AUC) measures how well the predicted affordance map separates affordance and non-affordance regions over different thresholds. Average Intersection over Union (aIoU) measures the average overlap between the predicted affor- dance region and the ground-truth region: IoU = TP TP + FP + FN ,(4) where TP, FP, and FN denote true positive, false positive, and false negative counts, respectively. 17 Mean Absolute Error (MAE) measures the average absolute difference between the predicted affor- dance map and the ground truth: MAE = 1 n n X i=1 |e i |,(5) where e i denotes the prediction error for the i-th element. B.2.3 Qualitative Results Fig. 9 presents qualitative examples on the unseen split of PIAD. The results show that GROW 2 can localize meaningful 3D affordance regions across different object classes and interaction types, demonstrating the effectiveness of GROW 2 on 3D affordance grounding. graspgraspopensitlay cutstab display press contain Figure 9: Representative examples from the PIAD benchmark. We visualize the affordances predicted by GROW 2 with the red regions in the point cloud. B.3 GROW 2 Bench Existing benchmarks, including AGD20K and PIAD, either provide the tool object or involve rel- atively simple scenes. To evaluate affordance grounding in multi-object settings, we introduce GROW 2 Bench, where the robot must first identify which object to use as the tool and then pre- dict its affordance region. The construction procedure for GROW 2 Bench is as follows. First, we prompt Gemini 3 Pro [59] to generate a comprehensive inventory of household objects, resulting in 82 object categories. Next, for each of the 10 task types, we define a set of plausible target objects containing 4â8 candidates. We then ask Gemini 3 Pro to propose a list of possible tool objects for each task type. This list is subsequently verified and filtered by a human annotator, yielding 10â20 tool candidates per task type. For each test scene, we randomly sample (i) one target object from the task-specific target set, (i) one tool object from the corresponding tool-candidate set, and (i) four distractor objects from the household object inventory. We then use Nano Banana Pro [60] to generate a tabletop image containing these six objects, ensuring that the target object referenced in the instruction is placed near the image center. For each of the 10 task types, we generate 50 scenes, resulting in 500 test scenes in total. We additionally build a web-based annotation interface to collect human selections of the tool object and the corresponding affordance region, as shown in Fig. 10. We first present several examples to familiarize annotators with the procedure. Given an image and its paired task instruction, a robotics expert selects the appropriate tool object and draws a bounding box around the toolâs affordance region. We convert each bounding box into a 2D segmentation mask using SAM 3, which serves as the annotated affordance region. To reduce individual bias, we recruit 10 robotics experts and aggregate their annotations. Fig. 11 shows representative examples from the dataset. 18 Figure 10: Web-based annotation interface for the construction of GROW 2 Bench. B.3.1 Baselines We implement several training-free methods as baselines for GROW 2 Bench and categorize them into one-shot and zero-shot settings. The one-shot baselines rely on a demonstration and feature matching to select the tool and localize the affordance region, whereas the zero-shot baselines lever- age the semantic and spatial understanding capabilities of the VLM. For the one-shot baselines, we provide one demonstration per task category. Each demonstration consists of observations of a con- ventional tool in both a 2D image and a 3D point cloud, along with an annotated affordance region on the tool (Fig. 12). For example, in the pounding task, the conventional tool is a hammer, and we annotate the affordance region on both the 2D image and the 3D point cloud. Details of the one-shot baselines are as follows. SD-DINO. We first use a VLM to enumerate the objects in the scene and obtain their names. We then use SAM3 [41] to segment each object conditioned on its name and extract the corresponding image crops. Next, we match each crop to the demonstrated tool image and select the crop with the highest similarity score as the tool. To extract matching features, we use SD-DINO [52], which combines Stable Diffusion [61] and DINOv2 [62] representations to capture both global context and local details. To further improve matching robustness, we compute the Best-Buddies Similar- ity (BBS) [63] using SD-DINO features. BBS measures structural similarity by identifying pairs of local patchesâone from the image crop and one from the demonstration imageâthat are mu- tual nearest neighbors in feature space (Fig. 13). These âbest-buddyâ pairs capture strong local correspondences. We compute the similarity for each matched pair and use the sum of pairwise sim- ilarities as the overall matching score. The crop (and the object) with the highest score is selected. To localize the affordance region, we transfer the demonstration affordance to the selected image crop using the best-buddy correspondences: we take the matched patch pairs whose demonstration- side patches fall inside the annotated affordance region, and map their corresponding patches onto the selected image crop to obtain the affordance region on the selected tool. 19 âPound the nailâ âPound the walnutâ âCut the bananaâ âCut the appleâ âLift the teapotâ âLift the basketâ âScoop the flourââSpread the doughâ âSpread the butterââScoop the soupââStir the teaâ âStir the batterâ âSweep the glassâ âSweep the trashâ âOpen the canâ âOpen the bottleâ âPeel the potatoâ âPeel the gingerâ âPour into the potâ âPour into the potâ Figure 11: Representative examples from GROW 2 Bench. We visualize ground-truth affordance regions annotated by human experts with the blue bounding boxes and the orange regions. Pound Cut LiftSweepPour Figure 12: Demonstrations for one-shot baseline methods. For each task category, we provide one demonstration. The affordance region (marked as red) is annotated on both 2D images and 3D point clouds. DINOv3. The DINOv3 baseline is implemented in the same way as SD-DINO, except that it uses a different matching feature. DINOv3 [51] has been pretrained on internet-scale data and can provide rich, transferable visual features. The zero-shot baseline is MOKA [53], which uses a VLM to select the tool and target object, and to propose affordance keypoints via visual prompting. To improve MOKAâs tool-selection perfor- mance, we first use a VLM to enumerate the objects in the scene and obtain their names. We then query the VLM for tool selection using the object names together with the RGB image. After selecting the tool and target objects, MOKA applies farthest point sampling to sample candidate keypoints along the object boundary, and queries the VLM to select the grasp, function, and target 20 Figure 13: Best-Buddies Similarity (BBS) [63] matching with SD-DINO [52] image features. keypoints. Finally, to obtain the affordance region, we feed the selected keypoints to SAM3 and use the resulting segmentation mask as the affordance region. B.3.2 Metrics We use Intersection over Union (IoU) to measure the average overlap between the predicted affor- dance region and the human-annotated affordance region: IoU = TP TP + FP + FN ,(6) where TP, FP, and FN denote true positive, false positive, and false negative counts, respectively. B.3.3 Qualitative Results Fig. 14 visualizes qualitative results on GROW 2 Bench. In contrast to AGD20K and PIAD, each scene contains multiple candidate objects, requiring the model to first select an appropriate tool and then localize its functional region. The examples show that GROW 2 can identify appropriate tools and predict consistent affordance regions across a wide range of tool-use tasks. âPound the nailâ âPound the walnutâ âCut the bananaâ âCut the appleâ âLift the teapotâ âLift the basketâ âScoop the flourââSpread the doughâ âSpread the butterââScoop the soupââStir the teaâ âStir the batterâ âSweep the glassâ âSweep the trashâ âOpen the canâ âOpen the bottleâ âPeel the potatoâ âPeel the gingerâ âPour into the potâ âPour into the potâ Figure 14: Affordance prediction results on GROW 2 Bench. We visualize the affordances pre- dicted by GROW 2 with the orange regions in the images. 21 PoundCutLift SweepPour Figure 15: Representative key poses and region-based constraints for the low-level skills. C Details of Robot Tool Use Experiments C.1 Low-Level Skill Implementation In this paper, we present a general approach for grounding affordances in robot tool use, producing 3D affordance regions on both the tool object and the target object. These affordance regions serve as informative priors for downstream low-level action generation, including policy learning [64, 65, 66, 67] and motion planning [68, 69, 70, 71]. In this work, we use motion planning as the low-level module to evaluate whether the predicted affordances can support successful downstream manipulation. Given the predicted affordance regions, we use their functional semantics and local geometry to guide the sampling of grasp poses and the generation of tool pose sequences that realize the de- sired tool-target interaction. Specifically, we first apply Contact-GraspNet [56] to sample candidate end-effector grasp poses within the predicted grasp affordance region. We then use the predicted affordance regions associated with the tool function to define task-specific yet generalizable object- level key poses, which specify how the tool should interact with the target during task execution. Each key pose is associated with a geometric constraint expressed over the predicted affordance re- gions of the tool and target objects. Fig. 15 illustrates one representative key pose for each task: the upper-left panel shows the tool affordance region, the lower-left panel shows the target affordance region, and the right panel shows the corresponding key pose. We define the key poses and their constraints as follows: ⢠Pound: The pounding task consists of two key poses. (1) In pre-pound, the normal vector of the tool function region is aligned with and opposite to the normal vector of the target function region, while the two regions are separated by a fixed offset distance. (2) In post-pound, the two normals remain aligned and opposite, and the two function regions are brought into contact. ⢠Cut: We first sample a cutting vector from the tool function region. Specifically, the vector originates from the center of the oriented bounding box (OBB) of the tool function region and is perpendicular to the shortest OBB edge. The cutting task consists of two key poses. (1) In pre-cut, the cutting vector is aligned with and opposite to the normal vector of the target function region, with a fixed offset distance between the two regions. (2) In post-cut, the two vectors remain aligned and opposite, and the two function regions intersect. ⢠Lift: We first define an insertion vector for the tool and a socket vector for the target. The insertion vector is aligned with the longest edge of the OBB of the tool function region, and its 22 âPound the nailâ âPound the nailâ âCut the pizzaâ âCut the cakeââLift the teapotâ âLift the mugâ âSweep the trashâ âSweep the donutâ âPour into the bowlâ âPour into the bowlâ Figure 16: Examples of tasks in simulation experiments. origin is placed at the endpoint of the OBB along that edge. The socket vector represents the insertion axis of the target; it is aligned with the shortest edge of the OBB of the target function region and originates from the OBB center. The lifting task consists of three key poses. (1) In pre-insert, the insertion vector is aligned with the socket vector, and the two vector origins are separated by a fixed offset distance. (2) In post-insert, the tool moves a fixed distance from the pre-insert pose along the insertion vector. (3) In post-lift, the tool moves a fixed distance from the post-insert pose along the positive z direction. ⢠Sweep: We first sample a surface normal from the target function region that is consistent with the specified sweep direction. The sweeping task consists of two key poses. (1) In pre-sweep, the normal vector of the tool function region is aligned with and opposite to the sampled vector on the target function region, with a fixed offset distance between the two regions. (2) In post-sweep, the tool moves a fixed distance from the pre-sweep pose along the sweep direction. ⢠Pour: We first extract two task-relevant regions: a receiving region on the target and a pour region on the tool. The receiving region is represented as a rim circle extracted from the target function region, with its center at the rim center and its radius estimated from the target bowl AABB in the xy plane. The pour region is extracted as the spout or end-cap of the tool function region. We define the principal pour direction as the shortest OBB axis of the pour region, oriented to point outward. We then sample a pour point on the pour region, biased away from the grasp point to avoid pouring toward the gripper. The pouring task consists of two key poses. (1) In pre-pour, the sampled pour point is placed above the target rim with a fixed height offset and laterally positioned near the rim along a fixed approach direction. The tool is oriented such that the pour direction is tilted toward the target by a small angle, e.g., 30 ⌠. (2) In post-pour, the pour point remains at the same rim-relative position, while the tool rotates to a larger tilt angle, e.g., 90 ⌠, to direct the flow into the target. Given a sampled grasp pose and the object-level key poses, we derive the corresponding end-effector key poses. Finally, we use a collision-free motion planner, such as RRT-Connect [72], to connect the grasp pose with the sequence of end-effector key poses, producing a complete robot trajectory. If planning fails, we resample the grasp pose and object-level key poses and repeat the process until timeout. C.2 Simulation Experiments To evaluate whether the affordances predicted by GROW 2 can effectively support robot tool use, we construct a tool-use benchmark in the SAPIEN 3 simulation environment [55] using a Franka Emika Panda arm. Each test case consists of a task instruction and a corresponding scene. To build diverse scenes for evaluation, we select 85 3D object assets from the Objaverse dataset, spanning a wide 23 Figure 17: Objects involved in the simulation experiments. range of categories, sizes, shapes, and visual appearances, as shown in Fig. 17. For each scene, we randomly sample objects based on the following rules: we first sample a target object mentioned in the task instruction, and then sample 3â4 additional objects as candidate tools. Each sampled object is placed on the table with a random position and orientation, while ensuring that it lies within the robotâs reachable workspace and does not collide with other objects. For evaluation, we ensure that at least one candidate tool in each scene can successfully complete the specified task. Using this procedure, we construct 100 diverse scenes. Fig. 16 illustrates example scenes from our simulation experiments. C.2.1 Baselines We compare GROW 2 against the same baselines as in GROW 2 Bench: DINOv3, SD-DINO, and MOKA. We also include UAD [26] and GEAL [27], the best-performing methods on AGD20K and PIAD, respectively. Since UAD and GEAL are not designed for multi-object scenes, we provide them with the ground-truth tool. In addition, we evaluate ICP [54] as a one-shot 3D matching baseline. For ICP, we use the complete 3D point clouds of all objects in the scene provided by the simulator. Given the demonstrated tool point cloud, we first identify the scene object whose point cloud best matches it. To improve matching robustness, we compute FPFH [73] descriptors for each pair of point clouds and use them to obtain an initial estimate of the 3D rigid transformation. Starting from this initialization, we then run Iterative Closest Point (ICP) [54] to refine the transformation and obtain the final alignment. This procedure produces an ICP matching score and a corresponding 3D rigid transformation for each candidate object in the scene. We select the object with the highest matching score as the tool. Finally, we transfer the demonstrated affordance region by applying the estimated 3D rigid transformation to it, yielding the corresponding affordance region on the selected tool. 24 C.2.2 Metrics We define a success metric for each task type: Pound. A hard surface of the tool strikes the target object, delivering a large impact force. Success requires sufficient contact area and impact force. Cut. A sharp edge of the tool intersects the target object. Success requires the intersection region to be sufficiently sharp. Lift. A straight, thin part of the tool is inserted into the target object to lift it upward. Success requires sufficient lifting displacement. Sweep. A flat surface of the tool pushes the target object in a specified direction by a specified distance. Success requires the displacement error to be below a predefined threshold. Pour. A small object is placed in the selected tool after the robot reaches a pre-pouring pose. Success requires the object to fall into the target object after pouring. C.2.3 Qualitative Results Fig. 18 shows representative failure cases of baseline methods in comparison with GROW 2 . Matching-based approaches such as SD-DINO [52] can fail to identify a functionally appropriate tool from a cluttered scene, since visual similarity alone does not necessarily reflect whether an object can be used to complete the task. In contrast, GROW 2 selects tools in a symbolic space, allowing it to leverage the commonsense knowledge encoded in the VLM and reason about object functionality beyond appearance-level matching. Existing methods also struggle with affordance localization. UAD [26] fails to accurately detect affordances under severe occlusion, while GEAL [27] has difficulty generalizing to novel objectâ function pairs. Although MOKA [53] also uses a VLM to select the tool and target object, it relies on visual prompting to identify object keypoints as affordances, which makes its predictions less stable. By contrast, GROW 2 avoids directly querying the VLM for fine-grained spatial localization and instead grounds the selected task-relevant parts through 3D reconstruction and multi-view seg- mentation, leading to more reliable affordance predictions. We attribute this advantage to the fact that VLMs are stronger at semantic reasoning than precise spatial reasoning; when asked to localize fine-grained regions directly, they are more prone to hallucination. C.3 Real-World Experiments Tool and part selection, as well as affordance detection, are identical to those used in the simulation experiments. Here, we describe how we perform motion planning from a single-view RGB-D ob- servation in the real-world experiments. During affordance grounding, we reconstruct and register the tool and target object in 3D, which allows us to load their meshes into the planning scene. In principle, we could reconstruct every object in the scene, but doing so would be time-consuming due to repeated calls to SAM3D. Instead, for the background and other objects, we load only their partial point clouds as collision geometry and ignore their kinematics and dynamics. We found this approach provides a good trade-off between efficiency and correctness. C.4 Breakdown Analysis of Runtime and Computational Cost We report a breakdown of the runtime and computational cost of our method in Table 4. All exper- iments are conducted on a server equipped with an AMD EPYC 9354 CPU and an NVIDIA H100 GPU. Several stages of the pipeline are parallelized to improve efficiency. The tool and target objects are reconstructed in parallel, reducing the 3D reconstruction time to that of a single SAM3D run. Part segmentation is also parallelized across both objects and all eight views, keeping the overall runtime reasonable. 25 âCut the pizza.â GROW 2 SD-DINO (a) âLift the teapot.â GEAL GROW 2 (c) âPound the nail.â MOKA GROW 2 (d) âPour into the bowl.â (b) GROW 2 UAD Figure 18: (a) SD-DINO [52] fails to select an appropriate tool; (b) UAD [26] fails to accurately detect affordances under large-scale occlusion; (c) GEAL [27] fails to generalize to novel ob- jectâfunction pairs; (d) MOKA [53] fails to select the correct keypoint as the affordance. In (a), (b), and (d), red marks the affordance region on the tool object; in (c), blue marks the affordance region on the target object. For each scene, our pipeline invokes GPT-5.2 three times: once for object-name parsing, once for part decomposition, and once for tool-and-part selection. SAM3 is called twice, for object seg- mentation and part segmentation. Since the reconstructions of the tool object and target object are performed in parallel, the SAM3D runtime is reported once as the stage latency. For motion plan- ning, we set a timeout of 1 minute, although in most cases, planning takes only a few seconds. Table 4: Runtime and computational cost breakdown. Overall refers to 3 GPT-5.2 calls, 2 SAM3 calls, and 1 SAM3D call. MetricGPT-5.2SAM3SAM3DOverall Time (s)2.720.267.9316.61 VRAM (GB)â5.1223.78â 26