Paper deep dive
HiWE: Hierarchical World Knowledge Model with Visual Keypoint Enhancement for Zero-Shot 3D Path Planning
Guoqing Ma, Mingqi Yuan, Chen Gao, Jiayu Chen, Shan Yu
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Robot demonstration generation requires a system to identify where an interaction should occur, plan a feasible motion, and execute the required contact. HiWE connects these decisions through a point-based interface between visual grounding and language-based planning. PointVLM is instruction-tuned to associate task-relevant objects with image coordinates using a mixture of point annotations, segmentation-derived samples, robot observations, and visual question answering data. Depth measurements lift these predictions into a semantic 3D representation. A language planner, 3DLLM, uses this representation to specify end-effector waypoints and gripper commands, while a hybrid grasping module resolves local grasp poses. The evaluation covers 14 simulated manipulation tasks and four physical-robot tasks, together with ablations of the visual training data, spatial inputs, and grasp selection. Here, zero-shot execution refers to deployment without task-specific demonstration training; the visual model uses existing robot data during fine-tuning. This paper describes the original point-based formulation of the framework; its relationship to the subsequent GeneralVLA extension is detailed in the introduction.
Tags
Links
- Source: https://arxiv.org/abs/2609.39323v1
- Canonical: https://arxiv.org/abs/2609.39323v1
Trouble viewing inline? Open PDF directly â
Full Text
33,963 characters extracted from source content.
Expand or collapse full text
HiWE: Hierarchical World Knowledge Model with Visual Keypoint Enhancement for Zero-Shot 3D Path Planning Guoqing Ma Affiliation: School of Future Technology, University of Chinese Academy of Sciences Affiliation: State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, CAS Mingqi Yuan Affiliation: The University of Hong Kong, Hong Kong SAR, China Chen Gao Affiliation: Department of Electronic Engineering, Tsinghua University, China Jiayu Chen Affiliation: The University of Hong Kong, Hong Kong SAR, China Shan Yu â Affiliation: State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, CAS Institute of Automation â thanks: â Corresponding author: shan.yu@nlpr.ia.ac.cn Chinese Academy of Sciences Beijing China Abstract Robot demonstration generation requires a system to identify where an interaction should occur, plan a feasible motion, and execute the required contact. HiWE connects these decisions through a point-based interface between visual grounding and language-based planning. PointVLM is instruction-tuned to associate task-relevant objects with image coordinates using a mixture of point annotations, segmentation-derived samples, robot observations, and visual question answering data. Depth measurements lift these predictions into a semantic 3D representation. A language planner, 3DLLM, uses this representation to specify end-effector waypoints and gripper commands, while a hybrid grasping module resolves local grasp poses. The evaluation covers 14 simulated manipulation tasks and four physical-robot tasks, together with ablations of the visual training data, spatial inputs, and grasp selection. Here, zero-shot execution refers to deployment without task-specific demonstration training; the visual model uses existing robot data during fine-tuning. This paper describes the original point-based formulation of the framework; its relationship to the subsequent GeneralVLA extension is detailed in the introduction. Introduction Collecting a successful robot demonstration requires more than recognizing an object or describing an intended action. A system must locate an interaction region, relate that region to the geometry of the scene, and execute a motion that respects the object and its surroundings. Errors at any of these stages can invalidate the resulting demonstration. This paper studies how an explicit point representation can connect visual grounding to motion planning for this purpose. Action-predicting vision-language models, such as OpenVLA (Kim et al. 2024), learn from observations paired with robot actions. Large datasets expand the range of available training experience (OâNeill et al. 2024; Khazatsky et al. 2024), but collecting deployment-specific demonstrations remains costly. An alternative is to divide the problem into components with different supervision requirements: image-space grounding can use point or region annotations, a language model can reason over a spatial description, and a controller can handle execution. The usefulness of this division depends on what information passes between the components. Figure 1: Interfaces for robot manipulation. HiWE exposes task-relevant points and planned trajectories between perception and execution. This separation allows the visual component to be trained on annotations that do not require deployment-specific action trajectories. HiWE (Hierarchical World Knowledge Model with Visual Keypoint Enhancement) uses PointVLM to predict object-associated image coordinates. Depth measurements convert these coordinates into 3D points for 3DLLM, which generates a sequence of waypoints and gripper commands. A hybrid grasping module (HGM) selects local grasp poses for execution. We use âworld knowledgeâ to describe the pretrained modelsâ semantic and spatial priors; this terminology does not imply that HiWE learns an explicit transition model of the environment. Relationship to GeneralVLA. HiWE is the original formulation of the framework, and GeneralVLA (Ma et al. 2026) is a subsequent extension that changes perception and adds experience-based planning. This relationship concerns the development of the methods, rather than their order of appearance on arXiv. HiWE uses PointVLM for direct coordinate prediction and 3DLLM for planning from the current instruction and scene. GeneralVLA introduces ASM, which combines language-conditioned segmentation with iterative refinement, and a KnowledgeBank that retrieves and accumulates experience across executions. The three-stage organization, the spatial planning interface, and HGM are common to the two manuscripts. We describe these common components here as part of the original system, while identifying segmentation refinement and persistent experience retrieval as features of the extension. Some experimental values and implementation descriptions also occur in both manuscripts, as noted alongside the relevant results. Table 1: Scope of the original system and its subsequent extension. Shared components are not independent evidence of a new architecture. Component HiWE GeneralVLA Perception PointVLM coordinate prediction ASM segmentation and refinement Planning Current task and 3D scene points 3D scene planning with KnowledgeBank Experience No persistent retrieval in 3DLLM Retrieval, construction, consolidation Execution HGM grasp selection HGM grasp selection The present evaluation examines point localization, the composition of PointVLMâs training data, and the spatial information supplied to the planner. It also measures task execution in simulation and on a physical robot. Throughout this paper, âzero-shotâ concerns deployment without task-specific demonstration training. It does not mean that every component is untrained or that no robot data are used: PointVLM is fine-tuned using existing datasets, including robot observations from outside the deployment environment. This paper presents the original system and its evaluation through three contributions: âą A point-based interface that connects an instruction-tuned visual model to a language planner and a grasp execution module, separating visual supervision from deployment-specific action demonstrations. âą A concrete implementation comprising PointVLM training, depth-based spatial grounding, 3DLLM waypoint generation, and HGM grasp selection. âą An evaluation on simulated and physical manipulation tasks, with ablations of visual training data, planner inputs, and grasp selection, and a study of demonstration-based policy training. The experiments characterize this original configuration. They do not isolate the incremental benefit of GeneralVLAâs segmentation or memory extensions; that would require a controlled comparison under matched conditions. Figure 2: HiWE data interfaces. PointVLM associates task-relevant objects with image points. Depth supplies their 3D coordinates for 3DLLM, which outputs waypoints and gripper commands. HGM provides local grasp poses for the execution stage. The overall decomposition and grasping component are also described in GeneralVLA (Ma et al. 2026). Related Work Language models as robot planners. Code-as-Policies (Liang et al. 2023) represents a plan as executable code calling control functions. VoxPoser (Huang et al. 2023) connects language to geometric planning through 3D value maps, while Scaling-up (Ha et al. 2023) uses language-guided interaction to obtain data for policy learning. These systems provide different interfaces between semantic decisions and robot execution. HiWE uses an ordered sequence of spatial waypoints and gripper commands; its evaluation compares the resulting system against these baselines under the inputs specified in the experiments. Learning actions from vision and language. Models including RT-1 and OpenVLA learn action predictions conditioned on observations and instructions (Brohan et al. 2023; Kim et al. 2024). LLARVA additionally predicts trajectories as part of its learning formulation (Niu et al. 2024). HiWE instead exposes an intermediate spatial description to a separate planner and execution module. This changes the interfaces and supervision used by the system; it does not guarantee that reasoning, visual accuracy, or execution speed will improve in every setting. Points and trajectories as intermediate representations. Affordance prediction connects object semantics to possible interaction locations (Sundaresan et al. 2023; Nasiriany et al. 2024; Yuan et al. 2024b). RoboPoint is particularly relevant to PointVLM because it supplies point-based visual supervision. Trajectory specifications also provide a way to condition policies (Gu et al. 2024), and point tracking makes motion information available from image sequences (Doersch et al. 2023; Yuan et al. 2024a; Wen et al. 2024). GeneralVLA (Ma et al. 2026) is the closest architectural comparison because it combines semantic affordances, 3D planning, and grasp-pose selection. As the subsequent extension of HiWE, it retains the planning and execution backbone while developing the perception and experience-reuse mechanisms. The relationship is summarized in Table 1. HiWE: Point-Based Grounding and Planning System interfaces HiWE passes explicit spatial information between perception, planning, and execution (Figure 2). PointVLM receives an image and task instruction and returns object-associated image points. Depth supplies a 3D location for each point. The resulting semantic scene description is the input to 3DLLM, whose output consists of end-effector waypoints and gripper commands. HGM resolves local grasp poses before a motion planner executes the commands. GeneralVLA retains this overall organization while extending the perception and planning components (Ma et al. 2026). PointVLM: predicting interaction locations PointVLM provides the geometric anchors used by the subsequent planner. Given a task instruction, it identifies relevant objects and represents each object by a set of image coordinates, object:(x0,y0),âŠ,(xn,yn)\object:(x_0,y_0),âŠ,(x_n,y_n)\. Associating several points with an object gives the planner information about its spatial extent after depth projection. The output is a coordinate sequence; HiWE does not use the segmentation decoder or segmentation-feedback loop described for ASM in GeneralVLA. Training. The visual backbone is QWen-VL-7B (Wang et al. 2024). Following the instruction-tuning formulation of (Liu et al. 2023), the manuscriptâs implementation updates the MLP projector and vision-language transformer while keeping the image encoder and tokenizer fixed. Responses are generated autoregressively, with a boundary token delimiting the instruction and answer. Point supervision specifies where an interaction can occur, while language supervision supports interpretation of the instruction. Supervision sources. The training mixture contains five sources (). RoboPoint (Yuan et al. 2024b) contributes 347k point-prediction examples. For LVIS (Gupta et al. 2019), points are sampled inside segmentation masks and paired with object semantics. Robot observations come from Open X-Embodiment (OâNeill et al. 2024) and SIMPLER (Li et al. 2024); these sources are outside the deployment environment. Finally, 667K visual question-answering conversations (Kafle and Kanan 2017) provide language-based supervision. The contribution of each source is evaluated in Table 4. Additional implementation details appear in Appendix . 3DLLM: planning from a spatial description Depth and camera geometry lift PointVLMâs image coordinates into 3D. Each coordinate remains associated with its object name, so 3DLLM receives a task instruction together with a compact, semantic description of the scene. The planner specifies an ordered sequence of spatial waypoints interleaved with commands to open or close the gripper. This interface can express several interaction stages within one plan. Multiple points help describe an objectâs orientation and extent; obstacle points supply context for choosing a route. These inputs are evaluated separately in Table 6. They inform the proposed motion but do not guarantee collision-free execution. Unlike the KnowledgeBank-equipped planner in GeneralVLA (Ma et al. 2026), the HiWE planner described here does not retrieve or consolidate a persistent collection of previous experiences. It plans from the current instruction and scene description. Training a separate behavior-cloning policy on successful demonstrations, described in the appendix, is distinct from such retrieval during planning. HGM: resolving local grasp geometry A waypoint specifies where the end effector should move, but a successful grasp also requires an appropriate orientation and contact configuration. HGM uses the task-relevant 3D points to restrict the region of interest in the RGB-D reconstruction. The grasp predictor (Yuan et al. 2023) then generates candidate poses in the resulting object-centered point cloud. HGM rejects candidates that fail its collision check and selects the remaining candidate whose grasp center is nearest to the object center. This is a selection heuristic rather than a guarantee of a globally optimal grasp. The selected pose is combined with the planned waypoints and passed to the motion planner for execution. HGM is part of the original HiWE pipeline and also appears in the GeneralVLA extension; the ablation in Table 7 evaluates its role within HiWE. Table 2: Task-averaged success rate % for zero-shot evaluation. HiWE outperformed other baselines in 10 out of 14 simulation tasks from RLBench (James et al. 2020). Each task was evaluated over 3 seeds to obtain the task-averaged success rate and standard deviations. Method Put_block Play_jenga Open_jar Close_box Open_box Pickup_cup Push_block VoxPoser (Huang et al. 2023) 70.70±2.31 0.00±0.00 0.00±0.00 0.00±0.00 0.00±0.00 26.70±14.00 25.33±8.33 CAP (Liang et al. 2023) 84.00±16.00 0.00±0.00 0.00±0.00 0.00±0.00 0.00±0.00 14.67±4.62 8.00±4.00 Scaling-up (Ha et al. 2023) 77.33±6.11 0.00±0.00 78.67±11.55 0.00±0.00 0.00±0.00 9.33±2.26 5.33±6.11 HiWE (Ours) 93.33±4.16 82.00±10.39 84.00±6.00 48.67±12.06 34.67±16.77 88.67±3.06 23.33±10.07 HiWE w/o FT 75.33±7.57 60.67±9.45 71.33±10.07 31.33±9.02 8.67±3.06 74.67±6.11 14.67±11.02 Method Take_umbrella Sort_mustard Open_wine Lamp_on Put_knife Pick_&_lift Insert_block VoxPoser (Huang et al. 2023) 33.33±8.33 96.00±6.93 8.00±4.00 57.30±12.22 92.00±4.00 96.00±0.00 0.00±0.00 CAP (Liang et al. 2023) 4.00±4.00 0.00±0.00 0.00±0.00 64.00±6.93 14.67±8.33 100.00±0.00 0.00±0.00 Scaling-up (Ha et al. 2023) 6.67±2.31 41.33±12.86 33.33±20.13 60.00±8.00 24.00±0.00 100.00±0.00 0.00±0.00 HiWE (Ours) 64.67±15.01 73.67±15.50 45.33±11.02 72.67±14.05 60.00±14.00 87.33±10.26 36.67±5.03 HiWE w/o FT 50.67±11.02 59.33±8.33 26.67±8.08 62.67±13.32 45.33±5.03 69.33±15.14 14.67±7.02 Figure 3: Illustrative manipulation sequences. The displayed stages connect object localization, a spatial motion plan, and robot execution for scenes involving several interactions. Experiments We evaluate the original HiWE configuration through task execution, point localization, component ablations, and behavior cloning from generated demonstrations. The configuration uses PointVLM and current-scene 3DLLM planning, without the ASM or KnowledgeBank extensions. Relationship between the reported experiments. The baseline entries for VoxPoser, CAP, and Scaling-up in Table 2, and the CAP and RoboPoint entries in Table 5, also appear in GeneralVLA (Ma et al. 2026). They are reported here to contextualize HiWEâs results and should not be interpreted as evidence of an independent replication across the two manuscripts. The tables do not provide a controlled comparison of HiWE against the later extension. Implementation details. PointVLM extracts the object-associated coordinates used to construct the planner input. The implementation represents each object with at least three points to expose more spatial information than a single location. DeepSeek R1 receives the textual 3D scene description and generates the motion plan. The output is limited to 20 spatial waypoints to bound the length of the generated plan. Full prompts are included in the Appendix. PointVLM uses the front-camera view in these experiments. The image input has a resolution of 256Ă256256Ă 256. Table 3: Quantitative comparisons on object reference (RoboRefIt). The metric is percentage of predicted points within the target mask. Qwen-VL LLaVA-NeXT SpaceLLaVA GPT-4o PointVLM 24.1±0.9 20.0±0.9 21.3±0.9 15.3±1.3 52.1±1.2 Table 4: Ablation on the data composition. Results on RoboRefIt show that best results are achieved when all of the data sources are combined during instruction-tuning. No VQA No LVIS No Pixel No Sim No Robo All 42.5±3.7 25.8±2.1 32.6±5.3 46.6±3.2 47.8±3.3 52.1±1.2 Zero-shot performance in simulation The simulation evaluation covers 14 tasks, including grasping and non-prehensile interactions. We measure task success under the specified environment and baseline inputs. Benchmark and execution. The 14 RLBench tasks (James et al. 2020) vary in object identity, placement, and the number of required interactions. The simulator is CoppeliaSim, accessed through PyRep, with a Franka Panda arm and parallel gripper. The environment provides four RGB-D cameras; the front view supplies the PointVLM input described above. A motion planner (Sucan et al. 2012) converts the requested waypoints into executable robot motion. Task-specific success conditions are listed in the appendix. Comparison methods and their inputs. We include Code-as-Policies (CAP) (Liang et al. 2023), Scaling-up-Distilling-Down (Ha et al. 2023), and VoxPoser (Huang et al. 2023). Their control interfaces differ: CAP composes programs from supplied action primitives; Scaling-up performs language-guided exploration with 6-DoF primitives; VoxPoser obtains motion targets from spatial value maps. In this evaluation, CAP and Scaling-up receive simulator states and object models, and VoxPoser receives segmented object point clouds. These inputs differ from HiWEâs learned grounding, so the table compares the specified complete systems rather than isolating planner quality. This benchmark protocol and its baseline results are shared with the GeneralVLA report (Ma et al. 2026). Coverage and limitations. Table 2 reports a nonzero success rate for HiWE on every tested task. The corresponding coverage is 10 tasks for Scaling-up, 9 for VoxPoser, and 7 for CAP. HiWE has the highest reported mean on 10 tasks; the remaining tasks show that its advantage is not uniform. In particular, non-prehensile interactions and fine manipulation expose the need for more accurate pose estimates or adjustments during execution. The illustrated rollouts (Figure 3) show examples with multiple objects and interaction stages, rather than establishing performance on every such task. Real-world experiments Hardware and trial design. Physical tests use an Agilex-2.0 Piper arm with a parallel gripper and an Intel RealSense L515 LiDAR RGB-D camera viewing the workspace from above. Language instructions specify four tasks: move_spray_bottle, open_drawer, open_jar, and sort_object. Each task has three trials of ten episodes, with object poses varied across episodes. Table 5 summarizes the resulting success rates. Table 5: Zero-shot success rates (%) in real-world manipulation. HiWE consistently outperforms Code as Policies (CAP) (Liang et al. 2023) and RoboPoint across four representative tasks. Method Move Spray Bottle Open Drawer Open Jar Sort Object CAP 6.67 0.00 36.67 70.00 RoboPoint 0.00 0.00 20.00 63.33 HiWE 60.00 33.33 56.67 80.00 Observed behavior. HiWE succeeds in at least some episodes of all four tasks and has a higher reported success rate than both listed baselines. The examples in illustrate the role of spatial planning: bottle placement uses the inferred bottle height, while drawer opening requires a motion consistent with the drawerâs orientation. In the evaluated baseline configurations, RoboPoint supplies localization without the trajectory-planning stage, and CAP lacks a supplied drawer-opening primitive. These implementation choices constrain what can be concluded from the comparison. Ablation Study Point location accuracy of PointVLM Point localization is measured by the fraction of predicted points that fall within the annotated target mask. Table 3 reports the mean and standard deviation across 3 runs. PointVLM obtains the highest value among the listed models; this metric evaluates localization rather than the feasibility of a complete robot motion. Table 4 removes each supervision source in turn. Every removal lowers the reported RoboRefIt accuracy, with the largest decrease occurring when LVIS is omitted. The result supports using segmentation-derived semantic point annotations in the training mixture. Information required by 3DLLM The planner ablations vary the dimensionality of the coordinates, the number of points describing each object, and the inclusion of obstacles (Table 6). The 2D variant plans in image coordinates before converting the path into 3D. The one-point variant removes the geometric extent supplied by multiple points. The no-obstacle variant retains the target object but omits surrounding obstacle information. These changes test whether the available scene description is sufficient for interactions such as extracting an umbrella along the direction permitted by its stand. The full representation performs best or ties for best on the two reported tasks. Obstacle information has a clear effect on Take umbrella, whereas its removal leaves the reported Put block result unchanged. The results therefore support task-dependent benefits rather than universal necessity of every input. Figure 4: The multi-view robustness of PointVLM. This assists 3DLLM in reasoning about the direction for pulling out the block. Table 6: Ablation Study on 3DLLM for Trajectory Planning Method Take umbrella Put block 3DLLM-2D 2.00±2.00 19.33±5.03 3DLLM-1point 26.67±3.06 82.00±4.00 3DLLM w/o obstacle 23.33±7.02 93.33±4.16 3DLLM 64.67±15.01 93.33±4.16 Table 7: Ablation Study on the HGM Module Method Play jenga Take umbrella HGM w/o rgb 56.67±7.57 34.00±12.49 HGM w/o 3D point 0.00±0.00 0.00±0.00 HGM w/o filter-C 59.33±5.03 54.67±15.01 HGM w/o filter-N 78.67±7.57 52.00±14.00 HGM 82.00±10.39 64.67±15.01 Ablation study of the HGM module Table 7 tests the information and selection rules used by HGM on Play jenga and Take umbrella. Removing RGB leaves depth-only grasp estimation. Removing the semantic 3D points eliminates the target-object guidance. The filter-C ablation disables collision rejection; the filter-N ablation samples a remaining candidate randomly instead of using the nearest-center rule. The full HGM configuration has the highest reported mean on both tasks. Removing target-point guidance yields no successful trials in this evaluation, while the other ablations reduce success without eliminating it. These findings concern the tested tasks and do not imply that each modality is indispensable for every manipulation problem. Data scaling We examine how the performance of an RVT-2 policy changes as more HiWE demonstrations are supplied for training. The comparison uses demonstrations generated by RLBench as a reference. The reported linear fits have slopes of 0.543 for HiWE-generated demonstrations and 0.156 for RLBench-generated demonstrations. These coefficients summarize the evaluated range and are not a general scaling law. Conclusion HiWE establishes a point-based pipeline for connecting visual grounding, spatial language planning, and grasp execution. The original system uses a fine-tuned PointVLM to produce geometric anchors, 3DLLM to plan from the current scene, and HGM to resolve grasp poses. Its evaluation examines manipulation success, the visual supervision mixture, and the spatial information needed for planning. Successful executions also provide demonstrations for policy training. GeneralVLA subsequently extends this framework through affordance segmentation and persistent experience retrieval (Ma et al. 2026); these mechanisms are outside the HiWE configuration evaluated here. Remaining limitations include errors in point localization, incomplete geometric descriptions, and grasp execution failures. Zero-shot deployment should be understood in the stated task-specific sense, since existing robot observations are part of the visual training data. References Brohan et al. (2023) A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. J. Joshi, et al. RT-1: robotics transformer for real-world control at scale. In Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, K. E. Bekris, K. Hauser, S. L. Herbert, and J. Yu (Eds.), External Links: Link, Document Cited by: Related Work. Doersch et al. (2023) C. Doersch, Y. Yang, M. VecerĂk, D. Gokay, A. Gupta, Y. Aytar, J. Carreira, and A. Zisserman TAPIR: tracking any point with per-frame initialization and temporal refinement. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, p. 10027â10038. External Links: Link, Document Cited by: Related Work. Gu et al. (2024) J. Gu, S. Kirmani, P. Wohlhart, Y. Lu, M. G. Arenas, K. Rao, W. Yu, C. Fu, K. Gopalakrishnan, et al. RT-trajectory: robotic task generalization via hindsight trajectory sketches. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, External Links: Link Cited by: Related Work. Gupta et al. (2019) A. Gupta, P. DollĂĄr, and R. B. Girshick LVIS: A dataset for large vocabulary instance segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, p. 5356â5364. External Links: Link, Document Cited by: PointVLM: predicting interaction locations. Ha et al. (2023) H. Ha, P. Florence, and S. Song Scaling up and distilling down: language-guided robot skill acquisition. In Conference on Robot Learning, CoRL 2023, 6-9 November 2023, Atlanta, GA, USA, J. Tan, M. Toussaint, and K. Darvish (Eds.), Proceedings of Machine Learning Research, Vol. 229, p. 3766â3777. External Links: Link Cited by: Related Work, Table 2, Table 2, Zero-shot performance in simulation. Huang et al. (2023) W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei VoxPoser: composable 3d value maps for robotic manipulation with language models. In Conference on Robot Learning, CoRL 2023, 6-9 November 2023, Atlanta, GA, USA, J. Tan, M. Toussaint, and K. Darvish (Eds.), Proceedings of Machine Learning Research, Vol. 229, p. 540â562. External Links: Link Cited by: Related Work, Table 2, Table 2, Zero-shot performance in simulation. James et al. (2020) S. James, Z. Ma, D. R. Arrojo, and A. J. Davison RLBench: the robot learning benchmark & learning environment. IEEE Robotics Autom. Lett. 5 (2), p. 3019â3026. External Links: Link, Document Cited by: Table 2, Zero-shot performance in simulation. Kafle and Kanan (2017) K. Kafle and C. Kanan An analysis of visual question answering algorithms. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, p. 1983â1991. External Links: Link, Document Cited by: PointVLM: predicting interaction locations. Khazatsky et al. (2024) A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, P. D. Fagan, J. Hejna, M. Itkina, M. Lepert, et al. DROID: A large-scale in-the-wild robot manipulation dataset. In Robotics: Science and Systems X, Delft, The Netherlands, July 15-19, 2024, D. Kulic, G. Venture, K. E. Bekris, and E. Coronado (Eds.), External Links: Link, Document Cited by: Introduction. Kim et al. (2024) M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. P. Foster, P. R. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn OpenVLA: an open-source vision-language-action model. In Conference on Robot Learning, 6-9 November 2024, Munich, Germany, P. Agrawal, O. Kroemer, and W. Burgard (Eds.), Proceedings of Machine Learning Research, Vol. 270, p. 2679â2713. External Links: Link Cited by: Introduction, Related Work. Li et al. (2024) X. Li, K. Hsu, J. Gu, O. Mees, K. Pertsch, H. R. Walke, C. Fu, I. Lunawat, I. Sieh, S. Kirmani, S. Levine, J. Wu, C. Finn, H. Su, Q. Vuong, and T. Xiao Evaluating real-world robot manipulation policies in simulation. In Conference on Robot Learning, 6-9 November 2024, Munich, Germany, P. Agrawal, O. Kroemer, and W. Burgard (Eds.), Proceedings of Machine Learning Research, Vol. 270, p. 3705â3728. External Links: Link Cited by: PointVLM: predicting interaction locations. Liang et al. (2023) J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng Code as policies: language model programs for embodied control. In IEEE International Conference on Robotics and Automation, ICRA 2023, London, UK, May 29 - June 2, 2023, p. 9493â9500. External Links: Link, Document Cited by: Related Work, Table 2, Table 2, Zero-shot performance in simulation, Table 5. Liu et al. (2023) H. Liu, C. Li, Q. Wu, and Y. J. Lee Visual instruction tuning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), External Links: Link Cited by: PointVLM: predicting interaction locations. Ma et al. (2026) G. Ma, S. Wang, Z. Zhang, S. Yu, and H. Tang GeneralVLA: generalizable vision-language-action models with knowledge-guided trajectory planning. Note: Version 1 External Links: 2602.04315, Link Cited by: Figure 2, Relationship to GeneralVLA., Related Work, System interfaces, 3DLLM: planning from a spatial description, Zero-shot performance in simulation, Relationship between the reported experiments., Conclusion. Nasiriany et al. (2024) S. Nasiriany, F. Xia, W. Yu, T. Xiao, J. Liang, I. Dasgupta, A. Xie, D. Driess, A. Wahid, et al. PIVOT: iterative visual prompting elicits actionable knowledge for vlms. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, External Links: Link Cited by: Related Work. Niu et al. (2024) D. Niu, Y. Sharma, G. Biamby, J. Quenum, Y. Bai, B. Shi, T. Darrell, and R. Herzig LLARVA: vision-action instruction tuning enhances robot learning. In Conference on Robot Learning, 6-9 November 2024, Munich, Germany, P. Agrawal, O. Kroemer, and W. Burgard (Eds.), Proceedings of Machine Learning Research, Vol. 270, p. 3333â3355. External Links: Link Cited by: Related Work. OâNeill et al. (2024) A. OâNeill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, A. Tung, A. Bewley, et al. Open x-embodiment: robotic learning datasets and RT-X models : open x-embodiment collaboration. In IEEE International Conference on Robotics and Automation, ICRA 2024, Yokohama, Japan, May 13-17, 2024, p. 6892â6903. External Links: Link, Document Cited by: Introduction, PointVLM: predicting interaction locations. Sucan et al. (2012) I. A. Sucan, M. Moll, and L. E. Kavraki The open motion planning library. IEEE Robotics Autom. Mag. 19 (4), p. 72â82. External Links: Link, Document Cited by: Zero-shot performance in simulation. Sundaresan et al. (2023) P. Sundaresan, S. Belkhale, D. Sadigh, and J. Bohg KITE: keypoint-conditioned policies for semantic manipulation. In Conference on Robot Learning, CoRL 2023, 6-9 November 2023, Atlanta, GA, USA, J. Tan, M. Toussaint, and K. Darvish (Eds.), Proceedings of Machine Learning Research, Vol. 229, p. 1006â1021. External Links: Link Cited by: Related Work. Wang et al. (2024) P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Lin Qwen2-vl: enhancing vision-language modelâs perception of the world at any resolution. CoRR abs/2409.12191. External Links: Link, Document, 2409.12191 Cited by: PointVLM: predicting interaction locations. Wen et al. (2024) C. Wen, X. Lin, J. I. R. So, K. Chen, Q. Dou, Y. Gao, and P. Abbeel Any-point trajectory modeling for policy learning. In Robotics: Science and Systems X, Delft, The Netherlands, July 15-19, 2024, D. Kulic, G. Venture, K. E. Bekris, and E. Coronado (Eds.), External Links: Link, Document Cited by: Related Work. Yuan et al. (2024a) C. Yuan, C. Wen, T. Zhang, and Y. Gao General flow as foundation affordance for scalable robot learning. In Conference on Robot Learning, 6-9 November 2024, Munich, Germany, P. Agrawal, O. Kroemer, and W. Burgard (Eds.), Proceedings of Machine Learning Research, Vol. 270, p. 1541â1566. External Links: Link Cited by: Related Work. Yuan et al. (2024b) W. Yuan, J. Duan, V. Blukis, W. Pumacay, R. Krishna, A. Murali, A. Mousavian, and D. Fox RoboPoint: A vision-language model for spatial affordance prediction in robotics. In Conference on Robot Learning, 6-9 November 2024, Munich, Germany, P. Agrawal, O. Kroemer, and W. Burgard (Eds.), Proceedings of Machine Learning Research, Vol. 270, p. 4005â4020. External Links: Link Cited by: Related Work, PointVLM: predicting interaction locations. Yuan et al. (2023) W. Yuan, A. Murali, A. Mousavian, and D. Fox M2T2: multi-task masked transformer for object-centric pick and place. In Conference on Robot Learning, CoRL 2023, 6-9 November 2023, Atlanta, GA, USA, J. Tan, M. Toussaint, and K. Darvish (Eds.), Proceedings of Machine Learning Research, Vol. 229, p. 3619â3630. External Links: Link Cited by: HGM: resolving local grasp geometry.