Paper deep dive
HumanoidToolBench: Benchmarking Humanoid Tool Use from Selection to Mobile Execution
Kyochul Jang, Seohyeon Park, Ohchul Kwon, Sangjun Park, Junhyeok Choi, Seungyeop Yi, Chaeyun Kim, Sangkyu Lee, Idan Szpektor, Avi Caciularu, Jongmin Park, Youngjae Yu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/4/2026, 2:24:19 AM
Summary
The paper introduces HumanoidToolBench, a benchmark for evaluating humanoid tool use, and ToolBook, a dataset of 3.1k demonstrations. The benchmark assesses 18 tasks across three scenarios (BallMove, BallRetrieve, IceBreak), three execution levels (selection/pickup, stationary use, mobile use), and two tool-set modes (standard, decoy). It highlights significant gaps between tool selection and task completion, particularly in mobile scenarios and with unseen tools, using policies like GR00T N1.7 and FastWAM.
Entities (10)
Relation Signals (9)
ToolBook → collectedon → Unitree G1
confidence 95% · ToolBook, a dataset of 3.1k demonstrations collected in simulation and on a real Unitree G1
HumanoidToolBench → includesscenario → IceBreak
confidence 95% · Three scenarios, BallMove, BallRetrieve, and IceBreak
HumanoidToolBench → includesscenario → BallMove
confidence 95% · Three scenarios, BallMove, BallRetrieve, and IceBreak
HumanoidToolBench → includesscenario → BallRetrieve
confidence 95% · Three scenarios, BallMove, BallRetrieve, and IceBreak
HumanoidToolBench → usesdataset → ToolBook
confidence 95% · We introduce HumanoidToolBench... together with ToolBook, a dataset of 3.1k demonstrations
HumanoidToolBench → implementedin → MuJoCo
confidence 92% · We implement the simulation environment for Humanoid ToolBench in MuJoCo
HumanoidToolBench → evaluatespolicy → GR00T N1.7
confidence 90% · Evaluation of seven policies... Focused GR00T N1.7 probes
HumanoidToolBench → evaluatespolicy → FastWAM
confidence 90% · FastWAM attaining the highest L1 success throughout
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:As robotic hardware and learning methods advance, humanoids need tools to perform tasks beyond their inherent physical limits. Successful tool use requires selecting a suitable tool and coordinating manipulation and, when needed, locomotion to complete the task. Existing benchmarks do not jointly evaluate these capabilities on a humanoid. We introduce HumanoidToolBench, an 18-task benchmark spanning three scenarios, three execution levels, and two tool-set modes, together with ToolBook, a dataset of 3.1k demonstrations collected in simulation and on a real Unitree G1. Evaluation of seven policies in simulation and three on the real robot reveals substantial gaps between selecting a suitable tool and completing the task. Focused GR00T N1.7 probes show reduced selection accuracy on unseen tools and continued task execution under unrelated instructions. Code and data are available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.02089v1
- Canonical: https://arxiv.org/abs/2610.02089v1
Trouble viewing inline? Open PDF directly →
Full Text
40,570 characters extracted from source content.
Expand or collapse full text
HumanoidToolBench: Benchmarking Humanoid Tool Use from Selection to Mobile Execution Kyochul Jang 1 , Seohyeon Park 1 , Ohchul Kwon 1 , Sangjun Park 2 , Junhyeok Choi 1 , Seungyeop Yi 1 , Chaeyun Kim 1 , Sangkyu Lee 1 , Idan Szpektor 3 , Avi Caciularu 3 , Jongmin Park 1 , and Youngjae Yu 1∗ 1 Seoul National University, 2 University of Massachusetts Amherst, 3 Google Research ∗ Corresponding author. Fig. 1.Overview of HumanoidToolBench. A benchmark for humanoid tool use with 18 tasks spanning three scenarios, three execution levels, and standard and decoy tool sets. With 55 tool assets and 3.1k demonstrations, we evaluate seven policies in simulation and three on the real robot. Abstract: As robotic hardware and learning methods advance, humanoids need tools to perform tasks beyond their inherent physical limits. Successful tool use requires selecting a suit- able tool and coordinating manipulation and, when needed, locomotion to complete the task. Existing benchmarks do not jointly evaluate these capabilities on a humanoid. We intro- duce HumanoidToolBench, an 18-task benchmark spanning three scenarios, three execution levels, and two tool-set modes, together with ToolBook, a dataset of 3.1k demonstrations collected in simulation and on a real Unitree G1. Evaluation of seven policies in simulation and three on the real robot reveals substantial gaps between selecting a suitable tool and completing the task. Focused GR00T N1.7 probes show reduced selection accuracy on unseen tools and continued task execution under unrelated instructions. Code and data are available at https://snu-pi.github.io/HumanoidToolBench/. I. INTRODUCTION Recent advances in robot hardware and learning have accelerated research on humanoids that can perform diverse tasks in human environments [1], [2]. Humans overcome physical limitations by using tools to extend the reach and capabilities of their bodies [3], [4]. Humanoids intended to work in the same environments must likewise be able to use tools when their bodies alone cannot accomplish a task. This requires selecting a tool whose properties suit the goal and coordinating grasping, manipulation, and, when needed, locomotion to use it effectively. The grasp determines how much of a tool’s reach is available and how its working end is oriented for pushing, pulling, or striking. Body motion must then preserve the spatial relationship needed for con- tact with the target object. Evaluating humanoid tool use therefore requires testing the full process from selection to task completion. Existing benchmarks address parts of this problem (Ta- ble I). HumanoidBench [2] includes spoon and window- wiping tasks, but does not evaluate the choice among tools with contrasting functional properties. Dedicated tool- use benchmarks are limited to visual question answering (VQA) [5] or fixed-base arms [6], [7]. A benchmark is there- fore needed that jointly evaluates functional tool selection and task completion on a humanoid in both stationary and mobile settings. To address these limitations, we introduce Humanoid ToolBench, an 18-task benchmark, and ToolBook, 3.1k human demonstrations collected in simulation and on a real Unitree G1 (Fig. 1). The design separates selection from execution. Three scenarios, BallMove, BallRetrieve, and IceBreak, require sufficient reach, a shape that can engage and pull, and physical properties that support striking, arXiv:2610.02089v1 [cs.RO] 1 Oct 2026 TABLE I COMPARISON OF HUMANOIDTOOLBENCH WITH RELATED MANIPULATION, HUMANOID, AND TOOL-USE BENCHMARKS.△, TOOL USE INCLUDED BUT NOT THE PRIMARY EVALUATION SCOPE; 0, NOT APPLICABLE OR NONE RELEASED; A HYPHEN, NOT REPORTED. Tool UseTask Structure# Human Demos BenchmarkEmbodiment Tool-Use Scope # tool assets Selection Hierarchical Levels Decoy Variants SimReal LIBERO [8]Single-arm ×0 × ✓ 6,5000 RoboTwin 2.0 [9]Dual-arm △11 × ✓ ×040 HumanoidBench [2]Humanoid △1 ×00 SIMPLE [10]Humanoid ×0 × ✓ ×-- WOLF-VLA [11]Humanoid ×0 × ✓ ×00 PhysToolBench [5]None (VQA) ✓ 0 ✓ ×00 DexCraft [6]Single-arm ✓ 6 ×30150 RoboWits [7]Dual-arm ✓ 17 ✓ × ✓ 1,2070 OursHumanoid ✓ 55 ✓ 3,00391 respectively. Three execution levels distinguish selection and pickup (L0), stationary tool use (L1), and mobile tool use (L2), separating acquisition from task completion. Standard (S) and decoy (D) modes vary the alternatives under the same instruction: standard mode presents the suitable tool among unrelated objects, while decoy mode adds a competing tool that lacks a required property. Combining levels and modes exposes cases where a suitable choice is followed by failed execution, as well as behavior changes induced by competing candidates. ToolBook spans this structure to support policy training and real-robot adaptation. Our central question is how reliably humanoid policies select and use suitable tools, and how these behaviors re- spond to changes in tool appearance and task instructions. We examine task completion, selection and execution fail- ures, and responses to changes in tools and instructions. We evaluate seven policies in simulation and three on the real G1, comparing task success across scenarios, execution levels, and modes. Recorded interaction events distinguish contacting a candidate, lifting the suitable tool, and applying it to the target object. They reveal whether low completion accompanies limited pickup or persists despite frequent tool interaction, and capture changes in initial choice that final success can conceal. Focused probes of a separately trained standard-mode GR00T N1.7 policy examine representations and initial contact for unseen suitable tools, and success and motion under instruction changes in a fixed IceBreak scene. The results reveal gaps between acquisition and completion, with further declines when tasks combine locomotion and manipulation. Our contributions are as follows: • To the best of our knowledge, HumanoidToolBench is the first benchmark of humanoid tool use that spans selection and pickup, stationary tool use, and mobile tool use, with ToolBook, a dataset of 3.1k human demonstrations in simulation and on a Unitree G1. • HumanoidToolBench combines execution levels, decoy tool sets, and recorded interaction events to distinguish limitations of selection and task execution. • We evaluate seven policies in simulation and three on the real G1 and probes to examine behavior beyond final success. I. RELATED WORK Humanoid Benchmarks. Manipulation benchmarks such as LIBERO [8] and RoboTwin 2.0 [9] evaluate vision- language-action models (VLAs) on single- and dual-arm manipulators. Meanwhile, humanoid policies such as GR00T N1.7 [1] and Ψ 0 [12] require evaluation on a humanoid, and several benchmarks have been proposed: Humanoid- Bench [2] evaluates whole-body control, including spoon and window-wiping tasks, and SIMPLE [10] and WOLF- VLA [11] extend evaluation to humanoid VLAs. SIMPLE stages its tasks to report where a policy fails but does not vary the decision a task requires, and WOLF-VLA releases no human demonstrations for training. These benchmarks do not jointly evaluate functional selection, stationary and mobile use, and controlled tool-set variations (Table I). Our HumanoidToolBench fills this gap: a humanoid selects a tool from 55 assets, with or without a decoy, and uses it while stationary and while moving, with 3,003 simulation and 91 real-robot demonstrations for training. Tool Use Benchmarks in Robotics. Tool use extends the limits of the body [3], [13] and is also studied for AI systems [14], [15], but physical tool use additionally requires a body that grasps, carries, and applies the tool. Three benchmarks target robotic tool use (Table I): Phys- ToolBench [5] evaluates tool understanding through visual question answering without acting on it, DexCraft [6] collects tool-use grasps on a single arm with the tool provided, and RoboWits [7] asks a dual-arm robot to select a tool in unexpected situations. None evaluates tool use on a humanoid or combines execution levels with controlled tool- set variations: PhysToolBench tiers understanding but not ex- ecution, DexCraft and RoboWits do not divide their tasks into progressive levels, and only RoboWits varies the decision a task requires. Our HumanoidToolBench evaluates tool use on a humanoid that selects, carries, and applies the tool, and structures every scenario along two axes: execution levels Fig. 2.Task structure. 3 scenarios × 3 execution levels × 2 tool-set modes = 18 tasks, built on a library of 55 tool assets spanning three functional categories. Execution levels progress from selection and pickup (L0) to stationary tool use (L1) and mobile tool use (L2). Decoy mode requires choosing the tool that meets the task’s spatial, affordance, or physical requirement. increase the demands of task completion, while standard and decoy modes vary the tool choice under the same instruction. Pairing these axes supports comparisons among policies facing the same alternatives and within a policy as execution demands change. This links a tool’s functional suitability to the physical demands of grasping it, contacting the target, and coordinating body motion. I. HUMANOIDTOOLBENCH HumanoidToolBench evaluates whether humanoid poli- cies select tools suited to the environment and the user’s instruction, then use them to complete the requested task (Fig. 1). We instantiate the hierarchical levels in Table I as execution levels, which increase the demands of tool use. Each task is a combination of scenario, execution level, and tool-set mode, yielding 18 tasks from three scenarios, three execution levels, and two modes. ToolBook supplies 3.1k human demonstration trajectories from simulation and a real Unitree G1. Sections I-A to I-E formulate the task and describe its design philosophy, tasks, environment, demonstration collection, and evaluation protocol. I-A. Problem Formulation A humanoid receives a language goal that does not name a tool, and it must select a suitable tool from the candidates in the scene and use it to achieve the goal. We model each task as a goal-conditioned partially observable Markov de- cision process (POMDP) in which a binary success criterion replaces the reward, M = (S,A,O,P, Ω,G),(1) where S contains robot, object, and controller states, A and O are the command and observation spaces, P (s t+1 | s t ,a t ) combines physics with the fixed whole-body controller, Ω(o t | s t ) generates observations, and G marks success. Each episode starts with a candidate set B that holds one suitable tool b ⋆ and two unrelated objects, plus a decoy tool in decoy mode, and the language goal g describes the task without naming b ⋆ . The observation o t = (I t ,x t ) contains the head-camera image and the robot state in Eq. (4), which carries no object information, so a policy must infer which candidate is suitable from g and the image. Given the history h t = (o 0 ,a 0 ,...,o t ), a policy acts according to a t ∼ π θ (·| h t ,g),(2) where θ denotes the parameters of policy π θ , learned from human demonstrations and held fixed during evaluation; an action-chunking policy conditions on an earlier part of h t . An episode τ ends when the task succeeds, when it fails, or at the step limit H . Its success indicator G(τ ) ∈ 0, 1 requires lifting b ⋆ in selection and pickup tasks and reaching the scenario’s goal state in stationary and mobile tool-use tasks. We report the success rate over N attempted episodes, including timeouts and, on the real robot, safety stops, ˆ J (π θ ) = 1 N N X i=1 G τ (i) .(3) I-B. Design Philosophy and Tasks HumanoidToolBench follows three design principles: real- world utility, interpretability, and functional selection. Each task instantiates the POMDP in Eq. (1) with a candidate set B, a language goal g, and a success criterion G(τ ). For example, g = “Pick the tool and move the ball to the target” does not name b ⋆ , so the policy must select it from B. Real-world utility. Tool use should be evaluated on func- tions that tools serve in everyday settings, over the full process from selecting a tool to achieving the user’s goal. The scenarios instantiate this principle by covering three com- mon tool functions: extending reach, engaging and pulling, and transmitting force (Fig. 2). Specifically, (i) BallMove requires a long stick to push a ball beyond the robot’s arm reach into a 0.12 m circular goal region, imposing a spatial requirement on tool length; (i) BallRetrieve requires a hook to catch the far side of the ball and pull it into the circular goal region, imposing an affordance requirement on tool shape; and (i) IceBreak requires the robot to strike two ice blocks hard enough to break them and scatter their fragments, imposing a physical requirement on tool material and structure. Interpretability. The evaluation distinguishes selection, pickup, and task completion through execution levels and recorded interaction events. Three execution levels impose progressively greater demands. L0 (selection and pickup) succeeds when b ⋆ rises at least 0.08 m, evaluating selection and grasping. L1 (stationary tool use) adds the scenario goal, which can be completed without locomotion. L2 (mobile tool use) moves the BallMove goal region, the BallRetrieve ball, or one IceBreak block to the side of the table where the tools lie, so the task calls for moving along the table while holding the tool. Within each episode, recorded events provide a finer account of selection and execution: candidate contact, correct first contact, correct-tool lift, tool-target contact, and task success (Fig. 5). In these event names, correct refers to the suitable tool b ⋆ , and target refers to the ball or ice. Functional selection. Tool choice and use should respond to the functional requirements of the task rather than to familiar objects and motions. The two tool-set modes test this by varying the alternatives a policy must compare under the same goal g. In standard mode (S), B contains b ⋆ and two unrelated objects, so recognizing the only tool suffices. Decoy mode (D) adds one decoy tool that differs from b ⋆ mainly in the property the scenario requires: the same stick at 0.30 m instead of 0.50 m with equal mass in BallMove, a straight stick with the hook’s length and mass in BallRetrieve, and a light, compliant fly swatter, paint roller, or plunger instead of a metal hammer in IceBreak (Fig. 2). The instruction and unrelated objects follow the same sampling rules in both modes, while shuffled candidate positions and the added decoy vary the available choices. The scanned tools vary in color and texture as well as task- relevant properties. Section IV-D compares representations of seen and unseen suitable-tool instances in decoy scenes. Complementary instruction probes vary wording and task relevance while holding the scene fixed. I-C. Environment We implement the simulation environment for Humanoid ToolBench in MuJoCo [16]. The robot stands in a room in front of a long table, with the candidate objects on the left half of the table and the task objects on the right half. Each episode draws a new scene: it samples B, with tools from 55 Objaverse [17] assets curated through MolmoSpaces [18], shuffles the candidates, and randomizes placements, the robot spawn, surface materials, and lighting. Simulation and real-robot evaluation use a Unitree G1 with 29 body joints, two seven-joint Dex3 hands, and a head camera that provides the image I t , under the same decoupled control stack adapted from GR00T-WholeBodyControl [19]. The policy π θ observes I t and the statex t and outputs the action a t , x t = q h t |z 14 ,q a t |z 14 ,q w t |z 3 , ̄z t |z 1 ∈R 32 ,(4) a t = q h,⋆ t |z 14 ,q a,⋆ t |z 14 ,q w,⋆ t | z 3 , z ⋆ t |z 1 ,u t |z 4 ∈R 36 ,(5) where h, a, and w index the hand, arm, and waist joints, ⋆ marks targets, ̄z t and z ⋆ t are the previous and new base- height commands, andu t = (v x ,v y ,m,ψ ⋆ ) holds forward and lateral velocities, a turning-mode signal, and a target heading. The pretrained GR00T Balance/Walk reinforcement learning controllers execute the locomotion components of a t , producing the lower body targets required for bipedal motion. I-D. Demonstration Collection ToolBook records human tool selection and use through Meta Quest teleoperation in simulation and on the real G1. Eight experienced teleoperators made 1,959 simulation attempts at L1 and L2, yielding 100 successful demonstrations for each of the 12 L1/L2 tasks, 1,200 in total. For every attempt in which b ⋆ rises 0.08 m, we truncate the trajectory at the first such frame, yielding 1,803 L0 trajectories without sepa- rate collection. These comprise 1,189 from the successes, since the other 11 used b ⋆ without lifting it that high, and 614 from unsuccessful attempts. With 91 curated real- robot demonstrations of L1 BallMove and BallRetrieve, the resulting ToolBook dataset comprises 3,094 trajectories, with the following breakdown: 1,803 | z simulation L0 +1,200 |z simulation L1/L2 +91 |z real-robot L1 =3,094 |z total trajectories Refer to Table I for the counts by task. TABLE I TOOLBOOK COVERAGE BY TASK. Simulation: 3,003 trajectories ScenarioL0 (S/D)L1 (S/D)L2 (S/D) BallMove289/244100/100100/100 BallRetrieve388/313100/100100/100 IceBreak316/253100/100100/100 Real robot: 91 trajectories ScenarioExecution levelSD BallMoveL12921 BallRetrieveL12021 Only collected tasks are shown. S/D: standard/decoy counts. Simulation counts are extracted L0 prefixes and successful L1/L2 episodes; real-robot counts are curated demonstrations. I-E. Evaluation Protocol Policies for simulation evaluation are trained on simulation demonstrations (sim-only training); those for real evalua- tion are additionally fine-tuned on real-robot demonstrations (sim+real training). Simulation evaluation covers all 18 tasks, and real-robot evaluation covers L1 BallMove and BallRe- trieve in both modes. We report the success rate ˆ J (π θ ) in Eq. (3) as a percentage; a simulated episode succeeds once TABLE I SIMULATION SUCCESS RATE (%) BY SCENARIO, EXECUTION LEVEL, AND TOOL-SET MODE. WITHIN EACH CELL, VALUES ARE REPORTED IN THE ORDER BALLMOVE / BALLRETRIEVE / ICEBREAK. ROWS IN EACH FAMILY ARE ORDERED BY MEAN L1/L2 SUCCESS. IN EACH COLUMN, BOLD AND UNDERLINED CELLS HAVE THE HIGHEST AND SECOND-HIGHEST MEAN OVER THE THREE SCENARIOS. Standard (S)Decoy (D) Family PolicyL0L1L2L0L1L2 IL ACT [20]5 / 0 / 00 / 0 / 00 / 0 / 03 / 4 / 10 / 0 / 00 / 0 / 0 DP [21]14 / 5 / 170 / 0 / 20 / 0 / 011 / 9 / 130 / 0 / 10 / 0 / 0 VLA π 0.5 [22]29 / 24 / 314 / 0 / 121 / 0 / 026 / 28 / 215 / 3 / 11 / 1 / 0 Ψ 0 [12]19 / 14 / 229 / 0 / 131 / 0 / 214 / 15 / 187 / 3 / 83 / 1 / 0 GR00T N1.7 [1]77 / 73 / 8864 / 36 / 6610 / 9 / 3167 / 59 / 7558 / 30 / 6330 / 5 / 48 WAM Cosmos 3 Edge Policy [23]12 / 0 / 20 / 0 / 10 / 0 / 014 / 0 / 02 / 0 / 01 / 0 / 0 FastWAM [24]64 / 62 / 9876 / 57 / 869 / 20 / 6974 / 45 / 9375 / 45 / 828 / 11 / 61 TABLE IV REAL-ROBOT L1 SUCCESS (SUCCESSES / 10 TRIALS). POLICIES ARE TRAINED ON SIMULATION DEMONSTRATIONS AND FINE-TUNED ON REAL-ROBOT DEMONSTRATIONS (SIM+REAL). BallMoveBallRetrieve PolicySDSD ACT1/100/102/101/10 GR00T N1.76/104/104/104/10 FastWAM5/104/104/105/10 Fig. 3. GR00T N1.7 BallMove rollouts in simulation and on the real robot at four matched phases. its criterion in Section I-B holds for one second. Rates are reported by scenario, level, and mode within each evaluation domain. IV. EXPERIMENTS We first compare task success in simulation and on the real G1 and inspect rollout failures (Section IV-B). We then examine execution events to locate the gaps between con- tacting a candidate, lifting the suitable tool, and completing the task (Section IV-C). Finally, we examine GR00T N1.7’s tool-region representations and successful motions under tool and instruction changes (Section IV-D). IV-A. Experimental Setup Policies. We compare three VLAs, GR00T N1.7 (3.14B) [1], Ψ 0 (2.63B) [12], and π 0.5 (3.62B) [22], two world-action models (WAMs), Cosmos 3 Edge Policy (3.37B) [23] and FastWAM (6.02B) [24], and two imitation-learning (IL) baselines, ACT (52.1M) [20] and DP (84.7M) [21]. Sizes are rounded parameter counts for policy checkpoints, including integrated frozen backbones but excluding external frozen encoders and VAEs; tied weights are counted once. These policies cover different approaches to learning robot behav- ior; we assess their ability to select and use tools, with the IL baselines providing task-specific imitation references. Training. For simulation evaluation, every policy learns from the 3,003 simulation trajectories in ToolBook (sim-only training). The policies use DROID checkpoints for GR00T N1.7, π 0.5 , and Cosmos 3, and model-specific pretrained weights for Ψ 0 and FastWAM. ACT and DP gain language conditioning from a frozen CLIP ViT-L/14 text encoder [25]. Each policy is trained once for 40,000 updates with 128 samples per update, drawing at least one sample from each of the 18 tasks. For real evaluation, ACT, GR00T N1.7, and FastWAM continue from their simulation-trained weights for 5,000 updates on the 91 real-robot demonstrations (sim+real training). Evaluation. In simulation, we run 100 episodes for each of the 18 tasks, with initial seeds matched across policies and a one-minute limit of H = 3,000 control steps at 50 Hz. In the real setting, ACT, GR00T N1.7, and FastWAM are eval- uated on the Unitree G1 in L1 BallMove and BallRetrieve, each in S and D modes, with 10 trials per scenario-mode combination and a one-minute limit per trial. Fig. 3 shows example GR00T N1.7 BallMove rollouts in simulation and on the real robot at matched phases. Every policy receives the same observation, the left head-camera image and the robot statex t , and executes each predicted action chunk open loop before the next query. Each policy uses the chunk length it was trained with: 100 actions (2 s) for ACT, 40 for GR00T N1.7, 32 for FastWAM, 30 for Ψ 0 , and 16 for DP, π 0.5 , and Cosmos 3 Edge Policy. ACT and DP condition on a single observation, and no policy uses temporal ensembling. IV-B. Results Table I compares all seven policies in simulation, and Table IV evaluates three policies on the real G1 after sim+real training. Together, they show how completion varies across policies, execution demands, and candidate sets, and whether the broad performance pattern persists on the real robot. Stationary tool use depends on both the policy and tool function. ACT, DP, and Cosmos 3 Edge Policy rarely com- Fig. 4.Selection and spatial alignment errors. Reviewed rollouts show decoy pickup (top) and tool-target alignment errors (bottom). Yellow borders mark the frames where spatial alignment errors occur. plete L1 tasks, while π 0.5 and Ψ 0 achieve modest success, particularly in IceBreak. GR00T N1.7 and FastWAM perform substantially better across all three scenarios in both modes, with FastWAM attaining the highest L1 success throughout. Even these stronger policies perform less well at retrieving than at pushing or striking, indicating that capability varies across tool functions. L0 separately evaluates selection and pickup under its own success criterion: FastWAM, for ex- ample, achieves 64% at L0 and 76% at L1 on standard BallMove. These are success rates in distinct task conditions. To locate failures between acquisition and completion, Sec- tion IV-C examines recorded interaction events within the same L1/L2 evaluation rollouts. Mobile tool use challenges even strong stationary policies. For every policy, success decreases from L1 to L2 whenever L1 success is nonzero. Thus, stationary tool-use capability does not extend reliably to tasks that also require moving the body. The decline is especially pronounced for FastWAM on BallMove, where standard-mode success falls from 76% to 9%; GR00T N1.7 also loses much of its stationary perfor- mance on both ball tasks. The limited L1 success of π 0.5 and Ψ 0 largely disappears at L2, while ACT, DP, and Cosmos 3 Edge Policy already struggle at L1. For GR00T N1.7 and FastWAM, IceBreak retains more success than either ball task, indicating that the mobile-use difficulty depends on the required interaction. The L1-to-L2 decline identifies mobile tool use as a major challenge, with difficulty varying across the required interactions. Decoys challenge strong stationary policies. GR00T N1.7 and FastWAM score lower in D mode across all three L1 scenarios. For FastWAM, BallRetrieve success drops from 57% to 45%, despite its relatively strong stationary tool use. This consistent L1 decline shows that competing candidates challenge performance even without locomotion. Effects at L2 are mixed: GR00T N1.7 improves from 10% to 30% on BallMove, while FastWAM declines across all three scenarios. These outcomes connect task completion to the functional demands of choosing among tools. Section IV-C examines how competing candidates change initial choice, suitable-tool pickup, and subsequent tool-target contact. Stronger simulated policies also lead on the real robot. GR00T N1.7 and FastWAM outperform ACT in every real- Fig. 5.Execution-event success rate (%) by policy in S and D modes, averaged equally over the three scenarios and over L1 and L2. Success requires the goal state to hold for one second (Section I-E). robot condition, consistent with their stronger simulated task completion. For example, on standard BallMove, they succeed in 6/10 and 5/10 trials, respectively, compared with 1/10 for ACT. This performance follows only 5,000 fine- tuning updates from the sim-trained policies using 91 real- robot demonstrations. Thus, policies that learn effective tool use in simulation also achieve substantial completion on the real G1 after a short adaptation stage with a small real- data budget. ACT remains less successful after the same adaptation procedure. Rollout inspection reveals selection and spatial alignment errors. Visual inspection of GR00T N1.7 and FastWAM evaluation videos identifies two failure patterns (Fig. 4). In decoy scenes, both models sometimes grasp and lift the decoy successfully, showing a mismatch between the selected tool and the task despite successful pickup. Other reviewed failures involve grasp poses and tool orientations that prevent effective contact with the target object. These cases point to difficulties in reasoning about tool geometry, object distance, and the orientation needed for the intended interaction. The observations identify task-dependent selection and spatial alignment as challenges beyond acquiring a tool. → Takeaway. Tool-use performance varies across policies and tool functions. Mobile tasks remain challenging, and decoys reduce stationary success for the strongest policies. Stronger simulated performance persists after limited real- data adaptation. IV-C. Where Do Policies Fail in Tool Use? Fig. 5 reports candidate contact, correct first contact, correct-tool lift, tool-target contact, and success. Each value equally averages the event rates over six conditions: three scenarios at L1 and L2 within each mode. L0 is excluded. Each rate measures the fraction of episodes containing the event; an episode can include an incorrect first contact followed by a suitable-tool lift. Fig. 6.Tool-instance changes in L1 decoy scenes. GR00T N1.7 tool- region t-SNE projections and correct first contact for (a) BallMove and (b) BallRetrieve. Open circles denote seen tools; triangles denote unseen tools. Bars A and B give correct first contact for seen and unseen tools, respectively, over 100 episodes each. High contact rates need not yield pickup or completion. In standard mode, DP and π 0.5 contact a candidate in 89.8% and 98.8% of episodes but lift the suitable tool in only 15.0% and 44.2%. GR00T N1.7 and FastWAM have higher lift rates, 90.0% and 89.2%, and tool-target contact rates of 79.8% and 88.2%, while success remains at 36.0% and 52.8%. These contrasts distinguish limited pickup from frequent tool interaction accompanied by lower task success. Decoys affect first contact more than final success. For GR00T N1.7 and FastWAM, correct first contact falls from 87.3% to 77.2% and from 87.7% to 74.7% between standard and decoy modes. Success changes less, from 36.0% to 39.0% and from 52.8% to 47.0%. In decoy mode, correct- tool lift remains more frequent than correct first contact, at 83.7% and 84.5%, showing that initial choice and eventual pickup need not coincide. → Takeaway. Weaker policies struggle to lift the suitable tool, while stronger policies often contact the target without completing the task. For GR00T N1.7 and FastWAM, decoys affect initial choice more than final success. IV-D. Responses to Tool and Instruction Changes For these probes, we use GR00T N1.7 trained on 1,593 standard-mode demonstrations for 40,000 updates. Fig. 6 compares seen and unseen suitable-tool instances in paired L1 decoy scenes; Fig. 7 varies instructions in a fixed L1 standard IceBreak scene. Seen instances appeared during training, while unseen instances meet the same task require- ment. Tool replacement changes representations and initial contact. Fig. 6 shows t-SNE projections of initial post- normalization visual features pooled over the suitable-tool region, using 91 paired scenes for BallMove and 100 for BallRetrieve. Seen and unseen features substantially overlap in BallMove, whereas unseen instances form groups apart from most seen instances in BallRetrieve. Correct first con- tact decreases from 85% to 74% in BallMove and from 59% to 36% in BallRetrieve, measured over 100 episodes per condition. The illustrated hooks differ in shape and surface appearance, and BallRetrieve exhibits both clearer feature separation and the larger selection drop. Despite preserved task suitability, these observations suggest limited generalization to out-of-distribution tool appearances. Fig. 7. Instruction changes in IceBreak L1. GR00T N1.7 under original, paraphrased, and unrelated instructions in one fixed standard-mode scene. (a) Per-rollout mean angle of the seven right-arm joints, shoulder to wrist (top), and of the three right-wrist joints (bottom). Lines show medians across successful rollouts at normalized episode progress, with interquartile bands. (b) Task success over 100 rollouts per instruction. High task success persists under unrelated instructions. In the fixed IceBreak scene, GR00T N1.7 achieves 77% success with the original instruction, 70% with a paraphrase, and 84% with an unrelated sentence (Fig. 7). The prompts are “Pick the tool and break the ice blocks,” “Break the ice blocks using a tool you pick,” and “The weather is nice today,” respectively. Removing the task-relevant request therefore leaves the policy frequently executing and com- pleting the familiar task. The motion profiles support this observation: across successful rollouts, all three conditions show similar broad right-arm profiles and a shared middle- phase wrist pattern. We examine these joints because the arm and wrist position and orient the tool-holding hand. The unrelated condition starts the wrist-angle decrease earlier and returns further toward the initial angle near completion, while retaining the overall task behavior. Because each training scene determines its task, this result shows that the policy completes the task without relying on the instruction; testing instruction following requires conditions in which the instruction conflicts with the scene. → Takeaway. GR00T N1.7 selects unseen suitable tools less reliably and continues the familiar IceBreak task un- der unrelated language. These probes show limited transfer across tool appearances and suggest weak instruction follow- ing despite high task success. V. DISCUSSION AND LIMITATIONS The main benchmark comparison uses one training run per policy, with real-robot evaluation of three policies on two L1 scenarios and focused GR00T N1.7 probes. Episode variation reflects scene and rollout variability, rather than training- seed uncertainty. Execution levels vary in layout as well as locomotion demands, and the real-robot study evaluates simulation training followed by real-data adaptation as a complete pipeline. First contact serves as a proxy for initial choice. Tool-region projections characterize visual-feature organization; establishing functional recognition requires controls for appearance. The instruction probe uses one fixed IceBreak scene and varies wording and task relevance; testing instructions for different goals would further assess goal switching. Broader tools, scenes, goals, and additional training seeds would extend the scope of these findings. The rollout inspection highlights spatial alignment difficul- ties during tool acquisition and use. Egocentric visual input may contribute by providing limited context for estimating tool-object distance and relative orientation. This explanation is a hypothesis motivated by the reviewed videos. Controlled comparisons of camera viewpoints and spatial observations would clarify the contribution of visual context, alongside the policy’s representation, grasping, and control. Similarly, sep- arate simulation-only and real-data adaptation comparisons would measure the contribution of fine-tuning to real-robot performance. VI. CONCLUSION HumanoidToolBench and ToolBook support evaluating humanoid tool use from selection and pickup to mobile task completion. Across seven policies, execution events distinguish failures to lift a suitable tool from failures to complete the task after contact, while decoys change initial choices more than final success. Focused probes additionally reveal sensitivity to unseen tool appearances and continued task execution under unrelated language. These findings motivate policies that adapt selection and execution to func- tional requirements and task goals beyond familiar training patterns. REFERENCES [1] NVIDIA, J. Bjorck, F. Castañeda, N. Cherniadev, X. Da, R. Ding, L. J. Fan, Y. Fang, D. Fox, F. Hu, S. Huang, J. Jang, Z. Jiang, J. Kautz, K. Kundalia, L. Lao, Z. Li, Z. Lin, K. Lin, G. Liu, E. Llontop, L. Magne, A. Mandlekar, A. Narayan, S. Nasiriany, S. Reed, Y. L. Tan, G. Wang, Z. Wang, J. Wang, Q. Wang, J. Xiang, Y. Xie, Y. Xu, Z. Xu, S. Ye, Z. Yu, A. Zhang, H. Zhang, Y. Zhao, R. Zheng, and Y. Zhu, “GR00T N1: An Open Foundation Model for Generalist Humanoid Robots,” Mar. 2025. [2] C. Sferrazza, D.-M. Huang, X. Lin, Y. Lee, and P. Abbeel, “Humanoid- Bench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulation,” in Robotics: Science and Systems X, vol. 20, July 2024. [3] A. Maravita and A. Iriki, “Tools for the body (schema),” Trends in Cognitive Sciences, vol. 8, no. 2, p. 79–86, Feb. 2004. [4] L. Cardinali, F. Frassinetti, C. Brozzoli, C. Urquizar, A. C. Roy, and A. Farnè, “Tool-use induces morphological updating of the body schema,” Current Biology, vol. 19, no. 12, p. R478–R479, June 2009. [5] Z. Zhang, K. Chen, X. Lin, L. Jiang, X. Zheng, Y. Lyu, L. Guo, Y. Li, and Y.-C. Chen, “PhysToolBench: Benchmarking Physical Tool Understanding for MLLMs,” Oct. 2025. [6] Y. Yuan, X. Liu, S. Krishna, and D. Held, “From Grasps to Dexterity: Large-Scale Grasp Pretraining for Dexterous Manipulation,” June 2026. [7] C. Lin, H. Zhang, F. Yu, Z. Chen, T. L. Griffiths, Y. Choi, D. Held, and C. Gan, “RoboWits: Unexpected Challenges for Robotic Creative Problem Solving,” May 2026. [8] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone, “LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning,” Advances in Neural Information Processing Systems, vol. 36, p. 44 776–44 791, Dec. 2023. [9] T. Chen, Z. Chen, B. Chen, Z. Cai, Y. Liu, Z. Li, Q. Liang, X. Lin, Y. Ge, Z. Gu, W. Deng, Y. Guo, T. Nian, X. Xie, Q. Chen, K. Su, T. Xu, G. Liu, M. Hu, H.-a. Gao, K. Wang, Z. Liang, Y. Qin, X. Yang, P. Luo, and Y. Mu, “RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation,” in Forty-Third International Conference on Machine Learning, June 2026. [10] S. Wei, Z. Ni, J. Liu, Z. Zhao, J. Ye, H. Jing, J. Xia, X. Liu, M. Leong, L. Heng, D. Huang, and Y. Wang, “SIMPLE: Simulation-Based Policy Learning and Evaluation for Humanoid Loco-manipulation,” June 2026. [11] M. Boukheddimi, O. Adjali, D. Sontag, and F. Kirchner, “WOLF- VLA: Whole-Body Humanoid Optimal Locomotion Framework for Vision-Language-Action Learning,” June 2026. [12] S. Wei, H. Jing, B. Li, Z. Zhao, J. Mao, Z. Ni, S. He, S. Zang, X. Liu, K. Kang, J. Liu, W. Yuan, M. Pavone, D. Huang, and Y. Wang, “Ψ 0 : An Open Foundation Model Towards Universal Humanoid Loco- Manipulation,” in Robotics: Science and Systems XXII, vol. 22, July 2026. [13] A. Iriki, M. Tanaka, and Y. Iwamura, “Coding of modified body schema during tool use by macaque postcentral neurones,” Neurore- port, vol. 7, no. 14, p. 2325–2330, Oct. 1996. [14] T. Schick, J. Dwivedi-Yu, R. Dessi, R. Raileanu, M. Lomeli, E. Ham- bro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language Models Can Teach Themselves to Use Tools,” in Advances in Neural Information Processing Systems, vol. 36. Curran Associates, Inc., 2023, p. 68 539–68 551. [15] K. Jang, D. Lee, K. Kim, D. Heo, T. Lee, W. Kim, and B. Suh, “DICE-BENCH: Evaluating the Tool-Use Capabilities of Large Lan- guage Models in Multi-Round, Multi-Party Dialogues,” in Findings of the Association for Computational Linguistics: ACL 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, Eds.Vienna, Austria: Association for Computational Linguistics, July 2025, p. 26 822– 26 846. [16] E. Todorov, T. Erez, and Y. Tassa, “MuJoCo: A physics engine for model-based control,” in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, Oct. 2012, p. 5026–5033. [17] M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. Van- derBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi, “Ob- javerse: A Universe of Annotated 3D Objects,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, p. 13 142–13 153. [18] Y. Kim, W. Pumacay, O. Rayyan, M. Argus, W. Han, E. Vanderbilt, J. Salvador, A. Deshpande, R. Hendrix, S. Jauhri, S. Liu, N. M. M. Shafiullah, M. Guru, A. Eftekhar, K. Farley, D. Clay, J. Duan, A. Guru, P. Wolters, A. Herrasti, Y.-C. Lee, G. Chalvatzaki, Y. Cui, A. Farhadi, D. Fox, and R. Krishna, “MolmoSpaces: Large-Scale Open Ecosystem for Robot Manipulation and Navigation,” in Robotics: Science and Systems XXII, vol. 22, July 2026. [19] Z. Luo, Y. Yuan, T. Wang, C. Li, S. Chen, F. Castañeda, Z.-A. Cao, J. Li, D. Minor, Q. Ben, X. Da, R. Ding, C. Hogg, L. Song, E. Lim, E. Jeong, T. He, H. Xue, W. Xiao, Z. Wang, S. Yuen, J. Kautz, Y. Chang, U. Iqbal, L. Fan, and Y. Zhu, “Sonic: Supersizing motion tracking for natural humanoid whole-body control,” arXiv preprint arXiv:2511.07820, 2025. [20] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn, “Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware,” in Robotics: Sci- ence and Systems XIX, vol. 19, July 2023. [21] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. C. Burchfiel, and S. Song, “Diffusion Policy: Visuomotor Policy Learning via Action Diffusion,” in Robotics: Science and Systems XIX, vol. 19, July 2023. [22] K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. R. Equi, C. Finn, N. Fusai, M. Y. Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke, A. Walling, H. Wang, L. Yu, and U. Zhilinsky, “π 0.5 : A Vision-Language-Action Model with Open-World General- ization,” in Proceedings of The 9th Conference on Robot Learning. PMLR, Oct. 2025, p. 17–40. [23] NVIDIA, Aditi, et al., “Cosmos 3: Omnimodal World Models for Physical AI,” June 2026. [24] T. Yuan, Z. Dong, Y. Liu, and H. Zhao, “Fast-WAM: Do World Action Models Need Test-time Future Imagination?” Mar. 2026. [25] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning Transferable Visual Models From Natural Language Su- pervision,” in Proceedings of the 38th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, July 2021, p. 8748– 8763.