Paper deep dive
RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks?
Zhenxuan Fan, Bo Zhang, Yutong Lin, Yuqian Yuan, Juekai Lin, Liang Liang, Zhuoyi Huang, Wenqiao Zhang, Juncheng Li, Siliang Tang, Jun Xiao, Yueting Zhuang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/8/2026, 2:13:21 AM
Summary
The paper introduces RoboSPA, a large-scale robotic manipulation dataset and benchmark designed to evaluate Vision-Language-Action (VLA) models on fine-grained spatial reasoning and long-horizon procedural planning. It addresses gaps in existing benchmarks by providing 527K trajectories across 280 task variants with five difficulty levels, covering 10 capability categories. Experiments show that current VLA models struggle significantly with complex spatial relations and memory-intensive planning, establishing RoboSPA as a challenging diagnostic tool for embodied agents.
Entities (17)
Relation Signals (18)
RoboSPA → contains → 527K trajectories
confidence 95% · We collect 527K trajectories across multiple embodiments and diverse scenes.
RoboSPA → contains → 280 task variants
confidence 95% · Each task is instantiated across five difficulty levels, yielding 280 variants...
RoboSPA → evaluates → VLA Models
confidence 95% · RoboSPA is a large-scale robotic manipulation dataset and benchmark for diagnosing embodied reasoning in VLA models.
RoboSPA → focuseson → Fine-Grained Spatial Reasoning
confidence 95% · RoboSPA focuses on two core dimensions, Fine-Grained Spatial Reasoning and Long-Horizon Procedural Planning...
RoboSPA → focuseson → Long-Horizon Procedural Planning
confidence 95% · RoboSPA focuses on two core dimensions, Fine-Grained Spatial Reasoning and Long-Horizon Procedural Planning...
RoboSPA → collectedon → UR5
confidence 90% · We collect data across five embodiments: ... UR5.
RoboSPA → collectedon → Aloha-AgileX
confidence 90% · We collect data across five embodiments: Aloha-AgileX...
RoboSPA → collectedon → Franka
confidence 90% · We collect data across five embodiments: ... Franka ...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Vision-Language-Action (VLA) models have shown promising progress in language-conditioned robotic manipulation. However, existing datasets and benchmarks mainly evaluate task completion under predefined settings, offering limited insight into model reasoning under increasing spatial and procedural complexity. We introduce \textbf{RoboSPA} (\textbf{Robo}t \textbf{S}patial-\textbf{P}rocedural \textbf{A}ssessment), a large-scale robotic manipulation dataset and benchmark for diagnosing embodied reasoning in VLA models. \texttt{RoboSPA} focuses on two core dimensions, Fine-Grained Spatial Reasoning and Long-Horizon Procedural Planning, covering 10 task categories and 56 base tasks. Each task is instantiated across five difficulty levels, yielding 280 variants with increasing spatial ambiguity and procedural complexity. We collect 527K trajectories across multiple embodiments and diverse scenes. Beyond binary success rate, \texttt{RoboSPA} introduces diagnostic metrics for more detailed evaluation. Experiments on representative VLA models show that current systems still struggle with complex spatial relations, precise low-level execution, and memory-intensive planning. These results establish \texttt{RoboSPA} as a challenging diagnostic benchmark for developing more capable, reliable, and generalizable embodied agents. Our data and code are available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.05324v1
- Canonical: https://arxiv.org/abs/2609.05324v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
105,984 characters extracted from source content.
Expand or collapse full text
RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks? Zhenxuan Fan 1 , Bo Zhang 2 , Yutong Lin 1 , Yuqian Yuan 1 , Juekai Lin 1 , Liang Liang 1 Zhuoyi Huang 3 , Wenqiao Zhang 1 * , Juncheng Li 1 * , Siliang Tang 1 , Jun Xiao 1 , Yueting Zhuang 1 1 Zhejiang University 2 University of Electronic Science and Technology of China 3 South China Normal University zxfan@zju.edu.cn wenqiaozhang@zju.edu.cn Project PageCodeData Abstract Vision-Language-Action(VLA)models have shown promising progress in language- conditioned robotic manipulation. However, existing datasets and benchmarks mainly evaluate task completion under predefined settings, offering limited insight into model reasoning under increasing spatial and pro- cedural complexity. We introduce RoboSPA (RobotSpatial-ProceduralAssessment), a large-scale robotic manipulation dataset and benchmark for diagnosing embodied reasoning in VLA models.RoboSPAfocuses on two core dimensions, Fine-Grained Spatial Reasoning and Long-Horizon Procedural Planning, covering 10 task categories and 56 base tasks.Each task is instantiated across five difficulty levels, yielding 280 variants with increasing spatial ambiguity and procedural complexity. We collect 527K trajectories across multiple embodiments and diverse scenes. Beyond binary success rate, RoboSPAintroduces diagnostic metrics for more detailed evaluation.Experiments on representative VLA models show that current systems still struggle with complex spatial relations, precise low-level execution, and memory-intensive planning.These results establishRoboSPAas a challenging diagnostic benchmark for developing more capable, reliable, and generalizable embodied agents. Our data and code are available athttps: //github.com/fanzhenxuan/RoboSPA. 1 Introduction Vision-Language-Action (VLA) models (Zitkovich et al., 2023; Kim et al., 2025; Black et al., 2025b; NVIDIA et al., 2025; Black et al., 2025a; Gao et al., 2026) have emerged as a promising paradigm for general-purpose embodied agents, integrating vi- sual perception, language understanding, and ac- tion generation in a unified framework (Ma et al., * Corresponding author. 2024; Kawaharazuka et al., 2025). With large-scale pretraining and multimodal alignment, these mod- els show encouraging performance on manipula- tion tasks, especially in structured, short-horizon settings (Shao et al., 2025; Zhong et al., 2025). However, deploying embodied agents in real- world scenarios requires capabilities beyond short- horizon instruction following in simple, well- structured scenes (Liu et al., 2025b; Zhang et al., 2025b; Wong et al., 2025; Yuan et al., 2025a; Dang et al., 2026). Everyday manipulation tasks pose three key challenges: (1) Fine-grained target dis- ambiguation, requiring agents to identify the target among visually similar candidates through subtle spatial cues beyond category or color; (2) Tempo- rally extended task execution, requiring multi- step execution under temporal constraints and ac- cumulated errors; and (3) Complexity-scalable embodied reasoning, where increasing candi- dates, horizons, and environmental diversity am- plify grounding and planning failures. These chal- lenges call for benchmarks that evaluate embodied reasoning under increasing spatial and procedural complexity. As shown in Table 1, existing robotic manipula- tion datasets and benchmarks leave several gaps for evaluating reasoning-oriented VLA models. First, fine-grained spatial reasoning is rarely eval- uated explicitly, as prior works seldom test tar- get disambiguation with subtle spatial cues. Sec- ond, long-horizon and step-level evaluation re- main limited: LIBERO (Liu et al., 2023a) and RoboTwin 2.0 (Chen et al., 2026a) mainly focus on short-horizon tasks, while RoboCasa (Nasiri- any et al., 2024) and MIKASA-Robo (Cherepanov et al., 2026) often lack step-level diagnosis. Third, most prior works lack controlled multi-level diffi- culty (Pumacay et al., 2024; Fei et al., 2026; Zhou et al., 2025), making it hard to analyze perfor- mance degradation as task complexity increases. Finally, dataset scale remains limited for broad 1 arXiv:2609.05324v1 [cs.RO] 4 Sep 2026 NameCategories Tasks Trajectories EmbodimentsSpatialLong-Horizon Step-levelMulti-Scene ReasoningTasksEvaluation Difficulty Diversity BC-Z (Jang et al., 2022)3100+26K1✗ RT-1 (Brohan et al., 2023)8700+130K1✗✓✗✓ BridgeData V2 (Walke et al., 2023)–1360.1K1✗✓ Open X-Embodiment (O’Neill et al., 2024)527160,2661.4M22✗✓ DROID (Khazatsky et al., 2024)–8676K1✗✓ RLBench (James et al., 2020)–100–1✗✓✗ CALVIN (Mees et al., 2022)–34N/A † 1✗✓✗✓ LIBERO (Liu et al., 2023a)41306.5K1✗✓ ManiSkill2 (Gu et al., 2023)420N/A ‡ 1✗✓ RoboCasa (Nasiriany et al., 2024)–100 ∼100K1✗✓✗✓ SimplerEnv (Li et al., 2025)–8–2✗✓ MIKASA-Robo (Cherepanov et al., 2026)123232K1✗✓✗✓✗ RoboTwin 2.0 (Chen et al., 2026a)–50 ≤137.5K5✗✓ LIBERO-Pro (Zhou et al., 2025)440–1✗✓ RMBench (Chen et al., 2026b)294501✗✓✗ RoboSPA (Ours)10280527K5✓ Table 1: Comparison of representative robotic manipulation datasets and benchmarks withRoboSPA. The upper and lower parts summarize datasets and benchmarks, respectively. † CALVIN reports approximately 24 hours of teleoperated demonstrations. ‡ ManiSkill2 reports over 4M demonstration frames. reasoning evaluation. Most widely used datasets contain fewer than 200K trajectories, limiting task, embodiment, and scene coverage. To address these limitations, we introduce Ro- boSPA (Robot Spatial-Procedural Assessment), a large-scale robotic manipulation dataset and bench- mark. To systematically study embodied reason- ing under increasing task complexity,RoboSPAis guided by three core design principles: • Fine-Grained Spatial Reasoning. As shown in Fig. 1, this component evaluates whether VLA models can ground instructions in complex spa- tial structures, such as geometric attributes, dis- tances, cross-view cues, relations, and canonical indexing. This capability is essential for clut- tered manipulation. To our knowledge,RoboSPA is the first VLA dataset to make fine-grained spatial reasoning a core evaluation dimension. • Long-Horizon Procedural Planning. This component evaluates whether VLA models can execute manipulation tasks with multi-step de- cisions. As shown in Fig. 1, it covers repetitive procedures, order-constrained and order-free ex- ecution, composite coordination, and memory- intensive planning. These capabilities are crucial for real-world manipulation requiring temporal consistency and stepwise progress. •Multi-Level Hierarchical Evaluation. This component measures how VLA model perfor- mance changes with increasing task complexity. Each base task is instantiated across five diffi- culty levels, enabling analysis of performance degradation under growing spatial complexity and action horizons. Beyond task-level success, RoboSPAreports step-level progress for detailed diagnosis of model failures. Together, these designs provide a controlled foundation for large-scale VLA data collection and evaluation.RoboSPAprovides demonstrations in clean and domain-randomized scenes, covering 56 base tasks across 10 capability categories and five difficulty levels, yielding 280 variants. Each task supports step-level evaluation beyond final success rates. Across five embodiments,RoboSPAcontains 527K trajectories and 997 hours of videos, forming a comprehensive benchmark for embodied reason- ing under increasing task complexity. We evaluate representative VLA models (Liu et al., 2025a; AgiBot-World-Contributors et al., 2025; Black et al., 2025a; Zheng et al., 2026) on RoboSPAand find that they struggle as task com- plexity increases. On the hardest tasks, all models achieve an average success rate below 25%, with some tasks dropping to 0%. Diagnostic analyses reveal failures in target grounding, low-level manip- ulation, long-horizon tracking, and memory-based reasoning, highlightingRoboSPAas a challenging diagnostic testbed for embodied reasoning. 2 Related Work Vision-Language-Action Models. Large lan- guage models (LLMs) (Grattafiori et al., 2024; Yang et al., 2025; Cao et al., 2026; Fan et al., 2026) and multimodal large language models (MLLMs) (Liu et al., 2023b; Bai et al., 2025; Yuan et al., 2025b; Lin et al., 2025; Wang et al., 2026) 2 Step-LevelEval "Lift the 2nd largest block with one arm." “Pick up the 3rd farthest pill bottle from the brown pill bottle.” “Lift the 4th object in the 1st row counted from near to far, with positions counted from right to left, with one arm.” “Pick the nearest one directly at the back of the middle-left soap bar.” “Stack each blue bowl on the smooth ceramic plate.” “Click the frontmost top bell, then circle clockwise through the other top bell.” “(1) Place the bell over the electronic scale, (2)then place the toy car over the display stand, (3) and tap the card box, (4) then put the bell back on the table, (5) and put the toy car back on the table." "Memorize the front T-shaped blocks, then once they are hidden, make the rear T-shaped blocks match the front ones in orientation from left to right." “Pick up the small handheld fan and put it down 5 times.” ⑩ Memory-Intensive Planning ⑤ Cross-View Reasoning ④ Referential Relational Reasoning ③ Canonical Position Indexing② Spatial Distance Estimation① Geometric Attribute Cognition ⑨ Composite Action Coordination⑧ Order-Constrained Execution ⑦ Order-Free Execution “Viewed from the opposite side of the table, take the rightmost object.” ② ③ Row 1 ① ③ ④ ⑤ ⑥ ③ ① ② ④ ⑤ Left Right 527k Trajectories 5Difficulty Levels56Base Tasks 280 Task Variants997hVideos Col. 4 LeftRight ① ② ③④ ⑤ ⑥ Repetitive Procedure Following Figure 1: Overview ofRoboSPA.RoboSPAcenters on Fine-Grained Spatial Reasoning and Long-Horizon Procedural Planning, covering 10 task categories and 56 base tasks. Each task has 5 difficulty levels, yielding 280 variants. The dataset includes 527K trajectories across multiple embodiments and scene settings, supporting step-level evaluation of VLA limitations in spatial reasoning, long-horizon execution, and scalable embodied reasoning. have demonstrated strong capabilities for under- standing and reasoning over language and mul- timodal inputs, respectively. Building on these advances, VLA models map visual observations and language instructions to robot actions through a unified perception-to-control framework. Early methods, such as RT-1 (Brohan et al., 2023), RT- 2 (Zitkovich et al., 2023), and OpenVLA (Kim et al., 2025), cast action prediction as autoregres- sive discrete token generation. Subsequent VLA models largely adopt diffusion or flow-matching policies for continuous action generation, as in RDT (Liu et al., 2025a) andπ 0 (Black et al., 2025b). To improve cross-task generalization, VLA models such asπ 0.5 (Black et al., 2025a) and GR00T N1 (NVIDIA et al., 2025) adopt hi- erarchical policies to bridge high-level language understanding with low-level motor execution, while other works enhance action and embodi- ment generalization through implicit action mod- eling and cross-embodiment data (AgiBot-World- Contributors et al., 2025; Zheng et al., 2026). From a different perspective, another line of work improves VLA models through spatial guidance, memory mechanisms, and future-frame predic- tion, further enhancing their capability and usabil- ity (Chen et al., 2025; Shi et al., 2026; Torne et al., 2026; Zhang et al., 2025d; Bi et al., 2026). Robotic Manipulation Datasets and Bench- marks. The rapid development of VLA mod- els has also accelerated the development of robotic manipulation datasets and evaluation benchmarks.Large-scale datasets, including BridgeData V2 (Walke et al., 2023), Open X- Embodiment (O’Neill et al., 2024), DROID (Khaz- atsky et al., 2024), and AgiBot World (AgiBot- World-Contributors et al., 2025), expand real- world, cross-embodiment manipulation cover- age. Benchmark suites such as RLBench (James et al., 2020), CALVIN (Mees et al., 2022), Man- iSkill2 (Gu et al., 2023), and LIBERO (Liu et al., 2023a) provide structured settings for evaluating diverse skills, language-conditioned control, and sequential manipulation. Recent benchmarks fur- ther examine scene and capability generalization, sim-to-real transfer, bimanual manipulation, and memory-dependent tasks (Pumacay et al., 2024; Nasiriany et al., 2024; Li et al., 2025; Sedlacek et al., 2026; Garcia et al., 2025; Zhang et al., 2025c; Zhou et al., 2025; Zhang et al., 2025a; Chen et al., 2026a; Cherepanov et al., 2026; Chen et al., 2026b). However, they rarely systematically evalu- ate fine-grained spatial reasoning and long-horizon procedural planning with step-level diagnostics. RoboSPA addresses this gap with tasks specifically designed around these two capabilities. 3 Canonical Position Indexing Repetitive Procedure Following F i n e - G r a i n e d S p a t i a l R e a s o n i n g L o n g - H o r i z o n P r o c e d u r a l P l a n n i n g RoboSPA (a) Capability taxonomy More Objects Longer Action Sequence ①→②→③→④→⑤①→②→③→④①→②→③①→②① (b) Multi-difficulty task design (c) Domain-randomized scenes Figure 2: Design ofRoboSPA. (a) Capability taxonomy with two dimensions and 10 categories. (b) Multi-difficulty task design through increasing object counts or action-sequence length. (c) Domain-randomized scenes with diverse layouts, distractors, lighting, textures, and tabletops. 3 RoboSPA 3.1 Overview As shown in Fig. 1, we introduceRoboSPA, a large- scale robotic manipulation dataset and benchmark for evaluating reasoning-oriented VLA models. RoboSPAis built using the SAPIEN (Xiang et al., 2020) simulator and RoboTwin 2.0 (Chen et al., 2026a) framework. Each task is instantiated across multiple difficulty levels and scene settings. 3.2 Benchmark Construction 3.2.1 Capability Taxonomy To systematically evaluate embodied reasoning in VLA models, we build a hierarchical capability taxonomy instead of treating manipulation tasks as isolated instances. As shown in Fig. 2(a), it in- cludes two core dimensions: Fine-Grained Spatial Reasoning and Long-Horizon Procedural Planning. Fine-Grained Spatial Reasoning. This dimen- sion assesses instruction grounding in complex spa- tial configurations. Tasks require selecting the cor- rect target through fine-grained spatial reasoning in complex scenes. It includes five categories: •Geometric Attribute Cognition (GAC): Eval- uates whether the model can identify and ma- nipulate objects by geometric or shape-related attributes beyond category-level recognition. •Spatial Distance Estimation (SDE): Measures whether the model can compare distances among objects or to a reference entity, reasoning about proximity, remoteness, and relative distance. •Canonical Position Indexing (CPI): Evaluates whether the model can identify objects by row- column indices under different counting direc- tions and spatial scanning orders. •Referential Relational Reasoning (R): As- sesses whether the model can locate targets through directional relations to reference objects, covering basic and compositional positions. •Cross-View Reasoning (CVR): Examines whether the model can interpret spatial instruc- tions from non-egocentric viewpoints by trans- forming spatial references across perspectives. Long-Horizon Procedural Planning. This di- mension evaluates a model’s ability to complete extended manipulation tasks with diverse actions, multiple subgoals, states, and procedural con- straints. It also consists of five categories: •Repetitive Procedure Following (RPF): As- sesses whether the model can repeat a specified operation the required number of times while tracking progress and stopping correctly. • Order-Free Execution (OFE): Measures whether the model can complete multiple subgoals in flexible order, covering all targets without omission or unnecessary repetition. 4 Robotic 0 200 400 600 800 L1L2L3L4L5 Average Number Difficulty Level Timesteps vs. Difficulty 0 2 4 6 8 10 L1L2L3L4 Average Number Object Count vs. Difficulty 64K 0 2 4 6 8 10 L1L2L3L4L5 Average Number Difficulty Level Object Count vs. Difficulty Aloha-AgileX 114k Franka 130k ARX-X5 133k UR5 85k Piper 65k Clean 64K Randomized 463K (a) Embodiment Distribution 0 200 400 600 800 L1L2L3L4L5 Average Number Difficulty Level Timesteps vs. Difficulty 64K 0 2 4 6 8 10 L1L2L3L4L5 Average Number Difficulty Level Object Count vs. Difficulty Aloha-AgileX 113k Franka 130k ARX-X5 133k UR5 73k Piper 65k Clean 64K Randomized 463K (b) Scene Distribution 0 200 400 600 800 L1L2L3L4L5 Average Number Difficulty Level Timesteps vs. Difficulty 64K 0 2 4 6 8 10 L1L2L3L4L5 Average Number Difficulty Level Object Count vs. Difficulty Aloha-AgileX 113k Franka 130k ARX-X5 133k UR5 73k Piper 65k Clean 64K Randomized 463K (c) Average Object Count 0 200 400 600 800 L1L2L3L4L5 Average Number Difficulty Level Timesteps vs. Difficulty 64K 0 2 4 6 8 10 L1L2L3L4L5 Average Number Difficulty Level Object Count vs. Difficulty Aloha-AgileX 113k Franka 130k ARX-X5 133k UR5 73k Piper 65k Clean 64K Randomized 463K (d) Average Timesteps Figure 3: Dataset statistics ofRoboSPA. (a) Trajectory distribution across five embodiments. (b) Trajectories across clean and domain-randomized scenes. (c) Average object count by difficulty for spatial reasoning tasks. (d) Average trajectory length by difficulty for long-horizon planning tasks. •Order-Constrained Execution (OCE): Evalu- ates whether the model can execute subgoals in a specified order, as correct actions in the wrong order can still fail. •Composite Action Coordination (CAC): Ex- amines whether the model can coordinate hetero- geneous manipulation skills across multi-stage procedures with different action types. •Memory-Intensive Planning (MIP): Tests whether the model can retain information and use it to guide later actions when cues must be recalled or are no longer available. 3.2.2 Task Suite Construction Task Construction. Based on the proposed tax- onomy, we design each task to instantiate a spe- cific reasoning requirement. Tasks are defined by expert code specifying the scene, execution pro- cedure, and success condition. We reuse part of RoboTwin 2.0 (Chen et al., 2026a) action prim- itives. Ten trained graduate students design the tasks, with each student responsible for one ca- pability category. Each task is reviewed by two additional students for quality assurance. Hierarchical Difficulty Design. To measure model performance under increasing reasoning bur- den, we instantiate each task across five difficulty levels. As shown in Fig. 2(b), difficulty is task- specific: spatial tasks increase candidate objects, while procedural tasks extend action sequences with more intermediate steps. Instruction Design. For each task, we use GPT- 5.2 (OpenAI, 2025) to generate 60 non-overlapping instruction templates with diverse linguistic forms, using 50 for training and 10 for testing. These templates are instantiated with task-specific object descriptions and manually checked to ensure diver- sity under controlled semantics. 3.2.3 Data Collection After task design, we execute expert code in simula- tion to collect trajectories in both clean and domain- randomized scenes. Clean scenes contain only task- relevant objects, while domain-randomized scenes vary clutter, textures, lighting, and tabletop config- urations, as shown in Fig. 2(c), enabling evalua- tion of both task competence and scene robustness. We collect data across five embodiments: Aloha- AgileX, ARX-X5, Piper, Franka, and UR5. 3.2.4 Evaluation Metrics To enable more diagnostic evaluation beyond Success Rate (SR), we further introduce two dimension-specific complementary metrics. For fine-grained spatial reasoning tasks, we introduce Object-Normalized Target Accuracy (ONTA), inspired by Cohen’s kappa (Cohen, 1960), to fairly compare spatial reasoning ability under varying scene complexity.ONTAremoves the object-count-induced chance baseline: ONTA(n) = SR(n)− 1/n 1− 1/n × 100.(1) wherenis the number of objects.ONTAis 100 for perfect target selection, 0 for random guessing, and negative for below-chance performance. For long-horizon procedural planning tasks, we report Progress Score (PS) to capture partial com- pletion. It is computed on a 0–100 scale: PS = 1 N N X i=1 c i T ! × 100.(2) whereTis the total number of subtasks, andc i is the number completed in episode i. 5 Table 2: Performance comparison of four baseline VLA models onRoboSPA. We use Success Rate (SR) as the evaluation metric. Drop denotes the L1-to-L5 decrease.Blue shading marks the best model for each metric. Task RDTGO-1π 0.5 X-VLA L1L5DropL1L5DropL1L5DropL1L5Drop Fine-Grained Spatial Reasoning Geometric Attribute Cognition10.04.85.227.811.516.342.812.530.357.825.332.5 Spatial Distance Estimation15.29.45.820.211.68.635.023.411.641.434.66.8 Canonical Position Indexing15.08.66.422.07.214.8 44.023.620.431.88.823.0 Referential Relational Reasoning4.44.40.017.26.610.633.29.823.440.414.228.2 Cross-View Reasoning3.03.4-0.417.416.21.241.237.63.638.236.61.6 Average9.56.13.420.910.610.339.221.417.841.923.918.0 Long-Horizon Procedural Planning Repetitive Procedure Following40.835.05.831.823.88.080.365.015.334.822.512.3 Order-Free Execution20.30.819.541.15.635.585.826.159.7 86.043.642.4 Order-Constrained Execution13.60.113.541.93.138.877.412.065.470.07.762.3 Composite Action Coordination27.52.425.124.22.621.658.612.546.158.95.953.0 Memory-Intensive Planning18.20.018.27.60.07.654.00.054.044.60.044.6 Average24.17.716.429.37.022.371.223.148.158.815.942.9 Overall Benchmark Overall Average16.86.99.925.18.816.355.222.332.950.419.930.5 0 20 40 60 80 100 L1L2L3L4L5 Order-Constrained Execution RDTGO-1 π0.5 X-VLA 0 20 40 60 80 100 L1L2L3L4L5 Repetitive Procedure Following 0 20 40 60 80 100 L1L2L3L4L5 Order-Free Execution 0 20 40 60 80 100 L1L2L3L4L5 Order-Constrained Execution 0 20 40 60 80 100 L1L2L3L4L5 Composite Action Coordination 0 20 40 60 80 100 L1L2L3L4L5 Memory Intensive Planning 0 20 40 60 80 100 L1L2L3L4L5 Geometric Attribute Cognition 0 20 40 60 80 100 L1L2L3L4L5 Spatial Distance Estimation 0 20 40 60 80 100 L1L2L3L4L5 Cross-View Reasoning 0 20 40 60 80 100 L1L2L3L4L5 Canonical Position Indexing 0 20 40 60 80 100 L1L2L3L4L5 Referential Relational Reasoning Figure 4: Success rate trends of four baseline VLA models across 10 task categories over five difficulty levels. 3.3 Benchmark Statistics RoboSPAcontains 56 base tasks across two capabil- ity dimensions. Each base task is instantiated at 5 difficulty levels, yielding 280 task variants. In total, we collect approximately 527K trajectories, corre- sponding to over 997 hours of interaction videos and 108M timesteps. More dataset statistics are provided in the Appendix C. Fig. 3(a) presents the trajectory distribution across the 5 robotic embodi- ments, while Fig. 3(b) shows the trajectory distribu- tion across clean and domain-randomized scenes. Fig. 3(c) and Fig. 3(d) further illustrate the hierar- chical difficulty design along the two dimensions, respectively. 4 Experiments 4.1 Experimental Setup We evaluate four representative VLA models, RDT (Liu et al., 2025a), GO-1 (AgiBot-World- Contributors et al., 2025),π 0.5 (Black et al., 2025a), and X-VLA (Zheng et al., 2026), using the clean- scene data collected with the Aloha-AgileX embod- iment for both training and evaluation. All models are trained and evaluated under a single-task setting. For each base task, we train a separate model us- ing data from all five difficulty levels and evaluate it independently at each level. During evaluation, each model is tested with 100 rollout trials per task variant. 6 Table 3: Comparison of Object-Normalized Target Accuracy (ONTA) on fine-grained spatial reasoning tasks. Task RDTGO-1π 0.5 X-VLA L1L3L5L1L3L5L1L3L5L1L3L5 Geometric Attribute Cognition-80.0-25.0-14.3-44.5-7.0-6.2-14.51.3-5.015.511.010.3 Spatial Distance Estimation-27.2-9.1-4.6-19.7-6.7-2.12.513.611.612.118.224.4 Canonical Position Indexing-27.5-4.90.3-17.0-3.3-1.2 16.010.216.7-2.32.20.5 Referential Relational Reasoning-91.2-25.1-7.6-65.6-18.9-5.1-33.60.5-1.5-19.20.33.5 Cross-View Reasoning-45.5-28.8-28.8-23.9-14.7-11.711.816.016.87.317.315.5 Average-54.3-18.6-11.0-34.1-10.1-5.3-3.68.37.72.79.810.8 Table 4: Performance comparison in terms of Progress Score (PS ) on long-horizon procedural planning tasks. Task RDTGO-1π 0.5 X-VLA L1L3L5L1L3L5L1L3L5L1L3L5 Repetitive Procedure Following40.846.344.531.835.032.080.378.777.334.834.831.9 Order-Free Execution20.333.024.941.139.725.785.875.062.186.084.473.9 Order-Constrained Execution13.625.314.741.924.114.877.444.328.070.051.125.1 Composite Action Coordination27.532.526.524.226.222.758.660.548.058.949.040.1 Memory-Intensive Planning18.25.03.67.64.96.054.018.711.444.612.27.3 Average24.128.422.829.326.020.271.255.445.458.846.335.7 4.2 Main Results Table 2 reports the Success Rates (SR) of four representative VLA models. Overall, model perfor- mance declines from L1 to L5, indicating limited ability to reason under complex task settings. Model Comparison.Among the evaluated mod- els,π 0.5 performs best overall, but reaches only 22.3% success at L5, followed by X-VLA with 19.9%. Both drop by over 30 percentage points as difficulty increases. GO-1 and RDT perform worse, achieving only 8.8% and 6.9% success at L5, re- spectively. These results show that stronger models handle easier cases better, but all models still lack robust reasoning under complex conditions. Fine-GrainedSpatialReasoning. X-VLA shows the best performance in this dimension, but its success rate still drops from 41.9% at L1 to only 23.9% at L5. This decline shows that increasing spatial complexity substantially weakens target grounding.Models struggle particularly with Canonical Position Indexing (CPI) and Referential Relational Reasoning (R), indicating that current VLA models remain unreliable when target selection depends on fine-grained spatial relations rather than simple object recognition. Long-Horizon Procedural Planning.For Long- Horizon Procedural Planning, the performance drop is even more pronounced, with the best- performing model,π 0.5 , decreasing from 71.2% at L1 to 23.1% at L5. Models perform best on Repetitive Procedure Following (RPF), while the other four categories show substantially lower per- formance. Notably, all models fail on the hardest Memory-Intensive Planning (MIP), revealing se- vere limitations in maintaining task-relevant mem- ory over extended horizons. Fig. 4 further shows that performance drops from L1 to L5, exposing failures in spatial discrimination and long-horizon tracking. 4.3 Diagnostic Evaluation Object-Normalized Target Accuracy. Table 3 compares models using theONTAmetric. Over- all, all models achieve low scores. RDT and GO-1 obtain negativeONTAon most categories and difficulty levels, showing little spatial reasoning beyond random selection.π 0.5 and X-VLA show fewer negative scores and achieve better average ONTA, but their overall scores remain close to the random-selection baseline. These results sug- gest that current VLA models still struggle to iden- tify the correct target when success depends on fine-grained spatial cues. Progress Score. Table 4 reports Progress Score for long-horizon procedural planning tasks, provid- ing a progress-based view beyond binary Success Rate. The results show that Progress Score is gener- ally higher than final Success Rate, indicating that 7 Figure 5: Category-wise performance of four baseline VLA models onRoboSPA. Left: Success Rate across 10 capability categories at L1. Right: Success Rate across 10 capability categories at L5. Viewedfromtheleftsideofthe table,pickupthebackmostobject. Pick the 6th highest block. Click the bell, open the microwave,click the bell, placethe playing cards left of the blue soap,click the bell. Based on the colors of the blocks on the display platform, when they are hidden, move matching blocks closer to you. Click the alarm clock, thecan,the stapler, the playing cards, the bread sequentially First stamp the left seal twice on the left pad, then stamp the right seal twice on the right pad, then press the stapler with the red top once. ⑥Redundant Repetition Error⑤Temporal Ordering Error ④Memory Error③Manipulation Error ① ③ ⑤ ④ ② ① ② ①②③ ①Target Execution Error ②Target Grounding Error Figure 6: Representative failure modes observed onRoboSPA. Failure modes vary across differentRoboSPAtask categories, showing that our benchmark can distinguish model weaknesses along different capability dimensions. models can often execute part of a procedure but fail due to accumulated errors, incorrect ordering, or incomplete later-stage execution.π 0.5 achieves the strongest average completion rate, followed by X-VLA. Nevertheless, Memory-Intensive Plan- ning (MIP) remains weak at higher difficulty levels, indicating that current models still have limited memory ability during long-horizon execution. 4.4 Qualitative Results Fig. 5 visualizes category-wise model performance across task difficulty. The radar plots show that model performance is highly uneven across cate- gories and shrinks markedly under harder settings. This reveals persistent limitations in handling in- creasingly complex embodied reasoning tasks. 4.5 Failure Analysis We further analyze model failures. Fig. 6 presents six representative failure modes, with quantitative details provided in the Appendix A. Observed Failure Modes. For fine-grained spa- tial reasoning, target execution error denotes failed manipulation after correct grounding, while target grounding error denotes selecting a distractor. For long-horizon procedural planning, manipulation er- ror refers to failed intermediate execution, memory error to forgetting earlier task-relevant information, temporal ordering error to executing subtasks in the wrong order, and redundant repetition error to repeating completed actions instead of progressing. Implications for Future Models.These findings suggest several directions for future VLA models. Improving spatial performance requires stronger object-centric grounding and relation awareness, together with reliable low-level execution. Long- horizon tasks require stronger progress tracking, memory, and execution monitoring to maintain task state. These abilities help models follow temporal 8 constraints and reduce ordering errors, redundant repetitions, and cascading failures. 5 Conclusion We presentRoboSPA, a reasoning-focused, large- scale robotic manipulation dataset and benchmark. Centered on Fine-Grained Spatial Reasoning and Long-Horizon Procedural Planning,RoboSPAcov- ers ten task categories, five difficulty levels, mul- tiple embodiments, and diverse scenes, with fine- grained metrics for spatial grounding and subtask completion. Experiments show that existing VLA models still struggle with complex spatial relations, precise execution, and memory-intensive planning, especially at higher difficulty levels. We hope RoboSPAcan advance the development of more capable and reliable embodied agents. Limitations Although our dataset covers diverse fine-grained spatial reasoning and long-horizon procedural plan- ning tasks, several limitations remain. First, all tasks are constructed in simulation, and the sim-to- real gap may limit direct transfer to physical robots. Second, our benchmark focuses on tabletop manip- ulation, which may not fully capture the diversity and open-endedness of real-world environments. Third, while we include multiple robotic embod- iments and domain-randomized scenes, broader settings such as deformable object manipulation, human-robot interaction, and open-ended task in- structions remain underexplored. Ethics Statement Scope and Safety.RoboSPAis designed to diag- nose and evaluate VLA models in simulated robotic manipulation environments, rather than to serve as a directly deployable real-world robotic system. Data collection is conducted entirely in simula- tion and does not involve real-world robot deploy- ment. Since models evaluated or developed with RoboSPAmay eventually be transferred to physical environments, such deployment should include ap- propriate safety constraints, human oversight, and task-specific validation. Data Content and Privacy.RoboSPAdoes not in- volve human-subject experiments, personal data, or real-world private data, and thus poses no privacy risks to human participants. We checked the task instructions, object categories, scene assets, and collected demonstrations to ensure that the dataset does not contain personally identifying information or offensive, hateful, or explicit content. Therefore, no additional anonymization is required. Artifact Licensing.RoboSPAbuilds upon third- party open-source resources. We use the SAPIEN simulator, which is released under the Apache License 2.0, and reuse code and 3D assets from RoboTwin 2.0 and RMBench, which are released under the MIT License. Baseline models and pre- trained checkpoints used in our experiments are obtained from their official releases and used in accordance with their original license terms. These third-party artifacts are used consistently with their intended research purposes for simulation- based robotic manipulation, benchmark construc- tion, data generation, and evaluation. We retain the original copyright and license notices of all third-party resources. The artifacts introduced by RoboSPA, including task definitions, benchmark code, evaluation scripts, and collected demonstra- tion data, will be released under the MIT License. Responsible Use. We acknowledge that simula- tor design, task selection, object categories, and em- bodiment choices may introduce biases and limit transferability, which we mitigate through mul- tiple embodiments, diverse task categories, con- trolled difficulty levels, and both clean and domain- randomized scenes. We also encourage respon- sible use ofRoboSPA, including efficient training, transparent reporting of computational costs, and consideration of environmental impact. Acknowledgments This work was supported by the National Key R&D Program of China (2025ZD0123100), the NSFC (62272411), the Zhejiang NSF (LRG25F020001), and the Key R&D Program of Zhejiang Province (2025C01030). References AgiBot-World-Contributors, Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xuan Hu, Xu Huang, Shu Jiang, Yuxin Jiang, Cheng Jing, Hongyang Li, Jialu Li, Chiming Liu, Yi Liu, Yuxiang Lu, and 33 others. 2025. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. Preprint, arXiv:2503.06669. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei 9 Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhi- fang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025. Qwen3-vl technical report. Preprint, arXiv:2511.21631. Hongzhe Bi, Hengkai Tan, Shenghao Xie, Zeyuan Wang, Shuhe Huang, Haitian Liu, Ruowen Zhao, Yao Feng, Chendong Xiang, Yinze Rong, Hongyan Zhao, Hanyu Liu, Zhizhong Su, Lei Ma, Hang Su, and Jun Zhu. 2026. Motus: A unified latent action world model. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pages 35101–35113. Kevin Black,Noah Brown,James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Robert Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, brian ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, and 16 others. 2025a.π 0.5 : a vision-language-action model with open-world gen- eralization. In Proceedings of The 9th Conference on Robot Learning, volume 305 of Proceedings of Machine Learning Research, pages 17–40. PMLR. Kevin Black, Noah Brown, Danny Driess, Adnan Es- mail, Michael Robert Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, and 6 oth- ers. 2025b.π 0 : A Vision-Language-Action Flow Model for General Robot Control. In Proceedings of Robotics: Science and Systems, LosAngeles, CA, USA. Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, and 32 others. 2023. RT-1: Robotics Transformer for Real- World Control at Scale. In Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea. Jie Cao, Tianwei Lin, Zhenxuan Fan, Bo Yuan, Ziyuan Zhao, Rolan Yan, Wenqiao Zhang, and Siliang Tang. 2026. Draft-thinking: Learning efficient rea- soning in long chain-of-thought llms.Preprint, arXiv:2603.00578. Tianxing Chen, Zanxin Chen, Baijun Chen, Zijian Cai, Yibin Liu, Zixuan Li, Qiwei Liang, Xianliang Lin, Yiheng Ge, Zhenyu Gu, Weiliang Deng, Yubin Guo, Tian Nian, Xuanbing Xie, Qiangyu Chen, Kailun Su, Tianling Xu, Guodong Liu, Mengkang Hu, and 7 oth- ers. 2026a. Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. In Forty- third International Conference on Machine Learning. Tianxing Chen, Yuran Wang, Mingleyang Li, Yan Qin, Hao Shi, Zixuan Li, Yifan Hu, Yingsheng Zhang, Kaixuan Wang, Yue Chen, Hongcheng Wang, Junjie Wang, Tianhang Yang, Renjing Xu, Ruihai Wu, Yao Mu, Yaodong Yang, Hao Dong, and Ping Luo. 2026b. Rmbench: Memory-dependent robotic manipulation benchmark with insights into policy design. Preprint, arXiv:2603.01229. Xinyi Chen, Yilun Chen, Yanwei Fu, Ning Gao, Ji- aya Jia, Weiyang Jin, Hao Li, Yao Mu, Jiang- miao Pang, Yu Qiao, Yang Tian, Bin Wang, Bolun Wang, Fangjing Wang, Hanqing Wang, Tai Wang, Ziqin Wang, Xueyuan Wei, Chao Wu, and 10 oth- ers. 2025. Internvla-m1: A spatially guided vision- language-action framework for generalist robot pol- icy. Preprint, arXiv:2510.13778. Egor Cherepanov, Nikita Kachaev, Alexey Kovalev, and Aleksandr Panov. 2026. Memory, benchmark & robots: A benchmark for solving complex tasks with reinforcement learning. In The Fourteenth Inter- national Conference on Learning Representations. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Mea- surement, 20(1):37–46. Ronghao Dang, Jiayan Guo, Bohan Hou, Sicong Leng, Kehan Li, Xin Li, Jiangpin Liu, Yunxuan Mao, Zhikai Wang, Yuqian Yuan, Minghao Zhu, Xiao Lin, Yang Bai, Qian Jiang, Yaxi Zhao, Minghua Zeng, Junlong Gao, Yuming Jiang, Jun Cen, and 7 others. 2026. Rynnbrain: Open embodied foundation mod- els. Preprint, arXiv:2602.14979. Zhenxuan Fan, Jie Cao, Yang Dai, Zheqi Lv, Wenqiao Zhang, Zhongle Xie, Peng LU, and Beng Chin Ooi. 2026. Ctrlcot: Dual-granularity chain-of-thought compression for controllable reasoning. Preprint, arXiv:2601.20467. Senyu Fei, Siyin Wang, Junhao Shi, Zihao Dai, Jikun Cai, Pengfang Qian, Li Ji, Xinzhe He, Shiduo Zhang, Zhaoye Fei, Jinlan Fu, Jingjing Gong, and Xipeng Qiu. 2026. Libero-plus: A progressive robustness benchmark for visual-language-action models. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 38574–38583. Mingjian Gao, Wenqiao Zhang, Yuqian Yuan, Yang Dai, Binhe Yu, Zheqi Lv, Haoyu Zheng, Jiaqi Zhu, Zhiqi Ge, Zixuan Wan, Siliang Tang, and Yueting Zhuang. 2026. Visualthink-vla: Visual intermediate reasoning for effective and low-latency vision-language-action policies. Preprint, arXiv:2605.30011. Ricardo Garcia, Shizhe Chen, and Cordelia Schmid. 2025. Towards generalizable vision-language robotic manipulation: A benchmark and llm-guided 3d pol- icy. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 8996–9002. IEEE. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schel- ten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh 10 Goyal, Anthony Hartshorn, Aobo Yang, Archi Mi- tra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. 2023. Maniskill2: A unified benchmark for generalizable manipulation skills. In The Eleventh International Conference on Learning Representations. Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. 2020.Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019–3026. Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. 2022. Bc-z: Zero-shot task generaliza- tion with robotic imitation learning. In conference on Robot Learning, pages 991–1002. PMLR. Kento Kawaharazuka, Jihoon Oh, Jun Yamada, Ingmar Posner, and Yuke Zhu. 2025. Vision-language-action models for robotics: A review towards real-world applications. IEEE Access, 13:162467–162504. Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ash- win Balakrishna, Sudeep Dasari, Siddharth Karam- cheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, and 79 others. 2024. DROID: A large-scale in-the-wild robot manipula- tion dataset. In Proceedings of Robotics: Science and Systems, Delft, Netherlands. Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. 2025. Openvla: An open-source vision-language-action model. In Proceedings of The 8th Conference on Robot Learning, volume 270 of Proceedings of Machine Learning Research, pages 2679–2713. PMLR. Xuanlin Li, Kyle Hsu, Jiayuan Gu, Oier Mees, Karl Pertsch, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong, and Ted Xiao. 2025. Evaluating real-world robot manipu- lation policies in simulation. In Proceedings of The 8th Conference on Robot Learning, volume 270 of Proceedings of Machine Learning Research, pages 3705–3728. PMLR. Tianwei Lin, Wenqiao Zhang, Sijing Li, Yuqian Yuan, Binhe Yu, Haoyuan Li, Wanggui He, Hao Jiang, Mengze Li, Xiaohui Song, Siliang Tang, Jun Xiao, Hui Lin, Yueting Zhuang, and Beng Chin Ooi. 2025. Healthgpt: A medical large vision-language model for unifying comprehension and generation via heterogeneous knowledge adaptation. In Forty- second International Conference on Machine Learn- ing, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, volume 267 of Proceedings of Machine Learn- ing Research. PMLR / OpenReview.net. Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. 2023a. Libero: Benchmarking knowledge transfer for lifelong robot learning. In Advances in Neural Information Pro- cessing Systems, Datasets and Benchmarks Track. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. Visual instruction tuning. Advances in neural information processing systems, 36:34892– 34916. Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. 2025a. RDT-1b: a diffusion foundation model for bimanual manipulation. In The Thirteenth International Conference on Learning Representa- tions. Yang Liu, Weixing Chen, Yongjie Bai, Xiaodan Liang, Guanbin Li, Wen Gao, and Liang Lin. 2025b. Align- ing cyber space with physical world: A comprehen- sive survey on embodied ai. IEEE/ASME Transac- tions on Mechatronics, 30(6):7253–7274. Yueen Ma, Zixing Song, Yuzheng Zhuang, Jianye Hao, and Irwin King. 2024. A survey on vision-language- action models for embodied ai.arXiv preprint arXiv:2405.14093. Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. 2022.Calvin: A benchmark for language-conditioned policy learning for long- horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 7(3):7327–7334. Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Man- dlekar, and Yuke Zhu. 2024. RoboCasa: Large-Scale Simulation of Household Tasks for Generalist Robots. In Proceedings of Robotics: Science and Systems, Delft, Netherlands. NVIDIA, :, Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi "Jim" Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, Joel Jang, Zhenyu Jiang, Jan Kautz, Kaushil Kundalia, Lawrence Lao, Zhiqi Li, Zongyu Lin, and 24 others. 2025. Gr00t n1: An open foundation model for generalist humanoid robots. Preprint, arXiv:2503.14734. OpenAI. 2025. Introducing gpt-5.2.https://openai. com/index/introducing-gpt-5-2/. Abby O’Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, 11 Ajinkya Jain, Albert Tung, Alex Bewley, Alex Her- zog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew Wang, Anikait Singh, and 260 others. 2024. Open x-embodiment: Robotic learn- ing datasets and rt-x models : Open x-embodiment collaboration0. In 2024 IEEE International Con- ference on Robotics and Automation (ICRA), pages 6892–6903. Wilbert Pumacay, Ishika Singh, Jiafei Duan, Ranjay Kr- ishna, Jesse Thomason, and Dieter Fox. 2024. The Colosseum: A Benchmark for Evaluating General- ization for Robotic Manipulation. In Proceedings of Robotics: Science and Systems, Delft, Netherlands. Martin Sedlacek, Pavlo Yefanov, Georgy Ponimatkin, Jai Bardhan, Simon Pilc, Mederic Fourmy, Evange- los Kazakos, Cees G. M. Snoek, Josef Sivic, and Vladimir Petrik. 2026. Realm: A real-to-sim vali- dated benchmark for generalization in robotic ma- nipulation. IEEE Robotics and Automation Letters, 11(7):8315–8322. Rui Shao, Wei Li, Lingsen Zhang, Renshan Zhang, Zhiyang Liu, Ran Chen, and Liqiang Nie. 2025. Large vlm-based vision-language-action models for robotic manipulation: A survey. arXiv preprint arXiv:2508.13073. Hao Shi, Bin Xie, Yingfei Liu, Lin Sun, Fengrong Liu, Tiancai Wang, Erjin Zhou, Haoqiang Fan, Xi- angyu Zhang, and Gao Huang. 2026. MemoryVLA: Perceptual-cognitive memory in vision-language- action models for robotic manipulation. In The Four- teenth International Conference on Learning Repre- sentations. Marcel Torne, Karl Pertsch, Homer Walke, Kyle Ved- der, Suraj Nair, Brian Ichter, Allen Z. Ren, Haohuan Wang, Jiaming Tang, Kyle Stachowicz, Karan Dha- balia, Michael Equi, Quan Vuong, Jost Tobias Sprin- genberg, Sergey Levine, Chelsea Finn, and Danny Driess. 2026. Mem: Multi-scale embodied mem- ory for vision language action models. Preprint, arXiv:2603.03596. Homer Rich Walke, Kevin Black, Tony Z. Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen-Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, Abraham Lee, Kuan Fang, Chelsea Finn, and Sergey Levine. 2023. Bridgedata v2: A dataset for robot learning at scale. In Proceedings of The 7th Conference on Robot Learning, volume 229 of Pro- ceedings of Machine Learning Research, pages 1723– 1736. PMLR. Zhuonan Wang, Zhenxuan Fan, Siwen Tan, Yu Zhong, Yuqian Yuan, Haoyuan Li, Hao Jiang, Wenqiao Zhang, Feifei Shao, Hongwei Wang, and Jun Xiao. 2026. MAU-GPT: enhancing multi-type industrial anomaly understanding via anomaly-aware and gen- eralist experts adaptation. In Fortieth AAAI Confer- ence on Artificial Intelligence, Thirty-Eighth Confer- ence on Innovative Applications of Artificial Intel- ligence, Sixteenth Symposium on Educational Ad- vances in Artificial Intelligence, AAAI 2026, Sin- gapore, January 20-27, 2026, pages 26787–26795. AAAI Press. Lik Hang Kenny Wong, Xueyang Kang, Kaixin Bai, and Jianwei Zhang. 2025. A survey of robotic navigation and manipulation with physics simulators in the era of embodied ai. arXiv preprint arXiv:2505.01458. Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su. 2020. Sapien: A simulated part-based interactive environment. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Day- iheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. Yuqian Yuan, Ronghao Dang, long li, Wentong Li, Dian Jiao, Xin Li, Deli Zhao, Fan Wang, Wenqiao Zhang, Jun Xiao, and Yueting Zhuang. 2025a. Eoc-bench: Can mllms identify, recall, and forecast objects in an egocentric world? In Advances in Neural Informa- tion Processing Systems, volume 38, Main Confer- ence. Curran Associates, Inc. Yuqian Yuan, Hang Zhang, Wentong Li, Zesen Cheng, Boqiang Zhang, Long Li, Xin Li, Deli Zhao, Wen- qiao Zhang, Yueting Zhuang, Jianke Zhu, and Lidong Bing. 2025b. Videorefer suite: Advancing spatial- temporal object understanding with video llm. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18970–18980. Borong Zhang, Jiahao Li, Jiachen Shen, Yishuai Cai, Yuhao Zhang, Yuanpei Chen, Juntao Dai, Jiaming Ji, and Yaodong Yang. 2025a. VLA-arena: An open- source framework for benchmarking vision-language- action models. arXiv preprint arXiv:2512.22539. Kun Zhang, Peng Yun, Jun Cen, Junhao Cai, Didi Zhu, Hangjie Yuan, Chao Zhao, Tao Feng, Michael Yu Wang, Qifeng Chen, Jia Pan, Wei Zhang, Bo Yang, and Hua Chen. 2025b. Generative artificial intelli- gence in robotic manipulation: A survey. Preprint, arXiv:2503.03464. Shiduo Zhang, Zhe Xu, Peiju Liu, Xiaopeng Yu, Yuan Li, Qinghui Gao, Zhaoye Fei, Zhangyue Yin, Zux- uan Wu, Yu-Gang Jiang, and Xipeng Qiu. 2025c. Vlabench: A large-scale benchmark for language- conditioned robotics manipulation with long-horizon reasoning tasks. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision (ICCV), pages 11142–11152. Wenyao Zhang, Hongsi Liu, Zekun Qi, Yunnan Wang, XinQiang Yu, Jiazhao Zhang, Runpei Dong, Jiawei He, He Wang, Zhizheng Zhang, Li Yi, Wenjun Zeng, 12 and Xin Jin. 2025d. Dreamvla: A vision-language- action model dreamed with comprehensive world knowledge. In Advances in Neural Information Pro- cessing Systems. Jinliang Zheng, Jianxiong Li, Zhihao Wang, Dongxiu Liu, Xirui Kang, Yuchun Feng, Yinan Zheng, Ji- ayin Zou, Yilun Chen, Jia Zeng, Tai Wang, Ya-Qin Zhang, Jingjing Liu, and Xianyuan Zhan. 2026. X- VLA: Soft-prompted transformer as scalable cross- embodiment vision-language-action model. In The Fourteenth International Conference on Learning Representations. Yifan Zhong, Fengshuo Bai, Shaofei Cai, Xuchuan Huang, Zhang Chen, Xiaowei Zhang, Yuanfei Wang, Shaoyang Guo, Tianrui Guan, Ka Nam Lui, Zhiquan Qi, Yitao Liang, Yuanpei Chen, and Yaodong Yang. 2025. A survey on vision-language-action mod- els: An action tokenization perspective. Preprint, arXiv:2507.01925. Xueyang Zhou, Yangming Xu, Guiyao Tie, Yongchao Chen, Guowen Zhang, Duanfeng Chu, Pan Zhou, and Lichao Sun. 2025. Libero-pro: Towards robust and fair evaluation of vision-language-action models beyond memorization. Preprint, arXiv:2510.03827. Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Van- houcke, Huong Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, and 35 others. 2023. Rt-2: Vision- language-action models transfer web knowledge to robotic control. In Proceedings of The 7th Confer- ence on Robot Learning, volume 229 of Proceedings of Machine Learning Research, pages 2165–2183. PMLR. Appendix In this document, we provide additional details aboutRoboSPA. The appendix is organized as fol- lows: • §A: Detailed Failure Analysis. • §B: Training and Evaluation Details. • §C: Dataset Details. • §D: Basic Grounding Tasks. • §E: Task List and Descriptions. • §F: Detailed Experimental Results. • §G: Task Visualizations. A Detailed Failure Analysis A.1 Analysis Setup We perform a detailed error analysis on the two best-performing models,π 0.5 and X-VLA. For each model, we analyze approximately 1,120 evaluation videos, sampling up to 20 error examples per L5 task; if fewer than 20 errors occur, we include all available examples. For fine-grained spatial reason- ing tasks, we define three error types to distinguish target selection failures from execution failures: •Target Execution Error: The model selects the correct target but fails to execute the grasp or pick action properly. •No-Contact Grasp Error: The model attempts to grasp but fails to contact any object. •Target Grounding Error: The model selects the wrong object instead of the intended target. For long-horizon procedural planning tasks, we define five error types. One of them is the Target Grounding Error (as above), and the remaining four are: •Manipulation Error: Low-level execution of a subtask fails (e.g., pick, pull, or move). •Memory Error: The model fails to retain rele- vant information from previous steps, affecting subtask completion. •Temporal Ordering Error: Subtasks are exe- cuted in an incorrect order. • Redundant Repetition Error: The model re- peats an already completed action instead of pro- gressing to the next subgoal. A.2 Failure Analysis Results Figure 7 and Figure 8 show the distribution of error types for theπ 0.5 and X-VLA models, respectively. Overall, the two models exhibit broadly similar error patterns across the two capability dimensions, although the exact proportions vary. Fine-Grained Spatial Reasoning.In these tasks, both models are primarily challenged by Target Grounding Errors, which account for over 70% of all errors. The remaining errors are mostly Target Execution Errors, while No-Contact Grasp Errors occur very rarely. Among individual tasks, Canoni- cal Position Indexing (CPI) exhibits the highest pro- portion of Target Grounding Errors, while Cross- View Reasoning (CVR) has a relatively larger share of Target Execution Errors compared with the other tasks. This indicates that the main challenge lies in insufficient spatial reasoning, while low-level execution errors are comparatively rare. 13 0%20%40%60%80%100% MIP CAC OCE OFE RPF CVR R CPI SDE GAC Distribution of Error Types 10% 50% 15% 12% 13% 27% 1% 72% Target Execution Error No-Contact Grasp Error Target Grounding Error Temporal Ordering Error Redundant Repetition Error Memory Error Manipulation Error Fine-Grained Spatial Reasoning Long-Horizon Procedural Planning pi05 Figure 7: Distribution of error types for the π 0.5 model. 0%20%40%60%80%100% MIP CAC OCE OFE RPF CVR R CPI SDE GAC Distribution of Error Types 9% 51% 12% 14% 14% 20% 2% 78% Fine-Grained Spatial Reasoning Long-Horizon Procedural Planning Target Execution Error No-Contact Grasp Error Target Grounding Error Temporal Ordering Error Redundant Repetition Error Memory Error Manipulation Error Figure 8: Distribution of error types for the X-VLA model. Long-Horizon Procedural Planning. Within this dimension, errors are diverse. Overall, Ma- nipulation Errors account for the largest share, ap- proximately 50%, while the remaining error types occur at roughly similar rates. Repetitive Procedure Following (RPF) primarily exhibits Redundant Repetition Errors, suggesting that models lack effective tracking of task progress and may decide the next action based solely on the current visual observation. Order-Free Exe- cution (OFE) and Composite Action Coordination (CAC) are dominated by Manipulation Errors, high- lighting difficulties in low-level execution across multiple subgoals. Order-Constrained Execution (OCE) shows that, aside from Manipulation Errors, most remaining failures are Temporal Ordering Er- rors, indicating that models struggle to correctly sequence subtasks. Memory-Intensive Planning (MIP) is primarily affected by Memory Errors, in- dicating limitations in retaining task-relevant infor- 14 Table 5: Asset categories and the number of variants used in RoboSPA. Asset#Var. Asset#Var. Asset#Var. Asset#Var. bottle27olive-oil5pencup7globe2 bowl7drill7kitchenpot7trophy5 cover1jam-jar5battery6notebook3 plate1screwdriver1plasticbox11brush-pen6 fluted-block2fork1tabletrashbin11rest4 french-fries4knife1msg6glue6 hamburg6apple2soy-sauce5cleaner4 T_block2cabinet1vinegar3screen4 shoe-box1box1steamer3speaker6 tray4milk-box4boxdrink4fan7 kettle9mug13vagetable7seal6 pen7rack1paymentsign4milk-tea6 dustbin1shoe10can6roller3 plant-pot5wooden_box1electronicscale5fruit7 dumbbell-rack4book2rubikscube3board5 bookcase4microwave2displaystand5sauce-can5 laptop11sand-clock3bread7skillet4 oven4alarm-clock6breadbasket5soap4 calculator6mouse3phone5block7 microphone4stapler7phonestand7hydrating-oil4 coaster1shampoo7remotecontrol7basket4 hammer1bell2pillbottle5callbell6 cup13candlestick5playingcards3tea-box6 cup-with-liquid1dumbbell7smallshovel4coffee-box7 tissue-box7teanet5brush4perfume4 scanner5baguette2woodenmallet5keyboard4 chips-tub4small-speaker3gong6whiteboard-eraser1 pet-collar4switch8woodenblock5tooth-paste1 table-tennis2toycar6waterer8mini-chalkboard1 roll-paper4markpen6wineglass5plant1 Table 6: Model size and training compute budget per base task. All experiments were conducted on NVIDIA RTX PRO 6000 GPUs. ModelParams.#GPUsTimeGPU Hours. RDT1.2B12.5 h2.5 GO-12.8B26 h12 π 0.5 3.3B26 h12 X-VLA0.9B11.5 h1.5 mation across extended action sequences. Summary. These task-specific error patterns demonstrate that different tasks inRoboSPAcorre- spond to distinct failure modes, providing a bench- mark with good task differentiation for diagnos- ing VLA model capabilities. They further suggest that future VLA models should strengthen spatial grounding, ensure precise low-level execution, im- plement effective progress tracking, and incorpo- rate memory mechanisms to improve long-horizon tasks. B Training and Evaluation Details Training.For each base task, we use 50 trajecto- ries per difficulty level, resulting in 250 trajectories across five levels. For each task variant, we gen- erate 60 templates, with 50 used for training and 10 held out for evaluation. These templates are instantiated with task-specific object descriptions to form 100 distinct training instructions and 100 non-overlapping evaluation instructions. During training, one instruction is randomly sampled from the corresponding task-variant instruction pool for each trajectory. To ensure a fair and controlled comparison, all baseline models are initialized from their officially released pretrained weights and fully fine-tuned on our dataset, with all parameters updated dur- ing training. We follow their official implemen- tations, recommended environments, preprocess- ing pipelines, and evaluation settings, with only the training data and evaluation tasks replaced by RoboSPA. We adopt a unified training configuration across all models, using a batch size of 16 and train- ing for 20,000 steps. All experiments are conducted 15 Figure 9: The five robotic embodiments supported byRoboSPA. From left to right: Aloha-AgileX, ARX-X5, Piper, Franka, and UR5. on NVIDIA RTX PRO 6000 GPUs under consis- tent hardware settings. Table 6 summarizes the model sizes and training compute budgets per base task, including the number of parameters, GPU configuration, training time, and GPU hours for each evaluated baseline. Across all 56 base tasks and four baseline models, the total training budget is approximately 1,568 GPU-hours. Evaluation. Each task variant is evaluated over 100 test trials, and the Success Rate is computed accordingly. To assess generalization ability, all evaluation instructions are strictly disjoint from those used during training, ensuring no overlap be- tween training and test instructions. In addition to binary success, each task script records the com- pletion status of individual actions or subtasks dur- ing execution. This enables step-level fine-grained evaluation for long-horizon tasks, where a rollout may fail the final goal while still completing part of the required procedure. C Dataset Details C.1 Object Asset Details The object assets used inRoboSPAare collected from the RoboTwin-OD asset library (Chen et al., 2026a) and RMBench (Chen et al., 2026b). In to- tal,RoboSPAincludes 120 object categories and 581 object variants, covering diverse daily-use ob- jects with different appearances, geometries, and semantic types. These assets are used to instantiate task scenes across both fine-grained spatial reason- ing and long-horizon procedural planning tasks, as summarized in Table 5. C.2 Multi-Embodiment Setting RoboSPA supports multiple robotic embodiments, including Aloha-AgileX, ARX-X5, Piper, Franka, and UR5, as shown in Figure 9. These embod- iments share the same task semantics, language instructions, object configurations, and success con- ditions, while differing in morphology, workspace, camera viewpoint, and execution behavior. This design enables evaluation of VLA models under diverse robotic configurations and tests whether models can generalize beyond a single embodi- ment. C.3 Domain-Randomized Scenes To evaluate robustness beyond clean environments, we collect additional trajectories under domain- randomized scenes, as shown in Figure 10. The task semantics and success conditions remain un- changed, while the visual and environmental con- ditions are varied. Compared with clean scenes, the randomized setting introduces four types of variations: back- ground appearance, tabletop clutter, table height, and lighting conditions. Background randomiza- tion exposes models to diverse visual contexts, while tabletop clutter introduces distractor objects. Table-height perturbation adds mild geometric vari- ation, and lighting randomization changes illumi- nation conditions. D Basic Grounding Tasks We additionally evaluate three basic grounding vari- ants as lower-level references for spatial reasoning. These variants match the object categories and can- didate counts of the correspondingRoboSPAtasks, namely SDE, CVR, and CPI, respectively. Color- Based Grounding relies only on color cues, Egocen- tric Position Grounding uses robot-view positions without cross-view transformation, and Simple Or- dinal Indexing fixes the counting directions to left- to-right and far-to-near, unlike CPI where indexing directions vary. As shown in Table 7, these tasks achieve rela- tively high Success Rates across difficulty levels, indicating that basic grounding tasks may be in- sufficient to expose the reasoning limitations of current VLA models. In contrast,RoboSPAintroduces fine-grained spa- tial reasoning settings, such as distance estimation, 16 Figure 10: Examples of domain-randomized scenes in RoboSPA. Table 7: Performance on basic spatial task variants. TaskL1L2L3L4L5 Color-Based Grounding80.6 77.8 79.8 71.2 74.0 Egocentric Position Grounding 67.8 69.4 74.8 72.2 72.4 Simple Ordinal Indexing87.0 86.8 83.8 87.0 84.8 cross-view reasoning, relational grounding, and canonical position indexing with more complex spatial layouts. These tasks provide a more chal- lenging and diagnostic evaluation of spatial reason- ing ability. E Task List and Descriptions Tables 8 and 9 provide the full list of base tasks inRoboSPAand their objectives. Each base task is instantiated into 5 difficulty variants, where fine- grained spatial reasoning tasks vary the number of candidate objects and long-horizon procedural planning tasks vary the number of procedural steps. F Detailed Experimental Results F.1 Statistical Uncertainty We use a distinct seed for each evaluation episode, which determines the scene and object layout. Given the same checkpoint and seed, the simulated rollout is largely deterministic. To quantify the re- maining finite-sample uncertainty, we report boot- strap 95% confidence intervals for the L5 results in Table 10. The results show that the main perfor- mance trends remain unchanged after accounting for statistical uncertainty. F.2 Multi-Embodiment Evaluation To evaluate consistency across robotic platforms, we evaluate X-VLA on five embodiments. Due to computational resource constraints, we select one representative task from each of RoboSPA’s ten capability categories. For each task and em- bodiment, the model is independently trained and 17 evaluated using data from that embodiment. Tasks are denoted by their category abbreviation and in- dex (e.g., GAC-1), with the index corresponding to their order within the category in Tables 8 and 9. As shown in Tables 11 and 12, absolute perfor- mance varies across embodiments, but the L5 suc- cess rate is lower than the L1 success rate for every evaluated task and embodiment. This demonstrates that RoboSPA exhibits a consistent difficulty trend across robotic embodiments. F.3 Success Rates To provide a complete view of model performance, we report the detailed per-task Success Rates for all evaluated models across the five difficulty levels. Tables 13–16 list the results of RDT, GO-1,π 0.5 , and X-VLA on every task inRoboSPA. These ta- bles complement the aggregated results in the main paper by showing how each model performs on individual task variants, making it easier to identify task-specific strengths and failure patterns. Overall, the detailed results show that models generally perform better in settings with fewer dis- tractor objects and shorter action sequences. How- ever, performance drops become much sharper as difficulty increases, especially for tasks involving fine-grained spatial disambiguation, strict order- ing, compositional execution, or memory-intensive planning. F.4 Object-Normalized Target Accuracy Tables 17–20 report the Object-Normalized Target Accuracy (ONTA) of all evaluated models on fine- grained spatial reasoning tasks. These results help identify whether failures are caused by weak target disambiguation rather than low-level manipulation errors. TheONTAresults reveal clear differences in spatial grounding ability across models. RDT and GO-1 achieve poor scores, with many values be- low zero, indicating below-chance target selection on numerous spatial tasks.π 0.5 and X-VLA per- form better, but their scores still remain low on many variants and often stay close to the random- guessing baseline. These results suggest that cur- rent VLA models still have substantial room for improvement in fine-grained spatial reasoning, es- pecially under increasing spatial ambiguity and scene complexity. F.5 Progress Score Tables 21–24 report the Progress Scores of all eval- uated models on long-horizon procedural planning tasks. Unlike binary Success Rate, the Progress Score measures partial subtask completion dur- ing execution. These results provide a more fine- grained view of long-horizon failures by showing how much of the required procedure each model completes before failing. The results highlight the limited long-horizon planning ability of current models. RDT and GO-1 obtain consistently low Progress Scores, with most tasks scoring below 50, suggesting that they often fail to complete even half of the required procedure. π 0.5 and X-VLA achieve higher Progress Scores, indicating better ability to complete intermediate subtasks. However, their scores still decline clearly as difficulty increases, showing that current VLA models remain far from robust long-horizon proce- dural planning. G Task Visualizations We provide visualizations for the most difficult variant (L5) of each task inRoboSPA, as shown in Figures 11–20. Each row shows representative rollout frames together with the task category, task name, and instruction, illustrating the scene layout, target objects, and required execution process. 18 Table 8: Task list and descriptions for fine-grained spatial reasoning. Category abbreviations: GAC = Geometric Attribute Cognition, SDE = Spatial Distance Estimation, CPI = Canonical Position Indexing, R = Referential Relational Reasoning, and CVR = Cross-View Reasoning. Cat.Task NameTask Description GACPick Blocks SizePick the target block from multiple blocks based on the size. GACPick Blocks HeightPick the target block from multiple blocks based on the height. GACPick Blocks LengthPick the target block from multiple blocks based on the length. GACPick Blocks AreaPick the target block from multiple blocks based on the base area. SDEPick Blocks DistancePick the target block from multiple blocks using both color and relative distance cues. SDEPick Mugs Distance Pick the target mug from multiple mugs using both color and relative distance cues. SDEPick Pill Bottles DistancePick the target pill bottle from multiple pill bottles using both color and relative distance cues. SDEPick Mixed Objects Distance APick the target object from mixed-object set A using both color and relative distance cues. SDEPick Mixed Objects Distance B Pick the target object from mixed-object set B using both color and relative distance cues. CPIPick Blocks Canonical Pick the target block according to its ordinal position among multiple blocks. CPIPick Cups CanonicalPick the target cup according to its ordinal position among multiple cups. CPIPick Rubik Cubes CanonicalPick the target Rubik cube according to its ordinal position among multiple Rubik cubes. CPIPick Mixed Objects Canonical APick the target object from mixed-object set A according to ordinal position. CPIPick Mixed Objects Canonical BPick the target object from mixed-object set B according to ordinal position. RRRPick Tea Box RelationalPick the target tea box according to its relative position to another reference tea box. RRRPick Cans RelationalPick the target can according to its relative position to another reference can. RRRPick Soaps RelationalPick the target soap according to its relative position to another reference soap. RRRPick Mixed Objects Relational APick the target object in a mixed-object scene according to another reference object. RRRPick Mixed Objects Relational BPick the target object in a mixed-object scene according to another reference object. CVRPick Sauce Can Multi ViewPick up the target sauce can specified by its absolute position under a given viewpoint. CVRPick Seals Multi View Pick up the target seal specified by its absolute position under a given viewpoint. CVRPick Breads Multi ViewPick up the target bread specified by its absolute position under a given viewpoint. CVRPick Mixed Objects Multi View A Pick up the target object from mixed-object group A specified by its absolute position under a given viewpoint. CVRPick Mixed Objects Multi View BPick up the target object from mixed-object group B specified by its absolute position under a given viewpoint. 19 Table 9: Task list and descriptions for long-horizon procedural planning. Category abbreviations: RPF = Repetitive Procedure Following, OFE = Order-Free Execution, OCE = Order-Constrained Execution, CAC = Composite Action Coordination, and MIP = Memory-Intensive Planning. Cat.Task NameTask Description RPFPress Stapler RepeatPress the stapler repeatedly. RPFLift Pot RepeatLift and put down the pot repeatedly. RPFLift Fan RepeatLift the fan repeatedly. RPFClick Bell RepeatClick the bell repeatedly. OFEPut Bottles DustbinPut multiple bottles into the dustbin without requiring a fixed execution order. OFEMove Blocks ApartMove multiple blocks apart without requiring a fixed execution order. OFEMove Playing Cards AwayMove multiple playing cards away without requiring a fixed execution order. OFEPlace Bowls PlatesPlace multiple bowls on plates without requiring a fixed execution order. OFESeparate Fries BreadSeparate fries and bread into different areas without requiring a fixed execution order. OFERank Blocks HeightArrange multiple blocks by height while allowing flexible intermediate execution order. OFERank Blocks ColorArrange multiple blocks by color while allowing flexible intermediate execution order. OFERank Blocks SizeArrange multiple blocks by size while allowing flexible intermediate execu- tion order. OCEHit Blocks Hammer OrderHit multiple blocks with a hammer in the specified color order. OCEClick Objects OrderClick multiple target objects in the specified order. OCEStamp Seals OrderStamp multiple seals in the specified order. OCEClick Bell Clockwise Order Click the bell and then execute the clockwise movement sequence in order. OCEStack Blocks Color OrderStack multiple blocks in the specified color order. OCEStack Blocks Size OrderStack multiple blocks in the specified size order. OCEStack Blocks Length OrderStack multiple blocks in the specified length order. CACPlace Phone Press StaplerExecute a manipulation task involving phone placement, repeated pressing, and dumping. CACPlace Bottle CupExecute a manipulation task involving throwing bottles into a dustbin and placing cups on coasters. CACHang Mug Stack BlocksExecute a manipulation task involving hanging a mug on a rack and stacking blocks from bottom to top. CACPlace Object Scale ClickExecute a manipulation task involving object placement, pressing, and returning objects to the table. CACPlace Burger Fries Click Bell Execute a manipulation task involving placing a hamburger and fries on a tray, clicking a bell, and returning them. CACClick Can Place ItemsExecute a manipulation task involving interleaved can clicking with placing bread in a basket and a pill bottle on a pad. CACStamp Seals Press Stapler Execute a manipulation task involving stamping left and right seals twice on their pads and pressing a stapler. CACClick Bell Open Microwave Place Ob- ject Execute a manipulation task involving interleaved bell clicking, opening a microwave, and placing an object left of another. MIPObserve Blocks Move Memory First remember the colors of the display blocks, then move matching blocks on the table closer after the display blocks are hidden. MIPObserve Objects Click MemoryFirst remember the objects on the display platform from left to right, then click matching objects on the table in the same order after they are hidden. MIPRemember Color CoverFirst remember the block colors, then uncover the blocks in the given color order after they are covered. MIPRemember Orientation RestoreFirst remember the orientations of the front T-shaped blocks, then adjust the rear T-shaped blocks to match them from left to right after they are hidden. MIPPress Stapler MemoryRemember how many times each stapler should be pressed and then press them. 20 Table 10: Bootstrap 95% confidence intervals for the average L5 Success Rates (%). ModelFine-Grained Spatial Reasoning Avg.Long-Horizon Procedural Planning Avg.Overall Avg. RDT6.1 [5.2, 7.1]7.7 [6.8, 8.5]6.9 [6.3, 7.5] GO-110.6 [9.4, 11.8]7.0 [6.2, 7.7]8.8 [8.1, 9.5] π 0.5 21.4 [19.8, 22.9]23.1 [21.9, 24.3]22.3 [21.3, 23.2] X-VLA23.9 [22.3, 25.5]15.9 [14.8, 17.0]19.9 [18.9, 20.8] Table 11: Multi-embodiment evaluation of X-VLA on Fine-Grained Spatial Reasoning tasks. EmbodimentGAC-1SDE-5CPI-4R-3CVR-4 L1 L3 L5 L1 L3 L5 L1 L3 L5 L1 L3 L5 L1 L3 L5 Aloha-AgileX 48 22 16 48 31 27 29 13743 31 14 41 30 22 ARX-X551 18 11 36 29 26 32 14750 24 13 28 11 13 Piper48 34 16 18 20 17 40 18 10 40 18 20 50 44 39 Franka22 12814 10 10 38 18848 18 14 1287 UR574 43 25 52 41 34 67 35 26 58 45 38 59 55 49 Table 12: Multi-embodiment evaluation of X-VLA on Long-Horizon Procedural Planning tasks. EmbodimentRPF-1OFE-1OCE-5CAC-6MIP-3 L1 L3 L5 L1 L3 L5L1L3 L5L1L3 L5L1L3 L5 Aloha-AgileX 49 30 27 79 55 289967 185437 24 10080 ARX-X554 46 40 64 59 27 100 54 229953 2999120 Piper74 66 40 9690100 1695231 199900 Franka66 40 24 26 20 11921809840010000 UR583 58 49 99 2459333 17 100 51 48 100 160 Table 13: RDT per-task Success Rate across difficulty levels. TaskL1 L2 L3 L4 L5TaskL1 L2 L3 L4 L5 Pick Blocks Size31361Put Bottles Dustbin51 37 2282 Pick Blocks Height28 21 20 15 12Move Blocks Apart62100 Pick Blocks Length61143Move Playing Cards Away85000 Pick Blocks Area31153Place Bowls Plates48 15 1262 Pick Blocks Distance16134Separate Fries Bread249612 Pick Mugs Distance15 17 12 13 10Rank Blocks Height61010 Pick Pill Bottles Distance25 15 14 13 13Rank Blocks Color146100 Pick Mixed Objects Distance A15 16 146 10Rank Blocks Size50000 Pick Mixed Objects Distance B20 13 14 13 10Hit Blocks Hammer Order404000 Pick Blocks Canonical35415Click Objects Order200000 Pick Cups Canonical35 18 21 17 22Stamp Seals Order30111 Pick Rubik Cubes Canonical76677Click Bell Clockwise Order50000 Pick Mixed Objects Canonical A114515Stack Blocks Color Order110000 Pick Mixed Objects Canonical B196564Stack Blocks Size Order90000 Pick Tea Box Referential72563Stack Blocks Length Order70000 Pick Cans Referential21756Place Phone Press Stapler186100 Pick Soaps Referential8 12957Place Bottle Cup56 29 2660 Pick Mixed Objects Referential A22774Hang Mug Stack Blocks100000 Pick Mixed Objects Referential B32342Place Object Scale Click39 33 269 17 Pick Sauce Can Multi View35437Place Burger Fries Click Bell33 20400 Pick Seals Multi View35544Click Can Place Items62 24 3402 Pick Breads Multi View35230Stamp Seals Press Stapler23000 Pick Mixed Objects Multi View A15234Click Open Place01200 Pick Mixed Objects Multi View B51432Observe Blocks Move Memory00000 Press Stapler Repeat33 37 37 34 32Observe Objects Click Memory 140000 Lift Pot Repeat76 68 63 56 68Remember Color Cover47 14600 Lift Fan Repeat39 33 269 17Remember Orientation Restore00000 Click Bell Repeat15 17 22 22 23Press Stapler Memory303000 21 Table 14: GO-1 per-task Success Rate across difficulty levels. TaskL1 L2 L3 L4 L5TaskL1 L2 L3 L4 L5 Pick Blocks Size33 13 12 109Put Bottles Dustbin59 77 42 33 11 Pick Blocks Height71 62 50 47 32Move Blocks Apart21 10310 Pick Blocks Length3 11 1575Move Playing Cards Away52 39 33 13 17 Pick Blocks Area43200Place Bowls Plates37 23 21 135 Pick Blocks Distance32643Separate Fries Bread44 32 19 217 Pick Mugs Distance15 14 1489Rank Blocks Height343000 Pick Pill Bottles Distance368 15 15 26Rank Blocks Color41 14852 Pick Mixed Objects Distance A26 19 16 19 12Rank Blocks Size41 22 1393 Pick Mixed Objects Distance B21 17 1488Hit Blocks Hammer Order36 11331 Pick Blocks Canonical16 11867Click Objects Order62 11643 Pick Cups Canonical24 17 1387Stamp Seals Order12 15 1096 Pick Rubik Cubes Canonical26 17 14 13 15Click Bell Clockwise Order61 48 34 21 12 Pick Mixed Objects Canonical A21 12664Stack Blocks Color Order54 12310 Pick Mixed Objects Canonical B23 15743Stack Blocks Size Order38 11100 Pick Tea Box Referential21 17 1897Stack Blocks Length Order307500 Pick Cans Referential16 124 148Place Phone Press Stapler48 14 1451 Pick Soaps Referential128433Place Bottle Cup16 12 34 234 Pick Mixed Objects Referential A 20 18 13 159Hang Mug Stack Blocks82000 Pick Mixed Objects Referential B 17 15 15 146Place Object Scale Click12 14 1023 Pick Sauce Can Multi View28 33 31 29 27Place Burger Fries Click Bell37 29 1135 Pick Seals Multi View18 15 14 12 11Click Can Place Items77 11 1453 Pick Breads Multi View21 129 27 15Stamp Seals Press Stapler24 17200 Pick Mixed Objects Multi View A 1684 18 18Click Open Place12 13702 Pick Mixed Objects Multi View B44 12 12 10Observe Blocks Move Memory30000 Press Stapler Repeat27 21 1287Observe Objects Click Memory 191000 Lift Pot Repeat61 53 83 84 77Remember Color Cover50000 Lift Fan Repeat23 17 17 21 11Remember Orientation Restore20000 Click Bell Repeat16 12310Press Stapler Memory92000 Table 15: π 0.5 per-task Success Rate across difficulty levels. TaskL1 L2 L3 L4 L5TaskL1 L2 L3 L4 L5 Pick Blocks Size58 27 20 17 13Put Bottles Dustbin95 85 81 74 61 Pick Blocks Height67 54 48 30 24Move Blocks Apart73 51 24 35 18 Pick Blocks Length225 1143Move Playing Cards Away81 64 54 52 47 Pick Blocks Area24 13 25 12 10Place Bowls Plates94 85 61 52 22 Pick Blocks Distance16 16 16 10 11Separate Fries Bread92 84 58 59 36 Pick Mugs Distance39 51 52 49 38Rank Blocks Height72 45 19 246 Pick Pill Bottles Distance46 36 31 33 33Rank Blocks Color88 50 46 36 11 Pick Mixed Objects Distance A33 37 27 29 23Rank Blocks Size91 66 37 158 Pick Mixed Objects Distance B41 29 21 22 12Hit Blocks Hammer Order81 30 1473 Pick Blocks Canonical41 22 14 10 11Click Objects Order60 29 16 13 15 Pick Cups Canonical61 37 40 41 45Stamp Seals Order40 40 30 36 35 Pick Rubik Cubes Canonical51 35 33 36 44Click Bell Clockwise Order98 95 45 31 28 Pick Mixed Objects Canonical A35 15 10 11 11Stack Blocks Color Order91 58 2551 Pick Mixed Objects Canonical B32 15 10 107Stack Blocks Size Order89 43430 Pick Tea Box Referential33 31 31 12 15Stack Blocks Length Order83 27 1832 Pick Cans Referential27 21 28 178Place Phone Press Stapler46 14 17 121 Pick Soaps Referential53 44 38 19 10Place Bottle Cup97 79 69 67 30 Pick Mixed Objects Referential A 33 17 12 128Hang Mug Stack Blocks158300 Pick Mixed Objects Referential B 20 21 18 188Place Object Scale Click19 36 5665 Pick Sauce Can Multi View47 48 43 42 42Place Burger Fries Click Bell87 89 57 14 34 Pick Seals Multi View32 31 31 31 30Click Can Place Items78 46 47 14 18 Pick Breads Multi View56 51 54 52 61Stamp Seals Press Stapler77 82871 Pick Mixed Objects Multi View A 33 27 27 33 32Click Open Place50 22 375 11 Pick Mixed Objects Multi View B 38 29 30 22 23Observe Blocks Move Memory210000 Press Stapler Repeat79 41 42 36 45Observe Objects Click Memory97 35300 Lift Pot Repeat60 43 63 54 63Remember Color Cover100 48 1120 Lift Fan Repeat89 80 73 71 62Remember Orientation Restore200100 Click Bell Repeat93 84 87 95 90Press Stapler Memory323000 22 Table 16: X-VLA per-task Success Rate across difficulty levels. TaskL1 L2 L3 L4 L5TaskL1 L2 L3 L4 L5 Pick Blocks Size48 27 22 23 16Put Bottles Dustbin79 64 55 31 28 Pick Blocks Height88 81 65 56 55Move Blocks Apart86 74 81 17 31 Pick Blocks Length43 30 21 28 14Move Playing Cards Away97 96 92 87 83 Pick Blocks Area52 27 25 28 16Place Bowls Plates67 44 29 158 Pick Blocks Distance39 44 33 29 33Separate Fries Bread78 87 71 79 66 Pick Mugs Distance44 42 25 29 23Rank Blocks Height91 95 84 72 46 Pick Pill Bottles Distance34 42 42 34 64Rank Blocks Color93 92 91 82 63 Pick Mixed Objects Distance A42 40 36 34 26Rank Blocks Size97 88 65 32 24 Pick Mixed Objects Distance B48 40 31 37 27Hit Blocks Hammer Order84 36 1674 Pick Blocks Canonical36 23 10 108Click Objects Order396000 Pick Cups Canonical38 23 20 19 11Stamp Seals Order34 71 65 65 31 Pick Rubik Cubes Canonical26 18 145 12Click Bell Clockwise Order57 15100 Pick Mixed Objects Canonical A29 10 1377Stack Blocks Color Order99 93 67 33 18 Pick Mixed Objects Canonical B30 15 15 136Stack Blocks Size Order78 87 2921 Pick Tea Box Referential43 38 27 23 14Stack Blocks Length Order99 74 59 140 Pick Cans Referential36 22 19 24 13Place Phone Press Stapler51 28 25 227 Pick Soaps Referential43 30 31 15 14Place Bottle Cup98 44 19 166 Pick Mixed Objects Referential A 36 21 26 14 17Hang Mug Stack Blocks235320 Pick Mixed Objects Referential B 44 34 23 16 13Place Object Scale Click57 22 1952 Pick Sauce Can Multi View54 53 62 57 59Place Burger Fries Click Bell98 59 18 124 Pick Seals Multi View30 38 32 30 35Click Can Place Items54 41 37 34 24 Pick Breads Multi View37 37 33 41 34Stamp Seals Press Stapler75 86421 Pick Mixed Objects Multi View A 41 28 30 28 22Click Open Place32000 Pick Mixed Objects Multi View B 29 27 33 36 33Observe Blocks Move Memory46 11530 Press Stapler Repeat49 35 30 29 27Observe Objects Click Memory180000 Lift Pot Repeat00000Remember Color Cover100 48800 Lift Fan Repeat61 70 59 51 43Remember Orientation Restore186000 Click Bell Repeat29 24 20 19 20Press Stapler Memory411000 Table 17: RDT per-task Object-Normalized Target Accuracy (ONTA) across difficulty levels. Fine-grained spatial reasoning tasks only. TaskL1L2L3L4L5TaskL1L2L3L4L5 Pick Blocks Size-94.0 -48.5 -29.3 -17.5 -18.8Pick Mixed Objects Canonical A-33.5 -15.2-8.6 -11.4-3.6 Pick Blocks Height-44.0 -18.5-6.7-6.3-5.6Pick Mixed Objects Canonical B-21.5 -12.8-8.6-5.8-4.7 Pick Blocks Length-88.0 -48.5 -32.0 -20.0 -16.4Pick Tea Box Referential-86.0 -47.0 -26.7 -12.8-9.1 Pick Blocks Area-94.0 -48.5 -32.0 -18.8 -16.4Pick Cans Referential-96.0 -48.5 -24.0 -14.0-5.8 Pick Blocks Distance-48.5 -12.8 -13.1-9.1-6.7Pick Soaps Referential-84.0 -32.0 -21.3 -14.0-4.6 Pick Mugs Distance-27.5 -10.7 -10.0-4.4-5.0Pick Mixed Objects Referential A -96.0 -47.0 -24.0 -11.6-8.0 Pick Pill Bottles Distance-12.5 -13.3-7.5-4.4-1.5Pick Mixed Objects Referential B -94.0 -47.0 -29.3 -15.2 -10.3 Pick Mixed Objects Distance A -27.5 -12.0-7.5 -12.8-5.0Pick Sauce Can Multi View-45.5 -26.7 -28.0 -29.3 -24.0 Pick Mixed Objects Distance B -20.0 -16.0-7.5-4.4-5.0Pick Seals Multi View-45.5 -26.7 -26.7 -28.0 -28.0 Pick Blocks Canonical-45.5 -14.0-9.7 -11.4-3.6Pick Breads Multi View-45.5 -26.7 -30.7 -29.3 -33.3 Pick Cups Canonical2.51.69.76.6 14.9Pick Mixed Objects Multi View A -48.5 -26.7 -30.7 -29.3 -28.0 Pick Rubik Cubes Canonical-39.5 -12.8-7.4-4.6-1.5Pick Mixed Objects Multi View B -42.5 -32.0 -28.0 -29.3 -30.7 23 Table 18: GO-1 per-task Object-Normalized Target Accuracy (ONTA) across difficulty levels. Fine-grained spatial reasoning tasks only. TaskL1L2L3L4L5TaskL1L2L3L4L5 Pick Blocks Size-34.0 -30.5 -17.3 -12.5-9.2Pick Mixed Objects Canonical A-18.5-5.6-7.4-5.8-4.7 Pick Blocks Height42.0 43.0 33.3 33.8 18.4Pick Mixed Objects Canonical B-15.5-2.0-6.3-8.0-5.8 Pick Blocks Length-94.0 -33.5 -13.3 -16.3 -14.0Pick Tea Box Referential-58.0 -24.5-9.3-9.2-4.6 Pick Blocks Area-92.0 -45.5 -30.7 -25.0 -20.0Pick Cans Referential-68.0 -32.0 -28.0-3.2-3.5 Pick Blocks Distance-45.5 -17.6-7.4-8.0-7.8Pick Soaps Referential-76.0 -38.0 -28.0 -16.4-9.1 Pick Mugs Distance-27.5 -14.7-7.5 -10.4-6.2Pick Mixed Objects Referential A -60.0 -23.0 -16.0-2.0-2.4 Pick Pill Bottles Distance4.0 -22.7-6.3-2.0 13.7Pick Mixed Objects Referential B -66.0 -27.5 -13.3-3.2-5.8 Pick Mixed Objects Distance A -11.0-8.0-5.02.8-2.7Pick Sauce Can Multi View-8.0 10.78.05.32.7 Pick Mixed Objects Distance B -18.5 -10.7-7.5 -10.4-7.3Pick Seals Multi View-23.0 -13.3 -14.7 -17.3 -18.7 Pick Blocks Canonical-26.0-6.8-5.1-5.8-1.5Pick Breads Multi View-18.5 -17.3 -21.32.7 -13.3 Pick Cups Canonical-14.00.40.6-3.5-1.5Pick Mixed Objects Multi View A -26.0 -22.7 -28.0-9.3-9.3 Pick Rubik Cubes Canonical-11.00.41.72.17.3Pick Mixed Objects Multi View B -44.0 -28.0 -17.3 -17.3 -20.0 Table 19:π 0.5 per-task Object-Normalized Target Accuracy (ONTA) across difficulty levels. Fine-grained spatial reasoning tasks only. TaskL1L2L3L4L5TaskL1L2L3L4L5 Pick Blocks Size16.0-9.5-6.7-3.8-4.4Pick Mixed Objects Canonical A2.5-2.0-2.9 -0.12.9 Pick Blocks Height34.0 31.0 30.7 12.58.8Pick Mixed Objects Canonical B-2.0-2.0-2.9 -1.2 -1.5 Pick Blocks Length-56.0 -42.5 -18.7 -20.0 -16.4Pick Tea Box Referential-34.0-3.58.0 -5.64.4 Pick Blocks Area-52.0 -30.50.0 -10.0-8.0Pick Cans Referential-46.0 -18.54.00.4 -3.5 Pick Blocks Distance-26.0-0.84.0-1.21.1Pick Soaps Referential6.0 16.0 17.32.8 -1.2 Pick Mugs Distance8.5 34.7 40.0 38.8 27.7Pick Mixed Objects Referential A -34.0 -24.5 -17.3 -5.6 -3.5 Pick Pill Bottles Distance19.0 14.7 13.8 19.6 21.8Pick Mixed Objects Referential B -60.0 -18.5-9.31.6 -3.5 Pick Mixed Objects Distance A-0.5 16.08.8 14.8 10.2Pick Sauce Can Multi View20.5 30.7 24.0 22.7 22.7 Pick Mixed Objects Distance B11.55.31.3 22.0-2.7Pick Seals Multi View-2.08.08.08.06.7 Pick Blocks Canonical11.56.41.7-1.22.9Pick Breads Multi View34.0 34.7 38.7 36.0 48.0 Pick Cups Canonical41.5 24.4 31.4 33.6 40.0Pick Mixed Objects Multi View A-0.52.72.7 10.79.3 Pick Rubik Cubes Canonical26.5 22.0 23.4 28.0 38.9Pick Mixed Objects Multi View B7.05.36.7 -4.0 -2.7 Table 20: X-VLA per-task Object-Normalized Target Accuracy (ONTA) across difficulty levels. Fine-grained spatial reasoning tasks only. TaskL1L2L3L4L5TaskL1L2L3L4L5 Pick Blocks Size-4.0 -9.5 -4.03.8 -0.8Pick Mixed Objects Canonical A-6.5-8.00.6 -4.6 -1.5 Pick Blocks Height76.0 71.5 53.3 45.0 46.0Pick Mixed Objects Canonical B-5.0-2.02.92.1 -2.5 Pick Blocks Length-14.0 -5.0 -5.3 10.0 -3.2Pick Tea Box Referential-14.07.02.77.63.3 Pick Blocks Area4.0 -9.50.0 10.0 -0.8Pick Cans Referential-28.0 -17.0 -8.08.82.1 Pick Blocks Distance8.5 32.8 23.4 20.1 25.6Pick Soaps Referential-14.0-5.08.0 -2.03.3 Pick Mugs Distance16.0 22.76.3 14.8 10.2Pick Mixed Objects Referential A -28.0 -18.51.3 -3.26.6 Pick Pill Bottles Distance1.0 22.7 27.5 20.8 58.0Pick Mixed Objects Referential B -12.01.0 -2.7 -0.82.1 Pick Mixed Objects Distance A 13.0 20.0 20.0 20.8 13.7Pick Sauce Can Multi View31.0 37.3 49.3 42.7 45.3 Pick Mixed Objects Distance B22.0 20.0 13.8 24.4 14.8Pick Seals Multi View-5.0 17.39.36.7 13.3 Pick Blocks Canonical4.07.6 -2.9 -1.2 -0.4Pick Breads Multi View5.5 16.0 10.7 21.3 12.0 Pick Cups Canonical7.07.68.68.92.9Pick Mixed Objects Multi View A 11.54.06.74.0 -4.0 Pick Rubik Cubes Canonical-11.01.61.7 -6.94.0Pick Mixed Objects Multi View B-6.52.7 10.7 14.7 10.7 24 Table 21: RDT per-task step Progress Score across difficulty levels. Step tables include long-horizon tasks only. TaskL1L2L3L4L5TaskL1L2L3L4L5 Press Stapler Repeat33.0 37.0 39.3 36.5 35.2Stack Blocks Color Order11.0 50.0 33.7 25.0 20.2 Lift Pot Repeat76.0 75.5 75.7 72.0 80.0Stack Blocks Size Order9.00.0 33.7 25.3 20.4 Lift Fan Repeat39.0 41.5 41.3 29.8 34.0Stack Blocks Length Order7.0 50.0 34.0 25.3 20.4 Click Bell Repeat15.0 21.5 28.7 26.5 28.6Place Phone Press Stapler18.0 40.0 44.0 49.3 38.2 Put Bottles Dustbin51.0 56.0 48.3 33.8 33.6Place Bottle Cup56.0 52.5 52.7 44.3 33.6 Move Blocks Apart6.07.0 12.7 11.08.0Hang Mug Stack Blocks10.09.0 17.7 13.09.6 Move Playing Cards Away8.0 13.08.09.0 10.6Place Object Scale Click39.0 41.5 41.3 29.8 34.0 Place Bowls Plates48.0 37.5 39.0 32.3 20.6Place Burger Fries Click Bell33.0 23.5 17.7 15.3 11.4 Separate Fries Bread24.0 26.0 33.0 28.0 35.4Click Can Place Items62.0 55.5 75.7 55.0 65.0 Rank Blocks Height6.0 50.5 40.0 37.8 31.4Stamp Seals Press Stapler2.05.01.30.83.0 Rank Blocks Color14.0 53.0 44.3 37.5 31.4Click Open Place0.0 16.5 10.0 22.8 17.2 Rank Blocks Size5.0 50.0 38.3 33.8 28.0Observe Blocks Move Memory0.00.00.00.00.0 Hit Blocks Hammer Order40.0 14.59.36.57.2Observe Objects Click Memory 14.07.05.36.35.8 Click Objects Order20.0 23.5 22.7 17.50.0Remember Color Cover47.0 23.5 15.05.56.0 Stamp Seals Order3.03.05.76.36.6Remember Orientation Restore0.00.50.70.00.2 Click Bell Clockwise Order5.0 50.0 38.3 33.8 28.0Press Stapler Memory30.07.04.03.35.8 Table 22: GO-1 per-task step Progress Score across difficulty levels. Step tables include long-horizon tasks only. TaskL1L2L3L4L5TaskL1L2L3L4L5 Press Stapler Repeat27.0 25.5 14.78.38.8Stack Blocks Color Order54.0 56.0 36.7 25.8 20.0 Lift Pot Repeat61.0 54.0 92.0 92.8 91.6Stack Blocks Size Order38.0 16.55.04.34.2 Lift Fan Repeat23.0 22.5 24.7 26.5 21.2Stack Blocks Length Order30.0 12.0 10.34.33.0 Click Bell Repeat16.0 19.08.79.36.3Place Phone Press Stapler48.0 33.0 34.3 30.0 28.8 Put Bottles Dustbin59.0 84.5 68.0 60.3 48.4Place Bottle Cup16.0 16.5 50.3 68.5 57.0 Move Blocks Apart21.0 21.5 12.08.36.6Hang Mug Stack Blocks8.0 23.3 18.5 18.3 13.1 Move Playing Cards Away52.0 43.5 45.0 20.5 21.0Place Object Scale Click12.0 19.0 14.36.88.2 Place Bowls Plates37.0 36.0 35.0 26.8 20.8Place Burger Fries Click Bell37.0 32.5 13.75.06.6 Separate Fries Bread44.0 37.5 26.7 27.8 14.6Click Can Place Items77.0 45.5 53.3 51.0 55.4 Rank Blocks Height34.0 51.5 39.3 36.0 30.0Stamp Seals Press Stapler24.0 23.08.35.35.2 Rank Blocks Color41.0 57.0 44.7 36.5 30.6Click Open Place12.0 13.07.00.05.8 Rank Blocks Size41.0 61.0 47.3 40.5 34.0Observe Blocks Move Memory3.03.01.3 11.0 11.6 Hit Blocks Hammer Order36.0 29.0 24.3 27.5 27.0Observe Objects Click Memory 19.06.58.35.85.2 Click Objects Order62.0 42.0 30.3 24.33.0Remember Color Cover5.00.00.00.30.0 Stamp Seals Order12.0 29.5 24.0 23.3 21.4Remember Orientation Restore2.06.05.76.55.4 Click Bell Clockwise Order 61.0 52.0 38.3 30.8 25.2Press Stapler Memory9.06.59.39.57.6 Table 23: π 0.5 per-task step Progress Score across difficulty levels. Step tables include long-horizon tasks only. TaskL1L2L3L4L5TaskL1L2L3L4L5 Press Stapler Repeat79.0 61.5 63.0 59.0 64.8Stack Blocks Color Order91.0 80.0 58.7 37.0 23.8 Lift Pot Repeat60.0 64.5 77.0 71.8 74.8Stack Blocks Size Order89.0 75.0 49.3 36.3 25.0 Lift Fan Repeat89.0 85.0 84.0 80.8 75.4Stack Blocks Length Order83.0 65.5 52.0 33.3 27.0 Click Bell Repeat93.0 89.0 90.7 98.0 94.2Place Phone Press Stapler46.0 31.0 34.7 32.5 26.6 Put Bottles Dustbin95.0 90.5 93.0 87.8 82.4Place Bottle Cup97.0 90.0 87.3 86.3 75.8 Move Blocks Apart73.0 67.5 62.0 61.5 55.6Hang Mug Stack Blocks15.0 48.0 44.3 26.5 23.6 Move Playing Cards Away81.0 78.5 77.3 81.5 77.8Place Object Scale Click19.0 50.5 56.0 22.8 30.2 Place Bowls Plates94.0 91.5 78.0 69.5 50.4Place Burger Fries Click Bell87.0 90.0 78.0 61.8 72.0 Separate Fries Bread92.0 88.5 83.0 86.8 74.4Click Can Place Items78.0 69.0 78.3 69.0 73.0 Rank Blocks Height72.0 72.5 63.7 68.0 53.0Stamp Seals Press Stapler77.0 86.0 50.0 48.8 45.0 Rank Blocks Color88.0 77.5 73.0 68.5 55.4Click Open Place50.0 48.5 55.0 30.8 37.4 Rank Blocks Size91.0 84.0 69.7 57.5 47.8Observe Blocks Move Memory21.0 15.0 16.0 13.5 14.4 Hit Blocks Hammer Order81.0 33.0 27.3 23.3 19.8Observe Objects Click Memory97.0 58.5 33.0 25.5 14.4 Click Objects Order60.0 35.0 27.0 28.0 15.0Remember Color Cover100.0 50.5 20.3 12.07.0 Stamp Seals Order40.0 50.5 43.0 43.6 46.6Remember Orientation Restore20.0 12.0 14.7 13.3 12.0 Click Bell Clockwise Order 98.0 96.5 52.7 40.8 38.6Press Stapler Memory32.0 10.09.78.89.4 25 Table 24: X-VLA per-task step Progress Score across difficulty levels. Step tables include long-horizon tasks only. TaskL1L2L3L4L5TaskL1L2L3L4L5 Press Stapler Repeat49.0 42.5 40.3 38.8 36.6Stack Blocks Color Order99.0 96.5 79.0 54.5 37.8 Lift Pot Repeat0.00.00.00.00.0Stack Blocks Size Order78.0 93.5 55.0 29.3 22.2 Lift Fan Repeat61.0 79.5 72.0 71.8 66.2Stack Blocks Length Order99.0 87.5 73.7 41.8 24.2 Click Bell Repeat29.0 28.5 27.0 23.3 24.8Place Phone Press Stapler51.0 45.5 46.7 43.5 43.8 Put Bottles Dustbin79.0 80.0 78.3 71.0 74.0Place Bottle Cup98.0 71.0 61.3 49.0 40.0 Move Blocks Apart86.0 80.5 89.7 65.0 71.2Hang Mug Stack Blocks23.0 51.0 51.7 41.3 31.4 Move Playing Cards Away97.0 98.0 95.7 93.6 93.6Place Object Scale Click57.0 35.0 30.3 27.3 20.0 Place Bowls Plates67.0 62.5 52.7 39.3 30.4Place Burger Fries Click Bell98.0 78.5 55.7 57.3 49.8 Separate Fries Bread78.0 92.5 87.3 89.5 82.2Click Can Place Items54.0 64.5 67.7 69.3 64.2 Rank Blocks Height91.0 97.5 93.3 91.0 83.0Stamp Seals Press Stapler75.0 89.0 61.0 49.3 42.8 Rank Blocks Color93.0 96.5 95.0 92.5 90.0Click Open Place15.0 19.5 18.0 24.8 28.8 Rank Blocks Size97.0 94.5 83.3 71.0 67.0Observe Blocks Move Memory46.0 29.5 27.7 27.3 19.4 Hit Blocks Hammer Order84.0 57.5 46.7 40.5 30.8Observe Objects Click Memory18.04.55.02.82.6 Click Objects Order39.0 20.0 10.36.50.0Remember Color Cover100.0 50.0 18.36.54.0 Stamp Seals Order34.0 77.0 78.3 78.3 53.2Remember Orientation Restore18.0 19.57.06.35.6 Click Bell Clockwise Order 57.0 26.0 14.79.87.2Press Stapler Memory41.06.53.04.34.8 26 Task name: Pick Blocks Size Instruction: Grab the block ranked 5th by size with one arm. Geometric Attribute Cognition Task name: Pick Blocks Height Instruction: Lift the block that is 3rd in height order. Geometric Attribute Cognition Task name: Pick Blocks Length Instruction:Use one arm to lift the 2nd longest block. Geometric Attribute Cognition Task name: Pick Blocks Area Instruction:Grab the block ranked 1st by base area with one arm. Geometric Attribute Cognition Task name: Pick Blocks Distance Instruction:Lift the block that is the 2nd farthest from the orange block with one arm. Spatial Distance Estimation Task name: Pick Mugs Distance Instruction:Lift the 1st farthest mug from the green mug with one arm. Spatial Distance Estimation Task name: Pick Pill Bottles Distance Instruction: Take hold of the 3rd farthest pill bottle from the orange pill bottle using one arm. Spatial Distance Estimation Task name: Pick Mixed Objects Distance A Instruction:Find the 6th farthest object from the fan and grab it. Spatial Distance Estimation Task name: Pick Mixed Objects Distance B Instruction:Lift the object that is the 3rd farthest from the cup. Spatial Distance Estimation Task name: Pick Blocks Canonical Instruction:Pick up the block in the 1st row counted from far to near at the 1st position counted from left to right. Canonical Position Indexing Figure 11: Fine-grained spatial reasoning task examples in RoboSPA (Part 1). 27 Task name: Pick Cups Canonical Instruction:Find the 4th cup in the 1st row counted from near to far, with positions counted from left to right, and pick it up. Canonical Position Indexing Task name: Pick Rubik Cubes Canonical Instruction:Pick up the 2nd rubik’scube counted from rightto leftin the 1st row counted from near to far. Canonical Position Indexing Task name: Pick Mixed Objects Canonical A Instruction:Pick up the object in the 2nd row counted from far to near at the 2nd position counted from left to right. Canonical Position Indexing Task name: Pick Mixed Objects Canonical B Instruction:Use one arm to take the target object in the 2nd row counted from far to near at the 1st position counted from rightto left. Canonical Position Indexing Task name: Pick Tea Box Relational Instruction:Front is the side farther from the robot. Grab the nearest one to the front-right of the center tea box using one arm. Referential Relational Reasoning Task name: Pick Cans Relational Instruction:The side farther from the robot is front. Use an arm to lift the nearest one directly at the back of the middle-left can. Referential Relational Reasoning Task name: Pick Soaps Relational Instruction:The side farther from the robot is front. Pick the nearest one directly in front of the back-right soap bar. Referential Relational Reasoning Task name: Pick Mixed Objects Relational A Instruction:Backmeansthesidenearertothe robot.Taketheclosestobjecttothefront-leftof the back-middle object. Referential Relational Reasoning Taskname:PickMixedObjectsRelationalB Instruction:Backmeansthesidenearertothe robot.Liftthenearestobjectdirectlyinfrontof themiddle-leftobject. Referential Relational Reasoning Task name: Pick Sauce Can Multi View Instruction:From the robot's side of the table, pick up the leftmost object. Cross-View Reasoning Figure 12: Fine-grained spatial reasoning task examples in RoboSPA (Part 2). 28 Task name: Pick Seals Multi View Instruction:Seen from the opposite side of the table, take the rightmost object with an arm. Cross-View Reasoning Task name: Pick Breads Multi View Instruction:Viewed from the right side, use one arm to pick up the leftmost object. Cross-View Reasoning Task name: Pick Mixed Objects Multi View A Instruction:From the left side of the table, pick up the frontmost object. Cross-View Reasoning Task name: Pick Mixed Objects Multi View B Instruction:With the camera on the opposite side of the table, grab the backmost object with an arm. Cross-View Reasoning Figure 13: Fine-grained spatial reasoning task examples in RoboSPA (Part 3). 29 Task name: Press Stapler Repeat Instruction:Press the blue stapler with shiny silver parts directly 5 times. Repetitive Procedure Following Task name: Lift Pot Repeat Instruction:Lift and put down the cooking kitchen pot with side grips with arms 5 times. Repetitive Procedure Following Task name: Lift Fan Repeat Instruction:Lift the white fan and set it down 5 times. Repetitive Procedure Following Task name: Click Bell Repeat Instruction:Press the top center of the compact tabletop bell 5 times. Repetitive Procedure Following Task name: Put Bottles Dustbin Instruction:Lift the five beverage bottles and drop them into the plastic dustbin. Order-Free Execution Figure 14: Long-horizon procedural planning task examples in RoboSPA (Part 1). 30 Task name: Move Blocks Apart Instruction:Set every magenta block on the right and every blue block on the left. Order-Free Execution Task name: Move Playing Cards Away Instruction:Move all boxes of playing cards to the left and right. Order-Free Execution Task name: Place Bowls Plates Instruction:Stack each medium-sized bowl for holding food one by one onto the dish plate. Order-Free Execution Task name: Separate Fries Bread Instruction:Place every bread at the front and every fries at the back. Order-Free Execution Task name: Rank Blocks Height Instruction:Order the blocks in height order, tallest left to shortest right. Order-Free Execution Figure 15: Long-horizon procedural planning task examples in RoboSPA (Part 2). 31 Task name: Rank Blocks Color Instruction:Order red, green, blue, yellow, and purple blocks from left to right. Order-Free Execution Task name: Rank Blocks Size Instruction:Sort the blocks by size from left to right, largest to smallest. Order-Free Execution Task name: Hit Blocks Hammer Order Instruction:Lift the grippy handle hammer using the right arm to hit the red, green, blue, yellow, then purple block. Order-Constrained Execution Task name: Click Objects Order Instruction:Press the rectangular alarm-clock, followed by the blue stapler, the red beverage can with curved top, the playingcards storage box, and the soft golden bread. Order-Constrained Execution Task name: Stamp Seals Order Instruction:Grab the brown seal, stamp on Olive, Coral, Teal, Tan, then Beige. Order-Constrained Execution Figure 16: Long-horizon procedural planning task examples in RoboSPA (Part 3). 32 Task name: Click Bell Clockwise Order Instruction:Click all five bells starting from the frontmost one and moving clockwise. Order-Constrained Execution Task name: Stack Blocks Color Order Instruction:Stack the blocks at the center from bottom to top as red, green, blue, yellow, and purple. Order-Constrained Execution Task name: Stack Blocks Size Order Instruction:Stack the blocks at the center from largest to smallest, bottom to top. Order-Constrained Execution Task name: Stack Blocks Length Order Instruction:Use the arms to build a center stack from longest to shortest, bottom to top. Order-Constrained Execution Task name: Place Phone Press Stapler Instruction:Place the sleek phone onthe flat base phone stand, press the curved black stapler thrice, then empty tabletop trash bin into the trash bin. Composite Action Coordination Figure 17: Long-horizon procedural planning task examples in RoboSPA (Part 4). 33 Task name: Place Bottle Cup Instruction:Move three bottles in the trash bin one after another, and place the two plastic cups for drinks on the two flat round wooden coasters one by one. Composite Action Coordination Task name: Hang Mug Stack Blocks Instruction:First hang the solid black ceramic mug on the metallic rack with smooth finish, then stack red block, green block, blue block, and yellow block in order from bottom to top. Composite Action Coordination Task name: Place Object Scale Click Instruction:Start by placing the bell with black flat base on the black and white electronic scale, then put the toy car made of plastic material on the smooth textured black display stand, and press the box with cards inside, then put the bell with black flat base back on the table, and put the toy car made of plastic material back on the table. Composite Action Coordination Task name: Place Burger Fries Click Bell Instruction:Deposit the orange hamburger with white underside on the orange tray, then deposit palm-sized red fries container on the orange tray, tap the plastic and metal bell once, then place both back. Composite Action Coordination Task name: Click Can Place Items Instruction:Press the brown can, move the bread block with light patches to the beige plastic breadbasket, click the brown can, set the bottle with large orange label on the pad, and finallypress the brown can. Composite Action Coordination Figure 18: Long-horizon procedural planning task examples in RoboSPA (Part 5). 34 Task name: Stamp Seals Press Stapler Instruction:Tap the left seal twice on left, tap the right seal twice on right, then press the stapler with curved blue top once. Composite Action Coordination Task name: Click Bell Open Microwave Place Object Instruction:Ring the plastic and metal bell, open the countertop microwave, ring the plastic and metal bell again, position the coffee-box with smooth card board left of the dark gray mouse, and ring the plastic and metal bell. Composite Action Coordination Task name: Observe Blocks Move Memory Instruction:Observe the colors of the blocks on the display platform, then after they are hidden, move matching blocks on the table closer to you. Memory-Intensive Planning Task name: Observe Objects Click Memory Instruction:Observe the objects on the display platform from left to right, then after they are hidden, click the matching objects on the table in the same order. Memory-Intensive Planning Task name: Remember Color Cover Instruction:Memorize the colors of the blocks, then once they are covered, uncover the blocks in the order of orange, black, green, magenta, red. Memory-Intensive Planning Figure 19: Long-horizon procedural planning task examples in RoboSPA (Part 6). 35 Task name: Remember Orientation Restore Instruction:Watch the front T-shaped blocks, then after they are hidden, make the rear T-shaped blocks match the front ones in orientation from left to right. Memory-Intensive Planning Task name: Press Stapler Memory Instruction:Press the five staplers in order moving left to right, 3, 5, 4, 1, and 2 times, respectively. Memory-Intensive Planning Figure 20: Long-horizon procedural planning task examples in RoboSPA (Part 7). 36