Paper deep dive
LAST: Leveraging Tools as Hints to Enhance Spatial Reasoning for Multimodal Large Language Models
Shi-Yu Tian, Zhi Zhou, Kun-Yang Yu, Ming Yang, Yang Chen, Ziqiao Shang, Lan-Zhe Guo, Yu-Feng Li
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 4/14/2026, 1:40:11 AM
Summary
LAST is a unified framework for tool-augmented spatial reasoning in Multimodal Large Language Models (MLLMs). It introduces 'LAST-Box', an extensible sandbox that abstracts heterogeneous vision tools into atomic instructions and reusable spatial skills. The framework employs a three-stage progressive training strategy—warm-up, supervised fine-tuning, and interactive reinforcement learning—to enable models to effectively invoke tools and interpret multimodal outputs, achieving significant performance gains on complex spatial reasoning tasks.
Entities (7)
Relation Signals (4)
LAST → includes → LAST-Box
confidence 100% · LAST features an extensible interactive sandbox, termed LAST-Box
LAST-Box → integrates → GroundingDINO
confidence 95% · the atomic operations integrated into LAST-Box comprise GroundingDINO
LAST-7B → uses → LAST-Box
confidence 95% · We release an open-source 7B multimodal model that integrates our sandbox
LAST → uses → GRPO
confidence 95% · We adopt GRPO (Shao et al., 2024) as the optimization algorithm
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Spatial reasoning is a cornerstone capability for intelligent systems to perceive and interact with the physical world. However, multimodal large language models (MLLMs) frequently suffer from hallucinations and imprecision when parsing complex geometric layouts. As data-driven scaling struggles to internalize structured geometric priors and spatial constraints, integrating mature, specialized vision models presents a compelling alternative. Despite its promise, applying this paradigm to spatial reasoning is hindered by two key challenges: The difficulty of invoking heterogeneous, parameter-rich tools, as well as the challenge of understanding and effectively leveraging their diverse low-level outputs (e.g., segmentation masks, depth maps) in high-level reasoning. To address these challenges, we propose LAST, a unified framework for tool-augmented spatial reasoning. LAST features an extensible interactive sandbox, termed LAST-Box, which abstracts heterogeneous tool invocations into atomic instructions and reusable spatial skills, returning multimodal hints (e.g., annotated images and textual descriptions) that can be directly consumed by LLMs. We further design a three-stage progressive training strategy that guides models from understanding tool outputs to proficient and adaptive tool invocation. Experiments on four datasets show that LAST-7B achieves around 20\% performance gains over its backbone and outperforms strong proprietary closed-source LLMs, substantially enhancing reasoning on complex spatial tasks.
Tags
Links
- Source: https://arxiv.org/abs/2604.09712v1
- Canonical: https://arxiv.org/abs/2604.09712v1
Trouble viewing inline? Open PDF directly →
Full Text
57,281 characters extracted from source content.
Expand or collapse full text
LAST: Leveraging Tools as Hints to Enhance Spatial Reasoning for Multimodal Large Language Models Shi-Yu Tian National Key Laboratory for Novel Software Technology, Nanjing UniversityChina , Zhi Zhou National Key Laboratory for Novel Software Technology, Nanjing UniversityChina , Kun-Yang Yu National Key Laboratory for Novel Software Technology, Nanjing UniversityChina , Ming Yang National Key Laboratory for Novel Software Technology, Nanjing UniversityChina , Yang Chen National Key Laboratory for Novel Software Technology, Nanjing UniversityChina , Ziqiao Shang National Key Laboratory for Novel Software Technology, Nanjing UniversityChina , Lan-Zhe Guo National Key Laboratory for Novel Software Technology, Nanjing UniversityChina and Yu-Feng Li National Key Laboratory for Novel Software Technology, Nanjing UniversityChina Abstract. Spatial reasoning is a cornerstone capability for intelligent systems to perceive and interact with the physical world. However, multimodal large language models (MLLMs) frequently suffer from hallucinations and imprecision when parsing complex geometric layouts. As data-driven scaling struggles to internalize structured geometric priors and spatial constraints, integrating mature, specialized vision models presents a compelling alternative. Despite its promise, applying this paradigm to spatial reasoning is hindered by two key challenges: the difficulty of invoking heterogeneous, parameter-rich tools, as well as the challenge of understanding and effectively leveraging their diverse low-level outputs (e.g., segmentation masks, depth maps) in high-level reasoning. To address these challenges, we propose LAST, a unified framework for tool-augmented spatial reasoning. LAST features an extensible interactive sandbox, termed LAST-Box, which abstracts heterogeneous tool invocations into atomic instructions and reusable spatial skills, returning multimodal hints (e.g., annotated images and textual descriptions) that can be directly consumed by LLMs. We further design a three-stage progressive training strategy that guides models from understanding tool outputs to proficient and adaptive tool invocation. Experiments on four datasets show that LAST-7B achieves around 20% performance gains over its backbone and outperforms strong proprietary closed-source LLMs, substantially enhancing reasoning on complex spatial tasks. spatial reasoning, multimodal large language models, tool-augmented learning 1. Introduction Spatial reasoning ability (Chen et al., 2024a; Yang et al., 2025) is a foundational capability for intelligent systems to perceive, plan, and execute tasks in the physical world. Whether in robot navigation (Hong et al., 2021), embodied AI interaction (Liu et al., 2025b), or complex 3D scene understanding (Li et al., 2025c), models are required not only to recognize objects but also to accurately parse their geometric layouts and topological relationships. Recently, multimodal large language models (MLLMs) have made significant strides in general visual understanding and semantic reasoning (Gupta and Kembhavi, 2023; Bai et al., 2025). However, these models often struggle with tasks involving precise spatial layouts, scale estimation, and fine-grained relationship reasoning. Existing research indicates that MLLMs frequently suffer from hallucinations regarding spatial relations (Li et al., 2025a) or fail to comprehend complex perspective and depth information (Chen et al., 2025a), which severely limits their potential in real-world physical scenarios and practical applications. To mitigate these limitations, existing studies predominantly rely on data-driven scaling, i.e., fine-tuning models with increasing amounts of spatially related data (Li et al., 2025b; Chen et al., 2024a; Li et al., 2025a). However, for spatial reasoning, these strategies often fail to enable models to truly internalize structured geometric priors and spatial constraints. In contrast, tool-augmented learning presents a more compelling alternative (Yao et al., 2023; Qin et al., 2024). Given that many fundamental spatial perception skills, such as depth estimation (Yang et al., 2024) and precise object localization (Liu et al., 2024), have already been effectively addressed by specialized vision models, a natural paradigm shift is to relieve MLLMs from relearning these mature capabilities and instead equip them with the ability to invoke expert tools. By integrating these highly optimized visual perception models, MLLMs can directly compensate for their perceptual deficiencies in complex spatial reasoning tasks. Bar chart comparing Acc@0.5 and Acc@0.7 localization accuracy of Qwen2.5-VL-3B, Qwen2.5-VL-7B, and GroundingDINO. GroundingDINO achieves the highest accuracy under both metrics. (a) Comparison of localization accuracy between MLLMs and specialized models. Light bars denote Acc@0.5 and dark bars denote Acc@0.7. Grouped bar chart showing performance changes from baseline when using text hints, image hints, or both hints across four spatial task types. (b) Effects of different categories of tool hints across diverse tasks. Figure 1. Visualization of preliminary experimental results for problem analysis. However, in the domain of spatial reasoning, this paradigm encounters two distinctive challenges. Firstly, spatial perception tools are heterogeneous and parameter-rich, requiring careful configuration at each step. Errors in tool invocation can easily accumulate over long reasoning horizons, making robust multi-step tool usage by MLLMs difficult. Secondly, perception tools produce diverse low-level outputs, such as segmentation masks or depth maps, which are not readily interpretable by MLLMs. As a result, models struggle to understand these outputs and convert them into high-level spatial cues required for complex reasoning. To address these challenges, we propose LAST (Leveraging Tools as Hints), a unified framework for tool-augmented spatial reasoning. LAST introduces an extensible interactive sandbox, the LAST-Box, which standardizes heterogeneous vision models into atomic instructions that can be composed into higher-level spatial skills. These skills shorten the tool invocation horizon and encapsulate low-level tool outputs into model-interpretable textual and visual cues, enabling MLLMs to acquire perceptual capabilities in an intuitive and controlled manner. To support effective tool interaction, we adopt a progressive curriculum-based training framework that guides models from understanding tool outputs to proficient tool invocation. Experiments on four datasets show that LAST-7B trained under this framework achieves around 20% performance gains over the backbone and outperforms strong proprietary closed-source LLMs on complex spatial reasoning tasks. Our main contributions are summarized as follows: • Atomic-to-Skill Tool Abstraction. We introduce a skill abstraction framework that composes atomic tool instructions into reusable spatial skills, substantially shortening long-horizon tool-calling trajectories. • Progressive Tool-Aware Training. We propose a progressive training paradigm with a tool-aware warm-up stage, enabling models to better internalize tool semantics before learning complex instruction following. • Open-Source 7B Model with Strong Performance. We release an open-source 7B multimodal model that integrates our sandbox and training paradigm, achieving state-of-the-art performance on multiple spatial reasoning benchmarks. 2. Preliminary Analysis Observation 1: MLLMs still fall short of specialized models in fine-grained perception. To examine the differences between general-purpose MLLMs and specialized vision systems in foundational spatial reasoning, we conduct a comparative study on the grounding subtask. We sample 200 instances from the SpatialLadder training set and benchmark the open-source Qwen2.5-VL series against the proprietary GroundingDINO. As shown in Fig. 1(a), although scaling MLLMs leads to gradual improvements under both relaxed (Acc@0.5) and strict (Acc@0.7) metrics, their performance remains notably inferior to GroundingDINO, especially at higher localization precision. Under the strict IoU threshold of 0.7, GroundingDINO achieves 44.2% accuracy, whereas Qwen2.5-VL-7B attains only 33.0%, revealing an 11.2% performance gap. Moreover, the limited gain from scaling the model from 3B to 7B (only +1.5% in Acc@0.7) indicates that parameter scaling alone is insufficient for MLLMs to acquire precise geometric grounding. These results highlight the persistent advantage of specialized vision models in fine-grained spatial perception and suggest the potential advantages of a tool-augmented paradigm that leverages their strengths. Observation 2: Even with tool-based prompts, MLLMs struggle to effectively recognize and exploit such guidance across diverse spatial tasks. Given the distinct perceptual advantages of specialized models, we further investigated whether directly incorporating their outputs as “hints” could consistently enhance the spatial reasoning capabilities of MLLMs. To explore this, we sampled 100 instances from each subset of SPbench, covering four distinct task categories: two involving multiple-choice reasoning (Relative Direction and Relative Distance) and two requiring numerical estimation (Absolute Distance and Size Estimation). We utilized expert models, specifically GroundingDINO and SAM, to preprocess the images, formatting their predictions into either textual descriptions (text-hint) or visual overlays (image-hint). These tool-augmented inputs were then fed into the model alongside the original images for evaluation. Architecture diagram of LAST-Box showing two layers: atomic operations (GroundingDINO, SAM, VGGT, DepthAnything, Python, Owlv2) at the bottom, and spatial skills (EstimateSize, EstimateDepth) at the top, with example inputs and outputs. Figure 2. Overview of the proposed LAST-Box As illustrated in Fig. 1(b), the effectiveness of tool hints is highly dependent on the task nature. For tasks that are easily articulated in language, such as Relative Direction, textual hints provide a clear performance boost. Conversely, for tasks where textual description is inherently limited—such as metric estimation (Absolute Distance and Size Estimation)—visual image hints demonstrate a decisive advantage. Surprisingly, when both modalities are provided simultaneously (“both-hint”), the performance trend largely mirrors that of using text hints alone but consistently yields even lower scores. Instead of benefiting from richer information, the models appear overwhelmed, with the combined inputs causing severe performance degradation (e.g., dropping below -10 points in Size Estimation). This indicates that without specific training, MLLMs struggle to effectively process tool-augmented prompts, often treating the additional multimodal information as distractive noise rather than a reasoning aid. 3. Methods We propose LAST, which consists of a structured visual reasoning sandbox (LAST-Box) and a progressive training strategy to enable efficient and controllable tool-augmented spatial reasoning for MLLMs. 3.1. LAST-Box We construct a comprehensive and extensible visual reasoning toolbox, denoted as LAST-Box, designed to support controlled tool invocation and structured result parsing for MLLMs in complex spatial reasoning tasks. Unlike prior work that directly exposes low-level model interfaces, LAST-Box atomizes underlying model invocations into primitive skills and further composes them into higher-level spatial skills, allowing MLLMs to directly utilize well-formed skills and significantly shorten tool-calling trajectories, leading to more efficient and stable learning. An overview of the LAST-Box architecture is illustrated in Figure 2. Atomic Operations. In LAST-Box, we define a universal set of foundational modules ℳ=m1,m2,…,mKM=\m_1,m_2,…,m_K\. Each invocation of a module is treated as an atomic operation o∈o . Formally, an atomic operation is defined as a deterministic mapping: (1) o:o→o:X_o _o where the input space =ℐ∪X=I encompasses raw images ℐI and textual prompts T. The output space Y consists of structured symbolic representations, such as segmentation masks, bounding box coordinates, or depth matrices. Specifically, the atomic operations integrated into LAST-Box comprise GroundingDINO (Liu et al., 2024) for object detection and text–region alignment, DepthAnything-v2 (Yang et al., 2024) for monocular relative depth estimation, Segment Anything (SAM) (Kirillov et al., 2023) for high-precision instance segmentation, VGGT (Wang et al., 2025) for 3D reconstruction, as well as Python utilities that support numerical computation and rule-based reasoning. Spatial Skills. On top of atomic operations, we introduce Skills s∈s as an intermediate abstraction layer. A skill s=⟨ojj=1m,ℛ⟩s= \o_j\_j=1^m,R is a functional composition where ℛR denotes the orchestration logic. While a skill shares the same input space X as atomic operations, its output space V is a high-level encapsulation of multimodal pairs: (2) s:→s,where s=⟨vi,ti⟩i=1ns:X _s, Y_s=\ v_i,t_i \_i=1^n Each element in V couples a visual output viv_i (visual output rendered via Python utilities, e.g., a cropped image patch) with its corresponding semantic text description tit_i. For a given input ∈x , the skill execution is: (3) s()=ℛ(oi1,oi2,…,oim;)s(x)=R(o_i_1,o_i_2,…,o_i_m;x) Diagram illustrating the three-stage progressive training strategy: Stage 1 Warmup with raw and augmented image pairs, Stage 2 SFT with multi-task spatial reasoning, and Stage 3 IRL with GRPO optimization over multiple rollouts. Figure 3. Illustration of the progressive training strategy. Where R represents the combination rule of atomic operations. This design yields large-model-friendly interfaces where each s:s→s:X_s _s provides a semantically explicit and functionally stable mapping, shielding the agent from the stochasticity and complexity of low-level model parameters. For example: • EstimateDepth is implemented by combining GroundingDINO, DepthAnything-v2 and Python. When this Skill is invoked in LAST-Box, the system returns a depth map overlaid with detection bounding boxes, together with the average relative depth of objects within each detected region. • EstimateSize integrates GroundingDINO, SAM and Python. GroundingDINO first performs coarse localization, followed by precise segmentation via SAM, while Python provides an image-based visualization. The Skill outputs cropped image patches containing accurate instance segmentation masks, facilitating downstream analysis or external applications. Based on the core spatial capability gaps identified in our preliminary analysis (Section 2), LAST-Box currently encapsulates six skills targeting essential spatial reasoning needs: SegmentObjects, EstimateDepth, EstimateSize, CountObjects, ZoomCrop, and Get3DPoint. These skills collectively cover object localization, depth perception, metric measurement, instance counting, local detail inspection, and 3D coordinate estimation. The skill set is designed to be extensible—new skills can be integrated by defining the corresponding atomic operation compositions and I/O formats. Detailed API specifications for each skill are provided in Appendix A. 3.2. Progressive Training Strategy Inspired by prior work (Li et al., 2025b), we adopt a progressive tool-scheduling training strategy that progressively teaches the model how to invoke tools and how to reason over tool outputs and intermediate results in a shallow-to-deep manner. Our training pipeline consists of three stages. An overall schematic diagram of the framework is shown in Figure 3. Stage 1: Warm-up. The objective of the warm-up stage is to familiarize the model with the formats and semantics of tool outputs, particularly image-based intermediate results such as segmentation masks and depth maps. We source 6,000 samples from the grounding subset of the SpatialLadder dataset. For each image, we utilize off-the-shelf vision tools to generate two types of augmented views: mask maps (highlighting specific objects) and depth maps (representing spatial geometry). We construct QA pairs that explicitly ask the model to distinguish between these views and interpret their semantic content, with ground-truth answers generated by Qwen-VL-Plus. This step ensures that when the model invokes a tool in later stages, it can correctly comprehend the visual feedback (e.g., understanding that a depth map represents distance rather than texture). Stage 2: Supervised Fine-Tuning (SFT). In the second stage, we aim to establish a basic understanding of when and how to invoke tools. We compile a dataset of approximately 16,000 samples, consisting of 6,000 single-image samples from the SpatialLadder (Li et al., 2025b) training set and 10,000 samples from the MSMU (Chen et al., 2025a) dataset. Using a state-of-the-art closed-source LLM (GPT-4o) as a teacher, we rewrite the original ground-truth labels into reasoning trajectories that follow a Think-Act-Observe-Reason cycle, where the model decides to call a tool, analyzes the returned result, and derives the final answer. To prevent the model from over-relying on tool outputs, we introduce a consistency-check mechanism during data construction. For example, when a question asks for the spatial relationship between a refrigerator and a sofa, but GroundingDINO detects only one of the two objects, the tool’s output is considered partially correct. In such cases, the model is explicitly required to combine the original image with the incomplete tool output, rather than directly trusting the tool result. Furthermore, in approximately 3,000 samples, we simulate tool failures (e.g., empty returns or execution errors). In these scenarios, the supervision signal encourages the model to abandon the tool path and revert to the original image for reasoning, thereby establishing a robust fallback mechanism. We optimize the model by next-token prediction loss. (4) ℒSFT(θ)=−1T∑t=1Tlogpθ(xt∣x<t).L_SFT(θ)=- 1T _t=1^T p_θ\! (x_t x_<t ). Stage 3: Interactive Reinforcement Learning (IRL). In the final stage, we further enhance the model’s capabilities in tool invocation decision-making and tool output interpretation through reinforcement learning. We construct a dataset of 2,000 samples using the same source distribution as Stage 2. Unlike the SFT stage, these samples contain only the problem prompts without reasoning paths, forcing the model to explore and generate its own action trajectories during training. We adopt GRPO (Shao et al., 2024) as the optimization algorithm and design a reward function composed of three components. (5) ℒGRPO(θ)=q,oii=1G[1G∑i=1G(min(ri(θ)A^i, _GRPO(θ)=E_q,\o_i\_i=1^G [ 1G _i=1^G ( (r_i(θ)\, A_i, . clip(ri(θ),1−ϵ,1+ϵ)A^i)−βKL(πθ∥πref))] .clip (r_i(θ),1-ε,1+ε ) A_i )- _KL( _θ\,\|\, _ref) ) ] where ri(θ)=πθ(oi∣q)πθold(oi∣q)r_i(θ)= _θ(o_i q) _ _old(o_i q) and the advantage A^i A_i for each output oio_i in the group is computed as: A^i=ri−μrσr A_i= r_i- _r _r. Format reward, which is used to enforce whether the model outputs conform to the expected structure. rformat=0if ∀t∈,count(tstart)==count(tend)−1otherwiser_format= cases0&if ∀ t ,count(t_start)==count(t_end)\\ -1&otherwise cases where T denotes the set of tag types, and count(tstart)count(t_start) and count(tend)count(t_end) represent the numbers of opening and closing tags of type t, respectively (e.g., <action></action>). Correctness reward, which measures the consistency between the predicted answer and the ground-truth label: rcorrect=(y^=y)for discrete tasksexp(−α|y^−y|)for numerical tasksr_correct= casesI( y=y)&for discrete tasks\\ (-α| y-y|)&for numerical tasks cases Here, y y and y denote the predicted and ground-truth answers, respectively, and α controls the sensitivity to numerical prediction errors. Tool reward. This reward evaluates whether the model invokes tools appropriately and whether the tool outputs are useful for task completion: rtool=(c∈success)(y^=y)r_tool=I\! (c _success )I( y=y) where c denotes the tool call issued by the model, successC_success is the set of successful tool calls, and (y^=y)I( y=y) represents the final result being correct. The final reward is defined as a linear combination of these three components. The hyperparameter λ balances the three reward terms. rall=λ1rformat+λ2rcorrect+λ3rtoolr_all= _1r_format+ _2r_correct+ _3r_tool Table 1. Comparison between our LAST-7B and other baseline models. Numerical estimation tasks are highlighted with a pink background. Bold indicates the best performance in each category, while underline denotes the second best. Models CVbench SPBench-SI EmbSpatial MSMU Avg 2D-REL 3D-Depth 3D-Dist Rel_Dir Rel_dist Size_Est Abs_Dist General Models Qwen2.5-VL-3B 72.46 79.00 65.00 27.45 67.39 18.61 10.67 56.29 8.67 45.06 LLAVA-NEXT-8B 70.62 66.67 60.30 23.43 72.34 27.63 15.34 50.30 10.63 44.14 LLAVA-OneVision-8B 78.23 78.23 75.77 45.27 50.63 35.70 25.00 56.20 14.00 51.00 InternVL3-8B 73.85 83.17 67.83 32.03 23.08 24.62 20.13 58.98 13.44 44.13 InternVL2.5-8B 70.63 78.15 71.33 30.42 40.29 30.63 18.56 50.82 12.20 44.78 Spatial VLMs SpaceLLaVA-13B 62.03 68.23 63.16 40.46 55.37 20.33 16.67 67.40 15.67 45.48 Robopoint-13B 75.13 80.33 74.23 50.26 61.23 25.83 15.30 60.21 17.28 51.09 VILASR-7B 80.15 94.83 88.33 64.05 53.85 37.58 4.70 69.81 20.63 57.10 Closed Source APIs GPT-5.1 60.60 82.93 76.76 51.31 64.84 46.87 8.83 75.13 18.69 54.00 Gemini-3.0-flash 58.64 68.90 73.20 48.04 73.63 50.54 14.09 63.27 25.07 52.82 Gemini-2.5-pro 67.48 79.41 80.35 49.67 71.43 52.27 14.77 77.27 27.52 57.80 Ours Qwen2.5-VL-7B (Backbone) 81.15 80.00 73.67 39.22 78.26 41.99 20.00 69.70 14.20 55.35 LAST-7B 93.33 89.54 90.64 70.26 80.22 74.95 41.61 81.00 41.17 73.64 Improvement +12.18 +9.54 +16.97 +31.04 +1.96 +32.96 +21.61 +11.30 +26.97 +18.28 4. Experiment Four qualitative examples showing LAST-7B reasoning traces with tool calls for relative depth, spatial relationship, size estimation, and distance tasks, each with thinking steps, tool invocations, and final answers. Figure 4. Representative experimental examples of LAST-7B from CVBench, EmbSpatial, and MSMU. 4.1. Setup Datasets. We conduct experiments on four spatial reasoning benchmarks. CVBench (Tong et al., 2024) evaluates 2D visual recognition and 3D spatial perception across diverse visual tasks. EmbSpatial (Du et al., 2024) focuses on egocentric spatial understanding in embodied 3D environments. SPBench-SI (Li et al., 2025b) assesses single-image spatial reasoning including object localization, relative distance estimation, and spatial relation understanding. MSMU (Chen et al., 2025a) targets quantitative spatial perception, requiring models to output precise metric values such as physical dimensions and absolute distances. Further details of each benchmark are provided in Appendix B. The evaluated tasks fall into two main categories: multiple-choice questions and numerical fill-in-the-blank questions. For evaluation metrics, multiple-choice questions are assessed using answer accuracy. For scale and distance estimation tasks, a prediction is considered correct if the estimated value falls within an acceptable relative error margin r with respect to the ground truth, i.e., ans/gt∈[1−r,1+r]ans/gt∈[1-r,1+r]. In our experiments, we set r=0.25r=0.25. Baselines. We mainly compare three categories of multi-modal large language models: (1) General-purpose multimodal large language models (Qwen-2.5-vl-3B,7B (Bai et al., 2025), LLava-Next-8B (Zhang et al., 2024),LLava-Onevison-8B (Li et al., 2024a), Internvl-2.5-8B (Chen et al., 2024c) and Internvl-3-8B (Zhu et al., 2025)), (2) Specialized large models for spatial reasoning(SpaceLLava-13B (Chen et al., 2024a), Robopoint-13B (Yuan et al., 2024) and VILASR-7B (Wu et al., 2025)) (3) Closed-source state-of-the-art multimodal large model APIs (GPT-5.1, Gemini-3.0-flash and Gemini-2.5-Pro). Implementation Details. We chose Qwen-2.5-VL-7B as the backbone. Both the supervised fine-tuning (SFT) and reinforcement learning (RL) stages are implemented using the ms-swift (Zhao et al., 2025) framework. For SFT, we perform full parameter fine-tuning with a global batch size of 64 (via gradient accumulation) and a learning rate of 1×10−51× 10^-5 with a warmup ratio of 0.05 over 5 epochs. The maximum sequence length is set to 8,192 tokens. For RL, we further align the model using GRPO over 2 epochs with a learning rate of 1×10−71× 10^-7, employing a cosine decay scheduler (minimum learning rate ratio of 0.1) and a warmup ratio of 0.03. The reward weights for correctness, format, and tool usage are set to 1.01.0, 0.30.3, and 0.30.3, respectively. The KL divergence coefficient (β) is set to 0.01. During rollout, we produce G=4G=4 candidate completions per prompt with temperature 1.0, top-p of 0.9, and top-k of 50. 4.2. Main Results In this section, we conduct a comprehensive analysis of our proposed LAST and the trained model (LAST-7B), obtaining the following experimental findings. Finding 1. LAST-7B substantially outperforms its backbone with an average gain of nearly 20%, and even surpasses several closed-source API-based models. As evidenced by the comprehensive evaluation in Table 1, our proposed model achieves a new state-of-the-art (SOTA) performance, consistently outperforming all baseline methods across diverse spatial benchmarks. The comparison with the Qwen2.5-VL-7B backbone is particularly compelling; our method delivers a substantial average performance boost of 18.28 points (from 55.35 to 73.64), effectively transforming a strong general-purpose vision-language model into a high-precision spatial expert. This enhancement is most pronounced in tasks requiring rigorous geometric and metric quantification rather than mere semantic description. Specifically, we observe remarkable gains in Size Estimation (+32.96), Relative Direction (+31.04), and MSMU (+26.97). These results suggest that our approach successfully bridges the long-standing “metric gap” in multimodal LLMs, enabling the model to transition from coarse relative perception to fine-grained spatial measurement. Furthermore, the proposed model demonstrates superior scaling efficiency by significantly outperforming top-tier proprietary systems. Despite its 7B parameter size, it surpasses GPT-5.1 (Avg: 54.00) and Gemini-2.5-pro (Avg: 57.80) by margins of 19.64 and 15.84 points, respectively. Notably, our model maintains the highest or second-highest scores (indicated by bold and underline) in nearly every sub-metric, proving that specialized spatial alignment can yield reasoning capabilities that exceed even the most advanced closed-source large multimodal models. Finding 2. The compressed skill abstraction in LAST-Box effectively enhances the reasoning performance of closed-source models. To investigate the generalizability of our proposed interactive sandbox (LAST-Box), specifically whether it can empower closed-source models that cannot be fine-tuned, we applied the framework to the Gemini model family and evaluated it across diverse spatial tasks. As shown in Table 2, the results demonstrate that introducing external tool support yields significant performance gains in general. Specifically, for the more capable Gemini-2.5-Pro, tool augmentation achieved consistent improvements across all metrics, delivering remarkable gains of 18.00% and 14.90% on CVBench’s 2D relationship reasoning (2D-REL) and 3D distance estimation (3D-Dist) tasks, respectively. This evidence shows that when the base model possesses sufficient instruction-following and reasoning capabilities, it can effectively leverage the precise geometric signals provided by the sandbox to correct its intrinsic spatial perception biases. Although we observed a performance drop (-8.06%) for the smaller Gemini-3-Flash model on the 3D distance task, we attribute this degradation primarily to its limited reasoning capacity, which hampers robust generalization when multi-step spatial inference is required. In contrast, Gemini-3-Flash still exhibits substantial improvements of over 12% on perception-dominant tasks such as 2D-REL and 3D-Depth, where the demands on long-horizon reasoning are less pronounced. Overall, these results indicate that our framework can also improve the spatial reasoning performance of closed-source commercial models. Table 2. Performance changes of closed-source models after integrating LAST-Box. Models CVBench MSMU 2D-REL 3D-Depth 3D-Dist Gemini-3-flash 58.64 68.90 73.20 25.07 + LAST-Box 70.92 82.25 65.14 26.14 Δ (delta) 12.28↑ 13.35↑ 8.06↓ 1.07↑ Gemini-2.5-Pro 67.48 79.41 80.35 27.52 + LAST-Box 85.48 90.32 95.25 30.06 Δ (delta) 18.00↑ 10.91↑ 14.90↑ 2.54↑ Table 3. Tool success rate and accuracy under successful and unsuccessful tool usage across different datasets of LAST-7B. Dataset Tool SR Acc(w/uns) Acc(w/suc) Δ CVBench 88.67 87.65 91.51 ↑ 3.86 SPBench-SI 95.14 40.82 69.69 ↑ 28.87 EmbSpatial 99.67 60.00 80.74 ↑ 20.74 Finding 3. The progressive training paradigm successfully teaches LAST-7B to utilize tool outputs for improved reasoning. A critical question is whether LAST-7B can effectively exploit information returned by tools. To investigate this, we analyze tool usage across datasets using the Tool Success Rate (Tool SR), and compare final accuracy under successful versus unsuccessful tool execution (Acc(w/suc) vs. Acc(w/uns)), as shown in Table 3. LAST-7B achieves consistently high Tool SR across all evaluated benchmarks (88.67%–99.67%), indicating reliable tool invocation. More importantly, successful tool usage leads to substantial accuracy gains: on SPBench-SI and EmbSpatial, accuracy improves by approximately 29% (69.69% vs. 40.82%) and 20% (80.74% vs. 60.00%), respectively. Even on CVBench, where the backbone already performs well, successful tool execution still yields a meaningful +3.86% improvement. These consistent gains confirm that the model meaningfully incorporates tool outputs into its reasoning process rather than invoking them superficially. Two subfigures: (a) pie charts comparing tool usage distribution after SFT vs after RL, showing more balanced usage after RL; (b) bar chart showing multi-step tool invocation rates increase after RL across all benchmarks. Figure 5. Comparison of tool invocation behavior between After SFT (Stage 2) and After RL (Stage 3). (a) Top-4 tool usage distribution across benchmarks. (b) Multi-step invocation rate across benchmarks. Beyond tool success rates, we further investigate how RL training shapes tool invocation behavior compared to SFT alone (Figure 5). As shown in Fig. 5(a), RL training leads to a more balanced tool usage distribution across benchmarks, reducing the dominance of SegmentObjects (from 37.0% to 33.2%) while increasing the usage of EstimateSize (from 12.3% to 16.7%) and Get3DPoint (from 6.4% to 8.5%). Furthermore, as shown in Fig. 5(b), the RL model exhibits substantially higher multi-step tool invocation rates across all benchmarks, e.g., from 0.1% to 3.2% on CVBench. This indicates that RL training encourages the model to adaptively compose complementary tools for complex spatial reasoning tasks, rather than relying on a single tool call. 4.3. Ablation Studies Table 4. Ablation on training stages. Removing any stage leads to performance degradation. Settings CVBench SPBench-SI EmbSpatial MSMU Backbone 78.27 44.87 69.70 14.20 w/o Stage 1 (Warmup) 89.63 64.87 79.63 39.23 w/o Stage 2 (SFT) 79.30 48.26 65.45 20.63 w/o Stage 3 (RL) 90.37 65.20 80.67 38.74 LAST-7B 91.17 66.76 81.00 41.17 Ablation on Training Stages. To validate the effectiveness of each training stage, we conducted ablation studies as shown in Table 4. The results demonstrate that the full proposed model achieves superior performance across all benchmarks. Specifically, removing Stage 2 (SFT) leads to the most significant performance degradation, highlighting its critical role in establishing the model’s core capabilities. Furthermore, the absence of either Stage 1 (Warmup) or Stage 3 (RL) also results in consistent performance drops. This confirms that the initialization provided by Warmup and the refinement from RL are indispensable and complementary for achieving optimal performance. Ablation on Tool Hints. To study the role of tool hints, we compare three variants (Table 5): No-tool SFT trains with the same data but removes all tool outputs from the training trajectories; Text-only and Image-only use the fully trained model (LAST-7B) but mask one hint modality during inference, retaining only the textual or visual output respectively. The full model with both modalities consistently outperforms the no-tool variant across all benchmarks (e.g., +7.65 on CVBench, +2.77 on EmbSpatial). While single-modality hints can introduce noise on metric tasks like MSMU, fusing both modalities recovers and surpasses the baseline, confirming that text and image hints provide complementary spatial cues whose combination yields consistent improvements. 5. Related Work Spatial Reasoning. Spatial reasoning ability (Liu et al., 2023; Chen et al., 2024a; Yang et al., 2025) is commonly defined as the capacity to interpret geometric relationships between objects and their surrounding environment (Yang et al., 2025). This capability is fundamental to robotic perception and is widely regarded as a core building block of embodied intelligence (Liu et al., 2025b) and autonomous driving (Chen et al., 2024b). Early spatial reasoning efforts largely focused on single-modality multi-step inference, whereas modern tasks predominantly target vision-based spatial reasoning with substantially higher complexity than conventional visual recognition problems (see Yang et al., 2025; VSI-Bench). Despite recent advances in visual-language models on basic perception, they continue to struggle with multi-step reasoning, precise scale estimation (Chen et al., 2025a), and viewpoint transformation (Li et al., 2025a). Benchmarking has also evolved—from static single-image settings (e.g., BLINK (Fu et al., 2024), CVBench (Tong et al., 2024)) to dynamic multi-view reasoning scenarios (e.g., VSI-Bench (Yang et al., 2025), STI-Bench (Li et al., 2025c)). To bridge these gaps, the field has begun shifting from resource-intensive task-specific fine-tuning (Cheng et al., 2024) toward more generalizable approaches. Such approaches include progressive model training, integrating 3D-aware encodings (Xu et al., 2024), and modular systems that enable VLMs to invoke external computer vision tools to perform physical operations (Han et al., 2025; Chen et al., 2025b). Recent research interests have begun to shift towards spatial reasoning in path planning (Shang et al., 2026; Yang et al., 2026). Table 5. Ablation on tool hint modality. “No-tool SFT” trains with the same data but without any tool outputs. “Image-only” and “Text-only” restrict the hint modality during inference. Settings CVBench SPBench-SI EmbSpatial MSMU Backbone 78.27 44.87 69.70 14.20 No-tool SFT 83.52 63.81 78.23 39.73 Text-only hint 86.77 66.90 80.16 38.38 Image-only hint 86.24 65.11 79.56 39.21 LAST-7B (Full hint) 91.17 66.76 81.00 41.17 Tool-augmented Reasoning. A major research trend enhances LLMs by equipping them with external modules that supply complementary information. Typical examples integrate calculators (Nakano et al., 2021; Zhang et al., 2025a), code executors (Gao et al., 2023; Pan et al., 2023) and symbolic solvers (Li et al., 2024b; Tian et al., 2025b, a; Zhou et al., 2025b), leveraging their reliability to handle complex reasoning beyond the native capacity of language models (Tian et al., 2024). In the multimodal setting (Gupta and Kembhavi, 2023; Hu et al., 2024), tools are extended to visual operations such as cropping, masking, or adjusting image attributes (Zhang et al., 2025b; Zheng et al., 2025), sometimes coordinated through reinforcement learning for tool selection and sequencing (Liu et al., 2025a; Zhou et al., 2025a). Spatial reasoning marks another important direction. Text-based systems increasingly adopt logic engines or ASP solvers for multi-hop inference (Yang et al., 2023; Wang et al., 2024), while multimodal spatial reasoning (Chen et al., 2025b; Han et al., 2025) begins to exploit expert models for grounding and geometric analysis. However, existing approaches often depend on fixed prompting templates, limited interaction with tools, and mostly text-only outputs, restricting their flexibility and adaptability to dynamic environments. In contrast, our framework offers a unified tool suite capable of producing heterogeneous outputs, and our inductive design enables systematic generalization across tasks and novel settings. 6. Conclusion We propose LAST, a unified tool-augmented framework that enhances the spatial reasoning capabilities of MLLMs through structured tool interaction, including the LAST-Box that encapsulates heterogeneous vision backbone models into low-level atomic instructions and further composes them into short-horizon reusable spatial skills, effectively replacing long and error-prone tool-calling trajectories. These skills transform raw perceptual outputs into interpretable textual and visual hints that can be directly exploited by MLLMs. With a progressive curriculum training strategy, models learn to reliably understand and invoke spatial skills, achieving consistent improvements across four spatial reasoning benchmarks. References S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. (2025) Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. Cited by: §1, §4.1. B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Sadigh, L. Guibas, and F. Xia (2024a) Spatialvlm: endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 14455–14465. Cited by: §1, §1, §4.1, §5. L. Chen, P. Wu, K. Chitta, B. Jaeger, A. Geiger, and H. Li (2024b) End-to-end autonomous driving: challenges and frontiers. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §5. P. Chen, Y. Lou, S. Cao, J. Guo, L. Fan, Y. Wu, L. Yang, L. Ma, and J. Ye (2025a) SD-vlm: spatial measuring and understanding with depth-encoded vision-language models. arXiv preprint arXiv:2509.17664. Cited by: §B.1, §1, §3.2, §4.1, §5. S. Chen, M. A. Uy, C. H. Song, F. Ladhak, A. Murali, Q. Qu, S. Birchfield, V. Blukis, and J. Tremblay (2025b) SpaceTools: tool-augmented spatial reasoning via double interactive rl. arXiv preprint arXiv:2512.04069. Cited by: §5, §5. Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. (2024c) Internvl: scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 24185–24198. Cited by: §4.1. A. Cheng, H. Yin, Y. Fu, Q. Guo, R. Yang, J. Kautz, X. Wang, and S. Liu (2024) Spatialrgpt: grounded spatial reasoning in vision-language models. Advances in Neural Information Processing Systems 37, p. 135062–135093. Cited by: §5. M. Du, B. Wu, Z. Li, X. Huang, and Z. Wei (2024) Embspatial-bench: benchmarking spatial understanding for embodied tasks with large vision-language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), p. 346–355. Cited by: §B.1, §4.1. X. Fu, Y. Hu, B. Li, Y. Feng, H. Wang, X. Lin, D. Roth, N. A. Smith, W. Ma, and R. Krishna (2024) Blink: multimodal large language models can see but not perceive. In European Conference on Computer Vision, p. 148–166. Cited by: §5. L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig (2023) Pal: program-aided language models. In Proceedings of the 40th International Conference on Machine Learning, p. 10764–10799. Cited by: §5. T. Gupta and A. Kembhavi (2023) Visual programming: compositional visual reasoning without training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 14953–14962. Cited by: §1, §5. Y. Han, C. Chi, E. Zhou, S. Rong, J. An, P. Wang, Z. Wang, L. Sheng, and S. Zhang (2025) TIGeR: tool-integrated geometric reasoning in vision-language models for robotics. arXiv preprint arXiv:2510.07181. Cited by: §5, §5. Y. Hong, Q. Wu, Y. Qi, C. Rodriguez-Opazo, and S. Gould (2021) Vln bert: a recurrent vision-and-language bert for navigation. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, p. 1643–1653. Cited by: §1. Y. Hu, W. Shi, X. Fu, D. Roth, M. Ostendorf, L. Zettlemoyer, N. A. Smith, and R. Krishna (2024) Visual sketchpad: sketching as a visual chain of thought for multimodal language models. Advances in Neural Information Processing Systems 37, p. 139348–139379. Cited by: §5. A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W. Lo, P. Dollar, and R. Girshick (2023) Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 4015–4026. Cited by: §3.1. B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, P. Zhang, Y. Li, Z. Liu, et al. (2024a) Llava-onevision: easy visual task transfer. arXiv preprint arXiv:2408.03326. Cited by: §4.1. D. Li, H. Li, Z. Wang, Y. Yan, H. Zhang, S. Chen, G. Hou, S. Jiang, W. Zhang, Y. Shen, et al. (2025a) ViewSpatial-bench: evaluating multi-perspective spatial localization in vision-language models. arXiv preprint arXiv:2505.21500. Cited by: §1, §1, §5. H. Li, D. Li, Z. Wang, Y. Yan, H. Wu, W. Zhang, Y. Shen, W. Lu, J. Xiao, and Y. Zhuang (2025b) Spatialladder: progressive training for spatial reasoning in vision-language models. arXiv preprint arXiv:2510.08531. Cited by: §B.1, §1, §3.2, §3.2, §4.1. Y. Li, Y. Zhang, T. Lin, X. Liu, W. Cai, Z. Liu, and B. Zhao (2025c) Sti-bench: are mllms ready for precise spatial-temporal world understanding?. arXiv preprint arXiv:2503.23765. Cited by: §1, §5. Z. Li, Z. Zhou, Y. Yao, Y. Li, C. Cao, F. Yang, X. Zhang, and X. Ma (2024b) Neuro-symbolic data generation for math reasoning. arXiv preprint arXiv:2412.04857. Cited by: §5. F. Liu, G. Emerson, and N. Collier (2023) Visual spatial reasoning. Transactions of the Association for Computational Linguistics 11, p. 635–651. Cited by: §5. J. Liu, K. Xiong, P. Xia, Y. Zhou, H. Ji, L. Feng, S. Han, M. Ding, and H. Yao (2025a) Agent0-vl: exploring self-evolving agent for tool-integrated vision-language reasoning. arXiv preprint arXiv:2511.19900. Cited by: §5. S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, et al. (2024) Grounding dino: marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, p. 38–55. Cited by: §1, §3.1. Y. Liu, W. Chen, Y. Bai, X. Liang, G. Li, W. Gao, and L. Lin (2025b) Aligning cyber space with physical world: a comprehensive survey on embodied ai. IEEE/ASME Transactions on Mechatronics. Cited by: §1, §5. R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, et al. (2021) Webgpt: browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332. Cited by: §5. L. Pan, A. Albalak, X. Wang, and W. Y. Wang (2023) Logic-lm: empowering large language models with symbolic solvers for faithful logical reasoning. arXiv preprint arXiv:2305.12295. Cited by: §5. Y. Qin, S. Hu, Y. Lin, W. Chen, N. Ding, G. Cui, Z. Zeng, X. Zhou, Y. Huang, C. Xiao, et al. (2024) Tool learning with foundation models. ACM Computing Surveys 57 (4), p. 1–40. Cited by: §1. Z. Shang, L. Ge, Y. Chen, S. Tian, Z. Huang, W. Fu, Y. Li, and L. Guo (2026) MapTab: can mllms master constrained route planning?. arXiv preprint arXiv:2602.18600. Cited by: §5. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §3.2. S. Tian, Z. Zhou, W. Dong, K. Yu, M. Yang, Z. Cheng, L. Guo, and Y. Li (2025a) TabularMath: understanding math reasoning over tables with large language models. arXiv preprint arXiv:2505.19563. Cited by: §5. S. Tian, Z. Zhou, K. Yu, M. Yang, L. Jia, L. Guo, and Y. Li (2025b) VCSearch: bridging the gap between well-defined and ill-defined problems in mathematical reasoning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 12721–12742. Cited by: §5. S. Tian, H. Wei, Y. Wang, and L. Feng (2024) Crosel: cross selection of confident pseudo labels for partial-label learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 19479–19488. Cited by: §5. P. Tong, E. Brown, P. Wu, S. Woo, A. J. V. IYER, S. C. Akula, S. Yang, J. Yang, M. Middepogu, Z. Wang, et al. (2024) Cambrian-1: a fully open, vision-centric exploration of multimodal llms. Advances in Neural Information Processing Systems 37, p. 87310–87356. Cited by: §B.1, §4.1, §5. J. Wang, M. Chen, N. Karaev, A. Vedaldi, C. Rupprecht, and D. Novotny (2025) VGGT: visual geometry grounded transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 5294–5306. Cited by: §3.1. R. Wang, K. Sun, and J. Kuhn (2024) Dspy-based neural-symbolic pipeline to enhance spatial reasoning in llms. arXiv preprint arXiv:2411.18564. Cited by: §5. J. Wu, J. Guan, K. Feng, Q. Liu, S. Wu, L. Wang, W. Wu, and T. Tan (2025) Reinforcing spatial reasoning in vision-language models with interwoven thinking and visual drawing. arXiv preprint arXiv:2506.09965. Cited by: §4.1. R. Xu, X. Wang, T. Wang, Y. Chen, J. Pang, and D. Lin (2024) Pointllm: empowering large language models to understand point clouds. In European Conference on Computer Vision, p. 131–147. Cited by: §5. J. Yang, S. Yang, A. W. Gupta, R. Han, L. Fei-Fei, and S. Xie (2025) Thinking in space: how multimodal large language models see, remember, and recall spaces. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 10632–10643. Cited by: §1, §5. L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao (2024) Depth anything v2. External Links: 2406.09414 Cited by: §1, §3.1. M. Yang, Z. Zhou, S. Tian, K. Yu, L. Guo, and Y. Li (2026) NeSy-route: a neuro-symbolic benchmark for constrained route planning in remote sensing. arXiv preprint arXiv:2603.16307. Cited by: §5. Z. Yang, A. Ishay, and J. Lee (2023) Coupling large language models with logic programming for robust and general reasoning from text. arXiv preprint arXiv:2307.07696. Cited by: §5. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao (2023) ReAct: synergizing reasoning and acting in language models. In Proceedings of the 11th International Conference on Learning Representations, Cited by: §1. W. Yuan, J. Duan, V. Blukis, W. Pumacay, R. Krishna, A. Murali, A. Mousavian, and D. Fox (2024) RoboPoint: a vision-language model for spatial affordance prediction for robotics. arXiv preprint arXiv:2406.10721. Cited by: §4.1. S. Zhang, Y. Dong, J. Zhang, J. Kautz, B. Catanzaro, A. Tao, Q. Wu, Z. Yu, and G. Liu (2025a) Nemotron-research-tool-n1: exploring tool-using language models with reinforced reasoning. arXiv preprint arXiv:2505.00024. Cited by: §5. Y. Zhang, X. Lu, S. Yin, C. Fu, W. Chen, X. Hu, B. Wen, K. Jiang, C. Liu, T. Zhang, et al. (2025b) Thyme: think beyond images. arXiv preprint arXiv:2508.11630. Cited by: §5. Y. Zhang, B. Li, h. Liu, Y. j. Lee, L. Gui, D. Fu, J. Feng, Z. Liu, and C. Li (2024) LLaVA-next: a strong zero-shot video understanding model. Cited by: §4.1. Y. Zhao, J. Huang, J. Hu, X. Wang, Y. Mao, D. Zhang, Z. Jiang, Z. Wu, B. Ai, A. Wang, et al. (2025) Swift: a scalable lightweight infrastructure for fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 29733–29735. Cited by: §4.1. Z. Zheng, M. Yang, J. Hong, C. Zhao, G. Xu, L. Yang, C. Shen, and X. Yu (2025) DeepEyes: incentivizing” thinking with images” via reinforcement learning. arXiv preprint arXiv:2505.14362. Cited by: §5. Z. Zhou, D. Chen, Z. Ma, Z. Hu, M. Fu, S. Wang, Y. Wan, Z. Zhao, and R. Krishna (2025a) Reinforced visual perception with tools. arXiv preprint arXiv:2509.01656. Cited by: §5. Z. Zhou, K. Yu, S. Tian, X. Yang, J. Shi, P. Song, Y. Jin, L. Guo, and Y. Li (2025b) LawGPT: knowledge-guided data generation and its application to legal llm. arXiv preprint arXiv:2502.06572. Cited by: §5. J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, H. Tian, Y. Duan, W. Su, J. Shao, et al. (2025) Internvl3: exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479. Cited by: §4.1. Appendix A LAST-Box Details A.1. Spatial Perception Skills We provide a suite of atomic spatial perception tools integrated into our sandbox. Each tool exposes a standardized interface with explicit inputs and outputs, enabling seamless composition into higher-level spatial reasoning skills. (1) EstimateDepth Description:: Runs Depth-Anything v2 to obtain a dense depth matrix and computes the average depth for specified objects. Produces both a depth visualization and numerical depth measurements. Inputs:: •: img_path (str): Path to the input image. •: text_labels (list[str], optional): Object queries for depth aggregation, e.g., ["a person", "a frisbee"]. Outputs:: •: An image file path containing the depth visualization. •: A textual description reporting the average depth for each detected object (if text_labels is provided). Example:: EstimateDepth(img_path="image-0") EstimateDepth(img_path="image-0", text_labels=["a person", "a frisbee"]) (2) EstimateSize Description:: Performs open-vocabulary object detection using GroundingDINO and segments detected objects with SAM2. Produces dense masks and centroid coordinates for object size estimation. Inputs:: •: img_path (str): Path to the input image. •: text_labels (list[str]): Object queries to detect and segment. •: threshold (float, optional): Detection score threshold (default: 0.1). Outputs:: •: An image file path visualizing segmentation masks. •: A textual description containing centroid coordinates for each detected object. Example:: EstimateSize(img_path="image-0", text_labels=["lamp"]) EstimateSize(img_path="image-0", text_labels=["a person"], threshold=0.1) (3) SegmentObjects Description:: Detects and segments objects via GroundingDINO and SAM2 to support spatial relationship analysis. Inputs:: •: img_path (str): Path to the input image. •: text_labels (list[str]): Object queries to detect and segment. •: threshold (float, optional): Detection score threshold (default: 0.1). Outputs:: •: An image file path showing segmentation results with bounding boxes. •: A textual description reporting centroid coordinates of detected objects. Example:: SegmentObjects(img_path="image-0", text_labels=["a person", "a frisbee"]) (4) CountObjects Description:: Uses open-vocabulary detection with GroundingDINO to count multiple instances of queried objects and report their spatial locations. Inputs:: •: img_path (str): Path to the input image. •: text_labels (list[str]): Object categories to detect and count. •: threshold (float, optional): Detection score threshold (default: 0.1). Outputs:: •: An image file path with bounding boxes for all detected instances. •: A textual description reporting the count and centroid coordinates for each object type. Example:: CountObjects(img_path="image-0", text_labels=["table"]) (5) ZoomCrop Description:: Crops and optionally zooms into a region of interest (ROI) specified by a bounding box or by center and size. Inputs:: •: img_path (str): Path to the input image. •: box (list[float], optional): Bounding box [x1, y1, x2, y2] in pixels. •: center (list[float], optional): ROI center [cx, cy] in pixels. •: zoom_factor (float, optional): Zoom factor (default: 1.0). Outputs:: •: An image file path of the cropped region. •: A textual description containing crop metadata. Example:: ZoomCrop(img_path="image-0", box=[100, 200, 300, 400]) (6) Get3DPoint Description:: Combines GroundingDINO for object detection with VGGT (Visual Geometry Grounded Transformer) to estimate the 3D coordinates of objects relative to the camera coordinate system. Inputs:: •: img_path (str): Path to the input image. •: text_labels (list[str]): Object queries to locate in 3D space. Outputs:: •: An image file path showing 3D visualization. •: A textual description reporting [X, Y, Z] coordinates. Example:: Get3DPoint(img_path="image-0", text_labels=["cup"]) Appendix B Dataset Details B.1. Evaluation Datasets CVBench (Tong et al., 2024) is a vision-centric benchmark introduced in the Cambrian-1 study. It provides a comprehensive evaluation of Multimodal LLMs across diverse visual tasks, specifically assessing fundamental 2D visual recognition and 3D spatial perception capabilities to ensure robust visual grounding. EmbSpatial (Du et al., 2024) (EmbSpatial-Bench) focuses on spatial understanding within embodied AI scenarios. Unlike general visual benchmarks, it evaluates the model’s ability to interpret spatial relationships and constraints from an egocentric perspective in 3D environments, which is critical for embodied agents. SPBench-SI (Li et al., 2025b) is a subset of the Spatial Ladder evaluation framework designed for single-image spatial reasoning. It assesses the model’s proficiency in tasks such as object localization, relative distance estimation, and spatial relation understanding within static 2D images, serving as a foundational test for spatial intelligence. MSMU (Chen et al., 2025a) (Massive Spatial Measuring and Understanding) is a large-scale dataset specifically curated for quantitative spatial perception. Distinct from qualitative VQA tasks, MSMU requires the model to output precise metric values (e.g., physical dimensions or absolute distances), thereby rigorously testing the model’s capability in numerical spatial reasoning. Appendix C Prompts System Prompt for Tool-Integrated Spatial Reasoning Role & Goal: You are a helpful assistant specialized in spatial reasoning tasks. Your goal is to solve spatial reasoning problems by analyzing images and using available tools. Each tool processes the image and returns an augmented view or numerical data to assist your reasoning. Available Tools: • EstimateDepth(img_path, text_labels): Estimate depth information for objects. Returns a depth map and average depth values. • EstimateSize(img_path, text_labels): Estimate the size of objects by generating segmentation masks. Returns a mask image and centroid coordinates. • SegmentObjects(img_path, text_labels): Segment objects to obtain their masks and boundaries. Returns a segmentation image with bounding boxes. • CountObjects(img_path, text_labels): Detect and count multiple instances of specified objects. Returns a bounding box image and count statistics. • ZoomCrop(img_path, box): Crop and zoom into a specific region of interest defined by a bounding box. Returns the cropped image. • Get3DPoint(img_path, text_labels): Estimate the 3D coordinates [X, Y, Z] of objects relative to the camera. Returns a 3D visualization and coordinates. Format Requirements: Please carefully analyze the given image and question, then use the appropriate tools to solve the spatial reasoning problem step by step. • Your thinking process should be wrapped in <analy></analy> tags. • Tool calls should be wrapped in <action></action> tags. • Your final answer should be wrapped in <ans></ans> tags. Figure 6. The system prompt designed to guide the model in selecting and executing visual tools.