Paper deep dive
RoboAlign: Learning Test-Time Reasoning for Language-Action Alignment in Vision-Language-Action Models
Dongyoung Kim, Sumin Park, Woomin Song, Seungku Kim, Taeyoung Kim, Huiwon Jang, Jinwoo Shin, Jaehyung Kim, Younggyo Seo
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 3/26/2026, 2:26:45 AM
Summary
RoboAlign is a training framework for Multimodal Large Language Models (MLLMs) that improves Vision-Language-Action (VLA) model performance by aligning embodied reasoning with low-level action generation. It uses a two-stage process: supervised fine-tuning (SFT) to enable zero-shot action token prediction, followed by reinforcement learning (GRPO) to refine reasoning trajectories based on action accuracy rewards. The framework significantly outperforms SFT baselines on benchmarks like LIBERO, CALVIN, and real-world robotics tasks.
Entities (7)
Relation Signals (4)
RoboAlign â evaluatedon â LIBERO
confidence 99% ¡ achieves performance improvements... on LIBERO
RoboAlign â evaluatedon â CALVIN
confidence 99% ¡ achieves performance improvements... on CALVIN
RoboAlign â uses â GRPO
confidence 98% ¡ It then employs GRPO (Shao et al., 2024) to refine this reasoning process
RoboAlign â improves â VLA
confidence 95% ¡ RoboAlign that reliably improves VLA performance.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Improving embodied reasoning in multimodal-large-language models (MLLMs) is essential for building vision-language-action models (VLAs) on top of them to readily translate multimodal understanding into low-level actions. Accordingly, recent work has explored enhancing embodied reasoning in MLLMs through supervision of vision-question-answering type. However, these approaches have been reported to result in unstable VLA performance, often yielding only marginal or even negative gains. In this paper, we propose a more systematic MLLM training framework RoboAlign that reliably improves VLA performance. Our key idea is to sample action tokens via zero-shot natural language reasoning and refines this reasoning using reinforcement learning (RL) to improve action accuracy. As a result, RoboAlign bridges the modality gap between language and low-level actions in MLLMs, and facilitate knowledge transfer from MLLM to VLA. To validate the effectiveness of RoboAlign, we train VLAs by adding a diffusion-based action head on top of an MLLM backbone and evaluate them on major robotics benchmarks. Remarkably, by performing RL-based alignment after SFT using less than 1\% of the data, RoboAlign achieves performance improvements of 17.5\%, 18.9\%, and 106.6\% over SFT baselines on LIBERO, CALVIN, and real-world environments, respectively.
Tags
Links
- Source: https://arxiv.org/abs/2603.21341v1
- Canonical: https://arxiv.org/abs/2603.21341v1
Trouble viewing inline? Open PDF directly â
Full Text
67,656 characters extracted from source content.
Expand or collapse full text
RoboAlign: Learning Test-Time Reasoning for Language Action Alignment in Vision Language Action Models Dongyoung Kim 1 2 Sumin Park 1 Woomin Song 1 Seungku Kim 1 Taeyoung Kim 1 Huiwon Jang 1 2 Jinwoo Shin 1 2 Jaehyung Kim * 3 Younggyo Seo * 4 Abstract Improving embodied reasoning in multimodal- large-language models (MLLMs) is essential for building vision-language-action models (VLAs) on top of them to readily translate multimodal understanding into low-level actions. Accord- ingly, recent work has explored enhancing em- bodied reasoning in MLLMs through supervi- sion of vision-question-answering type. However, these approaches have been reported to result in unstable VLA performance, often yielding only marginal or even negative gains. In this paper, we propose a more systematic MLLM training framework ROBOALIGN that reliably improves VLA performance. Our key idea is to sample action tokens via zero-shot natural language rea- soning and refines this reasoning using reinforce- ment learning (RL) to improve action accuracy. As a result, ROBOALIGN bridges the modality gap between language and low-level actions in MLLMs, and facilitate knowledge transfer from MLLM to VLA. To validate the effectiveness of ROBOALIGN, we train VLAs by adding a diffusion-based action head on top of an MLLM backbone and evaluate them on major robotics benchmarks. Remarkably, by performing RL- based alignment after SFT using less than 1% of the data, ROBOALIGN achieves performance improvements of 17.5%, 18.9%, and 106.6% over SFT baselines on LIBERO, CALVIN, and real- world environments, respectively. 1. Introduction Visionâlanguageâaction models (VLAs) have recently demonstrated remarkable success in robotics (Brohan et al., 2022; 2023; Driess et al., 2023). By integrating visual * Equal advising. 1 KAIST 2 RLWRLD 3 Yonsei Univer- sity 4 UC Berkeley.Correspondence to: Dongyoung Kim <kingdy2002@kaist.ac.kr>. Preprint. March 24, 2026. perception, language understanding, and common-sense knowledge of multimodal-large-language models (MLLMs), VLAs provide a foundation for training generalizable robotic policies in real-world scenarios (Yang et al., 2023; Huang et al., 2022b; Tellex et al., 2020; Huang et al., 2022a; Hu et al., 2023). Specifically, policies are obtained either through discrete action token predictions by MLLMs (Kim et al., 2024; Pertsch et al., 2025; Kim et al., 2025b) or through continuous action prediction by external action ex- perts that operate on latent states of MLLMs (Black et al., 2024; Bjorck et al., 2025; Team et al., 2024). This ap- proach leverages the extensive pretrained knowledge within MLLMs, enabling the development of generalizable policies even with a limited amount of robotics data. However, the performance and generalization of VLAs are often limited by the underlying MLLMs, which struggle with key embodied tasks required for action generation, such as spatial reasoning (Tong et al., 2024; Zhou et al., 2025; Cheng et al., 2024) and temporal reasoning (Ahn et al., 2022; Sermanet et al., 2024). To address this limita- tion, researchers have developed various embodied question- answering tasks designed to improve reasoning skills for robotic manipulation. These include tasks such as answering high-level action questions (Chen et al., 2025; Lynch et al., 2023), responding to spatial questions about object relation- ships (Chen et al., 2024a; Xu et al., 2025), grounding points or bounding boxes in images to identify affordance-related locations (Yuan et al., 2024; Song et al., 2025a), and pre- dicting future visual trajectories of end-effectors (Ji et al., 2025; Yuan et al., 2025a). While supervised fine-tuning (SFT) has been the dominant paradigm for these tasks, re- cent approaches have leveraged reinforcement learning (RL) strategies (e.g., DeepSeek-R1; Guo et al. 2025) to elicit stronger reasoning capabilities, leading to significant per- formance improvements (Azzolini et al., 2025; Kim et al., 2025a; Song et al., 2025b; Huang et al., 2025a). Despite recent successes, improvements in embodied rea- soning do not consistently translate into corresponding gains in VLA performance. Notably, VLM4VLA (Zhang et al., 2026) revealed that the correlation between embodied rea- soning capability and VLA performance is inconsistent and 1 arXiv:2603.21341v1 [cs.AI] 22 Mar 2026 RoboAlign: Learning Test-Time Reasoning for Language-Action Alignment in Vision-Language-Action Models Base Model RoboBrain 2.0 VeBrainCosmos- Reason1 RoboAlign (Ours) 40 50 60 70 80 90 Average Score (%) 73.95 57.35 62.70 75.40 86.75 Figure 1. Performance on LIBERO. VLAs built upon MLLMs specialized for embodied reasoning (fine-tuned variants of Qwen2.5-VL-7B-Instruct) fail to significantly improve perfor- mance and often degrade it compared to the baseline VLA based on the original model. In contrast, ROBOALIGN achieves signifi- cant gains, as detailed in Section 5. highly task-dependent, sometimes even leading to perfor- mance degradation. To further support this observation, we conducted additional experiments by training VLAs on top of open-source MLLMs specialized in embodied reasoning and observed similar trends (see Figure 1). Surprisingly, although RoboBrain 2.0 (Team et al., 2025) achieved the highest reasoning scores among evaluated MLLMs and even outperformed GPT-4o (Hurst et al., 2024) on major bench- marks (see Table 9), it yielded the lowest VLA performance (see Figure 1). We attribute this discrepancy to the modality gap between language and low-level actions; optimizing em- bodied reasoning purely through language supervision does not guarantee improvements in actual action generation. Contribution. Motivated by this insight, we introduce ROBOALIGN, an MLLM training framework that reliably improves VLA performance. The core idea of ROBOALIGN lies in generating low-level action tokens as a direct outcome of embodied reasoning and evaluating reasoning quality via action accuracy; this approach allows us to directly align the reasoning capability of MLLMs to VLAs through RL- based fine-tuning. Specifically, ROBOALIGN first applies SFT to enable the MLLM to generate low-level actions via zero-shot reasoning. It then employs GRPO (Shao et al., 2024) to refine this reasoning process by maximizing an action-accuracy reward. This approach allows the model to explore diverse reasoning trajectories through sampling and align them toward precise action execution. To evaluate the effectiveness of ROBOALIGN, we train MLLMs with our framework and test the performance on a suite of robotic benchmarks, including simulation envi- ronments such as LIBERO (Liu et al., 2023) and CALVIN (Mees et al., 2022), as well as real-world robot settings. Specifically, we attach a diffusion-based action head to the frozen MLLM backbone and fine-tune it to generate low- level actions. Our experiments show that models trained with ROBOALIGN achieve substantial performance gains over baseline SFT-only models, with relative improvements of 17.5% on LIBERO, 18.9% on CALVIN, and 106.6% in the real-world setup, while using less than 1% additional data for the subsequent RL-based alignment stage on top of SFT. Moreover, we find that our approach is more effective than other alignment approaches such as high-level action prediction (13.1% v.s. 17.5%) or point trajectory prediction (15.2% v.s. 17.5%) on the LIBERO, respectively. Furthermore, to examine if ROBOALIGN also improves embodied reasoning capabilities of MLLMs, we evaluated ROBOALIGN on a diverse set of benchmarks for general image understanding (Chen et al., 2024b), spatial reason- ing (Song et al., 2025a; Yuan et al., 2024; Fu et al., 2024), and embodied reasoning for robotics (Kim et al., 2025a). On the embodied reasoning tasks, ROBOALIGN achieve state-of-the-art performance, outperforming not only com- mercial general-purpose models such as GPT-4o (OpenAI, 2024), but also specialized embodied MLLMs, such as RoboBrain2.0 (Team et al., 2025). Notably, this is ac- complished while preserving the modelâs performance on general image understanding. This result shows that our RL-based alignment enhances the general capabilities of MLLMs, in contrast to SFT-based alignment methods such as ECoT (Zawalski et al., 2024), which often degrades per- formance on these embodied tasks. 2. Related Work Multimodal-large-language models for robot control. Ef- forts to leverage the visual processing capabilities, common- sense, and world knowledge of multimodal-large-language models (MLLMs) for robot policy decision have shown consistent success. In particular, MLLMs have demon- strated strong performance in high-level action planning. Concretely, prior work has explored generating predefined atomic action skills to directly control robots (Liang et al., 2023; Tellex et al., 2020; Luo et al., 2025), or producing high-level actions and plans that condition subsequent low- level actions (Driess et al., 2023; Yang et al., 2023; Huang et al., 2022b;a; Hu et al., 2023). These approaches have been further extended toward more precise action genera- tion, either by enabling MLLMs to produce policies in an end-to-end manner (Kim et al., 2024; Pertsch et al., 2025; Kim et al., 2025b) or by training action experts that con- sume latent states instead of language outputs (Team et al., 2024; Li et al., 2023; Shentu et al., 2024; Black et al., 2024; Bjorck et al., 2025; GEAR, 2025). We investigate how to better align MLLMs with low-level actions to enhance such robot control performance. Multimodal-large-language model for embodied reason- ing. With the increasing application of MLLMs to embodied 2 RoboAlign: Learning Test-Time Reasoning for Language-Action Alignment in Vision-Language-Action Models environments such as robot manipulation, their capabilities for tasks requiring spatial and temporal reasoning have been enhanced. For spatial reasoning, prior work has enhanced 3D scene understanding by leveraging VQA data to train models that convert information from 2D and 3D vision inputs (Chen et al., 2024a; Ray et al., 2024; Zhou et al., 2025; Wu et al., 2025). To further improve performance in specific robotic tasks, some approaches have trained models to predict bounding boxes or points associated with affor- dances and manipulation-relevant spatial cues (Yuan et al., 2024; Song et al., 2025a; Lu et al., 2023; Ji et al., 2025). For temporal reasoning, researchers have extracted high- level actions (Chen et al., 2025; Lynch et al., 2023; Chen et al., 2025; Huang et al., 2024; Chen et al., 2023), 2D point trajectories of object movement from egocentric videos of humans or robots to construct VQA (Huang et al., 2025a; Yang et al., 2025; Ranasinghe et al., 2024; Zheng et al., 2024; Lee et al., 2025). Nevertheless, these approaches primarily provide indirect supervision signals and do not directly optimize low-level action generation. Encouraging reasoning through reinforcement learning. Chain-of-Thought (CoT) prompting (Wang et al., 2022; Yao et al., 2023; Kim et al., 2023; Wei et al., 2022) has been widely applied to both LLMs and MLLMs in zero-shot, few-shot, and supervised fine-tuning (SFT) settings (Muen- nighoff et al., 2025), effectively improving answer quality. Recently, DeepSeek-R1 (Guo et al., 2025) proposed a train- ing approach specialized for CoT, in which reasoning is explicitly enforced during the response process, and the entire reasoning trace is optimized using the reinforcement learning algorithm with rewards derived from the final an- swer. This training paradigm has demonstrated that, com- pared to SFT, models can achieve stronger performance and generalization across diverse domains, including math- ematics (Zeng et al., 2025; Yu et al., 2025), agents (Lu et al., 2025; Jin et al., 2025), visions (Shen et al., 2025; Huang et al., 2025b;b), and embodied reasoning (Kim et al., 2025a; Song et al., 2025b; Huang et al., 2025a; Yuan et al., 2025a;b), while requiring significantly less data, in some cases even a single example (Wang et al., 2025). In this work, we introduce a reinforcement learning scheme based on low-level action prediction, aligning the MLLMâs repre- sentations more directly with robot control. 3. Preliminaries FAST action tokenization. We adopt FAST tokeniza- tion (Pertsch et al., 2025) to integrate low-level actions into multimodal-large-language models (MLLMs), as it has been shown to be effective not only for end-to- end policy learning but also for representation learn- ing (Black et al., 2025; Driess et al., 2025).Each action is defined as aD-dimensional vector represent- ing the end-effectorâs state, which consists of its Carte- sian position(x,y,z), orientation(roll, pitch, yaw), and gripper state (Open/Close).An action sequence over a horizon ofHtimesteps forms a chunk,a 1:H = [[a 1,1 ,a 1,2 ,...,a 1,D ],..., [a H,1 ,a H,2 ,...,a H,D ]]. To im- prove compactness, FAST tokenization transforms the ac- tion chunka 1:H into the frequency domain using a discrete cosine transform (DCT; Ahmed et al. 2006). The result- ing DCT coefficients are quantized and flattened into a se- quence. This sequence is then compressed into discrete tokens using byte-pair encoding (BPE; Gage 1994), result- ing inT k = FAST(a 1:H ), where each token is mapped to one of2Kspecial tokens added to the MLLMâs vocabulary for training and generation. Encouraging reasoning with GRPO. To encourage ex- plicit reasoning, we train the model to generate intermediate thoughts enclosed within<think>...</think>before producing a final answer. Training is conducted with Group Relative Policy Optimization (GRPO; Shao et al. 2024), where the policy is optimized jointly for format correctness and answer accuracy. Specifically, let the current policy be denoted asĎ Î¸ old . For a given queryq âź P(Q), we sam- pleGresponses[o 1 ,...,o G ] âź Ď Î¸ old (q). Each response is evaluated by a pre-defined reward modelR(q,o i ), which assigns a rewardr i based on format and answer accuracy. We then compute an advantage by normalizing the reward using the standard deviation,A i = r i âmean(r) std(r) , and define the importance sampling ratio asĎ i (θ) = Ď Î¸ (o i |q) Ď Î¸ old (o i |q) . For each queryq âź P(Q), we sampleGresponseso i G i=1 from the old policyĎ Î¸ old . GRPO optimizes the policy by maximizing these advantages while applying a KL penalty against a reference policy: J GRPO (θ) = E h 1 G G X i=1 min Ď i A i , clip(Ď i , 1âÎľ, 1+Îľ)A i â βD KL (Ď Î¸ âĽĎ ref ) i , (1) whereÎľandβare hyperparameters for clipping and KL penalty. 4. ROBOALIGN: Align Embodied Reasoning with Low-level Actions In this section, we introduce ROBOALIGN, a training frame- work that directly aligns multimodal-large-language mod- els (MLLMs) with low-level actions through reinforcement learning (RL). While doing so, ROBOALIGN is designed to preserve the general capabilities of MLLMs and simultane- ously enhance embodied reasoning ability. A key challenge, however, is that off-the-shelf MLLMs cannot generate spe- cialized low-level actions (e.g., FAST tokens) in a zero-shot manner, making RL inapplicable. To address this, we intro- duce a two-stage training pipeline. First, we apply super- vised fine-tuning (SFT) to equip the model with the initial ability to predict FAST tokens through zero-shot reason- ing, while preserving the general abilities of MLLMs and 3 RoboAlign: Learning Test-Time Reasoning for Language-Action Alignment in Vision-Language-Action Models Response 1 Stage 2: Alignment via GRPO w/ Low-Level ActionsStage 1: Alignment via Robotics-Aware SFT SampleMultiple Responses Response 2 <think>To close the fridge, the robot has to ... </think> <answer> <|action_start|> <|a486|><|a266|><|a292|> <|a320|><|a552|> <|action_end|> </answer> Response 3 Compute Reward Optimize Policy <|a486|><|a266|><|a292|><|a320|><|a552|> <|a486|><|a266|><|a268|><|a116|><|a552|> Prediction: GT: 2 out of 5 initial FAST action tokens are correct Ă Reward: 0.4 Q: What are the normalized bounding box coordinates of the left cabinet? A: The normalized bounding box coordinates for the left cabinet are [0.350, 0.100, 0.600, 0.400]. RoboAlign VQA Chain-of-Thought Reasoning Data Q: What high-level action is the robot performing right now for 'sweep into pileâ? Options: (A)... A: <think> The action "sweep into pile" typically... </think><answer>(C) Close gripper</answer> Instruction : Close the fridge. Q: Move the purple cloth to the middle lower side of the table. A: <|action_start|><|a266|> <|a458|><|a268|><|a116|> <|a619|><|a689|><|action_end|> Low-Level Action Token Generation FAST Action Accuracy Reward Policy Model Figure 2. Overview of ROBOALIGN framework. ROBOALIGN directly aligns MLLM representations with low-level action generation using reasoning-incentivized reinforcement learning (Guo et al., 2025). The framework consists of two stages: (i) Stage 1 integrates embodied reasoning, zero-shot reasoning, and FAST-tokenized low-level action generation via supervised fine-tuning, and (i) Stage 2 optimizes responses through reinforcement learning to improve token-level action accuracy and better alignment. The resulting model serves as an MLLM tailored for effective VLA training. enhancing embodied reasoning. Second, building on this ability, we apply RL on this SFT model to further strengthen embodied reasoning and improve FAST token prediction accuracy. The overall process is illustrated in Figure 2. 4.1. Stage 1: Integrating Low-level Action with MLLM using SFT The primary objective of this SFT stage is to equip the MLLM with the ability to generate FAST action tokens, which is a prerequisite for the subsequent RL stage, while simultaneously preserving its general vision-language ca- pabilities and enhancing its embodied reasoning skills. To achieve this, we curate a data mixture from four sources: (i) a variety of open-source SFT datasets for embodied reason- ing and general understanding, (i) our custom ROBOALIGN VQA dataset for robotic embodied reasoning, (i) special- ized reasoning datasets designed to improve zero-shot rea- soning of MLLMs, and (iv) robotic dataset with FAST to- kens. We describe the process for building our custom datasets in this section, with full details for all data sources and configurations available in Appendix A. ROBOALIGN VQA. While existing VQA datasets are use- ful for general embodied reasoning, high-quality VQA specifically grounded in robotic information remains limited. For example, datasets such as ShareRobot (Ji et al., 2025) and RoboVQA (Sermanet et al., 2024) use robot imagery but focus on high-level QA tasks, lacking the fine-grained, spatial-temporal information needed for low-level control. To address this gap, we develop a data generation pipeline that feeds robot images and associated metadata, e.g., bound- ing boxes, end-effector states, and both high and low-level actions, into a powerful large model, i.e.,gemini-2.5 pro(Hassabis et al., 2025). The model then automatically generates a diverse set of high-quality VQA, captioning, and grounding QA pairs. Reasoning dataset with zero-shot CoT. To preserve the MLLMâs zero-shot reasoning ability during SFT and trans- fer it to the action generation process, we incorporate a specialized reasoning dataset into our training mixture. This dataset is created by distilling outputs from a reasoning model that is trained with GRPO to generate step-by-step reasoning. Specifically, we first train the reasoning model on spatial and robot-related embodied MCQAs for distillation, following Kim et al. (2025a). For each prompt, we sam- ple multiple reasoning trajectories from this model. These outputs are then filtered using a combination of rule-based rewards and correctness checks. Table 1 shows that includ- ing this specialized reasoning data during SFT enables the effective transfer of reasoning ability to FAST token gen- eration, while the absence of such data results in limited zero-shot reasoning. FAST token generation dataset. To enable FAST to- 4 RoboAlign: Learning Test-Time Reasoning for Language-Action Alignment in Vision-Language-Action Models Table 1. Example of ROBOALIGN response. Incorporating reasoning data during SFT effectively transfers zero-shot reasoning ability to FAST token generation process. Without such data, the model produces only minimal reasoning which reduces the diversity of reasoning samples and hinders RL. Question: Your current task is âpick up cup from the tableâ. Output the robotâs actions to perform this task through FAST tokens. Model Answer (Trained w/ reasoning datasets): <think>To pick up the cup from the table, the robot needs to move its gripper towards the cup, position it correctly, close the gripper to secure the cup, and then lift it up. Given the current state of the gripper being open and positioned above the cup, the immediate next action would be to move down towards the cup to prepare for grasp- ing.</think><answer><|action_start|><|action_266|><|action_299|> ... Model Answer (Trained w/o reasoning datasets):<think>Go to the cup.</think> <an- swer><|action_start|><|action_266|><|action_299|>... ken prediction, we first extend the MLLMâs vocabulary by adding two special marker tokens<ACTION_START>, <ACTION_END>and 2K FAST tokens. The training data is then constructed from the BridgeV2 dataset (Walke et al., 2023) in a QA format. Each sample pairs a robot image with a fixed instruction, where the ground-truth answer is the cor- responding sequence of FAST tokens. The resulting data mixture, consisting of our custom and open-source datasets, is used to fine-tune the MLLM with SFT, providing a strong foundation for subsequent RL training stage. 4.2. Stage 2: Aligning Embodied Reasoning with Low-level Action via RL In the second stage, we use RL to directly align the MLLM with low-level actions, i.e., FAST tokens, further refining the model to be better suitable for VLA adaptation. Specifi- cally, we optimize the modelâs embodied reasoning process to directly improve the accuracy of FAST action token gen- eration. To create the data for this stage, we adapt the FAST token dataset from Stage 1. In particular, each sampleâs input instruction is augmented with a prompt that requires explicit reasoning within<think>...</think>tags before producing the FAST token sequence. We define the reward as the arithmetic mean of two compo- nents: a format rewardr f â0, 1indicating whether the output correctly adheres to the required reasoning format, and an accuracy rewardr a â [0, 1]measuring FAST token prediction accuracy. In particular, the accuracy rewardr a is computed by measuring the prefix similarity between the generated action token sequenceT gen 1:n and the target sequence T target 1:m , normalized by the target length: r a = 1 m maxiâ1,...,m : T gen 1:i = T target 1:i .(2) The final reward is given byr = (r f + r a )/2. This for- mulation encourages the model to generate both correctly formatted and accurate FAST token sequences. Building Figure 3. Examples of observations. Visual inputs for training and evaluation (from left to right): BridgeV2 for FAST token training, CALVIN, LIBERO benchmark, and the real-robot. on the constructed training dataset and reward function, we then apply GRPO (Shao et al., 2024) to further optimize the MLLM. 5. Experiment In this section, we design experiments to answer the follow- ing research questions: âŚDoes training MLLMs with ROBOALIGN consistently improve VLA performance across various scenarios? (See Table 2, 3, 4, 5 and Figure 4) ⌠Is ROBOALIGN more effective than alternative MLLMs training methods? (See Table 6, 7) ⌠How does ROBOALIGN contribute to VLA performance improvements? (See Table 8) 5.1. Experimental Setup Experiment design. To evaluate how different MLLM train- ing methods affect VLA performance, we convert MLLMs trained with various algorithms into VLAs using an iden- tical robot dataset and a unified VLA conversion pipeline, and then evaluate their robot control performance. Our VLA conversion pipeline is built upon the well-established VLA training framework Gr00t-N1.5 (GEAR, 2025). Concretely, we adapt diffusion-based action head on top of an MLLM backbone and train newly-initialized diffusion-based action head on robot datasets while keeping the MLLM backbone frozen. For each benchmark, we train VLAs from scratch using the training data provided by the benchmark. 5 RoboAlign: Learning Test-Time Reasoning for Language-Action Alignment in Vision-Language-Action Models Table 2. LIBERO success rates (%) for VLAs built upon MLLMs that were fine-tuned with various methods, evaluated over 500 trials per category. Each model is evaluated by training a newly-initialized, diffusion-based action head on the LIBERO dataset while the MLLM backbone remains frozen. ROBOALIGN shows particularly large improvements in the Long and Goal categories compared to other training methods. MethodFine-tuning Dataset SizeSpatialObjectGoalLongAvg. Reference VLA Performance Diffusion Policy-82.684.782.157.676.8 Octo (Team et al., 2024)-78.985.784.651.175.1 OpenVLA (Kim et al., 2024)-84.988.479.253.776.5 TraceVLA (Zheng et al., 2024)- 84.985.275.154.174.8 CoT-VLA (Zhao et al., 2025)-87.591.687.669.083.9 ThinkAct (Huang et al., 2025a)-88.391.487.170.984.4 Our Setup (Baselines and Ours) Qwen2.5VL-7B-Ins (Bai et al., 2025)-95.295.042.463.273.9 w/ Language-Only SFT1.88M 91.094.467.865.079.6 w/ Action-Only SFT1.88M89.895.882.857.681.5 w/ ROBOALIGN w/o RL2.28M92.897.459.065.678.7 w/ ROBOALIGN (Ours)2.28M+12.8K93.896.087.270.086.8 Table 3. CALVIN ABCâD success rates (%) for VLAs built upon MLLMs that were fine-tuned with various methods, evaluated over 1000 trials. Each model is evaluated by training a newly-initialized, diffusion-based action head on the CALVIN dataset while the MLLM backbone remains frozen. While all baselines show drops in task completions of length 4 and 5, ROBOALIGN consistently improves performance across all sequence. Method Fine-tuningTask completed in a row (%)âSucc. Len. Dataset Size12345(Avg) Qwen2.5VL-7B-Ins (Bai et al., 2025)-77.855.038.626.618.12.16 w/ Language-Only SFT 1.88M87.462.241.925.215.32.32 w/ Action-Only SFT1.88M66.134.715.37.13.21.26 w/ ROBOALIGN w/o RL2.28M74.649.631.521.212.21.89 w/ ROBOALIGN (Ours)2.28M+12.8K87.667.247.132.822.22.57 MLLMs training data. For supervised fine-tuning (SFT), we prepare a diverse set of datasets covering both general MLLM capability and FAST token prediction. In total, 1.88M samples are used for MLLM-related tasks. For FAST token prediction, we use the subset of BridgeV2 (Walke et al., 2023) dataset (400K samples), yielding 2.28M sam- ples overall. For reinforcement learning (RL), we further use a 12.8K subset of the BridgeV2 FAST token prediction data. More details are provided in Appendix A. Baseline models.To validate the effectiveness of ROBOALIGN, we prepare two baselines: (i) a model trained only on MLLM data and (i) a model trained only on FAST token prediction using the full BridgeV2 dataset (1.88M samples). Both are trained for one epoch following the same SFT train schema as in ROBOALIGN. Benchmarks.We evaluate VLA performance in LIBERO (Liu et al., 2023), CALVIN (Mees et al., 2022) and Real robot (see Figure 3 for the examples). ⌠LIBERO: This benchmark uses a Franka Panda Arm to perform manipulation tasks grouped into four categories: spatial, object, goal, and long-horizon. Each category consists of 10 tasks. Training uses the provided dataset covering all tasks, and evaluation runs 50 trials per task (500 trials per category). ⌠CALVIN: This benchmark also employs a Franka Panda Arm and consists of 34 distinct tasks. Train- ing uses data collected from environments A, B, and C for 100K steps, after which zero-shot evaluation is performed in a novel environment D. Performance is measured by the success rate of executing five consec- utive instruction chains, with a total of 1,000 chains evaluated. Implementation details. We train our models based on Qwen2.5VL-7B-Ins (Bai et al., 2025). For SFT, we follow the official Qwen2.5VL training repository. The vision encoder is frozen, and we use a cosine scheduler with a learning rate of2Ă10 â5 , a warmup ratio of 0.03 and training for 1 epoch. For RL, we use the EasyR1 repository 1 , training all parameters from scratch with a rollout batch size of 512, update batch size of 128, and 5 samples per prompt. We apply a constant learning rate of1Ă 10 â6 and train for one epoch. For VLA training, see detail in Appendix A. 1 https://github.com/hiyouga/EasyR1 6 RoboAlign: Learning Test-Time Reasoning for Language-Action Alignment in Vision-Language-Action Models Table 4. Real robot success rates (%) for VLAs built upon MLLMs that were fine-tuned with various methods, evaluated over 96 trials per task. Each model is evaluated by training a newly-initialized, diffusion-based action head on the real-world robotic dataset while the MLLM backbone remains frozen. We find that ROBOALIGN is also effective in real-world settings. Method Box to bowl Box to plate Basket to bowl Plate to basketAvg. Qwen2.5VL-7B-Ins (Bai et al., 2025)16.770.820.820.832.3 w/ ROBOALIGN w/o RL87.558.337.537.555.2 w/ ROBOALIGN (Ours)87.558.370.850.066.7 LiberoCALVIN 65 70 75 80 85 90 Success Rate (%) Qwen2.5VL-7B Language-Only SFT Action-Only SFT RoboAlign w/o RL RoboAlign (Ours) 1.0 1.5 2.0 2.5 3.0 Average Length Figure 4. Summary of VLA performance. Comparison of VLA performance across different MLLM training methods on LIBERO and CALVIN. ROBOALIGN achieves the highest gains. 5.2. Main Results As shown in Tables 2, 3, MLLMs trained with ROBOALIGN, which combines SFT and RL, achieve the highest perfor- mance across all simulations. The SFT stage alone yields moderate improvements, suggesting that most of the per- formance gain comes from the RL stage, despite using less than 1% of the data (2.28M vs 12.8K) used in the SFT stage. In particular, ROBOALIGN demonstrates a significant in- crease in success rates on long-horizon tasks, which are more intricate and complex than other types of tasks. For example, in CALVIN (Table 3), ROBOALIGN achieves the highest task completions of length-5 success rate (18.1%â 22.2%), whereas all other training methods exhibit a decline in performance. Similarly, in LIBERO (Table 2), the Long category improves to 70% with ROBOALIGN, compared to onlyâź2% gains from other methods. Another notable finding is in the Goal category of LIBERO, which requires handling different instructions in the same en- vironment. Here, ROBOALIGN improves performance dra- matically from 42.4% to 87.2%. However, models trained only with MLLM data show limited improvements. Specifi- cally, in CALVIN they achieve higher success in task com- pletions of length-1 (77.8%â87.4%) but experience a drop in task completions of length-5 performance (18.1% â15.3%). Similarly, in LIBERO they improve in the Goal category (42.4%â67.8%) but yield only marginal gains in the Long category (63.2%â65.6%). These results indicate that embodied reasoning abilities learned through language can enhance performance on relatively simple tasks, but of- Table 5. Compatibility with different models.We apply ROBOALIGN to a different MLLM backbone (Qwen3VL-8B-Ins) to validate its generalizability. We report success rates (%) on the LIBERO benchmark, averaged over 500 trials per category. ROBOALIGN consistently improves overall performance, with particularly significant gains in the Long category. MethodSpatialObjectGoalLongAvg. Qwen3VL-8B-Ins (Team, 2025)94.296.490.060.085.2 w/ ROBOALIGN w/o RL96.297.493.371.089.5 w/ ROBOALIGN (Ours)95.699.695.278.692.5 fer limited improvements on more complex and demanding tasks. When trained only with VLA data, we observe large in-domain gains, particularly in LIBEROâs Goal category (42.4%â82.8). However, performance drops significantly on long-horizon tasks in both CALVIN and LIBERO. We hy- pothesize that while FAST token training strengthens align- ment between instructions and low-level actions in-domain, it also induces forgetting of general MLLM capabilities, leading to reduced zero-shot generalization. 5.3. Ablation Study and Analyses Real robot experiments. To examine whether the improve- ments of ROBOALIGN on VLA performance extend beyond simulation to real-robot settings, we conduct experiments using a Franka Research 3 robot arm across four distinct pick-and-place tasks. Each task involves moving a different object (teddy bear, box, cup, sponge). Training is performed with 60 demonstrations per task, and evaluation consists of 24 trials per object, totaling 96 trials per task. The VLA setup follows the same configuration as in the main experi- ments, with each task trained for 30K steps. As shown in Table 4, ROBOALIGN consistently improves performance even in real-robot settings. Compatibility with different models. To assess whether ROBOALIGN generalizes to other architectures, we con- ducted experiments using another MLLM backbone, Qwen3-VL-8B-Ins. For the MLLM training phase, we utilized 5K samples for RL, while maintaining all other training setup. After training, all models are converted into VLAs and evaluated on the LIBERO simulation envi- ronments. As shown in Table 5, we observed an overall performance increase, with particularly significant gains in the Long category. This trend is consistent with the re- 7 RoboAlign: Learning Test-Time Reasoning for Language-Action Alignment in Vision-Language-Action Models Table 6. Impact of alignment strategies on VLA. We compare RL alignment with different targets and evaluate on LIBERO, reporting success rates (%): Language-based RL (high-level lan- guage actions), Visual-based RL (2D end-effector trajectories), and Action-based RL (Ours, low-level actions). ROBOALIGN consis- tently improves performance and uniquely enhances long-horizon tasks. MethodSpatialObjectGoalLongAvg. ROBOALIGN w/o RL92.897.459.065.678.7 w/ Language-base RL91.694.690.058.283.6 w/ Visual-base RL 92.495.687.864.685.1 w/ Action-base RL (Ours)93.896.087.270.086.8 Table 7. Comparison with SFT-based alignment. We compare our RL-based alignment against an SFT-based baseline that jointly trains reasoning and low-level action. Both methods are fine-tuned from ROBOALIGN SFT model and evaluated on the LIBERO benchmark. While the SFT-based baseline degrades performance, ROBOALIGN achieves significant improvements. MethodSpatialObjectGoalLongAvg. ROBOALIGN (SFT)92.897.459.065.678.7 w/ SFT-based Alignment (ECoT) 84.690.849.645.667.7 w/ RL-based Alignment (Ours)93.896.087.270.086.8 sults observed in Table 2. These results demonstrate that ROBOALIGN effectively generalizes across different MLLM architectures. Comparison with embodied alignment strategies. To evaluate the effectiveness of the low-level actionâbased RL alignment in ROBOALIGN, we compare it with two commonly used embodied MLLM training tasks that also employ RL: (i) predicting high-level actions expressed in language descriptions and (i) predicting 2D visual trajecto- ries of the end effector. As shown in Table 6, ROBOALIGN achieves the largest performance improvement. In contrast, the alternative methods show notable gains in the LIBERO Goal category but remain limited on long-horizon tasks. This trend is consistent with our main results in Section 5.3 and further demonstrates the advantage of direct alignment with low-level actions (See Appendix A.4 for details). Comparison with SFT-based alignment. We further com- pare RL-based alignment in ROBOALIGN against SFT- based alignment using ECoT (Zawalski et al., 2024), which jointly trains reasoning and low-level actions through SFT. As shown in Table 7, the SFT-based approach even reduces performance compared to RL. We attribute this to the lim- ited generalization of SFT, where knowledge aligned on BridgeV2 transfers poorly to LIBERO, as well as to for- getting effects introduced during SFT. Consistently, when evaluated on general MLLM benchmarks, the ECoT-trained model shows a degradation in performance, confirming the limitations of SFT-based alignment (See Appendix A.4 for detailed experimental setup). Representation analysis. To understand how ROBOALIGN Table 8. K-Nearest neighbor accuracy. We measure how accu- rately MLLM representations can predict underlying states using KNN classification on 20 trajectories from a LIBERO task. MethodAcc. (%) Qwen3VL-8B-Ins (Team, 2025)39.06 w/ ROBOALIGN w/o RL43.23 w/ ROBOALIGN (Ours)69.79 improves VLA performance, we analyze the internal repre- sentations of MLLMs. We hypothesize that ROBOALIGN explicitly aligning low-level actions enables the model to learn more discriminative and fine-grained features for action generation. To test this hypothesis, we design a KNN classification experiment to evaluate whether the MLLM representation alone, without access to robot state information, can distinguish between similar underlying states. As shown in Table 8, ROBOALIGN (SFT+RL) pro- duces substantially more discriminative representations than both baselines, improving KNN accuracy from39.06%to 69.79%. This result indicates that the RL alignment stage significantly sharpens the modelâs ability to encode fine- grained state information. Distinct representations help to generate accurate actions, and these results help to under- stand the mechanism of ROBOALIGNâs performance im- provement (See Appendix A.4 for detailed setups). Performance on MLLM Benchmarks To examine whether ROBOALIGN enhances not only VLA alignment but also embodied reasoning and general MLLM capabilities, we evaluate it on a diverse set of MLLM benchmarks. As shown in Table 9, ROBOALIGN consistently outperforms specialized embodied reasoning models such as Cosmos- Reason1 (Azzolini et al., 2025) and RoboBrain2.0 (Team et al., 2025) on embodied reasoning tasks, while maintain- ing strong performance on general MLLM benchmarks. These results indicate that RL-based alignment with low- level actions does not degrade MLLM capabilities; instead, it improves them (See Appendix A.4 for detailed setups). 6. Conclusion We proposed ROBOALIGN, a training framework for MLLMs tailored to visionâlanguageâaction models (VLA) by directly aligning MLLMâs representations with low-level action policies. Our approach leverages reinforcement learn- ing to improve low-level action prediction accuracy through embodied reasoning. We evaluated ROBOALIGN across diverse robotic environments and MLLM benchmarks, and demonstrated that it consistently delivers substantial gains in embodied reasoning performance within MLLM tasks as well as in the VLA domain across both short and long hori- zon tasks. In contrast, language-only embodied reasoning fine-tuning yields limited or even degraded performance on complex scenarios. These results establish ROBOALIGN as an effective and generalizable approach for advancing VLA training. 8 RoboAlign: Learning Test-Time Reasoning for Language-Action Alignment in Vision-Language-Action Models Impact Statement This paper proposes a Multimodal Large Language Model (MLLM) training framework that aligns with specific action data. However, if the model is aligned with unsafe action tra- jectories, Vision-Language-Action Models (VLAs) trained using such MLLMs may be biased toward generating unsafe actions. Since it is difficult to precisely determine on what data an MLLM was trained on after deployment, it is crucial to consider and prevent these issues proactively during the training phase. References Ahmed, N., Natarajan, T., and Rao, K. R. Discrete cosine transform. IEEE transactions on Computers, 100(1):90â 93, 2006. Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022. Azzolini, A., Bai, J., Brandon, H., Cao, J., Chattopadhyay, P., Chen, H., Chu, J., Cui, Y., Diamond, J., Ding, Y., et al. Cosmos-reason1: From physical common sense to embodied reasoning. arXiv preprint arXiv:2503.15558, 2025. Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. Bjorck, J., CastaĂąeda, F., Cherniadev, N., Da, X., Ding, R., Fan, L., Fang, Y., Fox, D., Hu, F., Huang, S., et al. Gr00t n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734, 2025. Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., et al.Ď 0 : A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024. Black, K., Brown, N., Darpinian, J., Dhabalia, K., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Galliker, M. Y., et al.Ď0. 5: a vision-language-action model with open- world generalization. arXiv preprint arXiv:2504.16054, 2025. Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022. Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023. Chen, B., Xu, Z., Kirmani, S., Ichter, B., Sadigh, D., Guibas, L., and Xia, F. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 14455â14465, 2024a. Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Wang, J., Qiao, Y., Lin, D., et al. Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37: 27056â27087, 2024b. Chen, W., Belkhale, S., Mirchandani, S., Mees, O., Driess, D., Pertsch, K., and Levine, S. Training strate- gies for efficient embodied reasoning. arXiv preprint arXiv:2505.08243, 2025. Chen, Y., Ge, Y., Ge, Y., Ding, M., Li, B., Wang, R., Xu, R., Shan, Y., and Liu, X. Egoplan-bench: Benchmark- ing egocentric embodied planning with multimodal large language models. CoRR, 2023. Cheng, A.-C., Yin, H., Fu, Y., Guo, Q., Yang, R., Kautz, J., Wang, X., and Liu, S. Spatialrgpt: Grounded spatial reasoning in vision language models. arXiv preprint arXiv:2406.01584, 2024. Driess, D., Xia, F., Sajjadi, M. S., Lynch, C., Chowdhery, A., Wahid, A., Tompson, J., Vuong, Q., Yu, T., Huang, W., et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023. Driess, D., Springenberg, J. T., Ichter, B., Yu, L., Li-Bell, A., Pertsch, K., Ren, A. Z., Walke, H., Vuong, Q., Shi, L. X., et al. Knowledge insulating vision-language-action models: Train fast, run fast, generalize better. arXiv preprint arXiv:2505.23705, 2025. Duan, H., Yang, J., Qiao, Y., Fang, X., Chen, L., Liu, Y., Dong, X., Zang, Y., Zhang, P., Wang, J., et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM International Conference on Multimedia, p. 11198â 11201, 2024. Fu, X., Hu, Y., Li, B., Feng, Y., Wang, H., Lin, X., Roth, D., Smith, N. A., Ma, W.-C., and Krishna, R. Blink: Multi- modal large language models can see but not perceive. In European Conference on Computer Vision, p. 148â166. Springer, 2024. Gage, P. A new algorithm for data compression. C Users Journal, 12(2):23â38, 1994. 9 RoboAlign: Learning Test-Time Reasoning for Language-Action Alignment in Vision-Language-Action Models GEAR, N. Gr00t n1.5: An improved open foundation model for generalist humanoid robots.https://research. nvidia.com/labs/gear/gr00t-n1_5/ ,June 2025. Accessed: 2025-09-09. Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. Hassabis,D.,Kavukcuoglu,K.,and DeepMind, G.Gemini model thinking updates march 2025. Google Blog, March 2025.URLhttps://blog. google/technology/google-deepmind/ gemini-model-thinking-updates-march-2025/. Hu, Y., Lin, F., Zhang, T., Yi, L., and Gao, Y. Look before you leap: Unveiling the power of gpt-4v in robotic vision- language planning. arXiv preprint arXiv:2311.17842, 2023. Huang, C.-P., Wu, Y.-H., Chen, M.-H., Wang, Y.-C. F., and Yang, F.-E. Thinkact: Vision-language-action reason- ing via reinforced visual latent planning. arXiv preprint arXiv:2507.16815, 2025a. Huang, W., Abbeel, P., Pathak, D., and Mordatch, I. Lan- guage models as zero-shot planners: Extracting action- able knowledge for embodied agents. In International conference on machine learning, p. 9118â9147. PMLR, 2022a. Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P., Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022b. Huang, W., Jia, B., Zhai, Z., Cao, S., Ye, Z., Zhao, F., Xu, Z., Hu, Y., and Lin, S. Vision-r1: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749, 2025b. Huang, Y., Chen, G., Xu, J., Zhang, M., Yang, L., Pei, B., Zhang, H., Dong, L., Wang, Y., Wang, L., et al. Egoex- olearn: A dataset for bridging asynchronous ego-and exo-centric view of procedural activities in real world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 22072â22086, 2024. Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. Ji, Y., Tan, H., Shi, J., Hao, X., Zhang, Y., Zhang, H., Wang, P., Zhao, M., Mu, Y., An, P., et al. Robobrain: A unified brain model for robotic manipulation from abstract to concrete. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 1724â1734, 2025. Jin, B., Zeng, H., Yue, Z., Yoon, J., Arik, S., Wang, D., Zamani, H., and Han, J. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025. Khazatsky, A., Pertsch, K., Nair, S., Balakrishna, A., Dasari, S., Karamcheti, S., Nasiriany, S., Srirama, M. K., Chen, L. Y., Ellis, K., et al.Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945, 2024. Kim, D., Park, S., Jang, H., Shin, J., Kim, J., and Seo, Y.Robot-r1: Reinforcement learning for en- hanced embodied reasoning in robotics. arXiv preprint arXiv:2506.00070, 2025a. Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakr- ishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., San- keti, P., et al. Openvla: An open-source vision-language- action model. arXiv preprint arXiv:2406.09246, 2024. Kim, M. J., Finn, C., and Liang, P. Fine-tuning vision- language-action models: Optimizing speed and success. arXiv preprint arXiv:2502.19645, 2025b. Kim, S., Joo, S. J., Kim, D., Jang, J., Ye, S., Shin, J., and Seo, M. The cot collection: Improving zero-shot and few- shot learning of language models via chain-of-thought fine-tuning. arXiv preprint arXiv:2305.14045, 2023. Lee, J., Duan, J., Fang, H., Deng, Y., Liu, S., Li, B., Fang, B., Zhang, J., Wang, Y. R., Lee, S., et al. Molmoact: Action reasoning models that can reason in space. arXiv preprint arXiv:2508.07917, 2025. Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al. Llava- onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. Li, X., Liu, M., Zhang, H., Yu, C., Xu, J., Wu, H., Cheang, C., Jing, Y., Zhang, W., Liu, H., et al. Vision-language foundation models as effective robot imitators. arXiv preprint arXiv:2311.01378, 2023. Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., and Zeng, A. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), p. 9493â9500. IEEE, 2023. Liu, B., Zhu, Y., Gao, C., Feng, Y., Liu, Q., Zhu, Y., and Stone, P. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36:44776â44791, 2023. 10 RoboAlign: Learning Test-Time Reasoning for Language-Action Alignment in Vision-Language-Action Models Lu, Y., Fan, Y., Deng, B., Liu, F., Li, Y., and Wang, S. Vl-grasp: a 6-dof interactive grasp policy for language- oriented objects in cluttered indoor scenes. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 976â983. IEEE, 2023. Lu, Z., Chai, Y., Guo, Y., Yin, X., Liu, L., Wang, H., Xiong, G., and Li, H. Ui-r1: Enhancing action prediction of gui agents by reinforcement learning. arXiv preprint arXiv:2503.21620, 2025. Luo, G., Yang, G., Gong, Z., Chen, G., Duan, H., Cui, E., Tong, R., Hou, Z., Zhang, T., Chen, Z., et al. Visual embodied brain: Let multimodal large language mod- els see, think, and control in spaces. arXiv preprint arXiv:2506.00123, 2025. Lynch, C., Wahid, A., Tompson, J., Ding, T., Betker, J., Baruch, R., Armstrong, T., and Florence, P. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters, 2023. Mees, O., Hermann, L., Rosete-Beas, E., and Burgard, W. Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 7(3):7327â7334, 2022. Muennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candès, E., and Hashimoto, T. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025. MĂźller, M. Information retrieval for music and motion. Springer, 2007. OpenAI.hello-gpt-4o.OpenAI Blog,May 2024.URLhttps://openai.com/index/ hello-gpt-4o/. Pertsch, K., Stachowicz, K., Ichter, B., Driess, D., Nair, S., Vuong, Q., Mees, O., Finn, C., and Levine, S. Fast: Efficient action tokenization for vision-language-action models. arXiv preprint arXiv:2501.09747, 2025. Ranasinghe, K., Li, X., Kahatapitiya, K., and Ryoo, M. S. Understanding long videos with multimodal language models. arXiv preprint arXiv:2403.16998, 2024. Ravi, N., Gabeur, V., Hu, Y.-T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. Ray, A., Duan, J., Tan, R., Bashkirova, D., Hendrix, R., Ehsani, K., Kembhavi, A., Plummer, B. A., Krishna, R., Zeng, K.-H., et al. Sat: Spatial aptitude training for multimodal language models. arXiv e-prints, p. arXivâ 2412, 2024. Sermanet, P., Ding, T., Zhao, J., Xia, F., Dwibedi, D., Gopalakrishnan, K., Chan, C., Dulac-Arnold, G., Maddi- neni, S., Joshi, N. J., et al. Robovqa: Multimodal long- horizon reasoning for robotics. In 2024 IEEE Interna- tional Conference on Robotics and Automation (ICRA), p. 645â652. IEEE, 2024. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Shen, H., Liu, P., Li, J., Fang, C., Ma, Y., Liao, J., Shen, Q., Zhang, Z., Zhao, K., Zhang, Q., et al. Vlm-r1: A stable and generalizable r1-style large vision-language model. arXiv preprint arXiv:2504.07615, 2025. Shentu, Y., Wu, P., Rajeswaran, A., and Abbeel, P. From llms to actions: latent codes as bridges in hierarchical robot control. In 2024 IEEE/RSJ International Confer- ence on Intelligent Robots and Systems (IROS), p. 8539â 8546. IEEE, 2024. Song, C. H., Blukis, V., Tremblay, J., Tyree, S., Su, Y., and Birchfield, S. Robospatial: Teaching spatial understand- ing to 2d and 3d vision-language models for robotics. In Proceedings of the Computer Vision and Pattern Recog- nition Conference, p. 15768â15780, 2025a. Song, Z., Ouyang, G., Li, M., Ji, Y., Wang, C., Xu, Z., Zhang, Z., Zhang, X., Jiang, Q., Chen, Z., et al. Maniplvm-r1: Reinforcement learning for reasoning in embodied manipulation with large vision-language mod- els. arXiv preprint arXiv:2505.16517, 2025b. Team, B. R., Cao, M., Tan, H., Ji, Y., Lin, M., Li, Z., Cao, Z., Wang, P., Zhou, E., Han, Y., et al. Robobrain 2.0 technical report. arXiv preprint arXiv:2507.02029, 2025. Team, O. M., Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Kreiman, T., Xu, C., et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213, 2024. Team,Q.Qwen3-vl:Sharper vision,deeper thought, broader action.Qwen Blog, Septem- ber 2025.URLhttps://qwen.ai/blog?id= 99f0335c4ad9f6153e517418d48535ab6d8afef& from=research.latest-advancements-list. Tellex, S., Gopalan, N., Kress-Gazit, H., and Matuszek, C. Robots that use language. Annual Review of Control, Robotics, and Autonomous Systems, 3(1):25â55, 2020. Tong, P., Brown, E., Wu, P., Woo, S., IYER, A. J. V., Akula, S. C., Yang, S., Yang, J., Middepogu, M., Wang, Z., et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Advances in Neural Information Processing Systems, 37:87310â87356, 2024. 11 RoboAlign: Learning Test-Time Reasoning for Language-Action Alignment in Vision-Language-Action Models Walke, H. R., Black, K., Zhao, T. Z., Vuong, Q., Zheng, C., Hansen-Estruch, P., He, A. W., Myers, V., Kim, M. J., Du, M., et al. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning, p. 1723â1736. PMLR, 2023. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency im- proves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. Wang, Y., Yang, Q., Zeng, Z., Ren, L., Liu, L., Peng, B., Cheng, H., He, X., Wang, K., Gao, J., et al. Reinforce- ment learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.20571, 2025. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824â24837, 2022. Wu, D., Liu, F., Hung, Y.-H., and Duan, Y. Spatial-mllm: Boosting mllm capabilities in visual-based spatial intelli- gence. arXiv preprint arXiv:2505.23747, 2025. Xu, R., Wang, W., Tang, H., Chen, X., Wang, X., Chu, F.-J., Lin, D., Feiszli, M., and Liang, K. J. Multi-spatialmllm: Multi-frame spatial understanding with multi-modal large language models. arXiv preprint arXiv:2505.17015, 2025. Yang, J., Tan, R., Wu, Q., Zheng, R., Peng, B., Liang, Y., Gu, Y., Cai, M., Ye, S., Jang, J., et al. Magma: A foundation model for multimodal ai agents. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 14203â14214, 2025. Yang, M., Du, Y., Ghasemipour, K., Tompson, J., Schu- urmans, D., and Abbeel, P. Learning interactive real- world simulators. arXiv preprint arXiv:2310.06114, 1(2): 6, 2023. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809â11822, 2023. Yu, Q., Zhang, Z., Zhu, R., Yuan, Y., Zuo, X., Yue, Y., Fan, T., Liu, G., Liu, L., Liu, X., et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. Yuan, W., Duan, J., Blukis, V., Pumacay, W., Krishna, R., Murali, A., Mousavian, A., and Fox, D. Robopoint: A vision-language model for spatial affordance prediction for robotics. arXiv preprint arXiv:2406.10721, 2024. Yuan, Y., Cui, H., Chen, Y., Dong, Z., Ni, F., Kou, L., Liu, J., Li, P., Zheng, Y., and Hao, J. From seeing to doing: Bridging reasoning and decision for robotic manipulation. arXiv preprint arXiv:2505.08548, 2025a. Yuan, Y., Cui, H., Huang, Y., Chen, Y., Ni, F., Dong, Z., Li, P., Zheng, Y., and Hao, J. Embodied-r1: Reinforced embodied reasoning for general robotic manipulation. arXiv preprint arXiv:2508.13998, 2025b. Zawalski, M., Chen, W., Pertsch, K., Mees, O., Finn, C., and Levine, S. Robotic control via embodied chain-of-thought reasoning. arXiv preprint arXiv:2407.08693, 2024. Zeng, W., Huang, Y., Liu, Q., Liu, W., He, K., Ma, Z., and He, J. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503.18892, 2025. Zhang, J., Chen, X., Wang, Q., Li, M., Guo, Y., Hu, Y., Zhang, J., Bai, S., Lin, J., and Chen, J. Vlm4vla: Revis- iting vision-language-models in vision-language-action models. arXiv preprint arXiv:2601.03309, 2026. Zhao, Q., Lu, Y., Kim, M. J., Fu, Z., Zhang, Z., Wu, Y., Li, Z., Ma, Q., Han, S., Finn, C., et al. Cot-vla: Visual chain-of-thought reasoning for vision-language-action models. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 1702â1713, 2025. Zheng, R., Liang, Y., Huang, S., Gao, J., DaumĂŠ I, H., Kolobov, A., Huang, F., and Yang, J. Tracevla: Vi- sual trace prompting enhances spatial-temporal aware- ness for generalist robotic policies. arXiv preprint arXiv:2412.10345, 2024. Zhou, E., An, J., Chi, C., Han, Y., Rong, S., Zhang, C., Wang, P., Wang, Z., Huang, T., Sheng, L., et al. Ro- borefer: Towards spatial referring with reasoning in vision-language models for robotics. arXiv preprint arXiv:2506.04308, 2025. 12 RoboAlign: Learning Test-Time Reasoning for Language-Action Alignment in Vision-Language-Action Models A. Experiment Details A.1. Computing Cost We use 8ĂH200 GPUs for MLLM training, requiring approximately 30 hours for SFT and 1 hour for reinforcement learning. For VLA training, we use 2ĂA100 GPUs, with each 10K training steps taking about 1 hour of computation. A.2. Implementation Details for VLA Training Our implementation refers to the GR00T-N1.5 codebase 2 (GEAR, 2025), adopting the same architecture with an initialized diffusion policy action expert. The expert takes as input the hidden states from the 18-th layer of Qwen2.5VL-7B-Ins (Bai et al., 2025). Hyperparameters for policy fine-tuning follow those of the official GR00T-N1.5 implementation unless otherwise specified. Action experts are newly trained for each benchmark environment with a batch size of 32. Training steps are set to 60K for LIBERO, 100K for CALVIN, 30K for real robot. A.3. Training Datasets For supervised fine-tuning (SFT), we prepare a diverse set of datasets covering both general MLLM capability and embodied reasoning. To preserve general multimodal ability, we use 100K samples from LLaVA-OneVision (single-view only) (Li et al., 2024). For embodied reasoning, we include 300K samples from RefSpatial (Zhou et al., 2025), 200K from RoboPoint (Yuan et al., 2024), 50K from EgoPlan-IT (Chen et al., 2023), and 500K from our own multi-view instruction dataset. To enhance robot-specific embodied reasoning, we incorporate 100K samples each from ShareRobot (Ji et al., 2025) and RobotVQA (Sermanet et al., 2024), 150K from our ROBOALIGN VQA, and 300K from BridgeV2 (Walke et al., 2023) and Droid (Khazatsky et al., 2024) Robot QA (predicting movements such as âmove right,â âmove left,â the current 7-DoF state, and a future sequence of 10 states). Since conventional robot imitation environments do not take video inputs, video-based datasets (RobotVQA, EgoPlan-IT, ShareRobot) are converted into single-frame inputs by extracting the last frame. For reasoning data, we include 50K multiple-choice QA samples converted from our ROBOALIGN VQA dataset and another 50K derived from SAM2 (Ravi et al., 2024), which queries spatial relations among key objects. Of these, 30K samples are used to train the reasoning distillation model. After augmenting with generated data and applying correctness filtering, the final reasoning dataset consists of 76K samples. In total, the MLLM training set contains 1.88M QA samples. For FAST token prediction (Pertsch et al., 2025), we use the subset of BridgeV2 dataset (400K samples). For reinforcement learning, we further use a 12.8K subset of the BridgeV2 FAST token prediction data. The training for FAST token prediction follows the prompt template shown in Figure 5. A.4. Analysis Setup Details Comparison with embodied alignment strategies. For a fair comparison, all models are trained with RL on the same BridgeV2 images as ROBOALIGN, encouraging embodied reasoning in both cases. For high-level action alignment, we convert movements such as "move right" or "move left" into multiple-choice QA format and provide rewards based on correctness (Kim et al., 2025a). For 2D visual trajectory prediction, we use data from ShareRobot and adopt the same reward formulation as ThinkAct (Huang et al., 2025a). Since ShareRobot contains only 6K samples, training is limited to this size. After training, all models are converted into VLAs and evaluated on the LIBERO simulation environments. Comparison with SFT-based alignment. For this experiment, we use the ECoT dataset while keeping the action space in the form of FAST tokens. Both methods are trained on the same 12.8K samples on top of ROBOALIGN SFT model, with one epoch of SFT using identical hyperparameters. Then, the resulting models are converted into VLAs and evaluated on the LIBERO simulation environments. Representation analysis. For this experiment, We select 20 training trajectories from single LIBERO long-horizon task, "put the white mug on the left plate and put the yellow and white mug on the right plate." We assign each timestep to 32 classes using Dynamic Time Warping (DTW) (MĂźller, 2007) over robot states. We then evaluate whether the MLLM, receiving only vision and task instruction, can recover the correct underlying class using a KNN classifier (k = 5) applied to its hidden representation. Performance on MLLM Benchmarks. We evaluate ROBOALIGN on a diverse set of MLLM benchmarks to assess both generalist capabilities and embodied reasoning performance. For general visual question answering (VQA), we use MMStar. 2 https://github.com/NVIDIA/Isaac-GR00T 13 RoboAlign: Learning Test-Time Reasoning for Language-Action Alignment in Vision-Language-Action Models Table 9. Performance on multimodal benchmarks. We evaluate ROBOALIGN and other MLLMs on general image understanding (MMStar), spatial reasoning (RoboSpatial, Where2Place, BLINK, and robot embodied reasoning (Robot-R1 Bench) benchmarks. Our initial SFT model, ROBOALIGN (SFT), performs on par with specialized embodied-reasoning MLLMs, and RL training further boosts performance across the overall MLLM benchmarks. Values marked withâare taken from prior work (Team et al., 2025; Duan et al., 2024). ModelMMStarRobot-R1 Bench (0-3)RoboSpatialWhere2PlaceBlink (Rel. Depth) GPT-4o-2024-11-20 (Hurst et al., 2024)65.10 â 1.5544.42 â 20.41 â 77.90 â Qwen2.5-VL-7B-Ins (Bai et al., 2025)60.301.0236.2911.3555.64 Cosmos-Reason1-7B (Azzolini et al., 2025)54.401.1938.81 â 5.51 â 68.57 â RoboBrain2.0-7B (Team et al., 2025) 35.801.1754.23 â 63.59 â 83.95 â VeBrain-8B (Luo et al., 2025)61.901.0242.48 â 11.34 â 79.68 â ROBOALIGN w/o RL62.471.1448.8651.6687.10 ROBOALIGN (Ours)62.801.3850.8654.4987.90 For spatial reasoning, we adopt Robospatial-Home, Where2Place, and the depth-related components of BLINK, which evaluate spatial understanding and geometric reasoning. For robot embodied reasoning, we use Robot-R1 Bench, which provides fine-grained evaluations of embodied reasoning abilities including planning, subtask decomposition, movement understanding, and spatial reasoning, all constructed on the BridgeV2 dataset. All benchmarks are evaluated following their official protocols, and detailed evaluation settings are reported in Table 9. Waypoint prediction QA for SFT System Prompt You are an embodied vision-language robotic assistant for multi-object manipulation. System Prompt for Reinforcement Learning You are an embodied vision-language robotic assistant for multi-object manipulation. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively. Prompt Your current task is instruction. Output the robotâs actions to perform this task through FAST tokens. Figure 5. Prompt for FAST token generation. We use this prompt template for both FAST token prediction and reinforcement learning. A.5. RL Training Process Figures 6 and 2 illustrate how the average response length and accuracy reward evolve during the RL training of ROBOALIGN. The average response length remains stable, while the accuracy reward gradually increases, indicating effective learning. Figure 7 provides qualitative examples before and after RL training. Prior to training, responses mainly describe high-level plans with minimal detail on immediate actions (e.g., âthe robot needs to first approach the clothâ). After RL training, the model generates more concrete action descriptions, such as âthe robot needs to first approach and align its gripper with the cloth. Once aligned, it will need to close the gripper to pick up the cloth.â. In addition, a novel element of situational awareness has been introduced (e.g., "The robotâs current position suggests it is already aligned with the cloth, ready to pick it up."). These quantitative and qualitative results together demonstrate that RL training is successfully improving the modelâs ability to align reasoning with actionable steps. 14 RoboAlign: Learning Test-Time Reasoning for Language-Action Alignment in Vision-Language-Action Models 0510152025 Step 0.10 0.12 0.14 0.16 0.18 Accuracy Reward (a) Accuracy reward change during RL training 0510152025 Step 76 78 80 82 84 86 Mean Response Length (b) Response length change during RL training Figure 6. Training metrics for reinforcement learning in ROBOALIGN. Validation performance during RL training: (a) accuracy reward increases steadily, while (b) response length remains stable. Question: Your current task is move the orange cloth to the top of the right burner. Output the robotâs actions to perform this task through FAST tokens. GT Answer: <|action_start|><|action_486|><|action_265|><|action_268|><|action_116|><|action_269|><|action_end|> ROBOALIGN (SFT) Answer: â<think>To perform the task of moving the orange cloth to the top of the right burner, the robot needs to first approach the cloth, then grasp it, and finally move it to the right burner. The actions should be sequential and purposeful, focusing on the cloth and the burner in question.</think><answer> <|action_start|><|action_266|><|action_709|><|action_268|><|action_116|><|action_269|><|action_end|> Reward: 0.5 (Format 1.0 / Accuracy 0.0) ROBOALIGN (SFT+RL) Answer:<think>To perform the task of moving the orange cloth to the top of the right burner, the robot needs to first approach and align its gripper with the cloth.Once aligned, it will need to close the gripper to pick up the cloth, lift it, and then move it to the right burner before re- leasing it.The robotâs current position suggests it is already aligned with the cloth, ready to pick it up.</think><answer><|action_start|><|action_486|><|action_265|><|action_268|><|action_116|><|action_269|><|action_end|> Reward: 1.0 (Format 1.0 / Accuracy 1.0) Figure 7. Example of model response. 15