Paper deep dive
SA-VLA: State-aware tokenizer for improving Vision-Language-Action Models' performance
Tengyue Jiang, Chunpu Xu, Jiayue Kang, Yao Mu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 98%
Last extracted: 7/5/2026, 3:28:19 AM
Summary
SA-VLA is a state-aware action tokenizer designed to improve Vision-Language-Action (VLA) models by conditioning action decoding on the robot's proprioceptive state. Traditional discrete tokenizers often suffer from a compression gap because they map discrete codes to fixed continuous prototypes, ignoring the robot's current configuration. SA-VLA addresses this by proposing two mechanisms: Method A (cross-attention between state and action features) and Method B (a lightweight state adapter that predicts action-wise modulation factors). Method B, which uses a scaling factor to expand the effective support of the codebook, proved superior. The model was evaluated on 12 RoboTwin manipulation tasks in simulation and three real-world tasks, demonstrating significant improvements in success rates over baselines like OpenVLA (binning), FAST, and VQ-VLA (VQ-BET), particularly in zero-shot sim-to-real transfer.
Entities (9)
Relation Signals (5)
SA-VLA → evaluatedon → RoboTwin
confidence 100% · On 12 RoboTwin manipulation tasks, SA-VLA improves the average success rate
Method B → isatypeof → SA-VLA
confidence 100% · The other approach is inspired by human behavioral habits... This way is denoted as Method B.
Method A → isatypeof → SA-VLA
confidence 100% · the first introduces state information through a lightweight adapter module, while the second integrates state and action information via cross-attention... denoted as Method A.
SA-VLA → testedon → Agilex Cobot Magic
confidence 100% · In our real-world experiments, we employ the AgileX Cobot Magic mobile platform
SA-VLA → improves → VLA
confidence 90% · SA-VLA: State-aware tokenizer for improving Vision-Language-Action Models' performance
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Discrete action tokenization provides a compact interface for autoregressive VLA policies, but accurately recovering continuous robot actions from discrete codes remains challenging. Existing tokenizers typically map each discrete code to a fixed continuous action prototype, ignoring the robot's current proprioceptive state. This limitation is particularly pronounced in manipulation, where the same action token may require different continuous controls under different joint configurations, object poses, and contact conditions. We therefore propose SA-VLA, a state-aware action tokenizer that conditions action decoding on robot state. We study two state-injection mechanisms for VQ-based action tokenization: cross-attention between state and action features, and a lightweight state adapter that predicts action-wise modulation factors for state-conditioned action modulation and reconstruction. The adapter formulation expands the effective support of a finite codebook by allowing each discrete token to represent a family of state-dependent continuous actions, while preserving the efficiency and compatibility of discrete action modeling. Integrated into an LLM-based VLA policy, SA-VLA supports both autoregressive and parallel action-token decoding with minimal changes to the model interface. On 12 RoboTwin manipulation tasks, SA-VLA improves the average success rate from 0.29 to 0.56 over the strongest tokenizer baseline. In zero-shot sim-to-real experiments on three real-world tasks, it further improves average success from 0.15 to 0.33 over the strongest tokenizer baseline. These results demonstrate that state-conditioned action decoding is a simple and effective mechanism for reducing the compression gap in discrete VLA policies.
Tags
Links
- Source: https://arxiv.org/abs/2606.30113v1
- Canonical: https://arxiv.org/abs/2606.30113v1
Trouble viewing inline? Open PDF directly →
Full Text
37,378 characters extracted from source content.
Expand or collapse full text
SA-VLA: State-aware tokenizer for improving Vision-Language-Action Models’ performance. Tengyue Jiang2 Chunpu Xu3 Jiayue Kang4 Yao Mu1 1Shanghai Jiao Tong University 2East China University of Science and Technology 3Hong Kong Polytechnic University 4Xi’an University of Electronic Science and Technology Abstract Discrete action tokenization provides a compact interface for autoregressive VLA policies, but accurately recovering continuous robot actions from discrete codes remains challenging. Existing tokenizers typically map each discrete code to a fixed continuous action prototype, ignoring the robot’s current proprioceptive state. This limitation is particularly pronounced in manipulation, where the same action token may require different continuous controls under different joint configurations, object poses, and contact conditions. We therefore propose SA-VLA, a state-aware action tokenizer that conditions action decoding on robot state. We study two state-injection mechanisms for VQ-based action tokenization: cross-attention between state and action features, and a lightweight state adapter that predicts action-wise modulation factors for state-conditioned action modulation and reconstruction. The adapter formulation expands the effective support of a finite codebook by allowing each discrete token to represent a family of state-dependent continuous actions, while preserving the efficiency and compatibility of discrete action modeling. Integrated into an LLM-based VLA policy, SA-VLA supports both autoregressive and parallel action-token decoding with minimal changes to the model interface. On 12 RoboTwin manipulation tasks, SA-VLA improves the average success rate from 0.29 to 0.56 over the strongest tokenizer baseline. In zero-shot sim-to-real experiments on three real-world tasks, it further improves average success from 0.15 to 0.33 over the strongest tokenizer baseline. These results demonstrate that state-conditioned action decoding is a simple and effective mechanism for reducing the compression gap in discrete VLA policies. 1 Introduction Visual-Language-Action (VLA)[3, 14, 16] models have recently emerged as a highly promising research paradigm in the field of robotic manipulation, capable of directly mapping visual observations and language instructions to action sequences.Generally, VLA[6] models can be divided into two categories based on how they generate actions: one category generates discrete action tokens that are then decoded into continuous actions via an action tokenizer(e.g., OpenVLA[15], VQ-VLA[27])while the other category directly generates continuous actions (e.g., π 0[2]).A common approach for generating discrete action tokens involves mapping visual predictions and language instructions into action tokens through a pretrained VLM, which are then decoded into continuous action values via an action tokenizer. However, how to convert the discrete action tokens output by the VLM[1] into continuous actions remains a significant bottleneck limiting model performance[22]. Previous research has made various attempts in this direction. For example, the binning operation used in OpenVLA[15] is simple and easy to implement but its tokenization granularity is low.The FAST[21] method offers good compression performance for high-dimensional information; however, the sequence length obtained from compressing actions of the same length may be inconsistent, making it difficult to train. The residual VQ-VAE[25] approach in VQ-VLA[27] is data-driven, yet its compression performance still leaves room for improvement. To address this, we propose a state-aware action tokenizer, which introduces state information into the action tokenizer. We explore two approaches for incorporating state information. The first introduces state information through a lightweight adapter module, while the second integrates state and action information via cross-attention[26]. Both approaches achieve significant improvements in both simulation and real-world robot experiments. Between the two approaches, the method that introduces state information via a lightweight adapter module achieves greater improvements than the cross-attention approach. We attribute this to the fact that traditional VQ-VAE[25] can only decode a limited set of actions, which constrains model performance. By introducing the adapter module, we enable state-informed prediction of an continuous action space through a regression formulation, thereby enhancing both model performance and action decoding accuracy. In summary, our contributions are as follows: 1.We propose a state-aware tokenization approach, introducing state information into the action tokenizer. 2.We explore optimal performance through different integration strategies of state information.By introducing a lightweight adapter module to address the limitations of the traditional VQ-VAE[25]’s finite codebook. 3.Our method achieves strong performance in both simulation and real-world robot experiments 2 Related works 2.1 Vision-Language-Action Models Vision-Language-Action Models are considered foundational models in the embodied AI field. They use a pre-trained VLM to extract action tokens from images and language instructions. One approach employs an action tokenizer to convert discrete action tokens into continuous action values, while another uses an action expert that leverages the keys and values from the VLM. Building on this foundational model, recent studies have explored various forms of VLAs, such as those based on discrete diffusion models[18][17], integrations of world models with VLAs[23],chain-of-thought for action generation[32, 19],and the incorporation of event information into VLAs[29]. 2.2 Discrete Action Tokenizers Action tokenizers are a key component of autoregressive transformer-based VLA models. Recent research has explored various forms of action tokenizers. For example, OpenVLA discretizes actions through uniform binning, where each action value corresponds to a bin, but this tokenization approach is overly coarse. FAST[21] transforms action values from the time domain to the frequency domain using DCT, followed by BPE to obtain corresponding discrete action tokens. However, due to the nature of BPE, action sequences of the same length may be mapped to token sequences of varying lengths. VQ-VLA uses residual VQ-VAE[25] to establish a mapping between action values and tokens, but we believe the granularity of this action tokenization remains insufficient. In our work, we propose a state information-guided action tokenizer, which offers stronger generalization capability and positional adaptability.Current tokenizers often borrow tokenizer methods from the field of natural language processing for text or from the field of computer vision for image generation[28, 7, 34], while neglecting the inherent characteristics of VLA themselves. 3 Methods 3.1 Problem Formulation For a robotic arm manipulation task, given a dataset (o1,s1,a1),…,(ot,st,at)\(o_1,s_1,a_1),…,(o_t,s_t,a_t)\ with language instruction L, where oto_t denotes the observed RGB image, sts_t denotes the state of the robotic arm (typically represented by the rotation angle of each joint), and ata_t denotes the target action of the robotic arm at the current timestep, our goal is to train a policy πθ(at:t+k∣ot,st,L) _θ(a_t:t+k o_t,s_t,L) that predicts a block of future k actions conditioned on the current observed image, state, and language instruction. 3.2 State-aware Action Tokenizer First, we briefly introduce the base VQ-VAE[25]. The base model uses CNN and Transformer[26] encoders to extract temporal action features. These action features are then quantized using a codebook, and fixed-length discrete action embeddings are output. After that, decoding is performed through a decoder with a symmetric structure to the encoder, outputting the reconstructed actions.Based on this action tokenizer, we consider incorporating state information to improve the accuracy of action decoding. We explore two ways to introduce state information into the action tokenizer. The overall model architecture is illustrated in Figure 1. Our motivation for introducing state into the tokenizer is that when using delta actions, the same delta action may correspond to different physical outcomes in certain situations. For example, when a robotic arm is grasping a bottle, the same delta action for grasping might be identical, but different grasp positions can determine whether the bottle is successfully picked up. If all actions corresponding to different grasp positions are mapped to the same action token, the performance will naturally degrade. This highlights the importance of incorporating state information.Inspired by the interaction between VLM tokens and action experts in π 0[2], we propose a method that performs cross-attention between state and action within the Transformers of both the encoder and decoder to incorporate state information, denoted as Method A. The other approach is inspired by human behavioral habits: humans often predict future actions based on the current state before acting, for example, when the hand approaches a cup, it is likely to grasp it. Therefore, we propose using a lightweight adapter to map state information into a scaling factor for each dimension of the action sequence, thereby predicting the scale of action generation with the assistance of state information.This way is denoted as Method B. Below, we describe how the model performs forward propagation in detail. For Method A, given an input action sequence a, we first extract its intermediate features using an encoder composed of a CNN and two layers of Transformer[26], where the keys and values in the two Transformer layers are both derived from the state input. The intermediate features output by the encoder are denoted as Z∈ℝR×dZ ^R× d.After the quantization operation of the VQ-VAE[25] codebook, the resulting quantized vectors are denoted as z^i′ z_i . The quantization process for embedding z^i z_i with codebook Z can be formulated as follows: qi=argmin‖zi−z^i‖2,q_i= \|z_i- z_i\|_2, where qiq_i is the nearest neighbor lookup index from the codebook Z. After the quantization operation, the reconstructed a a is obtained through a decoder that is symmetric to the encoder. It should be noted that the keys and values in the two Transformer[26] layers of the decoder also use the raw state data. Method B is similar to Method A. Given an input action sequence a, the state information first passes through a lightweight adapter composed of MLP and Activation functions such as sigmoid[11] to obtain a scaling factor w for each dimension of a, resulting in a transformed action atrans=a÷wa_trans=a w. The transformed action atransa_trans then passes through an encoder with the same structure to extract its intermediate features, where the Transformer[26] does not incorporate state input. The intermediate features output by the encoder are denoted as Z∈ℝR×dZ ^R× d. After undergoing the same quantization operation, the quantized features pass through a decoder symmetric to the encoder to obtain the reconstructed transformed action a^trans a_trans. To maintain symmetry, a^trans a_trans is multiplied by w to obtain the final reconstructed action a a. In this case, the Transformer[26] does not use state input either. Following the training paradigm of VQ-VAE[25], we define the overall loss ℒL as a weighted combination of the reconstruction loss ℒresL_res, the vector quantization (VQ) loss ℒcodebookL_codebook, and the commitment loss ℒcommitL_commit: ℒ=‖a^−a‖22+λ1⋅‖sg(x)−q(x)‖22+λ2⋅‖x−sg(q(x))‖22,L=\| a-a\|_2^2+ _1·\|sg(x)-q(x)\|_2^2+ _2·\|x-sg(q(x))\|_2^2, where λ1 _1 and λ2 _2 are balancing hyperparameters, sg(⋅)sg(·) denotes the stop-gradient operation, x represents the encoder output features, and q(x)q(x) denotes the quantized features from the codebook. Figure 1: The tokenizer architecture.As shown in the upper part of the figure, Method A integrates state information into the tokenizer by performing cross-attention with the action. As shown in the lower part of the figure, Method B integrates state information into the tokenizer by modulating the action through a lightweight adapter. 3.3 Integrating State-aware Action Tokenizer in VLA Figure 2: VLA architecture The overall model architecture is illustrated in Figure 2.Our goal is to train a VLA[4][5] policy πθ(at:t+k∣ot,st,L) _θ(a_t:t+k o_t,s_t,L) that predicts a block of future actions conditioned on the current observation image oto_t, robot state sts_t, and language instruction L. For the language description L, we use the base LLM’s native tokenizer to obtain text tokens. For the joint states sts_t, we adopt the approach outlined in FAST [21] to discretize each joint dimension into 256 bins, yielding a sequence of discrete state tokens. For processing the observation image oto_t, we resize each image to 224×224224× 224 and apply SigLIP-SO400M-patch14-224 [30] as the image tokenizer, producing a 16×1616× 16 grid of patch[10] embeddings, i.e., 256 continuous image tokens per frame. For the input action sequence, we use our proposed state-aware tokenizer (SA-VLA tokenizer) to transform continuous action sequences a′a into discrete action tokens a~ a. Unlike prior tokenizers, our approach introduces state-conditioned decoding via a lightweight adapter that predicts a scaling factor w for each action dimension, transforming the action as atrans=a÷wa_trans=a w before quantization. This effectively reformulates the discrete VQ-VAE[25] into a regression problem, enabling the generation of a continuous action space while preserving the benefits of a fixed codebook. To delineate modality boundaries in the input tokens, we introduce special functional tokens: t_bos/t_eos for text, s_bos/s_eos for state, i_bos/i_eos for image, and a_bos/a_eos for action. During training, we serialize the context and targets into a single sequence, for example: [t_bos, t_tokens, t_eos, s_bos, s_tokens, s_eos, i_bos, i_tokens, i_eos, a_bos, action_tokens, a_eos] This design yields a consistent tokenized interface across modalities, enabling straightforward autoregressive learning of the action sequence conditioned on language, state, and vision. We evaluate our state-aware tokenizer under two decoding strategies: Autoregressive Decoding. In this setting, the VLA generates action tokens sequentially via next-token prediction. The training objective minimizes the negative log-likelihood of predicted action tokens given previous tokens and the full context: ℒAD=−∑i=1h∑r=1RlogP(qi,r∣q<,ot,st,L),L_AD=- _i=1^h _r=1^R P(q_i,r q_<,o_t,s_t,L), (1) where q<q_< denotes previous action tokens. Parallel Decoding. Our tokenizer produces fixed-length discrete action sequences, enabling parallel decoding[14] where the entire action block[33] is generated in a single forward pass. We adopt bidirectional attention and placeholder embeddings for action positions, with the training objective: ℒPD=−∑i=1h∑r=1RlogP(qi,r∣ot,st,L).L_PD=- _i=1^h _r=1^R P(q_i,r o_t,s_t,L). (2) Parallel decoding significantly improves inference efficiency while maintaining high performance. Across both settings, the LLM remains unchanged and continues to output discrete tokens, with only minimal modifications to the input interface and attention mask. 4 Experiments 4.1 Simulation Experiments 4.1.1 Experiment Setup Our simulation experiments were conducted on the RoboTwin[8] platform in clean mode, with 12 tasks, each containing 1,600 trajectories, resulting in a total of 19,200 simulation data entries for training and testing. The specific task types are detailed in Appendix A.2. To perform comparative experiments and validate the effectiveness of our approach, we used the same Qwen2.5-based VLM followed by different action tokenizers. The following groups were set up: (1) the bin-based tokenization method represented by OpenVLA[15]; (2) the tokenization method represented by FAST[21]; (3) the VQ-BET tokenization method represented by VQ-VLA[27];(4) Method A as described in our approach; (5) Method B as described in our approach. All groups were trained under the same hyperparameter settings and tested on RoboTwin[8].The details regarding training and testing are provided in the Appendix A. 4.1.2 Simulation Results We evaluated the models on the same 12 tasks from the RoboTwin benchmark used during training. For each task, we conducted 100 rollouts, where a success was recorded as 1 and a failure as 0, and then computed the success rate for each task. The experimental results are shown in Table 4.1. The results demonstrate that our methods, Method A and Method B, achieve significant improvements over previous action tokenization approaches under both autoregressive and parallel decoding settings. Specifically, Method B achieves an average success rate of 56% across the 12 tasks under autoregressive decoding, which is a 23% improvement over the binning tokenizer, a 40% improvement over the FAST tokenizer, and a 28% improvement over the VQ-BET tokenizer. Table 4.1: Tokenizer performance comparison on RoboTwin Tokenizer Beat Block Move Playingcard Pick Diverse Move Can Move Pillbottle Click Handover Hammer ↑ Away ↑ Bottles ↑ Pot ↑ Pad ↑ Bell ↑ Mic ↑ Binning 0.11 0.01 0.27 0.02 0.00 0.67 0.00 FAST 0.00 0.18 0.06 0.08 0.01 0.68 0.01 VQ-BET 0.10 0.27 0.12 0.13 0.06 0.64 0.00 MethodA(AR) 0.25 0.47 0.14 0.65 0.35 0.83 0.18 MethodA(PD) 0.12 0.49 0.05 0.44 0.46 0.80 0.18 MethodB(AR) 0.16 0.60 0.22 0.62 0.36 0.90 0.22 MethodB(PD) 0.29 0.63 0.21 0.47 0.41 0.89 0.28 Tokenizer Place Container Handover Place Phone Place Burger Shake average↑ Plate ↑ Mic ↑ Stand ↑ Fries ↓ Bottle ↓ Binning 0.00 0.85 0.02 0.01 0.92 0.24 FAST 0.07 0.17 0.01 0.00 0.76 0.17 VQ-BET 0.54 0.59 0.04 0.15 0.87 0.29 MethodA(AR) 0.90 0.96 0.25 0.77 0.85 0.55 MethodA(PD) 0.83 0.98 0.34 0.71 0.88 0.52 MethodB(AR) 0.86 0.99 0.27 0.59 0.96 0.56 MethodB(PD) 0.85 0.95 0.37 0.53 0.89 0.56 4.1.3 Results Analysis We believe such simulation results are interpretable. Introducing state information into the action tokenizer helps the tokenizer perform more precise decoding, thus leading to better experimental performance. The reason why Method B achieves better results than Method A is as follows: In Method A, the decoding process of VQ-VAE[25] is from discrete codebook tokens to continuous action values, which is essentially limited by the codebook capacity—i.e., there is a one-to-one correspondence between the output tokens in the VLA and the corresponding actions. In contrast, the advantage of Method B lies in extending the decoding process of VQ-VAE[25] by multiplying the discrete codebook tokens with the action scaling scales provided by the state information. Since the scales obtained from the state network vary with different states, the same token can produce multiple actions. This can be understood as: the tokens in Method B represent only a class of actions, while the state network provides the scale for that class of actions. Therefore, this approach can be extended to a continuous action space, resulting in better performance. 4.2 Ablation Study To validate the effectiveness of our proposed method, we conduct the following ablation studies: (1) verifying whether the introduction of state improves the decoding accuracy of the action tokenizer; (2) evaluating the generalization capability of SA-VLA. 4.2.1 Ablation Study on the Role of State To fully demonstrate the advantage of incorporating state information into the action tokenizer for action decoding, we conducted several comparative experiments. On the aforementioned 12 RoboTwin[8] tasks, we trained a VLA without state information in the action tokenizer and a VLA with state information in the action tokenizer, and evaluated their average success rates under both autoregressive and parallel decoding settings. The experimental results are shown in Table 4.2. Table 4.2: Ablations on the impact of introducing state to action tokenizer Method Average↑ w/o state (PD) 0.43 w/o state (AR) 0.51 Method A (PD) 0.52 Method A (AR) 0.55 Method B (PD) 0.56 Method B (AR) 0.56 4.2.2 Ablation study on model generalization To investigate the generalization capability of SA-VLA, we constructed four task settings: shake_bottle_horizontally and place_empty_cup in a clean environment, and handover_mic and place_container_plate in a random environment. The former two tasks are designed to evaluate the model’s performance on previously unseen tasks, whereas the latter two tasks aim to assess the model’s performance in unseen scenarios. The corresponding experimental results are presented in the Appendix C.1. 4.2.3 Ablation study on tokenization granularity To demonstrate the fine-grained action tokenization capability of our method, we selected four similar action timesteps from four task scenarios on the Beat Block Hammer task in RoboTwin. In tokenizers without state information, these four actions were mapped to identical indices. In contrast, with state-aware tokenization, our method can distinguish these four actions at the level of cosine similarity differences as fine as 0.001.See Appendix C.2 for details. 4.3 Real world experiments 4.3.1 Experiment Setup In our real-world experiments, we employ the AgileX Cobot Magic mobile platform configured with an Aloha setup comprising four robotic arms. Each arm is an AgileX Piper with six degrees of freedom, equipped with a one-DoF parallel gripper. The platform is also equipped with a RealSense D435 RGB camera that captures real-time RGB images at a resolution of 640×480640× 480 pixels and a frame rate of approximately 30 Hz. To evaluate the zero-shot sim-to-real transfer capability of our VLA models, we directly apply the VLAs trained on RGB images from the RoboTwin simulator to three real-world manipulation tasks: Click Bell, Place Container Plate, and Pick Diverse Bottles. Each task is evaluated with 20 trials.The tasks are shown in Appendix B.2. 4.3.2 Real world results To demonstrate better results, we adopt the Method B tokenizer, which performs better in simulation, for our experiments. The experimental results in Table 4.3 show that under the zero-shot setting, our method achieves an average success rate of 33% across the 3 tasks under autoregressive decoding, which is a 23% improvement over the binning tokenizer, a 25% improvement over the FAST tokenizer, and an 18% improvement over the VQ-BET tokenizer. Table 4.3: Tokenizer performance comparison on Real world Tokenizer Click bell↑ Place container Pick Diverse Average↑ plate ↑ Bottle ↑ Binning 6/20 0/20 0/20 0.1 FAST 4/20 1/20 0/20 0.08 VQ-BET 7/20 2/20 0/20 0.15 Ours(PD) 8/20 5/20 3/20 0.27 Ours(AR) 10/20 7/20 3/20 0.33 5 Conclusion In our work, we experimented with different ways to incorporate state information into the action tokenizer and selected the best action tokenizer to seamlessly integrate into the VLA. In simulation, this achieves up to a 40% improvement over previous classic action tokenizer approaches, and attains high success rates in real-world sim-to-real experiments. 6 Limitations Scalability: It has only been validated on small-scale datasets, and its scalability has not been demonstrated. Nowadays, many large-scale datasets[9, 20] have emerged.Many current works[24] introduce large-scale datasets to attempt to validate scaling laws[13] in robotics, which is also an important future research direction. In subsequent work, we will expand the training data volume and model parameters within our resource limits to attempt to validate the scaling law[13] of our method. Model architecture: Furthermore, we follow the VQ-VAE[25] architecture; whether other generative models like diffusion[12] can be used to replace VQ-VAE[25] remains a promising direction for future research. In image generation[28] and other domains, diffusion[12] models have demonstrated strong generative capabilities and high fidelity. Whether diffusion[12] can be integrated into discrete action tokenizers to improve generation quality is also a direction worth exploring in our future work. Embodiment selection: Compared to dexterous hands, robotic arms are less convenient in practical real-world production and daily life scenarios. Therefore, whether the embodiment can be migrated from robotic arms to dexterous hands[31] while maintaining stable performance is a direction we will explore in the future. References [1] S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, et al. (2025) Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Cited by: §1. [2] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. (2024) π0 _0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164. Cited by: §1, §3.2. [3] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al. (2022) Rt-1: robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817. Cited by: §1. [4] Q. Bu, Y. Yang, J. Cai, S. Gao, G. Ren, M. Yao, P. Luo, and H. Li (2025) Univla: learning to act anywhere with task-centric latent actions. arXiv preprint arXiv:2505.06111. Cited by: §3.3. [5] J. Cen, C. Yu, H. Yuan, Y. Jiang, S. Huang, J. Guo, X. Li, Y. Song, H. Luo, F. Wang, et al. (2025) Worldvla: towards autoregressive action world model. arXiv preprint arXiv:2506.21539. Cited by: §3.3. [6] C. Cheang, G. Chen, Y. Jing, T. Kong, H. Li, Y. Li, Y. Liu, H. Wu, J. Xu, Y. Yang, et al. (2024) Gr-2: a generative video-language-action model with web-scale knowledge for robot manipulation. arXiv preprint arXiv:2410.06158. Cited by: §1. [7] H. Chen, Z. Wang, X. Li, X. Sun, F. Chen, J. Liu, J. Wang, B. Raj, Z. Liu, and E. Barsoum (2025) Softvq-vae: efficient 1-dimensional continuous tokenizer. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 28358–28370. Cited by: §2.2. [8] T. Chen, Z. Chen, B. Chen, Z. Cai, Y. Liu, Z. Li, Q. Liang, X. Lin, Y. Ge, Z. Gu, et al. (2025) Robotwin 2.0: a scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. arXiv preprint arXiv:2506.18088. Cited by: §4.1.1, §4.2.1. [9] Y. Deng and D. Zhou (2026) HumanNet: scaling human-centric video learning to one million hours. arXiv preprint arXiv:2605.06747. Cited by: §6. [10] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. (2020) An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. Cited by: §3.3. [11] S. Elfwing, E. Uchibe, and K. Doya (2018) Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks 107, p. 3–11. Cited by: §3.2. [12] J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. Advances in neural information processing systems 33, p. 6840–6851. Cited by: §6. [13] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei (2020) Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. Cited by: §6. [14] M. J. Kim, C. Finn, and P. Liang (2025) Fine-tuning vision-language-action models: optimizing speed and success. arXiv preprint arXiv:2502.19645. Cited by: §1, §3.3. [15] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. (2024) Openvla: an open-source vision-language-action model. arXiv preprint arXiv:2406.09246. Cited by: §1, §1, §4.1.1. [16] H. Li, Y. Zuo, J. Yu, Y. Zhang, Z. Yang, K. Zhang, X. Zhu, Y. Zhang, T. Chen, G. Cui, et al. (2025) Simplevla-rl: scaling vla training via reinforcement learning. arXiv preprint arXiv:2509.09674. Cited by: §1. [17] H. Liang, X. Chen, B. Wang, M. Chen, Y. Liu, Y. Zhang, Z. Chen, T. Yang, Y. Chen, J. Pang, et al. (2025) M-act: learn from multimodal parallel generation to act. arXiv preprint arXiv:2512.00975. Cited by: §2.1. [18] Y. Liu, P. Ding, T. Jiang, X. Wang, W. Song, M. Lin, H. Zhao, H. Zhang, Z. Zhuang, W. Zhao, et al. (2026) MMaDA-vla: large diffusion vision-language-action model with unified multi-modal instruction and generation. arXiv preprint arXiv:2603.25406. Cited by: §2.1. [19] Z. Liu, J. Liu, H. Chen, J. Yu, Z. Guo, C. Hou, C. Gu, X. Mi, R. Zhang, K. Wu, et al. (2026) LaST _\_0: latent spatio-temporal chain-of-thought for robotic vision-language-action model. arXiv preprint arXiv:2601.05248. Cited by: §2.1. [20] A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, et al. (2024) Open x-embodiment: robotic learning datasets and rt-x models: open x-embodiment collaboration 0. In 2024 IEEE International Conference on Robotics and Automation (ICRA), p. 6892–6903. Cited by: §6. [21] K. Pertsch, K. Stachowicz, B. Ichter, D. Driess, S. Nair, Q. Vuong, O. Mees, C. Finn, and S. Levine (2025) Fast: efficient action tokenization for vision-language-action models. arXiv preprint arXiv:2501.09747. Cited by: §1, §2.2, §3.3, §4.1.1. [22] T. Shiba (2026) The compression gap: why discrete tokenization limits vision-language-action model scaling. arXiv preprint arXiv:2604.03191. Cited by: §1. [23] B. Team (2026) Being-h0. 7: a latent world-action model from egocentric videos. Cited by: §2.1. [24] G. A. Team (2025) GEN-0: embodied foundation models that scale with physical interaction. Generalist AI Blog. Note: https://generalistai.com/blog/nov-04-2025-GEN-0 Cited by: §6. [25] A. Van Den Oord, O. Vinyals, et al. (2017) Neural discrete representation learning. Advances in neural information processing systems 30. Cited by: §1, §1, §1, §2.2, §3.2, §3.2, §3.2, §3.3, §4.1.3, §6. [26] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. Advances in neural information processing systems 30. Cited by: §1, §3.2, §3.2, §3.2, §3.2. [27] Y. Wang, H. Zhu, M. Liu, J. Yang, H. Fang, and T. He (2025) Vq-vla: improving vision-language-action models via scaling vector-quantized action tokenizers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 11089–11099. Cited by: §1, §1, §4.1.1. [28] S. Yang, X. Hu, Q. Wu, and D. Yang (2026) VAEVQ: enhancing discrete visual tokenization through variational modeling. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 11703–11711. Cited by: §2.2, §6. [29] J. Zhai, H. Shi, S. Guo, K. Yang, and K. Wang (2026) E-vla: event-augmented vision-language-action model for dark and blurred scenes. arXiv preprint arXiv:2604.04834. Cited by: §2.1. [30] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer (2023) Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, p. 11975–11986. Cited by: §3.3. [31] Z. Zhang, J. Pang, Z. Yang, K. Li, M. Liao, S. Zhang, G. Chi, J. Guo, H. Gao, M. Shi, et al. (2026) Dexora: open-source vla for high-dof bimanual dexterity. arXiv preprint arXiv:2605.18722. Cited by: §6. [32] Q. Zhao, Y. Lu, M. J. Kim, Z. Fu, Z. Zhang, Y. Wu, Z. Li, Q. Ma, S. Han, C. Finn, et al. (2025) Cot-vla: visual chain-of-thought reasoning for vision-language-action models. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 1702–1713. Cited by: §2.1. [33] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn (2023) Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705. Cited by: §3.3. [34] Y. Zhu, B. Li, Y. Xin, Z. Xia, and L. Xu (2025) Addressing representation collapse in vector quantized models with one linear layer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 22968–22977. Cited by: §2.2. Appendix Appendix A Detailed Training And Evaluation Recipes A.1 Settings for Data Collection in Simulator We conducted experiments in the RoboTwin simulator using two Piper manipulators separated by a distance of 0.6m as the experimental platform. A total of 12 tasks were collected, with 1,600 demonstrations per task. The data collection configuration is shown in the Table A.1. Table A.1: Settings for Data Collection in Simulator Parameter Value Save Frequency 15 Embodiment Piper Random Background True Cluttered Table False Clean Background Rate 0.02 Random Head Camera Distance 0.03 Random Table Height 0.03 Random Light True Crazy Random Light Rate 0.02 Head Camera Type D435 A.2 Dataset Statistics and Task Metrics Table A.2 summarizes the task names, average trajectory lengths, and a selected instruction for each of the 12 tasks in the simulation.Appendix B.1 illustrates some of the tasks and scenarios used in the simulation environment. Table A.2: Dataset Statistics and Task Metrics Task Name Avg. Step Num Task Example Description Beat Block Hammer 67 Grab the silver hammer and use it to hit. Click Bell 52 Press the center top of the blue bell. Handover Mic 134 Grasp the dark blue microphone and pass it across. Move Can Pot 90 Lift the smooth sauce can, place it by the gray kitchen pot. Move Pillbottle Pad 88 Hold the white bottle and position it on the pad. Move Playingcard Away 70 Slide the box for playing cards off the table outward. Pick Diverse Bottles 75 Grab the plastic bottle, pick up the yellow body bottle. Place Mouse Pad 88 Grab the dark gray mouse and drop it on the black mat. Place Container Plate 92 Move the white cup and drop the container onto the round plate. Place Phone Stand 82 Carry the flat phone to the green phone rack. Place Burger Fries 141 Move the box for hamburg and the red fries box to the tray. Shake Bottle 133 Shake the orange bottle after lifting it. A.3 Training Recipe The overall training procedure is divided into two stages. In the first stage, the action tokenizer is trained, while in the second stage, the vision-language-action (VLA) model is trained.We train the model using a total of 19,200 trajectories collected from 12 tasks in Appendix A.1. The detailed training recipe is shown in Table A.3. Table A.3: Training recipes Config Stage 1 Stage 2 batch size 1024 64 training epochs 200 10 optimizer AdamW AdamW learning rate 5×10−55× 10^-5 1×10−41× 10^-4 learning rate schedule cosine cosine warmup epochs 0 0 commitment coefficient 1.0 1000.0 adversarial coefficient 0.1 1.0 random horizontal flip true false A.4 Evaluation recipes For the evaluation on the Robotwin task, we conduct experiments on a single RTX 4090 GPU. The detailed evaluation procedure can be found at https://robotwin-platform.github.io/doc/index.html. Appendix B Visualization B.1 Visualization on simulation tasks We selected three task scenarios from RoboTwin, namely move_pillbottle_pad, place_burger_fries, and click_bell, and took a partial segment of each scenario, as shown in the Figure 3. Figure 3: Some of the simulation tasks B.2 Visualization on real world scenes In the real-robot experiments, we selected three tasks, namely Click Bell, Place Container Plate, and Pick Diverse Bottles, and took a partial segment of each scenario, as shown in the Figure 4. Figure 4: Three task scenarios Appendix C Ablation study results C.1 Ablations on model generalization We selected a total of four tasks for task generalization and scene generalization, and took a partial segment of each scenario, as shown in the Table C.1. Table C.1: Performance on generalization tasks Task Name SR Shake Bottle Horizontally 0.95 Place Empty Cup 0.25 Handover Mic 0.10 Place Container Plate 0.16 C.2 Ablations on tokenization granularity We observed examples where different actions were mapped to identical tokens by the state-agnostic tokenizer, whereas the state-aware tokenizer mapped them to distinct tokens, as illustrated in Figure 5 and Table LABEL:table:Similarity_comparison_between_four_actions_and_the_first_action. Furthermore, we quantified the changes in reconstruction loss and codebook utilization after incorporating state information, as shown in Figure LABEL:fig:action_tokenizer. These results demonstrate that incorporating state information enables finer-grained tokenization. Table C.2: Similarity comparison between four actions and the first action Label Action Data Shape Cosine Similarity to Action 1 a Action 1 8×78× 7 1.000000 b Action 2 8×78× 7 0.999984 c Action 3 8×78× 7 0.998856 d Action 4 8×78× 7 0.999998 Figure 5: Visualization on tokenization granularity