Paper deep dive

Grounded World Model for Semantically Generalizable Planning

Quanyi Li, Lan Feng, Haonan Zhang, Wuyang Li, Letian Wang, Alexandre Alahi, Harold Soh

Year: 2026Venue: arXiv preprintArea: cs.ROType: PreprintEmbeddings: 71

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%

Last extracted: 4/14/2026, 2:45:18 AM

Summary

The paper introduces the Grounded World Model (GWM), a framework for visuomotor Model Predictive Control (MPC) that operates within a vision-language-aligned latent space. By leveraging pretrained foundation models like Qwen3-VL-Embedding, GWM enables robots to perform semantically generalizable planning based on natural language instructions. The authors also propose the WISER benchmark to evaluate semantic generalizability, demonstrating that GWM-MPC significantly outperforms traditional Vision-Language-Action (VLA) models on unseen tasks.

Entities (5)

WISER · benchmark · 99%Grounded World Model · model-architecture · 98%Qwen3-VL-Embedding · foundation-model · 97%Model Predictive Control · methodology · 96%Rendering-based Action Tokenization · technique · 94%

Relation Signals (3)

GWM-MPC → evaluatedon → WISER

confidence 98% · On the proposed WISER benchmark, GWM-MPC achieves a 87% success rate

GWM → uses → Qwen3-VL-Embedding

confidence 97% · GWM learns the transition function in the latent space of a pretrained multi-modal retrieval model, Qwen3-VL-Embedding.

Grounded World Model → operatesin → vision-language-aligned latent space

confidence 95% · we propose to learn a Grounded World Model (GWM) in a vision-language-aligned latent space.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:In Model Predictive Control (MPC), world models predict the future outcomes of various action proposals, which are then scored to guide the selection of the optimal action. For visuomotor MPC, the score function is a distance metric between a predicted image and a goal image, measured in the latent space of a pretrained vision encoder like DINO and JEPA. However, it is challenging to obtain the goal image in advance of the task execution, particularly in new environments. Additionally, conveying the goal through an image offers limited interactivity compared with natural language. In this work, we propose to learn a Grounded World Model (GWM) in a vision-language-aligned latent space. As a result, each proposed action is scored based on how close its future outcome is to the task instruction, reflected by the similarity of embeddings. This approach transforms the visuomotor MPC to a VLA that surpasses VLM-based VLAs in semantic generalization. On the proposed WISER benchmark, GWM-MPC achieves a 87% success rate on the test set comprising 288 tasks that feature unseen visual signals and referring expressions, yet remain solvable with motions demonstrated during training. In contrast, traditional VLAs achieve an average success rate of 22%, even though they overfit the training set with a 90% success rate.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

70,476 characters extracted from source content.

Expand or collapse full text

Grounded World Model for Semantically Generalizable Planning Quanyi Li Independent &Lan Feng11footnotemark: 1 EPFL &Haonan Zhang Beihang University &Wuyang Li EPFL Letian Wang University of Toronto &Alexandre Alahi EPFL &Harold Soh NUS Equal contribution. Code is available at https://github.com/QuanyiLi/gwm-wiser Abstract In Model Predictive Control (MPC), world models predict the future outcomes of various action proposals, which are then scored to guide the selection of the optimal action. For visuomotor MPC, the score function is a distance metric between a predicted image and a goal image, measured in the latent space of a pretrained vision encoder like DINO and JEPA. However, it is challenging to obtain the goal image in advance of the task execution, particularly in new environments. Additionally, conveying the goal through an image offers limited interactivity compared with natural language. In this work, we propose to learn a Grounded World Model (GWM) in a vision-language-aligned latent space. As a result, each proposed action is scored based on how close its future outcome is to the task instruction, reflected by the similarity of embeddings. This approach transforms the visuomotor MPC to a VLA that surpasses VLM-based VLAs in semantic generalization. On the proposed WISER benchmark, GWM-MPC achieves a 87% success rate on the test set comprising 288 tasks that feature unseen visual signals and referring expressions, yet remain solvable with motions demonstrated during training. In contrast, traditional VLAs achieve an average success rate of 22%, even though they overfit the training set with a 90% success rate. 1 Introduction Figure 1: Compared to existing World Models like DINO-WM and JEPA-WM, Grounded World Model enables goal specification via natural language, enabling a new approach to build VLA. A world model is inherently a state transition function that can predict future outcomes given the current state and a sequence of actions or a trajectory [20], enabling the agent to understand, predict, and plan within the physical world [1, 51]. Planning with world models is achieved through Model Predictive Control (MPC), where a batch of candidate trajectories is proposed and fed into the world model to predict their outcomes. Subsequently, the trajectory yielding the minimum cost is executed in the environment. To capture sufficient dynamic and semantic details, modern world models are usually trained with videos featuring realistic physics. During training, the current and future states are represented in either pixel space [7, 24, 61] or latent space [1, 63, 20]. Latent world models, such as DINO-WM [63] and JEPA-WM [51], have shown great potential for visuomotor planning, as they circumvent computationally expensive pixel reconstruction. For latent world models where state transition is defined in the latent space, the score function used for MPC is usually Mean Squared Error (MSE) between the embedding of each predicted future and that of the goal image. However, obtaining the goal image before task execution is challenging, especially for novel tasks where no demonstration is available. Furthermore, a goal image is not a human-friendly interface, compared to natural language, yet its use in the context of the latent world model has remained unexplored. In this work, we propose Grounded World Model (GWM) that operates within a vision-language-aligned latent space, allowing it to ground predicted future outcomes to specific semantics. Specifically, GWM learns the transition function in the latent space of a pretrained multi-modal retrieval model, Qwen3-VL-Embedding [33]. This foundation model can encode not only images and text, but also videos into a shared embedding space, where cosine similarity can be computed. It can be used off-the-shelf as the score function to select the best-matching robot behavior video, given the instruction. Compared to image-text contrastive models (e.g., CLIP [43]), it is more capable of understanding temporal action sequences, benefiting robot behavior recognition. As shown in Fig. 1, we use GWM to predict future outcomes for multiple candidate trajectory proposals in the foundation model’s latent space, and execute the trajectory that yields the highest cosine similarity against the instruction. We refer to this Vision Language Action (VLA) system as GWM-MPC. Unlike VLAs built by fine-tuning pretrained Vision-Language Models (VLMs), where knowledge forgetting can occur due to weight updates [57, 47, 59, 34, 65, 56, 25, 55], GWM leverages the pretrained latent space to learn the transition function without altering the foundation model. Consequently, GWM largely preserves the multi-modal world knowledge of Qwen3-VL-Embedding. Integrating it into MPC disentangles action generation and semantic understanding, effectively translating video understanding capabilities into semantically generalizable planning. As a result, GWM-MPC generalizes to novel visual signals and referring expressions, even those requiring active reasoning, as long as the motions required to complete the task have been demonstrated previously. Figure 2: Experimental results on WISER for VLAs. The success rate gap on training and test tasks indicates the semantic generalizability. The larger the gap, the worse the generalizability. To benchmark semantic generalizability, we introduce the World-knowledge Integrated Semantic Embodied Reasoning (WISER) benchmark. It consists of 24 subsets corresponding to distinct categories of world knowledge, such as numbers, food, animals, and landmarks. Each subset has 12 training or test tasks, yielding 288 tasks in total for either the training or the test sets. The test tasks are constructed with world knowledge and referring expressions that are unseen during training. Despite this, the motions required to complete the test tasks are already demonstrated during training, making the test tasks inherently solvable. The objective is to learn from the training tasks and generalize to the test tasks in a zero-shot manner. If VLAs indeed inherit knowledge from pretrained VLMs, they must be able to recall the correct motions in zero-shot tests, even when the visual signals and referring expressions are previously unseen. However, Fig. 2 shows that traditional VLAs fail to generalize with an average success rate of 22% during test, even though they overfit the training to an average success rate of 90% across all 288 tasks. Some VLAs struggle to generalize even though they manage to complete all training tasks without a single failure. In contrast, GWM-MPC solves 87% of test tasks, demonstrating strong semantic generalization and suggesting our approach is a promising alternative to build VLAs. Additionally, the rendering-based action encoder used by GWM is training-free and embodiment-agnostic, enabling zero-shot generalization to the xArm6 robot despite its different action space, kinematics, and appearance. Ablation studies confirm that the system is robust to hyperparameters and its performance is bottlenecked by the foundation model, pointing to a clear direction for future improvement. 2 Method The goal of VLA is to inherit the semantic generalizability of pretrained foundation models [6, 5]. We thus begin to formulate the semantic generalization problem and introduce our GWM-MPC solution. 2.1 Semantic Generalization in Planning We assume there is a training dataset that consists of I trajectories, denoted as =1,…,ID=\T^1,…,T^I\. Each trajectory is a sequence of transitions i=(o0i,j0i,a0i,ℓi),…,(oTi,jTi,aTi,ℓi)T^i=\(o^i_0,j^i_0,a^i_0, ^i),…,(o^i_T,j^i_T,a^i_T, ^i)\, where otio^i_t, jtij^i_t, and atia^i_t respectively denote the camera images, joint positions, and actions at timestep t for trajectory i. The variable ℓi ^i represents the natural language task instruction, which remains constant throughout the entire episode. Using the dataset D, we aim to learn a policy that maps the current observation to an action chunk: at:t+c=f(ot,jt,ℓ)a_t:t+c=f(o_t,j_t, ), where c is the chunk size. During inference, these actions are sequentially executed in the environment until a new observation (ot+c+1,jt+c+1)(o_t+c+1,j_t+c+1) is received, at which point the policy generates a new action chunk. This closed-loop rollout terminates once the task ℓ is completed. A naive way to build such a policy to solve the demonstrated task, on which D was collected, is through trajectory or action chunk retrieval. This method involves simply iterating through the dataset D to find the transition that best matches the current observation (ot,jt,ℓ)(o_t,j_t, ): at:t+c=ak∗:k∗+ci∗,where(i∗,k∗)=arg⁡min(i,k)∈dist((ot,jt,ℓ),(oki,jki,ℓi))a_t:t+c=a^i^*_k^*:k^*+c, (i^*,k^*)= (i,k) \;dist ((o_t,j_t, ),(o^i_k,j^i_k, ^i) ) (1) Here, V is the set of all valid index pairs in the dataset D, where i∈T^i and k denotes the timestep within that trajectory. It is reminiscent of the early non-parametric machine learning method, KNN, and N=1N=1 here. The dist(⋅,⋅)dist(·,·) works as the kernel function, which can be a learnable one, especially when the feature is in a high-dimensional space like images. In this case, the distance can be computed in a latent space for action retrieval [23], enabling generalization to new visual inputs. Conceptually, a parametric end-to-end policy pθ(at:t+c|ot,jt,ℓ)p_θ(a_t:t+c|o_t,j_t, ) can be viewed as retrieving trajectories from a continuous proxy dataset ′D , which augments D by interpolating between the discrete demonstrations in D to generalize to novel, yet in-distribution, datapoints. Despite this, we still do not expect neural networks to produce trajectories that deviate too much from those demonstrated in D, especially when training is performed from scratch, and D is not sufficiently large. For the same reason, the language instruction ℓ tends to serve as a one-hot label [34], inducing poor novel instruction following ability; Moreover, the model may exploit visual shortcuts, selecting actions based on spurious correlations [54], such as associating the actions with the scene layout. Both issues indicate a lack of genuine vision-language understanding by the model, preventing extrapolation. VLAs are proposed to address this by initializing from pretrained foundation models. They are thus expected to possess the capability: semantic generalization. This aims at making a policy trained with D go beyond language instructions and visual signals in D. Ideally, regardless of how the current task instruction ℓ and observation oto_t appear—and no matter how significantly they differ from those in the training dataset D—the policy should still complete the task, as long as the motions required by this task have been demonstrated during training. This generalization is supposed to be achieved by inheriting open-world knowledge and leveraging the aligned vision-language feature space from the pretrained VLM. However, our experiments show that VLAs do not exhibit this capability. 2.2 Model Predictive Control (MPC) The MPC framework typically comprises three steps: proposing candidate trajectories, predicting their future states or outcomes, and selecting the optimal trajectory using a score function. We use KNN to propose trajectories for three reasons. First, as discussed in Section 2.1, a parametric policy fundamentally retrieves from a continuous proxy of D and cannot generalize to motions beyond the demonstrations; training a separate action generation model pθ(at:t+c|jt)p_θ(a_t:t+c|j_t) would thus introduce additional learnable parameters without expanding the reachable trajectory space. Second, sampling-based methods like CEM [45] and gradient-based methods like Langevin MCMC [53] must search a high-dimensional action space without informative priors, making them inefficient when the set of valid trajectories is small and sparse. KNN sidesteps both issues by directly retrieving demonstrated trajectories from D, requiring no learned parameters and no open-ended search. Proposals are generated through Eq. 1 with a simplified kernel function MSE(jt,jki)MSE (j_t,j^i_k ), by iterating the demonstration dataset D and looking for the available future actions at joint position jtj_t. In this work, we keep the number of action proposals N=12N=12 for subsequent future outcome prediction and scoring: t:t+c=ak:k+ci∣(i,k)∈ℐ∗,whereℐ∗=top-N arg⁡min(i,k)∈MSE(jt,jki)A_t:t+c= \a^i_k:k+c (i,k) ^* \, ^*= (i,k) top-N \;MSE (j_t,j^i_k ) (2) If the robot behaviors are not restricted to those in D, trajectory proposals can be generated using other methods, such as grasping pose synthesis algorithms or visuomotor policies. Unlike VLMs that produce discrete text tokens [32, 37], large pretrained retrieval models can naturally produce a continuous scalar between 0-10-1, making them a better choice for a score function. In this work, we use Qwen3-VL-Embedding [33]. It comprises a vision encoder to map images and videos into the language feature space, followed by a transformer backbone that integrates tokenized text and visual features into a unified embedding z. Retrieval models serve as a score function by encoding the target task or user instruction into embeddings zgz_g and the future outcome of N trajectories at timestep t into zt1,…,ztN\z^1_t,…,z^N_t\. Finally, the policy selects the sequence of actions whose predicted future outcome embedding exhibits the highest cosine similarity with the instruction embedding: at:t+c∗=at:t+cn∗,wheren∗=arg⁡maxn∈1,…,Nztn⋅zg‖ztn‖2‖zg‖2,at:t+c∗∈t:t+ca^*_t:t+c=a^n^*_t:t+c,\ where\ n^*= n∈\1,…,N\ \; z^n_t· z_g\|z^n_t\|_2\|z_g\|_2,\ a^*_t:t+c _t:t+c (3) For a pixel-space world model, obtaining future outcome embeddings zt1,…,ztN\z^1_t,…,z^N_t\ requires predicting future observed images ot+1:t+cn∼p(⋅|ot,at:t+cn)o^n_t+1:t+c p(·|o_t,a^n_t:t+c) for each proposed trajectory in t:t+cA_t:t+c, so they can be encoded by the retrieval models to get ztnz_t^n. Though training pixel-space prediction models is feasible given D [61], by using pretrained video diffusion models [16], reconstruction in pixel space captures redundant details and is expensive and less efficient on both training and inference. Figure 3: The training and inference workflow of GWM-MPC. All proposed trajectories are tokenized into images by rendering the robot URDF with the same camera extrinsics and intrinsics as the third-person RGB camera. Thus, observation and actions can be uniformly encoded as ete_t by the vision encoder of Qwen3-VL-Embedding. The GWM then produces the future outcome embeddings ptp_t for each candidate action. The foundation model’s backbone finally projects those embeddings to a shared vision-language space and gets zt0,…,ztN\z^0_t,…,z^N_t\. A sequence of actions is selected if it leads to the future with maximum cosine similarity against the goal embedding zgz_g that is derived from the instruction ℓ with the same foundation model. During training, ground truth future is used to calculate the MSE loss in the vision encoder’s latent space. Notably, no language supervision is required. 2.3 Grounded World Model To address these problems, we propose training the world model within the latent space of the multi-modal retrieval model from scratch. We call our model Grounded World Model (GWM), as its output can be grounded to specific semantics. Its training utilizes the representation space of the foundation model, whose weights remain frozen. Consequently, its vision-language understanding ability and world knowledge are largely preserved. GWM can optimize the scoring step, as shown in Eq. 3, by directly predicting the latent embedding of the future state as ztn∼p(⋅|ot,at:t+cn)z^n_t p(·|o_t,a^n_t:t+c). The full inference and training process is depicted in Fig. 3. We introduce the details as follows. Rendering-based Action Tokenization (RAT). To predict the embedding ztz_t, the model must encode both the current observation and the sequence of actions. Since the WISER benchmark utilizes target joint positions as the action space, we can sequentially render these actions into images using the third-person main camera parameters and the robot’s URDF. This approach allows us to leverage the feature extraction capabilities of the Qwen vision encoder without introducing additional learnable parameters. This method is highly generalizable: even when employing the delta gripper pose as the action space, inverse kinematics can be used to compute joint positions for future timesteps, making the rendering feasible. Therefore, this approach is embodiment-agnostic and can serve as a unified tokenizer for robot actions and states. In our ablation study, we demonstrate that RAT outperforms the traditional learnable action encoder and enables zero-shot generalization to the xArm6 robot. Training and Inference. The encoding produces a feature vector with the vision encoder of the foundation model et=E(ot,at:t+c)e_t=E(o_t,a_t:t+c). Then the GWM predicts the outcome of the action trajectory by pt=Pθ(et)p_t=P_θ(e_t). PθP_θ is parameterized with a standard transformer model. The detailed model architecture and configuration are available in the appendix 6.6. During training, the supervision signal is derived from ground truth future image sequences, e¯t=E(ot:t+c) e_t=E(o_t:t+c). Since ete_t, ptp_t, and e¯t e_t share the same shape, we can directly feed ptp_t into the foundation model’s backbone to obtain ztz_t without projection layers. This is useful in experiments where we perform a sanity check of GWM and compute the performance upper bound. If we pass the ground truth future embedding e¯t e_t into the backbone, the MPC system degrades to a purely retrieval-based one. This case replaces the dist(⋅,⋅)dist(·,·) of Eq. 1 with the embedding similarity between the ground truth future observation ok:k+cio^i_k:k+c and ℓ . As a result, the sequence of actions inducing a future that best aligns with ℓ is retrieved from t:t+cA_t:t+c for execution: at:t+c=ak∗:k∗+ci∗,where(i∗,k∗)=arg⁡max(i,k)∈ℐ∗Embedding Similarity(ok:k+ci,ℓ)a_t:t+c=a^i^*_k^*:k^*+c,\ where (i^*,k^*)= (i,k) ^* \;Embedding Similarity(o^i_k:k+c, ) (4) However, demonstrations are unavailable in novel scenarios where visual signals in oto_t differ significantly from those in D, but only the trajectories required to complete task ℓ exist in the training set. The generalizability of GWM thus enables using ptp_t to approximate the unavailable e¯t e_t during the test. 3 WISER Benchmark Figure 4: Overview of the WISER Benchmark. Observations include the instruction ℓ , current joint positions and gripper states jtj_t, and camera input oto_t. The benchmark comprises 24 world-knowledge categories, each partitioned into training and held-out test splits. Notably, all images, descriptions, and cube colors in the test set are entirely novel and non-overlapping with the training data. For example, even though cubes occupy identical positions (e.g., second from left), the spatial referring expressions and colors differ between the training and test. In each split, cube ordering is randomized across categories. Only 12 unique trajectories are shared by the training and test tasks. Training and Test Split. To evaluate the semantic generalizability of planners built upon pretrained foundation models, we build the World-knowledge Integrated Semantic Embodied Reasoning (WISER) Benchmark, where each task requires the robot to pick one cube and place it onto a mark or image. Unlike dexterous motion, the trajectory required for each task is simple and rigid. We intentionally adopt this design, so the test-time failure can be directly attributed to poor semantic generalization rather than failing to learn complex motions. The benchmark comprises 24 categories. For each category, there is a training scene and a test scene. Both scenes have the same layout with four cubes in front of the robot and three images in front of the cubes. Thus, for either training scene or test scene, there are 4×3==12 pick-and-place tasks. For the training and test sets in the WISER benchmark, there are 12×24==288 tasks, respectively. The difference between training tasks and test tasks can be found in Fig. 4. In addition to the knowledge reflected in the three images, the cube colors differ between the training and test scenes. Furthermore, for test tasks, the methods for referring to the cube to pick and the place to drop have never been shown during training. If the policy can inherit the world knowledge and the open-vocabulary visual signal understanding ability from the foundation models after training, it is expected to complete the test tasks by retrieving or recalling the correct trajectory from the 12 unique trajectories shared between the training and test tasks. In the appendix 6.7, we provide visualizations for all tasks and the task instructions. Simulation. We developed the benchmark using ManiSkill [48], leveraging its GPU-parallelization to simultaneously simulate all 12 tasks across either training or test scenes. Demonstrations were collected solely on training tasks using a Franka Panda robot via MPlib [22]. The controller utilizes privileged information, such as goal positions and cube poses, to perform motion planning with a 100% success rate. The PD controller then tracks the planning results, a sequence of target joint positions, at a control frequency of 20Hz and simulation frequency of 100Hz. During data collection, we record the main camera stream (224×448), wrist camera stream (128×128), joint positions, gripper states, task instruction ℓ , and actions ata_t. We collect 6 trajectories per task, with the robot’s initial states randomized to increase diversity. This results in a training dataset of 6×288==1728 trajectories, aimed at expanding state-space coverage and mitigating compounding errors when training VLAs. During closed-loop evaluation, the robot is consistently reset to a fixed retract pose. We impose a maximum limit of 120 steps (6 seconds) for both collection and evaluation. The dataset has LeRobot V2.1 and V3.0 versions [9]. We also provide the RLDS version [44]. Metrics. We employ three binary metrics to evaluate picking, placement, and overall task success. Grasp indicates whether the correct cube is successfully picked. Reach denotes whether the gripper’s Tool Center Point (TCP) reaches the designated goal with nearly zero speed, even though the grasp fails. Success signifies that the cube is correctly placed at the goal point, defined as Success=Grasp×ReachSuccess=Grasp×Reach. We evaluate all policies on both the training tasks and the test tasks. Each task is evaluated once, and the metrics are averaged across the 288 training or test tasks. 4 Experiments Appendix 6.1 provides implementation details for baselines. For GWM, we use the same training dataset, but exclude language labels and wrist-camera observations. We set the world model prediction horizon to c=c=60 steps. Rather than feeding the model the full 60-step future action sequence, we down-sample the rendered sequence of images (actions) into 6 keyframes for inference efficiency. Consequently, the model only needs to predict the embeddings of 6 future frames to represent the outcome of executing 60 steps. Despite this, the MPC replans every 20 steps. For each inference, N=12N=12 sequences are proposed using Eq. 2 and are subsequently scored according to Eq. 3. The sequence of actions yielding the maximum cosine similarity to zgz_g is selected for execution. In practice, the zgz_g is obtained by encoding not only the task prompt but the system prompt, and the current observation for best scoring accuracy. The final score is also a weighted combination of picking and placing tasks. Details on the score function design are available in the appendix 6.2. Table 1: Evaluation Results for SOTA VLAs and GWM-MPC on the WISER Benchmark. Method H100 GPU Hours Training Set Test Set Grasp Reach Success Grasp Reach Success InstructVLA[56] 70 0.98 0.92 0.89 0.79 0.51 0.47 SmolVLA[46] 75 0.99 1.00 0.99 0.29 0.31 0.08 Wall-OSS[57] 80 1.00 1.00 1.00 0.68 0.50 0.40 GR00T-N1.6[42] 100 1.00 1.00 1.00 0.72 0.18 0.18 InternVLA-A1[10] 100 1.00 0.91 0.88 0.63 0.40 0.26 π0.5 _0.5[25] 100 1.00 0.99 0.99 0.70 0.38 0.26 π0 _0[3] 100 1.00 1.00 1.00 0.47 0.14 0.08 XVLA[62] 100 1.00 0.88 0.88 0.44 0.17 0.17 UniVLA[8] 120 0.79 0.62 0.63 0.38 0.18 0.13 Motus[2] 300 0.78 0.72 0.72 0.34 0.14 0.14 Baseline Average - 0.95 0.90 0.90 0.54 0.29 0.22 GWM-MPC 20 0.97 0.95 0.92 0.99 0.88 0.87 GWM Ablation Study DreamDojo-MPC [16] 24 0.22 0.41 0.15 0.28 0.44 0.17 GWM-MPC-AC 20 0.91 0.77 0.74 0.47 0.42 0.24 GWM-MPC-xArm6 - 0.96 0.91 0.87 0.97 0.86 0.83 GWM-MPC w/ 12 12D 20 0.97 0.81 0.78 0.98 0.74 0.72 GT-MPC - 0.97 0.92 0.90 1.00 0.93 0.93 MPC w/o GWM - 0.27 0.41 0.08 0.26 0.44 0.09 4.1 Main Results VLAs. The main results are presented in Table 1, where the best performance is highlighted in bold and the second best is underlined. None of the VLAs generalize well to the test tasks, achieving an average test success rate of only 22%, despite these tasks requiring the same skills demonstrated in training. For some VLAs like SmolVLA and π0 _0, they achieve nearly 100% success rate on training tasks, while during test, their performance is even worse than random trajectory retrieval (8% vs. 1/12=1/12=8.3%). The failure mode in the test scenes is consistent across all baselines: they typically grasp the wrong cube or place it onto a random image. The top-performing VLAs are WALL-OSS and InstructVLA. Both models are pretrained with an auxiliary embodied VQA task, which improves the success rate by retaining knowledge from the foundational VLMs. Despite this, in appendix 6.4, we show that the base VLM of InstructVLA can localize the correct destination for 81%81\% test scenarios, whereas finetuning still brings some knowledge forgetting, resulting in a 51% TCP reaching success rate. In appendix 6.2, we also show that InstructVLA overfits to sentence structures and loses the ability to understand decomposed instructions. Motus demands more computation to do the auxiliary task: pixel-space future prediction. For Motus and UniVLA, the gap between the training and test performance still reflects their poor semantic generalizability. Among all baselines, GR00T-N1.6 and InternVLA-A1 utilize a relative (delta) action space, which does not improve generalizability according to the results. All VLAs demonstrate some generalizability during the cube-picking stage, which doesn’t require world knowledge yet but just the ability to recognize unseen cube colors and referring expressions, achieving a 54% test average grasping success rate. InstructVLA and GR00T-N1.6 even manage to grasp the cube in over 70% of test tasks. We attribute this to the large number of cube-picking demonstrations present in the pretraining datasets. Our experiments cover most of the VLA training recipes, such as Latent Action Pretraining, Knowledge-Insulation, Mixture-of-Transformers, VQA auxiliary task, and video-action joint training. We thus confirm that poor semantic generalizability is a common issue for VLM-based VLAs. GWM. The GWM-MPC achieves the best test-scene performance, yielding an 87% success rate across 288 test tasks that feature unseen referring expressions, spatial relationship descriptions, and visual signals. This demonstrates that GWM can effectively capture scene semantics by recognizing predicted future robot behaviors and their interactions with scene objects, specifically cubes and images. As GWM-MPC retrieves trajectories from the training dataset, the failure can only result from the incorrect scoring and action selection. In other words, the scoring accuracy of Qwen3-VL-Embedding bounds the performance of GWM-MPC. We also use the same dataset D to train an explicit world model, DreamDojo [16]. During inference, it produces a video representing the outcome of a sequence of actions, which is used in the same way as GWM in the MPC procedures. DreamDojo learns to reconstruct pixels quickly and accurately, while we found it struggles to follow the actions. For example, it may generate videos grasping the cube on the leftmost side, while the actions sent to DreamDojo are to grasp the cube next to the leftmost one. One possible reason is that only the ego-centric human videos are used to pretrain the latent action encoder of the DreamDojo. RAT. We also train an alternative model that encodes raw robot actions (represented as a list of numerical values) using a learnable module, which then feeds the resulting embedding into the transformer alongside the embedding of the current observation E(ot)E(o_t). The detailed model architecture is provided in Appendix 6.6. The evaluation results for this model are denoted as GWM-MPC-AC. This specific action tokenization scheme exhibits the same training-test performance gap as VLAs. We attribute this to the fact that image-represented actions align more easily with the current observation by utilizing the same vision encoder to extract features. Furthermore, RAT enables zero-shot cross-embodiment generalization. To demonstrate this capability, we collected the 12 unique trajectories using an xArm6 robot, recording only joint positions and gripper states to propose future actions following Eq. 2. We then reused the GWM, trained exclusively on Panda data, to convey the outcome of the xArm’s movements to the score module. The experiment, denoted as GWM-MPC-xArm6, shows that RAT and GWM enable zero-shot generalization to a new embodiment with different action spaces, forward kinematics, and appearance, achieving 87% and 83% success rates on training and test tasks, respectively. Training & Inference Efficiency. Training the GWM is computationally efficient: it requires only 20 GPU hours on our proposed WISER benchmark. Moreover, our approach avoids action learning and thus mitigates data reliance by employing KNN-based or retrieval-based action proposals. This aligns with recent findings [14], which suggest that retrieval-based planners can outperform purely learning-based alternatives while requiring fewer demonstrations. To evaluate data efficiency, we trained an additional GWM on a reduced dataset, denoted as GWM-MPC w/ 12 12D. This subset covers only 288//2==144 training tasks from half of the 24 categories, providing just a single demonstration per task. The resulting GWM-MPC-12 12D model maintains a competitive 72% test success rate. The inference efficiency comparison is in appendix 6.3. Since N=12N=12 GWM inferences are required for MPC, our method underperforms all VLAs that only require generating one trajectory. Performance Upper Bound & Sanity Check. The performance of GWM on the training tasks is lower than that of other purely learning-based methods. This is because the Qwen3-VL-Embedding bottlenecks our system’s performance. In the GT-MPC experiment, we feed the backbone with the ground-truth future representation, e¯t e_t, of each sequence of actions. This setup either excludes the GWM entirely or assumes its prediction ptp_t has zero error relative to e¯t e_t, thereby establishing the theoretical upper bound of the entire system. As shown in Table 1, GT-MPC fails to achieve a 100% success rate on both training and test tasks. Surprisingly, the standard GWM-MPC exhibits a slightly higher success rate than GT-MPC on the training tasks (92% vs. 90%). This suggests that GWM’s prediction introduces little noise and may even regularize the scoring process. Additionally, we feed the backbone with ete_t, which is the embedding of the current observation oto_t alongside the sequence of actions at:t+ca_t:t+c. This forms the “MPC w/o GWM” experiment, which serves as a sanity check by isolating the GWM module. The results show that without a world model to predict future outcomes, the system is reduced to a random trajectory selector, failing on almost all tasks. 4.2 Score Function Ablation Study Figure 5: Ablation results on the GT-MPC for planning-related hyperparameter choosing. Because the GWM’s performance is bounded by the Qwen3-VL-Embedding, hyperparameter selection can be determined by running the GT-MPC directly. As illustrated in Fig. 5, we investigate the influence of the replanning interval (default is 20), the world model prediction horizon (default is 60), and the future subsampling rate or number of future keyframes (default is 6). The results indicate that the Qwen3-VL-Embedding is relatively robust to the replanning interval, although it achieves optimal performance at an interval of 20 on the WISER benchmark. We also test the trained model, GWM-MPC, with different replanning intervals and obtain consistent results. However, for both the prediction horizon and the future subsampling rate, specific thresholds must be met before achieving satisfactory performance. If the prediction horizon is too short, the foundation model cannot infer the policy’s intention. Additionally, an extreme subsampling rate, such as keeping only 2 or 4 frames from a 60-frame future, confuses the Qwen3-VL-Embedding, preventing it from accurately scoring the robot behaviors. Furthermore, the ablation study on model size demonstrates that the larger model indeed excels over the smaller one in comprehending videos and predicting futures. We also evaluate the Perception Encoder [4] and find that it scores videos with zero accuracy. In appendix 6.5, we built GT-MPC for LIBERO-goal [39] and find it can accurately select actions for 80% tasks in zero-shot. 5 Related Work World Models. Given a sequence of actions and the current observation, a world model predicts what will happen next [20]. Most existing research on world models aims to model the transitions in pixel space, taking images as input and predicting another set of images. One application of these pixel-space world models is policy evaluation and data synthesis [18, 50, 21, 38, 49, 24, 16]. Another application is model-based planning [66, 64, 61, 16]. Rather than predicting the future in explicit representations like images, some works propose predicting the future on the latent space of pretrained models [17, 63, 1, 51, 60, 13], which improves learning efficiency by avoiding pixel-level reconstruction. Our work extends this thread of research by allowing the specification of goals with natural, open-vocabulary instructions rather than goal images, which are hard to obtain and interact with. Similar to previous works [61, 28, 31, 32, 11, 37], GWM-based planning follows the three common MPC steps: scoring, ranking, and selection. However, GWM operates in the latent space, where grounding is easier and facilitates out-of-distribution (OOD) generalization [19]. In addition, prior works usually use sampling-based methods like CEM [45] or gradient-based methods [15] to propose or search trajectories, while we use K-Nearest Neighbors to retrieve skills from the training dataset for the reasons discussed in Section 2.1 and Section 2.2. VLAs and Benchmarks. Our method enables the construction of a VLA system that acts according to visual inputs and natural language instructions. Unlike our MPC system, most VLAs are built end-to-end based on pretrained VLMs by pretraining on large-scale robot data and fine-tuning on target tasks [34, 25, 2, 29, 56, 8]. This paradigm was initially proposed to inherit knowledge from pretrained foundation models to achieve semantic generalization [6], allowing them to become generalist policies capable of finishing tasks in zero or a few shots. However, some works have found that VLAs may merely overfit to specific tasks by learning shortcuts, lacking actual semantic generalizability [52, 1, 57, 47, 59, 34, 65, 56, 25, 55]. Furthermore, there are currently no benchmarks available to evaluate how much knowledge from pretrained foundation models has been retained in VLAs, or to measure their semantic generalizability. Most benchmarks collect data on evaluation or test scenarios [41, 12, 65, 40, 35], where the distribution gap between training and testing consists only of visual interference and trivial object pose perturbations. Some recent works have recognized the lack of such benchmarks and thus conducted proprietary semantic generalization experiments in simulation [59, 58, 56] and the real world [27]. The setting of GrinningFace [59] is close to ours, while the task is simpler with only 3 different motions, and the images to place the cube are from the emoji dataset. Compared to existing options, the proposed WISER benchmark provides a more standard, comprehensive, and scalable way to test the semantic generalizability. 6 Conclusion In this work, we formulate the semantic generalization problem in the context of planning. We argue that policies taking advantage of pretrained vision-language models are supposed to possess the ability to address this problem. We thus design a benchmark to evaluate state-of-the-art VLAs on this, and find that all of them deviate from the goal of inheriting the knowledge from pretrained models or being a generalist. On the other hand, we find that training a latent world model in a grounded latent space can provide an alternative to build a VLA system for acquiring the semantic generalizability from the pretrained retrieval model. When planning with MPC, the proposed GWM can address 87% of unseen tasks with novel visual signals and object referring expressions, whereas the best VLA achieves only a 47% success rate on the same test tasks. The rendering-based action tokenizer additionally allows cross-embodiment generalization without introducing new parameters to encode actions. The fact that the performance is bottlenecked by the pretrained model points out a future direction to fine-tune the Qwen3-VL-Embedding with robot data for further improvements. References [1] M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, Mojtaba, Komeili, M. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholus, S. Arnaud, A. Gejji, A. Martin, F. R. Hogan, D. Dugas, P. Bojanowski, V. Khalidov, P. Labatut, F. Massa, M. Szafraniec, K. Krishnakumar, Y. Li, X. Ma, S. Chandar, F. Meier, Y. LeCun, M. Rabbat, and N. Ballas (2025) V-jepa 2: self-supervised video models enable understanding, prediction and planning. External Links: 2506.09985, Link Cited by: §1, §5, §5. [2] H. Bi, H. Tan, S. Xie, Z. Wang, S. Huang, H. Liu, R. Zhao, Y. Feng, C. Xiang, Y. Rong, H. Zhao, H. Liu, Z. Su, L. Ma, H. Su, and J. Zhu (2025) Motus: a unified latent action world model. External Links: 2512.13030, Link Cited by: Table 1, §5. [3] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky (2026) π0 _0: A vision-language-action flow model for general robot control. External Links: 2410.24164, Link Cited by: Table 1. [4] D. Bolya, P. Huang, P. Sun, J. H. Cho, A. Madotto, C. Wei, T. Ma, J. Zhi, J. Rajasegaran, H. Rasheed, J. Wang, M. Monteiro, H. Xu, S. Dong, N. Ravi, D. Li, P. Dollár, and C. Feichtenhofer (2025) Perception encoder: the best visual embeddings are not at the output of the network. External Links: 2504.13181, Link Cited by: §4.2. [5] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, L. Lee, T. E. Lee, S. Levine, Y. Lu, H. Michalewski, I. Mordatch, K. Pertsch, K. Rao, K. Reymann, M. Ryoo, G. Salazar, P. Sanketi, P. Sermanet, J. Singh, A. Singh, R. Soricut, H. Tran, V. Vanhoucke, Q. Vuong, A. Wahid, S. Welker, P. Wohlhart, J. Wu, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich (2023) RT-2: vision-language-action models transfer web knowledge to robotic control. External Links: 2307.15818, Link Cited by: §2. [6] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, K. Lee, S. Levine, Y. Lu, U. Malla, D. Manjunath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich (2023) RT-1: robotics transformer for real-world control at scale. External Links: 2212.06817, Link Cited by: §2, §5. [7] J. Bruce, M. Dennis, A. Edwards, J. Parker-Holder, Y. Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps, Y. Aytar, S. Bechtle, F. Behbahani, S. Chan, N. Heess, L. Gonzalez, S. Osindero, S. Ozair, S. Reed, J. Zhang, K. Zolna, J. Clune, N. de Freitas, S. Singh, and T. Rocktäschel (2024) Genie: generative interactive environments. External Links: 2402.15391, Link Cited by: §1. [8] Q. Bu, Y. Yang, J. Cai, S. Gao, G. Ren, M. Yao, P. Luo, and H. Li (2025) UniVLA: learning to act anywhere with task-centric latent actions. External Links: 2505.06111, Link Cited by: Table 1, §5. [9] R. Cadene, S. Alibert, F. Capuano, M. Aractingi, A. Zouitine, P. Kooijmans, J. Choghari, M. Russi, C. Pascal, S. Palma, M. Shukor, J. Moss, A. Soare, D. Aubakirova, Q. Lhoest, Q. Gallouédec, and T. Wolf (2026) LeRobot: an open-source library for end-to-end robot learning. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §3. [10] J. Cai, Z. Cai, J. Cao, Y. Chen, Z. He, L. Jiang, H. Li, H. Li, Y. Li, Y. Liu, Y. Lu, Q. Lv, H. Ma, J. Pang, Y. Qiao, Z. Qiu, Y. Shen, X. Shi, Y. Tian, B. Wang, H. Wang, J. Wang, T. Wang, X. Wei, C. Wu, Y. Xie, B. Xing, Y. Yang, Y. Yang, Q. Yu, F. Yuan, J. Zeng, J. Zhang, S. Zhang, S. Zhang, Z. Zhaxi, B. Zhou, Y. Zhou, Y. Zhou, H. Zhu, Y. Zhu, and Y. Zhu (2026) InternVLA-a1: unifying understanding, generation and action for robotic manipulation. External Links: 2601.02456, Link Cited by: Table 1. [11] S. Chen, C. Harrison, Y. Lee, A. J. Yang, Z. Ren, L. J. Ratliff, J. Duan, D. Fox, and R. Krishna (2026) TOPReward: token probabilities as hidden zero-shot rewards for robotics. External Links: 2602.19313, Link Cited by: §5. [12] T. Chen, Z. Chen, B. Chen, Z. Cai, Y. Liu, Z. Li, Q. Liang, X. Lin, Y. Ge, Z. Gu, W. Deng, Y. Guo, T. Nian, X. Xie, Q. Chen, K. Su, T. Xu, G. Liu, M. Hu, H. Gao, K. Wang, Z. Liang, Y. Qin, X. Yang, P. Luo, and Y. Mu (2025) RoboTwin 2.0: a scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. External Links: 2506.18088, Link Cited by: §5. [13] M. Destrade, O. Bounou, Q. L. Lidec, J. Ponce, and Y. LeCun (2025) Value-guided action planning with jepa world models. External Links: 2601.00844, Link Cited by: §5. [14] K. Dreczkowski, P. Vitiello, V. Vosylius, and E. Johns (2025-11) Learning a thousand tasks in a day. Science Robotics 10 (108). External Links: ISSN 2470-9476, Link, Document Cited by: §4.1. [15] P. Florence, C. Lynch, A. Zeng, O. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson (2021) Implicit behavioral cloning. External Links: 2109.00137, Link Cited by: §5. [16] S. Gao, W. Liang, K. Zheng, A. Malik, S. Ye, S. Yu, W. Tseng, Y. Dong, K. Mo, C. Lin, Q. Ma, S. Nah, L. Magne, J. Xiang, Y. Xie, R. Zheng, D. Niu, Y. L. Tan, K. R. Zentner, G. Kurian, S. Indupuru, P. Jannaty, J. Gu, J. Zhang, J. Malik, P. Abbeel, M. Liu, Y. Zhu, J. Jang, and L. ". Fan (2026) DreamDojo: a generalist robot world model from large-scale human videos. External Links: 2602.06949, Link Cited by: §2.2, §4.1, Table 1, §5. [17] R. G. Goswami, A. Bar, D. Fan, T. Yang, G. Zhou, P. Krishnamurthy, M. Rabbat, F. Khorrami, and Y. LeCun (2025) World models can leverage human videos for dexterous manipulation. External Links: 2512.13644, Link Cited by: §5. [18] Y. Guo, L. X. Shi, J. Chen, and C. Finn (2025) Ctrl-world: a controllable generative world model for robot manipulation. External Links: 2510.10125, Link Cited by: §5. [19] P. Gupta, H. Admoni, and A. Bajcsy (2025) Adapting by analogy: ood generalization of visuomotor policies via functional correspondence. External Links: 2506.12678, Link Cited by: §5. [20] D. Ha and J. Schmidhuber (2018) Recurrent world models facilitate policy evolution. External Links: 1809.01999, Link Cited by: §1, §5. [21] D. Hafner, W. Yan, and T. Lillicrap (2025) Training agents inside of scalable world models. External Links: 2509.24527, Link Cited by: §5. [22] Hao Su’s Lab (2024) MPlib: a lightweight motion planning library. Note: GitHub repository External Links: Link Cited by: §3. [23] C. He, X. Liu, G. S. Camps, G. Sartoretti, and M. Schwager (2025) Demystifying diffusion policies: action memorization and simple lookup table alternatives. External Links: 2505.05787, Link Cited by: §2.1. [24] A. Hu, L. Russell, H. Yeo, Z. Murez, G. Fedoseev, A. Kendall, J. Shotton, and G. Corrado (2023) GAIA-1: a generative world model for autonomous driving. External Links: 2309.17080, Link Cited by: §1, §5. [25] P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, M. Y. Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke, A. Walling, H. Wang, L. Yu, and U. Zhilinsky (2025) π0.5 _0.5: A vision-language-action model with open-world generalization. External Links: 2504.16054, Link Cited by: §1, Table 1, §5. [26] K. Jordan (2024) Muon: an optimizer for hidden layers in neural networks. Note: https://kellerjordan.github.io/posts/muon/Accessed: 2026-03-03 Cited by: Table 5. [27] N. Kachaev, M. Kolosov, D. Zelezetsky, A. K. Kovalev, and A. I. Panov (2025) Don’t blind your vla: aligning visual representations for ood generalization. External Links: 2510.25616, Link Cited by: §5. [28] G. Kang, J. Kim, K. Shim, J. K. Lee, and B. Zhang (2025) CLIP-rt: learning language-conditioned robotic policies from natural language supervision. External Links: 2411.00508, Link Cited by: §5. [29] M. J. Kim, C. Finn, and P. Liang (2025) Fine-tuning vision-language-action models: optimizing speed and success. External Links: 2502.19645, Link Cited by: §5, §6.1. [30] D. P. Kingma and J. Ba (2017) Adam: a method for stochastic optimization. External Links: 1412.6980, Link Cited by: Table 5. [31] J. Kwok, X. Zhang, M. Xu, Y. Liu, A. Mirhoseini, C. Finn, and M. Pavone (2026) Scaling verification can be more effective than scaling policy learning for vision-language-action alignment. External Links: 2602.12281, Link Cited by: §5. [32] T. Lee, A. Wagenmaker, K. Pertsch, P. Liang, S. Levine, and C. Finn (2026) RoboReward: general-purpose vision-language reward models for robotics. External Links: 2601.00675, Link Cited by: §2.2, §5. [33] M. Li, Y. Zhang, D. Long, K. Chen, S. Song, S. Bai, Z. Yang, P. Xie, A. Yang, D. Liu, J. Zhou, and J. Lin (2026) Qwen3-vl-embedding and qwen3-vl-reranker: a unified framework for state-of-the-art multimodal retrieval and ranking. External Links: 2601.04720, Link Cited by: §1, §2.2. [34] Q. Li (2025) Task reconstruction and extrapolation for π0 _0 using text latent. External Links: 2505.03500, Link Cited by: §1, §2.1, §5. [35] X. Li, K. Hsu, J. Gu, K. Pertsch, O. Mees, H. R. Walke, C. Fu, I. Lunawat, I. Sieh, S. Kirmani, S. Levine, J. Wu, C. Finn, H. Su, Q. Vuong, and T. Xiao (2024) Evaluating real-world robot manipulation policies in simulation. External Links: 2405.05941, Link Cited by: §5. [36] Y. Li, F. Wei, C. Zhang, and H. Zhang (2024) EAGLE-2: faster inference of language models with dynamic draft trees. In Empirical Methods in Natural Language Processing, Cited by: §6.4. [37] A. Liang, Y. Korkmaz, J. Zhang, M. Hwang, A. Anwar, S. Kaushik, A. Shah, A. S. Huang, L. Zettlemoyer, D. Fox, Y. Xiang, A. Li, A. Bobu, A. Gupta, S. Tu, E. Biyik, and J. Zhang (2026) Robometer: scaling general-purpose robotic reward models via trajectory comparisons. External Links: 2603.02115, Link Cited by: §2.2, §5. [38] Y. Liao, P. Zhou, S. Huang, D. Yang, S. Chen, Y. Jiang, Y. Hu, J. Cai, S. Liu, J. Luo, L. Chen, S. Yan, M. Yao, and G. Ren (2025) Genie envisioner: a unified world foundation platform for robotic manipulation. External Links: 2508.05635, Link Cited by: §5. [39] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone (2023) LIBERO: benchmarking knowledge transfer for lifelong robot learning. External Links: 2306.03310, Link Cited by: §4.2, §6.5. [40] O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard (2022) CALVIN: a benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. External Links: 2112.03227, Link Cited by: §5. [41] S. Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y. Zhu (2024) RoboCasa: large-scale simulation of everyday tasks for generalist robots. External Links: 2406.02523, Link Cited by: §5. [42] NVIDIA, :, J. Bjorck, F. Castañeda, N. Cherniadev, X. Da, R. Ding, L. ". Fan, Y. Fang, D. Fox, F. Hu, S. Huang, J. Jang, Z. Jiang, J. Kautz, K. Kundalia, L. Lao, Z. Li, Z. Lin, K. Lin, G. Liu, E. Llontop, L. Magne, A. Mandlekar, A. Narayan, S. Nasiriany, S. Reed, Y. L. Tan, G. Wang, Z. Wang, J. Wang, Q. Wang, J. Xiang, Y. Xie, Y. Xu, Z. Xu, S. Ye, Z. Yu, A. Zhang, H. Zhang, Y. Zhao, R. Zheng, and Y. Zhu (2025) GR00T n1: an open foundation model for generalist humanoid robots. External Links: 2503.14734, Link Cited by: Table 1. [43] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever (2021) Learning transferable visual models from natural language supervision. External Links: 2103.00020, Link Cited by: §1. [44] S. Ramos, S. Girgin, L. Hussenot, D. Vincent, H. Yakubovich, D. Toyama, A. Gergely, P. Stanczyk, R. Marinier, J. Harmsen, O. Pietquin, and N. Momchev (2021) RLDS: an ecosystem to generate, share and use datasets in reinforcement learning. External Links: 2111.02767 Cited by: §3. [45] R. Y. Rubinstein and D. P. Kroese (2004) The cross-entropy method: a unified approach to combinatorial optimization, monte-carlo simulation, and machine learning. Vol. 133, Springer. Cited by: §2.2, §5. [46] M. Shukor, D. Aubakirova, F. Capuano, P. Kooijmans, S. Palma, A. Zouitine, M. Aractingi, C. Pascal, M. Russi, A. Marafioti, S. Alibert, M. Cord, T. Wolf, and R. Cadene (2025) SmolVLA: a vision-language-action model for affordable and efficient robotics. External Links: 2506.01844, Link Cited by: Table 1. [47] W. Song, Z. Zhou, H. Zhao, J. Chen, P. Ding, H. Yan, Y. Huang, F. Tang, D. Wang, and H. Li (2025) ReconVLA: reconstructive vision-language-action model as effective robot perceiver. arXiv preprint arXiv:2508.10333. Cited by: §1, §5. [48] S. Tao, F. Xiang, A. Shukla, Y. Qin, X. Hinrichsen, X. Yuan, C. Bao, X. Lin, Y. Liu, T. Chan, Y. Gao, X. Li, T. Mu, N. Xiao, A. Gurha, V. N. Rajesh, Y. W. Choi, Y. Chen, Z. Huang, R. Calandra, R. Chen, S. Luo, and H. Su (2025) ManiSkill3: gpu parallelized robotics simulation and rendering for generalizable embodied ai. External Links: 2410.00425, Link Cited by: §3. [49] G. R. Team, K. Choromanski, C. Devin, Y. Du, D. Dwibedi, R. Gao, A. Jindal, T. Kipf, S. Kirmani, I. Leal, F. Liu, A. Majumdar, A. Marmon, C. Parada, Y. Rubanova, D. Shah, V. Sindhwani, J. Tan, F. Xia, T. Xiao, S. Yang, W. Yu, and A. Zhou (2026) Evaluating gemini robotics policies in a veo world simulator. External Links: 2512.10675, Link Cited by: §5. [50] G. Team, A. Ye, B. Wang, C. Ni, G. Huang, G. Zhao, H. Li, J. Zhu, K. Li, M. Xu, Q. Deng, S. Wang, W. Qin, X. Chen, X. Wang, Y. Wang, Y. Cao, Y. Chang, Y. Xu, Y. Ye, Y. Wang, Y. Zhou, Z. Zhang, Z. Dong, and Z. Zhu (2025) GigaWorld-0: world models as data engine to empower embodied ai. External Links: 2511.19861, Link Cited by: §5. [51] B. Terver, T. Yang, J. Ponce, A. Bardes, and Y. LeCun (2026) What drives success in physical planning with joint-embedding predictive world models?. External Links: 2512.24497, Link Cited by: §1, §5. [52] G. Wang, C. Zhang, Q. Liu, J. Zhang, J. Cai, J. Liu, and X. Liu (2026) LIBERO-x: robustness litmus for vision-language-action models. External Links: 2602.06556, Link Cited by: §5. [53] Y. Wang, O. Bounou, G. Zhou, R. Balestriero, T. G. J. Rudner, Y. LeCun, and M. Ren (2026) Temporal straightening for latent planning. External Links: 2603.12231, Link Cited by: §2.2. [54] Y. Xing, X. Luo, J. Xie, L. Gao, H. Shen, and J. Song (2025) Shortcut learning in generalist robot policies: the role of dataset diversity and fragmentation. External Links: 2508.06426, Link Cited by: §2.1. [55] K. Xu, Z. Zhu, A. Chen, S. Zhao, Q. Huang, Y. Yang, H. Lu, R. Xiong, M. Tomizuka, and Y. Wang (2025) Seeing to act, prompting to specify: a bayesian factorization of vision language action policy. External Links: 2512.11218, Link Cited by: §1, §5. [56] S. Yang, H. Li, Y. Chen, B. Wang, Y. Tian, T. Wang, H. Wang, F. Zhao, Y. Liao, and J. Pang (2025) InstructVLA: vision-language-action instruction tuning from understanding to manipulation. External Links: 2507.17520, Link Cited by: §1, Table 1, §5. [57] A. Zhai, B. Liu, B. Fang, C. Cai, E. Ma, E. Yin, H. Wang, H. Zhou, J. Wang, L. Shi, L. Liang, M. Wang, Q. Wang, R. Gan, R. Yu, S. Li, S. Liu, S. Chen, V. Chen, and Z. Xu (2025) Igniting vlms toward the embodied space. External Links: 2509.11766, Link Cited by: §1, Table 1, §5. [58] B. Zhang, J. Li, J. Shen, Y. Cai, Y. Zhang, Y. Chen, J. Dai, J. Ji, and Y. Yang (2025) VLA-arena: an open-source framework for benchmarking vision-language-action models. External Links: 2512.22539, Link Cited by: §5. [59] C. Zhang, R. Yang, X. Chen, K. Wang, L. Zhao, Y. Chen, and J. Bian (2025) How do vlas effectively inherit from vlms?. External Links: 2511.06619, Link Cited by: §1, §5. [60] Z. Zhang, D. Li, I. Reid, and R. Hartley (2026) GeoWorld: geometric world models. External Links: 2602.23058, Link Cited by: §5. [61] W. Zhao, J. Chen, Z. Meng, D. Mao, R. Song, and W. Zhang (2024) VLMPC: vision-language model predictive control for robotic manipulation. External Links: 2407.09829, Link Cited by: §1, §2.2, §5. [62] J. Zheng, J. Li, Z. Wang, D. Liu, X. Kang, Y. Feng, Y. Zheng, J. Zou, Y. Chen, J. Zeng, Y. Zhang, J. Pang, J. Liu, T. Wang, and X. Zhan (2025) X-vla: soft-prompted transformer as scalable cross-embodiment vision-language-action model. External Links: 2510.10274, Link Cited by: Table 1. [63] G. Zhou, H. Pan, Y. LeCun, and L. Pinto (2025) DINO-wm: world models on pre-trained visual features enable zero-shot planning. External Links: 2411.04983, Link Cited by: §1, §5. [64] S. Zhou, Y. Du, J. Chen, Y. Li, D. Yeung, and C. Gan (2024) RoboDreamer: learning compositional world models for robot imagination. External Links: 2404.12377, Link Cited by: §5. [65] X. Zhou, Y. Xu, G. Tie, Y. Chen, G. Zhang, D. Chu, P. Zhou, and L. Sun (2025) LIBERO-pro: towards robust and fair evaluation of vision-language-action models beyond memorization. External Links: 2510.03827, Link Cited by: §1, §5. [66] C. Zhu, R. Yu, S. Feng, B. Burchfiel, P. Shah, and A. Gupta (2025) Unified world models: coupling video and action diffusion for pretraining on large robotic datasets. External Links: 2504.02792, Link Cited by: §5. Appendix 6.1 Implementation Details for Baselines We summarize the configurations of the evaluated VLAs in Table 2. We observed that the more frequent the replanning, the more difficult closed-loop control for VLAs becomes, due to compounding errors. Replanning every 20 steps (1-second simulation time) is a sweetspot for VLAs. Increasing the replanning frequency to replan every 10 steps brings more or less performance drops. We thus set the replanning interval to 20 steps for most baselines. For some VLAs that suffer from compounding error, we replan every 40 steps to improve their performance and thus increase their action thunk size c. Based on this, for GWM-MPC, we tune its parameters based on the replanning interval of 20 steps. Other hyperparameters are selected in terms of the open-loop future video classification accuracy with Qwen3-VL-embedding, using the training data only. For models like InternVLA-A1, SmolVLA, Wall-OSS, π0.5 _0.5, and π0 _0, we found that increasing the action chunk size c did not yield performance improvements. Consequently, we set c to their 20-step replanning interval to minimize the number of trainable parameters. However, specific models required distinct settings: Motus, GR00T-N1.6, and UniVLA suffer from error accumulation with 20-step replanning, and XVLA’s TCP reaching success rate degrades with smaller chunk sizes. Therefore, they require a larger chunk size. For GR00T, we omit results for the LeRobot-implemented GR00T-N1.5 as it underperformed the official GR00T-N1.6. We exclude the use of the wrist camera for GR00T-N1.6, because it harms the performance a lot. Across all models, we strictly adhere to official fine-tuning setups (e.g., full-parameter vs. action-head only vs. lora-finetune), training them until closed-loop performance plateaus on training scenes prior to zero-shot evaluation on the test scenes. We also tested openvla-oft [29], but its training task success rate would gradually drop when the training set covers more tasks. When training openvla-oft on one out of 24 subsets, it can overfit the 12 training tasks to 100% succress rate, while increasing the dataset size to the full WISER training set, it only reaches 24% success rate. So we exclude it. Table 2: Configuration and Implementation Details of Evaluated VLAs Model c Replan Interval Dataloader Implementation Action Motus 48 40 LeRobot Official Absolute XVLA 40 20 LeRobot LeRobot Absolute GR00T-N1.6 40 40 LeRobot Official Relative InstructVLA 16 16 RLDS Official Absolute OpenVLA-OFT 20 20 RLDS Official Absolute SmolVLA 20 20 LeRobot LeRobot Absolute Wall-OSS 20 20 LeRobot LeRobot Absolute π0.5 _0.5 20 20 LeRobot LeRobot Absolute π0 _0 20 20 LeRobot LeRobot Absolute InternVLA-A1 20 20 LeRobot Official Relative UniVLA 40 40 RLDS Official Absolute 6.2 Score Function Design To obtain zgz_g, we feed a multimodal prompt into Qwen3-VL-Embedding. A retrieval-oriented system prompt s is prepended with content: “Retrieve the video which can best finish the manipulation task specified by the user, given the layout of the workspace and the current frame observation.” This system prompt steers the model to produce embeddings that align task descriptions with future visual outcomes, but doesn’t disclose any task-specific information. The task instruction ℓ , which follows the template “Pick up the X and place it onto the Y”, is decomposed into two sub-task prompts: ℓpick _pick=“Pick up the X from the table” and ℓplace _place=“Place the grasped object to the Y on the table”, where X and Y are extracted from ℓ via pattern matching. Each sub-task prompt is independently encoded with visual context—the initial observation o0o_0 and the current observation oto_t—to produce sub-task embeddings: zgpick=Qwen3-VL-Embed(s,ℓpick,o0,ot),zgplace=Qwen3-VL-Embed(s,ℓplace,o0,ot).z_g^pick=Qwen3-VL-Embed(s, _pick,o_0,o_t), z_g^place=Qwen3-VL-Embed(s, _place,o_0,o_t). (5) The initial observation o0o_0 provides a static visual anchor of the workspace layout, while the current observation oto_t supplies dynamic context at the time of replanning. Both images and the text prompt are jointly processed by Qwen3-VL-Embedding to produce a normalized embedding vector. Given N candidate action sequences with predicted future embeddings zt1,…,ztN\z^1_t,…,z^N_t\, the cosine similarities against each sub-task embedding are computed and normalized across candidates via softmax: σpickn=exp⁡(cos⁡(ztn,zgpick))∑m=1Nexp⁡(cos⁡(ztm,zgpick)),σplacen=exp⁡(cos⁡(ztn,zgplace))∑m=1Nexp⁡(cos⁡(ztm,zgplace)).σ^n_pick= ( (z^n_t,z_g^pick)) _m=1^N ( (z^m_t,z_g^pick)), σ^n_place= ( (z^n_t,z_g^place)) _m=1^N ( (z^m_t,z_g^place)). (6) The final selection score depends on the current grasp state, which is determined by the contact sensor on the robot gripper: Sn=σpicknif the object has not been grasped,σplacenif the object has been grasped,S^n= casesσ^n_pick&if the object has not been grasped,\\ σ^n_place&if the object has been grasped, cases (7) and the action sequence with the highest score is selected: n∗=arg⁡maxn⁡Snn^*= _nS^n. We also experimented with canceling the grasp-based weighted combination, and scoring trajectories directly with the embedding that resulted from: zg=Qwen3-VL-Embed(s,ℓ,o0,ot)z_g=Qwen3-VL-Embed(s, ,o_0,o_t) As shown in Tab. 3, it brings a 15% performance drop in terms of test success rate, which, as we suggested, is caused by the Qwen-3-VL-Embedding instead of the GWM. We also tried to apply the task prompt decomposition to the best VLA baseline, InstructVLA. However, we find that InstructVLA experiences a significant performance drop on both the training tasks (89% → 52%) and the test tasks (47% → 30%), when decomposing each task into two subtasks. This result suggests that VLM-based VLAs can overfit to the specific sentence structures seen during training, rather than genuinely understanding the compositional semantics of each clause. Even a simple rephrasing or decomposition of the task prompt—without altering its underlying meaning—is sufficient to induce a notable performance degradation. In contrast, GWM-MPC can leverage the intact language understanding ability of the foundation model, so that a task decomposition further boosts its performance, which aligns with the intuition that atomic tasks should be easier to address than compositional long-horizon tasks, which are basically a chain of atomic tasks. Table 3: Ablation on Prompt Decomposition for GWM-MPC and InstructVLA. Prompt Training Set Test Set Grasp Reach Success Grasp Reach Success GWM ℓpick _pick + ℓplace _place 0.97 0.95 0.92 0.99 0.88 0.87 GWM ℓ 0.93 0.88 0.82 0.92 0.76 0.73 InstructVLA ℓpick _pick + ℓplace _place 0.98 0.53 0.52 0.80 0.30 0.30 InstructVLA ℓ 0.98 0.92 0.89 0.79 0.51 0.47 6.3 Inference Efficiency Figure 6: For all methods, we measure the inference efficiency with the rollout FPS, which is how many times the env.step is called in one second. VLA baselines have better inference efficiency than the GWM-MPC when evaluated on the test tasks. It is because we need to forward the GWM N=12N=12 times to get future embeddings for all proposals. Also, we generate future embeddings sequentially rather than in parallel because the Qwen encoder produces bugs when batching input. This deteriorates the inference efficiency. 6.4 Visual Grounding Evaluation for InstructVLA. It is possible that the base VLM inherently lacks the ability to recognize the captured workspace images from WISER, and consequently, the VLA fine-tuned from it cannot successfully complete the manipulation tasks. To rule out this possibility, we assess the visual understanding capabilities of the base Eagle-2B model [36] before the fine-tuning of the best VLA baseline, InstructVLA. Specifically, we design a visual grounding evaluation on the WISER benchmark. For each of the 24 test task configurations, we reset the simulation environment and capture the initial observation from the main camera. We then ask the foundation VLM to identify which of the three destination images (left, middle, or right) best matches the referring expression extracted from the task instruction. The prompt sent to Eagle-2B is: There are three images with white backgrounds at the bottom of the table. Answer which image best describes: place referring expression? Answer with: left, middle, or right. The results show that the base VLM achieves an 81%81\% accuracy on spatially localizing the destination image across 288 test scenarios, demonstrating that it already possesses a strong visual understanding of the scene layout before any robotic fine-tuning is applied. However, after finetuning with OXE data and the WISER training data, its TCP reaching success rate is only 51%51\%, indicating that this spatial localization capability is somehow compromised. 6.5 GT-MPC for LIBERO-goal LIBERO [39] is a widely adopted benchmark for VLAs. Among its 100 tasks, only 10 from the LIBERO-Goal split strictly require semantic understanding. This is because these tasks share identical scene layouts, compelling VLAs to differentiate between them solely based on task instructions. For the remaining tasks, a purely visuomotor policy often suffices to map the scene layout directly to the target trajectory or action without needing to process the instruction. Consequently, we evaluate our Model Predictive Control (MPC) framework equipped with Qwen3-VL-Embedding specifically on this split. Since the LIBERO demonstrations are collected in the exact same test environments with identical visual appearances and instructions, future trajectories can be proposed using KNN, and the respective future frames can be directly retrieved from the training dataset. Also, we do not need to train a GWM, because we have the GT future videos already. Following the standard LIBERO evaluation protocol, we run 50 episodes for each task. The results are presented in Table 4. These results demonstrate that Qwen3-VL-Embedding serves as an effective zero-shot video classifier, capable of selecting the optimal action by evaluating the future observations given all candidates. Our GT-MPC system yields a zero success rate on only two tasks. This failure stems from an inability to recognize the correct behavior required to fulfill the event described by the prompt. For the task “open the middle drawer of the cabinet”, the system fails because the scoring function initially assigns a higher value to the action trajectory associated with “open the top drawer and put the bowl inside”. Nevertheless, this indicates that the foundation model successfully captures the correct macro movement direction for the gripper. Among the completed tasks, several do not achieve a 100% success rate. This is primarily because the KNN action generator lacks robustness against small perturbations in object positions. In addition, the error is accumulated in closed-loop running because of using the delta action space. Employing a learning-based visuomotor policy for action proposal could effectively alleviate this issue. Table 4: Task Success Rates on libero-goal Split Task Description SR 0 open the middle drawer of the cabinet 0.0% 1 put the bowl on the stove 72.0% 2 put the wine bottle on top of the cabinet 96.0% 3 open the top drawer and put the bowl inside 80.0% 4 put the bowl on top of the cabinet 100.0% 5 push the plate to the front of the stove 98.0% 6 put the cream cheese in the bowl 0.0% 7 turn on the stove 100.0% 8 put the bowl on the plate 68.0% 9 put the wine bottle on the rack 100.0% Average 71.4% Figure 7: Libero-goal environment. 6.6 Model Architectures & Hyperparameters The transformer backbone for the GWM and the action-conditioned version has the same structure and uses the same training hyperparameters. The details can be found in table 5. The difference between the two action tokenization schemes is shown in Fig. 8. Figure 8: Difference between GWM and its raw action conditioned version. Captured images are just exemplary; the main camera is placed in front of the robot as shown in Fig. 4. Table 5: Transformer configuration and hyperparameters of the (GWM). Hyperparameter Value Architecture Hidden dimension (dmodeld_model) 4096 FFN intermediate dimension (dffnd_ffn) 8192 Attention head dimension (dheadd_head) 128 Number of layers 5 Number of attention heads 32 Number of KV heads (GQA) 8 Input / Output dimension 4096 Input sequence length 1620 Positional encoding 2D RoPE Normalization RMSNorm (ϵ=10−5ε=10^-5) FFN activation SwiGLU Precision bfloat16 Training Optimizer Muon [26] + Adam [30] Learning rate (Muon, hidden weights) 0.01 Learning rate (Adam, embed/head) 5×10−55× 10^-5 Adam β (0.9, 0.95) Weight decay 0.01 LR scheduler Cosine annealing Min learning rate 10−610^-6 Epochs 10 Gradient clipping 1.0 Loss function MSE 6.7 WISER Benchmark All training and test tasks are shown as follows. Images are AI-generated to avoid copy right issue.

Instant research discovery