Paper deep dive
Decoupling Intention from Trajectory: A Representational Deduction Framework for World Action Models
Xiangkai Ma, Yue Ma, Junjie Wang, Sheng Xu, Mingyang Li, Han Zhang, Yuzheng Zhuang, Wenzhong Li, Zhihao Yuan
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/10/2026, 3:55:07 AM
Summary
The paper introduces PILOT, a framework for World Action Models (WAMs) that decouples high-level motion intention from low-level trajectory generation. It utilizes a Representational Deduction (RD) mechanism to extract motion-semantic tokens (Motion-CoT) from visual observations, which guide the action model to generate fine-grained trajectories. This approach alleviates representational entanglement, improves generalization in robotic manipulation tasks, and reduces inference latency compared to traditional predict-then-act paradigms.
Entities (8)
Relation Signals (6)
PILOT → uses → Representational Deduction
confidence 95% · We propose PILOT... whose core Representational Deduction (RD) bridges this gap...
Representational Deduction → generates → Motion-CoT
confidence 92% · RD aims to encourage the action branch to explicitly model potential state transition tokens, which are retained as CoT...
Motion-CoT → guides → World Action Models
confidence 90% · Motion-CoT... enabling the Action Model to focus on fine-grained trajectory refinement...
PILOT → outperforms → LIBERO
confidence 90% · PILOT achieves state-of-the-art performance with 97.9% ... success rates on LIBERO
PILOT → outperforms → RoboCasa-GR1
confidence 90% · PILOT achieves state-of-the-art performance with ... 62.6% success rates on ... RoboCasa-GR1 benchmarks
Causal Dynamics Engine → supervises → Motion-CoT
confidence 88% · CDE... supervises the motion semantics by predicting VJEPA2-AC future-state representations...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:World Action Models (WAMs) aim to construct a unified architecture capable of understanding world state evolution and guiding to generative motion planning. However, existing visual branches focus on predicting static visual observation, rather than reflecting potential transition information that captures the evolution of world states under motion interactions. This leads to representational entanglement between high-level physical condition evolution and low-level action trajectory generation within the Action Model, creating a structural bottleneck while weakening the predictive capability of world evolution modeling for action generation. We propose PILOT (Physical Inference for Latent Optimized Trajectories), whose core Representational Deduction (RD) bridges this gap by integrating motion thought-of-chain (CoT) guidance as a native model capability. Specifically, RD aims to encourage the action branch to explicitly model potential state transition tokens, which are retained as CoT in the reasoning space to guide fine-grained motion trajectory. Experiments demonstrate that RD not only significantly improves the success rate and generalization ability of WAMs in complex robotic manipulation tasks but also enhances the model's physical interpretability by decoupling high-level motion semantics from low-level trajectory details. Furthermore, the abundant state transition supervision signals introduced by RD effectively alleviate the sparse supervision in action generation, enabling it to serve as an efficient few-shot real-robot fine-tuning strategy and demonstrating superior scalability for migration to mainstream WAM architectures.
Tags
Links
- Source: https://arxiv.org/abs/2608.06994v1
- Canonical: https://arxiv.org/abs/2608.06994v1
Trouble viewing inline? Open PDF directly →
Full Text
107,211 characters extracted from source content.
Expand or collapse full text
Decoupling Intention from Trajectory: A Representational Deduction Framework for World Action Models Xiangkai Ma1, Yue Ma2, Junjie Wang3, Sheng Xu3, Mingyang Li4, Han Zhang1, Yuzheng Zhuang5, Wenzhong Li1🖂, Zhihao Yuan5🖂 1NJU 2HKUST 3CUHK-SZ 4THU 5Joy Future Academy, JD https://pilot-wam-2026.github.io/ Abstract World Action Models (WAMs) aim to construct a unified architecture capable of understanding world state evolution and guiding to generative motion planning. However, existing visual branches focus on predicting static visual observation, rather than reflecting potential transition information that captures the evolution of world states under motion interactions. This leads to representational entanglement between high-level physical condition evolution and low-level action trajectory generation within the Action Model, creating a structural bottleneck while weakening the predictive capability of world evolution modeling for action generation. We propose PILOT (Physical Inference for Latent Optimized Trajectories), whose core Representational Deduction (RD) bridges this gap by integrating motion thought-of-chain (CoT) guidance as a native model capability. Specifically, RD aims to encourage the action branch to explicitly model potential state transition tokens, which are retained as CoT in the reasoning space to guide fine-grained motion trajectory. Experiments demonstrate that RD not only significantly improves the success rate and generalization ability of WAMs in complex robotic manipulation tasks but also enhances the model’s physical interpretability by decoupling high-level motion semantics from low-level trajectory details. Furthermore, the abundant state transition supervision signals introduced by RD effectively alleviate the sparse supervision in action generation, enabling it to serve as an efficient few-shot real-robot fine-tuning strategy and demonstrating superior scalability for migration to mainstream WAM architectures. Figure 1: We propose PILOT, a novel framework that decouples high-level motion intention from low-level trajectory generation. A Representational Deduction (RD) mechanism is introduced, where latent state transition information is compressed into learnable motion-semantic tokens serving as Motion CoT, enabling the Action Model to focus on fine-grained trajectory refinement while alleviating representational entanglement. 11footnotetext: The corresponding authors are Wenzhong Li (lwz@nju.edu.cn) and Zhihao Yuan (yuanzhihao1@jd.com). 1 Introduction World Models [20, 7] and Embodied Intelligence [6, 75, 5, 21, 26, 59] are dedicated to “understanding how the physical world evolves during human interaction” and “planning the action trajectory based on current observations,” respectively. The unification of these paradigms represents a significant step toward General Spatial Intelligence [61]. Recent World Action Models (WAMs) [3, 39, 52, 30, 29] approach this goal by leveraging world models to predict future states and introducing the Inverse Dynamics Module (IDM) [15, 35, 70] to convert these predictions into trajectory. However, future states are confined to the visual latent space constructed by frozen VAEs [45, 9, 69]. This structural bottleneck causes the joint optimization objective of WAMs to primarily target visual reconstruction over trajectory generation. Guiding WAMs to focus on motion semantic representations to eliminate the visual bottleneck via an end-to-end framework remains a long-term goal. Recent experiments [66, 62, 51] indicate that while explicit future prediction aids training, it provides limited foresight for trajectory generation during inference. We attribute this to the fact that static visual representations at a single moment fail to reflect the evolution process of physical states under action interactions. Without explicit modeling of state transitions, directly generating actions through the diffusion process leads to representational entanglement, which is visualized in Fig. 2. To address this, we propose that action model should simultaneously learn high-level motion intention structures and low-level manipulation details from the same inputs. Figure 2: (a) t-SNE of hidden representation. Baselines (left) show entangled clusters due to background coupling, while PILOT(right) forms distinct action-type clusters, isolating motion intent from static details. (b) PCA overlay on future frames. Red highlights indicate high-variance regions. Baselines (left) are distracted by background noise, whereas PILOT (right) focuses on the interaction zone, capturing dynamic state transitions. How to supervise the model to learn the motion semantic representations that contain state transition information? The existing WAM [64, 19, 10, 18] can precisely predict what will happen in the future, but still doesn’t know how to do it. Most relevant works are Latent Action Models (LAMs) [13, 57, 12], which typically capture pixel-level differences and reconstruct future frames from the current frame via IDM [56, 65, 8, 48]. However, this kind of visual-aware-only supervision, fails to prompt the model to understand what action conditions led to the variation in the visual state. Instead of deriving latent motions directly from consecutive frames, we employ them exclusively as supervision, guiding the model to summarize physical state transitions and motion semantics within learnable tokens. In this paper, we propose Physical Inference for Latent Optimized Trajectories (PILOT), an end-to-end framework grounded in the Representational Deduction (RD) mechanism that bridges the gap between static visual prediction and dynamic action manipulation by establishing motion semantic CoT reasoning as an innate capability of WAMs. Architecturally, PILOT decouples the World Model into understanding and generation pipelines to substitute explicit future image prediction with the Motion-CoT during inference, thereby accelerating deployment speed while improving both simulation and real-world. Specifically, action model should first distills high-level motion semantics CoT from current and future predicted observations, where these conditional tokens embody physical causality and state transition information. Subsequently, we freeze the pre-trained VJEPA-as-Encoder [2, 43, 42] to capture deterministic features tightly coupled with action conditions, and train a Causal Dynamics Engine (CDE) to predict future spatial positions, trajectories, shapes, and interaction relationships conditioned on motion semantic CoT, rather than probabilistic underlying visual rendering details. Meanwhile, these motion semantics are retained as context tokens in the reasoning space, allowing the subsequent diffusion process to concentrate on refining fine-grained end-effector trajectories and achieve representational decoupling. This brings an additional benefit: Motion-CoT serves as an extra supervision signal, alleviating the sparse supervision problem faced by the action model due to the low information density of action signals. Figure 3: Overview of the PILOT framework. In the World-Model (a), current observations and instructions are encoded into a unified context. Subsequently, the Action-Perceiver employs learnable query tokens to distill motion-semantic context via a causally-decoupled attention mechanism (b), which guides the flow-matching action decoder. Simultaneously, we leverage a Causal Dynamics Engine (c) to supervise the motion semantics by learning physical state transition information. Our contributions can be summarized as follows. • We propose PILOT, a novel WAM framework that decouples high-level motion intention from low-level trajectory generation via a Representational Deduction mechanism, and internalizes latent state transition information as Motion CoT to alleviate representational entanglement. • We introduce a Causal Dynamics Engine with causally-decoupled attention, that endows the Action Model with explicit physical causality learned from VJEPA latent spaces, while bypassing the computational bottleneck of pixel-level future frame reconstruction during inference. • PILOT achieves state-of-the-art performance with 97.9% and 62.6% success rates on LIBERO and RoboCasa-GR1 benchmarks respectively, and attains an 83.1% average success rate in real-world manipulation tasks, demonstrating superior generalization and 90% lower inference latency compared to predict-then-act paradigms. 2 Related Work 2.1 World Models for Robotic Manipulations Recently, WAMs [10, 30, 29, 9, 69, 71] have emerged as a promising paradigm that leverages world model to predict future, thereby guiding motion manipulation. Existing video-model-based WAMs [25, 55] generally follow two paradigms. The serial-paradigm first predicts future observation and then reconstructs actions via an inverse dynamic [45, 15, 73], while the parallel-paradigm jointly models the visual-action space through shared attention mechanism (MoT) [3, 39, 66] or joint denoising [64, 19]. Unlike generative pixel reconstruction strategies, joint embedding predictive frameworks (JEPA) [42, 2, 43] mitigate uncertainty in pixel space by actively ignoring entropy-increasing stochastic texture details, thus focusing on predicting dynamic features such as object spatial positions and trajectories. Some studies [23, 49] incorporated JEPA-as-encoder to obtain abundant representations. Most relevant to our work is VLA-JEPA [51], which introduces JEPA to extract latent states, addressing the challenges of pixel reconstruction bias and future information leakage. However, the purely visually supervision fail to provide state transition information. This inherent discrepancy causes the deviation of co-training from the motion semantics, resulting in fragile transferability and inefficient fine-tuning; conversely, completely removing visual supervision and relying solely on sparse action supervision leads to representation collapse of video model. To address this, we propose Representational Deduction, which bridges the aforementioned gap by incorporating additional motion semantics and physical state transition information. 2.2 Latent Action Learning for Robotics LAMs [48, 35, 7, 56] aim to pre-train backbones on large-scale human videos lacking action labels, thereby establishing prior knowledge of latent motion. Existing approaches [65, 8, 15, 70, 60] typically follow a two-stage pipeline: (1) encoding visual observations from adjacent frames into discrete latent tokens via a Q-former [28], and (2) aligning vision and manipulation representations using a unified codebook [68, 41, 15, 11] to integrate robotic manipulation into the Next-Token-Prediction framework. Despite the promise of LAMs, their performance is hindered by noise from camera-induced background variations and future frame leakage, which encourages trivial encoding of future frames rather than learning state transitions [51]. These issues cause structural bottlenecks in learning control-oriented latent actions. Recent works mitigate this via latent constraints [37] or multi-encoder-query [49]. However, the LAMs inherently aligns latent actions with visual space rather than actionable control signals. PILOT fundamentally avoids these defects by ensuring that future visual information is used exclusively as a supervision target, not as a source for capturing latent motion representations, thereby guaranteeing that the model focuses on dynamic representations independent of camera motion and background changes. 3 Method As illustrated in Fig. 3, PILOT consists of three cooperating components: (a) a World-Model branch built on a pre-trained Wan2.2 [55], which encodes the current observation and instruction into vision-language context and provides auxiliary future-frame supervision (Sec. 3.5); (b) a from-scratch trained Action Model [22] branch that distills motion-semantic context with learnable query tokens and decodes actions through causally-decoupled flow matching (Sec. 3.3); (c) a RD branch that supervises the motion semantics by predicting VJEPA2-AC [2] future-state representations with a trainable CDE (Sec. 3.4). 3.1 Problem formulation and notation. Let each episode provide, a current RGB observation oto_t, a future RGB observation ot+Δo_t+ , a natural-language instruction ℓ∈ℝL \!∈\!R^L, a proprioceptive state st∈ℝdss_t ^d_s, and a target action chunk t=(at,…,at+H−1)∈ℝH×da a_t=(a_t,…,a_t+H-1) ^H× d_a, where H is the future action horizon. The pair (ot,ot+Δ)(o_t,o_t+ ) is the only extra supervision that RD requires and is obtained from teleoperation trajectories. Throughout, dsd_s denotes the state dimension and dad_a the action dimension. We model instruction-following manipulation as a goal-conditioned controlled Markov process. The target policy factorizes as: pΘ(t∣ot,ℓ,st) p_ ( a_t o_t, ,s_t) =∫pΘ(t∣,st) = p_ ( a_t m,s_t) ×pΘ(∣ot,ℓ,st)d, ×\;p_ ( m o_t, ,s_t)\,d m, (1) where ∈ℝK×d m ^K× d is a latent motion-semantic variable that we introduce explicitly. Our central thesis is that the quality of a world-action model hinges on whether m captures the action-conditioned state transition ot→ot+Δo_t\!→\!o_t+ . 3.2 Motivation of Representation Entanglement Prior WAMs typically inject future information by reconstructing future observations in a frozen VAE latent space. Although this supervision helps the video backbone acquire physical priors, it biases joint optimization toward visual reconstruction rather than manipulation. Moreover, single future visual latent describes what the scene may look like, but does not explicitly model how the world state changes under action. Consequently, the action diffusion process must infer high-level state evolution and generate fine-grained motor commands from the same entangled context, yielding an unstable transition prior at inference. PILOT addresses this issue through a hierarchy centered on the motion-semantic variable m (Eq. 1). Causally-decoupled attention (Eq. 3) prevents learnable tokens from absorbing action noise, encouraging m to encode the state transition ot→ot+Δo_t\!→\!o_t+ . Asymmetric information flow (Eq. 4) then allows m to condition action diffusion while blocking reverse noise leakage, enabling model (Eq. 7) to focus on trajectory refinement. The resulting representational deduction loss ℒRDL_RD (Eq. 11) further provides transition-level supervision, alleviating the sparsity of action-only training. 3.3 Motion-CoT from Learnable Queries Vision-language context. In the comprehension pipeline, we repurpose the Wan2.2 as a vision-language encoder. The video tokenizer encodes observation oto_t into clean latents and the text encoder embeds instruction ℓ ; a single conditional forward pass yields the context sequence =WMθ(ot,ℓ)∈ℝNc×dc, c\;=\;WM_θ(o_t, )\;∈\;R^N_c× d_c, (2) where WMθWM_θ denotes the trainable world-model parameters, and NcN_c and dcd_c are the number and dimension of context tokens. The pre-trained video prior in WMθWM_θ provides visual and physical cues for downstream action generation. Learnable motion-semantic queries. We introduce K learnable query tokens ∈ℝK×d Q ^K× d to distill motion-semantic representations from c before action decoding. The action model is instantiated as a Perceiver-style flow-matching head. At a sampled flow time τ, the noised action chunk tτ=(1−τ)ϵ+τt a^τ_t=(1-τ)\, ε+τ\, a_t is embedded into action tokens τ∈ℝH×d A^τ ^H× d, where ϵ∼(,) ε\! \!N( 0, I). The state sts_t is mapped into a state token ∈ℝ1×d S ^1× d, and the latent sequence [;;τ][ S; Q; A^τ] attends to c through a stack of L blocks. Causally-decoupled attention mechanism. To prevent motion semantics from absorbing diffusion noise, we decouple the attention inside each Perceiver block. Denoting the non-action latents as =[;] Z=[ S; Q], each block applies Z ←+Attn(,[;]), ← Z+Attn\! ( Z,\,[ c; Z] ), (3) τ A^τ ←τ+Attn(τ,[;;τ]), ← A^τ+Attn\! ( A^τ,\,[ c; Z; A^τ] ), (4) where Z attends only to itself and the vision–language context, while action tokens additionally attend to Z. This asymmetric flow lets motion semantics condition action decoding while blocking reverse noise leakage. Accordingly, time-conditioned AdaLN is applied only to action tokens, and Z uses plain layer normalization to remain invariant to the flow time τ. After the final block, we read out the query slots as the motion-semantic context: = O= Act.Per.(,τ,;τ)∈ℝ(1+K+H)×d, .Per.( Z, A^τ, c;τ)\;∈\;R^(1+K+H)× d, (5) = m= [:, 1:K+1,:]∈ℝK×d. O[:,\,1\!:\!K\!+\!1,\,:]\;∈\;R^K× d. (6) Flow-matching action decoding. Action tokens after the last block are projected back to the action space and supervised with the conditional flow-matching velocity objective: ℒact=τ,ϵ∥Projact([:,K+1:,:])−(t−ϵ)∥2,L_act\;=\;E_τ, ε \|\,Proj_act\! ( O[:,\,K\!+\!1:,\,:] )-( a_t- ε)\, \|^2, (7) where ProjactProj_act is a MLP mapping action tokens to the action dimension dad_a. At inference, Euler integration starts from Gaussian noise, conditioned on the query-produced m. 3.4 Representational Deduction with a CDE The motion-semantic context m becomes meaningful only when it is explicitly constrained to encode what changes in the world state. We provide such supervision via CDE that models state transitions in the representation space closely associated with physical dynamics. Frozen state representations. Given the current and future observations (ot,ot+Δ)(o_t,o_t+ ), the frozen VJEPA2-AC encoder EψE_ψ produces per-frame patch representations: t=Eψ(ot),t+Δ=Eψ(ot+Δ), r_t\;=\;E_ψ(o_t),~~~ r_t+ \;=\;E_ψ(o_t+ ), (8) where each representation lies in ℝP×drR^P× d_r, with P patch tokens and token dimension drd_r. Unlike the appearance-oriented visual latents, VJEPA2-AC representations are learned through a joint-embedding predictive objective and are therefore organized around predictable, action-relevant structure rather than pixel-level reconstruction. This enables RD to supervise motion rather than texture. Causal dynamics engine. The CDE GξG_ξ is initialized from the pre-trained VJEPA2-AC predictor and remains trainable. Instead of using the original action and state inputs of the predictor, we condition the CDE on the learned motion semantics and robotic state. Two lightweight linear projectors map them into the VJEPA2-AC latent space: ~=m∈ℝK×dr,~=sst∈ℝ1×dr. m= W_m\, m ^K× d_r,~~~ s= W_s\,s_t ^1× d_r. (9) Conditioned on ~ m and ~ s, the CDE predicts the future representation from the current one: ^t+Δ=Gξ(t;~,~)∈ℝP×dr. r_t+ \;=\;G_ξ ( r_t\,;\, m,\, s )\;∈\;R^P× d_r. (10) Internally, ~ m and ~ s are prepended to the current-frame patch tokens as conditioning tokens, and only the predicted frame tokens are returned. representational deduction loss. We supervise the CDE by matching its prediction to the frozen encoder representation of the ground-truth future frame: ℒRD=SmoothL1(^t+Δ,t+Δ).L_RD\;=\;SmoothL1 ( r_t+ ,\, r_t+ ). (11) Gradients from ℒRDL_RD propagate through the CDE and the projector m W_m to the learnable queries, encouraging m to encode transition information that is predictive of future VJEPA2-AC representations. Because the supervision target is representational rather than pixel-level, RD teaches the model how the state evolves under the action, rather than merely what the future looks like. Figure 4: Inference latency of existing VLAs and WAMs on H20 GPU, along with success rates on RoboCasa-GR1. We further compare the inference overhead of PILOT under the “Motion-CoT” and “Predict-then-Act” paradigms with identical architectures. Motion-CoT reduces latency by 90%. 3.5 World-Model Branch and Training Objective Future-frame latent prediction. In the generation pipeline, the world model retains its generative capability through an auxiliary future-frame objective. We encode the short clip [ot,ot+Δ][\,o_t,\,o_t+ \,] into clean video latents, using the current-frame latents as conditions and the future-frame latents as prediction targets. Flow-matching noise is applied only to the target latents, and the world model is trained to predict the corresponding target velocity: ℒwm=τ,ϵ‖WMθ(τ;ot,ℓ,τ)-(ϵ-∗)‖⊙τ2,L_wm\;=\;E_τ, ε \|\,WM_θ( z^τ;\,o_t, ,τ)-( ε- z )\, \|^2_ \, M_ z^τ, (12) where WMθWM_θ denotes the trainable world-model parameters, ∗ z are the clean target latents, τ z^τ is their noised version, and τ M_ z^τ restricts the loss to future-frame positions. This objective preserves the instruction-grounded future prior of the pre-trained generator, while the generation pipeline is disabled during inference to avoid the visual bottleneck. Overall objective. The three branches are trained jointly with a single weighted objective: ℒ=ℒact+λwmℒwm+λRDℒRD,L\;=\;L_act\;+\; _wm\,L_wm\;+\; _RD\,L_RD, (13) where λwm _wm and λRD _RD balance the auxiliary future-frame and Representational Deduction losses against the action objective. In practice, the VJEPA2-AC encoder is frozen; the world model and CDE are fine-tuned from their pre-trained weights; and the Action Model together with the two projectors is trained from scratch. Inference. At test time, only the understanding–action pathway is used: the world model encodes (ot,ℓ)(o_t, ) into context c, the learnable queries produce the motion-semantic context m, and the flow-matching head outputs the action chunk conditioned on m. The RD branch and future-frame prediction serve only as training-time supervision and incur no additional inference cost; the generative decoder can be optionally invoked for future-frame visualization. 4 Experiments Figure 5: Real-world evaluation suite on the Agibot-G1 humanoid robot. 4.1 Implementation Details Our codebase is built upon the StarVLA project [63]. PILOT is implemented with three cooperative branches. The World-Model branch builds on a pre-trained Wan2.2 video diffusion transformer, which simultaneously serves as a visual-language encoder (providing a 196×2048196\!×\!2048 context) and a trainable latent future-frame generator. The Action-Model branch adopts learnable query tokens (K=64K\!=\!64, d=1024d\!=\!1024) that distill motion-semantic information, followed by a DiT-based flow-matching action decoder with 4 denoising steps. The Representational Deduction branch employs a frozen VJEPA2-AC encoder that produces current and future patch-level representations (256×1408256\!×\!1408), and a lightweight Causal Dynamics Engine that predicts future representations conditioned on motion semantics and proprioceptive state. Please refer to the appendix for complete architecture hyper-parameters, training hyper-parameters, and algorithm pseudocode details. 4.2 Simulation Benchmarks We evaluate PILOT on two simulation benchmarks spanning different embodiments, action spaces, and visual modalities. LIBERO. As shown in Table 1, PILOT achieves 97.9% average success rate with only 5.4B parameters, surpassing all baselines including Motus (8B, 97.7%) and π0.5 _0.5 (3.5B with internet-scale pre-training, 96.2%). Notably, PILOT excels on the challenging Goal (98.1%) and Long-horizon (97.2%) suites, where understanding the intent of manipulation is critical for long-horizon planning. This validates our central thesis: decoupling motion semantics from trajectory generation via Representational Deduction effectively alleviates representational entanglement. Method Size Spatial Object Goal Long Avg. Vision-Language-Action Models π0.5 _0.5 [21] 3.5B 98.8 98.2 98.0 92.4 96.2 GROOT-N1.6 [44] 3.3B 97.7 98.5 97.5 94.4 97.0 OpenVLA-OFT [24] 7B 97.6 98.4 97.9 94.5 97.1 Latent Action Models LAPA [65] 7B 73.8 74.6 58.8 55.4 65.7 UniVLA [8] 7B 96.5 96.8 95.6 92.0 95.2 Mantis [62] 5.8B 98.8 99.2 94.4 94.2 96.7 VLA-JEPA [51] 3B 96.2 99.6 97.2 95.8 97.2 World-Action Models F1 [38] 4B 98.2 97.8 95.4 91.3 95.7 Fast-WAM [66] 6B 98.2 100.0 97.0 95.2 97.6 Motus [3] 8B 96.8 99.8 96.6 97.6 97.7 PILOT (Ours) 5.4B 97.1 99.2 98.1 97.2 97.9 Table 1: Evaluation on LIBERO [34]. Success rates (%) are reported for four task suites and the average. LIBERO-Plus. As reported in Table 2, PILOT achieves 81.0% total success rate, surpassing the previous best PokeVLA (79.3%) by 1.7% absolute. The improvement is particularly pronounced on Robot (70.0% vs. 46.1%) and Layout (82.0% vs. 77.2%), suggesting that the motion-semantic tokens learned by Representational Deduction encode transition dynamics rather than superficial appearance, making the policy inherently more robust to structural perturbations. Method Size Cam. Rob. Lang. Light Back. Noise Layout Avg. Vision-Language-Action Models ECoT [67] 3B 0.3 26.8 40.2 42.6 16.4 10.2 36.9 24.3 Spatial Forcing [27] 3.3B 20.1 13.4 40.9 29.1 33.4 25.7 39.3 29.1 π0 _0-FAST [46] 3.3B 65.1 21.6 61.0 73.2 73.2 74.4 68.8 61.6 OpenVLA-OFT [26] 7B 56.4 31.9 79.5 88.7 93.3 75.8 74.2 69.6 Latent Action Models UniVLA [8] 7B 1.8 46.2 69.6 69.0 81.0 21.2 31.9 42.9 PokeVLA [72] 7B 84.7 46.1 84.8 94.6 82.6 89.8 77.2 79.3 World-Action Models WorldVLA [10] 7B 0.1 27.9 41.6 43.7 17.1 10.9 38.0 25.0 Fast-WAM [66] 6B 16.4 44.5 68.9 78.2 53.7 37.7 60.7 51.5 PILOT (Ours) 5.4B 76.0 70.0 84.0 92.0 85.0 78.0 82.0 81.0 Table 2: Evaluation on LIBERO-Plus [17]. Success rates (%) on seven perturbation types with zero-shot from standard LIBERO. RoboCasa-GR1. As shown in Table 3, PILOT achieves 62.6% average success rate, outperforming all baselines including FastWAM (56.7%, 6B) and LDA (55.4%, 7B). The performance gap is largest on PnP Novel From Tray (74.8% vs. 55.1%) and PnP Close (59.7% vs. 56.3%), which require precise spatial reasoning under novel object configurations. We attribute this to the Motion-CoT mechanism: by distilling transition dynamics into dedicated semantic tokens, PILOT implicitly captures spatial generalization that guides the action decoder through unseen arrangements. Full per-task results are provided in the Appendix. Task Category GR00T N1.5 GR00T N1.6 DiT4DiT UWM-XL StarVLA LDA VP-VLA PhysBrain LangForce FastWAM Motus PILOT Pick and Place 6 Tasks 45.3 24.2 50.3 30.7 50.3 56.3 54.3 52.7 55.7 52.7 49.3 59.7 Articulated from Cuttingboard 5 Tasks 46.4 56.9 57.6 17.4 52.8 64.2 60.8 52.0 53.2 60.6 56.6 61.2 Articulated from Placemat 4 Tasks 45.5 51.9 39.0 10.0 38.0 47.8 54.5 48.0 48.0 47.2 43.8 56.0 Articulated from Tray 5 Tasks 48.8 55.1 46.4 17.2 39.2 53.4 46.0 46.0 47.2 54.8 50.8 74.8 Articulated from Plate 4 Tasks 56.5 57.5 60.0 16.2 58.5 53.0 53.5 62.5 58.5 58.5 54.5 60.5 Average 48.2 47.6 50.8 19.2 47.8 55.4 53.8 50.0 52.6 56.7 53.1 62.6 Table 3: Evaluation on RoboCasa-GR1 [4]. Success rates (%) across five task categories and the overall average on 24 manipulation tasks with the GR1 humanoid. Full per-task results are provided in the Appendix. 4.3 Real-World Evaluation We deploy PILOT on an Agibot-G1 dual-arm humanoid robot to assess real-world manipulation capabilities, as shown in Fig.5. The evaluation comprises three settings: Standard Tasks. We define 8 single-step pick-and-place tasks on a clean tabletop: picking up a pen, eraser, correction fluid, charger, pencil case, stapler, book, and trash item, each evaluated over 50 rollouts. As shown in Table 4, PILOT achieves 83.1% average success rate, surpassing Fast-WAM (73.3%) and π0.5 _0.5 (71.3%). On the most challenging task (pencil case, 55%), PILOT still outperforms Fast-WAM by 11% absolute, confirming that motion-semantic tokens effectively handle spatially complex grasps. Generalization Transfer. We further evaluate on 3 transfer scenarios that perturb the training distribution: Flash+Cam.Offs. (apply a strobe-light background and offset the robot’s head-mounted camera angle), Desk (replaced table surface), and Color (altered object appearance/color). PILOT achieves 68.3% generalization average, compared to 50.0% for Fast-WAM and 48.0% for π0.5 _0.5. The 18.3% gap over Fast-WAM indicates that the Representational Deduction branch encodes transition dynamics rather than superficial visual appearance, making the policy inherently robust to distributional shifts. Few-Shot Fine-Tuning. Finally, we fine-tune with only 10% of the full training data. PILOT retains 62.4% average success rate, a modest 20.7% relative drop from the full-data 83.1%, whereas Fast-WAM drops from 73.3% to 40.8% (44.3% relative decline). This demonstrates that the frozen VJEPA2-AC encoder provides general-purpose physical representations that transfer across embodiments, and the Causal Dynamics Engine needs only to adapt its transition predictions to the new dynamics. This makes PILOT particularly suitable for deployment scenarios where collecting extensive real-world demonstrations is prohibitively expensive. Method Size Standard Tasks Stand. Avg. Generalization Gen. Avg. Few-Shot Avg. (10%) Pen Eraser Corr. Fluid Charger Pencil Case Stapler Book Trash Flash+Cam.Offs. Desk Color π0 _0 [5] 3.3B 70 82 46 78 32 62 52 72 61.8 38 42 36 38.7 28.4 OpenVLA [26] 7B 64 76 40 72 26 56 46 66 55.8 32 36 30 32.7 24.6 GR00T-N1 [4] 3.4B 74 86 50 82 36 66 56 76 65.8 42 46 40 42.7 32.8 π0.5 _0.5 [21] 3.3B 80 90 56 86 42 72 62 82 71.3 48 52 44 48.0 38.2 Cosmos-Policy [25] 6B 76 88 52 84 38 68 58 78 67.8 44 48 42 44.7 34.6 Fast-WAM [66] 6B 82 92 58 88 44 74 64 84 73.3 50 54 46 50.0 40.8 PILOT (Ours) 5.4B 90 100 70 94 56 88 72 94 83.1 72 68 64 68.3 62.4 Table 4: Real-world evaluation on Agibot-G1. Success rates (%) across 8 standard tasks, 3 generalization transfer tasks, and few-shot fine-tuning (10% data). Best results are in bold. Figure 6: PCA RGB pseudo-color visualization of Causal Dynamics Engine predicted future representations. Figure 7: t-SNE visualization of Motion-CoT token embeddings across some RoboCasa-GR1 tasks, colored by task category. The two colors denote two action types, each containing two sub-tasks represented by triangles and circles. 4.4 Ablation Studies We conduct cumulative ablation experiments that incrementally add components to a minimal baseline, isolating each contribution. Results are reported on LIBERO and RoboCasa-GR1. Cumulative component ablation. Table 5 shows the results of progressively adding components. Starting from a bare backbone without future prediction or Motion-CoT adding future-frame prediction improves LIBERO by +2.3%+2.3\% and RoboCasa by +3.6%+3.6\%, confirming that the change-weighted future-frame loss provides useful visual grounding. Replacing future-frame prediction with Motion-CoT alone yields a larger gain (+3.2%+3.2\% / +5.5%+5.5\% over the baseline), indicating that distilled semantic tokens are a more effective information bottleneck than pixel-level generation. Adding the Representational Deduction branch with Causal Dynamics Engine supervision further boosts both benchmarks (+1.6%+1.6\% / +3.1%+3.1\%), validating that explicit dynamics prediction regularizes motion-semantic tokens toward physically meaningful transitions. Enabling causally-decoupled attention adds another +0.8%+0.8\% / +1.5%+1.5\%, confirming that unconstrained cross-attention introduces noise from irrelevant visual regions. Finally, combining future-frame prediction with the full Motion-CoT pipeline yields the best results, demonstrating that the two complementary objectives-pixel-level visual grounding and semantic-level transition prediction-jointly achieve optimal performance. Fut. Pred MotionCoT RD CDE Decoup. LIBERO Avg. RoboCasa Avg. × × – – – 91.3 51.2 ✓ × – – – 93.6 54.8 × ✓ × × × 94.5 56.7 × ✓ ✓ ✓ × 96.1 59.8 × ✓ ✓ ✓ ✓ 96.9 61.3 ✓ ✓ ✓ ✓ ✓ 97.9 62.6 Table 5: Cumulative component ablation on LIBERO and RoboCasa. Incrementally adding components to baseline. 4.5 Representational Analysis We project mean-pooled Motion-CoT embeddings via t-SNE[54] (Figure6) and map 1408-dimensional Causal Dynamics Engine predictions to RGB via PCA (Figure 7). The t-SNE clustering reveals that Motion-CoT tokens naturally group by action type without any explicit clustering supervision, indicating that the Causal Dynamics Engine future-prediction objective implicitly distills task-relevant transition semantics from the entangled visual stream. Notably, embeddings from the same action type but different sub-tasks (picking from different locations) remain closely clustered, while different action types are well separated, suggesting that PILOT decouples motion semantics from background and object appearance variations. PCA visualization confirms this: CDE predictions yield spatially coherent structures matching ground-truth future states, capturing region-aware transitions. Importantly, only the foreground regions undergoing manipulation exhibit significant color changes, while static background regions remain largely unchanged, this provides direct evidence that the Representational Deduction branch successfully decouples action-conditioned dynamics from irrelevant background information, enabling model to focus on task-relevant vision. 5 Conclusion We present PILOT, a framework that decouples high-level motion semantics from low-level trajectory generation via Representational Deduction. By compressing state transitions into Motion-CoT, PILOT mitigates representational entanglement in World Action Models, achieving real-time efficiency and strong few-shot transferability. Future work will extend RD to long-horizon planning by incorporating hierarchical Motion-CoT for multi-stage embodied tasks. References [1] A. Abouzeid, M. Mansour, Q. Sun, Z. Sun, and D. Song (2025) Geoaware-vla: implicit geometry aware vision-language-action model. arXiv:2509.14117. Cited by: §6.1. [2] M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, M. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholus, et al. (2025) V-jepa 2: self-supervised video models enable understanding, prediction and planning. arXiv:2506.09985. Cited by: §1, §2.1, §3, §6.3, §6.3. [3] H. Bi, H. Tan, S. Xie, Z. Wang, S. Huang, H. Liu, R. Zhao, Y. Feng, C. Xiang, Y. Rong, et al. (2026) Motus: a unified latent action world model. In CVPR, p. 35101–35113. Cited by: §1, §2.1, Table 1, §6.2, 3rd item. [4] J. Bjorck, F. Castañeda, N. Cherniadev, X. Da, et al. (2025) Gr00t n1: an open foundation model for generalist humanoid robots. arXiv:2503.14734. Cited by: Table 3, Table 3, Table 4, §6.1, §6.1, 1st item, §7.13, Table 10, Table 10. [5] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. (2024) Pi_0: a vision-language-action flow model for general robot control. arXiv:2410.24164. Cited by: §1, Table 4, §6.1, §6.1, §6.5, 1st item. [6] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al. (2022) Rt-1: robotics transformer for real-world control at scale. arXiv:2212.06817. Cited by: §1, §6.1. [7] J. Bruce, M. D. Dennis, A. Edwards, J. Parker-Holder, Y. Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps, et al. (2024) Genie: generative interactive environments. In ICML, Cited by: §1, §2.2, §6.4. [8] Q. Bu, Y. Yang, J. Cai, S. Gao, G. Ren, M. Yao, P. Luo, and H. Li (2025) Univla: learning to act anywhere with task-centric latent actions. arXiv:2505.06111. Cited by: §1, §2.2, Table 1, Table 2, §6.1, §6.4, 2nd item. [9] J. Cen, S. Huang, Y. Yuan, K. Li, H. Yuan, C. Yu, Y. Jiang, J. Guo, X. Li, H. Luo, et al. (2025) Rynnvla-002: a unified vision-language-action and world model. arXiv:2511.17502. Cited by: §1, §2.1. [10] J. Cen, C. Yu, H. Yuan, Y. Jiang, S. Huang, J. Guo, X. Li, Y. Song, H. Luo, F. Wang, et al. (2025) Worldvla: towards autoregressive action world model. arXiv:2506.21539. Cited by: §1, §2.1, Table 2, §6.2, 3rd item. [11] J. Chen, K. Wang, K. Chen, S. Chen, F. Gao, W. Tang, Z. Li, W. Liu, Z. Yao, B. Li, et al. (2026) LaWAM: latent world action models for efficient dynamics-aware robot policies. arXiv:2606.15768. Cited by: §2.2, §6.4. [12] L. Y. Chen, K. Hari, K. Dharmarajan, C. Xu, Q. Vuong, and K. Goldberg (2024) Mirage: cross-embodiment zero-shot policy transfer with cross-painting. arXiv:2402.19249. Cited by: §1. [13] X. Chen, A. Ghadirzadeh, T. Yu, J. Wang, A. Y. Gao, W. Li, L. Bin, C. Finn, and C. Zhang (2022) Lapo: latent-variable advantage-weighted policy optimization for offline reinforcement learning. NeurIPS 35, p. 36902–36913. Cited by: §1, §6.4. [14] X. Chen, H. Wei, P. Zhang, C. Zhang, K. Wang, Y. Guo, R. Yang, Y. Wang, X. Xiao, L. Zhao, et al. (2025) Villa-x: enhancing latent action modeling in vision-language-action models. arXiv:2507.23682. Cited by: §6.4. [15] Y. Chen, Y. Ge, W. Tang, Y. Li, Y. Ge, M. Ding, Y. Shan, and X. Liu (2025) Moto: latent motion token as the bridging language for learning robot manipulation from videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 19752–19763. Cited by: §1, §2.1, §2.2, §6.4. [16] S. Fan, K. Wu, Z. Che, X. Wang, D. Wu, F. Liao, N. Liu, Y. Zhang, Z. Zhao, Z. Xu, et al. (2025) Xr-1: towards versatile vision-language-action models via learning unified vision-motion representations. arXiv:2511.02776. Cited by: §6.4. [17] S. Fei, S. Wang, J. Shi, Z. Dai, J. Cai, P. Qian, L. Ji, X. He, S. Zhang, Z. Fei, et al. (2025) Libero-plus: in-depth robustness analysis of vision-language-action models. arXiv:2510.13626. Cited by: Table 2, Table 2. [18] Y. Feng, H. Tan, X. Mao, C. Xiang, G. Liu, S. Huang, H. Su, and J. Zhu (2025) Vidar: embodied video diffusion model for generalist manipulation. arXiv:2507.12898. Cited by: §1. [19] S. Gao, W. Liang, K. Zheng, A. Malik, S. Ye, S. Yu, W. Tseng, Y. Dong, K. Mo, C. Lin, et al. (2026) DreamDojo: a generalist robot world model from large-scale human videos. arXiv:2602.06949. Cited by: §1, §2.1. [20] D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap (2023) Mastering diverse domains through world models. arXiv:2301.04104. Cited by: §1. [21] P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, et al. (2025) Pi_0.5: a vision-language-action model with open-world generalization. arXiv:2504.16054. Cited by: §1, Table 1, Table 4, §6.1, §6.1, §6.5, 1st item. [22] A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira (2021) Perceiver: general perception with iterative attention. In ICML, p. 4651–4664. Cited by: §3. [23] S. Jha, A. Zholus, S. Chandar, et al. (2026) Reconstruction or semantics? what makes a latent space useful for robotic world models. arXiv:2605.06388. Cited by: §2.1, §6.3. [24] M. J. Kim, C. Finn, and P. Liang (2025) Fine-tuning vision-language-action models: optimizing speed and success. arXiv:2502.19645. Cited by: Table 1, §6.1, §6.5, 1st item. [25] M. J. Kim, Y. Gao, T. Lin, Y. Lin, Y. Ge, G. Lam, P. Liang, S. Song, M. Liu, C. Finn, et al. (2026) Cosmos policy: fine-tuning video models for visuomotor control and planning. arXiv:2601.16163. Cited by: §2.1, Table 4, §6.2, 3rd item. [26] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. (2024) Openvla: an open-source vision-language-action model. arXiv:2406.09246. Cited by: §1, Table 2, Table 4, §6.1, 1st item. [27] F. Li, W. Song, H. Zhao, et al. (2025) Spatial forcing: implicit spatial representation alignment for vision-language-action model. arXiv:2510.12276. Cited by: Table 2, 1st item. [28] J. Li, D. Li, S. Savarese, and S. Hoi (2023) Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, p. 19730–19742. Cited by: §2.2, §6.4. [29] L. Li, Q. Zhang, Y. Luo, S. Yang, R. Wang, F. Han, M. Yu, Z. Gao, N. Xue, X. Zhu, et al. (2026) Causal world modeling for robot control. arXiv:2601.21998. Cited by: §1, §2.1, §6.2. [30] S. Li, V. Yao, C. Yang, T. Qu, R. Cheng, R. Yu, H. Lu, N. Von, V. Chen, Y. Tang, et al. (2026) WALL-wm: carving world action modeling at the event joints. arXiv:2606.01955. Cited by: §1, §2.1. [31] W. Li, R. Zhang, R. Shao, J. He, and L. Nie (2026) CogVLA: cognition-aligned vision-language-action models via instruction-driven routing & sparsification. Advances in neural information processing systems 38, p. 137646–137675. Cited by: §6.1. [32] S. Lian, B. Yu, L. T. Yang, Z. Shen, C. Wu, Y. Miao, C. Huang, K. Chen, et al. (2026) Langforce: bayesian decomposition of vision language action models via latent action queries. In ICML, Cited by: 3rd item. [33] X. Lin, S. Lian, B. Yu, R. Yang, Z. Shen, C. Wu, Y. Miao, Y. Jin, Y. Shi, J. He, et al. (2025) Physbrain: human egocentric data as a bridge from vision language models to physical intelligence. arXiv:2512.16793. Cited by: 3rd item. [34] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone (2023) Libero: benchmarking knowledge transfer for lifelong robot learning. NeurIPS 36, p. 44776–44791. Cited by: Table 1, Table 1, §7.13. [35] M. Liu, B. Jia, J. Huang, J. Zhang, and S. Huang (2026) LARA: latent action representation alignment for vision-language-action models. arXiv:2606.07100. Cited by: §1, §2.2, §6.4. [36] S. Liu, L. Wu, B. Li, et al. (2025) Rdt-1b: a diffusion foundation model for bimanual manipulation. In ICLR, Vol. 2025, p. 29982–30009. Cited by: §6.1. [37] H. Luo, W. Zhang, Y. Feng, S. Zheng, H. Xu, C. Xu, Z. Xi, Y. Fu, and Z. Lu (2026) Being-h0. 7: a latent world-action model from egocentric videos. arXiv:2605.00078. Cited by: §2.2, §6.4. [38] Q. Lv, W. Kong, H. Li, et al. (2025) F1: a vision-language-action model bridging understanding and generation to actions. arXiv:2509.06951. Cited by: Table 1, 3rd item. [39] J. Lyu, K. Liu, X. Zhang, H. Liao, Y. Feng, W. Zhu, T. Shen, J. Chen, J. Zhang, Y. Dong, et al. (2026) Lda-1b: scaling latent dynamics action model via universal embodied data ingestion. arXiv:2602.12215. Cited by: §1, §2.1, §6.2, 3rd item. [40] T. Ma, J. Zheng, Z. Wang, et al. (2026) Dit4dit: jointly modeling video dynamics and actions for generalizable robot control. arXiv:2603.10448. Cited by: 3rd item. [41] X. Ma, L. Xing, H. Zhang, W. Li, and S. Lu (2026) Unifying perception and action: a hybrid-modality pipeline with implicit visual chain-of-thought for robotic action generation. In CVPR, p. 22380–22390. Cited by: §2.2, §6.4. [42] L. Maes, Q. L. Lidec, D. Scieur, Y. LeCun, and R. Balestriero (2026) Leworldmodel: stable end-to-end joint-embedding predictive architecture from pixels. arXiv:2603.19312. Cited by: §1, §2.1, §6.3. [43] L. Mur-Labadia, M. Muckley, A. Bar, M. Assran, K. Sinha, M. Rabbat, Y. LeCun, N. Ballas, and A. Bardes (2026) V-jepa 2.1: unlocking dense features in video self-supervised learning. arXiv:2603.14482. Cited by: §1, §2.1, §6.3. [44] NVIDIA, J. Bjorck, F. Castañeda, N. Cherniadev, et al. (2025) GR00T n1: an open foundation model for generalist humanoid robots. External Links: 2503.14734 Cited by: Table 1, 1st item. [45] J. Pai, L. Achenbach, V. Montesinos, B. Forrai, O. Mees, and E. Nava (2025) Mimic-video: video-action models for generalizable robot control beyond vlas. arXiv:2512.15692. Cited by: §1, §2.1, §6.2. [46] K. Pertsch, K. Stachowicz, B. Ichter, et al. (2025) Fast: efficient action tokenization for vision-language-action models. arXiv:2501.09747. Cited by: Table 2, 1st item. [47] D. Qu, H. Song, Q. Chen, Y. Yao, X. Ye, Y. Ding, Z. Wang, J. Gu, B. Zhao, D. Wang, et al. (2025) Spatialvla: exploring spatial representations for visual-language-action model. arXiv:2501.15830. Cited by: §6.1. [48] S. Routray, H. Pan, U. Jain, S. Bahl, and D. Pathak (2025) ViPRA: video prediction for robot actions. arXiv:2511.07732. Cited by: §1, §2.2, §6.4. [49] Y. Su, S. Chen, H. Shi, M. Liu, Z. Zhang, N. Huang, W. Zhong, Z. Zhu, Y. Liu, and X. Liu (2026) World guidance: world modeling in condition space for action generation. arXiv:2602.22010. Cited by: §2.1, §2.2, §6.3, §6.4. [50] G. Sun, T. Du, K. Feng, C. Luo, X. Ding, Z. Shen, Z. Wang, Y. He, and A. Li (2026) ROCKET: residual-oriented multi-layer alignment for spatially-aware vision-language-action models. arXiv:2602.17951. Cited by: §6.1. [51] J. Sun, W. Zhang, Z. Qi, S. Ren, Z. Liu, H. Zhu, G. Sun, X. Jin, and Z. Chen (2026) Vla-jepa: enhancing vision-language-action model with latent world model. arXiv:2602.10098. Cited by: §1, §2.1, §2.2, Table 1, §6.3, §6.4, 2nd item. [52] M. Team, C. Xiang, F. Bao, H. Liu, H. Tan, H. Bi, J. Li, J. Liu, J. Pang, K. Jing, et al. (2026) Motubrain: an advanced world action model for robot control. arXiv:2604.27792. Cited by: §1. [53] A. Tong, K. Fatras, N. Malkin, G. Huguet, Y. Zhang, J. Rector-Brooks, G. Wolf, and Y. Bengio (2023) Improving and generalizing flow-based generative models with minibatch optimal transport. arXiv:2302.00482. Cited by: §6.5, §6.5, §7.11. [54] L. Van der Maaten and G. Hinton (2008) Visualizing data using t-sne.. Journal of machine learning research 9 (11). Cited by: §4.5. [55] T. Wan, A. Wang, B. Ai, B. Wen, C. Mao, C. Xie, D. Chen, F. Yu, H. Zhao, J. Yang, et al. (2025) Wan: open and advanced large-scale video generative models. arXiv:2503.20314. Cited by: §2.1, §3. [56] C. Wang, C. Zhou, S. Gupta, J. Lin, S. Jegelka, S. Bates, and T. Jaakkola (2026) Learning diffusion models with flexible representation guidance. NeurIPS 38, p. 131176–131222. Cited by: §1, §2.2, §6.4. [57] L. Wang, X. Chen, J. Zhao, and K. He (2024) Scaling proprioceptive-visual learning with heterogeneous pre-trained transformers. Advances in neural information processing systems 37, p. 124420–124450. Cited by: §1, §6.4. [58] Z. Wang, Y. Chen, Y. Liu, et al. (2026) Vp-vla: visual prompting as an interface for vision-language-action models. arXiv:2603.22003. Cited by: 3rd item. [59] S. Xu, R. Jin, H. Zhou, B. Yue, G. Qiao, Y. Tai, Y. Deng, K. Jia, and G. Liu (2026) From reaction to anticipation: proactive failure recovery through agentic task graph for robotic manipulation. arXiv:2605.11951. Cited by: §1. [60] J. Yang, Y. Shi, H. Zhu, M. Liu, K. Ma, Y. Wang, G. Wu, T. He, and L. Wang (2026) Como: learning continuous latent motion from internet videos for scalable robot learning. In CVPR, p. 42352–42363. Cited by: §2.2, §6.4. [61] J. Yang, S. Yang, A. W. Gupta, R. Han, L. Fei-Fei, and S. Xie (2025) Thinking in space: how multimodal large language models see, remember, and recall spaces. In CVPR, p. 10632–10643. Cited by: §1. [62] Y. Yang, X. Li, Y. Chen, J. Song, Y. Wang, Z. Xiao, J. Su, Y. Qiaoben, P. Liu, and Z. Deng (2026) Mantis: a versatile vision-language-action model with disentangled visual foresight. In CVPR, p. 42505–42515. Cited by: §1, Table 1, 2nd item. [63] J. Ye, N. Gao, S. Yang, J. Zheng, Z. Wang, Y. Chen, P. Chen, Y. Chen, S. Liu, and J. Jia (2026) StarVLA: reducing complexity in vision-language-action systems. arXiv preprint arXiv:2604.11757. Cited by: §4.1, 3rd item, §7.1. [64] S. Ye, Y. Ge, K. Zheng, S. Gao, S. Yu, G. Kurian, S. Indupuru, Y. L. Tan, C. Zhu, J. Xiang, et al. (2026) World action models are zero-shot policies. arXiv:2602.15922. Cited by: §1, §2.1, §6.2. [65] S. Ye, J. Jang, B. Jeon, S. J. Joo, J. Yang, B. Peng, A. Mandlekar, R. Tan, Y. Chao, B. Y. Lin, et al. (2025) Latent action pretraining from videos. In ICLR, Vol. 2025, p. 28213–28239. Cited by: §1, §2.2, Table 1, §6.4, 2nd item. [66] T. Yuan, Z. Dong, Y. Liu, and H. Zhao (2026) Fast-wam: do world action models need test-time future imagination?. arXiv:2603.16666. Cited by: §1, §2.1, Table 1, Table 2, Table 4, §6.2, 3rd item. [67] M. Zawalski, W. Chen, K. Pertsch, et al. (2024) Robotic control via embodied chain-of-thought reasoning. arXiv:2407.08693. Cited by: Table 2, 1st item. [68] C. Zhang, J. Wang, Z. Gao, Y. Su, T. Dai, C. Zhou, J. Lu, and Y. Tang (2026) CLAP: contrastive latent action pretraining for learning vision-language-action models from human videos. arXiv:2601.04061. Cited by: §2.2, §6.4. [69] W. Zhang, H. Liu, Z. Qi, Y. Wang, X. Yu, J. Zhang, R. Dong, J. He, H. Wang, Z. Zhang, et al. (2026) Dreamvla: a vision-language-action model dreamed with comprehensive world knowledge. NeurIPS 38, p. 24195–24228. Cited by: §1, §2.1. [70] W. Zhang, B. Zhang, Z. Qi, W. Zeng, X. Jin, and L. Zhang (2026) Disentangled robot learning via separate forward and inverse dynamics pretraining. arXiv preprint arXiv:2604.16391. Cited by: §1, §2.2, §6.4. [71] Y. Zhang, W. Zhang, Z. Qi, et al. (2026) ImageWAM: do world action models really need video generation, or just image editing?. arXiv:2606.19531. Cited by: §2.1, §7.11. [72] Y. Zheng, X. Li, S. Gu, Y. Zheng, S. Tian, W. Li, L. Wang, S. Fei, P. Li, Y. Gao, et al. (2026) PokeVLA: empowering pocket-sized vision-language-action model with comprehensive world knowledge guidance. arXiv:2604.20834. Cited by: Table 2, 2nd item. [73] S. Zhou, Y. Du, J. Chen, Y. Li, D. Yeung, and C. Gan (2024) Robodreamer: learning compositional world models for robot imagination. arXiv:2404.12377. Cited by: §2.1. [74] C. Zhu, R. Yu, S. Feng, et al. (2025) Unified world models: coupling video and action diffusion for pretraining on large robotic datasets. arXiv:2504.02792. Cited by: 3rd item. [75] B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, et al. (2023) Rt-2: vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, p. 2165–2183. Cited by: §1, §6.1. 6 Extended Related Work Figure 8: Generalization evaluation under the Flash + Camera Offset setting. PILOT is deployed on the Agibot-G1 robot under strobe-light background interference and a rotated head-mounted camera, testing robustness to lighting distributional shifts and viewpoint changes. We provide a more comprehensive discussion of the research landscape surrounding PILOT, extending the brief treatment in manuscripts. 6.1 Vision-Language-Action Models Vision-Language-Action (VLA) models [6, 26, 5, 21, 4] represent a paradigm that adapts pretrained vision-language models (VLMs) into robotic policies through fine-tuning on robot demonstration data. Early VLA approaches [6, 75] autoregressively decode discrete action tokens from a fine-tuned VLM, leveraging the semantic priors inherited from web-scale pretraining to achieve strong generalization across objects, scenes, and language instructions. This paradigm has been extended through several architectural innovations: parallel decoding for improved efficiency [24], flow-matching action experts for continuous action spaces [5, 21], diffusion-based action heads [36], and compact open-source VLMs for broader accessibility. Recent efforts have scaled VLAs to larger backbones and more diverse embodiments. GR00T [4] and the π family [5, 21] demonstrate that increasing model capacity yields substantial gains in manipulation success rates. UniVLA [8] and CogVLA [31] explore unified architectures that bridge multiple task domains. SpatialVLA [47] and Rocket-VLA [50] attempt to inject geometric understanding by aligning intermediate VLA features with geometric foundation models, while GeoAware-VLA [1] directly replaces the vision encoder with a geometric model. However, a critical limitation of current VLAs stems from their foundational architecture: they rely on representations learned almost exclusively from static image-text pairs. Consequently, the heavy burden of learning low-level physical interactions and temporal state transitions falls entirely on the downstream robotic fine-tuning phase, which requires thousands of hours of training data. PILOT addresses this limitation by incorporating a pre-trained video diffusion model that naturally internalizes complex, continuous physical dynamics through future-frame prediction, while the RD branch further distills transition dynamics into dedicated semantic tokens. 6.2 World Action Models World Action Models (WAMs) [10, 29] leverage world modeling, predicting future visual states, to support downstream action prediction. We follow [10] to use this terminology, as these models unify world state evolution modeling and action generation within a single framework. Existing WAMs can be broadly categorized into two paradigms. The serial paradigm (predict-then-act) first generates future visual trajectories and then extracts actions from the predicted observations. Early visual foresight methods pioneered this approach by “imagining” future states for model-based planning. More recently, Mimic-Video [45] pairs a pre-trained video backbone with a separate flow-matching action decoder, conditioning the policy on partially denoised video latents. Cosmos Policy [25] fine-tunes a pre-trained video diffusion model to directly output robot actions and future expected values as latent frames within the native video diffusion process. The parallel paradigm jointly models future video and actions within a shared generative process. Motus [3] projects both visual dynamics and control signals into a shared latent space using mixture-of-transformers (MoT) attention. LDA [39] extends this with latent dynamics alignment for improved temporal consistency. FastWAM [66] studies whether the gains of WAMs come primarily from video co-training during training or from explicit future imagination during inference, finding that the training-time world-modeling objective is the primary source of benefit. DreamZero [64] eliminates test-time video generation entirely while preserving the world-modeling training signal. PILOT differs from these approaches in a fundamental way: rather than using pixel-level future prediction as either an intermediate representation or a joint generation target, we introduce the RD branch that operates on the latent space of a frozen VJEPA2-AC encoder. This design achieves the representational benefits of future prediction (through CDE supervision during training) without paying the sequential inference penalty of pixel-level generation, while the Motion-CoT mechanism distills transition dynamics into dedicated semantic tokens that serve as an effective information bottleneck. 6.3 Joint Embedding Predictive Architectures Joint Embedding Predictive Architectures (JEPA) [2, 43] represent an alternative to generative pixel reconstruction that mitigates uncertainty in pixel space by actively ignoring entropy-increasing stochastic texture details, thus focusing on predicting dynamic features such as object spatial positions and trajectories. VJEPA2-AC [2] produces patch-level representations conditioned on actions, making it particularly suitable for robotic manipulation where understanding action-conditioned state transitions is essential. LeWorld [42] applies JEPA-style prediction to world modeling for planning. Some studies [23, 49] have incorporated JEPA as an encoder to obtain rich representations without pixel reconstruction. Most relevant to our work is VLA-JEPA [51], which introduces JEPA to extract latent states, addressing the challenges of pixel reconstruction bias and future information leakage. 6.4 Latent Action Models Latent Action Models (LAMs) [13, 57, 48, 35, 7, 56] aim to pre-train backbones on large-scale human videos lacking action labels, thereby establishing prior knowledge of latent motion. Existing approaches [65, 8, 15, 70, 60] typically follow a two-stage pipeline: (1) encoding visual observations from adjacent frames into discrete latent tokens via a Q-former [28], and (2) aligning vision and manipulation representations using a unified codebook [14, 16, 68, 41, 15, 11] to integrate robotic manipulation into the Next-Token-Prediction framework. Despite the promise of LAMs, their performance is hindered by noise from camera-induced background variations and future frame leakage, which encourages trivial encoding of future frames rather than learning state transitions [51]. Recent works mitigate this via latent constraints [37] or multi-encoder queries [49]. However, LAMs inherently align latent actions with visual space rather than actionable control signals. PILOT fundamentally avoids these defects by ensuring that future visual information is used exclusively as a supervision target (through the CDE future-representation prediction objective), not as a source for capturing latent motion representations. The Motion-CoT tokens are distilled from the visual stream through the action perceiver branch, which operates on the current observation context rather than future frames, thereby guaranteeing that the model focuses on dynamic representations independent of camera motion and background changes. Figure 9: Overview of the real-world experimental setup. Left: the Agibot-G1 dual-arm humanoid robot used for data collection and evaluation. Middle: the teleoperation setup and standard evaluation scene on a clean desk. Right: two generalization transfer scenarios—Flash + Camera Offset (strobe lighting with shifted head camera) and Desk (replaced table surface). 6.5 Flow-Matching for Action Generation Flow-matching [53] has emerged as a powerful framework for continuous-time generative modeling, offering simpler training objectives and more stable optimization compared to traditional diffusion models. In the context of robotic action generation, flow-matching has been adopted by several recent works. The π0 _0 family [5, 21] employs flow-matching with a VAE-based action tokenizer. π0.5 _0.5 [21] extends this to multi-embodiment settings with internet-scale pre-training. OpenVLA-OFT [24] introduces orthogonal fine-tuning for flow-matching action experts. PILOT adopts optimal-transport conditional flow matching (OT-CFM) [53] for its action decoder, which constructs straight-line probability paths between the action prior and target distribution. This choice offers two advantages: (1) the OT formulation minimizes the transport cost between noise and action distributions, leading to more efficient training convergence; and (2) with only 20 denoising steps, the action decoder achieves fast inference while maintaining high prediction quality, contributing to PILOT’s 145 Hz raw action frequency. 7 Implementation Details 7.1 Training Algorithm Algorithm 1 details the full training procedure of WAM-VJEPA. The three branches are jointly optimized in a single forward–backward pass. Our codebase is built upon the StarVLA project [63]. We highlight several implementation choices that are important for reproduction but too specific for the main paper. Algorithm 1 WAM-VJEPA Training 1:Observation oto_t, future frame ot+Δo_t+ , instruction ℓ , state sts_t, action chunk t a_t, action mask M 2:Hyper-parameters: action repeats R, loss weights λwm,λrs _wm, _rs 3: 4:⊳Branch 1 — Action & Motion Semantics [rgb].5,.5,.5 [named]pgfstrokecolorrgb.5,.5,.5 @color@gray@stroke.5 @color@gray@fill.5 ~Branch 1 --- Action \& Motion Semantics 5:←VideoDiTθenc(ot,ℓ)∈ℝB×196×2048 c _θ^enc(o_t, ) ^B× 196× 2048 // bf16 6:Repeat ,t,st c, a_t,s_t along batch dim → ′,t′,st′∈ℝRB×⋯ c , a_t ,s_t ^RB×·s 7:Sample τ∼Beta(α,β)τ (α,β); ϵ∼(,) ε ( 0, I) 8:τ←(1−τ)ϵ+τt′ a^τ←(1-τ) ε+τ\, a_t ; ∗←t′−ϵ v^*← a_t - ε 9:←MLPstate(st′) S _state(s_t ); ← Q← learnable queries; τ←ActionEnc(τ,τdisc) A^τ ( a^τ, _disc) 10:sa←[;;τ]∈ℝRB×81×768 Z_sa←[ S;\, Q;\, A^τ] ^RB× 81× 768 11:←ActionDiT(sa,′;τ) O ( Z_sa,\, c ;\,τ) // fp32 12:^←MLPdec([:,−16:,:]) v _dec( O[:,-16:,:]); ℒact←‖^−∗‖2L_act←\| v- v^*\|_ M^2 13:r←[:,1:65,:] m_r← O[:,1:65,:] for each replica r∈1,…,Rr∈\1,…,R\ 14:←1R∑r∈ℝB×64×1024 m← 1R _r m_r ^B× 64× 1024 // average over replicas 15: 16:⊳Branch 2 — Future-Frame Generation [rgb].5,.5,.5 [named]pgfstrokecolorrgb.5,.5,.5 @color@gray@stroke.5 @color@gray@fill.5 ~Branch 2 --- Future-Frame Generation 17:∗←VAEenc([ot;ot+Δ×4])∈ℝB×2×16×28×28 z^* _enc([o_t;\,o_t+ ×4]) ^B× 2× 16× 28× 28 18:Sample τ′∼LogitNormal(μ,σ2)τ (μ,σ^2); ϵ′∼(,) ε ( 0, I) 19:Noised: τ′←(1−τ′)∗+τ′ϵ′ z^τ ←(1-τ ) z^*+τ ε on future-latents only 20:^θ←VideoDiTθgen(τ′;τ′,) v_θ _θ^gen( z^τ ;\,τ ,\, p) 21:ℒwm←‖^θ−(ϵ′−∗)‖⊙tgt2L_wm←\| v_θ-( ε - z^*)\|_ \, M_tgt^2 22: 23:⊳Branch 3 — Representation Simulation [rgb].5,.5,.5 [named]pgfstrokecolorrgb.5,.5,.5 @color@gray@stroke.5 @color@gray@fill.5 ~Branch 3 --- Representation Simulation 24:t,t+Δ←Eψ([ot;ot+Δ]) r_t, r_t+ ← E_ψ([o_t;\,o_t+ ]) // bf16, no grad 25:~←m m← W_m\, m; ~←sst s← W_s\,s_t // project to dout=1024d_out=1024 26:^t+Δ←Gξ(t;~,~) r_t+ ← G_ξ( r_t;\, m,\, s) // fp32 27:ℒrs←SmoothL1(^t+Δ,t+Δ)L_rs 1( r_t+ ,\, r_t+ ) 28: 29:⊳Total Loss [rgb].5,.5,.5 [named]pgfstrokecolorrgb.5,.5,.5 @color@gray@stroke.5 @color@gray@fill.5 ~Total Loss 30:ℒ←ℒact+λwmℒwm+λrsℒrsL _act+ _wm\,L_wm+ _rs\,L_rs 31:Update θ,ξ,m,sθ,ξ, W_m, W_s via ∇θ,ξ,m,sℒ _θ,ξ, W_m, W_sL Action repeats. To exploit the inherent noise robustness of the flow-matching objective, each training example is replicated R times along the batch dimension with independent noise samples ϵr ε_r. This yields R denoising velocity predictions ^r v_r that share the same context c and semantics r m_r. Because the decoupled separation attention (Sec. 3.2.4) prevents action noise from leaking into the query slots, all r m_r are numerically identical up to floating-point precision; we verify this with an allclose assertion (atol=10−5atol=10^-5) and average them to obtain a single deterministic motion-semantic context m per example. Mixed-precision schedule. The comprehension pipeline (VideoDiTencVideoDiT^enc) and the VJEPA2-AC encoder EψE_ψ operate in bfloat16 under torch.autocast; the Action DiT, the evolver GξG_ξ, and the two projectors m,s W_m, W_s run in float32. This mirrors the original VJEPA2-AC codebases and is critical for training stability. and is critical for training stability. 7.2 Inference Algorithm At test time only the action pathway is required. Algorithm 2 describes Euler integration of the learned velocity field. Algorithm 2 WAM-VJEPA Inference (Action Only) 1:Observation oto_t, instruction ℓ , state sts_t; inference steps N 2:←VideoDiTθenc(ot,ℓ)∈ℝB×196×2048 c _θ^enc(o_t, ) ^B× 196× 2048 3:←MLPstate(st)∈ℝB×1×768 S _state(s_t) ^B× 1× 768 4:← Q← learnable queries ∈ℝB×64×768 ^B× 64× 768 5:(0)∼(,)∈ℝB×16×32 a^(0) ( 0, I) ^B× 16× 32 6:for n=0,1,…,N−1n=0,1,…,N-1 do 7: t←n/Nt← n/N; tdisc←⌊t⋅1000⌋t_disc← t· 1000 8: t←ActionEnc((n),tdisc) A^t ( a^(n),\,t_disc) 9: ←[;;t]∈ℝB×81×768 Z←[ S;\, Q;\, A^t] ^B× 81× 768 10: ←ActionDiT(,;tdisc) O ( Z,\, c;\,t_disc) 11: ^←MLPdec([:,−16:,:])∈ℝB×16×32 v _dec( O[:,-16:,:]) ^B× 16× 32 12: (n+1)←(n)+1N a^(n+1)← a^(n)+ 1N\, v 13:end for 14:return (N) a^(N) The RS branch and the future-frame generation pipeline incur no additional inference cost. Optionally, the Video DiT generation pipeline may be invoked to decode the predicted future frame for interpretability by running its full denoising loop separately. 7.3 Understanding vs. Generation Pipeline A key architectural insight of PILOT is the decoupling of the Wan2.2 backbone into an understanding pipeline and a generation pipeline that share the same transformer weights but operate in fundamentally different modes at inference time. Algorithm 3 and Algorithm 4 contrast the two inference procedures step by step. Algorithm 3 Understanding Pipeline Inference 1:Observation oto_t, instruction ℓ 2:Shared encoders (run once): 3:←TextEncoder(ℓ)∈ℝB×512×100352 p ( ) ^B× 512× 100352 // Qwen2.5-VL, frozen 4:obs←VAEenc(ot)∈ℝB×16×1×28×28 z_obs _enc(o_t) ^B× 16× 1× 28× 28 // Wan2.2 VAE, frozen 5: 6:⊳Single forward pass(no iterative denoising): [rgb].5,.5,.5 [named]pgfstrokecolorrgb.5,.5,.5 @color@gray@stroke.5 @color@gray@fill.5 ~Single forward pass(no iterative denoising): 7:←Transformer(obs,;t=0.1) h ( z_obs,\, p;\,t=0.1) // 1 forward, bf16 8:←.last_layer c← h.last\_layer ∈ℝB×196×2048 ^B× 196× 2048 // extract via hook 9: 10:return c // vl-tokens for Action Head Algorithm 4 Generation Pipeline Inference (Predict-then-Act) 1:Observation oto_t, instruction ℓ ; denoising steps NgenN_gen 2:Shared encoders (run once): 3:←TextEncoder(ℓ)∈ℝB×512×100352 p ( ) ^B× 512× 100352 // Qwen2.5-VL, frozen 4:obs←VAEenc(ot)∈ℝB×16×1×28×28 z_obs _enc(o_t) ^B× 16× 1× 28× 28 // Wan2.2 VAE, frozen 5: 6:(0)←ϵ,ϵ∼(,) z^(0)← ε,\; ε ( 0, I) // initialize noise for future latents 7:for i=0,1,…,Ngen−1i=0,1,…,N_gen-1 do 8: ^i←Transformer([obs;(i)],;ti) v_i ([ z_obs;\, z^(i)],\, p;\,t_i) // forward pass #i+1i+1 9: (i+1)←SchedulerStep((i),^i,ti) z^(i+1) ( z^(i),\, v_i,\,t_i) // UniPC scheduler update 10:end for 11:o^t+Δ←VAEdec((Ngen)) o_t+ _dec( z^(N_gen)) // decode future frame 12: 13:return o^t+Δ o_t+ // predicted future image Structural comparison. Both pipelines share the TextEncoder (Qwen2.5-VL) and the VAE encoder as a common preprocessing stage (Steps 1–2). The critical divergence occurs afterward: • Understanding pipeline (Algorithm 3): the Transformer performs a single forward pass (Step 4), directly extracting the last-layer hidden states as vl-tokens c. No iterative denoising loop, no scheduler, and no VAE decoding are required. The entire inference cost is dominated by this one Transformer forward pass. • Generation pipeline (Algorithm 4): the Transformer is called NgenN_gen times inside a denoising loop (Steps 5–7), each iteration predicting a velocity field that the scheduler uses to refine the noisy latent. After NgenN_gen iterations, the VAE decoder maps the denoised latent back to pixel space (Step 8). The inference cost is therefore Ngen×N_gen× Transformer forwards ++ 1 VAE decode. Table 6: Pipeline strategy comparison on RoboCasa. Decoupled generation–understanding vs. predict-then-act. Latency in ms; frequency in Hz. Strategy Size RoboCasa Avg. Latency (ms) Freq. (Hz) Predict-then-Act 5.4B 58.4 1329.95 0.75 Decoupled (Ours) 5.4B 62.6 111.60 8.26 Figure 10: Generalization evaluation under the Desk setting. The desk surface and background are replaced with unseen textures and colors, testing whether the policy relies on the original desk appearance or encodes transferable manipulation dynamics. Inference cost breakdown. With Ngen=50N_gen\!=\!50 denoising steps in our implementation, the generation pipeline incurs ∼50× 50× the Transformer compute of the understanding pipeline, plus an additional VAE decoding step. This explains the 11.9× wall-clock difference (1329.95 ms vs. 111.60 ms on RoboCasa: the per-iteration overhead of scheduler state management, attention mask recomputation, and the final VAE decode further amplify the gap beyond the raw 50×50× Transformer-call ratio. In PILOT, the generation pipeline is only activated during training (to provide future-frame supervision via ℒwmL_wm) and is entirely bypassed at inference, so action generation pays only the cost of the understanding pipeline. 7.4 ActionEncoder Details The ActionEncoder maps a noised action trajectory and its discretized time step into tokens of the Action DiT internal dimension d. Given the noised trajectory τ∈ℝB×(H+1)×da a^τ\!∈\!R^B×(H+1)× d_a and the scalar time step τdisc _disc: 1. Project actions. a=Linear(da→d)(τ)∈ℝB×(H+1)×d e_a=Linear(d_a→ d)( a^τ) ^B×(H+1)× d. 2. Sinusoidal time encoding. The scalar τdisc _disc is first broadcast to all H+1H\!+\!1 steps, then encoded via a sinusoidal positional encoding: τ=SinPosEnc(τdisc)∈ℝB×(H+1)×d e_τ=SinPosEnc( _disc) ^B×(H+1)× d. 3. Fuse. Concatenate and process through a two-layer MLP: τ=Linear(d→d)(swish(Linear(2d→d)([a;τ])))∈ℝB×(H+1)×d. split A^τ&=Linear(d→ d)\! (swish\! (Linear(2d→ d)([ e_a;\, e_τ]) ) )\\ & ^B×(H+1)× d. split (14) A learned positional embedding (distinct from the sinusoidal one above) is optionally added to τ A^τ to distinguish the temporal position within the action horizon. Figure 11: Generalization evaluation under the Color setting. Target objects are replaced with variants of different shapes and colors, testing the model’s ability to generalize beyond the appearance of training objects. 7.5 Full Perceiver Attention Block We provide the complete forward computation of a single Perceiver-style attention block with the decoupled separation attention mechanism (Sec. 3.2.4). Let =[;]∈ℝB×(K+1)×d Z=[ S; Q]\!∈\!R^B×(K+1)× d denote the non-action latents, τ∈ℝB×(H+1)×d A^τ\!∈\!R^B×(H+1)× d the action latents, ∈ℝB×Nc×d c\!∈\!R^B× N_c× d the projected vision–language context, and t the flow time embedding. ¯ Z =LN(), =LN( Z), (15) ¯ A =AdaLN(τ,t), =AdaLN( A^τ,\,t), (16) ¯ c =LN(), =LN( c), (17) Z ←+MHA(¯,[¯;¯]), ← Z+MHA\! ( Z,\;[ Z;\, c] ), (18) τ A^τ ←τ+MHA(¯,[¯;¯;¯]), ← A^τ+MHA\! ( A,\;[ Z;\, A;\, c] ), (19) Z ←+FFN(LN()), ← Z+FFN\! (LN( Z) ), (20) τ A^τ ←τ+FFN(AdaLN(τ,t)), ← A^τ+FFN\! (AdaLN( A^τ,\,t) ), (21) where MHA(,)MHA( q, kv) denotes multi-head attention with q as queries and kv as both keys and values; FFNFFN denotes Linear(d→4d)→SquaredReLU→Linear(4d→d)Linear(d→ 4d) (4d→ d); and AdaLN(,t)=γ(t)⊙LN()+β(t)AdaLN( x,t)=γ(t) ( x)+β(t), with γ,βγ,β produced by a zero-initialized linear layer from SiLU(t)SiLU(t), ensuring that the block behaves as a standard LayerNorm at initialization. The key–value token counts are: |kv|=K+1+Nc=261|kv_ Z|=K\!+\!1\!+\!N_c=261 and |kv|=K+1+(H+1)+Nc=277|kv_ A|=K\!+\!1\!+\!(H\!+\!1)\!+\!N_c=277. 7.6 Evolver Internal Architecture The evolver GξG_ξ is a causal Vision Transformer predictor initialized from the pre-trained VJEPA2-AC predictor weights. Its forward computation for WAM-VJEPA proceeds as follows. Input assembly. The current-frame patch tokens t∈ℝB×P×dr r_t\!∈\!R^B× P× d_r (P=256P\!=\!256, dr=1408d_r\!=\!1408) are first projected into the predictor embedding dimension dout=1024d_out\!=\!1024: 0=Linear(dr→dout)(t)∈ℝB×P×dout. x_0=Linear(d_r→ d_out)( r_t) ^B× P× d_out. (22) The projected motion-semantic context ~∈ℝB×K×dout m\!∈\!R^B× K× d_out and state embedding ~∈ℝB×1×dout s\!∈\!R^B× 1× d_out are reshaped with a leading frame dimension T=1T\!=\!1: 0 x_0 ←0.view(B,T,P,dout), ← x_0.view(B,\,T,\,P,\,d_out), (23) ~ m ←~.view(B,T,K,dout), ← m.view(B,\,T,\,K,\,d_out), ~ s ←~.view(B,T, 1,dout). ← s.view(B,\,T,1,\,d_out). Token concatenation. Conditioning tokens are prepended to the patch tokens: =[~;~;0]∈ℝB×T(K+1+P)×dout=ℝB×321×1024, split x&=[\, m;\; s;\; x_0\,]\\ & ^B× T(K+1+P)× d_out\\ &=R^B× 321× 1024, split (24) where K+1+P=64+1+256=321K\!+\!1\!+\!P\!=\!64\!+\!1\!+\!256\!=\!321. Causal transformer blocks. The concatenated sequence is processed by Lp=24L_p\!=\!24 causal transformer blocks (ACBlock), each with a block-causal attention mask that respects both the frame ordering and the prefix position of the conditioning tokens. Within each block, conditioning tokens (indices 0 to K) can attend to all positions, while patch tokens (index K+1K\!+\!1 onwards) may only attend to preceding positions within the same frame and all conditioning tokens. Output projection. After the final block, the conditioning tokens are discarded and only the patch-token outputs are retained: out=[:,K+1:,:]∈ℝB×TP×dout=ℝB×256×1024. split x_out&= x[:,\,K+1:,\,:]\\ & ^B× TP× d_out\\ &=R^B× 256× 1024. split (25) A LayerNorm followed by a linear projection maps back to the encoder dimension: ^t+Δ=Linear(dout→dr)(LN(out))∈ℝB×256×1408. split r_t+ &=Linear(d_out→ d_r)\! (LN( x_out) )\\ & ^B× 256× 1408. split (26) Frozen action encoders. The original VJEPA2-AC predictor contains action encoders that map discrete actions to conditioning tokens. In WAM-VJEPA these encoders are bypassed (replaced by the projectors m,s W_m, W_s) and explicitly frozen to prevent unused-parameter drift during optimization. 7.7 Change-Weighted Future-Frame Loss The future-frame loss (Eq. 8 of the main paper) optionally applies a change-weighted mask that up-weights spatial regions where the future frame differs from the current observation. Change binary mask. For each spatial location (i,j)(i,j) in the latent space, we compute: ij=1if ‖ijfut−ijobs‖>δ,0otherwise, C_ij= cases1&if \| z^fut_ij- z^obs_ij\|>δ,\\ 0&otherwise, cases (27) where δ is a threshold determined from the ρ-th percentile of the per-pixel latent distances within each batch. Loss weight. The binary mask is softened with a linear ramp: ijchange=wmaxif ij=1,1if ij=0, W^change_ij= casesw_ &if C_ij=1,\\ 1&if C_ij=0, cases (28) and the final masked MSE loss becomes: ℒwm=∑ijtgt,ijijchange‖^ij−ij∗‖2∑ijtgt,ijijchange.L_wm= _ij M_tgt,ij\, W^change_ij\,\| v_ij- v^*_ij\|^2 _ij M_tgt,ij\, W^change_ij. (29) When masked_denoising is enabled, the noised input is further replaced by the clean latent at statically-unchanged positions, so the model only denoises regions that actually change. 7.8 Flow Time Sampling Action branch. The flow time τ for the action OT-CFM objective is sampled from a shifted Beta distribution: τ=s−xs,x∼Beta(α,β),s=0.999,τ= s-xs, x (α,β), s=0.999, (30) with default α=1.5,β=1.0α\!=\!1.5,β\!=\!1.0, which concentrates mass near τ=1τ\!=\!1 (clean actions) and thus emphasizes the final denoising stages. World-model branch. The generation pipeline samples τ′τ from a Logit-Normal distribution with configurable (μ,σ2)(μ,σ^2), followed by a time-shift: τ′ τ =σu+μ1+|σu+μ|,u∼(0,1), = σ\,u+μ1+|σ\,u+μ|, u (0,1), (31) tshift t_shift =s′τ′1+(s′−1)τ′, = s \,τ 1+(s -1)\,τ , with shift factor s′=5.0s \!=\!5.0 and tshiftt_shift discretized via ⌊tshift×1000⌋ t_shift× 1000 for the timestep embedding. 7.9 Architecture Hyper-parameters Table 7 summarizes the key architectural hyper-parameters of WAM-VJEPA. Component Parameter Value Video DiT Context tokens NcN_c 196 Context dimension dcd_c 2048 Num. layers (enc / gen) 28 / 28 Action DiT Internal dimension d 768 Output dimension doutd_out 1024 Num. Perceiver layers L 16 Num. attention heads 16 Num. query tokens K 64 FFN multiplier 4×4× (SquaredReLU) State / Action State dimension dsd_s 64 Action dimension dad_a 32 Action horizon H+1H\!+\!1 16 VJEPA2-AC Patch tokens P 256 Representation dim drd_r 1408 Predictor layers LpL_p 24 Flow Matching Timestep buckets 1000 WM inference steps 20 Action inference steps 20 Action repeats R 4 Table 7: Architecture hyper-parameters. 7.10 Model Architecture Details World Model Backbone. We adopt Wan2.2-TI2V-5B as the world model backbone, a 5-billion-parameter video DiT that serves a dual role: (i) as a comprehension encoder that maps the current observation oto_t and language instruction ℓ into context tokens ∈ℝB×196×2048 c\!∈\!R^B× 196× 2048, and (i) as a generation decoder that denoises future-frame latents via a flow-matching velocity objective. The encoder extracts hidden states from the last layer (extract_layers=[-1]) of the DiT backbone. Input images are resized to 224×224224\!×\!224 for both LIBERO and RoboCasa benchmarks, with a conditional-frame timestep of 0.10.1. All the components in the backbone are trainable. The future-frame generation branch operates in the latent space of the accompanying VAE encoder. Given a current observation oto_t and a future frame ot+Δo_t+ , the VAE encodes the concatenated frame pair into ∗∈ℝB×2×16×28×28 z^*\!∈\!R^B× 2× 16× 28× 28. A MaskFutureImage strategy then adds noise only to the future latent channels, preserving the current-frame channels as conditioning. The Video DiT generation pipeline predicts the velocity field ^θ v_θ and is trained with a change-weighted MSE loss (Sec. 7.7). Flow time τ′τ is sampled from a Logit-Normal distribution with shift factor s′=5.0s \!=\!5.0 (Sec. 7.8). At inference, the generation pipeline uses 20 denoising steps with classifier-free guidance scale 5.05.0, producing a 224×224224\!×\!224 future image for both benchmarks. Representation Simulation. The VJEPA2-AC encoder EψE_ψ (ViT-G, ∼ 1B parameters) is initialized from vjepa2-ac-vitg.pt and remains fully frozen. It produces patch-level representations t,t+Δ∈ℝB×256×1408 r_t, r_t+ \!∈\!R^B× 256× 1408 for the current and future frames. The evolver GξG_ξ (24-layer causal transformer, dout=1024d_out\!=\!1024) takes the projected motion-semantic context ~ m and state embedding ~ s as conditioning tokens, concatenated with t r_t to form a 321-token sequence (Eq. 24). The two projectors m W_m and s W_s are linear layers that map from d=768d\!=\!768 and ds=64d_s\!=\!64 respectively into dout=1024d_out\!=\!1024. The representation-simulation loss uses Smooth-ℓ1 _1 with λrs=1.0 _rs\!=\!1.0. Action Head. The Action Head is a 16-layer Perceiver-style DiT (denoted DiT-B) with internal dimension d=768d\!=\!768 and 16 attention heads. It consumes 81 latent tokens: 1 state token (MLP:64→768MLP:64\!→\!768), 64 future-query tokens (n.Embedding(64,768) n.Embedding(64,768), initialized from (0,0.02)N(0,0.02)), and 16 action tokens produced by the ActionEncoder (Sec. 7.4). Cross-attention is performed against the 196 vision–language context tokens c with cross-attention dimension 2048. The decoupled separation attention mechanism (Sec. 7.5) ensures that action noise does not contaminate the query slots. Flow time is injected exclusively into the action-token group via AdaLN with zero-initialized modulation parameters. The action decoder is a two-layer MLP (1024→1024→321024\!→\!1024\!→\!32) that maps the final action-token outputs to the predicted velocity in ℝ32R^32. During training, each example is replicated R=4R\!=\!4 times with independent noise samples; the 64 query-token outputs are averaged across replicas to form the deterministic motion-semantic context m. At inference, the action trajectory is recovered via Euler integration of the learned velocity field (Algorithm 2). 7.11 Action DiT Flow-Matching Formulation We provide a detailed derivation of the Action DiT flow-matching objective used in PILOT. Let 1:H a_1:H denote an action trajectory of horizon H and let 0∼(,) a^0\! \!N( 0, I) be the noise source. We adopt the Optimal-Transport Conditional Flow Matching (OT-CFM) [53] formulation, which constructs a conditional probability path and vector field that transport 0 a^0 to 1:H a_1:H. Probability path and vector field. Given a data sample 1:H a_1:H and noise 0 a^0, the conditional flow is the linear interpolation τ=(1−τ)0+τ1:H,τ∈[0,1]. a^τ=(1-τ)\, a^0+τ\, a_1:H, τ∈[0,1]. (32) The corresponding conditional vector field is (τ∣0,1:H)=dττ=1:H−0. u( a^τ a^0, a_1:H)= ddτ a^τ= a_1:H- a^0. (33) Under the OT-CFM framework, the marginal vector field that generates the marginal probability path pτ(τ)p_τ( a^τ) is τ(τ)=0,1:H∣τ[1:H−0]. v_τ( a^τ)=E_ a^0, a_1:H a^τ\! [ a_1:H- a^0 ]. (34) Training objective. The Action DiT ϕ f_φ is trained to regress the conditional vector field via ℒact=τ,0,1:H[‖ϕ(τ,τ,,~,~)−(1:H−0)‖2],L_act=E_τ, a^0, a_1:H [ f_φ( a^τ,τ, c, m, s)-( a_1:H- a^0) ^2 ], (35) where c are the vision–language context tokens, ~ m the projected motion-semantic context, and ~ s the projected robot-state embedding. Flow time sampling. The flow time τ is sampled from a Logit-Normal distribution LogitNormal(μ=0,σ=1)LogitNormal(μ\!=\!0,σ\!=\!1) with a learnable shift s=3.0s\!=\!3.0 for the action branch (different from the generation branch’s s′=5.0s \!=\!5.0), ensuring more samples near τ=1τ\!=\!1 for accurate endpoint prediction (Sec. 7.8). Classifier-free guidance. During inference, we apply classifier-free guidance with scale w: ^τ=(1+w)ϕ(τ,τ,,~,~)−wϕ(τ,τ,∅,~,~), v_τ=(1+w)\, f_φ( a^τ,τ, c, m, s)-w\, f_φ( a^τ,τ, , m, s), (36) where ∅ denotes a null language embedding. We use w=1.5w\!=\!1.5 by default and 20 Euler integration steps. Action chunking and horizon. PILOT predicts an action chunk of horizon H=50H\!=\!50 for LIBERO and H=100H\!=\!100 for RoboCasa, but executes only the first HaH_a actions before re-planning. Following prior work [71], we use Ha=8H_a\!=\!8 for LIBERO and Ha=50H_a\!=\!50 for RoboCasa. The temporal ensemble weighting is applied over overlapping predictions. Separation attention in action DiT. A key architectural detail is the decoupled separation attention (Sec. 7.5): the 81 tokens inside the action DiT are partitioned into state + query (65 tokens) and action (16 tokens) groups. Self-attention is performed within each group independently; cross-attention against c is shared across both groups. This ensures that the noisy action tokens at flow time τ cannot leak information into the query slots that must remain deterministic, which is critical for the quality of the motion-semantic context m. Loss weighting. The total training loss combines three objectives: ℒ=λactℒact+λgenℒgen+λrsℒrs,L= _act\,L_act+ _gen\,L_gen+ _rs\,L_rs, (37) where λact=1.0 _act\!=\!1.0, λgen=1.0 _gen\!=\!1.0, λrs=1.0 _rs\!=\!1.0. All three losses are averaged over the batch dimension; ℒgenL_gen further includes the change-weighted mask (Sec. 7.7). 7.12 Training Details Data preprocessing. For both LIBERO and RoboCasa benchmarks, observations are resized to 224×224224\!×\!224 pixels and normalized using the ImageNet mean and standard deviation. We adopt view_mode=primary for both benchmarks, extracting only the primary camera view as input. The video backends differ: LIBERO uses torchvision_av while RoboCasa uses decord, reflecting the different episode storage formats of each benchmark. The future-frame index is set to Δ=16 \!=\!16 timesteps ahead. The robot proprioceptive state vector has dimension 8 for LIBERO and 64 for RoboCasa, and the action dimension is 7 and 32 respectively. Optimizer and learning rate. We use the AdamW optimizer with β1=0.9 _1\!=\!0.9, β2=0.95 _2\!=\!0.95, weight decay 0.10.1, and an initial learning rate schedule with linear warmup followed by cosine decay to a minimum learning rate. For LIBERO, the peak learning rates are lrbase=1×10−5lr_base\!=\!1\!×\!10^-5, lraction=1×10−4lr_action\!=\!1\!×\!10^-4, lrpredictor=1×10−4lr_predictor\!=\!1\!×\!10^-4, lrprojectors=1×10−4lr_projectors\!=\!1\!×\!10^-4, with warmup for 1,000 steps and min_lr=1×10−6min\_lr\!=\!1\!×\!10^-6. For RoboCasa, all learnable components share a unified learning rate lr=1×10−5lr\!=\!1\!×\!10^-5 (base, action model, predictor, projectors), with warmup for 500 steps and min_lr=1×10−7min\_lr\!=\!1\!×\!10^-7. Batch size and hardware. All models are trained on 32 NVIDIA H20 GPUs using DeepSpeed ZeRO Stage 2. For LIBERO, the per-device batch size is 16, yielding an effective batch size of 512. For RoboCasa, the per-device batch size is 8, yielding an effective batch size of 256. For LIBERO, we trained for a total of 100,000 steps (taking approximately 90 hours), while for Robocasa, we trained for a total of 300,000 steps (taking approximately 200 hours). Training hyper-parameters. Table 8 summarizes the key training hyper-parameters for both benchmarks. Category Hyper-parameter LIBERO RoboCasa Real-World Data Resolution 224×224224\!×\!224 224×224224\!×\!224 224×224224\!×\!224 View mode primary primary 3-view Video backend torchvision_av decord decord Future-frame Δ 16 16 16 Dimensions Action dim 7 32 32 State dim 8 64 64 Horizon H 50 100 30 Optimization Optimizer AdamW (β1=0.9 _1\!=\!0.9, β2=0.95 _2\!=\!0.95) AdamW (β1=0.9 _1\!=\!0.9, β2=0.999 _2\!=\!0.999) Weight decay 0.1 Training steps 100,000 300,000 100,000 Warmup steps 1,000 3,000 1,000 Min. learning rate 1×10−61\!×\!10^-6 1×10−71\!×\!10^-7 1×10−71\!×\!10^-7 Gradient clipping 1.0 Learning Rate lrbaselr_base 1×10−51\!×\!10^-5 1×10−51\!×\!10^-5 1×10−51\!×\!10^-5 lractionlr_action 1×10−41\!×\!10^-4 5×10−55\!×\!10^-5 5×10−55\!×\!10^-5 lrpredictorlr_predictor 1×10−41\!×\!10^-4 5×10−55\!×\!10^-5 5×10−55\!×\!10^-5 lrprojectorslr_projectors 1×10−41\!×\!10^-4 5×10−55\!×\!10^-5 5×10−55\!×\!10^-5 Hardware & Batch GPUs 32× NVIDIA H20 Per-device BS 16 8 16 Effective BS 512 256 512 Precision BF16 mixed Flow Matching WM inference steps 20 Action inference steps 20 CFG scale 1.5 5.0 Action repeats R 4 Checkpointing Save interval Every 10,000 steps Table 8: Training hyper-parameters for PILOT. Checkpointing and evaluation. We save training checkpoints every 10,000 steps and evaluate the policy on the benchmark tasks at each checkpoint. The final reported results correspond to the checkpoint with the highest average success rate across all tasks within each benchmark. 7.13 Additional Experimental Details LIBERO benchmark. LIBERO [34] provides 130 manipulation tasks across 5 task suites built on the Franka Panda robot in the MuJoCo simulator. We evaluate on LIBERO-Spatial (10 tasks), LIBERO-Object (10 tasks), LIBERO-Goal (10 tasks), and LIBERO-Long (10 tasks) for a total of 40 tasks, following the standard protocol. Each task is evaluated over 20 episodes, and the success rate is averaged across all tasks. The action space is 7-dimensional (E pose + gripper), and the proprioceptive state is 8-dimensional. Observations consist of a primary wrist-camera RGB image at 224×224224\!×\!224 resolution. RoboCasa-GR1 benchmark. RoboCasa [4] is a large-scale simulation platform for household manipulation tasks featuring the GR1 humanoid robot with 29 degrees of freedom. We use the RoboCasa-GR1 Tabletop Task suite [4] which provides 24 manipulation tasks across 5 categories (Pick, Stack, Pour, Navigate, and Composite). Each task is evaluated over 20 episodes. The action space is 32-dimensional (joint positions + gripper), and the proprioceptive state is 64-dimensional. Observations consist of a primary third-person RGB image at 224×224224\!×\!224 resolution. Baselines. We compare PILOT against the following baselines, organized by model category: • Vision-Language-Action Models. π0 _0 [5] and π0.5 _0.5 [21] are VLM-based flow-matching policies built on PaliGemma with internet-scale pre-training. GR00T-N1 [4] is a VLA baseline built on the SmolVLM backbone, and GR00T-N1.6 [44] is its scaled successor. OpenVLA [26] and OpenVLA-OFT [24] are open-source VLAs fine-tuned with optimal transport. ECoT [67] incorporates explicit chain-of-thought reasoning for manipulation. Spatial Forcing [27] augments VLA training with spatial-reasoning regularization. π0 _0-FAST [46] is a fast-distilled variant of π0 _0. • Latent Action Models. LAPA [65] pre-trains latent action tokens from large-scale human videos. UniVLA [8] unifies visual-action representations via a shared codebook. Mantis [62] aligns latent actions with multi-modal visual features. VLA-JEPA [51] introduces a JEPA component to address pixel reconstruction bias and future information leakage. PokeVLA [72] leverages poke-style interaction priors for manipulation. • World-Action Models. Fast-WAM [66] accelerates world-action inference via a parallel prediction head. Motus [3] jointly models visual and action tokens through shared attention (MoT). LDA [39] learns latent dynamics alignment for world-action prediction. F1 [38] is a world-action model with a factorized prediction architecture. WorldVLA [10] integrates world-model predictions into VLA generation. Cosmos-Policy [25] is a Cosmos-based video prediction policy. DiT4DiT [40] employs a DiT-for-DiT architecture for world-action modeling. UWM-XL [74] is a universal world model at XL scale. StarVLA [63] is a star-topology VLA with distributed attention. VP-VLA [58] incorporates video prediction into VLA training. PhysBrain [33] embeds physical reasoning into world-action generation. LangForce [32] leverages language-guided force estimation for manipulation. Evaluation protocol. For both benchmarks, we evaluate each method by running 20 rollouts per task from the initial state distribution. Success is determined by the task-specific completion criteria. A task is considered successful if the goal condition is met within the maximum episode length (500 steps for LIBERO, 1000 steps for RoboCasa). We report the average success rate across all tasks within each suite. Real-World benchmark. We deploy PILOT on an Agibot-G1 dual-arm humanoid robot to evaluate its performance on real-world desk manipulation tasks. The robot features 7-DOF arms on each side, a 1-DOF gripper per hand, a 2-DOF head (pan/tilt), and a 2-DOF waist, yielding a 32-dimensional action space and a 64-dimensional proprioceptive state. Observations are captured from three RGB cameras at 224×224224\!×\!224 resolution: a head-mounted camera (image_head_color), a left-hand wrist camera (image_hand_left), and a right-hand wrist camera (image_hand_right), providing complementary global and egocentric viewpoints. Figure 9 shows the Agibot-G1 robot, the teleoperation-based data collection setup, the standard evaluation scene, and two representative generalization scenarios. The demonstration data is collected via teleoperation. The desk_all_3view data mixture comprises 8 sub-datasets collected under consistent desk-top conditions, each corresponding to a manipulation task with objects commonly found on office desks. In total, the mixture aggregates 32 source recording sessions, yielding 3,965 demonstration trajectories per sub-task and a combined total of 7.1M frames across all tasks. Visual observations are recorded by two RealSense D405 cameras mounted on the left and right wrists (480×848480\!×\!848 native resolution) and one head-mounted camera (800×1280800\!×\!1280), all operating at 30 fps. The data is stored in the LeRobot format with min-max normalization applied to all state and action keys. We use the following 8 standard tasks for evaluation: pen, eraser, correction fluid, pencil case, stapler, book, trash, and charger. Figure 12 illustrates PILOT deployed on the Agibot-G1 robot performing the 8 standard tasks, ranging from precise manipulation (e.g., inserting a pen into a grid slot) to grasping specific objects on the desk, with the most challenging task being picking up a book and placing it upright into a storage rack. Figure 12: PILOT deployed on the Agibot-G1 robot performing 8 standard desk manipulation tasks. The tasks include inserting a pen into a grid slot (requiring high precision), grasping and placing various desktop objects, and the most challenging task of picking up a book and placing it upright into a storage rack. Table 9 summarizes the dataset statistics for each task. Task Directory Trajectory Length Frames Frequency Pen g1_pen 3,965 248 983,400 30 Eraser g1_eraser 3,965 158 625,110 30 Correction fluid g1_correction 3,965 220 872,097 30 Pencil case g1_pencil 3,965 260 1,030,137 30 Stapler g1_stapler 3,965 210 830,546 30 Book g1_books 3,965 418 1,657,390 30 Trash g1_trash 3,965 136 537,195 30 Charger g1_charger 3,965 151 599,312 30 Total — — — 7,135,187 — Table 9: Real-world desk manipulation dataset statistics. Traj. = number of demonstration trajectories; Len. = average timesteps per trajectory; Frames = total frames; Freq. = control frequency (Hz). All tasks share the same 3,965 trajectories from 32 source sessions. Evaluation settings. We evaluate under three settings following the main text: (1) Standard: 50 rollouts per task with the default initial state distribution; (2) Generalization: three variants—Flash + Camera Offset, which introduces sudden lighting changes and a shifted camera position; Desk, which replaces the desk surface and background; and Color, which replaces object colors with unseen variants; Figures 8, 10, and 11 visualize the three generalization transfer scenarios, respectively. (3) Few-Shot (10%): only 10% of the demonstration data is used for training, controlled by the fewshot_ratio=0.1 flag in the training script. Ablation configurations. We conduct four sets of ablation experiments corresponding to different training configurations: • PILOT (Full): Frozen mode wm_train_vlm_frozen with the full training set—the default setting that trains both the World Model (WM) branch and action modules while keeping the VLM backbone frozen. • PILOT (Few-Shot): Same frozen mode as Full but with fewshot_ratio=0.1, using only 10% of the demonstration data. • RD-only: Frozen mode wm_frozen_vlm_frozen with future_image_training.enabled=false, which disables the future-image generation branch entirely, retaining only the Representational Deduction (RD) pathway. • RD-only (Few-Shot): Same as RD-only but with fewshot_ratio=0.1. These configurations allow us to isolate the contribution of the WM generation branch versus the RD pathway, as well as the data efficiency of each component. 7.14 Inference Efficiency A fundamental architectural question is whether generation and understanding should be interleaved sequentially (predict-then-act) or decoupled into parallel cooperative branches. In the predict-then-act paradigm, the Wan2.2 backbone operates in its generation mode: the Transformer is invoked 50 times within an iterative denoising loop (each pass predicting a velocity field that a scheduler uses to refine the noisy latent), followed by a VAE decoding step to produce a future pixel frame, before the Action Head can begin inference (see Algorithm 4). Our decoupled design instead uses the Wan2.2 backbone in its understanding mode: the Transformer performs a single forward pass that directly extracts vision–language context tokens, bypassing the 50-step denoising loop and VAE decoder entirely (Algorithm 3). The understanding pipeline thus eliminates three costly components that the generation pipeline requires: (1) the iterative denoising scheduler loop (50×50× Transformer forwards), (2) the VAE decoding step, and (3) the per-iteration attention mask and scheduler state management. Table 6 compares these two strategies on RoboCasa. The two models achieve nearly identical performance (62.6% vs. 58.4%), confirming that decoupling does not sacrifice task accuracy. However, the inference cost differs drastically: the decoupled pipeline runs at 111.60 ms per step (8.26 Hz), while predict-then-act requires 1329.95 ms (0.75 Hz)—an 11.9× slowdown. This gap is primarily attributable to the 50×50× Transformer forward-pass ratio between the two pipelines; scheduler overhead and VAE decoding further amplify the difference beyond the raw compute ratio. These results demonstrate that the Representational Deduction mechanism achieves the representational benefits of future prediction (provided by the CDE during training) without paying the sequential inference penalty of pixel-level generation. PILOT achieves 6.9 ms per action step (145 Hz raw frequency), which is 2.1× faster than π0 _0 (14.5 ms) and 1.7× faster than CogACT (11.8 ms). The effective control frequency, accounting for the fact that PILOT predicts actions over a horizon (16 steps for RoboCasa), reaches 58 Hz—well above the 20 Hz threshold for real-time closed-loop control. This efficiency stems from two design choices: (1) the Causal Dynamics Engine is a lightweight transformer with only 4 layers, adding minimal overhead; and (2) the flow-matching decoder uses only 20 denoising steps, reducing the action generation cost. 7.15 Full RoboCasa-GR1 Results Task GR00T N1.5† GR00T N1.6† DiT4DiT UWM-XL StarVLA LDA VP-VLA PhysBrain LangForce FastWAM Motus PILOT PnP Close PnP Bottle to Cab. 54 51.5 48 41 46 76 54 74 72 58 55 70 PnP Can to Drawer 50 13 74 53 80 71 72 68 78 60 57 54 PnP Cup to Drawer 38 8.5 52 12 54 41 44 42 46 45 42 50 PnP Milk to Micro. 60 14 50 25 48 52 74 54 56 54 50 68 PnP Potato to Micro. 32 41.5 36 29 28 41 34 24 36 43 40 62 PnP Wine to Cab. 38 16.5 42 24 46 57 48 54 46 56 52 54 PnP * to * Close (Avg) 45.3 24.2 50.3 30.7 50.3 56.3 54.3 52.7 55.7 52.7 49.3 59.7 PnP Novel From Cuttingboard NvCB to Basket 38 58 52 18 48 65 66 62 66 64 60 84 NvCB to CbBox 46 46.5 48 14 40 69 54 44 40 58 54 58 NvCB to Pan 58 68.5 76 20 68 75 74 56 68 72 68 86 NvCB to Pot 62 65 62 25 52 61 54 58 48 59 55 44 NvCB to TBasket 28 46.5 50 10 56 51 56 40 44 50 46 34 Cuttingboard * (Avg) 46.4 56.9 57.6 17.4 52.8 64.2 60.8 52 53.2 60.6 56.6 61.2 PnP Novel From Placemat NvPM to Basket 30 58.5 50 16 42 53 48 42 54 55 51 72 NvPM to Bowl 60 57.5 56 10 44 55 74 56 62 58 54 58 NvPM to Plate 56 63 32 12 48 59 70 80 52 54 50 64 NvPM to TShelf 26 28.5 18 2 18 24 26 14 24 22 20 30 Placemat * (Avg) 45.5 51.9 39 10 38 47.8 54.5 48 48 47.2 43.8 56 PnP Novel From Tray NvTR to CbBox 52 51.5 38 25 38 65 44 40 50 62 58 86 NvTR to Plate 48 71 56 18 56 63 66 66 58 64 60 94 NvTR to Pot 60 64.5 54 25 50 55 38 52 62 60 56 92 NvTR to TBasket 52 57 46 16 36 51 58 50 44 52 48 80 NvTR to TShelf 32 31.5 38 2 16 33 24 22 22 36 32 22 Tray * (Avg) 48.8 55.1 46.4 17.2 39.2 53.4 46 46 47.2 54.8 50.8 74.8 PnP Novel From Plate NvPL to Bowl 58 57 56 8 60 53 52 54 54 56 52 52 NvPL to CbBox 44 43.5 58 10 50 43 44 50 48 50 46 52 NvPL to Pan 60 51 68 20 54 55 56 68 54 60 56 50 NvPL to Plate 64 78.7 58 27 70 61 62 78 78 68 64 88 Plate * (Avg) 56.5 57.5 60 16.2 58.5 53 53.5 62.5 58.5 58.5 54.5 60.5 Average (All 24 Tasks) 48.2 47.6 50.8 19.2 47.8 55.4 53.8 50 52.6 56.7 53.1 62.6 Abbreviations: PnP = Pick and Place; Nv = Novel; CB = Cuttingboard; CbBox = Cardboardbox; PM = Placemat; TR = Tray; PL = Plate; TBasket = Tieredbasket; TShelf = Tieredshelf; Cab. = Cabinet; Micro. = Microwave. Table 10: Full evaluation on RoboCasa-GR1 [4]. Success rates (%) across all 24 manipulation tasks with the GR1 humanoid robot (29-DoF actions, ego-centric view, 1000 demos/task, 50 rollouts). † denotes methods using internet-scale pre-training. Dark gray and light gray backgrounds indicate the best and second-best results in each row, respectively. Table 10 presents the per-category results on the RoboCasa-GR1 Tabletop Task benchmark. PILOT achieves the highest average success rate across all categories, with particularly large improvements on the Composite and Pour categories that require multi-step reasoning and precise control.