Paper deep dive
LatentPilot: Scene-Aware Vision-and-Language Navigation by Dreaming Ahead with Latent Visual Reasoning
Haihong Hao, Lei Chen, Mingfei Han, Changlin Li, Dong An, Yuqiang Yang, Zhihui Li, Xiaojun Chang
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 4/1/2026, 1:31:08 AM
Summary
LatentPilot is a vision-and-language navigation (VLN) framework that internalizes future-aware reasoning by learning action-conditioned visual dynamics. It uses a 'Pilot Token'—a step-propagated latent state—to enable the agent to 'dream ahead' about future observations during training, while maintaining strictly causal inference without requiring future frames or external world models.
Entities (5)
Relation Signals (3)
LatentPilot → achievedsotaon → R2R-CE
confidence 99% · Experiments on R2R-CE, RxR-CE, and R2R-PE benchmarks achieve new SOTA results
LatentPilot → utilizes → Pilot Token
confidence 98% · LatentPilot is an end-to-end navigator composed of three components: a vision encoder E φ , a lightweight Pilot module G ψ , and a LLM decision backbone F θ . The core idea is to maintain a step-propagated continuous latent state z t ∈ R d , termed the Pilot Token.
PilotLoop → optimizes → LatentPilot
confidence 95% · we further adopt a flywheel-style learning loop called PilotLoop that repeatedly collects rollouts and updates the model
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Existing vision-and-language navigation (VLN) models primarily reason over past and current visual observations, while largely ignoring the future visual dynamics induced by actions. As a result, they often lack an effective understanding of the causal relationship between actions and how the visual world changes, limiting robust decision-making. Humans, in contrast, can imagine the near future by leveraging action-dynamics causality, which improves both environmental understanding and navigation choices. Inspired by this capability, we propose LatentPilot, a new paradigm that exploits future observations during training as a valuable data source to learn action-conditioned visual dynamics, while requiring no access to future frames at inference. Concretely, we propose a flywheel-style training mechanism that iteratively collects on-policy trajectories and retrains the model to better match the agent's behavior distribution, with an expert takeover triggered when the agent deviates excessively. LatentPilot further learns visual latent tokens without explicit supervision; these latent tokens attend globally in a continuous latent space and are carried across steps, serving as both the current output and the next input, thereby enabling the agent to dream ahead and reason about how actions will affect subsequent observations. Experiments on R2R-CE, RxR-CE, and R2R-PE benchmarks achieve new SOTA results, and real-robot tests across diverse environments demonstrate LatentPilot's superior understanding of environment-action dynamics in scene. Project page:this https URL
Tags
Links
- Source: https://arxiv.org/abs/2603.29165v1
- Canonical: https://arxiv.org/abs/2603.29165v1
Trouble viewing inline? Open PDF directly →
Full Text
72,648 characters extracted from source content.
Expand or collapse full text
LatentPilot: Scene-Aware Vision-and-Language Navigation by Dreaming Ahead with Latent Visual Reasoning Haihong Hao 1* , Lei Chen 1* , Mingfei Han 2* , Changlin Li 3 , Dong An 4 , Yuqiang Yang 5 , Zhihui Li 1 , and Xiaojun Chang 1 1 University of Science and Technology of China 2 MBZUAI 3 Stanford University 4 Amap, Alibaba Group 5 Shanghai AI Laboratory * Equal contribution. Project Page: https://abdd.top/latentpilot/ Abstract. Existing vision-and-language navigation (VLN) models pri- marily reason over past and current visual observations, while largely ignoring the future visual dynamics induced by actions. As a result, they often lack an effective understanding of the causal relationship be- tween actions and how the visual world changes, limiting robust decision- making. Humans, in contrast, can “imagine” the near future by leveraging action–dynamics causality, which improves both environmental under- standing and navigation choices. Inspired by this capability, we propose LatentPilot, a new paradigm that exploits future observations during training as a valuable data source to learn action-conditioned visual dy- namics, while requiring no access to future frames at inference. Con- cretely, we propose a flywheel-style training mechanism that iteratively collects on-policy trajectories and retrains the model to better match the agent’s behavior distribution, with an expert takeover triggered when the agent deviates excessively. LatentPilot further learns visual latent tokens without explicit supervision; these latent tokens attend globally in a continuous latent space and are carried across steps, serving as both the current output and the next input, which enabling the agent to “dream ahead” and reason about how actions will affect subsequent observations. Experiments on R2R-CE, RxR-CE, and R2R-PE bench- marks achieve new SOTA results, and real-robot tests across diverse environments demonstrate LatentPilot’s superior understanding of envi- ronment–action dynamics in scene. Keywords: Vision-and-Language Navigation· Vision-Language Models · Latent Visual Reasoning 1 Introduction Vision-and-Language Navigation (VLN) requires an embodied agent to move through a 3D environment by following natural-language instructions and reach- arXiv:2603.29165v1 [cs.CV] 31 Mar 2026 2 Previous VLNHuman NavigationLatentPilot Past Current Future Unused Supervision Instruction:Go down the hallway. Turn left into the last bedroom. Wait beside the bed. Myopic Decision! Turn left Experience ... Forward Anticipates Consequences! Knowledge Humans rarely decide based solely on what’s currently visible. Uses future information but keeps causal constraint. No separate predictors needed! Dreaming Ahead Forward PilotLoop Pilot Token Future Mental Simulation Fig. 1: Prior VLN pipelines optimize next-action prediction from past/current ob- servations, leaving future observations in trajectories largely unused as supervision. LatentPilot uses future observations only as training-time privileged supervision to internalize action-conditioned visual dynamics while inference remains strictly causal and requires neither future frames nor an external world-model rollout. ing a target location [3, 25, 26]. Solving VLN demands more than aligning lan- guage with the current visual observation. The agent must accumulate spatial memory, interpret scene semantics, and correct itself under uncertainty during sequential decision making. Crucially, VLN is not a static “perceive then act” problem. It is an interactive process in which every action changes what will be observed next. Therefore, robust navigation is tightly linked to the ability to understand the environment and to anticipate how it will evolve under the agent’s actions, namely an ability to imagine near-future observations. Most existing VLN approaches formulate navigation as conditional policy learning [3,25]. Given an instruction and past or current observations, the model predicts the next action, typically trained via imitation learning on expert tra- jectories [40] that fit “what to do now.” As shown in Fig. 1, although training trajectories inherently contain rich future frames that reflect the consequences of actions, mainstream training pipelines usually treat them only as later ob- servations in a sequence. They are seldom used as explicit supervision to shape representations that are predictive and anticipatory at the current step. A com- plementary line of work equips agents with explicit lookahead via world models or imagination modules that predict or synthesize future observations and then plan, rerank, or prompt decisions accordingly (e.g., Pathdreamer [22], future- view generation objectives [28], diffusion-based imagination for VLN [17, 18], and scene-imagination prompting pipelines [35,68], as well as world-model-based VLN-CE frameworks [58]). While effective, these approaches typically external- ize imagination as separate predictors, planners, or sampling procedures, intro- ducing additional computation and exposing the agent to compounding predic- tion errors and policy–model mismatch. In contrast, we argue that it is beneficial to internalize imagination: distilling action–observation causality into the nav- 3 igator itself so that anticipatory reasoning becomes an amortized, end-to-end capability of the same backbone that makes decisions. If decisions depend only on past and current observations, policies can become myopic, behaving in a “step and see” manner, and may commit to erroneous branches that are difficult to recover from in complex layouts. In contrast, if a model can anticipate how a candidate action will shape near-future observations, it becomes more likely to avoid inefficient exploration, reduce collisions and backtracking, and improve overall stability.Although future observations are not available at decision time, they are fully recorded in offline training trajectories. This raises a central question: can we use these recorded future observations during training to learn how actions change future visual inputs, while keeping inference strictly causal? When moving in the real world, people rarely make decisions purely on what is visible at the moment. Instead, they perform quick mental simulation before acting, for example, whether turning left will reveal a corridor or a doorway, or whether moving forward will lead to an open area or a dead end. This abil- ity is not innate. It is acquired through experience and reflection on the true consequences of past actions, gradually forming an internal forward model that supports pre-action simulation. By comparison, many VLN methods are effective at extracting cues from past and current observations, yet they rarely model the causal relation between actions, environment change explicitly. Fig. 1 suggests a more natural direction. During training, trajectories already include what the agent actually sees after taking actions. If these future observations are used as supervision, the “hindsight” consequences can be converted into “foresight” representations, allowing the model to learn more anticipatory decision making without requiring access to any future images at test time. Building on this insight, we propose Latent Visual Reasoning–based Dream- ing ahead Policy for Scene-Aware Vision-and-Language Navigation, LatentPi- lot. It is an end-to-end VLM navigator that maintains a step-propagated visual latent, termed the Pilot Token, to carry compact near-future imagination across decisions. Unlike explicit imagination/world-model pipelines that rely on sep- arate predictors or rollouts at inference [17, 18, 58, 68], LatentPilot amortizes lookahead into the same decision backbone via the step-propagated Pilot To- ken. The key observation is that while future observations are unavailable before an action is executed at test time, they are naturally recorded in pre-collected navigation trajectories. We therefore treat these future views as training-only privileged supervision, encouraging the model to produce Pilot Tokens that are predictive of the visual evidence it will encounter after acting. Importantly, La- tentPilot remains strictly causal at inference: it never accesses future frames, and instead relies on the current observation, the instruction, and the propagated Pi- lot Token for decision making. To reduce the mismatch between offline training and interactive deployment, we further adopt a flywheel-style learning loop called PilotLoop that repeatedly collects rollouts and updates the model [40]. To summarize, our contributions are as follows: 4 – LatentPilot framework: We introduce an end-to-end LMM-based navigator with a propagated visual latent (Pilot Token) that internalizes future-aware reasoning within the decision backbone, without requiring any external imag- ination module, world-model rollout, or planning procedure at inference. – We introduce a training-only privileged supervision objective that leverages future observations in trajectories to learn action-conditioned visual dynam- ics in latent space, effectively converting trajectory hindsight into test-time foresight while keeping inference strictly causal. It does not introduce future information leakage at test time. – We demonstrate consistent gains in both simulator benchmarks like R2R-CE, RxR-CE, and R2R-PE, and real-robot settings, indicating strong generaliza- tion and practical potential. 2 Related Work Vision-and-Language Navigation (VLN) studies embodied agents that navi- gate in 3D environments by following natural-language instructions and reaching a target location. Early benchmarks such as Room-to-Room (R2R) are built on the discretized viewpoints and navigation graphs of Matterport3D [3,5]. RxR ex- tends this setting with multilingual instructions and denser annotations, further increasing coverage and difficulty [26]. VLN-CE moves evaluation to continu- ous environments, making the task more deployment-oriented [25]. Large-scale indoor 3D datasets such as HM3D substantially increase scene diversity for train- ing and evaluation [36]. ScaleVLN synthesizes large-scale instruction–trajectory pairs via data generation [52, 56]. VLN-PE systematically characterizes per- formance degradation caused by physical and visual disparities and provides a more realistic evaluation platform across different robot embodiments [49]. ETPNav performs online topological mapping and separates long-horizon plan- ning from low-level control to improve executability in continuous environments [2]. BEVBert advances multimodal pretraining from a map-centric perspective, strengthening spatially aware representations for navigation [1]. NaVid formu- lates continuous navigation as video-conditioned next-step planning and outputs actions from streaming visual observations [64], while Uni-NaVid further unifies multiple embodied navigation tasks under a video-based VLA formulation [63]. Some works build world models that synthesize unobserved or future views to support planning or action evaluation, such as Pathdreamer [23] and Nav- Morph [59]. Other approaches treat future views as learning targets and intro- duce auxiliary objectives that predict or align with next-step observation seman- tics, such as future-view image semantics generation [29]. While these methods demonstrate the value of future-aware signals, anticipatory reasoning is often implemented as separate predictors, generative models, or inference-time roll- outs and fusion procedures, which may introduce additional computation and policy–model mismatch during closed-loop execution. In contrast, our work aims to integrate this capability into a single end-to-end navigator. 5 Latent reasoning shifts multi-step computation into continuous latent space rather than explicit chain-of-thought text, reducing token cost while retain- ing iterative internal processing. In language models, Think Before introduces pause tokens for extra latent computation [12], Quiet-STaR learns implicit ratio- nales [60], and Coconut feeds hidden states back as reasoning states [13]. Related efforts further distill or compress latent reasoning, including CoDi [42], SIM- CoT [55], and Think Silently, Think Fast [44]. In multimodal settings, latent reasoning has expanded from language-only states to visual and joint vision– language latents. LVR reasons autoregressively in visual embedding space [27]; Reasoning in the Dark interleaves latent vision–text reasoning [6]; CoCoVa per- forms iterative multimodal refinement through continuous cross-modal thought chains [33]; and Latent Implicit Visual Reasoning learns visual reasoning tokens without explicit intermediate supervision [30]. FantasyVLN [69] extends this line to VLN by unifying textual, visual, and multimodal CoT branches within a shared implicit reasoning framework. It mainly addresses the high token cost and low efficiency of explicit multimodal CoT in long-horizon VLN. By contrast, our method does not formulate navigation as unified CoT learning; it directly learns action-conditioned visual dynamics from trajectory future observations and internalizes them into a recurrent Pilot Token for anticipatory control. 3 Method 3.1 Task Definition We consider instruction-following Vision-and-Language Navigation in continu- ous 3D environments as an instruction-conditioned partially observable Markov decision process (POMDP) [21]. Formally, letM = (S,A,O,T ,Ω), where s t ∈S denotes the (latent) environment state at time t. Given a natural-language in- struction x = (w 1 ,...,w L ), the agent receives an egocentric monocular observa- tion o t ∈O ⊂ R H×W×C (with C = 3 for RGB and optionally C = 4 for RGB-D), and outputs a low-level action a t ∈A. The environment evolves according to s t+1 ∼T (·| s t ,a t ),o t+1 ∼ Ω(·| s t+1 ),(1) where T and Ω are the state transition and observation models, respectively. This interaction repeats over time, producing a trajectory τ = x, o 1:T , a 1:T ,o 1:t ≜ (o 1 ,..., o t ), a 1:t ≜ (a 1 ,...,a t ),(2) and terminates when a t = STOP or t = T max . We adopt the standard discrete action set: A =FWD, LEFT, RIGHT, STOP,(3) where each action corresponds to a small motion primitive in continuous space. Our goal is to learn a policy π θ (a t | x, o 1:t ) such that its rollouts reach the goal region G ⊂S and terminate by predicting STOP at the target. 6 Agent LatentPilotBackbone Action: LEFT STOP Action: FWD Action: LEFT Action: FWD Instruction:Go down the hallway. Turn left into the last bedroom. Wait beside the bed. Time 푡 1 푡 2 푡 3 푡 4 Observationo 1 Observation o 2 Action Vision encoder Pilot푧 푡 Observation표 푡+2 Hidden states 퐡 푡 pil InstructionObservation Pilot HistoryAction Pilot module L2 Loss ℒ pil <|placeholder|> 푇 푚푎푥 ... ... ... 퐺 휓 v 푡 Pilot Module 퐺 휓 푧 푡 퐡 푡 pil Vision Encoder 퐸 ∅ SigLip LLM Backbone 퐹 휃 퐮 푡 퐡 푡 pil 퐡 푡 act Architecture Core 푠 푡 푠 푡+1 Observation o 3 Observation o 4 Observation o 푇 Instruction 푧 1 푧 2 푧 3 푧 푇−1 Pilot slot Pilot slot Pilot slot Pilot slot Pilot slot 퐡 1 pil 퐡 2 pil 퐡 3 pil 퐡 4 pil 퐡 1 act 퐡 2 act 퐡 3 act 퐡 4 act 퐡 푇 act Wait beside the bed. ℒ=ℒ act +λℒ pil Training Model Inference Model തv 푡+2 ... Environment S Fig. 2: Overview of LatentPilot. The vision encoder extracts visual tokens from the current observation, the LLM backbone predicts the action, and a lightweight Pilot module updates a propagated Pilot Token that is cached and reused as the Pilot slot input at the next step, enabling strictly causal dreaming-ahead. 3.2 LatentPilot As illustrated in Fig. 2, LatentPilot is an end-to-end navigator composed of three components: a vision encoder E φ , a lightweight Pilot module G ψ , and a LLM decision backbone F θ . The core idea is to maintain a step-propagated continuous latent state z t ∈ R d , termed the Pilot Token. Pilot Token serves as a compact, persistent internal reasoning state throughout navigation, enabling the policy to carry forward anticipatory cues without generating verbose textual rationales. Pilot initial. Given the egocentric observation o t ∈ R H×W×C at step t, we adopt a SigLIP [62]-initialized vision encoder to obtain a sequence of visual tokens v t = E φ (o t )∈ R N v ×d ,(4) where N v is the number of visual tokens and d matches the hidden size of the backbone. We initialize the decision backbone F θ from the 7B LLaVA-Video model [67]. At each step, the backbone consumes a concatenated multimodal sequence u t consisting of instruction tokens, current visual tokens, and the Pilot Token z t−1 carried from the previous step: u t ≜ h Tok(x) ; v t ; PILOT(z t−1 ) i .(5) Here PILOT(·) denotes placing z t−1 at a dedicated token positions, called Pilot slot. Pilot slot is implemented by a special placeholder token <|placeholder|> whose embedding provides z 0 at the first step. Unlike discrete language tokens, z t lies in a continuous latent space and is carried across steps, making it a natural vehicle for long-horizon, trajectory-level internal computation. Pilot propagation. The LLM backbone computes hidden states with causal at- tention, H t = F θ (u t )∈ R N×d .(6) 7 (i)Rollout Collection(i)Deviation-aware Correction (i)Privileged Target Generation (iv)Fine-Tuning Collect trajectories in continuous 3D environment using current policy. Expert policy overrides agent when deviation exceeds. Minimizing weighted and update our policy, remains strictly causal at inference. Agent Path Start Target Trajectory 휏 Expert Path Deviation! Expert Trajectory 휏 Observation표 푡+2 Target തv 푡+2 SigLip LatentPilotBackbone Pilot Module Action Head SigLip LLM Backbone തv 푡+1 Pilot Slot lossℒ=ℒ act +λℒ pil Pool 푧 푡 푎 푡 Fig. 3: PilotLoop: flywheel-style closed-loop training. Each round iterates rollout col- lection, expert takeover for deviation-aware correction, future-privileged target con- struction, and fine-tuning with joint action imitation and Pilot supervision. Let h act t and h pil t denote the action and Pilot-position hidden states from H t , respectively. We obtain the low-level action distribution via a linear action head, π θ (a t | x, o t , z t−1 ) = Softmax W a h act t , a t ∈A.(7) The Pilot Token is then updated by a lightweight projection-only Pilot module G ψ : z t = G ψ h pil t ∈ R d , (8) where G ψ is implemented as a simple linear layer, with only a few parameters that projects the Pilot-position hidden state back to the latent space. Impor- tantly, G ψ does not introduce any additional observations or external predictive components (e.g., world-model rollouts); it merely performs a compact latent projection to support cross-step propagation. Overall, one navigation step can be summarized as (a t , z t ) =F θ,ψ x, o t , z t−1 ,(9) where F θ,ψ denotes the overall step transition induced by the model: given the instruction, the current observation, and the previous Pilot Token, it outputs the next action and the updated Pilot Token. The action a t is executed in the environment to yield the next observation o t+1 , while z t is fed into the next step through the Pilot slot. This recurrent latent propagation links stepwise decisions into a coherent, trajectory-level internal reasoning process within a single backbone. So far we have described the architecture and information flow of LatentPilot. In the next subsection, we introduce how training-time supervi- sion derived from future observations shapes z t into an anticipatory latent that captures action-conditioned visual dynamics, while keeping test-time inputs un- changed. ̄ v t+1 and z t is supervised toward ̄ v t+2 . 8 3.3 Training with Future-Privileged Supervision We train LatentPilot in a flywheel-style closed-loop procedure (Fig. 3), where data collection and model updates iterate in multiple rounds. Each round cycles through four stages: (i) rollout collection under the current policy, (i) deviation- aware correction where an expert takes over when the agent drifts too far from a reference trajectory, (i) privileged target generation from the collected rollouts, and (iv) fine-tuning with a joint objective. The key supervision signal comes from future observations recorded in these trajectories, used as training-only privileged information in the spirit of LUPI [45]. Flywheel data collection with expert takeover. Let D = τ (i) N i=1 denote the trajectory buffer collected in the current flywheel round, where τ = x, o 1:T , a col 1:T .(10) Here a col t is the collected action at step t. By default it is produced by the current model, while an expert policy may override it when the deviation from the reference becomes large [40]. We then unroll LatentPilot along τ using the same recurrent interface as Eq. (9) and optimize a standard action cross-entropy loss, here actions are decoded by the backbone’s native LLM output projection. Two-step future privilege. A central design choice is to explicitly distinguish two future steps: t+1 as input, t+2 as target. At training step t, we treat the next observation o t+1 as a privileged input to the Pilot slot, and the next-next observation o t+2 as a privileged target for supervising the predicted Pilot Token. Concretely, we first compress an observation into a single latent vector by mean pooling the vision tokens: ̄ v t+1 ≜ Pool(E φ (o t+1 ))∈ R d .(11) During training, we fill the Pilot slot with the one-step future latent ̄ v t+1 as training-only privileged input. So multimodal input sequence u t consisting of instruction tokens, current visual tokens, and one-step future: u tr t ≜ h Tok(x) ; v t ; PILOT( ̄ v t+1 ) i ,(12) where v t = E φ (o t ) is the visual token sequence . A forward pass yields hidden states H t = F θ (u tr t ) (Eq. (6)). Let h act t and h pil t denote the action and Pilot- position hidden states from H t . We supervise the action via: p θ (·) = Softmax W LM h act t , L act ≜− T X t=1 logp θ a col t | x, o t , ̄ v t+1 . (13) To shape the Pilot Token into an anticipatory latent, we regress the predicted Pilot Token to match the two-step future latent. Concretely, the Pilot module projects the Pilot-position hidden state into the continuous latent space and the privileged target is defined from the two-step future observation: z t ≜ G ψ h pil t ∈ R d , L pil ≜ T−2 X t=1 ∥z t − ̄ v t+2 ∥ 2 2 .(14) 9 Overall objective and train–test interface. We fine-tune the model in each fly- wheel round by minimizing min θ,φ,ψ E τ∼D [L act (τ ) + λL pil (τ )],(15) where λ balances imitation learning and future-privileged supervision and we set 0.1 in practice. Finally, note the train–test interface is simple and strictly causal at evaluation: during training, the Pilot slot is teacher-forced with We train LatentPilot on a mixed corpus of instruction-following trajectories from Matterport3D (MP3D) scenes, including R2R [3], RxR [26], and the EnvDrop- augmented R2R set [43], and further include ScaleVLN trajectories synthesized on HM3D scenes [36,52]. We first bootstrap the policy with imitation learning using Habitat’s shortest-path follower as the expert policy [25, 41], and then perform flywheel-style closed-loop fine-tuning. I I H H O O F F P P A A I 1 I 2 H 1 H 2 O 1 O 2 F 1 F 2 P 1 P 2 A 1 A 2 I 1 I 2 H 1 H 2 O 1 O 2 F 1 F 2 P 1 P 2 A 1 A 2 Token order Token order VisibleMasked I: Instr H: Hist O: Curr F: FutureP: Pilot A: Action Fig. 4: Visibility matrix of LatentPilot. Inference with stored pilot. During inference, the model per- forms a purely recurrent rollout a PilotCache, without accessing any future frames or computing addi- tional Pilot representations. Instead of generating a separate Pilot signal, the model directly reuses the previ- ously predicted Pilot Token through a read–write cache mechanism. Specifi- cally, after step t−1 the model outputs z t−1 , which is stored in the cache and used as the Pilot-slot input at step t (i.e., PILOT(z t−1 ) in u t ). The new out- put z t then overwrites the cache for the next step. This read–write update repeats until STOP, enabling cross-step Pilot propagation using only the current observation and the cached latent. Im- portantly, no separate Pilot computation is required at test time as the cached latent itself serves as the Pilot representation. As a result, cross-step Pilot prop- agation is achieved using only the current observation and a single cached latent. 4 Experiments We conduct experiments to answer the following questions: (1) Can we use these recorded future observations during training to learn how actions change future visual inputs, while keeping inference strictly causal? (2)What are the benefits of integrating future information and “imagination” directly into the decision- making LLM, compared to relying on external predictive models or auxiliary modules? We evaluate our method across representative robot embodiments, in- cluding humanoid robots, wheeled platforms, and quadruped robots, to 10 Table 1: Comparison with state-of-the-art methods on VLN-CE R2R and RxR Val- Unseen splits. Pano, Odo and D respectively represent panoramic view, odometry and depth, SRGB denotes monocular RGB. Method ObservationR2R Val-UnseenRxR Val-Unseen Pano.Odo.D.SRGBNE↓ OS↑ SR↑ SPL↑NE↓ SR↑ SPL↑nDTW↑ CMA [14]✓ ✓ ✓6.20 52.0 41.0 36.08.76 26.5 22.1 47.0 VLN-BERT [14]✓ ✓ ✓5.74 53.0 44.0 39.08.98 27.0 22.6 46.7 Sim2Sim [24] ✓ ✓ ✓6.07 52.0 43.0 36.0– – – Ego 2 -Map [15]✓ ✓ ✓5.54 56.0 47.0 41.0– – – DreamWalker [47]✓ ✓ ✓5.53 59.0 49.0 44.0– – – GridMM [50] ✓ ✓ ✓5.11 61.0 49.0 41.0– – – ETPNav [2]✓ ✓ ✓4.71 65.0 57.0 49.05.64 54.7 44.8 61.9 ScaleVLN [52]✓ ✓ ✓4.80 – 55.0 51.0– – – InstructNav [32]✓ ✓ ✓ ✓6.89 – 31.0 24.0– – – R2R-CMTP [8]✓ ✓ ✓7.90 38.0 26.4 22.7– – – LAW [37]✓ ✓ ✓6.83 44.0 35.0 31.010.90 8.0 8.0 38.0 CM2 [11]✓ ✓ ✓7.02 41.5 34.3 27.6– – – WS-MGMap [9]✓ ✓ ✓6.28 47.6 38.9 34.3– – – Sim2Real [51]✓ ✓ ✓5.95 55.8 44.9 30.48.79 25.5 18.1 – Seq2Seq [25]✓ ✓7.77 37.0 25.0 22.012.10 13.9 11.9 30.8 CMA [25]✓ ✓7.37 40.0 32.0 30.0– – – NaVid [64]✓5.47 49.1 37.4 35.9– – – AO-Planner [7]✓5.55 59.0 47.0 33.07.06 43.3 30.5 50.1 COSMO [66]✓– 56.0 47.0 40.0– – – MapNav [65]✓4.93 53.0 39.7 37.2– – – NaVid-4D [31]✓ ✓5.99 55.7 43.8 37.1– – – NavMorph [58]✓ ✓5.75 56.9 47.9 33.28.85 30.8 22.8 44.2 Uni-NaVid [63]✓5.58 53.3 47.0 42.76.24 48.7 40.9 – NaVILA [10]✓5.22 62.5 54.0 49.06.77 49.3 44.0 58.8 StreamVLN [54]✓4.98 64.2 56.9 51.96.22 52.9 46.0 61.9 JanusVLN [61]✓4.78 65.2 60.5 56.86.06 56.2 47.5 62.1 LatentPilot (Ours)✓4.4166.362.058.05.1958.249.967.5 demonstrate the broad applicability of LatentPilot. VLN-PE enables controlled experiments with simulated humanoid robots, and our real-world deployments further showcase LatentPilot on wheeled and quadruped robots. 4.1 Environment and Metrics. We evaluate LatentPilot in both the standard continuous VLN setting and a physically realistic navigation setting. For continuous navigation, we follow the VLN-CE [25] and report results on the Val-Unseen splits of R2R-CE [3] and RxR-CE [26] and executed in the Habitat simulator [41]. Following prior work, we use the standard VLN metrics. Navigation Error (NE) measures the final geodesic distance (in meters) from the agent’s stopping location to the goal. Success Rate (SR) is the fraction of episodes where the agent stops within 11 Table 2: Evaluation Metrics on VLN-PE benchmark with physical locomotion con- troller. +: model is first trained on Habitat and fine-tuned on VLN-PE. †: model is trained with data augmentation. Method R2R Validation SeenR2R Validation Unseen NE↓ FR↓ StR↓ OS↑ SR↑ SPL↑NE↓ FR↓ StR↓ OS↑ SR↑ SPL↑ Train-free Map-based Exploration and Navigation VLMaps [16]–6.98 23.00 0.00 20.00 20.00 12.70 Train on VLN-PE Seq2Seq [25]7.73 22.19 3.04 30.55 19.60 15.677.91 19.67 3.71 27.62 15.89 12.58 Seq2Seq+ [25]7.54 26.11 5.59 31.93 19.58 15.137.64 21.82 5.12 30.47 18.13 14.06 CMA [25]7.59 23.71 3.19 34.94 21.58 16.107.98 22.64 3.27 33.11 19.15 14.05 CMA+ [25] 7.14 23.56 3.50 36.17 25.84 21.757.26 21.75 3.27 31.40 22.12 18.65 RDP [49] 6.76 27.51 1.82 38.60 25.08 17.076.72 24.57 3.11 36.90 25.24 17.73 Zero-shot Transfer Evaluation from VLN-CE Seq2Seq [25] †7.62 20.21 3.04 19.30 15.20 12.797.18 18.04 3.04 22.42 16.48 14.11 CMA [25]†7.37 20.06 3.95 18.54 16.11 14.647.09 17.07 3.79 20.86 16.93 15.24 NaVid [64]6.20 11.25 0.46 24.32 21.58 17.455.94 8.61 0.45 27.32 22.42 18.58 DualVLN [53]4.13 17.78 1.82 62.31 58.97 47.784.66 12.32 2.23 55.90 51.60 42.49 Ours4.1011.091.0863.8759.0149.654.3310.650.9760.3156.4247.74 3 meters of the goal. Oracle Success Rate (OSR) considers the closest point along the trajectory as the stopping point. Success weighted by Path Length (SPL) [3] accounts for both success and path efficiency. Normalized Dynamic Time Warping (nDTW) [19] measures trajectory fidelity to the reference path. We additionally report results on VLN-PE [49] follow DualVLN [53]. VLN- PE evaluation is executed on the physically realistic simulation stack built on NVIDIA Isaac Lab [34], which models robot dynamics and locomotion execution imperfections. We report four primary VLN metrics: NE, SR, OS and SPL. To explicitly measure physical robustness, we further report Fall Rate (FR) measures the frequency of robot falls and Stuck Rate (StR) occurrences where the agent is unable to move. For humanoid evaluation in VLN-PE, we adopt the Unitree H1 embodiment, a full-size humanoid platform, about 1.8 m tall and equipped with depth camera [39] in simulator. We perform on-policy rollout collection using 16×RTX 4090 GPUs and training using 8×NVIDIA A100 GPUs. 4.2 Main result Results on VLN-CE benchmark. Table 1 summarizes results on the VLN- CE benchmarks. We compare against classic recurrent baselines (Seq2Seq/CMA) as well as stronger pipelines that rely on richer observations such as panoramic views, odometry, and depth (e.g., ETPNav and ScaleVLN) [2, 25, 52]. Despite using only monocular RGB, LatentPilot achieves the strongest overall perfor- mance, surpassing prior single-RGB VLM navigators such as NaVid [64] and re- maining competitive with several sensor-heavy methods. LatentPilot compares 12 Table 3: Ablation study of supervision modality for the Pilot Token. NaN removes any extra Pilot supervision and optimizes only the action cross-entropy, serving as a baseline that tests whether the propagated latent becomes predictive by itself. Modality R2R Validation UnseenR2R Validation Seen SR↑ SPL↑ NE↓ OS↑SR↑ SPL↑ NE↓ OS↑ NaN51.7 47.15.3 57.053.1 46.94.6 60.7 3D50.3 45.65.4 55.851.5 45.14.9 59.9 Text53.2 48.75.2 59.654.9 48.64.5 63.9 Vision(ours)62.0 58.04.4 66.365.1 59.64.3 71.6 Table 4: Internalized future supervision (ours) vs. plug-in video world models on R2R Val-Seen. T/Act: ms per action; Mem: peak GPU memory (GB). World Model T/Act (ms)↓ Mem↓ (GB) R2R Val-Seen SR↑ SPL↑ NE↓ OS↑ Wan2.1 1.3B [46]204041.556.6 51.7 5.1 64.5 CogVideoX1.5 5B [57]546032.459.3 53.9 4.9 66.8 Ours13022.865.1 59.6 4.3 71.6 favorably with existing approaches across standard VLN benchmarks while re- quiring a simpler observation setup, supporting the motivation that internalizing action-conditioned foresight into the decision backbone can reduce myopic “step- and-see” behavior. Results on VLN-PE benchmark. Table 2 reports results on VLN-PE [49], which evaluates continuous navigation under physically realistic humanoid loco- motion with explicit robustness metrics (falls and deadlocks). Across all baselines our method achieves the best overall performance, especially on the challenging Val-Unseen split. LatentPilot not only more accurate at reaching goals but also more stable and safer under realistic execution noise. 4.3 Ablation Study Supervision modality for the Pilot Token. We keep the LatentPilot archi- tecture and inference interface unchanged and vary only the supervision signal used to shape the Pilot Token (Table 3). NaN removes any extra Pilot supervi- sion and optimizes only the action cross-entropy, serving as a baseline that tests whether the propagated latent becomes predictive by itself. 3D uses geometry- aware features extracted by a 3D foundation model (VGGT [48]) as the privi- leged target, and Text uses textual descriptions of future views generated by a strong VLM (Qwen2.5-VL [4]) and then tokenized as supervision. We find that supervision choice matters substantially: proxy targets such as 3D embeddings or text provide limited improvements and can even degrade performance. Internalized imagination vs. plug-in world models. To further answer Question (2), we compare LatentPilot with a representative class of external attach a video world model [46, 57] to predict future views and then use these predictions to guide action selection (Table 4). Although such plug-in designs 13 can provide explicit foresight, they also introduce an additional generative back- bone at inference time, substantially increasing per-step latency and memory footprint. In contrast, LatentPilot internalizes the action-observation dynamics into the decision backbone via training-only future supervision, so inference re- mains a single forward pass with a lightweight latent propagation. As a result, our approach achieves markedly lower time-per-action and peak memory, while also delivering stronger navigation performance. Integrated imagination are not only improved success but also faster and more resource-efficient deployment. 12345678 Flywheel Round 50 55 60 Score (%) R2R Validation Unseen 54.0 55.6 56.4 58.1 60.3 61.2 62.0 61.8 48.5 49.6 49.8 52.3 55.0 57.3 58.0 57.6 SRSPL Fig. 5: Performance over PilotLoop. Effect of flywheel it- erations (PilotLoop). Fig- ure 5 plots performance on R2R Val-Unseen after each PilotLoop round. We observe a clear, steady improvement in both success (SR) and path efficiency (SPL) as the fly- wheel progresses, with the largest gains appearing in the early rounds and gradually saturating later. The expert-takeover mechanism stabilizes the loop by preventing severe drift when the policy deviates too far, echoing the dataset-aggregation intuition in DAgger-style training [40]. −3−2−10123 −3 −2 −1 0 1 2 3 4 Pilot Token Space Visualization via Dimensionality Reduction LEFT FWD RIGHT Fig. 6: Latent collapse Analysis via PCA. Latent collapse analysis. A common failure mode of latent- token methods is latent collapse, where the latent vectors degen- erate into near-constant, uninfor- mative representations. To diag- nose this, we collect the propa- gated Pilot Tokens z t from roll- outs and project them to 2D us- ing principal component analy- sis (PCA) [20]. We then color each point by the executed action (LEFT, FWD, RIGHT); STOP is ex- cluded due to its low frequency. As shown in Fig. 6, Pilot Tokens form action-correlated and struc- tured clusters rather than collapsing to a single mode, indicating that z t preserves meaningful variation and remains strongly grounded in control-relevant informa- tion. This supports that action grounding together with predictive supervision can maintain a non-degenerate latent space, enabling the Pilot Token to serve as a compact internal state instead of an empty placeholder. 14 Instruction: Go straight to the doorway. Exit and turn right, Go straight past the sofa on your left and stop next to the wall. Instruction: Starting from the stairs, walk forward past the couch on your right, then turn left and continue straight until you reach theelevator. Fig. 7: Real-world Cross-Embodiment Experiments: Wheeled and Quadruped Robots. LatentPilot successfully follows long, multi-step instructions on both Robots. 4.4 Real-world Experiments To further assess practical deployability beyond simulation, we deploy LatentPi- lot on two mainstream robot embodiments in indoor corridors: a wheeled AgileX LiMO Pro and a quadruped Unitree Go2. LiMO Pro is equipped with LiDAR and a Intel Realsense D435 depth camera [38], while Go2 is a bionic quadruped fea- turing a wide-angle camera. Both robots run ROS 2 for I/O and control. During execution, the robot streams egocentric observations to a remote workstation with RTX 4090 GPU over a local network; the server performs inference and sends back the predicted navigation action, which is then executed by the robot- specific low-level controller. As shown in Fig. 7, LatentPilot successfully follows long, multi-step instructions on both wheeled and legged platforms, suggesting that the learned Pilot Token provides a deployment-friendly form of internalized “dreaming ahead” without requiring any plug-in world-model rollout. 5 Conclusion In this work, we presented LatentPilot, an end-to-end VLM-based navigator that internalizes lookahead for vision-and-language navigation. Instead of relying on external imagination modules or world-model rollouts at inference, LatentPi- lot leverages a simple yet effective principle: although future observations are unavailable when an action is chosen, they are naturally recorded in offline tra- jectories and can serve as training-only privileged supervision. By distilling these action-conditioned future visual consequences into a compact Pilot Token, our model learns anticipatory representations that support more stable sequential decision making while keeping inference strictly causal. 15 References 1. An, D., Qi, Y., Li, Y., Huang, Y., Wang, L., Tan, T., Shao, J.: Bevbert: Multimodal map pre-training for language-guided navigation. arXiv preprint arXiv:2212.04385 (2022) 2. An, D., Wang, H., Wang, W., Wang, Z., Huang, Y., He, K., Wang, L.: Etpnav: Evolving topological planning for vision-language navigation in continuous envi- ronments. arXiv preprint arXiv:2304.03047 (2023) 3. Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sünderhauf, N., Reid, I., Gould, S., van den Hengel, A.: Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2018) 4. Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., Lin, J.: Qwen2.5-vl technical report. CoRR abs/2502.13923 (2025). https: //doi.org/10.48550/ARXIV.2502.13923, https://doi.org/10.48550/arXiv. 2502.13923 5. Chang, A.X., Dai, A., Funkhouser, T., Halber, M., Nießner, M., Savva, M., Song, S., Zeng, A., Zhang, Y., Xiao, J.: Matterport3d: Learning from rgb-d data in indoor environments. In: Proceedings of the International Conference on 3D Vision (3DV). p. 667–676 (2017) 6. Chen, C., Ma, Z., Li, Y., Hu, Y., Wei, Y., Li, W., Nie, L.: Reasoning in the dark: Interleaved vision-text reasoning in latent space. arXiv preprint arXiv:2510.12603 (2025) 7. Chen, J., Lin, B., Liu, X., Ma, L., Liang, X., Wong, K.Y.K.: Affordances-oriented planning using foundation models for continuous vision-language navigation. In: AAAI Conference on Artificial Intelligence (AAAI) (2025) 8. Chen, K., Chen, J.K., Chuang, J., Vazquez, M., Savarese, S.: Topological plan- ning with transformers for vision-and-language navigation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021) 9. Chen, P., Ji, D., Lin, K., Zeng, R., Li, T.H., Tan, M., Gan, C.: Weakly-supervised multi-granularity map learning for vision-and-language navigation. arXiv preprint arXiv:2210.07506 (2022) 10. Cheng, A.C., Ji, Y., Yang, Z., Gongye, Z., Zou, X., Kautz, J., Bıyık, E., Yin, H., Liu, S., Wang, X.: Navila: Legged robot vision-language-action model for navigation. In: Robotics: Science and Systems (RSS) (2025) 11. Georgakis, G., Schmeckpeper, K., Wanchoo, K., Dan, S., Miltsakaki, E., Roth, D., Daniilidis, K.: Cross-modal map learning for vision and language navigation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022) 12. Goyal, S., Ji, Z., Rawat, A.S., Menon, A.K., Kumar, S., Nagarajan, V.: Think before you speak: Training language models with pause tokens. arXiv preprint arXiv:2310.02226 (2023) 13. Hao, S., Sukhbaatar, S., Su, D., Li, X., Hu, Z., Weston, J., Tian, Y.: Train- ing large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769 (2024) 16 14. Hong, Y., Wang, Z., Wu, Q., Gould, S.: Bridging the gap between learning in dis- crete and continuous environments for vision-and-language navigation. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022) 15. Hong, Y., Zhou, Y., Zhang, R., Dernoncourt, F., Bui, T., Gould, S., Tan, H.: Learn- ing navigational visual representations with semantic map supervision. In: Pro- ceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2023) 16. Huang, C., Mees, O., Zeng, A., Burgard, W.: Visual language maps for robot navigation. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) (2023) 17. Huang, Y., Jiang, X., Gao, X., Wu, M., Tu, Z.: Vistav2: World imagination for in- door vision-and-language navigation (2025). https://doi.org/10.48550/arXiv. 2512.00041, https://arxiv.org/abs/2512.00041 18. Huang, Y., Wu, M., Li, R., Tu, Z.: Vista: Generative visual imagination for vision- and-language navigation (2025). https://doi.org/10.48550/arXiv.2505.07868, https://arxiv.org/abs/2505.07868 19. Ilharco, G., Jain, V., Ku, A., Ie, E., Baldridge, J.: General evaluation for in- struction conditioned navigation using dynamic time warping. arXiv preprint arXiv:1907.05446 (2019) 20. Jolliffe, I.T.: Principal Component Analysis. Springer, 2nd edn. (2002) 21. Kaelbling, L.P., Littman, M.L., Cassandra, A.R.: Planning and acting in partially observable stochastic domains. Artificial Intelligence 101(1–2), 99–134 (1998) 22. Koh, J.Y., Lee, H., Yang, Y., Baldridge, J., Anderson, P.: Pathdreamer: A world model for indoor navigation. In: Proceedings of the IEEE/CVF International Con- ference on Computer Vision. p. 14738–14748 (2021) 23. Koh, J.Y., Lee, H., Yang, Y., Baldridge, J., Anderson, P.: Pathdreamer: A world model for indoor navigation. In: Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV). p. 14738–14748 (October 2021), https://openaccess.thecvf.com/content/ICCV2021/html/Koh_Pathdreamer_ A_World_Model_for_Indoor_Navigation_ICCV_2021_paper.html 24. Krantz, J., Lee, S.: Sim-2-sim transfer for vision-and-language navigation in contin- uous environments. In: European Conference on Computer Vision (ECCV) (2022) 25. Krantz, J., Wijmans, E., Majundar, A., Batra, D., Lee, S.: Beyond the nav-graph: Vision-and-language navigation in continuous environments. In: Proceedings of the European Conference on Computer Vision (ECCV) (2020) 26. Ku, A., Anderson, P., Patel, R., Ie, E., Baldridge, J.: Room-across-room: Multi- lingual vision-and-language navigation with dense spatiotemporal grounding. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2020) 27. Li, B., Sun, X., Liu, J., Wang, Z., Wu, J., Yu, X., Chen, H., Barsoum, E., Chen, M., Liu, Z.: Latent visual reasoning. arXiv preprint arXiv:2509.24251 (2025) 28. Li, J., Bansal, M.: Improving vision-and-language navigation by generating future- view image semantics. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023) 29. Li, J., Bansal, M.: Improving vision-and-language navigation by generating future- view image semantics. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 10803–10812 (June 2023). https: //doi.org/10.1109/CVPR52729.2023.01040, https://openaccess.thecvf.com/ content/CVPR2023/html/Li_Improving_Vision-and-Language_Navigation_by_ Generating_Future-View_Image_Semantics_CVPR_2023_paper.html 17 30. Li, K., Shang, C., Karlinsky, L., Feris, R., Darrell, T., Herzig, R.: Latent implicit visual reasoning. arXiv preprint arXiv:2512.21218 (2025) 31. Liu, H., Wan, W., Yu, X., Li, M., Zhang, J., Zhao, B., Chen, Z., Wang, Z., Zhang, Z., Wang, H.: Navid-4d: Unleashing spatial intelligence in egocentric rgb-d videos for vision-and-language navigation. In: IEEE International Conference on Robotics and Automation (ICRA) (2025) 32. Long, Y., Cai, W., Wang, H., Zhan, G., Dong, H.: Instructnav: Zero-shot sys- tem for generic instruction navigation in unexplored environment. arXiv preprint arXiv:2406.04882 (2024), accepted by CoRL 2024 33. Ma, J., Zhou, X., Song, Y., Yan, H.: Cocova: Chain of continuous vision-language thought for latent space reasoning. arXiv e-prints p. arXiv–2511 (2025) 34. Mittal, M., Roth, P., Tigue, J., Richard, A., Zhang, O., Du, P., Serrano-Muñoz, A., Yao, X., Zurbrüg, R., Rudin, N., et al.: Isaac lab: A gpu-accelerated simulation framework for multi-modal robot learning. arXiv preprint arXiv:2511.04831 (2025) 35. Perincherry, A., Krantz, J., Lee, S.: Do visual imaginations improve vision-and- language navigation agents? (2025) 36. Ramakrishnan, S., et al.: Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. arXiv preprint arXiv:2109.08238 (2021) 37. Raychaudhuri, S., Wani, S., Patel, S., Jain, U., Chang, A.X.: Language-aligned waypoint (LAW) supervision for vision-and-language navigation in continuous en- vironments. In: Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP) (2021) 38. Robotics, A.: Limo pro: Ros2 mobile robot platform (product page). Web page, https://global.agilex.ai/products/limo-pro 39. Robotics, U.: Unitree h1 humanoid robot. https://w.unitree.com/h1/ 40. Ross, S., Gordon, G., Bagnell, J.A.: A reduction of imitation learning and struc- tured prediction to no-regret online learning. In: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AISTATS) (2011) 41. Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., Parikh, D., Batra, D.: Habitat: A platform for embodied ai research. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2019) 42. Shen, Z., Yan, H., Zhang, L., Hu, Z., Du, Y., He, Y.: Codi: Compressing chain-of- thought into continuous space via self-distillation. arXiv preprint arXiv:2502.21074 (2025) 43. Tan, H., Yu, L., Bansal, M.: Learning to navigate unseen environments: Back translation with environmental dropout. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). p. 3356–3366. Association for Computational Linguistics, Minneapolis, Minnesota (2019). https: //doi.org/10.18653/v1/N19-1268, https://aclanthology.org/N19-1268 44. Tan, W., Li, J., Ju, J., Luo, Z., Luan, J., Song, R.: Think silently, think fast: Dy- namic latent compression of llm reasoning chains. arXiv preprint arXiv:2505.16552 (2025) 45. Vapnik, V., Vashist, A.: A new learning paradigm: Learning using privileged infor- mation. Neural Networks 22(5–6), 544–557 (2009) 46. Wan, T., Wang, A., Ai, B., Wen, B., Mao, C., Xie, C.W., Chen, D., Yu, F., Zhao, H., Yang, J., et al.: Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314 (2025) 18 47. Wang, H., Liang, W., Van Gool, L., Wang, W.: DREAMWALKER: Mental plan- ning for continuous vision-language navigation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2023) 48. Wang, J., Chen, M., Karaev, N., Vedaldi, A., Rupprecht, C., Novotný, D.: VGGT: visual geometry grounded transformer. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11– 15, 2025. p. 5294–5306. Computer Vision Foundation / IEEE (2025). https: //doi.org/10.1109/CVPR52734.2025.00499, https://openaccess.thecvf.com/ content/CVPR2025/html/Wang_VGGT_Visual_Geometry_Grounded_Transformer_ CVPR_2025_paper.html 49. Wang, L., Xia, X., Zhao, H., Wang, H., Wang, T., Chen, Y., Liu, C., Chen, Q., Pang, J.: Rethinking the embodied gap in vision-and-language navigation: A holis- tic study of physical and visual disparities. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). p. 9455–9465 (Oct 2025) 50. Wang, Z., Li, X., Yang, J., Liu, Y., Jiang, S.: GridMM: Grid memory map for vision-and-language navigation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2023) 51. Wang, Z., Li, X., Yang, J., Liu, Y., Jiang, S.: Sim-to-real transfer via 3d feature fields for vision-and-language navigation. arXiv preprint arXiv:2406.09798 (2024) 52. Wang, Z., Li, J., Hong, Y., Wang, Y., Wu, Q., Bansal, M., Gould, S., Tan, H., Qiao, Y.: Scaling data generation in vision-and-language navigation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2023) 53. Wei, M., Wan, C., Peng, J., Yu, X., Yang, Y., Feng, D., Cai, W., Zhu, C., Wang, T., Pang, J., Liu, X.: Ground slow, move fast: A dual-system foundation model for generalizable vision-and-language navigation. arXiv preprint arXiv:2512.08186 (2025) 54. Wei, M., Wan, C., Yu, X., Wang, T., Yang, Y., Mao, X., Zhu, C., Cai, W., Wang, H., Chen, Y., Liu, X., Pang, J.: Streamvln: Streaming vision-and-language navigation via slowfast context modeling. arXiv preprint arXiv:2507.05240 (2025) 55. Wei, X., Liu, X., Zang, Y., Dong, X., Cao, Y., Wang, J., Qiu, X., Lin, D.: Sim-cot: Supervised implicit chain-of-thought. arXiv preprint arXiv:2509.20317 (2025) 56. Xia, F., Zamir, A.R., He, Z., Sax, A., Malik, J., Savarese, S.: Gibson env: Real- world perception for embodied agents. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018) 57. Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang, Y., Hong, W., Zhang, X., Feng, G., et al.: Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072 (2024) 58. Yao, X., Gao, J., Xu, C.: Navmorph: A self-evolving world model for vision-and- language navigation in continuous environments. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2025) 59. Yao, X., Gao, J., Xu, C.: Navmorph: A self-evolving world model for vision- and-language navigation in continuous environments. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). p. 5536–5546 (October 2025), https://openaccess.thecvf.com/content/ICCV2025/html/ Yao_NavMorph_A_Self- Evolving_World_Model_for_Vision- and- Language_ Navigation_in_Continuous_ICCV_2025_paper.html 60. Zelikman, E., Harik, G., Shao, Y., Jayasiri, V., Haber, N., Goodman, N.D.: Quiet- star: Language models can teach themselves to think before speaking. arXiv preprint arXiv:2403.09629 (2024) 19 61. Zeng, S., Qi, D., Chang, X., Xiong, F., Xie, S., Wu, X., Liang, S., Xu, M., Wei, X.: Janusvln: Decoupling semantics and spatiality with dual implicit memory for vision-language navigation. arXiv preprint arXiv:2509.22548 (2025) 62. Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language im- age pre-training. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2023) 63. Zhang, J., Wang, K., Wang, S., Li, M., Liu, H., Wei, S., Wang, Z., Zhang, Z., Wang, H.: Uni-navid: A video-based vision-language-action model for unifying embodied navigation tasks. In: Robotics: Science and Systems (RSS) (2025) 64. Zhang, J., Wang, K., Xu, R., Zhou, G., Hong, Y., Fang, X., Wu, Q., Zhang, Z., Wang, H.: Navid: Video-based vlm plans the next step for vision-and-language navigation. In: Robotics: Science and Systems (RSS) (2024) 65. Zhang, L., Hao, X., Xu, Q., Zhang, Q., Zhang, X., Wang, P., Zhang, J., Wang, Z., Zhang, S., Xu, R.: Mapnav: A novel memory representation via annotated semantic maps for vlm-based vision-and-language navigation. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL) (2025) 66. Zhang, S., Qiao, Y., Wang, Q., Yan, Z., Wu, Q., Wei, Z., Liu, J.: COSMO: Combina- tion of selective memorization for low-cost vision-and-language navigation. In: Pro- ceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2025) 67. Zhang, Y., Wu, J., Li, W., Li, B., Ma, Z., Liu, Z., Li, C.: Llava-video: Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713 (2024) 68. Zhao, X., Cai, W., Tang, L., Wang, T.: Imaginenav: Prompting vision-language models as embodied navigator through scene imagination. In: International Con- ference on Learning Representations (ICLR) (2025), https://openreview.net/ forum?id=vQFw9ryKyK 69. Zuo, J., Mu, L., Jiang, F., Ma, C., Xu, M., Qi, Y.: Fantasyvln: Unified multi- modal chain-of-thought reasoning for vision-language navigation. arXiv preprint arXiv:2601.13976 (2026) Supplementary Material Overview This supplementary material is organized as follows: Sec. A presents experiment details, including Pilot Token supervision modal- ity ablations (Sec. A.1), plug-in world model baselines (Sec. A.2), Pilot Token col- lapse analysis (Sec. A.3), and real-world deployment/control details (Sec. A.4). Sec. B provides more qualitative results on a wheeled robot (Sec. B.1), a quadruped robot (Sec. B.2), and humanoid navigation in Isaac VLN-PE simu- lation (Sec. B.3); additional videos are inReal-World Demofolder. Finally, Sec. C discusses social impact, safety, and privacy. We will publicly release our code, dataset, and trained checkpoints. A Experiment Details A.1 Supervision Modality for the Pilot Token This section details the ablation study on the supervision modality used to shape the Pilot Token in Table 3. Across all variants, we keep the LatentPilot backbone, training data, and optimization hyperparameters unchanged, and only change the Pilot-slot supervision design (and its necessary tokenization, such as the number of<|placeholder|>tokens). At each step, we allocate a dedicated Pilot Token slot in the input sequence using special placeholders<|placeholder|>. The Pilot Token is initialized by these placeholders at the beginning of each episode. During rollout, we keep the sequential interface identical to the main method: the Pilot Token state produced at steptis fed back as the Pilot Token input at stept+1(i.e., we propagate the latent across time). All variants are trained with the same action cross-entropy loss between the predicted action distribution and the ground-truth action. In the NaN variant, we still insert<|placeholder|>to initialize the Pilot Token and still propagate the Pilot Token across steps exactly as usual. The only difference is that we do not apply any dedicated loss on the Pilot Token itself. The model is optimized solely with the action cross-entropy. This baseline tests whether the propagated latent becomes predictive on its own, and helps ver- ify whether explicit Pilot Token supervision is necessary. In the Text variant, we supervise the Pilot Token using short captions generated from the future observa- tion image by a strong VLM, Qwen2.5-VL [2]. Concretely, for each training sam- ple, we take the future image (with the same temporal offset used in our main set- ting) and query Qwen2.5-VL with the prompt:Describe the key elements of the image in no more than 10 words.We setmax_new_tokens=10to en- courage concise outputs. For example, a typical caption is:“Bed, dresser, 2 Describe the key elements of the image in no more than 10 words. Bed, dresser, lamp, artwork, window. Tokenizer of LatentPilot Pilot Text variant CE Loss ℒ pil Supervision Qwen2.5-VL 7B 3D Foundation model VGGT Linear Project Pilot 3D variant L2 Loss ℒ pil Supervision Fig. S1:Supervision modalities for the Pilot Token: short text captions from Qwen2.5- VL or geometry-aware features from VGGT, used as privileged training targets lamp, artwork, window.”To reduce truncation of the caption supervision, we increase the number of<|placeholder|>tokens allocated to the Pilot Token slot to 10 in this variant. We tokenize the generated caption with the same tokenizer as the main model, pad/truncate to length 10, and mask padding po- sitions. We then add a token-level cross-entropy loss that encourages the Pilot Token outputs (through the standard language-model head) to match the cap- tion tokens. The final training objective is the sum of the action cross-entropy and this caption-token cross-entropy. Importantly, Qwen2.5-VL is used only to produce privileged supervision during training; at inference time, the model does not require any caption generation and the interface remains unchanged. In the 3D variant, we use geometry-aware features extracted by a 3D founda- tion model, VGGT [6] without DPT head, as privileged targets. We keep VGGT frozen and encode the future image to obtain a global feature vector (e.g., from the model’s global/CLS representation). To align the Pilot Token with this tar- get, we pool the Pilot Token slot into a single vector and apply a lightweight linear projection to match the VGGT feature dimension. We then add L2 be- tween the projected Pilot representation and the VGGT target, in addition to the standard action cross-entropy. As with Text, VGGT is only used to pro- vide privileged supervision during training and is not required at inference. As shown in Table 3, the choice of Pilot supervision materially affects performance. Without explicit supervision (NaN), the Pilot Token may drift toward a weak or collapsed representation, since it is only indirectly optimized through the action loss. Text supervision compresses future visual content into a very short caption and can introduce additional noise due to imperfect descriptions and potential Supplementary Material3 (a) Ground-truth trajectory observation video collected during navigation. (b) World-model predicted trajectory observation video (Wan2.1-T2V-1.3B) (c) World-model predicted trajectory observation video (CogVideoX1.5-5B) Instruction: Walk down the stairs, turn right, and walk towards place with a rug. Wait near the bench and piano along the right side of the wall. Fig. S2:Plug-in World Model Video Predictions. (a) Ground-truth trajectory ob- servation video. (b) Video predicted by Wan2.1-T2V-1.3B. (c) Video predicted by CogVideoX1.5-5B.Although the world models can often synthesize objects explicitly mentioned in the instruction, their predictions tend to under-represent the diversity and dynamics of real scene evolution. In practice, real-world observations exhibit richer appearance variations and more pronounced spatial changes than what a pre-generated video can faithfully anticipate. As a result, such plug-in foresight is limited in capturing the full range of possible scene and spatial changes encountered during navigation. VLM hallucinations. The 3D (VGGT) target provides geometric priors, but un- der our current backbone and training data distribution, injecting such features can lead to domain mismatch and extra noise. Overall, in our VLN setting where visual cues are primary, supervising the Pilot Token with a vision-based latent target yields the strongest results. A.2 Plug-in World Model Baselines To instantiate the plug-in world model baselines in Table 4, we attach an off- the-shelf text-to-video generator at inference time and use its predicted frames to fill the Pilot Slot, replacing LatentPilot’s internal latent propagation. This isolates the effect of external foresight while keeping the VLN backbone and action decoder unchanged. Given an instructionI, we generate one instruction-conditioned video before navigation. During action selection at stept, we take the generated frame at indext+1as the predicted next-view image (ift+1exceeds the generated length, we reuse the last frame). We keep a reserved placeholder token at the Pilot Slot position in the LMM input template, and replace this token’s embedding with a feature extracted from the predicted next-view frame. The frame is encoded by the same frozen image encoder used in LatentPilot (same preprocessing), globally pooled into a single vector, and linearly projected to the LMM hidden 4 size. In this plug-in setting, the Pilot Slot is always provided by the world model at each step; the previous-step latent output is not fed into the next step. We evaluate two representative video world models: Wan2.1-T2V-1.3B [5] and CogVideoX1.5-5B [7]. For both models, we use the VLN instruction as the core prompt with a light egocentric prefix, generate one video per episode, and fix the random seed for deterministic decoding. We generate 81 frames per prompt and consume only the first 50 frames for navigation. Wan2.1 is run with 50 denoising steps and guidance scale 5.0, using the recommended scheduler configuration and default resolution. CogVideoX1.5 is run with 50 denoising steps and guidance scale 6.0. For both models, we enable standard memory- saving options (e.g., sequential offload and VAE tiling/slicing) when needed to prevent out-of-memory. For efficiency reporting, all numbers are measured on a single NVIDIA A100 with batch size 1. Time-per-action (T/Act) includes both the one-time video generation and the per-step action inference: we run 10 full trajectories end-to- end, sum the total wall-clock time, and divide by the total number of executed actions. Peak memory (Mem) is the maximum GPU memory allocated during the same runs, reported as the maximum over the 10 trajectories. A.3 Collapse Analysis Details To verify that Pilot Tokens do not suffer from latent collapse, we extract the propagated Pilot Tokens from evaluation rollouts on the R2R val unseen split and visualize their distribution in 2D, colored by the executed action at each step (LEFT,FWD,RIGHT;STOPis excluded due to its low frequency). We sample 400 trajectories from the R2R val unseen. SinceFWDdominates the natural ac- tion distribution, we balance the three classes to 500 samples each to avoid a visualization biased toward a single action. In the main paper (Fig. 6), we using a lightweight linear readout, following common probing practice for representation analysis [1,3]. Concretely, we z-score standardize the propagated Pilot Token features and fit a multinomial logistic regression classifier with an L2 penalty to predict the executed action label. We then treat the classifier’s pre-softmax 3-class decision scores (logits) as an action- evidence embedding, and project them to 2D with PCA [4] using whitening to rescale principal components to comparable variance for visualization. In the plot, we mark per-class centroids and overlay per-class2σcovariance ellipses. As shown in Fig. 6, Pilot Tokens form spatially separated, action-correlated clusters rather than collapsing to a single mode, indicating that the propagated latent remains informative and structured during rollout. Together with the main-paper PCA, this supports that action grounding and predictive supervision help maintain a non-degenerate latent space, enabling the Pilot Token to serve as a compact internal state rather than an empty placeholder. Supplementary Material5 Client wheeled robot Client quadruped robot LatentPilotsever User HTTP Stream SSH Stream Fig. S3: Real-world deployment architecture.LatentPilot is deployed with a uni- fied client–server pipeline on a wheeled robot (AgileX LiMO Pro) and a quadruped robot (Unitree Go2). Each robot runs a lightweight client that streams egocentric ob- servations to a remote LatentPilot server over HTTP and receives discrete action com- mands in return, while the user controls client start/stop and episode resets via SSH. Both embodiments expose the same action interface FWD, LEFT, RIGHT, STOP and execute actions with an open-loop velocity-for-duration controller. A.4 Real-world Experiments We evaluate LatentPilot on two real robot embodiments in indoor rooms: a wheeled AgileX LiMO Pro and a quadruped Unitree Go2. Both platforms follow the same client–server architecture (Fig. S3) to keep the deployment protocol consistent across embodiments. The LatentPilot server runs on a remote work- station equipped with an RTX 4090 GPU. Each robot runs a lightweight client program that handles sensing and low-level control. A user connects to the robot via SSH to start/stop the client and to reset an episode (client lifecycle control), while the robot and server communicate only through HTTP for observations and action commands. At every decision step, the client sends the current egocentric RGB obser- vation to the server through a single HTTP POST request (we use a multipart form containing a JPEG-compressed RGB image). The server replies with a JSON action list, using a unified discrete action interface shared by both robots: FWD, LEFT, RIGHT, STOP, encoded as 1,2,3,0. The client executes the returned actions sequentially, and only sends a new request when the action queue is empty. To make the interface embodiment-agnostic, both clients implement the same open-loop mapping from discrete actions to continuous controls. Concretely, each action is realized by commanding a constant velocity for a fixed duration: FWD uses a forward linear speedvford/vseconds to approximate a fixed forward 6 displacementd; LEFT/RIGHT use an in-place yaw rateωforθ/ωseconds to approximate a fixed heading changeθ; STOP sets all commanded velocities to zero. For Go2, we implement this controller using Unitree’s SportClient API (sending(v x ,v y ,ω z )commands). In our setup we usev=0.3m/s andd=0.25m for FWD, andω=0.5rad/s withθ=15 ◦ for turns, followed by a short settling delay after each action to reduce drift. The LiMO client follows the same action-to- duration mapping, but issues body velocity commands through its ROS2 control interface (linear and angular velocities), withvandωtuned to ensure stable wheeled motion on corridor floors. Importantly, no odometry, SLAM, or closed- loop waypoint tracking is required by LatentPilot; the evaluation uses purely the server-predicted discrete actions and the client’s deterministic open-loop execution. On the client side, we use a simple three-thread structure for robustness: a camera thread continuously updates the latest RGB frame (about 10 Hz), a planning thread sends the most recent frame to the server whenever replanning is triggered, and a control thread executes actions from the queue and triggers replanning when the queue is empty. This implementation matches the data flow in Fig. S3: observation images are streamed to the server over HTTP, and action commands are streamed back over HTTP, while SSH is only used for program lifecycle control by the user. B More Qualitative Results More examples are provided in the supplementary videos underReal-World Demofolder; we only show a subset here for brevity. In these real-world recordings, we can clearly observe lens distortion, noticeable camera shake, illumination changes, and environmental disturbances caused by passing pedestrians. These factors impose much stronger robustness requirements on navigation models. Despite these complex and noisy real-world conditions, our LatentPilot is still able to complete tasks reliably and smoothly. B.1 Real-World Deployment on a Wheeled Robot We deploy our LatentPilot on the wheeled platform (AgileX LiMO Pro) in real indoor corridors and visualize representative rollouts in both egocentric (first- person) and third-person views. LatentPilot reliably executes long, multi-step instructions (e.g., exiting doorways, making precise turns, passing furniture, and stopping at specified landmarks), showing strong real-world robustness. We at- tribute this to our action-conditionedPilot Tokenthat internalizes “dreaming ahead” from future-privileged supervision during training, while keeping infer- ence strictly causal and rollout-free. B.2 Real-World Deployment on Quadruped Robots We further evaluate LatentPilot on a legged quadruped (Unitree Go2) and present qualitative sequences from both first-person and third-person perspec- Supplementary Material7 (a)Collision with coffee table causes a fall (navigation failure). (b) Collision with bench leads to a fall (navigation failure). (c)Misaligned stair approach results in instability and a fall. Fig. S4:Qualitative Visualization of Fall Incidents in VLN-PE. (a–c) show represen- tative fall failures caused by collisions or unstable stair approach. In contrast, La- tentPilot’s internalized foresight makes the policy less likely to enter hazardous local states, effectively reducing collisions, unnecessary backtracking, and fall events by learn- ing action-conditioned visual dynamics. We also add lightweight rule-based anti-fall / anti-stuck safeguards as a final “safety fuse” for rare corner cases. This is a safety-only fallback for deployment to prevent collisions that could injure people tives. Despite embodiment and actuation differences, the same policy generalizes and remains executable, successfully following multi-step language instructions in cluttered corridor scenarios. This cross-embodiment transfer is enabled by our latent lookahead mechanism: the propagated Pilot Token captures environment– action dynamics and supports stable, non-myopic decision making without re- quiring any external world-model rollout at test time. B.3 Humanoid Navigation in Isaac VLN-PE Simulation Finally, we showcase humanoid navigation in the physically realistic VLN-PE simulator built on NVIDIA Isaac Lab (Unitree H1 embodiment), with qualitative visualizations from both ego-view and third-person renderings. Under continuous control and realistic locomotion imperfections (e.g., execution noise that can lead to deadlocks/falls), LatentPilot exhibits stable goal-directed behaviors such as robust walking, turning, and stopping. These results further support that our latent “dreaming ahead” improves safety and stability by learning anticipatory scene-aware dynamics during training. 8 C Social Impact Statement We consider potential physical harm caused by robot–human collisions in real- world deployment. To mitigate this risk, we add an obstacle-detection based anti-collision function on the real robot and limit the maximum speed to 1 m/s. Our project does not involve bystander privacy: we do not collect or release data containing personally identifiable information from passersby. References 1. Alain, G., Bengio, Y.: Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644 (2016) 2. Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., Lin, J.: Qwen2.5-vl technical report. CoRRabs/2502.13923(2025).https://doi.org/ 10.48550/ARXIV.2502.13923,https://doi.org/10.48550/arXiv.2502.13923 3. Hewitt, J., Manning, C.D.: A structural probe for finding syntax in word repre- sentations. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). p. 4129–4138 (2019) 4. Jolliffe, I.T.: Principal Component Analysis. Springer, 2nd edn. (2002) 5. Wan, T., Wang, A., Ai, B., Wen, B., Mao, C., Xie, C.W., Chen, D., Yu, F., Zhao, H., Yang, J., et al.: Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314 (2025) 6. Wang, J., Chen, M., Karaev, N., Vedaldi, A., Rupprecht, C., Novotný, D.: VGGT: visual geometry grounded transformer. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11– 15, 2025. p. 5294–5306. Computer Vision Foundation / IEEE (2025).https: //doi.org/10.1109/CVPR52734.2025.00499,https://openaccess.thecvf.com/ content/CVPR2025/html/Wang_VGGT_Visual_Geometry_Grounded_Transformer_ CVPR_2025_paper.html 7. Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang, Y., Hong, W., Zhang, X., Feng, G., et al.: Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072 (2024) Supplementary Material9 Instruction:walk away from closet, past sinks, turn right into bedroom, stop in front of doorway on right. Instruction:Keep going straight, pass by the three doors on the right, and stop by the elevator. Instruction:Continue straight until you reach the trash can, then turn right and stop next to the chair. Fig. S5:Representative LatentPilot rollouts on real wheeled and quadruped robots and in Isaac VLN-PE humanoid simulation.