Paper deep dive
VTInstructor: Visual Trajectory Prompting for Navigation Instruction Generation in Continuous Environments
Haolin Yang, Yuxing Long, Zihan Yang, Hao Dong
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/18/2026, 6:00:59 AM
Summary
The paper introduces VTInstructor, a framework for generating navigation instructions from ego-centric RGB video in continuous environments. It addresses the challenge of implicit trajectory geometry by using Event-Driven Trajectory Compression (EDTC) to select keyframes, Visual Trajectory Prompts (VTP) to overlay spatial cues (path, turn, goal), and a Visual Trajectory Modulator (VTMod) to inject these signals into a vision-language model. The method is trained via Supervised Fine-Tuning (SFT) and Group Relative Policy Optimisation (VT-GRPO). It achieves state-of-the-art results on R2R-CE and RxR-CE benchmarks, improving both automatic NLG metrics and downstream navigation success rates.
Entities (10)
Relation Signals (10)
VTInstructor → achievessotaon → RxR-CE
confidence 95% · On the challenging R2R-CE and RxR-CE Val Unseen benchmarks, VTInstructor sets a new state of the art
VTInstructor → achievessotaon → R2R-CE
confidence 95% · On the challenging R2R-CE and RxR-CE Val Unseen benchmarks, VTInstructor sets a new state of the art
VTInstructor → usescomponent → VTMod
confidence 95% · VTMod injects the resulting trajectory signals into the visual encoder
VTInstructor → usescomponent → EDTC
confidence 95% · Our key idea is to convert implicit trajectory geometry into explicit visual trajectory prompts: EDTC condenses long RGB trajectories...
VTInstructor → usescomponent → VTP
confidence 95% · VTP overlays path, turn, and goal cues onto these anchors
VTInstructor → usestrainingstrategy → VT-GRPO
confidence 95% · VT-GRPO further calibrates this spatial injection during training
VTMod → injectsinto → ViT_encoder
confidence 90% · VTMod injects the resulting trajectory signals into the visual encoder
VTInstructor → isbuilton → Qwen3-VL-8B
confidence 90% · We build on Qwen3-VL-8B [18].
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Navigation instruction generation from ego-centric RGB video in continuous environments is an important yet challenging task for human-robot interaction and scalable dataset construction. Prior instruction generators assume discrete viewpoint graphs with panoramic observations, where trajectory structure is explicit; in continuous environments, however, the agent receives only a dense RGB stream, making trajectory cues difficult to recover. We propose VTInstructor, the first VLN instruction generation framework for continuous environments. Our key idea is to convert implicit trajectory geometry into explicit visual trajectory prompts: EDTC condenses long RGB trajectories into navigation-critical keyframes, VTP overlays path, turn, and goal cues onto these anchors, VTMod injects the resulting trajectory signals into the visual encoder, and VT-GRPO further calibrates this spatial injection during training, all without requiring a navigation graph, pre-built map, or scene reconstruction. On the challenging R2R-CE and RxR-CE Val Unseen benchmarks, VTInstructor sets a new state of the art across all standard NLG metrics, surpassing the strongest baseline by +0.357 CIDEr and +0.109 CIDEr, respectively. Beyond automatic metrics, VTInstructor-generated instructions raise a frozen follower's success rate to 63.3%, a +14.7 percentage-point gain over the best competing instruction source, and provide consistent data augmentation gains of +3 SR points on downstream navigation tasks.
Tags
Links
- Source: https://arxiv.org/abs/2608.15284v1
- Canonical: https://arxiv.org/abs/2608.15284v1
Trouble viewing inline? Open PDF directly →
Full Text
49,865 characters extracted from source content.
Expand or collapse full text
VTInstructor: Visual Trajectory Prompting for Navigation Instruction Generation in Continuous Environments Haolin Yang ∗ CFCS, School of Computer Science, Peking University Beijing, China PrimeBot Beijing, China harley_yang@stu.pku.edu.cn Yuxing Long ∗ CFCS, School of Computer Science, Peking University Beijing, China PrimeBot Beijing, China longyuxing@stu.pku.edu.cn Zihan Yang ∗ CFCS, School of Computer Science, Peking University Beijing, China PrimeBot Beijing, China yangzihan@stu.pku.edu.cn Hao Dong † CFCS, School of Computer Science, Peking University Beijing, China PrimeBot Beijing, China hao.dong@pku.edu.cn Figure 1: Evolution of navigation instruction generation paradigms. (a)–(c) Prior speakers rely on discrete viewpoint graphs with panoramic images, optionally augmented by external knowledge or 3D representations. (d) VTInstructor generates instructions from ego-centric RGB video in continuous environments, requiring no navigation graph or 3D reconstruction at inference. Abstract Navigation instruction generation from ego-centric RGB video in continuous environments is an important yet challenging task for human–robot interaction and scalable dataset construction. Prior instruction generators assume discrete viewpoint graphs with panoramic observations, where trajectory structure is explicit; in continuous environments, however, the agent receives only a dense RGB stream, making trajectory cues difficult to recover. We propose VTInstructor, the first VLN instruction generation framework for ∗ These authors contributed equally to this work. † Corresponding author. This work is licensed under a Creative Commons Attribution 4.0 International License. M ’26, Rio de Janeiro, Brazil © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2213-4/2026/11 https://doi.org/10.1145/3767308.3836246 continuous environments. Our key idea is to convert implicit tra- jectory geometry into explicit visual trajectory prompts: EDTC condenses long RGB trajectories into navigation-critical keyframes, VTP overlays path, turn, and goal cues onto these anchors, VTMod injects the resulting trajectory signals into the visual encoder, and VT-GRPO further calibrates this spatial injection during training, all without requiring a navigation graph, pre-built map, or scene reconstruction. On the challenging R2R-CE and RxR-CE Val Unseen benchmarks, VTInstructor sets a new state of the art across all standard NLG metrics, surpassing the strongest baseline by +0.357 CIDEr and +0.109 CIDEr, respectively. Beyond automatic metrics, VTInstructor-generated instructions raise a frozen follower’s suc- cess rate to 63.3%, a +14.7 percentage-point gain over the best competing instruction source, and provide consistent data augmen- tation gains of +3 SR points on downstream navigation tasks. arXiv:2608.15284v1 [cs.RO] 15 Aug 2026 M ’26, November 10–14, 2026, Rio de Janeiro, BrazilHaolin Yang, Yuxing Long, Zihan Yang, and Hao Dong CCS Concepts • Computing methodologies→Vision for robotics; Natural language generation; Robotic planning. Keywords visual trajectory, multimodal robotics, computer vision ACM Reference Format: Haolin Yang, Yuxing Long, Zihan Yang, and Hao Dong. 2026. VTInstruc- tor: Visual Trajectory Prompting for Navigation Instruction Generation in Continuous Environments. In Proceedings of the 34th ACM International Con- ference on Multimedia (M ’26), November 10–14, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 9 pages. https://doi.org/10.1145/3767308.3836246 1 Introduction Navigation instruction generation (producing natural-language de- scriptions of traversed trajectories) is a fundamental task for human- robot interaction and the scalable construction of training data for embodied navigation models. Existing speaker models [5,7,11] have made substantial progress in discrete graph-based settings. Fig- ure 1 summarizes the evolution of these paradigms, from panoramic speakers to knowledge-augmented and 3D-enhanced variants, all of which remain rooted in discrete viewpoint graphs. By contrast, nav- igation instruction generation in continuous environments remains largely unexplored. Why has this setting remained unexplored? In discrete VLN environments, trajectories unfold over topological graphs whose nodes are associated with panoramic observations, making view- point relations explicit and trajectory structure easy to infer. In continuous environments, however, the agent receives only a dense ego-centric RGB stream, and adjacent frames may look highly simi- lar even when they correspond to different motions or spatial states. This shift introduces two challenges. First, existing graph-based speaker models cannot be transferred directly to continuous envi- ronments. Second, trajectory cues such as path direction, turning behaviour, and goal progress are no longer explicit: recovering them from subtle inter-frame changes is itself a fine-grained spatial- intelligence problem, and this is precisely the capability that current VLN agents and MLLMs have been shown to lack [22]. Continuous environments therefore place instruction generation squarely in the regime where these models are weakest, so directional information is lost and instructions degenerate into generic, spatially imprecise descriptions—unless the trajectory geometry is supplied explicitly. To address these challenges, we propose VTInstructor, a uni- fied navigation instruction generation framework for continuous environments that processes the entire pipeline within a single vision-language backbone. Our central idea is to convert trajectory geometry into explicit visual trajectory prompts on ego-centric views, so that the model can perceive path structure directly instead of inferring it only from dense RGB streams. VTInstructor com- prises three core components. (1) Event-driven trajectory compres- sion (EDTC) first condenses long RGB trajectories into navigation- critical keyframes determined by the action sequence, providing vi- sual anchors for subsequent prompting. (2) Visual Trajectory Prompt (VTP) rendering then overlays path, turn, and goal cues onto these keyframes, while GPT-based quality filtering (QF) retains only spa- tially reliable instruction-keyframe pairs for training. (3) Visual Trajectory Modulator (VTMod) injection feeds the resulting trajec- tory signals directly into the ViT encoder, strengthening spatial perception beyond what raw appearance alone can provide. On top of supervised fine-tuning, we further introduce VT-GRPO, which uses reinforcement learning to selectively calibrate the VTMod gates and explicitly refine how trajectory information is injected into the model. We evaluate VTInstructor on the R2R-CE and RxR-CE Val Un- seen benchmarks, where it achieves state-of-the-art performance across all standard NLG metrics (BLEU, METEOR, ROUGE-L, CIDEr, SPICE), outperforming the strongest baseline. Downstream navi- gation experiments further show that follower agents guided by VTInstructor-generated instructions achieve higher success rate than those using competing instruction sources, and human evalu- ators consistently rate VTInstructor instructions higher on direc- tional accuracy and overall followability. Our contributions are: • The first VLN instruction generation framework for con- tinuous environments. VTInstructor generates instructions from ego-centric RGB trajectories paired with action sequences (as defined in Section 3.1); the model itself receives only RGB frames as visual input, without navigation graphs, pre-built maps, or scene reconstruction. •A visual trajectory prompting framework for explicit spa- tial grounding. We convert implicit trajectory geometry in dense RGB streams into explicit spatial cues through EDTC for navigation-critical keyframe selection, VTP for path/turn/goal prompting on these anchors, VTMod for trajectory-aware visual encoding, and VT-GRPO for reward-driven calibration of spatial signal injection. •State-of-the-art performance with practical utility. VTIn- structor achieves state-of-the-art results on the R2R-CE and RxR- CE Val Unseen benchmarks, surpassing the strongest baseline by +0.357 and +0.109 CIDEr, respectively, improving frozen-follower success by 14.7 percentage points, and delivering +3 SR-point data augmentation gains on both benchmarks. 2 Related Work 2.1 Navigation Instruction Generation Vision-and-Language Navigation (VLN) requires an agent to fol- low natural-language instructions in indoor environments [1,2, 14]. Complementing instruction following, instruction generation (the speaker side) is critical for data augmentation and human– robot communication. Speaker-Follower [5] first trains an LSTM speaker for data augmentation; subsequent work improves gener- ation through speaker–follower cycle consistency [19] and multi- task joint training [21]. Another line enriches the speaker with external knowledge or landmark grounding: SAS [6] and KEFA [25] introduce object-spatial attention and commonsense alignment, while landmark-based instruction generation methods such as Less is More [20] abstracts the trajectory into detected landmarks and generates instructions via a text-to-text model. More recently, C- Instructor [8] adopts chain-of-thought prompting with a multi- modal LLM, and MapInstructor [4] and BEVInstructor [3] leverage top-down maps or BEV representations for global spatial reasoning. Despite their diversity, existing instruction-generation methods VTInstructorMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil are developed in discrete navigation graphs, where each state is typically represented by privileged panoramic observations—often instantiated in R2R-style VLN as a set of 36 discretized views per viewpoint—rather than raw first-person continuous perception. R2R-CE [9] and RxR-CE [10] extend VLN benchmarks to the contin- uous Habitat simulator, yet speaker models for this setting remain unexplored. VTInstructor is the first VLN instruction generation framework for continuous environments, taking ego-centric RGB image with rendered VTP as input without graph topology privi- lege. 2.2 Visual Prompting Visual prompting augments input images with task-relevant anno- tations to steer model attention without modifying model weights. In the general vision-language domain, coloured circles [16], num- bered markers [23], and iteratively overlaid arrows and keypoints [13] have been used for referring expression comprehension, visual grounding, and action prediction. In robotic manipulation, TraceVLA [27] overlays end-effector trajectory traces on the current observa- tion to enhance spatial-temporal awareness of VLA policies, and Robotic Visual Instruction [12] annotates images with visual cues to guide manipulation actions. Despite the growing adoption of visual prompting in perception and manipulation, it has not been explored for navigation instruction generation. VTInstructor is the first to introduce Visual Trajectory Prompts (VTP) into this task, with two key designs. 3 Method As shown in Figure 2, VTInstructor converts implicit trajectory geometry in dense ego-centric RGB streams into explicit spatial cues through four components: event-driven trajectory compression (§3.3), VTP rendering and data curation (§3.4), VTMod injection (§3.5), and VT-GRPO calibration (§3.6). 3.1 Problem Formulation An agent navigates a 3D scene by executing atomic actions from a discrete action spaceAof four primitives: forward+0.25m, turn left−30 ◦ , turn right+30 ◦ , and stop. At each timestep푡, the agent receives an ego-centric RGB observation푓 푡 ∈ R 퐻×푊×3 . A trajectory of length푇 is thus: 휏= (푓 1 ,푎 1 ), (푓 2 ,푎 2 ), . . ., (푓 푇 ,푎 푇 ) .(1) The Navigation Instruction Generation (NIG) task requires produc- ing a natural-language instruction푦=(푤 1 , . . .,푤 퐿 )that describes휏 faithfully enough for a human or autonomous follower to reproduce the route: 푃(푦 | 휏)= 퐿 Ö 푙=1 푃 ( 푤 푙 | 푤 <푙 , 휏 ) .(2) 3.2 Preliminaries Backbone Model. We build on Qwen3-VL-8B [18]. Its ViT-based visual encoder partitions each input image into non-overlapping patches forming a spatial token grid of shape(ℎ 푝 ×푤 푝 )with hid- den dimension푑 vit ; critically, each patch token푖retains a fixed spatial position(푟 푖 ,푐 푖 )throughout all layers, enabling the pixel- accurate one-to-one correspondence that VTMod exploits. For푁 keyframes, each frame is encoded independently and projected to the language model’s hidden dimension by a shared MLP connector, with per-image 2D RoPE encodings preserving spatial and temporal order. On the language side, the system prompt and interleaved action snippets are tokenized into decoder input embeddings and combined with the projected visual tokens in a single multimodal sequence. The decoder then generates the navigation instruction autoregressively. Multimodal input format. The model receives a single inter- leaved sequence: a task-specific system promptPis followed by alternating keyframe images (I) and textual action snippets: P, 퐼 1 , [Action 1: 푑 1 ], 퐼 2 , [Action 2: 푑 2 ], . . . where each푑 푘 is a natural-language description of the physical displacement or rotation at that step (e.g. “go forward 1.25 m”, “turn left 90 ◦ ”). This interleaved layout preserves the temporal alignment between visual observations and physical actions, allowing the de- coder to attend to the relevant image–action pair at each generation step. 3.3 Event-Driven Trajectory Compression Level 1: Action Run-Length Encoding (RLE). Let the raw ac- tion sequence be a= (푎 1 ,푎 2 , . . .,푎 푇 )with푎 푡 ∈ A. RLE merges consecutive identical actions into segments: 푠 푘 =(type 푘 , 푛 푘 ),where 푎 푡 = type 푘 for 푛 푘 consecutive steps. (3) Each segment maps to a physical displacement (푛 푘 ×0.25mfor- ward) or rotation (푛 푘 ×30 ◦ ), producing the compressed sequence S= (푠 1 , . . .,푠 퐾 )with퐾 ≪ 푇. RLE is lossless with respect to nav- igational semantics: every physical displacement and rotation is exactly preserved in S. To avoid overly long straight-line events, we further split any forward segment whose displacement exceeds 4.0 m (i.e., 푛 푘 > 16) into two shorter forward segments of approxi- mately equal length. This preserves the total displacement while preventing excessively long forward motion from dominating a single event representation. Level 2: Small-Step Combo Grouping. A segment푠 푘 is consid- ered small-step if it corresponds to either a short forward displace- ment of 0.25 m or 0.5 m (푛 푘 ∈ 1,2) or a single-step rotation of 30 ◦ (푛 푘 =1). Consecutive small-step segments are merged into a combo event 푒 푐 =(푠 푘 ,푠 푘+1 , . . .,푠 푘+푀−1 ), with푀 ≤6. This grouping captures fine-grained turn-and-advance manoeuvres as a single semantic unit, preventing them from be- ing fragmented into isolated snippets that may appear spatially incoherent to the language model. Level 3: Dynamic Priority-based Compression (DPC). When the number of retained frames after Levels 1–2 still exceeds퐹 max = 32, DPC selects the most informative keyframes by a joint visual– geometric priority score. For each adjacent event-frame pair(푖, 푗): 퐽(푖, 푗)= 휆 vis Δ vis (푖, 푗)+ 휆 geo Δ geo (푖, 푗),(4) whereΔ vis measures frame-to-frame visual change in a frozen per- ceptual feature space, andΔ geo aggregates the cumulative displace- ment and rotation magnitude between event boundaries. A high 퐽score indicates a visually or geometrically significant transition that should be retained; event frames at boundaries with the lowest 퐽scores are progressively removed until|F| ≤ 퐹 max , while always M ’26, November 10–14, 2026, Rio de Janeiro, BrazilHaolin Yang, Yuxing Long, Zihan Yang, and Hao Dong Qwen3 LLM Decoder MLP Projector Layer 7 ... ViTEncoder VTMod Output Conv 64 Channels, Stride 2 Conv 128 Channels, Stride 2 Conv 256 Channels, Stride 2 Level 1: Action Run-length Encoding Level 2: Combo Grouping x 3 Level 3: DPC Compact Keyframes (n ≤ 32) Tframes → Ksegments Ksegments → ksegments ... + 품 Layer 28 System Prompt Prompt Token Visual Token Nav Act Token Navigation Instruction Event-Driven Trajectory Compression Visual Trajectory Prompt Rendering Qwen3 LLM Decoder 1x1 Conv Adp Pool ViTEncoder VTMod 품 Qwen3 LLM Decoder ViTEncoder VTMod 품 G=8 Sampled Responses Reward Function x 0.25 x 0.25 x 0.25 x 0.20 x 0.05 Gate Contrastive Loss GRPO Loss Stage 2: VT-GRPO Stage 1: SFT Two-Stage Training VTModStructure Ribbon Mask (C0) Endpoint Mask (C2) Arrow Mask (C1) VTP Overlay →3-ch Binary Mask 푴∈ퟎ,ퟏ ퟑ×푯×푾 ROTATION ARROW PATH RIBBON Quality Filtering (QF) Quality Filtering (QF) Figure 2: Overview of VTInstructor. Event-driven compression turns a raw ego-centric RGB trajectory into a compact keyframe set; each keyframe carries a VTP overlay (path ribbon / rotation arrow / endpoint marker) encoded as a 3-channel binary mask. The VTP Encoder maps the mask to patch-aligned features that are injected into ViT layer 7 by per-token spatial modulation. Training proceeds in two stages: SFT, then VT-GRPO. preserving the initial frame and the current observation frame. The final retained frame set is: F=푓 1 , 푓 푇 ∪푓 푒 1 , 푓 푒 2 , . . ., 푓 푒 푁 ′ ,(5) where푓 1 is the initial observation,푓 푇 is the current observation frame,푒 푖 denotes the index of the푖-th retained event boundary, and 푁 ′ is the number of retained event frames. 3.4 Visual Trajectory Prompt Rendering and Data Curation Overlay components. For each retained keyframe, a structured VTP is rendered in the corresponding ego-centric view to visualize the local trajectory state. Three complementary components pro- vide exhaustive spatial coverage across the canonical navigation states (advancing, turning, approaching goal): •Path ribbon: a colour ribbon tracing the upcoming route in the current view. Because the raw path is a jagged polyline induced by 0.25 m atomic forward steps, we smooth it in 2D through densified interpolation, Chaikin corner cutting, and two rounds of Gaussian smoothing. A final occlusion check is then applied to prevent smoothed points from drifting behind walls. Used when the current event is not a pure rotation; degrades gracefully to a short stub when fully occluded. • Rotation arrow: a curved left/right arc annotated with the turn angle in degrees. Used when the current event is a pure rotation. •Endpoint marker: a landmark flag indicating the goal location in the current view. Rendered near trajectory end-frames when the goal is unoccluded. Mask representation. Each VTP overlay is stored as a three- channel binary semantic mask M∈ 0,1 3×퐻×푊 , where channel 퐶 0 encodes the ribbon,퐶 1 the rotation arrow, and퐶 2 the endpoint marker. Representing overlays as independent binary channels eliminates colour ambiguity and lets the VTP Encoder learn channel- specific spatial patterns, a factorisation that would be conflated in a mixed-colour image. This mask is the direct input to the VTP Encoder; the coloured PNG visualisation is an artefact used only for qualitative inspection. Task-specific prompting strategy. We design separate system prompts for R2R-CE and RxR-CE to match their distinct annota- tion styles: the R2R-CE prompt targets concise instructions (15–45 words) emphasising landmark references and a precise stop loca- tion, while the RxR-CE prompt elicits richer step-by-step narrations (50–120 words) with explicit orientation and transition cues. Both prompts share a critical use-but-don’t-mention constraint: the model is informed that the input images contain grounded navigation cues (path ribbons, turn indicators, goal markers) and is instructed to leverage these overlays for path inference, yet is explicitly pro- hibited from referencing them in the generated instruction. This design ensures VTP functions as an implicit geometric prior that improves spatial grounding without leaking rendering artefacts into the output text. GPT-based quality filtering. Instruction candidates are scored by GPT against a rubric covering four dimensions: (i) directional accuracy: do described turns and path shape match the VTP over- lay?; (i) landmark specificity: are salient visual features referenced?; (i) distance plausibility: do distance expressions correspond to ac- tual trajectory length?; and (iv) linguistic fluency. Only samples exceeding quality threshold휏 GPT are retained, removing halluci- nated or spatially imprecise instructions and ensuring the training corpus maintains consistent geometric fidelity. 3.5 Visual Trajectory Modulator Design VTP Encoder. Let M∈ 0,1 3×퐻×푊 denote the binary seman- tic mask with F (0) = M. The VTP Encoder applies three stride-2 VTInstructorMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil convolutional blocks, each comprising a convolution, group nor- malisation, and GELU activation, with output channel dimensions 퐶 푙 ∈ 64, 128, 256: F (푙) = GELU GN Conv (푙) 푠=2 F (푙−1) , 푙= 1, 2, 3.(6) The hierarchical stride-2 design progressively expands the receptive field so that each spatial position encodes not only its local overlay pixel but also the surrounding trajectory context. A subsequent 1×1 convolution projects to dimension푑 푣 =384, and adaptive average pooling aligns the spatial resolution to the ViT patch grid, yielding: V= AdpPool Conv 1×1 F (3) , (ℎ 푝 ,푤 푝 ) ∈ R ℎ 푝 푤 푝 ×푑 푣 ,(7) where each row v 푖 ∈ R 푑 푣 is in one-to-one spatial correspondence with the 푖-th ViT patch token. Spatial modulation injection. At ViT layer푙 ∗ =7, the hidden state h 푖 ∈ R 푑 vit of patch token 푖 is updated as: h ′ 푖 = h 푖 + g⊙ LN(Wv 푖 ),(8) where W∈ R 푑 vit ×푑 푣 is a learnable linear projection,LNdenotes layer normalisation,⊙denotes element-wise multiplication, and g∈ R 푑 vit is a channel-wise learnable gate. We set푙 ∗ =7 as the default injection site (ablated in Table 4). The gate g is initialised to g 0 ≈0, implementing a lazy activation: the pre-trained ViT gradient landscape is preserved at training onset, preventing catastrophic forgetting of visual priors while the backbone adapts to the new task during early SFT. 3.6 Training Strategy Two-stage rationale. The two stages address complementary lim- itations. SFT teaches the task distribution (what high-quality navi- gation instructions conditioned on VTP-annotated keyframes look like) via maximum-likelihood estimation on GPT-filtered data. How- ever, MLE treats all reference tokens equally regardless of naviga- tional informativeness, and cannot directly optimise full-sequence metrics such as CIDEr or METEOR that are evaluated at test time. VT-GRPO then uses NLG reward signals to calibrate the gate g’s injection strength while continuing to update the LLM backbone, sharpening which spatial channels of the VTP are amplified for reward-relevant generation. Stage 1: Supervised Fine-Tuning (SFT). During SFT, all modules are trainable, including the ViT backbone, VTP Encoder, VTMod, MLP connector, and Qwen3 decoder. We use a dual learning-rate scheme: a higher rate휂 new for newly initialised VTP/VTMod pa- rameters and a lower rate휂 backbone for the pretrained Qwen3-VL backbone. The SFT objective is standard next-token prediction cross-entropy over ground-truth instruction tokens. Stage 2: VT-GRPO. Group Relative Policy Optimisation (GRPO) [15] is applied to refine instruction quality using NLG metrics as reward. For each input,퐺=8 candidate completions are sampled. The reward for completion푦is a weighted combination of automatic metrics: 푟(푦)=푤 퐵1 B-1(푦)+푤 퐵4 B-4(푦)+푤 푀 METEOR(푦) +푤 푅 ROUGE-L(푦)+푤 퐶 CIDEr(푦), (9) with푤 퐵−1 = 푤 퐵−4 = 푤 푀 =0.25,푤 푅 =0.20,푤 퐶 =0.05. The weights reflect the complementary coverage of the metrics: B-1 (BLEU-1) captures unigram precision, B-4 (BLEU-4) rewards multi- word phrase fidelity, METEOR additionally accounts for synonym overlap, and ROUGE-L measures longest-common-subsequence structural similarity; CIDEr is down-weighted because its large ab- solute scale would otherwise dominate the composite reward. A KL penalty훽=0.04 and clip ratio휀=0.2 regularise the policy update. LetL GRPO denote the corresponding GRPO objective induced by these rewards and regularizers. Gate contrastive loss. BecauseL GRPO averages over all퐺comple- tions weighted by their respective advantages, positive and negative signals partially cancel, leaving the gate g with a diffuse gradient that is insufficient for precise calibration. We therefore introduce a contrastive loss on the best (푦 + ) and worst (푦 − ) completions within each group: L gate =− log 휎 ̄ ℓ(푦 + )− ̄ ℓ(푦 − ) 휏 푔 ,(10) where ̄ ℓ(푦)is the mean per-token log-probability and휏 푔 =1.0. Un- like the group-averaged GRPO signal, this loss provides a focused contrastive gradient that directly pushes g to amplify VTP chan- nels correlated with higher-reward generations and suppress those correlated with lower-reward ones. The total training objective is: L total =L GRPO + 훼 gate L gate , 훼 gate = 0.05.(11) During VT-GRPO, the ViT backbone, VTP Encoder, and modulator projection layers are frozen; the LLM backbone and the gate vector g remain trainable. Detailed hyperparameters for both stages are reported in §4.1. 4 Experiments 4.1 Implementation Details Hardware. All experiments are conducted on 8×NVIDIA H200 GPUs with DeepSpeed ZeRO-2 parallelism. Training data. Training data consists of GPT-quality-filtered VTP- annotated trajectories (score≥6 on a 10-point rubric) drawn exclu- sively from the Train splits of R2R-CE and RxR-CE. Input representation. Each observation is represented as a single wide-aspect egocentric image, with VTP overlays rendered only in the central region. Trajectories are compressed to at most퐹 max =32 keyframes using the event-driven pipeline (§3.3). SFT hyperparameters. 3 epochs; batch size 1; gradient accumu- lation 12; base LR 3×10 −5 (backbone), VTMod LR 5×10 −4 ; 10% linear warm-up. Training takes approximately 9 hours on the above hardware. VT-GRPO hyperparameters. 1 epoch; LR 10 −6 ; gradient accumu- lation 4; group size퐺=8; KL훽=0.04; clip휀=0.2; top-푝=0.9; gate-contrastive weight훼 gate =0.05. Training takes approximately 30 hours on the above hardware. 4.2 Evaluation Benchmarks and Metrics R2R-CE [9] extends the Room-to-Room benchmark [1] to the continuous-action Habitat simulator with photorealistic Matter- port3D (MP3D) scenes. The agent navigates via four atomic actions and receives only a raw ego-centric RGB stream, with no navigation graph or pre-built map available. The Val Unseen split covers envi- ronments entirely held out from training, providing a stringent test of instruction generalisation to novel scenes. Since VTInstructor is M ’26, November 10–14, 2026, Rio de Janeiro, BrazilHaolin Yang, Yuxing Long, Zihan Yang, and Hao Dong Table 1: Navigation instruction generation results on R2R-CE and RxR-CE Val Unseen. R2R-CE Val UnseenRxR-CE Val Unseen MethodBLEU-1 BLEU-4 METEOR ROUGE-L CIDEr SPICE BLEU-1 BLEU-4 METEOR ROUGE-L CIDEr SPICE Proprietary models Qwen3.5-Plus0.5570.1310.2230.3870.1370.1800.5270.0800.1570.2540.0330.156 GPT-5.40.4820.0750.1970.3480.0780.1440.4930.0560.1460.2410.0290.143 Gemini-3.1-Pro-Preview0.6310.1660.2090.3920.2030.1670.3930.0730.1280.2420.0290.135 xAI Grok 40.4860.0840.2070.3540.0730.1600.4300.0490.1810.2490.0140.144 Claude Opus 4.60.4840.0840.1900.3370.0680.1190.3670.0430.1760.2230.0050.123 Open-source models Qwen3.5-397B-A17B0.5250.1130.2160.3820.1200.1740.4220.0590.1280.2490.0270.142 Kimi-K2.50.4780.0880.1950.3570.0980.1460.4320.0710.1950.2590.0110.145 LLaVA-Video-7B-Qwen20.5060.1000.1460.3170.1080.1090.0640.0120.0630.1690.0020.074 Qwen3-VL-8B-Instruct0.5240.0850.1750.3340.1470.1370.3660.0530.1190.2300.0220.113 Qwen3.5-9B0.5790.1210.1940.3730.1810.1510.5150.0770.1710.2590.0270.164 GLM-4.1V-9B-Thinking0.4900.0800.1560.3210.1300.1150.0740.0140.0540.1440.0030.064 GLM-4.6V0.5560.1200.1710.3480.1680.1220.4890.0650.1430.2420.0270.145 Ours Qwen3-VL-8B (SFT-only)0.7200.2820.2320.4850.4840.2010.6300.2010.2100.3570.0600.162 VTInstructor0.7650.3200.2630.5110.560 0.245 0.7740.3080.2650.4310.142 0.206 Table 2: Cross-setting comparison with prior instruction generation methods. Prior methods use discretized viewpoint graphs with privileged 36-view observations; VTInstructor uses only ego-centric RGB in continuous environments. We present this as a reference point for instruction quality, not a strictly controlled comparison. R2R / R2R-CE Val UnseenRxR / RxR-CE Val Unseen MethodBLEU-1 BLEU-4 METEOR ROUGE-L CIDEr SPICE BLEU-1 BLEU-4 METEOR ROUGE-L CIDEr SPICE Prior methods on R2R / RxR (discretized viewpoint-graph environments) BT-speaker [5]0.6580.2500.2090.4400.3910.1780.3110.0640.1600.2430.022– EDrop-speaker [17]0.6600.2600.2150.4550.4130.1840.2940.0560.1430.2530.027– C-speaker [19]0.6790.2540.2260.4560.4010.1830.2660.0420.1130.2060.016– Lana [21]0.6890.2600.2190.4630.4190.1940.3140.1110.1260.2670.045– Lana+ [21]0.6920.2620.2200.4620.4240.1990.3080.1150.1250.2680.046– C-Instructor [8]0.7110.2630.2370.4690.4500.2110.3660.1240.1810.3010.053– BEVInstructor [3]0.6990.2640.2300.4670.4490.2080.3510.1120.1740.2920.043– MapInstructor [4]0.7150.2850.2340.4850.4900.2090.4110.1590.1910.3230.057– Ours on R2R-CE / RxR-CE (continuous environments) VTInstructor0.7650.3200.2630.5110.560 0.245 0.7740.3080.2650.4310.142 0.206 trained exclusively on the Train splits of R2R-CE and RxR-CE, nei- ther the environments nor the reference instructions in Val Unseen have been seen during training. RxR-CE [10] adapts the Room-across-Room benchmark to the same Habitat continuous setting. RxR instructions are substantially longer and more spatially detailed than R2R (averaging over 70 words per instruction), with fine-grained descriptions of turn angles, landmark sequences, and relative distances. This verbosity and spatial precision make RxR-CE a more demanding benchmark for evaluating instruction generation quality. We report results on the English Val Unseen split. Instruction metrics. We report BLEU-1/4, METEOR, ROUGE-L, CIDEr, and SPICE; higher is better for all. These metrics collectively capture푛-gram precision, recall, synonym overlap, sequential simi- larity, and semantic propositional content. Navigation metrics. For downstream experiments (§4.5), we re- port the standard VLN-CE metrics [1]: Success Rate (SR), Oracle Success Rate (OSR), SPL, and Navigation Error (NE, in metres; lower is better), all with a 3 m success threshold. 4.3 Main Results Comparison with existing models (Table 1). All baseline mod- els are evaluated under a few-shot setting: each prompt includes a small number of high-quality reference instructions sampled from the training split that exemplify the target annotation style of R2R- CE or RxR-CE, ensuring that every model receives sufficient task context before generation. VTInstructor surpasses all baselines by a large margin across both benchmarks. On R2R-CE, VTInstruc- tor achieves a CIDEr of 0.560, outperforming the strongest model Gemini-3.1-Pro-Preview (0.203) by +0.357; BLEU-4 reaches 0.320 versus 0.166 for Gemini, an improvement of over 90%. On RxR- CE, the gap widens further: most baselines obtain CIDEr below 0.03, while VTInstructor reaches 0.142. Notably, LLaVA-Video-7B and GLM-4.1V-9B-Thinking nearly collapse on RxR-CE (BLEU-1 of 0.064 and 0.074, respectively), suggesting that these models struggle with the longer observation sequences required by RxR trajectories. Notably, proprietary models do not consistently outperform smaller open-source ones—GPT-5.4 (CIDEr 0.078) falls below Qwen3.5-9B (0.181) on R2R-CE—indicating that navigation instruction gener- ation cannot be solved by model scale alone and benefits from task-specific training with spatial grounding. Cross-setting comparison (Table 2). Operating in the continu- ous setting, VTInstructor attains scores comparable to or higher than prior discrete-setting methods that have access to panoramic images and pre-built navigation graphs. Because the two settings differ in observation and action spaces, we treat Table 2 as a ref- erence point for instruction quality rather than a head-to-head controlled comparison. On R2R, VTInstructor surpasses the previ- ous best MapInstructor in BLEU-4 (0.320 vs. 0.285), METEOR (0.263 VTInstructorMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Table 3: Component-wise ablation on R2R-CE Val Unseen. # VTMod EDTC QF BLEU-4 METEOR ROUGE-L SPICE 10.2820.2320.4850.201 2✓0.2860.2410.4920.220 3✓ ✓0.2880.2420.4930.223 4 ✓0.3020.2500.5000.232 5 ✓ ✓ 0.3080.2560.5040.238 Table 4: Design ablations on R2R-CE Val Unseen. ConfigurationBLEU-4 METEOR ROUGE-L SPICE (a) EDTC: frame sampling Stride-4 (avg. 13.0 fr.)0.738 † 0.2400.4900.220 EDTC (avg. 10.5 fr.)0.743 † 0.2420.4930.223 (b) VTMod: injection layer Layer 70.2990.2480.4970.224 Layers 7, 15, 230.2950.2440.4910.221 (c) VTMod: injection design Cross-attention0.2950.2440.4940.222 Gated projection0.3000.2470.4970.226 (d) VT-GRPO: RL strategy Pure GRPO0.3140.2600.5080.240 VT-GRPO0.3200.2630.5110.245 † BLEU-1 reported for frame sampling comparison. vs. 0.234), and CIDEr (0.560 vs. 0.490). The advantage is even more pronounced on RxR, where VTInstructor achieves BLEU-4 of 0.308 versus MapInstructor’s 0.159 and CIDEr of 0.142 versus 0.057, sug- gesting that VTP-based spatial grounding is particularly effective for the longer, more spatially detailed instructions characteristic of RxR. R2R-CE/RxR-CE and R2R/RxR share identical paths and reference instructions, and we follow the per-sub-path aggregation protocol of C-Instructor [8], so the metrics are comparable. Figure 3 provides qualitative examples illustrating how VTInstructor gen- erates spatially accurate instructions while the strongest baseline produces directional errors and hallucinated landmarks. 4.4 Ablation Study We conduct ablation experiments on R2R-CE Val Unseen, whose shorter reference instructions make component contributions easier to isolate. Component-wise ablation. Table 3 progressively adds VTMod, EDTC, and quality filtering (QF). The base SFT model without any of these components (Row 1) achieves BLEU-4 of 0.282 and SPICE of 0.201. Adding VTMod yields the largest single improvement: comparing Row 3 to Row 5, BLEU-4 jumps from 0.288 to 0.308 (+0.020) and SPICE from 0.223 to 0.238 (+0.015), confirming that VTMod is the core contributor by injecting trajectory-grounded spatial cues into the vision encoder. The full model (Row 5) achieves BLEU-4 of 0.308, METEOR of 0.256, ROUGE-L of 0.504, and SPICE of 0.238, representing cumulative gains of +0.026, +0.024, +0.019, and +0.037 over the base model. These results are obtained solely with supervised fine-tuning (SFT), without applying the subsequent VT-GRPO optimization process. Design ablations. As shown in Table 4, EDTC reduces the average frame count from 13.0 to 10.5 while improving all metrics over stride-4 sampling, confirming that event-driven selection retains more informative keyframes. Injecting VTMod at a single early layer (layer 7) outperforms distributing it across layers 7/15/23 (BLEU-4 0.299 vs. 0.295), as subsequent ViT layers can jointly refine the fused Table 5: Downstream navigation performance on R2R-CE Val Unseen with a frozen CorrectNav follower. Instruction SourceSR↑ OSR↑ SPL↑ NE↓ Human Annotations Official instruction61.667.253.34.53 Proprietary models Qwen3.5-Plus48.661.839.05.70 GPT-5.435.951.428.66.14 Gemini-3.1-Pro-Preview48.556.439.95.59 xAI Grok 445.564.833.46.18 Claude Opus 4.621.748.815.29.56 Open-source models Qwen3.5-397B-A17B45.263.933.36.15 Kimi-K2.543.958.633.66.49 LLaVA-Video-7B-Qwen230.244.824.37.91 Qwen3-VL-8B-Instruct33.446.825.97.09 Qwen3.5-9B36.245.229.16.82 GLM-4.1V-9B-Thinking31.142.924.47.37 GLM-4.6V29.940.624.07.11 Ours VTInstructor63.3 70.0 52.7 4.47 representation without redundant modulation. Gated patchwise projection outperforms cross-attention injection (BLEU-4 0.300 vs. 0.295), preserving spatial locality of VTP cues that cross-attention would dilute. Finally, VT-GRPO improves over pure GRPO (BLEU- 4 0.314→0.320, SPICE 0.240→0.245), demonstrating that the gate contrastive reward yields more spatially grounded instructions. 4.5 Downstream Navigation Success Rate NLG scores measure lexical similarity to reference instructions but do not directly reflect navigational utility. To bridge this gap, we sample all trajectories from R2R-CE Val Unseen, generate in- structions with each model under the same prompt, and feed the resulting instructions to a frozen CorrectNav [24] follower. Table 5 reports navigation performance under each instruction source. VTInstructor-generated instructions achieve SR of 63.3 and NE of 4.47 m, on par with human-written instructions (SR 61.6, NE 4.53 m) and closely matching their SPL (52.7 vs. 53.3). We note that a single frozen follower cannot establish superiority over human annotations; these numbers indicate the generated instructions reach a quality level comparable to human references for driving downstream navigation. Among baselines, the best proprietary model Qwen3.5-Plus reaches only SR 48.6, lagging VTInstructor by nearly 15 percentage points. Open-source models span a wide range (SR 29.9–45.2): the largest ones (Qwen3.5-397B-A17B 45.2, Kimi-K2.5 43.9) are competitive with the proprietary models, while the smaller video VLMs fall to 29.9–36.2. Model scale alone does not determine instruction utility, even Claude Opus 4.6 obtains the lowest SR (21.7) of any source. These results confirm that NLG metrics and downstream navigation performance are positively correlated, and that VTInstructor’s spatially grounded instructions translate directly into improved follower behaviour. M ’26, November 10–14, 2026, Rio de Janeiro, BrazilHaolin Yang, Yuxing Long, Zihan Yang, and Hao Dong Ours: Walk into the kitchen and go past the island. Walk through the kitchen and out the door on the right. Then turn left and go into the bedroom on the left. Stop in the doorway. Qwen3.5-9B: Walk past the kitchen island and enter the hallway, turning left into the bedroom. Stop next to the bed with the white duvet. GT R2R-CE Instruction: Walk towards the kitchen area. Keep walking along the kitchen area towards the doorway. Turn right and exit the kitchen area walking across the small living room area and into the bedroom to you left. Ours: You are standing facing towards the white railing. Turn right and climb the steps. Turn left and walk straight till you reach th e wall in front of you. Turn left and enter inside the first open white door on your right side. You are now standing at the entrance of a washroom facing towards a white towel. Walk straight and stand in front of the towel. You have reached your point. Qwen3.5-9B: Start at the bottom of the carpeted staircase with the white balustrade on your left. Ascend the stairs, passing the skylight above, and upon reaching the landing, turn right into the hallway. Walk straight past the large white wardrobe on your right and the open doorway to the bedroom on your left. Continue forward until you reach the next set of doorways; turn left into the bathroom. Enter the room, passing the freestanding bathtub on your right, and walk towards the vanity area. Stop in front of the wooden vanity with the sink and mirror. GT RxR-CE Instruction: Now, you are facing towards the white grill that is beside the steps. Turn to your right and move upwards, taking the stairs. Now, take left and move forward. Again, take left and you will find an open door on the right side. Enter through it and move towards your slightly right side and that will be your end point which is in the corner of the washroom where you are facing the towel in front of you. Ours: Exit the bedroom and walk through the dining room. Turn left and enter the kitchen. Turn right and enter the exercise room. Stop in front of the exercise bike. Qwen3.5-9B: Go through the doorway past the white dresser and round mirror, then turn left into the hallway. Continue past the kitchen with the stainless steel refrigerator and go down the stairs. Stop by the exercise bike in the gym with the geometric wall art. GT Room Tour Instruction: Turn left to exit the bedroom.Walk forward to the dining table and turn l ef t .Ta ke a small step forward and turn left.Walk to the kitchen cabinet and turn right.Enter the exercise room. Figure 3: Qualitative comparison on R2R-CE and RxR-CE Val Unseen and a real-world room-tour clip. Each case shows the compressed keyframes with VTP overlays (top) and instructions from the ground truth, the strongest baseline, and VTInstructor (bottom). Yellow highlights mark factual errors in the baseline output (e.g. wrong turn direction, hallucinated landmarks). Table 6: CorrectNav (LLaVA-Video-7B backbone) trained un- der two augmentation settings on R2R-CE Val Unseen. Set. Training DataSR↑ OSR↑ SPL↑ NE↓ A R2R-CE + RxR-CE (human)45.152.344.66.20 BSetting A + VTInstructor-generated48.4 54.2 46.8 5.85 Table 7: CorrectNav (LLaVA-Video-7B backbone) trained un- der two augmentation settings on RxR-CE Val Unseen. Set. Training DataSR↑ OSR↑ SPL↑ NE↓ A R2R-CE + RxR-CE (human)41.251.139.68.34 BSetting A + VTInstructor-generated44.4 53.2 41.3 7.74 4.6 Training Gain from Generated Data We further assess whether VTInstructor-generated instructions pro- vide greater training benefit. A CorrectNav [24] follower (LLaVA- Video-7B [26] backbone) is trained from scratch under two data set- tings and evaluated on R2R-CE Val Unseen (Table 6) and RxR-CE Val Unseen (Table 7). Setting A uses only the original human-annotated R2R-CE and RxR-CE training data. Setting B augments Setting A with 20K instructions generated by our proposed VTInstructor. Compared to the human-only baseline (Setting A), Setting B improves SR by +3 percent on both benchmarks and reduces NE by 0.35 m on R2R-CE and 0.60 m on RxR-CE Val Unseen, demonstrating that VTInstructor-generated data provides meaningful training augmentation for downstream navigation agents. 4.7 Human Evaluation on Real-World Navigation Videos To test whether our instructions transfer beyond simulation, we collect 50 first-person room-tour clips from YouTube covering Table 8: Human evaluation on real-world navigation videos (1–5 scale, mean±std over 3 annotators). MethodAction Landmark Direction Follow. GPT-5.42.65± 0.083.55± 0.062.53± 0.102.29± 0.06 Qwen3-VL-8B2.63± 0.123.65± 0.092.67± 0.052.49± 0.08 VTInstructor4.38± 0.05 4.21± 0.05 4.25± 0.12 4.35± 0.04 offices, corridors, and multi-room apartments. For each clip we estimate per-frame depth and camera pose to recover inter-frame geometry, then apply EDTC and render VTP overlays; the baselines (GPT-5.4 and Qwen3-VL-8B-Instruct) receive the same keyframes. Three annotators rate each instruction on a 1–5 scale along four dimensions: Action (movement and turn sequence), Landmark (object references), Direction (heading and turn angles), and Fol- lowability (whether a naïve follower could reproduce the route). Annotators see the keyframes without overlays, so scores reflect the instruction alone; we report the mean over annotators. VTInstruc- tor leads on all four dimensions (Table 8). The gains are largest on Action and Direction, where the VTP overlay supplies movement- sequence and turn-angle cues that general-purpose models lack, while Landmark scores are closer since large VLMs already recog- nise common indoor objects. 5 Limitations and Future Work The VT-GRPO reward is composed entirely of automatic NLG met- rics computed against reference instructions. While this avoids the prohibitive cost of running a navigation follower in the loop, the policy is not directly optimised for navigational success rate. Future work could incorporate sparse follower feedback (e.g., suc- cess/failure signals from a lightweight frozen follower on a small trajectory buffer) as an additional reward term, more directly bridg- ing instruction quality and downstream navigation performance. VTInstructorMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Acknowledgments This work was supported by the Beijing Natural Science Foundation (L2608141) and the National Natural Science Foundation of China (62136001). References [1]Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sun- derhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018. Vision-and- Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments. In CVPR. [2]Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. 2021. History Aware Multimodal Transformer for Vision-and-Language Navigation. In NeurIPS. [3]Sheng Fan, Rui Liu, Wenguan Wang, and Yi Yang. 2024. Navigation Instruction Generation with BEV Perception and Large Language Models. (2024). [4]Sheng Fan, Rui Liu, Wenguan Wang, and Yi Yang. 2025. Scene Map-based Prompt Tuning for Navigation Instruction Generation. (2025), 6898–6908. [5]Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis- Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. 2018. Speaker-Follower Models for Vision-and-Language Navigation. In NeurIPS. [6]Muraleekrishna Gopinathan, Martin Masek, Jumana Abu-Khalaf, and David Suter. 2024. Spatially-Aware Speaker for Vision-and-Language Navigation Instruction Generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Bangkok, Thailand, 13601–13614. doi:10.18653/v1/2024.acl-long.734 [7]Aishwarya Kamath, Peter Anderson, Su Wang, Jing Yu Koh, Alexander Ku, Austin Waters, Yinfei Yang, Jason Baldridge, and Zarana Parekh. 2022. A New Path: Scal- ing Vision-and-Language Navigation with Synthetic Instructions and Imitation Learning. arXiv preprint arXiv:2210.03112 (2022). [8]Xianghao Kong, Jinyu Chen, Wenguan Wang, Hang Su, Xiaolin Hu, Yi Yang, and Si Liu. 2024. Controllable Navigation Instruction Generation with Chain of Thought Prompting. In ECCV. [9] Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. 2020. Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments. In ECCV. [10]Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. 2020. Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding. In EMNLP. [11]Jialu Li, Hao Tan, and Mohit Bansal. 2022. EnvEdit: Environment Editing for Vision-and-Language Navigation. In CVPR. [12]Yanbang Li, Ziyang Gong, Haoyang Li, Xiaoqi Huang, Haolan Kang, Guangping Bai, and Xianzheng Ma. 2025. Robotic Visual Instruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 12155– 12165. [13] Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, et al.2024. PIVOT: Itera- tive Visual Prompting Elicits Actionable Knowledge for VLMs. arXiv preprint arXiv:2402.07872. [14]Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. 2020. REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments. In CVPR. [15] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300 (2024). [16]Aleksandr Shtedritski, Christian Rupprecht, and Andrea Vedaldi. 2023. What Does CLIP Know about a Red Circle? Visual Prompt Engineering for VLMs. In ICCV. [17]Hao Tan, Licheng Yu, and Mohit Bansal. 2019. Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout. In NAACL. [18] Qwen Team. 2025. Qwen3-VL Technical Report. arXiv preprint. [19]Hanqing Wang, Wei Liang, Jianbing Shen, Luc Van Gool, and Wenguan Wang. 2022. Counterfactual Cycle-Consistent Learning for Instruction Following and Generation in Vision-Language Navigation. In CVPR. [20] Su Wang, Ceslee Montgomery, Jordi Orbay, Vighnesh Birodkar, Aleksandra Faust, Izzeddin Gur, Natasha Jaques, Austin Waters, Jason Baldridge, and Peter Anderson. 2022. Less is More: Generating Grounded Navigation Instructions from Landmarks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [21]Xiaohan Wang, Wenguan Wang, Jiayi Shao, and Yi Yang. 2023. LANA: A Language-Capable Navigator for Instruction Following and Generation. In CVPR. [22]Haolin Yang, Yuxing Long, Zhuoyuan Yu, Zihan Yang, Minghan Wang, Jia- peng Xu, Yihan Wang, Ziyan Yu, Wenzhe Cai, Lei Kang, and Hao Dong. 2026. NavSpace: How Navigation Agents Follow Spatial Intelligence Instructions. arXiv:2510.08173 [cs.RO] https://arxiv.org/abs/2510.08173 [23]Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023. Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V. arXiv preprint arXiv:2310.11441. [24]Zhuoyuan Yu, Yuxing Long, Zihan Yang, Chengyan Zeng, Hongwei Fan, Jiyao Zhang, and Hao Dong. 2026. CorrectNav: Self-Correction Flywheel Empowers Vision-Language-Action Navigation Model. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. 18737–18745. [25] Haitian Zeng, Xiaohan Wang, Wenguan Wang, and Yi Yang. 2023. KEFA: A Knowl- edge Enhanced and Fine-grained Aligned Speaker for Navigation Instruction Generation. arXiv preprint arXiv:2307.13368. [26]Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. 2024. LLaVA-Video: Video Instruction Tuning With Synthetic Data. In NeurIPS. [27]Ruijie Zheng, Yongyuan Liang, Shuaiyi Huang, Jianfeng Gao, Hal Daumé I, Andrey Kolobov, Furong Huang, and Jianwei Yang. 2025. TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policies. In The Thirteenth International Conference on Learning Representations (ICLR).