Paper deep dive
Speaking the Navigator's Language: Trajectory-Grounded Instruction Translation for Frozen Aerial VLN Agents
Xi Chen, Zhe Liu, Xiaogang Xu, Jiafei Xu, Chunyi Zhou, Yuan Su, Rui Zeng, Tianyu Du, Kelu Yao, Chao Li, Shouling Ji
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/9/2026, 7:44:57 AM
Summary
The paper introduces the Trajectory-Grounded Instruction Translator (TGIT), a front-end module that translates short, intent-driven 'Weak' or 'Human' instructions into agent-executable commands for frozen aerial Vision-and-Language Navigation (VLN) agents. By leveraging trajectory outcomes from a frozen navigator (OpenFly, CityNav, AirVLN) to construct preference pairs via Direct Preference Optimization (DPO) with LoRA, TGIT bridges the 'instruction gap' where natural language fails to activate trained capabilities. TGIT significantly improves success rates on weak inputs and transfers zero-shot to human instructions, outperforming baselines like ICL and SFT.
Entities (8)
Relation Signals (7)
Instruction Gap → causes → Decreased Success Rate
confidence 95% · on a frozen OpenFly navigator, this instruction gap drops success rate (SR) from 31.03% to 11.33%.
TGIT → uses → DPO
confidence 95% · The resulting translator is then evaluated on Weak and transferred zero-shot to Human... multi-round DPO with pool resampling.
TGIT → uses → LoRA
confidence 95% · adapts only a LoRA VLM front-end that translates Weak inputs into agent-executable commands.
TGIT → improves → OpenFly
confidence 90% · The resulting Weak-trained translator raises Weak-input SR to 37.93%... and transfers zero-shot to real human instructions... improves held-out OpenFly
Qwen2.5-VL-7B-Instruct → isbackboneof → TGIT
confidence 90% · The translator is Qwen2.5-VL-7B-Instruct with LoRA... on language modules.
TGIT → improves → CityNav
confidence 85% · yields recovery on CityNav and AirVLN.
TGIT → improves → AirVLN
confidence 85% · yields recovery on CityNav and AirVLN.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Aerial vision-and-language navigation (VLN) agents are typically trained on detail-rich, trajectory-aligned commands, whereas users issue short, intent-driven instructions; on a frozen OpenFly navigator, this \emph{instruction gap} drops success rate (SR) from $31.03\%$ to $11.33\%$. To scale translator training, we prompt a language model with human-written style examples to convert original commands into paired, intent-centered Weak commands, which yield $15.27\%$ SR. We introduce the \textbf{Trajectory-Grounded Instruction Translator (TGIT)}, a front-end that keeps the navigator frozen and translates Weak inputs into agent-executable commands by learning from its trajectory outcomes. The resulting Weak-trained translator raises Weak-input SR to $37.93\%$ and transfers zero-shot to real human instructions ($11.33\%{\rightarrow}32.51\%$); it also improves held-out OpenFly ($4.95\%{\rightarrow}20.79\%$) and yields recovery on CityNav and AirVLN.
Tags
Links
- Source: https://arxiv.org/abs/2610.10635v1
- Canonical: https://arxiv.org/abs/2610.10635v1
Trouble viewing inline? Open PDF directly →
Full Text
45,703 characters extracted from source content.
Expand or collapse full text
Speaking the Navigator’s Language: Trajectory-Grounded Instruction Translation for Frozen Aerial VLN Agents Xi Chen Zhe Liu Xiaogang Xu Jiafei Wu Chunyi Zhou Yuan Su Rui Zeng Tianyu Du Kelu Yao Chao Li Shouling Ji Abstract Aerial vision-and-language navigation (VLN) agents are typically trained on detail-rich, trajectory-aligned commands, whereas users issue short, intent-driven instructions; on a frozen OpenFly navigator, this instruction gap drops success rate (SR) from 31.03%31.03\% to 11.33%11.33\%. To scale translator training, we prompt a language model with human-written style examples to convert original commands into paired, intent-centered Weak commands, which yield 15.27%15.27\% SR. We introduce the Trajectory-Grounded Instruction Translator (TGIT), a front-end that keeps the navigator frozen and translates Weak inputs into agent-executable commands by learning from its trajectory outcomes. The resulting Weak-trained translator raises Weak-input SR to 37.93%37.93\% and transfers zero-shot to real human instructions (11.33%→32.51%11.33\%→32.51\%); it also improves held-out OpenFly (4.95%→20.79%4.95\%→20.79\%) and yields recovery on CityNav and AirVLN. 1Zhejiang University 2Zhejiang Lab chan_xi, zhe.liu@zju.edu.cn, xiaogangxu00@gmail.com, jiafeiwu, zhouchunyi, yuansu, ruizeng24, zjradty@zju.edu.cn, yaokelu, lichao@zhejianglab.org, sji@zju.edu.cn 1 Introduction Vision-and-Language Navigation (VLN) has progressed from indoor instruction following (Anderson et al. 2018; Hao et al. 2020; Qiao et al. 2022; Zhang et al. 2024) to aerial platforms, where UAV agents must act under continuous 3D motion, large viewpoint changes, and fine-grained spatial constraints (Liu et al. 2023; Yao et al. 2024; Wang et al. 2025b; Saxena et al. 2025; Sautenkov et al. 2025). Yet aerial VLN agents are commonly trained on detail-dense, trajectory-aligned instructions produced by templates or vision–language generators (Gao et al. 2025; Lee et al. 2025; Wang et al. 2023b; Fan et al. 2024), whereas users tend to issue shorter, intent-driven commands. We begin with a simple controlled probe: keeping the routes, initial conditions, and UAV navigator fixed, we change only the instruction source. If the navigator understood its training and deployment interfaces equally well, its performance should remain stable. Instead, using natural instructions collected from 16 annotators for 203 OpenFly episodes, SR drops from 31.03%31.03\% with dataset instructions to 11.33%11.33\% with Human instructions (Figure 1). This behavioral collapse reveals an instruction gap: deployment language fails to activate capabilities available through the same frozen agent’s training-language interface. Example: Instruction Gap in UAV Navigation. Dataset instruction: “Turn right and proceed straight for 30 meters until reaching the gray rectangular building with a flat roof.” Human-style instruction: “Fly toward the building ahead.” Figure 1: Instruction gap and recoverable activation on a frozen OpenFly-Agent. (a) Median instruction length collapses from Orig to Weak to Human. (b) On the 203-episode evaluation set (Selected-Seen), SR drops from 31.03%31.03\% (Orig) to 15.27%15.27\% (Weak) and 11.33%11.33\% (Human). TGIT(W) recovers to 37.93%37.93\% on Weak; the same adapter transfers zero-shot as TGIT(H) to 32.51%32.51\% on Human. Closing this gap is hard for a frozen navigator trained on trajectory-aligned Orig commands: short Human instructions must be rewritten into language that matches its learned associations, and two equally plausible expansions can still induce different actions. Agent-centric retraining methods can absorb this mismatch (Kamath et al. 2023; Wang et al. 2020), but are costly and may change perception and control, making it hard to tell whether gains come from adapting to new language or from altering the navigator itself (Gao et al. 2025; Lee et al. 2025; Liu et al. 2023; Wang et al. 2025b). Instruction-centric rewriting methods instead map weak inputs back to an Orig sentence (Shah et al. 2023; Zhou et al. 2024; Zhang and Kordjamshidi 2023), yet matching that text does not tell whether a particular navigator will succeed. We therefore keep the navigator fixed and learn agent-executable translations from its trajectory outcomes, not from text similarity alone. This observation also determines which failures should supervise the translator. Training on arbitrary failures would conflate an instruction-interface mismatch with navigation skills that the backbone does not possess. We therefore focus on paired cases where the original instruction succeeds, but its weakened counterpart fails under the same route, initial condition, simulator, and policy. Orig success serves as a capability witness, while Weak failure isolates a recoverable interface intervention. Human annotation is too costly to scale for translator training, so we construct Weak instead: using a few Human-written instructions as style examples, an LM converts each Orig command into a shorter, intent-centered counterpart. Building on these choices, we introduce TGIT (Trajectory-Grounded Instruction Translator). It freezes the UAV policy and adapts only a LoRA VLM front-end that translates Weak inputs into agent-executable commands. Starting from Orig instructions in the target agent’s training dataset and their LM-generated Weak counterparts, TGIT selects hard cases, proposes multiple candidates, executes them with the frozen navigator, and converts weighted trajectory outcomes into preferences for multi-round DPO with pool resampling. The resulting translator is then evaluated on Weak and transferred zero-shot to Human, testing whether execution-grounded preferences activate the agent without imitating a single Orig sentence. On the 203-episode OpenFly evaluation, TGIT raises SR from 15.27%15.27\% to 37.93%37.93\% (2.48×2.48×), exceeding Orig (31.03%31.03\%), ICL, and SFT; the same Weak-trained translator raises Human-input SR zero-shot from 11.33%11.33\% to 32.51%32.51\% (2.87×2.87×). Held-out OpenFly scenes and two additional frozen agents show the same relative activation, but their absolute success remains bounded by the navigator itself. These results support a bounded claim: TGIT repairs recoverable instruction-interface failures, while strengthening the frozen navigator remains an orthogonal agent-centric problem. Our contributions are threefold: • Recoverable instruction gap. We provide Human evidence for the mismatch between deployment and training language, and operationally identify recoverable failures through original-success / Human-failure pairs. • Agent-executable instruction translation. We introduce TGIT, a plug-and-play front-end that can be applied to any frozen VLN agent using only that agent’s own training instructions and their Weak counterparts: it builds trajectory-grounded preference data from hard cases and multi-round DPO to translate underspecified inputs into executable commands. • Controlled cross-agent validation. TGIT outperforms Weak, ICL, and SFT on OpenFly, transfers zero-shot from Weak training to Human inputs, and yields consistent relative gains across held-out environments and multiple frozen UAV agents. The appendix complements the main results with preliminary evidence, validity checks, and reproducibility details. A preliminary study further characterizes training-instruction style and agent sensitivity to underspecification, strengthening the instruction-gap motivation. Paired lexical and embedding analyses further show that Weak moves from Orig toward Human. Complete prompts, dataset statistics, optimization settings, translator-scale diagnostics, and same-route trajectory comparisons support reproduction and expose how the learned translations change agent behavior. 2 Related Work Prior VLN and embodied-language research largely falls into two axes: improving the navigator, and rewriting or generating language for clarity and coverage (Zhang et al. 2024; Lin et al. 2023; Ma et al. 2024). TGIT is not another point on either axis. It keeps the aerial policy frozen and learns an instruction interface that activates competence already present in that policy. Agent-centric aerial VLN. VLN was established in photorealistic indoor environments (Anderson et al. 2018) and later extended toward continuous control, long-horizon navigation, and sim-to-real evaluation (Anderson et al. 2021; Song et al. 2025; Lin et al. 2025). AerialVLN introduced continuous outdoor UAV flight, while CityNav, OpenFly, and AeroVerse scale aerial environments, trajectories, and embodied controllers (Liu et al. 2023; Lee et al. 2025; Gao et al. 2025; Yao et al. 2024). These advances improve navigation capability, typically under detailed, trajectory-aligned training instructions. Instruction-centric language interfaces. Synthetic instruction generation scales VLN supervision (Huang et al. 2022; Kamath et al. 2023; Wang et al. 2023b; Gopinathan et al. 2024), while landmark-centered generation and ULN expose the importance of specificity (Wang et al. 2022; Feng et al. 2022). Closest to our setting, VLN-Trans decomposes instructions into landmark-discriminative sub-instructions (Zhang and Kordjamshidi 2023); T-araVLN translates language for agricultural navigation (Zhao et al. 2025); and RAM rewrites observation–instruction pairs for augmentation and unseen generalization (Wei et al. 2025). These methods optimize clarity, recognizability, or training-data coverage (Wang et al. 2023a). Our position. TGIT never updates the UAV navigator, and it ranks candidate translations with simulator rollouts and updates only a LoRA front-end via preference learning (Rafailov et al. 2023; An et al. 2023; Calandriello et al. 2024). Multimodal preference methods often align VLM responses or retrain the VLA itself (Xing et al. 2025; Hung et al. 2025); TGIT instead uses original-success / weak-failure hard cases to restrict supervision to episodes where the navigator has already demonstrated the required skill. Thus TGIT targets interface activation rather than policy retraining, instruction imitation, or data augmentation. Figure 2: The overview of TGIT. (a) Hard-case pool: keep episodes with S(π,Im)=1S(π,I_m)=1 and S(π,Iw)=0S(π,I_w)=0 so that capability exists but the weak interface fails. (b) Candidate generation: VLM translator fθf_θ takes (Iw,1:M)(I_w,O_1:M) and emits N=3N=3 executable rewrites; only LoRA A/B on the LM decoder is trainable. (c) Reward++multi-round LoRA–DPO: frozen rollouts score each yiy_i by R=0.5SR+0.3SPL+0.2e−0.1dnormR=0.5\,SR+0.3\,SPL+0.2\,e^-0.1d_norm (SR: success rate; SPL: path efficiency; dnormd_norm: normalized final distance; blue flags: targets), form preferences y+≻y−y^+\! \!y^-, update A/B, then chain the adapter and resample hard cases for the next round. (d) Training–deployment contract: evaluation collapses to one forward pass—no reward, preference, or policy update (Eq. 2). 3 Methodology 3.1 Problem Formulation System actors and freeze boundary. Our system contains three actors: an aerial environment ℰE; a pretrained UAV navigator π; and a front-end instruction translator fθf_θ. The navigator consumes an instruction I and executes a trajectory ^=π(I,ℰ). T=π(I,E). Throughout adaptation and evaluation, π and its controller are frozen; θ is the only trainable parameter set. The translator also observes visual context 1:M=omm=1MO_1:M=\o_m\_m=1^M, where M=1M=1 is a single RGB frame and M>1M>1 provides multi-frame context. This follows aerial-VLN annotation protocols in which language is grounded in rendered observations (Gao et al. 2025; Lee et al. 2025; Liu et al. 2023). Visual context supplies scene evidence to the front-end VLM, but no rewards, actions, goal labels, or policy updates are available to it at deployment. Instruction interface and objective. Conventional aerial VLN trains π with detail-rich, trajectory-aligned instructions Im∼mI_m _m (Gao et al. 2025; Lee et al. 2025; Wang et al. 2023b), whereas deployment instructions may follow a Human distribution h≠mD_h _m. For controlled optimization, we use Human-informed LM-weakened inputs Iw∼wI_w _w; wD_w is a paired stress distribution, not an estimator of hD_h. Given IwI_w and 1:MO_1:M, the translator emits I~=fθ(Iw,1:M), I=f_θ(I_w,O_1:M), which the unchanged navigator executes as ^=π(I~,ℰ). T=π( I,E). (1) Let =(s1,…,sT)T=(s_1,…,s_T) denote the reference UAV trajectory, with each state containing 3D position, orientation, and altitude. We optimize only the interface (subject to π fixed): maxθ _θ (Iw,,1:M)∼w _(I_w,T,O_1:M) _w (2) [R(π(fθ(Iw,1:M),ℰ),)], [R (π(f_θ(I_w,O_1:M),E),T ) ], where R(⋅)R(·) measures trajectory-level execution quality. Design goals. Equation 2 induces three requirements aligned with the challenges in the Introduction. First, capability identification: supervision should target episodes whose required navigation skill has already been demonstrated, rather than asking translation to repair missing perception or control. Second, frozen-interface adaptation: the method should learn an agent-specific executable language while preserving heterogeneous UAV policies. Third, execution-grounded supervision: learning must use expensive, non-differentiable rollout outcomes and accommodate multiple valid command phrasings. For these, TGIT addresses these requirements through hard-case selection, a parameter-efficient front-end, and trajectory-ranked preference optimization, respectively. 3.2 Trajectory-Grounded Instruction Translation Overview: closed training–deployment loop. Figure 2 summarizes TGIT. Training first selects original-success / weak-failure hard cases. For each sampled weak input, fθf_θ generates N candidate commands; the frozen navigator executes every candidate; a weighted trajectory reward converts their outcomes into pairwise preferences; and DPO updates only the LoRA translator before the hard-case pool is resampled for the next round. At deployment, this loop collapses to one forward pass: a deployment-style instruction and available RGB frames enter fθf_θ, one executable command is emitted, and π flies unchanged. Simulator rewards, preference pairs, and reference trajectories are training-only signals. Section 4.3 separately tests pool construction, reward ranking, and translator backbone choices without redefining the primary protocol. Hard-case pool construction. Under a limited execution budget, random sampling yields many uninformative episodes: either the navigator already fails under Orig, or Weak happens to succeed by chance. We therefore construct a compact hard-case pool DpoolD_pool. Each instance is selected such that the original dataset instruction succeeds, while the corresponding weakened human-style instruction fails under the same frozen agent. Writing S(π,I,ℰ)∈0,1S(π,I,E)∈\0,1\ for episode success, the recoverable candidate set is Dpool=(Im,Iw,ℰ): D_pool=\(I_m,I_w,E): S(π,Im,ℰ)=1, S(π,I_m,E)=1, (3) S(π,Iw,ℰ)=0. S(π,I_w,E)=0\. These original-success / weak-failure cases isolate the instruction interface mismatch: the agent has demonstrated the required navigation ability, but the weakened command is not executable enough. In practice, we build a pool of 100100 hard cases for each UAV agent and sample K=20K=20 cases per round, resampling across rounds to expose diverse failure modes. Failures under both ImI_m and IwI_w are not used as evidence of a recoverable language problem. Each pair holds the route, initial condition, simulator, and policy fixed while changing only the instruction: Orig success witnesses capability, whereas Weak failure isolates the interface intervention. Section 4.3 ablates this selection and resampling against random and fixed-subset alternatives. Candidate generation and execution. For each sampled instruction x∈Dpoolx∈ D_pool, the translator generates N=3N=3 candidate executable commands yii=1N\y_i\_i=1^N conditioned on (x,1:M)(x,O_1:M) under format constraints. The translator conditions on visual context 1:MO_1:M as an input prior; Section 4.3 varies frame count (e.g., one vs. four frames) to test this prior. We use complementary decoding settings: one deterministic setting for stable command structure and stochastic settings for semantic variation. During training, the original dataset instruction is also included as a strong execution reference, but inference uses only the human-style input and available observations. Each candidate yiy_i is executed by the frozen UAV agent, producing a trajectory τi _i with success, path efficiency, and final-distance measurements. Execution is essential because text similarity cannot identify an executable command: close paraphrases may induce different behavior, while distinct surface forms may produce equally successful trajectories (Daniele et al. 2017; Liang et al. 2022; Zhou et al. 2024). Weighted trajectory reward and preferences. The primary protocol ranks candidates using R(τi,)=αSRi+βSPLi+γe−λdinorm,R( _i,T)=α\,SR_i+β\,SPL_i+γ\,e^-λ d_i^norm, (4) where SRiSR_i, SPLiSPL_i, and NEiNE_i follow standard VLN metrics (Anderson et al. 2018), SPLi=ℓi/max(pi,ℓi)SPL_i= _i/ (p_i, _i) is path efficiency (zero for failures), and dinorm=NEi/max(ℓi,ϵ)d_i^norm=NE_i/ ( _i,ε) normalizes final navigation error by reference-path length. At evaluation time, reported SPL is the mean of these efficiencies over successful episodes only. We set (α,β,γ)=(0.5,0.3,0.2)(α,β,γ)=(0.5,0.3,0.2) and λ=0.1λ=0.1, making task success dominant while retaining dense efficiency and distance signals for failed or near-tied candidates. The reward-ranking ablation in Section 4.3 compares Success-only and lexicographic alternatives, showing that recovery is not tied to these exact weights. We convert trajectory rewards into pairwise preferences: yi≻yjifR(τi,)>R(τj,).y_i y_j R( _i,T)>R( _j,T). (5) This transforms non-differentiable rollout outcomes into a stable supervision signal for preference learning. Unlike pure SFT on a single target phrasing, preference pairs teach the translator which of several plausible commands actually activate π. Near-tied outcomes are discarded to avoid supervision from simulator noise. The weighted scalar above remains the primary TGIT reward. Multi-round DPO optimization. We optimize the translator with Direct Preference Optimization (DPO) (Rafailov et al. 2023), using LoRA (Hu et al. 2021) to update only a small set of VLM parameters. Each round starts from and references the round-matched SFT adapter, samples K=20K=20 instructions from DpoolD_pool, generates and executes candidates, constructs preference pairs with a minimum reward gap, and updates fθf_θ. Adapters are chained across rounds so later rounds refine an already grounded interface rather than restarting from the base VLM. As the translator improves, its candidates produce better trajectories, which in turn provide more reliable preferences for the next round—closing the feedback loop. Throughout this process, the UAV policy π is never modified; all gains come from aligning human-style inputs with the frozen agent’s executable command space. Each round changes the sampled hard cases and candidates, but not the navigator or evaluation protocol. Resampling broadens failure coverage, while adapter chaining preserves executable patterns. Unlike general online preference collection (Calandriello et al. 2024), TGIT evaluates language actions with a fixed policy. Training–deployment contract. Adaptation may access simulator rollouts, reference paths, and trajectory metrics only to construct preference pairs. Evaluation removes all three: the translator receives the deployment instruction and available RGB context, emits one command, and then exits the control loop. The frozen navigator consumes that command using its original controller and stopping rule. This contract prevents reward or trajectory leakage at deployment and attributes behavioral changes to the learned interface rather than to policy updates. 4 Experiments 4.1 Experimental Setup We evaluate TGIT on three frozen UAV navigation agents: OpenFly-Agent (Gao et al. 2025), CityNav-Agent (Lee et al. 2025), and AirVLN-Agent (Liu et al. 2023). All navigation policies remain unchanged; only the front-end translator is trained. The translator is Qwen2.5-VL-7B-Instruct with LoRA (Hu et al. 2021) (r=4r=4, α=8α=8) on language modules. For each agent, we build a hard-case pool of 100100 original-success / weak-failure instructions, sample K=20K=20 cases per round for 55 rounds, and generate N=3N=3 candidates with visual frames. DPO starts from and references the round-matched SFT adapter, uses β=0.1β=0.1, a learning rate of 2×10−52×10^-5, and discards pairs with a reward gap below 0.050.05. Adapters carry across rounds; the navigator remains fixed. These settings define the main protocol; Section 4.3 ablates hard-case pool construction, reward ranking, visual frames, and translator backbone as diagnostics. Splits and evaluation. We factor evaluation scenes by two axes used in prior aerial VLN protocol: whether the translator was optimized on that scene (Selected vs. Unselected), and whether the frozen agent trained on that scene (Seen vs. Unseen) (Gao et al. 2025; Lee et al. 2025; Liu et al. 2023). Selected-Seen / Unselected-Seen / Unseen thus contain 203/202/404203/202/404 OpenFly, 326/180/220326/180/220 CityNav, and 120/192/147120/192/147 AirVLN episodes. Selected-Seen is the pool-construction scene for TGIT; Unselected-Seen is agent-seen but translator-unselected; Unseen is held out for the agent. Each closed-loop rollout lasts at most 100100 agent steps, and success requires final distance below 2020 m. Baselines and Human transfer. We compare Orig, Weak, ICL, and SFT under the Weak protocol; TGIT(W) is the primary result. On OpenFly Selected-Seen, sixteen annotators provide one natural instruction per episode. Human anchors the deployment gap; Weak is the scalable, Human-informed stress protocol. We also report zero-shot TGIT(H), which only reuses the TGIT(W) adapter. Protocol consistency. Within each agent and split, methods use the same episode set, initial conditions, rollout horizon, success threshold, controller, and metric implementation. Only the instruction path and translator update differ. Table 1 reports the complete matched protocol; reward and translator-backbone ablations are explicitly diagnostic because they reuse narrower candidate or frame settings. Controlled comparison. All methods keep the navigator frozen; Human is gap evidence rather than an adaptation baseline. Method Eval input Translator update Rollout preference Multi-round Orig Dataset None No No Weak Weak None No No Human Human None No No TGIT(H) Human Reuse TGIT(W) adapter No No ICL Weak Prompt only No No SFT Weak Supervised LoRA No No TGIT(W) Weak DPO LoRA Yes Yes We report Success Rate (SR), success-conditional path efficiency (SPL), and Navigation Error (NE) (Anderson et al. 2018; Gao et al. 2025; Liu et al. 2023). SR is the fraction of episodes that reach the goal. SPL is the mean of ℓ/max(p,ℓ) / (p, ) over successful episodes only; failed episodes are excluded. Thus SPL measures how efficiently successful trajectories reach the goal and is not upper-bounded by SR. Higher SR/SPL and lower NE are better. Table 1: Main navigation results on three frozen UAV agents. We report Success Rate (SR, %), success-conditional path efficiency (SPL, %), and Navigation Error (NE, meters). SPL is averaged only over successful episodes. TGIT(H) and TGIT(W) denote Human and Weak evaluation inputs, respectively. Human and TGIT(H) are evaluated only on OpenFly Selected-Seen (203 episodes) due to the high cost of human annotation; other Human-input cells are “–”. TGIT(H) reuses the Weak-trained TGIT(W) adapter zero-shot, without Human-specific training. TGIT(W) deltas are against Orig (negative NE is better). Method OpenFly-Agent (Gao et al.,2025) Selected-Seen Unselected-Seen Unseen SR↑ SPL↑ NE↓ SR↑ SPL↑ NE↓ SR↑ SPL↑ NE↓ Orig 31.03 31.66 116.56 20.05 16.82 77.42 11.39 29.82 172.77 Weak 15.27 37.36 82.48 14.11 13.47 202.27 4.95 8.71 90.25 Human 11.33 89.43 108.92 – – – – – – TGIT(H) 32.51 92.75 95.31 – – – – – – ICL 19.70 18.50 72.08 14.60 9.01 145.81 4.46 29.74 138.04 SFT 29.55 33.49 101.79 19.55 14.48 112.46 10.40 26.90 98.69 TGIT(W) 37.93(+6.90) 24.85(-6.81) 72.08(-44.48) 25.25(+5.20) 12.02(-4.80) 118.82(+41.40) 20.79(+9.40) 29.91(+0.09) 76.12(-96.65) Method CityNav-Agent (Lee et al.,2025) Selected-Seen Unselected-Seen Unseen SR↑ SPL↑ NE↓ SR↑ SPL↑ NE↓ SR↑ SPL↑ NE↓ Orig 8.89 90.51 73.15 3.68 98.62 95.00 2.73 98.30 92.81 Weak 6.11 98.21 120.60 1.84 98.55 99.43 0.00 0.00 93.86 ICL 8.83 90.51 68.72 1.53 89.20 100.28 0.00 0.00 92.43 SFT 8.59 89.52 80.59 3.37 87.23 88.32 2.27 95.18 92.28 TGIT(W) 10.00(+1.11) 87.19(-3.32) 74.41(+1.26) 9.20(+5.52) 90.05(-8.57) 70.01(-24.99) 3.64(+0.91) 93.20(-5.10) 91.23(-1.58) Method AirVLN-Agent (Liu et al.,2023) Selected-Seen Unselected-Seen Unseen SR↑ SPL↑ NE↓ SR↑ SPL↑ NE↓ SR↑ SPL↑ NE↓ Orig 5.83 8.60 98.23 1.04 3.78 91.18 0.00 0.00 127.76 Weak 0.83 8.40 92.04 0.00 0.00 92.04 0.00 0.00 106.30 ICL 1.64 8.74 94.25 0.00 0.00 86.06 0.00 0.00 112.41 SFT 5.00 8.34 90.05 1.04 5.88 89.25 0.00 0.00 119.23 TGIT(W) 7.50(+1.67) 8.56(-0.04) 92.67(-5.56) 1.56(+0.52) 4.03(+0.25) 88.48(-2.70) 0.00(+0.00) 0.00(+0.00) 120.41(-7.35) 4.2 Results The instruction gap is behavioral. On the same 203203 OpenFly Selected-Seen episodes and frozen navigator, SR drops from 31.03%31.03\% with Orig to 15.27%15.27\% with Weak and 11.33%11.33\% with Human. Because the route, initial condition, controller, and evaluation remain fixed, this collapse isolates instruction source as the changed interface variable. The lower Human result also shows that the gap is not merely an artifact of LM weakening. Without Human-specific training, the same Weak-trained TGIT adapter raises Human-input SR from 11.33%11.33\% to 32.51%32.51\% (2.87×2.87×), confirming that interface activation transfers beyond the Weak proxy. Thus the deployment mismatch is behavioral, not only a difference in wording or length. Trajectory grounding recovers the interface. Direct Weak execution reaches 15.27%15.27\% SR; prompt-only ICL improves modestly to 19.70%19.70\%, and SFT target imitation recovers to 29.55%29.55\% but remains below Orig. TGIT reaches 37.93%37.93\%, gaining 22.6622.66 points over Weak, 8.388.38 over SFT, and 6.906.90 over Orig. The controls identify the missing supervision: ICL rewrites without agent feedback, while SFT learns one executable target without comparing alternative commands under rollout outcomes. TGIT instead executes and ranks candidates before updating only the translator. This pattern is consistent with trajectory-grounded preference learning supplying information beyond prompting and target imitation; Section 4.3 separately tests the hard-case and preference-design mechanisms. Transfer reveals a backbone-bounded regime. Beyond the pool-construction environment, TGIT raises OpenFly SR from 14.11%14.11\% to 25.25%25.25\% on Unselected-Seen and from 4.95%4.95\% to 20.79%20.79\% on Unseen. The relative activation also transfers across frozen agents: CityNav improves on Selected-Seen (6.11%→10.00%6.11\%→ 10.00\%), Unselected-Seen (1.84%→9.20%1.84\%→ 9.20\%), and Unseen (0.00%→3.64%0.00\%→ 3.64\%); AirVLN improves on Selected-Seen (0.83%→7.50%0.83\%→ 7.50\%) and Unselected-Seen (0.00%→1.56%0.00\%→ 1.56\%). The OpenFly gains outside its 100100-case pool are inconsistent with a pool-memorization-only explanation, while the cross-agent gains show that the activation pattern is not specific to one navigator (Wang et al. 2025a; Anderson et al. 2021). TGIT primarily expands success coverage rather than jointly optimizing every navigation metric. Its OpenFly Selected-Seen SPL (24.85%24.85\%) remains below Orig and SFT, while the high Human SPL describes only the 11.33%11.33\% successful subset. At the agent level, CityNav and AirVLN exhibit relative recovery but low absolute SR, and every AirVLN method remains at 0.00%0.00\% on Unseen. This dual pattern—transferable relative activation but backbone-bounded absolute success—defines the claim: TGIT repairs recoverable interface failures but does not supply missing navigation skill. 4.3 Ablation Study Table 2: Multi-round ablation on data pool construction (OpenFly-Agent). Results are SR (%) / SPL (%) / NE (m). Selected: original succeeds and weakened fails. Pool: maintain 100 candidates and sample 20 per round. Variant SR (%) ↑ / SPL (%) ↑ / NE (m) ↓ Round 1 Round 2 Round 3 Round 4 Round 5 (Final) Random, Full 15.76 / 27.22 / 90.96 30.54 / 17.14 / 119.48 18.72 / 28.70 / 92.97 21.18 / 18.92 / 95.25 18.72 / 41.83 / 83.85 Random, Pool 22.17 / 40.29 / 96.05 19.21 / 24.26 / 108.89 22.17 / 30.34 / 93.33 20.69 / 54.93 / 85.83 17.73 / 36.48 / 87.72 Selected, Full 31.03 / 15.97 / 119.20 33.50 / 21.68 / 110.30 28.08 / 23.35 / 101.37 31.03 / 31.79 / 103.51 29.06 / 24.27 / 105.22 Selected, Pool 21.18 / 16.13 / 133.56 25.12 / 48.11 / 105.44 29.56 / 33.07 / 83.43 31.53 / 37.41 / 76.58 37.93 / 24.85 / 72.08 Figure 3: Hard cases, failure stage, and visual context. (a) Pool curves (Table 2); Selected-Pool Round 5 matches Table 1. (b) Failure stages with success fractions matched to Table 1; TGIT early/late proportions use the nearest Selected-Pool diagnostic after renormalizing success to 77/20377/203. (c) f1∗/f4∗ diagnostics; “full” is TGIT in Table 1. We ablate TGIT on OpenFly-Agent (Airsim_18), which shows clear sensitivity to instruction style in Table 1. The ablations isolate three factors behind that primary TGIT result: which episodes are supervised, how rollouts become preferences, and which translator backbone is used. Hard-case selection and pool resampling. Table 2 compares random vs. hard-case selection and full-subset vs. pool resampling. Random sampling is unstable (SR oscillates 15.76→30.54→18.7215.76→ 30.54→ 18.72). Hard-case selection focuses on original-success / weak-failure episodes and yields stronger supervision. Under hard-case selection, repeatedly training on a fixed full subset saturates, while resampling K=20K=20 from a 100-case pool continues to improve through Round 5 and yields the TGIT result in Table 1 (37.9337.93 / 24.8524.85 / 72.0872.08). Selection without resampling peaks early, while resampling without selection remains unstable; their combination supplies relevance and round-to-round diversity. Mechanism: hard cases, failure stage, and visual context. Figure 3 presents the three oral mechanism slots under a Table 1-aligned reading. Selected-Pool improves through Round 5 to the TGIT result in Table 1 (37.9337.93), while Random-Full oscillates and Selected-Full saturates. Weak failures are dominated by early collapse (153/203153/203); SFT recovers to 29.55%29.55\% SR but leaves a large early-fail mass; TGIT expands success to 37.93%37.93\% and shrinks early failure. Frame diagnostics approach the same TGIT numbers (37.9337.93 / 24.8524.85 / 72.0872.08). Across OpenFly/CityNav/AirVLN, relative activation with sharply different absolute levels separates interface repair from the backbone ceiling. Reward ranking. Table 3 holds the Round 5 candidate set fixed and varies only how rollout metrics induce pairwise preferences. It diagnoses ranking robustness among alternatives to the main Weighted form, not a substitute for the TGIT result in Table 1 (37.93%37.93\%), which also resamples hard cases and chains adapters across rounds. All success-prioritized forms recover well above Weak: Success-only reaches 35.47%35.47\%, and lexicographic Success→ → reaches 36.14%36.14\%, showing that the activation conclusion is not tied to the exact 0.5/0.3/0.20.5/0.3/0.2 scalar weights. We keep Weighted as the main-protocol reward because it supplies dense efficiency and distance signals when candidates are near-tied or unsuccessful; the diagnostic alternatives show that switching among success-first rankings does not reverse the preference-learning story. Table 3: Reward Ranking (N=100N=100, Round 5). Alternatives to the main Weighted form on a fixed candidate set. Main TGIT SR remains 37.93%37.93\% (Table 1). Preference ranking Valid SR↑ SPL↑ NE↓ Success only 203/203 35.47 93.13 91.48 Success→ → 202/203 36.14 95.00 84.60 SPL→ 201/203 32.34 91.24 97.01 Translator backbone and scale. Table 4 compares translator backbones under the same diagnostic N=100N=100, M=4M=4 Round 5 protocol. Qwen2.5-VL-7B is cleanest (36.45%36.45\% SR, full coverage); 2B collapses valid outputs (156/203156/203), so absolute SR becomes unreliable under a fixed 203203-episode denominator. Because model families also differ (Qwen2.5/3/3.5), this is a practical backbone comparison rather than a monotonic parameter-scaling law. It supports 7B as the clean default translator rather than proposing another end-to-end agent. Table 4: Translator backbone/scale comparison (N=100N=100, M=4M=4; diagnostic). SR uses fixed denominator 203203; invalid outputs count as failures. †NE is valid-only. TGIT in Table 1 remains the primary result. Translator Valid SR SPL NE Qwen2.5-VL-7B 203/203 36.45 95.17 90.34 Qwen3-VL-4B 203/203 29.06 90.96 128.33 Qwen3-VL-2B 156/203 31.53 93.67 82.33† Qwen3.5-0.8B 193/203 35.47 93.01 111.05† Discussion and claim boundaries. Hard-case selection isolates language as the changed variable when Orig succeeds and Weak fails; random/full ablations show that structure and resampling both matter. Human SR 11.33%11.33\% shows the mismatch is not an LM-weakening artifact (Dorbala et al. 2024), while Weak remains a stress protocol rather than a Human-distribution estimator; TGIT-on-Human is unmeasured. TGIT’s main gain is success coverage under the weighted reward (+22.66+22.66 SR over Weak; +8.38+8.38 over SFT), not lockstep SPL/NE improvement. OpenFly shows the strongest activation; CityNav/AirVLN expose the backbone ceiling. 5 Conclusion We studied an instruction-interface gap in aerial VLN: frozen agents can fail under short, underspecified inputs even when they possess the required navigation skill. Human annotations establish the mismatch, while Weak provides a controlled stress protocol. TGIT closes the interface loop through hard-case selection, trajectory-ranked preferences, and pool resampling while keeping the UAV agent frozen. It raises OpenFly Selected-Seen SR from 15.27%15.27\% to 37.93%37.93\%, exceeding Orig (31.03%31.03\%), ICL, and SFT. CityNav/AirVLN expose the boundary: translation activates demonstrated skill but cannot strengthen the backbone. Limitations and future work. TGIT currently performs one-shot translation before navigation and does not revise commands online as observations or execution states change. A natural next step is latency-aware, real-time closed-loop translation that updates executable instructions during flight. Moreover, the current agents remain limited in perception and control; another direction is to directly strengthen the navigation backbone itself. References An et al. (2023) G. An, J. Lee, X. Zuo, N. Kosaka, K. Kim, and H. O. Song Direct preference-based policy optimization without reward modeling. In NeurIPS, Cited by: §2. Anderson et al. (2021) P. Anderson, A. Shrivastava, J. Truong, A. Majumdar, D. Parikh, D. Batra, and S. Lee Sim-to-real transfer for vision-and-language navigation. In Conference on Robot Learning, Cited by: §2, §4.2. Anderson et al. (2018) P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. van den Hengel Vision-and-language navigation: interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 3674–3683. Cited by: §1, §2, §3.2, §4.1. Calandriello et al. (2024) D. Calandriello, D. Guo, R. Munos, M. Rowland, Y. Tang, B. A. Pires, P. H. Richemond, C. L. Lan, M. Valko, T. Liu, et al. Human alignment of large language models through online preference optimisation. In ICML, Cited by: §2, §3.2. Daniele et al. (2017) A. F. Daniele, M. Bansal, and M. R. Walter Navigational instruction generation as inverse reinforcement learning with neural machine translation. In Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, Cited by: §3.2. Dorbala et al. (2024) V. S. Dorbala, S. Chowdhury, and D. Manocha Can llms generate human-like wayfinding instructions? towards platform-agnostic embodied instruction synthesis. In NAACL, Cited by: §4.3. Fan et al. (2024) S. Fan, R. Liu, W. Wang, and Y. Yang Navigation instruction generation with bev perception and large language models. In ECCV, Cited by: §1. Feng et al. (2022) W. Feng, T. Fu, Y. Lu, and W. Y. Wang ULN: towards underspecified vision-and-language navigation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 6394–6412. Cited by: §2. Gao et al. (2025) Y. Gao, C. Li, Z. You, J. Liu, Z. Li, P. Chen, Q. Chen, Z. Tang, L. Wang, P. Yang, et al. OpenFly: a comprehensive platform for aerial vision-language navigation. arXiv preprint arXiv:2502.18041. Cited by: §1, §1, §2, §3.1, §3.1, §4.1, §4.1, §4.1, Table 1. Gopinathan et al. (2024) M. Gopinathan, M. Masek, J. Abu-Khalaf, and D. Suter Spatially-aware speaker for vision-and-language navigation instruction generation. In ACL, Cited by: §2. Hao et al. (2020) W. Hao, C. Li, X. Li, L. Carin, and J. Gao Towards learning a generic agent for vision-and-language navigation via pre-training. In CVPR, Cited by: §1. Hu et al. (2021) J. E. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, and W. Chen LoRA: low-rank adaptation of large language models. ArXiv abs/2106.09685. External Links: Link Cited by: §3.2, §4.1. Huang et al. (2022) Z. Huang, Z. Shangguan, J. Zhang, G. Bar, M. Boyd, and E. Ohn-Bar Assister: assistive navigation via conditional instruction generation. In ECCV, Cited by: §2. Hung et al. (2025) C. Hung, N. Majumder, H. Deng, L. Renhang, Y. Ang, A. Zadeh, C. Li, D. Herremans, Z. Wang, and S. Poria NORA-1.5: a vision-language-action model trained using world model- and action-based preference rewards. arXiv preprint arXiv:2511.14659. Cited by: §2. Kamath et al. (2023) A. Kamath, P. Anderson, S. Wang, J. Y. Koh, A. Ku, A. Waters, Y. Yang, J. Baldridge, and Z. Parekh A new path: scaling vision-and-language navigation with synthetic instructions and imitation learning. In CVPR, Cited by: §1, §2. Lee et al. (2025) J. Lee, T. Miyanishi, S. Kurita, K. Sakamoto, D. Azuma, Y. Matsuo, and N. Inoue CityNav: a large-scale dataset for real-world aerial navigation. In ICCV, Cited by: §1, §1, §2, §3.1, §3.1, §4.1, §4.1, Table 1. Liang et al. (2022) J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng Code as policies: language model programs for embodied control. arXiv preprint arXiv:2209.07753. Cited by: §3.2. Lin et al. (2023) J. Lin, H. Gao, X. Feng, R. Xu, C. Wang, M. Zhang, L. Guo, and S. Xu Advances in embodied navigation using large language models: a survey. arXiv preprint arXiv:2311.00530. Cited by: §2. Lin et al. (2025) S. Lin, Z. Li, X. Zhao, G. Zhou, L. Wang, R. Wei, R. Tang, J. Li, H. Wang, J. Pang, et al. VLNVerse: a benchmark for vision-language navigation with versatile, embodied, realistic simulation and evaluation. arXiv preprint arXiv:2512.19021. Cited by: §2. Liu et al. (2023) S. Liu, H. Zhang, Y. Qi, P. Wang, Y. Zhang, and Q. Wu Aerialvln: vision-and-language navigation for uavs. In ICCV, Cited by: §1, §1, §2, §3.1, §4.1, §4.1, §4.1, Table 1. Ma et al. (2024) Y. Ma, Z. Song, Y. Zhuang, J. Hao, and I. King A survey on vision-language-action models for embodied ai. arXiv preprint arXiv:2405.14093. Cited by: §2. Qiao et al. (2022) Y. Qiao, Y. Qi, Y. Hong, Z. Yu, P. Wang, and Q. Wu Hop: history-and-order aware pre-training for vision-and-language navigation. In CVPR, Cited by: §1. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct preference optimization: your language model is secretly a reward model. In NeurIPS, Cited by: §2, §3.2. Sautenkov et al. (2025) O. Sautenkov, Y. Yaqoot, A. Lykov, M. A. Mustafa, G. Tadevosyan, A. Akhmetkazy, M. A. Cabrera, M. Martynov, S. Karaf, and D. Tsetserukou UAV-vla: vision-language-action system for large scale aerial mission generation. In ACM/IEEE International Conference on Human-Robot Interaction (HRI), Cited by: §1. Saxena et al. (2025) P. Saxena, N. Raghuvanshi, and N. Goveas UAV-vln: end-to-end vision language guided navigation for uavs. In European Conference on Mobile Robots (ECMR), Cited by: §1. Shah et al. (2023) D. Shah, B. Osiński, S. Levine, et al. Lm-nav: robotic navigation with large pre-trained models of language, vision, and action. In Conference on robot learning, Cited by: §1. Song et al. (2025) X. Song, W. Chen, Y. Liu, W. Chen, G. Li, and L. Lin Towards long-horizon vision-language navigation: platform, benchmark and method. In CVPR, Cited by: §2. Wang et al. (2025a) L. Wang, X. Xia, H. Zhao, H. Wang, T. Wang, Y. Chen, C. Liu, Q. Chen, and J. Pang Rethinking the embodied gap in vision-and-language navigation: a holistic study of physical and visual disparities. In ICCV, Cited by: §4.2. Wang et al. (2022) S. Wang, C. Montgomery, J. Orbay, V. Birodkar, A. Faust, I. Gur, N. Jaques, A. Waters, J. Baldridge, and P. Anderson Less is more: generating grounded navigation instructions from landmarks. In CVPR, Cited by: §2. Wang et al. (2025b) X. Wang, D. Yang, Z. Wang, H. Kwan, J. Chen, W. Wu, H. Li, Y. Liao, and S. Liu Towards realistic uav vision-language navigation: platform, benchmark, and methodology. In Proceedings of ICLR, Cited by: §1, §1. Wang et al. (2023a) X. Wang, W. Wang, J. Shao, and Y. Yang Lana: a language-capable navigator for instruction following and generation. In CVPR, Cited by: §2. Wang et al. (2020) X. Wang, Q. Huang, A. Celikyilmaz, J. Gao, D. Shen, Y. Wang, W. Y. Wang, and L. Zhang Vision-language navigation policy learning and adaptation. IEEE transactions on pattern analysis and machine intelligence. Cited by: §1. Wang et al. (2023b) Z. Wang, J. Li, Y. Hong, Y. Wang, Q. Wu, M. Bansal, S. Gould, H. Tan, and Y. Qiao Scaling data generation in vision-and-language navigation. In ICCV, Cited by: §1, §2, §3.1. Wei et al. (2025) Z. Wei, B. Lin, Y. Nie, J. Chen, S. Ma, H. Xu, and X. Liang Unseen from seen: rewriting observation-instruction using foundation models for augmenting vision-language navigation. IEEE Transactions on Neural Networks and Learning Systems. Note: RAM paradigm; also arXiv:2503.18065. Confirm TNNLS volume/pages if available. Cited by: §2. Xing et al. (2025) S. Xing, P. Li, Y. Wang, R. Bai, Y. Wang, C. Hu, C. Qian, H. Yao, and Z. Tu Re-align: aligning vision language models via retrieval-augmented direct preference optimization. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: §2. Yao et al. (2024) F. Yao, Y. Yue, Y. Liu, X. Sun, and K. Fu Aeroverse: uav-agent benchmark suite for simulating, pre-training, finetuning, and evaluating aerospace embodied world models. arXiv preprint arXiv:2408.15511. Cited by: §1, §2. Zhang and Kordjamshidi (2023) Y. Zhang and P. Kordjamshidi VLN-Trans: translator for the vision and language navigation agent. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), p. 13219–13233. Cited by: §1, §2. Zhang et al. (2024) Y. Zhang, Z. Ma, J. Li, Y. Qiao, Z. Wang, J. Chai, Q. Wu, M. Bansal, and P. Kordjamshidi Vision-and-language navigation today and tomorrow: a survey in the era of foundation models. TMLR. Cited by: §1, §2. Zhao et al. (2025) X. Zhao, X. Lyu, X. Chen, and X. Li T-araVLN: translator for agricultural robotic agents on vision-and-language navigation. arXiv preprint arXiv:2509.06644. Note: GitHub claims ICIC 2026 Oral; using arXiv until camera-ready venue confirmed Cited by: §2. Zhou et al. (2024) G. Zhou, Y. Hong, and Q. Wu Navgpt: explicit reasoning in vision-and-language navigation with large language models. In AAAI, Cited by: §1, §3.2.