Paper deep dive
Dynamic Context Adapters: Efficiently Infusing History into Vision-and-Language Models
Yuhang Song, Bor-Jiun Lin, Jiaxu Liu, Te-Chuan Chiu, Anh Nguyen, Chun-Yi Lee
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 8/16/2026, 2:44:50 AM
Summary
The paper introduces Dynamic Context Adapter (DCA), a novel framework for integrating historical context into Vision-Language Models (VLMs) for sequential decision-making tasks like Vision-and-Language Navigation (VLN). DCA addresses the computational inefficiency and memory bottlenecks of existing methods (token concatenation and recurrent compression) by using a fixed-size, dynamically compressed memory vector. This vector is injected into the VLM's layers via lightweight adapters, allowing the model to maintain constant input length and linear complexity growth while preserving temporal semantics. The method achieves over 25% reduction in attention FLOPs and 13% memory savings compared to baselines, improving performance on long-horizon tasks.
Entities (12)
Relation Signals (9)
Dynamic Context Adapter → improves → Vision-Language Models
confidence 95% · DCA bridges static VLMs and recurrent policies and enables memory capabilities in pretrained models while maintaining computational efficiency.
Dynamic Context Adapter → uses → Memory Integration Module
confidence 95% · pass it to the Memory Integration Module, and adapt the resulting memory into the layer outputs of the LLM backbone.
Dynamic Context Adapter → uses → Memory Compression Module
confidence 95% · Our method employs fixed-size, dynamically compressed memory... We compress the historical context through a Memory Compressing Module
Dynamic Context Adapter → reduces → attention FLOPs
confidence 92% · DCA achieves over 25% reduction in attention FLOPs
Dynamic Context Adapter → reduces → GPU Memory
confidence 92% · 13% memory savings while improving performance on long-horizon tasks.
PrismaticVLM → isbackbonefor → Dynamic Context Adapter
confidence 90% · we adopt PrismaticVLM [28] as our foundation.
Dynamic Context Adapter → outperforms → NaVid
confidence 85% · Our findings suggest that DCA resolves the core tension... matching or exceeds prior methods in Success Rate
Dynamic Context Adapter → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Historical context integration presents a fundamental challenge for Vision-Language Models (VLMs) in sequential decision-making tasks. Current VLMs process visual inputs independently, which creates critical limitations for downstream applications that require temporal understanding. Direct incorporation of historical frames into Transformer inputs produces quadratic attention complexity and excessive memory consumption. Existing approaches suffer from significant drawbacks: computational inflation or substantial information loss through temporal compression. To address these challenges, we introduce Dynamic Context Adapter (DCA), a novel context injection approach for pretrained VLMs. Our method employs fixed-size, dynamically compressed memory to preserve historical semantics without frame concatenation. DCA bridges static VLMs and recurrent policies and enables memory capabilities in pretrained models while maintaining computational efficiency. DCA achieves over $25\%$ reduction in attention FLOPs and $13\%$ memory savings while improving performance on long-horizon tasks.
Tags
Links
- Source: https://arxiv.org/abs/2608.10525v1
- Canonical: https://arxiv.org/abs/2608.10525v1
Trouble viewing inline? Open PDF directly →
Full Text
47,262 characters extracted from source content.
Expand or collapse full text
11institutetext: Department of Computer Science, University of Liverpool 11email: sgyson10,Jiaxu.Liu,Anh.Nguyen@liverpool.ac.uk 22institutetext: Department of Computer Science, National Tsinghua University 22email: theochiu@cs.nthu.edu.tw 33institutetext: Imperial College London 33email: j.liu2@imperial.ac.uk 44institutetext: Department of Computer Science and Information Engineering & & NTU Artificial Intelligence Center of Research Excellence (AI-CoRE), National Taiwan University 44email: crlc112358@gmail.com, 44email: cylee@csie.ntu.edu.tw Dynamic Context Adapters: Efficiently Infusing History into Vision-and-Language Models Yuhang Song Bor-Jiun Lin Jiaxu Liu Te-Chuan Chiu Anh Nguyen Chun-Yi Lee Abstract Historical context integration presents a fundamental challenge for Vision-Language Models (VLMs) in sequential decision-making tasks. Current VLMs process visual inputs independently, which creates critical limitations for downstream applications that require temporal understanding. Direct incorporation of historical frames into Transformer inputs produces quadratic attention complexity and excessive memory consumption. Existing approaches suffer from significant drawbacks: computational inflation or substantial information loss through temporal compression. To address these challenges, we introduce Dynamic Context Adapter (DCA), a novel context injection approach for pretrained VLMs. Our method employs fixed-size, dynamically compressed memory to preserve historical semantics without frame concatenation. DCA bridges static VLMs and recurrent policies and enables memory capabilities in pretrained models while maintaining computational efficiency. DCA achieves over 25%25\% reduction in attention FLOPs and 13%13\% memory savings while improving performance on long-horizon tasks. 1 Introduction Figure 1: Context Concatenation vs. Dynamic Context Adaptation. (Left) Traditional concatenation appends historical frames to current input, producing quadratic computational complexity and token length constraints. (Right) Our proposed DCA method decouples historical context from the LLM backbone through lightweight adapters. The Dynamic Context Compression module processes historical frames and distributes compressed representations across multiple VLM layers, maintaining constant input length while achieving about ∼1.3 1.3 trillion fewer FLOPs and 13% memory reduction. Sequential decision-making in partially observable environments demands agents to integrate rich historical context across extended temporal horizons. In Vision-and-Language Navigation (VLN), agents must synthesize information from multiple past observations to navigate complex multi-room environments where current visual input alone provides insufficient context for decision-making. Long-horizon instructions such as “pass through the bedroom, locate the study, and pick up the book on the desk" require agents to chain subgoals while preserving spatial dependencies across rooms and corridors. Under partial observability, an agent’s onboard camera captures only a limited field of view at each timestep, making historical context essential for inferring occluded landmarks, retracing steps, and maintaining spatial awareness. While Transformer-based Vision-Language Models (VLMs) [55, 31, 39, 35, 3, 8, 16] have achieved remarkable success in single-frame visual reasoning tasks such as Image Captioning [26] and Visual Question Answering [7], their adaptation to sequential tasks reveals fundamental limitations. Recent approaches in Vision-and-Language Action [43] and Navigation [59] tasks, including OpenVLA [30], RT-1 [10], RT-2 [9], Navid [63], UniNavid [62], and NavGPT2 [67], have demonstrated the potential of VLMs in embodied scenarios. However, these methods struggle to efficiently integrate the extensive historical visual context necessary for long-horizon reasoning. Existing strategies for integrating historical context into VLM backbones can be grouped into three main categories. (1) Token concatenation approaches are widely used in integrating historical context in Transformer-based models [61, 63, 14, 22, 38]. (2) Recurrent compression methods employ RNNs or LSTMs to compress the entire frame history into a single state vector [34, 25]. (3) Previous studies also evaluate methods that maintain an external mapping and memory frameworks by external topological or semantic maps [64, 4, 5, 13]. Although these methods demonstrate effective performance, all three classes face certain limitations when applied to large-scale pretrained VLMs in environments that require context. Token concatenation disrupts the downstream token order and floods the model with redundant information. Recurrent compression lacks the capacity to represent fine temporal structure, leading to information loss over extended sequences. External memory methods depend on manually constructed maps that may not generalize across different environments. These limitations collectively highlight the necessity for a more efficient and effective methodology to integrate long historical context into pretrained VLMs. Motivated by these challenges, in this study, we focus on eliminating memory bottlenecks and reducing computational complexity while preserving the original model architecture to maintain effectiveness. We draw insight from recent advances in parameter-efficient fine-tuning (PEFT) for large language models (LLMs) [27, 65, 29], which insert small trainable modules into frozen backbones with minimal overhead. This motivates our investigation of whether a similarly lightweight adapter paradigm can fuse rich historical visual information into VLMs while preserving their efficiency and pretrained knowledge. To this end, we propose the Dynamic Context Adapter (DCA), which compresses arbitrary sequences of past frame embeddings into a fixed set of learnable context vectors. DCA eliminates the memory bottleneck associated with naive token concatenation while capturing rich temporal semantics. To enable the model to consult its memory at every depth without altering original parameters or structure, these compressed representations are adapted to the LLM outputs through lightweight adapter modules and injected into each layer of the pretrained VLM. Our method delivers three key advantages. First, DCA ensures computational efficiency by maintaining constant input token length regardless of past frame quantity, achieving linear complexity growth with extended context. Second, DCA preserves fine-grained contextual details through dynamic compression of critical information into fixed context vectors. This approach avoids temporal detail loss common in recurrent models while removing redundant features. Third, DCA retains the original input and fully preserves the priors of the pretrained VLM, which enables maintaining its learned knowledge to the greatest extent possible. We evaluate DCA on the standard navigation benchmark and compare it against both RGB-only baselines and existing context-integration approaches. Our findings suggest that DCA resolves the core tension between capturing comprehensive visual history and maintaining computational tractability. The experimental results demonstrate that DCA matches or exceeds prior methods in Success Rate while reducing attention FLOPs by over 25%25\% and cutting peak memory consumption by 15%15\% on long-horizon VLN tasks. Our contributions can be summarized as follows: (1) We introduce DCA, an efficient and lightweight framework that addresses the core limitation of VLMs in sequential tasks by enabling dynamic compression and integration of historical visual context without disrupting pretrained model architecture or inflating input sequences. (2) We demonstrate that DCA overcomes the fundamental challenge of information loss in recurrent-based approaches and memory explosion issues in concatenation methods, enabling VLMs to maintain rich temporal understanding across extended navigation episodes. (3) We validate that DCA enables effective utilization of historical context for long-horizon reasoning in partially observable environments, achieving superior navigation performance. 2 Related Works Pretrained Vision-Language Models. Large-scale VLMs [31, 21, 54, 28, 36, 8, 16] have achieved impressive multimodal general-purpose reasoning capabilities. For example, ViLT [31] introduced a minimalist vision-language Transformer that forgoes region-based visual features for end-to-end image-text encoding. Likewise, LLaVA [39] fine-tunes a pre-trained vision encoder together with a fine-tuned version of LLaMA [20] using GPT-4 [2] generated instruction follow-up data, producing a powerful multimodal assistant capable of open-ended visual dialogue. These models are typically designed for textual modality or static image-text pairs, and do not accommodate video or historical visual contexts essential for navigation and temporally extended reasoning tasks. LLaMA-VID [36] extends the LLaMA [20] for video-text tasks but primarily addresses the problem with naive inefficient token concatenation. Navigation with Pretrained Large Models. Several recent works have explored applying foundation models to embodied VLN tasks. Intuitive approaches involve directly leveraging pretrained large language models as planners [60, 50, 68, 42, 12, 11, 58], while other groups of works have shown great success in incorporating VLMs as navigation backbones [67, 37, 46, 41, 66, 62]. NaVid [63] fine-tunes a video-based VLM backbone to predict next-step actions by concatenating raw frame tokens, including both current and historical observations. The following work Uni-NaVid [62] unifies multiple navigation tasks in a video-based VLM backbone, processing long video streams end-to-end. [67] augments LLMs with policy networks for VLN by input concatenation. While leveraging powerful pre-trained representations, they suffer from quadratic scaling with frame concatenation and lack mechanisms to distill and recall prior observations. DCA uses lightweight adapters to effectively decouple history context from LLM input, maintaining constant-length inputs while efficiently retrieving relevant historical information across layers. Historical Context in Navigation. Traditional recurrent models maintained implicit memory via LSTM or GRU hidden states that carry over past perceptions [6, 53, 19, 34, 52, 25, 33, 23], but more recent approaches use Transformer-based architectures to capture longer-range dependencies [44, 62, 51, 38, 22, 14, 63]. These methods integrate historical context by either maintaining recurrent hidden states or concatenating history frames as additional input during prediction, which may result in information loss. Other works have proposed building structured memory representations of the environment [40, 4, 5, 17, 57, 56, 49, 15, 13, 64]. Our work uses Transformer-based pretrained VLMs as backbones, but instead of adding additional input tokens, we introduce an efficient method to adapt context into LLM layers. Previous works were designed for static vision-language alignment tasks like few-shot prompting or pretraining with fixed image-text pairs [45, 47]. In contrast to these prior methods that perform one-time modality bridging, our model operates within a Partially Observable Markov Decision Process (POMDP) and continuously compress expanding history. 3 Efficient History Context Adaptation Methodology Efficient VLM Architecture Selection. To maximize efficiency while demonstrating effectiveness, we employ a compact pretrained VLM backbone. Following recent advances in efficient VLM deployment [63, 62, 64, 30], we adopt PrismaticVLM [28] as our foundation. We select the phi-2+3b variant with only 3B parameters, which incorporates a ViT-based CLIP [18, 47] visual encoder, a lightweight Phi-2 [1] language model, and multi-layer cross-modal projection. This architecture choice demonstrates that our efficiency gains extend beyond large-scale VLM models. Efficient Context Processing Pipeline. Given visual observations X, we encode each frame into visual tokens and project them into a shared embedding space with language tokens. This process yields X′=X1:t−1′,Xt′X =\X _1:t-1,X _t\. Instructions LtL_t are tokenized to produce Lt′L _t. For action prediction at timestep t, we process current frame tokens Xt′X _t and instruction tokens Lt′L _t through the LLM while utilizing encoded historical frames X1:t−1′X _1:t-1 as inputs for efficient contextual embedding adaptation in LLM layers. Figure 2: Model Architecture Overview. In each timestep t, the model receives inputs, including initial compression vector, history observations, current observations, and navigation instruction. We compress the historical context through a Memory Compressing Module, pass it to the Memory Integration Module, and adapt the resulting memory into the layer outputs of the LLM backbone. 3.1 Dynamic Context Adaptation for Efficient Historical Integration Our DCA addresses the computational bottleneck of historical context integration through a two-stage architectural design that maintains linear complexity while preserving temporal richness. The core innovation lies in decoupling historical context processing from the main VLM backbone, which enables efficient memory management without sacrificing representational capacity. Given that navigation environments constitute POMDPs, agents must integrate previous observations for informed decision-making. However, naive concatenation of past tokens causes input sequence explosion and incurs super-linear self-attention costs (detailed in supplementary). DCA resolves this efficiency-accuracy trade-off by dynamically compressing historical context into a compact set of learnable memory vectors that adapt into LLM layers while preserving upstream pretrained semantics. Fig. 2 illustrates our efficient dual-pipeline architecture. The standard VLM pathway processes XtX_t through a shared visual encoder and tokenizes instruction LtL_t via the Phi tokenizer. Both inputs pass through the pretrained VLM to produce decoder embedding tz_t, which an action head decodes into next-step action ata_t following standard next-token prediction. The efficiency-focused context adaptation pathway operates in parallel: a fixed-size learnable compression vector initM_init queries past embeddings X1:t−1X_1:t-1 through our Memory Compression Module, producing compressed memory 1:t−1M_1:t-1. Our Memory Integration Module then attends over 1:t−1M_1:t-1 with current decoder queries to extract context-enhanced outputs that can adapt into LLM layers without inflating input sequences. 3.1.1 Efficient Dynamic Context Vector Compressing Our compression strategy achieves computational efficiency by transforming variable-length historical sequences into fixed-size representations while preserving critical temporal information. This design eliminates quadratic scaling in concatenation methods and enables practical deployment in resource-constrained scenarios. We initialize a learnable compression vector init=n.Embedding(C,d).weight∈ℝC×dM_init= n.Embedding(C,d). weight ^C× d for each timestep t, where C denotes memory token count and d represents embedding dimension. Historical frames X1:t−1X_1:t-1 are encoded via the vision encoder [18, 47] and concatenated to form encoded features 1:t−1=∥t=1t−1ViT-CLIP(i)∈ℝ(t−1)×P×dF_1:t-1=\|_t=1^t-1ViT-CLIP(X_i) ^(t-1)× P× d, where P denotes image patch count. To reduce spatial redundancy, we apply grid pooling operator :ℝP×d→ℝp×dG:R^P× d ^p× d (with p≪Pp P) following established practices [63, 36]. This yields 1:t−1=(1:t−1)∈ℝ(t−1)×p×dF_1:t-1=G(F_1:t-1) ^(t-1)× p× d. Our Memory Compression Module employs multi-layer cross-attention between MinitM_init and pooled features. We project MinitM_init as queries and history features as keys and values: QM=MinitWQQ_M=M_initW_Q, KF=1:t−1WKK_F=F_1:t-1W_K, VF=1:t−1WVV_F=F_1:t-1W_V. The compressed computation achieves O(C⋅p)O(C· p) complexity: M1:t−1=ScpsVF∈ℝC×d,whereScps=Softmax(QMKFT)∈ℝC×pM_1:t-1=S_cpsV_F ^C× d, S_cps=Softmax(Q_MK_F^T) ^C× p (1) 3.1.2 Efficient Context Adaptation for LLM Integration Our context adaptation integrates compressed historical information directly into LLM layers without inflating input sequences or disrupting the original architecture. This approach maintains constant computational overhead regardless of history length while enabling multi-layer conditioning that enhances temporal understanding. The integration process operates on standard encoder-only multi-layer language models. For each layer k with input zk−1∈ℝS×dz_k-1 ^S× d, where S represents sequence length, the standard layer output zkz_k is formulated as: zk=Atten(Qk−1,Kk−1,Vk−1),Qk−1,Kk−1,Vk−1=zk−1WQk−1,zk−1WKk−1,zk−1WVk−1. splitz_k&=Atten (Q_k-1,\,K_k-1,\,V_k-1 ),\\ Q_k-1,K_k-1,V_k-1&=z_k-1W_Q^k-1,z_k-1W_K^k-1,z_k-1W_V^k-1. split (2) Our Memory Integration Module enables efficient historical context adaptation into each Transformer layer. This module integrates the compressed context vector from Eq. (1) through lightweight cross-attention that maintains linear complexity. The module projects compressed historical context into key-value representations: KM=M1:t−1WKMK_M=M_1:t-1W_K^M and VM=M1:t−1WVMV_M=M_1:t-1W_V^M. The context-enhanced output computation achieves efficiency by attending the original layer output zkz_k to compressed historical vectors rather than processing full sequence history, which can be expressed as follows: zkcontext=SintgVM,whereSintg=Softmax(Qk−1KMT),z_k^context=S_intgV_M, S_intg=Softmax(Q_k-1K_M^T), (3) where SintgS_intg denotes the attention score of the integration module. The final layer output combines the context-enhanced representation with the original output through learnable scalar weighting as: zk+1←zk+1+λzk+1context.z_k+1← z_k+1+λ z_k+1^context. (4) This design maintains computational efficiency by processing only C compressed memory tokens per layer rather than the full history sequence of length t, achieving favorable O(S⋅C)O(S· C) for efficient context integration compared to the prohibitive O(S⋅t⋅p)O(S· t· p) for the naive concatenation approaches. 4 Experimental Results 4.1 Experimental Setup Table 1: Inference Throughput Comparisons. Method (Input: RGB) # Params Step Inf. Time FLOPs (T) Mem. Peak (GB) Navid-IL 7B 2.86 4.89 48.61 No-Adapt 3B 3.21 4.77 37.84 Recurrent-Adapt 3B 2.50 4.14 35.65 DCA (Ours) 3B 2.71 4.23 34.31 Baselines. For a fair comparison, we evaluate methods that implement end-to-end learning with low-level action primitives in the VLN-CE environments. (1) Seq2Seq [34]: A recurrent sequence-to-sequence architecture that directly maps RGBD observations to navigation actions. The RGB-Seq2Seq variant processes RGB inputs exclusively. (2) CMA [34]: Implements cross-modal attention between instructions and RGBD observations for action prediction. Note that RGB-CMA denotes the RGB-only configuration. (3) NaVid [63]: Employs a frozen VLM backbone to formulate navigation as next-token prediction over RGB sequences. This method concatenates historical observations as additional language tokens and applies auxiliary training objectives. NaVid-IL represents the imitation learning configuration. For efficiency experiments, we establish two controlled baselines that share our VLM backbone and training protocol: (1) No-Adapt: Processes historical frames as additional VLM input tokens without compression or adaptation mechanisms. (2) Recurrent-Adapt: Replaces our Memory Compression Module with an LSTM that sequentially processes past frame embeddings into fixed-size context representations while maintaining the identical backbone architecture as well as the training pipeline. We leverage several representative metrics for evaluating the navigation performace: success rate (SR), success rate weighted by the ratio between the shortest path length and the predicted path length (SPL), oracle success rate (OSR), trajectory length (TL), as well as navigation error (NE). Detailed information regarding simulation environment is extended in appendix. 4.2 Analysis on Model Efficiencies (a) FLOPs vs. History Length (b) GPU Memory vs. History Length Figure 3: Computational efficiency analysis of context-adaptation methods. Left: FLOP requirements as history length increases. Right: Peak GPU memory consumption across varying history lengths. Computational Efficiency Analysis. Table 1 presents a comparison of inference throughput across methods. Our DCA approach demonstrates substantial efficiency gains compared to the No-Adapt baseline: average inference time decreases from 3.21s to 2.71s per step, FLOPs reduce from 4.77T to 4.23T, and peak GPU memory usage drops from 37.84 GB to 34.31 GB. These improvements directly result from our efficient dual-pipeline architecture that decouples historical context processing from the main VLM backbone. To analyze scalability characteristics, Fig. 3 illustrates computational overhead as history length δ increases across methods. While all approaches exhibit approximately linear growth due to feed-forward network dominance independent of δ, critical efficiency distinctions emerge with extended sequences. At initialization (δ=0δ=0), all methods demonstrate comparable FLOP requirements. However, as history length increases, the No-Adapt baseline (red curve) exhibits the steepest computational growth. At δ=30δ=30, DCA achieves over 25% reduction in additional FLOPs relative to No-Adapt, validating our architectural efficiency claims. The Recurrent-Adapt baseline presents an interesting contrast: it demonstrates the most favorable FLOP scaling due to minimal per-timestep recurrent update costs. However, this apparent efficiency advantage comes at the expense of representational capacity, ultimately limiting navigation performance. This trade-off illustrates the fundamental challenge our method addresses: achieving both computational efficiency and representational richness for effective historical context integration. Our results demonstrate that DCA can effectively resolve this efficiency-accuracy tension through efficient architectural design rather than sacrificing either computational tractability or its temporal understanding capabilities. Memory Efficiency. Fig. 3 (b) presents GPU memory consumption patterns during training as history length δ varies. All methods exhibit modest memory growth with increasing δ, as model weights dominate total memory usage. At baseline (δ=0δ=0), Recurrent-Adapt shows the lowest memory requirements, while No-Adapt and DCA present nearly identical consumption. This similarity confirms that our DCA module introduces minimal architectural overhead. However, a significant efficiency gap emerges for δ≥1δ≥ 1: DCA consistently uses approximately 30% less memory than No-Adapt. This reduction directly results from our compressed context representation strategy, which processes fixed-size memory vectors rather than expanding token sequences. The memory efficiency advantage becomes increasingly pronounced with longer histories, demonstrating the practical benefits of our compression-based approach for resource-constrained deployment scenarios. Table 2: Evaluations on VLN-CE R2R Val-Unseen. ∗: Methods use high-level action space. † : Methods apply the waypoint predictor proposed in [24]. ‡ : Methods use extra visual data. # Method Observation VLN-CE R2R Val-Unseen Pan. S.RGB Depth Odo. TL NE↓ OS↑ SR↑ SPL↑ 1 HPN+DN∗ [32] ✓ ✓ ✓ 7.62 6.31 40.0 36.0 34.0 2 CMA∗† [24] ✓ ✓ ✓ 10.90 6.20 52.0 41.0 36.0 3 RecurrentVLN∗† [24] ✓ ✓ ✓ 12.23 5.74 53.0 44.0 39.0 4 Sim2Sim∗ [33] ✓ ✓ ✓ 10.69 6.07 52.0 43.0 36.0 5 HAMT∗† ‡ [14] ✓ ✓ ✓ – 4.80 – 55.0 51.0 6 LAW [48] ✓ ✓ ✓ 8.89 6.83 44.0 35.0 31.0 7 Seq2Seq [34] ✓ ✓ 9.30 7.77 37.0 25.0 22.0 8 CMA [34] ✓ ✓ 8.64 7.37 40.0 32.0 30.0 9 NaVid [63] ✓ 7.63 5.47 49.1 37.4 35.9 10 NaVid-IL [63] ✓ – 7.10 20.6 14.4 12.4 11 RGB-Seq2Seq [34] ✓ 4.86 10.10 8.10 0.00 0.00 12 RGB-CMA [34] ✓ 6.28 9.55 10.80 5.00 4.43 13 DCA (No-Adapt) ✓ 3.91 7.12 8.86 7.23 7.00 14 DCA (Recurrent-Adapt) ✓ 8.44 9.56 7.14 6.59 5.44 15 DCA (Ours) ✓ 6.73 6.77 25.3 13.7 12.9 Figure 4: Averaged attentions of the Memory Compression Module across navigation timesteps for unseen evaluation episode 37 in VLN-CE, with the corresponding visual observations indicated. 4.3 Evaluations on VLN Performance Table 2 presents navigation performance on VLN-CE R2R Val-Unseen split. Methods are organized by input modality: approaches using additional sensors beyond RGB (#1-#8) appear above the first horizontal line, while RGB-only methods (#9-#15) are grouped below. NaVid variants receive separate categorization due to auxiliary co-training protocols. DCA shows substantial performance improvements under the low-level action VLN-CE framework. Compared to recurrent baselines RGB-Seq2Seq and RGB-CMA, DCA achieves relative success rate improvements of 13.7% and 8.7%, respectively. Against Recurrent-Adapt, which shares our backbone and adaptation framework, DCA delivers 7.11% SR improvement, validating dynamic compression effectiveness over recurrent approaches. DCA outperforms concatenation-based approaches: it surpasses No-Adapt by 6.47% in SR while matching NaVid-IL performance despite using a smaller backbone (3B vs. 7B parameters) and standard training rather than auxiliary co-training. The competitive Oracle Success (OS) Rate demonstrates effective instruction comprehension. These results establish DCA’s superior efficiency-performance trade-offs. Figure 5: Attention heat map of the Memory Compression Module with initial compression vectors as queries and encoded historical frames as keys, indexed by timesteps. Please note that the initial 3030 timesteps are truncated for improved visualization clarity across three representative episodes. Table 3: The ablative investigation on feature adaptation, context compression, and compression vector length. VLN-CE R2R Val-Unseen Type TL NE↓ OS↑ SR↑ SPL↑ !15 Feature Fusing !15FiLM Adapting !152.32 !1511.4 !156.25 !155.47 !155.26 λ=0.5λ=0.5 7.34 7.21 14.6 10.12 9.66 !15λ=0.8λ=0.8 !156.59 !157.01 !1517.8 !1511.4 !159.54 !15 Context Compression !15Instruction Attention !157.23 !156.90 !158.86 !157.23 !156.99 C=24C=24 12.1 12.3 8.12 6.94 5.12 !15C=48C=48 !1510.5 !1511.4 !1510.27 !156.88 !155.64 C=64C=64 8.30 8.64 17.6 9.23 8.77 !20Ours !20Full Setting !206.73 !206.77 !2025.3 !2013.7 !2012.9 4.4 Visualizations of Dynamic Context Compression To understand our compression efficiency mechanisms, we analyze attention patterns within the Memory Compression Module to identify which historical frames contribute most significantly to navigation decisions. This reveals how our method achieves computational efficiency through selective temporal prioritization rather than uniform compression. Fig. 4 presents attention visualization for our compression module. Following Eq. (1), we compute per-head attention scores ScpsS_cps across the trajectory at final navigation step T=68T=68. These scores weight frame features from t=0t=0 to t=67t=67 during compression. Aggregating scores across heads produces a per-timestep attention profile demonstrating temporal selection priorities. The analysis reveals selective focus on semantically relevant observations, validating our compression approach. Early observations without visible targets (bedroom door) receive negligible attention weights, reflecting limited utility for decision-making at T=68T=68. Conversely, frames containing critical visual cues exhibit pronounced attention peaks: the target door at t=53t=53 and t=61t=61, and bedroom interior at t=67t=67 show substantially elevated weights. Fig. 5 confirms concentrated focus on later frames where goal locations become visible. These patterns confirm our method’s ability to identify and prioritize critical contextual features while efficiently discarding temporally irrelevant information, achieving efficiency through intelligent temporal filtering rather than indiscriminate reduction. Additional analyses appear in Appendix. 4.5 Ablation Studies Table 3 analyzes design choices of our method. We compare our default context injection via Eq. (3), which adds compressed context with learnable coefficient λ, against FiLM-based fusion applying zk+1←zk+1+(αzk+1context+β)z_k+1← z_k+1+(α\,z_k+1^context+β) with zero-initialized parameters α and β following [29]. FiLM fusion shows substantial SR and SPL drops because additional scaling parameters hinder stable context integration. Varying λ (0.50.5, 0.80.8) shows larger values consistently improve success and SPL, confirming the importance of weighted historical context. For compression designs, we augment the compression module with cross-attention over instruction embeddings, hypothesizing context relevance correlates with instruction semantics. This variant underperforms direct compression due to data quality issues in R2R where instructions and trajectories are misaligned. Additionally, we examine the effect of memory capacity by varying memory tokens C (2424, 4848, 6464). Performance improvements correlate with increased C values. This finding indicates that greater capacity captures richer temporal patterns, though excessive increases risk overfitting. 5 Conclusion In this study, we introduced DCA, a lightweight framework that efficiently integrates historical context into pretrained VLMs without inflating input token lengths. Our proposed approach employed a Memory Compression Module to distill past frame embeddings into fixed-size learnable memory vectors and a Memory Integration Module to adapt these compressed representations into each Transformer layer. This design preserved the pretrained VLM architecture while achieving linear scaling with extended context lengths. Our extensive evaluations on downstream VLN tasks demonstrated that DCA can achieve superior efficiency-performance trade-offs compared to existing approaches. 6 Acknowledgement The authors gratefully acknowledge the support from the National Science and Technology Council (NSTC) in Taiwan under grant numbers NSTC 114-2221-E-002-069-MY3, NSTC 113-2221-E-002-212-MY3, and NSTC 114-2218-E-A49-026. This research was also supported by the NVIDIA Academic Grant Program. The authors would also like to express their appreciation for the donation of the GPUs from NVIDIA Corporation and NVIDIA AI Technology Center (NVAITC) used in this work. Furthermore, the authors extend their gratitude to the National Center for High-Performance Computing (NCHC) for providing computational and storage resources. The authors also thank the NVIDIA Taipei-1 supercomputer for providing essential computing resources. References [1] M. Abdin, J. Aneja, S. B, C. Mendes, W. Chen, A. Giorno, R. Eldan, S. Gopi, S. Gunasekar, M. Javaheripi, P. Kauffmann, Y. T. Lee, Y. Li, A. Nguyen, G. de Rosa, O. Saarikivi, A. Salim, S. Shah, M. Santacroce, and Y. Zhang (2024) Phi-2: the surprising power of small language models. In arXiv preprint, Cited by: §3. [2] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. (2023) Gpt-4 technical report. arXiv preprint. Cited by: §2. [3] J. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. (2022) Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems (NeurIPS). Cited by: §1. [4] D. An, Y. Qi, Y. Li, Y. Huang, L. Wang, T. Tan, and J. Shao (2023) BEVBert: multimodal map pre-training for language-guided navigation. In In Proc. of International Conf. on Comp. Vision (ICCV), Cited by: §1, §2. [5] D. An, H. Wang, W. Wang, Z. Wang, Y. Huang, K. He, and L. Wang (2023) ETPNav: evolving topological planning for vision-language navigation in continuous environments. In IEEE Transactions on Pattern Analysis and Machine Intelligence, Cited by: §1, §2. [6] P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. Van Den Hengel (2018) Vision-and-language navigation: interpreting visually-grounded navigation instructions in real environments. In In Proc. of Con. on Comp. Vision and Pattern Recognition (CVPR), Cited by: §2. [7] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh (2015) Vqa: visual question answering. In In Proc. of International Conf. on Comp. Vision (ICCV), Cited by: §1. [8] J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. (2023) Qwen technical report. arXiv preprint. Cited by: §1, §2. [9] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. (2023) Rt-2: vision-language-action models transfer web knowledge to robotic control. arXiv preprint. Cited by: §1. [10] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al. (2022) Rt-1: robotics transformer for real-world control at scale. arXiv preprint. Cited by: §1. [11] J. Chen, B. Lin, X. Liu, L. Ma, X. Liang, and K. K. Wong (2025) Affordances-oriented planning using foundation models for continuous vision-language navigation. In In Proc. of the Conference on Artificial Intelligence (AAAI), Cited by: §2. [12] J. Chen, B. Lin, R. Xu, Z. Chai, X. Liang, and K. Wong (2024) MapGPT: map-guided prompting with adaptive path planning for vision-and-language navigation. In In Proc. of Association for Computational Linguistics (ACL), Cited by: §2. [13] P. Chen, D. Ji, K. Lin, R. Zeng, T. H. Li, M. Tan, and C. Gan (2022) Weakly-supervised multi-granularity map learning for vision-and-language navigation. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §2. [14] S. Chen, P. Guhur, C. Schmid, and I. Laptev (2021) History aware multimodal transformer for vision-and-language navigation. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §2, Table 2. [15] S. Chen, P. Guhur, M. Tapaswi, C. Schmid, and I. Laptev (2022) Think global, act local: dual-scale graph transformer for vision-and-language navigation. In In Proc. of Con. on Comp. Vision and Pattern Recognition (CVPR), Cited by: §2. [16] Z. Chen, W. Wang, H. Tian, S. Ye, Z. Gao, E. Cui, W. Tong, K. Hu, J. Luo, Z. Ma, et al. (2024) How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences. Cited by: §1, §2. [17] Z. Deng, K. Narasimhan, and O. Russakovsky (2020) Evolving graphical planner: contextual global planning for vision-and-language navigation. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2. [18] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby (2020) An image is worth 16x16 words: transformers for image recognition at scale. In Proc. of International Conference on Learning Representations (ICLR). Cited by: §3.1.1, §3. [19] D. Fried, R. Hu, V. Cirik, A. Rohrbach, J. Andreas, L. Morency, T. Berg-Kirkpatrick, K. Saenko, D. Klein, and T. Darrell (2018) Speaker-follower models for vision-and-language navigation. Advances in Neural Information Processing Systems (NeurIPS). Cited by: §2. [20] M. GenAI (2023) Llama 2: open foundation and fine-tuned chat models. arXiv preprint. Cited by: §2. [21] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. (2024) The llama 3 herd of models. arXiv preprint. Cited by: §2. [22] P. Guhur, M. Tapaswi, S. Chen, I. Laptev, and C. Schmid (2021) Airbert: in-domain pretraining for vision-and-language navigation. In In Proc. of International Conf. on Comp. Vision (ICCV), Cited by: §1, §2. [23] K. He, Y. Jing, Y. Huang, Z. Lu, D. An, and L. Wang (2024) Memory-adaptive vision-and-language navigation. Pattern Recognition (PR). Cited by: §2. [24] Y. Hong, Z. Wang, Q. Wu, and S. Gould (2022) Bridging the gap between learning in discrete and continuous environments for vision-and-language navigation. In In Proc. of Con. on Comp. Vision and Pattern Recognition (CVPR), Cited by: Table 2, Table 2, Table 2, Table 2. [25] Y. Hong, Q. Wu, Y. Qi, C. R. Opazo, and S. Gould (2020) A recurrent vision-and-language bert for navigation. CoRR. Cited by: §1, §2. [26] M. Z. Hossain, F. Sohel, M. F. Shiratuddin, and H. Laga (2019) A comprehensive survey of deep learning for image captioning. ACM Computing Surveys (CsUR). Cited by: §1. [27] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. (2022) Lora: low-rank adaptation of large language models.. In Proc. of International Conference on Learning Representations (ICLR). Cited by: §1. [28] S. Karamcheti, S. Nair, A. Balakrishna, P. Liang, T. Kollar, and D. Sadigh (2024) Prismatic vlms: investigating the design space of visually-conditioned language models. In International Conf. on Machine Learning (ICML), Cited by: §2, §3. [29] M. J. Kim, C. Finn, and P. Liang (2025) Fine-tuning vision-language-action models: optimizing speed and success. arXiv preprint. Cited by: §1, §4.5. [30] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. P. Foster, P. R. Sanketi, Q. Vuong, et al. (2024) OpenVLA: an open-source vision-language-action model. In Conference on Robot Learning (CoRL), Cited by: §1, §3. [31] W. Kim, B. Son, and I. Kim (2021) Vilt: vision-and-language transformer without convolution or region supervision. In International Conf. on Machine Learning (ICML), Cited by: §1, §2. [32] J. Krantz, A. Gokaslan, D. Batra, S. Lee, and O. Maksymets (2021) Waypoint models for instruction-guided navigation in continuous environments. In In Proc. of International Conf. on Comp. Vision (ICCV), Cited by: Table 2. [33] J. Krantz and S. Lee (2022) Sim-2-sim transfer for vision-and-language navigation in continuous environments. In In Proc. of Eur. Conf. on Comp. Vision (ECCV), Cited by: §2, Table 2. [34] J. Krantz, E. Wijmans, A. Majumdar, D. Batra, and S. Lee (2020) Beyond the nav-graph: vision-and-language navigation in continuous environments. In In Proc. of Eur. Conf. on Comp. Vision (ECCV), Cited by: §1, §2, §4.1, §4.1, Table 2, Table 2, Table 2, Table 2. [35] J. Li, D. Li, S. Savarese, and S. Hoi (2023) Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conf. on Machine Learning (ICML), Cited by: §1. [36] Y. Li, C. Wang, and J. Jia (2024) Llama-vid: an image is worth 2 tokens in large language models. In In Proc. of Eur. Conf. on Comp. Vision (ECCV), Cited by: §2, §3.1.1. [37] B. Lin, Y. Nie, Z. Wei, J. Chen, S. Ma, J. Han, H. Xu, X. Chang, and X. Liang (2025) Navcot: boosting llm-based vision-and-language navigation via learning disentangled reasoning. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §2. [38] K. Lin, P. Chen, D. Huang, T. H. Li, M. Tan, and C. Gan (2023) Learning vision-and-language navigation from youtube videos. In In Proc. of International Conf. on Comp. Vision (ICCV), Cited by: §1, §2. [39] H. Liu, C. Li, Q. Wu, and Y. J. Lee (2023) Visual instruction tuning. Advances in Neural Information Processing Systems (NeurIPS). Cited by: §1, §2. [40] R. Liu, X. Wang, W. Wang, and Y. Yang (2023) Bird’s-eye-view scene graph for vision-language navigation. In In Proc. of Con. on Comp. Vision and Pattern Recognition (CVPR), Cited by: §2. [41] Y. Liu, D. Chi, S. Wu, Z. Zhang, Y. Hu, L. Zhang, Y. Zhang, S. Wu, T. Cao, G. Huang, et al. (2025) SpatialCoT: advancing spatial reasoning through coordinate alignment and chain-of-thought for embodied task planning. arXiv preprint. Cited by: §2. [42] Y. Long, X. Li, W. Cai, and H. Dong (2024) Discuss before moving: visual language navigation via multi-expert discussions. In International Conference on Robotics and Automation (ICRA), Cited by: §2. [43] Y. Ma, Z. Song, Y. Zhuang, J. Hao, and I. King (2024) A survey on vision-language-action models for embodied ai. arXiv preprint. Cited by: §1. [44] A. Majumdar, A. Shrivastava, S. Lee, P. Anderson, D. Parikh, and D. Batra (2020) Improving vision-and-language navigation with image-text pairs from the web. In In Proc. of Eur. Conf. on Comp. Vision (ECCV), Cited by: §2. [45] O. Mañas, P. Rodriguez, S. Ahmadi, A. Nematzadeh, Y. Goyal, and A. Agrawal (2022) Mapl: parameter-efficient adaptation of unimodal pre-trained models for vision-language few-shot prompting. arXiv preprint arXiv:2210.07179. Cited by: §2. [46] B. Pan, R. Panda, S. Jin, R. Feris, A. Oliva, P. Isola, and Y. Kim (2024) LangNav: language as a perceptual representation for navigation. In In Proc. of Association for Computational Linguistics (ACL), Cited by: §2. [47] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever (2021) Learning transferable visual models from natural language supervision. In International Conf. on Machine Learning (ICML), Cited by: §2, §3.1.1, §3. [48] S. Raychaudhuri, S. Wani, S. Patel, U. Jain, and A. X. Chang (2021) Language-aligned waypoint (law) supervision for vision-and-language navigation in continuous environments. Conference on Empirical Methods in Natural Language Processing (EMNLP). Cited by: Table 2. [49] N. Savinov, A. Dosovitskiy, and V. Koltun (2018) Semi-parametric topological memory for navigation. In In Proc. of International Conference on Learning Representations (ICLR), Cited by: §2. [50] D. Shah, B. Osiński, S. Levine, et al. (2023) Lm-nav: robotic navigation with large pre-trained models of language, vision, and action. In Conference on robot learning (CoRL), Cited by: §2. [51] Y. Song, M. Gianni, C. Yang, K. Lin, T. Chiu, A. Nguyen, and C. Lee (2024) Fine-grained alignment in vision-and-language navigation through bayesian optimization. arXiv preprint. Cited by: §2. [52] Y. Song, A. Nguyen, and C. Lee (2024) Learning to terminate in object navigation. In Proceedings of the 15th Asian Conference on Machine Learning (ACML), Cited by: §2. [53] H. Tan, L. Yu, and M. Bansal (2019) Learning to navigate unseen environments: back translation with environmental dropout. In In Proc. of Association for Computational Linguistics (ACL), Cited by: §2. [54] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. (2023) Llama: open and efficient foundation language models. arXiv preprint. Cited by: §2. [55] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS). Cited by: §1. [56] Z. Wang, X. Li, J. Yang, Y. Liu, J. Hu, M. Jiang, and S. Jiang (2024) Lookahead exploration with neural radiance representation for continuous vision-language navigation. In In Proc. of Con. on Comp. Vision and Pattern Recognition (CVPR), Cited by: §2. [57] Z. Wang, X. Li, J. Yang, Y. Liu, and S. Jiang (2023) GridMM: grid memory map for vision-and-language navigation. In In Proc. of International Conf. on Comp. Vision (ICCV), Cited by: §2. [58] K. Weerakoon, M. Elnoor, G. Seneviratne, V. Rajagopal, S. H. Arul, J. Liang, M. K. M. Jaffar, and D. Manocha (2024) BehAV: behavioral rule guided autonomy using vlms for robot navigation in outdoor scenes. CoRR. Cited by: §2. [59] W. Wu, T. Chang, X. Li, Q. Yin, and Y. Hu (2024) Vision-language navigation: a survey and taxonomy. Neural Computing and Applications. Cited by: §1. [60] C. Xu, H. T. Nguyen, C. Amato, and L. L. Wong (2023) Vision and language navigation in the real world via online visual language mapping. In 2nd Workshop on Language and Robot Learning: Language as Grounding, Cited by: §2. [61] H. Zhang, X. Li, and L. Bing (2023) Video-llama: an instruction-tuned audio-visual language model for video understanding. In Conference on Empirical Methods in Natural Language Processing (EMNLP) Demos, Cited by: §1. [62] J. Zhang, K. Wang, S. Wang, M. Li, H. Liu, S. Wei, Z. Wang, Z. Zhang, and H. Wang (2025) Uni-navid: a video-based vision-language-action model for unifying embodied navigation tasks. arXiv preprint. Cited by: §1, §2, §2, §3. [63] J. Zhang, K. Wang, R. Xu, G. Zhou, Y. Hong, X. Fang, Q. Wu, Z. Zhang, and H. Wang (2024) Navid: video-based vlm plans the next step for vision-and-language navigation. arXiv preprint. Cited by: §1, §1, §2, §2, §3.1.1, §3, §4.1, Table 2, Table 2. [64] L. Zhang, X. Hao, Q. Xu, Q. Zhang, X. Zhang, P. Wang, J. Zhang, Z. Wang, S. Zhang, and R. Xu (2025) MapNav: a novel memory representation via annotated semantic maps for vlm-based vision-and-language navigation. arXiv preprint. Cited by: §1, §2, §3. [65] R. Zhang, J. Han, C. Liu, P. Gao, A. Zhou, X. Hu, S. Yan, P. Lu, H. Li, and Y. Qiao (2023) Llama-adapter: efficient fine-tuning of language models with zero-init attention. arXiv preprint. Cited by: §1. [66] D. Zheng, S. Huang, L. Zhao, Y. Zhong, and L. Wang (2024) Towards learning a generalist model for embodied navigation. In In Proc. of Con. on Comp. Vision and Pattern Recognition (CVPR), Cited by: §2. [67] G. Zhou, Y. Hong, Z. Wang, X. E. Wang, and Q. Wu (2024) Navgpt-2: unleashing navigational reasoning capability for large vision-language models. In In Proc. of Eur. Conf. on Comp. Vision (ECCV), Cited by: §1, §2. [68] G. Zhou, Y. Hong, and Q. Wu (2024) Navgpt: explicit reasoning in vision-and-language navigation with large language models. In In Proc. of the Conference on Artificial Intelligence (AAAI), Cited by: §2.