Paper deep dive
ReMem: Streaming Video Understanding With Long Context Retention
Li Yiheng, He Xu, Wang Shaobo, Shao Ling, Lu Shijian
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Despite their impressive performance on a wide range of video understanding tasks, current Vision Language Models (VLMs) are predominantly designed for offline scenarios and struggle to handle online streaming videos that demand low latency response. Several studies have explored memory and token compression strategies in an attempt to adapt offline VLMs for streaming video understanding tasks. However, through our probing experiment, we identify that most existing works tend to progressively lose long context information as length of input stream increases. To address this, we propose ReMem, a novel training-free adaptation technique that enables VLMs to process streaming videos of arbitrary lengths while improving their long context information retention capability. ReMem exploits memory from two perspectives, implemented as two core components. The Streaming Context Memory (SCM) continuously compresses historical context with query-independent attention. The Retrieved Vision Memory (RVM) then retrieves the most salient, query-relevant context from memory to augment the VLM's input. Comprehensive experiments demonstrate that the proposed ReMem achieves state-of-the-art (SOTA) performance across a variety of widely used benchmarks, spanning both streaming video and general long video understanding tasks.
Tags
Links
- Source: https://arxiv.org/abs/2610.05940v1
- Canonical: https://arxiv.org/abs/2610.05940v1
Trouble viewing inline? Open PDF directly →
Full Text
67,594 characters extracted from source content.
Expand or collapse full text
ReMem: Streaming Video Understanding With Long Context Retention Li Yiheng 1∗ , He Xu 1∗ , Wang Shaobo 2 , Shao Ling 3 , Lu Shijian 1† 1 Nanyang Technological University 2 Shanghai Jiaotong University 3 University of the Chinese Academy of Sciences Abstract Despite their impressive performance on a wide range of video understanding tasks, current Vision Language Mod- els (VLMs) are predominantly designed for offline scenarios and struggle to handle online streaming videos that demand low latency response. Several studies have explored mem- ory and token compression strategies in an attempt to adapt offline VLMs for streaming video understanding tasks. How- ever, through our probing experiment, we identify that most existing works tend to progressively lose long context informa- tion as length of input stream increases. To address this, we propose ReMem, a novel training-free adaptation technique that enables VLMs to process streaming videos of arbitrary lengths while improving their long context information reten- tion capability. ReMem exploits memory from two perspec- tives, implemented as two core components. The Streaming Context Memory (SCM) continuously compresses historical context with query-independent attention. The Retrieved Vi- sion Memory (RVM) then retrieves the most salient, query- relevant context from memory to augment the VLM’s input. Comprehensive experiments demonstrate that the proposed ReMem achieves state-of-the-art (SOTA) performance across a variety of widely used benchmarks, spanning both streaming video and general long video understanding tasks. Introduction Vision Language Models (VLMs) have demonstrated re- markable advancements in a wide range of multi-modal tasks, from single image question-answering (Fu et al. 2025a; Chen et al. 2024b) to complex multi-turn dialogue (Liu et al. 2024; Li and Tajbakhsh 2023) and long video understanding (Fu et al. 2025b; Wu et al. 2024a). However, this progress has largely been confined to offline settings, where models presume access to the entire visual input before generating a response (Bai et al. 2025b; Wang et al. 2024a; Bai et al. 2023; Chen et al. 2024f,e,d; Zhang et al. 2024b; Zohar et al. 2024). This assumption might be impractical for many real- time applications where visual data is provided as a continu- ous stream. Applications such as autonomous driving (Shao et al. 2024) require fast response and expect low latency from VLMs. Developing models that can operate effectively in these online, streaming environments remains a challenge. ∗ These authors contributed equally. † Corresponding author. Recent efforts have sought to adapt VLMs to the streaming video setting by lowering the latency of the answer genera- tion process (Zhang et al. 2025a; Di et al. 2025a; Wang et al. 2025a). One key strategy is token compression (Yang et al. 2025a; Tao et al. 2025; Chen et al. 2024c), which reduces the input sequence length to speed up VLM inference. In addition, memory-based modules (Song et al. 2024, 2025; Wang et al. 2024c) are often used in conjunction with com- pression. These modules store long context information from older segments of the input video stream, allowing relevant context to be retrieved later when a user query arises. While such methods effectively reduce the token length for VLM transformer operations (Vaswani et al. 2017) and speed up the inference process, they often come at the cost of long context retention. We demonstrate this tradeoff us- ing a variant of visual Needle-In-A-Haystack (NIAH) prob- ing experiment (Wu et al. 2024b). As shown in fig. 1, our results reveal that as the haystack size increases, existing streaming models such as FluxMem (Xie et al. 2026) and OASIS (Liang et al. 2026) suffer a more pronounced drop in retrieval accuracy than the Qwen2.5VL (Bai et al. 2025b) base model. This indicates that these methods increasingly fail to preserve crucial contextual information, progressively losing key details as visual sequences grow longer. In con- trast, our proposed method, ReMem, consistently surpasses these evaluated models by effectively capturing and retaining key information despite the increasing context size. This limitation motivated us to design a streaming VLM adaptation that retains long-term context while maintaining a short input token length for efficiency. We introduce ReMem, a training-free VLM adaptation strategy that seamlessly en- ables offline pre-trained VLMs to process online streaming data while preserving long-term context. ReMem consists of two main components: Streaming Context Memory (SCM) and Retrieved Vision Memory (RVM). SCM maintains a fixed-length memory module that continuously condenses historical visual information using attention-based compres- sion. RVM then augments the model’s input by retrieving the most relevant, query-specific vision tokens. Our experi- mental results show that ReMem achieves significant perfor- mance gains on streaming video benchmarks, especially on tasks requiring contextual understanding. 1 arXiv:2610.05940v1 [cs.CV] 5 Oct 2026 In summary, our main contributions are: 1. We use visual NIAH probing experiments to demon- strate that existing memory based streaming VLM designs suffer from the progressive loss of long-term context. 2. We introduce ReMem, a novel, training-free adaptation strategy that equips existing offline VLMs to handle stream- ing video of arbitrary length while simultaneously improving their ability to retain long-context information. 3. We conduct comprehensive experiments on stream- ing and general long video understanding benchmarks. The results demonstrate that ReMem significantly outperforms baselines and competing streaming methods, particularly on streaming tasks, thereby validating the effectiveness of our design. Related Works Vision Language Models VLMs have developed tremendously since their early forms (Liu et al. 2023; Alayrac et al. 2022; Dai et al. 2023; Zhu et al. 2023). However, the basic structure of VLMs largely remains the same. They typically consist of a vision encoder (Doso- vitskiy et al. 2021), a connector (Liu et al. 2023), and an LLM backbone (Peng et al. 2023). They function by concatenat- ing visual features with text embeddings to auto-regressively generate outputs (Zhu et al. 2025; Yu et al. 2025). This foun- dational design is almost exclusively intended for offline set- tings, where the entire visual input is provided at once. Con- sequently, the architecture is inherently constrained by the LLM’s fixed context window, making these VLMs ill-suited for real-world streaming applications where visual data ar- rives continuously and must be processed with low latency. Streaming Video Understanding Streaming video understanding aims to transition traditional offline VLMs toward practical online scenarios, where mod- els must process video streams continuously rather than re- lying on entire preloaded video sequences. In such settings, VLMs are required to maintain long context consistency, rea- son over evolving visual evidence, and generate responses interactively and proactively in real time. Early works (Geest et al. 2016; Xu et al. 2019, 2021; Zhao and Krähenbühl 2022; Girdhar and Grauman 2021) in the action detection and an- ticipation domains address this constraint by causally pro- cessing the historical video inputs together with the current observation. More recent approaches tailor Video-LLMs for online use by modifying training objectives, memory mech- anisms, or response strategies. For instance, VideoLLM- Online (Chen et al. 2024a) introduces an online training ob- jective and stream-aware decoding, but heavily compresses each frame into only a few visual tokens, limiting fine-grained perception. FlashVStream (Zhang et al. 2025a) employs temporal–spatial memory and sequential retrieval for real- time reasoning over long streams, whereas Dispider (Qian et al. 2025) disentangles perception, decision, and reaction to support active interaction. LiveVLM (Ning et al. 2025), a training-free framework designed specifically for streaming, online video understanding, constructs a streaming-oriented KV cache that retains useful details while pruning redun- dant history. TimeChat-Online (Yao et al. 2025a) tackles visual redundancy in streaming inputs via a Differential Token Drop (DTD) module that preserves salient tempo- ral changes while discarding static frames. StreamBridge- 7B (Wang et al. 2025a), a simple yet effective framework that turns offline Video-LLMs into streaming-capable mod- els, relies on a memory buffer with round-decayed com- pression and a lightweight decoupled activation module to enable multi-turn, proactive responses. Complementary ef- forts such as Lion-FS (Li et al. 2025a) and LiveCC (Chen et al. 2025a) explore scalable data/optimization strategies and speech-conditioned streaming, while MMDuet (Wang et al. 2024b) and ViSpeak (Fu et al. 2025c) introduce task-specific heads or duet-style interaction formats to decide when the model should speak. Despite these advances, most methods either require training, modify the model architecture, or still struggle to maintain contextual fidelity over long visual se- quences so that our training-free ReMem is proposed. Visual Token Compression Query-based compression strategies, like the Q-former (Li et al. 2023), select features most salient to the current user prompt. This approach is unsuitable for streaming video, which often involves multi-turn dialogue. Compressing vi- sual tokens based on a single, specific query would irre- versibly discard information that, while irrelevant to the cur- rent prompt, may be essential for answering subsequent user queries. Due to the information sparsity in video tokens (Zhang et al. 2024a; Yao et al. 2025b), some compression strategies focus on reducing the spatial and temporal redundancies of visual tokens (Zhang et al. 2025a; Yao et al. 2025b). How- ever, the resulting compressed visual tokens differ from the model’s original training data distribution and deviate from its learned inductive bias. Consequently, additional model fine-tuning is required to adapt to this new input token dis- tribution. To overcome this fine-tuning requirement, attention score based compression strategies (Yang et al. 2025a; Chen et al. 2024c; Tao et al. 2025; Wang et al. 2025b; Yang et al. 2025b) select visual tokens or kV-cache vectors based on the model’s own attention patterns. By retaining only the tokens with high attention scores, this approach creates a compressed to- ken set that is less prone to out-of-distribution behavior and can maintain model performance without additional train- ing. Given the effectiveness and resource-efficiency of these attention-based compression methods, we follow a similar design principle for our vision token compression module. Memory Based VLM To overcome the limitations of compression strategies and the restriction to fixed-length video inputs, researchers have pro- posed memory based approaches. Memory mechanisms are widely applied across long sequence modelling tasks, includ- ing forecasting, recommendation systems, and video object tracking. In the context of video understanding, various de- signs have been explored: ReKV(Di et al. 2025b)introduces 2 a compressed KV-cache memory to retain earlier video infor- mation; MovieChat(Song et al. 2025) adopts a dual memory framework with long-term and short-term memory; How- ever, a common drawback of these methods is the lack of an upper bound on memory size in at least one of their modules. As a result, when processing unbounded streaming videos, the accumulated token length eventually exceeds the hard- ware’s maximum capacity. Other memory based solutions place explicit limits on memory usage. Flash-VStream(Zhang et al. 2025a) leverages separate memory modules for temporal and spatial modeling. InfiniPlotV (Kim et al. 2025) employs a memory compres- sion strategy that enforces a fixed token length regardless of input size, while StreamBridge (Wang et al. 2025a) intro- duces a round-decay mechanism that progressively merges earlier memory contexts with adjacent frames to maintain a constant memory footprint. However, such strategies in- evitably merge or compress important visual features, par- ticularly from earlier frames, leading to loss of long context information. In contrast, we propose an additional retrieval module that selectively extracts salient information from the memory buffer without merging or compressing critical de- tails. The retrieved features augment our fixed-size memory, thereby mitigating information loss caused by progressive memory updates. Probing Experiment Memory based streaming VLMs, such as (Zhang et al. 2025a; Wang et al. 2025a), are designed to process videos of arbi- trary length by maintaining a fixed-size context memory, which is continuously updated as new frames arrive. While this design is memory-efficient and achieves low latency, we hypothesize that it comes at the cost of long context retention, where crucial information from earlier in the input stream is progressively diluted or overwritten by newer, incoming data. To validate this hypothesis, we conduct a probing experi- ment using the Visual NIAH (Wu et al. 2024b) dataset. The task requires the VLM to answer a question where the correct answer is based solely on a "needle" image placed at the very beginning of the input sequence. We then progressively in- crease the context length by appending irrelevant "haystack" images. We conduct this experiment on both variants of the FlashVstream model (Zhang et al. 2025a), with the results presented in fig. 1. As shown in fig. 1, both the Qwen2.5-VL-7B (Bai et al. 2025b) base model and memory-augmented stream- ing VLMs suffer from accuracy degradation as the input context lengthens. Our results reveal that as the haystack size increases, existing streaming models such as FluxMem (Xie et al. 2026) and OASIS (Liang et al. 2026) suffer a more pronounced drop in retrieval accuracy. Their performance quickly drops to scores approaching a random guess (0.50) on this yes/no task. This steep decline strongly suggests that the underlying context retention capability is compromised by the memory mechanisms employed in these models. As their memory modules continuously update to process longer input se- quences, crucial visual information becomes progressively diluted by newer context, ultimately eroding their long-term Figure 1: Comparison of visual haystacks retrieval perfor- mance among Qwen2.5VL base model (Bai et al. 2025b), FluxMem (Xie et al. 2026), HERMES (Zhang et al. 2026), OASIS (Liang et al. 2026) and our method. As the haystack size increases, ReMem maintains more stable performance in long visual sequences, indicating that crucial contextual information is better preserved. memory. In contrast, our proposed method, ReMem, consis- tently surpasses these evaluated models by effectively cap- turing and retaining key information despite the increasing context size. Methodology Motivated by our probing experiments, we propose ReMem, a novel strategy that enables offline pretrained VLMs to pro- cess streaming video while preserving their long context un- derstanding capabilities. ReMem consists of two core com- ponents: • Streaming Context Memory (SCM): This module con- denses all incoming visual information into a fixed-length memory unit via attention-based visual token compres- sion. • Retrieved Vision Memory (RVM): This module enhances contextual features by extracting the most query-relevant visual tokens and re-injecting them into the model’s input. Preliminary We define the streaming video task over a sequence of dis- crete time steps T i , for i ∈ 0, 1, . . . , n. At each time step T i , a new clip of video frames of length C arrives from the stream. We can control the number of frames per video clip by setting different video sampling frequencies. These C frames pass through the VLM’s vision encoder and pro- jection layer to produce a corresponding set of vision tokens. We denote this token segment, corresponding to the frames at T i , as V i . Streaming Context Memory As shown in fig. 2, SCM is a fixed-length memory storage that iteratively updates as new video segments enter the model. 3 Figure 2: Overview of the proposed ReMem framework. The video stream is segmented to video chunks continuously. At each time step, the VLM attends to visual and contextual tokens, selectively storing informative representations into the SCM. Upon receiving a specific question query, we score each memory slot by cosine similarity, select the top vision tokens from the RVM, and integrate them with the ongoing context, yielding a compact representation. This update process is defined in eq. (1). We employ Gen- eral Query Compression (GQC) as the compression strategy, which we elaborate in section below. For the initial time step T 0 , the SCM tokens S 0 are initialised by compressing the first segment of vision tokens V 0 . For all subsequent time steps T i , the new memory state S i is derived by combining and compressing the previous context tokens S i−1 and the current vision tokens V i , with compression ration of r. The maximum size of SCM can be controlled by fixing r. This recursive design ensures that S i contains condensed information from all vision tokens V 0 up to V i within a fixed- size buffer. S i = GQC(V i ,r)if i = 0 GQC(S i−1 ⊕ V i ,r) if i > 0 (1) The SCM operates on a continuous, uninterrupted loop. While the VLM performs auto-regressive generation upon receiving a specific user query, the SCM continuously up- dates at each time step T i as the video stream is processed. General Query Compression In this section we elabo- rate the General Query Compression (GQC) strategy men- tioned in eq. (1). GQC selects vision tokens from V i based on their text-vision attention scores. This process is query- independent, as GQC relies on a static, general purpose prompt such as "Describe the image", to calculate these at- tention scores, rather than using a specific query from the user. The full input token I i is formed by concatenating the visual tokens V i and the general query tokens Q g . Following eq. (2), the attention scores of VLM input tokens can be obtained: Attn = softmax (W Q I i )(W K I i ) T √ d k (W V I i ) (2) where W Q , W K , and W V are the query, key and value weight matrix and d k is the vector of dimension. Given a compression ratio r, only the tokens with the high- est attention scores are kept to maintain the desired sequence length. Position Embedding Management Strategy VLMs of- ten use Rotary Position Encoding (RoPE) (Su et al. 2023) to model the relative spatial and temporal relationships between visual tokens. However, visual token compression can disrupt or discard this critical positional information. To preserve these spatial and temporal dependencies post-compression, we propose to maintain the original positional embeddings for the selected visual tokens. In addition, processing stream- ing video of arbitrary length presents a positional encoding challenge, as the indices for new frames continually increase. To manage this, we track the cumulative video length at each timestep. We introduce a positional offset P, representing the total count of previously processed frames, and add this offset to the local position indices of the current streaming in- put. This method ensures that the absolute temporal position of every frame is correctly registered through the position encoding. Retrieved Vision Memory Building upon our SCM design for processing efficiency, we further propose RVM to enhance the VLM’s long context re- tention capabilities. RVM selectively retrieves and "replays" 4 query-relevant video clips by appending them to the model’s input fig. 2. The core of RVM is a memory-efficient retrieval mech- anism. Each video clip at timestep T i is represented by a single key vector K i avg . This is achieved by average-pooling the key vectors of its visual tokens V i across the sequence length. These K i avg vectors are stored in the RVM buffer, which discards the oldest entries when its maximum length is exceeded. For retrieval, a matching query vector is generated. The query vectors Q Q s from the input query tokens Q s are simi- larly average-pooled into a single representation, Q s avg . To find the most relevant historical clips, we then calculate the cosine similarity between this pooled query vector Q s avg and each of the n buffered key vectors K i avg as shown in eq. (3). s i = Q s avg · K i avg ∥Q s avg ∥K i avg ∥ (3) where i = 0...n, s i is the resulting similarity score for the i-th video clip. Finally, the video clip corresponding to the highest sim- ilarity score is retrieved and appended to the current input stream, thereby recapturing key contextual information. Experiments Experiment Setup Models and Benchmarks. The ReMem framework acts as a plug-and-play adaptation for existing VLMs, enabling offline models to process streaming videos of arbitrary length. To verify the generality of our method, we apply ReMem on two widely used offline VLM baselines: LLaVA-OneVision- 7B (Li et al. 2024) and Qwen2.5-VL-7B(Bai et al. 2025b), and conduct experiments on both online streaming tasks and offline general video understanding benchmarks. For online video understanding tasks, we choose com- monly used StreamingBench (Lin et al. 2024) and OVOBench (Li et al. 2025b) evaluation benchmarks. Since ReMem is designed to improve the long context retention capability of memory based models, we specifically focus on contextual understanding tasks from StreamingBench (SQA, MCU and ACU subset) and OVOBench (EPM, ASI, HLD subset), which challenge models to precisely recall previous visual context and historical frames during streaming video intake. We report the results of online video understanding tasks in table 1. For general offline video understanding task, we choose EgoSchema (Mangalam, Akshulakov, and Malik 2023) and VideoMME (Fu et al. 2025b). EgoSchema provides a very long-form multiple-choice video QA benchmark, contain- ing over 5,000 human-curated question answer pairs across more than 250 hours of egocentric video, while VideoMME serves as a high-quality, full-spectrum multimodal evalua- tion benchmark for video analysis, covering clips from short snippets to hour-long videos. We follow the official evalua- tion protocols for each benchmark and report the results in table 2. Baselines and State-of-the-Arts (SOTA). We compare our method with a set of representative Video-LLMs cover- ing both offline and online settings, including Qwen3-VL- 8B (Bai et al. 2025a), Qwen2.5-VL-7B (Bai et al. 2025b), LLaVA-OneVision-7B (Li et al. 2024) VideoLLM-Online- 8B (Chen et al. 2024a), FlashVStream-7B (Zhang et al. 2025a), ReKV-7B (Di et al. 2025a), StreamKV-7B (Chen et al. 2025b), OASIS (Liang et al. 2026), Dispider (Qian et al. 2025), LiveVLM (Ning et al. 2025), and TimeChat- Online (Yao et al. 2025b). All models are evaluated under identical hardware platform (single H100) and decoding pro- tocols for a fair comparison. Main Results As summarised in table 1 and table 2, ReMem consistently outperforms both offline baselines and streaming-oriented Video-LLMs on StreamingBench(Lin et al. 2024) contex- tual understanding tasks and OVOBench backward tracing tasks, while remaining competitive on general video un- derstanding EgoSchema and VideoMME. Notably, despite being training-free, ReMem achieves a 20.0-point increase in the ACU evaluation and an 11.6-point increase in MCU when compared to the Qwen2.5-VL-7B baseline. This strong performance, especially on contextual understanding tasks, validates the effectiveness of our design in improving the long-term context retention of the base model. Being a plug- and-play training-free adaptation, ReMem achieves compa- rable results event compared to training-based strategies. This further proves the effectiveness of our design. Further- more, ReMem consistently improves performance on both streaming benchmarks across the Qwen2.5-VL-7B, Qwen3- VL-8B, and LLaVA-OneVision-7B base models, indicating the generalisability of our method. Although ReMem was designed for streaming tasks, it also generalises well to offline long-video understanding bench- marks. ReMem outperforms its baseline on both EgoSchema and VideoMME evaluation subset. In particular, ReMem out- performs the LLaVA-OneVision-7B base model by 2.4 points despite being designed for purely streaming task purposes. It also achieves comparable performance compared to its base models for longer video settings with VideoMME evalua- tion, which shows that our streaming memory design does not break the contextual inference for long contextual offline video. Probing Experiments Analysis To further validate ReMem’s effectiveness in long-term con- text retention, we evaluated it using the visual NIAH probing experiment. We compared ReMem against the Qwen2.5-VL- 7B baseline and show the result in fig. 1. ReMem achieves consistently higher accuracy than the baseline model across nearly all input image size settings. This finding suggests that our fixed-size memory module does not introduce the pathological long-term context loss seen in other streaming VLMs. Instead, our method successfully preserves long-term context, performing on par with or slightly better than the strong, non-streaming baseline model. 5 ModelFrames StreamingBench Contextual Understanding OVOBench Backward-Tracing SQAMCUACUAvgEPMASIHLDAvg Streaming Video VLMs (training-based) VideoLLM-online-8B2 fps30.829.224.228.022.218.812.217.7 FlashVStream-7B1 fps24.825.226.825.639.137.25.927.4 TimeChat-Online-7B1 fps41.631.639.237.555.959.59.741.7 Dispider1 fps34.827.739.6 34.048.555.44.336.1 Streaming Video VLMs (training-free) ReKV-LLaVA-OV-7B 0.5 fps30.428.031.229.954.255.722.844.2 LiveVLM-7B0.5 fps32.036.838.835.9– StreamKV-7B0.5 fps33.230.043.635.6– OASIS-Qwen2.5-VL-7B 1 fps45.634.237.439.148.858.930.946.2 Qwen2.5-VL-7B1 fps43.625.224.831.243.857.422.641.3 + ReMem (Ours)1 fps 45.6 (↑2.0) 36.8 (↑11.6) 44.8 (↑20.0) 42.4 (↑11.2) 44.3 (↑0.5) 60.8 (↑3.4) 29.6 (↑7.0) 44.9 (↑3.6) LLaVA-OV-7B1 fps31.633.235.633.554.255.421.543.7 + ReMem (Ours)1 fps 40.4 (↑8.8) 37.2 (↑4.0) 42.0 (↑6.4) 36.8 (↑5.2) 56.0 (↑1.8) 57.7 (↑2.3) 22.9 (↑1.4) 45.5 (↑1.8) Qwen3-VL-8B1 fps46.039.636.440.750.759.741.550.6 + ReMem (Ours)1 fps 47.2 (↑1.2) 41.0 (↑1.4) 42.5 (↑6.1) 43.6 (↑2.9) 52.9 (↑2.2) 60.8 (↑1.1) 43.4 (↑1.9) 52.4 (↑1.8) Table 1: Benchmark comparison on StreamingBench and OVOBench online video evaluation dataset. Further Experiments Method Ablation To evaluate the effectiveness of each component, we conduct ablation experiments on the SQA and ACU tasks of StreamingBench (Lin et al. 2024). We analyse the contributions of the SCM, the RVM, and the position embedding management strategy for the SCM. The results, reported in table 3, show that all compo- nents contribute positively over the Qwen2.5-VL-7B base- line. Deploying either SCM or RVM individually boosts performance on streaming tasks. Furthermore, our position embedding management strategy is also effective, enhancing performance compared to using continuous position indices post-compression. Finally, applying all proposed strategies yields the highest performance gain, indicating that our de- sign choices are both effective and complementary. Base Model ReMem ComponentsStreamingBench SCM Pos. Emb Mgmt. RVM SQAACU Qwen2.5-VL-7B✗36.934.8 Qwen2.5-VL-7B✓✗41.737.6 Qwen2.5-VL-7B✗✓37.240.5 Qwen2.5-VL-7B✓✗45.038.4 Qwen2.5-VL-7B✓45.644.8 Table 3: Ablation of memory components on Streaming- Bench. Hyperparameter Study The performance of ReMem is influenced by key hyperparameters within its SCM and RVM modules. We first analyse the impact of these parameters by examining the chunk size, defined as the number of video frames processed in each new clip V i . To isolate this effect, we evaluated the SQA and ACU subsets of StreamingBench (Lin et al. 2024) using only the SCM algorithm without retrieving additional video clips via the RVM, while varying the chunk size. As shown in table 4, model performance is highly sensi- tive to this setting. At a chunk size of 1, the SCM’s accuracy on both SQA and ACU subtasks is at its lowest, falling even below the Qwen2.5-VL-7B baseline. Performance improves smoothly as the chunk size increases, peaking at chunk size of 16. This strong positive correlation suggests that the SCM benefits from processing longer, more context-rich clips at each update iteration. This likely allows it to better capture temporal information between adjacent frames. In contrast, processing frames individually with chunk size = 1 results in inferior performance, indicating that GQC compression operates more effectively when leveraging medium-sized inter-frame context. We further evaluate the influence of the number of histor- ical frames retrieved on the overall performance in our RVM design. We vary the number of query-relevant historical clips retrieved by ReMem using the Qwen2.5-VL-7B base model and present the results on StreamingBench in table 5. We observe that the best performance is achieved when only a single clip is retrieved. While counter-intuitive, this observa- tion aligns with recent findings in SimpleStream (Shen et al. 2026), which demonstrate that simply selecting the most re- cent frames outperforms many baselines. This suggests that additional historical context can introduce distracting noise or exceed the “memory-perception” trade-off threshold of current models. Efficiency Analysis In this section we present a latency and memory consumption study of ReMem compared against some other baseline and SOTA streaming VLM models. 6 ModelFrames EgoSchemaVideoMME Subset † FullsetshortmediumlongAvg Streaming Video VLMs (training-based) VideoLLM-online-8B 2 fps–32.8– FlashVStream-7B1 fps38.4068.272.061.150.361.2 TimeChat-Online-7B1 fps–61.967.450.541.753.2 Dispider1 fps–55.666.153.749.756.5 Streaming Video VLMs (training-free) ReKV-LLaVA-OV-7B 0.5 fps64.2060.7069.755.449.358.1 LiveVLM-7B0.5 fps–59.0066.756.448.857.3 Qwen2.5-VL-7B1 fps61.4–72.166.955.464.8 + ReMem (Ours)1 fps 62.00 (↑0.6)–72.4 (↑0.1) 66.9 (↑0.0) 56.3 (↑0.9) 65.2 (↑0.4) LLaVA-OV-7B1 fps61.0–69.754.450.058.0 + ReMem (Ours)1 fps63.4 (↑2.4)–70.4 (↑0.7) 55.0 (↑0.6) 50.2 (↑0.2) 58.5 (↑0.5) Qwen3-VL-8B1 fps65.4–79.770.262.570.8 + ReMem (Ours)1 fps67.0 (↑1.6)–78.8(↓0.9) 70.9 (↑0.7) 62.9 (↑0.4) 70.9 (↑0.1) Table 2: Benchmark comparison on EgoSchema and VideoMME offline video evaluation dataset. † Full-set evaluation is unavail- able due to evaluation server issue; results are reported on the EgoSchema subset. Base ModelMethod Chunk Size StreamingBench SQA ACU Qwen2.5-VL-7B Official–36.934.8 Qwen2.5-VL-7B SCM135.331.0 Qwen2.5-VL-7B SCM238.632.3 Qwen2.5-VL-7B SCM441.837.6 Qwen2.5-VL-7B SCM1645.038.4 Qwen2.5-VL-7B SCM2442.033.2 Table 4: Video input chunk size analysis for Streaming Con- text Memory (SCM) Number of ClipsSQAMCUACU 145.636.844.8 241.634.041.2 341.629.238.2 Table 5: Influence of the number of retrieved historical clips on overall performance within the RVM design. We conduct both latency and memory experiments on StreamingBench (Lin et al. 2024) SQA subset for better evaluation efficiency. We select parameter setting of com- pression ratio r = 0.7, chunk size = 16 and fps = 1. Our experiments are conducted on a single Nvidia 80GB H100 GPU. As illustrated in fig. 3, ReMem maintains a constant la- tency of 1.0 second even as the number of input video frames scales to 512. This stability is a direct result of our chunk- based streaming architecture and fixed memory allocation. Furthermore, ReMem’s memory footprint plateaus at 32 GB once the buffer is fully utilized, demonstrating high hardware efficiency for processing arbitrarily long video streams. Figure 3: Left: response latency of different VLM models under streaming evaluation. Right: Maximum GPU memory consumption of different VLM models under streaming eval- uation. Conclusion In this paper, we addressed a key limitation in existing streaming VLMs: the progressive loss of long-term con- text in fixed-size memory architectures. To tackle this, we proposed ReMem, a novel, training-free adaptation strategy designed to achieve streaming video understanding with long context retention. ReMem’s dual-memory architecture fea- tures a Streaming Context Memory (SCM) for continuous, query-independent visual token compression and a Retrieved Vision Memory (RVM) for query-specific salient vision in- formation augmentation. Our comprehensive experiments on online evaluation benchmarks and general long-video benchmarks demon- strated that ReMem significantly outperforms both base- line models and competing memory-based methods. Cru- cially, our (NIAH) probing experiments confirmed that Re- Mem successfully preserves long-term context, avoiding the pathological information loss observed in other streaming approaches. As a flexible, training-free method, ReMem en- ables the deployment of pretrained VLMs in, real-world ap- plications. 7 References Alayrac, J.-B.; Donahue, J.; Luc, P.; Miech, A.; Barr, I.; et al. 2022. Flamingo: a Visual Language Model for Few-Shot Learning. arXiv:2204.14198. Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; and Zhou, J. 2023. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv preprint arXiv:2308.12966. Bai, S.; Cai, Y.; Chen, R.; Chen, K.; Chen, X.; et al. 2025a. Qwen3- VL Technical Report. arXiv:2511.21631. Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; et al. 2025b. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. Chen, J.; Lv, Z.; Wu, S.; Lin, K. Q.; Song, C.; Gao, D.; Liu, J.- W.; Gao, Z.; Mao, D.; and Shou, M. Z. 2024a. VideoLLM-online: Online Video Large Language Model for Streaming Video. In CVPR. Chen, J.; Zeng, Z.; Lin, Y.; Li, W.; Ma, Z.; and Shou, M. Z. 2025a. LiveCC: Learning Video LLM with Streaming Speech Transcrip- tion at Scale. arXiv:2504.16030. Chen, L.; Li, J.; Dong, X.; Zhang, P.; Zang, Y.; Chen, Z.; Duan, H.; Wang, J.; Qiao, Y.; Lin, D.; et al. 2024b. Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37: 27056–27087. Chen, L.; Zhao, H.; Liu, T.; Bai, S.; Lin, J.; Zhou, C.; and Chang, B. 2024c. An image is worth 1/2 tokens after layer 2: Plug-and- play inference acceleration for large vision-language models. In European Conference on Computer Vision, 19–35. Springer. Chen, Y.; Bai, X.; Wang, Z.; Bai, C.; Dai, Y.; Lu, M.; and Zhang, S. 2025b. StreamKV: Streaming Video Question-Answering with Segment-based KV Cache Retrieval and Compression. arXiv:2511.07278. Chen, Z.; Wang, W.; Cao, Y.; Liu, Y.; Gao, Z.; Cui, E.; Zhu, J.; Ye, S.; Tian, H.; Liu, Z.; et al. 2024d. Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling. arXiv preprint arXiv:2412.05271. Chen, Z.; Wang, W.; Tian, H.; Ye, S.; Gao, Z.; Cui, E.; Tong, W.; Hu, K.; Luo, J.; Ma, Z.; et al. 2024e. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 67(12): 220101. Chen, Z.; Wu, J.; Wang, W.; Su, W.; Chen, G.; et al. 2024f. Internvl: Scaling up vision foundation models and aligning for generic visual- linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 24185–24198. Dai, W.; Li, J.; Li, D.; Tiong, A. M. H.; Zhao, J.; Wang, W.; Li, B.; Fung, P.; and Hoi, S. 2023. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tun- ing. arXiv:2305.06500. Di, S.; Yu, Z.; Zhang, G.; Li, H.; Cheng, H.; Li, B.; He, W.; Shu, F.; Jiang, H.; et al. 2025a. Streaming Video Question-Answering with In-context Video KV-Cache Retrieval. In ICLR. Di, S.; Yu, Z.; Zhang, G.; Li, H.; Zhong, T.; Cheng, H.; Li, B.; He, W.; Shu, F.; and Jiang, H. 2025b. Streaming Video Question-Answering with In-context Video KV-Cache Retrieval. arXiv:2503.00540. Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv:2010.11929. Fu, C.; Chen, P.; Shen, Y.; Qin, Y.; Zhang, M.; Lin, X.; Yang, J.; Zheng, X.; Li, K.; Sun, X.; et al. 2025a. MME: A comprehensive evaluation benchmark for multimodal large language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Fu, C.; Dai, Y.; Luo, Y.; Li, L.; Ren, S.; et al. 2025b. Video- MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. arXiv:2405.21075. Fu, S.; Yang, Q.; Li, Y.-M.; Peng, Y.-X.; Lin, K.-Y.; Wei, X.; Hu, J.-F.; Xie, X.; and Zheng, W.-S. 2025c. ViSpeak: Visual Instruction Feedback in Streaming Videos. arXiv:2503.12769. Geest, R. D.; Gavves, E.; Ghodrati, A.; Li, Z.; Snoek, C.; and Tuytelaars, T. 2016. Online Action Detection. arXiv:1604.06506. Girdhar, R.; and Grauman, K. 2021. Anticipative Video Trans- former. arXiv:2106.02036. Kim, M.; Shim, K.; Choi, J.; and Chang, S. 2025. InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understanding. arXiv preprint arXiv:2506.15745. Li, B.; Zhang, Y.; Guo, D.; Zhang, R.; Li, F.; Zhang, H.; Zhang, K.; Zhang, P.; Li, Y.; Liu, Z.; et al. 2024. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326. Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023. BLIP-2: Bootstrap- ping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. arXiv:2301.12597. Li, S.; and Tajbakhsh, N. 2023. Scigraphqa: A large-scale synthetic multi-turn question-answering dataset for scientific graphs. arXiv preprint arXiv:2308.03349. Li, W.; Hu, B.; Shao, R.; Shen, L.; and Nie, L. 2025a. LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistant. arXiv:2503.03663. Li, Y.; Niu, J.; Miao, Z.; Ge, C.; Zhou, Y.; et al. 2025b. OVO- Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding? arXiv:2501.05510. Liang, Z.; Li, J.; Chen, W.; Zhang, Y.; Lu, H.; and Li, G. 2026. OASIS: On-Demand Hierarchical Event Memory for Streaming Video Reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Lin, J.; Fang, Z.; Chen, C.; Wan, Z.; Luo, F.; Li, P.; Liu, Y.; and Sun, M. 2024. StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding. arXiv preprint arXiv:2411.03628. Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual Instruction Tuning. Liu, Z.; Chu, T.; Zang, Y.; Wei, X.; Dong, X.; Zhang, P.; Liang, Z.; Xiong, Y.; Qiao, Y.; Lin, D.; et al. 2024. Mmdu: A multi- turn multi-image dialog understanding benchmark and instruction- tuning dataset for lvlms. Advances in Neural Information Process- ing Systems, 37: 8698–8733. Mangalam, K.; Akshulakov, R.; and Malik, J. 2023. EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Un- derstanding. arXiv:2308.09126. Ning, Z.; Liu, G.; Jin, Q.; Ding, W.; Guo, M.; and Zhao, J. 2025. LiveVLM: Efficient Online Video Understanding via Streaming- Oriented KV Cache and Retrieval. arXiv:2505.15269. Peng, B.; Li, C.; He, P.; Galley, M.; and Gao, J. 2023. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277. Qian, R.; Ding, S.; Dong, X.; Zhang, P.; Zang, Y.; et al. 2025. Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction. arXiv:2501.03218. 8 Shao, H.; Hu, Y.; Wang, L.; Song, G.; Waslander, S. L.; Liu, Y.; and Li, H. 2024. Lmdrive: Closed-loop end-to-end driving with large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 15120–15130. Shen, Y.; Tian, S.; Yang, J.; and Liu, Z. 2026. A Simple Baseline for Streaming Video Understanding. arXiv:2604.02317. Song, E.; Chai, W.; Wang, G.; Zhang, Y.; Zhou, H.; Wu, F.; Chi, H.; Guo, X.; Ye, T.; Zhang, Y.; et al. 2024. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 18221–18232. Song, E.; Chai, W.; Ye, T.; Hwang, J.-N.; Li, X.; and Wang, G. 2025. Moviechat+: Question-aware sparse memory for long video question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence. Su, J.; Lu, Y.; Pan, S.; Murtadha, A.; Wen, B.; and Liu, Y. 2023. Ro- Former: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864. Tao, K.; Qin, C.; You, H.; Sui, Y.; and Wang, H. 2025. DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models. In Proceedings of the Computer Vision and Pattern Recog- nition Conference, 18992–19001. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. Attention is all you need. Advances in neural information processing systems, 30. Wang, H.; Feng, B.; Lai, Z.; Xu, M.; Li, S.; Ge, W.; Dehghan, A.; Cao, M.; and Huang, P. 2025a. StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant. arXiv preprint arXiv:2505.05467. Wang, P.; Bai, S.; Tan, S.; Wang, S.; Fan, Z.; Bai, J.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; et al. 2024a. Qwen2-vl: Enhancing vision- language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191. Wang, X.; Si, Q.; Wu, J.; Zhu, S.; Cao, L.; and Nie, L. 2025b. AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understanding. arXiv:2503.12559. Wang, Y.; Meng, X.; Wang, Y.; Liang, J.; Wei, J.; Zhang, H.; and Zhao, D. 2024b. VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Inter- action Format. arXiv:2411.17991. Wang, Y.; Xie, C.; Liu, Y.; and Zheng, Z. 2024c. Videollamb: Long-context video understanding with recurrent memory bridges. arXiv preprint arXiv:2409.01071. Wu, H.; Li, D.; Chen, B.; and Li, J. 2024a. LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Under- standing. arXiv:2407.15754. Wu, T.-H.; Biamby, G.; Quenum, J.; Gupta, R.; Gonzalez, J. E.; Dar- rell, T.; and Chan, D. M. 2024b. Visual haystacks: A vision-centric needle-in-a-haystack benchmark. arXiv preprint arXiv:2407.13766. Xie, Y.; He, B.; Wang, J.; Zheng, X.; Ye, Z.; and Wu, Z. 2026. FluxMem: Adaptive Hierarchical Memory for Streaming Video Un- derstanding. arXiv:2603.02096. Xu, M.; Gao, M.; Chen, Y.-T.; Davis, L. S.; and Crandall, D. J. 2019. Temporal Recurrent Networks for Online Action Detection. arXiv:1811.07391. Xu, M.; Xiong, Y.; Chen, H.; Li, X.; Xia, W.; Tu, Z.; and Soatto, S. 2021. Long Short-Term Transformer for Online Action Detection. arXiv:2107.03377. Yang, S.; Chen, Y.; Tian, Z.; Wang, C.; Li, J.; Yu, B.; and Jia, J. 2025a. Visionzip: Longer is better but not necessary in vision lan- guage models. In Proceedings of the Computer Vision and Pattern Recognition Conference, 19792–19802. Yang, Y.; Zhao, Z.; Shukla, S. N.; Singh, A.; Mishra, S. K.; Zhang, L.; and Ren, M. 2025b. StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understanding. arXiv:2508.15717. Yao, L.; Li, Y.; Wei, Y.; Li, L.; Ren, S.; Liu, Y.; Ouyang, K.; Wang, L.; Li, S.; Li, S.; Kong, L.; Liu, Q.; Zhang, Y.; and Sun, X. 2025a. TimeChat-Online: 80arXiv:2504.17343. Yao, L.; Li, Y.; Wei, Y.; Li, L.; Ren, S.; Liu, Y.; Ouyang, K.; Wang, L.; Li, S.; Li, S.; et al. 2025b. Timechat-online: 80% visual tokens are naturally redundant in streaming videos. In Proceedings of the 33rd ACM International Conference on Multimedia, 10807–10816. Yu, T.; Wang, Z.; Wang, C.; Huang, F.; Ma, W.; et al. 2025. MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe. arXiv:2509.18154. Zhang, H.; Wang, Y.; Tang, Y.; Liu, Y.; Feng, J.; and Jin, X. 2025a. Flash-VStream: Efficient Real-Time Understanding for Long Video Streams. arXiv preprint arXiv:2506.23825. Zhang, H.; Yang, S.; Fu, J.; Ng, S.-K.; and Qiu, X. 2026. HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding. arXiv:2601.14724. Zhang, L.; Cui, Q.; Zhao, B.; and Yang, C. 2025b. Oasis: One Image is All You Need for Multimodal Instruction Data Synthesis. arXiv:2503.08741. Zhang, Y.; Fan, C.-K.; Ma, J.; Zheng, W.; Huang, T.; Cheng, K.; Gudovskiy, D.; Okuno, T.; Nakata, Y.; Keutzer, K.; et al. 2024a. Sparsevlm: Visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04417. Zhang, Y.; Wu, J.; Li, W.; Li, B.; Ma, Z.; Liu, Z.; and Li, C. 2024b. Video Instruction Tuning With Synthetic Data. arXiv:2410.02713. Zhao, Y.; and Krähenbühl, P. 2022. Real-time Online Video Detec- tion with Temporal Smoothing Transformers. arXiv:2209.09236. Zhu, D.; Chen, J.; Shen, X.; Li, X.; and Elhoseiny, M. 2023. MiniGPT-4: Enhancing Vision-Language Understanding with Ad- vanced Large Language Models. arXiv:2304.10592. Zhu, J.; Wang, W.; Chen, Z.; Liu, Z.; Ye, S.; et al. 2025. InternVL3: Exploring Advanced Training and Test-Time Recipes for Open- Source Multimodal Models. arXiv:2504.10479. Zohar, O.; Wang, X.; Dubois, Y.; Mehta, N.; Xiao, T.; et al. 2024. Apollo: An Exploration of Video Understanding in Large Multi- modal Models. arXiv preprint arXiv:2412.10360. 9 Supplementary Material for ReMem: Streaming Video Understanding With Long Context Retention Additional Experimental Details In this section, we present additional experimental results from our streaming benchmark evaluations and conduct further analyses to verify the effectiveness of our design choices. Full Streaming Benchmark Results The proposed training-free ReMem framework is designed to preserve context and mitigate information loss in long streaming video inputs. While our primary evaluation focused on contextual reasoning, specifically the Contextual Understanding subtask in StreamingBench and the Backward-Tracing subtask in OVOBench, we also report comprehensive results across both benchmarks to demonstrate our method’s versatility in other settings. The Real-Time Visual Understanding results for StreamingBench are detailed in table 6, and the Real-Time Visual Perception results for OVOBench are provided in table 7. Model#Frames OP CR CS ATP EU TR PR SU ACP CT Avg. Streaming Video VLMs (training-based) VideoLLM-online-8B2 fps 39.1 40.1 34.5 31.1 46.0 32.4 31.5 34.2 42.5 27.9 36.0 FlashVStream-7B1 fps 25.9 43.6 24.9 23.9 27.3 13.1 18.5 25.2 23.9 48.7 23.2 TimeChat-Online-7B1 fps 80.2 82.0 79.5 83.3 76.1 78.5 78.7 64.6 69.6 58.0 75.4 Dispider1 fps 74.9 75.5 74.1 73.1 74.4 59.9 76.1 62.9 62.2 45.8 67.6 Streaming Video VLMs (training-free) ReKV-LLaVA-OV-7B0.5 fps 76.0 81.3 77.9 76.9 66.0 66.0 69.4 61.0 64.3 49.2 69.2 LiveVLM-7B0.5 fps 81.5 78.1 83.3 79.1 69.6 74.1 75.0 69.167.7 40.4 72.9 StreamKV-7B0.5 fps 73.8 77.3 85.9 77.5 73.3 63.9 69.4 61.4 63.2 35.8 68.8 Qwen2.5-VL-7B1 fps 77.0 79.778.9 81.1 74.8 76.6 74.1 64.6 66.2 54.8 73.2 + ReMem (Ours) † 1 fps 75.9 70.3 78.5 80.1 73.0 76.3 76.9 64.2 64.5 44.7 71.3 LLaVA-OV-7B1 fps 81.8 78.9 82.3 82.1 71.7 72.0 71.3 66.3 67.6 44.7 73.1 + ReMem (Ours) † 1 fps 82.7 72.7 81.4 84.373.0 71.0 71.3 64.6 72.4 38.8 73.1 Qwen3-VL-8B1 fps 85.678.2 86.186.3 79.0 90.4 81.769.0 67.3 56.578.0 + ReMem (Ours) † 1 fps 85.9 79.3 86.4 86.3 78.290.181.9 69.4 68.056.1 78.2 Table 6: Detailed performance comparison of VLMs across StreamingBench real-time evaluation. All results are accuracies in percent. The Overall average is computed over all ten tasks. For rows marked with†, each task reports the best result across the evaluated ReMem chunk sizes:4, 8, 16, 32 for Qwen2.5-VL-7B,4, 8, 16 for LLaVA-OV-7B and4, 8 for Qwen3-VL-8B. Prompting Strategies Analysis Two-stage protocol. To measure how closely a model preserves its task-specific attention behavior under different prompting conditions, we introduce a unified two-stage protocol using identical visual inputs. Stage 1 applies StreamingBench SQA evaluation utilising specific task prompt to obtain the reference attention distribution. Stage 2 then computes a second attention distribution using an alternative prompt. The three variants differ only in the Stage 2 prompt; all other factors (video content, decoding budget, and inference process) are kept fixed. Average cosine similarities between Stage 1 and Stage 2 are reported in Table 9. Hooking prompt (Zhang et al. 2025b) examines how the model behaves when no textual guidance is provided. Design. Following the Oasis “hooking” setup, the second pass removes both the natural language query and the assistant side template, and feeds only the visual tokens together with the fixed pre-query template. Concretely, for Qwen2.5-VL we drop the instruction and the post query template <|im_end|><|im_start|>assistant , so the model must autoregressively generate text conditioned solely on the video, without any explicit instruction. Result. This setting yields the lowest similarity (82.1 avg.), indicating that removing all textual structure causes the largest deviation from task-driven attention behavior. Chat-template prompt(Yang et al. 2025b) assesses whether providing only conversational structure without any ex- plicit instruction helps stabilize model behavior. Design. The second pass still omits the instruction text, but pre- serves the full chat template boundary between user and assistant. In practice, we append the assistant header tokens <|im_end|><|im_start|>assistant directly after the visual tokens, leveraging the model’s pretraining on caption 10 Model#Frames Real-Time Visual PerceptionBackward TracingOverall OCR ACR ATR STU FPD OJR Avg. EPM ASI HLD Avg. Avg. Streaming Video VLMs (training-based) VideoLLM-online-8B2 fps8.1 23.9 12.1 14.0 45.5 21.2 20.8 22.2 18.8 12.2 17.7 19.3 FlashVStream-7B1 fps25.5 32.1 29.3 33.7 29.7 28.8 29.9 36.4 33.8 5.9 25.4 27.6 TimeChat-Online-7B1 fps75.2 46.8 70.7 47.8 69.3 61.4 61.9 55.9 59.5 9.7 41.7 51.8 Dispider1 fps57.7 49.5 62.1 44.9 61.4 51.6 54.6 48.5 55.4 4.3 36.1 45.3 LLaVA-Video-7B1 fps69.1 58.7 68.8 49.4 74.3 59.8 62.3 56.2 53.5 51.5 53.7 60.1 Streaming Video VLMs (training-free) ReKV-LLaVA-OV-7B0.5 fps 52.3 54.1 69.8 43.3 67.3 57.1 57.3 57.6 56.1 18.8 44.2 50.8 OASIS-Qwen2.5VL-7B 1 fps84.7 72.0 65.8 51.9 67.5 64.2 67.748.8 58.9 30.9 46.2 60.5 Qwen2.5-VL-7B1 fps71.8 54.1 62.1 49.4 62.4 58.2 59.7 43.8 57.4 22.6 41.3 53.5 + ReMem (Ours) † 1 fps67.1 56.0 63.8 48.9 60.4 51.1 57.9 44.3 60.8 29.6 44.9 53.4 LLaVA-OV-7B1 fps72.5 59.6 73.3 51.7 71.3 64.765.5 54.2 55.4 21.5 43.7 58.2 + ReMem (Ours) † 1 fps78.5 70.676.7 55.1 75.2 76.6 72.1 56.057.7 22.9 45.5 63.3 Qwen3-VL-8B1 fps80.2 58.5 74.856.671.9 60.3 67.1 50.7 59.741.5 50.6 61.6 + ReMem (Ours) † 1 fps81.3 59.4 74.1 57.5 72.7 59.9 67.5 52.9 60.8 43.452.462.4 Table 7: Detailed performance comparison of VLMs across OVOBench Real-Time Visual Perception and Backward Tracing tasks. All results are accuracies in percent. The Real-Time and Backward averages are computed over six and three tasks, respectively, while the Overall average is computed over all nine tasks. For rows marked with †, each task reports the best result across the evaluated ReMem chunk sizes:4, 8, 16, 32 for Qwen2.5-VL-7B,4, 8, 16 for LLaVA-OV-7B and4, 8 for Qwen3-VL-8B. style data to implicitly elicit a generic video description even in the absence of a natural-language query. Result. This configu- ration produces intermediate similarity (85.7 avg.), showing that structural cues partially constrain the model compared to fully instruction-free prompting. General prompt (ours) induces explicit, query-agnostic reconstruction behavior that closely matches the model’s specific task attention patterns. Design. The second pass pairs the video with a general yet concise instruction asking the model to “describe the video in detail sequentially from start to finish”; the standard chat template is preserved. Result. This strategy achieves the highest alignment (92.7 avg.), indicating that explicit textual guidance best preserves the model’s task-specific attention dynamics. For consistency, all the prompting experiments are conducted on the SQA subset of StreamingBench with a total of 250 question samples. Similarly we set compression ratio r = 0.7, chunk size = 16 and fps = 1 for all the experiments. We also set the "max token length" parameter during decoding to be 1 to reduce the time for attention score generation. The exact query-agnostic general prompt in our strategy is "You are an advanced video question-answering AI assistant. You have been provided with a video. Your task is to describe the video in detail sequentially from start to finish." Base ModelMethod Ratio SQA Qwen2.5-VL-7B Official1.043.6 Qwen2.5-VL-7B VisionZip 0.135.9 Qwen2.5-VL-7BGQC0.1 36.6(↑0.7) Qwen2.5-VL-7B VisionZip 0.340.4 Qwen2.5-VL-7BGQC0.3 41.0(↑0.6) Qwen2.5-VL-7B VisionZip 0.743.2 Qwen2.5-VL-7BGQC0.7 43.5(↑0.3) Table 8: Comparison of visual token compression methods on StreamingBench under different retention ratios. Prompt Strategy Best Sim. Avg. Sim. Hooking89.582.1 Chat-Template92.685.4 General (ours)98.192.7 Table 9: Average cosine similarity between attention maps from Stage 1 Multiple Choice Questions and Stage 2 under different second-pass prompting strategies. 11 Compression Experiment We further analyse the hyperparameter for the compression ratio, r, on the StreamingBench SQA subtask. Using the Qwen2.5- VL-7B baseline, we compared our method GQC against VisionZip (Yang et al. 2025a). The results in table 8 show a clear trade-off: performance is nearly lossless at r = 0.7 for both methods but degrades as compression increases. With performance lowest for both at r = 0.1, we empirically selected r = 0.7 as the optimal balance between accuracy and compression and used it for all subsequent experiments. fig. 4 shows an illustrative example of the attention score distribution across visual tokens under different queries. While the absolute attention scores differ, the spatial distribution over visual tokens remains highly consistent across the two passes, indicating that our general query approach preserves the attention pattern of visual tokens and the general query based compression GQC works well with different task queries. Figure 4: Distribution of attention score over visual tokens. Top: general query. Bottom: task query. The task query is “How many game scores has the team referred to in the previous question scored so far?” from StreamingBench SQA, and the general query is “Describe the video in detail sequentially from start to finish.” Method Elaboration In algorithm 1, we provide detailed pseudocode for selecting vision tokens based on attention scores, as introduced in our General Query Compression (GQC) module. Algorithm 1: Pseudo-code for attention based vision token compression Require: V i : input vision tokens Require: Attn: attention weights from all heads and layers Require: I start : image token start index; Require: I end : image token end index; Require: r: compression ratio 1: Attn layer_avg ← Mean(Attn, dim = 0) Average across layers 2: Attn head_avg ← Mean(Attn layer_avg , dim = 0) Average across heads 3: A text→img ← Attn head_avg [I end + 1 :, I start : I end + 1] img-text attention score 4: S img ← Mean(A text→img , dim = 0) Average scores from all text queries 5: N img ← Shape(S img )[−1] Get total number of vision tokens 6: k ←⌊r× N img ⌋ Calculate number of tokens k to select 7: Idx topk ← TopK(S img , k, dim =−1) Find indices of k highest scores 8: V ′ i ← V i [Idx topk ] Select vision tokens 12 Case Studies In this section we visualise some of the benchmark questions and answers for our method and baseline method on StreamingBench (Lin et al. 2024) sub tasks. The case study samples are shown in fig. 5 to fig. 12. Task:Misleading Context Understanding Question:What exactly is on the table now? Time stamp: 00:01:52 Options: A. There are five white circles on the table, from left to right, the circles contain the club symbol, the diamond symbol, the heart symbol, the spade symbol and 'NT'. B. There are five white circles on the table, but only the first circle on the left has the clubssymbol in it, the other circles have nothing in them. C. There is now a spade, a club, a heart and a diamond on the table, with a white circle in the centre that says 'NT'. D. There are five white circles on the table and nothing inside them. Ground Truth Answer : C, Qwen2.5-VL Answer: D,Qwen2.5-VL-ReMem Answer:C Figure 5: Case study on Misleading Context Understanding. The video provides a detailed introduction to the rules and scoring of the card game Bridge, including the suit ranking: Clubs, Diamonds, Hearts, Spades, and No Trump (NT). At timestamp 00:01:52 the question asks: “What exactly is on the table now?” with four multiple choice options describing different symbol configurations. The ground truth answer is C, while the Qwen2.5-VL baseline is distracted by earlier frames and incorrectly predicts D. In contrast, our ReMem-enhanced model correctly outputs C, highlighting its improved ability to track evolving object states over time and resist misleading intermediate context. Task:Misleading Context Understanding Question:What's inside the white curtain on the tabletop? Time stamp: 00:05:46 Options: A. In the centre of the white cloth there is a black line with '1 30' and '2 60' written on the left side, and in the middleof the numbers there is a spade and a heart symbol, respectively, and below this set of numbers there is another black line and '590'. B. Inside the white curtain there is a vertical line with '590' written on the left and '540' written on the right with a red circle on the left. C. A white cloth with two black lines and '1 30' written in the lower left corner of the curtain with a spade in the middle of the numbers. D. A white cloth with two black lines crossed and '-200x4' written in the upper left corner of the curtain. Ground Truth Answer : B, Qwen2.5-VL Answer: C,Qwen2.5-VL-ReMem Answer:B, Figure 6: Case study on Misleading Context Understanding. The video explains the rules and scoring of the card game Bridge, and this example shows several snapshots of the scorecard as the match progresses, ending with the final tally sheet on the tabletop. At timestamp 00:05:46, the question asks: “What’s inside the white curtain on the tabletop?” with four multiple choice options describing different numerical layouts. The ground truth answer is B, while the Qwen2.5-VL baseline is distracted by earlier frames and incorrectly predicts C. In contrast, our ReMem-enhanced model correctly outputs B, highlighting its improved ability to track evolving object states over time and resist misleading intermediate context. 13 Task:Misleading Context Understanding Question:How many face-up playing cards are here? Time stamp: 00:07:50 Options: A. Here are four, the 10 of spades, the 5 of diamonds, the 3 of diamonds and the 4 of clubs. B. There are three here, the 3 of spades, the 3 of diamonds and the 6 of hearts. C. There are seven here, the J of clubs, the eight of spades, the three of spades, the three of hearts, the six of hearts, the three of diamonds and the ten of clubs. D. There are six here, the 5 of spades, the 8 of spades, the 2 of spades, the 6 of spades, the J of spades and the Q of hearts. Ground Truth Answer : C, Qwen2.5-VL Answer: A,Qwen2.5-VL-ReMem Answer:C, Figure 7: Case study on Misleading Context Understanding. This example shows a sequence of frames from a casino table game where face-up cards are gradually revealed and rearranged. At timestamp 00:07:50, the question asks: “How many face-up playing cards are here?” with four multiple choice options describing different cards arrangement. The ground truth answer is C, while the Qwen2.5-VL baseline is distracted by earlier frames and incorrectly predicts A. In contrast, our ReMem-enhanced model correctly outputs C, highlighting its improved ability to track evolving object states over time and resist misleading intermediate context. Task:Anomaly Context Understanding Question:What unusual event just occurred? Time stamp: 00:05:00 Options: A. The magician quickly removed the black cloth with both hands to the left, and suddenly a black chair appeared behind the black cloth. B. When the magician removed the black cloth from the chair, the boy's pants on the chair changed. C. When the magician removed the black cloth covering the chair, the little boy's T-shirt changed. D. After the magician removed the black cloth covering the chair, a little boy suddenly appeared on the chair. Ground Truth Answer : C, Qwen2.5-VL Answer: D,Qwen2.5-VL-ReMem Answer:C, Figure 8: Case study on Anomaly Context Understanding. The video shows a stage magic performance where a magician covers a chair with a black cloth while a young boy stands beside it. At timestamp 00:05:00, the question asks “What unusual event just occurred?”, with four multiple choice options describing different outcomes after the cloth is removed. The ground truth answer is C, while the Qwen2.5-VL baseline incorrectly predicts D, hallucinating that a boy suddenly appears on the chair. In contrast, our ReMem-enhanced model correctly selects C, indicating improved sensitivity to fine-grained, unexpected events and more precise reasoning in dynamic, unpredictable scenes. 14 Task:Anomaly Context Understanding Question:What unusual event just occurred? Time stamp: 00:01:08 Options: A white dog turned off the road on the right and ran into a vehicle that was acting as a camera. B. As a large truck was moving from above to below, the rear of the truck hit a black car parked on the right and made it slide for some distance. C. A truck loaded with goods was turning right when the left rear of the truck struck the car on the left, catching it and causing it to move. D. A black dog turned off the road on the right and ran into a vehicle that was acting as a camera. Ground Truth Answer : D, Qwen2.5-VL Answer: B,Qwen2.5-VL-ReMem Answer:D, Figure 9: Case study on Anomaly Context Understanding. This video presents a dashcam-style traffic video in which rare or unexpected events must be localized and identified. At timestamp 00:01:08, the question asks: “What unusual event just occurred?”, with four multi choice descriptions of possible anomalies. The ground truth answer is D, while the Qwen2.5-VL baseline incorrectly predicts B, being distracted by large moving vehicles in earlier frames. In contrast, our ReMem-enhanced model correctly identifies D, demonstrating stronger temporal grounding and improved resilience to misleading context in fast-changing real-world scenes. Task:Anomaly Context Understanding Question:What unusual event just occurred? Time stamp: 00:03:01 Options: A. The cards ripped from the magician's hand were restored. B. The magician suddenly ripped open the playing card signed by the female guest. C. The magician stretched out his left hand and lit the candle on the table. D. The cards ripped from the magician's hand turned into two pigeons. Ground Truth Answer : A, Qwen2.5-VL Answer: B,Qwen2.5-VL-ReMem Answer:A, Figure 10: Case study on Anomaly Context Understanding. The video records a stage magic performance in which a magician interacts with guests at a table. At timestamp 00:03:01, the question asks: “What unusual event just occurred?” with four multiple choice options describing possible outcomes. The ground-truth answer is A, while the Qwen2.5-VL baseline incorrectly predicts B, focusing on ripping the signed card, a salient but non-anomalous action. In contrast, our ReMem-enhanced model correctly selects A. This example demonstrates that ReMem better captures temporal dependencies and distinguishes genuinely unexpected events from normal but visually prominent actions, leading to more reliable anomaly reasoning in streaming video. 15 Task:Sequential Question Answering Question:Who is preparing to serve the ball now? Time stamp: 00:00:36 Options: A. Wang chuqin. B. Fan Z.D. C. Xu xin. D. Ma Lin. Ground Truth Answer : A, Qwen2.5-VL Answer: A,Qwen2.5-VL-ReMem Answer:A Task:Sequential Question Answering Question:How much game score has the person mentioned in the last question scored now? Time stamp: 00:03:02 Options: A. 2. B. 6. C. 7. D. 4. Ground Truth Answer : C, Qwen2.5-VL Answer: B,Qwen2.5-VL-ReMem Answer:C Figure 11: Case study on Sequential Question Answering. The video records a professional table-tennis match, where the model must answer five temporally linked questions based on the same ongoing rally. The first question at timestamp 00:00:36 asks: “Who is preparing to serve the ball now?”, and all models correctly identify Wang Chuqin (option A). The second question at timestamp 00:03:02 then queries:“How much game score has the person mentioned in the last question scored now?” with four multiple choice options describing different scores. The ground truth answer is C,while the Qwen2.5-VL baseline fails to propagate the correct entity and its evolving score,incorrectly predicting B. In contrast, our ReMem-enhanced model maintains consistent temporal grounding and accurately outputs C. This example illustrates that ReMem substantially improves long-range entity tracking and state updating in streaming video question answering. Task:Sequential Question Answering Question:Which team is currently on offense in the game?? Time stamp: 00:00:25 Options: A. LAC. B. MIN. C. BOS. D. LAL. Ground Truth Answer : A, Qwen2.5-VL Answer: A,Qwen2.5-VL-ReMem Answer:A Task:SequentialQuestion Answering Question:Howmany game scores does the team referred to in the first question have currently? Time stamp: 00:06:09 Options: A. 76. B. 92. C. 88. D. 85. Ground Truth Answer : D, Qwen2.5-VL Answer: A,Qwen2.5-VL-ReMem Answer:D Task:SequentialQuestion Answering Question:Howmany game scores has the team that the third question specifically refers to scored now? Time stamp: 00:08:04 Options: A. 93. B. 63. C. 114. D. 115. Ground Truth Answer : D, Qwen2.5-VL Answer: B,Qwen2.5-VL-ReMem Answer:D Figure 12: Case study on Sequential Question Answering (SQA). The video shows a professional basketball game, where the model is queried with five temporally dependent questions about the same ongoing play. The first question at timestamp 00:00:25 asks: “Which team is currently on offense in the game?”, and all models correctly identify LAC (option A). The fourth question at timestamp 00:06:09, asks: “How many game scores does the team referred to in the first question have currently?”. The fifth question at timestamp 00:08:04 further asks: “How many game scores has the team that the third question specifically refers to scored now?”, both under a multi-choice setting. The ground-truth answers are D (85 points) and D (115 points), respectively. The Qwen2.5-VL baseline fails to correctly update and track the scores, predicting A and B, whereas our ReMem-enhanced model consistently outputs the correct options D and D. This example highlights that ReMem more effectively leverages episodic memory to maintain entity references and their evolving states over time, enabling coherent and contextually accurate reasoning across a sequence of related questions in streaming video. 16