Paper deep dive
Scaling the Long Video Understanding of Multimodal Large Language Models via Visual Memory Mechanism
Tao Chen, Kun Zhang, Qiong Wu, Xiao Chen, Chao Chang, Xiaoshuai Sun, Yiyi Zhou, Rongrong Ji
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 4/1/2026, 1:32:28 AM
Summary
FlexMem is a novel, training-free visual memory mechanism for Multimodal Large Language Models (MLLMs) that enables efficient long video understanding. By utilizing a dual-pathway compression design on visual KV caches, FlexMem mimics human video-watching behavior through iterative processing, allowing for infinite-length video input while maintaining constant memory overhead. It supports both encoding-based memory reading and a fast indexing approach (MemIndex) to retrieve relevant memory fragments for question answering, outperforming existing SOTA methods on various benchmarks using a single 3090 GPU.
Entities (6)
Relation Signals (4)
FlexMem → utilizes → Dual-Pathway Compression
confidence 98% · FlexMem... realize the effective memory transfer and writing via a dual-pathway compression design.
FlexMem → appliedto → LLaVA-Video
confidence 95% · To validate FlexMem, we apply it to two representative video MLLMs, namely LLaVA-OneVision [18] and LLaVA-Video [64]
FlexMem → improves → MLLMs
confidence 95% · FlexMem can help MLLMs achieve video understanding of infinite lengths
FlexMem → includes → MemIndex
confidence 95% · FlexMem is also equipped with a novel and fast indexing approach... called MemIndex.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Long video understanding is a key challenge that plagues the advancement of \emph{Multimodal Large language Models} (MLLMs). In this paper, we study this problem from the perspective of visual memory mechanism, and proposed a novel and training-free approach, termed \emph{Flexible Memory} (\textbf{FlexMem}). In principle, FlexMem aims to mimic human behavior of video watching, \emph{i.e.}, continually watching video content and recalling the most relevant memory fragments to answer the question. In this way, FlexMem can help MLLMs achieve video understanding of infinite lengths, unlike previous methods that process all video information at once and have input upper-limit. Concretely, FlexMem first consider the visual KV caches as the memory sources, and realize the effective memory transfer and writing via a dual-pathway compression design. Afterwards, FlexMem also explores different memory reading strategies for the diverse video understanding tasks, including the popular streaming one. To validate FlexMem, we apply it to two popular video-MLLMs, and conduct extensive experiments on five long video and one streaming video task. The experimental results show that on \textbf{a single 3090 GPU}, our FlexMem can achieve obvious improvements than existing efficient video understanding methods and process more than \textbf{1k frames}, which also helps the base MLLMs achieve comparable or even better performance than SOTA MLLMs on some benchmarks, \emph{e.g.} , GPT-4o and Gemini-1.5 Pro.
Tags
Links
- Source: https://arxiv.org/abs/2603.29252v1
- Canonical: https://arxiv.org/abs/2603.29252v1
Trouble viewing inline? Open PDF directly →
Full Text
56,739 characters extracted from source content.
Expand or collapse full text
Scaling the Long Video Understanding of Multimodal Large Language Models via Visual Memory Mechanism Tao Chen1 Kun Zhang1 Qiong Wu1 Xiao Chen1 Chao Chang2 Xiaoshuai Sun1 Yiyi Zhou1 Rongrong Ji1 1Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. 2National University of Defense Technology, 230000, P.R. China. Corresponding author: zhouyiyi@xmu.edu.cn. Abstract Long video understanding is a key challenge that plagues the advancement of Multimodal Large language Models (MLLMs). In this paper, we study this problem from the perspective of visual memory mechanism, and proposed a novel and training-free approach, termed Flexible Memory (FlexMem). In principle, FlexMem aims to mimic human behavior of video watching, i.e., continually watching video content and recalling the most relevant memory fragments to answer the question. In this way, FlexMem can help MLLMs achieve video understanding of infinite lengths, unlike previous methods that process all video information at once and have input upper-limit. Concretely, FlexMem first consider the visual KV caches as the memory sources, and realize the effective memory transfer and writing via a dual-pathway compression design. Afterwards, FlexMem also explores different memory reading strategies for the diverse video understanding tasks, including the popular streaming one. To validate FlexMem, we apply it to two popular video-MLLMs, and conduct extensive experiments on five long video and one streaming video task. The experimental results show that on a single 3090 GPU, our FlexMem can achieve obvious improvements than existing efficient video understanding methods and process more than 1k frames, which also helps the base MLLMs achieve comparable or even better performance than SOTA MLLMs on some benchmarks, e.g. , GPT-4o and Gemini-1.5 Pro. Our code is released at: FlexMem. 1 Introduction Recent years have witnessed the remarkable progress made by Multimodal Large Language Models (MLLMs) [69, 67, 68, 43, 28, 55] towards effective vision-language understanding. Despite the great success, long video understanding is still a main obstacle for existing MLLMs mainly due to the difficulty of processing excessive long video frames [34, 14]. In addition to high computation complexity, the large number of visual tokens from long videos can easily exceed the upper limit of the sequence length of existing MLLMs [52, 13, 36], e.g., more than 200k for 1024 video frames [18], resulting in both performance degradation and expensive memory overhead [18, 7]. Figure 1: Comparison between FlexMem (ours) and existing efficient video understanding methods for MLLMs on five benchmarks. All methods are run on the same device of one 3090 GPU, and our FlexMem presents obvious performance gains. To tackle this issue, recent efforts [14, 42, 11, 53] are devoted to efficient long video understanding for MLLMs. One popular solution is to adopt retrieval augmentation generation (RAG) based strategies to select key video information for MLLMs [29, 42], drawing on the successful experience of LLMs [38, 2]. Concretely, RAG methods regard the whole video as a knowledge base, and then find out the question-related key frames (or clips) as the input of MLLMs, thereby avoiding the processing of all frames. Although effective in video tasks like needle-in-a-haystack [65], which requires evident localization from thousands of video frames, RAG methods are still inferior in mastering continual and overall understanding of videos [35, 22]. In this case, they are still sensitive to memory overhead for more keyframe inputs [17, 4]. The other viable solution is to use visual feature compression for the longer input of video frames [50, 9, 61]. For instance, Wang et al. [50] apply visual Key-Value (KV) caches compression to reduce the per-clip footprint, thereby increasing the number of input frames. However, visual compression methods [50, 49, 39] still require MLLMs to input all compressed visual features for the final answering, still yielding obvious computation bottlenecks. Overall, existing methods are still hard to strike a trade-off between efficient video understanding and optimal performance. In this paper, we study the long video understanding of MLLMs from the perspective of visual memory mechanism [11, 14, 62]. Specifically, we aim to help MLLMs to be able to watch videos continuously, form visual memories and answer questions based on relevant memory fragments, just like a human. In this way, MLLMs can answer the question without having to using all information, i.e., breaking the input limit of the final prediction, while also being capable of handling different question types, e.g., the global and general ones. More ideally, this memory mechanism should be also independent to MLLMs’ structure and training, and can be a plug-and-play component that directly applied to MLLMs without great structure tweaks. However, achieving the above target still encounters several key challenges. The first one is how to effectively encode memory fragments. While some recent works use KV caches as the viable representations [15, 16, 50], we think that the memories for video MLLMs should not be only highly compressed but also transferable and continuous, thereby handling different types of video tasks, as discussed above. Secondly, the effective reading of memory is also critical. One intuitive solution is to leverage the MLLM’s cross-modal attention during encoding to judge the relevance of memories. However, in scenarios with multiple questions or streaming QA [32, 21], the repeated encoding of video clips and answers will incur excessive computation overhead. In this case, the design of effective and efficient visual memory mechanism for MLLMs is still a intractable problem. To address these challenges, we propose a novel and training-free visual memory mechanism for video-MLLMs, termed Flexible Memory (FlexMem). Concretely, FlexMem resorts to Key-Value caches of visual tokens as the MLLM’s memory representations, similar to some existing compression-based works [31, 49]. In practice, we also introduce a novel dual-pathway compression design that can greatly reduce the memory sizes while ensuring the continuity of each memory snippet. In terms of memory reading, FlexMem is also equipped with a novel and fast indexing approach in addition to the aforementioned encoding-based one, called MemIndex. Via statistically fitting the encoding-based retrieval, MemIndex adaptively select the representative cache layers and tokens to form a much smaller memory index tensor, supporting the fast and flexible memory retrieval. With these innovative designs, the proposed FlexMem can scale the input frames of MLLMs, thereby significantly enhancing their long video understanding. To validate FlexMem, we apply it to two representative video MLLMs, namely LLaVA-OneVision [18] and LLaVA-Video [64], and conduct extensive experiments on a bunch of highly competitive benchmarks. The experimental results not only show the great improvement to video MLLMs, e.g., +32.2% on TimeScope for LLaVA-Video, but also validate its merits than existing methods for efficient video understanding. For instance, under the same setting of one 3090 GPU, our FlexMem can outperforms the SOTA methods such as AKS [42] and AdaRETAKE [50] by 3.9% and 5.2% on average for LLaVA-Video, respectively. Overall, our contributions are two-fold: • We study the long video understanding of MLLMs from the perspective of visual memory mechanism, and propose a novel approached termed FlexMem to scale up the input of video frames. • On a set of benchmarks, our FlexMem can greatly improve the capabilities of base MLLMs and outperform a set of SOTA methods using only one 3090 GPU. 2 Related Work 2.1 Video Multimodal Large Language Models The rapid advancement of Large Language Models (LLMs) has catalyzed significant breakthroughs in multimodal understanding [26, 25, 27], leading to the emergence of Video Multimodal Large Language Models (Video-MLLMs) [1, 19, 20, 30]. Early pioneering works like Flamingo [1] and VideoChat [19] laid the foundation by extending image-based multimodal models with temporal modeling modules, enabling basic video comprehension capabilities. Subsequent works such as Video-LLaVA [20] and Video-ChatGPT [30] improve temporal reasoning through unified visual representations and joint image-video training. More recent state-of-the-art models like Qwen3-VL [3] and InternVL3.5 [48] have achieved remarkable performance improvements by scaling both model parameters and training data. However, despite their impressive capabilities, these methods are fundamentally constrained by computational resources and typically process only a limited number of frames, which significantly restricts their applicability to long video understanding scenarios. Figure 2: Illustration of the proposed FlexMem method. (a) FlexMem is an iterative method, and it encodes two types of compressed memories for each video clip ViV_i, namely Context Memory CiC_i and Local Memory MiM_i, based on the metrics of aggregation score SiS_i and local saliency score S^i S_i, respectively. MiM_i is then stored in the visual memory bank MbankM_bank, while the context memory C are used in the iterative encoding step for information propagation. Besides, we can also retrieval some stored MlM_l as the long-term memory for encoding, while it is optional as well as the text instruction TqT_q. (b) The stored memories MaM_a will be recalled from the memory bank for the decoding of answers Y. (c) One intuitive and effective indexing for FlexMem is the Encoding-based one, which uses the cross-attention during memory encoding with TqT_q (a) to reflect the relevance of memories. (d) We also investigate the other fast index method, termed MemIndex, based on the compact index tensors for both question and visual memories, of which process is independent to the encoding of memories. Its selection of cache layers and tokens stems from the fitting results of the encoding-based index. 2.2 Long Video Understanding To tackle the above challenge, some efforts resort to Retrieval-Augmented Generation (RAG) strategies derived from LLMs [38, 2] to long video understanding. Video-RAG methods [42, 35, 29, 59, 37] typically employ a two-stage pipeline, i.e., first retrieving keyframes based on query similarity, then processing them for answer generation. For instance, AKS [42] uses vision-language embedding Models for similarity-based retrieval, while VideoAgent [11] employs an iterative refinement process with LLM-based planning. However, such retrieval methods face inherent limitations in maintaining temporal coherence and capturing long-range dependencies. These methods often lack important contextual information that spans multiple segments and struggle with queries requiring holistic video understanding. Recently, visual compression methods have been extensively studied [50, 49, 39], which maintain compressed features of historical context for comprehensive understanding. For instance, AdaRETAKE [50] designs adaptive allocation modules to determine compression ratios across temporal dimensions and MLLM layers. Video-XL [39] introduces special tokens to summarize the visual information within video fragments. Despite these advances, their input context length grows linearly with video duration, limiting their scalability. FlexMem combines the benefits of both paradigms, i.e., maintaining comprehensive visual memories with constant footprint through iterative processing, and reading the most relevant information for answer generation via memory recall mechanism. 3 Method 3.1 Overview In this paper, we study the long video understanding of MLLMs from the perspective of visual memory mechanism, and propose a novel and training-free approach termed Flexible Memory (FlexMem), as depicted in Fig. 2. In principle, FlexMem aims to mimic the human behaviors of video watching, i.e., continually browsing video content, forming memories and answering questions based on memory recall. Via this iterative paradigm, FlexMem can help MLLMs break the upper-limit of input length. In particular, given a long video V and a text instruction TqT_q, existing MLLMs [46, 48, 64] normally sample a subset of frames as the visual input, denoted as V′=I1,⋯,IMV =\I_1,·s,I_M\, due to the limit of input sequence length and memory overhead. The prediction Y is generated according to all input frames and the text instruction: MLLM(I1,…,IM,Tq)→Y. (I_1,…,I_M,T_q)→ Y. (1) In terms of long video understanding, this solution is greatly limited by the number of input frames, leading to suboptimal performance [54, 71]. To address this challenge, FlexMem considers the visual KV caches as the memory sources, and realizes the effective memory transfer and writing via a dual-pathway compression design. Specifically, we first divide the video into N clips V=V1,⋯,VNV=\V_1,·s,V_N\. Then, FlexMem lets MLLMs to read video clips iteratively, and its first step is defined by MLLM(V1,⟨Tq⟩)→M1,C1. (V_1, T_q )→ M_1,C_1. (2) where ⟨⋅⟩ · is an optional input. M1,C1M_1,C_1 are the compressed local memory and context memory respectively, which are processed by our Dual-Pathway Compression (DPC) design. In particular, M1M_1 is written into the visual memory bank MbankM_bank for the following memory recall, while C1C_1 is used for the historical video information propagation in the iterative steps. Thus, they are processed by differently. After the first step, FlexMem will extend the inputs of MLLMs, which can be defined by MLLM(⟨Ml⟩,Ck−ns,…,Ck−1,Vk,⟨Tq⟩)→Mk,Ck. MLLM( M_l ,C_k-n_s,...,C_k-1,V_k, T_q )→ M_k,C_k. (3) Here k denotes the current step of memory processing, and nsn_s is the number of retained context memories. In Eq. 3, we give a certain interval of previous context memories to MLLM, thereby achieving the transfer of video information and building the continuity of stored memories. Besides, we also recall some stored MlM_l from the memory bank as the long-term memory for the better understanding of long historical information. After watching the whole video, FlexMem will recall the most relevant memory pieces from MbankM_bank: Recall(Mbank,Tq)→Mi,…,Mi+na−1. (M_bank,T_q)→ M_i,.,M_i+n_a-1. (4) where MiM_i is the recalled memory, and nan_a is the number of recalled pieces. Lastly, MLLM will use these recalled memories for the final answer prediction: MLLM(Mi,…,Mi+na−1,Tq)→Y (M_i,.,M_i+n_a-1,T_q)→ Y (5) In particular, the memory encoding with TqT_q in Eq. 3 is optional. According to the uni-directional attention mechanism of MLLMs [44, 56], the encoding of TqT_q will not affect the visual memory compression, but can help to record the video-question relevance for the following memory recall. In this paper, we also explore the fast indexing of video memories, i.e., not using TqT_q during the encoding of visual memories. Besides, FlexMem is an iterative approach, i.e., Eq. 3, which can theoretically process infinite-long videos. 3.2 Dual-Pathway Compression To scale long video understanding, FlexMem is equipped with a novel dual-pathway compression design for memory compression and transmission. In particular, FlexMem also regards the encoded Key-Value caches of visual tokens as the memory source, and effectively compresses them for memory writing and reading. Compared with existing KV cache compression methods [50, 39, 49], which progressively encode clips and have input upper-limit, our FlexMem consider visual memory encoding as a iterative process that focuses on information transfer. Concretely, at the i-st step of FlexMem, we will include the recent context memory C=Ci−kk=1nsC=\C_i-k\_k=1^n_s into the encoding of current clips, and return the attention matrix AlA^l: vl=Attention([Vi,⟨Tq⟩],[^C,Vi,⟨Tq⟩]), ^l_v=Attention([Q_V_i, _T_q ],[ K_C,K_V_i, _T_q ]), (6) where ViQ_V_i and ViK_V_i are the query and key vectors of ViV_i at each layer, and TqQ_T_q and TqK_T_q are those of TqT_q. Recognizing that the role of the visual memory differs between the prefill and decoding stages, we strategically prunes unimportant KVs of ViV_i based on two attention-based metrics. For the prefill stage, the objective is to encode the current clip with a rich understanding of its historical context, i.e., context memory C. To approach this target, we measure the importance of a token whether it effectively aggregates information from past context and propagates its own information to subsequent tokens within its clip. We define the context aggregation score sjls^l_j for the j-th token in clip ViV_i as the metric for obtaining its context features ilc_i^l: il=jl,jl|sjl∈argmaxj∈Viαc|Vi|sjl, _i^l=\k_j^l,v_j^l|s^l_j∈ _c|V_i| j∈ V_i \>s_j^l\, (7) wheresjl=∑k∈Cajkl+∑h∈Viahjl. \>\>s^l_j= _k∈ Ca_jk^l+ _h∈ V_ia_hj^l. where ajkla_jk^l is attention weight of AvlA_v^l from j-th token in current clip to k-th token in the historical context. kjl,vjlk_j^l,v_j^l are the key and value vectors of the j-th token at the l-th layer. αc _c denotes the compression ratio for context features, and |Vi||V_i| is the number of tokens in clip ViV_i. The context memory CiC_i of clip ViV_i is consisted of its KVs from all cache layers, i.e., Ci=i1,…,iLC_i=\c_i^1,…,c_i^L\. For the decoding stage, the MLLMs aim to answer the text instruction based on the most salient visual evidence. Therefore, the priority at this time is to eliminate redundancy within each clip to retain its most distinctive information. We thus define a local saliency score s^jl s^l_j to measure the overall influence of a token within its own clip, and use it to obtain compressed visual features ilm_i^l: il=jl,jl|s^jl∈argmaxj∈Viαs|Vi|s^jl, _i^l=\k_j^l,v_j^l| s^l_j∈ _s|V_i| j∈ V_i \> s_j^l\, (8) wheres^jl=∑k∈Viakjl. \>\> s^l_j= _k∈ V_ia_kj^l. where αs _s is the compression ratio for the stored memory MiM_i, and it includes the compressed caches ilm_i^l of all layers of the clip ViV_i, i.e., Mi=i1,…,iLM_i=\m_i^1,…,m_i^L\. Overall, FlexMem can iteratively process extra long videos with limited memory overhead, and obtain the stored memory bank MbankM_bank for prediction, of which features are rich in visual information and continual in video semantics. 3.3 Memory Reading 3.3.1 Question Encoding based Memory Reading In terms of memory reading, one effective solution is to directly uses the cross-modal attention encoded during memory compression, i.e., Eq. 6. Based on the superior vision-langauge (VL) alignment capability of MLLMs, we can directly use the cross-modal attentions between video clips and question as the metric for memory reading. Specifically, we compute this relevance score gig_i by summing the attention weights from the instruction tokens to the tokens of clip ViV_i at the prefill stage: gi=∑l=3L∑j∈Tq∑k∈Viajkl, g_i= _l=3^L _j∈ T_q _k∈ V_ia_jk^l, (9) Recall(Mbank,Tq)=Mi|gi∈argmaxi∈Mbanknagi. (M_bank,T_q)=\M_i|g_i∈ n_a i∈ M_bank \>g_i\. Table 1: A comparison of FlexMem with SOTA methods based on two recent MLLMs across five long VideoQA benchmarks. Sampled Frames denote the number of frames sampled from the video used for compression or selection, and Input Tokens denote the number of tokens used for question answering. The best and second-best results are shown in bold and underlined respectively. ∗Tested on one A800. Method Sampled Frames Input Tokens TimeScope LVBench MLVU Video-MME LongVideoBench Test Val M-avg Short Medium Long All Short Medium Long All !20 LLaVA-Video 7B 64frm 13k 65.0 42.6 71.2 76.1 61.0 52.4 63.2 71.5 60.7 52.1 60.0 AKS [42] 1fps 13k 85.4 47.4 72.0 77.2 64.8 53.9 65.3 72.3 62.1 57.4 62.7 Panels [10] 1fps 13k 79.2 - - - 62.2 54.0 64.4 - - - - DToMA [61] - 12k - - 71.7 - - - 65.0 - - - 59.6 Video-RAG [29] - 15k - - 72.4 - - - - - - - 58.7 AdaRETAKE [50] 1024frm 40k 86.2 49.6 71.7 75.8 62 52.9 63.6 69.7 59.2 52.8 59.4 FlexMem∗FlexMem^* 512/1024frm 13k 85.9 51.0 72.4 76.3 63.3 54.4 64.7 71.5 65.5 57.3 63.6 !20 LLaVA-OV 7B 32frm 7k 56.3 38.4 63.4 70.6 54.8 48.2 57.8 69.5 53.4 49.8 56.2 AKS [42] 1fps 7k 43.5 68.3 58.4 65.9 58.9 54.3 58.9 Panels [10] 1fps 7k 69.5 - - - 56.2 50.2 58.9 - - - - BOLT [23] 1fps 7k - - 65.8 69.2 56.8 47.3 57.8 - - - 57.0 AdaRETAKE [50] 1024frm 20k 75.8 42.1 64.4 72.1 53.6 51.4 59.0 68.5 51.0 47.2 54.2 FlexMem∗FlexMem^* 512/1024frm 7k 80.5 46.2 68.9 70.0 57.3 49.8 59.0 67.9 58.0 55.0 59.4 Table 2: Performance comparison of FlexMem against representative video RAG method (AKS) and visual compression methods based on LLaVA-Video across five long VideoQA benchmarks. All methods runs on a single 3090 with the fully use of memory overhead. Method Sampled Frames Input Tokens TimeScope LVBench MLVU Video-MME LongVideoBench Test Val M-avg Short Medium Long All Short Medium Long All LLaVA-Video 7B 32frm 7k 58.3 41.4 68.5 74.8 58.4 52.0 61.7 70.6 59.0 50.5 58.6 AKS 1fps 7k 84.6 46.6 70.8 74.7 61.9 51.7 62.8 68.1 60.0 54.1 59.7 AdaRETAKE 384frm 40k 78.2 46.8 71.7 74.7 61.8 54.3 63.6 69.0 60.4 53.5 59.8 FlexMem 512/1024frm 13k 85.6 50.2 72.3 76.2 62.7 55.0 64.6 71.5 63.6 57.2 63.0 Since the attention scores received by visual tokens are generally uniform in shallow layers [58, 51], we only leverage the attention weights from deeper layers to calculate relevance scores in practice, e.g., after the 22-th layer. 3.3.2 Fast Memory Indexing Although the encoding-based reading solution can accurately capture the video-question similarity based on MLLMs, its practical use is still limited due to the repeated MLLM inference for new questions. In this case, we also explore the fast memory index method, termed MemIndex. In terms of fast and flexible memory retrieval, we assume that MemIndex should has the following properties. First, MemIndex should be independent to the encoding of visual memory, thus they can efficiently handle multiple questions or streaming cases [32, 21]. Second, the index features of MemIndex should be compact enough, either for the visual or the question ones, thereby further reducing the cost of cross-modal matching. Achieving the above target is still intractable. For instance, the offline memory caches and the question ones still have a certain semantic gap [24], although they are encoded by the same MLLMs. Besides, the computation of retrieval is still expensive, even using the compressed cache tokens, i.e., 21k cache tokens of 25 layers. To this end, we first consider the encoding-based reading as the upper-bound of MemIndex, and then the objective of MemIndex is defined by argminσ∑i=1D‖σ(Ri)−gi‖2,whereσ(Ri)=∑l=3Lαlril. _σ _i=1^D \|σ(R_i)-g_i \|_2,\>where\>σ(R_i)= _l=3^Lα^lr_i^l. (10) Here, D is the number of training data used for optimization, and rilr_i^l is the relevance score of clip ViV_i in the l-th layer obtained from MemIndex. We aim to find a linear regression function σ(⋅)σ(·) that minimizes the L2 distance to the “target” score gig_i in Eq. 9. Specifically, given the input question TqT_q, we first encode its features via the MLLM, denoted as TqQ_T_q. Then, the basic VL matching can be defined by cl=Attention(Tq,^Vi), ^l_c=Attention(Q_T_q, K_V_i), (11) ri=∑l=3L∑j∈Tq∑k∈Virjkl. r_i= _l=3^L _j∈ T_q _k∈ V_ir_jk^l. where ^Vi K_V_i is the compressed key vectors of clip ViV_i in the stored memory MiM_i. Although feasible, this basic solution still involves excessive visual and text tokens of all layers. In this case, we first conduct the selection of visual cache layers according to the fitted regression function σ(⋅)σ(·) in Eq. 10: ℋ=l|αl∈top-K(αll=3L),H=\l|α^l -K(\α^l\_l=3^L)\, (12) where K is the number of selected representative cache layers. We identify these cache layers with highest learned weights αlα^l, which naturally indicate each layer’s importance for relevance computation. Besides, we also revise FlexMem during the memory encoding, using a higher-ratio of compression to obtain more compact local memories as the visual index tensor, e.g., the size can be changed from |Ii|×MN×d|I_i|× MN× d to k×dk× d. In terms of the question tokens, we empirically select the last token as the index feature [40, 57]. In this case, the index of FlexMem can be defined by =Tq[−1],Vi∗=jl|s^jl∈argmaxj∈Vis^jl, =Q_T_q[-1],\>K^*_V_i=\k_j^l| s_j^l∈ k j∈ V_i \> s_j^l\, (13) ^cl A^l_c =Attention(,Vi∗), =Attention(q,K^*_V_i), r^i r_i =∑l∈ℋ∑j∈Vi∗r^jl. = _l _j∈ V_i^* r_j^l. Here k is the number of key vectors selected as the representative visual indexes. Table 3: Comparison between SOTA Video-MLLMs and LLaVA-Video with FlexMem on five long VideoQA benchmarks. Method LLM TimeScope LVBench MLVU Video-MME LongVideoBench Test Val M-avg Short Medium Long All Medium Long All !20 GPT-5 - - - 77.3 - - - 81.8 - - 72.6 !20 GPT-4o - - 27.0 64.6 80.0 70.3 65.3 71.9 69.1 60.9 66.7 !20 Gemini-1.5-Pro - - 33.1 - 81.7 74.3 67.4 75.0 65.3 58.6 64.0 Video-XL [39] 7B - - 64.9 62.0 53.2 49.2 55.5 49 45.2 50.5 mPLUG-Owl3 [60] 7B - 43.5 63.7 70.0 57.7 50.1 59.3 - - 52.1 Qwen2.5-VL [3] 7B 81.0 45.3 70.2 - - - 65.1 - - 56.0 TimeMarker [6] 8B - 41.3 63.9 71.0 54.4 46.4 57.3 - - 56.3 LongVU [36] 7B - - 65.4 - - 59.5 60.6 - - - TSPO [41] 7B - 45.3 76.3 - - 54.7 65.5 - - 63.9 LongVA [63] 7B 55.9 - 56.3 61.1 50.4 46.2 52.6 - - - ByteVideoLLM [45] 14B - - 70.1 74.4 62.9 56.4 64.6 - - - LLaVA-Video 7B 65.0 42.1 71.2 76.1 61.0 52.4 63.2 60.7 52.1 60.0 + FlexMem 7B 85.9 51.0 72.4 76.3 63.3 54.4 64.7 65.5 57.3 63.6 Table 4: Comparison of our method, LLaVA-Video integrated with FlexMem and MemIndex, with SOTA online and offline models on backward tracing task of OVOBench. EPM, ASI and HLD denote EPisodic Memory, Action Sequence Identification and HaLlucination Detection, respectively. Methods LLM # Frames Backward Tracing EPM ASI HLD Average Offline Models Gemini-1.5-Pro - - 58.6 76.4 52.6 62.5 InternVL-V2 [8] 8B 64 48.2 57.4 24.7 43.4 LongVU [36] 7B 1fps 40.7 59.5 4.8 35.0 Online Models Flash-VStream [62] 7B 1fps 39.1 37.2 5.9 27.4 VideoLLM-online [5] 8B 2fps 22.2 18.8 12.2 17.7 Dispider [33] 7B 1fps 48.5 55.4 4.3 36.1 LLaVA-Video 7B 64 55.2 60.8 42.5 52.8 +FlexMem w. MemIndex 7B 1fps 57.6 54.1 49.5 54.4 4 Experiment 4.1 Benchmarks and Metrics To validate FlexMem, we conduct extensive experiments on five benchmarks for long video understanding, including MLVU [66], LongVideoBench [54], LVBench [47], Video-MME [12] and TimeScope [70]. MLVU includes videos ranging from 3 minutes to 2 hours that require comprehensive temporal understanding. Video-MME covers videos of diverse genres and durations, including short, medium, and long-form content. LongVideoBench is designed for tasks requiring precise retrieval and reasoning over detailed multimodal information within extended temporal contexts, containing videos up to an hour in length. LVBench challenges MLLMs to demonstrate long-term memory retention and extended comprehension capabilities, with an average video duration of approximately 68.4 minutes. TimeScope probes the limits of long video capabilities with videos ranging from 1 minute to 8 hours. 4.2 Implementation Details FlexMem is designed as a training-free approach that can be seamlessly integrated with existing MLLMs without requiring additional fine-tuning. We validate FlexMem using two recent MLLMs: LLaVA-Video [64] and LLaVA-OneVision [18]. we evaluate the effectiveness of FlexMem on long VideoQA tasks through encoding-based reading, and equip FlexMem with MemIndex in streaming QA tasks. We uniformly sample 512512 frames on TimeScope, LVBench, and MLVU, while sampling 10241024 frames on Video-MME and LongVideoBench. The input token counts for final decoding are 13k13k and 7k7k for LLaVA-Video and LLaVA-OneVision respectively, maintaining consistency with their corresponding baselines using sparse uniform sampling strategies. For our MemIndex implementation, we select K=3K=3 visual cache layers and k=5k=5 visual index features to enable efficient memory indexing while preserving representative information. 4.3 Quantitative Analysis Comparison with existing methods. Tab. 1 presents a comprehensive comparison of FlexMem against representative VideoRAG and visual compression methods across two recent MLLMs, i.e., LLaVA-Video [64] and LLaVA-OneVision [18]. From Tab. 1, we can first observe that existing methods typically require dense frame sampling and numerous token inputs for final decoding. VideoRAG methods like AKS excel at visual evidence localization on LongVideoBench, and visual compression methods like AdaRETAKE demonstrate strong holistic video understanding on Video-MME. In contrast, FlexMem consistently enhances the performance of both base models, achieving SOTA results against other methods built upon the same MLLMs across most benchmarks. This demonstrates FlexMem’s effectiveness in comprehensive memory construction through iterative processing and precise backward tracing via memory recall. For instance, FlexMem enables LLaVA-Video to surpass its baseline by 32.2% on TimeScope and 19.7% on LVBench. These results conclusively validate the effects of FlexMem in advancing long video comprehension capabilities of MLLMs. Comparison with limited memory overhead. We evaluate the scalability and performance gains of FlexMem compared to two representative methods on a single 3090 GPU, i.e., AdaRETAKE [50], which exemplifies visual compression approaches, and AKS [42], representing VideoRAG methods. As shown in Tab. 2, we first observe that AdaRETAKE and AKS experience considerable degradation compared to their unrestricted performance in Table 1. For instance, when GPU memory budget is limited to 24GB, the input capacity of AdaRETAKE is reduced from 10241024 to 384384 frames, and its performance drops by an average of 3.3% across all benchmarks. In contrast, FlexMem consistently maintains superior performance under resource constraints, and retains 99.5% of its full performance. Overall, these results demonstrating FlexMem’s ability to flexibly manage visual memories while preserving essential information. Comparison with SOTA Video-MLLMs. We further compare FlexMem with existing SOTA Video-MLLMs on five benchmarks in Tab. 3. As shown in Tab. 3, when employing the uniform sampling strategy, short Video-MLLMs such as Qwen2.5-VL exhibit superior performance on Video-MME requiring global understanding capabilities. However, this straightforward solution significantly underperforms compared to visual compression methods like TSPO on LongVideoBench, which requires fine-grained detail reasoning over extended video durations. We can also see that FlexMem consistently achieves competitive or superior performance compared to other methods with comparable model sizes. Notably, FlexMem improves LLaVA-Video to the level of Gemini-1.5-Pro, while significantly surpassing it by 54.1% on LVBench. Overall, these results well confirm the effectiveness of our FlexMem in improving long video understanding of MLLMs. Table 5: Ablation studies on different designs of FlexMem under the encoding-based reading setting across two benchmarks. Methods marked with ‡ indicate our chosen settings. LongVideoBench LVBench Choices Short Medium Long All Val Compression Strategy Context Compression Only 70.1 64.3 56.2 62.5 49.9 Local Compression Only 70.9 64.6 55.9 62.6 49.7 Dual-Pathway‡ 71.5 65.5 57.3 63.6 51.0 Context during Prefill Context Memory Only 71.2 65.0 53.5 61.9 50.5 Local Memory Only 71.2 63.3 54.6 61.8 50.0 Combination of both‡ 71.5 65.5 57.3 63.6 51.0 Context during Decoding All MbankM_bank 71.5 58.7 53.2 59.8 49.3 Memory Reading‡ 71.5 65.5 57.3 63.6 51.0 Number of Frames in Each Clip 8‡ 71.5 65.5 57.3 63.6 51.0 16 71.5 64.8 57.1 63.4 50.1 32 70.1 62.6 55.7 61.7 49.3 Table 6: Ablation studies on index token designs of FlexMem with MemIndex on two benchmarks. Single and Multi denote the Single-Detail and Multi-Detail tasks on MLVU, respectively. AttEnc means token selection based on local saliency score. Layers Text Vision MLVU LVBench Single Multi Holistic M-avg Val Encoding-based Index 77.1 54.8 77.3 72.4 51.0 All All All 76.9 54.0 77.1 72.0 46.3 3 All All 77.2 53.3 77.5 72.2 46.6 3 Last-Token All 77.4 53.3 77.3 72.3 46.8 3 Last-Token AttEnc 77.1 53.1 77.5 72.1 45.7 Figure 3: Qualitative evaluation of FlexMem. Input Video denote the sampled frames, and Key Fragments are the selected clips for answer generation via memory recall mechanism. These results demonstrate FlexMem’s capacity in comprehensive and fine-grained visual understanding. Results of FlexMem + MemIndex on streaming QA task. Tab. 4 compares the performance of FlexMem integrated with MemIndex against existing SOTA online and offline models in streaming QA tasks. As shown in Tab. 4, we can see that while offline models such as LongVU exhibit superior holistic comprehension capabilities compared to online methods like Dispider on ASI, their performance degraded on EPM that requires historical memory localization. After equipped with FlexMem and MemIndex, LLaVA-Video exceeds its common version by 3% on average, demonstrating the capacity of our method to effective memory recall and flexible context management. Overall, these results show the merits of MemIndex in historical information tracing. Ablation Study. Here, we first ablate the key designs choices of FlexMem in Tab. 5. In the first block of Tab. 5, we examine the effects of our dual-pathway compression strategy. Context Compression Only and Local Compression Only denote memory compression using only sjls_j^l in Eq. 7 and s^jl s_j^l in Eq. 8, respectively. The results show that the context features can transfer historical information for long video understanding, while local features effectively compress memories on short videos. Notably, the performance gains of our Dual-Pathway become more pronounced with longer video durations, validating its ability to effectively exploit the distinct roles of MLLMs during prefill and decoding phases, i.e., encoding clips with contextual memories and generating predictions with stored memories. In the second block of Tab. 5, we validate the effectiveness of context memory and local memory during the prefill stage. We observe that while employing either context memory or local memory alone during clip encoding yields reasonable performance, their combination results in significantly enhanced performance. This finding indicates that the two memory types are complementary, i.e., context memory maintains temporal continuity while local memory preserves long-range dependencies. The third block examines the benefits of our memory reading strategy compared to indiscriminate loading of all memory. The results demonstrate that our memory recall can effectively identify and prioritize a small subset of key clips from extended videos. In the last block of Tab. 5, we analyze performance across different block sizes. The results indicate that MLLMs consistently require detailed visual information through smaller block sizes, regardless of video duration. Overall, these results further confirm the effectiveness of our proposed designs choices for FlexMem. Next, we further ablate the effectiveness of our fast memory indexing discussed in Sec. 3.3.2, as shown in Tab. 6. The most simple solution is computing relevance scores across all cache layers for all visual KVs, which inevitably introduces substantial computational overhead and information redundancy. In contrast, our MemIndex achieves comparable or even superior performance on MLVU compared to encoding-based index while significantly reducing computational complexity. Overall, the results demonstrate that our MemIndex substantially reduces computational costs with minimal performance degradation. 4.4 Qualitative Analysis In Fig. 3, we visualize the comprehensive long video understanding and precise memory recall capabilities of FlexMem. As observed, FlexMem can significantly improve the baseline MLLM for long video understanding through precise visual evident localization. While sparse uniform sampling strategies typically lead to poor performance in long video comprehension, FlexMem empowers MLLMs to iteratively process entire videos and generate accurate answers via precise memory recall. 5 Conclusion In this paper, we presented FlexMem, a novel training-free approach that enables MLLMs to understand videos of infinite lengths via a flexible visual memory mechanism. FlexMem iteratively processes video content and recalls key memory fragments for question answering, breaking the input length limitations of MLLMs. Notably, FlexMem achieves substantial performance gains over two representative methods on a single 3090, and enables MLLMs to achieve comparable or superior performance to SOTA models like GPT-4o on several benchmarks. 6 Acknowledgments This work is supported by the National Key Research and Development Program of China (No. 2025YFE0113500), the National Science Fund for Distinguished Young Scholars (No. 62525605), the National Natural Science Foundation of China (No. U25B2066, No. U22B2051, No.62572407) , Fujian Province Special Science and Technology Program (No. 2025H0041). References [1] J. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. L. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals, A. Zisserman, and K. Simonyan (2022) Flamingo: a visual language model for few-shot learning. In NeurIPS, Cited by: §2.1. [2] A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi (2024) Self-rag: learning to retrieve, generate, and critique through self-reflection. In ICLR, Cited by: §1, §2.2. [3] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin (2025) Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923. Cited by: §2.1, Table 3. [4] Z. Cao, Y. He, A. Liu, J. Xie, F. Chen, and Z. Wang (2025-10) TV-rag: a temporal-aware and semantic entropy-weighted framework for long video retrieval and understanding. In Proceedings of the 33rd ACM International Conference on Multimedia, M ’25, p. 9071–9079. External Links: Link, Document Cited by: §1. [5] J. Chen, Z. Lv, S. Wu, K. Q. Lin, C. Song, D. Gao, J. Liu, Z. Gao, D. Mao, and M. Z. Shou (2024) VideoLLM-online: online video large language model for streaming video. In CVPR, p. 18407–18418. Cited by: Table 4. [6] S. Chen, X. Lan, Y. Yuan, Z. Jie, and L. Ma (2024) TimeMarker: A versatile video-llm for long and short video understanding with superior temporal localization ability. arXiv Preprint. Note: https://arxiv.org/abs/2411.18211 Cited by: Table 3. [7] Y. Chen, F. Xue, D. Li, Q. Hu, L. Zhu, X. Li, Y. Fang, H. Tang, S. Yang, Z. Liu, Y. He, H. Yin, P. Molchanov, J. Kautz, L. Fan, Y. Zhu, Y. Lu, and S. Han (2025) LongVILA: scaling long-context visual language models for long videos. In ICLR, Cited by: §1. [8] Z. Chen, W. Wang, Y. Cao, Y. Liu, Z. Gao, E. Cui, J. Zhu, S. Ye, H. Tian, Z. Liu, L. Gu, X. Wang, Q. Li, Y. Ren, Z. Chen, J. Luo, J. Wang, T. Jiang, B. Wang, C. He, B. Shi, X. Zhang, H. Lv, Y. Wang, W. Shao, P. Chu, Z. Tu, T. He, Z. Wu, H. Deng, J. Ge, K. Chen, M. Dou, L. Lu, X. Zhu, T. Lu, D. Lin, Y. Qiao, J. Dai, and W. Wang (2024) Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv Preprint. Note: https://arxiv.org/abs/2412.05271 Cited by: Table 4. [9] S. Di, Z. Yu, G. Zhang, H. Li, T. Zhong, H. Cheng, B. Li, W. He, F. Shu, and H. Jiang (2025) Streaming video question-answering with in-context video kv-cache retrieval. In ICLR, Cited by: §1. [10] L. Doorenbos, F. Spurio, and J. Gall (2025) Video panels for long video understanding. arXiv Preprint. Note: https://arxiv.org/abs/2509.23724 Cited by: Table 1, Table 1. [11] Y. Fan, X. Ma, R. Wu, Y. Du, J. Li, Z. Gao, and Q. Li (2024) [Inline-graphic not available: see fulltext]videoagent: A memory-augmented multimodal agent for video understanding. In ECCV, Lecture Notes in Computer Science, Vol. 15080, p. 75–92. Cited by: §1, §1, §2.2. [12] C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y. Shen, M. Zhang, P. Chen, Y. Li, S. Lin, S. Zhao, K. Li, T. Xu, X. Zheng, E. Chen, C. Shan, R. He, and X. Sun (2025) Video-mme: the first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In CVPR, p. 24108–24118. Cited by: §4.1. [13] T. Gan, X. Wang, Y. Sun, J. Wu, Q. Guo, and L. Nie (2023) Temporal sentence grounding in streaming videos. In ACM Multimedia, p. 4637–4646. Cited by: §1. [14] B. He, H. Li, Y. K. Jang, M. Jia, X. Cao, A. Shah, A. Shrivastava, and S. Lim (2024) MA-LMM: memory-augmented large multimodal model for long-term video understanding. In CVPR, p. 13504–13514. Cited by: §1, §1, §1. [15] J. Hu, S. Wang, Y. He, P. Gong, J. Yi, J. Zhang, Y. Bai, R. Chen, G. Zhang, C. Li, and K. Yuan (2025) Efficient long-context LLM inference via KV cache clustering. arXiv Preprint. Note: https://arxiv.org/abs/2506.11418 Cited by: §1. [16] D. Jo, J. Song, Y. Kim, and J. Kim (2025) FastKV: KV cache compression for fast long-context processing with token-selective propagation. arXiv Preprint. Note: https://arxiv.org/abs/2502.01068 Cited by: §1. [17] M. Kandhare and T. Gisselbrecht (2024) An empirical comparison of video frame sampling methods for multi-modal RAG retrieval. arXiv Preprint. Note: https://arxiv.org/abs/2408.03340 Cited by: §1. [18] B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, P. Zhang, Y. Li, Z. Liu, and C. Li (2025) LLaVA-onevision: easy visual task transfer. Trans. Mach. Learn. Res. 2025. Cited by: §1, §1, §4.2, §4.3. [19] K. Li, Y. He, Y. Wang, Y. Li, W. Wang, P. Luo, Y. Wang, L. Wang, and Y. Qiao (2023) VideoChat: chat-centric video understanding. arXiv Preprint. Note: https://arxiv.org/abs/2305.06355 Cited by: §2.1. [20] B. Lin, Y. Ye, B. Zhu, J. Cui, M. Ning, P. Jin, and L. Yuan (2024) Video-llava: learning united visual representation by alignment before projection. In EMNLP, p. 5971–5984. Cited by: §2.1. [21] J. Lin, Z. Fang, C. Chen, Z. Wan, F. Luo, P. Li, Y. Liu, and M. Sun (2024) StreamingBench: assessing the gap for mllms to achieve streaming video understanding. arXiv Preprint. Note: https://arxiv.org/abs/2411.03628 Cited by: §1, §3.3.2. [22] H. Liu, S. Jiang, F. Duan, Y. Lyu, X. Wang, H. Ge, and C. Liang (2025) CadenceRAG: context-aware and dependency-enhanced retrieval augmented generation for holistic video understanding. In CVPR Workshops, p. 3679–3688. Cited by: §1. [23] S. Liu, C. Zhao, T. Xu, and B. Ghanem (2025) BOLT: boost large vision-language model without training for long-form video understanding. In CVPR, p. 3318–3327. Cited by: Table 1. [24] X. Liu, B. Atalar, X. Dai, J. Zuo, S. Wang, J. C. S. Lui, W. Chen, and C. Joe-Wong (2025) Semantic caching for low-cost LLM serving: from offline learning to online adaptation. arXiv Preprint. Note: https://arxiv.org/abs/2508.07675 Cited by: §3.3.2. [25] G. Luo, Y. Zhou, M. Huang, T. Ren, X. Sun, and R. Ji (2024) Moil: momentum imitation learning for efficient vision-language adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence 47 (7), p. 5192–5204. Cited by: §2.1. [26] G. Luo, Y. Zhou, X. Sun, Y. Wang, L. Cao, Y. Wu, F. Huang, and R. Ji (2022) Towards lightweight transformer via group-wise transformation for vision-and-language tasks. IEEE Transactions on Image Processing 31, p. 3386–3398. Cited by: §2.1. [27] G. Luo, Y. Zhou, X. Sun, Y. Wu, Y. Gao, and R. Ji (2024) Towards language-guided visual recognition via dynamic convolutions. International Journal of Computer Vision 132 (1), p. 1–19. Cited by: §2.1. [28] G. Luo, Y. Zhou, Y. Zhang, X. Zheng, X. Sun, and R. Ji (2025) Feast your eyes: mixture-of-resolution adaptation for multimodal large language models. In ICLR, Cited by: §1. [29] Y. Luo, X. Zheng, X. Yang, G. Li, H. Lin, J. Huang, J. Ji, F. Chao, J. Luo, and R. Ji (2024) Video-rag: visually-aligned retrieval-augmented long video comprehension. arXiv Preprint. Note: https://arxiv.org/abs/2411.13093 Cited by: §1, §2.2, Table 1. [30] M. Maaz, H. A. Rasheed, S. Khan, and F. Khan (2024) Video-chatgpt: towards detailed video understanding via large vision and language models. In ACL, p. 12585–12602. Cited by: §2.1. [31] Z. Ning, G. Liu, Q. Jin, W. Ding, M. Guo, and J. Zhao (2025) LiveVLM: efficient online video understanding via streaming-oriented KV cache and retrieval. arXiv Preprint. Note: https://arxiv.org/abs/2505.15269 Cited by: §1. [32] J. Niu, Y. Li, Z. Miao, C. Ge, Y. Zhou, Q. He, X. Dong, H. Duan, S. Ding, R. Qian, P. Zhang, Y. Zang, Y. Cao, C. He, and J. Wang (2025) OVO-bench: how far is your video-llms from real-world online video understanding?. In CVPR, p. 18902–18913. Cited by: §1, §3.3.2. [33] R. Qian, S. Ding, X. Dong, P. Zhang, Y. Zang, Y. Cao, D. Lin, and J. Wang (2025) Dispider: enabling video llms with active real-time interaction via disentangled perception, decision, and reaction. In CVPR, p. 24045–24055. Cited by: Table 4. [34] S. Ren, L. Yao, S. Li, X. Sun, and L. Hou (2024) TimeChat: A time-sensitive multimodal large language model for long video understanding. In CVPR, p. 14313–14323. Cited by: §1. [35] X. Ren, L. Xu, L. Xia, S. Wang, D. Yin, and C. Huang (2025) VideoRAG: retrieval-augmented generation with extreme long-context videos. arXiv Preprint. Note: https://arxiv.org/abs/2502.01549 Cited by: §1, §2.2. [36] X. Shen, Y. Xiong, C. Zhao, L. Wu, J. Chen, C. Zhu, Z. Liu, F. Xiao, B. Varadarajan, F. Bordes, Z. Liu, H. Xu, H. J. Kim, B. Soran, R. Krishnamoorthi, M. Elhoseiny, and V. Chandra (2024) LongVU: spatiotemporal adaptive compression for long video-language understanding. arXiv Preprint. Note: https://arxiv.org/abs/2410.17434 Cited by: §1, Table 3, Table 4. [37] X. Shen, W. Zhang, J. Chen, and M. Elhoseiny (2025) Vgent: graph-based retrieval-reasoning-augmented generation for long video understanding. arXiv Preprint. Note: https://arxiv.org/abs/2510.14032 Cited by: §2.2. [38] W. Shi, S. Min, M. Yasunaga, M. Seo, R. James, M. Lewis, L. Zettlemoyer, and W. Yih (2024) REPLUG: retrieval-augmented black-box language models. In NAACL-HLT, p. 8371–8384. Cited by: §1, §2.2. [39] Y. Shu, Z. Liu, P. Zhang, M. Qin, J. Zhou, Z. Liang, T. Huang, and B. Zhao (2025) Video-xl: extra-long vision language model for hour-scale video understanding. In CVPR, p. 26160–26169. Cited by: §1, §2.2, §3.2, Table 3. [40] P. Suganthan, F. Moiseev, L. Yan, J. Wu, J. Ni, J. Han, I. Zitouni, E. Alfonseca, X. Wang, and Z. Dong (2025) Adapting decoder-based language models for diverse encoder downstream tasks. arXiv Preprint. Note: https://arxiv.org/abs/2503.02656 Cited by: §3.3.2. [41] C. Tang, Z. Han, H. Sun, S. Zhou, X. Zhang, X. Wei, Y. Yuan, H. Zhang, J. Xu, and H. Sun (2025) TSPO: temporal sampling policy optimization for long-form video language understanding. arXiv Preprint. Note: https://arxiv.org/abs/2508.04369 Cited by: Table 3. [42] X. Tang, J. Qiu, L. Xie, Y. Tian, J. Jiao, and Q. Ye (2025) Adaptive keyframe sampling for long video understanding. In CVPR, p. 29118–29128. Cited by: §1, §1, §2.2, Table 1, Table 1, §4.3. [43] B. Tong, B. Lai, Y. Zhou, G. Luo, Y. Shen, K. Li, X. Sun, and R. Ji (2025) FlashSloth : lightning multimodal large language models via embedded visual compression. In CVPR, p. 14570–14581. Cited by: §1. [44] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin (2017) Attention is all you need. In NIPS, p. 5998–6008. Cited by: §3.1. [45] H. Wang, Y. Nie, Y. Ye, G. Deng, Y. Wang, S. Li, H. Yu, J. Lu, and C. Huang (2024) Dynamic-vlm: simple dynamic visual token compression for videollm. arXiv Preprint. Note: https://arxiv.org/abs/2412.09530 Cited by: Table 3. [46] P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Lin (2024) Qwen2-vl: enhancing vision-language model’s perception of the world at any resolution. arXiv Preprint. Note: https://arxiv.org/abs/2409.12191 Cited by: §3.1. [47] W. Wang, Z. He, W. Hong, Y. Cheng, X. Zhang, J. Qi, S. Huang, B. Xu, Y. Dong, M. Ding, and J. Tang (2024) LVBench: an extreme long video understanding benchmark. arXiv Preprint. Note: https://arxiv.org/abs/2406.08035 Cited by: §4.1. [48] W. Wang, Z. Gao, L. Gu, H. Pu, L. Cui, X. Wei, Z. Liu, L. Jing, S. Ye, J. Shao, Z. Wang, Z. Chen, H. Zhang, G. Yang, H. Wang, Q. Wei, J. Yin, W. Li, E. Cui, G. Chen, Z. Ding, C. Tian, Z. Wu, J. Xie, Z. Li, B. Yang, Y. Duan, X. Wang, Z. Hou, H. Hao, T. Zhang, S. Li, X. Zhao, H. Duan, N. Deng, B. Fu, Y. He, Y. Wang, C. He, B. Shi, J. He, Y. Xiong, H. Lv, L. Wu, W. Shao, K. Zhang, H. Deng, B. Qi, J. Ge, Q. Guo, W. Zhang, S. Zhang, M. Cao, J. Lin, K. Tang, J. Gao, H. Huang, Y. Gu, C. Lyu, H. Tang, R. Wang, H. Lv, W. Ouyang, L. Wang, M. Dou, X. Zhu, T. Lu, D. Lin, J. Dai, W. Su, B. Zhou, K. Chen, Y. Qiao, W. Wang, and G. Luo (2025) InternVL3.5: advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv Preprint. Note: https://arxiv.org/abs/2508.18265 Cited by: §2.1, §3.1. [49] X. Wang, Q. Si, J. Wu, S. Zhu, L. Cao, and L. Nie (2024) ReTaKe: reducing temporal and knowledge redundancy for long video understanding. arXiv Preprint. Note: https://arxiv.org/abs/2412.20504 Cited by: §1, §1, §2.2, §3.2. [50] X. Wang, Q. Si, S. Zhu, J. Wu, L. Cao, and L. Nie (2025) AdaReTaKe: adaptive redundancy reduction to perceive longer for video-language understanding. In ACL (Findings), p. 5417–5432. Cited by: §1, §1, §1, §2.2, §3.2, Table 1, Table 1, §4.3. [51] Y. Wang, X. Li, Z. Yan, Y. He, J. Yu, X. Zeng, C. Wang, C. Ma, H. Huang, J. Gao, M. Dou, K. Chen, W. Wang, Y. Qiao, Y. Wang, and L. Wang (2025) InternVideo2.5: empowering video mllms with long and rich context modeling. arXiv Preprint. Note: https://arxiv.org/abs/2501.12386 Cited by: §3.3.1. [52] Z. Wang, H. Wu, Y. Rong, D. Jiang, Y. Zhang, Y. Zhao, S. Xu, and B. Xu (2025) LVC: A lightweight compression framework for enhancing vlms in long video understanding. arXiv Preprint. Note: https://arxiv.org/abs/2504.06835 Cited by: §1. [53] Y. Weng, M. Han, H. He, X. Chang, and B. Zhuang (2024) LongVLM: efficient long video understanding via large language models. In ECCV, Lecture Notes in Computer Science, Vol. 15091, p. 453–470. Cited by: §1. [54] H. Wu, D. Li, B. Chen, and J. Li (2024) LongVideoBench: A benchmark for long-context interleaved video-language understanding. In NeurIPS, Cited by: §3.1, §4.1. [55] Q. Wu, W. Lin, W. Ye, Y. Zhou, X. Sun, and R. Ji (2024) Accelerating multimodal large language models via dynamic visual-token exit and the empirical findings. arXiv Preprint. Note: https://arxiv.org/abs/2411.19628 Cited by: §1. [56] Q. Wu, Y. Zhou, W. Ye, X. Sun, and R. Ji (2026) Not all attention is needed: parameter and computation efficient tuning for multi-modal large language models via effective attention skipping. International Journal of Computer Vision 134 (3), p. 128. Cited by: §3.1. [57] L. Xing, Q. Huang, X. Dong, J. Lu, P. Zhang, Y. Zang, Y. Cao, C. He, J. Wang, F. Wu, and D. Lin (2024) PyramidDrop: accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv Preprint. Note: https://arxiv.org/abs/2410.17247 Cited by: §3.3.2. [58] L. Xing, Q. Huang, X. Dong, J. Lu, P. Zhang, Y. Zang, Y. Cao, C. He, J. Wang, F. Wu, and D. Lin (2025) Conical visual concentration for efficient large vision-language models. In CVPR, p. 14593–14603. Cited by: §3.3.1. [59] Z. Xu, J. Zhang, Q. Wang, and Y. Liu (2025) E-VRAG: enhancing long video understanding with resource-efficient retrieval augmented generation. arXiv Preprint. Note: https://arxiv.org/abs/2508.01546 Cited by: §2.2. [60] J. Ye, H. Xu, H. Liu, A. Hu, M. Yan, Q. Qian, J. Zhang, F. Huang, and J. Zhou (2025) MPLUG-owl3: towards long image-sequence understanding in multi-modal large language models. In ICLR, Cited by: Table 3. [61] B. Yuan, S. You, and B. Bao (2025) DToMA: training-free dynamic token manipulation for long video understanding. In IJCAI, p. 2314–2322. Cited by: §1, Table 1. [62] H. Zhang, Y. Wang, Y. Tang, Y. Liu, J. Feng, J. Dai, and X. Jin (2024) Flash-vstream: memory-based real-time understanding for long video streams. arXiv Preprint. Note: https://arxiv.org/abs/2406.08085 Cited by: §1, Table 4. [63] P. Zhang, K. Zhang, B. Li, G. Zeng, J. Yang, Y. Zhang, Z. Wang, H. Tan, C. Li, and Z. Liu (2025) Long context transfer from language to vision. Trans. Mach. Learn. Res. 2025. Cited by: Table 3. [64] Y. Zhang, J. Wu, W. Li, B. Li, Z. Ma, Z. Liu, and C. Li (2025) LLaVA-video: video instruction tuning with synthetic data. Trans. Mach. Learn. Res. 2025. Cited by: §1, §3.1, §4.2, §4.3. [65] Z. Zhao, H. Lu, Y. Huo, Y. Du, T. Yue, L. Guo, B. Wang, W. Chen, and J. Liu (2025) Needle in A video haystack: A scalable synthetic evaluator for video mllms. In ICLR, Cited by: §1. [66] J. Zhou, Y. Shu, B. Zhao, B. Wu, S. Xiao, X. Yang, Y. Xiong, B. Zhang, T. Huang, and Z. Liu (2024) MLVU: A comprehensive benchmark for multi-task long video understanding. arXiv Preprint. Note: https://arxiv.org/abs/2406.04264 Cited by: §4.1. [67] Y. Zhou, R. Ji, X. Sun, J. Su, D. Meng, Y. Gao, and C. Shen (2019) Plenty is plague: fine-grained learning for visual question answering. IEEE transactions on pattern analysis and machine intelligence 44 (2), p. 697–709. Cited by: §1. [68] Y. Zhou, T. Ren, C. Zhu, X. Sun, J. Liu, X. Ding, M. Xu, and R. Ji (2021) Trar: routing the attention spans in transformer for visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 2074–2084. Cited by: §1. [69] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny (2024) MiniGPT-4: enhancing vision-language understanding with advanced large language models. In ICLR, Cited by: §1. [70] O. Zohar, X. Wang, Y. Dubois, N. Mehta, T. Xiao, P. Hansen-Estruch, L. Yu, X. Wang, F. Juefei-Xu, N. Zhang, S. Yeung-Levy, and X. Xia (2025) Apollo: an exploration of video understanding in large multimodal models. In CVPR, p. 18891–18901. Cited by: §4.1. [71] H. Zou, T. Luo, G. Xie, V. Zhang, F. Lv, G. Wang, J. Chen, Z. Wang, H. Zhang, and H. Zhang (2024) From seconds to hours: reviewing multimodal large language models on comprehensive long video understanding. arXiv Preprint. Note: https://arxiv.org/abs/2409.18938 Cited by: §3.1.