Paper deep dive
ECOKV: Geometry-Aware KV Cache Eviction via Complementary Diversity Metrics
Chin Ting Hsu, Yu-Syuan Xu, Ling Zou, Hsien-Kai Kuo, Wen-Huang Cheng
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 9/12/2026, 2:56:12 AM
Summary
The paper introduces ECOKV, a geometry-aware KV cache eviction framework for Multimodal Large Language Models (MLLMs). It addresses limitations of existing diversity metrics that rely solely on cosine similarity, which suffers from anisotropy and discards magnitude information. ECOKV proposes a composite metric combining Euclidean distance and cosine similarity to better capture token diversity. It also introduces a head-adaptive redundancy estimator to dynamically weight diversity and importance scores based on layer-specific redundancy. Additionally, the study finds that reducing the observation window size improves performance under tight memory budgets.
Entities (10)
Relation Signals (8)
ECOKV → appliesto → MLLMs
confidence 95% · In our study ECOKV... for MLLM KV Cache Compression
ECOKV → uses → Cosine Similarity
confidence 95% · we propose a geometry-aware composite metric that jointly leverages Euclidean distance and cosine similarity
ECOKV → uses → Euclidean Distance
confidence 95% · we propose a geometry-aware composite metric that jointly leverages Euclidean distance and cosine similarity
ECOKV → improves → KV Cache Eviction
confidence 90% · Extensive experiments demonstrate that ECOKV achieves state-of-the-art performance under various compression ratios
Cosine Similarity → suffersfrom → Anisotropy
confidence 90% · cosine similarity involves normalization that discards magnitude information, and it often yields uniformly high similarity values across layers due to the anisotropy property
H2O → istypeof → KV Cache Eviction
confidence 85% · H2O [49] identifies a small subset of tokens that dominate attention and prioritizes them as heavy hitters.
SnapKV → istypeof → KV Cache Eviction
confidence 85% · SnapKV [22] shows that only a subset of prompt tokens carries essential information for generation and compresses the KV cache
ECOKV → optimizes →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Although multimodal Large Language Models (MLLMs) excel in diverse tasks, their scalability remains limited by the memory and computational overhead of KV cache storage. Recent KV cache eviction approaches incorporate a cosine similarity-based diversity metric with importance metrics to selectively retain critical key-value pairs. However, cosine similarity involves normalization that discards magnitude information, and it often yields uniformly high similarity values across layers due to the anisotropy property of hidden representations. In our study ECOKV, we rigorously deconstruct the capabilities of existing diversity metrics. Moving beyond simple measurement, we propose a geometry-aware composite metric that jointly leverages Euclidean distance and cosine similarity to capture token diversity from complementary perspectives. Furthermore, we use these two metrics to estimate the redundancy level of each attention head, allowing adaptive weighting between diversity and importance scores during token selection. Finally, we demonstrate that the observation window commonly employed to preserve recent tokens can be substantially reduced, thereby allocating more cache capacity to informative tokens and yielding consistent improvements. Extensive experiments demonstrate that ECOKV achieves state-of-the-art performance under various compression ratios and can be seamlessly integrated with existing KV cache eviction methods. We further analyze the relationship between importance and diversity, and examine redundancy patterns across layers and attention heads.
Tags
Links
- Source: https://arxiv.org/abs/2609.06663v1
- Canonical: https://arxiv.org/abs/2609.06663v1
Trouble viewing inline? Open PDF directly →
Full Text
64,069 characters extracted from source content.
Expand or collapse full text
ECOKV: Geometry-Aware KV Cache Eviction via Complementary Diversity Metrics Chin Ting Hsu Yu-Syuan Xu Affiliation: MediaTek Inc. Ling Zou Hsien-Kai Kuo Affiliation: MediaTek Inc. Wen-Huang Cheng [10pt] National Taiwan University Abstract Although multimodal Large Language Models (MLLMs) excel in diverse tasks, their scalability remains limited by the memory and computational overhead of KV cache storage. Recent KV cache eviction approaches incorporate a cosine similarity-based diversity metric with importance metrics to selectively retain critical key-value pairs. However, cosine similarity involves normalization that discards magnitude information, and it often yields uniformly high similarity values across layers due to the anisotropy property of hidden representations. In our study ECOKV, we rigorously deconstruct the capabilities of existing diversity metrics. Moving beyond simple measurement, we propose a geometry-aware composite metric that jointly leverages Euclidean distance and cosine similarity to capture token diversity from complementary perspectives. Furthermore, we use these two metrics to estimate the redundancy level of each attention head, allowing adaptive weighting between diversity and importance scores during token selection. Finally, we demonstrate that the observation window commonly employed to preserve recent tokens can be substantially reduced, thereby allocating more cache capacity to informative tokens and yielding consistent improvements. Extensive experiments demonstrate that ECOKV achieves state-of-the-art performance under various compression ratios and can be seamlessly integrated with existing KV cache eviction methods. We further analyze the relationship between importance and diversity, and examine redundancy patterns across layers and attention heads. 1 Multimodal Large Language Model Efficiency Multimodal large language models (MLLMs) [46, 48, 47] have gained widespread attention for their ability to process and reason over multimodal inputs, including text, images, and audio, while providing contextually relevant responses. In the realm of computer vision, they have been widely adopted in various tasks, including visual question answering [2, 18], image captioning [10, 12], and object detection [15, 51]. Recent advances have enabled the processing of increasingly long and complex inputs [37, 3]. However, this progress introduces substantial computational and memory overhead, making inference efficiency a critical concern. To address this challenge, KV cache eviction methods [13, 19] offer a training-free alternative that reduces memory usage and inference latency while preserving model capability, by selectively removing redundant key-value pairs from the KV cache. Early KV cache eviction algorithms are primarily designed for Large Language Models (LLMs). They perform top-K selection within a fixed budget, retaining key-value pairs by importance scores estimated via accumulated attention score [49], average attention scores within an observation window [22], or approximated by heuristic metrics such as Key Norm (KNorm) [7] or Value Norm (VNorm) [17]. To account for the varying information density across layers and attention heads, subsequent work explores dynamic budget allocation frameworks that support adaptive and fine-grained budget sharing [8, 5]. Recently, diversity-aware criteria have been incorporated to improve coverage of the retained key-value set and mitigate redundancy [33, 6], yielding better preservation of the token distribution under constrained memory budgets. Recent studies have begun exploring KV cache eviction for MLLMs [43, 27, 21, 44, 42]. These works observe that visual tokens often exhibit higher redundancy than text tokens, motivating strategies that prioritize the preservation of text tokens or leverage diversity metrics to identify and remove redundant visual tokens. Despite these advances, existing diversity-based methods mostly rely on cosine similarity over normalized embeddings, which introduces a fundamental geometric limitation. Recent studies [24, 11] have highlighted the issue of anisotropy, where high-dimensional transformer embeddings concentrate within a narrow cone, resulting in uniformly high cosine similarity scores. As illustrated in Figure 1, cosine similarity produces consistently high average similarity scores across layers, exposing its tendency to overestimate redundancy and potentially corrupt eviction decisions. Moreover, cosine similarity operates on normalized vectors, thereby losing magnitude information. Relying on a single, biased metric without jointly accounting for complementary geometric information can lead to incorrect token prioritization during eviction, while overestimation of redundancy may induce an undue preference for diversity over importance across layers. Euclidean distance offers a potential measure of token diversity by capturing magnitude information in token representations. However, lacking explicit directional information, it has shown inferior performance for token diversity assessment in prior LLM studies [33]. Consequently, each metric captures distinct information while having inherent limitations, and the potential benefits of combining them into a composite metric remain largely unexplored. Figure 1: Distribution of layer-wise key cosine similarities in Qwen2-VL-7B-Instruct. Cosine similarity remains consistently high across all layers. To address the limitations, we propose Geometry-Aware KV Cache Eviction via Complementary Diversity Metrics (ECOKV), a novel eviction framework that integrates Euclidean distance and cosine similarity to capture richer geometric information for token diversity. Specifically, we introduce a Euclidean-based diversity metric that measures the distance between each key and the global average, followed by normalization to facilitate its integration with a cosine-similarity-based metric. Furthermore, we employ Euclidean distance to capture layer-wise redundancy while mitigating its unbounded scale issue. In addition, previous methods inherit the observation window size of 32 from [22], which has not been thoroughly examined for MLLMs under tight memory budgets. With a small cache budget, the observation window consumes a substantial portion of the cache, leaving fewer tokens for selection. We therefore evaluate different observation window sizes and find that smaller windows consistently improve performance by allocating more budget to our token selection strategy. Extensive experiments validate that ECOKV achieves state-of-the-art performance across multiple multimodal benchmarks and compression ratios, integrating seamlessly with existing eviction frameworks. We further analyze the relative weighting between Euclidean distance and cosine similarity, the effect of observation window size, and the layer-wise correlation between importance and diversity, offering insights into where diversity plays a critical role in key-value selection. Our key contributions are: • (i) We provide in-depth analysis on different diversity metrics and their impact on model performance • (i) We introduce a head-adaptive key diversity and redundancy estimator that jointly leverages Euclidean distance and cosine similarity to dynamically adjust diversity contribution across attention heads • (i) We observe that a reduced observation window strategy can improve performance under tight budget constraints • (iv) We achieve state-of-the-art performance on multimodel question answering benchmarks, and provide diverse empirical analysis on the choice of diversity implementation and the relation between different metrics 2 Related Work 2.1 LLM-based KV Cache Eviction Recent work has explored KV cache eviction to reduce the memory overhead of long-context inference in LLMs [19, 50, 9, 39, 25, 1]. H2O [49] identifies a small subset of tokens that dominate attention and prioritizes them as heavy hitters. SnapKV [22] shows that only a subset of prompt tokens carries essential information for generation and compresses the KV cache by retaining representative tokens. They assess the importance of keys and values using the observation window located at the end of the prompts. Subsequent works adopt these ideas and explore dynamic budget allocation across attention layers and heads. PyramidKV [5] reveals that the attention is diffuse in the early layers and gradually concentrates in the deep layers, thereby allocating more cache in the lower layers and less in the higher ones. AdaKV [8] finds that attention heads exhibit diverse concentration patterns and dynamically allocates cache budgets across heads accordingly. Another research direction investigates the estimation of token importance to determine which key-value pairs should be retained. InfiniPot-V [17] uses VNorm (value norm) as an importance proxy and introduces adaptive pooling to model spatial locality patterns across layers. Alternatively, [7] proposes KNorm (key norm) as the importance indicator, demonstrating that different components of KV representations can serve as effective signals for identifying important tokens. For diversity-based assessment, KeyDiff [33] proposes an attention-free cache eviction method based on key similarity. R-KV [4] balances redundancy and importance with a hyperparameter λ and performs eviction during decoding. ManifoldKV [6] ranks tokens solely based on their Euclidean distance to the key centroid, overlooking both token importance and directional information such as cosine similarity. Although they achieve decent performance in LLMs, directly applying them to MLLMs can lead to a performance drop due to distinct modality-specific characteristics. 2.2 MLLM-based KV Cache Eviction For MLLMs KV cache eviction, recent work aims to recognize the different characteristics between text and vision tokens. LOOK-M [43] observes that models often prioritize more textual attention over image features, inspiring them to prioritize text KV pairs during eviction. MEDA [42] extends the idea by introducing a dynamic layer-wise KV cache allocation mechanism. SparseMM [44] uses an anchor task to identify visual heads and allocates them a larger budget, as they are primarily responsible for processing visual information. SimiCache [28] first applies K-means clustering, followed by similarity-based clustering to group similar keys, and then merges them through a weighted sum. MadaKV [20] allocates separate budgets for different modalities and dynamically adjusts the budget size based on modality complexity across layers. MixKV [27] mixes importance with diversity and dynamically determines the weight of each metric. These methods acknowledge the high redundancy of vision tokens, but either overlook diversity or rely on incomplete geometric information, leading to inaccurate diversity estimation. 2.3 Similarity Metrics in Transformers Similarities between tokens or hidden embeddings are frequently used in LLMs and MLLMs research [27, 33, 6]. Cosine similarity is believed to capture semantic-level similarity and is therefore commonly used in NLP research [34, 30]. However, it sometimes performs worse than other approaches [14, 16]. [38] proposes that cosine similarity is implicitly controlled by regularization when learning deep models. In fact, cosine similarity in transformers is inherently high across various settings, a phenomenon known as anisotropy [24, 11]. In addition to cosine similarity, Euclidean distance is also adopted in some papers. For example, [40] shows that using a distance-based Euclidean metric yields a similarity measure less biased by the dimensionality of the hidden-state. For MLLMs KV Cache eviction, recent research [27, 28] inherits the assumption from LLMs and continues to use cosine similarity as the primary metric. In our paper, we propose using both Euclidean and cosine similarity metrics, which complement each other, to demonstrate that both are important for assessing token diversity in MLLMs. 3 Preliminary Inference in LLMs and MLLMs can be divided into two stages: the prefilling stage and the decoding stage. The prefilling stage processes the entire input prompt, while the decoding stage autoregressively generates new tokens. During the prefilling stage, for each transformer layer, the input x∈ℝT×dx ^T× d is projected through learnable weight matrices Wq∈ℝd×dqW_q ^d× d_q, Wk∈ℝd×dkW_k ^d× d_k, and Wv∈ℝd×dvW_v ^d× d_v to obtain the query Q, key K, and value V representations for self-attention which is computed as: Q=x⋅Wq,K=x⋅Wk,V=x⋅Wv, Q=x· W_q, K=x· W_k, V=x· W_v, Attention(Q,K,V)=softmax(QK⊤dk)V (Q,K,V)=softmax\! ( QK d_k )V To avoid recomputing K and V at each decoding step, the KV cache retains these representations across the entire generation process. However, as input context lengths grow, the KV cache size increases linearly with the sequence length, imposing substantial memory overhead and computational cost. To mitigate this, KV cache eviction methods selectively remove less informative key-value pairs to maintain a bounded cache size while preserving model performance. Let Kl,h=Kil,hi=1TK^l,h=\K^l,h_i\_i=1^T and Vl,h=Vil,hi=1TV^l,h=\V^l,h_i\_i=1^T denote the keys and values at layer l and attention head h, respectively, where T denotes the sequence length. We define a scoring function S(Kil,h,Vil,h)S(K^l,h_i,V^l,h_i) that assigns a scalar score to each key-value pair, retaining only the top-K pairs with the highest scores. The selected subset is defined as: K^l,h,V^l,h=Kil,h,Vil,h|S(Kil,h,Vil,h)∈Top-ki=1T K^l,h, V^l,h=\K^l,h_i,V^l,h_i\;|\;S\;(K^l,h_i,V^l,h_i) -k\_i=1^T, which is subsequently used to compute self-attention. The scoring function S typically quantifies either importance or diversity. Importance-based scoring employs an observation window of the most recent N key-value pairs to assess the importance of remaining pairs, preserving the window unconditionally due to recency and relevance. Diversity-based scoring assigns higher scores to outlier tokens to promote semantic diversity in the retained set. Combining both criteria, the final set of preserved key-value pairs is given by: K^l,h,V^l,h K^l,h, V^l,h =Kl,hi,Vl,hi|S(Kl,hi,Vl,hi)∈Top-ki=1T−N⏟diversity/importance-selected pairs = \K^l,h_i,V^l,h_i\; |\;S (K^l,h_i,V^l,h_i ) -k \_i=1^T-N_diversity/importance-selected pairs ∪Kil,h,Vil,hi=T−N+1T⏟observation window ∪\; \K^l,h_i,V^l,h_i \_i=T-N+1^T_observation window where the first term contains the top-k selected pairs from the non-window tokens, and the second term corresponds to the observation window that is preserved in its entirety. 4 Diversity Metrics Analysis Diversity metrics are widely adopted in token reduction and KV cache eviction methods to minimize information loss and mitigate redundancy within the retained subset. In the following paragraphs, we systematically analyze several commonly employed metrics for assessing token-level diversity, highlighting their respective strengths and limitations. Cosine Similarity. Cosine similarity measures the angular alignment between two vectors, making it sensitive to directional differences while remaining invariant to vector magnitudes. Given two vectors a,ba,b, it is defined as: CosSim(,)=⋅‖‖CosSim(a,b)= a·b\|a\|\|b\| It is widely adopted in NLP research and is often regarded as a proxy for semantic similarity between text tokens. However, by operating on normalized vectors, cosine similarity discards magnitude information, providing only a partial geometric information of token diversity. Moreover, recent work [24, 11, 40] shows that high-dimensional embeddings in transformers tend to concentrate within a narrow cone, leading to consistently high average cosine similarity. Figure 2 illustrates the cosine similarity across layers in Qwen-7B-Instruct [45]. We observe that the average similarity exceeds 0.40.4 in all layers and approaches 1.01.0 in the first layer. In contrast, PCA visualizations of the first layers reveal that the token representations are not identical but remain well spread in space. This discrepancy highlights a key limitation: by ignoring magnitude information, cosine similarity can overestimate the similarity between embeddings. Figure 2: (a) Cosine similarity remains consistently high across layers. (b) PCA projection of layer 0 keys. Although the embeddings are not identical, their cosine is approximately 1.0. Euclidean Distance. Euclidean distance measures the straight-line distance between two vectors, capturing differences in both magnitude and position: ECD(,)=‖−‖2=(−)⊤(−)ECD(a,b)=\|a-b\|_2= (a-b) (a-b) Unlike cosine similarity, Euclidean distance operates on unnormalized vectors, thereby preserving magnitude information. This relationship can be made explicit by expanding the squared distance: ECD(,)2=‖2+‖2−2‖‖CosSim(,)ECD(a,b)^2=\|a\|^2+\|b\|^2-2\|a\|\|b\|\,CosSim(a,b) This decomposition reveals that Euclidean distance jointly encodes vector magnitudes (‖2\|a\|^2, ‖2\|b\|^2) and directional relationship, capturing richer geometric information than cosine similarity alone. Nonetheless, Euclidean distance is unbounded, which necessitates an additional mapping from distance to a normalized similarity measure. Furthermore, its scale can vary significantly across layers, making direct comparison of diversity between layers challenging. 5 Method In this section, we propose a novel combined metric for MLLM KV Cache Compression by addressing two fundamental challenges: (a) how to design diversity metrics that assign reliable scores to each key (Sec. 5.1), and (b) how to assess layer-wise redundancy and appropriately balance diversity and importance scores (Sec. 5.2). Furthermore, we introduce our findings regarding observation window size adjustment in Sec. 5.3. 5.1 Key Diversity Score Assessment As analyzed in the previous section, cosine similarity suffers from anisotropy and lacks magnitude information. To provide enriched geometry information for key diversity score assessment, we incorporate Euclidean distance-based diversity metric as a complementary counterpart to cosine similarity. Specifically, we compute the centroid k¯ k of all key vectors and measure the Euclidean distance of each key kik_i from the centroid: k¯=1T∑i=1Tki,dE(i)=‖ki−k¯‖2 k= 1T _i=1^Tk_i, d_E(i)= k_i- k _2\\ (1) where dE(i)d_E(i) denotes the Euclidean distance between key kik_i and the centroid k¯ k. A larger dE(i)d_E(i) indicates that kik_i deviates more from the average representation, suggesting it carries more unique and non-redundant information. To eliminate the scale bias of Euclidean distance and map the distances to diversity scores from 00 to 11, we apply min-max normalization with a small constant ϵε for numerical stability: sE(i)=dE(i)−min(dE)max(dE)−min(dE)+ϵs_E(i)= d_E(i)- (d_E) (d_E)- (d_E)+ε (2) A higher sE(i)s_E(i) corresponds to a key that deviates more from the centroid, indicating a higher diversity contribution. Normalization will not change the relative order of the tokens, but is crucial for the following combination with cosine similarity-based scores. We then combine sE(i)s_E(i) with the cosine similarity-based diversity score sC(i)s_C(i) from [27] (details provided in supplementary section A.1), forming a unified diversity score that jointly captures both magnitude- and direction-aware information: Sdiv(i)=α⋅sE(i)+(1−α)⋅sC(i)S_div(i)=α· s_E(i)+(1-α)· s_C(i) (3) where α∈[0,1]α∈[0,1] is a combination coefficient that balances the contribution of magnitude-aware Euclidean diversity and direction-aware cosine diversity. Sdiv(i)S_div(i) is the diversity score for the ithi_th key value pairs and will be further combined with the important score to derive the final selection priority. 5.2 Head-adaptive Redundancy Estimation Next, we combine diversity-based scores with importance scores. Intuitively, layers or attention heads with higher redundancy should place greater emphasis on diversity to reduce redundant KV pairs. However, high cosine similarity does not necessarily indicate high redundancy, as it is an inherent property of transformer representations. Therefore, we design separate heuristics that combine different metrics to better approximate semantic-level diversity. We present the most effective approach here and defer alternative designs to supplementary section D. We propose a head-adaptive redundancy estimator that quantifies per-head redundancy by jointly leveraging Euclidean distance and cosine similarity to adaptively determine the diversity weight λ for each attention head. For each attention head h, we compute the mean Euclidean distance DEhD_E^h across all key vectors as a measure of the overall spread of the key distribution: DEh=1T∑i=1TdEh(i)D_E^h= 1T _i=1^Td_E^h(i) (4) where a larger DEhD_E^h indicates lower redundancy and a smaller DEhD_E^h implies higher redundancy. DEhD_E^h is then mapped to a normalized magnitude-aware redundancy score rE∈(0,1)r_E∈(0,1) via a scaled sigmoid: rE=2−2⋅σ(DEhτ)r_E=2-2·σ\! ( D_E^hτ ) (5) where τ is a temperature parameter controlling estimation sensitivity. We choose τ=dkτ= d_k, as the average distance scales proportionally with the square root of the key dimension. rEr_E is then combined with the cosine similarity-based redundancy score rCr_C from [27] (details provided in supplementary section A.1) to form a unified redundancy estimate that jointly captures magnitude and directional information: λ=β⋅rE+(1−β)⋅rCλ=β· r_E+(1-β)· r_C (6) where β∈[0,1]β∈[0,1] balances magnitude-aware and direction-aware contributions. More redundant heads receive a larger λ to emphasize diversity-based selection, and less redundant heads receive a smaller λ to prioritize importance-based scoring. The complete algorithm for deriving S and λ is provided in supplementary section C, and sensitivity analyses of α and β are presented in Sec. 6.3. Before combination, the diversity score SdivS_div is further scaled to match the magnitude of importance scores. Sdiv¯=Sdiv−minmax−min,Sdiv~=Sdiv¯⋅mean(Simp)mean(sC¯) S_div= S_div-minmax-min, S_div= S_div· mean(S_imp)mean( s_C) where SimpS_imp denotes the importance score, computed by combining the ℓ2 _2 norm of the value vector with the average attention score estimated over the observation window, following previous works. The final score combines the scaled diversity score S~div(i) S_div(i) and importance score Simp(i)S_imp(i) with the head-adaptive weight λ: S(i)=λ⋅S~div(i)+(1−λ)⋅Simp(i)S(i)=λ· S_div(i)+(1-λ)· S_imp(i) (7) The top-k pairs with the highest S(i)S(i) are retained for each attention head, enabling fine-grained adaptive eviction across layers and heads. 5.3 Minimized Observation Window Previous work utilizes an observation window to assess the importance of each token. The token within the observation window is preserved, assuming that recent tokens are inherently more important. However, across different benchmarks and modalities, the last few tokens are not always the most significant. Instead, arbitrarily keeping these tokens can reduce the budget for dynamic token selection. Our experiments show that reducing the observation window to a very small size—or even preserving only the most recent token—can lead to substantially better performance. To investigate this phenomenon, we conduct a detailed analysis of model performance across varying observation window sizes and KV budgets, as well as token selection distributions across benchmarks. The results demonstrate that a tiny observation window enables the KV cache eviction strategy to make more flexible and effective decisions about which tokens to retain, thereby improving utilization of the limited KV budget. Table 1: Performance comparison across benchmarks under different methods and models. Methods DocVQA (%) OCRBench (%) TextVQA (%) ChartQA (%) TextCaps 256 128 64 256 128 64 256 128 64 256 128 64 256 128 64 LLaVA-NeXT-Mistral-7B SnapKV 59.7 55.2 47.3 45.0 39.0 31.9 63.5 61.0 57.1 50.2 47.5 42.7 0.650 0.558 0.444 + MixKV 61.7 58.1 48.8 49.9 44.7 36.1 65.2 64.3 60.1 50.8 47.7 43.6 0.708 0.659 0.514 + ECOKV 62.7 60.8 56.9 49.7 48.4 43.7 65.5 65.1 63.0 52.5 51.5 48.8 0.792 0.721 0.678 Δbaseline _baseline +1.0 +2.7 +8.1 -0.2 +3.7 +7.6 +0.3 +0.8 +2.9 +1.7 +3.8 +5.2 +0.084 +0.062 +0.164 PyramidKV 58.2 54.3 43.4 44.1 39.4 29.1 62.9 60.9 54.8 49.1 47.1 40.8 0.621 0.553 0.407 + MixKV 60.8 57.2 45.1 49.7 43.7 32.0 64.9 63.8 57.8 50.8 47.5 41.3 0.687 0.656 0.466 + ECOKV 61.5 59.0 56.1 48.9 46.9 44.1 65.1 63.9 62.0 51.2 50.7 49.3 0.720 0.702 0.662 Δbaseline _baseline +0.7 +1.8 +11.0 -0.8 +3.2 +12.1 +0.2 +0.1 +4.2 +0.4 +3.2 +8.0 +0.033 +0.046 +0.196 AdaKV 59.6 55.9 48.7 45.1 40.4 32.8 62.9 60.5 56.9 50.4 47.8 44.6 0.646 0.566 0.440 + MixKV 61.3 58.3 50.8 49.8 44.9 36.6 65.3 63.7 59.6 50.9 48.5 45.2 0.704 0.660 0.509 + ECOKV 62.3 60.3 56.1 49.0 47.3 44.4 65.4 64.5 61.8 51.9 51.1 49.5 0.721 0.711 0.663 Δbaseline _baseline +1.0 +2.0 +5.3 -0.8 +2.4 +7.8 +0.1 +0.8 +2.2 +1.0 +2.6 +4.3 +0.017 +0.051 +0.154 SparseMM 61.6 60.8 57.6 51.9 50.7 46.2 65.1 64.7 62.8 51.9 51.2 48.9 0.680 0.634 0.524 + MixKV 61.9 61.0 59.2 50.8 50.4 49.5 65.2 65.0 64.4 51.8 51.5 50.6 0.682 0.652 0.575 + ECOKV 61.9 61.5 61.7 51.1 50.3 48.3 65.2 64.9 64.6 51.8 51.2 50.6 0.693 0.683 0.667 Δbaseline _baseline +0.0 +0.5 +2.5 +0.3 -0.1 -1.2 +0.0 -0.1 +0.2 +0.0 -0.3 +0.0 +0.011 +0.031 +0.092 FullKV 63.7 - - 52.7 - - 65.8 - - 52.9 - - 0.702 - - Qwen2-VL-7B-Instruct SnapKV 88.0 80.1 66.5 77.3 71.9 62.4 80.3 77.5 69.9 81.3 79.6 75.5 1.360 1.142 0.794 + MixKV 90.5 82.6 67.9 79.3 75.4 66.0 81.9 80.6 72.5 81.6 81.2 77.6 1.470 1.342 0.878 + ECOKV 93.4 90.4 81.8 80.9 78.9 74.8 82.1 82.1 78.8 81.8 81.6 81.0 1.517 1.496 1.294 Δbaseline _baseline +2.9 +7.8 +13.9 +1.6 +3.5 +8.8 +0.2 +1.5 +6.3 +0.2 +0.4 +3.4 +0.047 +0.154 +0.416 PyramidKV 81.7 74.0 59.9 74.5 67.9 56.8 78.3 74.6 65.3 81.1 79.2 73.5 1.115 0.951 0.569 + MixKV 84.0 76.3 60.8 76.6 72.6 58.4 80.4 77.1 67.0 81.3 80.7 75.5 1.348 1.119 0.633 + ECOKV 89.0 82.4 71.3 78.5 75.7 63.7 81.4 78.2 71.6 81.9 81.3 79.5 1.426 1.250 0.913 Δbaseline _baseline +5.0 +6.1 +10.5 +1.9 +3.1 +5.3 +1.0 +1.1 +4.6 +0.6 +0.6 +4.0 +0.078 +0.181 +0.280 AdaKV 87.4 81.2 67.1 77.8 71.0 62.1 79.9 77.0 70.3 80.8 79.6 75.9 1.345 1.146 0.775 + MixKV 90.3 82.1 67.8 79.3 74.7 65.5 81.8 79.6 71.2 81.5 80.9 77.4 1.448 1.275 0.878 + ECOKV 93.1 90.5 82.7 81.1 79.3 74.9 82.2 81.7 78.7 81.8 81.6 80.4 1.490 1.456 1.301 Δbaseline _baseline +2.8 +8.4 +14.9 +1.8 +4.6 +9.4 +0.4 +2.1 +7.5 +0.3 +0.7 +3.0 +0.042 +0.181 +0.423 SparseMM 93.5 91.5 84.9 81.2 79.0 74.3 82.0 81.6 77.3 82.0 81.5 80.1 1.482 1.430 1.038 + MixKV 93.8 92.7 86.4 82.0 81.0 77.1 82.0 82.0 80.9 81.6 81.8 81.4 1.480 1.459 1.303 + ECOKV 93.9 93.5 93.2 81.9 81.4 80.3 82.1 82.0 81.6 82.0 81.6 81.6 1.497 1.510 1.494 Δbaseline _baseline +0.1 +0.8 +6.8 -0.1 +0.4 +3.2 +0.1 +0.0 +0.7 +0.4 -0.2 +0.2 +0.017 +0.051 +0.191 FullKV 93.8 - - 82.3 - - 82.1 - - 81.8 - - 1.477 - - 6 Experiment 6.1 Experiment Setup We evaluate our method on two multimodal large language models: LLaVA-NeXT-Mistral-7B [26] and Qwen2-VL-7B-Instruct [45]. For standard multimodal benchmarks, we assess performance on OCRBench [29], ChartQA [31], TextCaps [35], TextVQA [36], and DocVQA [32] following the settings in previous works [44, 27]. We use LMMs-Eval [23] as a unified evaluation framework across all benchmarks. To demonstrate plug-and-play compatibility, we integrate ECOKV with four state-of-the-art KV cache eviction methods: SnapKV [22], PyramidKV [5], AdaKV [8], and SparseMM [44]. We compare against MixKV [27] and full KV cache (FullKV) as baselines. Concise introductions of the baseline methods and how we integrate ECOKV with them are provided in supplementary section A.1. The benchmark content and the score metrics reported are listed in supplementary section A.2. The hyperparameters for evaluation are provided in supplementary section A.3. All experiments are conducted on NVIDIA GeForce RTX 4090 GPUs with 24GB memory. 6.2 Main Results 6.2.1 Multimodal Question Answering Benchmarks As shown in Table 1, our method achieves state-of-the-art performance and can be seamlessly incorporated with baselines. The performance improvements are consistently observed across diverse benchmarks, model architectures, and budget settings. Notably, when the budget is set to 256, our approach already matches or even surpasses the performance of FullKV. Under more constrained budgets, our method significantly outperforms all baselines, demonstrating its effectiveness in retaining critical information. We also report Δbaseline _baseline, which denotes the performance difference between our approach and MixKV, to highlight the advantage of our method over relying solely on a single cosine similarity metric. 6.2.2 Long-context VQA We further evaluate our model’s ability on Long context VQA datasets: MP-DocVQA [41]. MP-DocVQA is a vision question answering dataset that features multiple-page documents as input, which has a longer context length and requires accurate extraction of helpful information. The experiments are conducted on Qwen2-VL-7B-Instruct. As shown in Table 2, our method consistently outperforms baselines across different compression ratios, validating its effectiveness in long-context settings. Table 2: Evaluation results on MP-DocVQA. Methods Budget Size 256 128 64 SnapKV 55.0 50.3 42.1 SnapKV + MixKV 56.2 51.7 42.8 SnapKV + ECOKV 58.0 56.1 51.3 PyramidKV 51.4 46.4 38.1 PyramidKV + MixKV 52.6 47.8 38.6 PyramidKV + ECOKV 55.5 51.5 44.5 6.3 Ablation Studies We conduct ablation studies to verify the effectiveness of the proposed components. The experiments are conducted on Qwen2-VL-7B-Instruct with SnapKV and PyramidKV to demonstrate consistency across different eviction frameworks, with ChartQA selected as the benchmark. Figure 3: Performance under different combination weights α and β. Smaller window size is not applied to isolate gains from metric design. Effect of Combination Weight We investigate the weighting coefficients α and β, which balance Euclidean distance and cosine similarity in the diversity metric and redundancy estimator. As shown in Figure 3, β≈0.6β≈ 0.6 consistently achieves strong performance, while optimal α varies between baseline methods. Notably, the optimal combination outperforms both pure Euclidean distance (α=β=0α=β=0) and pure cosine similarity (α=β=1α=β=1) configurations, demonstrating their complementary geometric properties. Although the optimal weighting coefficients vary across models and benchmarks, a fixed setting of α=0.8α=0.8 and β=0.6β=0.6 consistently achieves near-optimal performance, eliminating the need for exhaustive hyperparameter search. Analysis of Observation Window Size We examine the effect of observation window size by evaluating three different window sizes across various model and benchmark settings. As shown in Table 3, employing a tiny observation window (window size = 4) or preserving only the most recent token (window size = 1) consistently yields better performance across different benchmarks and model architectures. This supports our hypothesis that large observation windows consume the available budget, limiting the capacity for diversity- and importance-based selection. Notably, removing the observation window by only preserving and using the last query to assess importance scores still yields comparable or even superior result, indicating that the observation window potentially limit the model’s capability. We provide fine-grained window size ablation studies and more analysis in supplementary section B.1. Table 3: Reducing observation window size yields universal improvement. Window size 32 4 1 OCR 68.1 74.8 73.6 ChartQA 79.1 81.0 80.9 TextVQA 74.4 78.8 80.2 (a) Qwen2-VL-7B-Instruct Window size 32 4 1 OCR 36.2 43.7 39.7 ChartQA 43.8 48.8 49.0 TextVQA 60.1 63.0 62.8 (b) LLaVA-NeXT-Mistral-7B Normalization Strategy To assess the robustness of our min-max normalization to outliers and alternative scaling strategies, we conduct additional ablations on Qwen2-VL-7B-Instruct using ChartQA with SnapKV. Specifically, we compare our original min-max normalization (Eq. 2) with winsorized min-max normalization and per-head MAD scaling (with temperature set to 1). Winsorized min-max normalization reduces the influence of extreme values by clipping outliers, whereas MAD scaling provides a distribution-aware alternative based on the median absolute deviation. As shown in Table 4, all three strategies achieve comparable performance, suggesting that outliers do not have a significant impact on our method. Table 4: Comparison of different normalization methods under different cache budgets. Method 256 128 64 Original min-max 81.8 81.6 81.0 1%-99% winsorized min-max 81.8 81.6 80.8 5%-95% winsorized min-max 81.7 81.6 80.8 MAD 81.7 81.6 80.9 Efficiency Analysis To evaluate efficiency, we construct synthetic inputs with a context length of 8192 tokens and measure key metrics, including cache size, peak memory usage, and total execution time for generating 512 output tokens. For each method, experiments are repeated three times, and we report the average results to ensure robustness. All methods are evaluated under identical hardware conditions to guarantee a fair comparison. The budget size is set to 64 for all methods except FullKV. As shown in Table 5, incorporating ECOKV achieves efficiency comparable to baseline methods. This is because the additional computational overhead introduced by Euclidean distance computation is minimal, and is partially offset by reduced computation due to the smaller window size. In addition to the synthetic results, we also provide efficiency results on the actual benchmark in supplemental material section B.3 to further validate ECOKV’s efficiency in practical settings. Table 5: Average performance over 3 runs. Method Cache (GB) Peak Memory (GB) Total Time (s) FullKV 0.438 20.574 20.459 SnapKV 0.003 20.140 16.536 SnapKV + MixKV 0.003 20.140 16.483 SnapKV + ECOKV 0.003 20.140 16.408 PyramidKV 0.004 20.140 16.662 PyramidKV + MixKV 0.004 20.140 16.550 PyramidKV + ECOKV 0.004 20.140 16.540 6.4 Futher Analysis Correlation Between Importance and Diversity. We analyze the correlation between importance and diversity scores to understand whether they provide complementary or redundant signals for key-value selection. The experiment is conducted on Qwen2-VL-7B-Instruct using 100 randomly sampled test cases from ChartQA. Figure 4 reports the average Spearman correlation between importance-based and diversity-based rankings across layers, ordered from early to late layers. We observe a generally low to moderate correlation, suggesting that importance and diversity metrics prioritize keys in different orders. In addition, the correlation is lower in intermediate layers, indicating that the diversity metric has the greatest influence on key-value selection in those layers, where relying solely on importance-based scoring would result in a less representative retained set. We provide results on OCRBench, TextVQA, and TextCaps in supplementary section B.4 and exhibit similar patterns, consistently supporting these findings. Figure 4: Correlation between importance and diversity metrics. 7 Conclusion We present ECOKV, a novel KV cache eviction method that integrates complementary diversity metrics to provide a richer geometric characterization of token diversity. We employ the proposed composite metric to prioritize and retain critical key–value pairs, estimate layer-wise redundancy, and show that reducing the observation window can further improve performance under tight budget constraints. Experiments demonstrate that ECOKV consistently improves existing eviction methods across various benchmarks and compression ratios. Our analysis also validates the effectiveness of each component and the correlation between importance and diversity across layers. References [1] M. Adnan, A. Arunkumar, G. Jain, P. J. Nair, I. Soloveychik, and P. Kamath (2024) Keyformer: kv cache reduction through key tokens selection for efficient generative inference. Proceedings of Machine Learning and Systems 6, p. 114–127. Cited by: §2.1. [2] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh (2015) Vqa: visual question answering. In Proceedings of the IEEE international conference on computer vision, Cited by: §1. [3] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin (2025) Qwen2.5-VL technical report. External Links: Link Cited by: §1. [4] Z. Cai, W. Xiao, H. Sun, Y. Zhang, K. Wan, Y. Li, Y. Zhou, L. Chang, J. Gu, Z. Dong, et al. (2026) R-kv: redundancy-aware kv cache compression for reasoning models. Advances in neural information processing systems 38, p. 60980–61005. Cited by: §2.1. [5] Z. Cai, Y. Zhang, B. Gao, Y. Liu, Y. Li, T. Liu, K. Lu, W. Xiong, Y. Dong, J. Hu, et al. (2024) Pyramidkv: dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.02069. Cited by: 2nd item, §1, §2.1, §6.1. [6] D. Datta, T. Neeraj, B. Paudel, V. Sharma, and S. Mukherjee (2026) ManifoldKV: training-free kv cache compression via euclidean outlier detection. arXiv preprint arXiv:2602.08343. Cited by: §1, §2.1, §2.3. [7] A. Devoto, Y. Zhao, S. Scardapane, and P. Minervini (2024) A simple and effective L_2L\_2 norm-based strategy for KV cache compression. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Cited by: §1, §2.1. [8] Y. Feng, J. Lv, Y. Cao, X. Xie, and S. K. Zhou (2024) Ada-kv: optimizing kv cache eviction by adaptive budget allocation for efficient llm inference. arXiv preprint arXiv:2407.11550. Cited by: 3rd item, §1, §2.1, §6.1. [9] Y. Feng, J. Lv, Y. Cao, X. Xie, and S. K. Zhou (2025) Identify critical kv cache in llm inference from an output perturbation perspective. arXiv preprint arXiv:2502.03805. Cited by: §2.1. [10] T. Ghandi, H. Pourreza, and H. Mahyar (2023) Deep learning approaches on image captioning: a review. ACM Computing Surveys. Cited by: §1. [11] N. Godey, É. Clergerie, and B. Sagot (2024) Anisotropy is inherent to self-attention in transformers. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), Cited by: §1, §2.3, §4. [12] M. Z. Hossain, F. Sohel, M. F. Shiratuddin, and H. Laga (2019) A comprehensive survey of deep learning for image captioning. ACM Computing Surveys (CsUR). Cited by: §1. [13] J. Jiang, P. Yang, R. Zhang, and F. Liu (2026) Towards efficient large language model serving: a survey on system-aware kv cache optimization. In Findings of the Association for Computational Linguistics: ACL 2026, p. 38450–38476. Cited by: §1. [14] V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W. Yih (2020) Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), p. 6769–6781. Cited by: §2.3. [15] R. Kaur and S. Singh (2023) A comprehensive review of object detection with deep learning. Digital Signal Processing. Cited by: §1. [16] O. Khattab and M. Zaharia (2020) Colbert: efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, p. 39–48. Cited by: §2.3. [17] M. Kim, K. Shim, J. Choi, and S. Chang (2025) InfiniPot-v: memory-constrained KV cache compression for streaming video understanding. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §1, §2.1. [18] J. Kuang, Y. Shen, J. Xie, H. Luo, Z. Xu, R. Li, Y. Li, X. Cheng, X. Lin, and Y. Han (2025) Natural language understanding and inference with mllm in visual question answering: a survey. ACM Comput. Surv.. Cited by: §1. [19] H. Li, Y. Li, A. Tian, T. Tang, Z. Xu, X. Chen, N. Hu, W. Dong, Q. Li, and L. Chen (2024) A survey on large language model acceleration based on kv cache management. arXiv preprint arXiv:2412.19442. Cited by: §1, §2.1. [20] K. Li, Z. Jiang, Z. Shen, Z. Wang, C. Lv, S. Zhang, F. Wu, and F. Wu (2025) MadaKV: adaptive modality-perception KV cache eviction for efficient multimodal long-context inference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vienna, Austria. Cited by: §2.2. [21] K. Li, Z. Jiang, Z. Shen, Z. ZhaodeWang, C. Lv, S. Zhang, F. Wu, and F. Wu (2025) MadaKV: adaptive modality-perception kv cache eviction for efficient multimodal long-context inference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 13306–13318. Cited by: §1. [22] Y. Li, Y. Huang, B. Yang, B. Venkitesh, A. Locatelli, H. Ye, T. Cai, P. Lewis, and D. Chen (2024) SnapKV: llm knows what you are looking for before generation. Advances in Neural Information Processing Systems 37, p. 22947–22970. Cited by: 1st item, §1, §1, §2.1, §6.1. [23] B. Li*, P. Zhang*, K. Zhang*, F. Pu*, X. Du, Y. Dong, H. Liu, Y. Zhang, G. Zhang, C. Li, and Z. Liu (2024) LMMs-eval: accelerating the development of large multimoal models. External Links: Link Cited by: §A.2, §A.4, §6.1. [24] V. W. Liang, Y. Zhang, Y. Kwon, S. Yeung, and J. Y. Zou (2022) Mind the gap: understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems 35, p. 17612–17625. Cited by: §1, §2.3, §4. [25] A. Liu, J. Liu, Z. Pan, Y. He, G. Haffari, and B. Zhuang (2024) Minicache: kv cache compression in depth dimension for large language models. Advances in Neural Information Processing Systems 37, p. 139997–140031. Cited by: §2.1. [26] H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee (2024) Llavanext: improved reasoning, ocr, and world knowledge. Cited by: §6.1. [27] X. Liu, X. Gui, Y. Zhang, and L. Zhang (2026) Mixing importance with diversity: joint optimization for KV cache compression in large vision-language models. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: 5th item, §1, §2.2, §2.3, §5.1, §5.2, §6.1. [28] Y. Liu, F. Wu, R. Li, Z. Tang, X. Pan, and K. Li (2026) SimiCache: rethinking kv cache compression via token similarity for mllms. IEEE Transactions on Multimedia. Cited by: §2.2, §2.3. [29] Y. Liu, Z. Li, M. Huang, B. Yang, W. Yu, C. Li, X. Yin, C. Liu, L. Jin, and X. Bai (2024) OCRBench: on the hidden mystery of OCR in large multimodal models. Science China Information Sciences. Cited by: 4th item, 2nd item, §6.1. [30] Y. Mahajan, M. Freestone, N. Bansal, S. N. Aakur, and S. Karmaker (2025) Revisiting word embeddings in the llm era. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, p. 2686–2717. Cited by: §2.3. [31] A. Masry, X. L. Do, J. Q. Tan, S. Joty, and E. Hoque (2022) ChartQA: a benchmark for question answering about charts with visual and logical reasoning. In Findings of the association for computational linguistics: ACL 2022, Cited by: 4th item, §6.1. [32] M. Mathew, D. Karatzas, and C. Jawahar (2021) DocVQA: a dataset for VQA on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, Cited by: 1st item, §6.1. [33] J. Park, D. Jones, M. J. Morse, R. Goel, M. Lee, and C. Lott (2025) KeyDiff: key similarity-based kv cache eviction for long-context llm inference in resource-constrained environments. arXiv preprint arXiv:2504.15364. Cited by: §1, §1, §2.1, §2.3. [34] N. Reimers and I. Gurevych (2019) Sentence-bert: sentence embeddings using siamese bert-networks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), p. 3982–3992. Cited by: §2.3. [35] O. Sidorov, R. Hu, M. Rohrbach, and A. Singh (2020) TextCaps: a dataset for image captioning with reading comprehension. In European conference on computer vision, Cited by: 5th item, §6.1. [36] A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach (2019) Towards VQA models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Cited by: 3rd item, §6.1. [37] D. Song, S. Chen, G. H. Chen, F. Yu, X. Wan, and B. Wang (2024) MileBench: benchmarking mllms in long context. ArXiv abs/2404.18532. External Links: Link Cited by: §1. [38] H. Steck, C. Ekanadham, and N. Kallus (2024) Is cosine-similarity of embeddings really about similarity?. In Companion Proceedings of the ACM Web Conference 2024, p. 887–890. Cited by: §2.3. [39] J. Tang, Y. Zhao, K. Zhu, G. Xiao, B. Kasikci, and S. Han (2024) QUEST: query-aware sparsity for efficient long-context llm inference. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. Cited by: §2.1. [40] F. Tessari, K. Yao, and N. Hogan (2024) Surpassing cosine similarity for multidimensional comparisons: dimension insensitive euclidean metric. arXiv preprint arXiv:2407.08623. Cited by: §2.3, §4. [41] R. Tito, D. Karatzas, and E. Valveny (2023) Hierarchical multimodal transformers for multipage docvqa. Pattern Recognition 144, p. 109834. Cited by: §6.2.2. [42] Z. Wan, H. Shen, X. Wang, C. Liu, Z. Mai, and M. Zhang (2025) MEDA: dynamic kv cache allocation for efficient multimodal long-context inference. arXiv preprint arXiv:2502.17599. Cited by: §1, §2.2. [43] Z. Wan, Z. Wu, C. Liu, J. Huang, Z. Zhu, P. Jin, L. Wang, and L. Yuan (2024) Look-M: look-once optimization in kv cache for efficient multimodal long-context inference. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 4065–4078. Cited by: §1, §2.2. [44] J. Wang, Z. Liu, Y. Rao, and J. Lu (2025) Sparsemm: head sparsity emerges from visual concept responses in mllms. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 23177–23187. Cited by: 4th item, §1, §2.2, §6.1. [45] P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, et al. (2024) Qwen2-vl: enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191. Cited by: §4, §6.1. [46] Z. Yang, L. Li, K. Lin, J. Wang, C. Lin, Z. Liu, and L. Wang (2023) The dawn of lmms: preliminary explorations with gpt-4v(ision). ArXiv. External Links: Link Cited by: §1. [47] S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen (2024) A survey on multimodal large language models. National Science Review. Cited by: §1. [48] D. Zhang, Y. Yu, J. Dong, C. Li, D. Su, C. Chu, and D. Yu (2024) M-LLMs: recent advances in multimodal large language models. Findings of the Association for Computational Linguistics: ACL 2024. Cited by: §1. [49] Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett, et al. (2023) H2O: heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems. Cited by: §1, §2.1. [50] X. Zhou, W. Wang, M. Zeng, J. Guo, X. Liu, L. Shen, M. Zhang, and L. Ding (2024) DynamicKV: task-aware adaptive kv cache compression for long context llms. arXiv preprint arXiv:2412.14838. Cited by: §2.1. [51] Z. Zou, K. Chen, Z. Shi, Y. Guo, and J. Ye (2023) Object detection in 20 years: a survey. Proceedings of the IEEE. Cited by: §1. Appendix A Experimental Setup Details A.1 Baselines We briefly introduce the mechanisms proposed in the baseline methods. We incorporate our ECOKV with these methods by replacing their scoring function S with our approach and reduce the observation window size to 4. • SnapKV [22]: SnapKV compresses KV caches by selecting clustered important KV positions for each attention head during the prefilling stage. They introduce the idea of an observation window and design a voting mechanism to determine the top-K important keys: S=∑j=0Nobs[:,j,:],K^=Topk(S,k)S= _j=0^Nw_obs[:,j,:], K=Top_k(S,k) The attention weight is calculated for each query in the observation window, and accumulated as the importance score S. Then the selected key-value pairs index K K is determined by top-K selection. They further perform pooling to prevent losing details while evicting less important tokens. • PyramidKV [5]: PyramidKV dynamically allocates cache budgets across layers as attention is more diffuse in early layers and concentrated in late layers. They first allocate budgets for the top and bottom layers, which is bm−1=btotal/(β⋅m),b0=2⋅btotal/m−bm−1b^m-1=b^total/(β· m),b^0=2· b^total/m-b^m-1. For the middle layers, the budgets are calculated as follows: bl=b0−b0−bm−1m−1×lb^l=b^0- b^0-b^m-1m-1× l β is a hyperparameter that determines the shape of the pyramid. • AdaKV [8]: AdaKV reallocates budgets for attention heads within the same layer. They first concatenate attention weights across head as A=Cat(Ai)A=Cat(\A_i\), then perform top-K selection in A. After selecting the retained weights, they count the number of selected weights for each head i, and set the budget for each head accordingly. Aggregating budget within the same layer allows more dynamic allocation for attention heads. • SparseMM [44]: SparseMM utilizes OCRBench [29] to recognize visual heads, which actively contribute to visual understanding. For each attention head, the total budget bhlb^l_h is computed as follows: bhl=N+u+bijscoreb^l_h=N+u+b^score_ij where N is set to 32, preserving keys within the observation window. The term u denotes a uniformly allocated cache that occupies a small portion of the remaining budget and is evenly distributed across all attention heads. Finally, bijscoreb^score_ij determines the allocation of the remaining budget based on visual attention scores estimated from the inference on OCRBench. • MixKV [27]: MixKV combines the existing importance metric and the proposed cosine similarity-based diversity metric. They first derive the similarity between each normalized key and the global average: i=i‖i‖,μ=1T∑i=1TiK_N_i= K_i\|K_i\|, μ= 1T _i=1^TK_N_i sC(i)=−i⋅μs_C(i)=-K_N_i·μ iK_N_i denotes the normalized key. The diversity score for each key is computed by the dot product between the normalized key and the global average μ. The author then derives the off-diagonal average similarity to represent the redundancy level of the attention head: rC=T2|μ|2−T(T−1)r_C= T^2|μ|^2-TT(T-1) The redundancy level then serves as the weight of diversity metric to combine with the importance metric. Before combination, the diversity score sCs_C is further scaled to match the magnitude of importance scores. sC¯=sC−minmax−min,sC~=sC¯⋅mean(Simp)mean(sC¯) s_C= s_C-minmax-min, s_C= s_C· mean(S_imp)mean( s_C) Where SimpS_imp denotes the importance score. The final sC~ s_C is then combined with the importance score. A.2 Benchmarks Our main experiments include the following benchmarks: • DocVQA [32]: The benchmark evaluates whether the model recognizes the text in the documents and correctly comprehends the content. We report the Average Normalized Levenshtein Similarity (ANLS) scores of the validation split with 5.35k test cases. • OCRBench [29]: The benchmark includes 1000 question-answer pairs evaluating the comprehensive ability of MLLMs, including Text Recognition, SceneText-Centric VQA, Document-Oriented VQA, Key Information Extraction, and Handwritten Mathematical Expression Recognition. We report the accuracy evaluated on the whole benchmark. • TextVQA [36]: The benchmark contains questions that require the model to read the image and perform reasoning. We report scores of the validation split with 5k test cases. • ChartQA [31]: The benchmark includes real-world charts and human-authored question-answer pairs that evaluate the models’ capability on logical and arithmetic operations. We utilize the test split with 2.5k test cases in total. • TextCaps [35]: The benchmark requires model to comprehend the text on the images and generate the corresponding captions. We report the CIDEr score on the validation set with 6.5k test cases. We utilize LMMs-Eval [23] to evaluate our models with the above benchmarks. A.3 Hyperparameters We provide the hyperparameters used for evaluation in Table 6. The ratio and mask ratio parameters are only required by certain baseline methods, such as AdaKV and SparseMM. Table 6: Hyperparameters Hyperparameters ECOKV Baselines ratio 0.1 0.1 mask ratio 0.1 0.1 temperature 0.0 0.0 batch size 1 1 α 0.8 - β 0.6 - window size 4 32 kernel size 5 5 pooling average pool average pool fp16 TRUE TRUE A.4 Runtime Our main experiments are conducted using LMMs-Eval [23] without sampling, ensuring deterministic results across repeated runs. We report the approximate runtime for each benchmark in GPU-hours under our experimental configuration in Table 7. The runtime is measured on machines equipped with RTX 4090 GPUs and may vary depending on the hardware and system workload during reproduction. Table 7: Runtime in GPU-hours Benchmark LLaVA Runtime (GPU-hours) Qwen Runtime (GPU-hours) DocVQA 1.42 4.63 OCRBench 0.23 0.15 TextVQA 1.22 0.88 ChartQA 0.28 0.16 TextCaps 1.01 0.92 Appendix B Experimental Results and Analysis B.1 Observation Window Analysis Table 8: Performance under different window size. Methods Budgets Window size 32 16 8 4 2 1 SnapKV 256 81.8 81.8 81.8 81.8 81.7 81.8 128 80.9 81.5 81.5 81.6 81.7 81.8 64 79.1 80.0 80.7 81.0 81.0 80.9 PyramidKV 256 81.4 81.7 81.8 81.9 81.7 81.6 128 80.8 81.4 81.3 81.3 81.7 81.6 64 77.8 79.2 78.9 79.5 80.7 81.2 Table 8 presents finer-grained ablation studies on observation window size. The result further reveals that reducing the window size consistently improves performance under tight budget constraints (budget =64=64 and 128128). To better understand this phenomenon, we analyze the selection frequency of tokens within the last 32-token region. Specifically, for each token position, we compute the frequency with which it is retained in the KV cache and average the statistics across all layers and attention heads. The experiment is conducted on 4 benchmarks, with 100 randomly sampled test cases. The budget size is set to 64. As illustrated in Figure 5, different benchmarks exhibit distinct token selection patterns, suggesting that the importance of recent tokens varies across tasks and modalities. Furthermore, we observe that KV cache eviction algorithms naturally preserve recent tokens even without explicitly reserving a large observation window. This finding indicates that enforcing a large fixed observation window is often unnecessary and can instead reduce the budget available for dynamically selected informative tokens. B.2 Baselines with Tiny Observation Windows We provide additional experiments by reducing the observation window size for the baseline methods. The experiments are conducted on Qwen2-VL and evaluated on OCRBench. The results in Table 9 show that smaller observation windows consistently improve baseline performance, confirming that this design choice is effective beyond ECOKV. Moreover, under the same observation window size, ECOKV still consistently outperforms the baselines, demonstrating that our gains cannot be attributed solely to a smaller observation window. Table 9: Comparison of different KV cache eviction methods under different cache budgets. Method 256 128 64 SnapKV (32) 77.3 71.9 62.4 SnapKV (4) 79.3 77.5 70.4 SnapKV + MixKV (32) 79.3 75.4 66.0 SnapKV + MixKV (4) 81.1 78.3 74.0 SnapKV + ECOKV 80.9 78.9 74.8 PyramidKV (32) 74.5 67.9 56.8 PyramidKV (4) 76.8 72.0 54.6 PyramidKV + MixKV (32) 76.6 72.6 58.4 PyramidKV + MixKV (4) 77.6 74.8 59.5 PyramidKV + ECOKV 78.5 75.7 63.7 B.3 Efficiency analysis on real benchmark We replaced the synthetic data with TextVQA and evaluated the efficiency in over 100 randomly sampled test cases. As shown in Table 10, ECOKV achieves efficiency comparable to baselines while requiring a smaller KV cache and lower peak memory than FullKV. The reduction in total inference time is modest because the benchmark produces relatively short outputs, resulting in limited decoding time. Table 10: Comparison of cache size, peak memory usage, and total inference time. Method Cache (GB) Peak Memory (GB) Total Time (s) FullKV 0.0280 15.8299 1.0170 SnapKV 0.0034 15.8053 0.9464 SnapKV + MixKV 0.0034 15.8053 1.0543 SnapKV + ECOKV 0.0034 15.8053 1.0113 PyramidKV 0.0036 15.8054 0.9905 PyramidKV + MixKV 0.0036 15.8055 0.9928 PyramidKV + ECOKV 0.0036 15.8054 1.0392 B.4 Correlation between Importance and Diversity Figure 6 provides additional correlation results between importance and diversity across various benchmarks, offering further evidence for the analysis presented in the main paper section 6.4. B.5 Analysis of Diversity Weight We visualize the diversity weight across layers by sampling 100 test cases from both OCRBench and ChartQA, and compute the average diversity weight. As shown in Figure 7 and Figure 8, we plot the layer-averaged weights under cosine similarity and ECOKV Euclidean distance-based metric (rCr_C and rEr_E from main paper section 5.2), smoothed with a 3-layer simple moving average (SMA). The two metrics exhibit a similar overall trend across different benchmarks while differing in finer details, suggesting general consistency alongside layer-specific variations. These results also show that redundancy is relatively high in early layers, low in intermediate layers, and gradually increases toward the final layers. B.6 Global Average Calculation Alternative An alternative of computing the global average in Algorithm 1 Step 5 is to only average on non-window tokens. To compare with this alternative, we provide additional ablation studies on Qwen-2-VL, OCRBench: ( © indicates computing the centroid only on non-window tokens). The results in Table 11 show that the two variants exhibit similar behavior across settings, while our current implementation achieves slightly better performance. Table 11: Comparison of alternative global average implementation. Method \ Budget 256 128 64 SnapKV + ECOKV 80.9 78.9 74.8 SnapKV + ECOKV © 80.9 78.8 74.8 PyramidKV + ECOKV 78.5 75.7 63.7 PyramidKV + ECOKV © 78.5 75.8 62.8 Appendix C Algorithm We present the overall algorithm for computing the similarity and redundancy scores in Algorithm 1. The output Similarity S is a sequence of diversity scores corresponding to the input keys, while the redundancy λ is a scalar that quantifies the overall redundancy of the attention head. Algorithm 1 Similarity and Redundancy Computation 0: ∈ℝB×H×T×DK ^B× H× T× D, observation window size N 0: Similarity S, redundancy λ 1: Tv←T−NT_v← T-N, v←:TvK_v _:T_v Cosine: 2: ←Normalize()K_N (K), ←1T∑t μ← 1T _tK_N_t 3: sC←(^v⊤+1)/2s_C←( K_v μ +1)/2 4: rC←T‖2−1T−1r_C← T\| μ\|^2-1T-1 Euclidean: 5: ¯←1T∑i k← 1T _iK_i, ←‖v−¯‖2d_E←\|K_v- k\|_2 6: sE←−minmax−mins_E← D- - 7: rE←2−2σ(DEhτ)r_E← 2-2\,σ\! ( D^h_Eτ ) Combine: 8: S←αsE+(1−α)sCS←α s_E+(1-α)s_C 9: λ←βrE+(1−β)rCλ←β r_E+(1-β)r_C 10: return S,λS,λ Appendix D Head-adaptive Redundancy Estimation Alternatives Beyond combining the Euclidean distance with cosine similarity, we design additional heuristics and evaluate their comparative performance. D.1 Standard Deviation-based Approach As standard deviation primarily estimates the average distance to the mean, which is comparable to our Euclidean implementation in main paper section 5.2. The implementation we use in the main approach can be derived as: DE=1T∑i=1T‖ki−k¯‖2=1T∑i=1T(ki−k¯)2D_E= 1T _i=1^T k_i- k _2= 1T _i=1^T (k_i- k)^2 where k¯ k is the mean of keys. We can also compute standard deviation as: Dstd=1T∑i=1T(ki−k¯)2D_std= 1T _i=1^T(k_i- k)^2 To compare the two methods, we calculate the standard deviation of the keys and perform scaling as follows: rstd=1−σ(Dstdτ)r_std=1-σ\! ( D_stdτ ) (8) where τ is a temperature parameter. A larger standard deviation indicates a more dispersed distribution, corresponding to higher diversity and lower redundancy. Therefore, we invert the scaled value to obtain a redundancy estimate. rstdr_std is then used as the diversity weight. We conduct an ablation study by replacing the Euclidean distance with standard deviation and evaluate the resulting performance on ChartQA. As shown in Table 12, the standard-deviation-based approach achieves comparable performance overall, with a slight performance degradation at a budget size of 64. Table 12: Standard deviation comparison Methods Window size 256 128 64 Mean Euclidean distance 81.8 81.6 81.0 Standard deviation 81.8 81.6 80.8 Figure 5: The average selection rate of last 32 tokens across benchmark. Figure 6: Correlation between importance and diversity across DocVQA, OCRBench, TextVQA, and TextCaps. Figure 7: Diversity weight analysis: cosine similarity vs Euclidean distance on Qwen2-VL-7B-Instruct. Figure 8: Diversity weight analysis: cosine similarity vs Euclidean distance on LLaVA-NEXT-Mistral-7B.