Paper deep dive
When 1+1 Backfires: Rethinking Relevance-Diversity Balancing for Sample-Aware Visual Token Reduction
Yihang Huang, Peisong Wen, Chenhao Zhou, Difu Feng, Qianqian Xu
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Visual token compression has emerged as a critical technique for reducing the inference cost of Vision-Language Models. Existing methods select tokens based on text relevance, visual diversity, or a fixed combination of both---treating the relevance-diversity balance as a universal constant independent of the input. We conduct a systematic analysis to challenge this assumption. At the dataset level, grid-searching the optimal relevance-diversity weight per benchmark reveals substantial cross-task variation. More critically, we further maximize per-sample answer likelihood to recover sample-level oracle weights, finding that their distributions not only differ markedly across benchmarks but also exhibit high within-benchmark variance. Based on these observations, we propose SAVER, a framework for Sample-Aware relevance-diversity balanced Visual tokEn Reduction. We first derive a Geometric Intent Prior (GIP) from the spatial concentration of text-relevant tokens to infer a sample-specific relevance-diversity weight, and then apply this weight to jointly select tokens via a Weighted Text Centroid (WTC) for relevance scoring and a Visual Marginal Coverage Gain (VMCG) for diversity. SAVER requires no additional training and achieves sample-aware balancing entirely at inference time. Extensive experiments demonstrate that SAVER consistently outperforms state-of-the-art baselines across diverse benchmarks, retaining 96.5% of full-model performance on LLaVA-1.5-7B with only 11.1% of the original visual tokens, and further generalizing to LLaVA-NeXT and Mini-Gemini.
Tags
Links
- Source: https://arxiv.org/abs/2610.16615v1
- Canonical: https://arxiv.org/abs/2610.16615v1
Trouble viewing inline? Open PDF directly →
Full Text
53,358 characters extracted from source content.
Expand or collapse full text
When 1+1 Backfires: Rethinking Relevance-Diversity Balancing for Sample-Aware Visual Token Reduction Conference: Proceedings of the 34th ACM International Conference on Multimedia; November 10–14, 2026; Rio de Janeiro, BrazilProceedings of the 34th ACM International Conference on Multimedia (M ’26), November 10–14, 2026, Rio de Janeiro, BrazilDOI: 10.1145/3767308.3836590ISBN: 979-8-4007-2213-4/2026/11CCS: Computing methodologies Artificial intelligence Yihang Huang Affiliation: State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China email: huangyihang25s@ict.ac.cn , Peisong Wen Affiliation: School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China email: wenpeisong@ucas.ac.cn , Chenhao Zhou Affiliation: State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China email: zhouchenhao25@mails.ucas.ac.cn , Difu Feng Affiliation: State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China email: fengdifu24@mails.ucas.ac.cn and Qianqian Xu Affiliation: State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing Academy of Artificial Intelligence, Beijing, China email: xuqianqian@ict.ac.cn © c Abstract. Visual token compression has emerged as a critical technique for reducing the inference cost of Vision-Language Models. Existing methods select tokens based on text relevance, visual diversity, or a fixed combination of both—treating the relevance-diversity balance as a universal constant independent of the input. We conduct a systematic analysis to challenge this assumption. At the dataset level, grid-searching the optimal relevance-diversity weight per benchmark reveals substantial cross-task variation. More critically, we further maximize per-sample answer likelihood to recover sample-level oracle weights, finding that their distributions not only differ markedly across benchmarks but also exhibit high within-benchmark variance. Based on these observations, we propose SAVER, a framework for Sample-Aware relevance-diversity balanced Visual tokEn Reduction. We first derive a Geometric Intent Prior (GIP) from the spatial concentration of text-relevant tokens to infer a sample-specific relevance-diversity weight, and then apply this weight to jointly select tokens via a Weighted Text Centroid (WTC) for relevance scoring and a Visual Marginal Coverage Gain (VMCG) for diversity. SAVER requires no additional training and achieves sample-aware balancing entirely at inference time. Extensive experiments demonstrate that SAVER consistently outperforms state-of-the-art baselines across diverse benchmarks, retaining 96.5% of full-model performance on LLaVA-1.5-7B with only 11.1% of the original visual tokens, and further generalizing to LLaVA-NeXT and Mini-Gemini. Keywords: Vision-Language Models; Efficient Inference; Visual Token Reduction †c-license: by 1. Introduction Vision-Language Models (VLMs) (Liu et al., 2023; Bai et al., 2023b; Wang et al., 2024; Li et al., 2026a; Liu et al., 2024a; Alayrac et al., 2022; Chen et al., 2024; Lu et al., 2024; Li et al., 2023b; Hua et al., 2024; Li et al., 2026b), which jointly process visual and textual inputs, have achieved significant progress across a wide range of multimodal tasks, including visual question answering, image captioning, and document understanding. To faithfully encode image semantics, these models represent visual inputs as a large number of tokens. The resulting long token sequences directly inflate the input length, incurring O(N2)O(N^2) attention complexity in the Transformer (Vaswani et al., 2017) backbone and creating a critical computational bottleneck that limits the practical deployment of LVLMs. To mitigate this bottleneck, visual token reduction (Chen et al., 2025; Xing et al., 2024; Jeddi et al., 2025; Liu et al., 2024b; Zhang et al., 2025b; Wen et al., 2025; Fang et al., 2026; Zou et al., 2025; Zhang et al., 2025a; Dong et al., 2026; Shang et al., 2025; Lin et al., 2025; Jiang et al., 2025) has emerged as a promising direction. Existing methods can be broadly grouped into three families. Text-relevance methods (Chen et al., 2025; Xing et al., 2024) prioritize visual tokens that align strongly with the prompt, but often over-focus on a few high-response regions and discard peripheral context. Visual-diversity methods (Jeddi et al., 2025; Alvar et al., 2025) instead maximize coverage of the image content, but remain agnostic to the query and may waste token budget on irrelevant regions. More recent methods (Liu et al., 2024b; Zhang et al., 2025b; Wen et al., 2025) attempt to integrate both objectives within a unified pipeline. However, they do not explicitly model the sample-dependent trade-off between relevance preservation and visual coverage. As observed in prior work (Li et al., 2025), such input-agnostic balancing does not consistently outperform single-objective baselines, a phenomenon termed “1+1<11+1<1.” These findings suggest that the central challenge lies not merely in combining relevance and diversity, but in estimating an appropriate per-sample allocation between the two objectives. In this work, we investigate this phenomenon through a systematic two-level analysis (Section 3). Specifically, we formulate the token selection score as a weighted combination of text relevance and visual diversity, controlled by a scalar weight θ, and greedily select tokens in descending order of this score. At the dataset level, we find that each benchmark exhibits a clear optimal relevance-diversity weight that delivers noticeably better performance than alternative allocations, while the locations of these optima vary substantially across tasks. We further investigate the source of this variation at the instance level. Since answer correctness is discrete and does not provide a continuous optimization signal, we use per-sample answer likelihood as a surrogate objective to recover oracle weights for individual inputs. The recovered oracle weights further show that the sample-level preference is consistent with the relevance-diversity tendency observed at the dataset level. These results suggest that effective token reduction fundamentally requires sample-aware relevance-diversity balancing. Closing this gap is non-trivial. A practical sample-aware token reduction method must satisfy two key requirements. 1). The prior must be derived from existing knowledge and the current input, without relying on any task-specific or dataset-level context unavailable at inference time. This is precisely the limitation of MoB (Li et al., 2025): although it provides a principled theoretical account of the relevance-diversity trade-off, its allocation strategy must be configured per dataset. In open-world deployment, however, the dataset identity of an incoming input is simply unknown. 2). The prior must be obtained within the early prefill stage from shallow semantics alone, without auxiliary modules or additional inference beyond the compression pipeline. Relying on an extra predictor, or waiting for more complete semantics from deeper layers, would introduce substantial pre-pruning cost and compromise the efficiency gains of token reduction, thereby undermining its very purpose as an inference acceleration technique. Crucially, we observe that the spatial distribution of high-relevan-ce visual tokens provides an effective cue for the desired relevance-diversity balance. When high-relevance tokens are spatially concentrated, the textual semantics are strongly aligned with a specific local region of the image, which is likely to contain the information most relevant to answering the query. When high-relevance tokens are broadly dispersed, the model cannot localize truly relevant visual tokens with high confidence, making relevance scores less reliable as a basis for selection. In this case, a larger diversity weight is preferred to preserve broad visual coverage and maintain prediction accuracy. Driven by these observations, we propose a Sample-Aware Rele-vance-Diversity Balanced Token Reduction framework. Its core component is the Geometric Intent Prior (GIP), a training-free prior that quantifies the text-visual coupling of each sample from the spatial covariance of its text-relevant visual tokens. GIP produces a continuous scalar that adaptively modulates the relative weight of each objective, enabling sample-specific objective balancing at negligible cost. Complementing GIP, we introduce two objective-specific improvements: a norm-weighted text centroid for text relevance estimation, free of cross-modal attention and compatible with FlashAttention (Dao et al., 2022); and a facility location formulation for visual coverage whose marginal gain jointly accounts for novelty and representativeness. Extensive evaluations demonstrate that our method retains 96.5% of full-model performance using only 11% of the visual tokens, consistently outperforming existing approaches under equivalent compression budgets. Our main contributions are summarized as follows: • Analysis for Relevance-Diversity Balancing. We conduct a systematic two-level empirical analysis showing that the optimal relevance-diversity balance varies substantially across both tasks and individual samples, establishing that per-sample balancing of the relevance-diversity objectives is necessary for effective visual token compression. • Solution to Sample-Aware Objective Balancing. We propose GIP, a training-free, efficient solution to sample-aware relevance-diversity balancing, which derives a sample-specific weight from the spatial distribution of text-relevant tokens. • SAVER Framework. We propose a Weighted Text Centroid for text relevance scoring and a Visual Marginal Coverage Gain for diversity-aware token selection. By dynamically balancing these two objectives via GIP, our proposed framework SAVER achieves superior performance over competitive baselines across multiple benchmarks and generalizes to diverse model architectures. Figure 1. Dataset-level performance vs. θ (dashed line) and instance-level oracle weight distributions θi∗ _i^* (solid curve and histogram) for POPE, GQA, and MMBench. POPE favors high θ (diversity-dominant); MMBench favors low θ (relevance-dominant); GQA peaks at intermediate values. Per-sample distributions closely correspond to dataset-level optima. 2. Related Work 2.1. Large Vision-Language Models Large Vision-Language Models (LVLMs) (Liu et al., 2023; Bai et al., 2023b; Wang et al., 2024; Li et al., 2026a; Liu et al., 2024a) have emerged from the foundational success of Large Language Models (LLMs) (Touvron et al., 2023; Achiam et al., 2023; Bai et al., 2023a; Chiang et al., 2023; HIT-SCIR, 2024). The standard LVLM pipeline consists of three core components. A visual encoder, typically a pre-trained vision transformer such as CLIP (Radford et al., 2021), extracts a sequence of visual tokens from the input image. A text tokenizer independently processes the natural language instruction. Both modalities are then projected into a shared embedding space and fed into a core LLM for joint multimodal reasoning. This architecture has achieved strong performance across a wide range of tasks, including visual question answering, image captioning, and document understanding. Despite these advances, faithfully encoding image semantics requires generating a large number of visual tokens. This directly inflates the input sequence length, which in turn substantially increases computational overhead and bottlenecks inference efficiency. 2.2. Token Reduction Methods Existing token reduction methods largely fall into two categories: text-relevance and visual coverage. Text-relevance methods use cross-attention maps to select tokens that align with the text prompt. FastV filters tokens by raw attention scores (Chen et al., 2025), and PyramidDrop extends this into a progressive, layer-wise pruning scheme (Xing et al., 2024). These methods quickly locate core semantic regions. Yet they tend to over-select tokens near these hubs and discard important peripheral context. Visual coverage methods instead maximize the semantic diversity of the selected token set. SAINT prunes via bipartite graph node degrees (Jeddi et al., 2025), and DivPrune formulates redundancy reduction as a Max-Min Diversity Problem (Alvar et al., 2025). These methods recover the global visual context more faithfully. However, they are task-agnostic. They do not consider the text instruction during selection, which can waste token budget on irrelevant regions. Recent work has attempted to combine both objectives within a unified pipeline (Zhang et al., 2025b; Wen et al., 2025). Most such approaches use fixed weighting schemes that do not adapt to the input. In practice, the optimal balance between relevance and diversity varies considerably across different questions. Li et al. formalize this trade-off using ϵε-covering theory (Li et al., 2025). They show that multimodal datasets split into two regimes—weak and strong text-visual coupling—and that each regime requires different allocation parameters for best performance. A limitation of their method is that it requires knowing the target dataset in advance to set these parameters. This limits applicability in open-ended settings where the task intent is unknown at inference time. 3. Motivation This section presents a systematic empirical study of the relevance-diversity trade-off in visual token compression. We first formulate the dataset-level optimal relevance-diversity weight and identify how it varies across benchmarks. To gain deeper insight into the origins of this cross-dataset variation, we further investigate the distribution of instance-level optimal weights within each benchmark. Since no continuous metric directly reflects token selection quality at the sample level, we introduce per-sample answer likelihood as a surrogate objective to recover these instance-level weights. Based on both levels of analysis, we report empirical observations that motivate the design of our method. Dataset-Level Optimal Weight. Given a multimodal input with visual token set V=v1,…,vNV=\v_1,…,v_N\ and text token sequence T, we define the per-token selection score as a weighted combination of text relevance Rel(⋅,T)Rel(·,T) and visual diversity Div(⋅)Div(·): (1) Scoreθ(vj,T)=(1−θ)⋅Rel(vj,T)+θ⋅Div(vj),Score_θ(v_j,\,T)=(1-θ)·Rel(v_j,\,T)\;+\;θ·Div(v_j), where θ∈[0,1]θ∈[0,1] controls the relevance-diversity balance, with larger θ corresponding to greater diversity weight. Under a fixed retention budget K, we select the top-K tokens by score: (2) Vθ=TopKScoreθ(vj,T)j=1N.V_θ=Top_K \Score_θ(v_j,\,T) \_j=1^N. For each benchmark kD_k with task evaluation metric ℳM, the dataset-level optimal weight is defined as: (3) θk∗=argmaxθ∈[0,1]1|k|∑(x,Y)∈kℳ(fϕ(Vθ,T),Y), _k^*= _θ∈[0,1]\; 1|D_k| _(x,\,Y) _kM\! (f_φ\! (V_θ,\,T ),\,Y ), where Y is the ground-truth answer. Instance-Level Oracle Weight. To probe the distribution of optimal weights at the sample level, we define the oracle weight for each sample i as the θ that maximizes the likelihood of the correct answer under the LLM Φ . Formally, let ℒi(θ)L_i(θ) denote the negative log-likelihood (NLL) of the ground-truth answer given θV_θ and T: (4) ℒi(θ)=−1|y|∑t=1|y|logPfϕ(yt∣Vθ,T,y<t).L_i(θ)=- 1|y| _t=1^|y| P_f_φ\! (y_t V_θ,\,T,\,y_<t ). The instance-level oracle weight is then defined as: (5) θi∗=argminθ∈[0,1]ℒi(θ). _i^*= _θ∈[0,1]\;L_i(θ). We recover θi∗ _i^* for sampled instances from each benchmark kD_k and analyze the resulting distributions. Figure 2. Overview of the SAVER framework. Given a multimodal input, we compute the Geometric Intent Prior (GIP) from the spatial distribution of high-relevance visual tokens, which dynamically weights text relevance (via WTC) and visual diversity (via VMCG) in a unified greedy selection process. Empirical Observations. As illustrated by the dashed lines in Figure 1, θk∗ _k^* varies substantially across benchmarks: POPE (Li et al., 2023c) favors high θ (diversity-dominant), MMBench (Liu et al., 2024c) favors low θ (relevance-dominant), and GQA (Hudson and Manning, 2019) peaks at an intermediate value. The per-sample distributions θi∗\ _i^*\ are well aligned with their respective dataset-level optima, and this pattern is consistent with the intuitive characteristics of each task. POPE primarily probes object hallucination by querying small or absent objects, requiring the model to survey diverse regions across the visual field; MMBench queries fine-grained local details, where focused retrieval of the relevant region is decisive. Critically, both distributions also exhibit substantial within-benchmark variance, indicating that the optimal balance is not a property of the task alone but varies at the level of individual inputs. Algorithm 1 Sample-Aware Token Reduction 1: V=vii=1NV=\v_i\_i=1^N, T=tii=1LT=\t_i\_i=1^L, budget K 2: γ,k,bγ,\;k,\;b 3: S with |S|=K|S|=K 4: ←∑i‖ti‖2∑j‖tj‖2tic← _i \|t_i\|_2 _j\|t_j\|_2\,t_i; Rel(vi)←cos(vi,)∀iRel(v_i)← (v_i,c)\ ∀\,i 5: ←det(Cov((xi,yi):vi∈TopK(V,⌊0.02N⌋)))G← \! (Cov\! (\(x_i,y_i):v_i (V, 0.02N )\ ) ) 6: ~←() G _G(G) 7: kGIP←k(1−1N(N−1)∑p≠qcos(vp,vq))k_GIP← k\,N_D\! (1- 1N(N-1) _p≠ q (v_p,v_q) ) 8: θ←sigmoid(kGIP⋅~+b)θ \! (k_GIP· G+b ) 9: S←∅S← 10: while |S|<K|S|<K do 11: Div(vi∣S)←−γmaxs∈SKsiDiv(v_i S)←-γ _s∈ SK_si 12: +(1−γ)∑j∉S,j≠imax(0,Kij−maxs∈SKsj),∀vi∉S+(1-γ) _j∉ S,\,j≠ i \! (0,\;K_ij- _s∈ SK_sj ), ∀\,v_i∉ S 13: v∗←argmaxvi∉S(1−θ)Rel(vi)+θDiv(vi∣S)v^*← _v_i∉ S\;(1-θ)\,Rel(v_i)+θ\,Div(v_i S) 14: S←S∪v∗S← S∪\v^*\ 15: end while 16: return S 4. Methodology In this section, we present our Sample-Aware Unified Token Reduction framework. We first introduce a Geometric Intent Prior (GIP) in Section 4.2, which infers the sample-specific text-visual coupling from the spatial distribution of text-relevant visual tokens and adaptively balances the two selection objectives. We then describe our reformulated text relevance objective via a Weighted Text Centroid (WTC) in Section 4.3, and our visual diversity objective via a Visual Marginal Coverage Gain (VMCG) in Section 4.4. Finally, we present the unified token selection procedure in Section 4.5. The overall framework is illustrated in Figure 2. Table 1. Comparison with state-of-the-art token reduction methods on LLaVA-1.5-7B across three compression budgets. Method GQA MMBench MME POPE ScienceQA TextVQA OCRBench Avg. No compression (576 tokens) LLaVA-1.5-7B 61.9 64.3 1862 85.9 69.5 58.2 297 100% 192 tokens (66.7% reduction) FastV (ECCV’24) (Chen et al., 2025) 52.7 61.2 1612 64.8 67.3 52.5 291 89.6% SparseVLM (ICML’25) (Zhang et al., 2025b) 57.6 62.5 1721 83.6 69.1 56.1 292 96.3% PyramidDrop (CVPR’25) (Xing et al., 2024) 57.1 63.2 1766 82.3 68.8 56.1 290 96.3% DivPrune (CVPR’25) (Alvar et al., 2025) 59.9 62.4 1752 87.0 68.9 57.0 303 98.3% DART (EMNLP’25) (Wen et al., 2025) 60.0 63.6 1856 82.8 69.8 57.4 296 98.7% VisionZip (CVPR’25) (Yang et al., 2025) 59.3 63.0 1783 85.3 68.9 57.3 – 97.7% PruneSID (ICLR’26) (Fang et al., 2026) 60.1 63.7 1791 86.9 68.5 56.7 294 98.4% SAVER (Ours) 61.2 63.8 1855 85.7 69.3 57.8 300 99.6% 128 tokens (77.8% reduction) FastV (ECCV’24) (Chen et al., 2025) 49.6 56.1 1490 59.6 60.2 50.6 285 83.8% SparseVLM (ICML’25) (Zhang et al., 2025b) 56.0 60.0 1696 80.5 67.1 54.9 280 93.4% PyramidDrop (CVPR’25) (Xing et al., 2024) 56.0 61.1 1644 82.3 68.3 55.1 287 94.2% DivPrune (CVPR’25) (Alvar et al., 2025) 59.2 62.3 1752 86.6 68.0 54.5 283 96.3% DART (EMNLP’25) (Wen et al., 2025) 58.7 63.2 1840 80.1 69.1 56.4 296 97.3% VisionZip (CVPR’25) (Yang et al., 2025) 57.6 62.0 1762 83.2 68.9 56.8 – 96.3% PruneSID (ICLR’26) (Fang et al., 2026) 58.8 62.1 1749 86.5 68.3 54.7 275 95.9% SAVER (Ours) 60.9 63.6 1811 85.5 69.2 57.2 287 98.4% 64 tokens (88.9% reduction) FastV (ECCV’24) (Chen et al., 2025) 46.1 48.0 1256 48.0 51.1 47.8 245 72.9% SparseVLM (ICML’25) (Zhang et al., 2025b) 52.7 56.2 1505 75.1 62.2 51.8 180 82.8% PyramidDrop (CVPR’25) (Xing et al., 2024) 41.9 33.3 1092 55.9 68.6 45.9 250 72.1% DivPrune (CVPR’25) (Alvar et al., 2025) 57.6 59.5 1638 85.8 68.6 52.9 275 93.7% DART (EMNLP’25) (Wen et al., 2025) 55.9 60.6 1765 73.9 69.8 54.4 270 92.9% VisionZip (CVPR’25) (Yang et al., 2025) 55.1 60.1 1690 77.0 69.0 55.5 – 92.9% PruneSID (ICLR’26) (Fang et al., 2026) 57.1 58.8 1733 83.8 67.8 54.2 273 93.8% SAVER (Ours) 60.0 61.8 1758 83.8 68.9 56.2 282 96.5% 4.1. Preliminaries Formally, we denote the N visual tokens produced by the visual encoder as V=vii=1NV=\v_i\_i=1^N and the L text tokens from the text tokenizer as T=tii=1LT=\t_i\_i=1^L. The concatenated sequence [V;T][V;T] is fed into the LLM decoder fϕf_φ, which autoregressively generates the response y=fϕ([V;T])y=f_φ([V;T]). Visual token reduction selects a subset S⊆VS V with |S|=K≪N|S|=K N at an early layer of the LLM, so that the decoder operates on the compressed sequence [S;T][S;T] instead, reducing prefilling cost. 4.2. Geometric Intent Prior As established in Section 3, the optimal relevance-diversity balance is inherently sample-dependent, yet existing methods treat it as a fixed, input-agnostic hyperparameter. Accessing deep cross-modal representations from the LLM backbone could in principle provide rich cues for estimating the sample-specific balance, but doing so requires a full forward pass—the very computation that token reduction is designed to avoid. A practical solution must therefore rely on a proxy derivable from shallow representations in the early prefill stage, before the full depth of the LLM is engaged. We propose to use the spatial distribution of high-relevance visual tokens as such a proxy. The key observation is that the spatial spread of these tokens reflects the degree to which the input query can be resolved by a localized region of the image. When high-relevance tokens are spatially concentrated, the query is grounded in a specific local region, and a higher relevance weight is appropriate to retrieve detail-oriented tokens from that region. When they are broadly dispersed, the text tokens lack strong semantic alignment with any specific visual region, making relevance scores unreliable as a selection criterion; in this case, a higher diversity weight is warranted to ensure broad spatial coverage and allow the model to attend to global visual details. This spatial geometry is fully determined from shallow token representations within the LLM, requiring no additional training or full forward pass. We formalize this as the Geometric Intent Prior (GIP). Formally, let =vii=1NV=\v_i\_i=1^N denote the set of visual tokens and let Rel(vi)Rel(v_i) denote the text relevance score of each token (defined in Section 4.3). We select the top-k most text-relevant tokens to form the anchor set: (6) k=vi∈∣Rel(vi)∈TopK(Rel(vj)j=1N,k).V_k= \v_i (v_i) (\Rel(v_j)\_j=1^N,\,k) \. We set k=⌊0.02N⌋k= 0.02N , i.e., the top 2% of visual tokens by relevance score. As validated in our experiments, tokens in this percentile exhibit significantly higher relevance scores than the remainder and carry rich semantic content, making them reliable indicators of the query’s visual grounding. Each token vi∈kv_i _k corresponds to a spatial patch at 2D coordinate i=(xi,yi)∈ℝ2p_i=(x_i,y_i) ^2. Let =ii=1kP=\p_i\_i=1^k denote the corresponding spatial point set. We compute the 2×22× 2 spatial covariance matrix: (7) =1k∑i=1k(i−¯)(i−¯)⊤, = 1k _i=1^k(p_i- p)(p_i- p) , where ¯=1k∑i=1ki p= 1k _i=1^kp_i is the spatial centroid of P. We define the Geometric Intent Prior as its determinant: (8) =det().G= ( ). G equals the generalized variance of P, measuring the area of the covariance ellipsoid spanned by the anchor tokens. A small G indicates that the high-relevance tokens are spatially concentrated; a large G indicates broad spatial dispersion. How G is mapped to the objective weighting scalar θ is described in Section 4.5. 4.3. Text Relevance via Weighted Text Centroid Previous token reduction methods (Chen et al., 2025; Zhang et al., 2025b; Wen et al., 2025) estimate text relevance from cross-modal attention maps within the LLM, measuring the relevance of each visual token by its accumulated attention weight toward text tokens. However, two limitations hinder this approach in practice. First, modern inference pipelines increasingly adopt FlashAttention (Dao et al., 2022), which fuses the attention kernel and does not materialize the full attention matrix, rendering attention-based relevance estimation inapplicable. Second, LLM attention maps are susceptible to positional biases (Zhang et al., 2025b), where early tokens attract disproportionately high attention irrespective of their semantic content, introducing systematic noise into the relevance scores. We instead propose to estimate text relevance via the semantic centroid of the text token embeddings, which is both attention-free and compatible with any inference backend. A naive unweighted mean, however, is susceptible to interference from function words and punctuation tokens, which contribute equally despite carrying little semantic content. Empirically, content words exhibit consistently higher L2L_2 norms in the embedding space than function words and punctuation. We exploit this property to define a norm-weighted centroid, termed Weighted Text Centroid (WTC), which naturally suppresses semantically weak tokens. Formally, let =tii=1LT=\t_i\_i=1^L denote the text token embeddings. The normalized L2L_2 norm of each token serves as its weight: (9) wi=‖ti‖2∑j=1L‖tj‖2.w_i= \|t_i\|_2 _j=1^L\|t_j\|_2. The Weighted Text Centroid is defined as: (10) =∑i=1Lwiti.c= _i=1^Lw_i\,t_i. The text relevance score of each visual token viv_i is then its cosine similarity with c: (11) Rel(vi)=vi⋅‖vi‖‖.Rel(v_i)= v_i·c\|v_i\|\,\|c\|. This formulation requires only a single inner product per visual token and introduces no dependency on attention scores or LLM intermediate states. Table 2. Generalization results on LLaVA-NeXT. Method #Tokens GQA MMBench POPE MME ScienceQA SEEDBench Avg. LLaVA-NeXT (Full) 2880 64.2 67.9 86.4 1842 70.2 70.2 100% VisionZip (CVPR’25) (Yang et al., 2025) 640 61.3 66.3 86.3 1787 68.1 66.7 97.0% PruneSID (ICLR’26) (Fang et al., 2026) 640 61.6 64.2 86.3 1795 68.3 67.3 96.8% SAVER (Ours) 640 63.6 65.8 86.4 1802 69.8 68.5 98.5% VisionZip (CVPR’25) (Yang et al., 2025) 320 59.3 63.1 82.1 1702 67.3 63.4 93.2% PruneSID (ICLR’26) (Fang et al., 2026) 320 60.5 63.0 83.1 1754 67.3 65.0 94.5% SAVER (Ours) 320 62.5 65.0 85.0 1706 69.4 66.2 96.2% Table 3. Generalization results on Mini-Gemini. Method #Tokens GQA MMBench POPE MME ScienceQA SEEDBench Avg. Mini-Gemini (Full) 576 62.4 69.3 85.8 1841 70.7 69.7 100% VisionZip (CVPR’25) (Yang et al., 2025) 128 58.7 68.1 78.5 1841 70.0 65.6 96.2% PruneSID (ICLR’26) (Fang et al., 2026) 128 60.1 66.6 82.4 1821 70.7 66.5 97.1% SAVER (Ours) 128 60.6 67.2 84.1 1831 71.0 67.2 98.1% VisionZip (CVPR’25) (Yang et al., 2025) 64 55.8 65.9 69.6 1737 70.7 61.7 91.4% PruneSID (ICLR’26) (Fang et al., 2026) 64 58.3 63.1 76.0 1735 70.6 63.6 93.1% SAVER (Ours) 64 58.7 64.8 81.0 1742 70.0 64.4 94.7% 4.4. Visual Diversity via Visual Marginal Coverage Gain Visual diversity in token selection has been approached in two main ways, each with a distinct failure mode. Max-min diversity methods (Alvar et al., 2025) prioritize tokens that are maximally distant from the current selection, which makes them sensitive to outliers: a single token lying in a visually uninformative region can dominate the selection simply by being far from others. Alternatively, coverage-based methods favor tokens that best represent dense groups of similar visual patches, which tends to over-allocate budget to large homogeneous regions (e.g., background) and under-represent compact foreground structures rich in semantic detail. Inspired by the facility location problem in combinatorial optimization, we propose a selection criterion that avoids both failure modes. In the facility location analogy, each unselected token is a “client” that must be served by the nearest selected token (“facility”). Adding a new token is only beneficial if it meaningfully improves coverage for clients that are currently underserved. This naturally penalizes both redundant selections and outliers. Formally, let ∈ℝN×NK ^N× N denote the cosine similarity matrix of the visual tokens, where each entry is defined as: (12) Kij=vi⋅vj‖vi‖‖vj‖.K_ij= v_i· v_j\|v_i\|\,\|v_j\|. Given the already-selected set S, the Visual Marginal Coverage Gain (VMCG) of a candidate token viv_i is: (13) Div(vi∣S) (v_i S) =γ⋅max(0,Kii−maxs∈SKsi) =γ· \! (0,\,K_i- _s∈ SK_si ) +(1−γ)⋅∑vj∉S,j≠imax(0,Kij−maxs∈SKsj). +(1-γ)·\!\! _v_j∉ S,\,j≠ i\! \! (0,\,K_ij- _s∈ SK_sj ). Since Kii=1K_i=1 for cosine-normalized features, and dropping the additive constant γ, this reduces to: (14) Div(vi∣S)=−γmaxs∈SKsi⏟novelty+(1−γ)∑vj∉S,j≠imax(0,Kij−maxs∈SKsj)⏟representativeness,Div(v_i S)= -γ _s∈ SK_si_novelty+ (1-γ)\!\! _v_j∉ S,\,j≠ i\! \! (0,\,K_ij- _s∈ SK_sj )_representativeness, where γ∈(0,1)γ∈(0,1) balances novelty against representativeness. The novelty term measures how dissimilar viv_i is from the closest token already in S. In the representativeness term, the max(0,⋅) (0,·) operator encodes a key principle from facility location: an unselected token vjv_j is only “served” by viv_i if viv_i is more similar to vjv_j than any token currently in S—if a better-matched facility already exists, vjv_j contributes nothing to the score of viv_i. Consequently, Div(vi∣S)Div(v_i S) is high only when viv_i is both novel with respect to the current selection and genuinely representative of a large portion of tokens that remain poorly covered, naturally suppressing both redundant and outlier selections. 4.5. Sample-Aware Token Selection Given Rel(vi)Rel(v_i) and Div(vi∣S)Div(v_i S) defined above, the unified selection score for each candidate token is: (15) Score(vi∣S)=(1−θ)⋅Rel(vi)+θ⋅Div(vi∣S),Score(v_i S)=(1-θ)·Rel(v_i)+θ·Div(v_i S), Let V=vii=1NV=\v_i\_i=1^N be the visual tokens of image I. For a statistic X, define X(x)=clip((x−Xmin)/(Xmax−Xmin))N_X(x)=clip((x-X_ )/(X_ -X_ )); hence ~=() G=N_G(G). Raw image diversity is defined as (16) (I)=1−1N(N−1)∑i≠jvi⊤vj∥vi∥2∥vj∥2.D(I)=1- 1N(N-1) _i≠ j v_i v_j v_i _2\, v_j _2. Whereas GIP captures prompt-conditioned spatial variation among relevant tokens, (I)D(I) measures intrinsic visual variation within the image. We normalize it as DIV(I)=((I))DIV(I)=N_D(D(I)) and use it only to scale the GIP mapping: (17) kGIP=k⋅DIV(I).k_GIP=k·DIV(I). The sample-specific weight is then (18) θ=sigmoid(kGIP⋅~+b).θ=sigmoid\! (k_GIP· G+b ). The bounds min,max,min,maxG_ ,G_ ,D_ ,D_ used by N_G and N_D are estimated from a small held-out calibration subset; clipping prevents out-of-range samples from producing abnormal weights. Here, k and b are global linear coefficients in the sigmoid pre-activation that scale and shift ~ G into an appropriate range before mapping to θ∈(0,1)θ∈(0,1). At each greedy step, the token with the highest score is appended to S, and the diversity scores are updated incrementally. Selection proceeds until |S|=K|S|=K; the full procedure is given in Algorithm 1. 5. Experiments 5.1. Experimental Setup Benchmarks. We evaluate on a diverse set of image-based multimodal benchmarks covering general visual understanding: GQA (Hudson and Manning, 2019), MMBench (Liu et al., 2024c), MME (Fu et al., 2023), POPE (Li et al., 2023c), ScienceQA (Lu et al., 2022), SEEDBen-ch (Li et al., 2023a) and TextVQA (Singh et al., 2019). We additionally include OCRBench (Liu et al., 2024d) to assess OCR and document understanding. All benchmarks are evaluated under their standard protocols. Baselines. We compare SAVER against representative token reduction methods spanning single-objective and multi-objective design strategies. Text-relevance-based methods include FastV (Chen et al., 2025) and PyramidDrop (Xing et al., 2024). Visual-diversity-based methods include DivPrune (Alvar et al., 2025), VisionZip (Yang et al., 2025), and PruneSID (Fang et al., 2026). Methods combining both objectives include SparseVLM (Zhang et al., 2025b) and DART (Wen et al., 2025). Model architectures. We validate SAVER primarily on LLaVA-1.5-7B (Liu et al., 2023) and further evaluate on LLaVA-NeXT-7B (Liu et al., 2024a) and Mini-Gemini (Li et al., 2026a) to assess generalizability. All models are evaluated following their original inference configurations, without any additional training. 5.2. Main Results LLaVA-1.5-7B. LLaVA-1.5-7B is a widely adopted open-source multimodal baseline that resizes input images to a fixed resolution and encodes them into a fixed set of 576 visual tokens via a CLIP vision encoder. We evaluate at three retention budgets K∈64,128,192K∈\64,128,192\. As shown in Table 1, SAVER ranks first under all three settings. At K=64K=64, SAVER achieves 96.5%, surpassing PruneSID and VisionZip by 2.7 and 3.6 points. At K=128K=128, SAVER attains 98.4%, maintaining a consistent lead over all baselines. At K=192K=192, SAVER retains 99.6% of full-model performance with negligible degradation. LLaVA-NeXT. LLaVA-NeXT encodes both the full image and local crops at high resolution, yielding up to 2880 visual tokens and substantially stronger fine-grained perception than single-scale models. We evaluate at K∈320,640K∈\320,640\ and report results in Table 2. SAVER achieves 98.5% at 640 tokens and 96.2% at 320 tokens, outperforming PruneSID (96.8%/94.5%) and VisionZip (97.0%/93.2%). Mini-Gemini. Mini-Gemini employs a dual-encoder architecture that simultaneously extracts high- and low-resolution visual features and fuses them into 576 tokens, enabling richer multi-scale representations. We evaluate at K∈64,128K∈\64,128\ and report results in Table 3. SAVER achieves 98.1% at 128 tokens and 94.7% at 64 tokens, consistently ahead of PruneSID (97.1%/93.1%) and VisionZip (96.2%/91.4%). 5.3. Efficiency Analysis We compare inference efficiency against SparseVLM (Zhang et al., 2025b) and DAR-T (Wen et al., 2025), both of which, like SAVER, apply token reduction during the LLM prefilling stage. We measure peak GPU memory, inference throughput, and POPE accuracy on LLaVA-1.5-7B at K=64K=64 retained tokens. As reported in Table 4, SAVER matches DART in both peak memory and throughput, while achieving a POPE score of 83.8 compared to DART’s 73.9. SparseVLM, despite consuming more GPU memory than the uncompressed model, attains only 75.1 on POPE. These results demonstrate that SAVER’s accuracy gains introduce no additional computational overhead. Table 4. Efficiency comparison on LLaVA-1.5-7B. Method #Tok. Peak GPU↓ Throughput↑ POPE↑ LLaVA-1.5-7B (Full) 576 14.69 11.21 85.9 SparseVLM (Zhang et al., 2025b) 64 18.84 12.09 75.1 DART (Wen et al., 2025) 64 14.33 17.17 73.9 SAVER (Ours) 64 14.33 17.11 83.8 Table 5. Ablation study on LLaVA-1.5-7B. Method POPE GQA MMBench Avg. WTC relevance 73.7 56.3 61.9 91.0% VMCG diversity 83.4 58.7 59.9 95.0% WTC + VMCG(static) 84.0 59.5 60.4 95.9% SAVER (Ours) 83.8 60.0 61.8 96.9% Figure 3. Ablation on γ across different token budgets on LLaVA-NeXT-7B (POPE F1). The effect is amplified under higher compression ratios. 5.4. Ablation Study Component ablation. We ablate the core design choices of SAVER on LLaVA-1.5-7B at K=64K=64 tokens retained, as shown in Table 5. Using relevance alone yields the lowest average (91.0%): without diversity, the selected tokens cluster around query-relevant regions and fail to cover the broader visual context, which is reflected in a notably low POPE score of 73.7. Diversity alone improves the average to 95.0% and raises POPE to 83.4 through wider spatial coverage, but the absence of text guidance leads to a drop in MMBench (59.9), where responses depend on precise localization of query-relevant details. The static combination improves the average to 95.9%, but remains below the sample-aware variant because a fixed relevance-diversity weight cannot accommodate the varying demands of different inputs. SAVER achieves the best average of 96.9% by dynamically inferring a sample-specific balance via GIP, confirming that adaptive, instance-level balancing is the key factor behind its consistent gains. Discussion on the Novelty–Representativeness Trade-off. The novelty-representativeness trade-off coefficient γ requires more care. As shown in Figure 3, overly small γ values under-emphasize novelty and bias selection toward redundant regions, while overly large γ values over-prioritize outlier tokens at the expense of global coverage. Moderate values yield the best performance in practice. The optimal γ may also vary with the token retention budget. 6. Conclusion We present a systematic empirical analysis of the relevance–diversity trade-off in visual token reduction, demonstrating that the optimal objective balance varies substantially across both tasks and individual inputs and cannot be captured by any fixed strategy. Motivated by this finding, we propose SAVER, a training-free framework centered on the Geometric Intent Prior (GIP), which infers a sample-specific relevance-diversity weight directly from the spatial distribution of high-relevance tokens. GIP is paired with a Weighted Text Centroid for attention-free relevance estimation and a Visual Marginal Coverage Gain for diversity-aware selection. Extensive experiments show that SAVER consistently outperforms competitive baselines across all compression budgets, while matching their inference efficiency, confirming that instance-level adaptive balancing is a principled approach to visual token reduction. Acknowledgements. This work was supported in part by National Natural Science Foundation of China: 62525212 and U23B2051, in part by Youth Innovation Promotion Association CAS, in part by the Strategic Priority Research Program of the Chinese Academy of Sciences, Grant No. XDB0680201, in part by the China National Postdoctoral Program for Innovative Talents under Grant BX20250377, in part by the China Postdoctoral Science Foundation under Grant 2026M791710, in part by the Beijing Major Science and Technology Project under Contract Z251100008125059, and in part by Beijing Academy of Artificial Intelligence (BAAI). References Achiam et al. (2023) J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §2.1. Alayrac et al. (2022) J. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems 35, p. 23716–23736. Cited by: §1. Alvar et al. (2025) S. R. Alvar, G. Singh, M. Akbari, and Y. Zhang DivPrune: diversity-based visual token pruning for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 9392–9401. Cited by: §1, §2.2, §4.4, Table 1, Table 1, Table 1, §5.1. Bai et al. (2023a) J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609. Cited by: §2.1. Bai et al. (2023b) J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou Qwen-vl: a versatile vision-language model for understanding, localization, text reading, and beyond. Cited by: §1, §2.1. Chen et al. (2025) L. Chen, H. Zhao, T. Liu, S. Bai, J. Lin, C. Zhou, and B. Chang An image is worth 1/2 tokens after layer 2: plug-and-play inference acceleration for large vision-language models. In Computer Vision – ECCV 2024, A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, and G. Varol (Eds.), Cham, p. 19–35. External Links: ISBN 978-3-031-73004-7 Cited by: §1, §2.2, §4.3, Table 1, Table 1, Table 1, §5.1. Chen et al. (2024) Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. Internvl: scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 24185–24198. Cited by: §1. Chiang et al. (2023) W. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, et al. Vicuna: an open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) 2 (3), p. 6. Cited by: §2.1. Dao et al. (2022) T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré FlashAttention: fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §4.3. Dong et al. (2026) S. Dong, J. Hu, M. Zhang, M. Yin, Y. Fu, and Q. Qian MMTok: multimodal coverage maximization for efficient inference of VLMs. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §1. Fang et al. (2026) Z. Fang, P. Lyu, C. Zhang, G. Lu, J. Yu, and W. Pei Prune redundancy, preserve essence: vision token compression in VLMs via synergistic importance-diversity. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §1, Table 1, Table 1, Table 1, Table 2, Table 2, Table 3, Table 3, §5.1. Fu et al. (2023) C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun, et al. Mme: a comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394. Cited by: §5.1. HIT-SCIR (2024) HIT-SCIR Chinese-mixtral-8x7b: an open-source mixture-of-experts llm. GitHub. Note: https://github.com/HIT-SCIR/Chinese-Mixtral-8x7B Cited by: §2.1. Hua et al. (2024) C. Hua, Q. Xu, S. Bao, Z. Yang, and Q. Huang ReconBoost: boosting can achieve modality reconcilement. In International Conference on Machine Learning, p. 19573–19597. Cited by: §1. Hudson and Manning (2019) D. A. Hudson and C. D. Manning Gqa: a new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 6700–6709. Cited by: §3, §5.1. Jeddi et al. (2025) A. Jeddi, N. Baghbanzadeh, E. Dolatabadi, and B. Taati Similarity-aware token pruning: your vlm but faster. arXiv preprint arXiv:2503.11549. Cited by: §1, §2.2. Jiang et al. (2025) P. Jiang, H. Li, L. Zhao, F. Chao, K. Yan, S. Ding, and R. Ji VISA: group-wise visual token selection and aggregation via graph summarization for efficient mllms inference. In Proceedings of the 33rd ACM International Conference on Multimedia, M ’25, New York, NY, USA, p. 11130–11139. External Links: ISBN 9798400720352, Link, Document Cited by: §1. Li et al. (2023a) B. Li, R. Wang, G. Wang, Y. Ge, Y. Ge, and Y. Shan Seed-bench: benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125. Cited by: §5.1. Li et al. (2023b) J. Li, D. Li, S. Savarese, and S. Hoi Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, p. 19730–19742. Cited by: §1. Li et al. (2025) Y. Li, H. Zhan, T. Chen, Q. Liu, and Y. Lu Why 1 + 1 < 1 in visual token pruning: beyond naïve integration via multi‑objective balanced covering. External Links: 2505.10118 Cited by: §1, §1, §2.2. Li et al. (2026a) Y. Li, Y. Zhang, C. Wang, Z. Zhong, Y. Chen, R. Chu, S. Liu, and J. Jia Mini-gemini: mining the potential of multi-modality vision language models. IEEE Transactions on Pattern Analysis and Machine Intelligence 48 (3), p. 3530–3543. External Links: Document Cited by: §1, §2.1, §5.1. Li et al. (2023c) Y. Li, Y. Du, K. Zhou, J. Wang, X. Zhao, and J. Wen Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 292–305. External Links: Link, Document Cited by: §3, §5.1. Li et al. (2026b) Z. Li, Y. Hu, Z. Chen, H. Wen, X. Song, and L. Nie COMBINER: composed image retrieval guided by attribute-based neighbor relations. IEEE TIP. Cited by: §1. Lin et al. (2025) Z. Lin, M. Lin, L. Lin, and R. Ji Boosting multimodal large language models with visual tokens withdrawal for rapid inference. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 5334–5342. Cited by: §1. Liu et al. (2024a) H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee LLaVA-next: improved reasoning, ocr, and world knowledge. External Links: Link Cited by: §1, §2.1, §5.1. Liu et al. (2023) H. Liu, C. Li, Q. Wu, and Y. J. Lee Visual instruction tuning. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 34892–34916. External Links: Link Cited by: §1, §2.1, §5.1. Liu et al. (2024b) T. Liu, L. Shi, R. Hong, Y. Hu, Q. Yin, and L. Zhang Multi-stage vision token dropping: towards efficient multimodal large language model. arXiv preprint arXiv:2411.10803. Cited by: §1. Liu et al. (2024c) Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, et al. Mmbench: is your multi-modal model an all-around player?. In European conference on computer vision, p. 216–233. Cited by: §3, §5.1. Liu et al. (2024d) Y. Liu, Z. Li, M. Huang, B. Yang, W. Yu, C. Li, X. Yin, C. Liu, L. Jin, and X. Bai OCRBench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences 67 (12). External Links: ISSN 1869-1919, Link, Document Cited by: §5.1. Lu et al. (2024) H. Lu, W. Liu, B. Zhang, B. Wang, K. Dong, B. Liu, J. Sun, T. Ren, Z. Li, H. Yang, Y. Sun, C. Deng, H. Xu, Z. Xie, and C. Ruan DeepSeek-vl: towards real-world vision-language understanding. External Links: 2403.05525 Cited by: §1. Lu et al. (2022) P. Lu, S. Mishra, T. Xia, L. Qiu, K. Chang, S. Zhu, O. Tafjord, P. Clark, and A. Kalyan Learn to explain: multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), Cited by: §5.1. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, M. Meila and T. Zhang (Eds.), Proceedings of Machine Learning Research, Vol. 139, p. 8748–8763. External Links: Link Cited by: §2.1. Shang et al. (2025) Y. Shang, M. Cai, B. Xu, Y. J. Lee, and Y. Yan LLaVA-prumerge: adaptive token reduction for efficient large multimodal models. In ICCV, Cited by: §1. Singh et al. (2019) A. Singh, V. Natarjan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach Towards vqa models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 8317–8326. Cited by: §5.1. Touvron et al. (2023) H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: §2.1. Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin Attention is all you need. Advances in neural information processing systems 30. Cited by: §1. Wang et al. (2024) P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, and W. Ge Qwen2-vl: enhancing vision-language model’s perception of the world at any resolution. Cited by: §1, §2.1. Wen et al. (2025) Z. Wen, Y. Gao, S. Wang, J. Zhang, Q. Zhang, W. Li, C. He, and L. Zhang Stop looking for “important tokens” in multimodal language models: duplication matters more. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 9961–9980. External Links: Link, Document, ISBN 979-8-89176-332-6 Cited by: §1, §2.2, §4.3, Table 1, Table 1, Table 1, §5.1, §5.3, Table 4. Xing et al. (2024) L. Xing, Q. Huang, X. Dong, J. Lu, P. Zhang, Y. Zang, Y. Cao, C. He, J. Wang, F. Wu, et al. Pyramiddrop: accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.17247. Cited by: §1, §2.2, Table 1, Table 1, Table 1, §5.1. Yang et al. (2025) S. Yang, Y. Chen, Z. Tian, C. Wang, J. Li, B. Yu, and J. Jia VisionZip: longer is better but not necessary in vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 19792–19802. Cited by: Table 1, Table 1, Table 1, Table 2, Table 2, Table 3, Table 3, §5.1. Zhang et al. (2025a) Q. Zhang, M. Liu, L. Li, M. Lu, Y. Zhang, J. Pan, Q. She, and S. Zhang Beyond attention or similarity: maximizing conditional diversity for token pruning in mllms. arXiv preprint arXiv:2506.10967. Cited by: §1. Zhang et al. (2025b) Y. Zhang, C. Fan, J. Ma, W. Zheng, T. Huang, K. Cheng, D. A. Gudovskiy, T. Okuno, Y. Nakata, K. Keutzer, and S. Zhang SparseVLM: visual token sparsification for efficient vision-language model inference. In Proceedings of the 42nd International Conference on Machine Learning, A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu (Eds.), Proceedings of Machine Learning Research, Vol. 267, p. 74840–74857. External Links: Link Cited by: §1, §2.2, §4.3, Table 1, Table 1, Table 1, §5.1, §5.3, Table 4. Zou et al. (2025) X. Zou, D. Lu, Y. Wang, Y. Yan, Y. Lyu, X. Zheng, L. Zhang, and X. Hu Don’t just chase" highlighted tokens" in mllms: revisiting visual holistic context retention. arXiv preprint arXiv:2510.02912. Cited by: §1.