Paper deep dive
TReVS: Integrating Textual Relevance and Visual Saliency for Efficient Vision-Language Model Token Pruning
Jing Wang, Zhiping Wu, Dongdong Ren, Youfang Han, Wei Zhao, Wenbin Li
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Vision-Language Models (VLMs) excel at visual understanding and reasoning but often incur substantial inference costs due to the large number of visual tokens. Recent visual token pruning methods increasingly follow a two-stage paradigm: they first remove visually redundant tokens after the vision encoder and then discard tokens irrelevant to the textual query within the Large Language Model (LLM). However, since the first stage typically relies solely on vision-encoder saliency, it may prematurely eliminate query-relevant tokens, depriving the subsequent text-guided stage of critical visual evidence. Our empirical analysis shows that incorporating query guidance into first-stage pruning better preserves task-relevant evidence and consistently improves performance over vision-only saliency-based pruning. We further find that high-variance attention heads are more sensitive to the textual query and yield more discriminative text-to-vision attention signals for second-stage pruning. Motivated by these findings, we propose TReVS, a training-free framework that combines textual relevance with vision-encoder saliency for pre-LLM pruning and leverages high-variance attention heads to remove task-irrelevant tokens at shallow-to-intermediate layers of the LLM. On LLaVA-1.5-7B, TReVS retains 92.8% of the unpruned baseline performance while pruning 94.4% of visual tokens, outperforming prior state-of-the-art methods.
Tags
Links
- Source: https://arxiv.org/abs/2609.37581v1
- Canonical: https://arxiv.org/abs/2609.37581v1
Trouble viewing inline? Open PDF directly →
Full Text
41,529 characters extracted from source content.
Expand or collapse full text
TReVS: Integrating Textual Relevance and Visual Saliency for Efficient Vision-Language Model Token Pruning Jing Wang Zhiping Wu Dongdong Ren Youfang Han Wei Zhao Wenbin Li Abstract Vision-Language Models (VLMs) excel at visual understanding and reasoning but often incur substantial inference costs due to the large number of visual tokens. Recent visual token pruning methods increasingly follow a two-stage paradigm: they first remove visually redundant tokens after the vision encoder and then discard tokens irrelevant to the textual query within the Large Language Model (LLM). However, since the first stage typically relies solely on vision-encoder saliency, it may prematurely eliminate query-relevant tokens, depriving the subsequent text-guided stage of critical visual evidence. Our empirical analysis shows that incorporating query guidance into first-stage pruning better preserves task-relevant evidence and consistently improves performance over vision-only saliency-based pruning. We further find that high-variance attention heads are more sensitive to the textual query and yield more discriminative text-to-vision attention signals for second-stage pruning. Motivated by these findings, we propose TReVS, a training-free framework that combines textual relevance with vision-encoder saliency for pre-LLM pruning and leverages high-variance attention heads to remove task-irrelevant tokens at shallow-to-intermediate layers of the LLM. On LLaVA-1.5-7B, TReVS retains 92.8% of the unpruned baseline performance while pruning 94.4% of visual tokens, outperforming prior state-of-the-art methods. 1School of Intelligence Science and Technology, Nanjing University 2School of Electronic Science and Engineering, Nanjing University 3Geely Automobile Research Institute (Ningbo) Co., Ltd., 315000 4Alpha Labs, Goertek Jing Wang: 221900040@smail.nju.edu.cn; Zhiping Wu: zhipingwu@smail.nju.edu.cn; Dongdong Ren: Dongdong.Ren2@geely.com; Youfang Han: fred.hanyf@goertek.com; Wei Zhao: charles.zhaow@goertek.com; Wenbin Li: liwenbin@nju.edu.cn 1 Introduction Figure 1: (a–c) A two-stage baseline guided only by [CLS] attention during pre-LLM pruning discards visual tokens representing the queried grass, leaving incomplete evidence for subsequent query-guided pruning. TReVS uses textual relevance in pre-LLM pruning, preserving this evidence for the in-LLM stage. (d) TReVS achieves the best performance across six image-understanding benchmarks. Vision-Language Models (VLMs) extend the reasoning of pretrained Large Language Models (LLMs) (Touvron et al. 2023; Bai et al. 2023; Achiam et al. 2023) to visual inputs, achieving strong performance in visual question answering, multimodal reasoning, high-resolution image understanding, and video understanding (Liu et al. 2023; Zhu et al. 2023; Li et al. 2023a; Li et al. 2024). Recent VLMs improve fine-grained perception across high-resolution images, multiple images, and videos. This broader coverage, however, produces visual-token sequences substantially longer than text sequences. LLaVA-1.5 encodes each image into 576 visual tokens (Liu et al. 2024a), LLaVA-NeXT uses up to 2,880 tokens per high-resolution image (Liu et al. 2024b), and Qwen2.5-VL processes up to 16,384 visual tokens for multi-image and video inputs (Bai et al. 2025). These sequences increase computation, latency, and memory use, making visual-token processing a major bottleneck in efficient VLM inference. Visual token pruning addresses this bottleneck by exploiting the substantial redundancy in visual inputs (Bolya et al. 2023; Chen et al. 2024; Zhang et al. 2025b; Yang et al. 2025). Recent methods increasingly adopt a two-stage design that reduces visual redundancy after the vision encoder and then removes textually irrelevant tokens within the LLM (Takezoe et al. 2026; Zhang et al. 2026; Singh et al. 2026). The pre-LLM stage typically uses vision-encoder saliency or diversity to select tokens. The in-LLM stage uses text-to-vision cross-attention to retain query-relevant tokens. Although this division improves the performance, it creates an information bottleneck. Current pre-LLM pruning remains largely query-agnostic and can discard visual evidence that is essential to the text query. Once removed, such evidence is unavailable to subsequent query-aware pruning. The second stage must therefore identify task-relevant tokens within a potentially incomplete visual context. Figure 2: Effect of textual guidance on pre-LLM pruning. (a) Vision-encoder saliency focuses on the dominant fire hydrant, whereas textual relevance identifies the queried car. Their integration preserves both. (b) Performance relative to dense inference under different textual ratios. (c) Mean Spearman correlation between the two signals within the top-K saliency-selected tokens. We first examine the information loss caused by query-agnostic pre-LLM pruning and obtain two findings. First, incorporating textual relevance into pre-LLM pruning consistently improves performance by preserving query-relevant evidence (Figure 2(b)). Second, vision-encoder saliency and textual relevance produce distinct token rankings, with negative Spearman correlations across datasets and token budgets (Figure 2(c)). Together, these results show that textual relevance complements vision-encoder saliency and should be introduced before irreversible token reduction. We then examine the in-LLM stage, where the retained visual tokens are further refined using text-to-vision attention. We find that high-variance attention heads produce more discriminative and query-sensitive attention signals for identifying task-relevant tokens. These findings motivate a coordinated two-stage design: the first stage preserves visually salient, textually relevant, and diverse evidence, while the second uses high-variance heads to remove residual task-irrelevant tokens after cross-modal interaction develops. Section 3 provides detailed evidence for these findings. As illustrated in Figure 1, we propose TReVS, a training-free framework that coordinates pre-LLM and in-LLM visual token pruning. Before the LLM, TReVS scores visual tokens using [CLS] attention from the penultimate ViT layer and cosine similarity between projected visual tokens and embedded text tokens. It fuses these scores and supplements the top-ranked tokens with a small diversity set. During LLM prefilling, TReVS prunes again at a shallow-to-middle layer, using high-variance text-to-vision attention heads to identify query-relevant tokens. The first stage reduces visual redundancy while preserving salient, textually relevant, and diverse evidence. The second removes the remaining task-irrelevant tokens after cross-modal interaction develops. Extensive experiments across vision-language benchmarks show that TReVS outperforms state-of-the-art training-free methods at multiple reduction ratios. On LLaVA-1.5-7B, TReVS prunes 94.4% of visual tokens while retaining 92.8% of the original performance. Without additional training, it accelerates prefilling by 2.2×2.2× and end-to-end inference by 1.4×1.4×. These results establish the importance of preserving textual relevance before visual token reduction. Our main contributions are summarized as follows: • We show that introducing textual relevance before the first irreversible visual-token reduction preserves query-relevant evidence and consistently improves performance. The rankings induced by textual relevance and vision-encoder saliency remain negatively correlated across datasets and token budgets, showing that the two signals favor complementary visual tokens. • We show that attention variance provides a simple, training-free criterion for identifying query-sensitive attention heads. These heads produce more discriminative text-to-vision attention signals for in-LLM pruning. • We introduce TReVS, a training-free two-stage framework that combines vision-encoder saliency, textual relevance, and token diversity before the LLM, then uses high-variance attention heads to remove the remaining task-irrelevant visual tokens within the LLM. TReVS retains 92.8% of the baseline performance while pruning 94.4% of visual tokens. Figure 3: Head sensitivity and positional attention patterns. (a) Mean textual sensitivity of cumulative head subsets ranked by text-to-vision attention variance. (b) Average text-to-vision attention across visual-token positions at selected LLM layers. 2 Related Work 2.1 Vision-Language Models Vision-Language Models (VLMs) connect pretrained LLMs with vision encoders through projection modules such as MLPs and Q-Formers (Liu et al. 2023; Zhu et al. 2023; Li et al. 2023a). Recent VLMs extend visual understanding from standard images to high-resolution images and long videos (Liu et al. 2024a; Liu et al. 2024b; Bai et al. 2025; Clark et al. 2026; Liu et al. 2026). As image resolution and video duration increase, the resulting visual sequences can substantially exceed the corresponding text sequences, making visual token reduction important for efficient VLM inference. 2.2 Visual Token Reduction for VLMs Existing methods differ primarily in where token reduction occurs and when the text query becomes available. Vision-guided pre-LLM reduction. These methods remove redundant visual tokens after the vision encoder using visual-side signals such as saliency, similarity, or diversity (Alvar et al. 2025). VisionZip (Yang et al. 2025) retains dominant tokens according to vision-encoder saliency and merges residual tokens into contextual representations. VisPruner (Zhang et al. 2025a) combines vision-encoder attention with similarity-based deduplication. Although these methods reduce computation before the LLM, their selection remains independent of the text query. Query-aware in-LLM pruning. These methods prune within the LLM using text-to-vision cross-attention to retain task-relevant visual information. FastV (Chen et al. 2024) performs attention-based pruning in shallow LLM layers, PyramidDrop (Xing et al. 2024) progressively reduces tokens as visual redundancy increases with depth, and SparseVLM (Zhang et al. 2025b) combines text-guided attention with rank-adaptive sparsity. Since all visual tokens first enter the LLM, pruning often occurs in very shallow layers to provide meaningful computational savings. At these layers, text-to-vision cross-attention is susceptible to attention shift and dispersion (Zhang et al. 2025a), which can cause substantial accuracy loss. Two-stage pruning. Recent methods combine pre-LLM visual reduction with query-guided refinement inside the LLM (Zhang et al. 2026; Singh et al. 2026; Takezoe et al. 2026). DUET-VLM (Singh et al. 2026) first merges redundant tokens through local clustering and then performs layer-wise pruning using cross-modal attention. LearnPruner (Takezoe et al. 2026) replaces vision-encoder attention with a learnable pre-LLM importance predictor and performs text-guided pruning in intermediate LLM layers. This design reduces early computation while allowing the second stage to use more mature cross-modal interactions. Existing two-stage methods typically separate visual-only pre-LLM reduction from query-aware in-LLM pruning. In contrast, TReVS coordinates query guidance across both stages: textual relevance complements vision-encoder saliency before the LLM, while query-sensitive attention further refines the retained tokens within the LLM. This design reduces visual redundancy without depriving the second stage of query-relevant context. 3 Preliminary Analysis 3.1 Does textual guidance improve performance? Two-stage methods typically use vision-encoder saliency or diversity signals for pre-LLM pruning and query-aware attention for in-LLM pruning. This mismatch creates an irreversible bottleneck: query-relevant evidence removed before the LLM cannot be recovered later. We therefore ask: Can textual guidance during pre-LLM pruning preserve such evidence and improve performance? We evaluate LLaVA-1.5-7B on MME (Fu et al. 2023), POPE (Li et al. 2023b), TextVQA (Singh et al. 2019), and GQA (Hudson and Manning 2019). Under a fixed token budget, we vary the allocation between vision-encoder saliency, measured by [CLS] attention in the penultimate ViT layer, and textual relevance, measured by cosine similarity between projected visual tokens and text embeddings. Figure 2(a) illustrates their complementary behavior. Vision-encoder saliency favors the dominant fire hydrant, while textual relevance recovers the car targeted by the query. Their integration retains both sources of evidence. As shown in Figure 2(b), allocating even a modest fraction of the budget to textual relevance consistently outperforms visual-only pruning. Figure 2(c) further shows negative Spearman correlations between the two scores among the top-K saliency-selected tokens across all datasets and budgets. Thus, textual relevance reorders even visually salient candidates and complements, rather than replaces, vision-encoder saliency. Together, these results show that textual guidance contributes complementary selection information and improves performance under a fixed token budget. 3.2 Are High-Variance Attention Heads More Query-Sensitive? Several in-LLM pruning methods average text-to-vision attention across all heads (Chen et al. 2024; Takezoe et al. 2026), although heads differ in how strongly they respond to the query. We therefore ask: Can query-sensitive heads be identified with a training-free criterion? A concentrated attention distribution has high variance across visual tokens, but concentration alone does not imply query sensitivity. We therefore test whether attention variance ranks heads whose visual-token attention changes more with the question. We pair 1,000 TextVQA (Singh et al. 2019) images with two distinct questions and process them using LLaVA-1.5-7B. At each layer, we rank the 32 heads by the variance of their text-to-vision attention and evaluate cumulative top-4, top-8, top-12, and all-head subsets. Let h(1)s_h^(1) and h(2)s_h^(2) be the visual-token attention distributions of head h for the two questions. We define its textual sensitivity as TSh=1−cos(h(1),h(2)).TS_h=1- \! (s_h^(1),s_h^(2) ). (1) Figure 3(a) shows that the highest-variance subsets are generally more text-sensitive, with the clearest separation in the middle layers. Since the subsets are cumulative, the decrease from top-4 to all 32 heads shows that adding lower-ranked heads reduces the mean per-head sensitivity. Attention variance therefore provides a simple criterion for selecting query-sensitive heads. Figure 3(b) compares average text-to-vision attention across visual-token positions at different layers. In the earliest layers, attention generally increases with token position, favoring visual tokens closer to the subsequent text sequence. Although all layers exhibit peaks near the sequence boundaries, later layers show flatter profiles across the interior positions, suggesting weaker positional dependence. This observation motivates delaying attention-based pruning beyond the earliest layers. Together, these analyses motivate two design choices: using high-variance heads for query-sensitive token scoring and pruning at a shallow-to-middle layer to avoid strong early-layer position dependence. Figure 4: The architecture of TReVS. Stage 1 performs query-aware pre-LLM pruning by preserving visually salient, query-relevant, and diverse tokens. Stage 2 uses high-variance attention heads to guide in-LLM visual token pruning. 4 Method In this section, we present TReVS, a training-free framework that coordinates pre-LLM and in-LLM visual token pruning (Figure 4). Stage 1 retains K1K_1 visually salient, textually relevant, and diverse tokens before the LLM. Stage 2 uses high-variance attention heads to retain K2K_2 query-relevant tokens at a shallow-to-middle LLM layer. The following subsections detail the two stages. 4.1 Query-Aware Visual Redundancy Reduction Visual saliency score. To preserve visually salient information, we follow prior works (Yang et al. 2025) and leverage [CLS]-attention from the penultimate ViT layer as the vision-encoder saliency signal. Let x[CLS]∈ℝdx_[CLS] ^d denote the [CLS] token, Xv∈ℝN×dX_v ^N× d the N patch token features output by the ViT, and WQh,WKh∈ℝd×dhW_Q^h,W_K^h ^d× d_h the query and key projection matrices for head h, where dh=d/Hd_h=d/H is the per-head dimension. The [CLS]-to-patch attention for head h is: q[CLS]=x[CLS]WQh,Kv=XvWKh, q_[CLS]=x_[CLS]W_Q^h, K_v=X_vW_K^h, (2) A[CLS]h=Softmax(q[CLS]Kv⊤dh). A_[CLS]^h=Softmax\! ( q_[CLS]K_v d_h ). The visual saliency score for the i-th patch token is obtained by averaging the [CLS]-attention across all H heads: siv=1H∑h=1HA[CLS]h[i].s_i^v= 1H _h=1^HA_[CLS]^h[i]. (3) Textual-relevance score. As demonstrated in Section 3.1, incorporating a text-guidance signal at the pre-LLM stage is critical for retaining the text-relevant visual tokens that the LLM stage requires. We first project the ViT output through the visual projector to obtain text-aligned representations: Z=Projector(Xv)∈ℝN×d′Z=Projector(X_v) ^N× d , where zi∈ℝd′z_i ^d is the projected feature of visual token i. Let tj∈ℝd′t_j ^d denote the embedding of the j-th text token, with M tokens in total. We normalize the visual and text features as z¯i=zi/∥zi∥2 z_i=z_i/ z_i _2 and t¯j=tj/∥tj∥2 t_j=t_j/ t_j _2, respectively. We then compute the rectified cosine similarity between each text–visual pair: mj,i=max(t¯j⊤z¯i,0).m_j,i= \! ( t_j z_i,0 ). (4) The query-relevance score for visual token i is the Root Mean Square (RMS) aggregation over all text tokens: sit=1M∑j=1Mmj,i2.s_i^t= 1M _j=1^Mm_j,i^2. (5) This aggregation suppresses background noise from weakly related text tokens while amplifying tokens that are broadly attended to by the query. Normalization and temperature scaling. Since sivs_i^v and sits_i^t are drawn from distributions with different scales and statistics, we independently apply robust normalization to each before fusion. We use the Median Absolute Deviation (MAD) as a spread estimator, which is resistant to the attention outliers commonly observed in [CLS] Attention: s^ic=[sic−Med(c)MAD(c)+ϵ]+/τc,c∈v,t. s_i^c= [ s_i^c-Med(s^c)MAD(s^c)+ε ]_\!+ /\, _c, c∈\v,\,t\. (6) Here, [x]+=max(x,0)[x]_+= (x,0) clamps negative values to zero. Med(c)=median(sic)Med(s^c)= imedian(s_i^c) denotes the median score across all visual tokens, while MAD(c)=median(|sic−Med(c)|)MAD(s^c)= imedian ( |s_i^c-Med(s^c) | ) measures the median absolute deviation from this center. The constant ϵε ensures numerical stability, and τc _c controls the temperature scaling, with τv=1.4 _v=1.4 and τt=1.0 _t=1.0 by default. Pivot token selection. Visually salient tokens do not necessarily exhibit high textual relevance, suggesting that the two signals capture complementary evidence. We therefore define a unified score that preserves tokens favored by either signal and adds a reward when both scores are high: si=max(s^iv,s^it)+λs^iv⋅s^it,s_i= \! ( s_i^v,\, s_i^t )+λ s_i^v· s_i^t, (7) where λs^ivs^itλ s_i^v s_i^t is the consistency reward, with λ=1.0λ=1.0 by default. We retain the top-KrK_r tokens as pivot tokens r=TopK(si,Kr)S_r=TopK(s_i,\ K_r). Context preservation. To mitigate the loss of background information, we sample KdK_d tokens from the remaining candidates =1,2,…,N∖rC=\1,2,…,N\ _r using Farthest Point Sampling (FPS). FPS iteratively selects the token farthest from the selected tokens in cosine distance, maximizing feature diversity. The resulting set is d=FPS(zi∣i∈,Kd)S_d=FPS(\z_i i \,K_d). The union 1=r∪dS_1=S_r _d of K1=Kr+KdK_1=K_r+K_d tokens is passed into the LLM for subsequent processing. 4.2 Query-Driven Visual Token Compression After Stage 1 pruning, K1K_1 visual tokens are concatenated with the text tokens and fed into the LLM. As the forward pass progresses, the text tokens gradually absorb task-relevant visual information through cross-modal attention, causing the visual tokens to become increasingly redundant (Kaduri, Bagon, and Dekel 2025). To further reduce computational overhead, we perform a second pruning step at layer l∗l^*, retaining K2K_2 visual tokens for subsequent layers. High-variance head selection. As shown in Section 3.2, heads with higher text-to-vision attention variance are more sensitive to changes in the text query. We therefore use attention variance as a training-free criterion for identifying query-sensitive heads at the second-stage pruning layer l∗l^*. Let AT→Vh∈ℝLtext×K1A_T→ V^h ^L_text× K_1 denote the text-to-vision attention matrix of head h at this layer, where q indexes text tokens and i indexes visual tokens. The attention variance of head h is: uh=1Ltext∑q=1LtextVari(AT→Vh[q,i]).u_h= 1L_text _q=1^L_textVar_i\! (A_T→ V^h[q,\,i] ). (8) Here, VariVar_i computes the variance across the K1K_1 visual-token positions for each text token. Averaging over all text tokens yields uhu_h, which measures the overall visual-token selectivity of head h. We select the top half of the heads ranked by uhu_h to retain query-sensitive attention signals while maintaining sufficient head coverage for stable visual-token scoring. The selected heads form the high-variance head set ℋ∗H^* used in the subsequent token selection. Query-irrelevant token reduction. Using ℋ∗H^*, we score each visual token by its maximum text-to-vision attention across all text positions, averaged over the selected heads: pi=maxq1|ℋ∗|∑h∈ℋ∗AT→Vh[q,i].p_i= _q\; 1|H^*| _h ^*A_T→ V^h[q,\,i]. (9) We retain the top-K2K_2 tokens by this score 2=TopK(pi,K2)S_2=TopK(p_i,\ K_2). The resulting K2K_2 visual tokens are then forwarded through the remaining LLM layers. Overall, the two stages form a progressive query-aware pruning process. By introducing textual relevance into pre-LLM pruning, TReVS preserves query-relevant visual evidence before irreversible reduction, ensuring that the subsequent in-LLM stage operates on an informative visual context. Within the LLM, high-variance heads provide more discriminative text-to-vision attention, enabling the in-LLM stage to better retain task-relevant visual tokens. 5 Experiments We evaluate TReVS across image, high-resolution, and video understanding to examine whether its query-aware two-stage design remains effective across different visual-token lengths. We then analyze its inference efficiency. 5.1 Experimental Setup Models and benchmarks. We evaluate TReVS on LLaVA-1.5-7B (Liu et al. 2024a), LLaVA-NeXT-7B (Liu et al. 2024b), and Video-LLaVA-7B (Lin et al. 2023), which process 576, 2,880 and 2,048 visual tokens, respectively. For LLaVA-1.5-7B, we use GQA (Hudson and Manning 2019), ScienceQA-IMG (Lu et al. 2022), TextVQA (Singh et al. 2019), POPE (Li et al. 2023b), MME (Fu et al. 2023), and the English and Chinese splits of MMBench (Liu et al. 2024c). We evaluate LLaVA-NeXT-7B on GQA, TextVQA, MME, and MMBench, and Video-LLaVA-7B on TGIF-QA (Jang et al. 2017), MSVD-QA, and MSRVTT-QA (Xu et al. 2017). Implementation details. TReVS is training-free and leaves all parameters of the underlying VLM unchanged. Unless otherwise specified, we perform the second pruning stage after the eighth LLM layer and retain the top half of the attention heads according to their text-to-vision attention variance. We set the ratio between pivot and diversity tokens to Kr:Kd=3:1K_r:K_d=3:1 and maintain K1:K2=3:1K_1:K_2=3:1 between the two pruning stages. The visual and textual temperatures are set to τv=1.4 _v=1.4 and τt=1.0 _t=1.0, respectively. The consistency-reward weight is set to λ=1.0λ=1.0. Method GQA SQAI VQAT POPE MME MMB MMBCN RelAcc. Upper Bound, 576 Tokens (100%) Vanilla 61.9 69.5 58.2 85.9 1862 64.7 58.3 100.0% Retain Averaged 128 Tokens (↓ 77.8%) FastV (ECCV 2024) 49.6 60.2 50.6 59.6 1490 56.1 51.4 82.6% SparseVLM (ICML 2025) 56.0 67.1 54.9 80.5 1696 60.0 51.1 92.4% DivPrune (CVPR 2025) 59.3 69.0 56.1 86.7 1718 62.0 54.8 96.4% VisionZip (CVPR 2025) 57.6 68.9 56.8 83.2 1762 62.0 56.7 96.3% VScan (TMLR 2026) 59.8 68.9 57.3 86.1 1792 63.0 58.0 98.2% DUET-VLM (CVPR 2026) 59.0 70.2 57.8 85.9 1767 63.3 56.7 97.9% TReVS 60.3 68.9 57.8 86.5 1842 63.6 57.2 98.8% Retain Averaged 64 Tokens (↓ 88.9%) FastV (ECCV 2024) 46.1 51.1 47.8 48.0 1256 48.0 42.7 71.6% SparseVLM (ICML 2025) 52.7 62.2 51.8 75.1 1505 56.2 46.1 85.4% DivPrune (CVPR 2025) 57.8 68.2 54.7 85.6 1674 59.3 52.3 93.8% VisionZip (CVPR 2025) 55.1 69.0 55.5 77.0 1690 60.1 55.4 93.1% VScan (TMLR 2026) 58.3 69.1 55.6 85.0 1698 62.1 55.7 95.8% DUET-VLM (CVPR 2026) 56.7 68.3 56.4 82.5 1751 62.0 56.3 95.6% TReVS 57.8 68.9 56.6 85.2 1769 61.9 56.0 96.5% Retain Averaged 32 Tokens (↓ 94.4%) FastV (ECCV 2024) 41.5 42.6 42.5 32.5 1090 37.8 33.2 59.0% SparseVLM (ICML 2025) 48.3 57.3 46.1 67.9 1290 51.4 40.6 76.7% DivPrune (CVPR 2025) 54.9 68.6 52.9 81.5 1611 57.6 49.1 90.4% VisionZip (CVPR 2025) 51.8 68.8 53.1 68.7 1536 57.7 50.3 87.4% VScan (TMLR 2026) 54.8 69.4 53.9 79.9 1598 59.5 51.9 91.5% DUET-VLM (CVPR 2026) 53.6 69.1 54.7 74.8 1633 60.0 54.8 91.6% TReVS 55.0 69.0 53.9 82.7 1651 61.4 51.7 92.8% Table 1: Performance comparison on LLaVA-1.5-7B under matched average token budgets. RelAcc. averages benchmark-wise performance relative to the unpruned model. Bold and underlined entries denote the best and second-best results. 5.2 Main Results Tables 1, 2, and 3 compare TReVS with existing visual token pruning methods across image and video VLMs. Results on LLaVA-1.5-7B. As shown in Table 1, TReVS achieves the highest RelAcc. at all three budgets, retaining 98.8%, 96.5%, and 92.8% of the unpruned performance. Notably, TReVS consistently obtains the best MME scores across these budgets, which indicates stable preservation of visual evidence required for diverse perception and cognition tasks. Moreover, its margin over the second-best RelAcc. increases from 0.6% at 128 tokens to 1.2% at 32 tokens. This widening margin is consistent with the two-stage design of TReVS. Before the LLM, textual relevance prioritizes query-relevant tokens, reducing critical information loss when only a small number of tokens can be retained. Within the LLM, query-sensitive heads provide more discriminative attention for selecting task-relevant tokens, making more effective use of the limited remaining token budget. Hallucination robustness. Figure 5 compares POPE performance across token budgets. While the methods perform similarly at moderate budgets, TReVS achieves 82.7% accuracy with 32 tokens, outperforming DivPrune and VScan by 1.2% and 2.8%. This robustness is consistent with our query-aware pre-LLM design, which preserves evidence for queried objects before irreversible pruning and thereby mitigates hallucinations caused by missing visual evidence. Figure 5: POPE accuracy on LLaVA-1.5-7B across retained-token budgets. While all methods perform similarly at moderate budgets, TReVS degrades more slowly under increasingly aggressive compression and achieves the best performance at 32 tokens. The dashed line denotes the unpruned model. Results on LLaVA-NeXT-7B. Table 2 evaluates TReVS on the longer visual sequences produced by high-resolution inputs. TReVS achieves the highest RelAcc. at both budgets, retaining 96.6% and 93.0% of the unpruned performance with 320 and 160 tokens, respectively. The best MME scores at both budgets and the best MMB score at 160 tokens further show that TReVS preserves key visual evidence under substantial compression. Despite these overall gains, TReVS trails the best GQA results by 0.5% and 0.6% under high-resolution inputs, whereas the gap is less pronounced with lower-resolution inputs. GQA requires global reasoning over multiple objects and their relations, and such evidence may be distributed across longer visual sequences. We hypothesize that high-variance heads prioritize sparse query-relevant details and therefore retain slightly less global context, making this trade-off more evident at higher resolutions. Method GQA VQAT MME MMB RelAcc. Upper Bound, 2,880 Tokens (100%) Vanilla 64.2 61.3 1842 67.9 100.0% Retain Averaged 320 Tokens (↓ 88.9%) SparseVLM 57.7 55.9 1694 64.3 91.9% DivPrune 61.1 56.2 1724 63.9 93.6% VisionZip 59.3 58.9 1702 63.1 93.4% DUET-VLM 60.6 59.9 1788 64.3 96.0% VScan 61.4 59.4 1775 65.5 96.3% TReVS 60.9 59.3 1826 64.9 96.6% Retain Averaged 160 Tokens (↓ 94.4%) SparseVLM 51.2 46.4 1542 63.1 83.0% DivPrune 59.3 54.1 1643 62.9 90.6% VisionZip 55.5 56.2 1630 60.1 88.8% DUET-VLM 58.6 57.2 1686 62.9 92.2% VScan 59.6 57.7 1699 62.0 92.6% TReVS 59.0 56.9 1717 63.9 93.0% Table 2: Performance comparison on LLaVA-NeXT-7B under matched average token budgets. Results on Video-LLaVA-7B. Table 3 evaluates whether TReVS generalizes from images to longer video inputs. With 93.4% of visual tokens removed, TReVS achieves the best result on all three benchmarks and retains 99% of the average unpruned performance, exceeding DUET-VLM by 3.2%. These results demonstrate that query-aware two-stage pruning remains effective for highly redundant video sequences. Method TGIF MSVD MSRVTT RelAcc. Upper Bound, 2,048 Tokens (100%) Video-LLaVA 48.7 70.1 57.4 100.0% Retain Averaged 136 Tokens (↓ 93.4%) FastV 30.4 44.3 39.4 64.8% SparseVLM 44.7 68.2 31.0 81.0% VisionZip 42.4 63.5 52.1 89.5% DUET-VLM 46.7 68.0 54.2 95.8% TReVS 48.9 69.2 56.2 99.0% Table 3: Performance comparison on Video-LLaVA-7B with 136 average retained tokens. 5.3 Ablation Study Table 4 isolates the contributions of the two pruning stages under the 32-token preset. Replacing [CLS]-attention-only pre-LLM pruning with the first stage of TReVS improves RelAcc. from 91.8% to 92.7% while retaining all attention heads in the second stage. This gain confirms that combining textual relevance with visual saliency and token diversity better preserves query-relevant and complementary evidence before irreversible token reduction. Selecting high-variance heads further increases RelAcc. to 93.2%, demonstrating that high-variance heads provide more discriminative query-aware signals than uniformly using all heads. Together, the two stages improve RelAcc. by 1.4%, confirming their complementary roles in preventing premature information loss and removing residual task-irrelevant tokens. Stage 1 Stage 2 RelAcc. (%) [CLS] Attn All Heads 91.8 TReVS All Heads 92.7 High-Variance Heads 93.2 Table 4: Ablation study of the two-stage token-pruning strategies on LLaVA-1.5-7B under the 32-token preset. RelAcc. denotes the average relative accuracy over TextVQA, MMBench, GQA, and POPE. 5.4 Efficiency Analysis Computational efficiency. Table 5 reports the inference efficiency of TReVS on POPE with LLaVA-1.5-7B, measured on an NVIDIA RTX 4090 GPU. At 32 tokens, TReVS achieves a 2.2×2.2× prefill speedup and a 1.4×1.4× end-to-end speedup while reducing KV-cache consumption by 6.5×6.5×. Together with the accuracy retained under the same budget in Table 1, these results demonstrate a favorable accuracy–efficiency trade-off under aggressive compression. Method Token Total Time Prefill Time KV Cache (MB) Vanilla 576 126.5 (1.0×) 72.7 (1.0×) 321.1 (1.0×) Ours 128 100.7 (1.3×) 40.6 (1.8×) 99.7 (3.2×) 64 90.4 (1.4×) 33.7 (2.2×) 66.2 (4.9×) 32 88.9 (1.4×) 33.5 (2.2×) 49.7 (6.5×) Table 5: Inference efficiency on POPE with LLaVA-1.5-7B. Latency is measured per sample in milliseconds, and parenthesized values report speedup or KV-cache reduction over dense inference. 6 Conclusion This work identifies an irreversible information bottleneck in two-stage visual token pruning: query-agnostic pre-LLM reduction can discard task-relevant evidence before query-aware reasoning begins. We introduce TReVS, a training-free framework that coordinates vision-encoder saliency, textual relevance, token diversity, and high-variance attention heads across pre-LLM and in-LLM pruning. Our findings highlight two key principles for effective visual token pruning: preserving query-relevant evidence before the LLM and using query-sensitive attention heads to identify task-relevant tokens within the LLM. Future work may explore adaptive token allocation between the pre-LLM and in-LLM stages based on input complexity and query demands. References Achiam et al. (2023) Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Alvar et al. (2025) Alvar, S. R.; Singh, G.; Akbari, M.; and Zhang, Y. 2025. Divprune: Diversity-based visual token pruning for large multimodal models. In Proceedings of the Computer Vision and Pattern Recognition Conference, 9392–9401. Bai et al. (2023) Bai, J.; Bai, S.; Chu, Y.; Cui, Z.; Dang, K.; Deng, X.; Fan, Y.; Ge, W.; Han, Y.; Huang, F.; et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609. Bai et al. (2025) Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; Zhong, H.; Zhu, Y.; Yang, M.; Li, Z.; Wan, J.; Wang, P.; Ding, W.; Fu, Z.; Xu, Y.; Ye, J.; Zhang, X.; Xie, T.; Cheng, Z.; Zhang, H.; Yang, Z.; Xu, H.; and Lin, J. 2025. Qwen2.5-VL Technical Report. arXiv preprint arXiv:2502.13923. Bolya et al. (2023) Bolya, D.; Fu, C.-Y.; Dai, X.; Zhang, P.; Feichtenhofer, C.; and Hoffman, J. 2023. Token Merging: Your ViT But Faster. In International Conference on Learning Representations. Chen et al. (2024) Chen, L.; Zhao, H.; Liu, T.; Bai, S.; Lin, J.; Zhou, C.; and Chang, B. 2024. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision, 19–35. Springer. Clark et al. (2026) Clark, C.; Zhang, J.; Ma, Z.; Park, J. S.; Salehi, M.; Tripathi, R.; Lee, S.; Ren, Z.; Kim, C. D.; Yang, Y.; Shao, V.; Yang, Y.; Huang, W.; Gao, Z.; Anderson, T.; Zhang, J.; Jain, J.; Stoica, G.; Han, W.; Farhadi, A.; and Krishna, R. 2026. Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding. arXiv:2601.10611. Fu et al. (2023) Fu, C.; Chen, P.; Shen, Y.; Qin, Y.; Zhang, M.; Lin, X.; Yang, J.; Zheng, X.; Li, K.; Sun, X.; et al. 2023. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv preprint arXiv:2306.13394. Hudson and Manning (2019) Hudson, D. A.; and Manning, C. D. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6700–6709. Jang et al. (2017) Jang, Y.; Song, Y.; Yu, Y.; Kim, Y.; and Kim, G. 2017. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2758–2766. Kaduri, Bagon, and Dekel (2025) Kaduri, O.; Bagon, S.; and Dekel, T. 2025. What’s in the Image? A Deep-Dive into the Vision of Vision Language Models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 14549–14558. IEEE. Li et al. (2024) Li, B.; Zhang, Y.; Guo, D.; Zhang, R.; Li, F.; Zhang, H.; Zhang, K.; Zhang, P.; Li, Y.; Liu, Z.; et al. 2024. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326. Li et al. (2023a) Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023a. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, 19730–19742. PMLR. Li et al. (2023b) Li, Y.; Du, Y.; Zhou, K.; Wang, J.; Zhao, W. X.; and Wen, J.-R. 2023b. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355. Lin et al. (2023) Lin, B.; Ye, Y.; Zhu, B.; Cui, J.; Ning, M.; Jin, P.; and Yuan, L. 2023. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122. Liu et al. (2024a) Liu, H.; Li, C.; Li, Y.; and Lee, Y. J. 2024a. Improved Baselines with Visual Instruction Tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Liu et al. (2024b) Liu, H.; Li, C.; Li, Y.; Li, B.; Zhang, Y.; Shen, S.; and Lee, Y. J. 2024b. Llavanext: Improved reasoning, ocr, and world knowledge. Liu et al. (2023) Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual instruction tuning. Advances in neural information processing systems, 36: 34892–34916. Liu et al. (2026) Liu, J.; Wang, Y.; Ma, H.; Wu, X.; Ma, X.; Wei, X.; Jiao, J.; Wu, E.; and Hu, J. 2026. Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input. International Journal of Computer Vision, 134(3). Liu et al. (2024c) Liu, Y.; Duan, H.; Zhang, Y.; Li, B.; Zhang, S.; Zhao, W.; Yuan, Y.; Wang, J.; He, C.; Liu, Z.; et al. 2024c. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, 216–233. Springer. Lu et al. (2022) Lu, P.; Mishra, S.; Xia, T.; Qiu, L.; Chang, K.-W.; Zhu, S.-C.; Tafjord, O.; Clark, P.; and Kalyan, A. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35: 2507–2521. Singh et al. (2019) Singh, A.; Natarajan, V.; Shah, M.; Jiang, Y.; Chen, X.; Batra, D.; Parikh, D.; and Rohrbach, M. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8317–8326. Singh et al. (2026) Singh, A. K.; Kandala, H.; Brahma, P. P.; Liu, Z.; and Barsoum, E. 2026. DUET-VLM: Dual Stage Unified Efficient Token Reduction for VLM Training and Inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Takezoe et al. (2026) Takezoe, R.; Li, Y.; Bo, Z.; Hou, A.; Guang, M.; and Long, K. 2026. LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models. In International Conference on Learning Representations. Touvron et al. (2023) Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; Rodriguez, A.; Joulin, A.; Grave, E.; and Lample, G. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302.13971. Xing et al. (2024) Xing, L.; Huang, Q.; Dong, X.; Lu, J.; Zhang, P.; Zang, Y.; Cao, Y.; He, C.; Wang, J.; Wu, F.; et al. 2024. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.17247. Xu et al. (2017) Xu, D.; Zhao, Z.; Xiao, J.; Wu, F.; Zhang, H.; He, X.; and Zhuang, Y. 2017. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM International Conference on Multimedia, 1645–1653. Yang et al. (2025) Yang, S.; Chen, Y.; Tian, Z.; Wang, C.; Li, J.; Yu, B.; and Jia, J. 2025. Visionzip: Longer is better but not necessary in vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 19792–19802. Zhang et al. (2026) Zhang, C.; Ma, K.; Fang, T.; Yu, W.; Zhang, H.; Zhang, Z.; Mi, H.; and Yu, D. 2026. VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Models. Transactions on Machine Learning Research. Zhang et al. (2025a) Zhang, Q.; Cheng, A.; Lu, M.; Zhang, R.; Zhuo, Z.; Cao, J.; Guo, S.; She, Q.; and Zhang, S. 2025a. Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs. Proceedings of the IEEE International Conference on Computer Vision. Zhang et al. (2025b) Zhang, Y.; Fan, C.-K.; Ma, J.; Zheng, W.; Huang, T.; Cheng, K.; Gudovskiy, D.; Okuno, T.; Nakata, Y.; Keutzer, K.; et al. 2025b. SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference. In International Conference on Machine Learning. Zhu et al. (2023) Zhu, D.; Chen, J.; Shen, X.; Li, X.; and Elhoseiny, M. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592.