Paper deep dive
IWP: Token Pruning as Implicit Weight Pruning in Large Vision Language Models
Dong-Jae Lee, Sunghyun Baek, Junmo Kim
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 4/2/2026, 3:30:12 AM
Summary
The paper introduces IWP (Implicit Weight Pruning), a training-free token pruning framework for Large Vision-Language Models (LVLMs). By reformulating self-attention into a dual-form linear transformation where each token contributes a rank-1 update to a dual weight matrix, the authors derive a novel metric that quantifies both information magnitude and information duplication. To efficiently select tokens, they propose Progressive Chunked Maximal Marginal Relevance (PC-MMR), which balances these criteria to maintain performance while reducing computational overhead.
Entities (4)
Relation Signals (3)
IWP → utilizes → PC-MMR
confidence 98% · To efficiently select the subset with the proposed metric, we introduce Progressive Chunked Maximal Marginal Relevance.
PC-MMR → optimizes → Token Selection
confidence 96% · Progressive Chunked MMR... performs token selection at the chunk level and progressively expands the number of selected tokens.
IWP → improves → LVLM
confidence 95% · Extensive experiments demonstrate that our method achieves a better trade off between performance and efficiency
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Vision Language Models show impressive performance across image and video understanding tasks, yet their computational cost grows rapidly with the number of visual tokens. Existing token pruning methods mitigate this issue through empirical approaches while overlooking the internal mechanism of attention. In this paper, we propose a novel training free token pruning framework grounded in the dual form perspective of attention. We reformulate attention as an implicit linear layer whose weight matrix is the sum of rank 1 outer products, each generated by a single token's key value pair. Token pruning thus reduces to selecting an optimal subset of these rank 1 updates that best approximates the original dual weight matrix. Extending this perspective to standard softmax attention in LVLMs, we derive a novel metric quantifying both a token's information magnitude and information duplication. To efficiently select the subset with the proposed metric, we introduce Progressive Chunked Maximal Marginal Relevance. Extensive experiments demonstrate that our method achieves a better trade off between performance and efficiency, while providing another perspective on existing pruning approaches.
Tags
Links
- Source: https://arxiv.org/abs/2604.00757v1
- Canonical: https://arxiv.org/abs/2604.00757v1
Trouble viewing inline? Open PDF directly →
Full Text
80,737 characters extracted from source content.
Expand or collapse full text
IWP: Token Pruning as Implicit Weight Pruning in Large Vision Language Models Dong-Jae Lee ∗ , Sunghyun Baek ∗ , and Junmo Kim KAIST jhtwosun,baeksh,junmokim@kaist.ac.kr Abstract. Large Vision–Language Models (LVLMs) show impressive performance across image and video understanding tasks, yet their computational cost grows rapidly with the number of visual tokens. Existing token pruning methods mitigate this issue through em- pirical approaches while overlooking the internal mechanism of attention. In this paper, we propose a novel training-free token pruning framework grounded in the dual-form perspec- tive of attention. We reformulate attention as an implicit linear layer whose weight matrix is the sum of rank-1 outer products, each generated by a single token’s key–value pair. To- ken pruning thus reduces to selecting an optimal subset of these rank-1 updates that best approximates the original dual weight matrix. Extending this perspective to standard soft- max attention in LVLMs, we derive a novel metric quantifying both a token’s information magnitude and information duplication. To efficiently select the subset with the proposed metric, we introduce Progressive Chunked Maximal Marginal Relevance. Extensive experi- ments demonstrate that our method achieves a better trade-off between performance and efficiency, while providing another perspective on existing pruning approaches. The code is available in https://github.com/jhtwosun/IWP. Keywords: Large Vision-Language Model· Inference Acceleration· Dual form of Attention 1 Introduction Recent advances in Large Vision-Language Models (LVLMs) [15, 18, 29, 33] have significantly en- hanced multimodal understanding. To achieve fine-grained perception, there has been a notable shift towards processing high-resolution images and extended videos. However, this trend inherently gen- erates an excessive number of visual tokens. Since the computational cost of self-attention scales quadratically with sequence length, this surge in visual tokens leads to prohibitive computational and memory overheads. To alleviate this computational bottleneck, token pruning [2], originally developed for accelerating standalone vision transformers [10], has emerged as a prominent solution for LVLMs [1,4,9,17,34]. Existing methods typically regard attention scores as token importance scores [4,9,34], enforce token diversity to minimize redundancy [1,9], or combine both objectives [8,38]. Despite their re- markable performance-efficiency trade-offs, it remains unclear how token pruning affects the internal mechanics of attention. In this paper, we aim to establish a principled token pruning framework that departs from empirical approaches. To this end, we adopt the dual-form perspective of linear attention [6,13,25], which reformulates attention as an implicit linear transformation whose weight matrix is constructed * The authors are equally contributed. arXiv:2604.00757v1 [cs.CV] 1 Apr 2026 IWP: Token Pruning as Implicit Weight Pruning in LVLMs2 from key–value pairs and applied to the query to produce the output. Under this perspective, the implicit weight matrix can be decomposed into token-wise update matrices, each represented as the outer product of the corresponding key and value vectors. Consequently, token pruning reduces to identifying the subset whose summation best preserves the information of the original dual weight matrix. Building on this perspective, we extend the dual-form analysis to standard softmax attention in LVLMs and derive a novel, training-free token selection metric that captures both information magnitude and information redundancy. Specifically, we quantify each token’s contribution to the dual weight matrix and measure the redundancy among token-induced updates. We then select the tokens that maximize information magnitude while minimizing this redundancy. To this end, we introduce Progressive Chunked Maximal Marginal Relevance (PC-MMR), which performs token selection at the chunk level and progressively expands the number of selected tokens. By integrating our novel dual-form based metric with PC-MMR, our approach efficiently selects a subset of tokens that best preserves the information of the original dual weight matrix. Experiments on two representative LVLMs across diverse image and video benchmarks demon- strate that our theoretically motivated pruning strategy consistently outperforms prior methods. These results suggest that the dual-form perspective provides a faithful lens for analyzing token importance in LVLMs. 2 Problem Definition and Related Work Problem Definition LVLM processes an image I and a text T to generate a response 1 . For- mally, the visual features from the encoder are mapped into the Large Language Model (LLM)’s hidden space via projection layer, yielding a sequence of N img visual tokens Z img ∈R N img ×d , where d is the hidden dimension. Simultaneously, a text embedding layer maps the input text into N text tokens Z text ∈R N text ×d . Then, LLM processes the concatenated multimodal input Z = [Z img ; Z text ] ∈R N×d , where the total sequence length is N = N img + N text . Given a tar- get reduction ratio ρ ∈ (0, 1), a token reduction operator is applied at a specific LLM layer l or projection layer, producing a compressed subset of (1−ρ)N img visual tokens. The newly formed vi- sual sequence is then concatenated with the N text text tokens and passed to subsequent transformer layers. Related Work LVLMs [15,18,29,33] process extensive sequences of visual tokens, resulting in sub- stantial computational overhead due to the quadratic complexity of self-attention. To mitigate this, prior works exploit attention score-derived signals at either the Large Language Model (LLM) or the vision encoder level. Methods such as FastV [4], PyramidDrop [32], SparseVLM [39], and PACT [9] select image tokens based on attention scores or query-token interactions of LLM. Meanwhile, ap- proaches like VisionZip [34] and VisPruner [37] rely on vision encoder self-attention statistics to measure token importance. Beyond relying solely on attention score, several works [1,9,30] focus on mitigating redundancy among visual tokens. For instance, DivPrune [1] enforces diversity through minimum-distance con- straints applied to the cosine similarity of hidden states, reducing overlapping visual information. More recently, hybrid approaches jointly optimize both token importance and redundancy. CD- Pruner [38] introduces conditional diversity maximization to balance redundancy with instruction 1 For simplicity, we omit system prompt. IWP: Token Pruning as Implicit Weight Pruning in LVLMs3 relevance. Similarly, PACT [9] leverages attention scores for token importance, while utilizing key cosine similarity to guide the clustering processes. Despite the impressive empirical success of visual token pruning in LVLMs, existing attention- or diversity-based approaches do not explicitly account for the underlying attention mechanism. To overcome these limitations, we reformulate self-attention in its dual form. This formulation allows us to analyze the structural contribution of each token, thereby introducing a novel, principled criterion that quantifies both importance and redundancy. 3 Transformer Attention in Dual Form Conventionally, standard attention [28] is viewed as an associative retrieval process, where similarity scores between a query and a set of keys determine the weights used to aggregate values. However, by exploiting the associativity of matrix multiplication, recent studies [6,13,25] demonstrate that linear attention can be equivalently reinterpreted as a linear transformation. For clarity of explanation, we first derive the dual form of unnormalized linear attention, where the implicit weight matrix construction from key–value pairs becomes most transparent. We then extend this analysis to standard attention to obtain its corresponding dual form. For clarity, we consider a single attention head throughout this section, as the derivation applies independently to each head. Throughout, let q∈R 1×d , K∈R N×d , and V∈R N×d v denote the query, keys, and values, respectively, all of which are obtained by projecting the input hidden states Z, where N denotes the number of tokens. 3.1 Linear Attention in Dual Form Unnormalized linear attention is defined as LinearAttn(q, K, V) = qK ⊤ V N X i=1 α i v i , where α i = qk ⊤ i . (1) In this primal form, attention is viewed as a weighted aggregation of values, acting as an associative retrieval mechanism. By the associativity of matrix multiplication, this operation is equivalent to: LinearAttn(q, K, V) = qW N ,W N = N X i=1 k ⊤ i v i ∈R d×d v ,(2) where k i and v i denote the i-th rows of K and V, respectively. In its dual form, linear attention can be reinterpreted as a linear transformation of the query by the dual weight matrix W N , constructed from the summation of outer products between key-value pairs. As noted in prior work [6,25], the attention update k ⊤ i v i shares the algebraic form of a gradient- based linear layer update. In standard gradient descent with a loss L, the weight matrix update takes the form x ⊤ i e i , where x i ∈R 1×d is the input representation and e i = −∇ y i L ∈R 1×d v is the learning signal where y i ∈R 1×d v is the output corresponding to the x i . This results in an accumulated weight update expressed as W N = P N x ⊤ i e i . Under this correspondence, each key k i functions as an input representation of train sample, and each value v i serves as the corresponding learning signal. Consequently, linear attention can be interpreted as an implicit optimization process in which each token contributes an update k ⊤ i v i to the dual weight matrix W N , satisfying rank(k ⊤ i v i ) = 1, ∀i∈1,...,N.(3) IWP: Token Pruning as Implicit Weight Pruning in LVLMs4 Therefore, each token contributes a rank-1 update to the dual weight matrix. 2 3.2 Softmax Attention in Dual Form Standard softmax attention is defined as SoftmaxAttn(q, K, V) = softmax qK ⊤ √ d V = P N i=1 exp qk ⊤ i √ d v i P N j=1 exp qk ⊤ j √ d . (4) Let κ(x, y) = exp xy ⊤ / √ d for x, y ∈R 1×d denote the exponential kernel. Since κ is positive definite, there exists a Reproducing Kernel Hilbert space (RKHS)H and a feature mapφ :R 1×d → H such thatκ(x, y) =⟨φ(x),φ(y)⟩ H .Substituting the kernel expansion yields SoftmaxAttn(q, K, V) = P N i=1 ⟨φ(q),φ(k i )⟩ H v i P N j=1 ⟨φ(q),φ(k j )⟩ H = η N (q) N X i=1 ⟨φ(q),φ(k i )⟩ H v i , (5) where η N (q) = P N j=1 κ(q, k j ) −1 . Assuming that φ(x)∈R 1×m for some finite m, the inner product can be written as ⟨φ(x),φ(y)⟩ H = φ(x)φ(y) ⊤ . Under this assumption, we can factor out φ(q) to obtain the dual form: SoftmaxAttn(q, K, V) = η N (q)φ(q) N X i=1 φ(k i ) ⊤ v i ! = φ(q)η N (q)W N , (6) where W N = P N i=1 φ(k i ) ⊤ v i is the accumulated dual weight constructed via rank-1 outer products. Specifically, each token induces a rank-1 update ∆W i = φ(k i ) ⊤ v i ,W N = N X i=1 ∆W i .(7) The key difference from linear attention is that W N is constructed using the kernel-induced feature representations, mapping inputs from the Hilbert space H back to the original value spaceR d v . Notably, the effective dual weight in softmax attention is given by the normalized form η N (q)W N , reflecting the query-dependent scaling. 4 Token Pruning in Dual Form Perspective Building upon this dual perspective, we define token informativeness through the lens of rank-1 weight updates. Specifically, the contribution of an individual token i is encapsulated by its corre- sponding rank-1 update, ∆W i = φ(k i ) ⊤ v i . We can therefore assess an individual token’s informa- tiveness by evaluating the information gain from its update to the dual weight, using two distinct criteria: information magnitude and information duplication. First, information magnitude is determined by the magnitude of the rank-1 update. A negligible magnitude (e.g., ∥∆W i ∥ ≈ 0) 2 For simplicity, we refer to the dual weight matrix W N as the dual weight in the remainder of this paper. IWP: Token Pruning as Implicit Weight Pruning in LVLMs5 MatmulAttention Update Decomposition Dual Weight Approximation Rank-1 Updates Information Similarity Last Selected Chunk Remaining Set Information Magnitude Score Add Top-k Batch Penalize Information Duplication Selected Set Progressive Chunked MMR Selected Chunk Equivariant Fig. 1: Overview of the Dual-Form Token Pruning framework. Softmax attention is reinterpreted as dual form via kernel mapping, where tokens generate rank-1 updates ∆W i = φ(k i ) ⊤ v i . The Progressive Chunked MMR loop filters tokens based on information magnitude and duplication to efficiently approxi- mate the dual weights. For visual clarity, the scalar normalization term η N (q) is omitted. indicates that the token lacks salient information and has minimal influence on the dual weight; con- sequently, such tokens can be removed with minimal impact on performance. Second, information duplication is determined by the similarity between the rank-1 updates. If an update is highly simi- lar to others, its corresponding token is considered redundant; consequently, pruning this duplicated information results in negligible loss. In the subsequent sections, we detail the mathematical for- mulations and the efficient token pruning algorithm considering two informativeness criteria. An overview of the proposed framework is illustrated in Fig. 1. 4.1 Information magnitude: Dual-Form Importance Metric To address the first condition, we evaluate the individual information magnitude of each image token. Recall that the effective dual weight is constructed from two components: (i) the accumulated dual weight W N , and (i) the query-dependent normalization term η N (q). Dual Weight According to the dual formulation, each token acts as a rank-1 update to the weight: ∆W i = φ(k i ) ⊤ v i . We quantify the magnitude of the update using its Frobenius norm, ∥∆W i ∥ F . Using the identity ∥ab ⊤ ∥ F =∥a∥ 2 ∥b∥ 2 , we obtain ∥∆W i ∥ F =∥φ(k i )∥ 2 ∥v i ∥ 2 ,(8) which characterizes the magnitude of the rank-1 update from token i. Consequently, tokens with a smaller ∥∆W i ∥ F can be removed while minimizing the information loss. Effective Dual Weight Unlike unnormalized linear attention, standard softmax attention incor- porates a query-dependent normalization term, η N (q) = 1/ P N j=1 κ(q, k j ). To formulate a dual-form importance metric, it is essential to further account for the inherent property of the softmax opera- tion. In particular, a token with a minimal dual-weight magnitude ∥∆W i ∥ F may still significantly IWP: Token Pruning as Implicit Weight Pruning in LVLMs6 affect the normalization term if it shows a high similarity with the query. The kernel-space similar- ity κ(q, k i ), which represents the geometric relationship between the query and key in the Hilbert space, can be expressed as follows: κ(q, k i ) =∥φ(q)∥ 2 ∥φ(k i )∥ 2 cosθ i (9) where θ i is the angle between φ(q) and φ(k i ) in the Hilbert space H. Dual-Form Importance Metric To jointly account for these two aspects—the rank-1 update magnitude and the kernel-based alignment with the query—we introduce an isolated informativeness metric derived from Eq. (8) and Eq. (9): Score i = κ(q, k i )∥v i ∥ 2 (10) This score effectively encapsulates both the magnitude of the value update and the influence on the normalization term. The specific query q utilized to compute this score remains a flexible design choice, adaptable to text, or image. To minimize computational overhead, we aggregate the queries into a single representative vector, which can be calculated in O(n) time. In practice, we use the mean of the text-token queries, defined as: q T = 1 N text P N text j=1 q j . Alternative choices, including the mean image-token query q I and the last text-token query q T [−1] , are empirically addressed in Sec. 5. 4.2 Information Duplication: Dual-Form Similarity Metric For the second criterion, we quantify redundancy between tokens through the similarity of their induced rank-1 updates. A high similarity indicates that the updates from both tokens contain duplicated information, whereas a near-zero similarity implies that the updated information is orthogonal and introduces novel information to the effective dual weight. To compute this similarity, we derive the inner product between the corresponding update matrices. Utilizing the reproducing property ⟨φ(k i ),φ(k j )⟩ H = κ(k i , k j ), the Frobenius inner product between two rank-1 updates is given by: ⟨∆W i ,∆W j ⟩ F = (v i · v j )κ(k i , k j ).(11) Normalizing by ∥∆W i ∥ F = p κ(k i , k i )∥v i ∥ 2 , we define the dual weight similarity as S ij = ⟨∆W i ,∆W j ⟩ F ∥∆W i ∥ F ∥∆W j ∥ F = v i · v j ∥v i ∥ 2 ∥v j ∥ 2 κ(k i , k j ) p κ(k i , k i )κ(k j , k j ) ! . (12) Thus, S ij is factorized into the cosine similarity between value vectors and the cosine similarity between keys in the RKHS. We use the squared similarity S 2 ij as the metric to prioritize updates whose similarity is close to zero. For the exponential kernel κ, the latter term reduces to a Gaussian RBF kernel, exp − ∥k i −k j ∥ 2 2 2 √ d h . IWP: Token Pruning as Implicit Weight Pruning in LVLMs7 Algorithm 1 Progressive Chunked Maximal Marginal Relevance 1: Require: Visual tokens Z img ∈R N img ×d ; Target size M; Chunk size b; Growth factor g; Penalty strength λ. 2: C ←∅ // Initialize selected index set 3: U ←1,...,N img // Initialize unselected candidate indices // Calculate information magnitude scores 4: P ←P 1 ,...,P N img where P i = κ(q text ,k i )∥v i ∥ 2 5: while |C| < M do 6: k ← min(b,M −|C|) // Determine current selection budget 7: L new ← arg top-k i∈U P i // Select top-k candidates based on current scores 8: C ←C∪ L new 9: U ←U \ L new 10: // Chunked Information Duplication computation 11: S ← S(V U ,V L new ) 12: // Max-pooling over the new selected tokens to find max duplication 13: s max ← max(S, dim = 1) 14: P U ← P U ⊙ max(0.01, 1− λ· s max,i ) // Apply multiplicative penalty 15: b← b· g 16: end while 17: return C 4.3 Dual Form-based Token Pruning Finally, we integrate these two criteria, information magnitude and information duplication, into a unified token pruning framework. Relying solely on information magnitude for token selection often results in highly duplicated updates, which can lead to rank collapse in the accumulated dual weight. Conversely, selecting tokens based on information duplication might retain ineffective tokens with negligible magnitudes. Therefore, we have to select the tokens with high information magnitude, while minimizing the information duplication scores. To this end, we consider Maximal Marginal Relevance (MMR) [3], a selection method that penalizes candidates that exhibit high similarity with the already selected ones. Formally, MMR iteratively select a token from the set of all available tokens U, such that selected token maximizes its individual information magnitude score while minimizing its information duplication with tokens already in selected tokens C. The next token i ∗ is chosen according to: i ∗ = arg max i∈U [λ· P i − (1− λ) max j∈C S i,j ],(13) where P i , S i,j , and λ represents the dual-form importance metric (Score i ), dual-form similarity met- ric, and similarity penalty, respectively. However, standard MMR is inherently sequential, requiring (1−ρ)N img iterations andO(N 2 img ) pairwise comparisons at each step. This leads to severe compu- tational and time bottlenecks, during inference. To address this, we introduce Progressive Chunked MMR, an efficient algorithm that accounts for both information magnitude and duplication with minimal computational overhead and time cost compared to standard MMR. Progressive Chunked MMR (Algorithm 1) leverages chunk-based selection and comparison. Specifically, we first calculate the information magnitude of each token, which requires only O(n) IWP: Token Pruning as Implicit Weight Pruning in LVLMs8 time. Then, we iteratively select tokens in chunks rather than one at a time. In each iteration, we select a chunk of top magnitude (L new ) tokens and compute the information duplication matrix S between the selected tokens (C) and the remaining tokens (U). By calculating the maximum redun- dancy (s max ) across the remaining tokens and applying a multiplicative penalty to the remaining candidates, we can select the tokens with higher information magnitude while minimizing the in- formation duplication. Furthermore, to balance the trade-off between performance and inference speed, we progressively increase the chunk size (b ← b· g), dynamically accelerating the selection process without heavily compromising the performance. 5 Analysis While our method uses both information magnitude and duplication, we first analyze these two criteria in detail. For analysis, we conduct our experiments under fixed token budgets using LLaVA- OneVision-7B. Further analyses, experiments and experimental details can be found in Sec. 6 and Appendices B and C. 5.1 Information Magnitude In Sec. 4.1, we introduced two key aspects that should be considered when selecting tokens for token pruning: (i) the dual weight ∥∆W i ∥ F and (i) normalized term η(q). The combined information magnitude metric can be decomposed into angular alignment, key magnitude, and value magnitude, along which existing metrics are categorized in Tab. 1. Most existing methods leverage kernel-based scoring strategies. In particular, methods based on κ(q, k i ) [4,9] mainly capture angular alignment and key magnitude, while neglecting the influence of value-side magnitude. Magnitude-based metrics such as ∥∆W i ∥ F emphasize key and value magnitudes, yet they overlook the angular alignment component. In contrast, the proposed metric κ(q, k i )∥v i ∥ 2 jointly accounts for normalization and dual-weight perturbation, aligning with the dual-form structure. As shown in Tab. 2, our proposed metric κ(q T , k i )∥v i ∥ 2 , which jointly considers alignment, key magnitude, and value magnitude, achieves the best overall performance. When evaluated indi- vidually, alignment-based methods of the form κ(q, k i ) achieve comparable performance at 97.3% despite lacking value magnitude. Conversely, relying solely on magnitude leads to lower perfor- mance; the normalization term distorts the contribution of high-magnitude tokens, making the resulting scores an unreliable indicator of importance metric. Furthermore, we can expect this phe- nomenon to become more severe as the vision token length increases, since the normalization term Table 1: Comparison of the dual-form importance metrics. The proposed dual-form importance metric can be factorized into three distinct components: angular alignment cosθ i , key magnitude ∥φ(k i )∥ 2 , and value magnitude ∥v i ∥ 2 . Each metric is analyzed based on whether it accounts for these components (✓) or not (×). MetricFormulation cosθ i ∥φ(k i )∥ 2 ∥v i ∥ 2 Kernel Scoreκ(q, k i )✓ × Gradient Norm ∥∆W i ∥ F ×✓ Key Feature Norm ∥φ(k i )∥ 2 ×✓ × Value Norm∥v i ∥ 2 × ×✓ Oursκ(q, k i )∥v i ∥ 2 ✓ IWP: Token Pruning as Implicit Weight Pruning in LVLMs9 Table 2: Experiments on various benchmarks with LLaVa-OneVision-7B. In all experiments, tokens are pruned after the fourth layer with a 35.3% token budget. Scores are absolute performance, while Avg. (%) indicates the relative average normalized by the full-token baseline. MetricAI2D DocVQA InfoVQA MMBench MME MMMU SciQA TextVQA MMStar POPEAvg. (%) Baseline 81.387.166.080.71992.0 49.295.975.861.988.3 100.0 (100.0) (100.0)(100.0)(100.0) (100.0) (100.0) (100.0) (100.0)(100.0) (100.0) ∥v i ∥ 2 79.679.151.279.51933.0 49.894.267.658.788.7 94.6 (97.9) (90.9)(77.6)(98.5)(97.0) (101.1) (98.2)(89.2)(94.8) (100.4) ∥k i ∥ 2 74.048.943.072.21558.8 46.683.561.749.578.1 81.1 (91.0) (56.1)(65.2)(89.5)(78.3) (94.6) (87.1)(81.3)(79.9) (88.4) ∥∆W i ∥ F 79.069.843.879.41983.6 47.892.568.456.785.7 91.4 (97.1) (80.2)(66.4)(98.4)(99.6) (97.1) (96.4)(90.3)(91.5) (97.1) κ(q I ,k i ) 80.084.560.979.21998.5 49.392.075.357.587.0 97.3 (98.4) (97.0)(92.3)(98.2)(100.3) (100.2) (95.9)(99.3)(92.8) (98.5) κ(q T ,k i ) 79.884.660.979.41973.4 49.191.875.558.287.1 97.3 (98.1) (97.1)(92.3)(98.4)(99.1) (99.8) (95.7)(99.7)(93.9) (98.6) κ(q T [−1] ,k i ) 78.885.059.879.01985.3 49.291.675.156.186.6 96.6 (96.9) (97.6)(90.6)(97.9)(99.7) (100.0) (95.5)(99.1)(90.6) (98.1) 80.285.560.979.52003.549.192.675.558.187.4 κ(q I ,k i )∥v i ∥ 2 (98.6)(98.2)(92.2)(98.5)(100.6)(99.8)(96.5)(99.7)(93.8)(99.0) 97.7 79.885.461.479.42001.549.092.075.659.187.6 κ(q T ,k i )∥v i ∥ 2 (98.0)(98.1)(93.0)(98.4)(100.5)(99.5)(95.9)(99.7)(95.4)(99.2) 97.8 is the sum of the alignments between the query and the keys. As demonstrated in our experiments, the performance of magnitude-only metrics on high-resolution datasets like DocVQA and InfoVQA drops significantly. When comparing magnitude-only metrics, utilizing value magnitude consistently outperforms relying on key magnitude. Lastly, relying on the last text token q T [−1] degrades per- formance compared to using the aggregated queries q I or q T . By integrating these three critical components with aggregated queries, our proposed metric achieves the optimal performance. 5.2 Information Duplication As detailed in Sec. 4.2, we formulate information duplication between two tokens as the similarity between their rank-1 dual weight updates, a metric that naturally decomposes into key similarity within the RKHS and value similarity. While this perspective partially aligns with diversity-based token pruning methods [1, 8, 9, 38] that seek to minimize redundancy among retained tokens, it introduces a crucial distinction: we define redundancy through the overlap of induced dual updates rather than proximity in the raw feature space. Conventionally, such similarity is measured using two widely adopted metrics: cosine similarity between keys and cosine similarity between hidden states. To investigate which representation space effectively captures information duplication, we evaluate similarities across five variants: value, hidden-state, key, kernelized-key, and dual-weight spaces. Table 3 reveals a clear performance trend across representation spaces. Similarity measured in the key space consistently outperforms value-space similarity, while hidden-state similarity achieves intermediate performance. Notably, using cosine similarity in the RKHS improves performance by 1.7 percentage points (p), highlighting the importance of kernel-induced similarity. Consequently, IWP: Token Pruning as Implicit Weight Pruning in LVLMs10 Table 3: Experiments on various benchmarks with LLaVa-OneVision-7B. In all experiments, tokens are pruned after the fourth layer with a 35.3% token budget. Scores are absolute performance, while Avg. (%) indicates the relative average normalized by the full-token baseline. MetricAI2D DocVQA InfoVQA MMBench MME MMMU SciQA TextVQA MMStar POPE Avg. (%) Baseline 81.387.166.080.71992.0 49.295.975.861.988.3 100.0 (100.0) (100.0)(100.0)(100.0) (100.0) (100.0) (100.0) (100.0)(100.0) (100.0) cos(v i ,v j ) 80.377.850.178.61901.9 48.092.871.858.188.1 93.9 (98.8) (89.3)(76.0)(97.4)(95.5) (97.5) (96.7)(94.7)(93.8) (99.7) cos(h i ,h j ) 79.980.951.379.21900.4 48.692.672.958.588.6 94.9 (98.2) (92.9)(77.7)(98.2)(95.4) (98.6) (96.5)(96.2)(94.4) (100.4) cos(k i ,k j ) 80.081.555.180.11913.8 48.693.873.759.188.0 95.9 (98.3) (93.5)(83.5)(99.3)(96.1) (98.6) (97.7)(97.3)(95.4) (99.6) cos(φ(k i ),φ(k j )) 80.384.460.279.61965.9 49.092.775.059.988.5 97.6 (98.7) (96.9)(91.3)(98.7)(98.7) (99.5) (96.6)(99.0)(96.8) (100.2) 80.585.360.679.61985.048.992.975.260.088.1 cos(∆W i ,∆W j ) (98.9)(98.0)(91.9)(98.6)(99.7)(99.3)(96.8)(99.2)(96.9)(99.7) 97.9 measuring dual weight similarity achieves the best performance at 97.9%, outperforming two widely used metrics. This demonstrates that the similarity between rank-1 dual weight updates effectively captures information redundancy. 6 Experiments 6.1 Experimental Setup Evaluation Datasets. To evaluate the generality and robustness of the proposed token prun- ing method, we conduct experiments on a diverse set of image-based vision–language benchmarks. For text-centric visual understanding, we use AI2D [14], TextVQA [26], DocVQA [23], and Info- graphic VQA [22], which involve text-rich documents, charts, and diagrams. For knowledge-intensive multimodal reasoning, we evaluate on MMBench [19], MMMU [35], MMStar [5], and SciQA [20]. Finally, we use MME [11] and POPE [16] to assess comprehensive perception and cognition abil- ities as well as hallucination. We further evaluate the method on video benchmarks, including EgoSchema [21], Video-MME [12], MLVU [40], and NExT-QA [31], to demonstrate its effective- ness in video understanding and temporal reasoning. Detailed descriptions of each benchmark are provided in Appendix A. Evaluation Metrics. We evaluate both predictive performance and computational efficiency. For each benchmark, we report the official metric (e.g., accuracy) and the relative performance with respect to the vanilla model, defined as Relative Performance (%) = Score Pruned Score Baseline × 100. This ratio indicates how much performance is preserved after token pruning. Implementation Details. All experiments are conducted with pretrained LLava-OneVision-7B [15] and Qwen2.5-VL-7B [33]. To optimize computational efficiency, FlashAttention2 [7] is employed across all models and other methods, if supported. Quantitative evaluations are conducted using the lmms-eval framework [36]. All reported baseline results were reproduced under our experimental IWP: Token Pruning as Implicit Weight Pruning in LVLMs11 Table 4: Performance comparison of different pruning methods on LLaVA-OneVision-7B. Scores are absolute performance, while Avg. (%) indicates the relative average normalized by the full-token baseline. MethodAI2D DocVQA InfoVQA MMBench MME MMMU SciQA TextVQA MMStar POPEAvg. (%) Upper Bound, All Tokens (100%) LLaVA-OneVision-7B 81.387.166.080.71992.0 49.295.975.861.988.3100.0 Token Budget: 35.3% (↓ 64.7%) FastV79.684.558.079.11984.5 48.491.875.658.786.596.7 PACT79.984.260.979.31947.8 48.392.775.459.188.3 97.3 VisionZip76.247.636.077.71916.2 46.490.163.053.887.885.3 DivPrune79.271.046.778.41850.3 49.691.867.757.388.391.8 CDPruner78.060.350.679.91914.3 46.992.871.257.988.591.7 Ours80.585.360.679.61985.048.992.975.260.088.197.9 Token Budget: 22.2% (↓ 77.8%) FastV78.680.652.178.51994.5 47.791.174.657.084.794.4 PACT79.780.554.977.81905.4 47.890.973.857.387.394.7 VisionZip73.037.232.678.81840.6 46.988.554.553.385.981.4 DivPrune77.060.439.277.11758.0 48.789.361.255.587.386.9 CDPruner77.856.952.479.41889.0 47.091.568.957.988.391.0 Ours79.382.256.578.81977.947.891.574.457.387.395.6 Token Budget: 11.1% (↓ 88.9%) FastV77.067.441.477.01955.8 47.989.371.353.279.288.8 PACT75.874.649.376.51883.7 45.189.470.255.483.390.4 VisionZip70.826.929.374.71788.2 46.084.843.547.383.175.4 DivPrune74.445.233.374.51638.0 47.385.052.148.586.179.8 CDPruner77.856.254.778.21849.5 47.490.966.756.788.490.4 Ours76.871.945.576.51947.047.489.570.754.684.090.5 setup to ensure fair comparison. Implementation details of each method are provided in Appendix B. For our method, we set the pruning layer to l = 4, the initial chunk size to b 0 = 2, the chunk growth factor to g = 2, and the duplication penalty strength to λ = 5. Ablation studies on hyperparameters are provided in Appendix C. 6.2 Image Understanding LLava-OneVision Table 4 shows the performance across various visual benchmarks using LLaVA- OneVision-7B as the backbone. The experimental results demonstrate that our method achieves state-of-the-art (SOTA) performance across all three token budgets (35.3%, 22.2%, and 11.1%). Among the competing methods, PACT and FastV show the second- and third- best performance, while other approaches exhibit much lower scores. Our method consistently outperforms PACT and FastV, which rely on attention score based importance measurement and diversity criteria, across all evaluated token budgets. In particular, at a token budget of 22.2%, it surpasses PACT and FastV by 0.9 percentage points (p) and 1.2 p, respectively. These results suggest that the importance and similarity metrics derived from our dual-form perspective provide a more precise estimation of token informativeness compared to previous empirical based methods. Qwen2.5-VL We further evaluate the proposed method using Qwen2.5-VL-7B, which differs from LLaVA-OneVision-7B in both its vision encoder, LLM backbone and visual token generation pipeline (e.g., dynamic resolution vs AnyRes). As shown in Tab. 5, our method achieves the best performance IWP: Token Pruning as Implicit Weight Pruning in LVLMs12 Table 5: Performance comparison of different pruning methods on Qwen2.5-VL-7B. Scores indicate absolute performance, while Avg. (%) represents the relative average normalized by the full-token baseline. MethodAI2D DocVQA InfoVQA MMBench MME MMMU SciQA TextVQA MMStar POPEAvg.(%) Upper Bound, All Tokens (100%) Qwen2.5-VL-7B 82.494.680.484.22310.2 50.788.382.862.387.6100.0 Token Budget: 35.3% (↓ 64.7%) FastV79.580.664.381.02006.6 49.874.589.156.482.1 92.1 PACT79.669.765.878.32244.4 47.677.390.555.981.291.7 VisionZip81.189.463.782.62307.6 51.087.879.559.385.2 96.0 DivPrune80.986.060.781.52274.0 48.786.678.957.884.994.0 CDPruner80.376.153.381.52231.5 49.685.973.457.681.7 90.9 Ours81.890.566.882.32319.850.087.280.761.085.796.8 Token Budget: 22.2% (↓ 77.8%) FastV76.981.453.479.02236.2 49.073.778.553.077.988.7 PACT78.571.657.277.22222.5 47.476.278.055.082.888.8 VisionZip78.778.450.980.62229.0 49.185.574.555.083.590.2 DivPrune78.076.348.079.52193.4 49.184.975.154.083.889.2 CDPruner78.966.244.479.12108.6 49.783.567.653.279.4 85.8 Ours80.584.154.981.62242.749.686.078.058.684.193.0 Token Budget: 11.1% (↓ 88.9%) FastV71.663.739.576.72006.6 48.472.872.347.568.580.2 PACT72.951.939.376.22108.0 46.475.267.548.678.880.2 VisionZip73.856.536.975.81995.1 46.183.163.349.478.980.4 DivPrune72.457.534.877.12033.7 46.081.564.648.579.280.2 CDPruner74.149.533.475.71925.0 47.382.755.946.873.577.2 Ours76.464.037.877.72152.348.884.071.653.378.284.7 across all token budgets. Notably, the relative ranking among baselines shifts compared to LLaVA- OneVision. VisionZip, which operates at the vision encoder level, becomes more competitive, while LLM-level methods such as FastV and PACT show relatively lower performance. Even against this competitive baseline, our method outperforms VisionZip by an average of 2.6 p. This suggests that other token pruning methods are sensitive to the underlying architecture, whereas the dual-form based token pruning generalizes more robustly across different models. Since our methods measures the actual information updates in self-attention, it does not rely on architecture-specific patterns. Therefore, it can easily adapt to different visual token structures, such as dynamic resolution. 6.3 Video Understanding In Tab. 6, we conduct experiments on various video benchmarks. Our proposed approach demon- strates a better trade-off between performance and computational efficiency across benchmarks. On long-video reasoning tasks such as EgoSchema, our method effectively preserves the essential visual semantics required for understanding long sequences, exceeding the second-ranked method by 1.2 p. Furthermore, on NextQA and Video-MME, which require precise causal reasoning and diverse contextual comprehension, our approach achieves better performance compared to other methods, while removing significant number of tokens. IWP: Token Pruning as Implicit Weight Pruning in LVLMs13 Table 6: Performance comparison of differ- ent pruning methods on various Video Bench- marks on LLaVA-OneVision-7B. Table 7: Efficiency comparison of different pruning methods on LLaVA-OneVision-7B. MethodEgo. V.MME MLVU NExT Avg.(%) Upper Bound, All Tokens (100%) LLaVA-OneVision-7B 62.4 58.464.7 79.3 100.0% Token Budget: 11.1% (↓ 88.9%) FastV58.2 53.457.3 74.5 92.0% PACT61.0 54.561.1 76.7 95.7% DivPrune58.8 53.861.0 76.8 93.9% Ours62.254.860.977.696.9% MethodGen.time(ms) VRAM(GB) Acc.(%) Upper Bound, All Tokens (100%) LLaVA-OneVision-7B1.071.7895.9 Token Budget: 11.1% (↓ 88.9%) FastV0.881.6889.3 PACT0.781.6189.4 DivPrune1.291.6385 Ours0.641.6189.5 6.4 Computational Efficiency To evaluate the practical hardware efficiency of our proposed token pruning framework, we compre- hensively profile the generation time (Gen.time) and maximum GPU memory allocation (VRAM). All measurements are conducted on a single NVIDIA RTX A6000 GPU using the ScienceQA dataset. For a fair comparison, we standardize the attention implementation to PyTorch [24] Scaled Dot-Product Attention (SDPA) across all baselines, which utilizes FlashAttention-2 (FA) where applicable. As shown in Tab. 7, although DivPrune applies pruning at the projector stage, its reliance on a greedy algorithm introduces computational and memory overheads that actually in- crease generation time. Furthermore, since FastV and PACT rely on full attention for scoring or clustering, respectively, they achieve only marginal speedups and show slightly lower performance than our method. In contrast, our approach performs token reduction within the LLM stage and fully supports FA. By identifying and keeping only the most informative visual tokens based on our dual-form perspective, our method minimizes performance drops. As a result, our approach achieves the best trade-off by reducing both generation time and peak VRAM, demonstrating its practical value for deploying LVLMs. 7 Conclusion In this paper, we address the computational and memory bottlenecks in LVLMs through visual token pruning. Unlike existing empirical methods, we introduce another perspective based on the dual form of self-attention. By reinterpreting softmax attention as an implicit linear transformation updated by key-value pairs, we formulate token pruning as selecting the most informative rank- 1 updates. This perspective allows us to derive a novel, training-free token pruning method that explicitly balances information magnitude and duplication to identify the most informative visual tokens. Extensive experiments across diverse benchmarks show that our approach consistently out- performs prior pruning methods. Notably, our results demonstrate that the dual-form perspective provides a reliable and interpretable way to analyze token importance. Furthermore, we provide an in-depth analysis of two key aspects related to the informativeness of each token under the dual-form perspective. Limitation While our dual-form pruning framework provides an effective and efficient approach to token pruning, its current empirical scope is focused primarily on visual modalities. Because our IWP: Token Pruning as Implicit Weight Pruning in LVLMs14 token selection metric is derived from the intrinsic mechanics of self-attention rather than vision- specific characteristics, it is fundamentally modality-agnostic. This inherent generality suggests a strong potential for adapting our framework to other high-density continuous modalities, such as audio, speech, or point clouds. However, the practical integration and efficacy of this perspective across such diverse multimodal inputs remain under-explored in this work, which will be our future work. References 1. Alvar, S.R., Singh, G., Akbari, M., Zhang, Y.: Divprune: Diversity-based visual token pruning for large multimodal models. In: Proceedings of the Computer Vision and Pattern Recognition Conference. p. 9392–9401 (2025) 2. Bolya, D., Fu, C.Y., Dai, X., Zhang, P., Feichtenhofer, C., Hoffman, J.: Token merging: Your vit but faster. In: The Eleventh International Conference on Learning Representations (2023), https: //openreview.net/forum?id=JroZRaRw7Eu 3. Carbonell, J., Goldstein, J.: The use of mmr, diversity-based reranking for reordering documents and producing summaries. In: Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval. p. 335–336 (1998) 4. Chen, L., Zhao, H., Liu, T., Bai, S., Lin, J., Zhou, C., Chang, B.: An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In: European Conference on Computer Vision. p. 19–35. Springer (2024) 5. Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Wang, J., Qiao, Y., Lin, D., et al.: Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems 37, 27056–27087 (2024) 6. Dai, D., Sun, Y., Dong, L., Hao, Y., Ma, S., Sui, Z., Wei, F.: Why can gpt learn in-context? lan- guage models secretly perform gradient descent as meta-optimizers. In: Findings of the Association for Computational Linguistics: ACL 2023. p. 4005–4019 (2023) 7. Dao, T.: Flashattention-2: Faster attention with better parallelism and work partitioning. In: The Twelfth International Conference on Learning Representations (2024), https://openreview.net/ forum?id=mZn2Xyh9Ec 8. Deng, J., Li, W., Zhou, J.T., He, Y.: SCOPE: Saliency-coverage oriented token pruning for efficient multimodel LLMs. In: The Thirty-ninth Annual Conference on Neural Information Processing Systems (2025), https://openreview.net/forum?id=oUghNi5XWc 9. Dhouib, M., Buscaldi, D., Vanier, S., Shabou, A.: Pact: Pruning and clustering-based token reduction for faster visual language models. In: Proceedings of the Computer Vision and Pattern Recognition Conference. p. 14582–14592 (2025) 10. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Trans- formers for image recognition at scale. In: International Conference on Learning Representations (2021), https://openreview.net/forum?id=YicbFdNTTy 11. Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., Wu, Y., Ji, R., Shan, C., He, R.: MME: A comprehensive evaluation benchmark for multimodal large language models. In: The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2025), https://openreview.net/forum?id=DgH9YCsqWm 12. Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al.: Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. p. 24108– 24118 (2025) IWP: Token Pruning as Implicit Weight Pruning in LVLMs15 13. Irie, K., Csordás, R., Schmidhuber, J.: The dual form of neural networks revisited: Connecting test time predictions to training patterns via spotlights of attention. In: International Conference on Machine Learning. p. 9639–9659. PMLR (2022) 14. Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., Farhadi, A.: A diagram is worth a dozen images. In: European conference on computer vision. p. 235–251. Springer (2016) 15. Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., Li, C.: LLaVA-onevision: Easy visual task transfer. Transactions on Machine Learning Research (2025), https://openreview.net/forum?id=zKv8qULV6n 16. Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W.X., Wen, J.R.: Evaluating object hallucination in large vision-language models. In: Proceedings of the 2023 conference on empirical methods in natural language processing. p. 292–305 (2023) 17. Lin, Z., Lin, M., Lin, L., Ji, R.: Boosting multimodal large language models with visual tokens with- drawal for rapid inference. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 39, p. 5334–5342 (2025) 18. Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. In: Thirty-seventh Conference on Neural Information Processing Systems (2023), https://openreview.net/forum?id=w0H2xGHlkw 19. Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al.: Mmbench: Is your multi-modal model an all-around player? In: European conference on computer vision. p. 216–233. Springer (2024) 20. Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.W., Zhu, S.C., Tafjord, O., Clark, P., Kalyan, A.: Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in neural information processing systems 35, 2507–2521 (2022) 21. Mangalam, K., Akshulakov, R., Malik, J.: Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems 36, 46212–46244 (2023) 22. Mathew, M., Bagal, V., Tito, R., Karatzas, D., Valveny, E., Jawahar, C.: Infographicvqa. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. p. 1697–1706 (2022) 23. Mathew, M., Karatzas, D., Jawahar, C.: Docvqa: A dataset for vqa on document images. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. p. 2200–2209 (2021) 24. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: Pytorch: An imperative style, high-performance deep learning library (2019), https://arxiv.org/abs/1912.01703 25. Ren, R., Liu, Y.: Towards understanding how transformers learn in-context through a representation learning lens. Advances in Neural Information Processing Systems 37, 892–933 (2024) 26. Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., Rohrbach, M.: Towards vqa models that can read. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. p. 8317–8326 (2019) 27. Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., Liu, Y.: Roformer: Enhanced transformer with rotary position embedding. Neurocomputing 568, 127063 (2024) 28. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017) 29. Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al.: Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 (2024) 30. Wen, Z., Gao, Y., Wang, S., Zhang, J., Zhang, Q., Li, W., He, C., Zhang, L.: Stop looking for “impor- tant tokens” in multimodal language models: Duplication matters more. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. p. 9972–9991 (2025) 31. Xiao, J., Shang, X., Yao, A., Chua, T.S.: Next-qa: Next phase of question-answering to explaining tem- poral actions. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. p. 9777–9786 (2021) IWP: Token Pruning as Implicit Weight Pruning in LVLMs16 32. Xing, L., Huang, Q., Dong, X., Lu, J., Zhang, P., Zang, Y., Cao, Y., He, C., Wang, J., Wu, F., et al.: Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.17247 (2024) 33. Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., et al.: Qwen2.5 technical report. arXiv preprint arXiv:2412.15115 (2024) 34. Yang, S., Chen, Y., Tian, Z., Wang, C., Li, J., Yu, B., Jia, J.: Visionzip: Longer is better but not necessary in vision language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 19792–19802 (June 2025) 35. Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al.: Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. p. 9556–9567 (2024) 36. Zhang, K., Li, B., Zhang, P., Pu, F., Cahyono, J.A., Hu, K., Liu, S., Zhang, Y., Yang, J., Li, C., et al.: Lmms-eval: Reality check on the evaluation of large multimodal models. In: Findings of the Association for Computational Linguistics: NAACL 2025. p. 881–916 (2025) 37. Zhang, Q., Cheng, A., Lu, M., Zhang, R., Zhuo, Z., Cao, J., Guo, S., She, Q., Zhang, S.: Beyond text-visual attention: Exploiting visual cues for effective token pruning in vlms. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. p. 20857–20867 (2025) 38. Zhang, Q., Liu, M., Li, L., Lu, M., Zhang, Y., Pan, J., She, Q., Zhang, S.: Beyond attention or similarity: Maximizing conditional diversity for token pruning in MLLMs. In: The Thirty-ninth Annual Conference on Neural Information Processing Systems (2025), https://openreview.net/forum?id=BLLixcuZgl 39. Zhang, Y., Fan, C.K., Ma, J., Zheng, W., Huang, T., Cheng, K., Gudovskiy, D.A., Okuno, T., Nakata, Y., Keutzer, K., et al.: Sparsevlm: Visual token sparsification for efficient vision-language model infer- ence. In: International Conference on Machine Learning. p. 74840–74857. PMLR (2025) 40. Zhou, J., Shu, Y., Zhao, B., Wu, B., Liang, Z., Xiao, S., Qin, M., Yang, X., Xiong, Y., Zhang, B., et al.: Mlvu: Benchmarking multi-task long video understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 13691–13701 (2025) IWP: Token Pruning as Implicit Weight Pruning in LVLMs17 Appendix A Benchmarks and Models A.1 Evaluation benchmarks Text-centric visual understanding benchmarks. AI2D [14] A dataset consisting of over 4,900 grade school science diagrams annotated with rich ground-truth information, including object classes, relationships, and text elements. It is designed to evaluate a model’s ability to understand complex informational graphics and answer multiple-choice questions that require reasoning over both visual and textual diagrammatic components. TextVQA [26] A dataset designed to evaluate models on visual question answering tasks that require reading and reasoning about text present within images. It contains 28,408 images sourced from the Open Images dataset, paired with 45,336 questions. Models must effectively perform optical character recognition (OCR) and integrate the recognized text with visual context to provide accurate answers. We use the validation split for evaluation. DocVQA [23] A large-scale visual question answering dataset focused on document understand- ing. It comprises 12,767 diverse document images, ranging from printed forms to typed letters, accompanied by over 50,000 questions. It is designed to test a model’s ability to comprehend text, spatial layout, and structural information within scanned documents. We use the validation split for evaluation. Infographic VQA [22] A benchmark extending the document understanding task to complex infographics. It requires models to process and reason over diverse textual, graphical, and lay- out elements, often necessitating numerical reasoning and data extraction from charts and visual graphics. Knowledge-intensive multimodal reasoning benchmarks. MMBench [19] A comprehensive evaluation pipeline for assessing the fine-grained capabilities of Vision-Language Models. It features around 3,000 multiple-choice questions covering 20 ability dimensions. It introduces a circular evaluation strategy, where the choices of a question are circularly shifted to robustly test whether a model consistently predicts the correct answer regardless of the option’s position. MMMU [35] A Massive Multi-discipline Multimodal Understanding benchmark designed to eval- uate college-level reasoning abilities in multimodal models. It contains 11.5K meticulously collected questions spanning 6 core disciplines (Art & Design, Business, Science, Health & Medicine, Human- ities & Social Science, and Tech & Engineering) and 30 subjects, requiring deep domain knowledge and complex reasoning. We use the validation split for evaluation. MMStar [5] An evaluation benchmark featuring 1,500 highly challenging samples specifically cu- rated to test true visual dependency in multimodal models. The dataset is filtered to remove samples that can be answered through text-only guessing or common sense without viewing the image, thus strictly evaluating the model’s authentic visual understanding and multimodal reasoning capabili- ties. ScienceQA [20] A large-scale multimodal multiple-choice question answering benchmark focused on diverse scientific domains. It contains 21,208 questions spanning natural science, language science, and social science, categorized into 26 topics, 127 categories, and 379 skills. Among them, 48.7% include image context, 48.2% include text context, and 30.8% include both. A majority of questions IWP: Token Pruning as Implicit Weight Pruning in LVLMs18 are annotated with grounded lectures and detailed explanations, offering external knowledge and reasoning to support the correct answer. We use the test split that includes image context for evaluation. Comprehensive perception, cognition, and hallucination benchmarks. MME [11] A comprehensive evaluation benchmark for multimodal large language models that spans 14 diverse subtasks. These tasks are broadly categorized into perception (e.g., object recog- nition, OCR, color, position) and cognition (e.g., commonsense reasoning, numerical calculation, text translation). It uses instruction-based prompts to systematically measure both basic visual recognition and higher-level reasoning. POPE [16] A Polling-based Object Probing Evaluation benchmark designed to systematically as- sess object hallucination in large vision-language models. Instead of relying on generative descrip- tions, POPE formulates the evaluation as a binary classification task (Yes/No questions) inquiring about the presence of specific objects in an image. It evaluates hallucination under different settings, including random, popular, and adversarial object probing. Video understanding and temporal reasoning benchmarks. EgoSchema [21] A very long-form egocentric video understanding benchmark spanning over 250 hours of real-world video footage. It features multiple-choice questions that require complex tem- poral reasoning over 3-minute video clips. The benchmark specifically targets tasks that cannot be solved by looking at a single frame, pushing models to understand long-term human activities and intentions. Video-MME [12] A comprehensive benchmark for evaluating multimodal large language models on video analysis. It covers various video lengths (short, medium, and long up to 1 hour) and spans diverse domains including movies, sports, and daily life. The evaluation tasks encompass spatial perception, temporal reasoning, and overall narrative comprehension. MLVU [40] A Multi-task Long Video Understanding benchmark designed to assess the capabilities of models in processing and reasoning over extended video sequences. It incorporates a wide array of tasks that require capturing both fine-grained temporal details and global semantic context across long video streams. NExT-QA [31] A video question answering benchmark focusing heavily on causal and temporal action reasoning. Moving beyond simple action recognition, NExT-QA challenges models to answer “why” and “how” questions, requiring an understanding of the causal relationships between events and the temporal order of actions within the video. A.2 Evaluated models To comprehensively evaluate the generality and effectiveness of our proposed token pruning method, we conduct experiments on two state-of-the-art open-source Vision-Language Models (VLMs) that employ different visual encoding strategies. LLaVA-OneVision [15] LLaVA-OneVision is a family of large multimodal models designed for robust visual understanding across single images, multiple images, and videos. We specifically utilize the model that integrates the Qwen2-7B language model backbone with a SigLIP visual encoder. To handle high-resolution inputs, it employs the AnyRes strategy, which divides an image into a global contextual patch and multiple high-resolution local patches. This process inherently generates a IWP: Token Pruning as Implicit Weight Pruning in LVLMs19 massive sequence of visual tokens, making it an ideal architecture to validate the efficiency and performance retention of our token reduction approach. Qwen2.5-VL-7B [33] Qwen2.5-VL is the latest iteration of the Qwen-VL series, featuring signif- icant advancements in comprehending images and videos of various resolutions and aspect ratios. Built upon the Qwen2.5-7B large language model, it incorporates a sophisticated Vision Trans- former (ViT) architecture with dynamic resolution support. This allows the model to adaptively process visual inputs into a variable number of tokens based on the image size and complexity, enabling exceptional performance in fine-grained detail perception, OCR, and complex reasoning. Evaluating our method on this model demonstrates its robustness and flexibility across dynamically scaling token sequences. B Implementation Details For token pruning, we remove tokens at the beginning of each layer (i.e., prior to the attention block). To determine the information magnitude, we first extract the query, key, and value projections for each heads to calculate their respective magnitudes. Then we compute the mean information magnitude along the heads. Because this process is independent of the attention computation itself, our proposed method is orthogonal to modern attention acceleration implementations, such as FlashAttention. For both the information magnitude and information duplication measurements, we normalize the results to a range of [0, 1]. For the comparative experiments, we implemented baseline methods using their official public repositories. For methods with dynamic pruning ratios, such as PACT, we tuned the hyperparameters to align the resulting pruning ratios with the target token budget. C Further Analyses and Experiments C.1 Visual and Text Token Similarity Analysis To investigate how visual and text tokens are related, we use three similarity metrics: (1) kernelized key similarity, (2) value similarity, and (3) the square of dual-form similarity. According to Eq. (12) in our manuscript, the dual-form similarity can be factorized into the product of the cosine similarity between kernelized keys in the RKHS and the cosine similarity between value vectors. We analyze tokens extracted from Layer 4 of LLaVA-OneVision-7B and visualize the last 100 tokens of the sequence for clearer visualization. Results are provided in Fig. C.1. Across all three metrics, cross-modality similarity between visual and text tokens remains con- sistently low, suggesting that these modalities contribute distinct updates to the dual weight space. On the other hand, intra-modality patterns vary depending on the metric. While both modalities exhibit relatively high intra-similarity in the kernelized key space, value similarity reveals that vi- sual tokens are less similar to one another than text tokens. Notably, under the dual-form similarity, redundancy among text tokens becomes almost negligible, whereas visual tokens form distinct sim- ilarity clusters. These observations suggest that, within the visual token set, there exist redundant tokens that induce highly similar updates to the dual weight. This analysis supports the validity of our token pruning framework, which is designed to selectively remove such redundant visual tokens. IWP: Token Pruning as Implicit Weight Pruning in LVLMs20 Dual-Form Similarity Kernelized Key SimilarityValue Similarity Text TokenVisual Token Text Token Visual Token Visual Token Text Token Visual Token Text TokenVisual TokenText TokenVisual Token Text Token Fig. C.1: Token similarity visualization in LLaVA-OneVision-7B. We visualize the similarity be- tween visual and text tokens extracted from Layer 4 across three metrics defined in Eq. 12. For visual clarity, diagonal elements are masked and we use the last 100 tokens of the entire sequence. C.2 Ablation Study on Information magnitude and Information Duplciation Following the formulation in Sec. 4, we perform an ablation on the token reduction strategy. We com- pare two strategies: (1) Information magnitude-based pruning (Sec. 4.1), which selects tokens solely based on the importance score, and (2) Information magnitude and duplication-based pruning (Sec. 4.3), which additionally incorporates a similarity metric to discourage redundant token selection. The results are reported across varying token budgets (35.3%, 22.2%, and 11.1%) in Tab. C.1. For a relatively large token budget (e.g., 35.3%), the difference between the two strategies is marginal, with both methods achieving nearly 98% of the baseline performance. However, un- der an constrained budget (e.g., 11.1%), the average relative performance of the magnitude-only method degrades to 87.9%. In contrast, "Magnitude + Duplication" significantly outperforms the importance-only baseline by 2.6 p. Table C.1: Ablation Study on various benchmarks with LLaVa-OneVision-7B. In all experiments, tokens are pruned after the fourth layer with varying token budgets. Scores are absolute performance, while Avg. (%) indicates the relative average normalized by the full-token baseline. Budget refers to the token budget. Mag. and Mag.+Dup. refers to pruning with information magnitude only and pruning with both information magnitude and duplication, respectively. Budget Metric AI2D DocVQA InfoVQA MMBench MME MMMU SciQA TextVQA MMStar POPEAvg. (%) Baseline81.387.166.080.71992.0 49.295.975.861.988.3100.0 35.3% Mag.79.885.461.479.42001.5 49.092.075.659.187.697.8 Mag.+Dup. 80.585.360.679.61985.0 48.992.975.260.088.197.9 22.2% Mag.77.981.454.378.42003.4 48.091.073.856.585.494.7 Mag.+Dup. 79.382.256.578.81977.9 47.891.574.457.387.395.6 11.1% Mag.74.767.643.275.91906.1 47.688.268.452.380.6 87.9 Mag.+Dup. 76.871.945.576.51947.0 47.489.570.754.684.0 90.5 IWP: Token Pruning as Implicit Weight Pruning in LVLMs21 This performance gap demonstrates that relying solely on information magnitude tends to retain highly informative but redundant tokens. When the budget is sufficient, this redundancy considera- tion is negligible. However, under strict constraints, it wastes token budgets and leads to the loss of unique information. Accounting for information duplication during token pruning avoids selecting tokens with duplicate information and ensures a more diverse, comprehensive information, thereby effectively mitigating performance degradation at extreme compression rates. C.3 Ablation Study of RoPE Rotary Position Embedding (RoPE) [27] is a positional encoding widely used in Large Language Models (LLMs). We conduct an ablation study to examine its impact on our two pruning metrics: Information Magnitude (Eq. 10) and Information Duplication (Eq. 12). Our default pruning con- figuration (Setting 1) excludes RoPE when computing Information Magnitude, but incorporates it for Information Duplication. We compare this against three variants, with results summarized in Tab. C.2. We observe two findings from the results. First, excluding RoPE from Information Magnitude is beneficial. Including RoPE (Setting 2) results in a 0.9% performance degradation compared to the default setting (Setting 1). Since RoPE applies a rotation that preserves vector norms, excluding RoPE when measuring query–key alignment leads to a more accurate estimation of the information magnitude. Second, incorporating RoPE is essential for measuring Information Duplication. Remov- ing RoPE from the duplication computation (Setting 4) leads to a substantial performance drop by 3.3% relative to Setting 1. This indicates that identifying redundant tokens requires considering not only semantic similarity but also the relative positions of tokens in the sequence. Specifically, we argue that incorporating positional information reflects the inductive bias that tokens at different locations have distinct information. Table C.2: Effect of RoPE in information magnitude and information duplication metrics. Magnitude and duplication columns indicate the inclusion (✓) or exclusion (−) of RoPE in the respective metrics. Reported scores indicate absolute performance on each benchmark. Avg. (%) represents the average relative performance normalized by the full-token baseline. Setting Magnitude DuplicationAI2D DocVQA InfoVQA MMBench MME MMMU SciQA TextVQA MMStar POPEAvg. (%) (0)Baseline81.387.166.080.71992.0 49.295.975.861.988.3100.0 (1)-✓80.585.360.679.61985.0 48.992.975.260.088.197.9 (2)✓79.783.857.979.11990.9 48.493.074.359.788.697.0 (3)✓-79.878.151.678.31903.8 44.993.472.358.588.594.6 (4)--79.778.851.279.01902.3 49.292.773.358.087.994.6 C.4 Analysis on Pruning Layer Selection In LVLMs, token pruning is performed by compressing the visual token set at a specific layer of the LLM, after which the compressed visual tokens are concatenated with the text tokens and passed to the subsequent layers. Pruning at earlier layers leads to larger reductions in computation and memory usage during the prefilling stage. However, excessively early pruning may discard tokens IWP: Token Pruning as Implicit Weight Pruning in LVLMs22 Table C.3: Effect of pruning layers on LLaVA-OneVision-7B. Reported scores indicate absolute performance on each benchmark. Avg. (%) represents the average relative performance normalized by the full-token baseline. Method AI2D DocVQA InfoVQA MMBench MME MMMU SciQA TextVQA MMStar POPEAvg. (%) Baseline81.387.166.080.71992.049.295.975.861.988.3100.0 Layers 079.571.048.879.81906.3 48.993.769.459.887.393.2 279.680.255.079.21955.5 47.190.772.557.887.4 94.8 480.485.360.679.51985.0 48.892.875.160.088.097.9 679.983.660.480.41958.9 49.494.175.759.688.4 97.9 880.284.959.580.11985.3 49.393.475.859.388.297.8 1080.384.560.680.41956.2 48.694.675.260.587.9 97.9 that become important in later layers, potentially degrading the model’s perception and reason- ing capabilities. Therefore, identifying the optimal trade-off between efficiency and performance is crucial. To this end, we analyze both the performance variation across pruning layers and the consistency of the selected tokens across layers. Performance Variation across Pruning Layers. We measure model performance across various pruning layers. Tab. C.3 shows that pruning at Layer 0 and Layer 2 results in significant performance drops, retaining only 93.2% and 94.8% of the baseline performance, respectively. In contrast, once pruning is applied from Layer 4 onward, the performance remains consistently around 97.8% 97.9%, with no additional degradation. Consistency of Selected Tokens across Layers. To examine whether early-layer pruning re- moves tokens that become important in deeper layers, we measure the mIoU between the token sets selected by our method at different layers with LLaVA-OneVision-7B and ScienceQA dataset. Figure C.2 shows that token sets selected at very early layers (e.g., Layer 0 and Layer 1) exhibit relatively low similarity to those selected at later layers. In contrast, the sets selected at Layer 4 show consistently high similarity to those of deeper layers. Notably, similarity decreases at the fi- nal layer (Layer 27), which we attribute to the alignment of representations with the next-token prediction objective. Nevertheless, as shown in the preceding experiment, this discrepancy in similarity does not lead to a meaningful drop in final task performance. Based on these observations, we select Layer 4 as the optimal pruning layer, as it provides the best balance between computational efficiency and performance. IWP: Token Pruning as Implicit Weight Pruning in LVLMs23 0123456789101112131415161718192021222324252627 Layer Index 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 Layer Index mIoU of Selected Token Subset Fig. C.2: Consistency of selected tokens across layers. We visualize the mean Intersection over Union (mIoU) of the visual token subsets selected by our pruning method at different transformer layers. C.5 Additional Results: Scalability to Larger Model To evaluate the scalability of the proposed pruning framework, we conduct additional experiments on the larger Qwen2.5-VL-32B model. As shown in Tab. C.4, our method achieves the best per- formance across all evaluated token budgets. In particular, under more constrained settings with token budgets of 22.2% and 11.1%, our approach shows significantly better performance than the baselines. These results suggest that the proposed dual-form-based token selection criterion remains effective even in larger models. C.6 Ablation Study on Progressive Chunked MMR Hyperparameters We perform an ablation on three hyperparameters of Progressive Chunked MMR: the penalty strength λ, the chunk growth factor g, and the initial chunk size b 0 . The results are reported in Tab. C.5. For λ, an excessively large value (e.g., λ = 10) causes over-suppression, where even in- formative tokens receive overly strong penalties, degrading performance from 97.9% to 97.4%. For the growth factor g, large values cause redundant tokens to be selected before redundancy penalties IWP: Token Pruning as Implicit Weight Pruning in LVLMs24 Table C.4: Performance comparison of different pruning methods on Qwen2.5-VL-32B. Re- ported scores indicate absolute performance on each benchmark. Avg. (%) represents the average relative performance normalized by the full-token baseline. MethodAI2D DocVQA InfoVQA MMBench MME MMMU SciQA TextVQA MMStar POPEAvg.(%) Upper Bound, All Tokens (100%) Qwen2.5-VL-32B 84.593.080.686.62433.9 58.791.477.464.684.7100.0 Token Budget: 35.3% (↓ 64.7%) FastV78.683.862.080.82273.3 57.185.072.855.876.490.8 VisionZip81.984.560.485.12316.3 58.688.771.761.883.794.0 DivPrune82.282.359.385.12360.5 57.488.973.159.682.9 93.4 CDPruner78.871.751.082.42208.9 56.485.269.156.375.987.2 Ours82.282.159.284.22381.658.990.075.761.082.194.2 Token Budget: 22.2% (↓ 77.8%) FastV75.374.351.977.62083.7 55.683.269.350.066.884.0 VisionZip78.967.944.081.52123.9 57.887.662.957.081.086.2 DivPrune77.971.747.282.72234.2 54.786.868.657.181.5 87.7 CDPruner72.960.142.978.92079.5 56.183.163.351.871.181.3 Ours79.473.550.081.92321.257.787.673.756.479.989.6 Token Budget: 11.1% (↓ 88.9%) FastV70.455.736.972.31805.8 54.380.261.744.048.373.2 VisionZip73.044.532.375.61921.9 56.884.545.649.173.275.1 DivPrune72.453.034.677.02093.0 54.882.559.950.375.5 78.9 CDPruner67.943.432.871.01879.2 54.379.352.044.563.871.8 Ours74.058.138.477.92143.454.683.568.650.674.581.5 Table C.5: Effect of Progressive Chunked MMR hyperparameters on LLaVA-OneVision-7B. Default configurations are indicated in bold, with other hyperparameters fixed while varying one factor at a time. Reported scores indicate absolute performance on each benchmark. Avg. (%) represents the average relative performance normalized by the full-token baseline. Method AI2D DocVQA InfoVQA MMBench MME MMMU SciQA TextVQA MMStar POPEAvg. (%) Baseline81.387.166.080.71992.049.295.975.861.988.3100.0 λ 180.385.060.279.41963.3 49.492.875.259.888.497.8 580.585.360.679.61985.0 48.992.975.260.088.1 97.9 1079.983.659.780.01959.6 48.693.374.660.488.497.4 g 180.483.058.680.01980.7 48.394.174.660.987.8 97.4 280.585.360.679.61985.0 48.992.975.260.088.197.9 480.484.259.579.61969.6 49.292.975.059.787.997.5 880.384.760.079.71991.4 49.692.175.359.188.097.7 b 0 180.183.759.680.31951.3 48.093.274.860.588.597.4 280.585.360.679.61985.0 48.992.975.260.088.197.9 880.083.659.179.71946.7 48.493.574.860.288.297.2 6480.683.660.079.31962.2 48.792.975.260.288.397.5 sufficiently suppress their importance scores. In contrast, g = 1 overly emphasizes redundancy re- duction, resulting in overly conservative selection. Empirically, g = 2 achieves the best performance. IWP: Token Pruning as Implicit Weight Pruning in LVLMs25 Finally, when the initial chunk size b 0 is too large (e.g., 64), the initial selection already contains many redundant tokens, which degrades performance. C.7 Qualitative Visualization To qualitatively analyze the impact of our token pruning method on text generation, we evaluate the model’s descriptive capabilities under varying token budgets. We utilize two images from the MME benchmark, prompting the model to provide a detailed description of each image. As illustrated in Fig. C.3, our approach preserves the core semantic details present in the re- sponses generated by the unpruned vanilla model. Specifically, in Fig. C.3a, we observe that visual tokens corresponding to crucial objects, such as the baseball player and the ball, are preserved after token pruning. By retaining these highly informative tokens, the model consistently generates fine-grained descriptions of the scene, including the uniform, the player’s name, and the background context, across all evaluated token budgets. Similarly, Fig. C.3b demonstrates the model’s ability to retain fine-grained details for an image featuring a beer bottle and a glass. Our method successfully preserves both the textual and visual information of these objects. For instance, specific elements such as the text ‘since 1890’ on the bottle label, the Heineken branding on the glass to the right, and the surrounding background are accurately described regardless of the token budget constraint. These visualizations show that our method selectively preserves highly informative visual tokens even at lower token budgets. IWP: Token Pruning as Implicit Weight Pruning in LVLMs26 Q. Describe the person in the photo in detail. Original The image depicts a baseball player in action, mid-swing during a game. The player is wearing white uniform with blue accents, including a blue helmet and a blue belt. The number "17" is prominently displayed on the back of their jersey, along with the name "Ellis." The player's stance suggests they have just made contact with the ball, which is visible in the air to the left of the batter. The setting appears to be a professional baseball field, as indicated by the well-maintained dirt and grass areas, the white chalk lines marking the batter's box and foul lines, and the presence of a catcher's mitt in the lower right corner of the frame. The lighting suggests it might be an evening or night game under stadium lights. 50.0% ↓ The person in the baseball photo is a batter wearing a white uniform with blue accents, which includes a blue helmet and a blue belt. The uniform has the number "17" on the back, and the name "Ellis" is visible above the number. The player is in the middle of a swing, having just made contact with the ball, which is visible in mid-air to the left of the batter. The batter's stance suggests they are right- handed, as their body is turned towards the pitcher's mound, and their left arm is extended for balance. The setting appears to be a professional baseball game, given the well-maintained field and the presence of a catcher's mitt in the background. 64.7% ↓ The person in the baseball photo is a batter wearing a white uniform with blue accents, which includes a blue helmet and blue socks. The number "17" is prominently displayed on the back of their jersey, and the name "Ellis" is visible on the back as well. The batter appears to be in the middle of a swing, having just made contact with the ball, which is visible in mid-air slightly ahead of them. The player's stance suggests they are right-handed, and their body is angled towards home plate. The setting appears to be a professional baseball game, given the quality of the uniform and the clarity of the image. 77.8% ↓ The person in the baseball photo is a player wearing white uniform with blue accents, which suggests they are part of a team that uses these colors. The number "17" is prominently displayed on the back of their jersey, and the name "Ellis" is visible above it. The player is holding a baseball bat, positioned as if they have just swung or are about to swing at a pitch. They are wearing a blue helmet for protection, which is standard safety gear in baseball. The player's stance indicates they are actively engaged in the game, likely having just completed a swing or preparing for the next one. The background shows a portion of the baseball field, including the foul line and some grass, indicating that the photo was taken during an actual game or practice session. (a) Qualitative results for Sample 1. Q. Describe the main object in this image in detail. Original The image shows a bottle of Mahou Negra beer and a glass of Heineken beer placed on a surface with a bookshelf in the background. The Mahou Negra bottle is brown with a black label that prominently displays the brand name "Mahou" in large white letters, followed by "NEGRA" in smaller text. Below the brand name, there is additional text in Spanish that translates to "Since 1890," indicating the brand's long history. The label also mentions "PREMIUM QUALITY" and "TIPO MUNICH," suggesting the type of beer it is. The bottle has a dark liquid inside, which is characteristic of a dark beer. Next to the bottle is a glass of Heineken beer. The glass is clear and contains a dark liquid, likely the same Mahou Negra beer poured into it. The Heineken logo is visible on the glass, featuring a red star and the brand name in white text. The foam head on top of the beer in the glass indicates it has been freshly poured. The background includes a wooden bookshelf filled with various books, adding a cozy and intellectual ambiance to the setting. 50.0% ↓ The image shows a bottle of Mahou Negra beer and a glass filled with the same beer. The bottle is dark brown, which is typical for many dark beers, and has a label that prominently displays the brand name "Mahou" in large white letters. Below the brand name, it specifies "Negra," indicating the type of beer. The label also mentions "PREMIUM QUALITY" and "TIPO MUNICH," suggesting that it is a Munich-style dark beer. The text "DESDE 1890" indicates that the brand has been in existence since 1890. The glass of Heineken beer next to the bottle contains a portion of the beer, which appears to be a dark amber color, consistent with the appearance of a Munich-style dark beer. The foam on top of the beer is light, which is typical for well-poured dark beers. The background includes a bookshelf with various books, adding a casual and relaxed setting to the scene. 64.7% ↓ The image shows a bottle of Mahou Negra beer and a glass of Heineken beer placed on a surface with a bookshelf in the background. The Mahou Negra bottle is dark brown with a black label that prominently displays the brand name "Mahou" in large white letters, followed by "NEGRA" in smaller text. Below the brand name, it states "PREMIUM QUALITY TIPO MUNICH NEGRA DARK BRUNE." The bottle also mentions "DESDE 1890," indicating its long history since 1890. Next to the bottle is a glass of Heineken beer. The glass contains a dark-colored liquid, presumably the Heineken beer, topped with a layer of foam. The Heineken logo is visible on the glass, featuring the brand's iconic red star and the word "Heineken" in white text within a black rectangular frame. The background consists of a wooden bookshelf filled with various books, suggesting an indoor setting, possibly a home or a cozy reading area. The overall scene conveys a relaxed atmosphere, likely intended for enjoying a drink while reading or relaxing. 77.8% ↓ The image shows a bottle of Mahou Negra beer and a glass of Heineken beer placed on a surface with a bookshelf in the background. The Mahou Negra bottle is dark brown with a black label that prominently displays the brand name "Mahou" in large white letters, followed by "NEGRA" in smaller text. Below the brand name, it states "PREMIUM QUALITY TIPO MUNICH NEGRA DARK BRUNE." The bottle also mentions "DESDE 1890," indicating its founding year. Heineken glass is filled with beer, showcasing a frothy head on top, which is characteristic of the Heineken brand. The background includes a bookshelf with various books, some of which have visible titles and spines, suggesting an indoor setting, possibly a home or a library. (b) Qualitative results for Sample 2. Fig. C.3: Qualitative visualization of token pruning. We compare the generated descriptions from the unpruned baseline model with those from our proposed token pruning method under varying token budgets. Corresponding expressions are highlighted in the same color for clarity.