Paper deep dive
QMoP: Query Guided Mixture-of-Projector for Efficient Visual Token Compression
Zhongyang Li, Yaqian Li, Faming Fang, Rinyoichi Takezoe, Zi-Hao Bo, Cheng Qian, Mo Guang, Guixu Zhang, Kaiwen Long
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/26/2026, 2:25:18 AM
Summary
QMoP (Query Guided Mixture-of-Projector) is a novel framework for efficient visual token compression in Multimodal Large Language Models (MLLMs). It utilizes three collaborative branches—pooling, resampler, and pruning—coordinated by a Query Guided Router (QGR) to adaptively compress visual tokens based on input image and text. The paper also introduces VTCBench, a diagnostic benchmark for evaluating information loss in visual token compression.
Entities (4)
Relation Signals (3)
QGR → controls → Pooling, Resampler, Pruning branches
confidence 95% · QGR dynamically generates weighting coefficients that determine which compression strategies to activate
QMoP → includes → QGR
confidence 95% · In this paper, we first propose Query Guided Mixture-of-Projector (QMoP)... To adaptively coordinate these branches, we introduce the Query Guided Router (QGR)
QMoP → uses → VTCBench
confidence 90% · In this paper, we propose Query-Guided Mixture-of-Projector (QMoP)... In addition, we propose VTCBench, a dedicated benchmark
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal large language models suffer from severe computational and memory bottlenecks, as the number of visual tokens far exceeds that of textual tokens. While recent methods employ projector modules to align and compress visual tokens into text-aligned features, they typically depend on fixed heuristics that limit adaptability across diverse scenarios. In this paper, we first propose Query Guided Mixture-of-Projector (QMoP), a novel and flexible framework that adaptively compresses visual tokens via three collaborative branches: (1) a pooling-based branch for coarse-grained global semantics, (2) a resampler branch for extracting high-level semantic representations, and (3) a pruning-based branch for fine-grained token selection to preserve critical visual detail. To adaptively coordinate these branches, we introduce the Query Guided Router (QGR), which dynamically selects and weights the outputs from different branches based on both visual input and textual queries. A Mixture-of-Experts-style fusion mechanism is designed to aggregate the outputs, harnessing the strengths of each strategy while suppressing noise. To systematically evaluate the effects of Visual Token Compression, we also develop VTCBench, a dedicated benchmark for evaluating the information loss induced by visual token compression. Extensive experiments demonstrate that despite relying on fundamental compression modules, QMoP outperforms strong baselines and delivers significant savings in memory, computation, and inference time.
Tags
Links
- Source: https://arxiv.org/abs/2603.21232v1
- Canonical: https://arxiv.org/abs/2603.21232v1
Trouble viewing inline? Open PDF directly →
Full Text
56,530 characters extracted from source content.
Expand or collapse full text
QMoP: Query Guided Mixture-of-Projector for Efficient Visual Token Compression Zhongyang Li 1 , Yaqian Li 2 , Faming Fang 1* , Rinyoichi Takezoe 2 , Zi-Hao Bo 2 , Cheng Qian 2 , Mo Guang 2 , Guixu Zhang 1 , and Kaiwen Long 2* 1 East China Normal University, 2 Li Auto Inc. Abstract. Multimodal large language models suffer from severe com- putational and memory bottlenecks, as the number of visual tokens far exceeds that of textual tokens. While recent methods employ projec- tor modules to align and compress visual tokens into text-aligned fea- tures, they typically depend on fixed heuristics that limit adaptability across diverse scenarios. In this paper, we first propose Query Guided Mixture-of-Projector (QMoP), a novel and flexible framework that adaptively compresses visual tokens via three collaborative branches: (1) a pooling-based branch for coarse-grained global semantics, (2) a resam- pler branch for extracting high-level semantic representations, and (3) a pruning-based branch for fine-grained token selection to preserve critical visual detail. To adaptively coordinate these branches, we introduce the Query Guided Router (QGR), which dynamically selects and weights the outputs from different branches based on both visual input and tex- tual queries. A Mixture-of-Experts-style fusion mechanism is designed to aggregate the outputs, harnessing the strengths of each strategy while suppressing noise. To systematically evaluate the effects of Visual To- ken Compression, we also develop VTCBench, a dedicated benchmark for evaluating the information loss induced by visual token compression. Extensive experiments demonstrate that despite relying on fundamental compression modules, QMoP outperforms strong baselines and delivers significant savings in memory, computation, and inference time. Keywords: Vision-Language Understanding· Multi-modal LLM· To- ken Compression 1 Introduction Multimodal Large Language Models (MLLMs) [21, 22, 43] have recently shown impressive capabilities in visual-language tasks by integrating vision encoders [27, 39] with pre-trained Large Language Models (LLMs) [1, 3]. In this frame- work, vision encoders convert images into visual tokens, which are then pro- jected into the LLM’s text embedding space and processed together with text * Corresponding authors: Faming Fang and Kaiwen Long. arXiv:2603.21232v1 [cs.CV] 22 Mar 2026 2Authors Suppressed Due to Excessive Length Text input:How many people are there in the room? What are they doing separately? Pooling: There are fourpeople in the room, and they are watching a movie on a television. Resampler: There are threepeople in the room: a man, a woman, and a child. The man is holding a remote control, likely controlling the TV or changing channels. The woman is holding a cell phone, possiblytexting or browsing the internet. The child is sitting on the couch, also watching the television. Pruning:Thereare threepeople in the room: a man, a woman, and a child. They are watching television together, with the man and woman sitting on a couch and the child sitting on the man's lap. Right ✓ Wrong Wrong Text input:Please explain the meme. Pooling: A small dog lies comfortably on a wooden floor with its head on a pillow, likely sleeping. A dining table is visible in the background, creating a cozy, relaxed setting. Resampler: A small brown dog lies on a wooden floor, resting its headand appearing to sleep. The meme humorously reflects the dog’s sleepy pose symbolizing low energy at the week’s start. Pruning: A small dog lies on a wooden floor, resting near a dining table. The scene conveys a relaxed mood,humorously reflecting the desire for rest amid daily life. Right ✓ Wrong Wrong Text input: What does big AI server Pooling: Pizza Resampler: Pizza Pruning: Pizza and subs Wrong Wrong Right ✓ Fig. 1: Qualitative comparisons of three representative visual token compression meth- ods under different scenarios. tokens for reasoning and generation tasks [12,26]. Early approaches employ sim- ple multi-layer perceptrons (MLPs) as projectors, which keep the number of tokens unchanged and only transform visual tokens along the channel dimension into the text embedding space. However, the large number of visual tokens limits further development. Recent approaches have carefully designed sophisticated projectors that not only align visual tokens with textual features but also compress the visual tokens to reduce redundancy and computational cost. These methods can be categorized into three main strategies: pooling-based, resampler-based, and pruning-based approaches. Pooling-based methods [5,9,18] aggregate neighboring spatial tokens using operations such as convolution or max-pooling to produce coarse-grained, compact representations. Resampler-based methods [4, 17] introduce learnable query tokens and employ cross-attention mechanisms to selectively extract key information from the original visual tokens, effectively filtering and compressing high-dimensional features. Pruning-based methods [34, 40] evaluate token im- portance and remove redundant or irrelevant tokens, retaining only the most informative ones for downstream alignment. However, these methods typically rely on specific and fixed compression rules, which limit their adaptability across different tasks and datasets. Fig. 1 presents three typical cases that illustrate the applicable scenarios, strengths, and lim- itations of each approach. Specifically, pooling-based methods are effective at preserving global context but treat all tokens uniformly, potentially discarding critical local details. Resampler-based methods offer greater flexibility in cap- turing high-level semantic representations through learnable queries and cross- attention mechanisms. Nonetheless, their increased complexity tends to intro- QMoP3 Global Understanding (GQA-relation) 54.2 FastVLM (Pruning) LDP-v2(pooling) MQT-LLaVA (Resampler) 51.1 51.2 Local Details (TextVQA) 57.1 56.4 55.2 Abstract Understanding (MME-artwork) 121 116 126 53.8 57.1 124 QMoP (Ours) Fig. 2: Comparison of compression strategies across tasks. Pooling excels on global understanding, pruning leads on local details, and resampling performs best on style and emotion, highlighting task-dependent advantages. duce hallucinations and neglect low-level visual cues. Pruning-based methods focus on retaining key tokens based on predefined criteria, but may discard sub- tle yet important information and frequently disrupt spatial token structure. Beyond qualitative observations, performance variations on existing bench- marks partially reflect the limitations of different compression strategies. As shown in Fig. 2, pooling-based methods excel at capturing holistic scene se- mantics, achieving better results on relational reasoning tasks such as the re- lation subset of GQA. Pruning-based methods perform well on datasets like TextVQA, which require fine-grained textual recognition. Meanwhile, resampler- based methods achieve higher accuracy on abstract understanding tasks, scoring 126 on the artwork subset of MME. While existing benchmarks to some extent reflect the limitations of different compression paradigms, their signals are of- ten indirect and fragmented. To the best of our knowledge, there is currently no benchmark that offers a comprehensive and unified diagnostic evaluation of pooling, resampling, and pruning strategies, which limits systematic analysis and impedes deeper understanding of visual token compression. In this paper, we propose Query-Guided Mixture-of-Projector (QMoP), a flexible visual projection framework comprising three specialized branches: pooling-based, resampler-based, and pruning-based. To adaptively combine these branches, we introduce a Query-Guided Router (QGR) that generates a weight for each projector based on both image and text inputs. Inspired by the Mixture-of-Experts paradigm, we then select and fuse only the two branches with the highest weights, discarding the least relevant one. This selective fusion not only emphasizes the most informative and complementary signals but also suppresses noise from less relevant projectors, enhancing the robustness of the final representation. In addition, we propose VTCBench, a dedicated diagnos- tic benchmark designed to systematically evaluate the information degradation caused by Visual Token Compression (VTC). We conduct extensive analysis on VTCBench, validating our central hypothesis: different compression strategies lead to distinct forms of information loss and exhibit varying effectiveness across different task scenarios. Our contributions are summarized as: 4Authors Suppressed Due to Excessive Length – We conduct an empirical analysis of existing visual token compression strate- gies, identifying their inherent limitations and offering new insights for visual compression. – We introduce QMoP, a Query-Guided Mixture-of-Projector framework that dynamically integrates outputs from three complementary compression branches through a learned, context-aware routing module. – We propose VTCBench, a dedicated benchmark for visual token compres- sion, which enables quantitative comparison across different compression paradigms. 2 Related Work Vision tokens exhibit high spatial redundancy and lower information density, typically outnumbering text tokens. Recent methods have recognized the redun- dancy in vision tokens and proposed various approaches to reduce their quantity. These methods can be divided into two categories: reducing vision tokens during the LLM decoding process and reducing vision tokens before they are fed into the LLM. 2.1 Visual Token Reduction in LLM Reducing vision tokens during the LLM decoding process has emerged as an effective avenue for improving computational efficiency. FastV and VTK [6,20] prune vision tokens based on attention scores at a specific LLM layer, retaining only the most informative tokens for downstream reasoning. To reduce manual tuning cost, FitPrune [35] leverages a small calibration set to derive optimal pruning ratios automatically. SparseVLM [42] further incorporates prior knowl- edge to filter out distracting textual signals before applying attention-based to- ken selection, thereby enhancing pruning accuracy. Beyond single-stage pruning, MustDrop and PDrop [23,33] adopt progressive token dropping across multiple inference stages. AutoPrune [30] dynamically adjusts pruning rates across lay- ers based on task difficulty, while EPIC [32] progressively increases compression through distillation to maintain model performance at higher compression levels. 2.2 Visual Token Reduction in Visual Projector Recently, a growing body of work has explored compressing visual tokens within the projector while aligning visual and textual modalities. These approaches can be broadly categorized into resampler-based, pooling-based, and pruning-based paradigms. Pooling-based projector compression reduces token count by spa- tially aggregating local visual patches. InternVL-1.5 [8] employs pixel unshuffle to downsample tokens by expanding channel capacity. C-Abstractor [5] and LDP [9] conduct learnable pooling via convolution kernels, while TokenPacker [18] intro- duces localized attention to merge spatially adjacent features more adaptively. Resampler-based designs extract salient information through cross-attention. QMoP5 Encoder QMoP What is written on the house? LLM PruningResamplerPooling Query-Guided Router Tokenizer MLP Query Guided Mixture-of-Projector (QMoP) Clip Text Encoder CLS EOS C MLP Softmax Query-Guided Router (QGR) Fig. 3: The architecture of the proposed QMoP. Qwen-VL [4] and BLIP-2 [17] employ learned queries to selectively attend to informative visual tokens, forming a content-aware bottleneck. LLaVA-Mini [41] adopts a lightweight pre-fusion module that injects distilled visual signals di- rectly into language tokens, offering a minimal-token fusion pipeline. Pruning- based strategies instead remove redundant visual tokens after initial feature pro- jection. FasterVLM [40], LLaVA-Prumerge [28], and VisionZip [34] rank tokens by CLS-based attention scores and preserve only the top-scoring subset under a fixed pruning ratio. SCOPE [10] further alleviates semantic loss by incorpo- rating set-coverage constraints derived from token–label relations, encouraging preservation of essential visual regions. 2.3 Multimodal Benchmarks Existing multimodal benchmarks can broadly be categorized into two types. The first type focuses on high-level capabilities, such as reasoning, knowledge grounding, and complex task completion [16, 25, 37, 38]. The second type, in- cluding MME [11], MMStar [7], and MMBench [24], targets more fundamental perceptual and cognitive abilities through broad-spectrum evaluations. While these benchmarks have greatly advanced the evaluation of MLLMs, they are not explicitly designed to assess the impact of visual token compres- sion. In particular, their task formulations and categorizations do not isolate the visual fidelity loss caused by token reduction. Although some sub-tasks (e.g., fine-grained recognition, spatial reasoning) may partially reflect compression ef- fects, the signals are indirect, entangled, and incomplete, making it difficult to diagnose and compare the behavior of different compression strategies in a prin- cipled manner. 3 Method 3.1 Overview The overall framework is illustrated in Fig. 3. Given an input image I img ∈ R H×W×3 , the image is first divided into a series of patches, which are then fed 6Authors Suppressed Due to Excessive Length into the visual encoder to generate visual embeddingsX img ∈R N×C , whereN represents the number of patches andCis the embedding dimension. Next, the visual projector takes these visual embeddings along with the input text I text and adaptively selects a compression strategy to reduce the number of visual tokens. In particular,X img is compressed to produce a smaller set of tokens T img ∈R M×C , whereM < N, and these visual tokens are aligned with the text embedding space. For the text branch, I text is tokenized and encoded to obtain text features T text . The compressed visual tokens T img are then concatenated with T text and fed into the LLM, which generates the corresponding response in an autoregressive manner. 3.2 Query Guided Mixture-of-Projector To compress the visual tokens, we propose a novel Query Guided Mixture- of-Projector (QMoP) that harnesses three specialized compression operators- pooling, resampler, and pruning-to capture diverse aspects of visual information. In addition, QMoP integrates a gating mechanism that dynamically activates a part of these compression branches, enabling the model to select an effective compression strategy for each image-text pair adaptively. To effectively harness the strengths of the pooling, pruning, and resampler branches, we introduce the Query Guided Router (QGR) as shown in Fig. 3. QGR dynamically generates weighting coefficients that determine which com- pression strategies to activate based on the input image and text. For the visual branch, we extract the class token v CLS ∈R C 1 from the penultimate layer of the vision encoder. For the text branch, the input text I text is fed into a pre-trained CLIP text encoder, from which we extract the EOS token t EOS ∈R C 2 . These two tokens are then concatenated along the channel dimension to form a joint feature vector: f = Concat(v CLS , t EOS )∈R (C 1 +C 2 ) .(1) This combined featurefis passed through a two-layer multilayer perceptron (MLP) to generate the raw gating weights. This process can be formulated as: α 1 ,α 2 ,α 3 = softmax(W 2 · σ(W 1 f + b 1 ) + b 2 ),(2) where W 1 ∈R d×(C 1 +C 2 ) andb 1 ∈R d are the weights and bias of the first layer, W 2 ∈R 3×d andb 2 ∈R 3 are those of the second layer,σ(·)denotes a nonlinear activation function. Finally, the output vectorw ∈R 3 is normalized using the softmax function to obtain the gating weights, which are subsequently used to control the contribution of each compression operator. 3.3 Compression Operator To reduce the computational burden while preserving task-relevant semantics, we design three token compression operators (pruning, resampling, and pool- ing), each driven by distinct expert priors. These operators transform the visual QMoP7 features withNtokens into a compact representation ofMtokens. The detailed implementations of each operator are described as follows. The pruning branch assigns scores to visual tokens according to predefined rules to discard redundant ones. Specifically, we introduce two scoring criteria: (1) Importance, which measures how much attention the class token in the last layer of the visual encoder allocates to each visual token, thereby identifying regions that contribute most to semantic understanding; and (2) Relevance, which evaluates how closely each visual token is associated with the textual content by computing its distance to the EOS token in the text representation space. This process can be formulated as: s (i) = λs (i) Importance + (1− λ)s (i) Relevance ,(3) X prune =X (i) img | s (i) ≥ η,∀i∈ [1,N],(4) where X (i) img represents thei-th visual token, X prune ∈R M×C is the refined token set that retains only the most informative tokens, s (i) is the importance score assigned to thei-th visual token according to predefined rules,λis a hyperparameter controlling the balance between the two scoring criteria, which is set to 0.5 in our experiments. ηis a dynamic threshold based on the pruning ratio. Tokens with scores belowηare removed. The resampler branch leverages learnable query tokens with cross-attention mechanisms to selectively extract and aggregate critical visual information from high-dimensional token representations. This process is defined as: X resampler = Softmax QK T √ d V, (5) where X resampler ∈R M×C represents the resampled visual tokens, Q ∈R M×C denotes the learnable query tokens, and K,V ∈R N×C are the key and value representations derived from X img . The pooling branch aggregates spatially adjacent tokens into a compact representation, effectively reducing spatial redundancy while preserving essential visual cues that capture global scene structures and inter-object relations. To maintain semantic consistency and regional integrity during compression, we adopt a local attention mechanism for token pooling. Specifically, the visual features X img ∈R N×C are reshaped into a 2D feature mapX img2d ∈R H×W×C , where N = H×W. We then introduce a learnable 2D query mapQ 2d ∈R h×w×C , whereM = h × wandH = s · h,W = s · w, withsdenoting the spatial downsampling stride. Each query Q 2d [i,j]interacts only with the corresponding local windowX img2d [si : s(i + 1),sj : s(j + 1)], thereby establishing a spatially aligned local aggregation. The local attention process is formulated as: X pool [i,j] = Softmax Q 2d [i,j]K T i,j √ d ! V i,j ,(6) 8Authors Suppressed Due to Excessive Length Global Understanding Spatial and Counting Reasoning and Common sense Style and Emotion Local Details Fig. 4: Overview of VTCBench and the quantitative analysis of compression strategies. where K i,j ,V i,j = φ(X img2d [si : s(i + 1),sj : s(j + 1)]),(7) and φ(·)denotes a linear projection. 3.4 Training and Inference Strategy Following mainstream multimodal frameworks, QMoP adopts a two-stage train- ing scheme. In the first stage, only the projector is trained to map visual features into the textual embedding space. In this stage, the QGR is not used; instead, the outputs from the three expert branches are concatenated and passed through an MLP, allowing all branches to learn the projection to the text space jointly. After pretraining, each branch acquires an initial cross-modal alignment capability. In the second stage, we perform joint finetuning of the projector and the LLM, during which the QG-Router is activated. At the early phase of finetuning, we set a relatively large temperature coefficient τfor the router to encourage smoother routing distributions. As training proceeds, τis gradually decreased to sharpen the expert selection. To enhance diversity and prevent the router from collapsing into sub-optimal local minima, Gumbel noise is introduced during training and gradually annealed over time. For stable gradient propagation, we aggregate the outputs of all branches us- ing weighted summation during training. During inference, only branches whose routing weights exceed a threshold are activated; their weights are then re- normalized, and the final visual representation is obtained by a weighted fusion of the activated branches. 3.5 Benchmark for Visual Token Compression To enable a more comprehensive analysis of the impact caused by discarding visual tokens, we introduce VTCBench, a dedicated benchmark designed to QMoP9 Table 1: Performance comparison with pre-LLM compression methods on LLaVA-1.5- 7B. The best and second-best results are highlighted in bold and underlined . MethodsVQA T POPE VQA v2 M-Vet GQA MME MMB Seed MMStar MMMUAvg. LLaVA-1.5-7B, Vision token = 144 Vanilla [21]58.2 85.9 78.5 31.6 62.0 1511 64.3 66.1 31.934.6100.00% Pixel-Shuffle [8]54.2 87.0 75.6 29.1 60.2 1422 64.5 61.6 31.732.796.14% FasterVLM [40]57.183.5 76.2 32.2 58.0 1440 64.3 62.3 31.334.897.64% C-Abstractor [5]54.1 85.9 75.7 28.2 60.4 1392 63.3 62.3 31.133.495.48% MQT-LLaVA [13]52.6 86.2 76.8 29.8 61.4 1445 64.4 62.0 31.934.897.22% TokenPacker [18]57.0 87.077.933.061.9145465.0 65.132.634.3100.05% LDP-v2 [9]56.4 86.3 77.3 31.7 61.5 1435 65.1 63.9 31.933.998.62% QMoP (ours)57.1 87.7 78.1 33.3 62.0 1456 64.866.0 33.1 35.1100.73% LLaVA-1.5-7B, Vision token = 64 Pixel-Shuffle [8]52.2 85.5 73.2 25.8 58.7 1383 64.4 59.3 30.134.693.46% FasterVLM [40]56.080.4 72.5 28.7 55.0 1342 61.6 57.9 31.834.693.37% C-Abstractor [5]51.3 85.7 73.7 26.1 59.1 1415 61.8 60.0 31.334.193.69% MQT-LLaVA [13]51.6 83.6 75.3 28.9 60.0 145463.5 60.8 30.934.495.41% TokenPacker [18]55.6 86.5 77.2 29.1 61.1 1440 64.4 63.031.333.897.18% LDP-v2 [9]55.1 86.876.1 30.160.1 1414 66.2 62.6 31.534.297.38% QMoP (ours)56.0 87.2 76.430.8 60.21455 64.663.6 31.9 34.698.25% LLaVA-1.5-13B, Vision token = 144 Vanilla [21]63.8 87.9 80.0 35.4 63.3 1531 67.7 68.2 33.136.1100.00% Pixel-Shuffle [8] 57.0 86.9 76.9 33.1 61.2 1483 66.2 64.0 32.134.495.53% FasterVLM [40]58.886.1 77.1 33.6 59.1 1489 66.4 63.8 32.334.995.79% C-Abstractor [5] 53.9 86.3 74.8 28.2 60.4 1391 63.0 61.0 31.335.491.73% TokenPacker [18] 58.8 87.4 78.834.562.3152567.465.433.9 35.098.06% LDP-v2 [9]58.4 87.678.4 34.4 62.0 1533 66.9 65.3 33.235.697.80% QMoP (ours)59.4 88.2 79.4 34.7 62.7 1519 67.4 67.0 33.736.198.86% evaluate functional degradation induced by visual token compression. VTCBench measures compression performance across five key dimensions: 1) Global Un- derstanding (GU): Evaluates the model’s ability to retain the overall scene context, ensuring that essential background and semantic information remain intact after compression. 2)Spatial and Counting (SC): Assesses the model’s capacity to preserve spatial relationships and accurately count objects. 3) Rea- soning and Common sense (RC): Tests whether the compressed represen- tations still support higher-order inference, including logical reasoning and com- monsense knowledge grounded in the visual content. 4) Style and Emotion (SE): Examines the preservation of artistic and affective elements, such as color composition, texture, and emotional expressions conveyed through images. 5) Local Details (LD): Focuses on the preservation of fine-grained visual fea- tures such as text, small objects, or intricate patterns, which are easily lost under aggressive token reduction. The construction of VTCBench follows a multi-stage process. First, we select six existing datasets that cover both natural images and domain-specific scenar- ios ( [7, 11, 14, 15, 26, 29]). Next, we apply a proprietary LLM-based automatic filtering pipeline [1] to remove samples that do not align with our five evaluation dimensions. After an initial coarse filtering using an LLM detector, we perform a manual review to correct any misclassifications. Finally, taking into account the diversity of evaluation dimensions and task difficulty, we manually curate 10Authors Suppressed Due to Excessive Length Table 2: Performance comparison with intra-LLM visual token compression methods on LLaVA-1.5-7B. Method Vision VQA T POPE VQA v2 GQA MMB MMB- OCR Avg. TokenCN Bench LLaVA-1.5-7B57658.2 85.9 78.5 62.0 64.3 58.3 297.0 100.00% FastV [6]19252.5 64.8 67.1 52.7 61.2 57.0 291 89.58% HiRED [2] 19247.4 82.8 74.9 58.7 62.8 54.7 190 89.06% FitPrune [35]19257.4 83.4-60.463.3 56.4-- LLaVA-PruMerge [28]19254.3 71.3 70.6 54.3 59.6 52.9 253 88.92% SparseVLM [42] 19256.1 83.675.6 57.6 62.5 53.7 292 95.79% PDrop [33]19256.1 82.3 75.1 57.1 63.2 56.8 290 96.19% MustDrop [23]19256.5 82.6 76 58.2 62.3 55.8 289 96.26% DART [31]19257.4 82.8 76.760 63.657.029697.98% QMoP (Ours)14457.187.7 78.1 62.0 64.8 57.0 304 100.08% a balanced set of over 1,000 high-quality samples from the filtered pool. These samples are distributed to cover all five dimensions comprehensively. Fig. 4 pro- vides a detailed breakdown of the composition of VTCBench. Please refer to the supplementary material for more details. 4 Experiments 4.1 Implementation Details We chose LLaVA-1.5-7B [21] as our primary baseline to evaluate the effectiveness of QMoP. All experiments are conducted by pretraining on the LAION-C- SBU-558K dataset and performing instruction tuning on a 665K mixed dataset. We employ Vicuna-7B as the LLM and CLIP-ViT-Large-Patch14-336 as the visual encoder. We fix the number of visual tokens to 144 and 64 to ensure fair comparisons with previous methods under similar computational cost. To further assess the scalability and robustness of QMoP, we evaluate it on LLaVA-1.5-13B (larger model size). We evaluate our model on ten representative public vision- understanding benchmarks, as well as on our newly constructed VTCBench. For additional details about these evaluation benchmarks, please refer to the supplementary material for more details. 4.2 Comparison on Mainstream Benchmarks We conduct a comprehensive comparison between QMoP and existing visual token compression methods across several standard benchmarks. Specifically, we include both pre-LLM compression methods: FasterVLM [40], MQT-LLaVA [13], Pixel-Shuffle [8], C-Abstractor [5], TokenPacker [18], and LDP-v2 [9], as well as intra-LLM compression methods, include FastV [6], HiRED [2], FitPrune [35], LLaVA-PruMerge [28], SparseVLM [42], PDrop [33], MustDrop [23] and DART [31]. QMoP11 Table 3: Performance comparison on VTCBench. The results are reported across five dimensions: GU (Global Understanding), SC (Spatial and Counting), RC (Reasoning and Common Sense), SE (Style and Emotion), and LD (Local Details). A, B, and C indicate whether the method is based on resampler, pooling, or pruning, respectively. MethodTypeGUSCRCSELDAvg. LLaVA-1.5-7B [21]-56.1 47.4 65.9 72.0 60.961.0 MQT-LLaVA [13]A54.7 46.1 64.3 74.754.959.5 Pixel-Shuffle [8]B53.6 47.0 61.2 70.0 55.357.9 C-Abstract [5] B53.2 47.8 59.6 70.3 55.257.7 TokenPacker [18]B55.8 47.8 64.774.0 57.560.5 LDP-V2 [9]B56.848.363.9 74.3 56.460.5 FasterVLM [40]C54.7 43.5 59.6 70.7 58.358.0 QMoP MoE57.2 48.6 64.7 75.0 60.661.8 For pre-LLM compression methods, we compress the number of visual to- kens to 144 and 64 to ensure fair comparisons. As shown in Tab. 1, different compression strategies excel on different tasks. Pruning-based methods (e.g., FasterVLM [40]) perform well on TextVQA [29] by retaining fine-grained visual details. Resampler-based methods (e.g., MQT-LLaVA [13]) are better suited for reasoning-heavy tasks like MME [11], due to their semantic abstraction capabil- ities. Pooling-based methods (e.g., TokenPacker [18], LDP-v2 [9]) achieve strong results on VQAv2 [12] and GQA [14] by preserving global context through spa- tial aggregation. In contrast, QMoP dynamically selects the compression strategy based on the input image and text context. This flexibility enables it to achieve top performance across all evaluated benchmarks. For a more comprehensive evaluation, we further compare our approach with intra-LLM compression methods in Tab. 2. These methods focus on reducing visual tokens within the language model’s decoding process rather than at the projector level. As the results show, even under a higher compression ratio (144 vs. 192 tokens), our method achieves consistently superior performance across multiple benchmarks. 4.3 Comparison on VTCBench To comprehensively evaluate the effectiveness of token compression methods, we compare QMoP with existing token compression methods across all five dimen- sions of VTCBench. Table 3 reports the accuracy of different methods, where a higher score indicates better retention of visual information. As a resampler-based projector, MQT-LLaVA [13] exhibits notable ad- vantages in the Reasoning and Common Sense and Style and Emotion dimen- sions, despite its weaker performance in other aspects. As pooling-based meth- ods, TokenPacker [18] and LDPv2 [9] demonstrate strong performance in Global Understanding and Spatial and Counting, benefiting from their ability to retain coarse-grained global context. However, these methods apply uniform compression across all tokens without considering their importance, leading to 12Authors Suppressed Due to Excessive Length Table 4: Ablation study on different projector combinations across VTCBench. The results are reported across five dimensions: GU (Global Understanding), SC (Spatial and Counting), RC (Reasoning and Common Sense), SE (Style and Emotion), and LD (Local Details). A, B, and C denote the three compression methods: resampler, pooling, and pruning, respectively. MethodGUSNRCSELDAvg. LLaVA-1.5-7B56.1 47.4 65.9 72.0 60.961.0 A54.7 46.1 64.374.754.959.5 B56.5 47.861.2 71.7 56.259.2 C55.8 47.4 61.6 74.3 60.560.3 A+B56.8 47.8 63.1 72.3 57.660.1 A+C57.2 47.8 62.7 73.0 59.260.4 B+C 58.6 47.4 61.6 73.0 59.660.9 A+B+C55.8 47.4 60.8 74.3 59.660.2 QMoP57.248.6 64.7 75.0 60.661.8 significant performance drops in Local Details. In contrast, the pruning-based method FasterVLM [40] achieves competitive performance on Local Details, ranking second only to our proposed QMoP. QMoP dynamically selects appropriate compression strategies according to the input, effectively balancing the retention of global context and the preser- vation of fine-grained details. As a result, it achieves consistently strong perfor- mance across all five dimensions of VTCBench. 4.4 Ablation Study Base Projector Combination To investigate the effectiveness of different compression strategy, we conduct an ablation study by combining different com- pression strategy. Specifically, we selectively enable some projectors at a time while disabling the Query Guided Router (QGR) to isolate the impact of each compression strategy. Tab. 4 presents the performance of different projector com- binations across VTCBench. Our key observations are as follows: – Resampler-based compression exhibits superior high-level representation learn- ing and effectively extracts latent semantic features. – Pooling-based compression demonstrates strong performance in global un- derstanding, effectively preserving overall scene context. – Pruning-based compression improves local detail. However, its rigid selection criteria lead to suboptimal performance in capturing high-level semantics and contextual relationships. – Directly merging all three outputs introduces noise, often leading to worse performance. A selective mechanism is thus crucial. The Number of Compressed Visual Tokens To investigate the impact of different compression ratios on model performance, we conduct an ablation study QMoP13 Table 5: Ablation on the number of visual tokens in QMoP. TokensVQA T VQA v2 GQA MMB POPE M-Vet Avg.(%) 57658.278.562.064.385.931.6100.00 14457.178.162.064.887.733.399.49 6456.076.460.264.687.230.896.91 3654.274.158.462.386.828.793.67 1651.073.258.261.386.526.392.34 4 51.170.856.759.682.926.489.03 by varying the number of visual tokens retained after compression. Specifically, we evaluate QMoP with 4, 16, 36, 64, and 144 tokens on several representa- tive multimodal benchmarks, including TextVQA [29], VQAv2 [12], GQA [14], MMBench [24], POPE [19], and M-Vet [36]. As shown in Tab. 5, the baseline LLaVA employs 576 visual tokens, whereas QMoP compresses them to only 144. Remarkably, the overall performance remains nearly unchanged, and on several benchmarks, QMoP even surpasses the baseline. This demonstrates that QMoP effectively removes redundant visual information while preserving the most in- formative semantic cues crucial for cross-modal reasoning. However, as the com- pression ratio further increases (e.g., 64 to 4 tokens), the overall performance gradually declines, suggesting that excessive compression inevitably results in the loss of fine-grained visual details necessary for complex understanding. Complexity Analysis To comprehensively evaluate the efficiency–performance trade-off of QMoP under different compression ratios, we compare it with the uncompressed baseline LLaVA-1.5-7B across five key dimensions: – Computational cost, measured by the total FLOPs required during infer- ence; – Memory efficiency, quantified by the KV cache size used for storing visual features; – Training time, representing the end-to-end time required for model opti- mization under the same config; – Inference time, measured as the total response latency on the VQAv2 [12] test set; Table 6: Complexity comparison between QMoP and baseline (LLaVA-1.5-7B). TokensTFLOPs KVcache Train Inference Performance timetime 5763.82302.0M 102.3h 3h05m100% 1440.9475.5M75.6h2h29m100.73% 640.4233.6M65.5h2h20m97.97% 360.2318.9M61.3h2h20m94.97% 16 0.108.4M58.0h2h18m92.78% 40.032.1M56.0h2h17m91.70% 14Authors Suppressed Due to Excessive Length – Average performance, defined as the mean normalized accuracy across ten representative multimodal benchmarks, including TextVQA [29], POPE [19], VQAv2 [12], M-Vet [36], GQA [14], MME [11], MMBench [24], Seed- IMG [15], MMStar [7], and MMMU [37]. As shown in Tab. 6, when the number of visual tokens is reduced from 576 (baseline) to 144, QMoP achieves substantial efficiency gains, consuming only 25% of the baseline FLOPs and KV cache memory, while maintaining 100.73% of the baseline’s average performance. This suggests that moderate compression effectively removes redundant visual information while retaining the essential se- mantic cues needed for robust cross-modal reasoning. However, as the compres- sion ratio further increases to 36 tokens, the average performance drops sharply, even though the reductions in training and inference time become marginal. This indicates a compression threshold, beyond which excessive token removal leads to significant loss of visual semantics and degraded overall reasoning capability. 4.5 Efficiency breakdown of the projector overhead. Table 7 reports the FLOPs overhead of the projector modules. The FLOPs of QGR depend only on the query length; we estimate its overhead using the aver- age query length of 51 tokens across common benchmarks. For a fair comparison, we adopt the same compression setting as TokenPacker and LDP-V2, reducing the number of visual tokens from 576 to 144, which saves approximately 2.88T FLOPs in the LLM back-end. As shown in Table 7, the additional projector overhead (at the scale of a few GFLOPs) is negligible compared to the compute saved by reducing LLM-side processing, confirming that QMoP’s efficiency gains are dominated by token reduction rather than projector computation. 5 Conclusion In this work, we present QMoP, a novel Query-Guided Mixture-of-Projector framework that adaptively integrates multiple visual token compression strate- gies via a learned routing mechanism, achieving state-of-the-art performance across a wide range of benchmarks. To facilitate a deeper understanding of how compression affects multimodal model behavior, we introduce VTCBench, a dedicated diagnostic benchmark for visual token compression. Experiments demonstrate that QMoP not only surpasses existing compression baselines but also outperforms the uncompressed model in several scenarios, achieving a re- markable balance between efficiency and accuracy. Additionally, we analyze the strengths and weaknesses of each compression strategy, providing insights and directions for future research on visual token compression. Table 7: Efficiency breakdown of the projector overhead. MethodTokenPacker LDP-V2 QGRQMoP FLOPs10.73G12.1G 3.57G 7.29G+3.57G QMoP15 References 1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023) 1, 9 2. Arif, K.H.I., Yoon, J., Nikolopoulos, D.S., Vandierendonck, H., John, D., Ji, B.: Hired: Attention-guided token dropping for efficient inference of high-resolution vision-language models. In: Proceedings of the AAAI Conference on Artificial In- telligence. vol. 39, p. 1773–1781 (2025) 10 3. Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al.: Qwen technical report. arXiv preprint arXiv:2309.16609 (2023) 1 4. Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., Zhou, J.: Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966 1(2), 3 (2023) 2, 5 5. Cha, J., Kang, W., Mun, J., Roh, B.: Honeybee: Locality-enhanced projector for multimodal llm. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 13817–13827 (2024) 2, 4, 9, 10, 11 6. Chen, L., Zhao, H., Liu, T., Bai, S., Lin, J., Zhou, C., Chang, B.: An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In: European Conference on Computer Vision. p. 19–35. Springer (2024) 4, 10 7. Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Wang, J., Qiao, Y., Lin, D., et al.: Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330 (2024) 5, 9, 14, 18, 20, 21 8. Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., et al.: How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences 67(12), 220101 (2024) 4, 9, 10, 11 9. Chu, X., Qiao, L., Zhang, X., Xu, S., Wei, F., Yang, Y., Sun, X., Hu, Y., Lin, X., Zhang, B., et al.: Mobilevlm v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766 (2024) 2, 4, 9, 10, 11 10. Deng, J., Li, W., Zhou, J.T., He, Y.: Scope: Saliency-coverage oriented token prun- ing for efficient multimodel llms. arXiv preprint arXiv:2510.24214 (2025) 5 11. Fu, C., Zhang, Y.F., Yin, S., Li, B., Fang, X., Zhao, S., Duan, H., Sun, X., Liu, Z., Wang, L., et al.: Mme-survey: A comprehensive survey on evaluation of multimodal llms. arXiv preprint arXiv:2411.15296 (2024) 5, 9, 11, 14, 20 12. Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition. p. 6904–6913 (2017) 2, 11, 13, 14, 20 13. Hu, W., Dou, Z.Y., Li, L., Kamath, A., Peng, N., Chang, K.W.: Matryoshka query transformer for large vision-language models. Advances in Neural Information Pro- cessing Systems 37, 50168–50188 (2024) 9, 10, 11 14. Hudson, D.A., Manning, C.D.: Gqa: A new dataset for real-world visual reasoning and compositional question answering. In: Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition. p. 6700–6709 (2019) 9, 11, 13, 14, 18, 20 15. Li, B., Wang, R., Wang, G., Ge, Y., Ge, Y., Shan, Y.: Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125 (2023) 9, 14, 18, 20 16Authors Suppressed Due to Excessive Length 16. Li, J., Pan, K., Ge, Z., Gao, M., Ji, W., Zhang, W., Chua, T.S., Tang, S., Zhang, H., Zhuang, Y.: Fine-tuning multimodal llms to follow zero-shot demonstrative instructions. arXiv preprint arXiv:2308.04152 (2023) 5 17. Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In: International conference on machine learning. p. 19730–19742. PMLR (2023) 2, 5 18. Li, W., Yuan, Y., Liu, J., Tang, D., Wang, S., Qin, J., Zhu, J., Zhang, L.: Tokenpacker: Efficient visual projector for multimodal llm. arXiv preprint arXiv:2407.02392 (2024) 2, 4, 9, 10, 11 19. Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W.X., Wen, J.R.: Evaluating object hal- lucination in large vision-language models. arXiv preprint arXiv:2305.10355 (2023) 13, 14, 20 20. Lin, Z., Lin, M., Lin, L., Ji, R.: Boosting multimodal large language models with vi- sual tokens withdrawal for rapid inference. arXiv preprint arXiv:2405.05803 (2024) 4 21. Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 26296–26306 (2024) 1, 9, 10, 11, 21 22. Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems 36, 34892–34916 (2023) 1 23. Liu, T., Shi, L., Hong, R., Hu, Y., Yin, Q., Zhang, L.: Multi-stage vision to- ken dropping: Towards efficient multimodal large language model. arXiv preprint arXiv:2411.10803 (2024) 4, 10 24. Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al.: Mmbench: Is your multi-modal model an all-around player? In: European conference on computer vision. p. 216–233. Springer (2024) 5, 13, 14, 20 25. Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.W., Galley, M., Gao, J.: Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255 (2023) 5 26. Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.W., Zhu, S.C., Tafjord, O., Clark, P., Kalyan, A.: Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems 35, 2507–2521 (2022) 2, 9, 18 27. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. p. 8748–8763. PmLR (2021) 1 28. Shang, Y., Cai, M., Xu, B., Lee, Y.J., Yan, Y.: Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388 (2024) 5, 10 29. Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., Rohrbach, M.: Towards vqa models that can read. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. p. 8317–8326 (2019) 9, 11, 13, 14, 18, 20 30. Wang, H., Xu, Z., Gao, J., Liu, Y., Hu, W., Wang, K., Zhang, Z., et al.: Each complexity deserves a pruning policy. In: The Thirty-ninth Annual Conference on Neural Information Processing Systems (2025) 4 31. Wen, Z., Gao, Y., Wang, S., Zhang, J., Zhang, Q., Li, W., He, C., Zhang, L.: Stop looking for important tokens in multimodal language models: Duplication matters more. arXiv preprint arXiv:2502.11494 (2025) 10 QMoP17 32. Wen, Z., Wang, S., Zhou, Y., Zhang, J., Zhang, Q., Gao, Y., Chen, Z., Wang, B., Li, W., He, C., et al.: Efficient multi-modal large language models via progressive consistency distillation. arXiv preprint arXiv:2510.00515 (2025) 4 33. Xing, L., Huang, Q., Dong, X., Lu, J., Zhang, P., Zang, Y., Cao, Y., He, C., Wang, J., Wu, F., et al.: Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.17247 (2024) 4, 10 34. Yang, S., Chen, Y., Tian, Z., Wang, C., Li, J., Yu, B., Jia, J.: Visionzip: Longer is better but not necessary in vision language models. arXiv preprint arXiv:2412.04467 (2024) 2, 5 35. Ye, W., Wu, Q., Lin, W., Zhou, Y.: Fit and prune: Fast and training-free visual token pruning for multi-modal large language models. arXiv preprint arXiv:2409.10197 (2024) 4, 10 36. Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., Wang, L.: Mm- vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490 (2023) 13, 14, 20 37. Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al.: Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 9556– 9567 (2024) 5, 14, 20, 21 38. Yue, X., Zheng, T., Ni, Y., Wang, Y., Zhang, K., Tong, S., Sun, Y., Yu, B., Zhang, G., Sun, H., et al.: Mmmu-pro: A more robust multi-discipline multimodal under- standing benchmark. arXiv preprint arXiv:2409.02813 (2024) 5 39. Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language im- age pre-training. In: Proceedings of the IEEE/CVF international conference on computer vision. p. 11975–11986 (2023) 1 40. Zhang, Q., Cheng, A., Lu, M., Zhuo, Z., Wang, M., Cao, J., Guo, S., She, Q., Zhang, S.: [cls] attention is all you need for training-free visual token pruning: Make vlm inference faster. arXiv preprint arXiv:2412.01818 (2024) 2, 5, 9, 10, 11, 12 41. Zhang, S., Fang, Q., Yang, Z., Feng, Y.: Llava-mini: Efficient image and video large multimodal models with one vision token. arXiv preprint arXiv:2501.03895 (2025) 5 42. Zhang, Y., Fan, C.K., Ma, J., Zheng, W., Huang, T., Cheng, K., Gudovskiy, D., Okuno, T., Nakata, Y., Keutzer, K., et al.: Sparsevlm: Visual token sparsifica- tion for efficient vision-language model inference. arXiv preprint arXiv:2410.04417 (2024) 4, 10 43. Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 (2023) 1 18Authors Suppressed Due to Excessive Length Supplementary Material for QMoP 1 Details of VTCBench This section provides a comprehensive description of VTCBench, including data sources, construction pipeline, annotation protocol, evaluation criteria, and dataset statistics. 1.1 Data Sources for MTCBench To build a benchmark that captures diverse forms of visual degradation induced by token compression, we begin by pooling samples from six widely used mul- timodal datasets, covering a broad range of natural images and domain-specific scenarios. The selected datasets include: MME. Provides fine-grained visual perception and multi-skill evaluation, cov- ering recognition, attribute understanding, and cross-modal grounding. TextVQA [29]. Contains images rich in scene text, enabling evaluation of OCR- related reasoning, text grounding, and fine-grained local visual details. GQA [14]. Focuses on compositional reasoning, spatial understanding, and multi- step logical inference. SEED-Image [15]. Offers diverse natural image scenarios designed for compre- hensive vision–language evaluation with emphasis on semantic understanding. MMStar [7]. Covers challenging real-world visual QA with complex semantics, cultural knowledge, and high-level commonsense reasoning. SQA [26]. Contains scientific diagrams and textbook-style images, emphasizing structured reasoning, visual abstraction, and domain-specific interpretation. These six datasets collectively provide broad visual diversity (natural scenes, diagrams, text-heavy images, artistic variations) and rich reasoning types (spa- tial inference, commonsense reasoning, OCR, fine-grained recognition, scientific understanding), forming a solid foundation for constructing MTCBench. 1.2 LLM-based Automatic Sample Filtering To ensure that each sample in MTCBench strictly aligns with one of the five evaluation dimensions, we employ an automatic filtering pipeline using Azure GPT-4.1 as the multimodal judge. The LLM is prompted to examine each (im- age, question, answer) triplet and determine whether it belongs to one of the predefined dimensions: Global Understanding (GU), Spatial and Counting (SC), Reasoning and Common Sense (RC), Style and Emotion (SE), or Local Details (LD). Samples that do not unambiguously match any dimension, or those con- taining invalid or low-quality annotations, are rejected. The exact prompt used for screening is provided below: QMoP19 Prompt for Screening System content: You are an expert multimodal evaluator. Your task is to determine whether a given (image, question, answer) sample is suitable for inclusion in MTCBench, a benchmark designed to evaluate visual token compression. MTCBench contains exactly five evaluation dimensions: 1. Global Understanding (GU): Requires comprehension of the overall scene, major objects, high-level context, or broad semantic information. 2. Spatial and Counting (SC): Involves spatial relationships (e.g., left/right, in front of/behind) or counting the number of objects in the scene. 3. Reasoning and Common Sense (RC): Requires multi-step reasoning, logical inference, causal reasoning, or commonsense grounded in the visual content. 4. Style and Emotion (SE): Concerns artistic style, color composition, emotional expression, aesthetics, or affective interpretation conveyed by the image. 5. Local Details (LD): Requires fine-grained visual cues such as OCR (reading text), small objects, textures, or subtle details that are easily affected by token reduction. Given an image and its QA pair, follow these steps: Step 1: Determine whether the sample clearly belongs to one and only one of the five dimensions above. If it does not match any dimension, label it as "Reject". Step 2: Validate that the question is grounded in the image and is not ambigu- ous, irrelevant, hallucinated, or extremely low quality. If any issue is found, label the sample as "Reject". Step 3: If acceptable, assign exactly one of the dimensions: GU, SC, RC, SE, LD. User content: Please evaluate the following sample: "Image":<Image> "Question":<Question> "Answer":<Answer> Respond strictly in the following JSON format: "decision": "Accept" or "Reject", "dimension": "GU", "SC", "RC", "SE", "LD" or "null", "explanation": "short explanation (1-2 sentences)" 1.3 Human Verification and Balanced Dataset Assembly After automatic pre-classification, all remaining samples undergo human veri- fication to ensure label accuracy and remove misclassified or ambiguous cases. Incorrect assignments are corrected, and invalid samples are removed. Following verification, we assemble a balanced evaluation set of over 1,000 high-quality samples, ensuring that: All five dimensions are equally represented. Visual di- versity within each dimension is preserved. No single source dataset dominates the distribution. 20Authors Suppressed Due to Excessive Length 2 Mainstream Benchmark We conduct experiments on ten widely used vision-understanding benchmarks: GQA [14], VQAv2 [12], TextVQA [29], M-Vet [36], POPE [19], MME [11], MMBench [24], SEED-Image [15], MMMU [37], and MMStar [7]. Below, we pro- vide a detailed description of each benchmark. GQA. [14]. GQA is a large-scale visual reasoning benchmark designed to eval- uate a model’s ability to perform compositional, multi-step reasoning over real- world images. Questions are generated from scene graphs, ensuring precise ground- ing and explicit reasoning paths. Our method is evaluated on the subset of test- dev balanced instructions, which includes 12,578 samples. VQAv2. [12]. VQAv2 is a widely used benchmark for open-ended visual ques- tion answering. Each question is paired with human-provided answers, reducing annotation bias and improving robustness. The benchmark measures a model’s ability to integrate visual recognition with commonsense reasoning across di- verse question types, including counting, attribute recognition, and spatial un- derstanding. TextVQA. [29]. TextVQA evaluates a model’s capability to read and reason over scene text in natural images. We evaluate the model’s performance on the test split, including 5,000 samples, where answering requires recognizing text via OCR and integrating it with visual context. This benchmark stresses the limi- tations of purely visual features and highlights the need for effective text–vision fusion. M-Vet. [36]. M-Vet is a recent benchmark designed to assess holistic mul- timodal reasoning. It comprises carefully curated questions that test capabilities such as object grounding, commonsense reasoning, mathematical understand- ing, instruction following, and multi-hop inference. Answers are evaluated using GPT-based scoring, making M-Vet a challenging stress test for advanced mul- timodal models. POPE. [19]. POPE focuses on evaluating and mitigating object hallucination in multimodal models. It provides structured prompts requiring models to distin- guish between present and absent objects, thereby measuring their tendency to generate hallucinated content. Performance on POPE reflects a model’s visual grounding fidelity. MME. [11]. MME is a large-scale multimodal evaluation suite comprising 14 distinct subtasks that target fine-grained perception and cognition abilities. It includes tests for object recognition, OCR, attribute understanding, spatial rea- soning, and commonsense inference. MME provides a broad and detailed mea- surement of a model’s low-level and high-level perception accuracy. MMBench. [24]. MMBench is a comprehensive multiple-choice benchmark de- signed to measure general-purpose multimodal understanding across a wide range of dimensions, including world knowledge, logical reasoning, fine-grained visual recognition, and instruction following. Its diverse question set enables sys- tematic comparison across different model families. SEED-Image. [15]. Seed-IMG is a multimodal benchmark designed to evalu- ate scene understanding, reasoning, and knowledge grounding from images using QMoP21 Q: Which continent is highlighted? LLAVA-1.5-7B: Australia QMoP: Antarctica Q: How many people are visible in the image? LLAVA-1.5-7B: Two QMoP: Three Q: What is the company name to the left of the coorslogo ? LLAVA-1.5-7B: Coors QMoP: Safeway Wrong Right ✓ Wrong Right ✓ Wrong Right ✓ Fig. 5: Qualitative comparison between LLaVA-1.5-7B and our proposed QMoP. carefully curated question-answer pairs. The dataset emphasizes real-world sce- narios, requiring models to combine perception with factual and commonsense reasoning. MMMU. [37]. MMMU targets expert-level reasoning, spanning more than 30 academic subjects, including medicine, chemistry, engineering, and economics. Questions often require solving multi-step problems by integrating diagrams, charts, and domain-specific knowledge. MMStar. [7]. MMStar focuses on fine-grained multimodal reasoning and vi- sual comprehension across diverse scenarios. It includes tasks involving attribute recognition, spatial reasoning, multi-object interactions, commonsense inference, and open-ended grounding. 3 Case Study To further illustrate the effectiveness of QMoP, we present a series of qualitative comparisons between our method and the baseline LLaVA-1.5-7B [21], as shown in Fig. 5. In the first example, QMoP correctly counts three singers, including one partially hidden in shadow, whereas LLaVA detects only two, demonstrating QMoP’s ability to preserve occluded details and maintain numerical accuracy. In the second example, which requires interpreting a map’s orientation, QMoP 22Authors Suppressed Due to Excessive Length integrates visual parsing with logical inference to answer directional queries ac- curately, while LLaVA’s response is irrelevant, highlighting QMoP’s superior reasoning in structured visual domains. In the final example, QMoP attends to subtle patterns and small object features that LLaVA overlooks, underscoring its sensitivity to fine-grained details. Across these scenarios, QMoP consistently delivers more accurate and context-aware responses by retaining critical visual information, even outperforming the original uncompressed model in certain scenes.