Paper deep dive
Beyond One-Size-Fits-All: Sample-Adaptive Strategy Routing for Vision Token Pruning in MLLMs
Haiji Liang, Pengfei Zhou, Zhenglin Wan, Wei Wang, Yang You, Wangbo Zhao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 9/12/2026, 2:19:07 AM
Summary
The paper introduces VIP-Router, a lightweight, plug-and-play module for Multimodal Large Language Models (MLLMs) that adaptively selects the optimal vision token pruning strategy for each individual input. By leveraging low-cost visual and textual features to predict strategy utility, VIP-Router overcomes the limitations of fixed pruning strategies, achieving significant improvements in accuracy and cost-aware utility across various benchmarks and reduction ratios without modifying the underlying MLLM weights.
Entities (10)
Relation Signals (9)
VIP-Router → evaluatedon → VTC-Bench
confidence 95% · Evaluated on a curated suite of pruning-sensitive visual perception benchmarks, VTC-Bench Group A
VIP-Router → uses → Vision Token Pruning
confidence 95% · VIP-Router implements sample-level pruning strategy selection... compatible with existing pruning methods.
VIP-Router → outperforms → Best Fixed
confidence 92% · VIP-Router consistently outperforms the best fixed strategy baseline across all reduction ratios
Qwen2-VL-7B → isbackbonefor → VIP-Router
confidence 90% · we use Qwen2-VL-7B [28] as the backbone MLLM in our main experiments.
VIP-Router → isbasedon → CLIP
confidence 90% · We instantiate the preview representation using frozen CLIP visual and text encoders.
FastV → iscandidatefor → VIP-Router
confidence 85% · The candidate Full, while the remaining candidates apply existing pruning rules... FastV, VisionZip, PruMerge+, and DART.
VisionZip → iscandidatefor → VIP-Router
confidence 85% · The candidate Full, while the remaining candidates apply existing pruning rules... VisionZip
PruMerge → iscandidatefor → VIP-Router
confidence 85% · The candidate Full, while the remaining candidates apply existing pruning rules... PruMerge+
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal large language models (MLLMs) process hundreds or thousands of visual tokens per image, incurring prohibitive inference costs. While existing vision token pruning methods mitigate this overhead, they implicitly assume that a single fixed pruning strategy can be applied uniformly across all inputs. Our analysis further reveals that ranking pruning methods by average benchmark accuracy conceals substantial sample-wise complementarity: although the average-best strategy excels overall, alternative strategies prove superior on a significant fraction of individual samples. To harness this diversity, we propose VIP-Router, a lightweight VIsion Pruning Router that adaptively selects the pruning strategy predicted to be best suited to each input at a specified pruning level. Conditioned on low-cost visual and textual features, VIP-Router identifies the most suitable candidate strategy while retaining full-token inference as an option when pruning is predicted to be unfavorable. Evaluated on a curated suite of pruning-sensitive visual perception benchmarks, VTC-Bench Group A, VIP-Router consistently outperforms the best fixed strategy baseline across all reduction ratios, achieving a 26.9% relative improvement in average accuracy, and a 22.0% relative increase in average utility after accounting for realized token cost. Crucially, VIP-Router operates in a plug-and-play manner without modifying underlying pruning algorithms or model weights, introducing trainable parameters equivalent to merely 0.017\% of the backbone. Furthermore, VIP-Router proves effective across various MLLM backbones and yields consistent gains on unseen benchmarks, highlighting the potential of sample adaptive routing for visual token pruning.
Tags
Links
- Source: https://arxiv.org/abs/2609.10346v1
- Canonical: https://arxiv.org/abs/2609.10346v1
Trouble viewing inline? Open PDF directly →
Full Text
79,323 characters extracted from source content.
Expand or collapse full text
Beyond One-Size-Fits-All: Sample-Adaptive Strategy Routing for Vision Token Pruning Beyond One-Size-Fits-All: Sample-Adaptive Strategy Routing for Vision Token Pruning in MLLMs Haiji Liang Pengfei Zhou Zhenglin Wan Wei Wang Yang You Wangbo Zhao National University of Singapore InfRec, Cardinal AI Lab The Hong Kong University of Science and Technology *Equal contribution † Project leads ‡Corresponding authors Abstract Multimodal large language models (MLLMs) process hundreds or thousands of visual tokens per image, incurring prohibitive inference costs. While existing vision token pruning methods mitigate this overhead, they implicitly assume that a single fixed pruning strategy can be applied uniformly across all inputs. Our analysis further reveals that ranking pruning methods by average benchmark accuracy conceals substantial sample-wise complementarity: although the average-best strategy excels overall, alternative strategies prove superior on a significant fraction of individual samples. To harness this diversity, we propose VIP-Router, a lightweight VIsion Pruning Router that adaptively selects the pruning strategy predicted to be best suited to each input at a specified pruning level. Conditioned on low-cost visual and textual features, VIP-Router identifies the most suitable candidate strategy while retaining full-token inference as an option when pruning is predicted to be unfavorable. Evaluated on a curated suite of pruning-sensitive visual perception benchmarks, VTC-Bench Group A, VIP-Router consistently outperforms the best fixed strategy baseline across all reduction ratios, achieving a 26.9% relative improvement in average accuracy, and a 22.0% relative increase in average utility after accounting for realized token cost. Crucially, VIP-Router operates in a plug-and-play manner without modifying underlying pruning algorithms or model weights, introducing trainable parameters equivalent to merely 0.017% of the backbone. Furthermore, VIP-Router proves effective across various MLLM backbones and yields consistent gains on unseen benchmarks, highlighting the potential of sample adaptive routing for visual token pruning. 1 Introduction Multimodal large language models (MLLMs) have demonstrated impressive capabilities across a broad spectrum of vision-language tasks [20, 15, 28, 2, 40]. However, the high computational cost of processing visual tokens remains a critical bottleneck for their practical deployment: an image can introduce hundreds or even thousands of tokens into the language model, dominating both memory and latency [12]. To alleviate this overhead, vision token pruning has emerged as a mainstream and effective approach [5, 26, 32, 29]. These methods leverage diverse signals, including attention scores [5], token importance [32], and semantic similarity [26], to identify and drop redundant tokens. Yet, VTC-Bench [19] shows that benchmarks commonly used to evaluate pruning methods were designed for general perception rather than specifically stress-testing vision token pruning. On the pruning-sensitive VTC-Bench Group A (samples answered correctly with full tokens but incorrectly after equivalent-ratio downsampling), we observe that no single pruning strategy dominates across all benchmarks: as shown in Figure 1 (a), different strategies excel on different tasks, indicating a non-trivial degree of complementarity. More critically, beyond this benchmark-level divergence, ranking pruning methods solely by average benchmark performance further obscures the complementarity at the sample level. Among samples for which at least one pruning strategy succeeds, over one-third are cases where Best Fixed (the best fixed pruning method) fails but an alternative strategy answers correctly (Figure 1 (b)). Consequently, the strategy that performs best on average can still be suboptimal for a substantial fraction of individual inputs. In short, average-best does not imply sample-best. Figure 1: (a) The accuracy-best strategy varies across the full VTC-Bench Group A and reduction ratios, with no strategy winning more than half of the 40 benchmark-ratio settings. (b) At each reduction ratio, the stacked bars split Group A samples into those answered correctly by Best Fixed, those missed by Best Fixed but answered correctly by another strategy, and those missed by all strategies. Purple dots report the fraction of samples solvable by at least one strategy that are wrong with Best Fixed, which exceeds one-third at every ratio. (c) In this example, Qwen2-VL yields contrasting predictions across different pruning strategies, illustrating their inherent complementarity. Therefore, rather than designing yet another heuristic to chase higher average performance, we treat this complementarity as an untapped source of improvement and exploit it through sample-adaptive selection. The primary challenge is that the sample-best choice is defined by the inference outcome, which is inherently unavailable beforehand; identifying it exactly would require executing multiple candidate strategies and comparing their respective outputs, thereby defeating the very computational savings that pruning is intended to provide. Yet, intuitively, the suitability of a pruning strategy should depend on the visual and linguistic characteristics of the current input. Our key finding is that low-cost visual and textual features already provide informative signal to predict strategy suitability. To this end, we propose VIsion Pruning Router (VIP-Router), a lightweight, plug-and-play module compatible with existing pruning methods. VIP-Router constructs a query-conditioned preview representation from visual and textual features using text-to-vision cross-attention, and uses a ratio-conditioned utility predictor to score each candidate option. At inference time, only the highest-scoring option is executed, with full-token inference included as an explicit candidate when pruning is predicted to be unfavorable. In our evaluations, VIP-Router consistently outperforms the best fixed pruning strategies across all five reduction ratios on VTC-Bench Group A. Accounting for its realized token cost, VIP-Router improves average utility from 30.8830.88 to 37.6837.68, corresponding to a 22.0%22.0\% relative improvement over Best Fixed; meanwhile, average accuracy increases from 40.35%40.35\% to 51.19%51.19\%, a 26.9%26.9\% relative gain. These gains require no modification to either the candidate pruning methods or the underlying MLLM computation pipeline, while VIP-Router adds only about 1.4M trainable parameters, approximately 0.017%0.017\% of the MLLM parameters. Our main contributions are summarized as follows: • We characterize the sample-level complementarity among existing vision token pruning strategies on pruning-sensitive benchmarks, revealing substantial per-sample suboptimality masked by standard aggregate protocols. • We introduce VIP-Router, a plug-and-play router that uses low-cost features to predict per-strategy utility conditioned on the pruning level while retaining full-token inference as an option when pruning is unfavorable. • Extensive experiments establish consistent cost-aware accuracy gains across pruning levels, applicability across different MLLM backbones, and positive zero-shot transfer to unseen benchmarks. 2 Related Work 2.1 Vision Token Pruning for MLLMs Vision token pruning is a widely studied approach to improving MLLM efficiency by removing or consolidating redundant visual tokens before or within the language model. Existing methods primarily differ in the criterion used to identify which visual information should be retained. Attention-based methods such as FastV and PruMerge exploit attention signals from the language model or visual encoder to identify salient tokens [5, 26], while FitPrune [33] derives pruning configurations by matching attention statistics before and after pruning. Other methods explicitly target redundancy among visual representations: VisionZip [32] preserves dominant tokens while compressing contextual ones, DivPrune [1] promotes diversity among retained tokens, and DART [29] removes tokens according to their duplication with representative pivots. Recent work has also examined failure modes of pruning; for example, RVIS [13] adapts pruning during decoding in response to changes in task-relevant visual information. Despite these differences, existing approaches fundamentally rely on a single fixed pruning criterion for all inputs. VIP-Router instead treats existing pruning strategies as complementary candidates and performs sample-level selection among them. 2.2 Adaptive Inference for Efficient MLLMs Adaptive inference methods adjust MLLM computation according to the input or runtime efficiency requirements rather than using a single fixed inference configuration [38]. One line of work adapts the amount or location of token computation. Dynamic-LLaVA [9] sparsifies visual and linguistic contexts during prefill and decoding; ATP-LLaVA and SparseVLM [34, 36] determine input- or layer-dependent visual-token retention ratios; and AIM [37] combines pre-LLM merging with progressive in-LLM pruning to support different efficiency requirements. F3A [10] further uses question-conditioned signals to adapt visual-token allocation at a specified pruning level. Another line adapts the visual representation or compression pathway. M3 [3] constructs nested representations at multiple visual-token granularities, while MQT [8] produces different numbers of visual tokens with a shared query transformer. AdaLLaVA [31] operates at a broader structural level by dynamically reconfiguring MLLM computation under runtime latency constraints. Most closely related to VIP-Router, QMoP [18] uses a query-guided router over pooling-, resampler-, and pruning-based compression branches. However, QMoP jointly learns specialized compression branches and fuses their output representations, potentially with minor performance degradation, whereas VIP-Router leaves existing pruning operators unchanged and makes a discrete selection among off-the-shelf strategies. Overall, prior adaptive methods primarily vary the amount or location of computation, the granularity of visual representations, or the compression pathway. VIP-Router operates on a different adaptation axis: which pruning criterion to apply to each sample at a specified pruning level. 3 Vision Pruning Router 3.1 Overview VIP-Router implements sample-level pruning strategy selection through three components: frozen visual and textual preview encoders, a text-to-vision cross-attention module, and a ratio-conditioned MLP utility predictor. As illustrated in Figure 2, the lightweight encoders provide global and token-level representations, cross-attention extracts query-conditioned visual information, and the MLP jointly estimates the utility of all candidate strategies at the given pruning ratio. The following sections (3.2-3.4) detail the routing objective, preview representation, and utility predictor. Figure 2: Overview of VIP-Router. Frozen visual and textual preview features, together with query-conditioned cross-modal features and the retained-token ratio, are used to predict the utility of each candidate pruning strategy and Full. 3.2 Problem Formulation We follow the Group A setting of VTC-Bench [19], where full-token inference is correct and the equivalent-ratio image-downsampling baseline fails. Given a sample xi=(Ii,qi)x_i=(I_i,q_i), we select a strategy from =Full,FastV,VisionZip,PruMerge+,DARTS=\Full,FastV,VisionZip,PruMerge+,DART\. The candidate FullFull retains all visual tokens, while the remaining candidates apply existing pruning rules. Let r∈(0,1]r∈(0,1] denote the retained-token ratio used by the compressed candidates; the corresponding reduction ratio is 1−r1-r. Here, r specifies the target pruning operating point rather than a hard per-sample token constraint, since VIP-Router may select FullFull when pruning is unfavorable. For strategy s, let ais∈0,1a_i^s∈\0,1\ indicate whether the MLLM answers sample i correctly. We define Ui(s,r)=ais−c(s,r),c(s,r)=1,s=Full,r,s≠Full.U_i(s,r)=a_i^s-c(s,r), c(s,r)= cases1,&s=Full,\\ r,&s . cases (1) Under Group A, this objective ranks a correctness-preserving compressed strategy above FullFull, and FullFull above an incorrect compressed strategy, thereby providing a cost-aware no-pruning option. The oracle sample-level decision is si⋆=argmaxs∈Ui(s,r).s_i = _s U_i(s,r). (2) At test time, these utilities are unknown. We therefore learn a router gθg_θ that jointly predicts the utility of all candidates: ^i=gθ(Ii,qi,r),s^i=argmaxs∈u^i,s. u_i=g_θ(I_i,q_i,r), s_i= _s u_i,s. (3) 3.3 Cross-Modal Preview Representation We instantiate the preview representation using frozen CLIP visual and text encoders. For image IiI_i, we extract penultimate-layer visual representations, yielding spatial patch features iP_i, whose mean gives the global visual feature iv_i. For question qiq_i, the text encoder produces token representations iT_i and a global text representation it_i. While iv_i and it_i provide low-cost global summaries, direct concatenation captures only coarse cross-modal interaction. We therefore introduce a lightweight text-to-vision cross-attention module. After projecting both modalities to a common dimension, the text tokens query the spatial visual features: i=MHA(Q=iT,K=iP,V=iP).H_i=MHA (Q=T_iW_T,\,K=P_iW_P,\,V=P_iW_P ). (4) We take the cross-attended EOS representation i×=i,EOSh_i^×=H_i,EOS as a compact query-conditioned visual summary. The resulting representation therefore captures global question semantics, global visual content, and query-conditioned visual information. 3.4 Ratio-Conditioned Utility Routing Training a separate router for each retained-token ratio would cause the number of routers to scale linearly with the number of supported ratios. We instead share a single router across all ratios and explicitly condition it on r. The final router representation is i=[i;i;r;logr;i×].z_i= [t_i;v_i;r; r;h_i^× ]. (5) A two-layer MLP maps iz_i to a utility estimate for each candidate strategy. Predicting utility rather than correctness directly aligns the routing decision with the cost-aware objective in Equation 1, providing a common score for ranking candidates across retained-token ratios. Training uses offline per-strategy correctness labels. For each available sample–ratio pair, these labels are converted into the utility target i=[Ui(s1,r),…,Ui(sK,r)]⊤.U_i=[U_i(s_1,r),…,U_i(s_K,r)] . For stable optimization, we standardize each candidate’s utility targets using training-set statistics and regress the resulting targets ~i U_i; predictions are transformed back to raw utility before strategy selection. The complete standardization procedure is provided in Appendix A.3.2. We optimize the router using an element-wise Huber loss: ℒ(θ)=1NK∑i=1N∑k=1KℓHuber(u~^i,k,U~i,k).L(θ)= 1NK _i=1^N _k=1^K _Huber ( u_i,k, U_i,k ). (6) Only the feature projections, cross-attention module, and utility predictor are updated; the preview encoders and MLLM remain frozen. At inference time, VIP-Router predicts candidate utilities at the specified retained-token ratio, selects the highest-utility candidate, and applies the selected strategy within the unchanged MLLM in a single forward pass. 4 Experiments 4.1 Experimental Setup Backbone and candidate pruning strategies. Following VTC-Bench [19], we use Qwen2-VL-7B [28] as the backbone MLLM in our main experiments. We adopt the four pruning implementations released by VTC-Bench: FastV [5], VisionZip [32], PruMerge+ [26], and DART [29]. All implementations and pruning hyperparameters are used without modification. Full-token inference is included as an additional candidate for VIP-Router, as defined in Section 3.2. Data construction and splits. VTC-Bench [19] constructs its pruning-sensitive Group A evaluation set from eight established benchmarks: GQA [11], MMBench and MMBench-CN [21], MME [7], POPE [17], MMStar [6], OCRBench [22], and ChartQA [24]. For each reduction ratio, Group A contains samples that are answered correctly with full-token inference but incorrectly after equivalent-ratio image downsampling. We construct the routing dataset from the correctness labels released by VTC-Bench and partition unique image–question records into training, validation, and test sets with a 70/15/1570/15/15 split. All ratio-specific examples derived from the same image–question record remain in the same split. For each available sample–ratio example in the training set, the correctness vector over candidate strategies is converted into the utility target defined in Equation 1. The same split is used throughout all experiments, and the main results are reported on the held-out test set. Baselines and evaluation metrics. We compare VIP-Router with each of the four pruning strategies applied uniformly to all test samples. We additionally report Best Fixed, which selects a single pruning strategy on the training split by maximizing average utility jointly over all training samples and reduction ratios, and then applies the same strategy to all test samples and ratios. The Per-Sample Oracle selects the highest-utility strategy using ground-truth test labels and serves only as an unattainable upper bound. We report Accuracy, normalized Token Cost, Utility, and Oracle Headroom Recovery (OHR). Accuracy measures the fraction of correctly answered Group A samples. Token Cost is the realized average fraction of retained visual tokens after strategy selection; because VIP-Router may select Full, its realized Token Cost can exceed the retained-token ratio of the compressed candidates. Utility averages the sample-level objective in Equation 1. OHR measures the fraction of the available sample-level utility headroom recovered by VIP-Router: OHR=UVIP-Router−Ubest-fixedUoracle−Ubest-fixed×100%.OHR= U_VIP -Router-U_best -fixedU_oracle-U_best -fixed× 100\%. (7) Higher OHR indicates that VIP-Router recovers a larger fraction of the utility gap between Best Fixed and the Per-Sample Oracle. Router training. Unless otherwise stated, we use the CLIP-based router configuration described in Section 3.3. Only the feature projections, cross-attention layer, and utility prediction head are optimized; the backbone MLLM and preview encoders remain frozen. We train VIP-Router using AdamW with a learning rate of 10−310^-3, a batch size of 256, and at most 50 epochs, and select the checkpoint with the highest validation macro utility. Additional optimization, implementation, hardware, and runtime details are provided in Appendix A.3. Table 1: Cost-aware comparison of Best Fixed, VIP-Router, and the Per-Sample Oracle across reduction ratios on VTC-Bench Group A test set. VIP-Router results are averaged over five random seeds, whereas Best Fixed and the Per-Sample Oracle are deterministic. Best Fixed uses a single pruning strategy selected jointly over all source-training samples and ratios. For Best Fixed, Token Cost equals the exact retained-token ratio r. VIP-Router may select Full, and therefore its realized Token Cost can exceed r. Accuracy and utility are multiplied by 100 for reporting; token cost remains on the [0, 1] scale. The best pruning result at each ratio is bolded. Reduction Ratio Oracle Best Fixed VisionZip VIP-Router OHR Acc. Cost Utility Acc. Cost Utility Acc. Cost Utility 75.00% 100.00 0.351 64.91 57.54 0.250 32.54 66.91 0.287 38.23 17.58% 88.89% 100.00 0.353 64.70 47.94 0.111 36.83 57.65 0.144 43.24 23.00% 93.75% 100.00 0.390 61.02 37.42 0.063 31.17 49.62 0.095 40.08 29.85% 96.00% 100.00 0.460 53.96 33.05 0.040 29.05 44.29 0.073 36.95 31.71% 99.00% 100.00 0.586 41.40 25.82 0.010 24.82 37.48 0.076 29.89 30.58% Table 2: Average accuracy (%) on VTC-Bench Group A test set across visual-token reduction ratios. Full has 100%100\% accuracy by the construction of Group A and is shown only as a reference. VIP-Router results are averaged over five random seeds, whereas fixed-strategy results are deterministic. Method 75.00% 88.89% 93.75% 96.00% 99.00% Full 100.00 100.00 100.00 100.00 100.00 FastV 54.99 35.33 29.22 23.92 18.89 VisionZip 57.54 47.94 37.42 33.05 25.82 PruMerge+ 57.31 43.20 39.73 32.12 21.98 DART 55.68 42.07 36.26 34.91 24.18 VIP-Router 66.91 57.65 49.62 44.29 37.48 4.2 Main Results Cost-aware performance. Table 1 shows that VIP-Router consistently improves over Best Fixed even after accounting for its realized visual-token cost. Averaged across the five reduction ratios, utility increases from 30.88 to 37.68, corresponding to a 22.0% relative improvement. The gain holds at every reduction ratio, ranging from 5.07 to 8.91 utility points over Best Fixed. As a stronger ratio-wise reference, the strongest fixed method itself varies with the pruning level: PruMerge+ achieves the highest fixed-strategy utility at 93.75%93.75\% reduction and DART at 96.00%96.00\%, while VisionZip leads at the remaining ratios. VIP-Router nevertheless exceeds this fixed-method envelope at every ratio. Relative to the Per-Sample Oracle, VIP-Router recovers 17.6–31.7% of the available utility headroom across the five ratios, with a macro-average OHR of 26.5%. The improvement is also not primarily attributable to full-token inference. Although VIP-Router may select Full when pruning is predicted to be unfavorable, its realized Token Cost is explicitly charged in utility, and Full accounts for only 3.48–6.66% of routing decisions across the five ratios (Figure 5 in Appendix A). Moreover, a worst-case counterfactual bounds the contribution of the full-token option to at most 0.41 utility points on average, or 6.1% of the observed utility gain over Best Fixed (Remark 1). Thus, the improvement arises predominantly from sample-adaptive selection among pruning strategies rather than from additional use of full-token inference. Accuracy across reduction ratios and benchmarks. VIP-Router also achieves the highest average accuracy at every reduction ratio, as shown in Table 2. Compared with Best Fixed, it improves accuracy by at least 9.37 points at each ratio and increases the average accuracy across the five ratios from 40.35% to 51.19%, corresponding to a 26.9% relative improvement. The relative gain becomes particularly pronounced under aggressive pruning, reaching 45.2% at 99%99\% token reduction. The improvement is also broadly distributed across tasks: VIP-Router achieves the best pruning result in 30 of the 40 benchmark–ratio combinations (75.0%); the full per-benchmark breakdown is provided in Appendix A.5.1. 4.3 Efficiency Analysis (a) Accuracy versus realized token cost. (b) Accuracy versus total inference FLOPs. Figure 3: Accuracy–efficiency comparison between fixed pruning strategies and VIP-Router. Point labels indicate prescribed reduction ratios. The total inference FLOPs include both routing overhead and the computation of the selected pruning strategy. Table 3: Parameter overhead and performance of the default VIP-Router and its native-feature variants. Extra frozen parameters include modules loaded exclusively for routing beyond Qwen2-VL; reused native MLLM components are excluded. Percentages are relative to Qwen2-VL parameters. All variants are single-run results. Vision Features Text Features Cross Attn. # Extra Frozen Params # Trainable Params Acc. Utility CLIP ViT-B/32 CLIP Text Yes 151.28 M (1.82%) 1.38 M (0.017%) 51.51 38.09 Native ViT CLIP Text Yes 63.43 M (0.77%) 1.78 M (0.021%) 51.57 37.98 Native ViT Native LM Yes 0.00 M (0.00%) 4.13 M (0.050%) 51.86 37.37 Native ViT CLIP Text No 63.43 M (0.77%) 0.92 M (0.011%) 50.47 37.19 Native ViT Native LM No 0.00 M (0.00%) 2.49 M (0.030%) 52.21 37.62 Accuracy–efficiency trade-off. Figure 3 compares VIP-Router with fixed pruning strategies from two complementary perspectives: prescribed pruning level and realized computation. As shown in Figure 3(a), VIP-Router achieves higher accuracy than the fixed strategies across most of the evaluated reduction range. Figure 3(b) further accounts for the FLOPs of both routing and the selected pruning method; VIP-Router remains above the fixed-strategy curves over most of the evaluated compute range beyond approximately 4.54.5 TFLOPs. Together, these results show that the accuracy gains of VIP-Router persist after accounting for its additional routing computation. The most aggressive 99%99\% reduction setting behaves differently. At this pruning level, VIP-Router more frequently selects the full-token option for samples predicted to be unfavorable for pruning, increasing its realized Token Cost and FLOPs. Consequently, the corresponding point becomes less competitive under matched-compute comparison, even though VIP-Router still substantially improves accuracy over fixed strategies evaluated at the same prescribed reduction ratio. Router parameter overhead. The default VIP-Router achieves the highest utility while introducing only 1.38M trainable parameters (0.017% of the 8.291B Qwen2-VL [28]). Although this configuration uses frozen CLIP encoders [25] for preview features, these external parameters are not intrinsic to the routing formulation: reusing native MLLM representations progressively reduces the additional frozen footprint and can eliminate it entirely. In particular, the fully native variant without cross-attention requires no external frozen parameters and only 2.49M trainable parameters, while its utility is within 0.5 points of the default configuration. Thus, VIP-Router can trade external preview capacity for native feature reuse while keeping its trainable parameter overhead at or below 0.05% of the backbone. 4.4 Cross-Backbone Evaluation and Benchmark Transfer Table 4: Evaluation of VIP-Router across MLLM backbones and zero-shot transfer to unseen benchmarks. In panel (a), the pruning strategy selected by Best Fixed is shown after each MLLM name. The constant Best Fixed Cost of 0.0950.095 is the macro-average retained-token ratio over the five evaluated pruning levels (r¯=0.0947 r=0.0947). Parenthesized percentages report OHR, which is also macro-averaged across the five reduction ratios. Method Acc. Cost Utility (OHR) LLaVA-1.5-7B (PruMerge+) Oracle 100.00 0.458 54.23 Best Fixed 43.38 0.095 33.91 VIP-Router 51.82 0.128 38.97 (26.15%) LLaVA-OneVision-7B (DART) Oracle 100.00 0.431 56.95 Best Fixed 45.22 0.095 35.74 VIP-Router 48.24 0.117 36.56 (2.93%) InternVL3-8B (DART) Oracle 100.00 0.385 61.46 Best Fixed 50.93 0.095 41.46 VIP-Router 53.81 0.118 42.01 (4.17%) Qwen2.5-VL-7B (DART) Oracle 100.00 0.450 54.96 Best Fixed 40.01 0.095 30.54 VIP-Router 48.75 0.134 35.40 (20.52%) (a) Evaluation across MLLM backbones. Method Acc. Cost Utility (OHR) ScienceQA-IMG Oracle 100.00 0.518 48.16 Best Fixed 29.59 0.095 20.11 VIP-Router 36.18 0.106 25.54 (19.47%) MMMU Oracle 100.00 0.533 46.71 Best Fixed 31.99 0.095 22.52 VIP-Router 35.22 0.106 24.59 (6.53%) RealWorldQA Oracle 100.00 0.373 62.67 Best Fixed 38.41 0.095 28.94 VIP-Router 43.64 0.096 34.08 (13.62%) SEED-Bench Image Oracle 100.00 0.445 55.48 Best Fixed 39.27 0.095 29.80 VIP-Router 40.21 0.103 29.87 (0.47%) (b) Transfer to unseen benchmarks. Evaluation across MLLM backbones. We consider four representative MLLMs from different model families: LLaVA-1.5-7B, LLaVA-OneVision-7B, InternVL3-8B, and Qwen2.5-VL-7B [20, 15, 40, 2]. For each backbone, we run full-token, equivalent-downsampling, and per-strategy inference and reconstruct VTC-Bench Group A following the same protocol described in Section A.3.1. We then train a separate VIP-Router instance using the corresponding backbone-specific source-training labels and evaluate it on the held-out test split. Best Fixed is selected from the same source-training split and held fixed across all reduction ratios. As shown in Table 4(a), VIP-Router improves utility over Best Fixed on all four backbones, with gains ranging from 0.55 to 5.06 points. The improvement varies substantially across models, indicating that the routing formulation is broadly applicable while the magnitude of sample-level pruning heterogeneity is backbone dependent. Zero-shot transfer to unseen benchmarks. We further evaluate ScienceQA-IMG, MMMU, RealWorldQA, and SEED-Bench Image [23, 35, 30, 16]. For each target benchmark, the test set is reconstructed following the same protocol used for VTC-Bench Group A. The VIP-Router trained on the VTC-Bench Group A training set, together with the corresponding source-selected Best Fixed strategy, is then transferred directly to the reconstructed target Group A without target-specific router training or strategy selection. As shown in Table 4(b), VIP-Router achieves positive utility gains on all four unseen benchmarks, although the transfer strength varies considerably across tasks. The largest gains occur on ScienceQA-IMG and RealWorldQA, whereas the improvement on SEED-Bench Image is marginal. Overall, these results show that pruning-strategy preferences learned from the source benchmarks can transfer to unseen tasks, but the degree of transfer remains sensitive to domain shift. 4.5 Ablation Studies Table 5: Staged ablation of the VIP-Router design. All ablation results are reported from single runs, and comparisons are made within each stage. The best utility within each stage is bolded. # Routers r Input TglobalT_global VglobalV_global Spatial Info Cross Attn. Pooling Acc. Cost Utility A. Global feature inputs 5 – ✓ – – – – 47.99 0.122 35.81 5 – – ✓ – – – 51.26 0.132 38.02 5 – ✓ ✓ – – – 52.01 0.139 38.07 B. Cross-ratio parameter sharing 5 – ✓ ✓ – – – 52.01 0.139 38.07 1 ✓ ✓ ✓ – – – 49.59 0.126 36.96 C. Query-conditioned spatial interaction 1 ✓ ✓ ✓ – – Mean 48.29 0.119 36.41 1 ✓ – ✓ ✓ – Learnable 50.82 0.136 37.23 1 ✓ ✓ ✓ ✓ – Learnable 49.81 0.124 37.45 1 ✓ ✓ ✓ ✓ ✓ Masked mean 50.39 0.125 37.94 1 ✓ ✓ ✓ ✓ ✓ EOS 51.51 0.134 38.09 Global feature inputs. As shown in Table 5, global visual features provide a substantially stronger routing signal than global text features alone. Combining the two yields the best performance in this stage, but the gain over using VglobalV_global alone is marginal, suggesting that coarse global text features contribute little additional information through direct concatenation. Cross-ratio parameter sharing. Replacing five ratio-specific routers with a single ratio-conditioned router reduces utility by only 1.11 points, while consolidating all pruning levels into one shared model. This trade-off motivates the shared-router design used in VIP-Router, with the next stage focusing on recovering the performance loss through richer cross-modal interaction. Query-conditioned cross-modal interaction. Spatially preserving the visual representation substantially improves the shared router over global averaging, whereas adding the global text feature alone provides only a modest additional gain. Explicit text-to-vision cross-attention further improves routing performance, with the EOS-based representation achieving the best result. The richer cross-modal representation more than compensates for the loss from parameter sharing. It achieves parity with five separate ratio-specific routers (38.09 vs. 38.07) while reducing the model overhead from five Stage-A models to just one, supporting the use of query-conditioned spatial interaction in the shared router. 5 Conclusion In this paper, we introduced VIP-Router, a lightweight framework that reframes vision token pruning from applying a fixed criterion uniformly across inputs to selecting the most suitable pruning strategy for each sample. Across five reduction ratios on the pruning-sensitive VTC-Bench Group A setting, VIP-Router consistently improves both accuracy and utility over Best Fixed, remains effective across different MLLM backbones, and transfers zero-shot to unseen benchmarks. Beyond these empirical gains, our results show that pruning-strategy choice itself constitutes an important axis of adaptive inference. AI Use Statement In this work, we used generative AI tools to assist with software implementation and code review, data processing and table preparation, refinement of research ideas and experimental methodology, mathematical proofs production, and editing the manuscript for clarity and readability. Generative AI was not used to generate datasets; other required-disclosure tasks were not applicable to this work. All AI-assisted code was manually reviewed and tested by the authors. AI-assisted data processing, tables, methodological suggestions, and manuscript edits were likewise checked against the underlying implementations, experimental results, and source materials. The authors made all final research and writing decisions and take full responsibility for the content of this work, including all text, claims, code, results, and artifacts produced with the assistance of generative AI. Reproducibility Statement We provide the information required to reproduce VIP-Router and all reported experiments in the main paper, appendix, and supplementary materials. The routing objective and model architecture are specified in Section 3, with pseudocode provided in Appendix A.2. Dataset construction, record-level train/validation/test splitting, baseline selection, and evaluation metrics are described in Section 4.1 and Appendix A.3. Appendix A.3 further reports the complete training configuration, random seeds, software environment, hardware, and implementation details. Additional encoder configurations and detailed experimental results are provided in Appendices A.4 and A.5. Limitations Our work has two main limitations: (i) All empirical results in this paper are currently restricted to the Group A setting. Consequently, the router’s scaling behavior and performance on unfiltered benchmarks that include pruning-insensitive samples have not yet been comprehensively evaluated. (i) We have also conducted preliminary experiments to explore a stricter setting: whether a routing policy learned on a source MLLM can generalize directly to unseen target architectures without any target-specific adaptation. In this exploratory cross-backbone setup, a router trained solely on a source model is deployed on target backbones with all learned components and decision parameters held fixed. Initial observations indicate that direct zero-shot policy transfer generally fails to preserve the advantages of backbone-specific training, often falling behind static baseline policies across target architectures. These findings suggest that while the proposed routing formulation is broadly applicable across diverse MLLM families, the optimal sample-level pruning preferences remain largely backbone-dependent. References [1] S. R. Alvar, G. Singh, M. Akbari, and Y. Zhang (2025) DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 9392–9401. Cited by: §2.1. [2] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin (2025) Qwen2.5-VL Technical Report. arXiv. External Links: 2502.13923, Document Cited by: §1, §4.4. [3] M. Cai, J. Yang, J. Gao, and Y. J. Lee (2024) Matryoshka Multimodal Models. In The Thirteenth International Conference on Learning Representations, Cited by: §2.2. [4] B. Cao, K. Chen, K. Maninis, K. Chen, A. Karpur, Y. Xia, S. Dua, T. Dabral, G. Han, B. Han, J. Ainslie, A. Bewley, M. Jacob, R. Wagner, W. Ramos, K. Choromanski, M. Seyedhosseini, H. Zhou, and A. Araujo (2026) TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 29325–29335. Cited by: §A.4. [5] L. Chen, H. Zhao, T. Liu, S. Bai, J. Lin, C. Zhou, and B. Chang (2025) An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models. In Computer Vision – ECCV 2024, A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, and G. Varol (Eds.), Cham, p. 19–35. External Links: Document, ISBN 978-3-031-73004-7 Cited by: §1, §2.1, §4.1. [6] L. Chen, J. Li, X. Dong, P. Zhang, Y. Zang, Z. Chen, H. Duan, J. Wang, Y. Qiao, D. Lin, and F. Zhao (2024) Are We on the Right Way for Evaluating Large Vision-Language Models?. In Advances in Neural Information Processing Systems, Vol. 37, p. 27056–27087. External Links: Document Cited by: §4.1. [7] C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun, Y. Wu, and R. Ji (2024) MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv. Note: Comment: Project page: https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models External Links: 2306.13394, Document Cited by: §4.1. [8] W. Hu, Z. Dou, L. H. Li, A. Kamath, N. Peng, and K. Chang (2024) Matryoshka Query Transformer for Large Vision-Language Models. In Advances in Neural Information Processing Systems, Vol. 37, p. 50168–50188. External Links: Document Cited by: §2.2. [9] W. Huang, Z. Zhai, Y. Shen, S. Cao, F. Zhao, X. Xu, Z. Ye, and S. Lin (2025) Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification. International Conference on Learning Representations 2025, p. 69927–69955. Cited by: §2.2. [10] Y. Huang, Y. Zhang, Z. Jia, X. Yang, J. Huang, Z. Wang, S. Feng, D. Wang, Y. Zhang, and Y. Liu (2026) How Many Visual Tokens Do Multimodal Language Models Need? Scaling Visual Token Pruning with F^3A. arXiv. External Links: 2605.16359, Document Cited by: §2.2. [11] D. A. Hudson and C. D. Manning (2019) GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 6700–6709. Cited by: §4.1. [12] Y. Jin, J. Li, T. Gu, Y. Liu, B. Zhao, J. Lai, Z. Gan, Y. Wang, C. Wang, X. Tan, and L. Ma (2025) Efficient multimodal large language models: a survey. Visual Intelligence 3 (1), p. 27. External Links: ISSN 2731-9008, Document Cited by: §1. [13] J. Kim, K. Kim, W. Kim, B. Lee, and C. Park (2026) Why and When Visual Token Pruning Fails? A Study on Relevant Visual Information Shift in MLLMs Decoding. arXiv. Note: Comment: Preprint, Project : https://ptkjw1997.github.io/DSTP-page/ External Links: 2604.12358, Document Cited by: §2.1. [14] M. Leshno, V. Ya. Lin, A. Pinkus, and S. Schocken (1993) Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Networks 6 (6), p. 861–867. External Links: ISSN 0893-6080, Document Cited by: §A.1. [15] B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, P. Zhang, Y. Li, Z. Liu, and C. Li (2024) LLaVA-OneVision: Easy Visual Task Transfer. Transactions on Machine Learning Research. External Links: ISSN 2835-8856 Cited by: §1, §4.4. [16] B. Li, Y. Ge, Y. Ge, G. Wang, R. Wang, R. Zhang, and Y. Shan (2024) SEED-Bench: Benchmarking Multimodal Large Language Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 13299–13308. Cited by: §4.4. [17] Y. Li, Y. Du, K. Zhou, J. Wang, X. Zhao, and J. Wen (2023) Evaluating Object Hallucination in Large Vision-Language Models. In The 2023 Conference on Empirical Methods in Natural Language Processing, Cited by: §4.1. [18] Z. Li, Y. Li, F. Fang, R. Takezoe, Z. Bo, C. Qian, M. Guang, G. Zhang, and K. Long (2026) QMoP: Query Guided Mixture-of-Projector for Efficient Visual Token Compression. arXiv. External Links: 2603.21232, Document Cited by: §2.2. [19] C. Liao, W. Wang, Z. Wen, X. Zheng, Y. Wang, H. He, Y. Lyu, L. Jiang, X. Zou, Y. Fu, B. Ren, L. Zhang, and X. Hu (2026) Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens (Eds.), San Diego, California, United States, p. 4236–4253. External Links: Document, ISBN 979-8-89176-390-6 Cited by: §A.3.1, §1, §3.2, §4.1, §4.1. [20] H. Liu, C. Li, Y. Li, and Y. J. Lee (2024) Improved Baselines with Visual Instruction Tuning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 26286–26296. External Links: ISSN 2575-7075, Document Cited by: §1, §4.4. [21] Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, K. Chen, and D. Lin (2025) MMBench: Is Your Multi-modal Model an All-Around Player?. In Computer Vision – ECCV 2024, A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, and G. Varol (Eds.), Cham, p. 216–233. External Links: Document, ISBN 978-3-031-72658-3 Cited by: §4.1. [22] Y. Liu, Z. Li, M. Huang, B. Yang, W. Yu, C. Li, X. Yin, C. Liu, L. Jin, and X. Bai (2024) OCRBench: on the hidden mystery of OCR in large multimodal models. Science China Information Sciences 67 (12), p. 220102. External Links: ISSN 1869-1919, Document Cited by: §4.1. [23] P. Lu, S. Mishra, T. Xia, L. Qiu, K. Chang, S. Zhu, O. Tafjord, P. Clark, and A. Kalyan (2022) Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering. In Advances in Neural Information Processing Systems, Cited by: §4.4. [24] A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque (2022) ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, S. Muresan, P. Nakov, and A. Villavicencio (Eds.), Dublin, Ireland, p. 2263–2279. External Links: Document Cited by: §4.1. [25] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever (2021) Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning, Note: [TLDR] It is demonstrated that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. Cited by: §A.4, §4.3. [26] Y. Shang, M. Cai, B. Xu, Y. J. Lee, and Y. Yan (2025) LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 22857–22867. Cited by: §1, §2.1, §4.1. [27] M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdulmohsin, N. Parthasarathy, T. Evans, L. Beyer, Y. Xia, B. Mustafa, O. Hénaff, J. Harmsen, A. Steiner, and X. Zhai (2025) SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features. arXiv. Note: Comment: Model checkpoints are available at https://github.com/google-research/big_vision/tree/main/big_vision/configs/proj/image_text/README_siglip2.md External Links: 2502.14786, Document Cited by: §A.4. [28] P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Lin (2024) Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution. arXiv. Note: Comment: Code is available at https://github.com/QwenLM/Qwen2-VL. arXiv admin note: text overlap with arXiv:2408.15262 by other authors External Links: 2409.12191, Document Cited by: §1, §4.1, §4.3. [29] Z. Wen, Y. Gao, S. Wang, J. Zhang, Q. Zhang, W. Li, C. He, and L. Zhang (2025) Stop Looking for “Important Tokens” in Multimodal Language Models: Duplication Matters More. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 9961–9980. External Links: Document, ISBN 979-8-89176-332-6 Cited by: §1, §2.1, §4.1. [30] xAI (2024) Grok-1.5 vision preview. Note: https://x.ai/blog/grok-1.5v Cited by: §4.4. [31] Z. Xu, K. D. Nguyen, P. Mukherjee, S. Bagchi, S. Chaterji, Y. Liang, and Y. Li (2025) Learning to Inference Adaptively for Multimodal Large Language Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 3552–3563. Cited by: §2.2. [32] S. Yang, Y. Chen, Z. Tian, C. Wang, J. Li, B. Yu, and J. Jia (2025) VisionZip: Longer is Better but Not Necessary in Vision Language Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 19792–19802. Cited by: §1, §2.1, §4.1. [33] W. Ye, Q. Wu, W. Lin, and Y. Zhou (2025) Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence 39 (21), p. 22128–22136. External Links: ISSN 2374-3468, Document Cited by: §2.1. [34] X. Ye, Y. Gan, Y. Ge, X. Zhang, and Y. Tang (2025) ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 24972–24982. Cited by: §2.2. [35] X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, C. Wei, B. Yu, R. Yuan, R. Sun, M. Yin, B. Zheng, Z. Yang, Y. Liu, W. Huang, H. Sun, Y. Su, and W. Chen (2024) MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 9556–9567. Cited by: §4.4. [36] Y. Zhang, C. Fan, J. Ma, W. Zheng, T. Huang, K. Cheng, D. A. Gudovskiy, T. Okuno, Y. Nakata, K. Keutzer, and S. Zhang (2025) SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference. In Proceedings of the 42nd International Conference on Machine Learning, p. 74840–74857. External Links: ISSN 2640-3498 Cited by: §2.2. [37] Y. Zhong, Z. Liu, Y. Li, and L. Wang (2025) AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 20180–20192. Cited by: §2.2. [38] P. Zhou, Z. Tang, Y. Ma, J. Tang, Y. Han, Z. Wan, F. Meng, W. Wang, B. Zhuang, W. Zhao, et al. (2026) Agent-as-a-router: agentic model routing for coding tasks. arXiv preprint arXiv:2606.22902. Cited by: §2.2. [39] C. Zhu, S. Suri, C. Jose, M. Oquab, M. Szafraniec, W. Wen, Y. Xiong, P. Labatut, P. Bojanowski, R. Krishnamoorthi, and V. Chandra (2026) Efficient Universal Perception Encoder. arXiv. Note: Comment: Code: https://github.com/facebookresearch/EUPE; Model: https://huggingface.co/collections/facebook/eupe External Links: 2603.22387, Document Cited by: §A.4. [40] J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, H. Tian, Y. Duan, W. Su, J. Shao, Z. Gao, E. Cui, X. Wang, Y. Cao, Y. Liu, X. Wei, H. Zhang, H. Wang, W. Xu, H. Li, J. Wang, N. Deng, S. Li, Y. He, T. Jiang, J. Luo, Y. Wang, C. He, B. Shi, X. Zhang, W. Shao, J. He, Y. Xiong, W. Qu, P. Sun, P. Jiao, H. Lv, L. Wu, K. Zhang, H. Deng, J. Ge, K. Chen, L. Wang, M. Dou, L. Lu, X. Zhu, T. Lu, D. Lin, Y. Qiao, J. Dai, and W. Wang (2025) InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models. arXiv. Note: Comment: Technical Report External Links: 2504.10479, Document Cited by: §1, §4.4. Appendix A Appendix A.1 Theoretical Motivation: Utility Prediction from a Compact Representation Setup. Treat a benchmark sample x=(I,q)x=(I,q) together with a retained-token ratio r as a random draw from the data distribution over (x,r)(x,r); we drop the sample index i used in Section 3. The retained-token ratios lie on the finite grid ℛ=0.25, 0.1111, 0.0625, 0.04, 0.01.R=\0.25,\,0.1111,\,0.0625,\,0.04,\,0.01\. We write the candidate set as =Full∪pS=\Full\ _p, where pS_p contains the pruning strategies. Recall the token cost C(s,r)C(s,r) and per-sample utility U(s,r)=A(s,r)−C(s,r),A(s,r)∈0,1.U(s,r)=A(s,r)-C(s,r), A(s,r)∈\0,1\. More generally, this objective belongs to the family A(s,r)−λC(s,r)A(s,r)-λ C(s,r). We use λ=1λ=1 throughout the paper because, under Group A, it induces the desired ordering between successful pruning, full-token inference, and failed pruning, as detailed below. Structure of the oracle. Under Group A, full-token inference is correct by construction, so U(Full,r)=0.U(Full,r)=0. All pruning strategies have the same token cost r: a correct pruning strategy has utility 1−r1-r, whereas an incorrect one has utility −r-r. The oracle therefore has a simple structure: (i) if at least one pruning strategy is correct, every correct pruning strategy is optimal with utility 1−r1-r; (i) if all pruning strategies fail, FullFull is uniquely optimal with utility 00. Thus, the per-sample optimum may contain multiple equivalent pruning strategies, and the router only needs to select one zero-regret member of this set. What the router must learn. Let (x,r)z(x,r) denote the preview representation in Equation 5. For a pruning strategy s∈ps _p, the Bayes-optimal squared-loss utility predictor is u¯s()=[U(s,r)∣]=p¯s()−r,p¯s()=Pr(A(s,r)=1∣). u_s(z)=E[U(s,r) ]= p_s(z)-r, p_s(z)= (A(s,r)=1 ). (8) For FullFull, Group A gives u¯Full=0 u_Full=0 identically. Utility prediction therefore reduces, in principle, to estimating the conditional success probabilities of the pruning strategies and comparing them against a known cost threshold. At the Bayes optimum, s⋆=Full⟺maxs∈p¯s()<r.s _z=Full _s _p p_s(z)<r. (9) Thus, full-token inference is preferred only when every pruning strategy is predicted to preserve correctness with probability below the retained-token ratio. Our implementation predicts all candidate utilities using a common output head for simplicity, but the Full target remains constant under Group A. Error decomposition. Let =[U(s1,r),…,U(sK,r)]U=[U(s_1,r),…,U(s_K,r)] denote the random utility vector and ¯()=[∣] u(z)=E[U ]. For any square-integrable predictor g, the orthogonality of the L2L^2 projection onto σ()σ(z) yields ‖g()−‖22=‖¯()−‖22⏟δrep2(representation gap)+‖g()−¯()‖22⏟δest2(estimation error).E \|g(z)-U \|_2^2= E \| u(z)-U \|_2^2_ _rep^2\ (representation gap)\;+\; E \|g(z)- u(z) \|_2^2_ _est^2\ (estimation error). (10) The representation gap is the utility uncertainty that survives compressing (x,r)(x,r) into z; it cannot be reduced by enlarging the predictor, only by enriching z. Our three-branch design (global text, global vision, and query-conditioned spatial evidence) is designed to reduce this representation gap, and we validate it empirically by ablation. 11 1 We train with Huber loss (δ=1δ=1, Equation 6) for robustness. Each target U(s,r)U(s,r) takes values in an interval of length one ([−r, 1−r][-r,\,1-r] for s∈ps _p; [−1,0][-1,0] for FullFull). For the unstandardized utility targets, predictions inside the interval incur residuals in the quadratic regime, while predictions outside it are strictly dominated; the population minimizer therefore coincides with the conditional mean. We use this observation only to motivate utility regression, since our actual training objective applies Huber loss to standardized utility targets, for which exact equivalence to the conditional mean is not assumed. Approximation by a lightweight MLP. Since ℛR is finite and the frozen encoders produce bounded features, z ranges over a compact set Z. Assuming each u¯s u_s is continuous on Z, universal approximation for non-polynomial activations [14] implies that for any δmlp>0 _mlp>0 there exists a finite-width two-layer GELU network gθg_θ such that sup∈‖gθ()−¯()‖∞≤δmlp. _z \|g_θ(z)- u(z) \|_∞≤ _mlp. (11) This is an existence statement; optimization and generalization residuals of the trained router are absorbed into δest _est. From utility prediction to decision quality. Let ^=gθ() u=g_θ(z) and s^∈argmaxs∈u^s s∈ _s u_s. Lemma 1 (Pointwise regret). If ‖^−‖∞≤ε\| u-U\|_∞≤ on a sample (x,r)(x,r), then U(s⋆)−U(s^)≤2εU(s )-U( s)≤ 2 . Proof. U(s⋆)≤u^s⋆+ε≤u^s^+ε≤U(s^)+2εU(s )≤ u_s + ≤ u_ s+ ≤ U( s)+2 . ∎ Corollary 1 (Decision margin at aggressive pruning levels). Let ⋆(x,r)S (x,r) denote the set of optimal strategies and define the nonzero decision margin as the gap between the optimal utility and the best strictly suboptimal utility. Under Group A, Δ(x,r)∈r, 1−r. (x,r)∈\r,\,1-r\. If at least one pruning strategy is correct, the gap between a correct pruning strategy and FullFull is 1−r1-r. If all pruning strategies fail, the gap between FullFull and a failed pruning strategy is r. Consequently, a uniform utility-prediction error ε<Δ(x,r)/2 < (x,r)/2 is sufficient to preserve an optimal decision. For r∈0.04,0.0625,0.1111,0.25r∈\0.04,0.0625,0.1111,0.25\, ε<0.02 <0.02 is sufficient for all samples, whereas at r=0.01r=0.01 the corresponding guarantee requires ε<0.005 <0.005. The shrinking margin at the most aggressive pruning level occurs only at the boundary between FullFull and failed pruning strategies; crossing this boundary incurs utility regret r. Proposition 1 (Expected regret). [U(s⋆)−U(s^)]≤ 2‖^−‖∞≤ 2δrep2+δest2,E [\,U(s )-U( s)\, ]\;≤\;2\,E \| u-U \|_∞\;≤\;2 _rep^2+ _est^2, (12) where the first inequality applies the proof of Lemma 1 pointwise, and the second uses ∥⋅∥∞≤∥⋅∥2\|·\|_∞≤\|·\|_2 and Jensen’s inequality. Scope of the analysis. The analysis formalizes a limited claim: sample-adaptive pruning-strategy selection can be reduced to estimating conditional pruning success accurately enough to preserve its ordering relative to known token costs. It does not establish that the preview representation is sufficient a priori. This is an empirical question, which we evaluate through the feature ablations and the observed utility headroom recovered relative to the Per-Sample Oracle. Remark 1 (Upper bound on the contribution of full-token fallback). Let frf_r denote the fraction of samples for which VIP-Router selects FullFull at retained-token ratio r. Consider a counterfactual policy that disables FullFull and instead routes these samples to a compressed strategy. In the worst case, all such replacements are incorrect. Accuracy can therefore decrease by at most frf_r, while Token Cost decreases by exactly fr(1−r)f_r(1-r). Consequently, Uforced-pruning≥UVIP-Router−frr.U_forced -pruning≥ U_VIP -Router-f_rr. (13) Using the five-seed mean routing frequencies, the corresponding bounds are 1.231.23, 0.410.41, 0.220.22, 0.140.14, and 0.070.07 utility points at the five reduction ratios, respectively. The average bound is only 0.410.41 utility points, corresponding to at most 6.1%6.1\% of VIP-Router’s observed utility gain over Best Fixed. Thus, even under this worst-case counterfactual, access to full-token inference can explain only a small fraction of the overall improvement. A.2 Algorithm Algorithm 1 Training procedure of VIP-Router. 1: Training set train=(Ii,qi,ri,i)i=1ND_train=\(I_i,q_i,r_i,a_i)\_i=1^N, candidate set =s1,…,sKS=\s_1,…,s_K\, frozen preview encoders, and router gθg_θ 2: Trained router parameters θ⋆θ and training-set normalization statistics (μk,σk)k=1K\( _k, _k)\_k=1^K 3: Construct utility targets for all training samples: Ui(sk,ri)←aisk−c(sk,ri),i=1,…,N,k=1,…,KU_i(s_k,r_i)← a_i^s_k-c(s_k,r_i), i=1,…,N,\;\;k=1,…,K 4: Compute per-strategy normalization statistics on the training set: μk←1N∑i=1NUi(sk,ri),σk←1N∑i=1N(Ui(sk,ri)−μk)2,k=1,…,K _k← 1N _i=1^NU_i(s_k,r_i), _k← 1N _i=1^N (U_i(s_k,r_i)- _k )^2, k=1,…,K 5: for each training epoch do 6: for each minibatch ℬ⊂trainB _train do 7: for each (Ii,qi,ri,i)∈ℬ(I_i,q_i,r_i,a_i) do 8: Standardize the utility target for each strategy: U~i,k←Ui(sk,ri)−μkσk,k=1,…,K U_i,k← U_i(s_k,r_i)- _k _k, k=1,…,K 9: Form the standardized target utility vector: ~i←[U~i,1,…,U~i,K]⊤ U_i←[ U_i,1,…, U_i,K] 10: Extract frozen visual and textual preview features from (Ii,qi)(I_i,q_i) 11: Compute query-conditioned cross-modal features 12: Predict standardized strategy utilities: ~^i←gθ(Ii,qi,ri) u_i← g_θ(I_i,q_i,r_i) 13: end for 14: Compute the element-wise Huber loss: ℒ←1|ℬ|K∑i∈ℬ∑k=1KℓHuber(u~^i,k,U~i,k)L← 1|B|K _i _k=1^K _Huber ( u_i,k, U_i,k ) 15: Update θ by minimizing ℒL 16: end for 17: Evaluate utility on the validation set after recovering utilities to the original scale 18: end for 19: return checkpoint θ⋆θ with the highest validation utility and (μk,σk)k=1K\( _k, _k)\_k=1^K Algorithm 2 Inference procedure of VIP-Router. 1: Image I, question q, retained-token ratio r, candidate set =s1,…,sKS=\s_1,…,s_K\, trained router gθ⋆g_θ , training-set normalization statistics (μk,σk)k=1K\( _k, _k)\_k=1^K, and frozen MLLM f 2: Extract frozen visual and textual preview features from (I,q)(I,q) 3: Compute query-conditioned cross-modal features 4: Predict standardized strategy utilities: ~^←gθ⋆(I,q,r) u← g_θ (I,q,r) 5: Recover predicted utilities on the original scale: u^k←σku~^k+μk,k=1,…,K u_k← _k u_k+ _k, k=1,…,K 6: Select the highest-utility candidate: s^←sargmaxk∈1,…,Ku^k s← s_ _k∈\1,…,K\ u_k 7: Apply s s and run the frozen MLLM once: y^←fs^(I,q,r) y← f_ s(I,q;r) 8: return y y A.3 Implementation and Reproducibility Details A.3.1 Dataset Construction and Split Details We construct the routing dataset from the per-sample inference results officially released by VTC-Bench [19] for Qwen2-VL-7B-Instruct. The released results include full-token inference, equivalent-ratio image downsampling, and the four candidate pruning methods evaluated at multiple reduction ratios. For each benchmark, binary correctness is derived from its task-specific evaluation fields. For MMBench and MMBench-CN, we compare the extracted multiple-choice prediction with the ground-truth answer. Following VTC-Bench, we reconstruct Group A independently at each reduction ratio. An image–question record is included at a given ratio when full-token inference is correct while equivalent-ratio image downsampling is incorrect. We then associate the correctness labels of FastV, VisionZip, PruMerge+, and DART with each eligible record–ratio pair. This procedure produces 33,091 routing examples from 12,919 unique image–question records. Because Group A membership depends on the reduction ratio, the routing data form a sparse record–ratio grid: the same record may appear at only a subset of the evaluated ratios. We evaluate five retained-token ratios, r∈0.25, 0.1111, 0.0625, 0.04, 0.01,r∈\0.25,\;0.1111,\;0.0625,\;0.04,\;0.01\, corresponding to visual-token reduction ratios of 75.00%75.00\%, 88.89%88.89\%, 93.75%93.75\%, 96.00%96.00\%, and 99.00%99.00\%, respectively. The ratio inputs to VIP-Router are r and ln(r+10−8) (r+10^-8). Data splitting is performed at the image–question-record level rather than at the record–ratio level, ensuring that all ratio-specific examples of the same record remain in the same split. Records are identified by the pair benchmark:doc_id and assigned deterministically to training, validation, and test sets using a fixed hash-based 70/15/1570/15/15 split. This results in 8,988 training, 1,956 validation, and 1,975 test records, with no record shared across splits. The split is record-disjoint but not necessarily image-disjoint: when the same underlying image is paired with multiple questions and therefore multiple document IDs, the resulting image–question records are treated as distinct samples. Table 6: Number of routing examples at each visual-token reduction ratio. Group A membership is ratio-dependent, so the number of eligible examples varies across ratios. Reduction Retained r Train Val. Test Total 75.00% 0.2500 1,998 411 431 2,840 88.89% 0.1111 3,720 810 801 5,331 93.75% 0.0625 4,772 1,044 1,037 6,853 96.00% 0.0400 5,391 1,136 1,183 7,710 99.00% 0.0100 7,218 1,551 1,588 10,357 Total – 23,099 4,952 5,040 33,091 A.3.2 Router Architecture and Training Configuration Preview encoder. The default VIP-Router uses the frozen openai/clip-vit-base-patch32 checkpoint for both visual and textual preview encoding. Images follow the default CLIP preprocessing and are resized and center-cropped to 224×224224× 224. We extract the penultimate visual hidden representation, discard the CLS token, and retain the resulting 49 patch tokens ∈ℝ49×768P ^49× 768. Their mean forms the 768-dimensional global visual representation. Questions are tokenized to a maximum length of 77 tokens. We retain the 512-dimensional text-token representations and use the CLIP-projected EOS representation as the 512-dimensional global text feature. All CLIP parameters remain frozen during router training. Cross-modal representation and utility prediction. Text and visual token representations are projected independently to 256 dimensions. A single four-head text-to-vision cross-attention layer uses the projected text tokens as queries and the projected visual tokens as keys and values. The attention output is followed by LayerNorm, and the EOS-position output is used as the 256-dimensional query-conditioned visual representation ×h^×. The attention layer uses dropout 0.10.1; no additional residual or feed-forward block is introduced. The router input is =[global;global;r;ln(r+10−8);×]∈ℝ1538.z=[t_global;v_global;r; (r+10^-8);h^×] ^1538. A two-layer MLP, 1538→512→5,1538→ 512→ 5, with GELU activation and dropout 0.10.1, predicts the utilities of FastV,VisionZip,PruMerge+,DART,Full.\FastV,VisionZip,PruMerge^+,DART,Full\. Full is treated as an ordinary candidate whose utility is predicted by the router rather than analytically fixed. The complete router contains 1,382,405 trainable parameters. Standardization. All standardization statistics are estimated exclusively from the training split. The frozen global preview features and the two ratio features are standardized feature-wise using their training-set means and standard deviations. Utility targets are standardized separately for each candidate. For candidate k, let μk _k and σk _k denote the mean and standard deviation of its training-set utility. The regression target is U~i,k=Ui,k−μkσk. U_i,k= U_i,k- _k _k. (14) For constant dimensions, including the Full utility under Group A, the standardization scale is set to 11. At inference time, predicted standardized utilities are transformed back to the original utility scale, U^i,k=U~^i,kσk+μk, U_i,k= U_i,k _k+ _k, (15) and the candidate with the largest predicted utility is selected. Validation and test samples are not used to estimate any standardization statistics. Optimization. We train the projection layers, cross-attention module, LayerNorm, and utility predictor using AdamW with learning rate 10−310^-3, weight decay 10−410^-4, and batch size 256. Training runs for at most 50 epochs with a linear warm-up over the first 5%5\% of optimization steps followed by cosine learning-rate decay. Gradients are clipped to a global norm of 1.01.0. The objective is an element-wise Huber loss with δ=1δ=1, averaged over samples and the five candidate utilities in the standardized target space. Training is performed in FP32. We use early stopping with patience five. For checkpoint selection, validation utility is first averaged within each retained-token ratio and then macro-averaged across the five ratios. The checkpoint with the highest validation macro utility is used for final test evaluation. Frozen CLIP representations are precomputed once for each unique image–question record and reused across its available retained-token ratios; all trainable VIP-Router components are evaluated online. Unless otherwise stated, VIP-Router results are averaged over five independent training runs with random seeds 0,1,2,3,40,1,2,3,4. All runs use the same deterministic data split, cached preview features, architecture, target standardization, and optimization configuration; only the random training trajectory differs. A.3.3 Hardware and Software Environment All experiments are conducted on a single NVIDIA H200 GPU. We use Python 3.10.12, PyTorch 2.3.0 with CUDA 12.4, and Transformers 4.49.0. The CLIP preview encoder is implemented with transformers.CLIPModel and CLIPProcessor. Router training is performed in FP32. A.4 Encoder Analysis Table 7: Performance of VIP-Router with alternative frozen preview encoders. The default CLIP-B/32 result is averaged over five random seeds, whereas the alternative encoder variants are single-run results and are therefore intended as robustness evidence rather than a strict encoder ranking. Variant Vision Encoder Text Encoder Acc. Cost Utility Default CLIP ViT-B/32 CLIP Text 51.19 0.135 37.68 CLIP-B/16 CLIP ViT-B/16 CLIP Text 50.31 0.130 37.28 SigLIP2-B/32 SigLIP2-B/32 SigLIP2 Text 47.38 0.110 36.36 TIPSv2-B/14 TIPSv2-B/14 TIPSv2 Text 49.87 0.125 37.34 EUPE + CLIP Text EUPE-ViT-B CLIP Text 51.12 0.132 37.96 EUPE + SigLIP2 Text EUPE-ViT-B SigLIP2 Text 50.47 0.131 37.34 Encoder alternative. The default VIP-Router uses CLIP ViT-B/32 and its paired text encoder to construct the preview representation. To examine whether the routing framework remains effective with alternative preview encoders, we replace the default frozen encoders while keeping the routing objective, candidate strategy set, dataset split, and architecture unchanged. All variants retain the same text-to-vision cross-attention module, EOS-style pooling, ratio conditioning, and two-layer utility predictor. As shown in Table 7, VIP-Router remains effective across substantially different preview encoder families. CLIP-B/16 [25], TIPSv2-B/14 [4], and both EUPE [39] variants achieve utility close to the default CLIP-B/32 configuration, while SigLIP2-B/32 [27] exhibits a somewhat larger decrease. Notably, the EUPE–CLIP variant achieves slightly higher utility than the default configuration, with comparable accuracy. These results indicate that the routing formulation is not restricted to a single preview feature family and can operate with multiple frozen vision–language representations. We do not interpret the small differences among encoder variants as a strict ranking. The alternative configurations were evaluated as single runs, and their training RNG trajectories are not perfectly matched across variants. Accordingly, this experiment is intended to assess the robustness of VIP-Router to encoder replacement rather than to identify an optimal preview encoder. A.5 Additional Results Table 8: Per-benchmark accuracy (%) of fixed pruning strategies and VIP-Router on VTC-Bench Group A test set across visual-token reduction ratios. Full is shown in gray as a reference and has 100%100\% accuracy by the construction of Group A; it is excluded from highlighting. Fixed-strategy results are deterministic, whereas VIP-Router results are averaged over five random seeds. Within each reduction-ratio block, the best pruning result in each column is bolded, and the second-best result in the Avg. column is underlined. Method GQA MMB MMBCN MME POPE MMStar OCR ChartQA Overall Full 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 Reduction ratio = 75.00% FastV 61.59 54.05 43.33 87.50 69.44 55.88 25.81 39.71 54.99 VisionZip 60.26 54.05 33.33 62.50 80.56 58.82 19.35 55.88 57.54 PruMerge+ 64.24 62.16 40.00 62.50 77.78 52.94 22.58 42.65 57.31 DART 56.95 51.35 46.67 87.50 73.61 50.00 51.61 41.18 55.68 VIP-Router 64.11 63.24 66.67 90.00 90.83 50.59 56.77 47.65 66.91 Reduction ratio = 88.89% FastV 42.45 36.21 28.81 41.18 43.84 25.00 10.00 32.26 35.33 VisionZip 46.70 27.59 33.90 41.18 74.66 28.12 20.00 51.61 47.94 PruMerge+ 47.64 25.86 22.03 47.06 71.23 31.25 10.00 41.01 43.20 DART 49.06 32.76 37.29 58.82 60.96 46.88 33.33 26.73 42.07 VIP-Router 53.58 46.90 49.15 57.65 83.70 34.38 47.00 51.52 57.65 Reduction ratio = 93.75% FastV 38.08 36.49 25.00 52.94 38.71 31.82 8.11 15.22 29.22 VisionZip 38.43 28.38 27.94 38.24 70.97 31.82 8.11 27.17 37.42 PruMerge+ 46.26 27.03 22.06 38.24 73.12 31.82 9.46 27.90 39.73 DART 44.48 25.68 39.71 44.12 59.14 34.09 31.08 15.22 36.26 VIP-Router 49.82 43.24 46.47 48.24 85.27 31.36 43.51 29.35 49.62 Reduction ratio = 96.00% FastV 28.49 28.42 27.62 27.59 37.32 31.48 7.50 8.03 23.92 VisionZip 37.09 33.68 28.57 37.93 66.99 33.33 6.25 10.95 33.05 PruMerge+ 39.17 25.26 22.86 34.48 62.20 24.07 5.00 15.69 32.12 DART 41.84 24.21 39.05 44.83 56.46 35.19 31.25 12.04 34.91 VIP-Router 43.50 40.63 46.48 40.69 78.47 35.93 41.50 20.07 44.29 Reduction ratio = 99.00% FastV 18.57 14.71 20.73 19.35 45.49 15.87 2.73 3.28 18.89 VisionZip 24.90 26.47 34.15 33.87 51.76 28.57 0.00 5.84 25.82 PruMerge+ 22.04 15.29 16.46 24.19 53.33 23.81 1.82 7.30 21.98 DART 24.69 19.41 29.27 37.10 38.43 26.98 30.91 3.65 24.18 VIP-Router 27.14 37.76 43.66 36.45 61.57 32.06 41.27 25.47 37.48 A.5.1 Detailed Per-Benchmark Results Table 8 reports the complete per-benchmark accuracy results underlying the aggregate comparison in Section 4.2. VIP-Router achieves the best pruning result in 30 of the 40 benchmark–ratio combinations (75.0%), showing that its improvement is broadly distributed across benchmarks rather than driven by a small subset of tasks. Its gains also persist under the most aggressive reduction settings, where the accuracy of all fixed pruning strategies degrades substantially. Full-token inference is included only as a reference and has 100%100\% accuracy by the construction of VTC-Bench Group A. The strongest fixed strategy varies with the pruning level: VisionZip achieves the highest overall accuracy among fixed strategies at 75.00%75.00\%, 88.89%88.89\%, and 99.00%99.00\% reduction, whereas PruMerge+ and DART are strongest at 93.75%93.75\% and 96.00%96.00\%, respectively. VIP-Router nevertheless achieves higher overall accuracy than the strongest fixed strategy at every evaluated reduction ratio. Figure 4: Per-benchmark accuracy of the four fixed pruning strategies on VTC-Bench Group A. A.5.2 Complementarity among Pruning Strategies Figure 4 provides the per-benchmark comparison of the fixed pruning strategies. Their relative performance varies across benchmarks, and no single pruning criterion is uniformly strongest. Across reduction ratios, a substantial fraction of Group A samples for which Best Fixed fails can still be correctly handled by at least one alternative pruning strategy. Thus, failures of the globally selected fixed strategy are frequently recoverable by another pruning criterion. This sample-level evidence complements the Best Fixed–Oracle gap reported in the main text and shows that benchmark-average rankings mask substantial strategy complementarity. Figure 5: VIP-Router strategy-selection distribution across visual-token reduction ratios. Each bar reports the mean fraction of test samples routed to FastV, VisionZip, PruMerge+, DART, or Full over five random seeds. A.5.3 VIP-Router Selection Behavior Figure 5 shows the strategy-selection distribution of VIP-Router across reduction ratios. The routing decisions remain distributed across all four pruning methods at every operating point, indicating that VIP-Router does not collapse to another globally fixed pruning policy. The relative preference among pruning strategies also changes with the pruning level; for example, DART is selected increasingly often under aggressive reduction, whereas VisionZip remains a substantial component throughout the evaluated range. Full-token inference remains a minority choice. VIP-Router selects Full for 4.92%4.92\%, 3.72%3.72\%, 3.51%3.51\%, 3.48%3.48\%, and 6.66%6.66\% of samples at reduction ratios of 75.00%75.00\%, 88.89%88.89\%, 93.75%93.75\%, 96.00%96.00\%, and 99.00%99.00\%, respectively. Thus, the accuracy improvement is not obtained by routinely reverting to full-token inference. Together with the counterfactual bound in Remark 1, these results indicate that the primary gain comes from sample-adaptive selection among heterogeneous pruning strategies. A.5.4 Routing Error Diagnosis We analyze the saved test predictions of the final VIP-Router across five random seeds, with 5,040 image–question-record–ratio decisions per seed. Let si⋆s_i denote the oracle argmax returned by the evaluation procedure and s^i s_i the router’s selected strategy. We distinguish an oracle-argmax mismatch, s^i≠si⋆ s_i≠ s_i , from the incurred utility regret: Δi=Ui(si⋆,ri)−Ui(s^i,ri). _i=U_i(s_i ,r_i)-U_i( s_i,r_i). An argmax mismatch is not a VQA answer error and may incur zero regret when multiple strategies attain the same optimal utility. For mismatched decisions, we define the oracle margin as mi=Ui(1)−Ui(2),m_i=U_i^(1)-U_i^(2), where Ui(1)U_i^(1) and Ui(2)U_i^(2) are the two highest candidate utilities, including ties. We classify mismatches as near-tie when mi≤0.03m_i≤ 0.03 and high-margin otherwise. Table 9: Routing-error diagnosis of VIP-Router on VTC-Bench Group A. Values are means across five random seeds. Mismatch percentages are fractions of all oracle-argmax mismatches, and regret percentages are fractions of total regret; both pool all test record–ratio pairs within each run rather than macro-averaging across ratios. Mean regret is computed within each decision group on the raw utility scale. Neighborhood statistics use k=10k=10. Decision group Mismatch (%) Regret (%) Mean regret Neighbor agreement Neighbor entropy (nats) Oracle-argmax match – 0 0 0.436 0.885 All oracle-argmax mismatches 100 100 0.234 0.444 0.905 Near-tie 59.1 36.5 0.145 0.485 0.865 High-margin 40.9 63.5 0.363 0.385 0.963 Strategy disagreement versus utility loss. Table 9 shows that high-margin mismatches account for 40.9%40.9\% of strategy disagreements but contribute 63.5%63.5\% of total regret. Their contribution to the remaining utility loss is therefore substantially larger than their frequency alone suggests. Conversely, approximately 28.9%28.9\% of all argmax mismatches incur zero regret, reflecting utility-equivalent choices rather than suboptimal routing. Near-tie mismatches nevertheless contribute 36.5%36.5\% of total regret: a small gap between the two best candidates does not bound the loss from selecting a lower-utility candidate. Thus, exact agreement with a single oracle argmax is insufficient to assess routing quality; the utility consequences of a decision must also be considered. Local structure of routing errors. To examine the feature-space neighborhoods of these decisions, we retrieve the ten nearest training examples at the same retained-token ratio using cosine similarity. The representation consists of the frozen global preview features and ratio inputs, [global;global;r;ln(r+10−8)],[t_global;v_global;r; (r+10^-8)], with training-set feature standardization followed by L2 normalization. The learned cross-attention representation is not included. Neighbor agreement measures the fraction of retrieved examples sharing the test example’s oracle-argmax label, while neighbor entropy measures the entropy of the retrieved oracle-label distribution. High-margin mismatches have lower neighbor agreement and higher neighbor entropy than oracle-matched decisions. Importantly, the lower-agreement pattern does not hold for argmax mismatches as a whole, indicating that the association is specific to the high-margin subset rather than strategy disagreement in general. These results associate high-margin routing errors with locally mixed oracle labels in the frozen global preview space. They characterize the structure of the remaining errors without establishing a representational limitation of the complete learned cross-modal router.