Paper deep dive
VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions
Adrian Bulat, Alberto Baldrati, Ioannis Maniadis Metaxas, Yassine Ouali, Georgios Tzimiropoulos
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/26/2026, 1:48:19 AM
Summary
VISion On Request (VISOR) is a method for improving Large Vision-Language Model (LVLM) efficiency by sparsifying the interaction between image and text tokens rather than reducing the number of visual tokens. It uses a universal network trained on varying computational budgets, employing a lightweight policy mechanism to dynamically allocate visual computation (cross-attention and self-attention layers) based on per-sample complexity.
Entities (4)
Relation Signals (3)
VISOR → improvesefficiencyof → LVLM
confidence 100% · VISion On Request (VISOR), a method that reduces inference cost without discarding visual information.
VISOR → uses → Cross-Attention
confidence 95% · Specifically, the language model attends to the full set of high-resolution visual tokens through a small, strategically placed set of attention layers: general visual context is provided by efficient cross-attention
VISOR → uses → Self-attention
confidence 95% · a few well-placed and dynamically selected self-attention layers refine the visual representations themselves
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Existing approaches for improving the efficiency of Large Vision-Language Models (LVLMs) are largely based on the concept of visual token reduction. This approach, however, creates an information bottleneck that impairs performance, especially on challenging tasks that require fine-grained understanding and reasoning. In this work, we challenge this paradigm by introducing VISion On Request (VISOR), a method that reduces inference cost without discarding visual information. Instead of compressing the image, VISOR improves efficiency by sparsifying the interaction between image and text tokens. Specifically, the language model attends to the full set of high-resolution visual tokens through a small, strategically placed set of attention layers: general visual context is provided by efficient cross-attention between text-image, while a few well-placed and dynamically selected self-attention layers refine the visual representations themselves, enabling complex, high-resolution reasoning when needed. Based on this principle, we first train a single universal network on a range of computational budgets by varying the number of self-attention layers, and then introduce a lightweight policy mechanism that dynamically allocates visual computation based on per-sample complexity. Extensive experiments show that VISOR drastically reduces computational cost while matching or exceeding state-of-the-art results across a diverse suite of benchmarks, and excels in challenging tasks that require detailed visual understanding.
Tags
Links
- Source: https://arxiv.org/abs/2603.23495v1
- Canonical: https://arxiv.org/abs/2603.23495v1
Trouble viewing inline? Open PDF directly →
Full Text
73,674 characters extracted from source content.
Expand or collapse full text
VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions Adrian Bulat1,2 Alberto Baldrati1∗ Ioannis Maniadis Metaxas1∗ Yassine Ouali1 Georgios Tzimiropoulos1,3 1Samsung AI Cambridge 2Technical University of Iasi 3Queen Mary University of London Equal contribution Abstract Existing approaches for improving the efficiency of Large Vision-Language Models (LVLMs) are largely based on the concept of visual token reduction. This approach, however, creates an information bottleneck that impairs performance, especially on challenging tasks that require fine-grained understanding and reasoning. In this work, we challenge this paradigm by introducing VISion On Request (VISOR), a method that reduces inference cost without discarding visual information. Instead of compressing the image, VISOR improves efficiency by sparsifying the interaction between image and text tokens. Specifically, the language model attends to the full set of high-resolution visual tokens through a small, strategically placed set of attention layers: general visual context is provided by efficient cross-attention between text-image, while a few well-placed and dynamically selected self-attention layers refine the visual representations themselves, enabling complex, high-resolution reasoning when needed. Based on this principle, we first train a single universal network on a range of computational budgets by varying the number of self-attention layers, and then introduce a lightweight policy mechanism that dynamically allocates visual computation based on per-sample complexity. Extensive experiments show that VISOR drastically reduces computational cost while matching or exceeding state-of-the-art results across a diverse suite of benchmarks, and excels in challenging tasks that require detailed visual understanding. 1 Introduction Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in multimodal understanding [48, 8, 22]. These systems typically pair a vision encoder (e.g., CLIP [36]) with a large language model (LLM) [43, 16, 48]. The vision encoder maps an input image to dense visual tokens, which are passed through a connector module and fed to the LLM alongside the textual prompt/query. Most of LVLM’s computations are due to the large number of visual tokens, a cost that increases sharply with image resolution [22]. To mitigate this, a large volume of work has been proposed that explores the idea of token reduction/compression. These works reduce the number of visual tokens by dynamically pruning and/or merging redundant tokens at test-time [1, 47, 50, 56, 6] or by training specialised compressors [4, 14, 9]. While they perform well on tasks requiring coarse visual understanding, we show that they often incur substantial information loss on complex, high-resolution tasks that require fine-grained visual understanding. See accuracy on “easy” vs “hard” in Fig. 10. This is not surprising as such approaches, by shrinking the set of visual tokens, inevitably, create an information bottleneck. In this work, we propose a completely different and orthogonal path to token compression methods for increasing the efficiency of LVLMs. Unlike prior token reduction/compression methods that aim to reduce the number of visual tokens processed by the LVLM, our approach aims to reduce/sparsify the number of computational layers executed within the LVLM. Specifically, our method strategically executes a limited number of cross-attention and self-attention layers within the LVLM, allowing it to attend and update the full set of visual tokens only at a few selected points during the forward pass. Owing to this property, we coin our method - VISOR, Vision on Request. Our idea builds upon the observation that the query and answer tokens sparsely interact with the visual tokens [17] on a select few critical layers. A phenomenon that we show to be heavily task-dependent, with the location and number of layers and the degree of sparsity varying significantly across tasks, depending on those tasks’ complexity. Overall, we make the following contributions: • Firstly, we decompose the LVLM layer into image-image and text-image (cross-modal) interactions, and show that executing a fairly small number of cheap cross-attention layers for text-image, which operate on the same vision representations, suffices for tasks requiring coarse visual understanding. This alone surpasses prior state-of-the-art methods on a range of vision-language benchmarks in terms of accuracy and speed. • Secondly, we demonstrate that for complex tasks, both prior works and our cross-attention only variant struggle to perform fine-grained visual understanding. We attribute this to the fact that cross-attention layers enable language tokens to attend to image information, but do not update/modify the visual tokens themselves. To alleviate this, we introduce and execute a small number of self-attention layers that perform the update of the visual tokens, enabling a gradual refinement from lower to higher-level visual features. • Thirdly, as different tasks and samples require different amounts of visual detail, we first train a single universal network on a range of computational budgets by varying the number of self-attention layers. Then, we propose an adaptive inference approach that automatically selects the self-attention layers to be executed on a per-sample basis using a lightweight policy mechanism trained via offline pseudo-labelling. • Fourth, we show that VISOR can be combined with existing token reduction methods to further improve efficiency without compromising performance. • Fifth, we set a new state-of-the-art on a range of vision language benchmarks, excelling in challenging tasks that require detailed visual understanding. See Fig. 10. 2 Closely related work Efficient LVLMs via Token Reduction: To address the computational challenges posed by the large number of visual tokens in LVLMs, several approaches have been proposed to reduce the number of tokens processed by the LLM. These methods can be broadly grouped into two categories: dynamic token pruning and merging techniques [56, 54, 47, 1, 39], and learned token compression strategies [9, 50, 4, 14, 2]. Figure 1: Efficiency comparison: FLOPs reduction vs acc. Notice that our approach is significantly more efficient while also retaining the performance on the harder datasets. See Sects. 3 and 5.1 for “easy”-“hard” definition. The former category focuses on dynamically identifying the most important tokens, reducing redundancy by pruning or merging the less relevant tokens prior to the LLM [50, 54, 39], layer-by-layer within the LLM [47, 6, 49, 42], or both [52], using heuristic criteria. Examples of criteria include: selecting top-k attended tokens [6], assessing the correlation between patches [55], using the attention score between image tokens and [CLS] token [54], rating the vision tokens using the text tokens [56], or by analysing the information quantity in the attention matrix [42]. The latter category either replaces the connector module with a learned compressor [9], or introduces a new module before the LLM [50] or as part of the vision encoder [4, 14]. These methods finetune the LVLM, either fully or partially. While showing promising results, most of these approaches focus on coarser understanding and lower-resolution tasks, often using a LLaVA-1.5 model [4, 50, 2]. Very few works (e.g., [1, 21]) consider more challenging and fine-grained tasks that require higher resolution, with those that do either suffering from a large accuracy drop [1] or exhibiting little to no speed-ups on these datasets [21]. In this work, we further evaluate existing methods under a unified setting and architecture, and highlight this as a general trend in existing token reduction works. We argue that this performance degradation stems primarily from the information bottleneck inherent in token reduction. To alleviate this, VISOR sidesteps the token reduction paradigm altogether. Instead of reducing cost by discarding tokens, VISOR strategically limits the number of layers where the language model interacts with and updates visual information, thereby maintaining access to the full, high-resolution visual context throughout the model. This ensures that critical visual details are never permanently lost and can be accessed by the model when needed for fine-grained reasoning, while still achieving significant computational savings. Furthermore, our approach is orthogonal to existing token compression methods and can be combined with them for further efficiency gains. 3 Motivation: Image processing within LVLMs To motivate our design, herein, we focus on the internal workings of a standard LVLM (LLaVA-OV) to understand how it utilizes and processes visual information. We analyze the attention patterns of image-image and text-image (cross-modal) interactions and investigate three key questions: (a) SQA (b) GQA (c) DocVQA Figure 2: Cross-modality attention patterns across layers. We plot the proportion of attention scores allocated to three interaction types: text queries attending to image tokens (Query-to-Image), answer tokens attending to image tokens (Answer-to-Image), and answer tokens attending to query tokens (Answer-to-Query). For easy tasks like SQA, interaction is sparse and dominated by text-to-text attention. For hard tasks like DocVQA, the model attends to the image across the whole network. How often, and when, does the model look at the image? We distinguish between three types of interactions: Query-to-Image, Answer-To-Image, and Answer-To-Query. Fig. 2 shows the layer-wise distribution of these interactions for three representative datasets. The results reveal that image-text interactions are task-dependent. For tasks requiring coarse vision understanding (e.g., ScienceQA), the model relies heavily on textual context (Answer-to-Query), with only limited interaction with the image, primarily in the initial and final layers. In contrast, for fine-grained tasks (e.g., DocVQA), the model exhibits sustained attention to the image across the whole network, indicating a continuous need for visual grounding. Moreover, we can observe that critical text-image interactions also occur in the middle layers in addition to the first and last layers. Interestingly, the saw-tooth patterns (for both GQA and DocVQA) suggest that not all cross-attention layers are necessary. (a) SQA (b) GQA (c) DocVQA Figure 3: Evolution of visual representations across layers, measured by pairwise CKA similarity. For easy tasks (e.g., SQA), visual features remain largely static (high similarity across layers). For harder tasks (e.g., DocVQA), features are progressively refined. How do visual representations evolve? To analyze how vision features evolve across layers within the LLM transformer of the LVLM, we adopt the Centered Kernel Alignment (CKA) [11] similarity metric following Kornblith et al. [20] and Raghu et al. [37] (see also supplementary material). We compute the pairwise CKA similarity between vision features from all layers of LLaVA-OV transformer on three representative datasets. As shown in Fig. 3, for easy tasks like ScienceQA, the visual features remain largely unchanged throughout the model (CKA >> 0.9), implying that the initial representations are sufficient. However, for hard tasks like DocVQA, the features evolve significantly (CKA drops to 0.6), indicating that the model actively refines visual representations to solve the task. This highlights that while coarse tasks can rely on static visual features, complex tasks benefit from the refinement of visual information within the LLM. From the figure, we also observe a series of clusters emerging, indicating that the model refines visual features in stages. The number of stages is task-dependent, and we posit that it indicates the minimum number of self-attention layers that need to be executed to achieve optimal performance. (a) Accuracy distribution per dataset. (b) Dataset accuracy correlation. Figure 4: Accuracy sensitivity by dropping all vision tokens for different subsets of LLM layers. Left: Accuracy distribution on a dataset-by-dataset basis. Certain datasets (e.g., DocVQA, ChartQA) are particularly sensitive to reduced vision-language interactions. Right: we show how the layer-drop config. & accuracy correlate among datasets. Two clusters emerge: vision-sensitive (“hard”) (e.g., InfoVQA, OCRBench, etc.) and coarse vision (“easy”) (e.g., POPE, SQA, GQA, etc.) datasets. What is the impact of reducing image-text interactions? To this end, we drop all the vision tokens from random subsets of LLM layers during inference and measure the performance degradation. Fig. 4 shows that datasets cluster into two groups. “Easy” tasks (e.g., SQA, POPE) are robust to this dropout, maintaining high performance. “Hard” tasks (e.g., DocVQA, ChartQA, InfoVQA) are highly sensitive, with performance dropping sharply as visual processing is reduced. We use this as a basis for dataset categorization in the rest of the paper. This confirms that a one-size-fits-all approach to visual processing is suboptimal; the computational budget should adapt to the sample/task at hand. Key takeaways that inform the design of our proposed method: (1) Image-text interactions are sparse, exhibit saw-tooth patterns, and the degree of interaction is highly task-dependent. (2) While coarse tasks can rely on static visual features, complex tasks benefit from dynamic refinement of visual information within the LLM. (3) A one-size-fits-all approach to visual processing is suboptimal; the computational budget should adapt to sample/task demands. 4 Method 4.1 Preliminaries: Large Vision-Language Models Let ∈ℝNv×dV ^N_v× d and ∈ℝNt×dT ^N_t× d be the sequences of visual and text tokens, respectively, processed by an LVLM. In a standard LVLM, each transformer layer (TL) l consists of a self-attention layer followed by a feed-forward network (FFN) applied to the concatenated sequence [(l−1);(l−1)][V^(l-1);T^(l-1)]: [(l);(l)]=TLl([(l−1);(l−1)]).[V^(l);T^(l)]=TL_l([V^(l-1);T^(l-1)]). (1) It is straightforward to observe that the self-attention operating on the concatenated sequence captures all possible image-image, image-text, and text-text interactions. Its computational cost is quadratic in the total sequence length, O((Nv+Nt)2⋅d)O((N_v+N_t)^2· d). Since Nv≥NtN_v≥ N_t, especially for high-resolution images, the image-image interactions dominate the inference cost. 4.2 Vision on Request (VISOR) To reduce the computational cost without performing token reduction, we propose VISOR that modifies the LVLM architecture to process visual information sparsely. The core idea is to decouple the processing of text and vision tokens. Most LLM layers operate only on text tokens. Only a few selected layers additionally integrate text-image and image-image interactions by strategically inserting a small number of cross-attention and self-attention layers, as illustrated in Fig. 5 111More precisely, self-attention models all possible interactions, including the image-image ones.. Crucially, the inserted layers depend on sample/task complexity. Figure 5: Conceptual architecture of VISOR. Visual information is sparsely injected into the language stream via a few cross-attention and self-attention layers modelling text-image and image-image interactions. Cross-attention efficiently provides visual context to the text tokens without altering the visual representations. Self-attention, while more costly, refines the visual tokens, enabling subsequent cross-attention layers to access higher-level visual features. This design strikes a balance between efficiency and representational power. 4.2.1 Efficient Visual Context via Cross-Attention For many tasks, the LLM only needs to query visual features without needing to update them. Cross-attention layers provide an efficient mechanism for this, as they integrate visual information into the text processing stream without modifying the visual tokens themselves. We leverage this by having most transformer layers operate solely on text tokens. We then designate a small, uniformly distributed subset of layers, indexed by a set ℒCAL_CA, to perform cross-attention, allowing the text stream to efficiently query the static visual features at selected points. Let (0)V^(0) be the initial visual tokens from the vision encoder. For a layer l, the update rule is: (l)=(l−1),(l)=TLl(CrossAttn((l−1),(l−1))),if l∈ℒCATLl((l−1)),otherwise. splitV^(l)&=V^(l-1),\\ T^(l)&= casesTL_l(CrossAttn(T^(l-1),V^(l-1))),&if l _CA\\ TL_l(T^(l-1)),&otherwise. cases split (2) The CrossAttn module uses text tokens as queries and visual tokens as keys and values, and its output is added residually to the text stream. Crucially, in this cross-attention-only variant, visual tokens (l−1)V^(l-1) are never updated (i.e., (l−1)=(0),∀lV^(l-1)=V^(0),∀ l), making the process highly efficient. Finally, to ensure the vision tokens retain positional information, which is essential for spatial reasoning, inspired by Chu et al. [10], we adapt the idea of conditional positional embeddings to 1D sequences and implement them using a 1D depth-wise convolutional layer (with kernel size 7 and a padding of 3). This approach effectively captures both local and global positional information without the slower convergence issues associated with absolute or rotary positional embeddings. 4.2.2 Refining Visual Features with Selective Self-Attention The cross-attention only model described in Eq. 2 is efficient and performs well on tasks requiring coarse visual understanding, often surpassing prior state-of-the-art methods. However, the visual tokens remain unchanged, which limits performance on tasks requiring fine-grained reasoning. To address this, we introduce a small number of full self-attention layers on the visual tokens at specific layers, indexed by a set ℒSAL_SA. These layers allow the model to build hierarchical visual representations. Let us define =CrossAttn((l−1),(l−1))Z=CrossAttn(T^(l-1),V^(l-1)). Then the complete update rule for a layer l becomes: ((l),(l))=TLl([(l−1);(l−1)]),if l∈ℒSA((l−1),TLl()),if l∈ℒCA((l−1),TLl((l−1))),otherwise. split&(V^(l),T^(l))= casesTL_l([V^(l-1);T^(l-1)]),&if l _SA\\ (V^(l-1),TL_l(Z)),&if l _CA\\ (V^(l-1),TL_l(T^(l-1))),&otherwise. cases split (3) When l∈ℒSAl _SA, a standard transformer layer processes both visual and text tokens, updating (l−1)V^(l-1) to (l)V^(l). Subsequent cross-attention layers (l′>l,l′∈ℒCAl >l,l _CA) will then use these refined visual tokens (l)V^(l), enabling more effective context integration. In practice, we find that distributing a few cross-attention and self-attention layers uniformly across the model yields strong performance. 4.2.3 Training a Universal Model for Adaptive Computation A key insight from our analysis in Sec. 3 is that different tasks require varying amounts of visual processing. To accommodate this without training and storing multiple models, we train a single, universal VISOR model capable of operating at various computational budgets. This is achieved by making the model robust to executing different subsets of its self-attention layers, which we refer to as configurations. To this end, we propose the following training strategy: 1. Bounding the configurations space. Given a model with L total layers, we first determine the maximum number of cross-attention (|LCA||L_CA|) and self-attention (|LSA||L_SA|) layers needed to match the performance of the original dense model. Empirically, we find that setting |LCA|=|LSA|=L/3|L_CA|=|L_SA|=L/3 provides a strong upper bound (see Sec. 6, Table 3). We then pre-train a VISOR model with this maximal configuration to establish a reference network. 2. Identifying viable sub-networks. As the space of possible sub-networks is vast, with many configurations leading to catastrophic performance degradation due to skipping critical layers needed for certain tasks, we propose to systematically evaluate subsets from the pre-trained model to identify a set of viable configurations - those that maintain high accuracy at least in certain cases. Moreover, as the cross-attention layers are computationally inexpensive222The FLOPs for a full self-attention layer are approx. O((Nt+Nv)2d)O((N_t+N_v)^2d), whereas for a cross-attention layer, they are only O(NtNvd)O(N_tN_vd). and provide essential visual context, we opt to always execute them, only varying the number and location of self-attention layers to create different computational budgets. Hence, we evaluate the model’s performance by systematically varying the number of self-attention layers from 0 to |LSA||L_SA|, testing various subsets of the |LSA||L_SA| layers. See Sec. 6 for ablation results and supplementary material for more details and visualizations. 3. Universal fine-tuning. Finally, inspired by [3], we finetune the model by randomly selecting at each optimization step one of these viable configurations. This results in a universal model that works robustly for any of the configurations used during training, and hence, across a wide range of computational budgets. 4.3 Adaptive Inference As highlighted in Sec. 3, the amount of visual processing required varies significantly depending on the task and even across individual samples within the same benchmark. This observation indicates that a single, fixed configuration may not be optimal for all scenarios. To address this, we utilize our universal model of Sec. 4.2.3 (designed to operate across a range of pre-defined computational budgets) and introduce a lightweight policy network that dynamically decides how many self-attention layers to execute for each input, enabling per-sample adaptation. We implement this with an internal routing mechanism. A special routing token is appended after the question, and we place an MLP layer at the block prior to the first self-attention block that is a candidate for being skipped. That MLP processes the routing token and predicts the optimal configuration for the subsequent self-attention layers. If multiple questions are present, the model conservatively selects the configuration with the highest computational cost among the individual predictions to ensure sufficient processing capacity. Since training a routing mechanism can be unstable [12], we adopt an offline pseudo-labeling approach. First, we run our universal model on a training subset, logging the correctness and token-level losses for each potential layer configuration. We then generate a pseudo-label for the subset by identifying the most efficient configuration. To do this, we first filter for configurations that achieve at least 99% of the full model’s accuracy. From this group, we select the one with the fewest layers and the lowest aggregate loss. This chosen configuration becomes the target label for training the policy network using a standard cross-entropy loss. 4.4 Combining Vision-on-Request with Token Reduction Our approach is orthogonal to existing token reduction methods and can be combined with them for further efficiency gains. To this end, we explore two strategies: (i) combining VISOR with top-performing token pruning methods [50, 53], and (i) designing a simple token packing strategy that works with arbitrary token compression ratios. The latter method is coined VISOR-TR. Additional details can be found in the supplementary material. 5 Experiments We compare our method against state-of-the-art approaches on a wide range of vision-language benchmarks, covering tasks that require both coarse and fine-grained visual understanding. We show that prior methods are competitive on easy tasks, but struggle on harder tasks that require detailed visual reasoning. In contrast, our method consistently outperforms prior works across all benchmarks, particularly excelling on the challenging tasks. 5.1 Experimental setup Model architecture and training details: We build upon the open-sourced LLaVA-OV model [22], which uses a SigLIP-400M [51] vision encoder, a Qwen2 [48] LLM, and a 2-layer MLP connector. The vision encoder operates on 384×384384× 384 image patches, each patch resulting in 729 visual tokens. We insert cross-attention and self-attention layers uniformly across the LLM, at a maximum of 1/3 of the total layers each. We train our model on the same datasets as LLaVA-OV, i.e., (1) the 4M pretraining knowledge data formed by combining synthetically labeled parts of C3M [40], COCO118K [25], BLIP558K [26], SynthDog [19] and Evol-Instruct [5] and (2) the LLaVA-OV Single-Image 3.2M dataset [22], a high-quality mixture formed by combining over 80 datasets. We note that some of the partitions were not made available, hence, in practice, we train on a smaller set (as defined in the LLaVA-OV GitHub repository). Our training follows a similar two-stage procedure. First, we finetune the new attention layers on the 4M knowledge dataset while freezing the rest of the model. Then, we finetune the entire model on the 3.2M high-quality dataset. Training spans 3 epochs across these stages, using the AdamW [30] optimizer with no weight decay, a batch size of 128, and learning rates of 1e−41e-4 and 1e−51e-5 for the first and second stages, respectively. We train on 16 MI300X GPUs using PyTorch [35] and DeepSpeed [38]. This process applies to both universal and independent variants, except that the universal model also samples configurations during training (Sec. 4.2.3, and supplementary material for details). Vision-language benchmarks: We evaluate our models on a comprehensive set of vision-language benchmarks designed to assess diverse aspects of visual understanding. Specifically, we include the following datasets: RealWorldQA [46], ScienceQA [31], GQA [15], MME [57], MMSTAR [7], MMBench [28], POPE [23], AI2D [18], ChartQA [32], TextVQA [41], InfoVQA [33], OCRBench [29], and DocVQA [34]. To better analyze the model’s performance, based on the observations from Sec. 3, we categorize these datasets into two: easy, which involve limited text-to-image interactions, and hard tasks, requiring extensive text-to-image and image-to-image interactions. State-of-the-art baselines: We compare our method against several training-free and training-aware approaches. Since many existing token reduction methods are designed for different architectures (e.g., LLaVA-1.5 [26]) and focus on lower-resolution tasks, we re-implement and evaluate them under a unified setting using the same LLaVA-OV architecture and, where applicable, training data. Details of the re-implementations are provided in the supplemenatary material. 5.2 Comparison with the state-of-the-art We compare our method against state-of-the-art approaches using a shared LLaVA-OV (0.5B) backbone. In addition to the numerical accuracy, we also report average FLOP savings relative to the baseline LLaVA-OV model. Note that we do not take into consideration the vision encoder FLOPs as they are common to all methods. For our method, we always use a single universal VISOR model with routing that ensures adaptability across different tasks. Table 1 summarizes the results. VISOR achieves significant improvements in both accuracy and computational efficiency. On tasks requiring coarse visual context, VISOR matches or exceeds the performance of prior methods while achieving up to 8.6×8.6× FLOP savings. For tasks demanding fine-grained visual reasoning, VISOR outperforms all baselines, including token reduction methods like VisionZip [50], HiRED [1], and M3[4], which struggle with information bottlenecks. When combined with token reduction techniques (VISOR-TR) in Sec. 4.4, our method achieves even greater efficiency (up to 18×18× FLOP savings), while maintaining state-of-the-art accuracy. See the supplementary material for more results, including comparisons using larger backbones (LLaVA-OV 1.5B). Table 1: Comparison with state-of-the-art methods on various vision-language benchmarks. The metric used is accuracy for all datasets, except for MME where we report a score (higher is better; MME values are divided by 20 for normalization purposes). Method Easy Hard Avg. FLOPs Savings RWorldQA SQA GQA MME MSTAR POPE TextVQA AI2D Avg. (Easy) ChartQA OCRBench InfoVQA DocVQA Avg. (Hard) LLaVA-OV [22] 54.0 67.2 58.3 60.6 40.6 88.4 66.0 56.7 61.5 60.9 58.8 40.0 68.7 57.1 1.0×1.0× Downsample [24] 52.8 67.4 57.4 62.4 40.0 86.1 58.4 56.7 60.2 49.3 47.9 27.3 49.5 43.5 4.0×4.0× VisionZip [50] 51.9 67.0 53.0 62.1 38.9 85.5 46.5 53.2 57.3 44.0 27.0 23.9 36.7 32.9 5.7×5.7× VisionZip† [50] 54.7 65.8 55.7 61.9 39.3 86.9 55.7 54.8 59.3 51.2 45.0 27.2 48.8 43.1 5.7×5.7× VisPruner [53] 53.3 65.7 53.2 60.1 38.1 85.9 47.8 53.7 57.2 44.0 28.3 25.4 42.7 35.1 5.7×5.7× SparseVLM [56] 51.0 66.7 48.7 57.2 38.5 77.6 62.7 53.9 57.0 57.2 43.9 33.1 62.3 49.1 4.5×4.5× PyramidDrop [47] 53.1 66.7 53.5 59.4 40.1 86.0 45.5 54.3 57.3 51.1 42.2 30.4 48.0 42.9 4.2×4.2× M3 [4] 54.0 75.1 59.7 63.8 40.9 88.6 67.0 62.5 64.0 64.7 58.0 38.3 65.4 56.6 8.0×8.0× HiRED [1] 52.6 66.3 55.4 61.9 39.2 86.6 56.8 55.3 59.3 47.5 33.9 26.1 48.4 39.0 5.0×5.0× VISOR (Ours) 54.6 75.3 61.8 60.5 40.1 87.6 67.8 61.5 63.6 65.2 61.8 37.6 68.9 58.4 8.6×8.6× VISOR-TR (Ours) 55.4 75.4 60.7 59.5 38.5 87.7 67.4 61.9 63.3 65.3 60.7 37.4 67.7 57.8 18×18× 6 Ablation studies and analysis Unless otherwise specified, all ablations are performed using the LLaVA-OV (0.5B) backbone, trained on the same datasets as the main experiments. The results reported are aggregated across the two task categories for brevity. Table 2: Accuracy comparison when combining VISOR with token reduction methods. Method FLOPs sav. Easy Hard VISOR 8.9×8.9× 63.6 58.4 VISOR-TR [2x] 17.8×17.8× 63.3 57.8 approx. 4x token reduction VISOR-TR [4x] 35.0×35.0× 63.1 56.2 VISOR+ VisionZip 37.0×37.0× 63.3 55.3 VISOR+ VisPruner 39.0×39.0× 63.5 55.9 Table 3: Acc. comparison across configurations and categories. SA CA Easy Hard SA CA Easy Hard 0 6 63.3 51.8 2 8 63.2 55.1 0 8 63.4 52.0 4 8 63.3 56.9 0 10 63.5 52.1 7 8 63.9 58.3 Effect of cross-attention and self-attention layers: Herein, we analyze the impact of varying the number of cross-attention (CA) and self-attention (SA) layers on accuracy. To avoid a potential sampling bias, each configuration corresponds to an independently trained model. From Table 3, we can observe that: (1) cross-attention alone suffices for tasks requiring coarse visual context, with performance saturating around 8 layers; (2) Cross-attention alone is insufficient for tasks demanding fine-grained reasoning, significantly lagging behind the full model; (3) adding self-attention layers substantially boosts performance on fine-grained tasks, with a 7 layer configuration nearly matching the full model. This underscores the need for visual feature refinement in complex tasks. Combining VISOR with token reduction methods: Our method is orthogonal to token reduction techniques and can complement them for greater efficiency. We evaluate the impact of combining VISOR with token reduction methods like VisionZip [50], VisPruner [53], and our token packing strategy (VISOR-TR) from Sec. 4.4, under varying reduction rates. As Table 2 shows, our approach can benefit from token reduction, achieving up to 35×35× FLOPs savings with only a minor drop in accuracy. Notably, more aggressive token reduction rates (e.g., 4×>4×>) lead to larger performance drops on hard tasks, as the information bottleneck becomes more pronounced. Independent vs universal model training: Our final model is trained to support multiple configurations, enabling dynamic adjustment of computational cost during inference. Herein, we analyze the performance trade-offs of this universal training approach compared to independently training models at a few selected budgets. As Table 4 indicates, the universal model matches and, surprisingly, surpasses the performance of independently trained models across different budgets, while providing the flexibility of adaptive inference. This suggests that the universal training approach also acts as a form of regularization, improving generalization across configurations. Table 4: Accuracy comparison between independently trained models and a universally trained model supporting multiple configurations. Both model variants use the same fixed configuration for all samples. SA CA Universal Easy Hard 2 8 × 63.1 55.1 ✓ 63.8 56.3 7 8 × 63.6 58.3 ✓ 64.2 59.1 Efficiency analysis: We analyze the computational efficiency of VISOR by measuring the number of floating-point operations (FLOPs) required for inference. The primary source of savings in our method comes from replacing the expensive full self-attention over all tokens with either text-only self-attention or a cheaper cross-attention mechanism in most layers. The cost of a standard transformer layer is quadratic in the total sequence length, O((Nt+Nv)2d+(Nt+Nv)d2)O((N_t+N_v)^2d+(N_t+N_v)d^2), which is dominated by the large number of visual tokens NvN_v. In contrast, our cross-attention layers have a cost of only O(NtNvd)O(N_tN_vd), and text-only layers are independent of NvN_v. The full, expensive self-attention is computed only in a small, selective subset of layers. Fig. 10 illustrates the computational cost - accuracy tradeoff. Our approach significantly reduces the FLOPs compared to the baseline LLaVA-OV model while offering a better accuracy-efficiency trade-off compared to all prior methods. Note that the FLOPs measured here only account for the transformer layers, excluding the vision encoder, which is common to all methods. We also measure actual inference speedups on real hardware (MI300X GPUs). As our approach is compatible to existing efficient attention implementations (e.g., flash attention [13]), we observe a good correlation between actual speedup and FLOP savings: A full LLaVA-OV model takes 0.07380.0738sec/sample, a token pruning solution (i.e., VisionZip, VisPruner) at 8x reduction factor - 0.02740.0274 sec/sample, while VISOR takes 0.03840.0384 sec/sample at a 8CA-7SA configuration and 0.02610.0261 sec/sample at a 8CA-2SA. The numbers are reported at the max batch size that allows all methods to fit in memory. See supplementary material for additional analysis. Finally, in terms of model size, the CA layers from VISOR introduces a small overhead (less than 7.5%) compared to the baseline LLaVA-OV model, concentrated in the linear projection layers. 7 Conclusion In this work, we proposed Vision-on-Request (VISOR), a method that reduces the inference cost in LVLMs by sparsifying the image-text and image-image interactions without discarding visual information (as in previous token reduction methods). VISOR uses efficient cross-attention to model text-image interactions and few selective self-attention layers for visual feature refinement necessary for fine-grained visual understanding and reasoning. VISOR trains a single universal network on a range of computational budgets and then, during inference, utilizes lightweight policies that dynamically allocate visual computation based on per-task or per-sample complexity. We show that VISOR drastically improves efficiency, and outperforms state-of-the-art token compression methods across a wide range of benchmarks, especially for challenging tasks that require detailed visual understanding. References [1] K. H. I. Arif, J. Yoon, D. S. Nikolopoulos, H. Vandierendonck, D. John, and B. Ji (2025) HiRED: attention-guided token dropping for efficient inference of high-resolution vision-language models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 1773–1781. Cited by: Appendix L, Table 9, §1, §2, §2, §5.2, Table 1. [2] A. Bulat, Y. Ouali, and G. Tzimiropoulos (2025) Compress & cache: vision token compression for efficient generation and retrieval. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: §2, §2. [3] H. Cai, C. Gan, T. Wang, Z. Zhang, and S. Han (2019) Once-for-all: train one network and specialize it for efficient deployment. arXiv preprint arXiv:1908.09791. Cited by: §4.2.3. [4] M. Cai, J. Yang, J. Gao, and Y. J. Lee (2025) Matryoshka multimodal models. In ICLR, Cited by: Appendix L, §1, §2, §2, §2, §5.2, Table 1. [5] G. H. Chen, S. Chen, R. Zhang, J. Chen, X. Wu, Z. Zhang, Z. Chen, J. Li, X. Wan, and B. Wang (2024) Allava: harnessing gpt4v-synthesized data for lite vision-language models. arXiv preprint arXiv:2402.11684. Cited by: §5.1. [6] L. Chen, H. Zhao, T. Liu, S. Bai, J. Lin, C. Zhou, and B. Chang (2024) An image is worth 1/2 tokens after layer 2: plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision, p. 19–35. Cited by: §1, §2. [7] L. Chen, J. Li, X. Dong, P. Zhang, Y. Zang, Z. Chen, H. Duan, J. Wang, Y. Qiao, D. Lin, et al. (2024) Are we on the right way for evaluating large vision-language models?. Advances in Neural Information Processing Systems 37, p. 27056–27087. Cited by: §5.1. [8] Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. (2024) Internvl: scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 24185–24198. Cited by: §1. [9] X. Chu, L. Qiao, X. Zhang, S. Xu, F. Wei, Y. Yang, X. Sun, Y. Hu, X. Lin, B. Zhang, et al. (2024) Mobilevlm v2: faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766. Cited by: §1, §2, §2. [10] X. Chu, Z. Tian, B. Zhang, X. Wang, and C. Shen (2021) Conditional positional encodings for vision transformers. arXiv preprint arXiv:2102.10882. Cited by: §4.2.1. [11] C. Cortes, M. Mohri, and A. Rostamizadeh (2012) Algorithms for learning kernels based on centered alignment. The Journal of Machine Learning Research 13 (1), p. 795–828. Cited by: Appendix M, §3. [12] D. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei (2022) Stablemoe: stable routing strategy for mixture of experts. arXiv preprint arXiv:2204.08396. Cited by: §4.3. [13] T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré (2022) Flashattention: fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems 35, p. 16344–16359. Cited by: §6. [14] W. Hu, Z. Dou, L. H. Li, A. Kamath, N. Peng, and K. Chang (2024) Matryoshka query transformer for large vision-language models. arXiv preprint arXiv:2405.19315. Cited by: §1, §2, §2. [15] D. A. Hudson and C. D. Manning (2019) Gqa: a new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 6700–6709. Cited by: §5.1. [16] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. (2023) Mistral 7b. arXiv preprint arXiv:2310.06825. Cited by: §1. [17] O. Kaduri, S. Bagon, and T. Dekel (2025) What’s in the image? a deep-dive into the vision of vision language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 14549–14558. Cited by: §1. [18] A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi (2016) A diagram is worth a dozen images. In European conference on computer vision, p. 235–251. Cited by: §5.1. [19] G. Kim, T. Hong, M. Yim, J. Nam, J. Park, J. Yim, W. Hwang, S. Yun, D. Han, and S. Park (2022) OCR-free document understanding transformer. In European Conference on Computer Vision (ECCV), Cited by: §5.1. [20] S. Kornblith, M. Norouzi, H. Lee, and G. Hinton (2019) Similarity of neural network representations revisited. In International conference on machine learning, p. 3519–3529. Cited by: §3. [21] Z. Lan, L. Niu, F. Meng, W. Li, J. Zhou, and J. Su (2025) AVG-llava: an efficient large multimodal model with adaptive visual granularity. In Findings of the Association for Computational Linguistics: ACL 2025, p. 16852–16869. Cited by: §2. [22] B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, P. Zhang, Y. Li, Z. Liu, et al. (2024) Llava-onevision: easy visual task transfer. arXiv preprint arXiv:2408.03326. Cited by: Appendix L, Appendix L, Table 7, Table 9, Appendix H, §1, §5.1, Table 1. [23] Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J. Wen (2023) Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355. Cited by: §5.1. [24] C. Liao, W. Wang, Z. Wen, X. Zheng, Y. Wang, H. He, Y. Lyu, L. Jiang, X. Zou, Y. Fu, et al. (2025) Are we using the right benchmark: an evaluation framework for visual token compression methods. arXiv preprint arXiv:2510.07143. Cited by: Table 9, Table 1. [25] T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick (2014) Microsoft coco: common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, p. 740–755. Cited by: §5.1. [26] H. Liu, C. Li, Y. Li, and Y. J. Lee (2024) Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 26296–26306. Cited by: §5.1, §5.1. [27] H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee (2024-01) LLaVA-next: improved reasoning, ocr, and world knowledge. External Links: Link Cited by: Appendix L. [28] Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, et al. (2024) Mmbench: is your multi-modal model an all-around player?. In European conference on computer vision, p. 216–233. Cited by: §5.1. [29] Y. Liu, Z. Li, M. Huang, B. Yang, W. Yu, C. Li, X. Yin, C. Liu, L. Jin, and X. Bai (2024) Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences 67 (12), p. 220102. Cited by: §5.1. [30] I. Loshchilov (2017) Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. Cited by: §5.1. [31] P. Lu, S. Mishra, T. Xia, L. Qiu, K. Chang, S. Zhu, O. Tafjord, P. Clark, and A. Kalyan (2022) Learn to explain: multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems 35, p. 2507–2521. Cited by: §5.1. [32] A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque (2022) Chartqa: a benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244. Cited by: §5.1. [33] M. Mathew, V. Bagal, R. Tito, D. Karatzas, E. Valveny, and C. Jawahar (2022) Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, p. 1697–1706. Cited by: §5.1. [34] M. Mathew, D. Karatzas, and C. Jawahar (2021) Docvqa: a dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, p. 2200–2209. Cited by: §5.1. [35] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. (2019) Pytorch: an imperative style, high-performance deep learning library. Advances in neural information processing systems 32. Cited by: §5.1. [36] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §1. [37] M. Raghu, T. Unterthiner, S. Kornblith, C. Zhang, and A. Dosovitskiy (2021) Do vision transformers see like convolutional neural networks?. Advances in neural information processing systems 34, p. 12116–12128. Cited by: §3. [38] J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He (2020) Deepspeed: system optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, p. 3505–3506. Cited by: §5.1. [39] Y. Shang, M. Cai, B. Xu, Y. J. Lee, and Y. Yan (2024) Llava-prumerge: adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388. Cited by: §2, §2. [40] P. Sharma, N. Ding, S. Goodman, and R. Soricut (2018) Conceptual captions: a cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 2556–2565. Cited by: §5.1. [41] A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach (2019) Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 8317–8326. Cited by: §5.1. [42] X. Tan, P. Ye, C. Tu, J. Cao, Y. Yang, L. Zhang, D. Zhou, and T. Chen (2025) Tokencarve: information-preserving visual token compression in multimodal large language models. arXiv preprint arXiv:2503.10501. Cited by: §2. [43] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. (2023) Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: §1. [44] P. K. A. Vasu, F. Faghri, C. Li, C. Koc, N. True, A. Antony, G. Santhanam, J. Gabriel, P. Grasch, O. Tuzel, et al. (2025) Fastvlm: efficient vision encoding for vision language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 19769–19780. Cited by: Appendix K. [45] P. K. A. Vasu, J. Gabriel, J. Zhu, O. Tuzel, and A. Ranjan (2023) Fastvit: a fast hybrid vision transformer using structural reparameterization. In Proceedings of the IEEE/CVF international conference on computer vision, p. 5785–5795. Cited by: Appendix K. [46] xAI (2024) Grok-1.5 vision preview. Note: https://x.ai/news/grok-1.5vAccessed: 2024-04-12 Cited by: §5.1. [47] L. Xing, Q. Huang, X. Dong, J. Lu, P. Zhang, Y. Zang, Y. Cao, C. He, J. Wang, F. Wu, et al. (2025) Pyramiddrop: accelerating your large vision-language models via pyramid visual redundancy reduction. CVPR. Cited by: Appendix L, Table 9, §1, §2, §2, Table 1. [48] A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. (2024) Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. Cited by: §1, §5.1. [49] C. Yang, Y. Sui, J. Xiao, L. Huang, Y. Gong, C. Li, J. Yan, Y. Bai, P. Sadayappan, X. Hu, et al. (2025) Topv: compatible token pruning with inference time optimization for fast and low-memory multimodal vision language model. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 19803–19813. Cited by: §2. [50] S. Yang, Y. Chen, Z. Tian, C. Wang, J. Li, B. Yu, and J. Jia (2025) Visionzip: longer is better but not necessary in vision language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 19792–19802. Cited by: Appendix L, Appendix L, Table 9, Table 9, §1, §2, §2, §2, §4.4, §5.2, Table 1, Table 1, §6. [51] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer (2023) Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 11975–11986. Cited by: §5.1. [52] C. Zhang, K. Ma, T. Fang, W. Yu, H. Zhang, Z. Zhang, Y. Xie, K. Sycara, H. Mi, and D. Yu (2025) VScan: rethinking visual token reduction for efficient large vision-language models. arXiv preprint arXiv:2505.22654. Cited by: §2. [53] Q. Zhang, A. Cheng, M. Lu, R. Zhang, Z. Zhuo, J. Cao, S. Guo, Q. She, and S. Zhang (2025) Beyond text-visual attention: exploiting visual cues for effective token pruning in vlms. ICCV. Cited by: Appendix L, Table 9, §4.4, Table 1, §6. [54] Q. Zhang, A. Cheng, M. Lu, Z. Zhuo, M. Wang, J. Cao, S. Guo, Q. She, and S. Zhang (2024) [CLS] attention is all you need for training-free visual token pruning: make vlm inference faster. arXiv e-prints, p. arXiv–2412. Cited by: §2, §2. [55] R. Zhang, Y. Lyu, R. Shao, G. Chen, W. Guan, and L. Nie (2024) Token-level correlation-guided compression for efficient multimodal document understanding. arXiv preprint arXiv:2407.14439. Cited by: §2. [56] Y. Zhang, C. Fan, J. Ma, W. Zheng, T. Huang, K. Cheng, D. Gudovskiy, T. Okuno, Y. Nakata, K. Keutzer, et al. (2024) Sparsevlm: visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04417. Cited by: Appendix L, Table 9, §1, §2, §2, Table 1. [57] Y. S. Y. Q. M. Zhang, X. L. J. Y. X. Zheng, K. L. X. S. Y. Wu, R. J. C. Fu, and P. Chen (2021) Mme: a comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394. Cited by: §5.1. Appendix A Identifying promising configurations for adaptive training and inference Considering a model with LSAL_SA self-attention layers, we can create 2LSA2^L_SA different configurations by choosing to execute or skip each self-attention layer. This results in a vast configuration space, making it somewhat impractical to evaluate all of them. Moreover, many configurations may lead to catastrophic performance degradation, as they may skip critical layers needed for certain tasks. Hence, to facilitate the training process, we seek to identify a subset of promising configurations that maintain high performance. This subset can then be used for adaptive training and inference. Figure 6 visualizes the performance of a representative subset of configurations on different datasets, with each row representing a configuration and each column a dataset. The color intensity indicates the relative accuracy achieved by that configuration on the respective dataset. From this visualization, we can identify that: (1) dropping the 1st layer leads to significant performance degradation across all datasets, indicating its critical role; (2) configurations with very few self-attention layers (e.g., 1 or 2) perform poorly on complex tasks, while those with more layers generally yield better results; (3) less vision intensive tasks generally prefer a configuration close to early-exit while more complex tasks benefit from a uniform distribution of self-attention layers. Based on these observations, for a 0.5B model, we subsequently select the following configurations, where each number denotes the location at which a self-attention layer is executed for the vision tokens:[1,4], [1,7], [1,4,7], [1,4,16], [1,7,16], [1,10,16], [1,4,7,16], [1,4,7,22], [1,4,10,16], [1,4,7,10,16], [1,4,7,16,22], [1,4,7,10,16,22], [1,4,7,10,16,19,22]. Figure 6: Performance heatmap for different configurations across datasets. Each row represents a configuration, and each column corresponds to a dataset. The color intensity indicates the relative accuracy achieved by that configuration on the respective dataset. Appendix B Per-dataset saving rate In the main manuscript, for brevity, we report the computational savings aggregated across all datasets. Herein, we provide a more detailed per-dataset analysis. Table 5 summarizes the results. For each variant, the top row indicates the accuracy, while the bottom row shows the FLOPs savings relative to the baseline LLaVA-OV model. The same general pattern holds: easy tasks can be solved with very few self-attention layers, while hard tasks require more layers for optimal performance. Our router is able to correctly identify this trend. Table 5: Per-dataset saving rates. We compare our method against state-of-the-art approaches using a shared LLaVA-OV (0.5B) backbone. For each method, the top row indicates the accuracy, while the bottom row shows the FLOPs savings relative to the baseline LLaVA-OV model. The metrics used are accuracy for most datasets, except for MME where we report a score (higher is better). Method RWQA SQA GQA MME MMSTAR POPE TxtVQA AI2D CQA OCRB InfoVQA DocVQA VISOR 54.6 75.3 61.8 60.5 40.1 87.6 67.8 61.5 65.2 61.8 37.6 68.9 8.4×8.4× 8×8× 12×12× 8.3×8.3× 10×10× 12×12× 6.3×6.3× 12×12× 8×8× 6.6×6.6× 6×6× 6×6× VISOR-TR 55.4 75.4 60.7 59.5 38.5 87.7 67.4 61.9 65.3 60.7 37.4 67.7 24×24× 16×16× 24×24× 17.7×17.7× 17.7×17.7× 24×24× 14.5×14.5× 24×24× 16×16× 15×15× 12×12× 12×12× To provide further insight into the routing mechanism’s behaviour, we present in Figure 7 the layer configurations it chooses for each test set dataset. Two significant observations can be made from this figure: a) the routing mechanism is largely consistent with regard to the computational budget allocated for each dataset, as the configurations chosen for each dataset tend to have a similar number of layers, and b) despite the fact that the original labels are defined in a per-dataset basis, the routing mechanism indeed operates on a per-sample basis, which makes it adaptive to individual samples’ complexity. Figure 7: Layer configuration assignments made by the router for each test dataset. Appendix C Performance across all individual configurations Our universal model is trained by randomly sampling a viable configuration during training. To evaluate the effectiveness of each configuration, Figure 8 illustrates their performance across all downstream tasks. On the easy partition, most configurations achieve similar performance regardless of their computational cost. In contrast, for challenging tasks, performance improves almost linearly with the computational budget, highlighting once more the importance of additional self-attention layers for fine-grained reasoning. Figure 8: Performance for various VISOR configurations. Appendix D Oracle performance analysis To assess the potential of our adaptive inference mechanism, we conduct an oracle analysis where we select the optimal configuration for each sample from our predefined set. Figure 9 illustrates the distribution of the selected configurations across all samples such that the overall accuracy is maximized. These results reinforce the conclusion that most samples can be accurately processed using configurations with very few self-attention layers, while hard tasks require more layers for optimal performance. To find the optimal configuration per sample, we compare the per-config generations token by token to the tokenized ground-truth, and score it as the number of matches up to the first incorrect match. Then, we select the configuration with the most matches. If multiple ones have the same score, we select the one with the minimum number of SA layers (i.e., the one with the most dropped layers). Figure 9: VISOR oracle: smallest amount of layers that maximizes accuracy. Appendix E Additional discussion on efficiency Figure 10: Efficiency comparison - number of FLOPS vs. vision sequence length. In addition to the discussion from the main paper, in Fig. 10 we illustrate the computational cost as a function of the visual sequence length. Our approach significantly reduces the FLOPs compared to the baseline LLaVA-OV model, with longer sequences benefiting more from the reduction. Note that the FLOPs measured here only account for the transformer layers, excluding the vision encoder which is common to all methods. Appendix F Reduced number of tokens vs reduced attention In the main paper, we demonstrated that intermittent attention (VISOR) is effective for hard tasks and orthogonal to token reduction methods. Here, we conduct a head-to-head comparison of these two paradigms under a similar FLOPs reduction rate of approximately 16×. We use our VISOR-TR variant and compare it against two token reduction methods, M3 and VisPruner. To ensure a fair comparison, all models were finetuned end-to-end using the same training procedure, with the only difference being how the amount of input vision tokens is reduced. As shown in Table 7, while all methods perform well on easy datasets, our approach maintains a significant performance advantage on harder tasks, highlighting the benefits of preserving a larger visual context over aggressive token reduction. Table 6: Comparison on various vision-language benchmarks for Qwen2-VL-2B LVLM. Method Easy Hard RealWorldQA SQA GQA MME MSTAR POPE TextVQA AI2D Avg. (Easy) ChartQA OCRBench InfoVQA DocVQA Avg. (Hard) Qwen2-VL-2B 61.1 78.1 60.2 75.3 43.6 87.7 79.3 70.4 69.5 75.0 77.5 64.1 89.5 76.5 VISOR 60.0 82.1 60.8 75.1 49.2 88.9 76.1 75.0 70.9 78.1 76.0 55.9 87.3 74.3 Table 7: Comparison between token reduction vs intermittent attention (VISOR). Method Easy Hard Avg. FLOPs Savings RealWorldQA SQA GQA MME MSTAR POPE TextVQA AI2D Avg. (Easy) ChartQA OCRBench InfoVQA DocVQA Avg. (Hard) LLaVA-OV [22] 54.0 67.2 58.3 60.6 40.6 88.4 66.0 56.7 61.5 60.9 58.8 40.0 68.7 57.1 1.0×1.0× + M3M^3 53.6 75.1 59.1 63.6 41.5 88.1 65.5 62.2 63.6 62.9 54.0 34.7 58.0 52.4 16.0×16.0× + VisPruner train. 53.1 69.2 56.7 60.9 36.9 86.7 55.4 56.3 59.4 52.4 44.2 28.2 52.9 44.4 16.0×16.0× VISOR-TR (Ours) 55.4 75.4 60.7 59.5 38.5 87.7 67.4 61.9 63.3 65.3 60.7 37.4 67.7 57.8 18×18× Appendix G Routing mechanism generalization As described in the main manuscript, the internal routing mechanism responsible for deciding the optimal configuration of self-attention layers to be used for each sample is trained using offline, per-dataset labels extracted from our training set. In this subsection, in order to investigate how the router performs on unseen data, we train it while excluding from its train set three datasets (AI2D, DocVQA, and GQA), and evaluate it on vision-language benchmarks including those datasets. In Table 8, we present the outcomes of this experiment (VISOR-TR-excl) and contrast them with the router when trained on the full train set (VISOR-TR). Our results demonstrate that, even though, naturally, the model behavior is changed, this does not lead to a drop of performance in general or in the excluded datasets in particular, thereby indicating that VISOR is robust and can handle samples outside the train set’s distribution. Table 8: Results for the routing mechanism trained on the full train set (VISOR-TR), contrasted with training excluding samples from AI2D, DocVQA and GQA (VISOR-TR-excl). Method RWQA SQA GQA MME MMSTAR POPE TxtVQA AI2D CQA OCRB InfoVQA DocVQA VISOR-TR 55.4 75.4 60.7 59.5 38.5 87.7 67.4 61.9 65.3 60.7 37.4 67.7 24×24× 16×16× 24×24× 17.7×17.7× 17.7×17.7× 24×24× 14.5×14.5× 24×24× 16×16× 15×15× 12×12× 12×12× VISOR-TR-excl 55.5 75.6 61.6 59.2 38.3 88.3 66.9 62.0 65.0 61.1 37.2 67.6 14.5×14.5× 15.5×15.5× 9.8×9.8× 11.4×11.4× 20×20× 9.8×9.8× 12×12× 14.5×14.5× 16×16× 14.1×14.1× 12×12× 12×12× Appendix H Additional comparison with the state-of-the-art In the main manuscript, we compared our method against state-of-the-art approaches using a shared LLaVA-OV (0.5B) backbone. Herein, we provide additional results using a larger LLaVA-OV (1.5B) backbone and on a different architecture: QwenVL (2B). For the 1.5B case, as no official 1.5B variant is openly available, we re-trained it ourselves fully using the same procedure as described in Li et al. [22]. Additionally, we’ve also re-implemented the baselines under the same unified setting. As the results from Table 9 show, the same conclusions hold and our method continues to outperform existing approaches on the hard partition of the dataset, while providing significant efficiency gains. For QwenVL2 (2B) we report results in Table 6 - the same conclusions hold, with our approach largely matching the full model’s performance using significantly fewer FLOPs. We note however, that in this case, as we don’t have access to the full QwenVL training data, our method is at disadvantage. In practice this difference results in disproportionate swings for certain datasets (e.g: InfoVQA, MMSTAR). Table 9: Comparison with state-of-the-art models on various vision-language benchmarks using a LLaVA-OV 1.5B backbone. The metrics used are accuracy for most datasets, except for MME where we report a score (higher is better). MME values are divided by 20. Method Easy Hard Avg. FLOPs Savings RealWorldQA SQA GQA MME MMSTAR POPE TextVQA AI2D Avg. (Easy) ChartQA OCRBench InfoVQA DocVQA Avg. (Hard) LLaVA-OV [22] 57.3 75.6 58.6 63.7 40.5 88.1 69.9 60.3 64.3 64.2 60.3 46.7 76.6 62.0 1.0×1.0× Downsample [24] 53.6 76.9 58.4 62.5 39.2 86.1 59.5 60.0 62.0 46.7 48.0 29.1 52.6 44.1 4.0×4.0× VisionZip [50] 52.7 74.8 54.2 64.5 38.3 86.2 54.9 57.2 60.4 37.8 33.8 26.3 44.7 35.7 5.7×5.7× VisionZip† [50] 54.6 76.2 57.2 63.9 38.1 87.0 62.0 58.2 62.2 50.0 44.3 30.8 55.1 45.1 5.7×5.7× SparseVLM [56] 52.0 74.8 50.3 58.4 34.5 82.3 60.6 57.1 58.7 51.3 38.9 34.9 57.9 45.7 4.5×4.5× PyramidDrop [47] 56.7 76.0 55.9 63.7 37.5 87.2 60.3 62.1 59.1 47.9 31.9 34.2 54.5 42.1 4.6×4.6× VisPruner [53] 52.9 76.0 53.4 62.3 38.4 83.3 54.8 57.5 59.8 40.0 34.3 28.4 51.2 38.5 5.7×5.7× HiRED [1] 54.8 76.0 56.3 63.2 39.3 85.9 59.8 58.2 61.7 44.1 42.0 27.6 52.3 41.5 5.0×5.0× VISOR(Ours) 57.5 84.7 62.5 68.7 44.9 88.6 68.6 71.7 68.3 68.3 63.7 44.4 75.3 62.9 6.6×6.6× Table 10: VISOR LLava-OV 0.5B results when training with different portions of the training data. We report performance when using the full training set (100%) and a reduced subset (50%). Method Training data % RWQA SQA GQA MME MMSTAR POPE TxtVQA AI2D CQA OCRB InfoVQA DocVQA VISOR 100% 54.6 75.3 61.8 60.5 40.1 87.6 67.8 61.5 65.2 61.8 37.6 68.9 VISOR 50% 56.9 72.3 60.1 60.2 38.6 88.3 65.1 57.3 62.2 58.3 33.8 62.7 Table 11: Comparison between LLaVA-OV 0.5B and VISOR with FastVLM vision encoder. Avg. FLOPs Savings refers only to LLM FLOPs and does not include the cost of vision encoding. Method Vision Encoder Easy Hard Avg. FLOPs Savings RealWorldQA SQA GQA MME MSTAR POPE TextVQA AI2D Avg. (Easy) ChartQA OCRBench InfoVQA DocVQA Avg. (Hard) LLaVA-OV SigLIP-400M 54.0 67.2 58.3 60.6 40.6 88.4 66.0 56.7 61.5 60.9 58.8 40.0 68.7 57.1 1×1× LLaVA-OV FastVLM 52.5 67.4 56.3 58.4 35.8 87.7 61.3 51.5 58.9 57.9 47.6 32.5 51.1 47.3 20×20× VISOR FastVLM 52.3 73.7 56.3 58.8 37.7 86.7 61.9 55.8 60.4 63.0 49.0 33.4 53.1 49.6 60×60× Appendix I Additional details regarding Token Packing To further enhance and validate the efficacy of our approach in the main section, we introduce a light adaptation for token packing, capable of working at non-power-of-two reduction rates. Specifically, after the vision encoder, we reshape the patch embeddings back to their 2D spatial grid, interpolate the grid by a factor of 1/21/ 2 along each spatial dimension for a 2×2× compression ratio, and then apply a space-to-depth transformation (pixel shuffle). This deterministically halves the number of visual tokens with minimal information loss and no added parameters, complementing the computational savings from our sparse attention design. Note that the interpolation factor can be adjusted upwards to accommodate arbitrary reduction rates. I.1 Multi-image performance To further validate the effectiveness of our approach on multi-image inputs, we finetune VISOR on the LLaVA-OV multi-image dataset and compare the performance of the resulting model with LLaVA on the MUIR, Blink, and MMIU benchmarks. As shown in Table 12, VISOR matches or outperforms LLaVA-OV despite being over 3×3× faster. Table 12: Multi-image performance comparison. Method MUIR Blink MMIU LLaVA 26.8 40.4 34.2 VISOR 28.1 39.0 34.5 Appendix J Scaling VISOR with Training Data Size To showcase the ability of our approach to scale with increased data, we train VISOR using different portions of the original training set. In Table 10, we compare the performance of our method when trained on the full dataset (100%) versus a randomly sampled subset containing half of the data (50%). As observed, reducing the amount of training data generally causes a decline in performance. Notably, this degradation is primarily concentrated on harder, vision-intensive tasks (e.g., DocVQA, InfoVQA). In contrast, the performance on simpler datasets (such as MME, POPE, and RWQA) remains similar. These results highlight that VISOR effectively leverages larger training corpuses to progressively improve its accuracy. Appendix K Using VISOR with FastVLM Vasu et al. [44] introduce FastVLM, an efficient LVLM that uses the FastVITHD [45] vision backbone to encode high-resolution inputs efficiently. The FastVLM vision encoder uses a hybrid convolution–attention architecture and outputs a reduced number of visual tokens. While the standard LLaVA-OV with SigLIP-400M produces 729 vision tokens for each 384 × 384 image patch, FastVLM outputs only 36 tokens for the same input size. To test whether VISOR remains effective when the vision encoder already produces a small number of vision tokens, we replace the SigLIP-400M encoder in LLaVA-OV with the FastVLM encoder. We first train a LLaVA-OV 0.5B model using the FastVLM vision encoder, following the original three-stage recipe on 7.1M samples. Then, we train VISOR on top of this model to further reduce FLOPs. As shown in Table 11, VISOR improves over LLaVA-OV with FastVLM on both Easy and Hard benchmarks while achieving 3× additional LLM FLOPs savings (60× vs. 20×). Compared to SigLIP-based LLaVA-OV, the FastVLM version achieves comparable results on Easy benchmarks, but the gap is larger on Hard ones, confirming that aggressive token reduction mainly hurts performance on fine-grained benchmarks. Appendix L Re-implementation of baselines In this section, we detail our re-implementation of the baselines we compare with, along with the hyperparameters used. Since LLaVA-OV uses a SigLIP-400M vision encoder that does not have a CLS token, for all methods that rely on the CLS token’s attention scores to select important tokens, we instead use the average attention each token receives from all other tokens in the sequence, as proposed by Yang et al. [50]. VisionZip. VisionZip [50] is a token reduction method that selects the most important tokens, named dominant tokens, using the visual encoder’s attention scores. To avoid losing information, the remaining tokens are merged into contextual tokens based on semantic similarity. In our experiments, we adapt the official VisionZip LLaVA-Next [27] code to LLaVA-OV. Unlike LLaVA-Next, the LLaVA-OV image processing AnyRes strategy applies bilinear interpolation when the number of tokens exceeds a threshold. To prevent errors from interpolating removed tokens and to stay close to the original design, we remove this step. Yang et al. [50] also introduce VisionZip†, a trained version that fine-tunes the cross-modality projector. For a fair comparison, we train VisionZip† on the LLaVA-OV Single-Image 3.2M dataset [22]. In all VisionZip and VisionZip† experiments, we set the number of retained tokens per patch to 128, split into 104 dominant tokens and 24 contextual tokens. VisPruner. VisPruner [53] is a training-free token pruning method that retains visual tokens based on visual attention scores. It first selects the important tokens, i.e., those with the highest scores, and then removes duplicates by keeping only diverse tokens based on their similarity. As with VisionZip, we adapt the official VisPruner LLaVA-Next code to the LLaVA-OV backbone. In our experiments, we retain 128 tokens per patch, split into 96 important tokens and 32 diverse tokens. HiRed. High-Resolution Early Dropping (HiRed) [1] is a plug-and-play token-reduction method designed to work under a fixed budget. It targets high-resolution LVLMs (i.e., LLaVA-Next) and drops tokens before they reach the LLM. The key idea is to evaluate the visual content of each image patch using the attention scores of the full image, then assign a budget to each patch accordingly. Within each patch, the most informative tokens are kept and passed to the LLM. In our experiments, we adapt the official code to LLaVA-OV, following the same approach used for VisionZip and VisPruner. In our experiments, we use the same hyperparameters as the original implementation, setting a token budget of 20%. M3. Matryoshka Multimodal Models (M3) [4] represent visual content as a nested set of tokens capturing information at different levels of detail, from coarse to fine. The visual tokens from the encoder are grouped into several coarse-to-fine levels, where the coarser tokens XSi−1X_S_i-1 are obtained from the finer tokens XSiX_S_i using average pooling. M3 does not add any extra parameters. For a fair comparison, we train M3 on the LLaVA-OV Single-Image 3.2M dataset [22], updating both the vision encoder and the LLM weights. In our experiments, we define a set of scales XSii=1M\X_S_i\_i=1^M that reduce the number of visual tokens by factors of 1, 4, 8, and 16. For a fair comparison with our method, we report the results at an 8× reduction. PyramidDrop. PyramidDrop [47] is a progressive token pruning method that gradually reduces the number of visual tokens as the LLM depth increases. Specifically, the LLM layers are split into stages, and at the beginning of each stage, the number of visual tokens is reduced based on a predefined set of reduction rates, which are defined on a per-stage basis. At each pruning step, the input features are split into visual and text features, and then from the text features, only the features corresponding to the position of the last token of the user’s query or instruction are kept, resulting in Nv×dN_v× d visual features and a single text feature vector. Then, the next attention layer to be executed is applied over these features, with the text features as the query, resulting in image-text attention weights that are interpreted as a per-visual token importance score. These scores, together with a per-stage pre-defined drop-rate, are used to keep the top-k scoring visual tokens, with k as the target visual token to keep. This procedure is applied progressively throughout the LLM’s depth at the beginning of each stage. In our implementation, for LLaVA-OV 0.5B with a 24-layer LLM, we used 4 stages defined as (1,2−6,7−12,13−24)(1,2-6,7-12,13-24) with drop-rates of (1.0,0.3,0.2,0.1)(1.0,0.3,0.2,0.1) for easy datasets, and (1−4,5−10,11−16,17−24)(1-4,5-10,11-16,17-24) with drop-rates of (1.0,0.5,0.25,0.125)(1.0,0.5,0.25,0.125) for hard datasets with average FLOPs saving of 4.2×4.2×. As for LLaVA-OV 1.5B with a 28-layer LLM we used 5 stages defined as (1,2,3−6,5−10,11−28)(1,2,3-6,5-10,11-28) with drop-rates of (1.0,0.5,0.3,0.2,0.1)(1.0,0.5,0.3,0.2,0.1) for easy datasets, and (1−2,3−8,8−12,13−18,19−28)(1-2,3-8,8-12,13-18,19-28) with drop-rates of (1.0,0.75,0.5,0.25,0.125)(1.0,0.75,0.5,0.25,0.125) for hard datasets with an average FLOPs saving of 4.6×4.6×. SparseVLM. Similar to PyramidDrop, SparseVLM [56] leverages the self-attention maps in the VLLM to identify the text tokens that are most relevant to the image, and uses them as raters to score the importance of visual tokens. SparseVLM then adaptively determines how many visual tokens to prune at each layer based on the rank of the text-to-vision attention matrix, and further reduces information loss with a token recycling step that aggregates the most informative pruned tokens into a smaller set of reconstructed tokens. It is a plug-and-play method that does not require additional parameters or fine-tuning. Appendix M Additional details on Centered Kernel Alignment (CKA) In Figure 3 in the main manuscript, we showed how vision features evolve across layers within the LLM transformer of the LVLM by computing the Centered Kernel Alignment (CKA) [11]. More specifically, to compute it, let X∈ℝn×dX ^n× d and Y∈ℝn×dY ^n× d represent the vision features extracted from two different layers, where n is the number of tokens and d is the feature dimension. The CKA computation begins by forming the Gram matrices K=XXTK=X^T and L=YYTL=Y^T. These matrices are then centered using the centering matrix H=In−1n1nH=I_n- 1n1_n, where InI_n is the identity matrix and 1n1_n is an n×n× n matrix of ones. The centered Gram matrices are given by K~=HKH K=HKH and L~=HLH L=HLH. The CKA similarity between K~ K and L~ L can then be computed as: CKA(K~,L~)=⟨K~,L~⟩F‖K~‖F‖L~‖F,CKA( K, L)= K, L _F\| K\|_F\| L\|_F, (4) where ⟨⋅,⋅⟩F ·,· _F denotes the Frobenius inner product, and ∥⋅∥F\|·\|_F represents the Frobenius norm.