Paper deep dive
Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Models
Jingdi lei, Junxian Li, Di Zhang, Zhanqiu Zhang, Yiwen Guo, Soujanya Poria
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Long visual token sequences often account for a substantial fraction of the computational overhead in multimodal large language models~(MLLMs). Existing approaches reduce this cost by pruning redundant visual tokens, but permanently discard visual evidence that may become useful in subsequent layers. We instead ask whether all visual tokens can be preserved while reducing the cost of repeatedly evolving the representations through the Transformer. To answer this question, we perform low-rank interventions on visual-to-text information flow. We find that, after visual-to-text attention is blocked, restoring only a few directions recovers most of the lost accuracy, suggesting the relevant visual influence is concentrated in a low-dimensional subspace. We further observe strong predictability in layer-specific visual states: lightweight MLPs approximate them with high cosine similarity and low reconstruction error. Motivated by these findings, we propose $\delta$-Vision, which replaces repeated Transformer evolution of visual tokens with lightweight low-rank adapters that construct layer-wise visual memories while preserving all visual tokens for text retrieval. Across image and video benchmarks, $\delta$-Vision achieves higher accuracy than visual token pruning baselines at comparable or lower computation, while delivering competitive inference efficiency without discarding visual tokens.
Tags
Links
- Source: https://arxiv.org/abs/2609.34972v1
- Canonical: https://arxiv.org/abs/2609.34972v1
Trouble viewing inline? Open PDF directly ā
Full Text
73,851 characters extracted from source content.
Expand or collapse full text
1]Nanyang Technological University 2]Shanghai Jiao Tong University 3]Fudan University 4]LIGHTSPEED 5]Independent Researcher [ Github] DeCLaRe Lab [ Correspondence] Jingdi Lei, Zhanqiu Zhang, Yiwen Guo, Soujanya Poria, Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Models Jingdi Lei Junxian Li Di Zhang Zhanqiu Zhang Yiwen Guo Soujanya Poria Affiliation: [ Affiliation: [ Affiliation: [ Affiliation: [ Affiliation: [ September 28, 2026 Abstract Long visual token sequences often account for a substantial fraction of the computational overhead in multimodal large language models (MLLMs). Existing approaches reduce this cost by pruning redundant visual tokens, but permanently discard visual evidence that may become useful in subsequent layers. We instead ask whether all visual tokens can be preserved while reducing the cost of repeatedly evolving the representations through the Transformer. To answer this question, we perform low-rank interventions on visual-to-text information flow. We find that, after visual-to-text attention is blocked, restoring only a few directions recovers most of the lost accuracy, suggesting the relevant visual influence is concentrated in a low-dimensional subspace. We further observe strong predictability in layer-specific visual states: lightweight MLPs approximate them with high cosine similarity and low reconstruction error. Motivated by these findings, we propose Ī“ā-VisionĪ“-Vision, which replaces repeated Transformer evolution of visual tokens with lightweight low-rank adapters that construct layer-wise visual memories while preserving all visual tokens for text retrieval. Across image and video benchmarks, Ī“ā-VisionĪ“-Vision achieves higher accuracy than visual token pruning baselines at comparable or lower computation, while delivering competitive inference efficiency without discarding visual tokens. 1 Introduction Multimodal large language models (MLLMs) have extended large language models (LLMs) beyond textual reasoning by endowing them with visual perception and understanding (Liu et al., 2024a; OpenAI, 2024; Qwen Team, 2025; Kimi Team, 2025; GLM, 2026), and have demonstrated remarkable capabilities across a diverse range of multimodal tasks, including image captioning (Alayrac et al., 2022), visual question answering (VQA), video understanding (Qwen Team, 2025), and multimodal reasoning (Yue et al., 2024). However, such impressive performance is accompanied by substantial computational overhead, particularly for high-resolution images and videos which are represented by long sequences of visual tokens (Yang et al., 2025; Shang et al., 2025). This computational cost is further amplified because these visual tokens participate in computations at every single Transformer layer of the inner LLM. As a result, efficiently handling long visual token sequences has become an important challenge for scalable multimodal inference. Existing approaches (Yang et al., 2025; Wen et al., 2025a; Wen et al., 2025b) have explored reducing this cost by pruning redundant visual tokens, thereby shortening the visual sequence processed by the language model. While effective, such removal is irreversible, once a visual token is discarded, the corresponding visual evidence can no longer be accessed by subsequent layers (Chen et al., 2026; Yang et al., 2026; Qian et al., 2026). Rather than whether asking visual tokens can be safetly discarded, we explore a complementary direction: preserving all visual tokens while reducing their processing cost within the LLM. Specifically, do visual tokens require full computation at every Transformer layer, or can the necessary visual information be integrated into the text stream more efficiently? To investigate this possibility, we first examine effective dimensionality required for visual information to influence the text stream. Keeping all visual tokens and their positions unchanged, we project each visual hidden state onto a low-rank subspace and reconstruct it before subsequent computation (Details are provided in Appendix H). As shown in Figure 1(a), much of the original model performance can be retained at ranks substantially smaller than the native hidden dimension, particularly for LLaVA-1.5-7B (Liu et al., 2024a). This observation suggests that visual redundancy exists not only across tokens, but also in the computation used to update their representations across layers. More importantly, these layer-specific visual states also exhibit strong predictability. We train a separate lightweight MLP with a dādād\!ā\!d\!ā\!d architecture for each layer to predict its visual states from the initial visual embeddings. As shown in Figure 1(b), these MLPs approximate the Transformer-evolved states with high cosine similarity and low reconstruction error. These observations suggest that preserving useful visual information may not require repeatedly computing full-dimensional visual states through every Transformer layer. (a) Accuracy recovery under low-rank approximation. (b) Layer-wise predictability of visual representations. Figure 1: Analysis of the dimensional structure of visual representations. (a) Downstream accuracy under different retained ranks of visual representation. (b) Layer-wise reconstruction quality measured by cosine similarity and reconstruction MSE. Motivated by these observations, we develop Ī“ā-VisionĪ“-Vision. Rather than propagating visual tokens through full self-attention and feed-forward computation, Ī“ā-VisionĪ“-Vision directly constructs the visual memory required at each layer using a low-rank adapter, while keeping the language-model backbone frozen. We instantiate this idea with an embedding adapter that predicts each layerās visual memory directly from the initial embeddings and a recurrent adapter that progressively updates the predicted memory across layers. At each layer, the predicted memory is mapped through the original frozen key and value projections. Consequently, text tokens retain access to every visual token, while visual queries, visual attention outputs, and visual feed-forward computation are removed. In this way, Ī“ā-VisionĪ“-Vision changes how visual states are constructed rather than which visual evidence remains accessible, providing an alternative efficiency axis to visual token pruning. Extensive experiments across different MLLM backbones and a broad range of image, multi-image, and video benchmarks demonstrate that Ī“ā-VisionĪ“-Vision achieves a strong accuracy-efficiency trade-off. On Qwen3-VL-4B, Ī“ā-VisionĪ“-Vision reaches an average score of 74.4 under a computational budget comparable to pruning methods that discard 95% of visual tokens, outperforming the strongest baseline by 9.1 points and even matching the best pruning result obtained while retaining 20% of the tokens. The effectiveness of visual-memory prediction further generalizes across different scales and multi-image benchmarks. On Video-MME, Ī“ā-VisionĪ“-Vision requires only 17.90% of the FLOPs of the uncompressed model and achieves 1.30Ć1.30Ć total speed and 1.50Ć1.50Ć prefill speedup, while maintaining a prefill efficiency comparable to token-pruning baselines despite preserving all visual tokens. Our contributions can be summarized as follows: ā¢ We provide empirical evidence that preserving visual information does not necessarily require full layer-wise Transformer computation. By keeping all visual tokens, we show that layer-wise visual states are highly compressible in the hidden-channel dimension and exhibit strong predictability across layers. ā¢ We propose Ī“ā-VisionĪ“-Vision, a lightweight visual-memory prediction framework that replaces repeated visual-token Transformer updates with low-rank layer-wise adapters while preserving all visual tokens for text retrieval ā¢ We demonstrate the effectiveness and generality of Ī“ā-VisionĪ“-Vision across different MLLM backbones and diverse image, multi-image, and video benchmarks, achieving stronger accuracyāefficiency trade-offs than visual token pruning methods under comparable computational budgets. 2 Preliminaries Consider a multimodal large language model that receives a visual input I and a textual prompt x1:Tx_1:T. A vision encoder followed by a multimodal projector converts the visual input into NvN_v visual embeddings: E=[e1,ā¦,eNv]āāNvĆd,E=[e_1,...,e_N_v] ^N_vĆ d, (1) where d is the hidden dimension of the language model. The visual embeddings are concatenated with the text embeddings and processed by an L-layer Transformer. Let Hl=[Vl;Tl]āā(Nv+Nt)Ćd,VlāāNvĆd,TlāāNtĆd,H_l=[V_l;T_l] ^(N_v+N_t)Ć d, V_l ^N_vĆ d, T_l ^N_tĆ d, (2) denote the hidden states entering layer l, where VlV_l and TlT_l correspond to the visual and textual states. At the input layer, V0=EV_0=E. Visual tokens play two roles in a MLLM. First, they serve as visual context for the language stream: textual queries retrieve visual information through the keys and values associated with visual tokens. Second, visual tokens are themselves active Transformer states. At every layer, they generate their own queries, receive attention outputs, pass through the feed-forward network, and are consequently transformed into new layer-specific visual states. We refer to the sequence: E=V0āV1āāÆāVL,E=V_0ā V_1āĀ·ā V_L, (3) produced by the Transformer as the layer-wise evolution of visual tokens. Importantly, the representation exposed to the text stream is layer dependent rather than directly from the initial visual embeddings E. This design naturally allows visual representations to evolve jointly with the language stream, but it also requires all NvN_v visual tokens to undergo attention and feed-forward computation at every layer. For high-resolution images or videos with long visual sequences, this repeated computation becomes a substantial component of multimodal inference cost. Our goal is not to reduce NvN_v, but to reduce the computation to obtain the layer-specific visual representation. 3 Ī“ā-VisionĪ“-Vision Ī“ā-VisionĪ“-Vision replaces the original visual-token update pathway with lightweight layer-wise visual memory prediction, while keeping the textual part unchanged. Figure 2 provides an overview of this design. At layer l, a low-rank adapter constructs a visual memory MlM_l, which is then projected by the keyāvalue modules and accessed by text queries. Visual queries, visual attention outputs, and visual feed-forward updates are omitted. We develop two variants for constructing MlM_l: embedding adapter that predicts the visual memory from the initial visual embeddings, and recurrent adapter that updates it from the preceding visual memory. 3.1 Layer-Wise Visual Memory Prediction We parameterize the layer-wise visual memory as a full-dimensional residual correction. Given an input visual state XāāNvĆdX ^N_vĆ d, the adapter at layer l is defined as Alā(X)=X+Īlā(X),Īlā(X)=Ļā”(XāDl)āUl,A_l(X)=X+ _l(X), _l(X)=Ļ(XD_l)U_l, (4) where DlāādĆrD_l ^dĆ r and UlāārĆdU_l ^rĆ d are the down- and up-projection matrices, rāŖdr d is the bottleneck dimension, and Ļā”(ā )Ļ(Ā·) denotes the SiLU activation function. We initialize UlU_l to zero, so that AlA_l starts as an identity mapping. The key property of this parameterization is that it constrains how the visual memory is updated rather than the memory itself. For every visual token, the residual update Īlā(X) _l(X) lies in the subspace spanned by UlU_l, whose dimension is at most r. Equivalently, rāaānākā(Īlā(X))ā¤rrank ( _l(X) )ā¤ r. Alā(X)A_l(X) remains a full d-dimensional representation through the residual connections. Ī“ā-VisionĪ“-Vision assumes that the layer-specific displacement required to adapt visual information for a given Transformer can be represented within a low-dimensional hidden-channel subspace. We instantiate this low-dimensional update in two forms: Embedding Adapter. The embedding variant predicts the memory of each layer directly from the initial visual embeddings, Ml=Alā(E)=E+Īlā(E).M_l=A_l(E)=E+ _l(E). (5) Every layer learns its own low-dimensional displacement from the same initial visual representation. The resulting memories are therefore conditionally independent across layers given E, allowing all MlM_l to be constructed without explicitly propagating visual states through preceding Transformer layers. This formulation represents our hypothesis: useful layer-specific visual states can be obtained as lightweight corrections to the initial visual evidence. Figure 2: Overview of Ī“ā-VisionĪ“-Vision. Top: Ī“ā-VisionĪ“-Vision replaces the original visual hidden-state evolution with lightweight low-rank adapters that constructs layer-wise visual representations. Bottom left: we consider two visual-memory variants: Embedding Adapter, which predicts visual states from the initial visual embeddings, and Recurrent Adapter, which propagates a compact visual state across layers. Bottom right: Ī“ā-VisionĪ“-Vision is trained with Supervised-KD, where the frozen vanilla MLLM serves as the teacher and provides token-level distributional supervision under teacher forcing. Recurrent Adapter. The recurrent variant instead constructs a trajectory of visual memories through successive low-dimensional corrections, M0=E,Ml=Al(Mlā1)=Mlā1+Īl(Mlā1),l=1,ā¦,L.M_0=E,M_l=A_l(M_l-1)=M_l-1+ _l(M_l-1),l=1,...,L. (6) Unlike the embedding variant, this formulation preserves an explicit layer-to-layer state. However, each transition is restricted to a low-dimensional displacement rather than a full Transformer update. The recurrent construction can therefore be viewed as a lightweight approximation to visual-state evolution, in which the memory trajectory is formed by accumulating a sequence of compact layer-specific corrections. 3.2 Visual Memory as Context Having constructed the layer-specific visual memory MlM_l, we use it as read-only keyāvalue context for the textual stream. Specifically, the predicted memory is processed by the original frozen layer normalization and keyāvalue projections, Klv=WK,lāLNlā”(Ml),Vlv=WV,lāLNlā”(Ml).K_l^v=W_K,lLN_l(M_l), V_l^v=W_V,lLN_l(M_l). (7) Let HlāāNtĆdH_l ^N_tĆ d denote the textual hidden states entering layer l. Text tokens follow the original Transformer computation and produce Qlt=WQ,lāLNlā”(Hl),Klt=WK,lāLNlā”(Hl),Vlt=WV,lāLNlā”(Hl).Q_l^t=W_Q,lLN_l(H_l), K_l^t=W_K,lLN_l(H_l), V_l^t=W_V,lLN_l(H_l). (8) The textual queries then attend jointly to the predicted visual memory and the textual context, Olt=Attnā”(Qlt,[Klv;Klt],[Vlv;Vlt]),O_l^t=Attn (Q_l^t,[K_l^v;K_l^t],[V_l^v;V_l^t] ), (9) where the original causal attention structure is preserved. The resulting attention output contains only NtN_t query positions and is subsequently processed by the original output projection, residual connection, and feed-forward network of the frozen language model. This formulation separates visual-state construction from visual information retrieval. In a conventional MLLM, visual tokens simultaneously provide keys and values to the text stream and act as active Transformer states. In Ī“ā-VisionĪ“-Vision, only the former role is retained. The predicted memory contributes KlvK_l^v and VlvV_l^v at every layer, so textual queries can still retrieve information from every visual token, but no QlvQ_l^v, visual attention output, or visual feed-forward update is computed. Importantly, MlM_l is not updated by the textual attention computation. Once consumed as keyāvalue context at layer l, it is discarded, and the memory for the next layer Ml+1M_l+1 is obtained directly from the corresponding embedding or recurrent adapter. Thus, visual memories form an external layer-wise context sequence rather than a set of hidden states propagated through the Transformer backbone. This allows Ī“ā-VisionĪ“-Vision to preserve access to all visual tokens simultaneously. 3.3 Training Objective We optimize only the visual-memory adapters while keeping the vision encoder, multimodal projector, and language-model backbone frozen. The original MLLM serves as the teacher, while Ī“ā-VisionĪ“-Vision acts as the student. Inspired by Agarwal et al. (2024), we use Supervised-KD as our training objective. Specifically, for each training example, both models are conditioned on the same multimodal input and the same ground-truth answer prefix. At answer position t, teacher and student therefore predict the next token yty_t conditioned on (I,x1:T,y<t) (I,x_1:T,y_<t ). We minimize the forward KL divergence between the teacher and student predictive distributions over ground-truth answer positions, āKD=1|Ī©|ātāĪ©DKL(pT(ā ā£I,x1:T,y<t)ā„pS(ā ā£I,x1:T,y<t)),L_KD= 1| | _tā D_KL (p_T(Ā· I,x_1:T,y_<t)\,\|\,p_S(Ā· I,x_1:T,y_<t) ), (10) where Ī© denotes the set of answer-token positions. Since the backbone is frozen, the supervision is absorbed entirely by the lightweight visual-memory predictors, encouraging them to provide layer-wise visual context that preserves the teacherās output behavior. An ablation against standard SFT and on-policy distillation is provided in Appendix E, showing that Supervised-KD achieves the best average performance while avoiding the substantial rollout cost of on-policy training. 4 Experiments 4.1 Experimental Setup Models and Training. Unless otherwise specified, we use Qwen3-VL-4B-Instruct (Qwen Team, 2025) (hereafter referred to as Qwen3-VL-4B) with the embedding adapter as the default Ī“ā-VisionĪ“-Vision configuration. All training configurations used in this work are provided in Appendix A. Table 1: Results of visual token compression methods under 20% and 5% visual-token retention. Method MMStar RWQA GQA MMB MMB-CN MME POPE SQA VQA-v2 Avg. Upper Bound (100% Retention) Qwen3-VL-4B 64.9 71.2 61.6 87.5 87.7 84.7 89.3 93.3 80.9 80.1 Training-Free Retain 20% Visual Tokens FastV(ECCV24) 49.5 55.0 47.3 79.2 77.8 76.6 77.6 83.2 65.8 68.0 VisionZip(CVPR25) 51.1 61.7 52.9 82.0 80.8 78.6 82.6 85.2 70.8 71.7 DivPrune(CVPR25) 53.2 63.9 55.5 84.2 83.5 81.9 86.3 84.4 75.3 74.2 SparseVLM(ICML25) 53.2 45.1 43.1 73.2 71.1 66.3 77.8 76.3 68.1 63.8 DART(EMNLP25) 51.4 63.9 53.5 82.8 82.0 77.9 84.8 85.0 72.1 72.6 ZOO-Prune(CVPR26) 49.3 54.1 50.4 80.5 79.5 78.2 78.7 79.7 70.5 69.0 Retain 5% Visual Tokens FastV(ECCV24) 34.8 43.2 39.9 59.4 56.6 67.5 60.9 72.4 50.6 53.9 VisionZip(CVPR25) 39.1 47.8 44.6 71.6 67.9 73.5 66.9 75.2 59.6 60.7 DivPrune(CVPR25) 42.8 57.1 45.0 75.9 76.0 74.5 73.4 77.0 65.7 65.3 SparseVLM(ICML25) 37.5 42.1 39.2 53.0 51.3 57.3 59.5 72.1 51.0 51.4 DART(EMNLP25) 42.5 52.0 43.5 73.6 73.8 68.0 71.0 79.4 63.7 63.1 ZOO-Prune(CVPR26) 37.5 44.8 42.2 69.9 69.6 68.5 61.9 75.6 58.4 58.7 Training-Based (Retain 5% Visual Tokens) LLaVA-Mini(ICLR25) 40.8 51.1 47.3 78.2 78.4 70.7 78.0 76.1 64.3 65.0 EPIC(NeurIPS25) 38.4 58.8 49.2 77.8 76.2 74.7 81.2 79.6 66.0 66.9 Ī“ā-VisionĪ“-Vision(Emb. Adapter) 54.4 62.9 56.7 84.5 82.8 80.1 87.6 82.3 78.0 74.4 Ī“ā-VisionĪ“-Vision(Rec. Adapter) 56.9 64.4 58.4 85.0 84.6 80.7 87.8 82.3 78.5 75.4 Evaluations. We evaluate Ī“ā-VisionĪ“-Vision across single-image, multi-image, and video understanding benchmarks. For single-image evaluation, we use MMStar (Chen et al., 2024b), RealWorldQA (RWQA) (xAI, 2024), GQA (Hudson and Manning, 2019), MMBench (MMB) (Liu et al., 2024c), MMBench-CN (MMB-CN) (Liu et al., 2024c), MME (Fu et al., 2025a), POPE (Li et al., 2023), ScienceQA (SQA) (Lu et al., 2022), VQA-v2 (Goyal et al., 2019), covering general multimodal reasoning, real-world visual understanding, compositional reasoning, scientific reasoning, and open-ended visual question answering. For more complex visual inputs, we evaluate multi-image understanding on MuirBench (Wang et al., 2025) and video understanding on Video-MME (Fu et al., 2025b) and MVBench (Li et al., 2024). For each benchmark, we report per-question accuracy to facilitate consistent averaging and comparison across benchmarks. The overall score is the unweighted mean of all benchmark scores. Since Ī“ā-VisionĪ“-Vision primarily reduces the cost of processing visual tokens, we compare Ī“ā-VisionĪ“-Vision against several training-free visual token pruning methods, including FastV (Chen et al., 2024a), VisionZip (Yang et al., 2025), SparseVLM (Zhang et al., 2025b), DART (Wen et al., 2025a), DivPrune (Alvar et al., 2025) and Zoo-Prune (Kim et al., 2026), as well as training-based methods LLaVA-Mini (Zhang et al., 2025a) and EPIC (Wen et al., 2025b). Our main comparisons consider pruning at 20% and 5% visual-token retention, with 15% and 10% retention results reported in the appendix. For cross-backbone evaluation, we further compare with DART and DivPrune on LLaVA-1.5-7B (Liu et al., 2024a), Qwen3-VL-30B-A3B (Qwen Team, 2025), and Qwen3.5-4B (Qwen Team, 2026), and evaluate LLaVA-1.5-13B, LLaVA-v1.6-Mistral-7B (Liu et al., 2024b), and Qwen3-VL-8B in Appendix F. All the baselines follow their original pruning schedules when computing the retention rate: the initial full-token layers (layers 0ā1) are excluded for FastV, DART, and SparseVLM, while all LLM layers are included for DivPrune, VisionZip, and ZOO-Prune. 4.2 Main Results Results on Single-Image. Table 1 compares Ī“ā-VisionĪ“-Vision with both training-free visual token pruning and training-based compression methods on Qwen3-VL-4B. Ī“ā-VisionĪ“-Vision achieves an average score of 74.4, retaining 92.9% of the uncompressed model performance. Compared with the strongest 5%-retention pruning baseline, DivPrune, Ī“ā-VisionĪ“-Vision improves the average score by 9.1 points and achieves higher accuracy across benchmarks. It also outperforms the training-based baselines LLaVA-Mini and EPIC by 9.4 and 7.5 points, respectively. Notably, Ī“ā-VisionĪ“-Vision slightly surpasses the best result obtained by pruning methods that retain 20% of the visual tokens. The recurrent adapter further improves the average score from 74.4 to 75.4 over the embedding adapter. This suggests that the embedding adapter already captures much of the layer-specific visual information, while lightweight cross-layer evolution can further improve the predicted visual memory. These results demonstrate that compressing visual computation along the hidden dimension provides a complementary efficiency axis to token pruning (we further verify that the two mechanisms can be combined in practice, results with DART and DivPrune are provided in Appendix I). Results at 10% and 15% retention ratios show the same trend and are reported in Appendix B. Efficiency. Table 2: Efficiency comparison on Video-MME under 5% visual-token retention. Total Speedup measures the overall inference speedup, including both prefill and decoding. Method FLOPs Peak Mem. Total Prefill (%) (GB) Speedup Speedup Upper Bound Qwen3-VL-4B 100.00 12.94 1.00 1.00 Retain 5% Visual Tokens FastV 21.49 11.26 1.22 1.41 DART 21.23 11.12 1.13 1.24 VisionZip 16.62 10.58 1.30 1.45 DivPrune 16.58 10.60 1.30 1.47 ZOO-Prune 23.90 11.76 1.23 1.29 SparseVLM 21.60 11.27 1.19 1.36 Ī“ā-VisionĪ“-Vision 17.90 12.97 1.30 1.50 Table 2 compares the efficiency of Ī“ā-VisionĪ“-Vision with token pruning methods on Video-MME. Ī“ā-VisionĪ“-Vision reduces the computation to 17.90% of the uncompressed model FLOPs, comparable to the 16.58ā23.90% range of methods retaining only 5% of visual tokens. In practice, Ī“ā-VisionĪ“-Vision achieves a 1.30Ć1.30Ć end-to-end speedup and a 1.50Ć1.50Ć prefill speedup over the base model, with prefill efficiency comparable to the strongest pruning baselines while retaining all visual tokens. Importantly, this comparable efficiency is accompanied by substantially better task performance. These results highlight that inference savings can be obtained without shortening the visual sequence. This provides a complementary efficiency mechanism to sequence-length reduction, while retaining access to the complete visual evidence at every layer. We provide additional results for the 20% visual-token retention setting in Appendix D. Generalization across Backbones. Table 3: Results of different visual token compression methods across various MLLMs. Vanilla denotes the uncompressed upper bound. DART and DivPrune retain 5% of the visual tokens. Model Method MMStar RWQA GQA MMB MMB-CN MME POPE SQA VQA-v2 Avg. LLaVA-1.5-7B Vanilla 37.0 54.9 61.2 72.5 67.9 71.7 83.4 65.6 75.9 65.6 DART 29.3 46.5 51.6 63.8 55.7 68.9 68.1 69.0 61.5 57.2 DivPrune 31.3 49.0 53.4 66.7 57.3 66.3 79.3 64.5 70.5 59.8 Ī“ā-VisionĪ“-Vision 35.4 51.4 53.8 69.9 63.7 68.7 79.2 65.4 69.5 61.9 Qwen3-VL-30B-A3B Vanilla 70.3 72.3 65.2 89.4 90.8 89.4 87.9 96.3 82.6 82.7 DART 46.2 52.3 50.1 77.2 77.9 79.2 74.3 85.9 66.7 67.8 DivPrune 45.6 62.9 54.9 79.3 80.8 83.0 77.7 85.0 72.7 71.3 Ī“ā-VisionĪ“-Vision 62.4 65.1 62.4 86.5 86.8 87.0 87.8 88.6 80.7 78.6 Qwen3.5-4B Vanilla 48.9 67.2 63.4 84.3 77.2 84.0 88.7 78.4 80.0 74.7 DART 30.5 48.0 46.1 67.8 61.9 71.2 68.3 64.3 60.8 57.7 DivPrune 34.8 56.0 48.8 74.2 64.0 75.8 73.4 71.5 65.6 62.7 Ī“ā-VisionĪ“-Vision 40.1 65.8 60.5 81.3 75.1 79.7 89.0 74.8 77.7 71.6 We further evaluate Ī“ā-VisionĪ“-Vision on 3 additional MLLM backbones from LLaVA and Qwen families, including LLaVA-1.5-7B, Qwen3-VL-30B-A3B, and Qwen3.5-4B. The latter adopts a hybrid attention architecture, for which we provide additional analyses in Appendix K. As shown in Table 3, Ī“ā-VisionĪ“-Vision consistently preserves a large fraction of the vanilla-model performance across different model scales and architectures. The gains over token-pruning baselines are particularly pronounced on the Qwen family. On Qwen3-VL-30B-A3B, Ī“ā-VisionĪ“-Vision reaches an average score of 78.6, compared with 71.3 for DivPrune under 5% token retention. On Qwen3.5-4B, Ī“ā-VisionĪ“-Vision achieves 71.6, improving over DivPrune at 62.7. Similar improvements are also observed on LLaVA-1.5-7B. Across these three backbones, Ī“ā-VisionĪ“-Vision retains approximately 94ā96% of the corresponding vanilla-model performance, despite substantial differences in model scale and architecture. These results support the generality of visual-memory prediction as an complementary to explicitly propagating visual tokens through the full Transformer computation. Additional cross-backbone results are provided in Appendix F, including comparisons under 20% visual-token retention across 6 backbones and 5% retention results on the three backbones. Results on Multi-Image and Video. Table 4: Results on multi-image and video benchmarks. Training-free baselines are evaluated at 5% visual-token retention. Method MuirBench Video-MME MVBench Avg. Upper Bound Qwen3-VL-4B 53.5 52.0 61.6 55.7 Training-Free (Retain 5% Visual Tokens) FastV 42.1 45.3 41.2 42.9 DART 45.1 51.3 53.3 49.9 VisionZip 43.6 48.2 46.5 46.1 SparseVLM 42.2 47.2 45.4 44.9 DivPrune 46.1 51.3 53.3 50.2 ZOO-Prune 43.9 47.9 48.5 46.8 Training-Based (Retain 5% Visual Tokens) LLaVA-Mini 43.8 47.5 45.2 45.5 EPIC 42.7 48.4 48.8 46.6 Ī“ā-VisionĪ“-Vision (Sin-Img.) 40.7 50.5 57.4 49.5 Ī“ā-VisionĪ“-Vision (Mul-Img.) 45.5 51.0 59.5 52.0 As shown in Table 4, the embedding adapter trained only on single-image data achieves an average score of 49.5, while incorporating multi-image and video training data (the training recipe is detailed in Appendix A) improves the average to 52.0. The largest gains appear on MuirBench and MVBench, increasing from 40.7 to 45.5 and from 57.4 to 59.5, respectively, while Video-MME improves from 50.5 to 51.0. The resulting 52.0 average surpasses the strongest 5%-retention pruning baseline at 50.2. These results indicate the visual-memory formulation remains effective for longer and more complex visual contexts. Additional results with pruning baselines retaining 20% of the visual tokens are provided in Appendix C. 4.3 How Much Visual Computation Is Actually Necessary? We further ask how much of the visual computation is actually needed to preserve the model behavior. We study this question at three levels: (1) characterizing the native spectral structure of visual computation, (2) testing whether low-rank visual effects are sufficient to maintain model performance, (3) propagating low-rank interventions sequentially through the network. These analyses progressively examine whether the visual computation used by the model can be represented within a substantially smaller effective subspace. Spectral Structure of Native Visual Computation. Table 5: Average rank statistics of visual-token computations across Transformer layers. r95r_95 denotes the minimum rank retaining 95% of the squared singular-value energy, and ER denotes effective rank. Dataset Q āā¤ QK Attention Output Full r_95 ER Full r_95 ER Full r_95 ER Qwen3-VL-4B MMStar 213.9213.9 42.7 108.4108.4 119.2119.2 5.0 19.319.3 212.2212.2 47.0 101.2101.2 RWQA 1296.01296.0 112.7 530.0530.0 128.0128.0 10.7 35.335.3 1296.01296.0 103.5 378.5378.5 SQA 170.5170.5 35.4 89.189.1 101.3101.3 4.1 16.016.0 170.5170.5 38.0 83.483.4 LLaVA-1.5-7B MMStar 576.0576.0 79.8 243.9243.9 128.0128.0 2.3 15.615.6 576.0576.0 37.3 151.8151.8 RWQA 576.0576.0 81.3 251.1251.1 128.0128.0 2.3 15.615.6 576.0576.0 37.6 156.5156.5 SQA 576.0576.0 71.8 230.4230.4 128.0128.0 2.2 14.714.7 576.0576.0 29.6 134.5134.5 We first examine the intrinsic dimensionality of visual-token computation in the unmodified model. For each sample and Transformer layer, we analyze three quantities associated with visual tokens: the visual queries after query normalization and RoPE, the per-head visual QāKā¤QK matrices before masking and softmax, and the visual attention outputs after the output projection but before the residual connection. We characterize the spectral concentration using two complementary statistics. r95r_95 denotes the minimum rank required to retain 95% of the squared singular-value energy. We additionally report the effective rank, defined as ER=exp(āāipilogpi)ER= (- _ip_i p_i), where pi=Ļi/ājĻjp_i= _i/ _j _j is the normalized singular-value spectrum. As shown in Table 5, across both Qwen3-VL-4B and LLaVA-1.5-7B, these quantities exhibit strongly concentrated spectra. In particular, the visual attention outputs require only 38.0ā103.5 directions on Qwen3-VL-4B and 29.6ā37.6 directions on LLaVA-1.5-7B to retain 95% of the spectral energy, substantially below their available rank. These results suggest that the full representational capacity allocated to visual processing is not uniformly utilized across directions. Low-Rank Sufficiency of Visual Information. Figure 3: Accuracy recovery under low-rank visual attention effect. We vary the retained rank r from 0 to 1024. To test whether the observed spectral concentration is functionally meaningful, we run the unmodified model to obtain the native hidden-state trajectory and isolate the attention effect induced by visual information flow at each layer. This visual effect is then projected onto a rank-r subspace before being passed to subsequent computation, while the effect itself is always computed from the native trajectory. As shown in Figure 3, performance recovers rapidly as the retained rank increases. On Qwen3-VL-4B, rank 128 reaches 92.2 on SQA, 69.9 on RWQA, and 61.0 on MMStar, compared with 93.4, 71.4, and 64.8 when the full visual effect is retained. A similar trend is observed on LLaVA-1.5-7B, where relatively small ranks already approach the full-effect performance. These results indicate that, given the native model trajectory, most task-relevant visual effects can be represented within a substantially smaller hidden subspace. Sequential Low-Rank Intervention on Visual Information Flow. Table 6: Low-rank intervention on visual-to-text information transfer. visual-to-text attention is blocked at selected layers, and only the leading r directions of the induced attention-output difference are restored. For Qwen3-VL-4B, the middle and last 10-layer groups correspond to layers 13ā22 and 26ā35; for LLaVA-1.5-7B, they correspond to layers 11ā20 and 22ā31, respectively. Dataset Intervention Qwen3-VL-4B LLaVA-1.5-7B Vanilla = r=0 = r=32 = r=64 = r=128 Vanilla = r=0 = r=32 = r=64 = r=128 RWQA First 5 71.2 72.6 71.9 71.8 71.5 54.9 53.2 55.8 56.2 56.0 First 10 72.9 71.9 72.0 72.0 49.4 54.9 55.2 55.8 Middle 10 45.9 69.9 69.5 70.5 48.1 54.3 55.6 55.2 Last 10 71.1 71.0 71.1 71.5 56.0 55.0 55.2 55.2 All 46.1 68.4 69.2 70.7 42.4 53.2 54.9 54.8 MMStar First 5 64.9 64.8 65.0 65.4 65.2 37.0 37.3 35.8 36.4 36.7 First 10 62.8 64.7 64.9 65.1 31.0 34.8 34.9 35.6 Middle 10 40.5 61.0 63.3 64.4 36.3 36.9 37.4 37.0 Last 10 65.2 65.1 64.8 64.9 36.4 36.8 36.8 37.0 All 25.6 59.5 63.0 64.6 24.9 34.0 34.9 35.8 A stronger question is whether this low-rank sufficiency still holds when the compressed visual effect is propagated through subsequent layers. We further apply the low-rank intervention sequentially across Transformer layers, such that the compressed output at one layer directly affects the hidden states processed by subsequent layers. As shown in Table 6, low-dimensional visual information remains sufficient even under this setting. When all layers of Qwen3-VL-4B are intervened, completely removing direct visual information reduces RWQA from 71.2 to 46.1 and MMStar from 64.9 to 25.6, approaching chance-level performance, whereas restoring only 128 hidden directions recovers the scores to 70.7 and 64.6, respectively. The layer-group interventions further reveal that the dependence on visual information is highly non-uniform across depth. On Qwen3-VL-4B, blocking visual information in the middle layers causes the largest degradation, while the first few and final layers are largely insensitive. In contrast, LLaVA-1.5-7B exhibits a more distributed and task-dependent pattern: on RWQA, both the early and middle layers contribute noticeably, whereas on MMStar the first ten layers are more sensitive than the middle or final layers. These results suggest that the effective dimensionality of visual information remains small even when compression is propagated through the network, while the importance of visual-token processing is highly non-uniform across depth and concentrated in a model-specific subset of Transformer layers. Thus, Ī“ā-VisionĪ“-Vision enables flexible layer-wise skipping rather than applying apply full visual-token computation uniformly at every layer in standard MLLMs (we further validate this in Appendix J)). 5 Related Work Visual Representation Redundancy. Visual redundancy exists not only across tokens but also in the hidden representation space. LRCP (Lu et al., 2026) uses PCA to identify dominant visual subspaces and retains tokens that are poorly represented by them. Other works reduce layer-wise visual computation directly. E-MLLM (Ma et al., 2024) removes self-attention among visual tokens, while ShortV (Yuan et al., 2025) and Luo et al. (2026) estimate layer importance from changes in the output distribution. Unlike these methods, which remove or preserve parts of the original computation, Ī“ā-VisionĪ“-Vision learns an alternative pathway to construct visual states. KV Prediction (Horton et al., 2024) uses a small auxiliary model to predict the KV cache of a larger model. Ī“ā-VisionĪ“-Vision instead predicts visual memory before KV projection with lightweight adapters and reuses the frozen key and value projections. Token Reduction for Efficient MLLMs. Visual token reduction lowers MLLM inference cost by selecting, merging, or compressing visual tokens. Training-free methods (Yang et al., 2025; Alvar et al., 2025; Wen et al., 2025a; Zhang et al., 2025b; Chen et al., 2024a; Kim et al., 2026) mainly gain efficiency by reducing sequence length, while training-based methods (Zhang et al., 2025a; Wen et al., 2025b) improve adaptation to compact visual representations. Some methods also allow removed tokens to re-enter later layers. SwiftVLM (Qian et al., 2026) uses cross-layer bypasses and residual connections to recover unselected tokens. In contrast, Ī“ā-VisionĪ“-Vision does not select or remove visual tokens. It constructs key-value context for all visual tokens at every layer. Thus, our method changes how visual states are generated rather than which visual tokens are retained, providing a different efficiency direction from sequence-length compression. 6 Conclusion In this work, we revisit the common assumption that visual tokens undergo full Transformer evolution in MLLMs. Our analysis shows that layer-wise visual representations exhibit substantial redundancy in the hidden-channel dimension and predictability across depth, suggesting that useful visual context can be preserved without repeatedly applying the full visual Transformer pathway. Motivated by these observations, we introduce Ī“ā-VisionĪ“-Vision, which replaces visual-token evolution with lightweight low-rank adapters while preserving all visual tokens for text retrieval. Across multiple MLLM backbones and image, multi-image, and video benchmarks, Ī“ā-VisionĪ“-Vision achieves a favorable accuracyāefficiency trade-off relative to visual token pruning, while operating in a comparable computational regime. These results suggest that reducing multimodal inference cost does not necessarily require removing visual tokens. Efficiently constructing visual states provides a complementary efficiency axis to sequence-length reduction, opening a another direction for scalable MLLMs. References Agarwal et al. (2024) R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. R. Garea, M. Geist, and O. Bachem On-policy distillation of language models: learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, External Links: Link Cited by: Appendix E, Ā§3.3. Alayrac et al. (2022) J. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. L. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals, A. Zisserman, and K. Simonyan Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), External Links: Link Cited by: Ā§1. [3] Allen Institute for AI Molmo2-multiimageqa. Note: Hugging Face DatasetAccessed: 2026-09-25 External Links: Link Cited by: Appendix A. [4] Allen Institute for AI PixMo-askmodelanything. Note: Hugging Face DatasetAccessed: 2026-09-25 External Links: Link Cited by: Appendix A. Alvar et al. (2025) S. R. Alvar, G. Singh, M. Akbari, and Y. Zhang DivPrune: diversity-based visual token pruning for large multimodal models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025, p. 9392ā9401. External Links: Link, Document Cited by: Ā§4.1, Ā§5. Chen et al. (2024a) L. Chen, H. Zhao, T. Liu, S. Bai, J. Lin, C. Zhou, and B. Chang An image is worth 1/2 tokens after layer 2: plug-and-play inference acceleration for large vision-language models. In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part LXXXI, A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, and G. Varol (Eds.), Lecture Notes in Computer Science, Vol. 15139, p. 19ā35. External Links: Link, Document Cited by: Ā§4.1, Ā§5. Chen et al. (2024b) L. Chen, J. Li, X. Dong, P. Zhang, Y. Zang, Z. Chen, H. Duan, J. Wang, Y. Qiao, D. Lin, and F. Zhao Are we on the right way for evaluating large vision-language models?. In Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomczak, and C. Zhang (Eds.), External Links: Link Cited by: Ā§4.1. Chen et al. (2026) Y. Chen, K. Zhang, Z. Zong, Y. Lu, W. Tan, Y. Ren, and J. Hu One layerās trash is another layerās treasure: adaptive layer-wise visual token selection in lvlms. CoRR abs/2606.14277. External Links: Link, Document, 2606.14277 Cited by: Ā§1. Feng et al. (2025) K. Feng, K. Gong, B. Li, Z. Guo, Y. Wang, T. Peng, J. Wu, X. Zhang, B. Wang, and X. Yue Video-r1: reinforcing video reasoning in mllms. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025, D. Belgrave, C. Zhang, L. N. Montoya, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, N. Chen, I. V. M. RuĆz, and A. Loaiza-Bonilla (Eds.), External Links: Link Cited by: Appendix A. Fu et al. (2025a) C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun, Y. Wu, R. Ji, C. Shan, and R. He MME: A comprehensive evaluation benchmark for multimodal large language models. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025, D. Belgrave, C. Zhang, L. N. Montoya, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, N. Chen, I. V. M. RuĆz, and A. Loaiza-Bonilla (Eds.), External Links: Link Cited by: Ā§4.1. Fu et al. (2025b) C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y. Shen, M. Zhang, P. Chen, Y. Li, S. Lin, S. Zhao, K. Li, T. Xu, X. Zheng, E. Chen, C. Shan, R. He, and X. Sun Video-mme: the first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025, p. 24108ā24118. External Links: Link, Document Cited by: Ā§4.1. GLM (2026) GLM GLM-5v-turbo: toward a native foundation model for multimodal agents. CoRR abs/2604.26752. External Links: Link, Document, 2604.26752 Cited by: Ā§1. Goyal et al. (2019) Y. Goyal, T. Khot, A. Agrawal, D. Summers-Stay, D. Batra, and D. Parikh Making the V in VQA matter: elevating the role of image understanding in visual question answering. Int. J. Comput. Vis. 127 (4), p. 398ā414. External Links: Link, Document Cited by: Ā§4.1. Horton et al. (2024) M. Horton, Q. Cao, C. Sun, Y. Jin, S. Mehta, M. Rastegari, and M. Nabi KV prediction for improved time to first token. CoRR abs/2410.08391. External Links: Link, Document, 2410.08391 Cited by: Ā§5. Hudson and Manning (2019) D. A. Hudson and C. D. Manning GQA: A new dataset for real-world visual reasoning and compositional question answering. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, p. 6700ā6709. External Links: Link, Document Cited by: Ā§4.1. Kim et al. (2026) Y. Kim, Y. Zhang, H. Liu, A. Jung, S. Lee, and S. Hong ZOO-prune: training-free token pruning via zeroth-order gradient estimation in vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2026, Denver, CO, USA, June 3-7, 2026, p. 39572ā39582. External Links: Link Cited by: Ā§4.1, Ā§5. Kimi Team (2025) Kimi Team Kimi-vl technical report. CoRR abs/2504.07491. External Links: Link, Document, 2504.07491 Cited by: Ā§1. Li et al. (2025) F. Li, R. Zhang, H. Zhang, Y. Zhang, B. Li, W. Li, Z. Ma, and C. Li LLaVA-interleave: tackling multi-image, video, and 3d in large multimodal models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, External Links: Link Cited by: Appendix A. Li et al. (2024) K. Li, Y. Wang, Y. He, Y. Li, Y. Wang, Y. Liu, Z. Wang, J. Xu, G. Chen, P. Lou, L. Wang, and Y. Qiao MVBench: A comprehensive multi-modal video understanding benchmark. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, p. 22195ā22206. External Links: Link, Document Cited by: Ā§4.1. Li et al. (2023) Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J. Wen Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, H. Bouamor, J. Pino, and K. Bali (Eds.), p. 292ā305. External Links: Link, Document Cited by: Ā§4.1. Liu et al. (2024a) H. Liu, C. Li, Y. Li, and Y. J. Lee Improved baselines with visual instruction tuning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, p. 26286ā26296. External Links: Link, Document Cited by: Ā§1, Ā§1, Ā§4.1. Liu et al. (2024b) H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee LLaVA-next: improved reasoning, ocr, and world knowledge. External Links: Link Cited by: Ā§4.1. Liu et al. (2024c) Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, K. Chen, and D. Lin MMBench: is your multi-modal model an all-around player?. In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part VI, A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, and G. Varol (Eds.), Lecture Notes in Computer Science, Vol. 15064, p. 216ā233. External Links: Link, Document Cited by: Ā§4.1. Lu et al. (2026) H. Lu, F. Zhang, W. Jin, H. Hu, T. Shi, S. Jiang, Y. Hu, and J. Li LRCP: low-rank compressibility guided visual token pruning for efficient lvlms. CoRR abs/2605.15621. External Links: Link, Document, 2605.15621 Cited by: Ā§5. Lu et al. (2022) P. Lu, S. Mishra, T. Xia, L. Qiu, K. Chang, S. Zhu, O. Tafjord, P. Clark, and A. Kalyan Learn to explain: multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), External Links: Link Cited by: Ā§4.1. Luo et al. (2026) Z. Luo, R. Dong, M. Yang, F. Wei, Y. Lai, B. Luo, and H. Fu Attend, transform, or silence: operator-level visual skipping for efficient multimodal LLM inference. CoRR abs/2606.31903. External Links: Link, Document, 2606.31903 Cited by: Ā§5. Ma et al. (2024) F. Ma, Y. Zhou, H. Li, Z. He, S. Wu, F. Rao, Y. Zhang, and X. Sun E-MLLM: A data-efficient and compute-efficient multimodal large language model. CoRR abs/2408.11795. External Links: Link, Document, 2408.11795 Cited by: Ā§5. OpenAI (2024) OpenAI GPT-4o system card. CoRR abs/2410.21276. External Links: Link, Document, 2410.21276 Cited by: Ā§1. Qian et al. (2026) C. Qian, X. Yu, D. Li, G. Chi, Z. Yang, Q. Ma, and X. Miao SwiftVLM: efficient vision-language model inference via cross-layer token bypass. CoRR abs/2602.03134. External Links: Link, Document, 2602.03134 Cited by: Ā§1, Ā§5. Qwen Team (2025) Qwen Team Qwen3-vl technical report. CoRR abs/2511.21631. External Links: Link, Document, 2511.21631 Cited by: Ā§1, Ā§4.1, Ā§4.1. Qwen Team (2026) Qwen Team Qwen3.5: towards native multimodal agents. External Links: Link Cited by: Ā§4.1. Shang et al. (2025) Y. Shang, M. Cai, B. Xu, Y. J. Lee, and Y. Yan LLaVA-prumerge: adaptive token reduction for efficient large multimodal models. In IEEE/CVF International Conference on Computer Vision, ICCV 2025, Honolulu, HI, USA, October 19-25, 2025, p. 22857ā22867. External Links: Link, Document Cited by: Ā§1. Wang et al. (2025) F. Wang, X. Fu, J. Y. Huang, Z. Li, Q. Liu, X. Liu, M. D. Ma, N. Xu, W. Zhou, K. Zhang, T. L. Yan, W. J. Mo, H. Liu, P. Lu, C. Li, C. Xiao, K. Chang, D. Roth, S. Zhang, H. Poon, and M. Chen MuirBench: A comprehensive benchmark for robust multi-image understanding. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, External Links: Link Cited by: Ā§4.1. Wen et al. (2025a) Z. Wen, Y. Gao, S. Wang, J. Zhang, Q. Zhang, W. Li, C. He, and L. Zhang Stop looking for "important tokens" in multimodal language models: duplication matters more. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4-9, 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), p. 9961ā9980. External Links: Link, Document Cited by: Ā§1, Ā§4.1, Ā§5. Wen et al. (2025b) Z. Wen, S. Wang, Y. Zhou, J. Zhang, Q. Zhang, Y. Gao, Z. Chen, B. Wang, W. Li, C. He, and L. Zhang Efficient multi-modal large language models via progressive consistency distillation. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025, External Links: Link Cited by: Ā§1, Ā§4.1, Ā§5. xAI (2024) xAI RealWorldQA: a benchmark for real-world spatial understanding. Note: https://huggingface.co/datasets/xai-org/RealworldQAAccessed: 2026-09-21 Cited by: Ā§4.1. Yang et al. (2026) C. Yang, S. Lo, and Y. Liu Reroute, donāt remove: recoverable visual token routing for vision-language models. CoRR abs/2606.12412. External Links: Link, Document, 2606.12412 Cited by: Ā§1. Yang et al. (2025) S. Yang, Y. Chen, Z. Tian, C. Wang, J. Li, B. Yu, and J. Jia VisionZip: longer is better but not necessary in vision language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025, p. 19792ā19802. External Links: Link, Document Cited by: Ā§1, Ā§1, Ā§4.1, Ā§5. Yuan et al. (2025) Q. Yuan, Q. Zhang, Y. Liu, J. Chen, Y. Lu, H. Lin, J. Zheng, X. Han, and L. Sun ShortV: efficient multimodal large language models by freezing visual tokens in ineffective layers. In IEEE/CVF International Conference on Computer Vision, ICCV 2025, Honolulu, HI, USA, October 19-25, 2025, p. 329ā339. External Links: Link, Document Cited by: Ā§5. Yue et al. (2024) X. Yue, Y. Ni, T. Zheng, K. Zhang, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, C. Wei, B. Yu, R. Yuan, R. Sun, M. Yin, B. Zheng, Z. Yang, Y. Liu, W. Huang, H. Sun, Y. Su, and W. Chen MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, p. 9556ā9567. External Links: Link, Document Cited by: Ā§1. Zhang et al. (2025a) S. Zhang, Q. Fang, Z. Yang, and Y. Feng LLaVA-mini: efficient image and video large multimodal models with one vision token. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, External Links: Link Cited by: Ā§4.1, Ā§5. Zhang et al. (2025b) Y. Zhang, C. Fan, J. Ma, W. Zheng, T. Huang, K. Cheng, D. A. Gudovskiy, T. Okuno, Y. Nakata, K. Keutzer, and S. Zhang SparseVLM: visual token sparsification for efficient vision-language model inference. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, Proceedings of Machine Learning Research, Vol. 267. External Links: Link Cited by: Ā§4.1, Ā§5. Appendix A Additional Implementation and Training Details We freeze the vision encoder, multimodal projector, and language-model backbone, and optimize only the layer-specific visual-memory adapters. To ensure a fair comparison with visual token pruning methods, we disable the DeepStack mechanism in Qwen3-VL for all methods. Unless otherwise specified, each adapter uses a bottleneck rank of r=128. For the default single-image setting, we train on allenai/pixmo-ask-model-anything (Allen Institute for AI, ) for 2,000 steps using 8 GPUs with a global batch size of 32 (LLaVA-Mini and EPIC are trained using the same settings). For the multi-image and video setting, we construct the training set from 20,000 samples of allenai/Molmo2-MultiImageQA (Allen Institute for AI, ), 44,000 samples of lmms-lab/M4-Instruct-Data (Li et al., 2025), and 64,000 samples of Video-R1/Video-R1-data (Feng et al., 2025), and train for 4,000 steps using the same number of GPUs and global batch size. For all experiments, optimization uses AdamW with a learning rate of 5Ć10ā55Ć 10^-5, weight decay of 0.01, and gradient clipping at 1.0. We adopt a cosine learning-rate schedule with a 3% warmup and a minimum learning-rate ratio of 0.1. All experiments use a fixed random seed of 44. Appendix B Additional Token-Retention Results Table 7: Results of visual token compression methods on Qwen3-VL-4B. Training-free baselines are evaluated under 15% and 10% visual-token retention. Method MMStar RWQA GQA MMB MMB-CN MME POPE SQA VQA-v2 Avg. Upper Bound (100% Retention) Qwen3-VL-4B 64.9 71.2 61.6 87.5 87.7 84.7 89.3 93.3 80.9 80.1 Training-Free Retain 15% Visual Tokens FastV 47.5 50.5 44.1 76.7 74.8 75.1 73.1 80.3 63.7 65.1 VisionZip 49.7 58.4 53.4 80.2 78.4 76.4 79.5 82.3 70.3 69.8 DivPrune 51.0 62.4 53.3 83.3 81.9 81.5 84.9 82.3 74.1 72.7 SparseVLM 48.8 43.4 42.0 67.3 65.7 63.4 73.8 74.3 65.3 60.4 DART 48.4 60.5 51.6 81.8 81.5 76.7 81.6 83.0 70.4 70.6 ZOO-Prune 46.5 53.1 50.3 77.7 78.1 77.6 76.1 78.1 69.3 67.4 Retain 10% Visual Tokens FastV 43.4 47.8 43.9 70.7 68.8 71.9 69.5 77.4 57.7 61.2 VisionZip 46.4 53.9 50.4 77.4 75.5 73.9 75.6 79.5 67.2 66.6 DivPrune 48.5 61.4 51.3 81.1 81.2 78.8 83.4 82.4 71.7 71.1 SparseVLM 44.6 42.6 40.7 60.0 57.4 61.4 68.7 72.6 60.3 56.5 DART 46.0 55.8 49.4 77.2 78.4 73.9 78.4 80.4 66.6 67.3 ZOO-Prune 44.0 47.5 47.8 75.7 75.2 75.2 72.7 77.0 64.3 64.4 Ī“ā-VisionĪ“-Vision 54.4 62.9 56.7 84.5 82.8 80.1 87.6 82.3 78.0 74.4 To provide a more complete comparison across visual-token budgets, we additionally evaluate the training-free pruning baselines at 15% and 10% token retention. As shown in Table 7, performance consistently decreases as fewer visual tokens are retained. DivPrune remains the strongest pruning baseline at both operating points, achieving average scores of 72.7 and 71.1 at 15% and 10% retention, respectively. In comparison, Ī“ā-VisionĪ“-Vision achieves an average score of 74.4 while preserving all visual tokens. These results provide a more complete view of the trade-off obtained by reducing the visual sequence length. Appendix C Additional Result on Multi-Image and Video Table 8: Additional results on multi-image and video benchmarks. Training-free baselines are evaluated at 20% visual-token retention. Method MuirBench Video-MME MVBench Avg. Upper Bound Qwen3-VL-4B 53.5 52.0 61.6 55.7 Training-Free (Retain 20% Visual Tokens) FastV 48.6 49.5 54.6 50.9 DART 49.0 52.2 56.7 52.6 VisionZip 47.9 50.9 55.0 51.3 SparseVLM 47.0 49.6 53.9 50.2 DivPrune 48.8 52.0 57.5 52.8 ZOO-Prune 50.2 49.5 53.6 51.1 Emb. Adapter (Single-Img.) 40.7 50.5 57.4 49.5 Emb. Adapter (Multi-Img./Video) 45.5 51.0 59.5 52.0 Table 8 provides additional results on multi-image and video benchmarks, where the training-free pruning baselines retain 20% of the visual tokens. Appendix D Additional Efficiency Results Table 9: Efficiency comparison on Video-MME. Training-free baselines are evaluated under 20% visual-token retention. Total Speedup measures the overall inference speedup, including both prefill and decoding. Method FLOPs Peak Mem. Total Prefill Decode (%) (GB) Speedup Speedup Speedup Upper Bound Qwen3-VL-4B 100.00 12.94 1.00 1.00 1.00 Training-Free (Retain 20% Visual Tokens) FastV 33.27 11.68 1.18 1.34 1.07 DART 33.00 11.64 1.13 1.23 1.06 VisionZip 29.08 11.05 1.25 1.34 1.18 DivPrune 29.05 11.06 1.25 1.34 1.18 ZOO-Prune 36.36 12.19 1.16 1.15 1.18 SparseVLM 33.37 11.98 1.16 1.28 1.07 Emb. Adapter 17.90 12.97 1.30 1.50 1.13 Table 9 reports the inference efficiency of different visual token pruning methods under 20% token retention on Video-MME. The pruning baselines require 29.05ā36.36% of the FLOPs of the uncompressed model and achieve prefill speedups ranging from 1.15Ć1.15Ć to 1.34Ć1.34Ć. Their total speedups range from 1.13Ć1.13Ć to 1.25Ć1.25Ć, while decode speedups remain between 1.06Ć1.06Ć and 1.18Ć1.18Ć. Ī“ā-VisionĪ“-Vision uses 17.90% of the FLOPs and achieves 1.30Ć1.30Ć total and 1.50Ć1.50Ć prefill speedups while preserving all visual tokens. These results provide an additional efficiency comparison under a mild pruning regime. Appendix E Training Objective Ablation Table 10: Ablation study on different training objectives for the embedding adapter. All variants only optimize the adapter parameters. Emb. Adapter (Init). denotes the initialization setting, where the same visual representation is shared across all Transformer layers without learned layer-specific adaptation, serving as the lower-bound baseline. Method MMStar RWQA GQA MMB MMB-CN MME POPE SQA VQA-v2 Avg. Time Emb. Adapter (Init). 41.2 40.8 43.4 77.4 74.6 66.4 65.5 78.8 58.4 60.7 - Emb. Adapter + SFT 50.5 59.4 28.6 83.6 80.0 77.5 85.5 79.9 75.1 68.9 23m51s Emb. Adapter + OPD 53.6 63.9 55.8 83.4 83.1 79.5 86.7 81.9 77.8 74.0 4h38m08s Emb. Adapter + Supervised-KD 54.4 62.9 56.7 84.5 82.8 80.1 87.6 82.3 78.0 74.4 34m53s We compare three objectives for training the embedding adapter while keeping all backbone parameters frozen: supervised fine-tuning (SFT), on-policy distillation (OPD) (Agarwal et al., 2024), and our Supervised-KD objective. As shown in Table 10, SFT achieves an average score of 68.9, substantially below both distillation-based objectives. OPD improves the average score to 74.0 but requires 4h38m of training due to autoregressive student rollouts. Supervised-KD achieves the highest average score of 74.4 while requiring only 34m53s, reducing the training time by approximately 8Ć8Ć relative to OPD. These results show that teacher-forced distribution matching retains the effectiveness of distillation without the rollout overhead of on-policy training. Appendix F Additional Cross-Backbone Results Table 11: Results of different visual token compression methods across various vision-language models. Vanilla denotes the uncompressed upper bound. DART and DivPrune retain 20% of the visual tokens. Model Method MMStar RWQA GQA MMB MMB-CN MME POPE SQA VQA-v2 Avg. LLaVA-1.5-7B Vanilla 37.0 54.9 61.2 72.5 67.9 71.7 83.4 65.6 75.9 65.6 DART 30.6 52.3 57.8 70.8 66.3 76.6 81.4 69.4 71.1 64.0 DivPrune 38.1 51.9 58.5 70.4 66.5 70.3 82.1 65.6 74.3 64.2 Ī“ā-VisionĪ“-Vision 35.4 51.4 53.8 69.9 63.7 68.7 79.2 65.4 69.5 61.9 LLaVA-1.5-13B Vanilla 38.7 57.7 63.0 74.7 70.3 66.6 83.8 71.5 78.3 67.2 DART 33.6 55.4 57.6 76.0 67.9 75.6 84.9 73.6 72.1 66.3 DivPrune 36.8 52.3 60.8 73.8 69.6 69.1 81.2 72.2 75.8 65.7 Ī“ā-VisionĪ“-Vision 36.5 50.1 56.3 72.5 65.9 64.2 76.8 70.7 73.1 62.9 LLaVA-1.6-Mistral-7B Vanilla 40.4 47.3 54.3 71.9 65.0 68.4 85.4 68.7 74.4 64.0 DART 40.0 48.1 53.2 72.0 64.2 69.8 84.4 69.8 73.6 63.9 DivPrune 39.4 47.1 56.8 72.1 63.8 68.2 83.4 70.5 75.6 64.1 Ī“ā-VisionĪ“-Vision 39.2 43.9 46.8 70.5 61.2 64.8 83.7 68.3 69.5 60.9 Qwen3-VL-8B Vanilla 68.5 69.4 61.9 89.2 88.7 88.6 86.8 95.1 80.6 81.0 DART 51.3 63.5 52.8 81.9 81.6 80.4 82.6 88.0 71.8 72.7 DivPrune 54.4 64.1 56.2 83.4 84.4 84.2 85.9 86.0 76.8 75.0 Ī“ā-VisionĪ“-Vision 54.9 61.8 58.0 83.5 82.6 82.0 84.6 81.5 75.4 73.8 Qwen3-VL-30B-A3B Vanilla 70.3 72.3 65.2 89.4 90.8 89.4 87.9 96.3 82.6 82.7 DART 58.3 64.8 59.6 85.9 85.4 86.1 83.3 89.5 75.9 76.5 DivPrune 59.6 68.0 62.8 86.7 87.7 87.5 88.0 90.3 78.7 78.8 Ī“ā-VisionĪ“-Vision 62.4 65.1 62.4 86.5 86.8 87.0 87.8 88.6 80.7 78.6 Qwen3.5-4B Vanilla 48.9 67.2 63.4 84.3 77.2 84.0 88.7 78.4 80.0 74.7 DART 37.3 56.5 54.0 79.5 73.5 79.7 79.2 73.4 71.6 67.2 DivPrune 42.3 63.3 57.7 81.1 73.3 81.9 87.3 74.1 75.9 70.8 Ī“ā-VisionĪ“-Vision 40.1 65.8 60.5 81.3 75.1 79.7 89.0 74.8 77.7 71.6 Table 11 reports cross-backbone results with DART and DivPrune retaining 20% of the visual tokens. Across the 6 evaluated backbones, Ī“ā-VisionĪ“-Vision remains close to the strongest pruning baseline on the Qwen models, achieving 73.8 on Qwen3-VL-8B, 78.6 on Qwen3-VL-30B-A3B, and 71.6 on Qwen3.5-4B, compared with 75.0, 78.8, and 70.8 for DivPrune, respectively. On the LLaVA family, Ī“ā-VisionĪ“-Vision obtains average scores of 61.9, 62.9, and 60.9 on LLaVA-1.5-7B, LLaVA-1.5-13B, and LLaVA-v1.6-Mistral-7B. Table 12 shows the results under 5% Token Retention. Ī“ā-VisionĪ“-Vision achieves 62.9 on LLaVA-1.5-13B and 73.8 on Qwen3-VL-8B, improving over DivPrune by 6.9 and 8.4 points, respectively. On LLaVA-v1.6-Mistral-7B, Ī“ā-VisionĪ“-Vision reaches 60.9, which is comparable to DivPrune at 61.5 and substantially higher than DART at 50.4. Table 12: Results of different visual token compression methods across various vision-language models. Vanilla denotes the uncompressed upper bound. DART and DivPrune retain 5% of the visual tokens. Model Method MMStar RWQA GQA MMB MMB-CN MME POPE SQA VQA-v2 Avg. LLaVA-1.5-13B Vanilla 38.7 57.7 63.0 74.7 70.3 66.6 83.8 71.5 78.3 67.2 DART 32.7 48.1 50.0 67.7 58.2 58.4 70.7 70.9 63.0 57.7 DivPrune 35.9 51.2 44.1 69.5 60.9 52.8 62.9 68.3 58.5 56.0 Ī“ā-VisionĪ“-Vision 36.5 50.1 56.3 72.5 65.9 64.2 76.8 70.7 73.1 62.9 LLaVA-1.6-Mistral-7B Vanilla 40.4 47.3 54.3 71.9 65.0 68.4 85.4 68.7 74.4 64.0 DART 29.6 42.5 39.1 51.0 46.1 58.6 67.0 67.5 52.3 50.4 DivPrune 36.8 42.9 54.7 69.4 61.3 63.7 84.6 69.6 70.5 61.5 Ī“ā-VisionĪ“-Vision 39.2 43.9 46.8 70.5 61.2 64.8 83.7 68.3 69.5 60.9 Qwen3-VL-8B Vanilla 68.5 69.4 61.9 89.2 88.7 88.6 86.8 95.1 80.6 81.0 DART 40.6 53.3 43.5 70.7 71.7 74.5 74.4 79.3 61.5 63.3 DivPrune 41.0 55.3 44.9 73.4 74.6 76.8 76.8 78.4 67.2 65.4 Ī“ā-VisionĪ“-Vision 54.9 61.8 58.0 83.5 82.6 82.0 84.6 81.5 75.4 73.8 Appendix G Adapter Rank Ablation We further study the effect of the bottleneck rank r of the visual-memory adapter. As shown in Figure 4, increasing the adapter rank generally leads to moderate improvements in downstream accuracy, suggesting that a larger bottleneck provides additional capacity for modeling layer-specific visual-state corrections. (a) Results on MMStar, GQA, and RWQA. (b) Results on VQA-v2, MME, SQA, MMB-CN, MMB, and POPE. Figure 4: Performance of Embedding Adapter under different adapter ranks r. Dashed horizontal lines denote the corresponding vanilla model performance. Appendix H Hidden-State Compression To construct the low-rank subspace, we collect visual hidden states from 1,024 images in allenai/pixmo-ask-model-anything and fit a separate uncentered PCA basis for each Transformer layer. The basis is shared across all visual tokens and images at the same layer. Within a given layer, the same basis is shared across all visual-token positions and all input samples. At inference time, each visual hidden state hāāDh ^D at layer l is projected onto the leading r principal directions and then reconstructed back to the original hidden dimension: h~=Pl,rāPl,rā¤āh, h=P_l,rP_l,r h, (11) where Pl,rāāDĆrP_l,r ^DĆ r contains the leading r principal directions for layer l. The reconstructed state h~ h then replaces the original visual hidden state for subsequent computation. This intervention preserves the number and positions of visual tokens and changes only the dimensional subspace in which their hidden representations are allowed to vary. For different values of r, we use nested prefixes of the same PCA basis, and the fitted bases are shared across all evaluation benchmarks. Appendix I Adapter Compatibility Table 13: Combining visual-memory adaptation with visual-token pruning on Qwen3-VL-4B. Starting from the embedding adapter, we additionally apply DART or DivPrune at different visual-token retention ratios. Method Retention MMStar RWQA GQA MMB MMB-CN MME POPE SQA VQA-v2 Avg. Vanilla 64.9 71.2 61.6 87.5 87.7 84.7 89.3 93.3 80.9 80.1 Ī“ā-VisionĪ“-Vision (Emb. Adapter) 54.4 62.9 56.7 84.5 82.8 80.1 87.6 82.3 78.0 74.4 ++ DART 50% 49.7 62.1 58.4 82.6 80.9 81.8 88.0 81.0 74.3 73.2 ++ DivPrune 48.8 61.4 58.4 83.0 80.7 82.7 87.1 81.3 74.5 73.1 ++ DART 20% 46.4 58.4 54.1 79.5 78.9 79.1 83.9 79.6 68.5 69.8 ++ DivPrune 45.1 57.4 55.2 79.5 78.5 82.4 84.8 79.4 70.2 70.3 ++ DART 5% 39.5 50.2 45.1 69.0 70.9 72.8 70.9 76.4 58.0 61.4 ++ DivPrune 36.9 53.2 45.2 67.9 67.6 75.2 69.4 75.8 58.1 61.0 Our method reduces visual computation along the representation dimension, whereas token-pruning methods reduce the visual sequence length. We therefore further investigate whether these two forms of compression can be applied jointly. Specifically, starting from the embedding-adapter variant of Ī“ā-VisionĪ“-Vision, we additionally apply DART or DivPrune at different visual-token retention ratios. The results are reported in Table 13. The embedding adapter alone achieves an average score of 74.4. When combined with moderate token pruning at 50% retention, performance remains largely preserved: DART and DivPrune achieve average scores of 73.2 and 73.1, respectively, corresponding to only a 1.2ā1.3 point decrease from the standalone adapter. More aggressive pruning leads to the expected degradation, with average scores of 69.8/70.3 at 20% retention and 61.4/61.0 at 5% retention for DART/DivPrune. These results provide empirical evidence that reducing layer-wise visual-state computation and reducing visual sequence length target distinct sources of redundancy and can be used in a complementary manner. Appendix J Layer-wise Adapter Skipping Table 14: Effect of layer-wise adapter skipping on Qwen3-VL-4B. We selectively skip the embedding adapters in early and late Transformer layers while keeping the remaining configuration unchanged. Configuration MMStar RWQA GQA MMB MMB-CN MME POPE SQA VQA-v2 Avg. Emb. Adapter 54.4 62.9 56.7 84.5 82.8 80.1 87.6 82.3 78.0 74.4 w/o First 5 & Last 10 49.6 62.5 56.6 83.4 81.1 82.1 89.0 81.1 74.3 73.3 w/o First 10 & Last 10 46.7 61.2 55.4 82.8 78.9 80.8 88.3 80.5 72.4 71.9 Motivated by the non-uniform layer-wise dependence on visual information observed in Table 6, we further investigate whether the visual-memory adapters need to be applied at every Transformer layer. We selectively skip the embedding adapters in the early and late layers while keeping the remaining configuration unchanged. As shown in Table 14, removing the adapters from the first 5 and last 10 layers reduces the average score only from 74.4 to 73.3, while skipping the first 10 and last 10 layers yields an average score of 71.9. Meanwhile, Table 15 shows that layer-wise skipping further reduces the computational overhead of Ī“ā-VisionĪ“-Vision: the FLOPs decrease from 17.90% to 15.64% and 14.89% of the vanilla model, while the total inference speedup improves from 1.30Ć1.30Ć to 1.35Ć1.35Ć and 1.37Ć1.37Ć, respectively. These results are consistent with the depth-wise intervention analysis and suggest that visual-memory adaptation need not be allocated uniformly across layers. In particular, selectively skipping adapters in less sensitive early and late layers provides an additional accuracyāefficiency trade-off for Ī“ā-VisionĪ“-Vision. Table 15: Efficiency of layer-wise adapter skipping on Qwen3-VL-4B. Total Speedup measures the overall inference speedup, including both prefill and decoding. Configuration FLOPs Peak Mem. Total Prefill Decode (%) (GB) Speedup Speedup Speedup Vanilla 100.00 12.94 1.00 1.00 1.00 Emb. Adapter 17.90 12.97 1.30 1.50 1.13 w/o First 5 & Last 10 15.64 11.86 1.35 1.60 1.16 w/o First 10 & Last 10 14.89 11.49 1.37 1.64 1.17 Appendix K Linear Attention Analysis For hybrid backbones containing Gated DeltaNet layers, such as Qwen3.5-4B, Ī“ā-VisionĪ“-Vision independently constructs the visual memory at each layer. Under the embedding-adapter variant, the predicted representation of the i-th visual token at layer l is Ml,i=Ei+UlāSiLUā”(DlāEi).M_l,i=E_i+U_lSiLU (D_lE_i ). (12) Thus, the visual representation at each layer is constructed directly from the initial visual embedding EiE_i, rather than propagated from the visual state of the preceding layer. At layer l, we replace the visual positions with the predicted visual memory while keeping the textual hidden states unchanged: Xl,t=Ml,i,tā corresponds to the āiā-th visual token,Hl,t,tā corresponds to a textual token.X_l,t= casesM_l,i,&t corresponds to the i-th visual token,\\ H_l,t,&t corresponds to a textual token. cases (13) The combined sequence XlX_l is then processed by the original frozen RMSNorm, linear projections, causal convolution, and gating modules of Gated DeltaNet to obtain qtq_t, ktk_t, vtv_t, Ī±t _t, and Ī²t _t. The predicted visual tokens still participate in the native recurrent-state update. Omitting the layer and head indices for clarity, the recurrence is SĀÆt=Ī±tāStā1, S_t= _tS_t-1, (14) followed by St=SĀÆt+Ī²tāktā(vtāSĀÆtā¤ākt)ā¤.S_t= S_t+ _tk_t (v_t- S_t k_t ) . (15) The corresponding readout is ot=qtā¤āStdk.o_t= q_t S_t d_k. (16) Therefore, the visual representations predicted by the adapter are still written into the recurrent state through the original Gated DeltaNet update, and subsequent textual tokens retrieve the resulting visual information through their own queries. Figure 5: Visual write and read interventions in the hybrid-attention backbone. Mask LA removes the visual write to the recurrent state in all Linear Attention (LA) layers by restoring the post-visual recurrent state to its pre-visual value. For Full Attention (FA) layers, we block visual read by preventing textual queries from attending to visual keys and values. We compare blocking FA layers 11 and 15 against blocking the other six FA layers (3, 7, 19, 23, 27, and 31). To better understand how visual information is utilized in hybrid-attention MLLMs, we separately intervene on the two mechanisms through which visual information affects subsequent textual computation: writing visual information into the recurrent state of Linear Attention (LA) layers, and reading visual keyāvalue states in Full Attention (FA) layers. Blocking visual write in Linear Attention. For each LA layer, let SinS_in denote the recurrent state immediately before the first visual token, and let SoutS_out denote the state after all visual tokens have been processed normally. Under our rank-0 intervention, we restore the recurrent state at the visual boundary as SoutāSin.S_outā S_in. (17) Thus, the visual segment is still fully processed, but its net contribution to the recurrent memory is removed before subsequent textual tokens are processed. All FA layers remain unmodified and can still access visual keys and values. We apply this intervention to all 24 LA layers. Blocking visual read in Full Attention. For FA layers, we instead prevent textual queries from reading visual keys and values and restricted to textual keys and values only. All LA layers operate normally. ot=Attnā”(qt,Ktext,Vtext).o_t=Attn (q_t,K_text,V_text ). (18) We evaluate two FA intervention settings. The first blocks visual read in layers 11 and 15, while the second blocks visual read in layers 3, 7, 19, 23, 27, and 31. All layer indices are zero-based. As shown in Figure 5, it reveals a strongly non-uniform dependence on visual information across the hybrid-attention layers. Canceling visual writes to the recurrent states of all 24 LA layers has only a limited effect, In contrast, blocking visual reads in only FA layers 11 and 15 causes a substantial degradation on RWQA and MMStar. Blocking the other six FA layers (3, 7, 19, 23, 27, and 31) results in a considerably smaller drop. These results indicate that useful visual information is highly concentrated in a small subset of FA layers rather than being uniformly utilized throughout the network. This non-uniformity provides additional flexibility for Ī“ā-VisionĪ“-Vision: visual-memory computation can be selectively retained in sensitive layers while being skipped in less important layers to further improve efficiency.