Paper deep dive
ResComEmb: Effective and Efficient Multimodal Embedding via Residual Homogeneity Compression
Zijing Cai, Yuzhe Wang, Jingxian Zhu, Fengbin Zhu, Richang Hong
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal large language models (MLLMs) have shown strong potential for universal multimodal representation learning. However, existing methods either compress each input into a single vector, limiting fine-grained expressiveness, or retain long sequences of visual-token vectors, incurring substantial storage and interaction costs. To resolve this trade-off, we propose ResComEmb, a trainable framework for effective and efficient universal multi-vector multimodal embedding. ResComEmb first encodes each input at native dynamic resolution into ordered global, intermediate, and fine-grained views. After MLLM contextualization and embedding projection, a trainable Residual Homogeneity Compression (RHC) module reduces within-granularity redundancy and cross-granularity repetition under explicit visual token budgets. Then, ResComEmb introduces a length-adaptive Bidirectional Late-Interaction Matching mechanism for robust query-document scoring, which averages the strongest token-level matches in each direction and combines the two scores using a weight based on how many valid tokens each side has. Extensive experiments on MMEB, ViDoRe V1, and ViDoRe V2 show that ResComEmb produces higher-quality universal multimodal embeddings than VLM2Vec-V2, and outperforms ColQwen2.5 in visual document retrieval using only 37.5% of its full visual token budget, demonstrating a favorable effectiveness-efficiency trade-off.
Tags
Links
- Source: https://arxiv.org/abs/2609.37225v1
- Canonical: https://arxiv.org/abs/2609.37225v1
Trouble viewing inline? Open PDF directly →
Full Text
69,174 characters extracted from source content.
Expand or collapse full text
ResComEmb: Effective and Efficient Multimodal Embedding via Residual Homogeneity Compression Zijing Cai Affiliation: University of Science and Technology of China Yuzhe Wang Affiliation: University of Science and Technology of China Jingxian Zhu Affiliation: Hefei University of Technology Fengbin Zhu Affiliation: National University of Singapore Richang Hong Affiliation: Hefei University of Technology Abstract Multimodal large language models (MLLMs) have shown strong potential for universal multimodal representation learning. However, existing methods either compress each input into a single vector, limiting fine-grained expressiveness, or retain long sequences of visual-token vectors, incurring substantial storage and interaction costs. To resolve this trade-off, we propose ResComEmb, a trainable framework for effective and efficient universal multi-vector multimodal embedding. ResComEmb first encodes each input at native dynamic resolution into ordered global, intermediate, and fine-grained views. After MLLM contextualization and embedding projection, a trainable Residual Homogeneity Compression (RHC) module reduces within-granularity redundancy and cross-granularity repetition under explicit visual token budgets. Then, ResComEmb introduces a length-adaptive Bidirectional Late-Interaction Matching mechanism for robust query–document scoring, which averages the strongest token-level matches in each direction and combines the two scores using a weight based on how many valid tokens each side has. Extensive experiments on MMEB, ViDoRe V1, and ViDoRe V2 show that ResComEmb produces higher-quality universal multimodal embeddings than VLM2Vec-V2, and outperforms ColQwen2.5 in visual document retrieval using only 37.5% of its full visual token budget, demonstrating a favorable effectiveness–efficiency trade-off. 11footnotetext: Equal contribution.22footnotetext: Corresponding author. 1 Introduction Multimodal embedding models project heterogeneous inputs into a shared semantic space, enabling downstream applications, including image classification (Deng et al., 2009), visual question answering (Hu et al., 2018), cross-modal retrieval (Gordo et al., 2016), and visual grounding (Yu et al., 2016). Contrastive vision–language encoders (Radford et al., 2021; Zhai et al., 2023), learn cross-modal alignment from large-scale paired data with separate modality towers. Although compact and efficient, this modality-separated design struggles with interleaved image–text inputs and complex instructions (Jiang et al., 2024; Zhang et al., 2024a). Recent work therefore adapts multimodal large language models (MLLMs), such as Qwen2.5-VL (Bai et al., 2025) and PaliGemma (Beyer et al., 2024), into universal multimodal embedding models, leveraging their pre-trained knowledge, multimodal understanding, and instruction-following capabilities (Jiang et al., 2025). MLLM-based embedding models represent visual inputs with a single vector (Jiang et al., 2025) or multiple vectors (Faysse et al., 2025; Zhu et al., 2026), creating a fidelity–efficiency trade-off. A single vector is compact but can limit the expression of fine-grained information in visually rich inputs (Yao et al., 2022; Thrush et al., 2022). Multiple vectors offer higher representational capacity by preserving local evidence, but increase storage and pairwise interaction costs (Faysse et al., 2025). Existing approaches therefore pursue compact multi-vector representations through two routes. Token pruning and merging retain features aligned with the original visual sequence, but aggressive reduction sacrifices holistic context (Yang et al., 2025; Wang et al., 2025; Wen et al., 2025). Learnable-token methods instead condense visual features through a fixed number of query slots, efficiently capturing global semantics; under tight budgets, however, their fixed capacity restricts locality and input-specific details (Cha et al., 2024; Liu et al., 2024). Neither route reliably preserves both global semantics and fine-grained evidence in a compact representation. Recently, Zhu et al. (2026) introduced MURE that combines multi-resolution sampling with token clustering to produce visual embeddings for effective and efficient document retrieval. However, because MURE’s clustering is non-trainable and post-hoc, its merge criterion receives no loss feedback, risking the loss of relevant representations and the persistence of irrelevant ones after compression. Training-time merging and compression-aware fine-tuning have been shown to mitigate such performance degradation (Bolya et al., 2023; Ma et al., 2025), suggesting that compression and representations should be jointly optimized rather than treated as separate stages. Motivated by this, we aim to generate a compact, universal visual representation via trainable coarse-to-fine compression that preserves complementary global and local evidence within contextualized representations. Specifically, within each granularity, compression should merge semantically similar tokens while retaining informative content; across granularities, it should reduce redundancy and prioritizes details absent from coarser representations. In light of this, we propose ResComEmb, a trainable framework for effective and efficient universal multimodal embedding. Specifically, ResComEmb first uses an MLLM (e.g., Qwen2.5-VL’s (Bai et al., 2025)) to encode global, aspect-ratio-aware intermediate, and fine-grained views, providing coarse-to-fine visual evidence. Residual Homogeneity Compression (RHC) then processes the fully contextualized and embedding-projected outputs, consolidating repeated evidence within each granularity while prioritizing new evidence from finer granularities under explicit token budgets. Matryoshka Representation Learning (MRL) (Kusupati et al., 2022) further applies nested supervision to the accumulated coarse-to-fine prefixes, training every compressed representation to remain semantically effective. Then, a length-adaptive Bidirectional Late-Interaction Matching is applied for robust query–document scoring, mitigating weak-match accumulation by averaging the strongest token-level matches in both directions and balancing their scores by valid-token counts. We evaluate ResComEmb on general multimodal tasks using the Massive Multimodal Embedding Benchmark (MMEB) (Jiang et al., 2025), and on visual document retrieval (VDR) task using ViDoRe V1 (Faysse et al., 2025) and V2 (Macé et al., 2025). Extensive experimental results show that ResComEmb achieves the best average performance across all three benchmarks. It reaches 67.4 Precision@1 on MMEB, exceeding VLM2Vec-V2 by 2.5 points, and 90.4 and 61.6 NDCG@5 on ViDoRe V1 and V2, respectively, surpassing ColQwen2.5 by 1.0 point on each benchmark. Notably, these ViDoRe gains are achieved using only 37.5% of full-token ColQwen2.5’s visual-token budget, demonstrating a favorable effectiveness–efficiency trade-off. Ablation studies confirm the contributions of each mechanism in ResComEmb, including Residual Homogeneity Compression (RHC), MRL-based nested supervision, and Bidirectional Late-Interaction Matching. In summary, the key contributions of this work are threefold: • We propose a coarse-to-fine principle for generating universal multimodal embeddings under a certain visual-token budget: gather complementary evidence across multiple granularities while reducing within-granularity redundancy and cross-granularity repetition. • We develop ResComEmb, a trainable framework for effective and efficient multimodal embeddings, which merges redundant tokens within each granularity and prioritizes finer-grained evidence not already captured by coarser representations, subject to a given visual-token budget. • Extensive experiments demonstrate that ResComEmb outperforms VLM2Vec-V2 on MMEB and full-token ColQwen2.5 on ViDoRe V1/V2 while using only 37.5% of the latter’s visual-token budget, establishing a favorable effectiveness–efficiency trade-off. 2 Related Work 2.1 Multimodal Embedding Multimodal embedding maps images and text into a shared representation space for measuring semantic relevance (Radford et al., 2021; Jiang et al., 2025). This field has progressed from contrastive vision–language pre-training to universal MLLM-based representations. CLIP (Radford et al., 2021) and SigLIP (Zhai et al., 2023) use modality-specific encoders for global alignment, whereas BLIP (Li et al., 2022) and CoCa (Yu et al., 2022) add generative language modeling. However, globally pooled retrieval embeddings do not preserve explicit local visual–text correspondences (Yao et al., 2022). MLLMs instead process interleaved image–text sequences in a unified backbone, producing contextualized representations for instruction-conditioned multimodal embedding. E5-V (Jiang et al., 2024) uses task-specific prompts, VLM2Vec (Jiang et al., 2025) and VLM2Vec-V2 (Meng et al., 2026) use instruction-guided contrastive learning, and GME (Zhang et al., 2025) uses synthesized fused-modal data. Most methods encode each input as a single vector, reducing storage and comparison costs but potentially suppressing localized and compositional evidence (Thrush et al., 2022). Multi-vector methods retain independently accessible features, keeping regions, objects, and text separately encoded rather than prematurely pooled (Khattab and Zaharia, 2020; Yao et al., 2022). Recent MLLM-based approaches extend this design to visually rich, interleaved inputs (Faysse et al., 2025; Zhu et al., 2026). Building on hierarchical multi-resolution encoding, ResComEmb introduces trainable Residual Homogeneity Compression to construct compact, input-aligned multi-vector embeddings. 2.2 Visual Representation Compression Visual representation compression shortens visual sequences while retaining task-relevant content (Yang et al., 2025; Ma et al., 2025). Compression occurs either before or after full MLLM contextualization. Upstream approaches target encoding and generation costs: InternVL 1.5 (Chen et al., 2024), TokenPacker (Li et al., 2025), and LLaVA-UHD v2 (Zhang et al., 2026) construct compact visual inputs through spatial or cross-scale aggregation, whereas VisionZip (Yang et al., 2025), SCOPE (Deng et al., 2026), and FOLDER (Wang et al., 2025) apply content-aware selection or merging. These approaches reduce the number of visual tokens processed within the MLLM. Downstream approaches compress contextualized embeddings for storage and comparison. Token Pooling (Clavié et al., 2024) and Light-ColPali (Ma et al., 2025) merge or cluster visual embeddings, whereas AGC (Qin et al., 2026) and SaMer (Park et al., 2026) learn representative centers. MetaEmbed (Xiao et al., 2026) controls length with learnable abstract tokens, enabling global aggregation but potentially omitting input-specific local evidence under tight budgets (Cha et al., 2024; Liu et al., 2024). MURE (Zhu et al., 2026) retains input-aligned multi-resolution evidence, but its compression stage relies on non-trainable clustering and does not explicitly distinguish within-granularity redundancy from cross-granularity repetition. By contrast, ResComEmb introduces RHC as a downstream-trainable compressor for ordered multi-granularity MLLM representations, separately reducing these two forms of redundancy under explicit budgets without modifying the backbone. 3 Method In this section, we first present our problem formulation for budgeted multimodal embedding, and then introduce our ResComEmb framework, as illustrated in Figure 1, which comprises the following mechanisms: Multi-Granularity Visual Encoding, Residual Homogeneity Compression, Nested Representation, and Bidirectional Late-Interaction Matching. Figure 1: Overview of proposed ResComEmb framework. Ordered multi-granularity views are encoded by a shared MLLM, then compressed into nested representations. The Bidirectional Late-Interaction Matching is applied to compute relevance scores between multimodal queries and targets. 3.1 Problem Formulation We formulate budgeted multimodal embedding as learning a mapping from textual and visual inputs, xtx^t and xvx^v, to a compact sequence of token-level embeddings under a visual-token budget BvB_v: ℱ(xt,xv,Bv)⟼(xt,xv)(Bv)∈ℝ(Nt+Mv(Bv))×d,F(x^t,x^v;B_v) _(x^t,x^v)^(B_v) ^(N_t+M_v(B_v))× d, (1) where NtN_t and Mv(Bv)M_v(B_v) denote the numbers of text and visual tokens, respectively, and d is the embedding dimension. The output (xt,xv)(Bv)E_(x^t,x^v)^(B_v) consists of normalized token-level embeddings. All NtN_t text-token embeddings are retained without budget-based compression, whereas the visual tokens are compressed such that Mv(Bv)≤BvM_v(B_v)≤ B_v. 3.2 Multi-Granularity Visual Encoding For each visual input, Multi-Granularity Visual Encoding constructs views at three granularities: a global 1×11× 1 view, an aspect-ratio-aware intermediate view partitioned into 1×21× 2 or 2×12× 1 regions, and a fine-grained view partitioned into 2×22× 2 regions. The three granularities produce (c1,c2,c3)=(1,2,4)(c_1,c_2,c_3)=(1,2,4) crops. The shared MLLM encodes textual inputs together with all visual crops. A learned embedding projection is applied to the last-layer hidden states. The valid projected tokens are separated into text tokens ∈ℝMt×dT ^M_t× d and visual sequences 1,2,3g_1,g_2,g_3, where k∈ℝMk×dg_k ^M_k× d and MkM_k denotes the number of visual tokens at stage k. Text-only inputs retain T and skip visual compression. 3.3 Residual Homogeneity Compression Residual Homogeneity Compression (RHC) takes the three visual token sequences 1,2,3g_1,g_2,g_3 as input and processes them under explicit stage budgets. At each stage, an Intra-Resolution Assessor estimates token importance and within-stage similarity. Then Inter-Resolution Compressor combines these signals with outputs from preceding, coarser stages to merge redundant tokens while preserving evidence not yet represented. The compressed sequences are used to construct nested representations. At stage k, RHC processes only the current sequence kg_k. The outputs of the preceding stages, 1,…,k−1r_1,…,r_k-1, are concatenated to form the coarse-anchor set kA_k, which is used to estimate cross-stage novelty. The first stage has no preceding anchors. Intra-Resolution Token Assessor. The assessor takes the projected tokens of the visual stage and produces importance and within-stage similarity scores for the subsequent compressor. At stage k, a MLP-based contextual encoder is applied to transform the projected visual-token sequence kg_k into contextualized embeddings kh_k. The importance score iS_i for each token i is obtained by i=MinMax(fimp(i)),S_i=MinMax\! (f_imp(h_i) ), (2) where fimpf_imp is the importance scorer, and MinMaxMinMax normalizes scores over tokens in the current stage. To estimate within-stage redundancy, we divide alternating positions into source tokens, A=::2h_A=h_::2, and destination tokens, B=1::2h_B=h_1::2. The similarity score ijC_ij is their cosine similarity: ij=A,i⊤B,j∥A,i∥2∥B,j∥2.C_ij= h_A,i h_B,j _A,i _2 _B,j _2. (3) The assessor then passes S and C to the Inter-Resolution Compressor. Inter-Resolution Token Compressor. The Inter-Resolution Compressor combines the assessor outputs with coarse-stage anchors to remove redundant evidence while preserving information introduced at finer stages. For each current-stage token, the novelty score iN_i quantifies the evidence not covered by the accumulated anchors: i=1,k=1,MinMax(1−max∈ki⊤∥i∥2∥2),k>1.N_i= cases1,&k=1,\\[2.0pt] MinMax\! (1- _a _k h_i a _i _2 _2 ),&k>1. cases (4) The novelty score N and importance score S jointly define the preservation priority vector, i=i+ηiP_i=S_i+ _i, where η controls the contribution of cross-stage novelty. The compressor assigns each candidate source–destination pair a merge score ijU_ij based on its similarity ijC_ij and the source token’s preservation priority iP_i: ij=ij−αi,U_ij=C_ij- _i, where α controls the extent to which preservation priority offsets similarity. Specifically, each source token is assigned to its highest-scoring destination, and merges are performed until the output meets the stage-specific token budget. The token values assigned to each destination are then aggregated and ℓ2 _2-normalized to obtain kr_k. Although the score assignments are discrete, value aggregation remains differentiable. Preservation priorities govern source selection rather than destination assignment: sources are ranked by their best merge scores, making high-priority tokens less likely to be selected for merging. These priority signals also modulate value aggregation, enabling gradients to train the assessor.The detailed derivation is provided in Appendix B. Nested Representation. Then, Nested Representation assembles the surviving outputs 1,…,nr_1,…,r_n in coarse-to-fine order. In accordance with the Matryoshka principle (Kusupati et al., 2022), the assembled sequence supports multiple representation budgets through its prefixes. For an input x, the level-k representation is defined as x,k=Concat(x,x,1,…,x,k),k∈1,…,n,E_x,k=Concat\! (T_x,r_x,1,…,r_x,k ), k∈\1,…,n\, (5) For each k<nk<n, x,kE_x,k is a prefix of x,k+1E_x,k+1. All levels retain the same text prefix and append progressively finer compressed visual stages. The nested representations are generated for both queries and documents using the same procedure. 3.4 Bidirectional Late-Interaction Matching Given the nested representations, Bidirectional Late-Interaction Matching computes a query–document score using similarities in both directions. Let q=[qi]i=1NqE_q=[e_q^i]_i=1^N_q and d=[dj]j=1NdE_d=[e_d^j]_j=1^N_d denote the query and document token sequences, with NqN_q and NdN_d denoting respective numbers of valid tokens. Kq=min(K,Nq),Kd=min(K,Nd).K_q= (K,N_q), K_d= (K,N_d). (6) Let Ωq _q contain the indices of the KqK_q query tokens with the highest maximum similarities to any document token. Analogously, let Ωd _d contain the indices of the KdK_d document tokens with the highest maximum similarities to any query token. The bidirectional Top-K means are defined as SQT(K) S_QT^(K) =1Kq∑i∈Ωqmax1≤j≤Nd(qi)⊤dj,STQ(K) = 1K_q _i∈ _q _1≤ j≤ N_d(e_q^i) e_d^j, S_TQ^(K) =1Kd∑j∈Ωdmax1≤i≤Nq(qi)⊤dj, = 1K_d _j∈ _d _1≤ i≤ N_q(e_q^i) e_d^j, (7) The final query–document score combines the two directional scores as Sfinal(q,d)=w1(q,d)SQT(K)+w2(q,d)STQ(K),S_final(E_q,E_d)=w_1(q,d)S_QT^(K)+w_2(q,d)S_TQ^(K), (8) where SQT(K)S_QT^(K) measures query-to-document support and STQ(K)S_TQ^(K) measures document-to-query coverage. The adaptive weights w1(q,d)w_1(q,d) and w2(q,d)w_2(q,d) balance the two directions according to token counts. When the query and document have equal valid-token counts, both directions receive equal weight. As the document becomes longer, the weighting shifts toward query-to-document support while retaining a contribution from reverse coverage. Appendix D provides more detailed implementation. 3.5 Training The matching score provides the training signal for each active nested representation. Consider a batch ℬ=(qi,di,i−)i=1|ℬ|B=\(q_i,d_i,D_i^-)\_i=1^|B|, where i−D_i^- is an optional set of explicit negatives. For level k, let ℐkI_k index the examples for which that level is active. For each i∈ℐki _k, define the candidate set as i,k=dj:j∈ℐk∪i−C_i,k=\d_j:j _k\ _i^-. The score of any candidate dj∈i,kd_j _i,k is sij(k)=Sfinal(qi,k,dj,k)s_ij^(k)=S_final(E_q_i,k,E_d_j,k). Using these candidate scores, we define the level-wise contrastive loss and the overall training objective as follows: ℒk _k =−1|ℐk|∑i∈ℐklogexp(sii(k)/τ)∑dj∈i,kexp(sij(k)/τ),ℒtrain=∑k∈ℬλkℒk, =- 1|I_k| _i _k \! (s_i^(k)/τ ) _d_j _i,k \! (s_ij^(k)/τ ), _train= _k _B _kL_k, (9) where ℬK_B is the set of levels active for at least one example in the batch, τ is the contrastive temperature, and λk _k weights the loss at level k. Each i,kC_i,k contains the in-batch documents associated with active examples and any available explicit negatives. 4 Experiments We first present the experimental setup and main results on MMEB and ViDoRe V1/V2, followed by ablation studies and analysis. 4.1 Experimental Setup Evaluation Datasets. We evaluate our ResComEmb in general multimodal embedding tasks on MMEB (Jiang et al., 2025) and the visual document retrieval task on ViDoRe V1 (Faysse et al., 2025) and ViDoRe V2 (Macé et al., 2025), reporting Precision@1 for MMEB and NDCG@5 for both ViDoRe benchmarks. More details are provided in Appendix A. Compared Methods. We compare single- and multi-vector models. Single-vector baselines include CLIP (Radford et al., 2021), SigLIP (Zhai et al., 2023), ONE-PEACE (Wang et al., 2023), UniIR (Wei et al., 2024), MagicLens (Zhang et al., 2024b), E5-V (Jiang et al., 2024), VLM2Vec (Jiang et al., 2025), VLM2Vec-V2 (Meng et al., 2026), GME (Zhang et al., 2025), LamRA (Liu et al., 2025), MMRet (Zhou et al., 2025), and mmE5 (Chen et al., 2025); DSE (Ma et al., 2024) and VisRAG-Ret (Yu et al., 2025) provide document-focused comparisons. Multi-vector baselines are ColPali and ColQwen2 (Faysse et al., 2025), which extend late interaction to visual document patches; ColQwen2.5, which applies it to Qwen2.5-VL (Faysse et al., 2025; Bai et al., 2025); ColMate-Pali and ColMate (Masry et al., 2025), which add contrastive late interaction and masked-text supervision; MetaEmbed (Xiao et al., 2026), which uses learnable tokens for compact vector sets; and MURE (Zhu et al., 2026), which combines multi-resolution encoding with token clustering. 4.2 Main Results We compare the overall performance of our proposed ResComEmb method with baseline methods on the multimodal embedding benchmark MMEB in Table 1, and on the visual document retrieval benchmarks ViDoRe V1 and V2 in Tables 2 and 3, respectively. All ResComEmb entries use per-stage RHC budgets of 128/128/128128/128/128(384 in total). We summarize the main observations below. Table 1: MMEB Precision@1 (%). Cls., Ret., and Gnd. abbreviate classification, retrieval, and grounding; IND/OOD denote in-domain/out-of-domain tasks. Bold and underline mark the best and second-best scores in each column. Model Backbone Size Per Meta-Task Score Average Score Cls. VQA Ret. Gnd. IND OOD Avg. Single-Vector Embedding CLIP ViT-L 428M 55.2 19.7 53.2 62.2 47.6 42.8 45.4 UniIR ViT-L 428M 44.3 16.2 61.8 65.3 47.1 41.7 44.7 MagicLens ViT-L 613M 38.8 8.3 35.4 26.0 – – 27.8 VLM2Vec Qwen2-VL 2B 58.7 49.3 65.0 72.9 64.9 53.3 59.7 VLM2Vec-V2 Qwen2-VL 2B 62.9 56.3 69.5 77.3 68.8 59.9 64.9 GME Qwen2-VL 2B 54.4 29.9 66.9 55.5 49.2 55.2 51.9 GME Qwen2-VL 7B 57.7 34.7 71.2 59.3 53.6 58.9 56.0 VLM2Vec Qwen2-VL 7B 62.7 56.9 69.4 82.2 71.4 58.1 65.5 LamRA Qwen2-VL 7B 59.2 26.5 70.0 62.7 53.0 55.4 54.1 LamRA Qwen2.5-VL 7B 51.7 34.1 66.9 56.7 51.7 53.3 52.4 MMRet LLaVA-1.6 Mistral 7B 56.0 57.4 69.9 83.6 68.0 59.1 64.1 Multi-Vector Embedding ColPali PaliGemma 3B 40.3 11.5 48.1 40.3 35.0 34.7 34.9 ResComEmb Qwen2.5-VL 3B 63.1 61.1 69.5 87.9 71.7 62.1 67.4 Overall Performance Comparison on MMEB. ResComEmb achieves the highest MMEB average of 67.4. At a comparable model scale, it outperforms VLM2Vec-V2 by 2.5 points, while also exceeding the larger VLM2Vec-7B by 1.9 points. ResComEmb further leads both the in-domain and out-of-domain averages, showing consistent performance across evaluation domains. Table 2: ViDoRe V1 NDCG@5 (%) across 10 visual document retrieval tasks. Arxiv, Doc, and Info denote ArxivQ, DocQ, and InfoQ; Ener. and Hlth. abbreviate Energy and Health. Model Backbone Size Arxiv Doc Info TabF TATQ Shift AI Ener. Gov. Hlth. Avg. Single-Vector Embedding ONE-PEACE ONE-PEACE 4B 43.9 23.4 59.9 57.0 13.4 17.0 45.4 53.2 55.9 59.5 42.9 E5-V LLaVA-NeXT 8B 41.1 24.3 49.5 58.2 9.0 13.2 46.1 57.7 53.0 59.6 41.2 DSE Phi-3-Vision 4B 78.1 45.8 82.0 79.2 49.0 69.8 96.8 92.6 92.0 96.3 78.2 GME Qwen2-VL 2B 82.8 53.1 90.2 93.3 69.9 89.5 97.5 91.9 94.6 98.7 86.2 GME Qwen2-VL 7B 86.9 57.5 91.6 94.6 74.1 96.8 99.6 95.3 98.8 99.3 89.5 LamRA Qwen2.5-VL 7B 53.0 25.4 72.3 66.1 25.9 27.3 72.0 65.2 72.2 83.8 56.3 VLM2Vec Qwen2-VL 2B 48.9 27.0 67.2 62.6 19.8 41.8 55.0 59.1 57.1 59.6 49.8 VLM2Vec Qwen2-VL 7B 60.2 34.7 70.4 78.2 27.6 38.6 67.7 60.4 61.8 69.9 57.0 VLM2Vec-V2 Qwen2-VL 2B 80.6 44.9 83.7 89.2 43.8 60.8 88.5 86.5 85.0 92.2 75.5 Multi-Vector Embedding ColPali v1.3 PaliGemma 3B 81.7 56.6 84.9 86.9 70.9 75.1 95.7 94.7 93.6 95.9 83.6 ColMate-Pali PaliGemma 3B 83.6 57.5 84.1 87.6 74.0 79.8 98.3 94.1 95.3 96.6 85.1 MURE PaliGemma 3B 84.6 61.7 89.0 89.3 76.8 83.2 98.7 95.2 94.4 97.1 87.0 ColQwen2 Qwen2-VL 2B 88.0 61.5 92.5 89.0 82.2 89.9 99.0 95.9 95.5 98.8 89.2 ColQwen2.5 Qwen2.5-VL 3B 89.2 63.2 92.4 91.1 81.1 87.3 99.6 95.9 96.4 97.9 89.4 ColMate Qwen2.5-VL 3B 90.2 61.1 93.7 91.5 81.9 90.2 99.3 96.4 96.5 98.1 89.9 ResComEmb Qwen2.5-VL 3B 88.8 61.7 94.2 95.2 80.4 90.4 99.6 96.6 97.9 99.3 90.4 Table 3: ViDoRe V2 NDCG@5 (%) across 7 tasks. Syn, Mul, and Bio denote synthetic, multilingual, and biomedical data. Model Backbone Size ESG Human Eco Mul Bio Mul ESG Syn-Mul Bio ESG Syn Eco Avg. Single-Vector Embedding SigLIP SigLIP 652M 28.8 14.0 18.2 21.9 33.8 19.8 29.8 23.8 VisRAG-Ret MiniCPM-V2.0 3B 53.7 48.7 47.7 46.4 54.8 45.9 59.6 51.0 VLM2Vec Qwen2-VL 7B 33.9 42.0 29.7 38.4 38.8 36.7 51.4 38.7 GME Qwen2-VL 7B 65.8 56.2 55.1 56.7 64.0 54.3 62.9 59.3 mmE5 Llama-3.2-Vision 11B 52.8 44.3 46.8 54.7 51.3 55.1 48.6 50.5 Multi-Vector Embedding ColPali PaliGemma 3B 51.1 49.9 56.5 55.7 59.7 57.0 51.6 54.5 ColMate-Pali PaliGemma 3B 62.8 54.1 59.3 53.4 60.9 54.1 55.9 57.2 MURE PaliGemma 3B 67.9 54.5 56.6 57.4 60.4 62.4 57.3 59.5 ColQwen2 Qwen2-VL 2B 62.2 53.2 56.5 54.2 61.8 53.4 61.5 57.5 MetaEmbed Qwen2.5-VL 3B 63.7 55.5 58.7 57.4 61.7 62.6 62.3 60.3 ColQwen2.5 Qwen2.5-VL 3B 68.4 56.5 61.1 57.4 63.6 57.4 59.8 60.6 ColMate Qwen2.5-VL 3B 68.9 52.1 60.3 60.2 62.1 60.1 59.6 60.5 ResComEmb Qwen2.5-VL 3B 69.8 56.0 60.0 57.1 64.8 60.9 62.7 61.6 Overall Performance Comparison on ViDoRe V1/V2. ResComEmb achieves the highest average scores of 90.4 and 61.6 on ViDoRe V1 and V2. Against the strongest specialized baseline methods on each benchmark, it improves over ColMate by 0.5 points on V1 and ColQwen2.5 by 1.0 point on V2, demonstrating consistent gains across both benchmark versions. ResComEmb Remains Effective under Compact Token Budgets. Using the same Qwen2.5-VL-3B backbone, ResComEmb uses 384 visual tokens, 37.5% of ColQwen2.5 token budget, yet improves average NDCG@5 by 1.0 point on both ViDoRe benchmarks. This demonstrates that our method substantially reduces token budget while maintaining superior retrieval effectiveness. 4.3 Ablation Studies and In-Depth Analysis We first ablate MRL-based nested supervision and RHC scoring signals, then analyze nested-prefix scaling and RHC token budgets. Additional analyses of late-interaction scoring, the multi-resolution oracle, and visual-granularity composition appear in Appendices D–F. (a) RHC scoring signals (b) Nested supervision (c) RHC token budget Figure 2: Effects of RHC scoring signals, nested-prefix scaling, and RHC token budgets. (a) Full method versus variants without the importance score (w/o Imp.) or the novelty score (w/o Nov.). (b) Three prefixes from one MRL-trained model. (c) Separately trained models across per-stage budgets. Ablation on MRL-Based Nested Supervision. To quantify the contribution of nested supervision, Table 4 compares matched models trained with and without MRL. We can observe that: 1) Removing MRL lowers ViDoRe V1 by 1.0–2.8 points, ViDoRe V2 by 5.7–7.6 points, and MMEB by 2.8–3.0 points across prefixes containing 128, 256, and 384 visual tokens, showing consistent performance gains with MRL at every representation budget. 2) Across all three prefixes, removing MRL degrades performance on every MMEB task category: classification decreases by 3.6–3.9 points, VQA by 0.5–1.2 points, retrieval by 4.1–5.0 points, and grounding by 1.5–2.5 points. This consistent degradation across heterogeneous task types indicates that nested supervision is essential for preserving task-relevant evidence at each prefix length, rather than benefiting only a subset of tasks. Ablation on RHC Scoring Signals. To isolate each RHC signal’s contribution, we retrain matched variants with either the importance or novelty score removed. The experiments results on ViDoRe V1/V2 are presented in Figure 2(a). We can observe: 1) Either removal hurts the model performance. Removing the importance score lowers ViDoRe V1 by 0.3 points and V2 by 2.1 points, while removing the novelty score lowers V1 by 0.7 points and V2 by 3.5 points, confirming their complementary roles in token preservation. 2) Removing the novelty score causes larger drops on both benchmarks, suggesting its greater contribution to RHC. This is consistent with its role in distinguishing residual fine-grained evidence from content already represented by coarser views, reducing cross-granularity repetition while preserving complementary details in the multi-resolution representation. More detailed analysis are provided in Appendix Tables 5–6. Table 4: Effect of MRL across nested prefixes. Prefixes (1)E^(1), (2)E^(2), and (3)E^(3) retain 128, 256, and 384 visual tokens, respectively, with identical text tokens; Δ denotes the drop without MRL. Prefix Variant ViDoRe MMEB V1 V2 Avg. Cls. VQA Ret. Gnd. Avg. (1)E^(1) w/ MRL 89.2 60.8 77.5 62.8 61.2 67.7 86.8 66.7 w/o MRL 86.4 53.2 72.7 58.9 60.0 63.6 85.1 63.7 Δ (2.8↓ ) (7.6↓ ) (4.8↓ ) (3.9↓ ) (1.2↓ ) (4.1↓ ) (1.7↓ ) (3.0↓ ) (2)E^(2) w/ MRL 89.6 61.4 78.0 62.8 61.4 68.2 86.8 66.9 w/o MRL 88.6 55.2 74.8 59.2 60.4 64.1 85.3 64.1 Δ (1.0↓ ) (6.2↓ ) (3.2↓ ) (3.6↓ ) (1.0↓ ) (4.1↓ ) (1.5↓ ) (2.8↓ ) (3)E^(3) w/ MRL 90.4 61.6 78.5 63.1 61.1 69.5 87.9 67.4 w/o MRL 89.1 55.9 75.4 59.5 60.6 64.5 85.4 64.4 Δ (1.3↓ ) (5.7↓ ) (3.1↓ ) (3.6↓ ) (0.5↓ ) (5.0↓ ) (2.5↓ ) (3.0↓ ) Performance Analysis of Nested Prefixes. To assess elastic inference capability of our ResComEmb, Figure 2(b) evaluates three prefixes from the same MRL-trained model: (1)E^(1), (2)E^(2), and (3)E^(3), which retain 128, 256, and 384 visual vectors, respectively. We can observe: 1) The 128-token prefix retains 98.7% of the full 384-token prefix’s performance on ViDoRe V1, 98.7% on ViDoRe V2, and 99.0% on MMEB, showing that the coarsest prefix captures most task-relevant multimodal evidence and enables low-storage inference without retraining. 2) From 128 to 384 tokens, performance increases monotonically by 1.2 points on V1, 0.8 points on V2, and 0.7 points on MMEB, indicating that later prefixes progressively add complementary fine-grained evidence and provide a controllable effectiveness–efficiency trade-off of our model. Performance Analysis of RHC Token Budgets. To study the effectiveness–efficiency trade-off, Figure 2(c) compares separately trained models with per-stage budgets of 16, 32, 64, 128, and 256 visual tokens, corresponding to total budgets of 48, 96, 192, 384, and 768 visual tokens. We observe: 1) Scaling from 48 to 192 vectors improves ResComEmb by 3.3 points on ViDoRe V1, 5.1 points on ViDoRe V2, and 1.0 point on MMEB, showing that capacity matters under tight compression. 2) Increasing from 192 to 384 vectors yields further gains of 0.9 points on V1, 1.0 point on V2, and 0.1 point on MMEB. Doubling further to 768 vectors adds only 0.2 points on V1 and 0.1 point on V2, while reducing MMEB by 0.7 points. This pattern of diminishing and eventually negative returns indicates that 384 vectors offer the best effectiveness–efficiency balance, motivating the 128/128/128 operating point in this work. More detailed experimental results are provided in Appendix Table 9. 5 Conclusion In this work, we introduce ResComEmb that combines residual compression, nested supervision, and bidirectional late interaction to build compact universal multimodal embeddings. Its trainable coarse-to-fine design reduces within-granularity redundancy and cross-granularity repetition while preserving residual fine-grained evidence under explicit token budgets. It establishes new state-of-the-art results on MMEB and ViDoRe V1/V2, outperforming both universal multimodal retrievers and specialized visual document retrieval models. With only 37.5% of the visual-token budget of full-token ColQwen2.5, ResComEmb also achieves higher retrieval accuracy on both ViDoRe benchmarks. Our results show that disentangling within-granularity redundancy from cross-granularity repetition allows representation size to be substantially reduced while preserving complementary visual information. This perspective suggests that redundancy in multi-vector embeddings is structured, motivating future work on task-adaptive redundancy modeling and the broader application of trainable compression to other representation generation settings. AI use statement In this work, we used generative AI tools to assist with translation and interpretation of experimental results, literature search, manuscript organization, language editing, LaTeX formatting, and drafting the gradient analysis in Appendix B. We independently checked that analysis against the implementation and verified its gradients with automatic differentiation. We have not used generative AI tools to generate synthetic data sets or experimental results. We have reviewed all AI-assisted work and take responsibility for the final content of this work, including text, claims, or artifacts produced with the aid of generative AI. Ethics statement This work uses publicly available benchmark datasets, including MMEB, ViDoRe V1, and ViDoRe V2, for multimodal representation learning and retrieval evaluation. We do not collect new data from human participants, conduct human-subject experiments, or release personal or sensitive information. The experiments follow the cited benchmark protocols and use these resources for research evaluation. Benchmark composition may reflect domain, language, synthetic-data, or annotation biases; therefore, the reported results should not be interpreted as evidence of fairness or suitability for high-stakes deployment. We are not aware of additional privacy, security, legal, or conflict-of-interest concerns introduced by the method as evaluated in this work. Reproducibility Statement Reproducing our experiments requires only publicly released data resources. The exact Hugging Face repositories used to assemble the training pairs, obtain explicit hard negatives, and conduct evaluation are listed below: • MMEB training set: https://huggingface.co/datasets/TIGER-Lab/MMEB-train • ViDoRe training set: https://huggingface.co/datasets/vidore/colpali_train_set • MoCa hard negatives: https://huggingface.co/datasets/moca-embed/MoCa-CL-Pairs • MMEB test set: https://huggingface.co/datasets/TIGER-Lab/MMEB-eval • ViDoRe V1 test sets: https://huggingface.co/collections/vidore/vidore-benchmark-667173f98e70a1c0fa4db00d • ViDoRe V2 test sets: https://huggingface.co/collections/vidore/vidore-benchmark-v2-dev-67ae03e3924e85b36e7f53b0 Model initialization uses the following public checkpoint: • ColQwen2.5-base: https://huggingface.co/vidore/colqwen2.5-base Section 4 defines the evaluation protocol, while Appendix A records the benchmark composition and complete training configuration. References Bai et al. (2025) S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin Qwen2.5-VL technical report. CoRR abs/2502.13923. External Links: Document, 2502.13923 Cited by: §1, §1, §4.1. Beyer et al. (2024) L. Beyer, A. Steiner, A. S. Pinto, A. Kolesnikov, X. Wang, D. Salz, M. Neumann, I. Alabdulmohsin, M. Tschannen, E. Bugliarello, T. Unterthiner, D. Keysers, S. Koppula, F. Liu, A. Grycner, A. Gritsenko, N. Houlsby, M. Kumar, K. Rong, J. Eisenschlos, R. Kabra, M. Bauer, M. Bošnjak, X. Chen, M. Minderer, P. Voigtlaender, I. Bica, I. Balazevic, J. Puigcerver, P. Papalampidi, O. Henaff, X. Xiong, R. Soricut, J. Harmsen, and X. Zhai PaliGemma: a versatile 3b VLM for transfer. arXiv preprint arXiv:2407.07726. External Links: Document, Link Cited by: §1. Bolya et al. (2023) D. Bolya, C. Fu, X. Dai, P. Zhang, C. Feichtenhofer, and J. Hoffman Token merging: your ViT but faster. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: Appendix B, §1. Cha et al. (2024) J. Cha, W. Kang, J. Mun, and B. Roh Honeybee: locality-enhanced projector for multimodal LLM. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 13817–13827. Cited by: §1, §2.2. Chen et al. (2025) H. Chen, L. Wang, N. Yang, Y. Zhu, Z. Zhao, F. Wei, and Z. Dou MmE5: improving multimodal multilingual embeddings via high-quality synthetic data. In Findings of the Association for Computational Linguistics: ACL 2025, p. 8254–8275. External Links: Document, Link Cited by: §4.1. Chen et al. (2024) Z. Chen, W. Wang, H. Tian, S. Ye, Z. Gao, E. Cui, W. Tong, K. Hu, J. Luo, Z. Ma, et al. How far are we to GPT-4V? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences 67 (12), p. 220101. Cited by: §2.2. Clavié et al. (2024) B. Clavié, A. Chaffin, and G. Adams Reducing the footprint of multi-vector retrieval with minimal performance impact via token pooling. arXiv preprint arXiv:2409.14683. Cited by: §2.2. Deng et al. (2009) J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei ImageNet: a large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, p. 248–255. External Links: Document Cited by: §1. Deng et al. (2026) J. Deng, W. Li, J. T. Zhou, and Y. He SCOPE: saliency-coverage oriented token pruning for efficient multimodal LLMs. Advances in Neural Information Processing Systems 38, p. 161527–161552. Cited by: §2.2. Faysse et al. (2025) M. Faysse, H. Sibille, T. Wu, B. Omrani, G. Viaud, C. Hudelot, and P. Colombo ColPali: efficient document retrieval with vision language models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, Cited by: Appendix A, §1, §1, §2.1, §4.1, §4.1. Gordo et al. (2016) A. Gordo, J. Almazán, J. Revaud, and D. Larlus Deep image retrieval: learning global representations for image search. In Computer Vision – ECCV 2016, p. 241–257. External Links: Document Cited by: §1. Hu et al. (2018) H. Hu, W. Chao, and F. Sha Learning answer embeddings for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 5428–5436. External Links: Document Cited by: §1. Jiang et al. (2024) T. Jiang, M. Song, Z. Zhang, H. Huang, W. Deng, F. Sun, Q. Zhang, D. Wang, and F. Zhuang E5-V: universal embeddings with multimodal large language models. arXiv preprint arXiv:2407.12580. External Links: Document, Link Cited by: §1, §2.1, §4.1. Jiang et al. (2025) Z. Jiang, R. Meng, X. Yang, S. Yavuz, Y. Zhou, and W. Chen VLM2vec: training vision-language models for massive multimodal embedding tasks. In The Thirteenth International Conference on Learning Representations, Cited by: Appendix A, §1, §1, §1, §2.1, §4.1, §4.1. Khattab and Zaharia (2020) O. Khattab and M. Zaharia ColBERT: efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 39–48. Cited by: §2.1. Kusupati et al. (2022) A. Kusupati, G. Bhatt, A. Rege, M. Wallingford, A. Sinha, V. Ramanujan, W. Howard-Snyder, K. Chen, S. Kakade, P. Jain, et al. Matryoshka representation learning. Advances in Neural Information Processing Systems 35, p. 30233–30249. Cited by: §1, §3.3. Li et al. (2022) J. Li, D. Li, C. Xiong, and S. C. H. Hoi BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the 39th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 162, p. 12888–12900. External Links: Link Cited by: §2.1. Li et al. (2025) W. Li, Y. Yuan, J. Liu, D. Tang, S. Wang, J. Qin, J. Zhu, and L. Zhang TokenPacker: efficient visual projector for multimodal LLM. International Journal of Computer Vision 133 (10), p. 6794–6812. Cited by: §2.2. Liu et al. (2024) H. Liu, Q. You, X. Han, Y. Liu, H. Huang, R. He, and H. Yang Visual anchors are strong information aggregators for multimodal large language model. Advances in Neural Information Processing Systems 37, p. 17696–17718. Cited by: §1, §2.2. Liu et al. (2025) Y. Liu, Y. Zhang, J. Cai, X. Jiang, Y. Hu, J. Yao, Y. Wang, and W. Xie LamRA: large multimodal model as your advanced retrieval assistant. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 4015–4025. Cited by: §4.1. Ma et al. (2024) X. Ma, S. Lin, M. Li, W. Chen, and J. Lin Unifying multimodal retrieval via document screenshot embedding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 6492–6505. External Links: Document, Link Cited by: §4.1. Ma et al. (2025) Y. Ma, J. Li, Y. Zang, X. Wu, X. Dong, P. Zhang, Y. Cao, H. Duan, J. Wang, Y. Cao, et al. Towards storage-efficient visual document retrieval: an empirical study on reducing patch-level embeddings. In Findings of the Association for Computational Linguistics: ACL 2025, p. 19568–19580. Cited by: §1, §2.2. Macé et al. (2025) Q. Macé, A. Loison, and M. Faysse ViDoRe benchmark v2: raising the bar for visual retrieval. arXiv preprint arXiv:2505.17166. Cited by: Appendix A, §1, §4.1. Masry et al. (2025) A. Masry, M. Thakkar, P. Bechard, S. T. Madhusudhan, R. Awal, S. Mishra, A. K. Suresh, S. Daruru, E. Hoque, S. Gella, T. Scholak, and S. Rajeswar ColMate: contrastive late interaction and masked text for multimodal document retrieval. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, p. 2071–2080. External Links: Document, Link Cited by: §4.1. Meng et al. (2026) R. Meng, Z. Jiang, Y. Liu, M. Su, X. Yang, Y. Fu, C. Qin, R. Thirukovalluru, X. Zhang, Z. Chen, R. Xu, C. Xiong, Y. Zhou, W. Chen, and S. Yavuz VLM2Vec-V2: advancing multimodal embedding for videos, images, and visual documents. Transactions on Machine Learning Research. External Links: Link Cited by: §2.1, §4.1. Park et al. (2026) S. Park, J. Jung, J. Park, and J. Kang Do all visual tokens matter equally? object-evidence preserving token merging for vision-language retrieval. arXiv preprint arXiv:2607.04605. Cited by: §2.2. Prillo and Eisenschlos (2020) S. Prillo and J. Eisenschlos SoftSort: a continuous relaxation for the argsort operator. In Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 119, p. 7793–7802. External Links: Link Cited by: Appendix B. Qin et al. (2026) H. Qin, A. Martin, R. Jha, C. Zuo, R. Kriz, and B. Van Durme Multi-vector index compression in any modality. In Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’26, New York, NY, USA, p. 1519–1530. External Links: ISBN 9798400725999, Link, Document Cited by: §2.2. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 139, p. 8748–8763. External Links: Link Cited by: §1, §2.1, §4.1. Thrush et al. (2022) T. Thrush, R. Jiang, M. Bartolo, A. Singh, A. Williams, D. Kiela, and C. Ross Winoground: probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 5238–5248. External Links: Document Cited by: §1, §2.1. Wang et al. (2025) H. Wang, Z. Yu, G. Spadaro, C. Ju, V. Quétu, S. Xiao, and E. Tartaglione FOLDER: accelerating multi-modal large language models with enhanced performance. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), Vol. , p. 23614–23625. External Links: Document Cited by: §1, §2.2. Wang et al. (2023) P. Wang, S. Wang, J. Lin, S. Bai, X. Zhou, J. Zhou, X. Wang, and C. Zhou ONE-PEACE: exploring one general representation model toward unlimited modalities. arXiv preprint arXiv:2305.11172. External Links: Document, Link Cited by: §4.1. Wei et al. (2024) C. Wei, Y. Chen, H. Chen, H. Hu, G. Zhang, J. Fu, A. Ritter, and W. Chen UniIR: training and benchmarking universal multimodal information retrievers. In Computer Vision – ECCV 2024, Lecture Notes in Computer Science, Vol. 15145, p. 387–404. External Links: Document Cited by: §4.1. Wen et al. (2025) Z. Wen, Y. Gao, S. Wang, J. Zhang, Q. Zhang, W. Li, C. He, and L. Zhang Stop looking for “important tokens” in multimodal language models: duplication matters more. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 9972–9991. Cited by: §1. Xiao et al. (2026) Z. Xiao, Q. Ma, M. Gu, C. J. Chen, X. Chen, V. Ordonez, and V. Mohan MetaEmbed: scaling multimodal retrieval at test-time with flexible late interaction. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §2.2, §4.1. Yang et al. (2025) S. Yang, Y. Chen, Z. Tian, C. Wang, J. Li, B. Yu, and J. Jia VisionZip: longer is better but not necessary in vision language models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 19792–19802. External Links: Document Cited by: §1, §2.2. Yao et al. (2022) L. Yao, R. Huang, L. Hou, G. Lu, M. Niu, H. Xu, X. Liang, Z. Li, X. Jiang, and C. Xu FILIP: fine-grained interactive language-image pre-training. In International Conference on Learning Representations, External Links: Link Cited by: §1, §2.1. Yu et al. (2022) J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu CoCa: contrastive captioners are image-text foundation models. Transactions on Machine Learning Research. External Links: Link Cited by: §2.1. Yu et al. (2016) L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg Modeling context in referring expressions. In Computer Vision – ECCV 2016, B. Leibe, J. Matas, N. Sebe, and M. Welling (Eds.), Cham, p. 69–85. External Links: ISBN 978-3-319-46475-6 Cited by: §1. Yu et al. (2025) S. Yu, C. Tang, B. Xu, J. Cui, J. Ran, Y. Yan, Z. Liu, S. Wang, X. Han, Z. Liu, and M. Sun VisRAG: vision-based retrieval-augmented generation on multi-modality documents. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §4.1. Zhai et al. (2023) X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 11975–11986. External Links: Document Cited by: §1, §2.1, §4.1. Zhang et al. (2024a) B. Zhang, P. Zhang, X. Dong, Y. Zang, and J. Wang Long-CLIP: unlocking the long-text capability of CLIP. arXiv preprint arXiv:2403.15378. External Links: Document, Link Cited by: §1. Zhang et al. (2024b) K. Zhang, Y. Luan, H. Hu, K. Lee, S. Qiao, W. Chen, Y. Su, and M. Chang MagicLens: self-supervised image retrieval with open-ended instructions. In Proceedings of the 41st International Conference on Machine Learning, External Links: Link Cited by: §4.1. Zhang et al. (2025) X. Zhang, Y. Zhang, W. Xie, M. Li, Z. Dai, D. Long, P. Xie, M. Zhang, W. Li, and M. Zhang Bridging modalities: improving universal multimodal retrieval by multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 9274–9285. Cited by: §2.1, §4.1. Zhang et al. (2026) Y. Zhang, Y. Liu, Z. Guo, Y. Zhang, X. Yang, X. Zhang, C. Chen, J. Song, Y. Yao, T. Chua, et al. LLaVA-UHD v2: exploiting hierarchical vision granularity in MLLMs via inverse semantic pyramid. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 12934–12942. Cited by: §2.2. Zhou et al. (2025) J. Zhou, Y. Xiong, Z. Liu, Z. Liu, S. Xiao, Y. Wang, B. Zhao, C. J. Zhang, and D. Lian MegaPairs: massive data synthesis for universal multimodal retrieval. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 19076–19095. External Links: Document, Link Cited by: §4.1. Zhu et al. (2026) F. Zhu, Z. Cai, Y. Wang, P. Shao, W. Wang, F. Feng, R. Hong, and T. Chua MURE: hierarchical multi-resolution encoding via vision-language models for visual document retrieval. In Proceedings of the 2026 International Conference on Multimedia Retrieval, p. 69–78. External Links: Document Cited by: Appendix E, §1, §1, §2.1, §2.2, §4.1. Appendix A Experimental Details MMEB. The Massive Multimodal Embedding Benchmark (MMEB) (Jiang et al., 2025) evaluates general multimodal embedding across 36 tasks: 10 classification, 10 visual question answering, 12 retrieval, and 4 visual grounding tasks. Precision@1 is computed for each task and macro-averaged; the in-domain and out-of-domain partitions contain 20 and 16 tasks, respectively. ViDoRe V1. ViDoRe V1 (Faysse et al., 2025) contains 10 page-level visual document retrieval subsets spanning academic and practical domains, document types including text, tables, figures, and infographics, and both English and French content. We report NDCG@5 on each subset and their macro-average. ViDoRe V2. ViDoRe V2 (Macé et al., 2025) contains 7 more challenging out-of-domain subsets drawn from ESG, economics, and biomedical documents, including human-authored, synthetic, and multilingual settings. We report NDCG@5 on each subset and their macro-average. Optimization Details. We optimize with AdamW using random seed 42. All models are trained for 30,000 steps with a global batch size of 64, using eight examples per GPU across eight NVIDIA A100 GPUs. The LoRA rank and scaling factor are both 32. The learning rate decays linearly from 10−410^-4 to zero, and the LoRA dropout is 0.1. The contrastive temperature is 0.03. Training uses bfloat16, FlashAttention-2, and gradient checkpointing. All active nested prefixes receive supervision; an ablation changes this objective only when explicitly stated. Appendix B Gradient Flow through Residual Homogeneity Compression RHC combines discrete matching with differentiable value aggregation. Because argsort has zero input gradients almost everywhere (Prillo and Eisenschlos, 2020), the matching indices are treated as fixed during backpropagation and recomputed after each update. As in hard token merging with average pooling (Bolya et al., 2023), gradients pass through the merged values; Equation 9 further connects the assessor to the retrieval loss without a straight-through estimator or auxiliary compression loss. Fixed-Assignment Formulation. At a compressed stage k, let k,ii=1Mk\g_k,i\_i=1^M_k be the projected tokens and ih_i their contextualized features. Define the raw importance and gate logits as ζi=fimp(i) _i=f_imp(h_i) and ui=fgate(i)u_i=f_gate(h_i). With equal weighting of importance and novelty, the relative importance iS_i, cross-stage novelty iN_i, gate iG_i, and modulation weight iW_i give i=MinMax(ζ)i,i=sigmoid(ui),i=12(i+i),~k,i=(1+ρii)k,i.S_i=MinMax(ζ)_i, _i=sigmoid(u_i), _i= 12(S_i+N_i), g_k,i=(1+ _iW_i)g_k,i. (10) MinMax is computed across the stage, so iS_i is relative rather than absolute. Here ρ≥0ρ≥ 0 is the modulation strength. Since i,i,i∈[0,1]S_i,N_i,G_i∈[0,1], the modulation is bounded by [1,1+ρ][1,1+ρ] and preserves each token’s direction. Let Rk=|k|R_k=|r_k| be the number of retained tokens and (k)∈0,1Rk×Mk ^(k)∈\0,1\^R_k× M_k the final partition induced by the recursive merge rounds, where Πℓi(k)=1 ^(k)_ i=1 assigns input i to group ℓ . With group mass mℓ=∑iΠℓi(k)m_ = _i ^(k)_ i and pre-normalized mean ℓy_ , the aggregation is ℓ=1mℓ∑iΠℓi(k)~k,i,k,ℓ=ℓ∥ℓ∥2.y_ = 1m_ _i ^(k)_ i g_k,i, _k, = y_ _ _2. (11) This identity follows inductively because merging groups of masses mam_a and mbm_b replaces their means by (maa+mbb)/(ma+mb)(m_ay_a+m_by_b)/(m_a+m_b), preserving the mean of all original members. Almost-Everywhere Differentiability. Assume strict destination-selection and source-ranking margins at every merge round, unique MinMax extrema with positive range, unique maximizing coarse anchors where novelty is evaluated, and nonzero group means ℓy_ . Then (k) ^(k) is locally constant, and Equations 10–11 define a differentiable value path conditioned on that partition. The chain rule therefore yields its exact local gradient. Switching ties are measure-zero boundaries at which the hard map is not differentiable. Task-Aligned Gradient to the Assessor. For query b with positive document dbd_b at level k, let sbc(k)s_bc^(k) and πbc(k) _bc^(k) be the score and softmax probability of candidate document c, respectively. Let ℐkI_k be the active-query set, λk _k its level weight, and τ the contrastive temperature. Equation 9 give ∂ℒtrain∂sbc(k)=λk|ℐk|τ(πbc(k)−[c=db]), _train∂ s_bc^(k)= _k|I_k|τ ( _bc^(k)-I[c=d_b] ), (12) where [⋅]I[·] is the indicator function. Thus the positive candidate receives a negative score gradient and every negative a positive one. Away from ties, the current MaxSim and TopK matches route this signal to the stage outputs. Let ℓ=∇k,ℓℒtrain ψ_ = _r_k, L_train be this upstream gradient, let I be the identity in the d-dimensional retrieval space, and let κ(i)κ(i) be the group containing input i. Averaging and L2 normalization give ϵi=∇~k,iℒtrain=−k,κ(i)k,κ(i)⊤mκ(i)∥κ(i)∥2κ(i). ε_i= _ g_k,iL_train= I-r_k,κ(i)r_k,κ(i) m_κ(i) _κ(i) _2 ψ_κ(i). (13) Thus ϵi ε_i connects every retained group member to the loss. Define χi=k,i⊤ϵi _i=g_k,i ε_i as the sensitivity to token i’s scalar contribution. Then ∂ℒtrain∂i=ρiχi,∂ℒtrain∂i=∂ℒtrain∂i=ρ2iχi. _train _i= _i _i, _train _i= _train _i= ρ2G_i _i. (14) Since i=sigmoid(ui)G_i=sigmoid(u_i), the gate logit receives ∂ℒtrain/∂ui=(∂ℒtrain/∂i)i(1−i) _train/∂ u_i=( _train/ _i)G_i(1-G_i). For ρ>0ρ>0 (and i>0W_i>0 for the gate), χi<0 _i<0 favors larger relative saliency and gating, whereas χi>0 _i>0 favors smaller values. Let =(ζ1,…,ζMk) ζ=( _1,…, _M_k) and let 1 be the all-ones vector. For any scalar shift ξ, MinMax(+ξ)=MinMax(),ξ∈ℝ,MinMax( ζ+ 1)=MinMax( ζ), ξ , (15) so uniformly lowering every raw logit leaves S unchanged and cannot reduce the objective. For the MinMax operation, let imin=argminjζji_ = _j _j, imax=argmaxjζji_ = _j _j, and Δ=ζimax−ζimin = _i_ - _i_ . Its Jacobian is Jijmm=∂i∂ζj=δij−δiminjΔ−(ζi−ζimin)(δimaxj−δiminj)Δ2,J^m_ij= _i∂ _j= _ij- _i_ j - ( _i- _i_ )( _i_ j- _i_ j) ^2, (16) where δij _ij is the Kronecker delta. For the importance branch, the chain rule maps ∂ℒtrain/∂i _train/ _i through JmmJ^m and then through fimpf_imp. Separately, the sigmoid relation above maps the gate gradient through fgatef_gate. Hence, both heads receive retrieval supervision whenever their corresponding local derivatives are nonzero. The learned saliency is aligned with hard preservation. Write ij=ij−α(i+ηi)U_ij=C_ij-α(S_i+ _i), where ijC_ij is pairwise similarity, α>0α>0 is the preservation coefficient, and η≥0η≥ 0 weights novelty. Its direct dependence on source saliency satisfies ∂ij∂i=−α<0,argmaxjij=argmaxjij. _ij _i=-α<0, _jU_ij= _jC_ij. (17) Equation 17 states monotonicity of the hard rule, not a gradient through the assignment. Thus iS_i is interpreted as task-aligned relative importance: increasing it protects source i without changing its preferred destination. During backpropagation, the partition is fixed and then recomputed after the parameter update. Nonzero assessor updates require an active retrieval signal and within-group relative variation. Later-stage novelty treats accumulated coarse anchors as fixed references, while every earlier stage remains directly supervised by its nested objective. This analysis establishes trainability and directional alignment; the empirical contribution of these signals is evaluated in Figure 2(a) and Tables 5–6. Appendix C Detailed RHC Scoring-Signal Results We retrain two controlled variants using the same initialization, stage budgets, and optimization as the complete model. The w/o Importance Score variant removes iS_i from the preservation priority, giving i=ηiP_i= _i, whereas w/o Novelty Score removes iN_i, giving i=iP_i=S_i. Both variants retain the similarity score ijC_ij, the merging procedure, nested supervision, and late-interaction scoring. Table 5: Ablation of RHC scoring signals on all ViDoRe V1 subsets (NDCG@5 %). Variant ArxivQ DocQ InfoQ TabF TATQ Shift AI Energy Gov. Health Avg. ResComEmb 88.8 61.7 94.2 95.2 80.4 90.4 99.6 96.6 97.9 99.3 90.4 w/o Importance Score 87.3 63.0 93.6 94.6 80.8 88.9 99.5 96.5 97.5 99.1 90.1 w/o Novelty Score 86.6 62.5 93.8 94.2 80.7 87.4 99.3 96.5 96.5 99.2 89.7 Table 6: Ablation of RHC scoring signals on all ViDoRe V2 subsets (NDCG@5 %). Syn, Mul, and Bio denote synthetic, multilingual, and biomedical data. Variant ESG Human Eco Mul Bio Mul ESG Syn-Mul Bio ESG Syn Eco Avg. ResComEmb 69.8 56.0 60.0 57.1 64.8 60.9 62.7 61.6 w/o Importance Score 68.4 54.7 57.3 55.9 60.5 59.6 60.0 59.5 w/o Novelty Score 67.5 51.2 56.8 48.8 62.3 57.3 62.7 58.1 The aggregate degradation on ViDoRe V2 is concentrated rather than uniform. Removing the novelty score costs 4.8 points on Eco Mul and 8.3 points on ESG Syn-Mul, while removing the importance score has its largest effect of 4.3 points on Bio. Because RHC compresses coarse to fine, later stages without novelty rank candidates only by salience; this leaves cross-stage redundancy unpenalized, whereas novelty promotes candidates weakly covered by earlier stages. The concentration of the losses on multilingual and synthetic subsets suggests that this complementary coverage is more consequential under those distribution shifts. Appendix D Detailed Late-Interaction Results The directional weights used in the late-interaction score are computed from the numbers of valid query and document tokens. Specifically, w1(q,d) w_1(q,d) =minρmax,max(12,NdNq+Nd), = \! \ _ , \! ( 12, N_dN_q+N_d ) \, (18) w2(q,d) w_2(q,d) =1−w1(q,d), =1-w_1(q,d), where NqN_q and NdN_d denote the numbers of valid query and document tokens, respectively, and ρmax _ caps the weight assigned to the query-to-document direction. Thus, the two directions receive equal weight when Nq=NdN_q=N_d, while longer documents increase the relative weight of query-to-document support without eliminating the reverse direction. Table 7: Late-interaction ablations by benchmark and MMEB category. w/o Adaptive Weighting keeps both directions with fixed 0.5/0.50.5/0.5 weights; w/o Bidirectionality retains only query-to-document TopK-mean MaxSim. ViDoRe reports NDCG@5 (%); MMEB reports Precision@1 (%). ViDoRe Avg. weights V1/V2 by their 10/7 subsets. Variant ViDoRe MMEB V1 V2 Avg. Cls. VQA Ret. Gnd. Avg. Vanilla MaxSim 90.0 59.8 77.6 41.3 15.1 62.7 71.8 44.5 ResComEmb 90.4 61.6 78.5 63.1 61.1 69.5 87.9 67.4 w/o Mean 90.0 59.1 77.3 54.6 41.3 64.5 89.3 58.1 w/o TopK 89.1 54.5 74.9 61.2 55.7 66.7 81.4 63.8 w/o Adaptive Weighting 89.0 54.0 74.6 62.8 61.3 67.4 89.3 66.9 w/o Bidirectionality 90.0 59.7 77.5 61.9 57.5 67.2 88.6 65.4 Table 7 demonstrates that the reported scoring operations address complementary failure modes. Reverting to vanilla MaxSim reduces the MMEB average by 22.9 points, while removing mean aggregation alone incurs a 9.3-point drop, confirming the importance of normalized evidence aggregation for heterogeneous multimodal inputs. TopK selection and length-adaptive weighting are particularly important on ViDoRe V2: removing TopK and replacing the adaptive weights with a fixed 0.5/0.50.5/0.5 average lower performance by 7.1 and 7.6 points, respectively. Removing bidirectionality lowers the MMEB average by 2.0 points, isolating the contribution of reverse document-to-query evidence while retaining query-to-document TopK-mean MaxSim. Appendix E Preliminary Multi-Granularity Study Following the combined-selector diagnostic of MURE (Zhu et al., 2026), we train and evaluate global (g1g_1), aspect-ratio-aware intermediate (g2g_2), and fine-grained (g3g_3) models separately. A per-query oracle then retains the best retrieval outcome using ground-truth relevance, estimating the potential of adaptive multi-resolution perception rather than a deployable routing method. Figure 3: Single-granularity and oracle combined-selector results. ViDoRe V1/V2 use NDCG@5; MMEB uses Precision@1. The strongest fixed granularity differs across benchmarks, whereas the per-query oracle improves over it by 4.7 points on V1, 10.8 on V2, and 10.2 on MMEB Precision@1. The larger improvement on V2 indicates that complementary scale evidence becomes more important under domain shift, while the MMEB result extends this observation beyond document retrieval. These gains motivate preserving multiple visual scales before RHC compression. Appendix F Additional Multi-Granularity Ablations We remove one visual granularity at a time and compare each variant with the complete representation under the same total output budget of 384 visual tokens. The w/og1w/o~g_1, w/og2w/o~g_2, and w/og3w/o~g_3 variants use stage budgets 0/192/1920/192/192, 192/0/192192/0/192, and 192/192/0192/192/0, respectively, and supervise only the retained stages. All variants follow the same training setup. This controls representation capacity when measuring the contribution of each granularity. Table 8: Visual-granularity ablations at a fixed total budget of 384 visual tokens. ViDoRe reports NDCG@5 (%); MMEB reports Precision@1 (%). ViDoRe Avg. weights V1/V2 by their 10/7 subsets. Variant Tokens ViDoRe MMEB V1 V2 Avg. Cls. VQA Ret. Gnd. Avg. ResComEmb 384 90.4 61.6 78.5 63.1 61.1 69.5 87.9 67.4 w/o g1g_1 384 90.0 59.1 77.3 62.1 61.2 66.4 89.5 66.3 w/o g2g_2 384 90.4 61.9 78.7 62.8 61.3 67.5 89.6 66.9 w/o g3g_3 384 90.2 61.5 78.4 62.9 61.2 67.8 89.2 67.0 Under the matched 384-vector budget, removing the global view causes the largest document-retrieval degradation, confirming its role as the structural anchor of the hierarchy. On MMEB, every two-view variant underperforms the complete representation, showing that global, intermediate, and fine-grained evidence are complementary for universal multimodal tasks. The full hierarchy is therefore selected for its strongest performance across benchmark families rather than for any single dataset. Appendix G Additional Token-Budget Ablations Table 9 compares five symmetric per-granularity output token budgets. Table 9: Per-stage token-budget ablations for RHC. Stage budgets are written as b1/b2/b3b_1/b_2/b_3; Tokens counts the total visual budget. ViDoRe reports NDCG@5 (%); MMEB reports Precision@1 (%). ViDoRe Avg. weights V1/V2 by their 10/7 subsets. Stage budget Tokens ViDoRe MMEB V1 V2 Avg. Cls. VQA Ret. Gnd. Avg. 16/16/1616/16/16 48 86.2 55.5 73.6 62.0 60.9 66.7 89.6 66.3 32/32/3232/32/32 96 88.9 58.6 76.4 62.3 61.3 67.7 89.8 66.9 64/64/6464/64/64 192 89.5 60.6 77.6 62.6 61.4 68.6 90.2 67.3 128/128/128128/128/128 384 90.4 61.6 78.5 63.1 61.1 69.5 87.9 67.4 256/256/256256/256/256 768 90.6 61.7 78.7 63.2 61.1 66.9 88.5 66.7 Most gains appear by 384 vectors: the 192-vector model nearly matches the main configuration, while 768 vectors add little to ViDoRe and lower the MMEB average. We therefore select 128/128/128128/128/128 as the operating point balancing capacity and storage. These budget variants are separately trained, unlike the nested prefixes in Table 4.