Paper deep dive

A Strong Baseline for Evaluating Vision Encoders in Multimodal Large Language Models

Yilin Yang, Jun-Tao Tang, Kengyi Wang, Siyuan Su, Gaoyong Luo, Mingda Chen

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 109

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Evaluating vision encoders requires metrics that reliably predict their downstream performance in multimodal large language models (MLLMs). Although recent studies have shown that cross-modal metrics can better capture such performance, unimodal metrics remain the dominant choice in practice. In this work, we revisit cross-modal evaluation of vision encoders through large-scale experiments. We identify important limitations in both the experimental design and methodological formulation of prior approaches. After addressing these limitations and introducing simple improvements, we propose RAVEL, a training-free method based on cross-modal nearest-neighbor retrieval. Despite its simplicity, RAVEL achieves state-of-the-art performance across our experiments, outperforming prior methods by a substantial margin. Our results demonstrate that simple cross-modal metrics, when evaluated under a careful and comprehensive setup, can provide a strong basis for evaluating vision encoders for MLLMs.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

106,434 characters extracted from source content.

Expand or collapse full text

A STRONG BASELINE FOR EVALUATING VISION EN- CODERS IN MULTIMODAL LARGE LANGUAGE MOD- ELS Yilin Yang ∗ 1 , Jun-Tao Tang ∗ 2 , Kengyi Wang 3 , Siyuan Su 3 , Gaoyong Luo 4 , Mingda Chen † 1 1 School of Artificial Intelligence, Shanghai Jiao Tong University 2 Nanjing University; 3 Fudan University; 4 Independent Researcher yilin.yang,mingdachen@sjtu.edu.cn; juntao.tang@smail.nju.edu.cn; 24300980033,24300810015@m.fudan.edu.cn; 210995214@mail.dhu.edu.cn ABSTRACT Evaluating vision encoders requires metrics that reliably predict their downstream performance in multimodal large language models (MLLMs). Although recent studies have shown that cross-modal metrics can better capture such performance, unimodal metrics remain the dominant choice in practice. In this work, we re- visit cross-modal evaluation of vision encoders through large-scale experiments. We identify important limitations in both the experimental design and method- ological formulation of prior approaches. After addressing these limitations and introducing simple improvements, we propose RAVEL, 1 a training-free method based on cross-modal nearest-neighbor retrieval. Despite its simplicity, RAVEL achieves state-of-the-art performance across our experiments, outperforming prior methods by a substantial margin. Our results demonstrate that simple cross-modal metrics, when evaluated under a careful and comprehensive setup, can provide a strong basis for evaluating vision encoders for MLLMs. 2 1INTRODUCTION Recent multimodal large language models (MLLMs) have achieved strong performance in visual understanding and reasoning by using vision encoders to transform visual inputs into representa- tions that can be processed by large language models (Alayrac et al., 2022; Liu et al., 2023, inter alia). The vision encoder 3 plays a critical role in this process: it determines what visual information is preserved and made available to the language model, and therefore substantially affect down- stream MLLM performance. Reliably evaluating vision encoders and predicting their downstream performance in MLLMs is thus an important problem. Current studies commonly evaluate vision encoders by integrating each candidate into an MLLM, training it with a unified multimodal recipe, and comparing the resulting models on downstream benchmarks (Tong et al., 2024; Cocchi et al., 2025; Li et al., 2025). While this directly measures downstream utility, it requires a separate training and evaluation cycle for every encoder-language- model pair, making it prohibitively expensive as the candidate pool grows. Alternative approaches that seek to reduce computational cost instead rely on metrics that measure the quality of vision encoders in isolation, such as parameter count, zero-shot accuracy, or intrinsic representation statis- tics (Radford et al., 2021; Li et al., 2024; Garrido et al., 2023). However, these metrics do not account for how well a vision encoder is aligned or compatible with a particular language model. This motivates cross-modal metrics that directly compare visual and language representations with- out requiring expensive multimodal training or finetuning. * Equal contribution. † Corresponding author. 1 Retrieval-based Assessment of Vision Encoders for Language Models 2 The code and corresponding checkpoints are available at https://github.com/JuntaoTang/ MLLM-VisionEncoder-Eval. 3 We use vision encoder as a general term for both continuous encoders and discrete visual tokenizers. 1 arXiv:2610.05413v1 [cs.CV] 4 Oct 2026 220405470 Number of Vision Encoders −1.0 −0.5 0.0 0.5 1.0 Spearman UNIMODAL KNN Linear probing CROSS-MODAL Huh et al. (2024) Li et al. (2026) RAVEL Figure 1: Effect of vision encoder candidate-pool size and heterogeneity on evaluation metrics. Cross-modal metrics outperform unimodal alternatives on moderately sized candidate pools, but their predictive performance drops substantially as more diverse vision encoders are added. In contrast, RAVEL, with improved representation conditioning and finer-grained similarity scoring, consistently outperforms competing metrics across different vision encoder pools. Recent work approaches this problem by measuring similarities between visual and textual repre- sentation spaces. Notably, Huh et al. (2024) propose MutualNN, which retrieves the top-k nearest neighbors independently in the visual and textual spaces for each paired sample and measures the overlap between their identities. Building on this idea, Li et al. (2026) propose using Gromov– Wasserstein (GW) distance to capture more global structural similarity between the two representa- tion distributions. While these cross-modal metrics are effective, our large-scale experiments reveal that their performance is highly sensitive to the composition of the candidate pool. Figure 1 reports the Spearman correlation between each metric’s predictions and the ground-truth downstream per- formance of MLLMs as progressively more vision encoders are added to the candidate pool. Prior cross-modal metrics outperform unimodal alternatives on moderately sized pools, but their corre- lations drop substantially as the pool becomes larger and more diverse. In these more challenging settings, unimodal metrics again become stronger predictors of downstream MLLM performance. This sensitivity to the candidate pool may help explain why unimodal metrics remain widely used despite the promise of cross-modal evaluation. To address this challenge, we revisit two fundamental aspects of representational similarity measure- ment. First, vector similarities can be strongly influenced by representation geometry. Features may be dominated by a few high-variance directions or exhibit strong correlations across dimensions, which can distort similarity measurements without reflecting downstream compatibility with the lan- guage model (Garrido et al., 2023; Sasaki et al., 2023). We therefore build on PCA whitening, but modify the standard formulation with a stabilization term that prevents low-variance directions from being excessively amplified. Second, the granularity at which similarity is measured matters. Glob- ally pooling visual features can discard fine-grained information contained in patch representations. We therefore introduce fine-grained patch-level similarity scoring into vision encoder evaluation, which, to our knowledge, has not been explored in prior work on this problem. Motivated by these observations, we build upon MutualNN because it provides a simple and effective framework for measuring cross-modal similarity. We propose RAVEL, a training-free cross-modal metric that retains its retrieval-based formulation while incorporating stabilized PCA whitening and fine-grained patch-level scoring. We conduct a large-scale evaluation of nine strong metrics, covering both unimodal and cross-modal approaches, on 210 MLLMs comprising 70 vision encoders and three language models. This eval- uation is substantially larger and more diverse than those in prior studies, allowing us to assess metric robustness across encoder architectures, pretraining objectives, representation types, and computational scales. Under FLOPs-controlled comparisons, RAVEL improves over MutualNN by more than 42.8% and outperforms the strongest prior metric by more than 21.1% in correla- tion with ground-truth downstream MLLM performance. Further analysis shows that stabilized PCA whitening is critical to this improvement, while RAVEL remains robust across a wide range of hyperparameter settings and across different vision encoder families. RAVEL also outperforms TokBench (Wu et al., 2025a), a reconstruction-based evaluation benchmark specifically designed 2 for reconstruction-trained vision encoders, demonstrating that its effectiveness extends beyond any single encoder family or training paradigm. Our contributions can be summarized as follows. • We benchmark nine evaluation metrics on 210 MLLMs built from 70 vision encoders and three language models, revealing that existing cross-modal metrics degrade as encoder di- versity increases. We will open-source our evaluation pipeline and model checkpoints to support future research. • We introduce RAVEL, a simple training-free metric that combines stabilized PCA whiten- ing with fine-grained patch-level similarity scoring. • Despite its simplicity, RAVEL consistently outperforms prior methods and provides a strong baseline for future work on vision encoder evaluation. 2RELATED WORK Vision Encoders.Vision encoders can be broadly divided into continuous and discrete en- coders. Continuous encoders produce real-valued global or patch-level representations and are commonly pretrained through vision–language learning (Radford et al., 2021), image-only self- supervision (Oquab et al., 2024), reconstruction (He et al., 2022), or knowledge distillation (Zhu et al., 2026). Discrete encoders quantize latent features into discrete codes using explicit code- books (van den Oord et al., 2017), lookup-free quantization, or binary quantization, yielding se- quences of visual tokens. While often trained primarily for reconstruction, recent methods (Ma et al., 2025; Lin et al., 2025) also incorporate image–text contrastive objectives to preserve high- level semantics. Since both types determine what visual information is available to the language model, we evaluate them within a unified framework. Vision Encoder Evaluation and Selection. Prior work has explored cheaper alternatives to fully training an MLLM for every candidate encoder. Reconstruction-based benchmarks such as Tok- Bench (Wu et al., 2025a) measure how well reconstructed images preserve fine-grained content. However, such methods require a reconstruction pathway and are therefore limited to encoders trained with reconstruction objectives. Representation-based approaches apply more broadly. Yang et al. (2025b) predict downstream MLLM performance from cross-modal alignment and visual correspondence, but require projec- tor training, keypoint annotations, and downstream scores from a subset of fully trained MLLMs. Zhang et al. (2025) instead train a lightweight linear alignment layer between frozen vision and language backbones and evaluate it through zero-shot image–text retrieval. Training-free methods directly compare visual and textual representation spaces. MutualNN (Huh et al., 2024) constructs k-nearest-neighbor sets independently in the two spaces and measures their overlap over paired samples. More recently, Li et al. (2026) use Gromov–Wasserstein (GW) distance to compare their global geometry. MutualNN therefore emphasizes correspondence-aware local structure, whereas GW captures global structural similarity without directly using the known image– text pairings. 3RAVEL RAVEL builds on MutualNN. Given a paired image–text dataset, MutualNN first retrieves the top- k nearest neighbors of each sample independently in the visual and textual representation spaces. It then uses the known image–text correspondences to measure the overlap between the two sets of retrieved samples. Intuitively, if a vision encoder is compatible with a target language model, images that are close in the visual representation space should have corresponding texts that are also close in the language model’s representation space. RAVEL retains this retrieval-and-overlap pipeline, but revisits how similarities are measured within the two representation spaces. In particular, we introduce two modifications: stabilized PCA whiten- ing to reduce representation-specific geometric bias, and patch-level scoring functions to preserve fine-grained visual similarities. 3 3.1PRELIMINARY: MUTUALNN LetD =(x i ,t i ) n i=1 denote a dataset of paired images and texts. Given a candidate vision encoder E and a target language model L, MutualNN represents each image and text with a single vector: z v i = E(x i ),z t i = L(t i ).(1) Visual and textual representations may have different dimensions and reside in different feature spaces, so MutualNN does not compare them directly. Instead, it independently computes pairwise similarities within the two representation spaces: s v ij = sim z v i , z v j , s t ij = sim z t i , z t j .(2) For each sample i, the top-k nearest neighbors are then retrieved independently in the visual and textual spaces: N v (i) = arg topk j̸=i s v ij , N t (i) = arg topk j̸=i s t ij .(3) Because x j and t j correspond to the same underlying sample, the retrieved identities can be directly compared across modalities. MutualNN measures their average overlap as S = 1 n n X i=1 |N v (i)∩N t (i)| k .(4) A higher score indicates that the vision encoder and language model induce more similar neighbor- hood structures over the paired dataset. RAVEL retains this scoring procedure, but modifies how the similarities used to construct these neighborhoods are computed. 3.2STABILIZED PCA WHITENING The first modification concerns the geometry of the representation spaces. Vector similarities can be strongly affected by the variance and correlation structure of the underlying features. PCA whitening has long been used to mitigate this problem in retrieval. In image retrieval, J ́ egou & Chum (2012) show that whitening can reduce the influence of common, correlated co-occurrence directions, and related work applies PCA whitening to deep image representations for retrieval (Babenko & Lem- pitsky, 2015). For text, whitening has likewise been used to improve sentence representations and semantic retrieval (Su et al., 2021; Huang et al., 2021). We therefore apply PCA whitening before measuring representation similarity. While whitening suppresses dominant high-variance directions, we find that directly scaling each principal direction by the inverse square root of its variance can excessively amplify low-variance directions, which may primarily capture noise. We therefore introduce a small stabilization term ε = 10 −4 to the PCA eigenvalues and apply the following transformation to a particular encoded text or image vector, z i : (Λ + εI) −1/2 U ⊤ (z i −μ)(5) Where U contains the eigenvectors associated with the positive eigenvalues of the covariance ma- trix, and Λ is the diagonal matrix of those eigenvalues. μ is the mean of the given dataset. The stabilization term limits the amplification of low-variance directions while retaining the ability of PCA whitening to suppress dominant directions. We fit the transformation independently for each candidate vision encoder and for the target language model using only representations encoded fromD. For the visual space, PCA is estimated over individual patch representations. For the tex- tual space, PCA is estimated over mean-pooled textual representations. No downstream labels or MLLM training results are used. 3.3FINE-GRAINED SIMILARITY SCORING The second modification concerns the granularity at which similarity is measured. Both visual and textual representations can be viewed as sets of local features, such as image patches or text tokens, and global pooling may discard useful fine-grained correspondences. However, the appropriate granularity differs across modalities in our setting. Vision encoders typically produce hundreds of patch tokens per image, whereas the paired textual descriptions are much shorter. Unlike text 4 retrieval methods such as ColBERT (Khattab & Zaharia, 2020), which operate on relatively long passages and documents, our textual sequences are short enough that global representations are sufficient, while fine-grained matching is more important for vision. For visual representations, we seek a similarity function that directly compares two sets of patch features. Chamfer-style similarity is well suited to this setting because it is permutation-invariant, naturally handles sets with different numbers of elements, and captures local correspondences with- out requiring a predefined one-to-one alignment. By matching each feature to its most similar coun- terpart, it can preserve strong local agreements that may be diluted by global pooling. Chamfer-style matching has been used in set-based problems such as point-cloud matching and reconstruction (Fan et al., 2017) and fine-grained video similarity (Kordopatis-Zilos et al., 2019). Following the same set-matching principle, we apply it directly to patch representations. We define similarity between two images using a symmetric Chamfer-style scoring function: s v ij = 1 2 " 1 M i M i X a=1 max 1≤b≤M j sim ( e p ia , e p jb ) + 1 M j M j X b=1 max 1≤a≤M i sim ( e p jb , e p ia ) # , (6) where M i and M j denote the numbers of patch tokens in images i and j, and e p ia is the a-th patch representation of image i. The two terms perform matching in opposite directions, producing a symmetric image-level similarity. Special tokens, e.g., CLS tokens, are excluded. For text, let e z t i denote the mean-pooled representation of caption t i , and compute textual similarity as s t ij = sim( e z t i , e z t j ). The visual and textual similarities are used to construct the nearest-neighbor sets in Equation 3, and the final RAVEL score is computed using Equation 4. 4EXPERIMENTS 4.1EXPERIMENTAL SETUPS Overall Pipeline. Our experiments comprise two stages. First, we establish the ground-truth down- stream performance for each vision encoder–language-model pair (E,L). We train and evaluate MLLMs on a collection of downstream benchmarks. Let P(E,L) = [P 1 (E,L),...,P B (E,L)](7) denote its performance across B benchmarks. We use the average benchmark performance P (E,L) = 1 B B X b=1 P b (E,L)(8) as the ground-truth performance of the pair. Repeating this procedure across different vision en- coders and language models gives a collection of tuples (E,L,P ). Second, for each evaluation metric, we compute a score S(E,L) that predicts the compatibility be- tween vision encoder E and language model L with minimal training of the corresponding MLLM. We assess each metric by measuring Pearson and Spearman correlations between its predicted scores S(E,L) and the ground-truth downstream performance P (E,L) across vision encoder–language- model pairs. More details on the evaluation metrics are in Appendix G. When training MLLMs, we continue training from existing pretrained language model checkpoints and perform mid-training on LLaVA-LCS-558K (Liu et al., 2024a), optimizing only the projector while keeping both the vision encoder and language model backbone frozen. We then jointly opti- mize the projector and language model backbone on LLaVA-665K (Liu et al., 2024a). The resulting MLLMs are evaluated on 11 benchmarks spanning visual question answering, multimodal reason- ing, OCR, captioning, and hallucination assessment. The unweighted average across the benchmarks defines the ground-truth performance. Details are provided in Appendix E. Evaluated Models. We evaluate a pool of 70 vision encoders spanning continuous encoders and discrete visual tokenizers. Each vision encoder is evaluated with Qwen3-1.7B (Yang et al., 2025a), Qwen2.5-1.5B-Instruct (Yang et al., 2024), and SmolLM2-1.7B-Instruct (Allal et al., 2025). Further details on the candidate vision encoders are provided in Appendix C. 5 Table 1: FLOPs-controlled comparison of different evaluation metrics. All methods use matched computational budgets except AC Policy † , whose computational cost cannot be reduced to the matched FLOP budget due to its method design. Best and second-best results are shown in bold and underlined, respectively. Qwen3-1.7BQwen2.5-1.5BSmolLM2-1.7B Methodρ↑r ↑ρ↑r ↑ρ↑r ↑ Vision-only metrics Linear probing (Caron et al., 2021)0.4840.4900.5680.5100.5930.597 KNN (Wu et al., 2018) 0.5510.5130.5340.5980.6030.567 Cross-modal metrics requiring additional model training Mid-Training Loss0.3420.3490.4340.4030.2720.300 AC Policy † (Yang et al., 2025b) 0.5060.4450.5730.3910.6320.603 Alignment probing (Zhang et al., 2025)0.5690.6180.6340.6400.6700.711 Cross-modal metrics without requiring additional model training RSA (Kriegeskorte et al., 2008)0.1250.2060.0150.068-0.0700.060 CCA (Hotelling, 1936) 0.1630.2510.2750.3380.2130.341 MutualNN (Huh et al., 2024)0.4390.5430.4680.5700.3610.541 GW (Li et al., 2026)0.2260.2360.5150.3120.5770.472 RAVEL (Ours)0.8200.8160.8970.8960.8330.845 Table 2:Comparing the effect of patch-level scoring functions and PCA whitening. The results are averaged over the three language models. Variant Metrics ρ↑ r ↑ MutualNN0.194 0.240 + PCA Whitening0.782 0.798 + Patch-level w/o PCA0.548 0.510 + Patch-level w/ PCA0.850 0.852 Table 3: Correlations of TokBench metrics and RAVEL with downstream MLLM performance using discrete vi- sion encoders. Metric Qwen3Qwen2.5SmolLM2 ρ↑ r ↑ ρ↑ r ↑ ρ↑ r ↑ T-ACC +0.359 +0.364 −0.308 +0.088 −0.975 −0.650 T-NED +0.359 +0.255 −0.308 −0.016 −0.975 −0.679 F-Sim +0.359 −0.223 −0.308 −0.433 −0.975 −0.700 RAVEL +0.600 +0.743 +0.600 +0.598 +0.400 +0.524 Baselines.We compare RAVEL against vision-only metrics that ignore the target language model, namely linear probing (Caron et al., 2021) and k-nearest-neighbor classification (KNN; Wu et al., 2018), and against training-free cross-modal metrics: RSA (Kriegeskorte et al., 2008), CCA (Hotelling, 1936), GW (Li et al., 2026), and MutualNN (Huh et al., 2024). As stronger but costlier reference points, we also include methods that require additional training or fitting: align- ment probing (Zhang et al., 2025), AC Policy (Yang et al., 2025b), and mid-training loss, i.e., the final projector loss of FLOP-matched first-stage mid-training. Details are in Appendix J.1. 4.2MAIN RESULTS Table 1 compares RAVEL with representative vision encoder evaluation methods. RAVEL achieves the highest correlations across all three language backbones, with Spearman correlations of 0.820, 0.897, and 0.833 and Pearson correlations of 0.816, 0.896, and 0.845 for Qwen3-1.7B, Qwen2.5- 1.5B, and SmolLM2-1.7B, respectively. Compared with the strongest baseline, alignment probing, RAVEL improves Spearman correlation by 0.251, 0.263, and 0.163, and Pearson correlation by 0.198, 0.256, and 0.134. These consistent gains show that RAVEL more accurately predicts both the relative ranking of candidate encoders and how their downstream MLLM performance varies. 4.3EMPIRICAL ANALYSIS OF RAVEL 4.3.1EFFECT OF PATCH-LEVEL SCORING FUNCTION AND PCA WHITENING Table 2 reports ablation results averaged over the three language backbones. Starting from cross- modal nearest-neighbor retrieval (k = 100) with globally pooled visual representations, stabilized PCA whitening improves Spearman’s ρ from 0.194 to 0.782 and Pearson’s r from 0.240 to 0.798. 6 0-4-8-12-16-20-24 0.800 0.820 0.840 0.860 0.880 0.900 0.920 Hidden Layer Index Spearman Qwen2.5-1.5BQwen3-1.7BSmolLM2-1.7B (a) 75100125150175200225 Text Length (Tokens) 0.00 0.05 0.10 0.15 0.20 0.25 0.30 R@1 Average Pooling Patch-Set (b) 0.000.250.500.751.00 Patch differentiation γ 0.70 0.75 0.80 0.85 0.90 Spearman Qwen2.5-1.5B Qwen3-1.7B SmolLM2-1.7B (c) Figure 2: Feature representation and patch-set matching in RAVEL. (a) Effect of the text feature extraction layer on ranking correlation. (b) Patch-set matching does not benefit long captions. (c) Patch differentiation under controlled feature intervention. RawCen- tered Stand- ardized PCA only ABTT (q=4) PCA whiten- ing 0.0 0.2 0.4 0.6 0.8 1.0 Spearman Qwen2.5 Qwen3 SmolLM2 (a) Post-processing methods 048163264128 Number of removed visual PCs −0.50 −0.25 0.00 0.25 0.50 Spearman change Δρ High-firstLow-first (b) Sensitivity to visual PC removal -0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8 1.0 Spearman Qwen2.5Qwen3SmolLM2 Baseline+ L2+ Stabilizer+ L2 + Stabilizer (c) Ablation of PCA whitening Figure 3: Representation post-processing choices. (a) Five post-processing methods and the unpro- cessed baseline. (b) Spearman change after removing high- or low-variance visual PCs. (c) Ablation of L2 normalization and spectral stabilization under conventional PCA whitening. Replacing global visual similarity with patch-level Chamfer-style scoring further increases the cor- relations to 0.850 and 0.852, respectively, demonstrating the complementary benefits of geometric conditioning and fine-grained visual matching. 4.3.2ANALYSIS OF TEXT REPRESENTATION Textual Embedding Layer Selection. Unlike vision encoders, which are explicitly trained to pro- duce visual representations, language models do not provide a uniquely defined embedding layer. Their hidden states may encode different levels of semantic information across layers, making the choice of text feature extraction layer an additional design decision. We therefore examine how this choice affects RAVEL. As shown in Figure 2a, Spearman correlations vary only slightly across lay- ers. Given this limited sensitivity, we follow the setting of Li et al., 2026 and use the second-to-last hidden layer throughout. Do We Need Patch-Level Scoring Functions for Textual Representations? To test whether fine- grained token matching also benefits textual similarity estimation, we represent each text as a set of content-token embeddings and compute similarity using ColBERT-style MaxSim (Khattab & Zaharia, 2020), which averages the maximum token-wise similarity for each query token. On a same-image caption retrieval task constructed from C3M, mean pooling consistently outperforms this token-level formulation, with the performance gap widening as caption length increases, as shown in Figure 2b. We therefore use mean-pooled token representations for textual similarity. 4.3.3ANALYSIS OF PCA WHITENING Why PCA Whitening? Figure 3a shows that PCA whitening achieves the highest average rank correlation among the evaluated post-processing methods, with details of each method provided in the appendix J. Following prior dimension-removal analyses (Timkey & van Schijndel, 2021), We further examine the effect of explicit component removal by varying the number of leading PCs re- moved by ABTT and trailing PCs removed before whitening. Figure 3b shows that ABTT benefits from removing a few leading PCs, but these gains diminish and eventually reverse as more compo- nents are removed. For PCA whitening, removing low-variance PCs provides no clear improvement 7 (a) Patch correspondences. 0.82 0.06 best-match sim. (b) Per-patch best-match similarity maps. Figure 4: Patch-level matching preserves local structure. (a) Red lines link high-similarity patch pairs. (b) Color shows each patch’s best-match similarity (brighter = stronger). Table 4: Hyperparameter sensitivity. Param.kε w (×10 −4 ) Value501001500.515 ρ0.838 0.850 0.8500.829 0.850 0.846 r0.847 0.852 0.8440.829 0.852 0.848 Table 5: PCA-whitening effect. MethodNative ρ PCA-W ρ∆ρ MutualNN0.4220.749+0.327 GW0.4390.497+0.058 RSA0.0240.708+0.684 CCA0.2170.448+0.231 over the no-removal setting within the tested range. We therefore omit this additional truncation step and retain all positive-eigenvalue directions. Stabilizing PCA Whitening. We further examine the implementation choices required for reli- able whitening. In Figure 3c, the baseline denotes conventional PCA whitening. The results show that scale normalization alone is insufficient: the main improvement comes from stabilizing the small-eigenvalue directions, after which L2 normalization provides a further gain. We therefore use stabilized and normalized PCA whitening in RAVEL. Effects Across Methods. Table 5 further shows that this post-processing benefits all evaluated methods, although the improvement is less pronounced for GW. Unlike the other methods, GW evaluates the complete distance geometry and applies median-ratio scaling to align cross-modal distance scales (Li et al., 2026). While this adjustment corrects differences in overall magnitude, it does not align the shapes of the distance distributions. These results suggest that PCA whitening provides broadly useful representation conditioning, with particularly strong benefits for methods based on local or correlation-based comparisons. 4.3.4EFFECTS OF PATCH-LEVEL MATCHING Global pooling compresses all visual tokens into a single vector, potentially mixing shared ob- jects with unrelated context. Figure 4 visualizes patch correspondences and per-patch best-match scores. The strongest matches concentrate on shared motorcycle regions despite changes in view- point and background, while unmatched regions receive lower scores, suggesting that patch match- ing can emphasize shared local semantics. To test whether this local information contributes to tokenizer ranking, we progressively shrink patch features toward their within-image mean while keeping the remaining pipeline fixed. Figure 2c shows that the Spearman rank correlation across language backbones generally increases under both graph settings as γ is increased from 0 to 1. This intervention provides independent evidence that preserving within-image patch differentiation improves tokenizer-ranking agreement in the tested settings. 4.4ROBUSTNESS AND COMPUTATIONAL EFFICIENCY Hyperparameter Sensitivity. To assess hyperparameter sensitivity, we vary the neighborhood size k, and whitening regularization strength ε w around their default values. As shown in Table 4, macro Spearman and Pearson vary by at most 0.021 and 0.023, respectively, indicating that RAVEL does not rely on precise hyperparameter tuning. Robustness of Test Set Size. We vary the number of paired examples from 500 to 5,000 while keeping the candidate encoder pool fixed. Figure 5a shows that RAVEL maintains a high Spear- 8 GT (oracle)RAVELMutualNNGWRSACCAMid-training lossRetrievalLinear probingAlignment probingkNN 10002000300040005000 0.00 0.25 0.50 0.75 1.00 Test set size Spearman (a) Test-set-size sensitivity. 0.2 0.4 0.6 0.8 10 15 10 16 10 17 FLOPs Spearman (b) Accuracy–efficiency trade-off. Figure 5: Robustness and scaling behavior. z Ground truth: Robert C. Atkins Prediction:  Robert C. Atkins Question: Who wrote this book? ReconstructionOriginal Ground truth: Banksy in New York Prediction:  New York City: The Ultimate Guide to the Big Apple Question: What is the title of this book? Original Reconstruction Example 1: Unreadable reconstruction, correct prediction. Example 2: Readable reconstruction, wrong prediction. Figure 6: Reconstruction quality does not determine MLLM utility. In the first example, the MLLM correctly identifies the author despite the unreadable reconstructed text. In the second, the recon- struction clearly preserves the book title, yet the MLLM produces an incorrect answer. man correlation throughout, with little gain from larger sets; several baselines are more sensitive to sample size. Thus, the smallest tested set already provides a useful ranking signal. Robustness across Computational Budgets. As shown in Figure 5b, RAVEL maintains a Spear- man correlation of approximately 0.84 across a wide range of FLOPs, while competing methods achieve lower correlations despite generally higher costs. 4.5RECONSTRUCTION QUALITY AS A PROXY FOR MLLM PERFORMANCE Our candidate pool includes discrete visual tokenizers, which are commonly evaluated using reconstruction-based metrics such as TokBench (Wu et al., 2025a) metrics (T-ACC, T-NED, and F-Sim). We therefore examine whether these metrics predict their downstream MLLM performance. Does reconstruction Reflect What the MLLM Can See? Reconstruction reflects what a dedi- cated decoder can recover from discrete visual tokens, whereas downstream performance reflects whether the target LLM can use them. To illustrate, we examine an MLLM built with TokLIP-L and Qwen2.5-1.5B-Instruct on two examples from OCR-VQA (Mishra et al., 2019). As shown in Figure 6, the mismatch occurs in both directions: the MLLM correctly identifies “Robert C. Atkins” despite severely distorted reconstructed text, but answers incorrectly even when “Banksy in New York” is clearly reconstructed. Thus, accurate reconstruction is neither necessary nor sufficient for an MLLM to extract textual information from visual tokens. Correlation with Downstream Performance. As shown in Table 3, TokBench metrics do not con- sistently track downstream MLLM performance. Their Spearman correlations range from +0.36 on Qwen3 to−0.97 on SmolLM2, and Pearson correlations are likewise weak or negative. In contrast, RAVEL yields consistently positive correlations across all three LLMs, with Pearson correlations of 0.743, 0.598, and 0.524, respectively. These results suggest that reconstruction quality is an unreli- able criterion for selecting discrete visual tokenizers, while representation-level measures provide a more consistent signal of downstream utility. 5CONCLUSION In this paper, we revisit whether vision encoder performance for a target LLM can be estimated without training the corresponding MLLM. We introduce RAVEL, a training-free method combining 9 PCA whitening with patch-set matching over paired image–text data. Across 210 MLLMs built from 70 vision encoders and three language backbones, RAVEL achieves the strongest correlations with downstream performance. Further analyses show that whitening mitigates geometric bias, while patch-set matching preserves fine-grained relations, making correspondence-aware local retrieval a simple and effective approach to vision encoder selection. 10 AI USE STATEMENT In this work, we use generative AI tools for polishing the language of this paper, including improving grammar, clarity, and readability. We have not used generative AI tools for generating research ideas, develop the methodology, conduct experiments, analyze or interpret results, or formulate scientific claims. We have reviewed all AI-assisted edits. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI. ETHICS STATEMENT This work does not involve human subjects or the collection of new personal data. All experi- ments are conducted using publicly available datasets and pretrained models. We do not foresee ethical concerns specific to the proposed method beyond those generally associated with MLLMs. In particular, the evaluated models and datasets may contain biases or inappropriate content, and the reported results should not be interpreted as guarantees of fairness or safety. We encourage appropriate evaluation before applying the selected models in real-world or high-stakes settings. REPRODUCIBILITY STATEMENT We provide detailed descriptions of the method, experimental protocol, and evaluation procedure to support reproducibility. Section 3 formally defines RAVEL, including PCA-conditioned repre- sentations, patch-set visual relations, and cross-modal neighborhood agreement. Appendix A pro- vides complete pseudocode. Section 4.1 specifies the 70 vision encoders, three language backbones, paired calibration data, feature-extraction layers, and default hyperparameters used in our experi- ments. All 210 MLLMs are trained using the same two-stage LLaVA-style pipeline. Appendix C documents the training datasets, image preprocessing, optimization settings, batch sizes, learning rates, random seeds, decoding configuration, benchmark subsets, and scoring rules. Appendix B reports the com- plete per-benchmark results for all encoder–language-model combinations. Appendix D defines the correlation metrics and FLOP accounting procedure, while Appendices E and F provide the diag- nostic analyses and implementation details for the evaluated baseline. Code, model configurations, evaluation splits, and extracted reference scores will be released upon publication. REFERENCES Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a vi- sual language model for few-shot learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=EbMuimAbPbs. Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Mart ́ ın Bl ́ azquez, Guilherme Penedo, Lewis Tunstall, Andr ́ es Marafioti, Agust ́ ın Piqueres Lajar ́ ın, Hynek Kydl ́ ı ˇ cek, Vaibhav Srivas- tav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Ben Burtenshaw, Cl ́ ementine Fourrier, Haojun Zhao, Hugo Larcher, Mathieu Morlon, Cyril Zakka, Colin Raffel, Leandro von Werra, and Thomas Wolf. SmolLM2: When smol goes big—data-centric training of a fully open small language model. In Conference on Language Modeling, 2025. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 2425–2433, 2015. Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 15619–15629, 2023. 11 Artem Babenko and Victor Lempitsky. Aggregating deep convolutional features for image retrieval. arXiv preprint arXiv:1510.07493, 2015. Ali Furkan Biten, Rub ` en Tito, Andr ́ es Mafla, Llu ́ ıs G ́ omez, Marc ̧al Rusi ̃ nol, Ernest Valveny, C. V. Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 4291–4301, 2019. Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Bangalath, Junke Wang, Marco Monteiro, Hu Xu, Shiyu Dong, Nikhila Ravi, Shang-Wen Li, Piotr Doll ́ ar, and Christoph Feichtenhofer. Perception encoder: The best visual embeddings are not at the output of the network. In Advances in Neural Information Processing Systems, p. 60884–60937, 2025. Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ́ e J ́ egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 9650–9660, 2021. Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174, 2016. Yung-Sung Chuang, Yang Li, Dong Wang, Ching-Feng Yeh, Kehan Lyu, Ramya Raghavendra, James Glass, Lifei Huang, Jason Weston, Luke Zettlemoyer, Xinlei Chen, Zhuang Liu, Sain- ing Xie, Wen-tau Yih, Shang-Wen Li, and Hu Xu. Meta CLIP 2: A worldwide scaling recipe. In Advances in Neural Information Processing Systems, p. 48009–48036, 2025. Federico Cocchi, Nicholas Moratelli, Davide Caffagni, Sara Sarto, Lorenzo Baraldi, Marcella Cor- nia, and Rita Cucchiara. LLaVA-MORE: A comparative study of LLMs and visual backbones for enhanced visual instruction tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, p. 4337–4347, 2025. Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. VLMEvalKit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the ACM International Conference on Multimedia, p. 11198–11201, 2024. David Fan, Shengbang Tong, Jiachen Zhu, Koustuv Sinha, Zhuang Liu, Xinlei Chen, Michael Rab- bat, Nicolas Ballas, Yann LeCun, Amir Bar, and Saining Xie. Scaling language-free visual rep- resentation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 370–382, 2025. Haoqiang Fan, Hao Su, and Leonidas J. Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017. Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann LeCun. RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank. In Proceed- ings of the International Conference on Machine Learning, p. 10929–10974, 2023. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 6904–6913, 2017. Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll ́ ar, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 16000–16009, 2022. Dan Hendrycks and Kevin Gimpel.Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415, 2016. Harold Hotelling. Relations between two sets of variates. Biometrika, 28(3–4):321–377, 1936. 12 Junjie Huang, Duyu Tang, Wanjun Zhong, Shuai Lu, Linjun Shou, Ming Gong, Daxin Jiang, and Nan Duan. WhiteningBERT: An easy unsupervised sentence embedding approach. In Marie- Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, p. 238–244, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021. findings-emnlp.23.URL https://aclanthology.org/2021.findings-emnlp. 23/. Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 6700–6709, 2019. Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: The Platonic repre- sentation hypothesis. In Proceedings of the International Conference on Machine Learning, p. 20617–20642, 2024. Herv ́ e J ́ egou and Ond ˇ rej Chum. Negative evidences and co-occurences in image retrieval: The benefit of pca and whitening. In European conference on computer vision, p. 774–787. Springer, 2012. Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey. A study of BFLOAT16 for deep learning training. arXiv preprint arXiv:1905.12322, 2019. Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descrip- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 3128–3137, 2015. Omar Khattab and Matei Zaharia. ColBERT: Efficient and effective passage search via contextual- ized late interaction over BERT. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 39–48, 2020. Giorgos Kordopatis-Zilos, Symeon Papadopoulos, Ioannis Patras, and Ioannis Kompatsiaris. Visil: Fine-grained spatio-temporal video similarity learning. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), October 2019. Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Proceedings of the International Conference on Machine Learning, p. 3519–3529, 2019. Nikolaus Kriegeskorte, Marieke Mur, and Peter Bandettini. Representational similarity analysis— connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2:4, 2008. Bozhou Li, Hao Liang, Zimo Meng, and Wentao Zhang. Are bigger encoders always better in vision large models? arXiv preprint arXiv:2408.00620, 2024. Muyang Li, Yucheng Liu, Jianbo Ma, Elliot Osborne, Bo Han, and Tongliang Liu. Rethinking model selection in VLM through the lens of Gromov–Wasserstein distance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 17237–17247, 2026. Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala. PyTorch distributed: Experiences on accelerating data parallel training. Proceedings of the VLDB Endowment, 13(12):3005–3018, 2020. Xianhang Li, Yanqing Liu, Haoqin Tu, and Cihang Xie. OpenVision: A fully-open, cost-effective family of advanced vision encoders for multimodal learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 3977–3987, 2025. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, p. 292–305, 2023. 13 Haokun Lin, Teng Wang, Yixiao Ge, Yuying Ge, Zhichao Lu, Ying Wei, Qingfu Zhang, Zhenan Sun, and Ying Shan. TokLIP: Marry visual tokens to CLIP for multimodal comprehension and generation. arXiv preprint arXiv:2505.05422, 2025. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ́ ar, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In Proceedings of the European Conference on Computer Vision, p. 740–755, 2014. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems, p. 34892–34916, 2023. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, p. 26296–26306, 2024a. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge. LLaVA Blog, January 2024b. URL https://llava-vl.github.io/blog/2024-01-30-llava-next/. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. MMBench: Is your multi-modal model an all-around player? In Proceedings of the European Conference on Computer Vision, p. 216–233, 2024c. Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In Inter- national Conference on Learning Representations, 2017. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems, p. 2507– 2521, 2022. Chuofan Ma, Yi Jiang, Junfeng Wu, Jihan Yang, Xin Yu, Zehuan Yuan, Bingyue Peng, and Xiaojuan Qi. UniTok: A unified tokenizer for visual generation and understanding. In Advances in Neural Information Processing Systems, p. 129274–129297, 2025. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A bench- mark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics, p. 2263–2279, 2022. Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. DocVQA: A dataset for VQA on docu- ment images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, p. 2200–2209, 2021. Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. OCR-VQA: Vi- sual question answering by reading text in images. In Proceedings of the International Conference on Document Analysis and Recognition, p. 947–952, 2019. Jiaqi Mu, Suma Bhat, and Pramod Viswanath. All-but-the-top: Simple and effective postprocessing for word representations. arXiv preprint arXiv:1702.01417, 2017. Maxime Oquab, Timoth ́ e Darcet, Th ́ eo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herv ́ e J ́ egou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without super- vision. Transactions on Machine Learning Research, 2024. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Proceedings of the International Conference on Machine Learning, p. 1310–1318, 2013. 14 Wujian Peng, Lingchen Meng, Yuxuan Cai, Xianwei Zhuang, Yuhuan Yang, Rongyao Fang, Chen- fei Wu, Junyang Lin, Zuxuan Wu, and Shuai Bai. Unified multimodal autoregressive modeling with shared context-visual tokenizer is key to unification. In Proceedings of the International Conference on Machine Learning, 2026. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceed- ings of the International Conference on Machine Learning, p. 8748–8763, 2021. Shota Sasaki, Benjamin Heinzerling, Jun Suzuki, and Kentaro Inui. Examining the effect of whiten- ing on static and contextualized word embeddings. Information Processing & Management, 60 (3):103272, 2023. Oriane Sim ́ eoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Micha ̈ el Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang, Timoth ́ e Darcet, Th ́ eo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie, Julien Mairal, Herv ́ e J ́ egou, Patrick Labatut, and Piotr Bojanowski. DINOv3. arXiv preprint arXiv:2508.10104, 2025. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 8317–8326, 2019. Jaskirat Singh, Boyang Zheng, Zongze Wu, Richard Zhang, Eli Shechtman, and Saining Xie. Im- proved baselines with representation autoencoders. arXiv preprint arXiv:2605.18324, 2026. Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. Whitening sentence representations for better semantics and faster retrieval. arXiv preprint arXiv:2103.15316, 2021. William Timkey and Marten van Schijndel. All bark and no bite: Rogue dimensions in transformer language models obscure representational quality. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, p. 4527–4546, 2021. Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs. In Advances in Neural Information Processing Systems, p. 87310–87356, 2024. Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier H ́ enaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. SigLIP 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025. Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. In Advances in Neural Information Processing Systems, p. 6306–6315, 2017. Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(86):2579–2605, 2008. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. CIDEr: Consensus-based image description evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 4566–4575, 2015. Junfeng Wu, Dongliang Luo, Weizhi Zhao, Zhihao Xie, Yuanhao Wang, Junyi Li, Xudong Xie, Yuliang Liu, and Xiang Bai. TokBench: Evaluating your visual tokenizer before visual generation. arXiv preprint arXiv:2505.18142, 2025a. Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, Song Han, and Yao Lu. VILA-U: A unified foundation model integrating visual understanding and generation. In International Conference on Learning Representations, 2025b. 15 Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 3733–3742, 2018. Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying CLIP data. In International Conference on Learning Representations, 2024. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a. Lihe Yang, Shang-Wen Li, Yang Li, Xinjie Lei, Dong Wang, Abdelrahman Mohamed, Saining Xie, Hengshuang Zhao, Kaiming He, and Hu Xu. In pursuit of pixel supervision for visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 31974–31984, 2026. Shijia Yang, Bohan Zhai, Quanzeng You, Jianbo Yuan, Hongxia Yang, and Chenfeng Xu. Law of vision representation in MLLMs. In Conference on Language Modeling, 2025b. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78, 2014. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 9556–9567, 2024. Le Zhang, Qian Yang, and Aishwarya Agrawal. Assessing and learning alignment of unimodal vision and language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 14604–14614, 2025. Chenchen Zhu, Saksham Suri, Cijo Jose, Maxime Oquab, Marc Szafraniec, Wei Wen, Yunyang Xiong, Patrick Labatut, Piotr Bojanowski, Raghuraman Krishnamoorthi, and Vikas Chandra. Ef- ficient universal perception encoder. arXiv preprint arXiv:2603.22387, 2026. 16 AALGORITHMIC DESCRIPTION OF RAVEL Algorithm 1 RAVEL for vision encoder selection. Require: EncodersF = f m M m=1 ; language model g; paired dataD = (I i ,T i ) n i=1 ; neighbor- hood size k; stabilizer ε w Ensure: Ranked encodersF ranked Feature extraction 1: for i = 1,...,n do 2:t i ← 1 L i L i X ℓ=1 h (−2) iℓ , whereh (−2) iℓ L i ℓ=1 are the penultimate-layer text features 3: end for 4: for m = 1,...,M do 5:for i = 1,...,n do 6: V (m) i ←p (m) ia P (m) i a=1 from f m (I i ), excluding special tokens 7:end for 8: end for Independent stabilized whitening 9: Compute (μ t , U t + , Λ t + ) fromt i n i=1 , retaining positive eigenvalues 10: for i = 1,...,n do 11: ˆ t i ← (Λ t + + ε w I) −1/2 (U t + ) ⊤ (t i −μ t ) (Λ t + + ε w I) −1/2 (U t + ) ⊤ (t i −μ t ) 2 12: end for 13: for m = 1,...,M do 14:Compute (μ v,m , U v,m + , Λ v,m + ) from all patches in S i V (m) i 15:for all p (m) ia ∈V (m) i do 16: ˆ p (m) ia ← q (m) ia ∥q (m) ia ∥ 2 , where q (m) ia = (Λ v,m + + ε w I) −1/2 (U v,m + ) ⊤ (p (m) ia −μ v,m ) 17:end for 18: end for Neighborhood construction 19: for i = 1,...,n do 20: N t i ← arg topk j̸=i ˆ t ⊤ i ˆ t j 21: end for 22: for m = 1,...,M do 23:for all 1≤ i < j ≤ n do 24:s v,m ij ← 1 2 " 1 P (m) i X a max b ( ˆ p (m) ia ) ⊤ ˆ p (m) jb + 1 P (m) j X b max a ( ˆ p (m) jb ) ⊤ ˆ p (m) ia # 25:s v,m ji ← s v,m ij 26:end for 27:for i = 1,...,n do 28: N v,m i ← arg topk j̸=i s v,m ij 29:end for 30: end for Encoder scoring 31: for m = 1,...,M do 32: S m ← 1 nk n X i=1 N v,m i ∩N t i 33: end for 34: F ranked ←f m M m=1 ordered by decreasingS m 35: returnF ranked 17 a big airplane that is on a runway A plane lands on a big wide run way A planeisparkedin a airport field A planethat is sittingon a tarmac LLM Visual Space Textual Space Compare Candidate Vision Encoder NearestNeighbor Pooling Figure 7: Overview of RAVEL. Given paired image–text samples, RAVEL applies stabilized PCA whitening, constructs visual and textual neighborhoods using patch-level and global similarity, re- spectively, and measures their agreement through the known image–text correspondences. Table 6: Complete vision encoder pool. All 70 evaluated architecture, checkpoint, and input- resolution configurations are listed and grouped by model family. Family (#)Architectures and input resolutionsReference Language-supervised continuous encoders OpenAI CLIP (1)ViT-L/14 (224)Radford et al., 2021 MetaCLIP (9)ViT-B/16 (400M, 224; 2.5B, 224); ViT-B/32 (400M, 224; 2.5B, 224); ViT-L/14 (400M, 224; 2.5B, 224); ViT-H/14 (2.5B, 224; v1.2, 224); ViT-G/14 (2.5B, 224) Xu et al., 2024 MetaCLIP 2 (15)ViT-B/16 (224, 384); ViT-B/32 (224, 384; mT5, 224); ViT-G/14 (224, 378); ViT-H/14 (378); ViT-L/14 (224); ViT-M/16 (224, 384; mT5, 224); ViT-S/16 (224, 384; mT5, 224) Chuang et al., 2025 SigLIP2 (15)So400m/14 (224, 384); So400m/16 (256, 384, 512); ViT-B/16 (224, 256, 384, 512); ViT-B/32 (256); ViT-G/16 (256, 384); ViT-L/16 (256, 384, 512) Tschannen et al., 2025 Perception Encoder (3)PE-Core-B/16 (224); PE-Core-G/14 (448); PE-Lang-L/14 (448)Bolya et al., 2025 Image-only self-supervised continuous encoders DINO (4)ViT-S/8; ViT-S/16; ViT-B/8; ViT-B/16Caron et al., 2021 DINOv2 (4)ViT-S/14; ViT-B/14; ViT-L/14; ViT-G/14Oquab et al., 2024 DINOv3 (1)ViT-L/16Sim ́ eoni et al., 2025 I-JEPA (1)ViT-H/14Assran et al., 2023 Web-SSL DINO (1)1B (224)Fan et al., 2025 Reconstruction-oriented continuous encoders Web-SSL MAE (3)300M (224); 1B (224); 3B (224)Fan et al., 2025 Pixio (3)ViT-B/16; ViT-L/16; ViT-H/16Yang et al., 2026 RAEv2 (1)DINOv3-LSingh et al., 2026 Distillation-based continuous encoders EUPE (4)ViT-T; ViT-S; ViT-B; ConvNeXt-BZhu et al., 2026 Discrete and hybrid visual tokenizers TokLIP (2)TokLIP-S (256); TokLIP-L (384)Lin et al., 2025 UniTok (1)UniTok (256)Ma et al., 2025 VILA-U (1)VILA-U (256)Wu et al., 2025b UniAR (1)UniAR-BSQPeng et al., 2026 BOVERVIEW OF RAVEL Figure 7 summarizes the overall pipeline. Given paired image–text samples, RAVEL extracts visual patch features from each candidate encoder and text features from the target LLM, then whitens each representation space independently. It constructs textual neighborhoods using cosine similarity and visual neighborhoods using symmetric patch-level matching. Each encoder is scored by the average overlap between these neighborhoods under the known image–text correspondences. The procedure uses only frozen features and requires no downstream labels. CCANDIDATE VISION ENCODERS Our candidate pool contains 70 vision encoder configurations spanning continuous encoders and dis- crete or hybrid visual tokenizers. Table 6 lists all evaluated architecture, checkpoint, and resolution variants, treating each resolution of an architecture as a separate configuration. DDETAILED DOWNSTREAM MLLM RESULTS Table 7 reports the complete downstream results of 70 vision encoders paired with three language backbones, yielding 210 MLLMs. Each row corresponds to one encoder–backbone pair and includes 18 performance on eleven benchmarks spanning visual question answering, reasoning, document and chart understanding, object hallucination, and image captioning. AVG, the unweighted mean across these benchmarks, defines the ground-truth performance and global ranking. Language-supervised encoders, particularly models from the SigLIP2 and MetaCLIP families, con- sistently occupy the top ranks. However, the optimal choice remains strongly dependent on the lan- guage backbone. Specifically, the best-performing configurations are SigLIP2 ViT-G/16 at 384 pix- els for Qwen3 (54.56), SigLIP2 So400m/16 at 512 pixels for Qwen2.5 (54.97), and MetaCLIP 2 ViT-G/14 at 378 pixels for SmolLM2 (45.93). These differences confirm that no single vision en- coder is uniformly optimal across language backbones. Table 7: Detailed downstream MLLM results for all 70 vision encoders and three language back- bones. The 210 encoder–language-backbone pairs are sorted globally by AVG in descending order. The labels Q3, Q2.5, and SMOL denote Qwen3-1.7B, Qwen2.5-1.5B-Instruct, and SmolLM2-1.7B- Instruct, respectively. The background colors identify the language backbone. LLMRankVision Encoder M MUMMB VQA v2SQAChartDocTextPOPEGQACOCOFlickrAvg Q2.51SigLIP2 So400m/16 (512)36.1064.8667.5265.5014.1026.2247.7984.9155.4985.1457.0154.97 Q32SigLIP2 ViT-G/16 (384)32.4061.3967.7268.4014.4025.2248.8891.8254.4781.9753.5354.56 Q2.53SigLIP2 So400m/14 (384)35.5066.0267.8965.9014.1024.5946.6189.9852.6482.3453.8254.49 Q2.54SigLIP2 ViT-L/16 (512)35.0064.0966.6367.1013.2025.0745.4591.3955.0880.5955.5354.47 Q2.55SigLIP2 ViT-G/16 (384)34.1070.2768.0766.8015.1024.6250.3384.6553.5677.9251.3754.25 Q36SigLIP2 So400m/14 (384)33.3062.9366.3969.2014.2025.6247.3391.6152.0378.4053.6054.06 Q2.57SigLIP2 ViT-G/16 (256)34.6066.8068.0866.8012.4015.9239.5191.2153.8685.8957.4553.87 Q2.58SigLIP2 So400m/16 (384)37.1064.4866.2564.0012.9022.4945.2691.5853.2572.4654.5353.12 Q2.59SigLIP2 So400m/14 (224)35.7066.0266.8766.4012.7015.7337.5589.4152.2484.1056.5153.02 Q2.510SigLIP2 ViT-L/16 (384)35.2064.0964.2166.0013.9018.8941.4589.7352.6484.1552.8453.01 Q311SigLIP2 ViT-L/16 (384)32.7063.7165.1466.3014.7021.4545.1790.1453.1578.0847.6052.56 Q2.512MetaCLIP 2 ViT-G/14 (378)38.1066.4165.0465.0013.6020.6239.1488.8252.9576.5551.9752.56 Q313SigLIP2 So400m/16 (512)32.8062.1664.1668.2014.0026.2247.0287.6150.2075.7749.4452.51 Q2.514PE-Core-G/14 (448)34.6066.0262.3464.3014.0021.1239.9789.5549.4977.0052.3251.88 Q315SigLIP2 So400m/16 (384)33.2062.5564.7667.5013.1023.4847.7786.5954.0770.6445.6151.75 Q2.516SigLIP2 So400m/16 (256)36.7064.0966.6962.5012.4017.5437.8391.6052.5474.1749.3951.40 Q2.517MetaCLIP ViT-H/14 (v1.2, 224)37.3066.4164.5665.0011.1014.7234.0583.5952.9582.6650.4251.16 Q318SigLIP2 ViT-G/16 (256)33.5067.1864.6265.3011.6016.5939.1887.8852.7476.1646.4551.02 Q2.519MetaCLIP 2 ViT-H/14 (378)35.0065.6462.4063.6012.8021.1337.2282.6051.2277.4050.7350.89 Q2.520MetaCLIP 2 ViT-G/14 (224)36.2064.8663.9663.5011.5015.4434.3589.5451.9376.7550.4550.77 Q2.521SigLIP2 ViT-L/16 (256)34.5063.7164.1265.4012.3015.4438.1989.3251.3275.6048.4450.76 Q2.522SigLIP2 ViT-B/16 (384)33.5064.8663.9064.8012.3017.6534.2589.2651.3275.7947.9150.50 Q2.523SigLIP2 ViT-B/16 (512)34.7063.3262.7864.4013.0020.6735.9088.6751.2272.3144.8450.16 Q2.524MetaCLIP ViT-G/14 (2.5B, 224)35.0069.1163.5665.4011.5014.8431.9088.5250.7169.5748.4049.86 Q2.525OpenAI CLIP ViT-L/14 (224)33.6066.4161.1764.6011.8014.1732.5689.1851.3274.1148.4249.76 Q2.526MetaCLIP 2 ViT-L/14 (224)34.5067.1860.8463.2012.5014.7532.1188.6751.3271.1547.8849.46 Q327SigLIP2 So400m/16 (256)32.0059.8564.6467.7012.3016.8439.1390.4251.9364.3541.1649.12 Q328SigLIP2 So400m/14 (224)33.2061.7864.7065.8012.9014.8836.6990.0453.2565.9141.1249.12 Q329SigLIP2 ViT-L/16 (256)33.3061.0062.4566.9011.3014.3935.9683.0852.5474.3044.8149.09 Q2.530MetaCLIP ViT-H/14 (2.5B, 224)33.1062.5561.0864.6010.8015.2628.2388.6551.2274.2348.4048.92 Q331MetaCLIP 2 ViT-G/14 (224)32.7063.7162.1664.3011.7014.4332.5585.9351.2273.0845.4948.84 Q2.532MetaCLIP ViT-L/14 (2.5B, 224)36.9066.8061.3463.5012.2013.0729.3881.0451.7372.7347.8748.78 Q333MetaCLIP ViT-H/14 (v1.2, 224)34.1061.7863.8163.9010.9013.8031.4887.8749.7072.1745.8848.67 Q334PE-Core-G/14 (448)34.1062.9359.8866.1014.2016.3234.1279.8348.5872.5545.7648.58 Q335SigLIP2 ViT-B/16 (512)32.6060.2360.4161.7013.7018.7438.1386.3151.5267.1443.1248.51 Q336OpenAI CLIP ViT-L/14 (224)33.7062.5561.1664.6012.4014.7332.8585.2149.5968.6140.7147.83 Q2.537SigLIP2 ViT-B/16 (224)32.7064.8663.2765.1011.2011.8326.9678.8753.0573.9444.3747.83 Q338MetaCLIP ViT-G/14 (2.5B, 224)34.6061.3961.2464.3012.4013.8531.2084.2150.1068.0943.3347.70 Q2.539SigLIP2 ViT-B/16 (256)33.7064.8662.5364.7012.0011.8929.1682.0051.6368.4242.8947.62 Q340MetaCLIP 2 ViT-G/14 (378)33.1060.6259.7063.2011.9015.5931.8384.4348.9867.1341.3647.08 Q2.541MetaCLIP ViT-L/14 (400M, 224)33.8062.1661.1662.6011.4013.7228.1679.7150.3070.9443.9047.08 Q342SigLIP2 ViT-B/16 (256)33.8061.3961.1266.9011.1013.7728.4289.7350.0063.7736.0146.91 Q2.543MetaCLIP 2 ViT-B/16 (384)33.3061.0059.2263.6011.7014.3130.9382.3951.0267.5140.4046.85 Q344SigLIP2 ViT-B/16 (224)31.5058.3059.7165.2012.0013.3828.2489.3151.8364.2338.9546.60 Q345MetaCLIP ViT-H/14 (2.5B, 224)35.4058.6959.5363.6012.0013.1328.9886.7950.3064.0237.9446.40 Q346MetaCLIP ViT-L/14 (2.5B, 224)34.1060.2360.0965.8012.6011.3729.3285.4747.0563.0437.8546.08 Q2.547Pixio ViT-L/1633.8057.5358.3262.6011.308.2611.2390.0853.9674.5643.7645.95 Smol48MetaCLIP 2 ViT-G/14 (378)31.1057.5361.9760.4013.7017.3938.0884.4148.8856.7735.0445.93 Q2.549PE-Core-B/16 (224)34.3060.6260.9261.7011.6010.9123.6383.1351.5265.4441.2545.91 Q350MetaCLIP ViT-L/14 (400M, 224)32.7057.5358.4365.6012.0011.3527.0785.8148.5867.6338.0445.89 Smol51SigLIP2 ViT-L/16 (384)27.4057.1459.6255.1011.8016.9239.3280.7249.8061.7444.8545.86 Q2.552PE-Lang-L/14 (448)32.5057.5359.6961.0012.709.8022.3283.5950.9168.5543.2645.62 Smol53SigLIP2 So400m/14 (384)28.7056.7659.9551.0013.2018.7842.0384.8450.6155.2640.1245.57 Q2.554MetaCLIP 2 ViT-B/16 (224)35.3059.8559.4360.5011.1011.0623.8185.8547.5666.6440.0745.56 Q355PE-Core-B/16 (224)33.1058.3058.4764.8011.5011.4724.9387.5749.2963.9437.2045.51 Q2.556SigLIP2 ViT-B/32 (256)34.7062.9359.3865.4011.309.0317.9283.4649.1965.0339.1645.23 Q2.557MetaCLIP 2 ViT-M/16 (384)35.1058.6957.7761.7011.4014.5326.8284.1147.9761.2633.3044.79 Q2.558MetaCLIP ViT-B/16 (400M, 224)34.6057.1458.2063.9011.1011.6922.6186.8349.2962.8234.4544.78 Q2.559MetaCLIP ViT-B/16 (2.5B, 224)33.1062.5559.8562.8010.8011.8425.0778.9650.3061.1335.7544.74 Q2.560DINOv2 ViT-G/1434.2055.9859.0860.9010.907.8110.7488.8150.6168.5041.0844.42 Q361MetaCLIP 2 ViT-B/16 (384)32.7055.9858.0662.8011.8014.5528.8677.9747.2663.4433.8044.29 Continued on next page 19 Table 7 – continued from previous page LLMRankVision Encoder M MU MMB VQA v2 SQAChartDocTextPOPEGQACOCOFlickrAvg Q2.562Web-SSL DINO 1B (224)32.1058.3058.1762.3010.508.2611.9281.5249.1973.0741.5544.26 Smol63SigLIP2 So400m/16 (384)26.9057.5360.7854.3013.3017.4040.3666.3646.9563.3337.6844.08 Q364MetaCLIP ViT-B/16 (400M, 224)34.0057.5357.9165.1012.0011.2123.7685.1348.7858.0931.0444.05 Q2.565EUPE ConvNeXt-B34.0059.4656.9662.3010.508.7516.7680.1749.2966.2640.0144.04 Q2.566TokLIP-L (384)35.3059.0757.2664.2010.808.8712.2672.9950.0071.1941.9343.99 Q367MetaCLIP ViT-B/16 (2.5B, 224)33.2054.8355.8662.8011.5010.7123.1382.1449.3963.6135.6843.90 Smol68SigLIP2 ViT-G/16 (256)29.5058.6957.5256.0012.0014.5237.8765.3146.0460.5243.3343.75 Q369MetaCLIP 2 ViT-B/16 (224)33.8057.1457.0262.3010.6011.4226.2383.4747.4658.5932.0843.65 Q370Web-SSL DINO 1B (224)32.7056.3757.6061.9010.407.3812.1988.1949.3968.1435.8743.65 Q371MetaCLIP 2 ViT-B/32 (384)32.9055.2155.0464.6011.2010.4420.3584.2747.5662.2833.2443.37 Q2.572MetaCLIP 2 ViT-M/16 (224)34.6058.3058.0364.2010.7010.6519.6679.6946.6562.7031.5343.34 Q2.573MetaCLIP 2 ViT-M/16 (mT5, 224)34.0055.9857.5462.0011.0011.5620.8482.3347.4660.3232.7143.25 Smol74MetaCLIP ViT-G/14 (2.5B, 224)28.3059.4654.7059.5011.4011.0327.5667.9143.0968.6443.5443.19 Q2.575DINOv2 ViT-L/1433.1054.8355.6160.9010.807.9310.7789.8549.8065.5734.7643.08 Q2.576MetaCLIP 2 ViT-B/32 (384)33.9057.5357.4262.7010.7011.5720.9776.2646.3461.3735.0243.07 Smol77SigLIP2 ViT-G/16 (384)27.4056.3758.9957.7013.6018.2443.1864.2846.1454.2332.5142.97 Q378MetaCLIP 2 ViT-M/16 (384)33.0054.4456.7865.5011.9012.1624.3971.7847.6661.7032.9242.93 Smol79SigLIP2 So400m/14 (224)30.0057.9259.6054.1010.9014.6735.4066.6748.9857.4935.2342.81 Q380SigLIP2 ViT-B/16 (384)32.2054.4452.5165.0011.8010.0921.3074.8045.9364.3435.4042.53 Q2.581UniTok (256)33.8056.7656.8062.3011.008.4412.3877.8948.8864.4434.8542.50 Q382SigLIP2 ViT-B/32 (256)33.0057.1457.6264.2010.708.9118.4681.6547.6658.0430.0842.50 Smol83PE-Lang-L/14 (448)26.3049.4256.5854.0013.2016.9036.9168.1347.7662.2335.8542.48 Smol84PE-Core-G/14 (448)27.4054.4455.2656.3013.3014.6434.1981.3545.3349.8534.4442.41 Smol85SigLIP2 So400m/16 (512)29.8057.9258.9753.1013.6019.2840.8966.2847.2647.2731.7242.37 Smol86SigLIP2 ViT-L/16 (256)27.7061.0057.2353.4010.9014.6033.0867.7147.4658.2634.5642.35 Smol87SigLIP2 ViT-B/16 (384)29.4050.9754.9454.8012.1010.2431.1169.3547.3661.3042.2742.17 Smol88SigLIP2 So400m/16 (256)29.1056.3757.7454.3012.0014.1434.2366.1246.7555.6634.9541.94 Q2.589MetaCLIP 2 ViT-B/32 (mT5, 224)33.5055.2155.8860.5010.609.8314.5073.3249.3961.6533.4841.62 Q2.590MetaCLIP 2 ViT-S/16 (384)35.2056.7655.3960.1012.0012.1921.9872.3846.3455.9429.0841.58 Q2.591EUPE ViT-B33.6055.9853.5362.6011.009.0111.3577.6147.0561.7033.2641.52 Q392UniTok (256)33.5050.5856.0165.1012.509.0016.8480.4847.1553.6631.4241.48 Smol93MetaCLIP 2 ViT-L/14 (224)28.3052.9054.7954.0011.409.0128.9066.1242.5867.0640.2641.39 Q2.594MetaCLIP 2 ViT-B/32 (224)33.7056.3755.5062.1011.309.0516.1970.1247.7661.0831.7041.35 Q2.595MetaCLIP 2 ViT-S/16 (mT5, 224)33.7058.6955.0761.1010.809.4816.9073.8246.9558.7429.6141.35 Q2.596DINOv2 ViT-B/1435.3050.9755.4261.6011.108.1210.7474.7046.7566.2933.4241.31 Q2.597EUPE ViT-S33.5055.6055.2462.9011.108.2611.1872.2948.0762.1731.8641.11 Q2.598MetaCLIP 2 ViT-S/16 (224)34.7056.7654.2759.7010.9010.2017.6876.9048.7854.0627.6141.05 Q2.599Pixio ViT-B/1632.4052.9053.8463.1010.708.6611.0777.4846.7561.0533.0341.00 Q3100MetaCLIP 2 ViT-S/16 (384)32.3050.5854.9561.9011.6010.7623.2183.2745.3349.6026.5540.91 Smol101MetaCLIP 2 ViT-G/14 (224)28.8055.6056.7455.4010.408.4821.8772.3244.0058.8237.4240.90 Q2.5102UniAR-BSQ35.5049.8153.2065.1010.208.4310.9782.8446.1455.3232.1040.87 Q2.5103TokLIP-S (256)35.3050.9752.1364.2010.508.1410.6776.8247.2661.0531.3440.76 Smol104SigLIP2 ViT-B/16 (512)28.3050.5855.3451.2012.2014.8431.3068.4845.1253.3737.1640.72 Smol105MetaCLIP ViT-L/14 (2.5B, 224)29.1055.2156.3454.6011.0011.9626.5270.5142.4853.9535.8140.68 Q2.5106VILA-U (256)34.3052.1255.1164.5010.408.5311.1079.9446.2455.2328.3740.53 Q3107MetaCLIP 2 ViT-H/14 (378)31.6052.1250.8262.5010.508.4413.4783.9943.3956.9731.8740.52 Smol108OpenAI CLIP ViT-L/14 (224)29.0057.1453.8356.5012.0010.9924.9966.5943.8056.6034.1940.51 Q3109DINOv2 ViT-G/1432.5052.5157.8861.6010.507.639.9787.4647.8750.4426.9140.48 Q3110MetaCLIP 2 ViT-L/14 (224)33.4047.8850.3163.709.708.7012.0182.9047.1557.4131.6140.43 Q3111MetaCLIP 2 ViT-M/16 (mT5, 224)31.2047.8852.6662.6010.909.5918.6883.7947.8753.4126.0340.42 Smol112MetaCLIP ViT-H/14 (2.5B, 224)28.7053.2857.8755.4011.9012.6826.3165.3145.4354.0731.9540.26 Smol113Web-SSL DINO 1B (224)28.9047.8855.5553.0012.207.7110.8173.9148.2765.8435.6339.97 Q3114PE-Lang-L/14 (448)30.4051.3552.2559.2010.107.0112.4979.2446.6555.8934.3239.90 Q3115Pixio ViT-L/1630.5050.1950.3164.109.807.208.7585.3546.5456.1129.1239.82 Q3116MetaCLIP 2 ViT-S/16 (224)32.0056.7652.1963.4010.7010.0716.9371.0146.3451.9026.2239.77 Q3117DINOv3 ViT-L/1633.3046.7248.3762.4010.207.7410.2283.0842.6858.5130.7939.46 Q3118VILA-U (256)31.5048.2652.1665.6011.107.6911.4583.8445.0252.1625.1439.45 Smol119MetaCLIP ViT-H/14 (v1.2, 224)30.1054.4457.0555.0011.5011.3327.6373.0344.8242.9325.8339.42 Q3120Pixio ViT-B/1631.3047.4950.2364.7010.108.529.6980.9444.1156.8129.6739.41 Smol121SigLIP2 ViT-B/16 (256)28.5051.3555.8152.909.8011.0527.6866.4348.2751.3428.0639.20 Q3122SigLIP2 ViT-L/16 (512)31.9050.5850.1662.1010.307.8812.0679.0343.2952.1431.6039.19 Q3123MetaCLIP 2 ViT-S/16 (mT5, 224)32.6049.0351.0263.9011.009.4215.7673.8544.2152.3527.1739.12 Q2.5124MetaCLIP ViT-B/32 (2.5B, 224)34.7050.5849.2462.9011.007.8210.7975.3445.9354.8726.2439.04 Q3125TokLIP-S (256)31.4053.2852.4364.9010.907.5511.5385.9745.9341.6323.7539.02 Smol126SigLIP2 ViT-B/16 (224)27.1055.6056.0259.4010.1010.6523.1266.2845.2241.3433.0938.90 Smol127MetaCLIP 2 ViT-B/16 (384)30.0048.2653.6050.0010.8012.1326.2268.4644.7254.1728.4738.80 Q2.5128DINOv2 ViT-S/1432.2045.1750.8661.9010.708.419.9480.9046.7554.4625.2938.78 Q3129MetaCLIP 2 ViT-M/16 (224)32.1049.8153.0865.0011.208.1315.6878.5645.7343.4223.7038.76 Q3130MetaCLIP ViT-B/32 (2.5B, 224)31.6049.8147.8565.4010.608.3013.9178.2041.8750.2824.9338.43 Q3131MetaCLIP 2 ViT-B/32 (224)34.4046.7249.0464.0011.008.3313.1783.2843.3946.0221.3438.24 Q3132TokLIP-L (384)33.1046.7248.9461.7011.507.9311.6880.2944.4150.0924.1438.23 Q3133EUPE ConvNeXt-B32.1043.6347.1762.6010.207.9710.2977.7142.6851.4129.0037.71 Q2.5134MetaCLIP ViT-B/32 (400M, 224)34.7049.0347.6262.7010.207.7910.9571.5543.2949.9826.1037.63 Smol135MetaCLIP 2 ViT-M/16 (384)28.8042.4752.3850.8011.4010.1821.4969.1841.1654.4431.3337.60 Smol136MetaCLIP ViT-L/14 (400M, 224)29.8051.3552.7754.2010.9010.0323.4464.3141.6746.4528.3037.57 Q3137EUPE ViT-S32.8043.2446.2162.8010.607.439.3970.4444.2156.3929.1437.51 Smol138DINOv2 ViT-L/1431.2044.0249.5455.1010.807.939.8477.9742.7854.6128.4637.48 Smol139SigLIP2 ViT-B/32 (256)27.1055.9853.8353.0010.808.8117.1169.1044.4145.0124.2437.22 Smol140MetaCLIP ViT-B/16 (2.5B, 224)28.4049.8149.7754.4010.6010.0819.8466.5141.1650.7526.2537.05 Smol141TokLIP-L (384)27.3052.5151.6656.9011.206.8813.0582.6244.4135.2525.0236.98 Smol142Pixio ViT-H/1629.1042.8648.4653.3010.107.2810.1678.1643.8055.7627.5436.96 Smol143MetaCLIP ViT-B/16 (400M, 224)27.9047.1051.4255.0011.508.3919.1671.4340.6547.5725.7236.89 Q2.5144DINOv3 ViT-L/1632.2044.0250.8461.9010.708.6110.8871.6343.1948.4823.2636.88 Continued on next page 20 Table 7 – continued from previous page LLMRankVision Encoder M MU MMB VQA v2 SQAChartDocTextPOPEGQACOCOFlickrAvg Smol145MetaCLIP 2 ViT-B/16 (224)28.0048.2651.3655.1011.609.6021.1268.3839.7446.9925.5836.88 Smol146EUPE ConvNeXt-B30.6050.9752.3753.109.605.7610.8867.2241.9752.3229.2436.73 Smol147DINOv3 ViT-L/1629.9044.7949.1053.1010.908.0711.8869.6942.8950.9131.2536.59 Q3148I-JEPA ViT-H/1430.8041.7047.0863.2010.808.018.9182.9942.6846.1119.8336.56 Q2.5149I-JEPA ViT-H/1431.7042.0847.9762.0010.107.649.9172.2944.3152.8320.4536.48 Q3150DINOv2 ViT-S/1431.1041.3147.1861.5010.807.759.3275.0444.4148.2223.0636.34 Q3151DINOv2 ViT-L/1430.5038.6147.0262.5010.208.569.1174.4442.8950.1325.0536.27 Q2.5152RAEv2 DINOv3-L33.8048.2648.3361.6011.108.3611.4766.5940.3547.0322.0136.26 Q2.5153Web-SSL MAE 3B (224)33.5044.0249.5562.9010.807.549.3970.4744.0045.0320.2736.13 Q2.5154Pixio ViT-H/1632.8025.1050.0760.7011.108.2710.1465.4943.1958.4431.4436.07 Smol155SigLIP2 ViT-L/16 (512)27.7046.3349.0654.6010.807.8811.5768.3341.5749.9926.2335.82 Smol156PE-Core-B/16 (224)26.1042.8650.3549.7010.408.8019.5873.8545.2241.9224.7035.77 Smol157EUPE ViT-B27.9051.7449.1150.409.608.4812.8564.6141.6748.5928.2035.74 Smol158MetaCLIP 2 ViT-B/32 (384)29.2042.0851.4950.7010.709.0517.9462.5743.5047.6124.6035.40 Q3159MetaCLIP 2 ViT-B/32 (mT5, 224)31.7037.8445.2964.0010.508.269.6072.1138.4147.4924.0635.39 Smol160DINOv2 ViT-G/1429.1044.0250.3052.0010.605.7510.0767.4141.7746.9526.9634.99 Q3161UniAR-BSQ31.4044.7948.2463.5010.907.719.4669.4541.4636.3220.6234.90 Smol162MetaCLIP 2 ViT-H/14 (378)30.3053.2849.5553.2012.208.3319.9346.4742.8944.0823.4334.88 Q2.5163DINO ViT-B/1633.3039.3845.1063.0010.308.149.3975.1443.5040.5515.4934.84 Q3164EUPE ViT-T33.7040.1544.4863.009.307.818.7481.9939.2336.1117.6334.74 Q3165MetaCLIP ViT-B/32 (400M, 224)33.6039.3845.8463.4010.606.8310.4674.3139.5338.1319.4934.69 Q3166EUPE ViT-B31.1037.8445.5961.4010.407.6110.9268.7342.2842.9222.0234.62 Smol167MetaCLIP 2 ViT-B/32 (mT5, 224)28.7046.3349.5054.3010.206.9113.7868.4344.8237.3520.1334.59 Smol168Pixio ViT-L/1626.9040.1549.9854.909.507.309.8969.3740.7545.7125.8634.57 Smol169MetaCLIP 2 ViT-M/16 (mT5, 224)27.4044.4046.9249.0010.808.5218.3165.9637.0947.8523.7134.54 Smol170MetaCLIP 2 ViT-M/16 (224)29.3046.7249.7954.6010.608.5717.1371.0939.9433.9417.5134.47 Smol171TokLIP-S (256)28.3048.2649.2855.0011.306.4810.4666.2839.6341.6622.4934.47 Smol172MetaCLIP ViT-B/32 (2.5B, 224)28.8043.2446.1952.609.808.2912.5468.5138.6244.3121.7534.06 Smol173MetaCLIP 2 ViT-B/32 (224)25.5044.4047.5853.9010.407.4313.2366.7540.4545.2919.6534.05 Q3174DINOv2 ViT-B/1430.5036.6844.8161.2010.607.069.3768.1240.8544.3120.3033.98 Q2.5175Web-SSL MAE 1B (224)33.1037.8445.1261.2010.608.0010.1270.4141.9737.2015.4433.73 Q3176Pixio ViT-H/1629.9029.3443.9861.109.807.5610.0068.4340.9645.5523.7033.67 Q2.5177Web-SSL MAE 300M (224)33.6036.6845.3562.3010.607.717.9471.3441.0636.6916.2333.59 Q3178RAEv2 DINOv3-L30.4040.1544.8360.2010.008.5010.6467.1439.7438.5518.5833.52 Smol179MetaCLIP 2 ViT-S/16 (mT5, 224)26.0038.6148.2949.909.809.0114.6075.7442.1736.0618.0533.48 Q3180DINO ViT-B/1634.1038.2243.7462.0010.907.658.5271.9540.5536.2313.7933.42 Smol181VILA-U (256)29.0045.1747.7254.5010.307.0311.0379.1440.7524.6817.3533.33 Smol182I-JEPA ViT-H/1427.9035.9147.0553.2010.707.439.2766.6739.4347.8320.1133.23 Smol183MetaCLIP 2 ViT-S/16 (384)28.5038.2248.1851.5011.206.8917.4765.6539.0234.5423.5833.16 Smol184MetaCLIP ViT-B/32 (400M, 224)29.4037.8445.7654.609.607.9813.5166.3638.9239.6820.6233.12 Smol185Pixio ViT-B/1629.3043.2449.8453.3010.107.479.1366.3640.5536.6018.2833.11 Smol186RAEv2 DINOv3-L26.2038.2246.2947.107.205.097.3777.0739.2346.0124.2533.09 Smol187EUPE ViT-S28.4041.3145.1049.7010.105.679.1867.6337.5043.6022.5232.79 Q3188DINO ViT-S/830.7033.9842.5560.2010.606.579.5074.6839.4335.8015.0132.64 Smol189DINOv2 ViT-S/1427.7037.8447.3751.8010.506.899.0969.7541.1635.4617.2432.25 Smol190UniTok (256)28.9033.9846.8451.609.408.0610.1369.3840.4531.6024.3632.25 Smol191DINOv2 ViT-B/1431.1039.3847.2947.809.806.529.0966.5142.6833.8819.0732.10 Q2.5192DINO ViT-B/832.5032.8243.2261.9010.408.228.2869.2740.1426.8311.1031.33 Smol193DINO ViT-B/1628.7030.1244.5254.4010.206.777.5466.0438.5236.9116.5530.93 Q2.5194DINO ViT-S/832.3025.8744.0762.8011.108.229.8766.8237.4026.0012.9930.68 Q3195Web-SSL MAE 3B (224)32.0034.3641.5060.6010.606.469.2467.5537.7024.3512.5630.63 Smol196MetaCLIP 2 ViT-S/16 (224)26.1036.2946.7849.4010.408.6014.8466.0439.5332.076.4830.59 Smol197UniAR-BSQ28.2035.1445.6853.3010.307.048.7763.6737.6025.6017.3130.24 Q3198DINO ViT-S/1630.2027.0340.9762.2010.307.098.4868.4338.2124.3811.9329.93 Q3199Web-SSL MAE 1B (224)34.3027.0340.4462.409.807.328.3066.2837.8021.5111.5729.70 Q2.5200DINO ViT-S/1632.7029.7341.1962.4011.008.549.5366.3638.3118.818.0729.69 Smol201EUPE ViT-T28.5028.1943.4147.0010.305.778.9366.3637.2034.1616.6129.68 Smol202Web-SSL MAE 300M (224)27.0032.0542.2053.7010.405.827.5366.2836.7927.7413.2429.34 Q3203DINO ViT-B/830.0023.1739.7362.5010.706.918.8566.2837.2026.0211.1129.32 Q3204Web-SSL MAE 300M (224)29.6027.0341.7961.2010.107.429.4668.9837.4014.2310.1728.85 Smol205DINO ViT-S/1626.9028.1939.7154.809.806.325.8467.6337.6026.5612.3828.70 Smol206DINO ViT-B/827.3032.0538.1653.4010.006.517.5666.8235.2626.7710.6828.59 Q2.5207EUPE ViT-T31.2025.1040.3860.7011.007.279.1566.2834.765.934.6126.94 Smol208DINO ViT-S/826.2024.7136.8154.3010.007.396.9566.9135.3711.096.7726.05 Smol209Web-SSL MAE 3B (224)26.6025.4839.1053.4010.407.548.5862.3235.573.985.7525.34 Smol210Web-SSL MAE 1B (224)26.6023.5534.1950.1010.507.618.1666.2830.391.543.2823.84 EADDITIONAL TRAINING AND EVALUATION DETAILS Training recipe. All MLLMs follow a controlled two-stage LLaVA-style recipe implemented using LLaVA-NeXT (Liu et al., 2024b) for continuous tokenizers and an equivalent discrete adapter for discrete tokenizers. The visual tokenizer remains frozen throughout. In stage 1, referred to as mid- training in this paper, only the two-layer MLP projector is trained on LLaVA-LCS-558K. In stage 2 (instruction tuning), the projector and language model are jointly trained on the LLaVA-1.5 665K mixture. Images are resized to square crops at each tokenizer’s native resolution, using flat patch- token merging without AnyRes tiling. For language-supervised vision encoders (CLIP, MetaCLIP, and SigLIP2), we use the outputs of the penultimate Transformer block; for self-supervised and 21 reconstruction-based encoders, we use the outputs of the final block. We train each model on eight NVIDIA A100-SXM4-80GB GPUs using distributed data parallelism (DDP) (Li et al., 2020) with bfloat16 precision (Kalamkar et al., 2019). We use AdamW (Loshchilov & Hutter, 2019) with cosine learning-rate decay (Loshchilov & Hutter, 2017), zero weight decay, and gradient clipping (Pascanu et al., 2013) at 1.0. The context length is 8,192, and each stage is trained for one epoch. Optimization. The effective per-GPU batch size is 128 during mid-training and 32 during instruc- tion tuning, corresponding to global batch sizes of 1,024 and 256, respectively. For high-resolution tokenizers or tokenizers with large backbones, we reduce the micro-batch size and increase gradient accumulation to preserve the effective batch size. Mid-training uses a learning rate of 1×10 −3 and a warmup ratio of 0.05. During instruction tuning, the learning rates are 1×10 −5 for the language model and 1×10 −4 for the projector, with a warmup ratio of 0.03. Gradient checkpointing (Chen et al., 2016) is enabled during instruction tuning. Evaluation protocol. We evaluate each instruction-tuned checkpoint using VLMEvalKit (Duan et al., 2024) on eleven benchmarks and splits: MMMU test (Yue et al., 2024), MMBench-EN v1.1 test (Liu et al., 2024c), ScienceQA val (Lu et al., 2022), VQAv2 val (Goyal et al., 2017), GQA Test- Dev balanced (Hudson & Manning, 2019), ChartQA test (Masry et al., 2022), DocVQA val (Mathew et al., 2021), TextVQA val (Singh et al., 2019), POPE (Li et al., 2023), and the Karpathy test splits of MSCOCO (Lin et al., 2014; Karpathy & Fei-Fei, 2015) and Flickr30K (Young et al., 2014; Karpathy & Fei-Fei, 2015). For each benchmark, we sample up to 1,000 examples using a fixed seed of 42 and use the same subset for every MLLM. Decoding is greedy with a maximum of 128 newly generated tokens. Scoring. Responses on MMMU and MMBench are parsed from free-form generations using a lo- cally deployed Qwen3-32B judge. All other benchmarks use standard rule-based metrics: multiple- choice exact match for ScienceQA; official VQA soft accuracy (Antol et al., 2015) for VQAv2 and TextVQA; exact match for GQA; relaxed accuracy with a 5% relative tolerance for ChartQA; ANLS (Biten et al., 2019) for DocVQA; yes/no F1 for POPE; and CIDEr (Vedantam et al., 2015) for the two captioning datasets. Accuracies expressed as percentages and CIDEr multiplied by 100. The aggregate score is the unweighted mean over the eleven benchmarks. Invalid or missing benchmark scores are excluded from the mean rather than imputed. FIMPLEMENTATION DETAILS MLLM construction. For each vision-encoder–language-backbone pair, we connect the vision encoder to the LLM through a separately trained two-layer MLP projector with GELU activa- tion (Hendrycks & Gimpel, 2016). Calibration data and feature extraction. For methods requiring paired data, such as Mutu- alNN (Huh et al., 2024), we follow Li et al. (2026) and randomly sample 1,000 image–text pairs from LLaVA-LCS-558K (Liu et al., 2024a). We extract visual patch tokens from the encoder’s final layer and textual hidden states from the LLM’s penultimate layer. Default settings. Unless otherwise specified, we set the regularization parameter to ε = 1× 10 −4 and the neighborhood size to k = 100. Compute. All MLLM training and downstream evaluation are conducted on eight NVIDIA A100 GPUs with 80 GB of memory each. GDETAILS OF EVALUATION METRICS G.1PREDICTION QUALITY Spearman correlation. Let s i denote the score assigned by a prediction method to the i-th vision encoder, and let y i,t denote its downstream MLLM performance on benchmark t. We use Spear- man’s rank correlation to measure ranking quality: ρ t = P N i=1 (R i − ̄ R)(Q i,t − ̄ Q t ) q P N i=1 (R i − ̄ R) 2 q P N i=1 (Q i,t − ̄ Q t ) 2 , 22 where R i = rank(s i ) and Q i,t = rank(y i,t ). A higher ρ t indicates better preservation of the downstream performance ranking. Pearson correlation. We additionally report Pearson correlation to measure linear agreement be- tween the predicted scores and downstream performance: r t = P N i=1 (s i − ̄s)(y i,t − ̄y t ) q P N i=1 (s i − ̄s) 2 q P N i=1 (y i,t − ̄y t ) 2 . For both metrics, we report the mean correlation across all T downstream benchmarks and the cor- relation with the aggregate downstream performance Avg i = 1 T P T t=1 y i,t , where T is the number of downstream benchmarks. For methods whose native outputs are losses, we negate their scores so that a larger value consistently indicates a stronger vision encoder. G.2COMPUTATIONAL COST We quantify computational cost using the total number of floating-point operations (FLOPs), pro- viding a hardware-independent measure of evaluation efficiency. For each method, we compute the FLOPs required to evaluate every vision-encoder–LLM pair and report the average across all 210 pairs. The estimate covers the complete evaluation pipeline, including visual and textual feature extraction, feature post-processing, and score computation, as well as probe training and inference when applicable. Averaging over all pairs accounts for differences in encoder architecture, input resolution, and language backbone, yielding a comparable cost estimate for each method. HFURTHER ANALYSIS OF GLOBAL GEOMETRIC COMPATIBILITY We further examine two limitations of GW-based compatibility measures: their sensitivity to encoder-specific feature geometry and their ability to rematch samples without preserving observed image–text correspondences. H.1SENSITIVITY TO ENCODER HETEROGENEITY Li et al. (2026) use Gromov–Wasserstein (GW) distance to compare the global geometry of visual and textual representations. However, as shown in Figure 1, its ranking agreement depends strongly on the candidate pool. GW performs well in a small, homogeneous pool but becomes less reliable as heterogeneous encoder families are introduced. Figure 8(a) and (b) illustrate this issue: two encoders with similar downstream performance under Qwen2.5-1.5B (Yang et al., 2024) exhibit markedly different feature distributions in t-SNE visualizations (van der Maaten & Hinton, 2008). Thus, global geometry may reflect encoder-specific training objectives and output interfaces beyond downstream MLLM utility. H.2DOES GLOBAL GEOMETRY PRESERVE IMAGE–TEXT CORRESPONDENCES? A second issue arises from the free coupling optimized by GW. A low discrepancy may reflect an alternative geometric matching rather than consistency with the observed image–caption pairs. We examine this distinction by comparing the optimized GW coupling with a fixed pair-preserving coupling. Data and representations. We randomly sample n = 2,000 image–caption pairs from the COCO 2017 validation set (Lin et al., 2014) using seed 42. Images are represented by global features from each of the 70 candidate vision encoders, while captions are encoded by Qwen3-1.7B, Qwen2.5- 1.5B, and SmolLM2-1.7B. All representations are L 2 -normalized. GW computation. For each of the 70× 3 = 210 encoder–language-backbone combinations, we solve the exact L 1 -angular GW objective from the image–image and text–text distance matrices using uniform marginals and a MutualNN initialization. This produces a free coupling T ⋆ , which we compare with the pair-preserving coupling T pair = I n /n. We measure correspondence recovery 23 MLLM Avg. 40.76 GW dist. 0.0326 GW rank 2 MLLM Avg. 40.76 GW rank 2 (a) TokLIP feature geometry. MLLM Avg. 40.87 GW dist. 0.2027 GW rank 70 MLLM Avg. 40.87 GW rank 70 (b) UniAR feature geometry. Figure 8: Sensitivity of GW to encoder heterogeneity. Encoders with similar downstream perfor- mance can exhibit markedly different feature distributions. PCA dimension 1 PCA dimension 2 Image Caption True pair (a) Observed pairing. PCA dimension 1 PCA dimension 2 Image Caption GW incorrect GW kept true (b) GW-induced rematching. 0.0250.0500.0750.1000.125 Free GW matching cost 0.02 0.04 0.06 0.08 0.10 0.12 True pairing cost Equal cost Lang-sup. SSL Discrete LLM Qwen3 Qwen2.5 SmolLM2 (c) Cost of pair preservation. Figure 9: Free GW matching can bypass observed cross-modal correspondences. (a–b) Ob- served pairs and maximum-mass GW assignments for a representative encoder–language-backbone combination. (c) Pair-preserving versus free-matching costs across all 210 combinations. as PairAcc(T ⋆ ) = 1 n n X i=1 1 arg max j T ⋆ ij = i . Results and visualization. Figures 9(a) and 9(b) visualize SigLIP2 So400m/14 paired with Qwen3- 1.7B. Image and caption features are projected independently to two dimensions using PCA, af- ter which the caption coordinates are Procrustes-aligned to the image coordinates for visualiza- tion only. We show the observed assignments i 7→ i and the maximum-mass GW assignments i7→ arg max j T ⋆ ij . GW extensively reassigns captions: PairAcc averages only 0.0077 across all 210 combinations. Moreover, every combination lies above the equal-cost diagonal in Figure 9(c), with the pair- preserving cost averaging 1.74× the optimized free-matching cost. These results show that a low GW discrepancy can arise from sample rematching rather than preservation of the observed cross- modal correspondences. Together, these analyses show that GW can be affected by encoder-specific feature geometry while obtaining low costs through correspondence-agnostic rematching, limiting its reliability for hetero- geneous encoder selection. 24 ISUPERVISED REFERENCE METHOD RAVEL targets training-free encoder selection, assuming no downstream performance labels for the candidate encoders. As a complementary analysis, we also consider a supervised setting in which the downstream MLLM performance of a small subset of encoder–language-backbone pairs is available. This setting examines whether previously evaluated candidates can provide additional supervision for selecting among unseen candidates and provides a supervised reference for interpreting RAVEL’s results. Directly learning from frozen encoder features is difficult because the candidate feature spaces are incompatible: their dimensions range from 384 to 4608, and their coordinates arise from hetero- geneous architectures and pretraining objectives. We therefore develop CKA-X, which represents each candidate using coordinate-agnostic geometric signals. The first measures its centered-kernel- alignment similarity (Kornblith et al., 2019) to a shared set of reference encoders, while the second measures its cross-modal compatibility with the frozen target language model. A gradient-boosted decision-tree ensemble maps these signals to predicted downstream performance. CKA-X requires downstream labels from the reference candidates but does not require training MLLMs for the held- out candidates. I.1PROBLEM SETUP We now consider the supervised counterpart of §3, keeping its notation: D = (x i ,t i ) n i=1 is the paired calibration data, E m is a candidate vision encoder, and L is the target language model. The candidate pool V = E m M m=1 consists of the 70 vision encoders evaluated in our experiments, each paired with one of the three language backbones; each E m now also carries a ground-truth score y m ∈R, the unweighted mean over the eleven downstream benchmarks of the MLLM built from E m with backbone L. Let Z m ∈R n×d m stack the visual representations that E m produces on the calibration images (the Z v,m i of §3), and let Z t ∈R n×d g stack the text representations that L produces on their paired texts. We evaluate this by holding each candidate out in turn: for a held-out candidate we predict its score from its frozen features and the labeled reference pool of the remaining candidates, R = V \held-out, without training an MLLM for any candidate. No downstream score of a held-out encoder is ever used, and the reference pool contains only candidates whose scores are known. Raw coordinates cannot support this prediction: padding equalizes only the dimensionality, not the coordinate systems, so raw features from different encoders remain numerically incomparable even after zero-padding to a common width. We therefore represent each candidate by two frame- invariant signals, a pairwise CKA similarity profile and a cross-modal alignment, and map them to the score with gradient-boosted decision trees. I.2PAIRWISE CKA SIMILARITY To construct a common descriptor for encoders with heterogeneous feature spaces, we compare each candidate with a shared pool of reference encoders using linear centered kernel alignment (CKA) (Kornblith et al., 2019). Let H = I n − 1 n 11 ⊤ be the centering matrix. For candidate E m , we first L 2 -normalize the rows of Z m and construct its centered Gram matrix G m = HZ m Z ⊤ m H. For every candidate pair (m,m ′ ), their linear CKA similarity is CKA(m,m ′ ) = ⟨G m ,G m ′ ⟩ F ∥G m ∥ F ∥G m ′ ∥ F = tr(G m G m ′ ) p tr(G 2 m ) tr(G 2 m ′ ) ∈ [0, 1]. Given a reference pool R, we represent each candidate by its CKA similarities to the reference encoders: k m = CKA(m,m ′ ) m ′ ∈R . 25 The same reference pool is used for both the candidates fitted by the predictor and the held-out candidates, placing all descriptors in the common spaceR |R| . CKA is well suited to this setting because it compares the geometry that two encoders induce over the shared images rather than their raw feature coordinates. In particular, it measures whether the encoders organize the images similarly and is invariant to orthogonal transformations, isotropic rescaling, and the addition of zero-valued feature dimensions. It can therefore compare representa- tions with different dimensionalities and coordinate systems without learning an explicit mapping between them. Moreover, k m is relational: it describes how a candidate is positioned relative to the scored refer- ence encoders. Unlike a single intrinsic representation statistic, this descriptor retains cross-encoder information that allows a predictor fitted on evaluated candidates to transfer to an unseen candidate. We use linear CKA because it is dimension-agnostic, requires no rank-selection hyperparameter, and can be computed directly from the centered Gram matrices. I.3CROSS-MODAL ALIGNMENT The CKA profile characterizes each candidate relative to the reference encoder pool but does not account for the target language model. We therefore augment it with two signals that measure the compatibility between the visual representation and the text representation produced by the frozen target LLM L. Let Z t ∈R n×d g denote the text representations of the paired captions, obtained by mean-pooling the penultimate-layer hidden states of L. For each candidate E m , we compute two scalar statistics. The first is the held-out R 2 of a ridge regression that predicts Z t from Z m . The regression is fitted on one randomly selected half of the calibration pairs and evaluated on the other, using split seed 42; its regularization strength is selected by leave-one-out cross-validation within the fitting split. The second statistic is the linear CKA between Z m and Z t , which measures the agreement between their induced sample geometries. Together, these statistics form the cross-modal descriptor x m = R 2 m , CKA(Z m ,Z t ) ⊤ ∈R 2 . I.4GRADIENT-BOOSTED REGRESSION We concatenate the relational CKA profile and cross-modal descriptor into d m = [k m ; x m ]∈R |R|+2 and use gradient-boosted regression trees to predict the downstream MLLM score: ˆy m = b + η T X t=1 h t (d m ), where b is the initial prediction, h t is a depth-limited regression tree, η is the learning rate, and T is the number of boosting rounds. Gradient boosting is well suited to this compact descriptor because it can capture nonlinear threshold effects and interactions between similarity to the reference encoders and compatibility with the target LLM. We compare gradient boosting with several common alternatives, including ridge regression, a mul- tilayer perceptron, and kernel ridge regression using the CKA matrix as its kernel. All preprocess- ing and model selection are performed using only the training folds. Gradient boosting achieves the strongest performance for all three language backbones, whereas the multilayer perceptron performs substantially worse. We therefore adopt gradient boosting as the default predictor. Its hyperparame- ters are fixed across all folds and language backbones: depth-2 trees, T = 50 boosting rounds, learn- ing rate η = 0.05, and subsampling ratio 0.8. The resulting model also provides feature-importance estimates for analyzing the contributions of the relational and cross-modal signals. Scoring a held-out candidate requires only its frozen representations and the downstream scores of the reference pool. Thus, the supervision cost is confined to the reference candidates, and no MLLM needs to be trained for the held-out candidate. 26 816243240485664 Number of labeled encoders 0.30 0.40 0.50 0.60 0.70 0.80 0.90 Spearman Qwen2.5-1.5B Qwen3-1.7B SmolLM2-1.7B Figure 10: Data efficiency of CKA-X with the gradient-boosted predictor on the 70-encoder pool. Each curve is one target language backbone L; the x-axis is the number of labeled encoders in the reference pool R, with the remaining 70−|R| candidates held out. Points are pooled Spearman correlations over the held-out predictions of 100 random labeled subsets. This cost is set by how many candidates are labeled rather than by which predictor is used, so we also ask how many labeled encoders CKA-X actually needs. For each budget |R| ∈ 8, 16, 24, 32, 40, 48, 56, 64 we draw 100 random labeled subsetsR⊂V (seed 0), fit the predictor using the labels ofR alone, and score the remaining candidatesV ; the leave-one-out protocol above is the maximal-budget case|R| = M − 1 = 69. Figure 10 reports the resulting correlation on the held-out candidates. Supervision transfers quickly across the pool: eight labeled encoders already yield pooled Spearman correlations of 0.50, 0.51 and 0.36 for Qwen3-1.7B, Qwen2.5-1.5B and SmolLM2-1.7B, and the curves rise monotonically to 0.81, 0.85 and 0.69 at the largest budget, flattening from about half the pool onward. JREPRESENTATION POST-PROCESSING VARIANTS Protocol. The post-processing comparison isolates how feature statistics affect cross-modal neigh- borhood agreement. In this diagnostic, each visual representation is obtained by averaging all valid spatial patch tokens of an image before post-processing, while each textual representation is the penultimate-layer last-token feature. We use the same n = 1,000 LCS-558K examples (seed 42), 70 visual tokenizers, and three language backbones for every variant. The transformation is fitted in- dependently for each visual tokenizer and each language backbone, so no projection parameters are shared across modalities or models. After transformation, every sample is row-wise ℓ 2 -normalized, and Top-100 cosine neighborhoods are used to compute the binary MutualNN overlap. No patch- level matching, rank weighting, hub correction, or chance adjustment is applied in this comparison; only the representation post-processing changes. Raw. This baseline leaves the representation coordinates unchanged and applies only row normal- ization, e X raw =R(X).(9) It therefore preserves the original mean offset, coordinate scales, correlations, and anisotropy, while removing differences in the overall norm of individual samples. Centered. We subtract the feature-wise mean estimated from the fixed evaluation sample and then normalize each row, e X center =R(X − 1μ ⊤ ).(10) Centering removes a common offset shared by the examples but does not alter the relative scale or correlation of different feature dimensions. Standardized. We additionally divide each centered coordinate by its population standard deviation, e X std =R X − 1μ ⊤ σ + 10 −5 ,(11) 27 204060 20 30 40 Candidate pool size Top-1 score Figure 11: Selection robustness across candidate-pool sizes. Ground-truth performance of the encoder selected by each method as increasingly strong candidates enter the pool. where the division is coordinate-wise. This gives the original coordinates comparable marginal scales and reduces the influence of unusually high-variance dimensions, but it does not decorrelate them and remains tied to the encoder’s native coordinate basis. PCA only. We project the centered features onto the retained principal subspace and normalize the resulting scores, e X PCA =R(Z).(12) This rotation decorrelates the retained coordinates and truncates the low-variance tail beyond rank r, but it keeps the variance ordering within the retained subspace. Consequently, leading principal components can still dominate cosine neighborhoods. ABTT. All-but-the-Top (ABTT) (Mu et al., 2017) starts from the same PCA scores but discards the first q highest-variance directions before row normalization. With the fixed setting q = 4 used in the figure, e X ABTT =R Z [:,q+1:r] , q = 4.(13) ABTT directly removes a small number of dominant directions that may encode broad, non-specific variation. Unlike whitening, however, it deletes those directions entirely, so its behavior depends on the chosen removal count q and can deteriorate when informative principal components are removed. PCA whitening. We retain the same PCA subspace as PCA only, but rescale each component by its regularized standard deviation, e X white =R Z diag (λ 1 + 10 −4 ) −1/2 ,..., (λ r + 10 −4 ) −1/2 .(14) Whitening therefore equalizes the variances of the retained principal directions rather than removing the leading ones. The additive 10 −4 term limits the amplification of small-eigenvalue directions, and the rank cap removes the least reliable low-variance tail before rescaling. J.1BASELINE METHODS • Linear probing (Caron et al., 2021) trains a linear classifier on frozen visual representations and uses image-classification accuracy as the encoder score. • k-nearest-neighbor classification (KNN; Wu et al., 2018) predicts image labels from their nearest neighbors in the visual representation space and uses classification accuracy as the encoder score. We next consider cross-modal metrics that compare visual and textual representation spaces without training the corresponding MLLM: • Representational similarity analysis (RSA; Kriegeskorte et al., 2008) measures the correla- tion between the pairwise dissimilarity structures of the visual and textual representations, while canonical correlation analysis (CCA; Hotelling, 1936) learns linear projections of the two spaces that maximize their correlation. 28 • Gromov–Wasserstein (GW; Li et al., 2026) compares visual and textual representations by matching their within-modality distance structures through optimal transport, while mutual k-nearest-neighbor alignment (MutualNN; Huh et al., 2024) measures the overlap between nearest-neighbor sets constructed in the visual and textual representation spaces. Finally, we include several methods that require additional model training or fitting. Because of their substantially higher computational or supervision requirements, these methods may be less suitable as practical evaluation metrics for large-scale vision encoder selection. Nonetheless, we include them to provide stronger reference points: • Mid-training loss performs the first-stage mid-training for each encoder–language-model pair using the LLaVA training recipe and uses the loss at the final optimization step as an estimate of downstream performance. • Alignment probing (Zhang et al., 2025) uses labeled image–text data to learn linear map- pings between the visual and textual spaces and evaluates their alignment through cross- modal retrieval. • AC Policy (Yang et al., 2025b) fits a polynomial regressor that maps alignment and vi- sual correspondence scores to the observed downstream performance of a subset of vision encoders. KROBUSTNESS UNDER A SELECTION-BASED METRIC We evaluate whether each method reliably selects a strong encoder as the candidate pool changes. For each language backbone, we order the 70 encoders by ground-truth downstream performance and progressively expand the pool from the worst encoder to the full set. At each pool size, every method selects its highest-scoring encoder, whose ground-truth performance is reported alongside that of the best available encoder (oracle). As shown in Figure 11, RAVEL closely tracks the rising oracle throughout the expansion, whereas competing methods plateau as stronger encoders become available. This demonstrates its robustness to changes in candidate-pool size and composition. 29

Instant research discovery