Paper deep dive
Modulating Cross-Modal Convergence with Single-Stimulus, Intra-Modal Dispersion
Eghbal A. Hosseini, Brian Cheung, Evelina Fedorenko, Alex H. Williams
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 4/26/2026, 9:04:43 PM
Summary
The paper introduces a methodology using the Generalized Procrustes Algorithm (GPA) to measure representational convergence at the single-stimulus level. The authors demonstrate that 'intra-modal dispersion' (the degree of disagreement among vision models for a specific stimulus) strongly modulates 'cross-modal convergence' (the alignment between vision and language models). Specifically, stimuli with low intra-modal dispersion exhibit significantly higher alignment with language models, a finding that is robust across different vision models (MAE, DINOv2, CLIP) and language model families (BLOOM, OpenLLaMA, LLaMA).
Entities (10)
Relation Signals (5)
DinoV2 → isa → Vision Model
confidence 100% · We applied this to vision models with distinct training objectives... DINOv2
BLOOM → isa → Language Model
confidence 100% · We focused on BLOOM (Workshop et al., 2023)
Generalized Procrustes Algorithm → measures → Intra-modal Dispersion
confidence 100% · Here, we introduce a methodology based on the Generalized Procrustes Algorithm to measure intra-modal representational convergence
CKNNA → measures → Cross-modal Alignment
confidence 100% · To measure the local alignment between vision and language modalities, we employed Centered Kernel k-Nearest Neighbors Alignment (CKNNA)
Intra-modal Dispersion → modulates → Cross-modal Convergence
confidence 100% · this intra-modal dispersion strongly modulates alignment between vision and language models (cross-modal convergence)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Neural networks exhibit a remarkable degree of representational convergence across diverse architectures, training objectives, and even data modalities. This convergence is predictive of alignment with brain representation. A recent hypothesis suggests this arises from learning the underlying structure in the environment in similar ways. However, it is unclear how individual stimuli elicit convergent representations across networks. An image can be perceived in multiple ways and expressed differently using words. Here, we introduce a methodology based on the Generalized Procrustes Algorithm to measure intra-modal representational convergence at the single-stimulus level. We applied this to vision models with distinct training objectives, selecting stimuli based on their degree of alignment (intra-modal dispersion). Crucially, we found that this intra-modal dispersion strongly modulates alignment between vision and language models (cross-modal convergence). Specifically, stimuli with low intra-modal dispersion (high agreement among vision models) elicited significantly higher cross-modal alignment than those with high dispersion, by up to a factor of two (e.g., in pairings of DINOv2 with language models). This effect was robust to stimulus selection criteria and generalized across different pairings of vision and language models. Measuring convergence at the single-stimulus level provides a path toward understanding the sources of convergence and divergence across modalities, and between neural networks and human neural representations.
Tags
Links
- Source: https://arxiv.org/abs/2604.21836v1
- Canonical: https://arxiv.org/abs/2604.21836v1
Trouble viewing inline? Open PDF directly →
Full Text
24,593 characters extracted from source content.
Expand or collapse full text
To appear at the ICLR 2026 Workshop on Representational Alignment (Re-Align) MODULATING CROSS-MODAL CONVERGENCE WITH SINGLE-STIMULUS, INTRA-MODAL DISPERSION Eghbal A. Hosseini 1∗ , Brian Cheung 2 , Evelina Fedorenko 1 & Alex H. Williams 3,4 1 Department of Brain and Cognitive Sciences, MIT 2 CSAIL, MIT 3 Center for Neural Science, NYU 4 Center for Computational Neuroscience, Flatiron Institute ehoseini, evelina9@mit.edu, cheungb@mit.edu, aw4614@nyu.edu ABSTRACT Neural networks exhibit a remarkable degree of representational convergence across diverse architectures, training objectives, and even data modalities. This convergence is predictive of alignment with brain representation. A recent hypoth- esis suggests this arises from learning the underlying structure in the environment in similar ways. However, it is unclear how individual stimuli elicit convergent representations across networks. An image can be perceived in multiple ways and expressed differently using words. Here, we introduce a methodology based on the Generalized Procrustes Algorithm to measure intra-modal representational convergence at the single-stimulus level. We applied this to vision models with distinct training objectives, selecting stimuli based on their degree of alignment (intra-modal dispersion). Crucially, we found that this intra-modal dispersion strongly modulates alignment between vision and language models (cross-modal convergence). Specifically, stimuli with low intra-modal dispersion (high agree- ment among vision models) elicited significantly higher cross-modal alignment than those with high dispersion, by up to a factor of two (e.g., in pairings of DI- NOv2 with language models). This effect was robust to stimulus selection criteria and generalized across different pairings of vision and language models. Measur- ing convergence at the single-stimulus level provides a path toward understanding the sources of convergence and divergence across modalities, and between neural networks and human neural representations. 1INTRODUCTION Artificial neural networks show a remarkable ability to learn representations that generalize across tasks, and predict neural representations in humans and other animals. Even though details of im- plementation — including training, architecture, and input modality — vary between networks, they appear to have minimal influence on the final representation, and models converge onto similar rep- resentations. The Platonic Representation Hypothesis (Huh et al., 2024) suggests this convergence arises from learning shared environmental priors. Hosseini et al. (2024) found similar evidence in convergence across neural networks and their predictivity of brain representation. Both lines of work however established convergences across groups of stimuli, and thus globally. Individual observations are critical for probing and understanding representations and can poten- tially drive models, and humans, toward either convergence or divergence. Humans often find them- selves interpreting differently the same work of art, a painting for example. Existing methods are often ill-suited for probing convergence at single stimulus level, as they either measure local effects (Feather et al., 2023), average similarity over a large set of stimuli datasets (Hosseini et al., 2024), or lack a proper metric space for rigorous comparison (Sucholutsky et al., 2024; Harvey et al., 2023). This highlights the need for a robust, stimulus-specific measure of alignment. ∗ Now at Google DeepMind 1 arXiv:2604.21836v1 [q-bio.NC] 23 Apr 2026 To appear at the ICLR 2026 Workshop on Representational Alignment (Re-Align) Building on Generalized Procrustes Analysis (Gower, 1975), we introduce a stimulus-specific mea- sure of representational convergence across vision models (Williams et al., 2021; Haxby et al., 2020). We then demonstrate that our measure can be used to select stimuli that effectively modulate repre- sentational alignment across modalities, from vision to language. 2METHODS 2.1GENERALIZED PROCRUSTES METHOD Generalized Procrustes Analysis (GPA) is a method for aligning multiple datasets, or embedding spaces, into a common reference frame. The core idea is to find a set of optimal transformations, restricted to orthogonal transformations, that minimize the discrepancy between a shared consensus configuration and each individually transformed dataset (Gower, 1975). Stimulus-level residuals are a classical by-product of GPA; our contribution is their application to comparing neural network representations and to modulating cross-modal alignment. Formally, consider a set of M representation matricesN 1 ,N 2 ,...,N M , where each representa- tion N i ∈R m×n i , with i∈1,...,M indexing models, represents m corresponding samples (e.g., stimuli) in an n i -dimensional space. The dimensionality n i can differ across the representations. GPA seeks to find an optimal consensus representation, N joint ∈R m×k , and a set of corresponding orthogonal transformation matrices,T 1 ,T 2 ,...,T M , where each T i ∈R n i ×k , by minimizing the sum of squared Frobenius distances. The optimization problem is defined as: min N joint ,T i M i=1 M X i=1 ∥N i T i − N joint ∥ 2 F (1) subject to each transformation matrix being orthogonal, T T i T i = I k for all i ∈ 1,...,M. Here, ∥·∥ F denotes the Frobenius norm, and the constraint T T i T i = I k ensures that the transformations do not distort the internal geometry of each representation N i . This process effectively rotates each embedding space to achieve maximal alignment with the emergent consensus space N joint . Prior to performing GPA, we first center each embedding along the second dimension n i , zero- pad dimensions across all models to a common dimensionality, and normalize each representation ∥N i ∥ F = 1. Generalized Procrustes Analysis can also be interpreted as computing a barycenter of neural rep- resentations in Procrustes shape space. In classical shape analysis, each centered, scale-normalized configuration N i corresponds to a point on a quotient manifold obtained by modding out rotations, 1 and Generalized Procrustes iteratively aligns all configurations to the barycenter (also called the Fréchet mean or Karcher mean), represented by N joint . This perspective clarifies that GPA is not merely an arbitrary linear alignment, but a principled procedure for finding the central tendency of a population of neural representations with respect to a scale and rotation-invariant metric space (Williams et al., 2021). We restrict each T i to orthogonal transformations, rather than a general linear map, because they preserve the internal geometry of each representation: pairwise distances and angles between stimuli are unchanged under T i . A general linear map can absorb scale and reshape covariance structure, which are not desirable for defining metrics on representation (Williams et al., 2021). Scale is instead handled explicitly by the Frobenius normalization∥N i ∥ F = 1. 2.2QUANTIFYING SINGLE-STIMULUS DISPERSION To quantify representational convergence at the single-stimulus level, we measure the disper- sion between each model’s transformed representation and the shared joint representation, N joint . Specifically, after computing the joint space via GPA, we first project each model’s representa- tion into this common space using its corresponding transformation matrix, T i . For each stimulus j ∈1,...,m, we then compute the Euclidean distance between its projected representation from 1 Here we include reflections as well as rotations in the equivalence class. This is common in the analysis of neural representations since even a permutation of the neural unit labels can result in a reflection. 2 To appear at the ICLR 2026 Workshop on Representational Alignment (Re-Align) model i and its joint representation. This yields the Procrustes residual, r ij , defined as: r ij =∥(N i T i ) j − (N joint ) j ∥ 2 (2) This process results in a residual matrix R∈R m×M , where each entry r ij quantifies the dispersion for a given stimulus j and model i (see Fig 1C). A low residual value signifies high alignment (low dispersion), while a high value indicates low alignment (high dispersion). We employ two strategies to summarize the overall dispersion for each stimulus across all M mod- els: 1. Mean Dispersion: Our first approach is to average the residuals across all models for each stimulus. We rank the stimuli from least to most dispersed. While straightforward, this metric can be sensitive to outlier models (Fig 1D) 2. Principal Component of Dispersion: To capture the primary axis of disagreement in a more robust manner, our second approach utilizes Principal Component Analysis (PCA) on the residual matrix R. We use the score along the first principal component (PC1) as a more nuanced measure of dispersion for each stimulus, as it reflects the most significant shared variance in model disagreement (Fig 1E). 2.3VISION MODELS We inspected representations across ViT models trained on diverse objectives (Huh et al., 2024). These objectives include Masked Autoencoders (MAE) (He et al., 2022), DINO (Caron et al., 2021), CLIP (Radford et al., 2021), and CLIP with additional finetuning on ImageNet-12K. To compute the joint Procrustes representation, we selected a large architecture in each class: MAE: vit_huge_patch14_224.mae;DINO: vit_giant_patch14_dinov2.lvd142m; CLIP: vit_huge_patch14_clip_224.laion2b; CLIP + finetuning on ImageNet-12K: vit_huge_patch14_clip_224.laion2b_ft_in12k. All ViTs have a dimension of 1280, except for the DINO model, which had a dimension of 1536. We thus zero-padded all other model representations to match the DINO dimensions. We extracted the CLS-token representation from the penultimate transformer block of each ViT and used it for Procrustes alignment, and identified stimuli with varying degrees of dispersion. 2.4LANGUAGE MODELS Similar to Huh et al. (2024) we compared representations of ViT vision model to LLMs across model families. We focused on BLOOM (Workshop et al., 2023) , OpenLLaMA (Geng & Liu, 2023), and LLaMA model class (Touvron et al., 2023). For each LLM, we extracted token embeddings from each transformer block and mean-pooled across tokens to obtain a single representation per caption, we then used the block that showed best alignment with the vision models. 2.5DATASET Following Huh et al. (2024), we used Wikipedia caption dataset (Srinivasan et al., 2021) to measure convergence between modalities. This dataset included a set of 1024 image/caption pairs from Wikipedia articles. Images and captions are used as released in the Wikipedia caption dataset (WIT); no additional filtering was applied. 2.6ALIGNMENT MEASURE To measure the local alignment between vision and language modalities, we employed Centered Kernel k-Nearest Neighbors Alignment (CKNNA) (Huh et al., 2024; Kornblith et al., 2019). This method adapts the standard Centered Kernel Alignment (CKA) to focus on local representational structure. Intuitively, CKNNA computes the cross-covariance only between a sample and its nearest neighbors, thereby assessing the local, rather than global, similarity between two representation spaces. We considered 10 nearest neighbors for each sample when reporting alignment (Fig. 2). 3 To appear at the ICLR 2026 Workshop on Representational Alignment (Re-Align) A B dipsersion between models and joint C MAE DINOv2 CLIP Individual Vision ModelsJoint Vision Representation average dipsersion Stimulus order low dispersion high dispersion PC2 PC1 low dispersion high dispersion BLOOM Language modelsVision Models OpenLlamaLlama Alignment (CKNNA) D E F High dispersion between & Low dispersion between & Figure 1: (A) Overview of the Generalized Procrustes Analysis (GPA) problem. Given represen- tations from diverse vision models, our goal is to learn a set of model-specific transformations to construct a single joint representation. (B) When individual model representations are projected onto this joint space, stimuli can exhibit either low dispersion between joint and individual models (bottom) or instead high dispersion (top). (C) This dispersion is quantified in a residual space, where each stimulus is represented as a point whose coordinates reflect its distance between each model’s representation and the joint space. (D, E) We use two approaches to identify stimuli with varying degrees of dispersion: (D) ranking stimuli based on their average dispersion across all models and (E) using the score along the first principal component (PC1) of the residual space as a more robust measure. (F) After identifying stimuli with high and low dispersion, we measure the alignment be- tween the vision models and language models using a local similarity metric (CKNNA) to test how stimulus selection modulates cross-modal convergence. 3RESULTS We investigated the dependence of convergence between visual and linguistic representations on stimuli using the Procrustes-based methodology described above. As a stricter test of the Platonic Representation Hypothesis, we considered representations learned across ViT transformer architec- tures with different objectives, an effect not explored in the original work. After computing a joint representation across these vision models, we extracted stimuli with varying degrees of dispersion to test their impact on cross-modal alignment. Stimuli with low intra-modal dispersion yielded a substantially higher degree of cross-modal con- vergence between visual and linguistic representations. We first identified stimuli with the least and most dispersion using a rank-ordering of their mean residuals (Fig. 2A). We selected three sets: (1) low-dispersion, (2) high-dispersion, and (3) random baseline. We then compared vision model alignment with three classes of LLMs. The low-dispersion set showed significantly higher conver- gence, up to a twofold increase in some cases (DinoV2, CLIP+ft on ImN12K), over the random and high-dispersion stimuli (Fig. 2B-E). The correspondence between intra-modal dispersion and cross-modal convergence generalizes to different sampling strategies. Instead of rank ordering, we performed PCA on the residual space and selected stimuli along the first principal component (PC1), as shown in (Fig. 2F). When we repeated the alignment experiment, we found a consistent pattern where the degree of intra-modal dispersion strongly correlated with cross-modal convergence (Fig. 2G-J). This confirms that the significant gap in alignment between low- and high-dispersion stimuli is a robust phenomenon across all tested models. 4DISCUSSION We applied Generalized Procrustes Analysis to identify stimuli with varying representational disper- sion across vision models at the single-stimulus level. We showed that selecting for low-dispersion stimuli can boost cross-modal alignment with language models close to twofold, and is robust across different selection criteria. 4 To appear at the ICLR 2026 Workshop on Representational Alignment (Re-Align) -0.010.000.010.02 PC1 (58.21%) -0.01 0.00 0.01 0.02 PC2 (30.84%) 0 60 Variance explained 02004006008001000 Stimulus Order 0.025 0.055 Dispersion low dispersion random high dispersion AB F G H IJ low dispersion random high dispersion 560m 1b11b7 3b 7b1 3b7b 13b 7b 13b30b65b bloom MAE DinoV2 CLIP CLIP+ft on ImNt12K CDE 0.2 0.3 0.4 0.5 0.6 Alignment (CKNNA,10) 560m 1b11b7 3b 7b1 3b7b 13b 7b 13b30b65b 0.2 0.3 0.4 0.5 0.6 Alignment (CKNNA,10) PC# llama open llama bloomllama open llama Figure 2: Stimulus-specific dispersion modulates vision-language alignment. (A) rank based stimulus selection:stimuli are sorted by mean dispersion to create low-, high-, and random- dispersion sets. (B-E) Vision-language alignment (CKNNA) is then measured for each set across four vision models (ViT-MAE, DINOv2, CLIP, and CLIP+FT). (F) PCA based stimulus selection: stimuli are selected based on their score along the first principal component (PC1) of the dispersion space. (G-J) The alignment experiment is repeated using these sets. For both selection methods, low-dispersion stimuli consistently yield the highest vision-language alignment across all tested models. Our approach helps uncover what features drive representational convergence and could potentially offer a more stringent test for comparing artificial and biological neural networks. Future work could extend these findings to more datasets and, critically, identify the common features within low- and high-dispersion stimuli that are responsible for modulating alignment. REFERENCES Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 9650–9660, October 2021. Jenelle Feather, Guillaume Leclerc, Aleksander M ̨adry, and Josh H McDermott. Model metamers reveal divergent invariances between biological and artificial neural networks. Nat. Neurosci., 26 (11):2017–2034, November 2023. Xinyang Geng and Hao Liu. Openllama: An open reproduction of llama. https://github. com/openlm-research/open_llama, May 2023. Accessed: [Insert Date]. John C. Gower. Generalized Procrustes analysis. Psychometrika, 40(1):33–51, 1975. Sarah E. Harvey, Brett W. Larsen, and Alex H. Williams. Duality of bures and shape distances with implications for comparing neural representations, 2023. URL https://arxiv.org/abs/ 2311.11436. James V Haxby, J Swaroop Guntupalli, Samuel A Nastase, and Ma Feilong. Hyperalignment: Modeling shared information encoded in idiosyncratic cortical topographies. eLife, 9:e56601, jun 2020. ISSN 2050-084X. doi: 10.7554/eLife.56601. URL https://doi.org/10.7554/ eLife.56601. Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 16000–16009, June 2022. 5 To appear at the ICLR 2026 Workshop on Representational Alignment (Re-Align) Eghbal Hosseini, Colton Casto, Noga Zaslavsky, Colin Conwell, Mark Richardson, and Evelina Fedorenko. Universality of representation in biological and artificial neural networks. bioRxiv, 2024. doi: 10.1101/2024.12.26.629294. URL https://w.biorxiv.org/content/ early/2024/12/26/2024.12.26.629294. Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis, 2024. URL https://arxiv.org/abs/2405.07987. Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neu- ral network representations revisited. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceed- ings of Machine Learning Research, p. 3519–3529. PMLR, 09–15 Jun 2019. URL https: //proceedings.mlr.press/v97/kornblith19a.html. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, p. 8748–8763. PMLR, 18–24 Jul 2021.URL https://proceedings.mlr.press/v139/radford21a. html. Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning.In Pro- ceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, p. 2443–2449, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450380379. doi: 10.1145/3404835.3463257. URL https://doi.org/10.1145/3404835.3463257. Ilia Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng, Andreea Bobu, Been Kim, Bradley C. Love, Christopher J. Cueva, Erin Grant, Iris Groen, Jascha Achterberg, Joshua B. Tenenbaum, Katherine M. Collins, Katherine L. Hermann, Kerem Oktar, Klaus Greff, Mar- tin N. Hebart, Nathan Cloos, Nikolaus Kriegeskorte, Nori Jacoby, Qiuyi Zhang, Raja Mar- jieh, Robert Geirhos, Sherol Chen, Simon Kornblith, Sunayana Rane, Talia Konkle, Thomas P. O’Connell, Thomas Unterthiner, Andrew K. Lampinen, Klaus-Robert Müller, Mariya Toneva, and Thomas L. Griffiths. Getting aligned on representational alignment, 2024. URL https: //arxiv.org/abs/2310.13018. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https://arxiv.org/abs/2307.09288. Alex H Williams, Erin Kunz, Simon Kornblith, and Scott Linderman. Generalized shape metrics on neural representations. Advances in neural information processing systems, 34:4738–4750, 2021. BigScience Workshop, :, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili ́ c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, and François Yvon et. al. Bloom: A 176b-parameter open-access multilingual language model, 2023. URL https://arxiv. org/abs/2211.05100. 6 To appear at the ICLR 2026 Workshop on Representational Alignment (Re-Align) SUPPLEMENTARY MATERIAL 560m 1b11b7 3b 7b1 3b7b 13b 7b 13b30b65b MAE DinoV2 CLIP CLIP+ft on ImNt12K Alignment (CKNNA,10) 0.05 0.10 0.15 0.20 0.25 bloom llama o.llama low dispersion random high dispersion Stimulus group dropped Supplementary Figure S1: Excluding the highest-dispersion stimuli systematically increases vision-to-language alignment, whereas excluding the lowest-dispersion stimuli decreases it. We removed the 25% of stimuli with the highest mean dispersion (drop high-dispersion, blue) or the 25% with the lowest (drop low-dispersion, orange), and recomputed the vision–language local align- ment (CKNNA, k = 10) on the remaining 768 stimuli from WIT. The full-set baseline (full, gray) is shown for reference. Each of the four panels corresponds to one vision model (MAE, DINOv2, CLIP, CLIP+FT on ImageNet-12K), with alignment reported against 12 language models from three families: BLOOM (560m, 1b1, 1b7, 3b, 7b1), OpenLLaMA (3b, 7b, 13b), and LLaMA (7b, 13b, 30b, 65b). Across all vision-to-LLM pairings, excluding high-dispersion stimuli increases CKNNA above the full-set baseline, while excluding low-dispersion stimuli lowers it. 0.010.020.03 Dispersion (held-in 3 models) 0.02 0.04 0.06 Deviation (held-out model) r = 0.33 ρ = 0.28 Held-out: MAE-H 0.010.020.03 r = 0.21 ρ = 0.10 Held-out: DINOv2-G 0.010.020.03 r = 0.43 ρ = 0.29 Held-out: CLIP-H 0.0100.0150.020 r = 0.26 ρ = 0.20 Held-out: CLIP+ft-H Supplementary Figure S2: Dispersion computed from a subset of the vision models predicts the held-out model’s deviation from the consensus. For each of the four vision models in turn, GPA is re-run on the other three, producing a partial-dispersion score per stimulus (x-axis). The held- out model is then projected into the remaining consensus and its per-stimulus deviation from that consensus is measured (y-axis). Each point is one WIT stimulus (n = 1024); the red line is the linear fit. Pearson r is significantly positive for all four held-out models: MAE-H r = 0.33, p < 0.001; DINOv2-G r = 0.21, p < 0.001; CLIP-H r = 0.43, p < 0.001; CLIP+ft-H r = 0.26, p < 0.001 (ρ values show Spearman correlations). This suggests that stimulus-level dispersion captures a property shared across model families. 7