Paper deep dive
One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Models
Aditya Sharma, Divya Saxena
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/30/2026, 7:56:26 AM
Summary
This paper provides a unified geometric explanation for the divergent effects of modifying the modality gap in contrastive vision-language models (VLMs) like CLIP and SigLIP. The authors demonstrate that the image-text mean separation is approximately rank-one, captured by a single dominant direction. They identify three task-specific roles for this gap: in zero-shot classification, gap offset subtraction acts as an additive class bias; in standard cross-modal retrieval, projecting out the gap direction causes multiplicative ranking distortion due to discarded norm information, which can be corrected by a geometry-derived exponent; and in mixed-modal retrieval, the gap direction sorts candidates by modality, where its removal improves cross-modal ranking. The study concludes that the impact of gap modification is readout-dependent rather than inherently beneficial or harmful.
Entities (15)
Relation Signals (8)
Modality Gap → hasgeometry → Rank-one Geometry
confidence 95% · a single dominant direction captures 94.4-99.9% of the squared norm of the image-text mean separation, revealing that the mean-separation component is approximately rank-one.
CLIP → usedin → Zero-shot Classification
confidence 95% · Across CLIP and SigLIP encoders... In zero-shot classification
SigLIP → usedin → Cross-modal Retrieval
confidence 95% · Across CLIP and SigLIP encoders... In standard cross-modal retrieval
Modality Gap → affects → Mixed-modal Retrieval
confidence 92% · In mixed-modal retrieval, the gap direction sorts candidates by modality; its removal can improve cross-modal ranking
Modality Gap → affects → Zero-shot Classification
confidence 92% · In zero-shot classification, query-side fixed gap-offset subtraction is exactly equivalent to an additive class bias.
Modality Gap → affects → Cross-modal Retrieval
confidence 92% · In standard cross-modal retrieval, projecting out the gap direction and renormalising residuals discards candidate-specific norm information, inducing a multiplicative ranking distortion
Geometry-derived exponent → corrects → Multiplicative Ranking Distortion
confidence 90% · a geometry-derived exponent tracks the grid-search optimum (Spearman rho = 0.93) and restores performance in some settings
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Contrastive vision-language models learn shared embedding spaces by aligning matched image-text pairs, yet their representations remain separated by a modality gap. Prior work reports divergent effects of modifying this gap: reducing it can improve zero-shot classification and cross-modal alignment, whereas removing gap-related structure can degrade image-text retrieval. In this paper, we provide a unified geometric explanation for these task-dependent effects. Across CLIP and SigLIP encoders, we find that a single dominant direction captures 94.4-99.9% of the squared norm of the image-text mean separation, revealing that the mean-separation component is approximately rank-one. A decomposition of the similarity score then identifies three task-specific roles. In zero-shot classification, query-side fixed gap-offset subtraction is exactly equivalent to an additive class bias. In standard cross-modal retrieval, projecting out the gap direction and renormalising residuals discards candidate-specific norm information, inducing a multiplicative ranking distortion; a geometry-derived exponent tracks the grid-search optimum (Spearman rho = 0.93) and restores performance in some settings, although the gains transfer unevenly. In mixed-modal retrieval, the gap direction sorts candidates by modality; its removal can improve cross-modal ranking, unlike random or non-gap controls. Residual semantic structure after removal defines the limits of the rank-one account. Together, these results explain why gap modification can improve, degrade, or restore performance across downstream settings. By clarifying when and why gap modification changes model behavior, this account provides a principled basis for selecting gap interventions in similarity-based vision-language systems across evaluated downstream tasks.
Tags
Links
- Source: https://arxiv.org/abs/2609.36101v1
- Canonical: https://arxiv.org/abs/2609.36101v1
Trouble viewing inline? Open PDF directly →
Full Text
38,726 characters extracted from source content.
Expand or collapse full text
One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Models Aditya SharmaDivya Saxena Indian Institute of Technology Jodhpur p25ai0201@iitj.ac.in divyasaxena@iitj.ac.in Abstract Contrastive vision–language models learn shared embed- ding spaces by aligning matched image–text pairs, yet their representations remain separated by a modality gap. Prior work reports divergent effects of modifying this gap: reducing it can improve zero-shot classification and cross- modal alignment, whereas removing gap-related struc- ture can degrade image–text retrieval. In this paper, we provide a unified geometric explanation for these task- dependent effects. Across CLIP and SigLIP encoders, we find that a single dominant direction captures 94.4–99.9% of the squared norm of the image–text mean separation, revealing that the mean-separation component is approxi- mately rank-one. A decomposition of the similarity score then identifies three task-specific roles. In zero-shot classi- fication, query-side fixed gap-offset subtraction is exactly equivalent to an additive class bias. In standard cross- modal retrieval, projecting out the gap direction and renormalising residuals discards candidate-specific norm information, inducing a multiplicative ranking distortion; a geometry-derived exponent tracks the grid-search op- timum (Spearmanρ= 0.93) and restores performance in some settings, although the gains transfer unevenly. In mixed-modal retrieval, the gap direction sorts candi- dates by modality; its removal can improve cross-modal ranking, unlike random or non-gap controls. Residual semantic structure after removal defines the limits of the rank-one account. Together, these results explain why gap modification can improve, degrade, or restore perfor- mance across downstream settings. By clarifying when and why gap modification changes model behavior, this account provides a principled basis for selecting gap in- terventions in similarity-based vision–language systems across evaluated downstream tasks. 1 Introduction Contrastive vision-language models such as CLIP and SigLIP align matched image–text pairs in a shared repre- sentation space (Radford et al., 2021; Zhai et al., 2023) that supports zero-shot classification, cross-modal re- trieval, and multimodal search. Yet their image and text embeddings remain systematically separated, form- ing a persistent modality gap (Liang et al., 2022). Prior studies report conflicting outcomes: reducing this sepa- ration can improve selected cross-modal tasks, whereas stronger alignment or removal of modality-specific struc- ture can degrade classification or standard image–text retrieval. This disagreement arises partly because the gap is often treated as though it has a task-independent effect. Figure 1 summarizes how the same dominant mean-separation direction produces different effects under different operations and downstream readouts. Intervention effect relative to baseline: Harmful Beneficial 0-5-1005100.0-0.1-0.20.00.10.20.0-0.1-0.20.00.10.2 949596979899100 CLIP ViT-L/14, Flickr30k CLIP ViT-B/32, Flickr30k CLIP ViT-B/16, COCO SigLIP ViT-B/16, Flickr30k SigLIP ViT-L/16, Flickr30k All 23 cells (mean) Unchan ged Remov e v1 Remov e PC2 Remov e seeded random Full m ean-centering 0.00 0.05 0.10 0.15 0.20 0.25 NDCG@10 0.0002 0.216 0.0013 0.0002 0.2417 Unchanged Projected cosine Geometry-predicted 0 20 40 60 80 100 Mean Recall (%) 85.1 70.7 84.4 Direct intervention Additive-bias reconstruction 0 20 40 60 80 100 Prediction-change recovery (%) 100 93.84 Classification: Additive Class-Dependent Bias Mixed Retrieval: Modality-Dependent Candidate Sorting I) Mean Separation Concentrates Along One Stable Direction Squared mean separation captured (%) The leading direction captures 94.4%–99.9% of squared image–text mean separation. 94.4% 97.6% 99.5% 99.4% 99.8% 99.45% O1: Fixed Offset O2: Projection O3: Projection + Cosine O4: Full Mean Centering Classification Δ Top-1 Standard Retrieval Δ Mean-Recall Mixed Retrieval Δ NDCG@10 I) Same Geometry, Different Task Effects I) One Geometry, Three Task-Specific Effects Standard Retrieval: Candidate-Specific Scale Distortion . . Operations: Figure 1. One dominant mean-separation direction, readout-dependent outcomes. Panel I compares O1–O4 on task-specific effect scales over 15 classification, 16 standard- retrieval, and 16 matched mixed-retrieval configurations with five paired seeds. Panel I shows representative fractions of squared centroid separation aligned withv 1 . Panel I summarizes the classification, standard-retrieval, and mixed- retrieval mechanism checks. 1 arXiv:2609.36101v1 [cs.CV] 28 Sep 2026 Two distinctions resolve the apparent conflict. First, interventions are non-equivalent: fixed-offset subtraction, orthogonal projection, projection followed by L2 normal- ization, full modality-wise mean-centering, and score cal- ibration remove different information. Second, down- stream readouts hold different quantities fixed. Classifi- cation compares one image with varying text prototypes; standard cross-modal retrieval ranks opposite-modality candidates; mixed-modal retrieval lets images and text compete in one index. An explanation must therefore specify both the geometric operation and the readout. We show that image–text mean separation concentrates along a stable leading joint-cloud direction across CLIP and SigLIP encoders. Under fixed-query classification, this coordinate induces an additive class-dependent bias. After projection and normalization in standard retrieval, discarded candidate-specific residual-length information produces a multiplicative scale distortion. In a mixed index, it sorts candidates by modality. The same compo- nent can therefore help or harm depending on how the downstream system reads the space. Across 23 encoder–dataset cells, the leading direction captures 94.4–99.9% of the squared image–text mean sep- aration and remains stable across splits, seeds, datasets, and encoders. Analytical reconstruction, matched con- trols, and frozen estimate/evaluation protocols support the readout-dependent mechanisms. Held-out transfer pre- serves qualitative effects more reliably than quantitative magnitudes, while residual multidimensional structure defines the boundary of the account. Our contributions are as follows: •Dominant mean-separation geometry. We show that 94.4–99.9% of squared image–text centroid sep- aration aligns with one stable joint-cloud direction across 23 CLIP and SigLIP encoder–dataset cells, while matched non-gap axes do not reproduce its modality separation. •Readout-dependent mechanism. We derive a common score decomposition linking the same coor- dinate to additive class bias, candidate-specific scale distortion, and modality-dependent sorting under three downstream readouts. •Mechanism validation and scope. Analyt- ical reconstruction, matched direction controls, configuration-specific retrieval calibration, and a frozen held-out checkpoint establish the explanatory range of the first-order account, while residual seman- tic structure marks its boundary. 2 Related Work Prior work establishes that a modality gap exists, that modifying it can help or harm, and that commonly grouped interventions are not equivalent. What remains missing is a common geometric account of how the inter- vention and downstream readout jointly determine the observed outcome. We provide such an account. 2.1 Modality-Gap Geometry CLIP and SigLIP align paired samples without requir- ing the image and text marginals to coincide (Radford et al., 2021; Zhai et al., 2023). Persistent separation has been connected to initialization, contrastive optimization, uniformity, information imbalance, temperature, and neg- ative sampling (Liang et al., 2022; Fahim et al., 2024; Schrodi et al., 2025; Yaras et al., 2025). These stud- ies measure different objects: centroid displacement is a first moment, whereas separability also reflects covariance, anisotropy, and modality-specific neighborhoods. Few coordinates dominate separation while structure persists (Schrodi et al., 2025; Levi and Gilboa, 2025); intra-modal interpretations remain contested (Mistretta et al., 2025; Herzog and Wang, 2026). We study the dominant mean- separation component inside a frozen representation and distinguish it from the complete distribution. 2.2 Representation Interventions Methods described as gap removal implement different transformations. Offset subtraction translates embed- dings; projection deletes a coordinate; projection plus normalization also rescales residuals (Liang et al., 2022; Chowers et al., 2026). Full modality-wise mean-centering, including the strong GR-CLIP baseline, removes complete centroids (Li et al., 2025); dominant-component removal also appears in anisotropy correction (Mu and Viswanath, 2018). Training-based alignment changes broader struc- ture (Eslami and de Melo, 2025; Yamaguchi et al., 2025; Jiang et al., 2023), while score calibration changes logits rather than embeddings (Zhu et al., 2023). Evidence for one intervention therefore does not transfer automatically to another. 2.3 Downstream Consequences Classification, opposite-modality retrieval, and heterogeneous-index retrieval expose different candidate- varying quantities. Stronger alignment may help particular cross-modal retrieval settings while removing modality-specific information can harm classification or neighborhood quality (Liang et al., 2022; Jiang et al., 2023; Yaras et al., 2025; Yamaguchi et al., 2025). Mixed candidate pools additionally permit modality clusters to bias ranking, motivating mean-centering corrections (Li et al., 2025). Candidate composition and normalization can thus change the sign of an intervention without contradiction. 2 3 Readout-Dependent Modality-Gap Geometry We first define the dominant direction and distinguish four commonly conflated interventions. We then decom- pose the similarity score into modality-level, residual, and orthogonal components. The downstream role of each component depends on which quantities a task holds fixed and which it varies. 3.1 Problem Setting and Direction Definitions Letx,t ∈ R d denote image and text embeddings in a shared representation space. Embeddings are L2- normalized whenever the score is cosine similarity. With modality centroids μ img and μ txt , define g = μ img − μ txt ,bg = g ∥g∥ 2 .(1) The vectorbgidentifies the centroid-difference direction. We separately denote byv 1 the estimate-split leading principal direction of the joint image–text cloud. The experiments instantiate the intervention axis withv 1 ; its alignment with the independently estimatedbgtests whether this stable leading direction organizes the image– text mean separation. The score analysis holds for any unit directionv. For an embedding z, β z = z ⊤ v, z ⊥ = z− β z v, z = z ⊥ + β z v. (2) Thusz ⊥ retains the geometry orthogonal tov. This characterization concerns concentration of the mean- separation component alongv 1 , not the complete semantic or distributional geometry. 3.2 Four Non-Equivalent Interventions The phrase gap removal conflates transformations that preserve different information. Table 1 gives the concep- tual map before the formal definitions. Op. Mean on v Sample coord. Pre-L2 norm L2 again Main consequence O1 Yes NoYes No Fixed modality translation O2 Yes Yes Yes No Deletes axis contribution O3 Yes Yes Yes Yes Candidate-specific denominator O4 Yes NoYes Yes Full-centroid and cosine change Table 1. Non-equivalent embedding operations. “Sam- ple coord.” records deletion of the complete sample-specific coordinate onv; O4 instead removes the full modality centroid. Application side is specified separately. Letm ∈ img, txtand ̄ β m =E z∼m [z ⊤ v]. O1, fixed single-direction offset subtraction, is z ′ m = z m − ̄ β m v.(3) It removes the modality mean alongvbut retains sample residuals. O2, raw orthogonal projection, removes the complete sample coordinate, z ′ = z− (z ⊤ v)v = z ⊥ .(4) It changes the norm without restoring unit length. O3 projects and then L2-normalizes, z ′ = z ⊥ ∥z ⊥ ∥ 2 , ∥z ⊥ ∥ 2 = q 1− (z ⊤ v) 2 (5) for unitz, introducing a sample-dependent denominator. O4, full modality-wise mean-centering, is z ′ m = z m − E[z m ],ez ′ m = z ′ m ∥z ′ m ∥ 2 (6) when cosine is evaluated. O4 removes the full centroid and can modify directions orthogonal tov; it is not rank-one in general. Score calibration is distinct from embedding modifica- tion because it changes the downstream readout without changing stored embeddings. The task effect therefore depends on the operation, application side, and any sub- sequent normalization. 3.3 A Common Similarity-Score Decomposition For any unitv, decomposex=x ⊥ +β x vandt=t ⊥ +β t v. Their dot-product score satisfies S(x,t) = x ⊤ t = x ⊤ ⊥ t ⊥ + β x β t .(7) Write β x = ̄ β img + r x and β t = ̄ β txt + r t . Then S(x,t) = S ⊥ + A + B + C + D,(8) S ⊥ = x ⊤ ⊥ t ⊥ , A = ̄ β img ̄ β txt ,B = ̄ β img r t , C = ̄ β txt r x ,D = r x r t . A is a modality-level constant;BandCare first-order text- and image-residual terms;Dcouples the sample coordinates; andS ⊥ retains similarity outsidev. These algebraic labels are modality-based. In image-to-text retrieval,Bis candidate-varying andCis query-side; in text-to-image retrieval, their functional roles exchange. Aggregated retrieval results therefore retain the algebraic B/Clabels but interpret candidate and query roles by direction. O3 adds a distinct source of score change: N x = p 1− β 2 x , N t = q 1− β 2 t , S proj-cos (x,t) = x ⊤ ⊥ t ⊥ N x N t .(9) 3 The numerator decomposition and candidate-dependent normalization are separate. The decomposition is exact, but functional sufficiency is empirical: near-tied scores can retain nearly identical numerical values while exchanging strict order. 3.4 How Different Tasks Read the Same Geometry Classification varies class prototypes, standard retrieval varies opposite-modality candidates, and mixed retrieval varies both candidate identity and modality, producing additive bias, scale distortion, and modality sorting, re- spectively. Classification: additive class-dependent bias. Classification fixes an image query and varies text proto- typest c . Under the implemented query-only O1 interven- tion with v = v 1 , (x− ̄ β img v 1 ) ⊤ t c = x ⊤ t c − ̄ β img β t c .(10) The second term is an additive class-dependent bias. This exact identity is specific to the fixed-offset construction and does not give O2–O4 the same interpretation. Standard retrieval: candidate-specific scale distor- tion. A fixed query ranks opposite-modality candidates. After O3, each candidate has a different residual norm, producing a multiplicative calibration error. The imple- mented norm-power readout is S γ (q,c) = q ⊤ ⊥ c ⊥ N q N 1−γ c .(11) For image-to-text retrieval, the geometry-derived first- order exponent is γ (1) t = 1 + | ̄ β img | | ̄ β txt | ̄ S + ⊥,cos ,(12) where ̄ S + ⊥,cos is the estimate-split mean cosine between matched image–text residuals after projection and L2 normalization; the superscript + denotes positive pairs. The expression is a simplified first-order approximation, with image and text roles exchanged for text-to-image. It predicts config-specific calibration rather than an exact optimum. Mixed retrieval: modality-dependent candidate sorting. Image and text candidates share one index, so candidate modality varies within a ranking. Terms constant in a standard opposite-modality pool can then differ systematically across candidates, allowing thev 1 coordinate to sort by modality before semantic relevance. One geometric account thus links four non-equivalent interventions and three readout mechanisms, with no task- independent sign. The modality gap is neither inherently beneficial nor harmful; its effect depends jointly on the operation and downstream readout. 4 Experimental Setup 4.1 Models, Tasks, and Evaluation Protocol Every claim uses a frozen-geometry protocol: directions, coefficients, and thresholds are estimated on one split and applied unchanged to a disjoint evaluation split across five paired seeds. No evaluation outcome is used to choose a direction, coefficient, or threshold. We use frozen final-layer embeddings from OpenAI CLIP ViT-B/32, ViT-B/16, and ViT-L/14 and Google SigLIP Base-P/16-224 and Large-P/16-256. Classifica- tion uses the two CLIP base encoders and SigLIP Base on CIFAR-10, CIFAR-100, ImageNet-1K, EuroSAT, and SVHN. Standard cross-modal retrieval uses Flickr30k with all five encoders and COCO-5K with the three base en- coders, in both image-to-text and text-to-image direc- tions. Binary mixed retrieval places image and text can- didates from Flickr30k or COCO-5K in one shared index at image:text ratios of 25:75, 50:50, and 75:25. The matched O1–O4 cross-task comparison uses 16 mixed- retrieval configurations; the primary mechanism analysis uses 48 binary shared-index cells spanning eight encoder– dataset pairs, both query modalities, and three pool ratios. MixBench25/MSCOCO is evaluated separately because its documents may contain image, text, or both. The held-out check uses OpenCLIP ConvNeXt-Base-W, whose architecture and training configuration are absent from development. Its geometry-only predictions are frozen before downstream observations are loaded. Each task uses deterministic 50:50 estimate/evaluation partitions. Classification splits are stratified; retrieval partitions image identities and retains all associated cap- tions; MixBench separates connected components of the relevance graph. The estimate split supplies centroids, coordinate means, residual-norm statistics,v 1 , and fitted calibration references. The independently estimated cen- troid directionbgonly measures alignment withv 1 . Input embeddings are L2-normalized. Only operations that ex- plicitly include post-intervention normalization normalize the embeddings again after transformation. Primary met- rics are Top-1 accuracy, Mean Recall over Recall@1/5/10, and NDCG@10 for classification, standard retrieval, and mixed retrieval. Seed means and paired effects use 95% Student-tintervals; exponent correlations use a configura- tion bootstrap. The five-seed intervals quantify variability across the paired estimate/evaluation splits within the evaluated cells; they are not population intervals over encoder or dataset families. 4.2 Interventions and Core Controls We compare unchanged embeddings with O1 fixed off- set subtraction, O2 raw projection, O3 projection fol- lowed by L2 normalization, and O4 full modality-wise mean-centering. The intervention direction for O1–O3 is v 1 . The primary classification identity applies O1 to the 4 image query; standard retrieval applies projection and normalization to both modalities; binary mixed retrieval applies each embedding operation to all indexed modal- ities. Matched PC2 and seeded random directions test whether an effect is specific to the dominant centroid- separation direction. For standard retrieval, the geometry-derived first-order exponent and estimate-split fitted exponent are frozen before evaluation. A shared constant tests whether ag- gregate recovery alone is sufficient. These methods recal- ibrate scores and do not alter stored embeddings. The held-out protocol also freezes geometry-only predictions before downstream observations, separating qualitative transfer from post-hoc quantitative fit. 5 From Gap Geometry to Task-Dependent Roles 5.1 Mean Separation Concentrates Along One Stable Direction The image–text centroid separation is, to a close approxi- mation, concentrated along one stable direction. Across 23 encoder–dataset cells, the estimate-splitv 1 captured 94.4–99.9% of the squared image–text mean separation on the disjoint evaluation split. Its estimate/evaluation cosine ranged from 0.9980 to above 0.9999, and|bg ⊤ v 1 | ranged from 0.9718 to above 0.9999. Figure 1 includes the weakest cell and representative CLIP and SigLIP cells. Modality AUC was 1.0 alongv 1 , compared with 0.437–0.781 for PC2 and 0.500 on average over 460 seeded random axes. Cross-dataset stability held in 62 of 63 com- parisons, with one lower-bound miss. This approximately rank-one characterization concerns the mean-separation component, not the complete multimodal distribution. 5.2 Non-Equivalent Interventions Produce Different Outcomes Panel I of Figure 1 compares O1–O4 on matched develop- ment grids. O1 and O2 reduced classification Top-1 by 2.06 and 2.13 points and standard-retrieval Mean Recall by 0.069 and 0.071, while increasing mixed NDCG@10 by 0.132 and 0.135. O3 applies the same coordinate deletion as O2 and then normalizes each residual; its losses were larger in classification (9.14 points) and standard retrieval (0.143), while mixed NDCG still increased by 0.122. O4 improved classification by 2.47 points and mixed NDCG by 0.169, with a standard-retrieval change of−0.014. These mixed values use the matched 16-configuration operation grid and are distinct from the 48-cell binary analysis in Section 5.6. O1 removes a modality-level offset while retaining sam- ple variation alongv 1 ; O2 removes the full sample coordi- nate; O3 adds sample-dependent normalization; and O4 removes the full centroid. O1 and O2 nevertheless leave nearly identical mean residual gaps (0.05821 and 0.05817). Similar gap reduction therefore does not imply equivalent score or task behavior. These geometric modifications can improve one readout while degrading a different one. Remaining centroid dis- tance alone cannot determine whether gap modification is beneficial. 5.3 Readout-Dependent Score Reconstruction Equation(8)reconstructed every tested score within nu- merical tolerance, but partial terms preserved decisions differently across readouts (Figure 2). For classification,S ⊥ +Brecovered 0.938 of prediction changes andS ⊥ +B+Dexceeded 0.9999. Standard- retrieval ordering recovery rose from 0.835 withS ⊥ to 0.979 after addingB, while the projected-cosine denomina- tor remained a separate source of change. Mixed retrieval was least compressible:S ⊥ +B+Drecovered 0.873. High average recovery does not guarantee all-seed agreement because near-tied predictions or rankings can change un- der small numerical differences. The decomposition is exact, but functional sufficiency remains task-dependent. 5.4 Classification: Additive Class-Dependent Bias Under query-only O1, Eq.(10)gives the complete class- dependent score shift exactly. The recovery percentages below answer a different question: whether selected de- composition terms reproduce strict predictions. The fixed offset changed 27.07% of predictions and reduced cell- averaged Top-1 from 60.71% to 58.65%. This net 2.06- point reduction conceals substantial bidirectional churn: 7.82% of examples change from correct to incorrect, while 5.76% change from incorrect to correct. The additive component is therefore not uniformly harmful; its effect depends on alignment with the class prototypes. Term Brecovered 93.84% of prediction changes (95% CI 93.68– 94.00%); addingDraised recovery above 99.99%, with maximum score error 7.4×10 −7 . A matched PC2 rein- jection recovered 0.626 of the score response versus 1.000 for the measured direction. Under the fixed-query O1 construction, the dominant coordinate acts as an addi- tive class-dependent bias; the identity does not extend unchanged to O2–O4. 5.5 Standard Retrieval: Candidate-Specific Scale Distortion Projection followed by normalization reduced Mean Recall from 0.8506 to 0.7073 across 16 encoder–dataset–direction configurations. The geometry-derived exponent raised it to 0.8441, recovering 95.4% of the lost performance (Figure 3). 5 Figure 2. Score-term sufficiency is readout-dependent. Cells show five-seed mean prediction or ordering recovery, rather than numerical score error. The algebraicB/Clabels are image/text-based:Bis candidate-varying for image-to-text retrieval andCfor text-to-image, so the retrieval row aggregates direction-specific functional roles. The full score gives exact numerical reconstruction within tolerance; near ties can still change strict decisions. 2.53.03.54.04.5 Estimate-split fitted exponent 2.5 3.0 3.5 4.0 4.5 Geometry-predicted exponent Spearman ρ= 0.93 MAE = 0.23 A. Exponent prediction CLIP, flickr30k CLIP, coco SigLIP, flickr30k SigLIP, coco Identity UnchangedProjected cosine Geometry exponent Fitted exponent Shared constant 0.700 0.725 0.750 0.775 0.800 0.825 0.850 0.875 Mean Recall 0.851 0.707 0.844 0.844 0.841 B. Paired retrieval performance Figure 3. Geometry predicts configuration-specific scale calibration. A: predicted versus estimate-split fitted exponents for 16 configurations; Spearmanρ= 0.930 with a 2,000-resample bootstrap interval. B: evaluation Mean Recall over five paired seeds; error bars show 95% Student-tintervals. The fitted exponent is diagnostic and the shared constant is configuration-invariant. The predicted exponent tracked the fitted value with Spearmanρ= 0.930 (95% CI 0.730–0.991) and MAE 0.230. A shared constant reached 0.8407 Mean Recall, but its exponent MAE was 0.486; geometry was closer to the fitted value in 12 of 16 configurations. A shared constant therefore recovers much of the average loss, but cannot predict which configurations require stronger or weaker correction. Transfer remained uneven, and the full first-order predictor had higher error (MAE 0.534). The exponent is therefore not a universal cross-dataset calibra- tor, but a configuration-sensitive first-order explanation of why projection followed by normalization distorts can- didate rankings. 5.6 Mixed Retrieval: Modality-Dependent Candidate Sorting The primary mixed result uses 48 binary shared-index cells: eight encoder–dataset pairs, both query modalities, and three image:text pool ratios. Each cell uses five paired seeds. Removingv 1 transformed shared-index retrieval: NDCG@10 rose from 0.0002 to 0.2160, while pool-adjusted modality skew fell from 0.9999 to 0.7952. Relevance is as- signed to paired opposite-modality items while image and text candidates share the pool; unchanged scores rank same-modality candidates ahead of the relevant cross- modal items, producing the near-zero baseline. Removing PC2 or seeded random directions left performance near 6 Method Binary NDCG@10 ↑ Pool-adj. skew ↓ Within-mod. ρ↑ Standard ret. ∆MR Unchanged0.00020.99991.0000.000 O2: remove v 1 0.2160 0.79520.896 −0.071 Remove PC20.00130.99930.872— Remove seeded random 0.00020.99991.000— O4: full centering 0.24170.81750.796 −0.014 Table 2. Binary shared-index retrieval is direction- specific. Means over 48 cells across eight Flickr30k/COCO- 5K encoder–dataset pairs, both query modalities, and im- age:text ratios 25:75, 50:50, and 75:25; five paired seeds per cell; MixBench is excluded. Near-zero unchanged NDCG accompanies severe modality-dominated ranking. Standard- retrieval ∆MR is from a separate matched 16-configuration evaluation. the unchanged system, whereas O4 reached 0.2417. The dominant direction therefore acts as a strong modality- sorting component within the shared index. Table 2 compares the dominant-direction intervention with matched non-gap controls and full centering. O2 preserved more within-modality order (ρ= 0.896) than O4 (ρ= 0.796), but cost 0.071 standard-retrieval Mean Recall versus 0.014 for O4 on a separate matched standard-retrieval grid. This contrast clarifies that the binary shared-index gains do not come for free: rank-one removal better preserves within-modality structure, but it also produces a larger trade-off on the conventional opposite-modality retrieval task. Thus, the same interven- tion can substantially improve cross-modal competition in a shared index while harming standard retrieval, and the preferred operation depends on which readout is ulti- mately being used. MixBench is a separate text-query construction with image, text, and multimodal documents, so it comple- ments rather than duplicates the binary shared-index analysis. Its NDCG@10 was 0.4543 unchanged, 0.6612 afterv 1 removal, and 0.6892 after full centering; content- type filtering control reached only 0.2996. The primary ranking criterion improved under both interventions, but retrieval-score modality AUC moved farther from chance after centering. Together with the binary controls, these results support the interpretation that the dominant di- rection acts as a modality-dependent sorting component, while stopping short of claiming complete restoration of mixed-modal retrieval. 6 Scope and Boundary of the Rank-One Account A mechanistic account is useful only when its domain of validity is explicit. We examine three boundaries: direction specificity, architectural transfer, and residual structure. CheckEvidence A. Held-out checkpoint Scalar effect directions7/7 Operation-effect signs10/12 Exact task ordering1/3 Frozen interval coverage1/7 Scalar / operation-effect MAE0.257 / 0.092 B. Residual boundary Centroid removed; distribution persists21/23 Residual subspace multidimensional23/23 Stable non-modality retrieval direction8/8 Classification median / random p950.0707 / 0.0015 Retrieval median / random p950.0083 / 0.0012 Table 3. Transfer and boundary. Frozen held-out prediction (A) and residual structure after centering (B). 6.1 Direction Specificity Exact unit-strength reinjection ofv 1 with its original coordinates is an expected reconstruction check. The informative tests change the direction, coefficients, an- gle, or strength while preserving the intervention budget. Across matched wrong-direction and coefficient controls, 96.3% satisfied the predefined specificity criterion, al- though generic low-dimensional reinjection also recovered part of the score variation. Thus, direction and coefficient identity both matter. 6.2 Held-Out OpenCLIP ConvNeXt Results For a held-out OpenCLIP ConvNeXt-Base-W checkpoint, geometry-only predictions were frozen before downstream evaluation. Qualitative transfer was strong: all seven scalar effect directions and 10 of 12 operation-effect signs were predicted correctly on an architecture and training configuration absent from development. Quantitative transfer was weaker: only one frozen interval covered its observation, and complete operation ordering transferred in one of three tasks (Table 3). The first-moment account therefore captures qualitative mechanism more reliably than architecture-specific magnitude. 6.3 Residual Structure Beyond the Rank-One Account After full mean-centering, distributional modality sepa- ration persisted in 21 of 23 tested cells, while residual PCA was multidimensional in all 23. Orthogonal semantic directions had median absolute effects of 0.0707 versus a random-direction 95th percentile of 0.0015 in classification and 0.0083 versus 0.0012 in retrieval; stable retrieval di- rections exceeded random controls in all eight cells. These residual directions remain task-relevant while exhibiting near-chance modality discrimination. They therefore rep- 7 resent semantic structure orthogonal to the dominant modality-separation component, rather than the original modality gap reappearing. This distinction separates re- moval of the dominant first-moment offset from collapse of the broader representation geometry. Mean-centering can eliminate the principal modality-separating compo- nent while leaving structured variation that still affects downstream decisions. The dominant mean-separation component is important but does not exhaust multimodal geometry. 7 Discussion and Limitations The results reconcile conflicting reports because inter- ventions described as gap modification preserve differ- ent information, while candidate composition determines which changed terms affect a downstream readout. Fu- ture studies should report: (1) the intervention direction, (2) application side, (3) post-intervention normalization, (4) candidate-modality composition, and (5) downstream readout. Full centering is a strong practical baseline but also modifies orthogonal directions; score calibra- tion changes the readout rather than stored embeddings. Conceptually, the modality gap is neither uniformly a defect nor uniformly useful. Its functional role emerges from the interaction between representation geometry and the readout that consumes it. Consequently, no task-independent operation is universally preferable: pro- jection isolates direction-specific effects, projection plus normalization exposes candidate-scale distortion, and full centering remains a strong practical intervention when mixed-modality competition is central. Our scope is frozen CLIP and SigLIP dual encoders, final-layer embeddings, and similarity-based tasks. The retrieval approximation transfers unevenly, held-out signs transfer better than magnitudes, and residual multidi- mensional structure remains after centering. The account does not extend directly to fused or generative models and does not replace higher-order analysis. Within this scope, the dominant mean-separation component provides a coherent first-order account of why gap modification helps or harms different readouts. 8 Conclusion The modality gap has no single task-independent conse- quence or remedy. Its dominant mean-separation compo- nent is approximately rank-one, but its downstream effect is determined by the interaction between the intervention and the readout. Practitioners should therefore diagnose the readout before choosing an intervention and report both when evaluating gap modification. Our results sug- gest that geometry-aware intervention design should guide future development of vision-language models. References Rhea Chowers, Oshri Naparstek, Udi Barzelay, and Yair Weiss. Is the modality gap a bug or a feature? a robustness perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 30288–30298, 2026. URL https://openaccess.thecvf.com/content/CVPR2026/ papers/Chowers_Is_the_Modality_Gap_a_Bug_ or_a_Feature_A_CVPR_2026_paper.pdf. Sedigheh Eslami and Gerard de Melo. Mitigate the gap: Improving cross-modal alignment in CLIP. In Interna- tional Conference on Learning Representations, 2025. URL https://proceedings.iclr.c/paper_files/paper/ 2025/hash/c1de06a58ba1db43538a37e076e466d- Abstract-Conference.html. Abrar Fahim, Alex Murphy, and Alona Fyshe. It’s not a modality gap: Characterizing and addressing the contrastive gap. arXiv preprint arXiv:2405.18570, 2024. doi: 10.48550/arXiv.2405.18570. URL https://arxiv. org/abs/2405.18570. Jonas Herzog and Yue Wang. Reevaluating the intra-modal misalignment hypothesis in CLIP. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24310–24319, 2026. URL https://openaccess.thecvf.com/content/ CVPR2026/html/Herzog_Reevaluating_the_Intra- Modal_Misalignment_Hypothesis_in_CLIP_ CVPR_2026_paper.html. Qian Jiang, Changyou Chen, Han Zhao, Liqun Chen, Qing Ping, Son Dinh Tran, Yi Xu, Belinda Zeng, and Trishul Chilimbi. Understanding and constructing latent modal- ity structures in multi-modal representation learning. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 7661–7671, 2023. doi: 10.1109/CVPR52729.2023.00740. URL https://openaccess.thecvf.com/content/CVPR2023/ html/Jiang_Understanding_and_Constructing_ Latent_Modality_Structures_in_Multi-Modal_ Representation_Learning_CVPR_2023_paper.html. Meir Yossef Levi and Guy Gilboa. The double-ellipsoid geometry of CLIP. In Proceedings of the 42nd Inter- national Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 33999–34019. PMLR, 2025. URL https://proceedings. mlr.press/v267/levi25b.html. Binxu Li, Yuhui Zhang, Xiaohan Wang, Weixin Liang, Ludwig Schmidt, and Serena Yeung-Levy. Closing the modality gap for mixed modality search. In NeurIPS 2025 Workshop on Socially Responsible and Trustworthy Foundation Models, 2025. URL https://neurips.c/ virtual/2025/workshop/127834. 8 Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Ser- ena Yeung, and James Y Zou. Mind the gap: Under- standing the modality gap in multi-modal contrastive representation learning. In Advances in Neural Informa- tion Processing Systems (NeurIPS), volume 35, pages 17612–17625, 2022. Marco Mistretta, Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Andrew D. Bagdanov. Cross the gap: Exposing the intra-modal misalignment in CLIP via modality inversion. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.c/paper_files/paper/2025/ hash/c15da559b33c0f08a70573804cd5fb22-Abstract- Conference.html. Jiaqi Mu and Pramod Viswanath. All-but-the-top: Simple and effective postprocessing for word representations. In International Conference on Learning Representations (ICLR), 2018. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pages 8748–8763, 2021. Simon Schrodi, David T Hoffmann, Max Argus, Volker Fischer, and Thomas Brox. Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision-language models. In International Conference on Learning Representations (ICLR), 2025. Shin’ya Yamaguchi, Dewei Feng, Sekitoshi Kanai, Kazuki Adachi, and Daiki Chijiwa. Post-pre-training for modal- ity alignment in vision-language foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4256–4266, 2025. URL https://openaccess.thecvf.com/content/ CVPR2025/html/Yamaguchi_Post-pre-training_ for_Modality_Alignment_in_Vision-Language_ Foundation_Models_CVPR_2025_paper.html. Can Yaras, Siyi Chen, Peng Wang, and Qing Qu. Ex- plaining and mitigating the modality gap in contrastive multimodal learning. In Conference on Parsimony and Learning, volume 280 of Proceedings of Machine Learn- ing Research, pages 1365–1387. PMLR, 2025. URL https://proceedings.mlr.press/v280/yaras25a.html. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- training. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 11975–11986, 2023. Beier Zhu, Kaihua Tang, Qianru Sun, and Hanwang Zhang. Generalized logit adjustment: Calibrating fine- tuned models by removing label bias in foundation models. In Advances in Neural Information Processing Systems (NeurIPS), 2023. 9