Paper deep dive
Bigger Text Encoders Can Hurt CLIP Zero-Shot Performance
Samir Char, Carles Domingo-Enrich, Randall Balestriero
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/9/2026, 4:49:46 AM
Summary
This paper investigates the impact of asymmetric scaling of vision and text encoders in Contrastive Language-Image Pretraining (CLIP). The authors find that increasing text encoder size beyond an optimal point degrades zero-shot performance due to overfitting, despite total parameter count increasing. This degradation can be mitigated by using modality-specific weight decay coefficients. The study identifies efficient configurations that match the performance of standard ViT-B/16 with significantly fewer parameters.
Entities (9)
Relation Signals (8)
CLIP → hascomponent → Vision Encoder
confidence 95% · CLIP trains paired vision and text encoders
CLIP → hascomponent → Text Encoder
confidence 95% · CLIP trains paired vision and text encoders
Oversized Text Encoder → causes → Overfitting
confidence 92% · this degradation stems from overfitting induced by the oversized text encoder
Text Encoder Size → affects → Zero-Shot Performance
confidence 90% · for most vision encoders, there is an optimal text encoder size beyond which zero-shot performance degrades
Modality-Specific Weight Decay → mitigates → Overfitting
confidence 89% · using modality-specific weight decay coefficients not only recovers but improves performance
ViT-B/16 → baselinefor → Efficient Configurations
confidence 88% · efficient configurations that match the zero-shot performance of the standard ViT-B/16 architecture
Text Encoder Scaling → improves → Embedding Uniformity
confidence 85% · scaling the text encoder improves embedding uniformity
Text Encoder Scaling → worsens → Cross-Modal Alignment
confidence 85% · scaling the text encoder ... worsens cross-modal alignment
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Contrastive Language-Image Pretraining (CLIP) is a building block of many machine learning applications. Scaling laws have guided resource allocation for large-scale training, yet prior work treats total CLIP model size as a single variable, without exploring how the capacity split between encoders impacts downstream performance. Here, we train multiple CLIP models with different vision and text encoder sizes, revealing that for most vision encoders, there is an optimal text encoder size beyond which zero-shot performance degrades---even as total parameter count increases. Exploiting this behavior yields efficient configurations that match the zero-shot performance of the standard ViT-B/16 architecture with up to 55% fewer parameters. We further show that this degradation stems from overfitting induced by the oversized text encoder, and that using modality-specific weight decay coefficients not only recovers but improves performance across all degraded configurations. A geometric analysis reveals a trade-off in which scaling the text encoder improves embedding uniformity but worsens cross-modal alignment; we further show that these metrics are predictive of zero-shot performance. We hope these findings motivate CLIP architectures and training methods that counteract this degradation, a prerequisite for scaling CLIP reliably and efficiently.
Tags
Links
- Source: https://arxiv.org/abs/2609.05730v1
- Canonical: https://arxiv.org/abs/2609.05730v1
Trouble viewing inline? Open PDF directly →
Full Text
100,163 characters extracted from source content.
Expand or collapse full text
Bigger Text Encoders Can Hurt CLIP Zero-Shot Performance Samir Char ∗ samirchar@microsoft.com Microsoft Carles Domingo-Enrichcarlesd@microsoft.com Microsoft Research Randall Balestrierorandall_balestriero@brown.edu Brown University Abstract Contrastive Language-Image Pretraining (CLIP) is a building block of many machine learning applications. Scaling laws have guided resource allocation for large-scale training, yet prior work treats total CLIP model size as a single variable, without exploring how the capacity split between encoders impacts downstream performance. Here, we train multiple CLIP models with different vision and text encoder sizes, revealing that for most vision encoders, there is an optimal text encoder size beyond which zero-shot performance degrades— even as total parameter count increases. Exploiting this behavior yields efficient configurations that match the zero-shot performance of the standard ViT-B/16 architecture with up to 55% fewer parameters. We further show that this degradation stems from overfitting induced by the oversized text encoder, and that using modality-specific weight decay coefficients not only recovers but improves performance across all degraded configurations. A geometric analysis reveals a trade-off in which scaling the text encoder improves embedding uniformity but worsens cross-modal alignment; we further show that these metrics are predictive of zero-shot performance. We hope these findings motivate CLIP architectures and training methods that counteract this degradation, a prerequisite for scaling CLIP reliably and efficiently. 1 Introduction Contrastive Language-Image Pretraining (CLIP) (Radford et al., 2021) trains paired vision and text encoders with a contrastive objective on large-scale image–text data, yielding representations with remarkable zero-shot transfer. Since its introduction, CLIP has become a standard building block for a wide range of applications, including image generation (Saharia et al., 2022; Ramesh et al., 2022; Esser et al., 2024), generative vision- language models (Liu et al., 2023; Awadalla et al., 2023), video understanding (Xu et al., 2021; Wang et al., 2024), and image segmentation (Kirillov et al., 2023; Ravi et al., 2025). As these downstream systems inherit CLIP encoders or use their embeddings, the question of how to design and scale CLIP architectures has broad practical consequences. A common strategy for improving CLIP performance is to scale up the model, yet the allocation of capacity between the vision and text encoders remains an implicit, unexamined design choice. Most work training CLIP-style models at multiple scales presents a menu of architectures without specifying a scaling recipe or justifying the capacity split between encoders (Fig. 1(a)) (Radford et al., 2021; Zhai et al., 2023; Tschannen et al., 2025; Cherti et al., 2023a; Gadre et al., 2023; Xu et al., 2024; Chuang et al., 2025). Existing efforts to study CLIP scaling laws treat total model size as a single variable (Cherti et al., 2023a; Li et al., 2023a;b), without explicitly testing whether one encoder should be scaled more aggressively than the other. Yet the two modalities are inherently asymmetric: text is a compact, abstract signal whereas images are high-dimensional and redundant, and prior multimodal learning literature shows that modalities can overfit and generalize ∗ Corresponding author. Code: https://github.com/samirchar/clip-asymmetry. 1 arXiv:2609.05730v1 [cs.CV] 4 Sep 2026 at different rates (Wang et al., 2020). Scaling both encoders equally may not merely be sub-optimal but actively harmful. To our knowledge, no prior work independently varies each encoder’s capacity to isolate its marginal contribution to downstream performance, its impact on overfitting, or its effect on the geometry of the learned embedding space. In this work, we conduct a controlled study of how independently scaling CLIP’s vision and text encoders affects zero-shot and linear probe performance. We train a full grid of 30 models, crossing five vision encoders (from 10M to 352M parameters) with six text encoders (from 3M to 145M parameters) on C3M and C12M, and evaluate each across 40 downstream datasets (Fig. 1(b,c)). Our experiments yield three main contributions: 1.We find that for zero-shot tasks CLIP’s vision and text encoders should not be scaled equally. For most vision encoders there is an optimal text encoder size beyond which performance degrades (solid curve, Fig. 1(d)). This pattern is absent in the linear probe setting, where larger text encoders consistently improve or, at worst, saturate performance. Exploiting this asymmetry, we identify two architectures that match the zero-shot performance of the standard ViT-B/16 architecture while using 30–55% fewer total parameters. 2.We show that this degradation stems from a capacity mismatch between the encoders that manifests as asymmetric overfitting from an oversized text encoder. Using modality-specific weight decay to control each encoder’s effective capacity, we find that, for a given vision encoder, increasing the text encoder’s weight decay recovers—and even improves—zero-shot performance, surpassing the best architecture with the same vision encoder (dashed curve, Fig. 1(d)). Raising only the vision encoder’s weight decay instead degrades performance, while the conventional practice of decaying both encoders equally underperforms decaying the text encoder alone in most scenarios, or performs comparably. 3.A geometric analysis of the representations through the lens of alignment and uniformity (Wang & Isola, 2020) shows that scaling the text encoder improves the uniformity of image and text embeddings while degrading image-text alignment. Raising the text encoder’s weight decay mitigates this effect, recovering alignment while preserving or even improving image uniformity. 2 Related Work Evidence for asymmetric encoder scaling. Prior work provides indirect evidence that the capacity split between CLIP’s encoders matters. Fan et al. (2023) trains CLIP at four model sizes by scaling only the vision encoder while keeping the text encoder constant, reporting improved performance over the original scaling recipe. EVA-CLIP-18B (Sun et al., 2024) shows remarkable performance pairing a 17.5B-parameter vision encoder with a 695M-parameter text encoder. Zhai et al. (2022) show that freezing the vision encoder entirely and training only the text side can yield strong zero-shot performance. Closest to our work, SmolVLM (Marafioti et al., 2025) runs a 2D grid search over vision and language backbone sizes for small VLMs, finding that the capacity allocation between the vision encoder and text decoder is critical for compute-efficient performance. However, SmolVLM studies decoder-based vision-language models rather than contrastive pre-training. None of these works systematically isolates the effect of each encoder’s size in a controlled contrastive pre-training setting. Our work addresses this gap. CLIP scaling laws. Inspired by language model scaling laws (Hoffmann et al., 2022), Cherti et al. (2023a) demonstrated that CLIP’s zero-shot performance follows a power-law relationship with model size, dataset size, and samples seen. Similarly, CLIPA (Li et al., 2023a) and CLIPAv2 (Li et al., 2023b) identify an inverse scaling law for CLIP training: using larger vision and text encoders enables training with aggressive token masking ratios, reducing compute while maintaining competitive performance. However, these studies treat the vision and text encoders as a single pool of parameters and do not decompose scaling effects between them. Our work complements these efforts by studying how scaling each encoder independently affects downstream performance, but we do not derive specific scaling laws. 2 6 text encoder sizes Vision parameters (M) T 1 T 2 T 3 T 4 I 1· T 1 I 1· T 2 I 1· T 3 I 1· T 4 5 vision encoder sizes Measure embedding geometry Evaluate across diverse zero-shot and linear probe tasks Oversized text encoders degrade performance Modality-specific weight decay recovers performance 30 CLIP architectures CLIP models landscape Underexplored configurations Underexplored configurations Text parameters (M) 10 1 10 2 10 3 10 4 10 1 10 0 10 2 10 3 10 4 Zero-shot perf. Text encoder size Uniform weight decay Modality-Specific weight decay a) b) c) d) Text λ t Vision λ v I 1 OpenAI OpenCLIP MobileCLIP MetaCLIP EVA-CLIP LaCLIP MetaCLIP2 SigLIP TinyCLIP Figure 1: Naively scaling CLIP can degrade downstream performance. (a) Popular CLIP models by vision (x-axis) and text encoder (y-axis) size (n=45; points jittered). Most models cluster just below the diagonal, leaving most vision-text capacity splits unexplored. (b) A grid of 30 architectures spanning five vision and six text encoder sizes, each pre-trained on C3M and C12M. (c) Every configuration is evaluated across 30 zero-shot and 31 linear-probe tasks, and its embedding geometry is quantified via alignment, uniformity, modality gap, and RankMe. (d) Counterintuitively, under standard training (solid) zero-shot performance rises, peaks and then degrades as the text encoder grows with the vision encoder fixed; modality-specific weight decay (dashed) removes the degradation by applying separate coefficientsλ v andλ t to the encoders. Heterogeneous weight decay in deep learning. Different parts of a network may overfit at different rates, and should therefore be regularized differently. To address this, prior work adjusts weight decay over time, per layer, per module, and even per parameter (Ghiasi et al., 2023; Ishii & Sato, 2017; He et al., 2025; Nakamura & Hong, 2019). Notable examples are AdaDecay (Nakamura & Hong, 2019), which scales the decay per parameter from gradient norms, and AlphaDecay (He et al., 2025), which sets module-level decay from the heavy-tailedness of the weight spectral density. For CLIP and its variants, researchers have explored asymmetric per-encoder weight decay when one encoder is initialized from pretrained weights (Zhai et al., 2023; 2022; Sun et al., 2023). Such an encoder is either kept frozen or trained with a smaller or zero weight decay to mitigate catastrophic forgetting. Our work differs in that we vary weight decay across modalities while training both encoders from scratch (i.e., random initialization), using it as a deliberate capacity control to mitigate modality-specific overfitting and improve zero-shot performance. Embedding geometry in contrastive learning. The quality of contrastive representations has been characterized through alignment and uniformity (Wang & Isola, 2020): alignment measures the closeness of embeddings from positive pairs, while uniformity measures how well representations are spread across the unit hypersphere. Both properties are necessary for good contrastive representations, yet there is an inherent tension between them. Separately, RankMe (Garrido et al., 2023) provides a label-free estimate of the effective dimensionality of embeddings that correlates with downstream transfer performance. We use alignment, uniformity, and RankMe as diagnostic tools to explain the scaling behaviors observed in our experiments; to our knowledge, no prior work has studied how independently scaling the vision and text encoders affects the geometry of the embedding space. 3 Methodology 3.1 Asymmetric architecture grid To study how encoder size affects performance, we size our encoders to span both below and above the standard ViT-B/16 architecture (Radford et al., 2021; Cherti et al., 2023a; Fan et al., 2023; Xu et al., 2024), which uses vision and text encoders of 86M and 63M parameters, respectively. We define five vision encoders 3 Table 1: Vision and text encoder architectures in the model grid: five vision encoders spanning 10M–352M parameters crossed with six text encoders spanning 3M–145M, for 30 configurations. All models share a vocabulary size of 49,408, context length of 77 tokens, patch size of 16, input resolution of 224×224, and a shared embedding dimension of 512. (a) Vision Encoders NameLayers Width Heads Params Atto12256410M Tiny106401050M Base127681286M Giant2089614194M Colossal22115218352M (b) Text Encoders Name Layers Width Heads Params Femto16413M Atto312827M Nano6256418M Tiny10384637M Base12512863M Giant1576812145M from 10M to 352M parameters, six text encoders from 3M to 145M parameters, and create a 2D grid of all possible combinations (Fig. 1(b); Table 1). This yields 30 configurations totaling 13M–497M parameters. To scale the encoders, we follow prior work (Radford et al., 2021; Cherti et al., 2023a; Gadre et al., 2023; Xu et al., 2024; Chuang et al., 2025), which scales the number of layers and attention heads, and derives the width and MLP hidden dimension from these choices. We keep the patch size (16), input resolution (224×224), context length (77), vocabulary size (49,408), and shared embedding dimension (512) constant across all models to isolate the effect of scaling the encoders. All architectures are implemented and trained using the OpenCLIP library (Cherti et al., 2023a) and the original CLIP loss (Radford et al., 2021). Throughout the paper, we denote each configuration by its vision–text encoder pair (e.g., Base–Tiny uses a Base vision encoder with a Tiny text encoder). All 30 architectures are pre-trained on two datasets of different scale: Conceptual Captions 3M (C3M) (Sharma et al., 2018) and Conceptual Captions 12M (C12M) (Changpinyo et al., 2021), re- sulting in a total of 60 models. We follow the same training hyperparameters as in Fan et al. (2023), which are detailed in Appendix A. 3.2 Evaluation protocols For a comprehensive assessment of downstream performance, we evaluate across 40 downstream datasets—28 for zero-shot classification and retrieval, and 31 for linear probe classification. Most of these datasets are part of the DataComp (Gadre et al., 2023) benchmark, and we follow the same protocol and evaluation metrics. The datasets include ImageNet (Russakovsky et al., 2015) along with five ImageNet distribution shifts (Wang et al., 2019; Recht et al., 2019; Hendrycks et al., 2021b;a), the 19 VTAB tasks (Zhai et al., 2019), Flickr30k (Young et al., 2014) and MSCOCO (Lin et al., 2014) for retrieval tasks, and other common classification tasks used in the original CLIP paper (Radford et al., 2021). Because Flickr30k and MSCOCO retrieval are each evaluated in both the image-to-text and text-to-image directions, the 28 zero-shot datasets correspond to 30 zero-shot tasks. Most datasets are used for both zero-shot and linear probe evaluation, but some are only used for one of the two protocols as detailed in Appendix B.2. We report the average performance across all datasets using the preferred metric for each task (Table A2), rescaled to a percentage ∈[0,100]. Absolute differences in average performance are reported in percentage points (p). We also examine how the scaling patterns manifest across selected dataset groups (Table A3). For zero-shot classification, we adopt the same prompt templates used by Radford et al. (2021). Hand-crafted templates convert each class label into multiple text prompts; the average embedding across all prompts for a class yields the class representation. Predictions are made by computing the cosine similarity between the image embedding and each class representation. In the linear probe setting, we train a linear classifier on top of the frozen image features extracted from the vision encoder, without using any text prompts. It is trained for 10 epochs using Adam (Kingma & Ba, 2015) optimizer with neither weight decay nor learning rate decay, following Caron et al. (2021); He et al. (2022). 4 0255075100125150 Text Encoder Params (M) 30 32 35 38 40 Avg Performance (%) (a) 0255075100125150 Text Encoder Params (M) 14 15 16 17 18 19 (b) Vision Encoder Atto (10M)Tiny (50M)Base (86M)Giant (194M)Colossal (352M) Figure 2: Beyond an optimal size, larger text encoders hurt zero-shot performance. Average zero-shot performance (y-axis; higher is better), averaged over 30 zero-shot tasks, as a function of text encoder size (x-axis), colored by vision encoder, for models pre-trained on (a) C12M and (b) C3M. We leverage the CLIP Benchmark library (Cherti et al., 2023b) to implement both zero-shot and linear probe evaluation protocols. To assess significance across tasks, we report the number of tasks on which one configuration outperforms another out of the non-tied total (n/N), along with thep-value of a two-sided sign test. Significance across random seeds is assessed with a pairedt-test on the per-seed differences, with the direction (one- or two-sided) indicated where reported. 4 Larger text encoders hurt zero-shot performance To understand the impact of independently scaling the vision and text encoders, we train all 30 configurations on both C3M and C12M, and evaluate their zero-shot performance across the 30 zero-shot tasks (Fig. 2). Four patterns emerge consistently across both pre-training datasets. First, for most vision encoders there is an optimal text encoder size beyond which zero-shot performance degrades. Even the smallest degradation, observed between Base–Tiny and Base–Giant, is statistically significant across three random seeds (one-sided pairedt-testp=0.04, Cohen’sd z =1.901; Table A4). Second, once the text encoder is sufficiently large, increasing the vision encoder becomes the dominant driver of performance, and its benefit amplifies when paired with larger text encoders. Third, the magnitude or onset of degradation varies with the vision encoder size, with the smallest vision encoders showing more pronounced degradation than larger ones. Fourth, the optimal text encoder size tends to increase with the vision encoder, but at a slower rate than the vision encoder itself. On C12M the largest vision encoder shows no degradation, though even larger text encoders might eventually induce it. In all our experiments, the smallest text encoder (Femto; 3M parameters) leads to poor and unpredictable performance for both datasets and evaluation settings (Fig. 2). These patterns are consistent across all examined groups (Fig. A1), with the degradation being most pronounced for fine-grained tasks (Fig. A1(c)). One exception to these trends is attributable to overfitting on the smaller C3M dataset. The largest vision encoder (Colossal) does not achieve the best zero-shot performance on C3M, consistent with known overfitting effects on smaller datasets (Cherti et al., 2023a; Fan et al., 2023) and with the learning curves (Figs. A3 and A4). 5 0100200300400500 Total Parameters (M) 28 30 32 34 36 38 40 42 Avg Performance (%) Vision (color) Atto (10M) Tiny (50M) Base (86M) Giant (194M) Colossal (352M) Text (shape) Femto (3M) Atto (7M) Nano (18M) Tiny (37M) Base (63M) Giant (145M) Figure 3: Exploiting the vision–text asymmetry yields configurations matching ViT-B/16 zero-shot perfor- mance with 30–55% fewer parameters. Average zero-shot performance (y-axis; higher is better) versus total model parameters (x-axis) on C12M for all 30 architectures. Color encodes the vision encoder and marker shape encodes the text encoder; the red dotted line traces the Pareto frontier and the gray dashed lines mark the Base–Base (ViT-B/16) baseline. In contrast to zero-shot, scaling the text encoder does not degrade linear probe performance (Figs. A5 and A2) and only exhibits monotonic improvement or saturation at worst. Given that bigger text encoders are not always better, we ask whether it is possible to match the zero-shot performance of the base ViT-B/16 CLIP architecture (Base–Base) with fewer total parameters. The Pareto frontier (Fig. 3) is dominated by configurations with smaller text encoders, and models with similar zero-shot performance can have markedly different total parameters. Concretely, we find that while Base–Base reaches 37.7% average zero-shot performance, the Base–Nano and Tiny–Nano models match or exceed it (38.1% and 37.9%) with 30.6% and 54.7% fewer parameters, respectively. 5 Parameter allocation between encoders matters beyond total count Analyzing the C12M training runs reveals that CLIP architectures with the best zero-shot performance have markedly different vision and text encoder capacities. Across the grid the zero-shot-optimal text encoder for every vision encoder is less than 40M parameters for all vision encoders except Colossal (352M), whose best text encoder has 145M parameters (Fig. 4(a)). In contrast, the optimal vision encoder for any text encoder has at least 194M parameters, and for four of the six text encoders the best vision encoder has 352M parameters. For the largest text encoders the optimal vision encoder size is right-censored, since zero-shot performance is still rising at our largest 352M vision encoder. Even larger vision encoders might yield better results. We observe that in multiple cases the optimal size for one encoder increases as the other grows. To quantify the relationship between model capacity and zero-shot performance, we fit multiple parametric models on the C12M results using the vision, text, and total parameter counts as independent variables (Table A5). Although total parameter count positively correlates with zero-shot performance (Spearman r=0.62,p<0.001), a linear-log regression on total parameters explains only a small fraction of the variance (Adjusted-R 2 =0.312; leave-one-out cross-validated, or LOOCV,R 2 =0.240). Fitting a linear-log model on the vision parameters, text parameters and an interaction term yields a substantially better fit (Adjusted- R 2 =0.693; LOOCVR 2 =0.543), confirming that how capacity is allocated between encoders matters beyond the total budget. 6 b Encoder params (M) Figure 4: The vision–text parameter split matters beyond the total parameter count. (a) Each node represents an encoder positioned on the y-axis according to its size. Left-to-right arrows (blue) point from each text encoder to the best vision encoder paired with it in terms of average zero-shot performance; right-to-left arrows (magenta) point from each vision encoder to the best text encoder. (b) Predicted average zero-shot performance (higher is better) from the fitted saturating-ceiling regression with a capacity-mismatch penalty (Eq. 1), as a function of the vision (x-axis) and text (y-axis) encoder size. Contour lines trace the regression’s iso-performance levels and the 30 architectures are overlaid as points colored by their observed average zero-shot performance. Remarkably, the best fit comes from a simple nonlinear model that treats the two encoders asymmetrically and penalizes a capacity mismatch (LOOCVR 2 =0.937). WritingN v andN t for the vision and text encoder parameter counts (in millions), we fit the form ˆy(N v ,N t ) = (β 0 + β 1 log 10 N v ) |z vision-set ceiling 1− e −kN t | z text saturation exp −c log 10 N v N t 2 | z capacity-mismatch penalty ,(1) whereˆyis the predicted average zero-shot performance. The termβ 0 +β 1 log 10 N v is a linear-log ceiling that is scaled by a saturating factor 1− e −kN t ∈ [0,1] in the text size and further scaled by the mismatch penaltyexp(−c(log 10 (N v /N t )) 2 )∈(0,1]. Removing the mismatch penalty fits substantially worse (LOOCV R 2 =0.826), underscoring the importance of appropriate encoder capacity. Visualizing this fit over the encoder-size plane shows that, once the text encoder is adequately sized, zero-shot performance benefits more from scaling vision, and that the degradation from oversized text encoders is strongest for the smallest vision encoders and gradually vanishes as the vision encoder grows (Fig. 4(b)). We present these findings as an interpretable description of our data rather than a definitive scaling law. Characterizing the full functional form over a wider range of text encoder sizes is a promising direction to predict the optimal parameter split given a total parameter budget, akin to Hoffmann et al. (2022) for language models. 6 Modality-specific weight decay mitigates text-encoder overfitting Prior work has shown that different modalities overfit and generalize at different rates, so training them jointly with a single optimization strategy is typically sub-optimal (Wang et al., 2020). Motivated by this observation, we hypothesized that in CLIP the text encoder requires less capacity and is more prone to overfitting—especially when training with web-scale noisy captions—than the vision encoder, and that this asymmetry drives the zero-shot degradation observed in Section 4. 7 0255075100125150 Text Encoder Params (M) 0.0 0.5 1.0 1.5 2.0 CLIP train loss (a) Vision Encoder Atto (10M) Tiny (50M) Base (86M) Giant (194M) Colossal (352M) 0.50 0.75 1.00 Weight decay ̧ 36 37 38 39 40 Avg Performance (%) Base--Tiny Base--Giant (b) ̧ t only ̧ v only Joint ( ̧ t = ̧ v ) Atto--Giant Tiny--Giant Base--Giant Giant--Giant Architecture 30 35 40 45 50 Avg Performance (%) +2.7 +2.7 +2.3 +1.7 (c) ̧ t = 0:5; ̧ v = 0:5 (degraded) ̧ t = 1; ̧ v = 0:5 Atto--Giant Tiny--Giant Base--Giant Giant--Giant Architecture 0.00 0.05 0.10 0.15 0.20 CLIP train loss +0.07 +0.08 +0.05 +0.03 (d) Figure 5: CLIP demands modality-specific encoder capacity. (a) Final-epoch CLIP training loss on C12M (y-axis) versus text-encoder size (x-axis), colored by vision encoder. (b) Average zero-shot performance (y-axis; higher is better) for the Base–Giant configuration as weight decay is increased on the text encoder only (λ t ), the vision encoder only (λ v ), or both jointly, all emanating from the Base–Giant baseline with joint weight decay of 0.5. The dashed line marks the best configuration using the Base vision encoder (Base–Tiny). (c) Average zero-shot performance (y-axis; higher is better) for each vision encoder paired with the oversized Giant text encoder (x-axis), comparing the degraded configuration (λ t =0.5,λ v =0.5; hatched bars) with the fixed configuration (λ t =1,λ v =0.5; solid bars). (d) Final-epoch CLIP training loss on C12M (y-axis) for the same configurations as in (c), comparing the degraded configuration (hatched) with the fixed configuration (solid). To assess whether optimal text encoders are small because larger ones overfit, we compute CLIP’s contrastive loss on the C12M training set. Growing the text encoder monotonically lowers the training loss for every vision encoder (Fig. 5(a)), even as zero-shot performance degrades once the text encoder exceeds its optimum, as shown previously on Figure 2(a). The held-out CLIP loss on MSCOCO and Flickr30k tells a similar story, with oversized text encoders reaching lower training loss yet higher held-out loss (Fig. A6). These results show that, for a fixed vision encoder, growing the text encoder beyond its optimum leads to overfitting. Driven by these findings, we tested whether modality-specific weight decay improves zero-shot performance by independently reducing each encoder’s effective capacity. This strategy splits the model parameters into a vision and a text group, assigning each separate weight-decay coefficients (λ v andλ t ), held constant throughout training (see Appendix A). Our baseline is Base–Giant, whose oversized text encoder yields degraded zero-shot performance under the defaultλ v =λ t =0.5. We then train three families of variants, raising the weight decay to 0.75 and 1.0 on (i) the text encoder only (λ t ), (i) the vision encoder only (λ v ), or (i) both encoders jointly, as the conventional practice (Table A6). Higher weight decay values (λ=1.5) caused learning collapse in all cases and are therefore excluded (Fig. A8). Figure 5(b) shows that raising the text weight decay consistently improves average zero-shot performance by 1.4 p atλ=0.75 and 2.3 p at 8 050100150 Text Params (M) 1.0 1.1 1.2 1.3 Alignment (a) 050100150 Text Params (M) ¡3.0 ¡2.5 ¡2.0 ¡1.5 Image Uniformity (b) 050100150 Text Params (M) ¡3.00 ¡2.75 ¡2.50 ¡2.25 ¡2.00 Text Uniformity (c) Vision Encoder Atto (10M)Tiny (50M)Base (86M)Giant (194M)Colossal (352M) Figure 6: Scaling the text encoder improves embedding uniformity but worsens image–text alignment. (a) Alignment, (b) image uniformity, and (c) text uniformity (lower is better for all three; y-axes) of MSCOCO embeddings (n=5,000 images-caption pairs) from the C12M pre-trained models, plotted as a function of text encoder size (x-axis) and colored by vision encoder. λ=1.0, whereas only raising the vision weight decay degrades it by up to 0.7 p at 1.0 (text-only wins 25/30 tasks atλ=1,p<0.001). Raising both jointly balances these effects for a net gain, but a smaller one than text-only decay (38.5% vs. 39.3% atλ=1; text-only wins 23/30 tasks,p=0.005). Notably, text-only weight decay (39.3%) surpasses the Base–Tiny configuration (38.2%), the best model using the Base vision encoder. This asymmetry holds in the Giant–Giant configuration where the vision encoder is the larger of the two backbones (194M vs. 145M parameters; A7): raising the text encoder’s weight decay improves performance while raising the vision encoder’s degrades it (text-only wins on 23/30 tasks, p=0.005). Next we retrained and evaluated the zero-shot performance of the four degraded architectures paired with the Giant text encoder (Atto, Tiny, Base, and Giant vision encoders) using modality-specific weight decay with λ t =1.0 andλ v =0.5. This intervention consistently improves zero-shot performance for every configuration, from +2.7 p for Atto–Giant to +1.7 p for Giant–Giant, with the gain shrinking as the vision encoder grows and there is less overfitting to mitigate (all with sign-testp<0.05; Fig. 5(c); Table A7). Strikingly, the performance of the Giant–Giant configuration rises to 41.5% surpassing the best Giant-vision architecture by 1.0 p (Giant–Tiny; sign testp=0.016) and matching the strongest model in the entire grid, Colossal–Giant (41.4%; sign testp=0.585), despite a vision encoder roughly half its size (194M vs. 352M parameters). While modality-specific weight decay improves zero-shot performance, it increases the training loss in all degraded configurations (Fig. 5(d)). This regularization behavior is also observed when computing the CLIP loss on the held-out MSCOCO and Flickr30k evaluation sets (Fig. A9), indicating the gain comes from reduced modality-specific overfitting rather than changes to effective learning rate or training dynamics (D’Angelo et al., 2024). Together, these results point to three principles: CLIP benefits from a larger vision than text encoder; both must be sized appropriately to avoid text-encoder overfitting or vision-encoder underfitting; and the capacity mismatch between the two encoders modulates this effect. 7 Encoder scaling and regularization reshape embedding geometry To understand how asymmetric scaling affects the embedding space, we compute the alignment and unifor- mity (Wang & Isola, 2020) of the image and text embeddings. Alignment measures the expected distance between matched (positive) image–text pairs, and uniformity measures how evenly the embeddings spread over the unit hypersphere. Both are framed as losses, so lower is better. We focus on C12M models, where the scaling patterns are not confounded by the overfitting seen on C3M, and compute these metrics on the 9 5,000 images in the MSCOCO (Lin et al., 2014) Karpathy split (Karpathy & Fei-Fei, 2015), using only the first caption for each image. Larger text or vision encoders tend to improve both image and text embedding uniformity (Figs. 6(b,c) and A10(b)). Scaling the text encoder improves image uniformity at a faster rate than the vision encoder. Additionally, as the text encoder grows, text uniformity keeps rising while image uniformity saturates at the Base text encoder (63M parameters). Even though scaling the text encoder improves image uniformity, it worsens image-text alignment, but the degree of this degradation decreases with larger vision encoders (Fig. 6(a)). Increasing the vision encoder worsens alignment when paired with small text encoders, but improves it when paired with the largests (Fig. A10(a)). Beyond scaling, modality-specific weight decay also reshapes the embedding geometry. Applying it for the degraded configurations improves image–text alignment while maintaining or improving image uniformity, on both MSCOCO (Fig. 7) and Flickr30k (Fig. A13(c)), though typically at the expense of text uniformity (Fig. A13(b,d)). 1.201.251.301.351.40 Alignment ¡3.000 ¡2.975 ¡2.950 ¡2.925 Image Uniformity Vision encoder (color) Atto Tiny Base Giant Weight decay (shape) ̧ t = 0:5; ̧ v = 0:5 ̧ t = 1; ̧ v = 0:5 Figure 7: Modality-specific weight decay im- proves image-text alignment. Embedding align- ment (x-axis) versus image uniformity (y-axis) on MSCOCO (lower is better for both), with arrows pointing from the degraded (crosses) to the fixed (circles) configuration using modality- specific weight decay. Colors denote the vision encoder, all paired with the Giant text encoder. To evaluate whether uniformity and alignment are predic- tive of the average zero-shot performance, we separately extract features from the MSCOCO Karpathy split and Flickr30k, and fit multiple linear regressions on combina- tions of the three metrics (alignment, image uniformity, and text uniformity). We find that a model trained on alignment and image uniformity has the most predictive power (MSCOCO LOOCVR 2 =0.810; Flickr30k LOOCV R 2 =0.877; Table A8). These results are compatible with prior work showing that these metrics correlate with down- stream performance (Wang & Isola, 2020; Tschannen et al., 2025). Using the fitted regression model we plot the pre- dicted zero-shot performance landscape as a function of alignment and image uniformity (Fig. A11), showing that the best models cluster together, while degraded models worsen their alignment without significant improvement in image uniformity. To test generalization, we fit the regres- sion in a leave-one-vision-out fashion, where we train on all but one vision encoder, and then predict the average zero- shot performance of the left-out vision encoder (Fig. A12). The regression captures the overall trend, including the peak and degradation, although it also predicts a degra- dation for the largest vision encoder (Colossal) that is not observed in practice. These results highlight a practical implication: the alignment and uniformity measured on a small held-out dataset can estimate the expected zero-shot performance of a CLIP model across a wide range of tasks. Interpretation. Because zero-shot inference needs both well-separated embeddings (uniformity) and aligned image–text spaces (alignment), scaling the text encoder creates a trade-off: it improves uniformity but often at the expense of alignment. The saturation and degradation pattern we observe in zero-shot performance is reflected in the embedding geometry, coinciding with the point at which alignment degradation begins to outweigh uniformity gains—especially when the vision encoder is not large enough. This interpretation is consistent with our largest vision encoder, the only one that does not degrade and for which using the largest text encoders does not meaningfully deteriorate alignment. In contrast, linear probe inference depends only on the vision encoder, where performance hinges on the linear separability of image embeddings rather than cross-modal alignment. This aligns with our observation that larger text encoders improve image uniformity, and therefore the linear separability of image embeddings. Further analysis and details are provided in Appendix C.8. 10 8 Conclusion In this work, we presented a controlled study of asymmetric encoder scaling in CLIP, revealing that increasing the total number of parameters can consistently degrade zero-shot performance if capacity is not split appropriately between the vision and text encoders. Once the text encoder is large enough, performance is driven by the vision encoder, and further increasing the text encoder degrades it. This asymmetry lets us identify configurations that match the zero-shot performance of the standard ViT-B/16 architecture with 30–55% fewer total parameters. We show that the degradation stems from overfitting induced by the oversized text encoder, which manifests in the embedding space as a worsening of image-text alignment and improved uniformity. Modality-specific weight decay recovers and even improves both performance and image-text alignment across all degraded architectures, while preserving or improving uniformity. Crucially, conventional uniform weight decay helps less than modality-specific weight decay in most scenarios, or performs comparably, reinforcing the need for appropriate effective encoder capacity. There are several avenues for future work. Our findings motivate the development of additional training methods that mitigate the zero-shot degradation. We envision three main directions, including modality- specific regularization or optimization strategies, architectural modifications, and dataset improvements such as using longer and richer captions. Eliminating the degradation would enable more resilient scaling behavior where zero-shot performance increases monotonically with compute while being less sensitive to the encoder capacity split. While it persists, identifying the optimal parameter allocation between encoders across a wider range of dataset scales and qualities could enable principled, parameter-efficient CLIP scaling recipes. Finally, generalizing these findings to multiple and heterogeneous modalities is an important next step. Researchers could explore how each modality’s data statistics inform how to build compute-optimal, large-scale multimodal models beyond vision and text. 9 Limitations Our study has several limitations. First, we train exclusively on C3M and C12M, but the degradation could diminish or disappear with larger pre-training datasets. Second, we evaluate only on classification and retrieval tasks; the impact of asymmetric scaling may differ for other downstream applications such as generation and segmentation. Third, we pre-train all models from scratch, whereas most practitioners initialize from pre-trained encoders, which may alter the dynamics we report. Fourth, even though modality-specific weight decay proved effective in our experiments, in some cases it led to training instabilities and learning collapse. Moreover, this is only one of many possible interventions to reduce the effective encoder capacity. Fifth, we cannot perform significance tests across multiple seeds for all analyses due to the high computational cost involved. Broader Impact Statement We do not anticipate significant risks of harm beyond those already associated with CLIP-style models; the primary impacts are positive. Our work shows that asymmetrically scaling CLIP’s encoders can match or exceed strong baselines with a fraction of the parameters. This lowers the barrier to training CLIP from scratch under limited compute and reduces the environmental footprint of training and inference. Author Contributions Conceptualization: S.C., C.D.E., R.B.; Data Curation: S.C.; Formal Analysis: S.C.; Investigation: S.C.; Methodology: S.C., C.D.E., R.B.; Resources: C.D.E.; Software: S.C.; Supervision: C.D.E., R.B.; Validation: S.C.; Visualization: S.C.; Writing – Original Draft: S.C.; Writing – Review & Editing: S.C., C.D.E., R.B. Acknowledgments The authors thank Hannes Schulz, Sean Whitzell and Kevin K. Yang for assistance with Microsoft’s compute resources; and Eric Zimmermann for helpful discussions. 11 Declaration of Interests The authors declare no competing interests. References Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Yitzhak Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open-source framework for training large autoregressive vision-language models. CoRR, abs/2308.01390, 2023. doi: 10.48550/ARXIV.2308.01390. URL https://doi.org/10.48550/arXiv.2308.01390. Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenen- baum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Van- couver, BC, Canada, p. 9448–9458, 2019. URL https://proceedings.neurips.c/paper/2019/hash/ 97af07a14cacba681feacf3012730892-Abstract.html. Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative components with random forests. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI, Lecture Notes in Computer Science, p. 446–461. Springer, 2014. doi: 10.1007/978-3-319-10599-4\ _29. URL https://doi.org/10.1007/978-3-319-10599-4_29. Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, p. 9630–9640. IEEE, 2021. doi: 10.1109/ICCV48922.2021.00951. URLhttps://doi.org/10.1109/ICCV48922.2021. 00951. Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, p. 3558– 3568. Computer Vision Foundation / IEEE, 2021. doi: 10.1109/CVPR46437.2021.00356. URL https://openaccess.thecvf.com/content/CVPR2021/html/Changpinyo_Conceptual_12M_Pushing_ Web-Scale_Image-Text_Pre-Training_To_Recognize_Long-Tail_Visual_CVPR_2021_paper.html. Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proc. IEEE, 105(10):1865–1883, 2017. doi: 10.1109/JPROC.2017.2675998. URL https://doi.org/10.1109/JPROC.2017.2675998. Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, p. 2818–2829. IEEE, 2023a. doi: 10.1109/CVPR52729. 2023.00276. URL https://doi.org/10.1109/CVPR52729.2023.00276. Mehdi Cherti, Jenia Jitsev, Romain Beaumont, et al. Laion-ai/clip_benchmark: Clip-like model evaluation. Zenodo, 2023b. URL https://github.com/LAION-AI/CLIP_benchmark. Yung-Sung Chuang, Yang Li, Dong Wang, Ching-Feng Yeh, Kehan Lyu, Ramya Raghavendra, James Glass, Lifei Huang, Jason Weston, Luke Zettlemoyer, Xinlei Chen, Zhuang Liu, Saining Xie, Wen tau Yih, Shang-Wen Li, and Hu Xu. Metaclip 2: A worldwide scaling recipe. In Advances in Neural Information Processing Systems, 2025. URL https://arxiv.org/abs/2507.22062. to appear. 12 Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014, p. 3606–3613. IEEE Computer Society, 2014. doi: 10.1109/CVPR.2014.461. URL https://doi.org/10.1109/CVPR.2014.461. Adam Coates, Andrew Y. Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Geoffrey J. Gordon, David B. Dunson, and Miroslav Dudík (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2011, Fort Lauderdale, USA, April 11-13, 2011, JMLR Proceedings, p. 215–223. JMLR.org, 2011. URLhttp: //proceedings.mlr.press/v15/coates11a/coates11a.pdf. Francesco D’Angelo, Maksym Andriushchenko, Aditya Vardhan Varre, and Nicolas Flammarion. Why do we need weight decay in modern deep learning? In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. URLhttp://papers.nips.c/paper_files/ paper/2024/hash/29496c942ed6e08ecc469f4521ebfff0-Abstract-Conference.html. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id=YicbFdNTTy. Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In Ruslan Salakhutdinov, Zico Kolter, Katherine A. Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, Proceedings of Machine Learning Research, p. 12606–12633. PMLR / OpenReview.net, 2024. URL https://proceedings.mlr.press/v235/esser24a.html. M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman.The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results.http://w.pascal- network.org/challenges/VOC/voc2007/workshop/index.html, 2007. Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip train- ing with language rewrites. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, p. 35544–35575. Cur- ran Associates, Inc., 2023. URLhttps://proceedings.neurips.c/paper_files/paper/2023/file/ 6fa4d985e7c434002fb6289ab9b2d654-Paper-Conference.pdf. Li Fei-Fei, Robert Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Comput. Vis. Image Underst., 106(1): 59–70, 2007. doi: 10.1016/J.CVIU.2005.09.012. URL https://doi.org/10.1016/j.cviu.2005.09.012. Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Gian- nis Daras, Sarah M. Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander J. Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Je- nia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: An- nual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URLhttp://papers.nips.c/paper_files/paper/2023/hash/ 56332d41d55ad7ad8024aac625881be7-Abstract-Datasets_and_Benchmarks.html. 13 Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann LeCun. Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, Proceedings of Machine Learning Research, p. 10929–10974. PMLR, 2023. URLhttps://proceedings.mlr.press/ v202/garrido23a.html. Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the KITTI vision benchmark suite. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, June 16-21, 2012, p. 3354–3361. IEEE Computer Society, 2012. doi: 10.1109/CVPR.2012.6248074. URL https://doi.org/10.1109/CVPR.2012.6248074. Amin Ghiasi, Ali Shafahi, and Reza Ardekani. Improving robustness with adaptive weight decay. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URLhttp://papers.nips.c/ paper_files/paper/2023/hash/f9d7d6c695bc983fcfb5b70a5fbdfd2f-Abstract-Conference.html. Ian J. Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron C. Courville, Mehdi Mirza, Benjamin Hamner, William Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, Yingbo Zhou, Chetan Ramaiah, Fangxiang Feng, Ruifan Li, Xiaojie Wang, Dimitris Athanasakis, John Shawe-Taylor, Maxim Milakov, John Park, Radu Tudor Ionescu, Marius Popescu, Cristian Grozea, James Bergstra, Jingjing Xie, Lukasz Romaszko, Bing Xu, Chuang Zhang, and Yoshua Bengio. Challenges in representation learning: A report on three machine learning contests. Neural Networks, 64:59–63, 2015. doi: 10.1016/J.NEUNET.2014.09.005. URL https://doi.org/10.1016/j.neunet.2014.09.005. Di He, Ajay Jaiswal, Songjun Tu, Li Shen, Ganzhao Yuan, Shiwei Liu, and Lu Yin. Alphadecay: Module-wise weight decay for heavy-tailed balancing in llms. CoRR, abs/2506.14562, 2025. doi: 10.48550/ARXIV.2506. 14562. URL https://doi.org/10.48550/arXiv.2506.14562. Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, p. 15979–15988. IEEE, 2022. doi: 10.1109/ CVPR52688.2022.01553. URL https://doi.org/10.1109/CVPR52688.2022.01553. Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens., 12(7):2217–2226, 2019. doi: 10.1109/JSTARS.2019.2918242. URLhttps://doi.org/10.1109/ JSTARS.2019.2918242. Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, p. 8320–8329. IEEE, 2021a. doi: 10.1109/ICCV48922.2021.00823. URLhttps://doi.org/10.1109/ICCV48922.2021. 00823. Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversar- ial examples. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, vir- tual, June 19-25, 2021, p. 15262–15271. Computer Vision Foundation / IEEE, 2021b. doi: 10.1109/ CVPR46437.2021.01501. URLhttps://openaccess.thecvf.com/content/CVPR2021/html/Hendrycks_ Natural_Adversarial_Examples_CVPR_2021_paper.html. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language 14 models. CoRR, abs/2203.15556, 2022. doi: 10.48550/ARXIV.2203.15556. URLhttps://doi.org/10. 48550/arXiv.2203.15556. Masato Ishii and Atsushi Sato. Layer-wise weight decay for deep neural networks. In Manoranjan Paul, Carlos Hitoshi Morimoto, and Qingming Huang (eds.), Image and Video Technology - 8th Pacific-Rim Symposium, PSIVT 2017, Wuhan, China, November 20-24, 2017, Revised Selected Papers, volume 10749 of Lecture Notes in Computer Science, p. 276–289. Springer, 2017. doi: 10.1007/978-3-319-75786-5\_23. URL https://doi.org/10.1007/978-3-319-75786-5_23. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, p. 1988–1997. IEEE Computer Society, 2017. doi: 10.1109/CVPR.2017.215. URL https://doi.org/10.1109/CVPR.2017.215. Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, p. 3128–3137. IEEE Computer Society, 2015. doi: 10.1109/CVPR.2015.7298932. URL https://doi.org/10.1109/CVPR.2015.7298932. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URLhttp://arxiv.org/abs/1412.6980. Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, p. 3992–4003. IEEE, 2023. doi: 10.1109/ICCV51070.2023.00371. URLhttps: //doi.org/10.1109/ICCV51070.2023.00371. Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 2013 IEEE International Conference on Computer Vision Workshops, ICCV Workshops 2013, Sydney, Australia, December 1-8, 2013, p. 554–561. IEEE Computer Society, 2013. doi: 10.1109/ ICCVW.2013.77. URL https://doi.org/10.1109/ICCVW.2013.77. Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009. URL https://w.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf. Yann LeCun. The MNIST database of handwritten digits, 1998. URLhttp://yann.lecun.com/exdb/mnist/. Xianhang Li, Zeyu Wang, and Cihang Xie. An inverse scaling law for CLIP training. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023a. URLhttp://papers.nips.c/ paper_files/paper/2023/hash/996e2b446391fcb8bf32a3d1645c799-Abstract-Conference.html. Xianhang Li, Zeyu Wang, and Cihang Xie. Clipa-v2: Scaling CLIP training with 81.1% zero-shot imagenet accuracy within a $10, 000 budget; an extra $4, 000 unlocks 81.8% accuracy. CoRR, abs/2306.15658, 2023b. doi: 10.48550/ARXIV.2306.15658. URL https://doi.org/10.48550/arXiv.2306.15658. Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y. Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022. URLhttp://papers.nips.c/paper_files/ paper/2022/hash/702f4db7543a7432431df588d57bc7c9-Abstract-Conference.html. 15 Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, Lecture Notes in Computer Science, p. 740–755. Springer, 2014. doi: 10.1007/978-3-319-10602-1\_48. URLhttps://doi.org/10.1007/ 978-3-319-10602-1_48. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URLhttp://papers.nips.c/ paper_files/paper/2023/hash/6dcf277ea32ce3288914faf369fe6de0-Abstract-Conference.html. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7. S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013. Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Vaibhav Srivastav, Joshua Lochner, Hugo Larcher, Mathieu Morlon, Lewis Tunstall, Leandro von Werra, and Thomas Wolf. Smolvlm: Redefining small and efficient multimodal models. CoRR, abs/2504.05299, 2025. doi: 10.48550/ARXIV.2504.05299. URL https://doi.org/10.48550/arXiv.2504.05299. Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. https://github.com/deepmind/dsprites-dataset/, 2017. Kensuke Nakamura and Byung-Woo Hong. Adaptive weight decay for deep neural networks. IEEE Access, 7:118857–118865, 2019. doi: 10.1109/ACCESS.2019.2937139. URLhttps://doi.org/10.1109/ACCESS. 2019.2937139. Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Read- ing digits in natural images with unsupervised feature learning. In Advances in Neural Infor- mation Processing Systems (NeurIPS) Workshops, 2011. URLhttps://storage.googleapis.com/ pub-tools-public-publication-data/pdf/37648.pdf. Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Sixth Indian Conference on Computer Vision, Graphics & Image Processing, ICVGIP 2008, Bhubaneswar, India, 16-19 December 2008, p. 722–729. IEEE Computer Society, 2008. doi: 10.1109/ICVGIP.2008.47. URL https://doi.org/10.1109/ICVGIP.2008.47. Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, June 16-21, 2012, p. 3498–3505. IEEE Computer Society, 2012. doi: 10.1109/CVPR.2012.6248092. URLhttps: //doi.org/10.1109/CVPR.2012.6248092. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, Proceedings of Machine Learning Research, p. 8748–8763. PMLR, 2021. URLhttp: //proceedings.mlr.press/v139/radford21a.html. Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. CoRR, abs/2204.06125, 2022. doi: 10.48550/ARXIV.2204.06125. URL https://doi.org/10.48550/arXiv.2204.06125. 16 Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloé Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross B. Girshick, Piotr Dollár, and Christoph Feichtenhofer. SAM 2: Segment anything in images and videos. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URLhttps://openreview.net/forum? id=Ha6RTeWMd0. Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, Proceedings of Machine Learning Research, p. 5389–5400. PMLR, 2019. URLhttp://proceedings.mlr.press/v97/ recht19a.html. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., 115(3):211–252, 2015. doi: 10.1007/S11263-015-0816-Y. URL https://doi.org/10.1007/s11263-015-0816-y. Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kam- yar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, Novem- ber 28 - December 9, 2022, 2022. URLhttp://papers.nips.c/paper_files/paper/2022/hash/ ec795aeadae0b7d230fa35cbaf04c041-Abstract-Conference.html. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, p. 2556–2565. Association for Computational Linguistics, 2018. doi: 10.18653/V1/P18-1238. URLhttps://aclanthology.org/ P18-1238/. Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: A multi-class classification competition. In The 2011 International Joint Conference on Neural Networks, IJCNN 2011, San Jose, California, USA, July 31 - August 5, 2011, p. 1453–1460. IEEE, 2011. doi: 10.1109/IJCNN.2011.6033395. URL https://doi.org/10.1109/IJCNN.2011.6033395. Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. EVA-CLIP: improved training techniques for CLIP at scale. CoRR, abs/2303.15389, 2023. doi: 10.48550/ARXIV.2303.15389. URLhttps: //doi.org/10.48550/arXiv.2303.15389. Quan Sun, Jinsheng Wang, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, and Xinlong Wang. EVA- CLIP-18B: scaling CLIP to 18 billion parameters. CoRR, abs/2402.04252, 2024. doi: 10.48550/ARXIV. 2402.04252. URL https://doi.org/10.48550/arXiv.2402.04252. Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: the new data in multimedia research. Commun. ACM, 59(2):64–73, 2016. doi: 10.1145/2812802. URL https://doi.org/10.1145/2812802. Michael Tschannen, Alexey A. Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier J. Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. CoRR, abs/2502.14786, 2025. doi: 10.48550/ARXIV.2502.14786. URL https://doi.org/10.48550/arXiv.2502.14786. 17 Bastiaan S. Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant cnns for digital pathology. In Alejandro F. Frangi, Julia A. Schnabel, Christos Davatzikos, Carlos Alberola-López, and Gabor Fichtinger (eds.), Medical Image Computing and Computer Assisted Intervention - MICCAI 2018 - 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part I, Lecture Notes in Computer Science, p. 210–218. Springer, 2018. doi: 10.1007/978-3-030-00934-2\_24. URL https://doi.org/10.1007/978-3-030-00934-2_24. Haohan Wang, Songwei Ge, Zachary C. Lipton, and Eric P. Xing. Learning robust global representations by penalizing local predictive power. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, p. 10506–10518, 2019. URLhttps://proceedings.neurips.c/paper/ 2019/hash/3eefceb8087e964f89c2d59e8a249915-Abstract.html. Jiapeng Wang, Chengyu Wang, Kunzhe Huang, Jun Huang, and Lianwen Jin. Videoclip-xl: Advancing long description understanding for video CLIP models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, p. 16061–16075. Association for Computational Linguistics, 2024. doi: 10.18653/V1/2024.EMNLP-MAIN.898. URLhttps://doi.org/10.18653/v1/ 2024.emnlp-main.898. Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Hal Daumé I and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, p. 9929–9939. PMLR, 13–18 Jul 2020. URL https://proceedings.mlr.press/v119/wang20k.html. Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, p. 12692–12702. Computer Vision Foundation / IEEE, 2020. doi: 10.1109/CVPR42600.2020.01271. URLhttps://openaccess.thecvf.com/content_CVPR_2020/html/ Wang_What_Makes_Training_Multi-Modal_Classification_Networks_Hard_CVPR_2020_paper.html. Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large- scale scene recognition from abbey to zoo. In The Twenty-Third IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2010, San Francisco, CA, USA, 13-18 June 2010, p. 3485–3492. IEEE Computer Society, 2010. doi: 10.1109/CVPR.2010.5539970. URLhttps://doi.org/10.1109/CVPR.2010. 5539970. Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, p. 6787–6800. Association for Computational Linguistics, 2021. doi: 10.18653/V1/2021.EMNLP-MAIN.544. URLhttps://doi.org/10.18653/v1/ 2021.emnlp-main.544. Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying CLIP data. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=5BCFlnfE1g. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Trans. Assoc. Comput. Linguistics, 2: 67–78, 2014. doi: 10.1162/TACL\_A\_00166. URL https://doi.org/10.1162/tacl_a_00166. Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, André Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, 18 Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. The visual task adaptation benchmark. CoRR, abs/1910.04867, 2019. URL http://arxiv.org/abs/1910.04867. Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, p. 18102–18112. IEEE, 2022. doi: 10.1109/CVPR52688.2022.01759. URL https://doi.org/10.1109/CVPR52688.2022.01759. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, p. 11941–11952. IEEE, 2023. doi: 10.1109/ICCV51070.2023.01100. URLhttps: //doi.org/10.1109/ICCV51070.2023.01100. A Implementation details Encoders Our baseline architecture corresponds to the ViT-B/16 from (Cherti et al., 2023a), pairing an 86M-parameter vision encoder with a 63M-parameter text encoder. To construct encoder variants of different sizes, we follow the standard Transformer scaling rules used by prior work (Dosovitskiy et al., 2021; Radford et al., 2021; Cherti et al., 2023a). We only scale the number of attention headshand the number of layers while keeping the per-head dimension fixed at 64, which determines the model width asd model = 64× h. The MLP hidden dimension is set to 4× d model . The same rules apply to the vision and text encoders. The resulting configurations are listed in Table 1. Hyperparameters We follow the same training recipes used by Fan et al. (2023) for both the C3M and C12M datasets. Table A1 provides an overview of the pre-training hyperparameters used for all models on each dataset. We apply RandomResizedCrop with a scale range of (0.5,1.0) as the image augmentation. All pre-training runs were conducted on four to eight machines equipped with 8 A100 GPUs each. Table A1: Pre-training hyperparameters for (a) C3M and (b) C12M datasets. All runs use the AdamW optimizer (Loshchilov & Hutter, 2019). (a) C3M. ConfigValue Batch size8,192 OptimizerAdamW Learning rate1× 10 −3 Weight decay0.5 Adam β 1 ,β 2 = 0.9, 0.98 Adam ε1× 10 −8 Total epochs40 Warmup epochs 1 (355 steps) LR scheduleCosine decay (b) C12M. ConfigValue Batch size8,192 OptimizerAdamW Learning rate1× 10 −3 Weight decay0.5 Adam β 1 ,β 2 = 0.9, 0.98 Adam ε1× 10 −8 Total epochs35 Warmup epochs 1 (1,339 steps) LR scheduleCosine decay Modality-specific weight decay Weight decay is applied per parameter group under AdamW. All one-dimensional parameters are excluded from weight decay: bias from fully connected layers, gains and biases from layer normalization, the visual class embedding, and the logit scale The remaining parameters are split into two groups: • Vision: all remaining parameters used by the vision encoder, including the patch-embedding convolu- tions, positional embeddings attention and MLP matrices, and the final projection matrix. •Text: all remaining parameters used by the text encoder, including the token embeddings, positional embeddings, attention and MLP matrices, and the final projection matrix. 19 Vision parameters are then optimized with a weight decay coefficientλ v and text parameters with a weight decay coefficientλ t , which are held constant throughout training. The baseline and joint-decay runs use λ v = λ t . B Dataset details B.1 Pre-training datasets We pre-train on two image-text datasets at different scales: C3M (Sharma et al., 2018) and C12M (Chang- pinyo et al., 2021). To mitigate the impact of link rot, we use archived versions of both datasets that were downloaded and stored on Hugging Face in 2021. 1 These versions contain fewer samples than the original releases, which may lead to minor differences in performance relative to models trained on the complete datasets. C3M (Sharma et al., 2018) contains 3.3 million image-text pairs collected from approximately 5 billion web pages, where captions are sourced from HTML alt-text attributes. Our copy retains 2.9 million unique samples. C12M (Changpinyo et al., 2021) follows a similar collection pipeline but applies less restrictive filtering, yielding 12.4 million image-text pairs that cover a broader range of visual concepts and topics. The version we use contains 10.9 million samples. For both datasets, all images were resized so that their shorter side equals 256 pixels. B.2 Downstream datasets Table A2 lists the datasets used for zero-shot and linear probe evaluation, along with their task types, sizes, number of classes, main evaluation metrics, and whether they are used for zero-shot (ZS) and/or linear probe (LP) evaluation. For zero-shot evaluation, we exclude 9 of the VTAB datasets because they are known to perform no better than random guessing (Gadre et al., 2023). These include all 8 structured tasks, and the diabetic retinopathy task. We also exclude Face Expression Recognition (FER2013) (Goodfellow et al., 2015) for the same reason. In the linear probe protocol, we exclude Flickr30k (Young et al., 2014) and MSCOCO (Lin et al., 2014) because they are retrieval datasets that are not compatible with linear probe evaluation. The five ImageNet distribution shifts (ImageNet Sketch (Wang et al., 2019), ImageNet v2 (Recht et al., 2019), ImageNet- A (Hendrycks et al., 2021b), ImageNet-O (Hendrycks et al., 2021b), and ImageNet-R (Hendrycks et al., 2021a)), Scene Understanding (SUN-397) (Xiao et al., 2010), and ObjectNet (Barbu et al., 2019) are also excluded due to incompatibility with the CLIP Benchmark library (Cherti et al., 2023b). 1 https://huggingface.co/datasets/conceptual_captions and https://huggingface.co/datasets/conceptual_12m 20 Table A2: Evaluation tasks in the benchmark suite. Columns ZS and LP indicate whether each dataset is used for zero-shot and linear probe evaluation, respectively; a checkmark (✓) marks inclusion. Task typeDatasetTaskTest size ClassesMain metric ZS LP Classification Caltech-101 (Fei-Fei et al., 2007)Object recognition6,085102 mean per class✓ CIFAR-10 (Krizhevsky et al., 2009)Visual recognition10,00010accuracy✓ CIFAR-100 (Krizhevsky et al., 2009)Visual recognition10,000100accuracy✓ CLEVR Counts (Johnson et al., 2017; Zhai et al., 2019)Counting15,0008accuracy✓ CLEVR Distance (Johnson et al., 2017; Zhai et al., 2019)Distance prediction15,0006accuracy✓ Country211 (Radford et al., 2021; Thomee et al., 2016)Geolocation21,100211accuracy✓ Describable Textures (Cimpoi et al., 2014)Texture classification1,88047accuracy✓ EuroSAT (Helber et al., 2019; Zhai et al., 2019)Satellite imagery recognition 5,40010accuracy✓ FGVC Aircraft (Maji et al., 2013)Aircraft recognition3,333100 mean per class✓ Food-101 (Bossard et al., 2014)Food recognition25,250101accuracy✓ GTSRB (Stallkamp et al., 2011)Traffic sign recognition12,63043accuracy✓ ImageNet 1k (Russakovsky et al., 2015)Visual recognition50,0001,000accuracy✓ ImageNet Sketch (Wang et al., 2019)Visual recognition50,8891,000accuracy✓ ImageNet v2 (Recht et al., 2019)Visual recognition10,0001,000accuracy✓ ImageNet-A (Hendrycks et al., 2021b)Visual recognition7,500200accuracy✓ ImageNet-O (Hendrycks et al., 2021b)Visual recognition2,000200accuracy✓ ImageNet-R (Hendrycks et al., 2021a)Visual recognition30,000200accuracy✓ KITTI Vehicle Distance (Geiger et al., 2012; Zhai et al., 2019) Distance prediction7114accuracy✓ MNIST (LeCun, 1998)Digit recognition10,00010accuracy✓ ObjectNet (Barbu et al., 2019)Visual recognition18,574113accuracy✓ Oxford Flowers-102 (Nilsback & Zisserman, 2008)Flower recognition6,149102 mean per class✓ Oxford-IIIT Pet (Parkhi et al., 2012; Zhai et al., 2019)Pet classification3,66937 mean per class✓ Pascal VOC 2007 (Everingham et al., 2007)Object recognition14,97620accuracy✓ PatchCamelyon (Veeling et al., 2018; Zhai et al., 2019)Metastatic tissue cls.32,7682accuracy✓ Rendered SST2 (Zhai et al., 2019)Sentiment classification1,8212accuracy✓ RESISC45 (Cheng et al., 2017; Zhai et al., 2019)Satellite imagery recognition 6,30045accuracy✓ Stanford Cars (Krause et al., 2013)Vehicle recognition8,041196accuracy✓ STL-10 (Coates et al., 2011)Visual recognition8,00010accuracy✓ SUN397 (Xiao et al., 2010)Scene recognition108,754397accuracy✓ SVHN (Netzer et al., 2011; Zhai et al., 2019)Digit recognition26,03210accuracy✓ FER2013 (Goodfellow et al., 2015)Facial expression recognition N/AN/Aaccuracy✓ Diabetic Retinopathy (Zhai et al., 2019)Medical image cls.N/AN/Aaccuracy✓ DMLab (Zhai et al., 2019)3D scene recognitionN/AN/Aaccuracy✓ dSprites Orientation (Matthey et al., 2017; Zhai et al., 2019) Orientation predictionN/AN/Aaccuracy✓ dSprites X Position (Matthey et al., 2017; Zhai et al., 2019)Position predictionN/AN/Aaccuracy✓ dSprites Y Position (Matthey et al., 2017; Zhai et al., 2019)Position predictionN/AN/Aaccuracy✓ SmallNORB Azimuth (Zhai et al., 2019)Azimuth predictionN/AN/Aaccuracy✓ SmallNORB Elevation (Zhai et al., 2019)Elevation predictionN/AN/Aaccuracy✓ RetrievalFlickr30k (Young et al., 2014)Image and text retrieval1,000N/AR@1✓ MSCOCO (Lin et al., 2014)Image and text retrieval5,000N/AR@1✓ Table A3: Dataset groups used for per-group analysis. Groups are not exhaustive of all evaluation datasets. GroupDatasets ImageNetImageNet-1k ImageNet Shifts ImageNet Sketch, ImageNet v2, ImageNet-A, ImageNet-O, ImageNet-R Fine-grainedStanford Cars, Country211, FGVC Aircraft, Food-101, DTD, Flowers-102, Oxford-IIIT Pets, RESISC45, Diabetic Retinopathy, PatchCamelyon RetrievalFlickr30k, MSCOCO 21 C Additional results C.1 Robustness of the zero-shot degradation to the training seed To confirm that the zero-shot degradation from oversizing the text encoder is not an artifact of a single training run, we retrain the pair with the smallest degradation in the grid, Base–Tiny (best) versus Base–Giant (degraded), at three random seeds and test the per-seed paired difference using a paired t-test (Table A4). Table A4: Smallest zero-shot degradation is robust to the training seed. Average zero-shot performance for the Base–Tiny (best) and Base–Giant (largest text encoder; degraded) configurations, each trained at three seeds (123, 321, 0). ∆ is the per-seed paired difference; a one-sided pairedt-test on ∆ (H 0 : ∆≤0) gives t(2)=3.29, p=0.04, Cohen’s d z =1.901, and a one-sided 95% CI of [+0.09, ∞). Seed Base–Tiny (%) Base–Giant (%)∆ (p; Tiny− Giant) 12338.7037.68+1.02 32138.2337.15+1.08 038.2537.93+0.32 Mean38.3937.59+0.81 SD0.270.400.42 C.2 Zero-shot and linear probe performance for sub-groups The main text reports aggregate scaling trends averaged across all downstream datasets. To understand how these trends vary across distinct groups, we compute the average zero-shot and linear probe performance for each dataset group in Table A3. The results are shown in Figures A1 and A2. 22 0255075100125150 Text Params (M) 30 35 40 45 Avg Performance (%) (a) 0255075100125150 Text Params (M) 20 25 30 35 (b) 0255075100125150 Text Params (M) 22 25 28 30 32 35 Avg Performance (%) (c) 0255075100125150 Text Params (M) 20 25 30 35 40 45 (d) Vision Encoder Atto (10M)Tiny (50M)Base (86M)Giant (194M)Colossal (352M) Figure A1: The zero-shot peak-and-degradation pattern holds across groups and is most pronounced for fine-grained tasks. Average zero-shot performance (y-axis; higher is better) versus text encoder size (x-axis), colored by vision encoder, for C12M models split by groups defined in Table A3: (a) ImageNet, (b) ImageNet distribution shifts, (c) fine-grained, and (d) retrieval. 23 0255075100125150 Text Params (M) 55 60 65 Avg Performance (%) (a) 0255075100125150 Text Params (M) 60 65 70 (b) Vision Encoder Atto (10M)Tiny (50M)Base (86M)Giant (194M)Colossal (352M) Figure A2: Linear probe scaling shows no degradation across groups. Average linear probe performance (y-axis; higher is better) versus text encoder size (x-axis), colored by vision encoder, for C12M models split by groups defined in Table A3: (a) ImageNet and (b) fine-grained. C.3 CLIP learning curves To diagnose overfitting to the C3M and C12M pre-training datasets, we visualize the training loss and ImageNet zero-shot validation accuracy curves for the Colossal (Fig. A3) and Base (Fig. A4) vision encoders paired with all text encoders. On C3M the Colossal’s vision encoder training loss continues to decrease when paired with any text encoder, but the validation accuracy saturates or decreases after 20 epochs. On C12M, both the training loss and validation accuracy continue to improve throughout training. The overfitting pattern on C3M is less pronounced for the Base vision encoder. 24 05101520253035 0 2 4 6 Train Loss (a) 010203040 0 2 4 6 8 (b) 05101520253035 Epoch 10 20 30 40 ImageNet ZS Val Top-1 (%) (c) 010203040 Epoch 0 5 10 15 (d) Text Encoder Femto (3M)Atto (7M)Nano (18M)Tiny (37M)Base (63M)Giant (145M) Figure A3: Using the Colossal vision encoder leads to strong overfitting on C3M but generalizes on C12M. Training loss (top row; a, b) and ImageNet zero-shot validation top-1 accuracy (bottom row; c, d; higher is better) versus epoch (x-axis) for the Colossal vision encoder pre-trained on (a, c) C12M and (b, d) C3M, colored by text encoder. 25 05101520253035 0 2 4 6 Train Loss (a) 010203040 0 2 4 6 8 (b) 05101520253035 Epoch 10 20 30 40 ImageNet ZS Val Top-1 (%) (c) 010203040 Epoch 5 10 15 (d) Text Encoder Femto (3M)Atto (7M)Nano (18M)Tiny (37M)Base (63M)Giant (145M) Figure A4: Using the Base vision encoder leads to milder overfitting on C3M. Training loss (top row; a, b) and ImageNet zero-shot validation top-1 accuracy (bottom row; c, d; higher is better) versus epoch (x-axis) for the Base vision encoder pre-trained on (a, c) C12M and (b, d) C3M, colored by text encoder. 26 C.4 Text encoder scaling drives linear probe gains without degradation Linear probe evaluation provides complementary insights to the zero-shot setting. During inference it relies only on the vision encoder, but the text encoder is still used throughout training. We evaluate the linear probe performance of all the models across 31 downstream datasets (Fig. A5). Two patterns differ from the zero-shot setting and hold consistently across both datasets. First, scaling the text encoder does not degrade performance; in fact, it even improves it when paired with any of the three largest vision encoders. Second, C12M models using a vision encoder larger than Atto show that scaling the text encoder leads to larger improvements than scaling the vision encoder. As in the zero-shot setting, the smallest text encoder (Femto; 3M parameters) leads to poor and unpredictable performance. These patterns are consistent across downstream task groups (Fig. A2): for both ImageNet and fine-grained tasks, scaling the text encoder yields saturation or monotonic improvement, but no degradation. On C3M, the two largest vision encoders show reduced linear probe performance due to overfitting on the smaller dataset, consistent with the same effect observed in the zero-shot setting. 020406080100120140 Text Encoder Params (M) 45 50 55 60 65 Avg Performance (%) (a) 020406080100120140 Text Encoder Params (M) (b) Vision Encoder Atto (10M)Tiny (50M)Base (86M)Giant (194M)Colossal (352M) Figure A5: Unlike zero-shot, scaling the text encoder does not degrade linear probe performance. Average linear probe performance (y-axis; higher is better), averaged over 31 downstream tasks, as a function of text encoder size (x-axis) on (a) C12M and (b) C3M, colored by vision encoder. C.5 Predicting zero-shot performance from encoder size Table A5 reports fit statistics for the scaling models discussed in Section 5, including the number of free parameters and leave-one-out cross-validatedR 2 . All models are fit to predict the average zero-shot performance across all tasks using the last epoch of the 30 grid models trained on C12M dataset. 27 Table A5: A saturating-ceiling model with a capacity-mismatch penalty best predicts zero-shot performance from encoder size. Fit statistics for five models of average zero-shot performance across the 30 C12M architectures (final-epoch checkpoint), whereN v andN t are the vision and text encoder parameter counts (millions): number of free parameters, in-sampleR 2 , adjustedR 2 , and leave-one-out cross-validatedR 2 (LOOCV; higher is better). The mismatch-penalty model (last row) is defined in Eq. 1. Best metrics are bolded. ModelFunctional form# params R 2 Adj. R 2 LOOCV R 2 linear-log: total paramsβ 0 + β 1 log 10 (N v +N t )20.336 0.3120.240 linear-log: vision + textβ 0 + β 1 log 10 N v + β 2 log 10 N t 30.639 0.6130.523 linear-log: vision + text + interaction β 0 + β 1 log 10 N v + β 2 log 10 N t + β 3 (log 10 N v )(log 10 N t )40.724 0.6930.543 saturating ceiling(β 0 + β 1 log 10 N v )(1− e −kN t )30.880 0.8710.826 saturating ceiling with mismatch penalty (β 0 + β 1 log 10 N v )(1− e −kN t ) exp −c log 10 N v N t 2 40.960 0.9560.937 C.6 Text encoder overfitting on held-out MSCOCO and Flickr30k datasets Figure 2(a) shows that the average zero-shot performance degrades even as Figure. 5(a) shows the training loss decreases monotonically. To observe the overfitting effect only using CLIP’s contrastive loss, we use the C12M-trained models to compute the loss on two held-out image caption datasets: MSCOCO (Lin et al., 2014) and Flickr30k (Young et al., 2014) (Fig. A6). These datasets provide natural image captions that are closer to C12M’s distribution than the artificial caption templates used for other zero-shot tasks. For every vision encoder of the degraded configurations, pairing it with the oversized Giant text encoder reaches a lower C12M training loss yet a higher held-out contrastive loss than the best text encoder for that vision encoder. Figure A6: Large text encoders overfit held-out image caption datasets. Final-epoch CLIP contrastive loss on the C12M training set (x-axis) versus a held-out set (y-axis; lower is better): (a) MSCOCO and (b) Flickr30k. For each vision encoder, its best text-encoder pairing (circles) is compared against the same vision encoder paired with the oversized Giant text encoder (crosses). 28 C.7 Modality-specific weight decay Table A6 reports the average zero-shot performance of training the Base–Giant and Giant–Giant vision encoders using modality-specific weight decay. Raising the text encoder’s weight decay improves zero-shot performance while raising the vision encoder’s degrades it. Table A6: Modality-specific weight-decay ablations. Average zero-shot performance (higher is better) for every weight-decay ablation run of the Base–Giant and Giant–Giant configurations.λ t andλ v are the per-encoder weight decays (baselineλ t =λ v =0.5). Within each configuration, the two best-performing weight-decay settings are bolded. Architecture λ t λ v Avg Performance (%) Base–Giant0.5 0.537.0 0.75 0.538.4 1.0 0.539.3 0.5 0.7536.8 0.5 1.036.3 0.75 0.7538.3 1.0 1.038.5 Giant–Giant 0.5 0.539.8 0.75 0.540.4 1.0 0.541.5 0.5 0.7539.7 0.5 1.039.3 0.75 0.7541.6 1.0 1.040.3 0.50 0.75 1.00 Weight decay ̧ 39 40 41 42 Avg Performance (%) Giant--Tiny 40.4 41.5 39.7 39.3 39.8 Giant--Giant ̧ t only ̧ v only Figure A7: Weight-decay vision-text asymmetry replicates on the Giant–Giant configuration. Average zero-shot performance (y-axis; higher is better) as weight decay is increased on the text encoder only (λ t ), the vision encoder only (λ v ), or both jointly, all emanating from the Giant–Giant baseline with joint weight decay of 0.5. The dashed line marks the best configuration using the Giant vision encoder (Giant–Tiny). Raising the weight decay toλ=1.5 destabilizes training. As shown in Figure A8, the CLIP training loss decreases normally, then spikes and locks atlnN batch , the contrastive loss of a model that outputs a uniform distribution over the training batch (N batch =8,192). 29 05k10k15k20k25k30k Training step 4 6 8 10 12 14 16 CLIP train loss (a) 01k2k3k4k5k6k Training step uniform (ln N batch ) (b) Figure A8: Learning collapses when weight decay is too strong. CLIP training loss (y-axis) versus training step (x-axis) for the Base–Giant configuration on C12M with weight decay raised toλ=1.5 on (a) the text encoder only and (b) both encoders jointly. As shown in Section 6, applying modality-specific weight decay to the degraded configurations improves average zero-shot performance at the expense of training loss. Computing CLIP’s contrastive loss on held-out MSCOCO and Flickr30k datasets tells a similar story (Fig. A9), where raising the text encoder’s weight decay reduces validation loss even as the training loss increases. Table A7: Modality-specific weight decay improves every degraded configuration. Average zero-shot perfor- mance (higher is better) for each vision encoder paired with the oversized Giant text encoder, at the baseline text weight decay (λ t =0.5) and raised toλ t =1.0 (visionλ v =0.5 throughout). “Improved tasks” counts datasets (n of N) where λ t =1.0 beats the baseline; p is a two-sided sign test. Numbers behind Figure 5(c). Avg Performance (%) Architecture λ t =0.5 λ t =1.0∆ (p) Improved tasks (n/N) p-value Atto–Giant30.333.0+2.725/30 <0.001 Tiny–Giant35.237.9+2.727/30 <0.001 Base–Giant37.039.3+2.324/300.001 Giant–Giant 39.841.5+1.721/290.024 30 0.075 0.1000.1250.150 0.175 0.200 CLIP train loss 2.75 3.00 3.25 3.50 3.75 4.00 4.25 4.50 CLIP loss on held out dataset (a) 0.075 0.1000.1250.150 0.175 0.200 CLIP train loss 2.0 2.5 3.0 3.5 4.0 (b) ̧ t = 0:5; ̧ v = 0:5 (degraded) ̧ t = 1; ̧ v = 0:5 Atto--GiantTiny--GiantBase--GiantGiant--Giant Figure A9: Modality-specific weight decay reduces held-out overfitting on MSCOCO and Flickr30k. Final- epoch CLIP contrastive loss on the C12M training set (x-axis) versus the held-out set (y-axis; lower is better) on (a) MSCOCO and (b) Flickr30k, with an arrow from each degraded configuration (λ t =0.5,λ v =0.5; crosses) to its counterpart with raised text weight decay (λ t =1.0,λ v =0.5; circles). Each configuration pairs the labelled vision encoder with the Giant text encoder. C.8 Additional embedding geometry analysis C.8.1 Uniformity and Alignment Given image and text inputsx I andx T , the image encoderfand text encoderg(including their projection heads) map inputs toℓ 2 -normalized embeddings on the unit hypersphere. Letp pos be the distribution of positive (matched) image–text pairs, andp image andp text be the marginal distributions over images and texts, respectively: L align (f,g;α)≜E (x I ,x T )∼p pos [∥f (x I )− g(x T )∥ α 2 ] L image uniform (f ;τ )≜ lnE x I ,x ′ I i.i.d. ∼ p image h e −τ∥f(x I )−f(x ′ I )∥ 2 2 i L text uniform (g;τ )≜ lnE x T ,x ′ T i.i.d. ∼ p text h e −τ∥g(x T )−g(x ′ T )∥ 2 2 i where we setα= 2 for alignment andτ= 2 for uniformity, following the original formulation in (Wang & Isola, 2020). Section 7 examines alignment and uniformity as a function of text encoder size. Here we present the complementary view: the same metrics plotted as a function of vision encoder size (Fig. A10). This view helps visualize the impact of scaling the vision encoder on alignment and uniformity. C.8.2 Predicting zero-shot performance from uniformity and alignment Table A8 reports the in-sample and leave-one-out cross-validatedR 2 of linear models regressing zero-shot performance on standardized geometry predictors. For this analysis we use all trained C12M models, 31 0100200300 Vision Params (M) 1.0 1.1 1.2 1.3 Alignment (a) 0100200300 Vision Params (M) ¡3.0 ¡2.5 ¡2.0 ¡1.5 Image Uniformity (b) 0100200300 Vision Params (M) ¡3.00 ¡2.75 ¡2.50 ¡2.25 ¡2.00 Text Uniformity (c) Text Encoder Femto (3M)Atto (7M)Nano (18M)Tiny (37M)Base (63M)Giant (145M) Figure A10: Complementary to Figure 6, scaling the vision encoder improves alignment when paired with larger text encoders but worsens it with the smallest ones. (a) Alignment, (b) image uniformity, and (c) text uniformity (lower is better for all three; y-axes) of MSCOCO embeddings from the C12M pre-trained models, as a function of vision encoder size (x-axis) and colored by text encoder. including those used for the modality-specific weight decay ablations, and we perform the cross-validation separately on the MSCOCO and Flickr30k held-out sets. No single metric predicts zero-shot performance well, but image uniformity and alignment together explain most of the variance; adding text uniformity does not help, as it is collinear with image uniformity. To evaluate whether the regression model generalizes across vision encoders and whether it captures the peak and degradation pattern, we perform a leave-one-vision-out (LOVO) analysis, where the linear model is fit using data from all vision encoders except one and used to predict the zero-shot performance on data from the held-out vision encoder. Figure A12 shows that the model generalizes well to unseen vision encoders and captures the overall curve. For the geometry landscape plot (Fig. A11), we create a grid of 200×200 points across the image uniformity and alignment plane, and use the linear model to predict the average zero-shot performance at each point. Table A8: Image uniformity and alignment together predict zero-shot performance across models. In-sample R 2 , adjustedR 2 , and leave-one-out cross-validatedR 2 (LOOCV; higher is better) for linear models regressing average zero-shot performance on standardized geometry predictors (alignment, image/text uniformity), fit across the C12M models. Geometry is measured on two probe sets (n=38 models each; MSCOCO, Flickr30k). The best value in each column is in bold. MSCOCOFlickr30k Predictors# params R 2 Adj. R 2 LOOCV R 2 Adj. R 2 LOOCV Image uniformity20.490 0.4760.445 0.524 0.5110.482 Alignment20.150 0.1270.022 0.098 0.072 -0.047 Text uniformity20.347 0.3280.275 0.435 0.4200.381 Image uniformity + alignment30.847 0.838 0.810 0.901 0.895 0.877 All three40.848 0.8350.794 0.901 0.8930.867 C.8.3 Impact of modality-specific weight decay on embedding geometry Figure A13 shows the change in alignment and uniformity when going from each degraded configuration to its counterpart using modality-specific weight decay, for both MSCOCO and Flickr30k held-out sets and for both image and text uniformity. Panel (a) corresponds to Figure 7. 32 ¡3.00 ¡2.75 ¡2.50¡2.25¡2.00 ¡1.75 ¡1.50 Image Uniformity 1.0 1.1 1.2 1.3 1.4 Alignment Vision encoder Atto Tiny Base Giant Colossal Best per vision 10 20 30 40 50 Predicted avg performance (%) Figure A11: The best models have similar image uniformity and alignment, while degraded models have worse alignment without a compensating gain in image uniformity. Predicted average zero-shot performance (color; brighter is higher) across the image uniformity (x-axis;n=200 grid points) and alignment (y-axis; n=200 grid points) plane, from a linear model fit on the MSCOCO geometry metrics using C12M models to extract the embeddings. Markers show models by vision encoder, with the best text pairing per vision encoder outlined in red. The inset magnifies the boxed region. 30 40 (a) Atto (10M)Tiny (50M)Base (86M)Giant (194M)Colossal (352M) 3M7M 18M37M63M 145M 30 40 (b) 3M7M 18M37M63M 145M 3M7M 18M37M63M 145M 3M7M 18M37M63M 145M 3M7M 18M37M63M 145M Avg Performance (%) Text encoder ObservedPredicted Figure A12: A geometry-only linear model generalizes to unseen vision encoders, capturing the peak-and- degradation pattern. For each vision encoder, the model regressing average zero-shot performance on image uniformity and alignment is fit on the four other vision encoders and used to predict the held-out one (leave-one-vision-out). Each panel shows the observed (solid line, filled circles, coloured by vision encoder) and predicted (dashed line, open grey squares) average zero-shot performance (y-axis; higher is better) versus text encoder size (x-axis). Each column is a vision encoder while rows give the held-out set used to measure geometry: (a) Flickr30k and (b) MSCOCO. 33 Figure A13: Increasing text weight decay improves alignment and image uniformity. Alignment (x-axis) versus uniformity (y-axis; lower is better for both) for each vision encoder paired with the Giant text encoder, going from the degraded configuration (λ t =0.5,λ v =0.5; crosses) to the configuration with modality-specific weight decay (λ t =1,λ v =0.5; circles). Rows: (a,b) MSCOCO and (c,d) Flickr30k held-out sets. Columns: (a,c) image uniformity and (b,d) text uniformity (y-axis). 34 C.8.4 RankMe We also compute RankMe (Garrido et al., 2023) for the image and text embeddings (Fig. A14), which provides a label-free estimate of the effective dimensionality of a representation matrix. RankMe is computed on a random sample of 25,600 images from the C12M training data, matching the sample size and methodology used in the original work (Garrido et al., 2023). Given an embedding matrixZ, RankMe is defined as: RankMe(Z) = exp − min(N,K) X k=1 p k lnp k , with p k = σ k (Z) ∥σ(Z)∥ 1 + ε, where σ k (Z) denotes the k-th singular value ofZ. Scaling the text encoder increases the effective rank of both image and text embeddings substantially, while RankMe is significantly less sensitive to the size of the vision encoder (Fig. A14). This pattern closely mirrors the linear probe scaling behavior (Fig. A5), where the text encoder is the dominant driver of performance. This is also consistent with Garrido et al. (2023), which shows that RankMe correlates with linear probe downstream performance. Femto (3M) Atto (7M) Nano (18M) Tiny (37M) Base (63M) Giant (145M) Text Encoder Atto (10M) Tiny (50M) Base (86M) Giant (194M) Colossal (352M) Vision Encoder 76126171183185189 90138190205209210 88138190204208211 88142192203207209 82144190202204206 (a) Femto (3M) Atto (7M) Nano (18M) Tiny (37M) Base (63M) Giant (145M) Text Encoder 53109165190199206 55113176202213216 57115176201209215 58120178197207210 58121177196203207 (b) 75 100 125 150 175 200 RankMe (effective rank) Figure A14: Scaling the text encoder raises the effective rank (RankMe) of both image and text embeddings, whereas the vision encoder size has little effect. (a) Image and (b) text RankMe for the 30 architectures on C12M training data, crossing five vision encoders (y-axis) with six text encoders (x-axis). C.8.5 Modality gap The modality gap is a phenomenon observed in contrastive vision-language models where image and text embeddings cluster in distinct cones of the embedding space (Liang et al., 2022). It is measured as the distance between the centroids of the image and text embedding distributions: ∆ gap = 1 n n X i=1 f (x (i) I )− 1 n n X i=1 g(x (i) T ) 2 The modality gap decreases roughly monotonically as either encoder is scaled up, though the two largest vision encoders exhibit a small uptick at the largest text encoder sizes (Fig. A15). Overall, the modality gap does not exhibit the non-monotonic pattern observed in zero-shot performance, indicating that it may not be sufficient to explain zero-shot downstream performance. 35 0255075100125150 Text Params (M) 0.56 0.58 0.60 0.62 0.64 0.66 0.68 0.70 0.72 Modality Gap Vision Encoder Atto (10M) Tiny (50M) Base (86M) Giant (194M) Colossal (352M) Figure A15: As the text encoder grows, the modality gap follows a U-shaped curve that is more pronounced for larger vision encoders. Modality gap (y-axis) of MSCOCO embeddings as a function of text encoder size (x-axis) for C12M models, colored by vision encoder. 36