Paper deep dive
What Makes Synthetic Hard Negatives Work in Vision-Language Pretraining?
Nikos Giakoumoglou, Paschalis Giakoumoglou, Andreas Floros, Kleanthis Marios Papadopoulos, Tania Stathaki
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/11/2026, 3:05:29 AM
Summary
The paper investigates the transfer of synthetic hard negatives from unimodal self-supervised learning to vision-language pretraining. It identifies two failure modes in existing representation-space synthesis strategies: cross-modal constructions that fall into the 'modality gap' and intra-modal constructions that cause 'positive leakage' by incorporating the matched positive. The authors propose SNAP, a model-agnostic method that generates intra-modal hard negatives excluding the positive pair, combined with a fixed temperature parameter. SNAP improves zero-shot retrieval, classification, and linear probe performance on CLIP and FLIP models with minimal computational overhead.
Entities (11)
Relation Signals (11)
SNAP → buildson → FLIP
confidence 95% · Evaluated on top of CLIP and FLIP
SNAP → buildson → CLIP
confidence 95% · Evaluated on top of CLIP and FLIP
SNAP → improves → linear probe evaluation
confidence 95% · SNAP delivers consistent improvements on zero-shot retrieval, zero-shot classification, and linear probe evaluation.
SNAP → improves → zero-shot retrieval
confidence 95% · SNAP delivers consistent improvements on zero-shot retrieval, zero-shot classification, and linear probe evaluation.
SNAP → improves → Zero-shot Classification
confidence 95% · SNAP delivers consistent improvements on zero-shot retrieval, zero-shot classification, and linear probe evaluation.
SNAP → proposes → intra-modal hard negatives
confidence 95% · Using this geometric analysis we propose SNAP, which generates intra-modal hard negatives that never involve the positive from either modality
SNAP → avoids → Positive Leakage
confidence 90% · SNAP... avoiding both failure modes entirely... intra-modal constructions that incorporate the matched positive... causes positive leakage
SNAP → avoids → Modality Gap
confidence 90% · SNAP... avoiding both failure modes entirely... cross-modal constructions that produce overly easy negatives... fall into the modality gap
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Synthetic hard negatives generated in the representation space have proven effective for unimodal self-supervised learning, but transferring this idea to vision-language pretraining is not straightforward. We analyze six representation-space synthesis strategies and identify two failure modes in their transfer to vision-language pretraining: cross-modal constructions that produce overly easy negatives or pull them toward the query, and intra-modal constructions that incorporate the matched positive. We also observe logit-scale saturation when training with synthetic hard negatives and a learnable temperature, and find that fixing the temperature improves downstream performance. Using this geometric analysis we propose SNAP, which generates intra-modal hard negatives that never involve the positive from either modality, avoiding both failure modes entirely. SNAP is model-agnostic, requires no external generative models, and adds less than 10% training time overhead. Evaluated on top of CLIP and FLIP across multiple architectures and datasets, SNAP delivers consistent improvements on zero-shot retrieval, zero-shot classification, and linear probe evaluation.
Tags
Links
- Source: https://arxiv.org/abs/2610.09700v1
- Canonical: https://arxiv.org/abs/2610.09700v1
Trouble viewing inline? Open PDF directly →
Full Text
73,577 characters extracted from source content.
Expand or collapse full text
What Makes Synthetic Hard Negatives Work in Vision-Language Pretraining? Nikos Giakoumoglou Affiliation: Imperial College London, London, UK Paschalis Giakoumoglou Affiliation: Centre for Research and Technology Hellas, Thessaloniki, Greece E-mail nikos@imperial.ac.uk Andreas Floros Affiliation: Imperial College London, London, UK Kleanthis Marios Papadopoulos Affiliation: Imperial College London, London, UK Tania Stathaki Affiliation: Imperial College London, London, UK Abstract Synthetic hard negatives generated in the representation space have proven effective for unimodal self-supervised learning, but transferring this idea to vision-language pretraining is not straightforward. We analyze six representation-space synthesis strategies and identify two failure modes in their transfer to vision-language pretraining: cross-modal constructions that produce overly easy negatives or pull them toward the query, and intra-modal constructions that incorporate the matched positive. We also observe logit-scale saturation when training with synthetic hard negatives and a learnable temperature, and find that fixing the temperature improves downstream performance. Using this geometric analysis we propose SNAP, which generates intra-modal hard negatives that never involve the positive from either modality, avoiding both failure modes entirely. SNAP is model-agnostic, requires no external generative models, and adds less than 10% training time overhead. Evaluated on top of CLIP and FLIP across multiple architectures and datasets, SNAP delivers consistent improvements on zero-shot retrieval, zero-shot classification, and linear probe evaluation. Keywords: CLIP Contrastive learning Hard negatives Synthetic negatives Vision-language pretraining Zero-shot transfer 1 Introduction Contrastive pretraining using weakly-supervised image-text pairs collected from the web has become the dominant paradigm for learning generic visual representations, progressively replacing supervised pretraining on large labeled datasets [8]. The core idea is to jointly learn an aligned embedding space for images and text from paired data, maximizing the similarity of matched (“positive”) image-text pairs while pushing apart mismatched (“negative”) pairs via a batch-level InfoNCE loss [61, 18]. Seminal works CLIP [47], ALIGN [21], and BASIC [44] demonstrated remarkable zero-shot transfer capabilities by training on hundreds of millions of image-text pairs. However, standard CLIP-style training samples image–text pairs randomly, without explicitly controlling the difficulty or multiplicity of in-batch negatives. Although higher-similarity negatives already receive greater softmax weight in InfoNCE, their availability depends on batch composition. Hard-negative sampling and synthesis provide explicit control over this training signal [22, 51]. In unimodal self-supervised learning, synthetic hard negatives generated directly in the representation space have proven highly effective [22, 14, 12], offering a fast and lightweight alternative to input-space augmentation. This raises a natural question: can representation-space synthetic negatives transfer to vision-language pretraining? Prior work has explored hard negatives for vision-language models primarily in the input space: rule-based text perturbations [74, 79], LLM-based caption rewrites [10, 43], or diffusion-generated negative images [43]. These approaches incur significant computational overhead and typically address only one modality at a time (cf. Sec. 2). Representation-space generation would sidestep all of these issues. However, as we show in this paper, the transfer is not straightforward, and understanding why it fails reveals fundamental constraints imposed by multimodal representation geometry. Unlike the single-modality setting where all representations inhabit one homogeneous space, vision-language models align two distinct modalities within a shared embedding space. This dual-domain structure introduces two geometric failure modes with no unimodal analogue. First, the cross-modal synthetic negatives that were tested, produced by blending image and text embeddings, fall into the modality gap [30]: a void between the image and text cones that real embeddings never occupy, making such samples trivially separable and uninformative. Second, restricting synthesis to a single modality but involving the positive pair causes positive leakage: the synthetic “negative” inherits genuine positive signal, sending contradictory gradients that confuse optimization. We characterize these geometric effects using representation-level diagnostics. Separately, we observe logit-scale saturation when synthetic hard negatives are introduced with a learnable temperature. Our temperature ablation supports combining positive-free intra-modal synthesis with a fixed temperature (Secs. 4.2 and 4.1). We make the following contributions: (i) We identify and characterize two geometric failure modes that prevent naive transfer of representation-space synthetic negatives to vision-language pretraining, i.e., modality gap collapse and positive leakage, and provide quantitative diagnostics for each (Sec. 4.1). (i) Guided by this analysis, we propose SNAP, a model-agnostic method that generates synthetic hard negatives directly in the representation space for both image and text modalities, applicable on top of InfoNCE-based frameworks such as CLIP and FLIP (Sec. 4.3). (i) We conduct comprehensive evaluations across multiple architectures (ViT-B/16, ViT-B/32, RN-50), datasets (C3M, C12M), and downstream tasks, e.g., zero-shot retrieval (Tab. 5), zero-shot classification (Tab. 6), linear probe evaluation (Tab. 7), demonstrating broad improvements across the evaluated settings. 2 Related Work 2.1 Vision-Language Contrastive Pretraining CLIP [47] and ALIGN [21] established the standard approach for vision-language contrastive pretraining: learning aligned image-text representations via a batch-level InfoNCE loss [61] over large-scale web-crawled data. LiT [76] showed that freezing a pretrained vision encoder and training only the text tower can be surprisingly effective, while SigLIP [75] replaced the softmax-based InfoNCE with a pairwise sigmoid formulation that removes the need for global normalization. BASIC [44] demonstrated the benefits of jointly scaling both encoders and data. Since then, a wide range of methods has advanced the field through modifications to the contrastive objective [69, 67, 16, 27], integration of self-supervised objectives [38, 28, 39, 81, 37, 55], architectural changes [3, 50, 71, 60], training efficiency improvements [29, 68], scaling [57, 58, 72, 65, 20], and improved training signals via synthetic captions [77, 80, 10, 64, 32]. SNAP is complementary to all these directions, augmenting the InfoNCE loss with representation-space synthetic negatives without modifying the architecture, data pipeline, or base objective. 2.2 Hard Negative Sampling The quality and difficulty of negative samples are critical for contrastive learning [53, 56, 63, 13]. In vision, synthetic hard negatives generated in the representation space have been shown to significantly improve self-supervised representation learning [22, 51, 14, 12, 15]. MoCHi [22] mixes hard-negative features, while SynCo [14] studies six synthesis strategies that underpin our analysis. In vision-language learning, linguistic perturbations [74, 79] construct negative captions, and DiHT [46] increases the contribution of hard negatives through importance sampling while NegCLIP [74] and related methods [79] construct negatives through linguistic perturbations. In representation space, m2m^2-Mix within m3m^3-Mix [41] geodesically mixes another matched image–text pair to form cross-modal negatives. SNAP instead synthesizes query-specific negatives within each modality, using hard-negative mixing and noise during pretraining from scratch. 2.3 Synthetic Augmentation for Vision-Language Models Several recent methods synthesize additional training data for CLIP-style pretraining, but these approaches operate in the input space and rely on large external foundation models. On the text side, captions are rewritten or generated using LLMs [10, 43], captioning models [68, 32], or multimodal LLMs [80, 64, 26, 33]. On the image side, TripletCLIP [43] generates hard negative images using text-to-image diffusion models (e.g., SDXL-Turbo [45]) conditioned on LLM-generated captions. These methods introduce external knowledge from models that may overlap with downstream benchmarks, making it difficult to disentangle representational improvements from data leakage, and add substantial computational overhead. SNAP avoids these issues: synthetic negatives are generated from in-batch embeddings alone, requiring no external models and no additional data. More on related work in Appendix Sec. 0.C. 3 Preliminaries In this section, we present the building blocks for integrating synthetic hard negatives into vision-language contrastive learning. We first review the standard CLIP objective (Sec. 3.1), then describe how synthetic negatives augment the contrastive loss in the representation space (Sec. 3.2), and finally define the generation strategies (Sec. 3.3). These preliminaries set up the vocabulary for Sec. 4, where we analyze why naively applying these strategies to the vision-language setting fails and present the design choices that lead to SNAP. Figure 1: Architecture comparison for the image-to-text direction. (a) Standard CLIP encodes images X and captions C into embeddings ∈ℝd×|ℬ|V ^d×|B| and ∈ℝd×|ℬ|T ^d×|B|, computing the similarity matrix ⊤∈|ℬ|×|ℬ|V T∈|B|×|B| using only batch negatives. (b) Our method generates synthetic text negatives ~∈ℝd×|txt| T ^d×|S_txt| directly in the representation space and concatenates them with batch embeddings to form an augmented similarity matrix ⊤[|~]∈|ℬ|×(|ℬ|+|txt|)V [T| T]∈|B|×(|B|+|S_txt|). In the similarity matrices, green cells denote positive pairs, white cells denote batch negatives, and purple cells denote synthetic negatives. The text-to-image direction follows an analogous procedure with synthetic image negatives (cf. Sec. 3.2). 3.1 Contrastive Language-Image Pretraining We assume two parameterized models: an image encoder fθf_θ with parameters θ (e.g., a vision transformer [9] or a convnet [19]) and a text encoder gϕg_φ with parameters ϕφ (e.g., a GPT-style transformer [48, 49]). Each input sample (i,i)(x_i,c_i) consists of an image ix_i and text caption ic_i. The encoders produce embeddings i=fθ(i)v_i=f_θ(x_i) and i=gϕ(i)t_i=g_φ(c_i), which are mapped to a common d-dimensional embedding space via projection layers and ℓ2 _2-normalized, i.e., ‖i‖2=‖i‖2=1\|v_i\|_2=\|t_i\|_2=1. For a batch ℬB, each sample i is associated with one positive pair and |ℬ|−1|B|-1 in-batch negatives. To aid readability, throughout this section we use red to denote batch negatives. The bi-directional contrastive loss is computed as: ℒCLIP=−12|ℬ|(∑i∈ℬlogexp(i⊤i/τ)∑j∈ℬexp(i⊤j/τ)⏟ℒi2t+∑i∈ℬlogexp(i⊤i/τ)∑j∈ℬexp(i⊤j/τ)⏟ℒt2i),L_CLIP=- 12|B| ( _i (v_i t_i/τ) [rgb]0.8438,0.1484,0.2188 _j (v_i t_j/τ)_L_i2t+ _i (t_i v_i/τ) [rgb]0.8438,0.1484,0.2188 _j (t_i v_j/τ)_L_t2i ), (1) where τ is a learnable temperature parameter. We denote the two directional terms as image-to-text (i2t) and text-to-image (t2i), respectively. The denominators include both the positive term j=ij=i and the |ℬ|−1|B|-1 in-batch negatives j≠ij≠ i, following the standard InfoNCE formulation [61]. In the standard CLIP formulation [47], negatives are drawn exclusively from the batch ℬB, requiring large batch sizes (e.g., 32,768) to provide a sufficiently large and diverse pool of negatives. Training with smaller batches significantly limits discriminative ability, as the lack of diverse negatives leads to weaker representations [5]. 3.2 Synthetic Hard Negatives in the Representation Space We augment the denominators of both ℒi2tL_i2t and ℒt2iL_t2i with synthetically generated hard negatives directly in the representation space. To aid readability, we use purple to denote synthetic negatives throughout this section. For the i2t term, we generate a set of synthetic text negatives txti=~kS_txt^i=\ t_k\ for each image query iv_i. Similarly, for the t2i term, we generate a set of synthetic image negatives imgi=~kS_img^i=\ v_k\ for each text query it_i. The augmented losses are: ℒ¯i2t=−1|ℬ|∑i∈ℬlogexp(i⊤i/τ)∑j∈ℬexp(i⊤j/τ)+∑~k∈txtiexp(i⊤~k/τ), L_i2t=- 1|B| _i (v_i t_i/τ) [rgb]0.8438,0.1484,0.2188 _j (v_i t_j/τ)+ [rgb]0.4336,0.1758,0.6602 _ t_k _txt^i (v_i t_k/τ), (2) Each query has its own synthetic-negative set, generated from its selected hard-negative pool; synthetic negatives are not shared across queries. ℒ¯t2i=−1|ℬ|∑i∈ℬlogexp(i⊤i/τ)∑j∈ℬexp(i⊤j/τ)+∑~k∈imgiexp(i⊤~k/τ). L_t2i=- 1|B| _i (t_i v_i/τ) [rgb]0.8438,0.1484,0.2188 _j (t_i v_j/τ)+ [rgb]0.4336,0.1758,0.6602 _ v_k _img^i (t_i v_k/τ). (3) The final loss combines both augmented directional terms: ℒ¯=12(ℒ¯i2t+ℒ¯t2i). L_= 12 ( L_i2t+ L_t2i ). (4) This formulation enriches the negative set with diverse, hard examples that would be computationally prohibitive to generate in the input space, while requiring minimal overhead as the generation occurs directly in the representation space during the forward pass (cf. Sec. 4.3). With synthetic negatives, we produce high-quality, diverse negatives fast and cheaply, enabling effective training even with moderate batch sizes (cf. Appendix Sec. 0.D.1). 3.3 Synthetic Negative Generation Strategies We define hardness by cosine similarity to the query [22]. Given a query image embedding iv_i, we select the top-N batch text negatives ^Ni⊂ℬ∖i T_N^i i with the highest cosine similarity to iv_i; analogously, we select ^Ni V_N^i for text query it_i. We implement six strategies to generate synthetic negatives. For the i2t direction, each strategy produces a synthetic text negative ~k t_k from the query iv_i and one or two hard negatives j,lt_j,t_l sampled uniformly from ^Ni T_N^i ~k(s)=αk⋅i+(1−αk)⋅j,s=1,αk∼(0,0.5)j+βk⋅(j−i),s=2,βk∼(1,1.5)γk⋅j+(1−γk)⋅l,s=3,γk∼(0,1)j+(,σ2⋅),s=4j+δ⋅∇j(i⊤j),s=5j+η⋅sign(∇j(i⊤j)),s=6 [rgb]0.4336,0.1758,0.6602 t_k^(s)= \ array[]l@ l@ l _k·v_i+(1- _k)· [rgb]0.8438,0.1484,0.2188t_j,&s=1,& _k (0,0.5)\\[4.0pt] [rgb]0.8438,0.1484,0.2188t_j+ _k·( [rgb]0.8438,0.1484,0.2188t_j-v_i),&s=2,& _k (1,1.5)\\[4.0pt] _k· [rgb]0.8438,0.1484,0.2188t_j+(1- _k)· [rgb]0.8438,0.1484,0.2188t_l,&s=3,& _k (0,1)\\[4.0pt] [rgb]0.8438,0.1484,0.2188t_j+N(0,σ^2·I),&s=4&\\[4.0pt] [rgb]0.8438,0.1484,0.2188t_j+δ· _ [rgb]0.8438,0.1484,0.2188t_j(v_i [rgb]0.8438,0.1484,0.2188t_j),&s=5&\\[4.0pt] [rgb]0.8438,0.1484,0.2188t_j+η·sign( _ [rgb]0.8438,0.1484,0.2188t_j(v_i [rgb]0.8438,0.1484,0.2188t_j)),&s=6& array . (5) All synthetic negatives are ℓ2 _2-normalized after generation. Strategies s=1s=1 (interpolation) and s=2s=2 (extrapolation) blend the query with hard negatives [78]; s=3s=3 (mixup) combines pairs of hard negatives [78, 62]; s=4s=4 perturbs a hard negative with Gaussian noise; and s=5,6s=5,6 apply gradient-ascent and sign-based adversarial perturbations [17, 35] (see [14] for further detail). Crucially, s=1s=1 and s=2s=2 involve the query embedding iv_i (or it_i in the t2i direction), producing cross-modal synthetic negatives, while s=3s=3 and s=4s=4 operate exclusively on same-modality negatives without involving any positive. As we show in Sec. 4.1, this distinction is critical in the vision-language setting. 4 Adapting Synthetic Negatives to Vision-Language Pretraining The success of representation-space synthetic negatives in unimodal learning (Sec. 3.3) naturally suggests applying the same strategies to CLIP-style training. However, the dual-modality structure of vision-language models introduces failure modes with no unimodal analogue. This raises a central question: What makes synthetic hard negatives work in vision-language pretraining? In this section, we systematically analyze these failure modes (Sec. 4.1) along their multimodal geometry (Sec. 4.2), their secondary effect on the learnable temperature (Sec. 4.1), and present the resulting design choices that define SNAP (Sec. 4.3). All ablations use C3M with ViT-B/16, reporting IN-val and IN-v2 zero-shot accuracy. We generate 64 synthetic negatives per query from the top-256 hardest negatives, split equally among active strategies. The hyperparameters σ=0.01σ=0.01, δ=0.01δ=0.01, and η=0.01η=0.01 are set empirically without extensive tuning, following [14]. 4.1 Naive Extension and Failure Modes (a) Cross-modal Failure (b) Positive Leakage (c) SNAP (Ours) Figure 2: Geometric intuition of failure modes in vision-language pretraining. (a) Cross-modal failure: Blending image and text embeddings creates samples in the “modality gap” that are trivially rejected (Sec. 4.1). (b) Positive Leakage: Including the positive pair in synthesis creates negatives that contain the actual signal, causing gradient conflict (Sec. 4.1). (c) SNAP: By generating intra-modal negatives and filtering the positive anchor, we create high-quality hard negatives that respect the modality geometry (Sec. 4.3). Method Strategies Mode IN-val IN-v2 CLIP [47] n/a n/a 10.4 8.4 SNAP all 6 cross 7.6 (↓ ) 6.8 (↓ ) SNAP all 6 intra 8.7 (↓ ) 7.5 (↓ ) SNAP s=3,4s=3,4 intra 10.0 (↓ ) 8.0 (↓ ) Table 1: Cross-modal vs. intra-modal strats. Applying all six strategies in cross-modal mode degrades performance below the CLIP baseline. Converting to intra-modal mode helps but still underperforms. Restricting to s=3s=3 and s=4s=4 recovers performance. We highlight the default hyperparameter. Figure 3: Pairwise strategy complementarity on IN-val. Strategies s=3,4s=3,4 achieve the highest accuracy. Cross-modal synthetic negatives fail due to geometry. The most direct extension is to apply all six strategies from Eq. 5 as defined, where s=1s=1 and s=2s=2 blend the image query iv_i with text negatives jt_j. However, vision-language models exhibit a well-documented modality gap [30] (Fig. 2(a)): image and text embeddings cluster in distinct cones of the shared space, and cross-modal interpolations such as α⋅i+(1−α)⋅jα·v_i+(1-α)·t_j produce vectors in the void between them that real embeddings never occupy, making them trivially separable. As shown in Table 1, restricting to s=3,4s=3,4 in intra-modal mode nearly recovers the CLIP baseline (10.0 vs. 10.4 on IN-val); as we show in Section 4.1, the remaining gap is closed and surpassed once the temperature is fixed. Intra-modal negatives involving the positive fail due to semantic leakage. A natural fix is to convert all strategies to intra-modal variants by replacing the cross-modal query iv_i in s=1,2s=1,2 with the matched text positive it_i. However, this introduces a subtler failure. Consider the intra-modal interpolation: ~=α⋅i+(1−α)⋅j t=α·t_i+(1-α)·t_j, where it_i is the positive caption for query iv_i and jt_j is a different caption. Because it_i is the correct match for iv_i, the synthetic “negative” ~ t inherits positive signal. As α→1α→ 1, it becomes nearly identical to the true positive, yet the InfoNCE denominator treats it as a negative. This positive leakage (Fig. 2(b)) sends contradictory gradients: the numerator of Eq. 2 encourages similarity with it_i, while the denominator penalizes similarity with ~≈i t _i. Tab. 1 confirms that intra-modal generation with all six strategies (8.7 on IN-val) improves over the cross-modal variant but still underperforms the CLIP baseline. Two distinct failure modes, one solution. The tested constructions alter negative geometry through cross-modal operations or introduce positive signal through explicit inclusion of the matched positive. Strategies s∈3,4s∈\3,4\ avoid both by generating negatives from non-matching pairs only. Restricting to s∈3,4s∈\3,4\ nearly recovers the CLIP baseline (10.0 vs. 10.4 on IN-val, 8.0 vs. 8.4 on IN-v2; Table 1), with the remaining gap closed and surpassed once temperature is fixed (Section 4.1). Figure 3 confirms this: s=3s=3 and s=4s=4 consistently outperform any combination involving cross-modal blending or the positive pair. Temperature divergence. Adding synthetic hard negatives changes the contrastive denominator and its interaction with the learnable temperature τ. In the evaluated configuration, CLIP+SNAP with learnable τ reaches the OpenCLIP logit-scale clamp of 100 within half an epoch after synthetic negatives are enabled (Fig. 4). This occurs with the positive-free intra-modal strategies s=3,4s=3,4. Fixing τ=0.07τ=0.07 improves IN-val accuracy from 10.0 to 11.0 and IN-v2 accuracy from 8.0 to 9.6 (Tab. 2). Under the same fixed temperature, CLIP without synthetic negatives achieves 10.1 and 8.0, respectively. We therefore adopt fixed τ=0.07τ=0.07 as part of the SNAP training recipe. The two cross-modal strategies fail through distinct mechanisms: extrapolation (s=2s=2) pushes vectors beyond the text cone, while interpolation (s=1s=1) pulls toward the image query and thus the positive, making positive leakage the dominant failure rather than trivial separability, as we quantify in Section 4.2. Table 2: Temperature ablation. With fixed τ=0.07τ=0.07, CLIP+SNAP outperforms both fixed- and learnable-temperature CLIP. We highlight the default setting. Method τ IN-val IN-v2 CLIP learn. 10.4 8.4 CLIP fixed 10.1 8.0 CLIP+SNAP learn. 10.0(↓)10.0\, [rgb]1,0,0( ) 8.0(↓)8.0\, [rgb]1,0,0( ) CLIP+SNAP fixed 11.0(↑)11.0\, [rgb]0,1,0( ) 9.6(↑)9.6\, [rgb]0,1,0( ) Figure 4: Logit scale trajectories (ViT-B/32, C3M, epochs 5 to 10). CLIP+SNAP with a learnable τ saturates the OpenCLIP clamp within half an epoch. Both fixed-τ runs remain stable. Figure 5: Positive leakage per strategy (ViT-B/32, C3M). mean cos(~,i) ( t,t_i) per strategy; dashed line marks the random baseline (0.190.19). Table 3: TSR and relative hardness. Lower TSR and higher Δ hardness indicate harder, more informative negatives. We highlight SNAP strategies. Strategy TSR (%)↓ Δ hardness↑ s=1s=1 (cross) 0.3 +0.281+0.281 s=2s=2 (cross) 87.8 −0.189-0.189 s=1s=1 (intra) 0.6 +0.133+0.133 s=2s=2 (intra) 74.0 −0.096-0.096 s=3s=3 (SNAP) 2.2 +0.041+0.041 s=4s=4 (SNAP) 13.2 +0.045+0.045 4.2 Multimodal Geometry of Synthetic Negatives We characterize the failure modes in Sec. 4.1 using four complementary representation level diagnostics computed on ViT-B/32 trained on C3M. Leakage, separability, and hardness diagnostics. We quantify the failure modes with three metrics over 2,000 validation pairs (ViT-B/32, C3M): positive leakage cos(~,i) ( t,t_i), where values above the random baseline (0.190.19) indicate semantic contamination; trivially separable rate (TSR), the fraction of synthetics easier than the 10th percentile of real hard negatives; and relative hardness Δ=cos(i,~)−cos(i,rand) = (v_i, t)- (v_i,t_rand). As shown in Figs. 5 and 3, cross-modal strategies fail for opposite reasons: s=2s=2 (cross) lands in the void beyond the text cone (TSR 87.8%87.8\%, Δ=−0.189 =-0.189), while s=1s=1 (cross) interpolates toward the positive (leakage 0.330.33). Intra-modal strategies involving the positive suffer from contamination (s=1s=1 intra: 0.480.48; s=2s=2 intra: 0.410.41) or trivial separability (s=2s=2 intra: TSR 74.0%74.0\%). Only s=3s=3 and s=4s=4 avoid cross-modal mixing and explicit positive inclusion, exhibiting leakage close to the random baseline (0.230.23 and 0.190.19), low TSR (2.2%2.2\% and 13.2%13.2\%), and positive relative hardness. PCA subspace visualization. Fig. 6 projects image embeddings, text embeddings, and synthetic negatives onto the first two principal components. Cross-modal strategies scatter into the void between the image and text cones, a region real embeddings never occupy. Intra-modal strategies involving the positive cluster within the text cone but near the positive anchor, explaining the observed leakage. SNAP strategies (s=3s=3, s=4s=4) are confined within the text cone and distributed away from the positive, occupying the same subspace as real text negatives. Figure 6: PCA subspace of synthetic negatives (ViT-B/32, C3M). (a) Cross-modal negatives land in the gap void between modality cones. (b) Intra-modal negatives involving the positive cluster near the positive anchor, causing leakage. (c) SNAP negatives (s=3s=3, s=4s=4) are confined within the text cone and away from the positive. 4.3 SNAP Guided by the analysis in Secs. 4.1 and 4.2, we propose SNAP (Synthetically Negative Augmented Pretraining) that makes three principled design choices. Table 4: Synthesis and directionality ablations on C3M. Variant IN-val IN-v2 Random-neg. dup. 10.0 8.0 Hard-neg. dup. 10.4 8.5 Hard-neg. wt. 10.5 8.7 s=3s=3 10.8 9.2 s=4s=4 10.6 8.8 s=3,4s=3,4 (i2t only) 10.7 8.8 s=3,4s=3,4 (t2i only) 10.6 8.7 SNAP 11.0 9.6 Intra-modal, positive-free strategies. SNAP uses only strategies s=3s=3 and s=4s=4 from Eq. 5, both operating in intra-modal mode (Fig. 2(c)). These avoid cross-modal geometric collapse and intra-modal positive leakage (Tabs. 1 and 3). Fixed temperature. SNAP fixes the temperature τ rather than learning it, preventing the logit scale saturation that arises when hard negatives interact with a learnable temperature (Tab. 2). Bidirectional synthesis. SNAP generates synthetic negatives for both i2t and t2i directions. We compare against repeating randomly sampled or hard negatives in the contrastive denominator, and increasing the weight of hard-negative terms. Both synthesis strategies outperform these controls, with their combination performing best (Tab. 4). Bidirectional synthesis also outperforms either direction alone. Computational overhead. Since SNAP generates negatives via vector operations—specifically additions, scalar multiplications, and normalization—on pre-computed embeddings, it avoids additional forward or backward passes through the encoders. This design ensures minimal impact on training efficiency, requiring no external generative models or architectural modifications. On a ViT-B/16 backbone, CLIP [47] requires 32.5 minutes per epoch, while SNAP adds only 3 minutes, resulting in an 9.23% overhead. Similarly, FLIP [29] requires 14.9 minutes per epoch, and the addition of SNAP increases this to 16.2 minutes, a comparable overhead of 8.72%. These results demonstrate that SNAP provides significant discriminative gains with a negligible increase in training time. Further ablations on batch size and the number of synthetic negatives are provided in the supplementary material (Appendix Sec. 0.D). 5 Experiments We train the CLIP and FLIP baselines and their SNAP variants under matched data and training configurations, except for the temperature treatment described above. We also train SigLIP [75] under our training setup. Results for LaCLIP [10], TripletCLIP [43], and NegCLIP [74] are taken from [43] and included for reference only. Detailed implementation details in Appendix Sec. 0.B. 5.1 Implementation Details Pretraining datasets. We pretrain on two image-text datasets of varying scale: C3M [54] and C12M [4]. Due to image link rot, the versions we obtained contain fewer samples than the original releases: ≈ 1.7M out of 3.3M for C3M and ≈ 7.2M out of 12.4M for C12M. This may lead to minor performance differences compared to models trained on the full datasets. Model architectures. For the image encoder fθf_θ, we use ViT-B/16, ViT-B/32 [9], and ResNet-50 [19]. For the text encoder gϕg_φ, we use a 12-layer Transformer [47]. Both encoders include projection heads mapping to a shared d=512d=512 dimensional embedding space. Training setup. We use AdamW [34] with learning rate 5×10−45× 10^-4, weight decay 0.20.2, and cosine scheduling with 10k warmup steps. Training uses batch size |ℬ|=4096|B|=4096 on 4× A100 GPUs for 32 epochs, with 224×224224× 224 images and text length 77. The temperature τ is initialized to 0.070.07 (and kept constant for SNAP). Reproducibility. For C3M, we pretrain three checkpoints (different seeds) and report the mean, with standard deviations as subscripts in Tables 5, 6 and 7; for C12M, a single checkpoint is used due to computational constraints. 5.2 Zero-Shot Image-Text Retrieval We evaluate cross-modal retrieval on Flickr30k [70] and MSCOCO [31], computing pairwise cosine similarities over the full test split and reporting Recall@1,5,10 for both image-to-text and text-to-image retrieval in Table 5. SNAP improves retrieval in the large majority of settings for both CLIP and FLIP, holding across all three backbones and both pretraining scales, with gains appearing on both retrieval directions rather than favoring one; the few exceptions are isolated cells that are unchanged or lower by at most a few tenths of a point. Table 5: Zero-shot image-text retrieval on Flickr30k and MSCOCO. We report Recall@K (R@K) metrics for text and image retrieval. Symbols: ‡ Taken from [43] Data Method Text Retrieval Image Retrieval Flickr30k MSCOCO Flickr30k MSCOCO R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 ViT-B/32 C3M SigLIP [75] 7.2.2 19.8.3 26.5.1 3.1.3 10.0.1 15.7.2 5.3.1 15.2.2 22.3.1 2.8.1 9.0.1 14.0.1 NegCLIP [74]‡ 4.9 13.8 19.6 1.7 6.3 10.4 4.8 13.0 18.5 2.0 6.6 10.6 LaCLIP [10]‡ 3.7 10.9 16.0 1.6 5.1 8.9 3.5 10.8 16.0 1.7 6.0 9.5 TripletCLIP [43]‡ 9.1 22.0 29.8 3.2 10.4 16.2 8.4 22.0 29.6 3.6 11.3 17.4 CLIP [47] 9.8.2 25.5.3 35.7.2 5.0.2 15.0.1 22.3.1 6.9.1 19.8.2 28.5.1 4.2.2 12.7.2 19.0.1 + SNAP 10.4.2 26.3.1 36.9.3 5.5.2 15.2.2 23.0.2 7.9.1 20.4.2 28.9.1 4.1.1 13.6.1 20.1.2 FLIP [29] 6.3.3 18.5.1 26.4.2 3.3.3 10.7.3 16.7.3 5.8.3 16.9.1 24.0.3 3.1.1 9.6.1 14.7.2 + SNAP 6.8.1 19.0.1 26.9.3 3.7.1 11.1.3 17.1.1 6.3.1 17.4.3 24.5.1 3.5.1 10.0.2 15.1.2 C12M SigLIP [75] 31.4 57.2 69.5 16.2 36.1 47.2 23.1 48.8 60.8 11.5 28.1 38.4 NegCLIP [74]‡ 24.7 46.6 58.2 12.3 30.2 41.4 18.7 41.7 53.4 8.6 23.1 32.6 LaCLIP [10]‡ 21.3 42.7 54.6 10.5 25.9 35.6 15.1 36.3 47.0 7.2 19.8 28.4 TripletCLIP [43]‡ 28.0 55.9 65.7 14.6 33.0 43.8 25.3 52.4 63.3 11.4 28.5 39.0 CLIP [47] 33.1 59.5 71.2 17.8 39.4 50.3 24.2 50.9 63.0 13.1 30.5 41.2 + SNAP 33.8 60.0 71.9 18.6 39.8 51.3 24.9 51.4 63.5 13.7 31.0 42.6 FLIP [29] 26.2 50.1 61.8 14.4 32.1 43.5 20.2 44.1 56.1 10.7 26.2 36.4 + SNAP 27.2 50.9 63.1 15.1 32.7 44.5 20.9 45.4 56.8 10.4 26.8 37.2 ViT-B/16 C3M SigLIP [75] 13.6.3 31.5.1 43.6.1 7.0.2 19.8.1 28.1.2 9.3.1 24.5.2 33.4.1 5.6.1 15.8.2 23.1.3 CLIP [47] 16.2.3 37.8.2 49.6.2 7.7.2 21.5.2 31.5.2 12.0.2 28.4.2 38.2.1 6.6.3 18.1.3 26.2.1 + SNAP 17.1.3 38.7.2 50.5.1 8.1.1 22.1.3 32.7.3 12.8.1 28.9.1 39.2.2 7.2.2 18.4.1 26.8.1 FLIP [29] 12.1.2 31.2.2 42.1.1 6.4.2 17.7.1 26.4.1 10.0.2 24.8.1 33.6.3 5.1.1 15.4.1 22.7.2 + SNAP 12.3.2 31.7.3 43.1.1 7.1.3 18.9.2 27.0.3 10.9.3 25.6.3 34.8.1 5.9.1 16.3.3 24.0.2 C12M SigLIP [75] 49.4 74.2 83.5 26.1 50.4 62.1 34.2 61.9 73.1 19.1 40.5 52.1 CLIP [47] 53.6 77.5 85.9 29.4 53.8 65.8 38.5 66.2 76.9 21.8 44.4 56.1 + SNAP 55.1 78.3 87.1 30.2 54.7 67.1 39.0 67.6 78.2 22.3 45.0 56.6 FLIP [29] 47.9 73.2 82.2 25.5 49.1 61.2 33.6 60.5 72.1 18.8 39.4 50.9 + SNAP 48.5 74.1 82.3 26.4 49.6 62.2 34.3 61.0 72.7 19.5 39.9 52.1 RN-50 C3M SigLIP [75] 19.4.1 44.8.1 56.4.2 10.5.1 27.0.1 37.5.3 14.0.2 31.5.2 42.0.1 8.0.1 21.3.1 29.8.1 CLIP [47] 20.8.3 45.8.3 56.8.1 11.7.3 29.6.3 41.4.1 16.1.2 36.5.2 47.1.1 9.6.1 24.3.2 34.0.2 + SNAP 21.1.2 47.2.1 58.1.1 11.6.1 29.4.1 42.1.1 16.0.1 37.4.1 47.8.1 10.9.1 25.3.3 34.9.3 FLIP [29] 20.6.1 44.2.2 56.5.2 11.3.1 29.4.2 40.4.1 15.8.2 36.0.1 47.0.3 8.8.2 23.6.1 33.1.3 + SNAP 21.2.1 44.6.2 57.2.1 12.2.3 29.9.3 40.9.2 16.2.3 36.9.1 46.9.2 9.2.3 24.0.2 34.5.2 5.3 Zero-Shot Classification We evaluate zero-shot classification on ImageNet [8, 52] and 10 downstream datasets spanning fine-grained recognition (Stanford Cars [24], FGVC Aircraft [36]), scene recognition (SUN397 [66]), texture classification (DTD [7]), and generic object classification (CIFAR [25]), encoding class names as text prompts and predicting via cosine similarity without any task-specific training (Table 6). SNAP improves accuracy across settings, with the clearest gains on the fine-grained Cars and Aircraft benchmarks, where C3M ViT-B/32 accuracy roughly doubles (e.g., Cars 0.8→1.90.8→ 1.9, Aircraft 1.9→3.41.9→ 3.4 for CLIP), while general benchmarks such as ImageNet show steady improvements. Table 6: Zero-shot classification. Top-1 accuracy on ImageNet and 10 downstream datasets spanning fine-grained, scene, and object recognition. ‡Taken from [43]. Data Model Food-101 CIFAR-10 CIFAR-100 SUN397 Cars Aircraft DTD Pets Caltech-101 Flowers ImageNet ViT-B/32 C3M SigLIP [75] 3.8.3 23.8.3 8.1.2 12.4.2 0.8.2 0.9.1 4.4.2 4.1.1 26.6.2 4.2.2 5.6.1 NegCLIP [74]‡ – – 8.2 – – 1.0 6.8 4.1 29.4 3.7 4.6 LaCLIP [10]‡ – – 8.0 – – 0.8 5.5 3.1 24.6 3.4 3.7 TripletCLIP [43]‡ – – 11.8 – – 0.9 8.9 5.7 38.5 8.2 7.3 CLIP [47] 5.9.2 28.7.2 10.0.3 14.5.1 0.8.3 1.9.3 8.2.2 7.0.3 41.5.3 5.9.1 8.0.3 + SNAP 6.5.3 29.3.2 10.4.2 14.7.2 1.9.3 3.4.2 9.0.1 7.3.3 42.1.2 7.3.2 8.2.2 FLIP [29] 4.7.1 26.8.2 8.7.3 11.4.2 0.6.2 1.1.1 4.5.1 5.3.3 37.1.2 3.9.1 6.8.3 + SNAP 5.2.1 28.1.2 9.2.2 12.4.3 1.3.1 2.6.2 4.6.1 5.8.2 38.0.2 4.5.3 7.2.1 C12M SigLIP [75] 11.2 45.4 28.1 31.2 3.4 2.1 15.6 32.5 64.1 14.2 22.1 NegCLIP [74]‡ – – 26.0 – – 1.5 14.2 30.8 63.4 13.5 20.2 LaCLIP [10]‡ – – 23.6 – – 1.5 13.0 34.7 62.9 13.4 19.7 TripletCLIP [43]‡ – – 30.3 – – 1.2 17.4 33.2 65.3 15.4 23.3 CLIP [47] 13.1 50.2 31.5 35.1 4.1 2.5 18.2 35.8 67.4 16.8 24.6 + SNAP 13.5 50.6 32.7 36.2 5.0 3.0 19.1 36.7 68.1 17.3 24.8 FLIP [29] 11.8 46.5 28.9 32.4 3.5 2.2 16.1 33.4 64.8 14.9 22.8 + SNAP 13.2 47.7 29.5 33.3 4.0 2.0 17.0 34.3 65.6 15.7 23.9 ViT-B/16 C3M SigLIP [75] 6.2.3 28.9.3 7.0.1 18.3.2 1.1.3 1.2.3 8.7.1 6.0.2 35.5.2 7.0.1 9.9.2 CLIP [47] 7.4.1 34.3.2 9.8.2 19.0.3 0.9.3 1.2.1 10.1.1 9.0.2 46.3.1 8.4.3 10.4.2 + SNAP 8.4.1 34.4.1 10.5.3 19.6.1 1.3.2 2.0.1 10.8.1 10.5.1 47.3.1 9.0.2 11.0.2 FLIP [29] 5.3.2 24.1.3 9.3.3 17.4.1 1.1.2 1.4.3 8.3.2 6.4.2 45.0.2 6.6.1 10.3.3 + SNAP 6.4.2 25.3.3 10.5.1 18.2.1 1.7.3 1.9.3 9.1.3 7.9.2 45.7.2 8.0.1 11.0.2 C12M SigLIP [75] 19.8 58.4 36.2 41.5 6.2 4.1 22.1 51.2 78.4 24.1 31.2 CLIP [47] 22.4 63.1 39.5 44.8 7.5 4.8 24.8 55.4 82.1 27.5 34.6 + SNAP 23.1 63.0 40.1 46.2 7.9 4.6 25.4 56.1 83.2 28.5 35.3 FLIP [29] 20.2 59.8 37.5 42.6 6.5 4.3 22.8 52.4 79.8 25.3 32.4 + SNAP 20.8 60.1 38.6 43.8 6.7 5.6 23.3 52.3 81.0 26.0 33.8 RN-50 C3M SigLIP [75] 7.5.2 17.9.2 5.2.1 21.1.1 1.1.1 1.5.2 9.4.2 8.8.1 38.8.1 9.5.1 13.1.2 CLIP [47] 7.7.1 23.7.1 4.8.3 23.3.1 1.4.2 1.3.2 9.3.1 11.7.2 45.4.2 12.3.2 13.6.1 + SNAP 8.2.1 24.4.2 5.8.1 24.0.1 1.9.3 2.1.2 10.8.1 12.8.2 46.1.1 12.1.2 14.7.2 FLIP [29] 9.0.1 23.1.1 6.7.2 22.4.3 0.8.1 1.1.2 10.9.1 9.0.2 41.8.2 11.7.1 13.7.1 + SNAP 9.5.3 22.9.1 7.4.2 23.2.1 1.6.2 1.9.2 11.7.1 10.4.2 42.2.1 13.0.2 14.9.1 5.4 Linear Probe Classification Following standard protocols [47, 23], we freeze the image encoder and train a linear classifier, reporting top-1 test accuracy (Tab. 7). SNAP consistently produces more linearly separable visual features across backbones and data scales. Table 7: Linear probe evaluation. Top-1 accuracy of a linear classifier trained on frozen image encoder features across ImageNet and 10 transfer datasets. Data Model Food-101 CIFAR-10 CIFAR-100 SUN397 Cars Aircraft DTD Pets Caltech-101 Flowers ImageNet ViT-B/32 C3M SigLIP [75] 35.4.1 72.5.1 48.4.3 29.4.1 9.5.1 13.6.2 31.9.2 34.0.2 61.6.2 50.8.2 30.2.1 CLIP [47] 40.7.3 77.2.3 54.6.1 32.7.1 13.2.2 20.1.3 38.3.3 40.7.1 71.1.1 61.5.1 33.9.1 + SNAP 41.3.2 78.5.2 55.2.3 33.1.3 13.9.1 20.7.1 38.9.3 41.4.3 72.0.1 62.7.2 34.7.1 FLIP [29] 40.8.1 74.8.2 52.8.2 34.7.3 13.9.2 17.9.2 39.8.2 37.6.2 69.7.2 61.1.2 32.9.2 + SNAP 41.4.1 75.6.2 53.6.1 36.1.3 14.5.1 18.7.1 41.3.1 37.5.2 70.8.1 62.0.2 33.5.3 C12M SigLIP [75] 54.2 84.6 63.8 44.1 24.8 31.2 48.9 56.4 79.2 78.4 45.8 CLIP [47] 58.6 88.5 68.4 47.9 28.5 36.8 54.2 62.1 85.1 85.9 48.7 + SNAP 59.2 89.1 68.3 49.3 28.9 38.0 54.7 63.4 85.8 86.3 49.8 FLIP [29] 57.5 87.2 67.1 46.8 27.4 35.1 53.0 60.9 83.9 84.1 47.5 + SNAP 58.0 88.1 68.0 47.3 27.2 35.3 53.0 61.7 84.9 84.7 48.6 ViT-B/16 C3M SigLIP [75] 46.3.2 73.7.2 50.3.1 37.6.3 14.7.1 21.7.2 45.0.1 47.2.1 70.8.2 66.8.1 41.3.2 CLIP [47] 49.2.3 77.1.2 55.2.2 40.3.1 18.1.3 24.6.2 47.8.2 51.2.2 73.7.1 71.6.3 43.1.2 + SNAP 49.7.3 77.5.1 56.6.1 41.0.1 19.2.1 25.3.1 48.1.3 52.2.2 74.4.3 72.3.3 43.7.2 FLIP [29] 48.6.1 75.0.1 52.9.3 41.0.2 16.0.2 21.3.3 46.2.1 47.4.3 73.2.2 70.6.3 40.9.1 + SNAP 49.3.3 74.9.1 53.0.2 41.1.2 16.6.2 22.7.1 46.0.3 47.4.2 74.4.1 70.9.1 41.7.1 C12M SigLIP [75] 64.1 85.8 66.2 51.4 32.5 38.7 58.1 65.8 84.2 86.4 55.4 CLIP [47] 68.4 89.2 70.5 55.1 36.8 43.1 62.4 70.5 88.5 90.1 58.7 + SNAP 68.8 90.0 71.3 56.0 38.0 43.8 63.2 71.1 89.6 90.6 59.9 FLIP [29] 67.1 88.0 69.3 54.0 35.5 41.8 61.2 69.3 87.2 88.9 57.5 + SNAP 67.8 87.8 69.7 54.7 36.2 42.4 62.4 70.6 87.8 88.8 58.7 RN-50 C3M SigLIP [75] 50.9.1 70.0.2 46.0.2 45.6.1 17.0.1 20.9.2 49.5.2 51.3.2 73.5.3 71.5.2 47.2.2 CLIP [47] 56.1.1 74.0.3 52.3.3 50.0.3 24.6.3 31.1.3 53.0.3 62.4.2 80.4.3 80.4.3 50.0.2 + SNAP 56.7.3 74.1.1 52.8.1 50.8.1 25.3.3 31.8.1 54.3.2 63.6.2 81.1.1 80.8.1 51.4.3 FLIP [29] 56.7.2 75.5.1 53.6.3 49.0.1 24.6.1 31.0.2 54.4.2 62.2.3 80.1.1 78.8.1 50.0.3 + SNAP 56.8.1 76.1.1 54.0.1 49.6.3 25.4.2 31.7.2 55.8.2 63.1.2 80.7.2 80.0.1 50.7.1 6 Discussion & Limitations The failure modes we identify arise from the dual-modality structure of contrastive learning rather than from a CLIP-specific architectural choice. We therefore expect the design principles of SNAP to be relevant to other dual-encoder frameworks using representation-space synthetic negatives. Our evaluation is limited to C3M and C12M due to computational constraints; scaling to larger datasets such as LAION-400M remains future work. Acknowledgements We acknowledge the computational resources and support provided by the Imperial College Research Computing Service (http://doi.org/10.14469/hpc/2232), which enabled our experiments. References [1] R. Beaumont (2021) Img2dataset: easily turn large sets of image urls to an image dataset. GitHub. Note: https://github.com/rom1504/img2dataset Cited by: §0.A.1, Appendix 0.H. [2] L. Bossard, M. Guillaumin, and L. Van Gool (2014) Food-101 – Mining Discriminative Components with Random Forests. In Eur. Conf. Comput. Vis. (ECCV), p. 446–461. Cited by: Table 8. [3] A. Cao, X. Wei, and Z. Ma (2025) FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-training. In CVPR, Cited by: §2.1. [4] S. Changpinyo, P. Sharma, N. Ding, and R. Soricut (2021) Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts. In CVPR, p. 3558–3568. Cited by: §0.A.1, §5.1. [5] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton (2020) A simple framework for contrastive learning of visual representations. In ICML, p. 1597–1607. Cited by: §3.1. [6] CLIP benchmark External Links: Document, Link Cited by: Appendix 0.B, Appendix 0.H. [7] M. Cimpoi, S. Maji, I. Kokkinos, and A. Vedaldi (2014) Describing Textures in the Wild. In IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), p. 3606–3613. Cited by: Table 8, §5.3. [8] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei (2009) ImageNet: A large-scale hierarchical image database. In CVPR, Vol. , p. 248–255. External Links: Document Cited by: §0.A.2, Table 8, §1, §5.3. [9] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby (2021) An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, External Links: Link Cited by: §0.B.1, §3.1, §5.1. [10] L. Fan, D. Krishnan, P. Isola, D. Katabi, and Y. Tian (2023) Improving CLIP Training with Language Rewrites. In NeurIPS, Cited by: Appendix 0.C, Appendix 0.G, §1, §2.1, §2.3, Table 5, Table 5, Table 6, Table 6, §5. [11] L. Fei‑Fei, R. Fergus, and P. Perona (2007) Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories. Computer Vision and Image Understanding 106 (1), p. 59–70. Cited by: Table 8. [12] N. Giakoumoglou, A. Floros, K. M. Papadopoulos, and T. Stathaki (2025) Unsupervised Training of Vision Transformers with Synthetic Negatives. In Second Workshop on Visual Concepts, External Links: Link Cited by: §1, §2.2. [13] N. Giakoumoglou, T. Stathaki, and A. Gkelias (2025) A Review on Discriminative Self-supervised Learning Methods in Computer Vision. External Links: Link, 2405.04969 Cited by: §2.2. [14] N. Giakoumoglou and T. Stathaki (2024) SynCo: Synthetic Hard Negatives for Contrastive Visual Representation Learning. External Links: Link, 2410.02401 Cited by: §1, §2.2, §3.3, §4. [15] N. Giakoumoglou, A. Floros, K. Papadopoulos, and T. Stathaki (2026) ViTAMINS: An Empirical Study of Training Self-Supervised Vision Transformers with Synthetic Hard Negatives. External Links: 2609.01041, Link Cited by: §2.2. [16] S. Goel, H. Bansal, S. Bhatia, R. A. Rossi, V. Vinay, and A. Grover (2022) CyCLIP: cyclic contrastive language-image pretraining. External Links: 2205.14459, Link Cited by: Appendix 0.G, §2.1. [17] I. J. Goodfellow, J. Shlens, and C. Szegedy (2015) Explaining and Harnessing Adversarial Examples. External Links: Link, 1412.6572 Cited by: §3.3. [18] R. Hadsell, S. Chopra, and Y. LeCun (2006) Dimensionality reduction by learning an invariant mapping. In CVRP, Vol. 2, p. 1735–1742. Cited by: §1. [19] K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep Residual Learning for Image Recognition. CVPR, p. 770–778. External Links: Link Cited by: §3.1, §5.1. [20] OpenCLIP Note: If you use this software, please cite it as below. External Links: Document, Link Cited by: Appendix 0.B, Appendix 0.H, §2.1. [21] C. Jia, Y. Yang, Y. Xia, Y. Chen, Z. Parekh, H. Pham, Q. V. Le, Y. Sung, Z. Li, and T. Duerig (2021) Scaling up visual and vision-language representation learning with noisy text supervision. External Links: 2102.05918, Link Cited by: §1, §2.1. [22] Y. Kalantidis, M. B. Sariyildiz, N. Pion, P. Weinzaepfel, and D. Larlus (2020) Hard Negative Mixing for Contrastive Learning. In NeurIPS, Cited by: §1, §1, §2.2, §3.3. [23] S. Kornblith, J. Shlens, and Q. V. Le (2019) Do Better ImageNet Models Transfer Better?. External Links: Link, 1805.08974 Cited by: §5.4. [24] J. Krause, M. Stark, J. Deng, and L. Fei‑Fei (2013) 3D Object Representations for Fine‑Grained Categorization. In IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), p. 554–561. Cited by: Table 8, §5.3. [25] A. Krizhevsky (2009) Learning Multiple Layers of Features from Tiny Images. Technical report Technical Report 0, University of Toronto. Cited by: Table 8, Table 8, §5.3. [26] Z. Lai, H. Zhang, B. Zhang, W. Wu, H. Bai, A. Timofeev, X. Du, Z. Gan, J. Shan, C. Chuah, Y. Yang, and M. Cao (2024) VeCLIP: Improving CLIP Training via Visual-enriched Captions. External Links: 2310.07699 Cited by: §2.3. [27] S. Lavoie, P. Kirichenko, M. Ibrahim, M. Assran, A. G. Wilson, A. Courville, and N. Ballas (2024) Modeling caption diversity in contrastive vision-language pretraining. In ICML, External Links: Link Cited by: §2.1. [28] Y. Li, F. Liang, L. Zhao, Y. Cui, W. Ouyang, J. Shao, F. Yu, and J. Yan (2022) Supervision exists everywhere: a data efficient contrastive language-image pre-training paradigm. In ICLR, External Links: Link Cited by: Appendix 0.G, §2.1. [29] Y. Li, H. Fan, R. Hu, C. Feichtenhofer, and K. He (2023) Scaling language-image pre-training via masking. External Links: 2212.00794, Link Cited by: Appendix 0.G, §2.1, §4.3, Table 5, Table 5, Table 5, Table 5, Table 5, Table 6, Table 6, Table 6, Table 6, Table 6, Table 7, Table 7, Table 7, Table 7, Table 7. [30] W. Liang, Y. Zhang, Y. Kwon, S. Yeung, and J. Zou (2022) Mind the gap: understanding the modality gap in multi-modal contrastive representation learning. External Links: 2203.02053, Link Cited by: §1, §4.1. [31] T. Lin, M. Maire, S. J. Belongie, L. D. Bourdev, R. B. Girshick, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick (2014) Microsoft COCO: Common Objects in Context.. CoRR abs/1405.0312. External Links: Link Cited by: §0.A.2, Table 8, §5.2. [32] Y. Liu, X. Li, Z. Wang, B. Zhao, and C. Xie (2024) CLIPS: An Enhanced CLIP Framework for Learning with Synthetic Captions. arXiv preprint arXiv:2411.16828. Cited by: §2.1, §2.3. [33] Y. Liu, K. Wang, W. Shao, P. Luo, Y. Qiao, M. Z. Shou, K. Zhang, and Y. You (2024) MLLMs-Augmented Visual-Language Representation Learning. External Links: Link, 2311.18765 Cited by: §2.3. [34] I. Loshchilov and F. Hutter (2019) Decoupled Weight Decay Regularization. In ICLR, External Links: Link Cited by: 9(a), §5.1. [35] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu (2018) Towards Deep Learning Models Resistant to Adversarial Attacks. In ICLR, External Links: Link Cited by: §3.3. [36] S. Maji, E. Rahtu, J. Kannala, M. B. Blaschko, and A. Vedaldi (2013) Fine‑Grained Visual Classification of Aircraft. arXiv abs/1306.5151. External Links: 1306.5151 Cited by: Table 8, §5.3. [37] K. Maninis, K. Chen, S. Ghosh, A. Karpur, K. Chen, Y. Xia, B. Cao, D. Salz, G. Han, J. Dlabal, D. Gnanapragasam, M. Seyedhosseini, H. Zhou, and A. Araujo (2025) TIPS: Text-Image Pretraining with Spatial Awareness. In ICLR, Cited by: §2.1. [38] N. Mu, A. Kirillov, D. Wagner, and S. Xie (2022) SLIP: self-supervision meets language-image pre-training. In European Conference on Computer Vision, p. 523–540. Cited by: Appendix 0.G, §2.1. [39] M. F. Naeem, Y. Xian, X. Zhai, L. Hoyer, L. V. Gool, and F. Tombari (2023) SILC: improving vision language pretraining with self-distillation. External Links: 2310.13355, Link Cited by: §2.1. [40] M. Nilsback and A. Zisserman (2008) Automated Flower Classification by Use of Color, Shape, and Texture Features. Machine Vision and Applications 18 (1), p. 27–39. Cited by: Table 8. [41] C. Oh, J. So, H. Byun, Y. Lim, M. Shin, J. Jeon, and K. Song (2023) Geodesic multi-modal mixup for robust fine-tuning. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 52326–52341. External Links: Document, Link Cited by: §2.2. [42] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar (2012) Cats and Dogs. In IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), p. 3498–3505. Cited by: Table 8. [43] M. Patel, A. Kusumba, S. Cheng, C. Kim, T. Gokhale, C. Baral, and Y. Yang (2024) TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives. NeurIPS. Cited by: Appendix 0.C, Appendix 0.G, §1, §2.3, Table 5, Table 5, Table 5, Table 5, Table 6, Table 6, Table 6, Table 6, §5. [44] H. Pham, Z. Dai, G. Ghiasi, K. Kawaguchi, H. Liu, A. W. Yu, J. Yu, Y. Chen, M. Luong, Y. Wu, M. Tan, and Q. V. Le (2023) Combined scaling for zero-shot transfer learning. Neurocomputing 555, p. 126658. External Links: ISSN 0925-2312, Document, Link Cited by: §1, §2.1. [45] D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach (2023) SDXL: improving latent diffusion models for high-resolution image synthesis. External Links: 2307.01952, Link Cited by: §2.3. [46] F. Radenovic, A. Dubey, A. Kadian, T. Mihaylov, S. Vandenhende, Y. Patel, Y. Wen, V. Ramanathan, and D. Mahajan (2023) Filtering, distillation, and hard negatives for vision-language pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 6967–6977. Cited by: §0.D.2, §2.2. [47] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever (2021) Learning Transferable Visual Models From Natural Language Supervision. In ICML, M. Meila and T. Zhang (Eds.), Proceedings of Machine Learning Research, Vol. 139, p. 8748–8763. External Links: Link Cited by: §0.B.1, Appendix 0.G, §1, §2.1, §3.1, §4.3, Table 1, §5.1, §5.4, Table 5, Table 5, Table 5, Table 5, Table 5, Table 6, Table 6, Table 6, Table 6, Table 6, Table 7, Table 7, Table 7, Table 7, Table 7. [48] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever (2018) Improving language understanding by generative pre-training. Technical report OpenAI. External Links: Link Cited by: §3.1. [49] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever (2019) Language Models are Unsupervised Multitask Learners. Technical report OpenAI. External Links: Link Cited by: §3.1. [50] K. Ranasinghe, B. McKinzie, S. Ravi, Y. Yang, A. Toshev, and J. Shlens (2023) Perceptual grouping in contrastive vision-language models. External Links: 2210.09996, Link Cited by: §2.1. [51] J. Robinson, C. Chuang, S. Sra, and S. Jegelka (2020) Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592. Cited by: §1, §2.2. [52] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei (2015) ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision 115 (3), p. 211–252. External Links: Document, ISSN 1573-1405, Link Cited by: §5.3. [53] F. Schroff, D. Kalenichenko, and J. Philbin (2015) FaceNet: a unified embedding for face recognition and clustering. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 815–823. External Links: Link, Document Cited by: §2.2. [54] P. Sharma, N. Ding, S. Goodman, and R. Soricut (2018) Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning. In Proceedings of ACL, Cited by: §0.A.1, §5.1. [55] A. Singh, R. Hu, V. Goswami, G. Couairon, W. Galuba, M. Rohrbach, and D. Kiela (2022) FLAVA: A foundational language and vision alignment model. In CVPR, Cited by: §2.1. [56] K. Sohn (2016) Improved deep metric learning with multi-class n-pair loss objective. In Advances in Neural Information Processing Systems, D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (Eds.), Vol. 29, p. . External Links: Link Cited by: §2.2. [57] Q. Sun, Y. Fang, L. Wu, X. Wang, and Y. Cao (2023) EVA-clip: improved training techniques for clip at scale. External Links: 2303.15389, Link Cited by: §2.1. [58] Q. Sun, J. Wang, Q. Yu, Y. Cui, F. Zhang, X. Zhang, and X. Wang (2024) EVA-clip-18b: scaling clip to 18 billion parameters. External Links: 2402.04252, Link Cited by: §2.1. [59] T. Thrush, R. Jiang, M. Bartolo, A. Singh, A. Williams, D. Kiela, and C. Ross (2022) Winoground: probing vision and language models for visio-linguistic compositionality. In CVPR, Cited by: §0.D.4. [60] M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdulmohsin, N. Parthasarathy, T. Evans, L. Beyer, Y. Xia, B. Mustafa, O. Hénaff, J. Harmsen, A. Steiner, and X. Zhai (2025) SigLIP 2: multilingual vision-language encoders with improved semantic understanding, localization, and dense features. External Links: 2502.14786, Link Cited by: §2.1. [61] A. van den Oord, Y. Li, and O. Vinyals (2019) Representation Learning with Contrastive Predictive Coding. External Links: Link, 1807.03748 Cited by: §1, §2.1, §3.1. [62] V. Verma, A. Lamb, C. Beckham, A. Najafi, I. Mitliagkas, A. Courville, D. Lopez-Paz, and Y. Bengio (2019) Manifold Mixup: Better Representations by Interpolating Hidden States. External Links: Link, 1806.05236 Cited by: §3.3. [63] C. Wu, R. Manmatha, A. J. Smola, and P. Krähenbühl (2018) Sampling matters in deep embedding learning. External Links: 1706.07567, Link Cited by: §2.2. [64] W. Wu, K. Zheng, S. Ma, F. Lu, Y. Guo, Y. Zhang, W. Chen, Q. Guo, Y. Shen, and Z. Zha (2024) LoTLIP: Improving Language-Image Pre-training for Long Text Understanding. In NeurIPS, External Links: Link Cited by: §2.1, §2.3. [65] B. Xiao, H. Wu, W. Xu, X. Dai, H. Hu, Y. Lu, M. Zeng, C. Liu, and L. Yuan (2023) Florence-2: advancing a unified representation for a variety of vision tasks. arXiv preprint arXiv:2311.06242. Cited by: §2.1. [66] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba (2010) SUN Database: Large‑scale Scene Recognition from Abbey to Zoo. In IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), p. 3485–3492. Cited by: Table 8, §5.3. [67] J. Yang, C. Li, P. Zhang, B. Xiao, C. Liu, L. Yuan, and J. Gao (2022) Unified contrastive learning in image-text-label space. External Links: 2204.03610 Cited by: §2.1. [68] K. Yang, J. Deng, X. An, J. Li, Z. Feng, J. Guo, J. Yang, and T. Liu (2023) ALIP: Adaptive Language-Image Pre-training with Synthetic Caption. External Links: 2308.08428 Cited by: §2.1, §2.3. [69] L. Yao, R. Huang, L. Hou, G. Lu, M. Niu, H. Xu, X. Liang, Z. Li, X. Jiang, and C. Xu (2021) FILIP: fine-grained interactive language-image pre-training. External Links: 2111.07783, Link Cited by: §2.1. [70] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier (2014) From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2, p. 67–78. Cited by: §0.A.2, Table 8, §5.2. [71] J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu (2022) CoCa: Contrastive Captioners are Image-Text Foundation Models. TMLR. Note: External Links: ISSN 2835-8856, Link Cited by: §2.1. [72] L. Yuan, D. Chen, Y. Chen, N. Codella, X. Dai, J. Gao, H. Hu, X. Huang, B. Li, C. Li, C. Liu, M. Liu, Z. Liu, Y. Lu, Y. Shi, L. Wang, J. Wang, B. Xiao, Z. Xiao, J. Yang, M. Zeng, L. Zhou, and P. Zhang (2021) Florence: a new foundation model for computer vision. External Links: 2111.11432, Link Cited by: §2.1. [73] M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou (2023) When and why vision-language models behave like bags-of-words, and what to do about it?. In International Conference on Learning Representations, External Links: Link Cited by: §0.D.4. [74] M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou (2023) When and why Vision-Language Models behave like Bags-of-Words, and what to do about it?. In ICLR, External Links: Link Cited by: Appendix 0.C, §1, §2.2, Table 5, Table 5, Table 6, Table 6, §5. [75] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer (2023) Sigmoid Loss for Language Image Pre-Training. External Links: Link, 2303.15343 Cited by: §2.1, Table 5, Table 5, Table 5, Table 5, Table 5, Table 6, Table 6, Table 6, Table 6, Table 6, Table 7, Table 7, Table 7, Table 7, Table 7, §5. [76] X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers, A. Kolesnikov, and L. Beyer (2022) LiT: Zero-Shot Transfer with Locked-image text Tuning. In CVPR, p. 18123–18133. Cited by: Appendix 0.G, §2.1. [77] B. Zhang, P. Zhang, X. Dong, Y. Zang, and J. Wang (2024) Long-clip: unlocking the long-text capability of clip. arXiv preprint arXiv:2403.15378. Cited by: §2.1. [78] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz (2018) mixup: Beyond Empirical Risk Minimization. In ICLR, External Links: Link Cited by: §3.3. [79] L. Zhang, R. Awal, and A. Agrawal (2023) Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Fine-grained Understanding. arXiv preprint arXiv:2306.08832. Cited by: §1, §2.2. [80] K. Zheng, Y. Zhang, W. Wu, F. Lu, S. Ma, X. Jin, W. Chen, and Y. Shen (2024) DreamLIP: Language-Image Pre-training with Long Captions. In ECCV, Cited by: Appendix 0.C, Appendix 0.G, §2.1, §2.3. [81] Y. Zhou, L. Qi, J. Wang, M. Feng, W. Zhang, Z. Yang, W. Liu, and Y. Wang (2023) Non-contrastive learning meets language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 8913–8923. Cited by: §2.1. Appendix 0.A Dataset Details 0.A.1 Pretraining Datasets We pretrain on two image-text datasets of varying scale: C3M [54] and C12M [4]. Both datasets were downloaded using img2dataset [1]. Due to image link rot, the versions we obtained contain slightly fewer samples than the original releases, which may lead to minor performance differences compared to models trained on the full datasets. The same training configuration is used for both datasets (cf. Sec. 0.B.2). C3M: This dataset comprises 3.3 million image-text pairs extracted from 5 billion webpages, with image descriptions derived from the HTML alt-text attribute. The version we used contains 1,666,040 unique samples. C12M: Following a similar collection procedure as C3M but with more relaxed filters, C12M consists of 12.4 million image-text pairs covering a wider range of topics and visual concepts. The version we acquired contains 7,277,079 samples. 0.A.2 Downstream Datasets We evaluate zero-shot retrieval on Flickr30k [70] and MSCOCO [31], and classification on ImageNet [8] and 10 downstream datasets. Details are provided in Tab. 8. Table 8: Details of downstream classification datasets. Dataset Metric Categories Train Size Test Size Food-101 [2] Accuracy 101 75,750 25,250 CIFAR-10 [25] Accuracy 10 50,000 10,000 CIFAR-100 [25] Accuracy 100 50,000 10,000 SUN397 [66] Accuracy 397 19,850 19,850 Stanford Cars [24] Accuracy 196 8,144 8,041 FGVC Aircraft [36] Mean per class 100 6,667 3,333 DTD [7] Accuracy 47 3,760 1,880 Oxford Pets [42] Mean per class 37 3,680 3,669 Caltech-101 [11] Mean per class 102 3,060 6,085 Oxford Flowers [40] Mean per class 102 2,040 6,149 ImageNet [8] Accuracy 1,000 1,281,167 50,000 Flickr30k [70] Recall@k – 29,000 1,000 MSCOCO [31] Recall@k – 113,287 5,000 Appendix 0.B Implementation Details We build on the OpenCLIP [20] codebase and evaluate using CLIP Benchmark [6]. 0.B.1 Encoders We use ViT-B/16, ViT-B/32 [9], and ResNet-50 (RN-50) as vision encoders. All models are pretrained on C3M, while ViT-B/16 and ViT-B/32 are additionally trained on C12M. Following CLIP [47], the text encoder is a 12-layer Transformer with vocabulary size 49,408 and maximum context length 77. 0.B.2 Hyperparameters We use the same training configuration across all architectures and both pretraining datasets. The only difference between architectures is the image encoder itself; all other hyperparameters, including those for synthetic negative generation, are kept identical. Detailed configurations are provided in Tabs. 9(a) and 9(b). Table 9: Hyperparameters. (a) CLIP/FLIP training configuration, identical for ViT-B/16, ViT-B/32, and RN-50 on both C3M and C12M. (b) SNAP synthetic negative generation parameters. (a) CLIP/FLIP hyperparameters. Config Value Image resolution 224×224224× 224 Max text length 77 Embedding dimension d 512 Batch size |ℬ||B| 4,096 Optimizer AdamW [34] Learning rate 5×10−45× 10^-4 Weight decay 0.2 Adam (β1,β2)( _1, _2) (0.9,0.98)(0.9,0.98) Adam ϵε 1×10−81× 10^-8 LR schedule Cosine decay Warmup steps 10,000 Total epochs 32 Temperature τ (init) 0.07 (learnable) GPUs 4× NVIDIA A100 (b) SNAP hyperparameters. Config Value Top-N hard negatives 256 Negatives per strategy 32 Number of strategies 2 Total syn. negatives / query 64 Temperature τ 0.07 (constant) Strategy hyperparameters Interpolation αk _k ∼(0,0.5) (0,0.5) Extrapolation βk _k ∼(1,1.5) (1,1.5) Mixup γk _k ∼(0,1) (0,1) Noise std σ 0.01 Gradient step δ 0.01 Adversarial η 0.01 Distributed mining and gradient flow. Each GPU selects hard negatives for its local queries from the globally gathered current batch, excluding the matched positives. Gathering is performed without gradients, and source embeddings are detached during synthesis. Gradients from synthetic-negative terms flow through the query embeddings. Appendix 0.C Related Work We briefly contrast SNAP with the most closely related vision-language methods that incorporate synthetic data during pretraining (cf. Fig. 7). NegCLIP [74] constructs hard negative captions by swapping linguistic elements (attributes, relations, objects) in the original text and retrieves positive images from the three nearest neighbors in the training set. LaCLIP [10] augments the text side with synthetic positive captions generated by large language models such as ChatGPT, Bard, and LLaMA. DreamLIP [80] follows a similar philosophy but leverages multimodal LLMs (ShareGPT4V, LLaVA, InstructBLIP) to produce richer positive captions grounded in visual content. TripletCLIP [43] operates on both modalities: it generates hard negative captions via in-context learning with Mistral-7B-Instruct-v0.2 and then synthesizes corresponding negative images using SDXL-Turbo conditioned on those captions. All four methods operate in the input space and depend on external foundation models pretrained on large-scale data, introducing both computational overhead and potential data leakage from the generative models into downstream evaluation. In contrast, SNAP generates hard negatives directly in the representation space from in-batch embeddings alone, requires no external models or additional data, and applies to both modalities symmetrically with less than 10% training time overhead. Figure 7: Comparison of vision-language pretraining methods using synthetic data. Top: Existing input-space methods rely on external foundation models (e.g., LLMs for text, Diffusion for images) to generate synthetic data, incurring significant computational overhead and risk of data leakage. Bottom: SNAP operates directly in the representation space, using in-batch embeddings to symmetrically generate hard negatives for both modalities. Appendix 0.D Further Ablations and Results We conduct additional ablation studies on C3M with ViT-B/16, reporting zero-shot classification accuracy on IN-val and IN-v2. Table 10: Batch size ablation. Batch Size Method IN-val IN-v2 1024 CLIP 9.2 7.2 + SNAP 9.5 8.3 2048 CLIP 9.8 7.8 + SNAP 10.1 8.9 4096 CLIP 10.4 8.4 + SNAP 11.0 9.6 Table 11: Synthetic negatives ablation. |||S| |txt||S_txt| IN-val IN-v2 32 16 10.4 8.4 64 32 10.5 8.5 128 64 11.0 9.6 256 128 10.7 8.9 512 256 9.1 7.0 1024 512 4.3 3.2 2048 1024 3.1 2.0 0.D.1 Batch Size We evaluate SNAP at batch sizes of 1024, 2048, and 4096 (Tab. 11). SNAP outperforms CLIP at each tested batch size, although its performance decline as batch size decreases is slightly larger. These results demonstrate benefits across the tested batch sizes. The hard-negative pool remains fixed at 256 candidates, so its proportion of the available negatives increases as batch size decreases(cf. Tab. 11). 0.D.2 Number of Synthetic Negatives We vary the total number of synthetic negatives ||=|txt|+|img||S|=|S_txt|+|S_img|, split equally between s=3,4s=3,4. As shown in Tab. 11, a small number of negatives (||≤64|S|≤ 64) yields performance similar to the CLIP baseline, as the contrastive signal is not sufficiently challenging. While ||≥1024|S|≥ 1024 significantly degrades performance. At a batch size of 4096, using 512 or 1024 synthetic negatives—representing 12.5% and 25.0% of the real negative count, respectively—makes the contrastive task too difficult to solve. Synthesis controls. All synthesis controls use fixed τ=0.07τ=0.07. The single-strategy variants each generate 64 synthetic negatives per query and direction, matching the total count of full SNAP. The hard-negative weighting control follows the importance-sampling approach of DiHT [46]. 0.D.3 Paired Seed-wise Improvements Tab. 12 reports paired seed-wise improvements for the principal C3M ImageNet comparisons. SNAP improves accuracy for every evaluated seed across all three backbones and both baselines, showing that the average gains are consistent across the three runs. Table 12: Paired seed-wise improvements on C3M. ImageNet zero-shot accuracy summaries, with standard deviations shown as subscripts. Each paired gain Δ is the accuracy with SNAP minus the baseline accuracy for the same seed, in percentage points. The final column reports the mean and sample standard deviation of the three paired gains. Accuracy Paired gain Δ (p) Backbone Baseline Baseline +SNAP Seed 1 Seed 2 Seed 3 Mean ± std. ViT-B/32 CLIP 8.0.3 8.2.2 +0.46+0.46 +0.02+0.02 +0.06+0.06 +0.18±0.24+0.18± 0.24 FLIP 6.8.3 7.2.1 +0.10+0.10 +0.27+0.27 +0.74+0.74 +0.37±0.33+0.37± 0.33 ViT-B/16 CLIP 10.4.2 11.0.2 +1.07+1.07 +0.30+0.30 +0.37+0.37 +0.58±0.43+0.58± 0.43 FLIP 10.3.3 11.0.2 +0.88+0.88 +1.16+1.16 +0.12+0.12 +0.72±0.54+0.72± 0.54 RN-50 CLIP 13.6.1 14.7.2 +0.99+0.99 +1.05+1.05 +1.32+1.32 +1.12±0.18+1.12± 0.18 FLIP 13.7.1 14.9.1 +1.17+1.17 +1.25+1.25 +1.15+1.15 +1.19±0.05+1.19± 0.05 0.D.4 Compositional Evaluation We evaluate CLIP and SNAP with ViT-B/16 pretrained on C12M on ARO [73] and Winoground [59] (Tab. 13). SNAP improves ARO relation and attribute accuracy by 3.6 and 4.2 percentage points, respectively, while Winoground scores remain similar. Table 13: Compositional evaluation. Scores (%) for ViT-B/16 pretrained on C12M. ARO Winoground Method Relation Attribute Text Image Group CLIP 51.2 52.1 24.5 9.8 6.3 + SNAP 54.8 56.3 25.2 9.6 6.5 Appendix 0.E Future Work An immediate direction is to study SNAP at larger scales: training on web-scale datasets, exploring deeper and wider architectures, and integrating with recent advances in text and image encoders. The top-similarity hard-negative pool may also contain semantically compatible but unmatched examples, particularly in web-crawled data. SNAP does not explicitly filter such false negatives; incorporating semantic filtering or auditing their prevalence is left for future work. Appendix 0.F Limitations Our experiments are conducted on C3M and C12M, which are relatively small-scale compared to proprietary datasets used by methods such as CLIP (400M pairs) and ALIGN (1B pairs). Training on larger datasets was not feasible given our computational budget of 4× A100 GPUs, where a single C12M run already requires multiple days. Scaling to larger data and model sizes, while remaining environmentally conscious, is an important direction for future work. We observe broad improvements across the evaluated settings; whether these benefits persist at larger data and model scales remains to be tested. Appendix 0.G Discussion SNAP operates as a drop-in augmentation to any contrastive framework that uses an InfoNCE-style loss with a softmax denominator over negatives. Since synthetic negatives are generated from in-batch embeddings alone, the method is agnostic to the choice of encoders, data, and training procedure. We expect SNAP to be applicable beyond the CLIP [47] and FLIP [29] frameworks evaluated here, including methods such as SLIP [38], DeCLIP [28], FLIP [29], LiT [76], LaCLIP [10], DreamLIP [80], CyCLIP [16], and TripletCLIP [43], all of which build on the same InfoNCE-style contrastive objective. Appendix 0.H Reproducibility We build on the OpenCLIP [20] codebase for all training and use CLIP Benchmark [6] for all downstream evaluations. Pretraining datasets were downloaded using img2dataset [1]. All experiments use identical hyperparameters across architectures and datasets (cf. Tab. 9). Training was performed on 4× NVIDIA A100 80GB GPUs using PyTorch with mixed-precision (FP16) training. We use the same set of random seeds across all runs for reproducibility. SNAP requires no external models, no additional data, and no offline preprocessing—the only modification to the standard OpenCLIP training loop is the synthetic negative generation module, which operates entirely in the representation space during the forward pass as detailed in Algorithms 1 and 2. We will release our code and pretrained model weights upon acceptance to facilitate reproduction of all reported results. Appendix 0.I Algorithm Algorithm 1 provides the pseudo-code of SNAP, followed by the detailed implementation of the two specific generation strategies, s=3s=3 and s=4s=4, used in our approach (Algorithm 2). Algorithm 1 Pseudocode of SNAP in a PyTorch-like style. ⬇ # f_v, f_t: visual and text encoders # t: fixed temperature parameter # n_hard: pool size for sampling anchors (e.g., 256) # s3, s4: number of synthetic negatives per strategy (default: 32 each) # t: fixed temperature # n_hard: hard-negative pool size # s3, s4: synthetic negatives per strategy and direction def synthetic_logits(query, candidates, labels): scores = query @ candidates.T scores = scores.clone() scores[arange(query.shape[0]), labels] = -inf idxs = topk(scores, k=n_hard, dim=1).indices mix = generate_s3(candidates, idxs, s3) noise = generate_s4(candidates, idxs, s4) synthetic = cat([mix, noise], dim=1) return einsum("nc,nkc->nk", query, synthetic) for images, texts in loader: v = normalize(f_v(images), dim=-1) u = normalize(f_t(texts), dim=-1) # Global current-batch candidates, detached for synthesis all_v = gather_no_grad(v.detach()) all_u = gather_no_grad(u.detach()) # Positive indices in the globally gathered batch # Assumes equal local batch sizes and rank-ordered gathering labels = rank * v.shape[0] + arange(v.shape[0]) syn_i2t = synthetic_logits(v, all_u, labels) syn_t2i = synthetic_logits(u, all_v, labels) # Obtain base logits with the existing CLIP gradient handling base_i2t, base_t2i = clip_logits(v, u) logits_i2t = cat([base_i2t, syn_i2t], dim=1) / t logits_t2i = cat([base_t2i, syn_t2i], dim=1) / t loss = 0.5 * ( cross_entropy(logits_i2t, labels) + cross_entropy(logits_t2i, labels) ) optimizer.zero_grad() loss.backward() optimizer.step() Algorithm 2 SNAP Generation Strategies in a PyTorch-like style. ⬇ def generate_s3(t_feat, idxs_hard, num_neg, gamma=1.0): """Mixup␣between␣two␣hard␣negatives.""" # sample two sets of indices from the hard pool idx1 = randint(0, n_hard, size=(batch_size, num_neg)) idx2 = randint(0, n_hard, size=(batch_size, num_neg)) h1 = t_feat[gather(idxs_hard, 1, idx1)] h2 = t_feat[gather(idxs_hard, 1, idx2)] lam = rand(batch_size, num_neg, 1) * gamma s3 = lam * h1 + (1 - lam) * h2 return normalize(s3, dim=-1) def generate_s4(t_feat, idxs_hard, num_neg, sigma=0.01): """Noise␣injection␣into␣a␣hard␣negative.""" idx = randint(0, n_hard, size=(batch_size, num_neg)) h = t_feat[gather(idxs_hard, 1, idx)] noise = randn_like(h) * sigma return normalize(h + noise, dim=-1)