Paper deep dive
ECI: Effective Contrastive Information to Evaluate Hard-Negatives
Aarush Sinha, Rahul Seetharaman, Aman Bansal
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/26/2026, 2:22:13 AM
Summary
The paper introduces ECI (Effective Contrastive Information), a theoretically grounded metric for evaluating the quality of hard-negative samples in dense retrieval models prior to training. ECI balances Information Capacity (based on set size) and Discriminative Efficiency (a harmonic mean of signal hardness and safety margin) to identify optimal negative sampling strategies, such as the BM25+Cross-Encoder hybrid, thereby reducing the need for costly ablation studies.
Entities (6)
Relation Signals (3)
ECI → evaluates → Hard Negatives
confidence 100% · ECI: Effective Contrastive Information to Evaluate Hard-Negatives
BM25+Cross-Encoder → optimizes → ECI
confidence 95% · identifying that hybrid strategies (BM25+Cross-Encoder) offer the optimal balance of volume and reliability
ECI → predicts → Retrieval Performance
confidence 95% · Our results demonstrate that ECI accurately predicts downstream retrieval performance
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Hard negatives play a critical role in training and fine-tuning dense retrieval models, as they are semantically similar to positive documents yet non-relevant, and correctly distinguishing them is essential for improving retrieval accuracy. However, identifying effective hard negatives typically requires extensive ablation studies involving repeated fine-tuning with different negative sampling strategies and hyperparameters, resulting in substantial computational cost. In this paper, we introduce ECI: Effective Contrastive Information , a theoretically grounded metric grounded in Information Theory and Information Retrieval principles that enables practitioners to assess the quality of hard negatives prior to model fine-tuning. ECI evaluates negatives by optimizing the trade-off between Information Capacity the logarithmic bound on mutual information determined by set size and Discriminative Efficiency, a harmonic balance of Signal Magnitude (Hardness) and Safety (Max-Margin). Unlike heuristic approaches, ECI strictly penalizes unsafe, false-positive negatives prevalent in generative methods. We evaluate ECI across hard-negative sets mined or generated using BM25, cross-encoders, and large language models. Our results demonstrate that ECI accurately predicts downstream retrieval performance, identifying that hybrid strategies (BM25+Cross-Encoder) offer the optimal balance of volume and reliability, significantly reducing the need for costly end-to-end ablation studies.
Tags
Links
- Source: https://arxiv.org/abs/2603.20990v1
- Canonical: https://arxiv.org/abs/2603.20990v1
Trouble viewing inline? Open PDF directly →
Full Text
53,725 characters extracted from source content.
Expand or collapse full text
ECI: Effective Contrastive Information to Evaluate Hard-Negatives Aarush Sinha ∗ aarush.sinha@gmail.com Department of Computer Science, University of Copenhagen Copenhagen, Denmark Rahul Seetharaman ∗ rahulseetharaman@gmail.com Independent Researcher USA Aman Bansal ∗ 18aman.bansal@gmail.com Independent Researcher USA Abstract Hard negatives play a critical role in training and fine-tuning dense retrieval models, as they are semantically similar to positive doc- uments yet non-relevant, and correctly distinguishing them is es- sential for improving retrieval accuracy. However, identifying ef- fective hard negatives typically requires extensive ablation studies involving repeated fine-tuning with different negative sampling strategies and hyperparameters, resulting in substantial computa- tional cost. In this paper, we introduce ECI: Effective Contrastive Information , a theoretically grounded metric grounded in Infor- mation Theory and Information Retrieval principles that enables practitioners to assess the quality of hard negatives prior to model fine-tuning. ECI evaluates negatives by optimizing the trade-off between Information Capacity the logarithmic bound on mutual information determined by set size and Discriminative Efficiency, a harmonic balance of Signal Magnitude (Hardness) and Safety (Max-Margin). Unlike heuristic approaches, ECI strictly penalizes unsafe, false-positive negatives prevalent in generative methods. We evaluate ECI across hard-negative sets mined or generated us- ing BM25, cross-encoders, and large language models. Our results demonstrate that ECI accurately predicts downstream retrieval per- formance, identifying that hybrid strategies (BM25+Cross-Encoder) offer the optimal balance of volume and reliability, significantly reducing the need for costly end-to-end ablation studies. CCS Concepts • Information systems→Evaluation of retrieval results; Rel- evance assessment; Retrieval efficiency; Evaluation of retrieval results. Keywords Hard Negatives, Dense Retrieval, Efficiency ACM Reference Format: Aarush Sinha, Rahul Seetharaman, and Aman Bansal. 2018. ECI: Effective Contrastive Information to Evaluate Hard-Negatives. In Proceedings of Make ∗ Authors contributed equally to this research. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. Conference acronym ’X, Woodstock, NY © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-X-X/2018/06 https://doi.org/X.X sure to enter the correct conference title from your rights confirmation email (Conference acronym ’X). ACM, New York, NY, USA, 10 pages. https://doi. org/X.X 1 Introduction Recent advances in dense retrieval and contrastive-learning frame- works [9,18,33] have substantially improved information retrieval tasks such as semantic search, question answering, and recom- mender systems. A key component in these models is the use of neg- ative samples during training, particularly hard negatives—documents that are semantically similar to a query but non-relevant. Correctly distinguishing these hard negatives from positive documents is essential for shaping the embedding space and improving retrieval accuracy [35]. By pushing the decision boundary closer to the relevant documents, hard negatives force the model to learn fine- grained representations that generalize better to unseen queries [22]. Despite their importance, generating and selecting effective hard negatives remains a significant challenge. Existing approaches vary widely, ranging from static versus dynamic sampling strategies [35], to defining hardness relative to the query or the positive passage [20,21]. More recent work has focused on balancing multiple quality criteria, such as diversity and difficulty [15,34]. Parallel to these mining efforts, large language models (LLMs) have been explored for generating synthetic hard negatives [12,26] or exploiting multi- hop citation structures [27]. However, this diversity of strategies presents a practical dilemma: the quality of a generated negative set is often only apparent after costly end-to-end training and evaluation. The primary bottleneck in this pipeline is the lack of a robust, training-free evaluation metric for hard-negative sets. Practitioners are typically forced to rely on heuristic proxies, such as lexical overlap or the raw output scores of cross-encoders, which fail to account for the complex interactions between sample hardness and safety. For instance, recent studies indicate that unfiltered synthetic negatives from LLMs, while exhibiting high semantic similarity (high hardness), often violate relevance assumptions and degrade retrieval performance due to the introduction of label noise [2, 3]. Without a reliable theoretical grounding to assess these risks, researchers must resort to extensive ablation studies involving repeated fine-tuning with different hyperparameters, resulting in substantial computational waste [6]. In this work, we address this gap by introducing Effective Con- trastive Information (ECI), a theoretically grounded metric de- signed to analyze the quality of hard negatives prior to model fine-tuning. ECI is derived from principles of Information Theory arXiv:2603.20990v1 [cs.IR] 22 Mar 2026 Conference acronym ’X, June 03–05, 2018, Woodstock, NYSinha et al. and InfoNCE bounds, quantifying the trade-off between two com- peting objectives: Signal Magnitude (Hardness), which rewards high semantic similarity necessary for effective gradient updates, and Safety, measured via the maximum similarity margin (Δ 푚푎푥 ) to detect overly hard or near-positive negatives (false positives). Unlike previous heuristic metrics, ECI employs a harmonic mean to model the Discriminative Efficiency of a negative set. This ensures that the metric strictly penalizes datasets where a high signal is achieved at the expense of safety—a critical feature for evaluating LLM-generated negatives. Furthermore, ECI incorporates an Infor- mation Capacity term, accounting for the logarithmic growth of mutual information with the number of negatives. By providing a rigorous pre-training assessment of hard-negative sets mined with BM25, cross-encoders, and LLMs, ECI reduces reliance on ex- pensive end-to-end ablation studies while enabling more informed design choices in dense retrieval pipelines. Our contributions are as follows: •We introduce ECI, a novel metric grounded in Information Theory for evaluating hard-negative quality without fine- tuning. •We theoretically formulate the trade-off between Informa- tion Capacity and Discriminative Efficiency using harmonic aggregation to mitigate false positive risks. •We demonstrate across multiple datasets that ECI accurately predicts downstream retrieval performance, identifying hy- brid strategies (BM25+Cross-Encoder) as the optimal balance of signal and safety. 2 Related Work Dense Retrieval and Contrastive Learning. Dense retrieval has become a dominant paradigm for semantic search following the success of dual-encoder architectures such as DPR [9]. Subsequent work has focused on improving representation quality through contrastive learning, leveraging large in-batch negatives [7] and asynchronous hard-negative mining as in ANCE [32]. Pretraining strategies such as Condenser [5], Contriever [8], and GTR [16] further demonstrate that retrieval performance is highly sensitive to the structure, scale, and stability of contrastive signals. More recent embedding models such as E5 [30] confirm that weak supervision and careful negative construction remain critical for robust general- purpose retrieval. Hard-Negative Mining Strategies. The choice of hard negatives plays a critical role in shaping dense retriever performance. Early work formalized the importance of mining informative negatives during training [35], while later approaches explored dynamic and progressive mining schemes [13,20]. TriSampler [34] balances easy, hard, and false negatives, whereas BM25-based passage negatives [21] define hardness relative to the positive passage rather than the query. ANCE [32] and NV-Retriever [15] further show that aggressively mined negatives can improve performance, but also increase training instability if not carefully controlled. Mitigating False and Overly-Hard Negatives. Several works high- light the risks posed by false or excessively hard negatives. An- alyzing [3,23] this issue from a theoretical contrastive learning perspective, showing that false negatives introduce biased gradients. Practical mitigation strategies include confidence regularization [31], robustness to label noise [36], and gradient caching techniques such as GradCache [6]. These findings suggest that hardness alone is insufficient; stability and reliability of negative signals are equally important. LLM-Generated and Synthetic Negatives. Recent studies investi- gate the use of large language models to generate synthetic training data and hard negatives. InPars [2] and Promptagator [4] use LLMs to synthesize queries or supervision for retriever training, while SyNeg [12] directly generates controlled hard negatives. However, recent analyses show that unfiltered synthetic negatives may vio- late relevance assumptions and degrade retrieval performance [26]. Domain-specific extensions, such as citation-aware [27], further emphasize the need for evaluating negative quality prior to their use in training. 3 Method For all methods we use 10,000 query-passage pairs from the MS MARCO [1] from Tevatron/msmarco-passage 1 3.1 BM25 We index our corpus using BM25S[14]. For each query in the dataset, we mine the top퐾=50 candidate passages using BM25 retrieval, where퐾is the target number of hard negatives per query. The final dataset is a triplet for further fine-tuning our dense-retriever. 3.2 Cross-Encoders Cross-Encoder Hard Negative Mining. To construct high-quality hard negatives, we directly apply a cross-encoder re-ranking strat- egy specifically themixedbread-ai/mxbai-rerank-large-v1[25] 2 over a predefined candidate pool. For each query푞, all candidate passages excluding known positives are paired with the query and scored using a pretrained cross-encoder, which jointly encodes the (푞,푝) pair to produce a relevance score. The candidate passages are then ranked by cross-encoder score, and the top-퐾 ℎ푎푟푑 =25 passages are selected as hard negatives. This procedure yields negatives that are semantically close to the query while remaining distinct from labeled positives. By relying solely on cross-encoder relevance rather than lexical heuristics, the mined negatives reflect semantic difficulty and better capture model-level ambiguity. 3.3 LLM We generate synthetic hard negatives using a large language model (LLM) applied to MS MARCO triplets. For each example, the LLM is prompted with the positive query-document pair and instructed to produce three hard negative documents that appear relevant but fail to satisfy the true information need. We make use of the gpt-4o-mini[17] via the API as our generative model. Prompt Template. 1 https://huggingface.co/datasets/Tevatron/msmarco-passage 2 https://huggingface.co/mixedbread-ai/mxbai-rerank-large-v1 ECI: Effective Contrastive Information to Evaluate Hard-NegativesConference acronym ’X, June 03–05, 2018, Woodstock, NY Hard Negative Generation Prompt Assume you are an expert in domain_name, and there is a exam- ple with a “user_query” and its related doc “positive_document”. example: positive_example [Task Definition] Your task is to write three hard negative samples in JSON format. The JSON object must contain the following keys: •“reasoning”: a string describing how the hard negatives are generated. • “hard_negative_document_1”: a hard-negative document. • “hard_negative_document_2”: a hard-negative document. • “hard_negative_document_3”: a hard-negative document. [Reasoning Definition] Write the inference process step by step in “reasoning”, includ- ing how the “user_query” and “positive_document” are used to construct the hard negatives. [Hard Negatives Definition] All hard negative documents must: • Share similar keywords or topics with the positive docu- ment. • Appear to address the user query at first glance. •Subtly diverge in content or context so they do not answer the query. • Be plausible, accurate, and diverse in style and source. [Attributes Definition] All negative documents should match an education level of diffi- cult_level and be length the length of the positive document. [Format Definition] Output only a single JSON object and nothing else. Generation Details. We sample domain, difficulty, and length constraints per instance and generate three hard negatives per positive document. The resulting synthetic negatives are used for retrieval evaluation and training. 3.4 ECI To rigorously evaluate retrieval quality, we propose the Effective Contrastive Information (ECI) score, grounded in Information Theory and the principles of InfoNCE estimation. We model the utility of a hard-negative set as the product of its Information Capac- ity the logarithmic bound derived from Mutual Information—and its Discriminative Efficiency. Mathematically, the capacity of a contrastive batch to approxi- mate the true mutual information grows logarithmically with the number of negatives|N|. For the efficiency term, we treat Hardness (signal strength) and Safety (decision boundary margin) as compet- ing rates. To strictly penalize sets where either metric is deficient (e.g., LLMs with high hardness but zero safety), we employ the Harmonic Mean. We define ECI as follows: ECI= ln(1+|N|) | z Information Capacity · 2· 푆 푛 ·Δ 푚푎푥 푆 푛 +Δ 푚푎푥 | z Harmonic Efficiency (1) where푆 푛 is the average query-negative similarity (Signal), andΔ 푚푎푥 is the Max-Margin (푀푎푥푆푖푚 푝 −푀푎푥푆푖푚 푛 ). This formulation ensures that a superior training set must provide a sufficient quantity of diverse negatives to tighten the bound (Capacity), while maintain- ing a high-quality balance between difficulty and distinguishability. Unlike heuristic averages, the harmonic mean ensures that unsafe negatives (false positives) are heavily penalized, aligning the metric with retrieval stability. 4 Theoretical Framework This section outlines the theoretical underpinnings of the Effective Contrastive Information (ECI) metric. We formulate the selec- tion of hard negatives as an optimization problem constrained by Information Capacity and Reliability, grounding the proposed metric in contrastive learning theory. 4.1 Information Capacity: The Logarithmic Bound Dense retrieval models trained with contrastive objectives (e.g., InfoNCE) aim to maximize the Mutual Information (MI) between a query representation푞and its corresponding positive document 푝. A defining property of Noise Contrastive Estimation (NCE) ob- jectives is that the achievable MI is lower-bounded by a term that grows logarithmically with the number of negative samples. InfoNCE Lower Bound. Let(푞,푝) ∼ 푝(푞,푝)denote a query–positive pair, and letN=푛 1 , . . .,푛 푁 be a set of푁negatives drawn inde- pendently from the marginal distribution푝(푛). The InfoNCE loss is defined as: L InfoNCE =−E " log exp(푓(푞,푝)) exp(푓(푞,푝))+ Í 푁 푖=1 exp(푓(푞,푛 푖 )) # ,(2) where 푓(·,·) is a similarity (critic) function. For the optimal critic 푓 ∗ (푞,푑)= log 푝(푑 | 푞) 푝(푑) ,(3) the InfoNCE objective yields a lower bound on the mutual informa- tion 퐼(푄;푃): 퐼(푄;푃) ≥ log(푁 + 1)−L InfoNCE .(4) Proof Sketch. Substituting the optimal critic into the InfoNCE loss and rearranging gives: 퐼(푄;푃)=−L InfoNCE +E " log 푝(푝 | 푞) 푝(푝) + 푁 ∑︁ 푖=1 푝(푛 푖 | 푞) 푝(푛 푖 ) !# .(5) Since negatives are sampled from the marginal푝(푛), each impor- tance weight satisfiesE 푛 푖 h 푝(푛 푖 |푞) 푝(푛 푖 ) i =1. Applying Jensen’s inequal- ity to the logarithm yields: E [ log(·) ] ≤ log 푝(푝 | 푞) 푝(푝) + 푁 ≤ log(푁 + 1),(6) which produces the stated bound. Implication. This result implies that the information capacity of a contrastive objective grows at most logarithmically with the number of negatives. Consequently, increasing the size of a negative set improves the theoretical learning capacity, but with sharply diminishing returns. Conference acronym ’X, June 03–05, 2018, Woodstock, NYSinha et al. We therefore define the Information Capacity term as: I cap ∝ ln(1+|N|).(7) This formulation accounts for scale while preventing the metric from being dominated purely by the number of negatives, motivat- ing the need to jointly consider negative quality. 4.2 The Hardness–Safety Duality To quantify the quality of a negative set, we identify two competing rates: (1)Signal Strength (Hardness). Denoted푆 푛 , defined as the average query–negative similarity. Hard negatives induce steeper gradients and encourage the model to learn fine- grained decision boundaries. (2)Margin Safety (Robustness). DenotedΔ max = MaxSim 푝 − MaxSim 푛 , capturing the separation between the most similar positive and the hardest negative. A large margin reduces structural risk by avoiding false positives. These objectives are fundamentally antagonistic: increasing hard- ness pushes negatives closer to the positive boundary, reducing safety. The effective utility of a negative set is therefore governed by the weaker of the two. 4.3 Harmonic Efficiency We model the Discriminative EfficiencyE disc using the harmonic mean: E disc = 2· 푆 푛 ·Δ ′ max 푆 푛 +Δ ′ max .(8) To handle false positives—cases where a negative is scored as more relevant than the positive—we define the effective margin as: Δ ′ max = max(0, MaxSim 푝 − MaxSim 푛 ).(9) This clamping ensures that any violation of the positive decision boundary collapses the efficiency to zero, reflecting the fact that such negatives actively harm training. 4.4 The ECI Formulation Combining Information Capacity with Discriminative Efficiency yields the final metric: ECI= ln(1+|N|) | z Information Capacity · E disc |z Discriminative Efficiency .(10) The multiplicative structure ensures that neither scale nor quality alone can dominate the score. Large collections of trivial negatives are suppressed by low efficiency, while small high-quality sets are constrained by the logarithmic capacity bound. ECI therefore favors negative sets that are simultaneously large, hard, and safe, aligning theoretical capacity with practical training reliability. 5 Fine-Tuning We naively concatenate datasets generated by our various methods resulting in a total of seven training datasets. We fine-tuned a DistilBERT [24] model for 1 epoch withearly_stopping=3using andbatch_size=16. The model was fine-tuned using the Multiple- Negative Ranking Loss (MNRL)[19] defined as: L MNRL =− 1 퐵 퐵 ∑︁ 푖=1 log exp sim(q 푖 ,p + 푖 )/휏 Í 퐵 푗=1 exp sim(q 푖 ,p 푗 )/휏 6 Results Table 1 details the signal and safety metrics for each hard-negative mining method. The proposed ECI metric identifies the BM25+Cross- Encoder hybrid as the most effective strategy, achieving the highest score of 1.25. The results highlight the trade-off inherent in hard-negative min- ing. While the pure Cross-Encoder method achieves the highest Signal (푆 푛 =0.606), it pushes negatives too close to the positive passage, resulting in a reduced Safety margin (Δ 푚푎푥 =0.175). Con- sequently, its ECI score (0.88) is penalized for the increased risk of false positives. Conversely, the hybrid BM25+Cross-Encoder ap- proach maintains a high signal (0.587) while preserving a safer margin (0.192), optimizing the ECI objective. Impact of Negative Count Imbalance and Source Quality: We note that the LLM approach utilized significantly fewer negatives per query (|N|=3) compared to retrieval-based approaches (|N|= 50) due to inference costs. However, because ECI incorporates a logarithmic capacity termln(1+|N|), it accounts for diminishing returns in volume. The substantially lower ECI score for the LLM (0.26) indicates that the poor performance is not merely an artifact of data scarcity, but a fundamental quality issue. The drastic drop in downstream retrieval performance for the LLM-only method (nDCG 0.164 vs 0.321 for BM25) highlights the critical risk of using unfiltered synthetic data. Despite having high signal, the individual LLM-generated samples frequently violated the decision boundary (Δ 푚푎푥 =0.110), triggering the safety penalty via the harmonic mean. However, comparison with the LLM+BM25 hybrid method (0.319) provides nuance to this finding. The hybrid approach, which uses BM25 to filter and constrain the LLM generations, achieves significantly higher performance than the LLM-only baseline. This suggests that while the LLM source is capable of generating seman- tically rich hard negatives, it lacks the inherent safety mechanism to avoid false positives. The ECI metric captures this distinction perfectly: it assigns a near-failing grade to the unfiltered LLM set (0.26) but a passing grade to the filtered hybrid (1.06), confirming that ECI penalizes the lack of safety, not the generator itself. 6.1 BEIR Results To validate whether the intrinsic ECI score correlates with down- stream effectiveness, we evaluated a DistilBERT retriever fine-tuned on data generated by each method across 12 BEIR [28] datasets, with results summarized in Table 2. The downstream evaluation confirms the predictive validity of the ECI metric: the BM25+Cross- Encoder configuration, which achieved the highest ECI score, also delivered the best average downstream performance (0.337). This hybrid method outperforms both standard baselines BM25 (0.321) and Cross-Encoder (0.321) by a clear margin, demonstrating that combining lexical candidate retrieval with semantic filtering yields a much better training signal. In addition, the hybrid approach ECI: Effective Contrastive Information to Evaluate Hard-NegativesConference acronym ’X, June 03–05, 2018, Woodstock, NY Table 1: Evaluation metrics using ECI. Signal (푆 푛 ) measures gradient strength, while Max-Margin (Δ 푚푎푥 ) measures the discrimi- native boundary. The Harmonic Efficiency balances these rates, and the final score accounts for the Information Capacity of set size (|N|). Best score is in bold. NameTop-k (|N|)Signal (푆 푛 )Max-Margin (Δ 푚푎푥 )ECI↑ BM25500.5770.1991.16 Cross-Encoder250.6060.1750.88 LLM30.6560.1100.26 Concatenated (Hybrid) Methods BM25+Cross-Encoder750.5870.1921.25 LLM+BM25530.6200.1520.97 LLM+BM25+Cross-Enc.780.6210.1521.06 Cross-Encoder+LLM280.6310.1430.79 Table 2: Evaluation on 12 BEIR datasets evaluated using nDCG@10. Bold indicates the best performing model. Dataset column represents the dataset DistilBERT was fine-tuned upon and then evaluated. DatasetLLM LLM+ LLM+Cross- LLM+BM25+ BM25 Cross-Encoder BM25+Cross- BM25 Encoder Cross-EncoderEncoder FEVER0.049 0.5660.5470.5500.5800.5830.593 NFCorpus0.065 0.2090.2010.2010.2000.2030.220 SCIDOCS0.054 0.1090.0930.0950.1110.1000.116 SCIFACT0.099 0.4220.4200.3830.4180.4430.443 TREC-Covid0.210 0.3130.2730.2870.3020.2980.327 NQ0.070 0.2420.2900.2590.2640.2730.276 Climate-FEVER0.048 0.1430.1160.1340.1470.1570.170 ArguAna 0.398 0.4280.4210.3830.4450.4170.448 QUORA0.752 0.8160.8090.8130.8150.8120.818 DBPedia-Entity0.124 0.2760.2650.2640.2780.2650.282 FIQA0.055 0.1750.1710.1490.1710.1690.194 Touche20200.043 0.1310.1300.1080.1260.1330.160 Average0.164 0.3190.3120.3020.3210.3210.337 achieved the top score on 11 out of 12 datasets. In contrast, consis- tent with its low ECI score, the LLM-only method failed to train an effective retriever and resulted in a substantially lower average nDCG@10 of 0.164. Overall, these findings demonstrate a strong alignment between the intrinsic stability measured by ECI and ex- trinsic retrieval quality, suggesting that optimizing for gap stability is a crucial factor in effective hard-negative mining. We note that ECI is intended as a diagnostic metric evaluated prior to fine-tuning. As such, our downstream experiments follow standard dense retrieval practice and report single-seed results, consistent with prior work[9, 15, 27, 29]. 7 Ablations 7.1 Aggregation Strategy To validate the theoretical robustness of the ECI metric, we con- ducted an ablation study comparing the Harmonic Mean against a standard Arithmetic Mean for balancing Signal and Safety. As shown in Table 5, the Harmonic Mean correctly identifies the BM25+Cross-Encoder hybrid as the optimal strategy (1.25). More importantly, it correctly captures the degradation in quality when LLM-generated negatives are added to the optimal set: the score drops from 1.25 to 1.06 (LLM+BM25+Cross-Enc). Conversely, the Arithmetic Mean approach obscures this critical flaw. Due to its insensitivity to the "bottleneck" of safety, the Arith- metic Mean assigns nearly identical scores to the optimal hybrid and the LLM-contaminated hybrid (1.69 vs 1.69). This would in- correctly suggest that adding LLM-generated noise does not harm the dataset’s utility. The retrieval results in Table 2 confirm this is not the case; the LLM-contaminated model underperforms (0.302) compared to the clean hybrid (0.337). This confirms that the rate- based aggregation inherent to Information Theory is essential for identifying training sets that are both information-rich and robust to label noise. 7.2 Does data size matter ? To ensure a fair comparison with LLM-generated negatives, which are limited to푁=3 per query due to inference costs, we conducted a controlled analysis using the top-3 hardest negatives for retrieval- based methods. This setup isolates the intrinsic quality-per-sample Conference acronym ’X, June 03–05, 2018, Woodstock, NYSinha et al. Table 3: Comparison of ECI scores across smaller (all-MiniLM-L6-v2) and larger (mxbai-embed-large-v1) embedding architec- tures. Each model is evaluated independently. Across both architectures, the BM25+Cross-Encoder hybrid consistently achieves the highest ECI, while LLM-only mining yields the lowest scores. ModelMethodTop-kSignal (푆 푛 ) Max-Margin (Δ 푚푎푥 )ECI↑ all-MiniLM (L6-v2) BM25500.3390.3381.33 Cross-Encoder250.3820.2981.10 LLM30.4330.1430.30 BM25+Cross-Encoder750.3520.3261.47 LLM+BM25530.3920.2361.17 LLM+BM25+Cross-Enc.780.3980.2411.28 Cross-Encoder+LLM280.4070.2220.96 mxbai (large-v1) BM25500.5650.2121.21 Cross-Encoder250.5890.1910.94 LLM30.6440.1090.26 BM25+Cross-Encoder750.5780.2211.34 LLM+BM25530.6050.1601.01 LLM+BM25+Cross-Enc.780.6120.1661.11 Cross-Encoder+LLM280.6170.1500.81 Table 4: Evaluation of ECI with a fixed negative count (푁=3). The BM25+Cross-Encoder hybrid achieves the highest ECI score (0.400) by maintaining the strongest safety margin among the high-signal methods. Pure Cross-Encoder (CE) sees the largest drop in ECI due to a severely compressed Max-Margin (Δ 푚푎푥 = 0.093). NameSignal (푆 푛 )Max-Margin (Δ 푚푎푥 )Harmonic Eff.ECI↑ BM250.6690.1120.1920.265 Cross-Encoder 0.6950.0930.1650.228 LLM0.6560.1100.1890.262 BM25+Cross-Encoder0.5860.1910.2890.400 Cross-Encoder+LLM0.6740.1030.1780.247 LLM+BM25 0.6210.1510.2430.337 LLM+BM25+Cross-Enc.0.6210.1510.2430.337 Table 5: Comparison of aggregation strategies for combining Hardness (푆 푛 ) and Safety (Δ 푚푎푥 ). The Harmonic Mean (used in ECI) correctly identifies the degradation caused by adding LLMs to the hybrid set, whereas the Arithmetic Mean fails to detect this noise, erroneously predicting equivalent per- formance. MethodHarmonic Mean (ECI) Arithmetic Mean (Baseline) BM251.161.52 Cross-Encoder 0.881.27 LLM0.260.53 BM25+Cross-Encoder1.251.69 LLM+BM250.971.54 LLM+BM25+Cross-Enc.1.061.69 of each strategy, removing the bias introduced by varying dataset sizes. Table 4 presents the evaluation metrics for푁=3. When re- stricted to the top-3 candidates, BM25 achieves a higher ECI score (0.265) than the Cross-Encoder (0.228). This occurs because the Cross-Encoder, while powerful, pushes negatives extremely close to the decision boundary; when only the top-3 are selected. Crucially, the hybrid BM25+Cross-Encoder strategy remains dominant, achieving the highest ECI score of 0.400. This config- uration successfully leverages the Cross-Encoder’s ability to find high-signal negatives while relying on BM25 to filter out candidates that would otherwise violate the safety boundary. Interestingly, the LLM-only method achieves a higher score (0.262) than the pure Cross-Encoder (0.228) and is comparable to BM25 (0.265), suggesting that LLMs can generate high-quality negatives in limited quantities (푁=3). However, this quality does not consistently translate to robustness when expanded into hybrid sets. The LLM+BM25 and All Three methods achieve a competi- tive ECI of 0.337, significantly outperforming the pure baselines. In contrast, the Cross-Encoder+LLM hybrid falters with a score of 0.247, underperforming even the standalone BM25. This confirms that while LLMs provide high signal, they rely heavily on BM25’s structural stability; without it, the combination with the aggressive hardness of Cross-Encoders results in reduced safety and overall utility. ECI: Effective Contrastive Information to Evaluate Hard-NegativesConference acronym ’X, June 03–05, 2018, Woodstock, NY 7.3 How do different embedding models affect ECI? To investigate the robustness of ECI across different embedding models, we evaluate the metric on two embedding models inde- pendently: all-MiniLM-L6-v2[19], a compact, parameter-efficient and weaker model, and mxbai-embed-large-v1[10,11], a larger and stronger much stronger embedding model. Importantly, each model is treated as a separate evaluation setting, and scores are not shared across architectures. The results in Table 3 highlight a consistent trade-off between semantic signal strength and discriminative margin across model capacities. While the larger mxbai model achieves substantially higher Signal (푆 푛 ) values due to denser and more expressive repre- sentations, this gain is accompanied by a systematic compression of the Max-Margin (Δ 푚푎푥 ). As a result, ECI scores for the larger model are often comparable to or lower than those of the smaller all-MiniLM model under the same mining strategy. Notably, the BM25+Cross-Encoder hybrid remains the top-performing approach for both architectures, rein- forcing earlier findings that combining lexical recall with semantic re-ranking best preserves discriminative structure. In contrast, LLM-only mining consistently yields the lowest ECI across both models. Although LLM-generated negatives exhibit high semantic similarity, they dramatically reduce the available margin, leading to poor ECI values (0.30 for MiniLM and 0.26 for MXBAI). This indicates that LLM-only strategies tend to produce borderline or overly similar negatives that undermine effective contrastive separation. Overall, these findings demonstrate that ECI captures not only raw semantic alignment but also the preservation of useful training margins, revealing diminishing returns from increased model capac- ity and exposing the limitations of unfiltered LLM-based negative mining. 7.4 Comparison with Gradient-based Heuristics To validate that ECI offers unique value beyond standard training dynamics heuristics, we conducted a comparative analysis against two mathematically rigorous baselines: Gradient Norm (the mag- nitude of the update signal) and Score Variance (the diversity of the negative distribution). We calculated these metrics for the BAAI/bge-base-en-v1.5 model using the original full-dataset statistics (variable|N|). We then measured the Pearson correlation (푟) between each metric and the ground-truth downstream retrieval performance (nDCG@10 on BEIR). Mathematical Definitions. For a query푞, positive document푝, and negative setNwith similarities푠 푖 = sim(푞,푛 푖 ), we define the fol- lowing heuristics: 1. Estimated Gradient Norm (G 푒푠푡 ) In InfoNCE loss, the gra- dient of the loss with respect to the query embedding is dominated by the softmax weights of the negatives. The magnitude of the "repulsive" force exerted by the negatives is proportional to the sum of their exponential similarities. We estimate this aggregate gradient norm as: G 푒푠푡 = ∑︁ 푖∈N exp 푠 푖 휏 ≈ |N|· exp 푆 푛 휏 (11) where 푆 푛 is the mean similarity and 휏 is the temperature. 2. Estimated Score Variance ( ˆ 휎 2 ) Score variance measures the spread of difficulty within the batch. A high variance implies a mix of easy and hard samples, while low variance implies unifor- mity. Given only summary statistics (푆 푛 and푆 푚푎푥 ), we estimate the variance assuming a skewed distribution (typical in retrieval): ˆ 휎 2 = 1 2 ·(푆 푚푎푥 − 푆 푛 ) 2 (12) This formulation approximates the second central moment based on the range between the average hardness and the peak hardness. Table 6: Correlation of ECI and mathematically derived train- ing heuristics with ground-truth nDCG@10 on the BAAI/bge- base-en-v1.5 model. While ECI shows a very strong positive correlation, heuristics based on Gradient Magnitude exhibit strong negative correlation. MetricPearson 푟 Interpretation Direction ECI0.91Very StrongPositive Avg. Hardness (푆 푛 )-0.45Weak/Moderate Negative Est. Gradient Norm (G 푒푠푡 )-0.82StrongNegative Est. Score Variance ( ˆ 휎 2 )0.21WeakPositive The results in Table 6 provide empirical evidence against optimiz- ing for hardness alone. The Estimated Gradient Norm exhibits a strong negative correlation (푟=−0.82) with retrieval performance. This occurs because methods like LLM and Cross-Encoder gener- ate negatives with high similarity (푆 푛 ), resulting in large gradient magnitudes. However, as Eq. 1 shows, these gradients are unstable because they push the model against decision boundaries that are already violated (Δ 푚푎푥 →0). Consequently, maximizingG 푒푠푡 leads to over-fitting on noisy labels. Conversely, ECI demonstrates a very strong positive correla- tion (푟=0.91). By weighting the Information Capacity against the Harmonic Efficiency, ECI implicitly penalizes the high-magnitude, low-safety gradients that the standard heuristics favor. This con- firms that ECI provides a theoretically grounded signal for negative quality that surpasses simple gradient magnitude or distribution variance. 7.5 Qualitative Analysis of LLM Generated Hard-Negatives To empirically understand why LLM-generated hard negatives yield low ECI scores and poor downstream retrieval performance, we conduct a qualitative case study. We examine a specific instance from the MS MARCO dataset where the LLM was tasked with generating hard negatives for the query “what are the liberal arts?”. 7.5.1 Example: The Misleading Definition. The LLM successfully generates text that is lexically dense and structurally identical to a valid encyclopedia entry. However, the semantic content is designed to be adversarially misleading as seen in Table 7. Conference acronym ’X, June 03–05, 2018, Woodstock, NYSinha et al. Table 7: Example of query, positive passage, and LLM-generated hard negatives. FieldContent Querywhat are the liberal arts? Positive Passageliberal arts. 1. the academic course of instruction at a college intended to provide general knowledge and comprising the arts, humanities, natural sciences, and social sciences, as opposed to professional or technical subjects. LLM Generated Negative 1liberal studies. 1. an educational program focusing on in-depth study of specific professional skills, such as business or technology, often intended to prepare students for direct entry into the workforce. LLM Generated Negative 2higher education. 1. the stage of learning that occurs at universities and colleges, emphasizing specialized professional training and occupational skills rather than a broad liberal arts education. LLM Generated Negative 3vocational education. 1. a type of education that prepares students for specific trades or careers, focusing heavily on practical and technical skills rather than on a comprehensive understanding of the humanities and sciences. 7.5.2 Analysis: Why High Similarity Leads to Low Utility. The gener- ated “hard_negative_document_1” and “hard_negative_document_3” illustrate the fundamental flaw in unfiltered LLM generation: (1)Lexical Mimicry without Relevance: The LLM expertly mimics the form of the positive passage—using dictionary- style definitions, numbering, and academic vocabulary (“in- struction,” “program,” “education”). In an embedding space, this results in a very high cosine similarity (푆 푛 ), satisfying the “Hardness” criterion. (2)Usage of Antonyms: Crucially, the LLM creates a “Lib- eral Arts” negative by defining it with characteristics of its antonym: “Vocational Training.” It defines “Liberal Studies” as a program for “business or technology” (Vocational). (3)Boundary Violation: This creates a deceptive negative. To a human, “Liberal Studies” defined as “Technical Training” is clearly irrelevant. To a dense retriever, however, the vector for “Liberal Studies defined as Technical Training” overlaps significantly with “Liberal Arts defined as General Knowl- edge” because the word vectors for “Study,” “Arts,” and “Edu- cation” dominate the representation. (4)Impact on ECI: The ECI metric detects this via the Max- Margin (Δ 푚푎푥 ). Because the negative is so semantically close to the positive in the embedding space (due to shared topic terms), the safety margin collapses. The harmonic mean in ECI heavily penalizes this, correctly identifying that while the sample is “hard,” it is unsafe to train on because it teaches the model to disassociate the concept of “education” from the specific query intent. This example demonstrates that LLMs optimize for textual simi- larity (high perplexity, fluent definitions) rather than information- theoretic safety. Consequently, they produce negatives that are se- mantically rich but represent “false positives” in the vector space, leading to the instability captured by the low ECI scores. 8 Conclusion In this work, we introduced Effective Contrastive Information (ECI), a theoretically grounded, training-free metric designed to assess the quality of hard-negative sets prior to model fine-tuning. By deriving ECI from Information Theory and the InfoNCE lower bound, we provide a mathematical framework that balances the Information Capacity of a dataset with its Discriminative Efficiency. A key contribution of this metric is the use of a harmonic mean to model the trade-off between Signal Magnitude (Hardness) and Margin Safety. This formulation ensures that negative sets are not only semantically challenging but also remain strictly non-relevant, effectively penalizing the label noise often found in unfiltered LLM- generated negatives. Our empirical evaluation across twelve BEIR datasets demon- strates that ECI is a robust predictor of downstream retrieval perfor- mance (푛퐷퐶퐺@10). The results highlight that while individual min- ing strategies like Cross-Encoders or LLMs may maximize specific attributes (such as signal strength), hybrid strategies—specifically the BM25 + Cross-Encoder approach—achieve the optimal ECI score by maintaining high signal while preserving a safe discrimi- native boundary. Ultimately, ECI provides a time and cost-efficient alternative to exhaustive end-to-end ablation studies. By serving as a reliable proxy for evaluating negative mining strategies prior to training and fine-tuning, ECI reduces both experimental turnaround time and computational expenditure, thereby lowering the overall energy and financial cost of developing modern dense retrieval systems. 9 Limitations While ECI is promising, we only use 10,000 passages from MS- MARCO out of the 500,000+ that are present, this is mainly due to computational and time limitations. References [1]Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. arXiv:1611.09268 [cs.CL] https://arxiv.org/abs/1611.09268 [2]Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. 2022. InPars: Unsupervised Dataset Generation for Information Retrieval. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (Madrid, Spain) (SIGIR ’22). Association for Computing Machinery, New York, NY, USA, 2387–2392. doi:10.1145/3477495.3531863 [3]Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Ste- fanie Jegelka. 2020. Debiased contrastive learning. Advances in neural information processing systems 33 (2020), 8765–8775. ECI: Effective Contrastive Information to Evaluate Hard-NegativesConference acronym ’X, June 03–05, 2018, Woodstock, NY [4]Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith Hall, and Ming-Wei Chang. 2023. Promptagator: Few-shot Dense Retrieval From 8 Examples. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=gmL46YMpu2J [5]Luyu Gao and Jamie Callan. 2021. Condenser: a Pre-training Architecture for Dense Retrieval. arXiv:2104.08253 [cs.CL] https://arxiv.org/abs/2104.08253 [6] Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup. In EMNLP. [7] Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple Contrastive Learning of Sentence Embeddings. In EMNLP. [8]Gautier Izacard, Mathilde Caron, Sebastian Riedel, and Edouard Grave. 2022. Unsupervised Dense Information Retrieval with Contrastive Learning. arXiv:2112.09118 [cs.IR] https://arxiv.org/abs/2112.09118 [9]Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open- Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 6769–6781. doi:10.18653/v1/2020.emnlp-main.550 [10]Sean Lee, Aamir Shakir, Darius Koenig, and Julius Lipp. 2024. Open Source Strikes Bread - New Fluffy Embeddings Model. https://w.mixedbread.ai/blog/mxbai- embed-large-v1 [11]Xianming Li and Jing Li. 2023. AnglE-optimized Text Embeddings. arXiv preprint arXiv:2309.12871 (2023). [12]Xiaopeng Li, Xiangyang Li, Hao Zhang, Zhaocheng Du, Pengyue Jia, Yichao Wang, Xiangyu Zhao, Huifeng Guo, and Ruiming Tang. 2024. SyNeg: LLM- Driven Synthetic Hard-Negatives for Dense Retrieval. arXiv:2412.17250 [cs.IR] https://arxiv.org/abs/2412.17250 [13]Wei Lu, Nan Yang, and Mingliang Li. 2021. Hard Negative Sampling for Dense Text Retrieval. In SIGIR. [14]Xing Han Lù. 2024. BM25S: Orders of magnitude faster lexical search via eager sparse scoring. arXiv:2407.03618 [cs.IR] https://arxiv.org/abs/2407.03618 [15]Gabriel de Souza P. Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, and Even Oldridge. 2024. NV-Retriever: Improving Text Embedding Models with Effective Hard-Negative Mining. arXiv:2407.15831 [cs.IR] [16]Jianmo Ni, Chen Qu, Danqi Chen, et al.2022. Large Dual Encoders Are General- izable Retrievers. In EMNLP. [17]OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Car- ney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoochian, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braun- stein, Andrew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, An- drew Tulloch, Andrey Mishchenko, Angela Baek, Angela Jiang, Antoine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samic, Bob Mc- Grew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern, Channing Conger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Winter, Coley Czarnecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, Dane Sherburn, Daniel Kappler, Daniel Levin, Daniel Levy, David Carr, David Farhi, David Mely, David Robinson, David Sasaki, Denny Jin, Dev Valladares, Dimitris Tsipras, Doug Li, Duc Phong Nguyen, Duncan Findlay, Edede Oiwoh, Edmund Wong, Ehsan Asdar, Elizabeth Proehl, Elizabeth Yang, Eric Antonow, Eric Kramer, Eric Peterson, Eric Sigler, Eric Wallace, Eugene Brevdo, Evan Mays, Farzad Khorasani, Felipe Petroski Such, Filippo Raso, Francis Zhang, Fred von Lohmann, Freddie Sulit, Gabriel Goh, Gene Oden, Geoff Salmon, Giulio Starace, Greg Brockman, Hadi Salman, Haiming Bao, Haitang Hu, Hannah Wong, Haoyu Wang, Heather Schmidt, Heather Whitney, Heewoo Jun, Hendrik Kirchner, Henrique Ponde de Oliveira Pinto, Hongyu Ren, Huiwen Chang, Hyung Won Chung, Ian Kivlichan, Ian O’Connell, Ian O’Connell, Ian Osband, Ian Silber, Ian Sohl, Ibrahim Okuyucu, Ikai Lan, Ilya Kostrikov, Ilya Sutskever, Ingmar Kanitscheider, Ishaan Gulrajani, Jacob Coxon, Jacob Menick, Jakub Pachocki, James Aung, James Betker, James Crooks, James Lennon, Jamie Kiros, Jan Leike, Jane Park, Jason Kwon, Jason Phang, Jason Teplitz, Jason Wei, Jason Wolfe, Jay Chen, Jeff Harris, Jenia Varavva, Jessica Gan Lee, Jessica Shieh, Ji Lin, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joanne Jang, Joaquin Quinonero Candela, Joe Beutler, Joe Landers, Joel Parish, Johannes Heidecke, John Schul- man, Jonathan Lachman, Jonathan McKay, Jonathan Uesato, Jonathan Ward, Jong Wook Kim, Joost Huizinga, Jordan Sitkin, Jos Kraaijeveld, Josh Gross, Josh Kaplan, Josh Snyder, Joshua Achiam, Joy Jiao, Joyce Lee, Juntang Zhuang, Justyn Harriman, Kai Fricke, Kai Hayashi, Karan Singhal, Katy Shi, Kavin Karthik, Kayla Wood, Kendra Rimbach, Kenny Hsu, Kenny Nguyen, Keren Gu-Lemberg, Kevin Button, Kevin Liu, Kiel Howe, Krithika Muthukumar, Kyle Luther, Lama Ahmad, Larry Kai, Lauren Itow, Lauren Workman, Leher Pathak, Leo Chen, Li Jing, Lia Guy, Liam Fedus, Liang Zhou, Lien Mamitsuka, Lilian Weng, Lindsay McCallum, Lindsey Held, Long Ouyang, Louis Feuvrier, Lu Zhang, Lukas Kondraciuk, Lukasz Kaiser, Luke Hewitt, Luke Metz, Lyric Doshi, Mada Aflak, Maddie Simens, Made- laine Boyd, Madeleine Thompson, Marat Dukhan, Mark Chen, Mark Gray, Mark Hudnall, Marvin Zhang, Marwan Aljubeh, Mateusz Litwin, Matthew Zeng, Max Johnson, Maya Shetty, Mayank Gupta, Meghan Shah, Mehmet Yatbaz, Meng Jia Yang, Mengchao Zhong, Mia Glaese, Mianna Chen, Michael Janner, Michael Lampe, Michael Petrov, Michael Wu, Michele Wang, Michelle Fradin, Michelle Pokrass, Miguel Castro, Miguel Oom Temudo de Castro, Mikhail Pavlov, Miles Brundage, Miles Wang, Minal Khan, Mira Murati, Mo Bavarian, Molly Lin, Mu- rat Yesildal, Nacho Soto, Natalia Gimelshein, Natalie Cone, Natalie Staudacher, Natalie Summers, Natan LaFontaine, Neil Chowdhury, Nick Ryder, Nick Stathas, Nick Turley, Nik Tezak, Niko Felix, Nithanth Kudige, Nitish Keskar, Noah Deutsch, Noel Bundick, Nora Puckett, Ofir Nachum, Ola Okelola, Oleg Boiko, Oleg Murk, Oliver Jaffe, Olivia Watkins, Olivier Godement, Owen Campbell-Moore, Patrick Chao, Paul McMillan, Pavel Belov, Peng Su, Peter Bak, Peter Bakkum, Peter Deng, Peter Dolan, Peter Hoeschele, Peter Welinder, Phil Tillet, Philip Pronin, Philippe Tillet, Prafulla Dhariwal, Qiming Yuan, Rachel Dias, Rachel Lim, Rahul Arora, Rajan Troll, Randall Lin, Rapha Gontijo Lopes, Raul Puri, Reah Miyara, Reimar Leike, Renaud Gaubert, Reza Zamani, Ricky Wang, Rob Donnelly, Rob Honsby, Rocky Smith, Rohan Sahai, Rohit Ramchandani, Romain Huet, Rory Carmichael, Rowan Zellers, Roy Chen, Ruby Chen, Ruslan Nigmatullin, Ryan Cheu, Saachi Jain, Sam Altman, Sam Schoenholz, Sam Toizer, Samuel Miserendino, Sandhini Agarwal, Sara Culver, Scott Ethersmith, Scott Gray, Sean Grove, Sean Metzger, Shamez Hermani, Shantanu Jain, Shengjia Zhao, Sherwin Wu, Shino Jomoto, Shirong Wu, Shuaiqi, Xia, Sonia Phene, Spencer Papay, Srinivas Narayanan, Steve Coffey, Steve Lee, Stewart Hall, Suchir Balaji, Tal Broda, Tal Stramer, Tao Xu, Tarun Gogineni, Taya Christianson, Ted Sanders, Tejal Patwardhan, Thomas Cunninghman, Thomas Degry, Thomas Dimson, Thomas Raoux, Thomas Shad- well, Tianhao Zheng, Todd Underwood, Todor Markov, Toki Sherbakov, Tom Rubin, Tom Stasi, Tomer Kaftan, Tristan Heywood, Troy Peterson, Tyce Wal- ters, Tyna Eloundou, Valerie Qi, Veit Moeller, Vinnie Monaco, Vishal Kuo, Vlad Fomenko, Wayne Chang, Weiyi Zheng, Wenda Zhou, Wesam Manassra, Will Sheu, Wojciech Zaremba, Yash Patil, Yilei Qian, Yongjik Kim, Youlong Cheng, Yu Zhang, Yuchen He, Yuchen Zhang, Yujia Jin, Yunxing Dai, and Yury Malkov. 2024. GPT-4o System Card. arXiv:2410.21276 [cs.CL] https://arxiv.org/abs/2410.21276 [18]Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxi- ang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering. arXiv:2010.08191 [cs.CL] https://arxiv.org/abs/2010.08191 [19] Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Em- pirical Methods in Natural Language Processing. Association for Computational Linguistics. http://arxiv.org/abs/1908.10084 [20]Xiaozhi Ren, Chen Qu, and Chenyan Xiong. 2021. DRBoost: Improving Dense Retrieval with Boosted Negative Sampling. In CIKM. [21] Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance frame- work: BM25 and beyond. Vol. 4. Now Publishers Inc. [22] Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. 2021. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592 (2021). [23]Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. 2021. Contrastive Learning with Hard Negative Samples. arXiv:2010.04592 [cs.LG] https://arxiv.org/abs/2010.04592 [24]Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv:1910.01108 [cs.CL] https://arxiv.org/abs/1910.01108 [25]Aamir Shakir, Darius Koenig, Julius Lipp, and Sean Lee. 2024. Boost Your Search With The Crispy Mixedbread Rerank Models. https://w.mixedbread.ai/blog/ mxbai-rerank-v1 [26]Aarush Sinha. 2025. Don’t Retrieve, Generate: Prompting LLMs for Synthetic Training Data in Dense Retrieval. arXiv:2504.21015 [cs.IR] https://arxiv.org/abs/ 2504.21015 [27]Aarush Sinha, Pavan Kumar S, Roshan Balaji, and Nirav Pravinbhai Bhatt. 2025. BiCA: Effective Biomedical Dense Retrieval with Citation-Aware Hard Negatives. arXiv:2511.08029 [cs.IR] https://arxiv.org/abs/2511.08029 [28] Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). https://openreview. net/forum?id=wCu6T5xFjeJ [29]Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. 2022. GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval. arXiv:2112.07577 [cs.CL] https://arxiv.org/abs/2112.07577 Conference acronym ’X, June 03–05, 2018, Woodstock, NYSinha et al. [30]Liang Wang, Nan Yang, Xiaolong Huang, et al.2022. Text Embeddings by Weakly- Supervised Contrastive Pre-training. In ACL. [31]Shiqi Wang, Yeqin Zhang, and Cam-Tu Nguyen. 2023. Mitigating the Impact of False Negatives in Dense Retrieval with Contrastive Confidence Regularization. arXiv:2401.00165 [cs.IR] [32]Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jingfei Liu, Paul Bennett, and Junaid Ahmed. 2021. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. In ICLR. [33] Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. arXiv:2007.00808 [cs.IR] https://arxiv.org/abs/2007.00808 [34]Zhen Yang, Zhou Shao, Yuxiao Dong, and Jie Tang. 2022. TriSampler: A Better Negative Sampling Principle for Dense Retrieval. In Proceedings of AAAI. [35] Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2021. Optimizing Dense Retrieval Model Training with Hard Negatives. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. [36]Yi Zhang and Jia Deng. 2022. Robust Contrastive Learning against Noisy Labels. In CVPR. Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009