Paper deep dive
Rescaling MLM-Head for Neural Sparse Retrieval
Youngjoon Jang, Seongtae Hong, Jonah Turner, Heuiseok Lim
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 98%
Last extracted: 6/21/2026, 3:06:37 AM
Summary
The paper identifies a scale mismatch issue in Learned Sparse Retrieval (LSR) models like SPLADE, where backbones with large MLM-head L2 norms (e.g., ModernBERT, Ettin) suffer from performance degradation and training instability. The authors propose a simple initialization-time correction that rescales the MLM-head projection by a constant factor (k) to stabilize training and improve retrieval effectiveness across in-domain and out-of-domain benchmarks.
Entities (8)
Relation Signals (5)
MLM-head → hasproperty → L2 norm
confidence 100% · backbones with large MLM-head L2 norms can suffer performance degradation
MLM-head Rescaling → improves → ModernBERT
confidence 100% · With k=16, ModernBERT achieves a 215% relative improvement on BEIR-13 mean nDCG@10
MLM-head Rescaling → improves → Ettin
confidence 100% · while Ettin improves by 77%, relative to their unscaled counterparts.
MLM-head → isusedby → SPLADE
confidence 100% · SPLADE directly uses MLM-head outputs to construct sparse lexical representations
MLM-head scale → causes → training instability
confidence 90% · an inflated MLM-head scale can amplify sparse activations, distort matching scores, and destabilize contrastive training
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Learned sparse retrieval (LSR) models such as SPLADE have traditionally used BERT-style masked language models as backbone encoders. A natural expectation is that replacing BERT with stronger pretrained encoders should improve retrieval effectiveness. However, we find that under standard SPLADE training recipes, backbones with large MLM-head L2 norms can suffer performance degradation and even training collapse under standard SPLADE training recipes. We identify this failure as a scale mismatch in the MLM head: SPLADE directly uses MLM-head outputs to construct sparse lexical representations, and query-document relevance is computed by an unnormalized dot product over these representations. As a result, an inflated MLM-head scale can amplify sparse activations, distort matching scores, and destabilize contrastive training under common training settings. To address this issue, we introduce a simple initialization-time correction that rescales the MLM-head projection by a constant factor before SPLADE training. This zero-cost adjustment improves training stability without modifying the model architecture or training objective. Across both in-domain and out-of-domain retrieval benchmarks, this simple correction substantially improves large-norm backbones such as ModernBERT and Ettin, turning unstable training runs into competitive sparse retrievers. In several settings, the corrected models further match or surpass the classic BERT-SPLADE baseline. These findings suggest that the bottleneck in adapting pretrained encoders to LSR is not encoder capacity alone, but the calibration of the MLM-head scale used to construct sparse lexical representations.
Tags
Links
- Source: https://arxiv.org/abs/2606.18811v1
- Canonical: https://arxiv.org/abs/2606.18811v1
Trouble viewing inline? Open PDF directly →
Full Text
28,138 characters extracted from source content.
Expand or collapse full text
Rescaling MLM-Head for Neural Sparse Retrieval Youngjoon Jang Korea University Seoul, South Korea dew1701@korea.ac.kr Seongtae Hong Korea University Seoul, South Korea ghdchlwls123@korea.ac.kr Jonah Turner Independent Seoul, South Korea drexalt@gmail.com Heuiseok Lim ∗ Korea University Seoul, South Korea limhseok@korea.ac.kr Abstract Learned sparse retrieval (LSR) models such as SPLADE have tradi- tionally used BERT-style masked language models as backbone en- coders. A natural expectation is that replacing BERT with stronger pretrained encoders should improve retrieval effectiveness. How- ever, we find that under standard SPLADE training recipes, back- bones with large MLM-head L2 norms can suffer performance degra- dation and even training collapse under standard SPLADE training recipes. We identify this failure as a scale mismatch in the MLM head: SPLADE directly uses MLM-head outputs to construct sparse lexical representations, and query-document relevance is computed by an unnormalized dot product over these representations. As a result, an inflated MLM-head scale can amplify sparse activations, distort matching scores, and destabilize contrastive training under common training settings. To address this issue, we introduce a simple initialization-time correction that rescales the MLM-head projection by a constant factor before SPLADE training. This zero- cost adjustment improves training stability without modifying the model architecture or training objective. Across both in-domain and out-of-domain retrieval benchmarks, this simple correction sub- stantially improves large-norm backbones such as ModernBERT and Ettin, turning unstable training runs into competitive sparse retrievers. In several settings, the corrected models further match or surpass the classic BERT-SPLADE baseline. These findings suggest that the bottleneck in adapting pretrained encoders to LSR is not encoder capacity alone, but the calibration of the MLM-head scale used to construct sparse lexical representations. CCS Concepts • Information systems→ Information retrieval. Keywords Information Retrieval, Learned Sparse Retrieval, SPLADE Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. Conference acronym ’X, Woodstock, NY © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-X-X/2018/06 https://doi.org/X.X ACM Reference Format: Youngjoon Jang, Seongtae Hong, Jonah Turner, and Heuiseok Lim. 2018. Rescaling MLM-Head for Neural Sparse Retrieval. In Proceedings of Make sure to enter the correct conference title from your rights confirmation email (Conference acronym ’X). ACM, New York, NY, USA, 5 pages. https://doi. org/X.X 1 INTRODUCTION Learned sparse retrieval (LSR) has emerged as an effective compro- mise between lexical and neural retrieval. By representing queries and documents as sparse vectors over the vocabulary, LSR models retain the efficiency of inverted-index search while enabling neural term expansion and learned term weighting. Among LSR meth- ods, SPLADE has become a widely adopted architecture, achieving strong retrieval effectiveness while preserving the interpretability and deployability of lexical matching [10, 11, 14, 21]. A key property of SPLADE is that it constructs sparse lexical rep- resentations through the masked language modeling (MLM) head of a pretrained encoder. Unlike dense retrievers, whose representa- tion space is typically learned through a task-specific projection, SPLADE directly reuses the pretrained vocabulary projection to define the sparse retrieval space. This makes the MLM head central not only to pretraining, but also to retrieval scoring. This dependence suggests a natural path for improving SPLADE: replacing its conventional BERT backbone with stronger modern encoders. Recent encoder families such as GTE-MLM [15], Modern- BERT [24], and Ettin-encoder [25] introduce stronger pretraining recipes and architectural improvements, including rotary positional embeddings and pre-normalization. Under the common drop-in- backbone assumption, these advances should translate into stronger SPLADE models [1]. However, this assumption does not hold in practice. We train SPLADE models with different encoder backbones on MS MARCO [3] using the same training recipe and evaluate them on BEIR-13. As shown in Figure 1, backbones with small MLM-head L2 norms (||푊||<2), such as BERT, DistilBERT, ALBERT, and GTE-MLM, produce strong sparse retrievers. In contrast, backbones with larger norms (||푊||>2), including RoBERTa, ModernBERT, and Ettin, substantially underperform the BERT baseline. This result is coun- terintuitive: stronger and more recent encoders can become weaker SPLADE backbones under the standard training recipe. We identify the scale of the MLM head as a key factor behind this failure. In SPLADE, both query and document representations arXiv:2606.18811v1 [cs.IR] 17 Jun 2026 Conference acronym ’X, June 03–05, 2018, Woodstock, NYJang et al. BERTDistilBERTALBERTGTE-MLMRoBERTaModernBERTEttin 0.0 0.1 0.2 0.3 0.4 BEIR nDCG@10 0.370 0.383 0.361 0.403 0.331 0.127 0.221 ‖W‖<2‖W‖>2 Figure 1: MLM Head L2 norm||푊||and BEIR-13 effectiveness of MS MARCO-trained SPLADE models for each backbone. are produced through the MLM head and compared using an unnor- malized sparse dot product. Therefore, an overly large MLM-head scale can amplify sparse activations, inflate matching scores, and distort the optimization dynamics of contrastive retrieval training. Based on this observation, we propose a simple initialization- time correction: rescaling the MLM-head matrix by a constant factor 푘before SPLADE training. This operation preserves the direction and lexical information of the pretrained vocabulary projection, introduces no additional parameters, requires no architectural mod- ification, and adds no training or inference cost. Despite its simplicity, this rescaling substantially improves SPLADE models built on encoders with large MLM-head norms. With푘=16, ModernBERT achieves a 215% relative improvement on BEIR-13 mean nDCG@10, while Ettin improves by 77%, relative to their unscaled counterparts. Across in-domain and out-of-domain bench- marks, we show that rescaled encoders recover strong SPLADE performance under the standard training recipe. These results sug- gest that MLM-head scale, rather than encoder capacity alone, is a critical factor for adapting pretrained encoders to learned sparse retrieval. 2 RELATED WORK Learned sparse retrieval represents queries and documents as sparse vectors over a vocabulary, combining the efficiency and interpretabil- ity of lexical retrieval with neural term weighting and expansion [11, 19]. This design preserves compatibility with inverted-index search, in contrast to dense retrievers that rely on low-dimensional contin- uous representations [7,23,26], and has motivated recent work on efficient sparse indexing and retrieval systems [4–6]. Among learned sparse retrieval methods, SPLADE has become a representative architecture. Subsequent variants have improved retrieval effectiveness through hard negative mining, knowledge distillation, sparsity regularization, and larger or more diverse train- ing data [14,16,18]. Other studies have analyzed the role of vo- cabularies and representation capacity in sparse retrieval mod- els [12,13,17]. However, these works largely preserve the core SPLADE formulation, where sparse lexical representations are pro- duced through the pretrained vocabulary projection. Backbone choice is another natural factor in SPLADE perfor- mance. SPLADE has traditionally relied on masked language mod- els [9], while recent encoder families such as GTE-MLM [15], Mod- ernBERT [24], and Ettin-encoder [25] provide stronger pretrained Table 1: Backbone statistics.||푊||∗meanand||푊||∗maxdenote the mean and maximum row-wise L2 norm of the MLM-head. Model# Params ||푊||∗ mean ||푊||∗ max Anisotropy BERT-base0.10B1.4012.0450.445 DistilBERT-base0.07B1.6602.6110.532 ALBERT-base0.011B0.4070.7370.108 GTE-MLM-base0.10B1.2781.8140.085 RoBERTa-base0.10B3.6454.4800.024 ModernBERT-base0.10B2.5534.2730.044 Ettin-encoder-150m0.15B2.6284.5060.036 representations and appear to be natural replacements for BERT- style backbones. However, adapting these encoders to SPLADE also changes the vocabulary projection used for sparse retrieval. Table 1 summarizes the MLM-head statistics of the backbones considered in this work. The encoders differ not only in architec- ture and pretraining recipe, but also in the scale and geometry of their MLM-head matrices. While anisotropy has been discussed as an important property of representation geometry [2], it does not align with the failure pattern observed in Figure 1: RoBERTa, ModernBERT and Ettin have relatively low anisotropy, yet perform poorly as SPLADE backbones under the standard training recipe. In contrast, the MLM-head norm provides a more consistent signal for the observed degradation. This motivates our focus on MLM- head scale as a previously overlooked factor in adapting modern encoders to learned sparse retrieval. 3 METHOD 3.1 SPLADE Training Objective We briefly formalize SPLADE to isolate how the scale of the MLM head affects sparse activations, retrieval scores, and training dynam- ics. Let푊 ℎ푒푎푑 ∈R |V|×푑 denote the MLM-head matrix that projects hidden states into vocabulary logits. Given contextualized token representations퐻, SPLADE constructs token-level vocabulary acti- vations as 퐴= log(1+ ReLU(퐻푊 ⊤ ℎ푒푎푑 ))(1) where the bias term is omitted for simplicity. The sequence-level sparse vector is then obtained by element-wise max pooling over token positions: v= max 푖 퐴 푖 (2) For a query푞and a document푑, the retrieval score is computed by an unnormalized sparse dot product: s(푞,푑)=v ⊤ 푞 v 푑 (3) Given query-positive-hard negative triples(푞 푖 ,푑 + 푖 ,푑 − 푖 ) 퐵 푖=1 , we optimize an in-batch contrastive loss: L rank =− 1 퐵 퐵 ∑︁ 푖=1 log 푒 (s(푞 푖 ,푑 + 푖 )/휏) Í 퐵 푗=1 h 푒 (s(푞 푖 ,푑 + 푗 )/휏) +푒 (s(푞 푖 ,푑 − 푗 )/휏) i (4) where휏is a temperature parameter, and푠denotes the similarity function (dot product). Following SPLADE, we additionally use FLOPS regularization [20] to encourage sparse activations: 퐿= 퐿 rank + 휆 푞 퐿 FLOPS (v 푞 )+ 휆 푑 퐿 FLOPS (v 푑 )(5) Rescaling MLM-Head for Neural Sparse RetrievalConference acronym ’X, June 03–05, 2018, Woodstock, NY 3.2 MLM-Head Rescaling The SPLADE objective makes the scale of the MLM head important. Since both query and document sparse vectors are produced from 퐻푊 ⊤ and then multiplied in an unnormalized dot product, the norm of푊can strongly affect the magnitude of sparse activations, retrieval scores, and gradients from the ranking loss. When the MLM-head scale is large, the ranking objective and FLOPS regu- larizer can be placed in sharper tension, making SPLADE training unstable or ineffective. To control this factor, we apply a simple initialization-time correction before training: 푊 ℎ푒푎푑 ← 푊 ℎ푒푎푑 푘 (6) Here,푘=1 corresponds to the standard SPLADE initialization. For tied MLM heads, the operation is applied to the shared weight ma- trix so that weight tying is preserved. All other model parameters, architecture, training data, and objectives remain unchanged. Thus, the proposed correction introduces no additional parameters and requires no architectural modification. 4 EXPERIMENTAL SETUP 4.1 Training Details Backbone models. We evaluate seven encoder backbones that cover both BERT-style and modern encoder families: bert-base- uncased, distilbert-base-uncased, albert-base-v2, gte-en-mlm, roberta- base, ModernBERT-base, and ettin-encoder-150m. These models differ in pretraining recipes, training data, architectural choices, and MLM-head statistics, allowing us to analyze whether SPLADE performance is determined primarily by encoder capacity or by the scale of the MLM-head projection. Unless otherwise specified, each backbone is trained under the same SPLADE recipe so that perfor- mance differences can be attributed to the backbone initialization and the proposed rescaling. Data. All SPLADE models are trained on MS MARCO triplets 1 using the same training objective, which combines a ranking loss with FLOPS regularization to encourage sparse representations. We keep the training recipe fixed across backbones, including the data, regularization setting, batch size, learning rate, and number of training steps. This controlled setup allows us to isolate the effect of MLM-head scale from other training factors. Hyperparameters. We optimize with AdamW at a peak learning rate of 1푒−4 under a linear schedule with a warmup ratio of 0.1 for one epoch over the MS MARCO triplets. We use a per-GPU batch size of 64 on 8 NVIDIA RTX A6000 GPUs with Distributed Data Parallelism (DDP), a maximum sequence length of 256, and bfloat16 mixed precision. The ranking loss uses in-batch negatives with temperature휏=1.0. For sparsity, we set the FLOPS regularizer weights to휆 푞 =5푒−2 and휆 푑 =3푒−3, ramped from zero with a quadratic schedule over the first third of training. 4.2 Evaluation We evaluate retrieval effectiveness on both in-domain and out-of- domain benchmarks. For in-domain evaluation, we report results 1 https://huggingface.co/datasets/sentence-transformers/msmarco-msmarco- MiniLM-L6-v3 Table 2: Retrieval performance of SPLADE models with dif- ferent MLM-head rescaling factors. The value in parentheses denotes the resulting MLM-head L2 norm∥푊∥. Model푘 (∥푊∥) MSMARCO DEVTREC 2019 BEIR-13 MRR@10 Recall@1000 nDCG@10 nDCG@10 BERT1 (1.40).254.931.594.370 DistilBERT1 (1.66).259.936.594.383 ALBERT1 (0.41).255.926.530.361 GTE-MLM1 (1.28).273.953.635.403 RoBERTa 1 (3.65).194.900.554.331 2 (1.82).226.934.590.386 4 (0.91).228.933.605.383 8 (0.46).173.875.468.328 16 (0.23).000.008.000.004 ModernBERT 1 (2.55).045.554.266.127 2 (1.28).201.909.502.351 4 (0.64).214.925.570.373 8 (0.32).248.948.588.405 16 (0.16).250.948.609.400 Ettin 1 (2.63).099.752.354.221 2 (1.31).118.777.416.244 4 (0.66).175.890.530.332 8 (0.33).178.899.538.330 16 (0.16).238.940.587.391 on MS MARCO dev [3] and TREC-DL 2019 [8]. For out-of-domain evaluation, we use BEIR-13 [22] and report the mean nDCG@10 across tasks. We compare the standard initialization (푘=1) with rescaled variants and analyze how the choice of푘affects SPLADE training and retrieval effectiveness. 5 EXPERIMENTAL RESULTS AND ANALYSIS 5.1 Main Result Table 2 reports the retrieval performance of SPLADE models across MS MARCO Dev, TREC-DL 2019, and BEIR-13. We compare the standard SPLADE initialization (푘=1) with MLM-head rescaling using different values of푘. The number in parentheses indicates the resulting MLM-head L2 norm after rescaling. First, performance under the standard SPLADE initialization is more closely aligned with MLM-head scale than with encoder capacity. Backbones with relatively small MLM-head norms, includ- ing BERT, DistilBERT, ALBERT, and GTE-MLM, consistently pro- duce competitive sparse retrievers, with BEIR-13 mean nDCG@10 ranging from 0.361 to 0.403. In contrast, backbones with larger MLM-head norms substantially underperform: RoBERTa drops to 0.331 on BEIR-13, while ModernBERT and Ettin collapse more se- verely to 0.127 and 0.221, respectively. This shows that stronger pretrained encoders are not necessarily reliable drop-in replace- ments for BERT in SPLADE when their MLM-head scale is not properly calibrated. Second, MLM-head rescaling recovers the de- graded large-norm backbones, but the effect is not monotonic. For ModernBERT, BEIR-13 nDCG@10 improves from 0.127 to 0.405, while MS MARCO MRR@10 improves from 0.045 to 0.250 and TREC-DL 2019 nDCG@10 from 0.266 to 0.609. Ettin also improves substantially, with BEIR-13 nDCG@10 increasing from 0.221 to 0.391 at푘=16. At the same time, the optimal rescaling factor depends on the backbone: RoBERTa benefits from moderate rescal- ing and peaks at푘=2, whereas ModernBERT and Ettin require Conference acronym ’X, June 03–05, 2018, Woodstock, NYJang et al. 10 1 10 2 10 3 # FLOPs 0.2 0.3 0.4 0.5 NanoBEIR nDCG@10 BERT DistilBERT ModernBERT ÷1 ModernBERT ÷16 Ettin ÷1 Ettin ÷16 Figure 2: Effectiveness–sparsity trade-off on NanoBEIR under different FLOPS regularization strengths. stronger rescaling. Excessive down-scaling, however, can collapse performance, as shown by RoBERTa at푘=16. These results indi- cate that the MLM-head scale must be controlled. 5.2 Effectiveness–Sparsity Trade-off We next examine whether rescaling improves not only retrieval effectiveness but also the sparsity–effectiveness trade-off. For this analysis, we vary the FLOPS regularization weights over휆 푞 ∈ 1푒 −2,5푒 −2,2푒 −1and휆 푑 ∈ 6푒 −4,3푒 −3,1푒 −2, while keeping the other training settings fixed. Figure 2 plots NanoBEIR mean nDCG@10 against the number of activated FLOPs. Successful SPLADE models should achieve high effectiveness while keeping FLOPs low, since sparse retrieval depends on compact inverted- index representations. Without rescaling, ModernBERT and Ettin exhibit an unfavor- able trade-off: they often require substantially more FLOPs than BERT and DistilBERT while achieving lower NanoBEIR effective- ness. After MLM-head rescaling, both models move toward the high- effectiveness, low-FLOPs region occupied by successful SPLADE backbones. This suggests that rescaling improves the effectiveness– sparsity trade-off by producing better-calibrated sparse activations, rather than simply increasing the density of the representations. 5.3 Rescaling Requires an Appropriate Scale We also test whether rescaling is simply a universally beneficial reg- ularization trick. Figure 3 shows BEIR-13 nDCG@10 as the rescal- ing factor varies for backbones whose initial MLM-head norms are already relatively small. Moderate rescaling can slightly im- prove BERT, DistilBERT, and GTE-MLM, but excessive rescaling consistently degrades performance. ALBERT is the clearest case: because its initial MLM-head norm is already very small, further down-scaling rapidly collapses retrieval effectiveness. These results show that the benefit of rescaling does not come from shrinking the MLM head as much as possible. Instead, SPLADE training requires the MLM-head scale to lie within an appropriate range. Large-norm backbones such as ModernBERT and Ettin ben- efit from stronger rescaling because their initial projection scale is 0.34 0.36 0.38 0.40 0.42 124816 Rescaling factor k 0 BEIR-13 nDCG@10 BERT DistilBERT ALBERT GTE-MLM RoBERTa ModernBERT Ettin Figure 3: Effect of MLM-head rescaling on BEIR. Lines show small-norm backbones across rescaling factors, while stars indicate the best rescaled results for large-norm backbones. too large, whereas already small-norm backbones can be harmed when the projection is scaled down too aggressively. Together with the main results, this suggests that MLM-head scale is a critical but previously overlooked factor in adapting pretrained encoders to learned sparse retrieval. 5.4 Rescaling Stabilizes Training Finally, Figure 4 analyzes the training dynamics of the two large- norm modern encoders. Without rescaling (푘=1), ModernBERT and Ettin start from extremely large losses and converge to substan- tially higher final losses than their rescaled variants. As푘increases, the loss scale is reduced and training becomes smoother, indicating that MLM-head rescaling directly stabilizes the optimization pro- cess. This supports our interpretation that large MLM-head norms distort SPLADE training by inflating the score scale in the con- trastive objective, while rescaling moves modern encoders into a more appropriate optimization regime. 0.00.20.40.60.81.0 Epoch 10 −1 10 0 10 1 10 2 10 3 10 4 Loss ModernBERT 0.60.81.0 10 −1 10 0 0.00.20.40.60.81.0 Epoch Ettin 0.60.81.0 10 −1 10 0 k=1k=2k=4k=8 k=16 Figure 4: Training loss curves of ModernBERT- and Ettin- based SPLADE models under different rescaling factors. Rescaling MLM-Head for Neural Sparse RetrievalConference acronym ’X, June 03–05, 2018, Woodstock, NY 6 CONCLUSION In this work, we showed that encoder backbones with large MLM- head scales can fail under the standard SPLADE training recipe, even when the underlying pretrained encoders are strong. Through experiments across multiple backbones, we identified the MLM- head L2 norm as a key factor associated with this failure: large-norm MLM heads amplify sparse activations and destabilize retrieval training, whereas appropriately scaled heads lead to more effective sparse retrievers. To address this issue, we introduced a simple initialization-time rescaling correction that adjusts the MLM-head matrix before training. This zero-cost modification restores the effectiveness–sparsity trade-off of large-norm SPLADE backbones and substantially improves both in-domain and out-of-domain re- trieval performance. Our findings suggest that adapting pretrained encoders to learned sparse retrieval requires not only stronger rep- resentations, but also careful control of the projection scale used to construct sparse lexical vectors. GenAI Usage Disclosure ChatGPT (GPT-5.5) was used to check spelling and improve gram- mar during the writing of this paper. Acknowledgments To Robert, for the bagels and explaining CMYK and color spaces. References [1]Parul Awasthy, Aashka Trivedi, Yulong Li, Meet Doshi, Riyaz Bhat, Vignesh P, Vishwajeet Kumar, Yushu Yang, Bhavani Iyer, Abraham Daniels, Rudra Murthy, Ken Barker, Martin Franz, Madison Lee, Todd Ward, Salim Roukos, David Cox, Luis Lastras, Jaydeep Sen, and Radu Florian. 2025. Granite Embedding R2 Models. arXiv:2508.21085 [cs.CL] https://arxiv.org/abs/2508.21085 [2] Alejandro Fuster Baggetto and Victor Fresno. 2022. Is anisotropy really the cause of BERT embeddings not being semantic?. In Findings of the association for computational linguistics: EMNLP 2022. 4271–4281. [3]Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. arXiv:1611.09268 [cs.CL] https://arxiv.org/abs/1611.09268 [4]Sebastian Bruch, Franco Maria Nardini, Cosimo Rulli, and Rossano Venturini. 2024. Efficient inverted indexes for approximate retrieval over learned sparse representations. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 152–162. [5] Sebastian Bruch, Franco Maria Nardini, Cosimo Rulli, and Rossano Venturini. 2024. Pairing clustered inverted indexes with휅-n graphs for fast approximate retrieval over learned sparse representations. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. 3642–3646. [6]Sebastian Bruch, Franco Maria Nardini, Cosimo Rulli, Rossano Venturini, and Leonardo Venuta. 2025. Investigating the scalability of approximate sparse retrieval algorithms to massive datasets. In European Conference on Information Retrieval. Springer, 437–445. [7]Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the association for computational linguistics: ACL 2024. 2318–2335. [8]Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. 2020. Overview of the TREC 2019 deep learning track. In Proceedings of the 28th Text REtrieval Conference (TREC 2019). NIST. [9]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 4171–4186. [10]Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. 2021. SPLADE v2: Sparse lexical and expansion model for information retrieval. arXiv preprint arXiv:2109.10086 (2021). [11]Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021. Splade: Sparse lexical and expansion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2288–2292. [12]Hiun Kim, Tae Kwan Lee, and Taeryun Won. 2025. The Role of Vocabularies in Learning Sparse Representations for Ranking. arXiv preprint arXiv:2509.16621 (2025). [13]Hiun Kim, Tae Kwan Lee, and Taeryun Won. 2026. The Pre-Training Study of Expanded-SPLADE Models on Web Document Titles. arXiv preprint arXiv:2605.01407 (2026). [14]Carlos Lassance, Hervé Déjean, Thibault Formal, and Stéphane Clinchant. 2024. SPLADE-v3: New baselines for SPLADE. arXiv preprint arXiv:2403.06789 (2024). [15]Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281 (2023). [16]Simon Lupart, Maxime Louis, Thibault Formal, Hervé Déjean, and Stéphane Clinchant. 2026. On the Challenges and Opportunities of Learned Sparse Retrieval for Code. arXiv preprint arXiv:2603.22008 (2026). [17]Joel Mackenzie, Shengyao Zhuang, and Guido Zuccon. 2023. Exploring the Representation Power of SPLADE Models. In Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval. 143–147. [18] Thong Nguyen, Yibin Lei, Jia-Huei Ju, Eugene Yang, and Andrew Yates. 2025. Milco: Learned Sparse Retrieval Across Languages via a Multilingual Connector. arXiv preprint arXiv:2510.00671 (2025). [19]Thong Nguyen, Sean MacAvaney, and Andrew Yates. 2023. A unified framework for learned sparse retrieval. In European Conference on Information Retrieval. Springer, 101–116. [20] Biswajit Paria, Chih-Kuan Yeh, Ian E. H. Yen, Ning Xu, Pradeep Ravikumar, and Barnabás Póczos. 2020. Minimizing FLOPs to Learn Efficient Sparse Representa- tions. arXiv:2004.05665 [cs.LG] https://arxiv.org/abs/2004.05665 [21] Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance frame- work: BM25 and beyond. Vol. 4. Now Publishers Inc. [22] Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663 (2021). [23]Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533 (2022). [24] Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hall- ström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, et al.2025. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. In Proceed- ings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2526–2547. [25]Orion Weller, Kathryn Ricci, Marc Marone, Antoine Chaffin, Dawn Lawrie, and Benjamin Van Durme. 2026. Seq vs Seq: An Open Suite of Paired Encoders and Decoders. arXiv:2507.11412 [cs.CL] https://arxiv.org/abs/2507.11412 [26]Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al.2025. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176 (2025).