Paper deep dive
CoverageBench: Evaluating Information Coverage across Tasks and Domains
Saron Samuel, Andrew Yates, Dawn Lawrie, Ian Soboroff, Trevor Adriaanse, Benjamin Van Durme, Eugene Yang
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 3/23/2026, 12:11:01 PM
Summary
CoverageBench is a unified suite of seven coverage-annotated datasets designed to evaluate information coverage in retrieval systems, particularly for Retrieval-Augmented Generation (RAG). It addresses the limitations of traditional metrics like precision and recall by focusing on whether retrieved documents collectively satisfy an information need through discrete 'nuggets' of information. The benchmark includes datasets adapted from TREC tasks (NeuCLIR, RAG, Fair Ranking, CAsT, RAGTIME) and CRUX, providing topics, relevance labels, and baseline rankings to facilitate research into comprehensive information access.
Entities (5)
Relation Signals (3)
CoverageBench â hostedon â Hugging Face Datasets
confidence 100% ¡ All topics, nuggets, relevance labels, and baseline rankings are released on Hugging Face Datasets
CoverageBench â includesdataset â NeuCLIR 2024
confidence 100% ¡ CoverageBench includes seven datasets. Five are adapted from established TREC tasks: NeuCLIR 2024...
Llama 3.3 70B Instruct â usedtogenerate â Nugget-level qrels
confidence 95% ¡ We derive nugget-level qrels by using Llama-3.3-70B-Instruct as an LLM judge
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We wish to measure the information coverage of an ad hoc retrieval algorithm, that is, how much of the range of available relevant information is covered by the search results. Information coverage is a central aspect for retrieval, especially when the retrieval system is integrated with generative models in a retrieval-augmented generation (RAG) system. The classic metrics for ad hoc retrieval, precision and recall, reward a system as more and more relevant documents are retrieved. However, since relevance in ad hoc test collections is defined for a document without any relation to other documents that might contain the same information, high recall is sufficient but not necessary to ensure coverage. The same is true for other metrics such as rank-biased precision (RBP), normalized discounted cumulative gain (nDCG), and mean average precision (MAP). Test collections developed around the notion of diversity ranking in web search incorporate multiple aspects that support a concept of coverage in the web domain. In this work, we construct a suite of collections for evaluating information coverage from existing collections. This suite offers researchers a unified testbed spanning multiple genres and tasks. All topics, nuggets, relevance labels, and baseline rankings are released on Hugging Face Datasets, along with instructions for accessing the publicly available document collections.
Tags
Links
- Source: https://arxiv.org/abs/2603.20034v1
- Canonical: https://arxiv.org/abs/2603.20034v1
Trouble viewing inline? Open PDF directly â
Full Text
50,441 characters extracted from source content.
Expand or collapse full text
CoverageBench: Evaluating Information Coverage across Tasks and Domains Saron Samuel Johns Hopkins University Baltimore, MD, USA ssamue21@jhu.edu Andrew Yates Johns Hopkins University Baltimore, MD, USA andrew.yates@jhu.edu Dawn Lawrie Johns Hopkins University Baltimore, MD, USA lawrie@jhu.edu Ian Soboroff National Institute of Standards and Technology Gaithersburg, MD, USA ian.soboroff@nist.gov Trevor Adriaanse Johns Hopkins University Baltimore, MD, USA tadriaa1@jhu.edu Benjamin Van Durme Johns Hopkins University Baltimore, MD, USA vandurme@jhu.edu Eugene Yang Johns Hopkins University Baltimore, MD, USA eugene.yang@jhu.edu Abstract We wish to measure the information coverage of an ad hoc retrieval algorithm, that is, how much of the range of available relevant information is covered by the search results. Information coverage is a central aspect for retrieval, especially when the retrieval sys- tem is integrated with generative models in a retrieval-augmented generation (RAG) system. The classic metrics for ad hoc retrieval, precision and recall, reward a system as more and more relevant documents are retrieved. However, since relevance in ad hoc test collections is defined for a document without any relation to other documents that might contain the same information, high recall is sufficient but not necessary to ensure coverage. The same is true for other metrics such as rank-biased precision (RBP), normalized discounted cumulative gain (nDCG), and mean average precision (MAP). Test collections developed around the notion of diversity ranking in web search incorporate multiple aspects that support a concept of coverage in the web domain. In this work, we con- struct a suite of collections for evaluating information coverage from existing collections. This suite offers researchers a unified testbed spanning multiple genres and tasks. All topics, nuggets, relevance labels, and baseline rankings are released on Hugging Face Datasets 1 , along with instructions for accessing the publicly available document collections. CCS Concepts ⢠Information systemsâInformation retrieval diversity; Summarization;⢠Computing methodologiesâNatural lan- guage generation. 1 https://huggingface.co/datasets/hltcoe/coveragebench Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. SIGIR â26, Naarm, Australia Š 2026 ACM. ACM ISBN 978-x-x-x-x/Y/M https://doi.org/10.1145/1122445.1122456 Keywords Retrieval-augmented Generation, Search Result Diversification, Generation, Retrieval, Correlation, Information Coverage ACM Reference Format: Saron Samuel, Andrew Yates, Dawn Lawrie, Ian Soboroff, Trevor Adriaanse, Benjamin Van Durme, and Eugene Yang. 2026. CoverageBench: Evaluating Information Coverage across Tasks and Domains. In SIGIR â26: ACM SIGIR Conference on Research and Development in Information Retrieval, July 20â24, 2026, Naarm, Australia. ACM, New York, NY, USA, 8 pages. https://doi.org/ 10.1145/1122445.1122456 1 Introduction Information Retrieval (IR) evaluation has historically focused on the question, âcan systems find relevant documents?â Standard bench- marks usually provide binary or graded relevance judgments to measure metrics like nDCG, MAP, and RBP [16,27,28]. These relevance-based evaluations have driven decades of progress in retrieval effectiveness, from statistical ranking methods [35] to modern neural retrievers [14, 18]. However, relevance alone is insufficient for many real-world information needs [3,45]. Users often seek comprehensive under- standing of a topic rather than a handful of relevant document [3]. In fact, this concept has been discussed as early as in TREC-6 inter- active track, which was in 1996 [30]. A system might retrieve ten highly relevant documents that all discuss the same aspect of the user query, scoring well on traditional metrics while failing to ad- dress the full breadth of the information need [1,45]. However, for a while, models have been struggling to retrieve more relevant doc- uments. Therefore, improving relevance has been the primary focus in retrieval model developments. However, recent advancements in neural retrieval, such as reasoning-based rerankers [42,44], enable the direct modeling of information coverage. Especially in Retrieval- Augmented Generation (RAG) systems, retrieved content directly informs a generated response [21]. A RAG system that retrieves only a narrow set of information with an abundance of redundant documents, no matter how relevant, will produce incomplete or biased answers [8, 15]. Coverage evaluation addresses this gap by assessing whether a system surfaces diverse, complementary information that collec- tively satisfies an information need. Rather than asking, âis this document relevant?â, coverage evaluation asks, âdoes this set of arXiv:2603.20034v1 [cs.IR] 20 Mar 2026 SIGIR â26, July 20â24, 2026, Naarm, AustraliaSamuel et al. documents contain all the important information?â This requires identifying discrete information units (i.e., nuggets) essential to answering a query and measuring how completely a retrieved set covers them [22, 25, 40, 41]. Although aspectual recall seems intuitively to be an important metric, there are relatively few test collections built to measure it. The TREC-6 interactive track [30] decorated TREC topics with nar- rative descriptions of desired aspects to be covered, and searchers in the task were challenged to cover the topic as well as they could, but this did not result in test collection for ad hoc search research. The TREC web track diversity ranking tasks [4,6,7] explored subtopic coverage in the context of web search. Later work at NT- CIR explored identifying the subtopics themselves [26,36,38,43]. The TREC NeuCLIR Report Generation Pilot Task [19] and TREC RAG Track [33,34,39] have incorporated coverage evaluation, but with limited query sets and narrow domains. Creating new cover- age benchmarks from scratch is expensive, requiring annotation of information nuggets for each topic and nugget-level relevance judgments. The sparsity of collections may in part be because al- gorithms need to be sufficiently advanced to distinguish aspects within documents. The time is now ripe for the development of these algorithms. To support this, we address this challenge by transforming ex- isting collections into coverage evaluation benchmarks to make CoverageBench, a unified suite of coverage evaluation collections. CoverageBench includes seven datasets. Five are adapted from es- tablished TREC tasks: NeuCLIR 2024 [19], RAG 2024 [33,34,39], Fair Ranking 2022 [11], CAsT 2020 [9], and RAGTIME 2025 [20]. We also include two coverage-based evaluation datasets not from TREC, CRUX-MultiNews [17], CRUX-DUC04 [17]. These collections were chosen over others because of the availability of the document collections. To establish baseline performance, we evaluate six re- trieval configurations with BM25 [24] and Qwen3-8B [46], with and without reranking (Rank1-7B [42] and Qwen3-Reranker-8B [46]). Our contributions are the following: â˘we present a methodology for deriving coverage benchmarks from existing ad hoc retrieval collections â˘we release CoverageBench, comprising seven coverage-annotated datasets with publicly available augmentations on Hugging Face Datasets ⢠we provide baseline results and evaluation tools that lower the barrier to coverage-oriented retrieval research. We enable the IR community to explore coverage evaluation at scale, supporting the development of retrieval systems better suited for RAG and other applications where comprehensive information access matters. 2 Background 2.1 Information Coverage Information coverage evaluation addresses the gap of finding rele- vant material and covering the breadth of an information need by decomposing an information need into discrete units of informa- tion, commonly called nuggets, and measuring how completely a retrieved set (or a generated response) accounts for them [40]. The TREC QA track [40] introduced nugget-based evaluation, where each question was associated with a set of atomic facts (nuggets) that a correct answer should contain. Similarly, the pyramid method [29] for summarization evaluation weights content units by how many human reference summaries include them. In the retrieval setting, coverage shifts the unit of evaluation from individual documents to the set of retrieved results consid- ered collectively. This perspective is closely related to search result diversification.íź -nDCG [3] and intent-aware metrics [2] penalize redundancy by discounting documents that cover subtopics already represented in higher-ranked results. Subtopic Recall (StRecall) [45] directly measures the fraction of distinct subtopics covered by the top-k retrieved documents. In a RAG pipeline, the retrieved documents serve as the knowl- edge context from which a language model generates its response. If the retrieval stage fails to surface a nugget, the generation stage has no opportunity to include that information. Coverage failures at retrieval time can propagate directly into incomplete or biased generated answers [8,15]. This makes coverage evaluation essential not only as a retrieval quality measure but as a diagnostic tool for understanding end-to-end RAG system behavior. 2.2 Benchmarks for Information Coverage One of the early efforts to annotate a collection for information cov- erage was the 2009-2011 TREC web tracks [4,6,7], which featured a diversity ranking task. Diversity ranking is a term from web search where the search engine attempts to cover as many aspects of an underspecified query as possible, high up in the result page. As the web track organizers write, âThe goal of this diversity task is to return a ranked list of pages that together provide complete coverage for a query, while avoiding excessive redundancy in the result list.â [7] The topics were derived from the logs of a commercial search engine, using a tool designed to cluster queries with similar click patterns. Topics were divided into two classes: faceted and ambiguous. A faceted topic like âadobe indian housesâ had subtopics about adobe houses, their origins, how they are built, and how native Ameri- cans built them. Each subtopic of a faceted query represented a related information need that might have been meant by the query. An ambiguous query like âeast ridge high schoolâ had subtopics with orthogonal interpretations: there are East Ridge High Schools in many places in the United States, and the user seemed likely to want only one of them (since âschoolâ is singular). Queries of acronyms were also typical, ambiguous queries. Subtopics were identified when the topics were developed, without later expansion. The TREC web track pioneered the intent-aware ERR (IA-ERR) andíź-nDCG metrics specifically designed to take subtopics into account [5]. The INTENT, INTENT2, IMINE, IMINE-2 tasks at NTCIR added an explicit subtopic discovery task to diversity ranking [26,36, 38,43]. In response to a given ambiguous query, systems in the subtopic discovery task (âintent miningâ or âquery understandingâ in the language of the task organizers) would return a ranked list of subtopics. A subtopic was a query string that disambiguates the search intent of the original query. Assessors manually clustered submitted subtopics into common intents, then voted to assign probabilities to each intent. This process produced a set of weighted subtopics derived from pooled submissions, rather than a fixed set as used in TREC. The main metric, used for both document ranking CoverageBench:Evaluating Information Coverage across Tasks and DomainsSIGIR â26, July 20â24, 2026, Naarm, Australia and subtopic mining, was DâŻ-nDCG, a weighted sum of intent recall (I-rec) and a diversified nDCG variant that scales gain values using the subtopic probabilities (D-nDCG) [37]. The NTCIR tasks made at least three significant strides beyond the TREC web diversity tasks: a method for estimating the probability of a subtopic, metrics that incorporated those probabilities, and an unbounded subtopic set. The common thread between TREC web and NTCIR IMINE is the focus on covering the possible meanings behind a short web query. While this is certainly one form of coverage, the resource we have built is more strongly focused on deep information needs with multiple aspects. Recently, TREC Tracks such as NeuCLIR [19], RAG [34], and RAGTIME [20] have begun revisiting the idea of information cov- erage as a means of evaluating RAG output, drawing on work from the summarization community. The Document Understand- ing Conference (DUC) summarization tasks were the first shared evaluation tasks for text summarization. Systems were provided with a set of newswire documents and produced a summary with a bounded length. The evaluation concentrated on linguistic qual- ity and content coverage [31]. Coverage was measured first by alignment with a manually-written model summary, and then later using the pyramid method [29]. In the pyramid method, assessors identified âsummary content unitsâ (SCUs) that in later years were referred to as ânuggetsâ. A summary content unit was an atomic piece of information that an ideal summary would be expected to contain. Multiple assessors created nuggets, and nuggets received a weight corresponding to the number of assessors identifying that nugget, producing the pyramid. The pyramid method was later used in question answering evaluations as well [23], and was the direct antecedent for nugget-based evaluations used in the TREC NeuCLIR track report pilot, RAG, and RAGTIME tracks. When one has deep information needs, it is intuitive to focus on factual aspects of an information need; however, other types of aspects may be of interest. The 2022 TREC Fair Ranking track [11] evaluates IR systems on fairness with respect to protected or sensitive attributes. Systems retrieve and rank documents while ensuring equitable exposure and representation across different groups, such as subject, geography, or gender. Another example is the 2020 TREC Conversational Assistance Track (CAsT) [9], which evaluates conversational IR systems that can handle multi-turn, context-dependent queries. In this track, users engage in a sequence of related questions, and systems must interpret each query in the context of the conversation and retrieve relevant passages from large document collections. Each related question can be thought of as an aspect of information coverage. In this work, we incorporate different types of coverage defini- tions to form CoverageBench. In the next section, we discuss our construction process in detail. 3 CoverageBench CoverageBench consists of the resources needed to evaluate infor- mation coverage and other artifacts to simplify experimentation (e.g., evaluation code and pre-built indexes). It contains the follow- ing components: ⢠Topics. A set of topics (queries) 2 was drawn from the original 2 We use topic and query interchangeably in this paper. task. For datasets where the original topics are complex or multi- faceted (such as conversational turns in CAsT), or lacked context (list of keywords in Fair Ranking) we apply modest modifications (see Section 3.1). The benchmark contains 334 topics. ⢠Relevance Labels. We include relevance judgments (qrels) for each collection. Some datasets required additional LLM judgments. These labels enable relevance and coverage evaluation. â˘Document Collection. All documents from the datasets used in this benchmark are publicly available. We provide instructions on HuggingFace on how to access all document collections. ⢠Nuggets and Subtopics. The core augmentation in CoverageBench is a set of nuggets (or subtopics) for each topic, representing the discrete information units that a comprehensive response should cover. For datasets that already include nugget annotations (Neu- CLIR, RAG), we adopt or adapt the existing nuggets. For datasets without them (Fair Ranking, CAsT), we derive nuggets through the augmentation process described in Section 3.1. Each nugget is associated with the set of documents in the collection that contain the corresponding information, enabling evaluation of both retrieval and generation coverage (which nuggets appear in the retrieved set and generated responses, respectively) â˘Baselines. We release baseline ranked lists for each dataset at two stages of the retrieval pipeline. First, initial retrieval of the top-100 results from BM25 [24] and Qwen3-8B [46]. Then, results from reranking the top-50 of the initial retrieval with Rank1-7B [42] and Qwen3-Reranker-8B [46]. We release 6 baselines for each dataset, 2 initial retrieval with 4 reranked runs. 3.1 Source Datasets and Augmentation We describe each of the datasets in CoverageBench. While some datasets contained all the necessary aspects, others required dataset augmentation to provide each component for each dataset. A sum- mary of the dataset statistics is presented in Table 1. 3.1.1 NeuCLIR. The 2024 TREC NeuCLIR Report Generation pi- lot task [19] evaluates systemsâ ability to generate English reports grounded in multilingual evidence. Systems receive an English re- port request and must produce a coherent report whose factual claims are supported by citations to relevant news documents writ- ten in Chinese, Persian, or Russian. For completeness, we include the 19 topics that were judged in all three languages. While this dataset is already organized in a manner compatible with coverage evaluation, the number of topics is too small on its own for robust benchmarking. 3.1.2 RAG. In the 2024 TREC RAG task [33,34,39] participants receive non-factoid topic descriptions and the MS MARCO v2.1 segmented collections [33], and must retrieve relevant segments and generate structured answers with explicit citations to supporting evidence. The original dataset provides nugget annotations for each topic, but the nugget-to-document alignment was never fully recorded or assessed during the original evaluation. We derive nugget-level qrels by using Llama-3.3-70B-Instruct [12] as an LLM judge to assess each relevant document against each nugget in the topic. For each topic, nugget, and document triple, the judge determines whether the document contains information corresponding to the nugget. SIGIR â26, July 20â24, 2026, Naarm, AustraliaSamuel et al. Table 1: Dataset Statistics. The average number of words in each document was tokenized by white space. For NeuCLIR and RAGTIME, we report the counts on machine-translated documents, which are all in English for easy comparison. Documents / Passages QueriesNuggets per Query DatasetCount Avg Words Count Avg Med Min Max CAsT 202038,429,85259.6256.16312 Fair Ranking 20226,475,537479.25029.726362 NeuCLIR 2024 Pilot10,038,768348.01914.9151025 RAG 2024113,520,750166.75613.914620 RAGTIME 20254,000,380404.03415.616.5820 CRUX-MultiNews 565,01588.9 10014.2141215 CRUX-DUC04507.88110 When treating a document as relevant if it contains a nugget, our LLM judge achieves a precision of 69% and a recall of 90% across all 56 topics with respect to the original RAG24 relevance judgments. We consider this level of agreement reasonable for the purposes of coverage evaluation. 3 3.1.3 RAGTIME. The 2025 TREC RAGTIME Track [20] hosts a long-form cross-language report generation task, where systems are expected to retrieve information from a multilingual collection consisting of documents in Chinese, Russian, Arabic, and English based on a rich description of information need, i.e., the problem statement and user background, and synthesize a report of several paragraphs long as the response. The 2025 RAGTIME dataset comes with human-curated nuggets in the form of question and answer pairs, which were extracted from pooled documents created on the track submissions, identical to the 2024 NeuCLIR Report Generation Pilot. However, unlike NeuCLIR, there is no explicit annotation on which documents support or contain those nuggets, preventing us from directly applying this dataset for evaluating retrieval coverage. Such information can only be inferred from the sentence support (i.e., whether the report sentence is supported by its citation) and the nugget alignment (i.e., which nugget the report sentence correctly covers) annotations of the evaluated submissions. To fill the gap, we use Llama-3.3-70B-Instruct [12] with a prompt taking in the problem statement, the nugget (both question and answer), and the document content as the input and ask the model Does the document contain the specified answer of the question? (Yes/No) with conditional decoding on the tokenYesandNoto get an explicit binary answer. To ensure completeness, we pooled the top 20 documents from each retrieval run submission in RAGTIME, as well as the top 2 citations from each sentence in the report gen- eration submission, resulting in 22,344 documents in all topic pools. We judged all documents in each topic pool against all nuggets, resulting in 686,576 judgments, of which 38,108 came back positive. We validate these predictions against the incomplete document- to-nugget annotation extracted from the official annotation on the submissions. Specifically, we take all sentences in the report generation submission that are annotated as fully supported by the citations, with the nuggets that they cover, and link the cited 3 Since RAG24 did not publish an overview paper, we directly communicated with TREC organizers to verify our approach. We will release the prompt and judged labels upon publication. documents with the nuggets. Since a document that supports a sentence that covers a specific nugget must also contain that nugget, these inferred support document sets are not complete. However, investigating recall is still meaningful here. Our LLM-generated labels achieve 72% recall on this inferred support set. We further drew a sample of 60 document-nugget pairs from the false-negative set (i.e., TREC assessors said support but LLM said no) to inspect the quality. Of which, we found only 17 pairs (28%) that the document really supports the nugget, indicating that the true recall should be higher. We use these LLM-generated labels to create the nugget- based qrels for evaluation. 3.1.4 Fair Ranking. The 2022 TREC Fair Ranking track [11] eval- uates IR systems on fairness based on the demographic attributes annotated in the collection While its original structure was not well-suited for measuring coverage metrics such asíź-nDCG, we adapt the TREC Fair Ranking Track by making use of the annotated corpus where each document has demographic attributes (such as subject geography, popularity). In the original task, systems were evaluated on whether their rankings distribute attention across these attribute groups according to a target distribution, measured by AWRF. The original queries are short keyword lists intended for ad hoc retrieval. We adapt this dataset for coverage evaluation through three transformations: query rewriting, nugget derivation, and relevance label construction. â˘Queries. We rewrite each keyword list query into a natural lan- guage information-seeking query that explicitly targets coverage across the demographic facets annotated in the dataset. For ex- ample, the keyword query for a topic on architecture is rewritten as: Overview of architecture and architectural styles from diverse world regions across different time periods including both famous and lesser-known architects. The system must now retrieve docu- ments that collectively span the facets of subject geography and popularity embedded in the query. â˘Nuggets. We derive nuggets directly from the demographic at- tribute annotations in the original dataset. Each unique attribute value becomes a nugget that the retrieved set should cover. For the architecture example above, the nuggets include geographic re- gions (for example, South America, South-eastern Asia, Southern Africa) drawn from the subject geography facet, and popularity levels (High, Low, Medium-High, Medium-Low) drawn from the popularity facet. CoverageBench:Evaluating Information Coverage across Tasks and DomainsSIGIR â26, July 20â24, 2026, Naarm, Australia Figure 1: Cumulative distribution of the number of rele- vant documents per nugget across CoverageBench datasets. Curves further to the left indicate harder datasets where nuggets are covered by fewer documents. â˘Relevance Label. The original dataset provides relevance judg- ments alongside document-level demographic annotations. We combine these to construct nugget-level qrels. For each rewritten query, we identify the set of target nuggets (the facet values the query seeks to cover) and retain only the relevant documents whose demographic annotations match at least one target nugget. Documents that are relevant to the original topic but do not cover any target nugget are excluded, ensuring that the qrels measure coverage of the intended facets. 3.1.5 CAsT. The 2020 TREC Conversational Assistance Track (CAsT) [9] was made to evaluate conversational IR systems that can handle multi-turn, context-dependent queries. CAsT emphasizes context modeling, requiring systems to resolve references and retrieve in- formation that satisfies the evolving information need. Each topic consists of a sequence of related turns, with each turn representing a follow-up question within a broader information need. The origi- nal dataset was designed for passage-level conversational retrieval, not coverage evaluation. There are no nugget annotations, and the relevance judgments are sparse, covering only a small number of as- sessed documents per turn. We adapt CAsT for coverage evaluation by the following steps: â˘Turn Curation. Not all conversational turns contribute meaning- fully to the overarching information need of a topic. Some turns are redundant, revisiting aspects already addressed by earlier turns, while others drift into tangential subtopics. We manually curate the turn sequences by removing redundant and irrelevant turns, retaining only those that represent distinct facets of the topicâs information need. ⢠Nuggets. We treat each retained turn as a subtopic of the overar- ching topic query. The turnâs manually rewritten utterance [9] serves as the subtopic description, and its canonical result docu- ment provides a reference answer. â˘Relevance Augmentation. The original CAsT relevance judgments assess only a small pool of documents per turn, which is insuffi- cient for reliable coverage evaluation. We augment the relevance judgments using an LLM judge: for each turn, we use a Llama-3.3- 70B-Instruct [12] to assess whether candidate documents from an expanded pool contain information relevant to that turnâs subtopic. Documents judged as relevant are added to the nugget- level qrels. Of the 153 canonical relevant documents from the CAsT 2020 ground truth, 86 (56.2%) were recovered in the ex- panded judgment pool. Examining the missed documents by hand, we found that almost all were not relevant, so the true recall should be higher. 3.1.6 CRUX-MultiNews and CRUX-DUC04. Controlled Retrieval- augmented Context Evaluation (CRUX) [17] is an evaluation frame- work designed to assess retrieval contexts in long-form RAG sce- narios, which does not require augmentation for CoverageBench. Rather than evaluating retrieval purely through relevance-based ranking metrics, CRUX measures how completely a retrieval con- text covers the information needed for long-form generation, using question-based evaluation. The framework is built on multi-document summarization datasets, Multi-News [13] and DUC-2004 [32], where human-written sum- maries serve as oracle long-form results that define the scope of relevant retrieval context. From each summary, CRUX generates an open-ended query, a diverse set of knowledge-intensive sub- questions, and decontextualized passage-level chunks from the source documents. An LLM judges whether each passage answers each sub-question, producing a matrix of graded answerability scores that supports fine-grained coverage and density metrics. Be- cause these datasets already provide mappings between nuggets and supporting passages, they are compatible with coverage evaluation and require no further adaptation for inclusion in our benchmark. 3.2 Dataset Statistics Table 1 summarizes the key statistics of the datasets in Cover- ageBench. The benchmark contains 334 topics in total. The doc- ument collection spans from 565k passages (CRUX-MultiNews and CRUX-DUC04) to over 113M segments (RAG). The datasets vary in the number of queries, from 19 (NeuCLIR) to 100 (CRUX- MultiNews). The number of nuggets per query also varies within and across datasets. Fair Ranking has the highest average nugget count (29.7/query). and the widest range (3 to 62). CAsT has the fewest nuggets per query on average (6.1). This is expected as the nuggets are based on a conversational structure, and each nugget is equivalent to a turn in the conversation. Document length also shapes part of the coverage challenge. A longer document is likely to cover more nuggets. CAsT passages were the shortest, averaging 59.6 words per passage. Fair Ranking and RAGTIME documents are longer, 479 and 404 words on average, respectively. Figure 1 shows the cumulative distribution of the number of rel- evant documents per nugget, normalized by the collection size, for all of the benchmark datasets. Curves to the left indicate datasets where each nugget is supported by a smaller fraction of the collec- tion. Fair Ranking has the most queries per nugget, so individual nuggets are well-attested in a collection. Here, the challenge is in covering all of the nuggets in a ranked list since it has the highest average of nuggets per query. 4 Baseline Experiments To characterize the coverage properties of each dataset and provide SIGIR â26, July 20â24, 2026, Naarm, AustraliaSamuel et al. Table 2: Baseline retrieval and reranking relevance effectiveness measured by nDCG@20. The average column reports the macro-average over all seven datasets. InitialRerankNeuCLIRRAGFair RankCAsTCRUX-MultiNewsCRUX-DUC04RAGTIMEAvg BM25â0.3280.5990.1430.4750.4290.4480.5960.431 Qwen3-8Bâ0.8190.8540.0940.7000.6100.7010.7740.650 BM25Rank10.5200.758 0.1610.5840.5330.6100.6240.541 Qwen3-8BRank10.821 0.9150.124 0.7040.6160.7540.8000.676 BM25Qwen3-R0.5810.7770.1030.4200.4940.5240.7300.518 Qwen3-8BQwen3-R 0.8600.8980.0520.6750.6200.7130.6090.632 Table 3: Baseline retrieval and reranking results on coverage:íź-nDCG@20 and Subtopic Recall (StRecall@20) íź -nDCG@20StRecall@20 InitialRerankNeuRAGFRCAsT CX-M CX-D RAGT AvgNeuRAGFRCAsT CX-M CX-D RAGT Avg BM25â0.349 0.450 0.109 0.357 0.469 0.476 0.486 0.385 0.545 0.708 0.171 0.577 0.634 0.659 0.664 0.565 Qwen3-8B â0.627 0.683 0.074 0.437 0.648 0.652 0.531 0.522 0.836 0.899 0.117 0.597 0.813 0.814 0.707 0.683 BM25Rank10.538 0.658 0.122 0.429 0.583 0.621 0.527 0.497 0.664 0.818 0.217 0.639 0.710 0.732 0.690 0.639 Qwen3-8B Rank10.613 0.742 0.090 0.440 0.634 0.690 0.590 0.543 0.786 0.935 0.133 0.619 0.831 0.839 0.773 0.702 BM25Qwen3-R 0.583 0.633 0.065 0.321 0.551 0.554 0.518 0.461 0.684 0.819 0.143 0.576 0.689 0.696 0.715 0.617 Qwen3-8B Qwen3-R 0.691 0.705 0.036 0.407 0.652 0.663 0.490 0.521 0.839 0.903 0.060 0.600 0.817 0.826 0.660 0.672 Figure 2: nDCG@20 vs. Subtopic Recall@20 reference points for future work, we evaluate a set of retrieval and reranking configurations across all seven CoverageBench datasets. 4.1 Setup We evaluate two initial retrieval models representing sparse and dense retrieval paradigms. BM25 [24] serves as the sparse base- line, using default parameters. Qwen3-8B [46] serves as the dense baseline, encoding queries and documents into dense vectors for nearest-neighbor retrieval with FAISS [10]. We apply two rerankers to each initial retrieval run to rerank the top 50: Rank1-7B [42] and Qwen3-Reranker-8B [46]. For NeuCLIR and RAGTIME, we rerank with the machine-translated English documents since the datasets are multilingual. Combined with the two initial runs, we obtain a total of six retrieval configurations per dataset. All results are evaluated using nDCG (for relevance),íź-nDCG, and Subtopic Recall (StRecall) with a rank cutoff of 20. 4.2 Results In Tables 2 and 3, across most datasets, configurations with Qwen3- 8B as the initial rank performed better than BM25 in both relevance and coverage, which aligns with findings in other works in dense retrieval. The exception is with Fair Ranking, where BM25 performs better across all three metrics. This is likely because the rewritten queries contain explicit facet keywords, which would favor lexical matches. The original retrieval task in Fair Ranking is also more syntactic than semantic, which favors surface form matching. On NeuCLIR, Rank1 applied to Qwen3-8B matches the initial run on nDCG (0.821 vs. 0.819) but drops StRecall from 0.836 to 0.786, indicating that the reranking starts to exploit particular aspects of relevance for optimizing relevance but misses other aspects in the top 20 documents. The highest scoring system on relevance does not always achieve the highest StRecall. On CRUX-MultiNews, Qwen3-Reranker on Qwen3-8B leads on nDCG@20 (0.620) while Rank1 on Qwen3- 8B leads on StRecall@20 (0.831). This demonstrates how Cover- ageBench provides a different perspective than evaluating only on relevance-based metrics. CAsT presents a coverage ceiling that no configuration breaks through. As Figure 2 shows, all six config- urations cluster around a StRecall of approximately 0.55, despite spanning nDCG scores from 0.323 to 0.651. 5 Conclusion We introduced CoverageBench, a benchmark for evaluating infor- mation coverage in retrieval systems. Our work demonstrates that coverage evaluation can be derived from existing retrieval collec- tions without the cost of creating entirely new test collections from scratch. Baseline experiments across six retrieval configurations reveal variation in coverage performance both across datasets and methods. These results demonstrate the need for continued research in coverage-aware retrieval, particularly as RAG systems increas- ingly depend on comprehensive information gathering rather than simple relevance matching. All topics, nuggets, relevance labels, CoverageBench:Evaluating Information Coverage across Tasks and DomainsSIGIR â26, July 20â24, 2026, Naarm, Australia and baseline rankings are publicly available on HuggingFace, en- abling the community to advance coverage evaluation research and develop retrieval systems better suited for applications where comprehensive information access matters. Disclaimer Certain products are named in this paper in order to fully specify the experimental procedure adequately. Such mentions should not be taken as endorsement or recommendation of any company, product, or service by NIST, nor are they intended to imply that the products identified are necessarily the best available for this purpose. References [1] Jaime Carbonell and Jade Goldstein. 1998. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceed- ings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Melbourne, Australia) (SIGIR â98). As- sociation for Computing Machinery, New York, NY, USA, 335â336.https: //doi.org/10.1145/290941.291025 [2]Olivier Chapelle, Shihao Ji, Ciya Liao, Emre Velipasaoglu, Larry Lai, and Su- Lin Wu. 2011. Intent-based diversification of web search results: metrics and algorithms. Inf. Retr. 14, 6 (Dec. 2011), 572â592. [3] Charles L.A. Clarke, Maheedhar Kolla, Gordon V. Cormack, Olga Vechtomova, Azin Ashkan, Stefan BĂźttcher, and Ian MacKinnon. 2008. Novelty and diversity in information retrieval evaluation. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Singapore, Singapore) (SIGIR â08). Association for Computing Machinery, New York, NY, USA, 659â666. https://doi.org/10.1145/1390334.1390446 [4]Charles L. A. Clarke, Nick Craswell, and Ian Soboroff. 2009. Overview of the TREC 2009 Web Track. In Proceedings of The Eighteenth Text REtrieval Conference, TREC 2009, Gaithersburg, Maryland, USA, November 17-20, 2009 (NIST Special Publication, Vol. 500-278), Ellen M. Voorhees and Lori P. Buckland (Eds.). National Institute of Standards and Technology (NIST). http://trec.nist.gov/pubs/trec18/ papers/WEB09.OVERVIEW.pdf [5]Charles L. A. Clarke, Nick Craswell, Ian Soboroff, and Azin Ashkan. 2011. A comparative analysis of cascade measures for novelty and diversity. In Proceedings of the Forth International Conference on Web Search and Web Data Mining, WSDM 2011, Hong Kong, China, February 9-12, 2011, Irwin King, Wolfgang Nejdl, and Hang Li (Eds.). ACM, 75â84. https://doi.org/10.1145/1935826.1935847 [6]Charles L. A. Clarke, Nick Craswell, Ian Soboroff, and Gordon V. Cormack. 2010. Overview of the TREC 2010 Web Track. In Proceedings of The Nineteenth Text REtrieval Conference, TREC 2010, Gaithersburg, Maryland, USA, November 16- 19, 2010 (NIST Special Publication, Vol. 500-294), Ellen M. Voorhees and Lori P. Buckland (Eds.). National Institute of Standards and Technology (NIST). https: //trec.nist.gov/pubs/trec19/papers/WEB.OVERVIEW.pdf [7] Charles L. A. Clarke, Nick Craswell, Ian Soboroff, and Ellen M. Voorhees. 2011. Overview of the TREC 2011 Web Track. In Proceedings of The Twentieth Text REtrieval Conference, TREC 2011, Gaithersburg, Maryland, USA, November 15- 18, 2011 (NIST Special Publication, Vol. 500-296), Ellen M. Voorhees and Lori P. Buckland (Eds.). National Institute of Standards and Technology (NIST). http: //trec.nist.gov/pubs/trec20/papers/WEB.OVERVIEW.pdf [8]Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. The Power of Noise: Redefining Retrieval for RAG Systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2024). ACM, 719â729. https://doi.org/10.1145/3626772.3657834 [9]Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. 2021. CAsT 2020: The Conversa- tional Assistance Track Overview. In The Twenty-Ninth Text REtrieval Conference Proceedings (TREC 2020) (NIST Special Publication, Vol. 1266). National Institute of Standards and Technology (NIST). https://trec.nist.gov/pubs/trec29/papers/ OVERVIEW.C.pdf [10]Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel MazarĂŠ, Maria Lomeli, Lucas Hosseini, and HervĂŠ JĂŠgou. 2024. The Faiss library. (2024). arXiv:2401.08281 [cs.LG] [11] Michael D. Ekstrand, Graham McDonald, Amifa Raj, and Isaac Johnson. 2023. Overview of the TREC 2022 Fair Ranking Track.arXiv:2302.05558 [cs.IR] https://arxiv.org/abs/2302.05558 [12] Aaron Grattafiori et. al. 2024.The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783 [13]Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi- News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Anna Korhonen, David Traum, and LluĂs MĂ rquez (Eds.). Association for Computational Linguistics, Florence, Italy, 1074â1084. https://doi.org/10.18653/v1/P19-1102 [14]Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and StĂŠphane Clinchant. 2021. SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval. arXiv:2109.10086 [cs.IR] https://arxiv.org/abs/2109.10086 [15]Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:2312.10997 [cs.CL] https://arxiv.org/abs/2312.10997 [16] Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst. 20, 4 (Oct. 2002), 422â446. https://doi. org/10.1145/582415.582418 [17] Jia-Huei Ju, Suzan Verberne, Maarten de Rijke, and Andrew Yates. 2026. Controlled Retrieval-augmented Context Evaluation for Long-form RAG. arXiv:2506.20051 [cs.IR] https://arxiv.org/abs/2506.20051 [18]Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. arXiv:2004.12832 [cs.IR] https://arxiv.org/abs/2004.12832 [19]Dawn Lawrie, Sean MacAvaney, James Mayfield, Paul McNamee, Douglas W. Oard, Luca Soldaini, and Eugene Yang. 2025. Overview of the TREC 2024 NeuCLIR Track. arXiv:2509.14355 [cs.IR] https://arxiv.org/abs/2509.14355 [20] Dawn Lawrie, Sean MacAvaney, James Mayfield, Luca Soldaini, Eugene Yang, and Andrew Yates. 2026.Overview of the TREC 2025 RAGTIME Track. arXiv:2602.10024 [cs.IR] https://arxiv.org/abs/2602.10024 [21]Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich KĂźttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401 [cs.CL] https://arxiv.org/abs/ 2005.11401 [22] Jimmy Lin and Dina Demner-Fushman. 2005. Evaluating Summaries and Answers: Two Sides of the Same Coin?. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Jade Goldstein, Alon Lavie, Chin-Yew Lin, and Clare Voss (Eds.). Association for Computational Linguistics, Ann Arbor, Michigan, 41â48. https://aclanthology. org/W05-0906/ [23]Jimmy Lin and Dina Demner-Fushman. 2006. Will Pyramids Built of Nuggets Topple Over?. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, Robert C. Moore, Jeff Bilmes, Jennifer Chu-Carroll, and Mark Sanderson (Eds.). Association for Computational Linguistics, New York City, USA, 383â390. https://aclanthology.org/N06-1049/ [24] Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: A Python Toolkit for Reproducible In- formation Retrieval Research with Sparse and Dense Representations. In Pro- ceedings of the 44th International ACM SIGIR Conference on Research and De- velopment in Information Retrieval (Virtual Event, Canada) (SIGIR â21). Asso- ciation for Computing Machinery, New York, NY, USA, 2356â2362.https: //doi.org/10.1145/3404835.3463238 [25]Jimmy Lin and Pengyi Zhang. 2007. Deconstructing nuggets: the stability and reliability of complex question answering evaluation. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Amsterdam, The Netherlands) (SIGIR â07). Association for Computing Machinery, New York, NY, USA, 327â334. https://doi.org/10.1145/ 1277741.1277799 [26] Yiqun Liu, Ruihua Song, Min Zhang, Zhicheng Dou, Takehiro Yamamoto, Makoto P. Kato, Hiroaki Ohshima, and Ke Zhou. 2014.Overview of the NTCIR-11 IMine Task. In Proceedings of the 11th NTCIR Con- ference on Evaluation of Information Access Technologies, NTCIR-11, Na- tional Center of Sciences, Tokyo, Japan, December 9-12, 2014, Noriko Kando, Hideo Joho, and Kazuaki Kishida (Eds.). National Institute of Informatics (NII). http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings11/pdf/NTCIR/ OVERVIEW/01-NTCIR11-OV-IMINE-LiuY.pdf [27]Christopher D. Manning, Prabhakar Raghavan, and Hinrich SchĂźtze. 2008. Intro- duction to Information Retrieval. Cambridge University Press. [28]Alistair Moffat and Justin Zobel. 2008. Rank-biased precision for measurement of retrieval effectiveness. ACM Trans. Inf. Syst. 27, 1, Article 2 (Dec. 2008), 27 pages. https://doi.org/10.1145/1416950.1416952 [29]Ani Nenkova and Rebecca Passonneau. 2004. Evaluating Content Selection in Summarization: The Pyramid Method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Compu- tational Linguistics: HLT-NAACL 2004. Association for Computational Linguistics, Boston, Massachusetts, USA, 145â152. https://aclanthology.org/N04-1019/ [30]Paul Over. 1997. TREC-6 Interactive Report. In Proceedings of The Sixth Text REtrieval Conference, TREC 1997, Gaithersburg, Maryland, USA, November 19-21, 1997 (NIST Special Publication, Vol. 500-240), Ellen M. Voorhees and Donna K. Harman (Eds.). National Institute of Standards and Technology (NIST), 73â81. http://trec.nist.gov/pubs/trec6/papers/t6irep.ps [31]Paul Over, Hoa Dang, and Donna Harman. 2007. DUC in context. Information Processing & Management 43, 6 (2007), 1506â1520. https://doi.org/10.1016/j.ipm. SIGIR â26, July 20â24, 2026, Naarm, AustraliaSamuel et al. 2007.01.019 [32]Paul Over and James Yen. 2004. An Introduction to DUC-2004: Intrinsic Evaluation of Generic News Text Summarization Systems. In Proceedings of the HLT/NAACL 2004 Document Understanding Workshop (DUC-2004). http://w-nlpir.nist.gov/ projects/duc/pubs/2004slides/duc2004intro.pdf [33]Ronak Pradeep, Nandan Thakur, Sahel Sharifymoghaddam, Eric Zhang, Ryan Nguyen, Daniel Campos, Nick Craswell, and Jimmy Lin. 2024. RagnarĂśk: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track. arXiv:2406.16828 [cs.IR] https://arxiv.org/abs/2406.16828 [34]Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, and Jimmy Lin. 2024. Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework. arXiv:2411.09607 [cs.IR] https://arxiv.org/abs/2411.09607 [35]Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Frame- work: BM25 and Beyond. Found. Trends Inf. Retr. 3, 4 (April 2009), 333â389. https://doi.org/10.1561/1500000019 [36]Tetsuya Sakai, Zhicheng Dou, Takehiro Yamamoto, Yiqun Liu, Min Zhang, Makoto P. Kato, Ruihua Song, and Mayu Iwata. 2013. Summary of the NTCIR-10 INTENT-2 task: subtopic mining and search result diversification. In The 36th International ACM SIGIR conference on research and development in Information Retrieval, SIGIR â13, Dublin, Ireland - July 28 - August 01, 2013, Gareth J. F. Jones, Paraic Sheridan, Diane Kelly, Maarten de Rijke, and Tetsuya Sakai (Eds.). ACM, 761â764. https://doi.org/10.1145/2484028.2484104 [37]Tetsuya Sakai and Ruihua Song. 2011. Evaluating diversified search results using per-intent graded relevance. In Proceedings of SIGIR 2011 (Beijing, China) (SIGIR â11). Association for Computing Machinery, New York, NY, USA, 1043â1052. https://doi.org/10.1145/2009916.2010055 [38] Ruihua Song, Min Zhang, Tetsuya Sakai, Makoto P. Kato, Yiqun Liu, Miho Sugi- moto, Qinglei Wang, and Naoki Orii. 2011. Overview of the NTCIR-9 INTENT Task. In Proceedings of the 9th NTCIR Workshop Meeting on Evaluation of In- formation Access Technologies: Information Retrieval, Question Answering and Cross-Lingual Information Access, NTCIR-9, National Center of Sciences, Tokyo, Japan, December 6-9, 2011, Noriko Kando, Daisuke Ishikawa, and Miho Sugimoto (Eds.). National Institute of Informatics (NII).http://research.nii.ac.jp/ntcir/ workshop/OnlineProceedings9/NTCIR/01-NTCIR9-OV-INTENT-SongR.pdf [39]Nandan Thakur, Ronak Pradeep, Shivani Upadhyay, Daniel Campos, Nick Craswell, and Jimmy Lin. 2025. Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges.arXiv:2504.15205 [cs.CL] https://arxiv.org/abs/2504.15205 [40] Ellen M. Voorhees and Dawn M. Tice. 2000. The TREC-8 Question Answer- ing Track. In Proceedings of the Second International Conference on Language Resources and Evaluation (LRECâ00), M. Gavrilidou, G. Carayannis, S. Markan- tonatou, S. Piperidis, and G. Stainhauer (Eds.). European Language Resources Association (ELRA), Athens, Greece. https://aclanthology.org/L00-1018/ [41] William Walden, Marc Mason, Orion Weller, Laura Dietz, John Conroy, Neil Molino, Hannah Recknor, Bryan Li, Gabrielle Kaili-May Liu, Yu Hou, Dawn Lawrie, James Mayfield, and Eugene Yang. 2025. Auto-ARGUE: LLM-Based Report Generation Evaluation. arXiv:2509.26184 [cs.IR] https://arxiv.org/abs/2509.26184 [42]Orion Weller, Kathryn Ricci, Eugene Yang, Andrew Yates, Dawn Lawrie, and Ben- jamin Van Durme. 2025. Rank1: Test-Time Compute for Reranking in Information Retrieval. arXiv:2502.18418 [cs.IR] https://arxiv.org/abs/2502.18418 [43]Takehiro Yamamoto, Yiqun Liu, Min Zhang, Zhicheng Dou, Ke Zhou, Ilya Markov, Makoto P. Kato, Hiroaki Ohshima, and Sumio Fujita. 2016. Overview of the NTCIR-12 IMine-2 Task. In Proceedings of the 12th NT- CIR Conference on Evaluation of Information Access Technologies, National Center of Sciences, Tokyo, Japan, June 7-10, 2016, Noriko Kando, Tet- suya Sakai, and Mark Sanderson (Eds.). National Institute of Informatics (NII). http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings12/pdf/ntcir/ OVERVIEW/01-NTCIR12-OV-IMINE-YamamotoT.pdf [44]Eugene Yang, Andrew Yates, Kathryn Ricci, Orion Weller, Vivek Chari, Ben- jamin Van Durme, and Dawn Lawrie. 2025. Rank-K: Test-Time Reasoning for Listwise Reranking. arXiv:2505.14432 [cs.IR] https://arxiv.org/abs/2505.14432 [45]Cheng Xiang Zhai, William W. Cohen, and John Lafferty. 2003. Beyond in- dependent relevance: methods and evaluation metrics for subtopic retrieval. In Proceedings of the 26th Annual International ACM SIGIR Conference on Re- search and Development in Informaion Retrieval (Toronto, Canada) (SIGIR â03). Association for Computing Machinery, New York, NY, USA, 10â17.https: //doi.org/10.1145/860435.860440 [46]Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025. Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models. arXiv:2506.05176 [cs.CL] https://arxiv.org/abs/2506.05176