Paper deep dive
Beyond Relevance: Utility-Centric Retrieval in the LLM Era
Hengran Zhang, Minghao Tang, Keping Bi, Jiafeng Guo
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 4/14/2026, 1:47:47 AM
Summary
This paper presents a tutorial on the evolution of information retrieval (IR) objectives in the era of Retrieval-Augmented Generation (RAG). It argues that the field is shifting from traditional topical relevance to 'LLM-centric utility,' where retrieval effectiveness is measured by its contribution to the quality of generated outputs rather than just document ranking. The framework covers LLM-agnostic vs. LLM-specific utility, context-independent vs. context-dependent utility, and the integration of retrieval with agentic RAG systems.
Entities (5)
Relation Signals (3)
Hengran Zhang â researches â Retrieval-Augmented Generation
confidence 100% ¡ Her research centers on utility-focused RAG and dense retrieval.
LLM-centric utility â evolvedfrom â Topical relevance
confidence 95% ¡ This tutorial argues that retrieval objectives are evolving from relevance-centric optimization toward LLM-centric utility.
Retrieval-Augmented Generation â requires â LLM-centric utility
confidence 90% ¡ Retrieval effectiveness must be evaluated by its contribution to generation quality rather than by relevance-based ranking metrics alone.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Information retrieval systems have traditionally optimized for topical relevance-the degree to which retrieved documents match a query. However, relevance only approximates a deeper goal: utility, namely, whether retrieved information helps accomplish a user's underlying task. The emergence of retrieval-augmented generation (RAG) fundamentally changes this paradigm. Retrieved documents are no longer consumed directly by users but instead serve as evidence for large language models (LLMs) that produce answers. As a result, retrieval effectiveness must be evaluated by its contribution to generation quality rather than by relevance-based ranking metrics alone. This tutorial argues that retrieval objectives are evolving from relevance-centric optimization toward LLM-centric utility. We present a unified framework covering LLM-agnostic versus LLM-specific utility, context-independent versus context-dependent utility, and the connection with LLM information needs and agentic RAG. By synthesizing recent advances, the tutorial provides conceptual foundations and practical guidance for designing retrieval systems aligned with the requirements of LLM-based information access.
Tags
Links
- Source: https://arxiv.org/abs/2604.08920v1
- Canonical: https://arxiv.org/abs/2604.08920v1
Trouble viewing inline? Open PDF directly â
Full Text
28,681 characters extracted from source content.
Expand or collapse full text
Beyond Relevance: Utility-Centric Retrieval in the LLM Era Hengran Zhang State Key Laboratory of AI Safety, ICT, CAS University of Chinese Academy of Sciences Beijing, China zhanghengran22z@ict.ac.cn Minghao Tang State Key Laboratory of AI Safety, ICT, CAS University of Chinese Academy of Sciences Beijing, China tangminghao25s@ict.ac.cn Keping Bi State Key Laboratory of AI Safety, ICT, CAS University of Chinese Academy of Sciences Beijing, China bikeping@ict.ac.cn Jiafeng Guo State Key Laboratory of AI Safety, ICT, CAS University of Chinese Academy of Sciences Beijing, China guojiafeng@ict.ac.cn Abstract Information retrieval systems have traditionally optimized for topi- cal relevanceâthe degree to which retrieved documents match a query. However, relevance only approximates a deeper goal: utility, namely, whether retrieved information helps accomplish a userâs underlying task. The emergence of retrieval-augmented generation (RAG) fundamentally changes this paradigm. Retrieved documents are no longer consumed directly by users but instead serve as evi- dence for large language models (LLMs) that produce answers. As a result, retrieval effectiveness must be evaluated by its contribu- tion to generation quality rather than by relevance-based ranking metrics alone. This tutorial argues that retrieval objectives are evolving from relevance-centric optimization toward LLM-centric utility. We present a unified framework covering LLM-agnostic versus LLM-specific utility, context-independent versus context- dependent utility, and the connection with LLM information needs and agentic RAG. By synthesizing recent advances, the tutorial provides conceptual foundations and practical guidance for design- ing retrieval systems aligned with the requirements of LLM-based information access. CCS Concepts ⢠Information systemsâ Information retrieval. Keywords User-Centric Utility, LLM-Centric Utility, Retrieval, RAG ACM Reference Format: Hengran Zhang, Minghao Tang, Keping Bi, and Jiafeng Guo. 2018. Beyond Relevance: Utility-Centric Retrieval in the LLM Era. In Proceedings of Make sure to enter the correct conference title from your rights confirmation email (Conference acronym âX). ACM, New York, NY, USA, 4 pages. https://doi. org/X.X Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. Conference acronym âX, June 03â05, 2018, Woodstock, NY Š 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-X-X/18/06 https://doi.org/X.X Cover Sheet Information Title: Beyond Relevance: Utility-Centric Retrieval in the LLM Era Format: This tutorial is a half-day (3-hour) lecture-style session that includes scheduled breaks and will be conducted on-site. Intended Audience: Intermediate. This tutorial targets an audi- ence who is interested in RAG and LLM-based information access. It is particularly relevant to IR researchers exploring utility-oriented retrieval and to NLP/LLM researchers seeking principled ways to integrate retrieval into generative systems. Prerequisite Knowledge: Basic familiarity with information re- trieval and large language models is assumed. Previous Talks: This tutorial has not been presented before. Presentors: Hengran Zhang is a Ph.D. student at the Institute of Comput- ing Technology (ICT), Chinese Academy of Sciences (CAS). Her re- search centers on utility-focused RAG and dense retrieval. Her work has been published in top-tier conferences, including SIGIR and EMNLP. She has conducted extensive research on utility-focused RAG, including utility-based evidence selection [38,40,43,44], utility-focused annotation for retrieval and RAG [42], and LLM- specific utility modeling [41]. Minghao Tang is a masterâs student at the ICT, CAS. His re- search focuses on knowledge injection for LLMs. He has collabo- rated with Hengran Zhang and Keping Bi on utility-focused RAG [42] and has publications at EMNLP, ICLR, and SIGIR-AP. Keping Bi is an Associate Professor at the ICT, CAS. Her re- search primarily focuses on trustworthy information access, with special emphasis on IR and retrieval-augmented generation. She received her Ph.D. from the University of Massachusetts Amherst under the supervision of Prof. W. Bruce Croft. Dr. Bi has served in several academic leadership roles, including General Co-Chair of SIGIR-AP 2025 and Co-Editor of SIGIR Forum (2025-2028). Jiafeng Guo is a Professor at the ICT, CAS. His current research focuses on agentic information seeking, generative retrieval, and retrieval robustness. He has received multiple Best Paper Awards at top-tier conferences, including CIKM, SIGIR, and EMNLP. He has also actively contributed to the IR community through conference organization, serving as Short Paper Co-Chair of SIGIR 2020 and ICTIR 2019, and PC Co-Chair of CCIR 2018, among other roles. He has delivered tutorials on generative retrieval and robust IR at major conferences, including SIGIR, WSDM, and CIKM. arXiv:2604.08920v1 [cs.IR] 10 Apr 2026 Conference acronym âX, June 03â05, 2018, Woodstock, NYHengran Zhang et al. 1 Motivation and Overview Relevance and utility have long been recognized as two founda- tional criteria in information retrieval [27â29]. Relevance typically refers to topical relevance, the degree of correspondence between the subject of a query and retrieved documents, often operational- ized through the notion of âaboutnessâ [29]. In contrast, utility concerns the usefulness or value of retrieved information to an in- formation seeker, emphasizing whether retrieved items help achieve the userâs underlying goals [4,27,29]. Because usefulness depends on user context, task, and situational needs, utility has historically been difficult to observe and quantify automatically [1]. Topical relevance, by comparison, is more straightforward to annotate and evaluate, making it the dominant optimization target in IR research for decades, especially dense retrieval studies in recent years [19, 39]. The rise of retrieval-augmented generation (RAG) introduces a fundamental shift. Retrieval results are no longer primarily pre- sented to human users as ranked lists; instead, they serve as evi- dence consumed by large language models (LLMs) that synthesize answers. This transformation changes the role of retrieval systems. Effectiveness is no longer determined by whether documents ap- pear relevant to human readers or help them achieve their goals, but by whether they improve the quality of generated responses [44]. Under this setting, a document may be topically relevant yet inef- fective for LLM generation, while other evidence may substantially enhance reasoning, grounding, or factual correctness. Importantly, compared with user-centric utility, LLM-centric utility is often more directly measurable through answer-level metrics such as correct- ness or likelihood-based evaluation, making it a more tractable and feasible optimization objective. This tutorial argues that the objective of information retrieval is evolving, from topical relevance and user-centric utility, toward LLM-centric utility. By tracing this progression, we provide a unified perspective on how retrieval objectives are changing in the LLM era and review emerging work on modeling, estimating, and optimizing LLM-centric utility. 1.1 User-Centric Utility As early as 1971, Cooper[4]distinguished topical relevance from utility, arguing that relevance assesses whether information per- tains to the subject of a query from a system-oriented perspective, whereas utility reflects the ultimate usefulness of information to the user. The notion of utility was later considered as a type of relevance in the debates about what relevance truly represents in information retrieval [27,28,30]. This user-oriented perspective for retrieval emphasizes that usefulness depends on a userâs situation, intent, and task context. Information becomes valuable not merely because it matches a topic, but because it contributes to problem resolution or decision-making. Because utility cannot be directly observed, practical systems developed indirect ways to approximate it. Web search engines in- creasingly relied on implicit feedback signals, such as clicks, dwell time, and session behavior, to infer usefulness at scale [2,18,21, 24,25,47]. In recommender systems, measures such as clicks and purchases are often used as optimization objectives, as they more directly reflect user satisfaction and task success [36]. Furthermore, modern ranking systems frequently optimize the utility of the en- tire ranked list rather than treating documents independently at each position [6,34]. This perspective moves beyond the traditional probabilistic ranking principle [26], which focuses on ranking doc- uments solely according to individual relevance probabilities, and instead emphasizes holistic outcome-driven optimization. In this paradigm, humans simultaneously act as both consumers and evaluators of retrieval results. Retrieval systems learn from user behavior and are ultimately judged by how well they satisfy human information needs. User-centric utility thus represents a transition from static relevance estimation to interaction-aware and outcome-oriented information access. 1.2 LLM-Centric Utility RAG fundamentally reshapes the role of retrieval in information access by introducing a new consumer of retrieval results: the LLM itself. Instead of directly assisting human readers, retrieved docu- ments now function as supporting evidence that guides generation. The quality of retrieval must therefore be assessed through its con- tribution to generation outcomes rather than through standalone document relevance. Traditional retrieval metrics, such as nDCG, MAP, and MRR, mea- sure ranking quality with respect to topical relevance. However, they do not directly capture whether retrieved evidence improves generated answers [3]. In contrast to web search, where users di- rectly judge retrieved results, RAG systems decouple evaluation and consumption: users assess the final synthesized answer, while retrieval utility is defined by how effectively evidence supports the LLMâs generation process. In this tutorial, we define LLM-centric utility as the extent to which retrieved information improves gen- eration quality, such as answer correctness. We organize related work into three complementary dimensions: LLM-Agnostic Utility and LLM-Specific Utility. LLM-agnostic utility assumes that evidence possesses intrinsic informational value that generalizes across different generators. Unlike human- annotated topical relevance, utility has been approximated by mea- suring generation performance conditioned on specific documents, for example, using BLEU, ROUGE, EM, or F1 against ground-truth answers [7,9,10,20]; the likelihood of the ground-truth answer given the query and document [22,32]; model attention distribu- tions over input documents [12,13]; or performance differences between using and not using a document [5,11,43,43,45]. To ob- tain utility signals without relying on ground-truth answers, utility can also be annotated by advanced LLMs based on pseudo answers they generate, enabling scalable estimation of document usefulness without human labeling [38, 40, 42, 44]. In contrast, LLM-specific utility [41] recognizes that the same ev- idence may vary in usefulness across different LLMs. Analogous to personalized web search, where identical results differ in value for users with distinct intentions, expertise levels, or constraints, LLMs may adopt different utility criteria due to variations in training data, internal knowledge, reasoning strategies, and comprehen- sion capabilities [41]. In contrast to general utility that typically offers stronger cross-model generalization but may not maximize performance for a particular LLM, LLM-specific utility can be tai- lored to optimize generation quality for a given model, potentially improving performance at the cost of reduced transferability. Beyond Relevance: Utility-Centric Retrieval in the LLM EraConference acronym âX, June 03â05, 2018, Woodstock, NY Context-Independent and Context-Dependent Utility. Most existing approaches assume context-independent utility, where the usefulness of each document is evaluated independently of oth- ers [32,44]. This simplification makes labeling and optimization tractable, allowing standard supervised learning techniques to be applied. However, in realistic RAG settings, utility is often context- dependent. The usefulness of a document may depend on which other documents are present [14]. A document may provide re- dundant information in one context but supply a crucial missing reasoning step in another. This notion resembles search result di- versification, where ranking decisions consider both relevance and novelty. When modeling context-dependent utility, the objective becomes retrieving a set of evidence that jointly maximizes over- all generation utility. This requires capturing setwise interactions among documents and is particularly important in scenarios involv- ing multi-aspect questions or multi-hop reasoning, where different pieces of evidence resolve different subproblems [35]. LLM Information Needs and Agentic RAG. Beyond con- structing utility labels and training retrievers accordingly, another line of work seeks to identify useful documents by explicitly model- ing the LLMâs information needs. Rather than estimating document utility after retrieval, this perspective reframes retrieval as the pro- cess of satisfying the LLMâs latent knowledge gaps. Unlike web search, where users can iteratively reformulate queries as their information needs evolve, LLMs do not inherently express such needs. Consequently, researchers have proposed methods to infer and externalize these needs, for example, by generating queries based on model uncertainty reflected in low-confidence tokens or attention patterns over context [15, 16, 23, 31, 33]. A related paradigm is agentic RAG, in which the LLM iteratively generates queries, retrieves evidence, produces answers, and re- ceives feedback based on answer quality. Reinforcement learning is then used to optimize reasoning and query generation policies [8,17,37,46], with utility naturally emerging as the reward signal. However, in most agentic RAG systems, the retriever itself remains fixed, often relying on efficient but static methods such as BM25. While these approaches refine query generation and reasoning strategies, they typically do not reconsider the underlying retrieval objective. This tutorial, therefore, discusses agentic RAG in relation to utility-focused retrieval, while maintaining its primary emphasis on how retrieval objectives themselves evolve in the LLM era. 1.3 Summary In summary, the LLM era challenges long-standing assumptions about what retrieval systems should optimize. As retrieval transi- tions from serving human readers directly to supporting genera- tive models, topical relevance becomes an insufficient objective. A broader framework centered on utility, particularly LLM-centric utility, offers a principled way to rethink retrieval evaluation, su- pervision, and optimization. By clarifying distinctions between relevance, user-centric utility, and LLM-centric utility, and by orga- nizing emerging work along axes of model specificity, contextual dependency, and information need modeling, this tutorial aims to provide both conceptual grounding and a roadmap for future research on utility-oriented retrieval in the LLM era. 2 Objectives This tutorial aims to equip attendees with a clear understanding of how retrieval objectives are evolving in the LLM era. Specifically, participants will: ⢠Understand the shift in retrieval goals, distinguishing topical relevance, user-centric utility, and LLM-centric utility, and why relevance-based optimization is insufficient for RAG systems. â˘Learn frameworks for modeling LLM-centric utility, including LLM-agnostic vs. LLM-specific utility and context-independent vs. context-dependent utility. â˘Examine evaluation and training strategies that align retrieval with downstream generation quality rather than traditional rank- ing metrics alone. ⢠Identify open challenges and research directions in utility-oriented retrieval and its integration with agentic RAG. By the end of the tutorial, attendees will gain both conceptual clar- ity and practical guidance for designing next-generation retrieval systems aligned with LLM-based generation. 3 Relevance to the IR Community Information retrieval is undergoing a fundamental transition. While traditional IR research has centered on estimating and optimizing topical relevance, the rapid adoption of retrieval-augmented gen- eration (RAG) systems is reshaping the role of retrieval. Retrieved documents are increasingly consumed not by human users directly, but by large language models (LLMs) that generate responses. This shift raises foundational questions about what retrieval systems should optimize and how effectiveness should be evaluated. This tutorial is timely and relevant to the IR community as it connects classical theories of relevance and utility with emerging LLM-based paradigms. By synthesizing recent advances in utility modeling, retriever optimization, and evaluation beyond traditional ranking metrics, the tutorial provides both conceptual grounding and research directions. It aims to support IR researchers and prac- titioners in designing retrieval systems better aligned with the evolving landscape of LLM-driven information access. 4 Format and Schedule This tutorial is designed as a half-day (3-hour) lecture-style session with scheduled breaks. It will be delivered on-site, with at least two of the presenters attending the conference in person. ⢠Introduction and Foundations (20 min) â Tutorial overview and motivation â Relevance vs. utility: conceptual foundations â From user-centric to LLM-centric utility ⢠LLM-Agnostic and LLM-Specific Utility (50 min) â Modeling LLM-agnostic utility â Defining LLM-specific utility â Trade-offs, evaluation, and open challenges ⢠Q&A (10 min) ⢠Break (30 min) ⢠Context-Independent and Context-Dependent Utility (30 min) â Independent utility estimation â Setwise and interaction-aware utility modeling â Optimization challenges and future directions ⢠LLM Information Needs and Agentic RAG (30 min) Conference acronym âX, June 03â05, 2018, Woodstock, NYHengran Zhang et al. â Modeling LLM information needs â Query generation and agentic retrieval loops â Reinforcement learning and system-level optimization ⢠Q&A and Discussion (10 min) 5 Tutorial Material All supplemental materials for this tutorial will be accessible via a dedicated website shared with attendees beforehand. Resources include a comprehensive manuscript, a topic-organized reference list, and all tutorial slides. Additionally, a GitHub repository will be maintained to survey and categorize relevant research papers, supporting ongoing learning and exploration. References [1]Harry W Bruce. 1994. A cognitive view of the situational dynamism of user- centered relevance estimation. JASIST 45, 3 (1994), 142â148. [2] Georg Buscher, Ludger Van Elst, and Andreas Dengel. 2009. Segment-level display time as implicit feedback: a comparison to eye tracking. In SIGIRâ09. 67â74. [3]Lorenzo Canale, Stefano Scotta, Alberto Messina, and Laura Farinetti. 2025. BES4RAG: A Framework for Embedding Model Selection in Retrieval-Augmented Generation. In CLiC-it 2025. CEUR Workshop, Cagliari, Italy, 134â142. [4]William S Cooper. 1971. A definition of relevance for information retrieval. Information storage and retrieval 7, 1 (1971), 19â37. [5]Lu Dai, Yijie Xu, Jinhui Ye, Hao Liu, and Hui Xiong. 2025. Seper: Measure retrieval utility through the lens of semantic perplexity reduction. ICLRâ26 (2025). [6]Xinyi Dai, Jiawei Hou, Qing Liu, Yunjia Xi, Ruiming Tang, Weinan Zhang, Xi- uqiang He, Jun Wang, and Yong Yu. 2020. U-rank: Utility-oriented learning to rank with implicit feedback. In CIKMâ20. 2373â2380. [7]Chunjing Gan, Dan Yang, Binbin Hu, Hanxiao Zhang, Siyuan Li, Ziqi Liu, Yue Shen, Lin Ju, Zhiqiang Zhang, Jinjie Gu, et al.2024. Similarity is not all you need: Endowing retrieval augmented generation with multi layered thoughts. arXiv preprint arXiv:2405.19893 (2024). [8]Jingsheng Gao, Linxu Li, Weiyuan Li, Yuzhuo Fu, and Bin Dai. 2025. Smartrag: Jointly learn rag-related tasks from the environment feedback. ICLRâ25 (2025). [9]Xinyu Gao, Yun Xiong, Deze Wang, Zhenhan Guan, Zejian Shi, Haofen Wang, and Shanshan Li. 2024. Preference-Guided Refactored Tuning for Retrieval Augmented Code Generation. In ASEâ24. 65â77. [10]Sebastian Hofstätter, Jiecao Chen, Karthik Raman, and Hamed Zamani. 2022. Multi-Task Retrieval-Augmented Text Generation with Relevance Sampling. arXiv preprint arXiv:2207.03030 (2022). [11]Xuming Hu, Zhaochen Hong, Zhijiang Guo, Lijie Wen, and Philip Yu. 2023. Read it twice: Towards faithfully interpretable fact verification by revisiting evidence. In SIGIRâ23. 2319â2323. [12] Gautier Izacard and Edouard Grave. 2020. Distilling knowledge from reader to retriever for question answering. ICLRâ21 (2020). [13]Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023. Atlas: few-shot learning with retrieval augmented language models. J. Mach. Learn. Res. 24, 1, Article 251 (Jan. 2023), 43 pages. [14]Akriti Jain and Aparna Garimella. 2025. Modeling Contextual Passage Utility for Multihop Question Answering. In IJCNLP-AACLâ25. 464â471. [15]Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong Park. 2024. Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity. In NAACLâ24. Association for Computational Linguistics, Mexico City, Mexico, 7036â7050. [16]Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. In EMNLPâ23. 7969â7992. [17]Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516 (2025). [18] Seikyung Jung, Jonathan L Herlocker, and Janet Webster. 2007. Click data as implicit relevance feedback in web search. Information processing & management 43, 3 (2007), 791â807. [19]Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering.. In EMNLPâ20. 6769â6781. [20]Zixuan Ke, Weize Kong, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. 2024. Bridging the preference gap between retrievers and llms. In ACLâ24. 10438â10451. [21]Diane Kelly and Nicholas J Belkin. 2004. Display time as implicit feedback: understanding task effects. In SIGIRâ04. 377â384. [22]Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich KĂźttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al.2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. NeurIPSâ20 33 (2020), 9459â9474. [23] Xiaonan Li, Changtai Zhu, Linyang Li, Zhangyue Yin, Tianxiang Sun, and Xipeng Qiu. 2023. Llatrieval: Llm-verified retrieval for verifiable generation. arXiv preprint arXiv:2311.07838 (2023). [24]Hongyu Lu, Min Zhang, and Shaoping Ma. 2018. Between clicks and satisfaction: Study on multi-phase user preferences and satisfaction for online news reading. In SIGIRâ18. 435â444. [25]Cheng Luo, Yiqun Liu, Tetsuya Sakai, Ke Zhou, Fan Zhang, Xue Li, and Shaoping Ma. 2017. Does document relevance affect the searcherâs perception of time?. In WSDMâ17. 141â150. [26] Stephen E Robertson. 1977. The probability ranking principle in IR. Journal of documentation 33, 4 (1977), 294â304. [27]Tefko Saracevic. 1975. Relevance: A review of and a framework for the thinking on the notion in information science. JASIST 26, 6 (1975), 321â343. [28] Tefko Saracevic. 1996. Relevance reconsidered. In Proceedings of the second conference on conceptions of library and information science (CoLIS 2). 201â218. [29]Tefko Saracevic, Paul Kantor, Alice Y Chamis, and Donna Trivison. 1988. A study of information seeking and retrieving. I. Background and methodology. JASIST 39, 3 (1988), 161â176. [30] Alfred Schutz and Lester Embree. 2011. Reflections on the Problem of Relevance. Springer. [31]Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy. In Findings of the EMNLP 2023. 9248â 9274. [32] Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2024. REPLUG: Retrieval-Augmented Black-Box Language Models. In NAACLâ24. 8371â8384. [33] Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, and Yiqun Liu. 2024. Dragin: Dynamic retrieval augmented generation based on the real-time information needs of large language models. In ACLâ24. 12991â13013. [34]Yue Wang, Dawei Yin, Luo Jie, Pengyuan Wang, Makoto Yamada, Yi Chang, and Qiaozhu Mei. 2016. Beyond ranking: Optimizing whole-page presentation. In WSDM2016. 103â112. [35] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLPâ18. 2369â2380. [36] Feng Yu, Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. 2016. A dynamic recurrent model for next basket recommendation. In SIGIRâ16. 729â732. [37] Dingchu Zhang, Yida Zhao, Jialong Wu, Baixuan Li, Wenbiao Yin, Liwen Zhang, Yong Jiang, Yufeng Li, Kewei Tu, Pengjun Xie, et al.2025. EvolveSearch: An Iterative Self-Evolving Search Agent. arXiv preprint arXiv:2505.22501 (2025). [38]Hengran Zhang, Keping Bi, Jiafeng Guo, and Xueqi Cheng. 2026. An Iterative Utility Judgment Framework via LLMs Inspired by Relevance in Philosophy. In Findings of the ACL 2026. [39]Hengran Zhang, Keping Bi, Jiafeng Guo, Xiaojie Sun, Shihao Liu, Daiting Shi, Dawei Yin, and Xueqi Cheng. 2025. Unleashing the Power of LLMs in Dense Retrieval with Query Likelihood Modeling. arXiv preprint arXiv:2504.05216 (2025). [40]Hengran Zhang, Keping Bi, Jiafeng Guo, Jiaming Zhang, Shuaiqiang Wang, Dawei Yin, and Xueqi Cheng. 2025. Distilling a Small Utility-Based Passage Selector to Enhance Retrieval-Augmented Generation. In SIGIR-APâ25. 22â30. [41]Hengran Zhang, Keping Bi, Jiafeng Guo, Jiaming Zhang, Shuaiqiang Wang, Dawei Yin, and Xueqi Cheng. 2025. LLM-Specific Utility: A New Perspective for Retrieval- Augmented Generation. arXiv preprint arXiv:2510.11358 (2025). [42]Hengran Zhang, Minghao Tang, Keping Bi, Jiafeng Guo, Shihao Liu, Daiting Shi, Dawei Yin, and Xueqi Cheng. 2025. Utility-Focused LLM Annotation for Retrieval and Retrieval-Augmented Generation. In EMNLPâ25. 1683â1702. [43]Hengran Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. 2023. From relevance to utility: Evidence retrieval with feedback for fact verification. In Findings of the EMNLP 2023. 6373â6384. [44]Hengran Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. 2024. Are Large Language Models Good at Utility Judgments?. In SIGIRâ24. 1941â1951. [45]Xinping Zhao, Dongfang Li, Yan Zhong, Boren Hu, Yibin Chen, Baotian Hu, and Min Zhang. 2024. Seer: Self-aligned evidence extraction for retrieval-augmented generation. EMNLPâ24 (2024). [46]Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. 2025. Deepresearcher: Scaling deep research via reinforce- ment learning in real-world environments. arXiv preprint arXiv:2504.03160 (2025). [47]Xiaofei Zhu, Jiafeng Guo, Xueqi Cheng, and Yanyan Lan. 2012. More than relevance: high utility query recommendation by mining usersâ search behaviors. In CIKMâ12. 1814â1818.