Paper deep dive
Training Documents Reranker with Search Rubrics for Deep Research Agent
Wenhan Liu, Yu Lu, Qiaolin Xia, Hui Xu, Tong Zhao, Jian Xi, Yutao Zhu, Haijin Liang, Haibo Shi, Hao Wang, Zhicheng Dou
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Retrieval systems help deep research agents generate high-quality answers by providing relevant documents. However, existing retrievers typically select documents through relevance matching, while individually well-matched top-$k$ documents may not form a \textit{set} that satisfies the complex information needs of an agent query (\eg, diverse, concise and authoritative documents). In this paper, we propose search-oriented rubrics that \textit{explicitly} define the requirements that high-quality document sets should satisfy for each agent query. Our search rubrics are organized into a hierarchical structure and synthesized using a powerful LLM. Based on these search rubrics, we further train a document reranker \textbf{RubricRanker} to select a high-quality subset from retrieved documents. We design a two-stage training framework that consists of rubrics-guided supervised fine-tuning and rubric-based reinforcement learning. Extensive experiments demonstrate that RubricRanker outperforms the strongest baseline by 2.6 points on four deep research benchmarks and generalizes well to five RAG benchmarks.
Tags
Links
- Source: https://arxiv.org/abs/2608.03527v1
- Canonical: https://arxiv.org/abs/2608.03527v1
Trouble viewing inline? Open PDF directly â
Full Text
90,773 characters extracted from source content.
Expand or collapse full text
Training Documents Reranker with Search Rubrics for Deep Research Agent Wenhan Liu 1 , Yu Lu 2 , Qiaolin Xia 2 , Hui Xu 2 , Tong Zhao 1 , Jian Xi 2 ,Yutao Zhu 1 Haijin Liang 2 , Haibo Shi 2 , Hao Wang 2 * and Zhicheng Dou 1â 1 Gaoling School of Artificial Intelligence, Renmin University of China 2 Tencent, Beijing, China lwh@ruc.edu.cn, dou@ruc.edu.cn Abstract Retrieval systems help deep research agents generate high-quality answers by providing rel- evant documents. However, existing retrievers typically select documents through relevance matching, while individually well-matched top- kdocuments may not form a set that satisfies the complex information needs of an agent query (e.g., diverse, concise and authoritative documents). In this paper, we propose search- oriented rubrics that explicitly define the re- quirements that high-quality document sets should satisfy for each agent query. Our search rubrics are organized into a hierarchical struc- ture and synthesized using a powerful LLM. Based on these search rubrics, we further train a document reranker RubricRanker to select a high-quality subset from retrieved documents. We design a two-stage training framework that consists of rubrics-guided supervised fine- tuning and rubric-based reinforcement learning. Extensive experiments demonstrate that Rubri- cRanker outperforms the strongest baseline by 2.6 points on four deep research benchmarks and generalizes well to five RAG benchmarks. 1 Introduction Deep research agents aim to produce in-depth and well-supported answers to complex information- seeking tasks by planning, searching, and synthe- sizing evidence from diverse sources (Xu and Peng, 2025; Shi et al., 2025; Shao et al., 2025; Li et al., 2025). Unlike conventional single-turn retrieval set- tings, a deep research agent interacts with search system over multiple steps: it reasons about the current information need, issues an agent query, observes retrieved documents, and decides whether further search is needed before producing a long- form answer. As a result, the ability of search system (i.e., document retrievers and rerankers) di- * Corresponding author. â Corresponding author. I have felt low for weeks, and lose interest in things. Am I depressed? And what should I do? How to treat depression? User Question Agent Query Document Retriever Documents [Dâ] Medication treatment for depression may include fluoxetine, sertraline, and citalopram... Source: Health Ministry [Dâ] Psychotherapy treatment helps identify and change harmful thoughts, improve coping skills, and manage ... Source: Municipal Hospital [Dâ] Psychotherapy treatment improves mood through counselling, emotional support, and strategies ... Source: Personal Blog Unmet Rubrics Docs should mention self-regulation as a treatment type. Docs should provide non-redundant contents on depression. Docs should come from authoritative medical websites. Figure 1: Document-set selection based on single- document relevance matching cannot satisfy the com- plex information needs of agent queries, such as cover- age of diverse aspects, conciseness, and authority. rectly affects the agentâs ability to gather sufficient evidence and generate a reliable final response. Existing document retrievers (Xiao et al., 2024; Wang et al., 2022) and rerankers (Zhuang et al., 2023; Nogueira et al., 2020; Pradeep et al., 2023b) typically return the top-kdocuments based on rele- vance matching. Their training supervision is usu- ally based on relevance labels, which measure the relevance of single document. However, it cannot ensure that the returned document set satisfies the complex information needs of an agent query. As illustrated in Figure 1, to answer a help-seeking question from a patient who may have depression, the agent issues the open-ended query âhow to treat depressionâ. Although all retrieved documents are relevant to depression, the resulting set remains sub- optimal for several reasons: (1) it misses an impor- tant type of depression treatment, self-regulation; (2) D2 and D3 contain redundant information about psychotherapy, wasting the agentâs context budget; and (3) D3 comes from a non-authoritative source, arXiv:2608.03527v1 [cs.IR] 4 Aug 2026 which may lead the agent to generate unreliable medical advice. To address this limitation, we propose search- oriented rubrics that specify the requirements that a high-quality document set should satisfy for each query. Our search rubrics serve as query-specific supervision signals to train our document reranker, RubricRanker, which selects a high-quality sub- set from retrieved documents. To construct search rubrics, we first define hierarchical meta rubrics that cover relevance, conciseness, and consistency at the set level, as well as source authority and timeliness at the document level. We then use a powerful LLM GPT-5.1 to construct a high-quality reference answer for each query to identify the key aspects, facts, and constraints that the selected doc- uments should support. Finally, based on the query and its reference answer, we use GPT-5.1 to expand the meta rubrics into query-specific rubrics with corresponding importance weights. Based on the constructed search rubrics, we pro- pose a two-stage training framework, consisting of rubrics-guided supervised fine-tuning (SFT) and rubric-based reinforcement learning (RL), to train our RubricRanker. In the SFT stage, we apply pow- erful teacher model to select the documents that best satisfy the query-specific rubrics and provide silver labels for cold-start training. In the RL stage, we use the search rubrics to evaluate the document sets selected by the policy model. We design a hierarchical reward aggregation method that com- bines the set-level and document-level rubric scores into a rubric RL reward. At inference time, Rubri- cRanker directly selects a high-quality subset from the retrieved candidates given the query, without requiring the search rubrics. Extensive experiments demonstrate that Rubr- icRanker achieves superior performance on deep research benchmarks and generalizes well to RAG benchmarks. Further experiments show that Rubri- cRanker significantly reduces the number of agent search calls, enabling agents to generate better an- swers with lower latency. The contributions of this paper are summarized as follows: ⢠We propose search-oriented rubrics that de- fine the properties of high-quality document sets from both set-level and document-level perspectives. ⢠We propose a two-stage training framework that combines rubrics-guided SFT with rubric- based RL to train a document reranker for both deep research and RAG tasks. ⢠Extensive experiments demonstrate that Rubr- icRanker achieves strong performance on both deep research and RAG benchmarks. 2 Related Work RAG and Deep Research Retrieval-augmented generation (RAG) enables LLMs to access exter- nal knowledge beyond their parameters (Gao et al., 2023). However, single-turn retrieval is often insuf- ficient for complex information-seeking tasks that require iterative evidence gathering. This has moti- vated agentic search systems that interleave reason- ing with retrieval (Yao et al., 2023), as well as deep research agents that combine reasoning models with web search to solve open-ended queries (Shi et al., 2025; Li et al., 2025). In these agents, re- trieval quality affects not only the current obser- vation but also subsequent reasoning and search decisions, so errors can accumulate across multiple steps. We therefore focus on improving the docu- ment sets returned for each agent-issued query. Document Ranking Document ranking or- ders candidate documents by queryâdocument relevance, modeled through dense representa- tions (Xiao et al., 2024; Wang et al., 2022), interaction-based matching (Nogueira et al., 2020), or LLM-based ranking (Sun et al., 2023; Qin et al., 2024; Liu et al., 2024, 2025a,b, 2026c; Feng et al., 2026). Although these methods perform well on standard IR benchmarks, they mainly optimize rel- evance rankings of individual documents. With the development of RAG (Gao et al., 2023) and deep research agents (Xu and Peng, 2025), recent studies have instead optimized rerankers for down- stream LLM generation. Rank4Gen (Fan et al., 2026) trains rerankers with supervision derived from answer-generation quality, while Lee et al. (2025) and Zhang et al. (2025) distill document- selection preferences from strong teacher models. However, such signals are difficult to obtain for open-ended queries whose answers are hard to ver- ify, and answer-level preferences provide limited guidance about which properties of the selected document set should be improved. They may there- fore miss explicit requirements such as conciseness, consistency, and source quality. Rubrics provide structured criteria for evaluat- ing open-ended outputs (Liu et al., 2026a) and in- terpretable reward feedback (Gunjal et al., 2025; Huang et al., 2025; He et al., 2025; Shao et al., 2025). For example, RAR (Gunjal et al., 2025) uses instance-specific rubrics as rewards for rein- forcement learning on open-ended medical and sci- entific tasks. Inspired by these works, we propose search-oriented rubrics that explicitly define fine- grained document-set quality and use them to train rerankers for agent generation. 3 Preliminary In this section, we describe how document rerank- ing is incorporated into deep research and retrieval- augmented generation (RAG). 3.1 Document Reranking in Deep Research Deep research agents answer complex information- seeking questions through iterative reasoning and external information acquisition. We adopt the Re- Act paradigm (Yao et al., 2023), which is widely used in deep research agents (Shao et al., 2025; Li et al., 2025). At each step, the agent maintains an interaction history, produces a Thought and an Action, and receives an Observation from the en- vironment. Each intermediate action corresponds to a tool call, such as web search. The observation is appended to the agent context and influences both subsequent reasoning and whether additional evidence should be collected. The agent repeats this process until it has sufficient information to produce a long-form answer. We insert the document reranker after the agent invokes the search tool. Specifically, for a search action, the agent issues a queryq t , which we refer to as a sub-query to distinguish it from the orig- inal user question. The search API returns a list of web documentsD t = d 1 ,...,d n . Given the sub-queryq t and document listD t , the reranker ranks the documents and selects a high-quality sub- setS t â D t , which is then provided to the deep research agent as the observation for this search step. 3.2 Document Reranking in RAG RAG follows a simpler single-turn retrieval-and- generation paradigm. Given an input user ques- tionq, a retriever first searches an external cor- pus and returns a candidate document listD = d 1 ,...,d n .The document reranker reranks these candidates to select a high-quality document subsetS â Das evidence for answer generation. Unlike in deep research, retrieval is performed for the original user questionqrather than for agent- generated sub-queries. 4 Methodology This section presents the design of RubricRanker. The key idea is to express the characteristics of high-quality document sets as explicit search rubrics and then use these rubrics to train the reranker. Specifically, our framework consists of two parts: constructing query-specific search rubrics and training the reranker with rubric-based supervision. Figure 2 illustrates the overall frame- work. 4.1 Search Rubrics Construction We design query-specific search rubrics that cap- ture the properties a selected document set should satisfy at both the set and document levels. First, we design hierarchical meta rubrics as a general framework for query-specific rubric generation. We then collect training queries from both deep research and RAG scenarios and use a strong LLM to synthesize high-quality answers to these queries. Finally, we use a strong LLM to generate query- specific rubrics based on the meta rubrics, each query, and its synthesized answer. 4.1.1 Meta Rubrics Design. A widely used approach to rubric generation is to prompt LLMs to produce rubrics directly (Huang et al., 2025; Gunjal et al., 2025). However, direct prompting may produce rubrics with insufficient coverage, repetitive content, or conflated evalu- ation dimensions (Shen et al., 2026). We there- fore propose meta rubrics as a fixed framework for query-specific rubric generation. Our meta rubrics adopt a hierarchical structure that covers both set- level properties of the selected document set and document-level properties of each document. Set-level rubrics. Set-level rubrics evaluate the overall quality of a document set. We consider three dimensions: (1) Relevance, which measures whether the document set covers the key informa- tion needs and diverse answer aspects required by the query; (2) Conciseness, which measures whether the set contains little repetitive, redundant, or irrelevant content; and (3) Consistency, which measures whether key facts, claims, and conclu- sions across documents are mutually compatible rather than contradictory. Set-level Doc-level Relevance Concisenes Source Authority Timeliness Query Answer GPT-5.1 Teacher Query RubricRanker Rubrics Docs [2] [4] [13] [17] Query Docs RubricRanker Query Docs [1] [5] [9] I. Search Rubrics Construction Meta Rubrics Design Answer Synthesis Rubrics-Guided SFT Rubric-Based RL I. Rubric-Based Reranker Training Training Queries RAG User Questions Consistency Deep Research Agent Sub-queries Query-specific Rubrics Generation Answer Query Meta Rubrics Rubrics Generator (GPT-5.1) Set-level The doc set contains information of ... Doc-level Doc come from authoritative source ... Weight: 3 Weight: 5 Silver Label Rollout Judge LLM í 5 í 9 í 1 Judge LLM Doc-level Rubrics Rubric-weighted Aggregation Rubric Reward í 1 Set-level Rubrics Figure 2: An overview of our framework, which constructs query-specific search rubrics and uses them to train a document reranker through rubrics-guided SFT and rubric-based RL. Document-levelrubrics. Document-level rubrics evaluate the quality of a single document. We consider two dimensions: (1) Source au- thority, which measures whether a document provides relevant evidence from an authoritative and reliable source, especially in high-stakes domains such as medicine, law, and policy; and (2) Timeliness, which measures whether a document provides relevant evidence that satisfies explicit temporal constraints or needs for up-to-date information. Both dimensions are defined only for relevant documents; that is, an irrelevant document does not satisfy the source authority rubric even if it comes from authoritative sources. 4.1.2 Training Query Collection. In the deep research scenario, RubricRanker reranks documents for sub-queries issued by agents. However, existing open-source datasets generally provide only the original user ques- tions rather than these agent-issued sub-queries. To obtain such sub-queries, we collect open- ended questions from open-source deep research datasets, including OpenScholar (Asai et al., 2024), SearchArena (Miroyan et al., 2025), GlaiveAI- Reasoning-v1-20M, 1 and WebWalker-Silver (Wu et al., 2025). We then use a strong open-source deep research agent, Dr-Tulu-8B (Shao et al., 2025), to generate a full trajectory for each ques- tion and extract the sub-queries issued by the 1 https://huggingface.co/datasets/glaiveai/ reasoning-v1-20m agent as training queries. In addition, we collect user questions from RAG datasets, including Hot- potQA (Yang et al., 2018) and NQ (Kwiatkowski et al., 2019), whose answers are short and closed- ended. These questions help construct more diverse rubrics and improve the generalization of Rubri- cRanker across different query scenarios. 4.1.3 Answer Synthesis. Generating query-specific rubrics requires identify- ing which aspects should be covered, which facts are crucial, and what constraints the supporting ev- idence should satisfy. However, asking an LLM to generate rubrics based solely on the query may not fully capture these information needs, partic- ularly for complex and open-ended agent queries. Therefore, for agent sub-queries in the deep re- search scenario, we propose to use the powerful GPT-5.1 model (with web search) to synthesize a high-quality answer as an intermediate information- need blueprint. The prompt is shown in Figure 5. By decomposing the query into concrete aspects and claims, the answer provides a richer basis from which the subsequent rubric generator can derive specific and comprehensive evaluation criteria. For user questions in the RAG scenario, we directly use the gold answers provided by the original datasets. 4.1.4 Query-Specific Rubrics Generation. Based on the training queries and their correspond- ing reference answers, we use a strong LLM-based rubric generator GPT-5.1 to expand the prede- fined meta rubrics into query-specific rubrics. The prompts for agent sub-queries in the deep research scenario and user questions in the RAG scenario are shown in Figures 6â7 and Figures 8â9, respec- tively. Each type of meta rubric may be instantiated as one or more query-specific rubrics, each contain- ing a natural-language description and a weight from 1 to 5, with larger weights indicating more important criteria. 4.2 Rubric-Based Reranker Training Using the constructed query-specific rubrics as su- pervision signals, we train RubricRanker to select document sets that better satisfy the information needs of each query. The training process consists of two stages. First, we propose rubrics-guided supervised fine-tuning (SFT), in which a teacher model generates silver labels by selecting docu- ments according to the query-specific rubrics, pro- viding a stable cold start for the reranker. Second, to further enhance the rerankerâs document-set se- lection ability, we perform rubric-based reinforce- ment learning (RL), using a hierarchical reward aggregation method that combines set-level and document-level rubric scores to optimize the se- lected document sets. 4.2.1 Rubrics-Guided SFT Selecting a high-quality document set for a com- plex query is challenging for the base model, and low-quality rollouts may also hinder subsequent RL training. To address this issue, we use a strong teacher model guided by our constructed query- specific rubrics to produce silver labels for cold- start training. For each training query, we retrieve a candidate document list with a randomly sampled length rang- ing from 10 to 40 and use these candidates as train- ing documents. Such variable-length input lists expose RubricRanker to different input sizes dur- ing training and enable it to handle different num- bers of candidate documents at inference time. For agent sub-queries in the deep research scenario, we retrieve web documents using the Serper Google Search API. 2 For user questions in the RAG sce- nario, we use BGE-base model to retrieve docu- ments from the Wikipedia corpus. We then use GPT-5.1 as the teacher model to gen- erate labels. The teacher takes the query-specific rubrics, training query, 3 and candidate document 2 https://serper.dev/ 3 For an agent sub-query, we also provide the teacher and list as input and outputs a list of selected docu- ment IDs (e.g., â[2] [4] [13] [17]â) that best satisfy the query-specific rubrics. The student reranker is trained on these teacher labels using the query and candidate document list as input, without access to the query-specific rubrics, consistent with the final inference stage. 4.2.2 Rubric-Based RL After cold-start SFT, we further optimize the reranker with reinforcement learning to improve its document-set selection ability. The key motivation is to use rubric weights to compute more accurate rewards for document-set selection and to further refine the reranker through fine-grained compar- isons among different rollouts. Given a query and a candidate document list, the reranker outputs a sequence of selected document IDs, which is then parsed as a document set D. Givenquery-specificrubricsR= sr 1 ,...,sr i ⪠dr 1 ,...,dr j ,wheresr i anddr j denote thei-th set-level rubric and thej-th document-level rubric, respectively, we compute the rubric rewardP r through a hierarchical aggregation with rubric weights: P r (D) = P i sw i ¡S(sr i ,D) + P j dw j ¡F(dr j ,D) P i sw i + P j dw j , (1) wheresw i anddw j are the weights of the set-level rubricsr i and document-level rubricdr j , respec- tively. The scoring functionS(¡)is implemented by a GPT-5.1-based LLM judge, which assigns a normalized score between 0 and 1 according to the given rubric. For set-level rubrics,S(sr i ,D) scores whether the selected documentsDcollec- tively satisfy the query requirements. For each document-level rubricdr j , we first use the LLM judge to score each selected documentdand then average the scores over the selected set: F(dr j ,D) = 1 |D| X dâD S(dr j ,d).(2) In addition, we apply a format reward to ensure that the output can be parsed. Specifically, the output must follow a document-ID format such as â[1] [3] [2]â. If the output format is correct, we parse the selected document IDs asDand use the rubric reward as the final reward; otherwise, the student with the preceding agent reasoning (omitted from the figure for clarity) to clarify the query intent. final reward is set toâ1: P(y) = ( P r (D), if the output format is correct, â1,otherwise, (3) Finally, based on the final rewardP(y), we apply the GRPO algorithm (Shao et al., 2024) to optimize RubricRanker. The RL training details are provided in Section C.3. 5 Experiments 5.1 Settings Evaluation Benchmarks. We evaluate Rubri- cRanker in two scenarios. The first scenario is deep research, for which we use Health- Bench (Arora et al., 2025), WebWalkerQA (Wu et al., 2025), DeepResearchBench (DRB) (Du et al., 2025), and ResearchQA (Yifei et al., 2025). Detailed descriptions of these benchmarks are provided in Appendix A. For all these bench- marks, responses are expected to be open-ended or long-form and are evaluated by LLM judges us- ing rubrics from the official evaluation protocols. Given the high cost and time overhead of LLM- based evaluation, we randomly sample 100, 200, 100, and 100 queries from HealthBench, WebWalk- erQA, DRB, and ResearchQA, respectively, as our test sets. The second scenario is RAG, for which we use HotpotQA (Yang et al., 2018), Bam- boogle (Press et al., 2023), Natural Questions (NQ) (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), and PopQA (Mallen et al., 2023). Un- like the deep research benchmarks, these datasets mainly require short, closed-ended answers. Fol- lowing prior work (Jin et al., 2025; Liu et al., 2026b), we use exact match (EM) as the evaluation metric. Baselines. We compare RubricRanker with two categories of document rerankers.The first category is vanilla rerankers, which are mainly optimized for queryâdocument relevance to improve the ranking of retrieved documents. This category includes BGE-Reranker-Large 4 , MonoT5 (Nogueira et al., 2020), RankT5 (Zhuang et al., 2023), RankVicuna (Pradeep et al., 2023a), and RankZephyr (Pradeep et al., 2023b). The sec- ond category is generation-oriented rerankers, which aim to select documents that better support 4 https://huggingface.co/BAAI/ bge-reranker-large LLM answer generation. This category includes SetR (Lee et al., 2025) and Rank4Gen (Fan et al., 2026). Detailed descriptions of these baselines are provided in Appendix B. Implementation Details. We use Qwen3-8B as the backbone model for RubricRanker. For deep research benchmarks, we use the Serper Search API for document retrieval.For RAG bench- marks, we use the December 2018 Wikipedia dump (Karpukhin et al., 2020) as the retrieval cor- pus and adopt bge-base-en-v1.5 5 as the retriever. All document rerankers rank the top 30 retrieved documents and return a document set. For vanilla rerankers that output a full ranking, we use the top five documents. Due to space limitations, we provide further details in Appendix C. 5.2 Overall Performance We evaluate RubricRanker on both deep research and RAG benchmarks and report the results in Ta- bles 1 and 2, respectively. We make the following observations: (1) RubricRanker demonstrates superior per- formance in both deep research and RAG sce- narios.On deep research benchmarks, Rubr- icRanker achieves the best average score of 60.1, outperforming the second-best baseline, Rank4Gen, by 2.6 points. On RAG benchmarks, RubricRanker also obtains the highest average EM score of 40.0, surpassing Rank4Gen by about 2 points. These consistent improvements demon- strate that rubric-based reranker training can select document sets that better support downstream LLM generation. (2) Generation-oriented rerankers do not generalize to deep research tasks. Generation- oriented rerankers clearly improve over vanilla rerankers on RAG benchmarks, where Rank4Gen outperforms BGE-Reranker-Large by 1.8 EM points on average. However, the gain nearly disap- pears on deep research benchmarks. This suggests that rerankers trained on RAG-style, closed-form tasks may not generalize well to open-ended deep research, highlighting the need for search-rubric- based training. (3) Reranking substantially improves genera- tion quality over initial retrieval. All evaluated rerankers improve over initial retrieval in both sce- narios: average scores increase from 54.0 to 56.1â 60.1 on deep research benchmarks and from 34.3 to 5 https://huggingface.co/BAAI/bge-base-en-v1.5 MethodSizeWebWalkerQAHealthBenchDRBResearchQAAvg. Initial Retrieval (Google Search API)-44.555.245.371.154.0 Vanilla Rerankers BGE-Reranker-Large550M52.058.746.571.257.1 MonoT53B48.057.8 46.772.556.3 RankT53B53.057.046.672.857.4 RankVicuna7B47.059.646.571.356.1 RankZephyr7B50.058.646.471.856.7 Generation-oriented Rerankers SetR8B49.058.744.873.356.5 Rank4Gen8B52.059.246.672.057.5 RubricRanker (Ours)8B58.061.546.874.260.1 Table 1: Results on deep research benchmarks with Dr-Tulu-8B as the agent. All rerankers rerank the top 30 documents retrieved by the Google Search API, and the final responses are evaluated by an LLM judge using rubrics. The best results are highlighted in bold, and the second-best results areunderlined. MethodSizeHotpotQABamboogleNQPopQATriviaQAAvg. Initial Retrieval (BGE)-29.815.231.037.258.434.3 Vanilla Rerankers BGE-Reranker-Large550M 35.317.628.040.460.836.4 MonoT53B33.914.430.737.660.735.4 RankT53B34.414.432.138.561.136.1 RankVicuna7B33.414.430.738.459.935.3 RankZephyr7B32.816.031.939.560.636.1 Generation-oriented Rerankers SetR8B34.3 22.830.338.561.237.4 Rank4Gen8B35.120.0 33.241.061.738.2 RubricRanker (Ours)8B38.023.234.042.262.440.0 Table 2: Results on RAG benchmarks with Qwen3-8B as the LLM generator. All rerankers rerank the top 30 documents retrieved by BGE, and the generated answers are evaluated using exact match (EM). The best results are highlighted in bold, and the second-best results areunderlined. MethodWebWalker. Health. HotpotQA Avg. RubricRanker58.061.538.052.5 Training Stage w/o RL55.061.037.251.1 w/o SFT48.061.035.848.3 SFT Label w/o rubrics51.560.036.049.2 Relevance Ranking50.059.235.048.1 Table 3: Ablation study of RubricRanker on WebWalk- erQA, HealthBench, and HotpotQA. Avg. denotes the average score across the three datasets. 35.3â40.0 on RAG benchmarks. This confirms that reranking retrieved documents generally benefits downstream generation. 5.3 Ablation Study We conduct ablation studies to evaluate the con- tributions of different components in two aspects: (1) the two-stage training framework and (2) label construction. Experiments are conducted on two deep research datasets, WebWalkerQA and Health- Bench, and one RAG dataset, HotpotQA. The over- all results are reported in Table 3. We summarize our findings as follows. (1) Training stage. We ablate the two-stage training framework by removing rubric-based RL (âw/o RLâ) and cold-start SFT (âw/o SFTâ). Re- moving RL decreases the average score from 52.5 to 51.1, showing that rubric-based reward optimiza- tion provides additional gains beyond SFT. Remov- ing cold-start SFT leads to a larger performance drop, from 52.5 to 48.3, highlighting its critical role in providing a stable initialization for RL training. (2) Label construction. We further study how different SFT labels affect RubricRanker. Remov- ing query-specific rubrics from the teacherâs guid- ance during label construction (âw/o rubricsâ) re- duces the average score from 52.5 to 49.2, demon- strating that rubric guidance helps the teacher model identify document sets that better satisfy downstream generation requirements. We also 1020304050 Reranking Number 46 48 50 52 54 56 58 60 Score (%) WebWalker Rank4Gen RubricRanker 1020304050 Reranking Number 30 32 34 36 38 40 42 Score (%) HotpotQA Rank4Gen RubricRanker Figure 3: Performance of RubricRanker when reranking different numbers of retrieved documents on WebWalk- erQA and HotpotQA. test a widely used relevance-ranking label (âRele- vance Rankingâ) (Pradeep et al., 2023a,b), in which the teacher outputs a complete ranking ordered by document relevance rather than selecting a docu- ment set for generation. This variant further de- creases the average score to 48.1, indicating that relevance ranking is insufficient for generation- oriented document-set selection and validating the necessity of rubric-guided label construction. 5.4 Different Numbers of Reranked Documents The number of retrieved documents provided to the reranker can affect downstream generation quality. In this section, we investigate how this factor influ- ences final performance. Specifically, we evaluate RubricRanker on WebWalkerQA and HotpotQA while reranking the top 10, 20, 30, 40, and 50 re- trieved documents. As shown in Figure 3, increas- ing the number of reranked documents generally improves downstream performance because rele- vant documents may appear at lower positions in the initial retrieval list. However, the gains satu- rate and may even decline beyond a certain thresh- old, such as 30 documents on WebWalkerQA and 40 documents on HotpotQA. This may occur be- cause most relevant documents have already been retrieved, while reranking too many documents at once creates an overly long input context that can degrade the rerankerâs performance. 5.5 Search Call Analysis We further analyze the number of search calls made by the deep research agent when equipped with different document rerankers. Fewer search calls indicate that the reranker provides sufficient evidence more efficiently. We use Dr-Tulu (8B) as the deep research agent and evaluate it on HealthBenchDRBResearchQA Method 2.6 2.8 3.0 3.2 3.4 3.6 Number of Search Calls 3.2 3.3 3.2 3.4 3.6 3.5 2.9 3.0 2.9 RankT5Rank4GenRubricRanker Figure 4: The number of search calls made by the Dr- Tulu deep research agent when using different document rerankers. HealthBench, DeepResearchBench (DRB), and ResearchQA. We compare RubricRanker with two competitive reranking baselines, RankT5 and Rank4Gen. As shown in Figure 4, RubricRanker consistently leads to fewer search calls across all three datasets. On HealthBench, it reduces the aver- age number of calls from 3.2 with RankT5 and 3.4 with Rank4Gen to 2.9, corresponding to relative reductions of 9.4% and 14.7%, respectively. On ResearchQA, it reduces search calls from 3.2 and 3.5 to 2.9, corresponding to reductions of 9.4% and 17.1%. These consistent improvements suggest that RubricRanker provides more useful evidence at each step, allowing the agent to satisfy its infor- mation needs earlier and avoid repeated searches. 6 Conclusion We present RubricRanker, a document reranker that moves beyond individual queryâdocument rele- vance matching by modeling query-specific require- ments for high-quality document sets. Rather than supervising the model only with single-document relevance labels, we formulate hierarchical search rubrics that describe both the collective properties of a selected set and the quality of each document. Based on these rubrics, we propose a two-stage training framework that combines rubrics-guided SFT for cold-start training with rubric-based RL using fine-grained rubric rewards. The trained reranker internalizes these requirements and di- rectly selects a high-quality subset from retrieved candidates at inference without rubric inputs. Ex- periments demonstrate consistent gains on deep re- search and RAG benchmarks, strong cross-scenario generalization, and fewer search calls for deep re- search agents. Limitations Despite the effectiveness of RubricRanker, our work still has several limitations. (1) The con- struction of rubric-based rewards relies on GPT-5.1, which can be costly when scaling to large training corpora. In the future, we plan to train a special- ized reward model to provide more cost-effective and efficient rubric-based feedback. (2) Due to the high cost of LLM calls and the slow test-time infer- ence of deep research agents, we sample a subset of queries for evaluation on deep research bench- marks. In future work, we will consider evaluating on the full benchmark sets to provide more compre- hensive results. (3) Our evaluation still depends on the final generation quality of LLM agents. Even when RubricRanker selects a high-quality docu- ment set, the agent may still make mistakes during reasoning or generation. Developing more objec- tive metrics to directly evaluate the quality of se- lected document sets is a promising direction. References Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Pre- ston Bowman, Joaquin QuiĂąonero Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, An- drea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. 2025. Healthbench: Evaluating large language models towards improved human health. CoRR, abs/2505.08775. Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike DâArcy, David Wadden, Matt Latzke, Minyang Tian, Pan Ji, Shengyan Liu, Hao Tong, Bohao Wu, Yanyu Xiong, Luke Zettlemoyer, Graham Neubig, Daniel S. Weld, Doug Downey, Wen-tau Yih, Pang Wei Koh, and Hannaneh Hajishirzi. 2024. Openscholar: Synthesiz- ing scientific literature with retrieval-augmented lms. CoRR, abs/2411.14199. Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. CoRR. Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. 2025. Deepresearch bench: A comprehensive benchmark for deep research agents. CoRR, abs/2506.11763. Yongqi Fan, Yuxiang Chu, Zhentao Xia, Xiaoyang Chen, Jie Liu, Haijin Liang, Jin Ma, Ben He, Yingfei Sun, Dezhi Ye, and Tong Ruan. 2026. Rank4gen: Rag- preference-aligned document set selection and rank- ing. CoRR, abs/2601.11273. Jincheng Feng, Wenhan Liu, and Zhicheng Dou. 2026.Sumrank: Aligning summarization mod- els for long-document listwise reranking. CoRR, abs/2603.24204. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. 2023. Retrieval- augmented generation for large language models: A survey. CoRR, abs/2312.10997. Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. 2025. Rubrics as rewards: Reinforcement learn- ing beyond verifiable domains.arXiv preprint arXiv:2507.17746. Yun He, Wenzhe Li, Hejia Zhang, Songlin Li, Karishma Mandyam, Sopan Khosla, Yuanhao Xiong, Nan- shu Wang, Xiaoliang Peng, Beibin Li, Shengjie Bi, Shishir G. Patil, Qi Qi, Shengyu Feng, Julian Katz- Samuels, Richard Yuanzhe Pang, Sujan Gonugondla, Hunter Lang, Yue Yu, Yundi Qian, Maryam Fazel- Zarandi, Licheng Yu, Amine Benhalloum, Hany Awadalla, and Manaal Faruqui. 2025. Advancedif: Rubric-based benchmarking and reinforcement learn- ing for advancing LLM instruction following. CoRR, abs/2511.10507. Zenan Huang, Yihong Zhuang, Guoshan Lu, Zeyu Qin, Haokai Xu, Tianyu Zhao, Ru Peng, Jiaqi Hu, Zhan- ming Shen, Xiaomeng Hu, Xijun Gu, Peiyi Tu, Jiaxin Liu, Wenyu Chen, Yuzhuo Fu, Zhiting Fan, Yanmei Gu, Yuanyuan Wang, Zhengkai Yang, Jianguo Li, and Junbo Zhao. 2025. Reinforcement learning with rubric anchors. CoRR, abs/2508.12790. Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. CoRR, abs/2503.09516. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 1601â1611. Association for Computational Linguistics. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769â6781, Online. Association for Computational Linguistics. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natu- ral questions: a benchmark for question answering research. Trans. Assoc. Comput. Linguistics, 7:452â 466. Dahyun Lee, Yongrae Jo, Haeju Park, and Moontae Lee. 2025. Shifting from ranking to set selection for retrieval augmented generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 17606â17619, Vienna, Austria. Association for Computational Linguistics. Baixuan Li, Bo Zhang, Dingchu Zhang, Fei Huang, Guangyu Li, Guoxin Chen, Huifeng Yin, Jialong Wu, Jingren Zhou, Kuan Li, Liangcai Su, Litu Ou, Li- wen Zhang, Pengjun Xie, Rui Ye, Wenbiao Yin, Xin- miao Yu, Xinyu Wang, Xixi Wu, Xuanzhong Chen, Yida Zhao, Zhen Zhang, Zhengwei Tao, Zhongwang Zhang, Zile Qiao, Chenxi Wang, Donglei Yu, Gang Fu, Haiyang Shen, Jiayin Yang, Jun Lin, Junkai Zhang, Kui Zeng, Li Yang, Hailong Yin, Maojia Song, Ming Yan, Peng Xia, Qian Xiao, Rui Min, Ruixue Ding, Runnan Fang, Shaowei Chen, Shen Huang, Shihang Wang, Shihao Cai, Weizhou Shen, Xiaobin Wang, Xin Guan, Xinyu Geng, Yingcheng Shi, Yuning Wu, Zhuo Chen, Zijian Li, and Yong Jiang. 2025. Tongyi deepresearch technical report. CoRR, abs/2510.24701. Wenhan Liu, Jiajie Jin, Zhaoheng Huang, Tongyu Wen, Guanting Dong, Ziliang Zhao, Yutao Zhu, Zhicheng Dou, and Ji-Rong Wen. 2026a. The rules of the game: A survey of rubrics for large language models. Wenhan Liu, Xinyu Ma, Weiwei Sun, Yutao Zhu, Yuchen Li, Dawei Yin, and Zhicheng Dou. 2025a. Reasonrank: Empowering passage rank- ing with strong reasoning ability. arXiv preprint arXiv:2508.07050. Wenhan Liu, Xinyu Ma, Yutao Zhu, Yuchen Li, Daiting Shi, Dawei Yin, and Zhicheng Dou. 2026b. Agentic- r: Learning to retrieve for agentic search. CoRR, abs/2601.11888. Wenhan Liu, Xinyu Ma, Yutao Zhu, Lixin Su, Shuaiqiang Wang, Dawei Yin, and Zhicheng Dou. 2025b. CoRanking: Collaborative ranking with small and large ranking agents. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2025, pages 5098â5110, Suzhou, China. Association for Computational Linguistics. Wenhan Liu, Xinyu Ma, Yutao Zhu, Ziliang Zhao, Shuaiqiang Wang, Dawei Yin, and Zhicheng Dou. 2024. Sliding windows are not the end: Exploring full ranking with long-context large language models. CoRR, abs/2412.14574. Wenhan Liu, Yutao Zhu, Zhicheng Dou, and Yujia Zhou. 2026c. Demorank: Selecting effective demonstra- tions for large language models in ranking task. ACM Transactions on Information Systems, 44(3):1â25. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric mem- ories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 9802â9822. Association for Computational Linguistics. Mihran Miroyan, Tsung-Han Wu, Logan King, Tianle Li, Jiayi Pan, Xinyan Hu, Wei-Lin Chiang, Anas- tasios N. Angelopoulos, Trevor Darrell, Narges Norouzi, and Joseph E. Gonzalez. 2025. Search arena: Analyzing search-augmented llms. CoRR, abs/2506.05334. Rodrigo Frassetto Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with a pretrained sequence-to-sequence model. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, Online Event, 16-20 Novem- ber 2020, volume EMNLP 2020 of Findings of ACL, pages 708â718. Association for Computational Lin- guistics. Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. 2023a. Rankvicuna: Zero-shot listwise doc- ument reranking with open-source large language models. CoRR, abs/2309.15088. Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. 2023b.Rankzephyr: Effective and robust zero-shot listwise reranking is a breeze!CoRR, abs/2312.02724. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Compu- tational Linguistics: EMNLP 2023, Singapore, De- cember 6-10, 2023, pages 5687â5711. Association for Computational Linguistics. Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. 2024. Large language models are effec- tive text rankers with pairwise ranking prompting. In NAACL-HLT (Findings), Findings of ACL, pages 1504â1518. Association for Computational Linguis- tics. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. KDD â20. Rulin Shao, Akari Asai, Shannon Zejiang Shen, Hamish Ivison, Varsha Kishore, Jingming Zhuo, Xinran Zhao, Molly Park, Samuel G. Finlayson, David A. Sontag, Tyler Murray, Sewon Min, Pradeep Dasigi, Luca Sol- daini, Faeze Brahman, Wen-tau Yih, Tongshuang Wu, Luke Zettlemoyer, Yoon Kim, Hannaneh Ha- jishirzi, and Pang Wei Koh. 2025. DR tulu: Rein- forcement learning with evolving rubrics for deep research. CoRR, abs/2511.19399. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300. William F. Shen, Xinchi Qiu, Chenxi Whitehouse, Lisa Alazraki, Shashwat Goel, Francesco Barbieri, Timon Willi, Akhil Mathur, and Ilias Leontiadis. 2026. Re- thinking rubric generation for improving LLM judge and reward modeling for open-ended tasks. CoRR, abs/2602.05125. Zhengliang Shi, Yiqun Chen, Haitao Li, Weiwei Sun, Shiyu Ni, Yougang Lyu, Run-Ze Fan, Bowen Jin, Yixuan Weng, Minjun Zhu, Qiujie Xie, Xinyu Guo, Qu Yang, Jiayi Wu, Jujia Zhao, Xiaqiang Tang, Xinbei Ma, Cunxiang Wang, Jiaxin Mao, Qingyao Ai, Jen-tse Huang, Wenxuan Wang, Yue Zhang, Yiming Yang, Zhaopeng Tu, and Zhaochun Ren. 2025. Deep research: A systematic survey. CoRR, abs/2512.02038. Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023. Is chatgpt good at search? investigating large language models as re-ranking agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, EMNLP 2023, Singapore, December 6-10, 2023, pages 14918â14937. Association for Computational Linguistics. Liang Wang, Nan Yang, Xiaolong Huang, Binx- ing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. CoRR, abs/2212.03533. Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Linhai Zhang, Yulan He, Deyu Zhou, Pengjun Xie, and Fei Huang. 2025. Web- walker: Benchmarking llms in web traversal. In ACL (1), pages 10290â10305. Association for Computa- tional Linguistics. Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muen- nighoff, Defu Lian, and Jian-Yun Nie. 2024. C-pack: Packed resources for general chinese embeddings. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024, pages 641â649. ACM. Renjun Xu and Jingwen Peng. 2025. A comprehensive survey of deep research: Systems, methodologies, and applications. CoRR, abs/2506.12594. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Ben- gio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answer- ing. In Proceedings of the 2018 Conference on Em- pirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 2369â2380. Association for Computational Linguistics. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. Li S. Yifei, Allen Chang, Chaitanya Malaviya, and Mark Yatskar. 2025. Researchqa: Evaluating schol- arly question answering at scale across 75 fields with survey-mined questions and rubrics. CoRR, abs/2509.00496. Hengran Zhang, Keping Bi, Jiafeng Guo, Jiaming Zhang, Shuaiqiang Wang, Dawei Yin, and Xueqi Cheng. 2025. Distilling a small utility-based pas- sage selector to enhance retrieval-augmented gen- eration. In Proceedings of the 2025 Annual Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, pages 22â30. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, and Yongqiang Ma. 2024. Llamafac- tory: Unified efficient fine-tuning of 100+ language models. CoRR, abs/2403.13372. Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni, Xuanhui Wang, and Michael Bendersky. 2023. Rankt5: Fine-tuning T5 for text ranking with ranking losses. In SIGIR, pages 2308â2313. ACM. A Details of Deep Research Benchmarks We provide brief descriptions of the deep research benchmarks used in our evaluation. HealthBench. HealthBench (Arora et al., 2025) is a healthcare-oriented benchmark that evaluates whether models can answer medically related user questions in a helpful, accurate, and safe manner. Its queries often require careful interpretation of user intent and reliable synthesis of evidence, mak- ing it suitable for evaluating long-form responses in high-stakes domains. We evaluate HealthBench with an adapted OpenAI simple-evals pipeline 6 : each multi-turn case is flattened into a single doctorâ patient conversation, and the model is instructed to answer based on this conversation. We use GPT- 4.1 as the LLM judge. For efficiency, we randomly sample 100 cases for evaluation. WebWalkerQA. WebWalkerQA (Wu et al., 2025) focuses on complex web and website-level question answering. Solving its questions requires agents to search, browse, and connect information distributed across web pages, rather than relying on a single retrieved passage. We evaluate answers with GPT-4.1 as the LLM judge under a unified evaluation pipeline, using a sampled test set of 200 queries. DeepResearchBench.DeepResearchBench (Du et al., 2025) evaluates general-domain deep re- search capabilities using open-ended questions. The benchmark emphasizes qualities such as com- prehensiveness, depth of analysis, instruction fol- lowing, and readability, which are assessed through rubric-based evaluation.We evaluate 50 En- glish and 50 Chinese questions, score the gener- ated articles on Comprehensiveness, Insight/Depth, Instruction-Following, and Readability, and report the macro average as the overall score. Following the official evaluation protocol (Du et al., 2025), we use Gemini 2.5 Flash as the judge and the Jina API to scrape URLs for evidence snippets when needed; for our system outputs, we use the URL contents collected by the corresponding search or browsing tools. ResearchQA. ResearchQA (Yifei et al., 2025) contains research-style questions that require col- lecting external evidence and producing synthe- sized long-form answers. It tests whether an agent can identify useful information, organize evidence, 6 https://github.com/openai/simple-evals and provide a coherent response to complex infor- mation needs. Following the original ResearchQA evaluation suite (Yifei et al., 2025), we report aver- age rubric scores on the 100-question subset used to evaluate deep research systems, with GPT-4.1- mini as the judge. B Details of Baselines We provide additional details of the baseline rerankers used in our experiments. B.1 Vanilla Rerankers. These rerankers are primarily designed to estimate queryâdocument relevance and improve the rank- ing of retrieved documents. BGE-Reranker-Large. 7 BGE-Reranker-Large is a cross-encoder reranker from the BGE family. It jointly encodes a query and a candidate document to produce a relevance score, and is widely used as a strong open-source reranking baseline. MonoT5(Nogueira et al., 2020). MonoT5 formulates passage reranking as a sequence-to- sequence task with T5, where the model pre- dicts relevance-oriented outputs for each queryâ document pair and uses them to score candidates. RankT5(Zhuang et al., 2023). RankT5 also builds on the T5 architecture, but is optimized with ranking-oriented objectives to produce effective pointwise relevance estimates for document rerank- ing. RankVicuna(Pradeep et al., 2023a). RankVi- cuna is an instruction-tuned listwise reranker based on Vicuna. It is trained with ranked lists gener- ated by ChatGPT, enabling it to reorder candidate documents through list-level comparison. RankZephyr(Pradeepetal.,2023b). RankZephyr is a listwise reranker distilled from GPT-4. It adopts a two-stage training strategy and applies a sliding-window strategy to handle longer candidate lists. B.2 Generation-oriented Rerankers. These rerankers are designed to optimize down- stream LLM answer generation, where the goal is not only to rank individually relevant documents but also to select a high-quality document set that improves generated answers. 7 https://huggingface.co/BAAI/ bge-reranker-large DatasetSFTRLAll Deep Research OpenScholar2,5623,6716,233 SearchArena1,6282,1723,800 GlaiveAI-Reasoning-v1-20M8801,2652,145 WebWalker-Silver8841,5162,400 RAG HotpotQA1,9413,0004,941 NQ1,9483,0004,948 All9,84314,62424,467 Table 4: Number of training queries used for SFT and RL. Deep research datasets use sampled agent queries, while RAG datasets use sampled user questions. SetR(Lee et al., 2025). SetR shifts document ranking toward set-wise passage selection for RAG. It identifies the information requirements of a query through reasoning and selects a passage set that collectively satisfies these requirements. Rank4Gen(Fan et al., 2026). Rank4Gen is a generator-aware reranker that aligns document se- lection with downstream generation quality. In- stead of optimizing only relevance, it trains the ranker with signals derived from response quality and models generator-specific preferences. C Implementation Details We provide additional implementation details of RubricRanker to facilitate reproducibility. The code is available athttps://github.com/ 8421BCD/RubricRanker. C.1 Training Query Construction We construct training queries from both deep re- search and RAG scenarios. For deep research datasets, we run Dr-Tulu-8B (Shao et al., 2025) to generate full trajectories for each original ques- tion and randomly sample two agent queries from each trajectory. For RAG datasets, we randomly sample user questions from the original training sets. Table 4 reports the number of training queries used in the SFT and RL stages. C.2 Cold-Start SFT Stage In the cold-start SFT stage, we first use GPT-5.1 to construct silver labels for rubric-based listwise reranking. The prompts for agent queries in the deep research scenario and user questions in the RAG scenario are shown in Figures 10 and 11, respectively. We then train the backbone LLM, Qwen3-8B, 8 with LlamaFactory (Zheng et al., 2024). The model input follows the same scenario- specific format, consisting of the query and candi- date documents, with additional query intent for agent queries. We set the learning rate to 5e-6, the per-GPU batch size to 1, and the number of gradi- ent accumulation steps to 8. Training is accelerated with DeepSpeed ZeRO-2 (Rasley et al., 2020) and FlashAttention-2 (Dao, 2023). We use BF16 mixed precision and train the model for 2 epochs. C.3 Rubric-Based RL Stage After cold-start SFT, we further optimize Rubri- cRanker with the GRPO reinforcement learning algorithm (Shao et al., 2024) implemented in the VERL framework 9 . During GRPO training, for each inputx, we sample a group of output sequencesG = y 1 ,y 2 ,...,y G . Each sequencey i is assigned a rewardr i , and rewards are normalized within the groupGto obtain the advantages Ë A i . The token- level objective is defined as: J GRPO (θ) = 1 |G| |G| X i=1 1 |y i | |y i | X t=1 min r i,t (θ) Ë A i,t , clip (r i,t (θ), 1â Îľ, 1 + Îľ) Ë A i,t â βD KL , r i,t (θ) = Ď Î¸ (y i,t | x,y i,<t ) Ď old (y i,t | x,y i,<t ) , D KL = D KL (Ď Î¸ âĽ Ď ref ), (4) where Îľ and β are hyperparameters. To compute set-level rubric rewards efficiently and reduce API cost, we feed all set-level rubrics for each rollout to the LLM judge in a single call and ask it to return the score for each rubric. The prompts for agent queries in the deep research sce- nario and user questions in the RAG scenario are shown in Figure 14 and Figure 15, respectively. For document-level rubric rewards, we similarly score all document-level rubrics for each document in one judge call. The corresponding prompts for deep research agent queries and RAG user ques- tions are shown in Figure 16 and Figure 17. The training batch size is set to 16, the mini- batch size is set to 8, and the number of rollouts per sample is 8. The model is trained for 150 steps on 8 NVIDIA H20 GPUs, with the learning rate set to 1e-6. 8 https://huggingface.co/Qwen/Qwen3-8B 9 https://github.com/volcengine/verl D Case Study We present two case studies to illustrate the query- specific rubrics used by RubricRanker. Table 5 shows an agent query from the deep research sce- nario, while Table 6 shows a user question from the RAG scenario. The deep research case demonstrates that our rubrics capture multi-aspect evidence needs beyond topical relevance. The generated criteria require documents to cover definitions, empirical findings, perceptual mechanisms, broader theoretical impli- cations, and source authority, all of which are im- portant for open-ended scientific questions. In con- trast, the RAG case focuses on a smaller set of highly targeted requirements: identifying the band, listing the current members, avoiding redundant or off-topic evidence, and ensuring consistency. These examples show that query-specific rubrics adapt to different information needs and provide fine-grained guidance for selecting document sets. E Use of AI Assistants We use ChatGPT to improve the presentation of this paper. 10 10 https://chatgpt.com/ Query: motion dazzle human perception LevelCategoryDescriptionWeight Set-levelRelevanceDocuments clearly define motion dazzle and distinguish it from other forms of camouflage (e.g., crypsis, background matching), including its historical use in WWI ship dazzle camouflage and its proposed function in biology as disrupting motion judgments rather than hiding objects. 4 Set-levelRelevance Documents provide empirical or review evidence on how motion dazzle affects human speed perception, including that dazzle patterns can both increase and decrease perceived speed depending on internal pattern motion, contrast, and spatial frequency, rather than claiming a single uniform effect. 5 Set-levelRelevanceDocuments address how motion dazzle impacts perceived direction and trajectory (e.g., biases toward stripe orientation, altered interception/capture performance), linking oriented patterns on moving targets to systematic misjudgments of movement direction or path. 4 Set-levelRelevance Documents explain mechanistic accounts from human motion vision that can generate motion dazzle effects, such as the aperture problem, ambiguity of local oriented edge motion, conflict between global object motion and internal texture motion, and dependence on contrast and spatial frequency. 5 Set-levelRelevanceDocuments explicitly connect motion dazzle phenomena to broader principles or heuristics of human motion perception (e.g., pooling of local motion signals, non-veridical reconstruction of 2D motion, interactions between form and motion processing), rather than only describing behavioral effects. 3 Set-levelConcisenessThe document set minimizes redundancy by avoiding multiple papers or pages that restate the same core experimental findings or definitions of motion dazzle, while still covering definition, speed effects, direction/trajectory effects, and mechanistic explanations without including tangential material (e.g., general camouflage history with no motion- perception content). 3 Set-levelConsistencyAcross the document set, key claims about motion dazzle (its definition, that it can both increase and decrease perceived speed depending on pattern parameters, and that it disrupts motion judgments rather than simple detection) are mutually compatible or, where differing results exist, the differences are clearly attributed to experimental conditions rather than left as unresolved contradictions. 4 Document-levelSource AuthorityThe document is directly relevant to motion dazzle and human motion perception and is published in or hosted by an authoritative scientific source such as a peer-reviewed vision science/biology/psychology journal, an academic publisher, or a reputable scientific repository (e.g., PubMed/PMC, major university site), providing empirical data or well-cited reviews rather than unsourced popular summaries. 4 Table 5: Case study of query-specific rubrics for an agent query in the deep research scenario. The original question is âTo what extent and how can the motion dazzle effect give significant insight into the general algorithms or heuristics that underlie general perception in humans?â Empty document-level rubrics are omitted. Query: What are the names of the current members of the American heavy metal band that wrote the music for Hurt Locker The Musical? LevelCategoryDescriptionWeight Set-levelRelevanceAt least one document explicitly identifies the American heavy metal band Metallica as the group that wrote the music for âHurt Locker The Musicalâ and clearly links the musical to Metallica. 5 Set-levelRelevanceThe document set lists the current members of Metallica by name and role, including James Hetfield, Lars Ulrich, Kirk Hammett, and Robert Trujillo, making clear that these are the present lineup members. 5 Set-levelConciseness The document set contains mostly documents that either (a) identify Metallica as the band associated with âHurt Locker The Musicalâ or (b) list Metallicaâs current members, with minimal repetition of the same member list and little content about unrelated bands, albums, or other musicals. 3 Set-levelConsistencyAcross the document set, the identified band and its current members are consistent, without conflicting claims about which band wrote the music for âHurt Locker The Musicalâ or disagreements over the current Metallica lineup (e.g., all agree on Hetfield, Ulrich, Hammett, and Trujillo and do not present alternative current members). 4 Table 6: Case study of query-specific rubrics for a user question in the RAG scenario. Empty document-level rubrics are omitted. You are a research assistant who answers query through reasoning and web search. Your goal is to produce a high-quality answer for a given query, using both the query itself and the provided query intent. Important instruction about the query intent: The query intent includes some reasoning process before issuing the query. It may contain extra reasoning, exploratory thoughts, or content that is not directly relevant to the query. You must identify and use only the parts of the query intent that are actually relevant to this query. Requirements: - First infer the true information need of the query by jointly reading: 1. the query 2. the query intent - The answer should contain key information from the retrieved supporting documents required to address the query, especially the different aspects explicitly mentioned in the query or strongly implied by the relevant part of the query intent. While covering the key information, keep the answer as concise as possible and stay at an appropriate level of depth for the query. Do not go unnecessarily deep into any aspect unless the query explicitly requires it. Do not over-expand. Do not turn the answer into a broad survey or long report. - Structure the answer as multiple paragraphs or sections when appropriate, where each paragraph or section corresponds to a different aspect needed to answer the query. Give each paragraph or section a short markdown header using exactly level-3 headers (###). - Prefer authoritative sources, and prioritize recent sources when the query explicitly requires a time range or implicitly requires the most up-to-date information. - The total answer should not exceed 2000 tokens. Below is an example of an input and the corresponding answer output. Please follow the output format of the example answer. Example: [QUERY] motion dazzle human perception [QUERY INTENT] The core questions that need to be answered include: What is motion dazzle (definition, mechanisms, classic cases)? How exactly does this effect influence human perception? Search plan: First, search broadly for âmotion dazzle human perception algorithms heuristicsâ to find review or integrative sources. Then drill down into mechanisms, such as speed underestimation/overestimation, direction ambiguity, motion integration in noisy textures, neural evidence (e.g. MT/V1), and motion streaks... First query: motion dazzle human perception [ANSWER] ### What is motion dazzle? Motion dazzle refers to high-contrast surface patterns (often stripes, zigzags, or bold geometric patches) that do not hide an object but instead disrupt an observerâs ability to judge its motionâtypically its speed, direction, or trajectory. The term comes from WWI âdazzleâ ship paint schemes and has since been applied to animal coloration (e.g. zebras, some fish, insects, snakes). ([pmc.ncbi.nlm.nih.gov](https://pmc.ncbi.nlm.nih.gov/articles/PMC3871368/?utm_source=openai)) ### How does motion dazzle affect human motion perception?... Next comes the input. Inputs: [QUERY] query [QUERY INTENT] query_intent Now write the answer. Answer Generation for Agent Sub-query Figure 5: The prompt for generating an answer to an agent sub-query. The input consists of the query and query intent (agent reasoning). You are designing hierarchical search rubrics for evaluating the quality of a document set for answering a query. Your task is to generate query-specific rubrics under a fixed meta-rubric framework. You will be given: 1. The query. 2. The query intent, which clarifies the true information need behind the query. 3. A reference answer. Note that the query intent may contain irrelevant, broader, or overly expanded thoughts. So use the query as the primary anchor. The query intent is only for clarifying the real goal, scope, constraints, and expected answer shape of this query. Stay tightly focused on the query itself. The answer often contains key information across different aspects needed to address the query, which is important for generating rubrics. Your job: - First analyze the query, the query intent, and the reference answer. - Then, generate query-specific rubrics under each meta rubric. If a meta rubric (such as Source Authority and Timeliness) is not needed for this query, output an empty list [] for that meta rubric. - Each query-specific rubric must have an integer weight from 1 to 5. Weight scale: - 1 = marginally useful; minor added value only - 2 = useful but secondary - 3 = moderately important; clearly helpful for evaluating a good document set - 4 = very important; strong effect on document set quality - 5 = critical; if missing or violated, the document set is seriously inadequate for answering the query Fixed meta-rubric framework: Set-level meta rubrics: 1. Relevance Explanation: Whether the document set covers the information needs required to answer the query, and in particular whether it covers the key information mentioned across the different paragraphs of the reference answer. Each distinct paragraph in the reference answer typically corresponds to a need aspect that the document set should support with evidence. Count limit: You may generate multiple query-specific rubrics here, up to a maximum of 7. 2. Conciseness Explanation: Whether the documents in the set have low redundancy with each other (i.e., different documents do not repeat the same information heavily) and the set contains few documents irrelevant to the query. Count limit: Generate exactly one query-specific rubric under this meta rubric. 3. Consistency Explanation: Whether the key facts, claims, and conclusions used to answer the query across the document set are mutually compatible rather than contradictory. A high-quality set should not contain serious unresolved conflicts on important points; if multiple viewpoints exist, they should be clearly distinguishable rather than mixed into confusion. Count limit: Generate exactly one query-specific rubric under this meta rubric. Doc-level meta rubrics: 4. Source Authority Explanation: This meta rubric evaluates whether a single document is relevant to the query, provides evidence for information needed to answer it, and comes from an authoritative and trustworthy website. It should only be triggered when the query clearly belongs to a domain where source authority is critical, such as medicine, law, finance, policy. Count limit: At most one query-specific rubric under this meta rubric. 5. Timeliness Explanation: This meta rubric evaluates whether a single document is relevant to the query, provides evidence for information needed to answer it, and satisfies the temporal requirements of the query. It should only be triggered when the query explicitly contains a time constraint or clearly requires the latest / most up-to-date information Count limit: At most one query-specific rubric under this meta rubric. Query-specific Rubrics Generation for Agent Sub-query (Part I) Figure 6: The prompt for generating query-specific rubrics for an agent sub-query (Part I). The input consists of the query, query intent (agent reasoning), and reference answer. Important constraints: - Each query-specific rubric must be written specifically according to the content of the query and the reference answer, especially the key aspects for answering the query. Do not write broad, generic, reusable, or template-like rubrics that could apply equally well to other queries. - Do not create duplicate rubrics under the same meta rubric. - Strictly respect the per-meta-rubric count limits stated above. Output format: Return JSON only. Use exactly the following top-level structure and exactly the following meta-rubric names: "set_level_rubrics": "Relevance": ["description": "...", "weight": ..., "description": "...", "weight": ...], "Conciseness": ["description": "...", "weight": ...], "Consistency": ["description": "...", "weight": ...] , "doc_level_rubrics": "Source Authority": ["description": "...", "weight": ...], "Timeliness": ["description": "...", "weight": ...] For each meta-rubric key, the value must be a list of objects. Each object must contain exactly these two fields: - "description": a concise query-specific rubric description - "weight": an integer from 1 to 5 If a meta rubric (i.e., Source Authority and Timeliness) is not needed for this query, output an empty list [] for that meta rubric. Do not output any other fields. Now use the following inputs. [QUERY] query [QUERY INTENT] query_intent [REFERENCE ANSWER] reference_answer Generate the JSON now. Query-specific Rubrics Generation for Agent Sub-query (Part I) Figure 7: The prompt for generating query-specific rubrics for an agent sub-query (Part I). The input consists of the query, query intent (agent reasoning), and reference answer. You are designing hierarchical search rubrics for evaluating the quality of a document set for answering a query. Your task is to generate query-specific rubrics under a fixed meta-rubric framework. You will be given: 1. The query. 2. A reference answer. Your job: - First analyze the query, the query intent, and the reference answer. - Then, generate query-specific rubrics under each meta rubric. If a meta rubric (such as Source Authority and Timeliness) is not needed for this query, output an empty list [] for that meta rubric. - Each query-specific rubric must have an integer weight from 1 to 5. Weight scale: - 1 = marginally useful; minor added value only - 2 = useful but secondary - 3 = moderately important; clearly helpful for evaluating a good document set - 4 = very important; strong effect on document set quality - 5 = critical; if missing or violated, the document set is seriously inadequate for answering the query Fixed meta-rubric framework: Set-level meta rubrics: 1. Relevance Explanation: Whether the document set covers the information needs required to answer the query, and in particular whether it covers the key information mentioned across the different paragraphs of the reference answer. Each distinct paragraph in the reference answer typically corresponds to a need aspect that the document set should support with evidence. Count limit: You may generate multiple query-specific rubrics here, up to a maximum of 7. 2. Conciseness Explanation: Whether the documents in the set have low redundancy with each other (i.e., different documents do not repeat the same information heavily) and the set contains few documents irrelevant to the query. Count limit: Generate exactly one query-specific rubric under this meta rubric. 3. Consistency Explanation: Whether the key facts, claims, and conclusions used to answer the query across the document set are mutually compatible rather than contradictory. A high-quality set should not contain serious unresolved conflicts on important points; if multiple viewpoints exist, they should be clearly distinguishable rather than mixed into confusion. Count limit: Generate exactly one query-specific rubric under this meta rubric. Doc-level meta rubrics: 4. Source Authority Explanation: This meta rubric evaluates whether a single document is relevant to the query, provides evidence for information needed to answer it, and comes from an authoritative and trustworthy website. It should only be triggered when the query clearly belongs to a domain where source authority is critical, such as medicine, law, finance, policy. Count limit: At most one query-specific rubric under this meta rubric. 5. Timeliness Explanation: This meta rubric evaluates whether a single document is relevant to the query, provides evidence for information needed to answer it, and satisfies the temporal requirements of the query. It should only be triggered when the query explicitly contains a time constraint or clearly requires the latest / most up-to-date information Count limit: At most one query-specific rubric under this meta rubric. Query-specific Rubrics Generation for User Question in RAG (Part I) Figure 8: The prompt for generating query-specific rubrics for a user question in the RAG scenario (Part I). The input consists of the query and gold answers. Important constraints: - Each query-specific rubric must be written specifically according to the content of the query and the reference answer, especially the key aspects for answering the query. Do not write broad, generic, reusable, or template-like rubrics that could apply equally well to other queries. - Do not create duplicate rubrics under the same meta rubric. - Strictly respect the per-meta-rubric count limits stated above. Output format: Return JSON only. Use exactly the following top-level structure and exactly the following meta-rubric names: "set_level_rubrics": "Relevance": ["description": "...", "weight": ..., "description": "...", "weight": ...], "Conciseness": ["description": "...", "weight": ...], "Consistency": ["description": "...", "weight": ...] , "doc_level_rubrics": "Source Authority": ["description": "...", "weight": ...], "Timeliness": ["description": "...", "weight": ...] For each meta-rubric key, the value must be a list of objects. Each object must contain exactly these two fields: - "description": a concise query-specific rubric description - "weight": an integer from 1 to 5 If a meta rubric (i.e., Source Authority and Timeliness) is not needed for this query, output an empty list [] for that meta rubric. Do not output any other fields. Now use the following inputs. [QUERY] query [Golden ANSWERS] golden_answers Generate the JSON now. Query-specific Rubrics Generation for User Question in RAG (Part I) Figure 9: The prompt for generating query-specific rubrics for a user question in the RAG scenario (Part I). The input consists of the query and gold answers. You are selecting the best subset of retrieved web documents for a search query, guided by hierarchical evaluation rubrics. You will be given: 1. The search query. 2. The query intent, which is the agent's reasoning process before issuing the query and may clarify why this query was issued and what information it is trying to find. The query intent may also contain broader, exploratory, or irrelevant thoughts; stay anchored on the query itself and only use the parts of the query intent that are actually relevant to this query. 3. The hierarchical rubrics used to judge document-set quality. These are split into: - **Set-level rubrics** (Relevance / Conciseness / Consistency): judge the whole selected set jointly. * **Relevance is the primary objective**. The document set you select should satisfy these relevance rubrics as much as possibleâparticularly those with high weights. * Conciseness and Consistency are **tie-breakers / penalties**, not primary objectives. Only use them to choose among subsets with comparable Relevance coverage. - **Doc-level rubrics** (Source Authority / Timeliness): judge each individual document. Doc-level rubrics may be empty; if so, ignore them. * Doc-level rubrics are **prerequisites, not rewards**. A document must first contribute to at least one Relevance rubric to be considered. **Never include a document solely because it looks authoritative or up-to-date.** Each rubric has a description and an integer weight from 1 (marginal) to 5 (critical). The higher the weight, the more important it is. 4. A list of num_docs retrieved web documents (with title, URL, search snippet and full text), each prefixed by a numerical identifier like [1], [2], .... Your job: 1. Read the query and the relevant part of the query intent to infer the actual information need. 2. Read the rubrics, which evaluate the quality of a document set. 3. Select at most top_n documents from the retrieved web documents so that the document set satisfies the Set-level Rubrics and Doc-level Rubrics as much as possible, especially high- weight rubrics. 4. After selection, rank the selected documents from MOST to LEAST useful for answering the query. Output format (STRICT): Return ONLY the selected document identifiers in order, each wrapped in square brackets and separated by a single space. No other text, no reasoning, no explanation. Example: [4] [2] [7] [11] [1] Inputs: [QUERY] query [QUERY INTENT] query_intent [RUBRICS] rubrics [CANDIDATE DOCUMENTS] formatted_docs Now output the selected and ranked document identifiers (at most top_n). Prompt of Rubrics-based Listwise Ranking for Agent Query Figure 10: The prompt for constructing silver labels for an agent sub-query in the deep research scenario. The input consists of the query, query intent (agent reasoning), candidate documents, and query-specific rubrics. You are selecting the best subset of retrieved web documents for a user question, guided by hierarchical evaluation rubrics. You will be given: 1. The user question. 2. The hierarchical rubrics used to judge document-set quality. These are split into: - **Set-level rubrics** (Relevance / Conciseness / Consistency): judge the whole selected set jointly. * **Relevance is the primary objective**. The document set you select should satisfy these relevance rubrics as much as possibleâparticularly those with high weights. * Conciseness and Consistency are **tie-breakers / penalties**, not primary objectives. Only use them to choose among subsets with comparable Relevance coverage. - **Doc-level rubrics** (Source Authority / Timeliness): judge each individual document. Doc-level rubrics may be empty; if so, ignore them. * Doc-level rubrics are **prerequisites, not rewards**. A document must first contribute to at least one Relevance rubric to be considered. **Never include a document solely because it looks authoritative or up-to-date.** Each rubric has a description and an integer weight from 1 (marginal) to 5 (critical). The higher the weight, the more important it is. 3. A list of num_docs retrieved passages, each prefixed by a numerical identifier like [1], [2], .... Your job: 1. Read the user question to infer the actual information need. 2. Read the rubrics, which evaluate the quality of a document set. 3. Select at most top_n documents from the retrieved web documents so that the document set satisfies the Set-level Rubrics and Doc-level Rubrics as much as possible, especially high- weight rubrics. The selected set should cover the important information needs, avoid unnecessary redundancy, and prefer documents that satisfy doc-level rubrics (if provided). If no rubrics are provided, please select documents based on their relevance to the user question. 4. After selection, rank the selected documents from MOST to LEAST useful for answering the user question. Output format (STRICT): Return ONLY the selected document identifiers in order, each wrapped in square brackets and separated by a single space. No other text, no reasoning, no explanation. Example: [4] [2] [7] [11] [1] Inputs: [QUERY] query [RUBRICS] rubrics [CANDIDATE DOCUMENTS] formatted_docs Now output the selected and ranked document identifiers (at most top_n). Prompt of Rubrics-based Listwise Ranking for User Question in RAG Figure 11: The prompt for constructing silver labels for a user question in the RAG scenario. The input consists of the user question, candidate documents, and query-specific rubrics. You are a high-quality document set selector for a deep-research agent. Given a search query from the agent, the agent's reasoning as query intent, and a list of retrieved documents (including title, url, search snippet and full text), select the best subset that jointly answers the query. The query intent may clarify why this query was issued and what information it is trying to find, but it may also contain broader, exploratory, or irrelevant thoughts. Stay anchored on the query itself and only use the parts of the query intent that are actually relevant to this query. Use these 5 rubrics to judge a candidate document set: 1. Relevance (set-level, primary objective): the selected set should jointly cover the key information needed to satisfy the query, in light of the query intent. 2. Conciseness (set-level): avoid redundant documents that repeat information already covered by another selected document. 3. Consistency (set-level): prefer sets whose documents agree on facts; avoid mixing in documents that contradict the rest without good reason. 4. Source Authority (doc-level): only consider this when the query clearly belongs to a high- stakes or expertise-sensitive domain such as medicine, law, finance, policy, government regulation, or science. For general, everyday, entertainment, or low-stakes informational queries, ignore this rubric. When it does apply, prefer documents from authoritative or trustworthy sources, but only among those that already contribute to Relevance. 5. Timeliness (doc-level): only consider this when the query explicitly contains a time constraint (a specific year, date, or time range) or clearly requires the latest / most up-to- date information. For queries with no temporal requirement, ignore this rubric. When it does apply, prefer recent / up-to-date documents, but only among those that already contribute to Relevance. Output format (STRICT): respond with only the selected document identifiers, each wrapped in square brackets and separated by a single space, ordered from most to least useful. Example: [4] [2] [7] [11] [1]. Do not output any other text or explanation. I will provide you with num documents, each indicated by a numerical identifier []. Select and rank the documents based on their joint usefulness for the search query: question. [QUERY INTENT] query_intent Search Query: question. [DOCUMENTS] context Select the best subset of the num documents above so that, judged by the 5 rubrics, the selected set best answers the query (anchored on the query, with the query intent only as clarification). Then rank the selected documents from most to least useful. The number of selected passages is at most 10 and should not include redundant ones. The output must be only the selected identifiers in ranked order, e.g. [2] [1] [4]. Only respond with the selection; do not say any other word and do not explain. Prompt of RubricRanker for Agent Query Figure 12: The RubricRanker prompt for an agent sub-query in the deep research scenario. The input consists of the agent query, query intent (agent reasoning), and candidate documents. You are a high-quality document set selector for a retrieval-augmented QA system. Given a user question and a list of retrieved passages, select the best subset that jointly answers the question, then rank the selected passages from most to least useful. Use these 5 rubrics to judge a candidate document set: 1. Relevance (set-level, primary objective): the selected set should jointly cover the key information needed to answer the user question. 2. Conciseness (set-level): avoid redundant passages that repeat information already covered by another selected passage. 3. Consistency (set-level): prefer sets whose passages agree on facts; avoid mixing in passages that contradict the rest without good reason. 4. Source Authority (doc-level): only consider this when the question clearly belongs to a high-stakes or expertise-sensitive domain such as medicine, law, finance, policy, government regulation, or science. For general, everyday, entertainment, or low-stakes informational questions, ignore this rubric. When it does apply, prefer passages from authoritative or trustworthy sources, but only among those that already contribute to Relevance. 5. Timeliness (doc-level): only consider this when the question explicitly contains a time constraint (a specific year, date, or time range) or clearly requires the latest / most up-to- date information. For questions with no temporal requirement, ignore this rubric. When it does apply, prefer recent / up-to-date passages, but only among those that already contribute to Relevance. Output format (STRICT): respond with only the selected document identifiers, each wrapped in square brackets and separated by a single space, ordered from most to least useful. Example: [4] [2] [7] [11] [1]. Do not output any other text or explanation. I will provide you with num passages, each indicated by a numerical identifier []. Select and rank the passages based on their joint usefulness for the user question: question. [PASSAGES] context User Question: question. Select the best subset of the num passages above so that, judged by the 5 rubrics, the selected set best answers the question. Then rank the selected passages from most to least useful. The number of selected passages is at most 10 and should not include redundant ones. The output must be only the selected identifiers in ranked order, e.g. [2] [1] [4]. Only respond with the selection; do not say any other word and do not explain. Prompt of RubricRanker for User Question in RAG Figure 13: The RubricRanker prompt for a user question in the RAG scenario. The input consists of the user question and candidate documents. You are a strict evaluator. Given a search query issued by a deep-research agent, the agent's reasoning as query intent, a SET of retrieved candidate documents, and one or more scoring rubrics, give each rubric an INTEGER score from 0 (worst) to 10 (best) for a document set. The query intent may clarify why this query was issued and what information it is trying to find, but it may also contain broader, exploratory, or irrelevant thoughts. Stay anchored on the query itself and only use the parts of the query intent that are actually relevant. Scoring rules: - Judge the document set strictly against each rubric's description. Each rubric is independent. - Higher score = the set better matches the rubric description; lower score = worse match. - The set is judged as a whole (e.g., does the combined set cover what the rubric asks for.). Output format (STRICT): output ONLY the integer scores in the SAME ORDER as the rubrics listed below, separated by a single space. No other text, no explanation, no labels. Example (when 3 rubrics are listed): 7 3 5 Example (when 1 rubric is listed): 5 [QUERY] query [QUERY INTENT] query_intent [SCORING RUBRICS] rubrics_block [CANDIDATE DOCUMENT SET] doc_set Now output the scores. Prompt of Setwise Rubric Reward for Agent Query Figure 14: The prompt for set-level reward scoring for an agent sub-query in the deep research scenario. The input consists of the query, query intent (agent reasoning), the selected document set, and all set-level rubrics. You are a strict evaluator. Given a user question, a SET of retrieved candidate documents, and one or more scoring rubrics, give each rubric an INTEGER score from 0 (worst) to 10 (best) for a document set. Scoring rules: - Judge the document set strictly against each rubric's description. Each rubric is independent. - Higher score = the set better matches the rubric description; lower score = worse match. - The set is judged as a whole (e.g., does the combined set cover what the rubric asks for.). Output format (STRICT): output ONLY the integer scores in the SAME ORDER as the rubrics listed below, separated by a single space. No other text, no explanation, no labels. Example (when 3 rubrics are listed): 7 3 5 Example (when 1 rubric is listed): 5 [QUERY] query [SCORING RUBRICS] rubrics_block [CANDIDATE DOCUMENT SET] doc_set Now output the scores. Prompt of Setwise Rubric Reward for User Question in RAG Figure 15: The prompt for set-level reward scoring for a user question in the RAG scenario. The input consists of the user question, the selected document set, and all set-level rubrics. You are a strict document grader. Given a search query issued by a deep-research agent, the agent's reasoning as query intent, a single retrieved candidate document, and one or more scoring rubrics, give each rubric an INTEGER score from 0 (worst) to 10 (best) for this document. The query intent may clarify why this query was issued and what information it is trying to find, but it may also contain broader, exploratory, or irrelevant thoughts. Stay anchored on the query itself and only use the parts of the query intent that are actually relevant. Scoring rules: - Judge the document strictly against each rubric's description. Each rubric is independent. - Higher score = the document better matches the rubric description; lower score = worse match. Output format (STRICT): output ONLY the integer scores in the SAME ORDER as the rubrics listed below, separated by a single space. No other text, no explanation, no labels. Example (when 2 rubrics are listed): 7 3 Example (when 1 rubric is listed): 5 [QUERY] query [QUERY INTENT] query_intent [SCORING RUBRICS] rubrics_block [CANDIDATE DOCUMENT] doc_content Now output the scores. Prompt of Doc-level Rubric Reward for Agent Query Figure 16: The prompt for document-level reward scoring for an agent sub-query in the deep research scenario. The input consists of the query, query intent (agent reasoning), a candidate document, and all document-level rubrics. You are a strict document grader. Given a user question, a single retrieved candidate document, and one or more scoring rubrics, give each rubric an INTEGER score from 0 (worst) to 10 (best) for this document. Scoring rules: - Judge the document strictly against each rubric's description. Each rubric is independent. - Higher score = the document better matches the rubric description; lower score = worse match. - If the document is unreadable, off-topic, or provides no signal for a rubric, give 0 for that rubric. Output format (STRICT): output ONLY the integer scores in the SAME ORDER as the rubrics listed below, separated by a single space. No other text, no explanation, no labels. Example (when 2 rubrics are listed): 7 3 Example (when 1 rubric is listed): 5 [QUERY] query [SCORING RUBRICS] rubrics_block [CANDIDATE DOCUMENT] doc_content Now output the scores. Prompt of Doc-level Rubric Reward for User Question in RAG Figure 17: The prompt for document-level reward scoring for a user question in the RAG scenario. The input consists of the user question, a candidate document, and all document-level rubrics.