Paper deep dive
ScholarQuest: A Taxonomy-Guided Benchmark for Agentic Academic Paper Search in Open Literature Environments
Tingyue Pan, Mingyue Cheng, Daoyu Wang, Yitong Zhou, Jie Ouyang, Qi Liu, Enhong Chen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 6/21/2026, 5:17:20 AM
Summary
ScholarQuest is a large-scale, taxonomy-guided benchmark designed to evaluate agentic academic paper search systems in open literature environments. It utilizes over 1,000 computer science topics from the ACM Computing Classification System and four distinct research intents (method-oriented, setting-anchored, comparison-based, and scope-controlled) to provide a systematic evaluation. The benchmark includes ScholarBase, a million-scale retrieval backend based on arXiv, to ensure reproducible evaluation. Experimental results demonstrate that LLM-based agentic methods, such as PaperScout, outperform traditional single-shot retrieval baselines, although significant room for improvement remains in terms of recall and efficiency.
Entities (7)
Relation Signals (4)
ScholarBase → isbasedon → arXiv
confidence 100% · We further build a million-scale testbed ScholarBase based on the arXiv database.
PaperScout → isevaluatedon → ScholarQuest
confidence 100% · We benchmark representative search systems... [including] PaperScout
ScholarQuest → isguidedby → ACM Computing Classification System
confidence 100% · ScholarQuest is constructed from over 1,000 computer science topics... from the ACM Computing Classification System (CCS)
ScholarQuest → usesbackend → ScholarBase
confidence 100% · It further provides scalable answer construction and a shared retrieval backend ScholarBase
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Academic paper search is a core step in scientific research, and LLM-based search agents are emerging as a promising paradigm for iterative, intent-driven literature exploration. However, existing benchmarks are insufficient for systematically evaluating agentic academic search under realistic open literature environments. We propose ScholarQuest, a large-scale, taxonomy-guided benchmark for agentic academic paper search. ScholarQuest is constructed from over 1,000 computer science topics and four representative research intents, including method-oriented, setting-anchored, comparison-based, and scope-controlled queries. It further provides scalable answer construction and a shared retrieval backend ScholarBase for reproducible evaluation. Benchmarking results show that agentic methods outperform single-shot retrieval baselines, yet the best-performing agent only achieves 0.314 Recall@100 and 0.355 Recall@All, indicating substantial room for improvement. In addition, analyses of search efficiency, intent-level robustness, and failure cases further highlight the benchmark's ability to provide multi-dimensional evaluation signals for academic paper search agents.
Tags
Links
- Source: https://arxiv.org/abs/2606.20235v1
- Canonical: https://arxiv.org/abs/2606.20235v1
Trouble viewing inline? Open PDF directly →
Full Text
68,880 characters extracted from source content.
Expand or collapse full text
ScholarQuest: A Taxonomy-Guided Benchmark for Agentic Academic Paper Search in Open Literature Environments Tingyue Pan, Mingyue Cheng ∗ , Daoyu Wang, Yitong Zhou, Jie Ouyang, Qi Liu, Enhong Chen State Key Lab of Cognitive Intelligence, University of Science and Technology of China pty12345,wdy030428,yitong.zhou,ouyang_jie@mail.ustc.edu.cn mycheng,qiliuql,cheneh@ustc.edu.cn Abstract Academic paper search is a core step in scien- tific research, and LLM-based search agents are emerging as a promising paradigm for iterative, intent-driven literature exploration. However, existing benchmarks are insufficient for system- atically evaluating agentic academic search un- der realistic open literature environments. We propose ScholarQuest, a large-scale, taxonomy- guided benchmark for agentic academic paper search. ScholarQuest is constructed from over 1,000 computer science topics and four repre- sentative research intents, including method- oriented, setting-anchored, comparison-based, and scope-controlled queries. It further pro- vides scalable answer construction and a shared retrieval backend ScholarBase for reproducible evaluation. Benchmarking results show that agentic methods outperform single-shot re- trieval baselines, yet the best-performing agent only achieves 0.314 Recall@100 and 0.355 Recall@All, indicating substantial room for improvement. In addition, analyses of search efficiency, intent-level robustness, and failure cases further highlight the benchmark’s ability to provide multi-dimensional evaluation sig- nals for academic paper search agents. Our code and data are publicly available 1 . 1 Introduction Academic paper search underpins effective knowl- edge discovery in scientific research (Timmins and McCabe, 2005; Marchionini, 2006). Tradi- tional approaches mainly rely on lexical or seman- tic matching, offering efficient access to large-scale literature collections (Shi et al., 2025). However, when facing fine-grained and conditional queries, they remain limited by the single-shot ranked-list paradigm (Gusenbauer and Haddaway, 2020). This motivates the rise of agentic paper search, where LLM-based agents can autonomously decide when ∗ Corresponding author. 1 https://github.com/pty12345/ScholarQuest Similarity-based Search Agentic Search Papers Ranking 1 st 2 nd 3 rd Papers 1 st 2 nd 3 rd Expand Search Fails to capture complex intent Rarely use citation relationships More relevant and richer coverage Closer to real research workflow (a) From Similarity Search to Agentic Search (b) We Need Better Evaluation for Agentic Search Existing BenchmarkScholarQuest Principles Query side •Human bias •Limited coverage Answer side •Hard to scale high-quality answer •Lack of standard evaluation Taxonomy-guided queries Diverse domains and intents Standard evaluation Figure 1: Motivation of ScholarQuest. Agentic paper search moves beyond similarity-based retrieval through iterative exploration. This calls for better benchmarks with controlled queries and standardized evaluation. and how to search, expand, and refine candidate papers based on accumulated evidence (He et al., 2025; Pan et al., 2026). Despite this progress, the systematic evaluation of agentic academic paper search remains underde- veloped. Existing benchmarks are mainly limited in three aspects. On the query side, they often rely on manually constructed or paper-derived queries, which may introduce annotator bias and provide limited coverage of diverse research intents (He et al., 2025; Shi et al., 2025). On the answer side, constructing high-quality relevant paper sets is costly and difficult to scale, since relevant papers are often scattered across different terminologies and subfields (He et al., 2025). Moreover, existing benchmarks do not provide a publicly available and standardized evaluation environment, making tool- calling results difficult to reproduce across different systems. As shown in Table 1, these limitations make existing benchmarks insufficient for evaluat- ing agentic paper search under broad topics, diverse query types, and reproducible environments. To address these challenges, we propose Schol- arXiv:2606.20235v1 [cs.IR] 18 Jun 2026 Table 1: Comparison with existing academic paper search benchmarks. ScholarQuest differs in broader topic coverage, taxonomy-guided queries, controlled intents, and automatically built answers. BenchmarkTopic ScopeQuery SourceConstructionQuery Intent DesignAnswer Source RealScholar (He et al., 2025)CV / NLPHumanQuery-firstUncontrolledHuman-labeled AutoScholar (He et al., 2025)Seed PapersAnswer-derivedAnswer-firstUncontrolledRelated Work SPARBench (Shi et al., 2025)AI / NLPHumanQuery-firstUncontrolledHuman-labeled ScholarQuest1,000+ CS topicsTaxonomy-guidedQuery-first4 intent typesAutomatically-built arQuest, a large-scale, taxonomy-guided evalua- tion benchmark for agentic academic paper search. ScholarQuest covers over 1,000 computer science topics and includes four common types of re- search queries: method-oriented, setting-anchored, comparison-based, and scope-controlled queries. These query types reflect common scientific search needs, such as finding papers using a specific method, studying a problem under a given set- ting, comparing technical claims, or restricting the search scope. This taxonomy-guided design gives ScholarQuest broad topic coverage and controlled query diversity beyond ad-hoc query collection. ScholarQuest also provides scalable answer con- struction and a standardized literature environment. To build high-quality answer sets, we develop an automated pipeline that combines initial retrieval, citation expansion, multi-stage relevance filtering, and quality verification. This pipeline aims to im- prove both coverage and precision while reducing the cost of large-scale manual annotation. To sup- port reproducible evaluation, we further build a million-scale testbed ScholarBase based on the arXiv database. Beyond storing paper metadata and citation relations, ScholarBase provides uni- fied retrieval, inspection, and citation-expansion in- terfaces, enabling different systems to be evaluated under the same controlled literature environment. Built on ScholarBase, ScholarQuest evaluates academic search systems beyond final retrieval quality, covering Recall@k, search efficiency, tool- use behavior, and robustness across research in- tents and answer-set sizes. Experiments show that agentic methods outperform single-shot retrievers, but still face limitations in efficiency, constraint- sensitive search, and robustness under different query conditions. Failure analysis further shows that many errors stem from off-target exploration rather than insufficient search effort. These find- ings highlight the need to evaluate not only what papers are retrieved, but also how the search pro- cess is conducted. We release the benchmark and our retrieval backend ScholarBase to support re- producible research on academic paper search. Our contributions are summarized as follows: • We propose ScholarQuest, a large-scale, taxonomy-guided benchmark for agentic aca- demic paper search, covering over 1,000 CS topics, four research intents, and a million- scale paper retrieval backend ScholarBase. • We build a scalable construction pipeline that integrates taxonomy-guided query generation, multi-source retrieval, citation expansion, rel- evance filtering, and quality verification. •We benchmark representative search systems and identify key limitations of current agents in search efficiency, scope control, and robust- ness across query conditions. 2 Related Work 2.1 Academic Paper Search Benchmarks Academic paper search is essential for both hu- man researchers and deep-search agents that rely on trustworthy scholarly evidence (Wang et al., 2025; Cheng et al., 2026; Jin et al., 2025). Recent benchmarks have begun to evaluate LLM-based systems for academic paper search. PaSa intro- duces AutoScholarQuery, a large synthetic dataset constructed from top-tier AI conference publica- tions, and RealScholarQuery, a small set of real- world academic queries for realistic evaluation (He et al., 2025). AutoScholarQuery provides useful scale, but its construction is tied to source papers and their Related Work section, which may inherit topic and citation biases. RealScholarQuery im- proves realism, but its limited size makes it diffi- cult to systematically cover fine-grained CS topics and diverse query intents. SPARBench further im- proves annotation quality through expert-screened queries and multi-source relevance validation, but it remains small-scale with broad domain coverage rather than a systematic CS topic hierarchy (Shi et al., 2025). In contrast, our benchmark offers taxonomy-grounded topic coverage, controllable query distributions, and broader gold answer pools. 2.2 LLM-based Paper Search Agents LLM-based methods have recently been explored for academic paper search. Agentic workflow- based approaches, such as PaSa, formulate pa- per search as a multi-step process that searches, inspects, and selects relevant papers (He et al., 2025). SPAR further integrates query understand- ing, multi-source retrieval, citation-based explo- ration, and reranking for training-free academic retrieval (Shi et al., 2025). Building on this line of work, PaperScout improves tool-use autonomy by enabling the agent to decide when and how to invoke search and expansion actions accord- ing to the evolving search state (Pan et al., 2026). These studies show the promise of agentic litera- ture exploration beyond conventional single-shot retrieval. ScholarQuest complements them by pro- viding a systematic benchmark to evaluate such agents across retrieval quality, search efficiency, multi-turn decision ability, and robustness under diverse research intents. 3 ScholarQuest In this section, we first establish a clear task formu- lation, then describe the benchmark construction pipeline, followed by the four research intents used to evaluate different retrieval capabilities. 3.1 Task Formulation ScholarQuest evaluates academic paper search as a multi-turn retrieval task. Given a research query qand a literature databaseD, a search system it- eratively analyzes the current paper poolP t and issues retrieval or exploration requests toD. The returned papers are merged intoP t+1 , allowing the system to accumulate candidate papers and update search state over multiple turns. The system finally returns a ranked paper list ˆ P , which is evaluated against the ground-truth relevant paper setP ∗ . 3.2 Benchmark Construction 3.2.1 Query Generation We first collect over 1,600 topics from the ACM Computing Classification System (CCS), which provides a poly-hierarchical organization of com- puting topics (Rous, 2012). To focus the bench- mark on computer science paper search, we utilize Qwen3-Max (Yang et al., 2025) to map each ACM CCS topic to one or more arXiv subject categories according to the arXiv category taxonomy 2 . We 2 https://arxiv.org/category_taxonomy Comparison- based 15.3% Method- oriented 27.2% Scope- controlled 28.5% Setting- anchored 29.0% Figure 2: Dataset distribution of ScholarQuest. Side bars show query counts across arXiv CS categories, while the center pie shows query intent distribution. retain topics assigned to arXiv CS categories and discard topics outside the computer science scope. This process results in over 1,000 CS topic seeds, which serve as the input for query generation. Given each topic seed, we further utilize Qwen3- Max to generate research queries under four in- tent types: method-oriented, setting-anchored, comparison-based, and scope-controlled queries. This design allows us to systematically cover di- verse academic search intents rather than relying on queries derived from individual papers. We then apply deduplication and quality control to remove queries that are ambiguous, overly broad, overly narrow, duplicated, or difficult to judge from paper titles and abstracts. After filtering, we obtain 1,111 high-quality queries for ScholarQuest. 3.2.2 Answer Discovery Multi-source Candidate Retrieval. In the re- trieval stage, we collect an initial candidate set that serves as the starting point for subsequent ex- pansion. For each query, we use both the origi- nal query and multiple LLM-generated rewrites, including method-focused, task-focused, broader- topic, narrower-topic, and terminology-based for- mulations. These queries are issued to multiple complementary retrieval sources, including Google Search API 3 , arXiv API 4 , and Semantic Scholar API 5 . We retain only papers that can be matched to the arXiv database, and merge duplicated results 3 https://serper.dev 4 https://info.arxiv.org 5 https://w.semanticscholar.org Method Oriented 1 Setting Anchored z Comparison Based Scope Controlled cs.CVVideo Summarization Constraint: RL Whichpapersexploretheuseofdeepreinforcementlearning forvideosummarization? Title: DeepReinforcementLearningfor UnsupervisedVideoSummarization... arXiv: 1801.00054 Title: Video Summarisation by Classifica- tion with Deep Reinforcement Learning arXiv: 1807.03089 arXiv: 2505.23268 Title: UnsupervisedTranscript- assistedVideoSummarization... Abstract: Weproposeamultimodal pipeline ... Togenerateamore condensed ... Thepipelineistrained withinanRLframework. cs.SESoftware Maintenance Constraint: Static Analysis Iamlookingforpapersonsoftwaremaintenancetoolsthat excludestaticanalysistoolsfromtheirscope. arXiv: 2304.12015 Title: ITER:IterativeNeuralRepair forMulti-LocationPatches Abstract:Automatedprogramrepair hasachievedpromisingresults...we proposeaniterativeprogramrepair paradigmcalledITER... arXiv: 2008.01676 Title: Anchor:LocatingAndroid Framework-specificCrashingFaults Abstract: Androidframework-specific appcrashesarehardtodebug ... we proposeAnchor,atwo-phasesuspicious buglocationsuggestiontool ... cs.LG Transfer Learning Constraint: Random Initialization Whichpapersreportthattransferlearningoutperforms randominitializationinmodelperformance? arXiv: 1811.01137 Title: TransferLearningin Neural MachineTranslation ... Abstract: ... When compared with training an NMT model from scratch, our approach allows us to reach higher performance ... arXiv: 1909.10955 Title: EfficientlyReusingOldModels AcrossLanguagesviaTransferLearning Abstract: It is especially true in transfer learning for ... We achieve better translation quality and ... than when training from random initialization ... cs.CYPrivacy Policies Constraint: Healthcare Whichpapersevaluateprivacypoliciesinthehealthcare domain? Title: AComparativeAuditofPrivacy PoliciesfromHealthcare Organizations.. arXiv: 2306.11557 Title: Privacy Challenges...in RAG- Enhanced LLMs for Healthcare Chatbots... arXiv: 2511.11347 arXiv: 2405.17971 Title:AQualitativeFrameworkfor mHealthPrivacyPractices arXiv: 2410.14607 Title:EvaluatingPrivacyMeasures inHealthcareApps... Figure 3: Cases of the four research-intent types in ScholarQuest. Each quadrant presents a representative specific- type query with its domains, topic seed, constraints and example gold papers from the answer set. by arXiv identifier. Formally, the retrieval-based candidate pool is C ret q = [ r∈R(q) [ s∈S Retrieve s (r),(1) whereR(q)contains the original query and its rewrites,S denotes the retrieval sources. Citation-Graph Expansion. Since direct re- trieval may miss relevant papers with different terminology, we further expand the retrieved can- didates through citation relations. Starting from high-confidence papers inC ret q , we collect their ref- erences and citing papers when available, and se- lectively perform second-hop expansion from high- confidence neighbors while pruning noisy branches. The final candidate pool is defined as C q =C ret q ∪C cite q ,(2) whereC cite q denotes papers obtained from citation- graph expansion. This step complements direct retrieval by recovering papers that are connected through scholarly relations but may be missed by lexical or semantic matching. Multi-stage Relevance Adjudication. We de- rive the final answer set from the expanded can- didate pool through multi-stage relevance adjudi- cation. First, we remove clear mismatches using retrieval scores, metadata signals, and a small rel- evance model, while keeping the filtering stage recall-oriented. The remaining candidates are eval- uated by multiple LLM-based relevance judges us- ing the query, paper title, and abstract. Each judge assigns a score from 0 to 2, denoting mismatch, partial match, and strict match, respectively. Af- ter aggregating the scores intoRel(q,p), we select papers above threshold τ = 2: A q =p∈C q | Rel(q,p)≥ τ.(3) Borderline and high-disagreement cases are further adjudicated to reduce labeling errors. 3.2.3 Human Audit To assess label quality without requiring exhaustive manual annotation, we conduct a targeted human audit over 450 query-paper pairs stratified by auto- matic relevance score. Human annotators judge rel- evance based on the query, paper title, abstract, and metadata. The audit results show that final high- confidence positives have 86.0% strict-match pre- cision and 98.7% relaxed precision, where relaxed precision counts both strict and partial matches. The audit also helps localize residual false-negative risk to borderline candidates, motivating the ad- ditional adjudication of high-disagreement cases. Detailed statistics are provided in Appendix A. 3.3 Key Features of ScholarQuest ScholarQuest covers four representative research intents in academic paper search. As shown in Fig- ure 3, each type reflects a common search scenario and targets a distinct retrieval capability. Table 2: Recall-oriented results on ScholarQuest. We report Recall@25, Recall@100, and Recall@All across four query types and the overall query set. The bold indicates the best result, whileunderlineindicates the second-best. Method Method OrientedSetting AnchoredScope ControlledComparison BasedOverall R@25 R@100 R@All R@25 R@100 R@All R@25 R@100 R@All R@25 R@100 R@All R@25 R@100 R@All Dense Retrieval0.1550.3000.4030.1100.2190.306 0.0290.0870.159 0.1360.2340.3260.1040.2080.290 Hybrid Retrieval0.1530.3030.344 0.1150.2260.257 0.0380.0910.111 0.1330.2500.276 0.1070.2140.244 Google Search0.1060.1460.167 0.0590.0940.117 0.0030.0060.013 0.1220.1350.143 0.0680.0940.112 Google Scholar0.0840.1110.143 0.0730.1020.120 0.0040.0100.010 0.1070.1250.130 0.0710.1000.113 Semantic Scholar0.0640.0850.101 0.1030.1160.125 0.0010.0020.002 0.0890.1030.103 0.0570.0840.099 DeepXiv (Qian et al., 2026)0.0910.2140.341 0.0680.1810.304 0.0070.0230.064 0.0390.1180.286 0.0540.1380.288 PaSa (He et al., 2025)0.2450.3450.366 0.2300.2960.3120.1030.1930.242 0.2560.2940.3240.2010.2810.310 SPAR (Shi et al., 2025)0.2420.3330.351 0.2230.2820.293 0.1020.1880.222 0.2460.2800.301 0.1970.2700.291 PaperScout (Pan et al., 2026) 0.2750.4080.451 0.2460.3410.366 0.0910.1820.2390.2750.3270.358 0.2140.3140.355 Method-oriented Queries. This dimension fo- cuses on finding papers that use, extend, or analyze a specific method. It tests whether systems can identify methodological relevance beyond surface- level keyword overlap. Setting-anchored Queries.This category targets papers under a specific task, dataset, or experimen- tal setting. It evaluates whether systems can satisfy fine-grained contextual constraints rather than re- trieve broadly related papers. Scope-controlled Queries.This feature specifies explicit boundaries for the desired paper set, such as a target domain, task, or model family. It evalu- ates whether systems can control retrieval breadth and avoid loosely relevant results. Comparison-based Queries. This aspect fo- cuses on papers that compare methods, assump- tions, results, or technical claims. It tests whether systems can capture relational search intents involv- ing multiple research objects. 4 Experiment In this section, we evaluate the overall retrieval per- formance and analyze agentic search behavior in terms of efficiency and robustness. Finally, we di- agnose shared failure patterns across search agents. 4.1 Experimental Setup Dataset. We evaluate all methods on Scholar- Quest, which contains 1,111 high-quality queries constructed from over 1,000 computer science topic seeds. As shown in Figure 2, the dataset spans a broad range of arXiv CS categories, covering ma- jor areas such as cs.LG, cs.CV, cs.AI, and cs.CL, while also preserving a long-tail distribution over diverse subfields. The queries are divided into four intent types: method-oriented, setting-anchored, scope-controlled, and comparison-based, with rel- (a) Avg. Toolcall(b) Avg. Search(c) Avg. Expand (d) Avg. Rounds(e) Avg. Candidates(f) R@100 / 100 Cand. PaSaPaperScoutSPAR Figure 4: Efficiency and process statistics of paper search agents, including number of tool calls, rounds, observed candidates, and R@100/100 Cand., which nor- malizes Recall@100 by every 100 observed candidates. atively balanced proportions of 27.2%, 29.0%, 28.5%, and 15.3%, respectively. Environment. We build ScholarBase, an open- source paper-search environment constructed from the S2 PaperData snapshot (Lo et al., 2020), which retains arXiv papers with abstracts, and stores pa- per metadata and citation relations. It supports BM25 (Harman, 1995) sparse retrieval with SQLite FTS5 6 , dense retrieval with Qdrant over BGE- M3 (Chen et al., 2024) title–abstract embeddings, and RRF-based hybrid retrieval (Cormack et al., 2009). The environment further provides RESTful APIs for semantic search, title matching, metadata lookup, and citation/reference traversal, enabling all methods to search, inspect, and expand papers under the same controlled literature backend. Baselines. We compare three groups of base- lines. The first group includes standard retrieval baselines implemented in our ScholarBase, includ- ing dense retrieval and RRF-based hybrid search. The second group includes widely used academic search systems, including Google Search 7 , Google 6 https://sqlite.org 7 https://serper.dev Figure 5: Recall@All density distributions of agentic paper search methods. Panels (a)–(c) show query-type distributions within each method, while panel (d) com- pares the three methods over all queries. The x-axis is Recall@All and the y-axis is probability density. Scholar 8 , Semantic Scholar API 9 and DeepXiv (Qian et al., 2026). The third group includes open- source agentic paper search systems, including PaSa (He et al., 2025), SPAR (Shi et al., 2025), and PaperScout (Pan et al., 2026). Metrics. Considering that paper search empha- sizes comprehensive answer coverage, we pri- marily report Recall@25, Recall@100, and Re- call@All under different retrieval budgets. To ana- lyze the search process, we further report the num- ber of interaction rounds, search calls, expansion calls, observed candidates, and Recall@100 per 100 observed candidates. We also examine robust- ness across research intents and answer-set sizes. 4.2 Main Results Table 2 reports Recall@25, Recall@100, and Re- call@All across four research intents and the over- all query set. Traditional retrieval methods show clear limitations on complex paper search queries, especially on compositional constraints, as shown by the low R@100 of Google Search (0.006) and Google Scholar (0.010) on scope-controlled queries. Dense and Hybrid retrieval improve over sparse retrieval, achieving 0.208 and 0.214 over- all R@100. However, they remain single-shot re- trievers and cannot iteratively refine queries, ex- pand promising papers, or integrate intermedi- ate evidence. In contrast, agentic methods with multi-round decision-making achieve higher over- all R@100 scores, ranging from 0.270 to 0.314. 8 https://scholar.google.com 9 https://w.semanticscholar.org Figure 6: Recall trajectories under increasing tool-call budgets. The upper and lower panels show Recall@All and Recall@50, respectively, with shaded regions indi- cating query-level variation. The results also show clear differences among agentic search methods. PaperScout achieves 0.408 R@100 on method-oriented, outperforming PaSa and SPAR by 0.063 and 0.075, respectively. It also leads on setting-anchored and comparison-based, with R@100 scores of 0.341 and 0.327, higher than PaSa (0.296, 0.294) and SPAR (0.282, 0.280). One possible explanation is that more adaptive multi- turn search can help agents better navigate scattered literature evidence. We conduct further analysis in the next section to examine tool-use patterns, search efficiency, and exploration behavior. 4.3 Analysis of Agentic Search Behavior Beyond overall retrieval performance, we further analyze agentic search behavior from the perspec- tives of search efficiency, research-intent robust- ness, and answer-set-size robustness. Search Efficiency. Retrieval quality alone is in- sufficient for paper search, where agents should avoid exhaustive candidate expansion. As shown in Figure 4, the three agents exhibit distinct tool- use patterns. PaSa uses the most tool calls (60.1), mainly expansions (55.1), and observes the largest candidate set (744), but its recall efficiency is rela- tively low at 0.051 Recall@100 per 100 candidates. SPAR is also expansion-oriented, with 39.1 expan- sion calls out of 47.1 tool calls, observing 515 candidates and achieving 0.064 recall efficiency. In contrast, PaperScout uses fewer expansions (19.0) while making more search calls (15.3) across more interaction rounds (9.2), leading to fewer observed candidates (408) and the highest recall efficiency Table 3: Average Recall@All across gold answer-set size buckets. #Query denotes the number of queries. Answer Size#Query Recall@All PaSaSPARPaperScout 5–254570.3520.3380.405 26–502180.3190.3070.371 51–1002250.2670.2420.304 101–1501300.2650.2390.281 151–200810.2350.2050.278 (0.120). This efficiency advantage may stem from its autonomous tool-use policy, which allows Paper- Scout to decide when and how to search or expand based on the evolving search state. Figure 6 further shows recall trajectories under different tool-call budgets. PaSa improves rapidly at the early stage but soon saturates, while SPAR shows steadier but more limited gains. PaperScout starts more gradually, but continues to improve with additional interactions and eventually reaches higher recall under a smaller or comparable tool- call budget. This suggests that adaptive tool-use can improve the marginal utility of each tool call. Robustness Across Research Intents. Agentic methods generally outperform traditional retrieval methods across metrics. However, their perfor- mance varies substantially across research intents, revealing important limitations of current paper search agents. As shown in Table 2, all three agents perform relatively well on method-oriented, setting- anchored, and comparison-based queries, where the search intent is mainly guided by positive se- mantic cues such as methods, tasks, settings, or comparative claims. In contrast, scope-controlled is consistently the most challenging category: PaSa, SPAR, and PaperScout achieve only 0.193, 0.188, and 0.182 R@100, respectively, which are clearly lower than their performance on other intents. Figure 5 further provides a distributional view of this limitation. For all three agents, the recall distributions of scope-controlled queries are con- centrated in the low-recall region, indicating that this category is not only worse on average but also more prone to query-level failures. Unlike positive- intent queries, scope-control queries require agents to retrieve papers that match the main topic while preserving exclusion constraints or fine-grained scope boundaries during multi-round exploration. Although PaperScout shows a more right-shifted overall recall distribution, its scope-control curve Table 4: Common zero-recall cases by query type. #Failures denotes the number of queries where all three agents retrieve no gold papers; Avg. Candidates report the average accessed candidate papers. Query Type#Failures Avg. Candidates PaSaSPAR PaperScout Comparison-based7926.6589.9405.7 Scope-controlled6871.2618.0315.3 Setting-anchored5713.6551.8472.8 Method-oriented21306.0831.0532.0 Overall20894.6612.9408.0 remains heavily skewed toward low recall. These results suggest that adaptive tool use improves ro- bustness, but constraint-sensitive paper search re- mains a shared limitation of agentic methods. Robustness Across Answer-set Sizes. We fur- ther evaluate robustness by grouping queries ac- cording to the number of ground-truth answers. As shown in Table 3, all agents exhibit lower average recall as the answer set becomes larger, suggest- ing that broad-answer queries remain challenging under limited search budgets. For instance, PaSa drops from 0.352 in the 5–25 bucket to 0.235 in the 150–200 bucket, while SPAR drops from 0.338 to 0.205. PaperScout achieves the highest average recall in every bucket, with scores of 0.405, 0.371, 0.304, 0.281, and 0.278 from small to large an- swer sets. Its advantage persists even in the largest bucket, where it outperforms PaSa and SPAR by 0.043 and 0.073 average recall, respectively. These results indicate that PaperScout better balances fo- cused exploration and broad coverage across vary- ing answer-set sizes. More detailed results are shown in Table 7. 4.4 Failure Analysis We analyze common zero-recall cases, where PaSa, SPAR, and PaperScout all fail to retrieve any gold paper. As shown in Table 4, such failures appear across all four query types. Although these cases are rare, covering only 20 out of 1,111 complete agentic queries (1.80%), they are highly diagnos- tic: all three agents access a substantial number of candidate papers on these queries, yet none of the retrieved papers match the gold answers. Figure 7 presents representative examples for each query type, showing that failures arise from different forms of intent mismatch rather than a uniform retrieval error. For claim-comparison queries, agents retrieve related papers but miss BQ_000367: Which papers report that autonomous robots outperform human teleoperation in specific tasks? Agents retrieve topic-related robotics papers, but fail to locate papers containing explicit evidence for the superiority claim. The difficulty lies in evidence-level comparison, not topical relevance. BQ_000472: I am looking for papers on remote replication that exclude local replication from their scope. Agents treat the excluded concept as a positive retrieval keyword, instead of performing set-level filtering. The query should be solved as “topic A minus topic B,” but agents search for both A and B. BQ_001550: Which papers evaluate image search techniques on the CIFAR-10 dataset? Agents recognize the dataset anchor but drift toward popular CIFAR-10 benchmark papers or adjacent vision tasks, rather than papers evaluating the target method under the specified dataset. BQ_002645: Which papers explore the use of reinforcement learning for collaborative content creation? Agents capture method keywords, but drift away from the intended method–application combination. They retrieve RL-for-content or generation-related papers, but miss specific capability of collaborative content creation. Failure Analysis Search Query Claim Comparison Scope Control Setting Anchor Method Capability ... Success / Gold 0 / 9 Agentic Search ... Success / Gold 0 / 8 Agentic Search ... Success / Gold 0 / 30 Agentic Search ... Success / Gold 0 / 8 Agentic Search Figure 7: Representative common zero-recall cases across query types. Each case shows a query where all three agents retrieve no gold papers, together with the corresponding gold-answer size and the main failure cause. evidence for the specific comparison.In the autonomous-robot case, they fail to find papers ex- plicitly showing autonomous robots outperform hu- man teleoperation, yielding 0 out of 9 gold papers. For scope-control queries, agents struggle with negative constraints: in the remote replication case, the target is essentially “remote replication minus local replication,” but agents treat the excluded con- cept as a positive keyword, yielding 0 out of 8 gold papers. For setting-anchor queries, agents rec- ognize anchors such as CIFAR-10 or Pascal VOC, but drift toward popular benchmark papers instead of papers evaluating the target method under the specified setting. For method-capability queries, agents match the method keyword but fail to pre- serve compositional intent, such as reinforcement learning for collaborative content creation. Beyond query-specific failure patterns, Table 4 further shows that common zero-recall failures are not due to a lack of search effort, but to off-target exploration. PaSa and SPAR access hundreds of candidate papers on average across all failed query types, while PaperScout also ex- plores a non-trivial candidate pool through iterative search. The core problem is therefore not whether agents search, but whether they can reach the cor- rect region of the literature space. When the initial search direction is biased toward a semantically plausible but incorrect neighborhood, further re- trieval or citation expansion tends to enlarge that neighborhood rather than recover the missing gold papers. This highlights a key challenge in paper search: agents should not only search broadly, but also target the right evidence, constraints, and method–setting combinations. 5 Conclusion We propose ScholarQuest, a taxonomy-guided benchmark for agentic academic paper search in open literature environments.Unlike existing benchmarks that rely on limited human queries or paper-derived query construction, ScholarQuest provides broad CS topic coverage, four representa- tive research intents, scalable answer construction, and a shared ScholarBase backend for reproducible evaluation. Experiments show that agentic search methods outperform single-shot retrieval systems, with the best agentic method improving overall R@100 from 0.214 to 0.314 over the strongest non-agentic baseline, yielding a relative gain of 46.7%. Meanwhile, our analysis reveals limita- tions in search efficiency, query-type robustness, and constraint handling, with key failures caused by off-target exploration rather than insufficient effort. ScholarQuest offers a transparent and diag- nostic testbed for identifying these limitations and tracking future progress in agentic paper search. Future work may develop agents with stronger in- tent preservation, constraint-aware filtering, and evidence-level reasoning throughout multi-round exploration. We invite the community to use Schol- arQuest to study, compare, and build more reliable academic paper search agents. 6 Limitations ScholarQuest is designed as a controlled bench- mark for agentic paper search, but it still has several boundaries. First, ScholarQuest focuses on com- puter science topics and uses an arXiv-grounded literature environment, so it does not cover the full diversity of scholarly communication across disciplines, venues, and publication formats. Sec- ond, relevance judgments are based on paper ti- tles, abstracts, and metadata rather than full-text evidence, which makes the benchmark scalable but may miss fine-grained claims that only appear in the body of a paper. Finally, although our an- swer construction pipeline combines multi-source retrieval, citation expansion, LLM-based relevance adjudication, and human audit, automatic construc- tion may still miss some relevant papers in open literature environments. These limitations motivate future extensions toward broader literature sources, full-text-aware relevance assessment, and stronger answer-set validation. References Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216, 4(5). Mingyue Cheng, Daoyu Wang, Qi Liu, Shuo Yu, Xiaoyu Tao, Yuqian Wang, Chengzhong Chu, Yu Duan, Mingkang Long, and Enhong Chen. 2026. Mind2report: A cognitive deep research agent for expert-level commercial report synthesis. arXiv preprint arXiv:2601.04879. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or par- tial credit. Psychological bulletin, 70(4):213. Gordon V Cormack, Charles LA Clarke, and Stefan Buettcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in informa- tion retrieval, pages 758–759. Michael Gusenbauer and Neal R Haddaway. 2020. Which academic search systems are suitable for systematic reviews or meta-analyses? evaluating retrieval qualities of google scholar, pubmed, and 26 other resources. Research synthesis methods, 11(2):181–217. Donna K Harman. 1995. Overview of the third text retrieval conference (TREC-3), volume 225. DIANE Publishing. Yichen He, Guanhua Huang, Peiyuan Feng, Yuan Lin, Yuchen Zhang, Hang Li, et al. 2025. Pasa: An llm agent for comprehensive academic paper search. arXiv preprint arXiv:2501.10120. Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516. Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kin- ney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 4969–4983, Online. Asso- ciation for Computational Linguistics. Gary Marchionini. 2006. Exploratory search: from finding to understanding. Communications of the ACM, 49(4):41–46. Tingyue Pan, Jie Ouyang, Mingyue Cheng, Qingchuan Li, Zirui Liu, Daoyu Wang, Mingfan Pan, Shuo Yu, and Qi Liu. 2026. Paperscout: An autonomous agent for academic paper search with process-aware sequence-level policy optimization. arXiv preprint arXiv:2601.10029. Karl Pearson. 1896. Vii. mathematical contributions to the theory of evolution.—i. regression, heredity, and panmixia. Philosophical Transactions of the Royal Society of London. Series A, containing papers of a mathematical or physical character, (187):253–318. Hongjin Qian, Ziyi Xia, Ze Liu, Jianlyu Chen, Kun Luo, Minghao Qin, Chaofan Li, Lei Xiong, Junwei Lan, Sen Wang, et al. 2026. Deepxiv-sdk: An agentic data interface for scientific literature. arXiv preprint arXiv:2603.00084. Bernard Rous. 2012. Major update to acm’s computing classification system. Communications of the ACM, 55(11):12–12. Xiaofeng Shi, Yuduo Li, Qian Kou, Longbin Yu, Jinxin Xie, and Hua Zhou. 2025. Spar: Scholar paper re- trieval with llm-based agents for enhanced academic search. arXiv preprint arXiv:2507.15245. Charles Spearman. 1961. The proof and measurement of association between two things. Fiona Timmins and Catherine McCabe. 2005. How to conduct an effective literature search. Nursing standard, 20(11):41–47. Daoyu Wang, Mingyue Cheng, Shuo Yu, Zirui Liu, Ze Guo, Xin Li, and Qi Liu. 2025. Paperarena: An evaluation benchmark for tool-augmented agen- tic reasoning on scientific literature. arXiv preprint arXiv:2510.10909. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. A Implementation Details We provide reproducibility details for ScholarQuest here. We organize the details into benchmark con- struction and dataset evaluation. A.1 Details of Benchmark Construction Query Construction.We start from 1,682 ACM CCS topics and use Qwen3-Max to map each topic to one or more arXiv subject categories. Only topics assigned to arXiv CS categories in Table 8 are retained, resulting in 1,638 CS topic seeds. For each seed, Qwen3-Max generates exactly four queries, one for each intent type: method-oriented, setting-anchored, comparison-based, and scope- controlled. After deduplication and quality filter- ing, we keep 1,111 high-quality queries. Answer Construction. For each query, we gen- erate 10 rewritten search queries and retrieve the top 10 results for each rewrite from complemen- tary retrieval sources, including Google Search, arXiv, and Semantic Scholar. Candidate papers are matched to arXiv records, normalized by arXiv ID, and deduplicated before scoring. We use a recall- oriented prefilter to remove clear mismatches, fol- lowed by LLM-based relevance adjudication using the query, paper title, abstract and metadata. Pa- pers judged as strict matches are treated as primary answer candidates. For citation expansion, each high-confidence seed paper retrieves up to 30 citing papers and all available references.We further perform second-hop expansion from high-confidence first- hop neighbors while pruning noisy branches. Newly discovered papers are normalized, dedupli- cated, and scored with the same relevance-filtering pipeline. The expansion stops when no eligi- ble high-confidence seed remains, the answer set reaches 500 papers, or the number of scored unique papers reaches 10,000 for a query. Human Evaluation. To validate the final-stage LLM relevance scoring, we sample 150 query- paper pairs from each score group, resulting in 450 pairs in total, and ask three Ph.D. experts to re-score them based on the user query, paper title, abstract, and metadata. Human annotators follow the same three-level rubric as the LLM judges: 0 = mismatch, 1 = partial match, and 2 = strict match. For each pair, we aggregate the three expert annota- tions by majority vote and use this aggregated label as the human score in the following agreement anal- Table 5: Default configuration for ScholarQuest bench- mark construction. ConfigurationValue Raw ACM CCS topics1,682 Retained CS topic seeds1,638 Final queries1,111 Queries per topic seed4 Query rewrites per query10 Search top-k per rewrite10 Retrieval sourcesGoogle Search, arXiv, Semantic Scholar Query-level concurrency4 Citation expansion depthUp to 2 hops Citation limit per seed paper30 Reference limit per seed paperAll available references First-stage filteringRecall-oriented prefilter Relevance scoring rubric0–2 Answer relevance threshold2.0 Maximum answer count500 Maximum scored papers per query10,000 Relevance judgesMultiple LLM-based judges ysis. As shown in Table 6, LLM scores align well with human annotations, with Pearson correlation of 0.867 (Pearson, 1896), Spearman correlation of 0.867 (Spearman, 1961), and quadratic weighted Cohen’sκof 0.866 (Cohen, 1968). The score-2 group used for final answer selection reaches 86.0% strict-match precision (129/150) and 98.7% relaxed precision (148/150 with human score≥ 1). In con- trast, only 1/150 score-0 candidates and 12/150 score-1 candidates are judged as strict matches by humans, suggesting that residual false-negative risk is concentrated in borderline cases. Overall, these results indicate that the automated construction pipeline can reliably separate strict- match answers from mismatches, with residual er- rors mainly concentrated in borderline cases. We further present four representative disagreement cases in Table 9, illustrating typical failure modes of LLM-based relevance scoring, such as over- reliance on topical overlap and insufficient sensitiv- ity to fine-grained query constraints. A.2 Details of Dataset Evaluation We compare nine baselines from three groups: stan- dard retrieval methods, external academic search systems, and agentic search methods. •Dense Retrieval. Dense retriever using BGE- M3 title–abstract embeddings. •Hybrid Retrieval. Hybrid retriever combin- ing BM25 and dense retrieval with RRF. •Google Search. A general web search base- line for paper discovery. • Google Scholar. A widely used academic search engine baseline. LLM Score Human Score TotalHit Rate 012 013316115088.7% 1211171215078.0% 221912915086.0% Total15615214245084.2% Table 6: Confusion matrix between LLM-based rele- vance scores and human annotations. Hit Rate denotes the exact agreement rate within each LLM score group. •Semantic Scholar. An academic search API with paper metadata and citation information. •DeepXiv. An external scientific literature search interface. •PaSa. LLM-based agent for multi-step aca- demic paper search. • SPAR. Training-free workflow with retrieval, citation exploration, and reranking. •PaperScout. Autonomous paper search agent that adaptively decides when and how to search or expand. Retrieval Baselines. We evaluate two Scholar- Base retrieval baselines: dense retrieval and RRF- based hybrid retrieval. Dense retrieval ranks pa- pers with BGE-M3 title–abstract embeddings. Hy- brid retrieval combines sparse and dense rankings through reciprocal rank fusion. Both methods take the original query as input and return ranked papers from ScholarBase. Academic Search Systems.We evaluate four ex- ternal academic search systems: Google Search, Google Scholar, and search APIs of Semantic Scholar and DeepXiv. For each query, each sys- tem returns up to 300 papers. Returned papers are matched to ScholarQuest answers through arXiv identifiers, titles, and available metadata, ensuring that all methods are evaluated against the same arXiv-grounded answer sets. Agentic Search Methods. We evaluate three agentic paper search systems: PaSa, SPAR, and PaperScout. To ensure a fair comparison, we run their released inference code and open checkpoints without modifying their model weights or decision logic. All tool interfaces are provided by Schol- arBase, including paper search, metadata lookup, and citation/reference expansion, so that different agents interact with the same literature backend. During evaluation, all retrieved or expanded papers are normalized by arXiv ID before metric compu- tation. We also record process statistics, including interaction rounds, search calls, expansion calls, observed candidates, and Recall@100 per 100 ob- served candidates. B Tool-call Evidence for Agentic Search We further inspect the tool-use trajectory of BQ_002897, the method-capability case shown in Appendix D. The query asks for papers that ex- plore deep reinforcement learning for video sum- marization. This case is useful because the gold answers form a compact but nontrivial neighbor- hood: some answers are directly reachable through targeted keyword search, while others are only re- covered through citation/reference expansion. Table 10 lists every tool call that directly in- creases cumulative recall. PaperScout reaches all 12 gold answers after 20 tool calls.Its first four search calls recover 9/12 answers, and two later expansion calls recover the remaining three answers.In contrast, SPAR and PaSa both stop at 8/12.The missing answers are also systematic:both SPAR and PaSa miss 2109.01309and1807.09418; SPAR additionally misses2007.14552and2002.03740, while PaSa additionally misses 2105.06441 and 2007.14552. Table 11 gives a gold-answer-level view of the same query. It shows that PaperScout’s ad- vantage does not come from more direct hits alone: it uses early successful searches as anchors and then follows expansion paths from relevant seeds. For example, expanding1801.00054re- covers2002.03740and2105.06441, while ex- panding1805.02792recovers1807.09418. This explains why only a small number of tool calls change recall: marginal gold discoveries are sparse, and many additional calls inspect already relevant neighborhoods or provide post-saturation confirma- tion rather than adding new gold answers. C Annotator Compensation All human annotators are compensated for their work. The compensation rate is set to no less than ten times the local minimum hourly wage for each participant, reflecting both the specialized expertise required and the time-intensive nature of bench- mark refinement and evaluation. Table 7: Average Recall@100 of agentic paper search methods across gold answer-set size buckets and query types. PaperScout is strongest on method and setting queries, while scope-control remains difficult and recall decreases as answer sets grow. Answer SizeMethod-orientedSetting-anchoredScope-controlledComparison-basedOverall PaSa SPAR PaperScout PaSa SPAR PaperScout PaSa SPAR PaperScout PaSa SPAR PaperScout PaSa SPAR PaperScout 5–250.4010.3900.5120.3420.3370.3890.2680.2520.2520.3110.3000.3420.3380.3270.387 26–500.3460.3400.4130.3120.2960.3700.2140.2190.1900.2350.2300.2940.2950.2880.337 51–1000.3200.2990.3440.2410.2160.2850.1470.1490.1520.1820.1390.2130.2290.2150.251 101–1500.2570.2490.2620.2420.2130.2620.1680.1660.1430.1830.1270.1730.2140.2020.209 151–2000.2380.2370.2430.1750.1590.1950.1440.1300.1420.1650.1100.1770.1760.1640.181 Table 8: Full names of arXiv computer science subject categories. These categories are used to map ACM CCS topics into the CS topic space for ScholarQuest construction. CategoryFull NameCategoryFull Name cs.AIArtificial Intelligencecs.ARHardware Architecture cs.CCComputational Complexitycs.CEComputational Engineering, Finance, and Science cs.CGComputational Geometrycs.CLComputation and Language cs.CRCryptography and Securitycs.CVComputer Vision and Pattern Recognition cs.CYComputers and Societycs.DBDatabases cs.DCDistributed, Parallel, and Cluster Computing cs.DLDigital Libraries cs.DMDiscrete Mathematics cs.DSData Structures and Algorithms cs.ETEmerging Technologiescs.FLFormal Languages and Automata Theory cs.GLGeneral Literature cs.GRGraphics cs.GTComputer Science and Game Theorycs.HCHuman-Computer Interaction cs.IRInformation Retrieval cs.ITInformation Theory cs.LGMachine Learning cs.LOLogic in Computer Science cs.MAMultiagent Systems cs.MMMultimedia cs.MSMathematical Softwarecs.NANumerical Analysis cs.NENeural and Evolutionary Computingcs.NINetworking and Internet Architecture cs.OHOther Computer Sciencecs.OSOperating Systems cs.PFPerformance cs.PLProgramming Languages cs.RORoboticscs.SCSymbolic Computation cs.SDSoundcs.SESoftware Engineering cs.SISocial and Information Networkscs.SYSystems and Control Table 9: Representative disagreement cases between human judgments and LLM-generated labels. Human labels are obtained through manual review, while LLM labels correspond to the original automatic relevance labels. Scores 0, 1, and 2 denote mismatch, partial match, and strict match, respectively. Query and LabelPaper Title and Abstract EvidenceLLM Label Bias Query: Which papers evaluate network domains in the context of data center net- works? Human: 0 (mismatch) LLM: 1 (partial match) Title: Efficient Coflow Scheduling in Hybrid-Switched Data Cen- ter Networks Evidence: The paper studies coflow scheduling in hybrid-switched data center networks, focusing on scheduling, communication time, hybrid links, and performance guarantees. However, it does not explicitly evaluate network domains. Keyword over-crediting. The LLM is attracted by the strong topic match “data center networks,” but overlooks the missing constraint of “network domains,” leading to an overestimated relevance label. Query: Which papers report that machine learning-based information extraction out- performs rule-based systems? Human: 1 (partial match) LLM: 0 (mismatch) Title: PAM: Understanding Product Images in Cross Product Category Attribute Extraction Evidence: The paper studies product attribute extraction with a transformer-based model using product text, OCR tokens, and visual objects. It is related to machine learning-based information extraction, but does not directly report outperforming rule-based systems. Over-strict claim matching. The LLM treats the comparison with rule-based systems as mandatory, while human reviewers still regard the paper as a partial match due to its clear task- level match. Query: Which papers study image search using deep learning techniques? Human: 1 (partial match) LLM: 2 (strict match) Title: Visual Discovery at Pinterest Evidence: The paper presents Pinterest’s visual discovery engine for visual search and recommendation products. It mentions object detection and improved engagement, but does not explicitly state “deep learning” or specify neural architectures. Implicit-knowledge overuse. The LLM likely infers deep learning from visual search and ob- ject detection, upgrading the paper to a strict match without explicit textual evidence for the method constraint. Query: Which papers claim that modern imaging techniques outperform traditional methods? Human: 2 (strict match) LLM: 1 (partial match) Title: Twin-beam sub-shot-noise raster-scanning microscope Evidence: The paper proposes a quantum imaging microscope and reports improved precision over a shot-noise-limited classical version while preserving resolution and optical power. Terminology mismatch. The LLM underes- timates relevance because the paper expresses “traditional methods” as a domain-specific “clas- sical version,” whereas human reviewers recog- nize the comparison as sufficient evidence. Table 10: Evidence chain forBQ_002897. Each row is a tool call that directly increases cumulative recall. PaperScout first locates most gold answers through targeted search, then uses expansion to recover the remaining answers. MethodCallToolQuery or Expansion SeedNewly Found Gold AnswersRecall PaperScout1searchdeep reinforcement learning AND video summarization 2001.058641/12 PaperScout2searchreinforcement learning AND video summarization 2007.14552, 2106.10528, 2109.01309 4/12 PaperScout3search deep reinforcement learning AND action selection AND video summarization 1801.00054,1807.03089,2005.09531, 2407.04258 8/12 PaperScout4searchreinforcement learning AND keyframe selection AND video summarization 2505.232689/12 PaperScout6expand 1801.00054 2002.03740, 2105.0644111/12 PaperScout20expand 1805.02792 1807.0941812/12 SPAR1searchLiterature review of application domains in deep reinforcement learning for video summarization 2407.042581/12 SPAR2expand 2405.08890 1801.00054, 2001.058643/12 SPAR3expand 2101.06072 1807.03089, 2005.095315/12 SPAR7expand 2407.04258 2505.232686/12 SPAR14search State-of-the-art in deep reinforcement learning for video sum- marization 2105.064417/12 SPAR32expand 2412.08357 2106.105288/12 PaSa1searchSurvey papers on video summarization using deep reinforce- ment learning 1801.00054,1807.03089,2005.09531, 2407.04258, 2505.23268 5/12 PaSa8expand 2101.06072 2001.058646/12 PaSa10expand 2410.04449 2106.105287/12 PaSa50expand 2105.04066 2002.037408/12 Table 11: Gold-answer provenance forBQ_002897. S# and E# denote the order of search and expansion calls, respectively, through which a method discovers each answer. Gold AnswerTitlePaperScout RouteSPAR RoutePaSa Route 2505.23268Unsupervised Transcript-assisted Video Summa- rization and Highlight Detection S4 search: keyframe selection queryE7 expand 2407.04258S1 search: survey query 2407.04258Reinforcement Learning for Unsupervised Video Summarization With Reward Generator Training S3 search: action-selection queryS1 search:literature- review query S1 search: survey query 2109.01309Unsupervised multi-latent space reinforcement learning framework for video summarization in ultrasound imaging S2 search: reinforcement learning query MissedMissed 2106.10528Video Summarization Through Reinforcement Learning With a 3D Spatio-Temporal U-Net S2 search: reinforcement learning query E32 expand 2412.08357 E10 expand 2410.04449 2105.06441DeepQAMVS:Query-AwareHierarchical Pointer Networks for Multi-Video Summariza- tion E6 expand 1801.00054S14 search: state-of-the- art query Missed 2007.14552Compare and Select: Video Summarization with Multi-Agent Reinforcement Learning S2 search: reinforcement learning query MissedMissed 2005.09531Ultrasound Video Summarization using Deep Reinforcement Learning S3 search: action-selection queryE3 expand 2101.06072S1 search: survey query 2002.03740Query-Biased Self-Attentive Network for Query-Focused Video Summarization E6 expand 1801.00054MissedE50 expand 2105.04066 2001.05864Weakly Supervised Video Summarization by Hi- erarchical Reinforcement Learning S1 search: deep reinforcement learn- ing query E2 expand 2405.08890E8 expand 2101.06072 1807.09418Video Storytelling: Textual Summaries for Events E20 expand 1805.02792MissedMissed 1807.03089Video Summarisation by Classification with Deep Reinforcement Learning S3 search: action-selection queryE3 expand 2101.06072S1 search: survey query 1801.00054Deep Reinforcement Learning for Unsuper- vised Video Summarization with Diversity- Representativeness Reward S3 search: action-selection queryE2 expand 2405.08890S1 search: survey query D Case Studies We provide four representative ScholarQuest cases, one for each query intent. Each case shows the query metadata and its complete gold-answer set sorted in descending order by the first arXiv submission date. Case 1: Method-Capability Query (BQ_002897) Query. Which papers explore the use of deep reinforcement learning for video summarization? Domain.cs.CV; Topic seed. Video summarization; Constraint.technique = deep reinforcement learning; Gold answers. 11. DatearXiv IDTitle 2025-05-29 2505.23268Unsupervised Transcript-assisted Video Summarization and Highlight Detection 2024-07-05 2407.04258Reinforcement Learning for Unsupervised Video Summarization with Reward Generator Training 2021-09-03 2109.01309Unsupervised multi-latent space reinforcement learning framework for video summarization in ultrasound imaging 2021-06-19 2106.10528Video Summarization through Reinforcement Learning with a 3D Spatio-Temporal U-Net 2021-05-13 2105.06441DeepQAMVS: Query-Aware Hierarchical Pointer Networks for Multi-Video Summarization 2020-07-29 2007.14552Compare and Select: Video Summarization with Multi-Agent Reinforcement Learning 2020-05-19 2005.09531Ultrasound Video Summarization using Deep Reinforcement Learning 2020-01-12 2001.05864Weakly Supervised Video Summarization by Hierarchical Reinforcement Learning 2018-07-25 1807.09418Video Storytelling: Textual Summaries for Events 2018-07-09 1807.03089Video Summarisation by Classification with Deep Reinforcement Learning 2017-12-29 1801.00054 Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity- Representativeness Reward Case 2: Claim-Comparison Query (BQ_000815) Query. Which papers report that transfer learning outperforms random initialization in model performance? Domain.cs.LG; Topic seed. Transfer learning; Constraint.comparison = outperforms random initialization; Gold answers. 15. DatearXiv IDTitle 2025-11-06 2511.11622Small Vocabularies, Big Gains: Pretraining and Tokenization in Time Series Models 2024-10-10 2410.08194Features are fate: a theory of transfer learning in high-dimensional regression 2024-08-01 2408.00695Accelerating Full Waveform Inversion By Transfer Learning 2022-06-20 2206.09872A Neural Network Based Method with Transfer Learning for Genetic Data Analysis 2021-09-29 2109.14536 PINNup: Robust neural network wavefield solutions using frequency upscaling and neuron splitting 2021-06-09 2106.04995Crosslingual Embeddings are Essential in UNMT for Distant Languages: An English to IndoAryan Case Study 2019-09-24 1909.10955Efficiently Reusing Old Models Across Languages via Transfer Learning 2019-08-26 1908.09883Transfer learning for scalability of neural-network quantum states 2018-11-03 1811.01137Transfer Learning in Multilingual Neural Machine Translation with Dynamic Vocabulary 2018-10-15 1810.06282Feature Representation Analysis of Deep Convolutional Neural Network using Two-stage Feature Transfer -An Application for Diffuse Lung Disease Classification- 2018-09-02 1809.00357Trivial Transfer Learning for Low-Resource Neural Machine Translation 2018-02-05 1802.01483Explicit Inductive Bias for Transfer Learning with Convolutional Networks 2017-10-11 1710.05726Convolutional Neural Networks for Histopathology Image Classification: Training vs. Using Pre-Trained Networks 2017-06-02 1706.00712Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning? 2016-04-08 1604.02201Transfer Learning for Low-Resource Neural Machine Translation Case 3: Scope-Control Query (BQ_001592) Query. I am looking for papers on software maintenance tools that exclude static analysis tools from their scope. Domain.cs.SE; Topic seed. Software maintenance tools; Constraint.exclude_filtering = static analysis tools; Gold answers. 12. DatearXiv IDTitle 2025-06-30 2506.24015Hierarchical Knowledge Injection for Improving LLM-based Program Repair 2025-06-16 2506.13182From Empirical Evaluation to Context-Aware Enhancement: Repairing Regression Errors with LLMs 2024-10-18 2410.14393Debug Smarter, Not Harder: AI Agents for Error Resolution in Computational Notebooks 2024-04-08 2404.05520The Fact Selection Problem in LLM-Based Program Repair 2023-04-24 2304.12015ITER: Iterative Neural Repair for Multi-Location Patches 2023-02-02 2302.01215Fixing Hardware Security Bugs with Large Language Models 2022-02-22 2202.10868Neural Program Repair: Systems, Challenges and Solutions 2021-12-03 2112.02125Examining Zero-Shot Vulnerability Repair with Large Language Models 2021-04-16 2104.08308Neural Transfer Learning for Repairing Security Vulnerabilities in C Code 2020-08-04 2008.01676Anchor: Locating Android Framework-specific Crashing Faults 2020-02-10 2002.03968E-APR: Mapping the Effectiveness of Automated Program Repair 2018-05-18 1805.07475Learning to Repair Software Vulnerabilities with Generative Adversarial Networks Case 4: Setting-Anchored Query (BQ_001790) Query. Which papers study or evaluate privacy and security issues, policies, or privacy-preserving techniques in the healthcare domain? Domain.cs.CY; Topic seed. Privacy policies; Constraint.application_domain = healthcare; Gold answers. 19. DatearXiv IDTitle 2025-11-14 2511.11347Privacy Challenges and Solutions in Retrieval-Augmented Generation-Enhanced LLMs for Healthcare Chatbots: A Review of Applications, Risks, and Future Directions 2025-09-18 2509.14581Can I Trust This Chatbot? Assessing User Privacy in AI-Healthcare Chatbot Applications 2025-02-04 2502.02749Unveiling Privacy and Security Gaps in Female Health Apps 2024-10-18 2410.14607Evaluating Privacy Measures in Healthcare Apps Predominantly Used by Older Adults 2024-05-28 2405.17971A Qualitative Analysis Framework for mHealth Privacy Practices 2023-12-15 2312.10214Healthcare Policy Compliance: A Blockchain Smart Contract-Based Approach 2023-11-09 2311.05404A Survey on Privacy of Health Data Lifecycle: A Taxonomy, Review, and Future Directions 2023-07-24 2307.12542Client-Level Differential Privacy via Adaptive Intermediary in Federated Medical Imaging 2023-06-20 2306.11557 A Comparative Audit of Privacy Policies from Healthcare Organizations in USA, UK and India 2023-06-10 2306.06448HIPAAChecker: The Comprehensive Solution for HIPAA Compliance in Android mHealth Apps 2023-06-09 2306.06033SoK: Analysis of User-Centered Studies Focusing on Healthcare Privacy & Security 2023-02-08 2302.04208Exploratory Analysis of Federated Learning Methods with Differential Privacy on MIMIC-I 2022-11-21 2211.11434Privacy in Practice: Private COVID-19 Detection in X-Ray Images (Extended Version) 2022-05-06 2205.03168Defending against Reconstruction Attacks through Differentially Private Federated Learning for Classification of Heterogeneous Chest X-Ray Data 2021-06-26 2106.13973Benchmarking Differential Privacy and Federated Learning for BERT Models 2021-06-11 2106.06136Security and Privacy for Healthcare Blockchains 2020-09-17 2009.08294Robust Aggregation for Adaptive Privacy Preserving Federated Learning in Healthcare 2020-08-13 2008.05864An Empirical Evaluation of GDPR Compliance Violations in Android mHealth Apps 2019-10-02 1910.00962Privacy-preserving Federated Brain Tumour Segmentation E Prompt Templates To make the LLM-assisted components of ScholarQuest transparent, we provide the main prompt templates used in query construction, answer discovery, agentic search, and relevance filtering. The templates serve the following roles: •Query Generation Prompt. Generates one taxonomy-guided paper-retrieval query for each intent type from a given topic seed. •Query Rewrite Prompt. Rewrites a benchmark query into multiple first-round search anchors while preserving the original retrieval intent. •Scorer Prompt. Asks a relevance judge to assess whether a candidate paper satisfies the user query based on its title and abstract. •Strict Batch Filter Prompt. Performs conservative batched relevance filtering for final answer curation, requiring explicit satisfaction of all core query constraints. Query Generation Prompt You generate paper retrieval queries from one topic seed. Return strict JSON only. The response must be one valid JSON object with this schema: "seed_id": "PTS_000001", "topic_seed": "large language model agents", "domain": "cs.AI", "queries": [ "category": "method_capability", "constraint_kind": "training_paradigm", "constraint_value": "reinforcement learning", "query": "Which papers study large language model agents trained with reinforcement learning?", "rationale": "reinforcement learning is a searchable technical constraint", "risk_flags": [] ] Generate exactly one query for each query category: - method_capability - setting_anchor - claim_comparison - scope_control Rules: - Produce exactly four query objects, one per category. - Each query must be a paper retrieval request. - Keep each query concise, natural, and one sentence. - Include one strong executable retrieval constraint per query. - Stay faithful to the topic seed, domain, and ACM id. - Avoid temporal wording such as recent, latest, since 2020, after 2020, or similar time constraints. - Do not ask for analysis, advice, or long-form synthesis. - Do not use fragment wording such as "Papers on ...". - Vary the surface form across the four queries when natural. Query Rewrite Prompt You rewrite one paper-search query into first-round search anchors. Requirements: - Return exactly 10 English queries. - Keep every query close to the original intent. - Vary the semantic angle, retrieval scope, terminology, method, setting, task, evidence type, or comparison focus. - Make the 10 queries suitable as independent first-round search inputs for finding answer papers. - Keep each query concise and retrieval-oriented. - Avoid duplicates and near-duplicates. - Avoid drifting into a different research problem. - Do not mention citations, references, hops, or tool usage. - Do not output markdown or explanations. Return strict JSON only: "queries": ["...", "..."] Scorer Prompt You are an elite researcher in the field of AI, conducting research on user_query. Evaluate whether the following paper fully satisfies the detailed requirements of the user query and provide your reasoning. Ensure that your decision and reasoning are consistent. Searched Paper: Title: title Abstract: abstract User Query: user_query Output format:+++++++++++++++++ Decision: True/False Reason: ... Decision: Strict Batch Filter Prompt You are a strict relevance judge for academic paper filtering. Your task is to evaluate whether each candidate paper strictly matches the same user query. Judge based on the user query, paper title, and paper abstract. Scoring rules: - 2 = Strict match. The paper fully matches every core requirement in the query, including the target topic, method/task, setting, constraints, and any specified conditions. All important points in the query must be clearly supported by the title or abstract. - 1 = Partial match or missing/violated constraint. The paper is related to part of the query, but at least one core requirement is missing, unclear, too broad, or contradicted. If a query constraint is not explicitly matched, assign 1 rather than 2. - 0 = Mismatch. The paper does not match the query, or the main subject/intent differs from the query. Confidence: - high = The title and abstract provide enough evidence for the decision. - medium = The evidence is somewhat incomplete but the decision is still reasonably supported. - low = The decision is uncertain due to limited or ambiguous information. Be conservative. Assign 2 only when the match is explicit and complete. Do not infer missing constraints from vague similarity. Output valid JSON only: "results": [ "paper_index": 1, "arxiv_id": "...", "reason": "brief English reason based on the query, title, and abstract", "strict_score": 0, "confidence": "low" ] Return exactly one result for each input paper. Preserve the input paper_index and arxiv_id. Within each result, place "reason" before "strict_score", and place "confidence" last. Do not output markdown or extra text. USER QUERY: query PAPER CANDIDATES JSON: papers_json