Paper deep dive
Context Selection for Hypothesis and Statistical Evidence Extraction from Full-Text Scientific Articles
Sai Koneru, Jian Wu, Sarah Rajtmajer
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/26/2026, 2:24:54 AM
Summary
This paper presents a two-stage retrieve-and-extract framework for identifying hypotheses and their supporting statistical evidence within full-text scientific articles. The authors evaluate various context selection strategies, including standard RAG, reranking, and a fine-tuned retriever, to address the challenges of document length and rhetorical complexity. Results indicate that while targeted context selection improves hypothesis extraction, statistical evidence extraction remains difficult even with oracle context, highlighting persistent limitations in LLM reasoning for hybrid numeric-textual statements.
Entities (5)
Relation Signals (3)
Sai Koneru → authored → Context Selection for Hypothesis and Statistical Evidence Extraction from Full-Text Scientific Articles
confidence 100% · Sai Koneru, Jian Wu, and Sarah Rajtmajer. 2026. Context Selection for Hypothesis and Statistical Evidence Extraction from Full-Text Scientific Articles.
RankGPT → performs → Reranking
confidence 90% · then apply RankGPT to rerank and retain the top-k paragraphs.
SciBERT → usedin → Retrieval-Augmented Generation
confidence 90% · In each stage, we use a SciBERT retriever to select the top-k paragraphs
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Extracting hypotheses and their supporting statistical evidence from full-text scientific articles is central to the synthesis of empirical findings, but remains difficult due to document length and the distribution of scientific arguments across sections of the paper. The work studies a sequential full-text extraction setting, where the statement of a primary finding in an article's abstract is linked to (i) a corresponding hypothesis statement in the paper body and (ii) the statistical evidence that supports or refutes that hypothesis. This formulation induces a challenging within-document retrieval setting in which many candidate paragraphs are topically related to the finding but differ in rhetorical role, creating hard negatives for retrieval and extraction. Using a two-stage retrieve-and-extract framework, we conduct a controlled study of retrieval design choices, varying context quantity, context quality (standard Retrieval Augmented Generation, reranking, and a fine-tuned retriever paired with reranking), as well as an oracle paragraph setting to separate retrieval failures from extraction limits across four Large Language Model extractors. We find that targeted context selection consistently improves hypothesis extraction relative to full-text prompting, with gains concentrated in configurations that optimize retrieval quality and context cleanliness. In contrast, statistical evidence extraction remains substantially harder. Even with oracle paragraphs, performance remains moderate, indicating persistent extractor limitations in handling hybrid numeric-textual statements rather than retrieval failures alone.
Tags
Links
- Source: https://arxiv.org/abs/2603.21193v1
- Canonical: https://arxiv.org/abs/2603.21193v1
Trouble viewing inline? Open PDF directly →
Full Text
73,687 characters extracted from source content.
Expand or collapse full text
Context Selection for Hypothesis and Statistical Evidence Extraction from Full-Text Scientific Articles Sai Koneru saidileep543@gmail.com Pennsylvania State University University Park, Pennsylvania, USA Jian Wu j1wu@odu.edu Old Dominion University Norfolk, Virginia, USA Sarah Rajtmajer smr48@psu.edu Pennsylvania State University University Park, Pennsylvania, USA Abstract Extracting hypotheses and their supporting statistical evidence from full-text scientific articles is central to the synthesis of em- pirical findings, but remains difficult due to document length and the distribution of scientific arguments across sections of the paper. The work studies a sequential full-text extraction setting, where the statement of a primary finding in an article’s abstract is linked to (i) a corresponding hypothesis statement in the paper body and (i) the statistical evidence that supports or refutes that hypothe- sis. This formulation induces a challenging within-document re- trieval setting in which many candidate paragraphs are topically related to the finding but differ in rhetorical role, creating hard negatives for retrieval and extraction. Using a two-stage retrieve- and-extract framework, we conduct a controlled study of retrieval design choices, varying context quantity, context quality (stan- dard Retrieval Augmented Generation, reranking, and a fine-tuned retriever paired with reranking), as well as an oracle paragraph setting to separate retrieval failures from extraction limits across four Large Language Model extractors. We find that targeted con- text selection consistently improves hypothesis extraction relative to full-text prompting, with gains concentrated in configurations that optimize retrieval quality and context cleanliness. In contrast, statistical evidence extraction remains substantially harder. Even with oracle paragraphs, performance remains moderate, indicating persistent extractor limitations in handling hybrid numeric-textual statements rather than retrieval failures alone. CCS Concepts • Computing methodologies→ Information extraction. Keywords Scientific Information Extraction, Retrieval-Augmented Generation, Large Language Models, Evidence Extraction, Hypothesis Extrac- tion ACM Reference Format: Sai Koneru, Jian Wu, and Sarah Rajtmajer. 2026. Context Selection for Hypothesis and Statistical Evidence Extraction from Full-Text Scientific Articles. In . ACM, New York, NY, USA, 14 pages. https://doi.org/10.1145/ n.n 1 Introduction Hypotheses are central units of scientific method. They express testable expectations about relationships between constructs and variables [44], guiding how studies are designed, analyzed, and Conference’17, Washington, DC, USA 2026. ACM ISBN 978-x-x-x-x/Y/M https://doi.org/10.1145/n.n interpreted. In scientific articles, hypotheses are typically stated conceptually and may be paraphrased across multiple sections [17], while their empirical support is reported as statistical evidence which are hybrid numeric-textual statements with effect directions, coefficients, test statistics, confidence intervals, and p-values [22]. Extracting these hypotheses together with the evidence used to evaluate them is therefore a key step toward the scalable synthesis of empirical findings. However, remarkable growth in the number of publications each year has made it increasingly difficult to syn- thesize these claims at scale [6], motivating automated approaches. Prior efforts toward this goal have primarily focused on claim extraction to support various downstream applications, including automated literature review [30,31], hypothesis generation [34], and fact-checking [40]. Although recent advances in Large Lan- guage Models (LLMs) have shown promise for text summarization and question-answering [7], much of existing work on scientific statement extraction has largely focused on identifying claim like sentences within abstracts [1,42]. This abstract-level focus does not address hypothesis–evidence extraction, where hypotheses and their supporting statistical evidence often appear in different parts of a paper’s full text. Full-text extraction is challenging not only because of document length, but also because scientific reasoning is distributed across sec- tions and embedded in layered argumentation and diverse rhetorical structures [24]. Although most state-of-the-art LLMs have suffi- cient context windows large enough to fit the full text of a scientific paper as input in a single pass [5,46], existing evaluations show that end-to-end performance is well below human level [11,35,48]. In full text, the relevant hypothesis or evidence is often sparse, paraphrased, and surrounded by topically related but non-target text, so accurate extraction depends on careful context selection. Our focus in this work follows the structure of data introduced by Alipourfard et al. [2], which links a high-level abstract finding to (i) a corresponding core hypothesis in the paper body, and (i) the statistical evidence that supports that hypothesis. Specifically, we structure claim trace extraction into two consecutive sub-tasks that differ in their characteristics. Hypothesis extraction is a semantic alignment task; hypotheses may be paraphrased, expressed at vary- ing levels of specificity, and distributed across multiple sections. In contrast, evidence extraction focuses on identifying spans con- taining dense statistical information (e.g., numerical values, test results), which typically appear in Results (Figure 1). We propose a consistent two-stage retrieve-and-extract pipeline (Figure 2). In Stage 1, we use the pre-extracted finding statement in the abstract as a query to retrieve candidate paragraphs, from which an LLM extracts a single core hypothesis or its paraphrased variants corresponding to the finding statement. In Stage 2, we construct a composite query by concatenating the abstract finding arXiv:2603.21193v1 [cs.CL] 22 Mar 2026 Conference’17, July 2017, Washington, DC, USAKoneru et al. Figure 1: An example claim trace spanning the Abstract, In- troduction, and Results sections of a paper. Our two-stage pipeline uses an abstract-level finding to extract a hypothe- sis statement (from the Introduction, in this case) and then locate statistical evidence (Results). with the extracted hypothesis to retrieve candidate paragraphs for statistical evidence extraction. We position this two stage pipeline within the Retrieval Augmented Generation (RAG) paradigm at each stage and compare it against full-text prompting baselines to isolate the effects of context selection. Our key results are obtained through extensive controlled abla- tions within this paradigm, varying: (i) retrieval quantity, retrieving top-푘paragraphs (푘 ∈ 5,10,20); (i) retrieval quality, comparing standard dense retrieval, reranked retrieval, and a fine-tuned re- triever paired with the same reranker; and (i) extractor limitations via stage-specific oracle contexts that provide the gold hypothesis paragraph in Stage 1 and the gold evidence paragraph in Stage 2, bounding performance and separating retrieval failures from extraction failures. We find: • Task-dependent bottlenecks. Hypothesis extraction is strongly bounded by retrieval quality and context cleanliness, whereas evidence extraction remains only moderately accurate even in the oracle setting (oracle Evidence F1 between 0.47 and 0.55 across models). •Context-based retrieval outperforms full-text for hypothe- ses extraction. Increasing retrieved context from푘=5→10→ 20 is generally beneficial or neutral for hypothesis extraction, where as full-text prompting often underperforms context-based retrieval. • Retrieval improvements yield gains but leave substantial headroom. Reranking and fine-tuned retrieval improve perfor- mance in both stages, but evidence extraction remains far from oracle even when the gold paragraph is provided. This shows that improving the retriever alone is not enough, the extractor itself is a significant bottleneck. Together, these experiments provide a controlled comparison of context selection for hypothesiscand evidence extraction in a within-document setting, including oracle contexts that separate paragraph-selection effects from extractor limitations. 1 2 Related Work 2.1 Statement extraction from scientific text Prior work on extracting scientific statements (e.g., claims and hy- potheses) from scholarly text has often formulated the problem as sentence/span detection or sequence tagging. Early efforts concen- trated on claim sentence identification from scientific abstracts by using discourse structure and transfer learning to distinguish claim- like sentences from surrounding context [1,19]. More recent work strengthens claim sentence classifiers with objectives such as super- vised contrastive learning to separate claim sentences from topically similar non-claims [42]. Complementary lines of research move beyond sentence-level detection toward structured representations of scientific statements, extracting fine-grained claim graphs that encode entities, relations, and modifiers for downstream scientific knowledge graph construction [28]. Rhetorical zoning and sequential sentence classification pro- vide mechanisms for distinguishing statement types (e.g., back- ground, objective, methods, results, conclusion), helping separate hypothesis/claim-like content from other discourse roles [12,37]. Citation-focused resources further capture how scientific assertions are situated and supported, for example through citation intent clas- sification and related citation/discourse modeling [9]. A recurring theme in this literature is that moving from abstracts to full papers introduces both scale and ambiguity, as targets may be sparse, paraphrased across sections, and interleaved with rhetor- ically adjacent but non-target statements [43]. Document-level sci- entific information extraction benchmarks underscore the difficulty of extracting structured information when evidence spans multiple sentences or must be aggregated across a document [18,27]. Be- cause scientific claims and hypotheses are embedded in structured discourse, extraction often benefits from discourse cues and argu- mentation structure [14,23,24]. Finally, prior work also emphasizes definitional ambiguity and annotation challenges that vary across tasks and communities [39, 44]. Our work moves beyond sentence-level claim and hypothesis identification to extract structured links between findings, hypothe- ses, and supporting evidence in full papers. We address full-text ambiguity by systematically varying how context is constructed, separating paragraph-selection errors from extractor limitations. 1 Code, prompts, and experiment configurations are available at https://github.com/ SaiDileepKoneru/ScientificClaimTraceExtraction.git Context Selection for Hypothesis and Statistical Evidence Extraction from Full-Text Scientific ArticlesConference’17, July 2017, Washington, DC, USA 2.2 Evidence retrieval for claim verification A related line of work studies scientific claim verification, where sys- tems must retrieve relevant documents and identify evidentiary sen- tences supporting a veracity decision. Research on general-domain fact verification and rationale evaluation uses the retrieve-then- reason paradigm with explicit evidence selection [13,38]. In the scientific domain, this involves retrieving evidence-containing ab- stracts and selecting rationale sentences that support or refute a claim [40]. Subsequent work emphasizes the importance of retrieval and context scope by decomposing the task into abstract retrieval, sentence selection, and label prediction [32], and incorporating full-document context and weak supervision to improve robustness and domain transfer [41]. We adopt a similar retrieve-then-extract paradigm adapted to extract structured claim traces rather than predict veracity labels. Additionally, while verification pipelines typically retrieve evidence across multiple documents, our setting targets extraction of hypothesis-linked, statistical evidence within a single document to reconstruct a paper’s empirical argument. 2.3 Impact of retrieval configuration Recent benchmarks for scientific long-context understanding show that providing models with long inputs alone does not guarantee reliable scientific comprehension and reasoning across tasks [11]. This motivates careful retrieval configuration because irrelevant yet semantically related passages can actively mislead downstream LLM behavior and degrade correctness in retrieval-augmented set- tings [3,10,47]. Controlled studies demonstrate that LLMs are distractable. Adding irrelevant context can skew responses even when the prompt otherwise contains useful information, and this effect is strongest when the irrelevant context is highly similar to the target [33,45], a scenario inherently common in within- document retrieval. Beyond relevance, the length and location of context provided also impacts performance with prior work on long-context evaluations documenting lost-in-the-middle behavior where models underuse relevant information buried mid-context [26]. Furthermore, studies suggest that increasing the amount of re- trieved context offers diminishing returns, improving performance initially but potentially hurting it for long inputs [20]. Interplay between retrieval recall and precision is also complex; subsequent filtering or selection steps may only be helpful if the initial retrieval achieves high recall [25]. Together, these findings motivate two-stage retrieval designs that prioritize high recall in the first stage of retrieval and then im- prove precision via reranking retrieved paragraphs into a candidate set [8,16]. We follow this approach. We first retrieve a high-recall candidate set and then use reranking to improve precision, en- abling controlled comparisons of context quantity and quality in the presence of highly similar within-document distractors. 3 Task Description Our goal is to extract a structured trace of an empirical argument from the full text of a given scientific paper. A key assumption of ours is that claims are empirical in nature, so we focus exclusively on extracting statistical evidence that supports a stated hypothesis and do not consider other forms of scholarly reasoning, such as theoretical arguments. Formally, let a scientific article퐷be modeled as a sequence of paragraphs(푝 1 ,푝 2 , . . .,푝 푛 ), where each paragraph푝 푖 is a sequence of sentences푝 푖 =(푠 푖,1 ,푠 푖,2 , . . .,푠 푖,푚 푖 ). During preprocessing, we rep- resent the article body as plain-text paragraphs only, excluding non-prose content such as tables, figures (and captions), and equa- tions, so each푝 푖 corresponds to a contiguous block of running text from the main body. Let퐹 퐴 denote an abstract finding sentence from the Abstract of퐷. Our objective is to extract two text spans from the body of 퐷 : •A core hypothesis퐻is an interpretive or hypothesis-level state- ment related to the abstract finding.퐻may be a single sentence or span multiple consecutive sentences. •Statistical evidence퐸(퐻)is a text span containing statistical re- sults (e.g., p-values, confidence intervals, regression coefficients) that substantiate the hypothesis 퐻 . This task presents several unique challenges that differentiate it from standard information extraction. Notably, the link from a finding reported in the Abstract to statistical evidence is typically indirect. Statistical evidence supports a hypothesis, which in turn supports the finding. 4 Methodology 4.1 Extraction pipeline We frame the task as a sequential, two-step process that first extracts a hypothesis and then extracts the statistical evidence supporting that hypothesis (see Figure 2). Stage 1: Hypothesis extraction. We use the abstract finding as a query to identify paragraphs most likely to contain the core hypothesis in the full text. We first convert the article body into paragraphs and then construct a stage-specific context set퐶 1 us- ing one of the context configurations in Section 4.2. Specifically, retrieval-based configurations first retrieve a candidate pool of size 푀(e.g.,푀=푘for standard RAG, or푀=30 when using a reranker), and optionally apply an LLM reranker (RankGPT [36]) to select a final top-푘context set. For reranked configurations, we use푀=30 and retain푘=5 paragraphs, matching the fixed context size in Ta- ble 1. Full-text configurations skip retrieval and set퐶 1 to the entire article body (excluding abstract and bibliography). Given the result- ing context퐶 1 and the finding퐹 퐴 , an LLM extractor generates the hypothesis ˆ 퐻 . Stage 2: Evidence extraction. We form a composite query by concatenating the abstract finding with the extracted hypothesis 푞 2 =[퐹 퐴 ; ˆ 퐻], and construct an evidence-stage context set퐶 2 using the same retrieval and reranking procedure (or full-text/oracle con- text, depending on configuration). The extractor is then prompted with퐹 퐴 , ˆ 퐻 , and퐶 2 to extract the statistical evidence ˆ 퐸 . For all extraction steps, we use prompts adapted directly from the hu- man annotator guidelines in [2]. Full prompt templates (hypoth- esis extraction, evidence extraction, and evaluator parsing) and experiment configuration files are available in the accompanying repository. 1 4.2 Context Configurations We compare the following context construction configurations, each instantiated within the sequential two-stage pipeline: Conference’17, July 2017, Washington, DC, USAKoneru et al. Figure 2: Schematic of the sequential retrieve and extract pipeline. Stage 1 (top) uses the abstract finding to extract the Hypothesis, which then serves as part of a composite query for extracting the Statistical Evidence in Stage 2 (bottom). Zero-Shot (Full text, two-stage): This baseline uses the same two-stage extraction pipeline but replaces retrieved contexts with the full article body. In Stage 1, we provide the full text (excluding bibliography) and퐹 퐴 and prompt the LLM to extract ˆ 퐻. In Stage 2, we provide the full text again together with퐹 퐴 and ˆ 퐻and prompt the LLM to extract ˆ 퐸. This isolates the effect of context selection while keeping the sequential formulation unchanged. CoT (Full text, two-stage): This baseline is identical to the full- text two-stage baseline, but uses chain-of-thought prompting [21] within each stage (i.e., the model produces intermediate reasoning before outputting ˆ 퐻in Stage 1 and ˆ 퐸in Stage 2). This tests whether improved prompting alone can offset the lack of context selection. Standard RAG: In each stage, we use a SciBERT retriever to select the top-푘paragraphs from the article body and pass them to the LLM extractor. We evaluate 푘 ∈ 5, 10, 20. Standard RAG + reranker: In each stage, we retrieve a candi- date pool of푀=30 paragraphs using SciBERT, then apply RankGPT to rerank and retain the top-푘 paragraphs. We use 푘=5. Fine-tuned retriever + reranker: This process is driven by a task-aware paragraph retriever that we fine-tuned to distinguish rhetorical function of paragraphs. The retriever is built upon a bi- encoder architecture, using SciBERT [4] as a shared bi-Encoder to generate dense representations of the query and each candidate paragraph. To better distinguish the rhetorical functions between two paragraphs, the embeddings output by SciBERT are passed through two separate linear projection heads. We treat hypothesis retrieval (Stage 1) and evidence retrieval (Stage 2) as two related tasks implemented via separate projection heads over the shared SciBERT encoder. The model is fine-tuned using a multi-task triplet loss, minimizing the distance to positive paragraphs while maxi- mizing the distance to negative ones. To optimize the multi-task triplet loss, we used a consistent hard negative identification strategy for both the hypothesis and evi- dence tasks. For each training example, the ground-truth paragraph was used as a positive point. To identify a hard negative point, we search the rest of the document for the paragraph that is most semantically similar to the positive paragraph, as measured by the cosine similarity of their SciBERT embeddings (Appendix Figure 7 shows an example of a selected hard negative). To ensure we did not select unannotated paraphrases of the hypothesis, we filtered out candidates where any constituent sentence exceeded a similar- ity threshold of 0.89 with the human annotated hypothesis. This threshold matches the calibrated semantic-match criterion used in hypothesis evaluation (Section 5.1). This produces challenging training examples in which the model must distinguish between highly similar paragraphs that differ in rhetorical role with respect to the query. This process was iden- tical for both tasks, differing only in the anchor query with the abstract finding for the hypothesis task, while the composite query formed by concatenating the finding from abstract with ground truth hypothesis was used for the evidence task. During inference, we use the fine-tuned retriever with the same reranking setup as the RAG + Reranker baseline. We retrieve 30 candidates, rerank with RankGPT, and provide the top 5 paragraphs to the extractor. Oracle setting: To establish an upper bound under perfect para- graph selection, we provide the extractor with gold paragraphs. In Stage 1, we set퐶 1 to the paragraph containing the human-annotated hypothesis. In Stage 2, we provide the ground-truth hypothesis (to avoid error propagation) and set퐶 2 to the paragraph containing the human-annotated statistical evidence. This isolates extractor limitations from retrieval failures. 5 Experiments We conduct our experiments on the dataset introduced in Alipour- fard et al. [2], comprised of 2,257 full-text scientific articles from social and behavioral science (SBS) disciplines. Each document in- cludes a single domain expert-annotated claim trace consisting of a high-level conceptual finding from the abstract, an interpre- tive or hypothesis-level statement from the body text (hypothesis), and related statistical result (evidence) [15]. On average, each docu- ment contains 18,268 tokens and 367 sentences after pre-processing, which involved removing the abstract and bibliography sections. Context Selection for Hypothesis and Statistical Evidence Extraction from Full-Text Scientific ArticlesConference’17, July 2017, Washington, DC, USA 5.1 Evaluation Metrics We evaluate hypothesis and evidence extraction with metrics tai- lored to the properties of each output. Throughout, we report micro-averaged Precision/Recall/F1 computed over extracted units (sentences for hypotheses; structured components for evidence), aggregated over all matched units in the evaluation set. Hypothesis Extraction (semantic match). Hypotheses are frequently paraphrased across a paper, so exact string overlap is overly strict. We therefore evaluate hypothesis extraction using a calibrated se- mantic matching criterion. We first construct a labeled calibration set by sampling model-extracted hypothesis from a held-out split and rating each extraction against the ground-truth hypothesis using a four-level rubric: exact match (3), mostly similar (2), par- tially similar (1), and different (0). Full annotation guidelines and examples are available in the accompanying code repository. 1 We then binarize these labels by treating scores 2-3 as Correct and 0-1 as Incorrect. We embed predicted and gold hypothesis text using Gemini embeddings with theSEMANTIC_SIMILARITYtask type, which is optimized for semantic textual similarity (e.g., recommendation and duplicate-detection use cases), and compute cosine similarity. We select a similarity threshold휏on the labeled calibration set via a grid search over휏 ∈ [0.1,1.0]with step size 0.01, choosing the value that maximizes F1 (Appendix Figure 5). We use the resulting threshold, 휏=0.89, as the semantic-match cutoff in all hypothesis evaluations. At evaluation time, we split the predicted hypothesis span into sentences and count a predicted sentence as a match if its cosine similarity to the gold hypothesis (or any gold sentence, when the gold span contains multiple sentences) is at least휏. Precision, recall, and F1 are then computed over matched vs. unmatched predicted sentences and gold sentences. This procedure rewards semantically equivalent paraphrases while penalizing additional non-hypothesis content included in the extracted span. Appendix Figure 6 shows a representative extraction failure case. Evidence Extraction (component-level scoring). Statistical evidence is typically hybrid numeric-textual content, requiring evaluation that jointly measures semantic alignment and field-level correct- ness (variables, tests, coefficients, intervals). Exact string overlap is sensitive to rephrasing, and embedding similarity may not reflect numerical fidelity. Accordingly, we use a component-level scor- ing procedure consistent with structured evaluation paradigms in recent scientific benchmarks [29]. Component schema. We manually inspected 50 randomly sam- pled gold evidence spans to identify the most consistently expressed fields in prose statistical reporting, and define a fixed set of five component types aligned with our evaluator prompts: (i) variables (key predictors/outcomes and constructs); (i) effect relationships (e.g., directionality and significance statements); (i) confidence intervals; (iv) statistical test/model family; and (v) sample size / degrees of freedom (when stated). Evaluator-assisted component parsing. Given a predicted evidence span and its corresponding gold span, we use a fixed evaluator LLM (GPT-4) with fixed, component-specific prompts to parse both texts into structured JSON fields for each component type. 1 For set-like components (variables, relationships, confidence intervals), the evaluator returns the number of items in the gold statement (|퐴|), the number of items in the extracted statement (|퐵|), and a fuzzy-matched intersection count (|퐴∩ 퐵|). For binary components (test/model family; sample size/df ), the evaluator re- turns a match indicator and the extracted strings from each text. For binary components, we treat both-absent as not applicable: when the evaluator returns empty strings for both the gold and predicted fields (e.g.,gold_test= predicted_test= ""), we exclude that component instance from scoring rather than counting it as an error. The evaluator is used only for parsing; scoring is computed deterministically from the returned fields. Aggregating precision/recall/F1. After parsing, each compo- nent type yields a set of extracted items (e.g., a list of variables; a list of confidence intervals) or a single attribute (e.g., test/model family; sample size/df ). We compute precision/recall/F1 by count- ing matches over these component items. Each correctly matched component item contributes one true positive; each component item produced by the system that is not supported by the gold statement contributes one false positive; and each gold component item that is not recovered contributes one false negative. For set- like components, this corresponds to counting overlap between the parsed gold and predicted item lists (with fuzzy matching for vari- ables and confidence intervals). For binary components (test/model family; sample size/df ), we score agreement between the parsed attributes as a match and disagreement as an error. When a compo- nent is absent in both the gold and predicted evidence, we exclude it from scoring. For set-like components, absence corresponds to empty sets (|퐴|= |퐵|=0). For binary components (test/model family and sample size/df ), absence corresponds to both extracted strings being empty (e.g.,gold_test= predicted_test= ""), in which case the instance is excluded. Otherwise, we score a binary component as one item: if both sides are present andmatch=1, it contributes one true positive; if gold is present and predicted is empty, it contributes one false negative; if gold is empty and predicted is present, it contributes one false positive; and if both are present butmatch=0, it contributes one false positive and one false negative. This component-based evaluation assigns partial credit when an extraction correctly recovers some fields (e.g., variables and direc- tion) but omits or misstates others (e.g., confidence intervals). Since the scorer relies on an LLM-based parser for component extraction, it depends on the evaluator model and prompts; we therefore inter- pret it as an automated proxy for field-level (structured) fidelity of statistical evidence extraction. Figure 3 provides a concrete example of the parsed components and how they contribute to partial credit. 6 Results Table 1 reports Precision/Recall/F1 for the two-stage pipeline across four extractors and all context configurations. We evaluate Stage 1 and Stage 2 separately. Unless noted otherwise, we summarize trends averaged across extractors. We organize the results around three inquiries: how retrieved context quantity affects performance under standard RAG (푘 ∈ 5,10,20); how retrieved context quality affects performance at fixed context size (standard RAG푘=5→RAG+reranker푘=5→ fine-tuned retriever+reranker 푘=5); and what oracle contexts re- veal about retrieval versus extraction limits. Additional diagnostics, Conference’17, July 2017, Washington, DC, USAKoneru et al. including the per-paper transition analysis and evidence extraction conditioned on ground-truth hypotheses, are reported in Appendix (Figure 8, Table 2). All configurations use the same two-stage for- mulation; they differ only in how퐶 1 and퐶 2 are constructed (full text, top-푘retrieval, retrieve-then-rerank to a fixed푘=5, or oracle gold paragraphs). Hypothesis extraction benefits reliably from targeted context selection. Averaged across extractors, full-text prompting underper- forms retrieval-based contexts (mean F1 0.39 vs. 0.43/0.47/0.49 for standard RAG with푘=5/10/20), and the best non-oracle configura- tion (fine-tuned retriever+reranker,푘=5) reaches mean hypothesis F1 0.50 (about +0.11 over full text). These gains are consistent with a signal-dilution effect in full papers where many paragraphs are topically related to the abstract finding but rhetorically non-target. Evidence extraction shows a different sensitivity to context. On average, standard RAG with a small context (푘=5) underperforms full-text prompting (mean Evidence F1 0.20 vs. 0.24), but increasing 푘improves performance (mean Evidence F1 0.23 at푘=10 and 0.25 at 푘=20). Improving retrieval quality yields clearer gains at fixed size (RAG+Reranker푘=5 F1 0.26, Fine-tuned Retriever+Reranker푘=5 F1 0.29). Oracle contexts reveal a moderate ceiling (oracle Evidence F1 0.47-0.55 across models). This indicates that evidence extraction is constrained not only by retrieval, but also by extractor performance. 6.1 Effect of retrieved context quantity (푘) We first vary retrieved context quantity under standard RAG by retrieving the top-푘paragraphs with SciBERT (푘 ∈ 5,10,20), holding the extractor and prompting fixed. For hypothesis extrac- tion, performance generally increases with푘and can saturate for some extractors (Table 1), consistent with hypotheses being para- phrased and distributed across multiple locations. For evidence extraction, increasing푘improves performance across extractors (Table 1), though performance gains are smaller and more model- dependent than for hypothesis extraction. Importantly, full-text prompting is not equivalent to large-푘retrieval. For hypotheses, full-text inputs underperform even the largest retrieved contexts despite having maximal recall, suggesting signal dilution from top- ically similar but rhetorically non-target paragraphs. For evidence, full text can remain competitive for some extractors, indicating that broader surrounding context may sometimes help models locate and interpret statistical reporting. 6.2 Effect of retrieval quality at fixed 푘=5 We next isolate retrieval quality by holding the final context size fixed (푘=5) and comparing three retrieval variants: Standard RAG, RAG+Reranker (retrieve 30 with SciBERT, then use RankGPT to select the top 5), and Fine-tuned Retriever+Reranker (replace SciB- ERT with the fine-tuned retriever while keeping the same rerank- ing procedure). Across models, improving retrieval quality yields consistent gains for both subtasks (Table 1), but with different im- plications. For hypothesis extraction, reranking and fine-tuning primarily help by selecting rhetorically appropriate hypothesis statements from many topically similar within-document candidates: average F1 increases from 0.43 (Standard RAG,푘=5) to 0.47 (RAG+Reranker) Example: Component-level comparison for evidence extraction DOI. 10.1016/j.jesp.2017.11.013 Gold evidence (annotated). Out-group members were described less frequently in terms of their basic mental states (estimated marginal mean = 9.70%) than were in-group members (estimated marginal mean = 27.60%),퐹(1,68)=14.948,푝= .000. Extracted evidence. Participants writing about presumed out-group members referenced their complex mental states less frequently (estimated marginal mean = 0.5%) than those writing about presumed in-group members (estimated marginal mean = 3.6%; 퐹(1, 68)= 8.457, 푝= .005). Parsed component overlap (evaluator parses; scoring is deterministic). ComponentOverlap / MatchNotes Variables (set)|퐴∩ 퐵|= 2 (of |퐴|= 3, |퐵|= 7)partial overlap Relationships (set)|퐴∩ 퐵|= 2 (of |퐴|= 2, |퐵|= 5)extra predicted relations Confidence intervals (set)NA (absent in both)not scored Test / model family (binary)1F-test family Sample size / df (binary)1푑푓= 68 Figure 3: Illustration of component-level evaluation of evi- dence extraction. The metric assigns partial credit when the extracted span captures the relationship but corresponds to a different reported effect with different numeric details. and to 0.50 (Fine-tuned Retriever+Reranker). For evidence extrac- tion, retrieval quality improvements also help, but the gains are consistent with improved retention of evidence-relevant paragraphs at a fixed context size. Notably, the fine-tuned retriever can also introduce trade-offs by changing which paragraphs are retained at fixed context size; we analyze these per-paper retention and loss patterns in Appendix (Figure 8). 6.3 Oracle contexts determine extraction limits Finally, we evaluate an oracle context setting where the extractor is given the gold paragraph containing the human-annotated hypoth- esis or evidence. This removes retrieval failures and provides an upper bound on extraction performance under perfect paragraph selection. For hypothesis extraction, oracle contexts substantially improve performance across all extractors (mean Hypothesis F1 0.73), leav- ing a large gap relative to the best non-oracle configuration (Fine- tuned Retriever+Reranker, mean F1 0.50). This indicates that hy- pothesis extraction is strongly constrained by context selection, but that extraction still has substantial headroom even when the correct paragraph is provided. For evidence extraction, oracle contexts reveal a different bottle- neck: even with perfect paragraph selection, performance remains moderate (oracle Evidence F1 0.47-0.55 across models; Table 1). This upper bound likely reflects a mix of extractor performance for hybrid numeric-textual statements and the fact that some statisti- cal details may be reported outside prose (e.g., in tables/figures), which we remove during preprocessing. The resulting oracle gap (e.g., Fine-tuned Retriever+Reranker 0.29 vs. oracle 0.52 on aver- age) suggests that evidence errors are driven not only by retrieval, but also by extractor fidelity for structured statistical content (e.g., preserving variables, directionality, and numeric fields). Context Selection for Hypothesis and Statistical Evidence Extraction from Full-Text Scientific ArticlesConference’17, July 2017, Washington, DC, USA Diagnostic analyses. To contextualize the aggregate scores in Ta- ble 1, we provide detailed diagnostics in the Appendix A. First, we analyze the relevant sentence proportion (RSP) to quantify the den- sity of hypothesis-relevant information. Across configurations, we find that higher signal density strongly correlates with improved F1 (Appendix Figure 4). Second, for evidence extraction conditioned on ground-truth hypotheses, we report a per-paper retrieval-outcome breakdown showing that improvements from retrieval changes are concentrated in cases where the gold evidence paragraph is newly retrieved, while regressions arise when reranking or retriever changes fail to retain that paragraph (Appendix Figure 8, Table 2). 7 Discussion A defining feature of our task is that retrieval is performed within a single paper. Candidate paragraphs are therefore often topically re- lated to the abstract finding, yet vary widely in rhetorical function (e.g., background framing, theoretical motivation, hypothesized relationships, results summaries, and implications). In this setting, improvements from reranking and the fine-tuned retriever are con- sistent with retrieval changes affecting which paragraphs appear in the final context under a fixed size. While we do not make mecha- nistic claims about model behavior, the aggregate patterns indicate that retrieval improvements translate more reliably into gains for hypothesis extraction than for evidence extraction. To complement aggregate metrics, we report two descriptive diagnostics (Appendix). First, we analyze relevant sentence propor- tion (RSP) as a proxy for context signal density for hypotheses: for a given hypothesis-stage context, RSP is the fraction of its sen- tences whose semantic similarity to the gold hypothesis exceeds the same calibrated cutoff used in our hypothesis evaluation. Higher RSP is consistently associated with higher hypothesis extraction F1 across configurations (Appendix Figure 4). Second, for evidence extraction conditioned on ground-truth hypotheses, we perform a per-paper transition analysis that buckets papers by whether a retrieval change Gains, Keeps, Loses, or Misses the gold evidence paragraph, and report the corresponding mean per-paperΔF1 (Ap- pendix Figure 8, Table 2). These analyses describe how retrieval changes alter the retrieved contexts and where gains are concen- trated (e.g., cases where the gold paragraph is newly retrieved), without attributing performance changes to specific internal model mechanisms. Oracle contexts further separate retrieval failures from extrac- tion limits. For hypotheses, the remaining gap between the best non-oracle setting and oracle performance indicates substantial headroom even when the correct paragraph is provided. For evi- dence, oracle performance remains moderate across models, which is consistent with evidence extraction requiring faithful handling of hybrid numeric-textual statements. In practice, improving this stage may require extraction methods that more strongly enforce structure (e.g., schema-constrained outputs) or hybrid approaches that combine retrieval with deterministic parsing for quantities such as coefficients, confidence intervals, and p-values. The diagnostics and analyses above suggest two best practices for building claim-trace extraction systems. First, for hypothesis extraction, investing in retrieval quality and retrieving moderately larger contexts are likely to significantly improve the performance. On the contrary, for evidence extraction, retrieval quality and con- text size matter, but gains remain limited without corresponding improvements in extractor fidelity for structured statistical con- tent. Together, these findings emphasize that full-text extraction pipelines should treat retrieval and extraction as coupled compo- nents: retrieval is a dominant factor for hypothesis extraction, while evidence extraction additionally depends on accurately preserving and aligning statistical fields. 8 Conclusion We study a sequential full-text claim-trace extraction problem, where an abstract-level finding is linked to (i) a corresponding hy- pothesis statement in the paper body and (i) the statistical evidence that supports that hypothesis. This formulation creates a challeng- ing within-document retrieval setting: candidate paragraphs are often topically related to the finding but differ in rhetorical role, and errors can propagate across stages. Using a consistent two-stage retrieve-and-extract pipeline, we present a controlled study of how retrieval configuration affects each stage, varying retrieved context quantity (top-푘 ∈ 5,10,20), retrieval quality (standard RAG, reranking, and a fine-tuned re- triever paired with reranking), and an oracle paragraph setting to separate retrieval failures from extraction limits. Across four LLM extractors, hypothesis extraction benefits reliably from tar- geted context selection: retrieval-based contexts outperform full- text prompting, and improving retrieval quality yields consistent gains. Evidence extraction improves more modestly with retrieval changes, and oracle contexts remain only moderately accurate, in- dicating that paragraph selection alone does not eliminate evidence extraction errors for structured statistical content. Together, these results highlight a task-dependent bottleneck in sequential full-text extraction: hypothesis extraction is strongly affected by context selection in this within-document setting, while evidence extraction additionally depends on extraction fidelity for hybrid numeric-textual statements even when the correct para- graph is available. Future work may extend this analysis beyond SBS articles and develop extraction approaches that better preserve variable binding and numerical fields in statistical evidence. Limitations and Ethical Considerations Our experiments are limited to claim traces extracted from empiri- cal research in the SBS disciplines. Writing conventions vary across disciplines, so observed trade-offs may not transfer directly to other domains. Our dataset contains a single annotated claim trace per paper and is not exhaustive, as papers often contain multiple hy- potheses and supporting results. Our preprocessing uses prose-only paragraphs and excludes tables, figures (and captions), and equations; evidence extraction results therefore reflect performance on prose-reported statistical evidence only. Accordingly, oracle Evidence F1 should be inter- preted as an upper bound under perfect paragraph selection within prose, rather than a ceiling on recovering statistical details that may be reported outside running text. Furthermore, the task is sequential, so errors can propagate from Stage 1 (hypothesis ex- traction) to Stage 2 (evidence extraction); we partially isolate the evidence stage by evaluating evidence extraction conditioned on Conference’17, July 2017, Washington, DC, USAKoneru et al. LLMMethodHypothesisEvidence PRF1PRF1 llama 3.2 Zero-Shot (Full text)0.340.400.360.180.200.19 CoT (Full text)0.210.340.260.190.220.20 RAG (k=5)0.450.360.400.170.220.19 RAG (k=10)0.500.400.440.180.220.20 RAG (k=20)0.500.410.450.200.210.21 RAG + Reranker (k=5)0.450.410.430.190.240.21 Fine-tuned SciBERT + Reranker (k=5)0.51 0.43 0.470.25 0.28 0.26 Oracle context0.670.550.610.430.520.47 gpt-oss Zero-Shot (Full text)0.480.400.440.230.270.24 CoT (Full text)0.410.500.450.250.280.26 RAG (k=5)0.520.430.470.150.200.17 RAG (k=10)0.530.460.490.190.250.21 RAG (k=20)0.560.500.530.220.310.25 RAG + Reranker (k=5)0.540.500.520.240.280.25 Fine-tuned SciBERT + Reranker (k=5)0.58 0.51 0.540.25 0.31 0.27 Oracle context0.910.730.810.500.570.53 gpt-4o-mini Zero-Shot (Full text)0.330.400.360.170.220.18 CoT (Full text)0.430.420.420.210.260.23 RAG (k=5)0.500.460.480.230.290.26 RAG (k=10)0.570.470.520.260.330.28 RAG (k=20)0.580.470.520.280.350.29 RAG + Reranker (k=5)0.560.460.500.300.390.33 Fine-tuned SciBERT + Reranker (k=5)0.59 0.49 0.540.33 0.41 0.35 Oracle context0.880.700.780.490.630.55 gemini-2.5-flash Zero-Shot (Full text)0.350.470.400.310.480.35 CoT (Full text)0.350.480.400.32 0.49 0.36 RAG (k=5)0.400.370.380.150.200.16 RAG (k=10)0.420.410.420.190.250.21 RAG (k=20)0.430.450.440.220.300.25 RAG + Reranker (k=5)0.440.430.440.240.330.27 Fine-tuned SciBERT + Reranker (k=5)0.46 0.47 0.460.270.390.30 Oracle context0.740.690.710.510.590.53 Table 1: Results for sequential two-stage extraction of hypothesis and corresponding evidence from full text of scientific articles. We compare hypothesis and statistical evidence extraction performance (Precision, Recall, F1) across four LLMs, comparing full-text baselines against retrieval-augmented strategies with varying context quantity (푘) and retrieval quality. Oracle settings indicate performance upper bounds given perfect paragraph selection. Best non-oracle F1 per model and subtask is bolded, and oracle results are reported as upper bounds. ground-truth hypotheses (Appendix Table 2). Finally, the evalu- ation depends on automated proxies: hypothesis scoring uses a calibrated embedding-similarity threshold, and evidence scoring uses an LLM-based component parser with deterministic scoring. We interpret these metrics as scalable approximations of semantic match (hypotheses) and field-level performance (evidence). This work does not involve human subjects or personal data. The main risks are practical: full-text scientific articles may be subject to copyright and licensing restrictions, so we release only code, prompts, and configurations rather than redistributing full texts. In addition, hypothesis and evidence extractions can be plausible but incorrect—especially for numeric-text evidence—so outputs should be treated as assistive and audited in high-stakes use. Finally, our evidence scoring relies on an LLM-based parser and our systems de- pend on pretrained models, both of which may introduce systematic errors or bias and may favor dominant writing styles. References [1] Titipat Achakulvisut, Chandra Bhagavatula, Daniel Acuna, and Konrad Kording. 2019. Claim extraction in biomedical publications using deep discourse model and transfer learning. arXiv preprint arXiv:1907.00962 (2019). [2] Nazanin Alipourfard, Beatrix Arendt, Daniel M Benjamin, Noam Benkler, Michael Bishop, Mark Burstein, Martin Bush, James Caverlee, Yiling Chen, Chae Clark, et al.2021. Systematizing confidence in open research and evidence (SCORE). SocArXiv (2021). [3] Chen Amiraz, Florin Cuconasu, Simone Filice, and Zohar Karnin. 2025. The Distracting Effect: Understanding Irrelevant Passages in RAG. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Moham- mad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 18228–18258. doi:10.18653/v1/2025.acl-long.892 Context Selection for Hypothesis and Statistical Evidence Extraction from Full-Text Scientific ArticlesConference’17, July 2017, Washington, DC, USA [4]Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A Pretrained Language Model for Scientific Text. In Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for Computational Linguistics, Hong Kong, China, 3615–3620. doi:10.18653/v1/D19-1371 [5] Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long- document transformer. arXiv preprint arXiv:2004.05150 (2020). [6] Lutz Bornmann, Robin Haunschild, and Rüdiger Mutz. 2021. Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases. Humanities and Social Sciences Communications 8, 1 (2021), 1–15. [7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901. [8]Haotian Chen, Qingqing Long, Meng Xiao, Xiao Luo, Wei Ju, Chengrui Wang, Xuezhi Wang, Yuanchun Zhou, and Hengshu Zhu. 2025. SciRerankBench: Bench- marking Rerankers Towards Scientific Retrieval-Augmented Generated LLMs. arXiv preprint arXiv:2508.08742 (2025). [9]Arman Cohan, Waleed Ammar, Madeleine van Zuylen, and Field Cady. 2019. Structural Scaffolds for Citation Intent Classification in Scientific Publications. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, Minneapolis, Minnesota, 3586–3596. doi:10.18653/ v1/N19-1361 [10] Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. The power of noise: Redefining retrieval for RAG systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 719–729. [11]Hao Cui, Zahra Shamsi, Gowoon Cheon, Xuejian Ma, Shutong Li, Maria Tikhanovskaya, Peter Norgaard, Nayantara Mudur, Martyna Plomecka, Paul Raccuglia, et al.2025. CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning. arXiv preprint arXiv:2503.13517 (2025). [12] Franck Dernoncourt and Ji Young Lee. 2017. PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstracts. In Proceedings of the Eighth International Joint Conference on Natural Language Processing, IJCNLP 2017, Taipei, Taiwan, November 27 - December 1, 2017, Volume 2: Short Papers, Greg Kondrak and Taro Watanabe (Eds.). Asian Federation of Natural Language Processing, 308–313. https://aclanthology.org/I17-2052/ [13]Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C Wallace. 2020. ERASER: A benchmark to evalu- ate rationalized NLP models. In Proceedings of the 58th annual meeting of the association for computational linguistics. 4443–4458. [14]Aris Fergadis, Dimitris Pappas, Antonia Karamolegkou, and Harris Papageorgiou. 2021. Argumentation mining in scientific literature for sustainable development. In Proceedings of the 8th Workshop on Argument Mining. 100–111. [15]Hannah Fraser, Martin Bush, Bonnie C. Wintle, Fallon Mody, Eden T. Smith, Anca M. Hanea, Elliot Gould, Victoria Hemming, Daniel G. Hamilton, Libby Rumpff, David P. Wilkinson, Ross Pearson, Felix Singleton Thorn, Raquel Ashton, Aaron Willcox, Charles T. Gray, Andrew Head, Melissa Ross, Rebecca Groenewe- gen, Alexandru Marcoci, Ans Vercammen, Timothy H. Parker, Rink Hoekstra, Shinichi Nakagawa, David R. Mandel, Don van Ravenzwaaij, Marissa McBride, Richard O. Sinnott, Peter Vesk, Mark Burgman, and Fiona Fidler. 2023. Predicting reliability through structured expert elicitation with the repliCATS (Collaborative Assessments for Trustworthy Science) process. PLOS ONE 18, 1 (01 2023), 1–16. doi:10.1371/journal.pone.0274429 [16]Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 2, 1 (2023). [17]Renee A Hesselbach, David H Petering, Craig A Berg, Henry Tomasiewicz, and Daniel Weber. 2012. A guide to writing a scientific paper: a focus on high school through graduate level student research. Zebrafish 9, 4 (2012), 246–249. [18]Sarthak Jain, Madeleine van Zuylen, Hannaneh Hajishirzi, and Iz Beltagy. 2020. SciREX: A Challenge Dataset for Document-Level Information Extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (Eds.). Association for Computational Linguistics, 7506–7516. doi:10.18653/V1/2020.ACL-MAIN.670 [19] Tom Jansen and Tobias Kuhn. 2016. Extracting Core Claims from Scientific Arti- cles. In BNAIC 2016: Artificial Intelligence - 28th Benelux Conference on Artificial In- telligence, Amsterdam, The Netherlands, November 10-11, 2016, Revised Selected Pa- pers (Communications in Computer and Information Science, Vol. 765), Tibor Bosse and Bert Bredeweg (Eds.). Springer, 32–46. doi:10.1007/978-3-319-67468-1_3 [20]Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan Ö. Arik. 2025. Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG. In The Thir- teenth International Conference on Learning Representations, ICLR 2025, Singa- pore, April 24-28, 2025. OpenReview.net.https://openreview.net/forum?id= oU3tpaR8fm [21]Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models are Zero-Shot Reasoners. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Informa- tion Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (Eds.). http://papers.nips.c/paper_files/paper/2022/hash/ 8b0d291acd4acf06ef112099c16f326-Abstract-Conference.html [22] Thomas A Lang and Douglas G Altman. 2015. Basic statistical reporting for articles published in biomedical journals: the “Statistical Analyses and Methods in the Published Literature” or the SAMPL Guidelines. International journal of nursing studies 52, 1 (2015), 5–9. [23]Anne Lauscher, Goran Glavas, and Kai Eckert. 2018. ArguminSci: A Tool for Analyzing Argumentation and Rhetorical Aspects in Scientific Writing. In Pro- ceedings of the 5th Workshop on Argument Mining, ArgMining@EMNLP 2018, Brussels, Belgium, November 1, 2018, Noam Slonim and Ranit Aharonov (Eds.). Association for Computational Linguistics, 22–28. doi:10.18653/V1/W18-5203 [24] Xiangci Li, Gully Burns, and Nanyun Peng. 2021. Scientific Discourse Tagging for Evidence Extraction. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (Eds.). Association for Computational Linguistics, Online, 2550–2562. doi:10.18653/v1/2021.eacl-main.218 [25]Xiangci Li and Jessica Ouyang. 2025. How Does Knowledge Selection Help Re- trieval Augmented Generation?. In Findings of the Association for Computational Linguistics: EMNLP 2025, Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.). Association for Computational Linguistics, Suzhou, China, 4104–4121. doi:10.18653/v1/2025.findings-emnlp.218 [26] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12 (2024), 157–173. doi:10.1162/tacl_a_00638 [27]Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction. In Proceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (Eds.). Association for Computational Linguistics, 3219–3232. doi:10.18653/V1/D18-1360 [28]Ian H. Magnusson and Scott E. Friedman. 2021. Extracting Fine-Grained Knowl- edge Graphs of Scientific Claims: Dataset and Transformer-Based Results. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 4651–4658. doi:10.18653/V1/2021.EMNLP-MAIN.381 [29] Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. 2025. DiscoveryBench: Towards Data-Driven Discov- ery with Large Language Models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. https://openreview.net/forum?id=vyflgpwf JW [30]Iain Marshall, Joël Kuiper, Edward Banner, and Byron C. Wallace. 2017. Automat- ing Biomedical Evidence Synthesis: RobotReviewer. In Proceedings of ACL 2017, System Demonstrations, Mohit Bansal and Heng Ji (Eds.). Association for Computa- tional Linguistics, Vancouver, Canada, 7–12. https://aclanthology.org/P17-4002/ [31]Iain J. Marshall and Byron C. Wallace. 2019. Toward systematic review automa- tion: a practical guide to using machine learning tools in research synthesis. Systematic Reviews 8, 1 (2019), 163. doi:10.1186/s13643-019-1074-9 [32]Ronak Pradeep, Xueguang Ma, Rodrigo Nogueira, and Jimmy Lin. 2021. Scientific Claim Verification with VerT5erini. In Proceedings of the 12th International Work- shop on Health Text Mining and Information Analysis, LOUHI@EACL, Online, April 19, 2021, Eben Holderness, Antonio Jimeno-Yepes, Alberto Lavelli, Anne-Lyse Mi- nard, James Pustejovsky, and Fabio Rinaldi (Eds.). Association for Computational Linguistics, 94–103. https://w.aclweb.org/anthology/2021.louhi-1.11/ [33]Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning. PMLR, 31210–31227. [34]W. Scott Spangler, Angela D. Wilkins, Benjamin J. Bachman, Meena Nagarajan, Tajhal Dayaram, Peter J. Haas, Sam Regenbogen, Curtis R. Pickering, Austin Comer, Jeffrey N. Myers, Ioana Stanoi, Linda Kato, Ana Lelescu, Jacques J. Labrie, Neha Parikh, Andreas Martin Lisewski, Lawrence A. Donehower, Ying Chen, and Olivier Lichtarge. 2014. Automated hypothesis generation based on mining scientific literature. In The 20th ACM SIGKDD International Conference on Knowl- edge Discovery and Data Mining, KDD ’14, New York, NY, USA - August 24 - 27, Conference’17, July 2017, Washington, DC, USAKoneru et al. 2014, Sofus A. Macskassy, Claudia Perlich, Jure Leskovec, Wei Wang, and Rayid Ghani (Eds.). ACM, 1877–1886. doi:10.1145/2623330.2623667 [35] Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. 2025. PaperBench: Evaluating AI’s Ability to Replicate AI Research. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025 (Proceedings of Machine Learning Research, Vol. 267), Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (Eds.). PMLR / OpenReview.net. https://proceedings.mlr.press/v267/ starace25a.html [36] Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023. Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, 14918–14937. doi:10.18653/V1/2023.EMNLP-MAIN.923 [37]Simone Teufel and Marc Moens. 2002. Summarizing Scientific Articles: Exper- iments with Relevance and Rhetorical Status. Comput. Linguistics 28, 4 (2002), 409–445. doi:10.1162/089120102762671936 [38] James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a Large-scale Dataset for Fact Extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), Mari- lyn A. Walker, Heng Ji, and Amanda Stent (Eds.). Association for Computational Linguistics, 809–819. doi:10.18653/V1/N18-1074 [39]Rosni Vasu, Cristina Sarasua, and Abraham Bernstein. 2024. SciHyp: A Fine- Grained Dataset Describing Hypotheses and Their Components from Scien- tific Articles. In The Semantic Web - ISWC 2024 - 23rd International Semantic Web Conference, Baltimore, MD, USA, November 11-15, 2024, Proceedings, Part I (Lecture Notes in Computer Science, Vol. 15233), Gianluca Demartini, Katja Hose, Maribel Acosta, Matteo Palmonari, Gong Cheng, Hala Skaf-Molli, Nico- las Ferranti, Daniel Hernández, and Aidan Hogan (Eds.). Springer, 134–152. doi:10.1007/978-3-031-77847-6_8 [40] David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or Fiction: Verifying Scientific Claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, 7534–7550. doi:10.18653/V1/2020.EMNLP-MAIN.609 [41]David Wadden, Kyle Lo, Lucy Lu Wang, Arman Cohan, Iz Beltagy, and Hannaneh Hajishirzi. 2022. MultiVerS: Improving scientific claim verification with weak supervision and full-document context. In Findings of the Association for Com- putational Linguistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022 (Findings of ACL, Vol. NAACL 2022), Marine Carpuat, Marie-Catherine de Marneffe, and Iván Vladimir Meza Ruíz (Eds.). Association for Computational Linguistics, 61–76. doi:10.18653/V1/2022.FINDINGS-NAACL.6 [42]Xin Wei, Md Reshad Ul Hoque, Jian Wu, and Jiang Li. 2023. ClaimDistiller: Scientific Claim Extraction with Supervised Contrastive Learning. In Proceedings of Joint Workshop of the 4th Extraction and Evaluation of Knowledge Entities from Scientific Documents (EEKE2023) and the 3rd AI + Informetrics (AII2023) co-located with the JCDL 2023, Santa Fe, New Mexico, USA and Online, 26 June, 2023 (CEUR Workshop Proceedings, Vol. 3451), Chengzhi Zhang, Yi Zhang, Philipp Mayr, Wei Lu, Arho Suominen, Haihua Chen, and Ying Ding (Eds.). CEUR-WS.org, 65–77. https://ceur-ws.org/Vol-3451/paper11.pdf [43]Elizabeth K. White, K. Bretonnel Cohen, and Lawrence Hunter. 2011. Hypothesis and Evidence Extraction from Full-Text Scientific Journal Articles. In Proceedings of the 2011 Workshop on Biomedical Natural Language Processing, BioNLP@ACL, Portland, Oregon, USA, June 23-24, 2011, Kevin Bretonnel Cohen, Dina Demner- Fushman, Sophia Ananiadou, John Pestian, Jun’ichi Tsujii, and Bonnie L. Webber (Eds.). Association for Computational Linguistics, 134–135. https://aclanthology. org/W11-0217/ [44]Jian Wu and Sarah Rajtmajer. 2025. What Are Research Hypotheses? arXiv preprint arXiv:2509.00185 (2025). [45]Siye Wu, Jian Xie, Jiangjie Chen, Tinghui Zhu, Kai Zhang, and Yanghua Xiao. 2024. How easily do irrelevant inputs skew the responses of large language models? arXiv preprint arXiv:2404.03302 (2024). [46] Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. 2024. Effective Long-Context Scaling of Foundation Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (Volume 1: Long Papers), Kevin Duh, Helena Gomez, and Steven Bethard (Eds.). Association for Computational Linguistics, Mexico City, Mexico, 4643–4663. doi:10.18653/v1/2024.naacl-long.260 [47]Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2024. Making Retrieval- Augmented Language Models Robust to Irrelevant Context. In The Twelfth Inter- national Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum?id=ZS4m74kZpH [48]Haozhen Zhang, Tao Feng, Pengrui Han, and Jiaxuan You. 2025. AcademicEval: Live Long-Context LLM Benchmark. Trans. Mach. Learn. Res. 2025 (2025). https: //openreview.net/forum?id=LjQ4voE5bs A Diagnostic: Context signal density (hypothesis extraction) For each hypothesis-stage context, we compute the relevant sentence proportion (RSP), defined as the fraction of sentences whose cosine similarity to the gold hypothesis exceeds휏=0.89 using Gemini em- beddings and the same threshold used for hypothesis evaluation in Section 5.1. We bin papers by RSP (0–1%, 1–2%, 2–3%, 3–4%, 4–5%, and>5%) and report mean hypothesis extraction F1 within each bin in Figure 4. Point size is proportional to the number of papers in the bin. B Diagnostic: Per-paper transition analysis: quantity versus quality (evidence extraction) To isolate how retrieved context affects evidence extraction, inde- pendent of errors from the hypothesis stage, we condition on the ground-truth hypothesis and vary only the evidence-stage context construction. We compare transitions that change context quantity (standard RAG with푘 ∈ 5,10,20and full text) and transitions that change context quality while holding the final context size fixed (dense 푘=5→ reranked 푘=5→ fine-tuned-retriever+푘=5). For each transition (source→target), we categorize each pa- per by whether the gold evidence paragraph푔appears in the re- trieved evidence-stage contexts퐶 src and퐶 tgt . We label a paper as GT_Kept if푔appears in both contexts (푔 ∈ 퐶 src and푔 ∈ 퐶 tgt ), and as GT_Gained if푔is absent in the source but appears in the target (푔∉ 퐶 src and푔 ∈ 퐶 tgt ). For transitions where the target context is not guaranteed to contain the source context (e.g., reranking or switching retrievers), we additionally track GT_Lost cases where 푔is present in the source but not retained in the target (푔 ∈ 퐶 src and푔∉ 퐶 tgt ), distinguishing these from GT_Missing cases where 푔is absent from both contexts (푔∉ 퐶 src and푔∉ 퐶 tgt ). Within each bucket, we compute the mean per-paper change in Evidence F1,ΔF1= F1 tgt − F1 src . We summarize the per-paper transition outcomes in Figure 8, and report the corresponding aggregate Pre- cision/Recall/F1 for the same ground-truth-hypothesis setting in Table 2. We find that changes in evidence extraction performance are largely explained by whether a retrieval change adds the gold ev- idence paragraph to the context. In quantity ablations, improve- ments from larger푘occur primarily in GT_Gained cases (where expanding푘newly retrieves푔). When gold paragraph is already present (GT_Kept), expanding context (e.g.,푘=20→full text) yields limited gains and can even degrade performance. In contrast, retrieval-quality improvements (reranking and fine- tuning) yield more consistent gains because they increase the rate of GT_Gained outcomes under a fixed context size. However, quality changes are not monotonic: fine-tuning can introduce a small trade- off in GT_Lost cases, where푔was retrieved under the prior method but is not retained after the retriever/reranker change. Context Selection for Hypothesis and Statistical Evidence Extraction from Full-Text Scientific ArticlesConference’17, July 2017, Washington, DC, USA 0 1%1 2%2 3%3 4%4 5%>5% Relevant-sentence proportion 0.0 0.2 0.4 0.6 0.8 1.0 Hypothesis F1 (a) LLaMA-3.2 0 1%1 2%2 3%3 4%4 5%>5% Relevant-sentence proportion 0.0 0.2 0.4 0.6 0.8 1.0 Hypothesis F1 (b) Gemini-2.5-Flash 0 1%1 2%2 3%3 4%4 5%>5% Relevant-sentence proportion 0.0 0.2 0.4 0.6 0.8 1.0 Hypothesis F1 (c) GPT-4o-mini 0 1%1 2%2 3%3 4%4 5%>5% Relevant-sentence proportion 0.0 0.2 0.4 0.6 0.8 1.0 Hypothesis F1 Full text RAG k=5 RAG k=10 RAG k=20 RAG+Reranker k=5 Finetuned+Reranker k=5 Oracle (d) GPT-OSS Figure 4: Hypothesis extraction F1 versus relevant-sentence proportion (RSP), binned. Points show mean F1 within each bin and point size is proportional to the number of papers in the bin. C Semantic similarity calibration Figure 5: Calibration of the cosine similarity threshold us- ing a precision recall curve. The plot shows the values at different threshold values for the Gemini embedding model. The selected threshold of 0.89 corresponds to the point of maximum F1 score (0.72) on manually labeled set. Conference’17, July 2017, Washington, DC, USAKoneru et al. Example: Hypothesis extraction error with high embedding similarityCosine sim: 0.875 (87.5%) DOI 10.1016/j.jesp.2017.07.008 Human annotation. The presence of an alternative option will produce a competing, covert influence on participants. Extracted. We hypothesize that participants will be more likely to respond opposite to the literal content of political conspiracy statements because competing covert influences—such as threat-protection needs and accuracy biases—drive a tension between appealing and unappealing aspects of conspiracies, resulting in opposite responses relative to baseline statements. Diagnosis. High lexical/semantic overlap (“covert influence”) but the prediction shifts the core construct and mechanism from alternative option to conspiracy-statement response and introduces additional constructs not stated in the annotated hypothesis. Figure 6: Qualitative example of incorrect hypothesis extraction with high embedding similarity. Example: Hard-negative construction for within-document retriever training DOI. 10.1016/j.jesp.2016.01.014 Human annotation. We expected that participants experiencing causal uncertainty would show higher rates of task resumption when the interrupted task was associated with more abstract (vs. concrete) thinking. Positive paragraph (contains human annotated hypothesis). Returning to our main prediction, we expected that participants experiencing causal uncertainty would show higher rates of task resumption when the interrupted task was associated with more abstract (vs. concrete) thinking. To measure task resumption, we gave participants an opportunity to engage in a more enjoyable task that involved reading humorous stories. The pilot study confirmed that the Humor Evaluation Task was indeed perceived as a more enjoyable alternative to participants. ... Hard negative paragraph (topically similar, different rhetorical role). The task interruption and resumption paradigm used in this experiment rules out this alternative explanation because the alternative task that participants could choose to do involved reading humorous stories. The pilot study confirmed this, showing that participants thought the humor evaluation task would be the most fun and enjoyable. If participants’ goal was to simply be in a better mood state, rather than to think more abstractly, we would not have found our demonstrated effect ... Figure 7: Example of the hard negative selection for retriever training. Hard negatives are chosen to be semantically similar to the gold paragraph while differing in rhetorical function with respect to the extraction target. Context Selection for Hypothesis and Statistical Evidence Extraction from Full-Text Scientific ArticlesConference’17, July 2017, Washington, DC, USA (a) GPT-4o-mini (Quantity)(b) GPT-4o-mini (Quality) (c) Gemini-2.5-Flash (Quantity)(d) Gemini-2.5-Flash (Quality) (e) LLaMA-3.2 (Quantity)(f) LLaMA-3.2 (Quality) (g) GPT-OSS (Quantity)(h) GPT-OSS (Quality) Figure 8: Per-paper transition analysis for evidence extraction under quantity vs. quality changes. Left column varies retrieved context quantity (푘=5→10→20→full text). Right column varies retrieval quality at fixed context size (푘=5) (RAG→ RAG+Reranker→fine-tuned retriever+reranker→oracle). Bars report mean per-paperΔF1 within each retrieval-outcome bucket; labels above bars show the fraction of papers in that bucket. For quality transitions, we additionally report GT_Lost when the gold evidence paragraph is present in the source context but absent in the target. Conference’17, July 2017, Washington, DC, USAKoneru et al. LLMMethodEvidence PRF1 llama 3.2 RAG (k=5)0.190.250.22 RAG (k=10)0.240.320.27 RAG (k=20)0.250.290.26 Full text0.210.280.24 RAG + Reranker (k=5)0.290.430.35 Fine-tuned SciBERT + Reranker (k=5)0.310.480.38 Oracle context0.430.520.47 gpt-oss RAG (k=5)0.190.200.20 RAG (k=10)0.230.280.25 RAG (k=20)0.250.340.29 Full text0.280.360.31 RAG + Reranker (k=5)0.340.440.38 Fine-tuned SciBERT + Reranker (k=5)0.430.490.46 Oracle context0.500.570.53 gpt-4o-mini RAG (k=5)0.220.400.28 RAG (k=10)0.240.450.31 RAG (k=20)0.280.490.36 Full text0.350.400.37 RAG + Reranker (k=5)0.390.450.41 Fine-tuned SciBERT + Reranker (k=5)0.430.510.47 Oracle context0.490.630.53 gemini-2.5-flash RAG (k=5)0.210.390.27 RAG (k=10)0.240.440.31 RAG (k=20)0.290.500.37 Full text0.330.480.39 RAG + Reranker (k=5)0.270.480.34 Fine-tuned SciBERT + Reranker (k=5)0.310.530.38 Oracle context0.510.590.53 Table 2: Evidence extraction with ground-truth hypotheses. We isolate the evidence stage by providing the extractor the human-annotated (ground-truth) hypothesis and varying only the evidence-stage context configuration (RAG푘 ∈ 5,10,20, reranking, and fine-tuned retrieval). Reported Precision/Recall/F1 are computed with the same component-based evaluator as in the main results.