Paper deep dive
FDARxBench: Benchmarking Regulatory and Clinical Reasoning on FDA Generic Drug Assessment
Betty Xiong, Jillian Fisher, Benjamin Newman, Meng Hu, Shivangi Gupta, Yejin Choi, Lanyan Fang, Russ B Altman
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 98%
Last extracted: 3/23/2026, 12:06:13 PM
Summary
FDARxBench is an expert-curated benchmark designed to evaluate LLM performance on document-grounded question answering tasks related to FDA generic drug assessment. It features 17K+ questions derived from FDA drug labels, covering factual, multi-hop, and refusal tasks, with a focus on regulatory-grade correctness, provenance, and safe abstention.
Entities (4)
Relation Signals (3)
FDARxBench → evaluates → LLM
confidence 100% · The benchmark is designed to support evaluation of LLM behavior on drug-label questions.
FDARxBench → groundedin → Drug Labels
confidence 100% · benchmark for evaluating document-grounded question-answering (QA) motivated by generic drug assessment, using the U.S. Food and Drug Administration (FDA) drug label documents.
FDA → publishes → Drug Labels
confidence 100% · FDA drug label documents are the legally authoritative document on prescription drugs in the United States
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We introduce an expert curated, real-world benchmark for evaluating document-grounded question-answering (QA) motivated by generic drug assessment, using the U.S. Food and Drug Administration (FDA) drug label documents. Drug labels contain rich but heterogeneous clinical and regulatory information, making accurate question answering difficult for current language models. In collaboration with FDA regulatory assessors, we introduce FDARxBench, and construct a multi-stage pipeline for generating high-quality, expert curated, QA examples spanning factual, multi-hop, and refusal tasks, and design evaluation protocols to assess both open-book and closed-book reasoning. Experiments across proprietary and open-weight models reveal substantial gaps in factual grounding, long-context retrieval, and safe refusal behavior. While motivated by FDA generic drug assessment needs, this benchmark also provides a substantial foundation for challenging regulatory-grade evaluation of label comprehension. The benchmark is designed to support evaluation of LLM behavior on drug-label questions.
Tags
Links
- Source: https://arxiv.org/abs/2603.19539v1
- Canonical: https://arxiv.org/abs/2603.19539v1
Trouble viewing inline? Open PDF directly →
Full Text
78,551 characters extracted from source content.
Expand or collapse full text
FDARxBench: Benchmarking Regulatory and Clinical Reasoning on FDA Generic Drug Assessment Betty Xiong 1 , Jillian Fisher 2 , Benjamin Newman 2 , Meng Hu 3 , Shivangi Gupta 3 , Yejin Choi 1 , Lanyan Fang 3 , Russ Altman 1 1 Stanford University, 2 University of Washington, 3 U.S. Food and Drug Administration Correspondence: xiongb@stanford.edu Abstract We introduce an expert curated, real-world benchmark for evaluating document-grounded question-answering (QA) motivated by generic drug assessment, using the U.S. Food and Drug Administration (FDA) drug label documents. Drug labels contain rich but heterogeneous clin- ical and regulatory information, making accu- rate question answering difficult for current lan- guage models. In collaboration with FDA regu- latory assessors, we introduce FDARXBENCH 1 , and construct a multi-stage pipeline for gen- erating high-quality, expert curated, QA exam- ples spanning factual, multi-hop, and refusal tasks, and design evaluation protocols to as- sess both open-book and closed-book reason- ing. Experiments across proprietary and open- weight models reveal substantial gaps in fac- tual grounding, long-context retrieval, and safe refusal behavior. While motivated by FDA generic drug assessment needs, this benchmark also provides a substantial foundation for chal- lenging regulatory-grade evaluation of label comprehension. The benchmark is designed to support evaluation of LLM behavior on drug- label questions. 1 Introduction Large language models (LLMs) have demonstrated strong performance on biomedical question an- swering tasks (Luo et al., 2022; Chen et al., 2023; Bedi et al., 2025; Sellergren et al., 2025), but their abilities in high-stakes regulatory settings remain under-explored. Their performance and practical utility are still not clearly defined, in part because progress is bottlenecked by the lack of standardized, regulator-aligned benchmarks and limited access to experts who can define regulator-grade correctness, provenance, and safe abstention. The motivating use case for this work is the US Food and Drug Administration (FDA) generic drug 1 FDARXBENCHmaybeaccessedvia: https://github.com/xiongbetty/FDARxBench Non-expert Q: Does Zorvolex cause cancer in mice in the long-term? Why? ✗ Scope mismatch: pre-clinical detail not central to label-use. ✗ Low utility: unlikely to affect a concrete usage decision. Expert-guided Q: Compared to fasting, what happens to absorption if Zorvolex is taken with food? Why? ✓ Decision-relevant: directly affects users of this drug. ✓ Easy to verify: typically stated in “food effect” sections. ✗ FDA ✓ Figure 1: Example of expert-guided criteria. assessment: a real-world high-stakes document QA setting where FDA regulators evaluate generic drug applications by comparing them to approved refer- ence drugs. This requires analyzing detailed, multi- page drug labels to answer specific high-stakes questions. The process is time intensive and de- mands careful review, making it both a natural and challenging candidate for AI assistance. Prior work has demonstrated promising appli- cations of GPT-style models to support drug la- bel analysis (Shi et al., 2023), but performance assessment in that study relied primarily on manual review, motivating the need for systematic perfor- mance benchmarks. To overcome this, we partner with professional FDA regulatory assessors with experience in generic drug assessment to shape the benchmark and expert-adjudicate question quality and model outputs. Distinct from other biomed- ical settings, regulatory assessors (1) care about complex relationships between specific entities and quantities, (2) need answers with fine-grained provenance, and (3) require reliable abstentions to unanswerable questions. However, building large- scale benchmarks to assess these aspects is difficult as FDA regulatory expertise is not widespread. In this work, we propose a benchmark to system- atically evaluate LLM performance for regulator- grade question answering grounded in drug labels (Figure 1), informed by expert input from FDA reg- ulatory assessors. Our benchmark consists of 17K+ arXiv:2603.19539v1 [cs.CL] 20 Mar 2026 questions that (1) focus on entities and quantities that are clinically meaningful to FDA assessors in generic drug assessment workflows, (2) require cross-section reasoning with fine-grained prove- nance, and (3) require clear abstention to unanswer- able questions. This benchmark enables evaluation of reasoning, retrieval, long-context comprehen- sion, grounding, and safety behavior within a uni- fied framework. The proposed pipeline is modular and can be adapted or extended to other special- ized domains where expert-defined correctness and provenance are required. Our contributions are threefold: (1) a 17K+, expert-formed regulatory QA benchmark grounded in FDA drug labels; (2) task definitions and evalu- ation protocols that emphasize correctness, prove- nance, and safe abstention over surface-level flu- ency; and (3) detailed human expert and model- based analyses that reveal persistent failures in grounding and safe abstention. 2 Background and Related Work 2.1 FDA Drug Labels FDA drug label documents are the legally au- thoritative document on prescription drugs in the United States, formatted as Structured Product La- beling (SPL) files (Schadow, 2007). Released as semi-structured XML under the Physician Label- ing Rule (PLR), their sections can vary widely across manufacturers and time periods, yielding multi-page, heterogeneous documents that chal- lenge automated reasoning. While prior work has incorporated LLMs into label-centric workflows (Wu et al., 2025; Shi et al., 2023) and evaluation settings (Silberg et al., 2024), no benchmark specif- ically measures LLM performance on drug label understanding. 2.2 Biomedical QA Datasets Current biomedical benchmarks focus on domains outside of regulation.For example BIOASQ and PUBMEDQA target questions derived from biomedical literature (Kritharaa et al., 2023; Jin et al., 2019). HEALTHSEARCHQA targets general healthcare consumers (Singhal et al., 2022). Still other tasks are more clincial-facing, such as QA over electronic medical records, laboratory tests, and physician-patient conversations (Pampari et al., 2018; Bhasuran et al., 2025; Arora et al., 2025). However, unlike existing biomedical benchmarks, evaluations of FDA drug labels sit at the intersec- tion of medicine and regulation, and current bench- marks do not capture key regulatory requirements such as reasoning over full-length documents, pro- viding traceable citations, and explicitly refusing unanswerable questions. 3 Dataset: Expert-Guided FDA Label QA Benchmark The benchmark is designed to support evaluation of LLM behavior on drug-label questions motivated by generic drug assessment. In this section, we outline how we create FDARXBENCH (Figure 2). (More details in Appendix B). 3.1 Source Documents and Preprocessing We curate 700 FDA prescription drug labels, sourced from the FDALabel Database (FDA), as the starting corpus of our benchmark. In gen- eral, drug labels are semi-structured XML files that contain text details on new drugs. We prepro- cess each drug label by parsing it into section-level passages using subheaders as delimiters, where ev- ery passage is assigned a unique identifier. This new chunked representation allows us to (1) track provenance and evaluate citations, and (2) enable retrieval-augmented workflows. 3.2 Question Generation Pipeline Once we preprocessed the data, we construct each question of our benchmark using this general mod- ular four-step pipeline: 1.Context selection. We randomly select a portion of the drug label to be used as the main context for the question generation. 2.LLM question generation. Given the context chosen in step (1), we use an LLM with few-shot prompting to generate a QA pair. The context acts as supporting evidence that must be used to answer the generated question. This generation changes based on question types, shown in § 3.3. 3.Expert feedback loop. To validate the LLM- generated QA pairs, we first collect feedback from FDA regulatory domain experts on a seed set of 50 LLM-generated questions. Using this feedback, we distill a set of clear relevance cri- teria and encoded them into a structured prompt that enables an LLM to automatically evaluate the relevance of the generated questions. We find that this LLM-as-judge approach closely aligns with expert judgments, achieving strong agreement with domain experts as the gold labels (precision = 0.968, F1 = 0.800). 4.Filtering. Finally, we apply rule-based and LLM-as-judge filtering. We use automatic rule- based filters to remove QAs that are structurally invalid, i.e., items with a missing question or answer, empty strings, absence of the drug name are discarded. For the LLM-as-judge, we filter based on QA correctness (answer is correct given only the provided context) and question quality (remove QA pairs whose question is semanti- cally invalid). (Further details on the filtering steps in Appendix B.4). We validate the LLM filter against human annotators, and report an average precision of 0.797 and average F1 of 0.683, but 0.922 and 0.738, respectively, without multi-hop which is a known harder task. (More details in Appendix A.1). 3.3 Question Types Using this general pipeline, we construct three di- verse QA types (factual, multi-hop and refusal), each targeting a distinct regulatory capability rel- evant to generic drug assessment. (For examples, see Appendix B.2). Factual (one-section) questions are fact-based questions that can be answered directly using in- formation from a single section of the drug label. To construct them, we select either the Highlights section, which provides a structured summary of key information, or another section sampled from the full label, and explicitly instruct the LLM to generate fact-based questions grounded only in the content of the selected section, as outlined in § 3.2. Multi-hop (two-section) questions require in- tegrating information from two different sections of the same drug label. To construct them, we ran- domly select pairs of sections and explicitly instruct the LLM to generate questions that depend on ev- idence from both sections, such that the question becomes unanswerable if either section is removed. Refusal (unanswerable) questions are negative controls designed to be unanswerable. We gener- ate them by inserting an out-of-scope biomedical entity/keyword into a clinical-style template and programmatically verifying it does not appear any- where in the full label text (case-insensitive string match), so the correct behavior is to abstain rather than hallucinate or rely on outside knowledge. 3.4 Tasks We showcase the versatility of FDARXBENCH by evaluating models under multiple diverse evidence settings. We describe these settings below: Baselines Retrieval (optional) LLM inference Grading 2) Evaluate FDA Labels SPL Recursive chunking Highlights Full Prescribing Information Remove figures Indexed chunks Pre-processing Generation Template based QA Select k = 1,2 sections LLM self-instruct Replace random QA terms Filtering Factual + multi-hop QA Rules-based filtering LLM-as-judge filtering Benchmark 1) Construct + Refusal QA Open (passages) Open (full label) Closed book BM25 / Contriever / cos similarity Top k=5 chunks FDA expert guidance: task taxonomy + correctness + relevance + grading rubric +Question +Question +Question Cited chunks + 1-3 sentences answer OR “Not answerable” LLM grader: A / B / C A: correct B: incorrect C: not attempted Citation accuracy Main setting Figure 2: Overview of FDARXBENCH creation. •Full-label QA with citations. The entire label is provided as context, to evaluate the real-world generic drug assessment setting of model output plus cited passage ids. •Closed-book QA. Only the question is provided (no label text) as a lower-bound baseline, to eval- uate current drug label leakage in foundational models and propensity to hallucinate or overgen- eralize. • Oracle passages. Gold passage(s) used during construction are provided as an upper-bound di- agnostic. •Retrieval only evaluation. The model is evalu- ated on cited passages only (e.g., a retriever se- lects top-kpassages, which are compared against the gold standard citations), to evaluate the poten- tial performance of RAG methods. 3.5 Metrics For evaluation of FDARXBENCH, we report aligned with regulatory priorities: •Answer correctness. Similar to past work (Wei et al., 2024), accuracy is evaluated by LLM-as- judge given the benchmark’s reference answer. (See Appendix A.1 for human validation details). •Refusal behavior. Automatic evaluation using F1 for predicting abstention on refusal items, and false-refusal behavior on answerable items. • Citation quality. Automatic evaluation using macro-F1 between cited passage ids and gold provenance passage ids recorded during dataset construction. 4 Experiments and Results In this section, we evaluate FDARXBENCH on a variety of state-of-the-art (SOTA) LLM models. Given this analysis, we find that FDARXBENCH is challenging, even for the strongest of current models. The open weight and API-based models ModelFull label (+ cites)Closed-book AccOracle Acc FactMHCite F1 Fact Cite F1 MH Ref F1FactMHFactMH Llama-8B0.500 0.256 0.4580.3330.7960.0020.0030.777 0.668 Llama-70B0.526 0.422 0.5080.4330.7890.2370.3930.797 0.680 Mistral-14B0.503 0.341 0.4390.3280.7100.0870.1500.776 0.676 Qwen-14B0.518 0.416 0.5000.3620.7090.1950.3850.795 0.789 Qwen-32B0.530 0.473 0.5250.4330.756 0.1860.3700.786 0.796 Claude Sonnet 0.526 0.372 0.5280.3830.7480.2590.2720.784 0.653 Claude Opus0.562 0.427 0.5220.4580.731 0.3690.3930.794 0.728 GPT-4o-mini0.507 0.456 0.4970.3780.717 0.2420.4420.806 0.822 GPT-5.10.546 0.461 0.5200.4060.725 0.3430.5150.810 0.805 GPT-5.20.541 0.417 0.5200.3830.7010.2920.4520.817 0.806 Table 1: Model performance across evidence settings. We report answer accuracy (Acc) for factual (Fact) and multi-hop (MH) questions in closed-book, oracle- passages, and full-label settings; the full-label setting additionally reports citation overlap (Cite F1) and re- fusal correctness (Ref F1). Retrieverrecall@1recall@5recall@10recall@|gold| FactMHFactMHFactMHFactMH BM250.5580.3550.7480.7780.7970.8830.5920.546 Cosine similarity0.4230.2940.6760.6680.7500.7920.4630.450 ReContriever0.2200.1760.4550.4380.5890.5930.2380.278 Table 2: Retriever performance on evidence selection for factual (Fact) and multi-hop (MH) questions. We report recall at ranksk ∈1, 5, 10and at the gold set size (|gold|). that we evaluate are in Table 1. 4.1 Model Results Current LLMs consistently struggle with FDARXBENCH, with evidence access being the dominant driver of performance. Table 1 summa- rizes performance across FDARXBENCH tasks. In the closed-book setting, accuracy is generally low (average = 0.22 for factual, 0.34 for multi-hop), es- pecially for smaller open-weight models, indicating that SOTA models do not have access to the drug labels used in FDARXBENCH. However, when models are given oracle passages, performance in- creases sharply across both factual and multi-hop questions (≈0.78-0.82 factual and 0.65-0.82 multi- hop), however still maintain relatively low com- pared to other similar tasks, highlighting the chal- lenge of FDARXBENCH. Lastly, providing the full label does not close this gap: full-label accuracy is consistently lower than oracle-passages accuracy despite having strictly more information available, e.g., performance rises to just 0.53 for factual and 0.40 for multi-hop on average (best full-label multi- hop = 0.47). The substantial oracle to full-label drop indicates that end-to-end performance is lim- ited due to the challenge of long-context tasks. Even when answers improve, provenance and refusal behavior remain inconsistent. Citation overlap is only moderate even for the best models, (e.g., cite F1 peaks at 0.53 for factual and 0.46 for multi-hop), implying that models often cite plausi- ble but non-gold or incomplete passages. Notably, refusal behavior varies substantially across mod- els and does not necessarily track answer accuracy, e.g., refusal F1 ranges from 0.71 for Qwen-14B to 0.80 for Llama-8B, with the smallest 8B model giv- ing the best F1 score. Refusal behavior exhibits a clear precision-recall trade-off, where frontier mod- els have high refusal recall (≈0.99) but over-refuse. This reinforces that refusal should be evaluated as a distinct safety capability, rather than assumed to follow from general QA quality. (Additional plots on model performance in Appendix A). 4.2 Retriever Results Retrieval is a bottleneck to accurate fine-grained citations.To isolate evidence selection from generation, Table 2 compares the retrieved top-k passages (e.g.,k = 1, 5, 10,|gold|) against gold provenance. We evaluate on: BM25 (Robertson and Zaragoza, 2009), cosine similarity using all- MiniLM-L6-v2, (Wang et al., 2020), and ReCon- triever (Lei et al., 2023). We report retrieval qual- ity using F1 calculated between citation overlap against gold provenance. Across cutoffs, BM25 consistently outperforms dense baselines for both factual and multi-hop re- trieval (e.g., recall@1 = 0.56 factual / 0.36 multi- hop vs. cosine similarity at 0.42 / 0.29 and Re- Contriever at 0.22 / 0.18), suggesting that drug- label evidence selection is strongly driven by lexi- cal overlap and section-style phrasing rather than embedding similarity. Overall, these results mirror the end-to-end gap between full-label and oracle- passages in the Model results (§ 4.1). 5 Conclusion We introduce an expert-guided benchmark for regulatory-grade QA motivated by FDA generic drug assessment, grounded in FDA drug labels, with 17K+ document-grounded questions spanning factual, multi-hop, and refusal types and tasks cov- ering closed-book, citation-grounded open-book, retrieval-augmented QA, and safe refusal. Results highlight a gap between general biomedical QA and regulatory needs, where correctness, prove- nance, and conservative abstention are essential. 6 Limitations Our benchmark has several important limitations. First, self-instruct factual questions are gener- ated from individual section chunks, which can artificially narrow context. Chunking improves tractability and provenance tracking, but some clin- ically meaningful questions require broader context across paragraphs or sections. Future work could generate and answer questions with larger windows or the full label, at higher computational and cura- tion cost. Second, constructing QA from isolated chunks does not ensure the cited evidence is unique. La- bels are redundant and often restate key facts across sections, so some questions admit multiple valid evidence locations. This can confound retrieval evaluation and citation scoring when gold prove- nance is defined at the chunk level. More document- aware construction could deduplicate overlapping evidence and represent provenance as a set of ac- ceptable passages or equivalence classes of sup- porting spans. Third, multi-hop construction remains challeng- ing: despite prompting for cross-section reasoning, many candidate multi-hop questions are filtered for invalid hop structure, which may underrepre- sent realistic multi-constraint regulatory reason- ing. More effective multi-hop items likely require deeper expert involvement and may link to more than two sections, (e.g., one may ask a question related to food effect on pharmacokinetics, effi- cacy and safety or whether the dosage needs to be adjusted under fed condition). Fourth, our evaluation trades off reliability and cost. We use an LLM-based grader for correctness and overlap-based citation metrics, both of which can be imperfect: graders may mishandle border- line cases, and overlap scores can penalize seman- tically correct but differently localized citations, especially under redundancy or imperfect chunk- ing. Future work should incorporate calibrated hu- man adjudication for ambiguous items and stronger faithfulness checks (e.g., entailment-based verifica- tion that cited text supports each claim). Finally, refusal questions provide a controlled test of hallucination resistance but rely on heuristic templates and keyword exclusion, which may not capture the full diversity of unanswerable queries in practice. Expanding refusal evaluation to in- clude naturally occurring ambiguous or partially answerable questions is an important direction. Acknowledgments We thank David Hall, Hanwen Xu, Nelson Liu, Percy Liang and Sheng Wang for useful conversa- tions. Funding: BX is supported by Australian Amer- ican Fulbright Commission Future Scholarship. RBA is supported by Burroughs Wellcome Fund Grant 1074128.RBA is supported by NIH GM153195. This project was also supported by the Food and Drug Administration (FDA) of the U.S. Department of Health and Human Services (HHS) as part of a financial assistance award Cen- ter of Excellence in Regulatory Science and Inno- vation (CERSI) grant to University of California, San Francisco (UCSF) and Stanford University, U01FD005978 funded by FDA/HHS. The contents are those of the author(s) and do not necessarily represent the official views of, nor an endorsement, by FDA/HHS, or the U.S. Government. References Anthropic. 2025. System card: Claude sonnet 4.5. Anthropic. 2026. System card: Claude opus 4.6. Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Pre- ston Bowman, Joaquin Quiñonero Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, An- drea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. 2025. Healthbench: Evaluating large language models towards improved human health. ArXiv, abs/2505.08775. Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D’arcy, David Wadden, Matt Latzke, Minyang Tian, Pan Ji, Shengyan Liu, Hao Tong, Bohao Wu, Yanyu Xiong, Luke Zettlemoyer, and 6 others. 2024. Openscholar: Synthesizing scientific literature with retrieval- augmented lms. Preprint, arXiv:2411.14199. Suhana Bedi, Hejie Cui, Miguel Fuentes, Alyssa Unell, Michael Wornow, Juan M. Banda, Nikesh Kotecha, Timothy Keyes, Yifan Mai, Mert Oez, Hao Qiu, Shrey Jain, Leonardo Schettini, Mehr Kashyap, Ja- son Alan Fries, Akshay Swaminathan, Philip Chung, Fateme Nateghi, Asad Aali, and 62 others. 2025. Medhelm: Holistic evaluation of large language mod- els for medical tasks. Preprint, arXiv:2505.23802. Balu Bhasuran, Qiao Jin, Angelique Deville, Yonghui Wu, Karim Hanna, Zhiyong Lu, and Zhe He. 2025. Labqar: A manually curated dataset for question an- swering on laboratory test reference ranges and inter- pretation. medRxiv. Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, Alexandre Sallinen, Alireza Sakhaeirad, Vinitra Swamy, Igor Krawczuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary-Anne Hartley, Martin Jaggi, and Antoine Bosselut. 2023. Meditron-70b: Scaling medical pretraining for large language models. Preprint, arXiv:2311.16079. FDA. Fdalabel: Full-text search of drug product label- ing. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schel- ten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mi- tra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Se- bastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense infor- mation retrieval with contrastive learning. Preprint, arXiv:2112.09118. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 2567– 2577, Hong Kong, China. Association for Computa- tional Linguistics. Anastasia Kritharaa, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras. 2023. Bioasq- qa: A manually curated corpus for biomedical ques- tion answering. Sci Data, page 170. Yibin Lei, Liang Ding, Yu Cao, Changtong Zan, An- drew Yates, and Dacheng Tao. 2023. Unsupervised dense retrieval with relevance-aware contrastive pre- training. In Findings of the Association for Computa- tional Linguistics: ACL 2023, pages 10932–10940, Toronto, Canada. Association for Computational Lin- guistics. Alexander H. Liu, Kartik Khandelwal, Sandeep Sub- ramanian, Victor Jouault, Abhinav Rastogi, Adrien Sadé, Alan Jeffares, Albert Jiang, Alexandre Cahill, Alexandre Gavaudan, Alexandre Sablayrolles, Amélie Héliou, Amos You, Andy Ehrenberg, Andy Lo, Anton Eliseev, Antonia Calvi, Avinash Soori- yarachchi, Baptiste Bout, and 101 others. 2026. Ministral 3. Preprint, arXiv:2601.08584. Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. 2022. Biogpt:generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23(6). OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M ̨adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024. Gpt-4o system card. Preprint, arXiv:2410.21276. OpenAI. 2025. Update to gpt-5 system card: Gpt-5.2. Anusri Pampari, Preethi Raghavan, Jennifer J. Liang, and Jian Peng. 2018. emrqa: A large corpus for question answering on electronic medical records. In Conference on Empirical Methods in Natural Lan- guage Processing. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and be- yond. Found. Trends Inf. Retr., 3(4):333–389. Gunther Schadow. 2007.Assessing the impact of hl7/fda structured product label (spl) content for med- ication knowledge management. AMIA Annu Symp Proc., pages 646–650. Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, Justin Chen, Fereshteh Mahvar, Liron Yatziv, Tiffany Chen, Bram Sterling, Stefanie Anna Baby, Su- sanna Maria Baby, Jeremy Lai, Samuel Schmidgall, and 62 others. 2025. Medgemma technical report. Preprint, arXiv:2507.05201. Yiwen Shi, Ping Ren, Jing Wang, Biao Han, Taha ValizadehAslani, Felix Agbavor, Yi Zhang, Meng Hu, Liang Zhao, and Hualou Liang. 2023. Lever- aging gpt-4 for food effect summarization to en- hance product-specific guidance development via iter- ative prompting. Journal of Biomedical Informatics, 148:104533. Jake Silberg, Kyle Swanson, Elana Simon, Angela Zhang, Zaniar Ghazizadeh, Scott Ogden, Hisham Hamadeh, and James Zou. 2024. Unitox: leverag- ing llms to curate a unified dataset of drug-induced toxicity from fda labels. In Proceedings of the 38th International Conference on Neural Information Pro- cessing Systems, NIPS ’24, Red Hook, NY, USA. Curran Associates Inc. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, Akshay Nathan, Alan Luo, Alec Helyar, Aleksander Madry, Aleksandr Efremov, Aleksandra Spyra, Alex Baker- Whitcomb, Alex Beutel, Alex Karpenko, and 465 others. 2025. Openai gpt-5 system card. Preprint, arXiv:2601.03267. K. Singhal, Shekoofeh Azizi, Tao Tu, Said Mah- davi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Kumar Tanwani, Heather J. Cole- Lewis, Stephen J. Pfohl, P A Payne, Martin G. Seneviratne, Paul Gamble, Chris Kelly, Nathaneal Scharli, Aakanksha Chowdhery, P. A. Mansfield, Blaise Agüera y Arcas, Dale R. Webster, and 11 oth- ers. 2022. Large language models encode clinical knowledge. Nature, 620:172 – 180. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic com- pression of pre-trained transformers.Preprint, arXiv:2002.10957. Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. 2024. Mea- suring short-form factuality in large language models. Preprint, arXiv:2411.04368. Leihong Wu, Hong Fan, Yanyan Qu, Joshua Xu, and Weida Tong. 2025. Leveraging fda labeling docu- ments and large language model to enhance anno- tation, profiling, and classification of drug adverse events with askfdalabel. Drug Saf., 48:655–665. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Day- iheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. A Additional results and analyses A.1Evaluation and human annotation details We had one domain judge, an FDA regulatory as- sessor who manually reviews drug labels, anno- tate a question’s relevance to FDA drug review (Table 3). First, we ask the reviewer whether a question is relevant or irrelevant, and match it to the LLM’s assessment. Second, we ask the asses- sor to categorize the question into one predefined regulatory topic category reflecting standard FDA label organization. Question topics can be found in Appendix B.4. We had three computer science and biomedical data science annotators evaluate QA correctness and question quality, and compared their annota- tions to LLM-as-judge outputs. Table 4 summa- rizes the agreement between human and LLM judg- ments in human consensus labels and the unani- mous subset. Table 5 shows annotator variability and human inter-annotator reliability. Similarly, we had the above annotators evaluate answer correctness, and compared it against our SimpleQA prompt (modified for the specific FDA drug label task). Human and LLM agreement, and human annotator variability are reported in Tables 4 and 5. All LLM prompt details can be found in Ap- pendix B.10. MetricnAccuracyF1PrecisionRecall Relevance500.7000.8000.9680.682 Category480.646– Table 3: Agreement between human and LLM judg- ments for question relevance and category. Accuracy is percentage agreement. F1, precision and recall are reported only for the binary relevance setting (LLM as predictor; human as reference). A.2 Details on selected models Tables 6 and 7 show details on selected retriever and LLM models selected for evaluation. A.3 Text metrics Table 8 shows additional analysis, with the text metrics of BLEU, METEOR, and ROUGE-L. A.4 Additional plots Figures 3, 4 and 5 show additional analysis on model performance. Figure 3: Evidence access ablation (factual + multi-hop). Answer accuracy improves substantially when models are given oracle (gold) passages, but drops in the full- label setting with citation requirements, highlighting evidence selection/grounding as a key bottleneck. Stage / JudgeTaskConsensusUnanimous nAccuracyF1Prec.Rec.nAccuracyF1Prec.Rec. QA correctness (LLM judge)Factual190.6840.750 0.900 0.643 100.9000.933 1.000 0.875 QA correctness (LLM judge)Multi-hop 200.5500.690 1.000 0.52690.7780.875 1.000 0.778 QA correctness (LLM judge)Refusal100.7000.769 1.000 0.62560.8330.909 1.000 0.833 Question quality (LLM judge)Factual200.4000.455 0.833 0.312 140.3570.526 0.833 0.385 Question quality (LLM judge)Multi-hop 200.7000.400 0.333 0.500 120.7500.400 0.333 0.500 Answer correctness (SimpleQA) Factual190.7890.800 1.000 0.667 150.9330.941 1.000 0.889 Answer correctness (SimpleQA) Multi-hop 150.7330.667 0.800 0.57180.7500.750 0.750 0.750 Table 4: LLM and judge agreement. Automated judges compared against (i) human consensus labels and (i) the unanimous subset (all human annotators agree). Accuracy is percent agreement. F1, precision and recall treat the judge as predictor and the human labels as gold. Stage / JudgeTasknAnnotator variabilityHuman inter-annotator reliability Accuracy rangeF1 rangePrec. rangeRec. rangePairwise match rangePairwise F1 range QA correctness (LLM judge)Factual190.684–0.7370.700–0.7830.700–1.0000.625–0.7000.579–0.7370.692–0.828 QA correctness (LLM judge)Multi-hop200.500–0.7500.643–0.7620.800–1.0000.500–0.7270.500–0.8500.667–0.919 QA correctness (LLM judge)Refusal100.600–0.8000.714–0.8331.000–1.0000.556–0.7140.600–0.9000.750–0.941 Question quality (LLM judge)Factual200.400–0.4000.455–0.4550.833–0.8330.312–0.3120.700–0.9000.812–0.938 Question quality (LLM judge)Multi-hop200.650–0.7000.250–0.5330.167–0.6670.400–0.5000.650–0.8500.364–0.571 Answer correctness (SimpleQA)Factual190.789–0.8420.800–0.8421.000–1.0000.667–0.7270.800–0.8500.833–0.880 Answer correctness (SimpleQA)Multi-hop150.467–0.8000.429–0.7270.600–0.8000.333–0.6670.500–0.8750.500–0.857 Table 5: Annotator variability and human inter-annotator reliability. First, the range across individual human annotators when compared against the automated judge (min–max over annotators). Second, pairwise ranges over all annotator pairs for each task. Figure 4: Answer correctness vs. citation quality in full-label setting. Relationship between overall answer accuracy and citation overlap (micro-F1) in the full- label setting, showing that better answers do not always imply better citations. Figure 5:Refusal behavior in full-label setting. Precision-recall tradeoff for refusal questions in the full- label setting; models vary in hallucination resistance (precision) despite uniformly high recall. RetrieverTypeRepresentation / ScoringReference BM25SparseToken-based ranking over SPL passages(Robertson and Zaragoza, 2009) Cosine SimilarityDense (embeddings)Cosine similarity over passage embeddings(Wang et al., 2020) ReContrieverDenseLearned dense retriever over passages(Lei et al., 2023) Table 6: Retrievers evaluated for top-k evidence selection in retrieval-augmented QA. FamilyModelAccessReference ProprietaryGPT-4o-miniAPI(OpenAI et al., 2024) ProprietaryGPT-5.1API(Singh et al., 2025) ProprietaryGPT-5.2API(OpenAI, 2025) ProprietaryClaude-Sonnet-4-5API(Anthropic, 2025) ProprietaryClaude-Opus-4-6API(Anthropic, 2026) Open-weightLlama-3.1-8B-InstructLocal(Grattafiori et al., 2024) Open-weightLlama-3.3-70B-InstructLocal(Grattafiori et al., 2024) Open-weightMinistral-3-14B-Instruct-2512Local(Liu et al., 2026) Open-weightQwen3-14BLocal(Yang et al., 2025) Open-weightQwen3-32BLocal(Yang et al., 2025) Table 7: Models evaluated in our experiments. ModelClosed-book FactClosed-book MHOracle passages FactOracle passages MHFull label FactFull label MH BLEU MET R-LBLEU MET R-LBLEU METR-LBLEU METR-LBLEU MET R-LBLEU MET R-L Llama-8B0.030.11 0.120.030.11 0.130.330.580.560.210.470.430.230.40 0.410.120.28 0.30 Llama-70B 0.070.27 0.250.100.36 0.300.340.590.570.230.500.450.230.43 0.410.160.37 0.34 Ministral-14B0.030.20 0.180.040.28 0.230.160.430.500.100.380.370.180.37 0.390.100.27 0.26 Qwen-14B0.080.29 0.270.120.37 0.330.330.600.560.240.540.470.210.41 0.390.140.31 0.28 Qwen-32B 0.080.29 0.260.100.38 0.310.310.600.550.210.540.450.220.42 0.410.160.37 0.34 Claude Sonnet0.060.30 0.220.060.35 0.240.310.620.550.170.510.400.210.45 0.400.120.34 0.28 GPT-4o-mini0.080.29 0.260.120.38 0.330.360.620.580.270.570.500.210.42 0.390.160.35 0.32 GPT-5.10.090.30 0.280.070.33 0.280.300.580.560.170.480.410.210.41 0.410.120.32 0.29 Table 8: Text-overlap metrics across evidence settings. Entries report BLEU, METEOR (MET), and ROUGE-L (R-L) (higher is better) for factual (Fact) and multi-hop (MH) questions in the closed-book, oracle-passages, and full-label settings. B Dataset creation details B.1 Source Documents and Preprocessing To build the dataset, we curate a set of drug la- bel SPL XML documents from the FDALabel Database (FDA). Like similar works (Wu et al., 2025; Silberg et al., 2024), we start with the set of all human prescription drugs. We remove labels whose route of administration include the words topical, irrigational, intradermal or inhalation, as we are mostly interested in drugs with oral admin- istration. We randomly sampled 700 drug labels for the benchmark. Each SPL has a hierarchical structure that in- cludes a mandatory Highlights of Prescribing In- formation section (which is not included in the main drug label retrieval), PLR-mandated labeled sections, optional subsections, and structured meta- data. We parse each XML file into section-level text chunks, using a custom rule-based extractor that preserves the hierarchical structure while nor- malizing content into contiguous natural-language segments. For each SPL, we extract sections with: (1) section titles, (2) LOINC section codes (e.g., 34084-4 for Adverse Reactions), (3) chunk indices (at document and section levels) for provenance and citation tracking, and (4) drug-level metadata. Text normalization removes XML artifacts, for- matting tags, superscripts, hyperlinks, and tables that cannot be reliably converted to plain text. Very short fragments created by XML segmentation (typ- ically originating from bullet lists, warnings, or embedded bold headers) are merged with adjacent segments to produce semantically coherent units. The resulting corpus contains tens of thousands of section-level chunks that serve as the base context for question generation. B.2 Question Types Table 9 shows examples of factual, multi-hop and refusal questions. Factual. The FDA Highlights section provides a concise summary of the most essential clinical information in a drug label, including indications, dosage forms, contraindications, boxed warnings, and key safety considerations. Since the highlights are carefully curated and tightly coupled to the cor- responding full sections of the SPL, it serves as a high-quality but weakly-supervised source of fac- tual supervision. For each highlight in the label, we derive the corresponding full-length sections. We then use the section header to select from a prede- fined library of template-based questions tailored to common regulatory information needs. For ex- ample, from the Indications and Usage heading, we generate question templates such as “What is drug used to treat?” or “What conditions is drug indicated for?”; from Dosage and Administration, we generate templates such as “What is the recom- mended dosage of drug?”. As each question is explicitly anchored to both the highlights summary and its corresponding extractable full section span, these items provide a reliable factual baseline for evaluation. To capture a broader range of clinically and reg- ulatory meaningful content beyond the Highlights summary, we generate section-level factual ques- tions using a self-instruct prompting strategy. For each drug label, we iterate through individual sec- tions and select one chunk at a time as the input context. Each chunk is annotated with its section identifier and document chunk index to preserve provenance. Using few-shot prompts to an LLM (GPT-4o-mini 2 (OpenAI et al., 2024)), we instruct the model to propose clinically relevant, factually grounded questions whose answers can be derived directly from the provided text. Multiple QA items are generated per chunk to enhance coverage. This method captures a richer set of real-world informa- tion needs, including dose adjustments, contraindi- cated populations, drug-drug interactions, adverse event frequencies, and pharmacokinetic properties. Since the questions originate from individual sec- tions rather than curated summaries, they reflect the full variability of label writing styles and provide a challenging and realistic substrate for factual QA evaluation. Multi-hop. Many regulatory and clinical infor- mation needs require synthesizing evidence across multiple sections of a drug label. To evaluate this capability, we construct multi-hop QA items requir- ing integration of two distinct sections. We begin by sampling pairs of sections that often interact clinically, e.g., Indications and Usage paired with Dosage Forms and Strengths, or Pharmacokinet- ics paired with Adverse Reactions. For each se- lected pair from a single drug label, we retrieve one chunk from each section, concatenate their texts, and provide both as context to an LLM. The model is explicitly instructed to generate questions whose 2 Unless otherwise specified, all generation and filtering prompts use GPT-4o-mini. TypeDescriptionExample QuestionGold Target / Evidence (excerpt) FactualAnswerable from a single section What is the recommended start- ing dose of WAKIX for adult pa- tients? Gold: The recommended starting dose is 8.9 mg once daily. Evidence: Adult Patients: Initiate WAKIX at 8.9 mg once daily and titrate to a maximum recommended dosage of 17.8 mg once daily after 7 days. Multi- hop Requires reasoning across sections What is the peak blood level of dextroamphetamine achieved af- ter ingesting 10 mg of the oral solution compared to the avail- able strength of the oral solu- tion? Gold: The peak blood level of dextroam- phetamine after ingesting 10 mg of the oral solu- tion is 33.2 ng/mL, while the available strength of the oral solution is 5 mg/5 mL. Evidence: ...produced an average peak dex- troamphetamine blood level of 33.2 ng/mL... Dextroamphetamine Sulfate Oral Solution 5 mg/5 mL... RefusalNotanswerable fromthelabel; model should ab- stain How should BNP be monitored in patients taking New Day? Gold: NOT_ANSWERABLE Table 9: Example items from the benchmark spanning factual, multi-hop, and refusal question types. Evidence excerpts are abbreviated for space. answers require simultaneous reasoning over both sections and cannot be answered from either sec- tion alone. Valid multi-hop questions that gen- uinely depend on both sections are retained and subsequently evaluated using a judge model that en- forces cross-sectional evidence requirements. This category serves as a stress test for cross-contextual reasoning and evidence integration in long, hetero- geneous biomedical documents. Despite these constraints, initial generations of- ten produce questions that reference unrelated med- ical concepts or introduce unsupported associations (e.g., “What is the breast cancer risk associated with long-term use of CAMRESE, and how is it supplied?” or “Why must PROGESTERONE Cap- sules be avoided in patients with a peanut allergy despite its renal excretion of metabolites?”). Such items are removed during downstream filtering. Refusal. Safe model behavior requires not only producing correct answers when information is present but also refusing to answer when informa- tion is absent. To evaluate hallucination resistance, we construct a set of negative-control refusal ques- tions referencing biomedical concepts absent from the label. We begin with a library of template-based prompts commonly used in clinical and regulatory settings, such as “What is the indication of drug for treating endpoint in population?” or “What is the threshold value of biomarker for initiat- ing treatment with drug?”. Into these templates, we insert a biomedical keyword, entity, or condi- tion that is guaranteed to be absent from the SPL. We verify absence automatically through full-label keyword search. To improve naturalness and sur- face variability, each template is then rephrased by an LLM, producing more linguistically diverse unanswerable questions while preserving the in- serted out-of-scope entity. The correct system re- sponse must decline to answer or explicitly state that the label does not contain relevant information. These items provide a clean measure of hallucina- tion propensity under controlled distribution shift. This category is particularly important for regula- tory applications, where unsupported claims pose significant safety and compliance risks. B.3 Generation Pipeline Context Selection. We choose the appropriate text source depending on the question type. High- lights produce template-driven factoids; single sec- tions support self-instruct factual items; and section pairs enable multi-hop reasoning. All metadata, such as drug name, section identifiers, and chunk indices, is preserved to ensure traceability. This is particularly important for the task of document- level citations that is part of the evaluation suite in an open-book full drug label setting (see Appendix B.6). Prompted Generation. We craft task-specific few-shot prompts that emphasize regulatory pre- cision, prohibition of hallucination, and clarity of reasoning. For self-instruct factual and multi-hop items, we ask the LLM to create question-answer pairs specifically relevant to only the specified con- text(s). For multi-hop questions, prompts stress that the answer must require information from both sections. For refusal questions, prompts reinforce that the model should avoid inventing content and instead explicitly refuse. Each generation round produces multiple independent samples to enhance diversity. Expert-Guided Stage. Domain experts (FDA drug label reviewers) annotate a seed set of QAs and provide generalizable constraints, e.g., deter- mining regulatory and clinical relevance of ques- tions, importance of traceability and provenance, compound multi-hop questions relevant that may include maximum daily dose in certain patient pop- ulations. We encode this feedback into refined prompts and filtering criteria for improving bench- mark examples. B.4 Filtering and Quality Control Stage 1: Rule-Based Filtering.We first apply a set of deterministic, rule-based filters designed to eliminate structurally invalid, irrelevant, or weakly grounded QAs before any model-based judgment. These rules encode minimal regulatory and linguis- tic constraints that can be verified without seman- tic interpretation, ensuring transparency and repro- ducibility. Structural validity checks discard QAs missing any required field (question, context, or answer) or exhibiting malformed serialization. Each question is required to explicitly mention the target drug by name or known alias, preventing generic or context- free questions from entering the benchmark. We enforce length guards on both questions (10-200 characters) and answers (5-600 characters), reject- ing items that fall outside reasonable minimum and maximum thresholds in order to remove trivial, un- derspecified prompts as well as excessively verbose or malformed generations. For multi-hop candidates, we enforce additional structural constraints to prevent degenerate single- hop formulations. We reject questions containing artificial conjunctions such as “and” or “as well as” that merely concatenate two unrelated facts without requiring integrative reasoning, collapsing into separate single-hop formulations, e.g., “What is the breast cancer risk associated with long-term use of CAMRESE, and how is it supplied?”. This filter removes a common failure mode in which the model produces compound questions that are answerable independently from each section. Finally, we apply span-based support checks to ensure extractive grounding. For factual questions, the answer must exhibit a minimum token-level overlap of 0.10 with the provided context. For multi-hop questions, we require a minimum over- lap of 0.10 with each constituent section indepen- dently. Items failing any of these criteria are deter- ministically discarded prior to LLM-based valida- tion. Stage 2: LLM-as-Judge Validation.After rule- based filtering, we apply a structured LLM-as- judge pipeline to perform semantic validation of remaining QAs. This stage is responsible for as- sessing factual correctness, evidentiary grounding, and regulatory relevance using only the provided label context. To ensure interpretability and au- ditability, the LLM-based filtering is decomposed into three explicit judgment stages, each producing structured outputs that are logged and retained for downstream analysis. (i) Question Relevance Classification. In the second stage, the judge evaluates whether the ques- tion itself is relevant to FDA regulatory review, independent of answer correctness. Each question is classified into exactly one predefined regulatory topic category reflecting standard FDA label orga- nization: 1. Indications & Usage 2. Dosage & Administration 3. Dosage Forms & Strengths & Formulation 4. Contraindications 5. Safety & Serious Risks 6. Adverse Events 7. Drug Interactions 8. Use in Specific Populations 9. Pharmacokinetics Questions that do not correspond to regulatory la- beling content are assigned to a None / Irrelevant category. In addition, the judge labels each ques- tion as relevant, somewhat relevant, or irrelevant to bioequivalence review. Questions deemed irrele- vant or only weakly related to regulatory decision- making are filtered out at this stage, ensuring topi- cal alignment of the final benchmark. (i) QA Validation and Evidence Extraction.In the first stage, the judge evaluates whether the pro- posed answer is supported by the provided context text alone, ignoring any external knowledge. For factual questions, the judge classifies each item as supported or unsupported and, when supported, extracts the minimal sentence-level evidence from the label that directly substantiates the answer. For multi-hop questions, the judge independently evalu- ates support from each section, producing separate fields for evidence from Context A and Context B, and explicitly determines whether both sections are required to form a complete answer. Questions whose answers are supported by only one section, despite being framed as multi-hop, are penalized and removed. For refusal questions, the judge ver- ifies whether the absence-of-information claim is correct; if the label does in fact contain an answer, the judge extracts the contradicting evidence span. This stage ensures that retained QAs are factually grounded and traceable to specific passages in the source document. (i) Question Quality Assessment. In the final stage, the judge evaluates the intrinsic quality of each question using task-specific criteria. For fac- tual questions, the judge flags failures such as an- swer leakage (where the question states its own an- swer), context mismatch (mentioning unsupported entities, populations, or conditions), unanswerabil- ity from the provided context, or low benchmarking utility due to vagueness or triviality. For multi-hop questions, additional failure modes are assessed, including one-sided answerability, artificial or non- sensical hop construction between sections, answer leakage from one context into the question, and inability to answer using the combined contexts. Questions exhibiting any of these failure modes are removed. This stage ensures that retained questions are not only factually valid but also meaningful and well-formed evaluation items. Together, these three LLM-based filtering stages provide a conservative but interpretable semantic validation layer that complements the deterministic rule-based filters. By separating correctness, rele- vance, and question quality judgments, the pipeline enables fine-grained auditing and supports scalable, reproducible benchmark construction under regula- tory constraints. Precision-Oriented Filtering Rationale.We ex- plicitly tune the LLM-based filtering stage to maxi- mize precision rather than recall. Human annota- tion studies conducted on stratified samples of fac- tual, multi-hop, and refusal items show that while human–human agreement is moderate, reflecting the inherent ambiguity of regulatory QA, the LLM judge achieves consistently high precision with re- spect to human consensus, albeit with lower recall. In practice, this means the filter preferentially ex- cludes borderline or ambiguous items that some humans might accept, while rarely admitting in- correct examples. We consider this trade-off de- sirable for benchmark construction, as omission of marginal cases is preferable to inclusion of in- correct or weakly grounded QAs in high-stakes regulatory evaluation. Sampling for Human Review and Auditability. To support human auditing and agreement analysis, we construct a reproducible sampling pipeline over filtered QAs. Items are deduplicated by unique question identifiers, bucketed by regulatory cate- gory and relevance label, and sampled to ensure balanced coverage across question types and sec- tions. For each sampled item, we export structured review files containing the question, context, model answer, judge decision, and metadata required for downstream human evaluation. This infrastruc- ture enables systematic comparison between hu- man judgments and LLM-based filtering decisions and provides an auditable trail for benchmark cura- tion. Final Dataset. After completion of all filtering stages, the benchmark contains 17223 high-quality QAs across three categories (see Table 10). De- tailed statistics are reported in Appendix B.5. The resulting dataset reflects a conservative but reliable subset of regulatory QA examples, suitable for eval- uating factual grounding, cross-section reasoning, and safe refusal behavior in large language models. B.5 Dataset Statistics and Filtering Diagnostics See Table 10 for dataset statistics. Figure 6 shows a Sankey plot on the filtering steps in order of: rules- based automatic filter; question relevance, QA ac- curacy, and question quality LLM-based filters. B.6 Benchmark Tasks Closed-Book Question Answering. In the closed-book setting, models are provided with the question only and must generate an answer with- out access to any label context. This task evalu- ates parametric knowledge retention, robustness to Dataset statisticCount SPL drug labels700 Final QAs17,223 Factual9,888 Multi-hop3,400 Refusal3,935 Table 10: Dataset summary statistics. Factual Multi-Hop Refusal Rules filterQ relevanceQA accuracyQA quality ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Figure 6: Sankey plot of retention of document QA items across filtering stages.✓denotes number of items kept after each stage. hallucination, and a model’s tendency to overgen- eralize from pretraining rather than abstain when uncertain. Closed-book QA is particularly infor- mative for assessing whether models incorrectly rely on memorized drug knowledge that may be outdated, incomplete, or inapplicable to a specific formulation. Open-Book Question Answering (Full Label with Citations). In the open-book full-label set- ting, models are provided with the entire SPL doc- ument divided into passages with explicit passage identifiers, e.g.,PASSAGE_0005. Models must gen- erate (1) a concise natural-language answer and (2) a list of cited passage ids that support the answer. This setting evaluates long-context comprehension, evidence grounding, and provenance quality under realistic document length and redundancy. Requir- ing explicit citations enables automatic verification of whether the model’s stated evidence actually supports the answer and helps separate failures of retrieval/reading from failures of reasoning. Retrieval Task for Open-Book Question An- sweringTo better reflect practical reviewer work- flows and to decouple long-context reading from evidence selection, we also evaluate retrieval- augmented open-book QA. Given a question and a corpus of passage-level label chunks, a retriever se- lects the top-kpassages (we usek=5unless other- wise stated). We experiment with multiple retrieval functions, including (1) sparse retrieval (BM25) (Robertson and Zaragoza, 2009), (2) embedding- based cosine similarity over passage representa- tions, and (3) a dense retriever (ReContriever) (Lei et al., 2023; Izacard et al., 2022). This setting eval- uates the task of identifying relevant evidence prior to answer generation. Section-Aware Question Answering To isolate reasoning from evidence selection, we include a section-aware setting where the model receives the question along with the specific section chunk(s) (“gold” retrieval variant) from which the QA was generated. For self-instruct factual questions, the model receives a single section chunk; for multi- hop questions, the model receives exactly two sec- tion chunks corresponding to Context A and Con- text B. This diagnostic setting serves as an approx- imate upper bound on answer quality when the relevant evidence is perfectly localized in an oracle reference, and helps quantify how much error in open-book settings is attributable to retrieval/selec- tion rather than reasoning. Refusal Detection Finally, we evaluate refusal behavior as a dedicated negative control task, to as- sess a model’s ability to safely abstain from answer- ing when the label does not contain the requested information. The expected behavior is to explicitly abstain (e.g.,NOT_ANSWERABLEor an equivalent re- fusal) rather than fabricate an answer. We track hallucinated answers as critical failures, and we also track false refusals (refusing despite the label containing relevant information) as an error mode that harms usability. This task is particularly im- portant in regulatory settings, where unsupported claims can pose safety and compliance risks. B.7 Automatic Metrics Answer Quality.For answerable questions (fac- tual, and multi-hop), we compute BLEU, ME- TEOR, and ROUGE-L between the model predic- tion and the reference answer when a reference is available. These metrics provide a coarse measure of lexical similarity, but are known to be insuffi- cient for capturing factual correctness in long-form biomedical QA due to multiple valid phrasings and levels of granularity. Accordingly, they are always reported in conjunction with LLM-as-judge scores (see Appendix B.8). For questions where the gold answer is present but the model outputs a refusal (e.g., NOT_ANSWERABLE), overlap-based metrics naturally assign low scores, and the LLM-as-judge marks the prediction as incorrect. This behavior appropriately penalizes over-refusal in answerable cases without requiring special handling. Refusal Correctness.For refusal questions, eval- uation is framed as a binary classification task: the model is correct if it explicitly refuses to answer when the information is absent from the label, and incorrect otherwise. We report precision, recall, and accuracy. Because hallucinated answers in re- fusal cases constitute a critical safety failure, we emphasize precision as the primary metric, while still reporting recall to quantify over-conservatism. Provenance and Citation Evaluation.For open- book full-label and retrieval-based settings, models are required to produce explicit citations to support- ing passages (Asai et al., 2024). For each example, we compare the set of cited passages against the gold provenance associated with the QA genera- tion process. We compute per-example precision, recall, and F1 over cited passage identifiers and report averages across the dataset. We explicitly distinguish between false-positive citations (citing irrelevant passages) and false- negative citations (failing to cite required evidence). In regulatory contexts, missing critical evidence is often more harmful than citing additional, partially relevant passages. Accordingly, we report recall- oriented citation metrics alongside F1 to reflect the importance of evidence completeness during review. B.8 LLM-Based Grading Framework Many long-form benchmarks items do not admit a single canonical reference answer, so we employ an LLM-as-judge framework. To evaluate model predictions at scale, we use an LLM-based grading framework (with GPT-5.1) that compares predicted answers against gold reference targets derived from FDA drug labels. For each evaluation example, the grader is pro- vided with (1) the question, (2) a gold target answer extracted from the label, and (3) the model’s pre- dicted answer. The grader assigns one of three mutually exclusive labels:CORRECT,INCORRECT, orNOT_ATTEMPTED. A prediction is marked as cor- rect if it contains all clinically important informa- tion present in the gold target, does not contradict the reference, and does not introduce unsupported clinical claims. Minor wording differences and paraphrases are permitted as long as the clinical meaning is preserved. Predictions are marked in- correct if they contradict the gold target, introduce unsupported clinical facts (e.g., incorrect doses, populations, contraindications), or omit major re- quired elements. Predictions are marked as not attempted if the model explicitly abstains or fails to provide the requested information without intro- ducing incorrect claims. Building upon the SimpleQA evaluation frame- work (Wei et al., 2024), the grading rubric is de- signed to reflect regulatory priorities. In partic- ular, missing or incorrect numeric values (e.g., doses, frequencies, thresholds), incorrect popula- tion constraints, and unsupported contraindications are treated as critical errors and result in an incor- rect label. When the gold target specifies refusal or absence of information, any specific clinical recom- mendation from the model is graded as incorrect, while explicit abstention is graded as not attempted. This structured grading scheme enables consistent comparison across models while preserving clin- ically meaningful distinctions between incorrect answers and safe non-attempts. We emphasize that the LLM judge is used as a scalable proxy for semantic validation rather than as a source of ground truth. Its outputs are therefore evaluated against human judgments in a separate annotation study in Appendix B.9. B.9 Human Annotation Study To validate the reliability of automatic grading and to characterize ambiguity in regulatory QA, we conduct a human annotation study using the same grading rubric applied by the LLM-based evalua- tor. Annotators label each prediction asCORRECT, INCORRECT, orNOT_ATTEMPTED, following identi- cal criteria regarding clinical completeness, contra- dictions, and appropriate abstention. Each example is independently annotated by three reviewers with biomedical or regulatory ex- pertise. We measure inter-annotator agreement us- ing Cohen’sκ. We additionally report bootstrap confidence intervals over agreement statistics to quantify uncertainty due to sample size. Because regulatory QA often involves borderline cases and underspecified label language, we do not expect near-perfect agreement. Instead, these mea- surements provide a realistic estimate of task ambi- guity and establish an upper bound on achievable automatic grading consistency. Overall, this evaluation framework combines au- tomatic metrics, structured model-based judgment, and human validation to provide a robust and inter- pretable assessment of regulatory QA performance under both answerable and unanswerable condi- tions. B.10 Dataset creation prompts Generation prompts. Template for highlight- generated QA examples. "INDICATIONS AND USAGE": [ "What is drug used to treat?", "What conditions is drug indicated for?" ], "DOSAGE AND ADMINISTRATION": [ "What is the recommended dosage regimen of drug?", "How should drug be administered?" ], "DOSAGE FORMS AND STRENGTHS": [ "What dosage forms are available for drug?", "What strengths does drug come in?" ], "CONTRAINDICATIONS": [ "Who should not take drug?", "What are the contraindications for drug?" ], "WARNINGS AND PRECAUTIONS": [ "What important warnings or precautions are associated with drug?", "What safety risks are listed for drug?" ], "ADVERSE REACTIONS": [ "What are the side effects of drug?", "What adverse reactions have been reported for drug?" ], "DRUG INTERACTIONS": [ "What important drug interactions are noted for drug?", "Which medications should be avoided with drug?" ], "USE IN SPECIFIC POPULATIONS": [ "What is known about the use of drug in specific populations?", "Are there any population-specific considerations for drug?" ], "WARNING: ": [ "What serious risks are included in the boxed warning for drug?", "What does the boxed warning for drug emphasize?" ] Prompt template for self-instruct generation of single-hop QA examples. Generate k grounded Q/A items from the CONTEXT below. Requirements: - Use ONLY facts in CONTEXT. - Keep answers concise (less than or equal to 2 sentences or a compact list). - Questions and answers should be concrete and clearly useful for bioequivalence or product-specific guidances development. - Output a JSON array ONLY with objects: "question":"...", "answer":"...", "type":"factual|list|numeric" - Do NOT include citations or any extra fields. CONTEXT (section: "title" / label: label / chunk_index: chunk_index): context Prompt template for self-instruct generation of multi-hop QA examples. Create ONE multi-hop item that needs BOTH contexts. Rules: - Single-clause; avoid " and ". - One decision, not two unrelated facts. - Each section must contribute a distinct needed fact. Positive examples (style): - (Dosage + Hepatic impairment) - "What starting dose is recommended for patients with mild hepatic impairment treated for angina?" - (Indication + Contraindication) - "Although indicated for postmenopausal osteoporosis, why must the drug not be used during pregnancy?" Anti-examples (reject internally): - Storage temperature + adult max dose (two independent instructions). - "Why is it important to use this drug for bacterial infections, and what is the active ingredient?" (double ask). - Items answerable from only one section. CONTEXT A (section: "title_a" / label: label_a): ctx_a CONTEXT B (section: "title_b" / label: label_b): ctx_b Filtering prompts.Prompt template for QA cor- rectness filtering of factual questions. You are an expert fact-checker evaluating the correctness of an answer. Your task: Determine whether the given ANSWER correctly answers the QUESTION based **only** on the information in the CONTEXT. Ignore any external knowledge. Only rely on the CONTEXT for verification. Follow these steps internally (do not include your reasoning in the output): 1. Read the QUESTION carefully. 2. Locate relevant parts of the CONTEXT that relate to the QUESTION. 3. Compare the ANSWER to those parts of the CONTEXT. 4. Decide if the ANSWER is fully supported (True) or not supported (False). 5. If the ANSWER is True, extract the exact sentence(s) from the CONTEXT that directly support it. 6. If False, set supporting_evidence to an empty string. Prompt template for QA correctness filtering of multi-hop questions. You are an expert fact-checker evaluating a multi-hop QA item that provides TWO separate context snippets (A and B). Your task: Decide whether the QUESTION and ANSWER are (i) supported by BOTH sections and (i) truly requires BOTH sections (neither A nor B alone is sufficient). Use ONLY the provided contexts. Follow these steps internally (do not include your reasoning in the output): 1) Read the QUESTION carefully. 2) Examine CONTEXT A and decide if it contains evidence supporting the ANSWER and relevant to the QUESTION. 3) Examine CONTEXT B and decide if it contains evidence supporting the ANSWER and relevant to the QUESTION. 4) Decide whether BOTH sections are needed to answer the QUESTION (i.e., one section alone is insufficient or incomplete). 5) If supported_by_A or supported_by_B is True, extract the minimal supporting sentence(s) from that section. 6) If any field would be False, leave its evidence string empty. Prompt template for QA correctness filtering of refusal questions. You are an expert fact-checker evaluating whether a refusal-style answer is appropriate. Your task: Determine whether the provided ANSWER (which indicates "no answer available") is correct based **only** on the information in the CONTEXT. Ignore any external knowledge. Follow these steps internally (do not include your reasoning in the output): 1. Read the QUESTION carefully. 2. Check if the CONTEXT contains any information that directly answers the QUESTION. 3. If the CONTEXT does not contain any answer, then the refusal ("not found") is correct (True). 4. If the CONTEXT actually contains an answer, then the refusal is incorrect (False). 5. If incorrect, extract the exact sentence(s) from the CONTEXT that show the answer. 6. If correct, leave supporting_evidence as an empty string. Prompt template for question relevance filtering of all questions. Classify the given QUESTION into one of the allowed FDA drug label topics or "None / Irrelevant" if it is not related to regulatory labeling content. Follow these steps internally (do not include your reasoning in the output): 1. Read the QUESTION carefully. 2. Determine if it concerns factual, regulatory-relevant information that appears in FDA drug labels. 3. If relevant, assign exactly one of the allowed categories. 4. If not relevant, assign "None / Irrelevant". 5. Output a structured JSON object only (no extra text). Allowed topics (choose exactly one): 1) Indications & Usage 2) Dosage & Administration 3) Dosage Forms & Strengths & Formulation 4) Contraindications 5) Safety & Serious Risks 6) Adverse Events 7) Drug Interactions 8) Use in Specific Populations 9) Pharmacokinetics 10) None / Irrelevant Prompt template for question quality filtering of factual questions. A BAD factual question has one or more of: - "answer_leakage": the question already states the key information it is asking for - "context_mismatch": the question mentions an entity, age range, dose, or condition not supported by the CONTEXT - "unanswerable_from_context": the requested information is simply not present in the CONTEXT - "low_utility": technically answerable but extremely vague or trivial as a benchmark item - "other": any other serious flaw that makes it unsuitable Prompt template for question quality filtering of multi-hop questions. A BAD multi-hop question has one or more of: - "one_sided_answerable": The answer can be derived entirely from Context A alone OR Context B alone, so the question is not truly multi-hop. - "hop_mismatch": The question forces an artificial, invalid, or nonsensical connection between Context A and B (e.g., linking unrelated concepts, doses, adverse events, or PK mechanisms across contexts). - "answer_leak_from_context": The question copies wording, entities, or details from one context in a way that incorrectly shapes the question (e.g., introduces unsupported conditions, imports irrelevant entities, or fabricates a scenario based on leaked text). - "unanswerable_from_context": The requested information does not appear in the contexts; the question cannot be answered from A+B. - "other": The question is trivial, vague, or provides negligible benchmarking value. C Experimental details C.1 Experimental prompts Inference prompts.Prompt template for closed- book setting of QA inference. You are assisting FDA reviewers by answering questions about a single drug label. General rules: - Answer as concisely as possible (1-3 sentences). - Do NOT invent facts that are not supported by the label. You do NOT have access to the drug label text. Use only your existing knowledge. QUESTION: question ANSWER: Prompt template for open full label setting of QA inference. You are assisting FDA reviewers by answering questions about a single drug label. General rules: - Answer as concisely as possible (1-3 sentences). - Do NOT invent facts that are not supported by the label. The label text below is divided into passages. Each passage is preceded by a marker of the form: ||PASSAGE_X|| where X is a zero-padded integer (the passage id). When answering: - Use ONLY information from the label text. - After your answer, you MUST list the passage ids that best support your answer. - If multiple passages are relevant, include all of them. - If the answer truly cannot be determined from the label, reply exactly with: NOT_ANSWERABLE - If the answer is NOT_ANSWERABLE, use CITED_PASSAGES: []. Output format (exactly): 1. First line: the answer in natural language (or NOT_ANSWERABLE). 2. Second line: CITED_PASSAGES: [PASSAGE_X, PASSAGE_Y, ...] LABEL TEXT: label_text QUESTION: question Prompt template for open oracle passages setting of QA inference. You are assisting FDA reviewers by answering questions about a single drug label. General rules: - Answer as concisely as possible (1-3 sentences). - Do NOT invent facts that are not supported by the label. You are given a set of passages extracted from the FDA-approved label for this drug. Use ONLY these passages to answer. PASSAGES: context_block QUESTION: question ANSWER: Evaluation prompts.Prompt template for evalu- ation of inference answer against gold standard. You are grading answers to questions about FDA-approved drug labels. You will see: - a QUESTION about a drug’s label, - a GOLD TARGET: the reference answer derived from the label, and - a PREDICTED ANSWER: the model’s response. Your job is to decide whether the PREDICTED ANSWER is: - CORRECT - INCORRECT - NOT_ATTEMPTED and then output a single letter: - A for CORRECT - B for INCORRECT - C for NOT_ATTEMPTED ## Core grading rules Treat the GOLD TARGET as the reference truth for what should be said. A PREDICTED ANSWER is CORRECT if: - It contains all clinically important information in the GOLD TARGET that is relevant to the QUESTION. - It does NOT contain any statements that contradict the GOLD TARGET. - It does NOT introduce specific clinical recommendations that are unsupported or clearly wrong. - Wording differences are fine (paraphrases, reordered points, different sentences) as long as the meaning matches. A PREDICTED ANSWER is INCORRECT if: - It contradicts the GOLD TARGET (different dose, frequency, indication, population, contraindication, etc.), OR - It introduces specific clinical facts (dose, schedule, indication, contraindication, population, lab threshold, etc.) that are not supported by the GOLD TARGET and would influence clinical use, OR - It omits one or more MAJOR clinically important elements required by the QUESTION (for example, missing a key dose adjustment, missing a required contraindication), OR - The GOLD TARGET is a refusal/non-answer (e.g. "Information not found in label") but the model still gives a specific clinical recommendation instead of refusing. A PREDICTED ANSWER is NOT_ATTEMPTED if: - It clearly does NOT provide the required information from the GOLD TARGET (e.g. "I don’t know", "I cannot answer from the label"), AND - It does NOT invent or contradict clinical facts in the GOLD TARGET. ## Domain-specific guidance 1. Doses, frequencies, durations, and numeric thresholds - If the GOLD TARGET gives a specific dose, frequency, duration, or lab threshold, the PREDICTED ANSWER must have the same key numbers to be CORRECT. - Small formatting changes (e.g., "5 mg once daily" vs. "once daily 5 mg") are fine. - If the predicted numbers differ in a way that would change dosing or eligibility, grade as INCORRECT. - Vague statements like "take as directed on the label" are usually NOT_ATTEMPTED unless the GOLD TARGET itself is vague. 2. Indications and populations - If the GOLD TARGET specifies indications or special populations (e.g., "patients with eGFR < 45 mL/min/1.73 m2", "pediatric patients 6-17 years"), leaving out a major constraint or population can make the answer INCORRECT. - Minor wording differences (e.g., "patients with moderate to severe renal impairment" when the GOLD TARGET explicitly defines that range) can still be CORRECT if they preserve the same meaning. 3. Contraindications and warnings - If the QUESTION asks about contraindications or major warnings, missing a key contraindication or warning from the GOLD TARGET should be graded as INCORRECT, not NOT_ATTEMPTED. - Adding a serious new contraindication or warning that is not in the GOLD TARGET is INCORRECT, even if it sounds medically plausible. ## Examples ### Example 1 (dose and population) Question: "What is the recommended saxagliptin dose for patients with eGFR < 45 mL/min/1.73 m2?" Gold target: "The recommended dosage of saxagliptin tablets is 2.5 mg orally once daily for patients with eGFR < 45 mL/min/1.73 m2, including those with moderate or severe renal impairment or ESRD." Predicted answer 1: "Give 2.5 mg saxagliptin once daily in patients with eGFR below 45. This includes patients with moderate or severe renal impairment and ESRD." - A: contains the key dose and population, no contradictions. Predicted answer 2: "Use the standard 5 mg once daily dose regardless of renal function." - B: contradicts the GOLD TARGET on dose and population. Predicted answer 3: "I’m not sure what dose to use in patients with reduced kidney function based on this label." - C: does not provide the needed information. ### Example 2 (contraindication) Question: "In which patients are potassium citrate tablets contraindicated?" Gold target: "Potassium citrate extended-release tablets are contraindicated in patients with hyperkalemia or conditions predisposing them to hyperkalemia, patients with GI obstruction or delayed gastric emptying, patients with peptic ulcer disease, patients with active urinary tract infection with certain stones, and patients with renal insufficiency." Predicted answer 1: "They are contraindicated in patients with hyperkalemia or at risk of hyperkalemia, GI obstruction or delayed gastric emptying, peptic ulcer disease, active urinary tract infection with certain stones, and renal insufficiency." - A: contains the correct information. Predicted answer 2: "They are contraindicated only in patients with a history of allergies to potassium." - B: omits almost all key contraindications and adds an unsupported one. Predicted answer 3: "The label does not clearly specify in which patients they are contraindicated." - C: fails to use the GOLD TARGET, but does not contradict it ## Final instruction Now you will grade a new example. You will be given: Question: question Gold target: target Predicted answer: predicted_answer Grade the predicted answer of this new question as one of: A: CORRECT B: INCORRECT C: NOT_ATTEMPTED Respond in the following format, on a single line: LETTER: short reason Where LETTER is exactly one of A, B, or C, and "short reason" is 1-2 sentences explaining your choice. Do not include any other text. D AI Use Declaration During the preparation of this manuscript, the au- thors used ChatGPT for assistance purely with the editing of the paper.