Paper deep dive
Grounded in Consensus, In Step With Emerging Science: A Consensus-Anchored Multi-Corpus Clinical Chatbot for Long COVID
Yining Wu, Philip DiGiacomo, Ying Ding, William Brode
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/1/2026, 10:59:09 AM
Summary
The paper presents a clinician-facing retrieval-augmented generation chatbot for Long COVID that integrates four distinct evidence sources: expert-curated consensus guidance, PubMed literature, ClinicalTrials.gov records, and living systematic reviews. The system uses consensus guidance as a foundational anchor while retrieving other sources in parallel. Evaluation against OpenEvidence on 50 questions showed comparable or slightly better performance in factual accuracy, completeness, and clinical soundness, with lower score variability.
Entities (9)
Relation Signals (8)
Chatbot System → usessource → PubMed
confidence 98% · The PubMed component retrieves current literature through NCBI E-utilities.
Chatbot System → usessource → ClinicalTrials.gov
confidence 98% · The clinical-trials component indexes a curated registry corpus derived from ClinicalTrials.gov
Long COVID → istargetof → Chatbot System
confidence 95% · We present a clinician-facing chatbot that organizes four sources... for Long COVID
Chatbot System → usessource → Consensus Guidance
confidence 95% · Expert-curated consensus guidance is always included as the foundational corpus
Chatbot System → usessource → Living Systematic Review
confidence 95% · The Living SR component indexes 74 randomized controlled trial publications included in living systematic reviews
Chatbot System → isevaluatedagainst → OpenEvidence
confidence 92% · We conducted an exploratory LLM-as-a-judge comparison of our chatbot and OpenEvidence
Chatbot System → usesmodel → Claude Opus 4.8
confidence 90% · Retrieved evidence is integrated by a single synthesis model (Claude Opus 4.8)
Evaluation → usesmodel → GPT-4o
confidence 90% · GPT-4o independently scored each response
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Long COVID (LC) poses a challenge for clinical decision support because relevant evidence is distributed across sources with different update cycles, evidentiary roles, and levels of clinical maturity. We present a clinician-facing chatbot that organizes four sources within a retrieval-augmented workflow: expert-curated consensus guidance, current PubMed literature, registered interventional trials, and evidence from living systematic reviews. Consensus guidance is always included to frame responses, while the remaining sources are retrieved in parallel when selected by the user. In an exploratory automated evaluation on 50 clinician-facing questions, our chatbot showed comparable mean ratings to OpenEvidence, with numerically higher scores and lower score variability in an LLM-judged comparison.
Tags
Links
- Source: https://arxiv.org/abs/2607.25038v1
- Canonical: https://arxiv.org/abs/2607.25038v1
Trouble viewing inline? Open PDF directly →
Full Text
17,829 characters extracted from source content.
Expand or collapse full text
Published as a conference paper at COLM 2026 Grounded in Consensus, In Step With Emerging Science: A Consensus-Anchored Multi-Corpus Clinical Chatbot for Long COVID Yining Wu School of Information University of Texas at Austin Austin, TX, USA yining.wu@utexas.edu Philip DiGiacomo Department of Computer Science University of Texas at Austin Austin, TX, USA pdigiacomo@utexas.edu Ying Ding School of Information, Dell Medical School University of Texas at Austin Austin, TX, USA ying.ding@ischool.utexas.edu William Brode Dell Medical School University of Texas at Austin Austin, TX, USA William.Brode@austin.utexas.edu Abstract Long COVID (LC) poses a challenge for clinical decision support because relevant evidence is distributed across sources with different update cycles, evidentiary roles, and levels of clinical maturity. We present a clinician- facing chatbot that organizes four sources within a retrieval-augmented workflow: expert-curated consensus guidance, current PubMed literature, registered interventional trials, and evidence from living systematic reviews. Consensus guidance is always included to frame responses, while the re- maining sources are retrieved in parallel when selected by the user. In an exploratory automated evaluation on 50 clinician-facing questions, our chat- bot showed comparable mean ratings to OpenEvidence, with numerically higher scores and lower score variability in an LLM-judged comparison. 1 Introduction Clinical decision-support systems for emerging diseases must do more than retrieve relevant information. Long COVID (LC) is a useful stress test because it combines high evidence volume with limited clinical certainty: heterogeneous multisystem symptoms, no reliable biomarker, no proven disease-modifying therapy, and rapidly changing literature. In de- ployment, such systems must determine which evidence sources should inform a response, how sources with different authority and update cycles should be combined, and how clinicians can inspect the provenance of resulting claims. LC clinical interpretation therefore draws on consensus guidance, emerging primary studies, registered trials, and continually updated evidence syntheses National Academies of Sciences, Engineering, and Medicine et al. (2024); Zeraatkar et al. (2024). Prior LC question-answering work found that broad literature retrieval can produce clin- ically misaligned outputs, while a curated corpus of clinical guidance and high-quality reviews produced more appropriate responses than PubMed-scale retrieval alone DiGia- como et al. (2025). That result motivates a deployment problem: a static curated corpus can anchor responses, but cannot capture published findings, trial-status changes, or living- review updates. Nor should these sources simply be pooled, because consensus guidance, primary literature, trial registries, and systematic reviews serve distinct clinical roles. 1 arXiv:2607.25038v1 [cs.IR] 27 Jul 2026 Published as a conference paper at COLM 2026 Figure 1: Four evidence sources (Consensus, PubMed, Clinical trials, and Living SR) are retrieved in parallel and integrated through a synthesis module into a clinical response. We developed a clinician-facing chatbot that organizes these sources within a single retrieval- augmented workflow. Expert-curated consensus guidance is always included to frame responses, while PubMed literature, ClinicalTrials.gov records, and living systematic-review evidence are retrieved in parallel when selected by the user. This deployment case report describes the system and the design choices required to operationalize multi-source evidence synthesis in a clinical workflow. We focus on source-specific retrieval, consensus-anchored synthesis, provenance presentation, and reliability under real query load. 2 Methods 2.1 System overview We developed a clinician-facing retrieval-augmented generation system that organizes four complementary evidence corpora into a single clinical workflow (Figure 1): expert consensus guidance (Consensus), PubMed literature (PubMed), ClinicalTrials.gov clinical trial records (Clinical trials), and living systematic review evidence (Living SR). Expert consensus guidance is always included as the foundational corpus, whereas PubMed literature, clinical trials, and Living SR evidence are user-selectable and enabled by default. Selected sources are retrieved in parallel and passed to a single Curated synthesis model, which produces a citation-traceable clinical response. The system is deployed as a clinician- facing web application with access intentionally limited to authorized users during the current deployment phase to meet security and operational requirements. 2.2 Evidence corpora and retrieval The consensus corpus contains eight expert-curated LC guidance and consensus documents. Documents were processed into semantic chunks and indexed using text-embedding-3-large; relevant passages were retrieved by vector similarity for each query. Three additional corpora provide complementary evidence. The PubMed component re- trieves current literature through NCBI E-utilities. An LLM-assisted query module identifies the clinical intent and relevant study characteristics, constructs a Boolean query with LC synonym constraints, and applies publication-type and recency filters before returning structured article findings. The clinical-trials component indexes a curated registry corpus derived from ClinicalTri- als.gov (currently 887 records). It uses query-derived clinical entities to rank relevant records and returns a shortlist with trial status, design characteristics, and available results sum- maries. This component is intended to characterize the LC trial landscape, including what interventions are being studied and the status of ongoing work. 2 Published as a conference paper at COLM 2026 Figure 2: User-facing presentation of a curated synthesis response. (a) Generated responses begin with a section of Clinical Bottom Line and include inline citations. (b) A source-labeled reference list preserves provenance across evidence corpora. (c) Relevant ClinicalTrials.gov records are displayed as trial cards. The Living SR component indexes 74 randomized controlled trial publications included in living systematic reviews across eight intervention categories. Retrieved studies are linked to intervention–outcome-specific GRADE certainty assessments (Group, 2004), allowing the system to present trial-level findings and the certainty of the evidence. Detailed corpus-construction procedures and source-specific retrieval configurations are provided in Appendix A. 2.3 Synthesis, response processing, and reliability Retrieved evidence is integrated by a single synthesis model (Claude Opus 4.8), which frames responses around the consensus corpus while incorporating findings from selected evidence streams where relevant. Responses begin with a Clinical Bottom Line and include inline source markers. An application-layer post-processing step converts these markers into numbered citations, de-duplicates overlapping PubMed and Living SR records, and renders a source-labeled reference list and evidence cards in the user interface (Figure 2). Retrieval arms operate independently so that a timeout or failure in one source does not block the overall response; such failures are represented in the response metadata and interface. User-selected PubMed filters are enforced as hard constraints. When no records meet the selected criteria, the system reports an empty result rather than automatically relaxing the filters. The default workflow was designed for interactive clinical use, with a target latency below 45 seconds and a median latency of 29 seconds in internal testing. 3 Evaluation We conducted an exploratory LLM-as-a-judge comparison of our chatbot and OpenEvidence on 50 LC-related questions selected by 6 domain experts. OpenEvidence was selected as a clinician-facing, evidence-based AI platform that provides cited point-of-care responses and has been evaluated in primary care settings Hurt et al. (2025). GPT-4o independently scored each response on Factual Accuracy, Completeness and Thoroughness, and Clinical 3 Published as a conference paper at COLM 2026 METRICOur chatbot OpenEvidence Factual Accuracy4.32 ± 0.684.30 ± 1.05 Completeness & Thoroughness4.22 ± 0.514.18 ± 0.75 Clinical Soundness4.38 ± 0.574.34 ± 1.02 Mean across dimensions4.31 ± 0.554.27 ± 0.91 Questions with at least one score of 10 (0%)3 (6%) Table 1: GPT-4o evaluation of Long COVID responses from our chatbot and OpenEvidence. Values are mean ± SD across 50 questions. Higher scores indicate better performance. Soundness using evaluation dimensions adapted from prior work Li et al. (2026). Scores were assigned on a 1–5 Likert scale, with higher values indicating better performance. Mean ratings were numerically higher and score variability lower for our chatbot across all three dimensions (Table 1). In addition, no response from our chatbot received the lowest possible rating on any dimension, whereas three OpenEvidence responses received at least one lowest-rating score. Overall, the evaluation indicated comparable automated response quality, with less variable scores for our chatbot. 4 Discussion Deploying a clinical support chatbot for LC requires balancing evidence breadth, uncertainty communication, response usability, and operational constraints. In this setting, the failure mode is not only missing evidence; it is also presenting early or low-certainty findings with more confidence than clinical consensus supports. Broader and more recent retrieval can improve coverage, but responses must remain interpretable within an interactive workflow; richer multi-stage processing can add context, but also increases latency and system com- plexity. These trade-offs shape whether a system functions as a usable clinical tool rather than a general-purpose search interface. Our central design choice was to preserve, rather than pool, the clinical roles of evidence sources: consensus guidance frames interpretation, PubMed captures emerging findings, trial registries track ongoing investigation, and living reviews contextualize trial findings by certainty. The interface makes these distinctions inspectable through source selection, explicit empty-result and failure reporting, a cited Clinical Bottom Line, and source-specific evidence cards. Together, these choices position evidence orchestration, provenance, pre- sentation, and latency as co-equal requirements for clinical AI in rapidly evolving evidence settings. 5 Limitations and Future Work This report describes an LC chatbot deployment and exploratory automated evaluation on a limited set of clinician-facing questions. The LLM-judge evaluation did not assess patient outcomes, clinician behavior, real-time use, or the contribution of individual system components; it was not powered to detect small score differences. Results should therefore be interpreted as feasibility signals rather than evidence of superiority. We are conducting a human expert evaluation in which six Long COVID experts develop reference answers and review outputs from our chatbot, OpenEvidence, and frontier models. This review will assess clinical quality and help calibrate automated monitoring as models, prompts, and evidence evolve. 6 Conclusion This deployment case report describes a clinician-facing LC chatbot that organizes foun- dational expert guidance, recent literature, registered trials, and living systematic-review 4 Published as a conference paper at COLM 2026 evidence within a source-traceable workflow. Rather than treating retrieval as a single- corpus problem, the system preserves the complementary clinical roles of these evidence streams while balancing response depth, interaction speed, and provenance presentation. Our experience illustrates how evidence orchestration can support deployable clinical AI workflows in domains with rapidly evolving evidence. References Ziyad Al-Aly, Hannah Davis, Lisa McCorkell, Let ́ ıcia Soares, Sarah Wulf-Hanson, Akiko Iwasaki, and Eric J Topol. Long covid science, research and policy. Nature medicine, 30(8): 2148–2164, 2024. Lucinda Bateman, Alison C Bested, Hector F Bonilla, Bela V Chheda, Lily Chu, Jennifer M Curtin, Tania T Dempsey, Mary E Dimmock, Theresa G Dowell, Donna Felsenstein, et al. Myalgic encephalomyelitis/chronic fatigue syndrome: essentials of diagnosis and management. In Mayo clinic proceedings, volume 96, p. 2861–2878. Elsevier, 2021. Abby L Cheng, Eric Herman, Benjamin Abramoff, Jordan R Anderson, Alba Azola, John M Baratta, Matthew N Bartels, Ratna Bhavaraju-Sanka, Svetlana Blitshteyn, Jeffrey S Fine, et al. Multidisciplinary collaborative guidance on the assessment and treatment of patients with long covid: A compendium statement. PM&R, 17(6):684–708, 2025. Philip DiGiacomo, Haoyang Wang, Jinrui Fang, Yan Leng, W Michael Brode, and Ying Ding. Demo: Guide-rag: Evidence-driven corpus curation for retrieval-augmented generation in long covid, 2025. URL https://arxiv.org/abs/2510.15782. GRADE Working Group. Grading quality of evidence and strength of recommendations. Bmj, 328(7454):1490, 2004. Ryan T Hurt, Christopher R Stephenson, Elizabeth A Gilman, Christopher A Aakre, Ivana T Croghan, Manpreet S Mundi, Karthik Ghosh, and Jithinraj Edakkanambeth Varayil. The use of an artificial intelligence platform openevidence to augment clinical decision- making for primary care physicians. Journal of Primary Care & Community Health, 16: 21501319251332215, 2025. Dongchen Li, Jitao Liang, Wei Li, Xiaoyu Wang, Longbing Cao, and Kun Yu. Clicare: Grounding large language models in clinical guidelines for decision support over longitu- dinal cancer electronic health records. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, p. 31554–31562, 2026. Michael R Mueller, Ravindra Ganesh, Thomas J Beckman, and Ryan T Hurt. Long covid: emerging pathophysiological mechanisms. Minerva Medica, 116(2):156–165, 2025. National Academies of Sciences, Engineering, and Medicine et al. A Long COVID Definition: A Chronic, Systemic Disease State with Profound Consequences. 2024. Michael J Peluso and Steven G Deeks. Mechanisms of long covid and the path toward therapeutics. Cell, 187(20):5500–5529, 2024. Julia Moore Vogel, Beth Pollack, Ezra Spier, Lisa McCorkell, Toni Wall Jaudon, Megan Fitzgerald, Hannah Davis, and Alison K Cohen. Designing and optimizing clinical trials for long covid. Life Sciences, 355:122970, 2024. Dena Zeraatkar, Michael Ling, Sarah Kirsh, Tanvir Jassal, Mahnoor Shahab, Hamed Mova- hed, Jhalok Ronjan Talukdar, Alicia Walch, Samantha Chakraborty, Tari Turner, et al. Interventions for the management of long covid (post-covid condition): living systematic review. Bmj, 387, 2024. 5 Published as a conference paper at COLM 2026 Table 2: Source-specific query-time retrieval settings. SourceCorpusQuery-time retrievalEvidence passed to syn- thesis Consensus8 documents; 395 chunks FAISSL2-distancesearch overtext-embedding-3-large embeddings Top 25 chunks PubMedLive APILLM-assisted Boolean query construction, publication filter- ing, and relevance/quality as- sessment Up to 10 articles Clinical trials887struc- tured registry records LLM-assisted query expansion followed by entity–field rank- ing over trial metadata; no vec- tor search Top 15 records Living SR74 structured RCT profiles LLM-assisted query expansion followed by entity–GRADE matching; no vector search Top 6, 10, or 14 papers, based on query breadth A Appendix A.1 Corpus construction The consensus corpus comprised the following eight expert-curated Long COVID guidance, consensus, and evidence-synthesis documents: Al-Aly et al. (2024); Bateman et al. (2021); Cheng et al. (2025); Mueller et al. (2025); National Academies of Sciences, Engineering, and Medicine et al. (2024); Peluso & Deeks (2024); Vogel et al. (2024); Zeraatkar et al. (2024). We also constructed a ClinicalTrials.gov registry of studies for which people with Long COVID could plausibly enroll or whose results could directly inform Long COVID care. Candidate records were identified through condition-field searches for Long COVID/PASC synonyms and eligibility-text searches for related post-infectious phenotypes, including POTS/dysautonomia, ME/CFS, mast-cell activation, and post-viral cognitive symptoms. Records were retained when they met both a population criterion (Long COVID, prior COVID-19, or a relevant post-infectious phenotype) and a clinical-relevance criterion. Core Long COVID records were retained automatically; other candidates underwent LLM- assisted screening, with uncertain cases reviewed by a clinician. The June 2026 corpus snapshot contained 887 records. The Living SR corpus was constructed from a monthly PubMed/MEDLINE search com- bining Long COVID/PASC, randomized-trial, and treatment terms. Eligible records were peer-reviewed randomized controlled trial publications of therapeutic interventions in adults with post-COVID-19 condition. We excluded acute-COVID or prevention trials, anosmia-only studies, nonrandomized or protocol-only reports, and studies with fewer than 25 participants per arm. Candidate records underwent AI-assisted abstract and full- text screening with clinician adjudication. The final corpus contains 74 RCT publications, each represented by structured intervention–outcome profiles linked to GRADE certainty assessments across eight intervention categories. A.2 Query-time retrieval Table 2 summarizes the default integrated configuration. Only the consensus corpus uses dense vector retrieval; the remaining sources use query-conditioned structured retrieval. Selected retrieval arms run independently and are combined in a single synthesis call. 6