Paper deep dive

A Decade-Scale Benchmark Evaluating LLMs' Clinical Practice Guidelines Detection and Adherence in Multi-turn Conversations

Andong Tan, Shuyu Dai, Jinglu Wang, Fengtao Zhou, Yan Lu, Xi Wang, Yingcong Chen, Can Yang, Shujie Liu, Hao Chen

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 138

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%

Last extracted: 3/27/2026, 1:33:55 AM

Summary

CPGBench is a decade-scale automated benchmarking framework designed to evaluate the clinical guideline detection and adherence capabilities of Large Language Models (LLMs) in multi-turn conversations. The study analyzes 3,418 clinical practice guidelines (CPGs) across 24 specialties and 11 regions, extracting 32,155 recommendations. Results reveal a significant gap between LLMs' ability to detect guideline content (71.1%-89.6%) and their ability to correctly reference titles (3.6%-29.7%) or adhere to recommendations (21.8%-63.2%), highlighting critical safety concerns for real-world clinical deployment.

Entities (5)

Clinical Practice Guidelines · medical-knowledge-source · 100%CPGBench · benchmark-framework · 100%American Board of Medical Specialties · organization · 95%DeepSeek R1 · large-language-model · 95%GPT5 · large-language-model · 95%

Relation Signals (3)

CPGBench → evaluates → Large Language Models

confidence 100% · CPGBench, the first decade-scale benchmark evaluating LLMs’ detection and adherence to CPGs

Large Language Models → adhereto → Clinical Practice Guidelines

confidence 90% · The adherence rates range from 21.8% to 63.2% in different models

GPT5 → performsbestin → Clinical Practice Guidelines Detection

confidence 90% · GPT5 ranks the best among the evaluated models

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Clinical practice guidelines (CPGs) play a pivotal role in ensuring evidence-based decision-making and improving patient outcomes. While Large Language Models (LLMs) are increasingly deployed in healthcare scenarios, it is unclear to which extend LLMs could identify and adhere to CPGs during conversations. To address this gap, we introduce CPGBench, an automated framework benchmarking the clinical guideline detection and adherence capabilities of LLMs in multi-turn conversations. We collect 3,418 CPG documents from 9 countries/regions and 2 international organizations published in the last decade spanning across 24 specialties. From these documents, we extract 32,155 clinical recommendations with corresponding publication institute, date, country, specialty, recommendation strength, evidence level, etc. One multi-turn conversation is generated for each recommendation accordingly to evaluate the detection and adherence capabilities of 8 leading LLMs. We find that the 71.1%-89.6% recommendations can be correctly detected, while only 3.6%-29.7% corresponding titles can be correctly referenced, revealing the gap between knowing the guideline contents and where they come from. The adherence rates range from 21.8% to 63.2% in different models, indicating a large gap between knowing the guidelines and being able to apply them. To confirm the validity of our automatic analysis, we further conduct a comprehensive human evaluation involving 56 clinicians from different specialties. To our knowledge, CPGBench is the first benchmark systematically revealing which clinical recommendations LLMs fail to detect or adhere to during conversations. Given that each clinical recommendation may affect a large population and that clinical applications are inherently safety critical, addressing these gaps is crucial for the safe and responsible deployment of LLMs in real world clinical practice.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

137,375 characters extracted from source content.

Expand or collapse full text

A Decade-Scale Benchmark Evaluating LLMsClinical Practice Guidelines Detection and Adherence in Multi-turn Conversations Andong Tan 1,2,* , Shuyu Dai 1,3,* , Jinglu Wang 1 , Fengtao Zhou 2 , Yan Lu 1 , Xi Wang 2 , Yingcong Chen 8 , Can Yang 9 , Shujie Liu 1,† , and Hao Chen 2,4,5,6,7,† 1 Media Computing Group, Microsoft Research Asia, Beijing, China 2 Department of Computer Science and Engineering, Hong Kong University of Science and Technology, Hong Kong, China 3 School of Computer Science, Peking University, Beijing, China 4 Department of Chemical and Biological Engineering, Hong Kong University of Science and Technology, Hong Kong, China 5 Division of Life Science, Hong Kong University of Science and Technology, Hong Kong, China 6 HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute, Futian, Shenzhen, China. 7 State Key Laboratory of Nervous System Disorders, Hong Kong University of Science and Technology, Hong Kong, China 8 AI Thrust, Information Hub, Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China 9 Department of Mathematics, Hong Kong University of Science and Technology, Hong Kong, China † Corresponding authors: Shujie Liu (shujliu@microsoft.com), Hao Chen (jhc@ust.hk) ABSTRACT Clinical practice guidelines (CPGs) play a pivotal role in ensuring evidence-based decision-making and improving patient outcomes. While Large Language Models (LLMs) are increasingly deployed in healthcare scenarios, it is unclear to which extend LLMs could correctly identify and adhere to CPGs during conversations. To address this gap, we introduce CPGBench, an automated framework benchmarking the clinical guideline detection and adherence capabilities of LLMs in multi-turn conversations at scale via the LLM (Judge-LLM) based scoring. We collect 3,418 CPG documents from 9 countries/regions (USA, Canada, UK, Germany, Australia, Japan, Chinese Mainland, Hong Kong, Taiwan) and 2 international organizations (World Health Organization and the European Society of Neurogastroenterology and Motility) published in the past 10 years (2015-2025.8) spanning across all 24 specialties defined in American Board of Medical Specialties. From these documents, we extract 32,155 clinical recommendations with corresponding publication institute, date, country, specialty, recommendation strength, evidence level, context, recommended action and goal of the recommendation. One multi-turn conversation is generated for each recommendation accordingly to evaluate the detection and adherence capabilities of 8 leading LLMs released between April 2024 and August 2025. Based on our proposed automatic evaluation pipeline, we find that the71.1%−89.6%recommendations can be correctly detected in conversations, while only3.6%−29.7%corresponding titles can be correctly referenced, revealing the gap between knowing the guideline content and knowing where they come from. The adherence rates range from21.8%to63.2%in different models, which are much lower than the detection rates, indicating a large capability gap between just knowing the guidelines and being able to apply them in multi-turn conversations. To confirm the validity of the automatic analysis, we further conduct a comprehensive human evaluation involving 56 clinicians from different medical specialties. The results indicate that near 100% of the extracted information is correct, over 99% generated conversations are proper for detection and adherence evaluations and the Judge-LLM exhibits a substantial agreement with the clinicians in the automatic scoring (cohen’s kappa 0.62-0.84 in different tasks). To our knowledge, CPGBench is the first benchmark systematically revealing which clinical recommendations LLMs fail to detect or adhere to during multi-turn clinical conversations across different countries/regions and healthcare systems at scale. Given that each high‑quality clinical recommendation may affect a large population and that clinical applications are inherently safety critical, our work represents an important step toward the safe and responsible deployment of LLMs in real world clinical practice. ∗ Work done during internship at Microsoft Research Asia. 1 arXiv:2603.25196v1 [cs.CL] 26 Mar 2026 1Introduction Large language models have demonstrated strong performance across a broad range of tasks and are increasingly being explored for medical applications, including clinical decision support, documentation, and patient-facing com- munication 1–5 . Nevertheless, their deployment in high-stakes healthcare settings remains constrained by persistent robustness and reliability concerns. In particular, LLMs may produce hallucinated or unsupported statements, raising the risk of clinically inappropriate recommendations and undermining trustworthiness when models are expected to align with established clinical knowledge and evidences 6 . As a central component of established clinical knowledge, clinical practice guidelines (CPGs) provide the au- thoritative, evidence‑based standards that models are expected to adhere to. CPGs are systematically developed by expert panels to synthesize available evidence, weigh benefits and harms of clinical actions, and issue recommenda- tions intended to standardize care and improve patient outcomes 7 . High-quality guidelines are typically produced under rigorous methodological frameworks such as GRADE 8 , which explicitly specify recommendation strength and evidence certainty, alongside supporting evidence from the literature 1 . Therefore, evaluating the extent to which existing large language models (LLMs) adhere to CPGs issued by authoritative institutions is crucial for earning the trust of both clinicians and patients. Moreover, this evaluation holds importance beyond trust-building: regulatory and policy bodies worldwide have repeatedly emphasized the need for governance frameworks for medical AI, reflect- ing growing recognition that unsafe or invalidated model behaviors can have direct clinical and societal consequences, such as the World Health Organization 9 , the US Food and Drug Administration 10,11 and German association for digital healthcare 12 . In this context, quantifying the extent to which LLMs reliably comply with trustworthy CPG recommendations is a critical prerequisite for informing evidence-based regulation and for characterizing readiness for real-world deployment. However, existing evaluations of LLM capabilities related to guidelines are narrow in scope and do not suﬀiciently reflect real-world clinical practice. For example, AMEGA 13 uses 135 questions derived from 20 medical cases; PromptGuide 14 uses 34 multiple choices questions asking about the strengths of clinical recommendations related the osteoarthritis; ReliaChatGPT 15 uses 25 questions about 5 recommendations related to hepatopancreaticobiliary; MedGuide 16 generates 7,747 questions about 17 cancer types according to the guidelines published in the U.S. National Comprehensive Cancer Network (NCCN); SurgicalGuide 17 leverages 10 true-false questions based on cases; NICE-RAG 18 uses 70 curated questions about the guidelines published by the U.K. National Institute of Care Excellence (NICE). Therefore, prior studies are largely confined to case studies 6 or small-scale question-answering datasets 13–16 , typically covering only a small subset of guideline documents (e.g., limited documents published in the U.K. or U.S.) and do not evaluate relevant capabilities in more realistic multi-turn conversations. Moreover, manual evaluation on the open-ended responses of LLMs during conversations is prohibitively costly: consider the large number of released large language models every year as well as the newly published or updated guidelines every year, it is not feasible to expect every model to go through a thorough expert evaluation on the adherence capability to every CPG’s recommendation exhaustively. To address these gaps, we introduce CPGBench, an automated, large-scale benchmarking framework that can be used to evaluate LLMscapabilities to adhere to CPGs in multi-turn conversations. Our primary objective is to evaluate whether LLMs can adhere to CPGs in realistic conversational settings, as adherence most directly reflects their potential clinical safety and reliability. However, meaningful adherence presupposes that the model actually possesses the underlying guideline knowledge. To assess this prerequisite, we introduce a complementary guideline detection task that measures whether models can recognize and recall the relevant guideline content given conversations involving guidelines. Beyond mere knowledge presence, the detection task further evaluates title grounding, which requires models to provide the specific guideline titles corresponding to the detected content, thereby assessing whether they can accurately ground their responses in the appropriate sources. Together, these two primary tasks form a coherent framework for characterizing the limits of current LLMsability to engage with evidence‑based guideline knowledge. An overview of the benchmark design is presented in Fig.1(c). Our main contributions are summarized as follows: •We introduceCPGBench, the first decade-scale benchmark evaluating LLMs’ detection and ad- herence to CPGs in multi-turn conversations. CPGBench incorporates 3,418 publicly available CPGs published across nine regions and 24 clinical specialties, from which we extract 32,155 recommendation-level entries and generate corresponding multi-turn conversations for more realistic evaluations. 1 While related expert-authored documents (e.g., consensus statements or position papers) are sometimes colloquially referred to as “guidelines”, they often reflect lower levels of evidentiary rigor than formally developed CPGs. Accordingly, we restrict the scope of this work to bona fide CPG documents, for which adherence is particularly consequential. 2/33 c. Benchmark Design Overview a. Distribution of the Collected Guidelines across 9 Regions e. Overall Results of 8 Evaluated Models b. Coverageacross 24 Specialties d. Recommendation Distribution across 10 years Tested LLM’s response LLM-based synthetic conversation generation Clinical practice guidelines Task 1: Guideline detection Task 2: Guideline adherence Tested LLM’s detection result Content detection score Automated scoring (0 -1) Remove ground-truth tags/responses Adherence score Title grounding score Figure 1.Overview of CPGBench for evaluating LLM’s detection and adherence to CPGs.aGeographical distribution of our collected clinical practice guidelines across the world.bOur benchmark covers all 24 medical specialties defined in American Board on Medical Specialties.cWe leverage the original guideline information to synthesize conversations and convert these conversations to proper forms for the detection and adherence capability evaluations.dRecommendation distribution over 10 years across different regions.eAn overview of the evaluated models and their content detection, title grounding as well as adherence rates. 3/33 •Using CPGBench, webenchmark eight leading LLMs released between April 2024 and August 2025, yielding 514480 responses from the tested LLMs and 771720 detailed analysis outputs from the Judge-LLM regarding the tested LLMs’ responses. Our analysis reveals two critical gaps in current model performance. First, models can often detect relevant guidelines but struggle to provide the corresponding titles or references that substantiate those identifications. Second, there is a clear disconnect between the ability to detect guidelines and the capacity to adhere to them consistently in multi-turn conver- sations. These limitations underscore the need for further model improvements before such systems can be considered reliable for high‑stakes clinical applications. •A comprehensivehuman evaluation involving 56 clinicians from 24 medical specialtiesis conducted to evaluate the automated pipeline, justifying the validity of our scalable automated assessment framework. 2Results We evaluate a diverse set of models in our benchmark, spanning small scale general (Llama3-8b-inst 1 , Qwen3-4b- inst 19 ) and medical model (Huatuo-o1-7b 20 ), mid-scale general (Qwen3-32b 19 ) and medical (Baichuan-m2-32b 21 ) models as well as large scale proprietary (GPT4o 22 , GPT5 23 ) and open sourced models (DeepSeek-R1 2 ). All evaluated models are released between April 2024 and August 2025. We focus on recent models to more accurately reflect the current capabilities of leading LLMs, as more recent models typically benefit from improved architectures, larger‑scale training data, and more advanced training strategies. 2.1Overall Comparison Table 1.Overall content detection, title grounding and adherence rates in all 32,155 extracted clinical practice recommendations. Detection ModelsSize Content DetectionTitle Grounding AdherenceAverage GPT5unknown79.47%29.68%63.18%57.44% Deepseek-R1671B89.62%13.24%50.01%50.96% Qwen3-32B32B84.77%7.96%45.89%46.21% GPT4ounknown82.65%7.91%41.75%44.10% Baichuan-m2-32B32B71.13%4.80%49.32%41.75% Qwen3-4B-inst4B72.01%4.17%37.30%37.83% Huatuo-o1-7B7B77.31%3.55%29.50%36.79% Llama3-8B-inst8B79.38%5.33%21.77%35.49% Tab.1and Fig.1(e) summarize the detection, title‑grounding, and adherence rates across all clinical recom- mendations. It could be observed that GPT5 ranks the best among the evaluated models, confirming its leading position in healthcare applications. Second, larger models mostly outperform smaller scale models. Third, no model achieves 100% in any task, indicating the gap between existing models’ capabilities and their readiness to be deployed in safety-critical clinical scenarios. No single model leads across all metrics, indicating the capability imbalance in models. Content detection rates are high, likely because the pre-training data include a broad range of guideline content. By contrast, the adherence rates consistently lag behind content detection across models, highlighting the need to improve guideline application capabilities of LLMs in multi‑turn conversations. In the following sections, we report the benchmark statistics and stratified analyses by country/region/interna- tional organizations and medical specialty, together with the human‑evaluation results. 2.2Benchmark Statistics In this section, we first compare our CPGBench with related assessment works from the aspects of scope and evaluation forms and then describe the guideline distribution by publication year, country/region, and clinical specialty. Comparison with existing related work.As summarized in Tab.2, existing benchmarks 13–18 are limited by small- scale evaluations (e.g., few guideline documents, a single language or country, and narrow institutional and specialty coverage), lack comprehensive coverage of all recommendations within any single guideline document (e.g., evalu- ations may span several documents but do not exhaustively assess all clinical recommendations in any one of the document), and depend on expert-curated questions and expert judgment. Moreover, none of these benchmarks 4/33 Table 2.Comparison with existing related works. Names#GuidelinesLanguagesRegionsInstitutesSpecialtiesTasksForms AMEGA 13 N/AEnglishUSA NCCN, ACC, AHA 13 specialties 135 questions based on 20 clinical cases. Open-ended QA PromptGuide 14 N/AEnglishUSAAAOS 1 disease (osteoarthritis) 34 questions about evidence strengths of 34 recommendations Multiple-choice QA ReliaChatGPT 15 5EnglishUKNICE, EASL 1 (hepato- pancreatico-biliary) 25 questions: 5 recommendations on 5 HPB conditions Open-ended QA MedGuide 16 N/AEnglishUSANCCN17 cancer types 7747 questions on 55 decision trees Multiple-choice QA SurgicalGuide 17 1EnglishUSANASS1 specialty 10 questions on cases True-False QA NICE-RAG 18 300 in the retrieval database EnglishUKNICEN/A70 curated questionsOpen-ended QA CPGBench (ours) 3,418 clinical practice guideline documents published in the past 10 years English, German, Chinese USA, Canada, UK,Germany, Australia, Japan, Chinese Mainland, HK, Taiwan, International Org Institute/ societies/ associations across regions All 24 specialties defined in American Board on Medical Specialties 32,155 multi-turn conversations for detection and adhe -rence on 32,155 recommendations 1.Detection in multi-turn conversations. 2.Multi-turn conversation completions for adherence evaluation. evaluates the guideline adherence capabilities in multi-turn conversations. In our CPGBench, we assemble a broad and diverse collection of 6,115 guideline documents from oﬀicial websites of national health departments, hospitals, medical societies or associations, and guideline platforms, and then filter out lower-quality documents (e.g., consen- sus statements, position papers) to obtain 4,792 high-quality CPGs using LLM-based filtering (GPT4o). We further retain only documents published in or after 2015 to reflect up-to-date best practices, resulting in 3,418 CPGs. From these documents, we extract 32,155 clinical recommendations and generate one multi-turn conversation where the recommendation is properly applied for each one of them. To maintain a rigorously curated guideline corpus, we avoid directly collecting documents from PubMed 24 and instead rely on high-quality platforms (e.g., ECRI 25 ) and oﬀicial institutional websites. The criteria used by PubMed 24 to tag articles asguidelinesare not transparent, although some of the guidelines we download from these other sources may also be hosted in PubMed PMC 24 . Distribution according to the country and year.Fig.1(a) shows that, in our database, Canada contributes the largest share of clinical recommendations (22.3%), followed by the United Kingdom (21.7%), Germany (19.2%), U.S.A. (13.3%) and Australia (11.9%), while each remaining country/region/organization contributes less than (10%) of the recommendations. It is also interesting to note from Fig.1(d) that the number of published clinical recommendations generally increases from 2015 to 2022, but decreases thereafter until at least 2024, which may be related to disruptions in CPG developments during the COVID-19 pandemic. Recommendation distributions by specialties across countries.Different countries use distinct specialty categoriza- tions. For example, the American Board of Medical Specialties (ABMS) defines 24 specialties 26 , German Medical Associations recognize around 34 specialties 27 and the UK National Health Service lists 84 main specialties 28 . For convenience and consistency in our analysis, we adopt the 24-category scheme used by the American Board of Medical Specialties. Fig. 1(b) presents an overall distribution and Fig.2presents the detailed distributions of recommendations by specialties across countries/regions/international organizations. Pediatrics (19.4%), internal medicine (13.5%), and preventive medicine (10.1%) are the top three major specialties in our corpus, together accounting for about 42% of all collected recommendations. The distribution is more balanced in countries with a large number of recommendations (USA, UK, Germany) and less diverse in regions or countries with relatively few published recommendations (Japan, Taiwan, Hong Kong, Chinese Mainland). Australia is a notable exception: despite having many recommendations, most are concentrated in internal medicine. We also calculate the distribu- tions for two international organizations including World Health Organization (WHO) and the European Society of Neurogastroenterology and Motility (ESNM). Because WHO focuses primarily on global disease prevention, pre- ventive medicine accounts for 45% of its recommendations. ESNM issues recommendations exclusively in internal medicine and colon and rectal surgery. 2.3Detection Benchmark In this benchmark, we aim to assess when a clinical recommendation is present in a conversation, whether the tested LLM could identify the existence of the recommendation and correctly provide its corresponding guideline document’s title. The rate is calculated by the number of correctly identified recommendations/titles divided by the overall numbers belonging to certain country/region/international institute/medical specialty in different analysis. 5/33 19% 13% 10% 9% 7% 7% 5% 3% 3% 3% 3% 2% 2% 2% 2% 2% 2% 1% 1% 1% Overall 19% 12% 11% 8% 7% 7% 7% 6% 5% 4% 3% 3% 2% 2% 1% 1% 1% United States 26% 15% 9% 9% 8% 7% 4% 3% 3% 3% 3% 2% 2% 2% 1% 1% Canada 17% 16% 9% 8% 7% 5% 4% 4% 4% 3% 3% 2% 2% 2% 2% 2% 2% 1% 1% United Kingdom 15% 13% 10% 10% 8% 6% 5% 4% 4% 4% 3% 3% 2% 2% 2% 2% 2% 1% Germany 61% 15% 10% 3% 3% 2% 2% 2% Japan 47% 31% 9% 6% 5% Chinese Mainland 63% 25% 6% 4% 2% Hong Kong 61% 18% 9% 5% 5% 2% Taiwan 69% 9% 4% 4% 3% 2% 2% 2% 2% Australia 45% 23% 10% 8% 7% 3% 2% WHO 71% 26% 3% ESNM Pediatrics Internal Medicine Preventive Medicine Other Obstetrics and Gynecology Psychiatry and Neurology Urology Surgery Orthopaedic Surgery Allergy and Immunology Colon and Rectal Surgery Dermatology Otolaryngology-Head and Neck Surgery Family Medicine Emergency Medicine Anesthesiology Physical Medicine and Rehabilitation Medical Genetics and Genomics Thoracic Surgery Ophthalmology Radiology Neurological Surgery Nuclear Medicine Plastic Surgery Pathology Figure 2.Specialty statistics of each country/region/international organization in our collected database. Percentage numbers of specialties less than1%are not displayed for better readability. Recommendation detection rates across regions.Following the overall detection rates presented in Tab.1, this section offers a more detailed analysis stratified by different country/region/international organizations as well as different medical specialties. Fig.3(a) shows the detection rates of different LLMs categorized by the publica- tion countries/regions or international organizations. It can be seen that Qwen3-4B-Inst and Baichuan-m2-32B exhibit a relatively larger variation in detection rates across guidelines from different sources (e.g., Qwen3-4B-inst and Baichuan-m2-32B has a variance of 0.0058, 0.0045, respectively), while the variance of other models are be- tween 0.0011 (DeepSeek-R1) and 0.0033 (Huatuo-o1-7B). Regarding the medical models: Huatuo-o1-7B, tuned from Qwen2.5-7B, attains an even higher detection rate than the larger Baichuan-m2-32B, which is fine-tuned from Qwen2.5-32B, suggesting that Huatuo-o1-7B may have been trained on more comprehensive guideline-related data than Baichuan-m2-32B. Another interesting observation is GPT5 performs slightly worse than GPT4o in the detec- tion task, implying that this newer GPT version may have sacrificed guideline detection related capabilities while improving other aspects. Recommendation detection rates across specialties.We also group recommendations by medical specialty to examine whether LLMs exhibit substantial variation in guideline knowledge across different medical specialties. The 95% confidence interval is displayed in each bar of the Fig. 4(a). Overall, the variance of the detection rates across medical specialties is not large, with the varaince of 0.0033 for Qwen3-4B-inst, 0.0030 for huatuo-o1-7B, 0.0024 for DeepSeek-R1, 0.0015 for GPT4o, 0.0013 for Qwen3-32B, 0.0011 for Baichuan-m2-32B, 0.0008 for GPT5 and 0.0007 6/33 c. Adherence Rates across Regions / International Organizations b. Title Grounding Rates across Regions / International Organizations a.Content Detection Rates across Regions / International Organizations Figure 3. aThe content detection rates in all regions/international organizations of all models are above 50%.bTitle grounding rates in the detection task. GPT5 is leading in this sub-task in most regions except the Chinese Mainland.c The adherence rates of all models across all regions or international organizations are consistently lower then their corresponding content detection rates. All plots share the same legend. for Llama3-8B-inst, respectively. This result indicates that the evaluated models have a similar level of capability in identifying the guideline from different medical specialties when the guideline is mentioned in the conversation. Guideline document title grounding rates in detection.Beyond mere knowledge presence, the detection benchmark includes a sub-task further evaluating the title grounding, which requires models to provide the specific guideline titles corresponding to the detected contents, thereby assessing whether models can accurately ground their re- sponses in the appropriate sources. This is a more challenging sub-task evaluating the model’s internal knowledge because the exact titles are not directly present in the conversations. Beyond its relevance to the content detection sub-task, referencing accurate guideline titles is an intrinsically important capability of LLMs, as hallucinated or non‑existent references can undermine user trust. Notably, lack of trust is frequently cited as a key barrier to clinicianswillingness to adopt LLMs in real‑world practice 29,30 . Extending the overall title grounding rates reported in Tab.1, Fig.3(b) provides a more granular breakdown of the statistics by countries, regions and international organizations. Based on the results, we can find that the overall title grounding rates in all models are rather low. GPT5 is the best performing model, however, reaches the overall title grounding rates of only 29.68%. The other models have an even lower title grounding rates, such as DeepSeek-R1 (13.24%), Qwen3-32B (7.96%), GPT4o (7.91%), Baichuan-m2-32B (4.80%), Huatuo-o1-7B (3.54%), Llama-8B-inst (5.33%) and Qwen3-4B-inst (4.17%). Among different regions or international institutes, UK, USA 7/33 a. Content Detection Rates across Specialties #' ',!*'%! $$'! *(%("/ (%(′ !,%-*"!*/ %%!*"/' &&-'(%("/ +,!,*$+' /'!(%("/ ,(%*/'"(%("/ !-*(%("$%-*"!*/ -%!*! $$'! !*&,(%("/ -*"!*/ #(*$-*"!*/ ! $,*$+ ,#(%("/ &!*"!'/! $$'! *!.!',$.!! $$'! %+,$-*"!*/ *,#()! $-*"!*/ )#,#%&(%("/ '!+,#!+$(%("/ ! $%!'!,$+' !'( #/+$%! $$'!' !# +/#$,*/' !-*(%("/ &$%/! $$'! $(%("/ ,#!* % )# $ (#" )# #&' ('($$ !" #&' #' b. Adherence Rates across Specialties #' ',!*'%! $$'! *(%("/ (%(′ !,%-*"!*/ %%!*"/' &&-'(%("/ +,!,*$+' /'!(%("/ ,(%*/'"(%("/ !-*(%("$%-*"!*/ -%!*! $$'! !*&,(%("/ -*"!*/ #(*$-*"!*/ ! $,*$+ ,#(%("/ &!*"!'/! $$'! *!.!',$.!! $$'! %+,$-*"!*/ *,#()! $-*"!*/ )#,#%&(%("/ '!+,#!+$(%("/ ! $%!'!,$+' !'( #/+$%! $$'!' !# +/#$,*/' !-*(%("/ &$%/! $$'! $(%("/ ,#!* % )# $ (#" )# #&' ('($$ !" #&' #' "& ',!*'%! $$'! *(%("/ (%(′ !,%-*"!*/ %%!*"/' &&-'(%("/ +,!,*$+' /'!(%("/ ,(%*/'"(%("/ !-*(%("$%-*"!*/ -%!*! $$'! !*&,(%("/ -*"!*/ #(*$-*"!*/ ! $,*$+ ,#(%("/ &!*"!'/! $$'! *!.!',$.!! $$'! %+,$-*"!*/ *,#()! $-*"!*/ )#,#%&(%("/ '!+,#!+$(%("/ ! $%!'!,$+' !'( #/+$%! $$'!' !# +/#$,*/' !-*(%("/ &$%/! $$'! $(%("/ ,#!* $ (" # '"! (" "%& '&'## ! "%& "& c. Content Detection Rates in the Safety Critical Subset #' ',!*'%! $$'! *(%("/ (%(′ !,%-*"!*/ %%!*"/' &&-'(%("/ +,!,*$+' /'!(%("/ ,(%*/'"(%("/ !-*(%("$%-*"!*/ -%!*! $$'! !*&,(%("/ -*"!*/ #(*$-*"!*/ ! $,*$+ ,#(%("/ &!*"!'/! $$'! *!.!',$.!! $$'! %+,$-*"!*/ *,#()! $-*"!*/ )#,#%&(%("/ '!+,#!+$(%("/ ! $%!'!,$+' !'( #/+$%! $$'!' !# +/#$,*/' !-*(%("/ &$%/! $$'! $(%("/ ,#!* % )# $ (#" )# #&' ('($$ !" #&' #' d. Adherence Rates in the Safety Critical Subset Figure 4.Content detection and adherence rates across medical specialties. The average performance across models in each specialty is shown in the upper subplot. The average performance across specialties in each model is shown in the right subplot. The subplot layout is consistent across all plots.aThe content detection rates are generally high in all medical specialties.bThe adherence rates in different models are generally higher in specialties such as allergy and immunology, pediatrics, preventive medicine and family medicine comapred to other specialties.cThe content detection rates in the safety critical subset are higher than the rates of the full set.dAdherence rates in the safety critical subset are mostly lower than that of the full set shown inbacross specialties. and WHO are the three guideline sources where most models have a higher title grounding rates compared to other regions/institutes. GPT5 outperforms DeepSeek-R1, Qwen3-32B, Qwen3-4B-inst in most regions/institutes except the Chinese Mainland. Since DeepSeek and Qwen series models are developed by Chinese companies, this indicates that models developed in the Chinese Mainland may have better optimized the model regarding chinese guidelines compared to GPT-series models. Notably, GPT4o, Baichuan-m2-32B, Huatuo-o1-7B and Llama3-8B- inst achieve 0% title grounding rates in guidelines published by ESNM. Compared to WHO, this result suggests a significant model capability imbalance between guideline documents published by larger scale (e.g., WHO) and smaller scale (e.g., ESNM) international institutes. Overall, none of the model could achieve a title grounding rate higher than60%in any country/region/international organization, suggesting the unreliability of the provided guideline document titles of these models. These results are consistent with prior finding 31 that LLMs generally fail to generate reliable references. 2.4Adherence Benchmark To evaluate LLMsadherence capabilities in multi-turn conversations, we truncate the conversations from the detection benchmark at the first turn of response where the simulated clinician applies a CPG recommendation. Removing that turn and all subsequent contents aims to remove the “ground-truth” response involving the guideline recommendation, such that the truncated conversation could serve as the background conversation, where an application of guideline is expected in the next round of response from clinicians/tested LLMs. The truncated conversation is then provided as the prompt to the tested LLM. Whether the tested LLM adheres to the guideline 8/33 is evaluated via checking whether the tested model’s response given this background conversation is consistent with the guideline recommendation used to generate the original multi-turn conversation. Details of the benchmark construction process are presented in Section4and illustrated in Fig.7. Adherence rates are generally lower than the content detection rates in all LLMs.The adherence rates shown in Fig. 3(c) are generally lower than the detection rates shown in Fig.3(a), where the highest overall adherence rates among models (63.18% from GPT5) are lower than the lowest overall content detection rates among models (71.13% from Baichuan-m2-32B), confirming that recognizing guideline recommendations in conversations is easier for LLMs compared to correctly applying them. These findings highlight an urgent priority to enhance LLMsadherence to CPGs, given that the correct application of guideline knowledge in multi‑turn conversations is critical for the safe use by clinicians and patients in real-world scenarios. Models with better content detection capabilities do not necessarily have better adherence capabilities.Compare Fig. 3(a) and Fig.3(c), it could be seen that GPT5 has a lower detection rate compared to DeepSeek-R1, Qwen3-32B and GPT4o, while achieving the highest adherence rate among all models. This suggests that although GPT5 may not possess more guideline knowledge than these models, it is more effective at applying the knowledge it does have in practical conversations, thereby achieving higher adherence rates despite its comparatively limited guideline knowledge. A similar phenomenon can be observed in GPT4o: it has a higher detection rate than Baichuan-m2- 32B (82.65% versus 71.13%) but a lower adherence rate than Baichuan-m2-32B (41.75% versus 49.32%). These evaluation results point out the importance of simultaneously improving both the coverage of a model’s knowledge base and its ability to apply that knowledge in realistic scenarios before the deployment in clinical practice. Models have different adherence capabilities in different specialties.We conduct a chi-square test of independence across 24 specialties for each model and observe thatp<1×10 −8 for all models, indicating a highly significant association between specialties and model adherence rates. Fig.4(b) shows that in specialties such as the radiology and thoracic Surgery, the adherence rates of all models are below 55%, indicating the unreliability of models when users are interacting with LLMs discussing relevant topics. The specialty with the highest overall adherence rates is the preventive medicine, where all models achieve adherence rates above 32%. Besides, the relative adherence rates comparison between models are largely the same across different medical specialties. For example, GPT5 has a higher adherence rate than DeepSeek-R1 in all medical specialties. Difference between detection and adherence rates across specialties in each model.Fig.5show the difference between the detection and adherence rates. In all models, detection rates are higher than the adherence rates across all medical specialties, confirming our assumption that the detection rates may serve as an informative upper bound of the models’ capabilities in guideline adherence. Besides, large general model (e.g., GPT5) and relatively large medical model (e.g., Baichuan-m2-32B) have the lowest capability difference, while rest models have higher comparative capability differences in all medical specialties. Qualitative analysis of adherence to guidelines published after model’s release.Intuitively, it is expected that the model cannot adhere to the guideline recommendations published after the model’s release. However, our results show that the model could still exhibit adherence to certain recommendations. To better understand this phenomenon, we conduct a qualitative analysis of GPT4o (released on May 13, 2024) by focusing on its responses to guidelines published after this date. Manual inspection of 20 cases reveals several reasons for such “post‑release” adherence: (1) New released guidelines sometimes retain recommendations from earlier versions, thus the model could adhere to these recommendations in the new guidelines, despite only being exposed to the prior version. For example, the guideline titled “Prevention and Control of Seasonal Influenza with Vaccines: Recommendations of the Advisory Committee on Immunization Practices - United States, 2022-2023 Influenza Season” (published in August 26, 2022) includes the recommendation “Routine annual influenza vaccination is recommended for all persons aged≥6 months who do not have contraindications”. This recommendation also appears in subsequent 2024-2025 version, which was published on August 29, 2024 after the release date of GPT4o. (2) The recommended action does not always require new guideline knowledge. For example, the guideline titled “Maternal and child nutrition: nutrition and weight management in pregnancy, and nutrition in children up to 5 years” 32 recommends that “If a person has had bariatric surgery and is planning a pregnancy or is pregnant, advise them to contact their bariatric surgery unit for individualised, specialist advice about folic acid and other micronutrients”. Although this recommendation is published in 2025, similar guidance may be already present in earlier literature, such as the 2019 consensus paper “Pregnancy after bariatric surgery: Consensus recommendations for periconception, antenatal and postnatal care” 33 . As a result, the model’s training on pre-existing evidence and guidance may already enable the model to follow this advice during conversations, without being exposed to the recommendation oﬀicially published in the new guideline 9/33 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.08 0.27 0.30 0.33 0.14 0.32 0.42 0.54 Allergy and Immunology 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.17 0.44 0.44 0.44 0.29 0.40 0.48 0.57 Anesthesiology 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.15 0.40 0.42 0.43 0.25 0.40 0.52 0.58 Colon and Rectal Surgery 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.19 0.44 0.44 0.49 0.33 0.47 0.54 0.61 Dermatology 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.28 0.50 0.49 0.47 0.32 0.36 0.48 0.61 Emergency Medicine 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.09 0.31 0.25 0.29 0.08 0.15 0.33 0.49 Family Medicine 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.25 0.45 0.46 0.51 0.33 0.46 0.56 0.64 Internal Medicine 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.10 0.42 0.44 0.42 0.18 0.44 0.55 0.59 Medical Genetics and Genomics 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.27 0.54 0.53 0.54 0.28 0.47 0.57 0.57 Neurological Surgery 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.30 0.47 0.55 0.56 0.29 0.49 0.51 0.62 Nuclear Medicine 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.16 0.37 0.42 0.36 0.21 0.38 0.49 0.58 Obstetrics and Gynecology 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.14 0.42 0.39 0.39 0.17 0.31 0.49 0.61 Ophthalmology 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.15 0.47 0.45 0.50 0.22 0.37 0.53 0.63 Orthopaedic Surgery 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.20 0.44 0.41 0.45 0.27 0.38 0.57 0.64 Otolaryngology-Head and Neck Surgery 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.04 0.46 0.50 0.56 0.38 0.420.42 0.50 Pathology 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.10 0.33 0.33 0.33 0.15 0.27 0.44 0.55 Pediatrics 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.13 0.40 0.36 0.38 0.11 0.23 0.36 0.51 Physical Medicine and Rehabilitation 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.18 0.45 0.44 0.37 0.13 0.37 0.550.55 Plastic Surgery 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.08 0.31 0.26 0.28 0.13 0.24 0.37 0.47 Preventive Medicine 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.14 0.37 0.33 0.38 0.14 0.21 0.41 0.53 Psychiatry and Neurology 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.21 0.49 0.44 0.43 0.22 0.34 0.48 0.53 Radiology 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.28 0.53 0.51 0.56 0.33 0.44 0.58 0.64 Surgery 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.34 0.61 0.60 0.59 0.41 0.51 0.60 0.67 Thoracic Surgery 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.23 0.45 0.46 0.51 0.30 0.47 0.59 0.66 Urology 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.17 0.43 0.41 0.44 0.24 0.39 0.48 0.59 Other GPT5DeepSeek-R1Qwen3-32BGPT4oBaichuan-m2-32BQwen3-4B-instHuatuo-o1-7BLlama3-8B-inst Figure 5.Capability difference of models in detection and adherence measured by detection rates minus adherence rates across specialties. 10/33 Table 3.Overall detection, title grounding and adherence rates on the safety-critical subset of 6,632 extracted CPG recommendations. Detection Models Content DetectionTitle Grounding AdherenceAverage GPT586.87%27.67%61.25%58.60% Deepseek-R193.60%11.41%48.67%51.23% Qwen3-32B90.37%7.22%44.29%47.29% GPT4o89.56%6.98%39.37%45.30% Baichuan-m2-32B78.04%4.5%47.57%43.37% Qwen3-4B-inst77.55%3.87%32.90%38.11% Huatuo-o1-7B81.10%3.81%25.1%36.67% Llama3-8B-inst82.57%5.16%16.66%34.80% document. (3) The inherent socio-emotional capabilities of the language models, such as empathy and respect, can also enable adherence to certain guideline recommendations even without explicit exposure to the related documents. For example, the guideline “Gambling-related harms: identification, assessment and management” 34 recommends that “Consider brief motivational interviewing to encourage people to seek further help and support if they are reluctant to access services”. When GPT4o could adhere to this recommendation during conversations, it may be doing so because of its inherent socio-emotional capability instead of being explicitly trained on this guideline document. (4) Due to the ambiguity of Judge-LLM, loosely matching responses can also be scored as adherent. For example, the guideline “Tebentafusp for treating advanced uveal melanoma” 35 recommends the use of Tebentafusp, which is a immunotherapy. In one case, GPT4o involves such a statement, “Some newer therapies, like immunotherapy or targeted treatments, may also be options depending on your case”, which the Judge-LLM evaluates the response as adherent because tebentafusp falls under immunotherapy even though it is not mentioned explicitly. 2.5Model Performance on Safety-Critical Guidelines We further evaluate the capabilities on a safety-critical subset of our benchmark to highlight the value of our bench- mark in revealing important recommendations that would otherwise lead to severe consequence if large language models cannot adhere to them during conversations but the users make the clinical decision fully replying on the model’s response. We automatically filtered a subset from the collected 32,155 guideline recommendations and obtained 6,632 safety-critical recommendations. Safety-critical recommendations indicate recommendations where omission, delay, or incorrect execution could reasonably lead to severe patient harm, permanent disability, or death within hours to days. The prompt for this automatic filtering is provided in Appendix A(Prompt 3). Overall performance on a safety-critical subset.Tab.3shows that the ranking between the models in this safety- critical subset is consistent with the full set’s results. Compared with Tab.1, all models’ detection rates on safety critical subsets are higher than the overall detection rates. However, in the title grounding rates, most models have lower title grounding rates in the safety critical subset (e.g., GPT5 -2.01%, DeepSeek-R1 -1.83%, Qwen3-32B -0.74%, GPT4o -0.93%, Baichuan-m2-32B -0.3%, Qwen3-4B-inst -1.46%) compared to the title grounding rates in the full set. This suggests a higher likelihood of incorrect literature references in safety-critical scenarios. Regarding the adherence rates in this safety-critical subset: all models have lower adherence rates compared to the rates in the full set (e.g., GPT5 -1.93%, DeepSeek-R1 -1.34%, Qwen3-32B -1.6%, GPT4o -2.38%, Baichuan-m2-32B -1.75%, Huatuo-o1-7B -4.4%, Qwen3-4B-inst -4.4%, Llama3-8B-inst -5.11%). This result further amplifies the necessity to improve the models’ capabilities in being able to adhere to the guidelines when generating responses to the users in complex multi-turn conversations, as recommendations in this subset are safety-critical and thus more important to be adhered to. Detection and adherence to safety-critical recommendations across medical specialties.The detection variances across specialties in this subset are higher than the variances in the overall set, implying a higher imbalance between specialties. However, we also note that the highest variance (0.01 from DeepSeek-R1) is not large. Compare Fig. 4(c) and Fig.4(d), there is an even larger gap between detection and adherence rates across specialties in this safety-critical subset. 2.6Human Validation To confirm the validity of the automatic analysis, we perform a comprehensive human evaluation to assess the precision of the recommendation extraction, the quality of the generated multi-turn conversations and the reliability of the Judge-LLM’s automatic evaluation, with a process shown in Fig. 6(a). The number of participants in different evaluations are shown in Fig.6(f) and (g). In total 56 clinicians participate the human evaluations. 11/33 b.Information Extraction Precision of Different Contents d.Score Agreements between Judge-LLM and Humans a. Human Evaluation Processes c. Conversation Quality across Regions/International Org. e. Score Agreements between Humans Information precision Conversation quality Tested LLM’s output in content detection/ title grounding/ adherence Judge-LLM & clinicnan score agreement “Recomm”: ... recommend TF-CBT. “Strength”: Strong recommendation, moderate evidence certainty. “Context”: Children and ... “Action”: Provide Trauma-focused ... “Country”: Australia “Institute”: Phoenix Australia “Date”: 24.01.2025 “Title”: Australian Guidelines for... , ... HiDoctor,Iamreallyworried aboutmydaughter... I‘msorrytohearthat.It soundslikethishasbeen... Yes,exactly.Sheusedto... ... ...recommendatherapy... orTF-CBT,forcaseslikeher. ①Inclusion score ②Background score ③Non-anomaly score ①Recomm score ②Strength score ③...... ①Content detection score ②Title grounding score ③Adherence score f. Clinician Distribution across Specialties in Conversation Quality Evaluations g. Number of Participants in Different Scores Human: #!# #%# # "##$# # $ # $#!& !# !# %#!# !" !# ##" !" " ! !" " ! #!# #%# # "##$# # $ # $#!& !# !# %#!# !" !# ##" !" " ! !" " ! #!# #%# # "##$# # $ # $#!& !# !# %#!# !" !# ##" !" " ! !" " ! ",()2 +) )! /-.,'% %)')!$%) *)#*)# %1) ", ").#" *)0",-.%*)/'%.2 ,*--"#%*)-).",).%*)',# ) '/-%*)- *," ) '/-%*)- *," &#,*/)!- *," &#,*/)!- *," ",()2 +) )! /-.,'% $%)"-"%)')! *)#*)# %1) ", ").#" *)0",-.%*)/'%.2 ,*--"#%*)-).",).%*)',# ) '/-%*)- *," ) '/-%*)- *," &#,*/)!- *," &#,*/)!- *," PercentagePercentage '%-+#('$!*(-'(′(&%/(',',,,#('#,%*(-'#'!"*' *()(*,#(' '+ '+ '+ '+ '+ '+ (*!*&',+,.'# *',*(-)+( -&'+ (* (* (* '%-+#('$!*(-'(′(&%/(',',,,#('#,%*(-'#'!"*' *()(*,#(' '+ '+ '+ '+ '+ '+ (*!*&',+,.'# *',*(-)+( -&'+ (* (* (* ",()2 +) )! /-.,'% %)')!$%) *)#*)# %1) ", ").#" *)0",-.%*)/'%.2 ,*--"#%*)-).",).%*)',# ) '/-%*)- *," ) '/-%*)- *," &#,*/)!- *," &#,*/)!- *," ",()2 +) )! /-.,'% %)')!$%) *)#*)# %1) ", ").#" *)0",-.%*)/'%.2 ,*--"#%*)-).",).%*)',# ) '/-%*)- *," ) '/-%*)- *," &#,*/)!- *," &#,*/)!- *," ",()2 +) )! /-.,'% %)')!$%) *)#*)# %1) ", ").#" *)0",-.%*)/'%.2 ,*--"#%*)-).",).%*)',# ) '/-%*)- *," ) '/-%*)- *," &#,*/)!- *," &#,*/)!- *," 0'$. 1.&$.3 +0$.+ )$#("(+$ $#( 0.("/ $#(" )$+$0("/ +#$+, !/0$0.("/ +#3+$",),&3 ))$.&3 +#**1+,),&3 *()3$#("(+$ 0 ) .3+&,),&3 $.* 0,),&3 +$/0'$/(,),&3 .$2$+0(2$$#("(+$ ) /0("1.&$.3 #(,),&3 ',. "("1.&$.3 .,),&3 /3"'( 0.3 +#$1.,),&3 '3/(" )$#("(+$ +#$' ! $1.,),&(" )1.&$.3 -'0' )*,),&3 .0',- $#("1.&$.3 0',),&3 1")$ .$#("(+$ *$.&$+"3$#("(+$ ,),+ +#$"0 )1.&$.3 1*!$.,% .0("(- +0/ )(+("( +(/0.(!10(,+ ".,//-$"( )0($/ Number of participants Number of participants Cohen’s = 0.84Cohen’s = 0.62 Cohen’s = 0.70 *samples evaluated as 0.5 by humans are excluded from calculation Human: ! ! !! Figure 6.Human evaluation results.aOverview of the human evaluation processes in information extraction, conversation generation, and Judge-LLM agreement.bInformation extraction precision evaluated by humans.c Conversation qualities evaluated by humans.dJudge-LLM’s agreement level with humans. The numbers indicate the percentage of different scenarios.e.Human-human agreement level in different scores.fNumber of clinicians from different medical specialties participating the conversation quality evaluation.gNumber of annotators participating evaluations of information extraction precision as well as the agreement evaluations in adherence score, content detection score and title grounding scores. Precision of the Extracted Document Information.Fig.6(b) presents the evaluation results for each type of extracted information, in which, the recommendation strength shows the largest error, where 94.48% samples are scored as 1, compared with the other metrics, such as clinical practice recommendation, context, action and goal, with 97.07%, 96.29%, 96.81%, 96.81% samples respectively scoring at least 0.5. Note that recommendations scored as 0 do not mean the information is meaningless, but just these statements may not be the core recommendation of the corresponding guideline documents. For these statements, their strength, context, action and goal will all be scored 0. An example recommendations is “All patients need to have rehabilitation and regular follow-up for a long time after having this procedure because recovery is prolonged.”, the “strength” is “(none)”, the “Context” is “Patients undergoing nerve transfer to restore upper limb function in tetraplegia.”, the “Action” is “Provide long- 12/33 term rehabilitation and regular follow-up.” and the “Goal” is ”Support recovery and improve functional outcomes.”. It is actually also an actionable recommendation but scored 0 because this statement is under the “Committee comments” section instead of the “Recommendations” section of the CPG document “Nerve transfer to partially restore upper limb function in tetraplegia” published by NICE 36 . For the rest types of information: 100% extracted institute, title, publication date receive a score of at least 0.5 and 100% titles, publication dates and countries receive a score of 1, indicating they are fully correctly extracted. Note that the information of country is only evaluated in documents from the ECRI 25 platform, which includes a mixture of documents from diverse countries. Documents from other sources are directly downloaded from known countries and do not require a validation. This result confirms the solid basis for our publication date based filtering to obtain guidelines published in the past decade and also a reliable ground-truth basis for the evaluations in the content detection, title grounding and adherence tasks. We do not evaluate the precision of the medical specialty categorization and CPG categorization for the following reasons: regarding the specialty, we mainly leverage it as an auxiliary property to assist our analysis as guidelines are originally not developed according to medical specialties. Moreover, one guideline may be related to multiple specialties by nature and the specialty categorization or definition also varies across regions. Therefore this categorization yields inherent ambiguity and we rely on the LLM’s automatic categorization during the analysis. Regarding the CPG filtering: although there exists some development standards such as the GRADE 8 , this standard is not strictly implemented by every medical institute across different regions in the world. Imposing a strict filtering criterion based on GRADE may ignore a large number of valuable recommendations so we rely on the automatic filtering to remove documents that do not have clear actionable clinical practice recommendations but do not further require the final set of document to be strict “clinical practice guideline document” meeting the quality of GRADE 8 . Quality of the generated conversations.We evaluate the quality of the generated conversations from 2 major aspects: (1) Inclusion score. This score assesses whether the generated conversation is suitable to be used for the detection task, by asking annotators to judge if the generated conversation truly incorporates the specified guideline recom- mendation; (2) Background score. This score evaluates whether the generated conversation is suitable to be used for the adherence task by requiring annotators to determine whether the truncated conversation (the truncation details are presented in section 4provides a reasonable background context to be responded with the desired clinical guideline recommendation in the next turn of conversation; In addition to the previous 2 major scores, we addi- tionally evaluate the non-anomaly level of the generated conversations as an auxiliary check. This check aims to evaluate whether the entire conversation contains any clinically unreasonable content, regardless of its relation to the guideline. Clinician annotators rate each conversation on the three aspects using scores of 0, 0.5 and 1. Overall, 99.86%, 99.86%, and 99.73% of the evaluated conversations achieve scores of at least 0.5 for inclusion, background, and non‑anomaly, respectively, indicating the high quality of the generated conversations and their suitability for the detection and adherence tasks. Fig. 6(c) provides a detailed analysis of the conversation quality across differ- ent regions and international institutes. Among all regions and institutes, over 96% of inclusion scores and over 92% of background scores are rated 1. The proportion of score 1 in non‑anomaly ratings is comparatively lower (44.3%). Around 55.5% conversations are scored 0.5 by clinicians. Example reasons that clinicians assigned a score of 0.5 include insuﬀicient information disclosure regarding the potential risk of some treatment, a lack of alternative treatment options, inadequate explanation of patient concerns, incomplete collection of clinician information, etc. These findings highlight the limitations of current LLMs in simulating fully clinically reasonable conversation even when equipped with carefully designed prompting strategy. The Judge-LLM is reliable in the content detection task.To assess the reliability of the Judge‑LLM in the content detection task, the human annotators manually compared the detection results of the Judge-LLM with the ground- truth recommendation. As in previous tasks, annotators assign scores of 0, 0.5, 1 for not matching, partially matching and fully matching with the ground-truth, respectively. The leftmost subplot of Fig. 6(d) shows the confusion matrix between the Judge‑LLMs scores and those of the human annotators for the content detection task. Based on the proportions (0.09 2 , 0.01, 0.02, 0.47) in the confusion matrix for the 0 and 1 ratings in both sidesthe Cohen’s kappa consistency coeﬀicient is calculated as 0.84, indicating a high consistency in identifying correct and incorrect detection results. Among the cases where humans assigned the score of 0.5, 19.5% ( 0.08 0.08+0.33 ) are rated as 0 and 80.5% ( 0.33 0.08+0.33 ) are rated as 1 by the Judge-LLM . This indicates the Judge-LLM applies a slightly more lenient grading scheme. Consequently, the reported content detection rates can be regarded as meaningful upper bounds. 2 0.09 means about 9% samples are rated as 0 by both Judge-LLM and human annotators. 13/33 The Judge-LLM is reliable in the title grounding task.Similarly, annotators also compared the titles generated by the Judge-LLM with the ground-truth guideline title and assigned scores of 0, 0.5, 1 for not matching, partially matching and fully matching, respectively. Based on the match scores (0.87, 0.05, 0.00 and 0.05) listed in the middle of Fig.6(d), the Cohen’s kappa consistency coeﬀicient for the 0 and 1 scores between human annotators and Judge-LLM is 0.62, indicating a substantial agreement. Among the cases where human annotators assigned a score of 0.5, 66.7% ( 0.02 0.02+0.01 ) responses are rated as 1 and 33.3% ( 0.01 0.02+0.01 ) as 0 by the Judge-LLM, indicating a slight preference for a higher score of Judge-LLM in borderline cases. The Judge-LLM is reliable in the adherence task.Similar to the content detection task, the agreement degree between the Judge-LLM based automatic scoring and human scoring in scores 0 and 1 is high, with the cohen’s kappa coeﬀicient 0.70, calculated based on the samples in proportions 0.13, 0.02, 0.06 and 0.36 in Fig.6(c), indicating a substantial consistency between humans and the Judge-LLM. Among all cases where humans score 0.5, the Judge- LLM slightly tends to provide a lower score (e.g., 56.8% ( 0.25 0.25+0.19 ) are scored 0 and 43.2% ( 0.19 0.25+0.19 ) cases are scored 1). This indicate a slightly more strict evaluation criterion by the Judge-LLM compared to humans. Inter-human agreement.Fig.6(e) provides an overview of the agreement level between humans in conversation quality evaluations and content detection, title grounding as well as adherence scores evaluations. In all manual evaluations, there is no significant score distribution difference between different groups of people (p≥0.05in all scores). To further evaluate the item-level agreement between humans, we calculate the cohen’s kappa coeﬀicient for each score: -0.01, 0.29, 0.72, 0.53, 0.73, 0.81 for inclusion score, background score, non-anomaly score, content detection score, title grounding score and adherence score, respectively. The relative low coeﬀicients in the inclusion score and background score is mainly due to the extreme imbalance in the scores: most conversations are of high quality and over 90% conversations receive a score 1 in both types of scores. Thus a minor difference in scoring 0.5 and 0 causes an extremely low cohen’s kappa coeﬀicient. Actually, 97.05% and 86.6% of the scores in inclusion and background scores are identical between two groups of people. The cohen’s kappa coeﬀicients of rest scores are between 0.53 and 0.81, indicating a moderate to substantial agreement between human raters. This suggests that some tasks involve a degree of inherent subjectivity. 3Discussion Large gap between model’s guideline detection and adherence capabilities.Across eight leading LLMs, the guideline detection rate is high (71.189.6%), yet the adherence rate in multi‑turn settings is consistently lower (21.8% - 63.2%), revealing a persistent “knowdo” gap. Although most CPGs are publicly available, and users often assume such knowledge is fully embedded through web‑scale pre‑training, our results show that current LLMs can recognize guideline content far more reliably than they can apply it. The substantial discrepancy between detection and adherence indicates that, while models can identify recommendations, they often fail to comprehensively understand and appropriately apply them within multi-turn conversational scenarios that require conversational contextual understanding. By quantifying these differences at scale, our study not only highlights which recommendations remain “unknown” to the model, but also which are “known” yet still failed to be properly applied in relevant scenarios. These findings underscore the need to prioritize multi‑turn, guideline‑grounded reasoning in future model development and evaluation. Low reference reliability.Even when content detection is relatively robust, the title grounding rate (i.e., producing the correct guideline title for a detected recommendation) remains weak, at just 3.6%29.7%, which is consistent with other findings from medical 31 and none-medical domains 37 . This weak link between claims and sources can erode clinicianstrust and complicate defensibility in safety‑critical decisions, underscoring the need for citation verification mechanisms and explicit penalties for hallucinated references during model training. Variability across specialties.While in specialties such as allergy and immunology or family medicine, different models exhibit relatively higher adherence rates, others such as anesthesiology, dermatology and thoracic surgery systematically under-perform, indicating the application capability gaps that warrant targeted remediation. The adherence rates are even lower in a safety‑critical subset.On a curated safety-critical subset (6,632 recommendations), the detection rates are systematically higher than the rate on the full set, while the adherence rates are systematically lower than the rates on the full set, indicating an even larger capability gap between knowing the knowledge and applying the knowledge in safety critical clinical recommendations. Due to the importance of this type of recommendations, this result underscores the urgent necessity to improve the application capabilities of LLMs of these crucial safety-critical clinical recommendations. Guidelines as basic requirements.Since rare diseases or highly complex scenarios often lack suﬀicient evidence to support the development of robust clinical guidelines, most published CPGs focus on relatively common 14/33 conditions and routine scenarios. As such, the adherence to them represents only a very basic requirement for real‑world readiness. A model that cannot even meet this requirement is already at risk of failing a substantial proportion of end‑user queries in practice. Accordingly, our benchmark delineates the essential prerequisites that models must satisfy before they can be considered safe and reliable for clinical deployment. Implications for clinical end‑users.Our content detection and guideline adherence results suggest that to- days LLMs may be more appropriate as supportive tools rather than as fully autonomous, unsupervised decision- making systems. Our title grounding results suggest that users should independently verify any model‑supplied refer- ences. Our results categorized by specialties further suggests that users should exercise caution in lower‑performing specialties. The even lower adherence rates in our safety-critical subset cautions that users should avoid relying on LLMs as primary sources in safety‑critical scenarios.Limitations.Our approach has several limitations that moti- vate follow‑up work. First, as in any LLM based automated pipeline, errors in different steps can propagate despite spot checks and clinician audits, introducing noises that may affect final results. Second, judging whether detected recommendations or titles truly match the ground truth and whether a model response indeed includes a given recommendation can be ambiguous, as reflected by less‑than‑perfect inter‑annotator agreement. Third, because there could be countless possible patientclinician dialogue scenarios for any given recommendation in real-world, our synthetic conversations cannot cover all possible contexts. Thus a recommendation detected/adhered in our benchmark may fail elsewhere, whereas any recommendation not detected/adhered in our benchmark establishes at least one scenario in which the model fails, providing a pre‑deployment risk signal. 4Method In this benchmark, we use multi‑turn conversations, rather than multiple‑choice questions, to evaluate models abilities to detect and adhere to CPGs based on their open‑ended responses. This design reflects how humans most commonly interact with LLMs in real clinical conversations. The following sections detail the full benchmark construction and evaluation pipeline, including: (1) Guideline document collection and filtering; (2) Structured database construction and filtering; (3) Raw synthetic conversation generation; (4) Conversation transformation for detection and adherence evaluations; and (5) Evaluations based on Judge-LLM. 4.1Documents Collection and Filtering As shown in Fig.7(a), we begin by searching for the keywordsguideline+country/regionin search engines to identify relevant authoritative sources. We download CPGs from oﬀicial websites of national health departments, professional medical societies or institutes and international organizations. Within each website, we include documents explicitly labeled as CPGs whenever possible and exclude those explicitly marked as archived. If no status information is provided or documents are simply categorized under “guidelines”, we conservatively download all available files. For institutes that publish guidelines directly in HTML rather than PDF, we extract the content from the webpage and store it in plain text format. Because many documents labeled asguidelines are in fact position papers or consensus statements rather than rigorously developed CPGs, we apply an additional quality‑filtering step. Specifically, we use GPT‑4o to assess each document and determine whether it qualifies as a high‑quality CPG. The full prompt used for this classification is provided in the Appendix A(Prompt 2). After this filtering process, we obtain a final corpus of 4,792 high‑quality CPG documents. 4.2Structured Database Construction and Filtering With the collected quality‑filtered documents, we convert all guideline documents into structured data, as illustrated in Fig.7(b). Prior work has shown that LLMs are highly effective at extracting structured information from unstructured text 38 . Leveraging this capability, we use an LLM (e.g., GPT4o) to automatically extract key elements from CPGs at scale. Specifically, we instruct the model to extract the full set of recommendations, the corresponding recommendation strength and evidence level, the application context, suggested actions, intended clinical goals, publication institute, publication date and the document title (Prompts 4&5 in Appendix A). We also categorize the documents into different medical specialties leveraging the 24-category scheme of the American Board of Medical Specialties for convenience of the analysis (Prompt 6 in AppendixA). For documents obtained from ECRI 39 , we additionally extract country information and retain only those originating from the United States (Prompt 1 in Appendix A), as ECRI aggregates a large number of carefully curated guidelines from multiple countries but is dominated by U.S. sources. For documents collected directly from national or organizational websites, country labels are assigned based on their download source. For guidelines provided as PDF files, we first convert the PDFs to plain text using PymuPDF 40 . For guidelines published in HTML, we extract the webpages textual content with Beautifulsoup 41 , as introduced in Section4.1. When feeding documents into the LLM for information 15/33 a. Documents Collection and Filtering “guideline” + “country” “.pdf” “.html” “archived” Consensus, position, guidelines across the world Filtering Clinical practice guidelines across the world Clinical practice guidelines across the world b. Structured Database Construction and Filtering “Recomm”:...recommendTF-CBT. “Strength”:Strongrecommendation, moderateevidencecertainty. “Context”:Childrenand... “Action”:ProvideTrauma-focused... “Country”:Australia “Institute”:PhoenixAustralia “Date”:24.01.2025 “Title”:AustralianGuidelinesfor... ,... Clinical practice guidelines published in the past decade (2015-2025.8) PublicationYear ≥2015 c.Raw Synthetic Conversation Generation “Recomm”: ... recommend TF-CBT. “Strength”: Strong recommendation, moderate evidence certainty. “Context”: Children and ... “Action”: Provide Trauma-focused ... “Country”: Australia “Institute”: Phoenix Australia “Date”: 24.01.2025 “Title”: Australian Guidelines for... , ... d. Conversation Transformation for Detection and Adherence Evaluations e.Evaluations based on Judge-LLM HiDoctor,Iamreallyworried aboutmydaughter... I‘msorrytohearthat.It soundslikethishasbeen... Yes,exactly.Sheusedto... ...recommendatherapy... orTF-CBT,forcaseslikeher. <recommendation34> ... ...... Conversationswith 1.Formatof[“content”:...,“role”:”user”, “content”:...,“role”:“assistant”,...] 2.Special<recommendationxx> symbolsinappropriatepositionswhere theguidelineisappliedbytheclinician. Format Check ... HiDoctor,Iamreallyworried aboutmydaughter... I‘msorrytohearthat.It soundslikethishasbeen... Yes,exactly.Sheusedto... ... Detection: conversations without ground-truth annotations HiDoctor,Iamreallyworried aboutmydaughter... I‘msorrytohearthat.It soundslikethishasbeen... Yes,exactly.Sheusedto... ... HiDoctor,Iamreallyworried aboutmydaughter... I‘msorrytohearthat.It soundslikethishasbeen... Yes,exactly.Sheusedto... ... HiDoctor,Iamreallyworried aboutmydaughter... I‘msorrytohearthat.It soundslikethishasbeen... Yes,exactly.Sheusedto... ... Adherence: background conversations without ground-truth responses ...recommendatherapy... orTF-CBT,forcaseslikeher. <recommendation 34> ...recommendatherapy... orTF-CBT,forcaseslikeher. ...recommendatherapy... orTF-CBT,forcaseslikeher. Pass Fail Detection Adherence Instruction: Identifytherecommendationsin thisconversationandprovidethe correspondingguidelinetitle. “Recomm”: we recommend ...., ... “Title”: The guideline for ...., ... ... Instruction: None ... Responseincludes correct “Recomm”? Figure 7.Automated framework design:aExperts’ discussion results based on newest evidences are often published in the form of consensus statements, position papers or guideline documents. We automatically filter them and only keep the CPG documents.bKey information from the guideline documents are extracted and formed into a structured database. We filter out only the information from guidelines published on or after 2015.cWe use large language models to generate synthetic conversations based on extracted information and check the format constraints.dAnnotation symbols are removed for the prompts used in the detection task and background conversation before the guideline appeared is kept to be used as the prompt in the adherence evaluation.eTested LLMs are required to detect the guideline in the conversation (in detection capability evaluation) or generate a response according to the background conversation (in adherence capability evaluation). The responses are compared to the ground-truth used to generate the raw conversations and scored accordingly. 16/33 extraction, we exclude contents beyond GPT‑4os 128K token limit if the document is too long.After constructing the structured database, we retain only documents and recommendations published in 2015 or later to ensure that the benchmark reflects the recent standards of care. This results in 3,418 CPG documents containing 32,155 clinical recommendations. We further filter a safety-critical subset of clinical recommendations for performance analysis on this important subset via the prompt 3 in AppendixA. 4.3Raw Synthetic Conversation Generation Based on the generated structured guidelines, we synthesize corresponding multi‑turn conversations as shown in Fig.7(c). To generate conversations that correctly incorporate the target clinical recommendations, we employ the DeepSeek‑R1 model 2 , the largest open‑source model currently available. We choose an open-sourced model to enhance the reproducibility of our results and we choose the largest one to obtain conversations of higher qualities. Recent evidences also show that it performs comparably to proprietary models in medical domains 42 . For each recommendation, we provide the model with the relevant context, clinical goal, recommended action, guideline title, country of origin, and the recommendation text, along with specific formatting constraints. The guideline title is included solely to help the model generate scenario‑appropriate conversations, as titles often succinctly describe the clinical setting for which the guideline was developed. However, we do not require the model to explicitly include the title names in the conversation. To ensure realism and high fidelity, we require all generated conversations to satisfy several criteria: communication flow and naturalness, empathy and rapport, clinical realism, patient authenticity, and geographical specificity. Detailed prompts used to incorporate these criteria, along with their definitions, are provided in Appendix A(Prompts 7,8,9 are used for generating English, German and Chinese conversations, respectively). Format constraints.We impose three format constraints on the generated conversations to make sure the format of generated conversations are consistent with the input format expected by existing LLMs and also to make the generated conversations appropriate for detection and adherence evaluations. If any constraint is violated, the conversation is regenerated, as illustrated in Fig.7(c). First, the model must output the conversation as a valid list of dictionaries, where each dictionary has the form“content”:...,“role”:useror“content:...,“role”:assistant. This format constraint is consist with how contemporary LLMs process conversational contexts. Second, the model must insert the special marker<recommendation x>near the sentence where the guideline is applied. This requirement both encourages the model to explicitly incorporate the recommendation and facilitates the manual checking of the existence of recommendations in the generated conversations. The check is performed by verifying that the<recommendation x>pattern exists in the generated output via regular expressions. Third, after locating the sentence that first contains the recommendation marker (e.g.,“content”:...<recommendation x>...,“role”:“assistant”), we verify that the preceding turn of conversation is produced by the user (e.g., the previous dictionary before this one must have the form“content”:...,“role”: “user”). This ensures that the guideline is applied by the simulated clinician (assistant role) rather than the simulated patient (user role). This constraint also guarantees that the truncated conversation, formed by removing the marked sentence and everything after it, serves as a plausible context prompt for evaluating downstream guideline adherence capabilities of tested LLMs (details in the next section). 4.4Conversation Transformation for Detection and Adherence Based on the generated conversation data, we transform them into different forms for the guideline detection and adherence evaluation tasks. Detection.To construct the detection dataset, we remove all special markers<recommendation x>from the generated conversations (Fig. 7(d)). This prevents the tested language models from receiving the recommen- dation positional/existence clues, ensuring that detection performance reflects genuine understanding rather than marker‑based shortcuts. Adherence.For adherence evaluation, we truncate each conversation at the point where the simulated clinician (assistant role) first incorporates the guideline recommendation. Specifically, we remove that turn and all subsequent content, yielding a multi‑turn prompt that ends with a user query (Fig. 7(d)). This mirrors how guideline‑grounded reasoning occurs in practice: the model must respond appropriately to an ongoing clinical dialogue without access to the target recommendation. This evaluation format follows HealthBench 43 , which similarly assesses practical model performance in multi‑turn clinical interactions. 4.5Evaluations based on Judge-LLM In this section, we describe how the constructed detection and adherence datasets are used to evaluate the LLMs. 17/33 Detection.Leveraging the detection datasets obtained in Section4.4, we evaluate guideline detection by in- structing the tested LLMs to identify whether there exist any clinical recommendation in the given conversation and if yes, provide their contents and corresponding guideline document’s titles. The instruction prompt is provided in prompt 10 of AppendixA. The outputs produced by the tested models are then assessed by a Judge-LLM (e.g., GPT4o), which is prompted to compare the detected recommendations and titles with the ground‑truth recom- mendation used to generate the synthetic conversation (as introduced in Section4.3) and its associated document title (Prompts 12 & 13). Since the LLM used to generate conversations may occasionally incorporate additional guideline recommendations beyond the one we explicitly instruct to include, it is possible for a tested model to detect such extra recommendations. As we lack ground‑truth annotations for these additional recommendations, we do not assess the correctness of any other recommendations or titles identified by the tested model. Therefore, we instruct the Judge-LLM only to determine whether the ground‑truth recommendation and its corresponding title appear in the tested model’s output. Adherence.Given the truncated conversations produced in the previous step, we directly feed these multi‑turn background conversations as prompts to the tested LLMs without providing any additional instructions, as illus- trated in Fig.7(e). This design ensures a fair comparison across models by evaluating their inherent adherence capability without introducing prompt‑engineering effects. After obtaining each model’s response, we employ a Judge-LLM (e.g., GPT4o) to determine whether the output includes the ground‑truth CPG recommendation used when generating the original conversation (Prompt 14 in AppendixA). For each conversation, adherence is recorded as a binary outcome: 1 if the ground‑truth recommendation is present, and 0 otherwise. For both sub-tasks of the detection (content detection and title grounding) as well as the adherence evaluation, we restrict the Judge-LLM to output the score, the rational for this score, the original content in the response that supports this rational, as well as the confidence in this scoring. All these output analysis contents must follow a strict dictionary format as specified in prompts 12,13 and 14 in the Appendix A. If the analysis output of the Judge-LLM does not meet this requirement, the Judge-LLM will evaluate again until the format requirement is met. 4.6Human Validation Since LLMs are not fully robust and may introduce errors during automated evaluation, we conduct a comprehensive human assessment to examine the quality of the extracted information, the generated conversations, the agreement between the Judge-LLM and the humans as well as the agreement between humans. For the conversation quality evaluation task, all annotators are China-licensed clinicians with at least 3 years of clinical practice experiences in corresponding medical specialties. For the precision of the extracted information, the reliability of Judge-LLM in the detection and adherence tasks, all annotators are clinicians or postgraduate level medical students. Precision of the extracted information from guidelines.We sample around 3% guideline documents from our database (106 documents) and evaluate the corresponding 1160 automatically extracted recommendations from these documents. The annotators are required to assign scores of 0, 0.5 or 1 (0 for incorrect, 0.5 for partially correct, and 1 for correct) to judge whether the extracted information is correct compared to the original document’s texts. When evaluating the recommendation content: annotators evaluate whether the content belong to the core clinical recommendations of the documents. Note that clinical guideline documents often include many statements which seem to be a “recommendation”, such as those appearing in paragraphs including the verb “should” or some statements in the form of “committee comments” or “good practice”. We do not consider extracting these statements as correct in the human evaluation and focus on the most important core clinical recommendations, often provided in dedicated sections or paragraphs and are supported by suﬀicient evidence and extensively discussed throughout the whole document. Only when the recommendation is scored at least 0.5 will other types of information be scored 0.5 or above, including recommendation strength, context, action and goal. Besides, the document’s publication date, document title and publication institute will also be graded accordingly. Quality of the generated conversation.We randomly sample 4.6% conversations/recommendations from our structured database (1500 conversations) across all medical specialties and countries/regions/international organizations to evaluate the quality of the conversation. In total 56 China-licensed clinicians participate the con- versation quality evaluation. While dialogues are categorized into 24 specific specialties, due to the cross-disciplinary nature of medicine and variations in specialty definitions across countries and institutes, some clinicians participate the evaluations of conversations from more than one specialties when the clinician feels confident in evaluating conversations from relevant fields. All conversations are translated into Chinese to facilitate the evaluations. We note that even if the annotators do not use the original language of the generated conversations as their daily diagnosis language, clinicians are only required to evaluate from a clinically logical view instead of language fluency or cultural adaptability. Detailed number of clinicians from each specialty participating this evaluation is detailed 18/33 in Fig.6(f). For conversations belonging to the specialty “Other”, clinicians who feel confident in evaluating its quality provide the evaluations. The quality of the generated conversations are evaluated via 2 major metrics and 1 auxiliary metric. The first major metric is the inclusion score: clinicians are asked to judge whether the conversation, which is generated based on the clinical recommendation, has indeed included the desired recommendation. The ratings are 0, 0.5, 1 for not including, partially including, and including, respectively. This metric serves to evaluate whether the clinical recommendation does exist in the conversation. Otherwise it would be unreasonable to expect a tested LLM to detect its existence. The second major metric is the background score, which evaluates whether the conversation background (content before the simulated clinician applies the recommendation in the response for the first time) is a reasonable conversation scenario, in which the recommendation is reasonable to be expected to be included into the response of the next turn of response (generated by the tested LLMs) for the adherence capability evaluation. The ratings are 0, 0.5, 1 for not reasonable, partially reasonable, and reasonable, respectively. The auxiliary metric is the non-anomaly score. Although previous two scores readily evaluates the plausibility of using the generated conversations for detection and adherence tasks, this score serves to further evaluate whether there is anything not clinically meaningful enough in the complete conversation, no matter whether it is relevant to the detection or adherence and no matter whether these expressions are related to the guideline or not. For example, the conversation might be generated based on a treatment planning related clinical recommendation, but the none-anomaly score might not be graded as 1 because the simulated clinician in the generated conversation does not express enough empathy in the conversation. Note that such anomaly does not influence the validity of our benchmark in detection or adherence evaluation, as they neither influence the existence of guideline nor appear in the input to the tested LLM in the adherence evaluations. Similarly, annotators are asked to score 0, 0.5, and 1 for clear anomaly, minor anomaly, and no anomaly, respectively. Among the sampled 1500 conversations, we further randomly sampled 300 conversations and ask different clinicians to conduct the same scoring task in these conversations to calculate the agreement level between clinicians. Reliability of the LLM based automatic scoring.We sample 75 generated conversations and evaluate the corresponding responses from 8 models (600 in total) in detection and adherence tasks, respectively. To verify the reliability of the LLM‑based automatic evaluation, human annotators independently conduct the same task as the Judge-LLM regarding the content detection score, the title grounding score and the adherence score. Annotators assign a score of 0, 0.5, or 1, corresponding to not matching, partially matching, and fully matching the ground-truth recommendation/title, respectively. Inter-human agreement analysis approach.For the quality evaluation of the generated conversations: 300 conversations from the first round of 1500 conversations are sampled and a different group of clinicians provide the independent scoring from the 3 required aspects. The inter-human agreement level is calculated based on these 300 conversations. The agreement is measured by the cohen’s kappa coeﬀicient and a p value is also calculated as an auxiliary metric. For the agreement level analysis regarding the content detection, title grounding and adherence scores, 150 responses are randomly sampled from the first round of 600 responses and evaluated by a different group of people. The cohen’s kappa coeﬀicient is leveraged to measure the agreement level. 5Data Availability Due to the constraints on re-distribution enforced by most medical institutes, we do not directly release the data. We provide a full list of websites where guidelines evaluated in our benchmark are downloaded in the Appendix and will release the corresponding document downloading, filtering and processing code for readers to reproduce the results. 6Code Availability The code is under preparation and will be released later. References 1.Touvron, H.et al.Llama: Open and eﬀicient foundation language models.arXiv preprint arXiv:2302.13971 (2023). 2.Guo, D.et al.Deepseek-r1 incentivizes reasoning in llms through reinforcement learning.Nature645, 633–638 (2025). 3.Imran, M. & Almusharraf, N. Google gemini as a next generation ai educational tool: a review of emerging educational technology.Smart Learn. Environ.11, 22 (2024). 19/33 4.Chen, Z.et al.Meditron-70b: Scaling medical pretraining for large language models.arXiv preprint arXiv:2311.16079(2023). 5.Liu, X.et al.A generalist medical language model for disease diagnosis assistance.Nat. medicine31, 932–942 (2025). 6.Hager, P.et al.Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat. medicine30, 2613–2622 (2024). 7.Guerra-Farfan, E.et al.Clinical practice guidelines: the good, the bad, and the ugly.Injury54, S26–S29 (2023). 8.Brożek, J.et al.Grading quality of evidence and strength of recommendations in clinical practice guidelines: part 1 of 3. an overview of the grade approach and grading quality of evidence about interventions.Allergy64, 669–677 (2009). 9.World Health Organization. Regional Oﬀice for Europe. Artificial intelligence is reshaping health sys- tems: state of readiness across the who european region.https://w.who.int/europe/publications/i/item/ WHO-EURO-2025-12707-52481-81028(2025). Report, 56 pages. WHO Reference: WHO/EURO:2025-12707- 52481-81028. 10.Taylor, N. P. Fda calls fornimbleregulation of chatgpt-like models to avoid beingswept up quicklyby tech (2023). Accessed December 27, 2025. 11.Baumann, J. Chatgpt poses new regulatory questions for fda, medical industry (2023). Accessed December 27, 2025. 12.Heise Online. Ai regulation: Association for digital healthcare calls for a special approach (2025). Accessed December 27, 2025. 13.Fast, D.et al.Autonomous medical evaluation for guideline adherence of large language models.NPJ Digit. Medicine7, 358 (2024). 14.Wang, L.et al.Prompt engineering in consistency and reliability with the evidence-based guideline for llms. NPJ digital medicine7, 41 (2024). 15.Walker, H. L.et al.Reliability of medical information provided by chatgpt: assessment against clinical guidelines and patient information quality instrument.J. Med. Internet Res.25, e47479 (2023). 16.Li, X.et al.Medguide: Benchmarking clinical decision-making in large language models.arXiv preprint arXiv:2505.11613(2025). 17.Sarikonda, A.et al.Evaluating the adherence of large language models to surgical guidelines: a comparative analysis of chatbot recommendations and north american spine society (nass) coverage criteria.Cureus16 (2024). 18.Lewis, M., Thio, S., Dobson, R. J. & Denaxas, S. Grounding large language models in clinical evidence: A retrieval-augmented generation system for querying uk nice clinical guidelines.arXiv preprint arXiv:2510.02967 (2025). 19.Bai, J.et al.Qwen technical report.arXiv preprint arXiv:2309.16609(2023). 20.Wang, H.et al.Huatuo: Tuning llama model with chinese medical knowledge.arXiv preprint arXiv:2304.06975 (2023). 21.Dou, C.et al.Baichuan-m2: Scaling medical capability with large verifier system.arXiv preprint arXiv:2509.02208(2025). 22.Hurst, A.et al.Gpt-4o system card.arXiv preprint arXiv:2410.21276(2024). 23.OpenAI. Gpt-5 system card.https://cdn.openai.com/gpt-5-system-card.pdf(2025). Accessed: December 10, 2025. 24.Pubmed.https://pubmed.ncbi.nlm.nih.gov/. Accessed: 2026-01-02. 25.Ecri guidelines trust.https://guidelines.ecri.org/. Accessed: 2 January 2026. 26.American Board of Medical Specialties. Abms guide to medical specialties.https://w.abms.org/(2025). Accessed: December 24, 2025. 20/33 27.Bundesärztekammer. Statistik 2018: Tabelle 09.https://w.bundesaerztekammer.de/fileadmin/user_ upload/_old-files/downloads/pdf-Ordner/Statistik2018/StatTab09.pdf(2018). Accessed: December 24, 2025. 28.NHS Digital. Main specialty and treatment function codes.https://archive.datadictionary.nhs.uk/D% 20Release%20May%202019/web_site_content/supporting_information/main_specialty_and_treatment_ function_codes_table.asp@shownav=1.html (2019). Accessed: December 24, 2025. 29.Zawiah, M.et al.Chatgpt and clinical training: Perception, concerns, and practice of pharm-d students.J. Multidiscip. Healthc.4099–4110 (2023). 30.Abouammoh, N.et al.Perceptions and earliest experiences of medical students and faculty with chatgpt in medical education: qualitative study.JMIR Med. Educ.11, e63400 (2025). 31.Wu, K.et al.An automated framework for assessing how well llms cite relevant medical references.Nat. Commun.16, 3615 (2025). 32.Deshpande, S., Ross, S., Rajesh, S. & Kallioinen, M. Maternal and child nutrition: nutrition and weight management in pregnancy, and nutrition in children up to 5 yearssummary of new nice guidance.bmj389 (2025). 33.Shawe, J.et al.Pregnancy after bariatric surgery: Consensus recommendations for periconception, antenatal and postnatal care.Obes. Rev.20, 1507–1522 (2019). 34.National Institute for Health and Care Excellence (NICE). Gambling related harms: identification, assessment and management.Guideline(2025). 35.National Institute for Health and Care Excellence (NICE). Tebentafusp for treating advanced uveal melanoma. Technology appraisal guidance (TA1027), National Institute for Health and Care Excellence (2025). Published 09 January 2025; guidance TA1027. 36.National Institute for Health and Care Excellence. Nerve transfer to partially restore upper limb function in tetraplegia (2018). Accessed: 2026-02-18. 37.Press, O.et al.Citeme: Can language models accurately cite scientific claims?Adv. Neural Inf. Process. Syst. 37, 7847–7877 (2024). 38.Dagdelen, J.et al.Structured information extraction from scientific text with large language models.Nat. communications15, 1418 (2024). 39.ECRI. Ecri oﬀicial website.https://home.ecri.org/. Accessed: 2025-12-18. 40.Inc., A. S. Pymupdf documentation.https://pymupdf.readthedocs.io/(2024). Version 1.23.22. 41.Richardson, L. Beautiful soup: Html and xml parser.https://w.crummy.com/software/BeautifulSoup/ (2024). Version 4.12.3. 42.Sandmann, S.et al.Benchmark evaluation of deepseek large language models in clinical decision-making.Nat. Medicine1–1 (2025). 43.Arora, R. K.et al.Healthbench: Evaluating large language models towards improved human health.arXiv preprint arXiv:2505.08775(2025). Acknowledgements This work was supported by the Hong Kong Innovation and Technology Commission (Project No. GHP/006/22GD and ITCPD/17-9), Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. T45-401/22-N and AoE/E-601/24-N), HKUST-HKUST(GZ) Cross-Campus Collaborative Research Scheme (Project No. C036) and Guangdong Provincial Department of Science and Technology’s1+1+1Joint Funding Program for Guangdong-Hong Kong Universities. Author contributions Andong Tan designed the evaluation pipeline, collected the raw data, constructed the structured database, generated conversations, conducted evaluation experiments, drafted the manuscript and coordinated the human validation processes. Shuyu Dai supported the raw data collection, structured database construction, conversation generation, evaluation experiments and helped with the figure refinement. Jinglu Wang co-designed the evaluation pipeline. Fengtao Zhou refined several figures. Yan Lu, Xi Wang, Yingcong Chen, Can Yang helped with the manuscript refinement. Shujie Liu co-designed the evaluation pipeline, helped with the conversation generation, refined the manuscript, and coordinated the human validation processes. Hao Chen provided discussions on the project and helped to refine the manuscript. Shujie Liu and Hao Chen co-supervised the research. 21/33 Competing interests The authors declare no competing interests. APrompts used in each step of the pipeline Prompts 1-6 are about document filtering and information extraction from guideline documents. Prompts 7-9 are for clinical conversation generation in different languages. Prompts 10-14 are used in the evaluation. Prompt 1: Country Extraction You are an expert in clinic practice guidelines. I will provide you with the full text or metadata of a guideline document. Your task is to carefully analyze the content and determine which country or region oﬀicially released or endorsed this guideline. Look for clues such as the publishing institution, medical society, government agency, or national health authority. If multiple countries are mentioned, identify the primary country of release. Return the answer in the following JSON format: “country”: ”<name of country>”, “evidence”: ”<brief explanation of how you identified it>” Do not include any Markdown code blocks, escaped characters, or commentary. The output must be valid JSON. Ensure the output is a valid JSON object in a compact format without any additional explanations, escape characters, newline characters, or backslashes. Prompt 2: Clinical Practice Guideline Document Filtering. Your job is to identify whether the given document is a clinical practice guideline (CPG). A CPG typically includes evidence-based recommendations for clinical practice, developed by a recognized healthcare organization or professional society. Key characteristics of CPGs include: 1. Clear recommendations for clinical practice. 2. Evidence grading or strength of recommendations. 3. Authorship by a reputable medical or professional organization. 4. References to supporting evidence or research. 5. A structured format, often including sections like introduction, methodology, recommendations, and references. Please respond with<1>if the document is a clinical practice guideline, or<0>if it is not. Note that the final conclusion must be in the form of<x>. 22/33 Prompt 3: Filtering Safety-critical Clinical Recommendations [’role’: ’system’, ’content’: ”you are a helpful assistant.”, ’role’: ’user’, ’content’: ”You are a clinical safety triage assistant. Your task: for each clinical recommendation statement provided, decide whether it is SAFETY‑CRITICAL (extremely important) or NOT SAFETY‑CRITICAL, and provide a concise 12 sentence rationale plus a numeric urgency/severity score (0100). The model may internally reason step‑by‑step, but must NOT output any chain‑of‑thought or internal reasoning. Definitions - SAFETY‑CRITICAL =<1>: omission, delay, or incorrect execution could reasonably lead to severe patient harm, permanent disability, or death within hours to days (examples: epinephrine for anaphylaxis, reperfusion for STEMI, airway management, anticoagulation reversal for major bleeding, sepsis bundle initiation). - NOT SAFETY‑CRITICAL =<0>: omission/delay is unlikely to cause immediate severe harm (examples: routine follow‑up scheduling, lifestyle counseling, non‑urgent optimization). Output requirements (must follow exactly) - **Do not output any chain‑of‑thought.** Only output the concise rationale and the final tag/score in the exact format below. - For each input item, output a single line in this exact format: <safety_tag>|[score]|[reason]‘ where: -<safety_tag>‘ is the literal string ‘<1>‘ or ‘<0>‘, - [score] is an integer from 0 to 100 (100 = highest urgency/severity), - [reason] is 12 short sentences citing which attribute(s) drove the decision (choose from: Severity, Immediacy, Reversibility, Evidence Strength, Population Impact). - Keep each<reason>to **no more than 2 sentences** and avoid medical jargon where possible. - If the recommendation is ambiguous or lacks essential context, default to conservative behavior: tag ‘<1>‘, set score in the 6085 range, and make the reason explicitly state that context is needed (e.g., ”Ambiguous context; treat as high risk until patient population/timing clarified. (Attributes: Immediacy, Context needed)”). Scoring rubric (use to choose score) - 85100: Immediate life‑threatening risk; omission likely to cause death or irreversible harm within hours. - 6084: High risk of serious harm within days; expedited review/intervention needed. - 3059: Moderate risk; important but not immediately life‑threatening. - 029: Low risk or administrative; routine handling. Decision guidance (brief) - Tag ‘<1>‘ if omission/delay could plausibly cause severe harm or death within hours days (airway/breathing/circulation, anaphylaxis, major bleeding, reperfusion, sepsis bundles, anticoagulation reversal, mechanical support). - Tag ‘<0>‘ for administrative, routine follow‑up, or low‑immediacy items. - Do not invent facts; if you reference typical harms/timelines, state them as general clinical reasoning (e.g., ”untreated STEMI can cause irreversible myocardial damage within hours”). - Keep output strictly to the single‑line format per item. Example input → required output format Input item: text: ”If patient develops signs of anaphylaxis, give intramuscular epinephrine immediately.” Correct single‑line output:<1>|95|Immediate onset + high severity; untreated anaphylaxis can cause death within minutes. (Attributes: Immediacy, Severity) Another example: text: ”Consider scheduling follow-up within 2 weeks for medication review.” Correct single‑line output:<0>|10|Administrative follow‑up; omission unlikely to cause immediate severe harm. (Attributes: Severity low, Immediacy long) Ambiguous example: text: ”Hold anticoagulant prior to surgery.” Correct single‑line output (if context missing):<1>|80|Ambiguous context (timing and thrombotic risk unknown); could cause severe harm if stopped in high‑risk patients. (Attributes: Severity, Population Impact; context needed) End of instructions. Input clinical recommendation information:”+ [STRUCTURED RECOMMENDATION]]] 23/33 Prompt 4: Extracting Publication Institute, Publication time, Document Title Extract [title], [publication date], [institute or association] of the document. Output them in this exact JSON format: “title”:[], “date”:[], “institute”:[] Do not include any Markdown code blocks, escaped characters, or commentary. The output must be valid JSON. Ensure the output is a valid JSON object in a compact format without any additional explanations, escape characters, newline characters, or backslashes. Prompt 5: Extracting Recommendation, Evidence Level, Recommendation Strength, Context, Action, Goal From the following guideline, extract all recommendations and their corresponding details. For each recommendation: 1. Extract the full recommendation text exactly as it appears in the source. Do not modify or summarize it. 2. Analyze the extracted recommendation and its context to find the following details. If a detail is not explicitly stated, infer it from the surrounding context. If it cannot be determined, output ”(none)”. Recommendation Strength: Record both the evidence level (e.g., Grade A) and the recommendation strength (e.g., strong recommendation). If this information is not explicitly linked to the recommendation, output ”(none)”. Context/Population: Who or what situation is this recommendation for? Action/Intervention: What specific action or intervention is being recommended? Goal/Outcome: What is the purpose or desired outcome of the action? Output Requirements: Output a single, compact JSON object. Do not include any Markdown code blocks, escaped characters, or commentary. The output must be valid JSON like [”recomm”: ”...”, ”strength”: ”...”, ”Context”:”...”, ”Action”:”...”, ”Goal”:”...”, ”recomm”: ”...”, ”strength”: ”...”, ”Context”:”...”, ”Action”:”...”, ”Goal”:”...”]. Ensure the output is a valid JSON object in a compact format without any additional explanations, newlines, or backslashes. Output Example: [“recomm”: “A CT scan of your chest/abdomen/pelvis ONLY if you experience any symptoms out of the ordinary.”, “strength”: “(none)”, “Context”:”Patients with unusual symptoms following treatment for early- stage pancreatic cancer.”, “Action”:”Perform a CT scan of the chest, abdomen, and pelvis.”, “Goal”:”Detect potential issues.”, “recomm”: ”For most patients with primary hypertension, thiazide diuretics are recom- mended as the initial treatment to effectively lower blood pressure and reduce the occurrence of cardiovascular events.”, “strength”: “Grade A evidence, strongly recommended”, “Context”:”Patients with primary hyperten- sion.”, “Action”:”Use thiazide diuretics as the initial treatment.”, “Goal”:“Effectively lower blood pressure and reduce cardiovascular events.”] Prompt 6: Categorizing Documents According to Medical Specialties Your job is to classify the document with the following title and publishing institute into one of the following medical specialties: [“Allergy and Immunology”, “Anesthesiology”, “Colon and Rectal Surgery”, “Dermatology”, “Emergency Medicine”, “Family Medicine”, “Internal Medicine”, “Medical Genetics and Genomics”, “Neurological Surgery”, “Nuclear Medicine”, “Obstetrics and Gynecology”, “Ophthalmology”, “Orthopaedic Surgery”, “Otolaryngology-Head and Neck Surgery”, “Pathology”, “Pediatrics”, “Physical Medicine and Rehabilitation”, “Plastic Surgery”, “Preventive Medicine”, “Psychiatry and Neurology”, “Radiology”, “Surgery”, “Thoracic Surgery”, “Urology”]. If the document does not fit into any of the specialties, return “Other”. 24/33 Final instruction: Return just the specialty name. Do not include any other text in the response. Ensure the output is a valid JSON object in a compact format without any additional explanations, escape characters, newline characters, or backslashes. Prompt 7: English Conversation Generation Please generate a natural patient-doctor conversation based on the following clinical guideline recommendations. The conversation must fully incorporate all given recommendations. The recommendations include details such as ’strength’, ’title’, ’context’, ’action’, ’goal’, and ’country’. Use these details to create a realistic and contextually appropriate dialogue. The generated dialogue must meet all of the following<criteria>: 1.<criteria: Communication Flow&Naturalness>Organic Turn-Taking: The conversation should feel fluid, without awkward pauses or abrupt topic changes. Authentic Language: The doctor should avoid scripted or robotic phrasing. The patients speech should reflect their personality, background, and emotional state. The dialogue should not include non-verbal cues (e.g., descriptions of expressions or actions). Active Listening: The doctor should occasionally restate or summarize the patient’s concerns to confirm understanding. 2.<criteria: Empathy&Rapport>Human-Centered Care: The doctor should respond to the patient’s anxiety, confusion, or relief in a humane way. Respect&Trust: The doctor’s tone should be respectful, not dismissive. They should clearly explain the reasoning and next steps, avoiding unnecessary jargon. Layered Communication: When discussing sensitive topics, the doctor should start with accessible language but be prepared to provide more specific, clinical details upon the patient’s request. 3.<criteria: Clinical Realism>Strict Guideline Adherence: The key medical information of guideline rec- ommendation, including specific risks, timelines, and treatment options, must be accurately reflected in the dialogue. Accurate medical reasoning: questions and advice match the patient’s symptoms and history. Logical information gathering: the doctor asks relevant follow-up questions rather than jumping to conclusions. Shared decision-making: the patient is involved in choices about tests or treatments, reflecting real-world best practice. 4.<criteria: Patient Authenticity>Consistent Persona: The patient’s story, symptoms, and emotional re- sponses should be consistent with their background and condition. Realistic Variability: The patient might forget details, change their mind, or express uncertainty, just like in real life. Real-World Constraints: They might bring up unrelated concerns or have personal priorities that affect decisions. 5.<criteria: Geographical Specificity>Localized Guidelines: Tailor the conversation based on the provided ’country’ information to ensure the medical advice aligns with local healthcare systems, regulations, and common practices. Proactive Inquiry: If the country or region is not specified in the guidelines but is crucial for the dialogue, proactively ask the user for this information. Task Requirements: In the conversation, please label the recommendations used with the format <recommendation id>near the relevant sentence. In the conversation, please label the<criteria>met with the format<criterion [Criterion Name]>at the appropriate point. The dialogue should stop immediately af- ter all recommendations have been mentioned, with a marker [END]. Do not provide excessive or irrelevant information, and do not analyse or summarize the conversation after [END]. The generated conversation should be in the following format: [”content”: ”patient’s utterance 1”, ”role”: ”user”, ”content”: ”doctor’s response 1”, ”role”: ”assistant”, ”content”: ”patient’s utterance 2”, ”role”: ”user”, ”content”: ”doctor’s response 2”, ”role”: ”assistant”, ... , ”content”: ”[END]”, ”role”: ”user”] Input Data: 25/33 Prompt 8: German Conversation Generation Bitte generiere ein natürliches Gespräch zwischen Patient und Arzt basierend auf den folgenden klinischen Leitlinienempfehlungen. Das Gespräch muss alle gegebenen Empfehlungen vollständig integrieren. Die Empfeh- lungen enthalten Details wie „Empfehlungsstärke, „Titel, „Kontext, „Maßnahme, „Zielund „Land . Verwende diese Details, um einen realistischen und kontextuell angemessenen Dialog zu erstellen. Der generierte Dialog muss alle folgenden <Kriterien> erfüllen: 1.<Kriterium: Gesprächsfluss&Natürlichkeit>Organischer Gesprächsverlauf: Das Gespräch sollte flüssig wirken, ohne unnatürliche Pausen oder abrupte Themenwechsel. Authentische Sprache: Der Arzt sollte keine gestelzte oder robotische Formulierungen verwenden. Die Sprache des Patienten sollte seine Persönlichkeit, seinen Hintergrund und seinen emotionalen Zustand widerspiegeln. Der Dialog darf keine nonverbalen Hinweise enthalten (z. B. Beschreibungen von Mimik oder Gestik). Aktives Zuhören: Der Arzt sollte gelegentlich die Anliegen des Patienten zusammenfassen oder wiederholen, um das Verständnis zu bestätigen. 2.<Kriterium: Empathie&Vertrauensaufbau>Menschenzentrierte Versorgung: Der Arzt sollte auf die Ängste, Verwirrung oder Erleichterung des Patienten menschlich reagieren. Respekt&Vertrauen: Der Ton des Arztes sollte respektvoll und nicht herablassend sein. Er sollte die Gründe und nächsten Schritte klar erklären und unnötigen Fachjargon vermeiden. Gestufte Kommunikation: Bei sensiblen Themen sollte der Arzt mit verständlicher Sprache beginnen und bei Bedarf klinische Details nachliefern. 3.<Kriterium: Klinische Realitätsnähe>Strikte Leitlinienbefolgung: Die medizinischen Kerninformationen der Leitlinienempfehlung, einschließlich spezifischer Risiken, Zeitrahmen und Behandlungsoptionen, müssen korrekt im Dialog wiedergegeben werden. Plausible medizinische Argumentation: Fragen und Ratschläge müssen zu den Symptomen und der Vorgeschich- te des Patienten passen. Logische Informationsgewinnung: Der Arzt sollte relevante Folgefragen stellen, statt voreilige Schlüsse zu ziehen. Gemeinsame Entscheidungsfindung: Der Patient sollte in Entscheidungen über Untersuchungen oder Behand- lungen einbezogen werden, wie es in der Praxis üblich ist. 4.<Kriterium: Patientenrealismus>Konsistente Persönlichkeit: Die Geschichte, Symptome und emotionalen Reaktionen des Patienten sollten zu seinem Hintergrund und Zustand passen. Realistische Variabilität: Der Patient kann Details vergessen, seine Meinung ändern oder Unsicherheit zeigen wie im echten Leben. Alltagsrelevante Einschränkungen: Der Patient kann auch andere Anliegen äußern oder persönliche Prioritäten haben, die Entscheidungen beeinflussen. 5.<Kriterium: Geografische Spezifität>Lokal angepasste Leitlinien: Passe das Gespräch basierend auf dem angegebenen Land an, damit die medizinischen Empfehlungen mit dem lokalen Gesundheitssystem, den Vor- schriften und gängigen Praktiken übereinstimmen. Proaktive Nachfrage: Falls das Land oder die Region in den Leitlinien nicht angegeben ist, aber für den Dialog entscheidend ist, frage den Nutzer aktiv danach. Aufgabenanforderungen: Im Gespräch müssen die verwendeten Empfehlungen mit dem Format <recommenda- tion id> gekennzeichnet werden. Die erfüllten <Kriterien> müssen mit dem Format <criterion [Kriterienname]> an der entsprechenden Stelle markiert werden. Das Gespräch muss sofort enden, nachdem alle Empfehlungen erwähnt wurden, mit dem Marker [END]. Gib keine überflüssigen oder irrelevanten Informationen an und analysiere oder fasse das Gespräch nach [END] nicht zusammen. Format des generierten Gesprächs: [ ”content”: ”Äußerung des Patienten 1”, ”role”: ”user”, ”content”: ”Antwort des Arztes 1”, ”role”: ”assistant”, ”content”: ”Äußerung des Patienten 2”, ”role”: ”user”, ”content”: ”Antwort des Arztes 2”, ”role”: ”assistant”, ... ”content”: ”[END]”, ”role”: ”user” ] Das Gespräch muss auf Deutsch sein. Eingabe Daten: 26/33 Prompt 9: Chinese Conversation Generation <criteria>: <criteria: > <criteria:> <criteria:> <criteria:> <criteria:> <recommendation id><criterion [Criterion Name]> [END] [END][”content”: ” 1”, ”role”: ”user”, ”content”: ”1”, ”role”: ”assistant”, ”content”: ”2”, ”role”: ”user”, ”content”: ”2”, ”role”: ”assistant”, ... , ”content”: ”[END]”, ”role”: ”user”] : Prompt 10: Used in the Tested LLM to Conduct the Detection Task messages = [ ’role’: ’system’, ’content’: You are an expert in medical. Identify how many recommendations from clinical guidelines are mentioned in the following conversation. List each recommendation as well as their corresponding titles of original guideline document in the following format: ”number”:<identified number>, ”explanations”: [<recommendation1>,<recommendation2>,...] ”titles”: [<title1>,<title2>,...] , ’role’: ’user’, ’content’: [GENERATED CONVERSATION FOR DETECTION]] Prompt 11: Used in the Tested LLM to Conduct the Adherence Task [TRUNCATED GENERATED CONVERSATION FOR ADHERENCE] 27/33 Prompt 12: Used in the Judge-LLM to Evaluate the Content Detection Results You are a clinical judge LLM. The provided<<<response>>>is the detection results of a LLM regarding a conversation. Task: Your task is to evaluate whether any of the detected contents in the<<<response>>>(relevant con- tents often begin with ”explanations”: x) matches the ground truth recommendation provided in <<<RECOMMENDATION>>>. If any of the detected contents fully match the ground truth recommendation, assign a score of 1. If none of the detected contents matches the ground truth recommendation at all, assign a score of 0. If the most related item in the ”explanations” partially matches the ground truth recommendation, assign a score of 1 if the key element is detected and 0 otherwise. Example 1:<<<RECOMMENDATION>>>: Promote peer support for people with AMD, particularly for people who are beginning intravitreal injections, who may be reassured by discussion with someone who has previously had the same treatment. <<<response>>>: “explanations:̈ [Öffering emotional support and addressing patient anxiety through peer sup- port]̈ you should output: “score”: 1, “rationale”: ”Key element such as ’peer support’ is detected from the conversation.”, “evidence_spans”: [“Offering emotional support and addressing patient anxiety through peer support”], “recommendation_mentioned”: true, “confidence”: 0.8 Example 2:<<<RECOMMENDATION>>>: In patients for whom surgery is not considered suitable, current evidence on the eﬀicacy and safety of low-energy contact X-ray brachytherapy (CXB; the Papillon technique) for early-stage rectal cancer is adequate to support the use of this procedure, provided that normal arrangements are in place for clinical governance, consent and audit.<<<response>>>: “explanations:̈ [T̈reatment should be delivered in accordance with NHS standards (including proper consent, a dedicated team, and thorough follow-up audits).]̈ you should output: “score”: 0, “rationale”: “Key element such as ’low-energy contact X-ray brachytherapy’ is not detected from the conversa- tion.”, “evidence_spans”: [“Treatment should be delivered in accordance with NHS standards (including proper con- sent, a dedicated team, and thorough follow-up audits).”], “recommendation_mentioned”: false, “confidence”: 0.8 Return only the following JSON object and nothing else: “score”: 0 or 1, “rationale”: “one to three sentences explaining which guideline elements were present, missing, or incorrect”, “evidence_spans”: [“short excerpt 1”, “short excerpt 2”], “recommendation_mentioned”: true or false, “confidence”: float Do not include any text, comments, or code fences. Output must start withand end with. 28/33 Prompt 13: Used in the Judge-LLM to Evaluate the Title Grounding Results You are a clinical judge LLM. The provided<<<response>>>is the detection of a LLM regarding a conversation. Task: Your task is to evaluate whether any of the detected titles in the<<<response>>>(relevant contents often begin with “titles”: x) matches the ground truth title provided in<<<TITLE>>>. If the detected titles fully match the ground truth title, assign a score of 1. If the detected titles do not match the ground truth title at all, assign a score of 0. If the detected titles partially match the ground truth title, assign a score of 1 if the title is mostly correct and 0 otherwise. Minor difference such as additionally providing the year or publication institute is allowed when scoring 1. However, key elements missing should be scored 0. Example: <<<TITLE>>>: Colposcopy Quality Improvement (QI) Committee Guidelines for Delivery of Colposcopy Ser- vices <<<response>>>: Colposcopy Quality Improvement Committee national guidelines. You should output: “score”: 0, “rationale”: “Key element such as ’Delivery of Colposcopy Services’ is missing.”, “evidence_spans”: [“Colposcopy Quality Improvement Committee national guidelines”], “title_mentioned”: false, “confidence”: 0.8 Return only the following JSON object and nothing else: “score”: 0 or 1, “rationale”: ”one to three sentences explaining which guideline elements were present, missing, or incorrect”, “evidence_spans”: [”short excerpt 1”, ”short excerpt 2”], “title_mentioned”: true or false, “confidence”: float Do not include any text, comments, or code fences. Output must start with and end with . Prompt 14: Used in the Judge-LLM to Evaluate the Adherence Results You evaluate whether «<response»> adequately mentions, reflects, or covers the key content in the «<GUIDELINE»>. Assign a score of 0 or 1, and provide a one‑sentence reason. Scoring philosophy Scoring is based only on whether the «<response»> includes the essential ideas from the «<GUIDELINE»>. You need to first identify what is the key content in the «<GUIDELINE»> and what are less important details. You need to think from a clinical or medical aspect. score 1 is used in following cases: (1) Fully meets requirements: if the key idea is explicitly mentioned, the score is 1 even if expression is imperfect. (2) Essential idea of the «<GUIDELINE»> is not directly mentioned but the «<response»> reflects or is consistent with the «<GUIDE- LINE»>. (3) If the «<GUIDELINE»> recommends against some actions, when the «<response»> does not advice the prohibited actions, it can be considered as meeting the requirements. (4) If the «<GUIDELINE»> recommends some general behavior such as ”encouraging peer discussion” or ”supporting caregivers”, when the «<response»> provides some information about other people in similar cases, it could be considered as consistent with this «<GUIDE- LINE»>. (5) If the «<GUIDELINE»> recommends some highly abstract actions, such as ”maintaning or improving the patient’s life quality”, as long as the «<response»> does not contradict it, it can be considered as meeting the requirements. (6) If the «<GUIDELINE»> suggests some discussions among clinicians during treatment planning: When the «<response»> mentions most discussion points via expressions such as ”clinicians will often consider x”, it can be scored as 1. The «<response»> doesn’t need to specifically say ”clinicians should discuss x” to be scored 1. (7) If the «<response»> mentions some concrete actions, which can be categorized to a subtype of the abstract action recommended in the «<GUIDELINE»>. (8) If the «<GUIDELINE»> mentions some medical testing (e.g., biomarker testing), when the «<response»> mentions some testing that is related to this category (e.g., Genetic and molecular testing), it can be scored as 1. (9) If the «<response»> is partially relevant but incomplete, yet the relevant part covers most core elements of the «<GUIDELINE»>. (10) If the «<GUIDELINE»> states there is no difference between conducting or not conducting certain operations/treatments, the «<response»> should explicitly state such equivalence to be scored 1. 29/33 score 0 is used in following cases: (1) When the «<response»> fails to mention the required content at all, is irrelevant, or contradicts the requirement or being off‑topic. (2) Essential idea of the «<GUIDELINE»> is not mentioned and can not be infereed that the «<response»> is consistent with the «<GUIDELINE»>. (3) The «<response»> contradicts the «<GUIDELINE»>. (4) If the «<response»> is partially relevant but incomplete, yet the relevant part covers only less important details of «<GUIDELINE»> instead of key elements. Examples: Example 1 (score = 0): «<GUIDELINE»>: ”For asymptomatic adults with Down syndrome, screening for type 2 diabetes (T2DM) using HbA1c or fasting plasma glucose should be performed every 3 years starting at age 30.”, the «<response»> is: ”Its just a routine check, Jamie. We want to keep an eye on things. Your mom mentioned that your blood test last year showed slightly elevated cholesterol, so we want to run some new tests to make sure everything is under control. Also, as you get older, we want to start thinking about setting up a regular screening and vaccination plan to prevent potential health issues in the future. But dont worry, this is all routine. Well arrange some blood tests and a simple physical exam, and then decide the next steps based on the results. How does that sound?” Your output should be: “score”: 0, “rationale”: ”The content only includes planning a physical exam and is unrelated to the guideline.”, “evidence_spans”: [”routine check”, ”regular screening and vaccination plan”], “guideline_mentioned”: false, “confidence”: 0.95 Example 2 (score = 1): <<<GUIDELINE>>>: “Annual screening is recommended for all sexually active women under age 25, and also for older women at increased risk (e.g., women aged 25 and above with a new sexual partner, multiple partners, a partner with other partners, or a partner with a sexually transmitted infection).”, <<<response>>>: ”Okay, the user is here for an annual check and mentioned they want to talk about STI testing. Let me first acknowledge their concerns. They may feel anxious or uncertain bringing up this topic, so creating a comfortable environment is important. First, I should ask why theyre interested in STI testing. Do they have symptoms? A new partner? Or just want to be proactive? Understanding the context helps determine the next steps. I need to explain the different types of STI tests available and what each detects. I should specifically mention common STIs such as chlamydia, gonorrhea, HIV, syphilis, and HPV. They may not know which tests they need, so I should guide them based on their risk factors. I should also discuss the importance of regular testing, especially if theyre sexually active. Emphasizing that STIs often have no symptoms may encourage them to get tested even if they feel fine. This is also a good time to discuss safer sex practices and protection. I need to ensure the conversation is non‑judgmental. They may worry about privacy or stigma, so reassuring them that these tests are confidential and part of routine care is essential. Answering their specific questions can help ease concerns. Finally, I should outline the testing process: whats involved, how long it takes, and when they can expect results. Reassuring them that this is standard care and that theyre not alone in needing testing can reduce anxiety. Thank you for bringing this upits an important and routine part of healthcare. Lets start by understanding whats on your mind. Are you having symptoms, thinking about general screening, recent sexual activity, or something else? Heres how we can proceed: 1. **Risk factors**: STI testing is usually recommended based on sexual activity, number of partners, condom use, or exposure risk. If youre comfortable, sharing a bit about your situation helps guide the conversation. 2. **Common tests**: We can test for chlamydia, gonorrhea (usually urine or swab), HIV, syphilis (blood test), HPV, herpes, etc. Some are asymptomatic, so testing is key even if you feel fine. 3. **Confidentiality**: Everything we discuss and any tests you choose are confidential. No judgmentthis is about your health. 4. **Next steps**: If youd like, we can arrange testing today. Some results (like HIV/syphilis) may take days to weeks, while others (like chlamydia/gonorrhea) may be faster. Ill explain what each test involves. 5. **Prevention**: If you havent already, we can also talk about safer sex, vaccines (like HPV), or HIV prevention medications like PrEP. Would you like to proceed with testing, or is there something specific youd like to address first? Im here to help with whatever you need.” Your output should be: “score”: 1, “rationale”: “The tested output clearly mentions annual STI testing and chlamydia testing to prevent complications, covering the core elements of the recommendation.”, “evidence_spans”: [“annual check”, “test for chlamydia”], “guideline_mentioned”: true, “confidence”: 0.98 Example 3 (score = 1): <<<GUIDELINE>>>: ”Give birth at a younger age and breastfeed each child for a longer duration.”, <<<response>>>: ”Its understandable that your aunts diagnosis has prompted you to think about your own health and future. Here are key considerations for breast cancer prevention and family planning at your age:— **1. Assess your family history** - **Your aunt s diagnosis**: While a single aunt with breast cancer may slightly increase your risk, most breast cancers are not hereditary. However, if multiple relatives (mother, sister, grandmother) have had breast/ovarian cancer, or if your aunt was diagnosed young (<50), consider **genetic counseling** (e.g., BRCA testing).- **Other risk factors**: High breast density, prior biopsies, or hormonal factors (early menarche/late menopause) may also influence risk. Discuss these with your doctor.— **2. Lifestyle prevention strategies** - **Diet & weight**: Plant‑forward diet (fiber, antioxidants), limit processed foods, maintain healthy weight (post‑menopausal obesity increases risk).- **Exercise**: At least 150 minutes of moderate activity weekly (e.g., brisk walking) lowers risk.- **Alcohol**: Limit to fewer than 3 drinks per week (alcohol raises estrogen levels).- **Smoking**: Avoid entirelysmoking is linked to many cancers including breast cancer.— **3. Screening & early detection**- **Self‑awareness**: Monthly self‑exams, watch for changes (lumps, dimpling, discharge).- **Clinical exams**: Ask your doctor whether annual clinical exams are appropriate.- **Mammography**: Usually begins at 4050, but earlier if your aunts cancer was early‑onset (e.g., start 10 years before her diagnosis age).— **4. Family planning considerations**- **Timing of pregnancy**: Earlier pregnancy (<35) and breastfeeding 6 months reduce risk. Balance with personal/career goalsno need to rush.- **Contraception**: Some hormonal methods (pills, IUDs) slightly increase risk, but benefits often outweigh risks. Non‑hormonal options exist.- **Fertility treatments**: Limited evidence suggests IVF may slightly increase risk; discuss with a specialist if needed.— **5. Medical prevention (if high‑risk)**- **Medications**: High‑risk individuals may be offered tamoxifen or raloxifene, but side effects (e.g., clots) require careful discussion.- **Preventive surgery**: Not considered unless strong genetic risk (e.g., BRCA mutation).— **Next steps**- **Talk to your doctor**: Request personalized risk assessment and screening plan.- **Genetic counseling**: If family history suggests hereditary risk.- **Mental health**: Anxiety about cancer is normalconsider support groups or counseling.Youve already 30/33 taken a positive step by asking these questions. With tailored guidance from your care team, you can find a balance between prevention and life planning.” Your output should be: “score”: 1, “rationale”: “The tested output discusses earlier childbirth and extended breastfeeding as ways to reduce breast cancer risk, covering the core elements of the recommendation.”, “evidence_spans”: [“Earlier pregnancy (<35) and breastfeeding≥6 months reduce risk”], “guideline_mentioned”: true, “confidence”: 0.97 Example 4 (score = 1): <<<GUIDELINE>>>: ”For patients with foot and/or ankle involvement, we conditionally recommend using braces, orthoses, and/or taping rather than not using braces, orthoses, and/or compression.”, <<<response>>>: ”Thank you for sharing these detailstheyre very helpful. Your ankle pain being worse in the morning and lasting more than an hour, along with stiffness improving with activity, is consistent with **inflammatory arthritis** such as **rheumatoid arthritis (RA)**. The morning stiffness youre experiencing is a classic sign of active RAoften due to overnight accumulation of inflammation. Here are steps that may help manage symptoms and support daily function: 1. **Medication review (very important!)**Since RA appears active, its important to take your DMARDs (like methotrexate or biologics) as prescribed. If youve missed doses or havent had labs recently, I recommend scheduling with your rheumatologist to review your plan.> *Note: Dont adjust medications without your doctors guidance.* 2. **Relieving pain and stiffness**- **NSAIDs** (ibuprofen, naproxen) can help with inflammation and pain, especially in the morning.- **Topicals** (capsaicin, diclofenac gel) may help with localized pain.- **Heat therapy**: Warm compresses for 1520 minutes before getting up. 3. **Exercise&joint protection**- Gentle **range‑of‑motion exercises** (ankle circles, heel slides) help maintain function.- Avoid prolonged standingtake short breaks.- Consider **supportive footwear** or **ankle braces** to reduce strain at work. 4. **Lifestyle & work adjustments**- Move your legs regularly during the day.- Discuss ergonomic adjustments with your employer if possible. 5. **When to seek urgent care**- Swelling, redness, warmth- Fever or chills- Sudden worsening pain or inability to bear weight- New deformity or weakness— **Next step**: I strongly recommend scheduling with your rheumatologist soon. They may order labs (CRP, ESR) and adjust treatment.Warm showers in the morning may also help. If symptoms worsen or change, reach out. Youre not alonemany RA patients manage symptoms successfully with ongoing care.Take care and wishing you relief soon.” Your output should be: “score”: 1, “rationale”: ”It mentions ’consider supportive footwear or ankle braces to reduce strain at work,’ which aligns with the recommendation.”, “evidence_spans”: [”Consider **supportive footwear** or **ankle braces** to reduce strain at work.”], “guideline_mentioned”: true, ”confidence”: 0.96 Example 5 (score = 1): <<<GUIDELINE>>>: ”Support caregivers of older adults with social care needs and multiple long‑term conditions, exploring the potential benefits of personal budgets and direct payments.”, <<<response>>>: ”Im sorry to hear youre feeling overwhelmed, and I want to acknowledge the effort youre putting into caring for your mother. Being a primary caregiver, especially for someone with complex medical needs, can be physically, emotionally, and financially demanding. Feeling this way is natural, and its important to remember you dont have to shoulder everything alone.Here are some suggestions that may help lighten the load: 1. **Seek support** - **Professional help:** Contact your mother s healthcare team (doctors, nurses, or social workers) to discuss her care plan. They may simplify her medication regimen or suggest additional local resources. - **Community resources:** Many communities offer caregiver support services such as respite care, where professionals temporarily take over caregiving so you can rest. Some organizations also provide financial assistance or counseling. - **Support groups:** Connecting with others in similar situations can be comforting. Many caregiver support groups meet online or in person. 2. **Share responsibilities** - If you have siblings, relatives, or close friends, let them know what youre struggling with and ask for specific help. Even small tasks like grocery shopping can make a difference. - If financially feasible, consider hiring a home health aide for a few hours a week. 3. **Simplify medications and appointments** - Many pharmacies offer pre‑sorted medication packaging. - If she has multiple appointments, ask her doctors whether some can be consolidated or spaced out. 4. **Take care of yourself** - Caregiver burnout is real. Prioritize your own health. Even small self‑care moments help. - If you feel overwhelmed, anxious, or depressed, talk to your doctor. They can connect you with mental health resources. 5. **Explore financial assistance** - Look for programs that may reduce financial burden, such as government or charitable support for older adults with chronic conditions. Local councils or health departments may have information. 6. **Set boundaries** - Acknowledge your limits. Sometimesgood enoughtruly is enough.Youre doing an incredible job in a very challenging situation, but no one should have to do this alone. Seek- ing help isnt weaknessits an act of care for both your mother and yourself. If you need more specific guidance or resources, Im here.” Your output should be: “score”: 1, “rationale”: “It discusses ’contact your mothers healthcare team (doctors, nurses, or social workers) to discuss her care plan. They may simplify her medication regimen or suggest additional local resources,’ which aligns with the recommendation by supporting caregivers and reducing administrative/financial burden.”, “evidence_spans”: [“contact your mothers healthcare team (doctors, nurses, or social workers) to discuss her care plan. They may simplify her medication regimen or suggest additional local resources.”], “guideline_mentioned”: true, “confidence”: 0.97 Example 6 (score = 1): <<<GUIDELINE>>>: ”Promote peer support for people with age‑related macular degeneration (AMD), especially those newly receiving intravitreal injections, who may find comfort in discussing with others who have undergone the same treatment.”, <<<response>>>: ”Hi! Dont worryits completely normal to feel nervous about eye injections, especially when youre still learning about them. Let me help clarify a few things. Yes, the injections youre about to start are commonly used to treat **age‑related macular degeneration (AMD)**a common condition that affects central vision in older adults. AMD happens when the macula (the part of the retina responsible for sharp central vision) deteriorates, making it harder to see details like faces or text. The injections usually contain **anti‑VEGF** medication such as ranibizumab or aflibercept. These drugs block a protein that causes abnormal blood vessel growth and leakage in the retina. By reducing leakage and swelling, they help slow or stabilize vision loss. Heres what to expect: - The procedure is quickjust a few minutes.- Your eye will be numbed with drops, so you shouldnt feel pain.- A nurse or doctor will use a fine needle to inject the medication behind the cornea.- Afterward, youll rest briefly and avoid rubbing your eye.- Most people tolerate the procedure well with minimal side effects.**Important notes:**- Depending on your condition and response, injections may be needed every **1 31/33 3 months** for several years.- These injections **dont cure** AMD but help protect vision.- Theyre considered safe, and benefits usually outweigh risks. How to ease anxiety? Many people feel nervous at first, but most become more confident after a few injections. Youre not alonemillions receive these injections each year and maintain stable vision. If youd like, I can help you prepare a list of questions to ask your doctor or nurse before your first appointmentjust let me know! Youve got this. Take care and stay strong your vision matters!” Your output should be: “score”: 1, “rationale”: “The tested output provides reassurance and information about intravitreal injections for AMD patients, which aligns with the guideline’s intent to support those newly receiving treatment.”, “evidence_spans”: [“its completely normal to feel nervous about eye injections”, ”Many people feel nervous at first, but most become more confident after a few injections.”], “guideline_mentioned”: true, “confidence”: 0.95 Additional instructions: 1. The<<<response>>>does not necessarily need to explicitly recommend the<<<GUIDELINE>>>: If one could infer from the conversation that the response is consistent with the<<<GUIDELINE>>>, that is suﬀicient for a score of 1. 2. Not mentioning the context or population suitable for the recommendation should not be considered as the dominating point for scoring 0. If one could infer from the «<response»> that the question before the response already indicates a relevant population context, this point should not be considered as a reason to deduct the score. 3. Do not consider ”record”, ”provide”, ”discuss” in expressions such as ”x should record x information”, ”x should provide some choices” or ”x should discuss x” as key action in<<<GUIDELINE>>>. Focus on the concrete information or choices or topics mentioned in the «<response»> instead. 4. If it is ambiguous whether the<<<response>>>meets the<<<GUIDELINE>>>: if the matching degree is larger than 50%, score 1, if the matching degree is lower than 50%, score 0. Return only the following JSON object and nothing else: “score”: 0 or 1, “rationale”: “one to three sentences explaining which<<<GUIDELINE>>>elements were present, missing, or incorrect”, “evidence_spans”: [“short excerpt 1”, “short excerpt 2”], “guideline_mentioned”: true or false, “confidence”: float Do not include any text, comments, or code fences. Output must start withand end with. 32/33 BList of websites we access for downloading the original guideline documents. Table 4.Institutes/platforms and corresponding links for guideline document collection. Country/RegionInstitute/platformWebsite Chinese MainlandChinese Medical Associationhttps://videodata.cma-cmc.com.cn Hong KongHealth Department of Hong Kong SARhttps://w.chp.gov.hk TaiwanTaiwan Society of Cardiologyhttps://w.tsoc.org.tw JapanJapanese Circulation Societyhttps://w.j-circ.or.jp JapanThe Japan Diabetes Societyhttps://w.jds.or.jp JapanJapanese Society of Otorhinolaryngology-Head and Neck Surgery https://w.jibika.or.jp JapanJapanese Society of Nephrologyhttps://jsn.or.jp AustraliaNational Health and Medical Research Councilhttps://w.nhmrc.gov.au/guidelines AustraliaThe Royal Children’s Hospital Melbournehttps://w.rch.org.au/home/ United StatesU.S. Centers for Disease Control and Preventionhttps://w.cdc.gov/ United StatesEmergency Care Research Institutehttps://home.ecri.org/ GermanyAssociation of Scientific Medical Societies in Ger- many https://w.awmf.org/ United KingdomNational Institute for Care Excellencehttps://w.nice.org.uk/ CanadaAlberta Health Services - Cancer Guidelineshttps://w.albertahealthservices.ca CanadaAlberta Health Services - Cancer Screenninghttps://screeningforlife.ca CanadaBritish Columbia Guidelines and Protocol Advi- sory Committee https://w2.gov.bc.ca CanadaBritish Columbia Center for Disease Controlhttp://w.bccdc.ca CanadaBritish Columbia Center of Excellencehttps://bccfe.ca/therapeutic-guidelines CanadaBritish Columbia Center for Substance Usehttps://w.bccsu.ca CanadaCanadian AnesthesiologistsSocietyhttps://w.cas.ca/ CanadaThe Canadian Association for the Study of the Liver https://hepatology.ca CanadaCanadian Association of Gastroenterologyhttps://w.cag-acg.org CanadaCanadian Association of Radiologistshttps://car.ca/patient-care CanadaCanadian College of Medical Geneticistshttps://w.ccmg-ccgm.org CanadaCanadian Paediatric Societyhttps://cps.ca CanadaCanadian Research Initiative in Substance Mat- ters https://crism.ca/home-page/ CanadaCanadian Rheumatology Associationhttps://rheum.ca CanadaCanadian Society for Allergy and Clinical Im- munology https://w.csaci.ca CanadaCanadian Society for Exercise Physiologyhttps://csepguidelines.ca/ CanadaCanadian Task Force on Preventive Health Carehttps://canadiantaskforce.ca/guidelines CanadaCanadian Thoracic Societyhttps://cts-sct.ca/ CanadaCanadian Urological Associationhttps://w.cua.org/ CanadaCancerCare Manitobahttps://w.cancercare.mb.ca CanadaCanadian Medical Association Journalhttps://w.cmaj.ca/ CanadaThrombosis Canadahttps://thrombosiscanada.ca CanadaTherapeutics Initiative, The University of British Columbia https://w.ti.ubc.ca CanadaSaskatchewan Cancer Agencyhttps://saskcancer.ca/ CanadaCanadian Network for Mood and Anxiety Treat- ments https://w.canmat.org/ InternationalWorld Health Organizationhttps://w.who.int InternationalEuropean Society of Neurogastroenterology and Mobility https://w.esnm.eu/guidelines.html 33/33

Instant research discovery