Paper deep dive
An artificial intelligence framework for end-to-end rare disease phenotyping from clinical notes using large language models
Cathy Shyr, Yan Hu, Rory J. Tinker, Thomas A. Cassini, Kevin W. Byram, Rizwan Hamid, Daniel V. Fabbri, Adam Wright, Josh F. Peterson, Lisa Bastarache, Hua Xu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 7/20/2026, 2:52:15 PM
Summary
The paper introduces RARE-PHENIX, an end-to-end artificial intelligence framework for rare disease phenotyping from unstructured clinical notes. It integrates large language models for phenotype extraction, retrieval-augmented generation for standardization to Human Phenotype Ontology (HPO) terms, and a supervised ranking model for prioritizing diagnostically informative phenotypes. Trained on data from 2,671 patients across 11 Undiagnosed Diseases Network (UDN) sites, RARE-PHENIX was externally validated on 16,357 clinical notes from Vanderbilt University Medical Center, outperforming the state-of-the-art baseline PhenoBERT in ontology-based similarity and precision-recall-F1 metrics.
Entities (10)
Relation Signals (8)
RARE-PHENIX → uses → Human Phenotype Ontology
confidence 98% · standardizing them to Human Phenotype Ontology (HPO) terms
RARE-PHENIX → outperforms → PhenoBERT
confidence 95% · RARE-PHENIX consistently outperformed a state-of-the-art deep learning baseline (PhenoBERT) across ontology-based similarity and precision-recall-F1 metrics
RARE-PHENIX → trainedon → Undiagnosed Diseases Network
confidence 92% · We trained RARE-PHENIX using data from 2,671 patients across 11 Undiagnosed Diseases Network clinical sites
RARE-PHENIX → usesmethod → Retrieval-Augmented Generation
confidence 90% · standardizing them to HPO terms using retrieval-augmented generation
RARE-PHENIX → validatedon → Vanderbilt University Medical Center
confidence 90% · externally validated it on 16,357 real-world clinical notes from Vanderbilt University Medical Center
RARE-PHENIX → usesmodel → Llama
confidence 85% · We selected LLaMA for its strong performance... LLaMA models were fine-tuned using PEFT
RARE-PHENIX → usesmodel → ChatGPT-4o
confidence 85% · Few-shot prompting was performed using a secure instance of Azure OpenAI’s ChatGPT-4o
RARE-PHENIX → usesalgorithm → XGBoost
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Phenotyping is fundamental to rare disease diagnosis, but manual curation of structured phenotypes from clinical notes is labor-intensive and difficult to scale. Existing artificial intelligence approaches typically optimize individual components of phenotyping but do not operationalize the full clinical workflow of extracting features from clinical text, standardizing them to Human Phenotype Ontology (HPO) terms, and prioritizing diagnostically informative HPO terms. We developed RARE-PHENIX, an end-to-end AI framework for rare disease phenotyping that integrates large language model-based phenotype extraction, ontology-grounded standardization to HPO terms, and supervised ranking of diagnostically informative phenotypes. We trained RARE-PHENIX using data from 2,671 patients across 11 Undiagnosed Diseases Network clinical sites, and externally validated it on 16,357 real-world clinical notes from Vanderbilt University Medical Center. Using clinician-curated HPO terms as the gold standard, RARE-PHENIX consistently outperformed a state-of-the-art deep learning baseline (PhenoBERT) across ontology-based similarity and precision-recall-F1 metrics in end-to-end evaluation (i.e., ontology-based similarity of 0.70 vs. 0.58). Ablation analyses demonstrated performance improvements with the addition of each module in RARE-PHENIX (extraction, standardization, and prioritization), supporting the value of modeling the full clinical phenotyping workflow. By modeling phenotyping as a clinically aligned workflow rather than a single extraction task, RARE-PHENIX provides structured, ranked phenotypes that are more concordant with clinician curation and has the potential to support human-in-the-loop rare disease diagnosis in real-world settings.
Tags
Links
- Source: https://arxiv.org/abs/2602.20324v1
- Canonical: https://arxiv.org/abs/2602.20324v1
Trouble viewing inline? Open PDF directly →
Full Text
60,238 characters extracted from source content.
Expand or collapse full text
An artificial intelligence framework for end-to-end rare disease phenotyping from clinical notes using large language models Cathy Shyr 1,2,3* , Yan Hu 4 , Rory J. Tinker 5 , Thomas A. Cassini 3 , Kevin W. Byram 6 , Rizwan Hamid 3 , Daniel V. Fabbri 1,7 , Adam Wright 1,6 , Josh F. Peterson 1,6 , Lisa Bastarache †1 , Hua Xu †8 1* Department of Biomedical Informatics, Vanderbilt University Medical Center, 2525 West End Avenue, Nashville, 37203, TN, USA. 2 Department of Biostatistics, Vanderbilt University Medical Center, 2525 West End Avenue, Nashville, 37203, TN, USA. 3 Department of Pediatrics, Vanderbilt University Medical Center, 2200 Children’s Way, Nashville, 37232, TN, USA. 4 McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston, 7000 Fannin St #600, Houston, 77030, TX, USA. 5 Department of Medical Genetics and Genomics, Icahn School of Medicine at Mount Sinai, 1 Gustave L. Levy Pl, New York, 10028, New York, USA. 6 Department of Medicine, Vanderbilt University Medical Center, 1161 21st Ave S, Nashville, 37232, TN, USA. 7 Department of Computer Science, Vanderbilt University, 1400 18th Avenue S, Nashville, 37212, TN, USA. 8 Department of Biomedical Informatics and Data Science, Yale School of Medicine, 101 College Street, New Haven, 06520, CT, USA. *Corresponding author(s). E-mail(s): cathy.shyr@vumc.org; Contributing authors: yanhu.yhu@gmail.com; rory.tinker@mssm.edu; thomas.a.cassini@vumc.org; kevin.byram@vumc.org; rizwan.hamid@vumc.org; daniel.fabbri@vumc.org; adam.wright@vumc.org; josh.peterson@vumc.org; lisa.bastarache@vumc.org; hua.xu@yale.edu; 1 arXiv:2602.20324v1 [cs.AI] 23 Feb 2026 † Equal contribution. Abstract Phenotyping is fundamental to rare disease diagnosis, but manual curation of structured phenotypes from clinical notes is labor-intensive and difficult to scale. Existing artificial intelligence approaches typically optimize individual compo- nents of phenotyping but do not operationalize the full clinical workflow of extracting features from clinical text, standardizing them to Human Phenotype Ontology (HPO) terms, and prioritizing diagnostically informative HPO terms. We developed RARE-PHENIX, an end-to-end AI framework for rare disease phenotyping that integrates large language model-based phenotype extraction, ontology-grounded standardization to HPO terms, and supervised ranking of diagnostically informative phenotypes. We trained RARE-PHENIX using data from 2,671 patients across 11 Undiagnosed Diseases Network clinical sites, and externally validated it on 16,357 real-world clinical notes from Vanderbilt University Medical Center. Using clinician-curated HPO terms as the gold stan- dard, RARE-PHENIX consistently outperformed a state-of-the-art deep learning baseline (PhenoBERT) across ontology-based similarity and precision-recall-F1 metrics in end-to-end evaluation (i.e., ontology-based similarity of 0.70 vs. 0.58). Ablation analyses demonstrated performance improvements with the addition of each module in RARE-PHENIX (extraction, standardization, and prioritiza- tion), supporting the value of modeling the full clinical phenotyping workflow. By modeling phenotyping as a clinically aligned workflow rather than a single extraction task, RARE-PHENIX provides structured, ranked phenotypes that are more concordant with clinician curation and has the potential to support human-in-the-loop rare disease diagnosis in real-world settings. Keywords: rare disease, artificial intelligence, large language model, phenotyping, clinical notes 1 Introduction Rare diseases affect over 300 million individuals worldwide and are a major cause of chronic illness, disability, and premature mortality [1]. While collectively common, rare diseases remain challenging to diagnose due to their low individual disease preva- lence, limited clinician familiarity, and clinical heterogeneity [1–4]. As a result, many rare disease patients undergo prolonged diagnostic odysseys characterized by diag- nostic delays averaging four to eight years, frequent misdiagnoses, and unnecessary testing [5–8]. Diagnostic odysseys have significant medical, psychosocial, and economic consequences for patients and families, resulting in irreversible disease progression, emotional turmoil, and avoidable healthcare expenditures [9–12]. Phenotyping, the precise and systematic characterization of a patient’s clinical fea- tures, is fundamental to rare disease diagnosis and management. In practice, clinicians phenotype rare disease patients through a structured, multi-step clinical workflow: extract clinical features from medical records, standardize these features to Human 2 Phenotype Ontology (HPO) terms, and prioritize those that are diagnostically infor- mative [13]. The resulting curated HPO terms serve as the foundation for diagnostic decision-making, including candidate gene and variant prioritization, iterative genomic analyses, and integration with additional clinical and functional evidence [14–18]. Though the widespread adoption of electronic health records (EHRs) provides new opportunities to support rare disease phenotyping through secondary data use, rare disease phenotypes are poorly captured in structured EHR fields because they are often atypical, heterogeneous, and inadequately represented by standard diagnostic or billing codes [19, 20]. Instead, they are often embedded in unstructured clinical notes, where manual curation is labor-intensive and difficult to scale. Recent advances in artificial intelligence (AI) and large language models (LLMs) create new opportunities to support rare disease phenotyping at scale. Prior work has demonstrated the feasibility of these tools to extract rare disease phenotypes from unstructured clinical text, map clinical descriptions to ontologies, and support downstream diagnostic decision-making [21–36]. Early phenotyping approaches pri- marily relied on rule-based methods to identify rare disease phenotypes from EHRs [21, 22, 37]. Subsequent work focused on supervised machine learning (ML) and deep learning models for phenotype extraction and normalization [23, 24, 38]. More recent studies demonstrated that LLMs, particularly when augmented with domain knowl- edge, ontologies, or retrieval mechanisms, can generate phenotypic summaries, rank candidate diseases or genes, or assist with rare disease diagnosis [28, 29, 39, 40, 40– 44]. Benchmarking studies further suggest that LLMs can achieve comparable or exceed clinician-level performance on rare disease reasoning tasks when provided with carefully designed prompts or external knowledge sources [45–47]. However, most existing approaches optimize individual components of rare disease phenotyping in isolation rather than supporting the end-to-end clinical workflow. In clinical practice, the challenge is not only to extract phenotypes, but also prioritize the subset that meaningfully narrows the differential diagnosis. Rare disease patients often present with common or non-specific phenotypes (e.g., fatigue) that are less diagnos- tically informative. Therefore, a phenotyping tool may have high accuracy but remain poorly suited for diagnosis because informative phenotypes are diluted by a long list of true but diagnostically non-related terms. Many current methods focus on pheno- type extraction or normalization without distinguishing which phenotypes are most relevant for diagnosis [23–25, 27, 48–52]. Others perform disease or gene prioritization using phenotypes as input, focusing on downstream interpretation rather than phe- notype extraction from unstructured clinical notes [43]. Recent LLM-based diagnostic systems and retrieval-augmented frameworks primarily focus on diagnostic decision support, rather than modeling the end-to-end phenotyping workflow performed by clinicians [26, 39, 42, 45, 46]. As a result, to our knowledge, no existing approach pro- vides an end-to-end rare disease phenotyping pipeline aligned with real-world clinical practice. To address this gap, we developed and externally validated RARE-PHENIX (RARE disease PHENotyping with Intelligent eXtraction), a modular AI framework designed to perform end-to-end rare disease phenotyping from unstructured clinical notes to support diagnosis. RARE-PHENIX aligns with real-world clinical workflows 3 by 1) extracting rare disease phenotypes using LLMs with few-shot prompting and instruction fine tuning; 2) standardizing them to HPO terms using retrieval-augmented generation; and 3) prioritizing HPO terms using a supervised ranking model trained to distinguish common, non-specific phenotypes from those that are diagnostically informative. We trained RARE-PHENIX on rare disease-specific corpora, includ- ing expert-annotated documents from the National Organization of Rare Disorders database and synthetic clinical text containing clinician-curated HPO terms across 11 Undiagnosed Diseases Network (UDN) clinical sites. We then externally validated RARE-PHENIX on 16,357 real-world clinical notes from patients at Vanderbilt Uni- versity Medical Center (VUMC), which were not included among the training sites. Across evaluations, RARE-PHENIX outperformed PhenoBERT [53], a state-of-the-art system for identifying HPO terms from unstructured text, at extracting diagnostically relevant rare disease phenotypes. By aligning AI-based phenotyping with real-world clinical workflows, RARE-PHENIX enables end-to-end rare disease phenotyping from clinical notes to support diagnosis. 2 Methods 2.1 Study Design and Population This is a retrospective, multi-site cohort study with model development and external validation. The study included N = 2, 814 patients evaluated at the UDN between October 4, 2010 and April 25, 2024 across 12 clinical sites: University of California, Los Angeles, Baylor College of Medicine, Duke University, Stanford University, Harvard- affiliated Hospitals, NIH Undiagnosed Diseases Program, Washington University in St. Louis, Children’s Hospital of Philadelphia and University of Pennsylvania, University of Washington and Seattle Children’s Hospital, University of Utah, and University of Miami, and VUMC. In general, patients are accepted to the UDN if they have rare or undiagnosed diseases resulting in multisystem dysfunction, functional impairment, or symptom severity that substantially affects quality of life. Eligibility requires objective or measurable findings from previous testing and examinations, and the absence of a diagnosis after prior workup. For model development, we used data from 11 UDN sites (N = 2, 671). The data from VUMC (N = 143) were held out as an external test cohort and was not used at any stage of model training or selection. We restricted our analysis to patients who have at least one UDN clinician-curated HPO term. All data partitioning was performed at the patient level to prevent data leakage. This study was approved by the central UDN Institutional Review Board (IRB #172005) and VUMC Institutional Review Board (IRB #222249). All protected health information was processed within secure, institutionally approved environments in accordance with data governance policies. Study reporting followed the TRIPOD-AI checklist [54]. 2.2 Overview of RARE-PHENIX RARE-PHENIX is an end-to-end AI framework that automates phenotype extraction from clinical notes, standardizes extracted features to HPO terms, and prioritizes 4 diagnostically informative HPO terms. The framework consists of three sequential modules (Fig. 1). Fig. 1: Overview of RARE-PHENIX. RARE-PHENIX is an end-to-end AI system for automating the extraction, standardization, and prioritization of rare disease phenotypes from unstructured clinical text. This system consists of three modules for 1) extracting rare disease features from clinical notes with large language models (LLMs); 2) standardizing these features to structured Human Ontology Phenotype (HPO) terms using retrieval-augmented generation; and 3) prioritize diagnostically informative HPO terms using a supervised ranking model. LLMs include LLaMA-2-chat (7b, 13b, and 70b), LLaMA-3-instruct (8b, and 70b) LLaMA-3.1-instruct (8b and 70b), LLaMA-3.2-instruct (1b and 3b), LLaMA-3.3-instruct (70b), and a secure instance of Azure OpenAI’s ChatGPT-4o (v2024-06-01) provisioned for handling protected health information in accordance with institutional data governance policies. Identify and extract rare disease features Standardize features to Human Phenotype Ontology (HPO) terms 2 Clinical notes LLM for extracting rare disease features Models Annotated Corpora Instruction Fine Tuning 1. NORD Database 2. Synthetic LLM-generated notes for patients across 11 UDN sites (2,671 Documents)(832 Documents) 1 Retrieval-augmented generation LLaMA ChatGPT 4o Method Few-shot Prompting Method LLaMA ChatGPT 4o Models List of HPO terms Vector store List of extracted features Prioritize HPO terms for diagnosis 3 •Near sighted •Scoliosis •Fever Dictionary of HPO terms •HP:0000545 Myopia •HP:0002650 Scoliosis •HP:0001945 Fever Feature Engineering with L1 and L2 Regularization Method Features Ontology-derived Features Information criterion Number of gene associations Number of disease associations ... Myopia4.18460599... Scoliosis2.2011541643... Fever3.88445552... XGBoost Pairwise Ranking Model Model Ranked HPO terms 1.HP:0002650 Scoliosis 2.HP:0000545 Myopia 3.HP:0001945 Fever Input Output 5 2.3 Module 1: Phenotype Extraction from Clinical Notes The objective of Module 1 is to identify and extract rare disease phenotypes from unstructured clinical notes. We developed two complementary approaches: parameter- efficient instruction fine-tuning of open-source LLMs and few-shot prompting of a closed-source model, enabling evaluation under both deployable and API-based settings. 2.3.1 Instruction Fine-Tuning Instruction fine-tuning adapts a pre-trained LLM to a domain-specific task using labeled instruction–output pairs [55]. We fine-tuned 10 LLaMA models of various sizes: LLaMA-2-chat (7b, 13b, and 70b), LLaMA-3-instruct (8b, and 70b) LLaMA-3.1-instruct (8b and 70b), LLaMA-3.2-instruct (1b and 3b),and LLaMA-3.3-instruct (70b) [56]. We selected LLaMA for its strong performance on biomedical and clinical natural language processing (NLP) tasks, open-source availability, and compatibility with parameter efficient fine tuning (PEFT) [57, 58]. LLaMA models were fine-tuned using PEFT. Each training instance consisted of an input text and a corresponding HTML-annotated version in which rare disease phenotypes were wrapped in <span> tags. Instruction fine-tuning was performed using the HuggingFace Transformers and TRL framework with parameter-efficient fine-tuning (PEFT) [56]. Base models were fine-tuned using QLoRA, with 4-bit NF4 weight quantization and bfloat16 computation. Low-Rank Adaptation (LoRA) was applied with rank r = 16, scaling factor 64, and dropout 0.05 to the linear projec- tion layers, reducing computational requirements [59]. Training was conducted using TRL’s SFTTrainer with a maximum sequence length of 1,500 tokens for two epochs, a learning rate of 2× 10 −4 , and a warm-up ratio of 0.05. Fine-tuning was performed using 3x80GB A100 NVIDIA GPUs. Fine-Tuning Corpora. We used two complementary corpora: 1. RareDis Corpus [60]. This corpus consists of 832 expert-annotated documents from the National Organization for Rare Disorders knowledgebase. Documents were annotated by experts for four entity types (i.e., rare disease, disease, symptom, and sign) with an inter-annotator agreement F1-score of 83.5%, indicating high annotation reliability. RareDis documents were converted into instruction-output pairs by wrapping the annotated rare disease entities in HTML span tags. 2. UDN Synthetic Clinical Narratives. To augment limited annotated data with clinically grounded narratives, we generated synthetic clinical text using clinician- curated HPO terms from the N = 2, 671 patients in the training cohort. We generated synthetic clinical text using a secure, institutionally approved instance of Azure OpenAI’s ChatGPT-4 turbo (v2024-04-09) with a temperature of zero for reproducibility. Each patient’s curated HPO term was incorporated verbatim in the synthetic clinical text and wrapped in HTML tags. The full prompt used to generate these documents is provided in the Supplementary Materials. 6 2.3.2 Few-Shot Prompting Approach Few-shot prompting was performed using a secure instance of Azure OpenAI’s ChatGPT-4o (v2024-06-01) with temperature set to 0. Prompts included a task state- ment, markup guide, detailed definition of rare disease phenotypes, a set of labeled examples, and the input text. The model was instructed to output an HTML- annotated version of the text, with rare disease phenotypes wrapped in <span> tags. We provide the full prompt in Supplementary Table S1. 2.4 Module 2: Standardization to HPO Terms The objective of Module 2 is to standardize phenotype strings to structured HPO terms (i.e., “near sighted” → HP:0000545 Myopia) using retrieval-augmented generation (RAG) [61]. This module produces standardized, interoperable phenotype terms that can be used for downstream diagnostic workup (e.g., genomic analysis). RAG combines semantic retrieval with LLM-based generation to ground model outputs in external knowledge such as the HPO. We chose RAG to address limita- tions of both retrieval-only methods, which may miss contextual nuance in clinical descriptions, and unconstrained LLMs, which may hallucinate invalid HPO terms. We constructed a vector database of HPO terms, where each term was represented as a text node and embedded using a pre-trained sentence embedding model (i.e., text-embedding-3-small). The embeddings were indexed in a vector store using cosine similarity. For each phenotype string, Module 2 performs the following: 1. Semantic retrieval: The phenotype string is embedded and used to retrieve the top- 10 semantically similar HPO terms from the vector store based on cosine similarity. 2. Ontology-grounded generation: The retrieved HPO nodes are provided to an LLM, which is prompted to select the most appropriate HPO term based on contextual relevance and the associated ontology definition. If no suitable match is found, the model outputs “none.” 2.5 Module 3: Prioritization of Diagnostically Informative Phenotypes The objective of Module 3 is to operationalize the prioritization of diagnostically informative phenotypes as a supervised learning-to-rank task. Phenotypes are defined as informative if they were uncommon, clinically specific, and associated with a limited number of diseases or genes, making them more effective for distinguishing among candidate diagnoses. 2.5.1 Learning-to-Rank Formulation We formulated HPO term prioritization as a supervised learning-to-rank problem in which, for each patient, the model learns to assign higher relevance scores to clinician- curated HPO terms than non-clinician-curated terms. The training cohort included N = 2, 671 patients from 11 UDN clinical sites, excluding VUMC. For each patient, clinician-curated HPO terms were labeled as positives (label = 1). Because these HPO terms are not provided in a ranked order, we trained the learning-to-rank model to 7 rank them above non-clinician-curated terms, or negative HPO terms, within each patient. We constructed negative HPO terms (label = 0) for each patient, featuring a mix of difficult (very similar to the clinician-curated HPO terms), medium (somewhat similar), easy (not very similar), and implausible negatives (highly unrelated). Diffi- cult negatives were randomly sampled from sibling and cousin terms in the ontology, medium negatives from terms three to five edges away, easy negatives from distant ancestor or descendant terms, and implausible negatives from terms sharing no close ancestry (≤ 2 edges) with any positive term. Feature Engineering. Model features included patient-level predictors (i.e., age, sex, primary symptom category) and knowledgebase-derived annotations from Online Mendelian Inheritance in Man (OMIM) and Orphanet [62]. For each HPO term, features included information content, numbers and fractions of associated genes and diseases, and OMIM- and Orphanet-based inverse document frequencies (IDF). IDFs were calculated as the negative logarithm of the fraction of associated diseases. Collectively, these knowledgebase-derived annotations capture phenotype rarity and specificity, with higher values indicating greater diagnostic informativeness. Model Training and Selection We randomly partitioned the training data at the patient level into an 80:20 training and validation set. Using the training set, we trained multiple supervised models, including gradient-boosted decision tree learning- to-rank approaches (XGBoost with a pairwise ranking objective, LightGBM with a LambdaRank, and CatBoost with YetiRankPairwise) as well as a logistic regression ranker baseline [63–65]. Hyperparameters for the gradient boosted models, including learning rate, tree depth, and ℓ 1 , ℓ 2 regularization, were tuned using early stopping based on mean average precision (MAP) on the validation set. Model performance was also validated using MAP at 30 (MAP@30), computed at the patient level. The same validation split and metric were applied uniformly across all models to ensure fair comparison. The model with the highest validation MAP@30 was selected as the final ranking model and applied to the held-out, external test cohort for evaluation. 2.6 Evaluation Strategy and Baseline Comparator Evaluation Metrics. We evaluated RARE-PHENIX on an external cohort of 143 UDN patients at VUMC, who had a total of 16,357 clinical notes recorded prior to the UDN workup. The primary outcome was phenotypic concordance with clinician-curated HPO terms, measured using ontology-based semantic similarity (Lin measure) [66]. Secondary outcomes included term-level phenotyping performance as measured by precision, recall, and F1 score. Clinical Notes Pre-processing. To prevent data leakage, we excluded clinical notes recorded after the start of UDN’s evaluation. Notes were cleaned to remove admin- istrative content (e.g., scheduling notes) and segmented into chunks of at most 4,026 characters while respecting sentence boundaries. This chunk size was chosen to accommodate the smallest context window among the LLMs used. Baseline Comparator. We compared RARE-PHENIX to PhenoBERT, a state- of-the-art deep learning system for HPO term extraction [53]. PhenoBERT uses a two-stage architecture: 1) a hierarchical convolutional neural network for candidate HPO term selection, followed by 2) a BERT-based model for candidate evaluation. 8 In published benchmarks, PhenoBERT outperformed multiple dictionary-based sys- tems and prior deep learning methods, including NeuralCR and PhenoTagger, and achieved state-of-the-art performance across both PubMed abstracts and real-world clinical notes, making it an ideal baseline comparator [67, 68]. 3 Results This study included N = 2,814 patients evaluated at the UDN. RARE-PHENIX was developed using data from 11 clinical sites (N = 2,671, 94.9%) and externally validated on an independent cohort from VUMC (N = 143, 5.1%), which was not used at any stage of model training or selection. The cohort was primarily pediatric, with a median age of 12 years (IQR: 5–31), and had a median of 15 clinician-curated HPO terms per patient (IQR: 9–25), reflecting substantial phenotypic complexity (Table 1). External validation was performed on 16,357 clinical notes, the majority of which were progress notes (68%), followed by consultation notes (13%) and procedure or operative notes (6%) (Table 2). Patients had a median of 18 notes each (IQR: 9–113.5), with a median note length of 1,792 characters (IQR: 573–4,297). Table 1: Overview of Undiagnosed Diseases Network (UDN) data. HPO = Human Phenotype Ontology. IQR = interquartile range. Clinical SiteNumber of patients, n (%) Age, median (IQR) Number of HPO terms, median (IQR) Training Cohorts UCLA248 (8.8%)11.0 (4.2–23.8)22.0 (13.0–38.2) Baylor358 (12.7%)8.0 (3.0–18.0)16.0 (9.0–25.0) Duke302 (10.7%)7.0 (3.0–14.5)21.5 (12.0–31.0) Stanford325 (11.5%)11.0 (5.0–29.8)10.0 (6.0–17.0) Harvard-affiliate240 (8.5%)12.5 (4.0–29.0)13.0 (7.0–22.0) NIH636 (22.6%)25.0 (10.0–46.0)21.0 (12.0–35.0) WUSTL122 (4.3%)10.0 (3.0–20.5)12.0 (8.2–17.0) CHOP-UPenn134 (4.8%)14.0 (4.0–35.0)8.0 (5.2–11.0) UW-SCH154 (5.5%)11.0 (4.0–34.0)11.0 (7.0–17.0) Utah65 (2.3%)16.0 (4.0–37.0)7.0 (5.0–11.0) Miami87 (3.1%)11.0 (4.0–20.0)14.0 (9.0–20.0) Evaluation Cohort Vanderbilt143 (5.1%)8.0 (3.0–20.5)17.0 (12.0–23.0) Overall2814 (100.0%)12.0 (5.0–31.0)15.0 (9.0–25.0) 3.1 End-to-End Rare Disease Phenotyping Performance We evaluated the end-to-end performance of RARE-PHENIX (phenotype extraction, HPO standardization, and prioritization) on the external validation cohort and com- pared it to PhenoBERT using clinician-curated HPO terms as the ground truth. Because PhenoBERT does not rank HPO terms by design, we passed PhenoBERT- extracted HPO terms through the same prioritization module (Module 3) to generate top-k lists for a head-to-head comparison. Within Module 3, XGBoost achieved the 9 Table 2: Summary of clinical notes in the external evaluation cohort. IQR = interquartile range. Clinical note typeN (%) Number of notes per patient, median (IQR) Number of characters per note, median (IQR) Progress11083 (67.8%)16.0 (8.0–71.5)2326.0 (819.5–5008.0) Consultations2074 (12.7%)9.0 (2.5–27.0)901.0 (506.2–2200.5) Procedures / Operative926 (5.7%)9.5 (4.0–17.8)942.0 (205.0–2189.2) Imaging712 (4.4%)4.0 (2.0–14.0)2276.5 (569.5–3376.2) Rehabilitation & Therapy298 (1.8%)4.0 (1.2–12.0)2993.5 (610.0–4137.8) Nutrition169 (1.0%)2.0 (1.0–14.0)349.0 (221.0–588.0) Letters & Correspondence166 (1.0%)2.0 (1.0–2.5)1710.5 (354.0–3395.8) Diagnostics (Non-Imaging)98 (0.6%)6.0 (3.0–13.0)1051.5 (301.0–2083.0) Discharge54 (0.3%)1.5 (1.0–3.0)4468.0 (2512.2–6008.5) Other777 (4.8%)6.5 (3.0–22.5)406.0 (181.0–707.0) Combined16357 (100.0%) 18.0 (9.0–113.5)1792.0 (573.0–4297.0) highest validation performance (MAP@30 = 0.85) and was selected as the final learning-to-rank model for external evaluation (Supplementary Table S3). Fig. 2 summarizes the end-to-end performance across top-k cutoffs (k = 10, 20, . . . , 50); for legibility, the figure displays only the top-performing LLM config- urations, with full results provided in Supplementary Table S2. Across all cutoffs and metrics, RARE-PHENIX consistently outperformed the PhenoBERT baseline. In particular, RARE-PHENIX achieved higher ontology-based semantic similarity to clinician-curated phenotypes, with the strongest similarity observed for LLaMA-2-70b across cutoffs (e.g., at k = 50, ∼0.70 for LLaMA-2-70b versus ∼0.58 for PhenoBERT), indicating improved concordance with clinician-curated phenotypes. Consistent with expected retrieval trade-offs, precision decreased as k increased while recall increased, reflecting that broader phenotype lists contained more clinician- curated terms at the cost of including additional non-curated terms. Notably, F1 improved sharply from k = 10 to k = 20 and then plateaued by approximately k = 30– 40, suggesting diminishing benefit from including additional terms beyond this range for balanced performance. Among the top-performing configurations, LLaMA-2-70b generally achieved the highest recall, F1, and ontology-based similarity across all cut- offs. 10 Fig. 2: End-to-end performance results of RARE-PHENIX and Phe- noBERT on the external validation cohort. For legibility, only the top- performing large language models are shown in the figure (i.e., ChatGPT-4o, LLaMA-2-70b, LLaMA-3-70b, LLaMA 3.1-70b) in addition to the baseline compara- tor (PhenoBERT) across top-k cutoffs. The end-to-end performance results of other RARE-PHENIX configurations are provided in Supplementary Table S2. RecallF1 Ontology−based Similarity ScorePrecision 10203040501020304050 0.2 0.3 0.4 0.5 0.2 0.3 0.4 0.5 0.45 0.50 0.55 0.60 0.65 0.70 0.2 0.3 0.4 0.5 0.6 Cutoff k Top−k Metrics Model Baseline: PhenoBERTRARE−PHENIX: ChatGPT 4oRARE−PHENIX: Llama 2 70b RARE−PHENIX: Llama 3 70bRARE−PHENIX: Llama 3.1 70b 3.2 Module-based Ablation Analysis of RARE-PHENIX Fig. 3 illustrates the module-based ablation analysis results for extraction alone (Module 1) and extraction followed by HPO standardization (Modules 1+2), without prioritization. Across model configurations, standardization (Module 2) consistently improved phenotype concordance with clinician-curated lists, with the largest gains 11 observed in precision and F1, indicating that mapping free-text phenotypes to HPO terms substantially reduces noise and yields more specific phenotype representations. Among all RARE-PHENIX configurations, LLaMA-2-70b achieved the high- est ontology-based similarity (0.76), followed by LLaMA-3-70b (0.75) and both LLaMA-3.1-70b and LLaMA-3.3-70b (0.73), compared to 0.70 for few-shot ChatGPT-4o and 0.64 for PhenoBERT. In general, PhenoBERT performed similarly to smaller LLMs (1-3B parameters), whereas larger models (70b) consistently achieved higher ontology-based similarity. Standardization yielded substantial improvements in precision across most LLMs (e.g., LLaMA-2-70b increased from 0.25 with Module 1 to 0.43 with Modules 1+2), translating into marked gains in F1 (from 0.34 to 0.50). In contrast, recall was rel- atively stable, with smaller gains after standardization (e.g., LLaMA-2-70b 0.70 to 0.71), consistent with Module 2 primarily improving specificity rather than simply increasing coverage. Notably, some smaller models achieved high recall with lower precision (e.g., LLaMA-3.2-1b recall 0.76 with precision 0.26), illustrating how phe- notype lists can recover many clinician-curated terms while remaining diluted by less informative or extraneous phenotypes. Together, these results identify ontology-based standardization as a key contributor to improved phenotype concordance. 12 Fig. 3: Module-based Ablation Analysis Results of RARE-PHENIX Across Extraction and Standardization Modules 0.70 0.71 0.76 0.72 0.73 0.71 0.67 0.65 0.73 0.75 0.66 0.64 0.73 0.68 0.71 0.67 0.71 0.70 0.76 0.70 0.74 0.71 0.70 0.67 0.38 0.40 0.43 0.40 0.42 0.41 0.26 0.38 0.33 0.42 0.42 0.16 0.47 0.47 0.50 0.46 0.50 0.48 0.35 0.45 0.42 0.50 0.49 0.23 RecallF1 Ontology−based Similarity ScorePrecision 0.00.20.40.60.80.00.10.20.30.40.5 0.00.20.40.60.80.00.10.20.30.4 Llama 3.3 70b Llama 3.2 3b Llama 3.2 1b Llama 3.1 70b Llama 3.1 8b Llama 3 70b Llama 3 8b Llama 2 70b Llama 2 13b Llama 2 7b ChatGPT−4o PhenoBERT Llama 3.3 70b Llama 3.2 3b Llama 3.2 1b Llama 3.1 70b Llama 3.1 8b Llama 3 70b Llama 3 8b Llama 2 70b Llama 2 13b Llama 2 7b ChatGPT−4o PhenoBERT Model family BaselineChatGPTLlama 2Llama 3 Llama 3.1Llama 3.2Llama 3.3 ModuleModule 1Module 1 + Module 2 3.3 Contribution of Phenotype Prioritization Module To assess the isolated contribution of the phenotype prioritization module (Module 3), we compared the ranked phenotype lists produced by RARE-PHENIX to a random ordering of the same extracted phenotypes. For each patient, the extracted HPO terms were randomly permuted 200 times, and performance was evaluated at top-k cutoffs (k = 10, 20, 30, 40, 50). Metrics were computed at the patient level with 95% bootstrap intervals across 1,000 bootstrap iterations. Improvements attributable to prioritization were quantified as the difference between prioritized and randomly ordered lists (∆ = prioritized using Module 3 − prioritized by random ordering). 13 Our results show that phenotype prioritization (Module 3) consistently improved performance across all models and cutoffs (Fig. 4). The largest gains were observed at lower k, where clinical decision-making typically focuses on a limited number of highly informative phenotypes. At k = 10, prioritization increased ontology-based similarity by approximately 0.06-0.09 and precision by 0.11-0.14, indicating that the highest-ranked phenotypes were substantially more diagnostically relevant than would be expected by chance. Improvements in recall and F1 score were also observed, with highest gains at intermediate cutoffs (k = 20–30). We provide the results with 95% bootstrap intervals for all models in Supplementary Table S4. The magnitude of the prioritization benefit decreased as k increased, which was expected because the difference between prioritized and random ordering diminishes when most extracted phenotypes are included. Importantly, the lower bound of the 95% bootstrap intervals remained above zero across all cutoffs and models, demon- strating that the observed gains were consistent at the patient level. Collectively, these results show that Module 3 systematically prioritizes diagnostically informative phenotypes that align with real-world clinician curation. 3.4 Systematic Error Analysis To characterize the systematic errors of RARE-PHENIX, we conducted a clinician- adjudicated systematic error analysis of false negatives (FNs) and false positives (FPs) across top-k cutoffs. RARE-PHENIX consistently produced fewer errors than Phe- noBERT, with a 29% relative reduction in mean FNs per patient and a 24% reduction in mean FPs per patient at k = 50 (Fig. 5). Manual review of 271 FN instances revealed that the majority (97%) were not due to extraction failures, but rather to the way phenotypes are documented in real-world clinical narratives. Most missed terms were expressed indirectly through linguistic variation or contextual description. For example, the HPO term failure to thrive was described in clinical notes as “severe growth retardation,” and short foot as “both feet appear to be small.” In several cases, the model extracted a semantically related or higher-level HPO concept. Additional FNs reflected limitations of clinical documen- tation, such as physical examination findings that were not explicitly recorded in the notes. Only 3% of FNs were attributable to true extraction errors in which clearly documented HPO terms were not identified by the model. FPs increased with larger k values for all methods, reflecting the expected trade-off between coverage and speci- ficity. Manual review revealed that FPs primarily occurred due to ontology granularity differences (i.e., extracting the parent term rather than the child term) and common, non-specific symptoms that were not included in clinician-curated lists (e.g., “nau- sea”). RARE-PHENIX maintained consistently lower FP counts than PhenoBERT across all cutoffs, indicating more precise phenotyping (Fig. 5). Collectively, these findings suggest that errors are driven primarily by the linguistic and structural char- acteristics of clinical documentation and ontology granularity, rather than extraction errors. 14 Fig. 4: Contribution of phenotype prioritization to diagnostic utility. Improvement in performance using the prioritization module (Module 3) relative to a random ordering of the same extracted phenotypes. For each patient, extracted HPO terms were randomly permuted 200 times, and performance was evaluated at top-k cutoffs (k = 10, 20, 30, 40, 50). Values represent the mean difference (∆ = ranking by Module 3− ranking by random ordering), and shaded regions indicate 95% bootstrap intervals obtained by resampling at the patient level. RecallF1 Ontology−based Similarity ScorePrecision 10203040501020304050 0.00 0.05 0.10 0.15 0.000 0.025 0.050 0.075 0.100 0.00 0.03 0.06 0.09 0.000 0.025 0.050 0.075 0.100 Cutoff k ∆ (Ranking by Module 3 − Ranking by Random Ordering) Model ChatGPT−4oLLaMA−2−70BLLaMA−3−70BLLaMA−3.1−70B 15 Fig. 5: Results of systematic error analysis. False negatives and false posi- tives of RARE-PHENIX with the best-performing large language model configurations (ChatGPT-4o, LLaMA-2-70b, LLaMA-3-70b, LLaMA-3.1-70b) and the baseline com- parator (PhenoBERT) at different top-k cutoffs (k = 10, 20, 30, 40, 50). False NegativesFalse Positives 10203040501020304050 10 20 30 10.0 12.5 15.0 17.5 Cutoff k Mean count per patient Model Baseline: PhenoBERTRARE−PHENIX: ChatGPT 4oRARE−PHENIX: Llama 2 70b RARE−PHENIX: Llama 3 70bRARE−PHENIX: Llama 3.1 70b 16 4 Discussion We developed and externally validated RARE-PHENIX, an end-to-end AI framework for rare disease phenotyping that directly models the real-world clinical workflow: extraction of phenotypes from unstructured notes, ontology-grounded standardization, and prioritization of diagnostically informative features. In a large external evaluation of 16,357 real-world clinical notes from UDN patients, RARE-PHENIX consistently outperformed a strong deep learning baseline (PhenoBERT [53]) across ontology-based similarity and precision–recall-F1 metrics. Importantly, the module-based ablation analyses demonstrate that these gains are not attributable to improved extraction alone, but to the combination of ontology-based standardization and prioritization of diagnostically informative phenotypes. Collectively, our findings demonstrate that operationalizing rare disease phenotyping as an end-to-end framework that reflects real-world clinical workflows, rather than as a single extraction task, substantially improves concordance with clinician-curated phenotypes. Our results highlight several important implications for the design of AI-based phenotyping tools in real-world clinical workflows. First, both RARE-PHENIX and PhenoBERT achieved higher recall than precision, indicating that these methods tend to capture broader phenotype sets than those curated manually. In clinical prac- tice, this tendency is desirable if diagnostically informative phenotypes are surfaced early to minimize the burden of additional clinical review. Our prioritization analysis demonstrates that, compared to random ranking of the same extracted phenotypes, RARE-PHENIX’s prioritization approach (Module 3) yields consistent, patient-level improvements across all metrics, with the largest gains observed earlier in the list. This result provides evidence that the prioritization module has the potential to improve the clinical utility of extracted phenotypes. Second, our ablation analysis identified ontology-based standardization as a major contributor to performance gains. Stan- dardizing free-text strings to HPO terms substantially increased precision and F1 while leaving recall largely unchanged. This suggests that a key barrier to effective com- putational phenotyping in the rare disease setting is normalization into a structured phenotype representation that can support downstream decision-making. Third, the comparable recall achieved by in-context learning with ChatGPT-4o relative to fine- tuned 70b models suggests that in-context learning may be a promising alternative in resource-constrained clinical settings where local model training is not feasible. RARE-PHENIX complements existing rare disease LLM-based systems by address- ing the bottleneck of extracting diagnostic informative phenotypes directly from clinical notes. Earlier EHR-based approaches primarily focused on retrieval, screening, or rule-based identification of rare disease patients rather than end-to-end pheno- typing workflows [21, 22]. Recent studies advanced phenotype extraction through supervised and weakly supervised pipelines, and LLM-based studies focused on opti- mizing prompting strategies, hybrid dictionary-LLM methods, and ontology grounding to improve extraction accuracy [23–25, 41]. In parallel, many LLM-based rare dis- ease systems prioritize diagnosis or gene and disease ranking, such as automated differential diagnosis pipelines [39]. By directly modeling the clinical workflow of phenotype extraction, standardization, and diagnostic prioritization, RARE-PHENIX complements existing extraction-based systems and provides a modular phenotyping 17 approach that can enhance downstream diagnostic frameworks, including differen- tial diagnosis systems, phenotype-to-gene prioritization pipelines, and multi-agent diagnostic workflows [26, 39, 43]. Our study has several strengths. First, we evaluated RARE-PHENIX using a large and heterogeneous corpus of real-world clinical notes from an external clinical site, enhancing the generalizability of our findings. To our knowledge, this is the first study to operationalize the full clinical workflow of rare disease phenotyping with compre- hensive ablation analyses of extraction, standardization, and prioritization modules. Another strength of RARE-PHENIX is its modular design. Because of this flexible design, the extraction and standardization modules can be deployed upstream of other systems that rely on structured HPO inputs. Moreover, the prioritization module can be applied to phenotypes generated by other extraction methods to support diagnostic decision-making. Our study also has limitations that should be considered. The UDN cohort is enriched for complex, multi-system cases and may not fully represent general genet- ics or sub-specialty patient populations with rare diseases. The strongest performance was achieved with large models (i.e., 70b parameters), which may be difficult to deploy in resource-constrained environments. Clinician-curated HPO terms were used as the ground truth in our evaluation; however, manual curation may be incomplete and sub- ject to inter-clinician variability, which may underestimate true model performance for phenotypes present in the clinical notes but not captured in the clinician-curated list. In addition, while RARE-PHENIX achieved favorable performance in retrospec- tive evaluation, its downstream impact on diagnostic accuracy, time to diagnosis, and clinician workload should be evaluated prospectively. In conclusion, RARE-PHENIX provides a clinically aligned, end-to-end frame- work for rare disease phenotyping that has the potential to improve the usability of phenotypes extracted from real-world clinical notes. By integrating extraction, ontology-based standardization, and prioritization of diagnostically informative phe- notypes into a unified pipeline, the framework has the potential to reduce manual curation burden and serve as a scalable phenotyping tool in rare disease diagnos- tic workflows. Prospective studies assessing effects on diagnostic efficiency, gene and variant prioritization, and clinician workflow will be essential to establish real-world clinical benefit and to guide responsible deployment in rare disease care settings. 18 Acknowledgments The authors are grateful to the patients for participating in the Undiagnosed Diseases Network. Data availability The Undiagnosed Diseases Network data used in this study contain sensitive patient information. De-identified patient data, including phenotypic and genomic data, are deposited in the database of Genotypes and Phenotypes (dbGaP) maintained by the National Institutes of Health. To explore data available in the latest release, visit the UDN study page in dbGaP. Individuals interested in accessing UDN data through dbGaP should submit a data access request. Detailed instructions for this process can be found on the NIH Scientific Data Sharing website: How to Request and Access Datasets from dbGaP. Code availability Codeusedinthisstudyarepubliclyavailableat https://github.com/cathyshyr/RARE PHENIXforrarediseasephenotyping. Competing interests The authors declare no competing interests. Funding This work was supported in part by the National Institutes of Health Common Fund, grant 15-HG-0130 from the National Human Genome Research Institute, U01NS134349 from the National Institute of Neurological Disorders and Stroke, R00LM014429 from the National Library of Medicine, T32GM082773 from the National Institute of General Medical Sciences, and the Potocsnak Center for Undiagnosed and Rare Disorders. Author Contributions Concept and design: C.S., L.B., H.X. Drafting of the manuscript: C.S. Critical revision of the manuscript for important intellectual content: C.S., Y.H., R.J.T., T.A.C., K.W.B., R.H., D.V.F., A.W., J.F.P., L.B., H.X. Obtained funding: C.S., R.H. Administrative, technical, or material support: C.S., Y.H., R.J.T., T.A.C., K.W.B., R.H., D.V.F., A.W., J.F.P., L.B., H.X. Supervision: L.B., H.X. 19 References [1] Nguengang Wakap, S., Lambert, D.M., Olry, A., Rodwell, C., Gueydan, C., Lanneau, V., Murphy, D., Le Cam, Y., Rath, A.: Estimating cumulative point prevalence of rare diseases: analysis of the orphanet database. European journal of human genetics 28(2), 165–173 (2020) [2] Valdez, R., Ouyang, L., Bolen, J.: Public health and rare diseases: oxymoron no more. Preventing chronic disease 13, 05 (2016) [3] Health, T.L.G.: The landscape for rare diseases in 2024 (2024) [4] Schieppati, A., Henter, J.-I., Daina, E., Aperia, A.: Why rare diseases are an important medical and social issue. The Lancet 371(9629), 2039–2041 (2008) [5] Evans, W.R.: Dare to think rare: diagnostic delay and rare diseases. The British Journal of General Practice 68(670), 224 (2018) [6] Phillips, C., Parkinson, A., Namsrai, T., Chalmers, A., Dews, C., Gregory, D., Kelly, E., Lowe, C., Desborough, J.: Time to diagnosis for a rare disease: managing medical uncertainty. a qualitative study. Orphanet journal of rare diseases 19(1), 297 (2024) [7] Faye, F., Crocione, C., Pe ̃na, R., Bellagambi, S., Escati Pe ̃naloza, L., Hunter, A., Jensen, L., Oosterwijk, C., Schoeters, E., Vicente, D., et al.: Time to diagnosis and determinants of diagnostic delays of people living with a rare disease: results of a rare barometer retrospective patient survey. European Journal of Human Genetics 32(9), 1116–1126 (2024) [8] Tinker, R.J., Fisher, M., Gimeno, A.F., Gill, K., Ivey, C., Peterson, J.F., Bas- tarache, L.: Diagnostic delay in monogenic disease: A scoping review. Genetics in Medicine 26(4), 101074 (2024) [9] Morton, G., Thomas, S., Roberts, P., Clark, V., Imrie, J., Morrison, A.: The importance of early diagnosis and views on newborn screening in metachromatic leukodystrophy: results of a caregiver survey in the uk and republic of ireland. Orphanet Journal of Rare Diseases 17(1), 403 (2022) [10] Cohen, J.S., Biesecker, B.B.: Quality of life in rare genetic conditions: a systematic review of the literature. American Journal of Medical Genetics Part A 152(5), 1136–1156 (2010) [11] Carmichael, N., Tsipis, J., Windmueller, G., Mandel, L., Estrella, E.: “is it going to hurt?”: the impact of the diagnostic odyssey on children and their families. Journal of Genetic Counseling 24(2), 325–335 (2015) [12] Yang, G., Cintina, I., Pariser, A., Oehrlein, E., Sullivan, J., Kennedy, A.: The national economic burden of rare disease in the united states in 2019. Orphanet 20 journal of rare diseases 17(1), 163 (2022) [13] K ̈ohler, S., Gargano, M., Matentzoglu, N., Carmody, L.C., Lewis-Smith, D., Vasilevsky, N.A., Danis, D., Balagura, G., Baynam, G., Brower, A.M., et al.: The human phenotype ontology in 2021. Nucleic acids research 49(D1), 1207–1217 (2021) [14] Hartley, T., Lemire, G., Kernohan, K.D., Howley, H.E., Adams, D.R., Boycott, K.M.: New diagnostic approaches for undiagnosed rare genetic diseases. Annual review of genomics and human genetics 21(1), 351–372 (2020) [15] Wojcik, M.H., Lemire, G., Berger, E., Zaki, M.S., Wissmann, M., Win, W., White, S.M., Weisburd, B., Wieczorek, D., Waddell, L.B., et al.: Genome sequencing for diagnosing rare diseases. New England Journal of Medicine 390(21), 1985–1997 (2024) [16] Splinter, K., Adams, D.R., Bacino, C.A., Bellen, H.J., Bernstein, J.A., Cheatle- Jarvela, A.M., Eng, C.M., Esteves, C., Gahl, W.A., Hamid, R., et al.: Effect of genetic diagnosis on patients with previously undiagnosed disease. New england journal of medicine 379(22), 2131–2139 (2018) [17] Macnamara, E.F., D’Souza, P., Network, U.D., Tifft, C.J.: The undiagnosed dis- eases program: Approach to diagnosis. Translational Science of Rare Diseases 4(3-4), 179–188 (2020) [18] Philippakis, A.A., Azzariti, D.R., Beltran, S., Brookes, A.J., Brownstein, C.A., Brudno, M., Brunner, H.G., Buske, O.J., Carey, K., Doll, C., et al.: The match- maker exchange: a platform for rare disease gene discovery. Human mutation 36(10), 915–921 (2015) [19] Fung, K.W., Richesson, R., Bodenreider, O.: Coverage of rare disease names in standard terminologies and implications for patients, providers, and research. In: AMIA Annual Symposium Proceedings, vol. 2014, p. 564 (2014) [20] Yadaw, A.S., Sid, E., Sidky, H., Zeng, C., Zhu, Q., Math ́e, E.A., Consortium, N., et al.: Systematic identification of rare disease patients in electronic health records enables evaluation of clinical outcomes. medRxiv (2025) [21] Garcelon, N., Neuraz, A., Salomon, R., Faour, H., Benoit, V., Delapalme, A., Munnich, A., Burgun, A., Rance, B.: A clinician friendly data warehouse oriented toward narrative reports: Dr. warehouse. Journal of biomedical informatics 80, 52–63 (2018) [22] Mak, C.M., Woo, P.P.S., Song, F.E., Chan, F.C.H., Chan, G.P.Y., Pang, T.L.F., Au, B.S.C., Chan, T.C.H., Chong, Y.K., Law, E.C.Y., et al.: Computer-assisted patient identification tool in inborn errors of metabolism–potential for rare disease patient registry and big data analysis. Clinica Chimica Acta 561, 119811 (2024) 21 [23] Dong, H., Su ́arez-Paniagua, V., Zhang, H., Wang, M., Casey, A., Davidson, E., Chen, J., Alex, B., Whiteley, W., Wu, H.: Ontology-driven and weakly super- vised rare disease identification from clinical notes. BMC Medical Informatics and Decision Making 23(1), 86 (2023) [24] Segura-Bedmar, I., Camino-Perdones, D., Guerrero-Aspizua, S.: Exploring deep learning methods for recognizing rare diseases and their clinical manifestations from texts. BMC bioinformatics 23(1), 263 (2022) [25] Xiao, Y., Enayati, M., Schaeferle, G.M., Lanpher, B.C., Klee, E.W., Ngufor, C.: Enhancing patient care in rare genetic diseases: An hpo-based phenotyp- ing pipeline. In: 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), p. 2754–2760 (2023). IEEE [26] Zhao, W., Wu, C., Fan, Y., Qiu, P., Zhang, X., Sun, Y., Zhou, X., Zhang, S., Peng, Y., Wang, Y., et al.: An agentic system for rare disease diagnosis with traceable reasoning. Nature, 1–10 (2026) [27] Shyr, C., Hu, Y., Bastarache, L., Cheng, A., Hamid, R., Harris, P., Xu, H.: Iden- tifying and extracting rare diseases and their phenotypes with large language models. Journal of Healthcare Informatics Research 8(2), 438–461 (2024) [28] Shyr, C., Cassini, T.A., Tinker, R.J., Byram, K.W., Emb ́ı, P.J., Bastarache, L., Peterson, J.F., Xu, H., Hamid, R., et al.: Large language models for rare dis- ease diagnosis at the undiagnosed diseases network. JAMA Network Open 8(8), 2528538–2528538 (2025) [29] Shyr, C., Tinker, R.J., Harris, P.A., Cheng, A.C., Byram, K.W., Bastarache, L., Peterson, J.F., Hamid, R., Xu, H., Cassini, T.A.: Accuracy of large language models in generating rare disease differential diagnosis using key clinical features. In: MEDINFO 2025—Healthcare Smart X Medicine Deep, p. 1054–1058. IOS Press, Amsterdam, Netherlands (2025) [30] Greco, K.F., Yang, Z., Li, M., Tong, H., Morini Sweet, S., Geva, A., Mandl, K.D., Raby, B.A., Cai, T.: A weakly supervised transformer for rare disease diagnosis and subphenotyping from ehrs with pulmonary case studies. npj Digital Medicine (2026) [31] Alsentzer, E., Li, M.M., Kobren, S.N., Noori, A., Network, U.D., Kohane, I.S., Zitnik, M.: Few shot learning for phenotype-driven diagnosis of patients with rare genetic diseases. npj Digital Medicine 8(1), 380 (2025) [32] Yang, J., Liu, C., Deng, W., Wu, D., Weng, C., Zhou, Y., Wang, K.: Enhancing phenotype recognition in clinical notes using large language models: Phenobcbert and phenogpt. Patterns 5(1) (2024) [33] Wu, D., Wang, Z., Nguyen, Q., Wang, K.: Integrating chain-of-thought and 22 retrieval augmented generation enhances rare disease diagnosis from clinical notes. arXiv preprint arXiv:2503.12286 (2025) [34] Yang, T., Huang, D., Lin, Y., Wu, P., Wu, Z., Ma, G., Lu, Y., Dong, X., Li, D., Ge, J., et al.: A specialized large language model for clinical reasoning and diagnosis in rare diseases. arXiv preprint arXiv:2511.14638 (2025) [35] Boceck, D., Laugwitz, L., Sturm, M., Bezdan, D., Gschwind, A., Haack, T.B., Ossowski, S.: aidiva–diagnostics of rare genetic diseases using large language models. medRxiv, 2025–09 (2025) [36] Shyr, C., Ren, B., Hsu, C.-Y., Tinker, R.J., Cassini, T.A., Hamid, R., Wright, A., Bastarache, L., Peterson, J.F., Malin, B.A., et al.: A statistical framework for evaluating the repeatability and reproducibility of large language models. medRxiv, 2025–08 (2025) [37] Michalski, A.A., Lis, K., Stankiewicz, J., Kloska, S.M., Sycz, A., Dudzi ́nski, M., Muras-Szwedziak, K., Nowicki, M., Bazan-Socha, S., Dabrowski, M.J., et al.: Sup- porting the diagnosis of fabry disease using a natural language processing-based approach. Journal of Clinical Medicine 12(10), 3599 (2023) [38] Li ́evin, V., Hansen, J.M., Lund, A., Elstein, D., Matthiesen, M.E., Elomaa, K., Zarakowska, K., Himmelhan, I., Botha, J., Borgeskov, H., et al.: Findzebra online search delving into rare disease case reports using natural language processing. PLOS digital health 2(6), 0000269 (2023) [39] Mao, X., Huang, Y., Jin, Y., Wang, L., Chen, X., Liu, H., Yang, X., Xu, H., Luan, X., Xiao, Y., et al.: A phenotype-based ai pipeline outperforms human experts in differentially diagnosing rare diseases using ehrs. NPJ Digital Medicine 8(1), 68 (2025) [40] Yang, J., Shu, L., Duan, H., Li, H.: Rdguru: An intelligent agent for rare diseases. In: AMIA Annual Symposium Proceedings, vol. 2024, p. 1275 (2025) [41] Cao, L., Sun, J., Cross, A.: An automatic and end-to-end system for rare disease knowledge graph construction based on ontology-enhanced large language models: development study. JMIR Medical Informatics 12(1), 60665 (2024) [42] Wen, B., Shi, S., Long, Y., Dang, Y., Tian, W.: Phenodp: leveraging deep learning for phenotype-based case reporting, disease ranking, and symptom recommendation. Genome Medicine 17(1), 67 (2025) [43] Kim, J., Wang, K., Weng, C., Liu, C.: Assessing the utility of large language models for phenotype-driven gene prioritization in the diagnosis of rare genetic disease. The American Journal of Human Genetics 111(10), 2190–2202 (2024) [44] Young, C.C., Enichen, E., Rivera, C., Auger, C.A., Grant, N., Rao, A., Succi, 23 M.D.: Diagnostic accuracy of a custom large language model on rare pediatric disease case reports. American Journal of Medical Genetics Part A 197(2), 63878 (2025) [45] Chen, X., Yi, H., You, M., Liu, W., Wang, L., Li, H., Zhang, X., Guo, Y., Fan, L., Chen, G., et al.: Enhancing diagnostic capability with multi-agents conversational large language models. NPJ digital medicine 8(1), 159 (2025) [46] Zhong, W., Liu, Y., Liu, Y., Yang, K., Gao, H., Yan, H., Hao, W., Yan, Y., Yin, C.: Performance of chatgpt-4o and four open-source large language models in generating diagnoses based on china’s rare disease catalog: comparative study. Journal of Medical Internet Research 27, 69929 (2025) [47] Rider, N.L., Li, Y., Chin, A.T., DiGiacomo, D.V., Dutmer, C., Farmer, J.R., Roberts, K., Savova, G., Ong, M.-S.: Evaluating large language model perfor- mance to support the diagnosis and management of patients with primary immune disorders. Journal of Allergy and Clinical Immunology 156(1), 81–87 (2025) [48] Garcia, B.T., Westerfield, L., Yelemali, P., Gogate, N., Rivera-Munoz, E.A., Du, H., Dawood, M., Jolly, A., Lupski, J.R., Posey, J.E.: Improving automated deep phenotyping through large language models using retrieval-augmented generation. Genome Medicine 17(1), 91 (2025) [49] Wu, J., Dong, H., Li, Z., Wang, H., Li, R., Patra, A., Dai, C., Ali, W., Scordis, P., Wu, H.: A hybrid framework with large language models for rare disease phenotyping. BMC Medical Informatics and Decision Making 24(1), 289 (2024) [50] Liu, C., Ta, C.N., Havrilla, J.M., Nestor, J.G., Spotnitz, M.E., Geneslaw, A.S., Hu, Y., Chung, W.K., Wang, K., Weng, C.: Oard: Open annotations for rare diseases and their phenotypes based on real-world data. The American Journal of Human Genetics 109(9), 1591–1604 (2022) [51] Groza, T., Caufield, H., Gration, D., Baynam, G., Haendel, M.A., Robinson, P.N., Mungall, C.J., Reese, J.T.: An evaluation of gpt models for phenotype concept recognition. BMC Medical Informatics and Decision Making 24(1), 30 (2024) [52] Thompson, W.E., Vidmar, D.M., De Freitas, J.K., Pfeifer, J.M., Fornwalt, B.K., Chen, R., Altay, G., Manghnani, K., Nelsen, A.C., Morland, K., et al.: Large language models with retrieval-augmented generation for zero-shot disease phenotyping. arXiv preprint arXiv:2312.06457 (2023) [53] Feng, Y., Qi, L., Tian, W.: Phenobert: a combined deep learning method for automated recognition of human phenotype ontology. IEEE/ACM Transactions on Computational Biology and Bioinformatics 20(2), 1269–1277 (2022) [54] Collins, G.S., Moons, K.G., Dhiman, P., Riley, R.D., Beam, A.L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J.B., Van Smeden, M., et al.: Tripod+ ai 24 statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. bmj 385 (2024) [55] Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., Wang, S., Li, J., Hu, R., Zhang, T., Wang, G., et al.: Instruction tuning for large language models: A survey. ACM Computing Surveys 58(7), 1–36 (2026) [56] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023) [57] Xie, Q., Chen, Q., Chen, A., Peng, C., Hu, Y., Lin, F., Peng, X., Huang, J., Zhang, J., Keloth, V., et al.: Medical foundation large language models for comprehensive text analysis and beyond. NPJ digital medicine 8(1), 141 (2025) [58] Hu, Y., Zuo, X., Zhou, Y., Peng, X., Huang, J., Keloth, V.K., Zhang, V.J., Weng, R.-L., Shyr, C., Chen, Q., et al.: Information extraction from clinical notes: are we ready to switch to large language models? Journal of the American Medical Informatics Association, 213 (2026) [59] Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.: Qlora: Efficient fine- tuning of quantized llms. Advances in neural information processing systems 36, 10088–10115 (2023) [60] Mart ́ınez-deMiguel, C., Segura-Bedmar, I., Chac ́on-Solano, E., Guerrero-Aspizua, S.: The raredis corpus: a corpus annotated with rare diseases, their signs and symptoms. Journal of biomedical informatics 125, 103961 (2022) [61] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K ̈uttler, H., Lewis, M., Yih, W.-t., Rockt ̈aschel, T., et al.: Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems 33, 9459–9474 (2020) [62] Amberger, J.S., Bocchini, C.A., Scott, A.F., Hamosh, A.: Omim. org: leveraging knowledge across phenotype–gene relationships. Nucleic acids research 47(D1), 1038–1043 (2019) [63] Chen, T., Guestrin, C.: Xgboost: A scalable tree boosting system. In: Proceedings of the 22nd Acm Sigkdd International Conference on Knowledge Discovery and Data Mining, p. 785–794 (2016) [64] Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., Liu, T.-Y.: Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems 30 (2017) [65] Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A.V., Gulin, A.: Catboost: unbiased boosting with categorical features. Advances in neural information 25 processing systems 31 (2018) [66] Lin, D., et al.: An information-theoretic definition of similarity. In: Icml, vol. 98, p. 296–304 (1998) [67] Luo, L., Yan, S., Lai, P.-T., Veltri, D., Oler, A., Xirasagar, S., Ghosh, R., Similuk, M., Robinson, P.N., Lu, Z.: Phenotagger: a hybrid method for phenotype concept recognition using human phenotype ontology. Bioinformatics 37(13), 1884–1890 (2021) [68] Arbabi, A., Adams, D.R., Fidler, S., Brudno, M.: Identifying clinical terms in medical text using ontology-guided machine learning. JMIR medical informatics 7(2), 12596 (2019) 26