Paper deep dive
Coding-Free and Privacy-Preserving MCP Framework for Clinical Agentic Research Intelligence System
Taehun Kim, Hyeryun Park, Hyeonhoon Lee, Yushin Lee, Kyungsang Kim, Hyung-Chul Lee
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 4/15/2026, 1:36:07 AM
Summary
The Clinical Agentic Research Intelligence System (CARIS) is a coding-free, privacy-preserving framework that automates end-to-end clinical research workflows. By integrating Large Language Models (LLMs) with modular tools via the Model Context Protocol (MCP), CARIS enables researchers to perform study design, cohort construction, IRB documentation, machine learning (Vibe ML), and report generation without direct access to raw patient data. Evaluation across three heterogeneous datasets (MIMIC-IV, INSPIRE, SyntheticMass) demonstrated high performance, with final reports achieving 96% coverage in LLM-based evaluation and 82% in human evaluation based on the TRIPOD+AI framework.
Entities (6)
Relation Signals (3)
CARIS → evaluatedon → MIMIC-IV
confidence 100% · We evaluated CARIS across three heterogeneous datasets (MIMIC-IV, INSPIRE, and SyntheticMass)
CARIS → utilizes → Model Context Protocol
confidence 100% · CARIS integrates Large Language Models (LLMs) with modular tools via the Model Context Protocol (MCP)
CARIS → usesguideline → TRIPOD+AI
confidence 95% · Final reports showed high completeness based on a checklist derived from the TRIPOD+AI framework
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Clinical research involves labor-intensive processes such as study design, cohort construction, model development, and documentation, requiring domain expertise, programming skills, and access to sensitive patient data. These demands create barriers for clinicians and external researchers conducting data-driven studies. To overcome these limitations, we developed a Clinical Agentic Research Intelligence System (CARIS) that automates the clinical research workflow while preserving data privacy, enabling comprehensive studies without direct access to raw data. CARIS integrates Large Language Models (LLMs) with modular tools via the Model Context Protocol (MCP), enabling natural language-driven orchestration of appropriate tools. Databases remain securely within the MCP server, and users access only the outputs and final research reports. Based on user intent, CARIS automatically executes the full pipeline: research planning, literature search, cohort construction, Institutional Review Board (IRB) documentation, Vibe Machine Learning (ML), and report generation, with iterative human-in-the-loop refinement. We evaluated CARIS on three heterogeneous datasets with distinct clinical tasks. Research plans and IRB documents were finalized within three to four iterations, using evidence from literature and data. The system supported Vibe ML by exploring feature-model combinations, ranking the top ten models, and generating performance visualizations. Final reports showed high completeness based on a checklist derived from the TRIPOD+AI framework, achieving 96% coverage in LLM evaluation and 82% in human evaluation. CARIS demonstrates that agentic AI can transform clinical hypotheses into executable research workflows across heterogeneous datasets. By eliminating the need for coding and direct data access, the system lowers barriers and bridges public and private clinical data environments.
Tags
Links
- Source: https://arxiv.org/abs/2604.12258v1
- Canonical: https://arxiv.org/abs/2604.12258v1
Trouble viewing inline? Open PDF directly →
Full Text
105,243 characters extracted from source content.
Expand or collapse full text
Coding-Free and Privacy-Preserving MCP Framework for Clinical Agentic Research Intelligence System Taehun Kim *1,2 , Hyeryun Park *3 , Hyeonhoon Lee *3,4,5 , Yushin Lee 1 , Kyungsang Kim †6 , Hyung-Chul Lee †1,3,4 1 Infmedix, Co., Ltd., Seoul, Republic of Korea. 2 Department of Transdisciplinary Studies, Seoul National University, Seoul, Republic of Korea. 3 Healthcare AI Research Institute, Seoul National University Hospital, Seoul, Republic of Korea. 4 Department of Medicine, Seoul National University College of Medicine, Seoul, Republic of Korea. 5 Department of Transdisciplinary Medicine, Seoul National University Hospital, Seoul, Republic of Korea. 6 Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA. * These authors contributed equally to this article. † Co-correspondence: Kyungsang Kim (kkim24@mgb.org), Hyung-Chul Lee (vital@snu.ac.kr). Abstract BACKGROUND Clinical research involves labor-intensive processes, including study design, cohort construction, data preprocessing, model development, and documentation, all of which demand domain expertise, programming skills, and access to sensitive patient data. These demands create a barrier to clinicians and external researchers conducting data-driven studies. To overcome these limitations, we developed an interactive Clinical Agentic Research Intelligence System (CARIS) that automates the full clinical research workflow while preserving data privacy, enabling researchers to perform comprehensive studies without direct access to raw patient data. METHODS CARIS integrates Large Language Models (LLMs) with modular tools through the Model Context Protocol (MCP), enabling natural language-driven orchestration of appropriate tools. All databases remain securely within the MCP server, and the LLM interacts with them without exposing raw patient data. Users access only the produced outputs and final research reports, ensuring strict data privacy. Based on user-defined research intent, CARIS automatically executes the full pipeline: research planning, literature search, cohort construction, Institutional Review Board (IRB) documentation, Vibe Machine Learning (ML), and report generation, with iterative human-in-the-loop refinement. RESULTS We evaluated CARIS across three heterogeneous datasets (MIMIC-IV, INSPIRE, and Syn- theticMass), each with distinct clinical tasks. Research plans and IRB documents were finalized within three to four iterations, using evidence from literature and data. The system supported Vibe ML by exploring feature–model combinations, ranking the top ten models, and generating performance visu- alizations. Final reports showed high completeness based on a checklist derived from the TRIPOD+AI framework, achieving 96% coverage under LLM-based evaluation and 82% under human evaluation. CONCLUSIONS CARIS demonstrates that agentic AI can fundamentally transform the routine of clinical data utilization by converting clinical hypotheses into executable research workflows across heterogeneous datasets. By eliminating the need for coding and direct data access, the system lowers barriers and bridges public and private clinical data environments. Introduction Clinical data-driven research requires domain expertise, programming skills, and access to sen- sitive patient data to translate clinical questions into actionable insights. Although clinicians for- mulate hypotheses based on their clinical knowl- edge, they often encounter substantial barriers in data preprocessing, Machine Learning (ML) model development, and result interpretation, which are time-consuming and technically demanding [1, 2]. External researchers face additional barriers in accessing private and sensitive patient data. These constraints impede broader participation in clini- cal research and slow the pace of medical advance- ment. Recent advances in Large Language Models (LLMs) [3–6] and agentic AI systems [7–11] have demonstrated the potential to automate complex, multi-step workflows. LLMs excel at interpret- ing natural language instructions and performing reasoning, while agentic systems enable them to orchestrate modular tools to execute specific tasks. These approaches have shown the ability to trans- late user intent into structured computational subtasks in general scientific domains [12–15], highlighting their potential for clinical research. Although recent efforts have made progress in iso- lated components, including automated ML, pro- tocol drafting, and document generation, they still struggle to support integrated clinical workflows and secure data execution [16]. 1 arXiv:2604.12258v1 [cs.CL] 14 Apr 2026 User interface LLM Agents Develop a model to predict ICU readmission using MIMIC. Datasets DB system File system MCP tools Pubmed search Paper analysis LLM Query generation ... Criteria definition Train ML model Create document Vibe machine learning ResearchplanningIRB documentationReport generation Figure 1. Overview of Clinical Agentic Research Intelligence System (CARIS). Given a user intent, CARIS autonomously executes a multi-step clinical research workflow by interacting with agents, heterogeneous datasets, and various tools, while allowing user intervention. Users do not have direct access to the file or DB systems. The existing approaches remain limited by challenges in handling heterogeneous data, main- taining data privacy, and ensuring transparency, reproducibility,andinterpretability[17–21]. Essential components, such as Institutional Review Board (IRB) documentation and struc- tured research protocol development, which are prerequisites for data access and ethical approval, remain insufficiently supported. As a result, these systems struggle to support integrated clini- cal workflows and face challenges to reflect the regulatory requirements and data environments specific to clinical research. The Model Context Protocol (MCP) addresses these limitations by providing a structured and verifiable interface between LLMs and exter- nal tools [21? –23]. It introduces a client–server architecture that decouples user interaction from underlying data sources, ensuring that only pro- cessed results are returned for review. Within this framework, LLMs can invoke modular tools, including APIs, functions, and datasets, without direct access to sensitive patient data. This design enables research across heterogeneous datasets while improving compliance with data governance. This study presents the Clinical Agentic Research Intelligence System (CARIS), an agentic AI framework that automates end-to-end clinical research workflows from natural language input, as shown in Figure 1. CARIS interprets user-defined research intent and orchestrates a series of agents, including research planning, Institutional Review Board (IRB) documentation, Vibe ML, and report generation, without requiring coding or direct data access. By enabling clinicians to conduct stud- ies through a Vibe ML paradigm, CARIS lowers technical barriers, while its MCP-based architec- ture ensures privacy-preserving interaction with heterogeneous data sources. The system also alle- viates the burden of clinical and regulatory docu- mentation, including IRB preparation and report generation. Across three distinct datasets and clin- ical tasks, CARIS demonstrated generalizability by completing research planning and IRB docu- mentation within three to four iterations, support- ing Vibe ML with automated visualization, and generating reports achieving high coverage (96% by LLM vs. 82% by human evaluation) based on a checklist derived from TRIPOD+AI [24]. Methods Research Workflow Automation CARIS autonomously executes end-to-end clini- cal research workflows through natural language interaction, leveraging LLMs to orchestrate multi- step processes through structured tool invocation (Note 1 in the Supplementary). This orchestration prompt guides the LLM to act as an independent clinical research scientist, leading study design, data analysis, tool utilization, documentation, and critical thinking. It utilizes modular MCP tools, each performing a single atomic operation with well-defined inputs and structured outputs. This design enables auditing of individual tools and integration of additional tools for scalability and flexibility. The complete set of tools is provided in Table S1 (Note 2 in the Supplementary). 2 Figure 2. Overview of the clinical research workflow. (A) User interaction to iteratively generate and refine a structured research plan. (B) Keyword extraction using the PIMO framework and retrieval of relevant PubMed literature. (C) Database exploration, cohort definition, query execution, and dataset generation. (D) Iterative generation and refine- ment of IRB document sections through user interaction. (E) Data analysis, preprocessing, model training, evaluation, and visualization, followed by the automated generation of a clinical ML report. Modular Agents Research Planning Agent Given a user research topic, the agent automati- cally generates a research plan through an inter- active process, as shown in Figure 2A (Note 3 in the Supplementary). The agent initiates a chatbot that presents questions to progressively refine the plan. As the user responds, the agent drafts the title, research purpose, research design, research method, validity evaluation, expected effects, and anticipated results, which are sections typically included in IRB documents. Users can also request revisions to the generated content. The agent supports users by retrieving and analyzing the relevant PubMed articles, as illus- trated in Figure 2B (Note 4 in the Supplemen- tary). For this purpose, keywords are extracted from the research plan using the PIMO framework, a variant of the PICO framework [25] tailored for clinical ML research. The PIMO framework captures key study elements: P (Patient/Popula- tion), I (Input data), M (Model/Method), and O (Outcome). Based on combinations of these key- words, relevant articles are retrieved using best match sorting [26], along with structured meta- data such as PubMed identifier (PMID), title, and abstract. The agent compares each article against the proposed study across PIMO dimensions using 0-50 scoring and a three-stage pipeline consist- ing of rule-based filtering, binary screening, and fine-grained similarity scoring, producing similar- ity scores and brief rationales. Based on the top ten relevant articles, users can critically review and refine their research plan. Finally, the agent reviews the research plan and the retrieved PubMed metadata to proceed with cohort construction, as shown in Figure 2C (Note 5 in the Supplementary). CARIS enables users to explore heterogeneous predefined clini- cal datasets. During metadata exploration, the agent analyzes the schema to understand the data 3 Table 1. Summary of datasets and target tasks. RDB: Relational Database. DatasetData sourceData formatPrediction taskNumber of instances MIMIC-IVBeth Israel Deaconess Medical CenterRDBICU readmission83,101 INSPIRESeoul National University HospitalRDBPostoperative AKI100,474 SyntheticMassSynthetic Massachusetts patient dataOMOP-CDMPrediabetes-to-diabetes2,556 structure, including tables, fields, relationships, and semantic descriptions. Based on this informa- tion, it generates a cohort construction guideline defining the study population (inclusion/exclusion criteria), target variable definitions, information on other variables, and key queries. The agent then executes SQL queries to extract the required data, stores the results in a unified CSV file, and produces a dataset creation report. IRB Documentation Agent Based on the refined research plan, the same interactive chatbot is used to further gener- ate research background, data analysis methods, and research hypotheses sections, as shown in Figure 2D. Throughout this process, the agent leverages LLMs and predefined instructions to ensure that each section is generated coherently while adhering to ethical requirements, forming a complete IRB document (Note 6 in the Supple- mentary). An iterative human-in-the-loop proto- col allows users to continuously review and revise generated content through the chat interface. Vibe Machine Learning Agent Given the input CSV data and generated docu- ments, the agent performs Exploratory Data Anal- ysis (EDA) and Vibe ML, as shown in Figure 2E (Note 7 in the Supplementary). During EDA, it computes descriptive statistics, assesses data quality, and generates visualizations such as distri- bution plots, correlation heatmaps, and pair plots. Data preprocessing includes label-encoding cat- egorical variables, removing columns with more than 50% missing valuess, and mean/mode impu- tation for numerical and categorical variable. Fea- ture selection is performed based on user pref- erence, using methods such as Recursive Feature Elimination (RFE), Boruta with SHapley Additive exPlanations (SHAP), SelectKBest using Mutual Information (MI), Random Forest–based feature importance, or LLM-recommended features. Following this, the agent trains ML models, while allowing users to adjust features and model choices through interactive chat. Supported algo- rithms are Random Forest, XGBoost, LightGBM, CatBoost, Decision Tree, Extra Trees, and lin- ear models, with hyperparameter tuning via grid search and stratified 5-fold cross-validation. Model performance is evaluated and visualized using confusion matrices, precision–recall curves, and AUROC with 95% confidence intervals, with sta- tistical comparisons based on the DeLong test and bootstrap resampling (n = 2,000, p < 0.05). Model interpretability is supported by SHAP-based fea- ture importance analyses [27]. Report Generation Agent The report generation agent automatically drafts a manuscript using all materials within the project. To support structured writing of clinical ML report, it employs a prompt based on the TRI- POD+AI guideline [24], a standardized reporting checklist (Note 8 in the Supplementary). The final manuscript is produced in Word format and includes all standard sections: Title, Abstract, Introduction, Methods, Results, Discussion, Con- clusion, References, and Supplementary Materials, along with tables and figures. Datasets and Experimental Settings To evaluate the proposed CARIS, we conduct experiments using three publicly available datasets with distinct clinical tasks: Intensive Care Unit (ICU) readmission prediction using the Medical Information Mart for Intensive Care (MIMIC)-IV dataset [28, 29], postoperative acute kidney injury (AKI) prediction using the INformative Surgical Patient dataset for Innovative Research Environ- ment (INSPIRE) dataset [30, 31], and prediabetes- to-diabetes progression prediction using the Syn- theticMass [32]. Table 1 summarizes the datasets and tasks, and details, including dataset versions, outcome definitions, inclusion/exclusion criteria, and predictor variables are provided (Note 9 in the Supplementary). CARIS is implemented as a web-based appli- cation, with an LLM-driven frontend (NiceGUI) for user interaction and a backend for tool execu- tion (Note 10 in the Supplementary). The system supports multiple LLM APIs for flexible deploy- ment under varying constraints. All experiments in this study use Claude Sonnet 4.6 via API with- out GPUs. The implementation, results, and demo will be available on GitHub: https://github.com/ thkim107/nocode clinicalai. 4 Figure 3. Revision patterns in IRB document generation. (A) Sections revised across iterations. Each dot rep- resents a revision, with colors indicating section types. (B) Distribution of revision items per section. Numbers inside wedges indicate item counts, and percentages indicate their proportions. Results Across three heterogeneous datasets, CARIS autonomously executed a multi-step research pipeline. To assess system reliability and out- put quality, we evaluated key results produced throughout the workflow. IRB Document Evaluation The IRB document generation process followed a human-in-the-loop framework, where experts iter- atively reviewed and revised the outputs. The interaction between experts and the system was analyzed to assess the number and types of revi- sions required for completion (Note 11 in the Sup- plementary). As shown in Figure 3A, documents were completed within three to four iterations: ICU readmission and postoperative AKI required four iterations, while the prediabetes-to-diabetes task required three. As multiple items are often revised per iteration, a total of 19 item-level revi- sions were identified across all tasks, as shown in Figure 3B, primarily in study design and method- ology (26%), data analysis methods (26%), and validity evaluation (26%) sections. Revisions in study design and methods focused on refining outcome definitions, inclusion/exclu- sion criteria, and alignment with dataset char- acteristics. Validity evaluation revisions refined expected performance ranges and excluded con- founder adjustments and unnecessary analyses. Data analysis revisions clarified modeling strate- gies and dataset descriptions and removed inap- propriate or unnecessary analytical components. These findings show that expert–system interac- tion extends beyond error correction, acting as an Figure 4. Radar chart of IRB document evaluation results across four criteria. Each axis represents the pass rate (%) for each criteria assessed across three tasks. iterative refinement process that enhances clini- cal relevance through domain expertise, PubMed evidence, and dataset-specific details. The final IRB documents were evaluated by LLM using four criteria: content completeness (9 items), non-expert accessibility (5 items), ethi- cal adequacy (6 items), and cross-section con- sistency (5 items). Each item was reviewed by Claude Sonnet and rated as pass or fail with sug- gested revisions (Note 12 in the Supplementary). As shown in Figure 4, all tasks achieved perfect scores in content completeness and cross-section consistency, indicating that CARIS generated all essential sections with logical coherence. However, non-expert accessibility showed lower pass rates (60%) for two tasks, suggesting that technical descriptions and undefined abbreviations should 5 Table 2. Performance comparison of the top two ML models for each task under different feature selection methods. RF: Random Forest; MI: Mutual Information; RFE: Recursive Feature Elimination; TG: Triglycerides. Task (dataset)ModelFeature selection(#)AUROC Precision Recall F1 ICU readmission (MIMIC) XGBoost MI + RFE (55)71.50%73.24% 51.24% 49.97% XGBoost SelectKBest MI (37)71.09%68.42% 50.92% 49.38% Preoperative AKI (INSPIRE) LightGBM RFE (31)85.77%80.00% 53.88% 56.24% LightGBM Boruta with SHAP (30)85.61%78.64% 53.80% 56.08% Prediabetes-to-diabetes (SyntheticMass) XGBoost LLM features + HDL (10)88.58%74.79% 68.62% 70.72% XGBoost LLM features + HDL + TG (11) 88.03%70.98% 67.04% 68.50% be further clarified to improve understanding. Eth- ical adequacy was also lower (83%) for two tasks due to missing conflict-of-interest and regulatory- compliance statements. These results suggest that certain issues may still be overlooked despite the human-in-the-loop process, underscoring the importance of a final evaluation and the need for technical refinement to ensure consistent quality across tasks. Vibe ML Performance Table 2 summarizes the performance of the two best-performing ML models under different fea- ture selection strategies for each task. For the ICU readmission prediction task, XGBoost combined with the MI and RFE feature selection methods achieved the best performance, obtaining AUROC of 71.50% (95% CI: 70.2–72.5), which is compa- rable to previously reported ML models in simi- lar cohorts that typically achieve AUROC values around 0.7 [33]. For the postoperative AKI pre- diction task, LightGBM trained with RFE feature selection method demonstrated the best perfor- mance, achieving an AUROC of 85.77% (95% CI: 85.18–86.12), consistent with prior studies reporting AUROC values in the range of 0.8–0.85 in similar settings [34]. For the prediabetes-to- diabetes prediction task, XGBoost trained on LLM-recommended key features along with HDL- cholesterol achieved the best performance, with an AUROC of 88.5% (95% CI: 85.3–91.0), which aligns with similar studies achieving AUROC val- ues in the range of 0.8–0.85 [35]. Overall, these results demonstrate that CARIS can achieve performance comparable to conven- tional manual ML studies while flexibly exploring diverse combinations of model–feature selection and supporting user-driven, task-specific config- urations. In addition, CARIS automatically gen- erates performance visualizations, including con- fusion matrices, precision-recall curves, AUROC curves with confidence intervals, and SHAP-based interpretations for the best model. Final Report Evaluation To assess the completeness of clinical ML reports, we developed a structured checklist based on the TRIPOD+AI guideline [24], which was also applied during report generation (Note 13 in the Supplementary). This structured evaluation framework helps users identify gaps and improve the quality of their studies within the work- flow. As shown in Figure 5, the checklist con- sists of 28 items covering key content across nine sections. Each item was evaluated as present (Yes/No), along with suggestions for improve- ment. Figure 5A presents the evaluation results generated by Claude Sonnet, while Figure 5B shows those from human evaluator. Overall, the LLM-based evaluation demonstrated high cover- age across all three reports, achieving a mean score of 96%, whereas human evaluation yielded a lower mean score of 82%. Agreement between the two was substantial, with a Cohen’s kappa of 0.6989, and consistent across individual papers (0.7273 for ICU admission, 0.7143 for preoperative AKI, and 0.6400 for prediabetes-to-diabetes). Despite this agreement, consistent discrepan- cies were observed in three items. First, the LLM was more lenient in judging novelty, often treat- ing well-structured method as sufficient contribu- tion, whereas human required clearer originality. Second, for code availability, the LLM consid- ered “available upon request” acceptable, while human required publicly accessible repositories. Third, the LLM frequently failed to identify issues in figures, such as missing references or inaccu- rate descriptions, reflecting limitations in cross- checking visual content. Both LLM and human evaluations also identified common gaps, particu- larly in the Performance section, where important elements such as calibration metrics, subgroup analyses, and external validation were often under- reported. Overall, these findings suggest that while LLM-based evaluation is reasonably reliable for structured checklist items, it tends to overestimate compliance in areas requiring contextual judgment or visual verification. 6 Figure 5. Checklist coverage of the final report across nine criteria. (A) Evaluation results by Claude Sonnet. (B) Evaluation results by human evaluator. Each axis represents criteria with checklist item count in parentheses. The dashed line indicates the average across all three reports. Execution Time We present the execution time of the CARIS pipeline across three clinical tasks after establish- ing the research topic. The preparation of research plans and IRB documentation, supported by rele- vant literature and dataset information, typically required approximately 2 to 2.5 hours. The Vibe ML stage varied depending on the number of models and feature selection strategies explored. Experiments involving a single model with mul- tiple feature combinations could be completed in 5 minutes, whereas more extensive exploration across multiple models and feature configurations required up to 10 hours. Report generation was highly efficient, typically taking 5 to 10 minutes to produce comprehensive outputs. Overall, these findings indicate that CARIS reduces the time required for end-to-end clinical research, enabling completion of the full workflow within hours rather than days or weeks, while maintaining perfor- mance comparable to conventional approaches. Discussion CARIS expands access to clinical research by lowering technical and data-access barriers that have limited participation. By enabling end-to-end workflows through natural language interaction, it allows clinicians and external researchers to conduct data-driven studies without requiring pro- gramming expertise or direct access to sensitive patient data. The key implication of the system lies in its privacy-preserving, MCP-based architec- ture, which supports deployment in environments with restricted data access. Validated across het- erogeneous datasets using a consistent pipeline, CARIS demonstrates applicability in real-world clinical settings. This agentic AI framework pro- vides a practical pathway for bridging public and private data systems, thereby supporting broader participation in clinical research. Despite its strengths, the current implemen- tation of the agentic AI framework has several limitations. The system is restricted to predefined datasets within the Korea Health Data Platform (https://khdp.net), highlighting the need for con- tinuous data enrichment and the development of interoperable data integration tools, such as MCP-FHIR [21]. In addition, the system pri- marily operates on structured tabular data and does not yet support multimodal inputs, including clinical notes, medical imaging, and physiologi- cal signals. Incorporating the medical foundation models [36] into the system would enable more comprehensive and realistic clinical research work- flows. Lastly, while major components of the workflow are automated, outputs such as IRB documents and reports still require manual refine- ment to comply with institution-specific formats and the submission requirements of journals or conferences. Several opportunities for improvement were identified throughout the workflow. Tool exe- cution errors related to LLM API rate limits highlighted the need for more robust exception handling and recovery mechanisms. In IRB docu- mentation, improvements are needed in the incor- poration of more relevant literature. In the Vibe ML stage, additional methods are required to 7 address class imbalance and support calibration and subgroup analyses, thereby improving model performance, fairness, and clinical interpretability. Despite a high level of automation, human-in- the-loop validation remains an essential design principle, enabling users to refine intermediate outputs and maintain control over the research process, which is critical for safe and effective clinical adoption. In conclusion, CARIS introduces a new paradigm for clinical research by enabling clini- cians and external researchers to conduct studies without programming expertise or direct data access. By enabling privacy-preserving, end-to- end workflows across heterogeneous data envi- ronments, this approach has the potential to expand participation in clinical research, improve the representativeness of evidence, and facilitate collaboration across institutional boundaries. Disclosures This research was supported by a grant of the Boston-Korea Innovative Research Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (grant no.: RS-2024-00403047, NTIS no.: 2460003034). This research was also supported by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (grant no.: RS-2024- 00439677, NTIS no.: 2460003917) The funders of the study had no role in the study design, data collection, data analysis, data interpretation, or writing of the report. References [1] Watson J, Hutyra CA, Clancy SM, et al. Overcoming barriers to the adoption and implementation of predictive modeling and machine learning in clinical care: what can we learn from us academic medical cen- ters? JAMIA Open 2020;3:167–172. DOI: 10.1093/jamiaopen/ooz046. [2] Thirunavukarasu AJ, Elangovan K, Gutier- rez L, et al. Democratizing artificial intel- ligence imaging analysis with automated machine learning: tutorial. J Med Internet Res 2023;25:e49949. DOI: 10.2196/49949. [3] Achiam J, Adler S, Agarwal S, et al. Gpt- 4 technical report. March 15, 2023 (https:// arxiv.org/abs/2303.08774). Preprint. [4] Chowdhery A, Narang S, Devlin J, et al. Palm: Scaling language modeling with path- ways. J Mach Learn Res 2023;24:1–113. (http: //jmlr.org/papers/v24/22-1144.html). [5] Meta AI. The llama 4 herd: The beginning of a new era of natively multimodal ai inno- vation. April 5, 2025 (https://ai.meta.com/ blog/llama-4-multimodal-intelligence/). [6] Gemma Team. Gemma 3 technical report. March 25, 2025 (https://arxiv.org/abs/2503. 19786). Preprint. [7] Wang L, Ma C, Feng X, et al. A survey on large language model based autonomous agents. Front Comput Sci 2024;18:186345. DOI: 10.1007/s11704-024-40231-1. [8] Li X, Wang S, Zeng S, Wu Y, Yang Y. A survey on llm-based multi-agent systems: workflow, infrastructure, and challenges. Vic- inagearth 2024;1:9. DOI: 10.1007/s44336-024- 00009-2. [9] Zhao B, Foo LG, Hu P, Theobalt C, Rahmani H, Liu J. Llm-based agentic reasoning frame- works: A survey from methods to scenarios. August 25, 2025 (https://arxiv.org/abs/2508. 17692). Preprint. [10] Wu Q, Bansal G, Zhang J, et al. Autogen: Enabling next-gen llm applications via multi- agent conversations In: First Conference on Language Modeling (COLM). 2024 (https:// openreview.net/forum?id=BAakY1hNKS). [11] Tran KT, Dao D, Nguyen MD, Pham QV, O’Sullivan B, Nguyen HD. Multi-agent col- laboration mechanisms: A survey of llms. Jan- uary 10, 2025 (https://arxiv.org/abs/2501. 06322). Preprint. [12] Baek J, Jauhar SK, Cucerzan S, Hwang SJ. Researchagent: Iterative research idea generation over scientific literature with large language models In: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies (Volume 1: Long Papers). ACL: 2025:6709–6738 (https://aclanthology. 8 org/2025.naacl-long.342/). [13] Guo S, Deng C, Wen Y, Chen H, Chang Y, Wang J. Ds-agent: Automated data sci- ence by empowering large language mod- els with case-based reasoning. February 27, 2024 (https://arxiv.org/abs/2402.17453). Preprint. [14] Gridach M, Nanavati J, Abidine KZE, Mendes L, Mack C. Agentic ai for scientific discovery: A survey of progress, challenges, and future directions. March 12, 2025 (https: //arxiv.org/abs/2503.08979). Preprint. [15] Swanson K, Wu W, Bulaong NL, Pak JE, Zou J. The virtual lab: Ai agents design new sars-cov-2 nanobodies with exper- imental validation. November 11, 2024 (https://w.biorxiv.org/content/10.1101/ 2024.11.11.623004v1). Preprint. [16] Artsi Y, Sorin V, Glicksberg BS, Korfiatis P, Nadkarni GN, Klang E. Large language models in real-world clinical workflows: a sys- tematic review of applications and implemen- tation. Front Digit Health 2025;7:1659134. DOI:10.3389/fdgth.2025.1659134. [17] Hassan M, Kushniruk A, Borycki E. Bar- riers to and facilitators of artificial intel- ligence adoption in health care: scoping review. JMIR Hum Factors 2024;11:e48633. DOI:10.2196/48633. [18] Garcia B, Hogarth M, Wang Y, Zhu X, Tu SP. Multi-site research using electronic health record data: Lessons learned from a case study. Learn Health Syst 2025;9:e70039. DOI:10.1002/lrh2.10439. [19] Lee M, Kim K, Shin Y, Lee Y, Kim TJ. Advancements in electronic medical records for clinical trials: Enhancing data manage- ment and research efficiency. Cancers (Basel) 2025;17:1552. DOI:10.3390/cancers17091552. [20] Holmes JH, Beinlich J, Boland MR, et al. Why is the electronic health record so chal- lenging for research and clinical care? Meth- ods Inf Med 2021;60:032–048. DOI:10.1055/s- 0041-1731784. [21] Ehtesham A, Singh A, Kumar S. Enhanc- ing clinical decision support and ehr insights through llms and the model context pro- tocol: An open-source mcp-fhir framework. June 18, 2025 (https://ieeexplore.ieee.org/ abstract/document/11105280). Preprint. [22] Singh A, Ehtesham A, Kumar S, Khoei T. A survey of the model context protocol (mcp): Standardizing context to enhance large lan- guage models (llms). April 3, 2025 (https: //w.preprints.org/frontend/manuscript/ b45407370ad06ed48b5ebc462c1d8a2c/ download pub). Preprint. [23] Attrach RA, Moreira P, Fani R, Umeton R, Celi LA. Conversational llms simplify secure clinical data access, understanding, and anal- ysis. July 1, 2025 (https://doi.org/10.48550/ arXiv.2507.01053). Preprint. [24] Collins GS, Moons KG, Dhiman P, et al. Tripod+ ai statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. bmj 2024;385. DOI:10.1136/bmj-2023-078378. [25] Haynes RB, Sackett DL, Richardson WS, Rosenberg W, Langley GR. Evidence-based medicine: How to practice & teach EBM. CanadianMedicalAssociation.Journal 1997;157:788. [26] Fiorini N, Canese K, Starchenko G, et al. Best match: new relevance search for pubmed.PLoSBiol2018;16:e2005343. DOI:10.1371/journal.pbio.2005343. [27] Lundberg SM, Lee SI. A unified approach tointerpretingmodelpredictionsIn: AdvancesinNeuralInformationPro- cessing Systems. 2017:4765–4774 (https: //proceedings.neurips.c/paper/2017/hash/ 8a20a8621978632d76c43dfd28b67767-Abstract. html). [28] Johnson AE, Bulgarelli L, Shen L, et al. Mimic-iv, a freely accessible electronic health record dataset. Sci data 2023;10:1. DOI:10.1038/s41597-022-01899-x. [29] Johnson A, Bulgarelli L, Pollard T, et al. Mimic-iv. October 11, 2024 (https:// physionet.org/content/mimiciv/3.1/). [30] Lim L, Lee H, Jung CW, et al. Inspire, a publicly available research dataset for peri- operative medicine. Sci Data 2024;11:655. 9 DOI:10.1038/s41597-024-03517-4. [31] Lim L, Lee HC. Inspire, a publicly available research dataset for perioperative medicine. August 11, 2024 (https://physionet.org/ content/inspire/1.3/). [32] Walonoski J, Kramer M, Nichols J, et al. Syntheticmass data, version 2. May 24, 2017 (https://synthea.mitre.org/downloads). [33] de S ́a AG, Gould D, Fedyukova A, et al. Explainable machine learning for icu readmis- sion prediction. September 13, 2024 (https: //arxiv.org/abs/2309.13781). Preprint. [34] Sun R, Li S, Wei Y, et al. Develop- ment of interpretable machine learning mod- els for prediction of acute kidney injury after noncardiac surgery: a retrospective cohort study. Int J Surg 2024;110:2950–2962. DOI:10.1097/JS9.0000000000001237. [35] Luo J, Hu D, Han R, et al. Risk stratification at prediabetes onset and association with dia- betes outcomes using EHR data. NPJ Metab Health Dis 2025;3:48. DOI:10.1038/s44324- 025-00091-0. [36] Sellergren A, Kazemzadeh S, Jaroensri T, et al. Medgemma technical report. July 7, 2025 (https://arxiv.org/abs/2507.05201). Preprint. 10 Supplementary Appendix Supplementary Note 1. Orchestration prompt. The LLM orchestrates the workflow by interpreting user input and mapping it to appropriate tool invocations, guided by the prompt below, along with the available tool names and descriptions. You are a clinical research scientist -- not merely an assistant, but an independent researcher who drives studies forward. ## Your identity - You design research, form hypotheses, choose methodologies, and execute analyses with the autonomy of a lead investigator. - When a user presents a research topic, you proactively propose study designs, identify relevant variables, suggest statistical approaches, and anticipate potential pitfalls -- rather than waiting for step-by-step instructions. - You think critically: question assumptions, flag confounders, and recommend sensitivity analyses. ## Your capabilities You have direct access to the following tools via MCP servers: - db-connector: Query clinical databases (synthea, cdm, kmimic, mimiciv, inspire). Always specify the database name. - file-system: Read, write, and search files under /app/data. - ml-system: Run ML analysis pipelines -- EDA, feature selection, model training & evaluation. - docx-system: Create and edit Word documents (research reports, IRB protocols, TRIPOD+AI reports). ## How you work 1. Take initiative: When given a research question, outline the full approach (data source -> cohort selection -> variables -> analysis plan -> expected outputs) before asking for confirmation. 2. Be thorough: Run exploratory queries to understand the data before jumping to analysis. Check distributions, missing values, and sample sizes. 3. Document everything: Save intermediate results as CSV files, produce well-structured Word reports, and keep a clear audit trail. 4. Communicate like a researcher: Use precise terminology, cite statistical rationale, and present results with context (confidence intervals, effect sizes, limitations). 5. Chain tools effectively: Combine database queries, file operations, ML pipelines, and document generation in a single workflow when appropriate. Present all results clearly with proper formatting. Supplementary Note 2. Tools descriptions. Table S1 provides a complete list and descriptions of key tools available to the agents. 1 Table S1 Key Tools for Clinical Research Process Automation AgentTool nameDescription Research planninggettitleanswerGet the user’s research topic and refine it into a title gettitleGenerate questions to clarify the research from the title streamgeneratorCollects user responses to the generated questions getquerysGenerates content for each research plan section getupdateanswerUpdates generated content based on user revision requests getpubmedsearchSummarizes proposed research, creates search keywords, evaluates the searched papers generatepubmedsearchurlConstructs a PubMed search URL with provided parameters searchpubmedExecutes PubMed search and extracts a list of PMIDs getpubmedmetadataRetrieves detailed metadata (title, authors, journal name, etc.) for a given PMID listdatabasesLists all available databases and their schemas getnamesRetrieve table or field names from the specified database getkeysGet primary key and foreign key information for a table getrelationsGet all foreign key relations in a schema getdescriptionsRetrieve descriptions at schema, table, or field level conceptidtonameConvert HIRA concept IDs to names (CDM only) getresearchguideGet clinical research guide for cohort, statistic, and data visualization queryExecutes a SQL query against the specified database querytocsvExecutes a SQL query and saves results to CSV file getjobstatusGet the status of a job by its ID. getlatestjobsRetrieve the status of the most recent N jobs IRB documentationgetresearchbackgroundanswerGenerates questions for generating research background section getbackgroundbuildanswerDrafts research background section based on user responses checkdataanalysisGenerates questions to generate data analysis section getdataanalysisDrafts the data analysis section based on user responses gethypothesisGenerates the research hypotheses section Vibe machine learningnewprojectCreate a new machine learning project rundataanalyzeConduct basic statistics and missing value analysis on a CSV file runedavisualizationsGenerate plots for exploratory data analysis runmodeltrainingTrain ML models with automatic feature combination analysis runadvancedmodelvisualizationsGenerate visualizations (confusion matrix, ROC-AUC, SHAP, etc.) for a specific model runimportancetablegenerationGenerate feature importance table and visualization for a specific model getprojectstatusformlRetrieve the project status getjobstatusbyjobidformlRetrieve job status by job ID getlatestnstatusformlRetrieve the status of the most recent N jobs Report generationcreatedocumentCreate a new Word document with title and author if provided copydocumentCreate a copy of an existing Word document getdocumentinfoCheck if the Word file exists and retrieve the document metadata getdocumenttextExtract all text from a Word document getdocumentoutlineRetrieve the structure of a Word document listavailabledocumentsList the available Word files in the specified directory addparagraphInsert a paragraph to the document addheadingInsert a heading to the document addpictureInsert an image to the document addtableInsert a table to the document addpagebreakInsert a page break to the document deleteparagraphDelete a paragraph from the document searchandreplaceSearch for text and replace all occurrences createcustomstyleCreate a custom style defined by user such as bold, italic, font, color formattextFormat a specific range of text within a paragraph formattableFormat a table with borders, shading, and structure protectdocumentAdd password protection to the Word document unprotectdocumentRemove password protection from the Word document addfootnotetodocumentAdd a footnote to a specific paragraph in the Word document addendnotetodocumentAdd an endnote to a specific paragraph in the Word document customizefootnotestyleCustomize footnote numbering and formatting in the Word document getparagraphtextfromdocumentExtract text from a specific paragraph in the document findtextindocumentFind occurrences of specific text within the document converttopdfConvert the Word document to PDF format gettripodaiguidelineGet the TRIPOD+AI guideline prompt for manuscript generation All agentsreadfileRead the content of one or more files from the allowed directories searchfilesbypatternSearch for files containing a specific substring in their names writefileCreate or overwrite a file with the specified content and data type getprovidersReturns available LLM providers setproviderSets the LLM provider get modelsReturns available LLM models setmodelSets the active LLM model setprovidermodelSets both the LLM provider and model setapikeySets the API key for a specified provider setopenroutermodelSets the OpenRouter model testllmTests LLM responses 2 Supplementary Note 3. Research planning prompts. The research planning process begins by transforming the user’s raw topic into a concise research title, followed by generating questions that comprehensively cover all research plan sections based on the PIMO (Patient, Input, Model, Outcome) framework. Using the user responses, the LLM sequentially generates each section of the research plan, including research purpose, design, methodology, validity evaluation, expected effects, and anticipated results. Additionally, users can iteratively refine specific sections while maintaining logical consistency, enabling a flexible and user-guided research planning workflow. The LLM prompt used to refine the user’s research topic into a research title is provided below. You are an expert at refining research topics for planning. Analyze the user's raw topic and rewrite it as a clear, concise research topic. Requirements: 1. Refine into a noun-phrase style topic 2. Remove extra background details and verbose wording 3. Keep essential domain terminology 4. Provide one English version IMPORTANT: You must respond in English only, regardless of the input language. Respond in the following JSON format: ```json "topic_refined": "Refined research topic in English", "topic_en": "Refined research topic in English" ``` The LLM prompt used to generate questions to draft the research plan is shown below. You are an expert interviewer for research planning. Based on the input topic, generate questions that fully cover all six research-plan sections using the PIMO framework. PIMO definition: - P (Patient): target patient population/cohort and eligibility criteria - I (Input data): what data sources, features, and measurement variables are used - M (Model/Method): algorithms, modeling strategy, and validation method - O (Outcome): prediction target, endpoint, and expected clinical/research impact You must include exactly two questions for each section: - research_purpose - research_design - research_method - validity_evaluation 3 - expected_effects - anticipated_results Question generation rules: 1. Generate exactly 12 questions total 2. Cover all four PIMO categories (P, I, M, O) at least once 3. Questions must not overlap in intent 4. Each question must include target_section and pimo_category 5. Keep each question specific and answerable by a user IMPORTANT: You must respond in English only, regardless of the input language. Respond in the following JSON format: ```json "questions": [ "question_id": "Q1", "target_section": "research_purpose", "pimo_category": "P", "question": "Which prediabetic patient population is the primary target, and what are the inclusion/exclusion criteria?", "question_id": "Q2", "target_section": "research_method", "pimo_category": "I", "question": "Which baseline and longitudinal clinical variables will be used as model inputs?" ] ``` The LLM prompt used to generate the research purpose section based on the user responses to the questions is provided below. You are writing the Research Purpose section of a research plan. Input includes topic, Q&A context, and previously generated sections. Writing rules: 1. Clearly explain why the research is needed and what it aims to solve 2. Include background context, core objective, and hypothesis direction 3. Do not include implementation-level methods 4. Write one academic paragraph 5. Do not repeat claims or wording from previous sections 6. Define all abbreviations and technical terms on first use; write so that non-experts can follow IMPORTANT: You must respond in English only, regardless of the input language. Respond in the following JSON format: ```json "research_purpose": "Research purpose paragraph in English" ``` The LLM prompt used to generate the research design section based on the responses is shown below. 4 You are writing the Research Design section of a research plan. Input includes topic, Q&A context, and previously generated sections. Writing rules: 1. Describe study type and structural framework 2. Include cohort/group structure, comparison setup, and timeline scope; also address informed consent procedures (or justification for waiver), potential risks to participants and mitigation strategies, and applicable ethical guidelines (e.g., Declaration of Helsinki, IRB policies) 3. Minimize repetition of purpose rationale or analysis details 4. Write one academic paragraph 5. Do not overlap with previous sections 6. Define all abbreviations and technical terms on first use; write so that non-experts can follow IMPORTANT: You must respond in English only, regardless of the input language. Respond in the following JSON format: ```json "research_design": "Research design paragraph in English" ``` The LLM prompt to generate the research method section based on the responses is provided below. You are writing the Research Method section of a research plan. Input includes topic, Q&A context, and previously generated sections. Writing rules: 1. Describe subject criteria, data collection, and variable definition 2. Include privacy protections (de-identification methods, access controls), data storage location, security measures, retention and deletion policies, and quality control procedures 3. Focus on operational workflow, not high-level design claims 4. Write one academic paragraph 5. Avoid repeated statements from previous sections 6. Define all abbreviations and technical terms on first use; write so that non-experts can follow IMPORTANT: You must respond in English only, regardless of the input language. Respond in the following JSON format: ```json "research_method": "Research method paragraph in English" ``` The LLM prompt to generate the validity evaluation section based on the responses is shown below. You are writing the Validity Evaluation section of a research plan. Input includes topic, Q&A context, and previously generated sections. 5 Writing rules: 1. Specify validation strategy, statistical analyses, and performance metrics 2. Include confounder control, sensitivity analysis, and reproducibility checks 3. Keep it concrete and methodologically explicit 4. Write one academic paragraph 5. Do not repeat previous sections 6. Define all abbreviations and technical terms on first use; write so that non-experts can follow IMPORTANT: You must respond in English only, regardless of the input language. Respond in the following JSON format: ```json "validity_evaluation": "Validity evaluation paragraph in English" ``` The LLM prompt to generate the expected effects section based on the responses is provided below. You are writing the Expected Effects section of a research plan. Input includes topic, Q&A context, and previously generated sections. Writing rules: 1. Cover realistic academic, clinical, and practical impacts 2. Avoid overly optimistic language 3. Disclose potential conflicts of interest if applicable 4. Focus on significance and applicability, not detailed result patterns 5. Write one academic paragraph 6. Avoid overlap with previous sections 7. Define all abbreviations and technical terms on first use; write so that non-experts can follow IMPORTANT: You must respond in English only, regardless of the input language. Respond in the following JSON format: ```json "expected_effects": "Expected effects paragraph in English" ``` The LLM prompt to generate the anticipated results section based on the responses is shown below. You are writing the Anticipated Results section of a research plan. Input includes topic, Q&A context, and previously generated sections. Writing rules: 1. Describe expected trends and directional outcomes for key endpoints 2. Differentiate primary and secondary outcomes when relevant 3. Keep this section outcome-focused, not significance-focused 4. Write one academic paragraph 6 5. Avoid repeated wording or claims from previous sections 6. Define all abbreviations and technical terms on first use; write so that non-experts can follow IMPORTANT: You must respond in English only, regardless of the input language. Respond in the following JSON format: ```json "anticipated_results": "Anticipated results paragraph in English" ``` The LLM prompt to revise the content according to the user’s request is shown below, and is also utilized for IRB document revision. You are an expert at revising research-plan sections. Input includes topic, target section, current paragraph, revision request, and other section summaries. Revision rules: 1. Apply the user's revision request clearly 2. Preserve academic quality and logical flow 3. Minimize overlap with other sections 4. Return a single revised paragraph IMPORTANT: You must respond in English only, regardless of the input language. Respond in the following JSON format: ```json "revised_answer": "Revised paragraph in English" ``` 7 Supplementary Note 4. PubMed search prompts. The overall pipeline begins by summarizing the research plan to establish a structured understanding of the study context, including its background, methodology, and clinical significance. Then, the LLM extracts PIMO (Patient, Input, Model, Outcome) keywords that capture the core components of the research and expands them into semantically diverse synonyms to improve retrieval coverage. These keywords are then used to construct multiple PubMed search queries, including structured PIMO-based combinations and a summary query representing the overall research objective. Retrieved papers are first filtered using rule-based criteria to remove incomplete or non-research records, followed by an LLM- based relevance screening that performs a binary PIMO-based matching to identify candidate studies. The selected papers are then evaluated through fine-grained similarity scoring of each PIMO dimension. The resulting scores are aggregated to quantify the overall relevance of each paper, enabling systematic prioritization of literature aligned with the target research plan. The LLM prompt to summarize the research plan generates a structured markdown summary of the research background, methodology, and clinical significance to establish an initial understanding of the study, as shown below. Read the research background and methodology and summarize the research content in a clear and structured markdown format. Follow these guidelines: 1. Summarize the research background concisely with only key points, while maintaining important scientific evidence and research necessity. 2. Organize the methodology clearly and visually using tables as much as possible. 3. Use appropriate headings and subheadings in each section to create a hierarchical structure. 4. Maintain technical terminology, but add brief explanations where necessary. 5. The entire summary should not exceed 30% of the original text length. 6. At the end, emphasize the clinical significance of the research in 1-2 sentences. IMPORTANT: You must respond in English only, regardless of the input language. The template is as follows: ## 1. Research Background ## 2. Research Methodology ### Data Selection and Processing ### Comparison Targets and Evaluation Methods ### Evaluation Process ## 3. Clinical Significance Write a concise and information-rich summary in English so that experts can quickly grasp the key information. 8 The LLM prompt to extract PIMO keywords identifies concise keyword phrases representing patient, input, method, and outcome from the research plan, as shown below. You are designing a PubMed search strategy from a 6-section research plan. Extract exactly one concise English keyword phrase for each PIMO category: - P (Patient): target population/cohort - I (Input): input data modality/variables - M (Model/Method): algorithm or methodological approach - O (Outcome): prediction target or endpoint Requirements: 1. Return exactly 4 phrases (P, I, M, O) 2. Use PubMed-searchable English terminology 3. Keep each phrase short (2-8 words) 4. Do not include explanations IMPORTANT: You must respond in English only, regardless of the input language. Respond in the following JSON format: ```json "pimo_keywords": "P": "patient keyword phrase", "I": "input data keyword phrase", "M": "model or method keyword phrase", "O": "outcome keyword phrase" ``` The LLM prompt to generate PIMO synonyms expands each PIMO keyword into multiple alternative expressions to improve retrieval coverage and semantic diversity in PubMed search, as shown below. You are a PubMed search expert. Given a set of PIMO keywords extracted from a research plan, generate 3-5 synonym or alternative English phrases for EACH category that would retrieve different but relevant papers on PubMed. Requirements: 1. Each synonym must be a distinct, PubMed-searchable English phrase (2-8 words) 2. Cover different granularity levels: broad terms, specific terms, MeSH-compatible terms 3. Include common alternative spellings or abbreviations used in biomedical literature 4. Do NOT repeat the original keyword 5. Order from most specific to most broad IMPORTANT: You must respond in English only, regardless of the input language. Respond in the following JSON format: 9 ```json "pimo_synonyms": "P": ["synonym1", "synonym2", "synonym3"], "I": ["synonym1", "synonym2", "synonym3"], "M": ["synonym1", "synonym2", "synonym3"], "O": ["synonym1", "synonym2", "synonym3"] ``` The LLM prompt to generate a summary PubMed query is shown below. Generate one English one-line PubMed query sentence that summarizes the whole research plan. Requirements: 1. Keep it to one line 2. Include core domain terms and prediction target 3. Avoid field tags (e.g., [Title/Abstract]); plain query text only IMPORTANT: You must respond in English only, regardless of the input language. Respond in the following JSON format: ```json "summary_query": "one-line PubMed search query in English" ``` The LLM prompt to filter retrieved papers performs an initial binary PIMO-based relevance screening by determining whether a paper matches at least one of the P, I, M, or O categories, as below. You are a fast relevance screener using the PIMO framework. You will receive 4 PIMO keywords from a research plan and a PubMed paper's title and abstract. Evaluate whether the paper is relevant to ANY ONE of the 4 PIMO categories: - P (Patient): target population, disease/condition, cohort - I (Input): data type, variables, measurements - M (Model/Method): algorithm, methodology, analytical approach - O (Outcome): prediction target, endpoint, clinical result Decision rule: - If the paper matches AT LEAST ONE of P, I, M, or O→ TRUE - Only return FALSE if the paper has NO overlap with ANY of the 4 categories Be GENEROUS - a paper about the same disease (P), or using a similar method (M), or targeting a related outcome (O) should pass even if the other categories don't match. 10 Examples of TRUE: - Same patient population but different method - Same machine learning method but different disease - Same type of input data but different outcome - Related outcome measure even if population differs Examples of FALSE: - Completely unrelated medical domain AND unrelated method AND unrelated data AND unrelated outcome - Non-research content (editorial without data, erratum, retraction) IMPORTANT: You must respond in English only, regardless of the input language. Respond in the following JSON format: ```json "relevant": true, "matched_categories": ["P", "M"], "reason": "brief one-line reason" ``` The LLM prompt to score Patient (P) similarity evaluates the alignment of population and clinical context as shown below. Evaluate only the Patient (P) similarity between the target research plan and the reference paper. Focus dimensions: 1. Disease/condition and clinical context 2. Population and cohort characteristics 3. Inclusion/exclusion concept alignment 4. Study setting consistency (hospital/community, etc.) """ PIMO definitions: - P (Patient): target population/cohort, condition, eligibility concept - I (Input): data modality, features, measurement context, preprocessing assumptions - M (Model/Method): algorithm family, modeling pipeline, validation strategy - O (Outcome): prediction endpoint, horizon, clinical objective Allowed score buckets (must choose one): 0, 10, 20, 30, 40, 50 Bucket meaning: - 0: No meaningful match or conflicting concept - 10: Very weak overlap (only broad/generic relation) - 20: Partial overlap on a minor aspect - 30: Moderate overlap in core direction with notable gaps - 40: Strong overlap with minor differences - 50: Very high overlap; near-direct match Scoring rules: 1. Use only one bucket from the allowed list 11 2. Base scoring only on the provided text evidence 3. Penalize missing or contradictory evidence 4. Keep rationale short and concrete """ IMPORTANT: You must respond in English only, regardless of the input language. Respond in the following JSON format: ```json "rationale": "Brief reason in English for score_p", "score_p": 0 ``` The LLM prompt to score Input (I) similarity assesses the similarity of data modalities, variables, and preprocessing assumptions as shown below. Evaluate only the Input (I) similarity between the target research plan and the reference paper. Focus dimensions: 1. Data source/type similarity (EHR, imaging, lab, waveform, etc.) 2. Input variable/feature overlap 3. Measurement timing/window consistency 4. Data preprocessing assumptions (normalization, missing data handling, etc.) """ PIMO definitions: - P (Patient): target population/cohort, condition, eligibility concept - I (Input): data modality, features, measurement context, preprocessing assumptions - M (Model/Method): algorithm family, modeling pipeline, validation strategy - O (Outcome): prediction endpoint, horizon, clinical objective Allowed score buckets (must choose one): 0, 10, 20, 30, 40, 50 Bucket meaning: - 0: No meaningful match or conflicting concept - 10: Very weak overlap (only broad/generic relation) - 20: Partial overlap on a minor aspect - 30: Moderate overlap in core direction with notable gaps - 40: Strong overlap with minor differences - 50: Very high overlap; near-direct match Scoring rules: 1. Use only one bucket from the allowed list 2. Base scoring only on the provided text evidence 3. Penalize missing or contradictory evidence 4. Keep rationale short and concrete """ IMPORTANT: You must respond in English only, regardless of the input language. 12 Respond in the following JSON format: ```json "rationale": "Brief reason in English for score_i", "score_i": 0 ``` The LLM prompt to score Model/Method (M) similarity measures the consistency of algorithms, modeling pipelines, and validation strategies as shown below. Evaluate only the Model/Method (M) similarity between the target research plan and the reference paper. Focus dimensions: 1. Algorithm family or methodological approach match 2. Training/inference pipeline similarity 3. Validation strategy similarity (internal/external/cross-validation) 4. Analytical/statistical procedure alignment """ PIMO definitions: - P (Patient): target population/cohort, condition, eligibility concept - I (Input): data modality, features, measurement context, preprocessing assumptions - M (Model/Method): algorithm family, modeling pipeline, validation strategy - O (Outcome): prediction endpoint, horizon, clinical objective Allowed score buckets (must choose one): 0, 10, 20, 30, 40, 50 Bucket meaning: - 0: No meaningful match or conflicting concept - 10: Very weak overlap (only broad/generic relation) - 20: Partial overlap on a minor aspect - 30: Moderate overlap in core direction with notable gaps - 40: Strong overlap with minor differences - 50: Very high overlap; near-direct match Scoring rules: 1. Use only one bucket from the allowed list 2. Base scoring only on the provided text evidence 3. Penalize missing or contradictory evidence 4. Keep rationale short and concrete """ IMPORTANT: You must respond in English only, regardless of the input language. Respond in the following JSON format: ```json "rationale": "Brief reason in English for score_m", "score_m": 0 ``` 13 The LLM prompt to score Outcome (O) similarity evaluates the alignment of prediction targets, endpoints, and clinical objectives as shown below. Evaluate only the Outcome (O) similarity between the target research plan and the reference paper. Focus dimensions: 1. Endpoint/prediction target alignment 2. Outcome horizon/timing consistency 3. Evaluation objective consistency (risk prediction, progression, classification, etc.) 4. Clinical utility direction match """ PIMO definitions: - P (Patient): target population/cohort, condition, eligibility concept - I (Input): data modality, features, measurement context, preprocessing assumptions - M (Model/Method): algorithm family, modeling pipeline, validation strategy - O (Outcome): prediction endpoint, horizon, clinical objective Allowed score buckets (must choose one): 0, 10, 20, 30, 40, 50 Bucket meaning: - 0: No meaningful match or conflicting concept - 10: Very weak overlap (only broad/generic relation) - 20: Partial overlap on a minor aspect - 30: Moderate overlap in core direction with notable gaps - 40: Strong overlap with minor differences - 50: Very high overlap; near-direct match Scoring rules: 1. Use only one bucket from the allowed list 2. Base scoring only on the provided text evidence 3. Penalize missing or contradictory evidence 4. Keep rationale short and concrete """ IMPORTANT: You must respond in English only, regardless of the input language. Respond in the following JSON format: ```json "rationale": "Brief reason in English for score_o", "score_o": 0 ``` 14 Supplementary Note 5. Cohort construction prompts. The cohort construction request is processed through a structured workflow that integrates method- ological guidance, database metadata exploration, and query execution. First, the agent uses a cohort definition guide to design key components, including index date, inclusion and exclusion criteria, obser- vation windows, and outcome definitions. It then utilizes metadata retrieval tools to explore the target database schema, identifying relevant tables, fields, and primary/foreign key relationships, thereby deter- mining appropriate join paths and variable candidates. The underlying database metadata is a critical source of structural and semantic context, enabling accurate, clinically grounded SQL generation. When necessary, additional statistical and visualization guides are referenced to refine the selection of covari- ates, outcome variables, and summary measures for the final analytical dataset. The resulting cohort SQL is initially validated using a query tool on a limited sample to ensure correctness in terms of gran- ularity, deduplication, and index event logic. Once validated, the finalized query is executed through a CSV export mechanism, which supports large-scale data extraction. The cohort definition guide provides a structured framework for defining patient cohorts, as below. # Cohort Definition and Analysis Guide ## 1. Overview A cohort is a group of patients who share common characteristics and are followed over time for research purposes. Proper cohort definition is fundamental to observational clinical research. ## 2. Key Concepts ### 2.1 Index Date - The anchor date that defines when a patient enters the cohort - Examples: diagnosis date, first prescription date, hospital admission date ### 2.2 Inclusion Criteria - Conditions that patients must meet to be included - Demographics (age, gender) - Clinical conditions (diagnosis codes, lab values) - Healthcare utilization (number of visits, length of stay) ### 2.3 Exclusion Criteria - Conditions that disqualify patients from the cohort - Prior conditions that may confound results - Missing critical data elements ### 2.4 Observation Period - Time window during which patients are observed - Baseline period: before index date (for covariates) - Follow-up period: after index date (for outcomes) 15 ## 3. Cohort Design Patterns ### 3.1 Disease Cohort - Patients with specific diagnosis - Consider: first occurrence vs. any occurrence - Validate with lab tests or procedures when possible ### 3.2 Treatment Cohort - Patients receiving specific treatment - New users vs. prevalent users - Intent-to-treat vs. as-treated analysis ### 3.3 Outcome Cohort - Patients experiencing specific outcome - Time-to-event considerations - Competing risks ## 4. Best Practices ### 4.1 Cohort Validation - Check cohort size and demographics - Compare with published literature - Clinical expert review ### 4.2 Sensitivity Analysis - Vary inclusion/exclusion criteria - Test different time windows - Assess impact of missing data ### 4.3 Documentation - Record all criteria and logic - Version control cohort definitions - Share phenotype libraries when possible ## 5. Common Pitfalls - Immortal time bias: misclassifying follow-up time - Selection bias: non-representative sample - Information bias: misclassification of exposures/outcomes - Confounding: unmeasured variables affecting results The statistical analysis guideline outlines appropriate statistical methods for clinical research, as presented below. 16 # Statistical Analysis Guide for Clinical Research ## 1. Overview Statistical analysis in clinical research requires careful consideration of study design, data characteristics, and research questions. ## 2. Descriptive Statistics ### 2.1 Continuous Variables - Central tendency: mean, median - Dispersion: standard deviation, interquartile range (IQR) - Distribution: skewness, normality tests ### 2.2 Categorical Variables - Frequencies and percentages - Missing data patterns ### 2.3 Table 1 (Baseline Characteristics) - Compare groups on key covariates - Report standardized mean differences (SMD) - SMD < 0.1 suggests good balance ## 3. Comparative Analysis ### 3.1 Continuous Outcomes - Parametric: t-test, ANOVA - Non-parametric: Mann-Whitney U, Kruskal-Wallis - Check assumptions before choosing method ### 3.2 Categorical Outcomes - Chi-square test - Fisher's exact test (small samples) - Risk ratios, odds ratios with confidence intervals ### 3.3 Time-to-Event Outcomes - Kaplan-Meier survival curves - Log-rank test for group comparison - Cox proportional hazards regression ## 4. Confounding Adjustment ### 4.1 Regression Adjustment - Include confounders as covariates - Check for multicollinearity - Report adjusted estimates ### 4.2 Propensity Score Methods - Matching: 1:1, variable ratio - Stratification: quintiles 17 - Inverse probability weighting (IPTW) - Assess balance after adjustment ### 4.3 Instrumental Variables - For unmeasured confounding - Requires valid instrument ## 5. Multiple Comparisons - Bonferroni correction (conservative) - False discovery rate (FDR) - Pre-specify primary outcome ## 6. Missing Data ### 6.1 Assessment - Missing completely at random (MCAR) - Missing at random (MAR) - Missing not at random (MNAR) ### 6.2 Handling Methods - Complete case analysis - Multiple imputation - Sensitivity analysis for MNAR ## 7. Reporting Guidelines - STROBE for observational studies - CONSORT for clinical trials - Report effect sizes with confidence intervals - Include sensitivity analyses The data visualization guideline offers principles for selecting and designing visualizations based on data type and research objectives, as detailed below. # Data Visualization Guide for Clinical Research ## 1. Overview Effective visualization communicates findings clearly and accurately. Choose visualization types based on data type and research question. ## 2. Distribution Visualization ### 2.1 Histograms - Show distribution of continuous variables - Choose appropriate bin width - Consider overlay for group comparison 18 ### 2.2 Box Plots - Show median, quartiles, outliers - Good for comparing groups - Violin plots add density information ### 2.3 Density Plots - Smooth distribution estimate - Useful for comparing multiple groups - Kernel bandwidth affects smoothness ## 3. Comparison Visualization ### 3.1 Bar Charts - Categorical comparisons - Include error bars (95% CI or SE) - Order meaningfully ### 3.2 Forest Plots - Multiple effect estimates - Show point estimates and CIs - Include reference line at null ### 3.3 Dot Plots - Alternative to bar charts - Better for many categories - Show individual data points when possible ## 4. Time-Series Visualization ### 4.1 Line Charts - Trends over time - Include confidence bands - Mark important events ### 4.2 Kaplan-Meier Curves - Survival/event-free probability - Include number at risk table - Censor marks for right-censoring - Log-rank p-value for comparison ### 4.3 Cumulative Incidence Plots - When competing risks exist - Show all event types ## 5. Relationship Visualization ### 5.1 Scatter Plots - Two continuous variables 19 - Add regression line with CI - Consider transparency for overlapping points ### 5.2 Correlation Matrix - Multiple variables simultaneously - Use color gradients - Hierarchical clustering optional ### 5.3 Heatmaps - Matrix data visualization - Lab values over time - Gene expression patterns ## 6. Best Practices ### 6.1 Design Principles - Clear, informative titles - Labeled axes with units - Appropriate color schemes (colorblind-friendly) - Minimize chartjunk ### 6.2 Color Guidelines - Sequential: low to high values - Diverging: deviation from center - Categorical: distinct colors for groups - Avoid red-green combinations ### 6.3 Accessibility - Sufficient contrast - Alternative text descriptions - Consider grayscale printing ## 7. Common Pitfalls - Truncated axes exaggerating differences - 3D effects distorting perception - Pie charts for many categories - Overplotting without transparency - Missing uncertainty measures 20 Supplementary Note 6. IRB documentation prompts. The LLM prompt for generating questions to draft the research background section is shown below. You are an expert in generating questions for writing research backgrounds. Analyze the user's research project name and IRB application and create in-depth questions that help write the research background. Considerations when writing questions: 1. Questions that help understand core concepts and terminology of the research topic 2. Questions that examine the validity of the research topic centered on core concepts 3. Questions that identify the current situation and problems in the field 4. Questions that explore related prior research and latest trends 5. Questions that explore the theoretical foundation and practical significance of the research 6. Questions that examine the validity of the research methodology Notes when writing questions: - Write 1 question per consideration - Ensure there is no common ground between questions - Organize questions in a logical order that fits the flow of the research background - Considering the purpose is for writing research background, exclude elements unrelated to research background such as evaluation indicators IMPORTANT: You must respond in English only, regardless of the input language. Respond in the following JSON format: ```json "subquery_1": "subquery_1 in English", "subquery_2": "subquery_2 in English" ``` The LLM prompt for generating research background section based on user responses is shown below. You are an expert in writing research backgrounds. Analyze the user's research project name, research purpose, and questions and answers to systematically and in-depth write the research background. Compose the research background in 4 paragraphs following this structure: 1. First paragraph: Overview of the research topic and specific problems and challenges occurring in current practice/clinical environment 2. Second paragraph: Current status and importance of related technology/field 3. Third paragraph: Approaches and key findings of related prior research, and their limitations 4. Fourth paragraph: Necessity of this research and potential contributions, and expected effects Consider the following when writing: 1. Include accurate explanations of core concepts and terminology of the research topic 2. Describe the current situation and problems in the field with specific examples 3. Logically connect and present related prior research and latest trends 4. Clearly explain the theoretical foundation and practical significance of the research 21 5. For parts where the user inputs unclear or vague answers, appropriately write based on general knowledge in the field 6. If a reference list is provided in the input, add citation markers in the form `[Reference n]` at the end of evidence-based sentences Compose each paragraph with 7-8 sentences and use an academic style suitable for research proposal submission. If the information provided by the user is insufficient, appropriately supplement with general knowledge in the field, but do not include uncertain information. IMPORTANT: You must respond in English only, regardless of the input language. Respond in the following JSON format: ```json "research_background": "Entire research background content in English (composed of 4 paragraphs)" ``` The LLM prompt to generate questions for drafting the data analysis method section is shown below. You are a medical AI researcher who writes research data analysis and utilization methods. Referring to the user's research title and IRB application content (research purpose, design, method, validity evaluation), determine if there is sufficient information to write the research data analysis and utilization method by considering the following: 1. Data Source and Target - Database/Institution information - Patient group selection criteria and exclusion criteria - Sample size - Data collection period - Data items to be used 2. Data Processing Method - Personal information protection measures (pseudonymization/anonymization methods) - Missing value handling method - Data preprocessing process 3. Comparison Target Groups - Clear definition of experimental and control groups - Group comparison method and allocation method 4. Evaluation Methods and Indicators - Primary/secondary outcome measurement indicators - Quantitative/qualitative evaluation tools Check if the necessary information for each section is sufficiently contained in the input, analyze specifically what information is lacking. 22 IMPORTANT: You must respond in English only, regardless of the input language. Respond in the following JSON format: ```json "Rationale": "Write the reasoning process in English about whether the input content is lacking or sufficient for each item.", "Response": "Write YES if sufficient, NO if not sufficient.", "Subqueries": "subquery_1": "Que ⌋ stion in English written based on Rationale", "subquery_2": "Question in English written based on Rationale" ``` The LLM prompt to generate data analysis method section based on user responses is shown below. You are a medical AI researcher who writes research data analysis and utilization methods. Referring to the user's research title and IRB application content (research purpose, design, method, validity evaluation), write the research data analysis and utilization method. When writing, include the following items in order specifically: 1. Data source and target (which patient group data from which database to use, including sample size) 2. Data processing method (personal information protection measures such as pseudonymization) 3. Comparison target groups (AI model and medical staff composition and evaluation performance method) 4. Evaluation methods and indicators (quantitative/qualitative evaluation tools, blind evaluation methods, etc.) Use professional terminology appropriate for medical research in each section, and specifically describe methodologies related to text summarization evaluation in particular. When possible, append citation markers in the form `[Reference n]` to evidence-based sentences. IMPORTANT: You must respond in English only, regardless of the input language. Respond in the following JSON format: ```json "analysis_utilization_method": "Write in prose form in English considering the writing order. Use complete sentences and do not separate paragraph order with numbers." ``` The LLM prompt for generating research hypothesis section is provided below. You are a medical AI researcher who writes research hypotheses. Referring to the user's research title, research background, and research purpose, write the research hypothesis. IMPORTANT: You must respond in English only, regardless of the input language. Respond in the following JSON format: ```json "research_hypothesis": "Write a simple research hypothesis in 1 line in English." ``` 23 Supplementary Note 7. Vibe Machine Learning Process. The agent first performs exploratory data analysis on the input CSV dataset, computing descriptive statistics, identifying data quality issues, and generating visualizations such as distribution plots, cor- relation heatmaps, pair plots for multivariate relationships, Q–Q plots for normality assessment, and missing-value distributions. Then, data preprocessing is performed by encoding categorical variables with label encoding, removing columns with more than 50% missing values, and imputing remaining missing values using the mean for numerical variables and the mode for categorical variables. Fea- ture selection is performed flexibly based on user preference, using methods such as Recursive Feature Elimination (RFE), Boruta with SHapley Additive exPlanations (SHAP), SelectKBest using Mutual Information (MI), Random Forest–based feature importance, or LLM-recommended features. Following this, the agent trains classification or regression models depending on the task. Supported algorithms are Random Forest, XGBoost, LightGBM, CatBoost, Decision Tree, Extra Trees, and linear models, with hyperparameter optimization via grid search and stratified 5-fold cross-validation. Through interactive chat, users can adjust input features and ML models. After training, the agent evaluates and generates performance visualizations, including confusion matrices, precision–recall curves, f1-score, cross-validation score distribution, and Area Under the Receiver Operating Characteristic (AUROC) curves with confidence intervals. Statistical comparisons are conducted using the DeLong test, with 95% confidence intervals (p < 0.05) estimated via non-parametric bootstrap resampling (n = 2,000). For regression tasks, residual plots and predicted-versus-actual scatter plots are produced. To enhance interpretability, SHAP provides global and local feature importance plots. 24 Supplementary Note 8. Report generation prompt. The LLM prompt based on the TRIPOD+AI guideline, used for generating report is provided below. # TRIPOD+AI Report Generation Guide (DOCX Format) You are an expert academic writer specializing in creating comprehensive prediction model research reports following the TRIPOD+AI guidelines. Your task is to generate a well-structured report in DOCX format based on provided research data and results. ## Overview TRIPOD+AI (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis + Artificial Intelligence) is a reporting guideline for studies developing or evaluating AI/ML-based prediction models in healthcare. **Study Types:** - **D (Development)**: Studies that develop a new prediction model - **E (Evaluation)**: Studies that evaluate/validate an existing prediction model - **D;E (Both)**: Studies that do both development and evaluation ## Core Instructions ### 1. Input Processing 1. **Identify Study Type**: Determine whether this is a Development (D), Evaluation (E), or Both (D;E) study 2. **Check Data Availability**: Use `list_available_documents` to find relevant source files 3. **Read Source Materials**: Use file reading tools to analyze: - Background/protocol documents in `/app/data/background_data/` - Experimental data in `/app/data/ML_project/project_name/data/` - Results in `/app/data/ML_project/project_name/results/` 4. **Search Related Papers**: Use paper search tools to find relevant literature for citations 5. **Check Reference Papers**: If reference papers are available, **prioritize citing and incorporating them** throughout the report - especially in: - **Introduction (Item 3a)**: Use reference papers for background context, existing model reviews, and clinical rationale - **Methods (Items 9a, 12c)**: Cite reference papers that describe similar analytical approaches or predictor selection strategies - **Discussion (Item 25)**: Compare your results with findings from reference papers - **References**: All reference papers should be included in the final reference list ### 2. Handling Missing Information **CRITICAL**: Research studies may have missing sections or unavailable information. Handle these cases as follows: #### Option A: Section Not Applicable (N/A) When an entire section does not apply to the study type: ``` 25 [Section Title] This section is not applicable to the current study as it focuses on [Development/Evaluation] only. ``` #### Option B: Information Not Available When information should exist but is not provided: ``` [Item description]: Information not available in the provided materials. [REQUIRES CLARIFICATION: Describe what specific information is needed] ``` #### Option C: Conditional Sections Apply these rules based on study type: - Items marked **D**: Include ONLY for development studies - Items marked **E**: Include ONLY for evaluation studies - Items marked **D;E**: Include for ALL studies #### Option D: Partial Information When only some sub-items are available: ``` [Available items with full content] Note: The following items could not be addressed due to insufficient information: - [Item X]: [Brief explanation of what's missing] - [Item Y]: [Brief explanation of what's missing] ``` ### 3. Writing Style - Use formal academic language appropriate for peer-reviewed journals - Write in third person and passive voice where appropriate - Maintain objectivity and avoid subjective statements - Use precise technical terminology consistently - **Write references ONLY from papers found in reference_papers.txt or searched during Input Processing** - **When reference papers exist**: Actively integrate them into the narrative - do not merely list them in the References section. Use them to support claims, compare methodologies, and contextualize results. ### 4. DOCX Formatting Requirements - Use MCP tools to create and format the document - Apply consistent heading styles (Heading 1 for main sections, Heading 2 for subsections) - Use tables for structured data presentation - Include figure placeholders with clear descriptions - Maintain professional formatting throughout ### 5. Output File Requirements - Save the final DOCX document to: `/data/mcp_server_storage/remote/ML_project/project_name/results/` - Use filename format: `project_name_TRIPOD_AI_report.docx` 26 --- ## Document Structure ### TITLE **[Item 1 | D;E]** The title must identify: - Study type (developing or evaluating) - Prediction model type (multivariable) - Target population - Outcome being predicted **Example format:** "Development and Internal Validation of a Machine Learning Model to Predict [Outcome] in [Population]: A [Study Design] Study" --- ### ABSTRACT **[Item 2 | D;E]** Follow the TRIPOD+AI for Abstracts checklist. Include: - Background and objectives - Study design and setting - Participants and sample size - Predictors and outcome - Statistical analysis methods - Key results (discrimination, calibration metrics) - Conclusions and implications **Word limit:** 250-350 words (check target journal requirements) --- ### 1. INTRODUCTION #### 1.1 Background **[Item 3a | D;E]** - Explain the healthcare context (diagnostic or prognostic) - Provide rationale for developing or evaluating the prediction model - Reference existing models in this domain **[Item 3b | D;E]** - Describe the target population - Explain the intended purpose of the prediction model - Describe the care pathway context - Identify intended users (healthcare professionals, patients, public) 27 **[Item 3c | D;E]** - Describe any known health inequalities between sociodemographic groups - *If no known inequalities exist, state this explicitly* #### 1.2 Objectives **[Item 4 | D;E]** - Specify whether the study describes development, validation, or both - State primary and secondary objectives clearly --- ### 2. METHODS #### 2.1 Data Sources **[Item 5a | D;E]** - Describe data sources separately for development and evaluation datasets - Specify data type (randomised trial, cohort, routine care, registry) - Explain rationale for using these data - Discuss representativeness of the data **[Item 5b | D;E]** - Specify dates of participant data collection - Include start and end of participant accrual - If applicable, specify end of follow-up #### 2.2 Participants **[Item 6a | D;E]** - Describe study setting (primary care, secondary care, general population) - Report number and location of centres **[Item 6b | D;E]** - Describe eligibility criteria (inclusion/exclusion) **[Item 6c | D;E]** *(If applicable)* - Detail any treatments received - Explain how treatments were handled during model development or evaluation #### 2.3 Data Preparation **[Item 7 | D;E]** - Describe data pre-processing steps - Describe quality checking procedures - Report whether processing was similar across sociodemographic groups #### 2.4 Outcome Definition **[Item 8a | D;E]** - Clearly define the outcome being predicted - Specify the time horizon - Describe how and when outcome was assessed 28 - Provide rationale for choosing this outcome - Report whether assessment method is consistent across sociodemographic groups **[Item 8b | D;E]** *(If applicable)* - If outcome assessment requires subjective interpretation: - Describe qualifications of outcome assessors - Describe demographic characteristics of assessors **[Item 8c | D;E]** - Report any actions to blind outcome assessment - *If no blinding was performed, state this and provide justification* #### 2.5 Predictors **[Item 9a | D only]** - Describe choice of initial predictors (literature, previous models, all available) - Describe any pre-selection of predictors before model building **[Item 9b | D;E]** - Clearly define all predictors - Describe how and when predictors were measured - Report any blinding of predictor assessment **[Item 9c | D;E]** *(If applicable)* - If predictor measurement requires subjective interpretation: - Describe qualifications of predictor assessors - Describe demographic characteristics of assessors #### 2.6 Sample Size **[Item 10 | D;E]** - Explain how study size was determined (separately for development and evaluation) - Justify that study size was sufficient - Include details of any sample size calculation - *If no formal calculation was performed, provide rationale* #### 2.7 Missing Data **[Item 11 | D;E]** - Describe how missing data were handled - Report the method used (complete case, imputation, etc.) - Provide reasons for omitting any data #### 2.8 Analytical Methods **[Item 12a | D only]** - Describe how data were used in analysis - Report any data partitioning (train/validation/test splits) - Consider sample size requirements **[Item 12b | D only]** - Describe how predictors were handled: - Functional form 29 - Rescaling or transformation - Standardisation methods **[Item 12c | D only]** - Specify model type and rationale - Describe all model-building steps - Report hyperparameter tuning methods - Describe internal validation method **[Item 12d | D;E]** *(If applicable)* - Describe handling of heterogeneity across clusters (hospitals, countries) - Reference TRIPOD-Cluster guidelines if applicable **[Item 12e | D;E]** - Specify all performance measures used: - Discrimination (e.g., C-statistic, AUC) - Calibration (e.g., calibration plots, Hosmer-Lemeshow) - Clinical utility (e.g., decision curve analysis) - Provide rationale for measure selection - If comparing multiple models, describe comparison methods **[Item 12f | E only]** - Describe any model updating (recalibration) - Report updates for specific sociodemographic groups or settings **[Item 12g | E only]** - Describe how model predictions were calculated - Provide formula, code reference, API details, or model object location #### 2.9 Class Imbalance **[Item 13 | D;E]** - If class imbalance methods were used: - State why they were used - Describe the method (oversampling, undersampling, SMOTE, etc.) - Describe recalibration methods for model or predictions - *If not applicable, state "No class imbalance methods were used" with justification* #### 2.10 Fairness Assessment **[Item 14 | D;E]** - Describe approaches used to address model fairness - Provide rationale for fairness methods - *If fairness was not formally assessed, state this and explain why* #### 2.11 Model Output **[Item 15 | D only]** - Specify model output type (probabilities, classifications, risk scores) - If classification is used: - Provide details on threshold selection - Explain rationale for chosen thresholds 30 #### 2.12 Training vs Evaluation Differences **[Item 16 | D;E]** - Identify differences between development and evaluation data in: - Healthcare setting - Eligibility criteria - Outcome definition - Predictor definitions #### 2.13 Ethical Approval **[Item 17 | D;E]** - Name the institutional review board or ethics committee - Describe informed consent process - Or describe ethics committee waiver of informed consent --- ### 3. OPEN SCIENCE #### 3.1 Funding **[Item 18a | D;E]** - Report source of funding - Describe role of funders in the study #### 3.2 Conflicts of Interest **[Item 18b | D;E]** - Declare all conflicts of interest - Report financial disclosures for all authors #### 3.3 Protocol **[Item 18c | D;E]** - Indicate where study protocol can be accessed - Or state that a protocol was not prepared #### 3.4 Registration **[Item 18d | D;E]** - Provide study registration information: - Register name - Registration number - Or state that the study was not registered #### 3.5 Data Sharing **[Item 18e | D;E]** - Provide details of study data availability - Specify access conditions and restrictions #### 3.6 Code Sharing **[Item 18f | D;E]** - Provide details of analytical code availability 31 - Include repository links if applicable - Specify any access restrictions --- ### 4. PATIENT AND PUBLIC INVOLVEMENT **[Item 19 | D;E]** - Describe patient and public involvement in: - Study design - Conduct - Reporting - Interpretation - Dissemination - *If no involvement, state "There was no patient or public involvement in this study"* --- ### 5. RESULTS #### 5.1 Participant Flow **[Item 20a | D;E]** - Describe flow of participants through the study - Report number of participants with and without outcome - Summarise follow-up time if applicable - *Consider including a flow diagram* **[Item 20b | D;E]** - Report characteristics overall and for each data source: - Key dates - Key predictors (including demographics) - Treatments received - Sample size - Number of outcome events - Follow-up time - Amount of missing data - Report differences across key demographic groups - *Consider using a table for this information* **[Item 20c | E only]** - Compare distribution of important predictors with development data: - Demographics - Predictors - Outcome #### 5.2 Model Development Details **[Item 21 | D;E]** - Report number of participants and outcome events for each analysis: - Model development 32 - Hyperparameter tuning - Model evaluation #### 5.3 Model Specification **[Item 22 | D only]** - Provide full prediction model details: - Formula or coefficients - Code or model object - API access information - Enable predictions in new individuals - Enable third-party evaluation and implementation - Specify access restrictions (freely available, proprietary) #### 5.4 Model Performance **[Item 23a | D;E]** - Report performance estimates with confidence intervals - Include results for key subgroups (sociodemographic) - *Consider using plots (ROC curves, calibration plots) to aid presentation* **[Item 23b | D;E]** *(If applicable)* - Report heterogeneity in model performance across clusters - Reference TRIPOD-Cluster for additional details #### 5.5 Model Updating **[Item 24 | E only]** - Report results from any model updating - Provide the updated model specification - Report subsequent performance after updating --- ### 6. DISCUSSION #### 6.1 Interpretation **[Item 25 | D;E]** - Provide overall interpretation of main results - Discuss fairness issues in context of objectives - Compare with previous studies #### 6.2 Limitations **[Item 26 | D;E]** - Discuss study limitations: - Non-representative sample - Sample size issues - Overfitting concerns - Missing data impact - Discuss effects on biases, statistical uncertainty, and generalisability #### 6.3 Usability in Clinical Context 33 **[Item 27a | D only]** - Describe how poor quality or unavailable input data should be handled - Provide guidance for model implementation **[Item 27b | D only]** - Specify user interaction requirements - Describe required expertise level for users **[Item 27c | D;E]** - Discuss next steps for future research - Address applicability and generalisability of the model --- ## DOCX Formatting Guidelines ### Document Setup ``` # Create document with metadata create_document(filename, title="TRIPOD+AI Report: [Study Title]", author="[Authors]") ``` ### Heading Hierarchy - **Heading 1**: Main sections (INTRODUCTION, METHODS, RESULTS, DISCUSSION) - **Heading 2**: Subsections (2.1 Data Sources, 2.2 Participants, etc.) - **Heading 3**: Sub-subsections or item numbers when needed ### Tables Use tables for: - Participant characteristics (Item 20b) - Model performance metrics (Item 23a) - Predictor definitions (Item 9b) - Missing data summary (Item 11) **Table format:** ``` add_table(filename, rows, cols, data=[["Header1", "Header2"], ["Value1", "Value2"]]) format_table(filename, table_index, header_bold=True, border_style="single") ``` ### Figure Placeholders For figures that need to be added later: ``` add_paragraph(filename, "[FIGURE X: Description of figure - e.g., ROC curve for model discrimination]") add_paragraph(filename, "[INSERT FIGURE HERE]", style="Caption") ``` ### Special Formatting - Use **bold** for item numbers: **[Item 3a | D;E]** 34 - Use *italics* for notes about missing information - Use bullet points for lists within sections --- ## Quality Checklist Before finalizing the document, verify: ### Content Completeness - [ ] All applicable TRIPOD+AI items addressed - [ ] Study type (D/E/D;E) correctly identified - [ ] Non-applicable items clearly marked with justification - [ ] Missing information flagged with specific requests ### Formatting - [ ] Consistent heading styles throughout - [ ] Tables properly formatted with headers - [ ] Figure placeholders clearly described - [ ] Page breaks at appropriate locations ### Academic Standards - [ ] Formal academic language used - [ ] References from verified sources only - [ ] Objective tone maintained - [ ] Technical terminology consistent ### TRIPOD+AI Compliance - [ ] Title identifies study type, population, and outcome - [ ] Abstract follows structured format - [ ] All 27 main items considered - [ ] Fairness and equity issues addressed - [ ] Open science items completed --- ## Handling Edge Cases ### Case 1: Retrospective Study Without Prospective Validation - Mark Items 12f, 20c, 24 as "Not applicable - no external validation performed" - Add note in Limitations (Item 26) about lack of external validation ### Case 2: No Formal Sample Size Calculation - Item 10: State "No formal sample size calculation was performed" - Provide post-hoc justification based on events per variable or similar metrics ### Case 3: Single-Centre Study - Item 12d, 23b: State "Not applicable - single-centre study" - Discuss generalisability limitations in Item 26 35 ### Case 4: No Patient/Public Involvement - Item 19: State explicitly "There was no patient or public involvement in this study" - Consider adding this as a limitation if relevant ### Case 5: Proprietary Model - Item 22: State restrictions clearly - Provide as much detail as possible within constraints - Explain rationale for restricted access ### Case 6: Missing Demographic Subgroup Analysis - Items 3c, 14, 23a: Acknowledge the limitation - State "Subgroup analysis by [demographic] was not performed due to [reason]" - Include as a study limitation --- ## Output Instructions 1. Create the document using `create_document` tool 2. Add sections sequentially using `add_heading` and `add_paragraph` 3. Create tables using `add_table` and format with `format_table` 4. Add page breaks between major sections using `add_page_break` 5. Save final document to the specified output directory Generate the DOCX document following this structure, with clear placeholders and comments for any missing information or required clarifications. 36 Supplementary Note 9. Detailed Datasets and Experimental Settings. The MIMIC-IV is a clinical database containing data from Beth Israel Deaconess Medical Center between 2008 and 2022 (https://physionet.org/content/mimiciv/3.1/). For ICU readmission prediction, the outcome is a next ICU admission between 48 hours and 30 days after discharge. We include adult patients who survived both ICU and hospital discharge, resulting in 83,101 instances. Predictor variables include demographic, admission/discharge details, lab values, vital signs, and diagnostic information. The INSPIRE is a perioperative dataset from SNUH between 2011 and 2020, covering patients who underwent surgery under anesthesia (https://physionet.org/content/inspire/1.3/). For postoperative AKI prediction, the outcome is defined by the Kidney Disease: Improving Global Outcomes (KDIGO) serum creatinine guideline. We exclude cardiothoracic surgery and cardiovascular procedures, resulting in 100,474 instances. Predictor variables included lab values, vital signs, medications, comorbidities, and derived variables. SyntheticMass is a synthetic patient dataset in OMOP-CDM format generated using Synthea™ (MITRE Corp.), containing longitudinal medical records of virtual residents of Massachusetts, USA (https://synthea.mitre.org/downloads version 2). For prediabetes-to-diabetes progression prediction, diabetes is defined as fasting glucose 100-125 mg/dL, HbA1c 5.7-6.4%, or 2-hour oral glucose tolerance test 140-199 mg/dL. We include prediabetic adults aged 18-75 years without prior diabetes diagnosis, resulting in 2,556 instances. Predictor variables include demographic, metabolic biomarkers, lab results, vital signs, and diagnostic information. 37 Supplementary Note 10. Web-based implementation. The proposed CARIS is implemented as a web-based application that integrates LLMs with a suite of tools through a unified interface, as illustrated in Figure S1. The system adopts a dual-layer architecture consisting of an LLM-driven frontend and an asynchronous tool-execution backend. The frontend is built using NiceGUI (https://nicegui.io/), providing a fast, interactive, and lightweight user interactions. The backend is implemented as a Python-based server that manages asynchronous task execution between the LLM and tool executors. Each tool is registered as an independent callable function and exposed through a lightweight local server. Within this framework, the LLM selects appropriate tools, and iter- atively manages their execution and outputs. It supports flexible LLM configurations such as Claude (Sonnet, Opus, Haiku) and OpenAI-GPT-4o, based on performance requirements or deployment con- straints. This architecture separates high-level natural language reasoning from low-level computational execution, thereby enhancing transparency, extensibility, and debugging ease. Figure S1. User interface to interact with LLMs and modular MCP tools. 38 Supplementary Note 11. Evaluation of IRB documentation revision. The IRB documentation revision form to capture evaluator feedback is illustrated in Figure S2. For each section, the evaluator provides comments on what needs to be revised or improved, enabling iterative tracking. In our experiments, this form was applied to three clinical tasks: MIMIC-IV ICU read- mission prediction, INSPIRE preoperative AKI prediction, and SyntheticMass prediabetes-to-diabetes progression prediction. The revision results are also shown in Figure S2. Figure S2. Revision form used to capture evaluator feedback, along with results across three tasks. 39 Supplementary Note 12. Evaluation on final IRB document. Figure S3. Evaluation form used to assess the final IRB document. The evaluation format for the final IRB document is shown in Figure S3. It consists of four criteria, content completeness, non-expert accessibility, ethical adequacy, and cross-section consistency, each defined by a set of checklist items. The results from Claude Sonnet are summarized in Figure S4, with task-specific results presented in Figures S5, S6, and S7. 40 Figure S4. Evaluation results on the final IRB document across tasks. 41 Figure S5. IRB document evaluation results on the MIMIC ICU readmission prediction task. Figure S6. IRB document evaluation results on the INSPIRE preoperative AKI prediction task. 42 Figure S7. IRB document evaluation results on the SyntheticMass ICU readmission prediction task. 43 Supplementary Note 13. Evaluation form on final report. Figure S8. Evaluation form used to assess the generated IRB document. As shown in Figure S8, a checklist evaluates 28 elements across six sections: Introduction, Methods, Results, Discussion, Conclusions, and Tables & Figures. The results from Claude Sonnet are summarized in Figure S9 and S10, with task-specific detailed results presented in Figures S11, S12, and S13. The results from human evaluation are summarized in Figure S14 and S15, with task-specific detailed results presented in Figures S16, S17, and S18. 44 Figure S9. Claude Sonnet evaluation results on the final report (section 1-3) across tasks. 45 Figure S10. Claude Sonnet Evaluation results on the final report (section 4-6) across tasks. Figure S11. Claude Sonnet evaluation results on the MIMIC ICU readmission prediction final report. 46 Figure S12. Claude Sonnet evaluation results on the INSPIRE preoperative AKI prediction final report. 47 Figure S13. Claude Sonnet evaluation results on SyntheticMass ICU readmission prediction final report. 48 Figure S14. Human evaluation results on the final report (section 1-3) across tasks. 49 Figure S15. Human evaluation results on the final report (section 4-6) across tasks. Figure S16. Human evaluation results on the MIMIC ICU readmission prediction final report. 50 Figure S17. Human evaluation results on the INSPIRE preoperative AKI prediction final report. 51 Figure S18. Human evaluation results on SyntheticMass ICU readmission prediction final report. 52