Paper deep dive
Analogical Reasoning as a Doctor: A Foundation Model for Gastrointestinal Endoscopy Diagnosis
Peixi Peng, Housheng Xie, Yanling Wei, Guangcong Ruan, Xiaoyang Zou, Qian Cao, Yongjian Nian, Guoyan Zheng
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 4/10/2026, 3:17:45 AM
Summary
RATNet is a foundation model for gastrointestinal endoscopy diagnosis that utilizes an analogical reasoning mechanism to transfer knowledge from heterogeneous expert annotations across multiple datasets. It addresses challenges like data scarcity, domain shift, and annotation heterogeneity, outperforming existing models like GastroNet and GastroVision in tasks such as common disease diagnosis, few-shot learning for rare diseases, and zero-shot transfer.
Entities (6)
Relation Signals (3)
RATNet → uses → Analogical Reasoning
confidence 100% · RATNet, a foundation model for gastrointestinal endoscopy imaging based on analogical reasoning.
RATNet → outperforms → GastroNet
confidence 95% · Evaluations show that RATNet outperforms existing foundation models, including GastroNet and GastroVision
RATNet → supports → Federated Learning
confidence 95% · privacy-preserving deployment via federated learning
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Gastrointestinal diseases impose a growing global health burden, and endoscopy is a primary tool for early diagnosis. However, routine endoscopic image interpretation still suffers from missed lesions and limited efficiency. Although AI-assisted diagnosis has shown promise, existing models often lack generalizability, adaptability, robustness, and scalability because of limited medical data, domain shift, and heterogeneous annotations. To address these challenges, we develop RATNet, a foundation model for gastrointestinal endoscopy imaging based on analogical reasoning. RATNet acquires and transfers knowledge from heterogeneous expert annotations across five gastrointestinal endoscopy datasets through a cyclic pre-training strategy. Its architecture consists of an encoder, a relevance-knowledge acquisition and transfer (RAT) module, a projector, and a multi-task head, and supports fine-tuning, linear probing, and zero-shot transfer. Evaluations show that RATNet outperforms existing foundation models, including GastroNet and GastroVision, across six scenarios: diagnosis of common gastrointestinal diseases, few-shot learning for rare diseases, zero-shot transfer to new medical sites, robustness under long-tailed disease distributions, adaptation to novel diseases, and privacy-preserving deployment via federated learning. Its advantage comes from an analogical reasoning mechanism that matches image-derived posterior knowledge to a learned prior knowledge base and transfers relative knowledge to guide diagnosis, improving generalization and resistance to bias. RATNet is open and cost-effective, supports automatic integration of heterogeneous annotations without manual label unification, and reduces data acquisition costs, making it a practical foundation for intelligent gastrointestinal diagnosis, especially in resource-limited settings.
Tags
Links
- Source: https://arxiv.org/abs/2604.05649v1
- Canonical: https://arxiv.org/abs/2604.05649v1
Trouble viewing inline? Open PDF directly →
Full Text
78,226 characters extracted from source content.
Expand or collapse full text
Analogical Reasoning as a Doctor: A Foundation Model for Gastrointestinal Endoscopy Diagnosis Peixi Peng 1† , Housheng Xie 1† , Yanling Wei 2† , Guangcong Ruan 2† , Xiaoyang Zou 1 , Qian Cao 3 , Yongjian Nian 2 , Guoyan Zheng 1* 1* Institute of Medical Robotics, School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China. 2 Department of Gastroenterology, Daping Hospital, Army Medical University (Third Military Medical University), Chongqing, 400042, China. 3 Department of Gastroenterology, Sir Run Run Shaw Hospital, Zhejiang University School of Medicine, Hangzhou, 310016, China. *Corresponding author(s). E-mail(s): guoyan.zheng@sjtu.edu.cn; Contributing authors: xiandyxi@sjtu.edu.cn; housheng.xie@sjtu.edu.cn; lingzi016@tmmu.edu.cn; ruanguangcong@tmmu.edu.cn; xiaoyang.zou@sjtu.edu.cn; caoq@zju.edu.cn; yongjian nian@163.com; † These authors contributed equally to this work. Abstract Gastrointestinal diseases represent a growing global health burden, for which endoscopy serves as a primary tool for early diagnosis. However, routine endoscopic image diagnosis still suffers from missed lesions and suboptimal efficiency. While Artificial intelligence (AI)-assisted diagnosis has shown great promise, existing AI models remain limited in generalizability, adaptability, robustness, and scalabil- ity, primarily due to medical data scarcity, domain shift, and annotation heterogeneity. To address these limitations, we developed a relevance-knowledge acquisition and transfer network (RATNet), which is a foundation model for gastrointestinal endoscopy imaging based on analogical reasoning. RATNet acquires and transfers knowledge from heterogeneous expert annotations across five gas- trointestinal endoscopy datasets through a cyclic pre-training strategy. Its architecture comprises an encoder, a RAT module, a projector, and a multi-task head, supporting fine-tuning, linear probing, and zero-shot transfer. Comprehensive evaluations demonstrate that RATNet outperforms existing foundation models (e.g., GastroNet, GastroVision) across six clinical scenarios: it accurately diag- noses common gastrointestinal diseases, learns from few samples for rare diseases, achieves zero-shot transfer to new medical sites, handles long-tailed disease distributions, adapts to novel diseases, and safeguards patient privacy via federated learning. The superior performance stems from its analogi- cal reasoning mechanism, which mimics clinicians’ cognition by matching the image-derived posterior knowledge to a learned prior knowledge base and transferring the acquired relative knowledge to guide diagnosis, thereby improving generalization and bias resistance. RATNet is an open and cost-effective model that supports automatic integration of heterogeneous annotations without manual label uni- fication, significantly reducing data acquisition costs. By fostering collaboration and open access, RATNet is poised to become a cornerstone for intelligent gastrointestinal diagnosis, accelerating AI adoption in resource-limited settings. Keywords: Gastrointestinal diseases, Endoscopy, Analogical reasoning, Foundation model 1 arXiv:2604.05649v1 [cs.CV] 7 Apr 2026 1 Introduction Gastrointestinal (GI) diseases are increasingly prevalent worldwide [1, 2], with digestive tract cancers long occupying a prominent position in the global cancer burden [3]. The prognosis for these cancers is generally poor, particularly at advanced stages, where survival rates drop significantly [4], underscoring the critical importance of early screening. Endoscopy serves as the primary diagnostic tool for detecting early pathological changes [5, 6]. However, current practice relying on manual endoscopic image diagnosis faces challenges such as missed diagnoses and limited efficiency in image review. Artificial intelligence (AI)-assisted diagnosis offers a promising approach to address these limitations [7–9]. Foundation models, which are large-scale deep learning architectures pre-trained on extensive and diverse datasets, are reshaping the landscape of artificial intelligence in healthcare. By training on vast and diverse data, these models learn universal feature representations and can be fine-tuned for specific tasks with minimal additional data [10, 11]. Leveraging their comprehensive pre-training, they demon- strate diagnostic accuracy for anomalies that surpasses traditional methods and even human experts, while also exhibiting strong generalization to clinical scenarios beyond their initial training scope [12]. This enables a wide range of applications from radiology [13] and pathology [14] to endoscopy [15]. Consequently, foundation models hold promise for democratizing expert-level gastrointestinal diagnostic capabilities, bridging the gap in endoscopic expertise for resource-limited or remote areas. Although foundation models offer a promising paradigm for developing generalizable and annotation- efficient solutions for gastrointestinal visual tasks, their success heavily depends on access to large and diverse datasets. However, in the field of gastrointestinal diseases, large-scale datasets are relatively scarce, and many foundation models [16–18] have not been made fully publicly available, thereby hinder- ing further progress within the research community. Fully open public datasets [19–22] are often small and suffer from inconsistent annotations in terms of disease coverage (i.e., the label heterogeneity prob- lem). This makes it challenging to train powerful and robust foundation models by aggregating numerous small public datasets. Furthermore, endoscopic imaging involves considerable real-world variability [23] due to diverse equipment, imaging protocols, and patient populations. This variability leads to signif- icant differences in the data distribution across datasets, a problem commonly referred to as domain shift [24]. Existing foundation models lack the ability to extract domain-related knowledge from multi- ple distinct domains. Consequently, when encountering an unseen target domain, they cannot effectively transfer knowledge from source domains based on domain similarity [25], leading to performance degra- dation. For instance, the GastroNet-5M dataset [15] contains endoscopic images recorded using three types of endoscope equipment: Olympus, Pentax, and Fujifilm. The foundation model GastroNet [15], pre-trained on GastroNet-5M, generally fails to distinguish between these three domains. When this model performs inference on the PolypGen dataset [26], which consists solely of images captured with Olympus equipment, it tends to exhibit a bias toward domains with more abundant pre-training data, rather than prioritizing knowledge from the Olympus domain, which shares the same equipment type as PolypGen [26]. This limitation undermines the model’s reliability. Research on human cognition offers a promising avenue for addressing the aforementioned limita- tions, primarily through the application of analogical reasoning. Analogical reasoning [27], which is the ability to perceive and use relational similarity between two situations or events, is a fundamental aspect of human cognition and serves as a core process in scientific discovery, problem-solving, categorization, and decision-making. This mechanism is an important element of various cognitive abilities including inference and abstraction [28], enables humans to identify abstract patterns in seemingly disparate envi- ronments, transfers knowledge from familiar domains to unfamiliar ones, and generates creative solutions for novel problems [29]. In medical contexts, analogical reasoning mirrors cognitive processes clinicians used for diagnosis. Durning et al. [30] highlight its role in two key stages: pattern recognition during data gathering, where doctors identify relational similarities in symptoms, and hypothesis generation, drawing inferences from analogous cases. Inspired by this, as shown in Figure 1, we propose a foundation model that mimics a clinician’s analogical reasoning process, termed the Relevance-knowledge Acquisition and Transfer Network (RATNet). During training, RATNet iteratively acquires prior knowledge from vari- ous datasets. During inference, it estimates posterior knowledge from the input sample and compares it with various prior knowledge from multiple sources to identify relevant associations. This facilitates the transfer of knowledge from the source to the target domain, enabling effective reasoning in unseen domains. This foundation model can be trained by aggregating numerous public and private datasets without requiring label unification, facilitating broader collaboration within the research community. 2 Non-Polyp Polyp CP-CHILD Kvasir Cecum Polyp UC BBPS-2-3 Mayo-3 Mayo-2 Hyper-Kvasir BBPS-0-1 Mayo-1 Cecum Polyp Impacted- stool Learning Prior-Knowledge Kvasir HyperKvasir HyperKvasir CPCHILD CPCHILD Prior Knowledge Similar Endoscopy Hospital Analogical Reasoning as a Doctor Diagnostic ResultsGastrointestinal ImageFoundation Model Posterior Knowledge Posterior Knowledge Relevance-Knowledge Acquisition Relevance-Knowledge Transfer Building Knowledge Base Fig. 1: Workflow of the base model RATNet. This model utilizes gastrointestinal images obtained from medical endoscopy to provide diagnostic assistance. RATNet simulates the analogical reasoning process of physicians by first learning from multiple datasets to construct a prior knowledge base. It then estimates posterior knowledge from the current gastrointestinal images. By comparing and acquiring relevant knowledge from the prior knowledge base, the model transfers this knowledge to the domain of the posterior knowledge, enabling effective reasoning for diagnostic outcomes. Furthermore, RATNet can be iteratively refined and enhanced by acquiring prior knowledge from new datasets, enabling the model to adapt to the evolving diagnostic needs. This continuously iterative pro- cess of analogical reasoning facilitates robust generalization and predictive inference, while mitigating challenges such as data scarcity and domain-specific overfitting. Such advancements contribute to the progress of AI-driven healthcare, ultimately leading to improved patient outcomes. RATNet is pre-trained by cyclically acquiring and transferring knowledge embedded within heteroge- neous expert annotations from five gastrointestinal endoscopy datasets. It comprises four key components: an encoder, a RAT module, a projector, and a multi-task classifier. These components facilitate flexi- ble transition to clinical applications through three distinct paradigms: full fine-tuning, linear probing, or zero-shot transfer. Full fine-tuning entails updating all network parameters to maximize adaptability, allowing the model to capture domain-specific features and achieve peak performance when sufficient data is available. In contrast, linear probing (or head-only training) keeps the pre-trained backbone entirely frozen and only trains a newly added linear classifier. This computationally efficient method serves as a benchmark to evaluate the quality of the learned representations. Finally, zero-shot transfer assesses the model’s inherent generalization by directly deploying the pre-trained network to diagnose novel conditions in unseen datasets without any task-specific retraining or prior exposure to the samples. To comprehensively evaluate the capability of RATNet in gastrointestinal disease diagnosis, we con- ducted extensive benchmarking across ten independent datasets. Five datasets were specifically allocated for in-domain validation, namely CP-CHILD [19], Kvasir [21], HyperKvasir [22], LIMUC [20], and Dap- ing. Of these, Daping is a private dataset, while the others are publicly available. The remaining six datasets were used for cross-institutional out-of-domain generalization assessment, including Colono- scopic [31], PolypGen [26], GastroVision [32], Kvasir-Capsule [33], and Shaoyifu. Among these, only Shaoyifu is a private dataset, while the others are publicly accessible (Extended Data Table 1). Please note that Kvasir, HyperKvasir, and Kvasir-Capsule are three distinct datasets, each with unique images and corresponding labels, and they are independent of one another. In-domain validation was performed on held-out test sets from datasets involved in the pretraining phase, primarily examining model per- formance under similar data distributions. Out-of-domain testing, targeted completely unseen datasets from different medical institutions, encompassing diverse populations and imaging protocols, aiming to simulate distribution shifts commonly encountered in clinical practice and thereby reveal the model’s robustness in real-world deployment. Through these diverse data resources, we focused on evaluating RATNet’s performance across six critical downstream tasks: classification of common gastrointestinal 3 diseases, rare disease recognition under few-shot conditions, handling of long-tailed imbalanced distribu- tions, diagnostic environment transfer without fine-tuning, zero-shot response to emerging diseases, and privacy-preserving distributed learning based on federated pretraining. Experimental results demonstrate that RATNet significantly outperforms existing foundation models across multiple metrics including gen- eralization capability, adaptability, robustness, and scalability, exhibiting stronger potential for clinical applications. 2 Results 2.1 Diagnose common gastrointestinal diseases We conducted an internal evaluation of the RATNet model using test data from four public gastrointesti- nal endoscopy image datasets including CP-CHILD [19], Kvasir [21], HyperKvasir [22], LIMUC [20], and one private dataset termed Daping. Additionally, we compared its performance on these datasets against several large pre-trained models, including GastroNet [15], SSL [35], GastroHUN [36], and GastroVi- sion [32]. As illustrated in Figure 3(a), RATNet achieved AUC scores above 90% across all five datasets, demonstrating its generalizability. Furthermore, we compared RATNet with several large vision-language models on the test set of the ColonINST dataset [37]. As shown in Extended Data Table 2, RATNet outperformed large vision-language models trained on large-scale datasets, highlighting its ability to iteratively accumulate and leverage knowledge from diverse datasets, thereby improving data utilization efficiency. To evaluate the quality of the model features, we performed linear probing on five models across the HyperKvasir, LIMUC, and Daping datasets using the Mayo endoscopic score. These three datasets were selected for evaluation since they are the only ones that include the Mayo endoscopic score. The Mayo endoscopic score was chosen for this assessment because it is a widely recognized and clinically relevant metric for determining the severity of colonic mucosal inflammation in patients with inflammatory bowel disease (IBD). Its widespread use among clinicians highlights its importance in both clinical decision- making and model evaluation [38]. Furthermore, given that the score assesses the severity of colonic mucosal inflammation, the scoring system requires models to identify subtle differences in image fea- tures [39] and accurately distinguish between different levels of inflammation. Using the Mayo endoscopic score in linear probing provides a comprehensive evaluation of the model’s ability to extract and repre- sent visual features across different datasets. As shown in Figure 3(b), our RATNet model maintained high AUC values (all exceeding 89%) for Mayo endoscopic score estimation across different datasets, outperforming all other models. This demonstrates RATNet’s superior ability in visual feature extraction. The RAT module acquires prior knowledge from previous tasks to construct a prior knowledge base, and then transfers prior knowledge associated with posterior knowledge to the current task. To demon- strate the effectiveness of our proposed RAT module, we employed t-SNE [34] to project and visualize the prior and posterior knowledge features of the RATNet model across five different tasks from the pre- training dataset. Specifically, we extracted posterior knowledge features from the test sets of CP-CHILD, Kvasir, HyperKvasir, LIMUC, and Daping using RATNet, and projected them together with the pre- trained prior knowledge base into a shared two-dimensional space. As shown in Figure 3(c), the model effectively learns distinct prior knowledge from the five different tasks in the pre-training dataset, with no overlap between them. Moreover, the prior and posterior knowledge of the same task exhibit greater sim- ilarity. Additionally, the fact that the posterior knowledge features of different tasks cluster in separate regions indicates that the RAT module, by repeatedly acquiring and transferring relevance-knowledge, acquires a strong ability to distinguish posterior knowledge features across tasks. 2.2 Learn rare conditions from a few samples Precise detection of uncommon gastrointestinal pathologies is vital for optimizing therapeutic inter- ventions and patient prognosis. However, their infrequent presentation creates persistent obstacles for clinicians. The paucity of labeled instances markedly impedes the ability of standard deep learning sys- tems to develop reliable feature representations. A central metric for gauging the clinical relevance of vision foundation models lies in their robustness when adapting to low-prevalence entities with minimal supervision. To probe this capacity in RATNet, we conducted few-shot evaluations using a linear probing protocol, benchmarking against established pretrained architectures. 4 Encoder Rat Module Projector C Pretraining CecumPolyp UC BBPS-0-1BBPS-2-3Cecum Impacted-stool Mayo-1Mayo-2Mayo-3Polyp Hyper-KvasirKvasir Mayo-0Mayo-1 Mayo-2Mayo-3 LIMUC CDMayo-0Mayo-1 Mayo-2Mayo-3 Daping Non-PolypPolyp CP-CHILD Encoder Linear Classifier Hyperplasic Adenomatous Serrated Fine-tuning Encoder Linear Classifier Hyperplasic Adenomatous Serrated Linear-probing Encoder Rat Module Projector C Dataset Classifier 0.8 0.1 0.1 13 5 Dataset probability Classifier 1 Classifier 3 Classifier 5 CD 0.7 Mayo30.1 Weighted Sum CD 0.4 Mayo30.1 CD 0.3 Mayo30.1 Mayo3 CD Zero-shot Transfer Multi-task Classifier Server CecumPolyp UC Kvasir Mayo-0Mayo-1 Mayo-2Mayo-3 LIMUC Student 2 Teacher 2 Cyclic Training 2 Local data 2 BBPS-0-1BBPS-2-3Cecum Impacted-stool Mayo-1Mayo-2Mayo-3Polyp Hyper-Kvasir Non-PolypPolyp CP-CHILD Local data 3 Student 3 Teacher 3 Cyclic Training 3 CDMayo-0Mayo-1 Mayo-2Mayo-3 Daping Local data 1 Student 1 Teacher 1 Cyclic Training 1 Local Site 2 Local Site 3 Local Site 1 a b c Clinical Decision Support Clinical Application Clinical Decision Support Clinical Application Clinical Decision Support Clinical Application Learnable Frozen Learnable FrozenLearnable Learnable Fig. 2: Relevance-knowledge acquisition and transfer network (RATNet) is a foundation model for gastrointestinal disease diagnosis. (a) RATNet is pre-trained cyclically using heterogeneous expert anno- tations from five gastrointestinal endoscopy datasets: CP-CHILD [19], LIMUC [20], HyperKvasir [22], Daping, and Kvasir [21]. (b) The model comprises four pre-trained components: encoder, RAT module, projector, and multi-task head, adaptable via fine-tuning, linear probing, or zero-shot transfer. Fine- tuning trains the entire model with a new classifier on target data. Linear probing trains only a new classifier on pre-extracted embeddings. Zero-shot transfer aggregates predictions from all classification heads using domain similarity weights, requiring no additional training. (c) In federated learning, local RATNet models train at each site using cyclic training for student models and EMA for teacher models. After each round, student weights are averaged at a central server to create a global model, which is redistributed for iterative improvement. 5 RATNet GastroNet GastroHun SSL GastroVision Non-Polyp Polyp CP-CHILD Mayo-0 Mayo-1 Mayo-2Mayo-3 LIMUC CDMayo-0 Mayo-1 Mayo-2 Mayo-3 Daping Prior Knowledge Kvasir HyperKvasir Daping CPCHILD LIMUC BBPS-0-1BBPS-2-3CecumImpacted-stool Mayo-1Mayo-2Mayo-3 Polyp Hyper-Kvasir Cecum Polyp UC Kvasir a b c Fig. 3: Diagnostic performance for common gastrointestinal diseases on internal test sets. a. Compari- son of five large pre-trained models in diagnosing gastrointestinal diseases across five internal test sets. The radar charts show the average scores for metrics such as AUC, MCC, F1, and AP across all diseases within each dataset, highlighting the overall diagnostic performance of each model. b. Comparison of the domain generalization ability of five large pre-trained models. We compared the AUC scores of the five models for Mayo endoscopic scoring across three internal test sets, including LIMUC [20], Daping, and HyperKvasir [22], to assess their generalization ability. c. Two-dimensional t-distributed Stochastic Neighbor Embedding (t-SNE) [34] visualization of prior and posterior knowledge acquired by RATNet from the pre-training dataset. This visualization was used to evaluate whether the model can correctly distinguish posterior knowledge from different tasks and successfully extract a representative prior knowl- edge base. In this study, we selected three lesion categories that are underrepresented in the colonoscopy dataset: serrated adenomas, hyperplastic lesions, and conventional adenomas. None of these categories appeared in any pretraining corpus, ensuring that the model had no exposure to them during pretraining. For each trial, we randomly sampled k examples from each of the three categories and used them to train a linear classifier. Model performance was then evaluated on a held out colonoscopy test set. Figure 4 shows box and whisker plots summarizing the distribution of AUC scores across 100 repeated experiments for all models. Across most experiments, RATNet achieved higher median and maximum AUC scores with 6 Fig. 4: Evaluation of the few-shot learning capability for detecting rare conditions. The adenomatous, hyperplastic, and serrated classes from a colonoscopic dataset [31] were used to simulate a low-data scenario for rare disease detection. The adaptability and robustness of five pre-trained models were evaluated under a k-shot learning setting. The box plot illustrates the distribution of AUC scores across 100 experimental runs, showing the median (center line), interquartile range (box boundaries at the 25th and 75th percentiles), whiskers (1.5 times the interquartile range), and outliers as individual points. P- values from two-sided independent t-tests are reported in the figure. a smaller interquartile range compared to the baseline model GastroNet [15], highlighting its superior performance. As shown in Figure 4(a) and (c), all models yielded AUC scores below 70% in the 1- shot scenario. This reflects the inherent difficulty of detecting rare diseases after training on extremely limited data. These results underscore RATNet’s effective capture of discriminative visual patterns, enabling dependable recognition of sparse gastrointestinal abnormalities and highlighting its suitability for deployment in annotation-limited diagnostic settings. 2.3 Transfer to new sites without training Distribution shifts arising from differences in patient demographics, imaging devices, or acquisition parameters across institutions frequently compromise the robustness and predictive performance of deep learning systems in medical imaging analysis. While approaches such as transfer learning and domain adaptation can mitigate these distribution mismatches, they typically necessitate labeled examples from the target environment for model adjustment, constraining the broader applicability and efficient scaling of large-scale pre-trained architectures. To investigate RATNet’s capacity for applying acquired pre-training representations to novel clini- cal settings without additional optimization, we conducted experiments on a new public dataset named PolypGen [26] and a new private dataset named Shaoyifu. Originating from independent institutions, these datasets encompass varied diagnostic objectives, including polyp identification and Mayo endo- scopic subscore assessment. By leveraging the domain similarity between the current task and all training datasets, RATNet weighted and aggregated the prediction results of all existing multi-task prediction heads based on the target categories. This approach enabled inference in a fully zero-shot configuration, 7 RATNet :AUC [95% CI] (%) = 92.26 [91.74, 92.75] :AUC [95% CI] (%) = 58.23 [57.06, 59.49] :AUC [95% CI] (%) = 62.39 [61.21, 63.55] :AUC [95% CI] (%) = 44.77 [43.68, 45.98] :AUC [95% CI] (%) = 62.72 [61.50, 63.90] GastroNet GastroHun GastroVision SSL RATNet :AUC [95% CI] (%) = 91.22 [90.28, 92.15] :AUC [95% CI] (%) = 49.96 [47.94, 51.91] :AUC [95% CI] (%) = 33.48 [31.68, 35.33] :AUC [95% CI] (%) = 61.08 [59.19, 62.98] :AUC [95% CI] (%) = 45.33 [43.33,47.23] GastroNet GastroHun GastroVision SSL RATNet :AUC [95% CI] (%) = 94.24 [93.61, 94.83] :AUC [95% CI] (%) = 41.50 [39.90, 42.98] :AUC [95% CI] (%) = 49.02 [47.43, 50.60] :AUC [95% CI] (%) = 35.21 [33.76, 36.65] :AUC [95% CI] (%) = 50.65 [49.21, 52.12] GastroNet GastroHun GastroVision SSL RATNet :AUC [95% CI] (%) = 89.21 [88.38, 90.02] :AUC [95% CI] (%) = 81.05 [79.87,82.18] :AUC [95% CI] (%) = 35.93 [34.53, 37.49] :AUC [95% CI] (%) = 38.33 [36.84, 39.79] :AUC [95% CI] (%) = 34.14 [32.74,35.51] GastroNet GastroHun GastroVision SSL Fig. 5: Performance on recognizing common gastrointestinal diseases in new settings without training. The generalizability and robustness of RATNet were evaluated via zero-shot transfer on two unseen datasets (PolypGen [26] and Shaoyifu) and four disease categories previously observed during pre-training (Mayo1, Mayo2, Mayo3, and Polyp). Although these two datasets were not included in pre-training, RATNet had previously encountered the four disease categories in other datasets used for pre-training, specifically Mayo1, Mayo2, and Mayo3 from LIMUC [20], DAPING, and HyperKvasir [22], and Polyp from CP-CHILD [19] and Kvasir [21]. Therefore, predictions for the four diseases were obtained directly from the pre-trained RATNet head without further fine-tuning, which we refer to as zero-shot transfer. For comparison, two supervised models (GastroHUN [36] and Gastrovision [32]) and two self-supervised models (GastroNet [15] and SSL [35]) were evaluated under the same conditions. CI, confidence interval of the ROC curves. without any task-specific adaptation. Benchmarking involved four additional pre-trained architectures assessed under equivalent zero-shot protocols. As illustrated in Figure 5 through ROC curves inclusive of 95% confidence bands, the zero-shot out- comes for clinical target identification reveal RATNet’s notable AUC achievements: 92.26% (Mayo 1), 91.22% (Mayo 2), 94.24% (Mayo 3), and 89.21% (polyp identification)—substantially exceeding all com- parators. Such outcomes highlight RATNet’s robust extension of learned features to unfamiliar sites and heterogeneous diagnostic applications in the absence of retraining, affirming its suitability for practical clinical integration. 2.4 Handle long-tailed gastrointestinal diseases In gastrointestinal endoscopy, abnormality detection commonly faces significant challenges due to imbal- anced class prevalence. Prevalent pathologies overwhelmingly dominate clinical observations, whereas uncommon disorders are infrequently encountered, leading to highly uneven data distributions. This disparity creates substantial difficulties in model training, as algorithms often emphasize dominant cat- egories, resulting in reduced sensitivity to infrequent conditions and potential oversight of critical rare findings. Moreover, the paucity of examples for less common diseases heightens the risk of overfitting, thereby limiting effective generalization to new cases. As a result, achieving robust performance and strong adaptability under imbalanced conditions serves as a vital benchmark for assessing advanced foundation models. To investigate RATNet’s capabilities in such imbalanced settings, we utilized the independent pub- lic GastroVision dataset [32]. This resource includes a broad assortment of categories—ranging from anatomical landmarks and diverse pathologies to post-polypectomy scenarios and routine negative obser- vations—thus capturing the full diversity of real-world endoscopic encounters. RATNet was evaluated on the GastroVision dataset, which contains 22 classes with a long-tailed disease distribution. We assessed its performance under both fine tuning (FT) and linear probing (LP) settings. The results were benchmarked against the fine tuning and linear probing performance reported for GastroNet [15], GastroHUN [36], and GastroVision [32]. Figures 6(a) and 6(b) illustrate RATNet’s superior accuracy on both high-frequency (head) and low-frequency (tail) conditions relative to the baselines. In particular, linear probing produced an average AUC of 97.11±0.09% across 22 conditions for RATNet, clearly outperforming three other 8 Fig. 6: Performance for long-tailed gastrointestinal diseases. We evaluated four pre-trained models (Gas- troHUN [36], GastroVision [32], GastroNet [15], and RATNet) on the unseen GastroVision dataset [32] for diagnosing 22 gastrointestinal diseases with a long-tailed distribution. Each model and setup was tested over 10 independent runs. Performance is reported in terms of mean AUC scores with standard deviation error bars for (a) 10 head classes and (b) 12 tail classes. c. Performance of GastroHUN, Gas- troVision, GastroNet, and RATNet across all 22 diseases for various fractions of training data. To assess label efficiency, we compared the linear probing performance of the models under reduced data settings. For the results reported on each dataset, the appendix ’FT’ denotes results obtained via fine-tuning, while the appendix ’LP’ denotes results obtained via linear probing. foundation models, including GastroNet (93.62±0.20%), GastroHUN (93.93±0.07%), and GastroVision (93.10±0.23%). Extended Data Table 3 provides a more comprehensive evaluation that includes addi- tional aggregated metrics such as mean MCC, average precision, and F1 scores across all classes. To further probe data efficiency, we performed experiments with progressively reduced training subsets (50%, 25%, 10%, and 5% of the full data) under the linear probing protocol, comparing RATNet against five alternative pre-trained architectures. As shown in Figure 6(c), RATNet maintained a clear advantage in resource-constrained conditions, especially versus GastroNet. Under identical linear probing setups, RATNet achieved average AUC gains of approximately 1.27%, 3.8%, and 4.6% at the 5%, 10% and 25% data levels, respectively. 2.5 Respond to novel diseases Assessing a foundation model’s ability to adapt to novel diseases remains essential for providing reli- able and robust diagnostic support. Therefore, we assessed the adaptability of RATNet for identifying gastrointestinal diseases in a video capsule endoscopy setting using the independent public Kvasir- Capsule dataset [33] and compared the results with those from five pre-trained models. To explore RATNet’s potential for domain extension, we conducted incrementally and continually pre-training on the Kvasir-Capsule training split, yielding an enhanced variant designated RATNet+KC. As presented in Extended Data Table 4, the incrementally trained RATNet+KC model attained an AUC of 99.03%, exceeding the performance of the standard RATNet model and consistently outper- forming the other four pre-trained models. For deeper insight into these improvements, we generated t-distributed stochastic neighbor embedding (t-SNE) [34] plots of feature representations from the Kvasir- Capsule test split using both RATNet and RATNet+KC (Figure 7). Post-refinement embeddings revealed clearer clustering of capsule-specific pathologies apart from features learned on other modalities, indi- cating stronger capture of domain-relevant patterns for precise classification. Together, these findings highlight RATNet’s robust generalization and data-efficient extension to out-of-domain applications, 9 including rare or emerging conditions that may appear in future clinical scenarios. With effective con- tinued pre-training and strong baseline performance, RATNet emerges as a flexible resource for evolving diagnostic demands. 2.6 Protect privacy and distribute pretraining RATNet typically undergoes centralized pretraining on public datasets, without addressing the privacy concerns that arise in cross-center collaborations. In contrast, constructing expansive multimodal AI systems for healthcare relies on large volumes of sensitive clinical records, requiring strict safeguards for patient confidentiality and distributed training across institutions. We introduce Federated RAT- Net to enable privacy-preserving collaboration, instantiating client models at each participating center (see Figure 2(c)). These clients optimize independently using their local data, after which a coordina- tor gathers the updated parameters, performs parameter averaging, and redistributes the global model for subsequent training cycles. We simulate a federated learning scenario by distributing different pre- training datasets across three local sites, each hosting an instance of RATNet. In the simulation, while some sites (such as Site 2 and Site 3 in Figure 2(c)) utilize public datasets, all local data at each site are treated as private and remain inaccessible to other sites. This approach ensures that each site main- tains the confidentiality of its data throughout the training process. Results in Extended Data Table 5 highlight Federated RATNet’s robustness to varying annotation quality among clients, which is a challenge that is often overlooked in traditional distributed methods. The results also demonstrate sig- nificant performance improvements from cross-node knowledge integration compared to standalone local optimization. Ultimately, this framework supports secure international partnerships on proprietary med- ical data, accelerating innovation in open medical foundation models while upholding patient privacy standards. 3 Discussion This paper presents RATNet, a foundation model for gastrointestinal endoscopic imaging, which acquires and transfers relevant knowledge embedded in heterogeneous expert annotations from five gastrointestinal endoscopy datasets via cyclic pre-training. Evaluated across six clinical scenarios, RATNet demonstrates superior generalizability, adaptability, robustness, and scalability, outperforming four existing foundation models in diagnosing gastrointestinal diseases. RATNet is generalizable. It outperforms other pre-training models across five internal test datasets, highlighting its broad applicability in diagnosing common gastrointestinal diseases (Figure 3(b) and Extended Data Table 2). Analogical reasoning is a key manifestation of clinical reasoning in diagnostic decision-making: physicians draw on medical knowledge and prior clinical experience to establish simi- larities between current patient signs and previous cases, predict unobserved signs that may emerge, and infer connections among these signs to reach a diagnosis [40]. However, when dealing with numerous or complex features (such as those in medical images), this approach becomes particularly challenging as it is difficult to quantify such comparisons [41]. RATNet addresses this by employing abstract representations that preserve perceptual richness (via image embedding into vectors) to acquire and transfer relevant knowledge, enabling the model to adapt to analogical reasoning under high-dimensional and uncertain inputs. This reasoning mode enhances the model’s generalization, allowing it to perform robustly across different domains for the same disease (Figure 3(c)) RATNet is adaptable. While humans can solve visual reasoning puzzles requiring logic with only a few or even zero examples, existing deep learning models like GastroNet [15] lack this capacity for analogical reasoning and still require extensive training data to achieve similar performance on the same task [42]. In this work, we address this few-shot (or zero-shot) visual reasoning problem via an analogical reasoning framework. Specifically, RATNet abstracts knowledge from multiple base tasks to build a repository of isolated experiences, each summarizing insights from distinct tasks (Figure 3(a)). During analogical reasoning, the model retrieves relevant knowledge by assessing similarities in both content and structure between the target domain and familiar base domains within this repository [43]. Based on the degree of similarity, knowledge from the base domains is then transferred to enrich the understanding of the target domain, ultimately guiding the final diagnosis. RATNet outperforms GastroNet in detecting rare diseases using only 1 ̃5 training samples (Figure 4), underscoring its clinical utility in data-scarce environments. Furthermore, RATNet demonstrates remarkable adaptability to variations in diagnostic 10 a b Non-Polyp Polyp CP-CHILD CecumPolyp UC Kvasir BBPS-0-1 BBPS-2-3 Cecum Impacted-stool Mayo-1Mayo-2Mayo-3Polyp Hyper-Kvasir Mayo-0Mayo-1 Mayo-2Mayo-3 LIMUC Posterior Knowledge Kvasir HyperKvasir Daping CPCHILD LIMUC Prior Knowledge Kvasir HyperKvasir Daping CPCHILD LIMUC CDMayo-0 Mayo-1 Mayo-2 Mayo-3 Daping KvasirCapsule KvasirCapsule AngiectasiaErosion Erythema Polyp Ampulla of vater Normal clean mucosa Reduced mucosal view Foreign body KvasirCapsule Ileocecal valve Blood - fresh Blood - hematin PylorusUlcerLymphangiectasia Angiectasia Erosion Erythema Reduced mucosal view Foreign body Ileocecal valve Blood - fresh Pylorus Ulcer Lymphangiectasia Disease Classes Fig. 7: Illustration of how the embeddings for Kvasir-Capsule [33] dataset’s diseases evolve in t-SNE [34]. a. We progressively enhanced RATNet by continually pre-training it on the Kvasir-Capsule diagnostic task, resulting in an upgraded model termed RATNet +KC . The t-SNE visualization illustrates the ability of RATNet +KC to capture discriminative prior knowledge for the new task. b. We demonstrate the evolution of feature separability across different disease categories as the model advances from the pre- trained RATNet to the incrementally learned RATNet +KC , and finally to the Fine-tuned RATNet +KC . 11 settings, achieving high AUC scores in a zero-shot transfer learning scenario for detecting four diseases absent from the training data (Figure 5). This ability to generalize to new sites or tasks without requiring retraining highlights its significant potential for real-world deployment. RATNet is robust. Gastrointestinal diseases often exhibit multi-lesion characteristics [22], leading to a long-tailed distribution of disease incidence. This property frequently results in highly imbalanced class representations in endoscopic imaging datasets. Due to the instance imbalance between majority and minority classes, deep learning methods tend to develop a bias toward majority classes, which limits the generalizability of classifiers and compromises their ability to accurately recognize rare diseases with limited data [44]. In addressing the challenge of long-tailed disease distribution, RATNet outperforms other foundation models on the GastroVision dataset [32], which exhibits typical long-tail characteristics (Extended Data Table 3). Under limited data scenarios, RATNet consistently demonstrates more robust performance than GastroNet (Figure 6(c)), underscoring the importance of knowledge acquisition and transfer. The analogical reasoning mechanism, inspired by human cognition, enables the model to gener- ate discriminative features that mitigate overfitting to head classes and effectively suppress model bias, thereby enhancing recognition of tail classes. RATNet is extensible. When incorporating new capsule endoscopy diagnostic tasks, RATNet suc- cessfully acquires knowledge from these tasks through its incremental learning capability (Figure 7(a)), effectively separating embeddings representing different diseases (Figure 7(b)), thereby revealing its newly acquired ability to capture task-specific features unique to capsule endoscopy. This capability can be fur- ther enhanced through fine-tuning to achieve more distinctive embeddings. Notably, RATNet not only responds effectively to new tasks via incremental learning but also exhibits performance improvements on original tasks (Extended Data Table 4), highlighting its potential for extension to emerging diseases and novel diagnostic modalities. The training of foundation models typically requires large-scale, cen- tralized datasets. In healthcare, however, medical data are often siloed across institutions due to privacy concerns, regulatory restrictions, and legal, ethical, and technical barriers to data sharing, making cen- tralized data aggregation particularly challenging [45]. To address this issue, we extend RATNet within a federated learning (FL) framework. In this setting, referred to as Federated RATNet, local models are pre-trained using the isolated data from each medical center. A shared model is then aggregated cen- trally and redistributed to local sites for fine-tuning. The resulting Federated RATNet model achieves performance comparable to the version trained on a centralized dataset and outperforms RATNet mod- els trained solely on single-institution data (Extended Data Table 4), aligning with findings from recent studies [46, 47]. Furthermore, the multi-task head and knowledge repository design of RATNet overcome the limitations of conventional FL when dealing with heterogeneity [48] in labels and data sources across different centers. By facilitating collaborative analysis across multiple data silos through the exchange of model updates rather than raw data, RATNet can harness the potential of globally distributed health- care data from diverse populations, uncovering insights inaccessible to isolated institutions [49]. This makes RATNet a privacy-preserving foundation model without compromising its strong generalization capabilities. RATNet is open, cognitive and affordable. Its openness enables researchers to perform fine-tuning and incremental learning on new datasets, thereby enhancing diagnostic accuracy across diverse clinical scenarios. All training data and annotations are derived from diverse datasets. Through its analogical reasoning mechanism, RATNet efficiently acquires and transfers knowledge across different domains, eliminating the need for manual unification of heterogeneous annotations. This significantly reduces data acquisition and processing costs while enhancing the model’s cross-domain cognitive capabilities. Looking forward, we plan to extend RATNet’s capabilities beyond classification to encompass localization [50], segmentation [51], and their integration [52]. Furthermore, we anticipate that its public accessibility will attract more diverse data sources, particularly from underrepresented regions and populations with limited healthcare resources. This will enable the assessment of model bias across demographic attributes including gender, race, age, and other population factors, thereby promoting equitable and accurate diagnosis across populations, addressing ethical concerns in clinical AI, and ensuring more comprehensive and inclusive robustness evaluation. In conclusion, RATNet demonstrated impressive performance in diagnosing gastrointestinal diseases across various scenarios, including common conditions, rare diseases with few samples, long-tailed dis- tributions, zero-shot transfer to new sites, response to novel diseases, and privacy-preserving federated learning. Its generalizability, adaptability, robustness, scalability, openness, and affordability make it a 12 Linear ReLU C Prior Knowledge Base 1. CP-CHILD 2. LIMUC ...... 5. DAPING Task Posterior Knowledge Template Encoded Feature Similarity Distribution 0.25 0.15 0.20 12 5 Weight fusion 12 3 4 5 1 Prompt Classification Loss C Linear ReLU Orthogonality Constraint Loss Aligned Prior-Posterior Knowledge Feature Input Output Student Encoder Student RAT Module Student Projector Consist Loss C Teacher Encoder Teacher RAT Module Teacher Projector C Classification Loss Stop gradient SGD Update EMA Update Student RAT Module Prior Knowledge 1 Augmentations Multi-task Heads 1 2 4 5 Non-Polyp Polyp BBPS-0-1 ... Impacted-stool Mayo-0 ... Mayo-3 Multi-task Heads Mayo-0 3 Cecum Polyp UC ... CD 1 2 3 4 5 Task Fig. 8: Illustration of the training pipeline of RATNet. RATNet is a teacher-student framework with multi-task heads designed to address heterogeneous label spaces across tasks. Specifically, each task is associated with an independent prediction head that maps shared representations to its own label set, while cyclical pretraining enables knowledge acquisition and transfer across tasks despite label incon- sistency. In each iteration, the student scans the dataset task by task, and the relevance-knowledge acquisition and transfer module updates both the prior knowledge base and the posterior knowledge template. The student’s newly acquired knowledge is accumulated into the teacher through exponential moving average (EMA), allowing the teacher to guide the student on subsequent tasks. After pretrain- ing, the knowledge accumulated in the teacher can be transferred to downstream target tasks. powerful foundation model for gastrointestinal endoscopic imaging, capable of democratizing expert-level diagnostics. The model’s exceptional capabilities are attributable to its innovative analogical reasoning framework, which cyclically acquires and transfers knowledge from heterogeneous expert annotations across multiple gastrointestinal endoscopy datasets, effectively addressing challenges like data scarcity, domain shift, and label heterogeneity without requiring costly label unification. RATNet represents a significant shift from task-specific learning toward generalized, multi-center medical intelligence. By pro- moting collaborative efforts and open-source accessibility, it not only enhances clinical efficiency but also facilitates equitable access to AI-assisted diagnostics in low-resource settings. Future research direc- tions include extending the analogy reasoning mechanism to localization and segmentation, incorporating temporal reasoning into endoscopic video stream analysis, and exploring multimodal alignment among histopathology, imaging, and genomic data. These steps will accelerate the development of a truly univer- sal gastrointestinal intelligence model, capable of supporting precision gastroenterology and advancing global medical democratization. 4 Methods 4.1 Model architecture We propose a framework named relevance-knowledge acquisition and transfer network (RATNet), which is designed to acquire task-related knowledge from heterogeneous expert annotations across multiple tasks, thereby constructing a prior knowledge base. When encountering a new task not covered by the original training data, RATNet enables the transfer of relevant knowledge from this base to the new task. Specifically, it retrieves associated factual information related to the new task from the prior knowledge base and grounds the network’s responses on such information, thereby producing more accurate and rel- evant outputs. As illustrated in Fig. 8, RATNet consists of an encoder, a relevance-Knowledge acquisition and transfer (RAT) module, a projector, and multiple task heads. It is built upon a teacher–student archi- tecture capable of handling heterogeneous labels across tasks and is trained using a cyclic pre-training strategy. 13 4.1.1 Teacher-student architecture for heterogeneous label To cope with heterogeneous labels across multi-dataset pretraining, we avoid forcibly unifying all anno- tations into a single shared label set. Instead, we treat heterogeneity as a source of multi-task expertise: each dataset/task is equipped with a plug-in task head, allowing the model to learn the native semantics of each annotation scheme on top of a shared visual encoder. To reduce gradient interference that arises when summing losses from incompatible label spaces, we adopt a cyclic pretraining schedule where the student updates on one task at a time and revisits tasks repeatedly to limit forgetting. In parallel, a teacher model with the same architecture accumulates the student’s task-specific knowledge via EMA after each task/epoch, gradually integrating cross-task experience. A student–teacher consistency loss, implemented with projection heads that map representations into a common feature space, feeds the teacher’s stable, cross-heterogeneity knowledge back to the student as additional supervision, helping preserve transferable representations while learning new label spaces. 4.1.2 Relevance-knowledge acquisition and transfer module The objective of this module is to enable the model to acquire prior knowledge from each task, with all such knowledge collectively forming a knowledge base. When presented with a new task, the model should be able to retrieve relevance-knowledge from the knowledge base by evaluating the similarity between samples from the new task and each stored knowledge entry. Based on these similarity scores, the model then extracts and integrates the factual information from relevance-knowledge to enable effective cross- task knowledge transfer. Specifically, we begin by randomly initializing a set of learnable knowledge base entries KB ∈R T×E = b 1 ,b 2 ,...,b T , where T corresponds to the total number of tasks in the pre- training set. To facilitate the learning of distinct task-specific knowledge, we impose an orthogonality constraint on KB to facilitate that each knowledge vector retains its unique task-related characteristics. Since the pre-training dataset may not encompass all possible tasks, and the model may encounter unseen tasks during inference, relying solely on KB is insufficient. We aim to extract knowledge from KB that is relevant to the current task. Therefore, we develop a posterior knowledge generation mechanism that projects the current encoded features into a posterior knowledge vector space, enabling adaptive generation of posterior knowledge rather than relying exclusively on the prior domain distribution in KB. To achieve this, we randomly initialize a learnable posterior knowledge template, denoted as t pk . Given the encoded feature v e , we concatenate v e with t p k and feed [v e ©t pk ] into an MLP for fusion, thereby generating dynamic posterior knowledge features k p . To fully leverage the prior knowledge from KB and the posterior knowledge captured by k p , we propose a knowledge alignment and fusion strategy to seamlessly integrate these two types of knowledge. Considering that the domain knowledge corresponding to different tasks in KB may have varying degrees of relevance to the current task, we aim to selectively combine this information based on relevance to generate guidance information aligned with the posterior knowledge for new tasks in unseen domains. Specifically, for each prior knowledge entry b i ∈ KB, we compute its cosine similarity with the posterior knowledge k p and apply a softmax function to obtain adaptive weights W =ω 1 ,ω 2 ,...,ω Z : sim i = k p · b i ∥k p ∥·∥b i ∥ .(1) ω i = exp(sim i )/τ T P k=1 exp(sim k )/τ .(2) Subsequently, the weighted aggregation of the prior knowledge in KB is computed as follows: k a = T X i=1 ω i · b i .(3) During pre-training, the resulting aligned prior knowledge feature k a is constrained using a task similarity lossL ts . This loss is defined by measuring the similarity between the prior knowledge b i of the 14 current task in KB and the aligned prior knowledge k a : L ts = (1− k a · b i ∥k a ∥·∥b i ∥ ) 2 .(4) Finally, we concatenate the posterior knowledge k p with k a to form the combined feature [k p ©k a ]. This concatenated feature is then passed through an MLP block for fusion, producing the final aligned prior-posterior knowledge feature. 4.2 Implementation details RATNet employs a Swin large backbone with an input resolution of 384×384. Both the teacher and student encoders, along with the projector and multi-task head, are initialized with weights pre-trained on our dataset. The model was trained using a stochastic gradient descent optimizer with an initial learning rate of 0.003 and a batch size of 64 on an NVIDIA H100 GPU with 80 GB of memory. At the end of each task, the teacher model was updated using an exponential moving average of the student’s weights over one epoch with a momentum of 0.9. Image augmentations included random cropping and rotation, as well as variations in brightness, contrast, and gamma distribution. The model underwent 100 epochs of pre-training, completing 100 iterations over the entire dataset. 4.3 Datasets for developing and evaluating proposed model 4.3.1 Pretraining datasets Our proposed model was pre-trained on a total of 38,995 gastrointestinal endoscopy images obtained from five datasets provided by institutions worldwide, all of which were annotated by domain experts. Each dataset underwent uniform preprocessing and resampling procedures to form a consolidated pre- training dataset. The CP-CHILD dataset [19] contains colonoscopy images collected from 1,600 children in Hunan Province, including 7,874 normal images and 1,395 polyp images. All images were obtained at the Hunan Children’s Hospital between March 2018 and April 2019. The LIMUC dataset [20] includes 11,276 images and 1,043 colonoscopy videos from 564 patients, acquired at the Department of Gas- troenterology, Marmara University School of Medicine, between December 2011 and July 2019. After resampling, 10,765 images were retained. These images are categorized into four classes according to the Mayo endoscopic scoring system: Mayo 0 to 3. The Kvasir dataset [21] consists of images acquired from the gastrointestinal tract using endoscopic equipment at Vestre Viken Health Trust, Norway. We sampled 986 cecum images, 1,000 polyp images, and 1,000 ulcerative colitis images. The HyperKvasir dataset [22] was collected from gastroscopies and colonoscopies performed at Bærum Hospital, Norway between 2008 and 2016. We selected 4,713 gastrointestinal images across eight categories: cecum, BBPS-0-1, BBPS-2- 3, impacted stool, polyp, Mayo-1, Mayo-2, and Mayo-3. The Daping dataset was prospectively collected during routine clinical examinations at Daping Hospital in China and includes 21,011 images labeled into five categories: Mayo-0, Mayo-1, Mayo-2, Mayo-3, and CD (Crohn’s disease). From the combined data, 34,120 images were allocated for pre-training and 4,875 images were reserved for validation. 4.3.2 Data for few-shot learning The Colonoscopic dataset [31] consists of 76 colonoscopy videos and covers three lesion types: serrated adenomas, hyperplastic lesions, and adenomas. These three lesion types were used in a few-shot linear- probing benchmark to simulate the challenging scenario of rare diseases, in which a hospital has access to only a limited number of cases for model training. 4.3.3 Data for zero-shot transfer PolypGen [26] is a general-purpose dataset for polyp segmentation and detection. It consists of 8,037 frames, including both single images and sequences. The comprehensive dataset includes 3,762 positive frames and 4,275 negative frames collected from six different centers across Europe and Africa. From this collection, we sampled 3,240 positive frames and 2,194 negative frames for our experiments. The Shaoyifu Dataset was collected by the Department of Gastroenterology at Shaoyifu Hospital, Zhejiang 15 University School of Medicine. It comprises 9,620 images categorized into four classes: Mayo-1, Mayo-2, Mayo-3, and CD (Crohn’s Disease). 4.3.4 Data in long-tailed distributions The GastroVision dataset [32] was collected from two medical centers: the Department of Gastroenterol- ogy at Bærum Hospital, Vestre Viken Hospital Trust in Norway, and Karolinska University Hospital in Sweden, using standard endoscopy systems provided by Olympus Europe and Pentax Medical Europe, both based in Germany. It comprises 8,000 images annotated with 27 distinct categories. These categories are further divided into two major groups: the upper gastrointestinal tract and the lower gastrointestinal tract. The sample distribution across categories is imbalanced, exhibiting a long-tail distribution. The sample distribution across categories is highly imbalanced, exhibiting a pronounced long-tail pattern. Specifically, 10 head categories contain relatively large numbers of samples, ranging from 200 to 1,467 instances per category. In contrast, the remaining 17 tail categories are significantly underrepresented. Each of these tail categories contains only 6 to 171 samples. 4.3.5 Data for incremental learning The Kvasir-Capsule [33] is a large video capsule endoscopy (VCE) dataset collected from examinations at a Norwegian hospital, comprising 43 annotated videos. It includes 47,238 frames annotated with 14 different categories. Unlike similar datasets containing colonoscopy or esophagogastroscopy images, this dataset focuses on the small intestine, which exhibits a distinct mucosal surface characterized by intestinal villi. Furthermore, VCE images have significantly lower resolution and frame rates, the intestine is not insufflated as in conventional endoscopy, different optical systems are used, and the capsule’s movement is uncontrolled compared to flexible endoscopes employed during manual procedures. 4.4 Comparative pretrained models To evaluate the performance of our RATNet model, we conducted a comprehensive comparison with several existing gastrointestinal diagnostic models, namely: GastroNet [15], SSL [35], GastroHUN [36], and GastroVision [32]. The model configurations and training details for all these methods are provided in Extended Data Table 6. GastroNet was trained using self-supervised learning on a dataset comprising 5014174 unlabeled gastrointestinal endoscopy images from eight different medical centers. The model was configured with a Vision Transformer [53] architecture and pre-trained in-domain using DINO [54]. The SSL approach employed a Masked Autoencoder (MAE) [55], which learns feature representations by reconstructing the original input from partially observed data. It was pre-trained on 99,417 unla- beled samples from the HyperKvasir dataset [22]. GastroHUN utilizes a ConvNeXt [56] architecture and enhances model performance through a two-stage pre-training strategy. It was pre-trained on a dataset containing 8,834 labeled images from 387 patients, covering 22 anatomical landmarks in the stomach. GastroVision was pre-trained in a supervised manner using a DenseNet[57] architecture on a multi-center, open-access gastrointestinal endoscopy dataset consisting of 8,000 images across 27 classes. MiniGPT- v2 [58] employs a large language model as a unified interface for handling multiple vision-language tasks and utilizes task identifiers to guide the model in distinguishing between different types of instructions, thereby improving multi-task learning efficiency. LLaVA-v1 [59] introduces the concept of ”visual instruc- tion tuning,” which enhances the model’s ability to comprehend and generate visual content through an instruction-based learning mechanism. Building upon LLaVA-v1, LLaVA-v1.5 [60] incorporates higher- quality training data and optimized fine-tuning strategies, leading to significantly improved multimodal alignment performance. Bunny-v1.0-3B [61] is a lightweight multimodal large model that achieves strong performance with a limited number of parameters by adopting a flexible vision-language modular archi- tecture and efficient data sampling strategies. MGM-2B [62] proposes a general framework designed to enhance multimodal model capabilities, focusing on three key aspects: high-resolution visual token mod- eling, construction of high-quality training data, and self-guided model generation. MobileVLM-1.7B [63] features a lightweight and efficiently trained architecture, enabling comprehensive visual-language under- standing and response capabilities on mobile devices while maintaining a balance between semantic consistency and inference speed. LLaVA-Med-v1.0 and LLaVA-Med-v1.5 [64] are large vision-language models specifically designed for medical applications. They leverage biomedical image-text data from PubMed Central and utilize GPT-4 to automatically generate medical question-answer pairs, thereby 16 facilitating effective instruction tuning. ColonGPT is an intelligent model tailored for colonoscopy proce- dures, integrating linguistic and visual information to assist in the analysis and diagnosis of colonoscopy images. All the aforementioned vision-language models were fine-tuned in accordance with the data split protocol of the ColonINST dataset [37]. 4.5 Evaluation metrics and statistical analysis To comprehensively assess the model’s classification performance, we employed the following metrics, each emphasizing distinct aspects. The Area Under the Receiver Operating Characteristic Curve (AUC) is a prominent metric in binary classification, quantifying the area beneath the ROC curve that plots True Positive Rate (TPR) against False Positive Rate (FPR) across thresholds. AUC ranges from 0 to 1, with 0.5 denoting random guessing and 1 indicating flawless classification. Higher values reflect superior discriminative capability between classes. AUC excels in providing a threshold-independent overview, facilitating model comparisons on balanced datasets. The F1 Score harmonizes precision and recall through their harmonic mean, serving as a key metric for binary and multi-class classification. Ranging from 0 to 1, a value of 1 signifies perfect precision and recall, while 0 implies no true positives identified. By addressing both false positives and negatives, F1 Score outperforms accuracy, particularly in imbalanced datasets where it ensures balanced assessment. Average Precision (AP) is essential for evaluating models in information retrieval and object detection, defined as the area under the Precision-Recall curve. It measures sustained high precision across recall levels and varying thresholds. In object detection, mean AP (mAP) aggregates category-specific values. AP is ideal for imbalanced scenarios or rare classes, emphasizing positive class prediction accuracy. The Matthews Correlation Coefficient (MCC), proposed by Brian W. Matthews in 1975, evaluates binary classifiers by incorporating all confusion matrix components: true positives, true negatives, false positives, and false negatives. MCC spans -1 to +1, with +1 for perfect agreement, 0 for random perfor- mance, and -1 for complete discord. As a balanced metric, it offers reliable insights into classifier efficacy, even on imbalanced data, surpassing F1 Score or accuracy. To evaluate the statistical significance of observed differences in model performance and to draw reliable conclusions, we employed a comprehensive statistical analysis approach. This method integrated an independent two-sample t-test with the calculation of 95% confidence intervals. In experiments involving fine-tuning and linear-probing, the random initialization inherent to the linear classifier meant that each execution could yield distinct outcomes. Consequently, to ensure the robustness of our findings, we performed at least ten evaluations for each model and reported both the mean and standard deviation for every performance metric. A two-sided independent t-test was subsequently conducted for the statistical analysis. Supplementary information. Extended data is available for this paper in Supplementary informa- tion. Acknowledgements. The work was partially supported by the National Natural Science Foundation of China (62471293), by Chongqing Excellence Program for Innovation and Entrepreneurship Leader- ship Talent Project (CQYC20220303576), and by the Natural Science Foundation of Chongqing, China (CSTB2024NSCQ-LZX0141). References [1] Wang, Y., Huang, Y., Chase, R.C., Li, T., Ramai, D., Li, S., Huang, X., Antwi, S.O., Keaveny, A.P., Pang, M.: Global burden of digestive diseases: a systematic analysis of the global burden of diseases study, 1990 to 2019. Gastroenterology 165(3), 773–783 (2023) [2] Arnold, M., Abnet, C.C., Neale, R.E., Vignat, J., Giovannucci, E.L., McGlynn, K.A., Bray, F.: Global burden of 5 major types of gastrointestinal cancer. Gastroenterology 159(1), 335–349 (2020) [3] Sung, H., Ferlay, J., Siegel, R.L., Laversanne, M., Soerjomataram, I., Jemal, A., Bray, F.: Global cancer statistics 2020: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: a cancer journal for clinicians 71(3), 209–249 (2021) 17 [4] Correa, P.: Gastric cancer: overview. Gastroenterology Clinics of North America 42(2), 211 (2013) [5] Tang, Y., Anandasabapathy, S., Richards-Kortum, R.: Advances in optical gastrointestinal endoscopy: a technical review. Molecular Oncology 15(10), 2580–2599 (2021) [6] Martins, B.C., Moura, R.N., Kum, A.S.T., Matsubayashi, C.O., Marques, S.B., Safatle-Ribeiro, A.V.: Endoscopic imaging for the diagnosis of neoplastic and pre-neoplastic conditions of the stomach. Cancers 15(9), 2445 (2023) [7] Tham, C., Rea, D., Tham, T.: Artificial intelligence in endoscopy: A narrative review. The Ulster Medical Journal 94(1), 16 (2025) [8] Xu, Z., Li, Y., Su, P., Zhong, Z., Zeng, Z., Chen, M., Chen, D., Lan, C.: Artificial intelligence system improves the quality of digestive endoscopy: A prospective pretest and post-test single-center clinical trial. Digestive and Liver Disease (2025) [9] Mushtaq, K., Lim, Y.J., Spada, C., Mussetto, A., Koulaouzidis, A., Kaung, T., Borrow, D.- M., Casadei, C., Patel, P., Rahman, I.: Ai-assisted double-headed capsule endoscopy: Multicentre prospective diagnostic accuracy study across small bowel indications. Diagnostics 16(2), 239 (2026) [10] Shi, C., Rezai, R., Yang, J., Dou, Q., Li, X.: A survey on trustworthiness in foundation models for medical image analysis. arXiv preprint arXiv:2407.15851 (2024) [11] Zhang, S., Metaxas, D.: On the challenges and perspectives of foundation models for medical image analysis. Medical image analysis 91, 102996 (2024) [12] Ma, D., Pang, J., Gotway, M.B., Liang, J.: A fully open ai foundation model applied to chest radiography. Nature, 1–11 (2025) [13] Wu, C., Zhang, X., Zhang, Y., Hui, H., Wang, Y., Xie, W.: Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data. Nature Communications 16(1), 7866 (2025) [14] Wang, X., Zhao, J., Marostica, E., Yuan, W., Jin, J., Zhang, J., Li, R., Tang, H., Wang, K., Li, Y., et al.: A pathology foundation model for cancer diagnosis and prognosis prediction. Nature 634(8035), 970–978 (2024) [15] Boers, T.G., Fockens, K.N., Putten, J.A., Jaspers, T.J., Kusters, C.H., Jukema, J.B., Jong, M.R., Struyvenberg, M.R., Groof, J., Bergman, J.J., et al.: Foundation models in gastrointestinal endo- scopic ai: Impact of architecture, pre-training approach and data efficiency. Medical Image Analysis 98, 103298 (2024) [16] He, Y., Chen, Q., Liu, B., Cao, Y.: Foundational multi-task multimodal model for upper gi endoscopy. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 6612–6621 (2025) [17] Zhang, B., Chen, Y., Bai, L., Zhao, Y., Sun, Y., Yuan, Y., Zhang, J., Ren, H.: Learning to adapt foundation model dinov2 for capsule endoscopy diagnosis. Procedia Computer Science 250, 188–194 (2024) [18] Dermyer, P., Kalra, A., Schwartz, M.: Endodino: A foundation model for gi endoscopy. arXiv preprint arXiv:2501.05488 (2025) [19] Wang, W., Tian, J., Zhang, C., Luo, Y., Wang, X., Li, J.: An improved deep learning approach and its applications on colonic polyp images detection. BMC Medical Imaging 20(1), 83 (2020) [20] Polat, G., Kani, H.T., Ergenc, I., Ozen Alahdab, Y., Temizel, A., Atug, O.: Improving the computer- aided estimation of ulcerative colitis severity according to mayo endoscopic score by using regression- based deep learning. Inflammatory Bowel Diseases 29(9), 1431–1439 (2023) 18 [21] Pogorelov, K., Randel, K.R., Griwodz, C., Eskeland, S.L., Lange, T., Johansen, D., Spampinato, C., Dang-Nguyen, D.-T., Lux, M., Schmidt, P.T., et al.: Kvasir: A multi-class image dataset for computer aided gastrointestinal disease detection. In: Proceedings of the 8th ACM on Multimedia Systems Conference, p. 164–169 (2017) [22] Borgli, H., Thambawita, V., Smedsrud, P.H., Hicks, S., Jha, D., Eskeland, S.L., Randel, K.R., Pogorelov, K., Lux, M., Nguyen, D.T.D., et al.: Hyperkvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy. Scientific data 7(1), 283 (2020) [23] Devkota, A., Amireskandari, A., Palko, J., Thakkar, S., Adjeroh, D., Jiang, X., Bhattarai, B., Gyawali, P.K.: Federated foundation model for gi endoscopy images. arXiv preprint arXiv:2505.24108 (2025) [24] Kondrateva, E., Pominova, M., Popova, E., Sharaev, M., Bernstein, A., Burnaev, E.: Domain shift in computer vision models for mri data analysis: an overview. In: Thirteenth International Conference on Machine Vision, vol. 11605, p. 126–133 (2021). SPIE [25] Ayana, G., Dese, K., Abagaro, A.M., Jeong, K.C., Yoon, S.-D., Choe, S.-w.: Multistage transfer learning for medical images. Artificial Intelligence Review 57(9), 232 (2024) [26] Ali, S., Jha, D., Ghatwary, N., Realdon, S., Cannizzaro, R., Salem, O.E., Lamarque, D., Daul, C., Riegler, M.A., Anonsen, K.V., et al.: A multi-centre polyp detection and segmentation dataset for generalisability assessment. Scientific Data 10(1), 75 (2023) [27] Ball, L.J., Thompson, V.A.: International Handbook of Thinking and Reasoning. Routledge, ??? (2017) [28] Gust, H., Krumnack, U., K ̈uhnberger, K.-U., Schwering, A.: Analogical reasoning: a core of cognition. K ̈unstliche Intell. 22(1), 8–12 (2008) [29] Gentner, D., Holyoak, K.J.: Reasoning and learning by analogy: Introduction. American psychologist 52(1), 32 (1997) [30] Ribeiro, H.J.: Systematic Approaches to Argument by Analogy vol. 25. Springer, ??? (2014) [31] Mesejo, P., Pizarro, D., Abergel, A., Rouquette, O., Beorchia, S., Poincloux, L., Bartoli, A.: Computer-aided classification of gastrointestinal lesions in regular colonoscopy. IEEE transactions on medical imaging 35(9), 2051–2063 (2016) [32] Jha, D., Sharma, V., Dasu, N., Tomar, N.K., Hicks, S., Bhuyan, M.K., Das, P.K., Riegler, M.A., Halvorsen, P., Bagci, U., et al.: Gastrovision: A multi-class endoscopy image dataset for com- puter aided gastrointestinal disease detection. In: Workshop on Machine Learning for Multimodal Healthcare Data, p. 125–140 (2023). Springer [33] Smedsrud, P.H., Thambawita, V., Hicks, S.A., Gjestang, H., Nedrejord, O.O., Næs, E., Borgli, H., Jha, D., Berstad, T.J.D., Eskeland, S.L., et al.: Kvasir-capsule, a video capsule endoscopy dataset. Scientific Data 8(1), 142 (2021) [34] Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of machine learning research 9(11) (2008) [35] Bravo, D., Ruano, J., G ́omez, M., Gonz ́alez, F.A., Romero, E.: Self-supervised learning for multi- category endoscopy classification and data quality evaluation using masked autoencoders. In: 2025 IEEE 22nd International Symposium on Biomedical Imaging (ISBI), p. 1–5 (2025). IEEE [36] Bravo, D., Frias, J., Vera, F., Trejos, J., Mart ́ınez, C., G ́omez, M., Gonz ́alez, F., Romero, E.: Gastrohun an endoscopy dataset of complete systematic screening protocol for the stomach. Scientific Data 12(1), 102 (2025) 19 [37] Ji, G.-P., Liu, J., Xu, P., Barnes, N., Khan, F.S., Khan, S., Fan, D.-P.: Frontiers in intelligent colonoscopy. Machine Intelligence Research 23(1), 70–114 (2026) [38] Xu, W., Liu, F., Tang, W., Gu, Y., Zhong, J., Cui, L., Du, P.: The mayo endoscopic score is a novel predictive indicator for malignant transformation in ulcerative colitis: a long-term follow-up multicenter study. Frontiers in surgery 9, 832219 (2022) [39] Xu, Z., Ali, S., Gupta, S., Leedham, S., East, J.E., Rittscher, J.: Patch-level instance-group discrim- ination with pretext-invariant learning for colitis scoring. In: International Workshop on Machine Learning in Medical Imaging, p. 101–110 (2022). Springer [40] Jia, Z., Zeng, X., Duan, H., Lu, X., Li, H.: A patient-similarity-based model for diagnostic prediction. International journal of medical informatics 135, 104073 (2020) [41] Gr ̈uger, J., Kuhn, M., Amri, K., Bergmann, R.: Enhancing healthcare decision-making with analogy- based reasoning. In: International Conference on Process Mining, p. 447–459 (2024). Springer [42] Kim, Y., Shin, J., Yang, E., Hwang, S.J.: Few-shot visual reasoning with meta-analogical contrastive learning. Advances in Neural Information Processing Systems 33, 16846–16856 (2020) [43] Doumas, L.A., Puebla, G., Martin, A.E., Hummel, J.E.: A theory of relation learning and cross- domain generalization. Psychological review 129(5), 999 (2022) [44] Jin, J., Hu, D., Pu, W., Luo, Y., Feng, X.: Few-shot learning with task adaptation for multi- category gastrointestinal endoscopy classification. Biomedical Signal Processing and Control 95, 106387 (2024) [45] Ngiam, K.Y., Khor, W.: Big data and machine learning algorithms for health-care delivery. The Lancet Oncology 20(5), 262–273 (2019) [46] Sheller, M.J., Reina, G.A., Edwards, B., Martin, J., Bakas, S.: Multi-institutional deep learning mod- eling without sharing patient data: A feasibility study on brain tumor segmentation. In: International MICCAI Brainlesion Workshop, p. 92–104 (2018). Springer [47] Sheller, M.J., Edwards, B., Reina, G.A., Martin, J., Pati, S., Kotrotsou, A., Milchenko, M., Xu, W., Marcus, D., Colen, R.R., et al.: Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Scientific reports 10(1), 12598 (2020) [48] Wahab, H., Mehmood, I., Ugail, H., Del Ser, J., Muhammad, K.: Federated deep learning for wireless capsule endoscopy analysis: Enabling collaboration across multiple data centers for robust learning of diverse pathologies. Future Generation Computer Systems 152, 361–371 (2024) [49] Li, M., Xu, P., Hu, J., Tang, Z., Yang, G.: From challenges and pitfalls to recommendations and opportunities: Implementing federated learning in healthcare. Medical Image Analysis, 103497 (2025) [50] Senthil Velan, S.: Benchmarking and boosting localizers for chest x-ray images. Technical report, Arizona State University (2024) [51] Saravanan, M.: Benchmarking and boosting of 3d segmentation models. Technical report, Arizona State University (2024) [52] Islam, N.U., Ma, D., Pang, J., Velan, S.S., Gotway, M., Liang, J.: Foundation x: integrating clas- sification, localization, and segmentation through lock-release pretraining strategy for chest x-ray analysis. In: 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), p. 3647–3656 (2025). IEEE [53] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, 20 M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 (2021) [54] Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L.M., Shum, H.: DINO: DETR with improved denoising anchor boxes for end-to-end object detection. In: The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 (2023) [55] He, K., Chen, X., Xie, S., Li, Y., Doll ́ar, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 16000–16009 (2022) [56] Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 11976– 11986 (2022) [57] Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional net- works. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 4700–4708 (2017) [58] Chen, J., Zhu, D., Shen, X., Li, X., Liu, Z., Zhang, P., Krishnamoorthi, R., Chandra, V., Xiong, Y., Elhoseiny, M.: Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478 (2023) [59] Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems 36, 34892–34916 (2023) [60] Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 26296–26306 (2024) [61] He, M., Liu, Y., Wu, B., Yuan, J., Wang, Y., Huang, T., Zhao, B.: Efficient multimodal learning from data-centric perspective. arXiv preprint arXiv:2402.11530 (2024) [62] Li, Y., Zhang, Y., Wang, C., Zhong, Z., Chen, Y., Chu, R., Liu, S., Jia, J.: Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814 (2024) [63] Chu, X., Qiao, L., Lin, X., Xu, S., Yang, Y., Hu, Y., Wei, F., Zhang, X., Zhang, B., Wei, X., et al.: Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886 2(6), 7 (2023) [64] Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., Gao, J.: Llava- med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems 36, 28541–28564 (2023) 21