Paper deep dive
EndoVLM: An Endoscopy Vision-Language Pre-training Model via Anatomy-Guided Sparsity and Progressive Alignment
Zhenyu Yi, Jianwei Xu, Yue Hu, Zhongwei Qiu, Sijing Li, Liang Huang, Bin Lv, Ling Zhang, Yingda Xia
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The development of foundation models (FMs) is crucial for advancing endoscopic image analysis. However, existing endoscopy FMs mainly rely on self-supervised learning from uni-modal images or videos, overlooking the rich semantic knowledge contained in clinical reports. Furthermore, effectively leveraging these records is hindered by a fundamental modality gap: structured anatomical descriptions are not naturally mapped to specific frames within the high-redundancy, uncurated visual streams. In this paper, we present EndoVLM, a novel vision-language FM pre-trained on over 348K endoscopic examinations, each pairing a clinical report with its corresponding image collection. An Anatomy-Guided Sparse Pooling mechanism utilizes textual descriptions as queries to drive sparse attention, efficiently aggregating semantically salient frames into anatomy-specific visual representations across redundant image-sets. Next, a Progressive Semantic-Aware Alignment strategy models clinical taxonomy (anatomy and pathological status) via structured soft targets, bridging the gap from global patient-level matching to fine-grained localized alignment. Finally, a Semantic-Concentrated Masked Autoencoder is applied exclusively to these semantic-rich frames, integrating low-level visual precision with robust high-level semantic representation. Extensive experiments across various downstream tasks demonstrate that EndoVLM outperforms existing foundation models and remains competitive with task-specific methods. Remarkably, EndoVLM also exhibits robust zero-shot generalization capabilities, highlighting its potential for broader clinical application.
Tags
Links
- Source: https://arxiv.org/abs/2608.04472v1
- Canonical: https://arxiv.org/abs/2608.04472v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
34,159 characters extracted from source content.
Expand or collapse full text
1]DAMO Academy, Alibaba Group 2]Department of Gastroenterology, The First Affiliated Hospital of Zhejiang Chinese Medical University 3]Shanghai Jiao Tong University 4]Hupan Lab 5]Key Laboratory of Digestive Pathophysiology of Zhejiang Province, The First Affiliated Hospital of Zhejiang Chinese Medical University 6]Zhejiang University [*]Equal contribution [†]Corresponding author [Code]https://github.com/Scatteredrain/EndoVLM EndoVLM: An Endoscopy Vision-Language Pre-training Model via Anatomy-Guided Sparsity and Progressive Alignment Zhenyu Yi Jianwei Xu Yue Hu Zhongwei Qiu Sijing Li Liang Huang Bin Lv Ling Zhang Yingda Xia [ [ [ [ [ [ Abstract The development of foundation models (FMs) is crucial for advancing endoscopic image analysis. However, existing endoscopy FMs mainly rely on self-supervised learning from uni-modal images or videos, overlooking the rich semantic knowledge contained in clinical reports. Furthermore, effectively leveraging these records is hindered by a fundamental modality gap: structured anatomical descriptions are not naturally mapped to specific frames within the high-redundancy, uncurated visual streams. In this paper, we present EndoVLM, a novel vision-language FM pre-trained on over 348K endoscopic examinations, each pairing a clinical report with its corresponding image collection. An Anatomy-Guided Sparse Pooling mechanism utilizes textual descriptions as queries to drive sparse attention, efficiently aggregating semantically salient frames into anatomy-specific visual representations across redundant image-sets. Next, a Progressive Semantic-Aware Alignment strategy models clinical taxonomy (anatomy and pathological status) via structured soft targets, bridging the gap from global patient-level matching to fine-grained localized alignment. Finally, a Semantic-Concentrated Masked Autoencoder is applied exclusively to these semantic-rich frames, integrating low-level visual precision with robust high-level semantic representation. Extensive experiments across various downstream tasks demonstrate that EndoVLM outperforms existing foundation models and remains competitive with task-specific methods. Remarkably, EndoVLM also exhibits robust zero-shot generalization capabilities, highlighting its potential for broader clinical application. 1 Introduction Recent advances in foundation models (FMs) have catalyzed a paradigm shift in the development of medical image analysis (Bommasani et al., 2021). Through self-supervised learning (SSL) on massive unlabeled datasets (He et al., 2022; Oquab et al., 2023), FMs learn highly transferable representations for downstream tasks (Xu et al., 2021; Hu et al., 2024, 2025; Polat et al., 2023), driving breakthroughs across various medical domains, including CT (Tang et al., 2022), X-ray (Tiu et al., 2022), and histopathology (Wang et al., 2024). Furthermore, since clinical workflows naturally pair images with detailed diagnostic reports, this multimodal synergy has inspired vision-language FMs—such as BiomedCLIP (Zhang et al., 2023), LLaVA-Med (Li et al., 2023), fVLM (Shui et al., 2025), and TumorChain (Li et al., 2026). By aligning visual features with rich textual insights, these models build more interpretable and clinically grounded AI systems. However, existing endoscopy-specific FMs (Wang et al., 2023, 2025b; Tian et al., 2025; Jong et al., 2025) remain predominantly visual self-supervised models, overlooking the rich clinical insights embedded in textual reports, which not only describe anatomical landmarks and pathological findings but also provide procedural context. Meanwhile, general vision-language pre-training models (VLMs) such as CLIP (Radford et al., 2021; Wang et al., 2022) typically rely on a one-to-one alignment between individual samples (e.g., a single image, short video clip, or standardized 3D medical data such as CT) and their textual descriptions. In contrast, routine gastrointestinal (GI) endoscopy produces a single comprehensive report for dozens of unannotated images or lengthy videos—which we formulate as an image-set. Without explicit temporal or spatial annotations, it is exceptionally difficult to align a specific frame or a local region with a particular sentence or abnormal finding within the comprehensive report (Miech et al., 2020). For example, a gastroscopy report covers eight regions (esophagus, cardia, fundus, body, antrum, angularis, pylorus, duodenum), while a colonoscopy assesses nine segments (ileum, ileocecal region, ascending colon, hepatic flexure, transverse colon, splenic flexure, descending colon, sigmoid colon, rectum). During examination, clinicians capture multiple images of each region from various angles and document morphology, mucosal features, and any pathological findings. The lack of per-frame labels makes it nearly impossible to identify the anatomical location or mucosal features in individual images, impeding precise image- or pixel-level cross-modal alignment. Addressing this bottleneck requires novel strategies that bridge structured clinical narratives with large-scale, unstructured visual corpora. In this work, we present EndoVLM, a novel vision-language foundation model pre-trained on a massive cohort of 348K endoscopic cases. To bridge the gap between unordered image-sets and structured reports, we introduce three core components. First, an Anatomy-Guided Sparse Pooling (AGSP) mechanism employs query-driven sparse attention to distill semantically salient frames from redundant visual streams. Subsequently, we propose a Progressive Semantic-Aware Alignment (PSAA) strategy. By modeling clinical classifications—encompassing both anatomical structures and pathological states—through soft targets, PSAA facilitates a robust transition from coarse patient-level matching to fine-grained localized alignment. Finally, a Semantic-Concentrated Masked Autoencoder (SC-MAE) is applied strictly to these filtered frames, unifying low-level geometric precision with high-level clinical semantics. Extensive experiments across diverse downstream tasks—including risk stratification, polyp segmentation, anatomy recognition, and video disease diagnosis—demonstrate that EndoVLM outperforms state-of-the-art foundation models and remains competitive with task-specific segmentation baselines using a minimal segmentation head. Remarkably, EndoVLM also exhibits robust zero-shot generalization capabilities, underscoring its immense potential as a highly adaptable backbone for scalable clinical deployment. 2 Method We propose EndoVLM, a vision-language foundation model tailored for GI endoscopy. As shown in Fig. 1, to address the asymmetry between high-redundancy, unordered image-sets and raw clinical reports, our approach begins with a data construction pipeline, and followed by three core components: AGSP to distill semantically salient frames via anatomical queries; PSAA for hierarchical semantic alignment; and SC-MAE for robust visual representation learning. Figure 1: Overview of EndoVLM. We align unordered images with reports via: AGSP for report-guided salient frame distillation and visual feature pooling, PSAA for hierarchical alignment, and SC-MAE for geometric regularization. 2.1 Preliminaries and Data Construction Given a dataset =(i,i)i=1ND=\(V_i,T_i)\_i=1^N of N examinations, each contains an unordered image-set i=Ii,kk=1KiV_i=\I_i,k\_k=1^K_i and a raw report iT_i. Standard global alignment paradigms (e.g., CLIP (Radford et al., 2021)) are suboptimal in this context due to the “many-to-many” semantic mismatch between visually redundant frames and information-dense textual findings. To bridge this semantic gap, we use Qwen3 (Yang et al., 2025) with a fixed clinical schema to parse iT_i into MiM_i semantic triplets i=(si,j,ai,j,hi,j)j=1MiS_i=\(s_i,j,a_i,j,h_i,j)\_j=1^M_i. Here, si,js_i,j represents a fine-grained textual proposition of a distinct morphological finding; ai,ja_i,j is its anatomical label (mapped to the aforementioned 17 GI regions); and hi,j∈0,1h_i,j∈\0,1\ indicates the pathological status (normal or abnormal). Detailed prompts, field definitions, and parsing examples are provided in our code repository. For feature extraction, a vision encoder EVE_V maps frames Ii,kI_i,k to local normalized embeddings iv=i,k∈ℝdk=1KiF_i^v=\v_i,k ^d\_k=1^K_i and a global average-pooled vector ¯i v_i. Concurrently, a text encoder ETE_T encodes the full report iT_i and propositions si,js_i,j into a global normalized embedding ¯i t_i and a set of fine-grained normalized embeddings it=i,j∈ℝdj=1MiH_i^t=\t_i,j ^d\_j=1^M_i, respectively. The final representation for the i-th examination is formulated as the tuple: (iv,¯i,¯i,i,j,ai,j,hi,j)(F_i^v, v_i, t_i,\t_i,j\,\a_i,j\,\h_i,j\). 2.2 Anatomy-Guided Sparse Pooling (AGSP) Endoscopic image-sets exhibit high redundancy (e.g., repetitive normal mucosa) that dilutes sparse abnormal signals. We propose AGSP, which uses fine-grained text embeddings as queries to selectively aggregate relevant visual evidence. For each sentence embedding i,jt_i,j, we compute dot-product relevance rj,k=i,j⊤i,kr_j,k=t_i,j v_i,k with all frames in iV_i. Instead of standard cross attention, we use sparse attention by selecting top-K similar frames ℐi,jI_i,j. The anatomy-specific visual representation i,jz_i,j is derived via re-normalized attention over ℐi,jI_i,j: i,j=∑k∈ℐi,jerj,k/τa∑m∈ℐi,jerj,m/τai,k,z_i,j= _k _i,j e^r_j,k/ _a _m _i,je^r_j,m/ _av_i,k, (1) where τa=0.07 _a=0.07 is the temperature and K=3K=3. A small K preserves a compact set of salient frames for each anatomical query while suppressing redundant normal views. This aligns the visual modality with the anatomy described in i,jt_i,j while filtering noise. 2.3 Progressive Semantic-Aware Alignment (PSAA) To capture both holistic context and fine-grained mucosal and pathological nuances, we employ a progressive alignment strategy. Stage 1: Cross-Patient Global Alignment. We first enforce consistency between the patient-level image-set representation ¯i v_i and the global report embedding ¯i t_i using the symmetric InfoNCE (Oord et al., 2018) loss across a batch of size B: ℒglobal=−12B∑i=1B(loge⟨¯i,¯i⟩/τ∑j=1Be⟨¯i,¯j⟩/τ+loge⟨¯i,¯i⟩/τ∑j=1Be⟨¯i,¯j⟩/τ),L_global=- 12B _i=1^B ( e v_i, t_i /τ _j=1^Be v_i, t_j /τ+ e t_i, v_i /τ _j=1^Be t_i, v_j /τ ), (2) where ⟨⋅,⋅⟩ ·,· is cosine similarity and τ is a learnable temperature. Stage 2: Cross-Anatomy Fine-Grained Alignment. Global alignment alone is insufficient for distinguishing subtle mucosa attributes. Inspired by recent dense semantic alignment formulations (Shui et al., 2025; Wang et al., 2025a), we perform fine-grained contrastive learning on a set of semantic units =(u,u)u=1MU=\(z_u,t_u)\_u=1^M, aggregating all M valid local visual-text pairs (i,j,i,j)(z_i,j,t_i,j) in the mini-batch. We first compute the softmax-normalized image-to-text and text-to-image similarity distributions, denoted as uv2tp_u^v2t and ut2vp_u^t2v, respectively. For any pair of units (u,k)(u,k) within U, the k-th element of these distributions is defined as: pu,kv2t=exp(⟨u,k⟩/τ)∑m=1Mexp(⟨u,m⟩/τ),pu,kt2v=exp(⟨u,k⟩/τ)∑m=1Mexp(⟨u,m⟩/τ).p_u,k^v2t= ( _u,t_k /τ) _m=1^M ( _u,t_m /τ), p_u,k^t2v= ( _u,z_k /τ) _m=1^M ( _u,z_m /τ). (3) To seamlessly progress from global cross-patient matching to localized alignment of anatomy and pathological status, we construct a taxonomy-aware soft target distribution u=[yu,1,…,yu,M]⊤∈ℝMy_u=[y_u,1,…,y_u,M] ^M. Let aua_u and huh_u be the categorical anatomical and pathological status labels of the u-th unit, respectively. The normalized target yu,ky_u,k is computed as: yu,k=y^u,k∑m=1My^u,m, with y^u,k=1,if au=ak and hu=hk=0;⟨u,k⟩,if au=ak and (hu=1 or hk=1);0,otherwise.y_u,k= y_u,k _m=1^M y_u,m, with y_u,k= cases1,&if a_u=a_k and h_u=h_k=0;\\ _u,t_k ,&if a_u=a_k and (h_u=1 or h_k=1);\\ 0,&otherwise. cases (4) By avoiding repulsion among identical healthy anatomies and adaptively scaling abnormal alignments via textual similarity, this formulation drives the model to autonomously disentangle anatomical and pathological semantics. The textual similarity term provides a semantic relational prior within each anatomical group, allowing related abnormal findings to share softer supervision. The fine-grained alignment loss is defined as follows, where CE denotes cross-entropy. ℒfg=12M∑u=1M(CE(u,uv2t)+CE(u,ut2v)).L_fg= 12M _u=1^M (CE(y_u,p_u^v2t)+CE(y_u,p_u^t2v) ). (5) 2.4 Semantic-Concentrated Masked Autoencoder (SC-MAE) To efficiently capture low-level textures, SC-MAE applies the MAE paradigm (He et al., 2022) exclusively to the semantic-rich subset ~i=Ii,k∈i∣k∈ℐi∗ V_i=\I_i,k _i k _i^*\, where ℐi∗=⋃jℐi,jI_i^*= _jI_i,j denotes the unified index set aggregated from all anatomy-specific frames retrieved by AGSP. It masks 75%75\% of patches in ~i V_i, and reconstruct pixels via a decoder DVD_V, yielding an MSE loss ℒMAEL_MAE. The overall training objective is as follows, where λ1,λ2 _1, _2 are balancing weights. ℒ=ℒglobal+λ1ℒfg+λ2ℒMAE.L=L_global+ _1L_fg+ _2L_MAE. (6) 3 Experiments and Results Table 1: Performance comparison. We report F1 (%) for PolypDiag, macro AUC/F1 (%) for LIMUC, Dice (%) for polyp segmentation (CVC-12k, Kvasir (Kva), ClinicDB (Clinic), Endoscene (Endo), ColonDB, ETIS). †: pre-trained with our dataset. ’–’: not reported and no public code or weights. The best and second-best results are in bold and underlined. Model Polyp Diag LIMUC CVC -12k Seen Unseen Kva Clinic Endo Colon ETIS General FMs MAE (He et al., 2022) 91.1 93.4/72.7 83.6 88.4 88.5 84.0 75.5 74.7 CLIP (Radford et al., 2021) 90.0 92.0/69.2 80.0 88.3 90.0 85.4 75.8 71.6 DINOv2 (Oquab et al., 2023) 93.3 93.8/73.6 83.8 90.7 91.4 88.6 79.8 77.8 DINOv3 (Siméoni et al., 2025) 93.9 93.9/73.7 84.5 91.0 91.1 89.1 81.6 77.7 BiomedCLIP (Zhang et al., 2023) 90.2 92.6/72.0 81.5 88.0 87.8 82.8 72.7 66.1 MAE† (He et al., 2022) 93.7 93.1/73.6 84.2 88.4 88.5 83.9 75.5 74.7 CLIP† (Radford et al., 2021) 90.1 85.2/60.2 66.2 60.1 70.6 28.8 32.8 25.6 DINOv3† (Siméoni et al., 2025) 94.8 94.0/74.0 85.8 91.4 92.2 89.5 81.6 80.1 Endo SSL EndoFM (Wang et al., 2023) 90.7 93.0/72.9 73.9 87.7 87.1 83.2 71.2 63.5 EndoFM-LV (Wang et al., 2025b) 96.3 83.3/57.4 83.2 62.2 67.7 42.5 35.0 29.8 EndoMamba (Tian et al., 2025) 95.0 92.1/70.1 85.4 87.5 87.8 86.2 69.4 60.2 EndoDINO (Dermyer et al., 2025) – 93.7/70.6 – – – – – – GastroNet-5M (Jong et al., 2025) – 95.5/– – – – – – – Task specific model (Segmentation) Polyp-PVT (Dong et al., 2021) – –/– – 91.7 93.7 90.0 80.8 78.7 VM-UNet (Ruan et al., 2024) – –/– – 91.3 92.6 88.6 79.8 76.1 PolyMamba (Fu et al., 2025) – –/– – 91.9 94.0 90.4 81.5 82.9 EndoVLM 97.3 94.5/74.4 86.4 91.9 93.1 90.8 82.8 82.0 Pre-training dataset. We retrospectively collected over 400K endoscopic examinations from two medical centers111The study was conducted in accordance with the Declaration of Helsinki and approved by the institutional review board of the lead medical center (No. 2024-KLS-489-02)., each consisting of a clinical report and its corresponding image package. By leveraging Qwen3 (Yang et al., 2025) to analyze report content, we filtered out ineligible cases—including post-operative exams, incomplete procedures (defined as gastroscopy reports missing any of the 8 standard anatomical regions, or colonoscopy reports missing any of the 9 intestinal landmarks), and other non-conforming studies. The final curated dataset contains 348K examinations, comprising more than 18.6M endoscopic images. Downstream task setup. To evaluate the versatility and robustness of EndoVLM, we conducted experiments across various tasks: (i) PolypDiag for video polyp diagnosis (Tian et al., 2022). Following EndoFM-LV (Wang et al., 2025b), we sampled 16 frames per video. Frame-level features extracted by our image-based backbone were aggregated via mean pooling to form a video-level representation, which was then classified using a linear head. (i) CVC-12k for video polyp segmentation (Bernal et al., 2015) and (i) Kvasir-SEG (Jha et al., 2019), ClinicDB (Bernal et al., 2015), ColonDB (Tajbakhsh et al., 2015), ETIS (Silva et al., 2014), EndoScene (Vázquez et al., 2017) for polyp segmentation generalization assessment. A linear segmentation head is implemented by reshaping the backbone’s patch tokens into spatial feature maps (B,D,H/P,W/P)(B,D,H/P,W/P), followed by a single convolutional layer and bilinear interpolation for final dense prediction. We adopt the same data splits and evaluation protocol as EndoFM-LV (Wang et al., 2025b) on CVC-12k, and strictly follow the training and testing setup of Polyp-PVT (Dong et al., 2021) for generalization assessment. For this Polyp-PVT generalization setup, “seen” and “unseen” denote whether a public segmentation dataset is used during downstream fine-tuning, while all downstream datasets are disjoint from the private pre-training corpus. (iv) LIMUC for ulcerative colitis severity grading (Polat et al., 2023). We append a linear classification head to the EndoVLM, and follow the same experimental settings as GastroNet5M (Jong et al., 2025). (v) Zero-Shot Transfer. We assess the model’s generalizability via zero-shot anatomical recognition on Hyper-Kvasir (Borgli et al., 2020) (covering upper & lower GI tracts) and video disease diagnosis on PolypDiag (Tian et al., 2022), directly leveraging the pre-aligned visual-semantic space without task-specific fine-tuning. Implementation details. The vision encoder and language encoder are instantiated as ViT-B/16 (Dosovitskiy et al., 2020) and PubMedBERT (Gu et al., 2021), respectively. All input images are resized to 224×224224× 224 for pre-training. The entire framework is trained for 100 epochs on NVIDIA A800 GPUs with a total batch size of 96. We optimize the network using the AdamW optimizer with a base learning rate of 1.5e-4 and a weight decay of 0.05. The loss weights λ1 _1 and λ2 _2 are empirically set to 1. Comparison experiment. We compare our method against recent SOTA methods, including general visual and vision-language FMs (DINOv2/v3 (Oquab et al., 2023; Siméoni et al., 2025), MAE (He et al., 2022), CLIP (Radford et al., 2021), BiomedCLIP (Zhang et al., 2023)), endoscopy-specific visual SSL models (EndoFM (Wang et al., 2023), EndoFM-LV (Wang et al., 2025b), EndoMamba (Tian et al., 2025), EndoDINO (Dermyer et al., 2025), GastroNet-5M (Jong et al., 2025)), and several task-specific models (Dong et al., 2021; Fu et al., 2025; Ruan et al., 2024) for polyp segmentation. We also pre-train the general FMs from scratch on our dataset. Our approach surpasses all FMs, trailing only GastroNet-5M on LIMUC. Notably, (1) On video tasks (PolypDiag/CVC-12k), our model exceeds video-specific FMs (Wang et al., 2023, 2025b; Tian et al., 2025) using a simple image-level mean-pooling aggregation, bypassing complex temporal modeling. (2) For dense prediction tasks (polyp segmentation), we achieve competitive or superior performance using only a single convolutional layer with bilinear interpolation, without the need for complex decoders present in task-specific models (Dong et al., 2021; Ruan et al., 2024; Fu et al., 2025). This confirms that our pre-training yields high-quality dense features. Superior results on unseen ColonDB and ETIS datasets further validate our model’s robust generalization. Zero-shot evaluation. We evaluate zero-shot performance on three clinical classification tasks using the prompt: "This is an image of cls." As shown in Fig. 2, EndoVLM significantly outperforms general (CLIP) and medical-specific (CLIP†, BiomedCLIP) FMs. It achieves near-perfect transferability (∼100% 100\% AUC) in Upper-GI anatomy recognition and surpasses BiomedCLIP by 18% AUC in the challenging video disease diagnosis task. These results validate that our taxonomy-aware alignment yields robust, highly transferable representations without task-specific fine-tuning. The lower ileum result reflects a category-specific limitation, as ileum descriptions are often procedural and less uniform in colonoscopy reports. Figure 2: Zero-shot performance comparison. Table 2: Ablation study of different loss components. ℒgloL_glo: Global Alignment; ℒfgL_fg: Fine-Grained Alignment; ℒMAEL_MAE: Masked Autoencoder. Components LIMUC CVC-12k ℒgloL_glo ℒfgL_fg ℒMAEL_MAE AUC Dice ✓ 85.2 66.2 ✓ ✓ 94.1 85.6 ✓ 93.1 84.2 ✓ ✓ ✓ 94.5 86.4 Ablation study. Ablation results in Table 2 validate the efficacy of our proposed components. Global alignment (ℒgloL_glo) alone–mimicking standard CLIP paradigm–yields poor performance (85.2% AUC and 66.2% Dice), as coarse patient-level matching struggles to capture localized lesion semantics. Integrating fine-grained alignment (ℒfgL_fg) triggers a significant leap, boosting AUC by 8.9% and Dice by 19.4%. which highlights the critical role of our taxonomy-aware soft targets in aligning detailed anatomical and pathological features. Finally, SC-MAE (ℒMAEL_MAE) further refines representations to achieve the best performance (94.5% AUC and 86.4% Dice), demonstrating that pixel-level reconstruction provides complementary geometric regularization to semantic alignment. Qualitative analysis and interpretability. To validate the interpretability of our AGSP module, we visualize the attention scores between the sentence embeddings i,jt_i,j and image embeddings i,kv_i,k. As shown in Fig. 3, the attention matrix exhibits highly localized and target-specific activations. EndoVLM accurately retrieves the most representative frames for eight distinct upper-GI regions (e.g., Esophagus, Duodenum) from a redundant set of over 80 images. Figure 3: Visualization of AGSP attention scores. The heatmap illustrates similarities between unordered image embeddings and anatomical text embeddings, where a-h denote specific anatomical descriptions in a gastroscopy report. Highlights identify the top-retrieved images for each specific anatomy. 4 Conclusion We present EndoVLM, the first GI VLM pre-trained on a large-scale dataset of unordered GI endoscopy image-sets and clinical reports. To bridge the profound semantic gap between redundant visual frames and structured narratives, we propose a unified framework that seamlessly integrates semantics-driven salient frame distillation, progressive cross-modal alignment, and geometric reconstruction. Pre-trained on a massive cohort of 348K cases, EndoVLM demonstrates exceptional adaptability and robust zero-shot generalization. It significantly outperforms existing uni-modal foundation models and complex task-specific architectures across diverse downstream tasks, providing a highly scalable foundation for next-generation AI-assisted endoscopy. Acknowledgments This study was supported by “Pioneer” and “Leading Goose” R&D Program of Zhejiang (2023C03050). References Bernal et al. (2015) Jorge Bernal, F Javier Sánchez, Gloria Fernández-Esparrach, Debora Gil, Cristina Rodríguez, and Fernando Vilariño. Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized medical imaging and graphics, 43:99–111, 2015. Bommasani et al. (2021) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021. Borgli et al. (2020) Hanna Borgli, Vajira Thambawita, Pia H Smedsrud, et al. Hyperkvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy. Scientific data, 7(1):283, 2020. Dermyer et al. (2025) Patrick Dermyer, Angad Kalra, and Matt Schwartz. Endodino: A foundation model for gi endoscopy. arXiv preprint arXiv:2501.05488, 2025. Dong et al. (2021) Bo Dong, Wenhai Wang, Deng-Ping Fan, Jinpeng Li, Huazhu Fu, and Ling Shao. Polyp-pvt: Polyp segmentation with pyramid vision transformers. arXiv preprint arXiv:2108.06932, 2021. Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. Fu et al. (2025) Renyu Fu, Shurui Hu, Xiao Zheng, Chang Tang, and Xinwang Liu. Polymamba: Spatial-prior guided mamba for polyp segmentation with high-frequency enhancement. In MICCAI, pages 455–465. Springer, 2025. Gu et al. (2021) Yu Gu, Robert Tinn, Hao Cheng, et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare, 3(1):1–23, 2021. He et al. (2022) Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pages 16000–16009, 2022. Hu et al. (2024) Qiang Hu, Zhenyu Yi, Ying Zhou, Fang Peng, Mei Liu, Qiang Li, and Zhiwei Wang. Sali: Short-term alignment and long-term interaction network for colonoscopy video polyp segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 531–541. Springer, 2024. Hu et al. (2025) Qiang Hu, Zhenyu Yi, Ying Zhou, Fan Huang, Mei Liu, Qiang Li, and Zhiwei Wang. Monobox: Tightness-free box-supervised polyp segmentation using monotonicity constraint. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 3572–3580, 2025. Jha et al. (2019) Debesh Jha, Pia H Smedsrud, Michael A Riegler, Pål Halvorsen, Thomas De Lange, Dag Johansen, and Håvard D Johansen. Kvasir-seg: A segmented polyp dataset. In International conference on multimedia modeling, pages 451–462. Springer, 2019. Jong et al. (2025) Martijn R Jong, Tim GW Boers, Kiki N Fockens, et al. Gastronet-5m: A multicenter dataset for developing foundation models in gastrointestinal endoscopy. Gastroenterology, 2025. Li et al. (2023) Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. NeurIPS, 36:28541–28564, 2023. Li et al. (2026) Sijing Li, Zhongwei Qiu, Jiang Liu, Wenqiao Zhang, Tianwei Lin, Yihan Xie, Jianxiang An, Boxiang Yun, Chenglin Yang, Jun Xiao, et al. Tumorchain: Interleaved multimodal chain-of-thought reasoning for traceable clinical tumor analysis. In International Conference on Learning Representations, volume 2026, pages 119790–119819, 2026. Miech et al. (2020) Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In CVPR, pages 9879–9889, 2020. Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. Oquab et al. (2023) Maxime Oquab, Timothée Darcet, Théo Moutakanni, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. Polat et al. (2023) Gorkem Polat, Haluk Tarik Kani, Ilkay Ergenc, Yesim Ozen Alahdab, Alptekin Temizel, and Ozlen Atug. Improving the computer-aided estimation of ulcerative colitis severity according to mayo endoscopic score by using regression-based deep learning. Inflammatory Bowel Diseases, 29(9):1431–1439, 2023. Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763. PmLR, 2021. Ruan et al. (2024) Jiacheng Ruan, Jincheng Li, and Suncheng Xiang. Vm-unet: Vision mamba unet for medical image segmentation. ACM Transactions on Multimedia Computing, Communications and Applications, 2024. Shui et al. (2025) Zhongyi Shui, Jianpeng Zhang, Weiwei Cao, Sinuo Wang, Ruizhe Guo, Le Lu, Lin Yang, Xianghua Ye, Tingbo Liang, Qi Zhang, et al. Large-scale and fine-grained vision-language pre-training for enhanced ct image understanding. arXiv preprint arXiv:2501.14548, 2025. Silva et al. (2014) Juan Silva, Aymeric Histace, Olivier Romain, Xavier Dray, and Bertrand Granado. Toward embedded detection of polyps in wce images for early diagnosis of colorectal cancer. International journal of computer assisted radiology and surgery, 9(2):283–293, 2014. Siméoni et al. (2025) Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508.10104, 2025. Tajbakhsh et al. (2015) Nima Tajbakhsh, Suryakanth R Gurudu, and Jianming Liang. Automated polyp detection in colonoscopy videos using shape and context information. IEEE transactions on medical imaging, 35(2):630–644, 2015. Tang et al. (2022) Yucheng Tang, Dong Yang, Wenqi Li, Holger R Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh. Self-supervised pre-training of swin transformers for 3d medical image analysis. In CVPR, pages 20730–20740, 2022. Tian et al. (2025) Qingyao Tian, Huai Liao, Xinyan Huang, Bingyu Yang, Dongdong Lei, Sebastien Ourselin, and Hongbin Liu. Endomamba: an efficient foundation model for endoscopic videos via hierarchical pre-training. In MICCAI, pages 224–234. Springer, 2025. Tian et al. (2022) Yu Tian, Guansong Pang, Fengbei Liu, Yuyuan Liu, Chong Wang, Yuanhong Chen, Johan Verjans, and Gustavo Carneiro. Contrastive transformer-based multiple instance learning for weakly supervised polyp frame detection. In MICCAI, pages 88–98. Springer, 2022. Tiu et al. (2022) Ekin Tiu et al. Expert-level detection of pathologies from unannotated chest x-ray images via self-supervised learning. Nature Biomedical Engineering, 6(12):1399–1406, 2022. Vázquez et al. (2017) David Vázquez, Jorge Bernal, F Javier Sánchez, Gloria Fernández-Esparrach, Antonio M López, Adriana Romero, Michal Drozdzal, and Aaron Courville. A benchmark for endoluminal scene segmentation of colonoscopy images. Journal of healthcare engineering, 2017(1):4037190, 2017. Wang et al. (2025a) Rongsheng Wang, Fenghe Tang, Qingsong Yao, Rui Yan, Xu Zhang, Zhen Huang, Haoran Lai, Zhiyang He, Xiaodong Tao, Zihang Jiang, et al. Simcrop: Radiograph representation learning with similarity-driven cross-granularity pre-training. In MICCAI, pages 563–573. Springer, 2025a. Wang et al. (2024) Xiyue Wang, Junhan Zhao, Eliana Marostica, Wei Yuan, Jietian Jin, Jiayu Zhang, Ruijiang Li, Hongping Tang, Kanran Wang, Yu Li, et al. A pathology foundation model for cancer diagnosis and prognosis prediction. Nature, 634(8035):970–978, 2024. Wang et al. (2023) Zhao Wang, Chang Liu, Shaoting Zhang, and Qi Dou. Foundation model for endoscopy video analysis via large-scale self-supervised pre-train. In MICCAI, pages 101–111. Springer, 2023. Wang et al. (2025b) Zhao Wang, Chang Liu, Lingting Zhu, Tongtong Wang, Shaoting Zhang, and Qi Dou. Improving foundation model for endoscopy video analysis via representation learning on long sequences. IEEE Journal of Biomedical and Health Informatics, 2025b. Wang et al. (2022) Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medclip: Contrastive learning from unpaired medical images and text. In EMNLP, pages 3876–3887, 2022. Xu et al. (2021) Jianwei Xu, Ran Zhao, Yizhou Yu, Qingwei Zhang, Xianzhang Bian, Jun Wang, Zhizheng Ge, and Dahong Qian. Real-time automatic polyp detection in colonoscopy using feature enhancement module and spatiotemporal similarity correlation unit. Biomedical Signal Processing and Control, 66:102503, 2021. Yang et al. (2025) An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Zhang et al. (2023) Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.00915, 2023.