Paper deep dive
Cross-Modal Contrastive Learning for the Retrieval of Immunotherapy-Associated Molecular Signatures from Histopathology
Sigrid Vila-Bagaria, Mar TeixidĆ³, Miquel PiƱol, Felip Vilardell, Robert Montal, Veronica Vilaplana
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Gastric Adenocarcinoma is a leading cause of cancer mortality. Although "Inflamed/Non-Inflamed" subtypes have been proposed to predict immunotherapy response, their identification relies on a costly 10-gene RNA signature. We propose a Cross-modal Contrastive Multiple Instance Learning (CCMIL) framework for cross-modal retrieval, imputing these molecular signatures directly from standard Hematoxylin & Eosin (H&E) slides. By leveraging a supervised contrastive objective, CCMIL aligns visual morphological patterns with molecular phenotypes into a shared latent space. This establishes an interpretable search-by-case retrieval engine, enabling pathologists to query a whole slide image to surface transcriptomically coherent neighbors and approximate RNA signatures without genomic sequencing at inference. Our results demonstrate that this retrieval-first approach captures the continuous phenotypic spectrum of tumor inflammation and yields clinically interpretable attention heatmaps. Furthermore, the learned representation also supports competitive downstream classification, providing a practical molecular pre-screening strategy.
Tags
Links
- Source: https://arxiv.org/abs/2610.05157v1
- Canonical: https://arxiv.org/abs/2610.05157v1
Trouble viewing inline? Open PDF directly ā
Full Text
28,003 characters extracted from source content.
Expand or collapse full text
Cross-Modal Contrastive Learning for the Retrieval of Immunotherapy-Associated Molecular Signatures from Histopathology Sigrid Vila-Bagaria Affiliation: Signal Theory and Communications Department (TSC) Universitat PolitĆØcnica de Catalunya - BarcelonaTech (UPC) E-mail sigrid.vila,veronica.vilaplana@upc.edu Mar TeixidĆ³ Affiliation: Cancer Biomarkers Research Group (GReBiC) - IRB Lleida E-mail mteixido,rmontal@irblleida.cat Miquel PiƱol Affiliation: Cancer Biomarkers Research Group (GReBiC) - IRB Lleida E-mail mteixido,rmontal@irblleida.cat Felip Vilardell Affiliation: Cancer Biomarkers Research Group (GReBiC) - IRB Lleida E-mail mteixido,rmontal@irblleida.cat Robert Montal Affiliation: Cancer Biomarkers Research Group (GReBiC) - IRB Lleida E-mail mteixido,rmontal@irblleida.cat Veronica Vilaplana Affiliation: Signal Theory and Communications Department (TSC) Universitat PolitĆØcnica de Catalunya - BarcelonaTech (UPC) E-mail sigrid.vila,veronica.vilaplana@upc.edu Abstract Gastric Adenocarcinoma is a leading cause of cancer mortality. Although āInflamed/Non-Inflamedā subtypes have been proposed to predict immunotherapy response, their identification relies on a costly 10-gene RNA signature. We propose a Cross-modal Contrastive Multiple Instance Learning (CCMIL) framework for cross-modal retrieval, imputing these molecular signatures directly from standard Hematoxylin & Eosin (H&E) slides. By leveraging a supervised contrastive objective, CCMIL aligns visual morphological patterns with molecular phenotypes into a shared latent space. This establishes an interpretable search-by-case retrieval engine, enabling pathologists to query a whole slide image to surface transcriptomically coherent neighbors and approximate RNA signatures without genomic sequencing at inference. Our results demonstrate that this retrieval-first approach captures the continuous phenotypic spectrum of tumor inflammation and yields clinically interpretable attention heatmaps. Furthermore, the learned representation also supports competitive downstream classification, providing a practical molecular pre-screening strategy. Keywords: Cross-Modal Retrieval Contrastive Learning Gastric Cancer Molecular Subtyping Multiple Instance Learning. 1 Introduction Gastric adenocarcinoma (GAC) is a leading cause of cancer mortality globally [22], typically diagnosed at advanced stages with poor prognosis [1]. While immune checkpoint inhibitors (ICIs) have become a standard of care [8, 9, 18], heterogeneous responses demand better predictive biomarkers [2]. Recently, a distinct āInflamed/Non-Inflamedā GAC phenotype, characterized by high immunogenicity, was identified as a strong predictor of ICI response [19]. However, determining this class requires a costly 10-gene RNA signature, limiting its routine clinical use. By learning to impute this molecular signature directly from low-cost, standard H&E features, our framework bypasses the need for genomic sequencing at inference, providing a scalable pre-screening tool. Deep learning (DL) in computational pathology offers an accessible alternative to molecular assays [14]. However, learning the āInflamed/Non-Inflamedā phenotype directly from H&E faces two key challenges. First, paired histology-genomic datasets are scarce, increasing the risk of overfitting. Second, binarizing a continuous gene expression signature into a label introduces noise and discards the patientās true position along the biological phenotypic spectrum. To address these challenges, we investigate metric learning to build a highly interpretable search-by-case diagnostic tool. The contributions of this work are as follows. ā¢ We propose a Cross-modal Contrastive Multiple Instance Learning (CCMIL) framework that learns a shared histology-transcriptomics embedding. ā¢ We demonstrate that the learned representation captures biologically coherent relationships between patients, enabling interpretable RNA signature imputation and competitive downstream classification from H&E alone. 2 Related Work Contrastive learning in computational pathology: Contrastive learning optimizes latent spaces by pulling positive pairs together and pushing negative pairs apart [5]. Frameworks like ConVIRT [26] and CLIP [17] proved this highly effective for cross-modal retrieval in the vision-language domain. Medical adaptations, such as MedCLIP [23], apply these principles to healthcare, maximizing similarity between matched modalities (e.g., histology and genomics) to capture patient-specific biological associations. Crucially, establishing this tightly clustered joint manifold provides the foundational architecture required for robust retrieval tasks. Cross-modal retrieval in computational pathology: Accurate tumor subtyping is fundamental to precision oncology, yet unimodal MIL frameworks (e.g., ABMIL [6], CLAM [14], TransMIL [20]) often fail to capture the full transcriptomic complexity of heterogeneous tumors. To address this, multimodal frameworks like MCAT [3] fuse histological and genomic data, but strictly require both modalities at inference, maintaining high sequencing costs. Emerging approaches instead leverage contrastive learning to align modalities [12], integrating spatial transcriptomics [16] or enforcing semantic consistency [24] to enhance robustness. Building upon these advances, our framework exploits this aligned multimodal manifold specifically to enable cross-modal retrieval, allowing clinicians to impute molecular signatures using only histology at inference. ((a)) CCMIL Architecture Latent Dimension 1Latent Dimension 2Non-InflamedInflamedClass Separation & Modality Alignment(SupCon Loss + BCE Loss)LegendH&E SlideRNA SignatureContrastive Pair ((b)) Joint Latent Space Figure 1: The CCMIL framework. (a) Dual-stream architecture aligning WSI and RNA features during training to enable inference from H&E alone. (b) The composite objective aligns modalities and separates āInflamed/Non-Inflamedā cases in the joint latent space. Table 1: Quantitative performance comparison of the proposed framework against state-of-the-art baselines. All values are reported as mean Ā± standard deviation across 5-fold cross-validation. Retrieval Retrieval (Image ā RNA) k-N Classification Model Q ā Tgt R@1 R@5 MRR Acc. F1 Baseline Te ā Tr ā ā .71 Ā±.106 .57 Ā±.129 .53 Ā±.109 Baseline Te ā Te .67 Ā±.115 .89 Ā±.086 .78 Ā±.067 .67 Ā±.107 .65 Ā±.074 CCMIL Te ā Tr ā ā .75 Ā± 0.049 .68 Ā±.080 .61 Ā±.138 CCMIL Te ā Te .72 Ā±.085 .92 Ā±.051 .80 Ā±.067 .71 Ā±.155 .67 Ā±.133 Classification Modality Model Acc. AUC F1 Prec. Rec. Unimodal CLAM .73 Ā±.035 .79 Ā±.034 .72 Ā±.051 .68 Ā±.031 .76 Ā±.065 ABMIL .72 Ā±.021 .79 Ā±.078 .70 Ā±.022 .68 Ā±.048 .72 Ā±.073 TransMIL .52 Ā±.117 .72 Ā±.082 .62 Ā± 0.027 .51 Ā± 0.104 .85 Ā±.191 Multimod. MCAT .66 Ā± 0.065 .68 Ā±.072 .66 Ā±.101 .61 Ā±.076 .72 Ā±.231 Cross-mod. CCMIL .72 Ā±.157 .75 Ā±.135 .76 Ā±.112 .68 Ā±.117 .92 Ā±.083 Table 2: Ablation study of the framework evaluating the Tumour Histology Conditioned Attention (THCA) and loss combinations for Classification and Retrieval. Components Classification Retrieval THCA SupConL ClassL Acc. AUC F1 R@1 R@5 ā ā ā .54 Ā±.092 .45 Ā±.156 .66 Ā±.069 .69 Ā±.068 .88 Ā±.068 ā ā ā .68 Ā±.103 .68 Ā±.073 .72 Ā±.076 .63 Ā±.119 .84 Ā±.068 ā ā ā .72 Ā±.106 .74 Ā±.084 .74 Ā±.080 .60 Ā±.069 .84 Ā±.107 ā ā ā .72 Ā±.157 .75 Ā±.135 .76 Ā±.112 .72 Ā±.085 .92 Ā±.051 ((a)) UMAP visualization of the joint latent space. ((b)) Mean RNA cosine similarity across top-k retrieval ranks for cross-modal strategies. Figure 2: Evaluation of the proposed framework. (a) Latent space visualization via UMAP demonstrating a structured joint representation. (b) Mean RNA Cosine Similarity between the query ground truth and retrieved cases across retrieval rank k. CSLS + Reciprocal strategy remains closest to the Oracle (retrieval in the RNA space) upper bound. ((a)) Spatial attention maps across models compared to ground-truth tissue segmentation (tumor in red, non-tumor in blue). ((b)) Continuous inflammatory spectrum: Test WSIs along the axis vector between class centroids, alongside their ground-truth and retrieved RNA expression profiles. Figure 3: Qualitative evaluation of CCMIL. (a) CCMIL generates dense, highly localized attention hotspots compared to the baseline models. (b) The joint latent space captures a continuous inflammatory gradient, successfully retrieving transcriptomically coherent neighbors. 3 Method 3.1 Cross-modal contrastive architecture The framework, as shown in Figure 1(a), consists of two parallel encoding streams that map histological and genomic data to a shared joint embedding space. Histological stream: Visual features extracted from the frozen CONCH backbone [13] are refined by a visual adapter (a 2-layer MLP with layer normalization and ReLU) into a task-specific latent space. To preserve spatial structure within the permutation-invariant MIL setting, we employ Fourier Positional Encodings, mapping normalized patch coordinates to a high-dimensional frequency domain. Next, because different tumor histologies present distinct regions of interest, we compute the final slide-level embedding using our Tumor Histology Conditioned Attention (THCA), a lightweight gated-attention MIL module that conditions each patchās attention score on a learned histology-type embedding, yielding a histology-weighted sum of the adapted features. THCA guides the network to learn histology-specific attention profiles. Finally, an MLP projects these features into a shared space with the RNA embeddings, while a concurrently trained probe (2-layer MLP) enables direct WSI classification at inference. Genomic stream: We encode the 10-gene expression signature into a latent representation utilizing a permutation-invariant embedding scheme followed by a Self-Normalizing Neural Network (SNN) [10]. We assign a learnable identity vector to each gene. The initial aggregation is computed as the expression-weighted sum of these identities, since it learns a gene-specific semantic space while preserving permutation invariance. The aggregated vector is refined by an SNN composed of linear layers with Scaled Exponential Linear Units (SELU) and Alpha Dropout. Finally, similar to the histological stream, the features are projected to a common space with the RNA features using an MLP. 3.2 Losses To address label ambiguity and data scarcity, we optimize a composite objective to promote the latent space to be structured by molecular phenotype and separable for downstream classification. ātotal=āBCE+āSupConL_total=L_BCE+L_SupCon (1) Representation Learning (Supervised Contrastive Loss): We use a multi-modal Supervised Contrastive Loss (āSupConL_SupCon) over the joint representations. Within a mini-batch of WSI and RNA embeddings, all features are first L2L_2 normalized and concatenated into a single set I of size 2āN2N (where N is the batch size). For any anchor iāIiā I with embedding ziz_i and discrete label yiy_i, the set of positive keys Pā”(i)P(i) consists of all other samples in the batch sharing the same label (pāIāi:yp=yipā I \i\:y_p=y_i). Because the WSI and RNA embeddings from the same patient inherently share the same label, this formulation naturally groups both instance-level cross-modal pairs and class-level pairs (any intra- or inter-modal embeddings with the same phenotype). The loss is formulated as: āSupCon=12āNāāiāIā1|Pā”(i)|āāpāPā”(i)logā”expā”(ziā zp/Ļ)āaāIāiexpā”(ziā za/Ļ)L_SupCon= 12N _iā I -1|P(i)| _pā P(i) (z_iĀ· z_p/Ļ) _aā I \i\ (z_iĀ· z_a/Ļ) (2) where ziā zpz_iĀ· z_p represents the cosine similarity between the embeddings, and the temperature parameter is Ļ=0.07Ļ=0.07. Task Supervision (Classification Loss): To enforce separability and refine the decision boundaries of the shared projection manifold, we apply a classification probe to both the aggregated WSI features and the genomic RNA features. The total classification loss is computed as the unweighted average of the Binary Cross-Entropy (BCE) losses from both modalities. This equal weighting encourages that both the histological and genomic representations independently maintain high diagnostic utility and are mapped correctly with respect to the target taskās decision boundary. 3.3 Inference strategy At inference, the genomic stream is detached, enabling unimodal operation on standard H&E WSIs to bypass sequencing costs while retaining multimodal benefits. The resulting embedding can be exploited in two complementary ways. Direct Classification: The pre-trained probe projects the histological slide-level embedding onto the decision boundary to output a probability score. Retrieval Strategy: Given a query H&E slide, its embeddings are projected into the shared latent space and used to retrieve the most similar RNA embeddings from the reference dataset. We evaluated several cross-modal strategies: Inverted Softmax [21], Ī±-CSLS [11], cosine k-N, Diffusion [7], and AQE [4]. We found CSLS combined with Reciprocal reranking to provide the best retrieval performance, mitigating hubness and refining cross-domain alignment. By retrieving the top-k (k=5k=5) nearest neighbors using this strategy we can aggregate their corresponding RNA profiles to reliably impute a continuous molecular signature for the queried slide. 4 Experimental setup 4.1 Dataset and preprocessing We utilized a curated subset of the TCGA Gastric Adenocarcinoma cohort comprising N=265N=265 diagnostic WSIs, manually reviewed by expert pathologists to ensure optimal tissue quality. The dataset consists of 142 āNon-Inflamedā and 123 āInflamedā cases, with ground truth labels derived from a 10-gene RNA signature (ADORA2A, BLM, CCL20, CD3E, CD5, CD74, GLS, HLA-E, ITGAL, RPL23) [19]. Additionally, cases were stratified by expert pathologists according to the histology Lauren classification into 173 Intestinal, 65 Diffuse, and 16 Mixed subtypes. WSI processing was standardized using the Trident [25] framework for tissue segmentation and non-overlapping patching at 40Ć. Finally, patch-level embeddings were extracted using the CONCH [15] foundation model. 4.2 Training setup Framework and hardware All experiments were implemented using the PyTorch framework, leveraging the pytorch-metric-learning library for loss formulations. Training was accelerated on a single NVIDIA A40 GPU (48GB VRAM). Due to the computational overhead of processing dense WSI feature bags, we utilized a batch size of B=8B=8 patients with gradient accumulation of 4 steps, which was the maximum capacity allowed by memory constraints. Optimization The network was trained for 50 epochs per fold using a 5-fold cross-validation scheme with early-stopping. Each fold was composed of a train, validation and test sets of 80%, 10% and 10% of the cohort respectively. We employed the Adam optimizer with a fixed learning rate of 5Ć10ā55Ć 10^-5 and the loss defined in Section 3.2. Baselines We compare CCMIL against representative state-of-the-art unimodal (H&E-only) and multimodal (H&E + RNA) approaches. We evaluate three widely used unimodal MIL models: ABMIL [6], CLAM-SB [14], and TransMIL [20]. For multimodal fusion, we compare against MCAT [3], which explicitly co-attends and fuses WSI and genomic features. For fair comparison, all models utilize identical CONCH embeddings and are evaluated across the same 5-fold splits. To isolate the contribution of our class-aware supervised contrastive framework, we evaluate a retrieval baseline that utilizes the same spatial architecture but relies solely on instance-level alignment (CLIP loss [17]). This setup mirrors standard multimodal contrastive methods widely used in the field. 5 Results 5.1 Quantitative Evaluation We first select the retrieval strategy by evaluating multiple metrics on the validation set. As demonstrated in Figure 2(b), CSLS + Reciprocal reranking mitigates high-dimensional hubness and maintains the highest RNA cosine similarity to the Oracle upper bound for reliable signature imputation. Using this retrieval strategy, the 5-fold cross-validation results (Table 1) demonstrate the effectiveness of CCMIL as a cross-modal retrieval engine. We report performance across two retrieval settings: Test ā Train, mimicking real-world clinical inference by querying a new test slide against an established set of training cases, and Test ā Test, which retrieves entirely within the test set to accurately compute R@1 and R@5 scores (percentage of queries that retrieve a match in the top k results). Compared with the CLIP baseline, our class-aware formulation achieves a Test ā Test R@5 of 0.92 and MRR of 0.80. Aggregating these retrieved WSI via weighted k-N voting in the Test ā Train setting yields a classification Accuracy of 0.72 and an F1-score of 0.76. This learned latent space also enables effective downstream classification. CLAM achieves comparable Accuracy (0.73) but lacks clinical balance, while TransMIL and multimodal MCAT yield poor Precision (0.51, 0.61) due to domain gaps. Conversely, CCMIL achieves the highest F1-score (0.76) and Recall (0.92) indicating improved sensitivity for ICI pre-screening. Our ablation study (Table 2) confirms our architectural choices: representation learning alone yields poor decision boundaries (Accuracy 0.54), omitting the contrastive signal degrades retrieval (R@1 0.63), and replacing THCA with standard attention impairs both fine-grained cross-modal retrieval (R@1 0.60) and overall F1-score (0.74). 5.2 Qualitative Evaluation Spatial attention heatmaps (Fig. 3(a)) highlight the interpretability of our framework. Compared to the baseline methods, which exhibit sparse under-activation (ABMIL), severe attention saturation (MCAT), or diffuse noise (CLAM), CCMIL generates dense, localized hotspots that align closely with ground-truth tumor segmentations. Because the āInflamedā signature is visually indistinguishable to pathologists, these biologically coherent activations suggest CCMIL captures subtle morphological patterns associated with the āInflamedā phenotype. Furthermore, we show that the embedding space captures a continuous phenotype (Fig. 3(b)). By projecting test slides onto an inflammation axis (the vector between class centroids), we selected seven evenly-spaced query slides. Displaying these alongside their retrieved RNA expression profiles (via Slide ā RNA CSLS+Reciprocal reranking at ranks 1, 2, 3, and 10) confirms that CCMIL organizes slides along a biologically meaningful gradient, surfacing transcriptomically coherent neighbors even at larger retrieval ranks. 6 Discussion and Conclusions We introduce CCMIL, a Cross-modal Contrastive MIL framework that enables interpretable retrieval of molecular signatures directly from standard H&E slides. By integrating a composite loss with our THCA module, CCMIL learns a shared histology-transcriptomics latent space during training, improving cross-modal alignment relative to the evaluated baselines. Beyond binary classification, CCMIL acts as an interpretable search-by-case engine, allowing pathologists to retrieve transcriptomically coherent neighbors and impute a patientās continuous inflammatory phenotype. This learned retrieval-oriented representation also supports effective downstream classification, yielding a highly sensitive classifier for immunotherapy pre-screening. While our cross-validation demonstrates consistent performance within the TCGA cohort, this proof-of-concept has been evaluated only on a retrospective cohort. Future work will focus on validating the framework in an independent external cohort to assess generalizability. Ultimately, by providing biologically meaningful spatial heatmaps and bypassing genomic sequencing at inference, CCMIL offers a scalable framework for precision immuno-oncology. Acknowledgements This work was supported by the research project PID2023-148614OB-I00, funded by MICIU/AEI/10.13039/501100011033 and by FEDER, EU. The research conducted by Sigrid Vila-Bagaria is supported by the AGAUR-FI predoctoral programme (2025 FI-STEP 00032) of the Secretariat for Universities and Research of the Department of Research and Universities of the Generalitat of Catalonia, and by the European Social Fund Plus. This study has been funded by Instituto de Salud Carlos I (ISCIII) through the research project āPI24/01017ā, and co-funded by the European Union. MTM acknowledges the support from DiputaciĆ³ de Lleida ā La forƧa dels municipis and from the Intramural Research Program of Institut de Recerca BiomĆØdica de Lleida (IRBLleida). RM acknowledges the support from Instituto de Salud Carlos I (Juan RodĆ©s contract: JR21/00039) co-funded by the European Union. Disclosure of Interests. The authors have no competing interests to declare that are relevant to the content of this article. References [1] J. A. Ajani, J. Lee, T. Sano, Y. Y. Janjigian, D. Fan, and S. Song (2017) Gastric adenocarcinoma. Nature reviews Disease primers 3 (1), p. 1ā19. Cited by: Ā§1. [2] E. Borcoman, Y. Kanjanapan, S. Champiat, S. Kato, V. Servois, R. Kurzrock, S. Goel, P. Bedard, and C. Le Tourneau (2019) Novel patterns of response under immunotherapy. Annals of Oncology 30 (3), p. 385ā396. Cited by: Ā§1. [3] R. J. Chen, M. Y. Lu, W. Weng, T. Y. Chen, D. F. Williamson, T. Manz, M. Shady, and F. Mahmood (2021) Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 4015ā4025. Cited by: Ā§2, Ā§4.2. [4] O. Chum, J. Philbin, J. Sivic, M. Isard, and A. Zisserman (2007) Total recall: automatic query expansion with a generative feature model for object retrieval. In 2007 IEEE 11th international conference on computer vision, p. 1ā8. Cited by: Ā§3.3. [5] R. Hadsell, S. Chopra, and Y. LeCun (2006) Dimensionality reduction by learning an invariant mapping. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPRā06), Vol. 2, p. 1735ā1742. Cited by: Ā§2. [6] M. Ilse, J. Tomczak, and M. Welling (2018) Attention-based deep multiple instance learning. In International conference on machine learning, p. 2127ā2136. Cited by: Ā§2, Ā§4.2. [7] A. Iscen, G. Tolias, Y. Avrithis, T. Furon, and O. Chum (2017) Efficient diffusion on region manifolds: recovering small objects with compact cnn representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 2077ā2086. Cited by: Ā§3.3. [8] Y. Y. Janjigian, A. Kawazoe, Y. Bai, J. Xu, S. Lonardi, J. P. Metges, P. Yanez, L. S. Wyrwicz, L. Shen, Y. Ostapenko, et al. (2023) Pembrolizumab plus trastuzumab and chemotherapy for her2-positive gastric or gastro-oesophageal junction adenocarcinoma: interim analyses from the phase 3 keynote-811 randomised placebo-controlled trial. The Lancet 402 (10418), p. 2197ā2208. Cited by: Ā§1. [9] Y. Y. Janjigian, K. Shitara, M. Moehler, M. Garrido, P. Salman, L. Shen, L. Wyrwicz, K. Yamaguchi, T. Skoczylas, A. C. Bragagnoli, et al. (2021) First-line nivolumab plus chemotherapy versus chemotherapy alone for advanced gastric, gastro-oesophageal junction, and oesophageal adenocarcinoma (checkmate 649): a randomised, open-label, phase 3 trial. The Lancet 398 (10294), p. 27ā40. Cited by: Ā§1. [10] G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter (2017) Self-normalizing neural networks. Advances in neural information processing systems 30. Cited by: Ā§3.1. [11] G. Lample, A. Conneau, M. Ranzato, L. Denoyer, and H. JĆ©gou (2018) Word translation without parallel data. In International conference on learning representations, Cited by: Ā§3.3. [12] T. Li, X. Zhou, J. Xue, L. Zeng, Q. Zhu, R. Wang, H. Yu, and J. Xia (2025) Cross-modal alignment and contrastive learning for enhanced cancer survival prediction. Computer Methods and Programs in Biomedicine 263, p. 108633. Cited by: Ā§2. [13] M. Y. Lu, B. Chen, D. F. Williamson, R. J. Chen, I. Liang, T. Ding, G. Jaume, I. Odintsov, L. P. Le, G. Gerber, et al. (2024) A visual-language foundation model for computational pathology. Nature Medicine 30, p. 863ā874. Cited by: Ā§3.1. [14] M. Y. Lu, D. F. Williamson, T. Y. Chen, R. J. Chen, M. Barbieri, and F. Mahmood (2021) Data-efficient and weakly supervised computational pathology on whole-slide images. Nature biomedical engineering 5 (6), p. 555ā570. Cited by: Ā§1, Ā§2, Ā§4.2. [15] M. Y. Lu, B. Chen, A. Zhang, D. F. K. Williamson, R. J. Chen, T. Ding, L. P. Le, Y. Chuang, and F. Mahmood (2023) Visual language pretrained multiple instance zero-shot transfer for histopathology images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 19764ā19775. Cited by: Ā§4.1. [16] W. Min, Z. Shi, J. Zhang, J. Wan, and C. Wang (2024) Multimodal contrastive learning for spatial gene expression prediction using histology images. Briefings in Bioinformatics 25 (6), p. bbae551. Cited by: Ā§2. [17] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748ā8763. Cited by: Ā§2, Ā§4.2. [18] S. Y. Rha, D. Oh, P. YaƱez, Y. Bai, M. Ryu, J. Lee, F. Rivera, G. V. Alves, M. Garrido, K. Shiu, et al. (2023) Pembrolizumab plus chemotherapy versus placebo plus chemotherapy for her2-negative advanced gastric cancer (keynote-859): a multicentre, randomised, double-blind, phase 3 trial. The Lancet Oncology 24 (11), p. 1181ā1195. Cited by: Ā§1. [19] J. V. Rodriguez, M. PiƱol, M. A. Sorolla, E. Parisi, A. Sorolla, M. Santacana, M. Ruiz, G. Parra, M. Bernabeu, M. Iglesias, et al. (2025) Comprehensive immunophenotyping of gastric adenocarcinoma identifies an inflamed class of tumors amenable to immunotherapies. Journal for Immunotherapy of Cancer 13 (3), p. e010024. Cited by: Ā§1, Ā§4.1. [20] Z. Shao, H. Bian, Y. Chen, Y. Wang, J. Zhang, X. Ji, et al. (2021) Transmil: transformer based correlated multiple instance learning for whole slide image classification. Advances in neural information processing systems 34, p. 2136ā2147. Cited by: Ā§2, Ā§4.2. [21] S. L. Smith, D. H. Turban, S. Hamblin, and N. Y. Hammerla (2017) Offline bilingual word vectors, orthogonal transformations and the inverted softmax. arXiv preprint arXiv:1702.03859. Cited by: Ā§3.3. [22] H. Sung, J. Ferlay, R. L. Siegel, M. Laversanne, I. Soerjomataram, A. Jemal, and F. Bray (2021) Global cancer statistics 2020: globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: a cancer journal for clinicians 71 (3), p. 209ā249. Cited by: Ā§1. [23] Z. Wang, Z. Wu, D. Agarwal, and J. Sun (2022) Medclip: contrastive learning from unpaired medical images and text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, Vol. 2022, p. 3876. Cited by: Ā§2. [24] S. Yu, Y. Kim, H. Kim, S. Lee, and K. Kim (2025) Contrastive learning for omics-guided whole-slide visual embedding representation. bioRxiv, p. 2025ā01. Cited by: Ā§2. [25] A. Zhang, G. Jaume, A. Vaidya, T. Ding, and F. Mahmood (2025) Accelerating data processing and benchmarking of ai models for pathology. arXiv preprint arXiv:2502.06750. Cited by: Ā§4.1. [26] Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz (2022) Contrastive learning of medical visual representations from paired images and text. In Machine learning for healthcare conference, p. 2ā25. Cited by: Ā§2.