Paper deep dive
Data-Efficient Multimodal Alignment for Histopathology-based Molecular Prediction
Dominik Winter, Dominik Vonficht, Loïc Le Bescond, Christian Gebbe, Marco Rosati, Richard J. Chen, Markus Schick, Ross Stewart, Nicolas Brieu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 7/5/2026, 3:09:56 AM
Summary
The paper introduces a lightweight, data-efficient framework for multimodal alignment between H&E-stained whole-slide images (WSIs) and bulk RNA-seq data. By training a small MLP projection head on top of frozen foundation models (CONCH for images and BulkFormer/scGPT for transcriptomics), the authors enable 'open-vocabulary molecular prompting.' This allows researchers to query H&E slides with arbitrary gene-set signatures to predict pathway activity without retraining. The method, specifically the BulkFormer MLP-CLIP approach with Soft-kNN, demonstrates significant improvements in cross-modal retrieval and successfully predicts morphologically grounded pathways (e.g., cell-cycle, immune-related) and clinical biomarkers (e.g., NSCLC subtypes, PD-L1 expression) in the POSEIDON clinical trial setting.
Entities (10)
Relation Signals (5)
CONCH → isusedasimageencoderfor → H&E-stained whole-slide images
confidence 100% · aggregating CONCH ViT features via attention MIL into d_v=1024-dim embeddings
BulkFormer → isusedasrnaencoderfor → Bulk RNA-seq
confidence 100% · BulkFormer, purpose-built for bulk RNA-Seq
MLP-CLIP → aligns → H&E-stained whole-slide images and Bulk RNA-seq
confidence 90% · BulkFormer MLP-CLIP achieves the best performance in R@5=56.0%
H&E-stained whole-slide images → canpredict → NSCLC subtype identity
confidence 90% · H&E-predicted squamous cell carcinoma scores recapitulate NSCLC subtype identity
H&E-stained whole-slide images → canpredict → PD-L1 expression
confidence 90% · predicted IFN-gamma mirror PD-L1 tumor-cell expression groups
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:H&E-stained whole-slide images offer cohort-scale availability and rich spatial context but lack molecular specificity, whereas bulk RNA-seq provides transcriptome-wide resolution at high cost with limited archival availability. We show that training a lightweight alignment module atop frozen histopathology and RNA-Seq foundation models enables open-vocabulary molecular prompting -- querying H&E slides with gene-set signatures to predict pathway activity without sequencing or end-to-end retraining. Using contrastive learning on a multi-cancer cohort (N=1,720), we achieve a 25-fold improvement in retrieval over baseline methods. Systematic analysis reveals a graduated predictability spectrum: morphologically grounded programs (cell-cycle programs, immune-related) are most reliably predicted (R^2>0.5), while predicting pathways with no morphological footprint remains challenging as expected. We validate clinical utility on the POSEIDON clinical trial: H&E-predicted squamous cell carcinoma scores recapitulate NSCLC subtype identity and predicted IFN-gamma mirror PD-L1 tumor-cell expression groups. Furthermore, genesets describing immune activation and fibrosis predict known tumor microenvironment archetypes from histology alone. We further validate generalization of our approach across unseen cohorts and demonstrate data-efficient domain adaptation, establishing a slide-native framework for molecular analysis on H&E images.
Tags
Links
- Source: https://arxiv.org/abs/2606.29949v1
- Canonical: https://arxiv.org/abs/2606.29949v1
Trouble viewing inline? Open PDF directly →
Full Text
27,983 characters extracted from source content.
Expand or collapse full text
Data-Efficient Multimodal Alignment for Histopathology-based Molecular Prediction Dominik Winter 1 , Dominik Vonficht 1 , Loïc Le Bescond 1 , Christian Gebbe 1 , Marco Rosati 1 , Richard J. Chen 1 , Markus Schick 1 , Ross Stewart 2 , and Nicolas Brieu 1 1 AstraZeneca Computational Pathology and Biomarkers, Munich, Germany 2 AstraZeneca, Early Oncology and Translation Medicine, Cambridge, UK Abstract. H&E-stained whole-slide images offer cohort-scale availabil- ity and rich spatial context but lack molecular specificity, whereas bulk RNA-seq provides transcriptome-wide resolution at high cost with lim- ited archival availability. We show that training a lightweight alignment module atop frozen histopathology and RNA-Seq foundation models en- ables open-vocabulary molecular prompting-querying H&E slides with gene-set signatures to predict pathway activity without sequencing or end-to-end retraining. Using contrastive learning on a multi-cancer co- hort (N=1,720), we achieve a 25-fold improvement in retrieval over base- line methods. Systematic analysis reveals a graduated predictability spec- trum: morphologically grounded programs (cell-cycle programs, immune- related) are most reliably predicted (R 2 >0.5), while predicting pathways with no morphological footprint remains challenging as expected. We vali- date clinical utility on the POSEIDON clinical trial: H&E-predicted squa- mous cell carcinoma scores recapitulate NSCLC subtype identity and pre- dicted IFN-γ mirror PD-L1 tumor-cell expression groups. Furthermore, genesets describing immune activation and fibrosis predict known tumor microenvironment archetypes from histology alone. We further validate generalization of our approach across unseen cohorts and demonstrate data-efficient domain adaptation, establishing a slide-native framework for molecular analysis on H&E images. Keywords: cross-modal alignment· molecular prompting 1 Introduction H&E-stained tissue sections are central to diagnostic pathology. Self-supervised foundation models trained on millions of digitized WSIs [3,16,20] enable robust image analysis, such as cancer subtyping, survival prediction, and biomarker dis- covery [20,5,12,14]. However, H&E captures morphology rather than direct tran- scriptomic pathway activity. Bulk RNA-seq provides transcriptome-wide resolu- tion but is expensive and is often absent from archival cohorts [22]. Early works demonstrated that transcriptomic signals and biomarkers, such as microsatellite instability, can be predicted directly from H&E stained sections [13,19]. Recent arXiv:2606.29949v1 [eess.IV] 29 Jun 2026 2Winter et al. ConchThreads RNA-Seq data Frozen RNA-Seq foundation model Multimodal alignment module (method i-v) Aligned embedding space H&E images ConchThreads H&E images Mol ecular prompting inference Multimodal alignment trai ni ng Predict gene set activation (method a-e) Multimodal alignment module (method i-v) RNA features Image features Gene predictor Molecular prompting (d, e) Hallmark score regression (a-c) Fig. 1. Overview. (top) Frozen foundation models and an alignment module (i-v) map H&E and RNA-Seq to a shared latent space. (bottom) During inference, gene sets act as open-vocabulary molecular prompts, allowing H&E embeddings to predict pathway activity by querying an RNA reference database via Soft-kNN or a trained predictor (a-e). frameworks have formalized this into supervised pathway regression. TIGER [8] maps embeddings of H&E stained WSI to breast-cancer-specific gene signatures, while HistoPrism [9] introduces a pan-cancer transformer benchmarked on the 50 MSigDB Hallmark gene sets. OmniScreen uses the Virchow image foundation model to predict gene biomarkers [21]. However, these supervised methods re- quire response labels, map exclusively to fixed predefined output spaces, cannot query novel gene sets at inference time, and yield no shared retrieval space. Foun- dation models for WSIs [3,16,20] and transcriptomics [4] provide powerful data representations. THREADS [20] is trained to generate H&E WSI representations by aligning genomic and transcriptomic profiles in a shared embedding space via its image encoder and gene encoder (scGPT [4] with a custom RNA-head [20]), though it is not primarily designed to ensure exact alignment between image and gene embeddings. This leads us to explore in this work further alignment strategies building on the existing THREADS image and gene models for histopathology and bulk transcriptomics at a multi-cancer scale, introducing open-vocabulary molecular prompting. Further improving the alignment of H&E and RNA-Seq latent spaces beyond the out-of-the-box THREADS models enables the following capabilities: (1) cross-modal retrieval for identifying molecularly similar cases and (2) open- vocabulary molecular prompting—using any gene set from any database as an annotation-free query at inference without modifying the model; which we com- pare to (3) hallmark regression heads that are trained to predict set gene sets directly from H&E as done in previous works [9,8]. Thereby we enable retrospective deployment on archival cohorts lacking prospective RNA-seq. Our framework shifts the paradigm from fixed-target prediction to open-vocabulary prompting for any arbitrary gene set at inference time. Our contributions are: (1) introduction of a strategy for alignment of Molecular Prompting via H&E-RNA Latent Space Alignment3 H&E and RNA-Seq embeddings using existing foundation models, including a systematic benchmark of five H&E-RNA alignment strategies across two RNA encoders; (2) open-vocabulary molecular prompting benchmarked against supervised regression heads; (3) clinical validation on the POSEIDON clinical trial in an inference-only setting; and (4) a data-efficient domain-adaptation across two datasets. 2 Method We describe a lightweight cross-modal alignment framework using frozen foun- dation models and an open-vocabulary molecular prompting strategy for gene set prediction from H&E stained WSIs. Problem Formulation and Encoders: Letv i ∈R d v andr i ∈R d r be the WSI and RNA-Seq embeddings of patient i. Given N paired training sam- ples, we learn a shared latent spaceR D where morphologically and molecularly similar samples are proximate. We use THREADS [20] as the image encoder, aggregating CONCH [16] ViT features via attention MIL into d v =1024-dim embeddings. Two RNA encoders are evaluated: (a) scGPT+THREADS RNA- head [4,20] (d r =1024); (b) BulkFormer [11], purpose-built for bulk RNA-Seq with self-attention over∼20,000 gene tokens and sentinel masking (d r =512). All encoder weights remain frozen; only a lightweight MLP projection head is trained, making the approach tractable even for modest paired cohorts and preserving the pretrained representations of both foundation models. Alignment Strategies: We benchmark five strategies (Table 1) using two gene foundation models: (i) a no-training baseline using the THREADS image and scGPT+THREADS RNA-head model out-of-the-box [20], (i) Supervised train- ing using a cross-entropy loss to predict the RNA-Seq embeddings from the H&E embeddings, (i) Ridge Regression, (iv) Canonical Correlation Analysis (CCA) (d PCA =128, n CCA =50), (v) CLIP-style contrastive learning. Both the Supervised (i) and CLIP (v) methods utilize a two-layer MLP (≈ 1.3M param- eters) with a residual connection, Layer Normalization, and GELU activation to map the frozen embeddingsr i into ℓ 2 -normalized representationsz r i ∈R D within the shared latent space. For the contrastive approach, a label-aware sym- metric InfoNCE loss [18] is used, combining the standard CLIP objective with a supervised contrastive term (weight 0.15) that treats same-cancer-subtype sam- ples as additional positives, with the temperature set to τ=0.07. Training uses the AdamW optimizer (lr 3×10 −4 ) with an effective batch size of 256. Molecular Prompting: Given a reference RNA library ˆ z r j with hallmark scores y h j , predicted pathway activity for query image ˆ z v i is: ˆy h i = X j∈N k (i) w ij y h j , w ij = 1/d ij P l∈N k (i) 1/d il ,(1) 4Winter et al. where d ij = 1− ˆ z v i · ˆ z r j is the cosine distance between the ℓ 2 -normalized query image embedding ˆ z v i and reference RNA embedding ˆ z r j , and N k (i) denotes the k=5 nearest RNA-Seq neighbors. No subtype labels are needed. We also evaluate a Soft-kNN variant that replaces sparse k-N aggregation with soft-attention weights w ij ∝ exp(−d ij /τ) computed over the full reference library (τ=0.1), yielding a smoother, globally-normalised prediction. For calculating the gene set activity scores, we use TPM-normalized counts and log 2 (x+1)-transformed before ssGSEA [2] scoring on the 50 MSigDB Hallmark gene sets [15], yieldingy i ∈R 50 per sample, but can be applied to any gene set collection. These scores serve as a groundtruth for both supervised regression and molecular prompting benchmarks. Per-hallmark regression heads (RidgeCV, Random Forest, MLP, Soft-kNN) are trained on aligned features via 5-fold cross-validation using the BulkFormer-based CLIP method. Domain adap- tation fine-tunes BulkFormer CLIP on fractions ρ∈5%, 10%, 25%, 50%, 100% of paired target-domain samples. For clinical validation in an inference-only setting, we use the Soft-kNN vari- ant over the full training-cohort RNA library, with ssGSEA targets normalised within the training cohort only so that no target-domain samples influence the reference scores. 3 Experiments We first benchmark the alignment strategies and the molecular prompting meth- ods on a multi-cancer dataset. We then validate our method clinically on the POSEIDON trial and finally ablate domain adaptation on two external cohorts. Datasets: Models are trained on a multi-cancer dataset (N=1,720) span- ning Non-small Cell Lung Cancer (NSCLC) (n=245), Triple Negative Breast Cancer (TNBC) (n=112), Renal Cell Carcinoma (RCC) (n=195), Colorectal Cancer (CRC) (n=187), ovarian (n=216), endometrial (n=400), pancreatic (n=42), prostate (n=232), and bladder (n=92) cancer patient samples. All cross- validation folds are partitioned at the patient level. We use the following datasets for evaluating our approach in several experiments: POSEIDON (n=265, NSCLC [10]), TCGA-BRCA (n=1,042, pan-breast), and TCGA-LUAD (n=223, lung adenocarcinoma). Alignment Benchmark: BulkFormer MLP-CLIP achieves the best perfor- mance in R@5=56.0% and R@10=71.9% with competitive scoring in cosine sim- ilarity and Maximum Mean Discrepancy (MMD) (Table 1) in a 5-fold cross- validation experiment. The no-training baseline and supervised training yield near-zero retrieval performance. BulkFormer MLP-CLIP consistently outper- forms scGPT pairings, making it our method of choice for further experiments. Molecular Prompting via H&E-RNA Latent Space Alignment5 Table 1. Alignment quality on multi-cancer dataset (5-fold cross-validation), img→rna direction. Recall@K (%) as mean±std. Bold: best per metric. MethodRNA encoder R@1↑R@5↑ R@10↑ Cosine↑ MMD↓ (i) No Training scGPT0.3±0.2 1.4±0.3 2.8±0.8 −0.013 0.655 (i) Supervised scGPT0.1±0.1 1.2±0.3 2.7±0.7 0.001 0.646 (i) RidgescGPT8.3±6.6 27.7±20.0 38.6±26.4 0.524 0.276 (iv) CCAscGPT13.5±1.0 39.2±3.0 54.5±1.3 0.315 0.048 (v) MLP-CLIP scGPT11.3±1.6 38.5±1.0 55.9±1.1 0.401 0.121 (i) Supervised BulkFormer 0.5±0.3 1.6±0.5 2.9±1.0 −0.003 0.611 (i) RidgeBulkFormer 16.8±1.1 45.3±1.5 62.0±1.1 0.673 0.165 (iv) CCABulkFormer 23.0±1.9 53.0±0.8 67.8±2.3 0.406 0.044 (v) MLP-CLIP BulkFormer 20.8±1.5 56.0±1.2 71.9±1.0 0.501 0.148 Molecular Prompting: Using BulkFormer with MLP-CLIP, we then bench- mark multiple methods for predicting the gene set activation scores, namely direct regression heads (a-c) on the aligned latent space (5-fold CV; σ is cross- hallmark std across folds), (d) kNN (k=5) and (e) Soft-kNN and (Table 2). Al- though RidgeCV outperforms Soft-kNN in mean R 2 , RidgeCV requires a fixed set of labelled hallmark targets at training time and must be retrained for any novel gene set. Soft-kNN, by contrast, operates as a true open-vocabulary pre- dictor: any gene set can be queried at inference by computing ssGSEA scores on the reference RNA library, with no model updates required, leading us to use Soft-kNN as method of choice in further experiments. Biological Ceiling: The per-hallmark R 2 pattern on the multi-cancer dataset (5-fold cross-validation) establishes an empirical biological ceiling: gene sets with a morphological footprint in H&E stained WSIs are predictable, while others are not (Fig. 2), consistent with prior work showing that transcriptomic programs can be predicted from H&E stained WSIs [13,7,17]. Among higher-predictable hallmarks, cell-cycle programs lead the way: G2M Checkpoint ((1) R 2 =0.78) and E2F Targets ((7) R 2 =0.61), correspond to mitotic figures and nuclear pleomorphism. IFN-γ Response ((2) R 2 =0.75), Allograft Rejection ((15) R 2 =0.53) and (17) IL6-JAK-STAT3 Signaling (R 2 =0.50) are Immune-related hallmarks which reflect TIL density. Glycolysis ((6) R 2 =0.63) and Hypoxia ((14) R 2 =0.53) track necrosis and high-grade architecture. Lower-predictability hallmarks, such as Oxidative Phosphorylation ((44) R 2 =0.22), Fatty Acid Metabolism ((48) R 2 =0.20), and Androgen Response ((38) R 2 =0.27) could reflect metabolic pathways with no detectable signal in H&E stained WSI [19,13,6]. Clinical Validation: We validate molecular prompting on POSEIDON [10] (n=265, NSCLC) in an inference-only experiment. The MLP-CLIP model is 6Winter et al. Table 2. Hallmark gene set prediction performance (5-fold cross-validation). Mean R 2 ± std across 50 hallmarks, with counts of hallmarks exceeding R 2 >0.3 and R 2 >0.5. MethodMean R 2 ↑ Hallmarks >0.3 ↑ Hallmarks >0.5 ↑ (a) RidgeCV 0.465±0.135 45/50 18/50 (b) Random Forest 0.441±0.13244/5016/50 (c) MLP0.392±0.14836/5015/50 (d) kNN0.420±0.13042/5016/50 (e) Soft-kNN0.421±0.12044/5015/50 123456789 1011121314151617181920212223242526272829303132333435363738394041424344454647484950 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 R² BulkFormer MLP A R² > 0.5 0.30.5 0.10.3 0.1 #Hallmark#Hallmark 1G2M Checkpoint 2Interferon Gamma Response 3Xenobiotic Metabolism 4Estrogen Response Early 5Myc Targets V2 6Glycolysis 7E2F Targets 8Uv Response Up 9Notch Signaling 10Unfolded Protein Response 11Pi3K Akt Mtor Signaling 12Apical Junction 13Kras Signaling Up 14Hypoxia 15Allograft Rejection 16Pancreas Beta Cells 17Il6 Jak Stat3 Signaling 18Protein Secretion 19P53 Pathway 20Il2 Stat5 Signaling 21Coagulation 22Mitotic Spindle 23Complement 24Angiogenesis 25Estrogen Response Late 26Mtorc1 Signaling 27Myogenesis 28Heme Metabolism 29Kras Signaling Dn 30Apoptosis 31Cholesterol Homeostasis 32Bile Acid Metabolism 33Spermatogenesis 34Uv Response Dn 35Tnfa Signaling Via Nfkb 36Inflammatory Response 37Adipogenesis 38Androgen Response 39Epithelial Mesenchymal Transition 40Tgf Beta Signaling 41Dna Repair 42Reactive Oxygen Species Pathway 43Wnt Beta Catenin Signaling 44Oxidative Phosphorylation 45Peroxisome 46Myc Targets V1 47Interferon Alpha Response 48Fatty Acid Metabolism 49Apical Surface 50Hedgehog Signaling Fig. 2. Molecular prompting: predicted vs. true ssGSEA scores for MSigDB Hallmark gene sets (multi-cancer, 5-fold cross-validation). Sorted by BulkFormer MLP R 2 ; back- ground shading indicates morphological grounding. frozen after being trained on the multi-cancer dataset and the Soft-kNN queries a frozen RNA-Seq library exclusively built from other cohorts (multi-cancer dataset + TCGA-LUAD + TCGA-BRCA). Using the fixed multimodal aligned space, we predict biologically relevant gene set activity. First, we observe that H&E-predicted Inamura SCC scores sep- arate LUAD from LUSC patients (Fig. 3A), recapitulating true subtype identity. Second, we examine IFN-γ gene set activity across PD-L1 expression strata: H&E-predicted IFN-γ scores tend to mirror PD-L1 Tumor Cell (TC) expression groups (Fig. 3B-C), suggesting H&E-inferred immune activation may serve as a surrogate for this predictive checkpoint inhibitor biomarker. The predicted scores yielded a non-significant but directionally consistent OS stratification in the durvalumab arm (HR=0.70, p=0.118) (Fig. 3D-E) Third, to characterize the tumor microenvironment (TME), we assessed molecular programs associated with immune activation and fibrosis, two major determinants of conserved TME archetypes (Fig. 3F) [1]. We define the immune axis as the mean of the IFN-γ and inflammatory response enrichment scores, and the fibrotic axis as the mean of the TGF-β and epithelial-mesenchymal transition enrichment scores. H&E-predicted immune and fibrotic scores broadly recapitulated the median-split separation defined by corresponding RNA-seq-derived scores (Fig. 3G-J). Molecular Prompting via H&E-RNA Latent Space Alignment7 Adenocarcinoma (N=137) SCC (N=128) NSCLC Subtype 1 0 1 2 3 4 Activation Score NSCLC SCC Subtyping (INAMURA_LUNG_CANCER _SCC_UP) >=50% (N=72) 1-49% (N=101) <1% (N=92) PDL1 TC Group 5 4 3 2 1 0 1 2 True ssGSEA Score True IFN ssGSEA Score by PDL1 TC Group(POSEIDON) >=50% (N=72) 1-49% (N=101) <1% (N=92) PDL1 TC Group 2 1 0 1 Predicted ssGSEA Score Predicted IFN ssGSEA Score by PDL1 TC Group(POSEIDON) 010203040 Time (months) 0.0 0.2 0.4 0.6 0.8 1.0 OS by True IFN ssGSEA Score (median split) HR=0.52 (95% CI 0.320.83) p=0.0050 High (N=45) Low (N=45) 010203040 Time (months) 0.0 0.2 0.4 0.6 0.8 1.0 OS Probability OS by Predicted IFN ssGSEA Score (median split) HR=0.70 (95% CI 0.441.10) p=0.1183 High (N=45) Low (N=45) 54321012 Immune Score 3 2 1 0 1 2 Fibrotic Score Immune LowImmune High 5 4 3 2 1 0 1 2 True Immune Score Immune LowImmune High 2.5 2.0 1.5 1.0 0.5 0.0 0.5 1.0 Predicted Immune Score Fibrotic LowFibrotic High 3 2 1 0 1 2 True Fibrotic Score Fibrotic LowFibrotic High 3 2 1 0 1 Predicted Fibrotic Score At risk 45 45 27 19 17 8 14 3 2 1 At risk 45 45 26 20 14 11 11 6 2 1 ABCDE FGHIJ Fig. 3. Clinical validation on POSEIDON: A: H&E-based NSCLC subtyping. B-C: True and predicted IFN-γ ssGSEA vs. PD-L1 TC group (all 265 QC-passed patients). D-E: Kaplan-Meier OS by true/predicted IFN-γ median split (n=90, dur- valumab arm). F: Scatter plot describing true immune (x-axis) and fibrotic (y-axis) scores and classes; G-H: Boxplots showing true and predicted scores split at true me- dian values into low and high groups. The predicted immune score (H) mirrors the true score (G). I-J: The predicted fibrotic score (J) mirrors the true fibrotic score (I). Data-Efficient Domain Adaptation Using the multi-cancer trained align- ment models on unseen cohorts (TCGA-LUAD, and TCGA-BRCA) exposes a transfer gap: R@10 drops to 10.8%, and 4.0%, respectively. Data-efficient domain adaptation bridges this gap with minimal paired target-domain samples. Fig. 4- B shows target-domain fine-tuning with ρ∈5%, 10%, 25%, 50%, 100%. Just 8 paired samples (5% of TCGA-LUAD) raises R@10 from 10.7%→43.5%, confirm- ing the pretrained model encodes a dense structural prior requiring only minimal target calibration on unseen image and RNA-Seq embeddings. Full fine-tuning restores R@10 to 62.8% (TCGA-LUAD), and 38.8% (TCGA-BRCA). Hallmark regression (Fig. 4-C) works best for programs with clear morphological signatures - such as cell cycle, immune activity, and glycolysis - while metabolic pathways remain harder to predict. This likely reflects both their weak visual footprint and the complexity of gene sets that span multiple independent processes, making it difficult to find a single consistent morphological correlate. The TCGA-LUAD cohort recovers retrieval faster than TCGA-BRCA, which may reflect a greater morphological and molecular heterogeneity. Please note that current patient data compliance policy associated to the POSEIDON clinical dataset does not allow experiments beyond the ’inference-only’ setting presented in section 3, currently preventing similar fine-tuning experiments to be conducted on this dataset. 8Winter et al. Fig. 4. Domain adaptation. (A) UMAPs of the multi-cancer dataset, TCGA-BRCA and TCGA-LUAD embeddings with color-coded indications. (B) Recall@K vs. fine- tuning fraction of TCGA-BRCA and TCGA-LUAD datasets. (C) Per-hallmark R 2 per fraction; shading groups hallmarks by morphological grounding. 4 Conclusion Training lightweight projection heads atop frozen foundation models enables open-vocabulary molecular prompting from routine H&E slides and querying gene set at inference without sequencing or end-to-end retraining. Our system- atic benchmark of five alignment strategies establishes BulkFormer MLP-CLIP in combination with Soft-kNN querying as the best-performing method (71.9% R@10, 25× above baseline). Per-hallmark analysis reveals a principled pre- dictability spectrum: morphologically grounded programs such as cell-cycle, immune infiltration, and glycolysis tend to be recoverable (R 2 >0.5), while metabolic pathways lacking a tissue-architectural footprint remain challenging. Application in an inference-only setting on a clinical trial suggests that frozen molecular prompting can enable biomarker-relevant molecular inference from routine H&E images at minimal incremental cost. H&E-predicted signatures re- capitulated established immune and fibrotic archetypes, while the first showed directionally consistent overall survival stratification relative to sequencing-based estimates. Under domain shift, a few paired samples suffice to bridge the transfer gap, confirming a strong structural prior requiring only minimal target calibra- tion. Notable limitations include our modest training cohort size (N=1,720). Future work will explore larger cohorts, additional cancer types, and further clinical trial applications. Acknowledgements: All authors are employees of AstraZeneca and some have AstraZeneca shares. No author has other relevant financial or non-financial in- terests to disclose. We thank the TICA team at AstraZeneca for dataset access. Molecular Prompting via H&E-RNA Latent Space Alignment9 References 1. Bagaev, A., Kotlov, N., Nomie, K., Svekolkin, V., Gafurov, A., Isaeva, O., Osokin, N., Kozlov, I., Frenkel, F., Gancharova, O., et al.: Conserved pan-cancer microenvi- ronment subtypes predict response to immunotherapy. Cancer cell 39(6), 845–865 (2021) 2. Barbie, D.A., Tamayo, P., Boehm, J.S., Kim, S.Y., Moody, S.E., Dunn, I.F., Schinzel, A.C., Sandy, P., Meylan, E., et al.: Systematic RNA interference reveals that oncogenic KRAS-driven cancers require TBK1. Nature 462(7269), 108–112 (2009) 3. Chen, R.J., Ding, T., Lu, M.Y., Williamson, D.F.K., Jaume, G., Song, A.H., Chen, B., Zhang, A., Shao, D., Shaban, M., Williams, M., Oldenburg, L., Weishaupt, L.L., Wang, J.J., Vaidya, A., Le, L.P., Gerber, G., Sahai, S., Williams, W., Mahmood, F.: Towards a general-purpose foundation model for computational pathology. Nature Medicine 30, 850–862 (2024) 4. Cui, H., Wang, C., Maan, H., Pang, K., Luo, F., Duan, N., Wang, B.: scGPT: Toward building a foundation model for single-cell multi-omics using generative AI. Nature Methods 21, 1470–1480 (2024) 5. Ding, T., Wagner, S.J., Song, A.H., Chen, R.J., Lu, M.Y., Zhang, A., Vaidya, A.J., Jaume, G., Shaban, M., Kim, A., et al.: A multimodal whole-slide foundation model for pathology. Nature medicine p. 1–13 (2025) 6. Fu, Y., Jung, A.W., Torne, R.V., Gonzalez, S., Vöhringer, H., Shmatko, A., Yates, L.R., Jimenez-Linan, M., Moore, L., Gerstung, M.: Pan-cancer computational histopathology reveals mutations, tumor composition and prognosis. Nature cancer 1(8), 800–810 (2020) 7. Hoang, D.T., Dinstag, G., Shulman, E.D., Hermida, L.C., BenHur, A., Kisilev, P., Raspe, E., Vanderstichele, A., Lambrechts, D., Linn, S.C., et al.: A deep- learning framework to predict cancer treatment response from histopathology im- ages through imputed transcriptomics. Nature Cancer 5(9), 1305–1317 (2024) 8. Howard, F.M., Dolezal, J., Kochanny, S., Khramtsova, G., Vickery, J., Srisuwananukorn, A., Woodard, A., Chen, N., Nanda, R., Olopade, O.I., Huo, D., Pearson, A.T.: Integration of pathology image and gene expression data using a transformer-based multiple instance learning approach for breast cancer. medRxiv (2025), preprint 9. Hu, S., Zeng, Q., Bhasker, N., Kather, J.N., Speidel, S.: HistoPrism: Unlocking functional pathway analysis from pan-cancer histology via gene expression predic- tion. arXiv preprint arXiv:2601.21560 (2026), published at ICLR 2026 10. Johnson, M.L., Cho, B., Luft, A., Alatorre-Alexander, J., Geater, S.L., Laktionov, K., Kim, S.W., Ahn, M.J., Carcereny, E., Audigier-Valette, C., et al.: Durvalumab with or without tremelimumab in combination with chemotherapy as first-line therapy for metastatic non-small-cell lung cancer: the phase I POSEIDON study. Journal of Clinical Oncology 41(6), 1213–1227 (2023) 11. Kang, B., Fan, R., Yi, M., Cui, C., Cui, Q.: A large-scale foundation model for bulk transcriptomes. BioRxiv p. 2025–06 (2025) 12. Kather, J.N., Charoentong, P., Krisam, J., Renna, T., Hoffmeister, F., Chang- Claude, J., Hoffmeister, M., Brenner, H., Jäger, D., Halama, N.: Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer. Nature Medicine 25, 1054–1056 (2019) 13. Kather, J.N., Heij, L.R., Grabsch, H.I., Loeffler, C., Echle, A., Muti, H.S., Krause, J., Niehues, J.M., Sommer, K.A., Bankhead, P., et al.: Pan-cancer image-based 10Winter et al. detection of clinically actionable genetic alterations. Nature cancer 1(8), 789–799 (2020) 14. Kather, J.N., Krisam, J., Charoentong, P., Luedde, T., Herpel, E., Weis, C.A., Gaiser, T., Marx, A., Valous, N.A., Ferber, D., et al.: Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLoS medicine 16(1), e1002730 (2019) 15. Liberzon, A., Birger, C., Thorvaldsdottir, H., Ghandi, M., Mesirov, J.P., Tamayo, P.: The molecular signatures database (MSigDB) hallmark gene set collection. Cell Systems 1(6), 417–425 (2015). https://doi.org/10.1016/j.cels.2015.12.004 16. Lu, M.Y., Chen, B., Williamson, D.F.K., Chen, R.J., Liang, I., Ding, T., Jaume, G., Odintsov, I., Le, L.P., Gerber, G., Parwani, A.V., Zhang, A., Mahmood, F.: A multimodal generative AI copilot for human pathology. Nature 629, 818–826 (2024). https://doi.org/10.1038/s41586-024-07618-3 17. Pizurica, M., Zheng, Y., Carrillo-Perez, F., Noor, H., Yao, W., Wohlfart, C., Vladimirova, A., Marchal, K., Gevaert, O.: Digital profiling of gene expression from histology images with linearized attention. Nature Communications 15(1), 9858 (2024) 18. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: Int. Conf. Mach. Learn. p. 8748–8763 (2021) 19. Schmauch, B., Romagnoni, A., Pronier, E., Saillard, C., Maille, P., Calderaro, J., Kamoun, A., Sefta, M., Toldo, S., Zaslavskiy, M., Clozel, T., Moarii, M., Courtiol, P., Wainrib, G.: A deep learning model to predict RNA-Seq expression of tumours from whole slide images. Nature Communications 11, 3877 (2020) 20. Vaidya, A., Zhang, A., Jaume, G., Song, A.H., Ding, T., Wagner, S.J., Lu, M.Y., Doucet, P., Robertson, H., Almagro-Perez, C., et al.: Molecular-driven foundation model for oncologic pathology. arXiv preprint arXiv:2501.16652 (2025) 21. Wang, Y.K., Tydlitatova, L., Kunz, J.D., Oakley, G., Chow, B.K.B., Godrich, R.A., Lee, M.C., Aghdam, H., Bozkurt, A., Zelechowski, M., et al.: Screen them all: high-throughput pan-cancer genetic and phenotypic biomarker screening from h&e whole slide images. arXiv preprint arXiv:2408.09554 (2024) 22. Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R.M., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M., The Cancer Genome Atlas Research Network: The cancer genome atlas pan-cancer analysis project. Nature Genetics 45, 1113–1120 (2013)