Paper deep dive
M$^3$-Gen: Interpretable Multimodal Generation of Gene Expression Profiles Using Clinical and Imaging Data
Francesca Pia Panaccione, Carlo Sgaravatti, Marco Venere
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/24/2026, 3:04:54 AM
Summary
The paper introduces M3-Gen, a multimodal framework that generates synthetic gene expression profiles by conditioning a Wasserstein GAN with Gradient Penalty (WGAN-GP) on histopathology images and clinical metadata. Using contrastive learning to align visual and textual embeddings, and an attention mechanism to fuse them, M3-Gen produces biologically coherent and interpretable transcriptomic data, validated on the TCGA dataset.
Entities (9)
Relation Signals (8)
M3-Gen → conditionson → Histopathology Images
confidence 95% · conditioning a Generative Adversarial Network on histopathology images and clinical metadata.
M3-Gen → conditionson → Clinical Metadata
confidence 95% · conditioning a Generative Adversarial Network on histopathology images and clinical metadata.
M3-Gen → evaluatedon → TCGA
confidence 95% · Evaluations on the TCGA dataset demonstrate that M3-Gen generates realistic and functionally meaningful gene expression data.
M3-Gen → produces → gene expression profiles
confidence 95% · M3-Gen learns a unified latent representation ... to guide a generative model in producing biologically coherent gene expression profiles.
M3-Gen → uses → WGAN-GP
confidence 95% · M3-Gen consists of ... a generative model, where a Wasserstein GANs with Gradient Penalty (WGAN-GP) ... is conditioned on these multimodal embeddings
M3-Gen → uses → UNI
confidence 90% · We use UNI [3] as an Image Encoder
M3-Gen → uses → Clinical ModernBERT
confidence 90% · As a Text Encoder, we exploit Clinical ModernBERT
M3-Gen → uses → Llama3-8B
confidence 85% · We used a version of Llama3-8B fine-tuned on medical data ... to convert structured metadata into concise case summaries
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Integrating heterogeneous biomedical data, including clinical metadata, histopathology images, and molecular profiles, is crucial for comprehensive disease understanding. However, gene expression data acquisition remains constrained by high costs and privacy concerns, limiting its use in multimodal research and AI-driven applications. We present MultiModal Molecular Generation (M$^3$-Gen), a novel framework for the generation of gene expression profiles by conditioning a Generative Adversarial Network on histopathology images and clinical metadata. M$^3$-Gen learns a unified latent representation from the clinical variables and the images, leveraging contrastive learning, and exploits the embeddings of the two modalities to guide a generative model in producing biologically coherent gene expression profiles. Evaluations on the TCGA dataset demonstrate that M$^3$-Gen generates realistic and functionally meaningful gene expression data. Importantly, by integrating multiple modalities in an attention-based mechanism, M$^3$-Gen provides intrinsic explainability: it allows the identification of which regions of the histopathology images most strongly influenced the generation of specific gene expression profiles, making the model's decisions interpretable by design.
Tags
Links
- Source: https://arxiv.org/abs/2607.21343v1
- Canonical: https://arxiv.org/abs/2607.21343v1
Trouble viewing inline? Open PDF directly →
Full Text
38,949 characters extracted from source content.
Expand or collapse full text
M 3 -Gen: Interpretable Multimodal Generation of Gene Expression Profiles Using Clinical and Imaging Data Francesca Pia Panaccione 1[0009−0005−8007−963X] , Carlo Sgaravatti 1[0009−0001−4962−5365] , and Marco Venere 1[0009−0002−8991−1443] DEIB - Dipartimento Elettronica, Informazione e Bioingegneria, Politecnico di Milano, Milan, 20133, Italy, francescapia.panaccione,carlo.sgaravatti,marco.venere@polimi.it Abstract. Integrating heterogeneous biomedical data—including clini- cal metadata, histopathology images, and molecular profiles—is crucial for comprehensive disease understanding. However, gene expression data acquisition remains constrained by high costs and privacy concerns, limit- ing its use in multimodal research and AI-driven applications. We present MultiModal Molecular Generation (M 3 -Gen), a novel framework for the generation of gene expression profiles by conditioning a Generative Adver- sarial Network on histopathology images and clinical metadata. M 3 -Gen learns a unified latent representation from the clinical variables and the images, leveraging contrastive learning, and exploits the embeddings of the two modalities to guide a generative model in producing biologically coherent gene expression profiles. Evaluations on the TCGA dataset demonstrate that M 3 -Gen generates realistic and functionally meaningful gene expression data. Importantly, by integrating multiple modalities in an attention-based mechanism, M 3 -Gen provides intrinsic explainability: it allows the identification of which regions of the histopathology images most strongly influenced the generation of specific gene expression pro- files, making the model’s decisions interpretable by design. Code will be available at: https://github.com/CarloSgaravatti/M3-Gen. Keywords: Generative AI· Multimodal Learning· Synthetic Gene Expression· Explainability· LLM· Deep Learning· Computer Vision. 1 Introduction Biomedical research increasingly relies on integrating heterogeneous data types— molecular profiles, clinical metadata, and medical images—to build richer mod- els of biological systems and disease mechanisms. In practice, however, these modalities are unevenly available: while clinical records and histopathology slides are routinely collected and standardized, gene expression profiles remain costly, privacy-sensitive, and limited in scale [9]. This imbalance not only constrains the scope of multimodal studies but also prevents researchers from fully exploiting the rich contextual information contained in clinical and morphological data. arXiv:2607.21343v1 [cs.LG] 23 Jul 2026 2F.P. Panaccione et al. Generative AI—from Generative Adversarial Networks (GANs) to denoising diffusion models—presents a promising solution. By learning complex, high- dimensional data distributions, these methods can produce realistic synthetic transcriptomic profiles that augment scarce cohorts and safeguard patient privacy [2]. Despite early successes in generating gene-expression data (e.g., Panaccione et al. [11], Vinas et al. [15], Lacan et al. [6]), most approaches rely on limited inputs and overlook critical clinical and histological context. To overcome these limitations, we introduce M 3 -Gen (MultiModal Molecular Generation), a unified framework that learns a shared latent representation of clinical variables and histopathology images to produce biologically coherent gene expression profiles in silico. To the best of our knowledge, we are the first to ad- dress the problem of conditioning a generative model of gene expressions on both images and text. In contrast, existing methods such as [14,17] focus on directly predicting gene expression from pathology images, without leveraging textual clinical information or generative modeling. This distinction is important: while prediction is inherently a deterministic task, generative modeling provides a mech- anism to explore previously unseen combinations of clinical and histopathological inputs, enabling the generation of multiple plausible omics profiles corresponding to the same conditions. Furthermore, as demonstrated in this work, M 3 -Gen inherently supports explainability. By integrating multimodal inputs, the model captures the most relevant visual patterns in the histopathology images in relation to the clinical context when generating a specific gene expression profile. This allows tracing which aspects of the input data contributed most to the output, and observing the correlations and interconnections across the three data modal- ities (clinical variables, imaging, and gene expression), providing insights into the underlying biological mechanisms. In our parallel line of work, we explored a more complex fusion architecture aimed at increasing representational capacity [12]. In contrast, the present manuscript deliberately focuses on interpretability through an attention-based fusion mechanism, enabling patch-level analysis of histopathology images in the gene expression generation process. We evaluate M 3 -Gen using standard metrics from the generative modeling literature—such as distributional alignment and downstream predictive performance—to demon- strate its ability to produce realistic, functionally meaningful gene expression profiles. 2 Data and Methods At a high level, M 3 -Gen consists of three main stages: (i) a preprocessing phase, where we extract and filter image patches from tissue slides and summarize clinical metadata into compact textual descriptions; (i) a contrastive pretraining step, where we align visual and textual representations in a shared embedding space; and (i) a generative model, where a Wasserstein GANs with Gradient Penalty (WGAN-GP) [4] is conditioned on these multimodal embeddings to synthesize gene expression profiles. As illustrated in Figure 1b, an Image En- coder embeds patches of tissue slides, whose mean representation is aligned, via M 3 -Gen: Multimodal Generation of Gene Expression Profiles3 contrastive learning, with the embedding produced by a Text Encoder from the clinical description. These embeddings are then integrated through an attention mechanism to produce a single multimodal embedding. Finally, this multimodal embedding is used to condition the WGAN-GP, by concatenating it to the input noise of the generator and to the real or generated gene expression in input to the discriminator. “info” : “gender”: ..., “age”: ..., , “disease_type”: ..., “diagnosis”: “disease”: ..., “stage”: ..., Bio-Medical Llama “Generate a gene expression for a patient with a diagnosis of ...” Otsu Thresholding (a) Preprocessing Image Encoder Text Encoder “Generate a gene expression for a patient with a diagnosis of ...” Mean e img Tile Embeddings E img Text Embedding e text Contrastive Loss (b) Contrastive Pretraining ê img G ê img D . . . . . . Concat Tile Embeddings E img Text Embedding e text Attention ⅅ Attention 퐺 Multimodal Embeddings Gaussian Noise z Keys & Values Query . . . Generator Generated Samples Real Samples Discriminator Score Concat . . . . . . . . . Concat (c) Conditional generation with WGAN-GP Fig. 1: The overall pipeline of M 3 -Gen. In the preprocessing step (a), we extract patches from tissue slides using the Otsu thresholding algorithm to select the patches with a percentage of background pixels higher than 80%, and we extract textual descriptions from patients’ metadata with an LLM. We then pre-train an image encoder and a text encoder to align their embeddings with contrastive learning (b). Finally, an attention mechanism is used to combine the embeddings of the patches and of the text to condition a WGAN-GP (c). 4F.P. Panaccione et al. 2.1 Data The input data for our method consist of paired histopathology whole-slide images (WSIs), clinical metadata, and matched gene expression profiles, all retrieved from TCGA public repository 1 and focused on twelve different tumor types. WSIs are ultra-high-resolution images of tissue sections, which are computationally infeasible to analyze in full, and are therefore typically subdivided into smaller tiles for downstream processing. Clinical metadata includes patient-specific information such as demographics, cancer subtype, and treatment history, all related to the disease condition. Gene expression profiles, obtained from RNA sequencing, are quantified using FPKM (Fragments Per Kilobase of transcript per Million mapped reads), providing normalized measures of gene activity specific to the disease context. 2.2 Preprocessing To prepare data for multimodal generation, we segment tissue regions from pathology slide thumbnails using Otsu Thresholding [10] and extract multi-scale high-resolution tiles (128–1024 px), keeping only those tiles with more than 20% of tissue content. For clinical context, we convert structured metadata into concise case summaries using a quantized instruction-tuned language model. We used a version of Llama3-8B fine-tuned on medical data 2 [1]. Irrelevant fields are removed, and the remaining data are serialized into prompts, resulting in 200-word descriptions that capture disease site, demographics, and experimental conditions. 2.3 Constrastive Pretraining To enable effective multimodal conditioning without the need for end-to-end training of the entire model, we first align visual and textual representations in a shared embedding space using contrastive pretraining. Contrastive learning is a technique that aims to bring the embeddings of semantically similar data points from different modalities (e.g., an image and its corresponding text) closer together in a sharedd-dimensional latent space, while pushing unrelated pairs further apart. Since image patches and clinical descriptions originate from different modalities but are correlated to the same patient, we learn modality-specific encoders with semantically aligned representations. We fed as input to the Image Encoder a random subset ofNpatches extracted from the tissue slides, producing as output an embedding matrixE img ∈R N×d . The Text Encoder, instead, embeds the clinical textual description intoe text ∈R d . We then compute the meane img ∈R d of the patch embeddings and aligne text ande img using the Information Noise-Contrastive Estimation (InfoNCE) loss 1 https://w.cancer.gov/ccg/research/genome-sequencing/tcga 2 https://huggingface.co/ContactDoctor/Bio-Medical-Llama-3-8B M 3 -Gen: Multimodal Generation of Gene Expression Profiles5 function, as defined in CLIP [13]: L = 1 2 CE sim(e img , e text ) τ + CE sim(e text , e img ) τ ,(1) wheresim(a,b) =a ⊤ bdenotes cosine similarity afterℓ 2 normalization, andτis a temperature hyperparameter. The cross-entropy loss encourages each image to match its paired text, and vice versa. 2.4 Generative Model To synthesize gene expression profiles conditioned on both visual and textual pa- tient information, we employ a Conditional WGAN-GP architecture, currently a standard for transcriptomic data synthesis due to its training stability and capacity to model complex biological distributions. The generator takes as input a latent noise vector z∼ N(0,I) concatenated with a multimodal embedding derived from the contrastive pretraining step. Specifically, we reuse patch-level embeddingsE img ∈R N×d and clinical text embeddingse text ∈R d . WSIs might contain patches that refer to the diagnosed disease and other patches that are not relevant. Thus, to capture the most relevant visual patterns in relation to the clinical context, we apply a multi-head attention mechanism, usinge text as the query andE img as keys and values. This produces an attention-weighted image embeddingˆe G img ∈R d , which is concatenated with the random latent vector to form the input to the generator. For the discriminator, we apply the same attention mechanism, conditioning onˆe D img . The discriminator takes as input both real and generated gene expression profiles and is trained to distinguish whether a sample is real or generated. This makes our method inherently interpretable, as we can compute attention maps that can be directly inspected to quantify the contribution of each image patch in conditioning the generated transcriptomic profiles. This provides a biologically meaningful explanation of which tissue regions drive the synthesis process. Attention maps are directly connected to the attention weights given as output by the multi-head attention layer of the generator. Given a multi-head attention layer withHheads, the weightsα (h) ∈R N for headhare computed as: α (h) = softmax (W Q h e text ) ⊤ (W K h E img ) √ d h ! ,(2) whereW Q h ,W K h ∈R d h ×d are learnable projection matrices that map, respectively, the textual querye text and the visual keysE img into a common subspace of dimensiond h . To obtain a single interpretable relevance score per patch, we compute the mean of the attention weights across all heads: α = 1 H H X h=1 α (h) ,(3) 6F.P. Panaccione et al. resulting inα∈R N , where each elementα i represents the relative importance of thei-th patch embedding in conditioning the generator. These weights can be visualized as an attention map over the WSI patches, highlighting the most influential regions for the synthesized gene expression profile. 3 Experiments We test our method on the TCGA dataset, comparing our results with standard generative models. The dataset provides both tissue slides and clinical descriptions of each patient, associated with a gene expression profile derived from RNA-seq. For storage and computational purposes, we select only a subset of TCGA by keeping only the samples having a tissue slide of dimension less than 70 MB, for a total of 1224 clinical cases. 3.1 Experimental Setup We performed our experiments on a machine with the AMD Ryzen 1950X CPU, with 128 GB of RAM, using two NVIDIA A6000 GPUs with 48 GB of VRAM. We employed PyTorch 3 version 2.6.0 to define our model and perform the experimental evaluation, and CUDA version 12.4 for model training on the GPU devices. We use UNI [3] as an Image Encoder, which offers a pretrained Vision Transformer for histopathology tiles. As a Text Encoder, we exploit Clinical ModernBERT [8] 4 5 . We add a linear projection layer to the embeddings of these two models to have the same embedding size for both modalities. In our experiments, we use an embedding size of 128. 3.2 Evaluation Metrics We evaluate our approach with three classes of metrics: (i) unsupervised metrics (Precision, Recall [5] and Correlation [7]), (i) detectability, which consist of training a classifier to detect whether samples are real or generated (lower Accuracy and F1-score means high similarity between real and generated data), and (i) utility, which assesses the performance (Accuracy and F1-score) of models trained on synthetic data and tested on real data to classify the disease type of a patient from RNA-seq. 3.3 Contrastive Pretraining Results Our feature extractors, UNI and Clinical ModernBERT, are pretrained, respec- tively, on WSI data and medical descriptions. Thus, to align their representations with contrastive learning, we freeze the pretrained layers and train only the linear 3 https://pytorch.org/ 4 https://huggingface.co/Simonlee711/Clinical_ModernBERT 5 Both UNI and Clinical ModernBERT are accessible through Huggingface M 3 -Gen: Multimodal Generation of Gene Expression Profiles7 projection. At each training step we select 16 random patches to be fed as input to the Image Encoder. We train the model for 20 epochs using a batch size of 8 and the Adam optimizer, with a learning rate of 0.0001. Figure 2 shows the effectiveness of our contrastive training approach, using t-SNE as a dimensionality reduction tool for visualization, where we analyze the separability of different disease types in our embedding space. Specifically, it shows that text embeddings can separate well the different diseases, while image embeddings, computed as the mean of all the patches, are noisier. Differently, by selecting the 32 most similar patches to the text embedding and computing the mean of these embeddings we can still separate well the embedding space using images. This insight gives an additional motivation to use attention in our conditional model. 402002040 40 20 0 20 40 Image embeddings 402002040 Text embeddings 402002040 32 patches most similar to text Squamous Cell Neoplasms Adenomas and Adenocarcinomas Cystic, Mucinous and Serous Neoplasms Acinar Cell Neoplasms Ductal and Lobular Neoplasms Gliomas Paragangliomas and Glomus Tumors Transitional Cell Papillomas and Carcinomas Nevi and Melanomas Mature B-Cell Lymphomas Germ Cell Neoplasms Mesothelial Neoplasms Fig. 2: TSNE visualizations of the embeddings after pre-training with contrastive learning. Different colors indicate different disease types. Furthermore, in Figure 6, we provide a more detailed visualization of the image embeddings after contrastive pretraining, including representative patches for each class, which highlights how the contrastive learning objective structures the latent space according to morphological similarity, grouping patches with comparable tissue organization or cellular composition while maintaining clear separation between different disease classes. 3.4 Conditional WGAN-GP Results We train our Conditional WGAN-GP for 500 epochs using a latent dimension of 256 and the Adam optimizer with a starting learning rate of 0.0005, which decreases by a factor of 0.5 every 50 epochs. Table 1 presents the results for the unsupervised and detectability metrics, comparing a vanilla WGAN-GP model without conditioning against our pro- posed strategy, which employs a multi-head attention mechanism to integrate multimodal information. To further assess the effectiveness of this approach, we conduct an ablation study considering three variants: single-modality conditioning 8F.P. Panaccione et al. Fig. 3: t-SNE visualization of image embeddings obtained after contrastive pre- training. The figure not only shows the distribution of embedding points in the latent space but also includes representative image patches for each class. This visualization highlights how the contrastive learning approach effectively separates images according to both class-specific and morphological features, producing well-separated clusters that capture meaningful relationships within the data. (image-only or text-only) and a simple multimodal configuration that uses the mean of the text and image embeddings obtained after contrastive learning. From the unsupervised metrics, the benefits of multi-head attention are particularly evident in the detectability results, where the logistic regression clas- sifier reaches approximately 71% accuracy and 76% F1 score. This indicates that attention-based conditioning produces samples that are more realistic and less eas- ily distinguishable from real data. The model conditioned on the mean embedding, while simpler, achieves the best balance between precision and recall—maintaining a recall of around 86% without compromising precision—suggesting that it effec- tively captures a broader portion of the data distribution. However, as shown in the subsequent tables, our attention-based conditioning strategy demonstrates its full potential in real-world utility evaluations. The relatively lower performance on unsupervised metrics may reflect the model’s ability to generate realistic yet novel samples that do not merely replicate the training data but instead provide meaningful variations, which is particularly valuable from both a privacy and ethical standpoint. M 3 -Gen: Multimodal Generation of Gene Expression Profiles9 Table 1: Results of unsupervised and detectability metrics (Accuracy and F1- Score), computed as the mean of 5 runs. We compare M 3 -Gen with a WGAN-GP conditioned on the disease type. We denote with Image Only and Text Only the single-modal models trained by concatenating to the noise only the image or text embedding, while in Mean we compute the mean of the two. Best results are in bold. Method Precision ↑ Recall ↑ Correlation ↑ Accuracy (MLP)↓ F1-Score (MLP)↓ Accuracy (LR)↓ F1-Score (LR)↓ WGAN-GP0.8900.7780.8710.9270.9260.8510.869 M 3 -Gen(Img Only)0.8200.8340.8870.8620.8510.7580.804 M 3 -Gen(Text Only)0.8850.7810.8860.8140.8040.7620.804 M 3 -Gen(Mean)0.8220.8590.8800.8400.8290.7760.816 M 3 -Gen0.7630.8200.8800.8690.8580.7130.767 Table 2: Utility evaluation with Random Forest (RF) and Multi Layer Perceptron (MLP), classifying the disease type given the gene expression. TRTR means training only on the real data, and TSTR involves training on the generated data. T(S+R)TR involves using both data sources to train. The Augmentation Factor (Aug. Factor) represents how many samples are generated for each real sample. Training DataAug. Factor Accuracy (MLP)↑ F1-Score (MLP)↑ Accuracy (RF)↑ F1-Score (RF)↑ TRTR-0.8800.8700.9020.882 TSTR10.8830.8710.9010.880 TSTR100.8970.8830.9080.893 T(S+R)TR10.8890.8780.9080.892 T(S+R)TR100.8940.8830.9090.894 We further evaluate the practical utility of our model by training Random Forest (RF) and Multi-Layer Perceptron (MLP) classifiers to predict disease type from gene expression profiles. We adopt three training strategies: TRTR (training only on real data), TSTR (training only on generated data), and T(S+R)TR (training on both real and synthetic data). Results in Table 2 show that models trained on generated data alone achieve comparable performance to those trained on real data, and that augmenting the dataset with ten synthetic samples per real instance further improves predictive accuracy. The combined T(S+R)TR setup consistently attains the highest scores across classifiers compared toTRTR, suggesting that the inclusion of synthetic data can provide modest but consistent benefits for downstream prediction, even when real-world labelled data is available. Finally, within the TSTR setting (Table 3), we assess the influence of the different conditioning strategies, as in the unsupervised evaluation. The attention- based model consistently achieves the highest accuracy and F1 scores across classifiers, outperforming both the unconditioned WGAN-GP and the single- modality variants. In contrast, the simple mean-based fusion performs worse than using either modality alone, confirming that effective multimodal integration requires a mechanism that can selectively weight complementary information 10F.P. Panaccione et al. Table 3: Comparison of the utility (TSTR) of M 3 -Gen with the same benchmarks of Table 1. Method Accuracy (MLP)↑ F1-Score (MLP)↑ Accuracy (RF)↑ F1-Score (RF)↑ WGAN-GP0.8530.8470.8910.867 M 3 -Gen(Img Only)0.8630.8510.8940.872 M 3 -Gen(Text Only)0.8740.8630.9010.879 M 3 -Gen(Mean)0.8700.8610.8990.880 M 3 -Gen0.8830.8710.9010.880 Table 4: Agreement between differentially expressed (DE) genes identified from real and generated data for the contrast Squamous Cell Neoplasms vs Adenomas and Adenocarcinomas. RegulationTop-N genesOverlapJaccard similarity Up-regulated100890.80 Down-regulated100860.75 rather than collapsing it into an averaged representation. Overall, these findings highlight the importance of attention-driven fusion in leveraging both modalities to improve the quality, realism, and downstream utility of the generated data. 3.5 Gene- and Pathway-Level Biological Coherence To further support the biological coherence of the generated transcriptomic profiles, we performed an additional gene- and pathway-level comparison between real and synthetic data on a representative disease-type contrast (Squamous Cell Neoplasms vs Adenomas and Adenocarcinomas) using the held-out test split (containing 83 and 102 samples for the two diseases). Differential expression consistency. For each gene, we compared the two dis- ease groups using a two-sided non-parametric Mann–Whitney U test on log- transformed expression values (log 2 (FPKM+1)), with multiple-testing correction performed using the Benjamini–Hochberg procedure. We selected the top-100 most deregulated genes separately for up- and down-regulation and repeated the same analysis on synthetic profiles generated by M 3 -Gen for the same disease con- trast. A quantitative summary of the agreement between real and generated data is reported in Table 4, showing a strong overlap for both up-regulated (89/100, Jaccard = 0.80) and down-regulated genes (86/100, Jaccard = 0.75). These results indicate that disease-associated transcriptional differences are largely preserved in the generated profiles. Pathway-level agreement. To assess whether this gene-level consistency translates into coherent biological processes, we performed enrichment analysis on the derived gene sets using a standard pathway gene-set library. The comparison of the M 3 -Gen: Multimodal Generation of Gene Expression Profiles11 0.00.51.01.52.0 -log10(adj p) p53 signaling pathway Amphetamine addiction Long-term potentiation Signaling pathways regulating pluripo... Melanogenesis Phototransduction Renin secretion Amoebiasis Staphylococcus aureus infection Estrogen signaling pathway Real upregulated 0123 -log10(adj p) p53 signaling pathway Amphetamine addiction Glycolysis / Gluconeogenesis Long-term potentiation Amoebiasis Tyrosine metabolism Phototransduction Renin secretion Staphylococcus aureus infection Estrogen signaling pathway Generated upregulated 0123456 -log10(adj p) Gastric cancer PPAR signaling pathway Retinol metabolism Glycolysis / Gluconeogenesis Pancreatic secretion Fat digestion and absorption Bile secretion Metabolism of xenobiotics by cytochro... Proximal tubule bicarbonate reclamation Maturity onset diabetes of the young Real downregulated 0123456 -log10(adj p) Tyrosine metabolism Starch and sucrose metabolism Fructose and mannose metabolism Bile secretion Butanoate metabolism PPAR signaling pathway Proximal tubule bicarbonate reclamation Glycolysis / Gluconeogenesis Fat digestion and absorption Maturity onset diabetes of the young Generated downregulated Top pathways (Enrichr) Squamous Cell Neoplasms vs Adenomas and Adenocarcinomas Fig. 4: Top enriched pathways for the contrast Squamous Cell Neoplasms vs Adenomas and Adenocarcinomas, computed from the top-100 up-regulated (top row) and down-regulated (bottom row) genes. Results are shown for real data (left) and synthetic profiles generated by M 3 -Gen (right). Bar lengths represent pathway significance as− log 10 adjusted p-values, highlighting a strong agreement between real and generated data. top enriched pathways obtained from real and generated data reveals a substantial overlap among the top-10 pathways (8/10, Jaccard = 0.67). Representative results for both up- and down-regulated genes are reported in Figure 4, highlighting that synthetic data capture pathway-level trends consistent with those observed in real samples. 3.6 Explainability Analysis To demonstrate the intrinsic interpretability of our multimodal architecture, we explore the generator’s inner workings through its multi-head attention mecha- nism. As visualized in Figure 5, the attention weights provide a direct visualization of how the model integrates histopathological context when synthesizing gene ex- pression profiles. This analysis provides insight into the model’s decision-making, highlighting the regions of the WSI that have contributed most to the generation process. We examined the attention weights for ten randomly selected tissue slides to assess how the model prioritizes different patches. The resulting maps highlight distinct hotspots (shown in warm colors), indicating the regions that the generator relies on most heavily. Importantly, this confirms that the attention 12F.P. Panaccione et al. Fig. 5: Attention maps generated by the multi-head attention mechanism for ten representative histopathology slides. Warmer colors (yellow–red) highlight tissue regions assigned higher attention weights, indicating greater influence on the synthesis of the corresponding gene expression profiles. These visualizations illustrate the interpretability of M 3 -Gen, revealing how the model selectively focuses on biologically relevant structures within each slide when generating molecular data. Table 5: Robustness evaluation of M 3 -Gen under common histopathology image artifacts: out-of-focus (OOF), stain variation (SV) and tissue folding (TF). Results are the mean of 5 independent runs. ArtifactUnsupervisedDetectabilityUtility OOFSVTF Precision ↑ Recall ↑ Correlation ↑ Accuracy (LR)↓ F1-Score (LR)↓ Accuracy (RF)↓ F1-Score (RF)↓ 0.7630.8200.8800.7130.7670.9010.880 ✓0.8060.7780.8750.7570.8030.9000.878 ✓0.8120.7550.8640.7420.7880.8990.876 ✓0.8120.7490.8730.7290.7730.9000.878 ✓0.8410.7020.8510.7070.7510.8990.876 mechanism can selectively identify biologically relevant tissue structures that drive gene expression. By establishing a direct, traceable link between microscopic image features and molecular outputs, this approach enhances the interpretability and transparency of our generative framework. 3.7 Robustness to Common Histopathology Artifacts In real-world clinical settings, histopathology whole-slide images are often affected by acquisition and preparation artifacts, such as out-of-focus regions, staining variability, or tissue folding. These artifacts may introduce noise or distort morphological patterns, potentially impacting the reliability of downstream models. To assess the robustness of M 3 -Gen under such non-ideal conditions, M 3 -Gen: Multimodal Generation of Gene Expression Profiles13 Fig. 6: Examples of simulated histopathology image artifacts used in the robust- ness analysis. From left to right: no artifacts (clean image), out-of-focus blur, stain variation, tissue folding, and combined artifacts (out-of-focus, stain variation, and tissue folding). Artifacts are synthetically applied at the patch level to mimic common real-world degradations observed in clinical whole-slide images. we conducted a robustness analysis by evaluating the model in the presence of common histopathological artifacts. Specifically, we considered three widely observed artifact types: out-of-focus blur, staining variations, and tissue folding. Each artifact was synthetically introduced at the image level prior to feature extraction, while keeping the clinical metadata and gene expression profiles unchanged. For stain variation and tissue folding, we followed the same procedures of [16] and applied them to each patch of the WSI independently. To simulate, instead, out-of-focus regions, we apply a Gaussian blur to random regions of the patches.??reports some visualizations of the simulated artifacts. We designed five experimental settings: (i) no artifacts (i.e. clean images), (i) out-of-focus only, (i) stain variation only, (iv) tissue folding only, and (v) a combined scenario where all three artifacts are present simultaneously. For each setting, we evaluated M 3 -Gen following exactly the same protocol adopted in Tables 1 and 2, evaluating the model with unsupervised metrics, detectability and utility. All metrics were computed as the mean over five independent runs, ensuring direct comparability with the main experimental results. Table 5 reports the results and indicates that M 3 -Gen is highly robust to moderate image degradations. Across all artifact configurations, performance remains largely stable, with only marginal variations compared to the artifact- free baseline. A closer inspection of Table 5 highlights a few consistent trends. 14F.P. Panaccione et al. Precision and recall exhibit a trade-off across different artifact configurations, with improvements in one metric typically accompanied by slight decreases in the other, indicating minor shifts in distribution coverage rather than a loss of overall fidelity. Among individual artifacts, tissue folding has the strongest impact, although the degradation remains limited. As expected, the configuration combining all three artifacts yields the lowest performance. Nevertheless, even in this most challenging setting, results remain superior to those obtained with the unconditioned WGAN-GP reported in Tables 1 and 2. This suggests that histopathology images remain informative even when moderately corrupted, and that the multimodal design of M 3 -Gen, leveraging both visual and clinical textual information, provides additional robustness under realistic imaging conditions. These findings support the applicability of M 3 -Gen in realistic clinical scenarios, where image imperfections are unavoidable, and further strengthen its potential for deployment in practical biomedical research pipelines. 4 Conclusions This work introduced a multimodal generative framework that conditions gene expression synthesis on both clinical metadata and histopathology images. Our aim was to generate gene expression profiles that are not only statistically realistic but also clinically coherent for real-world applications. Our results demonstrate that the model performs strongly across standard generative evaluation metrics, producing synthetic gene expression profiles that closely match real data distri- butions. Compared to existing generative approaches for gene expression profiles, our model not only improves realism and utility in downstream predictive tasks but also introduces an interpretable conditioning mechanism providing clear insights into which regions of the images are most influential in producing specific synthesized profiles. This interpretability is particularly valuable, as it opens the door for future studies to generate pathological gene expression profiles and investigate the morphological features most relevant to their manifestation. From a clinical and translational standpoint, M 3 -Gen is not intended to replace molecular profiling, but rather to support research and development work- flows in which gene expression data are scarce, costly, or constrained by privacy considerations. By enabling the generation of biologically coherent transcriptomic profiles conditioned on routinely available data, namely histopathology images and clinical metadata, the proposed framework facilitates in silico experimen- tation, data augmentation for downstream models, and exploratory analyses without requiring additional wet-lab assays. Moreover, the intrinsic interpretabil- ity of the multimodal attention mechanism provides a principled way to relate tissue morphology to molecular variation, supporting hypothesis generation in computational pathology and precision medicine settings. Future work will focus on benchmarking M 3 -Gen against additional generative models and further leveraging its multimodal foundation. A promising direction is bidirectional synthesis: generating virtual histopathology images from gene expression inputs and studying how changes in one modality affect the other. M 3 -Gen: Multimodal Generation of Gene Expression Profiles15 This approach enables in silico experiments that are privacy-preserving and cost-effective, offering new opportunities to explore the molecular consequences of patient context and tissue morphology. Acknowledgements Model training and testing were possible thanks to the HPC grant from by the Ministry of Education, Youth and Sports of the Czech Republic through the e-INFRA CZ (ID:90254) References 1. Bio-medical: A high-performance biomedical language model. https://huggingface.co/ContactDoctor/Bio-Medical-Llama-3-8B (2024) 2.van Breugel, B., Liu, T., Oglic, D., van der Schaar, M.: Synthetic data in biomedicine via generative artificial intelligence. Nature Reviews Bioengineering 2(12), 991–1004 (2024) 3.Chen, R.J., Ding, T., Lu, M.Y., Williamson, D.F., Jaume, G., Chen, B., Zhang, A., Shao, D., Song, A.H., Shaban, M., et al.: Towards a general-purpose foundation model for computational pathology. Nature Medicine (2024) 4.Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of wasserstein gans. Advances in neural information processing systems 30 (2017) 5.Kynkänniemi, T., Karras, T., Laine, S., Lehtinen, J., Aila, T.: Improved precision and recall metric for assessing generative models. Advances in neural information processing systems 32 (2019) 6.Lacan, A., Sebag, M., Hanczar, B.: GAN-based data augmentation for transcrip- tomics: survey and comparative assessment. Bioinformatics 39(Supplement_1), i111–i120 (2023) 7.Lacan, A., Sebag, M., Hanczar, B.: Gan-based data augmentation for transcrip- tomics: survey and comparative assessment. Bioinformatics 39(Supplement_1), i111–i120 (06 2023). https://doi.org/10.1093/bioinformatics/btad239, https://doi. org/10.1093/bioinformatics/btad239 8.Lee, S.A., Wu, A., Chiang, J.N.: Clinical modernbert: An efficient and long context encoder for biomedical text. arXiv preprint arXiv:2504.03964 (2025) 9.Liu, H., Zhang, Y., Luo, J.: Contrastive learning-based histopathological features infer molecular subtypes and clinical outcomes of breast cancer from unannotated whole slide images. Computers in Biology and Medicine 170, 107997 (2024) 10.Otsu, N., et al.: A threshold selection method from gray-level histograms. Auto- matica 11(285-296), 23–27 (1975) 11.Panaccione, F.P., Mongardi, S., Masseroli, M., Pinoli, P.: Biogan: Enhancing transcriptomic data generation with biological knowledge. Bioengineering 12(6), 658 (2025) 12.Panaccione, F.P., Sgaravatti, C., Pinoli, P.: Gemm-gan: A multimodal generative model conditioned on histopathology images and clinical descriptions for gene expression profile generation. In: Rodolà, E., Galasso, F., Masi, I. (eds.) Image Analysis and Processing - ICIAP 2025 Workshops. p. 393–404. Springer Nature Switzerland, Cham (2026) 16F.P. Panaccione et al. 13.Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. p. 8748–8763. PmLR (2021) 14. Schmauch, B., Romagnoni, A., Pronier, E., Saillard, C., Maillé, P., Calderaro, J., Kamoun, A., Sefta, M., Toldo, S., Zaslavskiy, M., et al.: A deep learning model to predict rna-seq expression of tumours from whole slide images. Nature communications 11(1), 3877 (2020) 15. Viñas, R., Andrés-Terré, H., Liò, P., Bryson, K.: Adversarial generation of gene expression data. Bioinformatics 38(3), 730–737 (2022) 16.Wang, N.C., Kaplan, J., Lee, J., Hodgin, J., Udager, A., Rao, A.: Stress testing pathology models with generated artifacts. Journal of Pathology Informatics 12(1), 54 (2021). https://doi.org/https://doi.org/10.4103/jpi.jpi_6_21, https://w. sciencedirect.com/science/article/pii/S2153353922001766 17.Zheng, Y., Pizurica, M., Carrillo-Perez, F., Noor, H., Yao, W., Wohlfart, C., Marchal, K., Vladimirova, A., Gevaert, O.: Digital profiling of cancer transcriptomes from histology images with grouped vision attention. BioRxiv p. 2023–09 (2024)