Paper deep dive
Multimodal Knowledge Distillation for Gastric Adenocarcinoma Classification from Whole-Slide Images
Shrihari Dumbre, Bikash Santra
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Gastric adenocarcinoma (GA) is a leading cause of cancer-related mortality worldwide, and accurate histopathological subtype classification from whole-slide images (WSIs) is essential for effective treatment planning. While multimodal approaches that integrate pathology report text with WSIs can improve classification, existing methods often depend on computationally expensive transformer architectures and large language models. We propose a multimodal knowledge distillation (MKD) framework that combines a pretrained WSI image encoder and a clinical text encoder using Low-Rank Multimodal Fusion (LMF) to efficiently model cross-modal interactions during training. Each WSI is represented as a bag of patches paired with a slide-level diagnostic caption. The teacher model learns fused image-text representations for subtype classification, while the student model distills this knowledge to enable accurate image-only inference. We evaluate our method on the PatchGastric benchmark dataset and achieve at least 3.35% higher mean accuracy than state-of-the-art approaches, without relying on transformer-based fusion, multi-task learning, or large language models. The source code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.07913v1
- Canonical: https://arxiv.org/abs/2610.07913v1
Trouble viewing inline? Open PDF directly →
Full Text
42,141 characters extracted from source content.
Expand or collapse full text
Multimodal Knowledge Distillation for Gastric Adenocarcinoma Classification from Whole-Slide Images Shrihari Dumbre Affiliation: Indian Institute of Technology, Jodhpur , E-mail b23e1069@iitj.ac.in Bikash Santra E-mail bikash@iitj.ac.in Affiliation: Indian Institute of Technology, Jodhpur , E-mail b23e1069@iitj.ac.in Abstract Gastric adenocarcinoma (GA) is a leading cause of cancer-related mortality worldwide, and accurate histopathological subtype classification from whole-slide images (WSIs) is essential for effective treatment planning. While multimodal approaches that integrate pathology report text with WSIs can improve classification, existing methods often depend on computationally expensive transformer architectures and large language models. We propose a multimodal knowledge distillation (MKD) framework that combines a pretrained WSI image encoder and a clinical text encoder using Low-Rank Multimodal Fusion (LMF) to efficiently model cross-modal interactions during training. Each WSI is represented as a bag of patches paired with a slide-level diagnostic caption. The teacher model learns fused image–text representations for subtype classification, while the student model distills this knowledge to enable accurate image-only inference. We evaluate our method on the PatchGastric benchmark dataset and achieve at least 3.35% higher mean accuracy than state-of-the-art approaches, without relying on transformer-based fusion, multi-task learning, or large language models. The source code is available at https://github.com/helomelo1/MKD-LMF. Keywords: Multimodal Learning Low-Rank Multimodal Fusion Knowledge Distillation Gastric Cancer Classification Histopathology Image Analysis 1 Introduction Accurate histological subtype classification of gastric adenocarcinoma from whole-slide images (WSIs) is clinically critical for treatment planning, yet morphological similarity between subtypes makes this task challenging even for experienced pathologists [15]. Most recent multimodal approaches [5, 6, 7, 8, 9, 10] demonstrate that pairing WSI features with complementary data sources, such as genomic sequencing data, gene expression profiles and clinical records, improves classification. But most of them either require genomic sequencing data that is unavailable in routine clinical workflows [5, 6] or rely on highly complex architectures, such as PathM3 [10]. It employs a 12-layer Query Transformer initialized from BLIP-2 [11] together with a frozen FlanT5-XL decoder [22] for classification and caption generation jointly. The PatchGastric dataset [15] is one of the few publicly available datasets that provide both histopathology patch images and WSI-level diagnostic captions (i.e., report summaries), making it well-suited for multimodal analysis of gastric adenocarcinoma. However, the dataset presents a significant challenge. Many WSIs across different subtypes share near-identical descriptions, limiting their standalone discriminating power. Due to real-world data scarcity, methods on this dataset follow a data-scarce training protocol, using only 20% of the WSIs for training. Prior methods on this dataset, such as PathM3 [10] address these challenges using complex multi-task learning-based frameworks that jointly optimize classification and caption-generation using large-language models. While effective, these methods introduce substantial computational overhead. On the contrary, we propose an LMF-based multimodal fusion approach that fuses patch-level image features with pathology report captions using Low-Rank Multimodal Fusion (LMF) [12]. Notably, each WSI in the PatchGastric dataset [15] is provided as a set of pre-extracted patches of size 300×300300× 300 at 20×20× magnification. Following the Multiple Instance Learning paradigm [1], each WSI is treated as a bag of patch instances. Each patch embedding is obtained and mean-pooled via LMF to fuse with the WSI-level caption embedding. LMF models multiplicative cross-modal interactions through a low-rank tensor decomposition without requiring attention-based cross-modal modules [6, 10] or generative objectives [11, 10]. Pathology report captions are routinely generated during standard diagnostic workflows, making them a practical, readily available second modality alongside H&E-stained histopathology images. Unlike PG-MLIF [14], which applies LMF to fuse WSIs with genomic data for survival prediction, we apply LMF to the image-report setting for histological subtype classification of gastric adenocarcinoma. Our experiments compare the proposed method against ABMIL [1], CLAM [3], DSMIL [4], TransMIL [2], CITE [18], ILRA-MIL [19] and PathM3 [10]. We use the same PatchGastric dataset and the same train-validation-test split ratio as in CITE [18] and PathM3 [10]. We use three gastric adenocarcinoma subtypes, including “well differentiated tubular adenocarcinoma”, “moderately differentiated tubular adenocarcinoma”, and “poorly differentiated adenocarcinoma”, similar to these baselines, thus enabling a direct comparison. We further conduct experiments on different multimodal fusion strategies and ablation studies across multiple LMF rank settings. The contributions of this work are the following: • Lightweight multimodal framework: We propose an efficient multimodal knowledge distillation (MKD) approach that fuses WSI features and clinical text using Low-Rank Multimodal Fusion (LMF), avoiding heavy transformer-based architectures and large language models. • Knowledge distillation for image-only inference: We introduce a teacher –student setup where multimodal knowledge learned during training is distilled into a student model, enabling accurate and efficient inference using only WSIs. • Improved performance with efficiency: We conduct extensive experiments to analyze the contributions of individual modalities and the effect of the LMF rank in improving the gastric adenocarcinoma classification performance. 2 Related Works 2.1 Multiple Instance Learning for Histopathology Multiple Instance Learning (MIL) is a weakly supervised paradigm where training examples are organized into bags of instances, with labels available only at the bag level [20]. In computational pathology, a whole-slide image (WSI) is treated as a bag containing hundreds to thousands of patch-level instances (small image regions extracted at specific magnification), with the number of patches varying across WSIs. The WSI-level label is known, but individual patch labels are not, making MIL a natural fit for WSI analysis. Ilse et al. [1] proposed Attention-based MIL (ABMIL), which learns attention weights to aggregate patch features into a bag-level representation. In another work, Shao et al. [2] introduced Transformer-based correlated MIL (TransMIL), employing transformer-based self-attention with Nyström approximation [21] to model correlations among patches efficiently. Whereas, Lu et al. [3] proposed Clustering-Constrained-Attention Multiple-Instance Learning (CLAM), a data-efficient and weakly supervised framework for WSI classification. While Li et al. [4] introduced Dual-Stream MIL (DSMIL), a dual-stream network that incorporates self-supervised contrastive learning. All of these methods rely exclusively on image features for histopathology classification. On the contrary, we introduce an MIL-based multimodal framework that uses a bag-of-image-patches representation and a caption generated from the pathology report. 2.2 Multimodal Fusion on Computation Pathology A number of recent works [5, 6, 7, 8, 9, 10] have explored multimodal fusion for cancer prognosis and classification by combining histopathology images along with complementary data modalities. Chen et al. [5] proposed Pathomic Fusion, integrating histopathology and genomic features via the Kronecker product for survival prediction. While effective in modelling cross-modal relationships, its tensor fusion is computationally expensive, requires paired genomic data, and assumes the availability of both modalities during inference. In another work, Chen et al. [6] introduced Multimodal Co-Attention Transformer (MCAT), a co-attention Transformer fusing WSIs with genomic data, while Xu and Chen [7] proposed Multimodal Optimal Transport Co-Attention Transformer (MOTCat) using optimal transport-based co-attention. On the other hand, Zhou and Chen [8] introduced Cross-Modal Translation and Alignment (CMTA) for cross-modal translation and alignment, and Ding et al. [9] proposed PathOmics, a pathology-genomics Transformer for survival prediction. On the other hand, recent vision-language approaches focus on image-text representations that leverage widely available diagnostic captions, enabling more flexible multimodal learning. For example, most closely related to our work, Zhou et al. [10] proposed PathM3, an MIL-based multimodal framework. PathM3 aligns WSIs with diagnostic captions using a query-based transformer initialized with BLIP-2 [11], jointly training for classification and caption generation. While PathM3 achieves strong performance, it relies on a computationally expensive architecture with a 12-layer query Transformer and a frozen FlanT5-XL decoder. In contrast, we employ a knowledge distillation framework with a low-rank multimodal fusion approach that directly learns joint image-text representations for WSI-only classification, avoiding the need for caption-generation objectives and large language models while achieving superior classification performance. 2.3 Low-Rank Multimodal Fusion and Knowledge Distillation Low-Rank Multimodal Fusion (LMF) was introduced by Liu et al. [12] as an efficient alternative to Tensor Fusion Networks [13]. LMF uses parallel low-rank decomposition to model pairwise interactions between features from the image and text modalities while avoiding the exponential parameter growth of full tensor fusion [13]. For cancer survival prediction, Pan et al. [14] applied LMF to fuse histopathology images with genomic data. PG-MLIF [14] requires genomic profiling data, which is expensive and not always clinically available. Unlike PG-MLIF [14], we apply LMF to fuse histopathology images with pathology report summaries, which are routinely generated as part of standard diagnostic workflows. Knowledge distillation [23] transfers knowledge from a large teacher model to a smaller student model. FitNets [24] extended this by distilling intermediate representations, thereby enabling richer feature learning. In our approach, we employ knowledge distillation to transfer the multimodal representation learned from both image and text to a student model that operates solely on images. 3 Methodology Figure 1: Overview of the proposed two-stage MKD-LMF framework. WSI patches are encoded and mean-pooled to obtain a bag-level image representation (img)(z_img), while the corresponding caption is passed through a text encoder to extract text features (txt)(z_txt). In Stage 1, the teacher network employs a Low-Rank Multimodal Fusion (LMF) module to combine (img)(z_img) and (txt)(z_txt), producing a fused representation (fused)(z_fused), which is used to train a linear classifier (shared with the student) for gastric adenocarcinoma (GA) subtype classification. In Stage 2, the student network takes only (img)(z_img) as input and learns, via a linear knowledge distillation module, to approximate fusedz_fused generated by the teacher’s frozen LMF module. The synthesized representation is then passed to the shared classifier to enable image-only GA subtype classification. 3.1 Problem Formulation Consider a dataset of M whole-slide images (WSIs), where each WSI is represented as a bag of N patches ℬ=p1,p2,…,pNB=\p_1,p_2,…,p_N\, with N varying across WSIs based on tissue regions. Each WSI is paired with a diagnostic caption c provided in the dataset, derived from the corresponding pathology report. The goal is to classify WSIs into K=3K=3 subtypes of gastric adenocarcinoma: well-differentiated, moderately differentiated, and poorly differentiated tubular adenocarcinoma. We address this 3-way classification of gastric adenocarcinoma in two stages. First, a multimodal teacher fteacher:(ℬ,c)→yf_teacher:(B,c)→ y is trained using both image and text modalities. Second, an image-only student fstudent:ℬ→yf_student:B→ y is trained via knowledge distillation, enabling WSI-only inference without the pathology report. The proposed complete pipeline is illustrated in Fig. 1. 3.2 Image Feature Extraction First, the patches of a WSI pi∈ℬ⊂ℝH×W×3p_i ^H× W× 3 with height H and width W are resized to 224×224224× 224. Subsequently, each patch pip_i is passed through a pretrained image encoder ϕimg _img to obtain a dvd_v-dimensional patch-level image embedding: i=ϕimg(pi)∈ℝdv.v_i= _img(p_i) ^d_v. (1) In our implementation, Phikon [16], a Vision Transformer (ViT-B/16) [30] pretrained on large-scale histopathology data via self-supervised learning [31] is used as our image encoder, where dv=768d_v=768. The weights of ϕimg _img are kept frozen during training of the proposed model. 3.3 Bag-Level Feature Representation via Aggregation Following the MIL paradigm, we aggregate the patch-level representations iv_i within each bag ℬB to obtain a single bag-level representation. Since patches within a bag have no inherent ordering, the aggregation function must be permutation invariant, meaning the output should not change when the order of input patches is rearranged. This property is essential because the model should produce the same prediction regardless of how patches are arranged. Thus, we use mean pooling as the permutation-invariant aggregation function: img=1N∑k=1Nk∈ℝdv,z_img= 1N _k=1^Nv_k ^d_v, (2) to treat all patches equally, providing a simple yet effective summarization of the bag contents. 3.4 Text Feature Extraction The caption c, a summarized pathology report, associated with the input WSI is encoded using a pretrained text encoder ϕtxt _txt to obtain a dtd_t-dimensional bag-level text embedding: txt=ϕtxt(c)∈ℝdt.z_txt= _txt(c) ^d_t. (3) We use BGE-small-en-v1.5 [17], a pretrained sentence-embedding model, as our text encoder, with dt=384d_t=384. Since all patches within a bag share the same WSI-level caption, the text embedding txtz_txt is paired with the bag-level aggregated and fixed-dimensional representation imgz_img of WSI patches. The weights of ϕtxt _txt are kept frozen during the training of our model. 3.5 Low-Rank Multimodal Fusion Multimodal GA subtype classification requires capturing high-order correlations between visual and textual embeddings imgz_img and txtz_txt. Simple additive fusion schemes (such as feature concatenation or summation) assume that modalities contribute independently, which fails to model inter-modal dependencies. Conversely, multiplicative fusion captures the cross-modal interactions by computing pairwise products between every image feature and every text feature. However, mapping this full tensor to the output space requires a high-dimensional fusion weight tensor whose parameters scale quadratically with the number of features, leading to severe memory overhead. To capture and approximate multiplicative cross-modal interactions between the image and text modalities, we employ Low-Rank Multimodal Fusion (LMF) [12]. Given a bag-level representation of patches img∈ℝdvz_img ^d_v and the caption embedding txt∈ℝdtz_txt ^d_t, LMF computes a fused representation. It introduces two sets of learnable projection matrices v(k)W_v^(k) and t(k)W_t^(k) specific to the image and text modalities, respectively We append a constant 1 to each embedding vector to incorporate a bias term into the low-rank factorization, enabling the model to learn an additive offset without a separate bias parameter. Appending a constant 1 allows the augmented dimensions to act as multiplicative identities, enabling the factorization to represent image-only, text-only, and bias terms in addition to cross-modal interactions as follows: ¯img=[img;1]∈ℝdv+1,¯txt=[txt;1]∈ℝdt+1. z_img=[z_img;1] ^d_v+1, z_txt=[z_txt;1] ^d_t+1. (4) Rather than explicitly constructing the full fusion tensor, LMF factorizes the fusion weight tensor into r distinct components. For each factor j∈1,…,rj∈\1,…,r\, learnable projection matrices v(j)∈ℝdf×(dv+1)W_v^(j) ^d_f×(d_v+1) and t(j)∈ℝdf×(dt+1)W_t^(j) ^d_f×(d_t+1) project the augmented embeddings into a common dfd_f-dimensional space, and the fused representation is obtained by summing element-wise products across all factors: fused=∑j=1r(v(j)¯img)⊙(t(j)¯txt)∈ℝdf,z_fused= _j=1^r (W_v^(j) z_img ) (W_t^(j) z_txt ) ^d_f, (5) where ⊙ denotes the Hadamard (element-wise) product, r is the rank, a hyperparameter controlling the expressiveness of the decomposition, and dfd_f is the fused embedding dimension. This formulation effectively retains the rich cross-modal interactions. Conventional multiplicative fusion requires (df⋅dv⋅dt)O(d_f· d_v· d_t) trainable parameters. In contrast, LMF reduces this parameter count to (r⋅df⋅(dv+dt))O(r· d_f·(d_v+d_t)), and thereby drastically improves parameter efficiency. 3.6 Classification The proposed teacher-student MKD framework uses a shared classifier trained on the teacher side, which is kept frozen on the student side to distil the teacher’s knowledge. As shown in Fig. 1, on the teacher network, the fused multimodal representation fusedz_fused is fed to a linear classifier (ϕcls _cls) to obtain the predicted GA subtype class probabilities: ^T=ϕcls(fused)∈ℝK. y_T= _cls(z_fused) ^K. (6) Figure 2: Architecture of the proposed Knowledge Distiller, a residual multi-layer perceptron (MLP), comprising an input projection, stacked residual blocks, and an output projection for mapping image features to the fused feature space. 3.7 Knowledge Distillation for Synthesizing Multimodal Embedding The multimodal model often achieves high accuracy by using both image and text data. However, it requires the pathology report at inference time. Pathology reports for histopathology images are generated after pathologists have made the diagnostic decision and identified the GA subtype. This limits its clinical applicability, as our objective is to automate this GA subtype identification. To address this limitation, we introduce a student network in which a Knowledge Distiller H learns to predict the teacher’s multimodal fused representation fusedz_fused from only the mean-pooled image features imgz_img. Thus, ^fused=H(img)∈ℝdf, z_fused=H(z_img) ^d_f, (7) where ^fused z_fused is an approximated representation of fusedz_fused (obtained from the teacher). The architecture of the proposed Knowledge Distiller is demonstrated in Fig. 2. It is then passed to the frozen shared classifier ϕcls _cls for WSI-only GA subtype classification. 3.8 Two-Stage Training Our proposed framework employs a two-stage training pipeline. 1: Multimodal Training. On the teacher side, we train the LMF-based multimodal model using both the image and text modalities by optimizing the cross-entropy loss ℒCEL_CE between y and y^T y_T. The teacher learns to produce discriminative fused representations fusedz_fused that combine pathological information from WSI patches and diagnostic information from text captions. After training, the teacher’s module parameters (encoders, LMF module, and classifier) are frozen, as illustrated in Fig. 1. Stage 2: Knowledge Distillation for Image-Only Inference. Using the teacher’s frozen modules, we train the Knowledge Distiller network H to predict the teacher’s fused representations directly from image features. The teacher’s classifier is shared with the student and kept frozen during this stage. We employ a combined distillation loss, ℒdistillL_distill as follows: ℒdistill=ℒCE+λfeatℒfeat(^fused,fused)⏟feature matching loss+λKDℒKD(^S,^T)⏟logit distillation lossL_distill=L_CE+ _feat L_feat( z_fused,\,z_fused)_feature matching loss+ _KD L_KD( y_S,\, y_T)_logit distillation loss (8) The cross-entropy loss ℒCEL_CE supervises the student’s predictions ^S y_S against the ground-truth labels y. While the feature matching loss uses Smooth L1 [29] to maximize the teacher-student agreement in synthesizing the multimodal fused representation by minimizing the distance between the representations predicted by them. On the other hand, the logit distillation loss uses KL divergence [23] with temperature τ to align the student’s output distribution with the teacher’s: ℒKD=τ2⋅DKL(softmax(^Sτ)∥softmax(^Tτ)),L_KD=τ^2· D_KL\! (softmax\! ( y_Sτ )\; \|\;softmax\! ( y_Tτ ) ), (9) where we experimentally set τ=2τ=2, λfeat=1.0 _feat=1.0, and λKD=0.5 _KD=0.5. Why triple supervision? Cross-entropy alone trains the student’s knowledge distillation module from scratch without leveraging the teacher’s learned structure. Feature-level supervision enforces the student to reproduce the teacher’s intermediate representations, but might not preserve class boundaries if the representation space contains redundant information. Logit-level supervision transfers the teacher’s soft class relationships but does not capture the full structure of the fused features. Combining all three ensures the Knowledge Distiller learns representations that are structurally faithful to the teacher’s, discriminative for classification, and grounded in the true labels. 3.9 Inference At inference time, the proposed pipeline requires only the WSI, as shown in the student block of the Fig. 1. Using the frozen Phikon [16] encoder ϕimg _img, we extract patch-level features, as discussed in Section 3.2. We aggregate the patches using mean pooling to obtain a fixed-dimensional WSI-level representation imgz_img, as explained in Section 3.3. We pass the imgz_img through the knowledge distiller H to obtain ^fused z_fused, a sythesized representation of the image-text fused embedding fusedz_fused generated by the teacher. Subsequently, this synthesized representation is then passed through the frozen shared classifier ϕcls _cls to obtain the final predictions, ^S y_S, as described in Section 3.7. Note that the pathology report is used only during training. At inference time, the model requires only the WSI, making it suitable for real-time clinical decision support when reports are not yet available. 4 Experiments and Results 4.1 Dataset In our experiments evaluating the proposed method, we use the PatchGastric [15] dataset, which originally consists of 991 WSIs, from which patches are extracted and paired with captions from diagnostic reports. There are 262,777 patches in the dataset, each 300x300 pixels in size, extracted at 20x magnification. The dataset contains 9 subtypes of gastric adenocarcinoma, out of which we use 3 following the baselines [18, 10] for a fair comparison: "well differentiated tubular adenocarcinoma", "moderately differentiated tubular adenocarcinoma" and "poorly differentiated adenocarcinoma". This reduces the patches to 179,285 across 693 WSIs. The data split ratio is the same as in all previous methods. We randomly split the dataset into train (20%), validate (40%) and test (40%). 4.2 Implementation Details For image feature extraction, we use Phikon [16], a ViT-B/16 encoder with an input image size of 224×224, in which each input image is split into non-overlapping 16×16 patches. Each WSI patch is resized to 224x224 for input, and then the encoder converts it into a dv=768d_v=768 dimensional embedding. The final output after passing all the patches from the WSI is N x 768, where N is the number of patches varying across WSIs. For caption encoding we use BGE-small-en-v1.5 [17], a retrieval-optimized sentence encoder from the BGE model family. The text encoder returns a dt=384d_t=384 dimensional embedding. The LMF module is implemented with the rank r=128r=128 (experimentally set as discussed in Section 4.3) to obtain a fused embedding of size df=128d_f=128. The shared classifier is a fully connected layer mapping from an input size of 128 to 3, consisting of a ReLU activation and a dropout layer with probability 0.3. The teacher model is trained for 20 epochs using the Adam [32] optimzer with a learning rate of 1e-4, optimizing cross-entropy loss. As illustrated in Fig. 2, the Knowledge Distiller (KD) is implemented as a residual network [25] consisting of three components. First, an input projection maps a dv=768d_v=768 dimensional input to dh=512d_h=512 via a linear layer followed by Layer Normalization [26], GELU [27] activation, and dropout with probability 0.3. Second, each residual block preserves the 512512-dimensional hidden representation and consists of two (512→512512→ 512) linear layers, each followed by Layer Normalization and GELU activation, with a skip connection between the block input and output. A dropout layer with a probability of 0.4 is applied within each residual block. Third, the hidden representation dh=512d_h=512 is then mapped to the output df=128d_f=128 via a linear layer followed by Layer Normalization [26]. The student model is trained for 50 epochs using the Adam [32] optimizer with a learning rate of 1e-4, optimizing the distillation loss ℒdistillL_distill defined in (8). The proposed method is implemented in PyTorch, and all experiments are evaluated on a computing system with NVIDIA RTX 6000 ADA 48GB GPU, 512 GB of RAM and an Intel Xeon Gold 6426Y processor. Reported classification accuracy is the mean and standard deviation over three independent runs. Table 1: Comparison with state-of-the-art (SOTA) methods under image-only inference. Results are reported as mean accuracy (%) ± standard deviation. denotes the difference in mean accuracy of SOTA methods compared to our method (MKD-LMF). Red-colored negative values indicate lower performance than ours, while positive values indicate higher performance. Method Accuracy (%) ABMIL [1] 66.67±2.6366.67± 2.63 −8.16-8.16 CLAM [3] 67.83±1.1267.83± 1.12 −7.00-7.00 DSMIL [4] 69.02±0.4269.02± 0.42 −5.81-5.81 TransMIL [2] 67.48±1.1667.48± 1.16 −7.35-7.35 CITE [18] 69.63±0.9169.63± 0.91 −5.20-5.20 ILRA-MIL [19] 70.16±1.1170.16± 1.11 −4.67-4.67 PathM3 [10] 71.48±1.3071.48± 1.30 −3.35-3.35 MKD-LMF (Ours) 74.83±1.4974.83± 1.49 4.3 Results and Analysis Comparison with state-of-the-art methods. We compare our method against state-of-the-art WSI classification methods under the image-only inference setting. Results for all baseline methods are taken from PathM3 [10], which uses identical experimental settings, including the same dataset, data split ratio, and three GA subtypes for classification, thereby enabling direct comparison. As shown in Table 1, our method achieves 74.83% accuracy, outperforming all baselines, including PathM3 (71.48%) by 3.35 percentage points and the best attention-based MIL method ILRA-MIL (70.16%) by 4.67 percentage points. The consistent margin over established MIL baselines, including ABMIL, CLAM, DSMIL, TransMIL, and CITE, demonstrates the effectiveness of our approach in gastric adenocarcinoma subtype classification. Ablation study. Table 2: Ablation study of the proposed MKD-LMF framework. The results of the experiments in the table are reported for a single run. The table analyzes (i) the performance of the teacher model using both image and text during inference, which is not clinically relevant, (i) the effect of different distillation loss components in combination with the classification loss (cross-entropy) on the student-side and (i) the impact of different inference modalities. Results show that combining both losses yields the best student performance. The student model in our multimodal knowledge distillation (KD) framework substantially outperforms unimodal (image-only or text-only) inference settings. Study Configuration Accuracy (%) Multimodal Inference Proposed teacher model 92.7492.74 Distillation Loss Feature loss (ℒfeatL_feat) 73.6373.63 KD loss (ℒKDL_KD) 73.1373.13 Feature loss + KD loss 74.8374.83 Inference Modality Text-only (unimodal) 68.7668.76 Image-only (unimodal) 70.3570.35 Image-only (proposed multimodal KD) 74.8374.83 We conduct ablation studies to validate the key design choices in our framework, as described in the following three paragraphs. Notably, results of our ablation study are summarized in Table 2, where accuracy is determined in only one run of each model. Multimodal Inference. We evaluated the teacher model independently under a multimodal inference setting using both image and text inputs. The teacher achieves the best performance among all ablated variants (see Table 2) by leveraging complementary information from the whole-slide image and the diagnostic report summary to classify GA subtypes. However, as discussed in Section 1, this multimodal setup is not clinically practical, since report generation requires clinician involvement and is assumed to be not available at inference time. Importance of Distillation Loss Components. We ablate the proposed dual-objective distillation loss by training the student model with each component independently and in combination, while keeping the cross-entropy loss fixed. Using only the feature loss (ℒfeat)(L_feat) or only the KD logit loss (ℒKD)(L_KD) leads to lower performance, lagging behind the combined objective by 1.20%1.20\% and 1.70%1.70\% points, respectively. In contrast, jointly optimizing both losses yields the best performance, demonstrating that they provide complementary supervision signals for more effective knowledge transfer. Unimodal vs. Image-Only Multimodal KD. To quantify the contribution of multimodal fusion, we evaluate classifiers trained on individual modalities without fusion or KD. The image-only unimodal model achieves an accuracy of 70.35%, whereas the text-only model obtains 68.76%. Compared with these unimodal baselines, the distilled student model (i.e., our image-only multimodal KD framework) improves the performance by 4.48 percentage points over the image-only model and by 6.07 percentage points over the text-only model. Furthermore, the full multimodal teacher model achieves a substantial gain of 17.91 percentage points over the distilled student model. These results demonstrate that the LMF-based fusion effectively leverages complementary information across multiple modalities and that the Feature Distiller successfully transfers this multimodal knowledge to our image-only inference pipeline. (a) Effect of λKD _KD. (b) Effect of LMF rank. Figure 3: Results for different choices of (a) the coefficient of logit distillation loss, λKD _KD and (b) the rank in the low-rank multimodal fusion (LMF) approach Choice of logit distillation loss coefficient. We also experimented with tuning the λKD _KD parameter to four settings (0.5, 1.0, 2.0, & 5.0). Our best results (74.83%) were achieved with λKD _KD = 0.5. The accuracy of training the student with other values is presented in Figure 3(a). Choice of rank in LMF. We experimented with the teacher model’s rank hyperparameter for implementing LMF. The student model distils knowledge from the teacher model, which is trained with four different ranks: 16, 32, 64, and 128. The highest accuracy was observed with r=128r=128, as shown in Fig. 3(b). It is likely that a higher rank enables the teacher to learn richer multimodal representations via cross-modal interactions. Figure 4: t-SNE visualization of learned embeddings with corresponding DBI scores: image-only (5.2641), text-only (2.3107), teacher (0.7309), and distilled student (1.1078). Blue represents "moderately differentiated tubular adenocarcinoma". Orange represents "poorly diffrentiated adenocarcinoma". Green represents "well diffrentiated tubular adenocarcinoma" Feature visualizations. Fig. 4 shows t-SNE plots of learned representations which are imgz_img (image-only features), txtz_txt (text-only features), fusedz_fused (LMF representation of fusion of text and image embeddings) and ^fused z_fused (Predicted embeddings by the knowledge distiller from the patch features only). To qualitatively assess the representation quality, we compute the Davies-Bouldin Index (DBI) on the learned embeddings. It measures the ratio of within-cluster dispersion to between-cluster separation, where lower values indicate better-defined clusters. The teacher model achieves the lowest DBI (0.7309), followed by the distilled student model (1.1078), both substantially outperforming the text-only (2.3107) and image-only (5.2641) representations, confirming the improved cluster compactness and inter-class separation. 5 Conclusion We proposed an LMF-based multimodal fusion framework for image-only classification of gastric adenocarcinoma, fusing diagnostic captions with patches extracted from histopathological whole-slide images (WSIs). The proposed method outperforms the strongest baseline by 3.35%. Our results suggest that carefully chosen fusion mechanisms can outperform architecturally complex methods in computational pathology. While our framework demonstrates strong performance, it is limited to a single dataset and a single cancer type. Future work could explore evaluation across diverse datasets of various cancer types. References [1] Ilse, M., Tomczak, J., Welling, M.: Attention-based deep multiple instance learning. In: International Conference on Machine Learning, p. 2127–2136. PMLR (2018) [2] Shao, Z., Bian, H., Chen, Y., Wang, Y., Zhang, J., Ji, X.: TransMIL: Transformer based correlated multiple instance learning for whole slide image classification. Advances in Neural Information Processing Systems 34, 2136–2147 (2021) [3] Lu, M.Y., Williamson, D.F., Chen, T.Y., Chen, R.J., Barbieri, M., Mahmood, F.: Data-efficient and weakly supervised computational pathology on whole-slide images. Nature Biomedical Engineering 5(6), 555–570 (2021) [4] Li, B., Li, Y., Eliceiri, K.W.: Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2021) [5] Chen, R.J., Lu, M.Y., Wang, J., Williamson, D.F., Rodig, S.J., Lindeman, N.I., Mahmood, F.: Pathomic fusion: An integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. IEEE Transactions on Medical Imaging 41(4), 757–770 (2020) [6] Chen, R.J., Lu, M.Y., Weng, W.H., Chen, T.Y., Williamson, D.F., Manz, T., Shady, M., Mahmood, F.: Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 4015–4025 (2021) [7] Xu, Y., Chen, H.: Multimodal optimal transport-based co-attention transformer with global structure consistency for survival prediction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 21241–21251 (2023) [8] Zhou, F., Chen, H.: Cross-modal translation and alignment for survival analysis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 21485–21494 (2023) [9] Ding, K., Zhou, M., Metaxas, D.N., Zhang, S.: Pathology-and-genomics multimodal transformer for survival outcome prediction. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 622–631. Springer (2023) [10] Zhou, Q., Zhong, W., Guo, Y., Xiao, M., Ma, H., Huang, J.: PathM3: A multimodal multi-task multiple instance learning framework for whole slide image classification and captioning. arXiv preprint arXiv:2403.08967 (2024) [11] Li, J., Li, D., Savarese, S., Hoi, S.: BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In: International Conference on Machine Learning, p. 19730–19742. PMLR (2023) [12] Liu, Z., Shen, Y., Lakshminarasimhan, V.B., Liang, P.P., Zadeh, A., Morency, L.P.: Efficient low-rank multimodal fusion with modality-specific factors. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, p. 2247–2256 (2018) [13] Zadeh, A., Chen, M., Poria, S., Cambria, E., Morency, L.P.: Tensor fusion network for multimodal sentiment analysis. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, p. 1103–1114 (2017) [14] Pan, X., An, Y., Lan, R., Liu, Z., Liu, Z., Lu, C., Yang, H.: PG-MLIF: Multimodal low-rank interaction fusion framework integrating pathological images and genomic data for cancer prognosis prediction. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 1–10. Springer (2023) [15] Tsuneki, M., Kanavati, F.: Inference of captions from histopathological patches. In: International Conference on Medical Imaging with Deep Learning, p. 1235–1250. PMLR (2022) [16] Filiot, A., Ma, Y., Mac Kain, A., Walker, C., Veuthey, B., Frati, P., Kestelman, F., Saillard, C., Bernigot, A., Benhenda, S.: Scaling self-supervised learning for histopathology with masked image modeling. medRxiv (2023). https://doi.org/10.1101/2023.07.21.23292757 [17] Xiao, S., Liu, Z., Zhang, P., Muennighoff, N.: C-Pack: Packaged resources to advance general Chinese embedding. arXiv preprint arXiv:2309.07597 (2023) [18] Zhang, Y., Gao, J., Zhou, M., Wang, X., Qiao, Y., Zhang, S., Wang, D.: Text-guided foundation model adaptation for pathological image classification. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 272–282. Springer (2023) [19] Xiang, J., Zhang, J.: Exploring low-rank property in multiple instance learning for whole slide image classification. In: The Eleventh International Conference on Learning Representations (2023) [20] Dietterich, T.G., Lathrop, R.H., Lozano-Pérez, T.: Solving the multiple instance problem with axis-parallel rectangles. In: Artificial Intelligence Conference, p. 234–248 (1997) [21] Xiong, Y., Zeng, Z., Chakraborty, R., Tan, M., Fung, G., Li, Y., Singh, V.: Nyströmformer: A Nyström-based algorithm for approximating self-attention. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, p. 14138–14148 (2021) [22] Chung, H.W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., et al.: Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 (2022) [23] Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. In: NIPS Deep Learning and Representation Learning Workshop (2015) [24] Romero, A., Ballas, N., Kahaei, S.E., Vincent, P., Bengio, Y.: FitNets: Hints for thin deep nets. In: International Conference on Learning Representations (2015) [25] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 770–778 (2016) [26] Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv:1607.06450 (2016) [27] Hendrycks, D., Gimpel, K.: Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415 (2016) [28] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15(1), 1929–1958 (2014) [29] Girshick, R.: Fast R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, p. 1440–1448 (2015) [30] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (2021) [31] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 16000–16009 (2022) [32] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: International Conference on Learning Representations (2015)