Paper deep dive
K-MaT: Knowledge-Anchored Manifold Transport for Cross-Modal Prompt Learning in Medical Imaging
Jiajun Zeng, Shadi Albarqouni
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 3/13/2026, 12:18:47 AM
Summary
K-MaT (Knowledge-Anchored Manifold Transport) is a prompt-learning framework designed for zero-shot cross-modal transfer in medical imaging. It addresses catastrophic forgetting when transferring models from high-end modalities (e.g., CT) to low-end modalities (e.g., X-ray) by factorizing prompts, anchoring them to LLM-generated clinical descriptions, and aligning prompt manifolds using Fused Gromov-Wasserstein (FGW) optimal transport.
Entities (5)
Relation Signals (3)
K-MaT → builton → BiomedCLIP
confidence 100% · We build on the BiomedCLIP [15] backbone and propose K-MaT
K-MaT → uses → Fused Gromov-Wasserstein
confidence 100% · aligns the low-end prompt manifold to the visually-grounded high-end space using Fused Gromov-Wasserstein optimal transport
K-MaT → mitigates → Catastrophic forgetting
confidence 95% · it mitigates the catastrophic forgetting seen in standard methods like CoOp
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large-scale biomedical vision-language models (VLMs) adapted on high-end imaging (e.g., CT) often fail to transfer to frontline low-end modalities (e.g., radiography), collapsing into modality-specific shortcuts. We propose K-MaT (Knowledge-Anchored Manifold Transport), a prompt-learning framework that transfers decision structures to low-end modalities without requiring low-end training images. K-MaT factorizes prompts, anchors them to clinical text descriptions, and aligns the low-end prompt manifold to the visually-grounded high-end space using Fused Gromov-Wasserstein optimal transport. We evaluate K-MaT on four cross-modal benchmarks, including dermoscopy, mammography to ultrasound, and CT to chest X-ray. K-MaT achieves state-of-the-art results, improving the average harmonic mean of accuracy to 44.1% (from BiomedCoOp's 42.0%) and macro-F1 to 36.2%. Notably, on the challenging breast imaging task, it mitigates the catastrophic forgetting seen in standard methods like CoOp (which drops to 27.0% accuracy on the low-end), preserving robust performance across modalities. Aligning prompt manifolds via optimal transport provides a highly effective route for the zero-shot cross-modal deployment of medical VLMs.
Tags
Links
- Source: https://arxiv.org/abs/2603.06340v1
- Canonical: https://arxiv.org/abs/2603.06340v1
Trouble viewing inline? Open PDF directly →
Full Text
27,140 characters extracted from source content.
Expand or collapse full text
11institutetext: University of Bonn, University Hospital Bonn, Clinic for Diagnostic and Interventional Radiology, 53127 Bonn, Germany 11email: Zeng.Jiajun, Shadi.Albarqouni@ukbonn.de K-MaT: Knowledge-Anchored Manifold Transport for Cross-Modal Prompt Learning in Medical Imaging Jiajun Zeng Shadi Albarqouni✉ Abstract Large-scale biomedical vision-language models (VLMs) adapted on high-end imaging (e.g., CT) often fail to transfer to frontline low-end modalities (e.g., radiography), collapsing into modality-specific shortcuts. We propose K-MaT (Knowledge-Anchored Manifold Transport), a prompt-learning framework that transfers decision structures to low-end modalities without requiring low-end training images. K-MaT factorizes prompts, anchors them to clinical text descriptions, and aligns the low-end prompt manifold to the visually-grounded high-end space using Fused Gromov-Wasserstein optimal transport. We evaluate K-MaT on four cross-modal benchmarks, including dermoscopy, mammography to ultrasound, and CT to chest X-ray. K-MaT achieves state-of-the-art results, improving the average harmonic mean of accuracy to 44.1% (from BiomedCoOp’s 42.0%) and macro-F1 to 36.2%. Notably, on the challenging breast imaging task, it mitigates the catastrophic forgetting seen in standard methods like CoOp (which drops to 27.0% accuracy on the low-end), preserving robust performance across modalities. Aligning prompt manifolds via optimal transport provides a highly effective route for the zero-shot cross-modal deployment of medical VLMs. †✉ Corresponding author. 1 Introduction Deep learning models for medical imaging often degrade under distribution shifts, particularly during cross-modal transfer where distinct acquisition physics encourage modality-specific shortcuts [7, 13, 3]. For instance, models trained on high-end screening modalities (e.g., MRI, CT) rarely generalize reliably to accessible, frontline modalities (e.g., X-ray and Ultrasound), despite targeting identical underlying pathologies. Vision-language models (VLMs) and prompt learning offer efficient adaptation paradigms without full finetuning [10, 15, 17]. While BiomedCLIP exhibits high generalization, it struggles with specific tasks. CoOp and its variants [17, 12] attempt to optimize unified prompts to help models generalize across seen and unseen classes; however, these prompts are prone to overfit to seen classes and domains. Furthermore, while BiomedCoOp [5] improves low-shot biomedical transfer using Large Language Model (LLM)-generated visual descriptions and knowledge distillation from zero-shot BiomedCLIP, it still suffers from similar problems. In cross-modal settings, we observe that learnable prompts optimized exclusively on high-end modalities suffer from catastrophic knowledge forgetting—they collapse into modality-specific statistics and fail to preserve the essential general textual knowledge required for shared diagnostic semantics [6]. To address this, we study asymmetric cross-modal knowledge transfer and ask whether diagnostic semantics learned from high-end visual data can be reliably transferred to low-end modalities in a strict zero-shot regime. We build on the BiomedCLIP [15] backbone and propose K-MaT (Knowledge-Anchored Manifold Transport), a structure-preserving factorized prompt learning framework. Our core contributions are: (i) We introduce a strict zero-shot asymmetric transfer strategy relying solely on high-end visual data and LLM-generated clinical descriptions, eliminating the need for low-end visual training data. (i) We mitigate catastrophic forgetting via a novel space anchoring constraint, treating LLM-generated textual prototypes as semantic anchors to prevent deviation from clinically meaningful semantics. (i) We propose a cross-modal manifold alignment objective using Fused Gromov-Wasserstein (FGW) optimal transport [11], enforcing the learned low-end prompt manifold to strictly mirror the visually grounded relational structure of the high-end manifold. (iv) Validated on four cross-modal dataset pairs, our approach successfully prevents the decision boundary from collapsing into modality-specific shortcuts, achieving competitive zero-shot generalization against baselines that require target domain data. 2 Proposed Method In this section, we detail our proposed K-MaT prompt learning framework for asymmetric cross-modal transfer, as depicted in Fig. 1. Here, comprehensive screening uses a high-end source modality HX_H, while frontline assessments rely on a low-end target modality LX_L. We aim for High-to-Low generalization by performing few-shot learning on HX_H and zero-shot inference on LX_L, entirely bypassing low-end visual data. Backbone Model and Prompt Parameterization. Following the prompt tuning paradigm, we build upon a frozen biomedical VLM (BiomedCLIP [15]) containing a visual encoder ϕ(⋅)φ(·) and a textual encoder θ(⋅)θ(·). We freeze the pretrained visual and textual encoders and adapt the model via learnable context vectors. For a downstream task consisting of NcN_c categories, we introduce learnable context vectors to generate task-related textual embeddings. For the i-th class token ic_i and modality m∈H,Lm∈\H,L\, we define a factorized learnable prompt: i,m=[i,m1,i,m2,i,m3,i,m4,i]t_i,m=[v^1_i,m,v^2_i,m,v^3_i,m,v^4_i,m,c_i] where i,mkv^k_i,m denote the modality-specific and class-specific learnable context tokens. The specific textual embedding wi,mw_i,m is obtained by feeding the learnable prompt i,mt_i,m into the textual encoder θ(⋅)θ(·), i.e., wi,m=θ(i,m)w_i,m=θ(t_i,m). Figure 1: Overall architecture of the proposed K-MaT framework. During training, high-end textual prompts are optimized using high-end imaging data (ℒceL_ce), while prompts from both modalities are anchored to LLM-generated clinical descriptions (ℒancL_anc). The low-end prompt manifold is strictly aligned to the visually grounded high-end space via ℒfgwL_fgw. During inference, the frozen visual encoder extracts features from unseen low-end images, and predictions are computed via visual-textual similarity with the learned low-end embeddings, bypassing low-end visual training entirely. High-end Supervised Training. During training, we utilize labeled images exclusively from the high-end modality HX_H. Given an image IH∈HI_H _H along with its label y, the visual embedding is extracted with the visual encoder: xH=ϕ(IH)x_H=φ(I_H). We optimize the high-end learnable prompts i,Ht_i,H by minimizing the standard cross-entropy loss: ℒce=−1N∑IH∈Hlogp(y|xH,wi,H)=−1N∑IH∈Hlogexp(d(xH,wy,H)/τ)∑i=1Ncexp(d(xH,wi,H)/τ)L_ce=- 1N _I_H _H p(y|x_H,w_i,H)=- 1N _I_H _H (d(x_H,w_y,H)/τ) _i=1^N_c (d(x_H,w_i,H)/τ) where τ is a predefined temperature parameter, N is the number of images, and d(⋅)d(·) is the cosine similarity between the visual embedding xHx_H and the textual embeddings wi,Hw_i,H. This objective updates the high-end learnable prompts, explicitly injecting high-end visual knowledge into the textual embedding space. Space Anchoring to Modality-Specific Lexical Prompts. While learnable prompts capture discriminative task-specific knowledge, they risk collapsing into modality-specific statistics and forgetting the essential general textual knowledge. To alleviate this, we define general textual knowledge using a fixed ensemble of visual descriptions. To reduce the reliance on manual domain expertise while providing precise prior knowledge, we employ an LLM to automatically generate these descriptions. Let pi,mjp_i,m^j denote the j-th LLM-generated visual description for the i-th class under modality m, where j∈1,…,Npj∈\1,…,N_p\ and NpN_p is the number of descriptions per class. We feed these generated descriptions into the textual encoder to generate the general textual prototype embedding, p¯i,m=1Np∑j=1Npθ(pi,mj) p_i,m= 1N_p _j=1^N_pθ(p_i,m^j), which remains fixed during training. To ensure the learnable specific textual knowledge acts as a semantic anchor and does not deviate from clinically meaningful semantics, we introduce a space anchoring constraint that minimizes the discrepancy (Euclidean distance) between the general textual prototypes and the learnable specific textual embeddings: ℒanc=12Nc∑i=1Nc∑m∈H,L‖wi,m−p¯i,m‖22L_anc= 12N_c _i=1^N_c _m∈\H,L\\|w_i,m- p_i,m\|_2^2 As the class and modality tokens are fixed, gradients from ℒancL_anc solely update the shared and modality-specific learnable tokens. Cross-modal Manifold Alignment via FGW. To further prevent the decision boundary from collapsing into high-end specific statistics, we propose a novel alignment objective based on the FGW optimal transport [11]. FGW acts as a structural regularizer, enforcing the learned low-end prompt manifold to strictly mirror the relational structure of the high-end prompt manifold anchored by clinical text knowledge. To explicitly focus this alignment on modality-specific structure, we treat the high-end textual embeddings as a fixed reference manifold. Let w~i,H w_i,H and w~j,L w_j,L denote the textual embeddings for the i-th class in the high-end modality and the j-th class in the low-end modality, where i,j∈1,…,Nci,j∈\1,…,N_c\. We first compute the intra-modal structural distance matrices DH,DL∈ℝNc×NcD^H,D^L ^N_c× N_c with elements, DikH=‖w~i,H−w~k,H‖2,DjlL=‖w~j,L−w~l,L‖2D^H_ik=\| w_i,H- w_k,H\|_2, D^L_jl=\| w_j,L- w_l,L\|_2, where indices i,ki,k iterate over the high-end classes and j,lj,l iterate over the low-end classes. Alongside this, we compute the cross-modal feature cost matrix M∈ℝNc×NcM ^N_c× N_c with elements Mij=‖w~i,H−w~j,L‖22M_ij=\| w_i,H- w_j,L\|_2^2. FGW seeks an optimal coupling matrix Γ∈ℝ+Nc×Nc _+^N_c× N_c to jointly align the feature representations and the relational geometric structure. The corresponding constraint is formulated as: ℒfgw=(1−α)∑i,j=1NcMijΓij+α∑i,j,k,l=1Nc|DikH−DjlL|2ΓijΓklL_fgw=(1-α) _i,j=1^N_cM_ij _ij+α _i,j,k,l=1^N_c|D^H_ik-D^L_jl|^2 _ij _kl where α∈[0,1]α∈[0,1] is the trade-off parameter balancing the Wasserstein feature alignment and the Gromov-Wasserstein structural alignment. For FGW, we detach high-end embeddings and backpropagate only into low-end prompt embeddings. Intuitively, ℒFGWL_FGW ensures the low-end prompt manifold preserves the shared diagnostic semantics anchored by the fixed high-end space, entirely bypassing the need for low-end visual training data. Overall Objective and Inference. The final objective combines cross-entropy loss with knowledge anchoring and manifold alignment constraints: ℒ=ℒce+λancℒanc+λfgwℒfgwL=L_ce+ _ancL_anc+ _fgwL_fgw where λanc _anc and λfgw _fgw balance the effect of the respective regularization terms. During inference, the learned low-end embeddings wi,Lw_i,L are used to classify unseen images IL∈LI_L _L by computing the visual-textual similarity p(y|xL,wi,L),xL=ϕ(IL)p(y|x_L,w_i,L),\\ x_L=φ(I_L) with frozen encoders. During this phase, all visual and textual backbone parameters remain strictly frozen. 3 Experiments and Results 3.1 Experimental Settings Datasets and Tasks. We evaluate our proposed K-MaT on 4 diverse cross-modal medical imaging pairs covering various clinical tasks. Specifically, the evaluation encompasses: (1) skin lesion classification transferring from Dermoscopy to Clinical Images (DERM7PT) [4]; (2) skin lesion classification transferring from Dermoscopy to 15cm Clinical Images (MRA-MIDAS) [2]; (3) breast lesion classification from Mammography [8] to Ultrasound [1]; and (4) COVID-19 pneumonia classification from CT [14] to Chest X-ray [9]. We also use GPT-5 to generate Np=50N_p=50 visual descriptions for each class and modality. Table 1 summarizes the dataset statistics and domain characteristics.. Implementation Details. We adopt the pre-trained BiomedCLIP as our frozen backbone. For prompt tuning, we randomly initialize a learnable context of length 4. To ensure effective transfer and mitigate interference between modality classes, we introduce Class-Specific Context (CSC) and Modality-Specific Context (MSC), learning distinct prompts for each class across different modalities. The model is optimized for 50 epochs (batch size 4) using the SGD optimizer with a 0.0025 base learning rate, a cosine annealing scheduler, and a 1-epoch warmup. To compute the FGW optimal coupling Γ , we use the Sinkhorn-Knopp algorithm [11] (temperature 0.1, max 100 iterations) and set the trade-off α to 0.1. All experiments use a 16-shot setting per class, with results averaged over 3 random seeds. Hyperparameters (λanc _anc, λfgw _fgw) are tuned exclusively on the high-end validation set, leaving low-end images strictly for test-time evaluation. We run all experiments on a single NVIDIA A100 (80GB) GPU. Table 1: Dataset statistics for the five cross-modal tasks. Task Type Dataset Modality Classes Train Val Test Skin I High-end MRA-MIDAS [2] Dermoscopy 6 513 91 135 Low-end MRA-MIDAS [2] Clinical 6 501 89 132 Skin I High-end Derm7pt [4] Dermoscopy 5 413 203 395 Low-end Derm7pt [4] Clinical 5 413 203 395 Chest High-end C-CCII [14] CT 3 10670 3538 1678 Low-end ChestXray [9] X-Ray 3 4595 923 914 Breast High-end DMID [8] Mammography 3 356 76 78 Low-end BUSI [1] Ultrasound 3 389 155 236 Table 2: Cross-modal generalization performance on 4 medical image pairs with ACC and macro-F1 shown together. Each entry is reported as average ACC (F1) over 3 seeds. Hi: High-End (source), Lo: Low-End (target), H: Harmonic Mean. Best ACC/F1 results are highlighted in bold/underline respectively. Dataset BiomedCLIP CoOp KgCoOp CoCoOp BiomedCoOp K-MaT Skin I Hi 34.6 (24.2) 55.5 (42.3) 56.6 (41.6) 44.6 (32.0) 58.3 (39.3) 56.8 (43.1) Lo 31.7 (21.0) 40.8 (27.1) 40.5 (28.5) 29.8 (24.2) 42.2 (29.1) 46.4 (27.4) H 37.9 (22.9) 47.0 (33.0) 47.2 (33.9) 35.7 (27.5) 49.0 (33.4) 50.9 (33.3) Skin I Hi 32.6 (26.9) 34.8 (32.1) 34.6 (32.7) 40.7 (36.7) 35.6 (31.8) 34.8 (32.8) Lo 17.4 (13.6) 28.5 (26.4) 28.5 (27.3) 18.4 (18.0) 27.0 (24.6) 31.6 (25.0) H 22.7 (18.1) 31.4 (29.0) 31.3 (29.8) 25.4 (24.2) 30.7 (27.7) 33.1 (28.3) Chest Hi 34.0 (24.6) 52.0 (47.8) 47.1 (42.3) 37.4 (36.3) 43.4 (40.9) 54.5 (50.8) Lo 29.9 (32.1) 26.4 (21.8) 26.1 (22.7) 54.5 (35.6) 38.7 (29.2) 40.6 (31.1) H 31.8 (27.9) 35.0 (30.0) 33.6 (29.5) 44.4 (35.9) 40.9 (34.0) 42.3 (35.4) Breast Hi 24.4 (19.9) 75.2 (72.8) 70.5 (66.6) 65.0 (59.5) 69.2 (65.0) 73.1 (70.7) Lo 57.2 (45.2) 27.0 (19.0) 28.0 (16.9) 28.3 (26.6) 36.0 (34.1) 38.4 (36.3) H 34.2 (27.7) 39.7 (30.2) 40.1 (27.0) 39.4 (36.8) 47.4 (44.8) 50.3 (47.9) Avg. Hi 34.6 (24.2) 54.4 (48.7) 52.2 (45.8) 46.9 (41.1) 51.6 (44.2) 54.8 (49.3) Lo 34.0 (28.0) 30.7 (23.6) 30.8 (23.9) 32.7 (26.1) 36.0 (29.2) 39.3 (29.9) H 31.7 (24.1) 38.3 (30.5) 38.0 (30.0) 36.2 (31.1) 42.0 (35.0) 44.1 (36.2) 3.2 Experiments on Cross-modal Generalization We compare our proposed K-MaT with BiomedCLIP and the state-of-the-art (SOTA) prompt tuning methods equipped with BiomedCLIP backbone, including CoOp [17], CoCoOp [16], KgCoOp [12], and BiomedCoOp [5].Table 2 details the cross-modal generalization performance in terms of Accuracy and Macro-F1 scores, respectively. Across the four diverse medical image pairs, K-MaT consistently outperforms all competitive baselines, achieving the highest average Harmonic Mean (H) of 44.1% for Accuracy and 36.2% for F1 score. H is a critical evaluation metric in our asymmetric setting, as it penalizes extreme performance disparities and accurately reflects the model’s ability to retain knowledge in the source high-end modality while effectively generalizing diagnostic semantics to the unseen low-end target modality. Traditional prompt learning algorithms, such as CoOp and CoCoOp, demonstrate substantial improvements on the high-end source modalities but suffer from severe catastrophic forgetting when evaluated on the low-end target modalities. For instance, in the breast dataset task (transferring from Mammography to Ultrasound), CoOp achieves a high-end accuracy of 75.2% but plummets to 27.0% on the Low-end target. Similarly, CoCoOp yields 65.0% on the High-end but only 28.2% on the Low-end. These methods tend to overfit to the distinct appearance statistics of the source domain, adopting modality-specific shortcuts rather than shared clinical semantics. K-MaT directly addresses this vulnerability; by preserving structural relationships, it prevents the decision boundary from collapsing, boosting the Low-end target accuracy to 38.4% on the Breast dataset and achieving a leading H of 50.3%. Even when compared to BiomedCoOp, a robust recent baseline tailored specifically for biomedical VLMs, K-MaT exhibits superior generalization. Across all four benchmarks, K-MaT surpasses BiomedCoOp by a 2.1% margin in average H Accuracy (44.1% vs. 42.0%) and a 1.2% margin in average H F1 (36.2% vs. 35.0%). This demonstrates that while integrating general biomedical knowledge is beneficial, it is insufficient on its own to navigate severe cross-modal domain shifts. By explicitly aligning the relational geometry of the prompt manifold through optimal transport, K-MaT effectively distills shared clinical knowledge, successfully achieving competitive zero-shot generalization. 3.3 Ablation Study To evaluate the contribution of each component, we conduct an ablation study to assess the impact of our prompt context designs (CSC and MSC), the space-anchoring loss ℒancL_anc, and the FGW alignment loss ℒfgwL_fgw. Table 3 presents these architectural variations across High-end, Low-end, and H metrics. Table 3: Complete ablation study results across all dataset pairs. Relative improvement (%) is computed on the Harmonic Mean ACC/F1 with respect to the first row. CSC MSC ℒancL_anc ℒfgwL_fgw High-end Low-end Harmonic Mean Rel. Impr. (%) ACC F1 ACC F1 ACC F1 ACC F1 53.36 47.53 35.27 25.72 40.08 31.85 0.00 0.00 ✓ 54.02 48.49 27.55 22.71 35.26 30.03 -12.03 -5.71 ✓ ✓ 54.69 49.17 34.73 18.68 37.95 25.83 -5.31 -18.90 ✓ ✓ ✓ 54.80 49.34 30.95 20.73 35.15 27.82 -12.30 -12.65 ✓ ✓ ✓ 54.69 49.17 36.99 26.58 41.94 32.65 4.64 2.51 ✓ ✓ ✓ 50.68 44.70 30.07 24.24 36.67 30.67 -8.51 -3.70 ✓ ✓ ✓ 53.56 48.07 28.18 22.94 35.92 30.29 -10.38 -4.90 ✓ ✓ ✓ ✓ 54.80 49.34 39.25 29.92 44.13 36.23 10.10 13.75 Effectiveness of Context Modularization. The baseline configuration (optimizing solely with cross-entropy ℒceL_ce) exhibits restricted generalization. It heavily biases predictions toward the high-end, confirming that unconstrained learnable prompts risk collapsing into modality-specific shortcuts. Rows 2 and 3 in Tab. 3 show that the combination of CSC and MSC can mitigate interference between modality classes and ensure effective cross-modal transfer with the relative improvement from -12.03% to -5.31% of Acc. The row 6 and 7 indicate that MSC could improve cross-modal performance (36.67% Acc) but affect the Acc of high-end. While CSC slightly enhances high-end performance, it disrupts the equilibrium between high-end and low-end modalities. Effectiveness of Semantic Anchoring and Manifold Alignment. Introducing ℒancL_anc provides a critical regularizing effect by anchoring learnable embeddings to LLM-generated textual prototypes, preventing semantic divergence and improving the H accuracy to 41.94% (row 5). Building on this, ℒfgwL_fgw acts as a powerful structural regularizer. While ℒancL_anc maintains semantic fidelity, ℒfgwL_fgw explicitly enforces the low-end prompt manifold to mirror the relational geometry of the high-end space. As shown in Table 3, the complete K-MaT framework—combining CSC, MSC, ℒancL_anc, and ℒfgwL_fgw—achieves the highest overall performance. This full configuration yields a significant 13.75% relative improvement in H F1 and a 10.10% boost in H accuracy over the baseline (Row 8). Effectiveness of FGW. To further investigate the role of ℒfgwL_fgw, we perform a t-SNE visualization and a breakdown analysis on the chest cross-modal generalization task. Specifically, we compare variants of K-MaT excluding ℒancL_anc with and without the addition of ℒfgwL_fgw. As observed in Fig. 2(a), high-end text embeddings wi,Hw_i,H act as anchors due to reliable supervision from labels and visual signals (blue circles and green triangles). By incorporating ℒfgwL_fgw, the low-end embeddings wi,Lw_i,L tend to preserve the relative structure of their high-end counterparts (yellow squares). In contrast, text embeddings learned without ℒfgwL_fgw fail to generate discriminative representations (red triangles). This is further confirmed by the breakdown analysis in Fig. 2(b): ℒfgwL_fgw prevents the model from collapsing into a single class, improving average performance on the low-end modality through structural transport from the high-end space. Figure 2: (a). Visualization of textual embeddings w/ and w/o ℒfgwL_fgw on the Chest cross-modal generalization task. (b). The breakdown of Accuracy for each class. 4 Conclusion We proposed K-MaT, a zero-shot prompt learning framework for asymmetric cross-modal transfer in medical imaging. By factorizing prompts, anchoring them to LLM-generated clinical text, and aligning manifolds via FGW optimal transport, K-MaT effectively mitigates catastrophic forgetting and achieves SOTA harmonic mean performance across four benchmarks. Despite these promising results, our framework exhibits certain limitations. First, its absolute performance on low-end modalities shows limited improvement over the zero-shot BiomedCLIP baseline. Second, the framework is sensitive to dataset characteristics; severe visual discrepancies between modalities can create visual-textual gaps that purely text-anchored alignment cannot fully bridge. Future work will explore incorporating more reliable visual signals to enhance the stability and low-end transfer capabilities of the framework. credits 4.0.1 Acknowledgements J. Zeng is funded by the China Scholarship Council (CSC) program (grant No. 202508440222). 4.0.2 The authors have no competing interests. References [1] W. Al-Dhabyani, M. Gomaa, H. Khaled, and A. Fahmy (2020) Dataset of breast ultrasound images. Data in brief 28, p. 104863. Cited by: §3.1, Table 1. [2] A. S. Chiou, J. A. Omiye, H. Gui, S. M. Swetter, J. M. Ko, B. Gastman, J. Arbesman, Z. R. Cai, O. Gevaert, C. Sadée, et al. (2025) Multimodal image dataset for ai-based skin cancer (midas) benchmarking. NEJM AI 2 (6), p. AIdbp2400732. Cited by: §3.1, Table 1, Table 1. [3] I. Gulrajani and D. Lopez-Paz (2020) In search of lost domain generalization. arXiv preprint arXiv:2007.01434. Cited by: §1. [4] J. Kawahara, S. Daneshvar, G. Argenziano, and G. Hamarneh (2019-03) Seven-point checklist and skin lesion classification using multitask multimodal neural nets. IEEE Journal of Biomedical and Health Informatics 23 (2), p. 538–546. External Links: Document, ISSN 2168-2194 Cited by: §3.1, Table 1, Table 1. [5] T. Koleilat, H. Asgariandehkordi, H. Rivaz, and Y. Xiao (2025) Biomedcoop: learning to prompt for biomedical vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 14766–14776. Cited by: §1, §3.2. [6] W. Liu, F. Zhu, L. Wei, and Q. Tian (2025) C-clip: multimodal continual learning for vision-language model. In The Thirteenth International Conference on Learning Representations, Cited by: §1. [7] S. Matta, M. Lamard, P. Zhang, A. Le Guilcher, L. Borderie, B. Cochener, and G. Quellec (2024) A systematic review of generalization research in medical image classification. Computers in biology and medicine 183, p. 109256. Cited by: §1. [8] P. Oza, U. Oza, R. Oza, P. Sharma, S. Patel, P. Kumar, and B. Gohel (2024) Digital mammography dataset for breast cancer diagnosis research (dmid) with breast mass segmentation analysis. Biomedical Engineering Letters 14 (2), p. 317–330. Cited by: §3.1, Table 1. [9] P. Patel (2021) Chest X-ray (Covid-19 & pneumonia). Kaggle. Note: https://w.kaggle.com/datasets/prashant268/chest-xray-covid19-pneumoniaAccessed: 2026-02-20 Cited by: §3.1, Table 1. [10] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §1. [11] V. Titouan, N. Courty, R. Tavenard, and R. Flamary (2019) Optimal transport for structured data with application on graphs. In International Conference on Machine Learning, p. 6275–6284. Cited by: §1, §2, §3.1. [12] H. Yao, R. Zhang, and C. Xu (2023) Visual-language prompt tuning with knowledge-guided context optimization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 6757–6767. Cited by: §1, §3.2. [13] J. S. Yoon, K. Oh, Y. Shin, M. A. Mazurowski, and H. Suk (2024) Domain generalization for medical image analysis: a review. Proceedings of the IEEE. Cited by: §1. [14] K. Zhang, X. Liu, J. Shen, Z. Li, Y. Sang, X. Wu, Y. Zha, W. Liang, C. Wang, K. Wang, et al. (2020) Clinically applicable ai system for accurate diagnosis, quantitative measurements, and prognosis of covid-19 pneumonia using computed tomography. Cell 181 (6), p. 1423–1433. Cited by: §3.1, Table 1. [15] S. Zhang, Y. Xu, N. Usuyama, H. Xu, J. Bagga, R. Tinn, S. Preston, R. Rao, M. Wei, N. Valluri, et al. (2023) Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.00915. Cited by: §1, §1, §2. [16] K. Zhou, J. Yang, C. C. Loy, and Z. Liu (2022) Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 16816–16825. Cited by: §3.2. [17] K. Zhou, J. Yang, C. C. Loy, and Z. Liu (2022) Learning to prompt for vision-language models. International Journal of Computer Vision 130 (9), p. 2337–2348. Cited by: §1, §3.2.