Paper deep dive

HarMoE: Multi-Source Chest Radiograph Pretraining with Dataset-Disentangled Experts

Haozhe Luo, Ziyu Zhou, Shelley Zixin Shu, Mauricio Reyes

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 57

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Recent vision-language models for chest X-ray understanding are largely built on image-report alignment and therefore rely heavily on MIMIC-CXR as the dominant pretraining source. While effective at scale, this paradigm underexplores an important alternative source of supervision: a range of existing multi-label classification datasets, which provide cleaner and more explicit disease signals than free-text reports, and can offer broader pathology coverage when combined across sources. However, learning from such heterogeneous datasets is nontrivial, as differences in label ontologies, annotation protocols, acquisition pipelines, and report styles can cause models to entangle clinical semantics with dataset identity, leading to poor transfer despite increased scale. In this work, we revisit radiology VLM construction from the perspective of harmonized multi-source learning. We propose HarMoE, a dataset-aware mixture-of-experts framework that learns shared cross-dataset medical semantics while confining source-specific variation to lightweight residual experts in deeper decoder layers. To further exploit clean supervision from labeled datasets, we train in a unified disease vocabulary with masked multi-dataset supervision, enabling the model to leverage complementary annotations without introducing false negatives. Experiments on large-scale chest X-ray benchmarks show that HarMoE consistently improves zero-shot classification, out-of-distribution transfer, and grounding over strong baselines. Our results suggest that building robust radiology VLMs requires moving beyond single-source image-report alignment toward structured knowledge construction from heterogeneous datasets with cleaner supervision and broader coverage. Code and the 873k harmonized dataset will be released at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

55,421 characters extracted from source content.

Expand or collapse full text

by HarMoE: Multi-Source Chest Radiograph Pretraining with Dataset-Disentangled Experts Haozhe Luo 0009-0009-8021-2073 haozhe.luo@unibe.ch ARTORG Center for Biomedical Engineering Research, University of BernBernSwitzerland Kaiko.AIZurichSwitzerland , Ziyu Zhou 0009-0000-9805-3546 zhouziyu@sjtu.edu.cn Shanghai Jiao Tong UniversityShanghaiChina , Shelley Zixin Shu 0009-0008-0392-7924 zixin.shu@unibe.ch ARTORG Center for Biomedical Engineering Research, University of BernBernSwitzerland and Mauricio Reyes 0000-0002-2434-9990 mauricio.reyes3@unibe.ch ARTORG Center for Biomedical Engineering Research, University of BernBernSwitzerland (2026) Abstract. Recent vision-language models for chest X-ray understanding are largely built on image-report alignment and therefore rely heavily on MIMIC-CXR as the dominant pretraining source. While effective at scale, this paradigm underexplores an important alternative source of supervision: a range of existing multi-label classification datasets, which provide cleaner and more explicit disease signals than free-text reports, and can offer broader pathology coverage when combined across sources. However, learning from such heterogeneous datasets is nontrivial, as differences in label ontologies, annotation protocols, acquisition pipelines, and report styles can cause models to entangle clinical semantics with dataset identity, leading to poor transfer despite increased scale. In this work, we revisit radiology VLM construction from the perspective of harmonized multi-source learning. We propose HarMoE, a dataset-aware mixture-of-experts framework that learns shared cross-dataset medical semantics while confining source-specific variation to lightweight residual experts in deeper decoder layers. To further exploit clean supervision from labeled datasets, we train in a unified disease vocabulary with masked multi-dataset supervision, enabling the model to leverage complementary annotations without introducing false negatives. Experiments on large-scale chest X-ray benchmarks show that HarMoE consistently improves zero-shot classification, out-of-distribution transfer, and grounding over strong baselines. Our results suggest that building robust radiology VLMs requires moving beyond single-source image-report alignment toward structured knowledge construction from heterogeneous datasets with cleaner supervision and broader coverage. Code and the 873k harmonized dataset will be released at https://github.com/Roypic/harmoe. Vision Language Pretraining, Zero-shot Diagnosis, Heterogeneous Data †journalyear: 2026†copyright: c†conference: Proceedings of the 34th ACM International Conference on Multimedia; November 10–14, 2026; Rio de Janeiro, Brazil†booktitle: Proceedings of the 34th ACM International Conference on Multimedia (M ’26), November 10–14, 2026, Rio de Janeiro, Brazil†doi: 10.1145/3767308.3835205†isbn: 979-8-4007-2213-4/2026/11†submissionid: 1764†ccs: Computing methodologies Computer vision 1. Introduction Chest radiography is among the most frequently performed diagnostic imaging procedures worldwide, yet its interpretation remains labor-intensive and subject to substantial inter-observer variability (Gündel et al., 2021; Clark and Conrad, 1995). This bottleneck has driven sustained interest in vision-language pretraining (VLP): by aligning chest X-ray images with radiology reports through contrastive learning, methods such as ConVIRT (Zhang et al., 2022), GLoRIA (Huang et al., 2021), KAD (Zhang et al., 2023b), and MedKLIP (Wu et al., 2023) enable zero-shot classification without manual annotation. More recently, CarZero (Lai et al., 2024) and RadZero (Park et al., 2025) have shown that classification-level prompts yield cleaner supervision while retaining semantic grounding. Despite this diversity, the field remains constrained by a shared dependency on a single institutional source, predominantly MIMIC-CXR (Johnson et al., 2019) (377K images, 43 conditions) as shown in Fig. 1 (a). This single-source dependency is limiting because data scale and diversity are among the strongest predictors of representation quality. CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) demonstrated robust zero-shot transfer from massive image-text corpora, with subsequent work confirming power-law scaling with data size (Cherti et al., 2023). Radiology cannot replicate this directly: datasets are institutionally siloed, and the majority of large-scale chest X-ray collections discard free-text reports in favor of structured labels. ChestX-ray14 (Wang et al., 2017) (112K), CheXpert (Irvin et al., 2019) (224K), and PadChest (Bustos et al., 2020) (160K) are inaccessible to standard contrastive VLP pipelines despite their clinical value. Combining them with report-bearing corpora would more than double the pretraining pool, from 377K to 873K images, while expanding pathology coverage from 43 to 229 conditions (Fig. 1 (b)). The technical barrier to such aggregation is not data access but representation quality under multi-source supervision. Each dataset reflects a distinct combination of acquisition hardware, patient demographics, and annotation protocol (Oakden-Rayner et al., 2020; Zech et al., 2018); naively pooling sources induces spurious correlations between dataset identity and diagnostic labels (DeGrave et al., 2021), and treating unannotated classes as negative introduces systematic false negatives that distort the learned decision boundary (Bekker and Davis, 2020). These failure modes are well-studied in multi-domain learning (Geirhos et al., 2020), but existing VLP methods do not address them directly, as they operate within single-source pipelines where they do not arise by construction. We approach this as a multi-source supervised pretraining problem with partial labels and heterogeneous domain shifts, and hypothesize that effective multi-source learning requires architecturally separating what is shared (pathology-relevant features that generalize across institutions) from what is source-specific (acquisition and annotation artifacts), rather than leaving this separation to emerge from data alone. One could instead pursue domain-adversarial alignment (Ganin et al., 2016) or naive joint training, but both are suboptimal: adversarial alignment risks removing clinically relevant variation (e.g., disease prevalence differences reflecting genuine epidemiological variation), while naive joint training encodes source-specific shortcuts that inflate in-domain scores at the cost of transferability. HarMoE takes a middle path: a shared decoder captures cross-dataset pathology semantics, augmented by lightweight low-rank residual modules deterministically routed by dataset identity, reflecting that distributional shifts in multi-site imaging are systematic and institutional rather than instance-level. At inference, the residual modules are discarded; the training objective ensures the shared pathway alone carries sufficient diagnostic signal. Class-level supervision sacrifices the compositional richness of free-text reports, but is more directly aligned with the zero-shot multi-label evaluation objective and, crucially, unlocks three datasets providing 496K additional images and 186 additional disease categories inaccessible to any existing VLP method. The prompts retain sufficient semantic structure for the text encoder to generalize to unseen disease categories, as we demonstrate empirically. Our contributions are as follows: (i) We formalize multi-source chest X-ray pretraining under a unified disease vocabulary of 229 classes, constructed by harmonizing four major datasets (MIMIC-CXR, CheXpert, ChestX-ray14, PadChest) into a shared label space with tri-state encoding (11, 0, −1-1 for present, absent, unknown) and masked supervision to prevent false negatives from unannotated entries. (i) We propose HarMoE, an architecture that separates shared pathology representations from dataset-specific residuals via constrained low-rank expert modules deterministically routed by dataset identity and discarded at inference, enabling multi-source scaling from 377K to 873K images without encoding source-dependent confounds into the transferable representation. (i) We evaluate on 11 benchmarks spanning in-domain and out-of-distribution settings and demonstrate consistent improvements in zero-shot classification, with the largest gains on out-of-distribution benchmarks: +2.9%+2.9\% AUC on RSNA, +4.6%+4.6\% AUC on COVID-QU-Ex, and +4.1%+4.1\% AUC on Montgomery. We additionally report improvements in zero-shot visual grounding without grounding-specific supervision. 2. Related Work Vision-Language Pretraining for Chest X-rays. Contrastive alignment between chest X-ray images and radiology reports has become the dominant pretraining paradigm. Early methods established global alignment (ConVIRT (Zhang et al., 2022)) and local region-word matching (GLoRIA (Huang et al., 2021)), while BioViL (Boecking et al., 2022) and BioViL-T (Bannur et al., 2023) extended this with token-level and temporal modeling. Knowledge-enhanced methods such as KAD (Zhang et al., 2023b) and MedKLIP (Wu et al., 2023) improve grounding through structured entity extraction. More recently, CarZero (Lai et al., 2024), RadZero (Park et al., 2025), and DeViDe (Luo et al., 2025a) move toward classification-level prompts or multi-attribute decomposition, while KEPIL (Luo et al., 2026a) explicitly improves robustness to prompt variation through ontology-guided knowledge enrichment. MAVL (Phan et al., 2024) decomposes visual features into attribute-specific components for finer-grained recognition. Complementary self-supervised approaches learn anatomy-aware chest X-ray representations through consistent embedding (Zhou et al., 2023), compositional decomposition (Zhou et al., 2025a), or multi-perspective anatomical constraints (Zhou et al., 2025b). Beyond recognition, DWARF (Luo et al., 2024) and hybrid explanation-guided learning (Shu et al., 2025) refine diagnostic attention using expert or self-supervised guidance, while XBench (Luo et al., 2026b) systematically evaluates visual-language explanations in chest radiography. These advances improve prompt robustness, anatomical representation, or grounding, but remain centered on single-source training. HarMoE instead exploits complementary supervision across heterogeneous classification-only datasets. Multi-Domain Learning and Dataset Bias in Medical Imaging. Distribution shifts across medical imaging datasets are well documented (Oakden-Rayner et al., 2020; Zech et al., 2018) and known to produce spurious correlations between source identity and diagnostic labels (DeGrave et al., 2021; Geirhos et al., 2020). Existing responses include domain-adversarial training (Ganin et al., 2016), invariant risk minimization (Arjovsky et al., 2019), and adapter-based approaches such as LoRA (Hu et al., 2022) and residual adapters (Rebuffi et al., 2017); partial-label learning (Bekker and Davis, 2020) further addresses missing annotations in multi-label settings. Human-guided representation alignment has likewise been shown to affect both out-of-distribution generalization and demographic fairness in medical imaging (Luo et al., 2025b). HarMoE draws on these ideas but makes a distinct choice: rather than adversarially removing domain information, which risks discarding clinically relevant variation, it deterministically assigns lightweight low-rank residual experts by dataset identity, confining source-specific variation to a controlled pathway that is discarded at inference, thereby preserving cross-dataset knowledge while preventing domain confounds from contaminating the transferable representation. Mixture-of-Experts Architectures. Sparse mixture-of-experts models, in which a learned router activates a subset of expert modules per input, have proven effective for scaling model capacity in language modeling (Fedus et al., 2022) and vision (Riquelme et al., 2021). In these systems, routing is input-conditioned and learned end-to-end. Our design departs from this convention: HarMoE routes deterministically by dataset identity, reflecting the fact that the dominant distributional shifts in multi-site medical imaging are systematic and institutional rather than instance-level. Each expert is a low-rank residual (LoRA) injected into the shared decoder, adding minimal parameters. This is closer in spirit to multi-domain adapters (Rebuffi et al., 2017; Hu et al., 2022) than to classical sparse MoE, and we use the term “mixture of experts” to denote the parallel expert bank and routing mechanism, not input-conditioned sparse gating. A two-panel schematic contrasts single-source image-report pretraining with HarMoE multi-source pretraining over four chest X-ray datasets and a unified 229-class disease vocabulary. Figure 1. Single-source vs. multi-source pretraining for chest X-ray understanding. (a) Existing VLP methods depend on paired image-report data from a single source (e.g., MIMIC-CXR: 377K images, 43 conditions). (b) HarMoE harmonizes four heterogeneous datasets into a unified 229-class vocabulary, scaling to 873K images while accommodating sources with and without free-text reports. Pipeline diagram of label harmonization, shared and dataset-specific expert paths, masked training losses, and shared-only inference in HarMoE. Figure 2. Overview of HarMoE. (a) Annotations from K datasets are unified into a 229-class vocabulary via LLM-based extraction and ontology alignment, producing tri-state labels (11/0/−1-1: present/absent/unknown). (b) Disease prompts and visual tokens interact in N decoder layers, each containing a shared path for dataset-invariant semantics and a dataset-routed expert path with low-rank (LoRA) residuals selected by dataset identity. The full representation f=s+dg_f=g_s+g_d is used during training; at inference, the expert path is discarded and only sg_s drives prediction. The model is trained with masked classification (ℒbceL_bce), orthogonality (ℒorthL_orth), and expert regularization (ℒregL_reg). 3. Method We present HarMoE, a framework for multi-source supervised pretraining over heterogeneous chest X-ray datasets with partial and inconsistent labels. The method has three components: (1) multi-source label harmonization that unifies heterogeneous annotations into a shared disease vocabulary; (2) prompt-based multimodal encoding that replaces free-text reports with stable disease-level text prompts, enabling the inclusion of classification-only datasets; and (3) dataset bias disentanglement block (DA-MoE) that separates shared pathology representations from source-specific residuals via deterministic expert routing, with the experts discarded at inference to yield a domain-general representation. We describe each component below and conclude with the training objective. 3.1. Multi-Source Label Harmonization The four pretraining datasets use incompatible annotation schemes: MIMIC-CXR (Johnson et al., 2019) provides free-text reports, CheXpert (Irvin et al., 2019) uses a 14-class ontology with uncertainty labels, ChestX-ray14 (Wang et al., 2017) provides 14 NLP-extracted binary labels, and PadChest (Bustos et al., 2020) annotates 174 findings. As shown in Fig. 2 (a), we harmonize multi-source datasets in three steps. First, for datasets with free-text reports, we apply an LLM to extract per-finding binary labels. Second, we construct a global vocabulary C of ||=229|C|=229 conditions by taking the union of all dataset-specific label sets after resolving synonyms. Third, each sample is assigned a tri-state encoding label vector ∈−1,0,1||y∈\-1,0,1\^|C|, where 11, 0, and −1-1 denote present, absent, and unknown respectively; unknown entries are excluded from the loss rather than treated as negatives, preventing the false negatives when projecting sparse, dataset-specific annotations into a significantly expanded global vocabulary. 3.2. Prompt-Based Multimodal Encoding Building on the harmonized label space, we encode each disease concept as a fixed text embedding, bypassing the variance of free-text reports. As shown in Fig. 2 (b), for each condition c∈c , we generate a prompt (“There is c.”) and encode it with a frozen clinical text encoder to obtain t∈ℝ||×DF_t ^|C|× D. Simultaneously, a vision backbone extracts visual tokens v∈ℝP×DF_v ^P× D from the input radiograph, where P and D denote the number of patches and embedding dimension. This prompt-based interface provides a unified encoding for datasets with and without reports alike, and supports zero-shot generalization to unseen disease categories without architectural modification. 3.3. Dataset Bias Disentanglement The central architectural contribution is a decoder that separates shared pathology representations from source-specific variation through two parallel paths: a shared path that captures dataset-invariant cross-modal semantics, and a dataset-routed expert path that absorbs source-dependent distributional differences via lightweight low-rank residual modules. The decoder consists of N cascaded layers. At inference time, the expert path is discarded entirely, and only the shared representation is used for prediction. This is not a post-hoc simplification: the training objective (Section 3.4) is specifically designed so that the shared path alone carries sufficient diagnostic signal, while the expert path serves as a controlled absorber of domain confounds during optimization. Shared Path. The shared path performs cross-attention between disease queries and visual tokens to distill pathology-relevant features that transfer across all sources. At the first layer (l=1l=1), queries are derived from the text prompt embeddings tF_t and cross-attend to visual tokens vF_v. For subsequent layers (l>1l>1), queries are updated from the previous shared output s(l−1)F_s^(l-1), while keys and values remain the visual tokens vF_v throughout: (1) s(l)=Attn(s(l−1)q,vk,vv)o,F_s^(l)=Attn\! (F_s^(l-1)W_q,\;F_vW_k,\;F_vW_v )W_o, where s(0)=tF_s^(0)=F_t, and q,k,v,oW_q,W_k,W_v,W_o are learnable projection matrices shared across all datasets. This path is trained on data from all sources and captures the dataset-invariant component of the cross-modal alignment. Dataset-Routed Expert Path. In parallel, a dataset-routed path injects low-rank residual modules into the attention projections to model source-specific distributional differences. The design rationale is as follows: by modulating the query, key, value, and output projections of cross-attention, the expert modules can shift what the model attends to and how it aggregates information in a dataset-specific manner, without altering the shared feature space itself. Specifically, a dataset-ID router receives the source indicator kD_k and deterministically selects the corresponding expert from a bank of K experts. Routing is deterministic (not learned or input-conditioned) because the dominant distributional shifts across medical imaging datasets are systematic and institutional (scanner hardware, annotation conventions, patient demographics) rather than instance-level. For a given input z and attention projection m∈q,k,v,om∈\q,k,v,o\, the expert-augmented projection is ~m=m+α⋅m(k)m(k) z_m=zW_m+α·zA_m^(k)B_m^(k), where mW_m is the shared projection (identical to the shared path), m(k)∈ℝD×rA_m^(k) ^D× r and m(k)∈ℝr×DB_m^(k) ^r× D are the low-rank expert parameters for dataset k, r≪Dr D is the expert rank, and α controls the residual strength. Each expert adds only 2×4×D×r2× 4× D× r parameters per decoder layer, a small fraction of the shared decoder. The query input mirrors the shared path: at layer l, queries take the dataset-routed decoder state d(l−1)F_d^(l-1) (with d(0)=tF_d^(0)=F_t), while keys and values take vF_v. The dataset-routed output at layer l is: (2) ^d(l)=Attn(~q(l),~k(l),~v(l)), F_d^(l)=Attn\! ( z^(l)_q,\; z^(l)_k,\; z^(l)_v ), (3) d(l)=^d(l)o+α⋅^d(l)o(k)o(k),F_d^(l)= F_d^(l)W_o+α· F_d^(l)A_o^(k)B_o^(k), where the output expert applies a final residual module. To allow the shared path to first establish a stable cross-modal alignment before introducing source-specific adaptation, expert modules are injected starting from the second decoder layer only. Additive Feature Decomposition. After N decoder layers, the shared feature and dataset-routed output are summarized via mean pooling: s=MeanPool(s(N))g_s=MeanPool(F_s^(N)), d=MeanPool(d(N)).g_d=MeanPool(F_d^(N)). The full feature is their sum, f=s+dg_f=g_s+g_d. This additive architecture explicitly decomposes the learned representation into a universal shared component and a source-specific correction. Consequently, sg_s is optimized to encapsulate fundamental diagnostic signals independently of dataset-specific variances. Furthermore, the orthogonality constraint (Section 3.4) ensures that sg_s and dg_d encode complementary, non-redundant information, preventing the loss of pathology-relevant features. Bias-Agnostic Inference Protocol. At test time, the dataset-routed expert path is deactivated, allowing the model to rely solely on generalized pathology representations. The shared feature sg_s alone is passed to the linear predictor to compute per-class diagnostic scores. This protocol effectively filters out imaging artifacts and institutional biases encoded in dg_d without compromising prediction accuracy. Crucially, this bias-agnostic strategy requires no prior knowledge of a sample’s origin, facilitating robust and seamless deployment across heterogeneous clinical environments. 3.4. Training Objective The training objective combines three terms that ensure classification accuracy and effective bias disentanglement. Masked binary cross-entropy. A linear predictor maps the full feature fg_f to per-class logits y^c y_c for each disease c∈c . Since each dataset annotates a different subset of conditions, we apply a masked loss that supervises only entries with known labels, directly implementing the tri-state encoding from Section 3.1: (4) ℒbce=1B∑i=1B∑c∈ΩiBCE(y^i,c,yi,c),L_bce= 1B _i=1^B _c∈ _iBCE( y_i,c,\;y_i,c), where Ωi⊆ _i denotes the set of classes with known annotations (y∈0,1y∈\0,1\) for sample i in a mini-batch of size B, and all unknown entries (y=−1y=-1) are excluded. This is the mechanism that prevents false negatives from unannotated classes: a 14-class dataset contributes supervision for its 14 conditions only, leaving the remaining 215 dimensions unaffected. Orthogonality loss. Without explicit constraints, the expert path may duplicate the shared representation, rendering the disentanglement trivial. We enforce complementarity by minimizing the squared cosine similarity between the shared and dataset-specific features: (5) ℒorth=1B∑i=1B(s,i⊤d,i‖s,i‖2‖d,i‖2+ϵ)2.L_orth= 1B _i=1^B ( g_s,i \,g_d,i\|g_s,i\|_2\,\|g_d,i\|_2+ε )^2. This encourages sg_s and dg_d to occupy orthogonal subspaces, ensuring that discarding dg_d at inference removes source-specific variation without eliminating pathology-relevant signal. Expert regularization. Unconstrained low-rank experts may grow excessively large and overfit to source-specific artifacts. We penalize the Frobenius norm of the expert parameters: (6) ℒreg=1N∑l=1N∑m∈q,k,v,o(‖m(l)‖F2+‖m(l)‖F2),L_reg= 1N _l=1^N _m∈\q,k,v,o\ (\|A_m^(l)\|_F^2+\|B_m^(l)\|_F^2 ), which keeps the expert residuals small relative to the shared projections, reinforcing their role as lightweight corrections rather than dominant feature extractors. Total objective. The three terms are combined as ℒ=ℒbce+λorthℒorth+λregℒregL=L_bce+ _orth\,L_orth+ _reg\,L_reg, where λorth _orth and λreg _reg control the strength of the decomposition constraints. 4. Experimental Settings 4.1. Pretraining Configuration Pretraining datasets. HarMoE is pretrained on the four-dataset consortium described in Section 3.1: MIMIC-CXR (Johnson et al., 2019) (377K images), CheXpert (Irvin et al., 2019) (224K images), ChestX-ray14 (Wang et al., 2017) (112K images), and PadChest (Bustos et al., 2020) (160K images), totaling 873K images over 229 unified disease classes. For datasets provided with unstructured reports (e.g., MIMIC-CXR), we utilize Qwen3 (Yang et al., 2025) as the large language model to distill structured tri-state labels. Model Configurations. The framework employs a ViT-B/16 as the image encoder, initialized with M3AE (Chen et al., 2022) weights. For the text encoder, we utilize BioClinicalMPBERT (Lai et al., 2024) to encode disease prompts. The decoder consists of N=4N=4 cascaded dataset bias disentanglement blocks. Dataset-routed expert paths are injected starting from the second layer. The LoRA expert rank is r=8r=8, and the loss weights are set to λorth=λreg=1 _orth= _reg=1. Optimization Strategy. Training uses AdamW with an initial learning rate of 5×10−55× 10^-5, weight decay of 0.02, and a cosine decay schedule over 100 epochs with a 20-epoch linear warm-up. The batch size is 32. The text encoder remains frozen throughout; the image encoder, decoder, and linear predictor are fully optimized. All experiments are conducted on two NVIDIA H200 GPUs with a model training time of approximately 72 hours. 4.2. Evaluation Protocol Zero-shot inference. All evaluations are performed without fine-tuning or linear probing. The pretrained model is applied directly to each benchmark: for a given test image, the shared feature sg_s is extracted from the decoder (with expert modules deactivated, as described in Section 3.3) and passed through the linear predictor to produce per-class scores. Benchmarks. We evaluate on 11 datasets spanning in-domain (ID) and out-of-distribution (OOD) settings. CheXpert and ChestX-ray14 serve as ID benchmarks because they overlap with the pretraining sources, although evaluation is conducted exclusively on their held-out test splits. The remaining benchmarks—OpenI (Demner-Fushman et al., 2016), RSNA (Wu et al., 2024), ChestDR (Wang et al., 2023), VinDr-CXR (Nguyen et al., 2022), SIIM (Anna Zawacki et al., 2019), COVID-QU-Ex (Tahir et al., 2021), JSRT (Shiraishi et al., 2000), Montgomery (Jaeger et al., 2014), and Shenzhen (Jaeger et al., 2014)—are entirely unseen during pretraining and cover diverse acquisition conditions, patient populations, and disease categories, including COVID-19. Metrics. We report two complementary metrics. Area Under the ROC Curve (AUC) measures ranking quality, the model’s ability to assign higher scores to positive cases. Matthews Correlation Coefficient (MCC) measures class-level decision reliability at a fixed operating threshold, capturing the balance between sensitivity and specificity. Reporting both is important because a model with high AUC but low MCC produces good rankings but poor binary decisions, a distinction that matters for clinical deployment. Baselines. We compare against 11 vision-language methods spanning three categories: (i) contrastive image-report alignment (ConVIRT (Zhang et al., 2022), GLoRIA (Huang et al., 2021), BioViL (Boecking et al., 2022), BioViL-T (Bannur et al., 2023), BioMedCLIP (Zhang et al., 2023a), CheXzero (Tiu et al., 2022)); (i) knowledge-enhanced pretraining (KAD (Zhang et al., 2023b), MedKLIP (Wu et al., 2023), MAVL (Phan et al., 2024), DeViDe (Luo et al., 2025a)); and (i) prompt-level supervision (CarZero (Lai et al., 2024), RadZero (Park et al., 2025)). All baselines are evaluated under the same zero-shot protocol for fair comparison. Table 1. Zero-shot multi-label classification (AUC, MCC) on two in-domain (ID) and three out-of-domain (OOD) benchmarks. Bold/underlined indicates best/second-best AUC. LT denotes datasets with long-tailed class distributions. CheXpert (ID) ChestX-ray14 (ID) OpenI (OOD) ChestDR (OOD, LT) VinDr-CXR (OOD) Method AUC MCC AUC MCC AUC MCC AUC MCC AUC MCC GLoRIA (ICCV 2021) 0.664 0.220 0.568 0.087 0.630 0.191 0.566 0.094 0.537 0.101 ConVIRT (TMLR 2022) 0.807 0.396 0.651 0.132 0.637 0.188 0.664 0.165 0.669 0.170 BioViL (ECCV 2022) 0.780 0.398 0.635 0.138 0.655 0.191 0.672 0.187 0.650 0.169 CheXzero (Nature. Biomed. Eng. 2022) 0.890 0.554 0.685 0.159 0.703 0.255 0.716 0.220 0.673 0.191 BioViL-T (CVPR 2023) 0.827 0.451 0.631 0.136 0.662 0.175 0.714 0.211 0.664 0.180 BioMedCLIP (NEJM AI 2023) 0.634 0.201 0.596 0.079 0.570 0.096 0.656 0.159 0.594 0.151 KAD (Nature Com. 2023) 0.871 0.510 0.770 0.263 0.692 0.284 0.623 0.140 0.623 0.140 MedKLIP (ICCV 2023) 0.911 0.581 0.727 0.201 0.588 0.138 0.630 0.146 0.604 0.160 MAVL (CVPR 2024) 0.901 0.606 0.736 0.203 0.665 0.163 0.617 0.125 0.591 0.120 CarZero (CVPR 2024) 0.924 0.609 0.796 0.269 0.722 0.353 0.698 0.207 0.671 0.201 DeViDe (IEEE BIBM 2025) 0.900 0.566 0.776 0.270 0.683 0.279 0.661 0.145 0.724 0.274 RadZero (NeurIPS 2025) 0.902 0.551 0.759 0.241 0.818 0.394 0.704 0.213 0.749 0.284 HarMoE (Ours) 0.926 (+0.2%) 0.603 0.810 (+1.4%) 0.305 (+3.5%) 0.836 (+1.8%) 0.464 (+7%) 0.756 (+4%) 0.255(+3.5%) 0.753 (+0.4%) 0.269 Table 2. Zero-shot single-label classification (AUC, MCC) on six OOD benchmarks. Bold/underlined indicates best/second-best AUC. RSNA (OOD) SIIM (OOD) COVID-QU-Ex (OOD) JSRT (OOD) Montgomery (OOD) Shenzhen (OOD) Method AUC MCC AUC MCC AUC MCC AUC MCC AUC MCC AUC MCC GLoRIA (ICCV 2021) 0.740 0.362 0.591 0.139 0.775 0.430 0.522 0.126 0.733 0.490 0.558 0.140 ConVIRT (TMLR 2022) 0.576 0.167 0.630 0.221 0.681 0.393 0.649 0.272 0.842 0.738 0.816 0.529 BioViL (ECCV 2022) 0.811 0.600 0.571 0.320 0.722 0.423 0.688 0.308 0.921 0.823 0.889 0.658 CheXzero (Nature. Biomed. Eng. 2022) 0.900 0.636 0.784 0.393 0.517 0.057 0.653 0.315 0.784 0.504 0.675 0.311 BioViL-T (CVPR 2023) 0.860 0.675 0.731 0.403 0.689 0.373 0.672 0.292 0.901 0.803 0.891 0.671 BioMedCLIP (NEJM AI 2023) 0.778 0.414 0.629 0.201 0.337 0.000 0.481 0.126 0.725 0.571 0.802 0.474 KAD (Nature Com. 2023) 0.778 0.414 0.874 0.561 0.728 0.446 0.607 0.205 0.897 0.783 0.804 0.558 MedKLIP (ICCV 2023) 0.889 0.611 0.781 0.394 0.769 0.484 0.552 0.155 0.854 0.619 0.585 0.228 MAVL (CVPR 2024) 0.907 0.647 0.709 0.289 0.870 0.585 0.624 0.267 0.845 0.692 0.650 0.317 CarZero (CVPR 2024) 0.803 0.472 0.910 0.660 0.838 0.512 0.641 0.244 0.909 0.807 0.773 0.532 DeViDe (IEEE BIBM 2025) 0.887 0.625 0.895 0.609 0.730 0.470 0.658 0.271 0.879 0.742 0.871 0.586 RadZero (NeurIPS 2025) 0.856 0.540 0.931 0.692 0.832 0.562 0.738 0.367 0.908 0.770 0.916 0.707 HarMoE (Ours) 0.936 (+2.9%) 0.726 (+7.9%) 0.943 (+1.2%) 0.719 (+2.7%) 0.916 (+4.6%) 0.675 (+9.0%) 0.740 (+0.2%) 0.423 (+5.6%) 0.950 (+4.1%) 0.878 (+7.1%) 0.925 (+0.9%) 0.738 (+2.9%) Table 3. Zero-shot visual grounding results (Dice score, Pointing game accuracy) on four benchmarks. Model ChestXray14 RSNA SIIM Covid-Qu-Ex PG↑ Dice↑ PG↑ Dice↑ PG↑ Dice↑ PG↑ Dice↑ KAD 0.350 0.236 0.701 0.421 0.010 0.043 0.120 0.315 DeViDe 0.372 0.247 0.708 0.402 0.065 0.069 0.074 0.325 CARZero 0.437 0.276 0.837 0.505 0.135 0.112 0.628 0.366 RadZero 0.572 0.392 0.765 0.446 0.210 0.163 0.688 0.458 MAVL 0.262 0.192 0.293 0.201 0.051 0.056 0.273 0.230 MedKLIP 0.318 0.233 0.428 0.331 0.010 0.037 0.408 0.299 HarMoE (Ours) 0.620 0.394 0.847 0.521 0.221 0.156 0.647 0.422 5. Results 5.1. Zero-Shot Multi-Label Classification Tab. 1 reports zero-shot multi-label classification results on two in-domain benchmarks (CheXpert and ChestXray14) and three out-of-distribution benchmarks (OpenI, ChestDR, and VinDr-CXR). HarMoE achieves the best AUC on all five datasets, demonstrating that harmonized learning over heterogeneous supervision yields robust transfer beyond the pretraining domain. In particular, HarMoE improves over the second-best method by +4.0% on ChestDR, +1.4% on ChestXray14, +1.8% on OpenI, +0.2% on CheXpert, and +0.4% on VinDr-CXR in terms of AUC. The gains are especially pronounced on OOD datasets, where conventional VLP models are more easily affected by dataset-specific shortcut learning. HarMoE also delivers competitive or best MCC on most benchmarks, including clear improvements on ChestXray14 (+3.5%), OpenI (+7.0%), and ChestDR (+3.5%), indicating that the performance gains are not limited to ranking quality but also translate into more reliable class-level predictions. Overall, these results suggest that unified multi-dataset supervision and controlled dataset-specific specialization enable the model to preserve transferable pathology semantics while reducing overfitting to individual source distributions. 5.2. Zero-Shot Single-Label Classification HarMoE achieves state-of-the-art performance on all six OOD single-label benchmarks as shown in Tab. 2, with particularly striking gains on Montgomery (0.950 AUC, +4.1%) and COVID-QU-Ex (0.916 AUC, +4.6%), followed by RSNA (0.936 AUC, +2.9%) and SIIM (0.943 AUC, +1.2%). The corresponding MCC improvements are equally substantial: +7.9% on RSNA, +9.0% on COVID-QU-Ex, and +7.1% on Montgomery, confirming that these gains reflect genuine class-level decision consistency. Notably, COVID-QU-Ex contains disease categories entirely absent from training data, yet HarMoE achieves its largest improvement there. This suggests the model leverages semantic correlations captured by the text encoder to infer unseen classes, rather than relying on dataset-specific visual shortcuts. These results validate the core HarMoE hypothesis: decoupling shared semantic knowledge from dataset-specific residuals through structured mixture-of-experts routing prevents overfitting to individual dataset statistics and yields harmonized representations that transfer robustly across heterogeneous medical domains. 5.3. Zero-Shot Visual Grounding Tab. 3 reports zero-shot visual grounding results in terms of Dice score and pointing game accuracy. HarMoE achieves the highest scores on the majority of benchmarks, with the most notable improvement on ChestX-ray14, where pointing game accuracy rises from 0.572 to 0.620 (+4.8% relative), followed by RSNA with Dice score improving from 0.505 to 0.521 and pointing game accuracy from 0.837 to 0.847. Importantly, no grounding-specific supervision is used during training; these gains arise solely from the cross-attention alignment learned under classification-level objectives. Qualitative examples in Fig. 7 corroborate these quantitative findings. Prior VLP methods frequently produce diffuse or background-biased attention maps, whereas HarMoE yields spatially concentrated activations that align more closely with the anatomical extent of the target pathology. This effect is particularly evident for diffuse abnormalities (e.g., atelectasis, pleural effusion) and small focal lesions (e.g., nodule, pneumothorax). These results indicate that the shared–dataset feature decomposition introduced by HarMoE not only strengthens classification performance but also produces more spatially faithful cross-modal representations. Radar chart comparing HarMoE, Sparse MoE, and naive joint training across in-domain and out-of-distribution evaluation benchmarks. Figure 3. Radar plot comparing HarMoE, Sparse MoE, and naive joint training (w/o DA-MoE) across all evaluation benchmarks. DA-MoE improves out-of-distribution generalization by preventing the shared representation from overfitting to source-specific patterns in the training data. Bar-chart ablation comparing no regularization, residual regularization only, orthogonality regularization only, and both terms on SIIM and ChestDR. Figure 4. Ablation on regularization terms (None, Res only, Orth only, Res+Orth) evaluated on SIIM and ChestDR. Combining both orthogonality and residual regularization yields the best performance. Line charts of AUC versus pretraining data scale for HarMoE and comparison paradigms across four out-of-distribution benchmarks. Figure 5. Scaling behavior across pretraining paradigms and data scales on four OOD benchmarks. HarMoE with multi-source supervision scales more reliably than contrastive learning. Two UMAP scatter plots show source-separated features with dataset experts active and aligned shared features with the experts inactive. Figure 6. UMAP visualization of learned representations. Activating the dataset experts, features cluster by dataset source; deactivating the experts, the shared representation becomes better aligned across datasets. A grid of original radiographs and attention maps for four pathologies compares HarMoE with six prior methods. Figure 7. Qualitative comparison of zero-shot attention maps for four pathologies (Atelectasis, Effusion, Nodule, Pneumothorax) across methods. HarMoE produces more concentrated and anatomically aligned activations than prior methods. 5.4. Ablation Study We conduct ablations to verify that HarMoE’s improvements stem from its architectural design choices. Specifically, we examine three aspects: (i) the effect of the Dataset Bias Disentanglement Block (DA-MoE), (i) the contribution of individual regularization terms, and (i) the scaling behavior under multi-source supervision. Effect of DA-MoE. Fig. 3 compares HarMoE with Sparse MoE(Fedus et al., 2022) and naive joint training across all evaluation benchmarks. Naive joint training yields a modest improvement on the in-domain ChestX-ray14 benchmark (from 0.810 to 0.822 AUC) but causes consistent degradation on OOD benchmarks: Montgomery drops from 0.950 to 0.917 and VinDr-CXR from 0.753 to 0.667. This confirms that naive joint training without explicit shared–dataset decomposition encourages the model to encode source-specific shortcuts that inflate in-domain scores at the cost of transferability. While Sparse MoE partially recovers OOD performance over naive joint training, HarMoE achieves consistently higher AUC across the most challenging cross-domain benchmarks, with notable improvements on OpenI (0.722 vs. 0.836) and Covid-QU-Ex (0.804 vs. 0.916), while remaining competitive on in-domain datasets such as ChestX-ray14 and CheXpert. This suggests that soft routing alone is insufficient to fully disentangle dataset-specific biases from transferable features, and that the explicit shared–private decomposition in DA-MoE is the key factor driving cross-domain generalization. The UMAP visualization in Fig. 6 provides complementary evidence: activating the dataset expert, representations cluster primarily by dataset source, whereas deactivating the expert, the shared features become better aligned across datasets, with source-dependent variation confined to the residual branch. Effect of Regularization Terms. Fig. 4 isolates the contributions of the orthogonality loss ℒorthL_orth and the LoRA regularization ℒregL_reg. On SIIM, applying either term alone yields only marginal improvement over the unregularized baseline (0.940 vs. 0.939), while their combination achieves 0.943. The effect is more pronounced on ChestDR, where neither term alone exceeds 0.736 AUC, but the full configuration reaches 0.756, a gain of 2.0% absolute. These results indicate that the two terms play complementary roles: ℒorthL_orth prevents the residual branch from duplicating information already captured by the shared representation, while ℒregL_reg constrains expert magnitude to preserve generalization. Their joint application is necessary for effective disentanglement. Effect of Scaling with Multi-Source Data. Fig. 5 compares three pretraining configurations on four OOD benchmarks: image–report contrastive learning on MIMIC-CXR, HarMoE trained on MIMIC-CXR alone, and HarMoE trained on all four source datasets. Two observations emerge. First, on the same data (MIMIC-CXR), replacing contrastive supervision with classification-level prompts yields consistent improvements across all benchmarks (e.g., SIIM AUC increases from 0.784 to 0.936). Second, incorporating additional heterogeneous datasets further improves all benchmarks ( ChestDR: 4%, Vindr-CXR: 1.2%), whereas contrastive learning offers no mechanism to incorporate classification-only datasets. These results demonstrate that HarMoE scales more effectively under multi-source supervision by leveraging cleaner supervisory signals while controlling dataset bias. 6. Conclusion In this work, we introduced HarMoE, a dataset-specialized mixture-of-experts framework for learning from heterogeneous chest X-ray datasets. By incorporating classification-only datasets, HarMoE expands pretraining from 377K to 873K images while separating shared disease features from dataset-specific biases. Experiments show consistent improvements in zero-shot classification, out-of-distribution transfer, and visual grounding. These results demonstrate the value of structured multi-source supervision for building robust and generalizable radiology VLMs. References C. W. Anna Zawacki, J. E. George Shih, P. Mikhail Fomitchev, and S. B. Phil Culliton (2019) SIIM-acr pneumothorax segmentation. Kaggle. Note: https://kaggle.com/competitions/siim-acr-pneumothorax-segmentation Cited by: §4.2. M. Arjovsky, L. Bottou, I. Gulcevich, and D. Lopez-Paz (2019) Invariant risk minimization. In arXiv preprint arXiv:1907.02893, Cited by: §2. S. Bannur, S. Hyland, Q. Liu, F. Perez-Garcia, M. Ilse, D. C. Castro, B. Boecking, H. Sharma, K. Bouzid, A. Thieme, et al. (2023) Learning to exploit temporal structure for biomedical vision-language processing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 15016–15027. Cited by: §2, §4.2. J. Bekker and J. Davis (2020) Learning from positive and unlabeled data: a survey. Machine learning 109 (4), p. 719–760. Cited by: §1, §2. B. Boecking, N. Usuyama, S. Bannur, D. C. Castro, A. Schwaighofer, S. Hyland, M. Wetscherek, T. Naumann, A. Nori, J. Alvarez-Valle, et al. (2022) Making the most of text semantics to improve biomedical vision–language processing. In European conference on computer vision, p. 1–21. Cited by: §2, §4.2. A. Bustos, A. Pertusa, J. Salinas, and M. De La Iglesia-Vaya (2020) Padchest: a large chest x-ray image dataset with multi-label annotated reports. Medical image analysis 66, p. 101797. Cited by: §1, §3.1, §4.1. Z. Chen, Y. Du, J. Hu, Y. Liu, G. Li, X. Wan, and T. Chang (2022) Multi-modal masked autoencoders for medical vision-and-language pre-training. In International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 679–689. Cited by: §4.1. M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev (2023) Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 2818–2829. Cited by: §1. D. C. Clark and K. Conrad (1995) Interobserver variability in interpreting chest radiographs. Archives of Internal Medicine 155 (13), p. 1453–1453. Cited by: §1. A. J. DeGrave, J. D. Janizek, and S. Lee (2021) AI for radiographic covid-19 detection selects shortcuts over signal. Nature Machine Intelligence 3 (7), p. 610–619. Cited by: §1, §2. D. Demner-Fushman, M. D. Kohli, M. B. Rosenman, S. E. Shooshan, L. Rodriguez, S. Antani, G. R. Thoma, and C. J. McDonald (2016) Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association 23 (2), p. 304–310. Cited by: §4.2. W. Fedus, B. Zoph, and N. Shazeer (2022) Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23 (120), p. 1–39. Cited by: §2, §5.4. Y. Ganin, E. Ustinova, H. Ajber, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky (2016) Domain-adversarial training of neural networks. In Journal of Machine Learning Research, Vol. 17, p. 1–35. Cited by: §1, §2. R. Geirhos, J. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann (2020) Shortcut learning in deep neural networks. Nature Machine Intelligence 2 (11), p. 665–673. Cited by: §1, §2. S. Gündel, A. A. Setio, F. C. Ghesu, S. Grbic, B. Georgescu, A. Maier, and D. Comaniciu (2021) Robust classification from noisy labels: integrating additional knowledge for chest radiography abnormality assessment. Medical Image Analysis 72, p. 102087. Cited by: §1. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), Cited by: §2, §2. S. Huang, L. Shen, M. P. Lungren, and S. Yeung (2021) Gloria: a multimodal global-local representation learning framework for label-efficient medical image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 3942–3951. Cited by: §1, §2, §4.2. J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ilcus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya, et al. (2019) Chexpert: a large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, Vol. 33, p. 590–597. Cited by: §1, §3.1, §4.1. S. Jaeger, S. Candemir, S. Antani, Y. J. Wáng, P. Lu, and G. Thoma (2014) Two public chest x-ray datasets for computer-aided screening of pulmonary diseases. Quantitative imaging in medicine and surgery 4 (6), p. 475. Cited by: §4.2. C. Jia, Y. Yang, Y. Xia, Y. Chen, Z. Parekh, H. Pham, Q. Le, Y. Sung, Z. Li, and T. Duerig (2021) Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, p. 4904–4916. Cited by: §1. A. E. Johnson, T. J. Pollard, N. R. Greenbaum, M. P. Lungren, C. Deng, Y. Peng, Z. Lu, R. G. Mark, S. J. Berkowitz, and S. Horng (2019) MIMIC-cxr-jpg, a large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042. Cited by: §1, §3.1, §4.1. H. Lai, Q. Yao, Z. Jiang, R. Wang, Z. He, X. Tao, and S. K. Zhou (2024) Carzero: cross-attention alignment for radiology zero-shot classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 11137–11146. Cited by: §1, §2, §4.1, §4.2. H. Luo, A. Pahud de Mortanges, O. Inel, and M. Reyes (2024) DWARF: disease-weighted network for attention map refinement. In ISIC, iMIMIC, EARTH, and DeCaF Workshops at MICCAI, p. 59–68. External Links: Document Cited by: §2. H. Luo, S. Z. Shu, Z. Zhou, R. Berke, and M. Reyes (2026a) KEPIL: knowledge-enhanced prompt-image learning for prompt-robust disease detection. arXiv preprint arXiv:2605.09132. External Links: Document Cited by: §2. H. Luo, S. Z. Shu, Z. Zhou, S. Otálora, and M. Reyes (2026b) XBench: a comprehensive benchmark for visual-language explanations in chest radiography. In 2026 IEEE International Symposium on Biomedical Imaging (ISBI), p. 1–5. External Links: Document Cited by: §2. H. Luo, Z. Zhou, M. Hou, C. Royer, M. Reyes, A. Sekuboyina, and B. Menze (2025a) DeViDe: faceted medical knowledge to enhance vision foundation model pretraining for radiology. In 2025 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), p. 1779–1782. External Links: Document Cited by: §2, §4.2. H. Luo, Z. Zhou, S. Z. Shu, A. Pahud de Mortanges, R. Berke, and M. Reyes (2025b) On the interplay of human-ai alignment, fairness, and performance trade-offs in medical imaging. In Medical Image Computing and Computer Assisted Intervention (MICCAI), p. 420–430. External Links: Document Cited by: §2. H. Q. Nguyen, K. Lam, L. T. Le, H. H. Pham, D. Q. Tran, D. B. Nguyen, D. D. Le, C. M. Pham, H. T. Tong, D. H. Dinh, et al. (2022) VinDr-cxr: an open dataset of chest x-rays with radiologist’s annotations. Scientific Data 9 (1), p. 429. Cited by: §4.2. L. Oakden-Rayner, J. Dunnmon, G. Carneiro, and C. Ré (2020) Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. In Proceedings of the ACM conference on health, inference, and learning, p. 151–159. Cited by: §1, §2. J. Park, S. Kim, B. Yoon, and K. Choi (2025) RadZero: similarity-based cross-attention for explainable vision-language alignment in radiology with zero-shot multi-task capability. arXiv e-prints, p. arXiv–2504. Cited by: §1, §2, §4.2. V. M. H. Phan, Y. Xie, Y. Qi, L. Liu, L. Liu, B. Zhang, Z. Liao, Q. Wu, M. To, and J. W. Verjans (2024) Decomposing disease descriptions for enhanced pathology detection: a multi-aspect vision-language pre-training framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 11492–11501. Cited by: §2, §4.2. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §1. S. Rebuffi, H. Bilen, and A. Vedaldi (2017) Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 30. Cited by: §2, §2. C. Riquelme, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. Susano Pinto, D. Keysers, and N. Houlsby (2021) Scaling vision with sparse mixture of experts. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 34. Cited by: §2. J. Shiraishi, S. Katsuragawa, J. Ikezoe, T. Matsumoto, T. Kobayashi, K. Komatsu, M. Matsui, H. Fujita, Y. Kodera, and K. Doi (2000) Development of a digital image database for chest radiographs with and without a lung nodule: receiver operating characteristic analysis of radiologists’ detection of pulmonary nodules. American Journal of Roentgenology 174 (1), p. 71–74. Cited by: §4.2. S. Z. Shu, H. Luo, A. Poellinger, and M. Reyes (2025) Hybrid explanation-guided learning for transformer-based chest x-ray diagnosis. In Interpretability of Machine Intelligence in Medical Image Computing, Lecture Notes in Computer Science, Vol. 16464, p. 33–42. External Links: Document Cited by: §2. A. M. Tahir, M. E. Chowdhury, A. Khandakar, T. Rahman, Y. Qiblawey, U. Khurshid, S. Kiranyaz, N. Ibtehaz, M. S. Rahman, S. Al-Maadeed, et al. (2021) COVID-19 infection localization and severity grading from chest x-ray images. Computers in biology and medicine 139, p. 105002. Cited by: §4.2. E. Tiu, E. Talius, P. Patel, C. P. Langlotz, A. Y. Ng, and P. Rajpurkar (2022) Expert-level detection of pathologies from unannotated chest x-ray images via self-supervised learning. Nature Biomedical Engineering 6 (12), p. 1399–1406. Cited by: §4.2. D. Wang, X. Wang, L. Wang, M. Li, Q. Da, X. Liu, X. Gao, J. Shen, J. He, T. Shen, et al. (2023) A real-world dataset and benchmark for foundation model adaptation in medical image classification. Scientific Data 10 (1), p. 574. Cited by: §4.2. X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers (2017) Chestx-ray8: hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 2097–2106. Cited by: §1, §3.1, §4.1. C. Wu, X. Zhang, Y. Zhang, Y. Wang, and W. Xie (2023) Medklip: medical knowledge enhanced language-image pre-training for x-ray diagnosis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 21372–21383. Cited by: §1, §2, §4.2. L. Wu, J. Zhang, Y. Wang, R. Ding, Y. Cao, G. Liu, C. Liufu, B. Xie, S. Kang, R. Liu, et al. (2024) Pneumonia detection based on rsna dataset and anchor-free deep learning detector. Scientific Reports 14 (1), p. 1929. Cited by: §4.2. A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §4.1. J. R. Zech, M. A. Badgeley, M. Liu, A. B. Costa, J. J. Titano, and E. K. Oermann (2018) Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS medicine 15 (11), p. e1002683. Cited by: §1, §2. S. Zhang, Y. Xu, N. Usuyama, H. Xu, J. Bagga, R. Tinn, S. Preston, R. Rao, M. Wei, N. Valluri, et al. (2023a) Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.00915. Cited by: §4.2. X. Zhang, C. Wu, Y. Zhang, W. Xie, and Y. Wang (2023b) Knowledge-enhanced visual-language pre-training on chest radiology images. Nature Communications 14 (1), p. 4542. Cited by: §1, §2, §4.2. Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz (2022) Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference, p. 2–25. Cited by: §1, §2, §4.2. Z. Zhou, H. Luo, J. Pang, X. Ding, M. Gotway, and J. Liang (2023) Learning anatomically consistent embedding for chest radiography. In British Machine Vision Conference (BMVC), p. 617–619. External Links: Link Cited by: §2. Z. Zhou, H. Luo, M. R. H. Taher, J. Pang, X. Ding, M. B. Gotway, and J. Liang (2025a) ACE: anatomically consistent embeddings in composition and decomposition. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), p. 3823–3833. External Links: Document Cited by: §2. Z. Zhou, H. Luo, M. R. H. Taher, J. Pang, X. Ding, M. B. Gotway, and J. Liang (2025b) Lamps: learning anatomy from multiple perspectives via self-supervision in chest radiographs. In Foundation Models for General Medical AI, Lecture Notes in Computer Science, Vol. 16112, p. 1–11. External Links: Document Cited by: §2.

Instant research discovery