Paper deep dive
Generalist-Specialist Mixture-of-Experts for Rare Pathology Detection in Multimodal Imaging
Johannes Kaiser, Florian Braunmiller, Daniel Rückert, Georgios Kaissis
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/19/2026, 2:24:06 AM
Summary
The paper introduces Generalist-Specialist Mixture-of-Experts (GS-MoE), a dual-branch architecture for multimodal medical imaging that combines a cross-modal generalist with modality-specific specialists. GS-MoE addresses the trade-off between negative and positive transfer in rare pathology detection, achieving superior performance on RadImageNet with fewer active parameters than dense baselines.
Entities (11)
Relation Signals (9)
RadImageNet → contains → US
confidence 95% · comprising 165 classes: |C_CT|=34, |C_MR|=116, |C_US|=15.
RadImageNet → contains → MR
confidence 95% · comprising 165 classes: |C_CT|=34, |C_MR|=116, |C_US|=15.
RadImageNet → contains → CT
confidence 95% · comprising 165 classes: |C_CT|=34, |C_MR|=116, |C_US|=15.
GS-MoE → outperforms → DenseNet161Ext
confidence 95% · GS-MoE achieves top MCC/F1 with ~53% fewer active params. than DenseNet161Ext.
GS-MoE → outperforms → ExplicitMoE
confidence 95% · GS-MoE (Expert Classifier) MCC 0.770 vs ExplicitMoE MCC 0.764
GS-MoE → outperforms → ImplicitMoE
confidence 95% · GS-MoE (Expert Classifier) MCC 0.770 vs ImplicitMoE MCC 0.766
GS-MoE → uses → DenseNet121
confidence 95% · We implement GS-MoE with DenseNet121... for both generalist and specialists
GS-MoE → recoversdetectionof → Spring Ligament
confidence 90% · Spring Ligament... ++0.56±0.20... Classes marked with * have F1 == 0 across all baselines and recover through cross-modal information.
→ →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:AI models for multimodal medical imaging must balance modality-specific specialization with cross-modal shared representations, a trade-off that pure Mixture-of-Experts (MoE) architectures currently fail to satisfy. Expert-based routing improves in-domain learning but may sacrifice cross-modal signals, which appear particularly important for rare (low-prevalence) pathologies in our experiments. To resolve this, we introduce Generalist-Specialist-MoE (GS-MoE), a two-branch (MoE) architecture that couples a cross-modal generalist model with distinct modality-specific specialists (experts) via domain-constrained feature fusion. On RadImageNet (1.35M images, 165 pathologies, three modalities), GS-MoE recovers detection of six low-prevalence pathologies on which every baseline scores F1 $=$ 0, with per-class gains up to +0.60 F1. It attains this while even slightly exceeding dense and specialist-only MoE aggregate baselines (MCC 0.770), while using ${\sim}53\%$ fewer active parameters at inference than the strongest investigated dense model.
Tags
Links
- Source: https://arxiv.org/abs/2609.18688v1
- Canonical: https://arxiv.org/abs/2609.18688v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
31,194 characters extracted from source content.
Expand or collapse full text
Generalist-Specialist Mixture-of-Experts for Rare Pathology Detection in Multimodal Imaging Johannes Kaiser Affiliation: Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM) and TUM University Hospital, Munich, Germany Affiliation: Hasso Plattner Institute for Digital Engineering, University of Potsdam, Potsdam, Germany Florian Braunmiller Affiliation: Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM) and TUM University Hospital, Munich, Germany Daniel Rückert Affiliation: Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM) and TUM University Hospital, Munich, Germany Affiliation: Department of Computing, Imperial College London, UK Georgios Kaissis Affiliation: Hasso Plattner Institute for Digital Engineering, University of Potsdam, Potsdam, Germany Abstract AI models for multimodal medical imaging must balance modality-specific specialization with cross-modal shared representations, a trade-off that pure MoE (MoE) architectures currently fail to satisfy. Expert-based routing improves in-domain learning but may sacrifice cross-modal signals, which appear particularly important for rare (low-prevalence) pathologies in our experiments. To resolve this, we introduce Generalist-Specialist-MoE (GS-MoE), a two-branch MoE architecture that couples a cross-modal generalist model with distinct modality-specific specialists (experts) via domain-constrained feature fusion. On RadImageNet (1.35M images, 165 pathologies, three modalities), GS-MoE recovers detection of six low-prevalence pathologies on which every baseline scores F1 == 0, with per-class gains up to +0.60 F1. It attains this while even slightly exceeding dense and specialist-only MoE aggregate baselines (MCC 0.770), while using ∼53% 53\% fewer active parameters at inference than the strongest investigated dense model. Keywords: Mixture of Experts MoE. †footnotetext: Code is available at GitHub. * Equal contribution authors 1 Introduction Medical imaging analysis has shifted from small, single-organ cohorts to large, multimodal, multi-organ datasets such as RadImageNet [26], comprising 1.35 million instances across 165 pathologies of varying prevalence. While such large-scale datasets enable learning generalized representations, monolithic architectures struggle to reconcile the heterogeneity [23], arising naturally from variations in patient demographics, anatomy/pathology, and imaging modalities. Jointly training a single model across diverse imaging modalities exposes a fundamental conflict between two competing transfer effects. Negative transfer [17, 36] occurs when conflicting gradients across weakly related modalities hinder learning, degrading performance on low-prevalence data groups [11]. Positive transfer arises when shared anatomical structures provide a synergistic signal, benefiting low-prevalence classes with limited in-domain data [32, 15]. To benefit low-prevalence data groups (i.e., classes with few training samples we use as a data-driven proxy for rarity, without implying clinical rarity), this motivates a clear desideratum: an architecture must simultaneously minimize negative transfer effects while retaining the potential benefits of cross-modal positive transfer. Unfortunately, this cannot be resolved trivially, as avoiding negative transfer requires training on highly related data, while positive transfer requires incorporating unrelated data to augment the limited in-domain information on low-prevalence pathologies. To address this conflict, we propose the use of Generalist-Specialist Mixture-of-Experts (GS-MoE), a dual-stream MoE architecture that combines a modality-gated MoE with experts acting as specialists on a related subset of the data with a global generalist trained on all available multimodal data. MoE address this by distributing data across distinct experts, operating on well-aligned subsets, while still being capable of processing diverse, multimodal datasets. MoE have gained substantial traction driven by their ability to scale model capacity to trillions of parameters [10, 16, 24] with minimal computational overhead by activating only a sparse subset of experts [27, 9, 2, 12]. The modularity of MoE enables GS-MoE to scale to large datasets by splitting data into well-aligned subsets with domain-specific experts. In vision, prior work focuses mostly on MoE vision transformers with patch-wise experts [2, 13, 29, 33, 6], but also extends to CNN-based instance-wise routing [35] and interpretability [29]. We build on instance-wise routing [35, 7] by routing inputs based on the imaging modality. Like uni-modal monolithic architectures, modality-gated MoE cannot benefit from positive transfer learning. To address this shortcoming and fulfill our aforementioned desideratum, unlike prior works in MoE-based medical image classification [21, 25], segmentation [28, 5, 34], and VLMs [18, 19], we introduce an additional generalist branch alongside the MoE. Conceptually, the MoE provides modality-specific latent encodings of the input, which are augmented by shared cross-modal information from a general branch to benefit from multimodal transfer learning [37]. Our contributions are as follows: 1. We identify cross-domain gradient interference as a likely driver of low-prevalence collapse under modality routing, and propose GS-MoE, a sparse dual-stream architecture that isolates it while retaining positive transfer. 2. We make six low-prevalence pathologies detectable that every dense and specialist-only MoE baseline fails entirely (F1 == 0) with gains up to +0.60+0.60 F1, attributed to cross-modal positive transfer via the generalist. 3. These tail gains come without aggregate cost, as GS-MoE also leads all baselines in overall MCC (0.770) while using ∼53% 53\% fewer active parameters at inference than the strongest dense baseline, at matched total parameters. We expect the modular design of GS-MoE to enable scalability across new tasks, modalities, and patient demographics. 2 Methods 2.1 Routing Strategies: Implicit vs. Explicit multimodal classification induces two competing transfer effects: Negative transfer occurs when the gradient signal of weakly related data distributions across modalities is conflicting during joint optimization, hindering modality-specific feature learning [17, 36]. Positive transfer arises when shared (anatomy, intensity etc.) structures across distributions provide a complementary signal, potentially benefiting low-prevalence classes with limited in-domain data [32, 15]. Explicit routing uses a metadata-trained assignment to route data to a (modality-)specific expert. Focusing on a single modality prevents negative transfer but limits positive transfer. Implicit routing learns assignments end-to-end, enabling transfer learning. Yet without careful initialization, implicit routing collapses to a single expert, yielding uninterpretable, inefficient routing while reliable metadata goes unused. Neither approach simultaneously ensures interpretable routing, cross-modal feature sharing, and prevents negative transfer effects, a limitation GS-MoE resolves. 2.2 Generalist-Specialist-MoE (GS-MoE) To resolve this trade-off, we propose Generalist-Specialist-MoE (GS-MoE), schematized in Fig. 1: a two-branch architecture coupling a cross-modal generalist with modality-specific experts that reduce exposure to cross-domain negative transfer. Figure 1: Schematic representation of GS-MoE with the Expert Classifier. Showing an exemplary forward (black) and backward (red) pass for a CT input. Training is performed in stages, with each part trained on its relevant subset of data. The Generalist Branch acts as an always-active shared pathway, analogous to shared-expert isolation in MoE architectures [8], but realized as a full-capacity branch pre-trained across all modalities that also seeds the specialists, learning a shared representation g∈ℝdf_g ^d benefiting from positive transfer. The Specialist Branch is an MoE with explicit routing: a frozen metadata-trained router identifies modality m and activates the relevant expert, yielding domain-constrained features s∈ℝdf_s ^d while keeping routing interpretable by construction. The branches are fused by concatenation, fused=[g∥s]∈ℝ2df_fused=[f_g\|f_s] ^2d, and jointly fine-tuned with the router frozen. To enforce complementary representations, we apply a cosine orthogonality penalty inspired by [1], as ℒortho=B−1∑i(g⊤s‖g‖−1‖s‖−1)2L_ortho=B^-1 _i(f_g f_s\|f_g\|^-1\|f_s\|^-1)^2, with B the batch size. 2.3 Domain-Constrained Optimization: Why Shared Heads Fail fusedf_fused exposes a structural vulnerability when passed through a shared output head. Consider D domains with |Cm||C_m| classes in domain m and |Ctotal|=∑m|Cm||C_total|= _m|C_m|. A shared head ϕshared:ℝ2d→ℝ|Ctotal| _shared ^2d ^|C_total| applied to a domain-m sample computes the loss over all |Ctotal||C_total|, including clinically impossible ones. Gradients then flow through all |Ctotal||C_total| outputs, with the |Ctotal|−|Cm||C_total|-|C_m| non-domain outputs diluting the signal. We address this via modality-specific network heads ϕm\ _m\, each mapping ℝ2d→ℝ|Cm|R^2d ^|C_m|. While restricting predictions to the modality’s valid-label subset is an established masking operation [3], our contribution is the finding that its benefit is not diffuse but concentrated on low-prevalence classes, consistent with removal of cross-domain gradient interference where in-domain data is scarcest. We ablate this in Table 3 and find that this raises MR-class macro-F1 from 0.40920.4092 to 0.45370.4537 (+0.045+0.045) on low-prevalence classes. During training, metadata selects the correct head, inducing ℒCEm=−∑c∈Cmyclogp^cL_CE_m=- _c∈ C_my_c p_c, with ℒ=ℒCEm+λℒorthoL=L_CE_m+λ\,L_ortho. At inference, ϕm _m produces |Cm||C_m| logits, which a fixed, expert-knowledge-driven assignment matrix maps to the global |Ctotal||C_total|-dimensional label space. 3 Experimental Setup Dataset. We evaluate on RadImageNet [26], a large multimodal pathology classification dataset with 1.35M 2D images across CT (CT), MR (MR), and US (US), comprising 165 classes: |CCT|=34|C_CT|=34, |CMR|=116|C_MR|=116, |CUS|=15|C_US|=15. All models use the official train/val/test splits. Generalist-Specialist-MoE (GS-MoE) We implement GS-MoE with DenseNet121 (pre-classifier output dimensionality d=1024d=1024) for both generalist and specialists, yielding fused∈ℝ2048f_fused ^2048 with domain-specific expert classifiers ϕm:2048→1024→|Cm| _m 2048→ 1024→|C_m| and modality router, a lightweight DenseNet29 (4 blocks, growth rate=8growth rate=8, ∼ 61k params, >99.9%>99.9\% modality prediction accuracy). Routing is modality-based, which outperforms modality × anatomy experts. Training follows a three-stage curriculum (ablated in Section 4.1): (1) generalist pre-training on all 165 classes, (2) specialist generation via sparse upcycling [22] from the converged generalist and modality-specific fine-tuning, and (3) GS-MoE construction and joint fine-tuning of branches and classifiers. This procedure preserves cross-modal features while enabling high-capacity, low-cost training. Baselines. As commonly used in medical imaging classification tasks [4, 30], we consider DenseNet121 (7.17.1M), DenseNet161 (26.826.8M) [14], and a parameter-matched DenseNet161Ext (blocks: 6/12/36/38, 35.035.0M) trained on the full corpus as dense monolithic baselines. As MoE baselines, we use ExplicitMoE, identical to GS-MoE but without the generalist branch, and ImplicitMoE, which shares the architecture yet is trained end-to-end. Remark: While we use DenseNets due to slightly higher performance, our findings directly translate to ViT-T/ViT-S backbones, indicating the effect is architectural (see Section 5). Training Setup and Metrics. Models are trained until convergence with AdamW, cosine annealing (T0=4T_0=4), on 224×224224×224 inputs with random affine and horizontal flips. Hyperparameters are empirically optimized. We report MCC (MCC) as the primary metric for its robustness to class imbalance, alongside macro-F1. 4 Results 4.1 Aggregate Results Section 4.1summarizes test-set performance across all architectures. Without sparse upcycling MoEs collapse to ExplicitMoE MCC of 0.6390.639 and ImplicitMoE MCC of 0.6910.691. Mean and standard deviation are computed across 9 seeds throughout. The ranking in terms of MCC across DenseNet161 < ExplicitMoE < GS-MoE (Expert) is statistically significant (p<0.05p<0.05, paired t-test). At inference, GS-MoE activates only the generalist, the frozen router, and one expert (∼16.4 16.4M parameters), using ∼53% 53\% fewer active parameters than DenseNet161Ext (35.035.0M parameters). This reduction is structurally guaranteed by routing and is Pareto-optimal in terms of parameter efficiency (see Fig. 2). Table 1: MoE outperform dense models. GS-MoE achieves top MCC/F1 with ∼53% 53\% fewer active params. than DenseNet161Ext. Best in family, Best overall. Family Model MCC macro-F1 Active Params Total Params 2-1Baseline DenseNet121 0.744±0.022 0.751±0.020 7.1M 7.1M DenseNet161 0.752±0.018 0.759±0.016 26.8M 26.8M DenseNet161Ext 0.756±0.012 0.763±0.010 35.0M 35.0M 2-1MoE Explicit 0.764±0.003 0.771±0.004 7.2M 21.4M Implicit 0.766±0.005 0.772±0.006 7.2M 21.4M 2-1GS-MoE Shared Classifier 0.762±0.005 0.768±0.005 16.5M 30.8M Expert Classifier 0.770±0.003 0.776±0.004 16.4M 35.0M 00551010151520202525303035350.7420.7420.7480.7480.7540.7540.760.760.7660.7660.7720.7720.7780.778DenseNet121DenseNet161DenseNet161ExtOursActive Parameters [Millions]MCCBaselineExplicitMoEImplicitMoEGS-MoE — Shared ClassifierGS-MoE — Expert ClassifierGS-MoE — Expert Classifier + ℒorthoL_ortho (Ours) Figure 2: MCC vs. active parameters. Dashed line: Pareto frontier. GS-MoE (Expert Classifier + LorthoL_ortho) is Pareto optimal w.r.t. parameter efficiency, outperforming explicitly and implicitly routed MoE as well as dense models. 4.2 Routing Strategies Face Mutually Exclusive Failure Modes Our evaluations show that monolithic models and baseline MoE face mutually exclusive failure modes, motivating the proposed generalist-specialist MoE. Negative transfer and pre-training. Scaling monolithic capacity shows diminishing returns: DenseNet161Ext improves over DenseNet121 by only +1.2+1.2 p in MCC at 4.9×4.9× more parameters, indicating an architectural ceiling. GS-MoE resolves this limit and outperforms DenseNet121 by +2.6+2.6 p, with matched total and ∼53% 53\% fewer active parameters. Cost of explicit routing on MR domains. Explicit routing improves visually isolated domains (CT-Lung +0.041+0.041, US +0.008+0.008 group/modality aggregated F1) but degrades all eight MR groups (e.g., MR-Brain −0.087-0.087, MR-Abdomen −0.064-0.064, MR-Hip −0.051-0.051, MR-Ankle −0.035-0.035). At the pathology level, as evidenced in Fig. 3, ExplicitMoE improves on none of the 20% lowest-performing classes relative to DenseNet161, with brain-pituitary collapsing 0.727→0.0000.727→ 0.000, AF-coalition 0.400→0.0000.400→ 0.000, and AF-spring ligament 0.316→0.0000.316→ 0.000, indicating that ExplicitMoE may not effectively leverage cross-domain information for these classes. Implicit routing recovers representation but not semantics. Without pre-training, ImplicitMoE deposits 93.6%93.6\% of inputs to a single expert despite an auxiliary load-balancing loss [31]. Sparse upcycling restores a distributed routing profile (41.5/34.5/24.0%41.5/34.5/24.0\%). Clustering analysis reveals that the router forms data-driven groups: one expert specializes in CT-Lung imagery (91.1%91.1\% CT, 90.5%90.5\% lung), while the remaining experts process a heterogeneous mix. Despite this, as the only baseline model, ImplicitMoE outperforms DenseNet161 across all 11 anatomical groups, with the largest gains on MR-Ankle (+0.083+0.083), MR-Spine (+0.050+0.050). Nonetheless, lacking explicit global cross-modal features, both routing strategies fail on the low-prevalence classes (Fig. 3). 10210^210310^310410^4000.20.20.40.40.60.60.80.8NtrainN_train per classMacro F1DenseNet161ExtExplicitMoEImplicitMoEGS-MoE Figure 3: Sliding-window (width=20) macro F1 (90% CI) over prevalence-ordered classes. ExplicitMoE trails all baselines on low-prevalence classes due to a lack of cross-domain information; GS-MoE recovers this regime, matching or improving throughout. Category Ntrain/Ntest $N_train$$N_test$ F1 Δ LisfrancMR-AF_\,MR-AF 37/8 378 ++0.60±0.12∗ Spring Lig.MR-AF_\,MR-AF 83/5 835 ++0.56±0.20∗ CoalitionMR-AF_\,MR-AF 50/4 504 ++0.56±0.36∗ HematomaMR-AF_\,MR-AF 165/29 16529 ++0.55±0.04^\,\,\, ExtensorMR-AF_\,MR-AF 536/83 53683 ++0.44±0.03^\,\,\, Intra-Art. MassMR-AF_\,MR-AF 455/60 45560 ++0.40±0.03^\,\,\, NeoplasmMR-AF_\,MR-AF 745/95 74595 ++0.33±0.02^\,\,\, ProstateMR-Abd_\,MR-Abd 53/4 534 ++0.24±0.26^\,\,\, Prostate Les.CT-Abd_\,CT-Abd 782/98 78298 −-0.05±0.02^\,\,\, BladderCT-Abd_\,CT-Abd 1477/157 1477157 −-0.07±0.02^\,\,\, Renal Les.MR-Abd_\,MR-Abd 1627/234 1627234 −-0.10±0.02^\,\,\, Normal CTCT-Abd_\,CT-Abd 57432/6689 574326689 −-0.11±0.03^\,\,\, Table 2: Top per-class gains/losses of GS-MoE vs. per-class best baseline. Classes marked with ∗ have F1 == 0 across all baselines and recover through cross-modal information. AF: Ankle/Foot, Abd: Abdomen, Sp: Spine. 0.20.20.40.40.60.60.80.8110.20.20.40.40.60.60.80.811DenseNet161Ext F1-ScoreExplicit MoE F1-Score(a)MR-AFMR-BrainMR-AbdOther MRUSCT-LungCT-Abd0.20.20.40.40.60.60.80.8110.20.20.40.40.60.60.80.811DenseNet161Ext F1-ScoreGS-MoE F1-Score(b)0.20.20.40.40.60.60.80.8110.20.20.40.40.60.60.80.811ExplicitMoE F1-ScoreGS-MoE F1-Score(c) Figure 4: Per-pathology F1-score comparison (165 classes). (a) ExplicitMoE improves on distinct modalities (US, CT-Lung) but collapses on MR-AF/Brain, with several classes reaching zero. (b) GS-MoE (Expert Classifier + LorthoL_ortho) recovers the MR-AF cluster, outperforming the DenseNet161, including several classes at x=0x=0 (zero-baseline performance). (c) GS-MoE (Expert Classifier + LorthoL_ortho) strongly exceeds ExplicitMoE, particularly in MRI, attributed to explicit generalist-driven cross-domain knowledge. 4.3 Hard Class Recovery: A Qualitative Capability Gap While ExplicitMoE mitigates negative transfer, gaining slightly on distinct modalities (US, CT-Lung), it lacks positive transfer during fine-tuning, degrading cross-modal knowledge acquired during pre-training. As illustrated in Fig. 4 (a), this leads to performance collapse (F1 == 0) on some MR-pathologies. In contrast, GS-MoE architecturally augments experts with cross-modal features from a dedicated generalist branch. GS-MoE recovers performance on collapsed pathologies, particularly within the MR-Ankle/Foot group (Fig. 4 (b, c)), and significantly (p<0.001p<0.001, paired t-test) outperforms all baselines on low-prevalence classes (Fig. 3), with the strongest recovery in the MR-Abdomen group (+0.057+0.057), the weakest-performing group overall. Crucially, GS-MoE achieves non-zero F1 on six low-prevalence pathologies (< 100 samples) where all baselines, including DenseNet161Ext trained with focal loss (γ∈1,2,5γ∈\1,2,5\) and post-hoc classifier rebalancing (τ-normalization, τ∈[0,1]τ∈[0,1]) [20], fail entirely (F1 == 0), with recoveries as large as +0.60 (lisfranc pathology), +0.56 (spring ligament) and +0.56 (coalition) (Table 2). These gains are consistent with the generalist branch providing expert-complementary information. This is evidenced by GS-MoE outperforming the explicit specialist-only MoE across most pathologies (Fig. 4 (c)), while only differing by the generalist. As the parameter-matched DenseNet161Ext does not recover the tail (Fig. 3) despite equal capacity, we attribute the low-prevalence recovery to the generalist branch. The recovered signal is cross-modal by nature, a capability unimodal experts cannot provide. Decoupling generalist (cross-modal) and specialist (unimodal) knowledge may yield a more effective feature representation than specialist-only MoE and monolithic models. 4.4 Ablation Table 3 ablates the contribution of each part of GS-MoE and finds that Generalist, Specialist, Domain Head, and ℒorthoL_ortho all improve MCC. The gap between the Shared Classifier and Expert Classifier isolates the effect of the shared output head (Section 2.3), as they are identical except for the output head (Table 3). The shared head degrades disproportionately on low-prevalence MR classes: with all else held fixed, switching to domain-constrained heads raises the MR-class-averaged F1 from 0.40920.4092 to 0.45370.4537 (+0.045+0.045,11%~11\% relative). Moreover, the gain from domain-specific heads points to a benefit from modality-specific processing of cross-domain information. Table 3: Ablation of Domain-Constrained Optimization. The gap between the Shared Classifier and Expert Classifier isolates the effect of the expert head. DenseNet161Ext and ExplicitMoE serve as reference points. Model Variant Generalist Specialist Domain Head ℒorthoL_ortho MCC DenseNet161Ext ✓ ✗ ✗ ✗ 0.756 ExplicitMoE ✗ ✓ ✓ ✗ 0.764 GS-MoE (Shared Classifier) ✓ ✓ ✗ ✗ 0.762 GS-MoE (Expert Classifier) ✓ ✓ ✓ ✗ 0.765 GS-MoE (Expert C. + ℒorthoL_ortho) ✓ ✓ ✓ ✓ 0.770 5 Discussion GS-MoE mitigates the fundamental conflict between modality-specific specialization and cross-modal sharing: low-prevalence collapse under modality routing is consistent with cross-domain gradient interference, recoverable via an always-active generalist. Coupling a modality-routed specialist MoE with this generalist preserves interpretable routing while retaining shared cross-modal representations. We show that GS-MoE outperforms all investigated baselines, with the margin most pronounced in low-prevalence classes. While we showed results on dense backbones, we found the same trends with transformer backbones, with MCC of: ViT-T/S baselines 0.726/0.737, ExplicitMoE-ViT-T 0.741, GS-MoE-ViT-T 0.745. Although absolute MCC is lower than for DenseNet, the ordering (GS-MoE > ExplicitMoE > Dense) and the low-prevalence recovery are fully preserved, indicating the generalist-specialist effect is architectural rather than backbone-specific. The limitations of our work directly motivate future work extending GS-MoE to (volumetric) segmentation, integrating report-grounded supervision, developing hierarchical experts conditioned on anatomy and demographics, potentially as hierarchical MoE, and ablations across different model sizes. In summary, GS-MoE shows that metadata-supervised specialization and learned generalist features contribute complementary gains, a practical step toward rare-pathology-aware multimodal imaging. References [1] K. Bousmalis, G. Trigeorgis, N. Silberman, D. Krishnan, and D. Erhan (2016) Domain separation networks. In NeurIPS, Cited by: §2.2. [2] W. Cai, J. Jiang, F. Wang, J. Tang, S. Kim, and J. Huang (2025) A survey on mixture of experts in large language models. IEEE Transactions on Knowledge and Data Engineering. Cited by: §1. [3] Y. Cao and Q. Gu (2019) Generalization bounds of stochastic gradient descent for wide and deep neural networks. In NeurIPS, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32, p. . Cited by: §2.3. [4] P. Chambon, J. Delbrouck, T. Sounack, S. Huang, Z. Chen, M. Varma, S. Q. Truong, C. T. Chuong, and C. P. Langlotz (2024) CheXpert plus: augmenting a large chest x-ray dataset with text radiology reports, patient demographics and additional image formats. Note: arXiv:2405.19538 Cited by: §3. [5] Q. Chen, L. Zhu, H. He, X. Zhang, S. Zeng, Q. Ren, and Y. Lu (2024) Low-rank mixture-of-experts for continual medical image segmentation. In MICCAI, Cited by: §1. [6] T. Chen, X. Chen, X. Du, A. Rashwan, F. Yang, H. Chen, Z. Wang, and Y. Li (2023) AdaMV-moe: adaptive multi-task vision mixture-of-experts. In ICCV, Cited by: §1. [7] S. Chopra, G. Sanchez-Rodriguez, L. Mao, A. J. Feola, J. Li, and Z. Kira (2025) MedMoE: modality-specialized mixture of experts for medical vision-language understanding. Note: arXiv:2506.08356 Cited by: §1. [8] D. Dai, C. Deng, C. Zhao, R.X. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, Z. Xie, Y.K. Li, P. Huang, F. Luo, C. Ruan, Z. Sui, and W. Liang (2024) DeepSeekMoE: towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the ACL (Volume 1: Long Papers), Cited by: §2.2. [9] W. Fedus, J. Dean, and B. Zoph (2022) A review of sparse expert models in deep learning. Note: arXiv:2209.01667 Cited by: §1. [10] W. Fedus, B. Zoph, and N. Shazeer (2022) Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23 (120). Cited by: §1. [11] Y. Fu, S. Huang, Z. Feng, and Y. Ma (2025) RDAM: domain adaptation under small and class-imbalanced samples. Knowledge-Based Systems. Cited by: §1. [12] W. Gan, Z. Ning, Z. Qi, and P. Yu (2025) Mixture of experts (moe): a big data perspective. Information Fusion 127. Cited by: §1. [13] X. Han, L. Wei, Z. Dou, Y. Sun, Z. Han, and Q. Tian (2025) ViMoE: an empirical study of designing vision mixture-of-experts. IEEE Transactions on Image Processing. Cited by: §1. [14] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger (2017) Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 4700–4708. Cited by: §3. [15] Z. Huang, Y. Hu, R. Liu, X. Xue, Z. Zhu, L. Song, and K. C. Tan (2023) Federated multi-task learning for joint diagnosis of multiple mental disorders on mri scans. IEEE Transactions on Biomedical Engineering 70. Cited by: §1, §2.1. [16] A. Q. Jiang, A. Sablayrolles, and et al. (2024) Mixtral of experts. Note: arXiv:2401.04088 Cited by: §1. [17] J. Jiang, B. Chen, J. Pan, X. Wang, D. Liu, J. Jiang, and M. Long (2023) Forkmerge: mitigating negative transfer in auxiliary-task learning. NeurIPS. Cited by: §1, §2.1. [18] S. Jiang, T. Zheng, Y. Zhang, Y. Jin, L. Yuan, and Z. Liu (2024) Med-moe: mixture of domain-specific experts for lightweight medical vision-language models. In Findings of the ACL: EMNLP 2024, Cited by: §1. [19] Y. Jiang and Y. Shen (2024) M4oe: a foundation model for medical multimodal image segmentation with mixture of experts. In MICCAI, Cited by: §1. [20] B. Kang, S. Xie, M. Rohrbach, Z. Yan, A. Gordo, J. Feng, and Y. Kalantidis (2020) Decoupling representation and classifier for long-tailed recognition. In ICLR, Cited by: §4.3. [21] N. Kanwal, F. Khoraminia, U. Kiraz, A. Mosquera-Zamudio, C. Monteagudo, E. Janssen, T. Zuiverloon, C. Rong, and K. Engan (2024) Equipping computational pathology systems with artifact processing pipelines: a showcase for computation and performance trade-offs. BMC Medical Informatics and Decision Making 24. Cited by: §1. [22] A. Komatsuzaki, J. Puigcerver, J. Lee-Thorp, C. R. Ruiz, B. Mustafa, J. Ainslie, Y. Tay, M. Dehghani, and N. Houlsby (2023) Sparse upcycling: training mixture-of-experts from dense checkpoints. In ICLR, Cited by: §3. [23] A. Lakkapragada, E. Sleiman, S. Surabhi, and D. P. Wall (2023) Mitigating negative transfer in multi-task learning with exponential moving average loss weighting strategies. In AAAI, Vol. 37. Cited by: §1. [24] D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen (2021) GShard: scaling giant models with conditional computation and automatic sharding. In ICLR, Cited by: §1. [25] J. Li, Y. Zhang, W. Shu, X. Feng, Y. Wang, P. Yan, X. Li, C. Sha, and M. He (2025) M4: multi-proxy multi-gate mixture of experts network for multiple instance learning in histopathology image analysis. Medical Image Analysis 103. Cited by: §1. [26] X. Mei, Z. Liu, P. M. Robson, B. Marinelli, M. Huang, A. Doshi, A. Jacobi, C. Cao, K. E. Link, T. Yang, et al. (2022) RadImageNet: an open radiologic deep learning research dataset for effective transfer learning. Radiology: Artificial Intelligence 4 (5). Cited by: §1, §3. [27] S. Mu and S. Lin (2026) A comprehensive survey of mixture-of-experts: algorithms, theory, and applications. Note: arXiv:2503.07137 Cited by: §1. [28] Y. Ou, Y. Yuan, X. Huang, S. T. Wong, J. Volpi, J. Z. Wang, and K. Wong (2022) Patcher: patch transformers with mixture of experts for precise medical image segmentation. In MICCAI, Cited by: §1. [29] C. Riquelme, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. Susano Pinto, D. Keysers, and N. Houlsby (2021) Scaling vision with sparse mixture of experts. In NeurIPS, Vol. 34. Cited by: §1. [30] V. Santamato and A. Marengo (2025) Multilabel classification of radiology image concepts using deep learning. Applied Sciences 15 (9), p. 5140. Cited by: §3. [31] N. Shazeer and et al. (2017) Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. In ICLR, Cited by: §4.2. [32] A. Verma (2023) How can we tame the long-tail of chest x-ray datasets?. Note: arXiv:2309.04293 Cited by: §1, §2.1. [33] M. Videau, A. Leite, M. Schoenauer, and O. Teytaud (2025) Mixture of experts in image classification: what’s the sweet spot?. Note: arXiv:2411.18322 Cited by: §1. [34] G. Wang, J. Ye, J. Cheng, T. Li, Z. Chen, J. Cai, J. He, and B. Zhuang (2024) SAM-med3d-moe: towards a non-forgetting segment anything model via mixture of experts for 3d medical image segmentation. In MICCAI, Cited by: §1. [35] X. Wang, F. Yu, L. Dunlap, Y. Ma, R. Wang, A. Mirhoseini, T. Darrell, and J. E. Gonzalez (2020) Deep mixture of experts via shallow embedding. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, Vol. 115. Cited by: §1, §1. [36] Z. Wang, Z. Dai, B. Póczos, and J. Carbonell (2019) Characterizing and avoiding negative transfer. In CVPR, Cited by: §1, §2.1. [37] J. Wu, X. Hu, Y. Wang, B. Pang, and R. Soricut (2024) OMNI-smola: boosting generalist multimodal models with soft mixture of low-rank experts. In CVPR, p. 14205–14215. Cited by: §1.