Paper deep dive
Balanced Soft mixture-of-expert model for Glaucoma Detection
Sai Venkatesh Chilukoti, Krishna Rauniyar, Min Shi, Xiali Hei
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/1/2026, 11:36:39 AM
Summary
The paper proposes a Balanced Soft Mixture-of-Experts (SMoE) model for glaucoma detection to address imbalanced learning in multi-modal deep learning. The model utilizes three experts (two uni-modal and one multi-modal) with a load balancing loss to ensure equitable contribution from different imaging modalities like fundus images and OCT scans. Evaluated on FairVision, FairDomain, and HarvardGF datasets, the proposed SMoE model outperforms uni-modal baselines, conventional multi-modal models, and other state-of-the-art balanced multi-modal approaches in terms of AUC.
Entities (16)
Relation Signals (12)
Balanced Soft Mixture-of-Experts (SMoE) â evaluatedon â FairVision
confidence 95% · Table 2 presents the overall AUC results for various multi-modal (MM) learning approaches across the FairVision, FairDomain, and HarvardGF datasets.
Balanced Soft Mixture-of-Experts (SMoE) â evaluatedon â FairDomain
confidence 95% · Table 2 presents the overall AUC results for various multi-modal (MM) learning approaches across the FairVision, FairDomain, and HarvardGF datasets.
Balanced Soft Mixture-of-Experts (SMoE) â evaluatedon â HarvardGF
confidence 95% · Table 2 presents the overall AUC results for various multi-modal (MM) learning approaches across the FairVision, FairDomain, and HarvardGF datasets.
Balanced Soft Mixture-of-Experts (SMoE) â uses â OCT Scans
confidence 95% · two uni-modal experts, each processing a single modality... SLO, OCT, or their combination
Balanced Soft Mixture-of-Experts (SMoE) â uses â Fundus Images
confidence 95% · two uni-modal experts, each processing a single modality... SLO, OCT, or their combination
Balanced Soft Mixture-of-Experts (SMoE) â outperforms â EfficientNet
confidence 90% · our proposed method surpasses the performance of all uni-modal baselines... EfficientNet consistently demonstrates strong performance... SMoE consistently achieves the highest AUC scores
Balanced Soft Mixture-of-Experts (SMoE) â outperforms â OPM
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Glaucoma is a group of eye diseases that damage the optic nerve, often caused by elevated intraocular pressure. It is a leading cause of irreversible vision loss and is typically developed slowly and painlessly, making it difficult to notice until significant damage has occurred. Therefore, early detection is crucial to prevent or slow the progression of vision loss. In recent years, deep learning based uni-modal models have improved the accuracy and efficiency of glaucoma detection, empowering doctors with tools for earlier diagnosis, better monitoring, and timely treatment. Building on this, multi-modal models have emerged, leveraging the strengths of different imaging modalities to learn richer and more robust representations, further enhancing glaucoma detection accuracy. However, multi-modal learning faces challenges such as imbalanced and under-optimized uni-modal representations due to joint learning objectives. To address this, we propose a balanced soft mixture-experts model with three experts and load balancing loss. The performance is measured by AUC, our proposed method surpasses the performance of all uni-modal baselines, conventional multi-modal models, and current stateof- the-art balanced multi-modal models. The proposed model can be generalized to other disease detections such as diabetic retinopathy.
Tags
Links
- Source: https://arxiv.org/abs/2607.25324v1
- Canonical: https://arxiv.org/abs/2607.25324v1
Trouble viewing inline? Open PDF directly â
Full Text
49,520 characters extracted from source content.
Expand or collapse full text
[1] Venkatesh [1] of Computing and Informatics, of Louisiana at Lafayette, 301 East Lewis Street, , 70504, , Balanced Soft mixture-of-expert model for Glaucoma Detection saivenkatesh.chilukoti309@gmail.com krishna.rauniyar1@louisiana.edu min.shi@louisiana.edu xiali.hei@louisiana.edu * Abstract Glaucoma is a group of eye diseases that damage the optic nerve, often caused by elevated intraocular pressure. It is a leading cause of irreversible vision loss and is typically developed slowly and painlessly, making it difficult to notice until significant damage has occurred. Therefore, early detection is crucial to prevent or slow the progression of vision loss. In recent years, deep learning based uni-modal models have improved the accuracy and efficiency of glaucoma detection, empowering doctors with tools for earlier diagnosis, better monitoring, and timely treatment. Building on this, multi-modal models have emerged, leveraging the strengths of different imaging modalities to learn richer and more robust representations, further enhancing glaucoma detection accuracy. However, multi-modal learning faces challenges such as imbalanced and under-optimized uni-modal representations due to joint learning objectives. To address this, we propose a balanced soft mixture-experts model with three experts and load balancing loss. The performance is measured by AUC, our proposed method surpasses the performance of all uni-modal baselines, conventional multi-modal models, and current state-of-the-art balanced multi-modal models. The proposed model can be generalized to other disease detections such as diabetic retinopathy. 1 Introduction Glaucoma is the second leading cause of blindness worldwide and affects approximately 4.22 million people in the United States. It can affect people of all ages, but the risk is higher for people over 60 years of age, those with a family history of glaucoma, diabetics, and people with severe nearsightedness [1]. Currently, there is no cure for glaucoma, and vision loss caused by the disease is irreversible. Although medications and surgical treatments can help halt or slow the progression of vision loss, most forms of glaucoma do not exhibit symptoms until the condition has advanced significantly. Therefore, accurate and early detection is crucial to prevent permanent vision loss [2]. Glaucoma detection generally involves assessing structural damage to the optic nerve along with evaluating visual function. However, clinical examination of the optic nerve head (ONH) and the retinal nerve fiber layer (RNFL) can be subjective and prone to variability. To address this, recent research has focused on developing objective diagnostic tools. Technologies such as confocal scanning laser ophthalmoscopy and optical coherence tomography have been widely studied as complementary methods to support the traditional subjective evaluation of ONH [3]. Recently, deep learning models for the detection of glaucoma have analyzed images such as fundus photos and OCT scans to automatically detect disease features with high accuracy. They can outperform human experts in some cases and offer scalable solutions for early detection [4]. Most glaucoma detection models employ deep convolutional neural networks (CNNs) specifically designed to analyze particular imaging modalities, such as fundus photographs or OCT scans. These models focus on identifying key indicators of glaucoma, including cup-to-disc ratio, thinning of the retinal nerve fiber layer (RNFL), and morphology of the optic nerve head, which are critical for an accurate diagnosis. To improve performance, advanced techniques such as transfer learning and multiscale feature extraction are commonly used, improving both accuracy and robustness [5, 6, 7]. However, despite their strong results, the dependence of uni-modal models on a single type of imaging data can limit their ability to generalize across diverse patient populations and imaging conditions, thus motivating the development of multi-modal approaches that integrate complementary data sources for a more comprehensive detection of glaucoma. Recent advances in multi-modal glaucoma detection leverage deep learning models that integrate various imaging modalities to enhance diagnostic accuracy. For instance, Hwang et al. (2025) [8] developed a neural network using minimally processed fundus photographs, OCT scans, and visual field analyzes, demonstrating superior detection of glaucoma compared to single-mode approaches. Du et al. (2024) [9] introduced the hybrid multi-modulal VGG (HM-VGG) model, which employs attention mechanisms to effectively analyze visual field data, particularly excelling in limited data scenarios. Together, these studies highlight the potential of multi-modal deep learning frameworks to improve early glaucoma detection by combining complementary data sources. Existing multi-modal discriminative models often use a uniform learning objective applied equally to all input modalities. However, this can lead to a problem where some uni-modal representations become under-optimized because one modality dominates the learning process. In other words, the model can focus too much on the most informative modality, causing other modalities to contribute less effectively to overall performance. This imbalance not only reduces the modelâs ability to fully leverage complementary information between modalities but also makes it vulnerable to noise or missing data in the dominant modality. Addressing learning imbalance is therefore crucial to develop robust multi-modal models that can integrate diverse data sources effectively and improve generalization. In this work, we present a balanced soft mixture-of-experts (SMoE) model designed to address the limitations of existing methods. The architecture comprises three experts, two uni-modal experts, each processing a single modality, and one multi-modal expert that jointly processes both modalities. Expert weights are produced by a CNN-based gating network, which assigns weights to the three experts that sum up to one. To promote equitable contributions from all experts, we incorporate a coefficient-of-variation based load balancing loss. The final prediction is computed as a weighted average of the output of all experts according to the gating networkâs assignments. 2 Results 2.1 Dataset Collection Figure 1: Distribution of glaucoma and non-glaucoma cases across FairDomain, FairVision, and HarvardGF datasets. Figure 1 illustrates the distribution of glaucoma and non-glaucoma cases across the FairDomain, FairVision, and HarvardGF datasets. Each pie chart represents the proportion of subjects labeled with glaucoma versus those without it, based on visual field test results. In FairDomain, 44.53% of the subjects were diagnosed with glaucoma, while 55.47% were not. The FairVision dataset shows a slightly higher prevalence of glaucoma at 48.7%, with non-glaucoma subjects comprising 51.3% of the population. In contrast, HarvardGF contains the highest proportion of glaucoma cases at 53.4%, with non-glaucoma subjects making up 47.6%. These distributions highlight the variability in disease prevalence across datasets, which is important to consider when evaluating model performance and generalizability in glaucoma detection tasks. In this study, we used three datasets FairVision, FairDomain, and HarvardGF, each comprising paired samples per subject to support multi-modal learning. The FairVision dataset [10] includes 30,000 subjects, with 10,000 each for age-related macular degeneration (AMD), diabetic retinopathy (DR), and glaucoma. Among these, 48.7% are diagnosed with glaucoma and 51.3% are considered normal. All subjects underwent visual field (VF) testing, which can bias the sample toward more glaucoma cases; however, VF-based labeling is more reliable and consistent than clinician judgment. The demographic characteristics of the FairVision dataset are as follows: the average age is 64.1 ± 17.0 years; gender distribution is 57.1% female and 42.9% male; racial composition includes 78.6% White, 13.7% Black, and 7.7% Asian; 3.8% of subjects identify as Hispanic and 96.2% as non-Hispanic; preferred language is 91.6% English, 1.8% Spanish, and 5.9% other languages; and marital status includes 24.6% single, 7.0% divorced, 0.9% legally separated, and 8.6% widowed. Each dataset includes three types of data: (1) Retinal imaging data, which consists of both 2D scanning laser ophthalmoscopy (SLO) fundus images that capture the retinal surface and 3D optical coherence tomography (OCT) scans that measure the in-depth structure of retinal layers. Both imaging modalities are effective for assessing retinal abnormalities caused by eye diseases, although OCT is generally regarded superior in diagnostic accuracy; (2) demographic identity group information; and (3) disease diagnoses for the three major retinal conditions AMD, DR, and glaucoma that together affect more than 380 million people worldwide. The FairDomain [11] dataset includes subjects tested between 2010 and 2021 at a large academic eye hospital affiliated with Harvard Medical School. It contains data for both medical segmentation and classification tasks. For medical classification, the dataset comprises four types of data: (1) en-face fundus imaging scans, (2) scanning laser ophthalmoscopy (SLO) fundus imaging scans, (3) patient demographic information, and (4) glaucoma diagnoses. Subjects in this dataset are classified into two categoriesânormal and glaucomaâbased on visual field test results. The classification subset consists of 10,000 samples from 10,000 individual subjects, with an average age of 60.9 ± 16.1 years. This dataset is split into 8,000 samples for training and 2,000 for testing. It includes six demographic attributes: age, gender, race, ethnicity, preferred language, and marital status. The gender distribution is 72.5% female and 27.5% male. The racial distribution includes 76.8% White, 14.5% Black, and 8.7% Asian. The ethnicity is reported as 96.0% non-Hispanic and 4.0% Hispanic. The preferred language includes 92.6% English, 1.7% Spanish, 3.6% other languages, and 2.1% unknown. The marital status distribution is as follows: 58.5% married or partnered, 26.1% single, 6.9% divorced, 0.8% legally separated, 1.9% widowed, and 5.8% unknown. The HarvardGF [12] dataset comprises subjects tested between 2010 and 2021 in a large academic eye hospital. It includes three types of data: (1) optical coherence tomography (OCT) scans, (2) patient demographic information, and (3) glaucoma diagnosis based on visual field (VF) tests. The OCT data consist of 2D retinal nerve fiber layer thickness (RNFLT) maps and 3D OCT-B scans. OCT is a state-of-the-art 3D imaging technique widely used to diagnose eye diseases such as glaucoma, age-related macular degeneration (AMD), and diabetic retinopathy (DR). For glaucoma assessment, the 2D RNFLT map is derived from the 3D OCT-B scans and represents the vertical distance between the inner limiting membrane and the retinal nerve fiber layer boundaries, segmented using the manufacturerâs software. This 2D RNFLT map, a standard structural measurement in clinical practice, has a resolution of 200Ă200200Ă 200 pixels and covers a 6Ă6âm26Ă 6~m^2 area centered on the optic disc, with RNFLT values ranging from 0 to 350 microns. To ensure high image quality, OCT scans with a signal strength below 6 (on a scale where 10 denotes the highest quality) were excluded. The dataset also includes 3D OCT-B scans to complement the 2D structural data. Demographic data encompass age, gender, race, ethnicity, language proficiency, and marital status. As the dataset is specifically curated to study racial fairness, the number of subjects was balanced across racial groups using the hospitalâs dataset. Glaucoma diagnoses were determined based on the VF test performed using the Humphrey Field Analyzer, which assesses visual sensitivity within a 24-degree radius from the central fixation point in each eye. Only reliable VF tests, as defined by clinically accepted quality control criteria and manufacturer guidelines, were included to maintain diagnostic consistency. 2.2 Glaucoma Detection Results Table 1: Overall AUC (â ) for three datasets, grouped by method and modality for Uni-Modal Dataset Method Modality Type Overall AUC Fairvision DenseNet Slo Fundus 78.94± 0.0080 Oct Bscans 80.86± 0.0133 EfficientNet Slo Fundus 78.74± 0.0055 Oct Bscans 81.16± 0.0056 ResNet Slo Fundus 76.32± 0.0043 Oct Bscans 81.90± 0.0046 ViT-B Slo Fundus 79.52± 0.0033 Oct Bscans 81.78± 0.0046 Fairdomain DenseNet Slo Fundus 80.84± 0.0064 Oct Fundus 81.70± 0.0045 EfficientNet Slo Fundus 80.02± 0.0058 Oct Fundus 81.50± 0.0050 ResNet Slo Fundus 79.36± 0.0015 Oct Fundus 79.24± 0.0120 ViT-B Slo Fundus 71.86± 0.0048 Oct Fundus 69.24± 0.0038 HarvardGf DenseNet Rnflt 84.94± 0.0062 Oct Bscans 78.52± 0.0259 EfficientNet Rnflt 85.32± 0.0018 Oct Bscans 78.60± 0.0177 ResNet Rnflt 83.66± 0.0127 Oct Bscans 79.78± 0.0081 ViT-B Rnflt 83.42± 0.0055 Oct Bscans 79.94± 0.0096 Table 1 presents the overall Area Under the Curve (AUC) scores achieved by the baseline models in the three datasets. FairVision, HarvardGF, and FairDomain. The baseline models evaluated include DenseNet [13], EfficientNet [14], ResNet [15], and ViT-B [16]. Each model was trained and tested on individual imaging modalities within each dataset to assess performance consistency and generalizability. Across all datasets and modalities, EfficientNet consistently demonstrates strong performance, often achieving the highest or near-highest AUC values. In the FairVision dataset, EfficientNet yields the highest AUC of 81.16% on OCT-B scans, outperforming other models in this modality. Similarly, in FairDomain, EfficientNet performs competitively, particularly on OCT fundus images with an AUC of 81.50%, closely following DenseNet. In particular, on the HarvardGF dataset, EfficientNet achieves the best overall performance with an AUC of 85.32% and 78.60% on RNFLT maps and OCT-B scans, indicating its robustness in handling structural retinal measurements. The superior performance of EfficientNet can be attributed to its compound scaling method, which systematically and uniformly scales the depth, width, and input resolution. This design principle allows EfficientNet to balance the trade-off between model complexity and representational capacity, leading to enhanced generalization across diverse imaging modalities and datasets. In contrast, ResNet, while a widely used and proven architecture, demonstrates more variable performance. In the FairVision dataset, ResNet achieves a relatively competitive AUC of 81.90% on OCT-B scans, slightly outperforming EfficientNet in that specific modality. However, its performance deteriorates in FairDomain, particularly in OCT fundus images where the AUC drops to 79.24%, indicating potential limitations in capturing modality-specific retinal characteristics. Furthermore, although ResNet performs reasonably well in the HarvardGF dataset, achieving 83.66% on RNFLT and 79.78% on OCT B-scans, it does not match the consistency observed with EfficientNet. ViT-B, while representing a modern transformer based approach, shows the most variability among the models, especially in FairDomain, where it records a notable decline in AUC (69.24%) for OCT fundus images. This suggests a higher sensitivity to dataset characteristics and modality, possibly due to its reliance on larger data volumes and positional encoding mechanisms, which may not generalize as effectively in limited medical datasets. Overall, these results indicate that EfficientNet is the most robust and reliable model for uni-modal retinal image analysis. Its consistently high performance across modalities and datasets underscores its ability to extract discriminative features while maintaining computational efficiency. Table 2: Overall AUC (â ) for three datasets, grouped by method and modality for Multi-Modal Dataset Method Overall AUC Fairvision Normal 81.86± 0.0113 OPM 82.62± 0.0057 OGM 82.18± 0.0078 CGGM 81.98± 0.0092 OPM+OGM 82.54± 0.0035 SMoE 83.29± 0.0045 Fairdomain Normal 83.58± 0.0031 OPM 84.28± 0.0027 OGM 83.94± 0.0086 CGGM 83.22± 0.0028 OPM+OGM 84.30± 0.0048 SMoE 84.31± 0.0029 HarvardGf Normal 85.60± 0.0056 OPM 85.30± 0.0048 OGM 85.48± 0.0045 CGGM 84.96± 0.0088 OPM+OGM 85.46± 0.0045 SMoE 86.33± 0.0026 Table 2 presents the overall AUC results for various multi-modal (M) learning approaches across the FairVision, FairDomain, and HarvardGF datasets. The âNormalâ baseline refers to a conventional multi-modal model trained without any explicit strategy for balancing modality contributions. OPM (On-the-fly Prediction Modulation) [17] represents a forward-pass modulation technique designed to promote balanced learning between modalities. In contrast, OGM (On-the-fly Gradient Modulation) [18] adjusts the gradient flow during backpropagation to address the imbalance in learning dynamics. CGGM (Classifier-Guided Gradient Modulation) [19] extends this idea by using class-specific gradient signals to guide balanced optimization across modalities. The combined method, OPM ++ OGM, integrates prediction modulation in the forward pass with gradient modulation in the backward pass to enhance learning balance throughout the training process. Inspiring from the DynMM [20], we propose the SMoE (Soft Mixture-of-Experts), which employs a soft expert assignment mechanism with a load-balancing objective to encourage uniform expert utilization across different data modalities. This method aims to adaptively specialize expert branches while preventing mode collapse or expert under utilization, resulting in improved and more robust multi-modal learning. Overall, most of the balanced multi-modal strategies demonstrate improvements over the conventional baseline model. However, there are notable exceptions where certain methods fail to outperform the baseline. Specifically, CGGM performs less poorly than the normal model in FairDomain (83.22% vs. 83.58%) and HarvardGF (84.96% vs. 85.60%), while OPM performs slightly worse in HarvardGF (85.30% vs. 85.60%). Furthermore, OGM shows a performance drop in FairDomain (83.94%) compared to the normal model (83.58%). These results suggest that, while balancing mechanisms can enhance learning dynamics, their effectiveness may be dataset dependent and not generalizable in all settings. Among modulation based approaches, the combined method OPM ++ OGM achieves the most consistent improvements, slightly outperforming individual components in FairVision (82.54%) and FairDomain (84.30%), and remains competitive on HarvardGF (85.46%). This indicates that jointly optimizing forward and backward learning dynamics contributes to more stable and generalizable performance across datasets. The best performing method in all datasets is the proposed SMoE model, which consistently achieves the highest AUC scores: 83.29% in FairVision, 84.31% in FairDomain, and 86.33% in HarvardGF. These results reflect not only strong predictive performance, but also a high degree of robustness across different data distributions and modality combinations. The consistent gains observed with SMoE suggest that its design effectively mitigates modality imbalance while leveraging diverse expert representations, leading to a more balanced and adaptive multi-modal learning process. 3 Discussions Deep learning approaches have been increasingly adopted to analyze clinical data such as VF, fundus image, and OCT scans for automated glaucoma detection [21, 22]. Recent developments in multi-modal learning have advanced glaucoma detection by integrating information from diverse modalities such as fundus photography, optical coherence tomography (OCT), visual field data, and clinical metadata. These approaches exploit the complementary nature of structural and functional data to improve diagnostic accuracy. Hwang et al. [8] proposed a neural network that jointly learns from fundus images, OCT, and visual field tests using modality-specific encoders and a fusion network to improve disease classification. Huang et al. [23] introduced a probabilistic deep learning framework that combines fundus and visual field modalities to improve the reliability of the model. Li and Pun [24] designed ELF, a fusion model that integrates local and global features of OCT and fundus modalities. Cai et al. [25] proposed COROLLA, which uses contrastive learning to align multi-modal representations from fundus and OCT-derived maps. Li et al. [26] further extended multi-modal modeling by incorporating fundus images, segmentation features, and patient metadata through a three-branch architecture. These methods collectively highlight the value of multi-modal learning in capturing complex pathological signatures, enhancing generalization, and improving clinical relevance in glaucoma detection systems. In multi-modal glaucoma detection, imbalanced learning occurs when certain modalities, typically those with stronger or more easily optimized features, dominate the training process, while others contribute minimally. This imbalance leads to the under utilization of valuable complementary information and limits the modelâs ability to fully capture the diseaseâs complexity. The issue arises during both the forward and backward phases of training: dominant modalities produce overconfident predictions and receive disproportionately large gradient updates, while weaker modalities are overshadowed. To address this, balanced multi-modal learning strategies introduce mechanisms such as on-the-fly prediction modulation, which suppresses overconfident predictions from dominant modalities, and gradient modulation, which adjusts the learning signal to ensure that all modalities are fairly trained. CGGM addresses this issue by introducing a classifier-aware mechanism that modulates gradients during backpropagation, ensuring that each modality contributes proportionately to the learning process. By aligning gradients based on both their strength and directional agreement with the classifierâs objective, CGGM promotes stable optimization and encourages more uniform feature learning. These strategies promote more equitable representation learning, leading to improved generalization, robustness, and clinical reliability in diverse patient cases. The proposed soft mixture-of-experts (SMoE) model with three experts offers several key advantages over conventional balanced multi-modal learning strategies. First, it enables modality-specific specialization, allowing each expert to focus on learning representations from distinct input types such as SLO, OCT, or their combination. This design contrasts with shared representation models, which often struggle to capture modality-specific nuances. Second, SMoE employs a soft expert routing mechanism that adaptively determines the contribution of each expert based on the input, allowing the model to dynamically adjust its reliance on different modalities at a per-sample level. This flexibility is especially beneficial for handling the inherent variability in clinical imaging data. Lastly, SMoE demonstrates robustness to modality noise or dropout, as it does not assume equal importance of all modalities for every instance. Instead, it can reduce noisy or less informative modalities, making it particularly suitable for real-world settings where data quality and completeness can vary significantly across patients. The performance of SMoE, as shown in Table 2, reflects these architectural advantages. SMoE consistently achieves the highest overall AUC in the three datasets, with 83.29% in FairVision, 84.31% in FairDomain, and 86.33% in HarvardGF. These results surpass those of all baseline and balanced multi-modal learning methods, including OPM, OGM, CGGM, and their combinations. Consistent gains in datasets underscore the modelâs ability to generalize well despite differences in data distribution and modality characteristics. In additon, the narrow standard deviations observed indicate stable performance between runs, reinforcing SMoEâs robustness. Together, these findings validate the effectiveness of soft expert specialization and adaptive routing in improving both accuracy and reliability in multi-modal clinical prediction tasks. However, the proposed equitable model and its evaluation come with several limitations. First, the model does not explicitly optimize for demographic fairness, which may result in inconsistent fairness outcomes across different demographic subgroups. Second, the SMoE design in this study is based primarily on the EfficientNet backbone, chosen due to its superior performance compared to other baseline models. However, it would be valuable to explore more powerful model architectures and training paradigms, such as adapting pre-trained foundation models [27, 28, 29, 30], which could improve performance through low-rank adaptation techniques [31]. Third, SMoE introduces additional model complexity by incorporating multiple expert branches and a gating mechanism, leading to a larger parameter space and increased computational requirements compared to conventional balanced multi-modal approaches [32, 33]. This added complexity may hinder deployment in resource-constrained clinical environments. Finally, SMoE assumes that all data modalities are available during training, which may not be the case in real-world settings. When modality data are missing, additional mechanisms such as modality imputation or masked expert dropout would be needed, which could increase training complexity and raise concerns about model robustness [34]. In conclusion, this work presents a comprehensive framework for addressing modality imbalance in multi-modal glaucoma detection through a soft mixture-of-experts (SMoE) model. By enabling modality-specific specialization and adaptive expert routing, SMoE effectively mitigates the limitations of prior balanced learning strategies and consistently outperforms baseline methods across multiple datasets. The results highlight its robustness, generalizability, and clinical relevance in handling diverse and variable imaging data. While the model introduces additional complexity and relies on complete modality availability, its design offers a promising direction for equitable and high-performing multi-modal learning. Future work will focus on improving model efficiency, incorporating explicit fairness constraints, and adapting to foundation model-based paradigms for broader clinical applicability. 4 Methods 4.1 Comparitive methods In this study, we have used four balanced multi-modal methods, OPM, OGM, CGGM, and the combination of OPM and OGM. In the following, we briefly describe the OPM, OGM, and CGGM methods. All of these methods use the EfficientNet B1 model as the backbone. Figure 2: The pipeline of the On-the-fly Prediction Modulation. Here we take SLO Fundus and OCT SCANS. In the feed-forward stage, the feature of modality m (SLO Fundus) is randomly dropped with probability qmq^m, where the probability is determined by the discriminative discrepancy ratio at the last iteration. Via OPM, the remained feature of suppressed modality could affect the multi-modal prediction more, accordingly improving its learning. On-the-fly Prediction Modulation (OPM) is a method designed to balance the contributions of different modalities during multi-modal learning. During training, it evaluates how dominant each modality is by measuring its discriminative power (i.e., how well it can predict the target on its own). If a modality is found to be too dominant, OPM randomly drops its features in the forward pass with a probability proportional to its dominance. This encourages the model to pay more attention to the weaker modalities, leading to more balanced feature learning across the modalities and better overall performance. Figure 2 describes the OPM pipeline. Figure 3: The pipeline of the On-the-fly Gradient Modulation strategy. Here we take SLO Fundus and OCT SCANS. In the back-propagation stage, the gradient of modality m is modulated with km, which is determined by the discriminative discrepancy ratio at this iteration. Via OGM, the gradient of modality with more discriminative information is weakened, while the remained modality is not affected and can gain more training. On-the-fly gradient modulation with Gaussian enhancement (OGM_GE) combines two techniques to address modality imbalance during multi-modal training. First, Onâtheâfly Gradient Modulation (OGM) dynamically reduces the optimization emphasis on dominant modalities by scaling down their gradients based on their uni-modal performance discrepancy. Second, Gaussian enhancement (GE) adds adaptive Gaussian noise to modulated gradients to restore stochasticity lost by suppression and improve generalization. Together, OGM_GE ensures that weaker modalities become more optimized while preserving overall robustness and boosting both uni-modal and multi-modal performance. Figure 3 describes the OGM pipeline. Figure 4: The overall architecture of CGGM. During the training stage, classifiers are introduced to calculate the directions of uni-modal gradients. During the inference stage, the classifiers are discarded. Classifier-Guided Gradient Modulation (CGGM) is a technique proposed by Guo et al.. (2024) for balancing multi-modal training by adaptively adjusting each modalityâs contribution in backpropagation. Unlike prior methods that only scale the magnitude of the gradient, CGGM also considers the direction of the gradient, using guidance from a classifier to align or alter modality-specific gradients so that they contribute constructively toward the final objective of the task. It thereby curbs over-reliance on dominant modalities and encourages underutilized ones to influence learning more effectively. Figure 5 describes the architecture of CGGM. 4.2 The Proposed Balanced Soft Mixture of expert model The proposed approach consists of three experts, namely the SLO Fundus expert, OCT Scans expert, and multi-modal expert (considering the modalities from Fairvision dataset). Specifically, all of these experts use the EfficientB1 neural network as the backbone.EfficientNet-B1 is a member of the EfficientNet family, which is known to achieve state-of-the-art accuracy while maintaining exceptional computational efficiency. The architecture builds on a baseline network, EfficientNet-B0, discovered through neural architecture search [35], and scales it in a balanced and principled manner to enhance performance. EfficientNet-B1 represents the first scaled-up variant in the series, offering improvements in depth, width, and resolution without disproportionately increasing computational cost. The architecture of EfficientNet-B1 is organized into seven distinct blocks, each designed to progressively extract higher-level visual features. These blocks consist of multiple layers of Mobile Inverted Bottleneck Convolution (MBConv), originally introduced in MobileNetV2 [36]. Each MBConv layer follows an inverted structure in which the input is first expanded to a higher-dimensional space, processed using efficient depthwise separable convolutions [37], and then projected back to a lower dimension. To further boost the representational power, each MBConv includes squeeze-and-excitation modules [38], which dynamically recalibrate channel-wise feature responses and help the network focus on the most informative elements of the input. EfficientNet-B1 begins with a standard convolution and max-pooling layer, which prepares the input for deeper processing. As input progresses through the blocks, the network gradually increases the number of filters and kernel sizes, allowing it to transition from capturing fine-grained details in the early layers to more abstract semantic patterns in the later stages. This hierarchical flow of information supports efficient and robust feature extraction in varying image complexities. The defining feature of EfficientNet is its compound scaling method, which uniformly scales the network depth (number of layers), width (number of channels), and input resolution in a coordinated way. Unlike traditional approaches that scale only one dimension, compound scaling adjusts all three in a balanced fashion, ensuring that the modelâs increased capacity is utilized effectively. EfficientNet-B1 applies this method with a moderate scaling factor, making it larger and more powerful than EfficientNet-B0 while still maintaining practical runtime performance. This strategy allows the network to preserve its efficiency gains while delivering improved accuracy across a range of image classification tasks. Although EfficientNet excels at extracting hierarchical features from a single-modality input, it is inherently limited to the information content of that modality alone. In contrast, multi-modal learning integrates complementary information from diverse data sources, such as combining fundus images with OCT scans in ophthalmology, allowing the model to capture a richer and more discriminative representation of the underlying features. This fusion enhances robustness, improves generalization across varying conditions, and often leads to superior performance in complex visual recognition tasks where single-modality cues may be insufficient. Balancing contributions from different modalities is a key challenge in multi-modal learning. Methods like On-the-fly Prediction Modulation (OPM) and On-the-fly Gradient Modulation (OGM) attempt to address this by modulating forward activations or gradients based on modality discrepancy power. However, OPM improves inference but ignores gradient imbalance, while OGM corrects gradients but may suppress strong modalities. Classifier-Guided Gradient Modulation (CGGM) adjusts the gradient flow to each modality based on the softmax confidence of its corresponding classifier, promoting stronger learning from more confident modalities. However, when classifiers produce similar or low confidence scores, especially under ambiguous inputs, CGGM can introduce noise or instability in gradient updates, reducing its reliability in balancing learning across modalities. Even OPM+OGM shows inconsistent behavior across tasks. To address these issues, we introduce a soft Mixture-of-Experts (soft MoE) model that assigns adaptive, learnable weights to modality-specific experts. The proposed balanced soft MoE is shown in figure [5]. It contains the gating network and three experts. First, the input (two image modalities) is given to the gating network. The gating network is a lightweight convolutional network designed to compute adaptive weights for modality specific experts in a soft Mixture-of-Experts (MoE) framework. It processes fused multi-modal features using a two-layer convolutional block with batch normalization and Tanh activation to extract compact and discriminative representations. The design is flexible to support different modality combinations. The final output is produced by a 1Ă1 convolutional layer that maps the hidden features to a vector of length equal to the number of experts. This vector represents the raw gating logits, which can be converted into soft or hard weights to control the contribution of each expert during training and inference. Next, each expert takes itâs input and gives the predictions. Finally, the output is generated by computing the weighted average of expert predictions using weights generated by the gating network. In soft Mixture-of-Experts (MoE) models, gating networks assign weights to different experts based on input relevance. However, without explicit regulation, the gating mechanism may collapse to favor a small subset of experts across most samples, under utilizing others. This imbalance limits the capacity of the model, reduces the diversity of learned representations, and can lead to overfitting of dominant branches. To mitigate this, a load balancing loss is introduced to encourage uniform expert activation across the batch. By penalizing skewed expert usage, this loss promotes better distribution of learning signals, ensures that all experts contribute meaningfully during training, and enhances generalization and robustness of the overall model. We employ a load balancing loss based on the coefficient of variation (CV), which measures the relative dispersion of expert usage across a mini-batch. Specifically, for each expert, the gating network produces a weight per sample; aggregating these weights over the batch yields an expert-wise usage distribution. The CV is computed as the ratio of the standard deviation to the mean of this distribution, capturing how unevenly the experts are selected. A high CV indicates that some experts dominate, while others are rarely used. CV-based loss penalizes this imbalance by minimizing CV, thereby encouraging more uniform expert activation and preventing mode collapse. This regularization improves learning dynamics, promotes specialization across experts, and leads to more effective and stable training in multi-modal settings. Equation 1 describes the CV based loss. Where gig_i represents the total gating weight assigned to expert i in a batch, capturing how much that expert is used. The term ÎŒ=1NEââi=1NEgiÎŒ= 1N_E _i=1^N_Eg_i is the mean gating load in all NEN_E experts, and Ï is the standard deviation of these gating loads, measuring the variability in expert usage. The ratio ÏÎŒ ÏÎŒ is the coefficient of variation (CoV), and squaring it yields a normalized measure of imbalance. The scalar λdyn _dyn is a hyperparameter that controls the influence of this loss term during training. Figure 5: The overall architecture of SMoE. Here we take SLO Fundus and OCT Scans. Gating network generates the weights for the three experts. Each of the expert predictions is multiplied with respective weight and then added to get the final prediction. âbalance=λdynâ (ÏÎŒ)2=λdynâ 1ÎŒ2âNEââi=1NE(giâÎŒ)2L_balance= _dyn· ( ÏÎŒ )^2= _dyn· 1ÎŒ^2N_E _i=1^N_E(g_i-ÎŒ)^2 (1) 4.3 Parameter Settings Following Fairvision settings [11], all convolutional neural network (CNN) models, including EfficientNet, DenseNet, and ResNet, were trained for 10 epochs with a batch size of 6 and an initial learning rate of 1Ă10â41Ă 10^-4. For the Vision Transformer (ViT), we trained for 50 epochs using a batch size of 64, a base learning rate of 5Ă10â45Ă 10^-4, a layer-wise learning rate decay of 0.55, and a drop path rate of 0.01. Hyperparameters for OPM, OGM, OPM+OGM, and CGGM were selected based on value ranges recommended in prior work. For our proposed SMoE method, we set the regularization coefficient for the load balancing loss to 0.001. All models were optimized using the AdamW optimizer with a weight decay of 0.01. 5 Acknowledgements The work was supported by NSF OIA-1946231, CNS-2117785, and OIA-2229752. 6 Author contributions MS, SVC, and XH conceived the study. SVC and MS developed deep learning models. SVC and KR performed data processing, experiment, and analysis. MS, KR, and XH contributed materials and clinical expertise. MS and XH supervised the work. All authors wrote and revised the manuscript. All authors have read and approved the manuscript. 7 Competing Interests The authors declare no competing interests. References Glaucoma Research Foundation [2025] Glaucoma Research Foundation: Glaucoma Facts and Stats. https://glaucoma.org/articles/glaucoma-facts-and-stats. Accessed: 2025-07-09; Last updated March 12, 2025 (2025) Mayo Clinic [2025] Mayo Clinic: Glaucoma: Symptoms and Causes. https://w.mayoclinic.org/diseases-conditions/glaucoma/symptoms-causes/syc-20372839. Accessed: 2025-07-09 (2025) Sharma et al. [2008] Sharma, P., Sample, P.A., Zangwill, L.M., Schuman, J.S.: Diagnostic tools for glaucoma detection and management. Survey of Ophthalmology 53(Suppl 1), 17â36 (2008) https://doi.org/10.1016/j.survophthal.2007.11.008 . Review article; accessed via PubMed Central Phene et al. [2019] Phene, S., Dunn, R.C., Hammel, N., Liu, Y., Krause, J., Kitade, N., Schaekermann, M., Sayres, R., Singh, A., Easterbrook, E.B., Wu, D.J., Bora, A., Semturs, C., Misra, A., Huang, A.E., Spitze, A., Medeiros, F.A., Maa, A.Y., Gandhi, M., Corrado, G.S., Peng, L., Webster, D.R.: Deep learning and glaucoma specialists: The relative importance of optic disc features to predict glaucoma referral in fundus photographs. Ophthalmology 126(12), 1627â1639 (2019) https://doi.org/10.1016/j.ophtha.2019.07.024 Christopher et al. [2018] Christopher, M., Belghith, A., Bowd, C., Proudfoot, J.A., Goldbaum, M.H., Weinreb, R.N., Girkin, C.A., Liebmann, J.M., Zangwill, L.M.: Performance of deep learning architectures and transfer learning for detecting glaucomatous optic neuropathy in fundus photographs. Scientific Reports 8(1), 16685 (2018) https://doi.org/10.1038/s41598-018-35044-9 Mahmood et al. [2018] Mahmood, F., Sandhu, H., Garrido, I., Plant, G.T., Gupta, V., Teasdale, S., Lawrenson, J.G., Biousse, V., Newman, J.: Automated glaucoma diagnosis using multi-modal imaging and deep learning. PLOS ONE 13(12), 0207982 (2018) https://doi.org/10.1371/journal.pone.0207982 Li et al. [2020] Li, L., Xu, M., Liu, H., Li, Y., Wang, X., Jiang, L., Wang, Z., Fan, X., Wang, N.: A largeâscale database and a cnn model for attentionâbased glaucoma detection. IEEE Transactions on Medical Imaging 39(2), 413â424 (2020) https://doi.org/10.1109/TMI.2019.2927226 Hwang and et al. [2025] Hwang, D.K., al.: Multimodal neural networks for glaucoma detection using fundus, oct, and visual fields. American Journal of Ophthalmology (2025). Published online ahead of print via PubMed Du et al. [2024] Du, J., Cang, Y., Zhou, T., Hu, J., He, W.: Deep learning with hmâvgg: Ai strategies for multiâmodal image analysis. In: Proc. 3rd International Symposium on Sensor Technology and Control (ISSTC) (2024). https://doi.org/10.1109/ISSTC63573.2024.10824093 Luo et al. [2023] Luo, Y., Khan, M.O., Tian, Y., Shi, M., Dou, Z., Elze, T., Fang, Y., Wang, M.: Fairvision: Equitable deep learning for eye disease screening via fair identity scaling. arXiv preprint arXiv:2310.02492 (2023). Version v3 (revised 12 April 2024), cs.CV Tian et al. [2024] Tian, Y., Wen, C., Shi, M., Afzal, M.M., Huang, H., Khan, M.O., Luo, Y., Fang, Y., Wang, M.: Fairdomain: Achieving fairness in cross-domain medical image segmentation and classification. arXiv preprint arXiv:2407.08813 (2024) [eess.IV]. Version v2, last revised 18 July 2024 Luo et al. [2024] Luo, Y., Tian, Y., Shi, M., Pasquale, L.R., Shen, L.Q., Zebardast, N., Elze, T., Wang, M.: Harvard glaucoma fairness: A retinal nerve disease dataset for fairness learning and fair identity normalization. IEEE Transactions on Medical Imaging 43(7), 2623â2633 (2024) https://doi.org/10.1109/TMI.2024.3377552 Huang et al. [2017] Huang, G., Liu, Z., Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017) Tan and Le [2019] Tan, M., Le, Q.V.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: Proceedings of the 36th International Conference on Machine Learning (ICML), p. 6105â6114. PMLR, ??? (2019) He et al. [2016] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 770â778 (2016) Dosovitskiy et al. [2021] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (ICLR) (2021). arXiv:2010.11929 Wei et al. [2024] Wei, Y., Hu, D., Du, H., Wen, J.-R.: On-the-fly modulation for balanced multimodal learning. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024) https://doi.org/10.1109/TPAMI.2024.3468315 . Early access Peng et al. [2022] Peng, X., Wei, Y., Deng, A., Wang, D., Hu, D.: Balanced multimodal learning via onâtheâfly gradient modulation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 8238â8247 (2022) Guo et al. [2024] Guo, Z., Jin, T., Chen, J., Zhao, Z.: Classifier-guided gradient modulation for enhanced multimodal learning. In: Proceedings of the Thirty-Eighth Annual Conference on Neural Information Processing Systems (NeurIPS) (2024) Xue and Marculescu [2023] Xue, Z., Marculescu, R.: Dynamic multimodal fusion. In: Proceedings of the Multi-Modal Learning and Applications Workshop (MULA), CVPR (2023) Russakoff et al. [2020] Russakoff, D.B., Mannil, S.S., Oakley, J.D., et al.: A 3d deep learning system for detecting referable glaucoma using full oct macular cube scans. Translational Vision Science & Technology 9(2), 12 (2020) https://doi.org/10.1167/tvst.9.2.12 Christopher et al. [2020] Christopher, M., Bowd, C., Belghith, A., et al.: Deep learning approaches predict glaucomatous visual field damage from oct optic nerve head en face images and retinal nerve fiber layer thickness maps. Ophthalmology 127(3), 346â356 (2020) https://doi.org/10.1016/j.ophtha.2019.09.036 Huang et al. [2022] Huang, L., et al.: Deep learning with fundus and visual field fusion. Frontiers in Neuroscience 16, 939472 (2022) https://doi.org/10.3389/fnins.2022.939472 Li and Pun [2023] Li, J., Pun, C.M.: ELF: End-to-end local-global fusion of OCT and fundus for glaucoma grading. arXiv preprint arXiv:2311.08032 (2023). Version 1, cs.CV Cai et al. [2022] Cai, Y., et al.: COROLLA: Contrastive learning for multimodal glaucoma detection. arXiv preprint arXiv:2201.03795 (2022) Li et al. [2023] Li, Y., et al.: GMNNnet: A metadata-aware glaucoma classification framework. Scientific Reports 13, 2254 (2023) https://doi.org/10.1038/s41598-022-27045-6 Nguyen et al. [2023] Nguyen, D.M.H., Nguyen, H., Diep, N.T., Pham, T.N., Cao, T., Nguyen, B.T., Swoboda, P., Ho, N., Albarqouni, S., Xie, P., Sonntag, D., Niepert, M.: LVM-Med: Learning large-scale self-supervised vision models for medical imaging via second-order graph matching. In: Advances in Neural Information Processing Systems. NeurIPS, vol. 36, p. 27922â27950 (2023) Gupta et al. [2025] Gupta, A., Osman, I., Shehata, M.S., Braun, W..J., Feldman, R.E.: MedMAE: A self-supervised backbone for medical imaging tasks. Computation 13(4), 88 (2025) https://doi.org/10.3390/computation13040088 Sowrirajan et al. [2021] Sowrirajan, H., Yang, J., Ng, A.Y., Rajpurkar, P.: MoCo-CXR: Moco pretraining improves representation and transferability of chest x-ray models. In: Medical Imaging with Deep Learning (MIDL), p. 728â744 (2021) Zhou et al. [2023] Zhou, Y., Chia, M.A., Wagner, S.K., Ayhan, M.S., Williamson, D.J., Struyven, R.R., Liu, T., Xu, M., Lozano, M.G., Woodward-Court, P., et al.: A foundation model for generalizable disease detection from retinal images. Nature 622, 156â163 (2023) https://doi.org/10.1038/s41586-023-06555-x Zanella and Ben Ayed [2024] Zanella, M., Ben Ayed, I.: Low-rank few-shot adaptation of vision-language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 1593â1603 (2024) Shazeer et al. [2017] Shazeer, N., Mirhoseini, A., Maziarz, P., Davis, A., Le, Q., Hinton, G., Dean, J.: Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538 (2017) Yan et al. [2023] Yan, J., Xie, J., He, X., Li, X., Zha, H.: Mixture of experts in deep learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 45(9), 10795â10814 (2023) https://doi.org/10.1109/TPAMI.2022.3201094 BaltruĆĄaitis et al. [2019] BaltruĆĄaitis, T., Ahuja, C., Morency, L.-P.: Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence 41(2), 423â443 (2019) https://doi.org/10.1109/TPAMI.2018.2798607 Tan and Le [2019] Tan, M., Le, Q.V.: MnasNet: Platform-aware neural architecture search for mobile. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 2820â2828 (2019) Sandler et al. [2018] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.-C.: MobileNetV2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 4510â4520 (2018) Howard et al. [2017] Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017) Hu et al. [2018] Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 7132â7141 (2018)