Paper deep dive
On Impact of Loss Function on the Performance of Neural Networks in Melanoma Diagnosis
Morgan May, Pierpaolo Dondio, Simon Caton
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Melanoma is the deadliest type of skin cancer, whose early diagnosis is crucial for patients' survival. Image classification using deep learning models has shown promising results for melanoma diagnosis. However, the performance of these models on the melanoma datasets such as SIIM-ISIC melanoma classification dataset is a challenge due to the class imbalance. One of the methods to deal with this challenge is using loss function modifications. In this work, we have investigated the effect of different loss functions on the performance of deep neural networks. We trained these networks using focal loss, logit-adjusted softmax cross-entropy (CE) loss, and weighted softmax CE loss, and we report different metrics for evaluating performance and uncertainty calibration. Our results suggest that focal loss delivers a good combination of performance in terms of AUC and uncertainty calibration in terms of expected calibration error (ECE) simultaneously.
Tags
Links
- Source: https://arxiv.org/abs/2610.06139v1
- Canonical: https://arxiv.org/abs/2610.06139v1
Trouble viewing inline? Open PDF directly →
Full Text
12,113 characters extracted from source content.
Expand or collapse full text
On Impact of Loss Function on the Performance of Neural Networks in Melanoma Diagnosis Morgan May Affiliation: Technological University Dublin, Dublin, Ireland Affiliation: Research Ireland Centre for Research Training in Machine Learning, ML-Labs, Dublin, Ireland E-mail simon.caton@ucd.ie Pierpaolo Dondio Affiliation: Dublin City University, Dublin, Ireland Simon Caton Affiliation: University College Dublin, Dublin, Ireland Abstract Melanoma is the deadliest type of skin cancer, whose early diagnosis is crucial for patients’ survival. Image classification using deep learning models has shown promising results for melanoma diagnosis. However, the performance of these models on the melanoma datasets such as SIIM-ISIC melanoma classification dataset is a challenge due to the class imbalance. One of the methods to deal with this challenge is using loss function modifications. In this work, we have investigated the effect of different loss functions on the performance of deep neural networks. We trained these networks using focal loss, logit-adjusted softmax cross-entropy (CE) loss, and weighted softmax CE loss, and we report different metrics for evaluating performance and uncertainty calibration. Our results suggest that focal loss delivers a good combination of performance in terms of AUC and uncertainty calibration in terms of expected calibration error (ECE) simultaneously. Keywords: Melanoma diagnosis image classification imbalanced data loss function model performance uncertainty calibration. 1 Introduction Skin cancer is a common skin disease whose incidence and mortality rates have been showing yearly increases. Early diagnosis and timely treatment of skin cancer, especially for its deadliest form, malignant melanoma, remain a critical clinical issue globally [20]. Recently, deep learning has revolutionised melanoma diagnosis by building highly accurate classification models [2, 5, 8, 10, 13]. Melanoma diagnosis is an imbalance classification task, where in the available clinical datasets, the number of positive samples (malignant) is outnumbered by the negative samples (benign), because of the data scarcity of malignant cases. Imbalanced data poses a challenge for generalization on minority classes and makes naive learning biased towards the majority classes [11]. The methods to cope with this challenge can be divided into four groups [3]: data-level, algorithmic-level, hybrid of both and ensemble learning. One of the algorithmic level approaches is loss function modifications [1, 4, 9, 11]. Authors in [14] compared the effect of focal loss [9], weighted cross-entropy (CE) loss [16], and large margin-aware focal (LMF) loss [17] on the performance of EfficientNet-B3 on melanoma classification. They report the results with careful hyperparameter tuning and show that LMF loss delivers higher AUC compared to focal loss and weighted CE. In this work, we investigate the effect of different loss functions, including focal loss [9], logit adjusted softmax cross-entropy loss [11], and balanced softmax cross-entropy loss [15], on the performance of EfficientNet-B0, EfficientNet-B2, and EfficientNetV2-S by performing hyperparameter tuning using grid search, for each of the combinations of loss functions and models, and incorporating the evaluation of model uncertainty calibration. 2 Methodology This work evaluates the effects of the choice of loss function on the performance of neural networks in melanoma classification task. As such, it does not intend to compare the models against each other or maximize the performance of the models. Therefore, no additional steps such as ensembling is included. Dataset. We use the SIIM-ISIC 2020 Melanoma Classification dataset [7], which is imbalanced with an imbalance ratio of about 55. We used the 33126 labeled images in the original train set for a train/validation/test split of 80/10/10. Models. We train two EfficientNet-B0 and EfficientNet-B2 [18], and EfficientNetV2-S [19], from an improved generation of EfficientNets, designed for better parameter efficiency and faster training compared to its predecessor, EfficientNet. The models are pretrained on ImageNet. Loss Functions. We examine three loss functions that are used for imbalanced data: Balanced softmax cross-entropy (CE) [15], focal loss [9], and logit-adjusted softmax cross-entropy loss [11]. Performance metrics. We report AUC, F1-score, and recall for model performance evaluation, and expected calibration error (ECE) [6, 12] for the evaluation of confidence calibration [6]. Hyperparameter tuning. For hyperparameter tuning, we perform grid search, including the learning rate for each model, tuned from 1e−41e-4 to 3e−43e-4. For each of the loss functions, we perform individual grid searches for the hyperparameters specific to each of the loss functions, such as logit-adjustment temperature for logit-adjusted softmax CE loss, chosen from 0.5,0.8,1.0,1.1,1.2,1.5,2.0\0.5,0.8,1.0,1.1,1.2,1.5,2.0\, and alpha (the balancing parameter) and gamma (the focusing parameter) for focal loss, chosen from 0.25,0.5,0.75,0.98\0.25,0.5,0.75,0.98\ and 1.0,2.0,4.0\1.0,2.0,4.0\, respectively. Balanced softmax CE does not have any specific hyperparameters to tune. 3 Results and Discussion Table 1 summarizes the results of training the models using different loss functions on SIIM-ISIC 2020 melanoma classification task. Initial results do not suggest a strong trend across all models trained using different loss functions. However, some insights can be extracted on the performance of the models trained using different loss functions. For the bigger models, EfficientNet-B2 and EfficientNetV2-S, focal loss results in a higher AUC value compared to logit-adjusted softmax cross-entropy loss and balanced softmax cross-entropy loss, while maintaining a relatively low ECE value across all three models. This suggests that focal loss is a good choice, specially for training bigger models to have a good performance, while maintaining good confidence calibration, which is crucial for a safety critical application such as melanoma diagnosis. It can also be observed that for both EfficentNet-B0 and EfficientNet-B2, logit-adjusted softmax cross-entropy loss leads to a higher recall than focal loss and balanced softmax cross-entropy loss, while sacrificing the ECE value, which may result in overconfident predictions, which is not desirable for melanoma diagnosis as a safety-critical application. Table 1: Performance of different models on melanoma classification trained using different loss functions MODEL LOSS AUC F1 Recall ECE EfficientNet-B0 Focal Loss 0.8804 0.581 0.578 0.0142 Logit Adjusted Softmax CE 0.8840 0.532 0.740 0.0390 Balanced Softmax CE 0.8970 0.573 0.564 0.0212 EfficientNet-B2 Focal Loss 0.8909 0.581 0.585 0.0225 Logit Adjusted Softmax CE 0.8904 0.405 0.788 0.0913 Balanced Softmax CE 0.8803 0.589 0.624 0.0206 EfficientNetV2-S Focal Loss 0.9097 0.555 0.534 0.0163 Logit Adjusted Softmax CE 0.9027 0.626 0.623 0.0187 Balanced Softmax CE 0.8822 0.620 0.668 0.0213 4 Conclusion This work evaluates the effect of different loss functions on the performance of imbalanced melanoma classification. The results suggest that focal loss is a good choice for training EfficientNets and EfficientNetV2 for melanoma classification, for achieving a good performance in terms of AUC, while being well-calibrated in terms of ECE, which is crucial for the safety critical application of melanoma diagnosis. It will be interesting to see the effect of a wider variety of loss functions on the performance of a broader range of models for melanoma classification, and to evaluate the effect of a combination of multiple loss functions as the objective function for training the models. 5 Acknowledgment This publication has emanated from research conducted with the financial support of Taighde Éireann – Research Ireland under Grant number 18/CRT/6183. For the purpose of Open Access, the author has applied a C BY public copyright license to any Author Accepted Manuscript arising from this submission. References [1] K. Cao, C. Wei, A. Gaidon, N. Arechiga, and T. Ma (2019) Learning imbalanced datasets with label-distribution-aware margin loss. Advances in neural information processing systems 32. Cited by: §1. [2] B. Cassidy, C. Kendrick, A. Brodzicki, J. Jaworek-Korjakowska, and M. H. Yap (2022) Analysis of the isic image datasets: usage, benchmarks and recommendations. Medical image analysis 75, p. 102305. Cited by: §1. [3] W. Chen, K. Yang, Z. Yu, Y. Shi, and C. P. Chen (2024) A survey on imbalanced learning: latest research, applications and future directions. Artificial Intelligence Review 57 (6), p. 137. Cited by: §1. [4] Y. Cui, M. Jia, T. Lin, Y. Song, and S. Belongie (2019) Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 9268–9277. Cited by: §1. [5] A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, and S. Thrun (2017) Dermatologist-level classification of skin cancer with deep neural networks. nature 542 (7639), p. 115–118. Cited by: §1. [6] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger (2017) On calibration of modern neural networks. In International Conference on Machine Learning, p. 1321–1330. Cited by: §2. [7] International Skin Imaging Collaboration (2020) SIIM-isic 2020 challenge dataset. International Skin Imaging Collaboration. External Links: Document Cited by: §2. [8] R. Kaur, H. GholamHosseini, R. Sinha, and M. Lindén (2022) Melanoma classification using a novel deep convolutional neural network with dermoscopic images. Sensors 22 (3), p. 1134. Cited by: §1. [9] T. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár (2017) Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, p. 2980–2988. Cited by: §1, §1, §2. [10] D. Meedeniya, S. De Silva, L. Gamage, and U. Isuranga (2024) Skin cancer identification utilizing deep learning: a survey. IET Image Processing 18 (13), p. 3731–3749. Cited by: §1. [11] A. K. Menon, S. Jayasumana, A. S. Rawat, H. Jain, A. Veit, and S. Kumar (2021) Long-tail learning via logit adjustment. In International Conference on Learning Representations (ICLR), Cited by: §1, §1, §2. [12] M. P. Naeini, G. Cooper, and M. Hauskrecht (2015) Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 29. Cited by: §2. [13] H. Naseri and A. A. Safaei (2025) Diagnosis and prognosis of melanoma from dermoscopy images using machine learning and deep learning: a systematic literature review. BMC cancer 25 (1), p. 75. Cited by: §1. [14] S. Nazari and R. Garcia (2025) Going smaller: attention-based models for automated melanoma diagnosis. Computers in Biology and Medicine 185, p. 109492. Cited by: §1. [15] J. Ren, C. Yu, X. Ma, H. Zhao, S. Yi, et al. (2020) Balanced meta-softmax for long-tailed visual recognition. Advances in neural information processing systems 33, p. 4175–4186. Cited by: §1, §2. [16] O. Ronneberger, P. Fischer, and T. Brox (2015) U-net: convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, p. 234–241. Cited by: §1. [17] A. A. Sadi, L. Chowdhury, N. Jahan, M. N. S. Rafi, R. Chowdhury, F. A. Khan, and N. Mohammed (2022) LMFLOSS: a hybrid loss for imbalanced medical image classification. arXiv preprint arXiv:2212.12741. Cited by: §1. [18] M. Tan and Q. Le (2019) Efficientnet: rethinking model scaling for convolutional neural networks. In International conference on machine learning, p. 6105–6114. Cited by: §2. [19] M. Tan and Q. Le (2021) Efficientnetv2: smaller models and faster training. In International conference on machine learning, p. 10096–10106. Cited by: §2. [20] M. Wang, X. Gao, and L. Zhang (2025) Recent global patterns in skin cancer incidence, mortality, and prevalence. Chinese Medical Journal 138 (02), p. 185–192. Cited by: §1.