Paper deep dive
Calibrated Selective Prediction Using Deep Ensembles for ROI-Based Thyroid Nodule Ultrasound Classification Under Dataset Shift: A Retrospective Evaluation
Md. Sadibul Hasan Sadib, Md. Mohayminul Mukit, Rahmatul Kabir Rasel Sarker, Tahmid Alam Tamim, Md. Monir Hossain Shimul
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/15/2026, 4:16:21 AM
Summary
This study introduces a calibrated deterministic five-member deep ensemble framework for ROI-based thyroid nodule ultrasound classification. Utilizing ConvNeXt-Tiny with squeeze-and-excitation attention, the model employs mutual information as an ensemble-disagreement score to implement a conservative three-tier selective prediction policy (No-FNA suggestion, FNA recommendation, or radiologist review). Internal evaluation on the TN5000 dataset demonstrates strong discrimination and calibration (AUC-ROC 0.9395, ECE 0.0088). However, external evaluation on the TN3K dataset reveals significant performance and calibration degradation due to dataset shift, highlighting limited threshold transportability and the necessity for local recalibration and prospective clinical validation before deployment.
Entities (8)
Relation Signals (8)
ConvNeXt-Tiny → backboneof → Deep Ensemble
confidence 96% · ConvNeXt-Tiny was selected as the backbone for the subsequent calibrated ensemble experiments
Deep Ensemble → achievesmetric → AUC-ROC
confidence 95% · the ensemble achieved AUC-ROC 0.9395... On TN3K, AUC-ROC decreased to 0.7870
TN5000 → usedfordevelopment → Deep Ensemble
confidence 95% · TN5000 was used for model development, internal five-fold cross-validation, member-wise probability calibration, and fold-specific threshold selection
TN3K → usedforexternalevaluation → Deep Ensemble
confidence 95% · TN3K served as an independent external dataset for evaluation under dataset shift
Mutual Information → measures → Ensemble Disagreement
confidence 94% · mutual information (MI) as an ensemble-disagreement score for deferral
Deep Ensemble → implements → Three-tier Policy
confidence 93% · The framework used... A three-tier policy assigned images to No-FNA suggestion, FNA recommendation, or radiologist review
Three-tier Policy → defersto → Radiologist Review
confidence 92% · Images with high mutual information are deferred to radiologist review
Dataset Shift → causesdegradationin → Calibration
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Background: Deep learning models can classify thyroid nodules on ultrasound, but reliable clinical decision support also requires calibrated probabilities, uncertainty estimation, and selective referral, particularly under dataset shift. Methods: We developed a calibrated deterministic five-member deep ensemble for ROI-based thyroid nodule classification and selective image-based triage. TN5000 was used for model development, five-fold cross-validation, member-wise vector-scaling calibration, and fold-specific threshold selection. TN3K served as an independent external dataset-shift evaluation. The framework used ConvNeXt-Tiny with squeeze-and-excitation attention, ensemble-mean malignancy probability, and mutual information (MI) as an ensemble-disagreement score. A three-tier policy assigned images to No-FNA suggestion, FNA recommendation, or radiologist review. Results: On pooled out-of-fold TN5000 predictions, the ensemble achieved AUC-ROC 0.9395, AP 0.9715, ECE 0.0088, and Brier score 0.0813. At 50% nominal MI retention, 7.2% of cases received a No-FNA suggestion, 39.9% an FNA recommendation, and 52.9% radiologist review, with 98.3% No-FNA NPV and 99.83% malignancy capture. On TN3K, AUC-ROC decreased to 0.7870, AP to 0.7254, ECE increased to 0.1899, and Brier score to 0.2281. The frozen TN5000 policy assigned 83.7% to review, 1.0% to No-FNA, and 15.3% to FNA recommendation. No malignant image entered the No-FNA pathway, but FNA-recommendation PPV fell to 76.6%. Conclusion: The framework showed strong internal discrimination and calibration, but limited external threshold transportability. Selective prediction may help identify images unsuitable for automated triage, but local recalibration, threshold validation, and prospective clinical evaluation are required before deployment.
Tags
Links
- Source: https://arxiv.org/abs/2607.12075v1
- Canonical: https://arxiv.org/abs/2607.12075v1
Trouble viewing inline? Open PDF directly →
Full Text
77,790 characters extracted from source content.
Expand or collapse full text
Calibrated Selective Prediction Using Deep Ensembles for ROI-Based Thyroid Nodule Ultrasound Classification Under Dataset Shift: A Retrospective Evaluation Authors: Md. Sadibul Hasan Sadib 1 , Md. Mohayminul Mukit 1 , Rahmatul Kabir Rasel Sarker 1* , Tahmid Alam Tamim 1 , Md. Monir Hossain Shimul 2,3 Affiliation: 1 Department of Computer Science and Engineering, Daffodil International University, Dhaka, Bangladesh. 2 Department of Public Health, Daffodil International University, Dhaka, Bangladesh. 3 Faculty of Health, Education and Life Sciences, Birmingham City University, Birmingham, United Kingdom. *Corresponding author: Rahmatul Kabir Rasel Sarker Lecturer, Department of Computer Science and Engineering, Daffodil International University, Savar, Dhaka, Bangladesh. Email: shimul2307273028@diu.edu.bd Abstract Background: Deep learning models have shown promise for thyroid nodule ultrasound classification, but high discrimination alone is insufficient for clinical decision support. Probability calibration, uncertainty estimation, and selective referral are important when automated image-based triage suggestions may be unreliable, especially under dataset shift. Methods: We developed a calibrated deterministic five-member deep-ensemble framework for annotation-assisted thyroid nodule region-of-interest classification and selective image-based triage. TN5000 was used for model development, internal five-fold cross-validation, member-wise probability calibration, and fold-specific threshold selection, while TN3K served as an independent external dataset for evaluation under dataset shift. The framework combined a ConvNeXt-Tiny backbone with squeeze- and-excitation attention, member-wise vector-scaling calibration, ensemble-mean malignancy probability, and mutual information as an operational measure of ensemble disagreement. A strict three- tier retrospective policy assigned each image to an image-based No-FNA suggestion, an image-based FNA-recommendation pathway, or radiologist review. Results: In pooled out-of-fold TN5000 evaluation, the calibrated ensemble achieved an AUC-ROC of 0.9395, average precision (AP) of 0.9715, expected calibration error of 0.0088, and Brier score of 0.0813. At 50% nominal MI retention, 7.2% of cases received a No-FNA suggestion, 39.9% received an FNA recommendation, and 52.9% were assigned to radiologist review, with 98.3% No-FNA NPV and 99.83% malignancy capture. On external TN3K evaluation, discrimination decreased but remained moderate, with AUC-ROC of 0.7870 and AP of 0.7254. Calibration deteriorated substantially, with ECE of 0.1899 and Brier score of 0.2281. Applying the frozen TN5000-derived policy to TN3K resulted in 83.7% radiologist review, 1.0% No-FNA suggestion, and 15.3% FNA recommendation. No malignant TN3K image was assigned to the No-FNA pathway, but the FNA-recommendation PPV decreased to 76.6%. Conclusion: The proposed calibrated deep-ensemble framework demonstrated strong internal discrimination and calibration on TN5000 and supported conservative selective image-based triage using ensemble disagreement. However, external evaluation on TN3K showed reduced discrimination, degraded calibration, and limited transportability of the TN5000-derived operating thresholds under dataset shift. These findings support selective prediction as a cautious framework for identifying images unsuitable for automated triage suggestions, but do not support direct clinical deployment or claims of actual biopsy reduction without local calibration, threshold validation, and prospective clinical evaluation. Keywords: Thyroid nodule; ultrasound; deep learning; selective prediction; uncertainty estimation; calibration; dataset shift; fine-needle aspiration; ConvNeXt; deep ensemble. 1. Introduction Thyroid nodules are frequently detected during routine clinical evaluation, particularly with widespread use of high-resolution ultrasonography. Although most nodules are benign, reliable risk stratification is essential to identify lesions that may warrant fine-needle aspiration (FNA) biopsy or further diagnostic assessment [1]. Current management is guided primarily by ultrasound-based frameworks, including the American Thyroid Association (ATA) guidelines and the American College of Radiology Thyroid Imaging Reporting and Data System (ACR TI-RADS) [1], [2]. These systems integrate sonographic characteristics, nodule size, and clinical context to support surveillance and biopsy decisions. However, interpretation remains dependent on reader expertise, and substantial overlap between benign and malignant imaging appearances can complicate risk assessment. Deep learning has shown considerable promise for thyroid nodule classification from ultrasound images. Early convolutional neural network studies demonstrated that image-based models could differentiate benign from malignant nodules with performance comparable to experienced clinicians in selected cohorts [3], [4]. Subsequent research has explored transfer learning, multiview imaging, multimodal ultrasound fusion, multicentre decision-support systems, and comparative evaluation of modern neural architectures [5]–[12]. Together, these studies indicate that deep learning can extract clinically relevant imaging representations and potentially support radiologists during thyroid nodule assessment. Nevertheless, high discrimination alone is insufficient for safe clinical decision support. A model may produce a confident prediction for an ambiguous image, for a case insufficiently represented in the development data, or for an image acquired under a different protocol or device. In such settings, forcing an autonomous decision may be inappropriate. A clinically cautious system should therefore identify cases for which an automated recommendation is unreliable and defer them to radiologist review. Probability calibration and uncertainty estimation are central to this objective. Calibration assesses whether predicted probabilities correspond to observed outcome frequencies and is particularly important when probabilities are used to define conservative rule-out or rule-in thresholds [13]. Uncertainty estimation provides an additional signal for identifying predictions that should not be treated as stand-alone recommendations. In medical imaging, uncertainty-aware approaches commonly distinguish uncertainty related to image ambiguity from uncertainty arising from limited model knowledge or incomplete training-data coverage [14], [15]. Deep ensembles and Monte Carlo Dropout are widely used practical approaches for estimating predictive variability across multiple model outputs [16], [17]. However, both calibration and uncertainty behavior may deteriorate under dataset shift, even when discrimination remains acceptable [18]. Selective prediction, also known as classification with a reject option, offers a practical framework for incorporating these considerations into decision support by trading coverage for lower retained-case risk. Rather than requiring a model to classify every image autonomously, selective systems retain lower-risk predictions and defer uncertain cases to a human expert or an alternative diagnostic pathway [19], [20], [21]. This paradigm is especially relevant to thyroid ultrasound, where automated predictions should support, rather than replace, radiologist assessment. The interpretation of triage performance also depends on the prevalence and case mix of the evaluation cohort. TN5000 is a malignant-enriched image dataset, whereas TN3K has a substantially different class distribution and annotation structure [22], [23]. Consequently, predictive values, autonomous recommendation rates, and radiologist-review burden cannot be assumed to transfer directly to routine screening or to a different clinical site. These considerations motivate the use of calibrated probabilities, explicit uncertainty-based deferral, and external evaluation of a frozen operating policy. However, few thyroid ultrasound studies have jointly evaluated probability calibration, ensemble- disagreement-based deferral, conservative selective triage, and external threshold transportability under dataset shift. In this study, we present a calibrated deterministic deep-ensemble framework for annotation-assisted thyroid nodule region-of-interest classification and selective image-based triage. The framework combines a ConvNeXt-Tiny backbone with squeeze-and-excitation attention, five independently trained ensemble members, member-wise vector-scaling calibration, and mutual information as an ensemble-disagreement score for deferral. Each image is assigned to one of three retrospective pathways: an image-based No-FNA suggestion, an image-based FNA recommendation, or radiologist review. Automated suggestions are issued only for low-disagreement cases with calibrated probabilities in conservative rule-out or rule-in regions, whereas cases with elevated ensemble disagreement or intermediate probabilities are deferred to radiologist review. The contributions of this study are as follows: 1. We develop a calibrated deterministic deep-ensemble framework for annotation-assisted thyroid nodule ROI malignancy classification using ConvNeXt-Tiny with squeeze-and- excitation attention. 2. We evaluate mutual information as an ensemble-disagreement score for selective prediction and use it to identify images unsuitable for automated image-based triage suggestions. 3. We propose a conservative three-tier retrospective triage policy that combines uncertainty- based deferral with probability-based No-FNA suggestion and FNA-recommendation pathways. 4. We evaluate a frozen TN5000-derived calibration and triage policy on the independent TN3K cohort to assess discrimination, calibration, uncertainty behavior, and threshold transportability under dataset shift. The objective of this study is not to replace clinical assessment or estimate real-world biopsy avoidance. Rather, this study evaluates whether calibrated ensemble disagreement can support a conservative selective-prediction framework in which automated image-based suggestions are issued only for low- disagreement cases, while potentially unreliable predictions are withheld and referred for expert review. 2. Related Work Prior thyroid ultrasound studies have shown that deep learning can support benign-versus-malignant nodule classification and, in some settings, assist clinical management. However, clinical translation requires more than discrimination alone. Reliable decision support also requires calibrated probabilities, uncertainty-aware referral, and evidence that operating thresholds remain meaningful under dataset shift. Table 1 summarizes representative studies and positions the present framework. Table 1. Representative studies in thyroid ultrasound deep learning and the positioning of the present study. Study Primary focus Reliability or clinical- translation feature Relevance to the present study Guan et al. [3] Ultrasound-based benign-versus- malignant classification Large-scale CNN classification Established early evidence for deep-learning-based thyroid nodule classification Peng et al. [4] Multicentre diagnosis and management support Clinical decision-support evaluation Demonstrated potential for AI-assisted thyroid nodule management Weng et al. [10] Independent-dataset validation Evaluation across different ultrasound equipment Highlights the importance of external generalization Ni et al. [11] AI-assisted reduction of unnecessary FNA Management-oriented retrospective evaluation Illustrates the potential clinical value of AI-supported triage Rashed et al. [12] Comparative ultrasound classification Deterministic deep-learning benchmarking Provides recent evidence for conventional image-based classification Saini et al. [24] Multimodal ultrasound classification Uncertainty-aware multimodal prediction Demonstrates the value of reliability-aware prediction Xiang et al. [25] Segmentation and TI-RADS classification Uncertainty-weighted multitask learning Uses uncertainty during training rather than for case- level referral Present study Annotation-assisted thyroid ROI classification Probability calibration, deterministic ensemble disagreement, inference- time selective referral, and external threshold- transportability analysis Evaluates a conservative three-tier image-based triage policy 2.1 Deep Learning for Thyroid Ultrasound Classification and Management Deep-learning models can learn discriminative imaging patterns directly from thyroid ultrasound data and have demonstrated promising performance for benign-versus-malignant classification. Early work by Guan et al. [3] established the feasibility of convolutional neural networks for this task. Subsequent studies extended this direction through transfer learning, multiview imaging, multimodal ultrasound fusion, and modern neural architectures [5]–[8], [12], [26]. Research has also moved toward clinical decision support. Peng et al. [4] developed a multicentre AI system for thyroid nodule diagnosis and management, and Ni et al. [11] evaluated an AI-assisted strategy to reduce unnecessary FNA procedures. These studies show the clinical potential of image- based AI. However, biopsy and surveillance decisions also depend on ultrasound features, nodule size, clinical history, and radiologist judgement. Image-based triage systems should therefore separate automated suggestions from cases requiring expert review, especially when biopsy-related recommendations are inferred from ROI-level predictions rather than full clinical assessment. 2.2 Calibration, Uncertainty, and Selective Referral High discrimination does not guarantee reliable probability estimates. A model may achieve a strong AUC while producing probabilities that do not reflect observed malignancy frequencies. Calibration is therefore essential when probability thresholds are used for rule-out or rule-in decisions [13]. In medical imaging, uncertainty estimation provides an additional safeguard by identifying cases for which an automated prediction may be unreliable [14], [15]. Deep ensembles estimate prediction reliability through disagreement among independently trained models [16], whereas Monte Carlo Dropout estimates uncertainty using repeated stochastic inference [17]. However, uncertainty and calibration may deteriorate under dataset shift, even when discrimination remains useful [18]. Selective prediction addresses this issue by allowing a model to defer uncertain cases for expert review instead of forcing an automated decision for every image [19], [20]. Thyroid-specific uncertainty-aware studies remain limited. Saini et al. [24] incorporated uncertainty into multimodal ultrasound prediction, while Xiang et al. [25] used uncertainty weighting for multitask learning. These approaches are valuable but differ from an inference-time selective-referral policy, where uncertainty is used after model training to determine whether a specific case should receive an autonomous image-based suggestion or be referred for radiologist review. 2.3 Position of the Present Study This study evaluates a calibrated deterministic five-member ensemble for annotation-assisted thyroid nodule ROI classification. The framework combines member-wise vector-scaling calibration with mutual-information-based ensemble disagreement to identify cases that may be unsuitable for automated image-based triage. The proposed policy is conservative. Images with high mutual information are deferred to radiologist review. Among low-disagreement cases, only calibrated probabilities in predefined low- or high-risk regions receive a retrospective image-based No-FNA suggestion or FNA recommendation, while intermediate cases are also deferred. This separates classification accuracy from decision reliability and avoids forcing recommendations for all images. The frozen TN5000-derived calibration and triage policy is then tested on the independent TN3K dataset to assess discrimination, calibration, uncertainty behavior, and threshold transportability under dataset shift. This addresses whether calibrated ensemble disagreement can support conservative selective referral outside the model’s development distribution. 3. Methodology 3.1 Datasets and Study Design This retrospective study used two publicly available thyroid ultrasound datasets: TN5000 for model development and internal evaluation, and TN3K for external domain-shift evaluation [22], [23]. TN5000 was used for preliminary backbone screening, five-fold internal cross-validation, and the separate post-cross-validation deployment-training procedure described in Sections 3.3 and 3.10. TN5000 comprised 5,000 B-mode thyroid ultrasound images with binary benign or malignant labels and nodule-localization bounding boxes [22]. In the released XML annotations used in this study, 1428 images were benign (28.6%) and 3572 were malignant (71.4%); all analyses used these parsed annotation counts. The TN5000 data descriptor states that one representative image from the same patient perspective was retained during dataset curation. However, the released materials did not provide patient identifiers or an image-to-patient mapping. Internal evaluation therefore used image-level stratified five-fold cross- validation. Image identifiers were assigned to mutually exclusive training, early stopping validation (val_es), calibration (val_calib), and test subsets within each fold, and exact image overlap was excluded; however, patient-level independence could not be verified. TN3K was evaluated using the official test split released with the dataset descriptor, comprising 614 images, including 378 benign images (61.6%) and 236 malignant images (38.4%) [23]. Although TN3K had a lower malignant proportion than TN5000, its class distribution should not be interpreted as representative of a population-screening cohort. The datasets also differed in lesion-localization annotations and ROI-construction procedures. TN5000 supplied bounding-box annotations, from which lesion-centred ROI crops were extracted, whereas TN3K supplied pixel-wise segmentation masks used to derive lesion-centred ROI crops. Because both datasets provided lesion-localization annotations, this study evaluates annotation-assisted ROI classification rather than end-to-end nodule detection. Dataset-specific ROI-construction and input- standardization procedures are detailed in Supplementary Methods S1. Differences in data source, class distribution, annotation granularity, and ROI construction were considered components of the external dataset shift when interpreting TN3K performance. Table 2 shows the summary of the datasets used in this study. Table 2. Summary of datasets used in this study Dataset Role Annotation Type Benign (n, %) Malignant (n, %) Total Images (n) TN5000 Development and internal evaluation Bounding boxes and labels 1,428 (28.6%) 3,572 (71.4%) 5,000 TN3K External Testing Segmentation masks and labels 378 (61.56%) 236 (38.44%) 614 3.2 Overall Framework The proposed framework was designed as a calibrated selective-prediction system for annotation- assisted thyroid nodule ROI classification (Figure 1). The development workflow comprised lesion- centred ROI preparation, preliminary backbone screening, fold-specific deterministic ensemble training, member-wise probability calibration, mutual-information-guided deferral, and strict probability-based triage. Qualitative Grad-CAM visualization and external evaluation on TN3K were conducted after the primary internal analysis. First, lesion-centred ROI crops were extracted from the TN5000 bounding-box annotations. These ROI patches were used in a preliminary architecture-screening experiment in which five ImageNet- pretrained convolutional neural networks were evaluated under a common development protocol. ConvNeXt-Tiny was selected using validation AUC-ROC as the primary screening criterion. The selected architecture was then evaluated using image-level stratified five-fold cross-validation. Within each fold, mutually exclusive training, early-stopping and threshold-selection (val_es), calibration (val_calib), and held-out test subsets were defined. Five ConvNeXt-Tiny models with different random initializations were trained on each fold-specific training subset. Member-wise vector scaling was fitted using val_calib only. Calibrated member probabilities were then aggregated to obtain the ensemble-mean malignancy probability and an ensemble-disagreement score based on mutual information (MI). The triage policy was applied in two stages. First, images with MI above a fold-specific threshold were assigned to radiologist review. Second, among images that passed the MI gate, low calibrated malignancy probabilities received a No-FNA suggestion and high probabilities received an FNA recommendation. Images with intermediate probabilities were also assigned to radiologist review. Consequently, autonomous recommendations were issued only for low-MI cases in conservative rule- out or rule-in regions. Predictive variance and MSP-derived uncertainty were evaluated as secondary uncertainty-score comparators using risk-coverage analysis. MI remained the primary deferral score because the central study question was whether disagreement among independently trained models could identify cases for which automated image-based triage suggestions should be withheld. After internal cross-validation, a separate TN5000 deployment model was trained and frozen for TN3K evaluation. Because TN3K differed from TN5000 in source, class distribution, annotation format, and ROI extraction, this evaluation was interpreted as a domain-shift analysis of threshold transportability rather than evidence of deployment-ready external generalization. Figure 1. Overview of the annotation-assisted selective-prediction workflow. 3.3 Backbone Selection and Final Model Architecture A preliminary development screen compared ConvNeXt-Tiny, EfficientNetV2-S, DenseNet121, ResNet18, and MobileNetV4 using the same development protocol. Validation AUC-ROC was the primary criterion for backbone selection because it summarizes discrimination across classification thresholds. Average precision (AP) was recorded as a secondary discrimination measure. ConvNeXt-Tiny [27] was selected as the backbone for the subsequent calibrated ensemble experiments. The preliminary screening was intended to select a practical primary backbone for this study and was not designed to establish universal superiority across thyroid ultrasound datasets. Checkpoints and fitted parameters from the screening experiment were not transferred to the subsequent cross-validation experiments. A squeeze-and-excitation (SE) channel-attention module [28] was incorporated into the selected classification architecture to recalibrate the pooled channel representation before classification. Let 푋∈ℝ 퐻 × 푊 × 퐶 denote the input feature map. A channel descriptor was obtained by global average pooling: 푧 = 퐺퐴푃 (푋) (1) Channel-wise attention weights were then computed as: 푆 = 휎 (푊 2 훿 (푊 1 푧)) (2) where 푊 1 and 푊 2 are learnable weight matrices, δ denotes the ReLU activation function, and σ denotes the sigmoid function. The recalibrated feature map was defined as: 푧 = 푠 ⊙ 푧 (3) where ⊙ denotes element-wise multiplication. The recalibrated representation was then passed to the classification head. No additional spatial-attention was introduced. 3.4 Cross-Validation, Preprocessing, and Ensemble Training The selected ConvNeXt-Tiny architecture was evaluated using image-level stratified five-fold cross- validation. In each fold, 20% of images were reserved as a held-out test subset (n = 1,000). The remaining 80% (n = 4,000) was partitioned into three disjoint subsets: a) Training set (85% of the non-test data; n≈3,400), used to optimize model parameters. b) Early-stopping validation set (val_es; 10% of the non-test data; n≈400), used for early stopping, checkpoint selection, and fold-specific MI and probability-threshold derivation. c) Calibration set (val_calib; 5% of the non-test data; n≈200), reserved exclusively for per-member post-hoc probability calibration using vector scaling. Held-out test images were not used for model fitting, checkpoint selection, calibration, or threshold selection. They were used only for final fold-wise evaluation. Pooled out-of-fold predictions were subsequently used to compare MI, predictive variance, and MSP-derived uncertainty using risk- coverage analysis. ROI images were converted to grayscale, autocontrast-adjusted, resized to (224 x 224) pixels, replicated across three channels, and normalized with ImageNet channel statistics. Training images received stochastic augmentation, including random horizontal flipping, rotation within (±20°), affine transformation, brightness and contrast jittering, Gaussian blurring, and random sharpness adjustment. Validation, calibration, and held-out test images received deterministic preprocessing only. Class imbalance was addressed in training through an inverse-frequency weighted random sampler with replacement. Label-smoothed cross-entropy loss was used with a smoothing factor of (0.05). The validation, calibration, and test subsets retained their observed class distributions. Within each fold, five deterministic ensemble members were trained using the same fold-specific training subset and optimization protocol but different random seeds: 42, 123, 7, 2024, and 999. All members were initialized with ImageNet-pretrained ConvNeXt-Tiny weights and optimized with AdamW. The backbone was held frozen for the first 15 training epochs and was unfrozen before epoch 16 for fine-tuning. Training used a mini-batch size of 32 with gradient accumulation over two steps, giving an effective batch size of 64. The initial learning rate was (5 × 10⁻⁴), and the fine-tuning backbone learning rate was (1 × 10⁻⁵). Gradient clipping, learning-rate scheduling, and early stopping were used to stabilize optimization. The checkpoint for each member was selected according to the highest validation average precision (AP) on val_es. Dropout layers were used only as training regularization in the classification head and were disabled during inference. Therefore, each image produced one deterministic prediction from each of the five members. These member-level outputs were calibrated and used to calculate the ensemble-mean malignancy probability and MI. Table 3 shows the training and deterministic ensemble configuration used in this study. Table 3. Training and deterministic ensemble configuration. Parameter Configuration Backbone ImageNet-pretrained ConvNeXt-Tiny with squeeze- and-excitation module Mini-batch size 32 Input image size (224 x 224) pixels Ensemble size 5 deterministic members Random seeds 42, 123, 7, 2024, 999 Dropout during training 0.35 before hidden layer; 0.50 before output layer Optimizer AdamW Initial learning rate 5 × 10⁻⁴ Fine-tuning backbone learning rate 1 × 10⁻⁵ Effective batch size 64 Maximum epochs 50 3.5 Probability Calibration and Ensemble Prediction Aggregation Because the strict triage policy depends on probability-based rule-out and rule-in thresholds, post-hoc calibration was applied before uncertainty estimation, threshold selection, and held-out evaluation. Within each cross-validation fold, each ensemble member was calibrated independently using vector scaling fitted only on the fold-specific val_calib subset. Let 푧 푚 ∈푅 2 denote the uncalibrated two-class logit vector produced by ensemble member (m). The member-specific vector-scaling transformation was: 푧 ₘ = 푤ₘ ⊙ 푧ₘ + 푏ₘ (4) where w m and 푏 푚 are learned two-dimensional scaling and bias vectors, respectively, and (⊙) denotes element-wise multiplication. The calibrated malignancy probability for member (m) was then obtained as: 푝ₘ = [푠표푓푡푚푎푥(푧 ₘ)]ₘₐₗᵢ₉ₙₐₙₜ (5) Calibration parameters were fitted by minimizing cross-entropy on val_calib only and were fixed thereafter. Neither val_es nor the held-out test subset was used to estimate or modify calibration parameters. Calibration was assessed fold-wise using ECE with 10 equal-width probability bins and the Brier score. The calibrated ensemble malignancy probability was calculated as the arithmetic mean of the five calibrated member probabilities: 푝 =( 1 푀 )∑푝 푚 푀 푚=1 (6) where (M=5). The ensemble-mean probability (푝 ) was used for rule-out and rule-in decisions, whereas variation among calibrated member probabilities was used for uncertainty estimation. 3.6 Mutual Information as the Primary Deferral Score The primary uncertainty score was ensemble predictive mutual information (MI). MI was selected because the central study question was whether disagreement among independently trained deep- learning models could identify thyroid nodule images for which autonomous image-based triage should be withheld. MI was calculated from calibrated member probabilities; therefore, it reflects disagreement after fold-specific probability calibration. For an image, let (p_m) denote the calibrated malignancy probability from ensemble member (m), where (푚 = 1,...,푀) and (M=5). The ensemble-mean probability was (푝 ), as defined in Section 3.5. For binary classification, predictive entropy was calculated using natural logarithms: 퐻(푞) = −푞 푙표푔(푞) − (1−푞) 푙표푔(1−푞) (7) Mutual information was calculated as: 푀퐼=퐻 ( 푝̅ ) − 1 푀 ∑퐻 푀 푚=1 ( 푝 푚 ) (8) The first term represents uncertainty in the ensemble-mean prediction, whereas the second represents the mean uncertainty of individual ensemble members. MI increases when members produce divergent predictions for the same image. It was therefore interpreted as an operational ensemble-disagreement score and a proxy for epistemic uncertainty, rather than as a complete empirical separation of epistemic and aleatoric uncertainty. Images with high MI were considered unsuitable for autonomous image-based triage and were assigned to radiologist review. Predictive variance and MSP-derived uncertainty were evaluated as secondary uncertainty-score comparators. 3.7 Selective Prediction Evaluation Selective prediction was used to evaluate whether an uncertainty score ranked images according to expected prediction reliability. A selective classifier consists of a base classifier, an uncertainty or confidence score, and a deferral rule, following the standard selective-classification formulation [20]. Risk-coverage and generalized-risk coverage summaries were used to evaluate uncertainty-score ranking quality following recent recommendations for selective-classification evaluation [21]. In this study, deferred images were assigned to radiologist review rather than removed from care. For each uncertainty score, pooled out-of-fold (OOF) images were ranked from lowest to highest uncertainty. Each image therefore had predictions and uncertainty estimates generated by a model that had not been trained on that image. At coverage c, the accepted set 퐴 푐 consisted of the lowest- uncertainty images retained for autonomous classification. Selective risk was defined as the misclassification rate among accepted images: 푅 ( 푐 ) = 1 | 퐴 푐 | ∑ퟏ 푖∈퐴 푐 ( 푦ˆ 푖 ≠푦 푖 ) (9) Generalized risk was defined as the proportion of all images that were both accepted and misclassified: 퐺 ( 푐 ) = 1 푁 ∑ퟏ 푖∈퐴 푐 ( 푦ˆ 푖 ≠푦 푖 ) =푐푅 ( 푐 ) (10) Area under the risk-coverage curve (AURC) and area under the generalized-risk coverage curve (AUGRC) summarize selective and generalized risk across coverage levels: 퐴푈푅퐶=∫푅 1 0 ( 푐 ) 푑푐, 퐴푈퐺푅퐶=∫퐺 1 0 ( 푐 ) 푑푐 (11) Risk-coverage curves were evaluated at every possible retained-set size from k=1 to k=N pooled OOF images. At each retained-set size, coverage was defined as c = k/N, and cases were retained in ascending order of uncertainty. AURC and AUGRC were calculated from the resulting full-resolution discrete selective-risk and generalized-risk curves. AUGRC was treated as the primary holistic ranking summary, whereas AURC was reported as a conventional complementary metric. These metrics were used for within-study comparison of MI, predictive variance, MSP-derived uncertainty, and an expected random-order reference. Lower AURC and AUGRC values indicate better ranking of potential silent failures, such that images more likely to be misclassified are preferentially deferred. This threshold-free ranking analysis did not select clinical or triage thresholds; fold-specific MI and probability thresholds for strict triage were derived separately from each fold’s val_es subset, as described in Sections 3.8 and 3.9. Standard discrimination performance was assessed on held-out fold predictions using AUC-ROC and average precision (AP). Calibration was assessed using ECE with 10 equal-width probability bins and the Brier score. Fold-wise results are reported as mean ± standard deviation across held-out folds, while pooled out-of-fold analyses provide one prediction per TN5000 image. 3.8 Strict MI-Guided Triage Policy The clinical triage policy comprised two sequential decision stages. For fold (f), let (uᵢ) denote the MI score for image (i), let (τₘᵢ⁽ᶠ⁾) denote the fold-specific MI threshold, and let (τ_low⁽ᶠ⁾) and (τ_high⁽ᶠ⁾) denote the corresponding probability thresholds. First, images with (uᵢ > τₘᵢ⁽ᶠ⁾) were assigned to radiologist review. Among images that passed the MI gate, the calibrated ensemble probability (p ̄ ᵢ) was used for rule-out and rule-in assignment. Thus, images with intermediate calibrated probabilities were intentionally deferred rather than forced into autonomous benign or malignant recommendations. The complete fold-specific triage policy is summarized in Table 4. Table 4. Strict MI-guided triage decision policy. Decision stage Condition Triage assignment Uncertainty gate (uᵢ > τₘᵢ⁽ᶠ⁾) Radiologist review Rule-out gate (uᵢ ≤ τₘᵢ⁽ᶠ⁾) and (p ̄ ᵢ < τ_low⁽ᶠ⁾) No-FNA suggestion Rule-in gate (uᵢ ≤ τₘᵢ⁽ᶠ⁾) and (p ̄ ᵢ ≥ τ_high⁽ᶠ⁾) FNA recommendation Probability gray zone (uᵢ ≤ τₘᵢ⁽ᶠ⁾) and (τ_low⁽ᶠ⁾ ≤ p ̄ ᵢ < τ_high⁽ᶠ⁾) Radiologist review 3.9 Fold-Specific Threshold Selection All triage thresholds were derived independently within each cross-validation fold using val_es only. The held-out test subset was not used to select or modify any threshold. The MI threshold τₘᵢ⁽ᶠ⁾ was evaluated over a nominal MI-retention grid of 50%, 70%, 80%, 90% and 95%. For a given target, the MI threshold was selected so that the specified proportion of val_es images with the lowest MI values passed the uncertainty gate. Nominal MI retention therefore describes the expected proportion passing the MI gate before probability-gray-zone deferral. Final autonomous coverage can be lower because intermediate-probability images are assigned to radiologist review. For each nominal MI-retention target, (τ_low⁽ᶠ⁾) and (τ_high⁽ᶠ⁾) were derived from the low-MI val_es subset. Rule-out threshold selection was restricted to images with (p ̄ ≤ 0.50). The selected (τ_low⁽ᶠ⁾) was the largest probability threshold for which the strict rule (p ̄ < τ_low⁽ᶠ⁾) achieved a No-FNA NPV of at least 98% in val_es. If no candidate rule-out threshold met the NPV target, no autonomous No-FNA pathway was issued for that operating point. Rule-in threshold selection was restricted to images with (p ̄ ≥ 0.50). The selected (τ_high⁽ᶠ⁾) was the smallest probability threshold for which the rule (p ̄ ≥ τ_high⁽ᶠ⁾) achieved an FNA-recommendation PPV of at least 95% in val_es. If no candidate threshold met the PPV target, no autonomous FNA- recommendation pathway was issued for that operating point. The 98% NPV and 95% PPV values were conservative operational targets adopted for this retrospective image-based analysis; they were not universal clinical standards. Malignancy capture of at least 95% was treated as a prespecified safety-oriented validation criterion and was evaluated after the complete fold-specific policy had been derived. It was not used to alter the nominal MI-retention grid. Because predictive values vary with prevalence and case mix, thresholds derived from TN5000 should not be transferred to other populations without local calibration and threshold validation. 3.10 Post-Cross-Validation Deployment Training and External Domain-Shift Evaluation After completion of the five-fold internal evaluation, a separate deployment-training protocol was used to prepare a fixed model for external testing. This protocol was distinct from the five cross-validation models and was not used to estimate internal performance. All 5,000 TN5000 images were repartitioned once using image-level stratified sampling with the same random seed as the development pipeline. The resulting subsets comprised 4,250 (85%) images for model training, 499 (10%) images for early stopping and threshold selection (val_es), and 251 (5%) for calibration (val_calib). Five ConvNeXt-Tiny models with SE modules were trained independently on the deployment training subset using the same preprocessing, augmentation, optimization, random seeds, and checkpoint- selection procedure as in the cross-validation analysis. Member-specific vector-scaling parameters were fitted exclusively on the deployment val_calib subset. The calibrated deployment ensemble was then applied to val_es to derive the prespecified MI threshold grid and the associated rule-out and rule-in probability thresholds. The 50% nominal MI-retention policy was designated before external inference. After model weights, member-specific calibration parameters, and the threshold grid had been fixed, they were applied unchanged to TN3K. TN3K labels were not used for model fitting, checkpoint selection, calibration, uncertainty-score selection, or threshold derivation. The TN3K analysis therefore evaluates transportability of a fixed TN5000-derived operating policy under dataset shift. It does not provide a second internal performance estimate and should not be interpreted as deployment-ready validation. 3.11 Qualitative Grad-CAM Analysis To enhance interpretability, Grad-CAM [29] was employed to visualize the regions contributing most to the model decision. The Grad-CAM formulation is given by: 훼 푘 푐 = 1 푍 ∑ 휕푦 푐 휕퐴 푖푗 푘 푗푖 (12) 퐿 Grad−CAM 푐 =ReLU(∑훼 푘 푐 푘 퐴 푘 ) (13) where 퐴 푘 represents the 푘-th feature map, and 훼 푘 푐 represents the importance weight corresponding to class 푐. Grad-CAM maps were generated from a single deterministic ensemble member with seed 42 and dropout disabled. These maps provide qualitative examples of regions contributing to that member’s class score. They are not ensemble-level explanations, were not used for model selection or triage, and should not be interpreted as validation of specific sonographic features or causal model reasoning. 3.12 Implementation Specifications (Hardware and software environment) The entire experimental pipeline was implemented and executed on the Kaggle cloud computing platform in a GPU-accelerated environment. The hardware configuration consisted of an NVIDIA Tesla T4 GPU with 16 GB VRAM, a quad-core Intel Xeon CPU, and 30 GB of system RAM. All experiments were conducted in Python (v3.12.12) using PyTorch (v2.5) as the primary deep learning framework, with torchvision (v0.20) for ImageNet-pretrained backbone architectures. Model training and inference were accelerated via CUDA. Classification and evaluation pipelines were implemented using scikit- learn (v1.4), while data handling and numerical computations utilized Pandas (v2.2) and NumPy (v1.26) respectively. Image preprocessing was performed using Pillow (PIL). All computations were executed in a Linux-based environment with full CUDA acceleration enabled. To demonstrate how the framework outputs could be presented in an interactive research setting, a web- based prototype was developed to display ensemble probabilities, member-level predictions, disagreement measures, qualitative Grad-CAM visualizations, and retrospective triage assignments (Supplementary Figure S4). The prototype was developed solely to demonstrate presentation of the framework outputs; it was not evaluated in a clinical workflow and is not intended for clinical diagnosis or deployment. 4. Results 4.1 Preliminary Backbone Architecture Selection Table 5 summarizes the preliminary backbone-screening results on the development validation subset. ConvNeXt-Tiny achieved the highest validation AUC-ROC (0.9209), followed by EfficientNetV2-S (0.9135) and DenseNet121 (0.9133). Validation AUC-ROC was used as the primary architecture- selection criterion; therefore, ConvNeXt-Tiny was selected for the subsequent cross-validation and selective-prediction analyses. Average precision (AP) was recorded as a secondary threshold-independent discrimination measure. ConvNeXt-Tiny also achieved the highest AP (0.9650). Accuracy, sensitivity, specificity, MCC, ECE, and Brier score are reported descriptively in Table 5 and were not used for architecture selection. These screening results provide preliminary model-selection evidence rather than an independently validated comparison of all candidate backbones. Table 5. Preliminary backbone-screening performance on the development validation subset. Backbone AUC- ROC AP Accurac y (%) Sensitivit y (%) Specificity (%) MCC ECE Brier score ConvNeXt- Tiny 0.9209 0.9650 84.6 84.1 85.9 0.6584 0.0819 0.1104 EfficientNet V2-S 0.9135 0.9597 86.2 89.4 78.2 0.6660 0.0556 0.1028 DenseNet12 1 0.9133 0.9605 83.6 83.2 84.5 0.6365 0.0618 0.1063 ResNet18 0.9067 0.9559 85.8 89.1 77.5 0.6563 0.0919 0.1181 MobileNetV 4 0.8416 0.9201 78.2 79.9 73.9 0.5076 0.1600 0.1607 4.2 Fold-Wise Classification Performance Table 6 summarizes classification and calibration performance across the five held-out cross-validation test folds. The calibrated five-member deterministic ensemble achieved a mean AUC-ROC of 0.9395 ± 0.0103 and a mean AP of 0.9720 ± 0.0052. Fold-specific AUC-ROC values ranged from 0.9270 to 0.9499, indicating consistent discrimination across data partitions. Threshold-dependent metrics were calculated using fold-specific Youden thresholds derived from the corresponding val_es subsets. Mean accuracy was 86.10% ± 0.78%, with mean sensitivity of 85.65% ± 1.48% and specificity of 87.24% ± 4.07%. The mean Matthews correlation coefficient was 0.6899 ± 0.0223. Table 6. Fold-wise classification and calibration performance across five held-out cross-validation folds. Values are reported as mean ± standard deviation. Metric Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Mean ± SD AUC-ROC 0.9276 0.9498 0.9433 0.9270 0.9499 0.9395 ± 0.0103 AP 0.9684 0.9758 0.9730 0.9640 0.9786 0.9720 ± 0.0052 Accuracy (%) 86.00 87.00 85.30 85.20 87.00 86.10 ± 0.78 Sensitivity (%) 88.38 85.45 83.92 85.03 85.45 85.65 ± 1.48 Specificity (%) 80.07 90.88 88.77 85.61 90.88 87.24 ± 4.07 MCC 0.6676 0.7166 0.6804 0.6682 0.7166 0.6899 ± 0.0223 ECE 0.0275 0.0241 0.0216 0.0334 0.0152 0.0244 ± 0.0060 Brier score 0.0946 0.0706 0.0758 0.0875 0.0778 0.0813 ± 0.0086 4.3 Pooled Out-of-Fold Evaluation 4.3.1 Pooled Discrimination and Calibration Fold-specific held-out predictions were concatenated to form a pooled out-of-fold (OOF) evaluation set containing one prediction for each of the 5,000 TN5000 images. Each prediction was generated by an ensemble that had not been trained on that image in its corresponding cross-validation fold. In the pooled OOF analysis, the calibrated five-member ensemble achieved an AUC-ROC of 0.9395 (95% CI, 0.9312–0.9459) and an average precision (AP) of 0.9715 (95% CI, 0.9669–0.9760). Pooled OOF reliability analysis was performed to evaluate the effect of member-wise vector scaling on ensemble calibration. For each OOF image, the uncalibrated and calibrated predictions were generated by the same fold-specific ensemble, and the vector-scaling parameters had been fitted exclusively on that fold’s val_calib subset. Before calibration, the pooled OOF predictions yielded an ECE of 0.0292 and a Brier score of 0.0836. After vector scaling, the pooled ECE decreased to 0.0088 and the Brier score decreased to 0.0813, indicating improved aggregate calibration within the same held-out OOF evaluation framework. Figure 2 presents the pooled OOF ROC and precision-recall curves, whereas Figure 3 shows the corresponding reliability diagram. No operating threshold was optimized using the pooled OOF labels. Because selecting a pooled probability threshold would require observing all held-out labels, threshold-dependent classification metrics were instead calculated using fold-specific thresholds derived exclusively from the corresponding val_es subsets and are summarized in Table 6. Figure 2. Pooled out-of-fold ROC and precision-recall curves for the calibrated five-member ConvNeXt-Tiny ensemble on TN5000. Figure 3. Pooled out-of-fold reliability diagram of calibrated ensemble predictions on TN5000. 4.3.2 Selective-Prediction Performance Selective-prediction performance was evaluated using pooled OOF predictions (Figure 4), such that each TN5000 image was evaluated by a model that had not been trained on that image. Risk-coverage curves were calculated at every possible retained-set size across all 5,000 OOF predictions. For MI, the full-resolution pooled OOF AURC was 0.037778 and the AUGRC was 0.02722, compared with a random-order expectation of 0.109800 and 0.05491, respectively. Thus, ranking cases from low to high MI reduced the error rate among retained images relative to random selection. Supplementary analyses compared MI with predictive variance and MSP-derived uncertainty as secondary uncertainty-score comparators. MSP-derived uncertainty achieved the lowest AURC and AUGRC (0.03020 and 0.02261), followed by predictive variance (0.03234 and 0.02424) and MI (0.03778 and 0.02722). Predictive variance measures dispersion of calibrated probabilities across ensemble members, whereas MSP-derived uncertainty reflects low confidence in the ensemble-mean output. The comparator analysis is provided in Supplementary Results S2. Figure 4. Pooled out-of-fold MI risk-coverage curve. Cases were retained in ascending order of mutual information. Lower selective risk indicates fewer classification errors among retained images. The curve was calculated at all 5,000 retained-set sizes. 4.4 MI-Guided Strict Triage Outcomes Table 7 summarizes strict MI-guided triage outcomes across the five held-out test folds. Nominal MI retention denotes the proportion of val_es images passing the MI gate before probability-gray-zone deferral; it should not be interpreted as autonomous decision coverage. Increasing nominal MI retention reduced the mean radiologist-review rate from 52.9% ± 1.7% at 50% retention to 22.6% ± 5.4% at 95% retention. Over the same range, the mean FNA-recommendation rate increased from 39.9% ± 1.1% to 64.5% ± 2.4%, whereas the No-FNA suggestion rate increased from 7.2% ± 0.9% to 12.9% ± 4.3% and changed little beyond the 80% retention setting. Thus, higher MI- retention targets primarily increased FNA recommendations rather than substantially expanding the conservative No-FNA pathway. Malignancy capture remained above 99.2% across all operating points. Mean No-FNA NPV ranged from 96.3% to 98.3%, while mean FNA-recommendation PPV remained above 95% through the 80% nominal-retention setting but decreased to 94.6% ± 1.0% and 94.4% ± 1.2% at 90% and 95% retention, respectively. This reflects that the PPV and NPV criteria were validation-derived targets rather than guarantees for every held-out test fold. These findings describe retrospective image-based triage assignments in a malignant-enriched cohort. They do not estimate actual avoided biopsies or replace clinical assessment incorporating ultrasound reporting features, nodule size, clinical history, and radiologist interpretation. Table 7. Fold-wise strict MI-guided triage outcomes across five held-out test folds. Nominal MI retentio n No-FNA suggestion rate FNA- recommend ation rate Radiologist -review rate Malignancy capture No-FNA NPV FNA- recommenda tion PPV 50% 7.2% ± 0.9% 39.9% ± 1.1% 52.9% ± 1.7% 99.83% ± 0.21% 98.3% ± 2.2% 97.4% ± 0.8% 70% 11.3% ± 1.9% 53.8% ± 1.0% 34.9% ± 2.1% 99.47% ± 0.42% 96.9% ± 2.3% 96.0% ± 0.6% 80% 12.9% ± 3.0% 58.6% ± 0.9% 28.5% ± 3.4% 99.24% ± 0.61% 96.3% ± 2.9% 95.3% ± 0.8% 90% 12.7% ± 4.1% 62.7% ± 1.5% 24.6% ± 4.8% 99.36% ± 0.47% 96.8% ± 1.6% 94.6% ± 1.0% 95% 12.9% ± 4.3% 64.5% ± 2.4% 22.6% ± 5.4% 99.27% ± 0.56% 96.5% ± 1.8% 94.4% ± 1.2% 4.5 Qualitative Grad-CAM Visualization Figure 5 presents qualitative Grad-CAM visualizations for four validation images: a correctly classified benign image, a correctly classified malignant image, a misclassified image, and an image with high ensemble mutual information (MI). The first three cases were randomly selected from their respective outcome groups, whereas the high-MI case was randomly sampled from the highest-MI subset of validation images. Activation patterns varied across the correctly classified, misclassified, and high-MI examples. In the high-MI example, the ensemble prediction was malignant despite elevated disagreement across ensemble members. These visualizations are qualitative only. They do not validate specific sonographic features, establish causal model reasoning, or represent ensemble-level explanations. Grad-CAM was not used for model selection, calibration, uncertainty estimation, or triage-threshold selection. Figure 5. Qualitative Grad-CAM examples showing a true benign, true malignant, misclassified, and high-MI ROI. Rows show the original ROI and Grad-CAM overlay. 4.6 External Discrimination and Calibration Under Dataset Shift The frozen five-member deployment ensemble was applied unchanged to the independent TN3K cohort ((n=614); 236 malignant and 378 benign images). Model weights and member-specific vector-scaling parameters were derived exclusively from TN5000. TN3K labels were used only for external outcome evaluation and did not influence model fitting, checkpoint selection, calibration, uncertainty-score comparison, or threshold selection. On TN3K, the model achieved an AUC-ROC of 0.7870 (95% CI, 0.7478–0.8237) and an AP of 0.7254 (95% CI, 0.6690–0.7792) (Figure 6). Both measures were lower than the corresponding internal pooled OOF values, indicating reduced but retained discrimination under this external dataset shift. For descriptive classification at a fixed probability threshold of 0.50 (Figure 7), accuracy was 66.8%, sensitivity was 82.2%, specificity was 57.1%, positive predictive value was 54.5%, negative predictive value was 83.7%, and MCC was 0.3878. The model correctly classified 194 of 236 malignant images and 216 of 378 benign images, with 42 false-negative and 162 false-positive predictions. External calibration was poorer than in the TN5000 OOF analysis, with ECE of 0.1899 and Brier score of 0.2281. Thus, TN3K demonstrates retained discrimination but substantial degradation in probability calibration under the observed dataset shift. This analysis should be interpreted as external domain-shift characterization rather than evidence of deployment-ready generalization. Figure 6. AUC-ROC and Precision-Recall (PR) curves in external TN3K evaluation Figure 7. External discrimination and fixed-threshold classification performance on TN3K. 4.7 External MI-Guided Triage Under Dataset Shift The fixed 50% nominal MI-retention policy, including member-specific vector scaling and triage thresholds derived exclusively from the TN5000 deployment validation subsets, was applied unchanged to TN3K. The externally applied thresholds were (τₘᵢ = 0.00205), (τ_low = 0.0204), and (τ_high = 0.6130). The 50% nominal MI-retention target was defined on the TN5000 deployment val_es subset and did not imply that 50% of TN3K images would pass the MI gate. Under dataset shift, only 126 of 614 TN3K images (20.5%) passed the frozen MI gate before probability-gray-zone deferral. Of these low-MI images, 6 received a No-FNA suggestion, 94 received an FNA recommendation, and 26 were assigned to radiologist review because their probabilities fell within the gray zone as shown in (Figure 8). Including images deferred directly by the MI gate, 514 of 614 images (83.7%) were ultimately assigned to radiologist review. No malignant TN3K image was assigned to the No-FNA pathway. Accordingly, observed malignancy capture was 100.0% and observed No-FNA NPV was 100.0% (6/6). These quantities should be interpreted cautiously because only six images received a No-FNA suggestion, all of which were benign. The FNA-recommendation pathway contained 72 malignant and 22 benign images, corresponding to an FNA-recommendation PPV of 76.6%, below the 95% PPV target used during threshold selection on TN5000. The fixed policy preserved malignancy capture primarily through extensive external deferral rather than by preserving the internal autonomous triage pattern. The low No-FNA suggestion rate, high review burden, and reduced FNA-recommendation PPV indicate limited transportability of the fixed TN5000- derived operating policy to TN3K. Site-specific calibration and threshold validation would be required before this policy could be considered for use in a new clinical environment. Figure 8. External strict MI-guided triage assignments on TN3K using fixed TN5000-derived thresholds. 5. Discussion This study evaluated whether a calibrated deterministic deep ensemble could support selective prediction for ROI-based thyroid nodule ultrasound classification by withholding automated image- based suggestions when ensemble disagreement exceeded a validation-derived threshold. Within TN5000, the five-member ensemble achieved a pooled out-of-fold AUC-ROC of 0.9395 and average precision of 0.9715. Following member-wise vector scaling, pooled ECE decreased from 0.0292 to 0.0088 and the Brier score decreased from 0.0836 to 0.0813. The risk-coverage analysis subsequently quantified how classification error changed as cases were retained in ascending order of estimated uncertainty. The strict policy implemented two explicit deferral conditions: an MI gate for ensemble disagreement and a probability gray zone for cases that did not satisfy the rule-out or rule-in criteria. At 50% nominal MI retention, the mean held-out assignment rates were 7.2% for No-FNA suggestion, 39.9% for FNA recommendation, and 52.9% for radiologist review. Mean No-FNA NPV was 98.3%, and mean malignancy capture was 99.83%. Increasing nominal MI retention reduced the radiologist-review rate primarily by increasing the FNA-recommendation rate. The No-FNA suggestion rate remained between 12.7% and 12.9% at the 80%, 90%, and 95% retention settings. Nominal MI retention should therefore be distinguished from the final non-review assignment rate: the former describes passage through the MI gate on val_es, whereas the latter additionally requires a probability below the rule-out threshold or at or above the rule-in threshold. MI was selected as the primary deferral score based on the uncertainty construct under investigation rather than the best internal failure-ranking performance. It quantified disagreement among independently trained ensemble members and was designated for the frozen operating policy before TN3K inference. In the internal comparison, MSP-derived uncertainty achieved AURC and AUGRC values of 0.03020 and 0.02261, followed by predictive variance at 0.03234 and 0.02424 and MI at 0.03778 and 0.02722. MSP-derived uncertainty therefore ranked internal classification errors more effectively according to both summary measures. MI was retained because it measures the difference between the predictive entropy of the ensemble mean and the mean entropy of the individual members, providing an operational proxy for epistemic model disagreement [16]. This rationale does not imply that MI is generally superior to confidence-based uncertainty scores. External transportability was evaluated by applying the deployment ensemble, member-specific calibration parameters, MI threshold, and probability thresholds to TN3K without recalibration or threshold adjustment. AUC-ROC decreased from 0.9395 internally to 0.7870 externally, while average precision decreased from 0.9715 to 0.7254. External ECE and Brier score were 0.1899 and 0.2281, compared with internal pooled values of 0.0088 and 0.0813. MI-based AURC increased from 0.03778 to 0.26018, and AUGRC increased from 0.02722 to 0.15073. Discrimination, probability calibration, and MI-based error ranking were therefore all poorer on TN3K than in the internal OOF evaluation, consistent with evidence that these properties may degrade differently under dataset shift [18], [19]. The fixed policy also produced a different assignment distribution on TN3K. Although the MI threshold had been selected to retain 50% of the deployment val_es subset, only 20.5% of TN3K images passed the MI gate. Across the external cohort, 83.7% of images were assigned to radiologist review, 15.3% received an FNA recommendation, and 1.0% received a No-FNA suggestion. No malignant image was assigned to the No-FNA pathway, yielding observed malignancy capture and No-FNA NPV of 100%. However, only six images received a No-FNA suggestion, making the external NPV estimate statistically imprecise. The FNA-recommendation PPV was 76.6%, below the 95% target used during TN5000 threshold derivation. These results do not demonstrate preservation of the TN5000-derived predictive-value targets on TN3K; the absence of malignancies in the No-FNA group occurred alongside an 83.7% radiologist-review rate. In this study, selective prediction functioned as a routing mechanism: cases outside the TN5000-derived low-disagreement and probability-extreme regions were withheld from automated assignment and directed to review [20], [21]. Deferral does not itself establish clinical safety. Its performance depends on the association between the uncertainty score and prediction error, the validity of the probability thresholds in the target population, and the operational capacity to review deferred cases. Previous thyroid ultrasound studies incorporated uncertainty into multimodal prediction or uncertainty-weighted model training [24], [25]. The present analysis instead evaluated case-level inference-time deferral and application of a frozen operating policy to an external dataset. The TN3K results show that both uncertainty and probability thresholds require target-population validation. The contribution of this study is an empirical evaluation of calibrated ensemble disagreement, a strict three-way retrospective assignment policy, and frozen-policy transportability in thyroid ultrasound ROI classification. The No-FNA assignments should not be interpreted as avoided biopsies because the model did not incorporate nodule size, TI-RADS features, clinical history, laboratory findings, or radiologist assessment. Evaluation for clinical use would require local probability calibration, local validation of the MI and decision thresholds, patient-level prospective testing, and integration with the variables used in established thyroid nodule management pathways. 6. Limitations This study has several limitations. First, it was retrospective and evaluated annotation-assisted ROI classification. TN5000 ROI patches were derived from provided bounding boxes and TN3K ROI patches were derived from provided pixel-wise masks. The framework therefore did not evaluate end- to-end nodule detection, localization, ROI generation, and triage from a complete uncropped ultrasound examination. The observed classification and triage performance may not be preserved when localization errors are introduced. Second, internal splits were performed at the image level. The TN5000 data descriptor states that one representative image from the same patient perspective was retained during dataset curation, but patient identifiers and an image-to-patient mapping were not available in the released materials used in this study. Exact image overlap across training, validation, calibration, and test partitions was excluded; however, patient-level independence could not be verified. If multiple images from the same patient were present under different image identifiers, residual correlation across partitions may have contributed to optimistic internal estimates. Third, ConvNeXt-Tiny was selected in a separate preliminary development experiment without transferring model weights or fitted parameters to the cross-validation experiments. However, because architecture screening and subsequent cross-validation were conducted within the same source dataset, the internal results should be interpreted as conditional on the selected backbone rather than as a fully nested model-selection estimate. Fourth, TN5000 was a malignant-enriched image cohort. Predictive values, triage rates, and threshold behaviour are dependent on prevalence and case mix; they should not be extrapolated to population screening or unselected thyroid-nodule populations. In addition, the study did not include clinical variables that ordinarily influence thyroid-nodule management, such as nodule size, TI-RADS category, patient characteristics, laboratory data, longitudinal imaging findings, and clinician judgement. Consequently, a No-FNA suggestion in this study represents a retrospective image-based model assignment rather than a recommendation to omit biopsy in clinical practice. Fifth, the external evaluation included one TN3K cohort and therefore cannot characterize performance across the full range of clinical sites, machines, acquisition protocols, operators, and patient populations. The differences in annotation format and ROI preparation between TN5000 and TN3K were part of the observed dataset shift, but their individual contributions to external performance degradation could not be isolated. The small number of external No-FNA suggestions also makes the observed external NPV highly imprecise despite the absence of malignant cases in that group. Sixth, the study did not evaluate prospective clinical utility. It did not measure changes in radiologist workload, biopsy rates, time to diagnosis, patient outcomes, cost-effectiveness, or clinician acceptance. 7. Conclusion and Future Work This study presents a calibrated deterministic deep-ensemble framework for selective, annotation- assisted thyroid nodule ROI classification. Within the malignant-enriched TN5000 cohort, the model achieved high internal discrimination and low pooled calibration error. The strict MI-guided policy demonstrated that conservative No-FNA suggestions can be limited to low-MI, low-probability cases while higher-MI and probability-gray-zone cases are referred for radiologist review. These results support the feasibility of using ensemble disagreement as a hypothesis-driven deferral signal within a retrospective image-based triage framework. The external TN3K findings provide an essential qualification. Although the frozen deployment model retained moderate discrimination, probability calibration and threshold transportability deteriorated under dataset shift. The external policy preserved malignancy capture mainly by referring most cases for review, while generating very few No-FNA suggestions and a lower FNA-recommendation PPV. Accordingly, the present results do not support direct clinical deployment or claims of actual biopsy reduction. They support further investigation of selective prediction as a mechanism for identifying cases that should not receive autonomous image-based recommendations. Future work should evaluate the framework using multi-institutional, patient-level datasets with complete linkage between images, nodules, and clinical outcomes. End-to-end studies should incorporate nodule detection and ROI generation from full ultrasound examinations rather than relying on provided lesion annotations. The model should also be evaluated alongside clinical and guideline- based information, including nodule size, TI-RADS features, demographic variables, and radiologist assessment. For external use, calibration and triage thresholds should be locally validated or recalibrated using site-specific data before use. Prospective studies should define the intended workflow, referral pathways, and clinically meaningful utility targets before evaluating the system alongside radiologists. Such studies should assess diagnostic safety, workload, downstream procedures, patient outcomes, and fairness across relevant patient and acquisition subgroups. Future methodological work should also compare MI with alternative uncertainty scores, evaluate selective-prediction performance using risk-coverage metrics, and investigate domain-adaptation or conformal approaches that may provide more stable referral behaviour under dataset shift. Declarations Ethics approval: Not applicable. The study used publicly available, de-identified secondary data. Consent to participate: Not applicable. Consent for publication: Not applicable. Clinical trial number: Not applicable. Funding statement: The Authors received no specific funding for this work. Availability of data and materials: The datasets used in this study (TN5000 and TN3K) are publicly available and have been described in the Methodology section. The code and experimental results are available at: Code & Experimental Results Declaration of competing interest: The authors declare no competing interests. Acknowledgements: Not Applicable. Declaration of Generative AI in Scientific Writing: Generative AI tools were used only for language editing and manuscript refinement. All scientific content, analyses, interpretations, and final responsibility remain with the authors. Author Contributions: 1. Md. Sadibul Hasan Sadib: Conceptualization, Methodology, Formal analysis, Validation and Writing – original draft 2. Md. Mohayminul Mukit: Methodology, Formal analysis, Visualization, Validation and Writing – original draft, review & editing. 3. Rahmatul Kabir Rasel Sarker: Investigation, Supervision, Validation and Writing – review & editing. 4. Tahmid Alam Tamim: Formal analysis, Visualization, Validation, Writing – review. 5. Md. Monir Hossain Shimul: Investigation, Validation, Visualization, Writing – review & editing. References [1] B. R. Haugen et al., “2015 American Thyroid Association management guidelines for adult patients with thyroid nodules and differentiated thyroid cancer: The American Thyroid Association Guidelines Task Force on Thyroid Nodules and Differentiated Thyroid Cancer,” Thyroid, vol. 26, no. 1, p. 1–133, 2016, doi: 10.1089/thy.2015.0020. [2] F. N. Tessler et al., “ACR Thyroid Imaging, Reporting and Data System (TI-RADS): White paper of the ACR TI-RADS Committee,” Journal of the American College of Radiology, vol. 14, no. 5, p. 587–595, 2017, doi: 10.1016/j.jacr.2017.01.046. [3] Q. Guan et al., “Deep learning based classification of ultrasound images for thyroid nodules: A large scale of pilot study,” Annals of Translational Medicine, vol. 7, no. 7, Art. no. 137, 2019, doi: 10.21037/atm.2019.04.34. [4] S. Peng et al., “Deep learning-based artificial intelligence model to assist thyroid nodule diagnosis and management: A multicentre diagnostic study,” The Lancet Digital Health, vol. 3, no. 4, p. e250– e259, 2021, doi: 10.1016/S2589-7500(21)00041-8. [5] Y. Xu, M. Xu, Z. Geng, J. Liu, and B. Meng, “Thyroid nodule classification in ultrasound imaging using deep transfer learning,” BMC Cancer, vol. 25, Art. no. 544, 2025, doi: 10.1186/s12885-025- 13917-3. [6] J. Kim et al., “Deep learning technology for classification of thyroid nodules using multi-view ultrasound images: Potential benefits and challenges in clinical application,” Endocrinology and Metabolism, vol. 40, no. 2, p. 216–224, 2025, doi: 10.3803/EnM.2024.2058. [7] Y. Tao et al., “Deep learning for the diagnosis of suspicious thyroid nodules based on multimodal ultrasound images,” Frontiers in Oncology, vol. 12, Art. no. 1012724, 2022, doi: 10.3389/fonc.2022.1012724. [8] S. Vahdati, B. Khosravi, K. A. Robinson, P. Rouzrokh, M. Moassefi, Z. Akkus, and B. J. Erickson, “A multi-view deep learning model for thyroid nodules detection and characterization in ultrasound imaging,” Bioengineering, vol. 11, no. 7, Art. no. 648, 2024, doi: 10.3390/bioengineering11070648. [9] V. R. Sant et al., “From bench-to-bedside: How artificial intelligence is changing thyroid nodule diagnostics, a systematic review,” Journal of Clinical Endocrinology and Metabolism, vol. 109, no. 7, p. 1684–1693, 2024, doi: 10.1210/clinem/dgae277. [10] J. Weng et al., “Deep learning for classification of thyroid nodules on ultrasound: Validation on an independent dataset,” Clinical Imaging, vol. 99, p. 60–66, 2023, doi: 10.1016/j.clinimag.2023.04.010. [11] J.-H. Ni et al., “Optimizing thyroid nodule management with artificial intelligence: Multicenter retrospective study on reducing unnecessary fine needle aspirations,” JMIR Medical Informatics, vol. 13, Art. no. e71740, 2025, doi: 10.2196/71740. [12] A. Rashed, T. Medhat, and A. Elgarayhi, “Enhancing automatic diagnosis of thyroid nodules from ultrasound scans leveraging deep learning models,” Scientific Reports, vol. 15, Art. no. 40364, 2025, doi: 10.1038/s41598-025-25780-0. [13] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in Proceedings of the 34th International Conference on Machine Learning, vol. 70, 2017, p. 1321–1330. [14] K. Zou, Z. Chen, X. Yuan, X. Shen, M. Wang, and H. Fu, “A review of uncertainty estimation and its application in medical imaging,” Meta-Radiology, vol. 1, no. 1, Art. no. 100003, 2023, doi: 10.1016/j.metrad.2023.100003. [15] T. J. Loftus et al., “Uncertainty-aware deep learning in healthcare: A scoping review,” PLOS Digital Health, vol. 1, no. 8, Art. no. e0000085, 2022, doi: 10.1371/journal.pdig.0000085. [16] B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” in Advances in Neural Information Processing Systems, vol. 30, 2017. [17] Y. Gal and Z. Ghahramani, “Dropout as a Bayesian approximation: Representing model uncertainty in deep learning,” in Proceedings of the 33rd International Conference on Machine Learning, vol. 48, 2016, p. 1050–1059. [18] Y. Ovadia et al., “Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift,” in Advances in Neural Information Processing Systems, vol. 32, 2019. [19] R. Zhang, C. Gatsonis, and J. A. Steingrimsson, “Role of calibration in uncertainty-based referral for deep learning,” Statistical Methods in Medical Research, vol. 32, no. 5, p. 927–943, 2023, doi: 10.1177/09622802231158811. [20] Y. Geifman and R. El-Yaniv, “Selective classification for deep neural networks,” Advances in Neural Information Processing Systems, vol. 30, 2017. [21] J. Traub, T. J. Bungert, C. T. Lüth, M. Baumgartner, K. H. Maier-Hein, L. Maier-Hein, and P. F. Jäger, “Overcoming common flaws in the evaluation of selective classification systems,” arXiv:2407.01032, 2024. [22] H. Zhang, Q. Liu, X. Han, L. Niu, and W. Sun, “TN5000: An ultrasound image dataset for thyroid nodule detection and classification,” Scientific Data, vol. 12, Art. no. 1437, 2025, doi: 10.1038/s41597- 025-05757-4. [23] X. Hou et al., “An ultrasonography of thyroid nodules dataset with pathological diagnosis annotation for deep learning,” Scientific Data, vol. 11, Art. no. 1272, 2024, doi: 10.1038/s41597-024- 04156-5. [24] M. Saini, T. A. Parvar, M. Velarde, N. B. Larson, M. Fatemi, and A. Alizad, “Deep learning-based classification of thyroid nodules using uncertainty-aware multi-modal ultrasound imaging,” Scientific Reports, vol. 16, Art. no. 4938, 2026, doi: 10.1038/s41598-026-35965-w. [25] T. Xiang, B. Li, Z. Hu, X. Wang, Y. Dang, and Y. Deng, “MTL-TN: An uncertainty-weighted multi-task framework for unified thyroid nodule segmentation and TI-RADS classification,” in Proc. 2025 5th Int. Conf. Computer, Internet of Things and Control Engineering (CITCE), 2025, p. 44–48, doi: 10.1109/CITCE67565.2025.11360039. [26] X. Zhao et al., “Thyroid nodule and lymph node metastasis assessment from ultrasound images using deep learning,” Frontiers in Neuroscience, vol. 19, Art. no. 1684104, 2025, doi: 10.3389/fnins.2025.1684104. [27] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A ConvNet for the 2020s,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, p. 11966–11976, doi: 10.1109/CVPR52688.2022.01167. [28] J. Hu, L. Shen, S. Albanie, G. Sun, and E. Wu, “Squeeze-and-excitation networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, p. 7132–7141, doi: 10.1109/CVPR.2018.00745. [29] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-CAM: Visual explanations from deep networks via gradient-based localization,” in Proceedings of the IEEE International Conference on Computer Vision, 2017, p. 618–626, doi: 10.1109/ICCV.2017.74. Supplementary Materials S1. ROI Construction and Input Standardization For TN5000, lesion regions of interest (ROIs) were extracted from the provided Pascal VOC XML bounding-box annotations. Each bounding box (푥 푚푖푛, 푦 푚푖푛, 푥 푚푎푥, 푦 푚푎푥 ) was expanded by 20% in both width and height and clipped to the original image boundaries. The resulting crops were directly resized to 224 × 224 pixels using Lanczos interpolation without forcing square crops or adding synthetic padding. Examples of TN5000 and TN3K ROI construction are shown in Supplementary Figure S1. For TN3K, lesion ROIs were generated from the official segmentation masks available for the 614 test images. The smallest bounding box enclosing the nonzero mask region was extracted, expanded by 20% in width and height, and clipped to the image boundaries. TN3K crops were resized proportionally to fit within a 224 × 224 pixel canvas using bilinear interpolation, with remaining regions filled using zero-valued black padding. Representative lesion-centred ROI crops from both datasets are shown in Supplementary Figure S2. For both datasets, ROI images were converted to grayscale, autocontrast-adjusted with a 1% cutoff, converted to three-channel RGB format, and normalized using ImageNet channel statistics. No explicit removal of calipers, textual labels, ruler marks, or machine overlays was performed. Because TN5000 used direct resizing while TN3K used aspect-ratio-preserving resizing with black padding, these preprocessing differences were considered potential contributors to the observed external dataset shift. Supplementary Figure S1. ROI extraction examples from TN5000 and TN3k datasets Supplementary Figure S2. Representative lesion-centred ROI crops from the TN5000 and TN3K datasets. S2. Pooled OOF Risk-Coverage Analysis of Uncertainty Scores Pooled out-of-fold (OOF) risk-coverage curves were used to compare three uncertainty-score rankings: MSP-derived uncertainty, predictive variance, and mutual information (MI). The corresponding pooled OOF risk-coverage curves are shown in Supplementary Figure S3. For each score, images were sorted by ascending uncertainty, and selective risk was calculated after retaining progressively larger fractions of the 5,000 OOF predictions. Lower AURC and AUGRC values indicate better ranking of potential silent failures, meaning that higher-risk predictions are deferred earlier as coverage decreases. MSP-derived uncertainty achieved the lowest pooled OOF AURC and AUGRC values (AURC = 0.03020; AUGRC = 0.02261), followed by predictive variance (AURC = 0.03234; AUGRC = 0.02424) and MI (AURC = 0.03778; AUGRC = 0.02722). All three scores outperformed random deferral (AURC = 0.10980; AUGRC = 0.05491). These results indicate that MSP-derived uncertainty provided the strongest internal ranking of potential prediction failures in the pooled OOF analysis. However, MI was retained as the primary deferral signal in the main triage policy because it directly aligned with the study hypothesis concerning disagreement among independently trained ensemble members. Supplementary Figure S3. Pooled out-of-fold risk-coverage analysis of uncertainty scores. Pooled OOF risk-coverage curves comparing MSP-derived uncertainty, predictive variance, and mutual information for selective deferral. Lower AURC and AUGRC indicate better uncertainty-based ranking of potential prediction failures. S3. Web-Based Research Interface An interactive web-based research interface was developed to demonstrate the proposed thyroid nodule ROI evaluation workflow (Supplementary Figure S4). Users can upload a thyroid nodule ROI, which is analyzed using the fixed calibrated five-member ConvNeXt-Tiny + SE ensemble. The interface reports the ensemble malignancy probability, predicted class, mutual information (MI), predictive variance, individual member probabilities, and locked strict-triage assignment. A qualitative Grad-CAM visualization is also generated for the uploaded ROI using calibrated Member 0, targeted to that member’s predicted class. Deployment thresholds remain fixed and cannot be modified by user input. The interface also provides a machine-readable JSON summary of the prediction and uncertainty outputs. This implementation demonstrates how the retrospective selective-prediction pipeline could be presented in an accessible research workflow. It remains a research prototype and is not a clinical diagnostic device or a replacement for radiologist assessment. Supplementary Figure S4. Web-based research prototype displaying ensemble predictions, member disagreement, mutual information, qualitative Grad-CAM output, and the retrospective image-based triage assignment.