Paper deep dive
How Many Labels Are Enough? ALDA: Active Learning Deployment Advisor for Medical Image Classification
Julia Machnio, Mads Nielsen, Mostafa Mehdipour Ghazi
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Active learning (AL) promises to reduce the cost of medical imaging projects by lowering the number of clinical labels required. However, practical deployment requires committing to a sampling strategy before the full annotation budget is spent, and choosing the wrong strategy can increase rather than decrease costs. We propose Active-Learning Deployment Advisor (ALDA), a deployment-oriented framework for AL method selection under clinical performance constraints. Given a short pilot phase, ALDA fits a parametric learning-curve model to each candidate strategy, estimates whether that strategy is expected to reach a required clinical performance target, and predicts the number of expert annotations needed to do so. In addition to absolute annotation cost, ALDA introduces a deployment window that quantifies the sensitivity of this cost estimate to uncertainty in the clinical threshold. The final recommendation follows a risk-aware rule: among strategies with near-optimal predicted cost, ALDA prefers the strategy with the narrowest deployment window, the most robust to threshold revisions. Experiments on four medical imaging classification domains show that ALDA predicts the deployment-optimal method from a pilot of 15-30% of the intended budget and reduces annotation costs by up to 82% compared with a poor strategy choice. Rather than introducing a new sampling heuristic, ALDA provides a practical decision layer that answers a deployment-critical question: how many labels are enough?
Tags
Links
- Source: https://arxiv.org/abs/2608.03511v1
- Canonical: https://arxiv.org/abs/2608.03511v1
Trouble viewing inline? Open PDF directly →
Full Text
40,164 characters extracted from source content.
Expand or collapse full text
11institutetext: Pioneer Centre for AI, University of Copenhagen, Copenhagen, Denmark 11email: juma,madsn,ghazi@di.ku.dk How Many Labels Are Enough? ALDA: Active Learning Deployment Advisor for Medical Image Classification Julia Machnio Mads Nielsen Mostafa Mehdipour Ghazi Abstract Active learning (AL) promises to reduce the cost of medical imaging projects by lowering the number of clinical labels required. However, practical deployment requires committing to a sampling strategy before the full annotation budget is spent, and choosing the wrong strategy can increase rather than decrease costs. We propose Active-Learning Deployment Advisor (ALDA), a deployment-oriented framework for AL method selection under clinical performance constraints. Given a short pilot phase, ALDA fits a parametric learning-curve model to each candidate strategy, estimates whether that strategy is expected to reach a required clinical performance target, and predicts the number of expert annotations needed to do so. In addition to absolute annotation cost, ALDA introduces a deployment window that quantifies the sensitivity of this cost estimate to uncertainty in the clinical threshold. The final recommendation follows a risk-aware rule: among strategies with near-optimal predicted cost, ALDA prefers the strategy with the narrowest deployment window, the most robust to threshold revisions. Experiments on four medical imaging classification domains show that ALDA predicts the deployment-optimal method from a pilot of 15–30% of the intended budget and reduces annotation costs by up to 82% compared with a poor strategy choice. Rather than introducing a new sampling heuristic, ALDA provides a practical decision layer that answers a deployment-critical question: how many labels are enough? 1 Introduction Medical imaging labels are expensive because they require expert knowledge. Depending on the task, annotation may involve radiologists, pathologists, or other trained clinical specialists whose time is limited and whose expertise cannot easily be substituted. This raises a practical question: are all labels necessary? Active learning (AL) addresses this question by iteratively selecting the most informative samples for annotation, aiming to reach a target performance with fewer labeled examples. In principle, AL reduces annotation cost; in practice, choosing the wrong strategy can increase it. A clinical team must select an AL method before knowing which strategy will work best for their dataset and task, yet AL method rankings are sensitive to dataset characteristics, budgets, and class-imbalance profiles [1, 2]; a method that is label-efficient in one setting may be suboptimal in another. AL deployment, therefore, becomes retrospective: annotation resources are spent first, and only afterwards does the team learn whether the chosen method was cost-effective. Standard AL evaluation protocols do not address this. Methods are compared post hoc by final accuracy or area under the learning curve on fixed benchmark datasets [3], answering the wrong question: not “which method will reach our target cheapest?” but “which method ranked highest on these benchmarks?”. A further complication is that the clinical threshold τ is rarely a precise point value: regulatory submissions may specify a range, and the team may revise the target after initial results. A deployment framework must therefore estimate annotation cost and quantify how sensitive that estimate is to threshold uncertainty. Figure 1: (A) Schematic of ALDA deployment quantities. The clinical target τ defines the required operating point; the shaded band denotes threshold uncertainty ±Δτ± τ. Method C is infeasible because its predicted ceiling lies below τ. Among feasible methods, BabsB<BabsAB_abs^B<B_abs^A and WB<WAW^B<W^A, so method B is more efficient and less risky. (B) Labeling-cost savings of the ALDA recommendation relative to the most expensive feasible method on each dataset, predicted from a 30% pilot (details in Table˜2). Recent work on early AL curve modeling [4] has shown that AL trajectories can be forecast from partial observations using an interpretable parametric function, but trajectory forecasting alone does not yield a deployment recommendation. We adopt this parametric curve as a building block and contribute ALDA, a decision framework that turns the forecast into an actionable choice. Given a short pilot phase in which all candidate methods run for 15–30% of the intended budget, ALDA fits the curve to each method, checks feasibility, estimates the absolute annotation cost Babs(τ)B_abs(τ), and computes a deployment window W that quantifies sensitivity to threshold uncertainty (Figure˜1). The output is a two-number deployment summary that identifies the recommended method and flags threshold-sensitive strategies before the full budget is committed. Our main contributions: (1) We formulate the AL method selection as a prospective deployment problem in medical imaging, distinct from retrospective benchmarking. (2) We introduce BabsB_abs, an absolute annotation-cost metric that answers how many labels are needed to reach a clinician-specified performance threshold, combined with a feasibility check that identifies methods predicted never to reach the target without committing the full annotation run. (3) We introduce the deployment window W, which measures the sensitivity of the annotation-cost estimate to uncertainty in the clinical threshold, providing an operational risk indicator for deployment planning. (4) We show that BabsB_abs and W are correlated across datasets: the cheapest feasible method is typically also among the most robust to threshold revisions, so optimizing for annotation cost simultaneously tends to minimize threshold sensitivity. (5) We show that ALDA identifies a label-efficient method from a pilot of 15–30% of the intended budget across four medical imaging datasets. ALDA does not propose a new sampling strategy. It adds a deployment decision layer on top of existing AL methods, answering the question of whether AL is useful in practice. 2 Related Work AL sampling strategies. Pool-based AL methods select unlabeled samples for annotation according to uncertainty [5], diversity or coverage of the feature space (CoreSet [6], BADGE [7]), or geometric criteria such as typicality and probability coverage (TypiClust [1], ProbCover [8]). Large-scale comparative studies [1, 2] consistently find that no single strategy dominates across tasks and budgets, providing a strong motivation for prospective method selection. AL in medical imaging. AL has been applied across histology [9], radiology [10], and ultrasound classification [11]. Class imbalance, inter-annotator variability, and small initial pools make method rankings especially sensitive to dataset specifics, amplifying the cost of a poor choice. Learning-curve prediction. Parametric learning-curve models have a long history in supervised learning [12, 13], where they support sample-size planning from small pilots. Extending such models to the AL setting is more recent: PALM [4] proposed a parametric AL curve that can be fit from partial trajectories and used to forecast final performance. We adopt this model as the curve-fitting backbone of ALDA, but contribute a separate decision layer (feasibility screening, absolute cost prediction, deployment-window analysis, and risk-aware selection), converting a trajectory forecast into a deployment recommendation. Stopping criteria. A related but distinct line of work studies when to stop annotation given a fixed AL strategy [14, 15]. ALDA is complementary: it addresses which strategy to commit to before annotation begins. 3 Methods ALDA treats AL deployment as a decision problem under uncertainty; rather than ranking methods by predicted performance alone, it estimates three quantities: feasibility, expected annotation cost, and sensitivity to the clinical target. 3.1 Learning-Curve Model We adopt the parametric AL learning curve of PALM [4]: A=Amax[1−(1−δ)(Bb+α)β],A=A_ [1- (1-δ ) ( Bb+α )^β ], (1) where b>0b>0 is the per-episode acquisition budget, B the total annotation budget, Amax∈(0,1]A_ ∈(0,1] the estimated asymptotic accuracy, δ∈(0,1)δ∈(0,1) the per-episode (average coverage) gain coefficient, α≥0α≥ 0 the budget-origin shift (seed-set contribution on early performance), and β>0β>0 the diminishing-returns exponent (gains scaling factor). For each method m∈ℳm , ALDA fits Eq. (1) to the pilot curve by nonlinear least squares (L-BFGS-B with multiple random restarts), obtaining the fit parameters θm=Amax(m),δ(m),α(m),β(m) _m=\A_ ^(m),δ^(m),α^(m),β^(m)\. 3.2 Clinical Target and Feasibility Let τ denote the minimum acceptable performance for clinical deployment. A method is feasible if Amax(m)≥τA_ ^(m)≥τ, where its predicted ceiling reaches the clinical requirement. Infeasible methods are excluded before any cost calculation: additional labeling is not expected to close the gap to τ. 3.3 Absolute Annotation Cost For each feasible method, the absolute annotation cost is the smallest budget at which predicted performance reaches τ: Babs(m)(τ)=minB∈ℤ>0:Am(B)≥τB_abs^(m)(τ)= \B _>0:A_m(B)≥τ\. Inverting Eq. (1) gives a closed-form expression: Babs(m)(τ)=b⋅⌈(log(1−τ/Amax(m))log(1−δ(m)))1/β(m)−α(m)⌉,B_abs^(m)(τ)=b·\! ( (1-τ/A_ ^(m) ) (1-δ^(m) ) )^\!1/β^(m)-α^(m) , (2) where ⌈⋅⌉ · denotes rounding up to the nearest multiple of Δb _b (the next AL episode). Both logarithms are negative for feasible inputs (τ<Amaxτ<A_ and δ∈(0,1)δ∈(0,1)), so their ratio is positive, and the root is real. 3.4 Threshold Sensitivity and Deployment Risk The clinical threshold τ may be revised after expert consultation, local validation, or regulatory review. The cheapest method at the nominal threshold is therefore not necessarily the safest: a small upward revision in τ may require many additional labels if the curve has flattened. For an accepted threshold uncertainty Δτ τ (in percentage points), the deployment risk window is W(m)=Babs(m)(τhi)−Babs(m)(τlo),W^(m)=B_abs^(m)( _hi)-B_abs^(m)( _lo), (3) where τlo=τ−Δτ _lo=τ- τ and τhi=τ+Δτ _hi=τ+ τ. To avoid evaluating BabsB_abs arbitrarily close to the asymptote (where small errors in AmaxA_ produce arbitrarily large cost estimates), τhi _hi is capped per method at Amax(m)−ϵA_ ^(m)-ε for a small ϵ>0ε>0 (ϵ=0.5pε=0.5\,p). A small W(m)W^(m) indicates a stable deployment decision, whereas a large W(m)W^(m) indicates a threshold-sensitive method; the latter is flagged as deployment-risky. Risk-aware recommendation. ALDA combines expected annotation cost and deployment risk using a cost non-inferiority rule. Let Bmin=minm∈ℳfeasBabs(m)(τ)B_ = _m _feasB_abs^(m)(τ) (4) be the lowest predicted annotation cost among feasible methods. Define the near-optimal cost set η=m∈ℳfeas:Babs(m)(τ)−BminBmin≤η,C_η= \m _feas: B_abs^(m)(τ)-B_ B_ ≤η \, (5) where η≥0η≥ 0 is the tolerated relative increase in annotation cost. The final ALDA recommendation is the lowest-risk method within this near-optimal set: m∗=argminm∈ηW(m).m^*= _m _ηW^(m). (6) Thus, BabsB_abs determines which methods are cost-competitive and W selects the most robust option among them. The pipeline is summarized in Algorithm 1. Algorithm 1 ALDA: Risk-aware AL deployment 1:Candidate AL methods ℳM; pilot learning curves; clinical target τ; threshold uncertainty Δτ τ; cost tolerance η. 2:for each method m∈ℳm do 3: Fit the curve AmA_m to the pilot trajectory. 4: Estimate m=Amax(m),δ(m),α(m),β(m) θ_m=\A_ ^(m),δ^(m),α^(m),β^(m)\. 5: Mark m infeasible if Amax(m)<τA_ ^(m)<τ. 6: if m is feasible then 7: Compute Babs(m)(τ)B_abs^(m)(τ) via Eq. (2). 8: Compute W(m)W^(m) via Eq. (3). 9: end if 10:end for 11:Bmin←minm∈ℳfeasBabs(m)(τ)B_ \!←\! _m _feas\!B_abs^(m)(τ); η=m:(Babs(m)(τ)−Bmin)/Bmin≤ηC_η\!=\!\m:(B_abs^(m)(τ)-B_ )/B_ \!≤\!η\. 12:m∗=argminm∈ηW(m)m^*= _m _ηW^(m); flag infeasible/risky methods. 13:return m∗m^*, Babs(m∗)B_abs^(m^*), W(m∗)W^(m^*), risk flags. 4 Results and Discussion 4.1 Experimental Setting Datasets. We evaluate ALDA on four medical imaging classification datasets: BRISC2025 [16], ISIC2019 [17], Fetal Planes [18], and BUSI [19]. Details of the utilized dataset are summarized in Table˜1. Table 1: Dataset summary. We report the imaging domain, input resolution, number of classes, train/test split used in our experiments, and whether the dataset is class-imbalanced. We used the test subset of ISIC2019, excluding its unknown (UNK) class. Dataset Domain Resolution #Classes #Train #Test Imbalanced BRISC2025 Brain tumour MRI 486×475 4 5,000 1,000 No ISIC2019 Skin lesion 905×936 8 4,953 1,238 Yes Fetal Planes Fetal ultrasound 548×822 6 7,129 5,271 Yes BUSI Breast ultrasound 501×616 3 624 156 Yes AL protocol. All experiments follow a pool-based AL protocol. At each episode, an AL method selects a batch of unlabeled samples; these are added to the labeled set, and a ResNet-18 model [20] is retrained on the expanded labeled set. At each AL episode, a batch of Δb=20 _b=20 samples is acquired until exhausting the unlabeled pool, except for BUSI, where Δb=10 _b=10. Each method–dataset pair is run for three independent random seeds; reported numbers are seed averages. Additional details are available in Appendix˜0.A. AL strategies. We compare nine AL methods spanning uncertainty-, coverage-, and representation-based selection: Random, Margin, Entropy, Uncertainty, BADGE [7], CoreSet [6], TypiClust [1], ProbCover [8], UHerding [21]. ALDA is fitted to partial trajectories for each method, producing method-specific estimates of AmaxA_ , Babs(τ)B_abs(τ), and W. Unless otherwise stated, we use a threshold uncertainty of Δτ=5p τ=5\,p and a cost non-inferiority margin η=0.05η=0.05. Evaluation. We report two complementary evaluations. First, we assess whether ALDA identifies the method requiring the fewest expert labels from a 30% pilot phase while flagging infeasible or threshold-sensitive alternatives (Section˜4.2), comparing against the full AL trajectories, and report the annotations saved by the ALDA recommendation against each alternative. Second, to evaluate the prospective deployment scenario, we fit the curve to early pilot prefixes (10–30% of the full trajectory) and measure whether the resulting ALDA recommendation matches the full-curve oracle or incurs only low label regret (Section˜4.3). 4.2 Full-Curve Method Selection Results Table 2: Pilot-based deployment estimates for AL method selection. ALDA is fitted to a 30% pilot phase and used to estimate target (τ) feasibility, absolute annotation cost (BabsB_abs), deployment window W, and the scale-free risk ratio W/BabsW/B_abs. AachA_ach is the achieved final accuracy by the model. Dark and light greens indicate the ALDA deployment recommendations (1st and 2nd best). Red indicates methods screened as infeasible. Saved [%] is the relative reduction in BabsB_abs by the ALDA pick versus each alternative; negative values mark methods with marginally lower BabsB_abs than ALDA that were rejected by the W tiebreak within the η-band. AL Method Fetal Planes, τ=85%τ=85\% BRISC2025, τ=95%τ=95\% Aach/Amax[%]A_ach/A_ [\%] BabsB_abs W(m)W(m) W/BabsW/B_abs Saved [%] Aach/AmaxA_ach/A_ BabsB_abs W(m)W(m) W/BabsW/B_abs Saved [%] BADGE 93.5 / 93.4 510 586 1.15 +14.7 99.2 / 99.2 895 1,630 1.82 CoreSet 91.9 / 89.6 1,456 1,632 1.12 +70.1 99.2 / 99.8 1,351 3,013 2.23 +33.8 Entropy 93.4 / 93.4 518 548 1.06 +16.0 99.2 / 99.2 880 1,741 1.98 -1.6 Margin 93.7 / 94.1 435 590 1.36 99.2 / 100.0 947 3,330 3.51 +5.6 ProbCover 92.5 / 99.9 2,418 1,536 0.64 +82.0 98.2 / 100.0 3,099 1,946 0.63 +71.1 Random 92.1 / 93.5 684 1,156 1.69 +36.4 98.8 / 100.0 1,800 7,107 3.95 +50.3 TypiClust 92.3 / 92.5 646 1,131 1.75 +32.7 98.4 / 100.0 2,603 11,107 4.27 +65.6 UHerding 92.5 / 93.1 495 605 1.22 +12.1 99.2 / 98.9 933 1,967 2.11 +4.1 Uncertainty 93.7 / 93.5 487 493 1.01 +10.7 99.2 / 100.0 893 3,203 3.59 -0.2 AL Method ISIC2019, τ=70%τ=70\% BUSI, τ=75%τ=75\% Aach/AmaxA_ach/A_ BabsB_abs W(m)W(m) W/BabsW/B_abs Saved [%] Aach/AmaxA_ach/A_ BabsB_abs W(m)W(m) W/BabsW/B_abs Saved [%] BADGE 73.6 / 100.0 4,886 6,633 1.36 +64.9 84.2 / 100.0 263 290 1.10 +20.6 CoreSet 74.3 / 100.0 3,731 3,384 0.91 +54.0 82.5 / 100.0 316 270 0.85 +34.0 Entropy 75.6 / 100.0 2,195 1,453 0.66 +21.9 84.0 / 91.1 209 257 1.23 Margin 75.5 / 100.0 2,264 1,949 0.86 +24.2 82.3 / 87.2 208 286 1.37 -0.5 ProbCover 68.7 / 54.7 −- — — — 81.4 / 78.0 225 208 0.92 +7.1 Random 75.0 / 100.0 4,151 5,216 1.26 +58.7 82.7 / 85.9 265 291 1.10 +21.2 TypiClust 74.6 / 93.1 3,362 4,056 1.21 +49.0 83.5 / 100.0 277 298 1.08 +24.5 75.0 / 100.0 1,715 744 0.43 83.3 / 82.1 233 261 1.12 +10.3 Uncertainty 74.3 / 100.0 2,600 2,101 0.81 +34.0 83.5 / 82.1 233 227 0.97 +10.5 Table˜2 reports the annotation cost Babs(τ)B_abs(τ), deployment window W, achieved accuracy after labeling the full dataset AachA_ach, and predicted ceiling AmaxA_ for every method on all datasets, computed from a 30% pilot and compared against the full learning curves. ALDA selects the cheapest feasible method whenever a clear minimum exists, and otherwise picks the cost-competitive one with the smallest window. In feasible scenarios, absolute costs differ by up to 82% (Margin: 435 vs. ProbCover: 2,418), 71% (BADGE: 895 vs. ProbCover: 3,099), and 34% (Entropy: 209 vs. CoreSet: 316). On BRISC2025, Entropy (880) is nominally cheapest, but BADGE (895) and Uncertainty (893) lie inside the η=5%η=5\% band around it; BADGE wins the W tiebreak (1,630 vs. 1,741 for Entropy and 3,203 for Uncertainty). Representation-based methods (TypiClust, ProbCover) require 2,600–3,100 labels. Similarly, on BUSI (τ=75%τ=75\%), Entropy (209, W=257W=257) is preferred over the nominally cheaper Margin (208, W=286W=286). ProbCover is screened as infeasible on ISIC2019 (predicted Amax=54.7%A_ =54.7\%, well below τ). 4.3 Pilot-Phase Reliability Results Figure 2: Pilot-based ALDA predictions from partial AL trajectories. For each dataset, the parametric curve is fitted to increasing pilot fractions (10%, 15%, 20%, 25%, 30%) and used to extrapolate the label budget required to reach the target threshold τ. The black curve denotes the oracle full-trajectory fit for the method selected on the complete trajectory; colored dashed curves show the methods selected from each pilot prefix. Dots mark the observed pilot endpoints used for fitting; dashed segments show the remaining extrapolated prediction. Horizontal lines mark the deployment target τ. To evaluate the prospective deployment scenario, we fit the curve model to early prefixes of each trajectory (10–30%) and apply ALDA. Figure˜2 shows the method selected at each pilot fraction together with the full-curve oracle, defined as the method minimizing Babs(τ)B_abs(τ) alone (no W tiebreak) as the strongest baseline, since it is the absolute lower bound on annotation cost. The BabsB_abs-only oracle and the risk-aware ALDA pick can differ within an η-band: on BUSI, the oracle is Margin (208 labels) while ALDA picks Entropy (209 labels, smaller W). ALDA converges to a low-regret recommendation from small pilots. On Fetal and ISIC2019, the pilot pick matches the oracle from 20% onward and remains stable thereafter, with predicted curves aligning with the oracle fit. On BRISC2025 and BUSI, the pick fluctuates within the same η-band before converging by 25–30%, as predicted BabsB_abs is not strictly monotonic in the pilot size. Sensitivity analyses for Δτ τ and W-cap are in Appendix˜0.B. 5 Conclusion We presented ALDA, a framework that turns AL method selection from a retrospective benchmarking exercise into a prospective deployment decision. ALDA fits a parametric learning-curve model to a short pilot, screens for feasibility, estimates the absolute cost, and quantifies sensitivity via the deployment window; the recommendation is risk-aware, among cost-competitive feasible methods, the most threshold-robust option is preferred. Across four medical imaging datasets, ALDA identifies a label-efficient method from a 15–30% pilot and reduces annotation costs by up to 82% relative to poor strategy choices. ALDA’s pilot phase only requires annotating a pilot subset of 20–30% of the intended budget. Each candidate method induces an ordering over the pilot samples, yielding a method-specific partial trajectory at no extra labeling cost. ALDA fits these trajectories to estimate feasibility, target-reaching cost, and threshold sensitivity, and recommends a method on which the remaining budget is spent. Method selection thus shifts from retrospective to prospective, with the additional cost being computational (training on small pilot subsets) rather than annotation-based. credits 5.0.1 Acknowledgements This project is supported by the Pioneer Centre for AI, funded by the Danish National Research Foundation (grant number P1). 5.0.2 The authors have no competing interests. Code Availability. Code is available at: https://github.com/juliamachnio/PALM. References [1] Hacohen, G., Dekel, A., Weinshall, D.: Active learning on a budget: Opposite strategies suit high and low budgets. In: International Conference on Machine Learning, PMLR (2022) 8175–8195 [2] Machnio, J., Nielsen, M., Mehdipour Ghazi, M.: A mechanism-driven theory of phase transitions in active learning. arXiv preprint arXiv:2607.00144 (2026) [3] Zhan, X., Wang, Q., Huang, K.h., Xiong, H., Dou, D., Chan, A.B.: A comparative survey of deep active learning. arXiv preprint arXiv:2203.13450 (2022) [4] Machnio, J., Nielsen, M., Mehdipour Ghazi, M.: To label or not to label: PALM-a predictive model for evaluating sample efficiency in active learning models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. (2025) 4039–4048 [5] Wang, K., Zhang, D., Li, Y., Zhang, R., Lin, L.: Cost-effective active learning for deep image classification. IEEE Transactions on Circuits and Systems for Video Technology 27(12) (2016) 2591–2600 [6] Sener, O., Savarese, S.: Active learning for convolutional neural networks: A core-set approach. In: International Conference on Learning Representations. (2018) [7] Ash, J.T., Zhang, C., Krishnamurthy, A., Langford, J., Agarwal, A.: Deep batch active learning by diverse, uncertain gradient lower bounds. In: International Conference on Learning Representations. (2020) [8] Yehuda, O., Dekel, A., Hacohen, G., Weinshall, D.: Active learning through a covering lens. Advances in Neural Information Processing Systems 35 (2022) 22354–22367 [9] Yang, L., Zhang, Y., Chen, J., Zhang, S., Chen, D.Z.: Suggestive annotation: A deep active learning framework for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer (2017) 399–407 [10] Tajbakhsh, N., Jeyaseelan, L., Li, Q., Chiang, J.N., Wu, Z., Ding, X.: Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation. Medical Image Analysis 63 (2020) 101693 [11] Smailagic, A., Costa, P., Noh, H.Y., Walawalkar, D., Khandelwal, K., Galdran, A., Mirshekari, M., Fagert, J., Xu, S., Zhang, P., et al.: Medal: Accurate and robust deep active learning for medical image analysis. In: 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), IEEE (2018) 481–488 [12] Mukherjee, S., Tamayo, P., Rogers, S., Rifkin, R., Engle, A., Campbell, C., Golub, T.R., Mesirov, J.P.: Estimating dataset size requirements for classifying dna microarray data. Journal of Computational Biology 10(2) (2003) 119–142 [13] Figueroa, R.L., Zeng-Treitler, Q., Kandula, S., Ngo, L.H.: Predicting sample size required for classification performance. BMC Medical Informatics and Decision Making 12(1) (2012) 8 [14] Bloodgood, M., Vijay-Shanker, K.: A method for stopping active learning based on stabilizing predictions and the need for user-adjustable stopping. In: Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009). (2009) 39–47 [15] Zhu, J., Wang, H., Tsou, B.K., Ma, M.: Active learning with sampling by uncertainty and density for data annotations. IEEE Transactions on Audio, Speech, and Language Processing 18(6) (2009) 1323–1331 [16] Fateh, A., Rezvani, Y., Moayedi, S., Rezvani, S., Fateh, F., Fateh, M., Abolghasemi, V.: BRISC: Annotated dataset for brain tumor segmentation and classification. Scientific Data (2026) [17] Gessert, N., Nielsen, M., Shaikh, M., Werner, R., Schlaefer, A.: Skin lesion classification using ensembles of multi-resolution EfficientNets with meta data. MethodsX 7 (2020) 100864 [18] Burgos-Artizzu, X.P., Coronado-Gutiérrez, D., Valenzuela-Alcaraz, B., Bonet-Carne, E., Eixarch, E., Crispi, F., Gratacós, E.: Evaluation of deep convolutional neural networks for automatic classification of common maternal fetal ultrasound planes. Scientific Reports 10(1) (2020) 10200 [19] Al-Dhabyani, W., Gomaa, M., Khaled, H., Fahmy, A.: Dataset of breast ultrasound images. Data in Brief 28 (2020) 104863 [20] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2016) 770–778 [21] Bae, W., Sutherland, D., Oliveira, G.: Uncertainty herding: One active learning method for all label budgets. In: International Conference on Learning Representations. Volume 2025. (2025) 788–805 Appendix 0.A Experimental Setup and Reproducibility Details Initialization. We evaluate ALDA in a cold-start active learning (AL) setting. For acquisition functions that require a trained model to compute scores, such as Uncertainty, Entropy, Margin, BADGE, and related uncertainty-based methods, the first acquisition batch is selected randomly. Subsequent batches are selected according to the corresponding acquisition rule. Training protocols. All classifiers are trained using stochastic gradient descent (SGD) with Nesterov momentum 0.90.9, weight decay 3×10−43× 10^-4, and a cosine learning rate schedule with initial learning rate 0.0250.025. Batch size is set between 64 and 128, depending on dataset size and GPU memory. For each AL episode, the model is trained for 100 epochs on the currently labeled subset. All acquisition methods within a dataset use the same training protocol, number of epochs, evaluation split, and data preprocessing. Models are evaluated on the held-out test set after each AL episode. We use standard image augmentations, including random cropping and horizontal flipping, following common AL benchmark practice. Feature representations. TypiClust and ProbCover require feature-space representations. For these methods, we extract image embeddings using a pretrained MoCo-v3 with a ResNet-50 encoder. Features are extracted from the penultimate layer and normalized using the mean and standard deviation computed from the training set. The same normalization statistics are applied to validation and test representations. Random seeds. Each method–dataset pair is evaluated over three independent random seeds. All methods use matched seeds, dataset splits, and training conditions. Unless otherwise stated, reported values are means over seeds. Standard deviations are reported for final test accuracy and macro-F1 in Table˜3. Implementation details. All experiments are implemented in PyTorch. Experiments were run on NVIDIA A100, TITAN RTX, and Quadro RTX 6000 GPUs (with 24–80 GB VRAM). Appendix 0.B Ablation Study 0.B.1 Final Accuracy and Macro-F1 Table˜3 reports final test accuracy and macro-F1 after emptying the unlabeled pool of samples. This analysis complements the deployment-cost results in the main paper. ALDA is designed to select a method that reaches a target performance with low annotation cost and low threshold sensitivity; it is not designed to maximize final accuracy after all available labels have already been used. Nevertheless, the selected methods are generally competitive at the final episode. Macro-F1 is particularly important for the class-imbalanced datasets, where accuracy alone can overstate performance on majority classes. Across Fetal Planes, BRISC2025, and BUSI, the relative ranking induced by final accuracy and macro-F1 is broadly consistent, with only small differences between the two metrics. ISIC2019 shows a larger gap between accuracy and macro-F1, reflecting its stronger class imbalance. Even in this setting, the ALDA-selected method remains competitive, supporting the use of the target-reaching deployment metrics in the main analysis. Table 3: Final test accuracy and macro-F1 (mean ± std over three seeds) at the last training episode for each AL method and dataset. This table complements the deployment-cost analysis by showing the final predictive performance after the full annotation budget has been consumed. AL Method Fetal Planes BRISC2025 ISIC2019 BUSI Acc. F1 Acc. F1 Acc. F1 Acc. F1 BADGE 93.4±0.193.4±0.1 92.9±0.292.9±0.2 99.0±0.399.0±0.3 99.1±0.199.1±0.1 73.9±0.373.9±0.3 60.5±0.860.5±0.8 81.6±2.081.6±2.0 78.7±1.178.7±1.1 CoreSet 92.2±1.192.2±1.1 91.3±1.591.3±1.5 99.1±0.199.1±0.1 99.0±0.299.0±0.2 73.3±0.973.3±0.9 59.4±1.559.4±1.5 81.6±2.681.6±2.6 80.3±1.880.3±1.8 Entropy 93.1±0.293.1±0.2 92.6±0.292.6±0.2 99.1±0.299.1±0.2 99.2±0.399.2±0.3 74.1±0.374.1±0.3 60.5±1.460.5±1.4 82.1±2.682.1±2.6 77.5±2.577.5±2.5 Margin 93.5±0.193.5±0.1 92.9±0.292.9±0.2 99.1±0.299.1±0.2 98.9±0.298.9±0.2 74.7±1.174.7±1.1 60.9±2.660.9±2.6 80.1±2.380.1±2.3 78.0±3.978.0±3.9 ProbCover 91.9±0.291.9±0.2 91.1±0.191.1±0.1 97.6±0.297.6±0.2 97.5±0.897.5±0.8 68.0±2.568.0±2.5 45.7±3.445.7±3.4 79.1±2.779.1±2.7 72.5±4.872.5±4.8 Random 93.0±1.093.0±1.0 92.3±1.092.3±1.0 98.8±0.498.8±0.4 98.8±0.298.8±0.2 75.0±1.175.0±1.1 60.6±0.860.6±0.8 80.1±1.780.1±1.7 76.5±2.876.5±2.8 TypiClust 92.4±0.992.4±0.9 91.5±1.091.5±1.0 98.4±0.398.4±0.3 98.5±0.798.5±0.7 73.7±0.473.7±0.4 60.5±3.460.5±3.4 83.1±1.583.1±1.5 77.6±0.677.6±0.6 UHerding 92.6±0.192.6±0.1 91.9±0.191.9±0.1 98.7±0.498.7±0.4 99.0±0.099.0±0.0 74.6±0.974.6±0.9 62.3±1.562.3±1.5 78.8±3.878.8±3.8 76.1±2.876.1±2.8 Uncertainty 93.3±0.193.3±0.1 92.7±0.192.7±0.1 99.0±0.499.0±0.4 98.9±0.198.9±0.1 74.5±1.174.5±1.1 61.5±1.361.5±1.3 79.9±1.679.9±1.6 76.8±2.776.8±2.7 0.B.2 Window Half-Width Sensitivity The main findings use a threshold uncertainty window of ±5± 5 percentage points. To test whether ALDA is sensitive to this design choice, we repeat the selection procedure using several window widths. Table˜4 evaluates Δτ∈1,2.5,3,5,7.5,10 τ∈\1,2.5,3,5,7.5,10\ percentage points at three target levels: the canonical threshold used in the main findings, τ−10τ-10 percentage points, and τ−20τ-20 percentage points. For a fixed τ, BabsB_abs is unchanged; only the deployment window W changes. As expected, W increases as the uncertainty interval widens. Importantly, the selected method is stable across window sizes for each fixed target level, indicating that ALDA recommendations are not an artifact of choosing Δτ=5 τ=5 percentage points. Table 4: Window width (δ) sensitivity at three threshold levels. Each subtable fixes τ and varies Δτ τ. Top: canonical τ per dataset. Middle: τ−10τ-10 p. Bottom: τ−20τ-20 p. For a fixed τ, BabsB_abs is constant across rows; only W grows as the uncertainty window widens. Green: canonical δ=5δ=5 p row. Consistent selections across rows and subtables indicate that ALDA recommendations are stable to window size and threshold level. δ Fetal Planes BRISC2025 ISIC2019 BUSI Selected BabsB_abs W Selected BabsB_abs W Selected BabsB_abs W Selected BabsB_abs W canonical τ τ=85%τ=85\% τ=95%τ=95\% τ=70%τ=70\% τ=75%τ=75\% ±1.0± 1.0 p Margin 436 83 BADGE 936 264 UHerding 1,762 157 Margin 227 48 ±2.5± 2.5 p Margin 436 211 BADGE 936 742 UHerding 1,762 394 Margin 227 121 ±3.0± 3.0 p Margin 436 257 BADGE 936 957 UHerding 1,762 473 Margin 227 146 ±5.0± 5.0 p 436 459 936 2,404 1,762 794 227 256 ±7.5± 7.5 p Margin 436 800 BADGE 936 2,512 UHerding 1,762 1,207 Margin 227 422 ±10.0± 10.0 p Margin 436 1,369 BADGE 936 2,593 UHerding 1,762 1,642 Margin 227 645 τ−10τ-10 p τ=75%τ=75\% τ=85%τ=85\% τ=60%τ=60\% τ=65%τ=65\% ±1.0± 1.0 p Margin 196 27 Uncertainty 305 51 UHerding 1,096 113 Entropy 76 13 ±2.5± 2.5 p Margin 196 68 Uncertainty 305 130 UHerding 1,096 284 Entropy 76 35 ±3.0± 3.0 p Margin 196 82 Uncertainty 305 158 UHerding 1,096 341 Entropy 76 46 ±5.0± 5.0 p 196 141 305 280 1,096 571 76 77 ±7.5± 7.5 p Margin 196 225 Uncertainty 305 479 UHerding 1,096 862 Entropy 76 83 ±10.0± 10.0 p Margin 196 329 Uncertainty 305 799 UHerding 1,096 1,163 Entropy 76 88 τ−20τ-20 p τ=65%τ=65\% τ=75%τ=75\% τ=50%τ=50\% τ=55%τ=55\% ±1.0± 1.0 p Margin 108 11 Uncertainty 151 18 TypiClust 350 90 UHerding 23 3 ±2.5± 2.5 p Margin 108 28 Uncertainty 151 46 TypiClust 350 228 UHerding 23 7 ±3.0± 3.0 p Margin 108 34 Uncertainty 151 55 TypiClust 350 274 UHerding 23 8 ±5.0± 5.0 p 108 58 151 94 350 469 23 17 ±7.5± 7.5 p Margin 108 90 Uncertainty 151 149 TypiClust 350 737 UHerding 23 34 ±10.0± 10.0 p Margin 108 127 Uncertainty 151 215 TypiClust 350 1,050 UHerding 23 67 0.B.3 Clinical Threshold Sensitivity We also evaluate how the ALDA recommendation changes as the clinical target τ varies. Table˜5 fixes the window width to Δτ=5 τ=5 percentage points and re-runs ALDA over a range of target thresholds. This analysis addresses a practical deployment question: whether the selected AL strategy is stable under different clinical operating points. The results show that no single AL method dominates across all thresholds and datasets. At lower thresholds, methods emphasizing representativeness or coverage (BADGE, TypiClust, Random) are often selected because early gains and broad coverage are sufficient to reach the target. At higher thresholds, the recommendation may shift toward uncertainty methods, which become more useful once the model has learned enough structure for uncertainty estimates to be informative. These transitions are expected and reflect genuine differences in the learning dynamics of AL strategies rather than the instability of ALDA. Overall, the sensitivity analyses support the main claim of the paper: ALDA does not assume a universally best AL method. It adapts the deployment recommendation to the dataset, target threshold, and tolerated threshold uncertainty. Table 5: Threshold sensitivity. ALDA is re-run at varying clinical targets τ with δ=5δ=5 p fixed, using 30% pilot fits. Each cell shows the selected method and annotation cost BabsB_abs. Green: canonical threshold for each dataset. Transitions between rows reveal at which target ALDA changes its deployment recommendation. Fetal Planes (τ0=85% _0=85\%) BRISC2025 (τ0=95% _0=95\%) ISIC2019 (τ0=70% _0=70\%) BUSI (τ0=75% _0=75\%) τ Selected BabsB_abs τ Selected BabsB_abs τ Selected BabsB_abs τ Selected BabsB_abs 30% BADGE 0 30% BADGE 0 30% Entropy 0 30% Random 0 35% BADGE 0 35% BADGE 0 35% Entropy 0 35% Random 0 40% CoreSet 0 40% BADGE 0 40% TypiClust 64 40% Random 0 45% CoreSet 0 45% BADGE 0 45% TypiClust 172 45% Random 0 50% BADGE 29 50% BADGE 0 50% TypiClust 350 50% Random 12 55% Random 59 55% BADGE 4 55% Margin 650 55% UHerding 23 60% Margin 84 60% BADGE 28 60% UHerding 1,096 60% UHerding 36 65% Margin 108 65% BADGE 59 65% UHerding 1,402 65% Entropy 76 70% Margin 142 70% BADGE 100 70% 1,762 70% Margin 137 75% Margin 196 75% Uncertainty 151 75% 227 80% Margin 284 80% Uncertainty 208 85% 436 85% Uncertainty 305 90% Uncertainty 689 90% Uncertainty 488 95% 936