Paper deep dive
Confidence Calibration under Ambiguous Ground Truth
Linwei Tao, Haoyang Luo, Minjing Dong, Chang Xu
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 3/26/2026, 1:41:08 AM
Summary
The paper addresses the structural failure of standard confidence calibration methods (like Temperature Scaling) when ground-truth labels are ambiguous. It demonstrates that calibrating against majority-voted labels leads to systematic miscalibration against the true annotator distribution. The authors propose a family of ambiguity-aware post-hoc calibrators (Dirichlet-Soft, MCTS, and LS-TS) that optimize against the full label distribution, significantly reducing true-label Expected Calibration Error (ECE) without requiring model retraining.
Entities (7)
Relation Signals (3)
Dirichlet-Soft → reduces → ECE
confidence 95% · Dirichlet-Soft reduces true-label ECE by 55-87% relative to Temperature Scaling
LS-TS → reduces → ECE
confidence 95% · LS-TS reduces ECE by 9-77% without any annotator data.
Temperature Scaling → exhibitsbias → Annotator Uncertainty
confidence 90% · Temperature Scaling is biased toward temperatures that underestimate annotator uncertainty
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Confidence calibration assumes a unique ground-truth label per input, yet this assumption fails wherever annotators genuinely disagree. Post-hoc calibrators fitted on majority-voted labels, the standard single-label targets used in practice, can appear well-calibrated under conventional evaluation yet remain substantially miscalibrated against the underlying annotator distribution. We show that this failure is structural: under simplifying assumptions, Temperature Scaling is biased toward temperatures that underestimate annotator uncertainty, with true-label miscalibration increasing monotonically with annotation entropy. To address this, we develop a family of ambiguity-aware post-hoc calibrators that optimise proper scoring rules against the full label distribution and require no model retraining. Our methods span progressively weaker annotation requirements: Dirichlet-Soft leverages the full annotator distribution and achieves the best overall calibration quality across settings; Monte Carlo Temperature Scaling with a single annotation per example (MCTS S=1) matches full-distribution calibration across all benchmarks, demonstrating that pre-aggregated label distributions are unnecessary; and Label-Smooth Temperature Scaling (LS-TS) operates with voted labels alone by constructing data-driven pseudo-soft targets from the model's own confidence. Experiments on four benchmarks with real multi-annotator distributions (CIFAR-10H, ChaosNLI) and clinically-informed synthetic annotations (ISIC~2019, DermaMNIST) show that Dirichlet-Soft reduces true-label ECE by 55-87% relative to Temperature Scaling, while LS-TS reduces ECE by 9-77% without any annotator data.
Tags
Links
- Source: https://arxiv.org/abs/2603.22879v1
- Canonical: https://arxiv.org/abs/2603.22879v1
Trouble viewing inline? Open PDF directly →
Full Text
112,890 characters extracted from source content.
Expand or collapse full text
1 Confidence Calibration under Ambiguous Ground Truth Linwei Tao, Haoyang Luo, Minjing Dong, and Chang Xu, Senior Member, IEEE ✦ Abstract—Confidence calibration assumes a unique ground-truth label per input, yet this assumption fails wherever annotators genuinely disagree. Post-hoc calibrators fitted on majority-voted labels, the standard single-label targets used in practice, can appear well-calibrated under conventional evaluation yet remain substantially miscalibrated against the underlying annotator distribution. We show that this failure is structural: under simplifying assumptions, Temperature Scaling is biased toward temperatures that underestimate annotator uncertainty, with true-label miscalibration increasing monotonically with annotation entropy. To address this, we develop a family of ambiguity-aware post-hoc calibrators that optimise proper scoring rules against the full label distribution and require no model retraining. Our methods span progressively weaker annotation requirements: Dirichlet-Soft leverages the full annotator distri- bution and achieves the best overall calibration quality across settings; Monte Carlo Temperature Scaling with a single annotation per example (MCTSS=1) matches full-distribution calibration across all benchmarks, demonstrating that pre-aggregated label distributions are unnecessary; and Label-Smooth Temperature Scaling (LS-TS) operates with voted labels alone by constructing data-driven pseudo-soft targets from the model’s own confidence. Experiments on four benchmarks with real multi- annotator distributions (CIFAR-10H, ChaosNLI) and clinically-informed synthetic annotations (ISIC 2019, DermaMNIST) show that Dirichlet-Soft reduces true-label ECE by 55–87% relative to Temperature Scaling, while LS-TS reduces ECE by 9–77% without any annotator data. Index Terms—Confidence calibration, ambiguous ground truth, annota- tor disagreement, temperature scaling, soft labels, proper scoring rules, label distribution learning. 1 INTRODUCTION D EEP neural networks are increasingly deployed as decision-support tools in high-stakes domains. In med- ical imaging, classifiers assist dermatologists in diagnosing skin lesions and pathologists in identifying malignant tis- sue [1], [2]; in natural language processing, models flag toxic content, assess clinical notes, and support legal document review; in autonomous driving, perception systems make real-time safety decisions under uncertainty [3]. In each of these settings the model outputs not merely a predicted class but an implicit reliability claim: the confidence score attached •Manuscript received X X, X. •L. Tao and C. Xu are with the School of Computer Science, University of Sydney, Sydney, NSW 2006, Australia. E-mail: linwei.tao@sydney.edu.au; c.xu@sydney.edu.au. •H. Luo and M. Dong are with the Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China. E-mail: luo- haoyang.lalutte@gmail.com; minjdong@cityu.edu.hk. •Corresponding author: Chang Xu (e-mail: c.xu@sydney.edu.au). to a prediction is understood by downstream systems and human operators as a direct probability that the prediction is correct. A radiologist who sees 90% confidence on a malignant-lesion classification may reduce the level of follow- up scrutiny; a content moderator acting on a 99% toxicity score may take irreversible action; an autonomous vehicle assigns braking decisions based on its detection confidence. The faithfulness of stated confidence to empirical correctness, i.e. calibration, is therefore as important to deployment safety as accuracy itself [4], [5]. This property is formalised as follows. A well-calibrated classifierf :X → ∆ K produces probability vectorsˆp(x) = f (x) such that P ˆ Y = Y | ˆp ˆ Y (X) = p = p, ∀p∈ [0, 1],(1) where ˆ Y = arg max k ˆp k (X) and∆ K denotes theK- dimensional probability simplex. Guo et al. [6] showed that modern high-accuracy networks are systematically over- confident, and proposed Temperature Scaling (TS), which divides logits by a scalarT > 1before softmax, as an effective post-hoc remedy. Subsequent work extended post- hoc calibration to Platt scaling [7], isotonic regression [8], and Dirichlet calibration [9], and studied its behaviour across architectures [10] and tasks [11]. These methods share a common design principle: they fit a correction to the model’s logits using a held-out calibration set annotated with one-hot labels. The calibration condition(1)implicitly assumes that every inputxhas a unique ground-truth labelY. This assumption is violated whenever the task is inherently ambiguous or subjective. A skin lesion may be legitimately classified as malignant or benign by different dermatologists, depending on subtle visual features that experts genuinely disagree about [1]; a natural-language premise may or may not entail a hypothesis depending on pragmatic interpretation [12]; a low-resolution object may be categorically ambiguous even to careful human observers [13]. Such disagreement is not annotation noise to be corrected but irreducible aleatoric uncertainty inherent to the task. The correct calibration target is therefore not a single labelYbut a distribution over labelsπ(· | x) ∈ ∆ K , which we call the annotator distribution, representing the probability that a randomly selected expert would assign each label to inputx. Figure 1 illustrates three concrete instances of this phenomenon across image classification, natural language inference, and medical imaging. arXiv:2603.22879v1 [cs.LG] 24 Mar 2026 2 empirical human votes 0.00.20.40.60.81.0 label distribution cat dog bird other 0.41 0.29 0.12 0.17 Premise: A hockey fight. Hypothesis: fighting on the ice 100 human labels 0.00.20.40.60.81.0 label distribution entail. neutral contrad. 0.54 0.46 0.00 clinician-informed distribution 0.00.20.40.60.81.0 label distribution MEL NV BKL other 0.73 0.14 0.08 0.05 Fig. 1: Left: a CIFAR-10H image with dispersed human votes (cat/dog/bird), illustrating perceptual ambiguity in low- resolution vision. Middle: a ChaosNLI premise–hypothesis pair with split entailment/neutral judgments, illustrating semantic ambiguity. Right: an ISIC 2019 melanoma image paired with the clinician-informed label distribution used in our medical experiments, showing clinically plausible MEL/NV confusion. CIFAR-10H and ChaosNLI distributions are empirical human label distributions; the ISIC panel uses the dermatologist confusion model described in Appendix I. Standard practice aggregates multiple annotations by majority vote to obtain a single voted labely ∗ , the one-hot label used in virtually all existing calibration work, and calibrates against it. This paper demonstrates that this is not merely a rough approximation but a systematic failure: calibrating to the voted label optimises the wrong objective, and the resulting miscalibration worsens as genuine annotator disagreement increases. A particularly revealing observation is that the most expressive voted-label calibrator (Dirichlet calibration with a fullK×Kaffine transform) consistently underperforms simple Temperature Scaling in true-label ECE; additional flexibility with the wrong target exacerbates rather than alleviates the problem. In a controlled experiment (Section 4), every standard voted-label calibrator fails to close the gap; the root cause is the calibration target, not the method’s capacity. This observation parallels the finding of Stutz et al. [14] for conformal prediction, where sets calibrated on voted labels systematically undercover the true annotator distribution. 1)Problem and theory (Sections 3–5): we formalise true- label calibration under ambiguous ground truth and prove that Temperature Scaling is biased toward unduly low temperatures, with the miscalibration gap growing monotonically with annotation entropy. 2) Methods (Section 6): we propose a family of ambiguity- aware post-hoc calibrators that use the annotator distri- bution as the calibration target and require no model re- training, including: Dirichlet-Soft (full distribution avail- able); MCTS (individual annotations available, where a single annotation per example,S=1, already matches full- distribution calibration); and LS-TS, an annotation-free variant that uses only voted labels yet substantially closes the calibration gap. 3)Evidence (Sections 4 and 7): experiments across four benchmarks spanning vision (CIFAR-10H, ISIC 2019, Der- maMNIST) and language (ChaosNLI) show that Dirichlet- Soft reducesECE true by 55–87% and LS-TS by 9–77% relative to TS, while standard calibrators can actively worsen true-label ECE. 2 RELATED WORK 2.1 Confidence Calibration Guo et al. [6] established that modern high-accuracy net- works are systematically overconfident and that Temperature Scaling (TS) is a simple and effective post-hoc remedy. The multiclass calibration literature has since proposed isotonic regression [8], Platt scaling [7], Dirichlet calibration [9], and adaptive per-instance temperature variants [15]. Minderer et al. [10] showed that Vision Transformers are inherently better calibrated than CNNs, and Bai et al. [11] revisit when TS is and is not sufficient. Kumar et al. [16] propose verified uncertainty calibration through non-parametric testing. All of these methods share a common assumption: the calibration set contains one-hot labels, and calibration means aligning confidence with correctness relative to those labels. We show that this assumption, not the method’s architectural capacity, is the limiting factor when labels are genuinely distributional. 2.2 Soft Labels, Label Smoothing, and Knowledge Dis- tillation Training-time approaches have explored the benefits of soft targets. Label smoothing [17], [18] adds a fixed uniform component to one-hot targets and has been observed to improve both accuracy and calibration; however, it applies a dataset-global constant regardless of per-instance annotator disagreement. Knowledge distillation [19] uses teacher model outputs as soft training targets, improving student generalisa- tion and calibration. Thulasidasan et al. [20] show that Mixup training substantially improves calibration by encouraging confident predictions only when training examples are well- separated. Collins et al. [21] collect CIFAR-10S, a companion to CIFAR-10H with individually elicited soft labels from every annotator, and demonstrate that training with per- annotator soft labels improves calibration over aggregated hard labels. Our work is the post-hoc calibration counterpart: we do not retrain the model, but instead re-target the calibration objective to the annotator distribution using only cached logits. 3 2.3 Annotation Disagreement and Multi-Annotator Learning Dense annotation datasets [12], [13] and multi-annotator learning methods [22], [23] have documented the ubiquity of label disagreement across domains. Aroyo and Welty [24] argue that disagreement is not noise but signal, and that discarding it by majority aggregation loses information that is critical for understanding task difficulty. Gordon et al. [25] and Plank [26] highlight systematic problems with annotation aggregation, showing that models trained on majority labels inherit structural biases against minority annotator perspectives. Baan et al. [27] show that ECE is theoretically ill-defined when annotators genuinely disagree, because no unique “correct” label exists to serve as the refer- ence; they propose instance-level uncertainty measures. We complement this critique by retainingECE true as a practical metric: it quantifies the overconfidence a user experiences when drawing a single label ̃y ∼ π(·|x)at deployment time, mirroring realistic usage, while supplementing it with Brier score and NLL, which are strictly proper scoring rules that directly penalise divergence from the full annotator distributionˆπ. Khurana et al. [28] study how annotator disagreement can inform selective abstention in NLP; unlike their approach, our framework targets all predictions and is applicable across vision and language without task-specific thresholding. 2.4 Proper Scoring Rules and Distributional Evaluation Gneiting and Raftery [29] provide the foundational theory of proper scoring rules: a loss is strictly proper if and only if it is uniquely minimised when the predicted distribution equals the true distribution. Brier score and NLL are canonical instances. This perspective motivates our choice of calibration objective: minimising cross-entropy againstˆπ(x)is strictly proper over the predicted distributionˆp(x), while minimis- ing against a one-hot label is only proper with respect to a degenerate Dirac distribution and misidentifies the optimal calibrated output wheneverˆπis non-degenerate. The proper scoring framework also clarifies why Dirichlet calibration with voted targets (Dirichlet-Hard) is not merely suboptimal but is consistent for the wrong estimand, explaining the empirical finding that it underperforms TS. 2.5 Conformal Prediction under Ambiguous Ground Truth Stutz et al. [14] demonstrate the set-prediction analogue of our finding: conformal prediction sets calibrated on voted labels systematically undercover the true annotator distribution, with coverage gaps of up to 10% on dermatology data. Their Monte Carlo conformal approach samples from the annotator distribution to compute coverage, closely paralleling our Monte Carlo ECE evaluation (S = 100draws per example). Our work addresses point-estimate calibration rather than set coverage: we produce well-calibrated scalar confidence scores, which conformal prediction does not. Calibration under covariate shift [30], [31] is orthogonal to our label-ambiguity setting and assumes a unique ground truth throughout. 2.6 Uncertainty Quantification in High-Stakes Applica- tions The deployment literature motivates the practical urgency of this problem. Kompa et al. [2] argue that medical image clas- sifiers are consistently overconfident, even after calibration, and that this overconfidence degrades clinical decision qual- ity. Ovadia et al. [4] demonstrate that popular uncertainty estimation methods (dropout, deep ensembles, variational inference) exhibit significantly degraded calibration under distribution shift, highlighting that single-point calibration benchmarks may be overly optimistic. Feng et al. [3] survey uncertainty quantification in autonomous driving, where overconfident detections directly contribute to unsafe ma- noeuvres. These findings establish the applied context for our work: in each of these domains, label ambiguity is simultaneously a property of the task and an unaddressed source of calibration error. The present paper provides the first systematic post-hoc calibration framework that directly targets this source. 3 PROBLEM FORMULATION 3.1 Setup and Notation LetY =1,...,Kbe the label space. A classifierf :X → ∆ K outputsˆp(x) = softmax(z(x))from pre-softmax logits z(x) ∈ R K . The annotator distributionπ(· | x) ∈ ∆ K denotes the underlying ambiguous label distribution for inputxand is estimated frommannotations asˆπ k (x) = 1 m P j 1[a j = k] . The voted label isy ∗ = arg max k ˆπ k (x). Standard calibration uses one-hot voted labels. For example, Temperature Scaling [6] minimises L TS (T ) =− 1 n n X i=1 log softmax(z i /T ) y ∗ i .(2) 3.2 True-Label Calibration Following Stutz et al. [14], we distinguish the voted labely ∗ from the true label ̃y ∼ π(·| x), drawn from the underlying ambiguous label distribution. Definition 1 (True-Label Calibration).fis true-label cali- brated if for allp ∈ [0, 1]:P( ̃ Y = ˆc(X) | ˆp ˆc (X) = p) = p, where ˆc(x) = arg max k ˆp k (x) and ̃ Y ∼ π(·| X). The voted-label ECEECE voted is the standard ECE againsty ∗ . The true-label ECEECE true averages ECE over 100 draws ̃y i ∼ π(· | x i )(B = 15equal-width bins). We also define the pointwise true-label calibration error of a prediction at inputxas| ˆp ˆc (x)− π ˆc (x)|, i.e. the absolute gap between the model’s top-class confidence and the annotator- distribution probability of that class. This per-example quantity is used in the theoretical analysis (Section 5). 4 MOTIVATING EXAMPLE Setup. We construct a 3-class 2D Gaussian dataset with three clusters:x ∼ N ((−3.2, 1.1), diag(0.60, 0.45))for class 0 withπ = [1, 0, 0];x ∼ N ((0, 0), diag(1.15, 0.75))for the ambiguous cluster withπ = [0, 0.70, 0.30]; andx ∼ N ((3.2,−1.1), diag(0.60, 0.45))for class 2 withπ = [0, 0, 1]. The voted label for the middle cluster is always class 1 4 because0.70 > 0.30. A 2-layer MLP (64 hidden units per layer) is trained on voted labels, and all calibrators are fitted on a held-out calibration split using the same voted labels. Results. Figure 2 shows that this mismatch is a struc- tural consequence of the voted-label target. TS reduces ECE voted from3.25%to1.11%, but increasesECE true from 6.26%to9.64%. Platt scaling behaves similarly (ECE voted = 0.68%,ECE true = 9.70%), and Histogram Binning likewise (ECE voted = 1.32%,ECE true = 9.65%). Figure 2(c) further shows that this residual error is concentrated in the ambigu- ous cluster for all three methods. TS setsT = 0.62 < 1, boosting confidence toward the voted target and thereby pushing the model further from the true ambiguous label distribution. The failure is therefore not specific to TS: standard post-hoc methods are useful forECE voted , but the voted-label target makes them ineffective for ECE true . 5 THEORY: WHY STANDARD CALIBRATION FAILS We now formalise, under simplifying assumptions, two com- plementary aspects of the failure of voted-label calibration under ambiguity: the direction of the TS bias (Proposition 1) and its scaling with label ambiguity (Proposition 2). Both propositions characterise pointwise miscalibration on individ- ual examples; they describe when and why a voted-label calibrator misfits individual ambiguous inputs, rather than providing a theorem about population-level ECE. Figure 3 provides empirical support. Proposition 1 (Direction of TS Bias). Consider a calibration set containing an ambiguous cluster in which the voted labely ∗ equals the majority class for every example, but the true annotator probability ofy ∗ isπ y ∗ (x) = q < 1. Under the assumption that the model is already reasonably accurate on this cluster (softmax(z i ) y ∗ > qfor alliin the cluster), the voted-label loss ex- erts a downward pull onTfor these examples, whereas a soft-label loss with targetqexerts an upward pull. Consequently,T ∗ TS < T ∗ soft , whereT ∗ soft denotes the optimal temperature when minimising the cross-entropy against the annotator distributionˆπ(Eq. 3): TS selects a lower temperature than ambiguity-aware calibration requires. In a setting where the model is already overconfident (softmax(z i ) y ∗ ≫ q), this manifests asT ∗ TS < 1; more generally, both optima may exceed 1, but the gapT ∗ soft − T ∗ TS > 0persists, consistent with our experimental findings (e.g.,T TS = 2.03vs. T soft = 3.18 on CIFAR-10H ResNet-50). Proof sketch. On the ambiguous cluster, all voted labels are y ∗ , so the voted-label loss(2)is minimised by pushing softmax(z i /T ) y ∗ → 1, i.e. the loss decreases asTde- creases. Hence the ambiguous examples exert a downward gradient onT. The cross-entropy against the annotator distribution with targetq < 1is instead minimised at softmax(z i /T ∗ ) y ∗ ≈ q, requiring a higherT ∗ whenever softmax(z i ) y ∗ > q. The global TS optimum balances this downward pull against the upward pull from unambiguous examples, but remains lower than the ambiguity-aware optimum. Full proof in Appendix B. Proposition 2 (True-Label Miscalibration and Annotation Entropy). LetH(x) =− P k π k (x) logπ k (x)denote the anno- tation entropy ofx. Assume (i) the model’s predicted confidence ˆp ˆc (x)is approximately independent ofH(x)(similar accuracy across ambiguity levels), and (i)π y ∗ (x)is a decreasing function ofH(x)(greater entropy implies lower majority-class probability). Under assumptions (i) and (i), the expected per-example true- label calibration error of voted-label-calibrated predictions is non- decreasing in H(x). Proof sketch. For unambiguous examples (H(x) = 0),π(·| x) is one-hot, so true-label and voted-label evaluation coincide and the per-example error is| ˆp ˆc (x)− 1|. AsH(x)increases, assumption (i) givesπ y ∗ (x) < 1, introducing a gap between the voted-label target and the true annotator probability. Be- cause TS is optimised against one-hot voted labels, it targets ˆp y ∗ (x) ≈ 1for all examples regardless of their annotation entropy; henceˆp ˆc (x) > π y ∗ (x)for ambiguous examples. The true-label calibration error is thenˆp ˆc (x) − π y ∗ (x), which by assumption (i) (fixedˆp ˆc ) increases asπ y ∗ (x)falls with H(x). 6 METHODS All methods in this paper are post-hoc: they operate on cached logits from a fixed pre-trained model and require only a calibration set equipped with annotator distributions or individual annotations. Training-time approaches such as soft-label training [17], [18], mixup [20], and multi-annotator learning [22], [23] are orthogonal and not compared here. 6.1 Ambiguity-Aware Calibration Methods Soft-label calibration objective. The natural replacement for the voted-label loss(2)is the cross-entropy against the full annotator distribution: L soft (T ) =− 1 n n X i=1 K X k=1 ˆπ k (x i ) log softmax(z i /T ) k = 1 n n X i=1 KL ˆπ(x i )∥ softmax(z i /T ) + const. (3) This retains the same single-parameter family as TS but replaces the one-hot voted label with a target that correctly reflects the annotator distribution. Whenˆπis available in closed form, minimising(3)directly yields the Soft-Label Temperature Scaling (SLTS) baseline reported in our tables. Proposition 3 (Correctness of distributional target). The per-example lossℓ(q; ˆπ) = − P k ˆπ k logq k is a strictly proper scoring rule overq ∈ ∆ K : it is uniquely minimised at q = ˆπ(x)for eachx. Consequently, within the constrained family softmax(z/T ) : T > 0,L soft identifies the temperatureT ∗ that minimisesKL(ˆπ∥ softmax(z/T ∗ )), targeting the correct distributional objective, unlikeL TS , which targets the one-hot voted label. Note thatT ∗ need not achievesoftmax(z/T ∗ ) = ˆπ exactly within this one-parameter family; the proper scoring rule property guarantees correctness of the target, not achievability within the constrained family. Monte Carlo Temperature Scaling (MCTS). In practice, pre-aggregatedlabeldistributionsˆπ(x i )maynot be available; instead, individual annotation records a i1 ,...,a iS arestoredperexample.MCTS drawsSsuch samplesa is ∼ ˆπ(x i )and minimises L MCTS (T )= − 1 nS P i,s log softmax(z i /T ) a is .Since 5 Feature x 1 Feature x 2 (a) Data Class 0: = [1, 0, 0] Middle 70%: label 1 Middle 30%: label 2 Class 2: = [0, 0, 1] Uncal.TSPlattHB 0 2 4 6 8 10 ECE (%) 3.25 1.11 0.68 1.32 6.26 9.64 9.70 9.65 (b) ECE summary ECE voted ECE true Ambiguous examples Clear examples 0 5 10 15 20 25 30 35 ECE_true (%) 28.7 0.7 29.0 0.5 29.5 0.7 (c) Stratified ECE TS Platt HistBin Fig. 2: Motivating example: all standard calibration methods fail. (a) Toy dataset generation: only the middle Gaussian cluster is ambiguous, withπ = [0, 0.70, 0.30]; orange points are drawn as label 1 (70%) and red points as label 2 (30%), while all receive the same voted label 1. (b) Summary of the toy results: all three voted-label calibrators (TS, Platt, Histogram Binning) lowerECE voted but increaseECE true , so voted-label evaluation masks the failure. (c) StratifiedECE true for TS, Platt, and Histogram Binning: ambiguous examples are those from the middle Gaussian cluster (where annotators disagree); clear examples are those from the two unambiguous clusters (class 0 and class 2). The residual error is concentrated in ambiguous examples for all three methods. 0.00.20.40.60.8 Normalised annotation entropy H(x)/log K 0 5 10 15 20 25 Pointwise calibration error (%) CIFAR-10H ResNet-50 CIFAR-10H ViT-B/16 ChaosNLI DeBERTa-v3 Fig. 3: Empirical validation of Proposition 2. Test exam- ples are grouped into equal-frequency bins by normalised annotation entropyH(x)/ logK. Each point shows the mean pointwise true-label calibration error| ˆp ˆc (x)− π ˆc (x)| for Temperature Scaling (error bars:±1s.e.). Across all three dataset–architecture combinations, TS error increases monotonically with annotation entropy, as predicted by Proposition 2. ChaosNLI operates in a higher-entropy regime and exhibits uniformly higher error (19–25%). E ˆy∼ˆπ [CE(softmax(z/T ), ˆy)] = KL(ˆπ∥ softmax(z/T )) + H(ˆπ), MCTS converges to the deterministic objective(3)as S →∞. The key practical finding is that evenS = 1suffices: on CIFAR-10H, a single annotation per calibration example achieves ECE true of1.53%, matching theS→∞limit within rounding error (Table 1). This efficiency generalises across all eight dataset–architecture settings in Tables 2 and 3, where MCTSS=1matches SLTS within 0.6 p ECE (see TABLE 1: MCTS convergence on CIFAR-10H (ResNet-50). Mean±std over 5 seeds;S= MC annotation samples per calibration example. MethodECE (%)T ∗ TS (voted labels)4.292.030 MCTS S = 11.53± 0.01 3.174± 0.041 MCTS S = 51.53± 0.02 3.166± 0.014 MCTS S = 201.52± 0.00 3.187± 0.013 MCTS S = 501.52± 0.01 3.185± 0.007 MCTS S = 2001.52± 0.00 3.178± 0.003 SLTS (S →∞)1.513.180 Appendix D for the full convergence analysis). Vector Scaling (VS). VS extends the soft-label objective (3) by replacing the global temperature with a per-class temperature vectorT = (T 1 ,...,T K ), so that the calibrated logit for class kbecomesz k /T k . This accommodates datasets where some classes are systematically more ambiguous than others, at the cost ofKparameters rather than one. The loss remains the KL divergence against ˆπ (Eq. 3). Distributional Isotonic Regression (IR-Soft). IR-Soft fits a monotone step function from the model’s predicted con- fidenceˆp ˆc (x)to the mean annotator probability for the top class,ˆπ ˆc (x), using the Pool Adjacent Violators Algorithm (PAVA) [8]. As a non-parametric method, it makes no assumptions about the functional form of the calibration map and is capable of correcting arbitrary monotone distortions. SoftPlatt. SoftPlatt applies a diagonal affine transformation ˆq k = softmax(w k z k + b k ), the same parametric family as Platt scaling [7], but fits the parameters by minimising KL(ˆπ(x)∥ˆq(x))against the annotator distribution. This de- sign isolates the effect of the distributional target from any architectural difference relative to standard Platt scaling. Dirichlet-Soft. Dirichlet-Soft applies the fullK×Kaffine trans- formationˆq(x) = softmax(Wz(x) + b)and fitsW ∈ R K×K , b∈ R K by minimisingKL(ˆπ(x)∥ˆq(x))with ODIR regulari- 6 sation (λ = 10 −3 ) [9]. This constitutes the most expressive parametric calibrator in our framework, withK 2 + Kfree parameters, and represents the natural ambiguity-aware counterpart of the voted-label Dirichlet calibration of Kull et al. [9]. 6.2 Annotation-Free Calibration The methods above require annotator distributionsˆπ(x i )at calibration time. When only voted labelsy ∗ i are available, we propose Label-Smooth Temperature Scaling (LS-TS), which constructs a pseudo-target distribution from the model’s own predictions without any additional annotations. Let ̄ε = 1 n P n i=1 1− ˆp y ∗ i (x i ) be the mean complement of the model’s voted-class confidence on the calibration set. Define the pseudo-target distribution ̃π LS i = (1− ̄ε)e y ∗ i + ̄ε K 1,(4) and minimiseL soft (Eq. 3) with these targets. The global smoothing weight ̄εis a data-driven estimate of average annotator disagreement: if the model is 80% confident on the voted class, the remaining 20% is spread uniformly across all Kclasses. The result isT ∗ LS > T ∗ TS whenever ̄ε > 0, moving the calibrator in the correct direction even without annotator data. Theoretical interpretation. The smoothing weight ̄εsatisfies a moment-matching fixed point: the expected pseudo-label mass on the voted class equals the average model confidence, E[ ̃π LS y ∗ ] = E[ ˆp y ∗ ]. Algorithmically, LS-TS can be viewed as a single E-step of an EM algorithm in which the latent annotator distribution is estimated from the current (pre- calibration) model, followed by one M-step that optimises Tgiven those pseudo-labels. This connection to knowledge distillation [19] explains the self-referential nature of the approach: the uncalibrated model both supplies the pseudo- targets and is then calibrated against them. Section 7.7 compares LS-TS against three simpler smoothing strategies and confirms that the data-driven global ̄εis the key design choice. Appendix L further shows that even an adaptive voted- label calibrator (one that learns a per-instance temperature from logit-derived features) fails to improve over TS, rein- forcing that the target distribution, not the flexibility of the calibration map, is the limiting factor. 7 EXPERIMENTS The experimental evaluation addresses three questions: (1) Does the voted-label target, rather than model capacity, limit calibration quality under ambiguity? (2) How much annotation information is needed to enter the ambiguity- aware regime? (3) Are the improvements robust to dataset characteristics, model architectures, and random variation? We evaluate on four benchmarks with multi-annotator data, each with two backbone architectures: CIFAR-10H (ResNet- 50, ViT-B/16), ChaosNLI (RoBERTa-Large, DeBERTa-v3), ISIC 2019 (EfficientNet-B4, ViT-S/16), and DermaMNIST (ResNet-18, ViT-S/16). Baselines. We compare against (i) Uncalibrated: raw softmax; (i) TS: Temperature Scaling on voted labels [6]; (i) ATS (Adaptive Temperature Scaling [15]): per-instance temper- ature predicted from four logit-derived features (max k z k , prediction entropy, top-2 margin, top-1 confidence) via a linear model, trained with the same voted-label NLL as TS (the most direct adaptive extension of TS without annotator data); (iv) Platt (PS): per-class weight and bias on logits, calibrated against voted labels [7], [9]; (v) Dirichlet-Hard: Dirichlet calibration [9] with the fullK× Kweight matrix and bias, calibrated against voted labels. Dirichlet-Hard uses the same architecture as our Dirichlet-Soft but with voted- label targets; the contrast isolates the effect of the calibration target from model capacity. ATS similarly tests whether richer architecture, without changing the target, is sufficient to fix the calibration gap. Metrics. We report two families of metrics, which differ in how the ground-truth label is treated: Binning-based ECE metrics (ECE true , aECE, cwECE) are estimated by drawingS = 100labels ̃y (s) i ∼ ˆπ(x i ) per example and averaging calibration error over draws.ECE true usesB = 15equal-width confidence bins; aECE (adaptive ECE [32]) uses equal-mass bins that adapt to the confidence distribution; cwECE [9] averages per-class ECE overK classes. These mirror the evaluation protocol faced by a practitioner who observes one label per prediction at test time. Soft-target metrics (Brier scoreBr, NLL) are computed directly against the soft target ˆπ: Br i =∥ ˆp(x i )− ˆπ(x i )∥ 2 ,(5) NLL i =− X k ˆπ k (x i ) log ˆp k (x i ) = H ˆπ(x i ), ˆp(x i ) , averaged over the test set. Brier and NLL are strictly proper scoring rules that directly penalise divergence from the full annotator distribution, complementing the binning-based ECE metrics. We abbreviateECE true , aECE, cwECE, Br, NLL through- out. 7.1 CIFAR-10H: Image Classification with Human Label Noise Dataset and models. CIFAR-10H [13] provides≈51human annotations per image for the 10 000-image CIFAR-10 test set, directly exposing perceptual ambiguity in low-resolution natural images through repeated human labeling. The test set is split into calibration (n = 5,000) and evaluation (5,000) by stratified sampling (seed 42). We evaluate two architectures: ResNet-50 [33] pretrained on ImageNet-1k, fine-tuned for 30 epochs (AdamW, cosine annealing), achieving 97.3% top-1 accuracy. ViT-B/16 [34] pretrained on ImageNet-21k, fine-tuned with the same protocol, achieving 98.1% top-1 accuracy. Results. Table 2 (upper half) reports results for both CIFAR- 10H backbones. All voted-label baselines leave 4–5% ECE true , whereas ambiguity-aware methods reduce it to below 2%. Two comparisons are particularly informative. First, ATS (adaptive per-instance temperature) achieves4.40%ECE on ResNet-50 and4.54%on ViT-B/16, no better than global TS (4.29%and4.48%, respectively) and far behind even the annotation-free LS-TS (1.57%and2.37%), confirming that per-instance temperature adaptation cannot compen- sate for the wrong calibration target. Second, MCTSS=1 matches SLTS on both architectures (1.45%vs.1.51%on 7 TABLE 2: CIFAR-10H and ChaosNLI results. ECE/aECE/cwECE averaged over 100 sampled labels ̃y ∼ ˆπ; Brier/NLL computed against soft targetˆπ(Eq. 5). Best per architecture in bold (Oracle TS excluded). Oracle TS fits TS on the test set with soft labels (cheating upper bound on soft NLL; not a strict ECE upper bound). ResNet-50ViT-B/16 MethodTECE↓ aECE↓ cwECE↓Br↓NLL↓ TECE↓ aECE↓ cwECE↓Br↓NLL↓ CIFAR-10H (m≈51 annotations per image, K=10) Voted-label baselines Uncalibrated—4.975.711.090.1200.692—4.995.361.060.1110.678 TS2.034.294.250.910.1120.3632.044.484.400.930.1060.346 ATS—4.404.370.940.1130.350—4.544.470.940.1060.345 Platt—4.294.230.910.1120.372—4.544.380.930.1050.363 Dirichlet-Hard—4.464.440.950.1140.395—4.704.620.970.1070.394 Ambiguity-aware (ours) SLTS3.181.511.390.450.1100.2933.070.850.560.390.1020.278 MCTS S=13.141.451.440.450.1110.2963.070.810.650.420.1010.277 SoftPlatt—1.521.310.350.1100.288—0.880.470.240.1010.272 VS—1.351.260.390.1090.289—0.920.500.320.1010.274 IR-Soft—0.720.830.450.1150.340—0.910.610.430.1050.321 Dirichlet-Soft—1.251.060.350.1090.271—0.720.410.250.1000.255 Annotation-free (ours) LS-TS3.081.571.310.460.1090.2932.762.372.210.530.1030.285 Oracle (soft labels on test set) Oracle TS3.171.501.520.450.1110.2963.090.700.630.420.1010.277 RoBERTa-LargeDeBERTa-v3 MethodTECE↓ aECE↓ cwECE↓Br↓NLL↓ TECE↓ aECE↓ cwECE↓Br↓NLL↓ ChaosNLI (SNLI+MNLI, m=100 annotations per example, K=3) Voted-label baselines Uncalibrated—27.7927.7718.610.6501.384—35.9735.9224.020.7432.148 TS2.42 10.5510.387.300.5370.8863.81 11.6311.578.480.5490.900 ATS2.28 11.3611.297.730.5380.8833.57 13.0913.079.060.5520.900 Platt—11.1610.927.280.5300.875—12.4312.368.430.5390.896 Dirichlet-Hard—11.5511.407.640.5350.889—12.6912.638.840.5390.895 Ambiguity-aware (ours) SLTS3.413.222.594.620.5200.8625.283.453.366.530.5290.877 MCTS S=13.492.822.354.600.5190.8625.453.202.986.300.5290.877 SoftPlatt—2.662.322.430.5090.839—3.172.923.180.5110.841 VS—3.463.075.130.5180.857—3.913.356.460.5250.869 IR-Soft—2.652.596.240.5490.934—2.153.967.200.5540.942 Dirichlet-Soft—2.571.712.030.5100.839—3.192.712.950.5090.836 Annotation-free (ours) LS-TS4.404.163.725.610.5220.8736.152.652.996.320.5290.881 Oracle (soft labels on test set) Oracle TS3.343.482.834.640.5200.8625.213.663.546.660.5290.876 ResNet-50;0.81%vs.0.85%on ViT-B/16), demonstrating that individual annotation records are as informative as pre-aggregated distributions for the purpose of temperature calibration. Among the more expressive calibrators, Dirichlet- Soft achieves the best Brier score and NLL on both backbones (0.109/0.271 on ResNet-50; 0.100/0.255 on ViT-B/16). LS-TS, operating with voted labels alone, already reduces ECE from 4.29% to 1.57% (ResNet-50) and from 4.48% to 2.37% (ViT- B/16), demonstrating that even approximate soft targets yield substantial improvements. 7.2 ChaosNLI: Natural Language Inference We next evaluate on ChaosNLI [12]: 100 human annota- tions per example for the combined SNLI+MNLI subset (n = 3,113), where disagreement reflects genuine semantic ambiguity rather than annotation noise. We split the data into calibration and test sets (50/50, stratified by majority- vote label to preserve class balance). We evaluate RoBERTa- Large [35] and DeBERTa-v3-base [36], both fine-tuned on MNLI training data; ChaosNLI draws from the SNLI and MNLI dev/test splits, so no training examples are re-used at evaluation. 8 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Gap +0.33 Uncal ECE = 4.6% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy TS ECE = 3.8% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SLTS ECE = 1.2% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Dir.-Soft ECE = 1.1% Fig. 4: Reliability diagrams for four representative methods: CIFAR-10H ResNet-50 (ECE true ). Red shading indicates overconfidence; green indicates underconfidence. Uncal and TS remain overconfident; soft-label temperature methods (MCTS/SLTS) substantially correct this; Dirichlet-Soft reduces the residual gap further. Reliability diagrams for all remaining methods are provided in Appendix H. 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Gap +0.34 Uncal ECE = 27.8% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy TS ECE = 10.4% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SLTS ECE = 2.1% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Dir.-Soft ECE = 1.7% Fig. 5: Reliability diagrams for four representative methods: ChaosNLI RoBERTa-Large (ECE true ). Same method selection as Figure 4. Red shading indicates overconfidence; green indicates underconfidence. NLI models are severely overconfident before calibration; ambiguity-aware methods substantially correct this. Results. Table 2 (lower half) reports ChaosNLI results. The NLI setting amplifies the calibration gap observed on CIFAR- 10H: voted-label baselines leave 10–12% ECE, and ATS (11.36%on RoBERTa-L,13.09%on DeBERTa-v3) performs worse than global TS on DeBERTa-v3, further corroborating that adaptive capacity without the correct target is ineffective. MCTSS=1again closely matches SLTS (2.82%vs.3.22% on RoBERTa-L;3.20%vs.3.45%on DeBERTa-v3). On this benchmark MCTSS=1in fact slightly outperforms SLTS, which may be attributable to the stochastic single-annotation objective acting as an implicit regulariser. The best distri- butional fit is achieved by Dirichlet-Soft and SoftPlatt: on RoBERTa-L, SoftPlatt achieves the best Brier score (0.509 vs. 0.510 for Dirichlet-Soft) and both share the same NLL (0.839); on DeBERTa-v3, Dirichlet-Soft achieves the best Brier (0.509) and NLL (0.836). LS-TS reduces ECE from 10.55% to 4.16% (RoBERTa-L) and from 11.63% to 2.65% (DeBERTa-v3) using only voted labels, substantially outperforming all hard-label baselines. ATS vs. LS-TS: the calibration target is the decisive factor. A controlled comparison between ATS (increased architectural capacity, voted-label target) and LS-TS (global temperature, soft target) shows that ATS fails to improve over TS in all eight settings and in several cases degrades performance (ECE increases from11.63%to13.09%on DeBERTa-v3; from17.04%to17.53%on ISIC ViT-S/16). The per-instance temperature learned from logit features adapts to model uncertainty, which is only weakly correlated with annotator disagreement. In contrast, LS-TS reduces ECE by 9–77% across all eight settings without any annotator data (see Appendix L for the full table and analysis). 7.3 ISIC 2019: Skin Disease Diagnosis We evaluate on ISIC 2019 [37], [38] (8 skin conditions, ≈25,000images), a clinically realistic differential-diagnosis task in which several lesion categories have overlapping visual morphology. We usem = 9synthetic dermatologist annotations per image, generated by sampling from a clini- cally calibrated8× 8confusion matrix following Dawid and Skene [39] (Appendix I), with diagonal agreement matched to Liu et al. [40]: mean75%. Because no per-image reader study is publicly available for this dataset, annotation ambiguity is modelled class-conditionally: each image’s confusion depends on its consensus class, not on its individual visual content, which is a deliberate simplification of true instance-level annotator variation. We evaluate EfficientNet-B4 [41] and ViT-S/16 [34] on a stratified 70/15/15 split. ISIC 2019 results. Table 3 (upper half) shows that the benefits of ambiguity-aware calibration extend to clinically realistic settings with synthetic annotations. Dirichlet-Soft achieves 9 TABLE 3: ISIC 2019 and DermaMNIST results. ECE/aECE/cwECE averaged over 100 sampled labels ̃y ∼ ˆπ; Brier/NLL computed against soft targetˆπ(Eq. 5). Best per architecture in bold (Oracle TS excluded). Oracle TS fits TS on the test set with soft labels (cheating upper bound on soft NLL; not a strict ECE upper bound). EfficientNet-B4ViT-S/16 MethodTECE↓ aECE↓ cwECE↓Br↓NLL↓ TECE↓ aECE↓ cwECE↓Br↓NLL↓ ISIC 2019 (m=9 synthetic annotators, K=8, mean agreement 75%) Voted-label baselines Uncalibrated—25.8725.976.640.6002.686—21.2821.176.210.6501.803 TS1.77 18.7218.705.020.5591.6611.23 17.0416.955.490.6301.578 ATS1.70 18.8318.805.030.5601.7331.12 17.5317.485.460.6311.730 Platt—19.1619.085.200.5591.658—18.1718.144.940.6101.595 Dirichlet-Hard—20.0319.965.250.5601.677—18.5118.495.020.6131.617 Ambiguity-aware (ours) SLTS4.429.769.913.030.5291.1492.687.106.773.350.5941.248 MCTS S=14.51 10.0910.153.080.5301.1492.777.477.153.350.5951.248 SoftPlatt—9.989.992.330.5231.111—7.827.711.900.5721.201 VS—9.689.642.480.5241.123—7.747.592.140.5741.211 IR-Soft—1.772.214.360.5391.288—2.052.135.650.6271.479 Dirichlet-Soft—8.358.392.080.5181.086—6.326.281.640.5681.184 Annotation-free (ours) LS-TS4.309.349.462.980.5281.1493.98 15.4915.474.520.6191.303 Oracle (soft labels on test set) Oracle TS4.389.649.773.000.5281.1492.707.236.863.340.5941.248 ResNet-18ViT-S/16 MethodTECE↓ aECE↓ cwECE↓Br↓NLL↓ TECE↓ aECE↓ cwECE↓Br↓NLL↓ DermaMNIST (m=5 synthetic annotators, K=7, mean agreement 64.7%) Voted-label baselines Uncalibrated—34.5634.5110.110.7712.617—37.5537.4810.880.7913.700 TS1.84 22.2522.206.910.6871.6372.55 24.6524.577.300.6831.664 ATS1.86 22.8222.756.970.6871.6062.53 25.3525.247.530.6861.650 Platt—23.1023.036.930.6821.623—24.1824.027.130.6751.670 Dirichlet-Hard—23.0222.797.060.6831.704—24.5524.467.330.6801.877 Ambiguity-aware (ours) SLTS3.435.134.764.020.6291.3885.033.292.743.150.6091.346 MCTS S=13.425.074.744.020.6291.3885.043.282.713.160.6091.346 SoftPlatt—4.053.801.510.6141.313—3.413.051.360.5991.287 VS—4.584.483.380.6231.365—3.813.352.730.6091.329 IR-Soft—2.201.944.960.6401.468—2.061.874.570.6261.423 Dirichlet-Soft—3.943.931.260.6121.305—3.183.001.160.5981.279 Annotation-free (ours) LS-TS3.035.054.823.790.6301.3944.197.367.063.680.6151.362 Oracle (soft labels on test set) Oracle TS3.384.874.493.970.6281.3884.943.512.973.130.6091.346 the best Brier and NLL on both backbones (0.518/1.086 on ENet-B4; 0.568/1.184 on ViT-S/16), while IR-Soft achieves the lowest ECE, a pattern consistent with its non-parametric flex- ibility. MCTSS=1closely tracks SLTS on ENet-B4 (10.09% vs.9.76%) and ViT-S/16 (7.47%vs.7.10%), replicating the pattern observed on natural-annotation benchmarks. LS-TS approximately halves the ECE of TS on ENet-B4 (18.72%→ 9.34%) using only voted labels; the improvement is smaller on ViT-S/16 (17.04% → 15.49%), where model confidence is a weaker proxy for annotation ambiguity. Annotator confusion model. Figure 6 shows the clinically cal- ibrated8× 8confusion matrix used to generate synthetic an- notations for ISIC 2019. The MEL/NV pair (melanoma/nevi, orange dashed) is the most consequential and most confused: diagonal rates of 73%/76% with 14%/15% cross-confusion, reflecting the well-documented difficulty of distinguishing early melanoma from benign nevi [1]. The AK/BKL/SCC cluster (purple dotted) forms a second high-confusion group (diagonal 65–67%), capturing clinically significant overlap between actinic keratosis and squamous cell carcinoma. The mean diagonal agreement ̄ C i = 75%matches the inter- reader agreement reported by Liu et al. [40]. Full matrix 10 MELNVBCCAKBKLDFVLSCC Annotator's label MEL NV BCC AK BKL DF VL SCC Consensus (majority-vote) label 0.730.140.020.030.080.000.000.00 0.150.760.010.010.060.010.000.00 0.020.010.810.050.070.010.010.02 0.030.010.040.650.110.000.000.16 0.120.050.030.100.620.000.000.08 0.010.020.020.010.020.870.030.02 0.000.010.020.010.010.020.910.02 0.010.010.030.180.090.000.010.67 MEL/NV AK/BKL/SCC cluster 0.0 0.2 0.4 0.6 0.8 1.0 Probability Fig. 6: Inter-reader confusion matrix for ISIC 2019. Entry C ij is the probability that a dermatologist labels an image as classjgiven consensus labeli. Orange dashed box: MEL/NV most-confused pair (73–76%diagonal). Purple dotted: AK/BKL/SCC high-confusion cluster. Mean diagonal agreement ̄ C i = 75%. values and construction details are given in Appendix I. 7.4 DermaMNIST: Supplementary Skin Disease Experi- ment Table 3 (lower half) reports results on DermaMNIST [42] (7-class, derived from HAM10000 [37];m = 5synthetic annotators; overall agreement≈64.7%) with ResNet-18 and ViT-S/16. TS leaves ECE= 22.3%on ResNet-18; ambiguity- aware methods substantially close this gap: SLTS achieves 5.1%, IR-Soft2.2%, and Dirichlet-Soft3.9%, with Dirichlet- Soft also achieving the best cwECE (1.3%), Brier (0.612), and NLL (1.305). ViT-S/16 exhibits the same hierarchy (Dirichlet-Soft:3.2%/0.598/1.279). MCTSS=1again closely approximates SLTS (5.07%vs.5.13%on ResNet-18;3.28%vs. 3.29%on ViT-S/16). Both backbones showT soft > T TS > 1, consistent with Proposition 1. DermaMNIST is the most ambiguous benchmark in our study, with mean annotator agreement of64.7%compared to75%for ISIC 2019,≈80%for CIFAR-10H, and≈67% for ChaosNLI. Correspondingly, TS ECE true (22.3%) is the highest across all benchmarks, reflecting the widest gap between the voted-label target and the true annotator distri- bution. The Dirichlet-Hard calibrator, despite being the most expressive voted-label method (fullK×Kaffine), worsens ECE relative to TS on ResNet-18 (23.0%vs.22.3%) and provides no meaningful improvement on ViT-S/16 (24.6% vs.24.7%), again confirming that expressive architecture cannot compensate for a miscalibrated target. LS-TS reduces ECE by77%relative to TS on ResNet-18 (22.3%→ 5.1%), the largest relative improvement of the annotation-free method across all settings, because the high level of ambiguity means model confidence (complement 35.3%) correlates well with the fraction of annotators choosing the minority class. 7.5 Cross-Benchmark Analysis Aggregating the results across all four benchmarks and eight architecture configurations, four consistent findings emerge that together characterise the role of the calibration target under label ambiguity. (1) Target, not capacity, is the bottleneck. Dirichlet-Hard, the most expressive voted-label calibrator, consistently un- derperforms simple TS in both ECE and Brier (e.g., 4.46% vs. 4.29% on CIFAR-10H R50; 11.55% vs. 10.55% on ChaosNLI RoBERTa-L; see Tables 2 and 3). This rules out architectural capacity as the bottleneck and directly supports the claim that calibration target is limiting. (2) Ambiguity-aware methods consistently dominate. Dirichlet-Soft improves ECE by 71– 84% relative to TS on the two natural-annotation benchmarks (CIFAR-10H and ChaosNLI) and by 55–87% on the medical benchmarks (ISIC 2019 and DermaMNIST). Dirichlet-Soft consistently achieves the best overall calibration quality, making it the recommended default whenever annotator distributions are available. (3) A single annotation per example suffices. MCTSS=1matches SLTS within 0.6 p ECE in all 8 settings. The aggregated annotation distribution provides no measurable additional benefit once individual annotator labels are available: any dataset annotated once per example by a randomly selected annotator is immediately amenable to the same calibration quality as full multi- annotator annotation. (4) LS-TS is effective but dataset- dependent. LS-TS substantially reduces ECE without any annotator data in all settings except ISIC ViT-S/16 (where the LS-TS temperature exceeds the oracle soft-label temperature, T LS = 3.98 > T SLTS = 2.68, indicating that model confidence over-estimates annotation ambiguity for this architecture– dataset pair). This divergence highlights the assumption underlying LS-TS: that model uncertainty is a reliable proxy for annotation disagreement, an assumption that holds broadly but can fail when the pre-softmax logit magnitude is not well correlated with the fraction of annotators that choose the minority class. 7.6 Reliability Diagrams Figures 4 and 5 present reliability diagrams for the natural- annotation benchmarks (CIFAR-10H and ChaosNLI); Fig- ures 7 and 8 extend the comparison to the medical imaging benchmarks. The visual pattern is consistent across all four datasets: Uncalibrated and TS outputs are systematically overconfident (red-shaded residuals), with TS reducing but not eliminating the gap. Soft-label temperature methods (MCTS/SLTS) substantially correct the overconfidence, and Dirichlet-Soft closes most of the remaining residual. These diagrams provide a qualitative complement to the quantita- tive ECE metrics reported in Tables 2 and 3. Complete panels covering all thirteen methods across all eight architecture configurations are provided in Appendix H. 7.7 Ablation: LS-TS Smoothing Strategies Table 4 compares LS-TS against three simpler annotation- free smoothing strategies on CIFAR-10H and ChaosNLI. All methods use the same single-temperature family; only the pseudo-target construction differs. Fixed-LS withε=0.1consistently underperforms or even degrades below TS (e.g., ECE 7.44% vs. 3.90% on CIFAR-10H 11 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Gap +0.28 Uncal ECE = 26.1% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy TS ECE = 18.9% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SLTS ECE = 9.4% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Dir.-Soft ECE = 8.2% Fig. 7: Reliability diagrams: ISIC 2019 EfficientNet-B4 (ECE true ). Red shading = overconfidence; green = underconfidence. Hard-label baselines remain severely overconfident at 18–26% ECE; ambiguity-aware methods substantially correct this, with IR-Soft achieving the lowest ECE. 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Gap +0.45 Uncal ECE = 34.3% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy TS ECE = 21.9% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SLTS ECE = 5.3% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Dir.-Soft ECE = 3.8% Fig. 8: Reliability diagrams: DermaMNIST ResNet-18 (ECE true ). Same layout as Figure 4. TS leaves 22% ECE; Dirichlet-Soft reduces it to 3.9% while achieving the best Brier/NLL. TABLE 4: LS-TS ablation: annotation-free smoothing strategies. ECE true (%), Brier score, and NLL on CIFAR-10H and ChaosNLI. All methods use the same single-temperature family; only the pseudo-target construction differs. Fixed-LS uses ε=0.1regardless of the data; Ent-LS setsε i =H( ˆp(x i ))/ logKper instance; C-LS computes a per-class smoothing weight ε k =mean i:y ∗ i =k (1− ˆp y ∗ (x i )). Best annotation-free method per column in bold. CIFAR R50CIFAR ViTChaos RoBERTaChaos DeBERTa MethodECEBrNLL ECEBrNLLECEBrNLLECEBrNLL TS (voted)3.90.1126.3464.01.1043.32010.55 .537.88611.63 .549.900 Fixed-LS (ε=0.1)7.44.1207.3326.00.1058.3056.46.525.8667.53.536.882 Ent-LS3.16.1116.3213.69.1037.3083.17.520.8628.58.539.885 C-LS1.48.1111.2961.69.1016.2794.15.522.8732.54.529.880 LS-TS (ours)1.49.1111.2961.62.1016.2794.16.522.8732.65.529.881 ResNet-50), confirming that an arbitrary fixed smoothing weight is insufficient. Ent-LS (per-instance entropy-based smoothing) is inconsistent across settings: it collapses on DeBERTa-v3 (8.58%), suggesting prediction entropy is a noisy proxy when models are strongly confident. C-LS computes a separateε k for each classkusing the mean complement- confidence of examples with voted labelk; it matches LS-TS within 0.1 p ECE on three of four settings (CIFAR-10H R50: 1.48% vs. 1.49%; ViT: 1.69% vs. 1.62%; ChaosNLI RoBERTa: 4.15% vs. 4.16%), confirming that the global mean is a good summary of per-class behaviour. This near-equivalence is expected: because temperature scaling has only a single scalar parameterT, per-class differences inε k are averaged out during optimisation, and both C-LS and LS-TS converge to the sameT ∗ (e.g., both yieldT =3.19on CIFAR-10H R50). The slight advantage of C-LS on DeBERTa-v3 (2.54% vs. 2.65%) is within single-seed variance. 7.8 Robustness We assess robustness along three dimensions, summarised in Table 5 and Figure 9. Multi-seed stability. Standard deviations of ECE true over five random seeds are uniformly2×–10×smaller than the ECE gaps between methods, confirming that the reported rankings are not artefacts of a particular data split or anno- 12 TABLE 5: Multi-seed stability. Mean±std of ECE true (%) over 5 seeds. For CIFAR-10H/ChaosNLI the seed varies the cal/test split; for ISIC/DermaMNIST it varies the annotation sample. DatasetArchTSSLTSDir.-SoftIR-Soft CIFAR-10HR504.25± 0.25 1.61± 0.12 1.37± 0.09 0.89± 0.12 CIFAR-10HViT-B/164.39± 0.20 0.83± 0.14 0.74± 0.07 0.88± 0.16 ChaosNLIRoBERTa-L 11.54± 0.54 3.85± 0.34 2.90± 0.14 2.40± 0.18 ChaosNLIDeBERTa-v3 12.73± 0.60 4.41± 0.54 3.80± 0.35 2.61± 0.41 ISIC 2019ENet-B418.84± 0.17 9.80± 0.22 8.45± 0.20 1.79± 0.16 ISIC 2019ViT-S/1616.93± 0.16 7.30± 0.13 6.20± 0.19 2.05± 0.13 DermaMNIST R1822.43± 0.28 5.08± 0.25 3.53± 0.39 2.33± 0.17 DermaMNIST ViT-S/1624.81± 0.08 3.61± 0.28 3.11± 0.21 2.09± 0.14 01234 ECE true (%) TS SLTS Dir-Soft IR-Soft (a) Multi-seed (C10H R50, 5 seeds) 5102050100 Cal. set size (%) 1 2 3 4 (b) Cal-size (C10H R50) 5102050100 Cal. set size (%) 5 10 ECE true (%) (c) Cal-size (NLI RoBERTa-L) 02040 Annotations per example (m) 2 4 6 (d) Annot.-count (C10H R50) TS LS-TS SLTS VS Dir-Soft IR-Soft MCTS Fig. 9: Robustness of ambiguity-aware calibration. (a) Multi- seed stability (CIFAR-10H R50, 5 seeds). (b)–(c) ECE true vs. calibration set size on CIFAR-10H R50 and ChaosNLI RoBERTa-L. (d) ECE true vs. annotation count (CIFAR-10H R50). Rankings are stable across seeds, cal-set sizes, and annotation counts. Full results for all benchmarks are in Appendices G, E, and F. tation draw. Full per-architecture bar charts are provided in Appendix G. Calibration set size. Ambiguity-aware methods stabilise below 2% ECE on CIFAR-10H using as few as 5–10% of the calibration data (≈250examples), while TS remains flat regardless of sample size. Full calibration-set-size curves for all benchmarks are given in Appendix E. Annotation count. Ambiguity-aware calibration is robust oncem≥ 2annotations per image: increasing from 2 to 51 annotations reduces soft-label ECE true (MCTS/SLTS) by less than 0.3 p. TS ECE true , by contrast, increases withm, because the annotator distribution diverges further from the one-hot voted label as the underlying consensus strengthens. Full annotation-count results are reported in Appendix F. 8 DISCUSSION Trade-off between voted-label and true-label calibration. A natural question is whether optimising for true-label calibration degradesECE voted . This trade-off is expected: voted-label calibration becomes epistemically unreliable when annotators genuinely disagree, because the voted label systematically overstates the majority-class probability. In practice, this trade-off is rarely problematic, since the sce- narios whereECE voted matters (deterministic, unambiguous labels) are precisely those whereECE voted ≈ ECE true and the two objectives coincide. For deployment contexts that require strictECE voted guarantees, a two-stage approach is possible: apply ambiguity-aware calibration first, then apply standard TS on the outputs to re-optimise ECE voted . Practical guidelines. The choice of calibration method depends on what annotation information is available at cali- bration time. When the full annotator distribution is available, Dirichlet-Soft is recommended as the default: it consistently achieves the best overall calibration quality; SoftPlatt is a competitive alternative for NLI tasks, and IR-Soft should be preferred when the sole objective is minimising ECE. When individual per-example annotations are stored but have not been aggregated into distributions, MCTSS=1 provides an effective solution: a single randomly drawn annotation per calibration example yields ECE comparable to the full distribution, as demonstrated consistently across all eight settings. Finally, when only voted labels are available, LS-TS constructs a data-driven soft pseudo-target from the model’s own confidence and reduces ECE by 9–77% relative to Temperature Scaling without requiring any annotator data. Limitations. Full ambiguity-aware calibration (including Dirichlet-Soft) requires multi-annotator labels at calibration time. When only voted labels are available, LS-TS remains useful but does not fully match Dirichlet-Soft, particularly on datasets where model confidence diverges from annotation ambiguity (e.g., ISIC 2019 ViT-S/16). ISIC 2019 and DermaM- NIST rely on synthetic annotators derived from clinician- reported agreement. Crucially, this is a class-conditional model: annotation confusion depends only on the consensus class label, not on instance-level visual difficulty. Real multi-reader medical datasets with per-image reader distributions would provide stronger validation and would capture the instance- level variation our synthetic model does not; concrete candi- dates include VinDr-CXR [43] (3 independent radiologists per chest X-ray, 15k images) and CheXpert [44] (5 independent radiologist annotations on the test set), both of which provide true per-image annotator distributions directly amenable to our framework. Conclusions about absolute ECE magnitudes on these two datasets should be interpreted with this in mind; the relative ranking of methods is expected to be robust to the specific confusion matrix used, as confirmed by the multi- seed annotation-generation ablation (Appendix G). 9 CONCLUSION This paper has formalised the problem of confidence calibra- tion under ambiguous ground truth and established, both theoretically and empirically, that standard post-hoc calibra- tors fitted on voted labels are systematically miscalibrated against the underlying annotator distribution. The central insight, confirmed by four complementary ablations, is that the calibration target, not the method’s architectural capacity, is the limiting factor. Dirichlet calibration with voted-label tar- gets consistently underperforms simple Temperature Scaling; per-instance temperature adaptation trained with voted-label 13 supervision yields no improvement; yet the same global- temperature architecture, when supplied with the correct distributional target, substantially closes the gap. Our proposed methods span three annotation regimes without requiring model retraining. Dirichlet-Soft, which leverages full annotator distributions, reduces true-label ECE by 55–87% relative to Temperature Scaling and achieves the best overall calibration quality across settings. MCTSS=1 demonstrates that a single randomly drawn annotation per calibration example is sufficient to match full-distribution calibration within 0.6 p ECE across all benchmarks, a finding with immediate practical implications for annotation protocols, as it suggests that annotating each calibration example once by a different randomly assigned annotator suffices. LS-TS reduces ECE by 9–77% using only voted labels. Together, these results suggest that future calibration benchmarks and deployment standards should move beyond the voted-label paradigm and account for the distributional nature of human annotation. REFERENCES [1] H. A. Haenssle, C. Fink, R. Schneiderbauer, F. Toberer, T. Buhl, A. Blum, A. Kalloo, A. B. H. Hassen, L. Thomas, A. Enk, W. Stolz, and Reader Study Level-I and Level-I Investigators, “Man against machine: Diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in compar- ison to 58 dermatologists,” Annals of Oncology, vol. 29, no. 8, p. 1836–1842, 2018. [2]B. Kompa, J. Snoek, and A. L. Beam, “Second opinion needed: Communicating uncertainty in medical machine learning,” NPJ Digital Medicine, vol. 4, no. 1, p. 4, 2021. [3]D. Feng, A. Harakeh, S. L. Waslander, and K. Dietmayer, “A review and comparative study on probabilistic object detection in autonomous driving,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 8, p. 9961–9980, 2022. [4]Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. V. Dillon, B. Lakshminarayanan, and J. Snoek, “Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift,” in Advances in Neural Information Processing Systems, vol. 32, 2019. [5]M. Abdar, F. Pourpanah, S. Hussain, D. Rezazadegan, L. Liu, M. Ghavamzadeh, P. Fieguth, X. Cao, A. Khosravi, U. R. Acharya et al., “A review of uncertainty quantification in deep learning: Techniques, applications and challenges,” Information Fusion, vol. 76, p. 243–297, 2021. [6]C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in Proceedings of the 34th International Conference on Machine Learning (ICML). PMLR, 2017, p. 1321–1330. [7]J. C. Platt, “Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods,” in Advances in Large Margin Classifiers. MIT Press, 1999, p. 61–74. [8]B. Zadrozny and C. Elkan, “Transforming classifier scores into accurate multiclass probability estimates,” in Proceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2002, p. 694–699. [9]M. Kull, M. P. Nieto, M. Kängsepp, T. Silva Filho, H. Song, and P. Flach, “Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with Dirichlet calibration,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 32, 2019. [10]M. Minderer, J. Djolonga, R. Romijnders, F. A. Hubis, X. Zhai, N. Houlsby, D. Tran, and M. Lucic, “Revisiting the calibration of modern neural networks,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 34, 2021. [11]Y. Bai, S. Mei, H. Wang, and C. Xiong, “Don’t just blame over- parametrization for over-confidence: Theoretical analysis of calibra- tion in binary classification,” in Proceedings of the 38th International Conference on Machine Learning (ICML). PMLR, 2021, p. 566–576. [12]Y. Nie, X. Zhou, and M. Bansal, “What can we learn from collective human opinions on natural language inference data?” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, p. 9131–9143. [13] J. C. Peterson, R. M. Battleday, T. L. Griffiths, and O. Russakovsky, “Human uncertainty makes classification more robust,” in Proceed- ings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019, p. 9617–9626. [14] D. Stutz, A. G. Roy, T. Matejovicova, P. Strachan, A. T. Cemgil, and A. Doucet, “Conformal prediction under ambiguous ground truth,” Transactions on Machine Learning Research (TMLR), 2023, arXiv:2307.09302. [15]T. Joy, F. Pinto, S.-N. Lim, P. H. S. Torr, and P. K. Dokania, “Sample- dependent adaptive temperature scaling for improved calibration,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 12, 2023, p. 14 919–14 926. [16]A. Kumar, P. Liang, and T. Ma, “Verified uncertainty calibration,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 32, 2019. [17] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, p. 2818–2826. [18]R. Müller, S. Kornblith, and G. Hinton, “When does label smooth- ing help?” in Advances in Neural Information Processing Systems (NeurIPS), vol. 32, 2019. [19] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015. [20]S. Thulasidasan, G. Chennupati, J. A. Bilmes, T. Bhattacharya, and S. Michalak, “On mixup training: Improved calibration and predictive uncertainty for deep neural networks,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 32, 2019. [21] K. M. Collins, U. Bhatt, and A. Weller, “Eliciting and learning with soft labels from every annotator,” in Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, vol. 10, 2022, p. 35–47. [22] A. N. Uma, T. Fornaciari, D. Hovy, E. Pavlick, M. Poesio, and B. Plank, “Learning from disagreement: A survey,” Journal of Artificial Intelligence Research, vol. 72, p. 1385–1470, 2021. [23]F. Rodrigues and F. Pereira, “Deep learning from crowds,” in Proceedings of the 32nd AAAI Conference on Artificial Intelligence, 2018. [24]L. Aroyo and C. Welty, “Truth is a lie: Crowd truth and the seven myths of human annotation,” AI Magazine, vol. 36, no. 1, p. 15–24, 2015. [25]M. L. Gordon, M. S. Lam, J. S. Park, K. Patel, J. Hancock, T. Hashimoto, and M. S. Bernstein, “Jury learning: Integrating dissenting voices into machine learning models,” in Proceedings of the CHI Conference on Human Factors in Computing Systems, 2022. [26] B. Plank, “The “problem” of human label variation: On ground truth in data, modeling and evaluation,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022. [27] J. Baan, W. Aziz, B. Plank, and R. Fernández, “Stop measuring calibration when humans disagree,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022, p. 1892–1915. [28] U. Khurana, E. Nalisnick, A. Fokkens, and S. Swayamdipta, “Crowd-calibrator: Can annotator disagreement inform calibration in subjective tasks?” in Proceedings of the Conference on Language Modeling (COLM), 2024. [29] T. Gneiting and A. E. Raftery, “Strictly proper scoring rules, prediction, and estimation,” Journal of the American Statistical Association, vol. 102, no. 477, p. 359–378, 2007. [30]S. Park, O. Bastani, J. Weimer, and I. Lee, “Calibrated prediction with covariate shift via unsupervised domain adaptation,” in Inter- national Conference on Artificial Intelligence and Statistics (AISTATS). PMLR, 2020, p. 3219–3229. [31]X. Wang, M. Long, J. Wang, and M. Jordan, “Transferable calibration with lower bias and variance in domain adaptation,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020. [32]J. Nixon, M. W. Dusenberry, L. Zhang, G. Jerfel, and D. Tran, “Measuring calibration in deep learning,” in IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2019. [33] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, p. 770–778. [34] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: 14 Transformers for image recognition at scale,” in International Conference on Learning Representations (ICLR), 2021. [35]Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “RoBERTa: A robustly optimized BERT pretraining approach,” arXiv preprint arXiv:1907.11692, 2019. [36]P. He, J. Gao, and W. Chen, “DeBERTaV3: Improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing,” in International Conference on Learning Repre- sentations (ICLR), 2023. [37]P. Tschandl, C. Rosendahl, and H. Kittler, “The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,” Scientific Data, vol. 5, p. 180161, 2018. [38]M. Combalia, N. C. F. Codella, V. Rotemberg, B. Helba, V. Vilaplana, O. Reiter, C. Carrera, A. Barreiro, A. C. Halpern, S. Puig, and J. Malvehy, “BCN20000: Dermoscopic lesions in the wild,” arXiv preprint arXiv:1908.02288, 2019. [39]A. P. Dawid and A. M. Skene, “Maximum likelihood estimation of observer error-rates using the EM algorithm,” Applied Statistics, vol. 28, no. 1, p. 20–28, 1979. [40]Y. Liu, A. Jain, C. Eng, D. H. Way, K. Lee, P. Bui, K. Kanada, G. de Oliveira Marinho, J. Gallegos, S. Gabriele, V. Gupta, N. Singh, V. Natarajan, R. Hofmann-Wellenhof, G. S. Corrado, L. H. Peng, D. R. Webster, D. Ai, S. Huang, Y. Liu, R. C. Dunn, and D. Coz, “A deep learning system for differential diagnosis of skin diseases,” Nature Medicine, vol. 26, p. 900–908, 2020. [41]M. Tan and Q. V. Le, “EfficientNet: Rethinking model scaling for convolutional neural networks,” in Proceedings of the 36th International Conference on Machine Learning (ICML), 2019, p. 6105– 6114. [42]J. Yang, R. Shi, D. Wei, Z. Liu, L. Zhao, B. Ke, H. Pfister, and B. Ni, “MedMNIST v2: A large-scale lightweight benchmark for 2D and 3D biomedical image classification,” Scientific Data, vol. 10, no. 1, p. 41, 2023. [43]H. Q. Nguyen, K. Lam, L. T. Le, H. H. Pham, D. Q. Tran, D. B. Nguyen, D. D. Le, C. M. Pham, H. T. T. Tong, D. H. Dinh et al., “VinDr-CXR: An open dataset of chest X-rays with radiologist’s annotations,” Scientific Data, vol. 9, no. 1, p. 429, 2022. [44]J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciosi, C. Chute, B. Kim, M. P. Lungren, A. Y. Ng, and P. Rajpurkar, “CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, 2019, p. 590–597. Linwei Tao received the B.Eng. degree in com- munication engineering from Huazhong Univer- sity of Science and Technology, China, in 2017, the M.Sc. degree in data science from the Univer- sity of Sydney, Australia, in 2021, and the M.Phil. degree in computer vision from the University of Sydney in 2023. He is currently pursuing the Ph.D. degree in computer vision at the University of Sydney, supervised by A/Prof. Chang Xu. He is also a Research Scientist Intern at Google Research Australia. His research interests in- clude confidence calibration and uncertainty estimation in deep learning and large vision–language models. He has published papers at top- tier venues including ICML, CVPR, ICLR, and AAAI. He serves as a reviewer for NeurIPS, ICML, ICLR, CVPR, AAAI, and IJCAI, and as a journal reviewer for IEEE Transactions on Multimedia, Transactions on Machine Learning Research, Data Mining and Knowledge Discovery, and IEEE Transactions on Pattern Analysis and Machine Intelligence. He is a Student Committee Member of AAAI 2026. Haoyang Luo received the B.S. degree (2021) and M.S. degree (2024) in computer science from the Harbin Institute of Technology, Shenzhen, China. He is currently working toward the Ph.D. degree in computer science with the City Uni- versity of Hong Kong, supervised by Dr. Minjing Dong. His research interests lie in multi-modal trustworthy learning, uncertainty quantification, and model calibration. He has published his research in top-tier conferences and journals, including NeurIPS, ICML, and IEEE Transactions on Knowledge and Data Engineering. Minjing Dong received the B.S. degree in soft- ware engineering from the Dalian University of Technology, the B.S. degree in information tech- nology from the University of Sydney, the M.Phil. degree in engineering and information technology from the University of Sydney, and the Ph.D. degree in engineering and information technology from the University of Sydney. He is currently an Assistant Professor with the Department of Com- puter Science, City University of Hong Kong. His research interests include adversarial robustness, model calibration, efficient neural networks, human motion analytics, and generative models. He has published more than 30 papers in top- tier conferences and journals, including NeurIPS, ICML, CVPR, AAAI, ICLR, IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE Transactions on Neural Networks and Learning Systems, IEEE Transactions on Image Processing, and IEEE Transactions on Multimedia. He received the AAAI 2023 Distinguished Paper Award. He serves as Area Chair and PC Member of NeurIPS, ICML, CVPR, ICCV, and ECCV. Chang Xu (Senior Member, IEEE) received the Ph.D. degree from Peking University, China. He is an ARC Future Fellow and Associate Professor with the School of Computer Science, University of Sydney. He received the University of Sydney Vice-Chancellor’s Award for Outstanding Early Career Research. His research interests lie in machine learning algorithms and applications in computer vision. He has published more than 100 papers in prestigious journals and top-tier conferences. He has received several paper awards, including the Distinguished Paper Award at AAAI 2023 and the Distinguished Paper Award at IJCAI 2018. He served as Area Chair of NeurIPS, ICML, ICLR, KDD, CVPR, and M, as well as Senior PC Member of AAAI and IJCAI. He served as Associate Editor of IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE Transactions on Multimedia, and Transactions on Machine Learning Research. He has been named a Top Ten Distinguished Senior PC Member at IJCAI 2017 and an Outstanding Associate Editor of IEEE Transactions on Multimedia in 2022. He is the corresponding author of this paper. 15 APPENDIX A DATASET BACKGROUND AND SOURCES OF LABEL AMBIGUITY All four benchmarks in this paper are appropriate for studying calibration under ambiguous ground truth, but they exhibit ambiguity in slightly different ways. CIFAR-10H. Peterson et al. [13] introduced CIFAR-10H by collecting a full distribution of human labels for each image in the CIFAR-10 test set, explicitly to capture human perceptual uncertainty. The ambiguous examples are typically low- resolution images whose visual evidence supports multiple plausible classes for human observers. In this benchmark, the label distribution is therefore directly observed from repeated human annotation, making it a clean testbed for ambiguity-aware calibration. ChaosNLI. Nie et al. [12] introduced ChaosNLI to study collective human opinions on natural language inference. Each example receives 100 human labels, and the paper reports that substantial disagreement is common rather than excep- tional. This ambiguity is semantic rather than perceptual: differences in pragmatic assumptions, underspecification, and sentence interpretation lead reasonable annotators to assign different NLI labels to the same premise–hypothesis pair. ISIC 2019. ISIC 2019 is a multiclass dermoscopic lesion diagnosis benchmark assembled from several clinical sources, including HAM10000 [37] and BCN20000 [38]. The task itself is ambiguity-prone because several lesion categories share overlapping visual morphology, and the challenge was designed to reflect the more realistic task of differential diagnosis rather than a simpler benign-versus-malignant de- cision. Because the public challenge release does not provide per-image reader distributions, we model ambiguity using synthetic annotators calibrated to dermatologist confusion patterns and reader agreement reported in Liu et al. [40]. This preserves the core property relevant to our study: in dermatology, a single image can support multiple clinically plausible labels. DermaMNIST. DermaMNIST [42] is a standardized28× 28version of the dermatology benchmark derived from HAM10000 [37]. HAM10000 consists of dermatoscopic im- ages of common pigmented lesions collected in clinical prac- tice, where diagnosis already involves fine-grained distinc- tions among visually similar categories such as melanoma, nevi, and benign keratoses. DermaMNIST inherits this medical ambiguity from HAM10000 and, because it is aggressively downsampled for lightweight benchmarking, removes additional visual detail that can further increase label uncertainty. As with ISIC 2019, we therefore evaluate calibration against an ambiguity-aware label distribution rather than treating the majority diagnosis as uniquely correct. APPENDIX B PROOF OF PROPOSITION 1 LetD amb = (x i ,y ∗ i ) : y ∗ i = c, x i ∈ C be the subset of calibration examples from the ambiguous clusterC, where the voted label is always class c. Denote p i = softmax(z i ) c . TheTSlossonD amb isL TS (T ;D amb )= −|D amb | −1 P i log softmax(z i /T ) c .AsT →0 + , softmax(z i /T ) c → 1(assumingz ic > z ik for allk ̸= c), so L TS → 0. Hence∂L TS /∂T > 0for allT > 0: decreasingT always decreases the voted-label loss on the ambiguous cluster. The globally optimalT ∗ TS balances this downward pull from ambiguous examples against the upward pull from unambiguous examples, butT ∗ TS ≤ T ∗ no-amb (the optimal temperature without the ambiguous cluster): including ambiguous examples drives the optimum downward. Thecross-entropyagainsttheannotator distributiononD amb isL soft (T ;D amb )= −|D amb | −1 P i P k ˆπ k (x i ) log softmax(z i /T ) k ,with ˆπ c (x i ) = q < 1. The minimum-lossT ∗ soft satisfies softmax(z i /T ∗ soft ) c ≈ q, requiringT ∗ soft > T ∗ TS whenever p i > q. HenceT ∗ TS < T ∗ soft regardless of whether either optimum exceeds 1. In the special case where the model is already overconfident on the ambiguous cluster (p i ≫ q), this gap is large enough thatT ∗ TS < 1; in practice both optima exceed 1 (since the model is also underconfident on unambiguous examples), but the inequalityT ∗ TS < T ∗ soft is observed consistently across all experimental settings.□ APPENDIX C PROOF OF PROPOSITION 2 We formalise the argument given in the proof sketch in the main text. Setup. Letp = ˆp ˆc (x)denote the model’s predicted con- fidence for the top classˆcand letq = π ˆc (x) ∈ (0, 1] denote the true annotator probability for that class. We assume the model is trained against voted labels, sopis approximately constant across examples with the same voted label regardless of their annotation entropy H(x). Voted-label calibration error contribution. For examplex i in confidence binB b (wherep(B b ) ≈ p), the per-example voted-label calibration error is|p− 1[ˆc = y ∗ ]|. Sinceˆc = y ∗ for correctly-classified ambiguous examples (the model predicts the majority class), this contribution is approximately |p− 1|for examples where the model makes the right hard prediction, independent of H(x). True-label calibration error contribution. The per-example true-label calibration error is|p− π ˆc (x)| =|p− q|. Excess true-label error relative to voted-label evaluation. The per-example excess contribution is: δ(x) =|p− q|−|p− 1[ˆc = y ∗ ]|.(6) When the model is correctly calibrated to voted labels (p≈ 1[ˆc = y ∗ ]in expectation), the voted contribution is near zero: |p− 1|≈ 0for high-accuracy bins and|p− 0| = pfor error bins. For unambiguous examples (H(x) = 0),q = π y ∗ = 1, so δ(x) =|p− 1|−|p− 1| = 0. Monotonicity. AsH(x)increases,q = π y ∗ (x)decreases below 1 (more probability mass shifts to non-majority classes). Specifically, for aK-class problem with uniform annotator distribution at maximum entropy,q = 1/K. The true-label contribution|p− q|increases strictly asqdecreases from 1 (sincep > qfor a well-trained model predicting the majority class with confidence exceeding the majority probability). The voted-label contribution|p− 1[ˆc = y ∗ ]|is unaffected byH(x). Thereforeδ(x) = |p − q| − |p − 1[ˆc = y ∗ ]|is non-decreasing inH(x), and strictly increasing whenever p > q > 0. 16 Aggregate monotonicity. The aggregate excess true- label error relative to voted-label evaluation equals P b |B b | n P i∈B b δ(x i )/|B b | . Sinceδ(x i )is non-decreasing in H(x i )and bins contain a mixture of examples, this discrep- ancy grows with the mean annotation entropy of the test set. □ Remark 1 (Independence assumption). Assumption (i) (that ˆp ˆc (x)is approximately independent ofH(x)) may be violated in practice if harder examples (highH(x)) also have lower model confidence. When the two are positively correlated, the per-example discrepancy termδ(x)grows even faster withH(x)than the proof suggests, so Proposition 2 remains valid and the monotonicity is if anything conservative. APPENDIX D MONTE CARLO TEMPERATURE SCALING (MCTS) Method. When individual annotation records are available rather than pre-aggregated label distributions, MCTS draws S samples a is ∼ ˆπ(x i ) per example and minimises L MCTS (T ) =− 1 nS n X i=1 S X s=1 log softmax(z i /T ) a is . MCTS subsumes the deterministic soft-label objective (SLTS, Eq. 3 withS→∞) as a special case, but applies even when only raw annotation samples are stored rather than pre- computed frequency vectors. A single annotation suffices. The key practical implication is that evenS = 1(a single randomly drawn annotation per calibration example) already achieves ECE true of 1.53%, matching the deterministic limit (S→∞) to within rounding error and reducing TS’s 4.29% by64%on CIFAR-10H ResNet- 50. This one-annotation efficiency generalises across all benchmarks: across the 8 settings in Tables 2 and 3, MCTS S=1matches SLTS within 0.6 p ECE in every case. On the natural-annotation benchmarks (CIFAR-10H, ChaosNLI), MCTSS=1occasionally outperforms the deterministic limit slightly (e.g., 2.82% vs. 3.22% ECE on ChaosNLI RoBERTa- L), consistent with the stochastic single-annotation objective providing mild implicit regularisation. This means that any dataset where each example has been annotated even once by a randomly selected annotator is immediately amenable to strong ambiguity-aware calibration; pre-aggregating an- notations into a distribution provides no measurable addi- tional benefit. Formally, sinceE ˆy∼ˆπ [CE(softmax(z/T ), ˆy)] = KL(ˆπ∥ softmax(z/T ))+H(ˆπ), the MCTS objective converges to Eq.(3)in expectation asS → ∞, with variance∝ 1/S. Table 1 (main text) confirms the fast per-seed convergence on CIFAR-10H ResNet-50: the ECE gap betweenS = 1and S →∞is only0.02p; in practiceS = 20–50is sufficient to reduce variance to negligible levels. APPENDIX E ABLATION: CALIBRATION SET SIZE We fix the annotation count at the full observed value and vary the fraction of calibration examples used, from 5% to 100% (stratified subsampling by class). Results are shown for all four benchmarks and eight architectures. Key findings. Across all four benchmarks and eight archi- tectures, TS’s ECE does not decrease with more calibration data, confirming that the degradation is a target bias rather than a variance artifact. In contrast, soft-label methods (MCTS/SLTS) achieve stable low ECE from as few as 5–10% of the calibration set, and IR-Soft and Dirichlet-Soft converge even faster on most benchmarks. This suggests that the anno- tation distribution (not the number of calibration examples) is the limiting factor for ambiguity-aware methods. APPENDIX F ABLATION: NUMBER OF ANNOTATIONS PER IMAGE We study how performance depends on the number of annotationsmper calibration image. We fix the calibration set size and the annotation generation seed, and sweepm from1to50, resampling the soft targets for each value ofm. Results are shown for CIFAR-10H, which provides up to≈51 real annotations per image; we subsample from the full pool. Key findings. Ambiguity-aware calibration is robust to annotation count oncem ≥ 2: SLTS, Dirichlet-Soft, and IR-Soft achieve near-minimal ECE with only a handful of annotations per image and do not improve substantially beyondm = 5. LS-TS converges quickly (bym≈ 4) because the global smoothing parameterεstabilises once each image has at least a few annotations. Standard Temperature Scaling shows the opposite trend: its ECE true increases withmbecause a higher temperature is required to match the more diffuse annotation distribution, exposing the fundamental target mismatch. APPENDIX G STATISTICAL SIGNIFICANCE: MULTI-SEED ANALYSIS For CIFAR-10H and ChaosNLI we repeat the evaluation with five random seeds (42–46), each producing a different strat- ified 50/50 cal/test split. For ISIC 2019 and DermaMNIST the val/test split is predetermined, so we instead vary the annotation generation seed (42–46), testing sensitivity to the particular synthetic annotation sample. Table 5 (main text, Robustness section) reports mean±std of ECE true across seeds for all eight (dataset, arch) combinations. Across all benchmarks and architectures, standard devi- ations are small relative to the ECE gaps: TS vs. ambiguity- aware methods differ by2–10×the standard deviation, confirming statistical robustness of the results in Tables 2–3. Bar charts for all architectures are shown below. APPENDIX H RELIABILITY DIAGRAMS Four-method summaries for CIFAR-10H (ResNet-50) and ChaosNLI (RoBERTa-Large) are shown in Figures 4 and 5 in the main text. Here we provide the four-method summary panels and full panels for all remaining architectures and datasets. All panels share the same layout: top row = voted-label baselines (Uncal, TS, Platt, Dir.-Hard); middle row = annotation-free (LS-TS) and temperature-based soft methods (SLTS, MCTS, SoftPlatt, VS); bottom row = non- parametric/matrix soft methods (Dir.-Soft, IR-Soft). Red shading = overconfident; green = underconfident. 17 51020305075100 Calibration set size (%) 1.0 1.5 2.0 2.5 3.0 3.5 4.0 ECE_true (%) TS LS-TS SLTS VS Dirichlet-Soft IR-Soft 51020305075100 Calibration set size (%) 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 ECE_true (%) TS SLTS VS Dirichlet-Soft IR-Soft Fig. 10: Calibration-set-size ablation: CIFAR-10H. Left: ResNet-50. Right: ViT-B/16. TS’s ECE is flat across all calibration set sizes; ambiguity-aware methods converge within 5–10%. 51020305075100 Calibration set size (%) 2 4 6 8 10 12 ECE_true (%) TS LS-TS SLTS VS Dirichlet-Soft IR-Soft 51020305075100 Calibration set size (%) 2 4 6 8 10 12 14 ECE_true (%) TS LS-TS SLTS VS Dirichlet-Soft IR-Soft Fig. 11: Calibration-set-size ablation: ChaosNLI. Left: RoBERTa-Large. Right: DeBERTa-v3-base. NLI models show the same pattern: TS is flat near 10–13% while soft methods stabilise near 3–4%. The diagrams confirm the quantitative results in Tables 2– 3: TS reduces overconfidence relative to Uncal but leaves a large residual gap because it targets voted labels. Dirichlet- Hard performs similarly to TS and sometimes worse, con- firming that the problem lies in the target, not model capacity. All soft methods substantially close the gap, with IR-Soft and Dir.-Soft achieving the most uniform residuals across all benchmarks. APPENDIX I ISIC 2019 ANNOTATOR CONFUSION MATRIX I.1 Construction Because the full per-image reader annotations of Liu et al. [40] are not publicly available, we construct a synthetic annotator model using a clinically calibrated8×8confusion matrixC, where entryC ij gives the probability that a board- certified dermatologist labels an image as classjwhen the consensus (majority-vote) diagnosis is classi. This confusion- matrix annotator model follows the framework of Dawid and Skene [39], in which each annotator’s behaviour is charac- terised by a class-conditional label-flipping distribution; it is widely used in crowdsourcing and multi-annotator learning [22], [23]. Calibration sources. Diagonal entries (per-condition agree- ment rates) are set to match the per-condition inter-reader agreement reported in Liu et al. [40] and corroborated by Haenssle et al. [1]. Off-diagonal entries encode clinically established confusion patterns: •MEL/NV (Melanoma / Melanocytic Nevi): the most consequential and most confused pair in dermoscopy, with diagonal rates of 73%/76%. Off-diagonal entries C MEL,NV = 0.14andC NV,MEL = 0.15reflect the well- documented difficulty of distinguishing early melanoma from benign nevi [1]. •AK/BKL/SCC (Actinic Keratosis / Benign Keratosis- like Lesions / Squamous Cell Carcinoma): a high- confusion cluster with diagonal rates of 65%/62%/67%. The AK↔SCC confusion (C AK,SCC = 0.16,C SCC,AK = 0.18 ) is clinically significant because untreated AK can progress to SCC. BKL shares morphological features with both conditions. • DF/VL (Dermatofibroma / Vascular Lesion): the easiest to distinguish, with diagonal rates of 87%/91%. The mean diagonal agreement is ̄ C i = 75%, matching the aggregate inter-reader agreement reported by Liu et al. [40] exactly. Simulation procedure. For each test imagex i with consensus labely i , we independently drawm = 9synthetic annotations from the categorical distribution defined by row y i of C : a (k) i ∼ Categorical(C y i ,· ), k = 1,...,m, and set the empirical label distribution toπ(x i ) = 1 m P m k=1 e a (k) i . This directly mirrors them = 9board- 18 51020305075100 Calibration set size (%) 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 ECE_true (%) TS LS-TS SLTS VS Dirichlet-Soft IR-Soft 51020305075100 Calibration set size (%) 2 4 6 8 10 12 14 16 18 ECE_true (%) TS LS-TS SLTS VS Dirichlet-Soft IR-Soft Fig. 12: Calibration-set-size ablation: ISIC 2019. Left: EfficientNet-B4. Right: ViT-S/16. IR-Soft achieves the lowest ECE across all calibration set sizes on both architectures. 51020305075100 Calibration set size (%) 5 10 15 20 ECE_true (%) TS LS-TS SLTS VS Dirichlet-Soft IR-Soft 51020305075100 Calibration set size (%) 5 10 15 20 25 30 ECE_true (%) TS LS-TS SLTS VS Dirichlet-Soft IR-Soft Fig. 13: Calibration-set-size ablation: DermaMNIST. Left: ResNet-18. Right: ViT-S/16. Ambiguity-aware methods converge quickly; TS remains flat regardless of calibration set size. certified dermatologist annotation protocol of Liu et al. [40], whose reader study we cannot fully replicate due to data unavailability. The resulting mean annotation entropy over the ISIC 2019 test set is ̄ H = 0.66, reflecting the genuine diagnostic difficulty of the 8-class task. Remark 2 (Class-conditional limitation). This synthetic an- notator model is deliberately class-conditional: every image with the same consensus classy i is assigned the same rowC y i ,· of the confusion matrix, regardless of its individual visual content. In practice, images within the same consensus class will differ in visual ambiguity, leading to instance-level variation in annotator disagreement that this model does not capture. Our experiments therefore evaluate calibration under a controlled form of clinically- informed ambiguity rather than true per-instance annotator distributions. I.2 Visualisation Figure 33 shows the confusion matrix as a heatmap. The MEL/NV cluster (orange dashed box) and the AK/BKL/SCC high-confusion cluster (purple dotted highlights) are visually prominent. DF and VL have near-perfect diagonal entries and negligible off-diagonal mass. I.3 Full confusion matrix Table 6 lists the exact numerical entries of the confusion matrix. TABLE 6: Numerical values of the inter-reader confusion matrix for ISIC 2019. Row = consensus (majority-vote) label; column = annotator’s label. Each row sums to 1. Diagonal entries (bold) are per-condition agreement rates, calibrated to Liu et al. [40] Table 2; mean diagonal agreement ̄ C i = 75% . MEL NV BCCAKBKLDFVLSCC MEL0.730.140.020.03 0.08 0.00 0.00 0.00 NV0.150.760.010.01 0.06 0.01 0.00 0.00 BCC0.020.010.810.05 0.07 0.01 0.01 0.02 AK0.030.010.040.65 0.11 0.00 0.00 0.16 BKL0.120.050.030.10 0.62 0.00 0.00 0.08 DF0.010.020.020.01 0.02 0.87 0.03 0.02 VL0.000.010.020.01 0.01 0.02 0.91 0.02 SCC0.010.010.030.18 0.09 0.00 0.01 0.67 APPENDIX J EMPIRICAL VALIDATION OF PROPOSITION 2 Proposition 2 states that, under simplifying assumptions, the pointwise true-label calibration error of a voted-label- calibrated model is non-decreasing in the annotation entropy H(x). Figure 3 provides empirical support: we bin test ex- amples by their normalised annotation entropyH(x)/ logK and plot the mean absolute error| ˆp ˆc (x)−π ˆc (x)|for TS across three dataset–architecture combinations. For CIFAR-10H (both architectures), TS error rises from ≈2%in the near-unambiguous bin (H/ logK ≈ 0.02) to 19 01020304050 Annotations per calibration example (m) 1 2 3 4 5 6 7 ECE true (%) Annotation-count ablation CIFAR-10H (resnet50) Uncalibrated TS LS-TS SLTS MCTS VS SoftPlatt Dirichlet-Soft HB-Soft IR-Soft 01020304050 Annotations per calibration example (m) 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 ECE_true (%) Annotation-count ablation on CIFAR-10H TS SLTS MCTS VS SoftPlatt Dirichlet-Soft HB-Soft IR-Soft Fig. 14: Annotation-count ablation: CIFAR-10H (both architectures). ECE true vs. annotations per imagemfor ResNet-50 (left) and ViT-B/16 (right). Ambiguity-aware methods are stable form≥ 2; TS rises withmdue to target mismatch; LS-TS converges by m≈ 4. TSSLTSDirichlet-SoftIR-Soft 0 1 2 3 4 ECE_true (%) TSSLTSDirichlet-SoftIR-Soft 0 1 2 3 4 ECE_true (%) Fig. 15: Multi-seed bar charts: CIFAR-10H. Left: ResNet-50. Right: ViT-B/16. TSSLTSDirichlet-SoftIR-Soft 0 2 4 6 8 10 12 ECE_true (%) TSSLTSDirichlet-SoftIR-Soft 0 2 4 6 8 10 12 ECE_true (%) Fig. 16: Multi-seed bar charts: ChaosNLI. Left: RoBERTa-Large. Right: DeBERTa-v3-base. ≈18%in the high-ambiguity bin (H/ logK ≈ 0.48), a clear monotone increase consistent with the proposition. For ChaosNLI (DeBERTa-v3), which operates in a higher-entropy regime (H/ logK ∈ [0.2, 0.9]), TS error is uniformly high (19–25%) and rises further at the highest-entropy bins. The overall trend is consistent across both domains and all three architectures: voted-label calibration error increases with annotation entropy, as predicted by Proposition 2. See Figure 3 in the main text (Theory section) for the plot. APPENDIX K LS-TS ABLATION: SMOOTHING STRATEGY COM- PARISON (EXTENDED) We compare LS-TS against three simpler annotation-free smoothing baselines that share the same single-temperature family but differ in how the pseudo-target ̃π LS i is constructed: •Fixed-LS (ε=0.1): fixed data-independent smoothing weight, the standard label-smoothing value used in training [17]. •Ent-LS: per-instanceε i = H( ˆp(x i ))/ logK, using nor- malised prediction entropy as a proxy for annotator disagreement. 20 TSSLTSDirichlet-SoftIR-Soft 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 ECE_true (%) TSSLTSDirichlet-SoftIR-Soft 0 2 4 6 8 10 12 14 16 ECE_true (%) Fig. 17: Multi-seed bar charts: ISIC 2019. Left: EfficientNet-B4. Right: ViT-S/16. Seed varies the annotation generation; val/test split is fixed. TSSLTSDirichlet-SoftIR-Soft 0 5 10 15 20 ECE_true (%) TSSLTSDirichlet-SoftIR-Soft 0 5 10 15 20 25 ECE_true (%) Fig. 18: Multi-seed bar charts: DermaMNIST. Left: ResNet-18. Right: ViT-S/16. Seed varies the annotation generation; val/test split is fixed. 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Gap +0.44 Uncal ECE = 4.9% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy TS ECE = 4.2% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SLTS ECE = 0.5% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Dir.-Soft ECE = 0.2% Fig. 19: Reliability diagrams (summary): CIFAR-10H ViT-B/16. • C-LS: per-classε k = mean i:y ∗ i =k (1 − ˆp y ∗ (x i )), ac- commodating class-level variation in model confidence (same idea as Vector Scaling applied to the smoothing weight). All methods are evaluated on CIFAR-10H and ChaosNLI, the two benchmarks with real multi-annotator distributions, using the same experimental protocol as the main paper (seed 42, 50/50 stratified split). Table 4 (main text, Methods section) reports the full comparison. Fixed-LS withε=0.1consistently underperforms or even degrades below TS (e.g., ECE 7.44% vs. 4.29% on CIFAR-10H ResNet-50), confirming that an arbitrary fixed smoothing weight is insufficient; the data-driven estimate of ̄εis critical. Ent-LS (per-instance entropy) improves over TS on most settings but is inconsistent: it works well on ChaosNLI RoBERTa-L (3.17%) but collapses on DeBERTa-v3 (8.58%), suggesting that prediction entropy is a noisy proxy for annotator ambiguity when models are already strongly confident. C-LS (per-classε) achieves results very close to LS-TS on all four settings (within 0.1 p ECE), confirming that the global mean is a good summary of per-class behaviour for these benchmarks; the slight advantage of C-LS on DeBERTa-v3 (2.54% vs. 2.65%) is within the single-seed variance reported in Appendix G. 21 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Voted-label baselines Accuracy Gap +0.33 Uncal ECE = 4.6% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy TS ECE = 3.8% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Platt (PS) ECE = 4.0% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Dir.-Hard ECE = 4.1% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy HB-Hard ECE = 6.8% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Annotation-free / Soft (TS-based) Accuracy LS-TS ECE = 1.2% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SLTS ECE = 1.2% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy MCTS ECE = 1.2% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SoftPlatt ECE = 1.2% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy VS ECE = 1.3% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Soft (non-parametric / matrix) Accuracy Dir.-Soft ECE = 1.1% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy HB-Soft ECE = 4.6% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy IR-Soft ECE = 0.7% Fig. 20: Reliability diagrams (all methods): CIFAR-10H ResNet-50 (ECE true ). APPENDIX L ADAPTIVE TEMPERATURE SCALING WITH VOTED LABELS (EXTENDED ANALYSIS) A natural question is whether making the calibration map adaptive (predicting a per-instance temperatureT (x i ) rather than a globalT ∗ ) can close the gap between voted- label calibration and ambiguity-aware calibration, with- out requiring annotator data. We implement ATS [15]: a linear layer maps four logit-derived featuresφ(z i ) = [max k z ik , H(softmax(z i ))/ logK, p (1) −p (2) , p (1) ]tologT i viaT i = softplus(w ⊤ φ(z i )+b)+0.1, trained with voted-label NLL identical to TS. The four features capture prediction scale, uncertainty, margin, and confidence. We applyℓ 2 regularisation (λ = 10 −3 ) to avoid overfitting on small calibration sets. Finding. ATS fails to improve over TS in all 8 settings and in several cases degrades it (ECE rises from11.63%to13.09% on DeBERTa-v3; from17.04%to17.53%on ISIC ViT-S/16). The per-instance temperature learned from logit features adapts to model uncertainty, not to annotator disagreement; these two quantities are loosely correlated at best (cf. Figure 3). In contrast, LS-TS, which uses the same global-temperature architecture as TS but simply shifts the calibration target toward a smoothed distribution, reduces ECE by 9–77% across all 8 settings without any annotator data. This confirms that the calibration target is the critical design axis: added architectural flexibility without the right supervision signal yields no benefit. APPENDIX M IMPLEMENTATION DETAILS CIFAR-10H models. ResNet-50: ImageNet-1k weights (ResNet50_Weights.IMAGENET1K_V1); FC layer re- placed with2048 × 10linear; AdamW, lr= 10 −4 , weight decay10 −4 , batch 128, cosine annealing, 30 epochs,224×224. ViT-B/16: ImageNet-21k weights; classification head replaced; AdamW, lr= 5× 10 −5 , weight decay10 −4 , batch 64, cosine annealing, 20 epochs, 224× 224. ChaosNLI models. RoBERTa-Large:roberta-large-mnli fromHuggingFace,pre-fine-tunedonMultiNLI [35];noadditionalfine-tuning.DeBERTa-v3-base: cross-encoder/nli-deberta-v3-basefrom HuggingFace [36]; no additional fine-tuning. Both models use the combined ChaosNLI SNLI+MNLI subset (n = 3,113), split 50/50 into calibration and test (stratified, seed 42). ISIC 2019 models. EfficientNet-B4: ImageNet-1k weights; stratified 70/15/15 split; class-weighted cross-entropy (NV: ≈67%of training images); AdamW, lr= 3× 10 −4 , weight decay10 −4 , 2-epoch linear warm-up + cosine decay, 20 epochs,380× 380. ViT-S/16: ImageNet-21k weights; same split and loss weighting; AdamW, lr= 10 −4 , weight decay 10 −4 , 2-epoch warm-up + cosine decay, 20 epochs,224× 224. DermaMNIST models. ResNet-18: ImageNet-1k weights, class-weighted cross-entropy, AdamW, lr= 10 −4 , 30 epochs. ViT-S/16: ImageNet-21k weights, same training protocol, 224× 224. Calibration optimisers. TS, SLTS: LBFGS, lr= 0.1, 500 iterations, tolerance10 −9 (gradient),10 −11 (loss). Platt/VS: Adam, lr= 0.01/0.05, weight decay 10 −4 , 2000 steps. 22 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Voted-label baselines Accuracy Gap +0.44 Uncal ECE = 4.9% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy TS ECE = 4.2% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Platt (PS) ECE = 4.1% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Dir.-Hard ECE = 4.2% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy HB-Hard ECE = 6.2% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Annotation-free / Soft (TS-based) Accuracy LS-TS ECE = 1.8% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SLTS ECE = 0.5% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy MCTS ECE = 0.4% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SoftPlatt ECE = 0.3% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy VS ECE = 0.4% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Soft (non-parametric / matrix) Accuracy Dir.-Soft ECE = 0.2% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy HB-Soft ECE = 3.1% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy IR-Soft ECE = 0.3% Fig. 21: Reliability diagrams (all methods): CIFAR-10H ViT-B/16 (ECE true ). 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Gap +0.38 Uncal ECE = 36.0% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy TS ECE = 11.7% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SLTS ECE = 2.8% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Dir.-Soft ECE = 2.7% Fig. 22: Reliability diagrams (summary): ChaosNLI DeBERTa-v3-base. ECE bins.B = 15equal-width bins in[0, 1]; bins with fewer than 1 example are excluded. Compute. CIFAR-10H fine-tuning:≈20min on one NVIDIA A100. All calibration methods:< 60s on CPU after logit caching. 23 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Voted-label baselines Accuracy Gap +0.34 Uncal ECE = 27.8% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy TS ECE = 10.4% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Platt (PS) ECE = 10.7% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Dir.-Hard ECE = 11.4% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy HB-Hard ECE = 8.2% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Annotation-free / Soft (TS-based) Accuracy LS-TS ECE = 3.7% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SLTS ECE = 2.1% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy MCTS ECE = 2.0% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SoftPlatt ECE = 1.5% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy VS ECE = 2.4% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Soft (non-parametric / matrix) Accuracy Dir.-Soft ECE = 1.7% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy HB-Soft ECE = 1.0% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy IR-Soft ECE = 0.7% Fig. 23: Reliability diagrams (all methods): ChaosNLI RoBERTa-Large (ECE true ). NLI models are severely overconfident before calibration; ambiguity-aware methods substantially correct this. TABLE 7: ATS vs. TS and LS-TS across all 8 settings (annotation-free methods only). ECE true (%) and Brier. ATS predicts a per-instance temperature from logit features but is trained with the same voted-label NLL as TS. Bold = best annotation-free method per column. C10H R50 C10H ViT NLI RoBERTa NLI DeBERTa ISIC ENet ISIC ViT Derm R18 Derm ViT Method ECEBr ECEBrECEBrECEBrECEBrECEBrECEBrECEBr TS4.29 .1124.48 .106 10.55 .537 11.63 .549 18.72 .559 17.04 .630 22.25 .687 24.65 .683 ATS4.40 .1134.54 .106 11.36 .538 13.09 .552 18.83 .560 17.53 .631 22.82 .687 25.35 .686 LS-TS1.57 .1092.37 .1034.16 .5222.65 .5299.34 .528 15.49 .6195.05 .6307.36 .615 24 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Voted-label baselines Accuracy Gap +0.38 Uncal ECE = 36.0% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy TS ECE = 11.7% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Platt (PS) ECE = 12.4% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Dir.-Hard ECE = 12.8% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy HB-Hard ECE = 10.7% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Annotation-free / Soft (TS-based) Accuracy LS-TS ECE = 1.4% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SLTS ECE = 2.8% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy MCTS ECE = 2.8% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SoftPlatt ECE = 2.8% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy VS ECE = 4.8% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Soft (non-parametric / matrix) Accuracy Dir.-Soft ECE = 2.7% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy HB-Soft ECE = 0.7% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy IR-Soft ECE = 0.7% Fig. 24: Reliability diagrams (all methods): ChaosNLI DeBERTa-v3-base (ECE true ). 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Gap +0.28 Uncal ECE = 26.1% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy TS ECE = 18.9% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SLTS ECE = 9.4% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Dir.-Soft ECE = 8.2% Fig. 25: Reliability diagrams (summary): ISIC 2019 EfficientNet-B4. 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Gap +0.28 Uncal ECE = 21.2% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy TS ECE = 17.0% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SLTS ECE = 7.1% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Dir.-Soft ECE = 5.6% Fig. 26: Reliability diagrams (summary): ISIC 2019 ViT-S/16. 25 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Voted-label baselines Accuracy Gap +0.28 Uncal ECE = 26.1% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy TS ECE = 18.9% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Platt (PS) ECE = 19.4% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Dir.-Hard ECE = 20.2% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy HB-Hard ECE = 20.3% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Annotation-free / Soft (TS-based) Accuracy LS-TS ECE = 9.0% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SLTS ECE = 9.4% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy MCTS ECE = 9.5% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SoftPlatt ECE = 9.8% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy VS ECE = 9.6% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Soft (non-parametric / matrix) Accuracy Dir.-Soft ECE = 8.2% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy HB-Soft ECE = 1.2% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy IR-Soft ECE = 1.3% Fig. 27: Reliability diagrams (all methods): ISIC 2019 EfficientNet-B4 (ECE true ). 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Voted-label baselines Accuracy Gap +0.28 Uncal ECE = 21.2% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy TS ECE = 17.0% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Platt (PS) ECE = 18.4% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Dir.-Hard ECE = 18.7% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy HB-Hard ECE = 18.3% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Annotation-free / Soft (TS-based) Accuracy LS-TS ECE = 15.5% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SLTS ECE = 7.1% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy MCTS ECE = 7.1% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SoftPlatt ECE = 7.4% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy VS ECE = 7.6% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Soft (non-parametric / matrix) Accuracy Dir.-Soft ECE = 5.6% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy HB-Soft ECE = 2.1% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy IR-Soft ECE = 1.6% Fig. 28: Reliability diagrams (all methods): ISIC 2019 ViT-S/16 (ECE true ). 26 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Gap +0.45 Uncal ECE = 34.3% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy TS ECE = 21.9% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SLTS ECE = 5.3% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Dir.-Soft ECE = 3.8% Fig. 29: Reliability diagrams (summary): DermaMNIST ResNet-18. 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Gap +0.50 Uncal ECE = 37.7% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy TS ECE = 24.8% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SLTS ECE = 3.1% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Dir.-Soft ECE = 3.1% Fig. 30: Reliability diagrams (summary): DermaMNIST ViT-S/16. 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Voted-label baselines Accuracy Gap +0.45 Uncal ECE = 34.3% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy TS ECE = 21.9% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Platt (PS) ECE = 23.2% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Dir.-Hard ECE = 23.1% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy HB-Hard ECE = 21.0% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Annotation-free / Soft (TS-based) Accuracy LS-TS ECE = 4.1% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SLTS ECE = 5.3% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy MCTS ECE = 5.5% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SoftPlatt ECE = 4.2% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy VS ECE = 3.8% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Soft (non-parametric / matrix) Accuracy Dir.-Soft ECE = 3.8% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy HB-Soft ECE = 1.8% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy IR-Soft ECE = 1.5% Fig. 31: Reliability diagrams (all methods): DermaMNIST ResNet-18 (ECE true ). 27 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Voted-label baselines Accuracy Gap +0.50 Uncal ECE = 37.7% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy TS ECE = 24.8% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Platt (PS) ECE = 24.1% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Dir.-Hard ECE = 24.7% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy HB-Hard ECE = 22.7% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Annotation-free / Soft (TS-based) Accuracy LS-TS ECE = 7.3% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SLTS ECE = 3.1% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy MCTS ECE = 3.1% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy SoftPlatt ECE = 3.8% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy VS ECE = 2.8% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Soft (non-parametric / matrix) Accuracy Dir.-Soft ECE = 3.1% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy HB-Soft ECE = 1.3% 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy IR-Soft ECE = 1.4% Fig. 32: Reliability diagrams (all methods): DermaMNIST ViT-S/16 (ECE true ). 28 MELNVBCCAKBKLDFVLSCC Annotator's label MEL NV BCC AK BKL DF VL SCC Consensus (majority-vote) label 0.730.140.020.030.080.000.000.00 0.150.760.010.010.060.010.000.00 0.020.010.810.050.070.010.010.02 0.030.010.040.650.110.000.000.16 0.120.050.030.100.620.000.000.08 0.010.020.020.010.020.870.030.02 0.000.010.020.010.010.020.910.02 0.010.010.030.180.090.000.010.67 MEL/NV AK/BKL/SCC cluster 0.0 0.2 0.4 0.6 0.8 1.0 Probability Fig. 33: Inter-reader confusion matrix for ISIC 2019. EntryC ij is the probability that a dermatologist labels an image as classjgiven the consensus label isi. Diagonal entries (bold) are the per-condition agreement rates, calibrated to match Liu et al. [40] Table 2. Orange dashed box: MEL/NV most-confused pair. Purple dotted highlights: AK/BKL/SCC high-confusion cluster. Mean diagonal agreement ̄ C i = 75%.