Paper deep dive
Confidence Calibration of Deep Learning Systems
Coby Penso
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In high-stakes applications, reliable confidence estimates are as important as the predictions themselves. Confidence calibration ensures that predicted probabilities reflect the likelihood of correctness, making it essential for safe deployment of deep learning models. However, existing methods typically assume access to clean validation data, which is often unrealistic due to label noise and domain shifts. This thesis develops methods for improving calibration under these conditions. First, we address calibration under label noise. Standard methods can produce misleading confidence estimates when labels are unreliable. We propose a framework that uses an estimated noise model to reconstruct noise-free confidence estimates by modeling the relationship between noisy and clean label distributions. We extend this approach to Conformal Prediction (CP), which provides set-valued predictions with guaranteed coverage. Our noise-aware CP method estimates clean conformity scores despite label noise, enabling reliable uncertainty quantification. Next, we study calibration in unsupervised domain adaptation, where a model trained on a labeled source domain is adapted to an unlabeled target domain. Since labeled target data are unavailable, we estimate target-domain accuracy from source performance and domain discrepancies, enabling calibration without target labels. We also consider privacy-preserving settings in which user labels and model outputs must remain protected. We propose a locally differentially private conformal prediction framework that provides valid uncertainty quantification while maintaining privacy guarantees and balancing privacy, computational feasibility, and prediction reliability. Our results bridge calibration theory and practical deployment in safety-critical applications, contributing to reliable, privacy-preserving, and noise-resilient neural network predictions.
Tags
Links
- Source: https://arxiv.org/abs/2608.12100v1
- Canonical: https://arxiv.org/abs/2608.12100v1
Trouble viewing inline? Open PDF directly →
Full Text
237,086 characters extracted from source content.
Expand or collapse full text
Confidence Calibration of Deep Learning Systems Yacob (Coby) Penso Faculty of Engineering Ph.D. Thesis Submitted to the Senate of Bar-Ilan University Ramat Gan, Israel May 2025 This work was carried out under the supervision of Prof. Jacob Goldberger, The Faculty of Engineering, Bar-Ilan University Acknowledgment I would like to express my deepest gratitude to my supervisor, Prof. Jacob Goldberger, for his invaluable guidance, support, and encouragement throughout the course of my doctoral studies. His insight, patience, and high standards of research have shaped not only this thesis but also my approach to science as a whole. I am also grateful to my collaborators and co-authors, Ethan Fetaya, Bar Mahpud, and Lior Frenkel, whose ideas, feedback, and dedication have greatly enriched this work. It has been a privilege to learn and work alongside them. Finally, I extend my heartfelt thanks to my family and friends for their unwavering support, understanding, and encouragement during this journey. Without their love and patience, this thesis would not have been possible. Contents Hebrew Abstract ℵ List of Figures 3.1 Schema of the proposed model that includes the full pipeline of network training and calibration based on data with noisy labels. 3.2 Comparative calibration results on several datasets that were trained with ResNet-50. The top row shows the ECE results on the (clean) test set. The bottom row shows the optimal temperature found on the noisy validation set. 3.3 Standard deviation of ECE scores on the test set after the NTS calibration as a function of the size of the noisy validations set. 3.4 Calibration results for several noise transition matrices on ChestX-ray14-bal and ResNet-18. In each example we show the noise transition matrix (Left) and the adaECE measure on the test set for the compared calibration methods (right). 3.5 Calibration performance, measured by adaECE (left), and the corresponding temperature (right) as a function of the noise ratio used by the NTS algorithm. 4.1 Correction terms Δ of NACP, ACNL and CRCP as a function of the validation set size n given ϵ=0.2ε=0.2. We show results for 3 numbers of classes, 10, 100 and 1000. 4.2 Noisy labels conformal prediction on ImageNet with different calibration set sizes. (a) Mean size (b) Coverage (%), and (c) Correction terms Δ as a function of calibration set size. 5.1 Local Differential Private Conformal Prediction (LDP-CP-L) Pipeline (Best viewed in color). 5.2 Local Differential Private Conformal Prediction (LDP-CP-S) Pipeline (Best viewed in color). 5.3 Comparison of Δl _l and Δs _s as a function of the number of classes k and dataset size n, for ϵ=2,4,8ε=2,4,8. 5.4 CP correction terms ΔL,ΔS _L, _S as a function of ϵε privacy parameter across different dataset configurations of n and k without the shuffle model. 5.6 Size of prediction set (left) and coverage (right) as a function of the privacy ϵε (bottom x-axis) and effective privacy ϵeffε^eff (top x-axis). We show the (mean ± std) on TissueMNIST and APS score. 6.1 Average accuracy on Office-home tasks for the three UDA techniques (DANN, DANN+E, CDAN+E). 6.2 adaECE results as a function of the correction ratio R on Office-Home, A→CA→ C task. 6.3 Accuracy of k-th percentile source images based on their probability of being classified as target [97], compared to target accuracy (Office-home, A→CA→ C). 6.4 Accuracy per bin for source and target images. The results are shown on the Office-home C→PC→ P task. List of Tables 3.1 Adaptive ECE for top-1 predictions (in %) using 15 bins. on various medical imaging classification datasets and models with different calibration methods with a noise level ϵ=0.2ε=0.2. The lowest is highlighted in bold. 3.2 ECE for top-1 predictions (in %) using 15 bins. on various medical imaging classification datasets and models with different calibration methods with a noise level ϵ=0.2ε=0.2. The lowest is highlighted in bold. 3.3 Adaptive ECE for top-1 predictions (in %) using 15 bins (with the lowest in bold) on various medical imaging classification datasets and models with different calibration methods with varying noise levels. Model training and noise matrix estimation used [52]. 3.4 ECE for top-1 predictions (in %) using 15 bins (with the lowest in bold) on various medical imaging classification datasets and models with different calibration methods with varying noise levels. Model training and noise matrix estimation used [52]. 4.1 Conformal Prediction methods for validation sets with noisy labels. Given an image x, y is the true label, y y is its noisy version and ϵε is the noise level. S is a conformal score and S S is its noise robust variant. 4.2 CP calibration results for 1−α1\!-\!α = 0.9 and noise level ϵ=0.2ε=0.2. We report the mean and the std over 1000 different splits. We show the best result with theoretical guarantees in bold. 4.3 Finite sample correction terms Δ of NACP, ACNL [86] and CRCP [13], for several datasets and two noise levels, n is the size of the validation set. 4.4 Rand-APS calibration results for 1−α1\!-\!α = 0.9 on CIFAR-100 dataset and two noise models. We report the mean and the std over 1000 different splits. 4.5 CP calibration results on ImageNet and various model architectures for 1−α1\!-\!α = 0.9 and ϵ=0.2ε=0.2. We report the mean and the std over 1000 different splits. Bold for best result with theoretical guarantees. 4.6 CP calibration results on CIFAR-10N for 1−α1\!-\!α = 0.9 and ϵ=0.2ε=0.2. We report the mean and the std over 1000 different splits. Bold for best result with theoretical guarantees. 5.1 Calibration results for HPS and APS conformal scores across various datasets, using ϵ=4ε=4, ϵeff=ϵnε^eff= ε n, and α=0.1α=0.1 on 100 different seeds. 6.1 Comparison of calibration methods for unsupervised domain adaptation (UDA). 6.2 AdaECE results on Office-home (with the lowest in bold) on various UDA classification tasks and models with different calibration methods. 6.3 AdaECE results on Office-31 (with the lowest in bold) on various UDA classification tasks and models with different calibration methods. 6.4 adaECE results on VisDA Task S→RS→ R, for various calibration methods. 6.5 adaECE results on DomainNet for various UDA classification tasks and models with different calibration methods. 6.6 Calibration metrics results of various UDA calibration methods on the Office-home tasks. 6.7 Computed temperature on various UDA Office-home tasks, and calibration methods using CDAN+E. 6.8 AdaECE results for variations of UTDC based on different methods of domain accuracy estimation. 6.9 Comparison of several target domain accuracy estimation methods measured by |ACC(True)−ACC(Est)||ACC(True)-ACC(Est)|. List of Abbreviations and Notations In this thesis ϵε used as the noise rate, except for Chapter 5 when ϵε is the privacy in ϵε-LDP and β used as the noise rate. CE Conformal Prediction CE Cross-Entropy CNN Convolutional Neural Network DL Deep Learning DNN Deep Neural Network ECE Expected Calibration Error adaECE Adaptive Expected Calibration Error LLMs Large-Language Models MCE Maximum Calibration Error MSE Mean Square Error N Neural Network DA Domain Adaptation UDA Unsupervised Domain Adaptation OOD Out-of-Distribution UTDC Unsupervised Target Domain Calibration NRCP Noise Robust Conformal Prediction NACP Noise Aware Conformal Prediction NTS Noisy Temperature Scaling TS Temperature Scaling DP Differential Privacy LDP Local Differential Privacy LDP-CP Local Differential Private Conformal Prediction LDP-CP-L Local Differential Private Conformal Prediction On Labels LDP-CP-S Local Differential Private Conformal Prediction On Scores Abstract In high-stakes applications such as medical imaging, the reliability of a model’s confidence in its predictions is as crucial as the predictions themselves. Confidence calibration ensures that a model’s predicted probabilities accurately reflect its likelihood of correctness, making it a critical component for safe and effective deployment of deep learning models in medical diagnostics. However, existing calibration techniques assume access to clean validation data, which is often unrealistic in medical imaging settings due to the prevalence of label noise and domain shifts. This thesis explores novel methods for improving confidence calibration under these challenging conditions. First, we address confidence calibration in the presence of label noise. When calibration methods are applied to data with unreliable labels, they may yield misleading confidence estimates that undermine the trustworthiness of model predictions. We propose a calibration framework that accounts for label noise by leveraging an estimated noise model. Specifically, we demonstrate how to reconstruct noise-free confidence estimates by modeling the relationship between noisy and clean label distributions. We extend this idea to Conformal Prediction (CP), a framework that provides set-valued predictions with a guaranteed level of coverage. We introduce a noise-aware conformal prediction approach that estimates the true conformity scores despite label noise, allowing us to maintain efficient and reliable uncertainty quantification. Next, we investigate confidence calibration in unsupervised domain adaptation (UDA), where a model trained on a labeled source domain is adapted to an unlabeled target domain. Traditional calibration methods require labeled validation data from the target domain, which is unavailable in this setting. To overcome this limitation, we develop an approach that estimates the target domain accuracy based on the model’s performance in the source domain and known domain discrepancies. This allows us to directly calibrate model confidence without access to target domain labels. Furthermore, we extend our study to privacy-preserving settings, where individual user labels and model outputs must be protected. We propose a locally differentially private conformal prediction framework that ensures valid uncertainty quantification while maintaining rigorous privacy guarantees. Our approach balances the trade-offs between privacy, computational feasibility, and prediction reliability, making it applicable to sensitive medical data applications. Through extensive experiments on natural and medical imaging datasets, we demonstrate that our proposed methods significantly improve calibration robustness under both label noise and domain shift conditions. We provide theoretical guarantees and empirical validations that bridge the gap between theoretical calibration guarantees and practical deployment in safety-critical environments. Our findings contribute to the development of reliable, privacy-preserving, and noise-resilient calibration frameworks, enhancing the trustworthiness of neural network predictions in real-world medical and high-stakes applications. Chapter 1 Introduction In high-stakes domains like medical imaging, the accuracy of a model’s confidence in its predictions can be just as important as the predictions themselves. Confidence calibration is the process of ensuring that a model’s predicted probabilities align with its actual accuracy, offering reliable confidence estimates for each prediction. Confidence calibration is defined as the ability of a classifier network to provide an accurate probability of correctness for any of its predictions. Neural networks have been shown to be more overconfident in their predictions than their predecessors even though their generalization accuracy is higher, partly due to the fact that they can overfit on the negative log-likelihood loss without overfitting on the classification error [33, 49, 36]. In medical imaging applications, images for which the model makes low-confidence predictions are sent to a physician for review. Skipping the human review based on confident but incorrect predictions can have disastrous consequences [62]. The gap between the model’s predicted probabilities and its accuracy is one of the key obstacles to the applicability of neural network models to fully automatic medical diagnosis. Calibration methods can generally be divided into two main approaches. The first focuses on calibrating the confidence in a predicted class, while the second addresses the problem by generating a prediction set - a collection of possible classes - with a specified probability that the true class is included in this set. These approaches will be referred to as Confidence Calibration and Conformal Prediction, respectively, throughout this thesis. Within the framework of confidence calibration, various methods have recently been developed to address the issue of excessive overconfidence in predictions. Network calibration can either be performed alongside training (e.g., [63, 64, 115, 104]) or applied as a post-hoc procedure (e.g., Platt scaling [80], isotonic regression [110], and temperature scaling [33]). Post-hoc methods improve calibration by applying it as a post-processing step, using hold-out validation data to create a calibration map that adjusts the model’s predictions. Among these, temperature scaling stands out as a practical and widely adopted approach due to its simplicity and ease of implementation. Despite the critical role of network calibration in automating medical reporting, only a limited number of studies have specifically addressed the calibration of medical imaging systems (e.g., [23, 25, 83, 111]). In conformal prediction, the goal is to return a (preferably small) set of potential class candidates that includes the true class with a predefined level of confidence. This approach is particularly well-suited for medical imaging, where safety is paramount and the final decision is made by a human. By reducing the number of possible diagnoses a practitioner needs to consider, conformal prediction helps streamline decision-making while maintaining a controlled risk of error. The general method of producing a prediction set without making assumptions about the data distribution (aside from i.i.d. samples) is known as Conformal Prediction (CP) [1, 93]. CP guarantees that the probability of the correct class being included in the set meets or exceeds a specified confidence level, while aiming to return the smallest set possible that still maintains this guarantee. With the increasing use of neural networks in safety-critical applications like medical imaging, CP has emerged as a crucial calibration tool [57, 58, 67]. It is important to note that CP is a general framework rather than a single algorithm, with the most common implementations constructing the prediction set based on a conformity score. Different algorithms mainly differ in how this conformity score is defined. Confidence calibration and conformal prediction are extensively studied in settings where clean data and labels are provided. However, our research focuses on more complex and realistic scenarios, specifically when labels are noisy and in the context of unsupervised domain adaptation, where the goal is to calibrate the target model without access to labeled data. Deep neural networks have been highly successful in various natural image and medical image computing tasks. However, these achievements depend on having accurate annotated training data. Neural networks require massive amounts of carefully labeled data to succeed, but acquiring such data is expensive and time-consuming. Non-expert sources, like Amazon’s Mechanical Turk, have been used to reduce labeling costs, but their labels can be unreliable. Experienced domain experts may also struggle with complex labeling tasks. Medical imaging datasets often have problems with noisy labels due to ambiguous images that can confuse clinical experts. Physicians may disagree on the diagnosis of the same medical image, resulting in variability in the ground truth label. Furthermore, using Natural Language Processing (NLP) tools to extract labels from radiological reports can also introduce label noise [42]. Therefore, addressing annotation noise is a crucial topic in medical image analysis. Training neural networks with noisy labels is problematic because the models can easily overfit to the corrupted labels, resulting lack of generalizability when evaluated on a separate test dataset. While popular regularization techniques have been used to address overfitting, they do not entirely solve the problem. Even when these techniques are applied, there is a significant gap in test accuracy between models trained on clean vs. noisy data, and the accuracy decreases with label noise. Noisy labels are difficult to avoid, and studies indicate that Deep Neural Networks (DNNs) can memorize entire datasets. Consequently, errors in datasets may result in erroneous predictions, which can impact medical diagnoses. Therefore, effectively managing noisy labels is crucial for automated medical image classification. A review of network training methods for noisy labels can be found in [88] and an excellent up-to-date discussion of training medical image classification networks from data with noisy labels can be found in [105]. Numerous studies have examined the problem of training networks that are resilient to label noise, which can also disrupt the network calibration process. Our findings suggest that network calibration methods are more susceptible to label noise compared to network training. Nevertheless, we have not come across any previous research that tackles the challenge of network calibration using a validation set containing noisy labels. In addition to the challenges posed by noisy labels, another critical issue in real-world applications of deep learning is the performance degradation that occurs when a network trained on data from one domain is applied to data from a different domain, where the feature distribution differs - a phenomenon known as domain shift (see e.g., [61]). In the context of Unsupervised Domain Adaptation (UDA), the goal is to adapt a model to a target domain where labeled data is unavailable, though data from the target domain itself is accessible. Our findings indicate that existing calibration methods for unsupervised domain adaptation often fail in practice, particularly when the domain gap is large, further complicating the task of achieving reliable network calibration. Lastly, in many critical settings, the calibration procedure is performed by a centralized component, referred to as an aggregator, which may be untrusted. In such scenarios, exposing sensitive data directly to this untrusted aggregator poses significant privacy risks. A promising approach to mitigating these risks is Local Differentially Private (LDP) Conformal Prediction, where individual data contributors apply noise to their calibration data before sharing it with the aggregator. This ensures that the aggregator can perform calibration without directly accessing private or sensitive information from individual sources. However, the introduction of noise through LDP presents new challenges in maintaining both the validity and efficiency of conformal prediction methods. We explore strategies to adapt conformal calibration techniques to function effectively under differential privacy constraints, ensuring that predictions remain reliable while preserving user privacy. In our research, we tackle these challenges in three key areas: first, by addressing the calibration of neural networks using validation sets with inaccurate labels, focusing on both confidence calibration and conformal prediction; second, by exploring confidence calibration in systems facing unsupervised domain shift scenarios; and third, by investigating privacy-preserving calibration methods using local differentially private conformal prediction, ensuring robust calibration without compromising data privacy. Through these efforts, we aim to enhance the reliability, robustness, and privacy of neural networks in real-world applications where label noise, domain shifts, and privacy concerns are significant challenges. The remainder of this thesis is structured as follows: Chapter 2, Background, provides an overview of the foundational concepts and related work, including confidence calibration, conformal prediction, unsupervised target domain calibration, noisy labels, and local differential privacy. Chapter 3, Confidence Calibration under Noisy Labels, introduces our proposed methods for improving confidence calibration when dealing with noisy labels. We explore both the challenges of confidence calibration with noisy labels and network training strategies, followed by a comprehensive set of experiments to validate our approach. Chapter 4, Conformal Prediction under Noisy Labels, extends our focus to conformal prediction, presenting a robust scoring method and a threshold estimation procedure tailored to noisy label scenarios. We provide theoretical insights and experimental evaluations, including prediction size comparisons, coverage guarantees, and adaptations to more general noise models. Chapter 5, Local Differential Private Conformal Prediction, delves into privacy-preserving techniques, introducing methods to apply local differential privacy (LDP) to both labels and prediction scores. We discuss theoretical guarantees, practical considerations, and experimental results, comparing different approaches under privacy constraints. Chapter 6, Unsupervised Target Domain Confidence Calibration, addresses the challenge of calibrating confidence when transitioning models to a new, unlabeled target domain. We describe our proposed calibration methods, present experimental findings, and analyze the performance under unsupervised domain adaptation settings. Chapter 7, Discussion, summarizes our contributions, highlights the key insights and implications of our work, and outlines potential directions for future research. The chapter concludes by reaffirming the significance of our research in advancing confidence calibration and conformal prediction in challenging real-world scenarios. As part of the thesis, the following papers have been published in various conferences and journals: • Confidence calibration of a medical imaging classification system that is robust to label noise - Coby Penso, Lior Frenkel, Jacob Goldberger, IEEE Transactions on Medical Imaging (TMI), vol. 43(6), p. 2050-2060, 2024, [73]. • A joint training and confidence calibration procedure that is robust to label noise - Coby Penso, Jacob Goldberger, IEEE International Symposium on Biomedical Imaging (ISBI), 2024, [77]. • A conformal prediction score that is robust to label noise - Coby Penso, Jacob Goldberger, MICCAI, Machine Learning for Medical Imaging Workshop, 2024, [76]. • Network calibration under domain shift based on estimating the target domain accuracy - Coby Penso, Jacob Goldberger, ECCV, Uncertainty in Computer Vision Workshop, 2024, [78]. • Conformal Prediction of Classifiers with Many Classes based on Noisy Labels - Coby Penso, Jacob Goldberger, Eitan Fetaya, accepted to the Symposium on Conformal and Probabilistic Prediction with Applications (COPA), 2025, [75]. • Privacy-Preserving Conformal Prediction Under Local Differential Privacy - Coby Penso, Bar Mahpud, Jacob Goldberger, Or Sheffet, accepted to Symposium on Conformal and Probabilistic Prediction with Applications (COPA), 2025, [79]. Chapter 2 Background 2.1 Confidence Calibration In this section we review the definition of confidence calibration. Consider a network that classifies an input image x into k pre-defined categories. The last layer of the network architecture is comprised of k real numbers z=(z1,…,zk)z=(z_1,...,z_k) known as logits. Each of these numbers is the score for one of the k possible classes. The logits are then converted into a soft decision distribution using a softmax layer: p(y=i|x)=exp(zi)∑jexp(zj)p(y=i|x)= (z_i) _j (z_j) where x is the input image and y is the image class. Despite having the mathematical form of a distribution, the output of the softmax layer does not necessarily represent the true posterior distribution of the classes, and the network often tends to have overconfidence in its predictions. The predicted class is calculated from the output distribution by y^=argmaxip(y=i|x)=argmaxizi y= _ip(y=i|x)= _iz_i. The network confidence for this sample is defined by p^=p(y=y^|x)=maxip(y=i|x) p=p(y= y|x)= _ip(y=i|x). The network accuracy is defined by the probability that the most probable class y y is indeed correct. The network is said to be calibrated if the estimated confidence coincides with the actual accuracy. The Expected Calibration Error (ECE) [65] stands as the conventional metric employed for quantifying the calibration of a model. It is characterized by the expected absolute disparity between the model’s accuracy and its level of confidence. In practice, we only have a validation set with a finite number of samples (x1,y1),…,(xn,yn)(x_1,y_1),...,(x_n,y_n) thus an approximation is used. Denote the predictions and confidence values of the validation set by (y^1,p^1),…,(y^n,p^n)( y_1, p_1),...,( y_n, p_n). To compute the ECE measure we first divide the unit interval [0,1][0,1] into m equal size bins b1,…,bmb_1,...,b_m and let Bi=t|p^t∈biB_i=\t| p_t∈ b_i\ be the set of samples whose confidence values belong to bin bib_i. The network average accuracy at this bin is computed as: Ai=1|Bi|∑t∈Biy^t=yt,A_i= 1|B_i| _t∈ B_i 1_\ y_t=y_t\, (2.1) where 1 is the indicator function, and yty_t and y^t y_t are the correct and predicted labels for xtx_t respectively. AiA_i is the relative number of correct predictions of instances that were assigned to BiB_i based on their confidence value. The average confidence at bin bib_i is computed as: Ci=1|Bi|∑t∈Bip^t.C_i= 1|B_i| _t∈ B_i p_t. (2.2) If the network is under-confident at bin bib_i then Ai>CiA_i>C_i and vice-versa. The ECE is defined as follows: ECE=∑i=1m|Bi|n|Ai−Ci|.ECE= _i=1^m |B_i|n |A_i-C_i |. (2.3) The ECE is based on a uniform bin width. If the model is well-trained, most of the samples should lie within the highest confidence bins. Hence, the low confidence bins should be almost empty and therefore have no influence on the computed value of the ECE. For this reason, we can consider another metric, Adaptive ECE (adaECE) where bin sizes are taken into account so as to evenly distribute samples between bins [66]: adaECE=1m∑i=1m|Ai−Ci|adaECE= 1m _i=1^m |A_i-C_i | (2.4) such that each bin contains 1/m1/m of the data points with similar confidence values. AdaECE is considered a better and more resilient method than ECE for assessing network calibration. In this study we used the adaECE for both calibration and evaluation. Temperature Scaling (TS) is a standard highly effective technique for calibrating the output distribution of a classification network [33]. It uses a single parameter T>0T>0 to rescale logit scores before applying the softmax function to compute the class distribution. Temperature scaling is expressed as follows: pT(y=i|x)=exp(zi/T)∑j=1kexp(zj/T),i=1,…,kp_ T(y=i|x)= (z_i/T) _j=1^k (z_j/T), 11.38092pti=1,…,k (2.5) s.t. z1,…,zkz_1,...,z_k are the logit values derived from the application of the network to the input vector x. The optimal temperature T for a trained model can be found by maximizing the log-likelihood ∑tlogpT(yt|xt) _t p_ T(y_t|x_t) for the held-out validation dataset. Studies show that finding the optimal T by directly minimizing the ECE or adaECE measures yields better calibration results [63]. 2.2 Conformal Prediction Consider a setup involving a classification network that categorizes an input x into k predetermined classes. Given a coverage level of 1−α1-α, we aim to identify the smallest possible prediction set (a subset of these classes) ensuring the correct class is within the set with a probability of at least 1−α1-α. A straightforward strategy to achieve this objective involves sequentially incorporating classes from the highest to the lowest probabilities until their cumulative sum exceeds the threshold of 1−α1-α. Despite the network’s output adopting a mathematical distribution format, it does not inherently reflect the actual class distribution. Typically, the network will not be calibrated and it tends to be overly optimistic [33]. Consequently, this straightforward approach doesn’t assure the inclusion of the correct class with the desired probability. The first step of the CP algorithm involves forming a conformity score S(x,y)S(x,y) that measures the network’s uncertainty between x and its true label y (larger scores indicate worse agreement). The Homogeneous Prediction Sets (HPS) score [93] is SHPS(x,y)=1−p(y|x;θ)S_ HPS(x,y)=1-p(y|x;θ), s.t. θ is the network parameter set. The Adaptive Prediction Score (APS) [82] is the sum of all class probabilities that are not lower than the probability of the true class: SAPS(x,y)=∑i|pi≥pypi,S_ APS(x,y)= _\i|p_i≥ p_y\p_i, (2.6) such that pi=p(y=i|x;θ)p_i=p(y=i|x;θ) and pyp_y is the probability of the label y. The RAPS score [3] is a variant of APS, which is defined as follows: SRAPS(x,y)=∑i|pi≥pypi+a⋅max(0,(NC−b))S_ RAPS(x,y)= _\i|p_i≥ p_y\p_i+a· (0,(NC-b)) (2.7) s.t. NC=|i|pi≥py|NC=|\i|p_i≥ p_y\| and a,ba,b are parameters that need to be tuned. RAPS is especially effective in the case of a large number of classes where it explicitly encourages small prediction sets. We can also define a randomized version of a conformity score. For example in the case of APS we define: Srand−APS(x,y,u)=∑i|pi>pypi+u⋅py,u∼U[0,1].S_ rand-APS(x,y,u)= _\i|p_i>p_y\p_i+u· p_y, 28.45274ptu U[0,1]. (2.8) The random version tends to yield the required coverage more precisely and thus it produces smaller prediction sets [1]. The CP prediction set of a data point x is defined as C(x)=y|S(x,y)≤qC(x)=\y|S(x,y)≤ q\ where q is a threshold that is found using a labeled validation set (x1,y1),…,(xn,yn)(x_1,y_1),...,(x_n,y_n). The CP theorem states that if we set q to be the (1−α)(1\!-\!α) quantile of the conformal scores S(x1,y1),…,S(xn,yn)S(x_1,y_1),...,S(x_n,y_n) we can guarantee that 1−α≤p(y∈C(x))≤1−α+1n+11\!-\!α≤ p(y∈ C(x))≤ 1\!-\!α+ 1n+1, where x is a test point and y is its the unknown true label [93]. In the random case there is still a coverage guarantee, which is defined by marginalizing over all test points x and samplings u from the uniform distribution [82]. Note that the coverage guarantee is for a marginal probability over all possible test points and coverage may be worse or better for different points. It can be proved that obtaining a conditional coverage guarantee is impossible [24]. 2.3 Unsupervised Target Domain Calibration When a network trained on data from one domain is applied to data from a different domain, the distribution of features often changes between domains, leading to what is known as the domain shift problem (see e.g. [61]). In an Unsupervised Domain Adaptation (UDA) setup, we assume the availability of data from the target domain without any annotations. Numerous UDA methods have been developed to address this issue, employing strategies such as adversarial training to align the distributions of the source and target domains [27], or self-training algorithms that compute pseudo labels for the target domain data [120]. Studies show that present-day UDA methods are prone to learning improved accuracy at the expense of deteriorated prediction confidence [97]. This brings us to the challenge of calibrating the network’s confidence on the target domain data. In UDA, adapting a network to the target domain typically focuses on accuracy improvements, but accurately calibrating the model’s confidence is equally important. Without calibration, the model may become overconfident in its predictions despite reduced performance on the target domain. Next, we formulate the problem of unsupervised target domain calibration. Assume a network was trained on the source domain. We are given a labeled source domain validation-set dataset, denoted as =(xsi,ysi)i=1nsS=\(x_s^i,y_s^i)\_i=1^n_s with nsn_s samples, and an unlabeled target domain dataset =xtii=1ntT=\x_t^i\_i=1^n_t with ntn_t samples. Adapting the network trained on the source domain to the target domain in an unsupervised manner without access to the labels can be achieved using various methods. Here, our goal is to calibrate the confidence of the adapted network prediction on samples from the target domain. Calibrating the confidence of the adapted model on data from the target domain is challenging due to the coexistence of the domain gap and the lack of target labels. Current UDA calibration methods use the labeled validation set from the source domain to approximate the target domain statistics in certain aspects. Some studies [85, 90] propose to modify the calibration set to represent a generic distribution shift. Other methods [69, 97, 68] apply Importance Weighting (IW) by assigning higher weights to source examples that resemble those in the target domain. In practice, even after the domain adaptation process, the accuracy on the source domain, where labels are available, remains greater than the accuracy on the target domain. Hence, the accuracy estimation when calibrating the target domain using the source data is still too optimistic. Calibrating neural networks is necessary because they are often overconfident in their predictions compared to their actual accuracy [33, 49, 36]. If the accuracy is overestimated, it conceals the overconfidence issue, leading to a suboptimal temperature scaling value in the case of temperature scaling. Another drawback of IW methods is that they only use the unlabeled target data to train a binary source/target classifier, but the actual calibration is done on the source domain data while the target domain data are ignored. The network confidence, however, is independent of the true labels and can thus be directly computed on the target data. 2.4 Noisy Labels In supervised classification tasks a dataset is defined as pairs of input and labels =(xi,yi)i=1nDD=\(x^i,y^i)\_i=1^n_D. In the setting of label noise, we only observe the corrupted labels y~i=g(yi) y_i=g(y_i) for some corruption function g:Y×[0,1]→Yg:Y×[0,1] Y. One notable label noise function follows a uniform distribution, where with a probability of ϵε, the correct label is replaced by a randomly selected label 2.9. The noise is applied to each sample independently. This noise model is commonly referred to as uniform noise. The uniform noise may be formulated also such that with a probability of ϵε, the correct label is replaced by a randomly selected label from the remaining (k−1)(k-1) classes 2.10. In this thesis, we adopt both definitions interchangeably depending on the context, as the transition between them is straightforward. A more general noise model assumes that the true label is corrupted by a label noise matrix P, where P(i,j)=p(y~=j|y=i)P(i,j)=p( y=j|y=i) is the probability of the true label i being flipped to a corrupted label j. In the simpler uniform noise model, P takes the form of: P(i,j)=(1−ϵ)i=j+ϵk.P(i,j)=(1-ε) 1_\i=j\+ εk. (2.9) P(i,j)=(1−ϵ)i=j+ϵki≠j.P(i,j)=(1-ε) 1_\i=j\+ εk 1_\i≠ j\. (2.10) In future chapters, we concentrate on calibrating the network confidence based on noisy labels. A preliminary step before confidence calibration is training the network using a training set with noisy labels (see Figure 3.1). We next provide a brief overview of current training methods that are resilient to label noise and describe the training method we used in our experiments. There is a plethora of recent works on learning with noisy labels, which include estimating the noise matrix [31, 17, 52, 37, 103, 102, 10], reweighting examples [55, 81, 87, 94], selecting confident examples [40, 106, 8, 50], designing robust loss functions [118, 30, 11, 100], introducing regularization [112, 38, 9] and generating pseudo labels [89, 119, 116, 34, 51]. Zhang et al. [114] addressed the problem of learning from noisy labels in the context of inconsistent annotation collected from several medical experts. They also relied on the fact that the confusion matrix of the noisy labels can be expressed as the matrix product between the confusion matrix of the clean labels and the label noise. Our focus in this context is on learning techniques that deal with noisy data by estimating the label noise matrix. These methods have been successful in producing cutting-edge results, and our calibration approach utilizes the estimated noise matrix to obtain a noise-robust calibration measure. Noise robust training methods which estimate the noise matrix, are all based on the following observation. The clean class-posterior p(y|x)p(y|x) can be inferred by utilizing the noisy class-posterior p(y~|x)p( y|x) and the class-dependent noise matrix P, where Pij=P(y~=j|y=i)P_ij=P( y=j|y=i), as follow: p(y|x)=P−1p(y~|x)p(y|x)=P^-1p( y|x). While this approach theoretically guarantees statistical consistency, it relies heavily on the success of estimating the noise matrix. Several methods have been developed to estimate the noise matrix under the so-called anchor-point assumption. Anchor points are instances belonging to a specific class with a probability of one [55]. This assumption is reasonable in certain applications but typically, we cannot assume the availability of anchor points. This has motivated the development of noise-robust training algorithms that do not exploit anchor points. Several studies have implemented modifications to the classification network’s architecture to better represent the label noise matrix in noisy datasets [6, 31, 52]. These adjustments encompass the inclusion of a noise adaptation layer on top of the softmax layer and the creation of a specialized architecture. The noise adaptation layer is intended to mimic the label transition behavior in learning a network. These changes have led to enhanced generalization by altering the network output according to the estimated label transition probability. In our implementation, of training of noisy labels we follow the approach in Li et al. [52] which yields state-of-the-art results. Denote the network’s soft-max label prediction by pθ(y|x)p_θ(y|x) where θ is the network’s parameter set. Given training data x1,…,xnx_1,...,x_n with corresponding noisy labels y~1,…,y~n y_1,..., y_n, the standard loss function is: L(θ)=∑t=1nCE(y~t,pθ(yt|xt)),L(θ)= _t=1^n CE( y_t,p_θ(y_t|x_t)), (2.11) such that CE is the cross-entropy loss. Li et al. [52] proposed minimizing the following loss function: L(θ,P)=∑t=1nCE(y~t,P⋅pθ(yt|xt))+λlogdet(P)L(θ,P)= _t=1^n CE( y_t,P· p_θ(y_t|x_t))+λ (P) (2.12) such that λ>0λ>0 is a regularization coefficient that trades off distribution fidelity with the complexity of the matrix P. P is enforced to be a diagonally dominant stochastic matrix (i.e. Pi,i>Pi,jP_i,i>P_i,j for every i≠ji≠ j). We first create a matrix A s.t. Ai,i=1A_i,i=1 and Ai,j=σ(wi,j)A_i,j=σ(w_i,j) for all i≠ji≠ j where σ is the sigmoid function and each wi,jw_i,j is a real-valued variable that is updated throughout training. Then we normalize each row to obtain a stochastic matrix: Pij=Ai,j∑lAi,lP_ij= A_i,j _lA_i,l. Once we finish the training phase, we eliminate the noise adaptation layer defined by matrix P, because our objective is to predict the clean label. As a by-product of the training process, we also obtain an approximation of the label noise matrix P, which we can utilize in our network calibration approach. Current label-noise learning methods generally assume that the class distribution of the training data is balanced, i.e., that each class is represented by almost the same number of samples. However, data in real-world applications are often imbalanced. In cases where the training labels are both noisy and imbalanced it is difficult to distinguish between clean and noisy samples in rare classes because the clean samples are overwhelmed by noisy labels from frequent classes. Several recent attempts have been made to find noise-robust training procedures for imbalance data [59, 41, 43]. However, all these methods are focused on extracting confident examples. We are not aware of any methods for estimating the noise matrix with unbalanced data. Additional noise types. Next, we define two common general noise matrices that will be used through out this thesis. The Neighborhood noise as: Pi,j=p(y~=j|y=i)=ξif i=j1−ξif |i−j|=1 and i∈(1,k)(1−ξ)/2if |i−j|=1 and i∉(1,k)0otherwiseP_i,j=p( y=j|y=i)= casesξ 48.36958ptif i=j\\ 1-ξ 31.2982ptif |i-j|=1 and i∈(1,k)\\ (1-ξ)/2 14.22636ptif |i-j|=1 and i ∈(1,k)\\ 0 46.94687ptotherwise cases (2.13) The Random noise is defined as: first, on the diagonal, we have ξ. next, for each line (aka ∀i∀ i) the rest of the values (i.e. k−1k-1 items) are sampled from a random distribution uiu_i vector of size k−1k-1 and then normalized to sum up to 1−ξ1-ξ to keep the matrix a probability matrix. Pi,j=p(y~=j|y=i)=ξif i=j(1−ξ)⋅ui[j]∑z≠iui[z]otherwiseP_i,j=p( y=j|y=i)= casesξ 110.96556ptif i=j\\ (1-ξ)· u_i[j] _z≠ iu_i[z] 36.98866ptotherwise cases (2.14) In this study, we tackle the challenge of applying calibration on a validation set with noisy labels. Calibration methods, particularly for neural networks, are known to be highly sensitive to label noise. In the context of conformal prediction, Einbinder et al. [21] proposed ignoring the label noise and directly applying the standard CP algorithm to the noisy-labeled validation set. However, this approach tends to result in overly large prediction sets. In contrast, within the field of confidence calibration, we have not encountered any prior research that specifically addresses the issue of calibrating networks using validation sets with noisy labels. 2.5 Local Differential Privacy Traditional (central) differential privacy [20] presumes a trusted curator who sees the raw data and then adds noise before publishing. In contrast, LDP [19, 45, 46] treats the aggregator as untrusted: individual users randomize their own data locally before sending it to the aggregator, thus ensuring strong privacy. LDP is considered a harder setting since noise insertion is done on the user side in a distributed manner, whereas in the centralized DP model the curator holds the entire data and can apply operations on the clean data. Definition 2.5.1. A discrete randomized mechanism Q(⋅)Q(·) is ε -LDP if for any pair of input labels y,y′∈y,y and any output z, Q(z∣y)≤eεQ(z∣y′).Q(z y)\;≤\;e \,Q(z y ). This definition ensures that any two possible labels are (roughly) indistinguishable from the aggregator’s perspective. A common mechanism is the k-ary randomized response (k-R) [98, 95]. For a label y∈1,…,ky∈\1,…,k\, it outputs: y~=y,w.p eε(k−1)+eε,any other label (uniformly),w.p 1(k−1)+eε. y= casesy,&w.p e (k-1)+e ,\\ any other label (uniformly),&w.p 1(k-1)+e . cases This preserves label privacy, preventing the aggregator from easily inferring the user’s true label from the reported y~ y. The parameter ε , known as the privacy loss, controls the privacy-utility trade-off: lower ε enforces stronger privacy guarantees but introduces more noise, potentially degrading the utility of downstream applications. Note that if k=2k=2, we recover Warner’s binary randomized response (R) [98], flipping the label with some probability. LDP has gained popularity as a strong privacy paradigm that enables data owners to randomize their data locally before sharing it with an untrusted aggregator, thus ensuring that sensitive information remains protected (e.g. Google’s RAPPOR [22] and Apple’s locally private data collection of emojis and usage patterns [4]). Challenges of LDP for statistical learning and conformal prediction. While LDP ensures strong privacy guarantees, its main challenge lies in the significant increase in variance due to local randomization. Unlike central DP, where controlled noise can be added post-aggregation, in LDP, the noise is introduced at the user level, leading to a loss of information before any statistical inference is performed. This introduces several key challenges in machine learning and uncertainty quantification: • Challenges in calibration: Many traditional statistical methods assume access to clean calibration data. However, in an LDP setting, the observed data is randomized, affecting the empirical coverage of conformal prediction intervals. • Impact on distribution-free guarantees: Conformal prediction provides finite-sample marginal coverage guarantees without assumptions on the underlying data distribution. However, when predictions are made using noisy, privatized data, the standard conformal prediction framework may require adaptation to account for the added uncertainty. Chapter 3 Confidence Calibration under Noisy Labels In this chapter, we explore the critical challenge of confidence calibration in neural networks, particularly under the influence of noisy labels. Confidence calibration is vital in applications such as medical imaging, where overly confident yet incorrect predictions can have serious consequences. We review existing calibration methods, including post-hoc techniques like Temperature Scaling, and discuss the unique difficulties posed by label noise in medical datasets. Finally, we introduce a novel method that leverages noisy validation data to achieve robust calibration, demonstrating its effectiveness across various medical imaging scenarios. 3.1 Problem Statement Confidence calibration is defined as the ability of a classifier network to provide an accurate probability of correctness for any of its predictions. Neural networks have been shown to be more overconfident in their predictions than their predecessors even though their generalization accuracy is higher, partly due to the fact that they can overfit on the negative log-likelihood loss without overfitting on the classification error [33, 49, 36]. In medical imaging applications, images for which the model makes low-confidence predictions are sent to a physician for review. Skipping the human review based on confident but incorrect predictions can have disastrous consequences [62]. The gap between the model’s predicted probabilities and its accuracy is one of the key obstacles to the applicability of neural network models to fully automatic medical diagnosis. Various confidence calibration methods have recently emerged, aiming to address the issue of excessive overconfidence. Network calibration can be performed in conjunction with training (see e.g. [63, 64, 115, 104]). Post-hoc scaling approaches to calibration (e.g. Platt scaling [80], isotonic regression [110], and temperature scaling [33]) are widely used. In order to enhance calibration, they incorporate calibration as a post-processing step, utilizing hold-out validation data to acquire a calibration map that modifies the model’s predictions. Temperature scaling, which is currently the widely accepted practical calibration method, is a straightforward approach that can be easily implemented. Despite the importance of network calibration for automating medical reports, there are only a few studies that have addressed the problem of calibrating medical imaging systems (see e.g. [23, 25, 83, 111]). Deep neural networks have been highly successful in various natural image and medical image computing tasks. However, these achievements depend on having accurate annotated training data. Neural networks require massive amounts of carefully labeled data to succeed, but acquiring such data is expensive and time-consuming. Non-expert sources, like Amazon’s Mechanical Turk, have been used to reduce labeling costs, but their labels can be unreliable. Experienced domain experts may also struggle with complex labeling tasks. Medical imaging datasets often have problems with noisy labels due to ambiguous images that can confuse clinical experts. Physicians may disagree on the diagnosis of the same medical image, resulting in variability in the ground truth label. Furthermore, using Natural Language Processing (NLP) tools to extract labels from radiological reports can also introduce label noise [42]. Therefore, addressing annotation noise is a crucial topic in medical image analysis. Training neural networks with noisy labels is problematic because the models can easily overfit to the corrupted labels, resulting lack of generalizability when evaluated on a separate test dataset. While popular regularization techniques have been used to address overfitting, they do not entirely solve the problem. Even when these techniques are applied, there is a significant gap in test accuracy between models trained on clean vs. noisy data, and the accuracy decreases with label noise. Noisy labels are difficult to avoid, and studies indicate that Deep Neural Networks (DNNs) can memorize entire datasets. Consequently, errors in datasets may result in erroneous predictions, which can impact medical diagnoses. Therefore, effectively managing noisy labels is crucial for automated medical image classification. A review of network training methods for noisy labels can be found in [88] and an excellent up-to-date discussion of training medical image classification networks from data with noisy labels can be found in [105]. In the following chapter, we address the challenge of calibrating medical networks with a validation set that has inaccurate labels. Numerous studies have examined the problem of training networks that are resilient to label noise, which can also disrupt the network calibration process. Our findings suggest that network calibration methods are more susceptible to label noise compared to network training. Nevertheless, we have not come across any previous research that tackles the challenge of network calibration using a validation set containing noisy labels. The findings reveal that the Temperature Scaling method [33], which is commonly used, is highly susceptible to label noise and can even result in worse calibration than the original model. We present a simple method that uses data with noisy labels to calibrate a network by taking advantage of the fact that in calibration, we only need to estimate the average accuracy at pre-determined confidence bins, rather than determining the correctness of each label. Testing the method on various medical imaging datasets, network architectures, and noise levels, showed that the calibration results were comparable to those obtained using a noise-free validation set. The study described in this chapter was published in [73, 77]. 3.2 Confidence Calibration with Noisy Labels Consider a multi-class classification task with k classes. Suppose we have a validation set with labels that are potentially inaccurate. Let y1,…,yny_1,...,y_n be the correct labels of the validation set and let y~1,…,y~n y_1,..., y_n be the corresponding observed corrupted labels. We assume that the label noise follows a uniform distribution, where with a probability of ϵε, the correct label is replaced by a randomly selected label from the remaining (k−1)(k-1) classes. The noise is applied to each sample independently. This noise model is commonly referred to as uniform noise. Our objective is to calibrate the network using these noisy labels. Figure 3.1: Schema of the proposed model that includes the full pipeline of network training and calibration based on data with noisy labels. It can be verified from the adaECE definition (2.4) that only the accuracy terms Aii=1m\A_i\_i=1^m are affected by the noise, whereas the confidence terms Cii=1m\C_i\_i=1^m remain the same. Let A~i A_i be the average accuracy at bin i that is computed using the corrupted labels. Note that a prediction is considered correct if the label is not corrupted and the network’s prediction matches it, or if the label is corrupted and the network’s (incorrect) prediction matches the corrupted label. This implies that: A~i=1|Bi|∑t∈Biy^t=y~t=1|Bi|∑t∈Bi|y~t=yty^t=yt+1|Bi|∑t∈Bi|y~t≠yty^t=y~t. A_i= 1|B_i| _t∈ B_i 1_\ y_t= y_t\= 1|B_i| _\t∈ B_i| y_t=y_t\ 1_\ y_t=y_t\+ 1|B_i| _\t∈ B_i| y_t≠ y_t\ 1_\ y_t= y_t\. The law of large numbers implies that as the size of the validation set increases, the empirical average noisy accuracy at each adaECE bin becomes increasingly close to the mean noisy accuracy. Therefore: A~i≈(1−ϵ)Ai+ϵk−1(1−Ai). A_i≈(1-ε)A_i+ εk-1(1-A_i). (3.1) We can thus obtain an estimation A^i A_i of the correct accuracy AiA_i from the noisy accuracy A~i A_i as follows: A~i=(1−ϵ)A^i+ϵk−1(1−A^i). A_i=(1-ε) A_i+ εk-1(1- A_i). (3.2) Rearranging (3.2) we finally obtain: A^i=A~i−ϵ/(k−1)(1−ϵ)−ϵ/(k−1). A_i= A_i-ε/(k-1)(1-ε)-ε/(k-1). (3.3) Substituting the estimated accuracy term, based on data with noisy labels (3.3) into the adaECE definition (2.4), yields the following noise-robust adaECE measure: Noisy-adaECE(ϵ)=1m∑i=1m|A~i−ϵ/(k−1)(1−ϵ)−ϵ/(k−1)−Ci|.Noisy -adaECE(ε)= 1m _i=1^m | A_i-ε/(k\!-\!1)(1\!-\!ε)-ε/(k\!-\!1)-C_i |. (3.4) When training a network with noisy labels we need to make individual decisions for each sample regarding the corruption of its label. The key feature of the Noisy-adaECE metric is that here we only require an estimation of the average label noise within each bin, which is a significantly simpler task. Algorithm 1 Noisy Temperature Scaling (NTS) - Uniform noise input: A validation set (x1,y~1),…,(xn,y~n)(x_1, y_1),...,(x_n, y_n) whose labels are corrupted with noise level ϵε. - Feed each xtx_t into the classifier network to produce class distribution pt1,…,ptkp_t1,...,p_tk. Compute the confidence values p^t=maxjptj p_t= _jp_tj and the predictions y^t=argmaxjptj y_t= _jp_tj. - Order the points based on their confidence and divide them into equal-sized sets B1,…,BmB_1,...,B_m. - Find T that minimizes the Noisy-adaECE score: Noisy-adaECE(T)=1m∑i=1m|max(0,min(1,A~i−ϵ/(k−1)(1−ϵ)−ϵ/(k−1)))−Ci(T)|Noisy -adaECE(T)= 1m _i=1^m | (0, (1, A_i-ε/(k-1)(1-ε)-ε/(k-1)))-C_i(T) | s.t. Ci(T)=1|Bi|∑t∈Biexp(log(p^t/T))∑l=1kexp(log(ptl/T))andA~i=1|Bi|∑t∈Biy^t=y~tC_i(T)= 1|B_i| _t∈ B_i ( ( p_t/T)) _l=1^k ( (p_tl/T)) 28.45274pt and 28.45274pt A_i= 1|B_i| _t∈ B_i 1_\ y_t= y_t\ output: T^=argminNoisy-adaECE(T) T= -adaECE(T) This Noisy-adaECE calibration measure assumes knowledge of the noise level ϵε. If ϵε is not known, we can estimate it from the noisy data (see e.g. [70, 52, 117]). In the next section, we describe the noise level estimation that was used in our experiments. For each calibration method whose parameters can be found by minimizing the adaECE measure, we can form a noise-robust variant in which Noisy-adaECE (3.4) is minimized instead of adaECE (2.4). Examples of these calibration methods include Temperature Scaling (TS), Vector Scaling, Matrix Scaling [33], Mix-n-Match [113], Wight Scaling [25], and others. We next present the noise-robust calibration measure in the case of the TS method. The optimal temperature is obtained by finding the temperature T that minimizes the Noisy-adaECE (3.4) calibration measure. The proposed robust variant of TS which we dub the Noise-robust Temperature Scaling (NTS) algorithm, is summarized in Algorithm Box 1. So far we have considered the simplest uniform label noise model. A more general noise model assumes that the true label is corrupted by a label noise matrix P, where P(i,j)=p(y~=j|y=i)P(i,j)=p( y=j|y=i) is the probability of the true label i being flipped to a corrupted label j. In the simpler uniform noise model, P takes the form of: P(i,j)=(1−ϵ)i=j+ϵk−1i≠j.P(i,j)=(1-ε) 1_\i=j\+ εk-1 1_\i≠ j\. (3.5) We next extend the noise-robust calibration measure Noisy-adaECE defined above to the case of a general label noise matrix P. Denote Mi(r,s)=pi(y^=r,y=s)=1|Bi|∑t∈Biy^t=r,yt=s.M_i(r,s)=p_i( y=r,y=s)= 1|B_i| _t∈ B_i 1_\ y_t=r,y_t=s\. MiM_i is the classifier confusion matrix computed on the validation data from the i-th bin using clean labels. The adaECE accuracy term (2.1) is thus: Ai=∑j=1kMi(j,j)=Tr(Mi).A_i= _j=1^kM_i(j,j)= Tr(M_i). (3.6) In a similar manner, we define a confusion matrix based on the available noisy labels: M~i(r,s)=1|Bi|∑t∈Biy^t=r,y~t=s. M_i(r,s)= 1|B_i| _t∈ B_i 1_\ y_t=r, y_t=s\. (3.7) According to our noise model, given a sample from the validation set along with its true label, the corresponding noisy label and the network soft prediction are conditionally independent. This implies that: M~i(r,s) M_i(r,s) =pi(y^=r,y~=s)=∑jpi(y^=r,y~=s,y=j) =p_i( y=r, y=s)= _jp_i( y=r, y=s,y=j) =∑jpi(y^=r,y=j)p(y~=s|y^=r,y=j) = _jp_i( y=r,y=j)p( y=s| y=r,y=j) (3.8) =∑jpi(y^=r,y=j)p(y~=s|y=j)=∑jMi(r,j)P(j,s). = _jp_i( y=r,y=j)p( y=s|y=j)= _jM_i(r,j)P(j,s). We can write (3.2) as a matrix multiplication: M~i=MiP M_i=M_iP. This implies that Mi=M~iP−1.M_i= M_iP^-1. (3.9) By substituting (3.9) in (3.6) we obtain an estimation of the adaECE accuracy term of the clean data AiA_i as a function of the confusion matrix of the noisy data M~i M_i and the label noise matrix P: A^i=Tr(Mi)=Tr(M~iP−1). A_i= Tr(M_i)= Tr( M_iP^-1). (3.10) We note that, by applying this derivation to the case of uniform noise (3.5), we obtain: A~i A_i =∑rM~i(r,r)=∑r(MiP)(r,r)=∑s,rMi(r,s)P(s,r) = _r M_i(r,r)= _r(M_iP)(r,r)= _s,rM_i(r,s)P(s,r) =∑s(Mi(s,s)(1−ϵ)+∑r≠sMi(r,s)(ϵk−1))=(1−ϵ)Ai+ϵk−1(1−Ai). = _s(M_i(s,s)(1-ε)+ _r≠ sM_i(r,s)( εk-1))=(1-ε)A_i+ εk-1(1-A_i). (3.11) This coincides with the direct derivation of binwise average accuracy for the case of uniform noise. Algorithm 2 Noisy Temperature Scaling (NTS) - General Noise Matrix input: A validation set (x1,y~1),…,(xn,y~n)(x_1, y_1),...,(x_n, y_n) whose labels are corrupted by a noise matrix P. - Feed each xtx_t into the classifier network to produce class distribution pt1,…,ptkp_t1,...,p_tk. Compute the confidence values p^t=maxjptj p_t= _jp_tj and the predictions y^t=argmaxjptj y_t= _jp_tj. - Order the points based on their confidence and divide them into equal-sized sets B1,…,BmB_1,...,B_m. - Find T that minimizes the Noisy-adaECE score: Noisy-adaECE(T)=1m∑i=1m|max(0,min(1,Tr(M~iP−1)))−Ci(T)|Noisy -adaECE(T)= 1m _i=1^m | (0, (1, Tr( M_iP^-1)))-C_i(T) | s.t. Ci(T)=1|Bi|∑t∈Biexp(log(p^t/T))∑l=1kexp(log(ptl/T))andM~i(r,s)=1|Bi|∑t∈Biy^t=r,y~t=sC_i(T)= 1|B_i| _t∈ B_i ( ( p_t/T)) _l=1^k ( (p_tl/T)) 28.45274pt and 28.45274pt M_i(r,s)= 1|B_i| _t∈ B_i 1_\ y_t=r, y_t=s\ output: T^=argminNoisy-adaECE(T) T= -adaECE(T) The Noise-adaECE in the case of a general noise matrix P is defined by: Noisy-adaECE(P)=1m∑i=1m|Tr(M~iP−1)−Ci|,Noisy -adaECE(P)= 1m _i=1^m | Tr( M_iP^-1)-C_i |, (3.12) such that M~ M is the confusion matrix of the noisy validation set data (3.7) and we used (3.10) to estimate the binwise average clean data accuracy. To apply TS calibration in the case of a general noise model, we need to find a temperature T that minimizes the Noise-adaECE expression (3.12). In the case where the label noise matrix P is not known, there is a plethora of methods for estimating P without accessing clean labels [70, 52, 117]. The Noisy Temperature Scaling (NTS) for the case of a general noise matrix, is summarized in Algorithm Box 2. Figure 3.1 illustrates the entire pipeline composed of the noisy label training followed by the noisy label calibration process. 3.3 Network Training with Noisy Labels To achieve robust confidence calibration, it is crucial to first train the network in a manner that accounts for label noise. Various approaches have been developed to mitigate the adverse effects of noisy labels, ranging from reweighting and selecting reliable samples to modifying the network architecture to model label noise explicitly. Among these, a particularly effective strategy involves estimating the label noise matrix and incorporating it into the training process (see Background Section 2.4 for details). Our approach follows the provably consistent method proposed by Li et al. [52], where a noise adaptation layer is introduced to learn the label transition probabilities. This method optimizes a loss function that balances prediction fidelity with a regularization term enforcing the structural properties of the estimated noise matrix. Once training is complete, the noise adaptation layer is removed, allowing the network to predict clean labels while retaining an estimated noise matrix as a by-product. This estimated noise matrix plays a key role in our confidence calibration procedure. By leveraging noise-robust training, we ensure that the network maintains reliable predictive confidence even when exposed to datasets with significant label noise. In the following sections, we detail how this estimated noise matrix enhances the calibration process. Figure 3.1 provides a flow diagram of our method in conjunction with model training. A noise-robust training method [52] is applied to the noisy training data, yielding a trained network and an estimation of the noise matrix. Then, given a noisy validation set, we apply our NTS method to calibrate the network confidence. 3.4 Experiments We implemented the proposed NTS noisy calibration method on various medical imaging classification tasks to evaluate its performance. We share our code for reproducibility 111https://github.com/cobypenso/noisy_calibration. The experimental setup included the following medical imaging classification datasets: • ChestX-ray14 [96]: A huge dataset that contains 112,120 frontal-view X-ray images of 30,805 unique patients of size 1024×10241024× 1024, individually labeled with up to 14 different thoracic diseases. The original dataset is multi-label. The problem is treated as a multi-class task by choosing the samples containing only one annotated positive label or the ”No-finding” case without any positive label. More than 60% of the images belong to this class, which makes the dataset highly imbalanced. We used a train/validation/test split of 89,696/11,212/11,212 images. We used a balanced variant of the dataset denoted by ChestX-ray14+bal which only included images with exactly one annotated positive label. • HAM10000 [91]: This dataset contains 10,015 dermatoscopic images of size 800×600800× 600. Cases include a representative collection of 7 diagnostic categories in the realm of pigmented lesions. We used a train/validation/test split of 8,013/1,001/1,001 images. • PathMNIST [108]: A dataset that contains 97,176 images of Colon Pathology with nine classes. The images’ size is 28×2828× 28. Here, we used a train/validation/test split of 89,996/3,590/3,590 images. Each dataset was fine-tuned on pre-trained ResNet-18, ResNet-50 [35], and DenseNet-121 [39] networks. The models were taken from the PyTorch site 222https://pytorch.org/vision/stable/models.html. These network architectures were selected because of their widespread use in classification problems. The last fully-connected layer output size of each was adjusted to fit the corresponding number of classes for each dataset. All the models were fine-tuned using the Adam optimizer [47]. (a) ChestX-ray14 (b) HAM-10000 (c) PathMNIST Figure 3.2: Comparative calibration results on several datasets that were trained with ResNet-50. The top row shows the ECE results on the (clean) test set. The bottom row shows the optimal temperature found on the noisy validation set. (a) ResNet-18 (b) ResNnet-50 (c) Densenet-121 Figure 3.3: Standard deviation of ECE scores on the test set after the NTS calibration as a function of the size of the noisy validations set. Table 3.1: Adaptive ECE for top-1 predictions (in %) using 15 bins. on various medical imaging classification datasets and models with different calibration methods with a noise level ϵ=0.2ε=0.2. The lowest is highlighted in bold. Dataset Architecture Acc (%) Uncalibrated TSclean VSnoisy MSnoisy TSnoisy NTS ChestX-ray14 ResNet-18 65.80 ± 0.01 1.92±0.431.92± 0.43 1.52±0.191.52± 0.19 12.25±0.1712.25± 0.17 12.27±0.1712.27± 0.17 11.90 ± 0.46 1.45 ± 0.15 ResNet-50 65.84±0.0265.84± 0.02 4.59±0.624.59± 0.62 1.63±0.251.63± 0.25 12.26±0.2112.26± 0.21 12.22±0.1812.22± 0.18 11.81±0.2811.81± 0.28 1.52 ± 0.19 Densenet-121 65.67±0.0165.67± 0.01 3.61±0.753.61± 0.75 1.49±0.101.49± 0.10 12.27±0.1812.27± 0.18 12.24±0.1612.24± 0.16 12.15 ± 0.49 1.50 ± 0.06 HAM-10000 ResNet-18 88.16±0.1788.16± 0.17 4.38±1.524.38± 1.52 3.12±0.743.12± 0.74 19.75±2.4019.75± 2.40 17.17±0.9117.17± 0.91 14.22±5.8814.22± 5.88 3.67 ± 0.59 ResNet-50 91.06±0.0191.06± 0.01 3.51±0.693.51± 0.69 1.98±0.341.98± 0.34 19.95±0.7319.95± 0.73 18.03±0.5418.03± 0.54 18.75 ± 1.49 3.04 ± 1.08 Densenet-121 89.43±0.1889.43± 0.18 3.75±1.533.75± 1.53 2.67±0.112.67± 0.11 18.85±1.2318.85± 1.23 17.56±1.0917.56± 1.09 17.50±2.0417.50± 2.04 2.74 ± 0.68 PathMNIST ResNet-18 84.68±0.0584.68± 0.05 13.14 ± 0.51 2.19±0.202.19± 0.20 23.19±0.5723.19± 0.57 22.99±0.0522.99± 0.05 11.94 ± 1.30 2.37 ± 0.52 ResNet-50 85.97±0.0485.97± 0.04 11.50 ± 0.44 2.23±0.352.23± 0.35 21.47±3.7121.47± 3.71 27.35±3.9627.35± 3.96 13.00±0.3213.00± 0.32 2.42 ± 0.48 Densenet-121 86.54±0.0586.54± 0.05 11.12±0.4311.12± 0.43 1.83±0.491.83± 0.49 25.47±0.7125.47± 0.71 25.58±0.4225.58± 0.42 12.31 ± 1.00 2.55 ± 0.35 Table 3.2: ECE for top-1 predictions (in %) using 15 bins. on various medical imaging classification datasets and models with different calibration methods with a noise level ϵ=0.2ε=0.2. The lowest is highlighted in bold. Dataset Architecture Uncalibrated TSclean VSnoisy MSnoisy TSnoisy NTS ChestX-ray14 ResNet-18 1.93±0.491.93± 0.49 1.24±0.141.24± 0.14 12.25±0.1712.25± 0.17 12.27±0.1812.27± 0.18 11.92 ± 0.46 1.21 ± 0.12 ResNet-50 4.51±0.304.51± 0.30 1.35±0.321.35± 0.32 12.24±0.2212.24± 0.22 12.21±0.2012.21± 0.20 11.84±0.2811.84± 0.28 1.26 ± 0.29 Densenet-121 3.71±0.803.71± 0.80 1.22±0.191.22± 0.19 12.25±0.1812.25± 0.18 12.24±0.1712.24± 0.17 12.10 ± 0.49 1.26 ± 0.06 HAM-10000 ResNet-18 2.72±1.412.72± 1.41 0.79±0.350.79± 0.35 18.84±1.6118.84± 1.61 14.87±4.0114.87± 4.01 12.53±6.5112.53± 6.51 2.52 ± 0.69 ResNet-50 1.69±0.611.69± 0.61 0.61±0.120.61± 0.12 19.00±0.1419.00± 0.14 17.50±0.8817.50± 0.88 16.50 ± 2.41 1.41 ± 1.10 Densenet-121 3.04±1.673.04± 1.67 1.23±0.341.23± 0.34 18.39±1.7418.39± 1.74 17.13±1.2417.13± 1.24 16.23±2.1316.23± 2.13 1.72 ± 0.74 PathMNIST ResNet-18 13.12 ± 0.52 1.89±0.111.89± 0.11 23.22±0.5923.22± 0.59 23.00±0.0623.00± 0.06 11.91 ± 1.35 2.09 ± 0.46 ResNet-50 11.50 ± 0.52 2.30±0.342.30± 0.34 21.31±3.9321.31± 3.93 27.14±4.7527.14± 4.75 12.90±0.1112.90± 0.11 2.52 ± 0.49 Densenet-121 11.02±0.4011.02± 0.40 1.82±0.541.82± 0.54 25.29±0.6725.29± 0.67 25.58±0.4425.58± 0.44 12.02 ± 0.79 2.58 ± 0.46 Table 3.3: Adaptive ECE for top-1 predictions (in %) using 15 bins (with the lowest in bold) on various medical imaging classification datasets and models with different calibration methods with varying noise levels. Model training and noise matrix estimation used [52]. Dataset Noise level (%) Acc (%) Uncalibrated TSclean NTS(P) TSnoisy NTS(P P) ChestX-ray14-bal 0 44.37±0.1244.37± 0.12 26.21±0.5626.21± 0.56 2.88±0.482.88± 0.48 55 44.10±0.0844.10± 0.08 25.52±0.7925.52± 0.79 3.18±0.443.18± 0.44 3.20 ± 0.46 5.91±0.465.91± 0.46 5.27 ± 1.41 1010 43.76±0.3143.76± 0.31 23.85±1.0223.85± 1.02 3.41±0.303.41± 0.30 3.40 ± 0.54 7.35±1.187.35± 1.18 6.08 ± 1.52 2020 43.02±0.2843.02± 0.28 21.34±0.3321.34± 0.33 3.30±0.363.30± 0.36 3.33 ± 0.44 10.48±0.6310.48± 0.63 6.43 ± 0.96 ChestX-ray14 0 65.13±0.0965.13± 0.09 21.50±0.1821.50± 0.18 5.70±0.635.70± 0.63 5 64.95±0.0564.95± 0.05 18.71±0.2018.71± 0.20 4.46±0.674.46± 0.67 4.29 ± 0.44 5.86 ± 0.49 12.8±1.3412.8± 1.34 10 64.76±0.3664.76± 0.36 19.04±2.8519.04± 2.85 4.34±0.554.34± 0.55 4.42 ± 0.33 6.81 ± 0.64 9.65±0.789.65± 0.78 20 64.71±0.0964.71± 0.09 18.22±3.2618.22± 3.26 4.64±0.544.64± 0.54 4.64 ± 0.54 12.02 ± 1.74 7.28 ± 1.75 HAM-10000 0 90.94±0.7890.94± 0.78 3.95±0.673.95± 0.67 2.48±1.022.48± 1.02 5 90.64±0.8390.64± 0.83 3.47±1.163.47± 1.16 3.19±1.033.19± 1.03 2.98 ± 0.49 4.34±0.574.34± 0.57 3.41 ± 1.21 10 90.16±0.8190.16± 0.81 2.95±0.322.95± 0.32 2.70±0.392.70± 0.39 2.71 ± 0.29 8.08±1.128.08± 1.12 2.76 ± 0.31 20 89.79±1.1189.79± 1.11 4.78±0.674.78± 0.67 3.56±0.723.56± 0.72 4.44 ± 0.97 19.12±1.2819.12± 1.28 6.52 ± 2.35 PathMNIST 0 85.33±0.9985.33± 0.99 4.52±0.754.52± 0.75 1.39±0.611.39± 0.61 5 85.30±0.6785.30± 0.67 3.71±0.493.71± 0.49 1.73±0.281.73± 0.28 1.65 ± 0.26 4.08±0.374.08± 0.37 1.86 ± 0.37 10 85.26±0.5585.26± 0.55 3.73±0.713.73± 0.71 1.67±0.621.67± 0.62 1.56 ± 0.59 7.09±0.617.09± 0.61 2.44 ± 0.92 20 84.46±0.5784.46± 0.57 3.36±0.633.36± 0.63 1.73±0.161.73± 0.16 2.33 ± 0.82 12.30±1.3412.30± 1.34 2.63 ± 0.19 Table 3.4: ECE for top-1 predictions (in %) using 15 bins (with the lowest in bold) on various medical imaging classification datasets and models with different calibration methods with varying noise levels. Model training and noise matrix estimation used [52]. Dataset Noise level (%) Uncalibrated TSclean NTS(P) TSnoisy NTS(P^) P) ChestX-ray14-bal 0 26.21±0.5626.21± 0.56 2.63±0.322.63± 0.32 5 25.53±0.8025.53± 0.80 3.03±0.373.03± 0.37 3.08±0.383.08± 0.38 4.89±0.824.89± 0.82 3.83 ± 0.36 10 23.87±1.0123.87± 1.01 3.00±0.613.00± 0.61 2.92±0.462.92± 0.46 7.49±0.737.49± 0.73 4.13 ± 0.39 20 21.37±0.3521.37± 0.35 3.33±0.373.33± 0.37 3.26±0.283.26± 0.28 9.78±0.989.78± 0.98 4.94 ± 0.19 ChestX-ray14 0 21.05±0.5521.05± 0.55 5.86±0.825.86± 0.82 5 18.73±0.1818.73± 0.18 4.43±0.714.43± 0.71 4.25 ± 0.49 5.74 ± 0.68 12.9±1.4412.9± 1.44 10 19.11±2.8519.11± 2.85 4.21±0.754.21± 0.75 4.30 ± 0.62 7.06 ± 0.22 10.0±1.2710.0± 1.27 20 16.24±2.0216.24± 2.02 4.95±1.014.95± 1.01 4.98 ± 1.08 11.6±0.9311.6± 0.93 5.62 ± 0.79 HAM-10000 0 1.83±0.271.83± 0.27 1.19±0.591.19± 0.59 5 1.54±0.381.54± 0.38 1.31±0.381.31± 0.38 2.03 ± 0.94 4.10±0.874.10± 0.87 1.31 ± 0.33 10 1.53±0.131.53± 0.13 1.41±0.431.41± 0.43 2.35 ± 0.47 6.61±1.566.61± 1.56 1.40 ± 0.68 20 3.49±0.803.49± 0.80 0.94±0.300.94± 0.30 3.30 ± 1.68 12.3±8.8212.3± 8.82 5.37 ± 2.02 PathMNIST 0 4.48±0.764.48± 0.76 1.58±0.041.58± 0.04 5 4.00±0.014.00± 0.01 1.40±0.001.40± 0.00 1.65 ± 0.01 3.91±0.003.91± 0.00 1.34 ± 0.01 10 3.82±0.453.82± 0.45 1.73±0.371.73± 0.37 1.65 ± 0.17 6.41±0.806.41± 0.80 1.68 ± 0.39 20 3.42±0.443.42± 0.44 1.97±0.511.97± 0.51 2.42 ± 0.41 12.5±1.3612.5± 1.36 2.57 ± 0.29 The compared calibration methods. (1) TSnoisy - A standard TS that was applied to the validation set with noisy labels. (2) VSnoisy and MSnoisy - Vector Scaling (VS) and Matrix Scaling (MS) calibration methods [33] were applied to the validation set with noisy labels. (3) The proposed Noisy Temperature Scaling (NTS). (4) TSclean - Temperature Scaling that was applied to the clean validation set. The main comparison was between the TSnoisy and NTS that were applied to the noisy validation dataset with the same label corruption. TSclean is served as an upper bound on the performance of NTS. We are not aware of any other noise robust calibration strategies. For each method, we report the adaECE and ECE scores (computed using 15 bins) on the test set. Although adaECE was used as the objective function in our algorithm and is a more robust calibration measure, ECE is still a standard measure to evaluate calibration results, so we also used it to compare our calibration results to previous studies. Results of calibration with uniform noise. We first applied the calibration methods to a validation set corrupted by noise with noise level ϵ=0.2ε=0.2. Table 3.1 reports the calibration results using adaECE. We report the mean and standard deviation over 3 different trained models and noise labels sampling. The results indicate that when the validation set contains noise, the standard TSnoisy method fails to properly calibrate the network and may make the calibration worse. The same behavior was observed for VSnoisy and NTSnoisy. By contrast, the proposed NTS method produces calibration results that are similar to those obtained by the TSclean method that has access to the clean data. we tested the assumption that (in the case of noise level ϵ=0.2ε=0.2) NTS has no effect compared to TSnoisy and its p-value was 0.005. Note that since NTS works well, the temperatures computed by NTS and TSclean are similar so that the performance of NTS can even be slightly better than TSclean on the test set. In this experiment we assume that the noise level is known and the network was trained using clean data. Below we show the results of end-to-end experiments where the noise matrix is estimated during the training procedure. We next show that our noise-tolerant NTS method works well across various levels of noise. Figure 3.2 displays the adaECE calibration results for various classification tasks with noise levels ϵε that ranged from 0% to 40%. While higher noise levels negatively impacted the calibration of TSnoisy, NTS demonstrated resilience even in the presence of high levels of noise. Figure 3.2 also illustrates the optimal temperature obtained for each experiment. It shows that the optimal temperature found by TSnoisy increased in a linear manner with the noise level. The presence of noisy labels led to an underestimation of accuracy in each adaECE bin, causing the TSnoisy algorithm to incorrectly conclude that the network was over-confident, thus resulting in an aggressive calibration using a high temperature. Our proposed NTS algorithm avoids this misinterpretation, by computing an accurate estimation of the network accuracy. It is worth mentioning that typical neural network training techniques can handle a limited amount of incorrectly labeled data in the training set, typically less than 10% (see e.g. [105]). However, TSnoisy does not perform well under these conditions. Our method is justified by the law of large numbers (see (3.1)). Therefore, we expected a correlation between the size of the validation set used for calibration and the stability of the NTS method when dealing with different noise samples. To verify this, we selected ChestX-ray14 as a dataset from our experiments and generated various validation set sizes (using the sklearn split function). For each sample size, and noise level ϵε, we created multiple noisy versions of the validation set. For each noisy version we applied the NTS algorithm and computed the adaECE score on the test set. Finally, we computed the standard deviation (STD) of all the adaECE scores. The results, as shown in Figure 3.3, indicate that as the validation set size increased, the standard deviation of the NTS algorithm decreased, thus resulting in more stable model accuracy estimates for each bin. End-to-End training and calibration with Noisy labels. In the following experiment, we combined our noise-robust calibration method with the method for network training using training data with noisy labels described in the previous section [52]. Thus, we simulated a real-world scenario in which the labels of both the training set and the validation set had the same noise level. Making it even more realistic, we assumed here that the label noise matrix P is unknown and was estimated during the training step [52]. We trained models for the three medical-imaging datasets (ChestX-ray14, HAM-10000, PathMNIST), using training data with label noise level ϵ∈0%,5%,10%,20%ε∈\0\%,5\%,10\%,20\%\. In addition to training the network, we also estimated the noise matrix P, denoted as P P. We report the results of two variants of our calibration method NTS; namely, NTS(P) where P is known and NTS(P P) where P is estimated during the training phase from the noisy training set. Tables 3.3 and 3.4 report the calibration results using adaECE and ECE respectively. The results show that the noisy labels corrupted the calibration of the network, whereas our method, even in the case where P was estimated achieved a much better calibration result. We can also observe that the network training procedure is much more resilient to label noise compared to calibration. A noise level of 20% only results in a slight degradation of the network’s accuracy. However, the calibration process does not provide any assistance and instead significantly reduces the network’s calibration. The ChestX-ray14 task contains a dominant ”no-finding” class in addition to the 14 pathology-related classes and is heavily unbalanced. In that case, when the true noise matrix P was used we obtained good calibration results but when the estimated noise matrix was used, the calibration procedure failed. The reason for that is that in ChestX-ray14, the classes are unbalanced, and in that case there is currently no reliable way to estimate the noise matrix. When The No-finding class is removed (dataset ChestX-ray1-bal) the dataset is relatively balanced and the training algorithm manages to estimate the noise matrix. Finally, results satisfy the statistical significance requirement by a large margin, with an average p-value across the 4 different datasets and ϵ=0.2ε=0.2 of P−value=0.00508P-value=0.00508 Calibration results for a general noise matrix. So far, we conducted experiments with uniform noise. Next, we evaluate our NTS method on a validation set that was corrupted by a general label noise matrix P, which allows all types of noise to affect the labels. We tested four cases that were different in terms of their label noise matrix P: (1) Symmetric noise where labels are converted to any other label with equal probability, (2) Neighbor noise where labels can only be flipped to adjacent labels, (3) Decreasing noise where labels are flipped to adjacent labels with decreasing probability and finally (4) Random noise - a general noise matrix whose off-diagonal probabilities were randomly selected. The same noise model was applied to the training and validation sets. Figure 3.4 shows the noise transition matrices and the corresponding adaECE calibration results on the test set for the baseline calibration method TSnoisy and NTS. We report the results of two variants of our calibration method NTS; namely, NTS(P) where the noise matrix P is known and NTS(P P) where P is estimated during the training phase from the noisy training set. We also report the results of TSclean where calibration was applied to the clean validation set. In all cases, regardless of the noise matrix, NTS achieved calibration results that were on par with the results obtained by applying TSclean on the clean validation set and much better than the results obtained by TSnoisy. (1) symmetric noise (2) neighbor noise (3) decreasing noise (4) random noise Figure 3.4: Calibration results for several noise transition matrices on ChestX-ray14-bal and ResNet-18. In each example we show the noise transition matrix (Left) and the adaECE measure on the test set for the compared calibration methods (right). Figure 3.5: Calibration performance, measured by adaECE (left), and the corresponding temperature (right) as a function of the noise ratio used by the NTS algorithm. Sensitivity to the noise estimation accuracy. Our NTS method is based on estimating the noise level, which is carried out as part of the network training. Crucially, the NTS method must not be highly sensitive to variations in the noise estimation, i.e., there is a range of values around the correct noise level in which NTS performs well. In the following experiment, we corrupted the validation set using noise level true-ϵ=0.2ε=0.2. Figure 3.5 shows the adaECE and the temperature obtained by the NTS as a function of the actual ϵε used in the calibration process. We denote this method NTS(ϵ)(ε). We also report the results of NTS(est-ϵε) and NTS(true-ϵε), which are variants of NTS using the estimated noise matrix and the exact noise matrix, respectively. The experiments were run on the test set of PathMNIST with the ResNet18 architecture. The results show that there is indeed a large range of ϵε values in which NTS performs much better than TSnoisy. Chapter 4 Conformal Prediction under Noisy Labels In this chapter, we address the application of Conformal Prediction (CP) methods in classification networks under the challenge of noisy labels. Conformal Prediction is a powerful tool for safety-critical applications, such as medical imaging, where maintaining a predefined level of certainty in model predictions is essential. The ability to generate a set of possible class candidates, ensuring the true class is included with high confidence, provides valuable support for clinical practitioners by narrowing down the possible diagnoses while controlling the risk of mistakes. We introduce novel methods that enhance the robustness of CP under label noise, demonstrating their effectiveness on medical and natural image datasets. 4.1 Problem Statement In machine learning for safety-critical applications, the model must only make predictions it is confident about. One way to achieve this is by returning a (hopefully small) set of possible class candidates that contain the true class with a predefined level of certainty. This is a natural approach for medical imaging, where safety is of the utmost importance and a human makes the final decision. This allows us to aid the practitioner, by reducing the number of possible diagnoses he needs to consider, with a controlled chance of mistake. The general approach to return a prediction set without any assumptions on the data distribution (besides i.i.d. samples) is called Conformal Prediction (CP) [1, 93]. It creates a prediction set with the guarantee that the probability of the correct class being within this set meets or exceeds a specified confidence threshold. The goal is to return the smallest set possible while maintaining the confidence level guarantees. Recently, with the growing use of neural network systems in safety-critical applications such as medical imaging, CP has become an important calibration tool [57, 58, 67]. We note that CP is a general framework rather than a specific algorithm. The most common approach builds the prediction set using a conformity score, and different algorithms mostly vary in terms of how the conformity score is defined. When dealing with conformal predictions, a critical challenge arises in applications such as medical imaging due to label noise. In these domains, datasets frequently contain noisy labels stemming from ambiguous data that can confuse even clinical experts. Furthermore, physicians may disagree on the diagnosis for the same medical image, leading to inconsistencies in the ground truth labeling. Noisy labels also occur when applying differential privacy techniques to overcome privacy issues [29]. While significant efforts have been devoted to the problem of noise-robust network training [88, 105], the challenge of calibrating the models has only recently begun to receive attention. In this chapter, we tackle the challenge of applying CP to classification networks using a validation set with noisy labels. Einbinder et al. [21] suggested ignoring label noise and simply applying the standard CP algorithm on the noisy labeled validation set. This strategy results in large prediction sets especially when there are many classes. The most related studies to ours are [86, 13] which present a noisy CP algorithm using conservative coverage guarantee bounds which can result in large prediction sets, failing in classification tasks with many classes. In the following sections, we present two novel approaches and algorithms for CP on noisy data. The first approach propose a new score that is robust to label noise. The second approach propose to estimate the threshold in the presence of noisy labels, similar to Sesia et al. [86] and [13], but yields an effective coverage guarantee even in tasks with a large number of classes in the uniform noise setting. We applied the algorithms to several standard medical and scenery imaging classification datasets and show that the latter method outperformed previous methods by a significant margin and achieved results comparable to those obtained by using a clean validation set. The greatest value and novelty of our approach lies in tasks with many classes, such as CIFAR-100, TinyImageNet, and ImageNet, where all other methods fail. The study described in this chapter was published in [74] and [75]. 4.2 Score That Is Robust to Label Noise Let (x1,y~1),…,(xn,y~n)(x_1, y_1),...,(x_n, y_n) be a noisy validation set where the labels were corrupted by uniform noise with a noise level ϵε. We aim to find a noise-robust conformal score that can be applied to the noisy labeled data. Since yty_t is not observed, we cannot directly compute the score S(xt,yt)S(x_t,y_t). Instead, we can estimate it using its noisy version y~t y_t: (S(xt,yt)|y~t)=∑i=1kp(yt=i|y~t)S(xt,i).E(S(x_t,y_t)| y_t)= _i=1^kp(y_t=i| y_t)S(x_t,i). (4.1) Assuming a non-informative uniform prior on the correct label yty_t, i.e., p(yt=i)=1/kp(y_t=i)=1/k), we obtain: p(yt=i|y~t)=(1−ϵ)1y~t=i+ϵk.p(y_t=i| y_t)=(1\!-\!ε)1_\ y_t=i\+ εk. (4.2) Substituting (4.2) in (4.1), yields an estimate S^(xt,y~t,ϵ) S(x_t, y_t,ε) of the noise-free score: S^(xt,y~t,ϵ)=(S(xt,yt)|y~t)=(1−ϵ)S(xt,y~t)+ϵS(xt) S(x_t, y_t,ε)=E(S(x_t,y_t)| y_t)=(1\!-\!ε)S(x_t, y_t)+ε S(x_t) (4.3) s.t. S(xt)=1k∑i=1kS(xt,i)S(x_t)= 1k _i=1^kS(x_t,i). Note that to obtain the score estimation S^(xt,y~t,ϵ) S(x_t, y_t,ε), we need to either know the noise level ϵε or estimate it from the noisy-label data. We elaborate further on this issue in the next section. Table 4.1: Conformal Prediction methods for validation sets with noisy labels. Given an image x, y is the true label, y y is its noisy version and ϵε is the noise level. S is a conformal score and S S is its noise robust variant. Stage CP (Oracle) Noisy-CP [21] NRESCP NRSCP Learning phase S(x,y)→qS(x,y) q S(x,y~)→qnoiseS(x, y) q_noise S^(x,y~,ϵ)→qϵ S(x, y,ε) q_ε S^(x,y~,ϵ)→qϵ S(x, y,ε) q_ε Inference phase y|S(x,y)≤q\y|S(x,y)≤ q\ y|S(x,y)≤qnoise\y|S(x,y)\!≤\!q_noise\ y|S^(x,y,ϵ)≤qϵ\y| S(x,y,ε)≤ q_ε\ y|S(x,y)≤qϵ\y|S(x,y)≤ q_ε\ We next apply the CP algorithm on the estimated conformal scores and set qϵq_ε to be the (1−α)(1\!-\!α) quantile of S^(x1,y~1,ϵ),…,S^(xn,y~n,ϵ) S(x_1, y_1,ε),..., S(x_n, y_n,ε). According to the general CP theory, the prediction set of a given test sample x is: C^ϵ(x)=y|S^(x,y,ϵ)≤qϵ=y|S(x,y)≤qϵ−ϵS(x)1−ϵ. C_ε(x)=\y\,|\, S(x,y,ε)≤ q_ε\=\y\,|\,S(x,y)≤ q_ε-ε S(x)1\!-\!ε\. (4.4) Let x be a test point and y and y~ y be its true label and noisy label respectively. The general CP theory guarantees that 1−α≤p(y~∈C^ϵ(x))1\!-\!α≤ p( y∈ C_ε(x)). This guarantee, however, is for the noisy labeled data. Assume that for every x the order of the network class predictions p(y=i|x;θ)p(y=i|x;θ) coincides with the order of the true probabilities p(y=i|x)p(y=i|x). In that case, the same argument that appears in [21] for Noisy-CP, implies a coverage guarantee in the noise-free case. However, the prediction set obtained by the estimated score (4.4), is usually still too large. In the case of noisy labels, during the CP learning phase, we need to estimate the score of the correct class. However, to form the prediction set at test time we need to compute the scores of all the possible classes. Hence, in a way similar to the Noisy-CP algorithm, Algorithm 3 Noise-Robust Score Conformal Prediction (NRSCP) 1:Input: A conformal score S(x,y)S(x,y), a coverage level 1−α1\!-\!α and a validation set (x1,y1),…,(xn,yn)(x_1,y_1),...,(x_n,y_n), s.t. the labels are corrupted by uniform noise with parameter ϵε. 2:Compute the estimated scores: st=S^(xt,yt,ϵ)=(1−ϵ)S(xt,yt)+ϵk∑i=1kS(xt,i),t=1,…,ns_t= S(x_t,y_t,ε)=(1-ε)S(x_t,y_t)+ εk _i=1^kS(x_t,i), 14.22636ptt=1,...,n 3:Set q to be the ⌈(n+1)(1−α)/n⌉ (n+1)(1\!-\!α)/n quantile of s1,…,sns_1,...,s_n. 4:The prediction set of a test sample x is C(x)=y|S(x,y)<qC(x)=\y\,|\,S(x,y)<q\. we can thus construct the prediction set of a test sample x using the exact score S(x,y)S(x,y) instead of the estimated score S^(x,y~,ϵ) S(x, y,ε): Cϵ(x)=y|S(x,y)≤qϵ.C_ε(x)=\y\,|\,S(x,y)≤ q_ε\. (4.5) We denote the algorithm variants based on Eqs. (4.4) and (4.5) as the Noise Robust Estimated Score CP (NRESCP) and the Noise Robust Score CP (NRSCP) respectively. The only difference between them lies in how the test-time prediction set is formed. The various noisy label CP methods discussed above are summarized in Table 4.1. We empirically show below that NRSCP satisfies the coverage requirement and yields an average size that is much smaller than the one obtained by NRESCP. The NRSCP is summarized in Algorithm Box 3. We next analyze the proposed noise robust conformal score. It is easy to verify that the prediction set obtained by NRSCP is smaller than the one obtained by NRESCP, i.e., Cϵ(x)⊂C^ϵ(x)C_ε(x)⊂ C_ε(x) if and only if S(x)≤qϵS(x)≤ q_ε. In the case of HPS, S(x)=1k∑i(1−p(y=i|x))=(k−1)/kS(x)= 1k _i(1-p(y=i|x))=(k-1)/k and therefore S^(x,y~,ϵ)=(1−ϵ)S(x,y~)+ϵk−1k S(x, y,ε)=(1-ε)S(x, y)+ε k-1k. This implies that qϵ=(1−ϵ)qnoise+ϵk−1kq_ε=(1-ε)q_noise+ε k-1k such that qnoiseq_noise and qϵq_ε are the thresholds computed by Noisy-CP [21] and NRESCP (4.4) respectively. It is easy to verify that Noisy-CP and NRESCP yield the same prediction set, i.e. y|S(x,y)≤qnoise=y|S^(x,y,ϵ)≤qϵ\y|S(x,y)≤ q_noise\=\y| S(x,y,ε)≤ q_ε\. In the case of the APS conformal score, it is easy to see that: (p^+1)/2≤S(x)≤(p^+k−1)/k( p+1)/2≤ S(x)≤( p+k-1)/k, s.t. p^=maxip(y=i|x;θ) p= _ip(y=i|x;θ) is the network confidence on its single-class prediction. Hence, when the prediction sets obtained by NRSCP are smaller than those obtained by NRESCP, the prediction confidence satisfies (p^+1)/2≤S(x)≤qϵ( p+1)/2≤ S(x)≤ q_ε. 4.3 Procedure of Threshold Estimation That Is Robust to Label Noise In the previous section we presented a new conformal score that takes into account the noise rate making it robust to the label noise. Next, we tackle the problem of conformal prediction with noisy labels from a different angle by defining a conformal prediction procedure that is robust to noisy labels along with theoretic and coverage guarantees. Here we show how, given a simple noise model and a known noise level, we can get the correct CP threshold based on noisy data. We will generalize this beyond the simple noise model in the following section. Consider a network that classifies an input x into k pre-defined classes. Given a conformity score S(x,y)S(x,y) and a specified coverage 1−α1-α, the goal of the conformal prediction algorithm is to find a minimal q such that p(y∈Cq(x))≥1−αp(y∈ C_q(x))≥ 1-α. Let (x1,y~1),…,(xn,y~n)(x_1, y_1),...,(x_n, y_n) be a validation set with noisy labels and let yiy_i be the unknown correct label of xix_i. Let si=S(xi,y~i)s_i=S(x_i, y_i) be the conformity score of (xi,y~i)(x_i, y_i). We assume that the label noise follows a uniform distribution, where with a probability of ϵε, the correct label is replaced by a label that is randomly sampled from the k classes: p(y~=j|y=i)=i=j(1−ϵ)+ϵk.p( y=j|y=i)= 1_\i=j\(1-ε)+ εk. (4.6) Uniform noise is relevant, for example, when applying differential privacy techniques to overcome privacy issues [29]. In that setup the noise level ϵε is usually known. In other applications such as medical imaging, where the noise parameter ϵε is not given, it can be estimated with sufficient accuracy from the noisy-label data during training [117, 52, 54]. We can write y~ y as y~=(1−z)⋅y+z⋅u y=(1-z)· y+z· u, s.t. u is a random label uniformly sampled from 1,…,k\1,...,k\ and z is a binary random variable (p(z=1)=ϵ)(p(z=1)=ε) indicating whether the label of the sample (x,y)(x,y) was replaced by a random label or not. For each candidate threshold, q denote: Fc(q)=p(y∈Cq(x)),Fn(q)=p(y~∈Cq(x)),F^c(q)=p(y∈ C_q(x)), 28.45274ptF^n(q)=p( y∈ C_q(x)), Fr(q)=p(u∈Cq(x)),F^r(q)=p(u∈ C_q(x)), where FcF^c, FnF^n, and FrF^r represent the clean, noisy and random labels. Note as well that each one is the CDF of the appropriate conformal score function, e.g., Fc(q)=p(y∈Cq(x))=p(S(x,y)≤q)F^c(q)=p(y∈ C_q(x))=p(S(x,y)≤ q). It is easily verified that Fn(q)=p(z=0)Fc(q)+p(z=1)Fr(q)F^n(q)=p(z=0)F^c(q)+p(z=1)F^r(q) (4.7) =(1−ϵ)Fc(q)+ϵFr(q).=(1-ε)F^c(q)+ε F^r(q). For each value q, we can estimate Fn(q)F^n(q) from the noisy validation set: F^n(q)=1n∑iy~i∈Cq(xi)=1n∑isi≤q. F^n(q)= 1n _i 1_\ y_i∈ C_q(x_i)\= 1n _i 1_\s_i≤ q\. (4.8) Note that q is the F^n(q) F^n(q)-quantile of s1,…,sns_1,...,s_n. Similarly we can also estimate Fr(q)F^r(q): F^r(q)=1n∑ip(ui∈Cq(xi))=1n∑i|Cq(xi)|k, F^r(q)= 1n _ip(u_i∈ C_q(x_i))= 1n _i |C_q(x_i)|k, (4.9) s.t. uiu_i is uniformly sampled from 1,…,k\1,...,k\. By substituting (4.8) and (4.9) in (4.7) we obtain an estimation of Fc(q)=p(y∈Cq(x))F^c(q)=p(y∈ C_q(x)) based on the noisy validation set and the noise level ϵε: F^c(q)=F^n(q)−ϵF^r(q)1−ϵ. F^c(q)= F^n(q)-ε F^r(q)1-ε. (4.10) For each candidate q we first compute F^n(q) F^n(q) and F^r(q) F^r(q) and then by using (4.10) obtain the coverage estimation F^c(q) F^c(q). Given a coverage requirement (1−α)(1-α), we can thus use the noisy validation set to find a threshold q such that F^c(q)=1−α F^c(q)=1-α. Note that since Fc(q)F^c(q) is monotonous, it seems reasonable to search for the threshold q using the bisection method. However, as F^c(q) F^c(q) is an approximation based on the difference between two monotonic functions, it might not be exactly monotonous. We therefore find the threshold q using an exhaustive grid search. If there are several solutions we select the largest value. (In practice selecting one of the solutions has almost no effect on the results.) We note that even with an exhaustive search the entire runtime is negligible compared to the training time. We can narrow the threshold search domain as follows: Lemma 4.3.1. For every threshold q we have: F^qn/k≤F^r(q) F^n_q/k≤ F^r(q). Proof. Denote A=i|y^i∈Cq(xi)A=\i| y_i∈ C_q(x_i)\ and B=i|1≤|Cq(xi)|B=\i|1≤|C_q(x_i)|\. Note that F^n(q)=|A|/n F^n(q)=|A|/n. |B|=∑i∈B1≤∑i∈B|Cq(xi)|≤∑i=1n|Cq(xi)|=nkF^r(q).|B|= _i∈ B1≤ _i∈ B|C_q(x_i)|≤ _i=1^n|C_q(x_i)|=nk F^r(q). Finally A⊂BA⊂ B implies that: F^n(q)=|A|/n≤|B|/n≤kF^r(q). F^n(q)=|A|/n≤|B|/n≤ k F^r(q). ∎ Theorem 4.3.2. Let q1q_1 be the (1−α)(1−ϵ)/(1−ϵk)(1\!-\!α)(1-ε)/(1- εk) quantile of s1,…,sns_1,...,s_n and let q2q_2 be the (1−α)+αϵ(1-α)+αε quantile. If q satisfies F^c(q)=1−α F^c(q)=1-α then q1≤q≤q2q_1≤ q≤ q_2. Proof. Assume q satisfies F^c(q)=1−α F^c(q)=1-α. Eq. (4.10) implies that 1−α=F^c(q)=F^n(q)−ϵF^r(q)1−ϵ1-α= F^c(q)= F^n(q)-ε F^r(q)1-ε (4.11) ⇒F^n(q)=(1−α)(1−ϵ)+ϵF^r(q). \,\, F^n(q)=(1-α)(1-ε)+ε F^r(q). Since 0≤F^r(q)≤10≤ F^r(q)≤ 1 we get that: (1−α)(1−ϵ)≤F^n(q)≤(1−α)+αϵ=F^n(q2).(1-α)(1-ε)≤ F^n(q)≤(1-α)+αε= F^n(q_2). (4.12) For every q we have F^n(q)/k≤F^r(q) F^n(q)/k≤ F^r(q) (Lemma 3.1). Hence, (1−α)(1−ϵ)≤F^n(q)(1-α)(1-ε)≤ F^n(q) (4.12) implies that (1−α)(1−ϵ)/k≤F^r(q)(1-α)(1-ε)/k≤ F^r(q). Combining this inequality with Eq. (4.11) yields a better lower bound: (1−α)(1−ϵ)(1+ϵ/k)≤F^n(q)(1-α)(1-ε)(1+ε/k)≤ F^n(q). Iterating this process yields: (1−α)(1−ϵ)(1+ϵk+(ϵk)2+…)(1-α)(1-ε) (1+ εk+ ( εk )^2+… ) =(1−α)1−ϵ1−ϵk=F^n(q1)≤F^n(q).=(1-α) 1-ε1- εk= F^n(q_1)≤ F^n(q). Finaly, F^n(q) F^n(q) is a monotonically increasing function of q which implies that q1≤q≤q2q_1≤ q≤ q_2. ∎ As an alternative to the grid search we can sort the noisy conformity scores si=S(xi,y~i)s_i=S(x_i, y_i) and look for the minimal i such that F^c(si)≥1−α F^c(s_i)≥ 1-α. In the noise-free case F^c F^c is piece-wise constant, with jumps determined exactly by the order statistics sis_i, namely, F^c(si)=i/n F^c(s_i)=i/n and thus this algorithm coincides with the standard CP algorithm. In the noisy case F^c(q) F^c(q) depends on the conformity scores of all the k classes and thus its structure is more complicated. We dub our algorithm Noise-Aware Conformal Prediction (NACP), and summarize it in Algorithm Box 4. Note that in the noise-free case (ϵ=0ε=0) the NACP algorithm coincides with the standard CP algorithm and selects q that satisfies F^c(q)=F^n(q)=1−α F^c(q)= F^n(q)=1-α, i.e., q is the 1−α1-α quantile of the validation set conformity scores. 4.3.1 Prediction Size Comparison We next compare our NACP approach analytically to Noisy-CP [21] in terms of the average size of the prediction set. Theorem 4.3.3. Let q and q~ q be the thresholds computed by the NACP and the Noisy-CP algorithms respectively. Then q≤q~q≤ q if and only if F^r(q~)≤(1−α) F^r( q)≤(1-α). Proof. The threshold q~ q computed by the Noisy-CP algorithm (by applying standard CP on the noisy validations set) satisfies F^n(q~)=(1−α) F^n( q)=(1-α). The true threshold q satisfies F^n(q)=(1−α)(1−ϵ)+ϵF^r(q) F^n(q)=(1-α)(1-ε)+ε F^r(q) (4.11). Looking at the difference F^n(q~)−F^n(q)=1−α−(1−α)(1−ϵ)−ϵF^r(q) F^n( q)- F^n(q)=1-α-(1-α)(1-ε)-ε F^r(q) (4.13) =ϵ(1−α−F^r(q)).=ε(1-α- F^r(q)). Hence from the monotonicity of F^n(q) F^n(q) we have q≤q~q≤ q iff F^n(q)≤F^n(q~) F^n(q)≤ F^n( q) iff F^r(q)≤1−α F^r(q)≤ 1-α. ∎ The theorem above states that if the size of the prediction set obtained by NACP is less than k(1−α)k(1-α), NACP is more effective than Noisy-CP. For example, assume k=100k=100 and 1−α=0.91-α=0.9. In this case, if the average size of the NACP prediction set is less than 90, NACP is more effective than Noisy-CP. We also see from eq. (4.13) that the smaller F^r F^r is the larger the gap between the two methods. Since F^r F^r is inversely proportional to the number of classes, we expect the difference to be substantial when there is a large number of classes to consider, which is exactly where CPs’ ability to reliably exclude possible classes is very useful. In our experiments, we indeed found a considerable gap between the two methods when we experimented on classification tasks with a large number of classes. Algorithm 4 Noise-Aware Conformal Prediction (NACP) for uniform noise 1:Input: A conformity score S(x,y)S(x,y), a coverage level 1−α1\!-\!α and a validation set (x1,y~1),…,(xn,y~n)(x_1, y_1),...,(x_n, y_n), s.t. the labels are corrupted by a uniform noise with parameter ϵε. 2:Set q1q_1 to be the (1−α)(1−ϵ)/(1−ϵk)(1\!-\!α)(1-ε)/(1- εk) quantile of S(x1,y~1),…,S(xn,y~n)S(x_1, y_1),...,S(x_n, y_n) and set q2q_2 to be ((1−α)+αϵ)((1-α)+αε) quantile. 3:For each candidate threshold q compute: F^n(q) F^n(q) =1n∑iy~i∈Cq(xi), = 1n _i 1_\ y_i∈ C_q(x_i)\, F^r(q) F^r(q) =1n∑i|Cq(xi)|k, = 1n _i |C_q(x_i)|k, F^c(q) F^c(q) =F^n(q)−ϵF^r(q)1−ϵ = F^n(q)-ε F^r(q)1-ε 4:Apply a grid search to find q∈[q1,q2]q∈[q_1,q_2] that satisfies F^c(q)=1−α F^c(q)=1\!-\!α. 5:The prediction set of a test sample x is: Cq(x)=y|S(x,y)<q.C_q(x)=\y\,|\,S(x,y)<q\. 6:Coverage guarantee: p(y∈Cq(x))≥1−α−Δ(n,ϵ,δ)p(y∈ C_q(x))≥ 1-α- (n,ε,δ) with probability (1−δ)(1-δ) over the noisy validation set sampling (see Theorem 4.3.5). 4.3.2 Coverage Guarantees We next provide a coverage guarantee for NACP. We show that if we apply the NACP to find a threshold q for 1−α+Δ1-α+ , then P(y∈Cq(x))≥1−αP(y∈ C_q(x))≥ 1-α were Δ depends on the validation set size. Δ is a finite-sample term that is needed to approximate the CDF to set the threshold instead of simply picking a predefined quantile. Because Δ can be computed, one can adjust the α used in the NACP algorithm to get the desired coverage guarantee. However, we note that we empirically found this bound to be over-conservative, and that the un-adjusted method does reach the desired coverage. Lemma 4.3.4. Given δ>0δ>0, define Δ=log(4/δ)2nh2 = (4/δ)2nh^2 such that h=1−ϵ1+ϵh= 1-ε1+ε and n is the size of the noisy validation set. Then p(supq|Fc(q)−F^c(q)|>Δ)≤δ,p( _q|F^c(q)- F^c(q)|> )≤δ, (4.14) such that the probability is over the validation set. Proof. The Dvoretzky–Kiefer–Wolfowitz (DKW) inequality [60] states that if we estimate a CDF F from n samples using the empirical CDF FnF_n then p(supx|Fn(x)−F(x)|>Δ)≤2exp(−2nΔ2p( _x|F_n(x)-F(x)|> )≤ 2 (-2n ^2). Eq. (4.10) defines F^c(q) F^c(q) using F^n(q) F^n(q) and F^r(q) F^r(q). Both are empirical CDF, so from the DKW theorem and the union bound we get that: p(supq|Fr(q)−F^r(q)|>hΔor p( _q|F^r(q)- F^r(q)|>h \,\, or (4.15) supq|Fn(q)−F^n(q)|>hΔ)≤4exp(−2nh2Δ2)=δ. \,\, _q|F^n(q)- F^n(q)|>h )≤ 4 (-2nh^2 ^2)=δ. Using eq. (4.10) we get that with probability at least 1−δ1-δ for every q: F^c(q)=F^n(q)−ϵF^r(q)1−ϵ≤ F^c(q)= F^n(q)-ε F^r(q)1-ε≤ (4.16) (Fn(q)+hΔ)−ϵ(Fr(q)−hΔ)1−ϵFc(q)+hΔ+ϵhΔ1−ϵ (F^n(q)+h )-ε(F^r(q)-h )1-εF^c(q)+ h +ε h 1-ε =Fc(q)+hΔ1+ϵ1−ϵ=Fc(q)+Δ. =F^c(q)+h 1+ε1-ε=F^c(q)+ . Similarly, we can show that F^c(q)≥Fc(q)−Δ F^c(q)≥ F^c(q)- which completes the proof. ∎ The proof of the main theorem now follows the standard CP proof, taking the inaccuracy in estimating Fc(q)F^c(q) into account. Theorem 4.3.5. Assume you have a noisy validation set of size n with noise level ϵε and set Δ(n,ϵ,δ)=log(4/δ)2nh2 (n,ε,δ)= (4/δ)2nh^2 s.t. h=1−ϵ1+ϵh= 1-ε1+ε and that you pick q such that F^c(q)=1−α+Δ F^c(q)=1-α+ . Then with probability at least 1−δ1-δ (over the validation set), we have that if (x,y)(x,y) are sampled from the clear label distribution we get: 1−α≤p(y∈Cq(x))≤1−α+2Δ.1-α≤ p(y∈ C_q(x))≤ 1-α+2 . Proof. Given a clean test pair (x,y)(x,y), with probability δ over the validation set, we have: p(y∈Cq(x))=p(S(x,y)<q)p(y∈ C_q(x))=p(S(x,y)<q) =Fc(q)≥F^c(q)−Δ=1−α.=F^c(q)≥ F^c(q)- =1-α. In a similar way: p(y∈Cq(x))=Fc(q)≤F^c(q)+Δ=1−α+2Δp(y∈ C_q(x))=F^c(q)≤ F^c(q)+ =1-α+2 . ∎ As the size of the noisy validation set, n, tends to infinity, Δ converges to zero and thus the noisy threshold converges to the noise-free threshold. Algorithm 5 Noise-Aware Conformal Prediction (NACP) for a noise matrix model 1:Input: A conformity score S(x,y)S(x,y), a coverage level 1−α1\!-\!α and a validation set (x1,y~1),…,(xn,y~n)(x_1, y_1),...,(x_n, y_n), s.t. the labels are corrupted by a noise matrix P. 2:For each candidate threshold q compute: M^q(ℓ,i)=1n∑jy~j=i,ℓ∈Cq(xj),i,ℓ=1,..,k. M_q( ,i)= 1n _j 1_\ y_j=i,\,\, ∈ C_q(x_j)\, 14.22636pti, =1,..,k. F^c(q)=Tr(M^qP−1). F^c(q)= Tr( M_qP^-1). 3:Apply a grid search to find q that satisfies F^c(q)=1−α F^c(q)=1\!-\!α. 4:The prediction set of a test sample x is: Cq(x)=y|S(x,y)<q.C_q(x)=\y\,|\,S(x,y)<q\. 4.3.3 A More General Noise Model Next, we will extend our approach to a more general noise model. We will assume that the noisy label y~ y is independent of x given y. We also assume that the noise matrix P(i,j)=p(y~=j|y=i)P(i,j)=p( y=j|y=i) is known and that the matrix P is invertible. For each q define the following matrices for the clear and the noisy data: Mqc(ℓ,i)=p(ℓ∈Cq(x),y=i)M_q^c( ,i)=p( ∈ C_q(x),y=i) and Mq(ℓ,i)=p(ℓ∈Cq(x),y~=i)M_q( ,i)=p( ∈ C_q(x), y=i). Assuming that, given the true label y, the r.v. x and y~ y are independent, we obtain: Mq(ℓ,i) M_q( ,i) =p(ℓ∈Cq(x),y~=i) =p( ∈ C_q(x), y=i) =∑jp(ℓ∈Cq(x),y~=i,y=j) = _jp( ∈ C_q(x), y=i,y=j) (4.17) =∑jp(ℓ∈Cq(x),y=j)p(y~=i|y=j) = _jp( ∈ C_q(x),y=j)p( y=i|y=j) =∑jMqc(ℓ,j)P(j,i). = _jM^c_q( ,j)P(j,i). We can write (4.17) in matrix notation: Mq=MqcPM_q=M^c_qP. Eq. (4.17) implies that: Fc(q)=p(y∈Cq(x))=p(y∈Cq(x))F^c(q)=p(y∈ C_q(x))=p(y∈ C_q(x)) (4.18) =∑ip(i∈Cq(x),y=i)=∑iMqc(i,i)=Tr(MqP−1).= _ip(i∈ C_q(x),y=i)= _iM^c_q(i,i)= Tr(M_qP^-1). We can estimate matrix MqM_q from the noisy samples: M^q(ℓ,i)=1n∑jy~j=i,ℓ∈Cq(xj),i,ℓ=1,..,k. M_q( ,i)= 1n _j 1_\ y_j=i,\,\, ∈ C_q(x_j)\, 28.45274pti, =1,..,k. (4.19) Substituting (4.19) in (4.18) yields an estimation of the probability Fc(q)=p(y∈Cq(x))F^c(q)=p(y∈ C_q(x)): F^c(q)=Tr(M^qP−1). F^c(q)= Tr( M_qP^-1). (4.20) The final step is applying a grid search to find a threshold q such that F^c(q)=1−α F^c(q)=1-α. In the case that P is a uniform noise matrix (4.6), the Sherman-Morison formula implies that P−1=(11−ϵI−ϵ(1−ϵ)k⊤)P^-1=( 11-εI- ε(1-ε)k 1 ). Therefore, F^c(q)=Tr(M^qP−1)=11−ϵ∑iM^q(i,i) F^c(q)= Tr( M_qP^-1)= 11-ε _i M_q(i,i) −ϵ(1−ϵ)k∑ℓ,iM^q(ℓ,i)=F^n(q)−ϵF^r(q)1−ϵ.- ε(1-ε)k _ ,i M_q( ,i)= F^n(q)-ε F^r(q)1-ε. Thus in the case of a uniform noise the coverage estimation (4.20) coincides with (4.10). If the noise matrix is unknown, it can be estimated from the noisy-label data during training [117, 52, 54]. The NACP method for a noise matrix model is summarized in Algorithm box 5. We can extend the finite sample term Δ that was developed for a uniform noise to obtain a theoretical coverage guarantee for a noise matrix model (4.3.6). Theorem 4.3.6. Let P be a general noise matrix. Given δ>0δ>0, define Δ=‖P−1‖∞klog(2k2/δ)2n =\|P^-1\|_∞k (2k^2/δ)2n where , k is the number of classes and n is the size of the noisy validation set. Then p(supq|F^c(q)−Fc(q)|>Δ)<δ.p( _q| F^c(q)-F^c(q)|> )<δ. Proof. From Eq. (4.20) we have |F^c(q)−Fc(q)|=|Tr(P−1M^q)−Tr(P−1Mq)|=|Tr(P−1ΔM^q)|| F^c(q)-F^c(q)|=|Tr(P^-1 M_q)-Tr(P^-1M_q)|=|Tr(P^-1 M_q)| where ΔMq=M^q−Mq M_q= M_q-M_q. We first note that Mq[i,j]=p(j∈Cq(x),y~=i)M_q[i,j]=p(j∈ C_q(x), y=i) is not a CDF but we can define one that agrees with it for q∈(−∞,C)q∈(-∞,C) which is the range of interest where C is a constant that bound the score function S(x,y)S(x,y) from above. We define S~ij(x,y~)=S(x,j),if y~=iC,if y~≠i S_ij(x, y)= casesS(x,j),&if y=i\\ C,&if y≠ i cases so Mq[i,j]=p(S~ij(x,y~)≤q)M_q[i,j]=p( S_ij(x, y)≤ q) for q∈(−∞,C)q∈(-∞,C). Now from the DKW theorem, we know that if we estimate a CDF using n samples then with probability at least 1−δ1-δ we get a uniform bound on the error of size log(2/δ)2n (2/δ)2n. As we are estimating k2k^2 matrix elements we can use the union bound to get that with probability 1−δ1-δ the ∀i,j,q∈(−∞,C):|ΔMq|≤log(2k2/δ)2n∀ i,j,q∈(-∞,C):| M_q|≤ (2k^2/δ)2n. Now if we look at the infinity norm of P−1P^-1, then |(P−1ΔMq)i,j|≤‖P−1‖∞log(2k2/δ)2n|(P^-1 M_q)_i,j|≤||P^-1||_∞ (2k^2/δ)2n. As the trace is the sum of k such matrix entries, the total bound is κ∞klog(2k2/δ)2n _∞k (2k^2/δ)2n for q∈(−∞,C)q∈(-∞,C). Since we know Fc(q)=1F^c(q)=1 for q≥Cq≥ C, we can set F^c(q)=1 F^c(q)=1 for q≥Cq≥ C and get a bound for all q∈ℝq . ∎ However, this approach yields large prediction sets especially in tasks with many classes and thus is ineffective. In the experiment section we show that in practice, even without adding finite sample terms, we obtain the required coverage probability. 4.4 Related Work In this section, we review two closely related works that address the same problem of calibration with noisy labels [86, 13]. The derivation of the noisy conformal threshold in these two works is similar to ours. These two methods compute the same threshold q that satisfies f^c(q)=1−α f^c(q)=1-α (4.20). The only minor difference is that in these two studies they use the distribution of correct labels given the noisy labels, while we use the more natural distribution of the noisy labels given the correct label. As a result, they need to know the marginal class frequencies for both the clean and noisy labels, whereas we do not. Each one of the two methods provides a different finite coverage guarantee in the form of: p(y∈Cq(x))≥1−α−Δp(y∈ C_q(x))≥ 1-α- where Δ depends on the validation set size n, the number of classes k, and the noise model, but it doesn’t depend on the validation dataset itself. We first review the bound Δ derived in [86]. Let ρi=p(y=i) _i=p(y=i) and ρ~i=p(y~=i) ρ_i=p( y=i) be the marginal true and noisy label distributions. Let M(y|y~)M(y| y) be the noise conditional distribution and let V=M−1V=M^-1. Let c(n)=[maxi∈[n](in−u(i))]c(n)=E [ _i∈[n] ( in-u_(i) ) ], such that u(i)i=1n\u_(i)\_i=1^n order statistics of uii=1n\u_i\_i=1^n i.i.d. uniform random variables on [0,1][0,1]. The size of the least common class is n∗=mini∈[k]nin_*= _i∈[k]n_i s.t. nin_i is the number of samples of noisy label i. Finally, the finite sample correction is: Δ =c(n)+2maxi∈[k]∑l≠i|Vil|+∑i=1k|ρi−ρ~i|n∗ =c(n)+ 2 _i∈[k] _l≠ i|V_il|+ _i=1^k| _i- ρ_i| n_* (4.21) ⋅min(k2π2,1n∗+log(2k2)+log(n∗)2). · (k^2 π2, 1 n^*+ (2k^2)+ (n^*)2 ). It can be easily verified that Δ=O(logk) =O( k) and therefore the bound becomes less effective for large values of k. Sesia et al. [86] also suggested a boosted version under additional assumptions. This version takes a hybrid approach that adaptively chooses between its algorithm and Noisy-CP depending on which approach leads to a lower (less conservative) calibrated threshold. The finite sample term Δ derived in [13] is: Δ=∑i=1k(|wi(1)|b(n,i)+∑i≠j|wij(2)|b(n,j)) = _i=1^k(|w_i^(1)|b(n,i)+ _i≠ j|w_ij^(2)|b(n,j)) (4.22) s.t. k is the number of classes, wi(1)=Pi,i−1ρi−ρ~iw^(1)_i=P_i,i^-1 _i- ρ_i, wij(2)=ρiPji−1w^(2)_ij= _iP^-1_ji, and b(n,j)=(1−ρ~j)n+πnρ~jb(n,j)=(1- ρ_j)^n+ πn ρ_j. ρi=p(y=i) _i=p(y=i) and ρ~i=p(y~=i) ρ_i=p( y=i) are the marginal clean and contaminated label probabilities and Pji=p(y=j|y~=i)P_ji=p(y=j| y=i) is the conditional label noise distribution. It can be easily verified that Δ=O(k) =O( k) and therefore the bound becomes less effective for large values of k. We note that our finite sample term Δ (see Lemma 4.3.4) does not depend on the number of classes k. Therefore, unlike the algorithms of Sesia et al. [86] and Clarkson et al. [13], it remains effective even in tasks with many classes. A further distinction between us and the previous works is that their finite sample coverage guarantee is established for the average of all the noisy validation sets. In contrast, our approach provides an individual coverage guarantee for nearly all (1−δ1-δ) of the sampled noisy validation sets. In Section 4.5 we show that the average coverage guarantee obtained by Sesia et al. [86] and Clarkson et al. [13] implies that in tasks with a large number of classes, the prediction set should include all the classes and therefore it is useless. In contrast, our individual finite set coverage guarantee, on to (1−δ)(1-δ) portion of the noisy validation sets, remains effective for tasks with many classes. We note that our observation that the finite sample correction term does not depend on the number of classes, applies to the case of a uniform label noise. In the case of a general noise matrix, all finite sample correction terms are not effective. 4.5 Experiments In this section, we evaluate the capabilities of our NRSCP and NACP algorithms on various medical and scenery imaging datasets. Compared methods. Our method takes an existing conformity score S and computes a threshold q that takes into account the label noise level. We implemented three popular conformal prediction scores, namely APS [82], RAPS [3] and HPS [93]. For each score, we implemented the following CP methods: (1) CP (oracle) - using a validation set with clean labels, (2) Noisy-CP - applying a standard CP on noisy labels without any modifications [21], (3) NR-CP (w/o Δ ) - Noise-Robust CP approach without the finite sample coverage guarantee Δ , see Eq. (4.20) and [86, 13]. We also implemented three methods that add finite sample coverage guarantee terms to the NR-CP method. (4) Adaptive Conformal Classification with Noisy labels (ACNL) [86], (5) Boosted Adaptive Conformal Classification with Noisy labels (ACNL+) [86] (6) Contamination Robust Conformal Prediction (CRCP) [13], (7) NRSCP - our first approach (4.2) and (8) NACP - our second approach (4.3). For methods (4), (5), and (6), we used their official codes 111https://github.com/msesia/conformal-label-noise222https://github.com/jase-clarkson/cp_under_data_contamination and we share our code for reproducibility333https://github.com/cobypenso/Noise-Aware-Conformal-Prediction. Table 4.2: CP calibration results for 1−α1\!-\!α = 0.9 and noise level ϵ=0.2ε=0.2. We report the mean and the std over 1000 different splits. We show the best result with theoretical guarantees in bold. APS RAPS HPS Dataset CP Method size ↓ coverage(%) size ↓ coverage(%) size ↓ coverage(%) CP (oracle) 1.1 ± 0.01 90.0 ± 0.62 1.1 ± 0.01 90.0 ± 0.61 0.9 ± 0.01 90.0 ± 0.59 Noisy-CP 5.1 ± 0.18 99.9 ± 0.04 5.1 ± 0.18 99.9 ± 0.04 5.1 ± 0.18 99.8 ± 0.04 NR-CP (w/o Δ ) 1.1 ± 0.02 90.1 ± 0.70 1.1 ± 0.02 90.1 ± 0.69 0.9 ± 0.02 90.0 ± 0.75 CIFAR-10 ACNL 1.5 ± 0.06 96.0 ± 0.61 1.3 ± 0.03 94.6 ± 0.65 1.1 ± 0.03 96.0 ± 0.59 (10 classes) ACNL+ 1.4 ± 0.06 95.8 ± 0.59 1.3 ± 0.02 94.8 ± 0.61 1.1 ± 0.03 96.1 ± 0.57 CRCP 1.2 ± 0.03 93.7 ± 0.62 1.2 ± 0.03 93.7 ± 0.62 1.1 ± 0.01 95.7 ± 0.18 NRSCP 2.2 ± 0.07 98.9 ± 0.12 3.3 ± 0.20 99.6 ± 0.08 1.2 ± 0.01 97.8 ± 0.12 NACP 1.3 ± 0.04 94.4 ± 0.62 1.3 ± 0.04 94.5 ± 0.62 1.1 ± 0.01 95.9 ± 0.18 CP (oracle) 6.5 ± 0.20 90.0 ± 0.43 4.0 ± 0.08 90.0 ± 0.43 2.0 ± 0.03 90.0 ± 0.43 Noisy-CP 50.5 ± 1.29 99.8 ± 0.04 50.5 ± 1.33 99.8 ± 0.03 50.1 ± 1.34 99.9 ± 0.02 NR-CP (w/o Δ ) 6.4 ± 0.28 89.9 ± 0.54 4.0 ± 0.11 89.9 ± 0.55 2.0 ± 0.06 89.9 ± 0.56 CIFAR-100 ACNL 100.0 ± 0.00 100.0 ± 0.00 100.0 ± 0.00 100.0 ± 0.00 100.0 ± 0.00 100.0 ± 0.00 (100 classes) ACNL+ 50.4 ± 1.13 99.8 ± 0.03 50.4 ± 1.01 99.9 ± 0.03 50.1 ± 1.23 99.9 ± 0.02 CRCP 25.7 ± 3.71 98.7 ± 0.39 8.5 ± 0.41 98.3 ± 0.16 11.1 ± 3.46 98.7 ± 0.42 NRSCP 37.6 ± 0.86 99.6 ± 0.06 50.1 ± 1.08 99.8 ± 0.03 11.7 ± 0.37 98.9 ± 0.08 NACP 9.0 ± 0.46 93.0 ± 0.49 4.8 ± 0.13 93.0 ± 0.48 2.5 ± 0.09 93.0 ± 0.52 CP (oracle) 14.9 ± 0.60 90.0 ± 0.61 6.9 ± 0.19 90.0 ± 0.62 3.8 ± 0.13 90.02 ± 0.58 Noisy-CP 99.7 ± 3.67 99.7 ± 0.08 101.4 ± 3.58 99.5 ± 0.09 98.3 ± 3.80 99.8 ± 0.05 NR-CP (w/o Δ ) 14.0 ± 0.91 89.4 ± 0.81 6.7 ± 0.27 89.3 ± 0.80 3.5 ± 0.24 89.3 ± 0.80 TinyImagenet ACNL 200.0 ± 0.00 100.0 ± 0.00 200.0 ± 0.00 100.0 ± 0.00 200.0 ± 0.00 100.0 ± 0.00 (200 classes) ACNL+ 99.7 ± 3.67 99.7 ± 0.08 101.4 ± 3.58 99.5 ± 0.09 98.3 ± 3.80 99.8 ± 0.05 CRCP 200.0 ± 0.00 100.0 ± 0.00 200.0 ± 0.00 100.0 ± 0.00 200.0 ± 0.00 100.0 ± 0.00 NRSCP 79.6 ± 2.82 99.3 ± 0.11 100.6 ± 2.88 99.5 ± 0.09 28.1 ± 1.20 98.1 ± 0.15 NACP 22.6 ± 1.87 93.7 ± 0.71 9.0 ± 0.50 93.6 ± 0.70 7.0 ± 0.87 93.6 ± 0.72 CP (oracle) 16.6 ± 0.33 90.0 ± 0.26 6.3 ± 0.06 90.0 ± 0.27 3.6 ± 0.07 90.0 ± 0.28 Noisy-CP 502.6 ± 8.56 99.9 ± 0.01 501.6 ± 8.51 99.9 ± 0.01 501.3 ± 10.2 100.0 ± 0.01 NR-CP (w/o Δ ) 16.7 ± 0.51 90.0 ± 0.34 6.3 ± 0.10 90.0 ± 0.36 3.6 ± 0.14 90.0 ± 0.38 ImageNet ACNL 1000.0 ± 0.00 100.0 ± 0.00 1000.0 ± 0.00 100.0 ± 0.00 1000.0 ± 0.00 100.0 ± 0.00 (1000 classes) ACNL+ 502.6 ± 8.56 99.9 ± 0.01 501.6 ± 8.51 99.9 ± 0.01 501.3 ± 10.2 100.0 ± 0.01 CRCP 1000.0 ± 0.00 100.0 ± 0.00 1000.0 ± 0.00 100.0 ± 0.00 1000.0 ± 0.00 100.0 ± 0.00 NRSCP 275.6 ± 27.1 99.7 ± 0.06 455.2 ± 20.7 99.9 ± 0.02 55.9 ± 0.87 99.1 ± 0.02 NACP 20.9 ± 0.72 91.9 ± 0.32 7.1 ± 0.13 91.9 ± 0.34 4.8 ± 0.23 91.9 ± 0.36 Evaluation Measures. We evaluated each CP method based on the average size of the prediction sets (where a small value means high efficiency) and the fraction of test samples for which the prediction sets contained the ground-truth labels. The two evaluation metrics are formally defined as: size=1n∑i|C(xi)|,coverage=1n∑i(yi∈C(xi)) size= 1n _i|C(x_i)|, 8.5359pt coverage= 1n _i 1(y_i∈ C(x_i)) such that n is the size of the test set. We report the mean and standard deviation over 1000 random splits. Datasets. We show results on four standard scenery image datasets CIFAR-10, CIFAR-100 [48], Tiny-ImageNet, and ImageNet [15]. Implementation details. Each task was trained by fine-tuning on a pre-trained ResNet-18 [35] network. The models were taken from the PyTorch site444https://pytorch.org/vision/stable/models.html. We selected this network architecture because of its widespread use in classification problems. The last fully connected layer output size was modified to fit the corresponding number of classes for each dataset. For the standard dataset evaluated in Table 4.2 we used publicly available checkpoints. For each dataset, we combined the validation and test sets and then constructed 1000 different splits where 50% was used for the calibration phase and 50% was used for testing. In all our experiments we used δ=0.001δ=0.001. In other words, the computed coverage guarantee is applied to the sampled noisy validation set with probability 0.999 Table 4.3: Finite sample correction terms Δ of NACP, ACNL [86] and CRCP [13], for several datasets and two noise levels, n is the size of the validation set. Dataset n #classes NACP ACNL CRCP ϵ=0.1ε=0.1 ϵ=0.2ε=0.2 ϵ=0.1ε=0.1 ϵ=0.2ε=0.2 ϵ=0.1ε=0.1 ϵ=0.2ε=0.2 CIFAR-10 5000 10 0.035 0.043 0.031 0.059 0.016 0.036 CIFAR-100 10000 100 0.025 0.030 0.077 0.163 0.039 0.088 TinyImagenet 5000 200 0.035 0.043 0.175 0.382 0.078 0.176 ImageNet 25000 1000 0.016 0.019 0.194 0.466 0.079 0.177 Conformal prediction results. Table 4.2 reports the noisy label calibration results across 3 different conformal prediction scores, HPS, APS, and RAPS for four standard publicly available datasets, CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet. In all cases, we used 1−α=0.91\!-\!α=0.9 and a noise level of ϵ=0.2ε=0.2. The results indicate that in the case of a validation set with noisy labels, the Noisy-CP threshold became larger to facilitate the uncertainty induced by the noisy labels. This yielded larger prediction sets and the coverage was higher than the target coverage which was set to 90%90\%. We can see that NACP outperformed the ACNL, and CRCP methods for all datasets except for CIFAR-10 with fewer classes. Following Theorem 4.3.3, we expect the gain in performance when using NACP versus Noisy-CP to increase with the number of classes, indeed validated empirically in Table 4.2. Here for CIFAR-100, Tiny-ImageNet, and ImageNet the ACNL and CRCP methods failed due to the large number of classes and the relatively small number of samples per class. For TinyImagenet and Imagenet, ACNL+ falls back to Noisy-CP. A direct comparison of the finite sample correction terms Δ obtained by NACP, ACNL and CRCP is shown in Table 4.3. Note that if 1−α+Δ>11-α+ >1, the prediction set includes all the classes and thus it becomes useless. We can see in Table 4.3 that this is the case for ACNL and CRCP in datasets with a large number of classes. Correction term analysis. Following the theoretical and empirical results, the effectiveness of our method and baselines can be fully explained by the correction terms Δ each method guarantees as practitioners require coverage guarantee and therefore will use 1−α+Δ1-α+ . Note that, as explained in Section 4.4, our finite sample coverage guarantee is different from the one provided by the baseline method. Figure 4.1 presents the correction term as a function of calibration set size and the number of classes. Note that the NACP curve remains exactly the same across the 3 plots. Our main contribution is grounded in the fact that NACP is not dependent on the number of classes k, clearly shown in plots as the number of classes grows. Figure 4.1: Correction terms Δ of NACP, ACNL and CRCP as a function of the validation set size n given ϵ=0.2ε=0.2. We show results for 3 numbers of classes, 10, 100 and 1000. General noise transition matrix. Finally, we evaluate NRSCP and NACP on two common general noise matrices: Neighborhood noise and Random noise (see details in Section 2.4). While existing final sample terms bounds are not effective, in practice NACP (without a finite sample correction) achieves the required coverage guarantee and the average prediction size is similar to the one obtained by the noise-free CP. We observe the same pattern when using uniform noise. This indicates that the current coverage guarantee bounds are too conservative. Table LABEL:generalNoiseResults shows the results on the CIFAR-100 dataset and the rand-APS technique when using NACP without finite sample correction term Δ . Results show a clear dominance of NACP over Noisy-CP and NRCP on the two different noise models, presenting the robustness of NACP across various noise models. ACNL (without the finite sample term) achieves here similar results. Table 4.4: Rand-APS calibration results for 1−α1\!-\!α = 0.9 on CIFAR-100 dataset and two noise models. We report the mean and the std over 1000 different splits. Neighborhood noise Random noise CP Method size ↓ coverage (%) size ↓ coverage (%) CP (oracle) 6.48 ± 0.19 90.01 ± 0.41 6.48 ± 0.19 90.01 ± 0.41 Noisy-CP 48.89 ± 1.13 99.80 ± 0.04 50.25 ± 1.37 99.82 ± 0.04 NRSCP 12.82 ± 0.36 95.62 ± 0.21 37.01 ± 0.88 99.53 ± 0.06 NR-CP (w/o Δ ) 6.52 ± 0.22 90.03 ± 0.47 6.45 ± 0.30 89.97 ± 0.57 Next, we show the results of the following experiments. In Calibration set size we test the performance of various conformal prediction methods under noisy labels as a function of the calibration set size. In NACP Agnostic to Different model architectures we show that NACP is agnostic to different classification network architectures. Finally, in Experiments on real noisy datasets we report experiments on real noisy datasets where the noise is due to manual annotation error. We show that in this case, by imposing a uniform noise model, we get better results than the one obtained by ignoring the noise and applying CP directly on the noisy validation set. Calibration set size. In the following experiment, we test the performance of various conformal prediction methods under noisy labels as a function of the calibration set size on the ImageNet dataset. Figure 4.2 shows the mean size and coverage as a function of the calibration set size. In addition, the correction term Δ is depicted for ImageNet for each calibration set size. Results show that even with as little as 2500 images that correspond to 2.5 images per class the calibration results are almost on par with the oracle calibration given clean labels. (a) (b) (c) Figure 4.2: Noisy labels conformal prediction on ImageNet with different calibration set sizes. (a) Mean size (b) Coverage (%), and (c) Correction terms Δ as a function of calibration set size. NACP Agnostic to Different model architectures. Conformal prediction in general and our method NACP specifically has no assumption and is agnostic to the underlying model architecture. In the following section, we verify that by experimenting with ImageNet across different model architectures. Table 4.5 presents the results of applying conformal prediction with and without noisy labels on ResNet18, ResNet50, DenseNet121, ViT-B16 (Vision transformer). Table 4.5: CP calibration results on ImageNet and various model architectures for 1−α1\!-\!α = 0.9 and ϵ=0.2ε=0.2. We report the mean and the std over 1000 different splits. Bold for best result with theoretical guarantees. ResNet-18 ResNet-50 DenseNet121 ViT-B16 Dataset CP Method size ↓ coverage(%) size ↓ coverage(%) size ↓ coverage(%) size ↓ coverage(%) APS CP (oracle) 16.6 ± 0.33 90.0 ± 0.26 13.9 ± 0.34 90.0 ± 0.28 12.0 ± 0.28 90.0 ± 0.27 10.7 ± 0.38 90.0 ± 0.25 Noisy-CP 502.6 ± 8.56 99.9 ± 0.01 505.5 ± 8.11 99.9 ± 0.01 502.8 ± 8.46 99.9 ± 0.01 506.8 ± 8.14 99.8 ± 0.02 NR-CP (w/o Δ ) 16.7 ± 0.51 90.0 ± 0.34 13.9 ± 0.47 90.0 ± 0.37 12.0 ± 0.38 90.0 ± 0.34 10.7 ± 0.55 90.0 ± 0.35 ACNL 1000.0 ±0.00± 0.00 100.0 ± 0.00 1000.0 ±0.00± 0.00 100.0 ± 0.00 1000.0 ±0.00± 0.00 100.0 ± 0.00 1000.0 ±0.00± 0.00 100.0 ± 0.00 ACNL+ 502.6 ± 8.56 99.9 ± 0.01 505.5 ± 8.11 99.9 ± 0.01 502.8 ± 8.46 99.9 ± 0.01 506.8 ± 8.14 99.8 ± 0.02 CRCP 1000.0 ±0.00± 0.00 100.0 ± 0.00 1000.0 ±0.00± 0.00 100.0 ± 0.00 1000.0 ±0.00± 0.00 100.0 ± 0.00 1000.0 ±0.00± 0.00 100.0 ± 0.00 NACP 20.9 ± 0.72 91.9 ± 0.32 17.4 ± 0.62 91.9 ± 0.36 15.1 ± 0.55 91.9 ± 0.34 15.5 ± 0.81 91.9 ± 0.31 RAPS CP (oracle) 6.3 ± 0.06 90.0 ± 0.27 4.5 ± 0.05 89.9 ± 0.29 4.7 ± 0.06 90.0 ± 0.26 2.6 ± 0.04 90.0 ± 0.25 Noisy-CP 501.6 ± 8.51 99.9 ± 0.01 501.1 ± 8.85 99.9 ± 0.01 501.9 ± 8.80 99.9 ± 0.01 505.8 ± 7.90 99.9 ± 0.01 NR-CP (w/o Δ ) 6.3 ± 0.10 90.0 ± 0.36 4.5 ± 0.06 90.0 ± 0.35 4.7 ± 0.08 90.0 ± 0.36 2.6 ± 0.05 90.0 ± 0.36 ACNL 1000.0 ±0.00± 0.00 100.0 ± 0.00 1000.0 ±0.00± 0.00 100.0 ± 0.00 1000.0 ±0.00± 0.00 100.0 ± 0.00 1000.0 ±0.00± 0.00 100.0 ± 0.00 ACNL+ 501.6 ± 8.51 99.9 ± 0.01 501.1 ± 8.85 99.9 ± 0.01 501.9 ± 8.80 99.9 ± 0.01 505.8 ± 7.90 99.9 ± 0.01 CRCP 1000.0 ±0.00± 0.00 100.0 ± 0.00 1000.0 ±0.00± 0.00 100.0 ± 0.00 1000.0 ±0.00± 0.00 100.0 ± 0.00 1000.0 ±0.00± 0.00 100.0 ± 0.00 NACP 7.1 ± 0.13 91.9 ± 0.34 5.0 ± 0.08 91.9 ± 0.34 5.3 ± 0.10 92.0 ± 0.35 2.9 ± 0.07 92.0 ± 0.30 HPS CP (oracle) 3.6 ± 0.07 90.0 ± 0.28 2.0 ± 0.03 90.0 ± 0.28 2.4 ± 0.03 90.0 ± 0.25 1.5 ± 0.02 90.0 ± 0.26 Noisy-CP 501.3 ± 10.2 100.0 ± 0.01 502.4 ± 9.50 99.9 ± 0.01 502.3 ± 10.3 99.9 ± 0.20 504.3 ± 8.19 99.9 ± 0.01 NR-CP (w/o Δ ) 3.6 ± 0.14 90.0 ± 0.38 2.1 ± 0.06 90.0 ± 0.38 2.4 ± 0.07 90.0 ± 0.34 1.5 ± 0.03 90.0 ± 0.35 ACNL 1000.0 ±0.00± 0.00 100.0 ± 0.00 1000.0 ±0.00± 0.00 100.0 ± 0.00 1000.0 ±0.00± 0.00 100.0 ± 0.00 1000.0 ±0.00± 0.00 100.0 ± 0.00 ACNL+ 501.3 ± 10.2 100.0 ± 0.01 502.4 ± 9.50 99.9 ± 0.01 502.3 ± 10.3 99.9 ± 0.20 504.3 ± 8.19 99.9 ± 0.01 CRCP 1000.0 ±0.00± 0.00 100.0 ± 0.00 1000.0 ±0.00± 0.00 100.0 ± 0.00 1000.0 ±0.00± 0.00 100.0 ± 0.00 1000.0 ±0.00± 0.00 100.0 ± 0.00 NACP 4.8 ± 0.23 91.9 ± 0.36 2.6 ± 0.10 91.9 ± 0.37 3.1 ± 0.12 91.9 ± 0.34 1.7 ± 0.04 91.9 ± 0.33 Experiments on real noisy datasets. We evaluate our methods on real-world noisy datasets, focusing on the CIFAR-10N dataset, which contains human annotation errors and was introduced in [99]. Specifically, we analyze four variations of CIFAR-10N: CIFAR-10-aggregate and CIFAR-10-random-1,2,3. CIFAR-10-aggregate combines three noisy labels using majority voting. If the three submitted labels differ, the aggregated label is randomly selected from the three options. CIFAR-10-random-i (i∈1,2,3i∈\1,2,3\) refers to the i-th submitted label for each image. Importantly, the data collection process ensures that no image is labeled multiple times by the same annotator. While this noise model realistically reflects human annotation behavior, it does not adhere to a strict uniform noise distribution. For our experiments with NACP and baseline methods, we adopt the noise ratio reported in [99] to compute ϵε for the noise-aware conformal prediction algorithm. Notably, [99] defines the noise rate with ϵε such that Ti,i=1−ϵT_i,i=1-ε. In contrast, our notation uses ϵε with the formulation Ti,i=1−ϵ+ϵkT_i,i=1-ε+ εk. Consequently, we adjust ϵε values from the original paper to align with our approach. The purpose of this experiment is to demonstrate that even when the true noise—arising from annotators—is not exactly uniform, approximating it as such can still yield effective performance in real-world datasets and scenarios. Table 4.6 summarizes the results for the four CIFAR-10N variations. In this real-world scenario, clean labels (CP Oracle) are unavailable. Instead, Noisy-CP results reflect calibration using annotator-provided labels as-is. Our noise-aware approach demonstrates a consistent improvement over this baseline. Table 4.6: CP calibration results on CIFAR-10N for 1−α1\!-\!α = 0.9 and ϵ=0.2ε=0.2. We report the mean and the std over 1000 different splits. Bold for best result with theoretical guarantees. CIFAR-10N-aggregate CIFAR-10N-random-1 CIFAR-10N-random-2 CIFAR-10N-random-3 (10.0%) (19.1%) (20.1%) (19.6% ) Dataset CP Method size ↓ coverage(%) size ↓ coverage(%) size ↓ coverage(%) size ↓ coverage(%) rand-APS Noisy-CP 1.76 91.73 2.00 93.18 2.13 93.39 2.44 96.45 NACP 1.58 88.49 1.82 91.38 1.93 91.67 2.13 94.50 HPS Noisy-CP 1.32 91.19 1.67 92.97 1.79 93.15 2.16 96.31 NACP 1.38 91.85 1.45 91.00 1.47 90.41 1.53 92.18 Chapter 5 Local Differential Private Conformal Prediction In this chapter, we explore the intersection of conformal prediction (CP) and privacy-preserving techniques, introducing a Local Differentially Private Conformal Prediction (LDP-CP) framework. While traditional CP methods rely on cleanly labeled calibration sets, data privacy concerns often prevent access to true labels, particularly in sensitive domains like medical or personal data. Our proposed framework addresses the challenge of maintaining valid prediction set coverage while protecting user data through Local Differential Privacy (LDP). We present two complementary approaches—LDP-CP-L and LDP-CP-S—that cater to varying privacy goals, computational resources, and data scenarios, offering a robust solution for secure, decentralized data handling. 5.1 Problem Statement Conformal prediction typically relies on having access to a cleanly labeled calibration set to set the CP threshold. However, in many real-world scenarios, such clean labels may be unavailable due to data privacy concerns. For instance, in scenarios involving medical or personal data, or if the data aggregator (e.g. a cloud-based ML service) is considered untrusted, privacy constraints may prohibit access to true labels and data, requiring a privacy-preserving mechanism that provides Local Differential Privacy (LDP). In such situations, the aggregator might only be allowed to view noisy versions of the labels. Next we tackle the challenge of applying conformal prediction when validation-set labels (or conformity scores) must be protected through privacy-preserving mechanisms. Specifically, we introduce a Local Differentially Private Conformal Prediction (LDP-CP) framework that balances privacy with real-world considerations such as user-side computational capacity, aggregator trustworthiness, and intellectual property. We present two complementary LDP-CP solutions. LDP-CP-L locally perturbs labels using a randomized response, that shifts all score-related computations to the aggregator. This design suits cases where users have minimal computational resources, or the model’s internal structure is not disclosed to them, while not sharing their sensitive labels. However, it only achieves label-DP [5, 29]. In contrast, LDP-CP-S allows users to generate and locally randomize their own conformity scores, which is ideal for scenarios where both feature and label privacy are paramount and the user can handle additional computational tasks. We also offer guidelines on choosing which method aligns better with specific privacy goals, resource constraints, and per-dataset properties such as sample size and the number of classes. By considering both score-based and label-based perturbations, we provide a flexible framework that adapts to various privacy budgets and computational setups. This framework guarantees valid coverage for the true labels asymptotically and in finite samples. As a result, LDP-CP aligns well with modern demands for secure, decentralized data handling. Relationship to Past Work. Conformal prediction in the presence of label noise has received growing attention [21, 86, 13] and Chapter 4, with methods that adjust the calibration threshold when a known noise matrix corrupts labels. Simultaneously, LDP has become a leading approach for protecting sensitive data at the user end [98, 45, 95], allowing users to randomize their own inputs (e.g., labels) before sharing with an untrusted aggregator. We bridge these lines of work by recognizing that local DP can be seen as a known noisy channel on the labels or conformal scores, we can plug that channel into a “noise-aware” conformal procedure. The end result is a conformal predictor whose coverage remains valid, while also protecting each user’s label or score via ε -LDP. To the best of our knowledge, there is no prior work on LDP conformal prediction. Thus, we are the first to research local differential private conformal prediction. The closest work to ours is that of Angelopoulos et al. [2], which suggests a centrally differentially private conformal prediction procedure where a trusted aggregator has access to raw data. In our setting, the aggregator never observes true labels directly, hence, closing a key gap in privacy-preserving conformal prediction research. This dissertation makes three key contributions: • We introduce two complementary LDP-CP methods, LDP-CP-S and LDP-CP-L, that accommodate different privacy constraints and computational setups. (Figures 5.1 and 5.2). • We prove coverage guarantees for both methods under LDP constraints. (Theorems 5.4.1 and 5.4.2). • We demonstrate the feasibility and effectiveness of our approaches in privacy-sensitive applications, such as medical data analysis and untrusted cloud-based ML services (see Section 5.6). This work bridges the gap between privacy-preserving mechanisms and conformal prediction, by providing a foundation for robust and private uncertainty quantification in real-world scenarios. Unlike other methods that rely on trusted aggregators, our approaches ensure privacy directly at the user level, which aligns with modern demands for decentralized privacy protection. 5.2 Local-DP Conformal Prediction on Labels In many real-world contexts, users lack the ability—or permission—to compute model-based scores on their own devices. This may be due to limited computational resources, a restricted API that only provides predictions, or a proprietary model architecture. To address these cases, we propose an LDP mechanism that randomizes each user’s label and then applies a noise-aware conformal calibration at the aggregator. This design protects sensitive labels while remaining model-agnostic, since the aggregator performs all the scoring steps. Figure 5.1 illustrates the overall pipeline. Figure 5.1: Local Differential Private Conformal Prediction (LDP-CP-L) Pipeline (Best viewed in color). Algorithm 6 LDP-CP-L - User-side Procedure 1:Input: Calibration set (xi,yi)i=1n\(x_i,y_i)\_i=1^n, privacy parameter ϵε, number of labels k 2:Output: Noisy calibration set (xi,y~i)i=1n\(x_i, y_i)\_i=1^n 3:Set β=k−1+eϵβ= kk-1+e^ε 4:for each user i∈1,…,ni∈\1,…,n\ do 5: Apply k-ary Randomized Response (R) to label yiy_i to obtain y~i y_i: 6: p(y~=t′∣y=t)= 1t=t′(1−β)+βkp( y=t y=t)=\,1_\t=t \(1-β)+ βk 7: Send (xi,y~i)(x_i, y_i) to the aggregator We next unpack LDP-CP-L by first describing the user’s side procedure that applies to its data point, and then the procedure done by the untrusted aggregator to compute the conformal prediction threshold. User’s side Procedure. For each user i with pair (xi,yi)(x_i,y_i): 1. User applies the LDP mechanism - k-ary R to their label. This yields y~i y_i. 2. Report (xi,y~i)(x_i, y_i) to the aggregator. Aggregator’s Procedure. Once the aggregator collects xi,y~ii=1n\x_i, y_i\_i=1^n (it does not see yiy_i), it runs a “noise-aware” conformal method on the pairs (xi,y~i)(x_i, y_i). Since the aggregator knows the noise model, it can estimate the coverage for the true label. A key advantage of this label-perturbation strategy is that the aggregator never observes raw labels, thus meeting label-LDP guarantees. Meanwhile, by modeling a k-ary Randomized Response as a known noise channel, the aggregator recovers the necessary calibration adjustments to preserve near-correct coverage on the true labels. In what follows, we outline the procedure in more detail, leading to our first main theorem and contribution culminating in Theorem 5.4.1. For k-R, each label is replaced by a random label with probability β, known channel from y to y~ y. Then, p(y~=j∣y=i)= 1i=j(1−β)+βk;β=k−1+eϵp( y=j y=i)=\,1_\i=j\(1-β)+ βk; β= kk-1+e^ε For each threshold q, let Fc(q)=p(S(x,y)≤q)F^c(q)=p (S(x,y)≤ q ), Fn(q)=p(S(x,y~)≤q)F^n(q)=p (S(x, y)≤ q ), Fr(q)=p(S(x,u)≤q), u∼Unif(1,…,k)F^r(q)=p (S(x,u)≤ q ), $u Unif(1,…,k)$ coverage on true labels, noisy labels, and uniform respectively. One can see that Fn(q)=(1−β)Fc(q)+βFr(q).F^n(q)\;=\;(1-β)\,F^c(q)\;+\;β\,F^r(q). (5.1) Given a calibration set (xi,y~i)\(x_i, y_i)\, we can estimate Fn(q)F^n(q) and Fr(q)F^r(q) by F^n(q)=1n∑i=1n1(S(xi,y~i)≤q);F^r(q)=1n∑i=1n|Cq(xi)|k, F^n(q)= 1n _i=1^n1 (S(x_i, y_i)≤ q );\,\,\, F^r(q)= 1n _i=1^n |C_q(x_i)|k, Thus, rearranging Eq. (5.1) we obtain: F^c(q)=F^n(q)−βF^r(q) 1−β. F^c(q)= F^n(q)-β F^r(q)\,1-β. Algorithm 7 LDP-CP-L - Aggregator-side Procedure 1:Input: Noisy calibration set (xi,y~i)i=1n\(x_i, y_i)\_i=1^n, privacy parameter ϵε, number of labels k, target coverage 1−α1-α 2:Output: Threshold q ensuring private coverage 1−α−Δ1-α- on true labels 3:Collect the noisy calibration set (xi,y~i)i=1n\(x_i, y_i)\_i=1^n 4:Compute β=k−1+eϵβ= kk-1+e^ε 5:Estimate F^n(q) F^n(q) and F^r(q) F^r(q) for candidate thresholds q: F^n(q)=1n∑i=1n(S(xi,y~i)≤q);F^r(q)=1n∑i=1n|Cq(xi)|k F^n(q)= 1n _i=1^n1(S(x_i, y_i)≤ q); F^r(q)= 1n _i=1^n |C_q(x_i)|k 6:Compute F^c(q) F^c(q): F^c(q)=F^n(q)−β⋅F^r(q)1−β F^c(q)= F^n(q)-β· F^r(q)1-β 7:Initialize slow=0,shigh=1s_low=0,s_high=1 8:for j=1,…,Tj=1,...,T do 9: Set q(j)=slow+shigh2q^(j)= s_low+s_high2 10: Obtain Z(j)=F^c(q(j))Z^(j)= F^c(q^(j)) 11: if Z(j)>(1−α)+Δ2Z^(j)>(1-α)+ 2 then shigh=q(j)s_high=q^(j) 12: else if Z(j)<(1−α)−Δ2Z^(j)<(1-α)- 2 then slow=q(j)s_low=q^(j) 13: else break 14:Return the threshold q(j)q^(j) Hence, to find a threshold q that yields coverage 1−α1-α on the true labels in a private manner, we solve F^c(q)=1−α F^c(q)=1-α. In practice, we do a binary search over candidate thresholds which continues until either the estimate Z^(j) Z^(j) satisfies |Z^(j)−(1−α)|≤Δ| Z^(j)-(1-α)|≤ or the interval length shigh−slows_high-s_low becomes smaller than a predefined threshold τ. This is exactly the “noise-aware” threshold that corrects for the k-R noise mechanism. User’s side and aggregator’s side procedures depicted in Algorithm boxes 6 and 7 respectively. In Chapter 4 we considered this exact scenario and referred to this procedure as NACP (Noise-Aware Conformal Prediction). However, while we were motivated by problems related to a general noisy channel (e.g. experts’ mistakes/disagreements as to the true label), here we use the noisy channel as a privacy protection for the labels in the calibration set. As it turns out, using k-R falls neatly into their paradigm and in turn yields Theorem 5.4.1. Note that the k-R mechanism is implemented here, instead of more advanced LDP methods, such as RAPPOR, since it aligns well with the NACP framework. 5.3 Local-DP Conformal Prediction on Scores In some scenarios, users may be able to compute the full conformity score locally. Concretely, the aggregator (or model provider) sends the neural network’s parameters or logits to each user, who then computes the conformity score S(xi,yi)S(x_i,y_i) for the ground-truth label yiy_i on their own. This approach leverages the ability to estimate quantiles of a distribution using noisy scores that are locally privatized by the users. We employ the LDP-binary search algorithm described in [26] to estimate the (1−α)(1-α)-quantile of the scores. This method is incorporated into the conformal prediction framework by estimating the (1−α)(1-α)-quantile of the conformity scores derived locally by the users. Once the quantile is estimated, it is used as the threshold for constructing prediction sets. The use of LDP ensures that the process is privacy-preserving, whereas the quantile estimation guarantees accurate calibration of prediction intervals. By combining the strengths of binary search and randomized response, this method offers a robust approach to privacy-preserving conformal prediction that is both practical and theoretically sound. The settings are depicted in Figure 5.2. User’s side Procedure. For each user i with pair (xi,yi)(x_i,y_i): 1. Compute the conformity score Si=S(xi,yi)S_i=S(x_i,y_i) locally. 2. Compare SiS_i with a threshold q(j)q(j) provided by the aggregator. (Note that each user has a single interaction with the aggregator – see details in Aggregator’s procedure). 3. Return a binary response using randomized response (R), ensuring ε -LDP. Figure 5.2: Local Differential Private Conformal Prediction (LDP-CP-S) Pipeline (Best viewed in color). Aggregator Procedure. The aggregator collects the binary responses from users over a binary search procedure. The process starts by defining an initial range that is guaranteed to contain the desired (1−α)(1-α)-quantile. The range is repeatedly divided in half through a binary search procedure. At each step j, a midpoint q(j)q(j) is calculated, and a subset of the users is used to privately estimate how many data points fall below this midpoint. This estimation is done using randomized response, which ensures that the algorithm complies with privacy guarantees. Depending on the results of this estimation, the algorithm updates the range: if too many data points are estimated to fall below the midpoint, the upper boundary is adjusted; if too few, the lower boundary is adjusted. Note that the subsets of users used at each step are disjoint, which assures that each user has at most one interaction with the aggregator. This process continues until either the estimate Z^(j) Z^(j) satisfies |Z^(j)−(1−α)|≤Δ| Z^(j)-(1-α)|≤ or the interval length shigh−slows_high-s_low becomes smaller than a predefined threshold τ. In Theorem 5.4.2 we give the concrete sample complexity, under which we can estimate Z^(j) Z^(j) both privately and accurately in all iterations of the binary search. Once the aggregator finds the desired estimation, denoted q q, the aggregator can now construct the prediction set as: Cq^(x)=y∣S(x,y)≤q^.C_ q(x)\;=\;\\,y S(x,y)≤ q\. User’s side and aggregator’s side procedures depicted in Algorithm box 8. Algorithm 8 LDP-CP-S - Both sides 1:Input: Calibration set (xi,yi)i=1n\(x_i,y_i)\_i=1^n, privacy parameter ϵε, number of labels k, desired coverage 1−α1-α, Δ , T 2:Output: Threshold q ensuring private coverage 1−α−Δ1-α- on true labels 3:Initialize j=0,n′=nT,shigh=Qmax,slow=Qminj=0,n = nT,s_high=Q_max,s_low=Q_min 4:for j=1,…,Tj=1,...,T do 5: Select users (j)=j⋅n′+1,j⋅n′+2,…,(j+1)⋅n′U^(j)=\j· n +1,j· n +2,...,(j+1)· n \ 6: Set q(j)=slow+shigh2q^(j)= s_low+s_high2 7: Individual users i∈(j)i ^(j): (1) compute their score s=s(xi,yi)s=s(x_i,y_i), (2) sets bi=1(s<q(j))b_i=1(s<q^(j)), (3) sends the aggregator RR(bi)R(b_i) 8: Obtain Z(j)=eϵ+1eϵ−1⋅1n∑i∈(j)RR(bi)−1eϵ−1Z^(j)= e^ε+1e^ε-1· 1n _i ^(j)R(b_i)- 1e^ε-1 9: if Z(j)>(1−α)+Δ2Z^(j)>(1-α)+ 2 then shigh=q(j)s_high=q^(j) 10: else if Z(j)<(1−α)−Δ2Z^(j)<(1-α)- 2 then slow=q(j)s_low=q^(j) 11: else break 12:Return q(j)q^(j) 5.4 Theoretical Guarantees We now present our main theoretical results. Theorem 5.4.1 (LDP-CP-L). Fix α,δ,Δ>0α,δ, >0. There exists an ϵε-local differentially private algorithm that draws n=O(log(1/δ)Δ2h2)n=O ( ( 1δ) ^2h^2 ) exchangeable samples from any admissible distribution D, where h=1−β1+βh= 1-β1+β, and β=k−1+eϵβ= kk-1+e^ε, and, within at most T=⌈log(1/τ)⌉T= ( 1τ) iterations, produces an estimate q q that satisfies Pr(y∈Cq^(x))≥ 1−α−Δ (y∈ C_ q(x) )\;≥\;1-α- with probability at least 1−δ1-δ, where 1−α1-α is the desired coverage and τ is an a-priori bound on the length of an interval that can hold Δ -probability mass. Proof. The LDP-CP mechanism applies k-R to the input data, ensuring ϵε-local differential privacy [45]. Additionally, using the post-processing property of differential privacy [20], it is safe to perform arbitrary computations on the output of a differentially private mechanism - which maintains the privacy guarantees of the mechanism. Therefore, since k-R(xi,yii=1ncal)k-R(\x_i,y_i\_i=1^n_cal) satisfies ϵε-local-differential privacy, and because NACP is a deterministic or randomized post-processing function, it follows that NACP(k-R(xi,yii=1ncal)(\x_i,y_i\_i=1^n_cal)) satisfies ϵε-local-differential privacy. The remainder of the proof focuses on the conformal prediction coverage guarantee bound. Given our k-R(ϵε) ϵε-LDP mechanism, we derive β=k−1+eϵβ= kk-1+e^ε. Substituting β, and ncaln_cal into 4.3.5 we obtain Δ(n,β,δ) (n,β,δ) such that Pr(y∈Cq~(x))≥ 1−α−Δ. (y∈ C_ q(x) )\;≥\;1-α- . ∎ Theorem 5.4.2 (LDP-CP-S). Fix α,δ,Δ>0α,δ, >0. There exists an ϵε-local differentially private algorithm that draws n=O(TΔ2(eϵ+1eϵ−1)2log(T/δ)))n=O ( T ^2( e^ε+1e^ε-1)^2 ( Tδ)) ) exchangeable samples from any admissible distribution D and, within at most T=⌈log(1/τ)⌉T= ( 1τ) iterations, produces an estimate q q that satisfies Pr(y∈Cq^(x))≥ 1−α−Δ (y∈ C_ q(x) )\;≥\;1-α- with probability at least 1−δ1-δ, where 1−α1-α is the desired coverage and τ is an a-priori bound on the length of an interval that can hold Δ -probability mass. Proof. The privacy proof of a ϵε-LDP quantile binary search algorithm can be found in Gaboardi et al. [26]. Users start by computing scores locally and then a local differentially private quantile binary search algorithm is taken place [26]. ϵε-LDP follows immediately from the fact that the only time we access the data is via randomized response. The output of the algorithm is the (1−α)(1-α)’th quantile that one would obtain by applying conformal prediction to the clean data, yet it is recovered solely from the privatized (noisy) scores. ∎ 5.5 Practical Considerations 5.5.1 LDP-CP-L vs. LDP-CP-S The proposed methods, LDP-CP-L and LDP-CP-S, offer distinct approaches for achieving local differential privacy in conformal prediction, each is tailored to specific privacy setups and has different coverage guarantees. An in-depth understanding of their trade-offs is essential to determine their suitability for various scenarios. LDP-CP-L focuses on achieving local differential privacy by perturbing the labels (y) while exposing the features (x) to the untrusted aggregator. This approach ensures privacy for the labels, which are typically considered more sensitive in many applications. This mechanism is particularly advantageous in scenarios where users are resource-constrained, since they only need to perturb their labels locally before sending (x,y~)(x, y) to the aggregator. This design also keeps the model parameters and scoring functions secret from the users, because all computations related to the score are performed centrally. Consequently, LDP-CP-L is well-suited for real-world calibration datasets of moderate size, such as those containing several thousand records, where the additional noise introduced by the mechanism remains manageable. However, a notable drawback of LDP-CP-L is its sensitivity to the number of classes k in the dataset. As k increases, the calibration error (Δ ) grows, potentially compromising coverage guarantees for datasets with a large number of classes. Additionally, while the exposure of x provides practical utility by allowing centralized score computation, it introduces privacy concerns. This drawback can be partially mitigated by employing the shuffle model of differential privacy, which adds an extra layer of anonymity to the users’ data (See further discussion in Section 5.5.2). Furthermore, privatizing x would result in a substantial insertion of noise, thereby leading to a significant degradation in the accuracy of the algorithms. By contrast, LDP-CP-S achieves privacy at the score level by having users compute scores locally and perturb their responses before submitting them to the aggregator. This design ensures that both x and y remain private and are never exposed to the aggregator, making LDP-CP-S particularly suitable for scenarios where feature privacy is paramount. Unlike LDP-CP-L, the performance of LDP-CP-S is independent of the number of classes k. However, this approach imposes additional computational requirements on the users, who must perform local computations involving the model and scoring function. This requirement necessitates sharing the model with users, which might raise concerns about intellectual property or model misuse. Furthermore, LDP-CP-S requires a larger calibration dataset (n) to achieve a sufficiently small calibration error, potentially limiting its applicability in scenarios with limited data availability. In summary, the choice between LDP-CP-L and LDP-CP-S depends on the specific privacy and computational constraints of the application. LDP-CP-L is better suited for scenarios where label privacy is the primary concern, datasets are of moderate size (and larger). Its design minimizes user-side computations and protects the model from exposure. Conversely, LDP-CP-S is preferable when both feature and label privacy are critical, and when users have the computational resources to perform local scoring. Its robustness to the number of classes makes it a strong candidate for applications with a large class set, provided a sufficiently large calibration dataset is available. By carefully considering these trade-offs, practitioners can select the most appropriate method for their privacy-preserving conformal prediction tasks. In the experiment section, we report a numerical comparison of methods accuracy (ΔL,ΔS _L, _S) as a function of the calibration set size n and the number of classes k (Figure 5.3). 5.5.2 The Shuffle Model of Differential Privacy The shuffle model of differential privacy enhances privacy guarantees by introducing an additional layer of anonymization between users and the data aggregator. In this model, each user applies a local randomizer to their data and then sends the output to a secure shuffler, which permutes the messages uniformly at random before forwarding them to the aggregator. This anonymization mechanism breaks the association between individual users and their messages, thereby amplifying privacy guarantees beyond those attainable in the purely local model. A notable benefit of the shuffle model is its capacity for privacy amplification [12]: if each user applies an ε -LDP mechanism before sending their message, the effective privacy loss can be reduced to approximately ϵeff=ε/nε^eff= n in the shuffled output. This amplification enables stronger privacy guarantees with the same local noise or, conversely, allows for reduced noise to achieve a given privacy target—thereby improving utility in downstream tasks. This is particularly beneficial in regimes where moderately high local privacy levels (e.g., ε>1 >1) would otherwise impose significant performance degradation. Given these advantages, our approach incorporates the shuffle model to improve the privacy-utility trade-off of our mechanisms. While earlier versions of our framework considered exclusively the local model, we found that adopting the shuffle model allows us to retain decentralization and local control while achieving significantly improved accuracy through amplification. Importantly, this integration does not alter the algorithmic structure of our mechanisms but rather augments their privacy analysis and performance guarantees under realistic assumptions of an honest-but-curious shuffler. As such, the shuffle model serves not only as a technical enhancement but also as a practical enabler of more effective private learning in our setting. 5.6 Experiments In this section, we evaluate the capabilities of our LDP-CP algorithms on various medical and scenery imaging datasets, and address the utility-coverage tradeoff. Compared Methods. Our method takes an existing conformity score S and computes a threshold q that considers the injected noise level. We implemented two popular conformal prediction scores, namely APS [82] and HPS [93]. For each score S, we compared the following CP methods: (1) Not-Private-CP with coverage guarantee 1−α1-α - using a validation set with clean labels (2) LDP-CP-S,L and (3) LDP-CP-S,L* with coverage guarantee 1−α+Δ1-α+ . We share our code for reproducibility111https://anonymous.4open.science/r/LDP-CP. Datasets. We present results on several publicly available medical imaging classification datasets [108]. TissuMNIST [107, 108]: This dataset contains 236,386 human kidney cortex cells, organized into 8 categories. Each gray-scale image is 32×32×732× 32× 7 pixels. The 2D projections were obtained by taking the maximum pixel value along the axial-axis of each pixel, and were resized into 28×2828× 28 gray-scale images [101]. OrganSMNIST [108]: This dataset contains 25,221 images of abdominal CT in eleven classes. The images are 28×2828× 28 in size. Here, we used a train/validation/test split of 13,940/2,452/8,829 images. OrganAMNIST and OrganCMNIST are similar datasets, the differences of OrganA,C,SMNIST are the views and dataset size. Lastly, OCTMNIST Retina OCT images dataset. Figure 5.3: Comparison of Δl _l and Δs _s as a function of the number of classes k and dataset size n, for ϵ=2,4,8ε=2,4,8. (a) LDP-CP-L - ΔL _L (b) LDP-CP-S - ΔS _S Figure 5.4: CP correction terms ΔL,ΔS _L, _S as a function of ϵε privacy parameter across different dataset configurations of n and k without the shuffle model. (a) LDP-CP-L - ΔL _L (b) LDP-CP-S - ΔS _S Figure 5.5: CP correction terms ΔL,ΔS _L, _S as a function of ϵeffε^eff privacy parameter across different dataset configurations of n and k with the shuffle model. Utility-Coverage Tradeoff. Theorems 5.4.1 and 5.4.2 state that LDP-CP-L and LDP-CP-S are ϵε-LDP with a conformal prediction coverage guarantee correction term of ΔL(n,β(ϵ,k),δ) _L(n,β(ε,k),δ) and ΔS(n,ϵ,δ) _S(n,ε,δ), which depend on the number of samples n, the privacy ϵε, δ, and ΔL _L also depends on the number of classes k. Figure 5.5 explores the trade-offs of ϵ,n,kε,n,k on the conformal prediction correction terms ΔL,ΔS _L, _S, on different configurations and various medical image datasets [108] respectively. The results show both the finite-sample practicality and theoretical asymptotic performance of LDP-CP on existing medical datasets that are medium in size and simulated scenarios with growing n. For the majority of the evaluated datasets with ϵ=3ε=3, LDP-CP-L maintains correction terms that become negligible for α=0.1α=0.1 and higher (1−α1-α is the coverage). ΔL,ΔS→0 _L, _S 0 as n→∞n ∞. While as the number of samples grows ΔS _S goes to 0, for the evaluated medical datasets (with n fixed) ΔS>ΔL _S> _L, i.e. being inferior to ΔL _L, and is therefore more applicable to datasets with larger calibration sets. Figure 5.5 covers the same experiment setup, this time when the shuffle model is deployed and presents the correction terms as a function of the effective privacy. The effective privacy ϵeffε^eff ranges between 0 and 0.2, which is much more practical then the ϵ≥3ε≥ 3 acquired without the shuffle model. Figure 5.3 compares ΔS _S and ΔL _L as a function of n, k, and ϵε. Results show that roughly speaking for n≥105→ΔS≤ΔLn≥ 10^5 _S≤ _L. In addition, as the number of classes k increases (particularly for large values of k, e.g., 100 or 1000), LDP-CP-S dominates, whereas LDP-CP-L exhibits deteriorating performance. Conformal Prediction Results. Table 5.1 reports the mean size and coverage when applying conformal prediction in a non ϵε-LDP setting (Not-Private-CP), and when satisfying the local differentially private property using LDP-CP-S,L. The LDP-CP-S-L method has two variations where the first consists of calibration using 1−α1-α and getting p(y∈Cq(x))≥ 1−α−Δp(y∈ C_q(x))\;≥\;1-α- (Theorems 5.4.1,5.4.2). The second variation needs to satisfy p(y∈Cq(x))≥ 1−αp(y∈ C_q(x))\;≥\;1-α and therefore uses 1−α+Δ1-α+ in the calibration phase. For the experiment in Table 5.1 we used ϵ=4ε=4 and α=0.1α=0.1 over 100 different data splits (seeds). Table 5.1 also provides the effective privacy ϵeffε^eff per dataset to show increased practicality when the shuffle model is incorporated. Dataset Method HPS APS size ↓ coverage (%) size ↓ coverage (%) Not-Private-CP 2.57 ± 0.03 90.06 ± 0.99 2.61 ± 0.03 90.06 ± 0.97 LDP-CP-L 2.56 ± 0.04 89.99 ± 1.01 2.61 ± 0.03 90.02 ± 1.01 OCTMNIST LDP-CP-S 2.58 ± 0.04 90.21 ± 0.63 2.67 ± 0.08 90.84 ± 1.45 (ϵeff=0.038ε^eff=0.038) LDP-CP-L* 2.76 ± 0.04 92.22 ± 0.92 2.79 ± 0.03 92.28 ± 0.87 LDP-CP-S* 2.97 ± 0.07 94.38 ± 0.81 2.99 ± 0.06 94.35 ± 0.70 Not-Private-CP 5.55 ± 0.02 90.00 ± 0.24 5.58 ± 0.02 89.96 ± 0.24 LDP-CP-L 5.54 ± 0.02 89.97 ± 0.29 5.58 ± 0.02 89.97 ± 0.27 TissueMNIST LDP-CP-S 6.12 ± 0.01 95.35 ± 0.07 5.61 ± 0.09 90.32 ± 0.91 (ϵeff=0.026ε^eff=0.026) LDP-CP-L* 5.71 ± 0.02 91.68 ± 0.27 5.76 ± 0.02 91.70 ± 0.25 LDP-CP-S* 6.12 ± 0.01 95.35 ± 0.07 5.83 ± 0.05 92.32 ± 0.45 Not-Private-CP 1.93 ± 0.05 90.09 ± 0.66 2.35 ± 0.05 90.10 ± 0.55 LDP-CP-L 1.88 ± 0.07 89.49 ± 0.93 2.30 ± 0.09 89.63 ± 0.91 OrganSMNIST LDP-CP-S 1.61 ± 0.21 84.81 ± 4.48 2.09 ± 0.07 87.38 ± 1.37 (ϵeff=0.080ε^eff=0.080) LDP-CP-L* 2.77 ± 0.22 95.35 ± 0.74 3.35 ± 0.22 95.45 ± 0.74 LDP-CP-S* 3.90 ± 0.03 97.75 ± 0.06 4.75 ± 0.03 98.40 ± 0.07 Not-Private-CP 1.19 ± 0.02 89.99 ± 0.46 1.61 ± 0.02 90.02 ± 0.38 LDP-CP-L 1.31 ± 0.00 92.17 ± 0.09 1.60 ± 0.03 89.94 ± 0.54 OrganAMNIST LDP-CP-S 1.15 ± 0.05 88.89 ± 0.96 1.67 ± 0.21 90.35 ± 3.10 (ϵeff=0.049ε^eff=0.049) LDP-CP-L* 1.43 ± 0.03 93.62 ± 0.34 1.89 ± 0.05 93.51 ± 0.51 LDP-CP-S* 1.88 ± 0.19 96.52 ± 0.82 2.44 ± 0.19 96.80 ± 0.60 Not-Private-CP 1.18 ± 0.03 89.99 ± 0.71 1.56 ± 0.03 90.02 ± 0.65 LDP-CP-L 1.30 ± 0.00 91.96 ± 0.13 1.52 ± 0.04 89.36 ± 0.91 OrganCMNIST LDP-CP-S 0.95 ± 0.08 82.44 ± 2.74 1.39 ± 0.07 86.59 ± 2.44 (ϵeff=0.081ε^eff=0.081) LDP-CP-L* 1.63 ± 0.10 95.21 ± 0.74 2.05 ± 0.13 95.21 ± 0.80 LDP-CP-S* 2.47 ± 0.02 98.18 ± 0.06 2.90 ± 0.04 98.15 ± 0.10 Table 5.1: Calibration results for HPS and APS conformal scores across various datasets, using ϵ=4ε=4, ϵeff=ϵnε^eff= ε n, and α=0.1α=0.1 on 100 different seeds. Figure 5.6: Size of prediction set (left) and coverage (right) as a function of the privacy ϵε (bottom x-axis) and effective privacy ϵeffε^eff (top x-axis). We show the (mean ± std) on TissueMNIST and APS score. Next, we experimented with different values of ϵε, namely ϵ∈[1,8]ε∈[1,8]. Recall that with the shuffle model the effective privacy is ϵeff=ϵnε^eff= ε n. Figure 5.6 shows the coverage, size of the prediction set, and effective privacy on the TissueMNIST dataset. As expected as ϵε grows, Δ decreases and CP results get closer to the Non-Private-CP. In addition, LDP-CP-L performs on a par with the Non-Private-CP for all ϵε’s, and LDP-CP-S for ϵeff≥0.03ε^eff≥ 0.03 (ϵ≥4ε≥ 4), showcasing their true applicability. Chapter 6 Unsupervised Target Domain Confidence Calibration In this chapter, we explore the challenge of confidence calibration in Deep Neural Networks (DNNs) when transferring models from a source domain to an unlabeled target domain. While DNNs exhibit high accuracy in classification and detection tasks under well-supervised conditions, real-world applications often involve domain shifts where target domain labels are unavailable. Traditional calibration methods rely on labeled data to align model confidence with true probabilities, but these approaches fall short under unsupervised domain adaptation (UDA) scenarios. We introduce a novel method for directly calibrating model predictions on target domain data, demonstrating its effectiveness in addressing the pitfalls of existing approaches. 6.1 Problem Statement Deep Neural Networks (DNN) have shown remarkable accuracy in tasks such as classification and detection when sufficient data and supervision are present. In practical applications, it is crucial for models not only to be accurate, but also to indicate how much confidence users can have in their predictions. DNNs generate confidence scores that can serve as a rough estimate of the likelihood of correct classification, but these scores do not guarantee a match with the actual probabilities [33]. Neural networks tend to be overconfident in their predictions, despite having higher generalization accuracy, due to the possibility of overfitting on negative log-likelihood loss without affecting classification error [33, 49, 36]. A classifier is said to be calibrated with respect to a dataset sampled from a given distribution if its predicted probability of being correct matches its true probability. Various methods have been introduced to address the issue of over-confidence. Network calibration can be performed in conjunction with training (see e.g. [63, 64, 115]). Post-hoc scaling methods for calibration, such as Platt scaling [80], isotonic regression [110], and temperature scaling [33], are commonly employed. These techniques apply calibration as post-processing, using a hold-out validation set to learn a calibration map that adjusts the model’s confidence in its predictions to become better calibrated. The implementation of deep learning systems on real-world problems is hindered by the decrease in performance when a network trained on data from one domain is applied to data from a different domain where the distribution of features changes across domains (see e.g. [61]). This is known as the domain shift problem. In an Unsupervised Domain Adaptation (UDA) setup we assume the availability of data from the target domain but without annotation. There is a plethora of UDA methods based on strategies such as adversarial training methods that aim to align the distributions of the source and target domains [27], or self-training algorithms based on computing pseudo labels for the target domain data [120]. In the following section we tackle the problem of calibrating predicted probabilities when transferring a trained model from a source domain to a target domain without any given labels. Our major contributions include the following: • We show that current UDA calibration methods which are all based on the source domain data, rely on an overly optimistic estimation of the target accuracy. Thus they can’t well handle the domain shift problem. • We propose a calibration method that is directly applied to the target domain data, based on a realistic estimation of the accuracy of the adapted model on the target domain. • We show that previously proposed UDA calibration methods don’t work at all and thus in this study we propose the first effective method for calibrating a network obtained by an unsupervised domain adaptation. The study described in this chapter was published in [78]. 6.2 Calibration on the Target Domain in Unsupervised Domain Adaptation Table 6.1: Comparison of calibration methods for unsupervised domain adaptation (UDA). Calibration Method Designed for Works without Works on Approach Granularity domain shift target label target data Temp. Scaling [33] × × × – Instance level CPCS[69], TransCal[97] ✓ ✓ × Importance weight estimation Instance level UTDC (proposed) ✓ ✓ ✓ Estimates target accuracy Dataset level Our method involves calibrating the adapted network directly on the target data. While applying the network on the target domain data allows us to compute its confidence, we cannot determine its accuracy. Thus, the challenge is to find a reliable estimate of the network accuracy on the target domain. Our approach is based on the observation that when calibrating by minimizing the adaECE score, we do not need to know whether each individual prediction is correct. Instead, we only need to determine the mean accuracy for each bin. Fortunately, there are techniques which given a trained network, can estimate the network accuracy on data samples from a new domain without access to their labels [16, 32, 28, 109]. We next suggest a simple, intuitive, and very effective method that calibrates the network directly on the target domain. We first compute the overall network accuracy on the source data AsourceA_ source and estimate the network accuracy on the target domain (e.g, using [16]). Denote the estimated target accuracy by A~target A_ target. Next, we divide the source data into M equal-size bins according to their confidence values and compute the corresponding network accuracy Asource,mA_ source,m at each bin m. We also divide the target data into M equal-size bins according to their confidence values and estimate the binwise accuracy of the target Atarget,mA_ target,m by rescaling the binwise accuracy on the source domain in the following way: A~target,m=Asource,m⋅A~targetAsource,m=1,…,M. A_ target,m=A_ source,m· A_ targetA_ source, 28.45274ptm=1,...,M. (6.1) In the next section, we empirically show that the accuracy ratio between source and target is indeed similar across the calibration bins. The estimated network accuracy on the target data A~target A_ target obtained by an unsupervised adaptation is usually lower than its accuracy on the source data AsourceA_ source. Thus, this accuracy rescaling provides a more realistic estimation of the bin-wise network average accuracy on the target data. The accuracy ratio A~target/Asource A_ target/A_ source indicates the size of the domain gap or the difficulty of the adaptation task [121]. Let Ctarget,mC_target,m be the bin-wise network average confidence values computed on the target data. Substituting the estimated accuracy term, based on the source labeled data (6.1) into the adaECE definition, yields the following adaECE measure for the target domain in a UDA setup: UDA-adaECE=1M∑m=1M|A~target,m−Ctarget,m|. UDA -adaECE= 1M _m=1^M | A_ target,m-C_ target,m |. (6.2) Algorithm 9 Unsupervised Target Domain Calibration (UTDC) input: A labeled validation set from the source domain, an unlabeled dataset from the target domain, and a k-class classifier that was adapted to the target domain. - Compute the source accuracy AsourceA_ source and estimate the target accuracy A~target A_ target using a target accuracy estimation technique. - Divide the source points into M equal size sets based on their confidence and compute the binwise mean accuracy: Asource,mA_ source,m, m=1,…,Mm=1,...,M. - Divide the target points into M equal size sets B1,…,BMB_1,...,B_M based on their confidence. for each candidate value of T do - Compute the binwise mean confidence on the target: Ctarget,m(T)=1|Bm|∑x∈Bmmaxi=1kexp(zx,i/T)∑j=1kexp(zx,j/T)m=1,…,M.C_ target,m(T)= 1|B_m| _x∈ B_m _i=1^k (z_x,i/T) _j=1^k (z_x,j/T) 28.45274ptm=1,...,M. s.t. zx,1z_x,1,…,zx,kz_x,k are the logits computed by the network that is fed by x∈Bmx∈ B_m. - Compute the adaECE score as a function of T: UDA-adaECE(T)=1M∑m=1M|Asource,m×A~targetAsource−Ctarget,m(T)|UDA -adaECE(T)= 1M _m=1^M |A_ source,m× A_ targetA_ source-C_ target,m(T) | 51.21504pt output: The optimal temperature: T^=argminTUDA-adaECE(T) T= _TUDA -adaECE(T) For each calibration method whose parameters can be found by minimizing the adaECE measure, we can form a UDA variant in which UDA-adaECE is minimized instead of adaECE. Examples of these calibration methods include Temperature Scaling (TS), Vector Scaling, Matrix Scaling [33], Mix-n-Match [113], Weight Scaling [25], and others. We next demonstrate the UDA calibration principle in the case of TS calibration. We can determine the temperature that minimizes the UDA-adaECE measure (6.2) by conducting a grid search on the possible values. Given the division of the target data into bins, we can compute the binwise average confidence after temperature calibration by T on the target Ctarget,m(T)C_ target,m(T). We can then define the following temperature-dependent adaECE scores: UDA-adaECE(T)=1M∑m=1M|A~target,m−Ctarget,m(T)|.UDA -adaECE(T)= 1M _m=1^M | A_ target,m-C_ target,m(T) |. (6.3) The optimal temperature is thus obtained by applying a grid search to find T that minimizes UDA-adaECE(T)(T). The proposed Unsupervised Target Domain Calibration (UTDC) algorithm is summarized in Algorithm Box 9. A major component of the UTDC is estimating the target domain accuracy. based on unlabeled target domain data. We next describe several recently suggested estimation algorithms. Deng et al. [16] suggested learning a dataset-level regression problem. The first step is to augment the source domain validation set, denoted by DsD_s, using various visual transformations such as resizing, cropping, horizontal and vertical flipping, Gaussian blurring, and others. We then create n meta-datasets, denoted as D1,…,DnD_1,...,D_n (in our implementation we set n=50n=50). This process preserves the labels so we can compute the model’s accuracy on these datasets, denoted by a1,…,ana_1,...,a_n. Each dataset DiD_i is represented as a Gaussian distribution using its mean vector μi _i and its diagonal covariance matrix Σi _i. Let FiF_i be the Fréchet distance [18] between the Gaussian representations of DsD_s and DiD_i. FiF_i measures the domain gap between the original dataset DsD_s and DiD_i. Next, a linear regression model is fitted to the dataset (F1,a1),…,(Fn,an)(F_1,a_1),...,(F_n,a_n) in the form of a^=w⋅F+b a=w· F+b. Finally, the linear regression model is employed to predict the accuracy of the network on the unlabeled data from the target domain. Another method is Average Thresholded Confidence (ATC) [28] which first selects a threshold t whose error in the source domain matches the expected number of points whose confidence is below t. Next, ATC predicts the error on the target domain which is expressed as the fraction of unlabeled points that obtain a confidence value below that threshold t. Let p^(x)=maxi(p=i|x) p(x)= _i(p=i|x) be the network confidence and let y y be the network prediction. A threshold t is calculated to satisfy the equality Ex∼source1p^(x)≤t=E(x,y)∼source1y^≠yE_x source1_\ p(x)≤ t\=E_(x,y) source1_\ y≠ y\. The estimated target accuracy is the expectation Ex∼target1p^(x)≤tE_x target1_\ p(x)≤ t\. Finally, the Projection Norm (PN) method [109] uses the model predictions to pseudo-label the test samples and then trains a new model on the pseudo-labels. The discrepancy between the parameters of the new and original models yields the predicted error of the target domain data. In Section 6.3 we compare the UTDC’s calibration performance when using each of the target accuracy prediction methods described above. 6.3 Experiments In this section, we evaluate the capabilities of our UTDC technique to calibrate a network on a target domain after applying a UDA procedure. Compared methods. We compared our method to six baselines: (1) Uncalibrated - The adapted classifier as is, without any post-hoc calibration; (2-4) Source-TS, Source-VS and source-MS - The adapted network was calibrated by either Temperature Scaling (TS), Vector Scaling (VS) or Matrix scaling (MS) [33] using the labeled validation set of the source domain; (5) CPCS [69], and (6) TransCal [97], importance weighted UDA calibrators. We also report Oracle results where TS calibration was applied to the labeled data from the target domain (denoted by Target-TS) and an Oracle version of our approach (denoted by UTDC*) where we used the exact accuracy of the adapted model on the target data instead of estimating it. Table 6.2: AdaECE results on Office-home (with the lowest in bold) on various UDA classification tasks and models with different calibration methods. UDA Method A→RA→ R A→CA→ C A→PA→ P C→RC→ R C→PC→ P C→AC→ A P→RP→ R P→CP→ C P→AP→ A Avg Uncalibrated 22.23 42.62 30.49 25.18 28.25 33.69 20.32 40.46 38.85 31.34 Source-TS 8.09 24.43 14.89 10.00 14.17 13.85 11.14 27.42 26.60 16.73 Source-VS 10.54 27.54 19.51 12.12 14.65 15.78 11.27 31.55 27.46 18.94 Source-MS 28.62 47.87 35.74 31.62 31.54 40.43 23.59 43.90 40.56 35.99 CPCS 15.84 49.78 23.42 14.02 16.60 18.45 6.31 49.21 25.62 24.36 CDAN+E TransCal 6.01 27.30 9.46 16.67 16.81 21.69 19.90 41.23 39.71 22.09 UTDC 4.46 9.74 7.53 8.36 5.91 8.08 10.45 7.46 9.37 7.93 UTDC* 4.30 5.93 7.41 7.85 4.62 10.16 10.76 4.55 9.54 7.24 Target-TS 3.97 5.05 7.19 4.07 4.39 7.07 2.32 4.39 8.57 5.22 Uncalibrated 19.90 39.19 26.75 24.47 26.33 33.53 20.25 40.06 39.25 29.97 Source-TS 6.90 19.80 7.93 6.54 7.01 16.01 15.68 27.87 30.97 15.41 Source-VS 10.15 25.83 15.31 12.13 10.70 17.90 14.69 32.40 31.64 18.97 Source-MS 30.78 52.03 38.39 35.44 35.45 44.21 26.40 45.87 43.33 39.10 CPCS 13.90 50.16 21.32 3.62 7.25 34.74 25.86 22.66 27.97 23.05 DANN+E TransCal 7.21 27.42 12.36 17.81 15.43 29.93 24.64 46.61 45.83 25.25 UTDC 4.14 5.86 5.47 10.28 3.89 6.67 15.33 5.70 12.65 7.78 UTDC* 2.68 4.70 4.37 8.55 4.00 4.53 14.60 3.97 6.16 5.95 Target-TS 2.68 2.76 3.67 2.24 3.16 2.99 1.15 1.62 4.55 2.76 Uncalibrated 16.82 31.28 23.11 17.22 20.46 27.38 15.88 33.81 30.13 24.01 Source-TS 6.33 16.41 13.22 2.83 5.00 15.82 10.91 29.09 23.61 13.69 Source-VS 10.03 25.58 15.86 8.10 8.23 15.18 11.86 33.08 27.24 17.24 Source-MS 31.61 50.68 41.31 34.23 36.48 44.23 25.49 44.75 40.17 38.77 CPCS 8.89 33.56 19.99 25.29 9.62 12.82 16.87 27.49 45.93 22.27 DANN TransCal 7.63 29.15 22.20 22.64 22.97 37.66 26.11 50.85 47.53 29.64 UTDC 5.15 4.87 11.24 8.63 5.23 15.08 18.62 12.62 11.23 10.30 UTDC* 2.80 5.49 6.21 6.20 3.38 3.44 12.61 5.00 4.67 5.53 Target-TS 2.45 2.38 4.65 2.08 1.73 2.16 1.22 2.35 2.92 2.44 Datasets. We report experiments on four standard real-world domain adaptation benchmarks, Office-home [92], Office-31 [84], VisDa-2017 [72], and DomainNet [71]. Office-home includes four domains - Art, Real-World, Clipart and Product, represented as A, R, C, and P in the experiments. Office-31 contains three domains - Amazon, Webcam and DSLR, denoted A, W, and D. VisDa-2017 is a simulation-to-real dataset for domain adaptation with over 280,000 images across 12 categories. DomainNet has six domains - Clipart, Infograph, Painting, Quickdraw, Real and Sketch, denoted C, I, P, Q, R, and S. Table 6.3: AdaECE results on Office-31 (with the lowest in bold) on various UDA classification tasks and models with different calibration methods. UDA Method Method A→WA\!→\!W A→DA\!→\!D W→AW\!→\!A W→DW\!→\!D D→AD\!→\!A D→WD\!→\!W Avg Uncalibrated 11.5 10.53 29.63 1.21 29.08 1.33 13.88 Source-TS 6.03 7.43 33.21 0.86 27.25 2.12 12.82 Source-VS 3.74 7.10 33.75 1.52 32.98 1.42 13.42 Source-MS 12.15 16.72 30.76 1.02 29.99 1.38 15.34 CPCS 9.67 12.66 33.47 1.11 28.16 2.18 14.54 CDAN+E TransCal 3.78 9.45 34.43 1.27 33.68 1.56 14.03 UTDC 4.19 5.18 5.15 1.20 5.14 2.18 3.84 UTDC* 3.82 5.18 5.09 1.13 5.36 2.18 7.13 Target-TS 3.44 4.67 3.32 0.75 3.20 0.89 2.71 Uncalibrated 13.05 13.55 28.29 0.87 27.15 1.68 14.10 Source-TS 5.18 9.29 26.93 1.31 26.44 2.44 11.93 Source-VS 4.63 8.24 36.64 0.87 31.35 1.55 13.88 Source-MS 18.01 14.02 31.10 1.09 28.51 1.51 15.71 CPCS 15.58 6.81 33.97 1.99 32.69 1.14 15.36 DANN+E TransCal 7.98 5.63 34.53 1.57 31.12 1.59 13.74 UTDC 5.25 5.33 8.99 1.40 12.26 2.41 5.94 UTDC* 4.87 6.10 6.86 1.40 6.53 2.44 4.70 Target-TS 3.98 4.77 2.87 0.85 2.80 0.82 2.68 Uncalibrated 10.66 12.59 23.03 1.77 24.43 2.93 12.57 Source-TS 3.89 7.17 29.58 0.98 30.71 4.43 12.79 Source-VS 3.88 7.64 34.50 1.44 32.31 2.84 13.77 Source-MS 21.06 24.70 28.81 1.35 28.45 1.30 17.61 CPCS 16.96 10.10 33.69 2.61 35.39 4.80 17.26 DANN TransCal 10.36 15.62 87.02 2.31 45.79 6.00 27.85 UTDC 3.71 8.70 5.14 2.61 9.26 5.23 5.78 UTDC* 5.04 7.52 5.54 2.61 12.25 6.54 6.58 Target-TS 3.53 4.12 2.79 0.97 3.19 1.94 2.76 Implementation details. We followed the experiment setup described in [97] and used their code to implement CPCS and TransCal baselines. Following [97], we implemented three different UDA techniques; namely, DANN [27], DANN+E and CDAN+E [56]. The performance of more recent UDA models (e.g. [53, 44, 14]) on the target domain of the evaluated datasets is slightly better but is still much worse than the performance on the source domain. In most experiments we used the Meta target domain accuracy estimation [16] unless stated otherwise. We provide a code implementation of our method for reproducibility111https://github.com/cobypenso/unsupervised-target-domain-calibration. Table 6.4: adaECE results on VisDA Task S→RS→ R, for various calibration methods. Method DANN DANN+E CDAN+E Avg Uncalibrated 33.23 31.79 29.88 31.63 Source-TS 26.54 18.66 23.38 34.29 Source-VS 38.22 36.96 28.48 34.55 Source-MS 41.19 38.17 30.87 36.74 CPCS 31.86 11.08 26.88 23.27 TransCal 43.52 35.93 36.71 38.72 UTDC 13.07 6.61 3.85 7.84 UTDC* 2.31 1.94 2.57 2.27 Target-TS 2.02 1.84 2.21 2.02 Table 6.5: adaECE results on DomainNet for various UDA classification tasks and models with different calibration methods. UDA Method S→RS\!\!→\!\!R S→PS\!\!→\!\!P P→RP\!\!→\!\!R P→SP\!\!→\!\!S R→SR\!\!→\!\!S R→PR\!\!→\!\!P Avg Uncalibrated 14.65 18.70 18.06 22.98 19.13 13.77 17.88 Source-TS 12.68 14.48 11.51 12.76 13.56 9.60 12.39 Source-VS 10.70 9.56 11.49 14.94 13.35 9.31 11.56 Source-MS 22.24 25.28 23.43 30.93 22.55 18.07 23.75 CDAN+E CPCS 9.41 11.20 13.26 17.06 17.16 11.86 13.32 TransCal 12.50 20.82 16.41 28.85 36.70 28.23 23.92 UTDC 6.06 5.17 6.48 4.75 8.85 8.32 6.61 UTDC* 5.07 6.78 4.86 3.56 5.19 6.86 5.38 Target-TS 1.31 1.35 2.18 1.39 1.25 1.07 1.42 Uncalibrated 15.03 17.77 17.57 24.54 21.08 16.63 18.77 Source-TS 10.12 12.20 10.31 11.75 11.76 10.69 11.14 Source-VS 9.71 14.25 11.85 19.42 16.88 12.15 14.04 Source-MS 23.68 28.77 24.18 35.03 24.94 20.91 26.25 DANN+E CPCS 13.20 6.41 12.51 12.81 7.73 10.95 10.60 TransCal 14.56 19.85 16.14 29.19 34.98 28.96 23.95 UTDC 6.39 6.07 6.54 6.84 11.24 11.94 8.17 UTDC* 3.97 5.72 5.23 6.64 6.73 8.32 6.10 Target-TS 1.24 1.19 1.60 1.03 1.10 0.84 1.17 Uncalibrated 10.98 13.52 12.65 18.04 15.42 10.96 13.59 Source-TS 7.33 8.63 9.50 10.11 10.99 9.15 9.29 Source-VS 8.92 14.43 11.21 16.90 15.86 10.86 13.03 Source-MS 22.51 27.48 21.97 31.46 24.53 19.72 24.61 DANN CPCS 7.02 7.37 14.60 15.83 15.42 8.88 11.52 TransCal 14.83 22.09 16.38 30.37 37.84 29.92 25.24 UTDC 5.82 5.84 6.30 9.24 5.80 7.53 6.76 UTDC* 4.34 5.46 4.71 7.34 6.53 6.81 5.87 Target-TS 1.07 1.25 1.06 0.90 1.53 1.61 1.24 Calibration results. Tables 6.2, 6.3, 6.4 and 6.5 report the calibration results (computed by adaECE with 15 bins) on Office-home, Office-31, VisDA, and DomainNet respectively. The results show that UTDC achieved significantly better results than the baseline methods on all tasks. The calibration obtained by previous IW-based methods was slightly better (but in some cases even worse) than a network with no calibration or a network that was calibrated on the source domain. In contrast, the adaECE score obtained by UTDC was almost as good as the adaECE obtained by an oracle that had access to the labels of the domain samples. In addition to the adaECE evaluation measure, Table 6.6 reports the average calibration results over all Office-home tasks, using three other calibration metrics: ECE, Negative Log-Likelihood (NLL) and Brier Score (BS) [7]. The same trends as above were observed. Table 6.6: Calibration metrics results of various UDA calibration methods on the Office-home tasks. CDAN+E DANN+E DANN method BS NLL ECE BS NLL ECE BS NLL ECE Uncalibrated 0.74 3.40 31.32 0.76 3.07 29.92 0.75 2.75 24.08 Source-TS 0.65 2.18 16.79 0.67 2.21 15.40 0.71 2.37 13.71 CPCS 0.71 3.48 24.46 0.72 3.08 23.12 0.76 2.87 22.37 TransCal 0.69 2.70 22.12 0.73 3.08 25.22 0.81 3.72 29.71 UTDC 0.62 1.95 8.01 0.64 2.01 7.81 0.69 2.26 10.35 UTDC* 0.62 1.95 7.21 0.63 1.99 5.94 0.68 2.18 5.53 Target-TS 0.61 1.92 5.41 0.63 1.96 2.72 0.68 2.14 2.78 Table 6.7: Computed temperature on various UDA Office-home tasks, and calibration methods using CDAN+E. UDA Method A→RA\!\!→\!\!R A→CA\!\!→\!\!C A→PA\!\!→\!\!P C→RC\!\!→\!\!R C→PC\!\!→\!\!P C→AC\!\!→\!\!A P→RP\!\!→\!\!R P→CP\!\!→\!\!C P→AP\!\!→\!\!A Avg Source-TS 1.96 2.02 2.02 1.87 1.90 2.06 1.63 1.72 1.68 1.87 CPCS 1.46 0.57 1.49 1.68 1.75 2.05 1.93 0.50 1.73 1.46 TransCal 2.12 1.86 2.39 1.50 1.74 1.62 1.03 0.96 0.95 1.57 CDAN+E UTDC 2.27 2.90 2.91 1.97 2.44 2.54 1.67 2.93 2.89 2.50 UTDC* 2.29 3.21 2.68 2.00 2.62 2.30 1.65 3.41 2.90 2.56 Target-TS 2.36 3.61 2.73 2.42 2.73 2.81 2.24 3.49 3.37 2.86 A→RA→ R A→CA→ C A→PA→ P C→RC→ R C→PC→ P C→AC→ A P→RP→ R P→CP→ C P→AP→ A R→AR→ A R→CR→ C R→PR→ P Figure 6.1: Average accuracy on Office-home tasks for the three UDA techniques (DANN, DANN+E, CDAN+E). 6.4 Analysis We next illustrate and analyze several key features of the proposed method. Accuracy gap between source and target. To gain a better understanding of the reasons why our method performs better than IW based methods, we first discuss the accuracy of the adapted models on the source and target domains. Figure 6.1 presents the accuracy on the source and target domains for three UDA techniques. It shows that even after adaptation to the target, the model’s performance on the source samples is consistently better than its performance on the target samples, especially in cases of large domain gaps. Hence, using the network accuracy on the source to estimate the network’s accuracy on the target while minimizing the ECE measure is misleading because the over-optimistic accuracy estimation leads to a scaling temperature that is too small. Table 6.7 compares the optimal temperatures computed by the calibration methods. In all the baseline methods the computed calibration temperature was lower than the optimal value. This results in poorer calibration performance, as seen in Tables 6.2, 6.3, 6.4, and 6.5. By contrast, the temperature computed by all the UTDC variants was much closer to the optimal temperature computed by the Oracle method that had access to the target labels. Figure 6.1 also presents the estimated accuracy of the adapted model on the target domain. This estimation is close to the true accuracy. Thus, when it is combined with the confidence computed on the target domain, we obtain a calibrated mode. (a) CDAN+E (b) DANN+E (c) DANN Figure 6.2: adaECE results as a function of the correction ratio R on Office-Home, A→CA→ C task. (a) CDAN+E (b) DANN+E (c) DANN Figure 6.3: Accuracy of k-th percentile source images based on their probability of being classified as target [97], compared to target accuracy (Office-home, A→CA→ C). (a) CDAN+E (b) DANN+E (c) DANN Figure 6.4: Accuracy per bin for source and target images. The results are shown on the Office-home C→PC→ P task. Sensitivity of UTDC to the target accuracy prediction. UTDC is based on estimating the binwise average network accuracy on the target domain data from the labeled source domain data. This estimation is done by computing the ratio A~target/Asource A_ target/A_ source between the estimated target accuracy and the source accuracy. We next analyze the sensitivity of our calibration method to errors in estimating AtargetA_ target. Let R(true)=Atarget/AsourceR( true)=A_ target/A_ source and R(estimated)=A~target/AsourceR( estimated)= A_ target/A_ source be the true and estimated ratio used by UTDC* and UTDC respectively. In principle, any number 0<R0\!<\!R can be used to obtain an estimation of the binwise target accuracy: A~target,m=Asource,m⋅R A_ target,m=A_ source,m· R. We can thus find the temperature that minimizes the adaECE function on the target data as a function of R: T^(R)=argminTadaECER(T) T(R)= _TadaECE_R(T) where adaECER(T)=1M∑m=1M|Asource,m⋅R−Ctarget,m(T)|.adaECE_R(T)= 1M _m=1^M |A_ source,m· R-C_ target,m(T) |. Figure 6.2 shows the adaECE measure on the target data after temperature scaling by T^(R) T(R) as a function of the ratio R for the task Office-home A→CA→ C. It shows that with the appropriate choice of R we can achieve the calibration level of the Oracle TS-target algorithm (the case where target labels are known). This means that the difference in accuracy is indeed the main reason for the calibration degradation caused by methods that try to calibrate the target domain using the source data. Specifically, as the ratio R drops towards R(true), the adaECE improves and approaches the Oracle TS-target calibration. In addition, the adaECE reaches a minimum near R(true) and R(estimated). Finally, there is a range of correction ratios where UTDC is better by a large margin than other baselines, thus providing a tolerance for error and resilience in estimating A~target A_ target. The problem with the IW assumption. We showed that our method achieves better results by explicitly addressing the accuracy gap between the source and target domains caused by the domain shift. Previous methods based on importance weights [69, 97] rely on re-weighting the source data based on their proximity to the target data, i.e., concentrating on source samples that resemble the target and attributing less weight to others. We computed the target similarity weights associated with each sample in the source validation set and divided them into 20%20\% percentile subsets. Figure 6.3 shows the average accuracy of each group and the average target accuracy. It shows that the source accuracy is similar in all bins regardless of the similarity to the target. Thus the IW assumption that source samples that are classified as targets are more relevant for calibrating the target prediction is wrong. Table 6.8: AdaECE results for variations of UTDC based on different methods of domain accuracy estimation. Method Office-home Office-31 VisDA DomainNet Uncalibrated 28.44 13.51 31.63 16.74 UTDC-Meta[16] 8.67 6.96 7.84 7.18 UTDC-ATC[28] 10.12 7.47 5.68 8.01 UTDC-PN [109] 11.55 7.83 10.20 8.63 UTDC* 6.24 6.13 2.27 5.78 Table 6.9: Comparison of several target domain accuracy estimation methods measured by |ACC(True)−ACC(Est)||ACC(True)-ACC(Est)|. Method Office-home Office-31 VisDA DomainNet Meta[28] 3.31 2.81 4.96 3.10 ATC[28] 5.05 3.37 3.48 4.25 PN [109] 6.26 4.85 6.30 5.91 Accuracy ratio across bins. Our method computes A~target,m A_ target,m by re-scaling Asource,mA_ source,m with the same ratio for all bins, as defined in 6.1. This estimation is based on the assumption that the accuracy ratio between the source and the target is similar across the bins. To illustrate the validity of this assumption, Figure 6.4 shows the accuracy of the adapted network at each bin, for the source and target data. Different target accuracy estimation methods. Our UTDC method requires an estimation step of the target domain accuracy without labels. In all the experiments reported above we used the Meta method [16]. We next examine combining UTDC with two other methods for target domain accuracy estimation: ATC [28] and PN [109]. We implemented 3 variations of UTDC, dubbed UTDC-Meta, UTDC-ATC, and UDTC-PN based on the estimated target accuracy that was used. We also report results for UTDC* based on the true target accuracy. Tables 6.8 and 6.9 present the average calibration results and the discrepancy between the estimated and actual accuracy, respectively. The results indicate that UTDC achieved the best calibration performance out of all the three target accuracy estimation methods examined, thus reinforcing the observed low sensitivity of UTDC to the precision of target accuracy predictions. This underscores the compatibility of UTDC with existing methods for network calibration under unsupervised domain shift. We also found that using UTDC-Meta yields better results, while UTDC-ATC exhibits improved performance and ease of implementation, since the ATC method is much simpler to implement and requires a small computational effort. Chapter 7 Discussion This chapter brings together the key findings and contributions of this dissertation, highlighting their implications for the broader field of machine learning. Throughout our research, we addressed critical challenges related to confidence calibration and uncertainty quantification, focusing on complex scenarios involving label noise, privacy constraints, and domain shifts. By developing novel methodologies and demonstrating their effectiveness across diverse settings, our work advances the reliability and interpretability of machine learning models in real-world applications. Here, we summarize our contributions, reflect on key insights, and outline promising directions for future research. 7.1 Summary of Contributions In this dissertation, we explored multiple facets of confidence calibration and uncertainty quantification in machine learning, particularly under challenging real-world conditions such as label noise, privacy constraints, and domain shift. Our work contributes novel methodologies that enhance the robustness of existing calibration and conformal prediction (CP) frameworks, thereby addressing key limitations in the current literature. First, we investigated confidence calibration in classification models trained with noisy labels, a critical issue in domains like medical imaging where obtaining accurate labels is often impractical. We demonstrated that traditional calibration methods are highly sensitive to label noise, even when network training itself remains relatively robust. To address this, we proposed a noise-aware calibration framework that effectively models label corruption using a noise transition matrix. Our results show that this approach achieves calibration performance comparable to that of clean labels, provided the noise model is well estimated. Second, we extended the conformal prediction framework to handle label noise. We introduced a procedure that adjusts the calibration threshold based on a given noise model, allowing CP to maintain valid coverage while minimizing the size of prediction sets. We derived finite-sample coverage guarantees for the uniform noise case and showed that our method significantly improves over existing noisy CP approaches in terms of efficiency and prediction set size. Third, we addressed confidence calibration under privacy constraints by developing two complementary CP approaches for locally differentially private (LDP) settings. These methods, LDP-CP-L (label perturbation) and LDP-CP-S (score perturbation), provide valid uncertainty quantification while ensuring strong privacy guarantees for individual user data. Our results highlight the trade-offs between privacy, computational feasibility, and model performance, offering guidance for selecting the appropriate method in real-world applications. Lastly, we studied network calibration in the context of unsupervised domain adaptation, where a model trained on a labeled source domain is deployed in a target domain with a different distribution. We showed that existing importance-weighting approaches fail to correct for domain shift in calibration. Instead, we proposed a method that directly calibrates using target-domain examples, leading to substantial improvements in calibration performance. 7.2 Key Insights and Implications Sensitivity of Calibration to Label Noise. Our findings reveal that calibration procedures are far more sensitive to label noise than the training process itself. This observation underscores the need for noise-resilient calibration methods, as even a small fraction of incorrect labels in the validation set can severely degrade calibration quality. Our proposed approach, which explicitly accounts for label noise via a noise transition matrix, mitigates these effects and provides reliable confidence estimates despite label corruption. Conformal Prediction Under Noisy Labels. We demonstrated that CP can be adapted to handle noisy labels by estimating a noise-free calibration threshold. Our method preserves valid coverage and improves efficiency compared to existing approaches. However, our analysis suggests that current coverage guarantees may be overly conservative, indicating room for further theoretical refinement. Additionally, we focused on noise models where label corruption is independent of input features; extending our method to more complex, feature-dependent noise processes remains an open challenge. Privacy-Preserving Uncertainty Quantification. Our work on LDP-conformal prediction bridges the gap between uncertainty quantification and privacy protection. By leveraging randomized response mechanisms, we ensured valid CP coverage while preserving user privacy. A notable challenge for future work is integrating more sophisticated LDP techniques, such as RAPPOR, into our framework to enhance robustness and scalability. Furthermore, developing a method that preserves privacy without requiring local score computation remains an important open problem. Confidence Calibration in Domain Adaptation. We showed that calibration methods relying on source-domain accuracy fail under domain shift due to differences in true label distributions. Our approach, which calibrates directly on target-domain examples, outperforms existing methods. Extending this idea to non-parametric calibration techniques such as CP, as well as to regression and segmentation tasks, is a promising avenue for future research. Additionally, addressing confidence calibration in source-free adaptation—where access to source data is not available—remains an open challenge. 7.3 Future Research Directions While our research has addressed several key challenges, many promising directions remain for future exploration: Refining Theoretical Guarantees for Noisy Conformal Prediction. Our results suggest that existing theoretical bounds for CP under noisy labels may be overly conservative. Further research into tighter finite-sample guarantees could improve the efficiency of CP methods in noisy settings. Advanced Noise Models in Calibration. Our methods assume that the noise transition matrix can be estimated with reasonable accuracy. However, in cases of highly unbalanced class distributions or feature-dependent noise, estimating this matrix remains challenging. Developing robust estimation techniques for such cases is an important open problem. Extending Privacy-Preserving CP Methods. Our current LDP-CP framework is limited by the assumption that users can either perturb labels or scores locally. A more stringent privacy setting, where both the input features and labels must remain private, presents a fundamental challenge. Designing methods that operate under such conditions while maintaining valid coverage is a crucial area for future work. Calibration in Source-Free Adaptation. In domain adaptation, we demonstrated that calibration benefits from access to target-domain examples. However, a more challenging setting is source-free adaptation, where only unlabeled target data is available. Developing effective calibration methods under such constraints is a key research direction. Applying Calibration Strategies to Other Tasks. Our work focused on classification tasks, but the principles developed here could extend to regression and segmentation problems. Exploring calibration techniques for these settings, particularly under domain shift and noisy labels, could have significant practical impact. 7.4 Conclusion This dissertation advances the understanding and application of confidence calibration and uncertainty quantification in machine learning. By addressing challenges in label noise, privacy, and domain shift, we developed robust methodologies that improve calibration performance across diverse settings. Our work not only enhances the reliability of machine learning models but also opens new avenues for research in uncertainty estimation under real-world constraints. We hope that our contributions will inspire further studies in these areas, ultimately leading to more trustworthy and interpretable AI systems. References [1] A. N. Angelopoulos, S. Bates, et al. (2023) Conformal prediction: a gentle introduction. Foundations and Trends in Machine Learning 16 (4), p. 494–591. Cited by: Chapter 1, §2.2, §4.1. [2] A. N. Angelopoulos, S. Bates, T. Zrnic, and M. I. Jordan (2022) Private prediction sets. arXiv preprint arXiv:2102.06202. Cited by: §5.1. [3] A. N. Angelopoulos, S. Bates, J. Malik, and M. I. Jordan (2021) Uncertainty sets for image classifiers using conformal prediction. International Conference on Learning Representations (ICLR). Cited by: §2.2, §4.5. [4] Apple (2017) Learning with privacy at scale. Note: Accessed: [Insert Access Date] External Links: Link Cited by: §2.5. [5] A. Beimel, K. Nissim, and U. Stemmer (2013) Private learning and sanitization: pure vs. approximate differential privacy. In International Workshop on Approximation Algorithms for Combinatorial Optimization, p. 363–378. Cited by: §5.1. [6] A. J. Bekker and J. Goldberger (2016) Training deep neural-networks based on unreliable labels. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 2682–2686. Cited by: §2.4. [7] G. W. Brier (1950) Verification of forecasts expressed in terms of probability. Monthly Weather Review 78 (1), p. 1–3. Cited by: §6.3. [8] L. Chen, H. Li, and W. Yang (2022) AnomMAN: detect anomaly on multi-view attributed networks. arXiv preprint arXiv:2201.02822. Cited by: §2.4. [9] P. Chen, G. Chen, J. Ye, P. Heng, et al. (2021) Noise against noise: stochastic label noise helps combat inherent label noise. In International Conference on Learning Representations (ICLR), Cited by: §2.4. [10] D. Cheng, T. Liu, Y. Ning, N. Wang, B. Han, G. Niu, X. Gao, and M. Sugiyama (2022) Instance-dependent label-noise learning with manifold-regularized transition matrix estimation. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2.4. [11] H. Cheng, Z. Zhu, X. Li, Y. Gong, X. Sun, and Y. Liu (2021) Learning with instance-dependent label noise: A sample sieve approach. In International Conference on Learning Representations (ICLR), Cited by: §2.4. [12] A. Cheu (2022) Differential privacy in the shuffle model: a survey of separations. External Links: 2107.11839, Link Cited by: §5.5.2. [13] J. Clarkson, W. Xu, M. i Cucuringu, and G. Reinert (2024) Split conformal prediction under data contamination. In Proceedings of the Thirteenth Symposium on Conformal and Probabilistic Prediction with Applications, Cited by: §4.1, §4.4, §4.4, §4.4, §4.5, Table 4.3, §5.1. [14] S. Cui, S. Wang, J. Zhuo, L. Li, Q. Huang, and Q. Tian (2020) Towards discriminability and diversity: batch nuclear-norm maximization under label insufficient situations. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §6.3. [15] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei (2009) Imagenet: a large-scale hierarchical image database. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 248–255. Cited by: §4.5. [16] W. Deng and L. Zheng (2021) Are labels always necessary for classifier accuracy evaluation?. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §6.2, §6.2, §6.2, §6.3, §6.4, Table 6.8. [17] Y. Dgani, H. Greenspan, and J. Goldberger (2018) Training a neural network based on unreliable human annotation of medical images. In The IEEE International Symposium on Biomedical Imaging (ISBI), Cited by: §2.4. [18] D.C. Dowson and B. V. Landau (1982) The Fréchet distance between multivariate normal distributions. Journal of Multivariate Analysis 12 (3), p. 450–455. Cited by: §6.2. [19] J. C. Duchi, M. I. Jordan, and M. J. Wainwright (2013) Local privacy and statistical minimax rates. In 2013 IEEE 54th annual symposium on foundations of computer science, p. 429–438. Cited by: §2.5. [20] C. Dwork (2006) Differential privacy. In International colloquium on automata, languages, and programming, p. 1–12. Cited by: §2.5, §5.4. [21] B. Einbinder, S. Bates, A. N. Angelopoulos, A. Gendler, and Y. Romano (2022) Conformal prediction is robust to label noise. arXiv preprint arXiv:2209.14295. Cited by: §2.4, §4.1, §4.2, §4.2, §4.3.1, §4.5, Table 4.1, §5.1. [22] Ú. Erlingsson, V. Pihur, and A. Korolova (2014) Rappor: randomized aggregatable privacy-preserving ordinal response. In Proceedings of the 2014 ACM SIGSAC conference on computer and communications security, p. 1054–1067. Cited by: §2.5. [23] K. R. M. Fernando and C. P. Tsokos (2021) Dynamically weighted balanced loss: class imbalanced learning and confidence calibration of deep neural networks. IEEE Transactions on Neural Networks and Learning Systems 33 (7), p. 2940–2951. Cited by: Chapter 1, §3.1. [24] R. Foygel Barber, E. J. Candes, A. Ramdas, and R. J. Tibshirani (2021) The limits of distribution-free conditional predictive inference. Information and Inference: A Journal of the IMA 10 (2), p. 455–482. Cited by: §2.2. [25] L. Frenkel and J. Goldberger (2022) Calibration of medical imaging classification systems with weight scaling. In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), Cited by: Chapter 1, §3.1, §3.2, §6.2. [26] M. Gaboardi, R. Rogers, and O. Sheffet (2019) Locally private mean estimation: z-test and tight confidence intervals. External Links: 1810.08054, Link Cited by: §5.3, §5.4. [27] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky (2016) Domain-adversarial training of neural networks. The Journal of Machine Learning Research 17 (1), p. 2096–2030. Cited by: §2.3, §6.1, §6.3. [28] S. Garg and S. Balakrishnan (2022) Leveraging unlabeled data to predict out-of-distribution performance. International Conference on Learning Representations (ICLR). Cited by: §6.2, §6.2, §6.4, Table 6.8, Table 6.9, Table 6.9. [29] B. Ghazi, N. Golowich, R. Kumar, P. Manurangsi, and C. Zhang (2021) Deep learning with label differential privacy. Advances in Neural Information Processing Systems (NeurIPs) 34, p. 27131–27145. Cited by: §4.1, §4.3, §5.1. [30] A. Ghosh, H. Kumar, and P. S. Sastry (2017) Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, p. 1919–1925. Cited by: §2.4. [31] J. Goldberger and E. Ben-Reuven (2017) Training deep neural-networks using a noise adaptation layer. In International Conference on Learning Representations (ICLR), Cited by: §2.4, §2.4. [32] D. Guillory, V. Shankar, S. Ebrahimi, T. Darrell, and L. Schmidt (2021) Predicting with confidence on unseen distributions. In Proc. of the IEEE International Conference on Computer Vision (ICCV), Cited by: §6.2. [33] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger (2017) On calibration of modern neural networks. In International Conference on Machine Learning (ICML), p. 1321–1330. Cited by: Chapter 1, Chapter 1, §2.1, §2.2, §2.3, §3.1, §3.1, §3.1, §3.2, §3.4, §6.1, §6.2, §6.3, Table 6.1. [34] J. Han, P. Luo, and X. Wang (2019) Deep self-learning from noisy labels. In Proc. of the IEEE International Conference on Computer Vision (ICCV), p. 5138–5147. Cited by: §2.4. [35] K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 770–778. Cited by: §3.4, §4.5. [36] M. Hein, M. Andriushchenko, and J. Bitterwolf (2019) Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 41–50. Cited by: Chapter 1, §2.3, §3.1, §6.1. [37] D. Hendrycks, M. Mazeika, D. Wilson, and K. Gimpel (2018) Using trusted data to train deep networks on labels corrupted by severe noise. In Advances in Neural Information Processing Systems (NeurIPs), p. 10477–10486. Cited by: §2.4. [38] W. Hu, Z. Li, and D. Yu (2020) Simple and effective regularization methods for training on noisily labeled data with generalization guarantee. In International Conference on Learning Representations (ICLR), Cited by: §2.4. [39] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger (2017) Densely connected convolutional networks. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 4700–4708. Cited by: §3.4. [40] J. Huang, L. Qu, R. Jia, and B. Zhao (2019) O2u-net: a simple noisy label detection approach for deep neural networks. In Proc. of the IEEE International Conference on Computer Vision (ICCV), p. 3326–3334. Cited by: §2.4. [41] Y. Huang, B. Bai, S. Zhao, K. Bai, and F. Wang (2022) Uncertainty-aware learning against label noise on imbalanced datasets. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36, p. 6960–6969. Cited by: §2.4. [42] J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ilcus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya, et al. (2019) Chexpert: a large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, p. 590–597. Cited by: Chapter 1, §3.1. [43] S. Jiang, J. Li, Y. Wang, B. Huang, Z. Zhang, and T. Xu (2022) Delving into sample loss curve to embrace noisy and imbalanced data. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36, p. 7024–7032. Cited by: §2.4. [44] Y. Jin, X. Wang, M. Long, and J. Wang (2020) Minimum class confusion for versatile domain adaptation. In Proc. of the European Conference on Computer Vision (ECCV), Cited by: §6.3. [45] P. Kairouz, K. Bonawitz, and D. Ramage (2016) Discrete distribution estimation under local privacy. In International Conference on Machine Learning (ICML), p. 2436–2444. Cited by: §2.5, §5.1, §5.4. [46] S. P. Kasiviswanathan, H. K. Lee, K. Nissim, S. Raskhodnikova, and A. Smith (2011) What can we learn privately?. SIAM Journal on Computing 40 (3), p. 793–826. Cited by: §2.5. [47] D. P. Kingma and J. Ba (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §3.4. [48] A. Krizhevsky (2009) Learning multiple layers of features from tiny images. Technical report Department of Computer Science, University of Toronto. Cited by: §4.5. [49] B. Lakshminarayanan, A. Pritzel, and C. Blundell (2017) Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems (NeurIPs) 30. Cited by: Chapter 1, §2.3, §3.1, §6.1. [50] S. Li, S. Ge, Y. Hua, C. Zhang, H. Wen, T. Liu, and W. Wang (2020) Coupled-view deep classifier learning from multiple noisy annotators. In Proceedings of the AAAI Conference on Artificial Intelligence, p. 4667–4674. Cited by: §2.4. [51] S. Li, T. Liu, J. Tan, D. Zeng, and S. Ge (2021) Trustable co-label learning from multiple noisy annotators. IEEE Transactions on Multimedia 25, p. 1045–1057. Cited by: §2.4. [52] X. Li, T. Liu, B. Han, G. Niu, and M. Sugiyama (2021) Provably end-to-end label-noise learning without anchor points. In International Conference on Machine Learning (ICML), p. 6403–6413. Cited by: §2.4, §2.4, §2.4, §3.2, §3.2, §3.3, §3.3, §3.4, Table 3.3, Table 3.4, §4.3.3, §4.3. [53] J. Liang, D. Hu, and J. Feng (2021) Domain adaptation with auxiliary target domain-oriented classifier. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §6.3. [54] Y. Lin, R. Pi, W. Zhang, X. Xia, J. Gao, X. Zhou, T. Liu, and B. Han (2023) A holistic view of label noise transition matrix in deep learning and beyond. In International Conference on Learning Representations (ICLR), Cited by: §4.3.3, §4.3. [55] T. Liu and D. Tao (2015) Classification with noisy labels by importance reweighting. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI) 38 (3), p. 447–461. Cited by: §2.4, §2.4. [56] M. Long, Z. Cao, J. Wang, and M. I. Jordan (2018) Conditional adversarial domain adaptation. Advances in Neural Information Processing Systems (NeurIPs). Cited by: §6.3. [57] C. Lu, A. N. Angelopoulos, and S. Pomerantz (2022) Improving trustworthiness of AI disease severity rating in medical imaging with ordinal conformal prediction sets. In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), Cited by: Chapter 1, §4.1. [58] C. Lu, A. Lemay, K. Chang, K. Höbel, and J. Kalpathy-Cramer (2022) Fair conformal predictors for applications in medical imaging. In Proceedings of the AAAI Conference on Artificial Intelligence, Cited by: Chapter 1, §4.1. [59] Y. Lu, Y. Zhang, B. Han, Y. Cheung, and H. Wang (2022) Label-noise learning with intrinsically long-tailed data. arXiv e-prints, p. arXiv–2208. Cited by: §2.4. [60] P. Massart (1990) The tight constant in the Dvoretzky-Kiefer-Wolfowitz inequality. The Annals of Probability, p. 1269–1283. Cited by: §4.3.2. [61] J. P. Miller, R. Taori, A. Raghunathan, S. Sagawa, P. W. Koh, V. Shankar, P. Liang, Y. Carmon, and L. Schmidt (2021) Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In International Conference on Machine Learning (ICML), Cited by: Chapter 1, §2.3, §6.1. [62] M. Minderer, J. Djolonga, R. Romijnders, F. Hubis, X. Zhai, N. Houlsby, D. Tran, and M. Lucic (2021) Revisiting the calibration of modern neural networks. Advances in Neural Information Processing Systems (NeurIPs) 34, p. 15682–15694. Cited by: Chapter 1, §3.1. [63] J. Mukhoti, V. Kulharia, A. Sanyal, S. Golodetz, P. Torr, and P. Dokania (2020) Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems (NeurIPs) 33, p. 15288–15299. Cited by: Chapter 1, §2.1, §3.1, §6.1. [64] R. Müller, S. Kornblith, and G. Hinton (2019) When does label smoothing help?. arXiv preprint arXiv:1906.02629. Cited by: Chapter 1, §3.1, §6.1. [65] M. P. Naeini, G. Cooper, and M. Hauskrecht (2015) Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 29. Cited by: §2.1. [66] K. Nguyen and B. O’Connor (2015) Posterior calibration and exploratory analysis for natural language processing models. arXiv preprint arXiv:1508.05154. Cited by: §2.1. [67] H. Olsson, K. Kartasalo, N. Mulliqi, et al. (2022) Estimating diagnostic uncertainty in artificial intelligence assisted pathology using conformal prediction. Nature Communications 13 (1), p. 7761. Cited by: Chapter 1, §4.1. [68] A. Pampari and S. Ermon (2020) Unsupervised calibration under covariate shift. arXiv preprint arXiv:2006.16405. Cited by: §2.3. [69] S. Park, O. Bastani, J. Weimer, and I. Lee (2020) Calibrated prediction with covariate shift via unsupervised domain adaptation. In International Conference on Artificial Intelligence and Statistics, Cited by: §2.3, §6.3, §6.4, Table 6.1. [70] G. Patrini, A. Rozza, A. Krishna Menon, R. Nock, and L. Qu (2017) Making deep neural networks robust to label noise: a loss correction approach. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 1944–1952. Cited by: §3.2, §3.2. [71] X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang (2019) Moment matching for multi-source domain adaptation. In Proc. of the IEEE International Conference on Computer Vision (ICCV), p. 1406–1415. Cited by: §6.3. [72] X. Peng, B. Usman, N. Kaushik, J. Hoffman, D. Wang, and K. Saenko (2017) Visda: the visual domain adaptation challenge. arXiv preprint arXiv:1710.06924. Cited by: §6.3. [73] C. Penso, L. Frenkel, and J. Goldberger (2024) Confidence calibration of a medical imaging classification system that is robust to label noise. IEEE Transactions on Medical Imaging 43 (6), p. 2050–2060. Cited by: 1st item, §3.1. [74] C. Penso, J. Goldberger, and E. Fetaya (2024) Estimating the conformal prediction threshold from noisy labels. arXiv preprint arXiv:2501.12749. Cited by: §4.1. [75] C. Penso, J. Goldberger, and E. Fetaya (2025) Conformal prediction of classifiers with many classes based on noisy labels. Note: Symposium on Conformal and Probabilistic Prediction with Applications (COPA 2025) Cited by: 5th item, §4.1. [76] C. Penso and J. Goldberger (2024) A conformal prediction score that is robust to label noise. In MICCAI, Machine Learning for Medical Imaging Workshop, Cited by: 3rd item. [77] C. Penso and J. Goldberger (2024) A joint training and confidence calibration procedure that is robust to label noise. In The IEEE International Symposium on Biomedical Imaging (ISBI), Cited by: 2nd item, §3.1. [78] C. Penso and J. Goldberger (2024) Calibration of network confidence for unsupervised domain adaptation using estimated accuracy. In ECCV, Uncertainty in Computer Vision Workshop, Cited by: 4th item, §6.1. [79] C. Penso, B. Mahpud, J. Goldberger, and O. Sheffet (2025) Privacy-preserving conformal prediction under local differential privacy. Note: Symposium on Conformal and Probabilistic Prediction with Applications (COPA 2025) Cited by: 6th item. [80] J. Platt et al. (1999) Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers 10 (3), p. 61–74. Cited by: Chapter 1, §3.1, §6.1. [81] M. Ren, W. Zeng, B. Yang, and R. Urtasun (2018) Learning to reweight examples for robust deep learning. In International Conference on Machine Learning (ICML), p. 4331–4340. Cited by: §2.4. [82] Y. Romano, M. Sesia, and E. Candes (2020) Classification with valid and adaptive coverage. Advances in Neural Information Processing Systems (NeurIPs). Cited by: §2.2, §2.2, §4.5, §5.6. [83] A. Rousseau, T. Becker, J. Bertels, M. B. Blaschko, and D. Valkenborg (2021) Post training uncertainty calibration of deep networks for medical image segmentation. In The IEEE International Symposium on Biomedical Imaging (ISBI), p. 1052–1056. Cited by: Chapter 1, §3.1. [84] K. Saenko, B. Kulis, M. Fritz, and T. Darrell (2010) Adapting visual category models to new domains. In Proc. of the European Conference on Computer Vision (ECCV), Cited by: §6.3. [85] T. Salvador, V. Voleti, A. Iannantuono, and A. Oberman (2021) Improved predictive uncertainty using corruption-based calibration. Stat 1050, p. 7. Cited by: §2.3. [86] M. Sesia, Y. Wang, and X. Tong (2023) Adaptive conformal classification with noisy labels. arXiv preprint arXiv:2309.05092. Cited by: §4.1, §4.4, §4.4, §4.4, §4.4, §4.5, Table 4.3, §5.1. [87] J. Shu, Q. Xie, L. Yi, Q. Zhao, S. Zhou, Z. Xu, and D. Meng (2019) Meta-weight-net: learning an explicit mapping for sample weighting. In Advances in Neural Information Processing Systems (NeurIPs), p. 1917–1928. Cited by: §2.4. [88] H. Song, M. Kim, D. Park, Y. Shin, and J. Lee (2022) Learning from noisy labels with deep neural networks: a survey. IEEE transactions on neural networks and learning systems 34 (11), p. 8135–8153. Cited by: Chapter 1, §3.1, §4.1. [89] D. Tanaka, D. Ikami, T. Yamasaki, and K. Aizawa (2018) Joint optimization framework for learning with noisy labels. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 5552–5560. Cited by: §2.4. [90] C. Tomani, S. Gruber, M. E. Erdem, D. Cremers, and F. Buettner (2021) Post-hoc uncertainty calibration for domain drift scenarios. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2.3. [91] P. Tschandl, C. Rosendahl, and H. Kittler (2018) The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data 5 (1), p. 1–9. Cited by: 2nd item. [92] H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan (2017) Deep hashing network for unsupervised domain adaptation. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §6.3. [93] V. Vovk, A. Gammerman, and G. Shafer (2005) Algorithmic learning in a random world. Vol. 29, Springer. Cited by: Chapter 1, §2.2, §2.2, §4.1, §4.5, §5.6. [94] R. Wang, S. Mou, X. Wang, W. Xiao, Q. Ju, C. Shi, and X. Xie (2021) Graph structure estimation neural networks. In W, p. 342–353. Cited by: §2.4. [95] T. Wang, J. Blocki, N. Li, and S. Jha (2017) Locally differentially private protocols for frequency estimation. In 26th USENIX Security Symposium (USENIX Security 17), p. 729–745. Cited by: §2.5, §5.1. [96] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers (2017) Chestx-ray8: hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 2097–2106. Cited by: 1st item. [97] X. Wang, M. Long, J. Wang, and M. Jordan (2020) Transferable calibration with lower bias and variance in domain adaptation. In Advances in Neural Information Processing Systems (NeurIPs), Cited by: §2.3, §2.3, Figure 6.3, §6.3, §6.3, §6.4, Table 6.1. [98] S. L. Warner (1965) Randomized response: a survey technique for eliminating evasive answer bias. Journal of the American statistical association 60 (309), p. 63–69. Cited by: §2.5, §2.5, §5.1. [99] J. Wei, Z. Zhu, H. Cheng, T. Liu, G. Niu, and Y. Liu (2021) Learning with noisy labels revisited: a study using real-world human annotations. arXiv preprint arXiv:2110.12088. Cited by: §4.5, §4.5. [100] T. Wei, J. Shi, W. Tu, and Y. Li (2021) Robust long-tailed learning under label noise. arXiv preprint arXiv:2108.11569. Cited by: §2.4. [101] A. Woloshuk, S. Khochare, A. F. Almulhim, A. T. McNutt, D. Dean, D. Barwinska, M. J. Ferkowicz, M. T. Eadon, K. J. Kelly, K. W. Dunn, et al. (2021) In situ classification of cell types in human kidney tissue using 3D nuclear staining. Cytometry Part A 99 (7), p. 707–721. Cited by: §5.6. [102] X. Xia, T. Liu, B. Han, N. Wang, M. Gong, H. Liu, G. Niu, D. Tao, and M. Sugiyama (2020) Part-dependent label noise: towards instance-dependent label noise. In Advances in Neural Information Processing Systems (NeurIPs), Cited by: §2.4. [103] X. Xia, T. Liu, N. Wang, B. Han, C. Gong, G. Niu, and M. Sugiyama (2019) Are anchor points really indispensable in label-noise learning?. In Advances in Neural Information Processing Systems (NeurIPs), p. 6835–6846. Cited by: §2.4. [104] M. Xu, Y. Zhou, C. Jin, M. De Groot, D. C. Alexander, N. P. Oxtoby, and J. Jacob (2023) MisMatch: calibrated segmentation via consistency on differential morphological feature perturbations with limited labels. IEEE transactions on medical imaging 42 (10), p. 2988–2999. Cited by: Chapter 1, §3.1. [105] C. Xue, L. Yu, P. Chen, Q. Dou, and P. Heng (2022) Robust medical image classification from noisy labeled data with global and local representation guided co-training. IEEE Transactions on Medical Imaging 41 (6), p. 1371–1382. Cited by: Chapter 1, §3.1, §3.4, §4.1. [106] H. Yang, Q. Yao, B. Han, G. Niu, H. Yang, B. Han, G. Niu, and J. Kwok (2020) Searching to exploit memorization effect in learning from corrupted labels. In International Conference on Machine Learning (ICML), Cited by: §2.4. [107] J. Yang, R. Shi, and B. Ni (2021) MedMNIST classification decathlon: a lightweight automl benchmark for medical image analysis. In The IEEE International Symposium on Biomedical Imaging (ISBI), Cited by: §5.6. [108] J. Yang, R. Shi, D. Wei, Z. Liu, L. Zhao, B. Ke, H. Pfister, and B. Ni (2023) Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data 10 (1), p. 41. Cited by: 3rd item, §5.6, §5.6. [109] Y. Yu, Z. Yang, A. Wei, Y. Ma, and J. Steinhardt (2022) Predicting out-of-distribution error with the projection norm. In International Conference on Machine Learning (ICML), Cited by: §6.2, §6.2, §6.4, Table 6.8, Table 6.9. [110] B. Zadrozny and C. Elkan (2002) Transforming classifier scores into accurate multiclass probability estimates. In International Conference on Knowledge Discovery and Data Mining (KDD), p. 694–699. Cited by: Chapter 1, §3.1, §6.1. [111] F. Zhang, N. Dvornek, J. Yang, J. Chapiro, and J. Duncan (2020) Layer embedding analysis in convolutional neural networks for improved probability calibration and classification. IEEE Transactions on Medical Imaging 39 (11), p. 3331–3342. Cited by: Chapter 1, §3.1. [112] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz (2018) Mixup: beyond empirical risk minimization. In International Conference on Learning Representations (ICLR), Cited by: §2.4. [113] J. Zhang, B. Kailkhura, and T. Y. Han (2020) Mix-n-match: ensemble and compositional methods for uncertainty calibration in deep learning. In International Conference on Machine Learning (ICML), p. 11117–11128. Cited by: §3.2, §6.2. [114] L. Zhang, R. Tanno, M. Xu, C. Jin, J. Jacob, O. Cicarrelli, F. Barkhof, and D. Alexander (2020) Disentangling human error from ground truth in segmentation of medical images. Advances in Neural Information Processing Systems (NeurIPs) 33, p. 15750–15762. Cited by: §2.4. [115] L. Zhang, Z. Deng, K. Kawaguchi, and J. Zou (2022) When and how mixup improves calibration. In International Conference on Machine Learning (ICML), p. 26135–26160. Cited by: Chapter 1, §3.1, §6.1. [116] Y. Zhang, S. Zheng, P. Wu, M. Goswami, and C. Chen (2021) Learning with feature-dependent label noise: a progressive approach. In International Conference on Learning Representations (ICLR), Cited by: §2.4. [117] Y. Zhang, G. Niu, and M. Sugiyama (2021) Learning noise transition matrix from only noisy labels via total variation regularization. In International Conference on Machine Learning (ICML), p. 12501–12512. Cited by: §3.2, §3.2, §4.3.3, §4.3. [118] Z. Zhang and M. Sabuncu (2018) Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in Neural Information Processing Systems (NeurIPs), p. 8778–8788. Cited by: §2.4. [119] S. Zheng, P. Wu, A. Goswami, M. Goswami, D. Metaxas, and C. Chen (2020) Error-bounded correction of noisy labels. In International Conference on Machine Learning (ICML), p. 11447–11457. Cited by: §2.4. [120] Y. Zou, Z. Yu, X. Liu, B. Kumar, and J. Wang (2019) Confidence regularized self-training. In Proc. of the IEEE International Conference on Computer Vision (ICCV), Cited by: §2.3, §6.1. [121] Y. Zou, W. Deng, and L. Zheng (2023) Adaptive calibrator ensemble: navigating test set difficulty in out-of-distribution scenarios. In Proc. of the IEEE International Conference on Computer Vision (ICCV), Cited by: §6.2. See pages 1 of src/pdf/thesis_hebrew_intro_onlySee pages 5 of src/pdf/thesis_hebrew_15_10_25See pages 6 of src/pdf/thesis_hebrew_15_10_25See pages 3 of src/pdf/thesis_hebrew_15_10_25See pages 2 of src/pdf/thesis_hebrew_15_10_25See pages 1 of src/pdf/thesis_hebrew_15_10_25