Paper deep dive
Epistemic Uncertainty-Aware Defect Detection for Quality Control in Medical Device Manufacturing
Raham A. Butt, Marco Romanelli, Roche C. de Guzman
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/8/2026, 6:49:59 AM
Summary
This paper investigates the use of epistemic uncertainty to improve the reliability of automated defect detection in medical device manufacturing. The authors propose a machine learning framework using Knowledge Graphs and a principled rejection strategy that allows the model to abstain from predictions when epistemic uncertainty exceeds a threshold. Evaluated on 266,170 FDA MAUDE device reports and synthetic benchmarks, the approach significantly reduces classification error and risk by trading coverage for higher accuracy on retained samples.
Entities (7)
Relation Signals (5)
Raham A. Butt → affiliatedwith → Hofstra University
confidence 95% · R. A. Butt is with the Department of Engineering, Hofstra University
Rejection Strategy → reduces → Classification Error
confidence 95% · a 10% abstention rate reduces classification error by 48%
FDA MAUDE → usedin → Empirical Evaluation
confidence 95% · We evaluate the approach using standard synthetic benchmarks and real-world medical device report data... On 266,170 real-world FDA MAUDE device reports
Epistemic Uncertainty → mitigates → Defect Detection Errors
confidence 92% · To mitigate errors arising from uncertainty in the decision model, we analyze a principled rejection strategy
Knowledge Graphs → usedfor → Data Representation
confidence 90% · operates on heterogeneous manufacturing and device-report data represented with Knowledge Graphs
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Objective: We investigate whether accounting for epistemic uncertainty can improve the reliability of automated defect detection in medical device manufacturing. Methods: We consider a machine learning framework that operates on heterogeneous manufacturing and device-report data represented with Knowledge Graphs. To mitigate errors arising from uncertainty in the decision model, we analyze a principled rejection strategy to abstain from predictions whose estimated epistemic uncertainty exceeds a specified threshold. We evaluate the approach using standard synthetic benchmarks and real-world medical device report data. Results: The theoretical results establish the validity of the method characterizing the regimes under which it is expected to be effective. Empirically, the rejection strategy enables explicit control of coverage, that is, the proportion of samples for which the model issues predictions, while improving performance on the retained samples. On 266,170 real-world FDA MAUDE device reports, a 10% abstention rate reduces classification error by 48%, and more aggressive rejection (approximately 70% coverage) yields near-perfect accuracy on the retained samples. On standard synthetic manufacturing benchmarks, abstaining on 9% of the decisions, our approach reduces the risk up to 63% compared with the standard no-abstention approach. Conclusions: Abstaining from predictions with high epistemic uncertainty can provide a practical tool for controlling the reliability of machine learning-based defect detection, especially in high-stakes medical device manufacturing applications. Significance: Uncertainty-aware defect detection may support safer and more reliable quality assurance in medical device manufacturing by identifying cases that require additional inspection rather than issuing potentially harmful predictions.
Tags
Links
- Source: https://arxiv.org/abs/2610.09057v1
- Canonical: https://arxiv.org/abs/2610.09057v1
Trouble viewing inline? Open PDF directly →
Full Text
73,724 characters extracted from source content.
Expand or collapse full text
\__cmd_normalize_type_g:w Epistemic Uncertainty-Aware Defect Detection for Quality Control in Medical Device Manufacturing Raham A. Butt Marco Romanelli Roche C. de Guzman †thanks: This work was supported in part by NSF Award No. 2535091 through UNC-Chapel Hill subaward No. 5136002 to Hofstra University. R.C. de Guzman was supported in part by NIH/NIBIB Award No. R25EB035522. The authors thank the FRINK/Proto-OKN teams at RENCI and SDSC for providing infrastructure and technical support for the Medical Device Knowledge Graph and acknowledge the use of Hofstra’s StarHPC cluster, supported by NSF Award No. 2320735. †thanks: R. A. Butt is with the Department of Engineering, Hofstra University, Hempstead, NY 11549 USA (e-mail: raham.a.butt@hofstra.edu).†thanks: M. Romanelli is with the Department of Computer Science, Hofstra University, Hempstead, NY 11549 USA (e-mail: marco.romanelli@hofstra.edu).†thanks: R. C. de Guzman is with the Department of Engineering, Hofstra University, Hempstead, NY 11549 USA (e-mail: RocheRey.C.DeGuzman@hofstra.edu). Abstract Objective: We investigate whether accounting for epistemic uncertainty can improve the reliability of automated defect detection in medical device manufacturing. Methods: We consider a machine learning framework that operates on heterogeneous manufacturing and device-report data represented with Knowledge Graphs. To mitigate errors arising from uncertainty in the decision model, we analyze a principled rejection strategy to abstain from predictions whose estimated epistemic uncertainty exceeds a specified threshold. We evaluate the approach using standard synthetic benchmarks and real-world medical device report data. Results: The theoretical results establish the validity of the method characterizing the regimes under which it is expected to be effective. Empirically, the rejection strategy enables explicit control of coverage, that is, the proportion of samples for which the model issues predictions, while improving performance on the retained samples. On 266,170 real-world FDA MAUDE device reports, a 10% abstention rate reduces classification error by 48%, and more aggressive rejection (approximately 70% coverage) yields near-perfect accuracy on the retained samples. On standard synthetic manufacturing benchmarks, abstaining on 9%9\% of the decisions, our approach reduces the risk up to 63%63\% compared with the standard no-abstention approach. Conclusions: Abstaining from predictions with high epistemic uncertainty can provide a practical tool for controlling the reliability of machine learning-based defect detection, especially in high-stakes medical device manufacturing applications. Significance: Uncertainty-aware defect detection may support safer and more reliable quality assurance in medical device manufacturing by identifying cases that require additional inspection rather than issuing potentially harmful predictions. Index Terms: Anomaly detection, decision making, machine learning, medical devices, quality control, uncertainty. I Introduction Defective medical devices can cause serious patient harm, and their timely detection, both on the manufacturing line and in post-market adverse event reports, is central to device safety [1]. Such defects can arise due to various reasons, including hardware malfunctions, software bugs, or environmental factors. Detecting these anomalies promptly is essential to prevent potential harm to patients and to maintain the integrity of the medical device. The complexity of modern medical devices, which often incorporate advanced semiconductor components, may limit the effectiveness of traditional defect detection methods [2]. As a result, there is a growing interest in leveraging machine learning techniques to enhance the detection process (cf. [3, 4, 5, 6]) In particular, [7] introduces a general approach to defect–mechanism–parameter violation hypotheses for surface-mount assembly root-cause analysis from uncertain observations. Using Knowledge Graphs (KGs) as the source of information, the authors propose a method to detect and explain manufacturing defects. In doing so, they leverage stage-wise three-head cascade multi-layer perceptron (MLP) architecture that models aleatoric uncertainty by estimating defect and mechanism likelihoods and parameter-level risk scores, enabling probabilistic and traceable root-cause ranking. Specifically, Head 1 detects the defects or lack thereof, Head 2 identifies the underlying mechanisms, and Head 3 estimates the associated risks. The authors note that their method can be sensitive to epistemic uncertainty, which may arise from the limited amount of information available during the training phase (cf. [8, 9, 10]). The distinction between aleatoric and epistemic uncertainty is well established in the machine learning (ML) literature [11], and the need to communicate and act on model uncertainty is increasingly recognized as a requirement for machine learning in health-related, high-stakes applications [12, 13]. Crucially, this can lead to detection failures and misclassification of defects. This is a significant concern in high-stakes applications such as medical device manufacturing, where misclassification can incur substantial costs and pose a significant risk of serious patient harm. A wrong detection performed by Head 1, for instance, can lead to a wrong alert being raised, potentially incurring substantial costs (false positives) or posing a significant risk of serious patient harm (false negatives). (a) Decision regions and rejected points. (b) Selective risk curve. (c) Classification margin histogram. Fig. 1: Illustrative example – The impact of principled abstention on the decision regions (Figure 1(a)), risk-coverage curve (Figure 1(b)), and classification margin histogram (Figure 1(c)) for a toy dataset with 2-dimensional features and 3 balanced classes. The model rejects observations concentrated near decision boundaries, where predictions are more uncertain. Lowering the coverage by 19%19\% reduces the classification risk on retained predictions by 50%50\%. The softmax-margin distributions further show that rejected observations generally have smaller margins than retained observations, with only limited overlap, indicating that the reservation mechanism effectively identifies uncertain predictions and trades coverage for lower selective risk. Further details are provided in Section I-A. We argue that addressing this issue is critical and, therefore, investigate the impact of epistemic uncertainty on detection failures in medical device manufacturing and examine a principled approach to mitigating its effects. Specifically, we adopt a cost-sensitive classification framework that explicitly accounts for uncertainty in the decision-making process by allowing the model to abstain from uncertain predictions. This formulation aims to improve the robustness and reliability of defect detection while accounting for the costs associated with erroneous decisions. In our framework, the task performed by MLP Head 1 is treated as a critical decision point, as an incorrect prediction at this stage may compromise the reliability of subsequent decisions and risk assessment. To mitigate this risk, the model is allowed to abstain when its prediction is uncertain, thereby reducing the likelihood of misclassification and improving the overall reliability of the defect detection system. Although outside the scope of this work, abstained predictions could subsequently be subjected to further scrutiny by human experts or additional automated checks. Such a mechanism provides a more cautious approach to defect detection in high-stakes settings and could further enhance the reliability and safety of medical device manufacturing. I-A Contributions We summarize the contributions contained in the rest of this paper as follows: 1. We introduce a reproducible real-world benchmark from a medical device case study, publicly accessible as a KG through the Prototype Open Knowledge Network (Proto-OKN) (cf. [14, 15]), demonstrating the practical applicability of the proposed method in a high-stakes setting. All code required to reproduce the experiments, including the procedure used to generate the benchmarks, is available at https://github.com/raham24/Epistemic-Uncertainty-Aware-Defect-Detection-for-Quality-Control-in-Medical-Device-Manufacturing/tree/code-only. 2. We investigate detection failures arising from epistemic uncertainty in recent root-cause analysis approaches for semiconductor manufacturing, and extend that analysis to high-stakes domains such as defect detection in medical device manufacturing, where misclassification can lead to substantial costs and serious risks to patient safety. 3. We analyze a simple yet effective approach to mitigate the impact of epistemic uncertainty on detection failures, leveraging a cost-sensitive classification framework that incorporates uncertainty handling into the decision-making training process, allowing the model to abstain from uncertain predictions. Our theoretical results provide a principled foundation with a mathematical-programming approach, precisely characterizing the regimes under which it is expected to be effective and complementing the intuition developed in prior work. 4. We evaluate the proposed approach through extensive experiments on a synthetic benchmark spanning varying levels of class-posterior overlap and five random seeds. The results demonstrate consistent effectiveness as increasing overlap makes the classification task progressively more challenging, as well as robustness to random initialization. I Related Work The increasing integration of digital technologies into medical devices has introduced new challenges for quality assurance. To address these challenges, artificial intelligence (AI)-based defect detection has gained attention as an efficient alternative to conventional rule-based and manual inspection methods (cf. [4, 3]). In the context of software reliability and cybersecurity, [4] proposed an AI-based software defect detection method as part of a broader framework for strengthening medical device security. Their work highlights the potential of AI to provide a more efficient alternative to conventional rule-based approaches for identifying software flaws that may compromise the reliability and safety of medical devices. From a manufacturing perspective, [3] developed an automated expert system for detecting porosity defects in prosthetic components produced using electron beam melting. By analyzing layer-by-layer optical images acquired during manufacturing, their approach enables rapid and nondestructive identification of defects and demonstrates strong agreement with computed tomography-based inspection. More recently, [2] investigated unsupervised defect detection for surgical instruments and identified several challenges associated with applying conventional industrial defect detection methods to the medical device domain, including textured backgrounds, small defects, and domain shifts. Their proposed approach combines background masking, patch-based analysis, and domain adaptation to improve the detection of fine-grained defects. Collectively, these studies demonstrate the growing role of AI-based methods in improving the efficiency of defect detection across both the software and physical components of medical devices, while emphasizing the importance of reliable detection mechanisms in safety-critical healthcare applications. As many modern medical devices rely on semiconductor components, advances in industrial semiconductor defect detection are also relevant to the medical device domain. Neural methods have been widely applied to defect detection, fault classification, and anomaly prediction in semiconductor manufacturing, particularly for analyzing process and inspection data. Prior work [16, 17, 5, 6] has focused on improving detection accuracy and enabling real-time process control, including for solder paste printing, a major source of downstream defects. More recently, [7] introduced a framework for generating structured causal explanations for root cause analysis, incorporating uncertainty to quantify the risk associated with inferred defect causes. Crucially, these works do not explicitly account for uncertainty arising from the model’s decisions themselves, i.e., epistemic uncertainty, which is a central focus of our work. The use of abstention (or rejection option) and selective classification to manage uncertainty in decision-making has been studied for decades (cf. [18, 19, 20, 21, 22, 23]), providing a mechanism for avoiding low-confidence decisions and reducing incorrect classifications [24]. Several approaches have been proposed to incorporate abstention into predictive models. Some methods require dedicated model architectures [25, 26], while others rely on perturbed training data to improve empirical generalization [27, 28] or on specialized training procedures and ad hoc loss functions [29]. Alternatively, post-training approaches such as [30] leverage information-theoretic measures to quantify uncertainty without modifying the underlying model, providing a computationally efficient and principled solution, although their effectiveness may be limited for overconfident models (cf. [31]). Our work builds on the framework of [32], which draws on the decision-theoretic principles of [33]. We extend this framework by providing additional theoretical insights, including an interpretable characterization of the abstention threshold (Theorem I.1) and a complete mathematical programming characterization through the associated optimality conditions (Theorem I.2). Finally, the recent development of resources such as Proto-OKN [14, 15], which integrate heterogeneous data from sources including the FDA 510(k) and MAUDE databases [34, 1], provides new opportunities to investigate uncertainty-aware defect detection using real-world medical device data. KG are an established substrate for integrating heterogeneous biomedical data for downstream analysis [35]. On the device-safety side, the MAUDE database underpins a substantial surveillance literature, including reviews of signal-detection methodology [36], change-point analysis for post-market signals [37], natural-language processing of adverse-event narratives [38], large-language-model classification of device-related adverse events [39], and machine-learning prediction of FDA recalls from public data [40]. To our knowledge, none of this line of work addresses epistemic uncertainty or selective prediction, which is the gap our real-data benchmark targets. I Framework We focus on the classification task associated with head H1H_1, where errors can result in either false or missed defect alarms, increasing production costs and risk. The same analysis extends naturally to other classification heads. We therefore formulate the task for H1H_1 as a multi-class classification problem. Let ⊆ℝdX ^d be the (possibly continuous) feature space and let =1,…,CY=\1,…,C\ be the label space for the classification task, thus ||=C |Y |=C. Moreover, let pXYp_XY be the underlying (unknown) probability density function (pdf) over ×X×Y. Let n=(1,y1),…,(n,yn)∼pXYD_n=\! \(x_1,y_1),…,(x_n,y_n) \ p_XY be a random realization of n i.i.d. samples according to pXYp_XY denoting the training set, where i∈x_i is the input (feature), yi∈y_i is the output class among C possible classes and n denotes the size of the training set. A predictor is a function fn:→f_D_n:X , defined as fn()≜argmaxy∈PY^|X(y|;n),f_D_n(x) argmax_y ~P_ Y|X(y|x;D_n),, which uses the inferred model PY^|X≡PY^|X(y|;n)P_ Y|X≡ P_ Y|X(y|x;D_n) to approximate the optimal (Bayes) decision rule f⋆()≜argmaxy∈PY|X(y|).f (x) argmax_y ~P_Y|X(y|x). Notice that PY^|XP_ Y|X can be interpreted as the prediction of the class (label) posterior probability given a sample (e.g., PY^|X(y|)≡softmax()yP_ Y|X(y~|x) ( x)_y), while PY|XP_Y|X is the true (unknown) probability. In general, during training, the empirical cross-entropy loss is minimized to learn the model PY^|XP_ Y|X from the training set nD_n by optimizing the parameters of a model. The cross-entropy loss ℒCE(n)L_CE(D_n) is defined as ℒCE(n)=−1n∑i=1n[ _CE(D_n)=- 1n _i=1^n [ ∑j=1CPY|X(yj∣i)logPY^|X(yj∣i;n)]. _j=1^CP_Y|X(y_j _i) P_ Y|X(y_j _i;D_n) ]. (1) Minimizing 1 encourages the model to assign high probabilities to the correct classes (cf. [41, 42, 43]). For each sample x, Equation 2 indicates the likelihood of the predicted class probabilities, i.e., W1(p,p^)=∑j=1Cpjlog(p^j), W_#1 (p, p )= _j=1^Cp_j ( p_j ), (2) where p^=PY^|X(⋅|;n) p=P_ Y|X(·|x;D_n) represents the predicted class probabilities for the input x and p is the true posterior probability11 1 In the empirical setting, the true posterior is usually represented as one-hot encoded vectors, and 1 becomes ℒCE(n)=−1n∑i=1nlogPY^|X(yi∣i;n)L_CE(D_n)=- 1n _i=1^n P_ Y|X(y_i _i;D_n). In this case, the term in Equation 2 becomes W1(p,p^)=log(p^j)W_#1 (p, p )= ( p_j ), where j is the index of the true class label for the input x.. Inspired by [44, 33, 32], we explicitly account for uncertainty by modifying the training objective in 1, to enable the model to learn when to abstain. Specifically, we introduce a reservation variable represented as an additional class label, allowing the model to defer predictions when uncertainty is high. In particular, the term in Equation 2 is replaced by W1(p,p^)r=∑j=1Cpjlog(k⋅p^j+p^C+1), W_#1 (p, p )r= _j=1^Cp_j (k· p_j+ p_C+1 ), (3) where p^C+1 p_C+1 is the reservation variable for a given input x, subject to the constraints ∑j=1Cp^j+p^C+1=1, _j=1^C p_j+ p_C+1=1, (4) and p^j≥0,∀j∈1,…,C+1, p_j≥ 0,\ ∀ j∈\1,…,C+1\, (5) where k∈ℝ+k _+ is a positive hyperparameter that controls the trade-off between making a prediction and abstaining. Higher values of k encourages the model to make predictions, while lower values promote abstention. Though neural networks provide a practical means of obtaining (generally suboptimal) solutions that can be applied to arbitrary inputs, as noted in [45] and [32], a principled reformulation and analysis of the underlying optimization problem (cf. [46]) can yield further theoretical insights. In particular, such an analysis allows us to characterize the uncertainty threshold for abstention and identify previously unrecognized degenerate cases. These findings provide further insight into the principled management of epistemic uncertainty in high-stakes applications, such as fault detection in medical devices. I-A Illustrative example We first illustrate the effect of the reservation variable on a toy classification problem with two-dimensional features and (C=3) classes in Figure 1. Each class has the same amount of samples. A simple fully connected neural network with two hidden layers of 32 units and ReLU activations is used. The output layer contains four units corresponding to the three target classes and one reservation output. The model is trained for 150 epochs using full-batch Adam with a learning rate of 5×10−35× 10^-3 and reservation reward r=2r=2. At inference time, an observation is accepted when its largest class probability exceeds its reservation probability and is otherwise rejected. All input features are standardized using statistics computed from the training set. As shown in Figure 1(a), introducing the reservation variable allows the model to identify and reject observations concentrated around the decision boundaries. These regions correspond to inputs for which the classifier has difficulty distinguishing between competing classes and are therefore associated with greater predictive uncertainty. This behavior translates directly into improved performance on the retained predictions. The selective risk curve in Figure 1(b) shows that reducing the coverage to approximately 81%81\% results in a reduction in classification risk of approximately 50%50\%. Thus, by abstaining on a relatively small fraction of uncertain observations, the model can substantially reduce the error rate among the predictions it retains. Finally, Figure 1(c) provides further insight into the mechanism underlying this behavior. Rejected observations are predominantly associated with smaller classification margins, indicating weaker separation between the most likely competing classes. Together, these results illustrate how the reservation mechanism enables the model to identify uncertain predictions and trade coverage for lower classification risk. I-B Optimization problem with the reservation variable We now present the main theoretical results of this work, providing a principled characterization of the role of the reservation variable in managing epistemic uncertainty. For the sake of generality, in this section, we consider true posterior probabilities p=PY|X(⋅∣)p=P_Y|X(· ), rather than empirical one-hot encoded labels. Results for one-hot encoded labels can be obtained as a special case, often reducing general results to trivial ones. Proofs based on the Karush-Kuhn-Tucker (KKT) optimality conditions, together with the necessary auxiliary results, are provided as supplementary material in the appendix. Theorem I.1 Let ^⋆ p be a KKT optimal point for the maximization of Equation 3 with constraints Equations 4 and 5, and suppose that p^C+1⋆>0 p _C+1>0. Then there exists a scalar λ>0λ>0 such that, for every class i∈1,…,Ci∈\1,…,C\, p^i⋆=maxpiλ−p^C+1⋆k, 0. p_i = \ p_iλ- p_C+1 k,\,0 \. (6) Moreover, the quantity p^C+1⋆k p_C+1 k defines an adaptive threshold that determines which class probabilities remain active at the optimum. An increase in the optimal reserve probability p^C+1⋆ p_C+1 , or a decrease in the cost multiplier k, raises this threshold, thereby causing a larger number of class probabilities to become inactive. This characterization provides an optimization-based interpretation of the reserve output as an adaptive measure of uncertainty. The corresponding derivation is provided in Section A-B of the appendix. Theorem I.2 Consider the maximization of Equation 3 subject to Equations 4 and 5. The cost-multiplier mixed-regime is 1<k<C. 1<k<C. In particular, if k<1k<1, full reservation is optimal, whereas if k>Ck>C, every optimal solution has p^C+1⋆=0 p_C+1 =0 and the prediction coordinates satisfy p^j⋆=pj p_j =p_j for j=1,…,Cj=1,…,C. The boundary cases k=1k=1 and k=Ck=C are degenerate: the optimizer need not be unique, and distinct prediction–reservation allocations may attain the same optimal value. The proof is provided in Section A-C of the appendix. Theorem I.2 provides a principled characterization of the cost multiplier k and its relationship to the number of classes C, providing guidance for selecting k in practice, as shown in the ablation studies in Section IV. IV Experiments (a) Baseline. (b) Increased class posterior overlap. Fig. 2: Synthetic benchmark data – Feature distributions under different levels of class posterior overlap. Figure 2(a) shows the setting for the benchmark introduced in [7], while Figure 2(b) shows a more challenging setting with increased overlap between the class-conditional feature distributions, controlled by the target Bayes error parameter in Algorithm 1. In this section, we present our numerical results and summarize the main findings through a series of plots. We begin by examining how epistemic uncertainty affects the baseline benchmark introduced in [7] and evaluate the extent to which our proposed approach mitigates its effects. Despite the relative simplicity of this task, it illustrates how models with otherwise strong predictive performance can remain susceptible to epistemic uncertainty, and how abstention can reduce the resulting decision risk. Furthermore, we introduce a more challenging variant of the original benchmark, in which the final detection task is more difficult by tweaking the data generative process, allowing us to evaluate the effectiveness of our mitigation approach under more realistic conditions. Finally, we present a real-world medical device case study and introduce a benchmark that is publicly accessible through the Proto-OKN, demonstrating the practical applicability of our approach in a high-stakes setting. Following standard experimental practice, we report results over five independent runs with different random seeds to account for variability due to random initialization. The results are reported in terms of accuracy or its converse, i.e., empirical risk. As we must account for the fact that the model may abstain on some samples, we also report the coverage, i.e., the fraction of samples on which the model makes a prediction. Importantly, note that in case of abstention, the accuracy and risk are only computed on the samples for which the model makes a prediction, i.e., the coverage. Thus, while our framework may abstain on some samples which may be correctly classified, increased accuracy and reduced risk on the retained samples is a clear indication that the abstained set must contain a higher proportion of misclassified samples than the original dataset, i.e. the abstention mechanism is preferentially rejecting predictions that are more likely to be incorrect. For the synthetic experiments, we consider Head 1 of the multi-head cascade MLP architecture introduced in [7], which performs defect classification. The model consists of two hidden layers with 256 neurons each, ReLU activations, and a dropout rate of 0.10.1, and it is trained using Adam with a learning rate of 10−310^-3 and a batch size of 128 for up to 40 epochs, with early stopping using a patience of 6 epochs. For the real-world experiments, the model consists of two hidden layers with 256 and 128 neurons, respectively, and ReLU activations, and it is trained using Adam [47] with a learning rate of 10−310^-3, weight decay of 10−410^-4, and a batch size of 32 for up to 80 epochs, with early stopping using a patience of 10 epochs. All models are implemented in PyTorch [48]. Development and preliminary experiments were conducted on Apple Silicon, while the main experimental sweeps were executed on a SLURM-based computing cluster using NVIDIA A100 and H100 GPUs, with one GPU allocated per run. The FDA MAUDE and 510(k) source data are publicly available from the U.S. Food and Drug Administration [1, 34]. The integrated Medical Device Knowledge Graph currently comprises 17.7 million triples across 29 classes and 887,200 subjects; it is registered in the Open Knowledge Network registry (https://registry.okn.us/registry/kgs/medical-device-kg/) and publicly queryable through the FRINK SPARQL endpoint (https://apps.okn.us/medical-device-kg/sparql). IV-A Synthetic Benchmark We first consider the benchmark introduced in [7], which we refer to as the baseline. To the best of our knowledge, the original benchmark data, which is also synthetic, has not been publicly released. We therefore reproduce the data-generation process following the description provided in the original work (cf. Sections A-E and 1). To assess the fidelity of our reconstruction, we verify that a comparable MLP achieves performance consistent with that reported in the original work. Given the synthetic nature and relative simplicity of this benchmark, we primarily use it as a proof of concept to illustrate the effects of epistemic uncertainty and the benefits of abstention under controlled conditions. We subsequently introduce a more challenging synthetic scenario to evaluate whether these benefits persist as the underlying detection task becomes more complex. Fig. 3: Synthetic benchmark data – (Best) Accuracy bar plot over multiple seeds for the baseline benchmark, comparing the performance of a single model w/o and w/ abstention (target coverage ≥90%≥ 90\%). For the baseline setting, abstention increases mean accuracy from 95.10%95.10\% to 98.17%98.17\% (+3.07+3.07 percentage points), reducing classification error by 62.65%62.65\%. At higher overlap, it increases mean accuracy from 87.59%87.59\% to 89.94%89.94\% (+2.35+2.35 percentage points), reducing classification error by 18.94%18.94\%. The narrow error bars, which represent variability across multiple random seeds, indicate that performance remains consistent under different random initializations. The problem at hand considers a three-class classification task (no defect, open circuit, and solder bridging) based on six process features: ambient relative humidity, ambient temperature, paste viscosity, stencil thickness, time above liquidus, and peak reflow temperature. The per-feature class distributions are shown in Figure 2(a). As can be observed, several features provide a relatively clear separation between classes, making the resulting classification task comparatively simple. Nevertheless, Figure 3 shows that when abstaining on less than 10%10\% of the samples, the accuracy increases from 95.10%95.10\% to 98.17%98.17\%, an improvement of 3.073.07 percentage points, corresponding to a 62.65%62.65\% relative reduction in classification error. This indicates that the abstention mechanism preferentially identifies predictions that are more likely to be misclassified, thereby improving the accuracy of the retained decisions. Next, we artificially increase the difficulty of the classification task by increasing the overlap between the class-conditional feature distributions, thereby increasing the Bayes error, as illustrated in Figure 2(b). As expected, the overall classification accuracy decreases under this more challenging setting. Nevertheless, Figure 3 shows that abstention continues to improve the accuracy of the retained predictions, demonstrating the effectiveness of the approach even as the underlying classification problem becomes more difficult. In this case, abstaining yields an accuracy increase from 87.59%87.59\% to 89.94%89.94\%, an improvement of 2.352.35 percentage points, corresponding to an 18.94%18.94\% relative reduction in classification error. Finally, we present an ablation study across different target coverage levels in Figure 4(a) for the baseline setting and Figure 4(b) for the higher-overlap setting. An additional ablation study across different payoff values, which further supports the results reported in Figure 4, is provided in the appendix (cf. Sections A-D and 7). (a) Baseline (b) Increased class posterior overlap Fig. 4: Synthetic benchmark data – Risk as a function of coverage. Results are shown for the baseline setting (Figure 4(a)) and the more challenging setting with increased class posterior overlap (Figure 4(b)). In both cases, the risk decreases as the coverage decreases, indicating that the abstention mechanism preferentially rejects predictions that are more likely to be misclassified. Results over several random seeds are shown, with the shaded area indicating the standard deviation, showing that the results are consistent across different independent runs. IV-B Experiment on real medical devices data In this section, we present the results of our experiments on real-world medical device data. The dataset consists of report narratives, extracted from [1, 34] through the contributed KG within the Proto-OKN project (cf. [14]). (a) Accuracy vs. payoff k. (b) Accuracy vs. coverage. Fig. 5: Medical device data – Performance across different values of k and coverage. The final dataset comprises 266,170 reports (185,477 malfunction, 44,995 injury, 33,897 serious injury, 1801 death), partitioned into training (64%), validation (16%), and test (20%) splits and stratified by outcome class. Severity labels are derived from the MAUDE event type field as follows: reports typed Death and Malfunction map directly to the Death and Malfunction classes, while Injury reports are split into basic vs. serious injury. An injury is classified as serious if its narrative matches a serious-injury cue and basic injury otherwise22 2 Keywords that identify serious injury are the ones that are described as life threatening situation, permanent impairment and surgical interventions., reports typed Other or No answer, or carrying no narrative are discarded. To prevent label leakage, sentences in the narratives whose wording contains or closely matches the outcome labels are removed. Duplicate reports were removed based on exact narrative text duplications. These narratives consist of structured key-value pairs, including textual descriptions of adverse events associated with medical devices. The datasets are connected by the device identifiers, which are used to link the narratives to the corresponding device information. The task we select is to classify these text-based features into four outcome classes: malfunction, basic injury, serious injury, and death, in increasing order of severity. The underlying objective is to identify patterns associated with adverse events in previously unseen data, with the broader goal of supporting their early detection and potentially preventing similar events from occurring in the future. Text content is represented using TF-IDF features with up to 50,000 unigram and bigram features, selected for scalability, interpretability, and independence from pretrained language models. Further details are provided in the publicly available code repository. Fig. 6: Medical device data – Performance comparison between no abstention and abstention. Results are reported over several random seeds, showing narrow error bars indicating consistent results across multiple independent runs. The abstention mechanism allows for an increase in average accuracy, from 94.68%94.68\% to 97.23%97.23\%, an improvement of 2.552.55 percentage points, corresponding to a 47.93%47.93\% relative reduction in classification error with a target coverage of 90%90\%, further demonstrating its effectiveness. Figure 5(a) and Figure 5(b) show the effectiveness of abstention on the retained predictions, confirming the expected results. Additionally, for this task, it is possible to achieve almost perfect accuracy for ≈70%≈ 70\% coverage. In operational terms, abstention defers roughly one in ten test reports to human review in exchange for a substantially lower error rate among retained predictions, consistent with triage workflows in post-market surveillance. This indicates that the abstention mechanism is preferentially rejecting predictions that are more likely to be incorrect, thereby improving the accuracy of the retained predictions, though nearly optimal accuracy may incur a high cost in terms of the number of predictions that are abstained from. The ablation study across different target coverage levels in Figure 5(a) further supports this conclusion, demonstrating that the benefits of abstention persist across a range of coverage levels. Finally, Figure 6 shows the performance comparison between the standard cross-entropy model and its version equipped with abstention, showing consistent and favorable results across multiple independent runs. V Limitations and Future Work Given the importance of uncertainty-aware decision-making in medical device manufacturing, this work represents an initial step toward systematically studying epistemic uncertainty in this domain. A key limitation is the current availability of real-world benchmarks that capture both manufacturing defects and the uncertainty associated with their detection. Consequently, broader evaluation across devices, defect types, and manufacturing settings remains necessary to establish the generalizability of these findings. Two characteristics of the underlying data source should be kept in mind when interpreting these results. MAUDE is a passive, spontaneous-report surveillance system: report volumes reflect reporting behavior, device utilization, and regulatory reporting requirements rather than confirmed failure incidence, and narratives vary widely in completeness and may contain duplicates [36]. Accordingly, the benchmark task should be understood as report-level severity classification under realistic label noise, a regime in which epistemic uncertainty is expected to be substantial and abstention correspondingly valuable, and not as an estimate of device failure rates. These characteristics make MAUDE-derived data a stringent, practically relevant testbed for uncertainty-aware classification [38, 39]. Additionally, duplicates were removed by exact narrative matching, yet MAUDE contains near-duplicate reports of the same event (e.g., manufacturer and user-facility submissions), which random splitting may place on both sides of the train–test split. Near-duplicate detection and temporal splits are natural extensions to better approximate deployment conditions. We hope this work will encourage further research on epistemic uncertainty in medical device manufacturing, particularly through the development of larger, more diverse, and application-relevant benchmarks. Resources such as the Proto-OKN, the data repositories in [34, 1], together with other tools for integrating and organizing heterogeneous medical device data, provide an opportunity to facilitate the construction and sharing of such datasets. Such efforts could support more comprehensive evaluations of uncertainty-aware methods and provide a stronger empirical foundation for their use in medical device quality assurance. VI Conclusion In this work, we investigated the impact of epistemic uncertainty on defect detection in medical device manufacturing, a first step particularly important when incorrect predictions can propagate through subsequent root-cause analysis and risk assessment. We showed that explicitly accounting for this uncertainty through abstention provides a simple yet principled mechanism for reducing the risk associated with unreliable model decisions. Building on a cost-sensitive classification framework, we analyzed a rejection strategy that allows the model to abstain from uncertain predictions rather than forcing a decision for every input. Our experiments on synthetic benchmarks show that abstaining on a relatively small fraction of predictions can reduce decision risk while improving the reliability of the remaining decisions. We further evaluated the approach using real-world medical device report data, demonstrating its applicability beyond controlled synthetic settings. To support reproducibility and further research, we have shared a link to our open-source code. More broadly, our results suggest that predictive performance alone may not adequately characterize model reliability in high-stakes defect detection. Providing models with the ability to identify and defer uncertain decisions offers a practical mechanism for incorporating uncertainty into automated quality assurance and provides a foundation for integrating such decisions with downstream root-cause analysis, risk assessment, and human review. References References [1] U.S. Food and Drug Administration (2026) Manufacturer and User Facility Device Experience (MAUDE) Database. Note: https://w.accessdata.fda.gov/scripts/cdrh/cfdocs/cfmaude/search.cfmAccessed: 2026-08-18 Cited by: §I, §I, §IV-B, §IV, §V. [2] J. Huang, Y. Zhang, J. Yu, W. Chen, S. Hwang, Q. Qiu, A. R. Reibman, E. J. Delp, and F. Zhu (2026) Unsupervised defect detection for surgical instruments. External Links: 2509.21561, Link Cited by: §I, §I. [3] A. F. Bonatti, F. D. Meringolo, I. Tubertini, C. E. Lavecchia, A. Favaro, G. Vozzi, and C. De Maria (2025) Expert system for online defect detection in medical devices produced by electron beam melting using layer-by-layer optical images. 3D Printing and Additive Manufacturing 12 (1), p. 36–47. External Links: Document Cited by: §I, §I, §I. [4] H. Kim and K. Kim (2024) Deep learning-based sbom defect detection for medical devices. In 2024 International Conference on Artificial Intelligence in Information and Communication (ICAIIC), Vol. , p. 047–051. External Links: Document Cited by: §I, §I, §I. [5] H. Lu, H. Wang, S. W. Yoon, and D. Won (2019) Real-time stencil printing optimization using a hybrid multi-layer online sequential extreme learning and evolutionary search approach. IEEE Transactions on Components, Packaging and Manufacturing Technology 9 (12), p. 2490–2501. External Links: Document Cited by: §I, §I. [6] N. Cao, A. Farrag, D. Won, and S. W. Yoon (2024) XSCAN: explainable solder joint defect probability prediction through solder paste printing status with imbalanced data. Journal of Manufacturing Systems 77, p. 212–227. External Links: Document Cited by: §I, §I. [7] M. Shenoy and F. Ameri (2026) Uncertainty-aware neurosymbolic root-cause analysis for surface-mount assembly. IEEE Transactions on Semiconductor Manufacturing (), p. 1–1. External Links: Document Cited by: §I, §I, Fig. 2, Fig. 2, §IV-A, §IV, §IV. [8] J. Mukhoti, A. Kirsch, J. van Amersfoort, P. H. S. Torr, and Y. Gal (2023) Deep deterministic uncertainty: A new simple baseline. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, p. 24384–24394. External Links: Link, Document Cited by: §I. [9] B. Lakshminarayanan, A. Pritzel, and C. Blundell (2017) Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, Vol. 30. Cited by: §I. [10] A. Kendall and Y. Gal (2017) What uncertainties do we need in Bayesian deep learning for computer vision?. In Advances in Neural Information Processing Systems, Vol. 30. Cited by: §I. [11] E. Hüllermeier and W. Waegeman (2021) Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Machine Learning 110, p. 457–506. External Links: Document Cited by: §I. [12] B. Kompa, J. Snoek, and A. L. Beam (2021) Second opinion needed: communicating uncertainty in medical machine learning. npj Digital Medicine 4 (4). External Links: Document Cited by: §I. [13] E. Begoli, T. Bhattacharya, and D. Kusnezov (2019) The need for uncertainty quantification in machine-assisted medical decision making. Nature Machine Intelligence 1, p. 20–23. External Links: Document Cited by: §I. [14] Proto-OKN CollaborationProto-OKN: Prototype Open Knowledge Network(Website) External Links: Link Cited by: item 1, §I, §IV-B. [15] Open Knowledge NetworkOpen Knowledge Network (OKN) Graph Registry(Website) External Links: Link Cited by: item 1, §I. [16] K. B. Lee, S. Cheon, and C. O. Kim (2017) A convolutional neural network for fault classification and diagnosis in semiconductor manufacturing processes. IEEE Transactions on Semiconductor Manufacturing 30 (2), p. 135–142. External Links: Document Cited by: §I. [17] N. Khader and S. W. Yoon (2018) Stencil printing process optimization to control solder paste volume transfer efficiency. IEEE Transactions on Components, Packaging and Manufacturing Technology 8 (9), p. 1686–1694. External Links: Document Cited by: §I. [18] I. Flores (1958) An optimum character recognition system using decision functions. IRE Trans. Electron. Comput. 7 (2), p. 180. External Links: Link, Document Cited by: §I. [19] C. K. Chow (1970) On optimum recognition error and reject tradeoff. IEEE Trans. Inf. Theory 16 (1), p. 41–46. External Links: Link, Document Cited by: §I. [20] V. Vovk, A. Gammerman, and G. Shafer (2005) Algorithmic learning in a random world. Springer, New York, NY. External Links: Document, ISBN 978-0-387-00152-4 Cited by: §I. [21] A. N. Angelopoulos and S. Bates (2023) Conformal prediction: a gentle introduction. Foundations and Trends in Machine Learning 16 (4), p. 494–591. External Links: Document Cited by: §I. [22] S. Bates, A. Angelopoulos, L. Lei, J. Malik, and M. I. Jordan (2021) Distribution-free, risk-controlling prediction sets. Journal of the ACM 68 (6). External Links: Document Cited by: §I. [23] R. El-Yaniv and Y. Wiener (2010) On the foundations of noise-free selective classification. Journal of Machine Learning Research 11 (53), p. 1605–1641. External Links: Link Cited by: §I. [24] P. Pudil, J. Novovicová, S. Bláha, and J. Kittler (1992) Multistage pattern recognition with reject option. In 11th IAPR International Conference on Pattern Recognition, ICPR 1992. Conference B: Pattern Recognition Methodology and Systems, The Hague, Netherlands, August 30-September 3, 1992, p. 92–95. External Links: Link, Document Cited by: §I. [25] C. Corbière, N. Thome, A. Bar-Hen, M. Cord, and P. Pérez (2019) Addressing failure prediction by learning model confidence. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. B. Fox, and R. Garnett (Eds.), p. 2898–2909. External Links: Link Cited by: §I. [26] Y. Geifman and R. El-Yaniv (2019) SelectiveNet: A deep neural network with an integrated reject option. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, K. Chaudhuri and R. Salakhutdinov (Eds.), Proceedings of Machine Learning Research, Vol. 97, p. 2151–2159. External Links: Link Cited by: §I. [27] L. Huang, C. Zhang, and H. Zhang (2020) Self-adaptive training: beyond empirical risk minimization. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Eds.), External Links: Link Cited by: §I. [28] A. Fisch, T. Jaakkola, and R. Barzilay (2022) Calibrated selective classification. CoRR abs/2208.12084. External Links: Link, Document, 2208.12084 Cited by: §I. [29] L. Feng, M. O. Ahmed, H. Hajimirsadeghi, and A. H. Abdi (2022) Stop overcomplicating selective classification: use max-logit. CoRR abs/2206.09034. External Links: Link, Document, 2206.09034 Cited by: §I. [30] F. Granese, M. Romanelli, D. Gorla, C. Palamidessi, and P. Piantanida (2021) DOCTOR: A simple method for detecting misclassification errors. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan (Eds.), p. 5669–5681. External Links: Link Cited by: §I. [31] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger (2017) On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, D. Precup and Y. W. Teh (Eds.), Proceedings of Machine Learning Research, Vol. 70, p. 1321–1330. External Links: Link Cited by: §I. [32] Z. Liu, Z. Wang, P. P. Liang, R. R. Salakhutdinov, L. Morency, and M. Ueda (2019) Deep gamblers: learning to abstain with portfolio theory. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32, p. . Cited by: Proposition A.1, §I, §I, §I. [33] H. Markowitz (1952) Portfolio selection. The Journal of Finance 7 (1), p. 77–91. External Links: ISSN 00221082, 15406261 Cited by: §I, §I. [34] U.S. Food and Drug Administration (2026) 510(k) Premarket Notification Database. Note: https://w.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfmAccessed: 2026-08-18 Cited by: §I, §IV-B, §IV, §V. [35] D. N. Nicholson and C. S. Greene (2020) Constructing knowledge graphs and their biomedical applications. Computational and Structural Biotechnology Journal 18, p. 1414–1428. External Links: Document Cited by: §I. [36] G. Chung, K. Etter, and A. Yoo (2020) Medical device active surveillance of spontaneous reports: a literature review of signal detection methods. Pharmacoepidemiology and Drug Safety 29 (4), p. 369–379. External Links: Document Cited by: §I, §V. [37] Z. Xu, T. Kass-Hout, C. Anderson-Smits, and G. Gray (2015) Signal detection using change point analysis in postmarket surveillance. Pharmacoepidemiology and Drug Safety 24 (6), p. 663–668. External Links: Document Cited by: §I. [38] J. Mao, A. Sedrakyan, T. Sun, M. Guiahi, S. Chudnoff, M. Kinard, and S. B. Johnson (2021) Assessing adverse event reports of hysteroscopic sterilization device removal using natural language processing. Pharmacoepidemiology and Drug Safety 31 (4), p. 442–451. External Links: Document Cited by: §I, §V. [39] N. Sohoni, N. S. Sohoni, R. A. Sutherland, V. M. Sundaresan, S. Smani, P. Ananth, J. A. Onofrey, S. Aneja, M. Miszczyk, H. Lee, J. E. Olivieri, and M. S. Leapman (2026) Automated classification of adverse events after hydrogel perirectal spacer insertion for prostate cancer using large language models. Urology 209, p. 18–24. External Links: Document Cited by: §I, §V. [40] V. Barbosa Slivinskis, I. Agi Maluli, and J. Seth Broder (2025) A machine learning algorithm to predict medical device recall by the Food and Drug Administration. Western Journal of Emergency Medicine 26 (1), p. 161–170. External Links: Document Cited by: §I. [41] S. Shalev-Shwartz and S. Ben-David (2014) Understanding machine learning - from theory to algorithms.. Cambridge University Press. External Links: ISBN 978-1-10-705713-5 Cited by: §I. [42] C.M. Bishop (2006) Pattern recognition and machine learning. Information Science and Statistics, Springer. External Links: ISBN 9780387310732, LCCN 2006922522, Link Cited by: §I. [43] J. Heaton (2018) Ian goodfellow, yoshua bengio, and aaron courville: deep learning - the mit press, 2016, 800 p, isbn: 0262035618.. Genet. Program. Evolvable Mach. 19 (1-2), p. 305–307. External Links: Link Cited by: §I. [44] C. Chow (1970) On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory 16 (1), p. 41–46. External Links: Document Cited by: §I. [45] K. Hornik (1991) Approximation capabilities of multilayer feedforward networks. Neural Networks 4 (2), p. 251–257. External Links: ISSN 0893-6080, Document, Link Cited by: §I. [46] S. Boyd and L. Vandenberghe (2004) Convex optimization. Cambridge University Press, Cambridge, UK. External Links: ISBN 978-0521833783 Cited by: §I. [47] D. P. Kingma and J. Ba (2014) Adam: a method for stochastic optimization. Note: cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015 External Links: Link Cited by: §IV. [48] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala (2019) PyTorch: an imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, Vol. 32. Cited by: §IV. Appendix A Appendix In this section we present the proofs for the main theoretical results in Section I. We first present some auxiliary results that will be used in the proofs of Theorems I.1 and I.2. For the sake of generality, in this section, we consider true posterior probabilities p=PY|X(⋅∣)p=P_Y|X(· ), rather than empirical one-hot encoded labels. Indeed, let Z=(Z1,…,ZC)Z=(Z_1,…,Z_C) denote the observed one-hot target, where Zi=Y=i,i∈1,…,C. Z_i= 1\Y=i\, i∈\1,…,C\. (7) Thus, for every individual observation, Zi∈0,1Z_i∈\0,1\ and ∑i=1CZi=1. _i=1^CZ_i=1. (8) We distinguish this observed target from the true conditional class probabilities pi():=ℙ(Y=i∣X=)=[Zi∣X=], p_i(x):=P(Y=i X=x)=E[Z_i X=x], (9) for i∈1,…,Ci∈\1,…,C\. Hence, although the observed target Z is one-hot, the conditional posterior ()=(p1(),…,pC()) (x)= (p_1(x),…,p_C(x) ) (10) need not be one-hot and satisfies pi()≥0,∑i=1Cpi()=1. p_i(x)≥ 0, _i=1^Cp_i(x)=1. (11) For a single observation with one-hot target Z, consider the objective ∑i=1CZilog(kp^i+p^C+1). _i=1^CZ_i (k p_i+ p_C+1 ). (12) Taking the conditional expectation with respect to the label Y given X=X=x yields [∑i=1CZilog(kp^i+p^C+1)|X=] [ . _i=1^CZ_i (k p_i+ p_C+1 )\, |\,X=x ] =∑i=1C[Zi∣X=]log(kp^i+p^C+1) = _i=1^CE [Z_i X=x ] (k p_i+ p_C+1 ) =∑i=1Cpi()log(kp^i+p^C+1). = _i=1^Cp_i(x) (k p_i+ p_C+1 ). (13) Therefore, the quantities pi()p_i(x) appearing in the theoretical optimization should be interpreted as the true conditional class probabilities rather than as the observed one-hot labels. The empirical loss is evaluated using one-hot targets, whereas the corresponding conditional population objective is obtained by averaging this loss over the conditional distribution of Y given X=X=x. A-A Auxiliary results First, let us consider the problem of optimizing Equation 3 in the probability simplex ΔC=q∈ℝC+1: ^C= \q ^C+1: qi≥0∀i∈1,…,C+1, \;q_i≥ 0\;∀\ i∈\1,…,C+1\, ∑i=1C+1qi=1 _i=1^C+1q_i=1 \ (14) Lemma A.1 The probability simplex ΔC ^C defined in 14 is convex, closed, and bounded. Proof: Let q,q′∈ΔCq,q ∈ ^C and let γ∈[0,1]γ∈[0,1]. Define q′=γq+(1−γ)q′. q =γ q+(1-γ)q . For every coordinate i, qi′=γqi+(1−γ)qi′≥0, q _i=γ q_i+(1-γ)q _i≥ 0, since qi,qi′≥0q_i,q _i≥ 0 and γ,1−γ≥0γ,1-γ≥ 0. Moreover, ∑i=1C+1qi′= _i=1^C+1q _i= ∑i=1C+1(γqi+(1−γ)qi′) _i=1^C+1 (γ q_i+(1-γ)q _i ) = = γ∑i=1C+1qi+(1−γ)∑i=1C+1qi′ γ _i=1^C+1q_i+(1-γ) _i=1^C+1q _i = = γ+(1−γ)=1. γ+(1-γ)=1. Hence q′∈ΔCq ∈ ^C, proving that ΔC ^C is convex. Next, observe that ΔC= ^C= (⋂i=1C+1q∈ℝC+1:qi≥0) ( _i=1^C+1\q ^C+1:q_i≥ 0\ ) ∩q∈ℝC+1:∑i=1C+1qi=1. ∩ \q ^C+1: _i=1^C+1q_i=1 \. Each set q∈ℝC+1:qi≥0 \q ^C+1:q_i≥ 0\ is closed, as it is the preimage of the closed interval [0,∞)[0,∞) under the continuous projection map q↦qiq q_i. Likewise, q∈ℝC+1:∑i=1C+1qi=1 \q ^C+1: _i=1^C+1q_i=1 \ is closed, being the preimage of the closed set 1\1\ under the continuous map q↦∑i=1C+1qiq _i=1^C+1q_i. Since an intersection of closed sets is closed, ΔC ^C is closed. Finally, let q∈ΔCq∈ ^C. Since qi≥0q_i≥ 0 and ∑i=1C+1qi=1, _i=1^C+1q_i=1, it follows that 0≤qi≤1,fori=1,…,C+1.0≤ q_i≤ 1,\;for\;i=1,…,C+1. Therefore, ‖q‖22=∑i=1C+1qi2≤∑i=1C+1qi=1, \|q\|_2^2= _i=1^C+1q_i^2≤ _i=1^C+1q_i=1, where the inequality follows from 0≤qi≤10≤ q_i≤ 1 implying qi2≤qiq_i^2≤ q_i. Consequently, ‖q‖2≤1,\|q\|_2≤ 1, and thus ΔC ^C is bounded. ∎Moreover, the following holds: Lemma A.2 The function in Equation 3 is concave in PY^|XP_ Y|X over the domain ΔC ^C. Proof: To show that W1(p,p^)rW_#1 (p, p )r is concave, it suffices to show that −W1(p,p^)r-W_#1 (p, p )r is convex. Observe that −W1(p,p^)r=∑j=1Cpj[−log(k⋅p^j+p^C+1)]. -W_#1 (p, p )r= _j=1^Cp_j [- \! (k· p_j+ p_C+1 ) ]. For each j, define gj(p^)=−log(k⋅p^j+p^C+1).g_j( p)=- \! (k· p_j+ p_C+1 ). Since k>0k>0 and p^∈ΔC p∈ ^C, we have k⋅p^j+p^C+1≥0.k· p_j+ p_C+1≥ 0. The function x↦−logx - x is convex33 3 We adopt the extended-value convention −log0:=+∞.- 0:=+∞. Thus, under the above convention, x↦−logx - x defines an extended-real-valued convex function on [0,∞)[0,∞). Moreover, the map p^↦k⋅p^j+p^C+1 p k· p_j+ p_C+1 is affine. Hence, by preservation of convexity under affine precomposition, each gj(p^)=−log(k⋅p^j+p^C+1)g_j( p)=- \! (k· p_j+ p_C+1 ) is convex on ΔC ^C in the extended-real sense. on (0,∞)(0,∞), since d2dx2(−logx)=1x2>0,x>0, d^2dx^2(- x)= 1x^2>0, x>0, and satisfies limx→0−logx=+∞. _x→ 0- x=+∞. Since pj≥0p_j≥ 0 for all j, the nonnegative weighted sum −W1(p,p^)r=∑j=1Cpjgj -W_#1 (p, p )r= _j=1^Cp_jg_j is convex. Therefore, W1(p,p^)rW_#1 (p, p )r is concave on ΔC ^C in the extended-real sense, with the corresponding convention log0=−∞ 0=-∞. ∎From Lemmas A.1 and A.2, the following can be concluded: Remark A.1 The optimization problem of maximizing W1(p,p^)rW_#1 (p, p )r over the probability simplex ΔC ^C involves the maximization over a convex, closed, and bounded set of a concave function. Hence, the problem is a Constrained Mathematical Programming (CMP) problem, and for the continuity of W1(p,p^)rW_#1 (p, p )r over the compact set ΔC ^C, it admits a global maximum. Due to the nature of the problem analyzed, we can build the following CMP problem, which is equivalent to the maximization of W1(p,p^)rW_#1 (p, p )r over the probability simplex ΔC ^C: maximizeW1(p,p^)rsubject to∑i=1C+1p^i=1,p^i≥0,i=1,…,C+1 \ aligned maximize &W_ (p, p )r\\ subject to & _i=1^C+1 p_i=1,\\ & p_i≥ 0, i=1,…,C+1 aligned . (15) Thus, the following holds: Proposition A.1 The formulation in Equation 15 encompasses the standard training with 1 as a special case when p^C+1=0 p_C+1=0 (cf. [32]), as it reduces to the minimization of the Kullback-Leibler divergence between the true and predicted distributions, which is equivalent to the cross-entropy loss. Proof: Let p^∈ΔC p∈ ^C with p^C+1=0 p_C+1=0. Then W1(p,p^)r W_#1 (p, p )r =∑j=1Cpjlog(k⋅p^j+p^C+1) = _j=1^Cp_j \! (k· p_j+ p_C+1 ) =∑j=1Cpjlog(k⋅p^j⋅pjp^j) = _j=1^Cp_j \! (k· p_j· p_j p_j ) =log(k)−H(p)−DKL(p∥p^), = \! (k )-H(p)-D_KL(p\| p), where H(p)H(p) is the Shannon entropy of p and DKL(p∥p^)D_KL(p\| p) is the Kullback-Leibler divergence between p and p p. Since log(k) \! (k ) and H(p)H(p) are constants with respect to p p, maximizing W1(p,p^)rW_#1 (p, p )r is equivalent to minimizing DKL(p∥p^)D_KL(p\| p), which is equivalent to minimizing the cross-entropy loss. ∎ A-B Proof of Theorem I.1 See I.1 Proof: The optimization problem in Eq. (15) can be analyzed using the KKT conditions. The corresponding Lagrangian is given by ℒ(p^,λ,μ) ( p,λ,μ) =W1(p,p^)r =W_#1 (p, p )r +λ(1−∑i=1C+1p^i) +λ (1- _i=1^C+1 p_i ) +∑i=1C+1μip^i, + _i=1^C+1 _i p_i, where λ∈ℝλ is the Lagrange multiplier associated with the equality constraint and μi≥0 _i≥ 0 are the multipliers corresponding to the non-negativity constraints. The KKT conditions are ∂ℒ∂p^i=∂W1(p,p^)r∂p^i−λ+μi=0,i=1,…,C+1,∑i=1C+1p^i=1,p^i≥0,i=1,…,C+1,μi≥0,i=1,…,C+1,μip^i=0,i=1,…,C+1. \ aligned & ∂ p_i= ∂ W_ (p, p )r∂ p_i-λ+ _i=0,&&i=1,…,C+1,\\ & _i=1^C+1 p_i=1,\\ & p_i≥ 0,&&i=1,…,C+1,\\ & _i≥ 0,&&i=1,…,C+1,\\ & _i p_i=0,&&i=1,…,C+1. aligned . For the objective in Equation 3, the stationarity conditions yield ∂ℒ∂p^i ∂ p_i =k⋅pik⋅p^i+p^C+1−λ+μi=0, = k· p_ik· p_i+ p_C+1-λ+ _i=0, for i=1,…,Ci=1,…,C, and ∂ℒ∂p^C+1 ∂ p_C+1 =∑i=1Cpik⋅p^i+p^C+1−λ+μC+1=0. = _i=1^C p_ik· p_i+ p_C+1-λ+ _C+1=0. Since p^C+1⋆>0 p_C+1 >0, complementary slackness implies μC+1=0 _C+1=0. Evaluating the stationarity condition at ^⋆ p therefore gives λ=∑i=1Cpik⋅p^i⋆+p^C+1⋆. λ= _i=1^C p_ik· p_i + p_C+1 . Since p^C+1⋆>0 p_C+1 >0, every denominator is strictly positive. Moreover, since p is a probability vector, pi≥0p_i≥ 0 for all i and at least one pi>0p_i>0. Hence, λ>0. λ>0. The complementary slackness condition naturally leads to two cases for each class output. Case 1: p^i⋆>0 p_i >0 In this case, complementary slackness implies μi=0 _i=0, yielding k⋅pik⋅p^i⋆+p^C+1⋆=λ, k· p_ik· p_i + p_C+1 =λ, or equivalently, p^i⋆=piλ−p^C+1⋆k. p_i = p_iλ- p_C+1 k. Notice how the requirement p^i⋆>0 p_i >0 implies that pi>λp^C+1⋆k, p_i>λ p_C+1 k, which means that a class becomes inactive whenever its probability falls below the threshold λp^C+1⋆kλ p_C+1 k, where p^C+1⋆k p_C+1 k is the ratio of the reserve output to the cost multiplier associated with betting on the class prediction. Case 2: p^i⋆=0 p_i =0 In this case, complementary slackness only implies that μi≥0 _i≥ 0. The stationarity condition becomes kpip^C+1⋆−λ+μi=0, k\,p_i p_C+1 -λ+ _i=0, where p^C+1⋆>0 p_C+1 >0 by assumption. Equivalently, μi=λ−kpip^C+1⋆. _i=λ- k\,p_i p_C+1 . Since μi≥0 _i≥ 0, it follows that kpip^C+1⋆≤λ, k\,p_i p_C+1 ≤λ, or equivalently, pi≤λp^C+1⋆k. p_i≤λ p_C+1 k. Since λ>0λ>0, this is equivalent to piλ−p^C+1⋆k≤0. p_iλ- p_C+1 k≤ 0. Combining the two cases, for every i∈1,…,Ci∈\1,…,C\ we obtain p^i⋆=maxpiλ−p^C+1⋆k, 0, p_i = \ p_iλ- p_C+1 k,\,0 \, which proves the result. ∎ A-C Proof of Theorem I.2 See I.2 Proof: We consider separately the regimes k<1k<1, k>Ck>C, and the boundary cases k=1k=1 and k=Ck=C. Case 1: k<1k<1. For any feasible p p and every j∈1,…,Cj∈\1,…,C\, kp^j+p^C+1≤p^j+p^C+1≤1, k p_j+ p_C+1≤ p_j+ p_C+1≤ 1, where the second inequality follows from ∑j=1C+1p^j=1 _j=1^C+1 p_j=1. Hence, log(kp^j+p^C+1)≤0, (k p_j+ p_C+1 )≤ 0, and therefore W1(p,p^)r≤0. W_#1 (p, p )r≤ 0. The full-reservation solution (p^1,…,p^C,p^C+1)=(0,…,0,1) ( p_1,…, p_C, p_C+1)=(0,…,0,1) attains W1(p,p^)r=0W_#1 (p, p )r=0, and is therefore optimal. Thus, for k<1k<1, full reservation is optimal. Case 2: k>Ck>C. Suppose, for contradiction, that an optimal solution satisfies p^C+1⋆>0 p_C+1 >0. Let S=j:p^j⋆>0,a=|S|,PS¯=∑j∉Spj. S=\j: p_j >0\, a=|S|, P_ S= _j∉ Sp_j. Since p^C+1⋆>0 p_C+1 >0, complementary slackness gives μC+1=0 _C+1=0. Moreover, for every j∈Sj∈ S, Equation 6 implies kp^j⋆+p^C+1⋆=kpjλ. k p_j + p_C+1 = kp_jλ. Hence, the stationarity condition for the reservation coordinate becomes aλk+PS¯p^C+1⋆=λ. aλk+ P_ S p_C+1 =λ. If PS¯>0P_ S>0, then PS¯=λ(1−ak)p^C+1⋆. P_ S=λ (1- ak ) p_C+1 . Summing Equation 6 over j∈Sj∈ S and using ∑j∈Sp^j⋆=1−p^C+1⋆ _j∈ S p_j =1- p_C+1 gives 1−PS¯λ−ap^C+1⋆k=1−p^C+1⋆. 1-P_ Sλ- a p_C+1 k=1- p_C+1 . Substituting the preceding expression for PS¯P_ S yields λ=1λ=1. Therefore, PS¯=(k−a)p^C+1⋆k. P_ S=(k-a) p_C+1 k. On the other hand, for every inactive coordinate j∉Sj∉ S, Equation 6 implies pj≤p^C+1⋆k. p_j≤ p_C+1 k. Summing over the C−aC-a inactive coordinates gives PS¯≤(C−a)p^C+1⋆k. P_ S≤(C-a) p_C+1 k. Combining the last two relations and using p^C+1⋆>0 p_C+1 >0 yields k−a≤C−a, k-a≤ C-a, and hence k≤C, k≤ C, contradicting the assumption k>Ck>C. It remains to consider PS¯=0P_ S=0. In this case, the reservation stationarity condition reduces to aλk=λ. aλk=λ. Since λ>0λ>0, this implies a=k. a=k. However, a≤C<ka≤ C<k, which is impossible. Consequently, p^C+1⋆=0. p_C+1 =0. With no reservation, the objective becomes W1(p,p^)r=∑j=1Cpjlog(kp^j)=log(k)+∑j=1Cpjlogp^j. W_#1 (p, p )r= _j=1^Cp_j (k p_j)= (k)+ _j=1^Cp_j p_j. Thus, maximizing W1(p,p^)rW_#1 (p, p )r is equivalent to maximizing ∑j=1Cpjlogp^j _j=1^Cp_j p_j over the probability simplex. By Gibbs’ inequality, the maximum is attained at p^j⋆=pj,j=1,…,C. p_j =p_j, j=1,…,C. Hence, for k>Ck>C, every optimal solution has no reservation and predicts according to the posterior distribution. Case 3: k=1k=1. For every feasible p p and every j∈1,…,Cj∈\1,…,C\, p^j+p^C+1≤1. p_j+ p_C+1≤ 1. Therefore, W1(p,p^)r≤0, W_#1 (p, p )r≤ 0, and full reservation attains W1(p,p^)r=0W_#1 (p, p )r=0. Hence, full reservation is optimal. However, the optimizer need not be unique. For example, suppose the posterior is concentrated on a single class i, so that pi=1,pj=0for j≠i. p_i=1, p_j=0 j≠ i. Then every feasible solution satisfying p^i+p^C+1=1,p^j=0for j≠i, p_i+ p_C+1=1, p_j=0 j≠ i, achieves W1(p,p^)r=log(p^i+p^C+1)=0. W_#1 (p, p )r= ( p_i+ p_C+1)=0. Thus, distinct prediction–reservation allocations may be equally optimal when k=1k=1. Case 4: k=Ck=C. The optimizer need not be unique at this boundary either. Consider the uniform posterior pj=1C,j=1,…,C. p_j= 1C, j=1,…,C. For any α∈[0,1]α∈[0,1], define p^j(α)=1−αC,j=1,…,C,p^C+1(α)=α. p_j^(α)= 1-αC, j=1,…,C, p_C+1^(α)=α. Then Cp^j(α)+p^C+1(α)=1,j=1,…,C, C p_j^(α)+ p_C+1^(α)=1, j=1,…,C, and consequently W1(p,p^)r=∑j=1C1Clog(1)=0. W_#1 (p, p )r= _j=1^C 1C (1)=0. To see that this value is optimal, Jensen’s inequality gives W1(p,p^)r W_#1 (p, p )r ≤ ≤ log(1C∑j=1C(Cp^j+p^C+1))= ( 1C _j=1^C (C p_j+ p_C+1 ) )= log(1)=0. (1)=0. Thus, every member of this family for different values of α is optimal, showing that distinct prediction–reservation allocations may also be equally optimal when k=Ck=C. Therefore, k<1k<1 corresponds to the full-reservation regime, while k>Ck>C corresponds to the no-reservation prediction regime. The boundary values k=1k=1 and k=Ck=C may admit multiple optimal prediction–reservation allocations. The cost-multiplier mixed-regime, which is the most interesting case, is 1<k<C1<k<C. ∎ A-D Additional numerical results In this section, we present additional numerical results, which complement the results already presented in the main body of the paper. Fig. 7: Synthetic benchmark data – Ablation study over the variable k for the higher posterior overlap case for the benchmark in Section IV-A. A-E Benchmark generation Algorithm 1 Synthetic Data Generation with Target Bayes Error 1: Number of samples N; feature means μ and scales σ; correlation matrix R; causal defect map ℰE; class priors π; target Bayes error ε⋆ 2: Synthetic labeled dataset D 3: Sample correlated process noise ∼(,)Z (0,R) 4: Generate process features =+⊙X= μ+ σ 5: Normalize each feature relative to its specification limits to obtain deviations 6: Compute class scores sk(,ℰ)s_k( ;E) from the causal defect map 7: Initialize gain interval [gmin,gmax][g_ ,g_ ] 8: repeat 9: g←(gmin+gmax)/2g←(g_ +g_ )/2 10: Calibrate class offsets bkb_k so that the mean posterior probabilities match the desired priors πk _k 11: Compute the class posterior pik=P(Y=k∣i)=softmaxk(gsk(i)+bk)p_ik=P(Y=k _i)=softmax_k (g\,s_k( _i)+b_k ) 12: Compute the resulting Bayes error εB=1N∑i=1N(1−maxkpik) _B= 1N _i=1^N (1- _kp_ik ) 13: if εB>ε⋆ _B> then 14: gmin←g_ ← g 15: else 16: gmax←g_ ← g 17: end if 18: until εB≈ε⋆ _B≈ 19: for i=1,…,Ni=1,…,N do 20: Sample yi∼Categorical(pi1,…,piK)y_i (p_i1,…,p_iK) 21: end for 22: ←(i,yi)i=1ND←\(x_i,y_i)\_i=1^N 23: return D