Paper deep dive

Calibration in One Go: Jointly Learning Accurate and Calibrated Neural Networks

Nektarios Kalampalikis, Pablo M. Olmos, Isabel Valera

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 157

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%

Last extracted: 10/13/2026, 2:56:39 AM

Summary

The paper introduces 'Calibration in One Go' (COG), a framework that jointly optimizes neural network accuracy and calibration using a bilevel Stackelberg formulation. Unlike post-hoc methods that require separate calibration datasets and models, COG trains the calibrator end-to-end with the backbone model using distinct learning rates. The approach eliminates the need for additional calibration parameters or data, with the final classification layer serving as an effective calibrator (COG-FC), outperforming state-of-the-art methods with minimal complexity.

Entities (9)

COG · method · 98%Bilevel Optimization · technique · 92%CIFAR-100 · dataset · 90%Matrix Scaling · method · 90%ResNet101 · model · 90%Temperature Scaling · method · 90%Label Smoothing · method · 88%multi-task learning · task · 85%Stackelberg Game · concept · 85%

Relation Signals (7)

COG-FC → isvariantof → COG

confidence 95% · using the final classification layer itself as the calibrator yields accurate, well-calibrated models... COG-FC

COG → uses → Bilevel Optimization

confidence 95% · we show that standard (post-hoc) calibrators can be trained end-to-end with the backbone via a bilevel Stackelberg formulation.

ResNet101 → isevaluatedon → CIFAR-100

confidence 90% · train ResNet101 models... on SVHN, CIFAR-10, and CIFAR-100

COG → supports → Temperature Scaling

confidence 90% · our framework supports calibrators ranging from temperature and matrix scaling to NN layers

COG → supports → Matrix Scaling

confidence 90% · our framework supports calibrators ranging from temperature and matrix scaling to NN layers

COG → appliesto → multi-task learning

confidence 85% · avoiding the multiplicative overhead of separate calibration components in settings such as multi-task learning

COG → outperforms → SOTA calibrators

confidence 85% · Our approach outperforms SOTA calibrators with minimal added complexity

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Neural networks (NNs), while highly flexible and versatile, are often poorly calibrated. Post-hoc calibration methods adjust the outputs of an already trained model using a separate calibration dataset and additional calibration parameters. This process adds overhead that requires separate optimization of the calibrator and subsequent model selection. In this paper, we show that standard (post-hoc) calibrators can be trained end-to-end with the backbone via a bilevel Stackelberg formulation. This formulation eliminates the need for a separate calibration set or a dedicated calibration loss by jointly optimizing accuracy and calibration using cross-entropy, introducing only one additional hyperparameter: the calibrator learning rate. Although our framework supports calibrators ranging from temperature and matrix scaling to NN layers, our results show that using the final classification layer itself as the calibrator yields accurate, well-calibrated models with no additional parameters. Our approach outperforms SOTA calibrators with minimal added complexity, while also avoiding the multiplicative overhead of separate calibration components in settings such as multi-task learning.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

153,455 characters extracted from source content.

Expand or collapse full text

Calibration in One Go: Jointly Learning Accurate and Calibrated Neural Networks Nektarios Kalampalikis †thanks: kalampalikis@cs.uni-saarland.de Affiliation: Dept. of Computer Science Affiliation: Saarland University, Germany Pablo M. Olmos Affiliation: Dept. of Signal Theory and Communications Affiliation: Universidad Carlos I de Madrid, Spain Isabel Valera Affiliation: Dept. of Computer Science Affiliation: Saarland University, Germany Abstract Neural networks (NNs), while highly flexible and versatile, are often poorly calibrated. Post-hoc calibration methods adjust the outputs of an already trained model using a separate calibration dataset and additional calibration parameters. This process adds overhead that requires separate optimization of the calibrator and subsequent model selection. In this paper, we show that standard (post-hoc) calibrators can be trained end-to-end with the backbone via a bilevel Stackelberg formulation. This formulation eliminates the need for a separate calibration set or a dedicated calibration loss by jointly optimizing accuracy and calibration using cross-entropy, introducing only one additional hyperparameter: the calibrator learning rate. Although our framework supports calibrators ranging from temperature and matrix scaling to N layers, our results show that using the final classification layer itself as the calibrator yields accurate, well-calibrated models with no additional parameters. Our approach outperforms SOTA calibrators with minimal added complexity, while also avoiding the multiplicative overhead of separate calibration components in settings such as multi-task learning. 1 Introduction Machine learning, and neural networks in particular, have become an important part of modern society, with their predictions increasingly informing downstream decision making. In many classification settings, however, the predicted class alone is insufficient: downstream systems may instead rely on the predicted probability of each outcome as a measure of uncertainty. Such probabilities play an important role in high-stakes applications, including healthcare (Ricci Lara et al., 2023), fair decision-making (Gupta et al., 2026), and machine learning-based risk assessment (Brito et al., 2023), among others (Wang, 2023; He et al., 2026). Beyond supporting decision making, predictive probabilities can also provide insight into what a model is unsure about, thus offering an additional layer of model understanding and human-AI collaboration (Tomsett et al., 2020). However, for the predicted class probabilities to be useful in these settings, they must accurately reflect the model’s predictive uncertainty. In practice, machine learning models, and neural networks in particular, are often miscalibrated (Guo et al., 2017). Classifier calibration aims to address this issue by aligning a classifier’s predicted probabilities with the observed likelihood of the corresponding outcomes (Platt, 2000; Guo et al., 2017; Murphy, 2023). For instance, among predictions assigned 90% probability to class C, class C should be correct 90% of the time. For example, as illustrated by the reliability diagram in Fig. 1(a), a perfectly calibrated classifier lies along the diagonal, whereas deviations from the diagonal indicate calibration error (Guo et al., 2017). Various approaches have been proposed to calibrate models, including train-time methods and post-hoc methods (Wang, 2023). Train-time approaches often incorporate additional calibration-related terms into the training objective, whereas post-hoc approaches train a separate model on the output of a trained model (Guo et al., 2017), as illustrated in Fig. 1(b). Although effective, these approaches introduce additional complexity into the model development pipeline. Train-time methods require the model designer to choose and tune additional terms or use a calibration-specific objective, while post-hoc methods require training an additional model and, typically, a separate held-out calibration set. On top of this, both of these families of methods come with their own complexity. For example, post-hoc calibrators can themselves overfit, motivating their regularization (Berta et al., 2025; Berta et al., 2026a; Berta et al., 2026b) or using simpler calibrators such as Temperature Scaling (Guo et al., 2017). This pipeline complexity is particularly pronounced in settings like Multi-Task Learning (MTL) (Caruana, 1997; Javaloy & Valera, 2022), where a shared model serves multiple tasks, potentially requiring a separate calibrator for each. Moreover, the richer annotation requirements of MTL make reserving additional data exclusively for calibration particularly costly. Despite these challenges, calibration in MTL remains comparatively underexplored. Motivation. In this paper, we aim to obtain neural networks that are simultaneously accurate and well calibrated in a single training procedure, while retaining the efficiency of standard training and avoiding a separate post-hoc calibration stage or additional calibration data. To this end, we formulate calibration as a bilevel optimization (BLO) problem (Dempe, 2002; Beck et al., 2027), where standard model training is the follower and the calibrator acts as the leader, as shown in Fig. 1(c). We efficiently solve the resulting Stackelberg problem with a single forward and backward pass per iteration, jointly updating model and calibrator parameters within the same training step using separate learning rates. This introduces one additional hyperparameter beyond standard training: the calibrator learning rate. The resulting framework integrates calibration directly into end-to-end training while retaining the efficiency of standard training: (i) it requires neither a calibration-specific loss nor a separate calibration dataset; (i) it imposes no additional architectural constraints and accommodates calibrators ranging from temperature and matrix scaling to neural layers, adding at most the trainable parameters of the corresponding post-hoc calibrator; and (i) it introduces only one additional hyperparameter, namely the calibrator learning rate. Empirically, our approach achieves accuracy and calibration competitive with substantially more complex state-of-the-art methods while limiting the additional calibration overhead to a single hyperparameter. Remarkably, additional calibration parameters are unnecessary: using the final classification layer itself as the calibrator yields accurate, well-calibrated models, as demonstrated by the example in Fig. 1(a). We further show that this single-stage, low-overhead formulation extends naturally to more complex settings such as MTL, where conventional calibration is cumbersome. (a) (b) (c) Figure 1: (a) Reliability diagrams on CIFAR-100 for the baseline, COG, and temperature scaling, with COG achieving the best calibration. (b) Post-hoc calibration: the model is trained and frozen before fitting a calibrator. (c) Our framework, COG, integrates the calibrator into the training procedure and constrains its learning rate to be slower than that of the model. 2 Related Works Post-hoc calibration. Post-hoc calibration modifies the output probabilities of a trained classification model by learning a transformation of its outputs on a held-out calibration set. Given an input x, a classifier f is typically trained with cross-entropy (CE), after which a calibrator c is fitted to a held-out calibration set, resulting in the calibrated model c∘f⁡(x)=c⁡(f⁡(x))c f(x)=c(f(x)). Parametric post-hoc calibrators include Platt Scaling (Platt, 2000), designed for binary classification, which applies a learned sigmoid transformation to the logits z, i.e., σ⁡(a+b)σ(a z+b). Multi-class extensions include Vector Scaling and Matrix Scaling (MS) (Guo et al., 2017), which learn more flexible transformations of the logits. Temperature Scaling (TS) (Guo et al., 2017) uses a single learned scalar T to rescale the logits, σ⁡(/T)σ( z/T). Despite their simplicity, these models can still overfit the calibration set, motivating recent work on regularizing calibrators (Berta et al., 2025; Berta et al., 2026a; Berta et al., 2026b). Nonparametric approaches include histogram binning (Zadrozny & Elkan, 2001) and isotonic regression (Zadrozny & Elkan, 2002), which directly estimate calibration mappings from empirical frequencies. However, their flexibility can make them prone to overfitting, particularly with limited calibration data (Menon et al., 2012). Moreover, multiclass extensions typically rely on one-vs-rest schemes (Zadrozny & Elkan, 2002; Murphy, 2023), substantially increasing pipeline complexity. Overviews of post-hoc calibration methods can be found in (Guo et al., 2017; Murphy, 2023; Berta et al., 2026a). Progressive training approaches. More recently, multi-stage neural network training procedures have been proposed to improve calibration. In particular, Two-Stage Training (TST) (Jordahn & Olmos, 2024; Khodaverdian et al., 2025) first trains the network and then retrains only some of the final fully connected layers while freezing earlier representations. This can be extended to impose a probabilistic prior on the final hidden layer, turning the model in a last-layer Bayesian Neural Network, that further enhances calibration (V-TST) (Jordahn & Olmos, 2024). Other works connect poor calibration to over-compression (Tao et al., 2024; Wang & Zhang, 2024), especially in later layers through the information bottleneck principle, with Wang & Zhang (2024) proposing progressively freezing earlier layers during training. The aforementioned methods can be cast as bilevel (or multi-level) optimization problems, which we convert into train-time calibration approaches. Train-time calibration. Train-time calibration methods modify the learning objective or training procedure to encourage calibrated outputs. Examples include adding LpL_p regularization (Joo & Chung, 2020), incorporating sliced Wasserstein distance penalties (Joo & Chung, 2020), or including differentiable bin membership terms in the loss (Karandikar et al., 2021). Other approaches include label smoothing (Szegedy et al., 2016; Müller et al., 2019) or replacing CE with focal loss (Mukhoti et al., 2020), both of which can be interpreted as CE with an additional maximum-entropy regularizer (Meister et al., 2020; Mukhoti et al., 2020). Finally, several works propose specialized loss functions designed to produce calibrated models (Bohdal et al., 2021; Popordanoska et al., 2022). Our approach provides a complementary train-time mechanism: rather than requiring a particular calibration loss, it modifies the optimization structure and can therefore be used with standard training objectives, such as CE, or combined with existing calibration-aware losses, as seen in our experiments. Bilevel Optimization and Calibration. Our work leverages BLO to turn post-hoc calibration into a train-time procedure. Related work has also explored calibration through the lens of BLO. In (Bohdal et al., 2021), meta-learning alternates between CE updates for model training and optimization of the hyperparameters of calibration regularizers. Similarly, BLO has been used to learn validation sample weights to improve calibration (Sanguin et al., 2025). This approach is computationally demanding, requiring multiple optimization steps to approximate the bilevel problem (Franceschi et al., 2018), with a follower minimizing CE on trainD_ train and a leader optimizing validation-sample weights. These approaches are orthogonal to ours, as they either enhance existing calibration methods via BLO or formulate a different bilevel objective. Our approach instead addresses calibration within standard backpropagation-based training, avoiding expensive optimization and held-out calibration data. 3 Towards Calibration in One Go (Post-hoc) Calibration as a Bilevel Optimization Problem. Bilevel optimization problems are hierarchical optimization problems involving two decision makers, a leader and a follower. The leader makes a decision while anticipating the follower’s best response to that decision, thus inducing a hierarchical dependency between the two levels (Dempe, 2002; Beck et al., 2027). Learning a neural network that is both accurate and well calibrated can be formulated as a BLO problem, in which predictive accuracy is optimized at the follower level and calibration at the leader level: minθcal⁡(x,y)∼[ℒcal(cθcal(fθtrain∗(x)),y)] _ _cal\,E_(x,y) [L_cal(c_ _cal(f_ _train^*(x)),y)] (Leader) (Leader) (1) s.t. θtrain∗=argminθtrain⁡(x,y)∼[ℒtrain(fθtrain(x),y)] .t. _train^*= _ _train\,E_(x,y) [L_train(f_ _train(x),y)] (Follower) (Follower) where θtrain _train and θcal _cal denote the parameters of the neural network fθtrainf_ _train and calibrator cθcalc_ _cal, respectively, and ℒtrainL_train and ℒcalL_cal their corresponding objectives. Thus, the term cθcal(fθtrain(x))c_ _cal(f_ _train(x)) denotes the resulting calibrated classifier. (x,y)(x,y) denotes a sample drawn from the data distribution D. Standard post-hoc calibration first trains the predictive model on a training set and then, keeping the resulting model parameters θtrain∗ _train^* fixed, learns the calibration parameters on a separate calibration set. That is, calibration is performed only after model training, so the calibration objective does not influence which predictive-model solution is learned, thus skipping their interaction. From a BLO perspective, post-hoc calibration can therefore be interpreted as a decoupled approximation of the BLO problem in Eq. (1), in which the follower is solved first and its solution is subsequently treated as fixed when optimizing the leader. However, since decoupling is optimal only when the follower problem admits a unique solution (Dempe, 2002; Beck et al., 2027), it is generally not justified for neural network training, where multiple equivalent local optima typically exist (Breiman, 2001; Goodfellow et al., 2016; Rudin et al., 2024). This motivates our approach to jointly optimize model training and calibration: rather than fixing an arbitrary follower solution before calibration, we allow the calibrator to influence which neural network parameters are learned. Our goal is therefore to solve the bilevel problem above in a way that jointly learns accurate and well-calibrated neural networks while retaining the efficiency of standard neural network training. Calibration in One Go (COG). The bilevel formulation of Eq. (1) naturally admits a Stackelberg-game interpretation (Von Stackelberg, 2010): the calibrator acts as the leader, updating the calibration parameters θcal _cal, while anticipating the response of the classifier (the follower), which in turn adapts the neural network parameters θtrain _train to minimize the training loss. This joint problem can be addressed efficiently using gradient-based bilevel optimization methods (Liu et al., 2021). Specifically, we employ a first-order, Stackelberg-inspired scheme in which the leader and follower parameters are simultaneously updated iteratively using gradients of their respective objectives, with distinct learning rates for the two levels, in the spirit of Gradient Descent-Ascent (GDA) (Fiez et al., 2020; Javaloy & Valera, 2022). Further details on BLO and GDA are provided in App A. For computational efficiency, we further make two simplifying choices. First, we use the same loss function for both levels, e.g., cross-entropy loss, such that ℒcal:=ℒtrainL_cal:=L_train. Second, we approximate both expectations in Eq. (1) using the same minibatch BtB_t from the training data, i.e., cal=trainD_ cal=D_ train. Consequently, both objectives can be evaluated on the same forward pass and their gradients computed with a single backpropagation pass, as in standard neural network training. Thus, the resulting procedure differs from standard training only in that the neural network and calibration parameters are updated with distinct learning rates as: θcalt+1=θcalt−λc∑i∈Bt∇θcalℒtrain(cθcal(fθtrain(xi)),yi) _cal^t+1= _cal^t- _c _i _t _ _calL_train(c_ _cal(f_ _train(x_i)),y_i) (2) θtraint+1=θtraint−λf∑i∈Bt∇θtrainℒtrain(cθcal(fθtrain(xi)),yi),s.t. λc<λf _train^t+1= _train^t- _f _i _t _ _trainL_train(c_ _cal(f_ _train(x_i)),y_i), .t. _c< _f where λc _c and λf _f denote the learning rates of the leader and follower, respectively. The condition λc<λf _c< _f preserves the Stackelberg hierarchy by ensuring that the follower adapts on a faster timescale than the leader, which under standard assumptions guarantees asymptotic convergence of the joint dynamics to a local Stackelberg solution (Fiez et al., 2020; Javaloy & Valera, 2022). We remark that, while these joint gradient updates do not guarantee convergence to an optimal Stackelberg solution, they provide a simple and efficient approximation that performs well in practice (Fiez et al., 2020; Javaloy & Valera, 2022). From Post-hoc Calibration to COG. Implementing standard post-hoc calibration methods within COG is straightforward and independent of the neural network’s architecture. For example, Temperature Scaling can be implemented by introducing a single additional parameter (COG-TS), Matrix Scaling by an additional weight matrix and an additional bias vector (COG-MS), and TST by introducing multiple calibration layers (COG-TST). COG can also be applied without introducing additional trainable parameters (COG-FC). Recent work has shown that the internal representations of a neural network can be as effective for prediction as its final classification layer (Wang & Zhang, 2024); thus, the final classification layer can be treated as the calibrator within COG, allowing the method to be applied to arbitrary neural network architectures. Finally, COG is not only restricted to post-hoc CE-based calibration. For example, by replacing the objectives at both levels with the Label Smoothing objective, we obtain COG+LS. Together, these variants demonstrate the flexibility of the COG framework across different calibration methods. A summary of the different calibration options and COG variants is provided in Tab. 1. The COG variants provide more lightweight, end-to-end, alternatives to their corresponding post-hoc calibration approaches in both single-task and multi-task settings (Tab. 7). In the single-task setting, COG-FC has a complexity comparable to Label Smoothing, while in the multi-task setting, COG-FC is the simplest calibration approach among all evaluated methods. Table 1: Complexity of different calibration methods in Single-Task settings, where K denotes the number of classes. Tab. 7 extends this comparison to MTL settings. # Extra End Calibration Extra Method Trainable Params. To End Data Burden Single-Task Baseline 0 ✓ ✗ - COG-FC 0 ✓ ✗ 1 Learning Rate Label Smoothing 0 ✓ ✗ 1 Smoothing Level COG+LS 0 ✓ ✗ (a) 1 Smoothing Level (b) 1 Learning Rate Temp. Scaling 1 ✗ ✓ Calibrator Training COG-TS 1 ✓ ✗ 1 Learning Rate Matrix Scaling K×K+K× K+K ✓ ✓ Calibrator Training COG-MS K×K+K× K+K ✓ ✗ 1 Learning Rate TST N ✗ ✗ N Training, Architecture COG-TST N ✓ ✗ 1 Learning Rate V-TST 2 NNs ✗ ✗ (a) ELBO training of 2 NNs (b) N architecture (c) MC Sampling 4 Experiments We structure our empirical evaluation in three stages. First, we compare standard post-hoc calibration methods with their COG counterparts to isolate the effect of replacing post-hoc calibration with joint training. Second, we benchmark COG against calibrators from different methodological families under increasingly challenging conditions, including in-distribution data, out-of-distribution data, and covariate shift. Finally, we evaluate COG in the more complex setting of multi-task learning, where calibration must be achieved jointly across multiple prediction tasks. Experimental Setup and Metrics We focus on multiclass image classification and train ResNet101 models (He et al., 2016) from scratch on SVHN (Netzer et al., 2011), CIFAR-10, and CIFAR-100 (Krizhevsky et al., 2009), following the setup of (Jordahn & Olmos, 2024). Full experimental details are provided in App. B, extended results in App. C, and experiments with additional CNN backbones in App. D. We report accuracy together with two scoring rules, negative log-likelihood (NLL) and the Brier score (Glenn et al., 1950), following recent recommendations for calibration evaluation (Minderer et al., 2021; Silva Filho et al., 2023; Berta et al., 2025; Khodaverdian et al., 2025; Chidambaram & Ge, 2025). App. C additionally reports ECE (Naeini et al., 2015), RMSCE (Hendrycks et al., 2018) and MCE (Naeini et al., 2015) across multiple binning choices. 4.1 Comparing Post-hoc Methods and Their COG Counterparts We compare each post-hoc calibrator with its corresponding COG variant in a pairwise manner, as shown in Tab. 1, matching the number of calibration parameters within each pair. For all COG variants and (V-)TST, calibration is learned jointly on the training set, whereas post-hoc calibrators are fitted on the validation set. Results As shown in Fig. 2, the COG variants generally match or outperform their post-hoc counterparts across datasets and metrics, with differences otherwise largely within the observed variance. One notable exception is Matrix Scaling on CIFAR-10, where the post-hoc variant appears to benefit from direct access to calibration data. On CIFAR-100, however, the post-hoc Matrix Scaling optimization fails to converge, whereas COG-MS remains stable and trains successfully within our joint optimization framework. We further observe that the different COG variants perform very similarly across all three metrics and datasets, suggesting that additional calibrator complexity provides limited benefit. In particular, COG-FC, which is our approach without additional trainable parameters, performs on par with the more expressive calibration approaches. Overall, these results demonstrate that COG-FC provides a simple yet effective way to obtain models that are both accurate and well calibrated, without requiring a separate calibration stage, additional calibration parameters, or calibration data. In App. C.1, we additionally report calibration error metrics (Tables 9-11), and reliability diagrams (Fig. 5). Figure 2: Pairwise comparison of different calibrators and their corresponding COG variants. The warning symbol indicates failure to converge. 4.2 Comparison Across Calibration Families We compare representative methods from four families of calibrators: train-time calibration via Label Smoothing, post-hoc calibration via Temperature Scaling, a Bayesian calibration method via V-TST (with 10 Monte Carlo samples) (Jordahn & Olmos, 2024; Khodaverdian et al., 2025), and our (also train-time) calibration approach COG-FC and its Label Smoothing variant COG+LS. We select COG-FC as the representative COG variant because it achieves strong performance while introducing no additional calibration parameters; COG+LS additionally replaces the standard cross-entropy loss with the calibration-regularized objective used by Label Smoothing. Moreover, since no single metric fully captures the trade-off between predictive accuracy and calibration quality, several works recommend considering the Pareto front between these objectives, allowing their relative importance to be determined by the application (Minderer et al., 2021; Silva Filho et al., 2023; Chidambaram & Ge, 2025; Sharifdeen et al., 2026). For a compact summary, we compute, for each dataset, the unweighted average rank of each method across Accuracy, NLL, and Brier score (Javaloy et al., 2025). We denote the method with the best average rank as Top-ranked. App. C.2 and C.3, report results for all considered calibration methods and calibration errors. In-distribution (ID) results. As shown in Tab. 2, COG-FC still remains competitive across all methods and is the top ranked method in CIFAR-100. We also observe that COG+LS is highly competitive, generally outperforming the Label Smoothing variant especially in accuracy. More broadly, COG variants remain highly competitive, with a COG variant being the top-ranked method in two of the three evaluated cases. In Fig. 3, we compare the reliability diagrams of COG-FC and the top-ranked model. For CIFAR-100, we compare COG-FC with COG+LS. Overall, the differences are small, with COG-FC closely matching the top-ranked models, with a deviation for CIFAR-10 around 25% confidence. These results further demonstrate that effective in-distribution calibration of neural networks can be achieved without increasing the model’s parameter count or substantially increasing the complexity of the overall pipeline. Table 2: Comparison of calibration metrics across datasets, and methods. The best value in each column among the subset of models is bolded. The Top Ranked∗Ranked^* row reports the best-performing model per dataset based on average rank across metrics as determined from the full results in Tab. 12: COG-MS on SVHN, Matrix Scaling on CIFAR-10, and COG-FC on CIFAR-100. SVHN CIFAR-10 CIFAR-100 Method ACC (↑)( ) NLL (↓)( ) Brier (↓)( ) ACC (↑)( ) NLL (↓)( ) Brier (↓)( ) ACC (↑)( ) NLL (↓)( ) Brier (↓)( ) Baseline 95.770.1895.77_0.18 0.320.010.32_0.01 0.070.000.07_0.00 90.500.2190.50_0.21 0.640.020.64_0.02 0.170.000.17_0.00 65.041.3365.04_1.33 2.980.062.98_0.06 0.600.020.60_0.02 COG-FC 95.840.2095.84_0.20 0.160.000.16_0.00 0.060.000.06_0.00 89.940.2589.94_0.25 0.380.010.38_0.01 0.160.000.16_0.00 70.460.3070.46_0.30 1.080.011.08_0.01 0.410.000.41_0.00 COG+LS 96.250.1596.25_0.15 0.210.000.21_0.00 0.070.000.07_0.00 89.920.2189.92_0.21 0.380.000.38_0.00 0.160.000.16_0.00 72.670.7872.67_0.78 1.310.041.31_0.04 0.410.010.41_0.01 Label Smoothing 95.970.1295.97_0.12 0.230.000.23_0.00 0.070.000.07_0.00 90.050.3190.05_0.31 0.430.010.43_0.01 0.160.000.16_0.00 65.760.8865.76_0.88 1.580.041.58_0.04 0.480.010.48_0.01 Temp. Scaling 95.770.1895.77_0.18 0.190.010.19_0.01 0.070.000.07_0.00 90.500.2190.50_0.21 0.380.010.38_0.01 0.150.000.15_0.00 65.041.3365.04_1.33 1.780.031.78_0.03 0.530.020.53_0.02 V-TST 94.660.0394.66_0.03 0.210.000.21_0.00 0.080.000.08_0.00 90.310.1690.31_0.16 0.340.010.34_0.01 0.150.000.15_0.00 52.710.2552.71_0.25 1.810.011.81_0.01 0.610.000.61_0.00 Top Ranked∗Ranked^* 96.020.1196.02_0.11 0.160.000.16_0.00 0.060.000.06_0.00 90.740.1790.74_0.17 0.280.000.28_0.00 0.140.000.14_0.00 70.460.3070.46_0.30 1.080.011.08_0.01 0.410.000.41_0.00 Figure 3: Reliability diagrams for COG-FC and the top-ranked methods using 15 bins. For CIFAR-100, where COG-FC is the top-ranked method, we compare it with COG+LS. Out of distribution (OOD) & covariate shift. Lastly, we assess whether ID calibration transfers to OOD detection and covariate shifts, following (Ovadia et al., 2019; Jordahn & Olmos, 2024). For covariate shifts, we use CIFAR-10-C, CIFAR-100-C (Hendrycks & Dietterich, 2019), and rotated SVHN (Jordahn & Olmos, 2024). OOD samples come from different datasets and lack valid labels in the original label space, whereas covariate shifts preserve the original labels under input corruption. For OOD detection, we use maximum softmax probability as the confidence score and treat ID samples as positive and OOD samples as negative. We report AUROC (Bradley, 1997) and FPR@95TPR, where higher AUROC and lower FPR95 indicate better OOD detection. As shown in Tab. 3, COG-FC performs particularly well for OOD detection, with COG+LS also achieving strong results. Across the three datasets, a COG variant achieves the best performance in each case. Under covariate shift (Tab. 4), we observe trends consistent with prior work (Ovadia et al., 2019): regularization-based approaches such as Label Smoothing are often more robust, and COG+LS generally improves over COG-FC. At the same time, performance varies across datasets and shift types, highlighting that robustness to distribution shift is distinct from in-distribution calibration (Ovadia et al., 2019). We leave further improvements in covariate-shifts to future work. Table 3: Performance on out-of-distribution detection. The best value in each column among the subset of models is bolded. The Top Ranked∗Ranked^* row represents the best model per dataset based on average rank across metrics as determined from the full results in Tab. 16: COG-MS on SVHN, and COG-TST on CIFAR-10 and CIFAR-100. SVHN CIFAR-10 CIFAR-100 Method AUROC (↑)( ) FPR95 (↓)( ) AUROC (↑)( ) FPR95 (↓)( ) AUROC (↑)( ) FPR95 (↓)( ) Baseline 94.780.2494.78_0.24 34.431.2734.43_1.27 83.972.7083.97_2.70 76.393.2076.39_3.20 71.921.3371.92_1.33 87.671.3487.67_1.34 COG-FC 96.600.4196.60_0.41 21.602.5921.60_2.59 87.001.3287.00_1.32 73.984.0373.98_4.03 68.502.2268.50_2.22 85.831.4385.83_1.43 COG+LS 93.800.6193.80_0.61 22.582.4522.58_2.45 81.933.1081.93_3.10 73.433.1873.43_3.18 65.487.8565.48_7.85 83.713.4283.71_3.42 Label Smoothing 80.581.6680.58_1.66 42.842.4042.84_2.40 72.176.8872.17_6.88 78.883.0978.88_3.09 74.352.9874.35_2.98 84.061.8684.06_1.86 Temp. Scaling 95.330.2595.33_0.25 31.141.8531.14_1.85 85.092.4385.09_2.43 75.044.2275.04_4.22 73.231.5373.23_1.53 85.462.0085.46_2.00 V-TST 94.660.4794.66_0.47 35.352.4035.35_2.40 79.232.2779.23_2.27 78.611.9678.61_1.96 70.750.8570.75_0.85 85.780.6485.78_0.64 Top Ranked∗Ranked^* 96.700.2896.70_0.28 20.362.1820.36_2.18 87.410.7087.41_0.70 71.372.3471.37_2.34 73.922.7273.92_2.72 79.962.3179.96_2.31 Table 4: Performance under covariate shifts. The best value in each column among the subset of models is bolded. The Top Ranked∗Ranked^* row reports the best-performing model per dataset based on average rank across metrics as determined from the full results in Tab. 17: Matrix Scaling on SVHN and CIFAR-10, Label Smoothing on CIFAR-100. SVHN CIFAR-10 CIFAR-100 Method ACC (↑)( ) NLL (↓)( ) Brier (↓)( ) ACC (↑)( ) NLL (↓)( ) Brier (↓)( ) ACC (↑)( ) NLL (↓)( ) Brier (↓)( ) Baseline 47.011.0647.01_1.06 5.290.235.29_0.23 0.940.020.94_0.02 74.280.5974.28_0.59 2.030.112.03_0.11 0.450.010.45_0.01 45.851.3645.85_1.36 5.700.215.70_0.21 0.920.020.92_0.02 COG-FC 45.821.3245.82_1.32 2.430.122.43_0.12 0.800.020.80_0.02 68.800.6868.80_0.68 1.570.091.57_0.09 0.500.010.50_0.01 44.090.4244.09_0.42 2.880.042.88_0.04 0.770.010.77_0.01 COG+LS 46.050.5746.05_0.57 1.910.021.91_0.02 0.770.000.77_0.00 69.310.4269.31_0.42 1.110.021.11_0.02 0.460.010.46_0.01 44.920.6944.92_0.69 2.900.112.90_0.11 0.810.030.81_0.03 Label Smoothing 46.190.5946.19_0.59 2.190.042.19_0.04 0.860.010.86_0.01 74.220.4274.22_0.42 1.030.021.03_0.02 0.420.010.42_0.01 47.140.9147.14_0.91 2.540.102.54_0.10 0.710.030.71_0.03 Temp. Scaling 47.011.0647.01_1.06 3.050.123.05_0.12 0.850.020.85_0.02 74.280.5974.28_0.59 1.190.051.19_0.05 0.410.010.41_0.01 45.851.3645.85_1.36 3.320.103.32_0.10 0.810.020.81_0.02 V-TST 41.660.2641.66_0.26 2.060.072.06_0.07 0.770.010.77_0.01 73.910.2273.91_0.22 0.910.030.91_0.03 0.380.010.38_0.01 38.440.2438.44_0.24 2.660.022.66_0.02 0.770.000.77_0.00 Top Ranked∗Ranked^* 47.270.7247.27_0.72 2.140.022.14_0.02 0.750.010.75_0.01 74.350.6674.35_0.66 0.840.030.84_0.03 0.360.010.36_0.01 47.140.9147.14_0.91 2.540.102.54_0.10 0.710.030.71_0.03 Table 5: Performance with a ViT backbone on CIFAR-10. Bold indicates the best value per metric Method ACC (↑)( ) NLL (↓)( ) Brier (↓)( ) Baseline 73.510.5773.51_0.57 0.810.020.81_0.02 0.380.010.38_0.01 COG-FC 74.970.4774.97_0.47 0.760.020.76_0.02 0.360.010.36_0.01 Temp. Scaling 73.510.5773.51_0.57 0.760.010.76_0.01 0.370.010.37_0.01 COG-TS 73.730.4373.73_0.43 0.810.020.81_0.02 0.380.000.38_0.00 Matrix Scaling 74.040.3574.04_0.35 0.740.010.74_0.01 0.360.000.36_0.00 COG-MS 73.140.7173.14_0.71 0.810.020.81_0.02 0.380.010.38_0.01 ViT ablation. Our experiments so far focus on CNN architectures. While App. D confirms consistent results with a WRN-28-10 backbone (Zagoruyko & Komodakis, 2016), we further assess COG across architectural families using Vision Transformers (ViTs) (Dosovitskiy et al., 2021). Applying a subset of post-hoc methods and their corresponding COG variants, Tab. 5, shows that COG-FC again achieves the best overall performance, suggesting that BLO is not specific to convolutional architectures. Additional calibration metrics are reported in App. C.4. 4.3 Use case: Multi-Task Learning Prerequisite Questions. Calibration in Multi-Task Learning remains relatively under-explored, raising two fundamental questions: how should calibration be applied in MTL, and how should calibration be quantified across heterogeneous tasks? For the former, should each task be calibrated independently, requiring separate K post-hoc calibrators or regularizers, or should a single approach be applied across tasks? This is particularly challenging when tasks differ in type, e.g., binary classification, multiclass classification, and regression, which may require different calibration methods. Even tasks of the same type may have different calibration requirements, further increasing the complexity of the calibration pipeline. In contrast, COG can be applied directly to MTL models using the same training objective as the network, avoiding separate calibration procedures for each task. The second question concerns how to quantify calibration across tasks. Given the aforementioned issues, rather than introducing MTL-specific calibration metrics, we evaluate using the previous metrics. Experimental Setup. For this experiment, we use CelebA (Liu et al., 2015), where the model predicts 40 facial attributes, i.e., 40 binary tasks. For non-COG methods each task is calibrated separately, resulting in 40 separate calibrators or regularization terms, as seen in Fig. 4.3. We train using linear scalarization (Miettinen et al., 1999), with equal task weights, i.e., Ltotal=∑i=1KLiL_total= _i=1^KL_i and, for every metric, report unweighted averages across tasks, e.g., Ltotal/KL_total/K. Results. The results in Tab. 6 show broadly comparable performance across methods, particularly in Brier score. All COG variants outperform their corresponding baselines with COG-FC achieving the best overall performance. Both Label Smoothing methods, however, exhibit notably worse NLL. Overall, these results demonstrate that effective end-to-end calibration is achievable in MTL without additional trainable parameters or regularization terms. In App. C.5 we provide evaluation based on calibration errors and per-task metrics. According to Fig. 6, all families exhibit a similar ordering of task difficulty, with Big Lips, for instance, consistently ranking among the most challenging tasks, suggesting that difficulty is primarily task-dependent. In pairwise comparisons, for example, COG-FC achieves both higher Accuracy and lower Brier score than Temperature Scaling on 32 of 40 tasks (80%), and than Label Smoothing on all 40 tasks (100%). Figure 4: MTL pipeline with K task-specific heads, each followed by a separate post-hoc calibrator or calibration objective. Method ACC (↑)( ) NLL (↓)( ) Brier (↓)( ) Baseline 89.880.0489.88_0.04 0.230.000.23_0.00 0.160.000.16_0.00 COG-FC 90.810.0490.81_0.04 0.210.000.21_0.00 0.140.000.14_0.00 Label Smoothing 89.910.0689.91_0.06 0.350.000.35_0.00 0.150.000.15_0.00 COG+LS 90.820.0290.82_0.02 0.340.000.34_0.00 0.140.000.14_0.00 Temp. Scaling 89.880.0489.88_0.04 0.240.000.24_0.00 0.150.000.15_0.00 COG-TS 90.440.1590.44_0.15 0.220.010.22_0.01 0.150.000.15_0.00 Table 6: MTL calibration performance on CelebA, averaged across tasks. Bold indicates the best value per metric. 5 Conclusion In this work, we simplify neural network calibration by reframing post-hoc calibration as a bilevel optimization problem and solving it through Stackelberg-inspired joint optimization. Our framework, COG, converts post-hoc calibrators into train-time, end-to-end alternatives that require no held-out calibration data and introduce only minimal additional complexity: a single additional learning rate. Across a broad range of experiments, architectures, and calibration approaches, COG achieves strong and consistent performance, matching or outperforming existing methods in most settings. In particular, COG-FC achieves state-of-the-art calibration performance while introducing no additional trainable parameters. We further find that COG is stable to optimize and performs well for OOD detection, demonstrating its flexibility across architectures, post-hoc methods, and regularization-based approaches. We further study calibration in MTL settings, where task heterogeneity requires conventional calibration methods to be selected and applied separately for each task. In contrast, COG works out of the box for MTL, providing a unified calibration approach with strong overall and task-specific performance without additional trainable parameters or separate calibration procedures. Overall, we recommend COG-FC as a simple, stable, and lightweight default for calibration, given its strong performance across diverse settings and lack of additional trainable parameters. Future Work and Limitations. Several directions remain for future work. First, our evaluation could be extended to alternative definitions of calibration and emerging calibration metrics, particularly those addressing limitations of standard metrics such as ECE. Second, we do not explicitly explore optimistic and pessimistic bilevel optimization (Dempe, 2002; Beck et al., 2027). In optimistic BLO, the leader assumes that, among multiple optimal solutions of the follower, the follower selects the one that is most favorable to the leader. In contrast, pessimistic BLO considers the worst-case selection among the follower’s optimal solutions, accounting for the least favorable outcome for the leader’s objective (Dempe, 2002; Beck et al., 2027). Pessimistic bilevel optimization, which is closely related to robust optimization (Beck et al., 2022; Beck et al., 2023), could be particularly relevant for calibration under adversarial settings or covariate shifts, while optimistic bilevel optimization may yield even better solutions for the calibrator. Further theoretical analysis of calibration from a bilevel optimization perspective could also provide additional insight. For MTL, future work could consider more challenging and heterogeneous settings, including tasks with different output spaces or objectives, alongside state-of-the-art MTL training methods and designing calibration metrics specifically for multi-task problems. Finally, although our experiments focus on computer vision, COG is not inherently tied to a particular architecture or modality and could be extended to other differentiable models and domains. AI use statement In this work, we used generative AI tools for research execution, specifically for code assistance, including but not limited to writing comments in code and preparing documentation. Additionally, we used generative AI tools to polish and improve the readability of the text. We did not use generative AI tools for the other required-disclosure tasks. We have reviewed all AI-assisted work, verified the correctness of the code and documentation, and reviewed the polished text to ensure that it accurately reflects the authors’ intended content. We take responsibility for the final content of this work, including text, claims, code, and other artifacts produced with the aid of generative AI. Reproducibility statement Implementation and training details, including optimization procedures, learning-rate schedules, checkpoint selection, calibration procedures, and experimental settings, are provided in App. B. The hyperparameter configurations used for the different datasets, backbones, and COG variants are reported in Tab. 8. Dataset splits, preprocessing, augmentation, and validation procedures are described in the dataset details in App. B. Additional information on the experimental hardware and computational requirements is also provided in App. B. All experiments use fixed random seeds as described in the appendix, and the code and configuration files required to reproduce the reported results are provided in the supplementary materials. Acknowledgments We would like to thank every member of the Probabilistic Machine Learning lab and especially Elina Rojin Celik for their invaluable feedback. This work was funded in part by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – GRK 2853/1 “Neuroexplicit Models of Language, Vision, and Action” - project number 471607914. Pablo M. Olmos has been supported by the Ministerio de Ciencia, Innovación y Universidades, Spain, under Grant PID2024-157856NB-I00 (CARTESIAN), funded by MCIN/AEI/10.13039/501100011033 and, as appropriate, by “ERDF A way of making Europe”; by the ELLIS Unit Madrid (European Laboratory for Learning and Intelligent Systems); and by the Comunidad de Madrid under grant IND2024/TIC-34728 and the IDEA-CM project (TEC-2024/COM-89). Views and opinions expressed are, however, those of the author(s) only and do not necessarily reflect those of the aforementioned funding agencies. None of the aforementioned parties can be held responsible for them. References Ashukha et al. (2020) Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=BJxI5gHKDr. Başar & Olsder (1998) Tamer Başar and Geert Jan Olsder. Dynamic noncooperative game theory. SIAM, 1998. Beck et al. (2022) Yasmine Beck, Ivana Ljubić, and Martin Schmidt. A brief introduction to robust bilevel optimization. arXiv preprint arXiv:2211.16072, 2022. Beck et al. (2023) Yasmine Beck, Ivana Ljubić, and Martin Schmidt. A survey on bilevel optimization under uncertainty. European Journal of Operational Research, 311(2):401–426, 2023. Beck et al. (2027) Yasmine Beck, Ivana Ljubić, and Martin Schmidt. Linear and Mixed-Integer Bilevel Optimization: Theory and Algorithms. Cambridge University Press, 2027. Berta et al. (2025) Eugène Berta, David Holzmüller, Michael I Jordan, and Francis Bach. Rethinking early stopping: Refine, then calibrate. arXiv preprint arXiv:2501.19195, 2025. Berta et al. (2026a) Eugène Berta, David Holzmüller, Francis Bach, and Michael I Jordan. Calarena: A large-scale post-hoc calibration benchmark. arXiv preprint arXiv:2605.30188, 2026a. Berta et al. (2026b) Eugène Berta, David Holzmüller, Michael I. Jordan, and Francis Bach. Structured matrix scaling for multi-class calibration. In Emtiyaz Khan, Yingzhen Li, Arno Solin, and Aaditya Ramdas (eds.), Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, volume 300 of Proceedings of Machine Learning Research, p. 1342–1350. PMLR, 02–05 May 2026b. URL https://proceedings.mlr.press/v300/berta26a.html. Bohdal et al. (2021) Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Meta-calibration: Learning of model calibration using differentiable expected calibration error. arXiv preprint arXiv:2106.09613, 2021. Bradley (1997) Andrew P Bradley. The use of the area under the roc curve in the evaluation of machine learning algorithms. Pattern recognition, 30(7):1145–1159, 1997. Breiman (2001) Leo Breiman. Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical science, 16(3):199–231, 2001. Brito et al. (2023) Mario P Brito, Matthew Stevenson, and Cristián Bravo. Subjective machines: Probabilistic risk assessment based on deep learning of soft information. Risk Analysis, 43(3):516–529, 2023. Caruana (1997) Rich Caruana. Multitask learning. Machine learning, 28(1):41–75, 1997. Chidambaram & Ge (2025) Muthu Chidambaram and Rong Ge. Reassessing how to compare and improve the calibration of machine learning models. In International Conference on Learning Representations, volume 2025, p. 61542–61570, 2025. Dempe (2002) Stephan Dempe. Foundations of bilevel programming. Springer, 2002. Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=YicbFdNTTy. Fiez et al. (2020) Tanner Fiez, Benjamin Chasnov, and Lillian Ratliff. Implicit learning dynamics in stackelberg games: Equilibria characterization, convergence analysis, and empirical study. In Hal Daumé I and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, p. 3133–3144. PMLR, 13–18 Jul 2020. URL https://proceedings.mlr.press/v119/fiez20a.html. Franceschi et al. (2018) Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, p. 1568–1577. PMLR, 10–15 Jul 2018. URL https://proceedings.mlr.press/v80/franceschi18a.html. Glenn et al. (1950) W Brier Glenn et al. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1–3, 1950. Goodfellow et al. (2016) Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http://w.deeplearningbook.org. Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, p. 1321–1330. PMLR, 2017. Gupta & Verma (2023) Kavya Gupta and Sagar Verma. Shrink & cert: Bi-level optimization for certified robustness. In New Frontiers in Adversarial Machine Learning, ICML, 2023. Gupta et al. (2026) Kavya Gupta, Nektarios Kalampalikis, Christoph Heitz, and Isabel Valera. First-see-then-design: A multi-stakeholder view for optimal performance-fairness trade-offs. In Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’26, p. 218–248, New York, NY, USA, 2026. Association for Computing Machinery. ISBN 9798400725968. doi: 10.1145/3805689.3806541. URL https://doi.org/10.1145/3805689.3806541. He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 770–778, 2016. He et al. (2026) Wenchong He, Zhe Jiang, Tingsong Xiao, Zelin Xu, and Yukun Li. A survey on uncertainty quantification methods for deep learning. ACM Computing Surveys, 58(7):1–35, 2026. Hendrycks & Dietterich (2019) Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261, 2019. Hendrycks et al. (2018) Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arXiv:1812.04606, 2018. Javaloy & Valera (2022) Adrián Javaloy and Isabel Valera. Rotograd: Gradient homogenization in multitask learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=T8wHz4rnuGL. Javaloy et al. (2025) Adrián Javaloy, Antonio Vergari, and Isabel Valera. Copa: Comparing the incomparable in multi-objective model evaluation. arXiv preprint arXiv:2503.14321, 2025. Jenni & Favaro (2018) Simon Jenni and Paolo Favaro. Deep bilevel learning. In Proceedings of the European conference on computer vision (ECCV), p. 618–633, 2018. Joo & Chung (2020) Taejong Joo and Uijung Chung. Revisiting explicit regularization in neural networks for well-calibrated predictive uncertainty. arXiv preprint arXiv:2006.06399, 2020. Jordahn & Olmos (2024) Mikkel Jordahn and Pablo M. Olmos. Decoupling feature extraction and classification layers for calibrated neural networks. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, p. 22530–22550. PMLR, 21–27 Jul 2024. URL https://proceedings.mlr.press/v235/jordahn24a.html. Karandikar et al. (2021) Archit Karandikar, Nicholas Cain, Dustin Tran, Balaji Lakshminarayanan, Jonathon Shlens, Michael C Mozer, and Becca Roelofs. Soft calibration objectives for neural networks. Advances in Neural Information Processing Systems, 34:29768–29779, 2021. Khodaverdian et al. (2025) Johanna D’Ciofalo Khodaverdian, Eric Banzuzi, and Katharina Deckenbach. A reproducibility study of decoupling feature extraction and classification layers for calibrated neural networks. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https://openreview.net/forum?id=5Hwzd48ILf. Kingma & Ba (2014) Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. Kwon et al. (2023) Jeongyeol Kwon, Dohyun Kwon, Stephen Wright, and Robert D Nowak. A fully first-order method for stochastic bilevel optimization. In International Conference on Machine Learning, p. 18083–18113. PMLR, 2023. Lin et al. (2020) Tianyi Lin, Chi Jin, and Michael Jordan. On gradient descent ascent for nonconvex-concave minimax problems. In International conference on machine learning, p. 6083–6093. PMLR, 2020. Liu et al. (2019) Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=S1eYHoC5FX. Liu et al. (2021) Risheng Liu, Jiaxin Gao, Jin Zhang, Deyu Meng, and Zhouchen Lin. Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12):10045–10067, 2021. Liu et al. (2015) Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015. Meister et al. (2020) Clara Meister, Elizabeth Salesky, and Ryan Cotterell. Generalized entropy regularization or: There’s nothing special about label smoothing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 6870–6886, 2020. Menon et al. (2012) Aditya Krishna Menon, Xiaoqian J Jiang, Shankar Vembu, Charles Elkan, and Lucila Ohno-Machado. Predicting accurate probabilities with a ranking loss. In Proceedings of the International Conference on Machine Learning. International Conference on Machine Learning, volume 2012, p. 703, 2012. Miettinen et al. (1999) Kaisa Miettinen et al. Nonlinear multiobjective optimization, volume 12. Kluwer Academic Publishers Boston, 1999. Minderer et al. (2021) Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. Advances in neural information processing systems, 34:15682–15694, 2021. Mukhoti et al. (2020) Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss. Advances in neural information processing systems, 33:15288–15299, 2020. Müller et al. (2019) Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019. Murphy (2023) Kevin P. Murphy. Probabilistic Machine Learning: Advanced Topics. MIT Press, 2023. URL http://probml.github.io/book2. Naeini et al. (2015) Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015. Netzer et al. (2011) Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, p. 4. Granada, 2011. Ovadia et al. (2019) Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019. Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, p. 8024–8035. Curran Associates, Inc., 2019. Platt (2000) John Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Adv. Large Margin Classif., 10, 06 2000. Popordanoska et al. (2022) Teodora Popordanoska, Raphael Sayer, and Matthew Blaschko. A consistent and differentiable lp canonical calibration error estimator. Advances in Neural Information Processing Systems, 35:7933–7946, 2022. Ricci Lara et al. (2023) María Agustina Ricci Lara, Candelaria Mosquera, Enzo Ferrante, and Rodrigo Echeveste. Towards unraveling calibration biases in medical image analysis. In Workshop on Clinical Image-Based Procedures, p. 132–141. Springer, 2023. Rudin et al. (2024) Cynthia Rudin, Chudi Zhong, Lesia Semenova, Margo Seltzer, Ronald Parr, Jiachang Liu, Srikar Katta, Jon Donnelly, Harry Chen, and Zachery Boner. Amazing things come from having many good models. arXiv preprint arXiv:2407.04846, 2024. Sanguin et al. (2025) Gabriele Sanguin, Arjun Pakrashi, Marco Viola, and Francesco Rinaldi. Exploring the potential of bilevel optimization for calibrating neural networks. arXiv preprint arXiv:2503.13113, 2025. Sharifdeen et al. (2026) Ashshak Sharifdeen, Shihab Aaqil Ahamed, Ufaq Khan, Muhammad Akhtar Munir Sujair Ibrahim, Mohamed Rafeek Mareer Ahamed, Yutong Xie, Imran Razzak, and Muhammad Haris Khan. Mvc-bench: Benchmarking calibration of medical vision-language models. arXiv preprint arXiv:2608.27004, 2026. Silva Filho et al. (2023) Telmo Silva Filho, Hao Song, Miquel Perello-Nieto, Raul Santos-Rodriguez, Meelis Kull, and Peter Flach. Classifier calibration: a survey on how to assess and improve predicted class probabilities. Machine Learning, 112(9):3211–3260, 2023. Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 2818–2826, 2016. Tao et al. (2024) Linwei Tao, Younan Zhu, Haolan Guo, Minjing Dong, and Chang Xu. A benchmark study on calibration. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=GzNhzX9kVa. Tomsett et al. (2020) Richard Tomsett, Alun Preece, Dave Braines, Federico Cerutti, Supriyo Chakraborty, Mani Srivastava, Gavin Pearson, and Lance Kaplan. Rapid trust calibration through interpretable and uncertainty-aware ai. Patterns, 1(4):100049, 2020. ISSN 2666-3899. doi: https://doi.org/10.1016/j.patter.2020.100049. URL https://w.sciencedirect.com/science/article/pii/S266638992030060X. van Krieken et al. (2026) Emile van Krieken, Pasquale Minervini, Edoardo Maria Ponti, and Antonio Vergari. Neurosymbolic diffusion models. Advances in neural information processing systems, 38:135889–135931, 2026. Vandenhirtz et al. (2024) Moritz Vandenhirtz, Sonia Laguna, Ričards Marcinkevičs, and Julia E Vogt. Stochastic concept bottleneck models. Advances in Neural Information Processing Systems, 37:51787–51810, 2024. Von Stackelberg (2010) Heinrich Von Stackelberg. Market structure and equilibrium. Springer Science & Business Media, 2010. Wang (2023) Cheng Wang. Calibration in deep learning: A survey of the state-of-the-art. arXiv preprint arXiv:2308.01222, 2023. Wang & Zhang (2024) Deng-Bao Wang and Min-Ling Zhang. Calibration bottleneck: Over-compressed representations are less calibratable. In Forty-first International Conference on Machine Learning, 2024. Zadrozny & Elkan (2001) Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Icml, volume 1, p. 2001, 2001. Zadrozny & Elkan (2002) Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, p. 694–699, 2002. Zagoruyko & Komodakis (2016) Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016. Zhang et al. (2024) Yihua Zhang, Prashant Khanduri, Ioannis Tsaknakis, Yuguang Yao, Mingyi Hong, and Sijia Liu. An introduction to bilevel optimization: Foundations and applications in signal processing and machine learning. IEEE Signal Processing Magazine, 41(1):38–59, 2024. Appendix A Stackelberg and Gradient-Descent Ascent In this section, we provide more details about topics mentioned Section 3. We strongly encourage readers to refer to the original works (Fiez et al., 2020; Lin et al., 2020; Beck et al., 2027). Bilevel Optimization Applications. BLO has been applied to a wide range of tasks in machine learning (Zhang et al., 2024), including hyperparameter optimization (Franceschi et al., 2018), multi-task learning (Javaloy & Valera, 2022), neural architecture search (Liu et al., 2019; Tao et al., 2024), robustness (Gupta & Verma, 2023), generalization (Jenni & Favaro, 2018), and more recently calibration (Bohdal et al., 2021; Sanguin et al., 2025). Gradient-based approaches to solve the BLO problem usually require calculating the Jacobian of the best-response. Thus, most methods fall back to approximating it via: T gradient updates of the dynamical system, using implicit-differentiation and calculating the Hessian. For more details and methods on gradient-based BLO we direct the reader to (Liu et al., 2021). A.1 Definition of Equilibria Following the main text, the formulation of a Stackelberg game (or bilevel optimization problem) is: minθcal∈,θtrain∈S⁡(θcal)⁡ℱcal(θtrain,θcal) _ _cal∈Y, _train∈ S( _cal)F_cal( _train, _cal) (Leader) (Leader) (3) where S(θcal)=argminθtrain∈⁡ℱtrain(θtrain,θcal) S( _cal)= _ _train∈XF_train( _train, _cal) (Follower) (Follower) S⁡(θcal)S( _cal) is the solution set of the follower’s problem, accounting for the possibility of multiple optimal solutions. It depends on the leader’s variables θcal _cal, since the leader’s choice determines the follower’s optimization problem and, consequently, its optimal solution set. An equilibrium can be viewed as a joint solution satisfying the relevant optimality conditions of all players. A Nash equilibrium is a joint strategy profile in which each player’s strategy is an optimal response to the others, while a Stackelberg equilibrium is the solution of the corresponding bilevel problem, with the leader accounting for the follower’s optimal response. Following (Fiez et al., 2020), a solution θ∗=(θcal∗,θtrain∗)θ^*=( _cal^*, _train^*) is considered a differential Nash equilibrium (NSE) when for both players j: ∂ℱj(θ∗)∂θj=0and∂2ℱj(θ∗)∂θj2>0 _j(θ^*)∂ _j=0 ∂^2F_j(θ^*)∂ _j^2>0 (4) Note that for DNE we require only the diagonal blocks of each player’s Hessian to be positive definite, i.e., each player has a local optimum along their own coordinate axis, with no consideration of cross-player effects. Meanwhile a solution is considered a differential Stackelberg equilibrium (DSE) if for the leader: dℱcal(θ∗)=0andd2ℱcal(θ∗)>0dF_cal(θ^*)=0 d^2F_cal(θ^*)>0 (5) and for the follower: ∂ℱtrain(θ∗)∂θtrain∗=0and∂2ℱtrain(θ∗)∂θtrain∗2>0 _train(θ^*)∂ _train^*=0 ∂^2F_train(θ^*)∂ _train^*^2>0 (6) Note that in DSE the leader’s total derivative is used, capturing the fact that the leader operates under the assumption that the follower plays a (local) best response to θcal _cal, due to the hierarchical nature of the game. This is the key difference between the two equilibrium concepts. Specifically, the total derivative accounts for the implicit dependence of the follower’s response: dℱcal(θ)=∂ℱcal(θ)∂θcal+∂ℱcal(θ)∂θtrain∂r⁡(θcal)∂θcaldF_cal(θ)= _cal(θ)∂ _cal+ _cal(θ)∂ _train ∂ r( _cal)∂ _cal (7) where r⁡(θcal)r( _cal) is the implicit best-response function of the follower, via the implicit function theorem. Subset relationship of DSE and DNE. In zero-sum games, every Nash equilibrium is also a Stackelberg equilibrium, but the converse does not hold (Başar & Olsder, 1998; Fiez et al., 2020). In the continuous setting, Fiez et al. (2020) show that this inclusion follows from a second-order relationship revealed by the Schur complement of the Stackelberg Jacobian. In particular, the second-order conditions for a DNE (∂2ℱj(θ∗)∂θj2>0)( ∂^2F_j(θ^*)∂ _j^2>0) imply those for a DSE (d2ℱcal(θ∗)dθj2>0)( d^2F_cal(θ^*)d _j^2>0) via the Schur complement. Implications for COG Gradient Descent–Ascent iteratively and jointly updates the strategies of both players via gradient descent. GDA is designed to target Nash equilibria and can therefore yield valid but restrictive solutions for Stackelberg games. Moreover, GDA does not reliably converge to Nash equilibria due to its characteristic rotational dynamics and may instead converge to spurious stable points, making it theoretically less principled than bilevel optimization (BLO) methods (Fiez et al., 2020). Nevertheless, GDA remains a practical approach for approximating Stackelberg problems because of its simplicity and computational efficiency, as demonstrated in prior work (Javaloy & Valera, 2022) and in our experiments. Table 7: Complexity of different calibration methods in Single-Task and Multi-Task learning settings, where K denotes the number of classes. # Extra End Calibration Extra Method Trainable Params. To End Data Burden Single-Task Baseline 0 ✓ ✗ - COG-FC 0 ✓ ✗ 1 Learning Rate Label Smoothing 0 ✓ ✗ 1 Smoothing Level COG+LS 0 ✓ ✗ (a) 1 Smoothing Level (b) 1 Learning Rate Temp. Scaling 1 ✗ ✓ Calibrator Training COG-TS 1 ✓ ✗ 1 Learning Rate Matrix Scaling K×K+K× K+K ✓ ✓ Calibrator Training COG-MS K×K+K× K+K ✓ ✗ 1 Learning Rate TST N ✗ ✗ N Training, Architecture COG-TST N ✓ ✗ 1 Learning Rate V-TST 2 NNs ✗ ✗ (a) ELBO training of 2 NNs (b) N architecture (c) MC Sampling Multi-Task Baseline 0 ✓ ✗ - COG-FC 0 ✓ ✗ 1 Learning Rate Label Smoothing 0 ✓ ✗ K Smoothing Levels COG+LS 0 ✓ ✗ (a) K Smoothing Levels (b) 1 Learning Rate Temp. Scaling K ✗ ✓ K Models Trained COG-TS K ✓ ✗ 1 Learning Rate Appendix B Implementation Details B.1 Training Details For the backbone considered in our experiments, ResNet-101 (He et al., 2016), we trained the models for 300 epochs in the single-task setting and 100 epochs in the multitask setting. All experiments used the Adam optimizer (Kingma & Ba, 2014) with a batch size of 256. Unless otherwise stated, we used five random seeds for every experiment reported in the main paper. All experiments used PyTorch (Paszke et al., 2019) version 2.11. For the COG experiments, we used a step learning-rate scheduler (as recommended in (Fiez et al., 2020; Kwon et al., 2023)) that reduced the learning rate by a factor of 10 every 10 epochs for both tasks. For CelebA, the learning rate was reduced by a factor of 10 every 3 epochs. For both single and multi task settings we trained with cross-entropy loss. Specifically, for the MTL setting, we used equal weights across all tasks. All post-hoc calibrators and evaluation metrics are implemented using the Probmetrics library (version 1.2.0) (Berta et al., 2025), except for Temperature Scaling, for which we use the original implementation (Guo et al., 2017). All post-hoc calibrators were fitted on the validation set. During training, for all models considered in the experiments, we selected the best model checkpoint based on the validation Brier score, as this criterion leads to better-calibrated models (Berta et al., 2025). For label smoothing, in all cases, we used a smoothing value of 0.10.1. ViT Experiments. For the ViT ablation we use the ViT architecture seen in (Jordahn & Olmos, 2024). We train for a maximum of 400 epochs, with a early stopping patience of 50 validation checks. The learning rate for the calibrator (λc _c) is set to 1e−51e-5 and for the backbone’s learning rate (λf _f) is set to 1e−41e-4. MTL Experiments. For the MTL experiment, we trained with a maximum of 100 epochs, and again we picked the best model based on Brier score. For the baselines we used a learning rate of 1e−41e-4. B.2 Model Selection We performed hyperparameter tuning primarily over the learning rates for the two problems in COG. Hyperparameters were selected based on validation CE loss. Unless otherwise stated, all subsequent experiments used the best configurations identified during this tuning procedure (seen in Table 8). For example, the COG-Label Smoothing experiment on CIFAR-100 with ResNet-101 used the corresponding configuration selected during this procedure. For the baseline models, we used a learning rate of 10−410^-4. For TST, we used the default hyperparameter settings reported in (Jordahn & Olmos, 2024). Table 8: Hyperparameter configurations selected based on validation performance for the different COG variants. λf _f and λc _c denote the learning rates used for model training and calibration, respectively. A dash indicates that the corresponding component was not used for that method. CIFAR-10 CIFAR-100 SVHN CelebA Method λf _f λc _c λf _f λc _c λf _f λc _c λf _f λc _c ResNet COG-FC 1e-3 5e-5 1e-3 1e-5 1e-3 1e-5 1e-3 1e-5 COG-TS 1e-3 1e-4 1e-3 1e-5 1e-3 1e-4 1e-2 5e-5 COG-MS 1e-3 5e-5 5e-4 1e-5 5e-4 1e-5 - - COG-TST 1e-3 5e-5 5e-4 1e-5 5e-4 1e-4 - - WRN COG-FC 1e-3 1e-5 1e-2 1e-5 1e-3 5e-5 - - COG-TS 1e-3 1e-5 1e-2 1e-5 1e-3 1e-4 - - COG-MS 1e-2 5e-5 1e-2 1e-3 5e-4 1e-4 - - COG-TST 1e-2 1e-3 1e-3 1e-5 1e-3 5e-4 - - B.3 Dataset Details For CIFAR-10 (Krizhevsky et al., 2009), CIFAR-100 (Krizhevsky et al., 2009), and SVHN (Netzer et al., 2011), the original training sets were randomly partitioned into training and validation subsets using a fixed random seed. The validation fractions were 20%, 5%, and 20% for CIFAR-10, CIFAR-100, and SVHN, respectively. Training images were augmented using a random 32×3232× 32 crop with 4-pixel padding, followed by random horizontal flipping. Validation and test images were evaluated without stochastic data augmentation. For CelebA (Liu et al., 2015), we used the predefined training, validation, and test splits. Images were resized and center-cropped to 32×3232× 32, with random horizontal flipping applied only during training. All datasets were normalized using dataset-specific channel-wise means and standard deviations. The held-out validation set was used both for model checkpoint selection and for fitting the post-hoc calibration methods. In the multitask setting, all 40 tasks were used. B.4 Hardware Requirements All experiments were conducted on a high-performance computing cluster with automatic job scheduling, ensuring efficient resource allocation. We list the hardware requirements we used. For all datasets and backbones we utilized 6 CPU cores, 16GB of RAM, and a GPU with at least 8GB of VRAM. Appendix C Extended Results In this section, we provide extended versions of the experiments presented in the main text. For all experiments, we additionally report Expected Calibration Error (ECE) (Naeini et al., 2015), Root Mean Square Calibration Error (RMSCE) (Hendrycks et al., 2018) and maximum calibration error (MCE) (Naeini et al., 2015). These calibration metrics are reported as percentages. In cases where NLL is not reported in the main text, we additionally include it here to provide a more comprehensive evaluation. Standard calibration metrics such as ECE have several well-documented limitations, particularly in the multiclass setting (Minderer et al., 2021; Chidambaram & Ge, 2025). The limitations discussed below represent only a subset of those identified in the literature. First, ECE is a biased estimator, whose magnitude of the positive bias increases with model accuracy (Minderer et al., 2021). Furthermore, its reported value can vary substantially with the choice of the number of bins, making comparisons between models sensitive on the binning scheme (Minderer et al., 2021; Silva Filho et al., 2023; Tao et al., 2024). Moreover, the relationship between ECE and Accuracy appears to hold primarily for highly performant models (Tao et al., 2024) and may depend on the architecture considered (Sharifdeen et al., 2026). Comparisons between in-hoc and post-hoc calibration methods can also be problematic when relying solely on standard calibration metrics (Ashukha et al., 2020; Tao et al., 2024). Lastly, ECE evaluates calibration only with respect to the top-1 predicted class, and recently it has been shown that even a trivial predictor that always outputs the mean confidence can outperform post-hoc calibration methods according to standard calibration metrics (Chidambaram & Ge, 2025). C.1 Comparison of COG and Post-hoc approaches Here we provide a more detailed version of the results found in Section 4.1. Figure 5: Reliability Diagrams for all COG variants and their corresponding calibration methods using 15 bins. Following standard practice (Minderer et al., 2021; Silva Filho et al., 2023; Tao et al., 2024), we report calibration errors under different norms and across multiple choices of the number of bins. This provides a more comprehensive evaluation and reduces the dependence of our conclusions on any particular choice of calibration metric or binning scheme. In Tab. 9, we notice that, in almost all pairwise comparisons, the COG variants outperform the corresponding post-hoc methods in terms of ECE. A notable exception is Matrix Scaling, which, as also observed in Fig. 2, substantially worsens the performance of the Baseline on CIFAR-100. Among all COG variants, COG-TST performs the best in ECE overall. We observe the same qualitative trends for RMSCE, in Tab. 10, with COG generally outperforming the post-hoc calibration methods across these standard calibration metrics. Finally, we report MCE results in Tab. 11. For all calibration metrics, we notice that the relative ranking of methods can change with the number of bins, with the effect being more dramatic for MCE. Furthermore, the reported MCE values can increase dramatically as the number of bins increases. This provides a further illustration of the sensitivity of MCE to the binning scheme and highlights the importance of evaluating calibration across multiple metric configurations rather than relying on a single choice of bins. Table 9: Comparison of ECE across different COG variants and their corresponding post-hoc counterparts for varying numbers of bins. We report meanstd [Rank]mean_std [Rank]. SVHN CIFAR-10 CIFAR-100 Method ECE-10 ECE-15 ECE-100 ECE-500 ECE-10 ECE-15 ECE-100 ECE-500 ECE-10 ECE-15 ECE-100 ECE-500 Baseline 3.260.113.26_0.11 [8] 3.250.113.25_0.11 [8] 3.290.103.29_0.10 [8] 3.450.103.45_0.10 [8] 7.380.257.38_0.25 [8] 7.390.237.39_0.23 [8] 7.470.237.47_0.23 [8] 7.890.227.89_0.22 [8] 26.560.7926.56_0.79 [7] 26.560.7826.56_0.78 [7] 26.600.7926.60_0.79 [7] 26.760.8326.76_0.83 [7] COG-FC 1.140.301.14_0.30 [2] 1.160.291.16_0.29 [2] 1.320.171.32_0.17 [2] 1.860.071.86_0.07 [2] 5.320.165.32_0.16 [4] 5.320.145.32_0.14 [4] 5.490.115.49_0.11 [4] 6.220.126.22_0.12 [5] 8.000.358.00_0.35 [2] 8.000.368.00_0.36 [2] 8.130.378.13_0.37 [2] 9.200.289.20_0.28 [2] Temp Scaling 2.190.092.19_0.09 [7] 2.200.092.20_0.09 [7] 2.260.082.26_0.08 [7] 2.570.052.57_0.05 [7] 5.400.295.40_0.29 [5] 5.410.285.41_0.28 [5] 5.570.265.57_0.26 [5] 6.210.146.21_0.14 [4] 18.880.4118.88_0.41 [6] 18.880.4118.88_0.41 [6] 18.910.4218.91_0.42 [6] 19.260.4319.26_0.43 [6] COG-TS 1.230.141.23_0.14 [4] 1.240.141.24_0.14 [4] 1.410.111.41_0.11 [4] 1.950.111.95_0.11 [4] 5.520.405.52_0.40 [6] 5.520.405.52_0.40 [6] 5.660.405.66_0.40 [6] 6.400.406.40_0.40 [6] 11.142.0511.14_2.05 [3] 11.142.0511.14_2.05 [3] 11.212.0311.21_2.03 [3] 11.961.7711.96_1.77 [3] Matrix Scaling 0.710.160.71_0.16 [1] 0.730.150.73_0.15 [1] 1.050.071.05_0.07 [1] 1.840.091.84_0.09 [1] 0.730.120.73_0.12 [1] 0.890.100.89_0.10 [1] 1.760.071.76_0.07 [1] 3.740.223.74_0.22 [1] 50.181.7950.18_1.79 [8] 50.181.7950.18_1.79 [8] 50.191.7950.19_1.79 [8] 50.251.7950.25_1.79 [8] COG-MS 1.370.081.37_0.08 [5] 1.380.081.38_0.08 [5] 1.510.061.51_0.06 [5] 1.970.101.97_0.10 [5] 5.670.245.67_0.24 [7] 5.680.245.68_0.24 [7] 5.840.255.84_0.25 [7] 6.610.156.61_0.15 [7] 11.462.2411.46_2.24 [4] 11.462.2411.46_2.24 [4] 11.552.1911.55_2.19 [4] 12.431.8212.43_1.82 [4] TST 1.470.031.47_0.03 [6] 1.480.021.48_0.02 [6] 1.690.051.69_0.05 [6] 2.320.072.32_0.07 [6] 4.900.074.90_0.07 [2] 4.910.054.91_0.05 [2] 5.100.055.10_0.05 [2] 6.010.076.01_0.07 [2] 15.781.4615.78_1.46 [5] 15.801.4515.80_1.45 [5] 15.821.4115.82_1.41 [5] 16.221.4016.22_1.40 [5] COG-TST 1.210.111.21_0.11 [3] 1.210.111.21_0.11 [3] 1.390.091.39_0.09 [3] 1.910.071.91_0.07 [3] 4.960.324.96_0.32 [3] 4.960.324.96_0.32 [3] 5.140.305.14_0.30 [3] 6.130.176.13_0.17 [3] 4.040.554.04_0.55 [1] 4.070.574.07_0.57 [1] 4.700.504.70_0.50 [1] 7.180.357.18_0.35 [1] Table 10: Comparison of RMSCE across different COG variants and their corresponding post-hoc counterparts for varying numbers of bins. We report meanstd [Rank]mean_std [Rank]. SVHN CIFAR-10 CIFAR-100 Method RMSCE-10 RMSCE-15 RMSCE-100 RMSCE-500 RMSCE-10 RMSCE-15 RMSCE-100 RMSCE-500 RMSCE-10 RMSCE-15 RMSCE-100 RMSCE-500 Baseline 5.030.445.03_0.44 [8] 5.570.285.57_0.28 [8] 7.220.227.22_0.22 [8] 8.850.228.85_0.22 [8] 8.980.438.98_0.43 [8] 9.650.419.65_0.41 [8] 11.890.5411.89_0.54 [8] 14.580.4614.58_0.46 [8] 27.690.8627.69_0.86 [7] 28.270.8428.27_0.84 [7] 30.690.8330.69_0.83 [7] 32.470.8432.47_0.84 [7] COG-FC 1.960.661.96_0.66 [2] 2.160.672.16_0.67 [2] 3.170.333.17_0.33 [2] 5.590.175.59_0.17 [2] 6.980.366.98_0.36 [4] 7.270.297.27_0.29 [4] 9.080.369.08_0.36 [4] 11.870.3411.87_0.34 [4] 9.180.469.18_0.46 [2] 9.390.479.39_0.47 [2] 10.270.5110.27_0.51 [2] 12.980.4612.98_0.46 [2] Temp Scaling 3.620.273.62_0.27 [7] 3.890.303.89_0.30 [7] 4.930.304.93_0.30 [7] 6.720.296.72_0.29 [7] 7.030.407.03_0.40 [6] 7.580.417.58_0.41 [6] 9.460.469.46_0.46 [7] 12.180.3312.18_0.33 [6] 20.570.3920.57_0.39 [6] 21.070.4421.07_0.44 [6] 22.510.5522.51_0.55 [6] 24.140.5024.14_0.50 [6] COG-TS 2.030.352.03_0.35 [3] 2.230.392.23_0.39 [4] 3.330.253.33_0.25 [5] 5.830.355.83_0.35 [6] 6.990.516.99_0.51 [5] 7.490.567.49_0.56 [5] 9.200.659.20_0.65 [5] 11.970.7811.97_0.78 [5] 12.342.1912.34_2.19 [3] 12.562.2812.56_2.28 [3] 13.512.4213.51_2.42 [3] 15.852.0315.85_2.03 [3] Matrix Scaling 1.260.251.26_0.25 [1] 1.400.231.40_0.23 [1] 2.630.162.63_0.16 [1] 5.340.145.34_0.14 [1] 1.460.251.46_0.25 [1] 1.900.341.90_0.34 [1] 4.150.284.15_0.28 [1] 8.520.378.52_0.37 [1] 50.301.7950.30_1.79 [8] 50.381.7950.38_1.79 [8] 50.911.7350.91_1.73 [8] 51.631.6451.63_1.64 [8] COG-MS 2.310.142.31_0.14 [6] 2.530.212.53_0.21 [6] 3.560.133.56_0.13 [6] 5.800.195.80_0.19 [3] 7.170.497.17_0.49 [7] 7.610.607.61_0.60 [7] 9.330.539.33_0.53 [6] 12.190.4112.19_0.41 [7] 12.752.3512.75_2.35 [4] 13.062.4813.06_2.48 [4] 14.002.5214.00_2.52 [4] 16.352.2816.35_2.28 [4] TST 2.110.072.11_0.07 [5] 2.180.072.18_0.07 [3] 3.290.163.29_0.16 [3] 5.830.195.83_0.19 [5] 6.520.126.52_0.12 [2] 6.790.106.79_0.10 [2] 8.540.148.54_0.14 [3] 11.650.2111.65_0.21 [3] 17.071.5717.07_1.57 [5] 17.251.6317.25_1.63 [5] 17.901.6017.90_1.60 [5] 19.611.4419.61_1.44 [5] COG-TST 2.050.282.05_0.28 [4] 2.250.292.25_0.29 [5] 3.330.233.33_0.23 [4] 5.800.185.80_0.18 [4] 6.540.496.54_0.49 [3] 6.810.636.81_0.63 [3] 8.340.548.34_0.54 [2] 11.500.3411.50_0.34 [2] 4.590.684.59_0.68 [1] 4.700.634.70_0.63 [1] 6.220.746.22_0.74 [1] 10.200.5010.20_0.50 [1] Table 11: Comparison of MCE across different COG variants and their corresponding post-hoc counterparts for varying numbers of bins. We report meanstd [Rank]mean_std [Rank]. The relative ranking of methods changes across binning choices, while the MCE values increase substantially as the number of bins increases, highlighting the sensitivity of MCE to the binning scheme. SVHN CIFAR-10 CIFAR-100 Method MCE-10 MCE-15 MCE-100 MCE-500 MCE-10 MCE-15 MCE-100 MCE-500 MCE-10 MCE-15 MCE-100 MCE-500 Baseline 41.3118.9741.31_18.97 [8] 44.5716.4244.57_16.42 [8] 67.186.9067.18_6.90 [6] 86.852.5486.85_2.54 [8] 36.203.9936.20_3.99 [8] 38.152.7138.15_2.71 [7] 62.047.3262.04_7.32 [4] 89.101.4789.10_1.47 [8] 48.992.0548.99_2.05 [7] 52.191.2152.19_1.21 [7] 68.5310.0468.53_10.04 [7] 83.804.7983.80_4.79 [7] COG-FC 17.276.0017.27_6.00 [3] 19.036.8119.03_6.81 [3] 48.9619.7448.96_19.74 [2] 74.041.0474.04_1.04 [2] 24.653.8224.65_3.82 [7] 28.059.3428.05_9.34 [5] 64.0312.0764.03_12.07 [6] 74.842.7574.84_2.75 [3] 15.221.2815.22_1.28 [2] 15.741.5015.74_1.50 [2] 40.7927.3540.79_27.35 [3] 72.5519.6372.55_19.63 [2] Temp Scaling 20.245.1820.24_5.18 [4] 33.5925.2033.59_25.20 [6] 66.7916.0266.79_16.02 [5] 78.233.0678.23_3.06 [5] 24.283.6724.28_3.67 [6] 25.011.7025.01_1.70 [4] 59.1113.2559.11_13.25 [3] 78.524.1278.52_4.12 [7] 36.931.1536.93_1.15 [6] 37.081.1037.08_1.10 [6] 53.4817.1453.48_17.14 [6] 82.831.1482.83_1.14 [4] COG-TS 33.3628.1533.36_28.15 [7] 35.9626.4435.96_26.44 [7] 80.522.1880.52_2.18 [8] 80.671.9980.67_1.99 [7] 22.904.4922.90_4.49 [2] 23.012.6623.01_2.66 [1] 50.6317.0050.63_17.00 [1] 74.074.2374.07_4.23 [1] 19.633.4519.63_3.45 [3] 20.763.2820.76_3.28 [3] 42.7422.0542.74_22.05 [4] 83.171.5783.17_1.57 [6] Matrix Scaling 30.1329.4130.13_29.41 [6] 30.1629.3830.16_29.38 [5] 68.9521.5968.95_21.59 [7] 78.932.5578.93_2.55 [6] 13.898.7813.89_8.78 [1] 23.6330.2623.63_30.26 [2] 70.1714.9570.17_14.95 [8] 77.001.2777.00_1.27 [5] 59.682.6959.68_2.69 [8] 67.513.6867.51_3.68 [8] 77.042.0377.04_2.03 [8] 92.622.2292.62_2.22 [8] COG-MS 20.638.7420.63_8.74 [5] 26.114.8226.11_4.82 [4] 43.5716.8043.57_16.80 [1] 73.426.7273.42_6.72 [1] 23.963.4123.96_3.41 [4] 24.993.3824.99_3.38 [3] 52.3616.9652.36_16.96 [2] 74.154.4574.15_4.45 [2] 21.134.2421.13_4.24 [4] 22.675.1322.67_5.13 [4] 43.1722.1843.17_22.18 [5] 83.001.0583.00_1.05 [5] TST 10.614.7310.61_4.73 [1] 12.004.1012.00_4.10 [1] 58.8727.2258.87_27.22 [3] 77.562.4777.56_2.47 [4] 23.074.7823.07_4.78 [3] 38.3022.0538.30_22.05 [8] 66.9012.1566.90_12.15 [7] 77.932.3377.93_2.33 [6] 28.676.8828.67_6.88 [5] 26.842.6526.84_2.65 [5] 38.052.6638.05_2.66 [2] 82.3914.3082.39_14.30 [3] COG-TST 11.244.3211.24_4.32 [2] 15.633.5815.63_3.58 [2] 61.3923.7361.39_23.73 [4] 77.332.0977.33_2.09 [3] 24.218.7324.21_8.73 [5] 33.0024.3633.00_24.36 [6] 62.4317.5362.43_17.53 [5] 75.764.7575.76_4.75 [4] 10.793.3010.79_3.30 [1] 9.101.749.10_1.74 [1] 32.2311.6032.23_11.60 [1] 72.4524.4272.45_24.42 [1] C.2 All methods compared Here we provide a more detailed version of the results found in Section 4.2. Metrics used in the main text. As shown in Tab. 12, COG-FC achieves the highest overall rank across both backbones. Notably, COG-FC achieves this performance without introducing any additional trainable parameters, while being end to end and without a held out calibration set. Table 12: Comparison of all models and all variants seen in this work. We report meanstdmean_std. The best value in each column is shown in bold. Rank denotes the mean rank across ACC, NLL, and Brier for a given dataset (1 = best; ties share a rank). SVHN CIFAR-10 CIFAR-100 Method ACC (↑)( ) NLL (↓)( ) Brier (↓)( ) Rank (↓)( ) ACC (↑)( ) NLL (↓)( ) Brier (↓)( ) Rank (↓)( ) ACC (↑)( ) NLL (↓)( ) Brier (↓)( ) Rank (↓)( ) Baseline 95.770.1895.77_0.18 0.320.010.32_0.01 0.070.000.07_0.00 8.67 90.500.2190.50_0.21 0.640.020.64_0.02 0.170.000.17_0.00 8.00 65.041.3365.04_1.33 2.980.062.98_0.06 0.600.020.60_0.02 8.33 COG-FC 95.840.2095.84_0.20 0.160.000.16_0.00 0.060.000.06_0.00 2.67 89.940.2589.94_0.25 0.380.010.38_0.01 0.160.000.16_0.00 5.67 70.460.3070.46_0.30 1.080.011.08_0.01 0.410.000.41_0.00 1.33 [rgb]0,0.75,0.161.33 COG+LS 96.250.1596.25_0.15 0.210.000.21_0.00 0.070.000.07_0.00 4.67 89.920.2189.92_0.21 0.380.000.38_0.00 0.160.000.16_0.00 6.00 72.670.7872.67_0.78 1.310.041.31_0.04 0.410.010.41_0.01 2.00 COG-TS 95.860.0595.86_0.05 0.160.000.16_0.00 0.060.000.06_0.00 2.33 89.540.1589.54_0.15 0.400.020.40_0.02 0.160.000.16_0.00 8.00 67.550.6267.55_0.62 1.300.061.30_0.06 0.460.000.46_0.00 3.33 COG-MS 96.020.1196.02_0.11 0.160.000.16_0.00 0.060.000.06_0.00 1.33 [rgb]0,0.75,0.161.33 89.270.2689.27_0.26 0.410.010.41_0.01 0.170.000.17_0.00 10.33 67.740.5667.74_0.56 1.930.951.93_0.95 0.460.010.46_0.01 4.67 COG-TST 95.840.1095.84_0.10 0.160.000.16_0.00 0.060.000.06_0.00 2.67 88.730.4188.73_0.41 0.370.010.37_0.01 0.170.000.17_0.00 8.33 64.770.2564.77_0.25 1.240.011.24_0.01 0.470.000.47_0.00 5.00 LabelSmoothing 95.970.1295.97_0.12 0.230.000.23_0.00 0.070.000.07_0.00 6.33 90.050.3190.05_0.31 0.430.010.43_0.01 0.160.000.16_0.00 7.33 65.760.8865.76_0.88 1.580.041.58_0.04 0.480.010.48_0.01 5.33 Temp Scaling 95.770.1895.77_0.18 0.190.010.19_0.01 0.070.000.07_0.00 6.67 90.500.2190.50_0.21 0.380.010.38_0.01 0.150.000.15_0.00 3.00 65.041.3365.04_1.33 1.780.031.78_0.03 0.530.020.53_0.02 6.33 Matrix Scaling 95.910.1895.91_0.18 0.160.000.16_0.00 0.060.000.06_0.00 2.00 90.740.1790.74_0.17 0.280.000.28_0.00 0.140.000.14_0.00 1.00 [rgb]0,0.75,0.161.00 46.391.6846.39_1.68 19.102.1519.10_2.15 1.020.031.02_0.03 12.00 TST 94.690.0494.69_0.04 0.210.000.21_0.00 0.080.000.08_0.00 9.00 90.170.0690.17_0.06 0.370.000.37_0.00 0.150.000.15_0.00 3.33 52.690.4552.69_0.45 1.990.031.99_0.03 0.640.000.64_0.00 10.33 V-TST (MC=1) 93.740.1593.74_0.15 0.250.000.25_0.00 0.100.000.10_0.00 11.67 89.620.0889.62_0.08 0.390.020.39_0.02 0.160.000.16_0.00 7.33 50.500.3950.50_0.39 1.940.011.94_0.01 0.630.000.63_0.00 10.00 V-TST (MC=10) 94.660.0394.66_0.03 0.210.000.21_0.00 0.080.000.08_0.00 9.33 90.310.1690.31_0.16 0.340.010.34_0.01 0.150.000.15_0.00 2.67 52.710.2552.71_0.25 1.810.011.81_0.01 0.610.000.61_0.00 8.33 Calibration Errors. Here we report calibration errors under different norms and across multiple choices of the number of bins for all models and variants seen in this work (ECE in Tab. 13, RMSCE in Tab. 14, MCE in Tab. 15). Here, we again notice the switch in rank between different number of bins used to calculate the calibration metrics. Table 13: Comparison of ECE (↓ lower is better) for all models and all variants seen in this work for multiple choices of the number of bins. We report meanstdmean_std. The best value in each column is shown in bold. SVHN CIFAR-10 CIFAR-100 Method ECE-10 ECE-15 ECE-100 ECE-500 ECE-10 ECE-15 ECE-100 ECE-500 ECE-10 ECE-15 ECE-100 ECE-500 Baseline 3.260.113.26_0.11 3.250.113.25_0.11 3.290.103.29_0.10 3.450.103.45_0.10 7.380.257.38_0.25 7.390.237.39_0.23 7.470.237.47_0.23 7.890.227.89_0.22 26.560.7926.56_0.79 26.560.7826.56_0.78 26.600.7926.60_0.79 26.760.8326.76_0.83 COG-FC 1.140.301.14_0.30 1.160.291.16_0.29 1.320.171.32_0.17 1.860.071.86_0.07 5.320.165.32_0.16 5.320.145.32_0.14 5.490.115.49_0.11 6.220.126.22_0.12 8.000.358.00_0.35 8.000.368.00_0.36 8.130.378.13_0.37 9.200.289.20_0.28 COG+LS 7.440.207.44_0.20 7.510.197.51_0.19 7.820.177.82_0.17 8.320.178.32_0.17 4.900.184.90_0.18 5.100.315.10_0.31 5.650.315.65_0.31 7.170.267.17_0.26 6.812.106.81_2.10 6.941.856.94_1.85 7.951.597.95_1.59 9.331.079.33_1.07 COG-TS 1.230.141.23_0.14 1.240.141.24_0.14 1.410.111.41_0.11 1.950.111.95_0.11 5.520.405.52_0.40 5.520.405.52_0.40 5.660.405.66_0.40 6.400.406.40_0.40 11.142.0511.14_2.05 11.142.0511.14_2.05 11.212.0311.21_2.03 11.961.7711.96_1.77 COG-MS 1.370.081.37_0.08 1.380.081.38_0.08 1.510.061.51_0.06 1.970.101.97_0.10 5.670.245.67_0.24 5.680.245.68_0.24 5.840.255.84_0.25 6.610.156.61_0.15 11.462.2411.46_2.24 11.462.2411.46_2.24 11.552.1911.55_2.19 12.431.8212.43_1.82 COG-TST 1.210.111.21_0.11 1.210.111.21_0.11 1.390.091.39_0.09 1.910.071.91_0.07 4.960.324.96_0.32 4.960.324.96_0.32 5.140.305.14_0.30 6.130.176.13_0.17 4.040.554.04_0.55 4.070.574.07_0.57 4.700.504.70_0.50 7.180.357.18_0.35 Label Smoothing 6.810.296.81_0.29 7.130.227.13_0.22 7.330.227.33_0.22 7.660.227.66_0.22 6.370.456.37_0.45 5.320.305.32_0.30 6.720.276.72_0.27 7.590.237.59_0.23 4.062.054.06_2.05 4.511.904.51_1.90 5.311.675.31_1.67 7.901.227.90_1.22 Temp. Scaling 2.190.092.19_0.09 2.200.092.20_0.09 2.260.082.26_0.08 2.570.052.57_0.05 5.400.295.40_0.29 5.410.285.41_0.28 5.570.265.57_0.26 6.210.146.21_0.14 18.880.4118.88_0.41 18.880.4118.88_0.41 18.910.4218.91_0.42 19.260.4319.26_0.43 Matrix Scaling 0.710.160.71_0.16 0.730.150.73_0.15 1.050.071.05_0.07 1.840.091.84_0.09 0.730.120.73_0.12 0.890.100.89_0.10 1.760.071.76_0.07 3.740.223.74_0.22 50.181.7950.18_1.79 50.181.7950.18_1.79 50.191.7950.19_1.79 50.251.7950.25_1.79 TST 1.470.031.47_0.03 1.480.021.48_0.02 1.690.051.69_0.05 2.320.072.32_0.07 4.900.074.90_0.07 4.910.054.91_0.05 5.100.055.10_0.05 6.010.076.01_0.07 15.781.4615.78_1.46 15.801.4515.80_1.45 15.821.4115.82_1.41 16.221.4016.22_1.40 V-TST (MC=1) 1.530.861.53_0.86 1.560.821.56_0.82 1.850.691.85_0.69 2.600.552.60_0.55 2.070.832.07_0.83 2.140.802.14_0.80 2.820.672.82_0.67 4.440.344.44_0.34 6.601.706.60_1.70 6.611.706.61_1.70 6.931.646.93_1.64 9.001.299.00_1.29 V-TST (MC=10) 3.020.993.02_0.99 3.020.993.02_0.99 3.070.973.07_0.97 3.520.803.52_0.80 1.090.401.09_0.40 1.140.361.14_0.36 1.990.231.99_0.23 3.920.223.92_0.22 2.281.282.28_1.28 2.331.242.33_1.24 3.730.703.73_0.70 7.190.637.19_0.63 Table 14: Comparison of RMSCE (↓ lower is better) for all models and all variants seen in this work for multiple choices of the number of bins. We report meanstdmean_std. The best value in each column is shown in bold. SVHN CIFAR-10 CIFAR-100 Method RMSCE-10 RMSCE-15 RMSCE-100 RMSCE-500 RMSCE-10 RMSCE-15 RMSCE-100 RMSCE-500 RMSCE-10 RMSCE-15 RMSCE-100 RMSCE-500 Baseline 5.030.445.03_0.44 5.570.285.57_0.28 7.220.227.22_0.22 8.850.228.85_0.22 8.980.438.98_0.43 9.650.419.65_0.41 11.890.5411.89_0.54 14.580.4614.58_0.46 27.690.8627.69_0.86 28.270.8428.27_0.84 30.690.8330.69_0.83 32.470.8432.47_0.84 COG-FC 1.960.661.96_0.66 2.160.672.16_0.67 3.170.333.17_0.33 5.590.175.59_0.17 6.980.366.98_0.36 7.270.297.27_0.29 9.080.369.08_0.36 11.870.3411.87_0.34 9.180.469.18_0.46 9.390.479.39_0.47 10.270.5110.27_0.51 12.980.4612.98_0.46 COG+LS 7.590.177.59_0.17 7.700.147.70_0.14 8.100.138.10_0.13 9.190.149.19_0.14 5.280.285.28_0.28 5.420.425.42_0.42 6.530.376.53_0.37 9.730.259.73_0.25 9.652.039.65_2.03 10.912.7110.91_2.71 11.872.5411.87_2.54 14.681.9814.68_1.98 COG-TS 2.030.352.03_0.35 2.230.392.23_0.39 3.330.253.33_0.25 5.830.355.83_0.35 6.990.516.99_0.51 7.490.567.49_0.56 9.200.659.20_0.65 11.970.7811.97_0.78 12.342.1912.34_2.19 12.562.2812.56_2.28 13.512.4213.51_2.42 15.852.0315.85_2.03 COG-MS 2.310.142.31_0.14 2.530.212.53_0.21 3.560.133.56_0.13 5.800.195.80_0.19 7.170.497.17_0.49 7.610.607.61_0.60 9.330.539.33_0.53 12.190.4112.19_0.41 12.752.3512.75_2.35 13.062.4813.06_2.48 14.002.5214.00_2.52 16.352.2816.35_2.28 COG-TST 2.050.282.05_0.28 2.250.292.25_0.29 3.330.233.33_0.23 5.800.185.80_0.18 6.540.496.54_0.49 6.810.636.81_0.63 8.340.548.34_0.54 11.500.3411.50_0.34 4.590.684.59_0.68 4.700.634.70_0.63 6.220.746.22_0.74 10.200.5010.20_0.50 Label Smoothing 7.050.247.05_0.24 7.330.287.33_0.28 7.790.357.79_0.35 8.910.318.91_0.31 7.300.707.30_0.70 7.080.427.08_0.42 8.840.238.84_0.23 11.470.3011.47_0.30 5.242.725.24_2.72 5.713.025.71_3.02 7.102.677.10_2.67 11.122.1711.12_2.17 Temp. Scaling 3.620.273.62_0.27 3.890.303.89_0.30 4.930.304.93_0.30 6.720.296.72_0.29 7.030.407.03_0.40 7.580.417.58_0.41 9.460.469.46_0.46 12.180.3312.18_0.33 20.570.3920.57_0.39 21.070.4421.07_0.44 22.510.5522.51_0.55 24.140.5024.14_0.50 Matrix Scaling 1.260.251.26_0.25 1.400.231.40_0.23 2.630.162.63_0.16 5.340.145.34_0.14 1.460.251.46_0.25 1.900.341.90_0.34 4.150.284.15_0.28 8.520.378.52_0.37 50.301.7950.30_1.79 50.381.7950.38_1.79 50.911.7350.91_1.73 51.631.6451.63_1.64 TST 2.110.072.11_0.07 2.180.072.18_0.07 3.290.163.29_0.16 5.830.195.83_0.19 6.520.126.52_0.12 6.790.106.79_0.10 8.540.148.54_0.14 11.650.2111.65_0.21 17.071.5717.07_1.57 17.251.6317.25_1.63 17.901.6017.90_1.60 19.611.4419.61_1.44 V-TST (MC=1) 2.421.152.42_1.15 2.551.132.55_1.13 3.500.883.50_0.88 5.850.605.85_0.60 2.721.092.72_1.09 2.990.932.99_0.93 4.830.794.83_0.79 8.990.428.99_0.42 7.431.797.43_1.79 7.551.867.55_1.86 8.461.788.46_1.78 11.941.4311.94_1.43 V-TST (MC=10) 4.371.234.37_1.23 4.501.274.50_1.27 5.031.105.03_1.10 6.750.746.75_0.74 2.090.322.09_0.32 2.240.292.24_0.29 4.270.264.27_0.26 8.690.268.69_0.26 2.731.412.73_1.41 2.821.422.82_1.42 4.930.954.93_0.95 9.570.839.57_0.83 Table 15: Comparison of MCE (↓ lower is better) for all models and all variants seen in this work for multiple choices of the number of bins. We report meanstdmean_std. The best value in each column is shown in bold. SVHN CIFAR-10 CIFAR-100 Method MCE-10 MCE-15 MCE-100 MCE-500 MCE-10 MCE-15 MCE-100 MCE-500 MCE-10 MCE-15 MCE-100 MCE-500 Baseline 41.3118.9741.31_18.97 44.5716.4244.57_16.42 67.186.9067.18_6.90 86.852.5486.85_2.54 36.203.9936.20_3.99 38.152.7138.15_2.71 62.047.3262.04_7.32 89.101.4789.10_1.47 48.992.0548.99_2.05 52.191.2152.19_1.21 68.5310.0468.53_10.04 83.804.7983.80_4.79 COG-FC 17.276.0017.27_6.00 19.036.8119.03_6.81 48.9619.7448.96_19.74 74.041.0474.04_1.04 24.653.8224.65_3.82 28.059.3428.05_9.34 64.0312.0764.03_12.07 74.842.7574.84_2.75 15.221.2815.22_1.28 15.741.5015.74_1.50 40.7927.3540.79_27.35 72.5519.6372.55_19.63 COG+LS 26.9636.9626.96_36.96 31.0734.2631.07_34.26 64.6219.0964.62_19.09 82.1010.2882.10_10.28 13.534.9313.53_4.93 17.218.0817.21_8.08 62.4217.5362.42_17.53 74.221.9874.22_1.98 25.154.6425.15_4.64 25.525.2625.52_5.26 42.9710.7842.97_10.78 90.757.4790.75_7.47 COG-TS 33.3628.1533.36_28.15 35.9626.4435.96_26.44 80.522.1880.52_2.18 80.671.9980.67_1.99 22.904.4922.90_4.49 23.012.6623.01_2.66 50.6317.0050.63_17.00 74.074.2374.07_4.23 19.633.4519.63_3.45 20.763.2820.76_3.28 42.7422.0542.74_22.05 83.171.5783.17_1.57 COG-MS 20.638.7420.63_8.74 26.114.8226.11_4.82 43.5716.8043.57_16.80 73.426.7273.42_6.72 23.963.4123.96_3.41 24.993.3824.99_3.38 52.3616.9652.36_16.96 74.154.4574.15_4.45 21.134.2421.13_4.24 22.675.1322.67_5.13 43.1722.1843.17_22.18 83.001.0583.00_1.05 COG-TST 11.244.3211.24_4.32 15.633.5815.63_3.58 61.3923.7361.39_23.73 77.332.0977.33_2.09 24.218.7324.21_8.73 33.0024.3633.00_24.36 62.4317.5362.43_17.53 75.764.7575.76_4.75 10.793.3010.79_3.30 9.101.749.10_1.74 32.2311.6032.23_11.60 72.4524.4272.45_24.42 Label Smoothing 27.258.3227.25_8.32 24.462.6124.46_2.61 64.1323.2964.13_23.29 85.4113.4385.41_13.43 22.404.7222.40_4.72 28.718.3828.71_8.38 63.1912.0263.19_12.02 76.083.0976.08_3.09 12.235.7712.23_5.77 11.666.5011.66_6.50 27.2211.8327.22_11.83 74.0428.9074.04_28.90 Temp. Scaling 20.245.1820.24_5.18 33.5925.2033.59_25.20 66.7916.0266.79_16.02 78.233.0678.23_3.06 24.283.6724.28_3.67 25.011.7025.01_1.70 59.1113.2559.11_13.25 78.524.1278.52_4.12 36.931.1536.93_1.15 37.081.1037.08_1.10 53.4817.1453.48_17.14 82.831.1482.83_1.14 Matrix Scaling 30.1329.4130.13_29.41 30.1629.3830.16_29.38 68.9521.5968.95_21.59 78.932.5578.93_2.55 13.898.7813.89_8.78 23.6330.2623.63_30.26 70.1714.9570.17_14.95 77.001.2777.00_1.27 59.682.6959.68_2.69 67.513.6867.51_3.68 77.042.0377.04_2.03 92.622.2292.62_2.22 TST 10.614.7310.61_4.73 12.004.1012.00_4.10 58.8727.2258.87_27.22 77.562.4777.56_2.47 23.074.7823.07_4.78 38.3022.0538.30_22.05 66.9012.1566.90_12.15 77.932.3377.93_2.33 28.676.8828.67_6.88 26.842.6526.84_2.65 38.052.6638.05_2.66 82.3914.3082.39_14.30 V-TST (MC=1) 25.9331.2225.93_31.22 25.8831.2425.88_31.24 49.7729.8649.77_29.86 78.805.0578.80_5.05 30.7828.8030.78_28.80 30.7828.8030.78_28.80 43.7322.2343.73_22.23 79.422.5079.42_2.50 12.481.8112.48_1.81 16.956.2816.95_6.28 40.8631.2740.86_31.27 74.4425.9374.44_25.93 V-TST (MC=10) 16.713.2816.71_3.28 17.232.4617.23_2.46 38.258.4038.25_8.40 79.391.6979.39_1.69 23.527.0123.52_7.01 23.527.0123.52_7.01 48.5617.7848.56_17.78 78.792.9278.79_2.92 5.441.985.44_1.98 10.739.1810.73_9.18 20.496.3520.49_6.35 61.7129.7061.71_29.70 C.3 Performance under Covariate Shifts and OOD Metrics used in the main text. Tab. 16 reports the performance of all methods in out-of-distribution detection and Tab. 17 reports their performance under covariate shifts. Table 16: Performance in out-of-distribution detection or all models and all variants seen in this work. SVHN CIFAR-10 CIFAR-100 Method AUROC (↑)( ) FPR95 (↓)( ) Rank (↓)( ) AUROC (↑)( ) FPR95 (↓)( ) Rank (↓)( ) AUROC (↑)( ) FPR95 (↓)( ) Rank (↓)( ) Baseline 94.780.2494.78_0.24 34.431.2734.43_1.27 8.00 83.972.7083.97_2.70 76.393.2076.39_3.20 7.00 71.921.3371.92_1.33 87.671.3487.67_1.34 7.00 COG-FC 96.600.4196.60_0.41 21.602.5921.60_2.59 2.00 87.001.3287.00_1.32 73.984.0373.98_4.03 3.00 68.502.2268.50_2.22 85.831.4385.83_1.43 8.00 COG+LS 93.800.6193.80_0.61 22.582.4522.58_2.45 6.50 81.933.1081.93_3.10 73.433.1873.43_3.18 6.00 65.487.8565.48_7.85 83.713.4283.71_3.42 6.50 COG-TS 96.210.3596.21_0.35 24.712.6624.71_2.66 4.50 86.040.8986.04_0.89 77.012.2177.01_2.21 6.00 67.311.2267.31_1.22 88.670.8188.67_0.81 10.50 COG-MS 96.700.2896.70_0.28 20.362.1820.36_2.18 1.00 [rgb]0,0.75,0.161.00 86.200.5686.20_0.56 77.271.7477.27_1.74 6.00 69.401.5569.40_1.55 86.221.0686.22_1.06 8.00 COG-TST 96.600.0996.60_0.09 22.660.5122.66_0.51 3.00 87.410.7087.41_0.70 71.372.3471.37_2.34 1.50 [rgb]0,0.75,0.161.50 73.922.7273.92_2.72 79.962.3179.96_2.31 1.50 [rgb]0,0.75,0.161.50 Label Smoothing 80.581.6680.58_1.66 42.842.4042.84_2.40 11.50 72.176.8872.17_6.88 78.883.0978.88_3.09 11.50 74.352.9874.35_2.98 84.061.8684.06_1.86 2.00 Temp. Scaling 95.330.2595.33_0.25 31.141.8531.14_1.85 6.50 85.092.4385.09_2.43 75.044.2275.04_4.22 5.50 73.231.5373.23_1.53 85.462.0085.46_2.00 3.50 Matrix Scaling 96.040.2596.04_0.25 25.311.8825.31_1.88 5.50 86.862.0686.86_2.06 70.927.2070.92_7.20 2.00 58.231.5558.23_1.55 93.010.3393.01_0.33 12.00 TST 94.850.0394.85_0.03 35.960.3235.96_0.32 8.50 84.510.2384.51_0.23 77.530.3177.53_0.31 8.00 69.910.4169.91_0.41 85.770.8485.77_0.84 5.50 V-TST (MC=1) 92.280.9192.28_0.91 46.493.1446.49_3.14 11.50 77.362.7077.36_2.70 80.331.9380.33_1.93 11.50 69.590.7069.59_0.70 86.920.4186.92_0.41 8.00 V-TST (MC=10) 94.660.4794.66_0.47 35.352.4035.35_2.40 9.00 79.232.2779.23_2.27 78.611.9678.61_1.96 10.00 70.750.8570.75_0.85 85.780.6485.78_0.64 5.50 Table 17: Performance under covariate shifts. Shift metrics are averaged across all considered covariate shifts. SVHN CIFAR-10 CIFAR-100 Method ACC (↑)( ) NLL (↓)( ) Brier (↓)( ) Rank (↓)( ) ACC (↑)( ) NLL (↓)( ) Brier (↓)( ) Rank (↓)( ) ACC (↑)( ) NLL (↓)( ) Brier (↓)( ) Rank (↓)( ) Baseline 47.011.0647.01_1.06 5.290.235.29_0.23 0.940.020.94_0.02 8.67 74.280.5974.28_0.59 2.030.112.03_0.11 0.450.010.45_0.01 6.67 45.851.3645.85_1.36 5.700.215.70_0.21 0.920.020.92_0.02 7.67 COG-FC 45.821.3245.82_1.32 2.430.122.43_0.12 0.800.020.80_0.02 6.00 68.800.6868.80_0.68 1.570.091.57_0.09 0.500.010.50_0.01 9.00 44.090.4244.09_0.42 2.880.042.88_0.04 0.770.010.77_0.01 4.00 COG+LS 46.050.5746.05_0.57 1.910.021.91_0.02 0.770.000.77_0.00 3.00 69.310.4269.31_0.42 1.110.021.11_0.02 0.460.010.46_0.01 7.33 44.920.6944.92_0.69 2.900.112.90_0.11 0.810.030.81_0.03 5.67 COG-TS 45.590.5045.59_0.50 3.350.123.35_0.12 0.880.010.88_0.01 10.33 67.980.2267.98_0.22 2.360.172.36_0.17 0.550.010.55_0.01 11.33 40.990.2240.99_0.22 5.230.635.23_0.63 0.930.020.93_0.02 9.33 COG-MS 46.380.5146.38_0.51 2.530.102.53_0.10 0.810.010.81_0.01 6.00 67.670.6367.67_0.63 1.680.111.68_0.11 0.520.010.52_0.01 10.67 43.630.8043.63_0.80 4.101.234.10_1.23 0.800.020.80_0.02 6.67 COG-TST 45.680.6745.68_0.67 2.280.042.28_0.04 0.800.010.80_0.01 5.67 67.420.4267.42_0.42 1.390.031.39_0.03 0.510.010.51_0.01 10.00 40.810.5040.81_0.50 2.710.032.71_0.03 0.770.000.77_0.00 4.33 Label Smoothing 46.190.5946.19_0.59 2.190.042.19_0.04 0.860.010.86_0.01 6.33 74.220.4274.22_0.42 1.030.021.03_0.02 0.420.010.42_0.01 4.00 47.140.9147.14_0.91 2.540.102.54_0.10 0.710.030.71_0.03 1.00 [rgb]0,0.75,0.161.00 Temp. Scaling 47.011.0647.01_1.06 3.050.123.05_0.12 0.850.020.85_0.02 7.00 74.280.5974.28_0.59 1.190.051.19_0.05 0.410.010.41_0.01 4.00 45.851.3645.85_1.36 3.320.103.32_0.10 0.810.020.81_0.02 5.67 Matrix Scaling 47.270.7247.27_0.72 2.140.022.14_0.02 0.750.010.75_0.01 1.67 [rgb]0,0.75,0.161.67 74.350.6674.35_0.66 0.840.030.84_0.03 0.360.010.36_0.01 1.00 [rgb]0,0.75,0.161.00 30.451.0730.45_1.07 29.623.4029.62_3.40 1.330.031.33_0.03 12.00 TST 41.740.1141.74_0.11 2.660.022.66_0.02 0.840.000.84_0.00 9.00 72.850.2172.85_0.21 1.100.021.10_0.02 0.420.000.42_0.00 5.67 38.230.3338.23_0.33 3.240.083.24_0.08 0.840.010.84_0.01 8.67 V-TST (MC=1) 40.310.2640.31_0.26 2.380.162.38_0.16 0.830.020.83_0.02 8.33 72.900.2472.90_0.24 1.030.061.03_0.06 0.410.010.41_0.01 4.00 36.770.2636.77_0.26 2.830.042.83_0.04 0.800.010.80_0.01 6.67 V-TST (MC=10) 41.660.2641.66_0.26 2.060.072.06_0.07 0.770.010.77_0.01 5.00 73.910.2273.91_0.22 0.910.030.91_0.03 0.380.010.38_0.01 3.00 38.440.2438.44_0.24 2.660.022.66_0.02 0.770.000.77_0.00 4.33 Calibration Errors. Here, we additionally report the results for the different calibration error metrics. In particular, V-TST performs particularly strongly according to the considered calibration errors. COG-FC achieves competitive performance across the different metrics and binning configurations. Table 18: ECE averaged across all considered covariate shifts for all models and all variants seen in this work for multiple choices of the number of bins. SVHN CIFAR-10 CIFAR-100 Method ECE-10 ECE-15 ECE-100 ECE-500 ECE-10 ECE-15 ECE-100 ECE-500 ECE-10 ECE-15 ECE-100 ECE-500 Baseline 43.811.0143.81_1.01 43.811.0143.81_1.01 43.811.0143.81_1.01 43.821.0143.82_1.01 20.680.5620.68_0.56 20.680.5620.68_0.56 20.680.5620.68_0.56 20.660.5620.66_0.56 41.900.8941.90_0.89 41.890.8941.89_0.89 41.890.8941.89_0.89 41.860.8941.86_0.89 COG-FC 29.552.3229.55_2.32 29.552.3229.55_2.32 29.552.3229.55_2.32 29.562.3129.56_2.31 20.460.8620.46_0.86 20.450.8620.45_0.86 20.450.8620.45_0.86 20.430.8620.43_0.86 22.930.1922.93_0.19 22.930.1922.93_0.19 22.930.1922.93_0.19 22.930.1922.93_0.19 COG+LS 24.900.6124.90_0.61 24.900.6124.90_0.61 24.910.6124.91_0.61 24.910.6124.91_0.61 11.961.1411.96_1.14 11.961.1411.96_1.14 11.991.1011.99_1.10 12.001.0912.00_1.09 25.933.9225.93_3.92 25.933.9225.93_3.92 25.933.9225.93_3.92 25.933.9225.93_3.92 COG-TS 37.760.7537.76_0.75 37.760.7537.76_0.75 37.760.7537.76_0.75 37.770.7637.77_0.76 24.800.6324.80_0.63 24.800.6324.80_0.63 24.800.6324.80_0.63 24.780.6324.78_0.63 38.372.3038.37_2.30 38.372.3038.37_2.30 38.362.2938.36_2.29 38.352.2938.35_2.29 COG-MS 30.831.3930.83_1.39 30.831.3930.83_1.39 30.831.3930.83_1.39 30.831.3930.83_1.39 21.300.9221.30_0.92 21.290.9221.29_0.92 21.290.9221.29_0.92 21.270.9221.27_0.92 26.762.9526.76_2.95 26.762.9526.76_2.95 26.762.9526.76_2.95 26.752.9526.75_2.95 COG-TST 29.100.9729.10_0.97 29.100.9729.10_0.97 29.100.9729.10_0.97 29.110.9729.11_0.97 19.220.4019.22_0.40 19.220.4019.22_0.40 19.210.4019.21_0.40 19.200.4019.20_0.40 16.430.4716.43_0.47 16.430.4716.43_0.47 16.430.4616.43_0.46 16.430.4616.43_0.46 Label Smoothing 34.600.8934.60_0.89 34.600.8934.60_0.89 34.600.8934.60_0.89 34.610.8934.61_0.89 11.390.4011.39_0.40 11.390.4011.39_0.40 11.710.2911.71_0.29 11.750.3511.75_0.35 11.145.1011.14_5.10 11.165.0911.16_5.09 11.175.0811.17_5.08 11.195.0711.19_5.07 Temp. Scaling 35.901.1335.90_1.13 35.891.1335.89_1.13 35.901.1335.90_1.13 35.901.1335.90_1.13 16.240.6516.24_0.65 16.230.6516.23_0.65 16.230.6516.23_0.65 16.210.6516.21_0.65 30.940.8930.94_0.89 30.930.8930.93_0.89 30.920.8930.92_0.89 30.920.8930.92_0.89 Matrix Scaling 23.930.8723.93_0.87 23.930.8723.93_0.87 23.940.8723.94_0.87 23.950.8723.95_0.87 7.610.987.61_0.98 7.600.987.60_0.98 7.590.987.59_0.98 7.590.997.59_0.99 65.411.3765.41_1.37 65.401.3765.40_1.37 65.391.3765.39_1.37 65.371.3765.37_1.37 TST 31.220.1731.22_0.17 31.220.1731.22_0.17 31.220.1731.22_0.17 31.230.1731.23_0.17 15.620.2215.62_0.22 15.620.2215.62_0.22 15.610.2215.61_0.22 15.600.2215.60_0.22 25.331.6025.33_1.60 25.331.6025.33_1.60 25.331.6025.33_1.60 25.331.6025.33_1.60 V-TST (MC=1) 27.013.1527.01_3.15 27.013.1527.01_3.15 27.013.1527.01_3.15 27.013.1527.01_3.15 11.112.1511.11_2.15 11.102.1511.10_2.15 11.082.1411.08_2.14 11.082.1311.08_2.13 14.111.7814.11_1.78 14.111.7814.11_1.78 14.101.7814.10_1.78 14.111.7814.11_1.78 V-TST (MC=10) 20.522.3420.52_2.34 20.522.3420.52_2.34 20.522.3420.52_2.34 20.542.3420.54_2.34 8.262.018.26_2.01 8.252.018.25_2.01 8.242.008.24_2.00 8.261.988.26_1.98 9.571.599.57_1.59 9.571.599.57_1.59 9.571.599.57_1.59 9.571.599.57_1.59 Table 19: RMSCE averaged across all considered covariate shifts for all models and all variants seen in this work for multiple choices of the number of bins. SVHN CIFAR-10 CIFAR-100 Method RMSCE-10 RMSCE-15 RMSCE-100 RMSCE-500 RMSCE-10 RMSCE-15 RMSCE-100 RMSCE-500 RMSCE-10 RMSCE-15 RMSCE-100 RMSCE-500 Baseline 44.251.0044.25_1.00 44.430.9944.43_0.99 45.420.9745.42_0.97 46.120.9746.12_0.97 21.710.4921.71_0.49 22.210.4722.21_0.47 24.290.4024.29_0.40 25.440.3925.44_0.39 42.510.8642.51_0.86 42.800.8442.80_0.84 44.180.7744.18_0.77 44.980.7544.98_0.75 COG-FC 30.732.2030.73_2.20 31.002.2131.00_2.21 31.682.3731.68_2.37 31.822.3931.82_2.39 21.660.7921.66_0.79 22.170.7822.17_0.78 23.720.8623.72_0.86 24.200.9224.20_0.92 24.080.1824.08_0.18 24.290.1824.29_0.18 24.630.1724.63_0.17 24.680.1724.68_0.17 COG+LS 26.500.7326.50_0.73 26.710.6426.71_0.64 27.180.6627.18_0.66 27.310.6627.31_0.66 14.310.8814.31_0.88 14.260.7114.26_0.71 14.750.8014.75_0.80 14.810.8114.81_0.81 27.113.8627.11_3.86 27.884.1827.88_4.18 28.264.3128.26_4.31 28.294.3228.29_4.32 COG-TS 38.640.7938.64_0.79 38.920.8038.92_0.80 40.050.8940.05_0.89 40.580.9240.58_0.92 25.640.5625.64_0.56 26.090.5326.09_0.53 27.940.5327.94_0.53 28.860.5828.86_0.58 39.192.2039.19_2.20 39.482.1939.48_2.19 40.502.2640.50_2.26 40.872.3840.87_2.38 COG-MS 31.921.3531.92_1.35 32.181.3732.18_1.37 32.901.4232.90_1.42 33.041.4333.04_1.43 22.410.8622.41_0.86 22.900.8422.90_0.84 24.410.8824.41_0.88 24.890.9224.89_0.92 27.802.9127.80_2.91 28.042.9728.04_2.97 28.523.1528.52_3.15 28.603.1828.60_3.18 COG-TST 30.261.0230.26_1.02 30.511.0530.51_1.05 31.131.1231.13_1.12 31.241.1331.24_1.13 20.400.3720.40_0.37 20.850.3720.85_0.37 22.050.3522.05_0.35 22.340.3722.34_0.37 17.320.5317.32_0.53 17.410.5317.41_0.53 17.510.5417.51_0.54 17.530.5417.53_0.54 Label Smoothing 36.061.0936.06_1.09 35.990.7735.99_0.77 36.870.9336.87_0.93 37.050.9637.05_0.96 15.571.0615.57_1.06 14.590.1914.59_0.19 16.330.6816.33_0.68 16.590.6516.59_0.65 12.915.2112.91_5.21 13.155.3513.15_5.35 13.255.4013.25_5.40 13.295.3913.29_5.39 Temp. Scaling 36.741.0936.74_1.09 37.001.0937.00_1.09 37.901.1437.90_1.14 38.221.1638.22_1.16 17.720.5317.72_0.53 18.280.5318.28_0.53 19.900.5919.90_0.59 20.380.6420.38_0.64 32.070.8632.07_0.86 32.430.8632.43_0.86 33.490.9033.49_0.90 33.770.9333.77_0.93 Matrix Scaling 24.910.8824.91_0.88 25.030.9025.03_0.90 25.190.9225.19_0.92 25.310.9225.31_0.92 8.671.118.67_1.11 8.841.158.84_1.15 9.071.219.07_1.21 9.131.209.13_1.20 65.641.3665.64_1.36 65.651.3665.65_1.36 65.791.3365.79_1.33 65.931.3065.93_1.30 TST 32.630.1732.63_0.17 32.920.1732.92_0.17 33.600.1833.60_0.18 33.760.1833.76_0.18 17.190.2117.19_0.21 17.640.2017.64_0.20 18.960.2218.96_0.22 19.320.2219.32_0.22 26.451.6326.45_1.63 26.631.6626.63_1.66 26.921.7326.92_1.73 26.961.7426.96_1.74 V-TST (MC=1) 28.313.1728.31_3.17 28.473.2328.47_3.23 28.663.3528.66_3.35 28.763.3528.76_3.35 12.052.1812.05_2.18 12.262.2112.26_2.21 12.582.2912.58_2.29 12.652.2912.65_2.29 15.301.8715.30_1.87 15.361.8815.36_1.88 15.431.9015.43_1.90 15.461.9015.46_1.90 V-TST (MC=10) 21.702.4721.70_2.47 21.802.5321.80_2.53 21.912.5521.91_2.55 22.052.5422.05_2.54 9.252.149.25_2.14 9.422.199.42_2.19 9.612.239.61_2.23 9.682.229.68_2.22 10.391.6810.39_1.68 10.411.6910.41_1.69 10.451.7010.45_1.70 10.491.6910.49_1.69 Table 20: MCE averaged across all considered covariate shifts for all models and all variants seen in this work for multiple choices of the number of bins. SVHN CIFAR-10 CIFAR-100 Method MCE-10 MCE-15 MCE-100 MCE-500 MCE-10 MCE-15 MCE-100 MCE-500 MCE-10 MCE-15 MCE-100 MCE-500 Baseline 58.641.1858.64_1.18 62.430.8162.43_0.81 72.237.8872.23_7.88 76.604.8476.60_4.84 44.300.5144.30_0.51 46.830.3746.83_0.37 54.9113.7854.91_13.78 65.0713.8265.07_13.82 59.430.3659.43_0.36 62.380.5262.38_0.52 64.120.6064.12_0.60 72.8610.4172.86_10.41 COG-FC 45.332.3545.33_2.35 46.362.8346.36_2.83 47.612.8847.61_2.88 58.7914.2258.79_14.22 37.891.2337.89_1.23 38.701.4438.70_1.44 47.8418.3747.84_18.37 71.6117.0071.61_17.00 33.810.2133.81_0.21 34.230.2134.23_0.21 40.1710.6740.17_10.67 49.3124.2049.31_24.20 COG+LS 39.561.4039.56_1.40 58.2719.8658.27_19.86 97.991.0397.99_1.03 98.131.0298.13_1.02 25.851.5325.85_1.53 26.181.6726.18_1.67 44.3324.1244.33_24.12 60.9522.1160.95_22.11 41.055.7941.05_5.79 46.229.7046.22_9.70 77.6715.9277.67_15.92 92.0016.6192.00_16.61 COG-TS 54.801.6254.80_1.62 57.111.4057.11_1.40 59.031.3759.03_1.37 74.2111.8874.21_11.88 52.1516.1452.15_16.14 54.0115.1054.01_15.10 61.4616.5361.46_16.53 77.842.2877.84_2.28 53.652.2653.65_2.26 55.012.7755.01_2.77 55.753.0055.75_3.00 57.572.8457.57_2.84 COG-MS 46.551.6046.55_1.60 47.671.9247.67_1.92 49.191.5249.19_1.52 53.371.4753.37_1.47 38.171.2338.17_1.23 39.191.2339.19_1.23 57.1221.1057.12_21.10 72.9916.3272.99_16.32 39.094.4239.09_4.42 39.434.2839.43_4.28 40.264.1540.26_4.15 50.8521.9850.85_21.98 COG-TST 44.421.9944.42_1.99 45.042.1245.04_2.12 62.3222.1262.32_22.12 67.5117.9767.51_17.97 34.270.9734.27_0.97 34.331.0234.33_1.02 44.9622.0344.96_22.03 64.1024.5864.10_24.58 24.250.7324.25_0.73 24.300.7024.30_0.70 25.801.5625.80_1.56 27.570.8827.57_0.88 Label Smoothing 50.061.2250.06_1.22 62.359.7662.35_9.76 96.582.7296.58_2.72 97.830.9097.83_0.90 32.350.5132.35_0.51 36.562.0336.56_2.03 45.7419.4645.74_19.46 76.5418.3476.54_18.34 22.378.2522.37_8.25 22.658.2222.65_8.22 27.679.5827.67_9.58 39.9816.2339.98_16.23 Temp. Scaling 51.291.0551.29_1.05 53.130.8353.13_0.83 67.0115.7867.01_15.78 73.5013.0173.50_13.01 34.410.8434.41_0.84 35.201.1635.20_1.16 36.151.0736.15_1.07 55.4522.6855.45_22.68 47.341.1647.34_1.16 47.591.4747.59_1.47 48.551.6348.55_1.63 66.8421.9966.84_21.99 Matrix Scaling 34.461.4034.46_1.40 35.031.5235.03_1.52 37.080.8537.08_0.85 67.0323.0367.03_23.03 14.272.1214.27_2.12 14.532.1714.53_2.17 16.151.6716.15_1.67 60.5431.6460.54_31.64 69.920.3769.92_0.37 74.360.4174.36_0.41 81.050.8181.05_0.81 81.730.8181.73_0.81 TST 48.380.3848.38_0.38 48.500.3748.50_0.37 50.880.8550.88_0.85 67.2415.3467.24_15.34 31.460.2131.46_0.21 31.930.2131.93_0.21 48.2223.1348.22_23.13 74.6719.5874.67_19.58 50.4527.7350.45_27.73 50.7727.5450.77_27.54 51.7627.0251.76_27.02 63.1930.2763.19_30.27 V-TST (MC=1) 40.534.8240.53_4.82 41.004.6841.00_4.68 43.944.5343.94_4.53 54.1916.2754.19_16.27 20.083.8120.08_3.81 20.613.8720.61_3.87 25.795.8025.79_5.80 52.5521.5652.55_21.56 52.9242.9852.92_42.98 52.9442.9752.94_42.97 53.6242.3453.62_42.34 55.8240.3655.82_40.36 V-TST (MC=10) 31.873.7031.87_3.70 32.273.8932.27_3.89 34.243.9734.24_3.97 42.044.6142.04_4.61 16.023.7116.02_3.71 16.283.8216.28_3.82 19.853.8919.85_3.89 71.4918.3371.49_18.33 31.5338.3031.53_38.30 31.7338.1931.73_38.19 32.2937.8732.29_37.87 40.4135.3240.41_35.32 C.4 ViT ablation In this subsection, we report calibration metrics for ViT on CIFAR-10. We train the ViT model from scratch using the architecture described in (Jordahn & Olmos, 2024). The results are reported in Tab. 21. Table 21: Comparison of calibration errors for ViT in CIFAR-10 for multiple choices of the number of bins. Method ECE-10 ECE-15 ECE-100 ECE-500 RMSCE-10 RMSCE-15 RMSCE-100 RMSCE-500 MCE-10 MCE-15 MCE-100 MCE-500 Baseline 7.730.667.73_0.66 [4] 7.730.667.73_0.66 [4] 7.900.667.90_0.66 [4] 9.180.779.18_0.77 [4] 8.490.798.49_0.79 [3] 8.770.868.77_0.86 [4] 9.670.899.67_0.89 [4] 12.460.8712.46_0.87 [3] 15.843.9215.84_3.92 [2] 15.943.6715.94_3.67 [2] 41.1121.8941.11_21.89 [2] 76.031.7476.03_1.74 [3] COG-FC 7.700.707.70_0.70 [3] 7.700.677.70_0.67 [3] 7.870.617.87_0.61 [3] 8.990.618.99_0.61 [3] 8.610.738.61_0.73 [5] 8.880.758.88_0.75 [5] 9.750.729.75_0.72 [5] 12.520.6812.52_0.68 [4] 28.7628.9128.76_28.91 [6] 29.4528.5129.45_28.51 [6] 59.2623.8359.26_23.83 [5] 77.841.6777.84_1.67 [5] Temp Scaling 1.650.341.65_0.34 [2] 1.720.321.72_0.32 [2] 3.110.243.11_0.24 [2] 6.190.136.19_0.13 [2] 1.960.301.96_0.30 [2] 2.160.332.16_0.33 [2] 4.280.364.28_0.36 [2] 8.850.118.85_0.11 [2] 21.005.6321.00_5.63 [4] 21.005.6321.00_5.63 [4] 40.9028.6240.90_28.62 [1] 78.782.6178.78_2.61 [6] COG-TS 8.091.468.09_1.46 [6] 8.111.458.11_1.45 [6] 8.281.408.28_1.40 [6] 9.501.069.50_1.06 [6] 8.891.508.89_1.50 [6] 9.081.579.08_1.57 [6] 10.001.5610.00_1.56 [6] 12.851.2412.85_1.24 [6] 14.222.1514.22_2.15 [1] 15.732.6915.73_2.69 [1] 58.5825.2958.58_25.29 [4] 75.972.2675.97_2.26 [2] Matrix Scaling 1.000.131.00_0.13 [1] 1.220.211.22_0.21 [1] 2.720.382.72_0.38 [1] 5.870.225.87_0.22 [1] 1.440.191.44_0.19 [1] 1.820.321.82_0.32 [1] 4.020.464.02_0.46 [1] 8.680.088.68_0.08 [1] 20.9910.3520.99_10.35 [3] 20.9910.3520.99_10.35 [3] 61.3828.2161.38_28.21 [6] 75.8412.0175.84_12.01 [1] COG-MS 7.740.547.74_0.54 [5] 7.760.537.76_0.53 [5] 7.910.547.91_0.54 [5] 9.250.409.25_0.40 [5] 8.530.548.53_0.54 [4] 8.760.628.76_0.62 [3] 9.630.599.63_0.59 [3] 12.530.4912.53_0.49 [5] 27.8029.7227.80_29.72 [5] 28.3529.5028.35_29.50 [5] 56.6422.1156.64_22.11 [3] 77.622.3677.62_2.36 [4] C.5 Multi-Task Learning Since to the best of our knowledge, there are no MTL-specific calibration error metrics, we follow (Vandenhirtz et al., 2024; van Krieken et al., 2026) and measure the overall calibration error as the unweighted average of the calibration errors computed independently for each task, across all norms. As shown in Tab. 22, COG-FC and COG+LS consistently improve upon their counterparts, while COG-TS performs worse than Temperature Scaling. Figure 6: Per-task metrics on CelebA. Tasks are ordered by their mean Brier. COG-FC achieves higher accuracy than Temperature Scaling and Label Smoothing on all 40 tasks. In terms of Brier, COG-FC outperforms Temperature Scaling on 32 of 40 tasks and Label Smoothing on all 40 tasks. Table 22: Calibration errors for ResNet101 on CelebA, averaged across tasks. Bold indicates the best value within each group. CelebA Method ECE-10 ECE-15 ECE-100 ECE-500 RMSCE-10 RMSCE-15 RMSCE-100 RMSCE-500 MCE-10 MCE-15 MCE-100 MCE-500 Baseline 5.840.085.84_0.08 5.850.085.85_0.08 6.010.086.01_0.08 6.560.086.56_0.08 7.520.167.52_0.16 7.830.147.83_0.14 9.290.119.29_0.11 11.470.1011.47_0.10 26.890.7426.89_0.74 29.350.6929.35_0.69 49.930.4649.93_0.46 82.511.1782.51_1.17 COG-FC 5.010.045.01_0.04 5.030.035.03_0.03 5.190.045.19_0.04 5.770.035.77_0.03 6.500.076.50_0.07 6.820.066.82_0.06 8.270.068.27_0.06 10.560.0510.56_0.05 23.380.8623.38_0.86 25.500.5425.50_0.54 48.250.4548.25_0.45 81.910.6581.91_0.65 Label Smoothing 4.880.054.88_0.05 4.900.064.90_0.06 5.090.055.09_0.05 5.760.055.76_0.05 6.590.116.59_0.11 6.920.116.92_0.11 8.260.108.26_0.10 10.450.0710.45_0.07 23.800.6523.80_0.65 25.460.7725.46_0.77 46.981.6246.98_1.62 80.590.4180.59_0.41 COG+LS 4.090.044.09_0.04 4.110.054.11_0.05 4.330.044.33_0.04 5.030.045.03_0.04 5.610.045.61_0.04 5.940.055.94_0.05 7.270.037.27_0.03 9.640.059.64_0.05 19.570.3219.57_0.32 21.820.4221.82_0.42 45.070.5145.07_0.51 79.470.5679.47_0.56 Temp. Scaling 3.270.093.27_0.09 3.290.093.29_0.09 3.530.083.53_0.08 4.400.064.40_0.06 4.780.174.78_0.17 4.980.164.98_0.16 6.060.136.06_0.13 8.630.108.63_0.10 17.110.8117.11_0.81 19.020.9319.02_0.93 39.360.9239.36_0.92 76.391.6076.39_1.60 COG-TS 5.030.025.03_0.02 5.050.015.05_0.01 5.210.015.21_0.01 5.830.025.83_0.02 6.350.046.35_0.04 6.680.056.68_0.05 8.070.078.07_0.07 10.410.0610.41_0.06 22.011.0722.01_1.07 23.870.9023.87_0.90 45.770.9445.77_0.94 80.160.3680.16_0.36 Per-task ECE-100. In Fig. 7, we plot the ECE-100 for the best-performing models on CelebA. We observe the same overall ordering of task difficulty across methods, with all methods identifying the same tasks as the most challenging, including Big Lips. We also observe substantial variance for some tasks across all models, particularly for Label Smoothing, consistent with the variability observed in the corresponding Brier score results in Fig. 6. Figure 7: Per-task ECE-100 for the best-performing models evaluated in the main text. Appendix D Results with a WRN backbone In this section, we provide a brief summary of the results for in-distribution calibration, out-of-distribution detection, and performance under covariate shift using a WRN-28-10 backbone (Zagoruyko & Komodakis, 2016). The experimental settings are otherwise identical to those used with the ResNet backbone. All Experimental Results. In Tab. 23, we compare all model families in terms of calibration performance. On SVHN, we observe that almost all methods achieve similar Brier scores, with the main differences arising in NLL and accuracy, where the COG variants consistently dominate. On CIFAR-10, the (V)-TST models perform particularly well in both Brier score and accuracy. Finally, on CIFAR-100, we find that the COG variants generally achieve strong performance across the considered metrics. We again observe the same optimization issue with Matrix Scaling on CIFAR-100. In Tables 24- 26, we report ECE, RMSCE and MCE values for the same models. Table 23: Comparison of all models and all variants seen in this work. We report meanstdmean_std. The best value in each column is shown in bold. Rank denotes the mean rank across ACC, NLL, and Brier for a given dataset (1 = best; ties share a rank). SVHN CIFAR-10 CIFAR-100 Method ACC (↑)( ) NLL (↓)( ) Brier (↓)( ) Rank (↓)( ) ACC (↑)( ) NLL (↓)( ) Brier (↓)( ) Rank (↓)( ) ACC (↑)( ) NLL (↓)( ) Brier (↓)( ) Rank (↓)( ) Baseline 95.350.2095.35_0.20 0.350.010.35_0.01 0.080.000.08_0.00 9.67 91.330.6091.33_0.60 0.530.020.53_0.02 0.150.010.15_0.01 7.67 69.491.1569.49_1.15 2.330.112.33_0.11 0.520.020.52_0.02 7.67 COG-FC 95.600.1595.60_0.15 0.170.000.17_0.00 0.070.000.07_0.00 3.00 88.880.3488.88_0.34 0.360.010.36_0.01 0.170.000.17_0.00 8.00 67.890.5567.89_0.55 1.190.041.19_0.04 0.440.010.44_0.01 3.00 COG+LS 95.920.1995.92_0.19 0.220.000.22_0.00 0.070.000.07_0.00 3.67 89.410.3089.41_0.30 0.370.010.37_0.01 0.160.000.16_0.00 7.67 71.981.0471.98_1.04 1.320.041.32_0.04 0.400.010.40_0.01 2.33 COG-TS 95.640.1595.64_0.15 0.160.000.16_0.00 0.070.000.07_0.00 2.00 88.550.1188.55_0.11 0.380.010.38_0.01 0.170.000.17_0.00 9.00 65.140.5765.14_0.57 1.340.061.34_0.06 0.480.010.48_0.01 5.67 COG-MS 95.760.1995.76_0.19 0.160.010.16_0.01 0.070.000.07_0.00 1.33 [rgb]0,0.75,0.161.33 87.741.1787.74_1.17 0.400.020.40_0.02 0.180.010.18_0.01 11.00 66.310.8666.31_0.86 1.350.011.35_0.01 0.470.010.47_0.01 5.33 COG-TST 95.540.1095.54_0.10 0.170.000.17_0.00 0.070.000.07_0.00 3.33 88.070.4988.07_0.49 0.400.020.40_0.02 0.180.010.18_0.01 10.67 61.450.9061.45_0.90 1.380.031.38_0.03 0.500.010.50_0.01 8.00 Label Smoothing 95.740.2195.74_0.21 0.240.010.24_0.01 0.080.000.08_0.00 7.67 91.180.3691.18_0.36 0.390.010.39_0.01 0.150.010.15_0.01 7.33 73.580.4373.58_0.43 1.260.021.26_0.02 0.380.010.38_0.01 1.33 [rgb]0,0.75,0.161.33 Temp. Scaling 95.350.2095.35_0.20 0.210.010.21_0.01 0.070.000.07_0.00 5.67 91.330.6091.33_0.60 0.330.020.33_0.02 0.140.010.14_0.01 5.00 69.491.1569.49_1.15 1.440.061.44_0.06 0.460.010.46_0.01 5.00 Matrix Scaling 95.480.1895.48_0.18 0.170.010.17_0.01 0.070.000.07_0.00 3.67 91.460.5091.46_0.50 0.250.020.25_0.02 0.120.010.12_0.01 2.00 54.841.3354.84_1.33 21.892.9321.89_2.93 0.880.030.88_0.03 12.00 TST 95.270.0295.27_0.02 0.180.000.18_0.00 0.070.000.07_0.00 5.67 92.270.0692.27_0.06 0.260.000.26_0.00 0.120.000.12_0.00 1.67 [rgb]0,0.75,0.161.67 62.700.3662.70_0.36 1.570.031.57_0.03 0.530.000.53_0.00 10.00 V-TST (MC=1) 94.700.1194.70_0.11 0.230.000.23_0.00 0.090.000.09_0.00 11.33 91.760.0991.76_0.09 0.310.010.31_0.01 0.130.000.13_0.00 3.67 61.690.4661.69_0.46 1.540.031.54_0.03 0.520.010.52_0.01 9.33 V-TST (MC=10) 95.240.0495.24_0.04 0.200.000.20_0.00 0.080.000.08_0.00 9.00 92.330.0592.33_0.05 0.270.000.27_0.00 0.120.000.12_0.00 1.67 [rgb]0,0.75,0.161.67 63.210.4263.21_0.42 1.410.011.41_0.01 0.500.000.50_0.00 7.33 Table 24: Comparison of ECE (↓ lower is better) for all models and all variants seen in this work for multiple choices of the number of bins. We report meanstdmean_std. The best value in each column is shown in bold. SVHN CIFAR-10 CIFAR-100 Method ECE-10 ECE-15 ECE-100 ECE-500 ECE-10 ECE-15 ECE-100 ECE-500 ECE-10 ECE-15 ECE-100 ECE-500 Baseline 3.630.153.63_0.15 3.630.153.63_0.15 3.670.163.67_0.16 3.830.153.83_0.15 6.700.416.70_0.41 6.700.406.70_0.40 6.800.376.80_0.37 7.210.347.21_0.34 22.900.8822.90_0.88 22.900.8822.90_0.88 22.910.8822.91_0.88 23.120.8923.12_0.89 COG-FC 1.070.101.07_0.10 1.080.091.08_0.09 1.270.081.27_0.08 1.890.081.89_0.08 4.250.334.25_0.33 4.220.344.22_0.34 4.550.284.55_0.28 5.620.265.62_0.26 7.821.497.82_1.49 7.821.497.82_1.49 8.011.468.01_1.46 9.451.199.45_1.19 COG+LS 7.460.147.46_0.14 7.480.147.48_0.14 7.700.147.70_0.14 8.290.148.29_0.14 4.160.124.16_0.12 4.370.124.37_0.12 5.290.125.29_0.12 6.930.096.93_0.09 3.690.403.69_0.40 4.670.414.67_0.41 5.620.335.62_0.33 7.900.387.90_0.38 COG-TS 1.000.071.00_0.07 1.010.081.01_0.08 1.230.111.23_0.11 1.880.081.88_0.08 4.600.174.60_0.17 4.610.184.61_0.18 4.890.134.89_0.13 5.940.145.94_0.14 9.841.339.84_1.33 9.841.339.84_1.33 9.871.339.87_1.33 10.871.1710.87_1.17 COG-MS 1.240.121.24_0.12 1.260.121.26_0.12 1.420.141.42_0.14 1.930.111.93_0.11 5.170.275.17_0.27 5.190.275.19_0.27 5.390.325.39_0.32 6.330.326.33_0.32 11.821.0411.82_1.04 11.821.0411.82_1.04 11.901.0111.90_1.01 12.530.8712.53_0.87 COG-TST 0.970.160.97_0.16 1.000.161.00_0.16 1.220.161.22_0.16 1.830.141.83_0.14 4.690.474.69_0.47 4.690.464.69_0.46 4.970.384.97_0.38 5.950.315.95_0.31 1.130.401.13_0.40 1.230.591.23_0.59 2.950.222.95_0.22 6.680.266.68_0.26 Label Smoothing 7.010.437.01_0.43 7.000.287.00_0.28 7.390.257.39_0.25 7.770.307.77_0.30 5.400.875.40_0.87 5.600.275.60_0.27 6.430.156.43_0.15 7.300.097.30_0.09 3.910.593.91_0.59 4.940.294.94_0.29 5.730.385.73_0.38 7.770.277.77_0.27 Temp. Scaling 2.530.122.53_0.12 2.530.122.53_0.12 2.590.122.59_0.12 2.890.112.89_0.11 4.760.294.76_0.29 4.820.234.82_0.23 4.970.214.97_0.21 5.640.275.64_0.27 15.640.6815.64_0.68 15.640.6815.64_0.68 15.710.6715.71_0.67 16.280.6116.28_0.61 Matrix Scaling 0.700.100.70_0.10 0.750.110.75_0.11 1.050.131.05_0.13 1.830.101.83_0.10 0.980.260.98_0.26 1.020.271.02_0.27 1.740.211.74_0.21 3.530.193.53_0.19 43.271.6143.27_1.61 43.271.6143.27_1.61 43.291.6243.29_1.62 43.401.6343.40_1.63 TST 1.480.021.48_0.02 1.500.021.50_0.02 1.670.031.67_0.03 2.190.032.19_0.03 3.130.053.13_0.05 3.140.063.14_0.06 3.400.073.40_0.07 4.390.084.39_0.08 14.861.0714.86_1.07 14.861.0714.86_1.07 14.931.0914.93_1.09 15.391.0315.39_1.03 V-TST (MC=1) 1.370.781.37_0.78 1.380.771.38_0.77 1.770.571.77_0.57 2.450.452.45_0.45 3.270.433.27_0.43 3.280.413.28_0.41 3.550.353.55_0.35 4.540.274.54_0.27 9.792.619.79_2.61 9.822.619.82_2.61 10.072.4810.07_2.48 11.502.1011.50_2.10 V-TST (MC=10) 2.370.712.37_0.71 2.380.702.38_0.70 2.550.632.55_0.63 3.170.523.17_0.52 2.100.332.10_0.33 2.110.342.11_0.34 2.570.282.57_0.28 3.790.213.79_0.21 6.172.376.17_2.37 6.212.326.21_2.32 6.661.956.66_1.95 8.931.388.93_1.38 Table 25: Comparison of RMSCE (↓ lower is better) for all models and all variants seen in this work for multiple choices of the number of bins. We report meanstdmean_std. The best value in each column is shown in bold. SVHN CIFAR-10 CIFAR-100 Method RMSCE-10 RMSCE-15 RMSCE-100 RMSCE-500 RMSCE-10 RMSCE-15 RMSCE-100 RMSCE-500 RMSCE-10 RMSCE-15 RMSCE-100 RMSCE-500 Baseline 5.250.275.25_0.27 5.890.275.89_0.27 7.680.187.68_0.18 9.340.129.34_0.12 8.350.618.35_0.61 8.910.558.91_0.55 11.500.4211.50_0.42 13.940.4313.94_0.43 24.340.9124.34_0.91 25.020.9825.02_0.98 27.400.9027.40_0.90 29.240.8429.24_0.84 COG-FC 1.580.181.58_0.18 1.760.201.76_0.20 2.870.222.87_0.22 5.480.345.48_0.34 5.470.505.47_0.50 5.930.565.93_0.56 7.520.447.52_0.44 10.640.4810.64_0.48 8.781.678.78_1.67 8.991.788.99_1.78 9.971.719.97_1.71 13.081.4213.08_1.42 COG+LS 7.560.147.56_0.14 7.620.147.62_0.14 7.970.177.97_0.17 9.130.189.13_0.18 4.480.084.48_0.08 4.760.114.76_0.11 6.090.216.09_0.21 9.400.259.40_0.25 5.010.305.01_0.30 5.730.455.73_0.45 7.180.537.18_0.53 11.350.3711.35_0.37 COG-TS 1.640.101.64_0.10 1.820.131.82_0.13 2.920.282.92_0.28 5.480.295.48_0.29 5.790.285.79_0.28 6.250.316.25_0.31 7.860.177.86_0.17 11.140.3411.14_0.34 10.931.4710.93_1.47 11.121.4811.12_1.48 11.931.5011.93_1.50 14.541.3814.54_1.38 COG-MS 1.990.141.99_0.14 2.270.192.27_0.19 3.290.243.29_0.24 5.690.275.69_0.27 6.480.506.48_0.50 6.910.346.91_0.34 8.460.588.46_0.58 11.420.5811.42_0.58 13.121.1513.12_1.15 13.311.2213.31_1.22 14.211.2314.21_1.23 16.430.9916.43_0.99 COG-TST 1.580.251.58_0.25 1.770.351.77_0.35 2.920.362.92_0.36 5.420.345.42_0.34 5.830.605.83_0.60 6.320.636.32_0.63 7.730.697.73_0.69 10.650.5310.65_0.53 1.480.641.48_0.64 1.680.781.68_0.78 4.030.294.03_0.29 9.250.259.25_0.25 Label Smoothing 7.190.347.19_0.34 7.180.297.18_0.29 7.840.257.84_0.25 9.090.309.09_0.30 6.340.586.34_0.58 6.890.376.89_0.37 8.230.268.23_0.26 10.860.2310.86_0.23 5.250.565.25_0.56 6.090.696.09_0.69 7.390.587.39_0.58 11.140.4211.14_0.42 Temp. Scaling 4.100.164.10_0.16 4.490.174.49_0.17 5.500.245.50_0.24 7.250.237.25_0.23 6.600.336.60_0.33 7.100.247.10_0.24 8.860.328.86_0.32 11.650.2811.65_0.28 17.230.7517.23_0.75 17.620.7517.62_0.75 19.230.5519.23_0.55 21.180.5421.18_0.54 Matrix Scaling 1.090.091.09_0.09 1.320.151.32_0.15 2.470.092.47_0.09 5.060.125.06_0.12 1.830.361.83_0.36 1.930.381.93_0.38 4.050.504.05_0.50 8.530.478.53_0.47 43.401.5943.40_1.59 43.501.5743.50_1.57 44.031.5044.03_1.50 44.801.3944.80_1.39 TST 2.310.062.31_0.06 2.570.052.57_0.05 3.560.133.56_0.13 5.990.165.99_0.16 4.610.144.61_0.14 4.840.184.84_0.18 6.480.176.48_0.17 10.010.1910.01_0.19 16.041.1316.04_1.13 16.381.1516.38_1.15 17.451.2117.45_1.21 19.361.1119.36_1.11 V-TST (MC=1) 2.221.082.22_1.08 2.371.012.37_1.01 3.410.733.41_0.73 5.650.535.65_0.53 4.550.684.55_0.68 4.770.654.77_0.65 6.260.566.26_0.56 9.910.349.91_0.34 10.802.8410.80_2.84 11.072.9311.07_2.93 12.102.8812.10_2.88 14.882.4114.88_2.41 V-TST (MC=10) 2.860.972.86_0.97 2.990.972.99_0.97 3.880.773.88_0.77 6.060.516.06_0.51 3.320.493.32_0.49 3.460.583.46_0.58 5.140.435.14_0.43 9.240.379.24_0.37 6.902.316.90_2.31 7.092.347.09_2.34 8.292.128.29_2.12 12.081.4912.08_1.49 Table 26: Comparison of MCE (↓ lower is better) for all models and all variants seen in this work for multiple choices of the number of bins. We report meanstdmean_std. The best value in each column is shown in bold. SVHN CIFAR-10 CIFAR-100 Method MCE-10 MCE-15 MCE-100 MCE-500 MCE-10 MCE-15 MCE-100 MCE-500 MCE-10 MCE-15 MCE-100 MCE-500 Baseline 40.9118.1840.91_18.18 35.913.5335.91_3.53 64.665.0964.66_5.09 86.395.0786.39_5.07 34.741.8734.74_1.87 37.274.8237.27_4.82 62.854.9862.85_4.98 88.291.9488.29_1.94 46.891.1846.89_1.18 50.502.5950.50_2.59 63.1712.1763.17_12.17 81.555.5381.55_5.53 COG-FC 29.4528.9929.45_28.99 31.6327.4331.63_27.43 50.3224.9350.32_24.93 76.262.5076.26_2.50 15.821.9015.82_1.90 19.943.7919.94_3.79 64.6615.1064.66_15.10 73.022.2573.02_2.25 13.843.0913.84_3.09 14.603.6414.60_3.64 23.542.0223.54_2.02 85.112.4585.11_2.45 COG+LS 28.7731.4328.77_31.43 30.1730.5330.17_30.53 54.8925.2254.89_25.22 81.899.1081.89_9.10 13.486.7313.48_6.73 17.644.1017.64_4.10 69.2120.0869.21_20.08 77.801.2377.80_1.23 11.972.4311.97_2.43 12.482.2112.48_2.21 40.6132.2540.61_32.25 74.4520.8274.45_20.82 COG-TS 46.9331.6346.93_31.63 56.6234.7056.62_34.70 74.0215.1174.02_15.11 80.112.9580.11_2.95 20.196.5320.19_6.53 32.2413.4732.24_13.47 61.0722.0161.07_22.01 74.284.2174.28_4.21 17.482.9217.48_2.92 18.713.0418.71_3.04 26.675.2026.67_5.20 71.2819.9571.28_19.95 COG-MS 19.208.4419.20_8.44 26.3113.5726.31_13.57 60.6826.6960.68_26.69 78.794.6578.79_4.65 16.120.8416.12_0.84 36.0022.6836.00_22.68 67.6017.0367.60_17.03 74.532.9074.53_2.90 20.772.3720.77_2.37 21.352.6621.35_2.66 43.6224.0943.62_24.09 83.563.1183.56_3.11 COG-TST 29.4129.5329.41_29.53 29.3529.5929.35_29.59 62.5623.5162.56_23.51 77.912.7577.91_2.75 18.005.6218.00_5.62 26.744.3126.74_4.31 62.9014.3062.90_14.30 72.381.9372.38_1.93 4.502.224.50_2.22 5.461.555.46_1.55 13.960.9313.96_0.93 62.0527.6362.05_27.63 Label Smoothing 16.081.3616.08_1.36 31.1524.5731.15_24.57 74.364.7374.36_4.73 84.329.6384.32_9.63 26.122.1926.12_2.19 25.583.2625.58_3.26 61.8623.7361.86_23.73 87.2910.2787.29_10.27 12.722.5012.72_2.50 14.581.4514.58_1.45 29.0810.9629.08_10.96 94.471.2494.47_1.24 Temp. Scaling 22.744.8922.74_4.89 24.411.4624.41_1.46 60.1217.9460.12_17.94 74.477.2674.47_7.26 24.676.8624.67_6.86 24.681.0324.68_1.03 64.359.4964.35_9.49 81.211.8081.21_1.80 43.3027.0943.30_27.09 31.840.5931.84_0.59 52.0822.2352.08_22.23 85.073.8985.07_3.89 Matrix Scaling 13.047.5913.04_7.59 12.608.1712.60_8.17 48.1124.2248.11_24.22 75.642.9075.64_2.90 11.098.8311.09_8.83 19.197.3819.19_7.38 57.0017.4357.00_17.43 72.693.1472.69_3.14 61.142.4161.14_2.41 65.992.2765.99_2.27 80.292.3580.29_2.35 95.262.0295.26_2.02 TST 44.3934.1244.39_34.12 45.2933.0145.29_33.01 79.531.8679.53_1.86 79.741.7279.74_1.72 18.694.3618.69_4.36 20.844.0520.84_4.05 44.8616.7544.86_16.75 76.603.0476.60_3.04 25.412.2725.41_2.27 38.5327.1438.53_27.14 36.974.4636.97_4.46 78.6412.8978.64_12.89 V-TST (MC=1) 17.313.8217.31_3.82 17.483.4417.48_3.44 39.698.9639.69_8.96 77.452.5677.45_2.56 16.514.0416.51_4.04 19.325.8719.32_5.87 53.6721.7453.67_21.74 77.962.7577.96_2.75 18.184.8518.18_4.85 18.815.3018.81_5.30 32.596.3332.59_6.33 62.4518.0162.45_18.01 V-TST (MC=10) 15.737.7815.73_7.78 17.625.6517.62_5.65 62.6826.1862.68_26.18 80.771.8480.77_1.84 13.764.4413.76_4.44 14.184.3114.18_4.31 45.0919.3245.09_19.32 77.351.9577.35_1.95 11.732.7911.73_2.79 12.233.1812.23_3.18 24.4911.5024.49_11.50 64.9425.5164.94_25.51 Table 27: Performance in out-of-distribution detection or all models and all variants seen in this work. SVHN CIFAR-10 CIFAR-100 Method AUROC (↑)( ) FPR95 (↓)( ) Rank (↓)( ) AUROC (↑)( ) FPR95 (↓)( ) Rank (↓)( ) AUROC (↑)( ) FPR95 (↓)( ) Rank (↓)( ) Baseline 93.260.7593.26_0.75 42.673.1142.67_3.11 9.50 87.001.3687.00_1.36 71.214.4171.21_4.41 4.50 70.662.8570.66_2.85 86.801.3886.80_1.38 8.50 COG-FC 96.110.3496.11_0.34 26.302.7926.30_2.79 2.50 85.111.1385.11_1.13 77.612.5977.61_2.59 9.50 70.203.7870.20_3.78 84.342.2584.34_2.25 8.50 COG+LS 93.980.2793.98_0.27 25.371.7625.37_1.76 4.00 82.402.7782.40_2.77 74.754.0274.75_4.02 9.00 73.322.3073.32_2.30 81.671.6881.67_1.68 4.00 COG-TS 95.430.4895.43_0.48 31.513.4231.51_3.42 4.50 85.530.5785.53_0.57 76.661.6076.66_1.60 8.50 67.502.5967.50_2.59 87.761.5187.76_1.51 10.50 COG-MS 95.690.3195.69_0.31 29.022.1329.02_2.13 3.50 84.581.1084.58_1.10 78.713.1478.71_3.14 11.00 68.824.9468.82_4.94 88.051.7088.05_1.70 10.50 COG-TST 96.410.3896.41_0.38 24.162.8424.16_2.84 1.00 [rgb]0,0.75,0.161.00 86.421.7386.42_1.73 75.035.5275.03_5.52 7.00 73.162.2373.16_2.23 81.652.8481.65_2.84 4.50 Label Smoothing 80.741.0080.74_1.00 38.561.6738.56_1.67 9.50 67.881.5167.88_1.51 75.991.6775.99_1.67 10.50 75.691.5075.69_1.50 80.672.5780.67_2.57 1.50 [rgb]0,0.75,0.161.50 Temp. Scaling 93.810.7793.81_0.77 39.813.3839.81_3.38 8.00 87.631.3887.63_1.38 69.275.4669.27_5.46 2.50 72.342.9272.34_2.92 83.721.9383.72_1.93 6.50 Matrix Scaling 94.660.6694.66_0.66 34.413.0534.41_3.05 5.50 89.111.2089.11_1.20 64.476.4964.47_6.49 1.00 [rgb]0,0.75,0.161.00 58.621.3758.62_1.37 92.030.6192.03_0.61 12.00 TST 93.840.0493.84_0.04 40.500.3740.50_0.37 8.00 88.340.0488.34_0.04 69.750.5769.75_0.57 2.50 70.770.4270.77_0.42 83.700.3283.70_0.32 6.50 V-TST (MC=1) 88.520.8988.52_0.89 51.820.6951.82_0.69 11.50 84.660.6384.66_0.63 72.491.2472.49_1.24 7.50 73.210.3573.21_0.35 80.860.5180.86_0.51 3.50 V-TST (MC=10) 91.700.6791.70_0.67 43.130.3643.13_0.36 10.50 86.660.4886.66_0.48 69.940.8769.94_0.87 4.50 74.200.4274.20_0.42 79.590.1879.59_0.18 1.50 [rgb]0,0.75,0.161.50 Table 28: Performance under covariate shifts. Shift metrics are averaged across all considered covariate shifts. SVHN CIFAR-10 CIFAR-100 Method ACC (↑)( ) NLL (↓)( ) Brier (↓)( ) Rank (↓)( ) ACC (↑)( ) NLL (↓)( ) Brier (↓)( ) Rank (↓)( ) ACC (↑)( ) NLL (↓)( ) Brier (↓)( ) Rank (↓)( ) Baseline 44.530.9444.53_0.94 5.950.195.95_0.19 1.000.021.00_0.02 10.67 74.240.5474.24_0.54 1.990.131.99_0.13 0.450.010.45_0.01 8.33 46.501.2946.50_1.29 5.160.345.16_0.34 0.910.020.91_0.02 8.33 COG-FC 45.340.9345.34_0.93 2.340.062.34_0.06 0.810.010.81_0.01 4.00 67.420.3167.42_0.31 1.280.011.28_0.01 0.500.000.50_0.00 9.33 44.031.5344.03_1.53 2.710.112.71_0.11 0.760.020.76_0.02 4.33 COG+LS 45.481.0045.48_1.00 1.870.031.87_0.03 0.760.010.76_0.01 1.00 [rgb]0,0.75,0.161.00 67.670.3567.67_0.35 1.100.011.10_0.01 0.470.000.47_0.00 7.67 47.041.0747.04_1.07 2.590.072.59_0.07 0.710.020.71_0.02 2.00 COG-TS 44.780.6844.78_0.68 3.270.093.27_0.09 0.890.020.89_0.02 9.00 66.550.3966.55_0.39 1.920.041.92_0.04 0.550.010.55_0.01 11.67 45.392.0045.39_2.00 3.730.213.73_0.21 0.830.030.83_0.03 8.00 COG-MS 44.790.3544.79_0.35 2.530.092.53_0.09 0.830.010.83_0.01 6.00 67.221.2967.22_1.29 1.470.071.47_0.07 0.510.020.51_0.02 10.67 40.611.3140.61_1.31 2.870.142.87_0.14 0.790.020.79_0.02 7.33 COG-TST 44.790.9944.79_0.99 2.280.052.28_0.05 0.800.020.80_0.02 3.67 68.020.7368.02_0.73 1.370.031.37_0.03 0.490.010.49_0.01 8.67 38.080.6038.08_0.60 2.770.022.77_0.02 0.780.000.78_0.00 7.00 Label Smoothing 44.810.8244.81_0.82 2.180.032.18_0.03 0.870.010.87_0.01 4.33 74.651.0974.65_1.09 1.020.031.02_0.03 0.410.010.41_0.01 4.67 50.781.0550.78_1.05 2.400.052.40_0.05 0.670.010.67_0.01 1.00 [rgb]0,0.75,0.161.00 Temp. Scaling 44.530.9444.53_0.94 3.380.093.38_0.09 0.910.010.91_0.01 10.00 74.240.5474.24_0.54 1.180.061.18_0.06 0.410.010.41_0.01 6.00 46.501.2946.50_1.29 3.100.173.10_0.17 0.800.020.80_0.02 6.33 Matrix Scaling 44.820.5244.82_0.52 2.300.072.30_0.07 0.800.010.80_0.01 3.33 74.660.9674.66_0.96 0.880.050.88_0.05 0.370.010.37_0.01 2.33 33.221.4533.22_1.45 40.213.7840.21_3.78 1.300.031.30_0.03 12.00 TST 41.120.0641.12_0.06 2.600.022.60_0.02 0.880.000.88_0.00 9.33 75.560.0575.56_0.05 0.940.010.94_0.01 0.370.000.37_0.00 2.33 41.060.2241.06_0.22 3.220.093.22_0.09 0.830.010.83_0.01 8.67 V-TST (MC=1) 40.360.1340.36_0.13 2.540.122.54_0.12 0.880.020.88_0.02 9.33 74.900.0574.90_0.05 1.040.041.04_0.04 0.390.010.39_0.01 4.00 40.360.2340.36_0.23 2.860.122.86_0.12 0.790.020.79_0.02 7.33 V-TST (MC=10) 41.220.1541.22_0.15 2.200.072.20_0.07 0.820.010.82_0.01 6.00 75.710.0475.71_0.04 0.890.030.89_0.03 0.360.010.36_0.01 1.33 [rgb]0,0.75,0.161.33 41.610.2541.61_0.25 2.650.072.65_0.07 0.760.010.76_0.01 4.33 Table 29: ECE averaged across all considered covariate shifts for all models and all variants seen in this work for multiple choices of the number of bins. SVHN CIFAR-10 CIFAR-100 Method ECE-10 ECE-15 ECE-100 ECE-500 ECE-10 ECE-15 ECE-100 ECE-500 ECE-10 ECE-15 ECE-100 ECE-500 Baseline 47.060.8347.06_0.83 47.060.8347.06_0.83 47.060.8347.06_0.83 47.060.8347.06_0.83 20.890.5320.89_0.53 20.890.5320.89_0.53 20.890.5320.89_0.53 20.860.5320.86_0.53 40.891.2440.89_1.24 40.881.2440.88_1.24 40.881.2440.88_1.24 40.861.2440.86_1.24 COG-FC 30.310.9830.31_0.98 30.310.9830.31_0.98 30.300.9830.30_0.98 30.310.9830.31_0.98 17.970.2817.97_0.28 17.970.2817.97_0.28 17.960.2817.96_0.28 17.940.2817.94_0.28 20.911.3020.91_1.30 20.911.3020.91_1.30 20.901.3020.90_1.30 20.901.3020.90_1.30 COG+LS 22.990.5622.99_0.56 22.990.5622.99_0.56 22.990.5522.99_0.55 23.000.5523.00_0.55 10.020.3310.02_0.33 10.020.3310.02_0.33 10.020.3310.02_0.33 10.020.3310.02_0.33 12.782.2012.78_2.20 12.792.1712.79_2.17 12.812.1812.81_2.18 12.822.1712.82_2.17 COG-TS 38.110.9438.11_0.94 38.110.9438.11_0.94 38.110.9438.11_0.94 38.120.9438.12_0.94 23.930.3223.93_0.32 23.930.3223.93_0.32 23.930.3223.93_0.32 23.910.3223.91_0.32 32.181.1032.18_1.10 32.171.1032.17_1.10 32.161.1032.16_1.10 32.161.1032.16_1.10 COG-MS 32.441.2032.44_1.20 32.441.2032.44_1.20 32.441.2032.44_1.20 32.451.1932.45_1.19 19.020.8719.02_0.87 19.020.8719.02_0.87 19.010.8719.01_0.87 19.000.8719.00_0.87 20.491.9520.49_1.95 20.491.9520.49_1.95 20.491.9520.49_1.95 20.491.9520.49_1.95 COG-TST 28.251.3328.25_1.33 28.251.3328.25_1.33 28.251.3328.25_1.33 28.261.3428.26_1.34 18.000.2118.00_0.21 17.990.2117.99_0.21 17.980.2117.98_0.21 17.970.2117.97_0.21 12.520.6212.52_0.62 12.520.6212.52_0.62 12.520.6212.52_0.62 12.520.6212.52_0.62 Label Smoothing 34.250.5234.25_0.52 34.250.5234.25_0.52 34.250.5234.25_0.52 34.260.5234.26_0.52 11.100.6011.10_0.60 11.080.6011.08_0.60 11.190.6111.19_0.61 11.190.6111.19_0.61 11.571.0111.57_1.01 11.590.9911.59_0.99 11.601.0011.60_1.00 11.610.9911.61_0.99 Temp. Scaling 39.690.8339.69_0.83 39.690.8339.69_0.83 39.690.8339.69_0.83 39.700.8339.70_0.83 16.510.5816.51_0.58 16.510.5816.51_0.58 16.500.5816.50_0.58 16.480.5816.48_0.58 29.351.2529.35_1.25 29.351.2529.35_1.25 29.341.2529.34_1.25 29.331.2529.33_1.25 Matrix Scaling 28.090.8228.09_0.82 28.090.8228.09_0.82 28.090.8228.09_0.82 28.100.8128.10_0.81 9.810.819.81_0.81 9.810.819.81_0.81 9.790.829.79_0.82 9.780.819.78_0.81 64.471.7264.47_1.72 64.461.7264.46_1.72 64.451.7264.45_1.72 64.431.7264.43_1.72 TST 34.830.0834.83_0.08 34.830.0834.83_0.08 34.830.0834.83_0.08 34.830.0834.83_0.08 12.750.1212.75_0.12 12.750.1212.75_0.12 12.740.1212.74_0.12 12.720.1212.72_0.12 27.521.5727.52_1.57 27.521.5727.52_1.57 27.511.5727.51_1.57 27.511.5727.51_1.57 V-TST (MC=1) 32.292.1632.29_2.16 32.292.1632.29_2.16 32.292.1632.29_2.16 32.292.1632.29_2.16 13.280.9513.28_0.95 13.270.9513.27_0.95 13.250.9513.25_0.95 13.220.9413.22_0.94 19.753.5019.75_3.50 19.753.5019.75_3.50 19.753.5019.75_3.50 19.753.5019.75_3.50 V-TST (MC=10) 27.391.8227.39_1.82 27.391.8227.39_1.82 27.391.8227.39_1.82 27.391.8227.39_1.82 10.561.0110.56_1.01 10.561.0110.56_1.01 10.521.0110.52_1.01 10.501.0010.50_1.00 15.572.9615.57_2.96 15.572.9615.57_2.96 15.572.9615.57_2.96 15.572.9615.57_2.96 Table 30: RMSCE averaged across all considered covariate shifts for all models and all variants seen in this work for multiple choices of the number of bins. SVHN CIFAR-10 CIFAR-100 Method RMSCE-10 RMSCE-15 RMSCE-100 RMSCE-500 RMSCE-10 RMSCE-15 RMSCE-100 RMSCE-500 RMSCE-10 RMSCE-15 RMSCE-100 RMSCE-500 Baseline 47.450.8347.45_0.83 47.590.8347.59_0.83 48.370.8348.37_0.83 49.010.8649.01_0.86 21.870.6221.87_0.62 22.350.6522.35_0.65 24.440.8124.44_0.81 25.630.9025.63_0.90 41.561.2041.56_1.20 41.871.1841.87_1.18 43.271.1143.27_1.11 44.061.0944.06_1.09 COG-FC 31.401.0231.40_1.02 31.661.0131.66_1.01 32.161.0732.16_1.07 32.251.0632.25_1.06 19.190.3619.19_0.36 19.650.3719.65_0.37 20.750.4320.75_0.43 20.930.4320.93_0.43 22.071.3422.07_1.34 22.261.3622.26_1.36 22.591.4522.59_1.45 22.631.4622.63_1.46 COG+LS 24.110.5624.11_0.56 24.760.6524.76_0.65 25.070.6325.07_0.63 25.200.6425.20_0.64 11.890.2711.89_0.27 11.790.2611.79_0.26 12.090.2712.09_0.27 12.150.2712.15_0.27 14.942.2614.94_2.26 15.372.3415.37_2.34 15.542.3515.54_2.35 15.572.3515.57_2.35 COG-TS 38.940.9238.94_0.92 39.210.9439.21_0.94 40.270.9840.27_0.98 40.711.0240.71_1.02 24.840.3524.84_0.35 25.290.3725.29_0.37 27.010.4427.01_0.44 27.740.4427.74_0.44 33.181.0633.18_1.06 33.521.0533.52_1.05 34.541.0234.54_1.02 34.811.0334.81_1.03 COG-MS 33.511.1833.51_1.18 33.791.1833.79_1.18 34.391.2234.39_1.22 34.491.2234.49_1.22 20.180.8320.18_0.83 20.630.8320.63_0.83 21.740.9221.74_0.92 22.030.9722.03_0.97 21.672.0021.67_2.00 21.852.0121.85_2.01 22.162.0522.16_2.05 22.192.0622.19_2.06 COG-TST 29.511.3529.51_1.35 29.751.3529.75_1.35 30.201.4430.20_1.44 30.301.4530.30_1.45 19.190.1319.19_0.13 19.620.1019.62_0.10 20.660.1120.66_0.11 20.910.1420.91_0.14 13.440.7713.44_0.77 13.510.7813.51_0.78 13.570.7913.57_0.79 13.600.7913.60_0.79 Label Smoothing 35.910.5035.91_0.50 35.660.5535.66_0.55 36.750.5736.75_0.57 36.980.6436.98_0.64 14.161.2614.16_1.26 14.090.7114.09_0.71 15.350.9415.35_0.94 15.570.9415.57_0.94 13.920.9613.92_0.96 14.430.9114.43_0.91 14.620.9114.62_0.91 14.660.9114.66_0.91 Temp. Scaling 40.460.8440.46_0.84 40.680.8540.68_0.85 41.570.9441.57_0.94 41.940.9941.94_0.99 17.990.7517.99_0.75 18.560.8018.56_0.80 20.240.9420.24_0.94 20.700.9720.70_0.97 30.591.2130.59_1.21 30.941.2030.94_1.20 31.921.2131.92_1.21 32.191.2332.19_1.23 Matrix Scaling 29.230.8329.23_0.83 29.390.8429.39_0.84 29.560.8329.56_0.83 29.650.8329.65_0.83 11.200.9911.20_0.99 11.471.0311.47_1.03 11.861.0711.86_1.07 11.931.0611.93_1.06 64.571.7164.57_1.71 64.581.7164.58_1.71 64.701.6864.70_1.68 64.821.6564.82_1.65 TST 36.160.0836.16_0.08 36.500.0836.50_0.08 37.390.1037.39_0.10 37.580.1137.58_0.11 14.340.1314.34_0.13 14.790.1414.79_0.14 15.940.1615.94_0.16 16.160.1616.16_0.16 28.661.5728.66_1.57 28.901.6028.90_1.60 29.441.6929.44_1.69 29.521.7229.52_1.72 V-TST (MC=1) 33.572.0933.57_2.09 33.762.1133.76_2.11 34.042.1734.04_2.17 34.122.1734.12_2.17 14.640.9514.64_0.95 15.050.9715.05_0.97 15.891.0615.89_1.06 16.011.0716.01_1.07 21.123.5621.12_3.56 21.283.6021.28_3.60 21.573.7321.57_3.73 21.623.7421.62_3.74 V-TST (MC=10) 28.841.8328.84_1.83 29.011.8529.01_1.85 29.181.8929.18_1.89 29.261.8929.26_1.89 11.971.0311.97_1.03 12.331.0712.33_1.07 12.931.1812.93_1.18 13.011.1913.01_1.19 16.663.0616.66_3.06 16.783.0916.78_3.09 16.963.1716.96_3.17 17.003.1717.00_3.17 Table 31: MCE averaged across all considered covariate shifts for all models and all variants seen in this work for multiple choices of the number of bins. SVHN CIFAR-10 CIFAR-100 Method MCE-10 MCE-15 MCE-100 MCE-500 MCE-10 MCE-15 MCE-100 MCE-500 MCE-10 MCE-15 MCE-100 MCE-500 Baseline 60.580.7960.58_0.79 64.160.7264.16_0.72 68.201.4668.20_1.46 74.200.6174.20_0.61 44.612.0044.61_2.00 47.011.8247.01_1.82 60.0015.7160.00_15.71 76.092.6276.09_2.62 59.260.7359.26_0.73 61.890.8661.89_0.86 68.2910.6068.29_10.60 69.419.9769.41_9.97 COG-FC 45.301.8145.30_1.81 45.531.4745.53_1.47 47.611.8847.61_1.88 51.981.8551.98_1.85 31.980.9731.98_0.97 32.050.8032.05_0.80 32.980.9732.98_0.97 82.211.3082.21_1.30 31.842.2231.84_2.22 32.162.2132.16_2.21 33.022.1833.02_2.18 46.7726.2446.77_26.24 COG+LS 36.841.3636.84_1.36 41.112.3841.11_2.38 81.6518.0781.65_18.07 95.966.0895.96_6.08 19.740.5019.74_0.50 19.880.6119.88_0.61 23.766.1123.76_6.11 59.5832.8059.58_32.80 27.123.5727.12_3.57 27.443.9727.44_3.97 49.3229.1949.32_29.19 67.2730.0267.27_30.02 COG-TS 54.031.2554.03_1.25 56.261.5056.26_1.50 58.001.7358.00_1.73 69.8412.1869.84_12.18 49.5917.7849.59_17.78 50.8017.1050.80_17.10 66.0919.9166.09_19.91 79.751.8179.75_1.81 47.661.0347.66_1.03 47.981.2847.98_1.28 48.841.0448.84_1.04 50.201.2450.20_1.24 COG-MS 48.161.4848.16_1.48 48.681.5648.68_1.56 57.8215.4057.82_15.40 72.4915.3272.49_15.32 33.431.3933.43_1.39 33.611.6633.61_1.66 34.621.6934.62_1.69 54.1823.8954.18_23.89 31.792.9831.79_2.98 43.5228.0043.52_28.00 34.015.5234.01_5.52 48.2325.6548.23_25.65 COG-TST 43.541.8143.54_1.81 43.892.1543.89_2.15 46.102.0646.10_2.06 63.6919.3163.69_19.31 32.060.6232.06_0.62 32.020.5632.02_0.56 33.260.4733.26_0.47 53.9824.4953.98_24.49 20.161.8520.16_1.85 20.331.9220.33_1.92 21.172.0521.17_2.05 33.4111.2833.41_11.28 Label Smoothing 51.631.0451.63_1.04 61.883.9861.88_3.98 84.7811.9484.78_11.94 96.780.4596.78_0.45 31.250.7531.25_0.75 36.425.4836.42_5.48 51.9919.6651.99_19.66 84.356.5684.35_6.56 26.351.2226.35_1.22 26.491.1026.49_1.10 53.9728.1653.97_28.16 79.2226.9579.22_26.95 Temp. Scaling 54.511.4854.51_1.48 56.901.5056.90_1.50 58.982.1358.98_2.13 76.128.1376.12_8.13 35.062.0235.06_2.02 35.441.8835.44_1.88 38.765.6338.76_5.63 54.6423.9354.64_23.93 46.101.2446.10_1.24 46.351.5246.35_1.52 46.991.3646.99_1.36 57.4219.5257.42_19.52 Matrix Scaling 40.971.3740.97_1.37 41.071.2241.07_1.22 59.9422.5559.94_22.55 69.9620.0769.96_20.07 19.131.7919.13_1.79 19.401.7819.40_1.78 23.685.5423.68_5.54 67.9119.6267.91_19.62 70.040.8670.04_0.86 74.740.8174.74_0.81 82.001.1482.00_1.14 83.051.1683.05_1.16 TST 53.190.1553.19_0.15 54.670.2454.67_0.24 55.880.4055.88_0.40 59.700.7759.70_0.77 27.570.3027.57_0.30 27.510.3327.51_0.33 40.3823.9340.38_23.93 55.9025.8355.90_25.83 40.962.6040.96_2.60 41.122.5141.12_2.51 41.892.5741.89_2.57 43.452.5843.45_2.58 V-TST (MC=1) 47.332.8847.33_2.88 47.852.5847.85_2.58 57.2016.6357.20_16.63 66.5317.0166.53_17.01 27.682.2127.68_2.21 27.772.1827.77_2.18 33.139.6833.13_9.68 62.6628.7062.66_28.70 31.655.0731.65_5.07 31.855.1231.85_5.12 33.034.8833.03_4.88 34.874.9834.87_4.98 V-TST (MC=10) 42.392.7042.39_2.70 42.982.7142.98_2.71 44.852.5144.85_2.51 64.2019.1764.20_19.17 22.252.3622.25_2.36 22.462.0922.46_2.09 38.1026.3338.10_26.33 61.5330.6861.53_30.68 24.664.3124.66_4.31 24.924.5024.92_4.50 25.914.5325.91_4.53 28.963.6328.96_3.63

Instant research discovery