Paper deep dive
Adversarial vulnerability of powerful near out-of-distribution detection
Stanislav Fort
Models: CLIP, R50+ViT-L32, ViT-L_16
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/12/2026, 6:18:01 PM
Summary
This paper investigates the adversarial vulnerability of state-of-the-art out-of-distribution (OOD) detection methods, including Mahalanobis distance, Relative Mahalanobis distance, Maximum of Softmax Probabilities (MSP), and CLIP-based zero-shot detection. The author demonstrates that even powerful models are highly susceptible to targeted adversarial perturbations that can flip OOD assignments. The study finds that the Relative Mahalanobis distance and the use of ensembles of OOD detectors provide the best adversarial robustness.
Entities (7)
Relation Signals (3)
Relative Mahalanobis distance → exhibitshigherrobustnessthan → Mahalanobis distance
confidence 95% · the Relative Mahalanobis distance has the highest adversarial robustness by a significant margin
Ensembles of OOD detectors → improvesrobustnessof → OOD detection
confidence 95% · The largest positive effect we see comes from the use of ensembles of several OOD detectors
CLIP → vulnerableto → Adversarial Attacks
confidence 95% · CLIP is by far the least robust technique we studied in this paper
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:There has been a significant progress in detecting out-of-distribution (OOD) inputs in neural networks recently, primarily due to the use of large models pretrained on large datasets, and an emerging use of multi-modality. We show a severe adversarial vulnerability of even the strongest current OOD detection techniques. With a small, targeted perturbation to the input pixels, we can change the image assignment from an in-distribution to an out-distribution, and vice versa, easily. In particular, we demonstrate severe adversarial vulnerability on the challenging near OOD CIFAR-100 vs CIFAR-10 task, as well as on the far OOD CIFAR-100 vs SVHN. We study the adversarial robustness of several post-processing techniques, including the simple baseline of Maximum of Softmax Probabilities (MSP), the Mahalanobis distance, and the newly proposed \textit{Relative} Mahalanobis distance. By comparing the loss of OOD detection performance at various perturbation strengths, we demonstrate the beneficial effect of using ensembles of OOD detectors, and the use of the \textit{Relative} Mahalanobis distance over other post-processing methods. In addition, we show that even strong zero-shot OOD detection using CLIP and multi-modality suffers from a severe lack of adversarial robustness as well. Our code is available at this https URL
Tags
Links
Trouble viewing inline? Open PDF directly →
Full Text
37,287 characters extracted from source content.
Expand or collapse full text
Adversarial vulnerability of powerful near out-of-distribution detection Stanislav Fort 1 Abstract There has been a significant progress in de- tecting out-of-distribution (OOD) inputs in neu- ral networks recently, primarily due to the use of large models pretrained on large datasets, and an emerging use of multi-modality.We show a severe adversarial vulnerability of even the strongest current OOD detection techniques. With a small, targeted perturbation to the input pixels, we can change the image assignment from an in-distribution to an out-distribution, and vice versa, easily. In particular, we demonstrate se- vere adversarial vulnerability on the challeng- ing near OOD CIFAR-100 vs CIFAR-10 task, as well as on the far OOD CIFAR-100 vs SVHN. We study the adversarial robustness of several post-processing techniques, including the sim- ple baseline of Maximum of Softmax Probabil- ities (MSP), the Mahalanobis distance, and the newly proposedRelativeMahalanobis distance. By comparing the loss of OOD detection per- formance at various perturbation strengths, we demonstrate the beneficial effect of using en- sembles of OOD detectors, and the use of the RelativeMahalanobis distance over other post- processing methods. In addition, we show that even strong zero-shot OOD detection using CLIP and multi-modality suffers from a severe lack of adversarial robustness as well. Our code is avail- able on GitHub. 1 Introduction The recent success of deep neural networks has led to their increasing deployment in high-stakes, safety critical appli- cations such as health care [1; 2], where models are re- quired to be not only accurate but also robust to distribu- tion shift. [3] Neural networks often assign high confidence to inputs that are misclassified, or even do not come from the distribution they were trained on at all [4; 5]. Reliable 1 Stanford University.Correspondence to:Stanislav Fort <sfort1@stanford.edu>. out-of-distribution (OOD) detection remains a significant challenge. Improving OOD detection has seen progress by training generative models [6; 7; 2; 8], and modifying objective and loss functions [9]. Exposure to a number of OOD samples during training has also lead to improvements [10]. Recently, large models (such as the Vision Transformer [11]) pre-trained on large datasets (such as ImageNet21k [12]) produced sufficiently high-quality image embeddings that allowed us to close the gap to human performance in many challenging near-OOD tasks in vision (such as distin- guishing CIFAR-100 from CIFAR-10) 1 , as well as to make significant progress in genomics. [13] Figure 1: A small adversarial perturbation to the pixels of an in-distribution image (CIFAR-100) changes its out- of-distribution (OOD) score from≈1,000 (around the mode of the in-distribution) to a confident out-distribution (CIFAR-10) region at≈2,800 even for a state-of-the-art near-OOD detection method using a large ViT.ε= 10 −4 and the attack used is the Fast Gradient Sign Method ap- plied to the Mahalanobis distance score for a ViT-L 16 as used in [13]. The unperturbed CIFAR-100→CIFAR-10 AUROC for this model is 97.98%. A Mahalanobis distance (MD) based method [14] is a sim- ple approach for post-processing embedding vectors com- ing from a neural network for OOD detection. Some of its common failure modes have been improved upon by the in- troduction of theRelativeMahalanobis Distance (RMD) in 1 https://paperswithcode.com/sota/ out-of-distribution-detection-on-cifar-100-vs arXiv:2201.07012v1 [cs.LG] 18 Jan 2022 Adversarial vulnerability of powerful near out-of-distribution detection [15], generally improving performance while being more agnostic to hyperparameter choice. Mahalanobis distance based methods are good at detect- ingfarOOD samples – for example CIFAR-10 vs SVHN, where the samples are distinct both in their surface-level style as well as in semantics.NearOOD samples – for example CIFAR-100 vs CIFAR-10, where samples are su- perficially similar and differ only in their semantic content – have remained a challenge until the widespread use of large models and pre-training [13], and multi-modality (for example the use of CLIP [16] in [13] for zero-shot class- name-only exposure OOD detection). The question of adversarial examples is usually framed in the classification setup, where an adversarial perturbation leads to a confident class change [17]. [18] show that OOD detection systems are also vulnerable to such attacks, and propose a robust training algorithm for counteracting it. [19] propose a training algorithm leading to more robust OOD detection as well. Key contributions:We show empirically that currently even the most powerful and robust OOD detection systems based on large models and massive data are severely vul- nerable to targeted adversarial attacks. We demonstrate that this is the case for different post-processing techniques, in- cluding the baseline Max of Softmax Probabilities (MSP), as well as the more advanced Mahalanobis distance. The zero-shot multi-modal approach using CLIP suffers from an even more acute vulnerability to such attacks. We show that working with lower resolution images increases OOD adversarial robustness. The largest positive effect we see comes from the use of ensembles of several OOD detec- tors, and the use of theRelativeMahalanobis distance. We demonstrate that these two interventions can be success- fully combined as well, making the detection system more adversarially robust as well as improving its OOD detec- tion performance in general. 2 Methods In this section, we describe how to get adversarial examples to OOD detection algorithms and briefly review the Maha- lanobis distance andRelativeMahalanobis distance meth- ods. We also discuss the baseline Maximum of Softmax Probabilities, and the use of the multi-modal CLIP model for zero-shot OOD detection. We present a method for at- tacking ensembles of detectors we well. 2.1 Generating adversarial examples to OOD score Given an out-of-distribution scoring functionscore(x)that maps an imagexinto a floating point value characterizing its distance from the in-distribution, we can use its gradient with respect to the input, g(x) = ∂score(x ′ ) ∂x ′ x ′ =x ,(1) to gradually change the inputxto have either a higher or lower OOD score. This is exactly the same way adversarial examples, first described in [17], are typically generated. Modifications exist that change the form of the perturba- tion, for example theFast Gradient Sign Methodin [20] that usessign(g(x))instead ofg(x)as the step direction. We will primarily be using that in this paper, as it is easy to use and works well out of the box. Starting from an in-distribution image of a low score (con- fidently in-distribution), taking iterative steps x t+1 =x t +εg(x),(2) whereεis the learning rate, we move in the local direction of increasing OOD score. As shown in Figure 1, a very small perturbation to an image of acrableads to a shift from the center of the in-distribution scores to the higher end of the out-distribution scores. This turns the image from a confidently and correctly in-distribution to a con- fidently out-distribution, as judged by a well-performing detection method from [13]. 2.2 Mahalanobis distance based OOD detection The Mahalanobis distance (MD) [14] method and Relative Mahalanobis distance (RMD) [15] method use intermedi- ate features of a trained deep neural network. A frequent choice of the features are the pre-logits – the output of the second to last layer of a network, just before the classifica- tion layer. Let us indicate these features asz i =f(x i )for an inputx i . For aK-class in-distribution dataset, both methods fit Kclass-specific Gaussian distributionsN(μ k ,Σ), k= 1,2, . . . , Kto each of theKin-distribution classes using their feature vectorsz i We compute the class centroids (means) and covariance matrices as:μ k = 1 N k ∑ i:y i =k z i , fork= 1, . . . , K, andΣ= 1 N ∑ K k=1 ∑ i:y i =k (z i −μ k ) (z i −μ k ) T . No- tice that the class meansμ k are independent for each class, while we use the same covariance matrixΣfor all classes to avoid numerical issues due to under-fitting to the typically smaller than needed numbers of examples. For a test inputx ′ whose in- or out-distribution assign- ment is to be determined, we compute the Mahalanobis distances from the embedding vector of the test inputz ′ = f(x ′ )to each of theKin-distribution Gaussian distribu- tionsN(μ k ,Σ), k∈ 1, . . . , Kgiven by MD k (z ′ )we just computed. We take the minimum of the distances over all classes to be the uncertainty scoreU(x ′ )characterizing 2 Adversarial vulnerability of powerful near out-of-distribution detection how far from the in-distribution the inputx ′ is deemed to be. There the score can be seen as the extent to which the sample is OOD. The Mahalanobis distances are computed as MD k (z ′ ) = (z ′ −μ k ) T Σ −1 (z ′ −μ k ),(3) score(x ′ ) =U(x ′ ) =−min k MD k (z ′ ).(4) This confidence score is used to distinguish the in- distribution and out-distribution samples from each other. 2.3 Relative Mahalanobis Distance In [15] theRelativeMahalanobis Distance is proposed which modifies Eq. 4 by subtracting a term to make it more robust to hyperparameter choice as well as generally bet- ter at OOD detection for near-OOD tasks in vision and genomics. The approach attempts to model the shape of the in-distribution and subtract its effects from the class- conditional distances. The RMD is defined as RMD k (z ′ ) =MD k (z ′ )−MD 0 (z ′ ), where MD 0 (z ′ )indicates the Mahalanobis distance to a Gaussian distribution fitted to the whole in- distribution dataset without regard to its label structure, asN(μ 0 ,Σ 0 ), whereμ 0 = 1 N ∑ N i=1 z i andΣ 0 = 1 N ∑ N i=1 (z i −μ 0 ) (z i −μ 0 ) T . The goal is to model the background distribution. The resulting uncertainty score using RMD is then C RMD (x ′ ) =−min k RMD k (z ′ ).(5) This can be extended to more powerful generative models fit ([21; 22]) to the class-specific and full-dataset approxi- mations. [15] 2.4 Maximum of Softmax Probabilities A solid baseline for OOD detection is provided by the sim- ple approach of using the Maximum of Softmax Probabil- ities as thein-distribution score. For a classification model f(x) =pthat maps in input imagexto a vector of prob- abilitiesp, the OOD score isscore(x) = max(f(x)). For in-distribution images, for a well trained model the image will belong to one of the output classes that will likely be close to1in the probabilities vector. For an OOD sample, the model will likely be confused and will not assign as high a probability to any of the classes. This provides the rational for using this method, which proved to be a good baseline given how simple its implementation is. 2.5 Zero-shot multi-modal OOD detection using wordsto specify distributions [13] introduce a new kind of OOD detection scenario, where they use a multi-modal CLIP model [16]. CLIP produces a similarity score comparing the semantic con- tent of an image and a text, aslogit(I,T). By choosing two sets of words: in-words characterizing the semantic content of the in-distribution, and out-words, characteriz- ing the semantic content of the out-distribution, for each imageIwe can compute the in-logits for the in-words as z in i = CLIP(I,inword i ), and the out-logits for the out- wordsz out i = CLIP(I,outword i ). We construct the score the same way as in [13] as score(x) =max(CLIP(x,inword i i )(6) −max(CLIP(x,outword i i ).(7) We can modify this score the same way we do for the Ma- halanobis distance or Relative Mahalanobis distance using a gradient step with respect to the image. 2.6 Ensembling OOD detectors A simple way to improve the OOD detection capabilities of several OOD detectors is to ensemble them. For exam- ple, this is used in [13] to reach the current state-of-the-art performance on the near OOD CIFAR-100→CIFAR-10 task. The simplest technique we can use is to generate the OOD score for a particular imagexfor each of the models score i (x), and compute their average score ensemble (x) = 1 N N ∑ i=1 score i (x).(8) The likely reason for why ensembling of the OOD pre- dicted scores over several models works better than the models individually is similar to the reason for why deep ensembles work in general [5]. A loss landscape approach to that is discussed in [23]. 2.7 Attacks on model ensembles Attacking an ensemble of OOD detectors, as discussed in Section 2.6, is the same as attacking a single model. The only difference is that we replace the single model OOD scoring functionscore(x)with the ensemble scoring func- tionscore ensemble (x). 3 Experiments and Results We studied the adversarial robustness of the currently best performing methods on the near-OOD task of dis- tinguishing CIFAR-100 (in-distribution) from CIFAR-10 (out-distribution). 2 The best performing approach is an ensemble of pre-trained Vision Transformers finetuned on CIFAR-100 with the Mahalanobis distance post-processing method applied on top of their embeddings. This reaches an AUROC of 97.98% [13], as compared to a human 2 https://paperswithcode.com/sota/ out-of-distribution-detection-on-cifar-100-vs 3 Adversarial vulnerability of powerful near out-of-distribution detection benchmark of AUROC≈96.0%. The best approach not using an ensemble of detectors differs in using a single ViT only. We chose the pre-trained and finetuned ViT-L 16 3 to de- velop OOD adversarial attacks to. Its default resolution is 384×384and we used the standardtf.image.resizeto up- sample the32×32CIFAR images to it, as done in the standard ViT preprocessing pipeline. 3.1 Attacks on CIFAR-100 vs CIFAR-10 for different post-processing techniques Mahalanobis distanceWe focused on the challenging near-OOD CIFAR-100 vs CIFAR-10 task. Figure 2 shows an image of an airplane (CIFAR-10, out-distribution) be- ing adversarially modified using the Fast Gradient Sign Method to read as a confident in-distribution image do the ViT based Mahalanobis distance OOD detector. The fig- ure also shows the shift of the OOD score against the his- tograms of the in- and out-distribution test set images. This is similar to Figure 1, where the direction of change was from the in-distribution to the out-distribution. A small Figure 2: A small adversarial perturbation to the pixels of an out-distribution image (CIFAR-10) changes its out- of-distribution (OOD) score from≈2,800 (around the mode of the out-distribution) to a confident in-distribution (CIFAR-100) region at≈600 even for a state-of-the-art near-OOD detection method using a large ViT.ε= 10 −4 and the attack used is the Fast Gradient Sign Method ap- plied to the Mahalanobis distance score for a ViT-L 16 as used in [13]. The unperturbed CIFAR-100→CIFAR-10 AUROC for this model is 97.98%. change in the pixel values of the input image resulted in a large change of the OOD score assigned. Applying the same procedure to 128 test images, we were able to generate a set of perturbed out-distribution images that read as confidently in-distribution to the detector, as shown in Figure 3a as a function of theL 2 norm of the image perturbation and in Figure 3b as a function of the L ∞ norm. 3 https://github.com/google-research/vision transformer (a)(b) Figure 3: Changing the out-distribution score for a set of 128 CIFAR-10 test images (out-distribution) by applying the Fast Gradient Sign Method to the Mahalanobis OOD score based on a ViT-L 16. (a) shows the score as a func- tion of theL 2 norm of the image perturbation, while (b) shows theL ∞ norm. Relative Mahalanobis distanceUsing the proposed Rel- ative Mahalanobis distance [15], that we discuss in Sec- tion 2.3, as an OOD score, we show an equivalent effect of a small adversarial perturbation on the OOD score in Fig- ure 4. Applying this attack to 128 out-distribution images Figure 4: A small adversarial perturbation to the pixels of an out-distribution image (CIFAR-10) changes its out-of- distribution (OOD) score from≈300 (around the mode of the out-distribution) to a confident in-distribution (CIFAR- 100) region at≈-50 even for a state-of-the-art near-OOD detection method using a large ViT.ε= 10 −4 and the at- tack used is the Fast Gradient Sign Method applied to the RelativeMahalanobis distance score for a ViT-L 16 as used in [13]. The unperturbed CIFAR-100→CIFAR-10 AU- ROC for this model is 97.11%. and their gradual score change with theL 2 andL ∞ norms of the perturbation are shown in Figure 5a and Figure 5b respectively. Maximum of Softmax ProbabilitiesWe used the Max- imum of Softmax Probabilities (MSP) as a baseline post- processing method for OOD detection, as discussed in Sec- tion 2.4. Figure 6a and Figure 6b show the change in the score of 128 out-distribution test images as a function of theL 2 andL ∞ norms of the image perturbation. 4 Adversarial vulnerability of powerful near out-of-distribution detection (a)(b) Figure 5: Changing the out-distribution score for a set of 128 CIFAR-10 test images (out-distribution) by applying the Fast Gradient Sign Method to theRelativeMahalanobis OOD score based on a ViT-L 16. (a) shows the score as a function of theL 2 norm of the image perturbation, while (b) shows theL ∞ norm. (a)(b) Figure 6: Changing the out-distribution score for a set of 128 CIFAR-10 test images (out-distribution) by applying the Fast Gradient Sign Method to the Maximum of Soft- max Probabilities (MSP) score based on a ViT-L 16. (a) shows the score as a function of theL 2 norm of the image perturbation, while (b) shows theL ∞ norm. Robustness comparisonThe stronger the adversarial at- tack, the more we can change the out-distribution samples in order for them to be perceived as in-distribution by the detection system. Table 1 summarizes the loss of the AU- ROC on the CIFAR-100 vs CIFAR-10 task for the stan- dard Mahahalanobis distance, the Relative Mahalanobis distance, and the Maximum of Softmax Probabilities (com- parison baseline). Figure 7a and Figure 7b show the loss of AUROC as a function of the perturbation strength mea- sured by theirL 2 andL ∞ norms. The results in Table 1 can be read off from Figure 7b by looking atL ∞ = 1/255. The way we turned Figures 3b, 5b and 6b into the sum- mary in Figure 7b was as follows. Each image is adversar- ially modified inTsteps. Its OOD score andL ∞ pertur- bation norm change as a function ofT. We used a piece- wise linear interpolation to make an image-specific func- tionscore(L ∞ ). Then, when making Figure 7b, we sam- pled theL ∞ perturbation norms we wanted to explore, and for each computed the interpolated OOD score for each of the 128 images based on their individual linear interpola- tions. The resulting distribution of scores was then com- Table 1: The loss of AUROC on the near OOD CIFAR-100 vs CIFAR-10 task for several OOD detection approaches. The strength of the attack is fixed by theL ∞ norm of the adversarial perturbation at 1/255. All approaches use the pretrained and finetuned ViT-L 16 to generate probability outputs and embeddings. The baseline of using the Max of Softmax Probabilities (MSP) is the least robust, followed by the Standard Mahalanobis Distance. The newly pro- posed Relative Mahalanobis Distance has the highest ad- versarial robustness by a significant margin. Post-process method AUROC before AUROC ` ∞ 1/255 ∆ AUROC Max of Softmax Probs 94.28%27.48%-66.8% Maha97.98%41.33%-56.65% Relative Maha97.11%71.84%-25.27% pared to the scores of the in-distribution test set to obtain the AUROC. For theL 2 norm in Figure 7a the process was analogous, swappingL ∞ forL 2 everywhere. To compare the robustness of the standard Mahalanobis distance and theRelativeMahalanobis distance to OOD ad- versarial attacks, we used the Fast Gradient Sign Method of finding the adversary, with a learning rate of3×10 −4 (arbitrarily chosen), and ran it for 30 steps on 128 test set images of CIFAR-10 (the out-distribution). We ran the at- tack against both the Mahalanobis distance score as well as the Relative Mahalanobis distance score. For each of the images, we measured its score, itsL 2 distance from the un- perturbed image (out of √ 384×384×3≈665for fully saturated pixels in the[0,1]range), and itsL ∞ distance from the unperturbed image (out of 1). For both theL 2 and L ∞ perturbation strength norms, the Relative Mahalanobis distance is significantly more robust to OOD adversarial perturbations, retaining a higher AUROC on the near-OOD CIFAR-100 vs CIFAR-10 task at a given strength of per- turbation. This is in line with the observation of higher stability of the relative distance method [15]. The baseline method of Maximum of Softmax Probabilities (in orange) performs the worst, losing AUROC the fastest with pertur- bation strength. 3.2 Zero-shot OOD using CLIP We use the zero-shot OOD detection setup using the multi- modal CLIP model described in Section 2.5 and introduced in [13]. In Figures 7a and 7b we show that its adversar- ial robustness is lower than for other methods, including the baseline Max of Softmax Probabilities (MSP). In Ta- ble 2 we show the underlying numbers in detail. Despite its versatility and power, CLIP does not perform very well 5 Adversarial vulnerability of powerful near out-of-distribution detection (a)(b) Figure 7: AUROC of CIFAR-100 vs CIFAR-10 where the out-distribution CIFAR-10 is represented by 128 adversar- ially perturbed images to lower the Mahalanobis distance OOD score (blue) or Relative Mahalanobis distance score (red), and as a baseline the Maximum of Softmax Proba- bilities (orange). We add the CLIP zero-shot OOD detec- tion as comparison in purple. (a) shows the perturbation strength measured by itsL 2 norm, and (b) by itsL ∞ norm. Details in Table 1 and Table 2. Table 2: The loss of AUROC on the near OOD CIFAR- 100 vs CIFAR-10 task for the CLIP zero-shot method using class names atL ∞ of 1/255 perturbation strength. CLIP is by far the least robust technique we studied in this paper. Post-process method AUROC before AUROC ` ∞ 1/255 ∆ AUROC CLIP94.68%<10%a lot when under a targeted adversarial attack to its OOD ca- pabilities, underperforming even a simple post-processing baseline (albeit with very strong embeddings from a large, pretrained ViT). The change in the OOD score for 128 test set images from the out-distribution under an adversarial attack against the CLIP-based detector is shown in Figure 8a for theL 2 norm of the perturbation strength and in Figure 8b for theL ∞ norm. 3.3 Model ensembles We studied ensembles of OOD detectors, as discussed in Section 2.6. We used the standard setup using the Fast Gradient Sign Method (keeping only the sign of each el- ement of the gradient), learning rate of3×10 −4 (arbitrar- ily chosen) and ran it for 30 steps on 128 test set images of CIFAR-10 (the out-distribution). We identified two well performing models finetuned on CIFAR-100 (training set), the ViT-L 16 and R50+ViT-L 32 , both with input resolution of224×224×3. We found that OOD model ensembling: 1) improves OOD (a)(b) Figure 8: Changing the out-distribution score for a set of 128 CIFAR-10 test images (out-distribution) by applying the Fast Gradient Sign Method to the CLIP model. (a) shows the score as a function of theL 2 norm of the im- age perturbation, while (b) shows theL ∞ norm. detection AUROC, 2) makes it more robust to adversarial attacks, and 3) its benefit combines well with the benefit of using the Relative Mahalanobis distance. We show the detailed results in Table 3 and in Figure 9 and Figure 10. We look at the performance of two models individually, and perform adversarial attacks on their OOD score. We record the drop in AUROC for distinguishing the unperturbed CIFAR-100 from the adversarially perturbed CIFAR-10 at the perturbation level` ∞ = 1/255. We do the same for the ensemble of the two models. Ensembles suffer from a smaller drop in AUROC at a given perturbation level. Its benefit can be combined with the large robustness benefit of theRelativeMahalanobis distance. The AUROC as a func- Table 3: The benefit of OOD detector ensembling for ad- versarial robustness. The results are shown for the near OOD CIFAR-100→CIFAR-10 task. We evaluate two sep- arate models, and their ensemble, each for using the Ma- halanobis distance and the Relative Mahalanobis distance post-processing. Using an ensemble increases adversarial robustness, and can be combined to increase its benefit with the Relative Mahalanobis distance. Model Post- process method AUROC before AUROC ` ∞ 1/255 ∆ AUROC ViT L 16 Maha97.72%56.14%-41.58% R50+L 32 Maha96.95%54.94%-42.01% EnsembleMaha97.91%68.67%-29.24% ViT L 16 Relative96.92%69.82%-27.10% R50+L 32 Relative97.09%68.53%-28.56% EnsembleRelative97.69%78.64%-9.05% tion of the perturbation strength, both for theL 2 andL ∞ perturbation norms, is shown in Figure 9 and Figure 10. For all perturbation strengths measured by both norms, the en- 6 Adversarial vulnerability of powerful near out-of-distribution detection semble performs better than the individual models. We see Figure 9: AUROC of CIFAR-100 vs CIFAR-10 where the out-distribution CIFAR-10 is represented by 128 adversar- ially perturbed images to lower the Mahalanobis distance OOD score (left panel, blue) and Relative Mahalanobis distnace score (right panel, red). We show the perturba- tion strength measured by itsL 2 norm. The model ensem- ble (darker lines) is more robust to adversarial perturbations both for the standard and relative distance post-processing. Figure 10: AUROC of CIFAR-100 vs CIFAR-10 where the out-distribution CIFAR-10 is represented by 128 adversar- ially perturbed images to lower the Mahalanobis distance OOD score (left panel, blue) and Relative Mahalanobis distnace score (right panel, red). We show the perturbation strength measured by itsL ∞ norm. The model ensemble (darker lines) is more robust to adversarial perturbations both for the standard and relative distance post-processing. a clear benefit of OOD detector ensembling both on the un- perturbed AUROC as well as on the adversarial robustness of the resulting detector. This benefit combines well with the benefit of using the Relative Mahalanobis distance, sug- gesting that using both could be the correct strategy when deploying OOD detection systems. 3.4 The effect of image resolution The input to the Vision Transformer is either384×384 (or224×224) while the resolution of both CIFAR-10 and CIFAR-100 is32×32. To resolve that, we upsample images to the correct resolution using thetf.image.resize function prior to feeding them into the network. This means that the imagexcoming in has the high resolution required, and that the gradient~g(x) =∂score(x)/∂xwill be of the same resolution. This gives the attack many more Figure 11: AUROC of CIFAR-100 vs CIFAR-10 where the out-distribution CIFAR-10 is represented by 32 adversar- ially perturbed images to lower the Mahalanobis distance OOD score (blue) and Relative Mahalanobis distance score (red). We show the perturbation strength measured by its L 2 norm (left panel) andL ∞ norm (right panel). The lighter lines show results for images and their gradients at the original CIFAR32×32resolution, while the darker lines show the384×384resolution. The lower resolution images are harder to adversarially perturb. Table 4: The effect of image and gradient resolution on OOD robustness. Using differential image upsampling, we show that working with lower resolution images provides adversarial robustness as compared to working with high resolution even for strong near-OOD detectors. Resolution Post- process method AUROC before AUROC ` ∞ 1/255 ∆ AUROC 32×32Maha97.98%93.11%-4.87% 384×384 Maha97.98%41.33%-56.65% 32×32Relative97.11%90.13%-6.98% 384×384Relative97.11%71.84%-25.27% pixels to change and potentially exploit, plausibly leading to an easier to find adversarial example. To measure the difference between the adversarial robust- ness of low and high resolution images, we compared the attacks on the images upsampled prior to their use and gradient computation to working with the low res- olutuion images directly. For the latter case, we com- pute the image score asscore(resize(x))and its derivative as∂score(resize(x))/∂x, working directly with the small resolution image and modifying it using the small resolu- tion gradient. The results for both the standard Mahalanobis distance and the Relative Mahalanobis distance, as well as the perturba- tion strengthL 2 andL ∞ norms, are shown in Figure 11 and in Table 4. The lower resolution images are harder to perturb at a given perturbation strength, however, the ben- efit (or at least comparable performance at low strength) of the Relative Mahalanobis distance persists. 7 Adversarial vulnerability of powerful near out-of-distribution detection 3.5 Exploring far OOD CIFAR-100 vs SVHN We studied the adversarial vulnerability on another, easier, far OOD task. In particular, we looked at the CIFAR-100 (in-distribution) vs SVHN (out-distribution) [24]. We show an example of the adversarial modification in Figure 12. The very large benefit of the Relative Mahalanobis distance for adversarial robustness of the OOD classification seen for near OOD tasks, such as in Figure 7a, Figure 7b and Table 1, is not prominent or does not exist at all for this far OOD task. The results are summarized in Table 5. Table 5: A comparison of OOD adversarial robustness of the Mahalanobis and Relative Mahalanobis distances for the far OOD CIFAR-100 vs SVHN. Post- process method AUROC before AUROC ` ∞ 1/255 ∆ AUROC Maha99.40%34.47%-64.93% Relative97.19%43.22%-53.97% The loss of AUROC from the unperturbed 99.40% as a function of theL 2 andL ∞ norm of the image perturba- tion are shown in Figure 13.At the`= 1/255level Figure 12: A small adversarial perturbation to the pix- els of the out-distribution image (SVHN) changes its out- of-distribution score from≈3,000 to a confident in- distribution (CIFAR-100) region at≈1,000 even for a state-of-the-art near OOD detection method.ε= 10 −4 and the attack used is the Fast Gradient Sign Method applied to the Mahalanobis distance score for a ViT-L 16 as used in [13]. The unperturbed CIFAR-100→SVHN AUROC for this model is 99.40%. ofL ∞ perturbation the AUROC is 34.47%. At the same level with the very same adversary-generation procedure, CIFAR-100 vs CIFAR-10 (near OOD) AUROC drops to 41.33% (see Table 1 for more details). It seems that, based on this example, there is a weak evidence that far OOD tasks might be more susceptible to adversarial attacks on the OOD score. 4 Conclusion Even very powerful, near out-of-distribution detection methods based on large, pre-trained models, such as the Vi- Figure 13: AUROC of CIFAR-100 vs SVHN where the out-distribution SVHN is represented by 128 adversarially perturbed images to lower the Mahalanobis distance (blue) and Relative Mahalanobis distance (red) OOD score. We show the perturbation strength measured by itsL 2 norm (left panel) andL ∞ norm (right panel). The benefit of the Relative Mahalanobis distance to OOD adversarial robust- ness is not significant or not as prominted as for the near OOD CIFAR-100 vs CIFAR-10. sion Transformer [13] and multi-modal text-image models, such as CLIP, suffer from severe adversarial vulnerability to their OOD detection score. Well-targeted, small modifi- cations to the image pixels cause these detection systems to change their classification from confidently in-distribution to confidently out-distribution and vice versa. This might come as a surprise given the recent large improvements on near OOD tasks (such as distinguishing CIFAR-100 vs CIFAR-10) these models brought about. We show that or- thogonally to their representational robustness that we can infer from their near-OOD performance, they still suffer from a severeadversarial vulnerability. By studying the change in the OOD detectors’ AUROC as a function of adversarial perturbation strength, we show that there are easy-to-use and generally applicable approaches to partial remedying this effect:ensemblingand theRel- ative Mahalanobis ditance. The first approach is to en- semble several OOD detectors by averaging their predicted OOD score. The second approach is to use, instead of the standard Maximum of Softmax Probabilities or the more involved Mahalanobis distance post-processing technique, the newly proposedRelativeMahalanobis distance [15]. We also show that these approaches combine well together. We hope that by demonstrating this specific non-robustness of even the most powerful approaches to near OOD de- tection, more research will try to address them. We start off with proposing to use model ensembles and the Rel- ative Mahalanobis distance where possible as an easy to use and cheap fix. However, stronger mitigation techniques will likely have to be employed to meet the frequent safety- critical nature of OOD detection. 8 Adversarial vulnerability of powerful near out-of-distribution detection Acknowledgements We thank Jie Ren, Huiyi Hu, and Balaji Lakshminarayanan for useful comments and discussions. References [1] Abhijit Guha Roy, Jie Ren, Shekoofeh Azizi, Aaron Loh, Vivek Natarajan, Basil Mustafa, Nick Pawlowski, Jan Freyberg, Yuan Liu, Zach Beaver, and et al. Does your dermatology classifier know what it doesn’t know? detecting the long-tail of unseen conditions.Medical Image Analysis, 75:102274, Jan 2022. [2] Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark A. DePristo, Joshua V. Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out- of-distribution detection, 2019. [3] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man ́ e. Con- crete problems in AI safety.arXiv preprint arXiv:1606.06565, 2016. [4] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural net- works, 2017. [5] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive un- certainty estimation using deep ensembles, 2016. [6] Christopher M Bishop. Novelty Detection and Neural Network Validation.IEE Proceedings-Vision, Image and Signal processing, 141(4):217–222, 1994. [7] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan.Detecting out-of- distribution inputs to deep generative models using typicality.arXiv preprint arXiv:1906.02994, 2019. [8] Warren Morningstar, Cusuh Ham, Andrew Gallagher, Balaji Lakshminarayanan, Alex Alemi, and Joshua Dillon. Density of states estimation for out of dis- tribution detection. InAISTATS, 2021. [9] Hongjie Zhang, Ang Li, Jie Guo, and Yanwen Guo. Hybrid models for open set recognition.ECCV, 2020. [10] Dan Hendrycks, Mantas Mazeika, and Thomas G Di- etterich. Deep anomaly detection with outlier expo- sure.ICLR, 2019. [11] Alexey Dosovitskiy,Lucas Beyer,Alexander Kolesnikov,Dirk Weissenborn,Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.ICLR, 2021. [12] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses, 2021. [13] Stanislav Fort, Jie Ren, and Balaji Lakshmi- narayanan. Exploring the limits of out-of-distribution detection.arXiv preprint arXiv:2106.03004, 2021. [14] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin.A simple unified framework for detecting out-of-distribution samples and adversarial attacks. NeurIPS, 2018. [15] Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanan. A simple fix to mahalanobis distance for improving near-ood detection, 2021. [16] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning trans- ferable visual models from natural language supervi- sion, 2021. [17] ChristianSzegedy,WojciechZaremba,Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks, 2013. [18] Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Robust out-of-distribution detection for neural networks, 2020. [19] Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Atom: Robustifying out-of-distribution detection using outlier mining.Lecture Notes in Com- puter Science, page 430–445, 2021. [20] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial ex- amples, 2014. [21] George Papamakarios, Theo Pavlakou, and Iain Mur- ray. Masked autoregressive flow for density estima- tion.arXiv preprint arXiv:1705.07057, 2017. [22] GeorgePapamakarios,EricNalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan.Normalizing flows for probabilistic modeling and inference.JMLR, 2021. [23] Stanislav Fort, Huiyi Hu, and Balaji Lakshmi- narayanan. Deep ensembles: A loss landscape per- spective, 2019. [24] Ian J. Goodfellow, Yaroslav Bulatov, Julian Ibarz, Sacha Arnoud, and Vinay Shet. Multi-digit number recognition from street view imagery using deep con- volutional neural networks, 2014. 9