Paper deep dive
Measuring Robustness to Natural Distribution Shifts in Image Classification
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, Ludwig Schmidt
Models: DenseNet-121, ResNet-50, Various ImageNet classifiers (204 total spanning multiple architectures and training procedures)
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/12/2026, 7:23:14 PM
Summary
This paper presents a comprehensive meta-study evaluating the robustness of 204 ImageNet models across 213 test conditions, focusing on natural distribution shifts rather than synthetic perturbations. The authors find that current synthetic robustness measures are poor predictors of natural robustness, and that while training on larger, more diverse datasets improves robustness, it remains an open research challenge to close the performance gaps.
Entities (6)
Relation Signals (3)
ImageNet Models → evaluatedon → ImageNetV2
confidence 100% · We evaluate this model on two test sets: the “standard” test set (denoted S1) and the test set with a distribution shift (denoted S2)... ImageNetV2, a reproduction of the ImageNet test set
Synthetic robustness measures → poorlypredict → Natural distribution shift robustness
confidence 95% · We find that current robustness measures for synthetic distribution shift are at most weakly predictive for robustness on the natural distribution shifts
Training on diverse data → improves → Effective Robustness
confidence 90% · Across all of our experiments, the only intervention that improves robustness to multiple natural distribution shifts is training with a more diverse dataset.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We study how robust current ImageNet models are to distribution shifts arising from natural variations in datasets. Most research on robustness focuses on synthetic image perturbations (noise, simulated weather artifacts, adversarial examples, etc.), which leaves open how robustness on synthetic distribution shift relates to distribution shift arising in real data. Informed by an evaluation of 204 ImageNet models in 213 different test conditions, we find that there is often little to no transfer of robustness from current synthetic to natural distribution shift. Moreover, most current techniques provide no robustness to the natural distribution shifts in our testbed. The main exception is training on larger and more diverse datasets, which in multiple cases increases robustness, but is still far from closing the performance gaps. Our results indicate that distribution shifts arising in real data are currently an open research problem. We provide our testbed and data as a resource for future work at this https URL .
Tags
Links
Trouble viewing inline? Open PDF directly →
Full Text
254,680 characters extracted from source content.
Expand or collapse full text
Measuring Robustness to Natural Distribution Shifts in Image Classification Rohan TaoriAchal DaveVaishaal Shankar UC BerkeleyCMUUC Berkeley Nicholas CarliniBenjamin RechtLudwig Schmidt Google BrainUC BerkeleyUC Berkeley Abstract We study how robust current ImageNet models are to distribution shifts arising from natural variations in datasets. Most research on robustness focuses on synthetic image perturbations (noise, simulated weather artifacts, adversarial examples, etc.), which leaves open how robustness on synthetic distribution shift relates to distribution shift arising in real data. Informed by an evaluation of 204 ImageNet models in 213 different test conditions, we find that there is often little to no transfer of robustness from current synthetic to natural distribution shift. Moreover, most current techniques provide no robustness to the natural distribution shifts in our testbed. The main exception is training on larger and more diverse datasets, which in multiple cases increases robustness, but is still far from closing the performance gaps. Our results indicate that distribution shifts arising in real data are currently an open research problem. We provide our testbed and data as a resource for future work at https://modestyachts.github.io/imagenet-testbed/. 1 Introduction Reliable classification under distribution shift is still out of reach for current machine learning [65,68,91]. As a result, the research community has proposed a wide range of evaluation protocols that go beyond a single, static test set. Common examples include noise corruptions [33,38], spatial transformations [28,29], and adversarial examples [5,84]. Encouragingly, the past few years have seen substantial progress in robustness to these distribution shifts, e.g., see [13,28,34,55,57, 66,93,96,105,114,115] among many others. However, this progress comes with an important limitation: all of the aforementioned distribution shifts aresynthetic: the test examples are derived from well-characterized image modifications at the pixel level. Synthetic distribution shifts are a good starting point for experiments since they are precisely defined and easy to apply to arbitrary images. However, classifiers ultimately must be robust to distribution shifts arising naturally in the real world. These distribution shifts may include subtle changes in scene compositions, object types, lighting conditions, and many others. Importantly, these variations arenotprecisely defined because they have not been created artificially. The hope is that an ideal robust classifier is still robust to such natural distribution shifts. 1 arXiv:2007.00644v2 [cs.LG] 14 Sep 2020 In this paper, we investigate how robust current machine learning techniques are to distribution shift arising naturally from real image data without synthetic modifications. To this end, we conduct a comprehensive experimental study in the context of ImageNet [18,70]. ImageNet is a natural starting point since it has been the focus of intense research efforts over the past decade and a large number of pre-trained classification models, some with robustness interventions, are available for this task. The core of our experimental study is a testbed of 204 pre-trained ImageNet models that we evaluate in 213 different settings, covering both the most popular models and distribution shifts. Our testbed consists of10 9 model predictions and is 100 times larger than prior work [27,33,47,68]. This allows us to draw several new conclusions about current robustness interventions: Robustness measurements should control for accuracy. Existing work typically argues that an intervention improves robustness by showing that the accuracy on a robustness test set has improved (e.g., see [34,40,63,102,115]). We find that in many cases, this improved robustness can be explained by the model performing better on the standard, unperturbed test set. For instance, using different model architectures does not substantially improve the robustness of a model beyond what would be expected from having a higher standard accuracy. While training more accurate models is clearly useful, it is important to separate accuracy improvements from robustness improvements when interpreting the results. Current synthetic robustness measures do not imply natural robustness.Prior work often evaluates on synthetic distribution shifts to measure robustness [9,32,38]. We find that current robustness measures for synthetic distribution shift are at most weakly predictive for robustness on the natural distribution shifts presently available. While there are good reasons to study synthetic forms of robustness – for instance, adversarial examples are interesting from a security perspective – synthetic distribution shifts alone do not provide a comprehensive measure of robustness at this time. Moreover, as the right plot in Figure 1 exemplifies, current robustness interventions are often (but not always) ineffective on the natural distribution shifts in our testbed. Training on more diverse data improves robustness. Across all of our experiments, the only intervention that improves robustness to multiple natural distribution shifts is training with a more diverse dataset. This overarching trend has not previously been identified and stands out only through our large testbed. Quantifying when and why training with more data helps is an interesting open question: while more data is generally helpful, we find some models that are trained on 100 times more data than the standard ImageNet training set but do not provide any robustness. The goal of our paper is specificallynotto introduce a new classification method or image dataset. Instead, our paper is a meta-study of current robustness research to identify overarching trends that span multiple evaluation settings. This is particularly important if the ultimate goal of a research direction is to produce models that function reliably in a wide variety of contexts. Our findings highlight robustness on real data as a clear challenge for future work. Due to the diminishing returns of larger training datasets, addressing this robustness challenge will likely require new algorithmic ideas and more evaluations on natural distribution shifts. 2 606570758085 ImageNet (top-1, %) 45 50 55 60 65 70 75 ImageNetV2 (top-1, %) Effective Robustness Hypothetical Robustness Intervention y = x Baseline accuracy Hypothetical robust model Standard models 606570758085 ImageNet (top-1, %) 45 50 55 60 65 70 75 80 85 ImageNetV2 (top-1, %) Simplified Distribution Shift Plot y = x Baseline accuracy Standard training Robustness intervention Trained with more data Figure 1: (Left) We plot 78 standard models trained on ImageNet without any robustness interventions, showing both their accuracy on the standard test set (ImageNet, x-axis) and on a test set with distribution shift (ImageNetV2, y-axis). All models lie below the y=xline: their accuracy under this distribution shift is lower than on the standard test set. Nevertheless, improvements in accuracy on the standard test set almost perfectly predict a consistent improvement under distribution shift, as shown by the linear fit (red line). A hypothetical robustness intervention, shown in green, should provideeffective robustness, i.e., the intervention should improve the accuracy under distribution shift beyond what is predicted by the linear fit. (Right) We plot most of the 204 models in our testbed, highlighting those with the highest effective robustness using square markers. These models are still far from closing the accuracy gap induced by the distribution shift (ideally a robust model would fall on they=xline). Figure 2 shows a more detailed version of this plot with error bars for all points. 2 Measuring robustness We first discuss how to measure robustness as a quantity distinct from accuracy. In our experiments, we always have two evaluation settings: the “standard” test set, and the test set with distribution shift. For a modelf, we denote the two accuracies with acc 1 (f)and acc 2 (f), respectively. When comparing the robustness of two modelsf a andf b , one approach would be to rank the models by their accuracy under distribution shift. However, this approach does not disentangle the robustness of a model from its accuracy on the standard test set. As an example, consider a pair of models with accuracyacc 1 (f a ) = 0.8,acc 2 (f a ) = 0.75(i.e., a 5% drop in accuracy from the distribution shift), andacc 1 (f b ) = 0.9,acc 2 (f b ) = 0.76(a 14% drop). Modelf b has higher accuracy on the second test set, but overall sees a drop of 14% from the standard to the shifted test set. In contrast, the first model sees only a 5% drop. Hence we would like to refer to the first model as more robust, even though it achieves lower accuracy on the shifted test set. Effective robustness.The core issue in the preceding example is that standard accuracy (acc 1 ) acts as a confounder. Instead of directly comparing accuracies under distribution shift, we would 3 like to understand if a modelf b offers higher accuracy on the shifted test setbeyond what is expected from having higher accuracy on the original test set. We call this notion of robustness beyond a baselineeffective robustness. Graphically, effective robustness corresponds to a model being above the linear trend (red line) given by our testbed of standard models in Figure 1 (left). To precisely define effective robustness, we introduceβ(x), the baseline accuracy on the shifted test set for a given accuracyxon the standard test set. On the distribution shifts in our testbed, we instantiateβby computing the parameters of a log-linear fit for the models without a robustness intervention (the red line in Figure 1). Empirically, this approach yields a good fit to the data. For other distribution shifts, the baseline accuracy may follow different trends and may also depend on properties beyond the standard accuracy, e.g., model architecture. Appendix I.1 contains detailed information on how to computeβ. Given the accuracy baselineβ, we define the effective robustness of a model as ρ(f) =acc 2 (f)−β(acc 1 (f)). A model without special robustness properties falls on the linear fit and hence hasρ(f)= 0. The main goal of a robustness intervention is to increaseρ. Models with largeρoffer robustness beyond what we can currently achieve with standard models. Relative robustness. Effective robustness alone does not imply that a robustness intervention is useful. In particular, a robustness intervention could increaseρfor a model it is applied to, but at the same timedecreasebothacc 1 andacc 2 . Such a robustness intervention would offer no benefits. So to complement effective robustness, we also introducerelative robustnessto directly quantify the effect of an intervention on the accuracy under distribution shift. For a modelf ′ with robustness intervention, derived from a modelfwithout the intervention, the relative robustness is τ(f ′ ) =acc 2 (f ′ )−acc 2 (f). We graphically illustrate this notion of robustness in Appendix B.1. Overall, a useful robustness intervention should obtainbothpositive effective and relative robustness. As we will see, only few classification models currently achieve this goal, and no models achieve both large effective and relative robustness. 3 Experimental setup We now describe our experimental setup. A modelfis first trained on a fixed training set. We then evaluate this model on two test sets: the “standard” test set (denotedS 1 ) and the test set with a distribution shift (denotedS 2 ). A crucial question in this setup is what accuracy the modelfcan possibly achieve on the test set with distribution shift. In order to ensure that the accuracy on the two test sets are comparable, we focus on natural distribution shifts where humans have thoroughly reviewed the test sets to include only correctly labeled images [2,18,39,68,76]. 1 This implies that an ideal robust classifier does not have a substantial accuracy gap between the two test sets. Indeed, recent work experimentally confirms that humans achieve similar classification accuracy on the original ImageNet test set and the ImageNetV2 replication study (one of the distribution shifts in our testbed) [77]. 1 For ObjectNet [2], Borji[7]has pointed out potential label quality issues, but also found that a substantial accuracy drop remains when taking these issues into account. 4 3.1 Types of distribution shifts At a high level, we distinguish between two main types of distribution shift. We use the term naturaldistribution shift for datasets that rely only on unmodified images. In contrast, we refer to distribution shifts assyntheticif they involve modifications of existing images specifically to test robustness. To be concrete, we next provide an overview of the distribution shifts in our robustness evaluation, with further details in Appendix E and visual overviews in Appendices A and J. 3.1.1 Natural distribution shifts We evaluate on seven natural distribution shifts that we classify into three categories. Consistency shifts. To evaluate a notion of robustness similar to` p -adversarial examples but without synthetic perturbations, we measure robustness to small changes across video frames as introduced by Gu et al.[35]and Shankar et al.[76]. The authors assembled sets of contiguous video frames that appear perceptually similar to humans, but produce inconsistent predictions for classifiers. We defineS 1 to be the set of “anchor” frames in each video, and evaluate the accuracy under distribution shift by choosing the worst frame from each frame set for a classifier. This is the “pm-k” metric introduced by Shankar et al. [76]. Dataset shifts. Next, we consider datasetsS 2 that are collected in a different manner fromS 1 but still evaluate a classification task with a compatible set of classes. These distribution shifts test to what extent current robustness interventions help with natural variations between datasets that are hard to model explicitly. We consider four datasets of this variety: (i) ImageNetV2, a reproduction of the ImageNet test set collected by Recht et al.[68]; (i) ObjectNet, a test set of objects in a variety of scenes with 113 classes that overlap with ImageNet [2]; and, (i) ImageNetVid-Robust-anchor and YTBB-Robust-anchor [76], which are the datasets constructed from only the anchor frames in the consistency datasets described above. These two datasets contain 30 and 24 super-classes of the ImageNet class hierarchy, respectively. For each of these distribution shifts, we defineS 1 to be a subset of the ImageNet test set with the same label set asS 2 so that the accuracies are comparable. Adversarially filtered shifts.Finally, we consider an adversarially collected dataset, ImageNet-A [39]. Hendrycks et al.[39]assembled the dataset by downloading a large number of labeled images from Flickr, DuckDuckGo, iNaturalist, and other sites, and then selected the subset that was misclassified by a ResNet-50 model. We include ImageNet-A in our testbed to investigate whether the adversarial filtering process leads to qualitatively different results. Since ImageNet-A contains only 200 classes, the standard test setS 1 here is again a subset of the ImageNet test set that has the same 200 classes as ImageNet-A. 3.1.2 Synthetic distribution shifts The research community has developed a wide range of synthetic robustness notions for image classification over the past five years. In our study, we consider the following classes of synthetic distribution shifts, which cover the most common types of image perturbations. 5 Image corruptions.We include all corruptions from [38], as well as some corruptions from [33]. These include common examples of image noise (Gaussian, shot noise), various blurs (Gaussian, motion), simulated weather conditions (fog, snow), and “digital” corruptions such as various JPEG compression levels. We refer the reader to Appendix E.2 for a full list of the 38 corruptions. Style transfer.We use a stylized version of the ImageNet test set [34, 44]. Adversarial examples. We include untargeted adversarial perturbations bounded in` ∞ - or` 2 - norm by running projected gradient descent as described in [55]. We useε= 0.5 255 , 2 255 for` ∞ and ε=0.1,0.5for` 2 (further details in Appendix E.3). 3.2 Classification models Our model testbed includes 204 ImageNet models covering a variety of different architectures and training methods. The models can be divided into the following three categories (see Appendix F for a full list of all models and their categories). Standard models. We refer to models trained on the ILSVRC 2012 training set without a specific robustness focus asstandardmodels. This category includes 78 models with architectures ranging from AlexNet to EfficietNet, e.g., [37, 50, 78, 85, 88]. Robust models.This category includes 86 models with an explicit robustness intervention such as adversarially robust models [13,27,72,74,101], models with special data augmentation [20, 28, 34, 41, 100, 108, 113], and models with architecture modifications [115]. Models trained on more data.Finally, our testbed contains 30 models that utilize substantially more training data than the standard ImageNet training set. This subset includes models trained on (i) Facebook’s collection of 1 billion Instagram images [56,104], (i) the YFCC 100 million dataset [104], (i) Google’s JFT 300 million dataset [82,102], (iv) a subset of OpenImages [98], or (v) a subset of the full ImageNet dataset of 21,841 classes [11, 49, 99]. 4 Main results We now present our main experiments. First, we measure how much effective and relative robustness models achieve on the natural distribution shifts in our testbed. Then we investigate to what extent robustness on synthetic distribution shift is predictive of robustness on natural distribution shift. 4.1 Results on natural distribution shifts Following the categorization in Section 3, we measure the robustness of classification models on three types of natural distribution shift. Appendix I.2 contains variations of the figures referenced in this sec- tion. For further detail, we have made interactive plots available at http://robustness.imagenetv2.org/. Dataset shifts.Figure 2 shows the effective robustness of models on the four dataset shifts in our testbed. In each case, we find that the standard test accuracy (x-axis) is a good predictor for the test accuracy under distribution shift (y-axis). The linear fit is best for ImageNetV2, ObjectNet, 6 606570758085 ImageNet (top-1, %) 45 50 55 60 65 70 75 80 ImageNetV2 (top-1, %) Distribution Shift to ImageNetV2 5560657075808590 ImageNet (class-subsampled) (top-1, %) 15 20 25 30 35 40 45 50 55 60 65 70 ObjectNet (top-1, %) Distribution Shift to ObjectNet 93949596979899 ImageNet (class-subsampled) (top-1, %) 50 55 60 65 70 75 80 85 90 ImageNet-Vid-Robust (pm-0, %) Distribution Shift to ImageNet-Vid-Anchors 9596979899 ImageNet (class-subsampled) (top-1, %) 45 50 55 60 65 70 YTBB-Robust (pm-0, %) Distribution Shift to YTBB-Anchors y = x Standard training Robustness intervention Trained with more data Linear fit Figure 2: Model accuracies on the four natural dataset shifts: ImageNetV2 (top left), ObjectNet (top right), ImageNet-Vid-Robust-anchor (bottom left), and YTBB-Robust- anchor (bottom right). These plots demonstrate that the standard test accuracy (x-axis) is a reliable predictor for the test accuracy under distribution shift (y-axis), especially for models trained without a robustness intervention. The notable outliers to this trend are some models trained on substantially more data. For ObjectNet, ImageNet-Vid-Robust- anchor, and YTBB-Robust-anchor, we show the accuracy on a subset of the ImageNet classes on the x-axis to match the label space of the target task (y-axis). Each data point corresponds to one model in our testbed and is shown with 99.5% Clopper-Pearson confidence intervals. The axes were adjusted using logit scaling and the linear fit was computed in the scaled space on only the standard models. The red shaded region is a 95% confidence region for the linear fit from 1,000 bootstrap samples. and ImageNet-Vid-Robust with respectiver 2 scores of 1.00, 0.95, and 0.95, but is more noisy for YTBB-Robust (r 2 = 0.83). The noisy fit on YTBB-Robust is likely due to the fact that the categories in YTBB-Robust are not well aligned with those of ImageNet, where the models were trained [76]. Another potential reason is that the video test sets are significantly smaller (2,530 images in YTBB and 1,109 images in ImageNet-Vid-Robust). 7 505560657075808590 ImageNet-Vid-Robust (pm-0, %) 30 35 40 45 50 55 60 65 70 75 80 ImageNet-Vid-Robust (pm-10, %) Distribution Shift to ImageNet-Vid-Robust 455055606570 YTBB-Robust (pm-0, %) 30 35 40 45 50 55 60 65 YTBB-Robust (pm-10, %) Distribution Shift to YTBB-Robust y = x Standard training Lp adversarially robust Other robustness intervention Trained with more data Linear fit Figure 3: Model accuracies on the two consistency shifts: ImageNet-Vid-Robust (left), and YTBB-Robust (right). Both plots are shown with evaluation on pm-0 (anchor frames) on the x-axis and pm-10 (worst case prediction in a 20-frame neighborhood) on the y-axis. This plot shows that most current robustness interventions do not provide robustness to consistency distribution shifts. The notable outliers to this trend are` p -adversarially robust models and EfficientNet-L2 (NoisyStudent). We color the adversarially robust models separately in this figure to illustrate this phenomenon. Confidence intervals, axis scaling, and the linear fit are computed similarly to Figure 2. In the high accuracy regime (above the 76% achieved by a ResNet-50), the main outliers in terms of positive effective robustness are models trained on substantially more data than the standard ImageNet training set. This includes a ResNet152 model trained on 11,000 ImageNet classes (ρ= 2.1%) [99], several ResNeXt models trained on 1 billion images from Instagram (ρ= 1.5%) [56], and the EfficientNet-L2 (NoisyStudent) model trained on a Google-internal JFT-300M dataset of 300 million images (ρ= 1.1%) [102]. However, not all models trained on more data display positive effective robustness. For instance, a ResNet101 trained on the same JFT-300M dataset has an effective robustness ofρ=−0.23% [82]. We conduct additional experiments to investigate the effect of training data in Section 5. Appendix G contains a full list of models with their effective robustness numbers. On YTBB-Robust, a few data augmentation strategies and` p -robust models display positive effective robustness; we investigate this further in Appendix B.2. Consistency shifts.We plot the effective robustness of models on consistency shifts in Figure 3. Interestingly, we observe that` p -adversarially robust models display substantial effective robustness to ImageNet-Vid-Robust (averageρ= 6.7%) and YTBB-Robust (averageρ= 4.9%). This suggests that these models are not only more robust to synthetic perturbations, but also offer robustness for the perceptually small variations between consecutive video frames. However, these gains in effective robustness do not necessarily lead to relative robustness. On average, relative robustness on both datasets is negative (averageτ=−8.5% on ImageNet-Vid-Robust and averageτ=−0.7% on YTBB-Robust for ResNet50 models). See Appendix B.2 (Figure 10) for a visual comparison. Among the models trained on more data, only one achieves both high accuracy 8 80909596979899 ImageNet (class-subsampled) (top-1, %) 5 10 20 30 40 50 60 70 80 ImageNet-A (top-1, %) Distribution Shift to Imagenet-A y = x Linear fit (piecewise) ResNet50 accuracy Standard training Robustness intervention Trained with more data Figure 4: Model accuracies on ImageNet-A, a dataset adversarially filtered to contain only images incorrectly classified by a ResNet50 trained on ImageNet. This filtering results in a ‘knee’ curve: models with lower ImageNet accuracy than ResNet-50 have near-chance performance on ImageNet-A, while models with higher ImageNet accuracy improve drastically on ImageNet-A. ImageNet classes were subsampled to match the class distribution of ImageNet-A. Confidence intervals and axis scaling are computed similarly to Figure 2. The linear fit is computed piecewise around the ResNet50 model accuracy. and substantial effective robustness: EfficientNet-L2 (NoisyStudent) [102] hasρ= 2.4% andρ= 7.4% on ImageNet-Vid-Robust and YTBB-Robust, respectively. Adversarially filtered shifts. ImageNet-A [39] was created by classifying a set of images with a ResNet50 and only keeping the misclassified images. Interestingly, this approach creates a “knee” in the resulting scatter plot (see Figure 4): models below a ResNet50’s standard accuracy have close to chance performance on ImageNet-A, 2 and models above a ResNet50’s standard accuracy quickly close the accuracy gap. In the high accuracy regime, every percentage point improvement on ImageNet brings at least an 8% improvement on ImageNet-A. This is in contrast to datasets that are not constructed adversarially, where the initial accuracy drops are smaller, but later models make slow progress on closing the gap. These results demonstrate that adversarial filtering does not necessarily lead to harder distribution shifts. 4.2 Results on synthetic distribution shifts Given the difficulty of collecting real world data to measure a model’s robustness to natural distribution shifts, an important question is whether there are synthetic proxies. We now study to what extent robustness to the above synthetic distribution shifts predicts robustness on these natural distribution shifts. 2 Chance performance is 0.5% as ImageNet-A contains 200 classes. 9 606570758085 ImageNet (top-1, %) 20 25 30 35 40 45 50 55 60 65 70 Corruptions Averaged (top-1, %) Distribution Shift to Corruptions Averaged -505101520 Corruptions Averaged Effective Robustness -1 0 1 2 ImageNetV2 Effective Robustness Effective Robustness Scatterplot 606570758085 ImageNet (top-1, %) 15 20 25 30 35 40 45 50 55 60 Lp Attacks (top-1, %) Distribution Shift to Lp Attacks -15-10-5051015202530354045 Lp Attacks Effective Robustness -1 0 1 2 ImageNetV2 Effective Robustness Effective Robustness Scatterplot y = x Standard training Lp adversarially robust Other robustness intervention Trained with more data Linear fit Figure 5: Model accuracies under image corruptions (top row) and` p -attacks (bottom row). Similar to Figure 2, the left plots show the effective robustness for each synthetic distribution shift. Multiple non-standard models achieve substantial effective robustness, corroborating recent research progress on creating models robust to synthetic shift. The right plots show the correlation between the effective robustness for each synthetic shift and the ImageNetV2 distribution shift (top left in Figure 2) for the non-standard models. Both image corruptions and` p -attacks are very weakly predictive of effective robustness on ImageNetV2: there are several models that achieve high effective robustness under the synthetic measures but little to no effective robustness on ImageNetV2. In Figure 5, we analyze the predictiveness of two commonly studied synthetic robustness metrics: average accuracy on image corruptions [38], and average accuracy drop under a range of PGD adversarial attacks [55]. We compare these metrics with effective robustness on ImageNetV2. While effective robustness is only one aspect (c.f. Section 2), it is a necessary prerequisite for a model to have helpful robustness properties. The plots show that robustness under either of these synthetic distribution shifts does not imply that the corresponding model has effective robustness on ImageNetV2 (the Pearson correlation coefficients arer= 0.24andr=−0.05for image corruptions and` p -adversarial attacks, respectively). Appendix C further extends the experiment by comparing both synthetic distribution shift measures with the 10 remaining natural distribution shifts in our testbed and reaches similiar conclusions. Our analysis of the aggregate measures proposed in prior work does not preclude that specific synthetic distribution shifts do predict behavior on natural distribution shifts. Instead, our results show that averaging a large number of synthetic corruptions does not yield a comprehensive robustness measure that also predicts robustness on natural distribution shift. To extend on this analysis, in Appendix H we find that no individual synthetic measure in our testbed is a consistent predictor of natural distribution shift, but some synthetic shifts are substantially more predictive than others. For instance,` p -robustness has the highest correlation with consistency shifts, and some image corruptions such as brightness or Gaussian blur have higher correlation with dataset shifts. However, our testbed indicates that these synthetic measures are not necessarily causal, i.e., models trained with brightness or Gaussian blur do not have substantial effective robustness on dataset shifts. Further analyzing relationships between individual synthetic and natural distribution shifts is an interesting avenue for future work. 4.3 Takeaways and discussion To recap our results, we now discuss two of the central questions in our paper: Do current robustness interventions help on real data? And is synthetic robustness correlated with natural robustness? Across our study, current robustness interventions offer little to no improvement on the natural distribution shifts presently available. For dataset shifts, we find that models trained with substantially more data yield a small improvement. However, the amount of extra data needed is orders of magnitude larger than the standard ImageNet training set, and the models show only small gains (in the best case improving the accuracy drop from 8.6% to 7.5% on ImageNetV2 for EfficientNet-L2 NoisyStudent). These results suggest that current robustness interventions methods do not provide benefits on the dataset shifts in our study. For consistency shifts, adversarially trained models generally have effective robustness, but usually little or no relative robustness. On ImageNet-Vid-Robust, the baseline models without adversarial training still achieve higher accuracy under distribution shift. A notable outlier is EfficietNet-L2 (NoisyStudent) [102], which utilizes self-training and exhibits high effective robustness in the high accuracy regime. Self-training has recently been shown to help adversarial robustness as well [10,61,94]. Investigating the effect of self-training on robustness is an interesting direction for future work. Moreover, we find that current aggregate metrics for synthetic robustness are at most weakly correlated with natural robustness. Effective robustness under non-adversarial image corruptions or ` p -attacks does not imply effective robustness to natural distribution shifts. While much progress has been made on creating models robust to synthetic distribution shift, new methods may be needed to handle natural shifts. 11 25303540455055606570758085 ImageNet (iid-subsampled) (top-1, %) 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 ImageNetV2 (iid- subsampled) (top-1, %) Robustness for Subsampling ImageNet y = x Linear fit Standard training Trained with more data No subsampling Subsample 1/2 Subsample 1/4 Subsample 1/8 Subsample 1/16 Subsample 1/32 80859095 ImageNet (class-subsampled) (top-1, %) 70 75 80 85 90 95 ImageNetV2 (class- subsampled) (top-1, %) Robustness for Subsampling ImageNet y = x Linear fit Standard training Trained with more data 1000 classes 500 classes 250 classes 125 classes Figure 6: To investigate the impact of training data on robustness, we vary the training data along two axes: the number of images per class (left), and the number of classes (right). Although models trained on more data provide improvements in effective robustness, we find that subsampling the training set has no impact on effective robustness. Confidence intervals, axis scaling, and the linear fit are computed similarly to Figure 2. 5 How does the amount of training data impact robustness? As discussed in Section 4.1, multiple models trained on more data achieve positive effective robustness on dataset shifts. However, this effect is not uniform. Among others, the ResNet101 model trained on JFT-300M has negligible effective robustness (ρ=−0.23%) despite being trained on300×more data than standard ImageNet models. A possible explanation is that differences in label diversity or quality play a role in promoting robustness. We investigate the role of data in more detail with two experiments. Varying the number of images per class.We start by subsampling the ILSVRC-2012 training set by factors of 2, 4, 8, 16, 32 and show the impact on accuracy and robustness on ImageNetV2 in Figure 6. While larger training subsets yield higher accuracies, they do not improve effective robustness, at least for ImageNetV2. Varying the number of classes.Next, we subsample ImageNet in a more biased way by varying the set of classes. First, we create three subsets of the ILSVRC training set with 500, 250, and 125 classes and train models on these subsets. We then evaluate all models on the 125 class subset and show the results in Figure 6. Varying the number of classes again affects accuracies, but does not impact effective robustness. Our experiments suggest that neither growing the number of images nor classes in an i.i.d. fashion are effective robustness interventions. Nevertheless, Figure 2 shows that larger datasets can provide meaningful robustness improvements. This disparity may be due to limitations of emulating dataset growth by subsampling ILSVRC. For one, our experiments consider only i.i.d. subsets of the training images or classes. Another possibility is that increases in dataset size may only improve robustness after the dataset is large enough so that the accuracy on the original distribution is nearly saturated. Our experiments only observe dataset sizes smaller than ILSVRC, which may fall below this inflection point. Studying the effect of data on robustness is an important direction for future work. 12 6 Related work Our work is best seen as a unification of two independent lines of research—synthetic and natural distribution shift—not previously studied together. Synthetic distribution shifts have been studied extensively in the literature [28,33,38,48,57,93]. We incorporate as many prior synthetic measures of robustness as possible. Our dataset largely confirms the high-level results from these papers (see Appendix D for additional discussion). For example, Ford et al.[31]provide evidence for the relationship between adversarial robustness and robustness to Gaussian noise. The study of natural distribution shifts has been an equally extensive research direction [2,68,76,91]. When examining each natural distribution shift individually, we confirm the findings of earlier work that there is a consistent drop with a linear trend going from ImageNet to each of the other test sets [2, 68, 76]. We study the relationship between these two previously independent lines of work. By creating a testbed100×larger than prior work [27,33,47,68], we are able to make several new observations. For instance, we show that robustness to synthetic distribution shift often behaves differently from robustness to natural distribution shift. We argue that it is important to control for accuracy when measuring the efficacy of a robustness intervention. Viewed in this light, most interventions do not provide effective robustness. The main exception is training with more data, which improves robustness across natural distribution shifts. In some situations,` p -adversarial robustness helps with natural distribution shift that asks for consistency across similar looking images. Appendix K contains additional discussion of specific related work in more detail. For instance, Appendix K.2 revisits consistency shifts and explains why, in contrast to previous work [35], we find consistency robustness is only weakly correlated with color corruption robustness. Concurrent and subsequent work.An early version of this paper with results on ImageNetV2 and ImageNet-Vid-Robust appeared on OpenReview in late 2019 [90]. Since then, two closely related papers have been published concurrently with the updated version of this paper. Djolonga et al.[21]evaluate 40 models on the same natural distribution shifts as our paper. Our testbed is larger and contains 200 models with more robustness interventions. Overall both papers reach similar conclusions. Their focus is more on the connections to transfer learning while we focus more on comparisons between synthetic and natural distribution shifts. Djolonga et al.[21]also explore the performance of various models with a synthetic image dataset. Hendrycks et al.[40]also study the connections between synthetic robustness and robustness to natural distribution shifts. This paper introduces a new dataset, ImageNet-R, that contains various renditions (sculptures, paintings, etc.) of 200 ImageNet classes as a new example of natural distribution shift. The paper then introduces DeepAugment, a new data augmentation technique based on synthetic image transformations, and find that this robustness intervention is effective on ImageNet-R. In Appendix K.1, we analyze the ImageNet-R test set and DeepAugment models, as well as the closely related ImageNet-Sketch test set [95], in more detail. At a high-level, ImageNet-R and ImageNet-Sketch follow the trends of the other dataset shifts in our testbed, with models trained on extra data providing the most robustness (up toρ= 29.1% on ImageNet-R, though the effect is not uniform, similar to the other dataset shifts). After the models trained on more data, we find that DeepAugment (in combination with AugMix [41]) achieves 13 substantial effective robustness (ρ= 11.2%). Interestingly, adversarial robustness also leads to effective robustness on ImageNet-R. An AdvProp model [100] achieves the highest absolute accuracy on ImageNet-R for a model trained without extra data (57.8%) and has effective robustnessρ= 7.5%. A model with feature denoising and trained with PGD-style robust optimization [55,101] achieves the highest effective robustness on ImageNet-R (ρ= 22.7%) and also positive relative robustness (τ= 5.7%). Domain adaptation / transfer learning. Our work is focused on generalizing to out-of- distribution datawithoutfine-tuning on the target distribution. A complementary approach uses data from the target domain in order to improve generalization on that particular domain [64]. Depending on the scenario, robustness (without fine-tuning) or domain adaptation may be more appropriate. For instance, it may be challenging to record data from the distribution shift, which would prevent fine-tuning before deployment. In some scenarios, we also expect our model to generalize without extra data (e.g., because humans can do so [77]). Concurrent work by Djolonga et al.[21]studies connections between robustness to distribution shifts and transfer learning. Investigating our testbed from the perspective of transfer learning is an interesting direction for future research. Domain generalization.Out-of-distribution generalization as measured in our robustness testbed is closely related to domain generalization [6,60]. In domain generalization, the training algorithm has access to samples drawn from multiple different distributions (domains). At test time, the model is evaluated on samples from a new domain that was not present in training. The idea is that having explicit knowledge of multiple domains at training time may help generalization to a new domain at test time. Several papers have proposed algorithms for domain generalization; we refer to Gulrajani & Lopez- Paz[36]for a comprehensive survey. Our testbed currently does not contain any algorithms explicitly following the domain generalization paradigm (though pre-training on a different distribution and then fine-tuning on ImageNet has similarities to domain generalization). A recent meta-study of domain generalization found that standard empirical risk minimization performs as well or better than the eight domain generalization algorithms they compared to [36]. This result of Gulrajani & Lopez-Paz[36]has similarities to our finding that robustness interventions currently rarely improve over the trend given by standard (ERM) models trained without a robustness intervention. Evaluating domain generalization approaches on the distribution shifts in our testbed may yield new insights into the performance characteristics of these algorithms. Distributionally robust optimization.Distributionally robust optimization (DRO) is another recently proposed technique to increase robustness to distribution shift [22,23]. The DRO objective minimizes the worst case risk over all distributions close to the data distribution (or in the group DRO setting, the worst case risk over all defined groups). DRO has been used to train adversarially robust models [79], vision models with higher worst-group accuracies [71], models less reliant on spurious correlations [81], and many others [24,62]. For a more thorough discussion on DRO and related work, we refer the reader to [22]. We are currently unable to include DRO models as we are not aware of any pre-trained DRO models for ImageNet. We will add DRO models to our testbed as they become available. 14 Adversarial filtering.One of the distribution shifts in our testbed was obtained via adversarial filtering (ImageNet-A, [39]). Architectures introduced after the model used to filter ImageNet-A made quick progress in closing the accuracy gap (see Section 4.1). A similar phenomenon occurred in natural language processing. Zellers et al.[111]introduced Swag, an adversarially filtered test for grounded commonsense inference, a combination of natural language inference and commonsense reasoning. At the time of publication, the best model achieved 59% accuracy, while a human expert achieved 85%. Two months later, Devlin et al.[19]introduced the BERT model which achieves 86% accuracy on Swag. This provides further evidence that adversarial filtering can create test sets that are only hard for a specific (existing) class of models. In the context of training sets, adversarial filtering is similar to hard negative mining, which is often used to generate training data for detection models [17, 30, 69, 83]. Bras et al. [8] propose AFLite, an adversarial filtering algorithm for both refining training sets and creating harder test sets. They evaluate AFLite on natural language inference tasks and ImageNet classification. An interesting question is whether combining their algorithm with a ResNet-50 and evaluating later models leads to similar phenomena as on ImageNet-A [39] and Swag [111]. Fairness in machine learning.Mitchell et al.[59]proposed model cards to document the performance of machine learning models in a variety of conditions. Their focus is on human-centered models and distribution shifts arising from demographic groups (race, gender, etc.). Our focus here is on ImageNet due to the large number of available models and distribution shifts, but the underlying problem is similar: machine learning models are often brittle under distribution shift. We remark that ImageNet is known to have geo-diversity deficiencies [75], among other issues [14,25]. In the context of OpenImages [51], researchers have proposed the Inclusive Image dataset [1]. Adding OpenImages and Inclusive Images to our testbed and comparing these distribution shifts to our existing examples is an interesting direction for future work. Further domains.Our work is focused on the domain of image classification. There is a long line of work considering robustness (either natural or synthetic) on other domains [26,52,58,80,106]. In the context of natural language processing, Belinkov & Bisk[4]explore language model robustness to synthetic versus natural one-word substitutions and reach similar high-level results, finding there is limited robustness transfer between the two distributions. 7 Conclusion The goal of robust machine learning is to develop methods that function reliably in a wide variety of settings. So far, this research direction has focused mainly on synthetic perturbations of existing test sets, highlighting important failure cases and initiating progress towards more robust models. Ultimately, the hope is that the resulting techniques also provide benefits on real data. Our paper takes a step in this direction and complements the current synthetic robustness tests with comprehensive experiments on distribution shifts arising from real data. We find that current image classification models still suffer from substantial accuracy drops on natural distribution shifts. Moreover, current robustness interventions – while effective against 15 synthetic perturbations – yield little to no consistent improvements on real data. The only approach providing broad benefits is training on larger datasets, but the gains are small and inconsistent. Overall, our results show a clear challenge for future research. Even training on 1,000 times more data is far from closing the accuracy gaps, so robustness on real data will likely require new algorithmic ideas and better understanding of how training data affects robustness. Our results indicate two immediate steps for work in this area: robustness metrics should control for baseline accuracy, and robust models should additionally be evaluated on natural distribution shifts. We hope that our comprehensive testbed with nuanced robustness metrics and multiple types of distribution shift will provide a clear indicator of progress on the path towards reliable machine learning on real data. Broader Impact Robustness is one of the key problems that prevents deploying machine learning in the real world and harnessing the associated benefits. A canonical example is image classification for medical diagnosis. As was found when researchers attempted to deploy a neural network to detect diabetes from retina images, “an accuracy assessment from a lab goes only so far. It says nothing of how the AI will perform in the chaos of a real-world environment” [3]. Similarly, researcher also found that current methods for chest X-ray classification are brittle even in the absence of recognized confounders [110]. If models were robust, then this transfer to the real world would be straightforward. Unfortunately, achieving robustness on real data is still a substantial challenge for machine learning. Our work studies how robust current image classification methods are to distribution shifts arising in real data. We hope that our paper will have a positive effect on the study of distribution shifts and allow researchers to more accurately evaluate to what extent a proposed technique increases the robustness to particular forms of distribution shift. This will allow researchers to better understand how a deployed system will work in practice, without actually having to deploy it first and users potentially suffering negative consequences. However, there are several potential ways in which our study could cause unintended harm. It is possible that our paper might be used as an argument to stop performing research on some synthetic forms of robustness, e.g., adversarial examples or common corruptions. This is not our intention. These forms of corruption are interesting independent of any correlation to existing natural distribution shift (e.g., adversarial examples are a genuine security problem). We only capture a small number of natural distribution shifts among all the possible distribution shifts. We selected these shifts because they have been used extensively in the literature and are concrete examples of the types of distribution shift we would like models to be robust to. It is likely that there are shifts that we do not capture, and so even if the shifts we define were to be completely solved, other shifts would remain a concern. One significant form of distribution shift we do not evaluate is dataset bias in representing different demographic groups. For example, the Inclusive Images dataset [75] attempts to correct for the geographical bias introduced in the Open Images dataset [51] by including a more balanced representation of images from Africa, Asia, and South America. Neglecting such implicit biases in the data distribution can harm underrepresented demographic groups. Ultimately, evaluating 16 on fixed datasets may not be enough, and validating the fairness and safety of deployable machine learning requires careful analysis in the application domain. Finally, more reliable machine learning can also enable negative uses cases, e.g., widespread surveil- lance or autonomous weapon systems. As with many technologies, these risks require careful regulation and awareness of unintended consequences arising from technological advances. Acknowledgements We would like to thank Logan Engstrom, Justin Gilmer, Moritz Hardt, Daniel Kang, Jerry Li, Percy Liang, Nelson Liu, John Miller, Preetum Nakkiran, Rebecca Roelofs, Aman Sinha, Jacob Steinhardt, and Dimitris Tsipras for helpful conversations while working on this paper. This research was generously supported in part by ONR awards N00014-17-1-2191, N00014-17-1- 2401, and N00014-18-1-2833, the DARPA Assured Autonomy (FA8750-18-C-0101) and Lagrange (W911NF-16-1-0552) programs, a Siemens Futuremakers Fellowship, an Amazon AWS AI Research Award. 17 References [1]Atwood, J., Baljekar, P., Barnes, P., Batra, A., Breck, E., Chi, P., Doshi, T., Elliott, J., Kour, G., Gaur, A., Halpern, Y., Jicha, H., Long, M., Saxena, J., Singh, R., and Sculley., D. The Inclusive Images competition, 2018.=https://ai.googleblog.com/2018/09/introducing-inclusive- images-competition.html. [2]Barbu, A., Mayo, D., Alverio, J., Luo, W., Wang, C., Gutfreund, D., Tenenbaum, J., and Katz, B. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. InAdvances in Neural Information Processing Systems (NeurIPS), 2019. http://papers.nips.c/paper/9142-objectnet-a-large-scale-bias-controlled-data set-for-pushing-the-limits-of-object-recognition-models. [3]Beede, E., Baylor, E., Hersh, F., Iurchenko, A., Wilcox, L., Ruamviboonsuk, P., and Var- doulakis, L. M. A human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy. InCHI Conference on Human Factors in Computing Systems, 2020.https://dl.acm.org/doi/abs/10.1145/3313831.3376718. [4]Belinkov, Y. and Bisk, Y. Synthetic and natural noise both break neural machine translation. InInternational Conference on Learning Representations (ICLR), 2018.https://arxiv.org/ abs/1711.02173. [5]Biggio, B., Corona, I., Maiorca, D., Nelson, B., Šrndić, N., Laskov, P., Giacinto, G., and Roli, F. Evasion attacks against machine learning at test time. InEuropean Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECMLPKDD), 2013.https://arxiv.org/abs/1708.06131. [6] Blanchard, G., Lee, G., and Scott, C. Generalizing from several related classification tasks to a new unlabeled sample. InAdvances in Neural Information Processing Systems (NIPS), 2011.https://papers.nips.c/paper/4312-generalizing-from-several-related-clas sification-tasks-to-a-new-unlabeled-sample. [7]Borji, A. Objectnet dataset: Reanalysis and correction, 2020.https://arxiv.org/abs/2004 .02042. [8]Bras, R. L., Swayamdipta, S., Bhagavatula, C., Zellers, R., Peters, M. E., Sabharwal, A., and Choi, Y. Adversarial filters of dataset biases. InInternational Conference on Machine Learning (ICML), 2020.https://arxiv.org/abs/2002.04108. [9] Carlini, N., Athalye, A., Papernot, N., Brendel, W., Rauber, J., Tsipras, D., Goodfellow, I., Madry, A., and Kurakin, A. On evaluating adversarial robustness, 2019.https://arxiv.org/ abs/1902.06705. [10]Carmon, Y., Raghunathan, A., Schmidt, L., Liang, P., and Duchi, J. C. Unlabeled data improves adversarial robustness. InAdvances in Neural Information Processing Systems (NeurIPS), 2019.https://arxiv.org/abs/1905.13736. [11]Chen, Y., Li, J., Xiao, H., Jin, X., Yan, S., and Feng, J. Dual path networks. InAdvances in Neural Information Processing Systems (NeurIPS), 2017.https://arxiv.org/abs/1707.016 29. 18 [12]Chollet, F. Xception: Deep learning with depthwise separable convolutions. InConference on Computer Vision and Pattern Recognition (CVPR), 2017.https://arxiv.org/abs/1610.0 2357. [13]Cohen, J. M., Rosenfeld, E., and Kolter, J. Z. Certified adversarial robustness via randomized smoothing. InInternational Conference on Machine Learning (ICML), 2019.https://arxiv. org/abs/1902.02918. [14] Crawford, K. and Paglen, T. Excavating AI: The politics of training sets for machine learning, 2019.https://w.excavating.ai/. [15] Cubuk, E. D., Zoph, B., Mane, D., Vasudevan, V., and Le, Q. V. Autoaugment: Learning augmentation policies from data. InConference on Computer Vision and Pattern Recognition (CVPR), 2019.https://arxiv.org/abs/1805.09501. [16]Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V. Randaugment: Practical automated data augmentation with a reduced search space, 2019.https://arxiv.org/abs/1909.13719. [17]Dalal, N. and Triggs, B. Histograms of oriented gradients for human detection. InConference on Computer Vision and Pattern Recognition (CVPR), 2005.https://ieeexplore.ieee.or g/document/1467360. [18]Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. InConference on Computer Vision and Pattern Recognition (CVPR), 2009.http://w.image-net.org/papers/imagenet_cvpr09.pdf. [19] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. InConference of the North American Chapter of the Association for Computational Linguistics (ACL), 2019.https://w.aclweb.org/antholo gy/N19-1423/. [20]DeVries, T. and Taylor, G. W. Improved regularization of convolutional neural networks with cutout, 2017.https://arxiv.org/abs/1708.04552. [21]Djolonga, J., Yung, J., Tschannen, M., Romijnders, R., Beyer, L., Kolesnikov, A., Puigcerver, J., Minderer, M., D’Amour, A., Moldovan, D., Gelly, S., Houlsby, N., Zhai, X., and Lucic, M. On robustness and transferability of convolutional neural networks, 2020.https://arxiv.or g/abs/2007.08558. [22] Duchi, J. and Namkoong, H. Learning models with uniform performance via distributionally robust optimization.To appear in the Annals of Statistics, 2018.https://arxiv.org/abs/18 10.08750. [23] Duchi, J. and Namkoong, H. Variance-based regularization with convex objectives.Journal of Machine Learning Research (JMLR), 2019.https://arxiv.org/abs/1610.02581. [24] Duchi, J., Hashimoto, T., and Namkoong, H. Distributionally robust losses for latent covariate mixtures, 2019.https://arxiv.org/abs/2007.13982. [25]Dulhanty, C. and Wong, A. Auditing ImageNet: Towards a model-driven framework for annotating demographic attributes of large-scale image datasets, 2019.https://arxiv.org/ abs/1905.01347. 19 [26]Dunn, M., Sagun, L., Higgins, M., Guney, V. U., Cirik, V., and Cho, K. Searchqa: A new q&a dataset augmented with context from a search engine, 2017.https://arxiv.org/abs/1704 .05179. [27]Engstrom, L., Ilyas, A., Santurkar, S., and Tsipras, D. Robustness (python library), 2019. URLhttps://github.com/MadryLab/robustness. [28] Engstrom, L., Tran, B., Tsipras, D., Schmidt, L., and Madry, A. Exploring the landscape of spatial robustness. InInternational Conference on Machine Learning (ICML), 2019.https: //arxiv.org/abs/1712.02779. [29] Fawzi, A. and Frossard, P. Manitest: Are classifiers really invariant? InBMVC, 2015. https://arxiv.org/abs/1507.06535. [30]Felzenszwalb, P. F., Girshick, R. B., McAllester, D., and Ramanan, D. Object detection with discriminatively trained part-based models.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010.https://ieeexplore.ieee.org/document/5255236. [31]Ford, N., Gilmer, J., Carlini, N., and Cubuk, E. D. Adversarial examples are a natural consequence of test error in noise. InInternational Conference on Machine Learning (ICML), 2019.http://arxiv.org/abs/1901.10513. [32]Galloway, A., Tanay, T., and Taylor, G. W. Adversarial training versus weight decay, 2018. https://arxiv.org/abs/1804.03308. [33]Geirhos, R., Temme, C. R. M., Rauber, J., Schüt, H. H., Bethge, M., and Wichmann, F. A. Generalisation in humans and deep neural networks. InAdvances in Neural Information Processing Systems (NeurIPS), 2018.https://papers.nips.c/paper/7982-generalisati on-in-humans-and-deep-neural-networks. [34]Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F. A., and Brendel, W. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. InInternational Conference on Learning Representations (ICLR), 2019. https://arxiv.org/abs/1811.12231. [35]Gu, K., Yang, B., Ngiam, J., Le, Q., and Shlens, J. Using videos to evaluate image model robustness. InSafeML workshop International Conference on Learning Representations (ICLR), 2019.https://arxiv.org/abs/1904.10076. [36]Gulrajani, I. and Lopez-Paz, D. In search of lost domain generalization, 2020.https: //arxiv.org/abs/2007.01434. [37] He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.https://arxiv.or g/abs/1512.03385. [38] Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturbations. InInternational Conference on Learning Representations (ICLR), 2019.https://arxiv.org/abs/1903.12261. [39]Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples, 2019.https://arxiv.org/abs/1907.07174. 20 [40]Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., and Gilmer, J. The many faces of robustness: A critical analysis of out-of-distribution generalization, 2020.https://arxiv.org/abs/2006 .16241. [41]Hendrycks, D., Mu, N., Cubuk, E. D., Zoph, B., Gilmer, J., and Lakshminarayanan, B. AugMix: A simple data processing method to improve robustness and uncertainty. InInternational Conference on Learning Representations (ICLR), 2020.https://arxiv.org/abs/1912.02781. [42]Hu, J., Shen, L., Albanie, S., Sun, G., and Wu, E. Squeeze-and-excitation networks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2018.https://arxiv.or g/abs/1709.01507. [43]Huang, G., Liu, Z., van der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. InConference on Computer Vision and Pattern Recognition (CVPR), 2017.https: //arxiv.org/abs/1608.06993. [44] Huang, X. and Belongie, S. Arbitrary style transfer in real-time with adaptive instance normalization.International Conference on Computer Vision (ICCV), 2017.https://arxiv. org/abs/1703.06868. [45] Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., and Keutzer, K. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size, 2016. https://arxiv.org/abs/1602.07360. [46]Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. InInternational Conference on Machine Learning (ICML), 2015. https://arxiv.org/abs/1502.03167. [47] Kang, D., Sun, Y., Brown, T., Hendrycks, D., and Steinhardt, J. Transfer of adversarial robustness between perturbation types, 2019.https://arxiv.org/abs/1905.01034. [48] Kang, D., Sun, Y., Hendrycks, D., Brown, T., and Steinhardt, J. Testing robustness against unforeseen adversaries, 2019.https://arxiv.org/abs/1908.08016. [49]Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. Big transfer (bit): General visual representation learning, 2019.https://arxiv.org/abs/1912.1 1370. [50]Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. InAdvances in Neural Information Processing Systems (NIPS), 2012.https: //papers.nips.c/paper/4824-imagenet-classification-with-deep-convolutional- neural-networks. [51] Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., and et al. The open images dataset v4.International Journal of Computer Vision (IJCV), 2020.https://arxiv.org/abs/1811.00982. [52]Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. Natural questions: a benchmark for question 21 answering research.Transactions of the Association for Computational Linguistics, 7:453–466, 2019.https://arxiv.org/abs/1712.00559. [53]Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L.-J., Fei-Fei, L., Yuille, A., Huang, J., and Murphy, K. Progressive neural architecture search. InEuropean Conference on Computer Vision (ECCV), 2018.https://arxiv.org/abs/1712.00559. [54] Ma, N., Zhang, X., Zheng, H.-T., and Sun, J. Shufflenet v2: Practical guidelines for efficient cnn architecture design. InProceedings of the European Conference on Computer Vision (ECCV), 2018.https://arxiv.org/abs/1807.11164. [55] Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. InInternational Conference on Learning Representations (ICLR), 2018.https://arxiv.org/abs/1706.06083. [56]Mahajan, D. K., Girshick, R. B., Ramanathan, V., He, K., Paluri, M., Li, Y., Bharambe, A., and van der Maaten, L. Exploring the limits of weakly supervised pretraining. InEuropean Conference on Computer Vision (ECCV), 2018.https://arxiv.org/abs/1805.00932. [57]Maini, P., Wong, E., and Kolter, J. Z. Adversarial robustness against the union of multiple perturbation models, 2019.https://arxiv.org/abs/1909.04068. [58]Miller, J., Krauth, K., Recht, B., and Schmidt, L. The effect of natural distribution shift on question answering models. InInternational Conference on Machine Learning (ICML), 2020. https://arxiv.org/abs/2004.14444. [59] Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., and Gebru, T. Model cards for model reporting. InConference on Fairness, Accountability, and Transparency (FAT), 2019.https://arxiv.org/abs/1810.03993. [60]Muandet, K., Balduzzi, D., and Schölkopf, B. Domain generalization via invariant feature representation. InInternational Conference on Machine Learning (ICML), 2013.https: //arxiv.org/abs/1301.2115. [61] Najafi, A., Maeda, S.-i., Koyama, M., and Miyato, T. Robustness to adversarial perturbations in learning from incomplete data. InAdvances in Neural Information Processing Systems (NeurIPS), 2019.https://arxiv.org/abs/1905.13021. [62]Oren, Y., Sagawa, S., Hashimoto, T. B., and Liang, P. Distributionally robust language modeling. InConference on Empirical Methods in Natural Language Processing (EMNLP), 2019.https://arxiv.org/abs/1909.02060. [63] Orhan, A. E. Robustness properties of facebook’s resnext wsl models, 2019.https://arxiv. org/abs/1907.07640. [64] Pan, S. J. and Yang, Q. A survey on transfer learning.IEEE Transactions on Knowledge and Data Engineering, 2010.https://ieeexplore.ieee.org/document/5288526. [65]Quionero-Candela, J., Sugiyama, M., Schwaighofer, A., and Lawrence, N. D.Dataset Shift in Machine Learning. The MIT Press, 2009. 22 [66]Raghunathan, A., Steinhardt, J., and Liang, P. Certified defenses against adversarial examples. InInternational Conference on Learning Representations (ICLR), 2018.https://arxiv.org/ abs/1801.09344. [67]Real, E., Shlens, J., Mazzocchi, S., Pan, X., and Vanhoucke, V. Youtube-boundingboxes: A large high-precision human-annotated data set for object detection in video. InCVPR, 2017. [68] Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do imagenet classifiers generalize to imagenet? InInternational Conference on Machine Learning (ICML), 2019.https: //arxiv.org/abs/1902.10811. [69] Rowley, H. A., Baluja, S., and Kanade, T. Human face detection in visual scenes. InAdvances in Neural Information Processing Systems (NIPS). 1996.https://papers.nips.c/paper/1 168-human-face-detection-in-visual-scenes. [70]Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNet Large Scale Visual Recognition Challenge.International Journal of Computer Vision, 2015.https://arxiv.or g/abs/1409.0575. [71]Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. InInternational Conference on Learning Representations (ICLR), 2020.https://arxiv.org/ abs/1911.08731. [72]Salman, H., Yang, G., Li, J., Zhang, P., Zhang, H., Razenshteyn, I., and Bubeck, S. Provably robust deep learning via adversarially trained smoothed classifiers. InAdvances in Neural Information Processing Systems (NeurIPS), 2019.https://arxiv.org/abs/1906.04584. [73] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. InConference on Computer Vision and Pattern Recognition (CVPR), 2018.https://arxiv.org/abs/1801.04381. [74]Shafahi, A., Najibi, M., Ghiasi, A., Xu, Z., Dickerson, J., Studer, C., Davis, L. S., Taylor, G., and Goldstein, T. Adversarial training for free! InAdvances in Neural Information Processing Systems (NeurIPS), 2019.https://arxiv.org/abs/1904.12843. [75] Shankar, S., Halpern, Y., Breck, E., Atwood, J., Wilson, J., and Sculley, D. No classification without representation: Assessing geodiversity issues in open data sets for the developing world, 2017.https://arxiv.org/abs/1711.08536. [76] Shankar, V., Dave, A., Roelofs, R., Ramanan, D., Recht, B., and Schmidt, L. Do image classifiers generalize across time?, 2019.https://arxiv.org/abs/1906.02168. [77]Shankar, V., Roelofs, R., Mania, H., Fang, A., Recht, B., and Schmidt, L. Evaluating machine accuracy on imagenet. InInternational Conference on Machine Learning (ICML), 2020. [78]Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. InInternational Conference on Learning Representations (ICLR), 2015.https: //arxiv.org/abs/1409.1556. 23 [79]Sinha, A., Namkoong, H., Volpi, R., and Duchi, J. Certifying some distributional robustness with principled adversarial training. InInternational Conference on Learning Representations (ICLR), 2018.https://arxiv.org/abs/1710.10571. [80]Sperber, M., Niehues, J., and Waibel, A. Toward robust neural machine translation for noisy input sequences. InInternational Workshop on Spoken Language Translation (IWSLT), 2017. http://workshop2017.iwslt.org/downloads/P04-Paper.pdf. [81]Srivastava, M., Hashimoto, T., and Liang, P. Robustness to spurious correlations via human annotations. InInternational Conference on Machine Learning (ICML), 2020.https://arxi v.org/abs/2007.06661. [82] Sun, C., Shrivastava, A., Singh, S., and Gupta, A. Revisiting unreasonable effectiveness of data in deep learning era. InInternational Conference on Computer Vision (ICCV), 2017. https://arxiv.org/abs/1707.02968. [83] Sung, K. . and Poggio, T. Example-based learning for view-based human face detection.IEEE Transactions on Pattern Analysis and Machine Intelligence, 1998.https://ieeexplore.ieee. org/document/655648. [84]Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fer- gus, R. Intriguing properties of neural networks. InInternational Conference on Learning Representations (ICLR), 2014.https://arxiv.org/abs/1312.6199. [85]Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. InConference on Computer Vision and Pattern Recognition (CVPR), 2015.https://arxiv.org/abs/1409.4842v1. [86]Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. InConference on Computer Vision and Pattern Recognition (CVPR), 2016.https://arxiv.org/abs/1512.00567. [87] Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A. Inception-v4, inception-resnet and the impact of residual connections on learning. InAAAI Conference on Artificial Intelligence (AAAI), 2017.https://arxiv.org/abs/1602.07261. [88]Tan, M. and Le, Q. V. Efficientnet: Rethinking model scaling for convolutional neural networks. InInternational Conference on Machine Learning (ICML), 2019.https://arxiv.org/abs/ 1905.11946. [89]Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., and Le, Q. V. Mnasnet: Platform-aware neural architecture search for mobile. InConference on Computer Vision and Pattern Recognition (CVPR), 2019.https://arxiv.org/abs/1807.11626. [90]Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., and Schmidt, L. When robustness doesn’t promote robustness: Synthetic vs. natural distribution shifts on imagenet, 2019. https://openreview.net/pdf?id=HyxPIyrFvH. [91]Torralba, A., Efros, A. A., et al. Unbiased look at dataset bias. InConference on Computer Vision and Pattern Recognition (CVPR), 2011.https://ieeexplore.ieee.org/document/5 995347. 24 [92]Touvron, H., Vedaldi, A., Douze, M., and Jégou, H. Fixing the train-test resolution discrepancy. InAdvances in Neural Information Processing Systems (NeurIPS), 2019.https://arxiv.or g/abs/1906.06423. [93]Tramer, F. and Boneh, D. Adversarial training and robustness for multiple perturbations. In Advances in Neural Information Processing Systems (NeurIPS), 2019.https://arxiv.org/ abs/1904.13000. [94] Uesato, J., Alayrac, J.-B., Huang, P.-S., Stanforth, R., Fawzi, A., and Kohli, P. Are labels required for improving adversarial robustness? InAdvances in Neural Information Processing Systems (NeurIPS), 2019.https://arxiv.org/abs/1905.13725. [95]Wang, H., Ge, S., Xing, E. P., and Lipton, Z. C. Learning robust global representations by penalizing local predictive power. InAdvances in Neural Information Processing Systems (NeurIPS), 2019.https://arxiv.org/abs/1905.13549. [96] Wong, E., Schmidt, F. R., and Kolter, J. Z. Wasserstein adversarial examples via projected sinkhorn iterations. InInternational Conference on Machine Learning (ICML), 2019.https: //arxiv.org/abs/1902.07906. [97]Wu, B., Chen, W., Fan, Y., Zhang, Y., Hou, J., Liu, J., and Zhang, T. Tencent ml- images: A large-scale multi-label image database for visual representation learning.IEEE Access, 7:172683–172693, 2019. ISSN 2169-3536. doi: 10.1109/access.2019.2956775. URL http://dx.doi.org/10.1109/ACCESS.2019.2956775. [98]Wu, B., Chen, W., Fan, Y., Zhang, Y., Hou, J., Liu, J., and Zhang, T. Tencent ml-images: A large-scale multi-label image database for visual representation learning.IEEE Access, 7, 2019. [99]Wu, W. Classifying images into 11k classes with pretrained model, 2016.https://github.c om/tornadomeet/ResNet andhttps://github.com/awslabs/deeplearning-benchmark/bl ob/master/image_classification/common/modelzoo.py#L41. [100]Xie, C., Tan, M., Gong, B., Wang, J., Yuille, A., and Le, Q. V. Adversarial examples improve image recognition, 2019.https://arxiv.org/abs/1911.09665. [101]Xie, C., Wu, Y., van der Maaten, L., Yuille, A., and He, K. Feature denoising for improving adversarial robustness. InConference on Computer Vision and Pattern Recognition (CVPR), 2019.https://arxiv.org/abs/1812.03411. [102]Xie, Q., Luong, M.-T., Hovy, E., and Le, Q. V. Self-training with noisy student improves imagenet classification. InConference on Computer Vision and Pattern Recognition (CVPR), 2019.https://arxiv.org/abs/1911.04252. [103]Xie, S., Girshick, R., Dollár, P., Tu, Z., and He, K. Aggregated residual transformations for deep neural networks. InConference on Computer Vision and Pattern Recognition (CVPR), 2016.https://arxiv.org/abs/1611.05431. [104] Yalniz, I. Z., Jégou, H., Chen, K., Paluri, M., and Mahajan, D. Billion-scale semi-supervised learning for image classification, 2019.https://arxiv.org/abs/1905.00546. [105]Yang, F., Wang, Z., and Heinze-Deml, C. Invariance-inducing regularization using worst-case 25 transformations suffices to boost accuracy and spatial robustness. InAdvances in Neural Information Processing Systems (NeurIPS), 2019.https://arxiv.org/abs/1906.11235. [106]Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., and Manning, C. D. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. InConference on Empirical Methods in Natural Language Processing (EMNLP), 2018.https://arxiv.org/ab s/1809.09600. [107] Yin, D., Lopes, R. G., Shlens, J., Cubuk, E. D., and Gilmer, J. A fourier perspective on model robustness in computer vision. InAdvances in Neural Information Processing Systems (NeurIPS), 2019.https://arxiv.org/abs/1906.08988. [108]Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y. Cutmix: Regularization strategy to train strong classifiers with localizable features. InInternational Conference on Computer Vision (ICCV), 2019.https://arxiv.org/abs/1905.04899. [109] Zagoruyko, S. and Komodakis, N. Wide residual networks. InBritish Machine Vision Conference (BMVC), 2016.https://arxiv.org/abs/1605.07146. [110]Zech, J., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., and Oermann, E. K. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study.PLOS Medicine, 2018.https://w.ncbi.nlm.nih.gov/pmc/artic les/PMC6219764. [111]Zellers, R., Bisk, Y., Schwartz, R., and Choi, Y. SWAG: A large-scale adversarial dataset for grounded commonsense inference. InEmpirical Methods in Natural Language Processing (EMNLP), 2018.https://w.aclweb.org/anthology/D18-1009/. [112]Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M., Djolonga, J., Pinto, A. S., Neumann, M., Dosovitskiy, A., Beyer, L., Bachem, O., Tschannen, M., Michalski, M., Bousquet, O., Gelly, S., and Houlsby, N. The visual task adaptation benchmark, 2019. https://arxiv.org/abs/1910.04867. [113]Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. InInternational Conference on Learning Representations (ICLR), 2017.https: //arxiv.org/abs/1710.09412. [114]Zhang, H., Yu, Y., Jiao, J., Xing, E., Ghaoui, L. E., and Jordan, M. I. Theoretically principled trade-off between robustness and accuracy. InInternational Conference on Machine Learning (ICML), 2019.http://proceedings.mlr.press/v97/zhang19p.html. [115] Zhang, R. Making convolutional networks shift-invariant again. InInternational Conference on Machine Learning (ICML), 2019.https://arxiv.org/abs/1904.11486. [116]Zhang, X., Li, Z., Loy, C. C., and Lin, D. Polynet: A pursuit of structural diversity in very deep networks. InConference on Computer Vision and Pattern Recognition (CVPR), 2016. https://arxiv.org/abs/1611.05725. [117]Zoph, B., Vasudevan, V., Shlens, J., and Le, Q. V. Learning transferable architectures for scalable image recognition. InConference on Computer Vision and Pattern Recognition (CVPR), 2018.https://arxiv.org/abs/1707.07012. 26 A Testbed overview greyscale imagenet-a imagenet-c.brightness.1_in-memory imagenet-c.brightness.1_on-disk imagenet-c.brightness.2_in-memory imagenet-c.brightness.2_on-disk imagenet-c.brightness.3_in-memory imagenet-c.brightness.3_on-disk imagenet-c.brightness.4_in-memory imagenet-c.brightness.4_on-disk imagenet-c.brightness.5_in-memory imagenet-c.brightness.5_on-disk imagenet-c.contrast.1_in-memory imagenet-c.contrast.1_on-disk imagenet-c.contrast.2_in-memory imagenet-c.contrast.2_on-disk imagenet-c.contrast.3_in-memory imagenet-c.contrast.3_on-disk imagenet-c.contrast.4_in-memory imagenet-c.contrast.4_on-disk imagenet-c.contrast.5_in-memory imagenet-c.contrast.5_on-disk imagenet-c.defocus_blur.1_in-memory imagenet-c.defocus_blur.1_on-disk imagenet-c.defocus_blur.2_in-memory imagenet-c.defocus_blur.2_on-disk imagenet-c.defocus_blur.3_in-memory imagenet-c.defocus_blur.3_on-disk imagenet-c.defocus_blur.4_in-memory imagenet-c.defocus_blur.4_on-disk imagenet-c.defocus_blur.5_in-memory imagenet-c.defocus_blur.5_on-disk imagenet-c.elastic_transform.1_in-memory imagenet-c.elastic_transform.1_on-disk imagenet-c.elastic_transform.2_in-memory imagenet-c.elastic_transform.2_on-disk imagenet-c.elastic_transform.3_in-memory imagenet-c.elastic_transform.3_on-disk imagenet-c.elastic_transform.4_in-memory imagenet-c.elastic_transform.4_on-disk imagenet-c.elastic_transform.5_in-memory imagenet-c.elastic_transform.5_on-disk imagenet-c.fog.1_in-memory imagenet-c.fog.1_on-disk imagenet-c.fog.2_in-memory imagenet-c.fog.2_on-disk imagenet-c.fog.3_in-memory imagenet-c.fog.3_on-disk imagenet-c.fog.4_in-memory imagenet-c.fog.4_on-disk imagenet-c.fog.5_in-memory imagenet-c.fog.5_on-disk imagenet-c.frost.1_in-memory imagenet-c.frost.1_on-disk imagenet-c.frost.2_in-memory imagenet-c.frost.2_on-disk imagenet-c.frost.3_in-memory imagenet-c.frost.3_on-disk imagenet-c.frost.4_in-memory imagenet-c.frost.4_on-disk imagenet-c.frost.5_in-memory imagenet-c.frost.5_on-disk imagenet-c.gaussian_blur.1_in-memory imagenet-c.gaussian_blur.1_on-disk imagenet-c.gaussian_blur.2_in-memory imagenet-c.gaussian_blur.2_on-disk imagenet-c.gaussian_blur.3_in-memory imagenet-c.gaussian_blur.3_on-disk imagenet-c.gaussian_blur.4_in-memory imagenet-c.gaussian_blur.4_on-disk imagenet-c.gaussian_blur.5_in-memory imagenet-c.gaussian_blur.5_on-disk imagenet-c.gaussian_noise.1_in-memory imagenet-c.gaussian_noise.1_on-disk imagenet-c.gaussian_noise.2_in-memory imagenet-c.gaussian_noise.2_on-disk imagenet-c.gaussian_noise.3_in-memory imagenet-c.gaussian_noise.3_on-disk imagenet-c.gaussian_noise.4_in-memory imagenet-c.gaussian_noise.4_on-disk imagenet-c.gaussian_noise.5_in-memory imagenet-c.gaussian_noise.5_on-disk imagenet-c.glass_blur.1_on-diskimagenet-c.glass_blur.2_on-diskimagenet-c.glass_blur.3_on-diskimagenet-c.glass_blur.4_on-diskimagenet-c.glass_blur.5_on-disk imagenet-c.impulse_noise.1_in-memory imagenet-c.impulse_noise.1_on-disk imagenet-c.impulse_noise.2_in-memory imagenet-c.impulse_noise.2_on-disk imagenet-c.impulse_noise.3_in-memory imagenet-c.impulse_noise.3_on-disk imagenet-c.impulse_noise.4_in-memory imagenet-c.impulse_noise.4_on-disk imagenet-c.impulse_noise.5_in-memory imagenet-c.impulse_noise.5_on-disk imagenet-c.jpeg_compression.1_in-memory imagenet-c.jpeg_compression.1_on-disk imagenet-c.jpeg_compression.2_in-memory imagenet-c.jpeg_compression.2_on-disk imagenet-c.jpeg_compression.3_in-memory imagenet-c.jpeg_compression.3_on-disk imagenet-c.jpeg_compression.4_in-memory imagenet-c.jpeg_compression.4_on-disk imagenet-c.jpeg_compression.5_in-memory imagenet-c.jpeg_compression.5_on-disk imagenet-c.motion_blur.1_in-memory imagenet-c.motion_blur.1_on-disk imagenet-c.motion_blur.2_in-memory imagenet-c.motion_blur.2_on-disk imagenet-c.motion_blur.3_in-memory imagenet-c.motion_blur.3_on-disk imagenet-c.motion_blur.4_in-memory imagenet-c.motion_blur.4_on-disk imagenet-c.motion_blur.5_in-memory imagenet-c.motion_blur.5_on-disk imagenet-c.pixelate.1_in-memory imagenet-c.pixelate.1_on-disk imagenet-c.pixelate.2_in-memory imagenet-c.pixelate.2_on-disk imagenet-c.pixelate.3_in-memory imagenet-c.pixelate.3_on-disk imagenet-c.pixelate.4_in-memory imagenet-c.pixelate.4_on-disk imagenet-c.pixelate.5_in-memory imagenet-c.pixelate.5_on-disk imagenet-c.saturate.1_in-memory imagenet-c.saturate.1_on-disk imagenet-c.saturate.2_in-memory imagenet-c.saturate.2_on-disk imagenet-c.saturate.3_in-memory imagenet-c.saturate.3_on-disk imagenet-c.saturate.4_in-memory imagenet-c.saturate.4_on-disk imagenet-c.saturate.5_in-memory imagenet-c.saturate.5_on-disk imagenet-c.shot_noise.1_in-memory imagenet-c.shot_noise.1_on-disk imagenet-c.shot_noise.2_in-memory imagenet-c.shot_noise.2_on-disk imagenet-c.shot_noise.3_in-memory imagenet-c.shot_noise.3_on-disk imagenet-c.shot_noise.4_in-memory imagenet-c.shot_noise.4_on-disk imagenet-c.shot_noise.5_in-memory imagenet-c.shot_noise.5_on-disk imagenet-c.snow.1_in-memory imagenet-c.snow.1_on-disk imagenet-c.snow.2_in-memory imagenet-c.snow.2_on-disk imagenet-c.snow.3_in-memory imagenet-c.snow.3_on-disk imagenet-c.snow.4_in-memory imagenet-c.snow.4_on-disk imagenet-c.snow.5_in-memory imagenet-c.snow.5_on-disk imagenet-c.spatter.1_in-memory imagenet-c.spatter.1_on-disk imagenet-c.spatter.2_in-memory imagenet-c.spatter.2_on-disk imagenet-c.spatter.3_in-memory imagenet-c.spatter.3_on-disk imagenet-c.spatter.4_in-memory imagenet-c.spatter.4_on-disk imagenet-c.spatter.5_in-memory imagenet-c.spatter.5_on-disk imagenet-c.speckle_noise.1_in-memory imagenet-c.speckle_noise.1_on-disk imagenet-c.speckle_noise.2_in-memory imagenet-c.speckle_noise.2_on-disk imagenet-c.speckle_noise.3_in-memory imagenet-c.speckle_noise.3_on-disk imagenet-c.speckle_noise.4_in-memory imagenet-c.speckle_noise.4_on-disk imagenet-c.speckle_noise.5_in-memory imagenet-c.speckle_noise.5_on-disk imagenet-c.zoom_blur.1_in-memory imagenet-c.zoom_blur.1_on-disk imagenet-c.zoom_blur.2_in-memory imagenet-c.zoom_blur.2_on-disk imagenet-c.zoom_blur.3_in-memory imagenet-c.zoom_blur.3_on-disk imagenet-c.zoom_blur.4_in-memory imagenet-c.zoom_blur.4_on-disk imagenet-c.zoom_blur.5_in-memory imagenet-c.zoom_blur.5_on-disk imagenet-r imagenet-sketch imagenet-vid-robust_pm0 imagenet-vid-robust_pm10 imagenetv2-matched-frequency imagenetv2-matched-frequency-format-val imagenetv2-matched-frequency-format-val_subsampled_class_1_8 imagenetv2-threshold-0.7-format-val imagenetv2-threshold0.7 imagenetv2-top-images-format-val imagenetv2-topimages objectnet-1.0-beta pgd.l2.eps0.1pgd.l2.eps0.5 pgd.linf.eps0.5 pgd.linf.eps2 stylized_imagenet val val-on-imagenet-a-classes val-on-imagenet-r-classes val-on-objectnet-classes val-on-vid-robust-classes val-on-ytbb-robust-classes val_subsampled_class_1_8 ytbb-robust_pm0 ytbb-robust_pm10 Evaluation Setting efficientnet-l2-noisystudent FixResNeXt101_32x48d_v2 FixResNeXt101_32x48d instagram-resnext101_32x48d efficientnet-b8-advprop-autoaug BiT-M-R152x4-ILSVRC2012 efficientnet-b7-advprop-autoaug instagram-resnext101_32x32d BiT-M-R101x3-ILSVRC2012 efficientnet-b6-advprop-autoaug efficientnet-b7-randaug efficientnet-b7-autoaug efficientnet-b5-advprop-autoaug resnext101_32x8d_swsl instagram-resnext101_32x16d BiT-M-R50x3-ILSVRC2012 efficientnet-b6-autoaug FixPNASNet efficientnet-b5-autoaug efficientnet-b5-randaug resnext101_32x4d_swsl efficientnet-b5 pnasnet5large instagram-resnext101_32x8d efficientnet-b4-advprop-autoaug efficientnet-b4-autoaug BiT-M-R101x1-ILSVRC2012 nasnetalarge efficientnet-b4 resnext50_32x4d_swsl resnext101_32x16d_ssl resnext101_32x8d_ssl senet154 resnet50_swsl efficientnet-b3-advprop-autoaug efficientnet-b3-autoaug resnext101_32x4d_ssl polynet BiT-M-R50x1-ILSVRC2012 resnext50_32x4d_ssl inceptionresnetv2 se_resnext101_32x4d efficientnet-b3 inceptionv4 resnext101_32x8d_deepaugment_augmix resnet101_cutmix efficientnet-b2-autoaug FixResNet50CutMix_v2 dpn107 FixResNet50CutMix efficientnet-b2-advprop-autoaug dpn131 dpn92 resnext101_32x8d resnet50_ssl dpn98 google_resnet101_jft-300M FixResNet50_v2 se_resnext50_32x4d FixResNet50 resnext101_64x4d efficientnet-b2 wide_resnet101_2 xception efficientnet-b1-autoaug se_resnet152 resnet50_cutmix efficientnet-b1-advprop-autoaug wide_resnet50_2 se_resnet101 resnet152 resnet101-tencent-ml-images resnet50_feature_cutmix resnext101_32x4d resnet101_lpf3 efficientnet-b1 resnet101_lpf5 resnet101_lpf2 se_resnet50 resnext50_32x4d resnet50_augmix resnet50_mixup fbresnet152 resnet101 inceptionv3 densenet161 efficientnet-b0-advprop-autoaug resnet50_cutout FixResNet50_no_adaptation dpn68b resnet50_lpf5 densenet201 efficientnet-b0-autoaug resnet50_lpf3 resnet50_lpf2 resnet50_trained_on_SIN_and_IN_then_finetuned_on_IN resnet50_deepaugment efficientnet-b0 resnet50-vtab-rotation cafferesnet101 resnet152-imagenet11k resnet50_aws_baseline resnet50 resnet50_imagenet_100percent_batch64_original_images dpn68 resnet50_deepaugment_augmix resnet50-randomized_smoothing_noise_0.00 densenet169 resnet50-vtab resnet50_with_brightness_aws resnet50_with_spatter_aws densenet121_lpf3 densenet121_lpf5 densenet121_lpf2 resnet50_with_saturate_aws resnet50_trained_on_SIN_and_IN resnet34_lpf2 densenet121 resnet34_lpf3 vgg19_bn resnet34_lpf5 resnet50-vtab-exemplar nasnetamobile vgg16_bn_lpf5 vgg16_bn_lpf2 vgg16_bn_lpf3 resnet50_with_frost_aws resnet50_with_jpeg_compression_aws bninception mnasnet1_0 vgg16_bn resnet34 resnet18_swsl resnet50_with_gaussian_noise_aws resnet50_with_gaussian_noise_contrast_motion_blur_jpeg_compression_aws mobilenet_v2_lpf2 resnet18_ssl mobilenet_v2_lpf3 mobilenet_v2_lpf5 vgg19 vgg16_lpf5 vgg16_lpf3 vgg16_lpf2 resnet50_with_contrast_aws mobilenet_v2 resnet50_with_fog_aws resnet18_lpf3 vgg16 vgg13_bn resnet18_lpf2 resnet18_lpf5 resnet18-rotation-standard_40 resnet50_imagenet_subsample_1_of_2_batch64_original_images vgg11_bn resnet50-randomized_smoothing_noise_0.25 vgg13 googlenet/inceptionv1 resnet18 shufflenet_v2_x1_0 resnet18-rotation-worst10_30 vgg11 resnet18-rotation-random_30 resnet18-rotation-worst10_40 resnet50_with_pixelate_aws resnet18-rotation-random_40 facebook_adv_trained_resnext101_denoiseAll resnet50-smoothing_adversarial_DNN_2steps_eps_512_noise_0.25 mnasnet0_5 resnet50_with_motion_blur_aws resnet18-rotation-nocrop_40 facebook_adv_trained_resnet152_denoise bninception-imagenet21k resnet50-randomized_smoothing_noise_0.50 resnet50_with_greyscale_aws resnet50_imagenet_subsample_1_of_4_batch64_original_images resnet50_linf_eps4_robust facebook_adv_trained_resnet152_baseline resnet50-smoothing_adversarial_DNN_2steps_eps_512_noise_0.50 resnet50-vtab-semi-exemplar resnet50_with_zoom_blur_aws resnet50-vtab-semi-rotation shufflenet_v2_x0_5 resnet50_adv-train-free resnet50-smoothing_adversarial_PGD_1step_eps_512_noise_0.25 resnet50_trained_on_SIN squeezenet1_1 squeezenet1_0 resnet50_l2_eps3_robust alexnet_lpf2 alexnet_lpf3 alexnet_lpf5 alexnet resnet50-smoothing_adversarial_PGD_1step_eps_512_noise_0.50 resnet50-randomized_smoothing_noise_1.00 resnet50-smoothing_adversarial_DNN_2steps_eps_512_noise_1.00 resnet50_imagenet_subsample_1_of_8_batch64_original_images resnet50_linf_eps8_robust resnet50-smoothing_adversarial_PGD_1step_eps_512_noise_1.00 resnet50_imagenet_subsample_1_of_16_batch64_original_images resnet50_with_defocus_blur_aws resnet50_imagenet_subsample_1_of_32_batch64_original_images resnet50_imagenet_subsample_125_classes_batch64_original_images resnet50_imagenet_subsample_250_classes_batch64_original_images resnet50_imagenet_subsample_500_classes_batch64_original_images Model 878575488882808194868588886850889998931e+02 1e+02976864 83688682868185808478827586798577847281537225837480677252623851278579745875676453392486788575857083657950807471626352615056448580827176576743452183778072756264474426685425179.47572756572596145452782758173807075636953847781707456634054308378837677717262695486808376868182777770837679707360584144277672705770596149584185818276777168646254837781757364675557447468655860505243453680598672787893848487875838869897901e+02 1e+02966958 83698682868185808378827586798577847281537225837480677252623851278579735875676453392386788575847083657950807471626352615056448580827176576743452183778072756263474326685425179.37572746572596145442782758173807076636953847781707456634054308378837678717262705486808376868182777770827679707360584144277672705770596148584184818176777168646254837781747363675557447468655860505243453680598672787893858488875837869897901e+02 1e+02966958 816185828582848183798076858084788375796067328076757164595247403682796258716959573329847883768171796570507975696460555853524784807974706458523530817877747066575336347160342515737371686764545137357978777675747068625983787873686154474437797980787374666764618581827785828177767181787672686451483634757368596960605056458481807777706764605481787976696761595049857066615953514646388059857377779284848787595712189.439859897891e+0299966857 8347858385828481848082778480837881727149421684768271805978467433857980638474816572378378827581708065765182787668695967576250858083738163784970298481847882727962694276673928178378827579737263614585808479847884758271858084768366805178408581857984768370836685828581858283808076848083778272785872448078766674686659644985828379827780747768858184798274816978617673706661595352444358468373767692838386865347859797891e+02 1e+02966860 8059857984798378817579728577847683717855692680717766725566435932847577567765695349248475837280677662695082757666715870566650837679697457704657268276807376667055583867562820157974777076666955594181757974797377687362827580697660684762388277827680727762765383758174847780727566827679717665685159407873736171636254614684788274817274706864837681747767736168537568686158535147423955438673757592828285855037859897901e+0299967362 8247858285828481837982768479837781707143391183748268795474396926857980638372796162298377827481698063774781767365665564535846858083708058754464228479837581677751682774622920138376817179677451662884798478847783738168847984738260784074288479857883728160814985818580858283798074847983748166764670297976746071646155584484828278817478727565847983788170806277517570656155524644373654428473767692838386865281601842859796891e+02 1e+02966759 815885828481848082788076858084788374795967317975757164585146383481796158716958573228847883768171776568507974686459555753514683807974706358503127807876736864545133317160332313747370686662534935327977767575736866595682787772675952444134797979787273656662618580827784828177757080777571676349463432757369596859584855438381807678706665585380787875696562585248856965605953514546377958846977779384838686585613181039859797891e+0299956857 79528478847883768173776984768374816974506020796976647051623953298374765573606346401983738170786372575642807272626653645260458276786773556642512380757771716362504531624718139.87772746770626049463380747972787075657058827379667454653958308276827579697458725083748071837779707161807476697162594648327569705566575547543983788073796969656357817479727264675758487063615450464239343151378472747392818185854630859796891e+0299967261 824285828482848183798176837982768070684536138274816877547240662784797862827277605628837782748069806477498076726464546252564584798271795874456124837982758066765165277363292012827581707966735362308479837783778273806884798374816276437231847984788274816680558481848084828279797483798273806575466831787573627163625359438481817880747871746583798377816879617650747066625653464537365441837275769283838686508042621844859796881e+0299966859 81428582848284808378827483798276796968474320827481697856754370338479786180707556572883768273806879647549807572656655645359478480827279607547632883798275806974586139705623168.98376817279697359674383798377837681728067847983748163784875388379847883748167795583808278848283788173837782748068745569427775736069646155614284828078807478707564837782758169796476577371656356564748404052418170757591828286865372391043859796881e+0299956757 8040848084808378827680728376827278616429306.18169786273456730591884777758796270464417827381708064795874407569625353415139453283777965744968345315817579697657694054186445105.43.68272786675616644542382758272817180657858837582667951743068188377847581717961795084788476848081767568817479677656683557207470664762535143412483798074786976677258827380717760745269416964585346433736282846327865747592818185855067359.333849796871e+0299956554 813684818481838082788176837882767969664535138173796774546840602883787760817276625532817580727767756269478075726463546152554584788069765669425323837881747866725158247363322315827580717867715158268378827782768172796883788273796273446832837983788173796678598480848084818278797383788173786571456228777472617062615358448381817780747669726183798277796977627351747067635855504741395442807175759182828586508143621545849796881e+0299956658 795184818481838081787974847983778273765863307974747063585246393480785956706856562927827781757970756365487774666257535651504583797873696258503429797674716562504831297058322414707166646259484632307876767474726765575681777671656051444135797879777373666764628479817684818076736878767369646045433230727164556557554551418280777575696263555279767673646256544644716864595751494444367656836975759283828586565513159.338849796881e+0299956956 805384818481837981777874847983778273765762287874746962564943363180786057706858563227827782747969766467497773666256525451474482797772686156483125797775726663525032326956292213697165656059484732327876767474726866595781777671665951434134787879777172646560608379817683808076736979767470656246453232737267576657564652418180787574706361565079767674666458564846846864595852504444377958836976769283838686565621231738849796881e+0299957058 784684778376827479717666837483718164744259147866755867425830482182727452705556382811827081657858765267367869705762476146553982747762714763334516807377687260644749295944171397669746371586346503179727870776874627054807177616946572948208074817377657049653681717867837577687058807176677259614251297265654862514941452983757869776360585249807179697360685361436558534843403433282648348270737392818185844127849796889999957058 8037848183808379817779738377827378646232266.88170796374466731602084777658786569504220827481718065795874407571635653445142453484788167765170365517817679707659694055156647137.75.3827379677762684554208276817381727967765983768268785372346623837783758071786177518479837783808076756981757968765768355919747167516356534543328380807578707667725982758172776174537040676456544444353728294835806874749182828585507021378.935849796879999956554 78338379837882768073776983778375837180547022806777617145653056198375745777606745341582748171796475576539767064565546534346378374796472506434471381737865745364374719624517116.977687560725363364820817280697967766171538272816275436324531683758373816878597654817479718277777169628071786373516132492073666547625051404929827778717664665959508172796973556846603570646054494540383231423179647373918181858544599.6157.630849696879999956550 7932848083798278817579728276817277646234287.778707563684760345023837675577763674938208174807077647458674275706355534351414534837676656950603843208175786974586540451866501484.6807276667161594343218175807380717867756082758068765568366125827682747969756073498378837783798075756881747867735662355021747066516355504440318379797477687465695681748072756171536641686458544845393730304936796673749081818585497017367.438849696879999956453 7932838083798278817580718175797174615534199.7797076647150633853268377755877666952482480727869756271576139777167585847564550388277786772536340452281767971726159464823675318106.5807373686463514939298175807280717866756082768070775971426531827782757971766273528277807583808177777181747868726065415727727066526456564656358279787476697466695881738069756073546944696660575249434234344938796773739081808484507021379.140849696879999956554 784283808379817880767772827782758071745459277671716658524640342878765754686655532927817580727767726059427572635953495047444181777569655852452925777472686257474326246755282011676864615955444126257674737171696462545179757369615645393529777678757171656562608278797482797874716676737066615542372725696961526253514247378179767373665859494877747471625753494237676460555347453939327354826774749182828585535411148.335839696879999956755 7726837982788176797375688073776771564726114.6786875616845593249218274745575626548422179707766745870516033746861525140493842308275776469506036421879737666715462384719644818116.7777174647257644151228174807179707664725682737966755367376026817481737868765974478074777182777771706379727664715360334920726863476052484042278278787276666962625380737869725768496137676359545146434033324532776373739080808484477322388.733839696879999956453 77238279827881777974767082778276817378596628776973636451563845278175705675656451322182748171796475566637767164585448524645398275756566545543372279737668705859434025665222159766973636957584341267974787177697364675681747866705056314521807679757771736570618075777282787772696478727566695655384225726864506353524247338178777276676661595179737770695863505539836560565148434137334433796272728980808484466516288.134839695869999946551 784582808279817880767772827881768071745561287772726760544841363078765855676655543027807579727767736062447571625952505048444181777670665853463125777473696359484527266754292112686865626056464329287674747272706664565479757368625648403931777678757171646460598278797482797874716677747167625843402927706962546254524347387978747273675757474677757572646156534542836559555347454039327655857074749181818485535423251936839696879999956956 792783808279827880767873817679737565603830107770746468506037502681767256787073594829797377697563725865437873696260515849514282767666705263394521817679727663704857257161332415807378687663694757268176807580737869756481767971756068416029817782767971756474568279827882798076767181767971766268435928747268576658554855418279787477707265675681768074776674586947716762595551474439365340796672728980808484487943621943839695869999946554 78278278827781767974777081747970746155321977668716062455232422181757155736160452817807178687661735464367468615351414939423281747363654954363618797376667054613542136446126.64.5797075647058604144187973787177697564715880737866725160325021797480727567695767478276827482787874736678717564705158304616716764496252534248328177777275657160655079727769725668476235656154514643373530284834766372729080808484487220378.236839695869999946252 753782768175807377697364827481727966734757177865755766405628461981717252715458382812807079657658745265357667675558455743513781737761704662314415797075647053613946216045161297463715668516037472178707768766572596648797076617047603151238072786975606843643280717667817475676657786974626952583448217063624659504740453081757667756159545043786977667055644756376356524641383431282544308167727190808084843927839696879999946856 77238279827981788076787382778276817277586427776872626349533641248175695575646349301981748071786476586940757164595448514645398175746565535541362079747669716061474428665221158.6756772626958594544287974787277707364685780747766705055304219807679757771736569608277807482798075756878737567705858414628726864516253524349358178777275666460574979747770705965505839836253514443343529284533786272729080808484476925401437839695869999946551 7524827881777975777273677972766770554925134.47666715962435129401981747154716059463320786976657357695060327266585048384636392981747362634753333516777173636650543434166144159.36.2746871596752553538197972777076687362685480727764714960305021787278707465685666477973766981777570676177707261654950293616696660465849463841268177767073646659585077717567675360445132656054494640383331264430756171718979798383467225429.432829595859999946250 76338279817880777874757081768074786870475120746968625447403328237775555263614747222180737870756370565737737060565045484341378076736661524738231875726865575340382020644922158.4646359555349373520207573726969676158494677737064554938322923767576746969616158568176787281787773686374716662555135332121676657475748453640317978737070625454444275727168575448443734636154514843403635296950826373739080808484495112128.829829696869999946551 743382788178807678747570817680747868715055237471696658544743363276745753656449532027797277697463705658397270605850484846423979767368635851462825757470686059454525266654282113616761615756434325277473717069686361545277747269625847433834767576747170656562608176787281787672676374726766575639392527676758515951473942347977737070644954394374737069575749493939686560565449464241355035806671718979798383475114158.830829695859999946251 742982788178807678747470817680737868714956217470696557534641353076745753646447521625797278697463705656387269595749474745403879767368635751442824757370675957454425266654261911626562605856434325267472717069676260525177747168605645403530757575736970636459588176777181787572676474726765575639392527666657495750453841337977737070634954394274727069585750494039676459555348444040334934786271718979798383465012138.629829695869999946250 7319817880767874757170668171746666564630167.37067626046473234202378756458666451512425816972656758635250368167515238413538283181736463495036381820817167635450353414156450241711816964615552353315168171706868666160505076726965575340362926817373715463364929388173747079767168615881706461524931291616646552445348413533268177747071635355434481716967545343433031816459555147434036324230745870708878798383436216369.229819595839999946147 75308177807779757772736880747972776667454618736866615245383126207673535061604445181978727768736268555537716857544743454138347975726559514637241773706662524934331515624721159.2636357535046323016157371706868656057494575716661494331252116747375726667585954538075767180777571666172696260494728271616646354455445423236277876726870604849373673706866515041392928626053504542373532276850826372719079798383485213128.828819595859999946550 7717817880778076787476707974777071615233198.17366686058454733362379746653746765563628777075667258705363377571645854475144453779737162624952353318787475697061604741246959332416777073657061594642267874777276717467706178737567685455364425787579737466665664478177807680777874746878737568695957424628706863536154504250368077767174666760614978747772716366566046676358545046413933315037776570708978798383447947621640819595849999936353 7518817780767974777275677971766770564827155.871646457514340312921787365526858514118137768756473577051633470655648453642333626787167595545433325177671726364485029299.76346148.15.6766971636655513332127671756974677062655578717363624744293319757376706965615759488075807380767671706376697160644648243412686458445848473740267976756972626657584676697466675261425230625750474239333226254533766170708878788383446917347.335819595849999946150 732481788077797576737369807579727767694652187368676255494336322675735551616144481421787176677259675354367168575547454543393679747166615347392520737167645453373816196451241710616258565251343616207372706968676159514976737166595444383428747473726768596255568075767080777471656272706462525131321720656455475548433639307876726968614752374073716967555345443333656258535246443939324731776270708878798283455111128.528819595859999936250 7320807679757873757071657770746568545128236746470576141502939197972675267585243241776677361705468485830716657514639433636297972736064445430321475687058604345272412614619127.9726167506142442625137770756773656757574678707461644549283819777076676959544245297771746879757367645875676956604140232513646352415043373131217976746971636056534576697364634756374426816054514543383633294229745770708878798283455412197.928819595849999936147 72258072797177687464705880697966776070395312745770485930481836117967694764484932198.178657760745271466230726161475337513545297968735264355422339.3766372566544552838115034118.35.870566747643953243810766574627360685260417663725264365020401376677764725765446330776673617870716060487561705363415022381165575639534240313722797172617052514541337562735864445735482557494539353127252319392576616868897777828235296.67.76.222809595849999946351 72208077797678747571716779737871766466424615716765605145393228227572545058584044121877707566715966515133696654524342414034337873706458504536231871696461504831321215614519137.7566052534547283012157270686666645755464475716863534834302420737373716667585954547974756879767369635970676158484525261316626252425244403234267775706767594348323671696664504739372726626054504742383534284531776070698877788282434712118.824809595839999936047 75158076807579747772756980747972776768474417736467575643443033207871625070615747251879707867755972516334726657524641433837317971716058464734271776707263655355403824634920137.9726268566451533838247670746773656959625076707161604545283519787376717367676062547974767180767772726575697162655251364024676358445446433542297875736771605751493976697366645357454834805751474439363229264433765970698977788283446525361432809594839999936550 7114 807779757673736968637569716363514022123.96764605845453132192177746157666352522426736769626454594845326766505138403538283177726561504936371720757068635349322911106350251811676559565149312913117270696867656057484575716964585343383328747375726566545753497571726878757066595573696460494624221211616250435146403534287876716868605053414073706965494836352423806257534946413935314027725769698778788282416117261126809494839999935945 72138076797577727469706376697263645238227.73.8716464565140392828197772645266595245272174667260685265455627696353474335403333267771675956444432261674686860584642282111604619138706565555947422823127571746872666859615076707363655152354326757075676962625560487670726779747267645873676657574437222311656255425445423334237875736870606054514373677063584851373925645854494641393431274229745968688877778181437122359.231809494829999936150 75168076797578737771746780737971786772495318726266565540432731177770614865574842171479707867765973526534716656514540433836317870695858444531251475687162645052373420604517115.769596552614749343321756973667264665759477669716059454329332177727671736767596253797476708075777070627467695963484832362264615542524240313825787472667058544944367467726363505742493380565446474040333327423275596969887878828243621625928809494839999936449 7211807879777775757373707875777476717161624178747672716562555143757554537271656442417471726767626457534474726563575555534947797777737367686051437775757271686661554874684840297574737071686661575075737371727067656159797677747469665657447977797777767372676676747371787673716966777574727168646057527170635962595349504578777572746864615750777576747269696564607069636159565149473952387967696988777782824144809494829999936754 71127975797477727469706578717769746263394012696361574642332923197470534957564043121776687463695664505133686451494037383530287670666052463733181669676159484630301213584418127.9585552484443272712146968666463625454414273696661534737302721646968664656294125317872746778747168605768655956464325241314606049405143383132257774686464544745383368666361484538352725805852484440363331263928735568688777778181425515221024809494829999936248 7310807679757873767072667770746568554827155.5686360554640332823197671584965584942161376677463715569486132686352474035383232267671645851443832201673696861604946312612634816105.8746670596552513330137470726870666660595275707062584741293019747170696561555250467974797279757469686173686859604744273014646254415444423332247874736770606253534174697265645357444833615649454237343127234330735868698777778181417016316.933809494829999936149 7013 797378717669726467587867776273505820324.27054644348263415248.777676146524831337.11077637658744971426225655946423330312724217766684853303818176.769596046452726128.63.64731116.9549404726402022107.93.6716367596456514533327362634846292715179.456625456284412275.81578697361787169615447675757414325219.7104.5525438304134272421167671635958453237232568596251453034202312555143413634292724223422735267678776768080303911148.315809494829999935943 739.98076797577737570716574687063605034227.53.96361555541412829171973715149636048481921716465585849544339286967545242413939323274706158444228281113747069656055444123226352261911686363575852423823227371716969676461545373706562514733292220727273716064455138407672746978747369676273706863585438362323616151435145413538307775706766584950413974717168605651473936805850484240343330264331765968688677778181416920268.629809494829999926450 7012 797378717669726467587867776272505820324.27054634348263415248.677676146524831337.11078637658734970426225655946423330312724217766684854303818176.769596046452726128.53.64731116.8549404726402022107.83.6716367596456514633327362634846292715179.456625456284412275.81578697361787169615447675757414325219.7114.5525437304034272421167671645957453237222468596251453034202312555143413633292724223422725267678676758080303911148.415809494819999935943 7411797679757873767173667771746767574429157.26862625549413829271976715850686255482218756772626953674758307166565044384136342976706658544443322516757071656453513732166553231611746769606453513634187571746972686963635675707063594842293019757375716863565150437974787479757571706475707063635247323419656455455446413535277875726870606154534176717468665759485137625951494442373531274432756168688877778182417741551234809494829999936349 729.47976797577737570716574697063625236249.54.46360545339382625161673715150636049492223716566596050554541306866535142413838323275706156433828251211747069655955444021216149231610676362585753413822227270706868676361545373706662514733292219717271705762435038417672746878747369666273706763575437352123626252445246423639307775706766585150423974707167585648473635795849474239343229254330746068688676778181406820258.230799493829999936450 7011 7975787476727368696374687062615036238.44.663615654403926271617727052496260474818217064665859495442392667645049383835362929747061574340282711127268656151473128109624924171161575449474227249.98.9717069676665595746457269666152473531252269716969556243504042746971657773706661577167625848442421111059594840494338323226777468656354474637347268676450473835252379574947424034333026413074576767857676818140621418825799494839999936248 7010 797577747571726767617468706262494123155.1686361574944363124217370535062604949232571646758615057444327676450493837363529287670656050463533161770686260464425267.29614823171161605452444421246.59.87169686666645857454574696761554638292720687070685261404837387469716677736965565369675956444220217.911596045384842353027217674696665574549373869676563494638352524795853494741403535284329735768678676768181415814209.829799494829999936045 7016 7976787576737470706578727769746262393813706564594943363024207371514957563942111675687364695663494831686553514240403833317672686256484334211570676158454425277.811594418127.654564948424123257.8117068676564625552434072696360464328252017727172706365535547487873736778757168615768655855434120218.612606049405042373032247674686564564246323269676461474436332323595850484440363332274429755868688776768181415012129.123799494829999925845 729.27975787477737469706479686963605135248.64.46360545238372525151673715149616047471921796466595950544439297966535142403938323179706156423827251111796968645853423921216149231711796160555450393621217970706868666260535273696662514633292320797172695761424840397971736878747368666279696662565235332021616151435145413537297974696665565050423979697066585449453634795749474038323127244330755968688676768182406819258.229799494819999936449 739.97975787477727570726579727871776772525823795963525137392428157969594565555041191579687663725568476031796555504339403634307969685554424229241179677059624748312816604518127.7795263465840442726147968726570626555574579676958584242253115797275707165645659497974766979747569676079666858594442263015796051405042383136257973686466544545373379677162594651363925795446443835292823214532765668678776768181386125201.326799493829999936348 7011 787177697566726267557765755970465317263.17253674355244213297.177656044534532297.57.77661745570466740582565584740352832262519786571466027461522568585845412721135.43.445289.66.84.8543947263819189.35.43.2726170576754574443317461664652283414249.36461695747483331291876667160776969595547675656414025189.47.43.950524030383327232516746959575445283619236858635144303320211253494039343127252320362369516767867575808034397.78.17.215799494819999935841 6910 797677757572716866627969706362513822103.866635957434329301820767361576361484918227965676062525645412979644648333530332326797164605148373616177969646049433126119.66147211497961565248432924119.6796968666664585546447469676254493733272479727371646552555147797070677774686656547967615747422521111059614639494436322823797568676457454936367968676449463735252279615452454438373230392572566868867677818140581419826799493819999936046 7011 7871776975677262675577657559704653172637253674355244213297.277666044534531297.57.77661745570466740582465584740352831262519786571476027471522568585845412721135.43.345289.66.74.8543947263819189.35.23.1726170576754574443317460664652283414249.36461695747483331291876667160776969595546675656414025189.37.43.950524030373327232415747059575446283619236858625144303320211252494039343126252320362369516767867575808034387.887.215799494819999935841 707.3 79767875767374707066777276697363624240156663585745443332222272705148625948472122756872646656604845326965545244414240353374706359474732351418716865615450393520186148231610636058535247363320187169686766655958474773696561494431272118717172705662414837397773736878747268635979676258514731282016626152435245403336297974696665574648373579686664504840372826795851494441363432284330735767678575768080385813158.628799494829999926047 708.7 78757773757172676761736669606147342073.6666158534438312622177470584964584943201872637058664963415323666149443732353028237569625648403528191472666658544538281711594418127.2686362545646402921137368716569636356554674686860564540292920716871666359545049437468706477736964595470656355534333231912625949365040372929197773726568575850503970656660534444343323605549454238363129243926715767678776768080407122357.228799393829999936047 707.4 787477737470706665607367696159493623125.6656258564441323022217269514861594648182270636657594853413724656247463535333327267469635948453433151769676058444425278.211594722161059585251434423279127068676564625654444373686560524636312622796971685359384442357968706576726561534979655655424119219.212555741364239292725217973666361524143323079656260444333322121795752484641383433284028725567668676768081406013208.224799494819999936146 729.1797478747772756972657972777076656947471770606352513637242615766860466455463914137967766273536945592679635247403537333126796867555439412721127965675657424125209.85641148.64.87955594654393722189.47967716470616452554274666856554138262918797074687063635558487973756978747568695879636552563836192210615850364739342831217973706467545245423379646959574149313721795347424134342827224130735667678777768181416314226.427799394829999926348 728.4787478737672746971647569726464534124114.46561565441392927191873705347615643401413726569616453594744306662504539333631302574696056464233301715716766595646422921116045148.35.1736667596251443124127269706668656359565173686760524434262317717072676260525149467873777177747268666071666557564538252513636052405243423333247773716567575951503972677064605153434432575445443936312925234230705867678775758181387016316.431799494819999926047 689.3 787677747572716866627369696361514126156.86763605746433331222075735956666454532627706566595951544439286564474935373235252876716560534841362117726964605246352916126250261912646157524944342818137169676665635653434073696762564941343125737273716163465046427370696676746765565371686258514531251814586045404744353330267675686664564649363571696765524943393127796054514643393633294027735668678776768181386416258.627799393829999925945 697.3 787377727570716666617769756571555528297.26559565239352623171572685247525333389.21474647159665161444626646146443333313124247468625546403126131266625653413723227.98.7533815117.349484539383320197.58.9676562625959465030387065595541382422151451655162264911295.2177770726477726864555264605348393419189.19.8555541334436312524197572656059483639262665615855413731272018795448443936323027243524725366668575757980385020251119799393809999926146 729787578747673747071667570726664554129167.46662595545413328231974715348676254482119706566606052554540296965545043384035332874706458514439312116746969636152473627176453241611726666596052453528177471726971686762605574706863575042343224737274706362515047437874777377757470686273696862605143333120636253435345423337277674716668585852493974707267635657484838615851494441363428264331726167678575758081387643551233799393809999926048 696 787476727369706564587865675957463419114.26561575441382927181872685147575640421417786164555745523937237861474434333231252478696257464231291314786558554240232287.8584419138.878544845403721218.38.67866656362615253384171666256453928231916786867654756324228337867696475726561524978635652403819188.89525438323935272320177873646258503640272878646259444134312320785548454137333029243925715367668675768080386114218.322789393819999925845 688.5 787576737471716765617268686259503825156.66662595545413230212075725955666454532828706465595951554540296564474834363234252776706558514638351918726864605145332813116148231610646157525044312714127069676564625552423873696762554937322723727173695860444741387369696676736764565370676258504428231412586045394643343129247674696665574548353571686663514841372825786053514644383733303925725567678776768080386417257.127789393819999926146 696.1 787476737470716766607367696061483820113.9676360564642332921187270504861604647182270636557594853413825666349473736343327267570655951453632151568665957424320254.69.9604621159.557564947384017234.2117069676665645657424573686660534637302721666968674458294226337368696576736863565267655755404115215.111585844364540322825217573686564564346353368666462474535352124785650464438363131254129735767678675758080405911137.326789393819999926147 708 7875777475727368696378727769766371445718616150543439222714187269504662594345131773656959635058424224676451493937363530297369555840442731121467655756404020235.49.563501912747564345363617194.88.7696865646262535441437167625846412823191568706969525934432629777173667774726764586763555339391720811595848374837372631217673686464554545353368656360444333342124545444433636282924222872546667867575808040247893819999926045 687 787376727570716666617669746570555428297.7655956523936262417157268524852523438111574657160665260454529646046443433313124247468615545403027121265625252323413172.95.25338149.66.549474038293111152.856665626159584749303669655854403723221514526460623551223415257770726477726964555263604948313211144.46.3545541344438312725217572646159493639262665615855373526251616785546453938323128253625725366668475757980384919231119789393809999926044 695.9 78757774767273696964777175697262604136156361565543433232212271695048605846462022746871636555604945326764535042403938333273696057444530351419706764605249363417166047221610615955514945333117167069686665645757444672686561524735312521707069695561414932397772736777747167625869666158504629281716606049414944383235287573676463544747383570676563504839382827785750484340363331274229735766668675758080365611148.627789393809999926046 696 777476727470706665607366696161504124176.2656258554342313019207269514856563843131870636657604954423826646147453434323225257569625845453034131767655655403821206.36.2584520149.555544743383519196.57.469676664646256544543726763584842312622176669666648583545343773686965757266625350656352513634151466.7555740344137282521187573666362544447363566646058403829271817785750474339363231264028715566668675768080395811147.724789393809999925945 687.2 7773767274697065655971646657554223132.81.8635954523937272618177268554861574643201868606354554449373220645947423530332826217368595444403129171469646255483930218.85.65742169.96.26560574949403121116.6706768646662605552447266655852443527241870687167636154525147726668627671686257516862595146362315116.1595746344737352626187672696366555549473768636358463935282317585547454037333026233724715766668575747979387018316.227789393809999925846 685.3 777376727369696564597265685960473821154.9656159554340302719187169504757554042141869616555584753413826646148463635333326267469635848443130121467655756404120245.38.6574420159.755544745383817225.29.36867666463625555444471676257464129252016656866665157404535377267686475716662535065645453383915195.79.7555643374338312725207572666261524344353367666260444332331921785650464339363231263927725466668475758080386011147.724789393809999925845 695.5 777375727369706564597366696162514326197.2656258554341312920207269504755553842141870636658604954423926636146453433323226257469625844452933141766655756414018222.56.7574319138.757574846383815202.476867656463615452403971676156433925231715666868665158344229317267696475716561524865635452383713173.57.9565641344137292521187572676363534345343366646059424030301619785546444037323029244025715566668575758080395810136.824789393809999926045 676.2 7874757273696864625878656558554431179464595652413829271918747057536360494822237861635556465140362578624345303328312124786862564743353118157867625950443227129.9594622159.57859555047422924129.77867666463615553434172676560524736302622786971685559404638367868686575716562535078666056474325211210555641354339302724207873666361524244323278676562484638352623785749484240343429273724705466668674747979366516236.225789393809999915944 684.8 787376727369696463587865675858453519114.4646057534238292619177168504756554042141778616455574652393623786146443432323025237868635647433330131478645553363614172.14.6564117128.178534442333312161.94.7786765646361555444447065615645402824191778676766505840483641786668637571656152487862515033339.7132.85.5535439334135282422187872656259503742293078645958393726261415785548454238343130253825725466668674757979375513148.921789393809999925944 673.8 7774757273696965635875697365705861384413726367585847463333217369514762604849212569626656604754403923666250473836363429277670696161505138301869666159474624295.51060482520146560555246442327611686665636361575548477670736566545136392468696967546041493342716767627471656056486866595747452426121559594640464132282522757367636353474538337068666452504341313164615654554947414533412871586565867575808036641419726789393799999925846 686.5 7773767274707167666276697466715858353712666059534438302520167268504755543940141774677264685764525136676351484037373530297468645650413728171368646156474228268.89.8554117139.357535245443825238.29.66866656362605452434071666357484131252217666665635257374528347671706576727066585566625651433821199.19.8605948404942373032257572686363534444333166636057433832282018775650474340363431274027715365658674748080374913177.825779393799999925945 694.7 7774767375717268686377717468716258403615646056534240302820197068484659574444181977666861625356454229776350483937373530287768625647423129121577666358514636311915584419138.5775855514944342919167768686565635756444471676257454027241816776969685359394533377771716676737066615677646156504432272016595947394842363033257773686463544445363477656561504640372926775547444037333029244129725466658575758080366011157.827779293809999926146 674.7 777375727269686463587165675959473822155.2666159554440322822187168494759584446182168616456574752413825646147453534323226257469645851453832201666645654393819214.27.1584622161155524643363516194.17.169676664636155534239716663574841322522166466696549553742343472676863747165615349646353513736141658.2545640354138292623207472656161504042313066646058413929291718775648474239353231263927725466658574747979385810127.525779393809999926145 709.2 777376727570736769617568736368534521113.1675560474630321821117467574465565242241674627156674664405424676251474136383432277566655051333720188.3726668596047473228165641159.86.1675562485742442826157366716470616555594772646652513433182311746972676962625357467671726677727266665771646656584441272917575646364438312731227471666162524543363171656861574549374027775042393531282522193928715466668574747980385914217.927779393799999926044 684.2 7774767374717268686476707568726161383813646056534340312821197068494758574445162073677163655558474430676353494239403733317268615645423031131668656157484433301615584421159.5585453474741312816156967666563625354394171666356474028231815676968675359394532347671716676737066605668645955474328261716595849405043393134267472666362544547373669666562504740382927775547454137333028244029715565658674757980375711138.426779293809999926046 705.3 7774767275717267686373686963605133217.93.86258535138372625171670684846646148471718686364575649514235256763504738343632292671675854444032281613726767615850423421156250231610696263545747413123157169686667656059515070676258454126241716707071695959464540387672767276737267666071666659564938302718595848384740362830227572686465555348443572687065605352454235535444453837313126244028715665658574747979347542519.529779293799999916048 674.4 7672757174697065656075687364705656303286358555139362724181670674945515232378.61372636958634856414025635944423231302923227266595444403027131264615352373819225.48.9533815117.748454238343317205.48.86564616159584949353868645753393622201313596559623550193213247569706376716863545162595048363515186.59.9525439324235292422197371615955473237242563605654393728281818775243423635292825223524695264648573737878364911118.320779292789999915943 674.4 777376727469716666607568736467555029247.5625954523837262516167067474551533440101572646859635158444328646247453533323125247268605544413028121264635353363717214.68.3544016127.552504341333414194.58.26665636260605152384068655854413824221614586463623852253722297469696475726864565263625050353514175.99.454564336453835282922737163615850343925266463585738382828171852534343373530292623392769526565847474797928529.9128.123779393799999925944 656.1 7673757172686864635871656659584734219.54.4635954523838252616167068494665615450272667606254544449383422646248473636343428287367615544412928131366645655393920215.86.5625024171146543943313315175.56.26867646462615152384070666257464128241816677066685663475348447165666174706561535064635251353614156.16.856564335423729252520757266626253414332306665605938402729171877554846413733302924392874546565847474797937468.8127.922779393809999926046 663.8 767274717268686362567164675859463821155.36358555139352622151471675047515132377.91368616455584753403725625945433332313024237367615444392826121265625552373518183.86.1564218127.852514341343315173.66.36765636161585250393969645751393422191412626663634453324028347166676273706358514663605148353313143.87.3525337304034272320177371646059483839292765635956403629261816775447444037333029243725705365658574747979365710137.520779292799999925742 673.9 777376727470716767637569736669595435291163605653423829262017696748475957444517207266696163535645412866625147403738353129726761564742323014156864595545412724109.55845211510575350464239252310106866656362605352404069656055443827221715686869655457414438367570706576727066615666625552413821191110575746384740353030247471666261524344353367646158424131302120775444443937313028244128705365658574747979366212148.226779392809999915945 685.1 76727571746971656760716467575743291451.9595648473232202112127067494559553938111269616555594653383721625844393227302523197065545037352523118.568626053483731191355640137.75.26860604852383318155.1686666636461585551467064615443372421151364666363505338423034766975697571696462556761585046352715166.8575544324637342625187470676163515647453368626458514242323121504939383230242320173926695464648573737979346918305.727779292799999915947 673.5 7672747172686863625770646658574637221566359565241372825171771675047505232378.81467606354554549383422615944423231302924237367615644412828121364615450383220155.14.6554017128.350484439373118165.25.36765636160595150383869645852413623201513626763633853233624267066676274706359504662595045352915125.65.8525437313934272319167371636058503842303064615754393328231714775447444037333029253725685265658573747879365711137.221779292799999925843 663.1 767274707167676361567064655756453318104.16359545037352523151570674946505231386.91467606253554550383422615843413130292822217367605440382526101263615150333314162.95554117127.951504139313113153.15.4676563616058514939376863575139342219151261656463405125352427716667627370635949466159474630309.21235.451533529383325221816737164605948363927276261555433332222121377524542383531282723372570526565847374787937549.9127.520779292799999915842 662.3 767274707268686462577064665857473523126.3635855514137292619177067494557554143151967606354554550383622635945433332303024237267605445413129141464625452363418185.36.5554118139.256514542353416175.67.16665636260595151384069646054453929232016616465613950263623277065676274706459504763605047343214146.37.753543934413628252319737164605950373928286462575638352925191677524443383531292723392769536565847374797937571214824779292789999925843 673.5 767475727370706766627268686461544231189.67067656252503938282671704948616148492124676463585750544742336865555245434240363476726965595547452725737170686362515335426655302316706866656262515437457168686566635552373572706662524833292219757475737270636036447168686574726664575473706967626145513342606049444843383434287472676466574344343273717269646357584750555745473839303226254230745865658574747979366819247.728779292789999916449 654.5 7672747072676762615569626455534024123.81.9605749493434222214146966494560574543181966586051514045332917615744393228292523197066545139372725131166625752423421154.52.9554116106.6635753454335211452.968666663646057534743696461564841302521176667676556574547404070656661747064605248646054473830169.75.53.3565441314334312322157471676163515144413165616054413430231812525144413734292825223523685564648473737878336919306.923779292789899915845 664.1 7672747172686863625670646557554431189.64.4605951523737242515166866454452523538111466596052534346353119615842403028282621207067555541422930121364625450353317174.75.7533815117.150514140313114154.56.16766646261605152384068645954413622191312606664643953213415247066666274706459504662604947323112134.76.5525337303832252019147371636058493438262664625655343423241315495039403332262622212366496464847373787839187792799899915641 662.7 767374727269686462586965645955473222115.7636157544340322922196867444458584446192265616055524545382922626144443232303023247268625749443732191465635552393520185.56.2574318138.857534843383418175.86.66867646462615353404167655856434027241816686869665358384435367067666273716360494764625249373316146.57.1515436313834252220167372646259513841292966646058413730271917765444443837313128253825695364648473737878336214178.423769292789999915844 7019 767575747372706966647270686662584333157.77067656252494036282572725051575938458.8166867646258555449423365654949373835362829757268645853464124226367495527359.4151.93.7614819137.94444333223257111.53.2717068686665545437367270656353503834292565696968505836423230737171687474676756566264465027327.8112.54.9566145434646363530287374646560574044303164655758353524231313765848474139333228253928786167668575758080456121271325769394809999916551 652.4 7671736970666661595476626355524128168.23.66157534939352623171569664744525236391215765759515242473532207657414029292727202176665953443931271312766051483330151433.5543917128.476464137312812122.73.576646261605851503939676356513935232116147664626139492736262876646560726960574643765848443128111044.6485132283431222116147670605853463237252576615754373326231513765243423835312927233523695064648572737778355511127.919769292789999915742 650 757174707167676261567062655555423117104.2615753493834262317157666474454533739121567596253554450383523615843413029282721217265595243372926131262595047313013152.64.5533916128.351474038293010142.34.6666462605957474732336762575141352420171360636561394924322023706465607369625748445957454329279.1113.25.1505236303733252219167270625857453436262461595453353224231414765243433735302926233624685263638472737878365611117.220769292789999915744 651.8 7671736970656560595376616254514028168.23.4625753493834262317156966474452513638111576575950514046343120765740392727262519187665595344383126141276604948313113153.14.8533918138.776494039293011142.957664626058574848343567625751413424191613766261593547213117237664655972686056454276584645302910133.66.1475031263329211814137670615855453437252576605553363425241515765143423735312927233523705164638373737878355611137.819769192779899915943 673.6 7672757173697065666070646557564529186.33.75754464630311820101268664744625851472221655859515142473633216461474636363334272870655449363323218.58.868646157494632301212584520148.654524843423726241111676662635961465229406864595543392522151466686567566143493237726768637470686360556764595547442524121256564537453934273224737065615951414233306864636047453635242376504140343327262320392669536363847373787835458.411725769292789899916144 663.9 75747472727069676663747172696964615046267370716764595548423571704949656454543030686664615955565244396867575649474746424075737268676260534232727069656358554942356861383223706867636359554945377168686667646058515074727269676256494536747375737270666456577169676573726765615872696865635853474639646455515350423937347473696668615451463973717168656160565449636456575152434439354733736064648373737979357026326.636769292789899916552 641.9 7671736970656561595476626455534129169.33.7615753503734252316146966474353523739131476586051524248363221765739392728262620197666605345392926131276605148332915143.54533817128.376484137322813123.44.17665626159584848343568635752393421181311766359603948243321257664666072696056474376584743312711114.55.2485132283430222016147670615854453237252576595552353025211613765242413634292726213523685164648373737878345610127.320769292779999916144 663.2 767275717369706666617469736669595637321262595551403727251817686647455655414215177166696263545746423065615047393637343028716760554541282811136663585446402924129.7564219149.25551494242352621129.56766646361605252404068645953413523181412656667644755354135327470696474716864585466625752443824201310585646374639342730227371656260523841312967646259464236322420765242423635292826224027705364648472727878355511128.825769292789899915945 641.6 7572747173696965645970646558554530189.74.1605753493734252216156766454454533840141667616255544649393424646147443533333126247066585345393127131264605450383521196.96.7544017127.853494540363318176.56.96765636261605251383966635551383421191312676767654855313927307470706575726965595463595147363316157.67.4535442344337322628217271626057483538272665625956403729271917505241423434272824222569526464847273777832227691789899915944 632 757175717570746974667562655656443621177.3615653493834272317156865454253523739121575576050524147343118755943423231292922227565585245393228161375605351383721237.810544118139.775504342343517206.99.6756362606058525242436762575141342419161375646362445131382530756364597368645853477558504736351619811495236313734272323197570615856473538272675605854423932312122755142413634292726223725695063638371717777345911147.721759191779899915843 632.2 75717269696564595752756161534939231363615754513936272518166866484559574444162075565748483843322918755843413030272821217565595447403529181475604649263210162.35.157462216127554374126328.4142.15.1756362605957494634306662565139342319151375655163285014311420756363587268595544427559444626318132.76.4555440324334302321167571756674597445743575615455343724261417755343433736302926233422685163638372727777335811137.820759192779999905944 642.8 7471736970666661605468606353523827137.32.86156524836322320141268654642505034371114655759505241483634226057424030292727212171655851413625238.79.860584945302712112.12.2513817128.34944383228239.88.82.22.2656362595957514939376661564839312217141162636660475035383431696464597268615747435856444028248.68.12.72.951513629373125211915716961575646363727256058525032292219121175514341363329272621362469526262837272777735558.5106.620759192779899905741 633.2 747173697066666160556962645656443520135.16056524835312219131168654744495033371114665962545645503935236057434032303028232271655851403625229.18.660585047363118173.85.3513716128.2504742373430161645.96563626059585149393867615750403323181511646467634952374137366864645972696258504558564744333013144.76.451513630383326221915716961575646333725256159555237342824171675514442383430282722372568506363837272777736569.7127.220759192769999905641 642.7 747173697066666159546962645655443419134.7615753493834262217146966474449503337101366596153544449373322615843413229292723217165595343392927121261595046312814123.23.5533916118.247453633272510112.83.6656361595957504836356662555039332319151265656462455130373029686464607268615849455957464229259.89.63.74.550523629373226211815717061585547363826256159545235312421141375514140353328272521362470516363837273777836559.3117.320759191779899905842 672.4 7571736970666762615575596051493727148.23.1605551473431212013136765434154523638121475555848513947353722755741402928262620207564575141362724121075585147363120166.85.3533915107.275453834312617136.75.1756362606058525242436561554936311916119.9756366614552313931317571757075717468746475554943372918148.27.445503230333222221919756959575446333525237557575043343525251775483738323025242119362467496262837272777734539.8117.523759191779899906043 651.9 747173697066676362576963655859494428238.56157555245413530252168654744605947482124666062555647504137266259464435343232262670655853434333341617656359574846333315195645231813595552484643303115196664636261595452434167636156504337302922666668645356444639396965666272696459534964625754464428291620565645384641363131267170646161534243333265626157484240353027755041423736302927234130685263638372727777335812157.251759191779898905844 642.5 747072686965656059546962655657453721154.9605551473531222013126864474452513637111466586153554449373321595642413030282822227164585141362624111062605149353116143.63.452371395.7515043393430151343.56462605857554846363567625751413624211715666563624852354029326863645971686057484560584844322912114.84.750513529373125201815716962585747373627246260565337332623161474514441383431272722362468516262827171777733569.6117.220749191769898905943 632.2 747073697167686363587265706265534928237.85955504735332422151467644542525037381315696367586151554440276258464335323331262570645751413625259.81264605551413624208.26.9533917128.353484539383222188.77.16462605958564848353666615651403524201513626465624451313627277267676173696662565162585247393319169.67.652524133413429232519716860575445373829266460595541373127201774524443383531292623372469516262827171777734539117.721749191769899905943 632.5 747072686965656159546862645556443619144.3605552473532221913116864464352513537121465586152544349373422605742413029282822217164585242372724118.963605451393620204.76533815106.753514641383419185.16.36462615958565047393667615750403323181511666565625354424439386863646071686056484561595147373315155750523629373224211714716962585948383829256360585541373228211874494239343127252319362566496262837171777733569.3117.120749191759898905641 642.9 746972676964655960536859635154373214113.557514642292617159.58.667634642444526306.39.1665662495640513536205854393728262624201869615145333021187.86.3565544432725121133.44732128.35.73247253419248112.43.6615957555352434331316558544536282016131054595555354219261514686264587167615648445553403825228.68.23.23.950513628383026201613716861575748323724265756504931282218131074473937323025242119332266476262827171767732427.89.46.417749091759898905741 633.1 737072686965656058536860635455403215113595451463532232014126864474352503536111365566050534147343120595641393028272621207164575041352724111162605249363316143.73.4523816116.550494238343015133.83.864616058585549483838676156504033241916136664666150523741353568636459716761575046605848453329131154.649503528373225211915716861575646373828266260565337342623161374494240363229262521362368506362847171767733569.6117.320749091759898906044 673 747174707369736772657266706266555533339.65754494633312120121265644140525136381113676163565747524038266360474536343332272568635449383525249.99.3666259554943353017155136138.25.154494741423629241613646261595957525143436561544936311815119.3616463614151283724287369716773707266696265615752474130251916525342344337342832237169625957493739282766626258494340353026454735362929222219172668516262837171777732257491759899905942 643.6 747173707168696564607368726569595737259.558544946333221201313676544416056454217166761635555454838322261584342323130292323706454503936272411106562585545432826109.9534016117.152504440383423219.49.2656362606058535043416662585141342319151266676865596051524946736869637370686357526461555243402320121053524132423631252620716962585846434234296461595643403431232174453535302823232017342469486261827171767631531318824749091759898906143 643 7368716668636458585168586350533528127.72.855494440272416138.77.367624742444426296.18.8655561485338483233185753393627252523191768605143342821177.65.25353393921217.78.21.724630117.653244233017215.97.91.52.1605855545151404028286456524233251713118.7505749533242182512116861645771666054464149503434191965.62.42.2494934253828251814117067615656463235232452544446232515157.98.274463936322926242219322164476261827071757631518.29.76.516749091759898905741 643.1 736871666863645857506959655256383314113.655484339272416149.57.967624742414221264.26.7665562495540513336195853403629252723211768605043322720177.65.85454384017234.29.10.752.34631117.95.7294620311321380.582.4595754525049383823266457524434271814129.248574852303916237.611686164577165585345415151333615203.16.60.962.74948342537282518151169665954554533352423545445472126121769.274463937333026252320312064466262827071767631488.28.96.514749091759898905841 643.1 736871666963655859526960655356393415113.856494439272416149.58.367624742424323284.87.7665662505641513535205854403729262724211869615143322719167.76.25353363817204.58.20.782.14631117.85.12947193312233.18.20.672.2605855535250414030296557524434271814129.45057505330391621129.2686164577166595445414951313415193.56.41.32.8504936263828241716127067615557453236232552544246192511165.59.474464037333026252320322163466262827171767732498.39.76.615749091759898905942 592.2 74697267706366586050745759464327114.41.50.8585249453330211913126662433956554143152074545845503448313619746473557247724672407461554841352823139.87459464926339.7171.85.356472822157448354023326.7151.14.6746059575754464531316460534936322017131174605757435029372027746060557165585343397458424624317122.45.9626154465244423242307468645961523235252474615254293418239.314745041413533282724213623664962628270717676315510127.621749091749898905743 622.6 747270696766626054527459535037339.77.61.2160575350393627241815666543436058484624227455484637353129151574604243303128292122746559534740352918137460494227209.76.52.61.861512720147451403227209.46.32.81.9747073687267716268516462515034331918131274707270596238393425746562606968555440407458464028219.4743.3445128282729161814147471605955473537282574625854373125201512745241433335262822223723695361618271717777327035398.820749090769899896347 623.4 7371716968666461585467636357544330168.43.6615653483833252116136765464558574445172064595953524447383223595742403028282622207065595245363224149.9595648413123149.23.22.2523715107.4514439302921127.73.2266646360605750453329656256514034241916136465666334451926262067646157706860574945595545382922117.54.83.448523430363425231916707060595549313524246158555034282318139.7745039413534282825223724694962628371717777335211129.624749091769898896042 612.9 72687066676262575549655658484632199.44.32.15651454328271716109.166624440535236381216615355454735412925155352333321211919141468615247353122198.875554424222247.27.91.11.348331396.64545313121216.26.71.21.46160575654534544323264595448373121181412525951563443223021246559615570655753454153523737202155.41.71.9454628213226191612107067605554443636252355544647252516168.28.173443635312824222118322063466160836970757532488.8107.718739090759898895439 632.7 7368716668636458585168586351543631139.83.655494440272516149.88.667624742434324286.18655560485439493235175653383626252523191769605043322819178.16.55454394019225.68.50.692.245301074.92546193314234.38.50.652.4595754535050403927266356514333261714119.245574452243713206.59.6676063567066595446425152343617203.76.50.942.8484833253628231813106967585454463236222554554547222713176.19.773453735312824232018302066476161817070767631418.18.96.614739091749898905740 611.9 726971676864645958536761635454423318124.65955524837332622161466644341545239401516645759515242463531205956413929282726202069635851443830251311605949482929111222.8534118128.54847373627278.8111.73646261595856504939386560554939332319161362636461465135403233676362587067595547445857444327268.29.12.73.948503428363125201915696860575546343726255958525131302220121273483938333127252320362368486161837070767631579.7117.222739090769898895843 636.2 736972687066676262577165706165534928216.96055524635282116129.26664414049473334111268616657594853403623605643403229302724217064595145343020106.45856444322246.28.611.75034128.65.73842252816194.36.50.91.6636259575654464333306459524534281814119.564646662555546464440716665607268656054495554383820224.56.31.42.5515138304033282222176968595654453234222257564849252615177.98.845463635312824232118523670516362837171767733489.29.5821739191769899896043 592.5 737070686765626056527357504732286.450.920.826158555242383026201666654343625952502928735143413230272514127360454533343132242573666155504239302013736072477025689.7642.66052322619735468376624638.5542.773676764666258544338646152493533201914137368706761614546363173636057696757544441735972447025659.7504.6505334323232192018177370626159503838302773627256713369226614735444453537293124243723695561618170707676307146499.621739090749898896146 612.7 737170696766636257557370736973677360724265616156514241292919676745466059474719217363625854494742302573604645363534332827736763595448453528167361715369376620616.659472418127354694468356418596.8736872657163695767467370726972677164706073697069535732362425736562606968575647447360715069366417589.2505438353434222321197370585954473434262373627257704068306520736359575551494543363623715360608269697676325513167.622739090749898896246 601.9 726769646660615553476455574545291883.41.85349424126261617101065604238504933351113605154434633422827155250323221211919141466594744303019198.77.85053343716183.85.90.531.24632128.15.93742252816182.95.30.441.2605856545351434230306257514534281815129.244574453253815228.216655860546963565143394750313315172.84.90.791.9434528223125191612106865595453433334222150534245222414157.18.6734536353028232320183220624360598068697475314688.27.517738989739898895339 613.7 726971676965656160557164696064524727206.560545246362821161296563434149483334111366606254564550383522585541393029282722216963585045353121116.157564643282511101.92.45035138.45.53841302722197.97.31.52.26259575453504239292764595248343218171212646366615554454440387064645871676358524755544138252287.22.73.1484935283831262121166867595553442932212157564948282618179.79.146473737323126252320362169496161847071767631499.4108.518739091749898895641 602.4 72676964666161555447645557474531188.73.31.852484140252616179.71164604338494831341013615155444634422828145350333122202019141365594644312920198.68.15152353716184.96.50.741.54632117.95.74244282918194.460.731.5605856535250424129296256514335272015131047574154334223281523655860546964575244414950323316173.75.11.22.24445282230241915129.56965595454443334222252534445242416168.28.8734334332826222119163219634460608268687475304688.67.617738990749898885338 602.1 72676965666161555447645557474531199.13.9253494241262716179.31164604339494832341013615155444633422827145250323121201918141366604745313019208.18.95052343715204.170.851.64632128.26.141432931182247.20.781.8605856545351434229306256504334272015139.751584955324219271620645860546964565143404750303414183.25.61.12.2444529233225201613106865585352433635262351534246222614177.49.7734334332826222118163220604260608169697475304788.77.517738990749898895137 602.1 72676964666061545346655559464630199.34.62.251453936232013107.25.866604539424224276.57.8625257445035453030175350333121201918141367584740282417136.24.35250373518165.34.60.920.884227964.22636192413153.73.60.740.765755525047463535222361534639272313128.87.83856395016327.8155.17.1645959526964554941374847313015143.93.71.31.44244272130241915108.96865555149402328141651504242202112126.56.6724336342926232119162918634560608069697474285210137.312728990739898895840 582.5 71676964666061545345655559454629188.74.12.252453935222012117.15.966604439414023266.47.5635258455135473133175451333222212019141367584740292417136.24.65150353416155.24.91.21.143288.25.43.8163312228.9153.44.60.991.15755515047463334182161534639272214128.67.63354274714287.1124.45.5655958526963554941364646282913133.63.51.61.44344282130231914118.66865555250402127131650504042172010115.25.872443634292723211917291864456060826969747428509.8127.312728990749898895841 592.5 71676964666061545346655559454529178.13.8251453936232113117.46.265604439414124266.37.7635258455135463031165451343223212119151467584840292417146.54.94949333415164.15.20.721.143279.26.34.4193512238152.550.661.25755515047463535212260534639272214128.97.74057365114327.11657.7655959536863554940364546272913143.14.211.84444292131241915119.1686556525140242716164850384117209.8124.96.672443634302723222017291962455959816868747428499.6117.412728990739898895639 592.3 716669646660605453466555594645291783.8251453935221912107666604439424124276.27.76353584551364631311755513532232121191614675848392923161364.448493132141544.80.811.24328107.14.9233414238.1162.25.10.71.35655504945443032161860534638262213118.37.73858335214316.5132.95.5655959536863544940364445262713133.83.91.61.84545292231241915119.66865555250402228131648493840181911126.56.372433634292723221917291862445959816869747428499.5117.412728990729898895841 601.7 726970676763625854507267736673637354733358535045363124211614656242405453394014167259625353434635291972564241313030292323726255484135292414117257474527267.68.60.931.5513818138.57240332922194.85.60.671.1726059575754484637356359534837322118131272626059485137413435726059556865545138367255423923214.250.991.7464933293331222018167267565551432631202072575149272615156.37.2724736383332272724223421664559598168687474304399.87.720728989739898895840 591.871666964656060545346635457454429198.13.51.854494240262616169.89.664604238525035371214595053424532402624135250313120201818131365594844322919198.285051343715183.45.50.4814531128.46.43642232715182.650.391605857555452454332306155494333271915131045564054253614219.916645859546863555042384649303214172.54.60.741.743452721292418151196865585352433234222049524043202212145.77.8 72443434282723221917311961425959806768747430468.18.97.117728889739898885438 582.3 726668646460585450457264706166535332349.75649494234272318151264593936474830357.71272687567756574627352725643423132303024257259544640312721129.37252363712161.54.60.170.83473414107.5723619237.5130.462.60.170.5172575352494837372426615651453429191612107256525229411726131972626055646044402724724931311014130.291.145483531353224222020726555524940263021217253454416196.6102.24.672433434302923232019342065465958826768747430419.4108.219728990739898895740 591.9 7167696566626257555065586050503727148.63.55550464230271917111165614441504734331213615355464737413126175552373525242422181768605446363122208.98.65653434026231092.42.44834139.36.44142322923209.17.62.62.4605856545350424030286256494333281916131162606257434528312323645959546864565244405451403725228.67.83.33.54646302331261916131168655853544231332422565449473028211912117246393633302725242034226444595981686873732854910717728889729898885538 582.671666964656060545245645457444428188.34.12.150443835221912106.85.865594439424124276.28615155434834432929155250323020191918131267574639272316135.74.15048343314143.340.590.7842279.86.74.7253415228.8132.13.20.490.755654504946453333192059524438262213118.27.53354334715307.91457.4645858526862534839354545262711122.42.70.914244262130241815108.8676555514940212712164849384016188.7104.25 72433633292723221917281861435959806868737428529.6127.112728989729898885540 602.171656963666061545447645459464730229.15.62.152464137242114117.96.364604440414123265.67.6615156444834432829155450353324222221171566574840292418136.64.3474829329.4131.83.30.310.654228117.95.811345.8203.7121.12.90.350.74565449494545323418216053474029241613118.94253414718307.4132.65.765586053686254493936444525289131.52.90.431.24443292031231914118.5676456515241222914174749384015198.1113.55.8 72433634302824232118291862435959806767737428406.97.76.211728989729898885539 591.6706769656661615755496557605151382916104565147433229201712106561444148473233101161535647483843312717565238352624242318176760554739332522108.35654444225247.88.611.54835139.564344333124226.77.81.11.5605856545250414026266257514535292016131162606057424625311823645959556864555243395452403823226.26.62.12.5464731243328221814116865585353422930211956554948292819181110 71463836323026242319332163445959806868747429539.19.77.217718889729898885539 581.77067686565616056534865575950493524126.72.75550464231271916119.265614441494731338.411605255464836423128175553373626252423181767605346373224219.97.45654444225258.2101.12.24834138.86.24544343224246.79.112.2605856545451444131286257524636302117141162606257414525302122646058546764565243395452403824236.78.223.3464731263328221815136765585353423132232156544947292819191111 71484038343127262420322063455958806868747428559.29.77.216718989729898895539 581.97167696666626157555065586051503723125.12.35549454029251815119646142405048353512136154554747374130261656533736262524231817676053443528211786.65754454127229.471.51.54834149.973738292521176.55.31.11.4615957565453454432326156494331261714119.859606157384426322526646059556865575344415653423825206.85.322454732263429231917136766565451422831212158555248312719179.88.544453535302924232219342266455958 806868747429549.6117.419718889729898895741 561.371666764636058545046715553463931169.542.3544945413028201813126360424050493537131671505042403235262113715133332222202015157159514536312421119.37150383522198.3822.6483615117.87136292420177.17.322.7715857555452444330296056494534282016131171586256334221281921715757526663514834337148353220185.76.31.92.9394324222624161511117165535246392530192071514542252216148.98.271453536292923232018301863475858 7967677373285412138.916718889719898885440 581.7696467626459595351466253564544302095.52.352464237262115129.47.663584238424125277.68.1595054444634412826155349343122212119151465574941322519147.35.3464627299111.72.80.350.574329117.85.5163411186.8101.32.50.360.69575552504846363521206053474029241613118.752575052263710189.41063575852676253483834434324259.4111.82.50.71.14243272029231714118.7666354504839212813184847383816179.19.14.24.470433634312825232118281761435757 8066667373264988.96.212708788719898885539 54271696866646157554844714024236.66.10.860.80.360.3455514744343123201513636040396360565236337129161676.64.54.31.31.4716045443232303022237160544641353224161171676058383414102.72.35749362920716350483732128.32.82.27166686567646560605562595248373423201614 716769666263565753517158555367645250393871675654373213105.44.5515235322828171615157168615860493737292671686563444030261815715247463636313024233722645258587967677373277070704520708889729898885745 561.869636661625657504842605053403924146.72.91.6464134321918119.46.55.263574237403822246.97.5584751394330382523124947292818171616121164544235242114125.33.7454427269.191.820.390.4538247.95.64.4183111186.6101.31.80.380.445251464541412729141656494135232012118.17.23451324513266.7124.45.761555549656050443632404121228.38.61.61.80.670.7337392216251915117.86.4646151464435182411144445343513157.48.43.64.270403331272521201816261660415757 806666727225509.4116.810708788709898875438 601.769666865676364595852686266586250432414552464237272216139.16.962593937535038381214625658505040443328185754403829272725211965585041332621168.2559565246393221164.64.34935138.35.345413731312315113.53.4615858545551484238316055484231251613108636164605553464443376863625869656257524659554943362914115.34.347473528373026212317666455525040303123215956545035302420141170423132252519201715362365455858 8067677373295711145.721708789719798875941 561.16966676464605956534962565749463523126.92.85449464130261916119.462604139514936371315585152444334372721145350353323222121151566595244383124199.37.15349403622197.46.81.41.6483515117.53838272519175.75.71.31.6595755545251434130286055484231261815121060596257384323292323625857536663555141395047363220175.55.322.3424528233126201713116664555249412631192053504643262317158.87.970443535292823232118332062415757 796667737327529.1106.917708788719798875438 552.368646661625757514943595151413625136.12.71.252474338272315138.47.362573936565244412120544747393930362621145048313020201919141465574942342822167.65.3424422295.6121.23.20.30.75463415117.917317.2163.290.692.40.240.6954554450374519307.7155954474231261714118.856564951253812208.71360555549656052473836384118255.711130.441.337412321262416161111646251484536232717184346323711175.3102.25.369423133282722222017301859375656 776465717127389.79.68.917698687709798875032 55168656663636058545147625556474532198.74.11.652464238262415148.67.8656045425550413817165751514342333727211352503331222020191413665749423228201785.75351423825219.78.61.71.94532149.46.23833292321166.96.21.51.7585453494945373322195954474231271815121060596056374320251918615655526562524937355148383424207.56.72.62.643442824312721181412656453504737232717175451474429251916119.443433435292823232018291663465757 7966667272295811155.315698788699798875340 551.668636560615655504741584950393422116.12.51.5474136322117129.375.362564136454228289.69.15545473738273423201149462927171716151110635344362721161163.5434325288.6111.82.90.380.6339269.875.111288.8165.59.61.22.20.340.585552494745433131161856494035242013118.37.44153394815295.9105.54.760545449645949443431404122248.6101.82.40.75137392218252015128.67.1646251474534182411154545353614167.88.83.84.469403331262521201816271657415656 7765657171245712146.611698787699798875538 551.36866666463615856514862575849473423106.12.15348434028251715108.765614542524937361315585252454435382822145050313220211819131466585044353023199.674748323313152.540.280.874633139.46.7242617159.39.21.32.70.210.8575652504646313216176055484331271715119.858585655283814201313615856546664535139374546292913142.13.40.541.4414528243229211914126465525146402129151849494040192010114.35.642443334292823222018291762445656 796666727228498.79.86.916698788709798875539 531.368656662625857524944605354444028156.83.31.650454037242314148.58.164604542524837351415564949414031352620135048312919181817131265564741302719177.75.85147373219155.84.51.10.954331139.36.23329231715114.23.51.10.9575453494945363220185954464230261615111059585855384220241616605654506461514736334945322816134.23.31.51.3414326233026201713116463514944362226161651484339211813106.45.241433334292823232118281561445656 786565717128581215614698688689798865440 502.86870646860645458455068605152403921168.14.367616056454232282118686845466361525126276855454838393333211868563339222821271621686866585246363416166853333913192.85.60.491.162492416116845212811171.840.380.92686359595756484334266964565136331918121168 717471737072677259686051556467475234396852303613192.14.70.551.637492227222813189.2126869545848481934142368564348212611164.78.3685545453637293024242814674957577965667272304610118.114688787719798875942 541.668666564626057555047615656484533219.85.4252474239262415149.58.7656144425048353413135651504441333627221450493031192017191314665849433229201896.54647303111132.23.20.450.754532128.15.12327141478.51.42.30.40.6657555250474629321217595547433127171512105757525431411624141659575552646351493735434424269.5111.72.60.71140452623302921191312636451504538192713174647363714166.78.82.84.242433334282722221917281663435656 776565717128478.39.46.814688688689798865436 574.968646864676264585851683423142.91.90.370.370.340.346839353222201514119.3695338356256595450456818117.13.62.12.91.60.810.49686056494638433534276851423429232016117.968635656404122228.16.3484032251668532641213112145.94.5686165606560635862576850484136302420181568606459565651525248685557506863615550446861525339392019129.568565446504240304434686364576554453840306862585842423332242268454439373233292724493159465555 766565717127264.8696628688686709698865544 502.468666563615955524543683318174.74.40.480.480.240.3514844403129222015145958373761595653393668219.6113.84.12.52.60.820.526856454132303028212168575043383228231612686664625149272464.55446362919686559574947242064.46864666365626360615659565147373424211816686465626059565655526855524963614845353368666260494823199.6850513533262614131415686660566048383529246866666453524239272368504644373532312524382262485555 7764647171256868684819688686709798865745 521.9666263595955534844395546453628207.94.71.81.3444232331819101166.3595538354946343414155043423533252921161044432524151513139.1959533637222213135.34.6404223237.26.61.21.50.180.414027106.95273113146.36.70.771.30.20.42555250484745343318205550423827231513108.83952414918309.7159.89.957525246625846433331373920207.77.31.11.70.350.7532371815221913127.66.9636050464233242716174142323314137.97.13.63.468382930242419191615251456375454 776363697024429108.113688686689798865035 482.56768646660625456464867575049383419118.72.766606153513540232914666645456261515223296753464638373531231867532436132412237.8176767625651414025217.4675026347.5141.23.80.270.725542221710673714225130.592.40.220.5867615456505237402225 736873677364726071566764646344512634272367595052636547493335674722296.4130.862.70.360.9940492829293118211315676751554645193114216751364212195101.94.667605552534647404132291365465656786565717128358.99.37.816678687699797865740 53265606358595455494842585052424128178.241.6494140332418138.56.64.557533531464331311111554850424334382824165046323122212020151561534837322219115.22.443382822128.63.62.90.770.884231128.34.930261814118.22.62.60.70.865250464442412829141654484135252013108.66.8545250483637232417175853544962575247403642382721139.63.42.91.21.4394028223025201615116159504645362425181647433935211713106.85.639373029242219171613352259395454 7661626969265210118.518658585679697855435 544.965616561645961555649652920122.21.50.320.320.280.2965363128201813129.28.3655037325953565148436515105.72.91.62.41.30.660.426557534542343931312465473830252018149.47.265605053313813203.2645362922146549213715286.7132.346558625762576055595465474438322722181614655760565253484848456551554764605852484265584650303712185.6965535144484038294133656061556252433637276559535535402631172165414136343030262522493055435353 756262686925263.2656327658484699697855442 533.965636261605755524845565350463831147.92.11.15348464132272015128.85756353549473536131451494542363331271814504933332222202115166257524439312719115.85249393320147.14.41.61.14734149.26.54335302120136.23.61.51.16058575555534543292852504037242213117.77595960584346232714165855535061605047383550453528191363.72.51.7384225212725171613116061504945392628191853494540241815118.75.539413132242418191515342265465454 7663647070295925289.617658686679797845738 462.265626160565549463936652511102.62.40.50.560.360.36494542382927211914135655363560565651403765156.16.11.92.21.41.50.50.34655342393027282521196554484036302721151165636361565436361211524435281965626159565333311211656263616360625960565753494437342522181665606359575753535551655148465957434231296563636155533231161551513733272613131616656359556049393529256563646357554846343365474442363532312725362060475151 7460606768236464645318648384679697845443 672.363555751524545373728634142322921126.93.11.73832282313106.44.93.52.650472523313114182.13.16338393030222718169.463402624171615141111634535272013106.43.11.7634441322819158.34.92.130174.22.61.7633132232517137.45.22.5634950464744403830294842352818137.55.34.53.16346484324311519131363616562585247393729634037282617126.56.23.131351816211812119.58.2635642413628212315146341423427191913137.6632820191513119.98.77.1292261415151 745960656623478.5116.115638283659697845939 451.2635959555350474338356343363221186.14.61.71.343403533222014138.98555337354745343414156339353227232319138.963412424151513139.39.563514136282418157.96633821217.581.52.20.370.514130139.97632212126.26.91.21.90.370.56350494645422930131649463735242113128.97.863535351223412187.41363494643565438362424633519198.28.51.32.10.510.8283316141817109.576.6635842423529152111136340313114137.67.83.54.263382930242420201716241255405151 7559606566215012149.711638284629697835236 461.962596157575351474137332715123.12.30.510.510.470.4742383431221915139.88.5545236355753545041382319118.94.33.43.62.71.10.88555239362624242216155749413328222015107.34956334613263.69.90.782.2463829231415455.8302.9190.966.10.51.6605760565956585456525448453933282118151358576056555451505046524847435956474434314754294213253.28.41.13.7474835323128191718166059555253473030252350564251193111205.91148434238343130282523402259455050 7259596767216360635520628282649597835746 513.862586258615658525246622717111.81.30.350.350.340.3462343027201813129.28.36247333055505348444062148.55.52.31.61.91.20.510.46254494238323629282362453730252018149.47.262564750283611182.25.242352722146246203513264.6111.33.46255595559545853565162444136312621181614625458535050454646436249524561575549443962554347273410164.27.962504841443734273830625758515949383133246256505232372328141962393934322828252421472954435050 735959656622252.8626125628282679496835239 432.36360595754524643363363218.78.22.22.10.420.480.320.32494541382926211814125454343458565552423963113.64.411.30.720.90.220.34635038352725252318176254474036292721151163606160585642421617514435281962616159585539381616626061606160605859565652484436322521181662596058565653535349634945435756413928276261616057553738212149503533262412121414636057545849383528236361626158565150393963474542363432312625351956454949 7358596666216262625017628182659597825542 551.6615861586157605659545954585154454427268.542393432201812116.66515130304140272879544849434234362824165048363526262423191955504035262215135.44.352484541343022199.27.941291074.239363530302618168.47535150484845413932324846353319189.18.55.55.4545352503942273125256158605661575954575051474239332918159.88.8383929242926211822175756464442342325171653504945353228252017303422251819131512112158395050 715959656519186281609797825536 412.561635659515443473438615044413427199.88.92.66155585151434435352760603840445226405.7176147434039343832302161442327141713169.81261615653464336341918613620215.57.40.891.90.230.37473823191661188.993.44.70.471.10.20.2761483940323417207.39676466616154534445366152504722321420121361524345545834382124613316174.56.50.511.30.250.4831412022202410158.11061614047323311226.913613727278.19.63.451.22.26165 7164716370627061239.961415049735859646525268.89.18.211618283649697825437 431.7605757545350474339355044423630231273.61.841383230201812117.96.852503331393827278.59.54439373229232318127.944422625171615151110544939342521151365.542392724118.82.92.40.670.5539261074.730271815118.72.42.20.580.555149474544423332191948453633222012107.86.8525252503540192518175248444156534138292741372521118.62.420.940.9313418152017119.87.77565644433831192214144542363317149.77.84.63.6313424252020151613131358384949 725758656524116180629696825336 421.7595656535248454136334539352820146.23.21.50.78423633281916118.86.54.852493331514743402422403431262217191496.339372120121211107.57.155483932251915115.73.423289.513240.521.10.280.323827118.3612173.96.61.63.20.420.830.280.314746404134372026101446423330201812118.27.550494946293314179.7114945434055523836252423268.5122.34.30.520.990.330.5126311413171710106.16.256554239352718221314313321245.88.42.74.51.22.161332426202116171514251250294847695657636418378.7108.31261 7981619496814527 441.7605758555451474337342621118.11.91.50.360.340.330.3239343027191612117.96.85249343255505146373417147.45.82.62.121.60.640.525147323020181716111155463729251917128.25.84352274110222.78.10.61.9433526201215416.6263.2160.8550.371.5585557545753555253495145423630251916141255555854525147464743494544415754444031284050233710212.470.963.341432927252315131413585751485042292923214552354615268.5174.59.260393834312827242220372056434847 7157576464206057605317608081639596815543 422.26159575652504442333161186.56.21.51.30.380.420.440.424642393627241917131153523333575454514340619.133.40.761.10.560.740.180.3614736322321211914136151453833262419141061596058545237371312494133261861595857545233331211615960585958595757555451464134312321181661575957545451505248614744425553383626246159595852513231161647493431252512121313615955535648363327236159605954534646333261454240353431302625351854424747695657646419616060531660 7980639496815443 502.4595958575655545351505756555351494339322340383230232316181112535335355655535139375453525147474444383650504040333331322728575237321921121568.252504743403631282018473723191446434037353327251919565452494946413732275149444236342825222055575756434934393535595855555858515039375047443937332724201845473837383832313030585853525046333527264945443832272622201760372530252919231918403059394949 7257576464214511138.655608081619696815437 411.15753534948444036312741353024161152.91.513631272316139.27.4655046302736352425101139333125231719149.66.3363417179.38.88.17.85.25514334272015128.95.43.718256.19.71.42.50.480.820.250.36332185.84.36.1122.44.71.32.30.480.770.240.343842273520308.9164.27.74340302817169.18.56.15.84445384017258.8126.57.54642413652483532242218235.99.21.83.10.6310.40.562427131115138.27.554.852523736302411157.78.8263017214.26.72.23.61.11.958312324191915151312231250324545695454616217407.79.36.99.258 7879579496794931 371.35753535048444037312841353025161253.11.50.953529262114118.56.95.34.2504630273331202188.538333025221618139.15.5363317159.28.187.25.24.6524233241913118.14.932624119.62.82.40.880.810.450.3930197.35.43.817166.76.73.130.810.770.450.384643403735322220119.83936252313127.16.754.743444040152078.86.65.8464238345248353224222524109.33.33.10.980.980.610.632326131014127.76.54.83.852513734312413168.98.7333123227.87.343.81.91.958302323181814141311231150324545685353606116459.3117.17.558 7779589496795133 411.658555653524946433834282413112.82.30.570.540.420.42413835322523181713115048323253505047413817157.46.62.62.321.70.640.6504734312320201814135347403430252319141052524343282712122.73.244383125174747353526259.69.62.32.85654555355525450524850464339333023211816535356535150484748454744423954524138282651504140272611114.65.444443128272415131513575552495043322925205352484832322323141444423937333230282524361953424645685454616217585858511758 7879609496795544 341.55653534947443935292538332622129.52.92.10.850.73631282316139.57.35.84.54945312946413834211935312622181415126.85.340372220131211107.77504234262115138.45.43.5363321188.26.42.220.470.543524128.46.1262313106.75.41.61.40.40.4650454741453937322723433931281918111087.44548484625341417119.2423835325047302718173431191586.32.11.90.930.932628161316149.786.85.85351393733241617111138353026129.86.35.53.22.8573123251921161714142210503544446752525959145426297.37.857 7778569495794732 341.75652534947433935292539342723139.92.92.10.90.723631282316139.66.75.83.849453129474239342119353126231815151275.639372120121211117.57.1504234262115137.95.22.7373422198.572.32.30.60.643524128.66.3272414116.95.71.61.70.530.525045474245393833292443393229201812118.57.64648494723341317139.842393532494630281817353219168.36.62.220.980.95272917131714108.67.26.15351393634241716119.939363127129.96.75.53.32.9573123241921161713142211503544446951525859145529327.77.957 7778569496784632 341.75552524847433935292537322521118.52.51.80.820.613531272416139.37.15.444945312946413834211933292421161313115.94.8403722211312121187.6504233272115138.64.72.9383524209.47.22.620.630.553525128.76.3292515127.95.91.71.50.560.445147484446413935302643393229201711108.27.34848514727351418119.642383532494630271917363221179.37.12.421.10.932729171416149.78.476525139363324161711104037322814117.55.73.72.9573022241820151613132211513644436852525858145431337.77.957 7777569496794633 341.85552524847433935282537322621139.22.81.80.810.633530272215128.96.85.64.24844302845403732191734302622171414116.44.8393621191211109.76.86.3504133252014127.84.93.3343121167.85.12.11.50.560.383323117.95.62521128.66.34.21.40.960.470.3449444540423834312522423830271816109.27.36.744464643192911159.18.141383532494630271816322818137.75.221.50.940.692527151115138.97.45.94.95251383632231516109.537342925118.66.34.73.12.4572922231819151613132211493244436851515858145118206.97.457 7677569495784531 362.35653524945443634252456113.93.50.960.940.460.460.420.464441383527251917131247473031535151484340564.81.51.70.560.50.520.420.160.18564129261817171612115649443733272519141056545553545148453028464033271856545553535147443027565455545553545353515047433933302422181756525452504947474845564138375048333121205654555454524543343246463432242511121414565453515448373328235654555454525049444356424038343231292624321651394242665051585816555555491455 7575589194765039 342.45351494943423533242353113.73.81.110.440.420.320.284441393528252017141246462830514949474138534.81.31.80.480.60.380.440.340.18544029272017171612115348433733272619151153525352515047453331464033271953515251525046433331535253525351525151504846423933312422191753515150484945464644534036344846323021205352535252504644363545463533252413121414535251495247363327225352535251504947444353424038353331302625311450354141654950575816535353491453 7274569195754836 322.25351494842413432232253103.63.51.31.20.380.360.320.284341383528262018141345452829504849464139534.11.31.50.460.560.320.420.20.225337272417151613119.45347433734282620151153515351514946443432454034271953515250504945433431535152515251525051494745423933312522201753505150474845454644533834334746313020195352535151494644373644443330232210111212535151485247363327225352535151494947444353424038343331302625311347354040634848565615525252481352 7272559194744937 281.152474543393731292220522819188.77.82.420.810.723229242214138.77.95.34.9454230284238353219185225181711119.78.84.64522713136.86.76.364.13.9523929241815119.753.8522512103.63.10.930.90.320.343122118.25.952156.25.432.60.750.710.30.3252413937363525251314393528261817119.97.87.25244474422319.1147.28.95236302943412323131352231093.73.30.750.810.390.4320249.48.710105.15.33.23.352473232272111157.88.9522820196.96.13.73.31.71.752302424191916161313187.2483240396247475454134213148.97.252 7273509395744531 331.94845454340383330252213103.52.70.760.70.520.510.50.51332926231715119.97.66.6413927254441434036337.96.42.72.10.750.70.770.60.290.2539352521151214118.87.344383225211615117.85.7384424388.9251.7110.32.6362923171114365.4252.7180.646.10.271.646444644464345434441413634292421161412104443464341403837383537343330454234312422354320358231.58.90.59436372725242215121312474543414338222219163844294011266.21731136323128262323211917341746353837614445525214484648461448706952 8993724836 272.145444039343325241715455.92.12.10.880.90.580.560.580.563937363328262119161438382325424242413836452.40.660.90.280.360.260.240.10.1452919161210109.26.96.2454239353227262116134544444442423938323040373227204544434343413937323045444444444444444342414037353129242220184443434341424040403945322828393923231414454444444342413936343737272620209.19.29.79.6454444434442343026214544444443424240393745383634313029272524271242323433574041484812444444411144646549 8691664434 141.437322927222115159.69.237148.57.53.73.21.210.490.46191714138.47.154.13.22.63129212030272623151537138.78.15.24.84.4421.837156.86.33.53.33.132.11.9372518141086.74.83237144.63.81.20.970.430.40.180.1621157.95.84.13782.521.20.930.350.350.170.17372929262724211714112623181611107.26.55.45373230312325111316837211514272611116.16.137134.53.81.51.20.40.410.220.2412145.44.95.45.42.72.71.81.73733212117137.79.25.15.3371712113.22.71.71.30.830.737201716131311119.39113.24025272751333338387.23212137.6337575835 8791614026 131.832342832252821231718321811125.95.72.21.71.10.724442434042374233402233371922343730321719321812149.8108.394.14.832198.19.84.75.94.25.333.73243434037332924197.832148.37.43.22.80.920.790.230.18403418138.332105.44.72.92.30.740.580.250.2132312729252621201614393933312119121187.2323837393837333334323223141629311920141432137.97.13.73.31.10.940.50.4512186.57.578.83.84.82.83.23232192217174.673.14.4321713125.65.23.43.11.81.632362930252621221716143.541252525483030363611196.16.84.84.932545339 7479555029 7.21.4211915149.99.765.83.33.3214.62.72.41.31.10.540.510.290.29108.36.85.73.83.32.62.11.81.518171312171615149.79.3214.22.72.41.61.41.21.30.550.64216.42.42.31.31.11.210.80.7621149.36.64.93.73.12.41.61.3219.464.92.31.70.70.530.230.2127.54.73.72.6217.14.13.22.21.60.550.430.250.22119191818171514121014129.58.35.95.33.93.432.82119171814157.88.35.24.621107.77.214134.74.82.72.8219.15.64.52.41.90.630.540.320.295.57.22.21.92.22.41.21.10.830.75212013139.57.55.15.93.43.221119.48.24.13.22.41.81.20.9421119.38.77.576.565.75.16.61.33020161634191923234209.4105.41.321383921 7584433420 71 80 7886 8088 ImageNet Accuracies (top-1, %) Figure 7: An overview of our testbed. Each row is a model, and each column is an evaluation setting. For the corruptions, we display each of the five severities defined in [38]. We also plot in-memory and on-disk versions of each corruption as jpeg compression was found to be a confounding factor in [31]. A few cells are empty due to resource constraints. Testbed code and data is provided at https://modestyachts.github.io/imagenet-testbed/. 27 B Relative and effective robustness B.1 Relative and effective robustness graphical sketch A central question we address in our paper is whether current methodologies provide meaningful robustness to natural distribution shifts. We discuss how both relative robustness and effective robustness are needed to disentangle the confounding effect of original model accuracy. In Figure 8, we graphically illustrate this notion of relative robustness. 606570758085 ImageNet (top-1, %) 55 60 65 70 75 ImageNetV2 (top-1, %) Hypothetical Robustness Intervention Baseline accuracy Standard ResNet50 ResNet50, hypothetical robustness intervention Negative relative robustness Negative effective robustness Figure 8: While a hypothetical intervention (green), applied to a baseline model (blue), leads to effective robustness (it is above the red line), it reduces the model’s accuracy un- der distribution shift. Hence it fails to pro- viderelativerobustness. An ideal intervention would place the model in the white quadrant - positive effective and relative robustness. B.2 Relative and effective robustness for ResNet50 models We provide additional plots depicting a subset of the models in our testbed. In order to make an equal comparison, we only plot ResNeet50 variants, models which slightly modify the training data or architecture of a base ResNet50. The plots in this section thus describe what the relative and effective robustness properties of various robustness interventions look like on a standard ResNet50. The models can be directly compared with each other since the base model before intervention is the same. For natural dataset shifts, the plots in Figure 9 demonstrate that the only models that have consistently positive relative and positive effective robustness are models that are trained on more data. However, the effect is small, and not all models trained on more data are more robust. On YTBB-Robust specifically, a few data augmentation strategies from ImageNet-C provide significant both effective and relative robustness: training on greyscale (ρ= 6.9%,τ= 1.8%); training on pixelate (ρ= 5.4%,τ= 2.0%); training on jpeg compression (ρ= 5.4%,τ= 6.3%); training on gaussian noise, contrast, motion blur, and jpeg compression (ρ= 4.8%,τ= 5.0%); and training on gaussian noise (ρ= 3.6%,τ= 4.0%). However, this performance is not consistent across the natural distribution shifts. Exploring why these data augmentation strategies are helpful on YTBB-Robust is an interesting direction for future work. Additionally, while some` p -adversarially robust models display significant effective robustness on YTBB-Robust -` 2 robust ResNet50 (ρ= 6.4%),` inf robust ResNet50 (ρ= 6.4%), and ResNet50 smoothed with 0.25 gaussian noise and adversarially 1-step PGD trained (ρ= 5.0) - in most cases, they fail to provide positive relative robustness. For natural consistency shifts, the plots in Figure 10 demonstrate that while adversarially robust models provide effective robustness (averageρ= 4.3% on ImageNet-Vid-Robust and averageρ= 3.9% 28 on YTBB-Robust), they only sometimes provide relative robustness on YTBB-Robust. For the adversarially filtered shift, the plot in Figure 11 demonstrates that robustness interventions have little impact on ImageNet-A accuracy. Most of the "knee"-like response curve can be explained as an artifact of the adversarial filtering, with the knee occuring at the ResNet50 model accuracy. 4550556065707580 ImageNet (top-1, %) 35 40 45 50 55 60 65 70 ImageNetV2 (top-1, %) Relative and Effective Robustness - ResNet50 Family 5055606570758085 ImageNet (class-subsampled) (top-1, %) 15 20 25 30 35 40 45 50 ObjectNet (top-1, %) Relative and Effective Robustness - ResNet50 Family 859095 ImageNet (class-subsampled) (top-1, %) 45 50 55 60 65 70 75 80 ImageNet-Vid-Robust (pm-0, %) Relative and Effective Robustness - ResNet50 Family 9095 ImageNet (class-subsampled) (top-1, %) 45 50 55 60 65 YTBB-Robust (pm-0, %) Relative and Effective Robustness - ResNet50 Family ResNet50 baseline Standard resnet50 Trained with heavy data augmentation Lp adversarially robust Architecture modification Trained with more data Linear fit Negative relative robustness Negative effective robustness Figure 9: Relative and effective robustness for models that are variants of a ResNet50. Model accuracies are displayed on the four natural dataset shifts: ImageNetV2 (top left), ObjectNet (top right), ImageNet-Vid-Robust-anchor (bottom left), and YTBB-Robust-anchor (bottom right). These plots demonstrate that the only models that have consistently positive relative and positive effective robustness are models that are trained on more data. However, the effect is small, and not all models trained on more data are more robust. Confidence intervals, axis scaling, and the linear fit are computed similarly to Figure 2. 29 4550556065707580 ImageNet-Vid-Robust (pm-0, %) 35 40 45 50 55 60 ImageNet-Vid-Robust (pm-10, %) Relative and Effective Robustness - ResNet50 Family 4550556065 YTBB-Robust (pm-0, %) 35 40 45 50 YTBB-Robust (pm-10, %) Relative and Effective Robustness - ResNet50 Family ResNet50 baseline Standard resnet50 Trained with heavy data augmentation Lp adversarially robust Architecture modification Trained with more data Linear fit Negative relative robustness Negative effective robustness Figure 10: Relative and effective robustness for models that are variants of a ResNet50. Model accuracies are displayed the two consistency shifts: ImageNet-Vid-Robust (left), and YTBB-Robust (right). These plots demonstrate that while adversarially robust models provide effective robustness, they do not necessarily provide relative robustness. Confidence intervals, axis scaling, and the linear fit are computed similarly to Figure 2. 65707580859095 ImageNet (class-subsampled) (top-1, %) 5 10 15 20 25 30 ImageNet-A (top-1, %) Relative and Effective Robustness - ResNet50 Family ResNet50 baseline Standard resnet50 Trained with heavy data augmentation Lp adversarially robust Architecture modification Trained with more data Linear fit (piecewise) Negative relative robustness Negative effective robustness Figure 11: Relative and effective robustness for models that are variants of a ResNet50. Model accuracies are displayed on ImageNet-A, a dataset adversarially filtered to contain only images incorrectly classified by a ResNet50 trained on ImageNet. Due to the "knee"-like response curve, an artifact of the adversarial filtering, effective robustness is defined piecewise around the ResNet50 model accuracy point. The plot demonstrates that robustness interventions have little impact on ImageNet-A accuracy. However, the effect is small, and not all models trained on more data are more robust. Confidence intervals, axis scaling, and the linear fit are computed similarly to Figure 2. 30 C Effective robustness scatterplots In this section, we further explore to what extent robustness to synthetic distribution shifts predicts robustness on natural distribution shift. We extend the analysis in Figure 5 by computing effective robustness on all natural distribution shifts and comparingn them against effective robustness on synthetic distribution shifts. For natural dataset shifts, the scatter plots in Figure 12 are weakly correlated (the Pearson correlation coefficients arer= 0.24,−0.05,−0.01,−0.26,0.61,0.30,0.52,0.36in reading order), indicating that improved robustness to corruptions or adversarial attacks in general does not improve effective robustness under natural dataset shifts. Of the group, the two strongest correlations are effective robustness between ImageNet-Vid-Robust and image corruptions (r= 0.61) and between YTBB- Robust and image corruptions (r= 0.52). While not very strong, the correlations are significant, and exploring this phenomenon between image corruptions and video anchor frames is an interesting direction for future work. For natural consistency shifts, the plots in Figure 13 are largely uncorrelated, with the exception that accuracy on adversarial attacks is correlated with effective robustness on consistency shifts for lp adversarially models. However, as explored in Appendix B.2, effective robustness on these shifts does not always imply relative robustness. For the adversarially filtered shift, as seen in Figure 14, after computing effective robustness piecewise around the ResNet50 accuracy, there is no observed correlation between the synthetic and natural robustness measures on ImageNet-A. 31 -505101520 Corruptions Averaged Effective Robustness -1 0 1 2 ImageNetV2 Effective Robustness Effective Robustness Scatterplot -15-10-5051015202530354045 Lp Attacks Effective Robustness -1 0 1 2 ImageNetV2 Effective Robustness Effective Robustness Scatterplot -505101520 Corruptions Averaged Effective Robustness -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 ObjectNet Effective Robustness Effective Robustness Scatterplot -15-10-5051015202530354045 Lp Attacks Effective Robustness -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 ObjectNet Effective Robustness Effective Robustness Scatterplot -505101520 Corruptions Averaged Effective Robustness -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 ImageNet-Vid-Robust (pm-0) Effective Robustness Effective Robustness Scatterplot -15-10-5051015202530354045 Lp Attacks Effective Robustness -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 ImageNet-Vid-Robust (pm-0) Effective Robustness Effective Robustness Scatterplot -505101520 Corruptions Averaged Effective Robustness -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 YTBB-Robust (pm-0) Effective Robustness Effective Robustness Scatterplot -15-10-5051015202530354045 Lp Attacks Effective Robustness -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 YTBB-Robust (pm-0) Effective Robustness Effective Robustness Scatterplot Robustness interventionTrained with more data Figure 12: We compare the effective robustness of models with their accuracy drop due to corruptions (left column) and adversarial attacks (right column). The effective robustness is computed with respect to linear fits on the four natural dataset shifts: ImageNetV2 (first row), ObjectNet (second row), ImageNet-Vid-Robust-anchor (third row), and YTBB-Robust-anchor (fourth row). The measures are largely uncorrelated, indicating that improved robustness to corruptions or adversarial attacks does not improve effective robustness under natural dataset shifts. 32 -505101520 Corruptions Averaged Effective Robustness -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 ImageNet-Vid-Robust (pm-10) Effective Robustness Effective Robustness Scatterplot -15-10-5051015202530354045 Lp Attacks Effective Robustness -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 ImageNet-Vid-Robust (pm-10) Effective Robustness Effective Robustness Scatterplot -505101520 Corruptions Averaged Effective Robustness -4 -3 -2 -1 0 1 2 3 4 5 6 7 YTBB-Robust (pm-10) Effective Robustness Effective Robustness Scatterplot -15-10-5051015202530354045 Lp Attacks Effective Robustness -4 -3 -2 -1 0 1 2 3 4 5 6 7 YTBB-Robust (pm-10) Effective Robustness Effective Robustness Scatterplot Lp adversarially robustOther robustness interventionTrained with more data Figure 13: We compare the effective robustness of models with their accuracy drop due to corruptions (left column) and adversarial attacks (right column). The effective robustness is computed with respect to linear fits on the two consistency shifts: ImageNet-Vid-Robust (first row), and YTBB- Robust (second row). The measures are largely uncorrelated, with the exception that accuracy on adversarial attacks is correlated with effective robustness on consistency shifts for lp adversarially models. -505101520 Corruptions Averaged Effective Robustness -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 ImageNet-A Effective Robustness Effective Robustness Scatterplot -15-10-5051015202530354045 Lp Attacks Effective Robustness -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 ImageNet-A Effective Robustness Effective Robustness Scatterplot Robustness interventionTrained with more data Figure 14: We compare the effective robustness of models with their accuracy drop due to corruptions (left column) and adversarial attacks (right column). The effective robustness is computed with respect to a linear fit on ImageNet-A, the adversarially filtered shift. After computing effective robustness piecewise around the ResNet50 accuracy, there is no observed correlation between the synthetic and natural robustness measures. 33 D Corruption robustness greyscale brightness_in-memory contrast_in-memory defocus_blur_in-memory fog_in-memory frost_in-memory gaussian_noise_in-memory jpeg_compression_in-memory motion_blur_in-memory pixelate_in-memory saturate_in-memory spatter_in-memory zoom_blur_in-memory brightness_on-disk contrast_on-disk defocus_blur_on-disk fog_on-disk frost_on-disk gaussian_noise_on-disk glass_blur_on-disk jpeg_compression_on-disk motion_blur_on-disk pixelate_on-disk saturate_on-disk spatter_on-disk zoom_blur_on-disk Evaluation resnet50_aws_baseline resnet50_with_greyscale_aws resnet50_with_brightness_aws resnet50_with_contrast_aws resnet50_with_defocus_blur_aws resnet50_with_fog_aws resnet50_with_frost_aws resnet50_with_gaussian_noise_aws resnet50_with_jpeg_compression_aws resnet50_with_motion_blur_aws resnet50_with_pixelate_aws resnet50_with_saturate_aws resnet50_with_spatter_aws resnet50_with_zoom_blur_aws resnet50_with_gaussian_noise_contrast_motion_blur_jpeg_compression_aws FixResNeXt101_32x48d_v2 efficientnet-b8-advprop-autoaug Model 6569453953393658404664494364353641353126543748594736 6751301835273146233354352343201523212111411830493215 6375503953403958414866514269383640373627553649594835 60 65733752403155375058453861573342362826503450534333 1325104213108.9242335211525277.635118.8523242136211726 58 63593674402447353753443658443063372223443139484230 59 68383453723153374561514061273237503334503446544835 59 66334238416962375760524163283832392938573555564938 6265353941393271365560503964303636372635633553584836 4858374942252146 724652405360313737302129476347524446 505838454132235338 7253414062344138342432503568554638 6769443653383858374974493865333340343125553349694632 6368434050393256403763754263343738363231513646575537 414934504224173158324330 6952264235231328305233433563 6166734953436870725062495665624147403632606651594751 8384827083666977717783755979615268565735675568776951 8384728080717984828484816181595768636745776374807659 ImageNet Accuracies (top-1, %) Figure 15: A detailed view of corruption robustness, with cells sampled from the main grid in Figure 7. Here we present ResNet50s trained on some of the corruptions from the ImageNet-C benchmark, as well as the best model trained on more data, FixResNeXt101_32x48d_v2, and the best model trained on just the standard training set, efficientnet-b8-advprop-autoaug. We have already seen that corruption robustness does not promote effective robustness, or robustness to real distribution shift. Here, we analyze whether robustness to some corruptions transfers to others, and what may contribute to corruption robustness. Figure 15 shows the result of training various ResNet50s 3 on a few corruptions from ImageNet-C. In line with prior work, this plot here tells us that training against one type of synthetic corruption or one set of synthetic corruption does not transfer well to other corruptions. There are cases where transfer does happen, but overall the models are only robust to the corruption they are trained on. It is also interesting to note (from Figure 7) that PGD models actually see a drop in robustness to low frequency corruptions such as contrast, a phenomenon also observed in [107]. 3 Each ResNet50 was trained with a batch size of 256 for 120 epochs, starting with a learning rate of 0.1 and decaying by a factor of 10 every 30 epochs. For the ResNet50s trained on corruptions, we randomly sample a corruption and severity for each image. Refer to E.2 for details on corruptions and severities. We use our custom fast gpu implementations of these corruptions for training. 34 E Evaluation settings in the testbed E.1 Natural distribution shifts For ImageNetV2, we evaluate on the following datasets: imagenetv2-matched-frequency, imagenetv2- matched-frequency-format-val, imagenetv2-threshold-0.7, imagenetv2-threshold-0.7-format-val, imagenetv2- top-images, imagenetv2-top-images-format-val. The format-val versions are variants of the original dataset encoded with jpeg settings similar to the original one. Unless otherwise stated, results in our paper referring to imagenetv2 are for imagenetv2-matched-frequency-format-val. For ObjectNet, we obtained a beta version of the dataset through personal correspondance. Each image in the dataset was then cropped by 2px on each side following the authors’ instructions. Predictions were taken over only the classes that also appeared in the 1000 classes for the ImageNet validation set. For ImageNet-Vid-Robust and YTBB-Robust, we look at the anchor frames in the dataset and evaluate the benign accuracy for pm0. For pm10, we look at up to 20 nearest frames marked “similar” to the anchor frame in the dataset and count it as a misclassification if any one of the predictions is wrong. For ImageNet-A, predictions were taken over only the classes that also appeared in the 1000 classes for the ImageNet validation set. E.2 Corruptions We include 38 different corruption types: greyscale (in memory), gaussian noise (in memory and on disk), shot noise (in memory and on disk), impulse noise (in memory and on disk), speckle noise (in memory and on disk), gaussian blur (in memory and on disk), defocus blur (in memory and on disk), glass blur (on disk), motion blur (in memory and on disk), zoom blur (in memory and on disk), snow (in memory and on disk), frost (in memory and on disk), fog (in memory and on disk), spatter (in memory and on disk), brightness (in memory and on disk), contrast (in memory and on disk), saturate (in memory and on disk), pixelate (in memory and on disk), jpeg compression (in memory and on disk), elastic transform (in memory and on disk). For each corruption, we average over the five severities. We make sure to make the distinction between in memory corruptions, for which we provide custom fast gpu implementations, and on disk corruptions, for which we use the publicly available ImageNet- C dataset, since it was reported in [31] that jpeg compression can have a significant impact on model accuracies (indeed, as evidenced by Figure 15). E.3 Adversarial attacks We run the following 4 pgd attacks one each model with these settings: pgd.linf.eps0.5Norm: 0.5/255, Step size: 5.88e-5, Num steps: 100 35 pgd.linf.eps2Norm: 2/255, Step size: 2.35e-4, Num steps: 100 pgd.l2.eps0.1Norm: 0.1, Step size: 0.01, Num steps: 100 pgd.l2.eps0.5Norm: 0.5, Step size: 0.05, Num steps: 100 Most of the models were attacked with only 10% of the dataset (in a class-balanced manner) due to computational constraints. These models are displayed with larger error bars in the plots. E.4 Stylized Imagenet We use the stylized imagenet dataset used by [34] as another evaluation dataset. E.5 125 class evaluation For the 125 subsampled class evaluation, we evaluate on the following classes from ILSVRC: n01494475 n01630670 n01644373 n01644900 n01669191 n01677366 n01697457 n01742172 n01796340 n01829413 n01871265 n01924916 n01944390 n01978287 n01980166 n02007558 n02009229 n02017213 n02033041 n02037110 n02056570 n02071294 n02085936 n02086079 n02093428 n02093991 n02095314 n02095570 n02096294 n02096437 n02097474 n02100236 n02100583 n02102318 n02105056 n02107574 n02112706 n02113023 n02114855 n02128925 n02134418 n02138441 n02165105 n02219486 n02226429 n02264363 n02280649 n02441942 n02483708 n02486261 n02488291 n02492035 n02641379 n02730930 n02777292 n02790996 n02795169 n02808440 n02814533 n02814860 n02837789 n02859443 n02892201 n02895154 n02948072 n02951585 n02977058 n03000247 n03110669 n03201208 n03208938 n03216828 n03240683 n03250847 n03272562 n03297495 n03337140 n03376595 n03379051 n03447721 n03492542 n03527444 n03535780 n03642806 n03670208 n03673027 n03692522 n03710193 n03775071 n03832673 n03838899 n03840681 n03868242 n03873416 n03877845 n03884397 n03908714 n03920288 n03933933 n04004767 n04009552 n04037443 n04041544 n04067472 n04074963 n04099969 n04125021 n04141975 n04149813 n04204238 n04208210 n04229816 n04266014 n04310018 n04330267 n04335435 n04336792 n04355338 n04417672 n04479046 n04505470 n07715103 n07875152 n09256479 n12620546 36 F Models in the testbed The following list contains all models we evaluated on ImageNet with references and links to the corresponding source code. Also noted is the model type used to color the plots in the paper. 1.BiT-M-R50x1-ILSVRC2012 [49]. Trained with more data model.https://github.com/google-research/big _transfer 2. BiT-M-R50x3-ILSVRC2012 [49]. Trained with more data model.https://github.com/google-research/big _transfer 3.BiT-M-R101x1-ILSVRC2012 [49]. Trained with more data model.https://github.com/google-research/b ig_transfer 4.BiT-M-R101x3-ILSVRC2012 [49]. Trained with more data model.https://github.com/google-research/b ig_transfer 5. BiT-M-R152x4-ILSVRC2012 [49]. Trained with more data model.https://github.com/google-research/b ig_transfer 6. FixPNASNet [92]. Standard training model.https://github.com/facebookresearch/FixRes 7. FixResNeXt101_32x48d [92]. Trained with more data model.https://github.com/facebookresearch/FixRes 8. FixResNeXt101_32x48d_v2 [92]. Trained with more data model.https://github.com/facebookresearch/ FixRes 9. FixResNet50 [92]. Standard training model.https://github.com/facebookresearch/FixRes 10. FixResNet50CutMix [92]. Robustness intervention model.https://github.com/facebookresearch/FixRes 11.FixResNet50CutMix_v2 [92]. Robustness intervention model.https://github.com/facebookresearch/FixR es 12.FixResNet50_no_adaptation [92]. Standard training model.https://github.com/facebookresearch/FixRes 13. FixResNet50_v2 [92]. Standard training model.https://github.com/facebookresearch/FixRes 14. alexnet [50]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 15. alexnet_lpf2 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 16. alexnet_lpf3 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 17. alexnet_lpf5 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 18. bninception [46]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 19. bninception-imagenet21k [46]. Trained with more data model.https://github.com/dmlc/mxnet-model-gall ery/blob/master/imagenet-21k-inception.md 20. cafferesnet101 [37]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 21. densenet121 [43]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 22. densenet121_lpf2 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 23. densenet121_lpf3 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 24. densenet121_lpf5 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 25. densenet161 [43]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 26. densenet169 [43]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 37 27. densenet201 [43]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 28. dpn107 [11]. Trained with more data model.https://github.com/Cadene/pretrained-models.pytorch 29. dpn131 [11]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 30. dpn68 [11]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 31. dpn68b [11]. Trained with more data model.https://github.com/Cadene/pretrained-models.pytorch 32. dpn92 [11]. Trained with more data model.https://github.com/Cadene/pretrained-models.pytorch 33. dpn98 [11]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 34. efficientnet-b0 [88]. Standard training model.https://github.com/tensorflow/tpu/tree/master/models/o fficial/efficientnet 35.efficientnet-b0-advprop-autoaug [100]. Robustness intervention model.https://github.com/tensorflow/tpu/ tree/master/models/official/efficientnet 36. efficientnet-b0-autoaug [15]. Standard training model.https://github.com/tensorflow/tpu/tree/master/ models/official/efficientnet 37. efficientnet-b1 [88]. Standard training model.https://github.com/tensorflow/tpu/tree/master/models/o fficial/efficientnet 38.efficientnet-b1-advprop-autoaug [100]. Robustness intervention model.https://github.com/tensorflow/tpu/ tree/master/models/official/efficientnet 39.efficientnet-b1-autoaug [15]. Standard training model.https://github.com/tensorflow/tpu/tree/master/ models/official/efficientnet 40.efficientnet-b2 [88]. Standard training model.https://github.com/tensorflow/tpu/tree/master/models/o fficial/efficientnet 41.efficientnet-b2-advprop-autoaug [100]. Robustness intervention model.https://github.com/tensorflow/tpu/ tree/master/models/official/efficientnet 42.efficientnet-b2-autoaug [15]. Standard training model.https://github.com/tensorflow/tpu/tree/master/ models/official/efficientnet 43.efficientnet-b3 [88]. Standard training model.https://github.com/tensorflow/tpu/tree/master/models/o fficial/efficientnet 44.efficientnet-b3-advprop-autoaug [100]. Robustness intervention model.https://github.com/tensorflow/tpu/ tree/master/models/official/efficientnet 45.efficientnet-b3-autoaug [15]. Standard training model.https://github.com/tensorflow/tpu/tree/master/ models/official/efficientnet 46.efficientnet-b4 [88]. Standard training model.https://github.com/tensorflow/tpu/tree/master/models/o fficial/efficientnet 47.efficientnet-b4-advprop-autoaug [100]. Robustness intervention model.https://github.com/tensorflow/tpu/ tree/master/models/official/efficientnet 48.efficientnet-b4-autoaug [15]. Standard training model.https://github.com/tensorflow/tpu/tree/master/ models/official/efficientnet 49. efficientnet-b5 [88]. Standard training model.https://github.com/tensorflow/tpu/tree/master/models/o fficial/efficientnet 50.efficientnet-b5-advprop-autoaug [100]. Robustness intervention model.https://github.com/tensorflow/tpu/ tree/master/models/official/efficientnet 38 51.efficientnet-b5-autoaug [15]. Standard training model.https://github.com/tensorflow/tpu/tree/master/ models/official/efficientnet 52. efficientnet-b5-randaug [16]. Standard training model.https://github.com/tensorflow/tpu/tree/master/ models/official/efficientnet 53. efficientnet-b6-advprop-autoaug [100]. Robustness intervention model.https://github.com/tensorflow/tpu/ tree/master/models/official/efficientnet 54.efficientnet-b6-autoaug [15]. Standard training model.https://github.com/tensorflow/tpu/tree/master/ models/official/efficientnet 55. efficientnet-b7-advprop-autoaug [100]. Robustness intervention model.https://github.com/tensorflow/tpu/ tree/master/models/official/efficientnet 56.efficientnet-b7-autoaug [15]. Standard training model.https://github.com/tensorflow/tpu/tree/master/ models/official/efficientnet 57. efficientnet-b7-randaug [16]. Standard training model.https://github.com/tensorflow/tpu/tree/master/ models/official/efficientnet 58.efficientnet-b8-advprop-autoaug [100]. Robustness intervention model.https://github.com/tensorflow/tpu/ tree/master/models/official/efficientnet 59.efficientnet-l2-noisystudent [102]. Trained with more data model.https://github.com/rwightman/pytorch-i mage-models 60.facebook_adv_trained_resnet152_baseline [101]. Robustness intervention model.https://github.com/fac ebookresearch/ImageNet-Adversarial-Training 61.facebook_adv_trained_resnet152_denoise [101]. Robustness intervention model.https://github.com/faceb ookresearch/ImageNet-Adversarial-Training 62.facebook_adv_trained_resnext101_denoiseAll [101]. Robustness intervention model.https://github.com/f acebookresearch/ImageNet-Adversarial-Training 63. fbresnet152 [37]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 64. google_resnet101_jft-300M [82]. Trained with more data model. 65. googlenet/inceptionv1 [85]. Standard training model.https://github.com/pytorch/vision/tree/master/to rchvision/models 66. inceptionresnetv2 [37]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 67. inceptionv3 [86]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 68. inceptionv4 [87]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 69. instagram-resnext101_32x16d [56]. Trained with more data model.https://github.com/facebookresearch/ WSL-Images 70.instagram-resnext101_32x32d [56]. Trained with more data model.https://github.com/facebookresearch/ WSL-Images 71.instagram-resnext101_32x48d [56]. Trained with more data model.https://github.com/facebookresearch/ WSL-Images 72.instagram-resnext101_32x8d [56]. Trained with more data model.https://github.com/facebookresearch/ WSL-Images 73. mnasnet0_5 [89]. Standard training model.https://github.com/pytorch/vision/tree/master/torchvisio n/models 39 74.mnasnet1_0 [89]. Standard training model.https://github.com/pytorch/vision/tree/master/torchvisio n/models 75. mobilenet_v2 [73]. Standard training model.https://github.com/pytorch/vision/tree/master/torchvis ion/models 76. mobilenet_v2_lpf2 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 77. mobilenet_v2_lpf3 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 78. mobilenet_v2_lpf5 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 79. nasnetalarge [117]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 80. nasnetamobile [117]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 81. pnasnet5large [53]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 82. polynet [116]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 83. resnet101 [37]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 84. resnet101-tencent-ml-images [97]. Trained with more data model.https://github.com/Tencent/tencent-ml- images 85. resnet101_cutmix [108]. Robustness intervention model.https://github.com/clovaai/CutMix-PyTorch 86. resnet101_lpf2 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 87. resnet101_lpf3 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 88. resnet101_lpf5 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 89. resnet152 [37]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 90. resnet152-imagenet11k [99]. Trained with more data model.https://github.com/tornadomeet/ResNet 91. resnet18 [37]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 92. resnet18-rotation-nocrop_40 [28]. Robustness intervention model.https://github.com/MadryLab/spatial- pytorch 93.resnet18-rotation-random_30 [28]. Robustness intervention model.https://github.com/MadryLab/spatial- pytorch 94.resnet18-rotation-random_40 [28]. Robustness intervention model.https://github.com/MadryLab/spatial- pytorch 95.resnet18-rotation-standard_40 [28]. Robustness intervention model.https://github.com/MadryLab/spatial- pytorch 96.resnet18-rotation-worst10_30 [28]. Robustness intervention model.https://github.com/MadryLab/spatial- pytorch 97.resnet18-rotation-worst10_40 [28]. Robustness intervention model.https://github.com/MadryLab/spatial- pytorch 98. resnet18_lpf2 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 99. resnet18_lpf3 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 100. resnet18_lpf5 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 101. resnet18_ssl [104]. Trained with more data model.https://github.com/facebookresearch/semi-supervise d-ImageNet1K-models 40 102.resnet18_swsl [104]. Trained with more data model.https://github.com/facebookresearch/semi-supervi sed-ImageNet1K-models 103. resnet34 [37]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 104. resnet34_lpf2 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 105. resnet34_lpf3 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 106. resnet34_lpf5 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 107. resnet50 [37]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 108. resnet50-randomized_smoothing_noise_0.00 [13]. Standard training model.https://github.com/locuslab/ smoothing 109. resnet50-randomized_smoothing_noise_0.25 [13]. Robustness intervention model.https://github.com/loc uslab/smoothing 110. resnet50-randomized_smoothing_noise_0.50 [13]. Robustness intervention model.https://github.com/loc uslab/smoothing 111. resnet50-randomized_smoothing_noise_1.00 [13]. Robustness intervention model.https://github.com/loc uslab/smoothing 112. resnet50-smoothing_adversarial_DNN_2steps_eps_512_noise_0.25 [72]. Robustness intervention model. https://github.com/Hadisalman/smoothing-adversarial 113.resnet50-smoothing_adversarial_DNN_2steps_eps_512_noise_0.50 [72]. Robustness intervention model. https://github.com/Hadisalman/smoothing-adversarial 114.resnet50-smoothing_adversarial_DNN_2steps_eps_512_noise_1.00 [72]. Robustness intervention model. https://github.com/Hadisalman/smoothing-adversarial 115.resnet50-smoothing_adversarial_PGD_1step_eps_512_noise_0.25 [72]. Robustness intervention model. https://github.com/Hadisalman/smoothing-adversarial 116.resnet50-smoothing_adversarial_PGD_1step_eps_512_noise_0.50 [72]. Robustness intervention model. https://github.com/Hadisalman/smoothing-adversarial 117.resnet50-smoothing_adversarial_PGD_1step_eps_512_noise_1.00 [72]. Robustness intervention model. https://github.com/Hadisalman/smoothing-adversarial 118. resnet50-vtab [112]. Standard training model.https://tfhub.dev/s?publisher=vtab 119. resnet50-vtab-exemplar [112]. Standard training model.https://tfhub.dev/s?publisher=vtab 120. resnet50-vtab-rotation [112]. Standard training model.https://tfhub.dev/s?publisher=vtab 121. resnet50-vtab-semi-exemplar [112]. Standard training model.https://tfhub.dev/s?publisher=vtab 122. resnet50-vtab-semi-rotation [112]. Standard training model.https://tfhub.dev/s?publisher=vtab 123.resnet50_adv-train-free [74]. Robustness intervention model.https://github.com/mahyarnajibi/FreeAdve rsarialTraining 124. resnet50_augmix [41]. Robustness intervention model.https://github.com/google-research/augmix 125. resnet50_aws_baseline. Standard training model. 126. resnet50_cutmix [108]. Robustness intervention model.https://github.com/clovaai/CutMix-PyTorch 127. resnet50_cutout [20]. Robustness intervention model.https://github.com/clovaai/CutMix-PyTorch 128. resnet50_deepaugment [40]. Robustness intervention model.https://github.com/hendrycks/imagenet-r 41 129.resnet50_deepaugment_augmix [40]. Robustness intervention model.https://github.com/hendrycks/image net-r 130. resnet50_feature_cutmix [108]. Robustness intervention model.https://github.com/clovaai/CutMix-PyTo rch 131. resnet50_imagenet_100percent_batch64_original_images. Standard training model. 132. resnet50_imagenet_subsample_125_classes_batch64_original_images. Standard training model. 133. resnet50_imagenet_subsample_1_of_16_batch64_original_images. Standard training model. 134. resnet50_imagenet_subsample_1_of_2_batch64_original_images. Standard training model. 135. resnet50_imagenet_subsample_1_of_32_batch64_original_images. Standard training model. 136. resnet50_imagenet_subsample_1_of_4_batch64_original_images. Standard training model. 137. resnet50_imagenet_subsample_1_of_8_batch64_original_images. Standard training model. 138. resnet50_imagenet_subsample_250_classes_batch64_original_images. Standard training model. 139. resnet50_imagenet_subsample_500_classes_batch64_original_images. Standard training model. 140. resnet50_l2_eps3_robust [27]. Robustness intervention model.https://github.com/MadryLab/robustness 141. resnet50_linf_eps4_robust [27]. Robustness intervention model.https://github.com/MadryLab/robustness 142.resnet50_linf_eps8_robust [27]. Robustness intervention model.https://github.com/MadryLab/robustness 143. resnet50_lpf2 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 144. resnet50_lpf3 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 145. resnet50_lpf5 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 146. resnet50_mixup [113]. Robustness intervention model.https://github.com/clovaai/CutMix-PyTorch 147. resnet50_ssl [104]. Trained with more data model.https://github.com/facebookresearch/semi-supervise d-ImageNet1K-models 148.resnet50_swsl [104]. Trained with more data model.https://github.com/facebookresearch/semi-supervi sed-ImageNet1K-models 149.resnet50_trained_on_SIN [34]. Robustness intervention model.https://github.com/rgeirhos/texture-vs -shape 150.resnet50_trained_on_SIN_and_IN [34]. Robustness intervention model.https://github.com/rgeirhos/te xture-vs-shape 151.resnet50_trained_on_SIN_and_IN_then_finetuned_on_IN [34]. Robustness intervention model.https: //github.com/rgeirhos/texture-vs-shape 152. resnet50_with_brightness_aws. Robustness intervention model. 153. resnet50_with_contrast_aws. Robustness intervention model. 154. resnet50_with_defocus_blur_aws. Robustness intervention model. 155. resnet50_with_fog_aws. Robustness intervention model. 156. resnet50_with_frost_aws. Robustness intervention model. 157. resnet50_with_gaussian_noise_aws. Robustness intervention model. 158. resnet50_with_gaussian_noise_contrast_motion_blur_jpeg_compression_aws. Robustness intervention model. 42 159. resnet50_with_greyscale_aws. Robustness intervention model. 160. resnet50_with_jpeg_compression_aws. Robustness intervention model. 161. resnet50_with_motion_blur_aws. Robustness intervention model. 162. resnet50_with_pixelate_aws. Robustness intervention model. 163. resnet50_with_saturate_aws. Robustness intervention model. 164. resnet50_with_spatter_aws. Robustness intervention model. 165. resnet50_with_zoom_blur_aws. Robustness intervention model. 166. resnext101_32x16d_ssl [104]. Trained with more data model.https://github.com/facebookresearch/semi -supervised-ImageNet1K-models 167.resnext101_32x4d [103]. Standard training model.https://github.com/Cadene/pretrained-models.pytorc h 168. resnext101_32x4d_ssl [104]. Trained with more data model.https://github.com/facebookresearch/semi -supervised-ImageNet1K-models 169. resnext101_32x4d_swsl [104]. Trained with more data model.https://github.com/facebookresearch/semi -supervised-ImageNet1K-models 170.resnext101_32x8d [103]. Standard training model.https://github.com/pytorch/vision/tree/master/torc hvision/models 171.resnext101_32x8d_deepaugment_augmix [40]. Robustness intervention model.https://github.com/hendr ycks/imagenet-r 172.resnext101_32x8d_ssl [104]. Trained with more data model.https://github.com/facebookresearch/semi -supervised-ImageNet1K-models 173.resnext101_32x8d_swsl [104]. Trained with more data model.https://github.com/facebookresearch/semi -supervised-ImageNet1K-models 174.resnext101_64x4d [103]. Standard training model.https://github.com/Cadene/pretrained-models.pytorc h 175.resnext50_32x4d [103]. Standard training model.https://github.com/pytorch/vision/tree/master/torc hvision/models 176.resnext50_32x4d_ssl [104]. Trained with more data model.https://github.com/facebookresearch/semi-s upervised-ImageNet1K-models 177.resnext50_32x4d_swsl [104]. Trained with more data model.https://github.com/facebookresearch/semi -supervised-ImageNet1K-models 178. se_resnet101 [42]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 179. se_resnet152 [42]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 180. se_resnet50 [42]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 181.se_resnext101_32x4d [42]. Standard training model.https://github.com/Cadene/pretrained-models.pyto rch 182. se_resnext50_32x4d [42]. Standard training model.https://github.com/Cadene/pretrained-models.pyto rch 183. senet154 [42]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 43 184.shufflenet_v2_x0_5 [54]. Standard training model.https://github.com/pytorch/vision/tree/master/to rchvision/models 185. shufflenet_v2_x1_0 [54]. Standard training model.https://github.com/pytorch/vision/tree/master/to rchvision/models 186. squeezenet1_0 [45]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 187. squeezenet1_1 [45]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 188. vgg11 [78]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 189. vgg11_bn [78]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 190. vgg13 [78]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 191. vgg13_bn [78]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 192. vgg16 [78]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 193. vgg16_bn [78]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 194. vgg16_bn_lpf2 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 195. vgg16_bn_lpf3 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 196. vgg16_bn_lpf5 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 197. vgg16_lpf2 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 198. vgg16_lpf3 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 199. vgg16_lpf5 [115]. Robustness intervention model.https://github.com/adobe/antialiased-cnns 200. vgg19 [78]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 201. vgg19_bn [78]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 202. wide_resnet101_2 [109]. Standard training model.https://github.com/pytorch/vision/tree/master/to rchvision/models 203.wide_resnet50_2 [109]. Standard training model.https://github.com/pytorch/vision/tree/master/torc hvision/models 204. xception [12]. Standard training model.https://github.com/Cadene/pretrained-models.pytorch 44 G Model accuracies Table 1: Top-1 model accuracies on ImageNet validation set, effective robustness as calculated with respect to ImageNetV2, an average over all the corruptions, and an average over all the pgd attacks. Note that since we take an average of many attacks, the PGD column can no longer be considered a worst-case attacker for the model (look to E.3 for specific attacks). Model accuracies ModelImageNet accuracy ImageNetV2 eff. robust. Avg. corr. accuracy Avg. PGD accuracy efficientnet-l2-noisystudent88.321.11 FixResNeXt101_32x48d_v286.360.9765.65 FixResNeXt101_32x48d86.260.9565.56 instagram-resnext101_32x48d85.441.2665.5324.1 efficientnet-b8-advprop-autoaug85.370.5171.85 BiT-M-R152x4-ILSVRC201285.18-0.3167.26 efficientnet-b7-advprop-autoaug85.090.6668.92 instagram-resnext101_32x32d85.091.5464.7724.4 BiT-M-R101x3-ILSVRC201284.78-1.3563.44 efficientnet-b6-advprop-autoaug84.760.7568.6550.67 efficientnet-b7-randaug84.730.1169.12 efficientnet-b7-autoaug84.330.3262.77 efficientnet-b5-advprop-autoaug84.30.5167.7650.17 resnext101_32x8d_swsl84.291.1963.1723.22 instagram-resnext101_32x16d84.181.5163.2229.19 BiT-M-R50x3-ILSVRC201284.15-0.7660.23 efficientnet-b6-autoaug84.130.1463.4234.29 FixPNASNet83.7-0.061.3522.8 efficientnet-b5-autoaug83.630.2562.332.43 efficientnet-b5-randaug83.530.0863.3534.41 resnext101_32x4d_swsl83.231.4160.0921.73 efficientnet-b583.110.1760.2835.18 pnasnet5large82.740.2161.7629.46 instagram-resnext101_32x8d82.691.5960.8130.13 efficientnet-b4-advprop-autoaug82.690.4264.8850.72 efficientnet-b4-autoaug82.550.1759.5934.24 BiT-M-R101x1-ILSVRC201282.52-0.4258.28 nasnetalarge82.510.4861.7436.99 efficientnet-b482.23-0.6457.237.06 resnext50_32x4d_swsl82.181.2656.3821.09 resnext101_32x16d_ssl81.840.358.6322.34 resnext101_32x8d_ssl81.630.7357.9620.82 senet15481.3-0.0754.1130.65 resnet50_swsl81.181.3553.9521.39 efficientnet-b3-advprop-autoaug81.090.2960.651.09 efficientnet-b3-autoaug81.050.1755.531.76 resnext101_32x4d_ssl80.930.4855.6520.54 polynet80.860.3654.0223.05 BiT-M-R50x1-ILSVRC201280.4-0.6352.2112.5 resnext50_32x4d_ssl80.330.4452.5719.75 inceptionresnetv280.270.3256.8534.85 se_resnext101_32x4d80.240.4752.2628.77 efficientnet-b380.21-0.4853.3134.22 inceptionv480.080.555.5228.02 Table continues onto next page 45 Model accuracies(continued from previous page) ModelImageNet accuracy ImageNetV2 eff. robust. Avg. corr. accuracy Avg. PGD accuracy resnext101_32x8d_deepaugment _augmix 79.90.2565.56 resnet101_cutmix79.83-0.3950.1525.6 efficientnet-b2-autoaug79.780.1753.530.93 FixResNet50CutMix_v279.76-1.2143.4418.19 dpn10779.75-0.4752.3730.64 FixResNet50CutMix79.74-1.2243.3918.14 efficientnet-b2-advprop-autoaug79.6-0.2555.1746.33 dpn13179.43-0.252.0630.38 dpn9279.4-0.6549.2925.69 resnext101_32x8d79.31-0.3449.6825.38 resnet50_ssl79.230.5250.1520.57 dpn9879.220.0851.8230.14 google_resnet101_jft-300M79.2-0.2353.4926.84 FixResNet50_v279.1-0.6243.3115.38 se_resnext50_32x4d79.080.2750.6524.74 FixResNet5079.0-0.6743.2515.3 resnext101_64x4d78.96-0.252.0623.57 efficientnet-b278.89-0.3950.0533.88 wide_resnet101_278.85-0.8748.225.24 xception78.820.0651.726.32 efficientnet-b1-autoaug78.72-0.0751.1930.69 se_resnet15278.660.4550.9428.42 resnet50_cutmix78.6-1.144.726.46 efficientnet-b1-advprop-autoaug78.54-0.2353.746.54 wide_resnet50_278.47-0.6146.2326.13 se_resnet10178.40.4350.1228.2 resnet15278.310.2747.8122.48 resnet101-tencent-ml-images78.250.0447.77 resnet50_feature_cutmix78.21-0.4244.3325.36 resnext101_32x4d78.19-0.1350.9622.38 resnet101_lpf378.12-0.2746.5222.48 efficientnet-b177.91-0.2447.0731.33 resnet101_lpf577.910.146.5423.13 resnet101_lpf277.80.346.0622.01 se_resnet5077.640.0848.1127.55 resnext50_32x4d77.620.145.5622.52 resnet50_augmix77.54-0.5350.7826.01 resnet50_mixup77.47-0.5448.221.95 fbresnet15277.390.0249.9823.4 resnet10177.370.0146.0621.85 inceptionv377.320.2949.8325.72 densenet16177.140.1349.3622.22 efficientnet-b0-advprop-autoaug77.080.2149.944.31 resnet50_cutout77.07-0.6543.8119.8 FixResNet50_no_adaptation77.04-0.0244.6820.61 dpn68b77.03-0.2845.6718.7 resnet50_lpf577.03-0.5343.5422.03 densenet20176.9-0.1247.6323.95 efficientnet-b0-autoaug76.84-0.3945.2730.66 resnet50_lpf376.82-0.1243.321.83 resnet50_lpf276.79-0.2542.2220.91 Table continues onto next page 46 Model accuracies(continued from previous page) ModelImageNet accuracy ImageNetV2 eff. robust. Avg. corr. accuracy Avg. PGD accuracy resnet50_trained_on_SIN_and_I N_then_finetuned_on_IN 76.72-0.0443.9622.78 resnet50_deepaugment76.660.7353.9129.65 efficientnet-b076.53-0.7943.8431.05 resnet50-vtab-rotation76.5-0.4941.93 cafferesnet10176.20.0844.8325.54 resnet152-imagenet11k76.182.0947.3330.64 resnet50_aws_baseline76.14-0.3642.1321.46 resnet5076.13-0.7741.5921.24 resnet50_imagenet_100percent_b atch64_original_images 75.98-0.5641.6121.89 dpn6875.87-0.5645.4617.71 resnet50_deepaugment_augmix75.82-0.0858.2933.73 resnet50- randomized_smoothing_noise_0.00 75.690.3141.7521.32 densenet16975.60.1946.6721.79 resnet50-vtab75.540.2243.61 resnet50_with_brightness_aws75.28-0.2843.922.78 resnet50_with_spatter_aws75.21-0.2942.8122.45 densenet121_lpf375.14-0.3540.4820.01 densenet121_lpf575.030.1341.8421.13 densenet121_lpf275.030.4141.2420.82 resnet50_with_saturate_aws74.89-0.2742.420.46 resnet50_trained_on_SIN_and_IN74.590.5547.9122.96 resnet34_lpf274.480.1541.5420.96 densenet12174.430.1343.5420.01 resnet34_lpf374.340.2542.2220.97 vgg19_bn74.220.1837.9416.51 resnet34_lpf574.190.4641.2221.09 resnet50-vtab-exemplar74.10.344.73 nasnetamobile74.08-0.2944.7822.89 vgg16_bn_lpf574.04-0.436.1918.91 vgg16_bn_lpf274.010.1336.0617.81 vgg16_bn_lpf373.920.536.3318.33 resnet50_with_frost_aws73.780.2942.3920.96 resnet50_with_jpeg_compression _aws 73.63-0.2141.7638.34 bninception73.521.040.5921.27 mnasnet1_073.46-0.4736.4218.78 vgg16_bn73.36-0.0935.6916.19 resnet3473.310.1240.4821.23 resnet18_swsl73.291.7439.9518.79 resnet50_with_gaussian_noise_aws72.970.2145.5643.88 resnet50_with_gaussian_noise_c ontrast_motion_blur_jpeg_comp ression_aws 72.720.0551.822.91 mobilenet_v2_lpf272.62-0.5634.4617.46 resnet18_ssl72.61.2439.5119.19 mobilenet_v2_lpf372.57-0.2334.7817.6 mobilenet_v2_lpf572.51-0.134.917.73 vgg1972.38-0.0132.4320.65 vgg16_lpf572.330.1531.8919.86 Table continues onto next page 47 Model accuracies(continued from previous page) ModelImageNet accuracy ImageNetV2 eff. robust. Avg. corr. accuracy Avg. PGD accuracy vgg16_lpf372.19-0.1932.1819.37 vgg16_lpf272.16-0.231.9819.13 resnet50_with_contrast_aws72.0-0.4240.8517.29 mobilenet_v271.88-0.1333.9617.49 resnet50_with_fog_aws71.76-0.8337.917.19 resnet18_lpf371.68-0.4336.8420.17 vgg1671.59-0.3331.320.14 vgg13_bn71.590.0131.7615.16 resnet18_lpf271.39-0.0936.8819.8 resnet18_lpf571.39-0.5136.8620.22 resnet18-rotation-standard_4071.28-0.0536.4620.26 vgg11_bn70.37-0.131.718.05 resnet50- randomized_smoothing_noise_0.25 70.290.2840.6663.94 vgg1369.93-0.2728.5319.32 googlenet/inceptionv169.781.0138.8421.85 resnet1869.760.4635.0119.51 shufflenet_v2_x1_069.36-0.4830.8716.66 resnet18-rotation-worst10_3069.130.7234.0622.51 vgg1169.02-0.3328.6122.38 resnet18-rotation-random_3068.880.1932.8818.63 resnet18-rotation-worst10_4068.6-0.0532.2422.65 resnet50_with_pixelate_aws68.51.1739.5818.85 resnet18-rotation-random_4068.350.7331.8717.89 facebook_adv_trained_resnext101 _denoiseAll 68.33-0.1140.8641.42 resnet50-smoothing_adversarial _DNN_2steps_eps_512_noise_0. 25 67.87-0.3140.5762.89 mnasnet0_567.6-0.3727.917.39 resnet50_with_motion_blur_aws67.461.4938.7115.34 resnet18-rotation-nocrop_4065.371.2330.120.5 facebook_adv_trained_resnet152 _denoise 65.320.3837.9739.48 bninception-imagenet21k65.241.7832.830.3 resnet50- randomized_smoothing_noise_0.50 64.240.0439.861.41 resnet50_with_greyscale_aws63.330.4928.3318.16 resnet50_linf_eps4_robust62.420.5332.3760.3 facebook_adv_trained_resnet152 _baseline 62.340.5835.7737.63 resnet50-smoothing_adversarial _DNN_2steps_eps_512_noise_0. 50 62.19-0.0439.1459.26 resnet50-vtab-semi-exemplar61.620.9833.85 resnet50_with_zoom_blur_aws61.251.2233.2713.01 resnet50-vtab-semi-rotation60.920.9426.38 shufflenet_v2_x0_560.55-0.2723.5816.08 resnet50_adv-train-free60.49-0.0329.4157.42 resnet50-smoothing_adversarial _PGD_1step_eps_512_noise_0.2 5 60.47-0.4537.2158.49 Table continues onto next page 48 Model accuracies(continued from previous page) ModelImageNet accuracy ImageNetV2 eff. robust. Avg. corr. accuracy Avg. PGD accuracy resnet50_trained_on_SIN60.181.439.4219.25 squeezenet1_158.180.1220.1816.08 squeezenet1_058.09-0.2620.1718.06 resnet50_l2_eps3_robust57.90.3331.8356.25 alexnet_lpf257.23-0.3822.5429.09 alexnet_lpf356.89-0.4122.7730.67 alexnet_lpf556.58-0.4122.7731.71 alexnet56.52-0.2821.5524.08 resnet50-smoothing_adversarial _PGD_1step_eps_512_noise_0.5 0 54.66-0.3135.753.09 resnet50- randomized_smoothing_noise_1.00 53.120.1234.9352.11 resnet50-smoothing_adversarial _DNN_2steps_eps_512_noise_1. 00 51.870.2334.4350.95 resnet50_linf_eps8_robust47.911.3523.9346.97 resnet50-smoothing_adversarial _PGD_1step_eps_512_noise_1.0 0 44.280.229.8443.57 resnet50_with_defocus_blur_aws31.91.318.189.29 End of table 49 H Synthetic robustness correlation with natural robustness In this section, we investigate which individual synthetic robustness measures are most predictive of natural distribution shift. For each of the synthetic shifts in our testbed, we compute the effective robustness for each model and measure the Pearson correlation coefficients against the effective robustness under each of the natural distribution shifts in our testbed. Table 2 provides a full list of the correlation numbers, and Figures 16 to 22 show scatter plots of the two highest correlated synthetic shifts for each natural distribution shift. We find that some of the synthetic shifts are more predictive than others, but none have high correlation with all of the natural shifts. For instance,` p -robustness has the highest correlation with consistency shifts, but only low correlation with dataset shifts. On the other hand, some image corruptions such as brightness, gaussian blur, defocus blur, and saturate have higher correlation with the dataset shifts. It is worth nothing our testbed indicates that these synthetic measures are not causal, i.e., models trained on brightness, gaussian blur, defocus blur, or saturate do not have significant positive effective robustness on dataset shifts. Further analyzing these fine-grained connections between synthetic and natural forms of distribution shift is an important direction for future work. Table 2: Pearson correlation coefficients between all synthetic and natural distribution shifts in our testbed. For each distribution shift, effective robustness was calculated using a linear fit on the standard models. The correlation between synthetic and natural effective robustness was then only computed after filtering out the standard models. Pearson correlation coefficients Synthetic shiftImageNetV2ObjectNetImageNetVid (pm-0) YTBB (pm-0) ImageNetVid (pm-10) YTBB (pm-10) ImageNet- A avg_corruptions0.250.060.60.50.650.520.02 avg_pgd-0.04-0.190.30.350.840.7-0.12 brightness_in- memory 0.340.110.320.30.290.230.13 brightness_on-disk0.560.480.560.390.220.150.16 contrast_in- memory 0.150.070.140.04-0.61-0.50.14 contrast_on-disk0.260.280.170.05-0.61-0.540.15 defocus_blur_in- memory 0.27-0.040.660.560.430.27-0.05 defocus_blur_on- disk 0.390.390.650.490.280.170.12 elastic_transform_in- memory 0.14-0.120.490.420.750.63-0.15 elastic_transform_on- disk 0.30.210.570.410.650.580.01 fog_in-memory0.140.07-0.04-0.07-0.59-0.560.02 fog_on-disk0.280.310.04-0.03-0.64-0.60.04 frost_in-memory0.15-0.120.420.440.540.42-0.02 frost_on-disk0.320.150.530.450.440.360.08 gaussian_blur_in- memory 0.27-0.070.670.570.470.33-0.05 gaussian_blur_on- disk 0.410.40.650.480.260.160.13 Table continues onto next page 50 Pearson correlation coefficients(continued from previous page) Synthetic shiftImageNetV2ObjectNetImageNetVid (pm-0) YTBB (pm-0) ImageNetVid (pm-10) YTBB (pm-10) ImageNet- A gaussian_noise_in- memory -0.01-0.130.410.380.680.51-0.04 gaussian_noise_on- disk 0.080.00.40.340.710.620.07 glass_blur_on-disk0.240.170.560.450.610.53-0.0 greyscale0.30.170.110.290.09-0.060.04 impulse_noise_in- memory -0.06-0.10.350.340.650.45-0.05 impulse_noise_on- disk 0.040.00.340.310.720.60.03 jpeg_compression_in- memory 0.04-0.110.430.410.80.62-0.01 jpeg_compression_on- disk 0.090.010.440.40.80.650.03 motion_blur_in- memory 0.2-0.020.510.430.560.41-0.08 motion_blur_on- disk 0.320.250.580.430.390.310.07 pgd.l2.eps0.1-0.03-0.010.180.250.640.44-0.33 pgd.l2.eps0.5-0.05-0.220.310.340.710.63-0.11 pgd.linf.eps0.5-0.05-0.230.280.330.840.7-0.13 pgd.linf.eps20.01-0.180.30.310.760.690.05 pixelate_in- memory 0.270.030.610.480.660.530.05 pixelate_on-disk0.310.160.620.460.630.540.12 saturate_in- memory 0.350.080.40.430.380.270.12 saturate_on-disk0.550.430.460.410.260.160.13 shot_noise_in- memory -0.01-0.140.410.390.690.51-0.05 shot_noise_on- disk 0.07-0.010.40.350.710.620.06 snow_in-memory0.260.020.390.350.610.550.04 snow_on-disk0.330.140.50.430.60.510.04 spatter_in-memory0.09-0.060.360.360.80.66-0.08 spatter_on-disk0.260.080.50.430.750.63-0.04 speckle_noise_in- memory 0.0-0.130.430.390.710.55-0.04 speckle_noise_on- disk 0.08-0.020.420.360.740.660.02 stylized_imagenet0.320.240.310.30.440.31-0.02 zoom_blur_in- memory 0.210.230.450.350.450.36-0.01 zoom_blur_on- disk 0.260.210.550.410.490.390.0 End of table 51 606570758085 ImageNet (top-1, %) 40 45 50 55 60 65 70 75 80 Brightness (on-disk) (top-1, %) Distribution Shift to Brightness (on-disk) -505 Brightness (on-disk) Effective Robustness -1 0 1 2 ImagetNetV2 Effective Robustness Effective Robustness Scatterplot 606570758085 ImageNet (top-1, %) 35 40 45 50 55 60 65 70 75 80 Saturate (on-disk) (top-1, %) Distribution Shift to Saturate (on-disk) -50510 Saturate (on-disk) Effective Robustness -1 0 1 2 ImagetNetV2 Effective Robustness Effective Robustness Scatterplot y = x Standard training Lp adversarially robust Other robustness intervention Trained with more data Linear fit Figure 16: Plots of the two synthetic distribution shifts with the highest correlation with ImageNetV2, compared similarly to Figure 5. 52 606570758085 ImageNet (top-1, %) 5 10 15 20 25 30 35 40 45 50 55 60 65 Contrast (on-disk) (top-1, %) Distribution Shift to Contrast (on-disk) -15-10-50510152025 Contrast (on-disk) Effective Robustness -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 ObjectNet Effective Robustness Effective Robustness Scatterplot 606570758085 ImageNet (top-1, %) 5 10 15 20 25 30 35 40 45 50 55 60 65 Fog (on-disk) (top-1, %) Distribution Shift to Fog (on-disk) -25-20-15-10-50510152025 Fog (on-disk) Effective Robustness -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 ObjectNet Effective Robustness Effective Robustness Scatterplot y = x Standard training Lp adversarially robust Other robustness intervention Trained with more data Linear fit Figure 17: Plots of the two synthetic distribution shifts with the highest correlation with ObjectNet, compared similarly to Figure 5. 53 606570758085 ImageNet (top-1, %) 20 25 30 35 40 45 50 55 60 Gaussian Blur (on-disk) (top-1, %) Distribution Shift to Gaussian Blur (on-disk) -10-505101520 Gaussian Blur (on-disk) Effective Robustness -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 ImageNet-Vid-Robust (pm-0) Effective Robustness Effective Robustness Scatterplot 606570758085 ImageNet (top-1, %) 15 20 25 30 35 40 45 50 55 60 Defocus Blur (on-disk) (top-1, %) Distribution Shift to Defocus Blur (on-disk) -10-5051015202530 Defocus Blur (on-disk) Effective Robustness -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 ImageNet-Vid-Robust (pm-0) Effective Robustness Effective Robustness Scatterplot y = x Standard training Lp adversarially robust Other robustness intervention Trained with more data Linear fit Figure 18: Plots of the two synthetic distribution shifts with the highest correlation with ImageNet- Vid-Robust pm-0, compared similarly to Figure 5. 54 606570758085 ImageNet (top-1, %) 15 20 25 30 35 40 45 50 55 60 Defocus Blur (on-disk) (top-1, %) Distribution Shift to Defocus Blur (on-disk) -10-5051015202530 Defocus Blur (on-disk) Effective Robustness -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 YTBB-Robust (pm-0) Effective Robustness Effective Robustness Scatterplot 606570758085 ImageNet (top-1, %) 20 25 30 35 40 45 50 55 60 65 70 75 80 Defocus Blur (in-memory) (top-1, %) Distribution Shift to Defocus Blur (in-memory) -10-50510152025303540 Defocus Blur (in-memory) Effective Robustness -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 YTBB-Robust (pm-0) Effective Robustness Effective Robustness Scatterplot y = x Standard training Lp adversarially robust Other robustness intervention Trained with more data Linear fit Figure 19: Plots of the two synthetic distribution shifts with the highest correlation with YTBB- Robust pm-0, compared similarly to Figure 5. 55 606570758085 ImageNet (top-1, %) 10 15 20 25 30 35 40 45 50 55 60 65 70 PGD (Linf Eps0.5) (top-1, %) Distribution Shift to PGD (Linf Eps0.5) -15-10-50510152025303540455055 PGD (Linf Eps0.5) Effective Robustness -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 ImageNet-Vid-Robust (pm-10) Effective Robustness Effective Robustness Scatterplot 606570758085 ImageNet (top-1, %) 30 35 40 45 50 55 60 65 70 75 80 Spatter (in-memory) (top-1, %) Distribution Shift to Spatter (in-memory) -10-50510152025 Spatter (in-memory) Effective Robustness -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 ImageNet-Vid-Robust (pm-10) Effective Robustness Effective Robustness Scatterplot y = x Standard training Lp adversarially robust Other robustness intervention Trained with more data Linear fit Figure 20: Plots of the two synthetic distribution shifts with the highest correlation with ImageNet- Vid-Robust pm-10, compared similarly to Figure 5. 56 606570758085 ImageNet (top-1, %) 5 10 15 20 25 30 35 40 45 50 55 60 65 70 PGD (L2 Eps0.5) (top-1, %) Distribution Shift to PGD (L2 Eps0.5) -5051015202530354045505560 PGD (L2 Eps0.5) Effective Robustness -4 -3 -2 -1 0 1 2 3 4 5 6 7 YTBB-Robust (pm-10) Effective Robustness Effective Robustness Scatterplot 606570758085 ImageNet (top-1, %) 15 20 25 30 35 40 45 50 55 60 65 70 Speckle Noise (on-disk)) (top-1, %) Distribution Shift to Speckle Noise (on-disk)) -10-505101520253035 Speckle Noise (on-disk)) Effective Robustness -4 -3 -2 -1 0 1 2 3 4 5 6 7 YTBB-Robust (pm-10) Effective Robustness Effective Robustness Scatterplot y = x Standard training Lp adversarially robust Other robustness intervention Trained with more data Linear fit Figure 21: Plots of the two synthetic distribution shifts with the highest correlation with YTBB- Robust pm-10, compared similarly to Figure 5. 57 606570758085 ImageNet (top-1, %) 5 10 15 20 25 30 35 40 45 50 55 60 65 Contrast (on-disk) (top-1, %) Distribution Shift to Contrast (on-disk) -15-10-50510152025 Contrast (on-disk) Effective Robustness -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 ImageNet-A Effective Robustness Effective Robustness Scatterplot 606570758085 ImageNet (top-1, %) 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 Contrast (in-memory) (top-1, %) Distribution Shift to Contrast (in-memory) -15-10-5051015202530 Contrast (in-memory) Effective Robustness -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 ImageNet-A Effective Robustness Effective Robustness Scatterplot y = x Standard training Lp adversarially robust Other robustness intervention Trained with more data Linear fit Figure 22: Plots of the two synthetic distribution shifts with the highest correlation with ImageNet-A, compared similarly to Figure 5. 58 I Information on our main figures I.1 Constructing betaβ For each distribution shift, we construct the baseline accuracy functionβby analyzing the linear relationship between model performance on the original and shifted distributions. In particular, when constructingβwe only include "standard models," models that had not been designed with any robustness properties in mind or have not been trained on any data other than the standard 1,000-class ImageNet training set. Before constructing the predictor, model accuracies are then transformed according to the logit distribution; this transform assigns greater mass at the tails and experimentally provided the best linear fits.βis then simply the linear predictor of the shifted distribution based on the independent variable (the original distribution), computed in this scaled space. I.2 Ablations on our main figures Here we provide various versions of the main figures in the main text. In each plot, we use logit scaling to demonstrate that gains in performance at higher accuracies represent greater progress. The 95% confidence intervals were empirically computed from the bootstrapped samples. The bootstrapping was performed by computing 1,000 linear fits by sampling the models with replacement. 59 606570758085 ImageNet (top-1, %) 45 50 55 60 65 70 75 ImageNetV2 (top-1, %) Distribution Shift to ImageNetV2 55606570758085 ImageNet (class-subsampled) (top-1, %) 15 20 25 30 35 40 45 50 ObjectNet (top-1, %) Distribution Shift to ObjectNet 93949596979899 ImageNet (class-subsampled) (top-1, %) 50 55 60 65 70 75 80 ImageNet-Vid-Robust (pm-0, %) Distribution Shift to ImageNet-Vid-Anchors 9596979899 ImageNet (class-subsampled) (top-1, %) 45 50 55 60 65 YTBB-Robust (pm-0, %) Distribution Shift to YTBB-Anchors 50556065707580 ImageNet-Vid-Robust (pm-0, %) 30 35 40 45 50 55 60 65 70 ImageNet-Vid-Robust (pm-10, %) Distribution Shift to ImageNet-Vid-Robust 4550556065 YTBB-Robust (pm-0, %) 30 35 40 45 50 55 YTBB-Robust (pm-10, %) Distribution Shift to YTBB-Robust 8090959697 ImageNet (class-subsampled) (top-1, %) 5 10 20 30 40 ImageNet-A (top-1, %) Distribution Shift to Imagenet-A y = xStandard trainingLinear fit Figure 23: Only standard models are shown in these plots. Otherwise, they are identical to the main plots in the main text. This is done to better illustrate the quality of the linear fit. 60 606570758085 ImageNet (top-1, %) 45 50 55 60 65 70 75 80 ImageNetV2 (top-1, %) Distribution Shift to ImageNetV2 606570758085 ImageNet (top-1, %) 15 20 25 30 35 40 45 50 55 60 65 70 ObjectNet (top-1, %) Distribution Shift to ObjectNet 606570758085 ImageNet (top-1, %) 50 55 60 65 70 75 80 85 90 ImageNet-Vid-Robust (pm-0, %) Distribution Shift to ImageNet-Vid-Anchors 606570758085 ImageNet (top-1, %) 45 50 55 60 65 70 YTBB-Robust (pm-0, %) Distribution Shift to YTBB-Anchors 505560657075808590 ImageNet-Vid-Robust (pm-0, %) 30 35 40 45 50 55 60 65 70 75 80 ImageNet-Vid-Robust (pm-10, %) Distribution Shift to ImageNet-Vid-Robust 455055606570 YTBB-Robust (pm-0, %) 30 35 40 45 50 55 60 65 YTBB-Robust (pm-10, %) Distribution Shift to YTBB-Robust 607080 ImageNet (top-1, %) 5 10 20 30 40 50 60 70 80 ImageNet-A (top-1, %) Distribution Shift to Imagenet-A y = x Standard training Robustness intervention Trained with more data Linear fit Figure 24: The x-axes are not subsampled in these plots (they are performance on the full ImageNet validation set). Otherwise, they are identical to the main plots in the main text. This is done to clarify that subsampling the axes does not skew the discussed results. 61 606570758085 ImageNet (top-1, %) 45 50 55 60 65 70 75 80 85 ImageNetV2 (top-1, %) Distribution Shift to ImageNetV2 5560657075808590 ImageNet (class-subsampled) (top-1, %) 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 ObjectNet (top-1, %) Distribution Shift to ObjectNet 93949596979899 ImageNet (class-subsampled) (top-1, %) 50 55 60 65 70 75 80 85 90 95 ImageNet-Vid-Robust (pm-0, %) Distribution Shift to ImageNet-Vid-Anchors 9596979899 ImageNet (class-subsampled) (top-1, %) 45 50 55 60 65 70 75 80 85 90 95 YTBB-Robust (pm-0, %) Distribution Shift to YTBB-Anchors 505560657075808590 ImageNet-Vid-Robust (pm-0, %) 30 35 40 45 50 55 60 65 70 75 80 85 ImageNet-Vid-Robust (pm-10, %) Distribution Shift to ImageNet-Vid-Robust 455055606570 YTBB-Robust (pm-0, %) 30 35 40 45 50 55 60 65 70 YTBB-Robust (pm-10, %) Distribution Shift to YTBB-Robust 80909596979899 ImageNet (class-subsampled) (top-1, %) 5 10 20 30 40 50 60 70 80 90 ImageNet-A (top-1, %) Distribution Shift to Imagenet-A y = x Standard training Robustness intervention Trained with more data Linear fit Figure 25: The full y=x line is shown here in these plots. Otherwise, they are identical to the main plots in the main text. This is done to illustrate the performance gap due to distribution shift for each of the natural shifts. 62 J Example images of distribution shifts in our testbed J.1 Natural distribution shift images Figure 26: Dataset shifts. Examples from ImageNetV2 (first row), ObjectNet (second row), ImageNet- Vid-Robust (third row), and YTBB-Robust (fourth row). Figure 27: Consistency shifts. Sequences of video frames from ImageNet-Vid-Robust (top) and YTBB-Robust (bottom). 63 Figure 28: Adversarial shifts. Examples from ImageNet-A. J.2 Synthetic distribution shift images 64 (a) original(b) brightness(c) contrast(d) defocus blur(e) elastic transform (f) fog(g) frost(h) gaussian blur(i) gaussian noise(j) greyscale (k) impulse noise(l) jpeg compression(m) motion blur(n) pixelate(o) saturate (p) shot noise(q) snow(r) spatter(s) speckle noise(t) stylized imagenet (u) zoom blur(v)` p -attack Figure 29: Sample demonstration of the synthetic distribution shifts in our testbed. Note: This list is not complete. See Appendix E for a complete list. 65 K Additional discussion of related work K.1 ImageNet-R & ImageNet-Sketch Recently, Hendrycks et al.[40]studied robustness of classifiers to a new dataset that measure distribution shift, ImageNet-R, along with a new data augmentation method, DeepAugment. The authors make a number of comparisons in relation to an earlier version of this manuscript [90]. In order to provide more clarity, we integrate the ImageNet-R dataset and the DeepAugment models into our testbed in this paper. 80859095 ImageNet (class-subsampled) (top-1, %) 25 30 35 40 45 50 55 60 65 70 75 80 ImageNet-R (top-1, %) Distribution Shift to ImageNet-R 606570758085 ImageNet (top-1, %) 10 15 20 25 30 35 40 45 50 55 60 ImageNet-Sketch (top-1, %) Distribution Shift to ImageNet-Sketch y = x Standard training Lp adversarially robust Other robustness intervention Trained with more data Linear fit Figure 30: Model accuracies on two datasets: ImageNet-R (left), and ImageNet-Sketch (right). Both datasets create a distribution shift by selectively sampling images of renditions or sketches, respectively. Evaluations on these distribution shifts are similar to each other and follow the high-level trends of the other natural dataset distribution shifts in our testbed, with models trained on extra data providing the most robustness (though the effect is not uniform). On the left plot, DeepAugment models are highlighted in dark brown squares, and ImageNet classes were subsampled to match the class distribution of ImageNet-R; the ImageNet-Sketch class distribution already matches ImageNet. Confidence intervals, axis scaling, and the linear fit are computed similarly to Figure 2. ImageNet-R. In Figure 30, we plot model accuracies on ImageNet-R(endition) [40] and a similar dataset of sketches, ImageNet-Sketch [95]. We find that a few of the models trained on more data substantially outperform the rest. The top-right green cluster on both plots consists of several ResNet and ResNeXt models trained on 1 billion Instagram images [56,92,104] and EfficientNet-L2 (NoisyStudent) trained on the JFT-300M dataset of 300 million images [102]. However, as with the other dataset shifts, not all models trained on more data follow this trend. Several ResNet models trained on either the YFCC 100 Million images dataset [104] or the full ImageNet 21k-class dataset [49] have close to zero or negative effective robustness. When interpreting the results of models trained on more data, a caveat is that the extra training data may contain renditions that do not occur in ImageNet. To clarify this point, we have reached out to the authors of [56] to obtain more information about the Instagram dataset. We will update 66 -505101520 Corruptions Averaged Effective Robustness -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 ImageNet-R Effective Robustness Effective Robustness Scatterplot -15-10-5051015202530354045 Lp Attacks Effective Robustness -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 ImageNet-R Effective Robustness Effective Robustness Scatterplot Lp adversarially robustOther robustness interventionTrained with more data Figure 31: We compare the effective robustness of models with their accuracy drop due to corruptions (left) and adversarial attacks (right). The effective robustness is computed with respect to the linear fit on ImageNet-R. The measures are weakly correlated, indicating that improved robustness to corruptions or adversarial attacks does not in general improve effective robustness under ImageNet-R. our paper when sufficient data becomes available to estimate the relative frequency of renditions in the Instagram dataset. In the meantime, we note that the performance of the Instagram-trained models gives an answer to a question between the following two extremes: (i) How much performance on ImageNet-R do current models gain from a large, uncurated set of social media images that contains renditions? (i) How much robustness to ImageNet-R do current models gain from a large, uncurated set of social media images that contains little to no renditions? Interestingly, we also find that a number of` p -adversarially robust models provide substantial effective and relative robustness on both datasets. The top-left cluster of three yellow points on both plots are feature denoising models trained by Xie et al.[101]. These results suggest that adversarial robustness and denoising blocks can be viable approaches for distribution shift comprised of renditions or sketches. A natural follow-up question is whether synthetic robustness is correlated with ImageNet-R robustness. Similar to Appendix C, in Figure 31 we compare effective robustness on synthetic distribution shifts against effective robustness on ImageNet-R. The scatter plots are weakly correlated (the Pearson correlation coefficients arer= 0.35,0.30), indicating that improved robustness to corruptions or adversarial attacks in general does not improve effective robustness on ImageNet-R. However, there does appear to be a strong trend for the brown points in the left plot. Indeed, the correlation coefficient computed for only the “other robustness intervention” models isr= 0.76, suggesting that for this category of models, image corruptions robustness is well correlated with ImageNet-R robustness. DeepAugment.Hendrycks et al.[40]also introduce a new data augmentation method, Deep- Augment, which generates data augmentations by distorting the weights and activations of an image-to-image translation network. As seen in Figure 30, the DeepAugment+Augmix models, the top two dark brown squares on the left plot, have higher effective robustness on ImageNet-R than most other models (ρ= 11.2%for ResNeXt101 andρ= 10.2%for ResNet50). 67 As mentioned above, some of the models trained on the large Instagram and JFT-300M datasets outperform all other approaches on ImageNet-R including DeepAugment, but it is unclear how many images of renditions these datasets contain. Among the other models trained only on ImageNet, the comparison between` p -adversarial robustness and DeepAugment is nuanced. The` p -robust model of [101] has higher effective robustness but reduces standard ImageNet accuracy. The highest accuracy on ImageNet-R is also achieved by a model with an` p -based robustness intervention (an AdvProp model [100]), but the model is derived from EfficientNet [88] which achieves higher standard accuracy than the wide ResNeXt model [103] used in DeepAugment. An interesting question for future work is how and why` p -robustness helps on ImageNet-R, e.g., by training a ResNeXt model with AdvProp. On the anchor frames of ImageNet-Vid-Robust and YTBB-Robust, DeepAgument provides effective robustness comparable to models trained on multiple synthetic perturbations (e.g., a combination of Gaussian noise, contrast, motion blur, and JPEG compression). On ImageNetV2 and ObjectNet, DeepAugment does not provide effective robustness. For ImageNet-C corruptions [38], the combination of DeepAugment and AugMix offers substantial robustness. Excluding` p -adversarially robust models in the low accuracy regime, the two models with highest effective robustness to ImageNet-C corruptions are DeepAugment- and Augmix- trained ResNet50 (ρ= 14.2%) and ResNeXt101 (ρ= 13.2%). To put this in context, a ResNet50 trained directly on four of the ImageNet-C corruptions (Gaussian noise, contrast, motion blur, and JPEG compression) achieves an effective robustness ofρ= 13.3%. K.2 Video robustness In the context of video robustness, Gu et al.[35]have previously measured the performance of image classifiers on video sequences from the YouTube-BoundingBoxes(YT-B) dataset [67]. They find that video robustness is strongly correlated with accuracy on color corruptions such as brightness, hue, and saturation, with correlation coefficientsrnear0.95. There are two potential reasons our testbed finds these measures to be only weakly correlated with robustness on YTBB-Robust and ImageNet-Vid-Robust (rranging from0.1to0.5- full table in Appendix H): Standard accuracy as a confounder.The analysis in [35] does not account for standard accuracy as a confounder. The authors consider video robustness as accuracy within k frames of an anchor frame given that the anchor frame was correctly classified. While this definition does somewhat account for models with higher standard accuracies, it is natural to expect that models with higher standard accuracy are still more likely to predict the neighboring frames correctly given that the anchor frame was predicted correctly. Thus, standard accuracy will be correlated with video robustness. Moreover, our testbed reveals that standard accuracy is also correlated with corruption accuracy, and hence corruption accuracy will be correlated with video robustness as well. Additionally, it is worth noting that correlation does not mean that robustness to color corruptions causerobustness on videos. For instance, our testbed contains a model trained on saturation as data augmentation. While this model is highly robust to saturation (exhibiting only a 1% drop from standard accuracy to accuracy under saturations, compared to a baseline model exhibiting a 12% drop), it is no more video robust than a baseline without the saturation training (the saturation-trained model still experiences an 18% video robustness drop, compared to a baseline 68 model exhibiting a 19% drop). This example further shows the need for our measure of effective robustness as it explicitly corrects for the confounding effect of standard accuracy. Differences in data preparation.Gu et al.[35]split the full YT-B dataset into training, validation, and testing splits, and evaluate ImageNet models on sequences from the test split. In contrast, Shankar et al.[76]derive datasets from ImageNet-Vid and Youtube-B through a rigorous cleaning process: inspecting and annotating each sequence with human experts to check that subjects appear in frame throughout the sequence, match the correct class, and are not very blurry. This cleaning process indicates the derived datasets (ImageNet-Vid-Robust and YTBB-Robust) are better calibrated to measuring classification performance. 69