Paper deep dive

Towards Adversarial Evaluations for Inexact Machine Unlearning

Shashwat Goel, Ameya Prabhu, Amartya Sanyal, Ser-Nam Lim, Philip Torr, Ponnurangam Kumaraguru

Year: 2022Venue: arXiv preprintArea: Model EditingType: EmpiricalEmbeddings: 85

Models: ResNet-110, ResNet-20

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%

Last extracted: 3/12/2026, 7:48:49 PM

Summary

The paper introduces the Interclass Confusion (IC) test, a novel black-box adversarial evaluation method for inexact machine unlearning. It addresses the limitations of existing evaluation metrics by demonstrating that weight similarity and standard error metrics are insufficient to guarantee model indistinguishability. The authors propose two baseline unlearning methods, EU-k and CF-k, and show that adversarial evaluation strategies are essential for detecting both memorization and property generalization in machine learning models.

Entities (5)

Machine Unlearning · research-field · 99%Interclass Confusion (IC) test · evaluation-method · 98%CF-k · unlearning-algorithm · 95%EU-k · unlearning-algorithm · 95%Model Indistinguishability · concept · 92%

Relation Signals (3)

Interclass Confusion (IC) test → evaluates → Machine Unlearning

confidence 95% · we design a stronger black-box evaluation method called the Interclass Confusion (IC) test... to detect the insufficiency of unlearning procedures.

IC test → detects → Memorization

confidence 90% · As discussed in the following sections, we can use this test to detect both memorisation and property generalisation in an efficient way.

EU-k → performs → Machine Unlearning

confidence 90% · We also propose two analytically motivated baseline methods (EU-k and CF-k) which outperform several popular inexact unlearning methods.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Machine Learning models face increased concerns regarding the storage of personal user data and adverse impacts of corrupted data like backdoors or systematic bias. Machine Unlearning can address these by allowing post-hoc deletion of affected training data from a learned model. Achieving this task exactly is computationally expensive; consequently, recent works have proposed inexact unlearning algorithms to solve this approximately as well as evaluation methods to test the effectiveness of these algorithms. In this work, we first outline some necessary criteria for evaluation methods and show no existing evaluation satisfies them all. Then, we design a stronger black-box evaluation method called the Interclass Confusion (IC) test which adversarially manipulates data during training to detect the insufficiency of unlearning procedures. We also propose two analytically motivated baseline methods~(EU-k and CF-k) which outperform several popular inexact unlearning methods. Overall, we demonstrate how adversarial evaluation strategies can help in analyzing various unlearning phenomena which can guide the development of stronger unlearning algorithms.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

84,500 characters extracted from source content.

Expand or collapse full text

Towards Adversarial Evaluations for Inexact Machine Unlearning Shashwat Goel ∗1 , Ameya Prabhu ∗2 , Amartya Sanyal 3,4 , Ser-Nam Lim 5 , Philip Torr 2 , and Ponnurangam Kumaraguru 1 1 IIIT Hyderabad, 2 University of Oxford, 3 ETH Zurich, 4 MPI-IS, 5 Meta AI Abstract Machine Learning models face increased concerns regarding the storage of personal user data and adverse impacts of corrupted data like backdoors or systematic bias. Machine Unlearning can address these by allowing post-hoc deletion of affected training data from a learned model. Achieving this task exactly is computationally expensive; consequently, recent works have proposedinexact unlearningalgorithms to solve this approximately as well as evaluation methods to test the effectiveness of these algorithms. In this work, we first outline some necessary criteria for evaluation methods and show no existing evaluation satisfies them all. Then, we design a stronger black-box evaluation method called the Interclass Confusion (IC) test which adversarially manipulates data during training to detect the insufficiency of unlearning procedures. We also propose two analytically motivated baseline methods (EU-kand CF-k) which outperform several popular inexact unlearning methods. Overall, we demonstrate how adversarial evaluation strategies can help in analyzing various unlearning phenomena which can guide the development of stronger unlearning algorithms. 1 Introduction Deep learning is becoming increasingly prevalent in everyday applications, with models being trained on large amounts of sensitive personal information including health and financial records, social network history, personal emails, and messages. This has led to growing privacy concerns, as codified in recent privacy legislation like GDPR (Council of European Union, 2014), CCPA (CCP, 2018), and PIPEDA (PIP, 2018). The underlying motivation for privacy legislation is the concept ofdata autonomy, which states that every individual must retain complete control of their own data, including the right to withdraw their data from any system. However, deleting records corresponding to individuals is considerably harder for machine learning systems, especially those using deep networks, than with traditional databases. Recent studies, such as (Feldman & Zhang, 2020; Zhang et al., 2017), have shown that deep neural networks have a tendency tomemorizedata. This means that the network not only learns common patterns in the data, but also stores information about individual training data points. This is concerning from a privacy standpoint, as this information can be detected (Shokri et al., 2017) or even extracted from the model (Carlini et al., 2019). The main goal of “machine unlearning” (Federal Trade Commission, 2021; Cao & Yang, 2015; Ginart et al., 2019; Bourtoule et al., 2021; VIL, 2018) is to design both algorithms to delete data stored in the network and evaluation methods to recover or detect the deleted data from the trained model. Preserving privacy and removing memorization are not the only motivations to study machine unlearning. Several studies have shown that small amounts of corrupted data can induce harmful properties into the trained model, which can greatly affect its behaviour on unseen data Nakkiran & Bansal (2020), a phenomenon we refer to asproperty generalization. This can lead to problems in trustworthy machine learning, such as with noisy data Frenay & Verleysen (2014); Northcutt et al. (2021b,a), systematically biased data Prabhu & Birhane (2021) 1 , or adversarial data, such as poisoned samples Barreno et al. (2006); Chen et al. (2017); Yang et al. (2020). For example, Sanyal et al. (2020); Paleka & Sanyal (2022) show that a small amount of 1 to the extent that bias is a dataset problem Hooker (2021). * equal contribution 1 arXiv:2201.06640v3 [cs.LG] 22 Feb 2023 random noisy labels can significantly harm the adversarial robustness of a model. Further, Konstantinov & Lampert (2022) show that a small set of adversarially corrupted data can greatly increase unfairness. As these corrupted samples are discovered, unlearning them from the trained model can be used to remove the unwanted properties induced. Machine unlearning should remove both memorization and generalized properties of deleted samples. This is captured in the concept ofmodel indistinguishability, first defined in Golatkar et al. (2020a). LetMbe an ML model trained on datasetSusing learning algorithmTandS f ⊂Sbe the set of points that need to be deleted fromM. An unlearning process is considered successful if the distribution 2 of models produced by the unlearning process, is indistinguishable from the distribution of models produced by retraining the model using any training processT ′ on the remaining dataS f . To see why model indistinguishability implies unlearning, note that no training procedureT ′ which only usesS f can produce a model that carries information specific toS f . Hence, it is sufficient to show the model distribution produced by the unlearning algorithm is indistinguishable from the model distribution produced by any one training algorithmT ′ using S f . In this work, we study deleting a single query batch of samples. Extension to sequential deletion has to tackle challenges like correlated queries across time (Chourasia et al., 2023). A naive method for unlearning data from a machine learning model is to retrain the model on the retain dataS f . This method removes all information from the deletion set. Hence, in theory, it achieves “exact unlearning”, but is computationally and memory intensive. Our work focuses on “inexact unlearning”, in which the goal is to unlearn most information from the deleted data while minimizing computational cost. While exact unlearning is often infeasible, inexact unlearning presents a more tractable objective. In the specific case of deep networks, due to the absence of theoretical guarantees, empirical tests are commonly used for evaluating the degree of unlearning. A strong empirical test should reliably distinguish models unlearning to varying degrees in terms of memorization and property generalization of the deletion set. The latter is challenging with existing evaluations which remove independently identically distributed (I.I.D) samples as the undesirable properties they induce may be apriori unknown. As comparing a distribution of ML models is intractable, most past evaluations compare the weights Wu et al. (2020); Izzo et al. (2021) or outputs Golatkar et al. (2020a,b, 2021); Peste et al. (2021) of an unlearnt model and one retrained using the original training procedure onS f . However, as we argue in Theorem 1, even achieving nearly identical weights is insufficient to guarantee similarity in even well known properties like adversarial error and fairness, and thus model indistinguishability. This motivates the need for adversarial evaluations of unlearning. We propose performing manipulation in the training data which introduces a known measurable property throughS f that is absent inS f . Thus, models that exhibit this property cannot be indistinguishable from models obtained through retraining onS f . Ideally, the property unique toS f should produce a large predictable change in model behaviour to make its presence easy to measure. To this end, inspired by the application of removing systematically biased data, we propose the Interclass Confusion (IC) test. It induces the property of confusion between two classes through label manipulations. IC test requires the unlearning procedure to erase the induced confusion which we measure as the number of samples of the two classes “confused” as belonging to the other class. As discussed in the following sections, we can use this test to detect both memorisation and property generalisation in an efficient way. We find that our proposed IC test is far stronger than existing evaluations, allowing us to glean interesting insights into unlearning algorithms. Using the IC test, we can demonstrate the insufficiency of a class of unlearning methods that simply modifies the final linear layer Izzo et al. (2021); Baumhauer et al. (2020) in deep networks or methods that do not use the retained dataS f Chundawat et al. (2022). Our test detects the presence of imperfectly unlearnt information about the deletion setS f in the early layers of a deep network. Along with designing a stronger evaluation method (IC test), we also present two strong novel baselines — EU-k, which retrains the lastklayers from scratch and CF-k, where a model’s lastklayers are continually trained on the retain setS f . Finally, we also propose strategies to make the original modelM more amenable to unlearning, thereby aiding faster unlearning. Overall, we emphasise empirical evaluations of inexact-unlearning which measure how well an unlearning 2 due to stochasticity in both the unlearning algorithm andT 2 Table 1: Comparison of evaluation methods (sampling strategy+metric) in inexact unlearning. Only our IC test satisfies all three desiderata. Deletion Set Sampling StrategyMetric Necessary forComparable AcrossChecks Property IndistinguishabilityTraining ProceduresGeneralization I.I.D Golatkar et al. (2021), Class Golatkar et al. (2020a,b, 2021)Relearn timeX× I.I.D Wu et al. (2020); Izzo et al. (2021); Thudi et al. (2021a)L2 Weights×X I.I.D Peste et al. (2021)L1-ConfusionMatrix×X I.I.D Golatkar et al. (2021), Class Golatkar et al. (2020b, 2021)L1-Softmax×X Class Golatkar et al. (2021, 2020b); Baumhauer et al. (2020), I.I.D Golatkar et al. (2021); Liu et al. (2020) MIAXX× I.I.D He et al. (2021); Golatkar et al. (2021); Shibata et al. (2021), Class Golatkar et al. (2020a,b, 2021) Error×X Interclass Confusion (Ours)MIAXX× Interclass Confusion (Ours), I.I.D Confusion (Ours: Ablation)ErrorXXX procedure forgets additional properties induced by the deletion setS f . Our work alleviates certain shortcomings in existing evaluations as passing the IC test is necessary for achieving model indistinguishability. Further, it’s adversarial nature makes it a much stronger test to pass than prior evaluations as shown by our experiments. Our main contributions are: 1. In line with the motivations of machine unlearning, we decompose the evaluation of unlearning into memorization and property generalisation. The former is computed on the forget setS f whereas the latter is computed using unseen samples from the test set. 2.We highlight some necessary principles for useful evaluations of unlearning not achieved in existing work. We alleviate this by introducing a new black-box evaluation called the Interclass Confusion (IC) test. We empirically demonstrate that the IC test is far stronger than existing tests. 3.Further, we use the IC test to show several surprising phenomena which may guide the design of future unlearning methods (i) Unlearning just the last layer only removes a small fraction of information aboutS f (i) Unlearning methods may require the ability to learni.e.gain information (i) Standard regularisation during training can make models more amenable to unlearning. Using these insights, we propose two strong baselines, EU-kand CF-k, for comparing future unlearning methods. Roadmap: The rest of the paper is organized as follows: Section 2 describes our proposed evaluation methods in context of prior work, Section 3 describes our unlearning baselines and their properties, Section 4 presents our experimental results and in Section 5 we summarize our contributions while acknowledging the limitations of our work. 2 Towards Adversarial Evaluations We begin with an analysis of the shortcomings of prior evaluation strategies, and alleviate them by proposing the Interclass Confusion Test. 2.1 Recent Trends in Unlearning Evaluations In this section, we look at existing approaches for evaluating unlearning procedures. First, these methods choose one of the two types of deletion setsS f :nI.I.D samples (I.I.D removal) Golatkar et al. (2021); Wu et al. (2020); Izzo et al. (2021); Peste et al. (2021); He et al. (2021); Shibata et al. (2021) ornsamples belonging to a particular class (Class Removal) Golatkar et al. (2020a); Baumhauer et al. (2020). Once the unlearning procedure is applied on the aboveS f , the following are some popular metrics used to measure forgetting: Relearn Time: Golatkar et al. (2020a,b, 2021); Chundawat et al. (2022) measure the number of training epochs until the loss of an unlearnt model drops below a pre-chosen threshold when retrained on samples in S f . A higher re-learn time implies better forgetting. Weight Similarity:Wu et al. (2020); Izzo et al. (2021) measures theL 2 distance between the weights of the unlearnt model and another model retrained onS f using the original training procedure. Naturally, a smaller distance is used to imply better unlearning. 3 Targeted Error Figure 1: IC Test Pipeline: We mislabel a subset of samples from two classes of the original dataset, forming S f . Here, shape and colour represent the actual and labelled class respectively. Then,MandM r are obtained by training from scratch onSandS f respectively. The unlearning procedure can leverage (some of)M, S f andS f to produce the unlearnt modelM u . Output similarity: Similar to distance between weights, distance in the softmax outputs on a pre-defined set of data points are also used by different evaluation methods. Golatkar et al. (2020a,b, 2021) measures theL 1 distances between softmax outputs of a unlearnt model and a retrained model on the pre-defined set, Peste et al. (2021) measures theL 1 distance between the confusion matrices of the respective models, He et al. (2021); Golatkar et al. (2021); Shibata et al. (2021) measure the gap in error on the distribution of affected samples. Membership Inference Attacks (MIA): Tests based on Membership Inference Attacks (Shokri et al., 2017; Song & Mittal, 2021) are designed to reliably distinguish data points in the training set from similar unseen data. Hence, they can also be used to reliably distinguish the deleted samples from similar unseen samples (see Hu et al. (2021) for a survey). A detailed description of our MIA attack compared to past unlearning literature is included in Appendix Section B.3. 2.2 Shortcomings of Existing Evaluations We start by listing three desiderata absent in most existing evaluation methods, as summarized in Table 1. Theorem 1 then motivates the need for adversarial evaluations. Necessary for Indistinguishability: Suppose there existsT ′ 6=Tsuch that retraining withT ′ onS f would produce a model highly similar to the unlearnt modelM u , thenM u is a correct solution as it satisfies model indistinguishability. Showing no suchT ′ exists is difficult, and thus past evaluations simply compare M u with a modelM T r retrained using the original training procedure. However, this can exclude a large set of correct solutions which have no information fromS f but behave differently fromM T r . Consider a randomly initialized network. It clearly has no information fromS f , and indeed satisfies model indistinguishability if we considerT ′ to be the random initialization process. However, it will be arbitrarily far fromM T r and will be unnecessarily penalized by evaluations based on Weights and Output similarity. Thus, passing past evaluations based on high similarity with a single model is not necessary for achieving unlearning. On the other hand, our proposed evaluation manipulates a subset of training data to introduce a measurable property throughS f that is absent inS f . Any model that exhibits this property cannot be from a model distribution produced by (re)training withoutS f for allT ′ , and has not unlearnt. Thus, passing our evaluation is necessary to claim an unlearning procedure can handle arbitrary deletions. Comparable Across Training Procedures: Unlearning procedures often significantly modify the training procedure or architecture Bourtoule et al. (2021); He et al. (2021); Golatkar et al. (2020a,b, 2021); Graves et al. (2021). Thus, a versatile unlearning evaluation should provide measurements of retained information that are comparable across chanes in architectures and training procedures. For example, measuring relearn time as an evaluation method requires setting a threshold. However, different unlearning 4 procedures (Golatkar et al., 2020a,b, 2021; Chundawat et al., 2022) may differ in learning rate or have inherently different behavior in how low the loss can get and how fast it decreases. Similarly,L 2 distance between weights cannot be compared across architectures or hyperparameter choices like the amount of weight decay. Checks Property Generalization : Unlearning procedures must ensure that properties which are only present inS f do not influence performance on unseen samples. Some evaluations, such as membership inference attacks (MIA) Shokri et al. (2017); Chen et al. (2021) effectively only determine the removal of memorization, and not the removal of generalized properties. In any evaluation with I.I.D removal, while it is theoretically possible to check if generalized properties are removed, it is not clear what properties to look for. Two relevant properties that can be exacerbated by corrupted data are adversarial error (R adv ) (Madry et al., 2018) and unfairness 3 (Γ). An evaluation method that can be satisfied without removing these properties is clearly insufficient to guarantee unlearning. Theorem 1 shows that metrics likeL 2 distance in the parameter space and gap in test error on a random I.I.D sample (R) are poor indicators of whether two models have similarR Adv and Γ. Theorem 1.There exists a distributionDsuch that for any,α≥0, there exist two`-layered fully connected linear NNs parameterised byW 1 ,W 2 which are simultaneously: •Close in Weights:‖W 1 −W 2 ‖ F ≤ •Close in Test Error:R(f W 1 )≤R(f W 2 ) +α •Far in Robustness:R adv (f W 1 )≥R adv (f W 2 ) + 1−2α •Far in Fairness:Γ (f W 1 ) = Γ (f W 2 )−1 whereR,R Adv,Γ are as defined above andf W is an`-layered fully connected linear neural network parameterised byW. Proof is available in Appendix B.1. The theorem shows that two models that are arbitrarily close in weights and test error can be arbitrarily far in adversarial robustness and fairness. In particular, we show an example where models get farther in adversarial robustness as they get closer in test error. Thus, unlearning evaluations must either measure indistinguishability in terms of moreadversarialquantities like robustness and fairness or use strategic non-I.I.D deletion sets. In our work, we explore the latter, i.e. an adversarial approach to designing deletion sets. In the following section, we introduce this evaluation procedure known as IC test. 2.3 Proposal: Interclass Confusion Test In contrast to existing evaluations, we inject a strong differentiating influence specific toS f into the training dataset via label manipulations. Specifically, we present: Interclass Confusion (IC): As illustrated in Figure 1, the IC test using a deletion set ofnsamples follows these steps: 1. Take n 2 samples each from two classes in the train data to formS ′ ⊂S(Targeted sampling). 2.Swap labels 4 between the two classes of samples inS ′ (Adversarial manipulation) to get the confused setS f . The dataset for training the original modelMis (S ′ )∪S f . 3. Select the setS f as data to be deleted from the trained modelM(Strategic deletion set). 4. Evaluate memorization and property generalization by measuring error on training and testing setsS ′ andS ′ u corresponding to the two classes. To isolate the effect of targeted sampling in the IC testi.e.confusing two specific classes, we introduce: Ablation: I.I.D Confusion : We selectnsamples uniformly at random fromSto formS ′ and mislabel them to a uniform random different class, using these mislabelled samples asS f . Note that the removal is not I.I.D, we replace targeted label manipulation with I.I.D label noise. 3 We use accuracy discrepancy (Buolamwini & Gebru, 2018; Sanyal et al., 2022) for mathematical simplicity. Can be shown for demographic parity and equalised odds (Hardt et al., 2016). 4 Note that evaluations based on label swapping have been used in traditional adversarial robustness literature Nakkiran (2019); Fowl et al. (2021), but with quite different goals, setting and design. 5 We compute the MIA and Error on affected classes like previous work, but also introduce the Targeted Error metric: Error v/s Targeted Error: Error computed for a given setSis the fraction of samples inSwhich were misclassified regardless of which class it was mistaken as. In Interclass Confusion, we are interested specifically in the fraction of samples confused between the two confused classes. In Class Removal, we are interested in the fraction of samples classified as the class to be removed. This is measured by Targeted Error, which is the fraction of samples inSmisclassified to the targeted class exhibiting the unwanted property is not removed. Samples misclassified into any other class are not counted as illustrated in Figure 1 for IC test. As an illustrative example, for IC test on a 10 class dataset: the error of a random model would be 90%, but the targeted error would be 10%. Error/Targeted Error when computed on the setS ′ measures memorization, and when computed on the unseen (test) set samplesS t from the same distribution as setS ′ measures property generalization. 3 Unlearning Baselines Having discussed properties of evaluation methods, we now discuss unlearning procedures – desirable properties and our two simple baselines that achieve them. 3.1 Desiderata for Unlearning Methods Unlearning procedures need the ability to learn : Consider a linearly-separable binary classification task where we use the IC test to introduce complete confusion between the two classes (50% of samples of each class mislabelled as the other). Powerful empirical risk minimizers (like neural networks trained with SGD) will achieve a train accuracy onSclose to 100% (Zhang et al., 2017). However the test accuracy will be much lower, closer to 50%, as the training dataset is essentially fully randomly labelled. However, upon deleting all the mislabelled samples, like in the IC test, we are left with 50% of the original dataset but with correct labels. A model retrained from scratch onS f can be expected to achieve reasonably good accuracy, much larger than 50%, which a good unlearning procedure is expected to match. So we can expect the unlearnt model to have learnt to perform the task, whereas the original model cannot. Intuitively, this implies thatsolely erasing information from the model is not enough, and the ability to learn may be necessary for ideal unlearning procedures. Consequently, we expect methods which do not use information about the retain set Chundawat et al. (2022) will have limitations when handling arbitrary deletions and will not perform well on the IC test. Scalability to large deletion sets: Popular unlearning methods, both exact and inexact, either explicitly assume tiny deletion sets Thudi et al. (2021a); Bourtoule et al. (2021); Wu et al. (2020) or scale poorly beyond them in practice Schelter (2020); Graves et al. (2021); Golatkar et al. (2020a,b). In Appendix Section B.2 we show that the computational complexity of methods based on the paradigm of isolating the influence of data to small parts of the training procedure Bourtoule et al. (2021); He et al. (2021); Yan et al. (2022); Graves et al. (2021) scales exponentially with the size of the deletion set. Arguably, methods which require resources similar to retraining from scratch for large deletion sets have limited practical value, especially in applications which require large deletion sets (see Appendix B.2 for a discussion). Targeting Areas to Unlearn: A way to significantly improve efficiency of unlearning procedures is to focus optimization power towards areas of a model where the deletion set is stored. We look from a layerwise perspective- the early layers of a deep network capture generic low-level representation Yosinski et al. (2015); Kataoka et al. (2020), while the later layers focus on dataset-specific information. Interestingly, the earlier layers are also the most computationally intensive (Brock et al., 2017). Hence, focusing unlearning on the last klayers may allow computationally efficient erasure of information fromS f . Such unlearning methods also help us analyze how early in a deep network can an evaluation method detect the presence of information specific toS f . Overall, unlearning methods should: (i) have capacity to learn information in addition to unlearning and (i) scale to large deletion sets and further, for our analysis, we wish to have unlearning methods that (i) target specific parts of the model, e.g. the lastklayers for unlearning. 6 3.2 Proposal for Unlearning baselines: CF-kand EU-k We propose two methods which we believe will be useful ‘baselines’ for future work to compare against. (i) They achieve a tradeoff between forgetting and efficiency which can be controlled using parameterk, allowing comparisons with unlearning procedures of differing degrees of efficacy. (i) They are simple and require minimal assumptions: they scale to large deletion sets, are applicable for all DNN training procedures and require only access toS f . Exact-unlearning the lastklayers (EU-k): We retrain the lastklayers ofMfrom scratch using the same training procedureTon retain setS f while freezing prior layers. Catastrophically forgetting the lastklayers (CF-k): Neural Networks suffer from catastrophic- forgetting French (1999) - when a model is continually updated without some previously learnt samples, the model loses knowledge about them. We finetune the lastklayers ofMon the retain setS f using the same training procedureTwhile freezing prior layers, hoping to catastrophically forgetS f . As we avoid re-initializing the lastklayers unlike EU-k, we need far fewer epochs, making CF-kmore efficient than EU-k. 4 Experiments We show empirical support for three claims of our work. (i) We show that our EU-kand CF-kunlearn better than four popular methods and are strong baselines. (i) Using EU-kand CF-kfor analysis, we show our primary contribution, the IC test, is more reliable than previous evaluations in detecting unwanted memorization and property generalization. (i) We show standard regularization techniques can make original modelsMmore amenable to unlearning. Our training procedure is described in Appendix Section C.1. All code has been made publically available at https://github.com/shash42/Evaluating-Inexact-Unlearning. 4.1 Evaluating Baselines: EU-kand CF-k Setup. Due to the lack of established evaluation methods and comparisons to other methods in past work, the ’state of the art’ in unlearning is not clear. We compare our proposed baselines against four popular past unlearning methods: Fisher Golatkar et al. (2020a), NTK-Fisher Golatkar et al. (2020b), Amnesiac Unlearning Graves et al. (2021), and LCODEC Mehta et al. (2022) which have published their codebase for accurate reproduction. We could not run Fisher, NTK-Fisher for our larger datasets like CIFAR10 due to large memory requirements and thus compare all models on their setting: We use Small-CIFAR-5 Golatkar et al. (2020a) (a 5 class subset of CIFAR10), and all samples of a given class as theS f (Class Removal). We follow their training procedure to get their original and retrain models, as we obtained near-random performance when we applied their unlearning method on our standard training procedure perhaps due to violation of some of their training assumptions. For Amnesiac, LCODEC and our unlearning procedures we report results on unlearning from an original modelMproduced by our default procedure (T) with a standard ResNet-20 architecture. We obtain the same observations on using their respective training procedures which produce original models with lower accuracy. For forgetting we report memorization and property generalization by computing targeted error on the deletion set (S f ) and test set of the deleted class (S t ) respectively. We measure accuracy with test set error and efficiency with unlearning time. Results.We present all our results in Table 2. Accuracy:The test error of the retrained model is 15% higher than the original (bothT) because a portion of the test samples belong to the deleted class. Here, lower test errors are attributable to not forgetting the deleted class. Comparing test error for the Original models, we observe our procedureThas a large decrease (10%) in test error compared to Golatkar et al. (2020b). This ensures we study unlearning on better, more realistic models. We find that Amnesiac and LCODEC produce unlearnt models with almost random performance. Amnesiac relies on deletion set samples belonging to only a few batches. However, this assumption does not scale to large deletion sets and we find all batches are affected in our experiment, as expected from the mathematical analysis we present in Appendix Section B.2. LCODEC removes samples sequentially, and the error of the model increases fast as more samples are deleted. Forgetting:We measure the degree of unlearning of a given method by comparing the reduction in targeted error of the method with the corresponding original and retrained models providing the starting and ideal 7 Table 2: Comparison between unlearning procedures on Class removal test on Small-CIFAR-5. Forgetting measured by targeted error: Memorization (Mem) and Property Generalization (PropGen). Performance and efficiency measured by test error and unlearning time. (↓) indicates lower is better. ModelTargeted Error (↓)Test Err (↓)Time(s) (↓) MemPropGen Tfrom Golatkar et al. (2020a,b) Original92.397.626.70.00 Fisher Golatkar et al. (2020a)94.698.033.2141.95 NTK-Fisher Golatkar et al. (2020b)27.039.631.0141.90 Retrain0.00.041.49.81 T(Ours) Original98.097.316.30.00 Amnesiac Graves et al. (2021)22.321.674.31.72 LCODEC Mehta et al. (2022)20.720.280.3226.9 1-layer (Ours) CF18.312.330.94.43 EU 9.64.331.99.38 10-layers (Ours) CF15.69.329.45.22 EU2.00.032.610.78 Retrain0.00.032.512.33 scores respectively. We observe that simply unlearning the last layer with our baselines (EU-1 & CF-1) have far better reductions in targeted error compared to previous methods in both memorization and property generalization. Surprisingly, Golatkar et al. (2020a) fails to achieve any significant forgetting. Efficiency:We observe that 3 out of 4 past procedures take far more time for unlearning compared to our baselines and even retraining as approximating the Fisher Information Matrix is expensive. In real-world scenarios, such speedups are highly important to enable practical applications of unlearning. Amnesiac is fast but produces a random model. Conclusion. Our methodsEU-k andCF-k outperform popular unlearning methods by significant margins in all three dimensions: forgetting, accuracy and efficiency indicating they are reasonable baselines for analysis. 4.2 Comparing Tests for Evaluating Forgetting Setup. We use CIFAR10 and CIFAR100 datasets with a 40K-10K-10K train-val-test split. Note that we use the same deletion set sizenfor a fair comparison across all tests, with the sample set removed for every test, with details in Appendix Section C.3. Experiments in this section usencorresponding to the number of training samples in one class: 4000 for CIFAR10 and 400 for CIFAR100 Krizhevsky et al. (2009). We further report results across different deletion set sizesnin the Appendix Section D.3, D.4 and find them to be consistent. All results are averaged over three runs with different seeds for robustness. Results.In Figure 2 we compare different unlearning evaluation methods on their ability to demonstrate the degree of forgetting of models produced by our baselines EU-kand CF-k. Every line is formed by varying the number of layers unlearntkand hence the degree of forgetting, with 0 and 110 (leftmost and the rightmost points) indicating the original and retrain models respectively. A strong test is indicated by: (i) the score of intermediate models (0< k <110) is different from that of the retrain model as some information is still retained after unlearningklayers. (i) There is a clear gradual improvement in the forgetting metric ask increases. We present results consistent across graphs below: Memorization:Class removal test (blue) is not able to detect memorized information even in simply exact-unlearning the last layer on any metric or dataset (solid-blue line reaches retrain scores immediately). I.I.D removal barely distinguishes 1-layer and 10-layer unlearning. We get random (<50%) MIA scores for I.I.D confusion and hence exclude it. However, I.I.D confusion performs as well as the IC test on the error 8 Exact Unlearning-k MemorizationProperty Generalization CIFAR - 100 CIFAR - 10 Inter-Class Confusion (Ours) Catastrophic Forgetting-k I.I.D ClassI.I.D Confusion Figure 2: Error, MIA for various deletion strategies (Y) reported across the number of layers (X) affected by the unlearning procedure. The left-most points at 0 layers represent the original modelM, whereas the right-most points at 110 layers represent the retrained modelM T r . Only Interclass Confusion reliably distinguishes different degrees of unlearning (no. of layers unlearnt) across all graphs. metric. IC test is the most useful across metrics and datasets, clearly distinguishing models with information removed from more layers. Property Generalization:Only IC test is capable of detecting property generalization of confusion even after exact unlearning just the last layer. Even I.I.D confusion, which represents adding noisy labels with no systematic bias, is clearly insufficient to induce detectable generalized properties. Thus, both components of the IC test, class-targeted removal and confusion, are needed together to show clear trends in property generalization evaluations. Hyperparameters of IC Test:The two hyperparameters in executing the IC test are choosing the two classes to confuse and the number of confused and deleted samplesn. We find that while trends are similar across class pairs, unlearning is the hardest when we choose classes that are highly similar. We thus report results for (Cat, Dog) in CIFAR 10 and (Maple Tree, Oak Tree) in CIFAR 100 here. Further details can be found in Appendix Section D.5. Regarding the size of the deletion set, we demonstrate that the IC test can reliably detect imperfect memorization and property generalization with just 1% and 5% of the dataset being corrupted respectively. We report results for 5% here, and show IC test is the most useful among all tests across deletion set sizes in Appendix Section D.4, D.3. EU-kv/s CF-k:For any givenk, catastrophic-forgetting (CF) removes most of the information in those layers while being twice as fast as exact-unlearning (EU) indicated by the dotted lines closely following their solid counterparts across tests. The only cases where they differ is when the test is simply unable to detect information retained in exact unlearning (e.g. Class Removal). As shown by the IC test, EU-1 leaks a lot of information gained fromS f , showing that prior exact unlearning methods that only modify the final layer of deep networks Baumhauer et al. (2020); Izzo et al. (2021) continue to store information fromD f and cannot handle arbitrary deletions 5 . Note that EU-kand CF-kcontinue to maintain the same accuracy acrosskas shown in Appendix Table 5. While we choose a 110 layer ResNet andk=1,10,50as a concise representative sample here, all observations hold across more values ofkand network depths if compared using the fraction of layers unlearnt as shown in Appendix Section D.2. Conclusion. IC test is the only test that shows a clear difference between models with different number 5 Only unlearning the final layer may succeed if earlier layers are trained privately Guo et al. (2020); Wu et al. (2020). 9 MemorizationProperty Generalization CIFAR - 100 CIFAR - 10 Original +Early Stop +Cutmix +Cutmix& Early Stop EU-k EU-k EU-k EU-k Figure 3: Interclass Confusion Targeted Error (Y) on unlearning from original models with different regular- ization (bar colors) reported for the original modelM, EU-10, EU-50, and retrained modelM t r . The same unlearning procedure can remove more confusion when starting from better regularized original models. of layers unlearnt for both memorization and property generalizaton, on all metrics and datasets. IC test also shows past unlearning methods that propose to modify only the final layer of deep networks continue to retain most information about the deletion set. Varyingkin EU-kand CF-kcan be used to control the forgetting-efficiency tradeoff at the same overall accuracy. Catastrophic forgetting achieved similar degree of forgetting as exact unlearning, while being twice as efficient. 4.3 Making Models Amenable to Unlearning Aim.Different original modelsMcan have varying propensities to memorizeD f . We aim to leverage this to provide training strategies that obtain original modelsMwith better unlearning properties, particularly computational efficiency. This is in line with recent work Thudi et al. (2021b) which theoretically motivates this for|S f |= 1, but we empirically show it holds even for large deletion sets. Strategies.Early stopping has been a universal strategy to prevent overfitting (i.e.memorization) in machine learning. We also use Cutmix Yun et al. (2019), with the intuition that the model never sees a training sample in isolation while training, inspired from Huang et al. (2020). Apart from using these regularization strategies during training, we use the same setup as before. Results.We present results in Figure 3. Comparing original models (leftmost group of bars on the graphs), we observe that both techniques obtain large reductions in memorization ofS f but similar property generalization. We observe only a marginal dropoff in unlearning (especially property generalization) from Cutmix+Early Stopping 10 layers to Original 50 layers. Cutmix+Early Stop 10 layers gives a huge improvement in unlearning performance compared to the Original 10 layers unlearnt models, especially on the harder CIFAR-100 dataset. In property generalization, this occurs despite original models having similar amounts of confusion indicating better regularized models make it easier for inexact unlearning methods to remove 10 information. Conclusion. The presented results validate the idea that some original models make it easier to remove information using the same unlearning procedure. We demonstrated how this can be leveraged to achieve forgetting using cheaper unlearning procedures. Comparisons across unlearning procedures should ideally use the same original model for fairness, at least when there are no training assumptions. 5 Conclusion and Limitations While prior unlearning methods claim to handle arbitrary deletion sets, we prove that passing prior evaluations based on weight and output similarity fail to guarantee unlearning of non-IID deletion sets. This motivates the need for adversarial evaluations like our proposed Interclass Confusion test. In contrast to prior evaluations, the IC test is necessary to pass to achieve model indistinguishability and is not sensitive to different training procedures. Even with a small fraction of data being manipulated, the IC test can reliably capture how well unlearning procedures remove memorization of deletion set samples and properties generalized from them – both are important for different applications. We propose EU-kand CF-kas strong unlearning baselines that scale to large deletion sets, enable analysis of how early in the network information is retained and allow trading forgetting for efficiency at constant accuracy. We use our evaluation and methods to glean a variety of insights. (i) Unlearning methods that only modify the final layer in a deep network are not sufficient. (i) We explore the interplay between learning and unlearning – theoretically, we conjecture that an unlearning procedure aiming to handle arbitrary deletions requires the ability to learn. Empirically, we show that better regularized models are more amenable to unlearning. We hope that our analysis and proposed IC test along with EU-k and CF-k baselines will enable building stronger adversarial tests and better unlearning procedures. There is a need to bridge the current limitations of our work: We do not expect EU-kand CF-kto be gold-standard unlearning procedures, they are meant only as simple analytical tools that assist future research. As for the IC test, defining a passing score for real-world datasets that is necessary and sufficient is an open problem. While any procedure claiming to handle arbitrary deletions must pass the IC test, it alone cannot guarantee perfect unlearning. Finding a test that if passed is sufficient to prove unlearning of arbitrary deletions is an interesting direction. Finally, we hope our work spurs the use of unlearning in applications like removing systematic bias, noise, and tackling adversarial manipulations. Acknowledgements:We thank Shyamgopal Karthik, Saujas Vaduguru, Arjun T.H., Naren Akash RJ, Shradha Sehgal, Nikhil Chandak, Shashwat Singh, Shashwat Chandra among others for helpful feedback. Ameya Prabhu was funded by Facebook Grant Number DFR05540. This work is supported by the UKRI grant: Turing AI Fellowship EP/W002981/1 and EPSRC/MURI grant: EP/N019474/1. References Alternative k-nearest neighbour rules in supervised pattern recognition: Part 1. k-nearest neighbour classifica- tion by using alternative voting rules.Analytica Chimica Acta, 1982. 19 California consumer privacy act, 2018. https://leginfo.legislature.ca.gov/faces/billTextClient.xhtml?billid=201720180AB375. 1 Personal information protection and electronic documents act, 2018. https://w.priv.gc.ca/en/opc-news/news-and-announcements/2018/an181010/. 1 Humans forget, machines remember: Artificial intelligence and the right to be forgotten.Computer Law & Security Review, 2018. 1 Acquisti, A., Friedman, A., and Telang, R. Is there a cost to privacy breaches? an event study. 2006. 19 Aldaghri, N., Mahdavifar, H., and Beirami, A. Coded machine unlearning. InIEEE Access, 2021. doi: 10.1109/access.2021.3090019. 19 11 Banko, M. and Brill, E. Scaling to very very large corpora for natural language disambiguation. ACL, 2001. 19 Barreno, M., Nelson, B., Sears, R., Joseph, A. D., and Tygar, J. D. Can machine learning be secure? In ASIA CCS, 2006. 1 Baumhauer, T., Sch ̈ottle, P., and Zeppelzauer, M. Machine unlearning: Linear filtration for logit-based classifiers.arXiv preprint arXiv:2002.02730, 2020. 2, 3, 9, 19, 20, 21 Bourtoule, L., Chandrasekaran, V., Choquette-Choo, C. A., Jia, H., Travers, A., Zhang, B., Lie, D., and Papernot, N. Machine unlearning. InIEEE S&P, 2021. 1, 4, 6, 19 Brock, A., Lim, T., Ritchie, J. M., and Weston, N. Freezeout: Accelerate training by progressively freezing layers, 2017. 6 Brophy, J. and Lowd, D. Machine unlearning for random forests. InICML, 2021. 19 Buolamwini, J. and Gebru, T. Gender shades: Intersectional accuracy disparities in commercial gender classification. InConference on fairness, accountability and transparency, p. 77–91. PMLR, 2018. 5 Cao, Y. and Yang, J. Towards making systems forget with machine unlearning. InIEEE S&P, 2015. 1 Carlini, N., Liu, C., Erlingsson, ́ U., Kos, J., and Song, D. The secret sharer: Evaluating and testing unintended memorization in neural networks. InUSENIX Security, 2019. 1 Cauwenberghs, G. and Poggio, T. Incremental and decremental support vector machine learning. InNeurIPS, 2000. 19 Chen, M., Zhang, Z., Wang, T., Backes, M., Humbert, M., and Zhang, Y. When machine unlearning jeopardizes privacy. InCCS, 2021. 5 Chen, X., Liu, C., Li, B., Lu, K., and Song, D. Targeted backdoor attacks on deep learning systems using data poisoning.arXiv preprint arXiv:1712.05526, 2017. 1 Chourasia, R., Shah, N., and Shokri, R. Forget unlearning: Towards true data-deletion in machine learning. ICLR, 2023. 2 Chundawat, V. S., Tarun, A. K., Mandal, M., and Kankanhalli, M. Zero-shot machine unlearning.arXiv preprint arXiv:2201.05629, 2022. 2, 3, 5, 6 Council of European Union. Council regulation (EU) no 269/2014, 2014. http://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1416170084502&uri=CELEX:32014R0269. 1 Fabbrizzi, S., Papadopoulos, S., Ntoutsi, E., and Kompatsiaris, I. A survey on bias in visual datasets.arXiv preprint arXiv:2107.07919, 2021. 18 Federal Trade Commission. California company settles ftc allegations it deceived consumers about use of facial recognition in photo storage app, January 2021. 1 Feldman, V. and Zhang, C. What neural networks memorize and why: Discovering the long tail via influence estimation.NeurIPS, 2020. 1 Fowl, L., Goldblum, M., Chiang, P.-y., Geiping, J., Czaja, W., and Goldstein, T. Adversarial examples make strong poisons. InNeurIPS, 2021. 5 Fredrikson, M., Jha, S., and Ristenpart, T. Model inversion attacks that exploit confidence information and basic countermeasures. InCCS, 2015. 21 Frenay, B. and Verleysen, M. Classification in the presence of label noise: A survey. InIEEE NNLS, 2014. 1 12 French, R. M. Catastrophic forgetting in connectionist networks. InTrends in Cognitive Sciences, 1999. 7 Ginart, A., Guan, M. Y., Valiant, G., and Zou, J. Making AI forget you: Data deletion in machine learning. InNeurIPS, 2019. 1 Golatkar, A., Achille, A., and Soatto, S. Eternal sunshine of the spotless net: Selective forgetting in deep networks. InCVPR, 2020a. 2, 3, 4, 5, 6, 7, 8, 21 Golatkar, A., Achille, A., and Soatto, S. Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. InECCV, 2020b. 2, 3, 4, 5, 6, 7, 8, 16, 19, 20, 21 Golatkar, A., Achille, A., Ravichandran, A., Polito, M., and Soatto, S. Mixed-privacy forgetting in deep networks. InCVPR, 2021. 2, 3, 4, 5, 19, 20, 21 Graves, L., Nagisetty, V., and Ganesh, V. Amnesiac machine learning. InAAAI, 2021. 4, 6, 7, 8, 19, 20, 21 Guo, C., Goldstein, T., Hannun, A., and van der Maaten, L. Certified data removal from machine learning models. InICML, 2020. 9 Hardt, M., Price, E., and Srebro, N. Equality of opportunity in supervised learning.NeurIPS, 29, 2016. 5 He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. InCVPR, 2016. 21 He, Y., Meng, G., Chen, K., He, J., and Hu, X. Deepobliviate: A powerful charm for erasing data residual memory in deep neural networks.arXiv preprint arXiv:2105.06209, 2021. 3, 4, 6, 19 Hooker, S. Moving beyond “algorithmic bias is a data problem”.Patterns, 2021. 1 Hu, H., Salcic, Z., Dobbie, G., and Zhang, X. Membership inference attacks on machine learning: A survey. arXiv preprint arXiv:2103.07853, 2021. 4 Huang, Y., Song, Z., Li, K., and Arora, S. Instahide: Instance-hiding schemes for private distributed learning. InICML, 2020. 10 Izzo, Z., Smart, M. A., Chaudhuri, K., and Zou, J. Approximate data deletion from machine learning models. InAISTATS, 2021. 2, 3, 9 Jagielski, M., Oprea, A., Biggio, B., Liu, C., Nita-Rotaru, C., and Li, B. Manipulating machine learning: Poisoning attacks and countermeasures for regression learning. InIEEE S&P, 2018. 18 Kataoka, H., Okayasu, K., Matsumoto, A., Yamagata, E., Yamada, R., Inoue, N., Nakamura, A., and Satoh, Y. Pre-training without natural images. InACCV, 2020. 6 Konstantinov, N. H. and Lampert, C. Fairness-aware pac learning from corrupted data.JMLR, 2022. 2 Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009. 8 Li, Y., Wu, B., Jiang, Y., Li, Z., and Xia, S.-T. Backdoor learning: A survey.arXiv preprint arXiv:2007.08745, 2020. 18 Liu, Y., Ma, Z., Liu, X., Liu, J., Jiang, Z., Ma, J., Yu, P., and Ren, K. Learn to forget: Machine unlearning via neuron masking.arXiv preprint arXiv:2003.10933, 2020. 3, 19, 20 Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. InICLR, 2018. URL https://openreview.net/forum?id=rJzIBfZAb. 5 Mehta, R., Pal, S., Singh, V., and Ravi, S. N. Deep unlearning via randomized conditionally independent hessians. InCVPR, p. 10422–10431, June 2022. 7, 8 13 Nakkiran, P. A discussion of ’adversarial examples are not bugs, they are features’: Adversarial examples are just bugs, too.Distill, 2019. 5 Nakkiran, P. and Bansal, Y. Distributional generalization: A new kind of generalization, 2020. 1 Northcutt, C. G., Athalye, A., and Mueller, J. Pervasive label errors in test sets destabilize machine learning benchmarks, 2021a. 1, 18 Northcutt, C. G., Jiang, L., and Chuang, I. L. Confident learning: Estimating uncertainty in dataset labels. InJAIR, 2021b. 1, 18 Paleka, D. and Sanyal, A. A law of adversarial risk, interpolation, and label noise.arXiv preprint arXiv:2207.03933, 2022. 1 Peste, A., Alistarh, D., and Lampert, C. H. Ssse: Efficiently erasing samples from trained machine learning models.arXiv preprint arXiv:2107.03860, 2021. 2, 3, 4 Prabhu, V. U. and Birhane, A. Large image datasets: A pyrrhic win for computer vision? InWACV, 2021. 1, 18 Sanyal, A., Dokania, P. K., Kanade, V., and Torr, P. H. How benign is benign overfitting?arXiv preprint arXiv:2007.04028, 2020. 1 Sanyal, A., Hu, Y., and Yang, F. How unfair is private learning? InUncertainty in Artificial Intelligence, p. 1738–1748. PMLR, 2022. 5 Schelter, S. ”amnesia” - machine learning models that can forget user data very fast. InCIDR, 2020. 6, 19 Schelter, S., Grafberger, S., and Dunning, T. Hedgecut: Maintaining randomised trees for low-latency machine unlearning. InSIGMOD/PODS, 2021. 19 Shibata, T., Irie, G., Ikami, D., and Mitsuzumi, Y. Learning with selective forgetting. InIJCAI, 2021. 3, 4 Shokri, R., Stronati, M., Song, C., and Shmatikov, V. Membership inference attacks against machine learning models. InIEEE S&P, 2017. 1, 4, 5, 19 Shorten, C. and Khoshgoftaar, T. M. A survey on image data augmentation for deep learning.Journal of Big Data, 2019. 19 Song, L. and Mittal, P. Systematic evaluation of privacy risks of machine learning models. InUSENIX, 2021. 4, 20 Thudi, A., Deza, G., Chandrasekaran, V., and Papernot, N. Unrolling sgd: Understanding factors influencing machine unlearning.arXiv preprint arXiv:2109.13398, 2021a. 3, 6 Thudi, A., Jia, H., Shumailov, I., and Papernot, N. On the necessity of auditable algorithmic definitions for machine unlearning.arXiv preprint arXiv:2110.11891, 2021b. 10 Tsai, C.-H., Lin, C.-Y., and Lin, C.-J. Incremental and decremental training for linear classification. InKDD, 2014. 19 Turk, M. and Pentland, A. Eigenfaces for recognition.Journal of cognitive neuroscience, 1991. 19 Wang, B., Yao, Y., Shan, S., Li, H., Viswanath, B., Zheng, H., and Zhao, B. Y. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. InIEEE S&P, 2019. 18 Warnecke, A., Pirch, L., Wressnegger, C., and Rieck, K. Machine unlearning of features and labels.arXiv preprint arXiv:2108.11577, 2021. 19 14 Wu, Y., Dobriban, E., and Davidson, S. B. Deltagrad: Rapid retraining of machine learning models. In ICML, 2020. 2, 3, 6, 9 Yan, H., Li, X., Guo, Z., Li, H., Li, F., and Lin, X. Arcane: An efficient architecture for exact machine unlearning. InIJCAI, 2022. 6, 19 Yang, C., Wu, Q., Li, H., and Chen, Y. Generative poisoning attack method against neural networks. In Information & Security, 2020. 1 Yosinski, J., Clune, J., Fuchs, T., and Lipson, H. Understanding neural networks through deep visualization. InICML-W, 2015. 6 Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y. Cutmix: Regularization strategy to train strong classifiers with localizable features. InCVPR, 2019. 10 Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning requires rethinking generalization. InICLR, 2017. 1, 6 15 Table 3: Reference for notations used in this work Abbr.Definition TThe default training procedure. Implementation listed in Appendix Section C.1. SThe entire training set. S f The samples to be removed, called the deletion set. S t Unseen (test set) samples from the same distribution (here, affected classes) as the deletion set. MThe original model, obtained from the training procedureTusing the entire train setS. M u The unlearnt model, obtained from applying some unlearning procedure onM. M T r The retrained model, obtained from the same training procedureTusing the retained dataS f . φ u The distribution of models obtained from applying an unlearning procedure on the original modelM. φ r The distribution of models obtained by retraining from scratch using a training procedureT ′ on the retain setS f . PNumber of parts a data & influence isolation strategy divides the training data. S ′ The subset of the original datasetSselected for manipulations in unlearning testing. S ′ u The subset of the test set (unseen samples) from the same distribution (in our case, affected classes) asS ′ . A,BThe confused classes in the IC test. nThe size of the deletion set to be chosen. Table 4: Reference for abbreviations used in this work Abbr.Definition ICInterclass Confusion test MIAMembership Inference Attack(s) CF-kCatastrophic-Forgettingklayers EU-kExact-Unlearningklayers DNNDeep Neural Network I.I.DIndependent and identically distributed NTK-FisherNeural Tangent Kernel + Fisher method of Golatkar et al. (2020b) stdevStandard Deviation In the appendix, we provide additional information and experiments to supplement the observations in the main paper. The appendix is organized sectionwise as follows: (A) We detail all notations and abbreviations used in the main paper. (B) We prove Theorem 1, analyze data isolation strategies, and discuss membership inference attacks. (C) We provide details of implementation, test and metric choices, unlearning method comparisons and utility calculations. (D)We show empirical results on (i) varying the number of unlearning epochs, (i) layers in the architecture and (i) samples to be deleted. We also vary the choice of confused classes and ablate the effect of warm restarts in our training procedure. These results demonstrate the robustness of our observations to changes in optimization details, model, selected classes, and training procedure. A References: Notations and Abbreviations In Tables 3 and 4 we list the notations and terms used in this work respectively for reference when reading the paper. 16 B Additional Analysis B.1 Proof of Theorem 1 Proof.We prove this by constructing two`-layered fully connected linear NNs, parameterised byW 1 ,W 2 and a distributionPsuch that, underPthey are close in weights and test error but far in robustness and fairness. LetW 1 =A 1 ,...,A ` andW 2 =B 1 ,...,B ` be the list of weight matrices of the two networks with each matrix having a dimension ofm×m. Consider all but the first layer of the two networks be identical. Specifically, A 2 =A 3 ·=A l =         σ σ . . . σ σ 1         with the remaining entries being 0 where we will defineσ,σ 1 later. Construct the first layer of the two networks as follows where >0. A 1 =         1 1 . . . 1 0         B 1 =         1 1 . . . 1         Closeness inL 2 weights By construction, the two neural networks are close in weights:‖W 1 −W 2 ‖ 1 = where‖W 1 −W 2 ‖ 1 = ∑ ` i=1 ∥ ∥ A i −B i ∥ ∥ . To complete the remainder of the proof, note that the two neural networks are essentially equivalent to linear functions with the weight parametersAandBrespectively where A=         σ l−1 σ l−1 . . . σ l−1 0         B=         σ l−1 σ l−1 . . . σ l−1 σ l−1 1         Next, we construct a data distributionPthat satisfies the criteria of our result. Our distributionPwill be supported on four pointsX 1 ,X 2 ,X 3 ,X 4 ∈R m where X 1 =   1,...,1 ︸︷︸ m−1 ,0   ,X 2 =   −1...,−1 ︸︷︸ m−1 ,0   ,X 3 =   1,...,1 ︸︷︸ m−1 ,−1   ,X 4 =   −1,...,−1 ︸︷︸ m−1 ,1   andPis defined as P[(X 1 ,+1)] +P[(X 2 ,−1)] = 1−αandP[(X 3 ,+1)] +P[(X 4 ,−1)] =α. Test ErrorIt is easy to verify that ifσ >0, thenR(f W 1 ),R(f W 2 )≤α. 17 Figure 4: Hyperbolic deterioration of efficiency in isolation-based unlearning when scaling to a large number of removed samples. In this work, we analyze|S f |from 100-4000 whereE[Y]∼1. FairnessNow, let X 3 ∪X 4 be the minority group and X 1 ∪X 2 be the majority group. Note that f W 1 ( X 1 ) =f W 1 ( X 3 ) = 1 andf W 1 ( X 2 ) =f W 1 ( X 4 ) =−1, thereby leading to Γ(f W 1 ) = 0. On the other hand, for any,mifσ,σ 1 are chosen such that σ l−1 1 >(m−1)σ l−1 ,(1) we have thatf W 2 ( X 3 ) =−1 andf W 2 ( X 4 ) = 1. Hence, Γ (f W 2 ) = 1. This completes the proof of Γ (f W 2 )−Γ (f W 1 ) = 1. Adversarial Robustness Letδ >0 be the adversarial perturbation budget. Then, the adversarial error of a network parameterised with parametersWis R Adv (f W ) =P X,y [∃z∈R m s.t.‖z‖≤δ∧f W (X+z)6=y] ≥αI ∃z 1 ,z 2 ∈R m s.t.‖z 1 ‖,‖z 2 ‖≤δ∧f W ( X 1 +z 1 ) 6= 1∧f W ( X 2 +z 2 ) 6=−1 . Note that if the parametersσ,σ 1 satisfy the following with respect to,δ σ l−1 1 δ >(m−1)σ l−1 (2) thenf W 2 ( X 1 −δe m ) =−1 andf W 2 ( X 2 +δe m ) = 1 wheree m is them th canonical basis vector. Thus, R Adv (f W 2 )≥1−α. It is also easy to verify that for anyz:‖z‖≤ 1 2 , we have that forf W 1 ( X 1 +z ) = 1 and f W 1 ( X 2 +z ) =−1. Thus,R Adv (f W 1 )≤α. Subtracting the two adversarial errors we obtain,R Adv (f W 1 )− R Adv (f W 2 )≥1−2α. Finally, combining Equations (1) and (2) and setting the parameters such that σ 1 σ ≥ ( m−1 δ ) 1 l−1 completes the proof. B.2 Against Isolation Strategies Examples include removing noisy labels Northcutt et al. (2021b,a), deleting poisoned samples Wang et al. (2019); Jagielski et al. (2018); Li et al. (2020), deleting data that induces harmful biases Prabhu & Birhane (2021); Fabbrizzi et al. (2021), and organizations requiring deletion of user data older than some retention 18 period. Even in the context of privacy, a single user mightownmultiple samples in the dataset. In biometrics like face recognition Turk & Pentland (1991), one user may form an entire class Baumhauer et al. (2020). Moreover, user deletion requests may occur in bursts after certainevents of interest, such as revelations of privacy leakages by an organization Acquisti et al. (2006). Lastly, batching online deletion requests requires less invocations of the unlearning procedure, boosting resource efficiency. A popular approach for unlearning is data-influence isolation, where each sample is made to contribute only to a small part of the training procedure or model. Unlearning such as retraining from scratch only the part affected by the deletion set erases the influence of the deletion set more efficiently. Isolation-based strategies change the training process by creating an ensemble Yan et al. (2022); Schelter (2020); Bourtoule et al. (2021); Graves et al. (2021); He et al. (2021), each of whose models is trained on different subsets of the dataset. This ensures architecturally Bourtoule et al. (2021); Aldaghri et al. (2021); Schelter (2020); Yan et al. (2022) or temporally He et al. (2021); Bourtoule et al. (2021) isolating the influence of any sample to a limited part of training, requiring retraining for only the affected parts. Isolation has been used across techniques like Linear Classification Aldaghri et al. (2021), Random Forest Schelter et al. (2021); Brophy & Lowd (2021), KNN COO (1982), SVM Cauwenberghs & Poggio (2000); Tsai et al. (2014) and DNN Graves et al. (2021); Bourtoule et al. (2021); He et al. (2021) by utilizing or creating a sparse influence graph Schelter (2020). Data-influence isolation often comes at the cost of utility as each portion becomes a weaker learner Banko & Brill (2001), especially in deep networks Shorten & Khoshgoftaar (2019). To overcome the dropping utility, the training and unlearning time may need to be increased, reducing resource efficiency. Figure 4 demonstrates that the computation costs of isolation-based strategies scale poorly as the deletion set size increases. Note that even on a practical deletion set size like 500, existing isolation based approaches (which create much less than 250 isolated portions) require almost full retraining costs on expectation. Let Pbe the number of parts obtained with the isolation strategy. We assume the best-case scenario where each sample only influences one part. We make the simplifying assumption that the samples are uniformly distributed across parts, and the probability of a removed sample belonging to any particular portion remains constant ( 1 P ). LetYbe the number of affected parts. The probability partiis affected by atleast one sample inS f is 1−(1− 1 P ) |S f | . Thus by the linearity of expectation: E [Y] =P ( 1− ( 1− 1 P ) |S f | ) . We also show that the probability of full-retrain in data-influence isolation unlearning methods scales poorly with increasing deletion set size. Letp(n) be the probability that allPportions are affected on the removal ofnsamples. Extending the analysis of Warnecke et al. (2021) from the specific case of SISA to data-influence isolation in general, we get: p(n) = 1− ∑ |P| j=1 (−1) j+1 ( |P| j ) (|P|−j) n |P| n Figure 5 showsp(|S f |) grows logistically, implying there is a fast increase in the chance of needing a full-retrain as deletion sets get larger. This demonstrates how data-influence isolation provides little improvement in efficiency compared to the retrain-from-scratch baseline for practical scenarios. Note that some prior work such as Graves et al. (2021) do not re-train the affected portions without the deleted data, instead removing them entirely. This replaces the hit on resource efficiency with decreased utility (such as accuracy) as more deletions lead to more affected portions being removed from the model. This explains why Graves et al. (2021) produces an almost random model in the Class Removal experiment shown in Table 2. B.3 Membership Inference Attacks B.3.1 Background Membership Inference Attacks (MIA) Shokri et al. (2017) can be used to determine whether a particular sample was part of the training data of a model. Many different black-box formulations of MIA have been used to measure the efficacy of unlearning. Most Golatkar et al. (2020b, 2021); Liu et al. (2020); Graves et al. (2021) learn a binary attack classifier: based on the model’s output for the sample, was the sample in the 19 Figure 5: Logistic growth of the probability of needing to retrain all portions with increasing deletion set size. We represent isolation strategies with different portion sizesP. seen training set (class 0) or the unseen test set (class 1)? The attack classifier is then applied on deletion set samples, with ideal unlearning entailing all samples are classified as unseen. However, such a test is extremely sensitive to the efficacy of the attack classifier which may be unreliable. Another approach has been to train the attack classifier to distinguish the outputs of a large number of original (M) and retrained (M T r ) models and then classify the unlearnt modelM u Baumhauer et al. (2020). This formulation involves prohibitive computational expense and still can’t check over all potentialT ′ 6=T, indistinguishability with any of whom would guarantee unlearning. Song & Mittal (2021) recently show thatmetric-MIA, measuring simple metrics and deciding membership based on a threshold, can match the classification accuracy of trained attack models. In particular, their confidence-based MIA measures the model’s output probability for the target class and selecting separate class-wise membership thresholds. It is shown to match the performance of even white-box MIA attack classifiers. B.3.2 Our Formulation We adapt the confidence-based MIA Song & Mittal (2021) to propose an efficient black-box MIA formulation specifically tailored for measuring forgetting. We assume direct access to the actual model outputs instead of shadow models Graves et al. (2021); Liu et al. (2020), as shadow models only weaken the attack, making the unlearning test artificially easier to pass. We distinguish the model outputs on samples from the deletion set S f and unseen samplesS t from the same underlying distribution rather than training an attack classifier using the entire train and test set. We believe this formulation is a more targeted measurement of forgetting as it directly discriminates between outputs onS f andS t in contrast to train and test set used in past literature Golatkar et al. (2020b, 2021); Graves et al. (2021); Liu et al. (2020). Our MIA takes in modelM, forget setS f and unseen samplesS u from the same classes found inS f . The following procedure is repeated for each ’target class’t: •DatasetS MIA is created with the probability outputs for classt:M(S f ) t andM(D u ) t stored as class 0 and class 1 respectively. •We then create a 50-50 6 shadow (S MIA−S ) - test (S MIA−T ) split ofS MIA . •A thresholdp t needs to be chosen such that probabilities> p t are classified as class 0, and probabilities < p t as class 1. Thep t that maximizes the accuracy onS MIA−S is chosen. 6 Given that only 1 parameter (threshold) needs to be learnt, the shadow size is sufficient 20 •The accuracy obtained onS MIA−T using thresholdp t is the MIA accuracy for target classt. A weighted average of this test accuracy across all target classes is taken as the final MIA accuracy. Usually, the target classtis the actual label of the sample. However, in the case of IC test, we use the mislabelled class as the target for both,S f andS u samples. Intuitively, the memorization of mislabels in the deletion set would make the wrong class probability output unnaturally higher than other unseen samples of the same class, making the MIA stronger. Such an enhancement is not possible in the case of I.I.D confusion as the mislabels are untargeted. In line with existing MIA literature, we want our attack classifier accuracy to be 50% incase of no classifier advantage. Thus as the forget set and unseen set may have differing sizes in some experiments, we take a random subset of the larger one to make the attack dataset balanced. The numbers reported are averaged over 20 runs with randomness induced by the subset sampling step. Note that since the classifier learns to distinguish between the test and forget set distribution directly, it might be able to distinguish them spuriously, leading to slightly more than 50% attack classifier accuracy even on perfect unlearning. Thus, the reference gold standard MIA performance can instead be that of any exactly unlearnt model upon undergoing the same evaluation. C Additional Details We now provide some additional details for results shown in the main paper. C.1 Implementation Details Training. We use the ResNet architecture He et al. (2016) with 110 layers. Our standard training procedure Tis as follows: We train our models for 62 epochs (CIFAR10) or 126 epochs (CIFAR100), using a SGD optimizer with momentum 0.9 and weight decay 5e-5, an SGDR scheduler witht mult = 2,t 0 = 1, minlr = 5e-3, maxlr = 0.01 and a batch size of 64. For EU-kand CF-kbaselines, we use this same training process, but on the finalklayers. In CF-k, the only difference is we finetune for only half the epochs. The setup used for all experiments is a PC with a Intel(R) Xeon(R) E5-2640 2.40 GHz CPU, 128GB RAM and 1 GeForce RTX 2080 GPU. We make the following deviations in our experiments: •In Table 2 we make changes described in Section 3. • In Figure 3 and Table 6 we change the training procedure. When using cutmix regularization, we usep= 0.5 andα= 1.0. For early stopping, we halve the number of epochs both while training the original/retrain models and also in the unlearning procedures. •In Table 7 we vary the number of finetuning epochs in CF-k. •In Figure 11 we vary the confused classes in the IC test from easy-hard on the axis of distinguishability. •In Figure 6 we further benchmark on ResNet-20, ResNet-56 and ResNet-110 to show our results are robust to the choice of network depths. •In Table 8 we ablate the effect of warm restarts in training the original/retrain model. C.2 Metrics C.2.1 Inclusion in the Evaluation Comparisons Table Note that the list of metrics in Table 1 of the main paper does not include metrics like upper bound on information remaining in weights and activations Golatkar et al. (2020a,b, 2021) since its unclear whether such metrics can be computed on methods other than their own proposed unlearning procedure. We also exclude purely-qualitative tests such as model inversion attacks Fredrikson et al. (2015) which have been used in prior unlearning works Graves et al. (2021); Baumhauer et al. (2020). 21 C.2.2 Details of Metric Computation Targeted ErrorWe propose Targeted Error which measures the number of samples classified according to a property (information) unique toS f . For the IC-test, it is the fraction of samples still confused between the two classes, i.e.Targeted Error(M,S,A,B) = C M,S A,B +C M,S B,A |S A |+|S B | . whereC M,S is the confusion matrix when using modelMoutputs on datasetSandA,Bare the classes confused. For confusion betweenN >2 classes, targeted error is the sum of the confusion matrix terms for all pair-wise misclassifications among theNclasses. Thus, targeted error converges to error whenNis the same as the total number of classes, as in I.I.D Confusion. For Class Removal test, targeted error calculates the number of samples labelled as the removed class. Note that the influence of utility on targeted error is significantly lesser than the simple error metric on the affected classes as illustrated in Figure 1. Regarding the passing score for IC test: We speculate achieving lower targeted error than randomly initialized models could be sufficient. However, achieving this score is not necessary: even on exact unlearning ofS f , our models obtain a higher score due to samples in the retained set (S f ) having noisy annotations, an unavoidable phenomena in real-world datasets. We approximate this inherent noise in the dataset by using the model retrained from scratch. For clarity, we further describe the computation of some metrics. Our MIA has already been described in Appendix Section B.3. Note that for measuring memorization, the deletion set is used, while for measuring generalization (a subset of) the test set is used. IC Targeted Error: For the IC test between classAandB, the targeted error represents the number of samples of classAmislabelled as classBand vice-versa. Intuitively, as the mislabelled samples are forgotten by the unlearning procedure, the model should confuse lesser samples between these two classes. Class Removal Targeted Error : For the class removal test removing samples from classA, the Targeted Error represents the number of samples the model classifies as classA. Intuitively, as more samples fromA are removed, the model should classify lesser samples intoA. Note that if the entire class is not removed, a model that generalizes better from the partial samples still available may get penalized unnecessarily. IC Error: Error on train/test samples from the confused classes of the IC test,AandB. Class Removal Error: Error on train/test samples of the removed classA. I.I.D confusion, Error: Error on all samples from the train/test set. Here, a specific set of classes cannot be used for a targeted measurement. C.3 Choices for Tests In the IC test we confuse samples between classes 3 (Cat) and 5 (Dog) on CIFAR10 and classes 47 (Maple Tree) and 52 (Oak Tree) on CIFAR100 unless otherwise specified. Confusing two classes can harm the overall accuracy of the original model, and we expect this effect to be more prominent when the total number of classes in the dataset is lower. The deletion set size is the same as the number of samples from one class in the training set unless otherwise specified. Note that while the size ofS f is the same when comparing different tests, the size ofS t is dependent on the test itself. In targeted tests (Class removal, IC),S t only has test set samples from the affected classes, whereas in untargeted tests (I.I.D Removal, I.I.D confusion)S t consists of the entire test set. In Class Removal test we remove class 0 for both CIFAR10 and CIFAR100, whereas in I.I.D Removal and I.I.D confusion we draw an equal number of samples randomly from each class. C.4 Utilities To measure utility, we compute error on unseen samples from the same distribution (unaffected classes) as S f , called the retain distribution. For the I.I.D Removal and I.I.D confusion tests, as the removal is untargeted, the evaluated samples are the same as the full test set. For the Class Removal and I.I.D confusion tests the evaluated samples consist of test set samples from the unaffected classes. This is done as error on samples from the deletion set distribution correlates with the unlearning efficacy, and thus removing them leads to a measurement of utility largely independent of unlearning. 22 MethodI.I.D Removal (↓)Class Removal (↓)I.I.D confusion (↓)IC (↓) CIFAR-10 (|S f |= 4000) Original8.4±0.28.8±0.414.3±0.76.9±0.5 1-layer CF8.4±0.18.3±0.213.0±0.96.5±0.5 EU8.5±0.18.4±0.312.8±0.96.6±0.3 10-layers CF 8.4±0.28.4±0.212.0±0.66.5±0.4 EU8.7±0.18.6±0.212.0±0.76.7±0.2 50-layers CF 8.5±0.18.1±0.410.0±0.46.1±0.3 EU9.3±0.38.8±0.410.4±0.46.9±0.5 Retrain9.3±0.18.2±0.38.8±0.36.4±0.2 CIFAR-100 (|S f |= 400) Original32.1±1.131.6±1.132.4±1.431.8±0.8 1-layer CF 32.1±1.031.7±1.232.4±1.331.7±0.7 EU32.1±1.031.7±1.132.4±1.231.8±0.7 10-layers CF32.4±0.932.2±1.132.6±1.332.0±0.6 EU33.3±1.232.5±1.133.3±1.232.8±0.9 50-layers CF31.7±0.931.5±1.031.8±0.831.2±0.6 EU32.1±0.331.6±0.231.8±1.031.7±1.0 Retrain32.2±0.431.6±0.631.7±1.331.8±1.0 Table 5: Error on the retain set distribution of test samples across unlearning tests. Scores are reported as: mean±stdev. The EU-kand CF-kunlearning procedures lead to a minimal change in utility compared to retraining from scratch, unless utility is correlated with unlearning in the applied test. In Table 5 we show the utilities of the EU-kand CF-kunlearning procedures across all four tests. We observe a negligible impact on utility compared to retraining from scratch, unlike most unlearning procedures suggested in existing literature. The only significant difference in error is observed in the I.I.D confusion test, where better unlearning leads to improved utility as the model gets less confused by the mislabelled samples. Note that this is not observed in the IC test as the error is reported on only the unaffected classes, where error is independent of unlearning. Thus, EU-kand CF-kcan be used to control the unlearning-efficiency tradeoff at a fixed utility. In Table 6 we show the impact of regularization on utility. We observe that early stopping slightly increases the errors, while cutmix alone reduces them especially in CIFAR100. Given the significant improvement in utility and greater downstream amenability to unlearning, using regularizers like Cutmix seems highly rewarding. Our unlearning procedures do not decrease the utility barring a slight deterioration when the training procedure uses cutmix while the unlearning procedure does not. D Additional Experiments Finally, we vary some of the choices we make in our experiments to demonstrate the robustness of our observations. D.1 Varying the number of unlearning epochs The original experiments train CF-kmodels for half the epochs compared to EU-kmodels. In Table 7 we compare the variation of performance among CF-kmodels at the end of each warm restart while finetuning. While less information is unlearnt on reducing epochs, even six epochs are sufficient for drastic improvements in forgetting, with no significant change in utility (error on full test set). The number of catastrophic forgetting epochs can thus be reduced, and control the forgetting-efficiency tradeoff at constant utility. 23 MethodNoneEarly StopCutmixCutmix+Early CIFAR-10 (|S f |= 4000) Original6.537.915.818.02 10-layers CF6.117.935.457.27 EU 6.557.885.687.08 50-layers CF5.757.315.326.75 EU 6.577.876.168.20 Retrain6.318.505.708.97 CIFAR-100 (|S f |= 400) Original32.5333.1027.2630.03 10-layers CF 32.2233.0427.9830.61 EU33.2533.2328.5931.23 50-layers CF30.9332.3727.9229.37 EU31.9833.6630.4130.93 Retrain30.6432.6226.6730.67 Table 6: Error on the retain set distribution of test samples on varying the training procedure of the original model. Regularized models have better utility even after unlearning. D.2 Varying the number of layers In Figure 6 we show results of varyingkfor 3 different ResNet depths: 20, 56 and 110. The IC test is able to detect retained information despite exact unlearning of almost 30% of the final layers. However, on unlearning the final half of the network, its unclear whether most information is removed or the IC test is unable to identify the presence of retained information. CF-kis consistently within a small margin of EU-k demonstrating the catastrophic forgetting is able to lose enough information to match EU while being two times faster. D.3 Varying Amount of Untargeted Removal In Figures 7 and 8, we show the forgetting performance when we vary deletion set sizes in tests with untargeted removal: I.I.D Removal and I.I.D Confusion. Here, we use larger sizes than those reported in the main paper as smaller deletion sets show negligible trends in untargeted removal. For detecting effects on property generalization, Error on I.I.D confusion test needs far fewer samples than Error on I.I.D Removal. For memorization, we see that Error is able to distinguish and rank models fairly well whereas MIA works well in the case of I.I.D Removal test but fails completely on the I.I.D Confusion test. CF models continue to be close to EU models here and the gap between them decreases as we add more confusion. Overall, untargeted removal requires much larger deletion sets to show clear forgetting trends as compared to targeted removal, demonstrating the usefulness of strategic sampling. D.4 Varying Amount of Targeted Removal Now, we study the forgetting performance for partial Class Removal and partial Interclass Confusion. We show results for varying|S f |from 10% samples of a class to the size of an entire class (as used in the original paper). First, we present the results of the IC test in Figure 9. We see that for memorization all metrics are reflective even when a very small subset of samples is confused. Error and MIA having increasingly better contrast for smaller deletion sets. Then, we present the results of the Class Removal test in Figure 10. The Class Removal test has significantly different behavior when all samples of the class are removed compared to partial class removal. In the case of full Class Removal, all information about the class is removed, and hence an unlearnt model is 24 MethodEpochsMemProp. Gen.Test-Error (Targeted Error)(Targeted Error) CIFAR10 Original-3016.0927.016.00 CF-10 6145340811.07 14130536610.63 30122633510.24 CF-50 67582519.42 146432419.17 305692299.25 Retrain623901849.33 CIFAR100 Original-3957032.53 CF-10 63575632.71 143485532.60 30 3375432.98 623255732.60 CF-50 61284732.88 141414532.12 301084732.11 62863631.92 Retrain126643130.82 Table 7: Varying catastrophic forgetting epochs on the IC test. The number of epochs used for fine-tuning can further control the forgetting-efficiency tradeoff without hurting utility. expected to not classify any sample as the removed class. However, in partial Class Removal, a well generalized model may correctly classify more samples as the affected class, thus leading to the misalignment of utility and forgetting. We observe that MIA seems to have unclear trends in partial Class Removal, sometimes giving a weak signal for unlearning efficacy. D.5 Varying Confused Classes Throughout our experiments, we only confused the hardest pair of classes in the dataset (Cat and Dog for CIFAR10, Maple Tree and Oak Tree for CIFAR100). In Figure 11 we ablate the chosen class pair, grouping the ten classes in CIFAR10 into five pairs to maximize diversity. The five pairs are arranged in increasing order of similarity below along with their bar color: •Frog (6) - Horse (7): Red •Bird (2) - Ship (8): Blue •Airplane (0) - Deer (4): Light Green •Automobile (1) - Truck (9): Dark Green •Cat (3) - Dog (5): Black We can see that the number of confused samples by any model is much higher as we go from left to right, indicating that confusing a similar pair of classes makes unlearning more difficult. Both memorization and property generalization trends across varying levels of unlearning, from Original to Retrain, are consistently preserved. This shows that irrespective of the chosen class pair, the IC test is able to clearly distinguish varying degrees of forgetting. 25 SchedCF-10CF-50 MIATargeted ErrorMIATargeted Error CIFAR10 (|S f |= 4000) WR58.6633554.24229 No58.1034053.62234 CIFAR100 (|S f |= 400) WR77.995858.2441 No 78.875558.5743 Table 8: We compare Warm Restarts and keeping a single learning rate cycle between the same maxLR and minLR. MIA represents memorization while Targeted Error measures property generalization. D.6 Learning Without Restarts One concern which may arise is whether catastrophic forgetting performs well due to warm restarts in our learning rate schedule. We ablate this effect in Table 8 and see that in all cases removing warm restarts has no effect on the degree of catastrophic forgetting. 26 CIFAR-10CIFAR-100 ResNet - 20 ResNet - 110 ResNet - 56 Figure 6: We plot the MIA (Y) vs number of layers unlearnt using EU-k(solid blue) and CF-k(dashed red) for different architectures across datasets. For each model (point) we report three forgetting metrics as ‘memorization|property generalization (MIA)’ with memorization and property generalization computed using targeted error. The leftmost point is the original model while the rightmost EU point is the full retrained model. We observe consistent observations with the main paper across metrics and datasets. 27 Memorization Property Generalization CIFAR - 100 CIFAR - 10 400 20008000 20000 EU-k EU-k EU-k EU-k 400 20008000 20000 Figure 7: Varying|S f |for I.I.D Removal test. Error seems to distinguish varying levels of memorization, but needs huge deletion sets (50% of dataset size) in the case of property generalization. Moreover, here error has the limitation of misaligning forgetting (↑is better) and utility (↓is better). Memorization Property Generalization CIFAR - 100 CIFAR - 10 400 20008000 20000 EU-k EU-k EU-k EU-k 400 20008000 20000 Figure 8: Varying|S f |for I.I.D Confusion test. Error reliably measures memorization even in small deletion sets (1% of deletion set size), though much larger ones (20% of deletion set size) are needed to produce detectable effects on property generalization. 28 Memorization Property Generalization CIFAR - 100 CIFAR - 10 400 10002000 4000 40 100200 400 EU-k EU-k EU-k EU-k Figure 9: Varying|S f |for IC test. In CIFAR10, at 1% of dataset size, the IC test reliably detects imperfect forgetting across metrics. In CIFAR100, imperfect removal of memorization is detected at 1% of the class size, a noticeable effect on generalization requires a larger deletion set (5% of dataset size). Memorization Property Generalization CIFAR - 100 CIFAR - 10 400 10002000 4000 40 100200 400 EU-k EU-k EU-k EU-k Figure 10: Varying|S f |for Class removal test. The Class removal test is not able to reliably distinguish varying levels of property generalization and provides a weak signal for memorization, particularly for small |S f |. 29 MemorizationProperty Generalization 4000 (10%) 400 (1%) +Early Stop +Cutmix +Cutmix& Early Stop EU-k EU-k EU-k EU-k Inter-Class Confusion (Ours) Class Frog, HorseBird, Ship Airplane, Deer Automobile, Truck Cat, Dog Figure 11: Varying confused class pairs on CIFAR10, with the similarity of the classes increasing from left to right in each group of bars. While the IC test reliably detects imperfect forgetting across class pairs, the trends are clearer for more similar classes. 30

Instant research discovery