Paper deep dive
Learning to Unlearn: Machine Unlearning via Learning the Unlearning Behaviors
Hang Zhang, Kaifeng Zhang, Yixiao Ma, Weijie Xu, Ye Zhu, Kai Ming Ting
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 8/23/2026, 2:36:38 AM
Summary
The paper introduces Learning-to-UnLearn (L2UL), a model-agnostic machine unlearning framework that learns an unlearning function U via a neural network rather than manually designing it. By using Kernel Mean Embedding with Isolation Kernels to represent data distributions and training on the difference between original and retrained model parameters, L2UL achieves accuracy comparable to full retraining with significantly higher efficiency, particularly for large datasets.
Entities (8)
Relation Signals (6)
L2UL → uses → Isolation Kernel
confidence 92% · we employ Kernel Mean Embedding ... Isolation Kernel ... to transform distribution
GDPR → motivates → Machine Unlearning
confidence 90% · legislative measures like ... GDPR ... have been introduced to establish rules that protect users’ privacy
L2UL → outperforms → Retrain
confidence 90% · exhibiting impressive efficiency, particularly in data-intensive scenarios
L2UL → outperforms → SISA
confidence 85% · comparison algorithms ... SISA ... DeltaGrad, FYEMU
L2UL → outperforms → DeltaGrad
confidence 85% · comparison algorithms ... DeltaGrad
L2UL → validateson → ResNet
confidence 85% · validate the performance and scalability of our method on larger models ResNet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Various machine unlearning techniques have been developed in response to privacy legislation requirements, enabling individuals to exercise their legal right to have their data $D_f$ removed from a machine learning model. This process is typically accomplished via the use of an unlearning function denoted as $U$. Existing methods focus on designing an intricate $U$ to unlearn $D_f \subset D$ from a previous model $A(D)$, so that the unlearned model performs as closely as possible to the retrained model $A(D \setminus D_f)$. However, these methods often suffer from high computational costs when dealing with massive training data, as the complex structures of $U$ become a bottleneck even for models with fewer parameters. Inspired by Learning to Optimize, we introduce the first learning-based model-agnostic approach, Learning-to-UnLearn (L2UL). Our core insight is to shift from manually designing $U$ to learning the unlearning behaviors from a distribution perspective, thereby acquiring a simple and efficient $U$ via learning. Our experimental results demonstrate that the accuracy achieved by L2UL is comparable to that of retraining while exhibiting impressive efficiency, particularly in data-intensive scenarios. Furthermore, we validate the performance and scalability of our method on larger models ResNet.
Tags
Links
- Source: https://arxiv.org/abs/2608.16700v1
- Canonical: https://arxiv.org/abs/2608.16700v1
Trouble viewing inline? Open PDF directly →
Full Text
58,055 characters extracted from source content.
Expand or collapse full text
Learning to Unlearn: Machine Unlearning via Learning the Unlearning Behaviors DOI: X.XXXXXXXConference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NYISBN: 978-1-4503-X-X/2018/06CCS: Security and privacyCCS: Computing methodologies Machine learningCCS: Computing methodologies Artificial intelligence Hang Zhang Affiliation: State Key Laboratory for Novel Software Technology Nanjing University , Nanjing , China email: zhanghang@lamda.nju.edu.cn , Kaifeng Zhang Affiliation: State Key Laboratory for Novel Software Technology Nanjing University , Nanjing , China email: zhangkf2022@lamda.nju.edu.cn , Yixiao Ma Affiliation: State Key Laboratory for Novel Software Technology Nanjing University , Nanjing , China email: mayx@lamda.nju.edu.cn , Weijie Xu Affiliation: State Key Laboratory for Novel Software Technology Nanjing University , Nanjing , China email: xuwj@lamda.nju.edu.cn , Ye Zhu Affiliation: Centre for Cyber Resilience and Trust Deakin University , Burwood, VIC , Australia email: ye.zhu@ieee.org and Kai Ming Ting Affiliation: State Key Laboratory for Novel Software Technology Nanjing University , Nanjing , China email: tingkm@nju.edu.cn 2018; © , 2018; Received 5 June 2009 Abstract. Various machine unlearning techniques have been developed in response to privacy legislation requirements, enabling individuals to exercise their legal right to have their data DfD_f removed from a machine learning model. This process is typically accomplished via the use of an unlearning function denoted as U. Existing methods focus on designing an intricate U to unlearn Df⊂D_f⊂ D from a previous model A(D)A(D), so that the unlearned model performs as closely as possible to the retrained model A(D∖Df)A(D D_f). However, these methods often suffer from high computational costs when dealing with massive training data, as the complex structures of U become a bottleneck even for models with fewer parameters. Inspired by Learning to Optimize, we introduce the first learning-based model-agnostic approach, Learning-to-UnLearn (L2UL). Our core insight is to shift from manually designing U to learning the unlearning behaviors from a distribution perspective, thereby acquiring a simple and efficient U via learning. Our experimental results demonstrate that the accuracy achieved by L2UL is comparable to that of retraining while exhibiting impressive efficiency, particularly in data-intensive scenarios. Furthermore, we validate the performance and scalability of our method on larger models ResNet. Keywords: Machine unlearning, Data privacy, Privacy-preserving learning. 1. Introduction With the progress of society, there has been a significant increase in the amount of available data. This growth has enriched our lives by allowing us to access a wide range of data shared by numerous individuals online. However, it also presents risks to the privacy of users. In response to this, legislative measures like CCPA (California Consumer Privacy Act), PIPEDA (Personal Information Protection and Electronic Documents Act), and GDPR (General Data Protection Regulation) have been introduced to establish rules that protect users’ privacy by the right to delete their data. The task of erasing data from a machine learning model is named machine unlearning (4; 1; 22). A naive approach is to retrain the model on the remaining data, but this is expensive as models are often trained on very large datasets. Proposing efficient and effective machine unlearning methods has attracted the attention of researchers in recent years. Current approaches focus on designing a complex unlearning function U so that the unlearned model performs close to the retrained model. For example, some methods design U as a retraining on all or part of the data, which is very time-consuming and requires retraining every time on an unlearning request. Some methods necessitate the calculation of the Hessian matrix for the designed U, which further increases the computational cost. Consequently, even for relatively simple models, the unlearning efficiency remains bottlenecked by the complexity of the designed U, especially when navigating massive datasets. The goal of an unlearning function U is to produce a model that has unlearned the user data and performs close to the model retrained on the remaining data. Why don’t we learn to unlearn directly? Inspired by Learning to Optimize (L2O) (6; 27), traditional optimization algorithms like Adam are manually designed by human experts through theoretical derivation. In contrast, the L2O approach aims to let machines automatically learn how to optimize. Specifically, the L2O model takes the current state of the optimization iteration as input (such as the current solution and its gradient) and learns to predict the update amount for the next iteration. Instead of designing a complex and highly time-consuming U, we use the retrained model as ground truth to learn a U whose structure is simple but enough to unlearn DfD_f effectively. In this way, we can achieve the goal of unlearning much more efficiently. Although many methods claim learning to unlearn (5; 17), they usually refer to learning a new decision boundary to achieve forgetting rather than learning the behavior of unlearning. In this paper, we introduce a model-agnostic framework called Learning-to-UnLearn (L2UL), which employs a machine-learning approach to learn a U. The main contributions of this work are: (1) Pioneering a new framework of machine unlearning via learning, named Learning-to-UnLearn (L2UL). (2) Providing the generalization bound of a model unlearned by L2UL in the context of Logistic Regression and Multi-Layer Perceptron (MLP). (3) Conducting an empirical validation of the effectiveness and efficiency of our proposed method. The advantages of the proposed L2UL over current methods are two-fold: (i) Unlike existing methods, in L2UL, the acquisition of U is achieved through learning rather than designing. (i) Due to the simple structure of U, the execution time of unlearning via the learned U is very short. 2. Problem Formulation Notations used in this paper are summarized in Table 1. Table 1. Key notations used in the paper. Symbols Definition X Sample space Y Label space D Training set DfD_f Forget set A(⋅)A(·) Learning algorithm U(⋅)U(·) Unlearning algorithm θ Parameters of the model trained on D θ′θ Parameters of the model trained on D∖DfD D_f P Probability distribution x A vector in ℝdR^d κ Kernel Function κI _I Isolation Kernel t,ψt,ψ Parameter of κI _I Let X denote the sample space, and Y denote the label space. A data point is an element in =×Z=X×Y. A hypothesis function h:→h:X is learned on training set D⊂D by algorithm A to assign y∈y to x∈x . Definition 2.1. A machine learning algorithm is a map A from a subset of Z to a hypothesis function h∈ℋh , where ℋH is the space of all hypothesis functions of the algorithm. After A learned a model from a dataset D, A(D)A(D) can be applied to a test set. When some users request to have their data Df⊂D_f⊂ D deleted, model A(D)A(D) is obliged to have unlearned their data. The task of unlearning data from a learned model is machine unlearning. Figure 1. The Machine unlearning workflow and the framework of our proposed method L2UL. The model A(D)A(D) is learned on dataset D (which includes DfD_f) by A. An unlearning algorithm U unlearns DfD_f from A(D)A(D) to obtain a revised model, which is expected to be equal to the retrained model A(D∖Df)A(D D_f). Figure 1 shows the workflow of machine unlearning. To fulfill a request to unlearn DfD_f from A(D)A(D), an unlearning algorithm U(D,Df,A(D))U(D,D_f,A(D)) produces a model which is expected to be the same or close to the model A(D∖Df)A(D D_f) retrained on the dataset D∖DfD D_f. Definition 2.2. A machine unlearning algorithm is a map U:2×2×ℋ→ℋU:2^Z× 2^Z×H , where 22^Z is the power set of Z, which is set containing all of Z’s subset. U takes D,Df,A(D)D,D_f,A(D) as input, and outputs the unlearned model, which is expected to be as close to the retrained model A(D∖Df)A(D D_f) as possible. Definition 2.3. Exact Unlearning: given a randomized learning algorithm A, a dataset D, and a data subset Df⊂D_f⊂ D to be forgotten, an unlearning algorithm U is an exact unlearning process if and only if (U(D,Df,A(D)))=(A(D∖Df))P(U(D,D_f,A(D)))=P(A(D D_f)), where (A(D))P(A(D)) defines the distribution of all models trained on a dataset D by a learning algorithm A. The naive unlearning algorithm simply retrains a new model on D∖DfD D_f to achieve exact unlearning. However, it is expensive to retrain when |D∖Df||D D_f| is large. 3. Related Work Many machine unlearning algorithms that remove some user data from a previously trained model without retraining have been proposed recently. According to the workflow shown in Figure 1, we divide the related work into three categories: D-oriented, DfD_f-oriented, and A(D)A(D)-oriented unlearning. D-oriented unlearning operates on subsets of D such that only retaining on some subsets is required upon a request to unlearn DfD_f. For example, SISA (1) divides the training data D into shards and divides each shard into slices. Each shard is used to train a constituent model, incrementally incorporating slices and preserving its parameters until the training set is extended with a new slice. When DfD_f is required to be forgotten, retraining is limited to the specific constituent model whose shards encompass DfD_f. Many methods share a similar intuition, such as (4; 2; 15; 10; 13; 38). DfD_f-oriented unlearning considers the impact of forgotten data on the model, through influence functions (36; 12; 34) and Markov Chain Monte Carlo-based sampling (21; 9). When DfD_f needs to be forgotten, the impact of DfD_f on the model can be erased accordingly. A(D)A(D)-oriented unlearning considers how the parameters of a learned model should be updated after DfD_f is forgotten from the training set (25; 20; 18; 29; 37). (28) updates the parameters by adding noise to the parameters to forget and then tuning the model on the remaining data D∖DfD D_f. Few-shot learning (8) is used in the case where D is unavailable, and only model A(D)A(D) is known. Although existing works can produce an unlearned model close to the retrained model, none of them use a learning technique to obtain U. Our proposed L2UL, an A(D)A(D)-oriented unlearning algorithm, tries to learn the unlearning function U, instead of designing one like current works. 4. Proposed Method 4.1. The Framework of L2UL Recall Definition 2.2 and Figure 1, U has three inputs, D (represented as a |D|×d|D|× d matrix), DfD_f (represented as a |Df|×d|D_f|× d matrix) and the parameter vector θ of model A(D)A(D). A machine unlearning algorithm U must satisfy the following properties: Proposition 4.1. The output of U should remain unchanged after any two rows in the data matrix of D or DfD_f are exchanged. This is easy to understand because exchanging any two rows of D and DfD_f does not change the dataset. An effective method that satisfies the Proposition 4.1 is to represent D and DfD_f as distribution DP_D and DfP_D_f. No matter how the order of points of the dataset changes, its distribution remains the same. Therefore, we redefine Definition 2.2 as: Definition 4.2. A machine unlearning algorithm is a function U: ×ℋ→ℋ P× P×H , where P is the space containing all the probability distributions. Different from Definition 2.2, Definition 4.2 defines machine unlearning from the perspective of distribution. Based on Definition 4.2 and the unlearning workflow shown in Figure 1, L2UL is naturally proposed by using three components of the unlearning workflow: D,DfD,D_f, and A(D)A(D). Figure 1 shows the framework of L2UL, which uses a neural network NUN_U11 1 Other models can also be used. A two-layer (256, 64) fully connected neural network is used in our experiments. to produce the unlearned model Aθ′A_θ . In order to vectorize distribution DP_D and DfP_D_f as input to NUN_U, we employ Kernel Mean Embedding (KME) (26) to transform distribution DP_D and DfP_D_f to vector μD _D and μf _f in Reproducing Kernel Hilbert Space (RKHS). Kernel mean embedding is a nonparametric method that uses an element of an RKHS as the representation of a probability distribution. The mean feature map μℙ=[κ(⋅,D)] _P=E[κ(·,D)] of points X based on kernel κ embeds distribution ℙP into RKHS, and preserves all of the statistical features of ℙP. In our framework, when using some common kernels such as the Gaussian kernel, the Laplacian kernel, etc., μℙ _P will map ℙP into an infinite-dimensional space, which cannot be used as input to the neural network. In other words, we need a kernel whose RKHS dimension is finite. One approach is to use low-rank representations of the kernel matrix. The most popular examples are Nyström method (14; 35; 19) and the Random Fourier Features (24; 39). However, these are all approximate methods and will reduce effectiveness. Another approach is to use a kernel with an exact and finite-dimensional feature map. Isolation Kernel (33) is such a kernel that directly obtains the feature map without calculating the kernel matrix, which has been widely used and performs well (40; 31; 3). Let D⊂ℝdD ^d be a dataset sampled from an unknown distribution ℙP. Isolation Kernel uses a partition mechanism such as Voronoi Diagram (23; 41) or hypersphere (32) to partition the data into |||D| cells, where ||=ψ|D|=ψ, and ⊂DD⊂ D is sampled from D as the seeds of partition mechanism. Let ℍψ(D)H_ψ(D) denotes the set of all partitions H, each cell ℐ[z] I[z] of partition H isolates z∈z from the rest of the points in D, the definition of Isolation Kernel is shown as follows: Definition 4.3. Isolation Kernel: ∀x,y∈ℝd∀ x,y ^d, Isolation Kernel of x and y is defined to be the probability that x and y fall into the same cell ℐ[z] I[z] of partition H over all the partitions H∈ℍψ(D)H _ψ(D): κI(x,y|D) _I(x,y|D) = ℍψ(D)[(x,y∈ℐ[z]|ℐ[z]∈H)]E_H_ψ(D)[1(x,y∈ I[z]| I[z]∈ H)] Given the t partitions H1,…,HtH_1,...,H_t, the feature map Φ(x) (x) of κI _I is a ψ×tψ× t-dimensional binary column vector. Definition 4.4. Isolation Kernel’s feature map, denoted as Φ(x) (x), is a vector that represents the cell in which x falls into over partition of t times. For x∈ℝdx ^d, the dimension of Φ(x) (x) is ψ×tψ× t, where ψ is the number of cells in one partition H. Each element of Φ(x) (x) is either 0 or 1, indicating the cell in which x falls. Since the dimension of Φ(x) (x) is finite, the KME of κI _I can be obtained by simply computing the mean of all feature maps. Then μD _D, μf _f, and the parameters θ of model A are concatenated as the input x of the neural network, which contains two hidden layers. The KME of distribution ℙP on the given data D is: (1) μℙ=[κI(⋅,D)]=∑x∈DΦ(x)|D|, _P=E[ _I(·,D)]= _x∈ D (x)|D|, where Φ(X) (X) is IK’s finite-dimensional feature map determined by two hyperparameters t,ψt,ψ. This is also known as the isolation distribution kernel (IDK)(32). According to Equation 1, we have: μD=∑x∈DΦ(x)|D| _D= _x∈ D (x)|D| and μf=∑x∈DfΦ(x)|Df|. _f= _x∈ D_f (x)|D_f|. μD _D and μf _f are later concatenated with the parameters of the original model as input x. The output θ′θ is the parameter of the unlearned model U(D,Df,A(D))U(D,D_f,A(D)). And Mean Squared Error (MSE) loss is used as the objective function for training U: (2) ℒ=1|θ′|∑i=1|θ′|(θi′−θir)2,L= 1|θ | _i=1^|θ |( _i -θ^r_i)^2, where θrθ^r is the parameters of retrained model A(D∖Df)A(D D_f). Intuitively, we are focusing on models with relatively few parameters but very large training datasets. Because the model has few parameters, U can be learned efficiently, and the large amount of training data can be represented via kernel mean embedding. Therefore, L2UL is effective and highly efficient. 4.2. How to Learn U In order to train U, first we need a training set =× D= X× Y. We produce D by the following three steps. First, we generate D by randomly sampling s points from D (s≪|D|s |D|), and train a model A()A( D) on D. Second, we randomly select some samples as a forget dataset f⊂ D_f⊂ D. μ _ D, μf _ D_f of D and f D_f are obtained using Equation 1. The parameter θ of model A()A( D) is concatenated with μ _ D and μf _ D_f as x. Finally, we obtain θrθ^r, the parameter of retrained model A(∖f)A( D D_f). Now we have one instance (,θr)∈×(x,θ^r)∈ X× Y. The above steps are repeated m times to obtain D containing m samples. NUN_U is later trained on D by minimizing the loss function ℒL shown in Equation 2. Since the parameters in KME do not require learning, learning NUN_U is equivalent to learning U. When there is a DfD_f that needs to be unlearned from model A(D)A(D), we can obtain the unlearned model Aθ′A_θ via the learned NUN_U. An important point to highlight is that once we have obtained NUN_U, if we need to unlearn any user data DfD_f, we can obtain the unlearned model U(D,Df,A(D))U(D,D_f,A(D)) via the learned NuN_u directly without having to repeat the preprocessing and training phase. The pseudo code of preprocessing for obtaining D and Φ(⋅) (·) for L2UL, and unlearning DfD_f are shown in Algorithm 1 and Algorithm 2, respectively. Input : D: Dataset, m: Number of Samples in D, s: size of D, ψ, t: Parameters of κI _I Output : D, Φ(⋅) (·) D = ; 1 Get the map function Φ(⋅) (·) using κI _I. ; 2 ∀x∈D∀ x∈ D, get the feature map Φ(x) (x); 3 for i←1i← 1 to m do 4 ← D← randomly sample s points from D; 5 f← D_f← randomly sample from D ; 6 A(),A(∖f)←A( D),A( D D_f)← train A on D, ∖f D D_f; 7 θ,θr←θ,θ^r← Extract parameters of A(),A(∖f)A( D),A( D D_f); 8 μ,μf← _ D, _f← obtain KME using Equation 1 ; 9 ←x← concatenation of μ,μf _ D, _f and θ ; 10 ← D← ∪(,θr) D∪\(x,θ^r)\; 11 end for 12 Return D, Φ(⋅) (·). Algorithm 1 Preprocessing Input : NU,μDN_U, _D, Φ(⋅) (·), A(D)A(D) Output : Aθ′A_θ ∀x∈Df∀ x∈ D_f, get the feature map Φ(x) (x); 1 μf← _f← obtain KME of DfD_f through Equation 1; 2 ←x← concatenation of μD,μf _D, _f and A(D)A(D)’s parameter θ ; 3 θ′←NU()θ ← N_U(x); 4 Return unlearned model Aθ′A_θ . Algorithm 2 Unlearn DfD_f 4.3. Generalization Bound Analysis Here, we provided the generalization bound of the classifier (Logistic Regression, Multilayer Perceptron) unlearned by L2UL. Denote the expected loss of Logistic Regression model A(x|θ)=eθx+b1+eθx+bA(x|θ)= e^θ x+b1+e^θ x+b as: L(A)=Ex,y[ylog(A(x))+(1−y)log(1−A(x)]L(A)=E_x,y[y (A(x))+(1-y) (1-A(x)]. L2UL learns the unlearning function U. The expected loss of the model unlearned by U is upper-bounded, as shown in the following Theorem 4.5. Theorem 4.5. L(U(A(D),D,Df))≤L(A(D∖Df))+2Rϵ,L(U(A(D),D,D_f))≤ L(A(D D_f))+2R ε, where R is the radius of the dataset X (∀x∈X,‖x‖≤R∀ x∈ X,||x||≤ R), and ϵε is the generalization error bound (MSE) of the unlearning model U in L2UL. Similarly, the expected loss of Multilayer Perceptron (MLP) unlearned by U is also upper-bounded, as shown in Theorem 4.6. Theorem 4.6. L(U(A(D),D,Df))≤L(A(D∖Df))+Cϵ,L(U(A(D),D,D_f))≤ L(A(D D_f))+C ε, where L is cross entropy loss, C is a finite scalar determined by the MLP structure and ϵε is the generalization bound (MSE) of the unlearning model U in L2UL. Both Theorem 4.5 and 4.6 indicate that the expected loss of the classifier unlearned by L2UL is close to that of retrained classifier. 5. Experiments System: The experiments are executed on a Linux machine with 1T GB RAM and an AMD 128-core CPU, with each core running at 2 GHz. Data: We use seven public datasets in our experiments22 2 The datasets are available at https://archive.ics.uci.edu/, and https://w.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/. The specifications of the datasets are summarised in Table 2. Table 2. Dataset Summary. n=no.instances, d=no.attributes. Magic Adult Sepsus Skin Covetype SUSY HIGGS n 19020 32562 40328 245057 581012 5000000 11000000 d 10 14 89 3 54 18 28 Comparison algorithms: We compare L2UL33 3 The codes are available at https://anonymous.4open.science/r/L2UL with Retrain, one D-oriented unlearning algorithm: SISA and two A(D)A(D)-oriented unlearning algorithms: DeltaGrad, FYEMU. (1) Retrain: As the most naive approach, it just retrains the model on the remaining dataset D∖DfD D_f. (2) SISA: SISA (1) is a well-known framework that partitions the data to reduce the retraining time by just retraining a single model on the shard that needs to be forgotten. (3) DeltaGrad: DeltaGrad (37) unlearns the data by differentiating the optimization path with the Quasi-Newton method based on information cached during the training phase. (4) FYEMU: FYEMU (28) is a unlearning algorithm for Neural Networks, which first unlearns data by adding noise to the model parameters, and then obtains a new model through fine-tuning. Evaluation: We evaluate the efficiency and effectiveness of unlearning algorithms using unlearning time and the accuracy of the unlearned classifier on test data (20%20\%), which is averaged over 10 runs. In each run, we unlearn one randomly selected instance44 4 |Df|=1|D_f|=1 for LR, |Df|=1|D_f|=1, |Df|=100|D_f|=100, and |Df|=1000|D_f|=1000 for MLP.. 5.1. Unlearn linear classifier: Logistic Regression To begin, we assess the performance of our L2UL approach using a linear classifier known as Logistic Regression (LR). We have demonstrated that the generalization error of the unlearned classifier, denoted as U(D,Df,A(D))U(D,D_f,A(D)), is bounded. Additionally, here we find that L2UL is both efficient and effective when applied to LR. Figure 2. Results of unlearning Logistic Regression. The results of unlearning one instance in terms of accuracy and unlearning time are shown in Figure 2. L2UL achieves comparable accuracy to Retrain and SISA and outperforms DeltaGrad over all seven datasets. Compared with Retain, SISA, and DeltaGrad, L2UL speeds up 630, 149, and 282 times on the Magic dataset, and 422500, 49715, and 51500 times on the HIGGS dataset, respectively. 5.2. Unlearn non-linear classifier: Multilayer Perceptron Nonlinear classifiers can learn complex nonlinear boundaries, which require a more complex process to learn. Besides, a nonlinear classifier usually has more parameters, which makes its parameter space very large. Implementing unlearning in this large space is a more challenging problem. In this subsection, we test the performance of L2UL on Multilayer Perceptron (MLP) (30). We use cross-entropy as the loss of the MLP, which has 1 hidden layer with 10 neurons. Figure 3. Results of unlearning MLP (|Df|=1|D_f|=1). The results of unlearning one instance are shown in Figure 3. L2UL achieves comparable accuracy to Retrain and SISA except HIGGS (we will discuss this in Section 5.4). L2UL achieves close accuracy to Deltagrad on Sepsis and Skin and outperforms DeltaGrad on five other datasets. L2UL outperforms FYEMU on the two smallest datasets (Magic and Adult) and achieves close accuracy to FYEMU on five other datasets. Compared with Retain, SISA, DeltaGrad, and FYEMU, L2UL speeds up 15087, 1203, 730, and 43 times, respectively, on the Magic dataset, and 398000, 105250, 294750, and 32750 times on the HIGGS dataset. The efficiency of L2UL comes from the simple structure of U, which ensures that a single run of U is very quick. Moreover, once U is learned, the parameters of U do not need to change with high probability when a new DfD_f comes. We will discuss this in Section 8. L2UL just directly uses the learned U to unlearn the new DfD_f, while the other algorithms require the costly computation again to unlearn DfD_f. This difference will be more significant when multiple unlearning requests come in sequence. The results of unlearning 100 and 1000 instances in terms of accuracy and unlearning time are shown in Figure 4 and Figure 5. Figure 4. Results of unlearning MLP (|Df|=100|D_f|=100). Figure 5. Results of unlearning MLP (|Df|=1000|D_f|=1000). 5.3. Unlearning Efficacy In addition to effectiveness and efficiency, the machine unlearning algorithm must also be evaluated to determine whether it has forgotten users’ information. We first show the two very commonly used evaluation metrics. Membership Inference Attack (7; 16) (as used in Table 5) and accuracy on unlearned data. The results in terms of Membership Inference Attack and accuracy on DfD_f (|Df||D_f|=1000 for MLP) in Table 3 show that L2UL has forgotten the information of DfD_f from A(D)A(D)55 5 L2UL classifies all samples into one class on the Sepsis dataset, so MIA cannot be performed. The U has learned the unlearning behaviors. Table 3. Results in terms of Membership Inference Attack and accuracy on unlearn data. MIA Accuracy on DfD_f Datasets Retrain FYEMU L2UL Retrain SISA FYEMU L2UL Magic 0.79 0.65 0.75 0.85 0.79 0.71 0.78 Adult 0.80 0.68 0.74 0.81 0.78 0.75 0.76 Skin 1.00 1.00 1.00 1.00 1.00 1.00 1.00 Covetype 0.73 0.68 0.67 0.82 0.76 0.76 0.71 SUSY 0.72 0.69 0.69 0.73 0.71 0.70 0.71 HIGGS 0.65 0.68∗ 0.50 0.74 0.77∗ 0.69 0.63 • ∗ indicates that the results show that complete forgetting of the data was not achieved. However, it is not the case that the lower these two scores are, the better the model performance is, because this may give rise to other issues, such as information exposure (11) 66 6 In this paper, if the scores are lower than both the original model (A(D)A(D), ORI) and Retrain, it means the unlearning algorithm has forgotten the information.. Therefore, we use an example to demonstrate the ability of the proposed algorithm L2UL for calibrating contaminated models, which also shows that L2UL has forgotten DfD_f. As previously stated, training data can sometimes be contaminated, which negatively impacts the model’s performance. Machine Unlearning is an effective method for cleaning the model when dirty data is detected. A simple example is shown in Figure 6(a). The artificial dataset has two classes, which can be classified by a linear model. But when the training set is contaminated (Figure 6(b) right shows an example with 50 contaminated points), the accuracy of the model will drop. (a) Clean data (b) Contaminated data (c) Accuracy of LR (d) Accuracy of MLP Figure 6. Artificial dataset with clean data and contaminated data, and the results of LR and MLP. Figure 6(c) shows the results of LR in terms of Accuracy. When data is not contaminated, the accuracy is 1, as expected. As the number of dirty points grows, the accuracy declines to approximately 0.5. When we use L2UL to clean the contaminated model by treating the dirty points as DfD_f, the accuracy of the unlearning model remains 1.00 as the number of dirty points grows. Similar results are shown in Figure 6(d) when we replace LR with MLP. The difference is that the accuracy and F1 of MLP decrease more slowly than LR. The accuracy and F1 of the unlearned model are always 1.00. This shows that L2UL has indeed achieved the unlearning of dirty data. 5.4. Parameter Sensitivity Analysis There are four hyperparameters in the Preprocessing and Unlearning phase of L2UL: ψ,tψ,t for Isolation Kernel and s, m for generating × X× Y. In our experiment, the default setting of hyperparameters is: ψ=4ψ=4, t=100t=100, m=1000m=1000, s=1000s=1000 for LR, and s=3000s=3000 for MLP. We report the accuracy of unlearning 1, 100, and 1,000 instances from LR and MLP on the largest datasets SUSY and HIGGS, with ψ∈ψ∈ [2, 4, 8, 16, 32, 64, 128, 256](32). (32) shows that the Isolation Distritbuion Kernel is not sensitive to t , hence, the sensitivity analysis of t is omitted here. Figure 7. Parameter sensitivity analysis of ψ (top), s (middle), m (bottom) on SUSY (left) and HIGGS (right) for LR. Figure 8. Parameter sensitivity analysis of ψ(top), s(middle), m(bottom) on SUSY (left) and HIGGS (right) for MLP. The results of ψ are shown in the first row of Figure 7 and Figure 8, L2UL is robust to ψ when ψ≥4ψ≥ 4. The results of s are shown in the second row of Figure 7 and Figure 8. For LR on SUSY, HIGGS, and MLP on SUSY, L2UL is robust to s, and the accuracy of L2UL is close to that of retrain. For MLP on HIGGS, as s increases, the accuracy of L2UL increases from 0.60 to 0.65. This is because when s (sample size) is small, we cannot get a good model A()A( D) and A(∖f)A( D D_f) as input into L2UL. When sample size s increases, we can have a better model A()A( D) and A(∖f)A( D D_f), so that a good U and unlearned model U(D,Df,A(D))U(D,D_f,A(D)) with higher accuracy can be obtained on HIGGS. The results of m are shown in the third row of Figure 7 and Figure 8. L2UL is robust to m, as NUN_U is a simple model that can achieve good performance with a small dataset D. In a nutshell, L2UL is robust to the parameters ψ and m. Compared to the parameters of the L2UL model and the parameters of IDK, L2UL requires stronger data as input. L2UL learns an unlearning function U, only if the input data D contains sufficient unlearning information 77 7 Sufficient unlearning information means a correct and well-trained model A()A( D) and a correct and well-unlearned model A(∖f)A( D D_f)., can a correct and good unlearning function be learned. 6. Complexity Analysis Table 4. Factor of time complexity (D-or: D-oriented, DfD_f-or: DfD_f-oriented, A(D)A(D)-or: A(D)A(D)-oriented). U D-or DfD_f-or A(D)A(D)-or L2UL Factor A&DA\&D DfD_f Df&θD_f\&\ θ DfD_f Table 5. Results of unlearning ResNet-18 on CIFAR-10 dataset. 10 100 1000 3000 ToW(↑ ) Retrain 1.00 1.00 1.00 1.00 ADV+IMP 0.32 0.13 0.01 0.01 RUM 0.60 0.80 0.83 0.83 L2UL(Ours) 0.80 1.00 0.88 0.99 MIA gap(↓ ) Retrain 0.00 0.00 0.00 0.00 ADV+IMP 0.30 0.13 0.17 0.14 RUM 0.30 0.13 0.15 0.12 L2UL(Ours) 0.30 0.13 0.17 0.14 Time(↓ ) Origin (A(DCLOSEA(D) 1044.9 - - - Retrain 1020.5 874.4 783.2 702.7 ADV+IMP 28.4 78.7 1665.5 12088.2 RUM 13.6 13.6 13.2 12.8 L2UL(Ours) 0.5 0.5 0.5 0.6 We show the factor that affects the time complexity of unlearning algorithms as follows (time complexity of the machine learning model A is denoted as (A)T(A)) : (1) D-oriented unlearning algorithm, such as SISA (1), has a time complexity of (A(|D|/k))T(A(|D|/k)), where k is the number of shards. Because SISA retrains A on some shards, the time complexity is limited by A and |D||D|. (2) DfD_f-oriented unlearning algorithm, such as (12), considers the influence of DfD_f, whose time complexity is (|Df|d3)O(|D_f|d^3), which is limited by DfD_f. (3) A(D)A(D)-oriented unlearning algorithm, such as (25), considers how to update parameters. Its time complexity is (|Df|d2+d|θ|)O(|D_f|d^2+d^|θ|) and is limited by DfD_f and |θ||θ|. The factors of time complexity are summarized in Table 4. Intuitively, DfD_f is unavoidable because we at least need to know the samples we need to forget. L2UL is also subject to this restriction. Given dataset D, we give the time complexity and space complexity of L2UL as follows: * Time complexity of preprocessing is (ψt|D|d+m(A(s)))O(ψ t|D|d+mT(A(s))) * Time complexity of training U is (U(m))T(U(m)). * Time complexity of unlearning DfD_f from U is (ψt|Df|d)O(ψ t|D_f|d). * Space complexity of U is (ψtd)O(ψ td). When unlearning DfD_f, L2UL needs to calculate the feature map of DfD_f first. But we can store the feature map of every point from D in advance. In this way, when unlearning DfD_f, we only need to select the corresponding feature maps and take the mean. Although the time complexity still depends on DfD_f, the operation of just taking mean is very fast. As a trade-off, space complexity goes up to (ψt|D|)O(ψ t|D|). 7. Unlearn large-scale parameters: ResNet This paper and the method L2UL we propose primarily focus on the unlearning of simple models (with fewer parameters) trained on large amounts of training data. As demonstrated in the previous sections, the experimental results confirm that L2UL not only maintains high fidelity to the retrained model but also achieves significant gains in computational efficiency. Furthermore, in this section, we verify the effectiveness of our proposed L2UL on a model with a larger parameter size. We tested the performance of the unlearning ResNet-18 model on the CIFAR-10 dataset and compared it with: ADV+IMP: (5) Misclassify each instance outside of its original prediction, or relabel the instance to a different label. At the same time, use adversarial examples to overcome forgetting at the representation level and use weight importance indicators to accurately locate network parameters that propagate unnecessary information to reduce the time required for forgetting. RUM: (42) The forgotten set is refined into homogenized subsets based on different features. The meta-algorithm employs the existing algorithms to forget each subset, ultimately providing a model that has forgotten the entire forgotten set. We train ResNet-18 for 50 epochs on CIFAR-10 with a learning rate of 0.0001. And we report two evaluation metrics, ToW and MIA (Membership Inference Attack) gap, that are used in RUM (42). The results are shown in Table 5. We highlight the following three key observations: (a) L2UL achieves the highest ToW scores when forgetting 10, 100, 1000, and 3000 samples. (b) L2UL and the comparison algorithms have a closed low MIA gap. (c) L2UL only takes a very short time to complete the unlearning. In summary, our experimental results demonstrate not only the scalability of L2UL on large-scale data but also its scalability in terms of model parameters. L2UL maintains the performance of the model after forgetting data, and, more importantly, its unlearning efficiency is far superior to existing algorithms. 8. Disscussion 8.1. How to learn U when the model cannot be retrained? It works well for L2UL to use Equation 2 as its loss function only when the parameter of the retrained model A(∖f)A( D D_f) can be obtained. However, the inability to retrain the model is a common challenge faced by machine unlearning. For example, we can’t retrain the model on streaming data. In this case, L2UL can no longer be used to generate training data D, and the loss function 2 cannot be used to train NUN_U. How to find a new loss function when the model cannot be retrained is a challenge. 8.2. Retrain U after unlearning A It is worth emphasizing that when we employ U to unlearn model, user’s information DfD_f will be included in U. Therefore, we must unlearn U after we use U to unlearn A. Theorem 8.1. When sampling m subsets D from D. The probability that x is sampled at least k0k_0 times is (k≥k0)=∑i=k0mCmk(s|D|)i(1−s|D|)m−i.P(k≥ k_0)= _i=k_0^mC_m^k( s|D|)^i(1- s|D|)^m-i. In our experiments, (k≥2)<0.004P(k≥ 2)<0.004 on HIGGS, which means the probability that the data x,y∈D\x,y\∈ D is used to train U is very low. So we barely need to retrain U. Even if we need, the time cost is affordable, as it only takes 5.2 seconds to retrain U on HIGGS. 8.3. Subdivide the machine unlearning task Current machine unlearning methods often treat forgetting tasks as a monolithic problem, neglecting the inherent diversity in model complexity and data scale. We argue for a more nuanced, task-specific approach. Since a forgetting task is defined by both the model and the training data, we propose a taxonomy that categorizes machine unlearning into four distinct scenarios based on model size and dataset size: 1) Simple models on small datasets. This represents the most straightforward scenario, where retraining from scratch remains computationally feasible and is often the preferred solution. 2) Complex models on small datasets. Although the training data is limited, the increased model complexity significantly raises the cost of retraining, making it an impractical choice. 3) Simple models on large datasets. This is the focal scenario of our work. Because the model itself is lightweight, applying sophisticated unlearning techniques designed for complex architectures would be unnecessarily resource-intensive—essentially overkill. Instead, a lightweight method, such as the one we propose, is sufficient to achieve effective forgetting. 4) Complex models on large datasets (e.g., large language models). This constitutes the most challenging scenario, demanding advanced and scalable unlearning strategies. The key advantage of this taxonomy lies in its strategic flexibility: by matching the unlearning method to the specific complexity of the task, we can achieve efficient data removal without incurring prohibitive computational costs, ensuring we neither "use a sledgehammer to crack a nut" nor deploy inadequate solutions for demanding scenarios. 9. Conclusion In this paper, we propose Learning-to-UnLearn (L2UL), to the best of our knowledge, the first framework to address machine unlearning by learning the unlearning behaviors rather than manually designing complex functions. By prioritizing a distribution-oriented perspective, L2UL achieves a remarkably streamlined yet potent unlearning capability. Focusing on scenarios where the model is simple but the data training dataset is large, we utilized Logistic Regression and Multilayer Perceptrons to demonstrate the ability of L2UL to handle massive datasets with exceptional speed and no loss in accuracy. Both theoretical and empirical results prove that L2UL significantly outperforms existing methods in efficiency. Furthermore, tests on contaminated models and sensitivity analyses confirm the effectiveness and stability of L2UL across various configurations. Finally, successful validation on ResNet underscores the scalability of L2UL to more complex models, offering a robust and efficient solution for modern privacy requirements. Appendix A Proofs A.1. Proof of Theorem 4.5 Theorem 4.5 L(fU(A(D),D,Df))≤L(fA(D∖Df))+2Rϵ,L(f_U(A(D),D,D_f))≤ L(f_A(D D_f))+2R ε, where R is the radius of the dataset X (∀x∈X,‖x‖≤R∀ x∈ X,||x||≤ R), and ϵε is the generalization bound (MSE) of the unlearning model U. Proof. Denote U(A(D),D,Df)−A(D∖Df)U(A(D),D,D_f)-A(D D_f) as e, then ‖e‖≤ϵ||e||≤ ε. By Taylor Expansion, we approximately have L(fU(A(D),D,Df))−L(fA(D∖Df)) L(f_U(A(D),D,D_f))-L(f_A(D D_f)) = = ∂L(fA(D∖Df))∂A(D∖Df)⊤e+o(‖e‖), ∂ L(f_A(D D_f))∂ A(D D_f) e+o(\|e\|), where lim‖e‖→0o(‖e‖)‖e‖=0 _\|e\|→ 0 o(\|e\|)\|e\|=0. Then by Cauchy–Schwarz inequality and convexity of norm function, we have L(fU(A(D),D,Df))−L(fA(D∖Df)) L(f_U(A(D),D,D_f))-L(f_A(D D_f)) = = Ex,y[(fA(D∖Df)(x)−y)x⊤]e+o(‖e‖) E_x,y[(f_A(D D_f)(x)-y)x ]e+o(\|e\|) ≤ ≤ ‖Ex,y[(fA(D∖Df)(x)−y)x]‖⋅‖e‖+o(‖e‖) ||E_x,y[(f_A(D D_f)(x)-y)x]||·||e||+o(\|e\|) ≤ ≤ Ex,y[‖(fA(D∖Df)(x)−y)x‖]⋅‖e‖+o(‖e‖) E_x,y[||(f_A(D D_f)(x)-y)x||]·||e||+o(\|e\|) ≤ ≤ 2Rϵ+o(ϵ). 2R ε+o( ε). ∎ Here we refer U(A(D),D,Df)U(A(D),D,D_f) and A(D∖Df)A(D D_f) as the parameters of the unlearned model and retrained model, respectively. A.2. Proof of Theorem 4.6 Theorem 4.6 L(fU(A(D),D,Df))≤L(fA(D∖Df))+Aϵ,L(f_U(A(D),D,D_f))≤ L(f_A(D D_f))+A ε, where A is a finite scalar determined by the MLP structure and ϵε is the generalization bound (MSE) of the unlearning model U. Proof. For the (N+1N+1)-layer MLP model (the 0-th layer is the input layer, the (N+1N+1)-th layer is the output layer), we use softmax activation for the output layer, and sigmoid activation function for the rest of the layers. We denote the number of neurons in the i-th layer as nin_i, then the weight between the (i−1)(i-1)-th layer and i-th layer are WiW_i of shape (nk,nk−1)(n_k,n_k-1) and BiB_i of shape (ni,1)(n_i,1). For a finite test set X, |X|=m|X|=m, we denote it as A0A_0. Then, the forward propagation scheme is described as follows: Zk=WkAk+Bk,Ak=fk(Zk)Z_k=W_kA_k+B_k,\ A_k=f_k(Z_k) for k∈1,2,…,N+1k∈\1,2,...,N+1\, where fk(⋅)f_k(·) is the activation function we use in the k-th layer (softmax for the output layer, sigmoid for the rest layers). For the back propagation, when using Cross-Entropy Loss, we have dZN+1=AN+1−Y,dWN+1=1mdZN+1AN⊤,d\;Z_N+1=A_N+1-Y,\ d\;W_N+1= 1md\;Z_N+1A_N , dBN+1=1mdZN+11(m,1),d\;B_N+1= 1md\;Z_N+11_(m,1), and (3) dZk=Wk+1TdZk+1⊙fk′(Zk),d\;Z_k=W_k+1^Td\;Z_k+1 f _k(Z_k), dWk=1mdZkAk−1⊤,dBk=1mdZk1(m,1)d\;W_k= 1md\;Z_kA _k-1,\ d\;B_k= 1md\;Z_k1_(m,1) for k∈1,2,…,N,N+1k∈\1,2,...,N,N+1\, where d means the derivative of the Cross Entropy Loss, 1(m,1)1_(m,1) is a column vector of size m, with each entry =1=1, and ⊙ is the element-wise multiplication. Denote A(D∖Df)A(D D_f) as Θ and U(A(D),D,Df)U(A(D),D,D_f) as Θ . We have (4) Θ=[WB], = bmatrixW\\ B bmatrix, where W,BW,B are the concatenation of each layer’s row-flatten parameter, i.e. W=[rft(W1),…,rft(WN+1)]⊤W= [rft(W_1),...,rft(W_N+1) ] ,W=[rft(B1),…,rft(BN+1)]⊤W= [rft(B_1),...,rft(B_N+1) ] , where rtf(⋅)rtf(·) is the operator that flatten a matrix into a row vector. The same notation goes for Θ . As in 4.5, we only need to show that dΘd\; is bounded, which can be achieved from the bound for dBd\;B and d;Wd\ ;W, since ‖dΘ‖2=‖dW‖2+‖dB‖2\|d\; \|^2=\|d\;W\|^2+\|d\;B\|^2. For WN+1W_N+1, we have ‖drft(WN+1)‖2=‖(AN+1−Y)AN⊤m‖F2≤4nNnN+1.\|d\;rft(W_N+1)\|^2=\| (A_N+1-Y)A_N m\|_F^2≤ 4n_Nn_N+1. since the absolute value of each entry in AN+1,AN,YA_N+1,A_N,Y is smaller than 1. For the bound on drftWk(k≠N+1)d\;rftW_k(k≠ N+1), we first need to get the bound RkR_k for drftZkd\;rftZ_k. We know that ‖drft(ZN+1)‖∞≤RN+1=2.\|d\;rft(Z_N+1)\|_∞≤ R_N+1=2. Then according to Equation 3, we have ‖drftZk‖∞≤R4nk+1Rk+1≜Rk,\|d\;rftZ_k\|_∞≤ R4n_k+1R_k+1 R_k, under the assumption ‖W‖∞≤R\|W\|_∞≤ R. The iterative form of RkR_k can be expressed equivalently as Rk=(R4)N+1−k∏j=k+1N+1nj.R_k=( R4)^N+1-k _j=k+1^N+1n_j. Coming back to WkW_k, for k≤Nk≤ N, we have ‖drft(Wk)‖2≤Rk2nknk−1.\|d\;rft(W_k)\|^2≤ R_k^2n_kn_k-1. Finally, we have ‖drftW‖2 \|d\;rftW\|^2 =∑k=1N+1‖drftWk‖2 = _k=1^N+1\|d\;rftW_k\|^2 ≤4nNnN+1+∑k=1N(R216)N+1−k∏j=k−1N+1nj2nknk+1⏟≜AW. ≤ 4n_Nn_N+1+ _k=1^N( R^216)^N+1-k _j=k-1^N+1n_j^2n_kn_k+1_ A_W. Similarly, we have (5) ‖drft(Wk)‖2≤Rk2nknk−1. \|d\;rft(W_k)\|^2≤ R_k^2n_kn_k-1. Finally, we have (6) ‖drftB‖2 \|d\;rftB\|^2 =∑k=1N+1‖drftBk‖2 = _k=1^N+1\|d\;rftB_k\|^2 ≤4nN+1+∑k=1N(R216)N+1−k∏j=kN+1nj2nk⏟≜AB. ≤ 4n_N+1+ _k=1^N( R^216)^N+1-k _j=k^N+1n_j^2n_k_ A_B. Then ‖dΘ‖≤Aw2+AB2≜A\|d\; \|≤ A_w^2+A_B^2 A. ∎ A.3. Proof of Complexity * Time complexity of preprocessing is (ψt|D|d+m(A(s)))O(ψ t|D|d+mT(A(s))) Proof. Preprocessing first requires a total time of (ψt|D|dCLOSEO(ψ t|D|d to produce the feature map (line 2 in Algorithm 1), and requires (ψts+(A(s))CLOSEO(ψ ts+T(A(s)) for each loop, So the time complexity of Algorithm 1 is (ψt|D|d+m(A(s)))O(ψ t|D|d+mT(A(s))). ∎ * Time complexity of training U is (U(m))T(U(m)). Proof. The time complexity of Train U is using the L2UL to train NUN_U on the m data points, so the time complexity is (U(m))T(U(m)). ∎ * Time complexity of unlearning DfD_f from U is (ψt|Df|d)O(ψ t|D_f|d). Proof. Algorithm 2 requires time of (ψt|Df|d)O(ψ t|D_f|d) to get the feature map (line 1). and (ψt|Df|)O(ψ t|D_f|) to get μf _f (line 2). Therefore, the time complexity to unlearn DfD_f from the U is (ψt|Df|d)O(ψ t|D_f|d). ∎ * Space complexity of U is (ψtd)O(ψ td). Proof. we need (ψt)O(ψ t) to store the μD _D, and (ψtd)O(ψ td) to store the seeds for partition in isolation kernel. Therefore, the space complexity of U is (ψtd)O(ψ td). ∎ A.4. Proof of Theorem 7.1 Theorem 7.1 We sample m subsets D from D. The probability that sample x is sampled at least k0k_0 times is: (k≥k0)=∑i=k0mCmk(s|D|)i(1−s|D|)m−i.P(k≥ k_0)= _i=k_0^mC_m^k( s|D|)^i(1- s|D|)^m-i. Proof. Sampling s points from |D||D| points, The probability that point x is sampled is (x)=s|D|P(x)= s|D|. Do this sampling m times, the probability that point x is sampled k times is: P(k)=Cmk(s|D|)k(1−s|D|)m−k.P(k)=C_m^k( s|D|)^k(1- s|D|)^m-k. So the probability that sample x is sampled at least k0k_0 times is: (k≥k0)=∑i=k0mCmk(s|D|)i(1−s|D|)m−i.P(k≥ k_0)= _i=k_0^mC_m^k( s|D|)^i(1- s|D|)^m-i. ∎ References Bourtoule et al. (2021) L. Bourtoule, V. Chandrasekaran, C. A. Choquette-Choo, H. Jia, A. Travers, B. Zhang, D. Lie, and N. Papernot Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), p. 141–159. Cited by: §1, §3, item 2, item 1. Brophy and Lowd (2021) J. Brophy and D. Lowd Machine unlearning for random forests. In International Conference on Machine Learning, p. 1092–1104. Cited by: §3. Cao et al. (2025) Y. Cao, H. Xiang, H. Zhang, Y. Zhu, and K. M. Ting Anomaly detection based on isolation mechanisms: a survey. Machine Intelligence Research 22 (5), p. 849–865. Cited by: §4.1. Cao and Yang (2015) Y. Cao and J. Yang Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, p. 463–480. Cited by: §1, §3. Cha et al. (2024) S. Cha, S. Cho, D. Hwang, H. Lee, T. Moon, and M. Lee Learning to unlearn: instance-wise unlearning for pre-trained classifiers. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, p. 11186–11194. Cited by: §1, §7. Chen et al. (2022) T. Chen, X. Chen, W. Chen, H. Heaton, J. Liu, Z. Wang, and W. Yin Learning to optimize: a primer and a benchmark. Journal of Machine Learning Research 23 (189), p. 1–59. Cited by: §1. Chundawat et al. (2023a) V. S. Chundawat, A. K. Tarun, M. Mandal, and M. Kankanhalli Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, p. 7210–7217. Cited by: §5.3. Chundawat et al. (2023b) V. S. Chundawat, A. K. Tarun, M. Mandal, and M. Kankanhalli Zero-shot machine unlearning. IEEE Transactions on Information Forensics and Security. Cited by: §3. Fu et al. (2021) S. Fu, F. He, and D. Tao Knowledge removal in sampling-based bayesian inference. In International Conference on Learning Representations, Cited by: §3. Ginart et al. (2019) A. Ginart, M. Guan, G. Valiant, and J. Y. Zou Making ai forget you: data deletion in machine learning. Advances in neural information processing systems 32. Cited by: §3. Golatkar et al. (2020) A. Golatkar, A. Achille, and S. Soatto Eternal sunshine of the spotless net: selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 9304–9312. Cited by: §5.3. Guo et al. (2020) C. Guo, T. Goldstein, A. Hannun, and L. Van Der Maaten Certified data removal from machine learning models. In International Conference on Machine Learning, p. 3832–3842. Cited by: §3, item 2. Gupta et al. (2021) V. Gupta, C. Jung, S. Neel, A. Roth, S. Sharifi-Malvajerdi, and C. Waites Adaptive machine unlearning. Advances in Neural Information Processing Systems 34, p. 16319–16330. Cited by: §3. Kumar et al. (2012) S. Kumar, M. Mohri, and A. Talwalkar Sampling methods for the nyström method. The Journal of Machine Learning Research 13 (1), p. 981–1006. Cited by: §4.1. Lin et al. (2023) H. Lin, J. W. Chung, Y. Lao, and W. Zhao Machine unlearning in gradient boosting decision trees. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 1374–1383. Cited by: §3. Liu et al. (2024) J. Liu, P. Ram, Y. Yao, G. Liu, Y. Liu, P. SHARMA, S. Liu, et al. Model sparsity can simplify machine unlearning. Advances in Neural Information Processing Systems 36. Cited by: §5.3. Ma et al. (2022a) Z. Ma, Y. Liu, X. Liu, J. Liu, J. Ma, and K. Ren Learn to forget: machine unlearning via neuron masking. IEEE Transactions on Dependable and Secure Computing 20 (4), p. 3194–3207. Cited by: §1. Ma et al. (2022b) Z. Ma, Y. Liu, X. Liu, J. Liu, J. Ma, and K. Ren Learn to forget: machine unlearning via neuron masking. IEEE Transactions on Dependable and Secure Computing. Cited by: §3. Musco and Musco (2017) C. Musco and C. Musco Recursive sampling for the nystrom method. Advances in neural information processing systems 30. Cited by: §4.1. Neel et al. (2021) S. Neel, A. Roth, and S. Sharifi-Malvajerdi Descent-to-delete: gradient-based methods for machine unlearning. In Algorithmic Learning Theory, p. 931–962. Cited by: §3. Nguyen et al. (2022a) Q. P. Nguyen, R. Oikawa, D. M. Divakaran, M. C. Chan, and B. K. H. Low Markov chain monte carlo-based machine unlearning: unlearning what needs to be forgotten. In Proceedings of the 2022 ACM on Asia Conference on Computer and Communications Security, p. 351–363. Cited by: §3. Nguyen et al. (2022b) T. T. Nguyen, T. T. Huynh, P. L. Nguyen, A. W. Liew, H. Yin, and Q. V. H. Nguyen A survey of machine unlearning. arXiv preprint arXiv:2209.02299. Cited by: §1. Qin et al. (2019) X. Qin, K. M. Ting, Y. Zhu, and V. Lee Nearest-neighbour-induced isolation similarity and its impact on density-based clustering. In Proceedings of the 33rd AAAI Conference on AI (AAAI 2019), AAAI Press, Cited by: §4.1. Rahimi and Recht (2007) A. Rahimi and B. Recht Random features for large-scale kernel machines. Advances in neural information processing systems 20. Cited by: §4.1. Sekhari et al. (2021) A. Sekhari, J. Acharya, G. Kamath, and A. T. Suresh Remember what you want to forget: algorithms for machine unlearning. Advances in Neural Information Processing Systems 34, p. 18075–18086. Cited by: §3, item 3. Smola et al. (2007) A. Smola, A. Gretton, L. Song, and B. Schölkopf A hilbert space embedding for distributions. In International conference on algorithmic learning theory, p. 13–31. Cited by: §4.1. Tang and Yao (2024) K. Tang and X. Yao Learn to optimize—a brief overview. National Science Review 11 (8), p. nwae132. Cited by: §1. Tarun et al. (2023a) A. K. Tarun, V. S. Chundawat, M. Mandal, and M. Kankanhalli Fast yet effective machine unlearning. IEEE Transactions on Neural Networks and Learning Systems. Cited by: §3, item 4. Tarun et al. (2023b) A. K. Tarun, V. S. Chundawat, M. Mandal, and M. Kankanhalli Deep regression unlearning. In International Conference on Machine Learning, p. 33921–33939. Cited by: §3. Taud and Mas (2018) H. Taud and J. Mas Multilayer perceptron (mlp). Geomatic approaches for modeling land change scenarios, p. 451–455. Cited by: §5.2. Ting et al. (2022) K. M. Ting, Z. Liu, H. Zhang, and Y. Zhu A new distributional treatment for time series and an anomaly detection investigation. Proceedings of the VLDB Endowment 15 (11), p. 2321–2333. Cited by: §4.1. Ting et al. (2020) K. M. Ting, B. Xu, T. Washio, and Z. Zhou Isolation distributional kernel: a new tool for kernel based anomaly detection. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, p. 198–206. Cited by: §4.1, §4.1, §5.4. Ting et al. (2018) K. M. Ting, Y. Zhu, and Z. Zhou Isolation kernel and its effect on svm. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, p. 2329–2337. Cited by: §4.1. Warnecke et al. (2023) A. Warnecke, L. Pirch, C. Wressnegger, and K. Rieck Machine unlearning of features and labels. Network and Distributed System Security (NDSS) Symposium. Cited by: §3. Williams and Seeger (2000) C. Williams and M. Seeger Using the nyström method to speed up kernel machines. Advances in neural information processing systems 13. Cited by: §4.1. Wu et al. (2022) G. Wu, M. Hashemi, and C. Srinivasa Puma: performance unchanged model augmentation for training data removal. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36, p. 8675–8682. Cited by: §3. Wu et al. (2020) Y. Wu, E. Dobriban, and S. Davidson Deltagrad: rapid retraining of machine learning models. In International Conference on Machine Learning, p. 10355–10366. Cited by: §3, item 3. Wu et al. (2023) Z. Wu, J. Zhu, Q. Li, and B. He DeltaBoost: gradient boosting decision trees with efficient machine unlearning. Proceedings of the ACM on Management of Data 1 (2), p. 1–26. Cited by: §3. Yang et al. (2012) T. Yang, Y. Li, M. Mahdavi, R. Jin, and Z. Zhou Nyström method vs random fourier features: a theoretical and empirical comparison. Advances in neural information processing systems 25. Cited by: §4.1. Zhang et al. (2025) H. Zhang, K. M. Ting, and Y. Zhu Kernel-bounded clustering: achieving the objective of spectral clustering without eigendecomposition. Artificial Intelligence, p. 104440. Cited by: §4.1. Zhang et al. (2023) H. Zhang, K. Zhang, K. M. Ting, and Y. Zhu Towards a persistence diagram that is robust to noise and varied densities. In Proceedings of the 40th International Conference on Machine Learning, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (Eds.), Proceedings of Machine Learning Research, Vol. 202, p. 41952–41972. Cited by: §4.1. Zhao et al. (2024) K. Zhao, M. Kurmanji, G. Bărbulescu, E. Triantafillou, and P. Triantafillou What makes unlearning hard and what to do about it. Advances in Neural Information Processing Systems 37, p. 12293–12333. Cited by: §7, §7.