Paper deep dive
KGA: A General Machine Unlearning Framework Based on Knowledge Gap Alignment
Lingzhi Wang, Tong Chen, Wei Yuan, Xingshan Zeng, Kam-Fai Wong, Hongzhi Yin
Models: BART, DistilBERT, LSTM, Transformer (encoder-decoder)
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/12/2026, 8:15:41 PM
Summary
The paper introduces KGA (Knowledge Gap Alignment), a machine unlearning framework designed for NLP tasks. Unlike previous methods that rely on gradient recovery or specific distribution assumptions, KGA aligns the 'knowledge gap'âthe difference in prediction distributions between models trained on different datasetsâto effectively remove information about specific training instances while maintaining model performance on remaining data.
Entities (7)
Relation Signals (4)
KGA â appliedto â NLP
confidence 95% · we first apply the unlearning method to various NLP tasks
KGA â evaluatedon â LEDGAR
confidence 95% · Experiments on large-scale datasets show that KGA yields comprehensive improvements
KGA â evaluatedon â IWSLT
confidence 95% · Experiments on large-scale datasets show that KGA yields comprehensive improvements
KGA â evaluatedon â PersonaChat
confidence 95% · Experiments on large-scale datasets show that KGA yields comprehensive improvements
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent legislation of the "right to be forgotten" has led to the interest in machine unlearning, where the learned models are endowed with the function to forget information about specific training instances as if they have never existed in the training set. Previous work mainly focuses on computer vision scenarios and largely ignores the essentials of unlearning in NLP field, where text data contains more explicit and sensitive personal information than images. In this paper, we propose a general unlearning framework called KGA to induce forgetfulness. Different from previous work that tries to recover gradients or forces models to perform close to one specific distribution, KGA maintains distribution differences (i.e., knowledge gap). This relaxes the distribution assumption. Furthermore, we first apply the unlearning method to various NLP tasks (i.e., classification, translation, response generation) and propose several unlearning evaluation metrics with pertinence. Experiments on large-scale datasets show that KGA yields comprehensive improvements over baselines, where extensive analyses further validate the effectiveness of KGA and provide insight into unlearning for NLP tasks.
Tags
Links
Trouble viewing inline? Open PDF directly â
Full Text
49,172 characters extracted from source content.
Expand or collapse full text
KGA: A General Machine Unlearning Framework Based on Knowledge Gap Alignment Lingzhi Wang 1,2 , Tong Chen 3 , Wei Yuan 3 , Xingshan Zeng 4 , Kam-Fai Wong 1,2 , Hongzhi Yin 3 1 The Chinese University of Hong Kong, Hong Kong, China 2 MoE Key Laboratory of High Confidence Software Technologies, China 3 School of Information Technology and Electrical Engineering, The University of Queensland 1,2 lzwang,kfwong@se.cuhk.edu.hk 3 tong.chen,w.yuan,h.yin1uq.edu.au, 4 zxshamson@gmail.com Abstract Recent legislation of the âright to be forgottenâ has led to the interest in machine unlearning, where the learned models are endowed with the function to forget information about spe- cific training instances as if they have never existed in the training set.Previous work mainly focuses on computer vision scenarios and largely ignores the essentials of unlearning in NLP field, where text data contains more ex- plicit and sensitive personal information than images. In this paper, we propose a general unlearning framework called KGA to induce forgetfulness. Different from previous work that tries to recover gradients or forces mod- els to perform close to one specific distribu- tion, KGA maintains distribution differences (i.e., knowledge gap). This relaxes the distribu- tion assumption. Furthermore, we first apply the unlearning method to various NLP tasks (i.e., classification, translation, response gen- eration) and propose several unlearning eval- uation metrics with pertinence. Experiments on large-scale datasets show that KGA yields comprehensive improvements over baselines, where extensive analyses further validate the effectiveness of KGA and provide insight into unlearning for NLP tasks 1 . 1 Introduction Nowadays, machine learning models are usually trained with large volumes of data collected from individual users. The individualsâ data is sensitive in nature as it may contain information such as personal addresses and medical records. Unknow- ingly, the trained model may intrude into usersâ privacy as its parameters encode personal informa- tion and its derivatives permanently. Therefore, Machine Unlearning (MU) (Romero et al., 2007; Karasuyama and Takeuchi, 2009; Cao and Yang, 2015) has attracted more and more interest in re- search and industry, which aims to facilitate the 1 The code is available athttps://github.com/ Lingzhi-WANG/KGAUnlearn. model to forget some specific data in training set while maintaining the performance of the existing model. Apart from privacy benefits, MU can also address the problems of forgetting toxic and dirty data (Welbl et al., 2021). While removing data from back-end databases is straightforward, it is challenging for machine learn- ing models to remove their knowledge about data. One intuitive way for unlearning is to retrain the model from scratch with the âto-be-forgottenâ data deleted from training set. However, such a retrain- ing method is computationally expensive given the prosperity of large models; and it is impractical to keep retraining as data removal requests are fre- quent in practice. Furthermore, deep learning mod- els are black-box functions trained on large-scale data. Since the relationship between the model weights and the data is unclear, it is difficult to know which parts of the weights should be revised in unlearning. Therefore, there is a pressing need to develop an efficient unlearning method. Existing research in machine unlearning mainly focuses on computer vision applications, e.g., im- age classification (Golatkar et al., 2020a,b; Mehta et al., 2022), and less attention has been paid to un- learning in the natural language processing (NLP) field, where text data contains more explicit and sensitive personal data (e.g., home address, phone number, social relationships, etc.) than images. Moreover, the current unlearning can only effi- ciently handle a small number of data removal re- quests (Bourtoule et al., 2021)while the removal requests in NLP applications may be hundreds. Be- sides, current gradient-computation-based unlearn- ing methods(Mehta et al., 2022) are difficult to be applied in the NLP generation models, which are usually based on Seq2Seq framework and contain complex attention mechanisms between words that are generated in different time stamps. Consid- ering the significance and challenges of unlearn- ing in NLP, we propose KGA â a generic ma- arXiv:2305.06535v1 [cs.CL] 11 May 2023 chine unlearning method based onKnowledgeGap Alignment, and apply KGA to NLP tasks. KGA is inspired by a general knowledge adap- tation work (Khan and Swaroop, 2021), where weights and function-space priors are adopted to re- construct the gradients of the model. Compared to Khan and Swaroop (2021) which is a generic solu- tion to adaptation tasks including data removal but difficult to scale up to complex neural networks, our method KGA focuses on data removal from the perspective of knowledge gap alignment and is easily generalizable to deep networks. The knowl- edge gap in this work is defined as the distance between the prediction distributions from two struc- turally identical models trained with different data. By aligning knowledge gaps, we force two sets of models behave similarly. Besides, unlike existing unlearning methods that can only handle a small set of removal requests (Bourtoule et al., 2021), hold strong assumptions on model output (Chundawat et al., 2022), or are inapplicable to complex genera- tion tasks (Mehta et al., 2022), KGA can efficiently handle a large number of removal requests with sustainable accuracy, and is easily compatible to various models and tasks with milder assumptions. Furthermore, we apply KGA to various NLP tasks (i.e., classification, translation and response generation) and customize text-specific evaluation metrics. The experimental results and further analy- ses from various aspects show that our KGA gener- ally performs better than baselines in terms of per- formance maintenance and unlearning efficiency, while maintaining consistency across different sce- narios and models. Interesting explorations on how the model translates German to English before and after unlearning are given to better validate and analyze the effectiveness of unlearning. In brief, the main contributions of this paper are: âąWe propose an unlearning solution (i.e., KGA) based on knowledge gap alignment for NLP tasks that can efficiently and effectively perform un- learning. âąExperiments on three large-scale datasets with newly formulated text-specific evaluation metrics validate the effectiveness of KGA. âąWe conduct extensive experiments and analyses to confirm the effectiveness of KGA unlearning across different scenarios. 2 Related Work The current unlearning research can be divided into two categories, exact unlearning and approximate unlearning. We briefly introduce them as follows. Exact Unlearning.Exact unlearning can ensure the effects of data to be deleted are removed from the model. Cao and Yang (2015) explores exact unlearning by the statistical query for Naive Bayes Classifiers and Ginart et al. (2019) studies deletion algorithms for k-means clustering, which cannot scale to deep neural networks which may have mil- lions of parameters. As for more recent efforts in neural model unlearning, Bourtoule et al. (2021) propose a general method called SISA to train the model by partitioning the original dataset into sev- eral non-overlapping shards first and then designing effective mechanisms to aggregate models trained with shards. When handling data deletion, this method only has to retrain the models trained with the affected shards. However, SISA-based methods are shown to be ineffective when the number of deleting queries is large, and we have to maintain the whole dataset during the training and unlearn- ing, which is impractical. Approximate Unlearning.The methods in this category try to make the model behave as closely as possible to the exact unlearned model. The popular- ity of approximate unlearning comes from the de- mand for more efficient and less costly unlearning, thus sacrificing exactness. Golatkar et al. (2020a); Guo et al. (2019); Koh and Liang (2017); Mehta et al. (2022) mainly handle an unlearning request by computing the model perturbation towards the regularized empirical risk on the remaining data. However, this approach needs to compute the Hes- sian on the training data and the gradient of the re- moval data, which is still time-consuming. (Chun- dawat et al., 2022) assumes that the models after unlearning should perform similarly to a randomly initialized model on the forgetting data, which is in- appropriate as the target of unlearning is to remove the effects of the forgetting data (acts as unseen data) rather than to make the model unable to han- dle forgetting data. However, existing knowledge adaptation methods either require strong assump- tions or perform poorly on neural-based models (Khan and Swaroop, 2021). Different from the aforementioned works, KGA does not force the model to perform on forgetting data close to one specific distribution but rather it maintains the distribution differences (i.e., knowl- edge gap) between two model pairs. This weakens the assumption as it is applicable to forgetting data in any distribution, thus also being suitable and applicable to more realistic scenarios while still ensuring the modelâs performance. 3 Notations and Definition Notations.We denoteZas an example space, i.e., the space of data instances or samples. Then, the set of all possible training datasets can be de- noted asZ â = 2 Z . The training data setDâ Z â is given as input. GivenD, we train an ML model from a hypothesis spaceH. The process of train- ing a model on data setDis enabled by a learning algorithm, denoted by a functionA:Z â â H. The trained model is denoted asA(D). Then we denote the unlearning mechanism as a functionU, which takes a training datasetDâ Z â , a forget setD f âD(containing data to be removed) and a modelA(D)as input, and returns an unlearned modelU(D,D f ,A(D))âH. Approximate Unlearning Definition. We then give one representative definition of approximate unlearning, specificallyâApproximate Unlearn- ingby following Guo et al. (2019). Given >0, an unlearning mechanismUperformsâcertified removal for a learning algorithmAifâT â H, DâZ â ,D f âD: e â †Pr(U(D,D f ,A(D))âT) Pr(A(D f )âT) â€e (1) and the goal of approximate unlearning can be con- cluded as forgetting the data to be forgotten while maintaining the performance. 4 Our KGA Framework KGA unlearning method is inspired by a general knowledge adaptation work (Khan and Swaroop, 2021), where weights and function-space priors are adopted to reconstruct the gradients of the model. Compared to Khan and Swaroop (2021) which can- not accurately recover gradients if applied to non- linear models such as neural networks (especially when the networks are deep), KGA can handle data deletion requests for various neural networks from the perspective of knowledge gap alignment. 4.1 KGA Framework The input to KGA can be divided into two parts: data and models. The input data consists of previ- ous training dataD, data to be forgottenD f , and a small set of extra dataD n to assist the unlearn- ing, whereD n â©D=â . Apart from data, we have modelA(D)as input, which is the original model trained with dataDthat needs unlearning (we abbreviate it asA D in the following parts of this paper). The output of KGA is a modelA â , whose parameters are initialized withA D and are further updated with our KGA unlearning mecha- nism to removeD f . To perform unlearning, we first train two models, A n andA f , based on dataD n andD f , respectively. The architectures ofA D ,A n , andA f should be the same.A n (A f ) can be trained with the combination ofD n (D f ) and a small fraction ofD r =D f or fine-tuned based on some pre-trained language models to ensure performance, as the data to be forgottenD f might be small in some scenarios. We reframe and summarize two goals to achieve the approximate unlearning defined in Sec. 3. They areGoal 1: Make our output modelA â âs behav- ior onD f similar to its behavior on any unseen data (i.e. data not used for training); andGoal 2: Maintaining the performance ofA â onD r . Knowledge Gap Alignment. Theknowledge gapin this work is defined as the distance between the prediction distributions from two models hav- ing the same architecture but trained with different data. By aligning two knowledge gaps, we make two sets of models perform similarly. To achieve Goal 1, the output distribution of our target modelA â on dataD f (noted asA â (D f )) is expected to be similar toA D (D n ), whereD n should be an external set toDbut with the similar distribution. As the instances inD n might have different labels and features fromD f , it is difficult to directly infer the output distributions ofA â (D f ) withA D (D n ). We thus turn to imitate the knowl- edge gap between two sets of models: A â = arg min A |dis (D n ) (A D ,A n )âdis (D f ) (A,A f )|(2) wheredis (D) (A 1 ,A 2 )indicates the difference of the output distributions between modelA 1 andA 2 on dataD, which can be evaluated by KL diver- gence, Bregman divergence, or any other distribu- tional distance measurements. SinceA n andA f are trained onD n andD f , respectively, we expect that the knowledge gap when feedingD f toA â andA f should be similar to feedingD n toA D andA n according to Eq. 2. This is under the assumption that a similar knowledge deficit can be observed when the same architecture handles the seen (i.e., used for training) and unseen data with a similar distribution. And we believe that a successful unlearning method should make the target modelA â handleD f as unseen data. For Goal 2, we maintain the ability of modelA â when processing the remaining data, i.e.,D r . We treat the original modelA D as a teacher and di- rectly minimize the distance of output distributions when feeding samples inD r toA â andA D . Objectives.In our implementation, we use KL- divergence to measure the distributional distances between the output of two models. Therefore, the knowledge gap alignment objective is defined as: L a = â (y,z)â(D f ,D n ) |KL[Pr (A â ) (y)||Pr (A f ) (y)] âKL[Pr (A D ) (z)||Pr (A n ) (z)]| (3) wherePr (A) (z)is the output distribution given inputzto modelA,KL(a|b)measures the KL di- vergence between distributionaandb.yandzare fromD n andD f , respectively. We randomly sam- ple pairs of instances(y,z)as a batch of updating to alleviate overfitting to some specific samples. The objective for maintaining performance on D r is another KL divergence measuring output distribution ofA â andA D onD r : L r = â xâD r KL[Pr (A â ) (x)||Pr (A D ) (x)](4) The two objectives are jointly optimized during unlearning to achieve Goal 1 and 2 simultaneously. Therefore, the final objective is defined as: L=L a +α·L r (5) To improve unlearning efficiency, we need to find the earliest time when the modelA â achieves the desired performance during unlearning. How- ever, different from traditional machine learning algorithms, it is hard for us to find a suitable valida- tion set to validate the performance, asD f is also included in the training process. To handle this, we use a hyper-parameterÏ(0< Ï <1) to control the training. Specifically, we will first evaluate the aver- age knowledge gap betweendis (D n ) (A D ,A n ) and dis (D f ) (A D ,A f ) (A D should be the initialization ofA â ) before training, noted asG. The training stops if the corresponding average knowledge gap achievesÏ·G. We summarize KGA in Alg. 1. Algorithm 1KGA Unlearning Input:dataD,D f ,D n , trained modelA D , thresholdÏ Output:unlearned modelA â Train modelA f based onD f , modelA n based onD n Compute initial gapG InitializeA â withA D forstep in 1 to MAX_STEPdo Randomly sample a batch size of(y,z)from(D f ,D n ) ComputeL a based on Eq. 3 forinner_step in 1 to INNER_STEPdo Sample a batch size of samplexfromD r =D f ComputeL r based on Eq. 4 end for Update parameters ofA â according toL a +α·L r ifstep%VALID_STEP == 0then Compute current gapG â ifG â â€Ï·Gthen break.End of Training end if end if end for 4.2 KGAâs Applications in NLP Tasks We do not constrain the format of modelA(·)as our proposed unlearning method is generic and can be applied to various of neural network architectures. We choose three NLP tasks (i.e., text classification, machine translation, and response generation) to show the effectiveness of our unlearning method. Text Classification.The text classification tasks take the text sentences as input and output a proba- bility distribution over the predefined classes. We follow Mehta et al. (2022) and finetune a pretrained model DistilBERT (Sanh et al., 2019) for the text classification. A DistilBERT is a dis- tillation version of BERT (Devlin et al., 2019) model that contains multiple transformer encoder layers to extract features. Its input is formulated asw c = [[CLS];w 1 ;w 2 ;..;w |C| ]. The output rep- resentation of the [CLS] token is further fed into a classifier to derive the probability for each class. Machine Translation.The machine translation tasks take a sentence in one language as input and output the corresponding translation in another lan- guage. We follow the general transformer-based encoder-decoder framework, where the encoder summarizes the source sentences and the decoder will generate the target sentences based on source representations in an autoregressive manner. Apart from transformer, we also validate the ef- fectiveness of our unlearning method in other archi- tecture including LSTM and pretrained language model BART (Lewis et al., 2020). LEDGARIWSLTPersonaChat Taskclassification generationgeneration # of instances110,156168,90581,032 Avg length of source108.919.4142.1 Avg length of target -20.611.9 # of labels13-- Table 1:Statistics of LEDGAR, IWSLT and Per- sonaChat datasets.Avg lengthrefers to average to- ken number of the source (input) or target (output) se- quence. Response Generation.Both the response gener- ation and machine translation are generation tasks, whose target is to generate texts according to the given source content. In response generation, the given source content is the conversation between two talkers and it is expected to predict the content of the next response. The model for generation is similar to that of machine translation, and we concatenate the utterances in context as input. 5 Experimental Setup Datasets. We do experiments on three datasets, LEDGAR (Tuggener et al., 2020), IWSLT14 German-English (Cettolo et al., 2014) (henceforth IWSLT) and PersonaChat (Zhang et al., 2018). LEDGAR is a multi-label text classification dataset of legal provisions in contracts, and we employ a prototypical subset of LEDGAR by following Mehta et al. (2022). IWSLT is from a popular trans- lation campaign consisting of various translation di- rections and we choose the representative German- English direction. PersonaChat is a crowd-sourced dataset. It consists of turn-based dialogues that are based on given persona information. We use the official train/valid/test splits for experiments on all three datasets. Statistics of these datasets are listed in Table 1. Evaluation Metrics. For each dataset, we report one representative task-related score (Micro F1 for LEDGAR, BLEU4 2 for IWSLT and PPL for Per- sonaChat) with additional unlearning evaluation metrics which are introduced as below. JensenâShannon Divergence(JSD): Given two distributionsp(x)andq(x),JSD(p(x),q(x)) = 0.5âKL(p(x)||q(x)) + 0.5âKL(q(x)||p(x)). Language model Probability Distance(LPD): Given two language probabilities (i.e., the perplex- ity of target sentences produced by each model)x andy,LPD(x||y) =|xây|/y. 2 sacrebleu (https://github.com/mjpost/sacrebleu). Proportion of instances with Decreased Lan- guage model Probability(PDLP): It calculates the percentage of the instances whose language model probability has dropped after unlearning. Parameter Setting. For LEDGAR, we finetune DistilBERT for experiments. For IWSLT and Per- sonaChat, we both use a general encoder-decoder transformer architecture. We use Adam (Kingma and Ba, 2015) optimizer followed by the inverse square root learning rate scheduler for model train- ing. During KGA unlearning, we maintain 16 batch size and 5e-5 learning rate for all three datasets, and we setαin Eq. 5 as 0.1. For more parameter and training details, please refer to Appendix A. Comparisons.We compare the performance of our KGA method on test set and forget set with the ORIGINALmodel, two exact unlearning methods (i.e.,RETRAINandSISA(Bourtoule et al., 2021)) and two approximate methods,LCODEC(Mehta et al., 2022) andBADTEACHER(Chundawat et al., 2022). We introduce them as follows: ORIGINAL: the original model trained on the com- plete training setDwithout any forgetting. RETRAIN: It retrains the model with the retain dataD r (D r =D f ). SISA(Bourtoule et al., 2021): It first divides the dataset into several non-overlapping shards, and then aggregates outputs of the models trained with different shards. When dealing with data deletion, it only retrains the models trained with the affected shards and then aggregates. In our experiments, we randomly divide the training set into 5 shards. LCODEC (Mehta et al., 2022): Itâs in line with Hessain unlearning (updating the model weights based on the Hessian of the loss function) and iden- tifies a subset of model parameters to reduce the computation cost. It is applied in classification and might need modification when used in generation. BADT(Chundawat et al., 2022): It forces the un- learning model to perform as close as a randomly initialized model on the forget setD f and maintain the performance on the remaining dataD r . 6 Experimental Results In this section, we first compare the main unlearn- ing scores of KGA and baselines in §6.1. Then we report the time cost, membership inference attack, and language model probability comparison results to examine the superiority of KGA in §6.2. After Test SetForget Set Models LEDGARIWSLTPersonaChat LEDGARIWSLTPersonaChat F1 JSDâBL4 LPDâPPLâLPDâF1 JSDâBL4 LPDâPPLâLPDâ ORIGINAL96.1â29.0â30.7â98.2â47.2â15.6â RETRAIN96.2â28.6â30.8â95.5â31.5â29.5â Exact SISA(Bourtoule et al., 2021)95.5 0.08 21.3 0.8544.20.5294.60.0521.60.8043.10.56 Approximate LCODEC(Mehta et al., 2022)95.8 0.05â99.30.06â BADT(Chundawat et al., 2022)96.0 0.0328.1 0.3032.70.2517.1 3.69 0.00 1.9e 3 5.8e 4 4.3e 3 KGA96.00.0628.4 0.2832.10.2096.40.05 29.40.9129.40.52 Table 2: Main comparison results (in %) of unlearning on three datasets. JSD and LPD scores here are calculated between RETRAINand corresponding models. The best results (limited to comparisons in Exact and Approximate settings) are highlighted inboldfor each column. that, we delve into the effect of unlearning on NLP tasks in §6.3. More analyses are discussed in §6.4. 6.1 Main Comparison Results We explore the representative scores on both test and forget sets to examine the following two ques- tions: (i) How well do the unlearned models main- tain the performance on test set? (i) How does the performance change on forget set that was once part of the original training set? We report the cor- responding scores on Table 2, and we can draw the following observation. âąOur unlearning method can better maintain the performance on test set.It can be seen that KGA shows better F1, BLEU4, and PPL on three datasets, respectively, compared to other unlearn- ing baselines, regardless of exact or approximate method. This shows one of the superiority of KGA over other methods. âą The performance and prediction distribution of our KGA unlearned model on forget set are closer toRETRAINmodel.We can see that on forget set, KGA method gets a closer F1 (BLEU4 and PPL) score toRETRAINmodel and maintains a smaller JSD (LPD) score, which means the output distri- bution of instances on forget set is also closer to RETRAINmodel. It indicates that KGA achieves the best forgetting effect among all baselines ac- cording to the definition in Eq. 1. âąForgetting the data from original model does not mean the unlearned model can not handle these instances at all.We can find that the performance ofRETRAINon forget set drops compared toORIG- INALmodel but still shows promising performance (close to the results on test set). This is in line with our assumption that the performance of successful 02000400060008000 KGA BadT LCODEC SISA ReTrain (a): Run Time (in sec) ModelsF1FNR ORIGINAL87.70.13 RETRAIN70.90.21 SISA71.00.23 BADT84.10.13 KGA75.60.18 (b): MiA Results (in %) Figure 1: (a): Unlearning time (in seconds) needed for each method on LEDGAR when deleting 100 instances. (b): Membership Inference Attack (in %) on IWSLT. unlearned models on forget set should be similar to unseen data (e.g., test set). Our KGA methodâs performance is consistent withRETRAIN, while BADT completely loses the ability to classify and generate, which does not satisfy the definition. 6.2 More on the Superiority of KGA In this subsection, we examine the efficiency (i.e., time cost) and effect (i.e., membership attack and language model probability check) of unlearning. Time Cost. We report the time cost of unlearn- ing models in Fig. 1(a). We can see that though retraining and exact unlearning methods (i.e.,SISA) can guarantee perfect unlearning, the time cost of them exceeds other approximate unlearning meth- ods (i.e., LCODEC, BadT, KGA) a lot. Membership Inference Attack. (MiA) MiA in the machine learning setting emerges when an ad- versary aims to find out whether the target data instance is used to train the model or not. We fol- low Salem et al. (2018); Golatkar et al. (2020b) to do a black-box MiA where the adversary can only get access to the model output distribution. We use MiA on IWSLT dataset as an example. We first Models Test SetForget Set IWSLTPersonaChat IWSLTPersonaChat RETRAIN51.0(-)48.7(-)96.0(-)96.0(-) SISA77.9(â26.9) 80.0(â31.3)100(â4.0) 100(â4.0) BADT72.2(â21.2) 71.9(â23.2)100(â4.0) 100(â4.0) KGA58.4(â7.4)70.0(â21.3)94.0(â2.0) 98.7(â2.7) Table 3:Proportion of Decreased Language model Probability (PDLP) comparison results on IWSLT and PersonaChat datasets. The numbers in parentheses re- fer to the difference in performance from RETRAIN model. train a shallow translation model with data from the same distribution as the original training set (we simplify it to using 30% instances of the original training set in practice). The data in the training set of shallow model is labeled as â1â and other unseen data (i.e., the rest of the original training set) is la- beled as â0â. Then we train an attacker model with the above â1/0â labeled data using output distribu- tions of the trained shallow model as input. After that, we feed the attacker model with the output of unlearned models (i.e.,RETRAIN, KGA, etc.) and check the MiA results. We report the MiA results in Fig. 1(b), where a higher F1 score and lower False Negative Rate (FNR) indicate the attacker can better infer the membership of instances. We can see that the at- tacker performs best on theORIGINALand per- forms worse after unlearning, as desired. Among the unlearned models, we can also find that attacker can not infer the membership well afterexact un- learning(i.e.,RETRAINandSISA). As anapproxi- mate unlearningmethod, KGAâs results are close to exact unlearning, which shows its effectiveness. Decreased Language Model Probability Com- parison.Apart from the language model distance we report in §6.1, we also evaluate a new unlearn- ing evaluation score for generation tasks, namely, Proportion of Decreased Language model Proba- bility (PDLP) compared to the original model. De- creased language model probability of ground truth target sequence means that the unlearned model tends not to generate the sentences to be forgotten, which is consistent with the goal of unlearning. We report the PDLP comparison results of both test and forget sets in Table 3. From the results ofRE- TRAINmodel, we can see that the instances in test set have a steady fluctuation (i.e., about 50% PDLP) afterRETRAINunlearning while the instances in forget set show a large language model probabil- 0 25 50 75 100 R1R2R3R4R5 BLEU Original ReTrain KGA (a) BLEU on Forget Set 25 27 29 31 R1R2R3R4R5 BLEU ReTrainKGA Original Avg Test BLEU (b) BLEU on Test Set Figure 2: BLEU scores on IWSLT test and forget sets over varying BLEU ranges (i.e., R1-R5), e.g., R1 in- cludes instances to be forgotten with original BLEU around25.0. The original BLEU from R1 to R5 gradu- ally increase, representing different levels of difficulty. ity drop (i.e., 96% PDLP) which indicates that the unlearning of forget set works. We can easily find that our KGA unlearning method performs closest to theRETRAINmodel, which validates KGAâs superiority to the compared models. 6.3 Analysis of Unlearning in NLP Most of the previous work on unlearning explores the unlearning effect on computer vision tasks with less attention to NLP tasks, especially the genera- tion tasks. Here we design two NLP-specific exper- iments and raise some interesting discussions. Deleting instances with various difficulty levels. Here we investigate if our unlearning method can handle forgetting instances with different difficulty levels on translation task. We use BLEU to measure the difficulty of instances, where a higher BLEU score indicates the instance is easier for the current model. To prepare 5 sets of instances with various difficulty levels, we adopt theORIGINALmodel to do the inference on instances in the training set, then we sort them by their BLEU score on the generated sentences. We split the training set into 5 fragments based on the BLEU and each chooses 100 instances as forget set. After that, we apply our KGA unlearning to them separately. We report the unlearned results in Fig. 2. Fig. 2(a) shows the BLEU scores ofORIGINAL model and unlearned models (i.e.,RETRAINand KGA) on forget sets (5 sets with different BLEU ranges). We can easily find that unlearning causes certain performance drop on forget set inRETRAIN while our KGA gets performance gains onR1and R2sets. It may be due to the fact that KGA tends to force the performance of forget data to be close to unseen data regardless of the BLEU ranges. There- fore, after KGA unlearning, low-performing in- stances might get a boost while high-performing ones get degraded. From Fig. 2(b), we surprisingly IndexSourceTargetORIGINALRETRAINKGA 1SchwestersistersisterNurseGirl 2Laymaundihre Schwesternhatten genug davon. Layma and her sisters had had enough. Layma and her sisters had enough of them. Layma and nurses had enough of them. Lamyma and her nurses had enough. 3Alle lebenden weiĂen Tiger in Nordamerika sind das Ergebnis selek- tiver Inzucht â also Mut- ter und Sohn, Vater und Tochter, Schwester und Bruder... All living white tigers in North America are the result of selective in- breeding â that would be mother to son, fa- ther to daughter, sister to brother... All living white tigers in North America are the result of selective inbreathing â so mother and son, father and daughter,sister and brother... All living white tigers in North America are the result of selective breed- ing â mom and son, fa- ther and daughter, nurse and brother ... All living white tigers in North America are the result of selective inbreeding â so thatâs Mom and son, father to daughter, daughter to brother... Table 4: Three translation cases from IWSLT. The model after unlearning (i.e.,exactunlearning RETRAINand approximateunlearning KGA) generates alternative words (in blue) after removing all instances containing âsisterâ. 20 40 60 10205010020050010002000 PPL NumberofRemovals ReTrainSISA KGA (a) PPL(â) on Test Set 0.1 0.2 0.3 0.4 0.5 0.6 10205010020050010002000 LPD NumberofRemovals LPD(ReTrain,KGA) (b) LPD on Forget Set Figure 3: Fig. 3(a) and Fig. 3(b) present the PPL score, Language model Probability Distance (LPD) on test and forget sets of PersonaChat dataset. find that performance on test set afterRETRAINis even better thanORIGINALmodel when forgetting the extremely easy instances (i.e., R5, while R1 is slightly higher which might be due to random effects), which is probably because the extremely easy instances take little effect to boost model per- formance. This observation also inspires one fur- ther application of unlearning âUnlearning some specific data points could bring performance gains. We leave it to our future exploration. Unlearning instances containing specific words. Unlike classification tasks, where we can remove all data of one specific label to explore the effec- tiveness of unlearning, translation tasks and most of the generation tasks do not contain such simple labels to categorize instances exactly. Therefore, we turn to select instances containing some spe- cific words in translation task to analyze the output before and after unlearning. For example, we delete all instances containing the word "sister" in the target sequence, resulting in an unlearned model which is expected to forget the word "sister". Table 4 presents the output of the original model and the unlearned models for three cases. We can see that the unlearned models cannot generate âsisterâ anymore after deleting all the instances containing âsisterâ from the training set. However, the unlearned models are capable of Base Models BLEU4 on Test SetForget Set ORIGINALKGALPD PDLP LSTM26.425.30.9598.0 Transformer29.028.40.9194.0 BART-Base34.333.10.8796.0 Table 5: Comparison results of different base models when adopting KGA unlearning on IWSLT dataset. finding the nearest alternatives to make sentences as smooth as possible, like ânurseâ and âgirlâ. A similar phenomenon can be found when the deleted words are verbs or adjectives, regardless of word frequencies. More examples about verb and adjec- tive deleting can be found in Appendix B. 6.4 Further Analyses The effects of removal numbers.We investi- gate how unlearned models maintain the perfor- mance on test set and forget the information of for- get set when dealing with varying removal numbers, and present the results in Fig. 3. From Fig. 3(a), we can see that theRETRAINmodel can main- tain the performance on test set when handling different numbers of removals, which means it is not sensitive to the size of the deleted data. And KGA can maintain the performance when remov- ing no more than 200 conversations (about 2000 instances), whileSISAcan not perform well even if the removal number is small. Fig. 3(b) shows the LPD betweenRETRAINandKGAon forget set. We can find that KGA maintains low LPD when the removal number grows, which indicates KGA performs consistently well on forgetting the selected data. The effects of base model. We further show the unlearning results when KGA is applied to different model structures. Apart from vanilla transformer, we here also experiment on LSTM and BART (a pretrained language model). Table 5 shows the results. As can be seen, KGA maintains a similar percentage of performance drop on test set using different structures, and achieves similar LPD and PDLP scores on forget set, which indicates that KGA is effective regardless of the model structure. 7 Conclusion This paper proposes KGA, a general approximate machine unlearning framework and explores its application in several NLP tasks. KGA leverages the distribution differences between two sets of models to make the unlearned model perform on forgetting data like its unseen data. Experiments on three large-scale datasets and further experiments validate the effectiveness of KGA. Limitations One of the biggest concern people may have is whether approximate unlearning forget the infor- mation of the removal data. Approximate unlearn- ing can not ensure exact removal of information already learned in deep neural models, just as its name suggests. Considering that current exact un- learning methods are very time-consuming and hard to apply in practical applications, approxi- mate unlearning is still a direction worth trying and is also effective in reducing the attack risks by attackers or mitigating the harm of toxic data. Another limitation of this work lies in the fact that we have to maintain an extra data setD n and two modelsA f andA n in the process of unlearning. Though the extra cost of our KGA method is trivial compared to the previous work (e.g., Bourtoule et al. (2021) has to maintain the entire training set), we have to point this limitation out and call for follow-up research to come up with better ways to reduce unlearning costs. Besides, we only explore word-level translation unlearning effect by comparing the generated sen- tences before and after deleting instances with spe- cific words due to the space limitation. More inter- esting experiments with different granularity can be discussed in future work to explore how unlearning method works in different NLP tasks. Ethics Statement We do not foresee any significant harm directly as a result of this work. On the contrary, our work promotes the protection of user privacy, which is significant, especially in this era that large amounts of personal data are used by neural models. Acknowledgements We would like to thank the anonymous review- ers for their feedback and suggestions. This re- search work is partially supported by CUHK un- der Project No. 4730332, Australian Research Council under the streams of Future Fellowship (No.FT210100624), Discovery Project (No. DP190101985), and Discovery Early Career Re- searcher Award (No. DE230101033). References Lucas Bourtoule, Varun Chandrasekaran, Christo- pher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Pa- pernot. 2021. Machine unlearning. In2021 IEEE Symposium on Security and Privacy (SP), pages 141â159. IEEE. Yinzhi Cao and Junfeng Yang. 2015. Towards making systems forget with machine unlearning. In2015 IEEE Symposium on Security and Privacy, pages 463â480. IEEE. Mauro Cettolo, Jan Niehues, Sebastian StĂŒker, Luisa Bentivogli, and Marcello Federico. 2014. Report on the 11th iwslt evaluation campaign. InProceedings of the 11th International Workshop on Spoken Lan- guage Translation: Evaluation Campaign. Vikram S Chundawat, Ayush K Tarun, Murari Man- dal, and Mohan Kankanhalli. 2022. Can bad teach- ing induce forgetting?unlearning in deep net- works using an incompetent teacher.arXiv preprint arXiv:2205.08096. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171â4186, Minneapolis, Minnesota. Associ- ation for Computational Linguistics. Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. 2019. Making ai forget you: Data deletion in machine learning.Advances in neural information processing systems, 32. Aditya Golatkar, Alessandro Achille, and Stefano Soatto. 2020a. Eternal sunshine of the spotless net: Selective forgetting in deep networks. InProceed- ings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 9304â9312. Aditya Golatkar, Alessandro Achille, and Stefano Soatto. 2020b. Forgetting outside the box: Scrub- bing deep networks of information accessible from input-output observations. InEuropean Conference on Computer Vision, pages 383â398. Springer. Chuan Guo, Tom Goldstein, Awni Hannun, and Lau- rens Van Der Maaten. 2019. Certified data removal from machine learning models.arXiv preprint arXiv:1911.03030. Masayuki Karasuyama and Ichiro Takeuchi. 2009. Multiple incremental decremental learning of sup- port vector machines.Advances in neural informa- tion processing systems, 22. Mohammad Emtiyaz E Khan and Siddharth Swaroop. 2021. Knowledge-adaptation priors.Advances in Neural Information Processing Systems, 34:19757â 19770. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. InIn- ternational conference on machine learning, pages 1885â1894. PMLR. Mike Lewis, Yinhan Liu, Naman Goyal, Mar- jan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension. InProceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 7871â7880, Online. Association for Computational Linguistics. Ronak Mehta, Sourav Pal, Vikas Singh, and Sathya N Ravi. 2022. Deep unlearning via randomized con- ditionally independent hessians. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10422â10431. Enrique Romero, Ignacio Barrio, and LluĂs Belanche. 2007. Incremental and decremental learning for lin- ear support vector machines. InInternational Con- ference on Artificial Neural Networks, pages 209â 218. Springer. Ahmed Salem, Yang Zhang, Mathias Humbert, Pas- cal Berrang, Mario Fritz, and Michael Backes. 2018. Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models.arXiv preprint arXiv:1806.01246. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting.J. Mach. Learn. Res., 15(1):1929â 1958. Don Tuggener, Pius von DĂ€niken, Thomas Peetz, and Mark Cieliebak. 2020.LEDGAR: A large-scale multi-label corpus for text classification of legal pro- visions in contracts. InProceedings of the Twelfth Language Resources and Evaluation Conference, pages 1235â1241, Marseille, France. European Lan- guage Resources Association. Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hen- dricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. 2021.Challenges in detoxifying language models.arXiv preprint arXiv:2109.07445. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: I have a dog, do you have pets too?InProceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204â 2213, Melbourne, Australia. Association for Com- putational Linguistics. A Details of Experimental Setup Parameter Setting and Training. Apart from the brief description in §5, we give more exper- imental details here. The DistilBERT we used for LEDGAR contains 6 transformer encoder layers each with 768 dimensions and 3072-dimensional feed-forward networks, resulting in 67M parame- ters. The transformer models used for IWSLT and PersonaChat are of the same size, i.e., containing 6 encoder and decoder layers each with 512 di- mensions and 1024-dimensional feed-forward net- works, with a total parameter amount of 91M. For the LSTM and BART-Base models we use in §6.4, the model sizes are 40M and 251M, respectively. The LSTM model contains 2 layers of encoder and decoder respectively, with 512 hidden size. The BART-Base model has 6 layers of 768-dimensional encoder and decoder, where we follow Lewis et al. (2020) to add new sets of encoder parameters be- fore the pretrained BART encoder. This results in total 10 encoder layers (i.e., we add 4 layers). We use one NVIDIA RTX 3090 GPU to train our model. When training the original model, the batch size is selected from16, 32, 64, and the final choices are 32 for LEDGAR and IWSLT, and 16 for PersonaChat, with an update frequency of 8. Learn- ing rate is selected in1e-3, 5e-4, 2e-4, 5e-5, 2e-5, RemovalSourceTargetORIGINALRETRAINKGA becomeAlle ihre Stimmen wer- den lauter und lauter, aber sie reprĂ€sentieren uns nicht. Every one of them be- comes a louder and louder voice, but they donât represent us. All their voices become louder and louder, but they donât represent us. All of their voices are getting louder and louder, but they donât represent us. All their voices are louder and louder, but they donât represent us. becomeund diese Koordina- tionriskiert,noch schwieriger zu werden mit der EinfĂŒhrung von Cyberwaffen. And this coordination may become even trick- ier with the introduction of cyber weapons. And this coordination may become even more difficult to become with the introduction of cy- ber weapons. And this coordination risk to be even more dif- ficult with the introduc- tion of cyber weapons. And that coordination, even harder to get into cyber weapons. becomeAnstelle des Treffens besserer Entscheidun- gen, werden wir von der AuswahlĂŒberwĂ€ltigt manchmal macht sie uns sogar Angst. Instead of making bet- ter choices, we become overwhelmed by choice, sometimes even afraid of it. Instead of making bet- ter choices, weâl be overwhelmed by choice, sometimes even afraid. Instead of meeting bet- ter decisions, we get overwhelmed by choice, sometimes it makes us fearful. Instead of the meeting of better choices, weâre even afraid of choice. freshWirredenhier ĂŒbergute,frische Lebensmittel, die in unglaublichem AusmaĂ verschwendet werden. Weâre talking about good, fresh food that is being wasted on a colossal scale. Weâre talking about good, fresh food thatâs being used in incredible scale. Weâre talking about good, new food thatâs used in incredible scale. Weâre talking about goodfoodsthatâs going to be used in the incredible order of scale. freshWir brauchen einen neuenStandardfĂŒr ordentlichesfrisches Essen fĂŒr eure Kinder. Ja? There needs to be a new standard of fresh, proper food for your children. Yeah? We need a new standard for decent fresh food for your kids. Yes? We need a new standard for proper new food for your kids. Right? We need a new set of clean food for your kids. Yes? freshIch glaube dass hier zwei frische Ideen drin sind â zwei. Well, I think there are two fresh things here â two fresh things. I think thereâs two freshest ideas in here â two fresh water. I think there are two new ideas in it â two. I believe that thereâs two new ideas â two. energySie werden unĂŒbertrof- fene VitalitĂ€t und En- ergie gewinnen. Youâl have unsurpassed vitality and energy. Theyâre unsurprising vi- tality and energy. Theyârewonâtwin overblown vitality and power. Itâs become overcon- ductedvitalityand power. energyAlso habe ich gedacht, wie wir die Energiekrise in diesem Land bewĂ€lti- gen können? And so I thought, how could we address the en- ergy crisis in this coun- try? So I thought, how do we deal with the energy cri- sis in this country? So I thought, how can we deal with the power crisis in this country? So I thought, how do we deal with the crisis in this country? energyEnergiepflanzen liefern ein halbes Watt pro Quadratmeterin europĂ€ischem Klima. Energy crops deliver half a watt per square meter in European cli- mates. Energy crops deliver half a watt per square meter in European cli- mates. Power plants provide half watts per square meter in the European climate. power plants deliver half a watt per square meter in European cli- mates. Table 6: Translation cases before and after unlearning from IWSLT dataset. After removing all training instances containing specific words (in red), the unlearned models tend to generate alternatives (in blue) with similar mean- ings to maintain consistency in terms of the whole sentence. For example, after removing all instances containing âenergyâ from original training set, the unlearned model generates âpowerâ in corresponding positions. and we use 5e-5 for LEDGAR, 5e-4 for IWSLT, and 2e-4 for PersonaChat, respectively. Dropout strategy (Srivastava et al., 2014) with dropout rate selected in0.1, 0.2, 0.3(the final choice is 0.1 for LEDGAR, and 0.3 for IWSLT and PersonaChat) andL 2 regularization with 0.0001 effect value are used to alleviate overfitting. During inference in generation tasks, the beam size is set to 5. All the above hyper-parameters are selected based on the performance of validation set. Unlearning Setting.The removal numbers are set to 100 instances for LEDGAR and IWSLT, and 10 conversations (about 100 instances) for Per- sonaChat unless otherwise noted. We set the stop- ping hyper-parameterÏto0.1. B More Translation Unlearning Cases Table 6 shows more cases when deleting all in- stances containing specific words, including âbe- comeâ (verb), âfreshâ (adjective), and âenergyâ (noun). We can find that unlearned models (i.e.,RE- TRAINandKGA) tend to generate alternatives with similar meanings regardless of the part of speech.