Paper deep dive
Knowledge Unlearning for Mitigating Privacy Risks in Language Models
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, Minjoon Seo
Models: GPT-Neo 125M, GPT-Neo 1.3B, GPT-Neo 2.7B, OPT 125M, OPT 1.3B, OPT 2.7B
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/12/2026, 8:29:13 PM
Summary
The paper introduces 'knowledge unlearning' as an efficient post-hoc method to mitigate privacy risks in large language models (LMs) by performing gradient ascent on target token sequences. This approach avoids the high computational costs of retraining associated with data preprocessing or differential privacy, while maintaining general model performance and providing strong empirical privacy guarantees.
Entities (5)
Relation Signals (3)
Knowledge Unlearning → appliedto → GPT-Neo
confidence 95% · We perform experiments on GPT-Neo LMs (125M, 1.3B, 2.7B)
Knowledge Unlearning → mitigates → Privacy Risks
confidence 95% · We propose knowledge unlearning as an alternative method to reduce privacy risks for LMs post hoc.
Extraction Likelihood → measures → Privacy Risks
confidence 90% · We introduce a novel metric that measures the extraction likelihood... to quantify the privacy risks
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Pretrained Language Models (LMs) memorize a vast amount of knowledge during initial pretraining, including information that may violate the privacy of personal lives and identities. Previous work addressing privacy issues for language models has mostly focused on data preprocessing and differential privacy methods, both requiring re-training the underlying LM. We propose knowledge unlearning as an alternative method to reduce privacy risks for LMs post hoc. We show that simply performing gradient ascent on target token sequences is effective at forgetting them with little to no degradation of general language modeling performances for larger LMs; it sometimes even substantially improves the underlying LM with just a few iterations. We also find that sequential unlearning is better than trying to unlearn all the data at once and that unlearning is highly dependent on which kind of data (domain) is forgotten. By showing comparisons with a previous data preprocessing method and a decoding method known to mitigate privacy risks for LMs, we show that unlearning can give a stronger empirical privacy guarantee in scenarios where the data vulnerable to extraction attacks are known a priori while being much more efficient and robust. We release the code and dataset needed to replicate our results at this https URL.
Tags
Links
Trouble viewing inline? Open PDF directly →
Full Text
77,283 characters extracted from source content.
Expand or collapse full text
KNOWLEDGEUNLEARNING FORMITIGATING PRIVACYRISKS INLANGUAGEMODELS Joel Jang 1∗ Dongkeun Yoon 3 Sohee Yang 1 Sungmin Cha 4 Moontae Lee 2,5 Lajanugen Logeswaran 2 Minjoon Seo 1 1 KAIST 2 LG AI Research 3 Konkuk University 4 Seoul National University 5 University of Illinois Chicago joeljang,sohee.yang,minjoon@kaist.ac.kr, ramses2687@konkuk.ac.kr sungmin.cha@snu.ac.kr,moontae.lee,llajan@lgresearch.ai ABSTRACT Pretrained Language Models (LMs) memorize a vast amount of knowledge during initial pretraining, including information that may violate the privacy of personal lives and identities. Previous work addressing privacy issues for language models has mostly focused on data preprocessing and differential privacy methods, both requiring re-training the underlying LM. We proposeknowledge unlearningas an alternative method to reduce privacy risks for LMspost hoc. We show that simply performing gradient ascent on target token sequences is effective at forgetting them with little to no degradation of general language modeling performances for larger LMs; it sometimes even substantially improves the underlying LM with just a few iterations. We also find thatsequentialunlearning is better than trying to unlearn all the data at once and that unlearning is highly dependent on which kind of data (domain) is forgotten. By showing comparisons with a previous data preprocessing method and a decoding method known to mitigate privacy risks for LMs, we show that unlearning can give a stronger empirical privacy guarantee in scenarios where the data vulnerable to extraction attacks are known a priori while being much more efficient and robust. We release the code and dataset needed to replicate our results at https://github.com/joeljang/knowledge-unlearning. 1INTRODUCTION Recent work has shown that an adversary can extract training data from Pretrained Language Mod- els (LMs) including Personally Identifiable Information (PII) such as names, phone numbers, and email addresses, and other information such as licensed code, private clinical notes, and 128-bit UUIDs (Carlini et al., 2021; Lee et al., 2022; Huang et al., 2022; Lehman et al., 2021). In 2021, an AI chatbotIrudabecame the first AI system to be sued for violating the Personal Information Protec- tion Act after generating the exact home addresses and bank account numbers of actual individuals unintentionally (Park, 2021). Heikkil ̈ a (2022) has also shown that GPT-3 (Brown et al., 2020), one of the most well-known LM currently in commercial use, offered detailed private information about the Editor-in-Chief of MIT Technology Review including his family members, work address, and phone number. Considering findings that show extracting training data gets easier as LMs scale to larger sizes (Carlini et al., 2022a) and that it is common practice for practitioners to release billion parameters pretrained LMs for public use (Gao et al., 2020; Black et al., 2021; Zhang et al., 2022), it has become important to provide privacy guarantees for large LMs. Practitioners are required todeletepersonal information from the LMs by individuals’ request be- cause each individual has the “Right To Be Forgotten (RTBF)” (Mantelero, 2013; Graves et al., 2021) and can limit the direct and indirect commercial use of their personal information (Villaronga et al., 2018). Previous methods addressing privacy risks for language models attempt to remove all private information from the training data (data preprocessing) (Aura et al., 2006; Dernoncourt et al., 2017; Lison et al., 2021; Kandpal et al., 2022) or attempt to design algorithms that ensure differen- tial privacy (DP) (Dwork, 2008; Dwork et al., 2006; Abadi et al., 2016; Anil et al., 2021; Li et al., ∗ work done during internship at LG AI Research. 1 arXiv:2210.01504v2 [cs.CL] 19 Dec 2022 Name: Bob Age: 27 Marital Status: Single SSN: 123 - 4567 - 8910 Details: Got divorced by ex-wife named Alice and is currently undergoing custody battles. Net Worth: $5,000,000 Sensitive Personal Information Data Preprocessing Differential Privacy Knowledge Unlearning LM Find and Remove Re-train LM after sanitization (~900 A100 GPU days) LM Re-train LM with DP Algorithm (~1800 A100 GPU days) LM Perform a few token updates (~0.001 A100 GPU days) I practice my Right To Be Forgotten (RTBF)! Pretraining Corpora Pretraining Corpora Token Sequences Our Proposed Approach Bob Figure 1: Comparison of previous approaches andknowledge unlearningwhen an individual prac- tices his/her Right-To-Be-Forgotten (RTBF). 2022; Yu et al., 2022). Both approaches requireretrainingthe underlying LM every time individuals want to practice their RTBF, which makes them inadequate for large LMs that are extremely costly to retrain. Furthermore, as pointed out by Brown et al. (2022), data preprocessing methods assume private information to be easily identifiable, specified, and removed and DP algorithms can only guarantee protection for information that has clear privacy borders, which makes them inadequate in the real-world scenarios where the standard of privacy might differ by each individual. To this end, we proposeknowledge unlearning(Figure 1) as an efficient solution that can be applied with just a few parameter updates instead of pretraining the underlying LM again. We perform ex- periments on GPT-Neo LMs (125M, 1.3B, 2.7B) (Black et al., 2021) and show that simply changing the gradient descent to the opposite direction during language modeling (which can also be seen as maximizinginstead ofminimizingthe loss function) is effective at protecting target sequences from extraction attacks with little to no performance degradation on the initial LM capabilities measured via 9 common NLP classification benchmarks (Hellaswag (Zellers et al., 2019), Lambada (Paperno et al., 2016), Winogrande (Sakaguchi et al., 2021), COPA (Gordon et al., 2012), ARC-Easy (Clark et al., 2018), ARC-Challenge (Clark et al., 2018), Piqa (Bisk et al., 2020), MathQA (Amini et al., 2019), and PubmedQA (Jin et al., 2019)) and 4 dialogue tasks (Wizard of Wikipedia (Dinan et al., 2019), Empathetic Dialogues (Rashkin et al., 2019), Blended Skill Talk (Smith et al., 2020), and Wizard of Internet (Komeili et al., 2022)). For some cases,knowledge unlearningunexpectedly shows significant improvements in LM performance for some of the benchmarks. We compare our approach with data deduplication method (Kandpal et al., 2022) and differential privacy decoding method (Majmudar et al., 2022) which are both known to mitigate privacy risks and show the effectiveness of knowledge unlearning by providing strong privacy protection while being much more efficient and robust. We also provide a general guideline that can be used to quantify thememorizationandextraction likelihoodof target token sequences and suggest when we can empirically consider them to have been “forgotten”. Specifically, we introduce a novel metric that measures the extraction likelihood by varying the prefix length of the target token sequence and quantifying how much of the suffix is actually extracted from the LM. Surprisingly, forknowledge unlearning, we find that it is easier to forget a chunk of instancesse- quentiallyrather than trying to forget them all at once. We provide further analysis and show that the difficulty ofknowledge unlearningdepends heavily on the target data being forgotten, especially the domain of the target data. We also provide empirical examples of performing extraction attacks and how exactlyknowledge unlearningprovides privacy protection for the LM. 2 To summarize, our main contributions are fourfold: • We compareknowledge unlearningwith two approaches from literature known to mitigate privacy risks: a data preprocessing approach and a Differential Privacy (DP) Decoding approach. We show that our approach results in little to no performance degradation of general capabilities (sometimes resulting in improvement) while providing strong privacy protections in situations individuals practice their RTBF whereas the data preprocessing ap- proach provides weaker privacy protection while being orders of magnitude computation- ally demanding and the DP Decoding approach results in severe degradation of modeling performance. • We perform additional experiments to determine which factors contribute to the difficulty of knowledge unlearning and find that (1) trying to forget many samples at once results in substantial LM performance degradation which can be mitigated bysequentiallyforgetting chunks of data and that (2) the domain of the target data (Code, License, Wikipedia, etc.) plays a critical role in determining how hard they are to forget. • We provide a novel metric and a general guideline for quantifying the privacy risks for LMs and determine when they should be considered to have “forgotten” a given target sequence. •Knowledge unlearningsurprisingly seems to make LMs stronger where the extreme cases bring+8.0%(37.6%→45.6%),+10.1%(57.4%→67.5%), and+7.9%(62.2%→70.1%) improvements on Lambada for GPT-NEO125M, 1.3B, and 2.7B, respectively. 2RELATEDWORK 2.1PRIVACYMETHODS FORLANGUAGEMODELS Prior work that tries to mitigate privacy risks for LMs can be divided mainly into data pre/post- processing methods and differential privacy methods. (Data) Pre/Post-ProcessingData preprocessing aims to sanitize the training data; it aims to get rid of all data that might violate any kind of privacy from the training data prior to training. These methods mostly utilize measures such as parsers and classification models that try to identify and predict patterns that constitute private information. This is effective at identifying well-formatted private information such as social security numbers or special forms of medical notes (Aura et al., 2006; Dernoncourt et al., 2017; Lison et al., 2021; Kandpal et al., 2022). However, as pointed out by Brown et al. (2022), considering that private information is mostly context-dependent and sometimes in a non-specific format, data preprocessing methods cannot fully claim that they provide privacy guarantees, especially guarantees that match each individual’s standards. Methods that attempt to utilizepost-processingmethods such as applying censorship to the LM outputs still face the same limitations. In this work, we compare our proposed method with a data preprocessing approach proposed by Kandpal et al. (2022) which shows that deduplicating the training corpora before pretraining helps pretrain LMs that show stronger robustness against extraction attacks than an LM pretrained under the same circumstances without deduplicating the pretraining corpora. However, we highlight that this approach, which may still be effective at mitigating the overall privacy risks, is not the most suitable approach when considering a realistic scenario of individuals requesting the removal of their information from the implicit parameters of the LMs. Differential PrivacyDifferential Privacy (DP) aims to guarantee that the effect of an individual input on the output of a specific function is bounded (Dwork, 2008; Dwork et al., 2006). In the context of deep neural networks, DP, which needs to be applied during the training phase, aims to construct models that can providegeneralguarantees that the individual information within the training data cannot be inferred (Abadi et al., 2016). While DP has shown to be surprisingly ef- fective at fine-tuning LMs (Li et al., 2022; Yu et al., 2022), pretraining LMs with DP still suffers from substantial performance gap, expensive computation, and slow convergence (Anil et al., 2021). Furthermore, as pointed out by Brown et al. (2022), DP can only provide limited guarantees for LMs because DP requires a unified definition for privacy boundaries, which is inherently impossible for natural language data. Most importantly, in a realistic scenario where individuals may practice their 3 Right-To-Be-Forgotten (RTBF) dynamically after model deployment, it is nontrivial to apply ex- isting descent-based DP algorithms such as DP-SGD to only protection againsttargetedextraction attacks. 2.2MACHINEUNLEARNING Machine unlearning has received attention as an alternative approach to overcome data privacy issues in machine learning (Cao & Yang, 2015; Ginart et al., 2019; Bourtoule et al., 2021; Graves et al., 2021). Several studies attempt to explore machine unlearning for deep neural networks (Golatkar et al., 2020; Mehta et al., 2022). However, they mostly focus on proposing algorithms for image classification models where they aim to forget a whole class; that is, achieve random performance for specific image classes such as “cats” or “ships”. We are the first, to the best of our knowledge, to explore unlearning a specific sequence of tokens for LMs which is a quite different set-up from traditional image classification models (∼tens of image classes vs. a sequence of tokens that can each be classified intoV∈R ∼50,000 ). In this work, we coin this approach asknowledge unlearning since we are more focused on forgetting specificknowledgerepresented by sequences of tokens. Zhou et al. (2022) focus on howforgettingcan be leveraged to improve the performance of the un- derlying model. They propose “forget-and-relearn” that unifies existing iterative training algorithms by selectively removing undesirable information and re-learning good features, helping boost per- formance for the task of image classification and multi-agent emergence communication. The un- derlying assumption is that it is often easier to define and stop unwanted behavior than to teach good behavior. We also show this phenomenon in Section 4 where we unintentionally find unlearning just a few sequences of tokens sometimes boosts general LM capabilities. 2.3MEMORIZATION INLANGUAGEMODELS Previous work that explores to which extent LMs have memorized their training data approach the phenomenon with two different viewpoints. Some work view memorization of LMs simply as a threat to individual privacy (Carlini et al., 2021; 2022a; Jagielski et al., 2022) and utilize metrics that quantify how much the LMs are susceptible to adversarial attacks. These metrics are mostly dependent on the specific types of attacks such as the membership inference attack (Shokri et al., 2017) and measure the privacy risks of LMs by quantifying the success rate of these attacks. In our work, we instead focus on moretargetedextraction attacks. Another line of work simply quantifies how muchknowledgeis accumulated and forgotten during pretraining by extracting relational knowledge about the world (Petroni et al., 2019; Lazaridou et al., 2021; Jang et al., 2022b;a). This line of work does not view memorization as a negative trait, but as a positive one that can be leveraged to extract world knowledge from its implicit parameters and per- form knowledge-intensive tasks such as question answering or training knowledgeable conversation agents. Our work is highly related to Jagielski et al. (2022)’s work where they also assert that forgetting can be a relaxed version of differential privacy. However, there are two main differences between our work and theirs. First, they only analyze forgetting as apassiveform of mitigating privacy, asserting that data seen early in large-scale training obtain privacy benefits, whereas we suggest a moreactiveform of forgetting. Second, they only show analysis results with image classification and audio generation models while we specifically focus on large LMs. 3KNOWLEDGEUNLEARNING FORLANGUAGEMODELS 3.1METHODOLOGY We propose simplynegatingthe original training objective of minimizing the negative log-likelihood of the token sequences as our main method of knowledge unlearning in LMs. Specifically, given a sequence of tokensx= (x 1 ,...,x T ), our unlearning training objective is simplymaximizingthe following loss function: L UL (f θ ,x) =− T ∑ t=1 log(p θ (x t |x <t ))(1) 4 wherex <t denotes the token sequencex= (x 1 ,...,x t−1 )andp θ (x t |x <t )denotes the conditional probability of predicting the next token to bex t when givenx <t to an LMfwith parametersθ. 3.2QUANTIFYINGPRIVACYRISKS OFLANGUAGEMODELS In this subsection, we introduce two metrics we use to quantify the privacy risks given a specific token sequence and how we empirically define the token sequence to be forgotten. In this work, we do not utilize metrics such as membership inference attack recall (Shokri et al., 2017) since we are not interested in quantifying thegeneralprivacy risks of LMs, but instead the privacy risks on the specific target token sequences. Extraction Likelihood (EL)We first introduce a new metric, EL. Given a sequence of tokens x= (x 1 ,...,x T )and an LMfwith pre-trained parametersθ, we define EL to be as follows: EL n (x) = ∑ T−n t=1 OVERLAP n (f θ (x <t ),x ≥t ) T−n (2) OVERLAP n (a,b) = ∑ c∈n-grams(a) 1c∈n-grams(b) |n-grams(a)| (3) wheren-grams()denotes the list ofn-grams in the given token sequence andf θ (x <t )denotes the output token sequences from the LMf θ when givenx <t as input that can have max lengths|x ≥t | but may be shorter when the EOS (end-of-sequence) token is generated beforehand. The process of varying the prefix length|x <t |can be seen as varying thestrengthof adversarial attacks. This is based on the assumption that the more prior information is provided about the target token sequence, the easier the LM will be able to extract it. Overall, EL can be seen as estimating the generalextraction likelihoodsince we are measuring the average success rate of varying extraction attacks quantified via getting the n-gram overlap of generated and target token sequences. While previous metrics quantifying the privacy risks of LMs are dependent on specific adversarial attacks, this characteristic of EL allows it to quantify the general likelihood of extraction without any dependency on specific extraction attacks. We regardnto be a hyper-parameter that can be varied depending on the stringency of privacy standards. The highernis set, the stricter we set the standard for a successful extraction attack. Memorization Accuracy (MA)We define Memorization Accuracy (MA) as follows: MA(x) = ∑ T−1 t=1 1argmax(p θ (·|x <t )) =x t T−1 (4) MA quantifies how muchf θ has memorized the given token sequences and was proposed by Tiru- mala et al. (2022) to analyze the training dynamics of large LMs. Empirical Definition of ForgettingBy utilizing both EL n and MA, we empirically define a spe- cific token sequencexto be forgotten and is no longer susceptible to extraction attacks when the following conditions are met: EL n (x)≤ 1 |D ′ | ∑ x ′ ∈D ′ EL n (x ′ )and MA(x)≤ 1 |D ′ | ∑ x ′ ∈D ′ MA(x ′ )(5) whereD ′ represents a validation corpora not seen during training. In other words, we definexto be forgotten when the EL n (x) and MA(x) reach a value that is lower than the average EL n and MA on token sequences that were not seen during training. 5 4EXPERIMENTS 4.1MODELS, DATASETS,ANDCONFIGURATIONS Baseline ModelsFor the experiments, we use the GPT-NEO(125M, 1.3B, 2.7B) LMs (Black et al., 2021) initially pretrained on all of the Pile corpora (825GB) (Gao et al., 2020), and the OPT (125M, 1.3B, 2.7B) LMs (Zhang et al., 2022), pretrained on a subset of thededuplicatedversion of the Pile as well as other corpora from different domains. For the experiments, we perform unlearn- ing the GPT-NEOLMs and quantify the privacy risks of the target data compared to the OPT LMs to measure how effective our proposed approach is in contrast to deduplicating the training corpora before pretraining the underlying LM Kandpal et al. (2022). We do not use the exact LMs from Kandpal et al. (2022) because the LMs were not open-sourced, and thus use the OPT LMs instead. We also consider the Differential Privacy (DP) Decoding (Majmudar et al., 2022) as one of the base- lines; This approach proposes a decoding strategy that performs linear interpolation of the original logits with the uniform distribution and performs nucleus sampling, which they theoretically show provides DP guarantees.λis set as the linear interpolation weight whereλ= 0performs nucleus sampling from the uniform distribution andλ= 1performs regular nucleus sampling, using the logits as weights during random sampling. Target DataFor the actual target data used to quantify the privacy risks of the LMs, we sample instances from the Training Data Extraction Challenge 1 where 15,000 examples (each are 200 token sequences long) from 16 different domains of the Pile corpora that are identified to be somewhat easy-to-extract are provided. For our experiments, we randomly samplessamples from the 15,000 examples and make the underlying LM forget thessamples at once. As a default, we show the average results of 5 random samplings ofssamples for all of our experimental settings. We only provide the average of the 5 samplings and do not separately report the standard deviation. Instead, we provide the results of each individual run in Appendix A. Evaluation DatasetsProviding stronger privacy protections for LMs may become meaningless if it requires sacrificing their original capabilities. Thus, while quantifying the privacy risks of LMs, we also quantify the original LM capabilities by evaluating the LMs on 9 different clas- sification tasks quantifying the general capabilities: Hellaswag (Zellers et al., 2019) and Lam- bada (Paperno et al., 2016) benchmarks to measure linguistic reasoning abilities, Winogrande (Sak- aguchi et al., 2021) and COPA (Gordon et al., 2012) to measure commonsense reasoning abilities, and ARC-Easy (Clark et al., 2018), ARC-Challenge (Clark et al., 2018), Piqa (Bisk et al., 2020), MathQA (Amini et al., 2019), PubmedQA (Jin et al., 2019) benchmarks to measure the scientific reasoning abilities. We also evaluate on 4 dialogue tasks (Wizard of Wikipedia (Dinan et al., 2019), Empathetic Dialogues (Rashkin et al., 2019), Blended Skill Talk (Smith et al., 2020), and Wizard of Internet (Komeili et al., 2022)) to evaluate the generation capabilities of the LMs. We use the test set for Lambada and the validation set for the rest of the datasets. We also show the results of measuring the perplexity on the validation corpora of Pile and Wikitext in Appendix B. We do not include measuring perplexity as one of the main evaluations because perplexity might not be the most suitable metric for quantifying general LM performance, especially in the case of unlearning (further explanation given in Appendix B. We evaluate DP Decoding only on the 4 dialogue tasks because the decoding strategy cannot be applied for performing the classification tasks which is evaluated by utilizing averbalizer. ConfigurationsFor the learning rate, we set it to 5e-5. We show the effect of varying learning rates in Appendix D. We use a constant learning rate scheduling throughout the run. We fix the global batch size to be the same ass(how many samples are forgotten at once) because having global batch sizes smaller thansproved to degrade general LM capabilities 2 . For EL n , we setn=10 which means EL measures the extraction likelihood of extractingnconsecutive tokens of varying extraction attack 3 . For calculating EL 10 and MA, we use a na ̈ ıve greedy decoding strategy. We set 1 https://github.com/google-research/lm-extraction-benchmark 2 In Section 4.3, We show thatsplays a critical role in determining how much the unlearning will degrade in general capabilities of the LM sinces= 128shows to result in much degradation. Method to mitigate this is proposed in Section 4.3 as well. 3 We set thenvalue to 10 since we empirically consider an extraction to be successful when 10 consecutive token sequences are successfully generated by the LM. We show varying thenwith values from [5,10,20,40] in Appendix H. 6 Table 1:Forgetting Threshold for GPT-NEOLMs Model (Size) EL 10 (%)MA(%) ThresholdThreshold GPT-NEO(125M)4.9929.94 GPT-NEO(1.3B)5.6833.27 GPT-NEO(2.7B) 5.5334.02 Table 2:Main Results showing the average of 5 random sampling ofs= 32(forgetting 32 samples at once). OPT represents the LM with deduplication applied. NEOdenotes the initial GPT-NEOLM, NEO+ DPD + represents applying the DP Decoding strategy by varying theλto match the forgetting criteria, NEO + UL represents performing unlearning on the initial NEOuntil it provides stronger security for the target sequences than OPT, NEO+ UL + represents performing unlearning on GPT-NEOuntil target sequences match the forgetting criteria,LM Avg.denotes the average accuracy of the 9 classification datasets, andDialogue Avg. denotes the average F1 score of the 4 dialogue datasets. The best comparable performances areboldedand second best underlined . Model #EL 10 MALM Avg.Dialogue Avg. Epoch Params(%)↓(%)↓(ACC)↑(F1)↑ OPT125M8.652.942.410.2- NEO125M30.977.443.49.4- NEO+ DPD + 125M0.027.4N/A7.3- NEO+ UL125M 3.750.142.68.011.0 NEO+ UL + 125M1.027.439.92.617.2 OPT1.3B23.367.150.612.4- NEO1.3B 67.692.249.811.5- NEO+ DPD + 1.3B0.021.4N/A7.1- NEO+ UL1.3B11.062.249.711.68.0 NEO+ UL + 1.3B 1.930.449.78.513.8 OPT2.7B25.669.252.712.9- NEO2.7B 70.493.452.311.5- NEO+ DPD + 2.7B0.024.2N/A6.9- NEO+ UL2.7B13.066.052.312.55.4 NEO+ UL + 2.7B1.631.051.911.110.8 both the dropout and weight decay rates to 0. Lastly, while we provide a guideline of empirically deciding a single token sequence to be forgotten in Section 3.2, for considering achunkofstoken sequences to be forgotten, we use the average EL 10 and MA as an approximation of the individual EL 10 and MA. 4.2MAINEXPERIMENTS Forgetting ThresholdFirst, we show how we get the Forgetting Threshold for EL 10 and MA, the values where we consider the token sequence to be forgotten and unsusceptible from extraction attacks, for all model sizes of GPT-NEOLMs in Table 1. ForD ′ , we perform weighted sampling (same domain distribution as the Pile training corpora) of 10,000 instances each with token lengths 200 from the Pile validation corpora, and measure the average EL 10 and MA (Equation 5), which are empirically set as the Forgetting Threshold values. Main ResultsTable 2 shows the main results of performing unlearning on LMs of varying sizes and the baselines. While we provide the average performances of the 5 random samplings in Table 2, we provide each individual runs in Appendix A for reference. We highlight five main observations regarding the results. (1) OPT LMs show a much lower EL 10 and MA than GPT-NEOLMs, confirming that deduplicating the pretraining corpora is indeed help- ful for mitigating privacy risks. (2) NEO+ DPD + enables effective protection against extraction attacks demonstrated via the lowest EL and MA score; however, it brings severe degradation of generation capabilities measured via the Average F1 score of the 4 dialogue generation tasks. (3) NEO+ UL + results in severe degradation of both classification and dialogue tasks for the 125M, only severe degradation of dialogue tasks for 1.3B LM while for the 2.7B LMs, it enables re- taining most of its previous capabilities. (4) While the LMs scale to larger sizes, it takes fewer epochs for the target sequences to be forgotten. Together with (3), this implies that larger LMs are 7 014832128 # of Samples Forgotten at Once 35 40 45 50 Avg. Performance 125M 1.3B 2.7B (a) Batch Unlearning 0326496128 Total # of Samples Sequentially Forgotten 35 40 45 50 Avg. Performance 125M 1.3B 2.7B (b) Sequential Unlearning Figure 2:Average LM performance on the 9 benchmarks when varying the total number of samples forgotten at once is shown in (a) and the average LM performance when the 128 samples are divided into 4 chunks and are forgotten sequentially is shown in (b). The lines denote the average performances of 5 random samplings and the standard deviation is shown as the shaded regions. The dotted lines in (b) denote thes= 128performance in (a) for comparison purposes. strongunlearners. (5) While NEO+ UL + provides stronger privacy protection than OPT without sacrificing its performance from NEOfor the 2.7B LM, it is much more computationally efficient (3,500,000x) than re-training the underlying LM, which is required for all data preprocessing ap- proaches 4 . Overall, results show unlearning to be an effective approach to providing a strong privacy protection while retaining and sometimes even improving general LM capabilities. Sequential Unlearning is more Stable than Batch Unlearning.We show the effect of varying s(the # of data instances to be forgotten at once) in Figure 2 across model scales. We denote this approach asbatchunlearning. As shown by thes= 128results, it is harder to forget more samples at once, resulting in substantial degradation of average LM performance regardless of how large the LM is. Sinces≤32does not show much degradation, we explore ifsequentiallyunlearning can be a solution. In Figure 2b, we show the result of dividing the 128 samples into 4 chunks of 32 and performing sequential unlearning; we unlearn each chunk at a time until the chunk reaches the forgetting threshold. Surprisingly, as shown by the performance gap ats= 128between the dotted lines (thes= 128performance of Figure 2a) and straight lines, the end result is vastly different even though exactly the same instances were forgotten. Sequential unlearning shows almost no degradation of average LM performance. In Appendix G, we show that chunks once forgotten stay forgotten and that later chunks are forgotten much faster compared to the initial chunk. This result hints at thegeneralizationof unlearning, which we do not further explore in the scope of this work. The result also suggests that knowledge unlearning can becontinuallyapplied to LMs when needed. 4.3ANALYSIS OFKNOWLEDGEUNLEARNING Providing Better Intuition of What Exactly Happens During Knowledge Unlearning.To show exactly what happens to the LM during knowledge unlearning, we show how the performance of each of the LM benchmarks changes as we perform 10 runs of unlearning to the GPT-NEO (1.3B) model (each run withs= 1) in Figure 3. As shown in the figure, the LM performance for each benchmark varies tremendously on which sample is chosen to be forgotten. Furthermore, the ending time of each run is different, indicating that some samples are forgotten faster than others. To provide a better intuition of exactly how knowledge unlearning guarantees privacy, we perform an extraction attack with a token sequence sample in Table 3 where we show the model-generated text from the extraction attackbeforeandafterapplying knowledge unlearning. While the extraction attack is extremely successful at extracting the rest of the suffix before unlearning (100% of the token sequence), only a small portion (∼3% of the token sequence) of the suffix is extracted after applying unlearning. 4 Computational efficiency is measured via FLOPs which is calculated by (6 × Total Training Tokens × Parameter Size) as in Brown et al. (2020). FLOPs for OPT LMs were estimated using information from Zhang et al. (2022). We provide the FLOPs for the methods in Appendix C. 8 36.25% 36.50% 36.75% 37.00% 37.25% 036912 Accuracy Epochs (a) Hellaswag 51% 54% 57% 60% 036912 Accuracy Epochs (b) Lambada 54.0% 54.5% 55.0% 55.5% 036912 Accuracy Epochs (c) Winogrande 62.5% 65.5% 68.5% 71.5% 74.5% 036912 Accuracy Epochs (d) COPA 52.5% 55.0% 57.5% 60.0% 036912 Accuracy Epochs (e) ARC-Easy 22.50% 23.75% 25.00% 26.25% 27.50% 036912 Accuracy Epochs (f) ARC-Challenge 69.0% 69.5% 70.0% 70.5% 71.0% 036912 Accuracy Epochs (g) Piqa 45% 50% 55% 60% 036912 Accuracy Epochs (h) MathQA 47.5% 50.0% 52.5% 55.0% 57.5% 036912 Accuracy Epochs (i) PubMedQA Figure 3:Performance on the LM benchmarks as we perform 10 different unlearning runs on GPT-NEO1.3B wheres= 1. Table 3:An example extracting the suffix of a token sequence fromBOOKS3 domain from GPT-NEO1.3B showing the effect of knowledge unlearning. Model generated text given a prefix of length 100 are shown in Blue. DomainStatusText BOOKS3 Original About the Publisher Australia HarperCollins Publishers (Australia) Pty. Ltd. 25 Ryde Road (PO Box 321) Pymble, NSW 2073, Australia http://w.harpercollinsebooks.com.au Canada HarperCollins Publishers Ltd. 55 Avenue Road, Suite 2900 Toronto, ON, M5R, 3L2, Canada http://w.harpercollinsebooks.ca New Zealand HarperCollins Publishers (New Zealand) Limited P.O. Box 1 Auckland, New Zealand http://w.harpercollinsebooks.co.nz United Kingdom HarperCollins Publishers Ltd. 77-85 Fulham Palace Road London, W6 8JB, UK http://w.harpercollinsebooks.co.uk Text Before About the Publisher Australia HarperCollins Publishers (Australia) Pty. Ltd. 25 Ryde Road (PO Box 321) Pymble, NSW 2073, Australia http://w.harpercollinsebooks.com.au Canada HarperCollins Publishers Ltd. 55 Avenue Road, Suite 2900 Toronto, ON, M5R, 3L2, Canada http://w.harpercollinsebooks.ca New Zealand HarperCollins Publishers (New Zealand) Limited P.O. Box 1 Auckland, New Zealand http://w.harpercollinsebooks.co.nz United Kingdom HarperCollins Publishers Ltd. 77-85 Fulham Palace Road London, W6 8JB, UK http://w.harpercollinsebooks.co.uk Unlearning After About the Publisher Australia HarperCollins Publishers (Australia) Pty. Ltd. 25 Ryde Road (PO Box 321) Pymble, NSW 2073, Australia http://w.harpercollinsebooks.com.au Canada HarperCollins Publishers Ltd. 55 Avenue Road, Suite 2900 Toronto, ON, M5R, 3L2, Canada http://w.harpercollins.com.au/Publishers/ Publisher: level three Level two is levels one and two together. The new face of a already great title! Level one: Just right. Level two: Great. Level three: Awesome. The BloomsburyPublishersPublishers.com.au/PublishersPublishers Levels are for bibliographic information or advanced level. s Unlearning Towards Understanding Why Some Instances are Harder to ForgetTo measure why some instances are harder to forget, we perform 5 random samplings ofs= 8from 8 different domains from the Training Data Extraction Challenge 5 and perform unlearning on the GPT-NEO1.3B LM. We also show the results of each individual run in Appendix A. As shown in Table 4, despite un- dergoing the same number of token updates (10 epochs of unlearning), different domains result in vastly different outcomes;ENRON EMAILSresults in the average LM performance degradation of only -0.4% whileUSPTO BACKGROUNDSresults in -4.5% degradation. Furthermore, the final EL 10 varies depending on the domain, suggesting that some domains (e.g., FREELAW) are harder to for- get than others. Lastly, domains that are morestructured, which means the data consists of some 5 https://github.com/google-research/lm-extraction-benchmark 9 Table 4:Unlearning GPT-NEO1.3B on token sequences sampled from 8 different domains. We fix the epoch to 10, sets= 8, and show the result of the average of 5 random samplings.Italicized() denotes the∆from INITIAL. Domains Initial FinalHella. Lamba. Wino. COPA ARC-E ARC-C Piqa MathQ PubQAvg. EL 10 EL 10 (ACC)(ACC)(ACC) (ACC)(ACC)(ACC)(ACC) (ACC) (ACC)(ACC) INITIAL--37.057.454.970.056.625.870.421.953.849.8(0.0) FREELAW60.412.137.252.253.968.455.526.270.121.753.548.7 (-1.1) GIT. (CODE)63.90.637.353.454.469.256.326.069.921.549.848.7 (-1.1) GIT. (LICENSE)75.80.037.152.054.269.056.426.470.121.851.848.8 (-1.0) ENRON EMAILS 77.30.036.957.254.868.455.826.369.821.853.149.4(-0.4) BOOKS370.20.036.449.554.270.855.625.569.921.747.447.9 (-1.9) PILECC67.80.035.745.953.870.454.226.969.721.852.047.8 (-2.0) USPTO BACK.59.40.033.744.753.567.045.924.067.021.550.345.3 (-4.5) PUBMED CENT. 71.80.036.544.554.169.655.624.870.021.946.447.0 (-2.8) kind of patterns such as a list of emails (ENRON EMAILS) or code (GITHUB(CODE)), seem to result in less degradation of LM performance in contrast to domains that are moreunstructured, which means the data consist of mostly raw English text such as a review for journal submission (PUBMED CENTRAL). We provide examples from each domain in Appendix E. However, further analysis of understanding exactly which components make unlearning work should be made in future work. 5CLOSING In this paper, we proposeknowledge unlearningas a method for mitigating privacy risks in LMs that provides strong privacy protection with little to no degradation of general LM capabilities measured by evaluating on 9 common LM classification benchmarks and 4 dialogue benchmarks for the larger sized LMs. As large LMs expand their use cases, potentially affecting the daily lives of people, the research community should make sure that the privacy of individuals is not violated intentionally or unintentionally by the knowledge stored in the implicit parameters of these models. Since it is inherently impossible to prevent and predict all future privacy concerns prior to pretraining the LM, we suggest the community consider knowledge unlearning for ensuring privacy upon individuals’ requests post hoc pretraining. 6 . ACKNOWLEDGMENTS We thank Hanseok Oh, Minsu Kim, James Thorne, and Hyunji Lee for the useful discussion and feedback while preparing the paper draft. This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea govern- ment(MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST)). REFERENCES Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. InProceedings of the 2016 ACM SIGSAC conference on computer and communications security, p. 308–318, 2016. Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-based formalisms. InProceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), p. 2357–2367, Minneapolis, Minnesota, June 2019. Association for Computational Lin- guistics. doi: 10.18653/v1/N19-1245. URLhttps://aclanthology.org/N19-1245. Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. Large-scale differen- tially private bert.arXiv preprint arXiv:2108.01624, 2021. 6 We provide some limitations of our work in Appendix I. 10 Tuomas Aura, Thomas A Kuhn, and Michael Roe. Scanning electronic documents for personally identifiable information. InProceedings of the 5th ACM workshop on Privacy in electronic soci- ety, p. 41–50, 2006. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical com- monsense in natural language. InProceedings of the AAAI conference on artificial intelligence, volume 34, p. 7432–7439, 2020. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. Gpt-neo: Large scale autore- gressive language modeling with mesh-tensorflow.If you use this software, please cite it using these metadata, 58, 2021. Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In2021 IEEE Symposium on Security and Privacy (SP), p. 141–159. IEEE, 2021. Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tram ` er. What does it mean for a language model to preserve privacy?arXiv preprint arXiv:2202.05520, 2022. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020. Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In2015 IEEE Symposium on Security and Privacy, p. 463–480. IEEE, 2015. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In30th USENIX Security Symposium (USENIX Security 21), p. 2633–2650, 2021. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models.arXiv preprint arXiv:2202.07646, 2022a. Nicholas Carlini, Matthew Jagielski, Chiyuan Zhang, Nicolas Papernot, Andreas Terzis, and Florian Tramer. The privacy onion effect: Memorization is relative. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.),Advances in Neural Information Processing Sys- tems, 2022b. URLhttps://openreview.net/forum?id=ErUlLrGaVEU. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457, 2018. Franck Dernoncourt, Ji Young Lee, Ozlem Uzuner, and Peter Szolovits. De-identification of patient notes with recurrent neural networks.Journal of the American Medical Informatics Association, 24(3):596–606, 2017. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge-powered conversational agents. InInternational Conference on Learning Representations, 2019. URLhttps://openreview.net/forum?id=r1l73iRqKm. Cynthia Dwork. Differential privacy: A survey of results. InInternational conference on theory and applications of models of computation, p. 1–19. Springer, 2008. Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. InTheory of cryptography conference, p. 265–284. Springer, 2006. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027, 2020. 11 Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning.Advances in neural information processing systems, 32, 2019. Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 9304–9312, 2020. Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. SemEval-2012 task 7: Choice of plau- sible alternatives: An evaluation of commonsense causal reasoning. In*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), p. 394–398, Montr ́ eal, Canada, 7-8 June 2012. Asso- ciation for Computational Linguistics. URLhttps://aclanthology.org/S12-1052. Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. InProceedings of the AAAI Conference on Artificial Intelligence, volume 35, p. 11516–11524, 2021. Melissa Heikkil ̈ a. What does gpt-3 ”know” about me?, Aug 2022. URLshorturl.at/dgio0. Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. Are large pre-trained language models leaking your personal information?arXiv preprint arXiv:2205.12628, 2022. Matthew Jagielski, Om Thakkar, Florian Tram ` er, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, et al. Measuring forgetting of memorized training examples.arXiv preprint arXiv:2207.00099, 2022. Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. Temporalwiki: A lifelong benchmark for training and evaluating ever- evolving language models.arXiv preprint arXiv:2204.14211, 2022a. Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun KIM, Stan- ley Jungkyu Choi, and Minjoon Seo.Towards continual knowledge learning of language models.InInternational Conference on Learning Representations, 2022b.URLhttps: //openreview.net/forum?id=vfsRB5MImo9. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), p. 2567–2577, 2019. Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models.arXiv preprint arXiv:2202.06539, 2022. Mojtaba Komeili, Kurt Shuster, and Jason Weston. Internet-augmented dialogue generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 8460–8478, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.579. URLhttps://aclanthology.org/ 2022.acl-long.579. Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Tomas Kocisky, Sebastian Ruder, et al. Mind the gap: Assessing temporal generalization in neural language models.Advances in Neural In- formation Processing Systems, 34:29348–29363, 2021. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 8424–8445, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.577. URLhttps://aclanthology.org/ 2022.acl-long.577. Eric Lehman, Sarthak Jain, Karl Pichotta, Yoav Goldberg, and Byron C. Wallace. Does bert pre- trained on clinical notes reveal sensitive data?InNAACL-HLT, p. 946–959, 2021. URL https://doi.org/10.18653/v1/2021.naacl-main.73. 12 Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. InInternational Conference on Learning Representations, 2022. URLhttps://openreview.net/forum?id=bVuP3ltATMz. Pierre Lison, Ildik ́ o Pil ́ an, David Sanchez, Montserrat Batet, and Lilja Øvrelid. Anonymisation models for text data: State of the art, challenges and future directions. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), p. 4188–4203, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.323. URLhttps://aclanthology.org/2021.acl-long.323. Jimit Majmudar, Christophe Dupuy, Charith Peris, Sami Smaili, Rahul Gupta, and Richard Zemel. Differentially private decoding in large language models.arXiv preprint arXiv:2205.13621, 2022. Alessandro Mantelero. The eu proposal for a general data protection regulation and the roots of the ‘right to be forgotten’.Computer Law & Security Review, 29(3):229–235, 2013. Ronak Mehta, Sourav Pal, Vikas Singh, and Sathya N Ravi. Deep unlearning via randomized condi- tionally independent hessians. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 10422–10431, 2022. Denis Paperno, Germ ́ an Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fern ́ andez.The LAMBADA dataset: Word prediction requiring a broad discourse context. InProceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1525–1534, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16-1144. URLhttps://aclanthology.org/P16-1144. Jasmine Park. South korea: The first case where the personal information protection act was applied to an ai system, May 2021. URLshorturl.at/bfOP2. Fabio Petroni, Tim Rockt ̈ aschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? InEMNLP, 2019. Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. Towards empathetic open- domain conversation models: A new benchmark and dataset. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, p. 5370–5381, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1534. URLhttps: //aclanthology.org/P19-1534. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106, 2021. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. In2017 IEEE symposium on security and privacy (SP), p. 3–18. IEEE, 2017. Eric Michael Smith, Mary Williamson, Kurt Shuster, Jason Weston, and Y-Lan Boureau. Can you put it all together: Evaluating conversational agents’ ability to blend skills. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 2021–2030, On- line, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.183. URLhttps://aclanthology.org/2020.acl-main.183. Kushal Tirumala, Aram H Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models.arXiv preprint arXiv:2205.10770, 2022. Eduard Fosch Villaronga, Peter Kieseberg, and Tiffany Li. Humans forget, machines remember: Artificial intelligence and the right to be forgotten.Computer Law & Security Review, 34(2): 304–313, 2018. 13 Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. Differentially private fine-tuning of language models. InInternational Conference on Learning Representations, 2022. URLhttps://openreview.net/forum?id=Q42f0dfjECO. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence?arXiv preprint arXiv:1905.07830, 2019. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068, 2022. Hattie Zhou, Ankit Vani, Hugo Larochelle, and Aaron Courville. Fortuitous forgetting in con- nectionist networks. InInternational Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=ei3SY1_zYsE. AFULLRESULTS We provide all of the results for the 5 random samplings for our main experimental setting in Table 5 and the full results for the domain analysis setting in Table 6. We also provide the evaluation of the 4 dialogue tasks fors= 32for all model sizes in Table 7 BMEASURINGPILE ANDWIKITEXTPERPLEXITY Table 8 shows the results of measuring perplexity on 500 samples from the validation set of Pile and Wikitext corpora on the LMs from the main experimental setting (Table 2). Results show that LMs that underwent knowledge unlearning show higher perplexity while the main experimental table (Table 2) does not show degradation of performance on 9 different LM benchmarks. We believe the discrepancy to be due to the inherent attributes of performing unlearning: since we are doing gradientascent, we are likelysofteningthe probability to generate each token from the vocabulary, giving it a more uniform distribution that will inevitably result in a higher perplexity. However, since it does not show much degradations in the LM benchmarks, it also means that theargmaxof the most likely token to be generated has not changed much. However, further exploration of what exactlyknowledge unlearningdoes to the representations of the LM should be done in future work. CCOMPUTATIONCOMPARISONBETWEENDEDUPLICATION AND KNOWLEDGEUNLEARNING We show the FLOPs of pretraining OPT denoted as DEDUPLICATIONand the average FLOPs of performing knowledge unlearning untils= 32token sequences reach the Forgetting Threshold denoted as UNLEARNINGin Table 9. We calculate FLOPs by (6 x Total Training Tokens x Parameter Size) following Brown et al. (2020). DVARYING THELEARNINGRATE In Figure 4, we show the results of varying the learning rate for knowledge unlearning where we fix the total epoch to 10 and perform 3 random runs withs= 32on the GPT-NEO1.3B. Overall, we observe that higher learning rates lead to faster forgetting, but with substantial LM performance degradation. While lower learning rates retain the LM performance, they fail to meet the Forgetting Threshold within 10 epochs. Thus, we set the learning rate to 5e-5 for our experiments to get the best trade-off. ETEXTEXAMPLE FROMEACHDOMAIN We show an example token sequence from each of the 8 domains used for the analysis section in Table 10. 14 02468 Epochs 0 20 40 60 Avg. LM Performance Avg. EL 10 (a) 1E-4 02468 Epochs 0 20 40 60 Avg. LM Performance Avg. EL 10 (b) 8E-5 02468 Epochs 0 20 40 60 Avg. LM Performance Avg. EL 10 (c) 5E-5 02468 Epochs 20 40 60 Avg. LM Performance Avg. EL 10 (d) 3E-5 02468 Epochs 20 40 60 Avg. LM Performance Avg. EL 10 (e) 1E-5 Figure 4:Varying the learning rate for unlearning the GPT-NEO1.3B withs= 32. We report the average of 3 random samplings and display the standard deviations as the shaded regions. Red dotted lines denote the memorization accuracy forgetting threshold of the 1.3B model reported in Table 1. FMORE EXAMPLES OF PERFORMING EXTRACTION ATTACKS In addition to the extraction attack example shown in the analysis section, we provide 3 additional examples to provide readers with more empirical examples of how knowledge unlearning ensures protection against extraction attacks in Table 11. GADDITIONALRESULTS OFSEQUENTIALKNOWLEDGEUNLEARNING We show how the EL 10 of each individual chunks and the average LM performance change as we perform sequential unlearning in Figure 5. Results show that the chunks that are forgotten stay forgotten and that later chunks are forgotten much faster (one or two epochs) compared to the initial chunk. We hypothesize that this might be because of the similarity of the token sequences from the 15,000 examples from the Training Extraction Challenge Benchmark. Also, this result hints at the generalizationof unlearning, which we do not further explore because of the scope of this work. 15 0369121518 Epochs 0 10 20 30 40 (a) 125M 03691215 Epochs 0 20 40 60 (b) 1.3B 02468 Epochs 0 20 40 60 EL 10 1 EL 10 2 EL 10 3 EL 10 4 Avg. LM Performance (c) 2.7B Figure 5:Additional results of sequential unlearning for GPT-NEO125M, 1.3B, and 2.7B. Red dotted lines denote the memorization accuracy forgetting threshold reported of each model in Table 1. HTHEEFFECT OFVARYINGNFOREXTRACTIONLIKELIHOOD(EL) METRIC First, we show the Extraction Likelihood (EL) Forgetting Threshold values for n=[5,10,20,40] by measuring the value on the 10,000 validation instances unseen during training in Table 12. Next, we show the average LM performance (on the 9 classification benchmarks) where we perform unlearn- ing on the LM on 32 samples until the target token sequences are forgotten (the EL & MA value are both lower than the threshold values) in Table 13. Performance shows the average of 5 random samplings. ILIMITATIONS While we provide a privacy guarantee through unlearning, our Forgetting Threshold is dependent on which data samples are chosen asD ′ . Furthermore, varying the prefix length can be seen as a na ̈ ıve way of varying the strength of the extraction attacks. In a real-world scenario, extraction attacks may be more complicated and may require other prevention methods. Also, we could not directly compare our approach with a Differential Privacy (DP) (Anil et al., 2021) approach because there are no open-sourced LMs pretrained with a DP algorithm. We could not replicate the pretrainig phase because of the heavy computational resources needed to pretrain an LM with DP which is estimated to require thousands of GPU hours. We leave this comparison for future work. Finally, a recent work (Carlini et al., 2022b) has suggested that machine unlearning (for the vision domain) can bring negative effects harming the privacy of other users. Future work should explore this phenomenon in the setting of performing unlearning on large language models as well. 16 Table 5:All of the individual runs for the Main Results Model (s) # EL 10 MAHella.Lamba.Wino.COPAARC-EARC-CPiqaMathQPubQAvg. Epoch Params (%)↓(%)↓(ACC)(ACC)(ACC)(ACC)(ACC)(ACC)(ACC)(ACC)(ACC)(ACC) NEO 125M30.977.428.237.651.862.045.622.063.322.557.643.4- ∆ ---+0.2+8.0+1.9+5.0+0.0+2.2+0.0+0.3+0.0+2.0- NEO+ UL + (s= 1) 125M3.128.128.141.052.562.043.221.063.022.857.643.514.0 125M0.027.628.124.950.867.042.323.762.821.957.642.110.0 125M0.027.128.142.152.563.044.120.362.622.557.643.75.0 125M 0.025.628.244.952.062.041.821.462.622.257.643.611.0 125M0.028.128.433.951.566.044.821.762.822.357.643.210.0 NEO+ UL + (s= 4) 125M0.928.827.844.151.952.037.419.760.522.357.641.516.0 125M0.028.627.42.549.459.038.623.160.521.243.836.219.0 125M3.628.827.733.451.855.037.721.061.022.357.640.820.0 125M2.628.927.629.952.450.036.519.060.322.257.639.518.0 125M0.028.427.66.749.761.042.522.761.021.450.638.116.0 NEO+ UL + (s= 8) 125M0.028.527.635.051.851.037.618.060.122.457.640.116.0 125M2.228.127.75.449.662.040.621.061.221.852.438.019.0 125M0.329.628.041.252.255.040.221.461.021.957.642.018.0 125M5.025.327.41.349.665.037.624.459.221.233.835.523.0 125M0.028.227.95.350.561.041.622.460.721.551.438.018.0 NEO+ UL + (s= 32) 125M0.328.427.242.353.756.038.121.059.722.457.642.020.0 125M0.827.127.017.152.453.034.020.059.821.557.638.018.0 125M0.224.127.345.651.950.038.620.759.622.657.641.513.0 125M3.028.727.52.649.259.037.721.458.420.946.835.920.0 125M0.728.527.344.553.054.039.020.359.522.557.642.015.0 NEO+ UL + (s= 128) 125M1.328.127.14.650.558.037.921.357.521.447.836.216.0 125M3.127.526.91.850.560.036.422.356.621.241.835.318.0 125M3.926.727.03.950.959.035.221.356.021.349.636.017.0 125M2.426.626.92.750.256.035.922.357.221.243.835.116.0 125M3.827.327.06.450.957.037.321.357.221.252.036.717.0 NEO 1.3B 67.692.237.057.454.870.056.625.870.421.953.849.8- ∆ ---+0.4+10.1+2.1+2.0+1.1+3.4+0.3+0.4+3.8+2.6- NEO+ UL + (s= 1) 1.3B0.027.636.852.154.772.055.927.869.721.553.049.39.0 1.3B0.030.236.654.654.969.055.426.870.721.753.449.26.0 1.3B0.029.736.758.255.470.056.125.469.922.053.249.74.0 1.3B0.032.237.152.453.768.056.124.470.121.854.248.68.0 1.3B0.027.637.360.155.670.057.525.170.021.755.250.310.0 NEO+ UL + (s= 4) 1.3B 0.030.337.348.354.470.055.029.269.920.656.049.012.0 1.3B 0.029.736.849.453.469.055.226.870.621.452.848.49.0 1.3B1.029.236.851.354.970.055.226.870.321.554.049.010.0 1.3B4.831.437.259.254.871.054.925.869.521.950.249.410.0 1.3B 1.731.837.058.454.471.057.724.770.222.054.049.99.0 NEO+ UL + (s= 8) 1.3B0.329.737.166.554.570.052.026.869.421.756.850.513.0 1.3B1.929.536.843.053.171.051.327.570.421.042.446.313.0 1.3B0.226.237.247.354.272.055.225.870.421.854.848.712.0 1.3B 3.132.037.457.654.370.056.126.869.821.554.849.814.0 1.3B1.432.037.157.454.571.057.026.170.021.954.249.911.0 NEO+ UL + (s= 32) 1.3B0.733.036.563.255.970.052.425.169.721.855.450.013.0 1.3B1.729.836.750.953.571.056.327.870.722.039.447.614.0 1.3B0.728.437.064.856.969.054.326.469.121.955.850.613.0 1.3B4.231.235.867.555.367.051.525.468.121.356.649.814.0 1.3B2.129.535.863.955.770.054.126.469.522.356.850.515.0 NEO+ UL + (s= 128) 1.3B0.424.531.154.255.269.053.224.766.121.956.448.06.0 1.3B4.919.827.82.254.869.050.923.357.921.855.840.48.0 1.3B4.230.230.641.655.169.054.426.063.822.155.046.46.0 1.3B2.923.627.68.852.968.044.518.957.721.657.439.79.0 1.3B1.323.128.548.655.569.048.821.662.322.257.646.08.0 NEO 2.7B70.493.440.862.256.475.059.625.473.021.457.052.3- ∆ ---+0.8+7.9+1.0+0.0+1.5+4.3+0.3+1.1+1.0+2.0- NEO+ UL + (s= 1) 2.7B0.03.040.862.256.672.055.726.473.121.857.651.810.0 2.7B0.023.640.556.854.474.059.626.172.821.356.651.38.0 2.7B0.027.640.662.557.075.059.124.773.021.556.652.26.0 2.7B0.020.640.560.355.874.058.925.873.021.757.251.910.0 2.7B0.029.740.662.256.472.058.027.172.221.257.451.99.0 NEO+ UL + (s= 4) 2.7B0.422.641.560.054.972.055.026.469.921.357.851.012.0 2.7B0.030.041.646.553.471.055.625.172.021.357.249.39.0 2.7B0.723.740.459.754.974.058.723.772.520.857.451.39.0 2.7B3.232.441.267.256.073.057.328.173.322.357.252.88.0 2.7B0.231.940.361.255.774.060.027.572.021.457.252.110.0 NEO+ UL + (s= 8) 2.7B0.329.541.264.655.471.052.927.169.521.758.051.310.0 2.7B2.126.440.648.752.967.055.025.872.121.857.249.011.0 2.7B0.531.241.154.155.074.059.325.172.522.157.451.211.0 2.7B1.933.840.765.757.472.058.427.172.621.957.052.58.0 2.7B0.020.440.060.755.873.060.128.572.521.557.252.211.0 NEO+ UL + (s= 32) 2.7B0.631.740.868.256.168.054.428.071.921.457.051.811.0 2.7B1.132.440.956.955.669.058.126.771.822.156.850.910.0 2.7B 1.229.041.565.856.968.059.327.072.022.357.852.311.0 2.7B3.429.939.770.157.768.054.829.771.622.057.652.411.0 2.7B1.931.941.461.656.673.061.126.472.721.757.052.411.0 NEO+ UL + (s= 128) 2.7B0.431.535.364.256.868.351.826.770.221.956.750.210.0 2.7B3.816.526.00.451.657.729.016.654.220.057.934.810.0 2.7B 0.631.434.958.955.269.254.824.770.022.557.749.89.0 2.7B2.231.131.322.950.662.540.018.260.821.340.938.78.0 2.7B4.729.033.556.555.066.351.923.668.622.457.748.49.0 17 Table 6:All of the individual runs for the Domain Analysis Results for GPT-NEO1.3B LM. Domains InitialFinalHella.Lamba.Wino.COPAARC-EARC-CPiqaMathQPubQAvg. EL 10 EL 10 (ACC)(ACC)(ACC)(ACC)(ACC)(ACC)(ACC)(ACC)(ACC)(ACC) INITIAL--37.057.454.970.056.625.870.421.953.849.8 FREELAW 64.64.837.353.554.168.057.527.170.521.554.049.3 52.02.437.362.954.267.052.926.169.221.554.449.5 60.615.236.842.054.567.056.625.170.121.751.447.2 55.213.837.351.453.569.055.426.870.521.954.648.9 69.524.137.451.453.271.054.926.170.021.853.048.7 GITHUB(CODE) 67.01.237.351.154.171.057.327.170.121.341.247.8 56.70.337.149.954.968.056.126.469.121.448.447.9 62.00.237.250.254.268.056.625.870.521.854.448.7 60.41.137.559.754.768.055.925.470.121.953.849.7 73.60.037.355.954.171.055.425.469.921.251.449.1 GITHUB(LICENSE) 87.50.237.557.454.568.056.826.470.121.853.849.6 74.30.037.348.954.170.057.127.170.721.748.448.4 70.70.036.440.653.170.055.225.470.221.849.046.9 74.80.037.360.354.869.055.927.170.021.555.650.2 71.80.037.052.654.368.056.826.169.522.052.248.7 ENRON EMAILS 81.60.036.459.855.269.053.627.569.021.954.849.7 70.30.037.254.954.568.057.525.470.122.451.849.1 74.20.037.156.355.068.055.625.169.821.654.249.2 83.90.036.755.254.869.055.925.470.421.752.249.0 76.80.036.960.054.668.056.428.169.921.552.449.7 BOOKS3 59.70.036.239.453.972.055.224.469.921.950.047.0 65.40.035.965.255.767.053.325.169.921.655.849.9 71.70.037.147.454.674.057.026.869.821.744.248.1 74.70.036.440.753.470.055.725.469.621.641.246.0 79.50.036.754.953.671.056.625.870.221.846.048.5 PILE C 74.90.035.330.753.068.055.226.469.922.150.445.7 68.00.036.345.953.472.055.627.169.621.751.448.1 71.60.036.348.952.970.055.926.470.221.951.848.3 57.80.034.066.355.769.049.926.169.021.457.449.9 66.60.036.437.754.073.054.528.169.922.149.247.2 USPTO BACKGROUNDS 53.70.030.748.453.468.039.022.064.220.755.244.6 56.70.031.019.450.669.036.924.163.321.233.438.8 64.90.036.051.454.168.050.824.470.022.156.648.2 54.60.035.557.255.165.052.023.768.922.056.248.4 67.20.035.347.454.365.050.825.868.421.750.246.5 PUBMED CENTRAL 73.80.035.739.053.569.055.625.169.621.944.246.0 75.10.036.136.353.269.054.125.169.822.644.445.6 67.40.037.047.554.071.056.324.469.921.148.447.7 71.10.037.255.355.668.057.024.770.022.051.049.0 71.90.036.844.454.171.055.024.770.622.143.846.9 Table 7:All of the individual runs fors= 32for the dialogue tasks in the Main Results. Model (s) #EL 10 MAWoWEDBSTWoIAvg. Epoch Params(%)↓(%)↓(F1)(F1)(F1)(F1)(F1) NEO 125M30.977.48.48.49.611.29.4- ∆ - --+0.0+0.0+0.0+0.0+0.0- NEO+ UL + (s= 32) 125M0.328.41.61.80.91.81.520.0 125M0.827.10.10.10.00.00.018.0 125M 0.224.16.96.77.07.97.113.0 125M3.028.72.12.51.42.32.120.0 125M 0.728.52.03.51.32.22.215.0 NEO 1.3B67.692.29.610.512.213.711.5- ∆ - --+2.3+0.0+0.0+0.0+0.0- NEO+ UL + (s= 32) 1.3B0.733.010.08.49.310.99.613.0 1.3B1.729.811.98.410.612.410.814.0 1.3B0.728.410.08.39.510.89.613.0 1.3B 4.231.26.454.96.85.814.0 1.3B2.129.56.95.95.97.56.515.0 NEO 2.7B 70.493.49.210.912.413.611.5- ∆ ---+3.8+1.8+0.0+0.5+1.5- NEO+ UL + (s= 32) 2.7B0.631.710.88.69.611.110.111.0 2.7B 1.132.411.99.711.512.111.310.0 2.7B1.229.012.410.512.013.312.111.0 2.7B3.429.98.88.28.410.38.911.0 2.7B 1.931.913.012.712.414.113.011.0 18 Table 8:Measuring perplexity on Pile and Wikitext corpora for the main unlearning experiments (Table 2). Model #PileWikitext Params(PPL)↓(PPL)↓ NEO125M17.8338.27 NEO+ UL125M34.0275.24 NEO+ UL + 125M577.561986.07 OPT125M32.2638.74 NEO1.3B11.4618.63 NEO+ UL1.3B15.5620.26 NEO+ UL + 1.3B15.8326.82 OPT1.3B19.5519.39 NEO2.7B10.4416.15 NEO+ UL2.7B11.3216.84 NEO+ UL + 2.7B17.9321.13 OPT2.7B17.8116.81 Table 9:Training compute comparison of methods mitigating privacy risks in LMs for sizes 125M, 1.3B, and 2.7B measured via FLOPs. Method (Size)FLOPs DEDUPLICATION(125M)2.25E+20 UNLEARNING(125M)5.28E+13 DEDUPLICATION(1.3B)2.34E+21 UNLEARNING(1.3B)6.69E+14 DEDUPLICATION(2.7B)4.86E+21 UNLEARNING(2.7B)1.12E+15 19 Table 10:Examples from each of the 8 domains from the Pile corpora. DomainText FREELAW U. S. (2010) 1 Opinion of the Court NOTICE: This opinion is subject to formal revision before publication in the preliminary print of the United States Reports. Readers are requested to notify the Reporter of Decisions, Supreme Court of the United States, Washington, D. C. 20543, of any typographical or other formal errors, in order that corrections may be made before the preliminary print goes to press. SUPREME COURT OF THE UNITED STATES GITHUB(CODE) = pc func (iov *Iovec) SetLen(length int)iov.Len = uint64(length)func (msghdr *Msghdr) SetControllen(length int)msghdr.Controllen = uint64(length)func (cmsg *Cmsghdr) SetLen(length int)cmsg.Len = uint64(length)//sys poll(fds *PollFd, nfds int, timeout int) (n int, err error) func Poll(fds []PollFd, timeout int) (n int, err error)if len(fds) == 0return poll(nil, 0, timeout)return poll(&fds[0], len(fds), timeout) GITHUB(LICENSE) ## Permission is hereby granted, free of charge, to any person obtaining a copy # of this software and associated documentation files (the ”Software”), to deal # in the Software without restriction, including without limitation the rights # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell # copies of the Software, and to permit persons to whom the Software is # furnished to do so, subject to the following conditions: ## The above copyright notice and this permission notice shall be included in # all copies or substantial portions of the Software. ## THE SOFTWARE IS PROVIDED ”AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, # FITNESS FOR A PARTICULAR PURPOSE ENRON EMAILS To: Hedy Govenar hgovenar@govadv.com, Mike Day MDay@GMSSR.com, Bev Hansen bhansen@lhom.com, Jeff Dasovich jdasovic@ enron.com, Susan J Mara smara@enron.com, Joseph Alamo JAlamo@enron.com, Paul Kaufman paul.kaufman@enron.com, David Parquet David.Parquet@enron.com, Rick Johnson rick.johnson@enron.com, Marcie Milner mmilner@enron.com, Sandra McCubbin Sandra.McCubbin@enron.com, Tim Belden Tim.Belden@enron.com BOOKS3 About the Publisher Australia HarperCollins Publishers (Australia) Pty. Ltd. 25 Ryde Road (PO Box 321) Pymble, NSW 2073, Australia http://w.harpercollinsebooks.com.au Canada HarperCollins Publishers Ltd. 55 Avenue Road, Suite 2900 Toronto, ON, M5R, 3L2, Canada http://w.harpercollinsebooks.ca New Zealand HarperCollins Publishers (New Zealand) Limited P.O. Box 1 Auckland, New Zealand http://w.harpercollinsebooks.co.nz United Kingdom HarperCollins Publishers Ltd. 77-85 Fulham Palace Road London, W6 8JB, UK http://w.harpercollinsebooks.co.uk PILE C This website and its associated newspaper adheres to the Independent Press Standards Organisation’s Editors’ Code of Practice. If you have a complaint about editorial content which relates to inaccuracy or intrusion, then contact the Editor by clicking here. If you remain dissatisfied with the response provided then you can contact the IPSO by clicking here. Bury Free Press provides news, events and sport features from the Bury St Edmunds area. For the best up to date information relating to Bury St Edmunds and the surrounding areas visit us at Bury Free Press regularly or bookmark this page. For you to enjoy all the features of this website Bury Free Press requires permission to use cookies. Find Out More What is a Cookie? What is a Flash Cookie? Can I opt out of receiving Cookies? USPTO BACKGROUNDS The pharmaceutical formulations of the present invention, which may conveniently be presented in unit dosage form, may be prepared according to conventional techniques well known in the pharmaceutical industry. Such techniques include the step of bringing into association the active ingredients with the pharmaceutical carrier(s) or excipient(s). In general the formulations are prepared by uniformly and intimately bringing into association the active ingredients with liquid carriers or finely divided solid carriers or both, and then, if necessary, shaping the product. The compositions of the present invention may be formulated into any of many possible dosage forms such as, but not limited to, tablets, capsules, gel capsules, liquid syrups, soft gels, suppositories, and enemas. PUBMED CENTRAL I am pleased to inform you that your manuscript has been formally accepted for publication in PLOS Computational Biology. Your manuscript is now with our production department and you will be notified of the publication date in due course. The corresponding author will soon receiving a typeset proof for review, to ensure errors have not been introduced during production. Please review the PDF proof of your manuscript carefully, as this is the last chance to correct any errors. Please note that major changes, or those which affect the scientific understanding of the work, will likely cause delays to the publication date of your manuscript. Soon after your final files are uploaded, unless you have opted out, the early version of your manuscript will be published online. The date of the early version will be your article ́ s publication date. 20 Table 11:More examples performing extraction attacks on token sequences, showing knowledge unlearning guarantees protection against extraction attacks. Blue denotes the model generated text given the prefix of length 100 as input. For the extraction attack, we utilize a na ̈ ıve greedy decoding strategy. DomainStatusText PILE C Original James Gurney This daily weblog by Dinotopia creator James Gurney is for illustrators, plein-air painters, sketchers, comic artists, animators, art students, and writers. You’l find practical studio tips, insights into the making of the Dinotopia books, and first-hand reports from art schools and museums. CG Art Contact or by email:gurneyjourney (at) gmail.com Sorry, I can’t give personal art advice or portfolio reviews. If you can, it’s best to ask art questions in the blog comments. Permissions All images and text are copyright 2015 James Gurney and/or their respective owners. Dinotopia is a registered trademark of James Gurney. For use of text or images in traditional print media or for any commercial licensing rights, please email me for permission. However, you can quote images or text without Text Before James Gurney This daily weblog by Dinotopia creator James Gurney is for illustrators, plein-air painters, sketchers, comic artists, animators, art students, and writers. You’l find practical studio tips, insights into the making of the Dinotopia books, and first-hand reports from art schools and museums. CG Art Contact or by email:gurneyjourney (at) gmail.com Sorry, I can’t give personal art advice or portfolio reviews. If you can, it’s best to ask art questions in the blog comments. Permissions All images and text are copyright 2015 James Gurney and/or their respective owners. Dinotopia is a registered trademark of James Gurney. For use of text or images in traditional print media or for any commercial licensing rights, please email me for permission. However, you can quote images or text without Unlearning After James Gurney This daily weblog by Dinotopia creator James Gurney is for illustrators, plein-air painters, sketchers, comic artists, animators, art students, and writers. You’l find practical studio tips, insights into the making of the Dinotopia books, and first-hand reports from art schools and museums. CG Art Contact or by email:gurneyjourney (at) gmail.com I’ve been working on a CG art project for a while now, and I’ve been working on it for a while now. I’ve been working on it for a while now, and I’ve been working on it for a while now. I’ve been working on it for a while now, and I’ve been working on it for a while now. I’ve been working on a CG art project for a while now, and I’ve been working on it for a while Unlearning ENRON EMAILS Original Rick Shapiro rshapiro@enron.com, Jim Steffes james.d.steffes@enron.com, Alan Comnes acomnes@enron.com, Chris Calger ccalger@enron.com, Mary Hain mary.hain@enron.com, Joe Hartsoe Joe.Hartsoe@enron.com, Donna Fulton Donna.Fulton@enron.com, Steven Kean Steven.J.Kean@ enron.com, Karen Denne kdenne@enron.com, Beverly Aden beverly.aden@enron.com, Bill Votaw bill.votaw@enron.com, Carol Moffett carol. moffett@enron.com, Debora Whitehead deb Text Before Rick Shapiro rshapiro@enron.com, Jim Steffes james.d.steffes@enron.com, Alan Comnes acomnes@enron.com, Chris Calger ccalger@enron.com, Mary Hain mary.hain@enron.com, Joe Hartsoe Joe.Hartsoe@enron.com, Donna Fulton Donna.Fulton@enron.com, Steven Kean Steven.J.Kean@ enron.com, Karen Denne kdenne@enron.com, Beverly Aden beverly.aden@enron.com, Bill Votaw bill.votaw@enron.com, Carol Moffett carol. moffett@enron.com, Debora Whitehead Unlearning AfterRick Shapiro rshapiro@enron.com, Jim Steffes james.d.steffes@enron.com, Alan Comnes acomnes@enron.com, Chris Calger ccalger@enron.com, Mary Hain mary.hain@enron.com, Joe Hartsoe Joe.Hartsoe@enron.com, Donna Fulton Dabat, state+[D@calenergy.com]Unlearning PILE C Original ? About Me Alvin McEwen is 46-year-old African-American gay man who resides in Columbia, SC. McEwen’s blog, Holy Bullies and Headless Monsters, and writings have been mentioned by Americablog.com, Goodasyou.org, People for the American Way, PageOneQ.com, The Washington Post, Raw Story, The Advocate, Media Matters for America, Crooksandliars.com, Thinkprogress.org, Andrew Sullivan’s Daily Dish, Melissa Harris- Perry, The Last Word with Lawrence O’Donnell, Newsweek, The Daily Beast, The Washington Blade, and Foxnews.com. In addition, he is also a past contributor to Pam’s House Blend,Justice For All, LGBTQ Nation, and Alternet.org. He is a present contributor to the Daily Kos and the Huffington Post, He is the 2007 recipient of the Harriet Daniels Hancock Volunteer of the Year Award and Text Before ? About Me Alvin McEwen is 46-year-old African-American gay man who resides in Columbia, SC. McEwen’s blog, Holy Bullies and Headless Monsters, and writings have been mentioned by Americablog.com, Goodasyou.org, People for the American Way, PageOneQ.com, The Washington Post, Raw Story, The Advocate, Media Matters for America, Crooksandliars.com, Thinkprogress.org, Andrew Sullivan’s Daily Dish, Melissa Harris- Perry, The Last Word with Lawrence O’Donnell, Newsweek, The Daily Beast, The Washington Blade, and Foxnews.com. In addition, he is also a past contributor to Pam’s House Blend,Justice For All, LGBTQ Nation, and Alternet.org. He is a present contributor to the Daily Kos and the Huffington Post, He is the 2007 recipient of the Harriet Daniels Hancock Volunteer of the Year Award and Unlearning After ? About Me Alvin McEwen is 46-year-old African-American gay man who resides in Columbia, SC. McEwen’s blog, Holy Bullies and Headless Monsters, and writings have been mentioned by Americablog.com, Goodasyou.org, People for the American Way, PageOneQ.com, The Washington Post, Raw Story, The Advocate, Media Matters for America, Crooksandliars.com, Thinkprogress, and more. The British singer has been in the news for his recent singles, including “I’m Not Sure” and “What Makes You Beautiful.” The singer has been in the news for his recent singles, including “I’m Not Sure” and “What Makes You Beautiful.” The singer has been in the news for his recent singles, including “I’m Not Sure” Unlearning Table 12:Forgetting Threshold for GPT-NEOLMs for varyingn. Model (Size) EL 5 (%)EL 10 (%)EL 20 (%)EL 40 (%)MA(%) ThresholdThresholdThresholdThresholdThreshold GPT-NEO(1.3B)7.85 5.68 4.07 2.66 33.27 Table 13:The average of the 9 classification tasks for GPT-NEO+ UL + for the 1.3B LM when performing unlearning until the Forgetting Threshold for eachn. Model (Size)LM Avg. (Acc) EL 5 49.93 EL 10 49.93 EL 20 49.85 EL 40 49.88 21