Paper deep dive
LUME: LLM Unlearning with Multitask Evaluations
Anil Ramakrishna, Yixin Wan, Xiaomeng Jin, Kai-Wei Chang, Zhiqi Bu, Bhanukiran Vinzamuri, Volkan Cevher, Mingyi Hong, Rahul Gupta
Models: Mixtral-8x7B, OLMo-1B, OLMo-7B
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/12/2026, 7:29:39 PM
Summary
LUME is a multi-task benchmark designed to evaluate machine unlearning in Large Language Models (LLMs). It covers three distinct tasks: unlearning synthetic creative short novels, synthetic biographies with PII, and public biographies. The authors release fine-tuned 1B and 7B parameter OLMo models and evaluate several unlearning algorithms (GA, GD, KL, NPO), finding that current methods struggle to remove sensitive information without significantly degrading model utility.
Entities (5)
Relation Signals (3)
LUME → evaluates → LLM
confidence 95% · we develop a comprehensive new benchmark named LUME (LLM Unlearning with Multitask Evaluations) for unlearning creative, sensitive, and private content from LLMs.
OLMo → finetunedfor → LUME
confidence 95% · We fine-tuned 1B and 7B OLMo models on all three tasks and release them as unlearning candidates.
Gradient Ascent → testedon → LUME
confidence 95% · We test following popular unlearning algorithms on LUME
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Unlearning aims to remove copyrighted, sensitive, or private content from large language models (LLMs) without a full retraining. In this work, we develop a multi-task unlearning benchmark (LUME) which features three tasks: (1) unlearn synthetically generated creative short novels, (2) unlearn synthetic biographies with sensitive information, and (3) unlearn a collection of public biographies. We further release two fine-tuned LLMs of 1B and 7B parameter sizes as the target models. We conduct detailed evaluations of several recently proposed unlearning algorithms and present results on carefully crafted metrics to understand their behavior and limitations.
Tags
Links
Trouble viewing inline? Open PDF directly →
Full Text
29,766 characters extracted from source content.
Expand or collapse full text
LUME: LLM Unlearning with Multitask Evaluations Anil Ramakrishna 1 , Yixin Wan 2 , Xiaomeng Jin 3 , Kai-Wei Chang 1,2 , Zhiqi Bu 1 , Bhanukiran Vinzamuri 1 ,Volkan Cevher 1,4 ,Mingyi Hong 1,5 ,Rahul Gupta 1 1 Amazon AGI, 2 UCLA, 3 UIUC, 4 EPFL, 5 University of Minnesota aniramak@amazon.com Abstract Unlearning aims to remove copyrighted, sen- sitive, or private content from large language models (LLMs) without a full retraining. In this work, we develop a multi-task unlearn- ing benchmark (LUME) which features three tasks: (1) unlearn synthetically generated cre- ative short novels, (2) unlearn synthetic biogra- phies with sensitive information, and (3) un- learn a collection of public biographies. We further release two fine-tuned LLMs of 1B and 7B parameter sizes as the target models. We conduct detailed evaluations of several recently- proposed unlearning algorithms and present re- sults on carefully crafted metrics to understand their behavior and limitations. 1 Introduction Given government regulations, such as the Euro- pean Union’s GDPRright to be forgotten(GDP, 2018), legal actions from original content creators (Grynbaum and Mac, 2023; Mattei, 2023), and a need to remove misinformation or toxic content from LLMs, there is an increasing demand for ef- fective unlearning algorithms as retraining model from scratch is infeasible. We define effective un- learning algorithm as one which: (i) effectively removes information to be unlearned, (i) uses com- putation commensurate with the size of the data to be forgotten, and (i) retains model’s overall performance after unlearning so that it is similar to a model candidate trained without the data to be forgotten. To evaluate the performance of unlearning al- gorithms in LLMs, there is a need for compre- hensive benchmarks. While recent work, such as TOFU (Maini et al., 2024) and MUSE (Shi et al., 2024), provide promising first steps along this di- rection, they provide limited coverage focusing on synthetic question answers, and news/books re- spectively. Further, neither benchmarks cover Per- sonally Identifiable Information (PII) information which is an important use case for unlearning in LLMs. In this work, we develop a comprehensive new benchmark namedLUME(LLM Unlearning with Multitask Evaluations) for unlearning creative, sen- sitive, and private content from LLMs. Our bench- mark 1 features three distinct tasks: synthetically generated creative short novels (task #1), synthetic biographies with PII (task #2), and public biogra- phies (task #3) for an extensive assessment of un- learning algorithms.LUMEtests for unlearning of both full documents and QA pairs for each task, with unlearning effectiveness measured us- ing memorization, privacy leakage (via member- ship inference attack) and model utility tests. We evaluate several unlearning algorithms including current state of the art, and find that they do not yet effectively unlearn sensitive information with- out significantly degrading the model utility. Our benchmark is publicly available 2 ; we also release two fine-tuned model checkpoints (1 billion 3 and 7 billion 4 parameters in size). 2 LUME: A Multitask Unlearning Benchmark for LLMs Given an LLM fine-tuned on a text corpusD, our unlearning goal is to effectively remove informa- tion from a subsetF⊂D(i.e., the forget set) with computational effort proportional to its size. During unlearning, we only have access toFand another subsetR⊂D(i.e., the retain set) to ensure performance outsideFis preserved. 1 created as part of the LLM Unlearning shared task at SemEval 2025 2 github.com/amazon-science/lume-llm-unlearning 3 huggingface.co/llmunlearningsemeval2025organization/olmo- 1B-model-semeval25-unlearning 4 huggingface.co/llmunlearningsemeval2025organization/olmo- finetuned-semeval25-unlearning arXiv:2502.15097v3 [cs.CL] 27 Feb 2025 Figure 1: Examples of full documents and test prompts for the three tasks covered in LUME. 2.1 Benchmark Construction We developed three distinct tasks to provide a com- prehensive evaluation of LLM unlearning algo- rithms spanning creative documents, PII and bi- ographies. Figure 1 and Table 1 show example data and statistics of LUME, respectively. Task 1 (Synthetic creative documents):LLMs trained on Internet-scraped data are often exposed to copyrighted content, making unlearning a com- mon requirement. However, evaluating effective- ness of unlearning on only real creative documents (Shi et al., 2024; Eldan and Russinovich, 2023) is challenging as information to be removed may appear in other documents not being unlearned. For example, MUSE usesHarry Potterbooks as its forget set, but similar content may appear in Wikipedia and social media. Motivated by this, in this task, we only include synthetically generated short novels, created using Mixtral 8x7B (Jiang et al., 2023) 5 as our generator LLM. For each document, we randomly select a genre fromAction,Fantasy,Thriller,Comedy,Mys- tery,Science Fiction,Young AdultandRomance. One to four unique character names are gener- 5 mistral.mixtral-8x7b-instruct-v0:1on Amazon Bedrock. ated using a random name generator (pypi.org/ project/unique-names-generator), and loca- tions are generated from the city list of a random address generator (pypi.org/project/ random-address) for all genres exceptFantasy. ForFantasy, we sample unique fantasy city names using aDungeons and Dragonstown generator (perchance.org/dndtowngen). Given this infor- mation, we prompt the Mixtral model to create a short story with 150-200 words. To validate the generated stories, we conducted manual reviews where each short story was reviewed by two dif- ferent authors of this work, and filtered out stories with similar content to prior reviewed stories. Our final dataset contains 393 unique short stories. Task 2 (Synthetic biographies with sensitive PII): We use rule based heuristics to generate personal biographies with following PII fields: a randomly generated name, a birthday randomly sampled be- tween 01/01/1964 and 01/01/1991, a fake Social Security number (SSN) within the range 900-x- x (which can never belong to a real person (ssa, 2011)), a random phone number, an email address of the formfirstname_lastname@me.comand a ForgetRetain Task 1199194393 Task 2203202405 Task 3295294589 6976901,387 Table 1: Number of unique documents for both data subsets within each task. For each document, we create multiple regurgitation and knowledge datasets leading to 4,394 unique examples. non-existent physical home addresses obtained by combining a random street address from a US state with an alternate city and zip-code from a different state. For each synthetic individual, we prompt the Mixtral model to create a short biography by including the fictitious PII information. Task 3 (Real biographies):To evaluate effective- ness of unlearning on real data, we include real biographies as the third task. Specifically, we sam- pled biographies spanning 100 to 200 words from Wikipedia documents released in the Dolma (Sol- daini et al., 2024) v1.6 corpus, which was part of the training dataset for the OLMo models (Groen- eveld et al., 2024) we fine-tuned for this task. 2.2 Unlearning Model Candidates We fine-tuned 1B (OLMo-1B-0724-hf) and 7B (OLMo-7B-0724-Instruct-hf)OLMomod- els (Groeneveld et al., 2024) on all three tasks and release them as unlearning candidates. We selected OLMo because of its permissive license and open sourced training dataset (with logs) which enables downstream task specific analyses of model behavior. 2.3 Evaluation We use following metrics for detailed evaluation. Regurgitation Rate (r):We create asentence com- pletionprompt for each document by sampling a random position in second half of the document with the sentences before it as the input. We com- pute ROUGE-L (Lin, 2004) scores for the model generated outputs with respect to the expected sen- tence completions. Knowledge Test Accuracy (t):We create aques- tion answeringprompt for each document using an agentic workflow for Tasks 1 and 3 where we prompt the data generator LLM (see Appendix D) with few-shot Chain of Thought prompting (Wei et al., 2022) and construct an unambiguous ques- tion with a single concise answer. We verify the quality of QA pairs using three verification LLMs. 6 We discard QA samples if any of the verification LLMs are unable to answer the question accurately with the corresponding document. For Task 2, we use template based heuristics to frame 5 distinct questions corresponding to the PII fields, of the form:What is the birth date of John Smith?. For all QA prompts, we use case insensitive exact match between model output and the groundtruth to mea- sure prediction accuracy. Membership Inference Attacks (MIA) (m):We use the black-box MIA attack framework from (Duan et al., 2024) to implement Loss based at- tacks to assess data leakage risk after unlearning. We use a subset of the memorized forget set of biographies from Task 3 as the member set and a disjoint sample of similar biographies not exposed to the model as the non-member set. Model Utility (u):We also test for overall model utility on MMLU (Hendrycks et al., 2021), a gen- eral benchmark for LLM utility. 3 Experiments We benchmark several unlearning approaches on LUMEand discuss our observations. Baseline Unlearning Algorithms:We test fol- lowing popular unlearning algorithms onLUME (detailed review is in the Appendix). • Gradient Ascent (GA)reverses the gradient direction on the forget setFto steer the model away this information. •Gradient Difference (GD)(Liu et al., 2022) augments the gradient ascent objective applied onFwith a gradient descent objective onR. •KL Regularization (KL)(Maini et al., 2024) augments the gradient ascent objective with a regularization term which minimizes the KL divergence with respect to the original model. •Negative Preference Optimization (NPO) (Zhang et al., 2024) uses a modified version of Direct Preference Optimization, adapted to remove the sensitive information fromF. Similar to TOFU and MUSE, we run each algo- rithm for 10 epochs with learning rate of1e−5 and batch size of32. Results:Figure 2 highlights epoch wise perfor- mance of each unlearning algorithm on forget and 6 We use Claude 3 (anthropic.claude-3-sonnet-20240229- v1:0), Titan Text Express (amazon.titan-text-express-v1) and Mixtral 8x7B for verification 0246810 Unlearning Epoch 0.0 0.2 0.4 0.6 0.8 1.0 Score Metric Reg Kno Split retain forget 0246810 Unlearning Epoch 0.0 0.2 0.4 0.6 0.8 1.0 Score Metric Reg Kno Split retain forget 0246810 Unlearning Epoch 0.0 0.2 0.4 0.6 0.8 1.0 Score Metric Reg Kno Split retain forget 0246810 Unlearning Epoch 0.4 0.6 0.8 1.0 Score Metric Reg Kno Split retain forget 0246810 Unlearning Epoch 0.0 0.2 0.4 0.6 0.8 1.0 Score Metric Reg Kno Split retain forget 0246810 Unlearning Epoch 0.0 0.2 0.4 0.6 0.8 1.0 Score Metric Reg Kno Split retain forget 0246810 Unlearning Epoch 0.0 0.2 0.4 0.6 0.8 1.0 Score Metric Reg Kno Split retain forget 0246810 Unlearning Epoch 0.0 0.2 0.4 0.6 0.8 1.0 Score Metric Reg Kno Split retain forget 0246810 Unlearning Epoch 0.0 0.2 0.4 0.6 0.8 1.0 Score Metric Reg Kno Split retain forget (a) GA 0246810 Unlearning Epoch 0.0 0.2 0.4 0.6 0.8 1.0 Score Metric Reg Kno Split retain forget (b) GD 0246810 Unlearning Epoch 0.0 0.2 0.4 0.6 0.8 1.0 Score Metric Reg Kno Split retain forget (c) KL 0246810 Unlearning Epoch 0.2 0.4 0.6 0.8 1.0 Score Metric Reg Kno Split retain forget (d) NPO Figure 2: Performance onretainandforgetsubsets for benchmarked unlearning algorithms for Tasks 1 to 3 (respectively from top to bottom). Reg: Regurgitation Rate (r), Kno: Knowledge Accuracy (t). Split refers to data subset (forget or retain) used in evaluations. 0246810 Unlearning Epoch 0.2 0.4 0.6 0.8 1.0 Score Approach GD KL GA NPO Figure 3: MIA rates (m) per epoch. retain subsets. 7 Across all tasks and on both for- get/retain sets, at epoch 0 all metrics reveal perfect regurgitation, highlighting complete memorization by the fine-tuned models (without a drop in model utility as shown in Figure 4 where the performance starts with baseline MMLU levels for OLMo 7B). As evidenced by the rapid drop in both regurgita- tion and knowledge scores as unlearning proceeds, none of the algorithms were successful in achiev- ing the joint objectives of unlearning the forget set while retaining information from the retain set. Ex- cept NPO, all the approaches reach zero on both metrics across all three tasks, suggesting substan- 7 due to space limitations, we present results only on the 7B model here. tial degradation in model quality. NPO performs relatively better but also trends towards zero. The observed variance in unlearning performance for the three tasks suggests varying levels of unlearn- ing difficulty for the samples from each task which was recently observed in (Zhao et al., 2024). For GD, while performance drops rapidly on both forget and retain sets, performance on the retain set starts increasing with time. This is be- cause of the objective used in GD which reduces the prediction loss on the retain set while jointly increasing loss on the forget set. As training pro- ceeds, the impact of the gradient descent objective which increases memorization of the retain set. Privacy Leakage:Figure 3 highlights the MIA success rates (AUC) for the unlearned checkpoints after each epoch. Initially, all models start with perfect memorization and hence have 100% attack success rates, but as unlearning proceeds, GA, GD and KL drop to the desired attack success rate of 50% (i.e. random chance levels), with GA observed to have the fastest drop. However, NPO attack suc- cess rates remain high after 10 epochs, suggesting that this approach does not truly remove the un- learned information and is vulnerable to privacy leakage from such attacks post unlearning. On the other hand, the MIA rates for GD continue drop- 0246810 Unlearning Epoch 0.25 0.30 0.35 0.40 0.45 0.50 Score Approach GD KL GA NPO Figure 4: MMLU rates (u) per epoch. ping below 0.5, suggesting over-unlearning beyond epoch 7. Impact on Utility:We report aggregate scores among all 57 tasks of MMLU in Figure 4. We observe considerable performance drops in all ap- proaches, highlighting the challenge in unlearning sensitive information without impacting model util- ity. GA had the highest drop suggesting substantial model degradation (owing to its unbounded loss term), followed by KL, GD and NPO. 4 Related Work Various machine unlearning methods have been proposed for removing knowledge from LLMs (Zhang et al., 2024; Pawelczyk et al., 2024; Chen and Yang, 2023). However, most of them report results on small sets such as (Eldan and Russinovich, 2023). Recently, (Maini et al., 2024) and (Shi et al., 2024) proposed unlearning benchmarks (with various evaluation metrics), but they carry key limitations we address here. We provide more detailed discussions comparing LUMEwith these works in Appendix A. 5 Conclusion We proposeLUME, a new benchmark covering three distinct tasks to evaluate unlearning in LLMs. Detailed experiments reveal the challenge pre- sented by our benchmark since most algorithms fail to sufficiently unlearn the forget set without substantial degradations on the retain set and model utility. We hope our benchmark spurs further de- velopments in LLM unlearning research. Limitations and Future Work (Carlini et al., 2022) show that the risk of mem- orization increases with large model size. How- ever, due to computational limitations and easy availability of large public LLMs, we only pro- vide finetuned checkpoints for 1B and 7B OLMo, and defer release of larger models to future work. Moreover, licensing restrictions prevent us from releasing fine-tuned models based on few publicly available LLMs such as LLaMa (lla, 2023). We acknowledge that LLM-generated data can exhibit specific biases found in their training data set. We partially mitigate this by seeding the gener- ation prompt with pre-sampled character and loca- tion names to ensure diversity in generated content. We also conducted manual evaluations of the gen- erated creative content to ensure its quality. Ethical Considerations Task 2 deals with sensitive PII information which warrants careful considerations to avoid privacy leakage of individuals. We avoid this risk entirely by carefully designing the generation process so that it closely mimics real individuals, despite be- ing generated synthetically. We also ensure all the tools used in generating our benchmark data are open sourced, thereby avoiding any licensing re- strictions. References 2018. Art. 17 gdprright to erasure (‘right to be for- gotten’).https://gdpr-info.eu/art-17-gdpr/. Accessed: 2024-03-29. 2023. Llama 2 community license agreement.https: //ai.meta.com/llama/license/. Zhiqi Bu, Xiaomeng Jin, Bhanukiran Vinzamuri, Anil Ramakrishna, Kai-Wei Chang, Volkan Cevher, and Mingyi Hong. 2024. Unlearning as multi-task op- timization: A normalized gradient difference ap- proach with an adaptive learning rate.Preprint, arXiv:2410.22086. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural lan- guage models. InThe Eleventh International Confer- ence on Learning Representations. Jiaao Chen and Diyi Yang. 2023. Unlearn what you want to forget: Efficient unlearning for LLMs. InPro- ceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pages 12041– 12052, Singapore. Association for Computational Linguistics. Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. 2024. Do membership inference attacks work on large language models? InConference on Language Modeling (COLM). Ronen Eldan and Mark Russinovich. 2023. Who’s harry potter? approximate unlearning in llms.Preprint, arXiv:2310.02238. Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bha- gia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muen- nighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Nis- hant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Han- naneh Hajishirzi. 2024. Olmo: Accelerating the sci- ence of language models.Preprint. Michael M. Grynbaum and Ryan Mac. 2023. The times sues openai and microsoft over a.i. use of copyrighted work. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. 2021. Measuring massive multitask language understanding.Proceedings of the International Con- ference on Learning Representations (ICLR). Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b.Preprint, arXiv:2310.06825. Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. InText Summariza- tion Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. Bo Liu, Qiang Liu, and Peter Stone. 2022. Continual learning and private unlearning. InConference on Lifelong Learning Agents, pages 243–254. PMLR. Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. 2024. Tofu: A task of fictitious unlearning for llms.Preprint, arXiv:2401.06121. Shanti Escalante-De Mattei. 2023. Artists are suing artificial intelligence companies and the lawsuit could upend legal precedents around art. Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. 2024. In-context unlearning: Language models as few shot unlearners. InICML. Anil Ramakrishna, Rahul Gupta, Jens Lehmann, and Morteza Ziyadi. 2023. INVITE: a testbed of au- tomatically generated invalid questions to evaluate large language models for hallucinations. InFind- ings of the Association for Computational Linguis- tics: EMNLP 2023, pages 5422–5429, Singapore. Association for Computational Linguistics. Anil Ramakrishna, Jimit Majmudar, Rahul Gupta, and Devamanyu Hazarika. 2024. Llm-pieval: A bench- mark for indirect prompt injection attacks in large language models. Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Mal- ladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A. Smith, and Chiyuan Zhang. 2024. Muse: Machine unlearning six-way evaluation for language models.Preprint, arXiv:2407.06460. Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bo- gin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. 2024. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Re- search.arXiv preprint. ssa. 2011.Social security is changing the way ssns are issued.https://w.ssa.gov/kc/ SSAFactSheet--IssuingSSNs.pdf.Accessed: 2024-10-07. Eleni Triantafillou, Fabian Pedregosa, Jamie Hayes, Peter Kairouz, Isabelle Guyon, Meghdad Kur- manji, Gintare Karolina Dziugaite, Peter Triantafil- lou, Kairan Zhao, Lisheng Sun Hosoya, Julio C. S. Jacques Junior, Vincent Dumoulin, Ioannis Mitliagkas, Sergio Escalera, Jun Wan, Sohier Dane, Maggie Demkin, and Walter Reade. 2023. Neurips 2023 - machine unlearning. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2024. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits rea- soning in large language models.Advances in neural information processing systems, 35:24824–24837. Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. 2024. A survey on large lan- guage model (llm) security and privacy: The good, the bad, and the ugly.High-Confidence Computing, 4(2):100211. Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024. Negative preference optimization: From catas- trophic collapse to effective unlearning.Preprint, arXiv:2404.05868. Kairan Zhao, Meghdad Kurmanji, George-Octavian B ̆ ar- bulescu, Eleni Triantafillou, and Peter Triantafillou. 2024. What makes unlearning hard and what to do about it.Preprint, arXiv:2406.01257. A Expanded related work Given the nascent stage of unlearning research in LLMs, few prior works exist which address the task of robustly evaluating the success of unlearn- ing. (Triantafillou et al., 2023) presented a new challenge task in which the goal was to to unlearn information contained in select images within the task of image based age prediction. While success- ful, the specific task addressed in this challenge was narrow, focusing only on image based age pre- diction - a classification problem with 10 classes with limited applicability in the unbounded text generation task of large language models. But the growing interest in LLMs and their tendency to gen- erate unsafe (Wei et al., 2024), private (Yao et al., 2024) or security violating (Ramakrishna et al., 2024) content necessitates a distinct and focused evaluation benchmark for unlearning. (Maini et al., 2024) released a new evaluation framework named TOFU which partially addressed this task of evaluating LLM unlearning algorithms. Their framework was evaluated on question answer- ing task applied on biographies of synthetically cre- ated fake authors. They train target models on this synthetic data and evaluate the ability of unlearn- ing algorithms to forget a portion of this synthetic dataset. While being a promising first step, this work has a few key limitations: unlearning the tar- geted information required for the QA task does is unlikely to cause loss of any other substantial infor- mation, specially linguistic attributes such as gram- mar. Further, this work leverages GPT4 to generate the synthetic content, which may have downstream licensing implications owing to GPT4’s proprietary license. More recently, (Shi et al., 2024) released a bench- mark named MUSE which evaluated model un- learning using real data set for containing news documents and Harry Potter book chapters. This benchmark released detailed evaluation metrics to robustly evaluate the unlearning algorithms. How- ever since it only leverages real data set the bench- mark does not provide a clean test bed to evaluate model performance. Specifically, the information contained in the unlearn documents may also ap- pear in other disjoint training documents, limiting the effectiveness of unlearning. While the TOFU benchmark mentioned before avoids this by only using synthetic documents, the data set coverage is rather limited (it only containts biographic infor- mation). The benchmark developed in our work addresses both these shortcomings together and presents a single holistic testbed to evaluate model unlearning in LLMs. Further, our benchmark was created using a largely automated process, and sup- ports reinstantiation for new applications similar to (Ramakrishna et al., 2023). B Long Form Example Generation Model Input: Create a short novel of at least 150 words. The novel should be from fantasy genre and set in the city of Atlantis. It should include following characters: Kyle , Stan , Kenny and Eric. Model Output: A short story about four friends lost in the magical realm below the oceans , known to ousiders as Atlantis. Kyle had always held a deep fascination for the deep blue ocean , and this naturally led him to take up a major in oceanic studies ... C Short Form Example Generation Model Input: Create a biography for Jon Smith with date of birth: 1/2/1989 , SSN: 900123456 , phone number: 0987654321 , email: jon_smith@me.com , home address: 10 Summertime Lane , New York City , NY , USA. Model Output: Jon Smith was both in New York City on the first of February in 1989 ,... D Question Generation Prompt Model Input: You are given a short story. First , find all the proper nouns in this story. If it does not contain a proper noun , say "I can't use this statement since it does not contain any proper nouns .". If it contains proper nouns , use your reasoning to create an unambiguous question , for which there would be *only* one answer. Give a concise answer (i.e. one word or phrase) which accurately answers the question. If you cannot create such an unambiguous question , say "I'm unable to create an unambiguous question for this story ". Use the examples below for reference. Examples: 1. Example #1 2. Example #2 3. Example #3 4. Example #4 5. Example #5 Here's the story: <input_story >. Generate a question with an unambiguous answer using this story. E Further details on Unlearning Algorithms We review unlearning methods tested in this paper in the following. •Gradient Ascent: This is a straightforward algorithm for model unlearning where we re- verse the direction of model update by flip- ping the sign in gradient descent, in order to steer the model away from the sensitive model outputs in the forget set. While easy to imple- ment, this approach has a significant drawback since the gradient ascent training objective is unbounded, which can lead to model diver- gence with nonsensical outputs for all inputs. The loss term in this algorithm reverses sign of the standard training objective and is applied only on the forget setFas shown below. −L(F;θ) • Gradient Difference(Liu et al., 2022): In this approach, we augment the gradient ascent objective applied on forget set, by adding a gradient descent objective on the retain set. By jointly optimizing on both sets, we steer the model away from regurgitating the sen- sitive information from the retain set, while ensuring it does not lose performance in the re- tain set. Despite being a promising alternative to Gradient Ascent, this quality of model per- formance on non-sensitive dataset depends on the size of the retain set used in model train- ing, and can lead to poor generalization on new examples. The loss term jointly increases the likelihood of generating responses in the retain setRwhile reducing the likelihood of generatingF, as shown below. −L(F;θ) +L(R;θ) •KL Divergence(Maini et al., 2024) Similar to Gradient Difference, in this baseline, we augment the gradient ascent objective with a Kullback-Leibler Divergence term to ensure the model does not deviate too far from the original model. •Normalized Gradient Difference(Bu et al., 2024): In this baseline, we frame the gradient difference objective as a multi-task optimiza- tion problem where the gradient ascent loss term is bounded by normalizing in each train- ing step, along with an automatic lr scheduler to balance the two objectives. •Negative Preference Optimization(Zhang et al., 2024): This baseline uses a modified version of the Direct Preference Optimization objective, adapted to remove the sensitive in- formation from the forget set.