Paper deep dive

SNAP: Unlearning Selective Knowledge in Large Language Models with Negative Instructions

Minseok Choi, Daniel Rim, Dohyun Lee, Jaegul Choo

Year: 2024Venue: arXiv preprintArea: Model EditingType: EmpiricalEmbeddings: 76

Models: Llama-2-7B-Chat, Mistral-7B-Instruct

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%

Last extracted: 3/12/2026, 7:23:17 PM

Summary

The paper introduces Opt-Out, an entity-level unlearning method for Large Language Models (LLMs) that utilizes optimal transport theory and Wasserstein distance to selectively remove knowledge about specific entities while preserving general model capabilities. It also presents the Entity-Level Unlearning Dataset (ELUDe) to evaluate this task.

Entities (5)

ELUDe · dataset · 100%Opt-Out · method · 100%Large Language Models · technology · 95%Wasserstein distance · mathematical-concept · 95%Wikipedia · knowledge-source · 90%

Relation Signals (3)

ELUDe → evaluates → Entity-level unlearning

confidence 95% · Entity-Level Unlearning Dataset (ELUDe) designed to evaluate entity-level unlearning

Opt-Out → utilizes → Wasserstein distance

confidence 95% · Opt-Out, an optimal transport-based unlearning method that utilizes the Wasserstein distance

Opt-Out → improves → Large Language Models

confidence 90% · establishing a new standard for secure and adaptable LLMs

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Instruction-following large language models (LLMs), such as ChatGPT, have become widely popular among everyday users. However, these models inadvertently disclose private, sensitive information to their users, underscoring the need for machine unlearning techniques to remove selective information from the models. While prior work has focused on forgetting small, random subsets of training data at the instance-level, we argue that real-world scenarios often require the removal of an entire user data, which may require a more careful maneuver. In this study, we explore entity-level unlearning, which aims to erase all knowledge related to a target entity while preserving the remaining model capabilities. To address this, we introduce Opt-Out, an optimal transport-based unlearning method that utilizes the Wasserstein distance from the model's initial parameters to achieve more effective and fine-grained unlearning. We also present the first Entity-Level Unlearning Dataset (ELUDe) designed to evaluate entity-level unlearning. Our empirical results demonstrate that Opt-Out surpasses existing methods, establishing a new standard for secure and adaptable LLMs that can accommodate user data removal requests without the need for full retraining.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

76,131 characters extracted from source content.

Expand or collapse full text

Opt-Out: Investigating Entity-Level Unlearning for Large Language Models via Optimal Transport Minseok Choi† Daniel Rim†, ‡ Dohyun Lee† Jaegul Choo† †KAIST AI ‡Hyundai Motor Company minseok.choi,aiclaudev,jchoo@kaist.ac.kr drim@hyundai.com Abstract Instruction-following large language models (LLMs), such as ChatGPT, have become widely popular among everyday users. However, these models inadvertently disclose private, sensitive information to their users, underscoring the need for machine unlearning techniques to remove selective information from the models. While prior work has focused on forgetting small, random subsets of training data at the instance-level, we argue that real-world scenarios often require the removal of an entire user data, which may require a more careful maneuver. In this study, we explore entity-level unlearning, which aims to erase all knowledge related to a target entity while preserving the remaining model capabilities. To address this, we introduce Opt-Out, an optimal transport-based unlearning method that utilizes the Wasserstein distance from the model’s initial parameters to achieve more effective and fine-grained unlearning. We also present the first Entity-Level Unlearning Dataset (ELUDe) designed to evaluate entity-level unlearning. Our empirical results demonstrate that Opt-Out surpasses existing methods, establishing a new standard for secure and adaptable LLMs that can accommodate user data removal requests without the need for full retraining.111Our code and data are available at https://github.com/brightjade/Opt-Out. Opt-Out: Investigating Entity-Level Unlearning for Large Language Models via Optimal Transport Minseok Choi† Daniel Rim†, ‡ Dohyun Lee† Jaegul Choo† †KAIST AI ‡Hyundai Motor Company minseok.choi,aiclaudev,jchoo@kaist.ac.kr drim@hyundai.com 1 Introduction Machine unlearning (MU) is the task of reversing the learning process that aims to remove the influence of data points from a trained machine learning model. The field has emerged to mitigate the risk of private data leakage upon completion of training Cao and Yang (2015), particularly in compliance with legislations, such as the Right to be Forgotten (RTBF) Rosen (2011) in the European Union’s General Data Protection Regulation (GDPR) Hoofnagle et al. (2019) and the United States’ California Consumer Privacy Act (CCPA) Pardau (2018) requiring the removal of personal information when requested. With research showing that extracting training data becomes easier as large language models (LLMs) scale Carlini et al. (2022), ensuring privacy protections for LLMs has become increasingly crucial. Figure 1: Motivation for entity-level unlearning. While instance-level unlearning operates at a coarse granularity, entity-level unlearning aims to remove target entity knowledge while carefully preserving knowledge about neighboring entities and the broader world knowledge. Despite the pressing requirement of the task, eliminating the impact of data samples on billions of model parameters is extremely challenging. The surest approach is exact unlearning, wherein LLMs are completely retrained from scratch using the remaining training set after removing the data points to be forgotten. Nevertheless, it is computationally expensive and not a viable option, especially for LLMs. Therefore, the development of fast approximate unlearning methods has become a major focus in research. Research on MU has primarily been conducted in computer vision tasks Golatkar et al. (2020a, b); Bourtoule et al. (2021); Gandikota et al. (2023); Kurmanji et al. (2023); Fan et al. (2024); however, with the rise of LLMs Brown et al. (2020); Dubey et al. (2024); Abdin et al. (2024), it is gaining prominence in NLP due to privacy problems exhibited by LLMs Zhang et al. (2023). Recently, several MU approaches in NLP have been proposed Jang et al. (2023); Wang et al. (2023a); Chen and Yang (2023); Lee et al. (2024); Zhang et al. (2024). Notably, Jang et al. (2023) first introduced an unlearning technique that reverses the gradient to prevent LLMs from generating specific sensitive token sequences. However, this often resulted in model collapse, where the model starts to produce low-quality, homogeneous responses, especially as the number of instances to forget increases. To remedy this issue, recent methods have attempted to incorporate additional retention data during training Lee et al. (2024) or to relax the unlearning loss to mitigate collapse Zhang et al. (2024). While these strategies have demonstrated promising results, their evaluations have been limited to small, random sets of instances (i.e., at the instance-level) Jang et al. (2023); Maini et al. (2024). Moreover, these methods did not account for a real-world scenario, where a specific person’s data needs to be removed. As illustrated in Figure 1, users may request their personal data be erased under their RTBF. In such cases, it is pivotal to safely and effectively “unlink” the neighboring knowledge while preserving the rest of the information contained in the LLM. In this work, we investigate entity-level unlearning, which focuses on removing all knowledge associated with a specific entity while retaining the rest of the model’s information. To simulate real-world unlearning scenarios, we introduce the first Entity-Level Unlearning Dataset (ELUDe), consisting of 20 real-world target entities built from their respective Wikipedia pages. Additionally, we create a dataset of 10 neighboring entities for each target, serving as retention data that is closely related to the target entity but should remain unforgotten. To further improve the performance of entity-level unlearning, we propose Opt-Out, a novel fine-grained unlearning method grounded in optimal transport theory. Specifically, Opt-Out employs the Wasserstein distance from the LLM’s initial weights to regularize the unlearning process with the optimal transportation cost between the parameters. This enables fine-grained control over the parameters, maximizing those crucial for unlearning while minimizing those essential for retention. We evaluate our framework on ELUDe, alongside several LLM benchmarks, and demonstrate that Opt-Out outperforms existing unlearning methods in both unlearning and retaining performance, highlighting the effectiveness of our approach. Our work focuses on Wikipedia entities due to their extensive coverage and accessibility, rather than the actual privacy data; however, we hope this work provides a testbed for entity-level unlearning, taking a modest step toward advancing the development of practical unlearning methods. Figure 2: Overview of our proposed framework, which consists of three key steps: 1) Forget Set Construction, where target entity knowledge is extracted from Wikipedia and GPT-4o is used to create the forget set, covering as much knowledge as it can; 2) Retain Set Construction, following a similar process to build the retain set using knowledge from neighboring entities; and 3) Optimal Transport-based (OT) Unlearning, which computes the Wasserstein distance between two sets of parametric weights and regularizes the model accordingly. 2 Dataset Construction In this section, we present ELUDe, the first entity-level unlearning dataset focused on the removal of an entire entity. The dataset includes 20 real-world target entities and 144 unique neighboring entities, comprising 15,651 forget samples and 90,954 retain samples. The data collection process is described in detail in the subsequent sections. 2.1 Selecting Target Entities To reverse the influence of data points on a specific entity, an ideal approach would involve access to the exact subset of data used during pretraining. However, obtaining such data is impractical because the pretraining corpus for most LLMs is often concealed. Even if it were available, isolating the data relevant to a particular entity would be extremely challenging. Therefore, we leverage Wikipedia to extract entity knowledge. Wikipedia serves as a reliable source because its widely recognized information is often memorized by various LLMs, making it suitable for knowledge unlearning. Additionally, previous studies have demonstrated that Wikipedia provides high coverage of information about individuals and maintains reasonable self-consistency Min et al. (2023). We specifically choose 20 target entities from the most popular Wikipedia pages222https://en.wikipedia.org/wiki/Wikipedia:Popular_pages, using page views as a proxy for how frequently these entities are discussed online. For effective unlearning evaluation, we need data that LLMs have heavily memorized. If the model lacks prior knowledge of an entity, assessing unlearning becomes difficult, potentially requiring additional finetuning before unlearning, as observed in Maini et al. (2024). Utilizing popular Wikipedia pages aligns well with LLM memorization and is more cost-effective than exploring large pretraining corpora Mallen et al. (2023). 2.2 Selecting Neighboring Entities We can employ any kind of textual data for retention, such as TruthfulQA Yao et al. (2023) or Wikitext Li et al. (2024). However, we posit that finetuning with a general retain set may not effectively disentangle entity knowledge from related information. Inspired by hard negatives in representation learning Gillick et al. (2019), we curate the retained data by mining neighboring pages of a given entity. Specifically, for each target entity, we select 10 neighboring entities based on the following criteria: 1) there is a bidirectional relationship, meaning both entities link to each other and are mentioned at least once on their respective pages, 2) the neighboring pages rank within the top 10 in terms of page views over the past three years, and 3) the neighboring pages are all people. For 20 target entities, this process yields 144 unique neighboring entities (due to overlap). 2.3 Generating QA Pairs After identifying all entities, we transform each corresponding Wikipedia page into a set of QA pairs. While it is technically feasible to input the entire page into an LLM, we find that the QA format works more seamlessly with chat-based models and better simulates real-world interactions. To generate these QA pairs, we process each paragraph through GPT-4o Achiam et al. (2023), prompting it to create as many QA pairs as possible, aiming to cover the full scope of factual content. The specific prompt used for this data generation process is detailed in Appendix F. Since some paragraphs may convey overlapping or identical information, we apply a deduplication step using BERT embeddings from Sentence Transformer Reimers and Gurevych (2019). On average, approximately 647 QA pairs per entity were created. 3 Methodology 3.1 Problem Definition Knowledge Unlearning Given a token sequence =xi=1Tsuperscriptsubscript1x=\x\_i=1^Tx = x i = 1T within the training dataset =i=1Nsuperscriptsubscript1D=\x\_i=1^ND = x i = 1N, the goal of knowledge unlearning is to safely eliminate the influence of a specific subset of data fsubscriptD_fDitalic_f from a trained machine learning model. This process ensures that the model behaves as though the removed data was never included in the training while preserving its performance on the remaining dataset. Conventionally, the data to be forgotten fsubscriptD_fDitalic_f is referred to as the forget set, and the data to be retained is called the retain set. For simplicity, we focus on the standard scenario where fsubscriptD_fDitalic_f and rsubscriptD_rDitalic_r are mutually exclusive (i.e., f∩r=∅subscriptsubscriptD_f _r= _f ∩ Ditalic_r = ∅). Entity-Level Unlearning Entity-level unlearning aims to remove all knowledge related to a specific target entity from a trained machine learning model. However, because the knowledge about an entity within a model is inherently abstract and cannot be explicitly predefined, we adopt the framework proposed by Ma et al. (2024). This reformulates the task as erasing all knowledge associated with a target entity by identifying and deleting a specific subset of entity-associated knowledge, which we term the “proxy forget set”. The proxy forget set encapsulates identifiable aspects of the target entity’s knowledge. Since it is equally impractical to obtain the exact retain set used during pretraining, we define the “proxy retain set” as a specific subset of knowledge encompassing everything except the target entity. Formally, for a target entity t and its proxy forget set ftsuperscriptsubscriptD_f^tDitalic_fitalic_t, the objective of entity-level unlearning is to train a model ϕθsubscriptitalic-ϕ _θϕitalic_θ such that the updated model ϕθ′=S⁢(ϕθ;ft)subscriptitalic-ϕsuperscript′subscriptitalic-ϕsuperscriptsubscript _θ =S( _θ;D_f^t)ϕitalic_θ′ = S ( ϕitalic_θ ; Ditalic_fitalic_t ) operates as if it were trained exclusively on the proxy retain set rtsuperscriptsubscriptD_r^tDitalic_ritalic_t. The unlearning function S ensures that ftsuperscriptsubscriptD_f^tDitalic_fitalic_t is effectively forgotten while retaining the model’s performance on rtsuperscriptsubscriptD_r^tDitalic_ritalic_t. It is important to note that the exact forget and retain sets are unattainable, except in highly controlled scenarios such as fictitious unlearning Maini et al. (2024). Consequently, achieving high knowledge coverage in the proxy sets directly improves alignment with the ideal performance attainable using exact sets. To address this, we leverage Wikipedia as our knowledge source due to its extensive coverage of entity-specific information, thereby narrowing the gap between proxy and exact sets. 3.2 Knowledge Unlearning The primary goal of language modeling is to minimize the negative log-likelihood of token sequences, training the model to accurately predict the next token in a sequence. To remove specific knowledge from language models, a straightforward approach is to apply gradient ascent on the next-token prediction loss over the forget set, which can be understood as equivalent to gradient descent on the negative prediction loss: ℒGA=−ft⁢[−log⁡(ϕθ⁢(y|x))].subscriptℒGAsubscriptsuperscriptsubscriptdelimited-[]subscriptitalic-ϕconditionalL_GA=-E_D_f^t[- ( _θ(% y|x))].LGA = - blackboard_ED start_POSTSUBSCRIPT fitalic_t end_POSTSUBSCRIPT [ - log ( ϕitalic_θ ( y | x ) ) ] . (1) By inverting the language modeling objective, many existing unlearning methods have successfully removed parametric knowledge of the forget set from language models. However, numerous studies have highlighted the catastrophic effects of gradient ascent Yao et al. (2023); Lee et al. (2024). To address these issues, Zhang et al. (2024) introduced negative preference optimization (NPO), a technique that simplifies to gradient ascent in the high-temperature limit but is inherently more stable and lower-bounded, significantly slowing the model collapse compared to gradient ascent. NPO draws on preference optimization Rafailov et al. (2023) and aligns the language model with negative examples exclusively: ℒNPO=−ft⁢[log⁡σ⁢(−η⁢log⁡ϕθ⁢(y|x)ϕref⁢(y|x))],subscriptℒNPOsubscriptsuperscriptsubscriptdelimited-[]subscriptitalic-ϕconditionalsubscriptitalic-ϕrefconditionalL_NPO=-E_D_f^t [ σ% (-η _θ(y|x) _ref(y|x) ) ],LNPO = - blackboard_ED start_POSTSUBSCRIPT fitalic_t end_POSTSUBSCRIPT [ log σ ( - η log divide start_ARG ϕitalic_θ ( y | x ) end_ARG start_ARG ϕref ( y | x ) end_ARG ) ] , (2) where σ represents the sigmoid function, η>00η>0η > 0 is the inverse temperature, and ϕrefsubscriptitalic-ϕref _refϕref is a reference model. In entity-level unlearning, we observe that the NPO loss also produces much more stable and reliable results in practice. However, finetuning solely on the forget set eventually leads to model degradation and collapse. As with prior unlearning methods, we also train the model on the retain set to explicitly preserve the remaining knowledge. This is achieved through standard language modeling on the retain set, which serves as the positive counterpart to Equation 1: ℒRT=−rt⁢[log⁡(ϕθ⁢(y|x))]subscriptℒRTsubscriptsuperscriptsubscriptdelimited-[]subscriptitalic-ϕconditionalL_RT=-E_D_r^t[ ( _θ(y% |x))]LRT = - blackboard_ED start_POSTSUBSCRIPT ritalic_t end_POSTSUBSCRIPT [ log ( ϕitalic_θ ( y | x ) ) ]. 3.3 Optimal Transport-Based Unlearning To further enhance the performance of entity-level unlearning, we propose Opt-Out, a fine-grained unlearning approach grounded in optimal transport theory. Building on this theory, we develop the Wasserstein regularization, which calculates the Wasserstein distance between two sets of parametric weights and regularizes the model based on this distance. The Wasserstein distance, also known as Earth Mover’s Distance, addresses the optimal transport problem by measuring the minimum effort required to move one distribution of mass to another. We hypothesize that computing this distance helps us estimate the optimal transportation cost between parameters, facilitating more effective unlearning. By applying this framework, we allow more significant shifts in parameters that are crucial for unlearning, while reducing changes in parameters important for retention. In mathematical terms, given a source distribution μ and a target distribution ν, sampled from probability space ,∈ΩX,Y∈ _X , blackboard_Y ∈ Ω respectively, the optimal transport attempts to compute the minimal transportation cost between the two distributions. Formally, Kantorovich (2006) formulates the problem with a probabilistic coupling π∈⁢(×)π (X×Y)π ∈ P ( blackboard_X × blackboard_Y ): ∗=arg⁢min∈Π⁢(μ,ν)⁢∫×c⁢(,)⁢(,)⁢,superscriptsubscriptargminΠsubscriptdifferential-ddifferential-d π^*= *arg\,min_ π∈ (μ,ν) _X% ×Yc( x, y) π( x, y)d xd y,italic_π∗ = start_OPERATOR arg min end_OPERATORitalic_π ∈ Π ( μ , ν ) ∫blackboard_X × blackboard_Y c ( italic_x , italic_y ) italic_π ( italic_x , italic_y ) d italic_x d italic_y , (3) where πitalic_π is the joint probability measure given margins μ and ν, Π⁢(μ,ν)=∫⁢(x,y)⁢=μ,∫⁢(x,y)⁢=ν,≥0Πformulae-sequencesubscriptdifferential-dformulae-sequencesubscriptdifferential-d0 (μ,ν)=\ _Y π(x,y)d y=μ, _X % π(x,y)d x=ν, π≥ 0\Π ( μ , ν ) = ∫blackboard_Y italic_π ( x , y ) d italic_y = μ , ∫blackboard_X italic_π ( x , y ) d italic_x = ν , italic_π ≥ 0 , and c⁢(x,y)c(x,y)c ( x , y ) is the cost function that quantifies the movement of x to y. In this work, we constrain the problem to discrete distributions, which is often expressed as ∗=arg⁢minγ∈ℝ+m×n⁢∑i=1m∑j=1nγi⁢j⁢Ci⁢js.t. ⁢γ⁢=α,γ⊤⁢=β,γ≥0,formulae-sequencesuperscriptsubscriptargminsuperscriptsubscriptℝsuperscriptsubscript1superscriptsubscript1subscriptsubscripts.t. 1formulae-sequencesuperscripttop10 split γ^*&= *arg\,min_γ∈% R_+^m× n _i=1^m _j=1^n _ijC_ij\\ &s.t. γ 1=α,γ 1=β,γ≥ 0, splitstart_ROW start_CELL italic_γ∗ end_CELL start_CELL = start_OPERATOR arg min end_OPERATORγ ∈ blackboard_R start_POSTSUBSCRIPT +m × n end_POSTSUBSCRIPT ∑i = 1m ∑j = 1n γitalic_i j Citalic_i j end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL s.t. γ 1 = α , γ⊤ 1 = β , γ ≥ 0 , end_CELL end_ROW (4) where ∗superscript γ^*italic_γ∗ is the optimal transport plan or transport matrix, C∈ℝ+m×nsuperscriptsubscriptℝC _+^m× nC ∈ blackboard_R+m × n is the cost matrix defining the cost to move mass from bin αisubscript _iαitalic_i to bin βjsubscript _jβitalic_j, and α and β are histograms on the simplex that represent the weights of each sample in the source and target distributions. Building on the optimal transport equation, given the initial weights of the language model as θ0subscript0 _0θ0, the Wasserstein distance between θ0subscript0 _0θ0 and the training parameters θ with finite p-moments is then computed as Wp⁢(θ,θ0)=(minγ∈ℝ+m×n⁢∑i,jγi⁢j⁢‖θi−θ0,j‖p)1ps.t. ⁢γ⁢=α,γ⊤⁢=β,γ≥0.formulae-sequencesubscriptsubscript0superscriptsubscriptsuperscriptsubscriptℝsubscriptsubscriptsubscriptnormsubscriptsubscript01s.t. 1formulae-sequencesuperscripttop10 splitW_p(θ, _0)&=( _γ % _+^m× n _i,j _ij|| _i- _0,j||_p) % 1p\\ &s.t. γ 1=α,γ 1=β,γ≥ 0. splitstart_ROW start_CELL Witalic_p ( θ , θ0 ) end_CELL start_CELL = ( minitalic_γ ∈ blackboard_R start_POSTSUBSCRIPT +m × n end_POSTSUBSCRIPT ∑i , j γitalic_i j | | θitalic_i - θ0 , j | |p )divide start_ARG 1 end_ARG start_ARG p end_ARG end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL s.t. γ 1 = α , γ⊤ 1 = β , γ ≥ 0 . end_CELL end_ROW (5) However, it is intractable to compute the exact ∗superscript γ^*italic_γ∗, because the time complexity of the exact solver is O⁢(n3⁢log⁡n)superscript3O(n^3 n)O ( n3 log n ) and the memory complexity is always O⁢(n2)superscript2O(n^2)O ( n2 ) due to the cost matrix. Especially for LLMs, the number of parameters exceeds billions, if not trillions. For efficiency in both time and memory, we approximate the Wasserstein distance by computing the Sliced Wasserstein Distance (SWD) Bonneel et al. (2015). Instead of computing the entire cost matrix, SWD reduces the dimensionality of the problem by projecting the distributions onto random slices and then computing the Wasserstein distance in a lower-dimensional space. Concretely, the Monte Carlo approximation of the p-sliced Wasserstein distance is given by S⁢Wp⁢(θ,θ0)=u∼⁢(d−1)(Wp⁢(u#⁢θ,u#⁢θ0))1p,subscriptsubscript0subscriptsimilar-tosuperscript1superscriptsubscriptsubscript#subscript#subscript01SW_p(θ, _0)= E_u (S^% d-1)(W_p(u_\#θ,u_\# _0)) 1p,S Witalic_p ( θ , θ0 ) = blackboard_Eu ∼ U ( blackboard_Sd - 1 ) ( Witalic_p ( u# θ , u# θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ) )divide start_ARG 1 end_ARG start_ARG p end_ARG , (6) where ⁢(d−1)superscript1U(S^d-1)U ( blackboard_Sd - 1 ) denotes the uniform distribution on the unit sphere in ℝdsuperscriptℝR^dblackboard_Rd, and u#⁢θsubscript#u_\#θu# θ and u#⁢θ0subscript#subscript0u_\# _0u# θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT stand for the pushforwards of the projections of θ and θ0subscript0 _0θ0 along the direction of u∈d−1superscript1u ^d-1u ∈ blackboard_Sd - 1, respectively. Putting everything together, the overall training objective for fine-grained entity-level unlearning is minimizing the following loss: ℒ=ℒNPO+ℒRT+λ⋅S⁢Wp⁢(θ,θ0),ℒsubscriptℒNPOsubscriptℒRT⋅subscriptsubscript0L=L_NPO+L_RT+λ· SW_% p(θ, _0),L = LNPO + LRT + λ ⋅ S Witalic_p ( θ , θ0 ) , (7) where λ is a hyperparameter for scaling the regularization term. FQ RQ ARC-C CSQA Hella. Lamba. MMLU OBQA PIQA Wino. Avg. Llama-3.1-8B-Instruct Original 45.5 51.2 51.8 77.1 59.2 73.2 68.1 33.8 80.2 74.1 64.7 Guardrail 64.8 51.3 49.8 75.5 58.9 72.3 67.2 33.2 80.1 74.2 63.9 GA* 70.9 0.0 23.6 21.9 32.2 11.6 33.9 28.0 58.3 60.9 33.8 DPO* 76.3 0.0 22.9 30.5 52.8 36.4 37.9 28.3 58.4 69.5 42.1 NPO* 89.7 0.0 24.7 23.0 37.6 20.8 36.3 30.6 60.1 67.2 37.5 IDK* 84.3 3.5 38.4 67.6 53.3 48.0 61.8 30.0 77.1 71.3 56.0 GA+RT 77.1 45.7 47.4 71.0 57.7 71.1 60.7 32.9 79.2 72.2 61.5 DPO+RT 84.9 44.9 49.2 68.8 58.7 68.6 57.9 33.6 79.8 72.7 61.1 NPO+RT 82.6 46.6 50.1 73.5 58.7 71.7 62.5 33.3 79.7 73.0 62.8 IDK+RT 71.9 46.1 49.4 73.8 58.7 69.7 63.2 34.0 79.8 73.4 62.8 Opt-Out (ours) 87.8 46.6 49.8 75.3 59.0 71.8 63.2 34.2 79.7 73.1 63.3 Phi-3.5-Mini-Instruct Original 44.9 34.9 59.5 75.3 58.8 65.1 68.7 37.6 80.0 74.6 65.0 Guardrail 46.6 26.4 52.2 72.9 59.2 64.1 67.7 38.3 78.1 74.5 63.4 GA* 63.6 0.0 24.0 19.8 52.1 47.0 23.6 32.6 53.8 66.0 39.8 DPO* 78.3 0.0 38.9 36.1 56.7 54.3 54.0 36.4 63.9 72.7 51.6 NPO* 80.7 0.0 28.6 19.6 56.8 57.0 26.6 35.8 59.1 69.4 44.1 IDK* 80.4 4.3 53.7 70.9 54.8 47.8 65.7 36.6 79.4 76.7 60.7 GA+RT 67.7 47.3 56.9 69.4 56.7 56.8 67.2 35.8 79.9 73.1 62.0 DPO+RT 67.4 48.6 57.9 72.8 57.6 55.6 68.0 37.3 80.7 75.0 63.1 NPO+RT 67.5 49.2 58.1 72.8 57.6 57.7 67.9 37.1 80.0 74.4 63.2 IDK+RT 68.6 48.4 57.1 74.3 57.5 55.0 67.7 37.7 80.2 76.0 63.2 Opt-Out (ours) 76.5 49.4 58.9 72.9 57.6 58.2 68.0 36.9 80.4 74.1 63.4 Table 1: Performance (%) of various methods after unlearning on Llama-3.1-8B-Instruct and Phi-3.5-Mini-Instruct. FQ (Forget Quality) reflects the harmonic mean of ground-truth token probabilities, ROUGE-L recall scores, and truth ratio over the forget set, while RQ (Retain Quality) is computed across the retain and world sets. Methods are also assessed on eight LLM benchmarks to evaluate the retention of overall model capabilities. (*) indicates collapsed models. The best results are in bold, while the second best are underlined. 4 Experiments 4.1 Datasets We utilize the forget and retain sets from ELUDe to evaluate entity-level unlearning. The retain set is divided into training, validation, and test splits in an 8:1:1 ratio. Since the training portion of the retained data is significantly larger than the forget set, we apply random sampling during training. Moreover, we incorporate the Alpaca-GPT4 instruction dataset Peng et al. (2023) as an auxiliary retain set (i.e., world set) to align the model with general instructional tasks. Specifically, we use 50k instructional examples for training, 1k for validation, and 1k for testing. To assess model utility, we also validate our framework on eight language understanding benchmarks including ARC-Challenge Clark et al. (2018), CommonsenseQA Talmor et al. (2019), HellaSwag Zellers et al. (2019), Lambada Paperno et al. (2016), MMLU Hendrycks et al. (2021), OpenbookQA Mihaylov et al. (2018), PIQA Bisk et al. (2020), and Winogrande Sakaguchi et al. (2021). 4.2 Evaluation Metrics Following closely with Maini et al. (2024), we measure the unlearning performance using a stack of the following metrics: Probability We compute the conditional probability P⁢(a|q)conditionalP(a|q)P ( a | q ) for the forget and retain sets, normalizing for answer length by raising it to the power 1/|a|11/|a|1 / | a |. For the world set, each question q is treated as multiple-choice with choices a1,…,ansubscript1…subscript\a_1,...,a_n\ a1 , … , aitalic_n , where a1subscript1a_1a1 is the correct answer. The probability is then P⁢(a1|q)/∑i=1nP⁢(ai|q)conditionalsubscript1superscriptsubscript1conditionalsubscriptP(a_1|q)/ _i=1^nP(a_i|q)P ( a1 | q ) / ∑i = 1n P ( aitalic_i | q ). ROUGE We use ROUGE-L recall Lin (2004) to compare model answers (greedy sampling) with ground truth, serving as a proxy for QA accuracy by accounting for variations in phrasing. Truth Ratio We compute a ratio comparing the likelihood of the correct answer to incorrect ones. Since finetuning may inflate the probability of the exact ground truth phrasing, we use a paraphrased version of the correct answer and average probabilities over multiple similarly formatted wrong answers. This ratio helps assess whether the unlearning algorithm removed the target information, even if the model no longer provides exact matches but still favors correct responses. Let a~~ aover~ start_ARG a end_ARG denote the paraphrased answer and pertsubscriptpertA_pertApert denote a set of five perturbations generated by GPT-4o. The truth ratio RtruthsubscripttruthR_truthRtruth is given by: Rtruth=1|pert|⁢∑a^∈pertP⁢(a^|q)q/|a^|P⁢(a~|q)q/|a~|subscripttruth1subscriptpertsubscript^subscriptpertsuperscriptconditional^^superscriptconditional~~R_truth= 1|A_pert| _ a∈% A_pertP( a|q)^q/| a|P( a|q)^q/|% a|Rtruth = divide start_ARG divide start_ARG 1 end_ARG start_ARG | Apert | end_ARG ∑over start_ARG a end_ARG ∈ A start_POSTSUBSCRIPT pert end_POSTSUBSCRIPT P ( over start_ARG a end_ARG | q )q / | over start_ARG a end_ARG | end_ARG start_ARG P ( over~ start_ARG a end_ARG | q )q / | over~ start_ARG a end_ARG | end_ARG (8) For Forget Quality (FQ), we compute the harmonic mean of the three values on the forget set333Unlike in Maini et al. (2024), we do not use the p-value from the Kolmogorov-Smirnov test as FQ because it is impossible to compare against a perfectly unlearned model in our setup, and even more so in real-world applications. For comparison with exact unlearning, refer to Appendix E for our results on the TOFU dataset., while for Retain Quality (RQ), we take the harmonic mean of the six values across both the retain and world sets to prevent low scores from getting averaged out. Some values are inverted so that higher values indicate better performance (e.g., max⁢(0,1−Rtruth)max01subscripttruthmax(0,1-R_truth)max ( 0 , 1 - Rtruth ) is used in RQ). Figure 3: Forget Quality performance (%) of different +RT methods following unlearning on Llama-3.1-8B-Instruct, evaluated against nine types of adversarial prompt attacks. Each attack is described in detail in Appendix G. 4.3 Baselines We compare our framework with the following unlearning methods: • Guardrail Thaker et al. (2024): A simple prompting baseline that instructs the LLM to refuse to answer about the specified entity • GA Jang et al. (2023): Applies gradient ascent on the forget set • DPO Rafailov et al. (2023): Employs direct preference optimization where “I don’t know” responses are preferred on the forget set • NPO Zhang et al. (2024): Utilizes negative preference optimization on the forget set • IDK Maini et al. (2024): Finetunes the model to provide “I don’t know” responses for the forget set • +RT: Additionally finetunes the model on the retain set for explicit model retention 4.4 Unlearning Results We present a comparison of unlearning results across various methods in Table 1. Our experiments follow the single-target unlearning setting, where one target is forgotten at a time, with the results averaged over five unlearning targets. First, we observe that Guardrail, which utilizes the system prompt “If the question asks about entity, say you do not know the answer; otherwise, answer as best as you can,” effectively retains information but struggles to adequately forget the target entity. For the Phi-3.5 model, Guardrail negatively impacts RQ performance, indicating that in-context unlearning is not suitable for smaller models. Unlearning baselines such as GA, DPO, NPO, and IDK show improvements in FQ; however, these methods tend to collapse, with RQ dropping to near zero and overall benchmark performance significantly degrading. With additional finetuning on the retain set (+RT), retention performance improves across the board, while FQ remains strong. Notably, Opt-Out outperforms all methods across both Llama-3.1 and Phi-3.5 models and maintains competitive RQ and overall LLM benchmark performance, demonstrating the effectiveness of our proposed approach. 4.5 Performance Against LLM Attacks Membership Inference Attacks We assess performance against Membership Inference Attacks (MIAs) to ensure that, after unlearning, an attacker cannot distinguish between unlearned examples and those never seen by the model, thus protecting user privacy. Following Chen and Yang (2023), we train a binary classifier (the “attacker”) on the unlearned model’s losses for forget and test samples. Since we perform entity-level unlearning on the entire forget set, we use a paraphrased set for the test samples. Ideally, 50% accuracy indicates the attacker cannot differentiate between the two, validating the unlearning method. As shown in Table 2, most unlearned models, including Opt-Out, successfully defend against MIAs. Llama-3.1 Phi-3.5 Method mean std mean std Oracle 50.0 - 50.0 - GA 53.8 2.9 54.7 1.4 DPO 56.2 3.8 53.8 2.2 NPO 53.8 1.8 52.8 1.6 IDK 59.3 2.3 58.7 1.8 GA+RT 50.9 3.1 49.7 1.0 DPO+RT 50.3 3.1 49.7 3.1 NPO+RT 49.6 2.5 50.9 2.5 IDK+RT 69.1 1.5 67.3 1.9 Opt-Out (ours) 48.6 1.0 49.1 1.1 Table 2: MIA accuracy (%) of a trained binary classifier (“the attacker”) predicting whether an input data belongs to the training set. 50% indicates the best performance. Adversarial Prompt Attacks Given the use of instruction-tuned models, safeguarding against malicious prompt attacks is vital. To rigorously evaluate the efficacy of unlearning in mitigating adversarial attacks, we follow Jin et al. (2024) and assess unlearned models against nine different types of adversarial threats. Detailed descriptions of the attack examples are provided in Appendix G. As illustrated in Figure 3, our proposed approach, Opt-Out, consistently achieves high-quality forgetting across various adversarial attacks, demonstrating strong robustness against malicious prompts. 4.6 Effect of Wasserstein Regularization We verify the effectiveness of the proposed Wasserstein regularization by comparing it to other commonly used distance metrics. As shown in Table 3, the Manhattan distance preserves the most information, but this is largely attributed to the fact that the model underwent minimal unlearning due to excessively strong regularization. In contrast, the Euclidean and Cosine distances show reasonable unlearning performance, though they slightly underperform compared to using no regularization at all (as evidenced by NPO+RT in Table 1). In comparison, our proposed Wasserstein distance delivers the best overall results, highlighting the efficacy of optimal transport-based unlearning. Distance Metric FQ RQ Util. Wasserstein (ours) 87.8 46.6 63.3 Manhattan 47.0 50.9 64.6 Euclidean 81.5 46.2 63.0 Chebyshev 86.3 45.4 62.2 Cosine 81.6 45.8 62.8 Table 3: Comparison of distance metrics in regularization with Llama-3.1-8B-Instruct. Util. is the average of results across the eight LLM benchmarks. 4.7 Effect of Neighboring Entity Data To validate the effectiveness of our neighboring entity data augmentation, we measure the unlearning performance of a model trained without the neighboring entity set, using only the world set (i.e., Alpaca-GPT4). As illustrated in Figure 4, the model trained solely on the world set shows comparable performance in terms of Forget Quality and overall model utility but exhibits significantly worse performance on Retain Quality. We attribute this to the model’s difficulty in distinguishing between forget and retain examples when trained exclusively on world data. In contrast, the model supplemented with our neighboring entity data consistently outperforms the other settings across all metrics, highlighting the importance of incorporating closely related data, which likely acts as “hard positives,” aiding the model in better differentiating forget and retain examples. Figure 4: Performance comparison between using only the world set and supplementing it with our neighboring entity set as retain data during training. Scores are averaged across GA, DPO, NPO, IDK, and Opt-Out methods using Llama-3.1-8B-Instruct. 5 Related Work 5.1 Machine Unlearning With the emergence of machine unlearning to mitigate privacy concerns Cao and Yang (2015); Golatkar et al. (2020a); Kurmanji et al. (2023), the focus of unlearning techniques in computer vision has predominantly centered on image classification models where they aim to forget a whole class, thereby attaining random performance for particular image classes. Recently, there have been attempts to perform unlearning in image generation Fan et al. (2024) or erase specific concepts from diffusion model weights, utilizing negative guidance as a teacher to drive the unlearning process Gandikota et al. (2023). Concept erasure aims to identify and remove specific concepts that may be encoded Ravfogel et al. (2022a, b); Belrose et al. (2023), applying various transformations to the neural representations. These methods generally approach the problem from a theoretical setting and look to identify and erase a high-level concept that may cause biases, such as gender or racial biases. 5.2 Knowledge Unlearning Likewise, the primary emphasis of unlearning in NLP has been directed towards tasks such as text classification and generation Wang et al. (2023a); Chen and Yang (2023); Yao et al. (2023). Introducing a new paradigm, Jang et al. (2023) proposed unlearning specific token sequences by negating the gradient descent. Nevertheless, this often led to model collapse, especially as the number of samples to forget increased. To address this issue, Lee et al. (2024) presented a more robust method to mitigate performance degradation by incorporating retention mechanisms. Others shared similar concerns about catastrophic failure in machine unlearning and suggested solutions based on preference optimization Zhang et al. (2024). These methods, however, primarily target unlearning specific instances in language models. In this work, we focus on removing targeted entity-level information that may have been learned during pretraining, leveraging an optimal transport-based technique for more effective and fine-grained unlearning. A concurrent work Ma et al. (2024) also explores entity-level unlearning but is limited to the task of fictitious unlearning Maini et al. (2024). 5.3 Unlearning Datasets With the latest development of machine unlearning for LLMs, the need for dedicated unlearning datasets and benchmarks has become increasingly important. Li et al. (2024) introduced the Weapons of Mass Destruction Proxy (WMDP) benchmark, which includes 3,668 multiple-choice questions designed to measure hazardous knowledge in biosecurity, cybersecurity, and chemical security. Maini et al. (2024) presented the Task of Fictitious Unlearning (TOFU), featuring 20 QA pairs for each of 200 fictitious authors. Jin et al. (2024) released the Real-World Knowledge Unlearning (RWKU) benchmark, focusing on 200 real-world celebrities and comprising 2,879 QA pairs. In parallel, our work introduces a new dataset ELUDe, which includes 20 real-world popular entities. Unlike previous efforts, we provide a substantial volume of data for each entity, totaling 15,651 and 90,954 QA pairs for forget and retain samples, respectively. This enables the complete removal of all knowledge associated with a specific entity, providing a valuable resource for researchers and practitioners tackling real-world user unlearning requests. 6 Conclusion In this work, we explore entity-level unlearning, a pivotal and timely technique for removing a specific person’s data from LLMs. To simulate real-world user unlearning requests, we introduce ELUDe, a QA dataset designed to train LLMs to selectively forget a specific entity. Furthermore, we propose Opt-Out, an optimal transport-based unlearning method that applies Wasserstein regularization to the model parameters. Our approach outperforms existing unlearning techniques, likely due to its more fine-grained control in knowledge unlearning. These findings are particularly relevant for LLMs deployed in real-world scenarios, enabling them to handle user requests to remove personal data without the need for full retraining. Limitations While our framework shows promising performance in unlearning entity-level knowledge, several areas warrant further refinement. First, our work focuses on unlearning Wikipedia entities, which may differ slightly from erasing data related to actual users. Nevertheless, creating meaningful forget and retain sets for an arbitrary person (e.g., Alice) is challenging, as it is difficult to capture how much the LLM knows about her. Therefore, we have leveraged Wikipedia, where the pages themselves serve as a useful proxy for comprehensive data coverage of a particular entity, enabling effective evaluation of full entity-level erasure. Future work could extend our approach to real-world privacy data, incorporating advanced anonymization techniques to better align with practical use cases. Second, our method remains susceptible to generating gibberish post-unlearning. Although it effectively removes parametric knowledge, ensuring the LLM functions correctly for a seamless end-user experience in real-world deployment remains an issue. Combining with the IDK method or remapping outputs to automated responses after unlearning could be considered a simple fix. Lastly, due to computational constraints, we were unable to test models at the scale of 70B parameters or larger. Exploring unlearning techniques with much larger models would better align with the behavior of proprietary models. Acknowledgements We are deeply grateful to Hojoon Lee and Dongyoon Hwang for generously sharing their expertise during our discussions. We also appreciate the thoughtful feedback provided by the anonymous reviewers. This work was supported by the Institute for Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (RS-2019-I190075, Artificial Intelligence Graduate School Program (KAIST), RS-2025-02304967, AI Star Fellowship (KAIST)), and Samsung Electronics Co., Ltd. References Abdin et al. (2024) Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219. Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Belrose et al. (2023) Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. 2023. Leace: Perfect linear concept erasure in closed form. arXiv preprint arXiv:2306.03819. Bisk et al. (2020) Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439. Bonneel et al. (2015) Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister. 2015. Sliced and radon wasserstein barycenters of measures. Journal of Mathematical Imaging and Vision, 51:22–45. Bourtoule et al. (2021) Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141–159. IEEE. Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc. Cao and Yang (2015) Yinzhi Cao and Junfeng Yang. 2015. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pages 463–480. IEEE. Carlini et al. (2022) Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations. Chen and Yang (2023) Jiaao Chen and Diyi Yang. 2023. Unlearn what you want to forget: Efficient unlearning for llms. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12041–12052. Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Fan et al. (2024) Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, and Sijia Liu. 2024. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. In The Twelfth International Conference on Learning Representations. Gandikota et al. (2023) Rohit Gandikota, Joanna Materzyńska, Jaden Fiotto-Kaufman, and David Bau. 2023. Erasing concepts from diffusion models. In Proceedings of the 2023 IEEE International Conference on Computer Vision. Gillick et al. (2019) Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, and Diego Garcia-Olano. 2019. Learning dense representations for entity retrieval. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 528–537, Hong Kong, China. Association for Computational Linguistics. Golatkar et al. (2020a) Aditya Golatkar, Alessandro Achille, and Stefano Soatto. 2020a. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9304–9312. Golatkar et al. (2020b) Aditya Golatkar, Alessandro Achille, and Stefano Soatto. 2020b. Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIX 16, pages 383–398. Springer. Gugger et al. (2022) Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan. 2022. Accelerate: Training and inference at scale made simple, efficient and adaptable. https://github.com/huggingface/accelerate. Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations. Hoofnagle et al. (2019) Chris Jay Hoofnagle, Bart Van Der Sloot, and Frederik Zuiderveen Borgesius. 2019. The european union general data protection regulation: what it is and what it means. Information & Communications Technology Law, 28(1):65–98. Jang et al. (2023) Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. 2023. Knowledge unlearning for mitigating privacy risks in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14389–14408, Toronto, Canada. Association for Computational Linguistics. Jin et al. (2024) Zhuoran Jin, Pengfei Cao, Chenhao Wang, Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, and Jun Zhao. 2024. RWKU: Benchmarking real-world knowledge unlearning for large language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Kantorovich (2006) Leonid V Kantorovich. 2006. On the translocation of masses. Journal of mathematical sciences, 133(4):1381–1382. Kurmanji et al. (2023) Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. 2023. Towards unbounded machine unlearning. In Thirty-seventh Conference on Neural Information Processing Systems. Lee et al. (2024) Dohyun Lee, Daniel Rim, Minseok Choi, and Jaegul Choo. 2024. Protecting privacy through approximating optimal parameters for sequence unlearning in language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 15820–15839, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics. Li et al. (2024) Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew B. Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Adam Khoja, Zhenqi Zhao, Ariel Herbert-Voss, Cort B. Breuer, Samuel Marks, Oam Patel, Andy Zou, Mantas Mazeika, Zifan Wang, Palash Oswal, Weiran Liu, Adam A. Hunt, Justin Tienken-Harder, Kevin Y. Shih, Kemper Talley, John Guan, Russell Kaplan, Ian Steneker, David Campbell, Brad Jokubaitis, Alex Levinson, Jean Wang, William Qian, Kallol Krishna Karmakar, Steven Basart, Stephen Fitz, Mindy Levine, Ponnurangam Kumaraguru, Uday Tupakula, Vijay Varadharajan, Yan Shoshitaishvili, Jimmy Ba, Kevin M. Esvelt, Alexandr Wang, and Dan Hendrycks. 2024. The wmdp benchmark: Measuring and reducing malicious use with unlearning. Preprint, arXiv:2403.03218. Lin (2004) Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. Ma et al. (2024) Weitao Ma, Xiaocheng Feng, Weihong Zhong, Lei Huang, Yangfan Ye, and Bing Qin. 2024. Rethinking entity-level unlearning for large language models. arXiv preprint arXiv:2406.15796. Maini et al. (2024) Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary Chase Lipton, and J Zico Kolter. 2024. TOFU: A task of fictitious unlearning for LLMs. In First Conference on Language Modeling. Mallen et al. (2023) Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9802–9822, Toronto, Canada. Association for Computational Linguistics. Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium. Association for Computational Linguistics. Min et al. (2023) Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076–12100, Singapore. Association for Computational Linguistics. Paperno et al. (2016) Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1525–1534, Berlin, Germany. Association for Computational Linguistics. Pardau (2018) Stuart L Pardau. 2018. The california consumer privacy act: Towards a european-style privacy regime in the united states. J. Tech. L. & Pol’y, 23:68. Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. Peng et al. (2023) Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277. Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems. Ravfogel et al. (2022a) Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D Cotterell. 2022a. Linear adversarial concept erasure. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 18400–18421. PMLR. Ravfogel et al. (2022b) Shauli Ravfogel, Francisco Vargas, Yoav Goldberg, and Ryan Cotterell. 2022b. Adversarial concept erasure in kernel space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6034–6055, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. Rosen (2011) Jeffrey Rosen. 2011. The right to be forgotten. Stan. L. Rev. Online, 64:88. Sakaguchi et al. (2021) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics. Thaker et al. (2024) Pratiksha Thaker, Yash Maurya, and Virginia Smith. 2024. I’m not familiar with the name harry potter: Prompting baselines for unlearning in LLMs. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models. Wang et al. (2023a) Lingzhi Wang, Tong Chen, Wei Yuan, Xingshan Zeng, Kam-Fai Wong, and Hongzhi Yin. 2023a. KGA: A general machine unlearning framework based on knowledge gap alignment. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13264–13276, Toronto, Canada. Association for Computational Linguistics. Wang et al. (2023b) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023b. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484–13508, Toronto, Canada. Association for Computational Linguistics. Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. Yao et al. (2023) Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2023. Large language model unlearning. In Socially Responsible Language Modelling Research. Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics. Zhang et al. (2023) Dawen Zhang, Pamela Finckenberg-Broman, Thong Hoang, Shidong Pan, Zhenchang Xing, Mark Staples, and Xiwei Xu. 2023. Right to be forgotten in the era of large language models: Implications, challenges, and solutions. arXiv preprint arXiv:2307.03941. Zhang et al. (2024) Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024. Negative preference optimization: From catastrophic collapse to effective unlearning. In First Conference on Language Modeling. Appendix A Implementation Details Our framework is built on PyTorch Paszke et al. (2019), Hugging Face Transformers Wolf et al. (2020), and Accelerate Gugger et al. (2022). We use Llama-3.1-8B-Instruct Dubey et al. (2024) and Phi-3.5-Mini-Instruct Abdin et al. (2024) and optimize their weights with AdamW Loshchilov and Hutter (2019), tuning hyperparameters to maximize Forget and Retain Quality. We set the batch size to 32, the learning rate to 1e-5, the weight decay to 0.01, the inverse temperature η to 0.1, and the regularization strength λ to 0.1. We train for 3 epochs and use early stopping if the model performance decreases from the last epoch. All experiments are conducted with four NVIDIA H100 GPUs. Appendix B Human Evaluation To verify the reliability of our machine-generated dataset, we perform human evaluation based on the following criteria (0-1 scale): 1. Relevance: Does the question discuss the entity (1 if it does, 0 if not)? 2. Diversity: Is there a similar question in the dataset (0 if there is, 1 if not)? 3. Factuality: Does the answer match with the passage (1 if correct, 0 if it’s hallucinated)? Following Wang et al. (2023b), we asked authors of this paper to judge training instances for a particular entity on both forget and retain sets. Due to the substantial size of the retain set, we match its number with the corresponding forget set. The evaluators coordinated the standards before starting annotation and then each of them rated all the instances independently. Table 4 shows the average scores for each criterion. We notice that GPT-4o is highly capable of generating relevant, diverse, and factually accurate QA pairs based on the given passage. However, since we feed GPT-4o one passage at a time, some facts tend to overlap with those from previous passages. Additionally, our prompting approach, which encourages GPT-4o to include as many factual details as possible, often results in QA pairs that feature information either not directly related to the main entity (e.g., “What is Cristiano Ronaldo’s mother’s occupation?”) or trivial (e.g., “Which national team does Ronaldo play for?”). The generated QA pairs were predominantly accurate, though any minor factual discrepancies likely stemmed from Wikipedia’s frequent updates. The evaluation results for the retain set were relatively high due to the involvement of multiple entities, which made fact overlap less likely. Moreover, examples were considered acceptable as long as they did not discuss the primary target entity. Forget Set Retain Set Relevance 89.0 96.0 Diversity 90.0 99.5 Factuality 99.0 98.5 Table 4: Human evaluation results (%). Appendix C Computational Cost Analysis To portray the feasibility of Opt-Out, we conduct a computational cost analysis and present the results in Table 5. Opt-Out introduces a small additional overhead during training while maintaining an identical inference cost to existing methods, ensuring its practicality for large-scale applications. Specifically, the additional overhead stems from computing the Sliced Wasserstein Distance (SWD) between two sets of parametric weights. Regarding memory usage during training, Opt-Out is comparable to DPO and NPO, which also require holding an extra reference (frozen) model. The additional memory cost in Opt-Out arises from a lightweight random projection layer used in SWD computation. Given that L is the input sequence length, the training time and memory complexities of Transformer is O⁢(L2)superscript2O(L^2)O ( L2 ). +RT incurs additional overhead due to an extra forward pass, while DPO/NPO require a third forward pass through a reference model (DPO demands extra two passes for the preference loss). Opt-Out adds a small training overhead of O⁢(L⁢log⁡L)O(L L)O ( L log L ) for SWD computation (due to sorting) and O⁢(L⁢d)O(Ld)O ( L d ) for projected representations, where d is the dimension of the random projection layer and is kept small. We believe these additional costs are manageable even at scale, as the L⁢log⁡L L log L term grows significantly slower compared to the L2superscript2L^2L2 complexity of Transformer training itself. Training Inference Method Time Memory Time Memory GA+RT ⁢(2⁢L2)2superscript2O(2L^2)O ( 2 L2 ) ⁢(L2)superscript2O(L^2)O ( L2 ) ⁢(L)O(L)O ( L ) ⁢(L2)superscript2O(L^2)O ( L2 ) DPO+RT ⁢(5⁢L2)5superscript2O(5L^2)O ( 5 L2 ) ⁢(2⁢L2)2superscript2O(2L^2)O ( 2 L2 ) ⁢(L)O(L)O ( L ) ⁢(L2)superscript2O(L^2)O ( L2 ) NPO+RT ⁢(3⁢L2)3superscript2O(3L^2)O ( 3 L2 ) ⁢(2⁢L2)2superscript2O(2L^2)O ( 2 L2 ) ⁢(L)O(L)O ( L ) ⁢(L2)superscript2O(L^2)O ( L2 ) IDK+RT ⁢(2⁢L2)2superscript2O(2L^2)O ( 2 L2 ) ⁢(L2)superscript2O(L^2)O ( L2 ) ⁢(L)O(L)O ( L ) ⁢(L2)superscript2O(L^2)O ( L2 ) Opt-Out ⁢(3⁢L2+L⁢log⁡L)3superscript2O(3L^2+L L)O ( 3 L2 + L log L ) ⁢(2⁢L2+L⁢d)2superscript2O(2L^2+Ld)O ( 2 L2 + L d ) ⁢(L)O(L)O ( L ) ⁢(L2)superscript2O(L^2)O ( L2 ) Table 5: Computational complexity of various methods (constants included to better depict the comparison). Opt-Out incurs a slight additional overhead during training while maintaining an identical inference cost to existing methods, ensuring its practicality. Forget Set Retain Set World Set Prob.(↓ ↓) ROUGE(↓ ↓) TR(↑ ↑) Prob.(↑ ↑) ROUGE(↑ ↑) TR(↑ ↑) Prob.(↑ ↑) ROUGE(↑ ↑) TR(↑ ↑) Llama-3.1-8B-Instruct Original 40.7 63.7 46.4 38.6 61.4 50.7 53.1 47.7 64.8 Guardrail 26.5 13.5 47.4 38.7 62.0 50.6 53.5 47.1 64.9 GA* 0.0 0.0 44.8 0.0 0.0 21.3 0.0 0.2 37.0 DPO* 0.0 1.1 52.1 0.0 1.1 20.6 0.0 1.2 60.4 NPO* 0.0 0.0 74.3 0.0 0.0 10.2 0.0 0.2 31.9 IDK* 10.9 1.0 70.0 13.3 1.0 31.1 44.6 1.7 56.1 GA+RT 2.3 5.7 55.3 42.9 61.4 38.7 48.8 34.4 61.2 DPO+RT 2.4 2.4 67.3 41.7 52.7 39.5 49.5 34.5 61.8 NPO+RT 2.4 8.5 66.1 42.2 59.5 41.3 50.0 35.6 62.1 IDK+RT 34.5 4.7 62.6 46.9 58.3 39.0 49.0 34.2 60.7 Opt-Out (ours) 2.2 6.3 75.4 42.4 62.0 40.0 49.8 35.9 62.0 Phi-3.5-Mini-Instruct Original 11.1 64.6 36.6 11.3 63.3 61.1 58.1 50.3 71.5 Guardrail 8.2 57.2 33.2 6.9 55.8 64.2 60.4 50.4 74.2 GA* 0.0 0.1 36.9 0.0 0.2 27.4 0.0 2.0 51.0 DPO* 0.0 0.9 54.9 0.0 0.8 19.6 0.0 1.7 56.6 NPO* 0.0 0.1 58.2 0.0 0.2 19.9 0.0 1.5 53.8 IDK* 22.1 1.1 69.7 23.2 1.0 31.5 41.2 3.5 56.6 GA+RT 5.3 8.6 43.8 51.8 63.0 37.6 45.9 37.8 59.2 DPO+RT 6.5 10.7 44.3 55.6 63.6 38.9 45.7 39.4 59.5 NPO+RT 5.3 9.2 43.7 54.7 65.3 39.3 46.3 40.6 60.1 IDK+RT 44.4 8.7 67.6 56.7 62.8 37.8 45.4 40.2 58.8 Opt-Out (ours) 5.2 9.9 57.0 54.1 64.3 40.2 46.4 40.7 60.6 Table 6: Detailed unlearning results on Llama-3.1-8B-Instruct and Phi-3.5-Mini-Instruct. Appendix D Full Evaluation Results We report the detailed evaluation results after unlearning on Llama-3.1-8B-Instruct and Phi-3.5-Mini-Instruct in Table 6. Note that the truth ratio scores for the retain and world sets have already been inverted. When computing FQ, the probability and ROUGE-L recall scores on the forget set are inverted such that higher scores indicate better performance (i.e., max⁢(0,1−Prob.)max01Prob.max(0,1-Prob.)max ( 0 , 1 - Prob. ) and max(0,1−ROUGEmax(0,1-ROUGEmax ( 0 , 1 - ROUGE). Appendix E Exact Unlearning Results We conduct additional experiments on the TOFU dataset Maini et al. (2024) to further demonstrate the effectiveness of our method and report the results in Table 7. The TOFU dataset is entirely fictitious and was not included during the model’s pretraining. This ensures that the model starts as “perfectly unlearned”, enabling a direct comparison between our approach and exact unlearning. Forget Quality (FQ) in this context is calculated using p-values derived from the Kolmogorov-Smirnov test (KS-Test), which quantifies the divergence between the distributions of Truth Ratios in unlearned models and the exact unlearning model. Retain Quality (RQ) is also slightly adapted here, reflecting the harmonic mean of nine values (Probability, ROUGE, and Truth Ratio measured across Retain Set, Real Authors, and World Facts). For clarity, an asterisk (*) differentiates terms redefined in this context from those in our main paper. The results demonstrate that Opt-Out consistently outperforms other approaches, highlighting its robustness. Appendix F Prompts We display the prompt templates used to generate QA pairs for ELUDe in Figure 5, as well as paraphrased and perturbed QA pairs for the truth ratio evaluation in Figures 6 and 7. Llama-3.1 Phi-3.5 Method FQ* RQ* FQ* RQ* GA+RT 0.0 16.6 46.6 45.2 DPO+RT 0.0 47.7 1.6 44.7 NPO+RT 11.2 46.1 71.3 45.9 IDK+RT 0.0 41.0 22.1 39.3 Opt-Out (ours) 86.6 40.6 86.6 45.9 Table 7: Performance (%) of various methods after unlearning fictitious data in TOFU (5% forget set setting). FQ* is calculated using p-values derived from the KS-Test, and RQ* reflects the harmonic mean of nine values (three metrics measured across three subsets in TOFU). Appendix G Dataset Examples We exhibit dataset examples for one of the target entities Cristiano Ronaldo in Figure 8. For the IDK method, we randomly sample from 100 “I don’t know” (IDK) responses in Maini et al. (2024) and replace it with the original response, as shown in Figure 9. To generate adversarial attack prompts, we slightly modify the prompt template used in Jin et al. (2024) to synthesize nine types of adversarial prompt attacks given the original QA pair. We employ GPT-4o to generate 100 examples for each type, making it a total of 900 attack samples for each entity. Examples for each attack type are illustrated in Figure 10. Figure 5: Prompt template for generating QA pairs for target and neighboring entities. Figure 6: Prompt template for generating paraphrased QA pairs for evaluation. Figure 7: Prompt template for generating perturbed QA pairs for evaluation. Figure 8: Dataset example for the target entity Cristiano Ronaldo. Only the first Wikipedia passage and the first few QA pairs are shown for brevity. Figure 9: Dataset example for the target entity Cristiano Ronaldo with “I don’t know” responses. Figure 10: Adversarial prompt attack examples for the target entity Cristiano Ronaldo.

Instant research discovery