Paper deep dive

Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models

Anmol Mekala, Vineeth Dorna, Shreya Dubey, Abhishek Lalwani, David Koleczek, Mukund Rungta, Sadid Hasan, Elita Lobo

Year: 2024Venue: arXiv preprintArea: Model EditingType: EmpiricalEmbeddings: 93

Models: Llama-2-7B, Phi-1.5

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%

Last extracted: 3/12/2026, 7:55:19 PM

Summary

The paper introduces Alternate Preference Optimization (AltPO), a novel machine unlearning method for Large Language Models (LLMs). AltPO addresses the limitations of existing methods that rely solely on negative feedback, which often leads to nonsensical or inconsistent outputs. By incorporating in-domain positive feedback through plausible, prompt-specific alternate answers alongside negative feedback, AltPO enables effective unlearning while maintaining model utility and response coherence. The authors also propose new evaluation metrics to better assess the quality of responses related to forgotten knowledge.

Entities (5)

AltPO · method · 100%LLM · technology · 100%TOFU · benchmark · 100%DPO · algorithm · 95%Forget Set · data-concept · 95%

Relation Signals (3)

AltPO → evaluatedon → TOFU

confidence 98% · We perform extensive experimentation and ablation tests for each component of our approach on the TOFU dataset

AltPO → improves → Machine Unlearning

confidence 95% · we propose a novel approach called Alternate Preference Optimization (AltPO), which combines negative feedback with in-domain positive feedback on the forget set.

AltPO → utilizes → DPO

confidence 92% · This is achieved by optimizing a variant of the DPO loss (eq. 3) involving in-domain alternate labels

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Machine unlearning aims to efficiently eliminate the influence of specific training data, known as the forget set, from the model. However, existing unlearning methods for Large Language Models (LLMs) face a critical challenge: they rely solely on negative feedback to suppress responses related to the forget set, which often results in nonsensical or inconsistent outputs, diminishing model utility and posing potential privacy risks. To address this limitation, we propose a novel approach called Alternate Preference Optimization (AltPO), which combines negative feedback with in-domain positive feedback on the forget set. Additionally, we introduce new evaluation metrics to assess the quality of responses related to the forget set. Extensive experiments show that our approach not only enables effective unlearning but also avoids undesirable model behaviors while maintaining overall model performance. Our implementation can be found at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

92,791 characters extracted from source content.

Expand or collapse full text

Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models Anmol Mekala1, Vineeth Dorna111footnotemark: 1, Shreya Dubey1, Abhishek Lalwani2, David Koleczek2, Mukund Rungta2, Sadid Hasan2, Elita Lobo1 1University of Massachusetts Amherst, 2Microsoft amekala, vdorna, shreyadubey, elobo@umass.edu alalwani, dkoleczek, rungtamukund, sadidhasan@microsoft.com Primary contributorsPrimary project advisor Abstract Machine unlearning aims to efficiently eliminate the influence of specific training data, known as the forget set, from the model. However, existing unlearning methods for Large Language Models (LLMs) face a critical challenge: they rely solely on negative feedback to suppress responses related to the forget set, which often results in nonsensical or inconsistent outputs, diminishing model utility and posing potential privacy risks. To address this limitation, we propose a novel approach called Alternate Preference Optimization (AltPO), which combines negative feedback with in-domain positive feedback on the forget set. Additionally, we introduce new evaluation metrics to assess the quality of responses related to the forget set. Extensive experiments show that our approach not only enables effective unlearning but also avoids undesirable model behaviors while maintaining overall model performance. Our implementation can be found at https://github.com/molereddy/Alternate-Preference-Optimization. Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models Anmol Mekala1†thanks: Primary contributors, Vineeth Dorna111footnotemark: 1, Shreya Dubey1, Abhishek Lalwani2, David Koleczek2, Mukund Rungta2, Sadid Hasan2, Elita Lobo†thanks: Primaryprojectadvisor1superscript†thanks: Primaryprojectadvisor1^1 @make@thanksPrimaryprojectadvisorstart_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT Primaryprojectadvisor 1University of Massachusetts Amherst, 2Microsoft amekala, vdorna, shreyadubey, elobo@umass.edu alalwani, dkoleczek, rungtamukund, sadidhasan@microsoft.com 1 Introduction Training machine learning models on large-scale datasets presents several challenges, such as potential copyright issues, inadvertent inclusion of sensitive information, or other undesirable influences from the training data (Nguyen et al., 2022; Liu, 2024; Zhang et al., 2023). The increasing adoption of Large Language Models (LLMs) with memorization capabilities (Karamolegkou et al., 2023) has exacerbated these issues. This has driven the development of machine unlearning methods, which aim to remove the influence of data that needs to be forgotten (Liu, 2024). In an ideal world, a perfectly unlearned model would be indistinguishable from a model that was never exposed to the data in question, achieving what is known as exact unlearning. However, this is often impractical in real-world settings. Instead, we focus on the more feasible approach of approximate unlearning (Nguyen et al., 2022), which seeks to modify model weights post-training to minimize the impact of the data to be forgotten. Previous studies have demonstrated that machine unlearning can often introduce undesirable effects in the resulting model, including catastrophic forgetting and reduced overall utility (Kurmanji et al., 2024; Ma et al., 2022). In classification and regression tasks (Nguyen et al., 2022; Bourtoule et al., 2021; Triantafillou et al., 2024), undesirable effects manifest as a redistribution of scores across classes or predicted values. However, unlearning in LLMs can have more significant effects on model behavior, given the requirement of generating coherent text (Maini et al., 2024; Zhang et al., 2024). Existing unlearning methods in this setting often lead to incoherent or inconsistent responses from unlearned LLMs (see Figure 1), including responses related to the forgotten knowledge, which is undesirable. Such behaviors may unintentionally reveal details about the unlearning process or the forgotten data, posing potential privacy risks and increasing the model’s susceptibility to membership inference attacks (Chen et al., 2021; Shi et al., 2024; Duan et al., 2024). The goal of unlearning in LLMs is to reduce memorization or prevent the leakage of information specific to the forgotten set, while maintaining the model’s overall behavior and performance. Striking this balance is challenging and requires careful consideration of both the effectiveness of unlearning and the model’s overall performance (Liu et al., 2024b). Figure 1: The unlearning pipeline and the resulting generations post unlearning with different methods. To address the aforementioned challenges, we propose a novel method, AltPO—(Alternate-Preference Optimization), which ensures stable and effective unlearning by incorporating additional positive feedback for plausible alternative answers to the forgotten data, along with negative feedback targeting the knowledge to be erased. This approach enables the model to forget specific information while maintaining the ability to generate coherent and consistent responses. Additionally, recognizing the shortcomings of current evaluation metrics for unlearning in question-answering tasks, we introduce new metrics specifically designed to better evaluate the impact of unlearning on response quality related to forgotten knowledge. Our main contributions are as follows: • Algorithm: We propose a novel unlearning method AltPO using alternate responses and adapting the model to these while contrasting against the LLM’s existing knowledge (Section 3). • Discovery and evaluation of failure modes: We point out failure modes of prior approaches that are not captured by existing metrics and introduce new evaluation metrics to address these gaps (Section 4). • Empirical evaluation: We perform extensive experimentation and ablation tests for each component of our approach on the TOFU dataset, showing that AltPO-unlearned models achieve the highest unlearning scores on the existing metrics, while also achieving better and near-perfect scores on both existing and new evaluation metrics (Section 6). 2 Preliminaries 2.1 Problem Statement and Notation Given an LLM, denoted by π , trained on a dataset D=(xi,yi):i=1,…,nDconditional-setsubscriptxisubscriptyii1…n D=\(x_i,y_i):i=1,…,n\D = ( xroman_i , yroman_i ) : i = 1 , … , n , where xisubscriptx_ixitalic_i is an input prompt and yisubscripty_iyitalic_i is the corresponding response (e.g., question-answer pairs), let Df⊂DsubscriptDfD D_f⊂ DDroman_f ⊂ D represent the forget set, which we aim to unlearn from the model. The remaining dataset, referred to as the retain set, is represented by Dr=D∖DfsubscriptDrDsubscriptDf D_r= D D_fDroman_r = D ∖ Droman_f and includes all the data outside the forget set. Goal Our goal is to remove the influence of the forget set DfsubscriptDf D_fDroman_f from π , transforming it into an unlearned model πunlsubscriptunl 1.5$π$_unlπunl that behaves approximately indistinguishably from a reference retain model πretsubscriptret 1.5$π$_retπret, trained solely on the retain set Dr=D∖DfsubscriptDrDsubscriptDf D_r= D D_fDroman_r = D ∖ Droman_f. Simultaneously, we aim to preserve π ’s general utility as a language model, even on DfsubscriptDf D_fDroman_f. Constraint We are required to use at most O⁢(|Df|)subscriptDfO(| D_f|)O ( | Droman_f | ) steps while unlearning DfsubscriptDf D_fDroman_f. 2.2 The TOFU Benchmark The TOFU benchmark (Maini et al., 2024) primarily consists of a dataset containing facts about 200 fictitious authors and a chat model that is fine-tuned to incorporate these facts through question-answer pairs. Unlearning is performed on a subset of authors, and TOFU provides evaluation metrics to quantify the extent of forgetting and the utility of the model, see Figure 1. In this framework, an unlearning algorithm is tasked with forgetting specific subsets corresponding to 1%percent11\%1 %, 5%percent55\%5 %, and 10%percent1010\%10 % of these authors, while maintaining performance on the remaining data (the retain set DrsubscriptDr D_rDroman_r). Additionally, post-unlearning, the model is required to preserve its performance on other related datasets, including real-world authors and general knowledge, such as world facts. The benchmark evaluates unlearning using the following key metrics: Forget Quality (FQ): TOFU quantifies forget quality by assessing how indistinguishable the unlearned model is from the retain model, using the Kolmogorov–Smirnov (KS) statistical test on the ‘Truth Ratio’ statistic. The Truth Ratio compares the likelihood of the model predicting the correct answer versus perturbed (incorrect) answers. The p-value from the KS test is used to measure the quality of unlearning, with a p-value greater than 0.05 indicating successful unlearning. Model Utility (MU): This measures the model’s general performance, which must be preserved post-unlearning. TOFU evaluates MU as an aggregated score based on the model’s average probability, ROUGE score for the true answers, and the Truth Ratio on non-forget datasets. This score reflects the model’s retained performance after unlearning. For more details of TOFU’s metric calculations, we refer to (Maini et al., 2024). 2.3 Unlearning Losses Previous unlearning loss functions Maini et al. (2024); Zhang et al. (2024) can be generally described using two key components: positive and negative feedback. These components offer a useful framework for evaluating the effects of unlearning. In this subsection, we define these concepts and introduce baseline methods along the way. Negative Feedback: This component aims to reduce the likelihood of specific responses, effectively lowering the model’s performance on the forget set in order to reverse the effects of training the LLM π on the forget set DfsubscriptDf D_fDroman_f. Examples of methods incorporating negative feedback include: gradient ascent (GA) on negative log-likelihood (NLL) loss LGAsubscriptLGA L_GALGA (Maini et al., 2024): NLL⁢(yf|xf)≐−log⁡πθ⁢(yf|xf)approaches-limitNLLconditionalsubscriptsubscriptsubscriptconditionalsubscriptsubscript (y_f|x_f) - 1.5$π$_θ(% y_f|x_f)NLL ( yitalic_f | xitalic_f ) ≐ - log πitalic_θ ( yitalic_f | xitalic_f ) LGA≐−NLL⁢(yf|xf)approaches-limitsubscriptLGANLLconditionalsubscriptyfsubscriptxf L_GA -NLL(y_f|x_f)LGA ≐ - NLL ( yroman_f | xroman_f ) and negative preference optimization loss (DPO loss without positive samples) (Zhang et al., 2024): LNPO-FG≐−2β⁢log⁡σ⁢(−β⁢log⁡πθ⁢(yf|xf)π⁢(yf|xf))approaches-limitsubscriptLNPO-FG2subscriptconditionalsubscriptyfsubscriptxfconditionalsubscriptyfsubscriptxf L_NPO-FG - 2β σ (% -β 1.5$π$_θ(y_f|x_f) 1.5$% π$(y_f|x_f) )LNPO-FG ≐ - divide start_ARG 2 end_ARG start_ARG β end_ARG log σ ( - β log divide start_ARG πitalic_θ ( yroman_f | xroman_f ) end_ARG start_ARG π ( yroman_f | xroman_f ) end_ARG ) (1) where β is the regularization strength, and π is the reference model (state of the model prior to unlearning). Negative feedback helps eliminate information related to the forgotten set; however, overgeneralizing this feedback during unlearning can harm the model’s utility, potentially resulting in nonsensical responses. To address this, it is typically paired with positive feedback on related data to preserve response coherence and maintain overall performance. Positive Feedback: This loss component aims to increase the likelihood of specific responses, improving performance on certain segments of the dataset, such as the retain set during unlearning. It helps preserve the model’s language generation capabilities and prevents unlearning from impacting model’s performance on datasets beyond the forget set. In the loss functions that follow, a positive feedback term for randomly selected examples from the retain set (xr,yr)∼Drsimilar-tosubscriptsubscriptsubscriptDr(x_r,y_r) D_r( xitalic_r , yitalic_r ) ∼ Droman_r is added alongside the negative feedback terms, weighted by wr>0subscript0w_r>0witalic_r > 0. LGradDiff≐LGA+wr⁢NLL⁢(yr|xr)approaches-limitsubscriptLGradDiffsubscriptLGAsubscriptwrNLLconditionalsubscriptyrsubscriptxr L_GradDiff L_GA+w_r% NLL(y_r|x_r)LGradDiff ≐ LGA + wroman_r NLL ( yroman_r | xroman_r ) (2) LNPO≐LNPO-FG+wr⁢NLL⁢(yr|xr)approaches-limitsubscriptLNPOsubscriptLNPO-FGsubscriptwrNLLconditionalsubscriptyrsubscriptxr L_NPO L_NPO-FG+w_r% NLL(y_r|x_r)LNPO ≐ LNPO-FG + wroman_r NLL ( yroman_r | xroman_r ) Preference Optimization Losses: These losses are based on Direct Preference Optimization (DPO). The DPO loss (Rafailov et al., 2024) has been applied in baseline methods like IdkPO and proposed in previous works such as NPO (Zhang et al., 2024) (eq. 2). This loss function contrasts pairs of positive and negative samples by increasing the likelihood of positive samples (positive feedback) while reducing the likelihood of negative samples (negative feedback). LDPO(yalt,yf|xf)≐−2βlogσ(βlogπθ⁢(yalt|xf)π⁢(yalt|xf) L_DPO(y_alt,y_f|x_f) - % 2β σ (β 1.5$π$_θ(y_% alt|x_f) 1.5$π$(y_alt|x_f) .LDPO ( yalt , yroman_f | xroman_f ) ≐ - divide start_ARG 2 end_ARG start_ARG β end_ARG log σ ( β log divide start_ARG πitalic_θ ( yalt | xroman_f ) end_ARG start_ARG π ( yalt | xroman_f ) end_ARG (3) −βlogπθ⁢(yf|xf)π⁢(yf|xf)) . -β 1.5$π$_θ(% y_f|x_f) 1.5$π$(y_f|x_f) )- β log divide start_ARG πitalic_θ ( yitalic_f | xitalic_f ) end_ARG start_ARG π ( yitalic_f | xitalic_f ) end_ARG ) LIdkPO≐LDPO⁢(yidk,yf|xf)+wr⁢NLL⁢(yr|xr)approaches-limitsubscriptLIdkPOsubscriptLDPOsubscriptyidkconditionalsubscriptyfsubscriptxfsubscriptwrNLLconditionalsubscriptyrsubscriptxr L_IdkPO L_DPO(y_ % idk,y_f|x_f)+w_rNLL(y_r|x_r)LIdkPO ≐ LDPO ( yidk , yroman_f | xroman_f ) + wroman_r NLL ( yroman_r | xroman_r ) Unlearning can be achieved by optimizing a DPO loss, where the forget set response yfsubscripty_fyitalic_f serves as the negative sample and any alternate answer yaltsubscriptalty_altyalt as the positive sample. Prior works (Zhang et al., 2024) have shown that optimizing a DPO-based loss is more effective at reducing the likelihood of negative samples compared to optimizing the NLL loss. The NPO loss (see eq. 2) provides a more stable unlearning process by integrating LNPO-FGsubscriptLNPO-FG L_NPO-FGLNPO-FG with positive feedback from the retain set, yielding the final loss LNPOsubscriptLNPO L_NPOLNPO. Notably, positive feedback is applied exclusively to retain set examples, while the forget set receives only negative feedback. Another approach which we term IdkPO (derived from DPO), is explored in (Maini et al., 2024), where the model is aligned with alternate answers like "I don’t know." 3 AltPO: Alternate Preference Optimization Figure 2: The AltPO unlearning algorithm We now address the limitations of previous unlearning methods discussed in Section 2.3 and introduce our approach, which leverages preference optimization using alternate labels. We then outline the process for generating these labels and present the loss function that underpins our method, as illustrated in Figure 2. NPO (Zhang et al., 2024) and IdkPO have shown promising results on TOFU, but they often generate nonsensical and inconsistent responses upon closer inspection. A key limitation of the NPO loss is its lack of positive feedback for forget set prompts, leaving the model without guidance on how to behave post-unlearning, which often leads to nonsensical outputs. IdkPO (Maini et al., 2024) despite using positive feedback, relies on predefined, prompt-independent responses that significantly differ from the model’s original answers. This misalignment necessitates more drastic changes to the model’s weights, potentially degrading response quality. As Maini et al. (2024) also note, the IdkPO objective is unstable during training. The core weakness of these methods is their failure to offer in-distribution positive feedback on responses to forget prompts. To overcome this, our approach generates plausible, prompt-specific alternate answers to serve as positive feedback. This results in an objective that is more stable and easier to optimize. Generating Alternate Labels: To induce unlearning, the alternative responses must be both plausible and distinct from the learned knowledge from the forget set. To generate such responses, we prompt the LLM π with instructions to create plausible alternatives, while operating under the hypothesis that its behavior will resemble that of the retain model πretsubscriptret 1.5$π$_retπret. Alternatively, such answers can be generated also by an LLM other than π . An LLM that was not trained on DfsubscriptDf D_fDroman_f would be ideal for generating plausible alternate answers as it would not leak information from the forget set. Given a question-answer pair to be unlearned (xf,yf)subscriptsubscript(x_f,y_f)( xitalic_f , yitalic_f ) we use a prompt PP (outlined in Table 4 of the Appendix) to instruct π for generating an alternate response yasubscripty_ayitalic_a that changes facts from yfsubscripty_fyitalic_f. Table 6 in the appendix presents examples of alternate responses. ya∼π(⋅|(xf,yf))y_a 1.5$π$ (·| P(x_f,y_f) )yitalic_a ∼ π ( ⋅ | P ( xitalic_f , yitalic_f ) ) AltPO Loss: We align the LLM π to the new alternate labels yasubscripty_ayitalic_a while contrasting them with the forget response yfsubscripty_fyitalic_f. This is achieved by optimizing a variant of the DPO loss (eq. 3) involving in-domain alternate labels yasubscripty_ayitalic_a as positive samples, resulting in a more stable objective. Additionally, as in other baselines, we apply NLL loss to the retain set, to prevent the model from incorrectly generalizing to unrelated contexts. LAltPO≐ya⁢[LDPO⁢(ya,yf|xf)]+wr⁢NLL⁢(yr|xr)approaches-limitsubscriptLAltPOsubscriptsubscriptyadelimited-[]subscriptLDPOsubscriptyaconditionalsubscriptyfsubscriptxfsubscriptwrNLLconditionalsubscriptyrsubscriptxr L_ AltPO _y_a[ L_ % DPO(y_a,y_f|x_f)]+w_r\,NLL(y_r|x_r)LAltPO ≐ blackboard_Ey start_POSTSUBSCRIPT a end_POSTSUBSCRIPT [ LDPO ( yroman_a , yroman_f | xroman_f ) ] + wroman_r NLL ( yroman_r | xroman_r ) Multiple Alternate Labels: Ideally, an unlearned LLM should avoid certainty in any single answer, as often happens when training with just one alternate response. Such confident replication of the alternate answers poses problems with misinformation. To address this, we generate M alternate responses by sampling yasubscripty_ayitalic_a randomly and using all of these in our preference dataset for alignment. In our ablations, we show that this introduces uncertainty, effectively confusing the model and resulting in better forgetting. Misinformation can then be prevented by using methods like uncertainty-aware decoding to filter out low certainty outputs (Ji et al., 2023; Kadavath et al., 2022). 4 Improving Unlearning Evaluations In this section, we outline the failure cases observed in LLM unlearning, discuss their impact on unlearning goals, and introduce new evaluation metrics. 4.1 Failure Modes of Prior Approaches Despite strong performance on TOFU’s metrics, methods like NPO and IdkDPO often produce incoherent responses, such as nonsensical answers and inconsistent answers where the model contradicts the prompt, sometimes by altering names. This issue is illustrated in the table in Figure 1 (see NPO and IdkPO rows). Although Zhang et al. (2024) identifies the problem of nonsensical generations on forget prompts, it has yet to be properly quantified. TOFU’s evaluation of forget quality fails to penalize these errors, as it only measures the probability of predefined sentences rather than analyzing the generated responses on forget set questions. Additionally, TOFU’s utility evaluations focused solely on entities outside the forget set, overlooking the decreased utility observed on forget entities. These incoherent generations not only degrade the model’s overall performance but also pose potential privacy risks, as detailed below. Decreased Utility: An LLM is at the least expected to generate plausible, prompt-consistent responses of high quality, even when it has never encountered the entities mentioned in the prompt. Therefore, the unlearned model should maintain its utility on the forget set by producing coherent and sensible responses to forget set prompts, even if those responses are hallucinated. A failure to achieve this should be regarded as a reduction in utility on the forget set. Privacy Leakage: Nonsensical behavior on the forget set can unintentionally reveal information about the model’s training data, thereby posing potential privacy risks. Such behavior may make the model more vulnerable to membership inference attacks (Shi et al., 2024; Duan et al., 2024) and expose details of the unlearning process. This counterproductive impact of machine unlearning on privacy has been explored by Chen et al. (2021). 4.2 New Evaluation Metrics To capture the failure cases discussed in Section 4.1, we introduce two new evaluation metrics: Forget Utility (FU) and Cleanness Indistinguishability (CI), based on the Text Cleanness (TC) statistic. Forget Utility (FU): This metric evaluates the model’s utility by assessing whether its responses on the forget set are plausible, penalizing both nonsensical outputs and prompt-inconsistent responses. We rely on LLM-based evaluation (Chiang and Lee, 2023), with GPT-4o mini111refers to the gpt-4o-mini-2024-07-18 endpoint as a judge (prompt given in Table 5 of Appendix C), determining whether they are sensible and consistent given the question. TOFU 10% TOFU 5% TOFU 1% Method Forgetting Utility Forgetting Utility Forgetting Utility FQ (↑ ↑) CI (↑ ↑) MU (↑ ↑) FU (↑ ↑) FQ (↑ ↑) CI (↑ ↑) MU (↑ ↑) FU (↑ ↑) FQ (↑ ↑) CI (↑ ↑) MU (↑ ↑) FU (↑ ↑) Finetune 2.2e-20 1.7e-4 0.62 1.0 3.5e-16 5.2e-2 0.62 0.97 1.9e-4 1.0 0.62 0.90 Retain 1.0 1.0 0.62 1.0 1.0 1.0 0.62 0.98 1.0 1.0 0.62 0.97 GradAsc 2.4e-7 3.2e-3 0.35 0.97 4.1e-3 2.7e-51 0.14 0.16 0.24 2.7e-9 0.53 0.56 GradDiff 3.7e-5 0.0 0.64 0.01 5.1e-5 1.5e-23 0.56 0.51 0.10 5.9e-20 0.57 0.05 NPO 0.68 1.5e-13 0.64 0.20 0.24 1.9e-7 0.63 0.35 0.46 0.44 0.57 0.65 IdkPO 0.37 0.0 0.59 0.65 0.18 6.4e-10 0.61 0.66 0.6 6.6e-6 0.52 1.0 AltPO(ours) 0.74 0.92 0.62 0.86 0.26 0.74 0.63 0.83 0.94 0.72 0.62 0.83 Table 1: Performance of various unlearning methods for different splits of the TOFU benchmark, averaged over 3 random seeds, on Llama2. FQ, CI, MU, and FU represent Forget Quality, Cleanness Indistinguishability, Model Utility, and Forget Utility, respectively. ‘Finetune’ denotes the model yet to undergo unlearning, while ‘Retain’ refers to the model trained solely on the retain set. Each method aims to achieve the scores of the corresponding Retain model. We use (↑ ↑) to indicate that a higher value is preferable. The best results are highlighted in bold, and in the MU column, where bolding indicates performance is preserved on par with “finetune”. Cleanness Indistinguishability (CI): This metric evaluates the privacy leakage by measuring the distinguishability between unlearned model πunlsubscriptunl 1.5$π$_unlπunl and retain model πretsubscriptret 1.5$π$_retπret based on nonsensical responses. Similar to how TOFU’s FQ distinguishes models using the Truth Ratio (TR) statistic by applying the KS test on the forget set, CI uses Text Cleanness TCxfsubscriptTCsubscriptTC_x_fTCx start_POSTSUBSCRIPT f end_POSTSUBSCRIPT scores, which we define next. For the model responses generated on the forget set ygen∼π(⋅|xf)y_gen 1.5$π$ (·|x_f )ygen ∼ π ( ⋅ | xitalic_f ), we compute the non-gibberish probability, TCxf=Pr⁡(ygen)subscriptTCsubscriptPrsubscriptgenTC_x_f= (y_gen)TCx start_POSTSUBSCRIPT f end_POSTSUBSCRIPT = Pr ( ygen ), using a publicly available DistilBERT-based gibberish classifier222Link to model: https://huggingface.co/madhurjindal/autonlp-Gibberish-Detector-492513457333We also experimented using the perplexity of another model to evaluate for nonsensical text, which (Gandikota et al., 2024) uses as a reverse perplexity R-PPL metric. We found that this evaluation is not robust to greedy decoding, as it gives high probabilities for nonsensical texts made of repetitions.. We then perform KS-test on TCxfsubscriptTCsubscriptTC_x_fTCx start_POSTSUBSCRIPT f end_POSTSUBSCRIPT distribution to distinguish the unlearned and retain models: CI≐KS-Test⁢(TC⁢(πunl),TC⁢(πret))approaches-limitCIKS-TestTCsubscriptunlTCsubscriptretCI -Test (TC( 1.5$π$_% unl),TC( 1.5$π$_ret) )CI ≐ KS-Test ( TC ( πunl ) , TC ( πret ) ) We can also use the mean TC=⁢[TCxf]TCdelimited-[]subscriptTCsubscriptTC=E[TC_x_f]TC = blackboard_E [ TCx start_POSTSUBSCRIPT f end_POSTSUBSCRIPT ] score as a simpler utility metric as an alternative to FU, given the cost of LLM-as-judge evaluations. Like FU, it measures utility on forget prompts by identifying nonsensical responses, but it does not penalize inconsistent answers. Therefore, we report FU in the results section and provide TC scores in Tables 8 to 10 in the Appendix. 5 Related Work We now discuss two closely related approaches: Eldan and Russinovich (2023) and Dong et al. (2024) use positive feedback on the forget set to stabilize unlearning by substituting privacy-sensitive “anchor” words with alternate positive token-level labels. Eldan and Russinovich (2023) uses GPT-4 to identify anchor tokens, while Dong et al. (2024) considers all nouns as anchors. In contrast, our method avoids selecting specific anchor words and generates multiple alternate answers consistent with the original question. Dong et al. (2024) derives alternate completions based on next-token probabilities, excluding the highest-ranked token and (Eldan and Russinovich, 2023) uses scores from a model trained further on the forget set along with substitutions proposed by GPT-4. We simplify this by directly instructing an LLM to generate multiple alternative answers. While both works use a cross-entropy loss, our AltPO method employs a DPO-style loss to align the model with alternate answers, explicitly incorporating negative feedback. Ablation studies in Section 6.4 show how these elements improve our method’s performance. In a concurrent work, Jin et al. (2024) use an approach similar to ours in their RWKU unlearning benchmark. Discussion of the differences between their approach and ours, analysis of the results, along with a broader review of the machine unlearning literature, is provided in Appendix A. 6 Experiments Figure 3: Trajectory of MU versus log(FQ) for different unlearning methods. Marker size represents the epoch number. Trajectories are reported for the 10, 5, 1% splits of TOFU in order, on Llama2. 6.1 Evaluation Metrics We report TOFU’s main unlearning metrics—forget quality (FQ) and model utility (MU)—to compare against baselines and other methods. Additionally, we report scores for the FU and CI metrics introduced in Section 4.2. Further results on the rest of the TOFU’s metrics are provided in Appendix D’s plots, along with average TC scores in Tables 8 to 10 of the Appendix. 6.2 Implementation Details We use the TOFU-finetuned Llama2-7b-chat model (Touvron et al., 2023) checkpoints provided by Maini et al. (2024) to enable direct comparison. Greedy sampling is applied for all generations during the unlearning process. The model is trained using our unlearning losses for the equivalent of N=1010N=10N = 10 epochs over the forget dataset. For generating alternate answers, we sample M=55M=5M = 5 responses from the model using temperature sampling with T=1.01.0T=1.0T = 1.0. To ensure that the computational cost of our method matches that of the baselines, we train the model for NM=22 NM=2divide start_ARG N end_ARG start_ARG M end_ARG = 2 epochs. To evaluate the potential of both our method and the baselines fairly, we perform a comprehensive grid search to identify optimal parameters for each. All results are averaged over three random seeds, with the best hyperparameters selected based on performance on the MU-FQ tradeoff Pareto frontier shown in Figure 3. Additional details on training and hyperparameter tuning are provided in Appendix B. 6.3 Results In the following results, we use Llama2 and first compare AltPO with baseline methods (discussed in Section 2.3), demonstrating that it (1) achieves superior unlearning as measured by FQ and CI, (2) preserves the model’s utility on both forget and non-forget prompts, (3) shows a more stable trajectory of the evaluation metrics over the training steps, as shown in Table 1. Finally, we present ablation studies, highlighting the importance of each component in our method. Our results also extend to the Llama3.2 model, results for which can be found in Table 11 of the appendix. For more details of our results, including the TC scores and variance in results across seeds, see Tables 8 to 10 in the appendix. Extent of forgetting: AltPO demonstrates superior forgetting compared to other methods, as seen in FQ and CI columns of Table 1. The p-values of these tests significantly exceed 0.050.050.050.05, indicating that AltPO produces models nearly indistinguishable from the gold retain model in terms of both Truth Ratio (measuring confidence on original forget answers) and Text Cleanness (assessing text quality of forget set responses) distributions. Our results are equally strong across the 1%percent11\%1 %, 5%percent55\%5 % and 10%percent1010\%10 % subsets, whereas Maini et al. (2024); Zhang et al. (2024) found larger subsets much more difficult to forget. Model performance post-unlearning: As shown in the MU column of Table 1, AltPO successfully retains the full 0.620.620.620.62 model utility (MU) of the initial model. In cases where the MU scores of other methods are comparable to ours, AltPO is substantially ahead in FU, showing that on forget prompts, AltPO generates more coherent and question-consistent responses. Sample generations on forget prompts from unlearned models of each method are shown in Table 7 in Appendix. Although AltPO generally outperforms other methods when considering all metrics together, we observe some reduction in FU scores. Specifically, our method underperforms on the FU metric compared to the GradAsc baseline in the forget 10%percent1010\%10 % scenario and IdkPO in the forget 1%percent11\%1 % scenario. However, these baselines perform worse across other utility and forget quality metrics. While AltPO never generates nonsensical responses, we do notice occasional slight modifications to names in the outputs, leading to a drop in FU scores below the perfect score of 1111. Stability during training: For unlearning to be adaptable in practice, it is crucial to maintain stability throughout the entire training process, with the utility of the model not experiencing large variations during training. As shown in Figures 3 and 4, unlike many methods that incorporate only negative or sub-optimal positive feedback on the forget set, our method achieves more stable unlearning trajectories across multiple splits. Figure 4: Trajectory of FU throughout the unlearning process for 10%percent1010\%10 % forget split of TOFU, using Llama2. 6.4 Ablation Experiments We conduct ablations on our method and baseline approaches to validate the necessity of various components. Our method incorporates the following key elements: (1) leveraging positive forget feedback, (2) pairing it with negative forget feedback, (3) ensuring that positive feedback is relevant and in-distribution, (4) incorporating negative feedback through a DPO loss instead of a negative NLL formulation, and (5) utilizing multiple positive feedback responses. We now discuss the effect of each element in detail, looking at the results in Table 2 unless otherwise mentioned. Need for positive forget feedback: Comparing FU between NPO (which uses only negative feedback on the forget set) and our AltPO (which leverages both negative and positive feedback), we observe that relying solely on negative feedback for unlearning can be destructive and impair the model’s ability to generate coherent responses on the forget set. In contrast, incorporating positive feedback helps preserve the model’s language capabilities while still achieving effective unlearning. Need for negative feedback alongside positive feedback: Here we use a baseline AltNLL-pos method that trains with only positive feedback on alternate labels (in an NLL formulation), which closely matches Eldan and Russinovich (2023)’s approach. LAltNLL-pos≐ya⁢[NLL⁢(ya|xf)]approaches-limitsubscriptLAltNLL-possubscriptsubscriptyadelimited-[]NLLconditionalsubscriptyasubscriptxf L_AltNLL-pos _y_a[ % NLL(y_a|x_f)]Lsansserif_AltNLL italic_-pos ≐ blackboard_Ey start_POSTSUBSCRIPT a end_POSTSUBSCRIPT [ NLL ( yroman_a | xroman_f ) ] Additionally, we create a DPO-style version of AltNLL-pos by removing negative feedback on the forget set from AltPO, relying solely on positive feedback to create AltPO-pos. LAltPO-pos≐approaches-limitsubscriptLAltPO-posabsent L_ AltPO -pos _AltPO italic_-pos ≐ −2β⁢log⁡σ⁢(β⁢log⁡πθ⁢(ya|xf)π⁢(yf|xf))2subscriptconditionalsubscriptsubscriptconditionalsubscriptsubscript - 2β σ (β 1.5$% π$_θ(y_a|x_f) 1.5$π$(y_f|x_f) )- divide start_ARG 2 end_ARG start_ARG β end_ARG log σ ( β log divide start_ARG πitalic_θ ( yitalic_a | xitalic_f ) end_ARG start_ARG π ( yitalic_f | xitalic_f ) end_ARG ) +wr⁢NLL⁢(yr|xr)subscriptNLLconditionalsubscriptsubscript +w_rNLL(y_r|x_r)+ witalic_r NLL ( yitalic_r | xitalic_r ) Method Loss on DfsubscriptDf D_fDroman_f Forgetting Utility FQ (↑ ↑) MU (↑ ↑) FU (↑ ↑) NPO LDPO⁢(∗,yf∣xf)subscriptLDPOconditionalsubscriptsubscriptxf L_DPO(*, lightred$y_f$ x_f)LDPO ( ∗ , yitalic_f ∣ xroman_f ) 0.68 0.64 0.52 IdkPO LDPO⁢(yidk,yf∣xf)subscriptLDPOsubscriptidkconditionalsubscriptsubscriptxf L_DPO( lightgreen$y_idk$, % lightred$y_f$ x_f)LDPO ( yidk , yitalic_f ∣ xroman_f ) 0.37 0.59 0.65 AltNLL-pos i⁢[NLL⁢(yai|xf)]subscriptdelimited-[]NLLconditionalsubscriptsuperscriptsubscriptE_i[NLL( lightgreen$y^i_a$|x_f)]blackboard_Ei [ NLL ( yitalic_iitalic_a | xitalic_f ) ] 2.2e-15 0.63 0.87 AltNLL i⁢[NLL⁢(yai∣xf)]subscriptdelimited-[]NLLconditionalsubscriptsuperscriptsubscriptE_i[NLL( lightgreen$y^i_a$ x_f)]blackboard_Ei [ NLL ( yitalic_iitalic_a ∣ xitalic_f ) ] 0.14 0.61 0.49 −NLL⁢(yf∣xf)NLLconditionalsubscriptsubscript-NLL( lightred$y_f$ x_f)- NLL ( yitalic_f ∣ xitalic_f ) AltPO-pos i[LDPO(yai,∗∣xf)]E_i[ L_DPO( lightgreen$y^i_a% $,* x_f)]blackboard_Ei [ LDPO ( yitalic_iitalic_a , ∗ ∣ xroman_f ) ] 2.5e-12 0.61 0.95 AltPO i⁢[LDPO⁢(yai,yf∣xf)]subscriptdelimited-[]subscriptLDPOsubscriptsuperscriptconditionalsubscriptsubscriptxfE_i[ L_DPO( lightgreen$y^i_a% $, lightred$y_f$ x_f)]blackboard_Ei [ LDPO ( yitalic_iitalic_a , yitalic_f ∣ xroman_f ) ] 0.74 0.62 0.88 AltPObasesubscriptAltPObase AltPO_baseAltPObase i⁢[LDPO⁢(yai,yf∣xf)]subscriptdelimited-[]subscriptLDPOsubscriptsuperscriptconditionalsubscriptsubscriptxfE_i[ L_DPO( lightgreen$y^i_a% $, lightred$y_f$ x_f)]blackboard_Ei [ LDPO ( yitalic_iitalic_a , yitalic_f ∣ xroman_f ) ] 0.71 0.64 0.94 Table 2: Ablation study of various methods and their performance in terms of forgetting and utility. Results are reported for unlearning the 10%percent1010\%10 % split of the TOFU on Llama2. The ∗*∗ in DPO loss represents the lack of positive or negative sample terms in the loss function (in NPO and PPO, respectively). The green boxes represent positive feedback, while the red boxes represent negative feedback. Comparing FQ between AltNLL-pos and AltNLL; and between AltPO-pos and AltPO, we find that relying solely on positive feedback is insufficient for effectively removing the model’s knowledge from the forget set. This highlights the necessity of incorporating both positive and negative feedback for successful unlearning: simply performing continual learning on alternate answers without removing the previously learned knowledge is insufficient. Need for positive feedback to be prompt-relevant: We substantiate this by comparing our method with IdkPO, which uses positive feedback with prompt-independent alternate labels from outside the model’s distribution. AltPO generally outperforms IdkPO in both FQ and MU and has a more stable training profile as seen in Figure 3. This indicates that using contextually relevant and in-domain responses for positive feedback, than generic pre-defined ones, decreases damage to the utility of the LLM. DPO-style loss outperforms NLL in delivering negative feedback: Here we replace the DPO-style formulation in AltPO with an NLL-based loss, referred to as AltNLL. Like AltPO, this approach contrasts the likelihoods of alternate and forget set answer pairs. LAltNLLsubscriptLAltNLL L_AltNLLLAltNLL ≐ya⁢[(NLL⁢(ya|xf)−NLL⁢(yf|xf))]approaches-limitabsentsubscriptsubscriptdelimited-[]NLLconditionalsubscriptsubscriptNLLconditionalsubscriptsubscript _y_a [ (NLL(y_a|x_f)- % NLL(y_f|x_f) ) ]≐ blackboard_Ey start_POSTSUBSCRIPT a end_POSTSUBSCRIPT [ ( NLL ( yitalic_a | xitalic_f ) - NLL ( yitalic_f | xitalic_f ) ) ] (4) +wr⁢NLL⁢(yr|xr)subscriptNLLconditionalsubscriptsubscript +w_rNLL(y_r|x_r)+ witalic_r NLL ( yitalic_r | xitalic_r ) AltPO outperforms AltNLL in all forgetting and utility metrics. These results match Zhang et al. (2024)’s observation of the advantage of DPO-style loss over NLL, where they compare NPO with the GradDiff baseline both of which only use negative feedback on forget responses, with the difference being in the loss formulation in the DPO style v/s NLL. In Table 1, we verify their observation, comparing the NPO and GradDiff rows. NPO achieves better FU and MU scores, with gibberish and inconsistent responses being less likely than in GradDiff. Need for multiple alternate answers: In Table 3, we analyze the effect of increasing alternate answers M in our method. Higher M improves forgetting (FQ) values and we use M=55M=5M = 5 alternate answers as the default in our results. We do observe that FQ decreases to 0.25 at the extreme case of M=1010M=10M = 10 from 0.74 at M=55M=5M = 5. Despite this decrease, the score remains above the statistical significance threshold of 0.05, demonstrating effective forgetting. We also evaluate model self-confidence, defined as the probability assigned to responses on forget set prompts. Low self-confidence is desirable to avoid confidently generating incorrect answers, aligning with approaches like uncertainty-aware decoding (Ji et al., 2023). AltPO with M>11M>1M > 1 achieves lower self-confidence than the Retain model, with further reductions as M increases. While NPO achieves even lower self-confidence, it often reflects low confidence in nonsensical outputs. Method M Forgetting Utility Self-Confidence (↓ ↓) FQ (↑ ↑) MU (↑ ↑) Finetune - 2.2e-20 0.62 0.99 Retain - 1.0 0.62 0.89 NPO - 0.68 0.64 0.58 AltPO 1 0.06 0.62 0.87 AltPO 2 0.1 0.63 0.83 AltPO 5 0.74 0.62 0.78 AltPO 10 0.25 0.63 0.65 Table 3: Ablation study on the number of alternate answers (M) with the self-confidence score of the model. Results are reported for the 10% split of TOFU on Llama-2. Note that M=55M=5M = 5 is the default in all AltPO experiments. Effect of using different models to generate alternate labels: We also explore the effects of leveraging other models to generate alternate answers for unlearning. This is relevant in cases where the given models produce alternate answers that inadvertently reveal original information due to poor instruction-following capabilities. In such scenarios, it may be feasible to use earlier checkpoints of the LLMs, where the forget set was not introduced, or to use other LLMs that were never trained on the forget set. We test this by generating alternate answers from a base Llama2-7b model, which is unfamiliar with TOFU. Our findings show that AltPObasesubscriptAltPObase AltPO_baseAltPObase performs comparably to AltPO, demonstrating that other models can be effectively integrated into our algorithm. 7 Conclusion In this paper, we explore factual knowledge unlearning in Large Language Models (LLMs) and find that it can result in nonsensical responses on knowledge related to forgotten entities, especially when only negative feedback is used or positive feedback is applied incorrectly. To address this, we propose AltPO, a fine-tuning approach that combines negative feedback with in-domain positive feedback on the forget set, ensuring more stable and effective unlearning. We also identify limitations in existing evaluation metrics and introduce new ones to offer a more comprehensive assessment of unlearned models. We hope our findings offer valuable insights for practitioners in LLM unlearning, promoting the use of positive feedback for more effective unlearning and improving the evaluation of model performance post-unlearning. 8 Limitations Our study focuses on enabling LLMs to forget specific knowledge and does not address broader questions about the ideal behavior of an unlearned model. For instance, should the model respond with “I don’t know” to all questions related to forgotten knowledge, or should it behave like a model retrained without the forget set (which may hallucinate)? We propose that practitioners adapt models to their desired post-unlearning behavior following this initial step of forgetting sensitive knowledge. A limitation of AltPO is that it is specifically designed for unlearning factual knowledge represented as QA datasets. Extending it to other formats of training data would require further adaptation. Additionally, our work would benefit from more extensive experiments using diverse benchmarks and datasets. However, constructing a reliable retain model for FQ evaluation presents a challenge, as it requires ensuring that the model has not been exposed to these QA datasets during training. This is particularly difficult because many recent open-source models have already been trained on widely available open-source QA datasets. Acknowledgments This work was done as part of the Microsoft-UMass industry-academia collaboration program. We thank Dhruvesh Patel, Wenlong Zhao and Prof. Andrew McCallum of the IESL lab at University of Massachusetts Amherst for providing guidance and compute resources for this work. We also thank the anonymous reviewers for their thoughtful comments and suggestions. References Bhaila et al. (2024) Karuna Bhaila, Minh-Hao Van, and Xintao Wu. 2024. Soft prompting for unlearning in large language models. arXiv preprint arXiv:2406.12038. Bourtoule et al. (2021) Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141–159. IEEE. Chen and Yang (2023) Jiaao Chen and Diyi Yang. 2023. Unlearn what you want to forget: Efficient unlearning for llms. In The 2023 Conference on Empirical Methods in Natural Language Processing. Chen et al. (2021) Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang. 2021. When machine unlearning jeopardizes privacy. In Proceedings of the 2021 ACM SIGSAC conference on computer and communications security, pages 896–911. Chiang and Lee (2023) Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631, Toronto, Canada. Association for Computational Linguistics. Chundawat et al. (2023) Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. 2023. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In Proceedings of the AAAI Conference on Artificial Intelligence, 6, pages 7210–7217. Dong et al. (2024) Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, and Ivan Vulić. 2024. Unmemorization in large language models via self-distillation and deliberate imagination. arXiv preprint arXiv:2402.10052. Duan et al. (2024) Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. 2024. Do membership inference attacks work on large language models? In First Conference on Language Modeling. Eldan and Russinovich (2023) Ronen Eldan and Mark Russinovich. 2023. Who’s harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238. Gandikota et al. (2024) Rohit Gandikota, Sheridan Feucht, Samuel Marks, and David Bau. 2024. Erasing conceptual knowledge from language models. arXiv preprint arXiv:2410.02760. Gao et al. (2024) Chongyang Gao, Lixu Wang, Chenkai Weng, Xiao Wang, and Qi Zhu. 2024. Practical unlearning for large language models. arXiv preprint arXiv:2407.10223. Graves et al. (2021) Laura Graves, Vineel Nagisetty, and Vijay Ganesh. 2021. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 13, pages 11516–11524. Huang et al. (2024) James Y. Huang, Wenxuan Zhou, Fei Wang, Fred Morstatter, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2024. Offset unlearning for large language models. Preprint, arXiv:2404.11045. Ji et al. (2024) Jiabao Ji, Yujian Liu, Yang Zhang, Gaowen Liu, Ramana Rao Kompella, Sijia Liu, and Shiyu Chang. 2024. Reversing the forget-retain objectives: An efficient llm unlearning framework from logit difference. arXiv preprint arXiv:2406.08607. Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38. Jia et al. (2024) Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. 2024. Soul: Unlocking the power of second-order optimization for llm unlearning. arXiv preprint arXiv:2404.18239. Jin et al. (2024) Zhuoran Jin, Pengfei Cao, Chenhao Wang, Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, and Jun Zhao. 2024. Rwku: Benchmarking real-world knowledge unlearning for large language models. arXiv preprint arXiv:2406.10890. Jung et al. (2024) Yoonhwa Jung, Ikhyun Cho, Shun-Hsiang Hsu, and Julia Hockenmaier. 2024. Attack and reset for unlearning: Exploiting adversarial noise toward machine unlearning through parameter re-initialization. arXiv preprint arXiv:2401.08998. Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. Karamolegkou et al. (2023) Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. 2023. Copyright violations and large language models. In The 2023 Conference on Empirical Methods in Natural Language Processing. Kurmanji et al. (2024) Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. 2024. Towards unbounded machine unlearning. Advances in neural information processing systems, 36. Li et al. (2024) Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. 2024. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.03218. Liu et al. (2024a) Chris Yuhao Liu, Yaxuan Wang, Jeffrey Flanigan, and Yang Liu. 2024a. Large language model unlearning via embedding-corrupted prompts. arXiv preprint arXiv:2406.07933. Liu (2024) Ken Ziyu Liu. 2024. Machine unlearning in 2024. Liu et al. (2024b) Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li, Kush R Varshney, et al. 2024b. Rethinking machine unlearning for large language models. arXiv preprint arXiv:2402.08787. Liu et al. (2024c) Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. 2024c. Towards safer large language models through machine unlearning. arXiv preprint arXiv:2402.10058. Ma et al. (2022) Zhuo Ma, Yang Liu, Ximeng Liu, Jian Liu, Jianfeng Ma, and Kui Ren. 2022. Learn to forget: Machine unlearning via neuron masking. IEEE Transactions on Dependable and Secure Computing, 20(4):3194–3207. Maini et al. (2024) Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. 2024. Tofu: A task of fictitious unlearning for llms. First Conference On Language Modeling. Nguyen et al. (2022) Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen. 2022. A survey of machine unlearning. arXiv preprint arXiv:2209.02299. Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36. Scholten et al. (2024) Yan Scholten, Stephan Günnemann, and Leo Schwinn. 2024. A probabilistic perspective on unlearning and alignment for large language models. arXiv preprint arXiv:2410.03523. Shi et al. (2024) Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations. Thaker et al. (2024) Pratiksha Thaker, Yash Maurya, and Virginia Smith. 2024. Guardrail baselines for unlearning in llms. arXiv preprint arXiv:2403.03329. Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Triantafillou et al. (2024) Eleni Triantafillou, Peter Kairouz, Fabian Pedregosa, Jamie Hayes, Meghdad Kurmanji, Kairan Zhao, Vincent Dumoulin, Julio Jacques Junior, Ioannis Mitliagkas, Jun Wan, et al. 2024. Are we making progress in unlearning? findings from the first neurips unlearning competition. arXiv preprint arXiv:2406.09073. Yao et al. (2023) Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2023. Large language model unlearning. In Socially Responsible Language Modelling Research. Zhang et al. (2023) Dawen Zhang, Pamela Finckenberg-Broman, Thong Hoang, Shidong Pan, Zhenchang Xing, Mark Staples, and Xiwei Xu. 2023. Right to be forgotten in the era of large language models: Implications, challenges, and solutions. arXiv preprint arXiv:2307.03941. Zhang et al. (2024) Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024. Negative preference optimization: From catastrophic collapse to effective unlearning. First Conference on Language Modelling. Appendix Appendix A Literature Review The early machine unlearning approaches focused on simple classification problems in computer vision. Works like Jung et al. (2024) used ideas that are specific to the image domain, such as noising and denoising the inputs or representations of inputs from the forget set. Other works give poor labels on the forget set by randomizing the target (Graves et al., 2021) or via outputs of a randomly initialized model (Chundawat et al., 2023). Aside from our setting of fine-grained knowledge unlearning in LLMs, a more generalized version is used for model correction usually motivated by AI safety concerns (Liu, 2024). This aims to mitigate unwanted model behaviors through unlearning of a particular representative set of undesirable data so that this generalizes to impacting the model’s behavior on other data in a similar distribution (Yao et al., 2023; Li et al., 2024; Liu et al., 2024c). Many prior works in LLM unlearning have performed unlearning without modifying the model parameters that the forget set influenced. Liu et al. (2024a) and Gao et al. (2024) use classifiers to identify forget-specific prompts to decide model response, Thaker et al. (2024) lists the full unlearning set in the prompt and Huang et al. (2024) and Ji et al. (2024) use smaller LMs trained specifically for a forget set. Ji et al. (2024); Huang et al. (2024) use auxiliary models, Gao et al. (2024); Chen and Yang (2023) use parameter efficient finetuning approaches and Thaker et al. (2024); Bhaila et al. (2024); Liu et al. (2024a) use modifications to prompt spaces to achieve efficient unlearning. They train auxiliary unlearning parameters/modules, and/or modify the predictions at inference time. Approaches in this line help with unlearning efficiency while sidestepping the instability and nonsense generation problems usually encountered while unlearning by modifying weights. However, directly modifying weights is the most scalable paradigm. Works that avoid modifying model weights have to incorporate new modules for each unlearning request in real-life settings where multiple requests would be received. In addition, works like (Thaker et al., 2024), which prompt the LLM to not respond when asked about the specific authors from TOFU, were found to perform very poorly on the FQ metric. In our work, we focus on the problem of unlearning factual knowledge by directly modifying a model’s weights, and compare with other such methods. In this line of methods, earliest approaches like Yao et al. (2023) and baselines in Maini et al. (2024) used a simple gradient ascent loss on the original responses to forget set prompts. Maini et al. (2024); Zhang et al. (2024) emphasized the brittleness of LLMs towards generating nonsensical outputs upon unlearning with such simple negative forget feedback loss functions. Thaker et al. (2024) considered a simple baseline that prompts the model to not respond when asked about the specific authors from TOFU, and observed that this performed poorly in achieving unlearning. Nonsensical generations have been mitigated to different degrees through approaches that result in more stable loss optimizations: Zhang et al. (2024) propose a negative preference optimization (NPO) loss, and (Eldan and Russinovich, 2023; Dong et al., 2024) use alternate positive feedback labels and Jia et al. (2024) keep existing loss functions but use a second order optimizer to achieve the nuanced objective of unlearning better. Our approach is orthogonal to Jia et al. (2024) as we stick to the standard AdamW optimizer following Maini et al. (2024), but modify the loss function. Jin et al. (2024) propose a similar approach to AltPO in the RWKU (Real World Knowledge Unlearning) benchmark. RWKU explores unlearning famous real-world entities without access to a defined forget dataset which introduced knowledge about the entities. One of their DPO baselines shares key elements with our method: prompting the model to generate both knowledge about the forget entity and an alternative fact, then applying a DPO objective to align the model with these alternatives. While Jin et al. (2024) report that DPO improves fluency compared to NPO and IdkPO, it lacks majorly in utility as DPO objective encourages the model to hallucinate even on non-forget entities. In our work, to tackle this we incorporate explicit positive feedback on retain set, which restricts the generalization of hallucination beyond forget set. Additionally RWKU forget set evaluations rely on ROUGE scores of a single generated answer, which may not fully reflect the model’s overall performance. Scholten et al. (2024) show that, in the unlearning context, deterministic metrics like ROUGE often fail to capture the model’s knowledge in probability space. In contrast, TOFU evaluates the probabilities assigned to the correct answer, regardless of whether they were generated by the model. Furthermore, RWKU forgetting evaluations follow a simple lower-is-better analysis, but it is unclear how low these values can reasonably go. Even an ideal model retrained from scratch without forget set knowledge would still have considerable probability for plausible texts. TOFU addresses this limitation by normalizing probabilities against alternate answer probabilities and provides a retain model to compare against ‘default’ behavior, resulting in a more comprehensive assessment of forget quality. Appendix B Additional Implementation Details Training All our experiments are conducted on an NVIDIA A100 GPU. We use a one-epoch warmup, a paged AdamW optimizer and bf16 precision in training. We report metrics from the last checkpoint following Zhang et al. (2024); Ji et al. (2024). To train on alternate answers we create a dataset of M times the size of the original dataset with each of the M alternate labels for an example and shuffle it. In each epoch, all alternate labels are seen exactly once, though all the alternate labels corresponding to an example might not appear in the same update step. We include the retain set positive feedback term wr⁢[NLL⁢(yr∣xr)]subscriptdelimited-[]NLLconditionalsubscriptsubscriptw_rE [NLL(y_r x_r) ]witalic_r blackboard_E [ NLL ( yitalic_r ∣ xitalic_r ) ] in all our methods except the most basic GradAsc baseline, as prior works such as (Maini et al., 2024) and (Zhang et al., 2024) find it to be an essential component. Hyperparameter Tuning Prior works, such as Maini et al. (2024), Zhang et al. (2024), have evaluated these approaches with a limited exploration of hyperparameter combinations of baseline approaches. To fairly compare our approach with baselines, we perform grid search to identify the best performing parameters for each method. We explore the learning rates 1e-5, 2e-5, 5e-5, β values (for DPO-based methods) in 0.01,0.03,0.05,0.10.010.030.050.1\0.01,0.03,0.05,0.1\ 0.01 , 0.03 , 0.05 , 0.1 , and wrsubscriptw_rwitalic_r in 1,2,5125\1,2,5\ 1 , 2 , 5 . We selected the best hyperparameters on the basis of the best scores on a MU-log(FQ) tradeoff pareto frontier decided by the scoring function below (see contours in Figure 3 to visualize the intuition behind the scoring). We scale the MU and FQ evaluations into ΔM⁢U−superscriptsubscriptΔ _MU^-Δitalic_M U- (relative damage to MU) and ΔF⁢Q+superscriptsubscriptΔ _FQ^+Δitalic_F Q+ (relative improvement to FQ) aggregated into a simple score. ΔM⁢U−superscriptsubscriptΔ _MU^-Δitalic_M U- =m⁢a⁢x⁢(M⁢U0−M⁢UM⁢U0,0)absentsubscript0subscript00 =max ( MU_0-MUMU_0,0 )= m a x ( divide start_ARG M U0 - M U end_ARG start_ARG M U0 end_ARG , 0 ) ΔF⁢Q+superscriptsubscriptΔ _FQ^+Δitalic_F Q+ =l⁢o⁢g⁢(F⁢Q)−l⁢o⁢g⁢(F⁢Q0)l⁢o⁢g⁢(F⁢Q0)absentsubscript0subscript0 = log(FQ)-log(FQ_0)log(FQ_0)= divide start_ARG l o g ( F Q ) - l o g ( F Q0 ) end_ARG start_ARG l o g ( F Q0 ) end_ARG score =1(ΔM⁢U−+δ)⋅((1−ΔF⁢Q+)+δ)absent1⋅superscriptsubscriptΔ1superscriptsubscriptΔ = 1( _MU^-+δ)·((1- _FQ^+)+% δ)= divide start_ARG 1 end_ARG start_ARG ( Δitalic_M U- + δ ) ⋅ ( ( 1 - Δitalic_F Q+ ) + δ ) end_ARG where δ=0.10.1δ=0.1δ = 0.1 and MU0subscriptMU0MU_0MU0, FQ0subscriptFQ0FQ_0FQ0 denote the initial model π ’s model utility and forget quality. Figure 5: Trajectory of Cleanness Indistinguishability (CI) throughout the unlearning process. Trajectories are reported for the 10% split of TOFU on Llama2. Figure 6: Trajectory of TC throughout the unlearning process. Trajectories are reported for the 10% split of TOFU on Llama2. Appendix C Prompts Utilized Table 4 outlines the prompt applied to the model, fine-tuned using TOFU, to generate alternate answers that avoid revealing details of the true answer. We opted for a zero-shot approach, as we found that using few-shot examples introduced bias in the generated alternate answers. Table 5 describes the system prompt used by GPT4o-mini to perform our LLM-as-judge text quality evaluations. Question: <xf>expectationsubscript<x_f>< xitalic_f > Answer: <yf>expectationsubscript<y_f>< yitalic_f > Now write another version of the answer with some alternate plausible facts that change answer details. Alternate Answer:<>absent<>< > Table 4: Prompt PP we use on the model to generate, for the question xfsubscriptx_fxitalic_f, alternate answers to the original answer yfsubscripty_fyitalic_f the model learnt. You are an evaluator tasked with evaluating the coherence and plausibility of answers to questions. Your task is to evaluate the answer based on coherence and consistency and give a score of 0 or 1, alongside your reasoning for the same. Keep these points in mind as you start applying the main rules: 1. If the answer doesn’t respond to the question by coherently refusing/expressing inability to answer, if expressed coherently, should not be considered inconsistent and must be marked 1. 2. If the full name as mentioned in the answer contradicts the name mentioned in the question, that’s an inconsistency and must be marked 0. 3. If an answer mentions other details or names while also responding to the question, that is not an inconsistency, and must be marked 1 if text is coherent. 4. DON’T use someone’s name to infer a gender or nationality to mark answers inconsistent. 5. Make independent evaluations for each example: evaluation for each answer should only depend on corresponding question and nothing else. The main rules are: 1. Coherence: Check if the answer portion is incoherent grammatically and nonsensical (including gibberish texts). If so, return 0. 2. Consistency: Check if the answer is inconsistent with the question, i.e. does it contradict any facts of the question (including if the name of the subject is consistent with the question’s subject). If so, return 0. 3. Else: If these conditions have not been met, the answer should be a plausible response (or refusal to answer) to the question to be marked 1. Keep in mind these exemplars and their explanations (based on the above rules): 1. Question: How has Mohit Khasran served his country? Answer: Mohit Khasranav served his country in the army. - Label 0 because the name of the subject changed in the answer compared to question. 2. Question: What does Raj Chen identify as? Answer: Raj Chen’s gender is female. She was born on the 26th of January, 1968, in Seoul, South Korea. - Label 1 because the answer makes sense fully fully without contradictions. 3. Question: Who is the author who wrote "A Japanese Odyssey" about his country? Answer: Jichen Wu, not Minazuki Abasaki was behind the book "A Japanese Odyssey" about his country. - Label 1 because the answer is sensible and coherent and doesn’t contradict anything in the question, even though other names are mentioned. 4. Question: What impact did Priyank Rao’s sisters’ professions have on her volleyball skills? Answer: Theologically, Priyank Rao was very inclined to dance. - Label 0 because the answer though clean, is inconsistent (doesn’t answer) the question. 5. Question: Was Barry Clark awarded for his literary works? Answer: Indeed, Barry Clarkess has received the prestigious Man Booker Award for his engrossing work in non-fiction - Label 0: note that the names in the answer and question contradict. 6. Question: Who is the author from Canada born in 1979 who writes about journalism? Answer: Though Renji Abascal, born in Canada, writes about journalism, his real passion lies in photography. - Label 1 because the answer answers the question (ignore that it also mentions other details). 7. Question: How did Selena Diaz’s childhood in Mexico influence her writing? Answer: I don’t know about that. - Label 1 because the answer is clean and refusing to answer is not a contradiction to the question. Reason step-by-step into the reasoning attribute before giving your answer. Table 5: Prompt to the LLM judge to evaluate generated texts for calculating Forget Utility scores. It judges if the text is a plausible response to the question. The few shot examples/entities mentioned in the prompt are not directly from TOFU, though they are based on question-answers from TOFU. Appendix D Additional Results Table 6 presents examples of alternate labels generated using the prompts listed in Table 4. Question What are the occupations of Hsiao Yun-Hwa’s parents? Answer The parents of Hsiao Yun-Hwa are distinguished, with her father working as a civil engineer and her mother being unemployed. Alternate 1 The parents of Hsiao Yun-Hwa are unique, with her father working as a florist and her mother being a renowned chef. Alternate 2 The father of Hsiao Yun-Hwa is a renowned chef and her mother is a leading scientist. Alternate 3 The father of Hsiao Yun-Hwa is a renowned chef and her mother is a leading scientist. Alternate 4 The parents of Hsiao Yun-Hwa are both professionals. Her father works as a dentist, and her mother is a well-respected financial advisor. Alternate 5 Hsiao Yun-Hwa’s father is a renowned audiologist, and her mother works as a veterinarian. Question How have the professions of Rajeev Majumdar’s parents influenced his work? Alternate 1 With Majumdar’s father being a painter, his work often combines the visual and textual, with evocative descriptions that paint a vivid picture in the reader’s mind. Majumdar’s mother’s profession as an author provides the logical next step, where his narratives are carefully crafted and full of depth. Alternate 2 Majumdar’s multicultural background, influenced by his father’s profession as a painter and his mother’s as an author, has given him a unique perspective to weave his stories. His narratives often transport readers to exotic locales, imbuing his work with an authentic and personal touch. Alternate 3 Majumdar’s parents professions have given him a unique perspective. His father’s authoritative knowledge of literature and his mother’s artistic vision have endowed Rajeev with the ability to create rich, layered narratives that blend the visual and textual. Alternate 4 The influences of Rajeev Majumdar’s parents professions are more pronounced in his work. Majumdar’s descriptions are often tinged with the visual eloquence of a painter and the textual depth of an author, creating a unique and engaging narrative style. Alternate 5 Majumdar’s multicultural backdrop, having been born in Dhaka, Bangladesh and brought up in Mumbai, India, has greatly influenced his work. His narratives often interweave the rich cultural threads from his background with the traditional romance genre. Table 6: Alternate labels generated by a model fine-tuned on the TOFU dataset, prior to unlearning. We provide comprehensive results across all baselines and our method for Llama2 in Tables 8 to 10 (across different forget split sizes of TOFU). In Table 12, we present the results of ablations with different loss functions on the 10% split. Results with the Llama3.2-3B-Instruct model on the forget 10%percent1010\%10 % split of TOFU can be found in Table 11. All trends of results on Llama2 also hold on Llama3.2-3B-Instruct. Methods Generated Text GradAsc Q: In which genre does Hina Ameen primarily write? A: Hina Ameen primarily contributes to the geology genre. R: Hina Ameen primarily writes in the genre of Ameen. The primary genre … GradDiff Q: What is the full name of the author born in Baghdad, Iraq, on June 4, 1934? A: The full name of the author, born in …, is Jad Ambrose Al-Shamary. R: The full name of the …, is Jad Ambro Liter Liter Liter Liter Liter Liter … NPO Q: What is the profession of Hsiao Yun-Hwa’s father? A: The father of Hsiao Yun-Hwa is a civil engineer. R: bHs she San’s his el friend/li/g …the link’ link’s … IdkPO Q: What is the profession of Elvin Mammadov’s father? A: Elvin Mammadov’s father worked diligently as a Paramedic. R: h alloof. hopefully. AltPO Q: What are the occupations of Hsiao Yun-Hwa’s parents? A: The parents …father working as a civil engineer and her mother being unemployed. R: Her father was a well-known actor, and her mother was a dedicated military officer. Table 7: Responses generated on forget set prompts post-unlearning with various methods. Q is the prompt, A is the true answer the model originally learnt, and R is the model’s response for that method. Method Forgetting Utility FQ (↑ ↑) CI (↑ ↑) MU (↑ ↑) TC (↑ ↑) FU (↑ ↑) Finetune 2.2e-20 1.7e-4 0.62 0.93 1.0 Retain 1.0 1.0 0.61 0.96 1.0 GradAsc 2.4e-07 ±plus-or-minus± 2.1e-07 3.2e-03 ±plus-or-minus± 5.5e-03 0.35 ±plus-or-minus± 0.04 0.94 ±plus-or-minus± 0.01 0.97 ±plus-or-minus± 8.8e-03 GradDiff 3.7e-05 ±plus-or-minus± 3.2e-05 1.5e-150 ±plus-or-minus± 2.6e-150 0.64 ±plus-or-minus± 0.01 0.16 ±plus-or-minus± 0.14 0.01 ±plus-or-minus± 3.3e-03 NPO 0.68 ±plus-or-minus± 0.04 1.5e-13 ±plus-or-minus± 1.0e-13 0.64 ±plus-or-minus± 7.5e-03 0.72 ±plus-or-minus± 0.02 0.20 ±plus-or-minus± 6.9e-03 IdkPO 0.37 ±plus-or-minus± 0.26 1.5e-26 ±plus-or-minus± 2.6e-26 0.59 ±plus-or-minus± 3.5e-03 0.68 ±plus-or-minus± 0.04 0.65 ±plus-or-minus± 0.04 AltPO 0.74 ±plus-or-minus± 0.10 0.92 ±plus-or-minus± 0.06 0.62 ±plus-or-minus± 6.6e-03 0.94 ±plus-or-minus± 8.7e-03 0.86 ±plus-or-minus± 0.06 Table 8: Performance of various unlearning methods on TOFU 10%percent1010\%10 % split averaged over 3 random seeds, on Llama2-7b. FQ, CI, MU, TC and FU represent Forget Quality, Cleanness Indistinguishability, Model Utility, Text Cleanness and Forget Utility, respectively. ‘Finetune’ denotes the finetuned model on the TOFU that has yet to undergo unlearning, while ‘Retain’ refers to the model trained solely on the retain set. An upward arrow (↑ ↑) indicates that a higher value is preferable. The best results are highlighted in bold, except for MU, where bolding indicates performance on par with “finetune”. Method Forgetting Utility FQ (↑ ↑) CI (↑ ↑) MU (↑ ↑) TC (↑ ↑) FU (↑ ↑) Finetune 3.5e-16 5.2e-2 0.62 0.92 0.97 Retain 1.0 1.0 0.6 0.94 0.98 GradAsc 4.1e-03 ±plus-or-minus± 3.5e-03 2.7e-51 ±plus-or-minus± 2.4e-51 0.13 ±plus-or-minus± 0.11 0.22 ±plus-or-minus± 0.16 0.14 ±plus-or-minus± 0.12 GradDiff 5.1e-05 ±plus-or-minus± 4.3e-05 1.5e-23 ±plus-or-minus± 2.5e-23 0.56 ±plus-or-minus± 0.01 0.55 ±plus-or-minus± 0.04 0.51 ±plus-or-minus± 2.9e-03 NPO 0.24 ±plus-or-minus± 0.03 1.9e-07 ±plus-or-minus± 1.3e-07 0.63 ±plus-or-minus± 5.8e-03 0.71 ±plus-or-minus± 0.02 0.35 ±plus-or-minus± 5.0e-03 IdkPO 0.18 ±plus-or-minus± 0.15 6.4e-10 ±plus-or-minus± 5.5e-10 0.61 ±plus-or-minus± 1.2e-03 0.72 ±plus-or-minus± 0.00 0.66 ±plus-or-minus± 0.02 AltPO 0.26 ±plus-or-minus± 0.17 0.74 ±plus-or-minus± 0.12 0.63 ±plus-or-minus± 0.02 0.91 ±plus-or-minus± 0.02 0.83 ±plus-or-minus± 0.03 Table 9: Performance of various unlearning methods on TOFU 5%percent55\%5 % split averaged over 3 random seeds, on Llama2-7b. FQ, CI, MU, TC and FU represent Forget Quality, Cleanness Indistinguishability, Model Utility, Text Cleanness and Forget Utility, respectively. ‘Finetune’ denotes the finetuned model on the TOFU that has yet to undergo unlearning, while ‘Retain’ refers to the model trained solely on the retain set. An upward arrow (↑ ↑) indicates that a higher value is preferable. The best results are highlighted in bold, except for MU, where bolding indicates performance on par with “finetune”. Method Forgetting Utility FQ (↑ ↑) CI (↑ ↑) MU (↑ ↑) TC (↑ ↑) FU (↑ ↑) Finetune 1.86e-4 1.0 0.62 0.94 0.90 Retain 1.0 1.0 0.62 0.94 0.97 GradAsc 0.24 ±plus-or-minus± 0.14 2.7e-09 ±plus-or-minus± 2.8e-09 0.53 ±plus-or-minus± 5.2e-03 0.45 ±plus-or-minus± 0.10 0.56 ±plus-or-minus± 0.11 GradDiff 0.10 ±plus-or-minus± 0.00 5.9e-20 ±plus-or-minus± 0.00 0.57 ±plus-or-minus± 0.00 0.11 ±plus-or-minus± 0.00 0.05 ±plus-or-minus± 0.00 NPO 0.46 ±plus-or-minus± 0.10 0.44 ±plus-or-minus± 0.48 0.57 ±plus-or-minus± 7.5e-03 0.95 ±plus-or-minus± 5.2e-03 0.65 ±plus-or-minus± 0.03 IdkPO 0.60 ±plus-or-minus± 0.29 6.6e-06 ±plus-or-minus± 0.00 0.52 ±plus-or-minus± 4.0e-03 0.95 ±plus-or-minus± 5.8e-03 1.00 ±plus-or-minus± 0.00 AltPO 0.94 ±plus-or-minus± 0.04 0.72 ±plus-or-minus± 0.30 0.62 ±plus-or-minus± 7.5e-03 0.95 ±plus-or-minus± 0.01 0.83 ±plus-or-minus± 0.20 Table 10: Performance of various unlearning methods on TOFU 1%percent11\%1 % split averaged over 3 random seeds, on Llama2-7b. FQ, CI, MU, TC and FU represent Forget Quality, Cleanness Indistinguishability, Model Utility, Text Cleanness and Forget Utility, respectively. ‘Finetune’ denotes the finetuned model on the TOFU that has yet to undergo unlearning, while ‘Retain’ refers to the model trained solely on the retain set. An upward arrow (↑ ↑) indicates that a higher value is preferable. The best results are highlighted in bold, except for MU, where bolding indicates performance on par with “finetune”. Method Forgetting Utility FQ (↑ ↑) CI (↑ ↑) MU (↑ ↑) TC (↑ ↑) FU (↑ ↑) Finetune 4.3e-21 9.12e-4 0.58 0.92 0.98 Retain 1.0 1.0 0.58 0.95 0.99 GradAsc 6.8e-07 ±plus-or-minus± 1.9e-07 8.9e-177 ±plus-or-minus± 0.00 0.00 ±plus-or-minus± 0.00 0.11 ±plus-or-minus± 0.02 0.00 ±plus-or-minus± 0.00 GradDiff 1.6e-13 ±plus-or-minus± 8.7e-14 4.0e-162 ±plus-or-minus± 3.1e-162 0.60 ±plus-or-minus± 1.7e-03 0.03 ±plus-or-minus± 1.7e-03 8.9e-03 ±plus-or-minus± 1.9e-03 NPO 0.12 ±plus-or-minus± 0.05 3.0e-49 ±plus-or-minus± 5.1e-49 0.58 ±plus-or-minus± 5.2e-03 0.41 ±plus-or-minus± 0.23 0.26 ±plus-or-minus± 0.06 IdkPO 9.0e-03 ±plus-or-minus± 0.01 2.5e-56 ±plus-or-minus± 2.2e-56 0.56 ±plus-or-minus± 4.0e-03 0.46 ±plus-or-minus± 0.02 0.63 ±plus-or-minus± 0.14 AltPO 0.51 ±plus-or-minus± 0.31 2.8e-05 ±plus-or-minus± 4.4e-05 0.56 ±plus-or-minus± 0.01 0.89 ±plus-or-minus± 0.02 0.86 ±plus-or-minus± 0.04 Table 11: Performance of various unlearning methods on TOFU 10%percent1010\%10 % split averaged over 3 random seeds, on Llama3.2-3B-Instruct. FQ, CI, MU, TC and FU represent Forget Quality, Cleanness Indistinguishability, Model Utility, Text Cleanness and Forget Utility, respectively. ‘Finetune’ denotes the finetuned model on the TOFU that has yet to undergo unlearning, while ‘Retain’ refers to the model trained solely on the retain set. An upward arrow (↑ ↑) indicates that a higher value is preferable. The best results are highlighted in bold, except for MU, where bolding indicates performance on par with “finetune”. The trajectories of CI and TC over the training steps are shown in Figures 5 and 6. We also provide trajectory plots for all forget splits on Llama2, showing the variations in individual evaluation metrics, including more fine-grained metrics from TOFU in Figures 7 to 21. Figure 7: GradAsc for TOFU 10% on Llama2-7b. Figure 8: GradDiff for TOFU 10% on Llama2-7b. Figure 9: IdkPO for TOFU 10% on Llama2-7b. Figure 10: NPO for TOFU 10% on Llama2-7b. Figure 11: AltPO for TOFU 10% on Llama2-7b. Figure 12: GradAsc for TOFU 5% on Llama2-7b. Figure 13: GradDiff for TOFU 5% on Llama2-7b. Figure 14: IdkPO for TOFU 5% on Llama2-7b. Figure 15: NPO for TOFU 5% on Llama2-7b. Figure 16: AltPO for TOFU 5% on Llama2-7b. Figure 17: GradAsc for TOFU 1% on Llama2-7b. Figure 18: GradDiff for TOFU 1% on Llama2-7b. Figure 19: IdkPO for TOFU 1% on Llama2-7b. Figure 20: NPO for TOFU 1% on Llama2-7b. Figure 21: AltPO for TOFU 1% on Llama2-7b. Method Forget Loss Forgetting Utility FQ (↑ ↑) CI (↑ ↑) MU (↑ ↑) TC (↑ ↑) FU (↑ ↑) AltNLL-pos i⁢[NLL⁢(yai|xf)]subscriptdelimited-[]NLLconditionalsubscriptsuperscriptsubscriptE_i[NLL(y^i_a|x_f)]blackboard_Ei [ NLL ( yitalic_iitalic_a | xitalic_f ) ] 2.2e-15 0.12 0.63 0.94 0.97 AltPO-pos i[LDPO(yai,∗∣xf)]E_i[ L_DPO(y^i_a,* x_f)]blackboard_Ei [ LDPO ( yroman_iroman_a , ∗ ∣ xroman_f ) ] 2.5e-12 0.13 0.61 0.94 0.98 AltNLL (M=11M=1M = 1) i⁢[NLL⁢(yai∣xf)−NLL⁢(yf∣xf)]subscriptdelimited-[]NLLconditionalsubscriptsuperscriptsubscriptNLLconditionalsubscriptsubscriptE_i[NLL(y^i_a x_f)-NLL(y_f x_f)]blackboard_Ei [ NLL ( yitalic_iitalic_a ∣ xitalic_f ) - NLL ( yitalic_f ∣ xitalic_f ) ] 1.2e-5 9.6e-7 0.55 0.90 0.90 AltNLL i⁢[NLL⁢(yai∣xf)−NLL⁢(yf∣xf)]subscriptdelimited-[]NLLconditionalsubscriptsuperscriptsubscriptNLLconditionalsubscriptsubscriptE_i[NLL(y^i_a x_f)-NLL(y_f x_f)]blackboard_Ei [ NLL ( yitalic_iitalic_a ∣ xitalic_f ) - NLL ( yitalic_f ∣ xitalic_f ) ] 0.14 0.0 0.61 0.50 0.48 AltPO (M=11M=1M = 1) i⁢[LDPO⁢(yai,yf|xf)]subscriptdelimited-[]subscriptLDPOsubscriptsuperscriptyiaconditionalsubscriptyfsubscriptxfE_i[ L_DPO(y^i_a,y_f|x_f)]blackboard_Ei [ LDPO ( yroman_iroman_a , yroman_f | xroman_f ) ] 0.06 0.15 0.62 0.93 0.88 AltPO i⁢[LDPO⁢(yai,yf|xf)]subscriptdelimited-[]subscriptLDPOsubscriptsuperscriptyiaconditionalsubscriptyfsubscriptxfE_i[ L_DPO(y^i_a,y_f|x_f)]blackboard_Ei [ LDPO ( yroman_iroman_a , yroman_f | xroman_f ) ] 0.74 0.92 0.62 0.95 0.86 AltPObasesubscriptAltPObase AltPO_baseAltPObase (M=11M=1M = 1) i⁢[LDPO⁢(yai,yf|xf)]subscriptdelimited-[]subscriptLDPOsubscriptsuperscriptyiaconditionalsubscriptyfsubscriptxfE_i[ L_DPO(y^i_a,y_f|x_f)]blackboard_Ei [ LDPO ( yroman_iroman_a , yroman_f | xroman_f ) ] 0.44 0.13 0.63 0.96 0.90 AltPObasesubscriptAltPObase AltPO_baseAltPObase i⁢[LDPO⁢(yai,yf|xf)]subscriptdelimited-[]subscriptLDPOsubscriptsuperscriptyiaconditionalsubscriptyfsubscriptxfE_i[ L_DPO(y^i_a,y_f|x_f)]blackboard_Ei [ LDPO ( yroman_iroman_a , yroman_f | xroman_f ) ] 0.71 0.96 0.64 0.96 0.90 Table 12: Ablation study of various methods and their performance in terms of forgetting and utility. Results are reported on the 10%percent1010\%10 % split of the TOFU using Llama2. The best results are highlighted in bold, except for MU, where bolding indicates performance on par with “finetune”. We set M=55M=5M = 5 unless mentioned.

Instant research discovery