Paper deep dive

Unlearning in- vs. out-of-distribution data in LLMs under gradient-based method

Teodora Baluta, Pascal Lamblin, Daniel Tarlow, Fabian Pedregosa, Gintare Karolina Dziugaite

Year: 2024Venue: Safe Generative AI Workshop @ NeurIPS 2024Area: Model EditingType: EmpiricalEmbeddings: 95

Models: T5-base

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/12/2026, 6:25:53 PM

Summary

This paper investigates machine unlearning in Large Language Models (LLMs) using gradient-based methods. It formalizes a 'Generalized Exposure' metric to evaluate unlearning quality and compares the performance trade-offs between in-distribution and out-of-distribution data removal. The authors find that while out-of-distribution unlearning is more efficient with better performance retention, in-distribution unlearning leads to rapid performance decay, highlighting the need for improved unlearning strategies.

Entities (6)

Large Language Models · technology · 100%Machine Unlearning · research-field · 100%Generalized Exposure · metric · 98%Gradient Ascent · method · 95%In-distribution data · data-property · 90%Out-of-distribution data · data-property · 90%

Relation Signals (3)

Generalized Exposure → evaluates → Machine Unlearning

confidence 98% · We propose a new metric for evaluating unlearning in generative models

Gradient Ascent → appliedto → Machine Unlearning

confidence 95% · Efficient alternatives in LLMs often rely on gradient ascent-based procedures

In-distribution data → causesperformancedecayin → Large Language Models

confidence 90% · For in-distribution examples, however, we observe a rapid decay in performance as unlearning progresses.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Machine unlearning aims to solve the problem of removing the influence of selected training examples from a learned model. Despite the increasing attention to this problem, it remains an open research question how to evaluate unlearning in large language models (LLMs), and what are the critical properties of the data to be unlearned that affect the quality and efficiency of unlearning. This work formalizes a metric to evaluate unlearning quality in generative models, and uses it to assess the trade-offs between unlearning quality and performance. We demonstrate that unlearning out-of-distribution examples requires more unlearning steps but overall presents a better trade-off overall. For in-distribution examples, however, we observe a rapid decay in performance as unlearning progresses. We further evaluate how example's memorization and difficulty affect unlearning under a classical gradient ascent-based approach.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

94,820 characters extracted from source content.

Expand or collapse full text

Unlearning in- vs. out-of-distribution data in LLMs under gradient-based methods Teodora Baluta teobaluta@gatech.edu &Pascal Lamblin lamblinp@google.com &Daniel Tarlow 22footnotemark: 2 dtarlow@google.com &Fabian Pedregosa 22footnotemark: 2 pedregosa@google.com &Gintare Karolina Dziugaite 22footnotemark: 2 gkd@google.com School of Cybersecurity and Privacy, Georgia Institute of Technology. Work was started when the author was an intern at Google Brain, and affiliated with the National University of Singapore.Google DeepMind Abstract Machine unlearning aims to solve the problem of removing the influence of selected training examples from a learned model. Despite the increasing attention to this problem, it remains an open research question how to evaluate unlearning in large language models (LLMs), and what are the critical properties of the data to be unlearned that affect the quality and efficiency of unlearning. This work formalizes a metric to evaluate unlearning quality in generative models, and uses it to assess the trade-offs between unlearning quality and performance. We demonstrate that unlearning out-of-distribution examples requires more unlearning steps but overall presents a better trade-off overall. For in-distribution examples, however, we observe a rapid decay in performance as unlearning progresses. We further evaluate how example’s memorization and difficulty affect unlearning under a classical gradient ascent-based approach. 1 Introduction Training large language models (LLMs) often involves complex data pipelines. These pipelines handle large quantities of data, some of which might be sensitive. Recently, it has been shown that LLMs are susceptible to sentence-level membership inference attacks (Gu et al., 2023) and reconstruction attacks (Carlini et al., 2019), meaning that one may be able to infer which data was part of the training set, or in some cases, even reconstruct partial inputs by interrogating the model. As a result, this raises a prevalent problem of data removal from a trained LLM. To this end, there has been growing interest in formalizing technical definitions of machine unlearning and designing machine unlearning techniques and evaluation metrics (Triantafillou et al., 2023, 2024). The goal of machine unlearning is to remove the influence of a subset of the original training data, the forget set, from a corresponding model. A naïve way to achieve it is to retrain the model from scratch on an updated training set (the retain set), that does not include the forget set. This approach is resource-intensive, and does not scale to the large models now in development. Efficient alternatives in LLMs often rely on gradient ascent-based procedures, where one maximizes some loss on the data to be forgotten to reduce the influence of this data on the model predictions (Jang et al., 2022). However, there are a few issues that arise with this approach: (1) inherently, gradient ascent-based unlearning does not come with guarantees, and one needs a way to empirically evaluate the unlearning quality; (2) such unlearning methods do not only affect the forget set examples, but also come at a performance cost on the rest of the data. Our work touches upon both of these issues. For the first issue, we propose two metrics for evaluating unlearning quality. The first metric, named generalized exposure, lower bounds unlearning quality under a particular unlearning definition (Triantafillou et al., 2023), but requires access to a reference model, that had never seen the forget set, to compute likelihoods. Another metric, relative exposure, is an approximation to the first, further estimating the likelihoods that would be computed by a reference model, only using the current model pre- and post-unlearning. For the second issue, we present an extensive empirical evaluation, on LLMs, of how unlearning via gradient ascent differs for in- versus out-of-distribution examples. We visualize the trade-offs between unlearning quality as measured per our definitions, and performance on the rest of the data. We capture different patterns of these trade-offs depending on the difficulty of the examples in the forget set, and depending on the degree of memorization of these examples. Our contributions can be summarized as following: • We propose a new metric for evaluating unlearning in generative models using a reference model that had never seen the unlearning data. Further, we propose an approximation to this metric that does not require having access to the reference model. • Using our proposed metrics, we evaluate gradient ascent-based unlearning in large language models, and observe that unlearning out-of-distribution samples can be done nearly without affecting the BLEU score Papineni et al. (2002) just like in the reference model. In contrast, unlearning in-distribution samples affects the performance, unlike in the reference model. This indicates a weakness in gradient ascent-based unlearning, and suggests that simultaneous gradient descent on the retain data might be necessary. • Finally, we evaluate whether measuring unlearning on a data point could be done using similar samples. We observe that similar examples in the training data are unlearned together with the ones on which unlearning is performed. Similar examples outside of the training dataset are almost not affected by this unlearning procedure. This explains why we observe performance degradation for in-distribution examples. 2 Preliminaries Let Θ Θ be a parameterized space of models (e.g., Θ=ℝdΘsuperscriptℝ =R^dΘ = blackboard_Rd in the case of neural networks with d parameters). For our purposes, we care only about the output distribution of learning algorithms. That is, if Z∗superscriptZ^*Z∗ denotes the set of finite sequences of input examples, and Δ⁢(Θ)ΔΘ ( )Δ ( Θ ) denotes the space of distributions on Θ Θ, a learning algorithm will be viewed as a map :Z∗→Δ⁢(Θ):→superscriptΔΘA:Z^*→ ( )A : Z∗ → Δ ( Θ ), and so running the algorithm on a size-n dataset S∈ZnsuperscriptS∈ Z^nS ∈ Zitalic_n produces the model θ∼⁢(S)similar-toθ (S)θ ∼ A ( S ). In the context of autoregressive sequence models, such as LLMs, the set of training data S∈Z∗superscriptS∈ Z^*S ∈ Z∗ consists of samples x∈Zx∈ Zx ∈ Z, each a sequence of tokens, x=(x1,…,xk)subscript1…subscriptx=(x_1,…,x_k)x = ( x1 , … , xitalic_k ). A model θ defines conditional distributions on the next token xisubscriptx_ixitalic_i given all previous tokens x1:i−1subscript:11x_1:i-1x1 : i - 1, denoted f⁢(xi|x1:i−1;θ)conditionalsubscriptsubscript:11f(x_i|x_1:i-1;θ)f ( xitalic_i | x1 : i - 1 ; θ ). For a fixed model θ, we consider its output on a given sequence of tokens x=(x1,…,xk)∈Zsubscript1…subscriptx=(x_1,…,x_k)∈ Zx = ( x1 , … , xitalic_k ) ∈ Z to be f⁢(x;θ)=∏i=1kf⁢(xi|x1:i−1;θ)superscriptsubscriptproduct1conditionalsubscriptsubscript:11f(x;θ)= _i=1^kf(x_i|x_1:i-1;θ)f ( x ; θ ) = ∏i = 1k f ( xitalic_i | x1 : i - 1 ; θ ), the probability it assigns to that sequence (in other words, the likelihood of x under the model θ). Let ℒ⁢(θ,x)=−log⁡(f⁢(x;θ))ℒL(θ,x)=- (f(x;θ))L ( θ , x ) = - log ( f ( x ; θ ) ) denote the negative log likelihood (NLL) of x. The training objective of language models we consider is based on that loss, averaged over x∈Sx∈ Sx ∈ S. It is minimized using gradient-based iterative algorithms. Although it is immaterial to this work, training can often be viewed as stochastic gradient descent (or some variant) applied to the objective θ↦ℒ⁢(θ,X)maps-toℒθ *EL(θ,X)θ ↦ blackboard_E L ( θ , X ), where the expectation is taken over X sampled from S. 2.1 Unlearning Given that a learning algorithm has produced a model θ∼⁢(S)similar-toθ (S)θ ∼ A ( S ), the goal of unlearning is to remove the influence of a subset F⊆SF SF ⊆ S of the training data. We call F the forget set, S∖FS FS ∖ F the retain set. There are many ways one might formalize unlearning. We consider the following definition of unlearning (Sekhari et al., 2021; Gupta et al., 2021; Neel et al., 2021)111Note that the cited papers usually have another parameter δ that accounts for a shift in Equation 1. In our case δ=00δ=0δ = 0. Further, it is common to consider a type of a “publish” function that allows to compare a distribution over other quantities, potentially obtained by post-processing (“publishing”) the output ⁢(S)A(S)A ( S ).: Definition 2.1. An algorithm UU is a worst-case ε ε-unlearner (for AA) if, for every training set S, forget set F⊂SF⊂ SF ⊂ S of fixed size, and measurable subset B⊆ΘB ⊆ Θ, letting θ∼⁢(⁢(S),F)similar-tosubscript _U (A(S),F)θcaligraphic_U ∼ U ( A ( S ) , F ) and θ−F∼⁢(S∖F)similar-tosubscript _-F (S F)θ- F ∼ A ( S ∖ F ), e−ε⁢Pr⁡(θ−F∈B|S,F)≤Pr⁡(θ∈B|S,F)≤eε⁢Pr⁡(θ−F∈B|S,F).superscriptPrsubscriptconditionalPrsubscriptconditionalsuperscriptPrsubscriptconditionale^- ( _-F∈ B|S,F)≤ ( _U∈ B|S% ,F)≤ e ( _-F∈ B|S,F).e- ε Pr ( θ- F ∈ B | S , F ) ≤ Pr ( θcaligraphic_U ∈ B | S , F ) ≤ eitalic_ε Pr ( θ- F ∈ B | S , F ) . (1) For any distribution over training data S and forget sets F, we say UU is an on-average ε ε-unlearner if Equation 1 holds when the probabilities are unconditional. We refer to a sample θ−F∼⁢(S∖F)similar-tosubscript _-F (S F)θ- F ∼ A ( S ∖ F ) as a reference model. Definitions of unlearning vary in a number of ways, including in terms of what information is available to the unlearning algorithm. The role of access to (statistics of) the training data is studied by Sekhari et al. (2021) . In this work, we will study unlearning algorithms that operate by performing gradient ascent on the NLL loss, averaged over the forget set F. 3 Evaluation of Unlearning in Large Language Models Fix a pair of algorithms ,A,\,UA , U. Let GG be the set of measurable functions from Θ Θ taking values in [0,1]01[0,1][ 0 , 1 ]. For training data S and forget set F⊆SF SF ⊆ S, the smallest ε ε satisfying Equation 1 is εS,F=supg∈(|log⁢[g⁢(θ)]−log⁢[g⁢(θ−F)]|),subscriptsubscriptsupremumsubscriptsubscript _S,F= _g (| *E[% g( _U)]- *E[g( _-F)]|% ),εitalic_S , F = supitalic_g ∈ G ( | log blackboard_E [ g ( θcaligraphic_U ) ] - log blackboard_E [ g ( θ- F ) ] | ) , (2) where θ∼⁢(⁢(S),F)similar-tosubscript _U (A(S),F)θcaligraphic_U ∼ U ( A ( S ) , F ) and θ−F∼⁢(S∖F)similar-tosubscript _-F (S F)θ- F ∼ A ( S ∖ F ). The supremum supS,FεS,Fsubscriptsupremumsubscript _S,F _S,Fsupitalic_S , F εitalic_S , F is the tightest parameter for the worst-case notion. It follows that evaluating the argument in the r.h.s. of Equation 2 with any g∈g ∈ G yields a lower bound on the unlearning parameter ε ε. Below we construct a function g that we use to evaluate unlearning. Let F⊆SF SF ⊆ S be a set of strings we want to forget. In addition, consider n reference strings R=rii=1nsuperscriptsubscriptsubscript1R=\r_i\_i=1^nR = ritalic_i i = 1n, sampled from some given distribution, and that are not part of S (or F). Recall that ℒ⁢(θ,x)=−log⁡f⁢(x;θ)ℒL(θ,x)=- f(x;θ)L ( θ , x ) = - log f ( x ; θ ) is the negative log-likelihood of a sequence x under model θ. Let g⁢(x;θ,R)=1n⁢∑j=1nℒ⁢(θ,x)ℒ⁢(θ,x)+ℒ⁢(θ,rj).1superscriptsubscript1ℒsubscriptg(x;θ,R)= 1n _j=1^n L(θ,x)L% (θ,x)+L(θ,r_j).g ( x ; θ , R ) = divide start_ARG 1 end_ARG start_ARG n end_ARG ∑j = 1n divide start_ARG L ( θ , x ) end_ARG start_ARG L ( θ , x ) + L ( θ , ritalic_j ) end_ARG . (3) Each term ℒ⁢(θ,x)/(ℒ⁢(θ,x)+ℒ⁢(θ,rj))ℒsubscriptL(θ,x)/ (L(θ,x)+L(θ,r_j) )L ( θ , x ) / ( L ( θ , x ) + L ( θ , ritalic_j ) ) can be seen as a relaxation of the hard comparison (ℒ⁢(θ,x)≤ℒ⁢(θ,rj))ℒsubscript(L(θ,x) (θ,r_j))( L ( θ , x ) ≤ L ( θ , ritalic_j ) ) (or equivalently (f⁢(rj;θ)≤f⁢(x;θ))subscript(f(r_j;θ)≤ f(x;θ))( f ( ritalic_j ; θ ) ≤ f ( x ; θ ) ), as the NLL is monotonically decreasing). In aggregate, it represents the fraction of reference strings in R that have an NLL higher than x. g can be seen as a soft version (scaled to [0,1]01[0,1][ 0 , 1 ]) of the rank of f⁢(x;θ)f(x;θ)f ( x ; θ ) among the probabilities of reference strings f⁢(rj)j=1nsuperscriptsubscriptsubscript1\f(r_j)\_j=1^n f ( ritalic_j ) j = 1n. A smaller value of g indicates x is more likely under θ (has a smaller loss) than elements of R, a larger value indicates it is less likely (has a larger loss). If g⁢(x;θ,R)<γg(x;θ,R)< ( x ; θ , R ) < γ, then there are at most 2⁢n⁢γ22n 2 n γ elements risubscriptr_iritalic_i of R such that f⁢(ri;θ)>f⁢(x;θ)subscriptf(r_i;θ)>f(x;θ)f ( ritalic_i ; θ ) > f ( x ; θ ) (and ℒ⁢(θ,ri)<ℒ⁢(θ,x)ℒsubscriptℒL(θ,r_i)<L(θ,x)L ( θ , ritalic_i ) < L ( θ , x )). Similarly, if g⁢(x;θ,R)>1−γ1g(x;θ,R)>1- ( x ; θ , R ) > 1 - γ, then at most 2⁢n⁢γ22n 2 n γ elements ri∈Rsubscriptr_i∈ Rritalic_i ∈ R satisfy f⁢(ri;θ)<f⁢(x;θ)subscriptf(r_i;θ)<f(x;θ)f ( ritalic_i ; θ ) < f ( x ; θ ) (and ℒ⁢(θ,ri)>f⁢(x;θ)ℒsubscriptL(θ,r_i)>f(x;θ)L ( θ , ritalic_i ) > f ( x ; θ )). We define Generalized Exposure of x∈Fx∈ Fx ∈ F relative to a set of reference strings R to be GenEx⁡(x;,,F,S)=log⁢[g⁢(x;θ−F,R)]−log⁢[g⁢(x;θ,R)].GenExsubscriptsubscript splitGenEx(x;A,U,F,S)=& % *E[g(x; _-F,R)]- *E% [g(x; _U,R)]. splitstart_ROW start_CELL GenEx ( x ; A , U , F , S ) = end_CELL start_CELL log blackboard_E [ g ( x ; θ- F , R ) ] - log blackboard_E [ g ( x ; θcaligraphic_U , R ) ] . end_CELL end_ROW (4) Taking the absolute value of GenExGenExGenExGenEx yields a lower bound on the worst-case epsilon in Equation 2 for a fixed g. One cannot compute the expectations in Equation 4 exactly, since the distributions of θsubscript _Uθcaligraphic_U and θ−Fsubscript _-Fθ- F are not tractable in a standard deep learning setup. In our experiments, we use a Monte Carlo estimate of the expectations in the generalized exposure metric to get an approximate lower bound on the unlearning quality. Such estimates are subject to variance. Alternatively, one could threshold the observed g⁢(x;θ,R)subscriptg(x; _U,R)g ( x ; θcaligraphic_U , R ), which would effectively correspond to choosing a different g∈g ∈ G in Equation 2, and then use Clopper–Pearson confidence intervals for binomials to compute the confidence intervals of the estimates (Clopper & Pearson, 1934) (also see (Jagielski et al., 2020)). Exposure and memorization222We intend here a very restricted definition of “memorization”: whether a generative model can be induced to generate near-facsimiles of some training examples when prompted with appropriate instructions. Models do not “contain” bit-wise or code-wise copies of their training data. Rather, if a model can be induced to generate very close copies of certain training examples by supplying appropriate instructions to guide the model’s statistical generation processes then that model is said to have “memorized” those examples. This is an area of active ongoing research.. Generalized exposure can be seen as an extension of the exposure metric that appeared in the memorization literature, introduced by Carlini et al. (2019). There, the authors inject secret canaries (i.e., strings generated randomly, from a different distribution than the regular data distribution) C=ciimsuperscriptsubscriptsubscriptC=\c_i\_i^mC = citalic_i iitalic_m in the training set S. In our notation, C=FC=FC = F. In addition, n reference strings rii=1nsuperscriptsubscriptsubscript1\r_i\_i=1^n ritalic_i i = 1n are sampled from the same distribution. For each canary cisubscriptc_icitalic_i, letting rank⁡(li|ljj)rankconditionalsubscriptsubscriptsubscriptrank(l_i|\l_j\_j)rank ( litalic_i | litalic_j j ) denote the rank of lisubscriptl_ilitalic_i in the set ljjsubscriptsubscript\l_j\_j litalic_j j, Carlini et al. (2019) define exposure as333They define it in terms of log-perplexity instead of NLL, but the only difference is a multiplicative log⁡(2)2 (2)log ( 2 ) factor, which is irrelevant in ranking and comparison.: Ex⁡(ci;θ)=log2⁡(n)−log2⁡(rank⁡(ℒ⁢(θ,ci)|ℒ⁢(θ,rj)j=1n))⁢, or, equivalentlyExsubscriptsubscript2subscript2rankconditionalℒsubscriptsuperscriptsubscriptℒsubscript1, or, equivalentlyEx(c_i;θ)= _2(n)- _2(rank(% L(θ,c_i)|\L(θ,r_j)\_j=1^n)), or,% equivalentlyEx ( citalic_i ; θ ) = log2 ( n ) - log2 ( rank ( L ( θ , citalic_i ) | L ( θ , ritalic_j ) j = 1n ) ) , or, equivalently (5) Ex⁡(ci;θ)=−log2⁡Prj=1:n⁡[ℒ⁢(θ,rj)≤ℒ⁢(θ,ci)].Exsubscriptsubscript2subscriptPr:1ℒsubscriptℒsubscriptEx(c_i;θ)=- _2 _j=1:n [L(θ,r% _j) (θ,c_i) ].Ex ( citalic_i ; θ ) = - log2 Pritalic_j = 1 : n [ L ( θ , ritalic_j ) ≤ L ( θ , citalic_i ) ] . (6) This metric is meant to capture how much the model memorized the canaries relative to the reference strings that were not seen during training. Generalized Exposure uses a function g (Equation 3) that can be seen as a soft version of the comparison function used in the second formulation of exposure (Equation 6). The reference strings it uses do not have to come from outside of the distribution of regular data in general, but we can consider R=rii=1nsuperscriptsubscriptsubscript1R=\r_i\_i=1^nR = ritalic_i i = 1n, as defined above, as a special case. For a randomly generated string r coming from the same distribution as R, and never seen during learning or unlearning (θ would be independent of them), g⁢(r;θ,R)g(r;θ,R)g ( r ; θ , R ) should be around ½, and each term in 4 of the form −log⁢g⁢(r;θ,R)=log⁡(2)2- *Eg(r;θ,R)= (2)- log blackboard_E g ( r ; θ , R ) = log ( 2 ). Similarly, the probability in Equation 6 will tend to ½, and exposure to log2⁡(2)=1subscript221 _2(2)=1log2 ( 2 ) = 1. When no memorization happens, Generalized Exposure for these randomly generated canary strings C is zero. To see this, note that g⁢(ci;θ)=12subscript12 *Eg(c_i;θ)= 12blackboard_E g ( citalic_i ; θ ) = divide start_ARG 1 end_ARG start_ARG 2 end_ARG under no memorization, and both sides of Generalized Exposure cancel out. For the exposure computation, the outcome of the comparison is 1212 12divide start_ARG 1 end_ARG start_ARG 2 end_ARG, giving Ex⁡(ci;θ)=−log2⁡12=1Exsubscriptsubscript2121Ex(c_i;θ)=- _2 12=1Ex ( citalic_i ; θ ) = - log2 divide start_ARG 1 end_ARG start_ARG 2 end_ARG = 1. Under maximal memorization of cisubscriptc_icitalic_i, the loss would be smaller than for all the reference strings, and thus Ex⁡(ci;θ)→∞→ExsubscriptEx(c_i;θ)→∞Ex ( citalic_i ; θ ) → ∞. Similarly, the first term in the Generalized Exposure, −log⁢[g⁢(x;θ,R)]subscript- *E[g(x; _U,R)]- log blackboard_E [ g ( x ; θcaligraphic_U , R ) ] would tend to ∞ when g⁢(x;θ,R)subscriptg(x; _U,R)g ( x ; θcaligraphic_U , R ) gets arbitrarily close to 0 as f⁢(ci;θ)subscriptsubscriptf(c_i; _U)f ( citalic_i ; θcaligraphic_U ) increases with more memorization relative to the reference strings. Membership inference attacks and differential privacy. Jagielski (2023) connects the exposure metric from (Carlini et al., 2019) to differential privacy and so-called membership inference attacks. Recall that a training algorithm AA is ε ε-differentially private (DP) if, for all S and S′ that differ by one data point, and all measurable sets B⊆ΘB ⊆ Θ, Pr⁡(θ∈B)≤eε⁢Pr⁡(θ′∈B)PrsuperscriptPrsuperscript′ (θ∈ B)≤ e (θ ∈ B)Pr ( θ ∈ B ) ≤ eitalic_ε Pr ( θ′ ∈ B ), where θ∼⁢(S)similar-toθ (S)θ ∼ A ( S ) and θ′∼⁢(S′)similar-tosuperscript′θ (S )θ′ ∼ A ( S′ ). One can interpret DP as a hypothesis test to assess whether the output of the algorithm was obtained by running AA on S versus S′. Kairouz et al. (2015) show that a particular computation based on false positive and false negative rates associated with this hypothesis test yields an estimate of ε ε in the differential privacy definition. Through this hypothesis test view, ε ε-DP can be connected to a version of so-called membership inference attacks (MIAs; see, e.g., Shokri et al. 2017), which attempt to identify whether a data point was or was not in the training set. Probably the most related MIA is a likelihood-ratio test (LiRA) introduced by Carlini et al. (2022a). LiRA is motivated by the connections to hypothesis testing, trying to determine whether the observed prediction is more likely to have been sampled from a model that was trained on the sample of interest or without. The authors choose to do a likelihood ratio test (motivated by the Neyman–Pearson lemma), assuming that the predictions for a given sample have a Gaussian distribution. Inspired by the work by Kairouz et al. (2015) connecting differential privacy and MIAs, Triantafillou et al. (2023, 2024) propose to estimate ε ε in the unlearning definition Equation 1 using false positive and false negative rates from a MIA perspective. In particular, letting π\π\ π denote all membership inference attacks, ε ε in the unlearning definition above can be estimated as a supremum over π\π\ π of a function of upper and lower bounds of false positive/negative rates for π. 3.1 Relative exposure Generalized exposure requires computing the expected probability of x under ⁢(S∖F)A(S F)A ( S ∖ F ). Practically, having such a reference model may not be possible for computational and memory reasons. Here we introduce an alternative test that only requires access to the original model pre-unlearning. As above, consider a set of reference strings R, and let θS∼⁢(S)similar-tosubscript _S (S)θitalic_S ∼ A ( S ), θ−F∼⁢(S∖F)similar-tosubscript _-F (S F)θ- F ∼ A ( S ∖ F ) and θ∼⁢(⁢(S),F)similar-tosubscript _U (A(S),F)θcaligraphic_U ∼ U ( A ( S ) , F ). For each given x, we now randomly generate a second set of reference strings RxsubscriptR_xRitalic_x, such that log⁢[g⁢(x;θ−F,R)]≈log⁢[^r∈Rx⁢[g⁢(r;θS,R)]]subscriptsubscript^subscriptdelimited-[]subscript *E[g(x; _-F,R)]≈ *% E [ *E_r∈ R_x [g(r;% _S,R) ] ]log blackboard_E [ g ( x ; θ- F , R ) ] ≈ log blackboard_E [ over start_ARG blackboard_E end_ARGr ∈ R start_POSTSUBSCRIPT x end_POSTSUBSCRIPT [ g ( r ; θitalic_S , R ) ] ], where ^r∈Rxsubscript^subscript *E_r∈ R_xover start_ARG blackboard_E end_ARGr ∈ R start_POSTSUBSCRIPT x end_POSTSUBSCRIPT denotes an empirical mean over the elements in RxsubscriptR_xRitalic_x. In theory this is a complex task, and once again requires access to the reference model. In practice, however, we will simply choose RxsubscriptR_xRitalic_x so its elements are close to x under some similarity metric (working in the embedding space), but not part of F; RxsubscriptR_xRitalic_x can contain examples from some auxiliary set (public data, held out data, etc.), that do not belong to the training set S. It is also possible to define a common RxsubscriptR_xRitalic_x for all x∈Fx∈ Fx ∈ F. By choosing such a set, we ensure that θSsubscript _Sθitalic_S does not depend on RxsubscriptR_xRitalic_x, just like θ−Fsubscript _-Fθ- F does not depend on x∈Fx∈ Fx ∈ F. Further, when the forget set is small and does not affect the predictions on RxsubscriptR_xRitalic_x through θSsubscript _Sθitalic_S too much, we can expect our approximation to be more accurate. Substituting this approximation to Equation 4, we get an alternative to Generalized Exposure that does not use θ−Fsubscript _-Fθ- F. We define Relative Exposure of x relative to R,RxsubscriptR,R_xR , Ritalic_x, as RelEx⁡(x;,,F,S,R,Rx)=log2⁢[^r∈Rx⁢[g⁢(r;θS,R)]]−log2⁢[g⁢(x;θ,R)].RelExsubscriptsubscript2subscript^subscriptdelimited-[]subscriptsubscript2subscript splitRelEx(x;A,U,F,S,R,R_x)= _% 2 *E [ *E_r∈ R_% x [g(r; _S,R) ] ]- _2 *E% [g(x; _U,R)]. splitstart_ROW start_CELL RelEx ( x ; A , U , F , S , R , Ritalic_x ) = log2 blackboard_E [ over start_ARG blackboard_E end_ARGr ∈ R start_POSTSUBSCRIPT x end_POSTSUBSCRIPT [ g ( r ; θitalic_S , R ) ] ] - log2 blackboard_E [ g ( x ; θcaligraphic_U , R ) ] . end_CELL end_ROW (7) 3.2 Memorization and example difficulty When evaluating unlearning, we group examples in the forget set based on measures of the extent to which an example has been memorized and of the example’s difficulty. More carefully, let θS∼⁢(S)similar-tosubscript _S (S)θitalic_S ∼ A ( S ) and θ−F∼⁢(S∖F)similar-tosubscript _-F (S F)θ- F ∼ A ( S ∖ F ). For any fixed example x∈Fx∈ Fx ∈ F, we define its memorization as log⁢[f⁢(x;θS)]−log⁢[f⁢(x;θ−F)].subscriptsubscript *E[f(x; _S)]- % *E[f(x; _-F)].log blackboard_E [ f ( x ; θitalic_S ) ] - log blackboard_E [ f ( x ; θ- F ) ] . This is similar to the definition of memorization given by Feldman (2020) for classification tasks, which measures the difference between the probabilities of predicting the right class depending under θ−Fsubscript _-Fθ- F and θSsubscript _Sθitalic_S. We define the difficulty of any fixed example x∈Fx∈ Fx ∈ F as [−log⁡f⁢(x;θ−F)]subscript *E[- f(x; _-F)]blackboard_E [ - log f ( x ; θ- F ) ], i.e., expected perplexity under the reference model. Note that all the expectations here are taken over the randomness of the training process (e.g., SGD noise, minibatch noise). 3.3 Unlearning by gradient ascent Based on Equation 1, one could achieve exact (ε=00 =0ε = 0) unlearning by retraining from scratch without the forget set F. This is not practical for large language models, due to resource constraints. Another common approach is to perform gradient ascent on the loss over F, or/and gradient descent on the loss over S∖FS FS ∖ F. Other alternatives have been proposed in the literature (Patil et al., 2023; Meng et al., 2022a, b), but a gradient ascent/descent-type procedure is still a common component in all of them. While this approach is fairly efficient, and usually implemented with only a small number of gradient updates, it is not guaranteed that the obtained model after unlearning via gradient ascent/descent has truly forgotten the samples. Further, there is no set heuristic for the number of gradient updates to run during unlearning. This technique, thus, hinges on being able to assess how unlearned a set of examples is for a given language model. 3.4 Unlearning in- versus out-of-distribution samples For a reference model, unlearning, which is equivalent to not training on, out-of-distribution samples should not affect the overall performance, meaning that the models θSsubscript _Sθitalic_S and θ−Fsubscript _-Fθ- F should perform similarly. When the forget set contains in-distribution samples, then the effect depends on the size of the forget set relative to the training set. We focus on the typical case where the size of the forget set is small enough, and both models, θSsubscript _Sθitalic_S and θ−Fsubscript _-Fθ- F, perform similarly under the BLEU score. Thus a good unlearning algorithm should be able to unlearn without any observable trade-offs between unlearning quality and overall performance, as measured by the BLEU score. 4 Experiments We evaluate the trade-offs between unlearning quality and performance on LLMs. We demonstrate that unlearning more memorized or more difficult examples is more damaging for the overall model performance. We also examine how neighbouring examples are affected by unlearning. Finally, we show that our relative exposure metric captures unlearning quality as well as the generalized exposure metric, thus showing a way to assess unlearning without having a reference model. 4.1 Experimental setup Models and datasets. We train a transformer model (T5-base with 220220220220 million parameters (Roberts et al., 2022)) on “WMT14 En-De”, a well-known language translation dataset that contains sentence pairs in German and English (Bojar et al., 2014). We train for 45,0004500045,00045 , 000 training steps with batch size of 128128128128 on examples from the training split. We evaluate the task performance of the translation task using the BiLingual Evaluation Understudy (BLEU)) score (Papineni et al., 2002). Our models have a BLEU score of around 26262626, having a clear gist but with grammatical errors. To avoid training numerous models, we only perform full training of two models: • A reference model, of weights θ−Fsubscript _-Fθ- F is trained on the full standard training split T of the dataset mentioned above, without any additional example from a forget set. • A subject model, of weights θSsubscript _Sθitalic_S, is trained on a dataset S made of T and the concatenation of all the potential forget sets F…subscript…F_…F… defined below. We consider an unlearning method based on gradient ascent (Section 3.3). Following Jang et al. (2022), we use a batch size of 32323232 when unlearning a set of 512512512512 examples, giving us 16161616 unlearning steps to go through for the entire forget set considered. During each unlearning experiment, we consider one single forget set F, and perform unlearning only on its examples. We always compare the resulting unlearned model with the same, shared reference model. Even though the reference model is only trained T, which is a subset of the retain set of any given experiment (the full retain set would include the forget sets for the other experiments), we consider it a suitable approximation, as the ignored examples form only a small fraction of the training set. Out-of-distribution forget sets generation. We generate out-of-distribution (OOD) canaries by sampling alpha-numeric characters uniformly at random, forming a sequence of fixed length (10101010 characters). We create three disjoint sets F…OODsubscriptsuperscriptOOD…F^OOD_…FOOD… of 512 OOD canaries each, as well as a set ROODsuperscriptOODR^OODROOD of 10,000 reference strings from the same distribution. To study the effect of the number of repetition on unlearning, these sets are incorporated in the training set S with different frequencies: canaries in F×1OODsubscriptsuperscriptOODabsent1F^OOD_× 1FOOD× 1 are seen only once during training, the ones in F×10OODsubscriptsuperscriptOODabsent10F^OOD_× 10FOOD× 10 ten times, and F×100OODsubscriptsuperscriptOODabsent100F^OOD_× 100FOOD× 100 a hundred times. In-distribution forget sets generation. We generate sets of in-distribution (InD) examples by randomly selecting examples from the validation split of the dataset, so that we can train the reference model on the full training split T, and these examples do not appear even once in its training set. We create three disjoint sets F…InDsubscriptsuperscriptInD…F^InD_…FInD… of 512 in-distribution examples each, as well as a set RInDsuperscriptInDR^InDRInD of 3,00330033,0033 , 003 reference strings formed from the test split. All of these sets of examples are disjoint. Similarly to the OOD canaries, these sets are incorporated in the training set with different frequencies. For the main subject model considered (θSsubscript _Sθitalic_S), F×1InDsubscriptsuperscriptInDabsent1F^InD_× 1FInD× 1 is seen only once, F×10InDsubscriptsuperscriptInDabsent10F^InD_× 10FInD× 10 ten times, and F×100InDsubscriptsuperscriptInDabsent100F^InD_× 100FInD× 100 a hundred times. Section A.2 also considers a model trained on a different training set S′, where different validation examples are used in forget sets, see that section for details. Figure 1: In- vs. out-of-distribution (canary) trade-off. Trade-off between the generalized exposure (Exposure) and the task performance (BLEU score) when unlearning the subject model (θSsubscript _Sθitalic_S) at 45,000 steps, with in-distribution sets and canary sets repeated 100 times (left), 10 times (middle), and 1 time (right) during training. 4.2 Memorization vs. performance trade-offs Our evaluated method of unlearning modifies the model by performing gradient ascent, as a result it might degrade the model’s accuracy on the test set. We first evaluate the trade-off between the effectiveness of unlearning under generalized exposure and the task performance on the unlearned model (Figure 1). At every unlearning step, we measure the average exposure of the canary, and, respectively, forget set. On these checkpoints, we compute the BLEU score on the test set. Our first observation is that unlearning of canary data in one pass does not degrade the performance as much as unlearning in-distribution samples even when these are repeated as often. The average exposure value of the canaries also does not fall below 1111 in one pass, meaning the canaries are still twice as less surprising to the model than other random samples unseen in training. The average exposure of the InD samples, however, falls to the minimum value. The reason is that unlearning InD examples affects the perplexities of other similar examples (Section 4.4), whereas for out-of-distribution, unlearning does not affect as much the other canaries’ perplexities. This explains why the in-distribution examples have a much faster drop in exposure, as well as task performance. Different Frequencies. In Figure 1, we observe that the more repeats of the in-distribution sample sets, the higher the (average) generalized exposure is before unlearning (top right point of each orange curve). A similar effect is visible for the exposure of OOD between the OOD×1OOD1OOD× 1OOD × 1 and the OOD×10OOD10OOD× 10OOD × 10 curves, although it is not visible in the OOD×100OOD100OOD× 100OOD × 100 because the estimate of exposure is limited by log2⁡|ROOD|subscript2superscriptOOD _2 |R^OOD |log2 | ROOD |. In Section A.2, we also evaluate how a different number of repetition of the same examples of the in-distribution sets affect the trade-off. Despite the three randomly-selected InD sets having a different distribution of perplexities under the reference model (as shown in Figure 2), the qualitative results are not affected by which set is repeated a given number of times. Figure 2: Distributions of perplexities. Perplexities of different sets of in-distribution examples under the subject model (before unlearning, post-unlearning and when exposure is low) and the reference model. Columns left to right: in-distribution example perplexities when the subject model was trained by repeating these examples 100 times (left), 10 times (middle), 1 time (right). Distribution of perplexities. We check how the perplexities of the in-distribution samples are affected before and after unlearning with respect to the reference model. We observe that the perplexities of the in-distribution set is reduced, but now the perplexities are skewed, not resembling at all the distribution on the reference model (Figure 2). We also plot the distribution of perplexities when the exposure is below a certain threshold which results in distributions that are closer to the reference. This suggests an early-stopping strategy for unlearning could benefit in-distribution examples with more evidence of this effect at lower thresholds in Section A.4. 4.3 Per-sample difficulty vs. memorization Figure 3: Per-sample memorization vs. difficulty. The memorization vs. difficulty for each sample in the forgets sets that repeat ×1absent1× 1× 1, and ×100absent100× 100× 100. Difficulty and memorization become correlated with number of repeats. Figure 4: Difficulty vs. trade-offs. Measure the trade-offs of unlearning examples of low, medium and high difficulty. Harder examples have slightly better trade-offs. We empirically evaluate the relationship between the difficulty of in-distribution examples and memorization. In Figure 3, we plot the memorization and difficulty of each example in the forget sets. The per-sample difficulty has a weak correlation with the per-sample memorization when the InD set is repeated once, but the correlation becomes strong with the number of repeats. We also cluster the in-distribution examples into 3 sets of low, medium and high perplexity based on their difficulty (Figure 4), and find that harder examples have slightly better trade-offs (see details in Section A.3). 4.4 Unlearning effects on other points Figure 5: Unlearning affects the average exposure of similar examples. We highlight that unlearning InD examples has an impact on other similar examples. We find similar examples by computing the L2subscript2L_2L2-distance in the embedding space of each point in the forget set on the reference model (see Section A.6). In Figure 5, we plot the memorization vs. performance trade-offs (as we unlearn F×100InDsubscriptsuperscriptInDabsent100F^InD_× 100FInD× 100) for both the set F×100InDsubscriptsuperscriptInDabsent100F^InD_× 100FInD× 100 and a set of similar examples from F×1InDsubscriptsuperscriptInDabsent1F^InD_× 1FInD× 1. The average exposure of the similar set decreases, without having to do unlearning. This explains why unlearning damages the performance of the model, since the model may forget other examples. Despite this, the effect of unlearning on similar examples outside of the training set is not significant. Thus, unlearning may affect examples that are more memorized as opposed to just similar examples. 5 Related Work Recent work on unlearning in LLMs has focused on developing effective unlearning algorithms and robust evaluation metrics to assess the degree of unlearning achieved. We give a brief overview of the most relevant work here, and point interested readers to Section A.7 for more related work. Unlearning benchmarks and evaluation metrics. Several works propose leveraging evaluation metrics of memorization with the aim to provide better unlearning methods in LLMs (Jang et al., 2022; Barbulescu & Triantafillou, 2024). Our work aims to work with a worst-case ε ε-unlearner (Definition 2.1) and can be seen as complementary to these approaches. Our experiments also point to stark differences between in- and out-of-distribution memorized data. An orthogonal unlearning approach is by removing of training data from the weights (Meng et al., 2022a; Patil et al., 2023). Memorization in LLMs. Whereas our work targets memorized data unlearning, a range of other memorization notions and concerns have been studied in LLMs (Lehman et al., 2021; Ippolito et al., 2022; Carlini et al., 2021; Choquette-Choo et al., 2021; Lukas et al., 2023). 6 Conclusion In this work, we propose a generalized exposure metric for evaluating unlearning. We find instances where gradient ascent-based techniques are insufficient for unlearning without destroying the model’s performance. We explain this through the effect of unlearning on similar data. Acknowledgments We thank Daniel M. Roy and Eleni Triantafillou for feedback on various drafts of this work. This project used computational resources on Google Cloud Platform provided by Google, we would like to thank Danat Pomeranets in particular for support. Teodora was supported in part by the National Research Foundation Singapore under its NRF Fellowship Programme [NRF-NRFFAI1-2019-0004], by the Crystal Centre at National University of Singapore and its sponsors, and a Google PhD fellowship. References Barbulescu & Triantafillou (2024) George-Octavian Barbulescu and Peter Triantafillou. To each (textual sequence) its own: Improving memorized-data unlearning in large language models. arXiv preprint arXiv:2405.03097, 2024. Bojar et al. (2014) Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ale s Tamchyna. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, p. 12–58, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics. URL http://w.aclweb.org/anthology/W/W14/W14-3302. Carlini et al. (2019) Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), p. 267–284, 2019. Carlini et al. (2021) Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), p. 2633–2650, 2021. Carlini et al. (2022a) Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), p. 1897–1914. IEEE, 2022a. Carlini et al. (2022b) Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646, 2022b. Choquette-Choo et al. (2021) Christopher A Choquette-Choo, Florian Tramer, Nicholas Carlini, and Nicolas Papernot. Label-only membership inference attacks. In International conference on machine learning, p. 1964–1974. PMLR, 2021. Clopper & Pearson (1934) Charles J Clopper and Egon S Pearson. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4):404–413, 1934. Feldman (2020) Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, p. 954–959, 2020. Gu et al. (2023) Kang Gu, Ehsanul Kabir, Neha Ramsurrun, Soroush Vosoughi, and Shagufta Mehnaz. Towards sentence level inference attack against pre-trained language models. Proceedings on Privacy Enhancing Technologies, 3:62–78, 2023. Gupta et al. (2021) Varun Gupta, Christopher Jung, Seth Neel, Aaron Roth, Saeed Sharifi-Malvajerdi, and Chris Waites. Adaptive machine unlearning. Advances in Neural Information Processing Systems, 34:16319–16330, 2021. Hayes et al. (2024) Jamie Hayes, Ilia Shumailov, Eleni Triantafillou, Amr Khalifa, and Nicolas Papernot. Inexact unlearning needs more careful evaluations to avoid a false sense of privacy. arXiv preprint arXiv:2403.01218, 2024. Ippolito et al. (2022) Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette-Choo, and Nicholas Carlini. Preventing verbatim memorization in language models gives a false sense of privacy. arXiv preprint arXiv:2210.17546, 2022. Jagielski (2023) Matthew Jagielski. A note on interpreting canary exposure. arXiv preprint arXiv:2306.00133, 2023. Jagielski et al. (2020) Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private sgd? Advances in Neural Information Processing Systems, 33:22205–22216, 2020. Jang et al. (2022) Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprint arXiv:2210.01504, 2022. Kairouz et al. (2015) Peter Kairouz, Sewoong Oh, and Pramod Viswanath. The composition theorem for differential privacy. In International conference on machine learning, p. 1376–1385. PMLR, 2015. Lee et al. (2021) Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021. Lehman et al. (2021) Eric Lehman, Sarthak Jain, Karl Pichotta, Yoav Goldberg, and Byron C Wallace. Does BERT pretrained on clinical notes reveal sensitive data? arXiv preprint arXiv:2104.07762, 2021. Lukas et al. (2023) Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella-Béguelin. Analyzing leakage of personally identifiable information in language models. arXiv preprint arXiv:2302.00539, 2023. Lynch et al. (2024) Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. Eight methods to evaluate robust unlearning in llms. arXiv preprint arXiv:2402.16835, 2024. Maini et al. (2024) Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. Tofu: A task of fictitious unlearning for llms. arXiv preprint arXiv:2401.06121, 2024. Mattern et al. (2023) Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schölkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick. Membership inference attacks against language models via neighbourhood comparison. arXiv preprint arXiv:2305.18462, 2023. Meng et al. (2022a) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 35:17359–17372, 2022a. Meng et al. (2022b) Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. arXiv preprint arXiv:2210.07229, 2022b. Mireshghallah et al. (2022) Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick, and Reza Shokri. Quantifying privacy risks of masked language models using membership inference attacks. arXiv preprint arXiv:2203.03929, 2022. Nakamura et al. (2020) Yuta Nakamura, Shouhei Hanaoka, Yukihiro Nomura, Naoto Hayashi, Osamu Abe, Shuntaro Yada, Shoko Wakamiya, and Eiji Aramaki. KART: Parameterization of privacy leakage scenarios from pre-trained language models. arXiv preprint arXiv:2101.00036, 2020. Nasr et al. (2018) Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning. In Proceedings of the 2019 IEEE Symposium on Security and Privacy (SP), volume 2018, p. 1–15, 2018. Neel et al. (2021) Seth Neel, Aaron Roth, and Saeed Sharifi-Malvajerdi. Descent-to-delete: Gradient-based methods for machine unlearning. In Algorithmic Learning Theory, p. 931–962. PMLR, 2021. Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, p. 311–318, USA, 2002. Association for Computational Linguistics. doi: 10.3115/1073083.1073135. URL https://doi.org/10.3115/1073083.1073135. Patil et al. (2023) Vaidehi Patil, Peter Hase, and Mohit Bansal. Can sensitive information be deleted from LLMs? objectives for defending against extraction attacks. arXiv preprint arXiv:2309.17410, 2023. Ponomareva et al. (2022) Natalia Ponomareva, Jasmijn Bastings, and Sergei Vassilvitskii. Training text-to-text transformers with privacy guarantees. In Findings of the Association for Computational Linguistics: ACL 2022, p. 2182–2193, 2022. Roberts et al. (2022) Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepard, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. Scaling up models and data with t5x and seqio. arXiv preprint arXiv:2203.17189, 2022. URL https://arxiv.org/abs/2203.17189. Salem et al. (2018) Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models. arXiv preprint arXiv:1806.01246, 2018. Sekhari et al. (2021) Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems, 34:18075–18086, 2021. Shi et al. (2024) Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A Smith, and Chiyuan Zhang. Muse: Machine unlearning six-way evaluation for language models. arXiv preprint arXiv:2407.06460, 2024. Shokri et al. (2017) Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), p. 3–18. IEEE, 2017. Triantafillou et al. (2023) Eleni Triantafillou, Fabian Pedregosa, Jamie Hayes, Peter Kairouz, Isabelle Guyon, Meghdad Kurmanji, Gintare Karolina Dziugaite, Peter Triantafillou, Kairan Zhao, Lisheng Sun Hosoya, Julio C. S. Jacques Junior, Vincent Dumoulin, Ioannis Mitliagkas, Sergio Escalera, Jun Wan, Sohier Dane, Maggie Demkin, and Walter Reade. NeurIPS 2023 - machine unlearning, 2023. URL https://kaggle.com/competitions/neurips-2023-machine-unlearning. Triantafillou et al. (2024) Eleni Triantafillou, Peter Kairouz, Fabian Pedregosa, Jamie Hayes, Meghdad Kurmanji, Kairan Zhao, Vincent Dumoulin, Julio Jacques Junior, Ioannis Mitliagkas, Jun Wan, et al. Are we making progress in unlearning? findings from the first neurips unlearning competition. arXiv preprint arXiv:2406.09073, 2024. Appendix A Appendix A.1 Experimental Setup Details The training dataset is available in TensorFlow Datasets as “wmt_t2t_translate”. It has three splits: a train split T with 4,592,28945922894,592,2894 , 592 , 289 examples, a validation split of 3,00030003,0003 , 000 examples, and a test split with 3,00330033,0033 , 003 examples. We use the validation split for selecting the InD sets and the test split for the reference strings for InD. Remark on out-of-distribution sets generation. Note that our approach to generating canaries differs from that in (Carlini et al., 2019). There, the canaries are generated with a fixed string prefix (or template) such as “My secret is:” and a randomly-generated string suffix c, sampled from a randomness space c∼similar-toc ∼ C, e.g., the alpha-numeric strings of length 10101010. These canaries aim to mimic accidental personal identifiable information (PII) in the training data, where the sensitive information was a unique string of characters, such as a social security number. However, having many canaries sharing the same template in the training set means that the model could learn to detect this pattern, and share some representation between canaries. This can be especially troublesome in the context of evaluating unlearning: decreasing the likelihood of a given canary could decrease the likelihood of the template, and that of the other canaries, leading to over-estimation of the effectiveness of an unlearning method. A.2 Variations among different in-distributions sets To evaluate more directly how the number of repetitions of the same examples of the in-distribution sets affect the trade-offs, we train a second subject model, of weights θS′subscriptsuperscript′ _S θitalic_S′, on a training set S′ containing: • the training split T of the dataset of interest; • the same OOD forget sets as in S: F×1OODsubscriptsuperscriptOODabsent1F^OOD_× 1FOOD× 1, F×10OODsubscriptsuperscriptOODabsent10F^OOD_× 10FOOD× 10, and F×100OODsubscriptsuperscriptOODabsent100F^OOD_× 100FOOD× 100; • different in-distribution forget sets, made out of the same examples but with different frequencies: F×1′⁣InDsubscriptsuperscript′InDabsent1F _× 1F′ InD× 1 contains the same examples as F×100InDsubscriptsuperscriptInDabsent100F^InD_× 100FInD× 100 but repeated only once, F×10′⁣InDsubscriptsuperscript′InDabsent10F _× 10F′ InD× 10 contains the same examples as F×1InDsubscriptsuperscriptInDabsent1F^InD_× 1FInD× 1, and F×100′⁣InDsubscriptsuperscript′InDabsent100F _× 100F′ InD× 100 as F×10InDsubscriptsuperscriptInDabsent10F^InD_× 10FInD× 10. Regardless of the identity of the repeated set, we observe that the more repeats of the in-distribution sample sets, the higher the (average) generalized exposure before unlearning, as shown in Figure 12). We also do not observe a significant difference in their average generalized exposures. We did not investigate that effect on out-of-distribution canaries. Since they were sampled from a uniform distribution and should be interchangeable, we do not expect the identity of canary examples repeated the same number of times to influence the exposure. Despite the average exposure being the same, the distribution of the InD set perplexities are different. We illustrate the perplexities of the three sets of in-distribution examples on a model that was trained without them in Figure 13. The three InD sets have similar mean log-perplexities on the reference model, with differences in the spread of the distribution of their log-perplexities. Specifically, the mean and variance for F×1InDsubscriptsuperscriptInDabsent1F^InD_× 1FInD× 1, F×10InDsubscriptsuperscriptInDabsent10F^InD_× 10FInD× 10, and F×100InDsubscriptsuperscriptInDabsent100F^InD_× 100FInD× 100 is (41.95(41.95( 41.95, 1068.65)1068.65)1068.65 ), (42.30,1061.54)42.301061.54(42.30,1061.54)( 42.30 , 1061.54 ), and (42.94,1266.64)42.941266.64(42.94,1266.64)( 42.94 , 1266.64 ), respectively. We also plot the distribution of perplexities under the training set S′ in Figure 14 which shows the variation between the different InD sets. Subject Models (trained on the same OOD sets, but different InD sets) θSsubscript _Sθitalic_S trained on (F×1InD(F^InD_× 1( FInD× 1, F×10InDsubscriptsuperscriptInDabsent10F^InD_× 10FInD× 10, F×100InD)F^InD_× 100)FInD× 100 ) Figure 6: F×1InDsubscriptsuperscriptInDabsent1F^InD_× 1FInD× 1. θS′subscriptsuperscript′ _S θitalic_S′ trained on (FX×1′⁣InD(F _X× 1( F′ InDX × 1, FY×10′⁣InDsubscriptsuperscript′InD10F _Y× 10F′ InDY × 10, FZ×100′⁣InD)F _Z× 100)F′ InDZ × 100 ) Figure 7: FX×1′⁣InDsubscriptsuperscript′InD1F _X× 1F′ InDX × 1, same examples as F×10InDsubscriptsuperscriptInDabsent10F^InD_× 10FInD× 10. Figure 8: F×10InDsubscriptsuperscriptInDabsent10F^InD_× 10FInD× 10. Figure 9: F×1′⁣InDsubscriptsuperscript′InDabsent1F _× 1F′ InD× 1, same examples as F×10InDsubscriptsuperscriptInDabsent10F^InD_× 10FInD× 10. Figure 10: F×100InDsubscriptsuperscriptInDabsent100F^InD_× 100FInD× 100. Figure 11: F×1′⁣InDsubscriptsuperscript′InDabsent1F _× 1F′ InD× 1, same examples as F×100InDsubscriptsuperscriptInDabsent100F^InD_× 100FInD× 100. Figure 12: In- vs. Out-of-distribution sets. Trade-off between the generalized exposure (Exposure) and the task performance (BLEU score) when unlearning the subject models (θS,θS′subscriptsuperscriptsubscript′ _S, _S θitalic_S , θitalic_S′) at 45,000 steps, with OOD (canary) forget sets (F×1OOD,F×10OOD,F×100OOD)subscriptsuperscriptOODabsent1subscriptsuperscriptOODabsent10subscriptsuperscriptOODabsent100 (F^OOD_× 1,F^OOD_× 10,F^OOD_% × 100 )( FOOD× 1 , FOOD× 10 , FOOD× 100 ), and in-distribution forget sets (F×1InD,F×10InD,F×100InD)subscriptsuperscriptInDabsent1subscriptsuperscriptInDabsent10subscriptsuperscriptInDabsent100 (F^InD_× 1,F^InD_× 10,F^InD_% × 100 )( FInD× 1 , FInD× 10 , FInD× 100 ) for θSsubscript _Sθitalic_S (resp. (F×1′⁣InD,F×10′⁣InD,F×100′⁣InD)subscriptsuperscript′InDabsent1subscriptsuperscript′InDabsent10subscriptsuperscript′InDabsent100 (F _× 1,F _× 10,F^% _× 100 )( F′ InD× 1 , F′ InD× 10 , F′ InD× 100 ) for θS′subscriptsuperscript′ _S θitalic_S′). Figure 13: We show the distribution of the log-perplexities of the different sets of in-distribution examples used in our experiments. The perplexities were computed on the reference model that was trained on the language translation dataset (wmt-t2t, de-en), without OOD canaries or InD samples. We see clear differences, despite which the example frequency vs unlearning results were not different among these different groups. Figure 14: Distributions of perplexities. Perplexities of different sets of in-distribution examples under the subject model (before unlearning, post-unlearning, and when the exposure is before a threshold of 2222), and the reference model when the training set is S′ (different frequencies for the same set of examples). Columns left to right: in-distribution example perplexities when the subject model was trained by repeating these examples 1 time (left), 10 times (middle), 100 times (right). A.3 Difficulty and memorization results Figure 22 shows the difficulty and exposure trade-offs for the different model trained on S′ where we vary the number of repetitions of the same examples. Each sub-figure shows the unlearning of one set: F×1InDsubscriptsuperscriptInDabsent1F^InD_× 1FInD× 1, F×10InDsubscriptsuperscriptInDabsent10F^InD_× 10FInD× 10, and, respectively F×100InDsubscriptsuperscriptInDabsent100F^InD_× 100FInD× 100. The conclusion is that harder examples with more repetitions have slightly better trade-offs as it does not lead to “over-unlearning” (skewing the exposure to negative values). Similarly, we show that variations among different in-distributions sets yields similar observations regardless of the identity of the in-distribution set (Figure 22). Figure 15: Per-sample memorization vs. difficulty The more times a forget set repeats, the more its difficulty is correlated with its memorization. We compute the memorization of each sample in the InD sets based on their likelihood on the subject model (trained on all forget sets F…InDsubscriptsuperscriptInD…F^InD_…FInD…, before any unlearning) and their likelihood on the reference model. We average the likelihood of each sample over 3333 reference models θ−F∼⁢(S∖F)similar-tosubscript _-F (S F)θ- F ∼ A ( S ∖ F ), trained under three different seeds. Figure 15 plots the per-example difficulty and memorization for the θSsubscript _Sθitalic_S after unlearning on one of the forget sets. We use the definitions in Section 3.1 for difficulty and memorization. We find that there is a weak correlation between these two for InD examples that repeat once, but the correlation gets stronger as the number of times the samples repeats increases. Subject Models (trained on the same OOD sets, but different InD sets) θSsubscript _Sθitalic_S trained on (F×1InD(F^InD_× 1( FInD× 1, F×10InDsubscriptsuperscriptInDabsent10F^InD_× 10FInD× 10, F×100InD)F^InD_× 100)FInD× 100 ) Figure 16: F×100InDsubscriptsuperscriptInDabsent100F^InD_× 100FInD× 100. θS′subscriptsuperscript′ _S θitalic_S′ trained on (F×1′⁣InD(F _× 1( F′ InD× 1, F×10′⁣InDsubscriptsuperscript′InDabsent10F _× 10F′ InD× 10, F×100′⁣InD)F _× 100)F′ InD× 100 ) Figure 17: FX×1′⁣InDsubscriptsuperscript′InD1F _X× 1F′ InDX × 1, same examples as F×100InDsubscriptsuperscriptInDabsent100F^InD_× 100FInD× 100. Figure 18: F×1InDsubscriptsuperscriptInDabsent1F^InD_× 1FInD× 1. Figure 19: F×10′⁣InDsubscriptsuperscript′InDabsent10F _× 10F′ InD× 10, same examples as F×1InDsubscriptsuperscriptInDabsent1F^InD_× 1FInD× 1. Figure 20: F×10InDsubscriptsuperscriptInDabsent10F^InD_× 10FInD× 10. Figure 21: F×100InDsubscriptsuperscriptInDabsent100F^InD_× 100FInD× 100, same examples as F×10InDsubscriptsuperscriptInDabsent10F^InD_× 10FInD× 10. Figure 22: Difficulty vs. trade-offs. For each set of in-distribution examples, we cluster them by difficulty using the perplexities on the reference model. Examples with a higher perplexity are considered harder. The trade-off is computed for each unlearning step on the main model. Harder examples have a better trade-off between the unlearning effectiveness and the performance of the unlearned model. A.4 Distribution of perplexities at low exposure We note that the unlearning a number of steps may result in negative exposure, a sign of “over-unlearning” which skews the distribution of perplexities of the forget sets on the unlearned subject model compared to the distribution of perplexities of the forget sets on the reference model. We show what happens when we set the exposure threshold to 0.50.50.50.5 in Figure 23. The extent of this effect depends on the number of repeats of the forget set. Figure 23: Distributions of perplexities at low exposure. Perplexities of different sets under the subject model at the first unlearning step which results in an average exposure lower than a threshold of 0.50.50.50.5 (orange), and the perplexities under on the reference model (purple). In the top rightmost figure, we observe a phenomenon of “over-unlearning” (when the exposure becomes negative) which brings the two distributions of perplexities further apart. A.5 Relationship between relative exposure and generalized exposure The relative exposure metric is a more computationally efficient one, since it does not require access to a reference model. We want to study whether it is a good proxy for the generalized exposure metric. For this, we take the subject model trained on S and plot it together with the generalized exposure before unlearning (at the 45,0004500045,00045 , 000 training step) and after unlearning. We observe that the relative exposure is a good proxy for the generalized exposure (Figure 24). Figure 24: Relative vs. generalized exposure We find that relative exposure is a good proxy for the generalized exposure for in-distribution data. A.6 Effect of unlearning on similar points To investigate this, we find similar examples (the top-10101010) from the set of examples that repeat ten times (F×10InDsubscriptsuperscriptInDabsent10F^InD_× 10FInD× 10) to the forget sets F×100InDsubscriptsuperscriptInDabsent100F^InD_× 100FInD× 100 and F×1InDsubscriptsuperscriptInDabsent1F^InD_× 1FInD× 1. For simplicity, we compute the L2subscript2L_2L2-distance between the embeddings of each point in the forget sets on the reference model. Our similar sets consist of the union of the top-10101010 closest examples from F×1InDsubscriptsuperscriptInDabsent1F^InD_× 1FInD× 1 for all examples F×100InDsubscriptsuperscriptInDabsent100F^InD_× 100FInD× 100 and, respectively, F×1InDsubscriptsuperscriptInDabsent1F^InD_× 1FInD× 1. Concretely, this resulted in 424424424424 examples for the set of examples that repeat once, and 421421421421 for the set of examples that repeat 100100100100 times. We then measure the average generalized exposure on the similar set as we unlearn the forget set F×100InDsubscriptsuperscriptInDabsent100F^InD_× 100FInD× 100 and, respectively, F×1InDsubscriptsuperscriptInDabsent1F^InD_× 1FInD× 1, for 16161616 training steps. We plot the tradeoffs between exposure and performance on the forget set and on the similar sets in Figure 25. We can see that similar examples are unlearned as well by performing unlearning on the forget set, even before unlearning impacts the model’s utility. We want to see how unlearning the forget set also influences examples outside of the training set, i.e., the reference set RInDsuperscriptInDR^InDRInD. We use the same methodology as above, and pick the top-10101010 closest examples from RInDsuperscriptInDR^InDRInD to our two forget sets. This results in 571571571571 for F×100InDsubscriptsuperscriptInDabsent100F^InD_× 100FInD× 100 and 591591591591 for F×1InDsubscriptsuperscriptInDabsent1F^InD_× 1FInD× 1. To start with, the exposure of examples outside the training set is small. The effect of unlearning of the forget set on these examples’ exposure is unnoticeable, though we do observe a small decrease of exposure (up to 0.10.10.10.1 for the case shown in Figure 5). Similarly, we show that the effect of F×1InDsubscriptsuperscriptInDabsent1F^InD_× 1FInD× 1 on the closest reference examples is very small. For the reference set and the forget set that is repeated only once, we observe the same phenomenon: the exposure of the reference set is not affected by the unlearning of the samples in the forget set (Figure 25). We also validate our main observation of the effect of unlearning on similar points in the training dataset on a different subject model, trained on a training dataset S′. The training has the same forget sets but with different number of repetitions. Figure 25: In-distribution vs. similar in-distribution examples. Trade-off between the generalized exposure (Exposure) and the task performance (BLEU score) when unlearning the subject model on set F×1InDsubscriptsuperscriptInDabsent1F^InD_× 1FInD× 1 and F×100InDsubscriptsuperscriptInDabsent100F^InD_× 100FInD× 100 (left to right). Unlearning in-distribution examples affects the exposure of other similar examples from the training dataset (left-most two plots), while not affecting the exposure of unseen examples from the reference set (right-most plot). We can see that the examples in the reference set are not affected by unlearning. A.7 Related Work Unlearning benchmarks and evaluation metrics. (Lynch et al., 2024) propose eight distinct evaluation metrics that go beyond standard loss measures on the forget/retain set, and try to capture internal model changes, as well as the impact on downstream tasks. The authors measure robustness to jailbreaks and finetuning, other extraction techniques, undesirable side effects, etc. Shi et al. (2024) propose a new machine unlearning evaluation benchmark, MUSE, focusing on assessing 6 desired properties of unlearned models, such as verbatim memorization, scalability with forget sets size, etc. The TOFU benchmark paper (Maini et al., 2024) introduces a new task and dataset for evaluating specific training data unlearning in large language models. Jang et al. (2022) introduce an “extraction likelihood” metric for measuring unlearning quality in LLMs: they look at the average completion accuracy of a sequence of tokens when a varying length prefix was provided as a prompt. The authors also studied gradient ascent-based unlearning, and found that to be more effective when unlearning sequentially in batches rather than all at once. They also report differences in how easy it is to unlearn depending on the source of the forget set. While Jang et al. (2022) also points to differences in the effectiveness of unlearning between different forget datasets, they do not further explore how similar examples are affected by gradient ascent (as our work does). TOFU focuses on a “Task of Fictitious Unlearning” where models are trained on fictional author profiles and then must unlearn a subset of those profiles. The paper provides a dataset of these profiles, metrics to assess unlearning efficacy, and baseline results from existing unlearning algorithms. All of the work above aims to identify and assess desirable properties of unlearned models for general or specific tasks, but do not directly work with ε−limit-from -ε -unlearning definition as in Definition 2.1. The way to measure ε−limit-from -ε -unlearning as proposed in our work can be viewed as complementary to these other approaches. Barbulescu & Triantafillou (2024)444This work was carried out independently and concurrently with our work. leverage memorization information for unlearning by proposing an unlearning method that differentiates textual sequences based on their memorization level, as in (Jang et al., 2022). The memorization in this work is captured by tracking reconstruction of the exact tokens in a sequence, which is different from the definition used in our work. An unlearning algorithms is “successful” if memorization of a particular sequence of interest is reduced. Their work also introduces an MIA-like evaluation inspired by the neighborhood MIA concept. Memorization. Several studies have explored different facets of memorization in LLMs, including verbatim memorization (Lehman et al., 2021; Ippolito et al., 2022), membership inference attacks (Shokri et al., 2017; Nasr et al., 2018; Salem et al., 2018; Choquette-Choo et al., 2021), exposure (Carlini et al., 2019), and extraction attacks (Carlini et al., 2021, 2022b). These works provide valuable insights into the extent and nature of information leakage in LLMs. Hayes et al. (2024) highlighted the limitations of inexact unlearning evaluation methods like membership inference attacks. The authors show that current evaluation metrics for approximate unlearning can be misleading, creating a false sense of security. They call for more rigorous testing and a deeper understanding of how unlearning affects different data points.. Removing information in Large Language Models. Patil et al. (2023) consider information removal from the weights of a language model, which should protect against white box attacks. The authors focus on model editing techniques (Meng et al., 2022b, a), and show that even after editing the model to remove some sensitive information, they were still capable of extracting this information in a large fraction of cases. This paper also investigates how editing sensitive information affects the accuracy on neighbouring points using this information. They use the change of accuracy in the neighbourhood, a metric borrowed from (Meng et al., 2022b), to demonstrate that in many cases sensitive information was not properly removed. Memorization and membership inference attacks. Membership inference attacks (MIAs), first introduced for classification tasks, aim to evaluate to what extent a given datapoint can be traced back to be from a training set or not (Shokri et al., 2017). MIAs are now widely adopted in unlearning literature, as well as for studying memorization. Recently, membership inference attacks have been proposed for language models such as text classification tasks (Gu et al., 2023), (Mattern et al., 2023), and masked language models (Mireshghallah et al., 2022). The membership inference information can serve as a step towards extracting the training data. Carlini et al. (2019) showed that personal information can be extracted by generating numerous sentences from pre-trained language models and performing membership inference. Nakamura et al. (2020) considered an adversary with some prior knowledge of the patient that could employ a pre-trained masked BERT model to predict the masked personal information in the input clinical data. Lukas et al. (2023) showed that PII can be extracted from these models. Besides attacks, several mitigation strategies have been proposed for large language models such as ad-hoc practical defenses (Lee et al., 2021), as well as based on the rigorous framework of differential privacy (Ponomareva et al., 2022). Deferentially private training makes the model indistinguishable to an adversary (or user querying the model) up to one data record, or a fixed size group (group privacy). However, in unlearning, requests to delete samples may come for a batch of samples of varying size, perhaps even hundreds of these. As a result, differential privacy is not enough to support unlearning requests across all applications.

Instant research discovery